One of the biggest process shifts for teams building AI-powered apps is baking in time to create a golden dataset before building the system.

What is a golden dataset? In a nutshell, it’s a curated set of real user questions paired with verified correct (or “good”) answers. The questions your users will actually ask, matched with what a great response looks like.

It feels unnecessary. But there are two important benefits:

1️⃣ It forces you to define what success looks like for the system.

Writing down these golden scenarios creates alignment about what the system is meant to do, and what good looks like. For LLM-powered apps, this is surprisingly fuzzy and fluid, which means developers are at risk of building a app that is completely useless. Getting concrete forces agreement and sets expectations.

2️⃣ It lets you measure progress (and regressions)

AI systems usually improve over time, and tweaks can have unexpected impact. You need to know that your time is improving the system (I just changed an embedding model for a RAG agent to one that should in theory perform much better, but performance degraded on average).

Here’s an analogy: Imagine you’re a coach trying to improve a basketball player’s free throw percentage, but you’ve decided not to track whether shots go in. You can watch them shoot, tweak their form, give feedback, but you never actually count makes vs. misses. After a month of coaching, someone asks “are they better now?” and you can only shrug.

That’s what evaluating an AI system without a golden dataset looks like. The golden dataset converts subjective “this seems better” into quantifiable metrics: success went from 0.72 to 0.82, answer faithfulness improved by 6%, etc. Now you can actually engineer improvements instead of guessing.

Why do teams resist this?

Because it requires answering hard questions upfront: What counts as a “good” answer? How do we handle ambiguous queries? What’s out of scope?

But skipping them doesn’t make the questions go away, it just means you’ll discover the disagreements later, after you’ve already built the wrong thing. The golden dataset becomes your evaluation set - the scorecard you run every time you make changes.

So here’s the challenge: start with 8 scenarios. That’s it!

I’ve put together a template you can use to get started, check it out here.