The Hardest Problem in AI Today Isn’t Generation, It’s Evaluation
The Hardest Problem in AI Today Isn’t Generation, It’s Evaluation
Over the past few years, the AI industry has been consumed by a single question: How do we build better models? Progress was measured by larger parameter counts, longer context windows, lower latency, and increasingly capable reasoning. Every major breakthrough focused on generation—making AI systems that could write more fluently, reason more effectively, and perform increasingly complex tasks. It was an exciting race, and one that fundamentally changed what software is capable of.
That race, however, is beginning to mature. While foundation models continue to improve, the conversations I’m having with enterprise AI teams have shifted in a noticeable way. Very few people are asking whether they can build an AI agent anymore. Between powerful frontier models, open-source frameworks, and rapidly improving developer tooling, building an agent has become remarkably accessible. Instead, the question has become much more practical and, in many ways, much more difficult: How do we know whether an agent is actually good?
I think this marks an important transition for our industry. Every engineering discipline eventually reaches a point where the challenge is no longer creating something, but reliably measuring its quality. Manufacturing didn’t become transformative because factories learned how to produce products; it became transformative because they learned how to produce products consistently. Software engineering didn’t mature because developers wrote more code; it matured because they built testing frameworks, deployment pipelines, and quality assurance processes that made software dependable at scale.
AI is reaching that same moment.
The challenge is that AI systems don’t behave like traditional software. Conventional software is deterministic. If a function is given the same input, we expect the same output. When something goes wrong, the failure is usually objective and relatively easy to identify. An API returns an error. A request times out. A unit test fails. Engineers have spent decades building tools to detect and diagnose these kinds of problems because correctness can often be defined with precision.
AI systems operate in a fundamentally different way. A language model can produce multiple responses that are all technically correct, yet only one actually helps the user accomplish their goal. An agent may retrieve the right information but present it in the wrong context. It may complete nine steps of a workflow perfectly before making a poor decision on the tenth. It may sound thoughtful and confident while quietly misunderstanding what the user was trying to achieve all along.
These aren’t software failures in the traditional sense. They’re failures of judgment.
That distinction becomes even more important as we move from chatbots to autonomous agents. Evaluating a chatbot answering a single question is relatively straightforward. We can compare its response against an expected answer, check for hallucinations, measure factual accuracy, or evaluate whether it followed a particular instruction. Those metrics remain useful, but they become increasingly incomplete as agents take on more responsibility.
An enterprise agent isn’t simply generating text. It’s making decisions over the course of an interaction. It decides when to retrieve information, which tools to call, how to interpret ambiguous requests, and how to adapt when a conversation changes direction. Success depends less on producing an eloquent response than on consistently making good decisions throughout an entire workflow. A customer doesn’t judge an agent based on whether one answer was well written; they judge it based on whether the task they came to accomplish was actually completed.
This is why evaluation has become so much more complicated than generation. Measuring language quality is one problem. Measuring behavior is an entirely different one.
The industry’s first response has been to use AI itself as the evaluator. LLM judges have quickly become one of the most promising developments in AI evaluation because they make it possible to assess enormous volumes of conversations that would be impossible for humans to review manually. Instead of asking people to inspect every interaction, another language model evaluates qualities such as relevance, factuality, policy adherence, or task completion. It’s an elegant solution that dramatically increases the scale at which evaluation can happen.
But almost immediately, another question emerges.
How do we know the evaluator is right?
An LLM judge is still a probabilistic model. It has biases, blind spots, and inconsistencies of its own. Two evaluators can disagree about the same conversation. The same evaluator can produce different judgments under slightly different conditions. In highly specialized industries, an experienced domain expert may reject an answer that an automated evaluator confidently approves. Before long, the challenge is no longer evaluating the agent—it’s evaluating the evaluator.
In many ways, we’ve recreated the original problem one level higher.
This is why I believe the next frontier in AI isn’t simply better generation. It’s better measurement. More specifically, it’s developing evaluation systems that can be trusted because they’ve been calibrated against human expertise rather than operating as generic judges. The organizations deploying AI successfully are increasingly recognizing that evaluation is not a one-time exercise or a benchmark run before launch. It’s an ongoing process of defining quality, refining standards, collecting feedback, and continuously improving how those standards are applied.
Human judgment plays a central role in that process. Rather than disappearing, subject matter experts become even more valuable because they establish what “good” actually means within a particular business context. An insurance company, a financial institution, and a healthcare provider each operate under different policies, regulations, and expectations. A universal evaluator cannot fully capture those nuances. Reliable evaluation comes from aligning automated judges with the standards that matter to the organization deploying the system.
I suspect this shift will shape the next phase of AI development more than any incremental improvement in model capability. Larger models and longer context windows will continue to matter, but they won’t solve the problem organizations increasingly care about most. Enterprise teams don’t simply want more capable agents—they want agents they can trust. Trust doesn’t come from impressive demos or benchmark scores. It comes from confidence that an agent will behave correctly across thousands of real-world situations, including the ones no one explicitly anticipated.
Ultimately, progress in engineering has always depended on measurement. We improve what we can reliably evaluate, and we trust what we can consistently measure. AI is no different. The remarkable advances in generation have brought us to the point where building intelligent systems is no longer the primary obstacle. The harder challenge—the one that will define the next generation of AI infrastructure—is determining whether those systems are actually performing as well as we believe they are.
Generation brought us this far. Evaluation is what will determine where we go next.
How is your team evaluating AI agents today? What’s been the biggest challenge?