The AI release that stopped relying on gut feel

The team has an AI feature that people want to use. It works well in the demo and usually handles the examples everyone knows. Then a prompt changes, a model is replaced, or a new document source is added. Some answers improve. Others get worse. Nobody can say by how much or whether the release is safe to ship.

This is where many AI products get stuck. The model is not obviously broken, so there is no single bug to fix. Quality is discussed through screenshots and recent memories. The loudest example in the room sets the direction.

The first job is not to add a dashboard. It is to turn good into a set of decisions the team can repeat.

We begin with real examples. Some come from production traces, some from support conversations, and some from the edge cases the team already worries about. Together they need to cover the important jobs of the product, not every possible question a user could ask.

Each example gets a reason for being there. A correct answer may need to use the right source, preserve a required fact, refuse an unsafe request, follow a format, or hand the task to a person. Those reasons become a failure taxonomy the whole team can understand.

This changes the conversation. Instead of asking whether a new version feels better, the team can see which types of work improved, which regressed, and where the evidence is still thin.

Not every quality check should be automated. Some rules are deterministic. A required field is present or it is not. A source was cited or it was not. Other checks need model-based scoring, and the most important or ambiguous cases still need a person.

We design those layers together. Automated evaluators handle the repeatable checks and help the team cover more examples. Human review calibrates the parts where context matters. When the two disagree, the workflow records the disagreement instead of hiding it inside one average score.

The result is an evaluation system tied to product decisions, not a collection of numbers with no owner.

The product then needs a record of what changed. Prompts, model settings, retrieval behaviour, and application versions should be visible beside the evaluation result. Production traces need enough context to show where a failure came from and which customers or workflows it affected.

We connect those records to a release check. The check can block a clear regression, request review when the evidence is mixed, or allow the release when the agreed thresholds are met. The team chooses those thresholds. Estma helps make them explicit and technically enforceable.

Production completes the loop. New failures enter the review queue, useful examples become regression tests, and alerts point to a decision someone can make.