A Vibe Check Is Not an Eval

A Vibe Check Is Not an Eval

Read five AI outputs and decide they look good, and you have run a vibe check, not an eval. An eval has a known right answer, decided in advance, and a rule for what counts as a pass.

AAA runs two real versions of this discipline. One is a held-out test. The other is a constraint that makes a wrong answer harder to produce in the first place.

The Cheapest Real Eval AAA Runs

The research script behind AAA’s 25-restaurant Field Note needed to be trusted before its findings went out under AAA’s name. Three cases got chosen first, before a single real restaurant was checked.

AAA’s own site was expected to show clean AI-crawler access. Wikipedia was expected to show the same. A site already publicly known to block AI crawlers was expected to show exactly that. All three matched what was already known to be true.

That order is what makes it an eval and not a vibe check. The right answer got decided first. The check ran after.

A Constraint Is an Eval Too, Just Cheaper

A held-out test runs once, before launch. A constraint runs every time, on every output, forever. Active GEO and Active Pulse use one directly inside the prompt that generates every report.

Every recommendation the model writes has to carry a related_to field. That field has to name a real finding or dimension already in the same report, exactly, or the recommendation does not match anything real in the output.

The second version cannot exist without a real finding behind it. That is the constraint working, not a writing style.

Where Else This Pattern Applies

Both techniques generalize past AI reports. None of the four examples below are real AAA client work. They show where the same discipline applies outside GEO and Pulse.

• Support. A held-out set of ten real past tickets with an agreed correct response, run against any new model version before it replaces the old one.
• Legal. A constraint that makes a drafting tool cite a real clause number for every claim it makes, the same way a recommendation must cite a real finding.
• Recruiting. A held-out set of resumes with an agreed shortlist decision, checked against a new screening prompt before it goes live.
• Marketing. A constraint that blocks a generated headline from shipping unless it names a real product feature, not a generic claim.

Each example above is illustrative, not a client engagement. The restaurant-audit validation and the related_to constraint are the two instances of this pattern that are real and running today.

What This Article Proves

This piece has its own Download PDF button. It was generated and rendered to a real PNG with the same pipeline described in the Active Ship piece before it shipped. Same check, different article.

Active Eval is the name for this practice area: a known answer decided in advance, or a rule enforced on every output, not a vibe check. The next capability up is Active Prompt, which depends on Eval to know whether a prompt change actually helped.