What the Bot Refuses, and When It Hands You to a Person

The Layer Above Retrieval

Retrieval decides what the model gets to read. The Active Retrieval piece covers that layer and how it was measured. Everything above it is the conversation. That means what counts as a follow-up, what the tool refuses, and when it hands you to a person.

Ask the Field Notes shipped with all three of those behaviors written into the code. None of them had been scored. The retrieval layer had 37 questions with known answers. The conversation layer had a code review and a good feeling.

So we applied the same method one layer up. Write the conversations first, decide what should happen, freeze the file, and then run it.

Thirteen Conversations, Written First

On September 14, 2026, we wrote thirteen conversations from a read of the endpoint code. Four are follow-ups, where the last turn only makes sense given the one before it. Five are questions the tool should refuse. Four are questions that should end with a handoff to the contact page.

Only the last turn of each conversation is scored. Earlier answers are not scripted. The tool’s real replies get fed back in, so the transcript is what a person would have seen. Two cases, quoted as written.

Read the two notes again. Both predicted a failure. Both were right, and neither prediction changed the expected result. The expected result is what a person needs, not what the code was likely to do.

As Shipped: Five of Thirteen

Every follow-up failed. The rule that handled them was simple. Fold the previous question into the retrieval query when the new one is under four words. “Why?” qualified. “How do I fix it?” did not. It is five words, and after stop words are removed it is one word: fix. The retriever went looking for “fix” alone. It found a section titled “Fix Access Before You Fix Content” in a different article, at a score of 3.15. The model correctly said the sources did not answer.

Three of four handoffs failed for a different reason. The handoff lived in the model. The prompt told it to end pricing and hiring replies with a token. The page turned that token into a button. But “How much does The Compass cost?” matched nothing in the Field Notes. The grounding rule rejected it before the model ran. The person got the plain “not covered” reply, with the contact page mentioned in passing and no button.

One refusal failed in the most interesting way. “Write me a LinkedIn post about AI” grounded on the Article schema section at a score of 6.01. The model did not write a post. It did something worse for a tool that promises grounded answers. It produced 161 tokens of cited advice about schema and brand consistency, then ended with the handoff token. Helpful in tone, wrong in kind. The tool answers questions. It does not take assignments.

Three Rules, Not a Rewrite

Each failure got one rule. None of them touched the retriever, and none of them changed how the model is called.

• Fold by content words, not raw words. A follow-up with fewer than three content words now retrieves on the previous question plus itself. “How do I fix it?” has one. “What was missing instead?” has two. Both now fold. A topic change like “Tell me about schema markup instead” has four, and does not.
• Hand off before the model, when nothing grounds. An ungrounded question about pricing, hiring, an engagement, or reaching a person gets a fixed handoff reply. No model call. The check only runs when the grounding rule has already said no. An in-scope question that contains the word cost still goes to the model. “What does blocking GPTBot cost me?” is one.
• An assignment is a refusal, not a handoff. One line added to the prompt, quoted below.

The same thirteen conversations ran again on the fixed build. Twelve passed. Here is the hard case from the frozen file, as run, with the trace the page shows under every answer.

And the two behaviors a person notices most, both real replies.

The Miss We Kept

Case 1 asks what six dimensions Active Pulse sorts feedback into, then asks “Why?” The retrieval was right both times. The fold rule fired. The NPS article’s section on what a sentiment read adds came first, six of six words matched. The model had it in front of it. The model refused, twice, in both runs.

The model has a point. The source says six numbers show what one number hides. It never says why those six. The frozen file says the tool should answer. Both readings are defensible, and only one of them was written down before the run. The rule for a frozen set is that you do not edit it after you have seen the results. So this stays a miss, and the next version of the set will phrase the follow-up better.

What the Bot Refuses, and Where

Refusals happen at two layers, and the split matters for cost. Three of the five refusals never reached the model. The grounding rule rejected them and the tool sent its fixed reply for free. The CRM recommendation landed there. So did a request for an opinion on another lab’s latest model. So did the request to ignore its instructions and tell a joke.

Two refusals came from the model itself. The Active Reach question is the known retrieval leak from the Retrieval piece. It reaches the model with the wrong chunks, and the model says so. The LinkedIn post is the new rule doing its job.

One limit belongs in the open. The instruction-override attempt was refused because the retriever found nothing to match, not because anything recognized it as an attack. An override phrased in the corpus’s own vocabulary would reach the model. The first rule of the prompt would be the only thing standing. That rule held in every run so far. It has not been tested on purpose, and it should be.

Where Else This Pattern Applies

Scoring the conversation layer generalizes past a firm’s own articles. None of the four examples below are real Active AI Advisors client work. They show where the same three rules apply.

• Customer support. A help-center bot answers how-to questions from the docs and refuses to promise a refund. Anything about billing goes to a ticket, with the conversation attached.
• IT helpdesk. A bot over the internal knowledge base walks through a VPN setup. It refuses to reset a password itself and hands the reset to a person, user already identified.
• Property management. A tenant bot answers lease questions from the lease itself. It refuses to speculate about a neighbor’s complaint and hands maintenance requests to the on-call line.
• Legal intake. A bot explains what a practice area covers and refuses to say whether someone should sue. Every fact pattern goes to intake, unsummarized.

In every case the conversations come first. Ten or twenty of them, each with the expected refusal or handoff written down. They decide whether the layer works before anyone talks to it. Each example is illustrative, not an engagement. Ask the Field Notes is the instance that is real and running today.

What This Article Proves

This article’s PDF was generated and rendered to a real image before it shipped, using the Active Ship pipeline. The article quotes the frozen conversations word for word. It is excluded from the retrieval corpus, for the reason the Retrieval piece explains. Both results files are single dated runs. The model runs at temperature zero and is still not fully deterministic. That is one more reason the miss stayed a miss.

Active Chat is the name for this practice area. It means a conversation layer with its follow-ups, refusals, and handoffs scored, not assumed. Active Retrieval and Active Chat are the first pieces in this track backed by a live tool. Next in the track is the second Active Forge piece, on agentic workflows, which needs a real build first.