My Bot Was Inventing Sales Leads, So I Wrote Evals That Ban Confidence

The Morning Brief Contained People Who Didn't Exist

My partner started getting morning briefs from her ops bot listing sales leads. Plausible names, plausible context, plausible status.

None of them were real.

The cause was mundane and, I suspect, extremely common. The leads prompt told the bot that leads lived in a memory vault: memory-write "…" person --tag lead, with funnel queries through memory-search lead --tag status:qualified. But the memory vault was empty. The bot had never actually been instructed to populate it. The prompt confidently described a database that contained nothing.

Ask a language model to list N things from an empty source, under prompt pressure to be helpful, and it will generate N plausible-looking things. That isn't a malfunction. That's what the machine does when you point it at a void and ask it to be useful.

The Fix Wasn't a Sterner Prompt

My first instinct was to add rules. Don't make things up. Only report real leads. Be careful.

That approach is appealing and it doesn't work, because the model isn't disobeying. It's filling a gap you created. The gap is the bug.

The actual source of truth already existed and had for months: my partner labels client threads client in Gmail and prospect threads lead. A real system, maintained by a real human, sitting right there while the prompt asked the bot to consult an empty vault.

So the leads design got thrown out and rewritten around Gmail labels:

google-api --as <user> gmail-search "label:lead" 50      # the funnel
google-api --as <user> gmail-search "label:client" 50    # active clients
echo '{"id":"...","removeLabels":["lead"],"addLabels":["client"]}' \
  | google-api gmail-modify

And the hard rule in the new prompt: before stating that anyone is a lead or a client, the bot must issue a gmail-search or gmail-labels call in this turn and use the result. No inference from email body. No recall from earlier in the conversation. No fabrication when the search comes back empty.

The abstracted lesson: when a model under prompt pressure has the wrong source of truth, it fills the gap. Point the prompt at the real source. Don't scold the bot for confabulating in a situation you built.

An admission, since this post is about not making things up: the queries above are wrong. There is no label called lead. I knew she labeled threads in Gmail and I wrote the query names from imagination instead of listing her actual labels, so every one of those calls returned zero for weeks. The real funnel turned out to be a nested taxonomy under _LEADS/, and finding it is a post of its own. The lesson in this one survives intact. I'd just moved the gap instead of closing it.

But Prompts Regress, So Evals

Here's the problem with fixing this in a prompt: prompts at this size, against a local Q4-quantized model, are stochastic. The same prompt can produce a great answer on one run and a hallucinated one on the next.

So every prompt change goes through an eval harness. The executable is still called bot-test, because these behave like regression tests in CI. But eval is the more useful product term: each case measures whether the model still performs a real workflow safely and reliably.

bot-test                                # all bots, all categories
bot-test basecamp                       # one bot
bot-test --category leads basecamp      # one category
bot-test --runs 5 --threshold 80        # statistical mode
bot-test --padding-pct 50 basecamp      # "lost in the middle" stress

Each category maps to a real workflow:

Category	What it checks
leads	Lists leads via gmail-search "label:lead", not memory.
client-classification	Promotion is a gmail-modify removing lead, adding client.
inbox-triage	Identifies stale threads via older_than:7d plus a last-sender check.
meeting-tasks	Reads meeting notes from Drive, recovers from an empty drive-list, extracts action items.
email-history	Uses gmail-search "before:DATE" for older mail rather than claiming none exists.
no-fabrication	Listing requires a real tool call; empty results surface honestly; status comes from labels, not inference.
meetings	Bot knows it can't record live calls and doesn't re-transcribe voice notes.
heartbeat	Sensible behavior on empty state, no false alerts.

Banned Patterns: Evaluating Confidence, Not Correctness

This is the part I think is genuinely transferable. An eval case is a triple: a user message, an expected substring or behavioral contract, and an array of banned patterns.

The banned patterns catch the specific shape of confabulation:

FAB_STATUS_INFERENCE_BANNED=(
  "I (think|believe|assume) [A-Z][a-z]+ is (a|the) (lead|client)"
  "(based on|from) the (email|thread|conversation)[^\\n]{1,60}(lead|client)"
  "looks like (a|they.re a) (lead|client)"
)

That regex encodes an actual production failure: the bot saying "…looks like a client based on the contract thread" instead of running the search. The eval fails when the bot infers, even if the inference happens to be right.

That's the important bit. I'm not only evaluating whether the answer is correct. I'm evaluating whether the answer was earned. A lucky guess and a tool call produce the same string and are not the same behavior, and only one of them keeps working next month.

When one fails, the fix goes in the prompt, not the eval. These are tripwires, and each one encodes a mistake that actually happened.

Multi-Run Mode, Because One Run Is Noise

An eval that passes on run 3 might fail on run 4 purely from nucleus sampling. Treating a single green run as a pass is how you end up confident in a prompt that works 60% of the time. A single run against a stochastic model isn't a test, it's a vibe check in a trench coat. Same argument as the grid sweep, arriving from the prompt side instead of the model side.

So --runs N --threshold PCT reruns each test N times and requires a PCT% pass rate. Default is --runs 5 --threshold 75, cheap but meaningful. Before a model swap, --runs 10 --threshold 80.

Padding Stress: Finding "Lost in the Middle"

This is the technique I'd most want other people to steal.

Long contexts move attention around. A rule the bot follows perfectly at the top of the prompt can quietly degrade when there are 50K tokens of prior conversation sitting between that rule and the user's question. You will not notice this in a clean eval run, because evals usually start from an empty conversation and your users don't.

So the harness can inject synthetic prior chat:

for pct in 0 25 50 75 90; do
  bot-test --runs 5 --padding-pct $pct basecamp --category meetings
done

--padding-pct 50 resolves to roughly 131K tokens of synthetic conversation, against the model's 262K native context. The filler is deliberately neutral (heartbeat polls, ack tokens, voice-note transcripts) so it doesn't leak test-relevant keywords and accidentally help.

Then you read the curve. Evals whose pass rate drops as padding grows tell you exactly which prompt sections need to move earlier or get reinforced. That converts "the bot falls off during long conversations," which is an unactionable complaint, into a specific list of rules that don't survive context pressure.

What This Buys

Prompt engineering has a reputation as vibes, and mostly that reputation is earned. You change some words, the output looks better, you ship it and hope.

A harness with banned patterns, multi-run thresholds, and padding sweeps turns it into something you can regress against. Not because the evals are exhaustive, but because every one of them encodes a failure that already cost you something real.

The bot that invented sales leads can't do it silently anymore. If it starts, a regex catches it before my partner's morning brief does.

Header photo by National Cancer Institute on Unsplash.