What happened
I shipped an AI support system for the people who had bought my job search automation. 13 agents, grounded in the actual workflow files, answering questions on demand so I wouldn’t have to.
The next morning it looked perfect. No errors. No crashes. Every question answered.
Then I read the daily summary email I had set up. One customer had burned through all 30 messages he had, in a single session. The agents had answered every one of them confidently, about a file they had never seen.
He set his system up wrong following those instructions. He never came back.
I found out because 30 was his ceiling and 30 seemed like a lot. The conversation was stored, so I retrieved it and had it read back to me. Nothing in the system told me. The report I had built arrived on time, was accurate, and didn’t contain the failure.
What was actually missing
I had scope. Every agent had a defined set of things it was allowed to answer, and the system refused questions outside it. That refusal fired during his session and showed up in the summary as a healthy signal.
I had verification. A daily report, built on purpose, arriving every morning.
What I did not have was anything in between. No point where an agent that could not ground a claim in a file it actually had was required to stop. No person who got summoned for that. Nothing the customer saw that told him the answer he was reading had no source behind it.
Scope is a boundary you draw before the agent runs. Verification is a check you run after it finishes. Both were in place. The failure happened in the middle, while the system was working, and there was nothing there.
What it cost
The customer built his system wrong. He followed instructions that sounded authoritative and had nothing behind them, and the setup he ended up with did not work.
He may have stopped trusting the product. I don’t know why he didn’t come back, and I’m not going to claim I do. But he paid for a system, asked it 30 questions, got 30 confident answers, and the thing he built from them failed. That is a reasonable place to walk away from.
And this was one customer on one morning. The same system was live for everyone who had bought it. Unverifiable instructions carrying the authority of a support agent, going out at whatever volume people happened to use it. I caught this one because 30 messages in one session was strange enough to look at. I wouldn’t have caught a customer who asked three questions and quietly got one wrong.
The shape of that failure does not change with size. Put the same three gaps inside a company with a real support organization: no required grounding, nothing that summons a person, and no way for the user to tell a sourced answer from an invented one. Then run it at ticket volume. Every one of those answers carries the authority of the company that sent it, and the layer where it breaks is invisible from outside for the same reason mine was. What scales is the bill, not the failure.
15 questions
These are the questions I ask now. The groups are not equal, and that is the point.
Before the agent runs
1. What is this agent allowed to do without asking, and who decided that?
2. Which of those actions cannot be undone?
3. For the actions that cannot be undone, what grounding does the agent need before it acts?
During, while it runs
4. When the agent cannot ground a claim in a source it actually has, does the agent stop and say so, and is that stopping behavior tested?
5. When the agent stops, who is summoned, and can that person both authorize and repair the issue?
6. Does the summoned human receive enough to act on without reconstructing the session?
7. Is the stop respected, or is it approved without review?
8. When the summoned human is not available, does the user learn that, and when help will arrive, before deciding how to proceed?
9. Throughout, does the user keep working, know what the agent can and cannot do, and decide whether to summon the human now?
After it finishes
10. How does anyone know this succeeded?
11. How does anyone know this failed, and does that arrive before or after the output is used?
12. Is the success check separate from the agent that did the work?
13. Who acts on the output, and can that person tell whether anything checked it?
14. Can this act be reversed, and is reversal something the system does or something a person has to do manually?
15. When it fails silently, what surfaces it, and how long does that take?
Why the groups are uneven
Three questions in the before section, six during, and six after. That imbalance is not an accident of drafting.
Before is cheap because it is planning. You are deciding what an agent may do while nothing is running and nothing is at stake, and you can change your mind for free.
After has six because that is where the cost arrives. Detection, reversal, and the distance between a thing failing and anyone finding out. Most teams build something here, even if all it measures is throughput.
During also has six, and that is the part worth sitting with. The middle of the flow carries exactly as much weight as the end, and it is the section almost nobody builds. Every one of those six requires something to exist at runtime: a tested stop, a person who can be reached, a handoff that carries context, a review that means something when it happens, a signal when that person isn’t there, and a user who isn’t left staring at a spinner. Scope is common. Monitoring is common. The gap is here.
A note on question 4: a prompt instruction telling the agent to stop is a request, not a guarantee. It only counts if you have built the condition that should trigger it and watched what the agent actually did.
A note on question 7: a checkpoint that is approved every time it fires is not a checkpoint. If you removed it and the outcomes would be identical, it was a signature line, not a review.
A note on question 8: the mechanism differs by context. For one person it can be as simple as a switch. For a team it could look like an on-call rotation. The requirement is the same either way. The user should learn whether help is coming, and when, before deciding how to proceed.
Scoring my own system
Here is that support system against all 15 questions, as it stands today. Every mark carries a line of evidence. A mark without one is a guess.
Five marks: Pass, Partial, Fail, Not visible from outside, for questions that cannot be answered without access the reader does not have, and Stated, not verified, used only when auditing a system I did not build, where the company publicly claims a behavior I have no way to confirm.
Before the agent runs
3 questions
PASS
1. What is this agent allowed to do without asking, and who decided that?
Component: Agent prompt definitions, one per agent, with the scope guard prepended to all 13.
Evidence: Each of the 13 agents has a defined answer surface. I decided it, and it is written down.
PASS
2. Which of those actions cannot be undone?
Component: The response stream itself. Once a chunk is sent to the buyer there is no recall.
Evidence: Answering is the act, and once a customer reads an answer and builds from it, it cannot be taken back.
FAIL
3. For the actions that cannot be undone, what grounding does the agent need before it acts?
Component: Specialist system prompts. The workflow files were referenced, but no rule required a claim to trace back to one.
Evidence: Agents had the workflow files available but nothing required a claim to be traced to one before answering.
During, while it runs
6 questions
PARTIAL
4. When the agent cannot ground a claim in a source it actually has, does the agent stop and say so, and is that stopping behavior tested?
Component: Hedge phrase detection and the Sonnet to Opus retry.
Evidence: Out-of-scope questions were refused and that fired correctly. A hedge cascade existed for low-confidence answers but was never tested, and it never fired on ungrounded in-scope claims.
PARTIAL
5. When the agent stops, who is summoned, and can that person both authorize and repair the issue?
Component: The escalation cascade: hedge, stronger model, refine request, email to me.
Evidence: An escalation path existed: hedge, then a stronger model, then a request to refine, then an email to me. It never fired, because it triggers on an agent reporting low confidence, not on a missing source. I can both authorize and repair, and did, but only because I went looking.
PARTIAL
6. Does the summoned human receive enough to act on without reconstructing the session?
Component: The escalation email body builder, which carries buyer identity and the last 10 messages.
Evidence: The escalation email carries buyer identity, the last 10 messages, and the hedged answer. It did not fire here, so I retrieved the stored conversation and reconstructed the session myself.
FAIL
7. Is the stop respected, or is it approved without review?
Component: None. No approval step exists in the request path. Answers stream straight to the buyer.
Evidence: There was no review step to respect. Nothing paused for a person, so nothing could be approved or refused.
FAIL
8. When the summoned human is not available, does the user learn that, and when help will arrive, before deciding how to proceed?
Component: None. No availability state exists for me, so there is nothing the system could show him.
Evidence: I am the only person who can authorize or repair, and the system had no way to tell him whether I was reachable. He got no signal either way.
FAIL
9. Throughout, does the user keep working, know what the agent can and cannot do, and decide whether to summon the human now?
Component: None. No user-initiated summon exists. The only interruptions are ones the agent decides to make.
Evidence: He kept working, which was the problem. Nothing told him the answers had no source, and he had no way to reach me from inside the session.
After it finishes
6 questions
FAIL
10. How does anyone know this succeeded?
Component: The daily digest, which is the only success instrument. It counts messages, buyers, scope refusals, and tokens. Nothing in the system, and no person, checked whether an answer was correct.
Evidence: Messages answered without error counted as success. That is throughput, not correctness.
FAIL
11. How does anyone know this failed, and does that arrive before or after the output is used?
Component: The digest cron, which runs once a day against the previous day’s logs.
Evidence: The daily summary arrived the next morning. He had already built from the answers.
PARTIAL
12. Is the success check separate from the agent that did the work?
Component: Daily summary generator, runs outside the agents.
Evidence: The summary is generated outside the agents, but it reports what they logged about themselves.
FAIL
13. Who acts on the output, and can that person tell whether anything checked it?
Component: The rendered answer. It shows which specialist replied and nothing about grounding, so a sourced answer and an invented one look identical to the buyer.
Evidence: The customer acted on it. A grounded answer and an invented one looked identical.
FAIL
14. Can this act be reversed, and is reversal something the system does or something a person has to do manually?
Component: None. No reversal path exists. The correction was an email I wrote by hand.
Evidence: Not reversible. Repair was manual: I wrote and sent a correction email myself.
PARTIAL
15. When it fails silently, what surfaces it, and how long does that take?
Component: The digest, plus the gap log, which only records when an agent hedges.
Evidence: The daily digest surfaced it, a day after the fact. I noticed 30 messages in one session and went looking.
Total for my own system: 2 passes, 5 partials, 8 fails.
Every row here resolves to pass, partial, or fail, because I built this system and can see all of it. That isn’t true of most agent products. When I run these questions against someone else’s, the during questions usually come back as not visible from outside, because who gets summoned and what they receive is not something a customer can observe. That is worth noticing on its own. The layer where my system failed is the layer nobody outside the company can check.
That is worse than my own read of it before I wrote the evidence column. Several of these I would have marked pass or partial until I had to write the line underneath. Question 12 is the clearest case. The summary is independent software reporting dependent data, which isn’t a real separation.
What I’m changing
The gap I care about most is question 9, and it is the one I only saw clearly after writing these questions down.
The customer never had a way to reach me from inside the session. The design I’m building now works like this. When an agent cannot ground a claim in a file it actually has, it refuses instead of answering. It says what it does not have. It stays in the session rather than ending it, tells the customer what it can still cover, points at the setup guide for the rest, and then asks one question: do you want me to contact Renee now?
The customer decides. That matters more than it looks. I don’t have to guess whether a given question is urgent enough to interrupt me, and the customer doesn’t have to guess whether he is allowed to ask. Whoever holds the problem is the one who says how much it is worth.
Question 8 changed that design after I wrote it. Asking whether the customer wants me contacted is only half of it, because a yes that goes nowhere is worse than no offer at all. So the offer carries my status. Available now, or unavailable until a time. The customer sees which one before deciding, and either way the agent stays in the session and keeps covering what it can. How that status gets set is an implementation detail and it scales differently depending on who is behind it. For one person it can be as simple as a switch. For a team it could look like an on-call rotation. The requirement is the same either way.
Designed 08-31-2026, revised 09-03-2026. Not shipped yet.
Where this came from
I built the system. I shipped it. It failed in a way nothing in it could see, and one customer paid for that.
The 15 questions came out of working backward from that morning. They aren’t a maturity model. A mark only means as much as the evidence line sitting next to it. They are the questions I ask before I trust an agent to act without me in the room, and the reason there are more of them in the during section than most checklists have is that the during section is where my own system broke.
I’m applying these to other people’s products as public teardowns, marking what I cannot see from outside as exactly that.
If you are building agents and the middle of your flow is unowned, I would like to hear how you are handling it.
Renee Romero