Picture the scene. A risk memo comes back from the AI tool in four seconds, formatted, confident, with a clean summary at the top. The reviewer's name goes in the sign off field forty seconds later. Nobody read it closely. Nobody was supposed to have time to.
That is what “human in the loop” looks like in most companies right now. A name attached to a field. Not a check. A liability transfer wearing the costume of a safeguard.
What the phrase is actually doing
Every AI governance document says it. Every vendor pitch deck has the slide. “Human in the loop” sounds like oversight, so it gets written into policy and everyone moves on, satisfied that the box is checked.
But a checkbox with a name on it does not catch anything. It just gives you someone to point to later. That is not the same job, even though they look identical on paper.
Bottom line: if the point of oversight is to catch what the model got wrong, and nobody is actually positioned to catch anything, you do not have oversight. You have a signature.
Why the checkbox version fails, and it is not the reviewer's fault
There is a well documented effect in how people work with automated systems. The more often a system is right, the less carefully people check it, right up until the one time it matters. Psychologists call it automation complacency. It is not laziness. It is what happens to attention when a machine keeps being correct.
Here is the part that should worry you more. The better the output looks, the less scrutiny it tends to get. A confident, clean, well formatted paragraph reads as trustworthy whether or not it is true. The same reviewer who would catch a junior analyst's sloppy draft in ten seconds will wave through the identical error if an AI wrote it, because the packaging did the persuading.
That is not a training gap you fix with a memo telling people to stay vigilant. Vigilance decays. It is a known property of human attention, not a character flaw. If your process depends on nobody ever getting tired or trusting the tool a little too much, your process is already broken.
What a real check actually requires
A real human in the loop role has three things a checkbox does not.
First, a specific job. Not “review the output and see how you feel about it.” A named thing being checked, against a named standard. What exactly is this person confirming, and how would they know if it were wrong.
Second, the actual authority to say no. If sending something back costs the reviewer time, blows a deadline, or makes them look difficult, the checkbox wins by default every time. Oversight that has no real cost to bypassing it is not oversight. It is theatre with paperwork.
Third, training on how this specific system fails, not a general warning that AI can be wrong. A useful reviewer knows the tool's actual failure patterns: where it tends to invent a citation, where it smooths over a gap in the data instead of flagging it, where it agrees with whatever framing the question implied. General caution does not catch a specific, predictable failure mode. Knowing the pattern does.
For what it is worth, the EU AI Act's human oversight requirement is built around roughly this idea. Oversight has to be capable of catching and correcting a problem, not merely present in the org chart. That is one sentence, not the whole subject. It deserves its own post.
What it looks like done right
A named check, a specific thing being verified, and a real path to sending it back that actually gets used sometimes. If the send it back option has never once been used, that is not a sign the system is flawless. It is a sign nobody is really looking.
None of this is expensive to build. It is a job description, a short list of known failure patterns, and permission to say no without paying for it. Plenty of demos show the approval gate. Nobody demos the moment someone actually uses it to say no.
We think about this because it is the actual work, not the interesting part. Building the workflow that makes oversight real, instead of decorative, is most of what a serious AI adoption plan is.
One more thing. This post was drafted with an AI model, then read, cut, and signed off by a human before it went up. Named check. Real ability to send it back. Someone willing to say so.