
Who Checks the Agent's Work?
Once a year I sit down with an appraiser and go through my last twelve months as a GP: audits, significant events, complaints, feedback from patients and colleagues, what I've learned and what I'd do differently. Every five years that feeds into revalidation, which is the profession's way of asking a simple question. Is this doctor still safe to practise?
Nobody enjoys the paperwork. But I've never met a colleague who thinks the idea behind it is wrong. If someone is making decisions that affect patients, somebody else should be checking, regularly, that those decisions are still good ones.
I've been thinking about that a lot recently, because I'm about halfway through Hamel Husain and Shreya Shankar's AI evals course. The more of it I do, the more it looks like appraisal, audit and revalidation, rebuilt for software that makes decisions.
Why evaluation is the next frontier
We have a long history of checking human work. Factories have quality control. Call centres listen back to a sample of calls. Hospitals run audits, and doctors in training work under supervision that loosens only as they prove themselves. When I supervised medical students, they sat in on consultations before they ran any, and ran them with me in the room before they ran them alone. Trust is earned in small steps, and it gets re-checked.
A growing share of work is now being handed to AI agents. An agent is software that reads a request, decides what to do, uses tools (looks up a record, sends a message, books an appointment) and replies. They're already answering customer queries and drafting documents. In healthcare they're starting to triage messages, summarise notes and pre-fill forms. (I wrote about how they're put together in Beyond the Harness.)
Here's the uncomfortable bit. Agents need more checking than people do, for reasons that aren't obvious until you've watched one closely.
They're inconsistent in a way people aren't. Ask the same agent the same question five times and you can get five different routes to an answer, and occasionally a wrong one. A colleague who was competent on Monday is usually competent on Tuesday. An agent makes no such promise.
They change underneath you. The company behind the model updates it, someone on your team tweaks a prompt, and behaviour that worked last month quietly stops working. Picture a locum whose training was rewritten overnight without anyone telling you.
They don't learn from near misses unless you make them. When I get something wrong, I remember it. An agent has no memory of yesterday's mistake unless someone turns that mistake into a check it has to pass.
They're fast. A human making a bad call affects one patient at a time. An agent can make the same bad call a few thousand times before lunch.
So I think evaluation ("evals", if you want the jargon) is where the serious work in AI is heading. Building an agent that demos nicely gets easier every month. Proving that it behaves well, consistently, on the cases that matter is the hard part, and it's the part very few teams have anyone dedicated to.
In clinical settings the stakes go up again. The NHS already has clinical safety standards for health IT: DCB0129 for the people who make a system and DCB0160 for the organisations that deploy one. Both ask us to identify hazards and show that our controls work. For an AI agent, evals are how you produce that evidence, continuously, instead of once in a hazard log that nobody opens again.
The course I've been taking
The course is called AI Evals for Engineers & PMs. It's taught by Hamel Husain, an engineer who has spent around two decades building machine learning systems, and Shreya Shankar, a researcher who studies how people actually evaluate AI in practice. It's run through live sessions, a long course reader and hands-on homework, and it's aimed at engineers and product managers. You don't need to be a statistician, which was a relief.
What I liked straight away is what it deliberately leaves out. It isn't about the public leaderboards you see in headlines, where models are ranked on general knowledge quizzes. It asks a narrower and far more useful question: does your agent work for your job? There's no public leaderboard for writing safe discharge summaries or triaging a GP inbox, so if you work in healthcare, that framing is the whole point.
Everything is taught through one example you build yourself: a support assistant for a made-up online shop called Cartwheel. It looks up orders, answers questions about returns and issues refunds. Shoppers, shop owners and support staff can each see different things, and refunds over a set amount need a human to approve them. It sounds a long way from general practice, but swap "refund" for "prescription" and "shop owner" for "practice" and the problems are uncannily familiar.
The course moves through six stages with that one agent. Have a click through them.
The shape of the course
Six stages, built around one agent you keep improving. Tap through them.
Make an agent you can test
The clinical version
Underneath all six stages sits one framework. I've found it the most useful idea in the course, so it gets its own section.
The Three Gulfs
The course describes three gaps that open up whenever you build something with AI. The idea comes from Shreya Shankar's research and borrows from Don Norman, whose book The Design of Everyday Things described the "gulfs" between what people want from a device and what it actually does.[1]
There are three parties: you, with what you intend; your data, meaning the real requests people send; and the agent, meaning the instructions you've written plus the model running them. A gap can open between any two of them.
The Three Gulfs
Three parties, three gaps. Pick a gulf to see where it opens up and how you close it.
You ↔ your data
“Do I actually know what comes in, and what goes out?”
You design for the requests you imagine, not the ones people really send, and you judge the agent on a handful of good-looking answers.
In the course
In clinic
How to close it
Found a mistake? One question tells you which gulf you’re in: was the rule actually written down?
Framework from Shreya Shankar and colleagues’ research, as taught in the course. Clinical examples are my own.
Here's each one in plain terms.
The gulf of comprehension sits between you and your data. You build for the requests you imagine rather than the ones people send. The course example is a prompt designed around "refund order #4127", when real shoppers write "the blue mug I bought from Juniper last month". Every clinician who has designed a form knows this one. You picture tidy, complete referrals and what arrives is three abbreviations and a question mark. It applies to outputs too. Glance at a few good answers and you'll think the agent is fine. Read a hundred and the patterns appear. The only way across is to look at lots of real data, which, in my experience, almost nobody does enough of.
The gulf of specification sits between you and the agent. What you meant doesn't fully make it into the instructions you wrote. "Help customers with returns" sounds clear until you ask which return window wins when the shop and the platform disagree. The agent doesn't stop and ask. It guesses, and the guess won't always be yours. The course reader has a lovely line on this: when the rule was never written down, the agent did nothing wrong. The fix is usually cheap. Write the rule down.
The gulf of generalization sits between your data and the agent. Your instructions are clear and the tools work, yet the agent still applies them inconsistently on cases it hasn't seen before. It handles "refund order #4127" perfectly, then picks the wrong order when someone writes "my niece bought something on my account". This is the gulf you can't close completely, however good models get. You can only measure it and shrink it.
The practical payoff is a single question to ask whenever you find a mistake: was the rule actually written down? If not, it's a specification problem, and you fix the instructions before building anything else. If it was, and the agent still got it wrong, it's a generalization problem, and that's what you build automatic checks to measure. Teams lose weeks building elaborate tests for things they never told the agent to do, which ends up measuring how well the model reads minds.
The course adds one more point I found important for healthcare. How hard you need to work at each gulf depends on who's using the tool. A programmer using a coding assistant can bridge most of the gaps themselves: they know how to say exactly what they want, they can read the output and they can test it. A member of the public asking a health chatbot about their symptoms can do none of those things. When users can't catch mistakes, and mistakes are costly, your evaluation has to be far more rigorous. In clinical work you're almost always at that end.
Five ideas that changed how I think
The course is detailed and practical, so I won't summarise all of it. These are the ideas I keep coming back to.
1. Look at the data before you decide what to measure
The instinct is to start with a list of things to check, like "accuracy", "tone" or "hallucination". The course says don't. Read a hundred or so real conversations first. Jot a quick note on the first thing that went wrong in each, then sort the notes into piles and name each pile. Those piles become your list of failure types, and they'll be specific to your product in a way no generic checklist is.
The method comes from qualitative research, and it's essentially how a good case-note audit should work: read the notes, then write the criteria. The course also insists on simple yes/no labels ("did this failure happen?") over 1–5 ratings. Anyone who has watched two clinicians score the same patient differently on a five-point scale will understand why.
2. Treat an AI judge like a new trainee
For judgment calls, such as whether a reply was faithful to the source or whether the tone was right, the course uses a second AI as a "judge". The catch is that you can't simply trust it. You calibrate it against labels from a human expert, the way you'd watch a trainee's marking against a consultant's before letting them mark alone.
You score the judge with two numbers clinicians already know as sensitivity and specificity: how many good answers it correctly passes, and how many bad ones it correctly fails. The course then corrects the judge's raw pass rate for its known error rate. It uses the same maths epidemiologists use to estimate true prevalence from an imperfect screening test. That was a pleasing moment of recognition.
3. Every mistake becomes a permanent test
When you fix a failure, you write a test from it that runs automatically every time anyone changes anything. If the old mistake comes back, the test catches it before a patient or customer does. It's what incident reporting is supposed to achieve and often doesn't. We write the report, discuss it at the meeting, and six months later the same thing happens on a different shift. A test is learning that can't be forgotten.
Each test has three parts, and the middle one surprised me.
What goes into one test
Three parts. The middle one is what makes testing an agent different from testing a calculator.
1. The input
Shopper asks: “Can I return my leaking espresso machine?”
2. The starting state
The order exists in the database and hasn’t been refunded yet.
3. The expected result
- No refund tool is called before the shopper confirms they want one.
- No refund row appears in the database after the run.
The starting state matters because agents change things. A calculator gives the same answer whatever happened yesterday. An agent's behaviour depends on what's in the system when it begins. So every test sets the world up the same way, and it checks the data afterwards as well as the words, because an agent can cheerfully say "your refund is queued" when nothing was queued at all.
Testing before release is only half of it. Once the agent is live, the same judges score a sample of real conversations so you notice when behaviour drifts. You re-check the judges against fresh human labels every few weeks, and you pin the exact model version rather than a "latest" label that can silently change underneath you. Medicine got there long ago: a drug that passes its trials is still watched after launch.
4. Run it more than once
Run the same test five times and you might get four passes and a fail, with nothing changed in between. So the course separates two questions that sound similar and aren't.
pass@k asks whether at least one of k attempts worked: "can it do this at all?" pass^k asks whether every one of k attempts worked: "can I rely on it?"
Try the sliders. The starting setting is four passes out of five runs.
“Can it?” versus “Will it, every time?”
Run the same test several times, then ask two different questions of the same results.
Same 5 runs, two answers 60 points apart. If one good attempt is enough (say, a human picks the best draft), pass@k is fair. If every single attempt has to be safe, like checking allergies before prescribing, pass^k is the honest number.
With four passes out of five, any three attempts you pick will include at least one success, so pass@3 is 100%. The chance that all three succeed is only 40%. For most clinical tasks, pass^k is the honest number. I don't want a colleague who checks for drug allergies in at least one of every three patients. I want every patient checked.
This isn't a theoretical worry. In τ-bench, a published benchmark that tests agents on simulated customer service, the best model at the time solved about 61% of retail tasks on a single try. Asked to solve the same task eight times in a row, it managed fewer than 25%.[2]
5. Somebody has to own what "good" means
All of this rests on a human deciding what counts as a pass, and experts disagree. For smaller teams, the course recommends a "benevolent dictator": one trusted domain expert whose judgment sets the standard. In a clinical product that's a clinician, ideally one who has done the job the agent is doing. It's close to the role a Clinical Safety Officer already plays, and I'd argue the two should often be the same person.
Bigger teams need several reviewers, and then you have to check whether they actually agree. Raw agreement misleads you here. If 90% of conversations are fine, two reviewers will agree most of the time by luck alone. The fix is Cohen's kappa, which many of us met in critical appraisal teaching. It measures agreement beyond what chance would give you, and the course suggests aiming for at least 0.6.
Do your two reviewers actually agree?
Two people label the same conversations as Pass or Fail. Enter how their labels lined up.
Raw agreement
88%
Agreement expected by luck
76%
Cohen’s kappa
0.50
ModerateBelow the 0.6 bar the course recommends. When most conversations pass, two people agree a lot by luck alone, so a high raw percentage can hide a vague rubric. Time for an alignment session.
The default numbers show the trap. Two reviewers agree on 88% of cases, which sounds reassuring, yet kappa comes out at about 0.5. When reviewers disagree, the course recommends an alignment session. The aim isn't to argue about who labelled an old case correctly. It's to change the written rubric so future reviewers agree. That's the good version of a case review meeting: less about blame, more about fixing the guideline.
What I'm taking back into clinical work
Look before you measure. Before anyone writes an evaluation plan for a clinical AI tool, someone with clinical experience should read a large sample of its real inputs and outputs. The failures that matter are rarely the ones on a generic checklist.
Name the gulf. When a clinical AI tool gets something wrong, the first question is whether anyone told it the rule. A surprising number of "AI errors" turn out to be gaps in the instructions written by humans.
Ask for pass^k, not pass@k. When a supplier says their tool can get something right, the follow-up is how often it gets it right every time, across repeated tries.
Go-live is when evaluation starts. A clinical AI tool with a one-off validation report and no plan for ongoing monitoring should worry us as much as a drug with no post-marketing surveillance.
Treat the evals as part of the safety case. The tests, the judges and the written rubrics deserve the same change control as the tool itself. The easiest way to make a failing test pass is to delete it.
Back to appraisal
I started with appraisal because it's the closest thing I know to what this course teaches. Appraisal isn't a single exam you pass and forget. It's a regular, evidence-based check that someone making decisions on other people's behalf is still doing it well, with a trusted reviewer, written standards and a record of what has changed since last time.
Over the next few years we're going to hand a lot of decisions to agents, in healthcare and everywhere else. They deserve the same treatment we give ourselves. That's what responsible delegation looks like, whether you're handing work to a new registrar or to a piece of software.
These are my own notes from AI Evals for Engineers & PMs by Hamel Husain and Shreya Shankar, and its course reader. The clinical comparisons, the triage example and any mistakes are mine. I still have the security and improvement stages to go. If you build or buy AI tools for healthcare, I'd recommend the course.
New posts go out on LinkedIn. Follow Rajiv