The Limits Set Before You Ask

Case of the Week, Min Wu, PhD · ai-public-health.com

I was the student physics professors liked to call on. Perfect scores in the theory courses, a GPA with nothing on it but the top mark. Then came my mechanical workshop internship.

I passed the training. I learned how to run the machines and how to use most of the tools. And then I made parts that were not good enough to keep. Some came out low-grade. Some were simply wasted.

The one I remember involved a dent whose depth I had to measure. The vernier caliper in my hand had a thin rod that slides out of its tail, built for exactly that job. I measured with the jaws instead. The part came out with large errors, and the perfect GPA I had carried into the workshop did not come out at all.

Today, when my sons talk about their pre-med plans, my advice is always the same: double-check your labs. That is where a good record breaks.

In my previous essay, I Don't Know, and It Depends, I argued that whether you should trust an AI answer depends on three things on the user's side: what kind of task it is, how much you already know, and which tool you're using.

But there is another side to "it depends," and it comes earlier. Before you ever type a question, someone else has already decided what the tool was built for, what test it had to pass, and what it is allowed to do. Those decisions set limits you inherit without seeing them.

Who Set the Test

I passed two tests before that workshop part failed. The theory exams said I understood physics. The workshop training said I knew how to use the equipment. Neither one was dishonest. Each measured exactly what it was designed to measure. What neither measured was whether I could produce a part that met the drawing.

AI tools go through something similar before release. Builders test them against collections of questions and tasks, often called benchmarks, and a high score is a real achievement. But the builder chooses the exam. A tool can score well on a test and still fail at your job if your job was never on the test.

I ran into this gap directly in my most recent paper, on AI agents that act on personal health records. In the evaluation work I reviewed, the focus was mostly on whether the agent's answer is correct, usually tested on data that had already been cleaned. So I sketched a scenario to show what that leaves out. Imagine an agent that correctly predicts a person with type 2 diabetes is heading toward a dangerous low after a skipped meal and a demanding workday. The prediction is right, but the agent still has to choose what to do next. Suggesting a walk sounds healthy and would make the low worse. A correct reading does not guarantee a part that fits.

So the first question to ask about any AI tool is not "how well did it score?" but "what was it tested on, and does that look anything like what I need it to do?"

What the Job Actually Required

The dent decided what "accurate enough" meant. Not my grades, not my training certificate, not how confident I felt holding the caliper. The part had a tolerance, someone else had written it on the drawing, and the only question that mattered was whether my measurement could meet it.

AI tools carry the setting they were built for in the same way. A tool designed around one kind of population, one kind of record-keeping, or one kind of clinic brings those assumptions wherever it goes. In a setting that matches, the limits may never show. In a setting that doesn't, they show up as errors nobody on the builder's side was positioned to see.

What the Tool Was Built to Do

Here is the part of the story I understand better now than I did then. For measuring that dent, there were really three options, and each one had a limit decided by a designer long before I walked in:

1. The caliper's jaws. The wrong function for the job. This is what I used.
2. The caliper's tail. A depth rod built into a general-purpose tool. Convenient, because one instrument does several jobs, but less precise than a tool made for depth alone.
3. A depth micrometer. A separate instrument built for one measurement and nothing else.

No amount of skill turns the caliper's tail into a depth micrometer. The ceiling was built into the tool. If the job needed more precision than the design allowed, the only option was a different instrument.

AI works the same way. A general-purpose assistant does many things reasonably well. A purpose-built tool does one thing with more precision. And builders also decide whether a tool can be adjusted after release: whether someone downstream can correct a weakness once it appears, or whether the design is sealed and users simply live with it. Neither choice is wrong. But it is made by the builder, not by the person holding the tool.

Where It Actually Gets Risky

Put the three together and the risky combination becomes clear: a general-purpose tool, certified by a test that doesn't resemble the job, used where precision decides the outcome, and designed so that no one downstream can change it. Every piece of that can be true while the tool still looks polished and scores well. Nothing in the setup is positioned to catch the mismatch until the part comes out wrong.

The mirror image is the strongest combination: a builder who knew the setting, tested the tool there, and left room for correction after release. That doesn't make the tool perfect. It makes its limits closer to where the real work happens, and easier to fix when they show.

The Test Makes the Top Student.

Observer's Insight

I think often about why passing two tests didn't protect me from a wasted part. The tests weren't wrong. They were answering a different question from the one the workshop was asking, and I had no reason to notice the difference until the part was in my hand.

What I'm less sure of is the advice I give my own sons. "Double-check your labs" is good advice. But part of what I'm telling them is how to protect a score. I hope I'm also teaching them to do the work the score is supposed to stand for, and I'm not certain those are always the same lesson. The workshop taught me they can come apart.

For AI, the limits I've described are set by builders, and most users will never see those decisions directly. I don't yet know how much of that can be made visible to the people who inherit it. But I know which question to start with. Not "how good is this tool?" but "what was it built for, and who decided?"

As a member of the Springer Nature Author Affiliate Program, I may earn a small commission from purchases made through this affiliated link to my book. Support the author by checking out my textbook, Artificial Intelligence in Public Health: https://tidd.ly/4mH9389