The Expertise Illusion: How AI is Eroding the Professions Built on Judgment

By Courtney Turrin · July 6, 2026 · 12 min read read

Category: AI

Fired for Social Media Posts About Charlie Kirk, Workers Win Six-figure Settlements — "Courtroom One Gavel" by Joe Gratz is marked with CC0 1.0. To view the terms, visit https://creativecommons.org/publicdomain/zero/1.0/.

AI is eroding professional judgment, creating confident amateurs who may be less skilled than before. This is seen in fabricated legal citations and reduced medical diagnoses.

Last October, a plaintiff's attorney in the U.S. District Court for the Eastern District of Oklahoma filed eleven pleadings built on 28 false or misleading citations produced by AI. The presiding judge wrote, “Generative technology can produce words, but it cannot give them belief. It cannot attach courage, sincerity, truth, or responsibility to what it writes. That remains the sacred duty of the lawyer who signs the page.” The ultimate responsibility stayed with the lawyer who had signed the filings, machine or no machine.

That case is one of over 1,700 documented in a database tracking AI-generated fabrications in court filings worldwide, a number that has grown roughly eightfold in about a year. Most of the people submitting these filings aren't lawyers at all. They're laypeople representing themselves, often because they can't afford legal representation, leaning on a free tool that sounds exactly as confident when it's wrong as when it's right.

Everyone is watching for AI to take jobs. The damage it's already doing inside them looks different.

The people building this technology describe a coming replacement: models capable enough to run without us, an economy that no longer needs most of the people currently working in it. The evidence from the professions furthest along in adopting these tools shows something narrower: professional judgment eroding inside the jobs AI hasn't replaced. A confident amateur is being manufactured, and the trained expert underneath is losing ground while it happens.

The pattern across professions

In the courtroom. Fabricated citations remain the single most common error type in the hallucination database, and the growth curve is steep: from roughly 200 documented cases in mid-2025, to 368 by November, to more than 1,700 today. (Figure 1)

[datawidget key="hallucination-cases"]

In the published record. The same failure shows up further upstream. A Columbia-led audit of 2.5 million biomedical papers found the rate of fabricated references in peer-reviewed medical literature has climbed more than twelvefold since 2023, from 1 in 2,828 papers to 1 in 277 in the first weeks of 2026. Doctors build treatment decisions on studies that, in a growing number of cases, don't exist. Almost none of the flagged papers have been corrected at the time of publication.

In the endoscopy suite. The impact of AI on the published record is one failure. What happens to the clinicians themselves is another. Colonoscopy is one of the few places in medicine where AI assistance has been studied on the same clinicians over time. A multi-location study in the Lancet Gastroenterology & Hepatology tracked adenoma detection at four Polish endoscopy centers before and after AI tools were introduced. Detection during standard, non-AI colonoscopies fell from 28.4% to 22.4%, a six-point drop, in doctors who had each performed more than 2,000 procedures. Without the AI running, the same doctors caught fewer precancerous growths than they had before the AI arrived. (Figure 2)

[datawidget key="colonoscopy-adr"]

In the codebase. In 2025 an independent research group called METR (Model Evaluation and Threat Research), ran a randomized controlled trial with 16 experienced open-source developers completing 246 real tasks in code they knew well. Before starting, the developers predicted AI tools would cut completion time by 24%. Afterward, they believed it had, by 20%. The measured result: work done with AI took seasoned developers 19% longer. Every estimate of how well it was working was wrong, including the one made after the work was finished. (Figure 3)

[datawidget key="metr-perception"]

In the financial plan. An economics professor at Pomona College tested three leading LLMs against eleven ordinary financial-decision questions. Given a plain choice between a one-year loan at 9% APR and a ten-year loan at 1% APR, a comparison most borrowers could eyeball, all three LLMs recommended the 9% loan, comparing raw total payments and ignoring the time value of money entirely. On a deferred annuity question, one model produced a 67% rate of return, a number that only works if the buyer lives forever. Sam Altman flagged the exact failure mode more than a year earlier, before most of these tools were in daily use: ChatGPT, he wrote, is “good enough at some things to create a misleading impression of greatness.” A planner who can catch these errors doesn't need the tool. A client who can't catch them is exactly who gets hurt by trusting it.

The safety culture AI skipped

Human factors researchers have documented human failure from over-reliance on automation since the 1970s, mostly in cockpits. Automation complacency, insufficient monitoring of a system once it's trusted to run itself, shows up in both novices and experts and can't be trained away. The failure isn't a lack of skill, but a lack of attention at the exact moment attention would catch the problem.

The aviation industry didn't simply accept that problem. It built mandatory manual-flying hours, simulator retraining, and a reporting culture designed to catch automation failure before it became a crash. None of that infrastructure exists for someone representing themselves in a legal proceeding, a patient reading a chatbot's opinion on a biopsy result, or a client taking a robo-advisor's word on an annuity. The automation shipped, but the safety culture didn't come with it.

The tool is built to tell you you're right

Part of the confidence gap is a documented, trained property of these models, not an accident. Anthropic's own researchers found that leading AI assistants consistently favor responses that match a user's existing views, and that both human raters and the models used to grade helpfulness prefer a convincing sycophantic wrong answer over a correct one a meaningful share of the time. Stanford researchers extended the finding: models systematically preserve a user's self-image through validation, endorsement, and accepting whatever framing the user brought in, a pattern that can lock people into conclusions the evidence doesn't support. When prompted with an open-ended question, LLMs preserve the user’s face 47% more than humans. OpenAI rolled back an update to GPT-4o in 2025 after users noticed the model had become excessively agreeable in exactly this way.

The reward signal explains why LLMs are built to flatter. A model trained to be rated highly by users learns that agreement scores better than correction. The result is a tool that validates your judgment at the exact moment your judgment should be challenged.

A real link isn't the same as a real claim

Fabricated citations are the easy AI failure to catch. A source that doesn't exist gets found out the first time someone clicks through. The harder-to-identify failure is a citation of a real source that misrepresents what was said.

The Tow Center for Digital Journalism at Columbia conducted 200 tests on eight AI search tools (ChatGPT Search, Perplexity, Perplexity Pro, Gemini, DeepSeek Search, Grok-2 Search, Grok-3 Search, and Copilot) to assess their ability to cite news accurately and found the problem wasn't limited to fake links. The AI tools failed to retrieve the correct information in more than 60% of test queries, citing syndicated or copied versions of articles instead of the originals and answering questions confidently rather than declining to answer when they had no basis on which to ground their response.

A separate 2026 comparison of six major chatbots found almost no relationship between how often a model cites a source and how accurate its answer is. One model cited something in nearly every response and still ranked near the bottom for correctness. The inclusion of a link creates the appearance of a receipt without there being one.

BBC researchers found the effect compounds on itself: audiences trust an answer more when it's attributed to a familiar outlet, "even if it's incorrect." Misrepresentation without fabrication does more damage than an invented source, because nothing about it looks wrong.

Big tech is selling you the replacement story

The people building these systems have been explicit about wanting exactly this outcome, though the story keeps changing. Musk has said that in his best case scenario, probably none of us will have a job, replaced by what he calls “universal high income”.

Altman spent two years pushing his own version, first bankrolling a universal basic income pilot with $14 million of his own money, then pivoting to a "collective ownership" model instead. That's shifted too. In April, Altman told The Atlantic he no longer believes in universal basic income as much as he once did. A month later, at a banking conference in Sydney, he said he was delighted to be wrong about the entry-level jobs apocalypse he'd spent the past year predicting.

Musk hasn't walked anything back. He's still selling the no-jobs future, and he's raised the objection to it himself: what happens to meaning once a machine can do everything better than you can. His answer was that humans might find purpose in giving AI meaning instead.

Musk is still selling it. Microsoft CEO Satya Nadella is one of the few people who builds this technology and is willing to say the story has a business model attached.

Nadella spent much of June arguing the opposite case. In an essay posted on X, he warned that there is no societal permission for an AI future that hollows out entire industries. Days later, he made the same case in an interview with the Wall Street Journal that also ran on The Journal podcast: AI companies can't keep claiming all white-collar jobs are gone while asking for unlimited capital to build the infrastructure behind that claim.

Nadella runs one of the companies building the thing he's warning about, and his proposed fix, companies building their own proprietary AI systems instead of depending entirely on a handful of frontier models, happens to be good for Microsoft's business too. That doesn't make the strategy wrong. Companies outside the frontier-model business are already doing exactly this, building rather than renting, for reasons that have nothing to do with Nadella’s essay. His financial stake in the argument is real. So is the argument.

What we're actually deciding

The evidence out of law, medicine, software, and finance doesn't argue against using AI tools. It argues against using them without knowing what they cost. Trained judgment is eroding underneath a tool that sounds like judgment, deployed to people who were never given the training or the guardrails to know the difference.

That outcome is still malleable. It's driven by a set of choices being made, about what gets automated, who gets trained to catch the automation's failures, and whether the people doing the work that actually requires a human still get paid to do it. That's the choice to make intentionally.

Start with the group most likely to be sold the replacement story: new graduates. Unemployment among college grads ages 22 to 27 hit 5.8% this year, well above the national rate, and 41.5% of employed grads are working jobs that don't require the degree they paid for, according to the New York Fed. The New York Fed's own researchers attribute most of that shift to remote work, which makes it harder for managers to mentor new hires, not generative AI specifically. But remote work and AI adoption aren't separate stories. Both remove the same thing: a junior person sitting next to someone experienced, absorbing how the work actually gets caught when it's wrong. Whatever the primary cause, the tasks disappearing first are the junior analyst work, the first-draft coding, the research grunt work that used to teach a 22-year-old how to become a 32-year-old who can catch a bad AI output. Remove the on-ramp before judgment develops, and there's nobody downstream left to notice when the model is wrong.

The white collar case is worse, because it isn't hypothetical. The people affected already have the judgment AI is supposedly going to replace, and it's degrading anyway. A licensed attorney filed fake case law. Doctors are building treatment plans based on studies that don't exist. Developers who know their own codebase got slower while believing they'd gotten faster. That's a workforce that still has to supervise the tool, and the supervision itself is getting worse in real time.

The people selling work as optional, a future of universal income and machine-run abundance, are pointing at a real shift and misnaming it. What's actually being decided is smaller and more urgent: whether we keep training people to have judgment at all, while the tool built to replace them still can't tell a real citation from a misrepresented one, still can't outperform the professionals using it, and still hasn't answered the question its own biggest advocates keep raising out loud: what happens to a person's purpose once the reason to get good at something disappears.