Opens in a new tab
needsahuman.

The jagged frontier: what AI does well, and where it fails

Does AI actually work, or is it hype? The best evidence says both, depending on the task. Researchers call this the jagged frontier of AI: excellent at some tasks, misleading on others that look almost the same. In this lesson What counts as proof that AI helps, and what does not. What six well-known studies…

Does AI actually work, or is it hype? The best evidence says both, depending on the task. Researchers call this the jagged frontier of AI: excellent at some tasks, misleading on others that look almost the same.

In this lesson

  • What counts as proof that AI helps, and what does not.
  • What six well-known studies of real people at work found, in plain words.
  • The “jagged frontier”: why AI can be excellent at one task and poor at a similar-looking one next to it.
  • Why the gains are often biggest for less experienced staff, and why skilled people can end up slower.
  • How this links to the evidence grades on our job pages.

What counts as proof

You will hear a lot of claims about what AI does for businesses. Most of them come from people selling it. Before you rely on a claim, ask what kind of evidence is behind it.

  • Strong: a study with a comparison group. Some people get the AI and some do not, ideally chosen at random, and both do the same work. Then any difference is likely down to the AI.
  • Strong: independent experts grading AI’s work against professionals’ work without being told which is which (“blind” grading).
  • Useful for the big picture: official surveys of businesses and workers, such as those run by the US Census Bureau and the UK’s Office for National Statistics. They tell you how many firms use AI, not whether it works.
  • Weak: a vendor’s case study, a testimonial, or “customers save ten hours a week”. These may be true, but they are marketing, and they rarely say what happened to the firms where it did not work.

This course only uses the first three kinds. So does the rest of this site.

What the studies found

Here are six studies often cited in discussions of AI at work. Each card gives who ran it, when, what was measured and what was found. Figures are as the authors report them; links are under Sources.

Erik Brynjolfsson, Danielle Li and Lindsey Raymond, The Quarterly Journal of Economics · May 1, 2025 · Finds an effect

Generative AI at Work

15% more issues resolved per hour on average (the 2023 working paper version reported 14%, with 34% for novice and low-skilled workers)

With the AI assistant, agents resolved more issues per hour. Less experienced and lower-skilled agents gained the most, in both speed and quality; the most experienced saw small gains in speed and small declines in quality.

Caveat: One kind of work (customer support chat). The AI assisted agents; it did not replace them in the study.

Source
Dell'Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and Lakhani (Harvard Business School working paper 24-013, with Boston Consulting Group) · September 22, 2023 · Mixed

Navigating the Jagged Technological Frontier

Inside the frontier: 12.2% more tasks, 25.1% faster, more than 40% higher quality. Outside it: 19 percentage points less likely to be correct

On 18 tasks inside what the authors call AI's 'jagged technological frontier', consultants with AI completed more tasks, faster and to higher quality. On one task chosen to sit just outside it, consultants with AI were less likely to get the right answer than those without.

Caveat: A working paper. One firm; tasks set by the researchers; the AI was GPT-4 as of 2023.

Source
Shakked Noy and Whitney Zhang (MIT), Science · July 14, 2023 · Finds an effect

Experimental evidence on the productivity effects of generative artificial intelligence

Average time taken fell 40%; output quality rose 18%

Those with ChatGPT took less time and produced work rated higher, and the gap between stronger and weaker writers narrowed.

Caveat: Short, self-contained writing tasks set for the experiment, not the participants' real work.

Source
Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer (arXiv preprint) · February 13, 2023 · Finds an effect

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

55.8% faster

The group with Copilot finished the task faster than the group without it.

Caveat: One small, well-defined coding task. A preprint; GitHub makes Copilot.

Source
METR (Model Evaluation and Threat Research) · July 10, 2025 · Finds an effect

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

19% longer with AI; developers had expected a 24% speed-up and afterwards believed it was 20%

With AI tools allowed, developers took longer. Before the study they expected AI to speed them up; afterwards they still believed it had.

Caveat: The authors say it does not show AI fails to speed up most developers, or people in other fields, and that their sample may not be representative.

Source
OpenAI (Patwardhan et al.), arXiv 2510.04374 · October 5, 2025 · Mixed

GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

Claude Opus 4.1: 47.6% of deliverables graded as better than or as good as the professional's

The strongest models were approaching the experts' quality on these tasks. The authors note the tasks are precisely specified and one-shot, not interactive, unlike most real work.

Caveat: Run by an AI company (though it graded other makers' models too). Blinding was imperfect. Results for models of mid-2025.

Source

In brief:

  • Customer support. Economists studied a generative AI assistant rolled out to 5,172 customer-support agents. In the version published in 2025, agents with the assistant resolved 15% more issues an hour on average. “Less experienced and lower-skilled workers improve both the speed and quality of their output, while the most experienced and highest-skilled workers see small gains in speed and small declines in quality” (Brynjolfsson, Li and Raymond, Quarterly Journal of Economics). The 2023 working paper reported 14%, and 34% for novice and low-skilled workers (NBER).
  • Consultants. In an experiment with 758 consultants at Boston Consulting Group, those given GPT-4 “completed 12.2% more tasks on average, and completed tasks 25.1% more quickly,” with “more than 40% higher quality,” on tasks inside AI’s capabilities. “For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI” (Dell’Acqua et al., Harvard Business School working paper).
  • Writing. In an experiment with 453 college-educated professionals doing writing tasks from their own occupations, “the average time taken decreased by 40% and output quality rose by 18%” for those given ChatGPT (Noy and Zhang, Science).
  • Coding, a set task. Developers asked to build a small web server with the GitHub Copilot assistant “completed the task 55.8% faster than the control group” (Peng et al.).
  • Coding, real work. In a 2025 study, 16 experienced open-source developers worked on 246 real issues in projects they knew well. “When developers are allowed to use AI tools, they take 19% longer to complete issues.” They had “expected AI to speed them up by 24%,” and afterwards “still believed AI had sped them up by 20%” (METR).
  • Head-to-head with experts. In OpenAI’s GDPval, experts with “an average of 14 years of experience” compared AI’s work with professionals’ work on real tasks from 44 occupations. In the first results, “47.6% of deliverables by Claude Opus 4.1 were graded as better than (wins) or as good as (ties) the human deliverable.” The authors note the tasks are “precisely-specified and one-shot, not interactive” (Patwardhan et al.).

What they add up to

Read together, the studies say four things a business owner can use.

  1. On the right tasks, the gains are real and large. Drafting, summarizing, answering routine questions and well-defined pieces of analysis came out faster, and often better, in several independent experiments.
  2. Less experienced staff often gain the most. In the customer support study the assistant seemed to pass on what the best agents did. That is good news for training, but it also means your most experienced people may see little benefit and should not be forced into tools that slow them down.
  3. Skilled people on familiar, messy work can be slowed down, and may not notice. The METR result is the important counterweight to the headline gains: people felt faster while being slower. Measure; do not rely on how it feels. Lesson 11 shows a simple way.
  4. Lab tasks flatter AI. The biggest gains come on tasks that are clearly set out and self-contained. Real work is full of missing context, changing requirements and people. GDPval’s own authors make this point.

The jagged frontier of AI

The consultants study gave us the most useful picture of where AI helps. Its authors suggest that “the capabilities of AI create a ‘jagged technological frontier’ where some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI.”

Picture a row of tasks your business does, and a line running across them. Above the line, AI does well. Below it, AI produces something that looks just as good but is wrong. The line is not smooth: it zigzags, and the tasks either side of a zig can look almost the same.

IllustrativeWhere each task sits is an example, not a measurement.
The jagged frontier An uneven line runs across eight ordinary business tasks. Above the line, AI does well: summarize a meeting, draft a reply to a customer, translate a product page, sort and tag invoices. Below the line, AI misleads: count stock from a photo, check a contract clause, forecast next month's cash, answer a policy question it wasn't given. Placement is illustrative. Above the line: AI does well Below the line: AI misleads Summarize ameeting Count stockfrom a photo Draft a replyto a customer Check acontract clause Translate aproduct page Sort and taginvoices Forecast nextmonth's cash Policy questionit wasn't given The jagged frontier An uneven line runs down through eight ordinary business tasks. Right of the line, AI does well: summarize a meeting, draft a reply to a customer, translate a product page, sort and tag invoices. Left of the line, AI misleads: count stock from a photo, check a contract clause, forecast next month's cash, answer a policy question it wasn't given. Placement is illustrative. Left of the line:AI misleadsRight of the line:AI does well Summarize ameeting Count stockfrom a photo Draft a replyto a customer Check acontract clause Translate aproduct page Sort and taginvoices Forecast nextmonth's cash Policy questionit wasn't given
AI is excellent at some tasks and poor at neighboring ones that look much the same. Solid boxes: AI does well. Dashed boxes on the hatched side: AI misleads, often confidently. The line is jagged because you cannot guess it from how hard a task feels; you find it by testing on your own work.

For example, an assistant may summarize a long supplier contract well, and then misstate the one clause about who pays for late delivery. It may draft a friendly reply to a complaint and then promise a refund your policy does not allow. It may tidy a spreadsheet perfectly and then get a total wrong. The danger is not the obvious failure; it is the confident, well-written answer on a task just past the line, which in the consultants study made people more likely to get it wrong than if they had not used AI at all.

Three practical lessons follow:

  • Test it on your own work. You cannot tell where the line runs from a vendor’s demo. Lesson 4 gives a test for choosing tasks, and Lesson 12 a way to pilot them.
  • Keep checking even when it is usually right. The tasks most likely to fool you are the ones where it is right nine times in ten.
  • The line moves. Models improve, and tasks that sat below the line last year may be above it now. Re-test every few months.

Where it fails

Across the studies and our own task ratings, AI is weakest where a task needs:

  • Judgment on thin information: a decision about a customer, an employee or a risk, where the facts are incomplete and the stakes are real.
  • Physical work: anything done with hands, tools or in a place, from fixing a boiler to serving a table. A language model cannot touch anything, and robots are still rare outside factories (see our guide to humanoid robots and physical jobs).
  • Accountability: a signature, a professional sign-off, or a decision a customer, regulator or court expects a named person to own.
  • Live facts it was not given: today’s stock level, this week’s prices, what the customer said on the phone yesterday.
  • Trust and relationships: calming an angry customer, winning a new client, managing a team through change.

How this links to our scores

Every job page on this site asks “Is it better than a person?” and gives an evidence grade from A to D. Grade A means there is a blind, expert-graded head-to-head test of AI against professionals on that job’s own work, such as GDPval. Only some jobs have one: 0 of the 923 jobs we score have grade A or B evidence in this release. Where there is none, we say so on the page rather than guess. How the grades work is on our quality parity method page.

These are the jobs whose quality score currently rests on grade A evidence, live from our data:

No jobs have grade A evidence in this release.

How many businesses use it

Official surveys give the wider picture. The latest figures from the US Census Bureau and the UK’s Office for National Statistics, updated as each survey is released:

In the two weeks to September 6, 2026, 23.8% of US businesses said they used AI in at least one business function (Census Bureau), and in an ONS survey run June 15 to 28, 2026, 28.9% of UK businesses said they currently used at least one AI technology.

Our business AI adoption tracker charts both surveys over time and by industry.

Key terms

Every term is explained in the glossary.

Try this

Pick one task your team does every week that AI might help with, such as replying to routine customer emails. For the next ten times it comes up, have the person who normally does it write down how long it takes. For the ten after that, have them use an AI assistant and write down the time again, including the time spent checking and fixing. Also note any mistakes the AI made that would have gone out if nobody had checked. You have just run a small version of the studies above, on your own work, which is the only test that counts.

Sources