Does AI actually work, or is it hype? The best evidence says both, depending on the task. Researchers call this the jagged frontier of AI: excellent at some tasks, misleading on others that look almost the same.
In this lesson
- What counts as proof that AI helps, and what does not.
- What six well-known studies of real people at work found, in plain words.
- The “jagged frontier”: why AI can be excellent at one task and poor at a similar-looking one next to it.
- Why the gains are often biggest for less experienced staff, and why skilled people can end up slower.
- How this links to the evidence grades on our job pages.
What counts as proof
You will hear a lot of claims about what AI does for businesses. Most of them come from people selling it. Before you rely on a claim, ask what kind of evidence is behind it.
- Strong: a study with a comparison group. Some people get the AI and some do not, ideally chosen at random, and both do the same work. Then any difference is likely down to the AI.
- Strong: independent experts grading AI’s work against professionals’ work without being told which is which (“blind” grading).
- Useful for the big picture: official surveys of businesses and workers, such as those run by the US Census Bureau and the UK’s Office for National Statistics. They tell you how many firms use AI, not whether it works.
- Weak: a vendor’s case study, a testimonial, or “customers save ten hours a week”. These may be true, but they are marketing, and they rarely say what happened to the firms where it did not work.
This course only uses the first three kinds. So does the rest of this site.
What the studies found
Here are six studies often cited in discussions of AI at work. Each card gives who ran it, when, what was measured and what was found. Figures are as the authors report them; links are under Sources.
Generative AI at Work
With the AI assistant, agents resolved more issues per hour. Less experienced and lower-skilled agents gained the most, in both speed and quality; the most experienced saw small gains in speed and small declines in quality.
SourceNavigating the Jagged Technological Frontier
On 18 tasks inside what the authors call AI's 'jagged technological frontier', consultants with AI completed more tasks, faster and to higher quality. On one task chosen to sit just outside it, consultants with AI were less likely to get the right answer than those without.
SourceExperimental evidence on the productivity effects of generative artificial intelligence
Those with ChatGPT took less time and produced work rated higher, and the gap between stronger and weaker writers narrowed.
SourceThe Impact of AI on Developer Productivity: Evidence from GitHub Copilot
The group with Copilot finished the task faster than the group without it.
SourceMeasuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
With AI tools allowed, developers took longer. Before the study they expected AI to speed them up; afterwards they still believed it had.
SourceGDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
The strongest models were approaching the experts' quality on these tasks. The authors note the tasks are precisely specified and one-shot, not interactive, unlike most real work.
SourceIn brief:
- Customer support. Economists studied a generative AI assistant rolled out to 5,172 customer-support agents. In the version published in 2025, agents with the assistant resolved 15% more issues an hour on average. “Less experienced and lower-skilled workers improve both the speed and quality of their output, while the most experienced and highest-skilled workers see small gains in speed and small declines in quality” (Brynjolfsson, Li and Raymond, Quarterly Journal of Economics). The 2023 working paper reported 14%, and 34% for novice and low-skilled workers (NBER).
- Consultants. In an experiment with 758 consultants at Boston Consulting Group, those given GPT-4 “completed 12.2% more tasks on average, and completed tasks 25.1% more quickly,” with “more than 40% higher quality,” on tasks inside AI’s capabilities. “For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI” (Dell’Acqua et al., Harvard Business School working paper).
- Writing. In an experiment with 453 college-educated professionals doing writing tasks from their own occupations, “the average time taken decreased by 40% and output quality rose by 18%” for those given ChatGPT (Noy and Zhang, Science).
- Coding, a set task. Developers asked to build a small web server with the GitHub Copilot assistant “completed the task 55.8% faster than the control group” (Peng et al.).
- Coding, real work. In a 2025 study, 16 experienced open-source developers worked on 246 real issues in projects they knew well. “When developers are allowed to use AI tools, they take 19% longer to complete issues.” They had “expected AI to speed them up by 24%,” and afterwards “still believed AI had sped them up by 20%” (METR).
- Head-to-head with experts. In OpenAI’s GDPval, experts with “an average of 14 years of experience” compared AI’s work with professionals’ work on real tasks from 44 occupations. In the first results, “47.6% of deliverables by Claude Opus 4.1 were graded as better than (wins) or as good as (ties) the human deliverable.” The authors note the tasks are “precisely-specified and one-shot, not interactive” (Patwardhan et al.).
What they add up to
Read together, the studies say four things a business owner can use.
- On the right tasks, the gains are real and large. Drafting, summarizing, answering routine questions and well-defined pieces of analysis came out faster, and often better, in several independent experiments.
- Less experienced staff often gain the most. In the customer support study the assistant seemed to pass on what the best agents did. That is good news for training, but it also means your most experienced people may see little benefit and should not be forced into tools that slow them down.
- Skilled people on familiar, messy work can be slowed down, and may not notice. The METR result is the important counterweight to the headline gains: people felt faster while being slower. Measure; do not rely on how it feels. Lesson 11 shows a simple way.
- Lab tasks flatter AI. The biggest gains come on tasks that are clearly set out and self-contained. Real work is full of missing context, changing requirements and people. GDPval’s own authors make this point.
The jagged frontier of AI
The consultants study gave us the most useful picture of where AI helps. Its authors suggest that “the capabilities of AI create a ‘jagged technological frontier’ where some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI.”
Picture a row of tasks your business does, and a line running across them. Above the line, AI does well. Below it, AI produces something that looks just as good but is wrong. The line is not smooth: it zigzags, and the tasks either side of a zig can look almost the same.
For example, an assistant may summarize a long supplier contract well, and then misstate the one clause about who pays for late delivery. It may draft a friendly reply to a complaint and then promise a refund your policy does not allow. It may tidy a spreadsheet perfectly and then get a total wrong. The danger is not the obvious failure; it is the confident, well-written answer on a task just past the line, which in the consultants study made people more likely to get it wrong than if they had not used AI at all.
Three practical lessons follow:
- Test it on your own work. You cannot tell where the line runs from a vendor’s demo. Lesson 4 gives a test for choosing tasks, and Lesson 12 a way to pilot them.
- Keep checking even when it is usually right. The tasks most likely to fool you are the ones where it is right nine times in ten.
- The line moves. Models improve, and tasks that sat below the line last year may be above it now. Re-test every few months.
Where it fails
Across the studies and our own task ratings, AI is weakest where a task needs:
- Judgment on thin information: a decision about a customer, an employee or a risk, where the facts are incomplete and the stakes are real.
- Physical work: anything done with hands, tools or in a place, from fixing a boiler to serving a table. A language model cannot touch anything, and robots are still rare outside factories (see our guide to humanoid robots and physical jobs).
- Accountability: a signature, a professional sign-off, or a decision a customer, regulator or court expects a named person to own.
- Live facts it was not given: today’s stock level, this week’s prices, what the customer said on the phone yesterday.
- Trust and relationships: calming an angry customer, winning a new client, managing a team through change.
How this links to our scores
Every job page on this site asks “Is it better than a person?” and gives an evidence grade from A to D. Grade A means there is a blind, expert-graded head-to-head test of AI against professionals on that job’s own work, such as GDPval. Only some jobs have one: 0 of the 923 jobs we score have grade A or B evidence in this release. Where there is none, we say so on the page rather than guess. How the grades work is on our quality parity method page.
These are the jobs whose quality score currently rests on grade A evidence, live from our data:
How many businesses use it
Official surveys give the wider picture. The latest figures from the US Census Bureau and the UK’s Office for National Statistics, updated as each survey is released:
Our business AI adoption tracker charts both surveys over time and by industry.
Key terms
- GDPvalA test by OpenAI in which experts compared AI’s work with professionals’ work on real tasks from 44 jobs, without knowing which was which.It is run by an AI company, so we weight it as a vendor study.More in the glossary
- Evidence gradeHow strong the proof is, from A to D. A is a blind, expert-marked test of AI against people; D means none yet.B is a test on closely related work; C is a benchmark with no human comparison.The four grades · More in the glossary
- Is it better than a person?How AI’s work compares with a qualified person’s on this job, from 0 to 100, where 50 means just as good.Also called quality parity. Most jobs have no direct test yet, so their pages say “Not yet measured.”How we judge quality · More in the glossary
- Automation and augmentationAutomation is AI doing a task in place of a person; augmentation is AI helping a person do it.On our job pages, AI does it is closest to automation and AI helps to augmentation.How we rate tasks · More in the glossary
Every term is explained in the glossary.
Try this
Pick one task your team does every week that AI might help with, such as replying to routine customer emails. For the next ten times it comes up, have the person who normally does it write down how long it takes. For the ten after that, have them use an AI assistant and write down the time again, including the time spent checking and fixing. Also note any mistakes the AI made that would have gone out if nobody had checked. You have just run a small version of the studies above, on your own work, which is the only test that counts.
Sources
- Brynjolfsson, E., Li, D. and Raymond, L., “Generative AI at Work”, The Quarterly Journal of Economics 140(2), May 2025, pp. 889–942. Read 6 October 2026. https://academic.oup.com/qje/article/140/2/889/7990658. Working paper version: NBER w31161, April 2023, revised November 2023. https://www.nber.org/papers/w31161
- Dell’Acqua, F. et al., “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality”, Harvard Business School Working Paper 24-013, 22 September 2023. Read 6 October 2026. https://mitsloan.mit.edu/sites/default/files/2023-10/SSRN-id4573321.pdf (SSRN 4573321)
- Noy, S. and Zhang, W., “Experimental evidence on the productivity effects of generative artificial intelligence”, Science 381(6654), 14 July 2023, pp. 187–192. Abstract read on PubMed, 6 October 2026. https://pubmed.ncbi.nlm.nih.gov/37440646/
- Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, arXiv 2302.06590, 13 February 2023. Read 6 October 2026. https://arxiv.org/abs/2302.06590
- METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 10 July 2025. Read 6 October 2026. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- Patwardhan, T. et al., “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks”, OpenAI, arXiv 2510.04374, 5 October 2025. Read 6 October 2026. https://arxiv.org/abs/2510.04374
- US Census Bureau Business Trends and Outlook Survey and ONS Business Insights and Conditions Survey: live figures from our business AI adoption tracker, which lists its sources.