🔍 Read the full analysis: Does ChatGPT Build Data Science Skills That Last? on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. In a later test without ChatGPT, researchers found no evidence the treated group had retained more technical knowledge.
A randomized trial with Boston Consulting Group consultants found that access to ChatGPT improved performance on coding, statistics and prediction tasks outside many participants’ usual expertise, but researchers found no evidence of better technical knowledge when participants later answered questions without the tool. Published in Nature Human Behaviour on October 26, the study suggests that AI assistance can expand what workers accomplish while it is available, without showing that the skills persist independently.
The trial assigned consultants to use either GPT-4 or conventional online resources, including Google and Stack Overflow. The ChatGPT group received 15 to 20 minutes of prompting instruction; the control group received training on finding information through search engines and sites such as Stack Overflow and Khan Academy. Participants then completed two of three 90-minute tasks: Python data cleaning, statistical analysis, or prediction modeling.
Across the tasks, ChatGPT users scored 49 percentage points higher in coding, 20 points higher in statistics and 17 points higher in prediction than the control group. These are differences between the study groups, not percentage increases over a stated baseline. The researchers also compared results with 44 BCG data scientists who completed the tasks without AI. On average, consultants using ChatGPT remained below that expert benchmark on all three tasks.
The study’s main analysis included 487 consultants who submitted work on both assigned tasks. Afterward, participants answered five technical questions on related topics without ChatGPT. The authors report that the treated group did not perform better than the control group on this follow-up test, which covered probabilities, machine learning and coding.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Performance Is Not Skill Retention
The findings separate doing a task with AI from learning how to do it unaided. For employers, that distinction affects how AI-assisted work should be described in training records, hiring decisions and assessments of staff capability. A strong result produced with a chatbot may show that a worker can use the tool effectively; it does not, by itself, establish that the worker has acquired the underlying technical skill.
The immediate performance gains could still matter to organizations. Consultants completed coding and prediction tasks faster on average with ChatGPT, by about nine minutes and 14 minutes, respectively, according to the study. The researchers found no significant time difference for statistics. But relying on AI-supported output also makes it important to check accuracy and determine whether staff can recognize errors or handle work when the tool is unavailable.
The paper also reports that ChatGPT users became more confident that the system could solve problems it could not. The authors describe this part of their analysis as exploratory, so it should not be treated as a settled finding about all users or workplace settings. It nevertheless highlights a practical concern: perceived capability and actual performance may not always match.
Products Worth Considering
BowFlex Results Series SelectTech Adjustable Dumbbells, Pair
Levoit Core 400S-P Smart Air Purifier for Large Rooms
Levoit Eleva 300X Air Purifier for Home Bedroom
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment took place at BCG in late March and early April 2024 and was preregistered before data collection. Researchers allocated 986 consultants to the two groups, though 573 began the survey and 487 submitted work on both tasks. Participants all had at least a bachelor’s degree, and the authors say most reported no coding experience.
BCG data scientists and OpenAI’s Economic Impacts Research team designed tasks intended to make ChatGPT useful without allowing it to answer every question correctly on its own. The setup aimed to test how people worked with the model, rather than whether they could simply copy a complete chatbot solution. The expert benchmark came from 44 BCG data scientists working without AI.
The authors describe the result as temporary upskilling and compare generative AI to an exoskeleton: a system that extends a person’s abilities while being used. That framing is the researchers’ interpretation of this trial, not proof that all AI use fails to teach lasting skills. The study examined particular tasks, a particular employer and a short period of tool use.
“The authors find no evidence that the ChatGPT group did better on technical questions after the tool was removed.”
— The study authors
Limits of the Skill Retention Test
The follow-up measured performance on five technical questions after the work tasks; it did not establish how participants would perform after longer training, repeated practice or different forms of instruction. The reported lack of evidence for an advantage is not proof that no participant learned anything, or that learning could never occur through AI-assisted work.
The study also focused on consultants at one firm, specific data-science tasks and GPT-4. It does not settle whether results would be similar in other jobs, with other AI systems, or over longer periods. The source material does not provide enough information to determine how durable any effects might be after extended use or whether a different training program would change the outcome.
The authors’ finding about increased confidence in ChatGPT’s ability to solve problems it could not is explicitly described as exploratory. Its scale and implications need further evidence. The study’s findings should not be generalized into a claim that AI cannot support learning; they show that this trial did not detect an advantage on its unaided follow-up test.
Testing Longer-Term Learning
The reported study establishes a short-term comparison between AI-assisted task performance and an unaided technical test. The next evidence needed is whether longer-term training and practice lead to retained skills, and whether workers can apply them to new tasks without assistance. The source material does not announce a follow-up study or a date for further results.
For organizations applying these findings, a practical next step is to distinguish AI-supported output from independently demonstrated skill when evaluating training or assigning technical work. Separate assessments without the tool could help measure what workers can do on their own, while checks of AI-assisted work could assess whether they can identify errors. The trial does not prescribe a company policy, and its results do not establish which approach works best.
Key Questions
Did ChatGPT improve consultants’ data-science task scores?
Yes. In the trial, the ChatGPT group scored 49 percentage points higher in coding, 20 points higher in statistics and 17 points higher in prediction than the group using conventional online resources.
Did the ChatGPT group reach the data-scientist benchmark?
No. On average, consultants using ChatGPT remained below the 44 BCG data scientists who completed the tasks without AI. The study reports that some participants with coding experience came closer on coding and statistics.
Did the consultants retain more knowledge after ChatGPT was removed?
Researchers found no evidence of an advantage for the ChatGPT group on five technical questions completed without the tool. That result does not prove that no individual learned anything.
Can the results be applied to every workplace or AI tool?
The trial involved consultants at one firm, specific data-science tasks and GPT-4. It does not establish that the same results would hold across other occupations, models or longer training programs.
Source: ThorstenMeyerAI.com
