← Back to blog

The Biggest AI Tutor Trial Found Lower Grades. Here Is What That Means.

A University of Maryland randomized trial with 2,379 students found that access to an AI tutor lowered final grades and participation. What the study shows, what it does not, and what a tutor should do instead.

AJTeam HeyPapin·6 min read·143 views
The Biggest AI Tutor Trial Found Lower Grades. Here Is What That Means.

In October, a team led by Jing Liu at the University of Maryland released the results of a large randomized trial of an AI tutor in real university courses. The tutor did not help. On average, students who were given access to it finished with lower grades than students in comparable sections without it.

That is an uncomfortable result for anyone building AI tutors, including us. However, it is also the most useful study on the subject this year, because it tells you what not to build.

What the study actually did

The researchers worked with 2,379 undergraduates and 30 instructors at a large U.S. public university, across STEM, the social sciences, the humanities and education. In some sections of a course, students got a generative AI tutor built into the course website; in comparable sections of the same course, they did not.

The results, in the authors' words: tutor access "reduced final grades by 0.37 standard deviations." Participation in the online course activities fell by 0.90 standard deviations. Furthermore, the losses were larger for first-generation students, the group a free tutor is usually meant to help most.

The paper is "The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial", published as an Annenberg EdWorkingPaper in October 2026.

What it does not show

Before you read this as "AI tutors make students worse," three details matter.

It is a working paper. It has not been through peer review yet, and numbers can change.

It measures access, not use. According to Connect Education's summary, only about 15 per cent of students in the tutor sections used it even once. So the drop describes what happened to whole sections that had a tutor available, not what happens to a student who uses one well.

It tested one kind of tutor. It was a GPT-4o study assistant placed inside the course website. A different design could produce a different result, and in fact, earlier studies with more structured tutors were more positive. ETC Journal's look back at the 2025 Harvard physics study is a good reminder that one trial rarely settles a question in either direction.

Why a helpful tutor can make things worse

The most interesting detail is not the grade drop. It is the participation drop, which was more than twice as large.

Students who used the tutor were also more likely to say they asked their instructor fewer questions. Connect Education summarises the authors' reading of this: the tutor "may have replaced the effort and contact that normally drive learning."

In other words, the tutor did not add to the course. It quietly replaced the parts of the course that were doing the teaching: struggling with the exercise, asking the instructor, talking it through with classmates. When a tool answers instantly, all of that starts to feel optional.

We have written before about the illusion of competence: reading a clear answer feels like understanding, even when nothing has moved into your own head. This trial looks like that effect, measured at the scale of a whole university.

What a tutor should do instead

If the problem is a tutor that replaces effort, the fix is a tutor that asks for it. That idea shapes several things we build into HeyPapin, and we think they matter more than any single feature.

Ask before telling. When you start a new topic, Papin opens with a simple picture from everyday life, then asks you one question about it and waits. The explanation builds on your answer, so you do part of the thinking before you get the formal version.

Make the steps visible. Papin draws the idea one piece at a time while it talks. You see the structure being built, which gives you something to check your own understanding against, rather than a finished answer to copy.

Slow down when you are lost. In one of our own test sessions, held in a noisy classroom, the learner told Papin, "I don't understand your question." Papin replied, "No problem, I asked it in a confusing way," rewrote it as a simple choice between two options, and pointed at the part of the board it meant. The next answer was correct. A tool that only gives answers has no reason to do that.

To be clear, none of this has been tested in a trial like Maryland's. Our own numbers come from a small group of early testers, and we do not know yet whether these choices protect grades at scale. Nevertheless, the trial gives us a clear standard to aim for: a tutor should leave you doing more of the thinking, not less.

If you study with AI

You do not need to wait for the next study to use this.

Answer first, then check. Try the problem yourself before you ask the AI. Then use the AI to find where your attempt went wrong, rather than to produce the attempt.

Keep your instructor in the loop. If an AI tutor means you stop asking questions in class, that is exactly the pattern the trial found. Use the tutor to prepare better questions, not to replace them.

Redraw it from memory. After a session, close the tool and sketch the idea yourself. Whatever you cannot rebuild is the part you have not learned yet, no matter how clear it looked on screen.

Sources

#AITutor #EdTech #LearningScience #StudyTips #AIWhiteboard #HeyPapin