What happens when an AI detector makes a mistake?

As artificial intelligence becomes increasingly widespread, teachers want to be able to tell if students are using AI to write their assignments for them. An AI detector seems like the natural solution. But many teachers are hesitant. What happens if the software makes a mistake?

That concern makes sense. Until now, AI detectors have been fairly unreliable. Numerous studies have found that these tools often fail to recognize when a writing sample was produced by AI, and that a determined cheater can throw them off by weaving in some light paraphrasing or misspellings. These false negatives are troublesome enough, since they allow some computer-generated writing to slip through undetected.

But more worrying is early detectors’ high rate of false positives. These occur when a piece that was actually written by a human gets flagged as AI, leading to stress and unwarranted discipline for students who have done nothing wrong, unnecessary policing by teachers, and a breakdown in trust at school.

The problem is particularly acute among students for whom English is not their first language. Several AI detection tools exhibit bias against non-native speakers. One 2023 paper from Stanford found that several detectors unanimously (and falsely) identified 1 in 5 essays written by a non-native English speaker as AI-generated. Nearly all of them were mistakenly flagged by at least one of the detectors.

How often do AI detectors make errors?

Most popular AI detectors acknowledge that they make these kinds of mistakes fairly often. TurnItIn, for instance, advertises a false positive rate of about 1 in 200, meaning for every 200 papers a teacher runs through it, one student’s original work will be falsely marked as AI-generated. Other tools advertise false positive rates between 1 in 500 and 1 in 100, while independent studies have found that the numbers can creep even higher.

Pangram’s false positive rate, on the other hand, is just 1 in 10,000, measured through testing on tens of millions of documents. Our model is particularly reliable when it comes to texts of more than a few hundred words written in complete sentences—exactly the kind of writing students usually submit for big assignments.

How can teachers know if an AI detector has mistakenly flagged student writing?

When an AI detector flags a piece of text as AI-generated, a teacher has a few options to help confirm the result. First, they should simply ask the student about AI use, approaching the conversation with humility. If the result actually was a mistake, the student may be able to show evidence of their writing process, like a robust revision history in Google Docs or copies of early drafts. In this case, teachers can acknowledge that they’ve likely found an extremely rare false positive. The student should also be able to discuss their writing process in detail. This conversation could illuminate a deep understanding of a submitted assignment, suggesting the student did indeed write it themselves. On the other hand, it could reveal that the student used AI in a way they didn’t realize was wrong, confirming the detector’s findings.

If the student continues to insist that they did not use AI, but can’t provide proof or speak about their work in a way that makes sense, it’s still ok to give them the benefit of the doubt. After all, it would be incredibly damaging if they were punished for something they didn’t do. In this case, teachers can instruct students to keep records of their writing process in the future, which will help clear up any further misunderstandings. If the student is knowingly lying about using AI, they will likely think twice about doing so going forward. But if their work keeps being flagged by an accurate AI detector like Pangram, it’s probably time to escalate the situation. The odds of one mistake are already small; the odds of multiple mistakes are miniscule.