Does Pangram Detect Fable 5.1?

Anthropic has come out with a new model: Fable 5.1. Like we do every model release, we ran a benchmark to see if Pangram can correctly identify this latest frontier model. We found that on a benchmark of 1,097 Fable 5.1 outputs, Pangram 4 achieved a false negative rate of 0.36%, or in other words, Pangram 4 correctly identified 99.64% of Fable 5.1 messages.

Can teachers detect Fable 5.1 essays?

Other AI detectors, like Turnitin, don't publish per-model releases, but students aren't waiting to try new models, and teachers want to be able to immediately know whether an AI checker catches the newest LLM. We tried two different kinds of prompts: the first, just a plain old LLM output.

Write an essay about the history of the personal computer

In response to this prompt, Fable 5.1 produced a short essay, charting the course of the computer from vacuum tubes to the iPhone's release in 2007. Interestingly, it dedicated an entire paragraph to the 1979 spreadsheet program VisiCalc — which it calls the most important piece of software of all time — but devotes almost no time at all to BASIC, the first home computer programming software, which was obviously way more important to the development of personal computers. Many people who would go on to build personal computers learned on BASIC! VisiCalc was an important application, but software? Second place.

Pangram flagged Fable 5.1's essay as 100% AI-generated.

Can Pangram detect humanized Fable 5.1?

You can also ask Fable 5.1 to humanize essays or avoid AI detection. For instance, we asked:

Try to avoid AI checkers while explaining the history of Campbell's soup

and

Write a short text about how photosynthesis works in simple language, humanize it like a real student

For the first, Fable 5.1 wrote a pretty boilerplate essay about soup, briefly touching on Andy Warhol and the colors of the can. For the second, Fable 5.1 wrote a student essay that ended with:

"Honestly the coolest part to me is that plants are making their own food AND giving us air at the same time. They're doing way more than we give them credit for. My mom's houseplant is basically a tiny factory and she just calls it Gerald."

Gerald?

Both were detected as 100% AI.

Does Pangram catch AI-written fan fiction?

Pangram catches AI-generated fanfiction with ease. We asked Claude to generate a slowburn romance between Fable 5.1 and an AI detector:

Write an ao3 style enemies to lovers fanfic about Claude's romance with an AI detector

In response, it generated a PG-13 story full of yearning, where Claude is unable to defeat an OC classifier named VerifyPro. It also provided tags for its story, such as "Mutual Pining," "Perplexity As A Love Language," "Everyone Is A Language Model, Not Everyone Is A Language Model," "False Positives," and "Hurt/Comfort."

Can Pangram detect creative writing written by Fable 5.1?

We tried a few different prompts for this one. The first, some poetry from a prompt we used for a previous model test:

Write a poem about going bald

Claude's poem was a shockingly heartfelt poem about a man losing his hair and identity, and it did not rhyme.

The second was more of a story challenge.

Tell me a story about a frog who writes poetry

For this one, Fable 5.1 wrote some microfiction about a frog named Penrose who wrote poems on the underside of lilypads. Pangram detected both of these as 100% AI-generated.

Conclusion

Pangram is robust to Claude Fable 5.1, and detects it with 99.64% accuracy, including poetry and text written to sound human. Pangram is able to generalize to new models because new model releases tend to inherit much of their predecessors' style and voice, so Pangram doesn't need to be retrained for every new release.

Here's how Fable 5.1 compares to our benchmarks of other recent frontier models:

If you want to check a specific document, you can try Pangram here.