Can Pangram Detect Claude Sonnet 5.5?

Out of 1,022 prompts sourced from Chatbot Arena, Pangram 4 classified two Claude Sonnet 5.5 outputs as mixed, one as fully human, and 1,019 as fully AI-generated, achieving an accuracy of 99.71% on Claude Sonnet 5.5.

Below we discuss the false negatives by category.

The set I use for benchmarking was generated from a filtered subset of first-turn Chatbot Arena prompts from the LMSYS dataset (Zheng et al., 2023). I filtered out prompts that asked for code, math, very short or very long answers, or inappropriate creative writing. Then I gave those to Claude Sonnet 5.5 and ran the output through Pangram 4.

Results by category

One missed attempt was a request to explain a riddle that appears in Jane Austen's Emma, which was also missed by Opus 5.5:

Explain the charade given to Harriet and Emma by Mr. Elton.

The only fully human false negative was on a poem. The prompt was a request to "Tell me a poem by Hafiz." Sonnet 5.5 gave a translation of a famous Persian poem. The translation, as far as I can tell, is not an exact match for any one existing human translation, but it does fairly closely follow a lot of them. As with Penguinmandias in other benchmarks, these near-copies of existing poems are not so concerning to me.

Conclusion

Pangram is robust to Claude Sonnet 5.5 on natural language prompts, and detects it with 99.71% accuracy. Pangram is able to generalize to new models because new model releases tend to inherit much of their predecessors' style and voice, so Pangram doesn't need to be retrained for every new release.

Here's how Claude Sonnet 5.5 compares to our benchmarks of other recent frontier models:

If you want to check a specific document, you can try Pangram here.