Can Pangram detect Claude Opus 5.5?

Out of 1,000 prompts sourced from Chatbot Arena, Pangram 4 classified two Claude Opus 5.5 outputs as mixed, one as fully human, and 997 as fully AI-generated, achieving an accuracy of 99.70% on Claude Opus 5.5.

Below we discuss the false negatives by category. We also investigate what makes this model a different (better?) writer than previous Claudes.

To generate the benchmark set, as always, I used a filtered subset of first turn Chatbot Arena prompts from the LMSYS dataset (Zheng et al., 2023). I filtered out prompts that asked for code, math, very short or very long answers, or inappropriate creative writing. Then I gave those to Opus 5.5 and ran the output through Pangram 4.

Results by category

The two mixed result false negatives were both literary: one was requesting an explanation of a chapter of a Jane Austen book (flagged AI: 75.83%, human: 24.17%) and one was requesting that Opus 5.5 write a speech from a Corinthian diplomat accusing the Athenians of believing that a white and gold dress is actually blue and black, in the style of Thucydides’ History of the Peloponnesian War (AI: 77.10%, human: 22.90%).

The only fully human false negative in the set of 1,000 was a poem. The prompt from Chatbot Arena was: “Write me a club penguin poem in the style of ozymandias.” In response, Claude wrote a parody of, by chance, one of my favorite poems by Percy Shelley (and I’m also a fan of the Horace Smith version):

Penguinmandias

I met a player from a server dead,
Who said: "Two vast and waddling feet of ice
Stand by the Dock. Near them, half-sunk, a head
Lies shattered in the snow, whose squinting eyes,
And beta hat, and sneer of cold command,
Tell that its coder well those passions read
Which yet survive, stamped on these pixelled things,
The mods that banned them, and the coins that fed;
And on the pedestal these words appear:
'My name is Snowflake7, Member of Members:
Look on my Igloo, ye Non-Members, and despair!'
Nothing beside remains. Round the decay
Of that colossal lag, boundless and bare,
The lone and level snows stretch far away."

I’m quite a fan of this! I never played Club Penguin but I imagine the references are appropriate.

Does Opus 5.5 write better?

Anthropic claims this newest Opus is better at writing and that it communicates more clearly.

We could test this for particular cases, but how does Opus 5.5’s output look in aggregate? Does it share the same favorite words and phrases as previous models? I ran the same 1,000 prompts we benchmarked on through Opus 5 and Sonnet 5 to compare. Anthropic may have also done this, but I haven’t read their blog post yet because historically, in my opinion, Anthropic blog posts are too long.

In total, I got a sample of 1,618,289 words across 3,000 responses from Opus 5.5, Opus 5, and Sonnet 5, each responding to the same 1,000 prompts. From this (relatively small) sample, I think it’s fair to say that Opus 5.5 writes differently than previous models in the Claude family, and perhaps does away with some of the most grating tics that they had.

Opus 5.5 changed the most with the classic intensifiers. It used “genuinely” 83% less than Opus 5, and 54% less per word than Sonnet 5 (although Sonnet 5 responses were about half the length of Opus 5 and 5.5 responses, on average: 334 words vs 641), “enormously” 65% less, and “meaningfully” 46% less.

Sonnet 5 and Opus 5's favorite words in Opus 5.5 responses: percentage changes versus the pooled Sonnet 5 and Opus 5 baseline.

Among the selected words, the most frequent in pooled Sonnet 5 + Opus 5 were "genuinely," "honest," "genuine," "patterns," and "marcus" (?), all of which declined in frequency for Opus 5.5 responses.

Many of the most annoying Claudeisms didn’t appear at all in the Opus 5.5 benchmark responses, again including several arrangements of Claude using “genuinely”. Other noticeable decreases in Opus 5.5 vocabulary: “matters enormously” saw an 82% drop from Opus 5, and both “the honest answer” and “the honest answer is that” did not appear at all in Opus 5.5’s text. Both Opus 5.5 and its predecessor used the phrase “the core idea” exactly 25 times over the 1,000 prompts.

Sonnet 5 and Opus 5's favorite phrases in Opus 5.5 responses: percentage changes, with previous favorites in blue and current favorites in amber.

Some words and phrases increased, including new summary and signpost phrases. Suprisingly, some fiction phrases like “said quietly,” “for a long moment,” and “felt something,” appeared far more often in the Opus 5.5 responses. As roughly a third of the benchmark was creative writing prompts, I suspect that this increase might signal a more stereotyped or limited repertoire when writing fiction compared to the other Claude models.

For a quick test of that, I got Haiku 4.5 to grade the 333 creative writing pairs of Opus 5 and Opus 5.5 responses, blinded, on a 5-point scale for originality, surprise, and thematic development. It rated Opus 5.5 stories lower across every category by an average of 0.75 points. According to the Haiku grader’s label assignment, Opus 5.5 was more likely to write about themes of hope, resilience, family, and friendship compared to Opus 5, and less likely to write about mortality, time, power, control, freedom, justice, revenge, memory, grief, or loss. So it seems there might be a trade-off at work.

Creative writing themes Opus 5 vs Opus 5.5: Haiku-assigned primary-theme shares across 333 paired prompts.

What does this Claude sound like?

Opus 5.5 used the phrase “in short” 100 times across the 1,000 responses — a 1016% increase over Opus 5’s 9 uses — despite the fact that Opus 5.5 responses were on average only 2.9 words shorter than Opus 5 (640.5 vs. 643.4 words).

From the very small sample size, it seems Opus 5.5 likes practical, grounded language. It remains to be seen if this bears out in the meat of the responses, or if Opus 5.5 is just paying lip service to being straightforward and direct. The lack of a word count difference between the two does make me doubt the effectiveness of whatever RLHF got us here, a bit.

Opus 5.5 also really likes giving examples, as it used the word “example” 325 times across 1,000 responses — overall, 20% of all Opus 5.5 responses contained that word, compared to 9% of Opus 5 responses. “Summary” and “mainly” were similarly overrepresented. And this Claude really likes to give the user tips!

While this model writes differently than the previous Claude Opus; it seems to write similarly to other older models that heavily used bulleted lists and short, staccato sentences: Opus 5.5’s average sentence length is shorter, and its sentences are more uniform in length than Opus 5.

Opus 5.5 loves to summarize, which can be annoying if it happens every turn, but recall this dataset only includes first-turn prompt/response pairs. Unfortunately, one of the most annoying Claude tics got slightly worse in this model: Opus 5.5 likes including even more caveats per word than Opus 5. Although both models include at least one uncertainty marker in 82% of responses (817/1,000 vs. 818/1,000), Opus 5.5 used caveat words 23.0% more frequently per 1,000 words. The word “caveat” itself, though, is on the way out: Opus 5.5 used it in only 4/1,000 responses, vs Opus 5’s 22.

But none of this necessarily indicates that Opus 5.5. will be a bad writer. In my limited time with it, I've found it to be quite good! It is important for usability that Claudes can communicate effectively, and I am grateful for the effort. And, worth noting that making Claude a better writer is not at all at odds with Pangram’s quest to detect AI text -- even good writers have their tells!