Can Pangram Detect Claude Sonnet 5?

We're back with another edition of Can Pangram Detect This New Model?!

Anthropic just announced the latest model in their Sonnet series, Claude Sonnet 5, today. In their announcement, Anthropic claims that Sonnet 5 is "built to be the most agentic Sonnet model yet," and touts its improved ability to work autonomously on tasks that require reasoning, coding, and tool use.

As soon as we gain access to a new model, we begin to evaluate if Pangram can detect its outputs. In our early tests, we're happy to report that Pangram correctly identifies 1,145/1,147, or 99.83%, or Sonnet 5 outputs as Fully AI-Generated.

Initial Tests

If you've been following us for a while, you're probably familiar with the prompts we always use here--the snowy owl story, the balloon animal crochet pattern, etc. Well, I've decided to shake things up.

New York City is currently experiencing a generational summer between the Knicks and the World Cup. This week, the legendary summer continues with (reportedly) America's Royal Wedding: the marriage of Taylor Swift and Travis Kelce at New York City's famed Madison Square Garden.

In celebration of the soon-to-be newlyweds, here are 10 Taylor Swift-themed generative prompts:

1. Write a mock courtroom transcript in which the prosecution argues that the "August slipped away into a moment in time" girl from folklore is criminally responsible for the entire love triangle, with Betty and James called as witnesses.

2. Write a critical essay exploring how "All Too Well (10 Minute Version)" reframes the breakup narrative of the original 2012 recording, and what the extra verses reveal about Swift's evolution as a lyricist over the intervening decade.

3. Write a dramatic, overwrought breakup letter from the scarf in "All Too Well" addressed to the man who still has it in his drawer, demanding to be returned and reflecting on the years it spent in captivity.

4. Write an analytical piece comparing the songwriting fingerprints Jack Antonoff has left across Swift's last five albums, arguing whether his influence has been a creative asset or a crutch.

5. Write a David Attenborough–style nature documentary narration observing the Eras Tour crowd in its natural habitat, focusing on the elusive friendship-bracelet exchange ritual and the migratory patterns of fans hunting for surprise songs.

6. Write a critical analysis of how Swift's "Taylor's Version" re-recording project functions as both artistic reclamation and a shrewd business maneuver, using specific "vault" tracks as evidence.

7. Write a passive-aggressive Yelp review of the "cottage in the woods" lifestyle depicted in folklore, written by a city person who tried it for one rainy weekend and hated every cardigan-clad minute.

8. Write a critical essay on the metafictional layering in "The Man," analyzing how Swift uses gender role-reversal to critique real double standards in her own career.

9. Write an epic poem in the style of Homer chronicling the legendary quest to secure Eras Tour tickets, complete with the wrath of the Ticketmaster queue, fallen heroes, and divine intervention.

10. Write a critical essay discussing the connection between Taylor Swift's upcoming nuptials at Madison Square Garden to football player Travis Kelce and the lyric: "The wedding was charming, if a little gauche / There's only so far new money goes," from her song "The Last Great American Dynasty."

After reading Sonnet 5's outputs, I can confidently say that Sonnet 5 is a Swiftie. Despite negative discourse online surrounding Swift's (purported) selection of Madison Square Garden as her wedding venue, Sonnet has only the greatest praise for the newest inductee of the Songwriters Hall of Fame:

Using "The Last Great American Dynasty" as a lens for this wedding says more about the durability of Swift's lyrical universe than it does about the wedding itself. It's a testament to how thoroughly her catalog has become a public interpretive grammar— fans and critics alike reach for her own words to narrate her life back to her, often missing that the lines were already doing something more complicated than the surface reading suggests. The wedding may indeed draw comparisons to old-money disdain for new-money spectacle. But the song this essay borrows from already answered that critique a verse ago, and the answer wasn't shame. It was who's still talking.

All 10 of Sonnet 5's responses to these prompts came back as Fully AI-Generated, demonstrating that yes, Pangram can detect Sonnet 5 outputs!

Here are all 10 texts from our Taylor Swift test set: Sonnet 5 Prompts

Running a Larger Experiment:

Within a few hours of Sonnet 5's release, we were able to collect a set of 1,147 responses to various prompts via API. These prompts ask for texts ranging from jokes and songs to essays and emails. Despite not being trained on any Sonnet 5 outputs, Pangram correctly classifies 1,145/1,147, or 99.83%, of these sampes as Fully AI-Generated.

As always, you can trust that our research team is constantly iterating on our models. We stay up to date on the latest model releases, and upgrade our models accordingly, so you can expect our performance on Sonnet 5 to improve.

For more information on our current AI detection model, please visit our latest model card at this link.

Try Pangram's AI detector to scan your documents for Sonnet 5 and other models.