Can Pangram Detect GPT-6 Astra?

Out of 986 prompts sourced from Chatbot Arena, Pangram 4 classified three GPT-6 Astra outputs as mixed, one as fully human, and 982 as fully AI-generated, achieving an accuracy of 99.59% on GPT-6 Astra.

Below we discuss the false negatives by category.

The benchmark was generated using a filtered subset of first-turn Chatbot Arena prompts from the LMSYS dataset (Zheng et al., 2023). I filtered out prompts that asked for code, math, very short or very long answers, or inappropriate creative writing. Then I gave those to GPT-6 Astra and ran the output through Pangram 4.

Results by category

The three mixed results were a poem, a short explanation, and an email.

Astra tends to answer short questions very briefly. Altogether, I sent 1,032 prompts in total for this benchmark set, but 46 of Astra 6's responses were under Pangram's 50-word minimum and weren't scored.

Conclusion

Pangram is robust to GPT-6 Astra on natural language prompts, and detects it with 99.59% accuracy. Pangram is able to generalize to new models because new model releases tend to inherit much of their predecessors' style and voice, so Pangram doesn't need to be retrained for every new release.

Here's how GPT-6 Astra compares to our benchmarks of other recent frontier models:

If you want to check a specific document, you can try Pangram here.