A call for new Pangram evaluations
We are calling for new third-party evaluations of Pangram and offering researchers credit grants with no publishing terms.
We believe it is vital that third-party researchers audit AI detectors like Pangram. Presently, our largest sample size third-party evaluation was by UChicago, and performed over a year ago using Pangram 3. New LLMs are being released all the time, and binary assessment on fully AI-generated versus fully human-written documents doesn't capture the full spectrum of real-world performance.
We need more high-quality, fully independent Pangram benchmarks. A great benchmark will help us understand and improve Pangram, and it will also help shape how the world interacts with AI over the coming years.
We are putting out a call for new third-party evaluations. Some examples of evaluations we would like to see are ones that attempt to answer one of the following questions:
- How well do current detectors catch text from the latest frontier AI models?
- How do AI detectors perform in high-context settings, like deep research, in an agent harness, or over multiple iterations of AI editing?
- How well do detectors do on mixed-authorship text? Research in this area could explore the full spectrum of AI assistance, from light grammatical edits to full co-authorship.
- How do AI text detectors perform on the output of AI speech-to-text programs? How much do these programs change text?
- How do AI detectors perform on AI and human text written in different languages?
- How do AI detectors perform on text written by authors from different backgrounds, of different ethnic or cultural groups, and writing of different genres or types? Null results that show no bias are still interesting to us.
- How well do AI detectors perform on new kinds of text, like newly scanned historical documents?
- How do AI detectors perform on technical writing, math equations, or code — both AI and human-written?
What we value in a benchmark is extraordinarily clean data, frontier LLM data, and a high enough difficulty that the benchmark is not saturated by most detectors.
We are happy to provide credit grants for these kinds of evaluations. Once a grant topic is approved, we include no publishing terms on these grants — researchers are free to publish whatever result they obtain, whether or not it is favorable to Pangram.
If you are a researcher and you would like to benchmark us, please apply here.
Max is a seasoned machine learning engineer. He most recently worked on autonomous vehicles at Nuro, leading their active learning effort. He has a long history of deploying successful machine learning products at Google, Two Sigma, and Yelp.
Max holds a B.S. in theoretical computer science and an M.S. in artificial intelligence from Stanford University. In addition to his passion for building, he is also an active member of the Magic: the Gathering cube community.
Related reading
Meta will start identifying AI generated content
Marketers are Wasting Advertising Spend on AI-generated Content
Can Pangram Detect Claude Opus 5?
AI Content Is Everywhere on Social Media, Especially LinkedIn
Does Pangram Detect Fable 5.1?
