No, Pangram does not rely on perplexity and burstiness for our model predictions. Pangram is a type of neural network called a classifier model. Pangram is trained using ‘synthetic mirrors’ of the hardest documents to classify and then it is retrained over and over again. That makes it adaptable to new models, and means that as Pangram gets more experience with human text through our active learning algorithm, it gradually gets better.
What are perplexity and burstiness?
Drawn from our blog post on perplexity and burstiness - feel free to reference this for in-depth explanations on the shortcomings of these methods.
https://www.pangram.com/blog/why-perplexity-and-burstiness-fail-to-detect-ai
Perplexity is how unexpected, or surprising, each word in a piece of text is, when looked at from the perspective of a particular language model or LLM.
For example, here are two sentences. Let's focus on the last word of each sentence, for demonstration purposes. In the first example, the last word has low perplexity, while in the second example, the last word has high perplexity.
Low perplexity:
For lunch today, I ate a bowl of *soup*.
High perplexity:
For lunch today, I ate a bowl of *spiders*.
The reason that the second sentence is high perplexity is because very rarely would a language model see examples of people eating bowls of spiders in its training dataset, and so it is very surprising to the language model that the sentence ends with "spiders", as opposed to something like "soup" or "a sandwich" or "a salad".
Burstiness is the change in perplexity over the course of a document. If some surprising words and phrases are interspersed throughout the document, one would say that it is high in burstiness.
Why doesn't Pangram use perplexity and burstiness?
There are many drawbacks to using perplexity and burstiness to detect AI writing. For an in depth explanation of the following, please refer to our blog post on the topic: https://www.pangram.com/blog/why-perplexity-and-burstiness-fail-to-detect-ai. As language models are explicitly trained to minimize perplexity on their training data, frequently reproduced human texts like the Declaration of Independence and Wikipedia articles get misclassified as AI, a problem that only worsens as models ingest more of the web. These metrics are also relative to a specific model, so a detector calibrated on one generator produces inaccurate results on another, and many commercial models don't expose the token probabilities needed to compute perplexity at all. They systematically penalize non-native English speakers, whose more limited vocabulary and simpler sentence structures naturally produce lower perplexity and burstiness. And most fundamentally, they are static heuristics that cannot improve with more data and compute the way a deep-learning approach can - they cannot deliver the reliability that high-stakes applications demand.
