Is Pangram accurate?

Accuracy is a measure of how effective Pangram is at correctly labeling text as either AI-generated, AI-assisted, or human-authored. Accuracy is defined by two different kinds of errors, each of which is measured separately:

1. False-negative rate: How often Pangram misses an AI-written or AI-assisted document, mistakenly labeling it as human-written
2. False-positive rate: How often Pangram mistakenly labels a human-written document as AI-generated or AI-assisted

Pangram is a tool that makes statistical predictions. We deliberately tune Pangram to be more inclined to produce false negatives rather than false positives; we have made this choice because while an occasional false negative can be annoying, a false positive can cause real harm.

If you encounter a false positive or false negative, you can report it using our feedback form. User reports are used for manual human review and diagnostics only — we never train on or benchmark on user data.

How is Pangram’s accuracy measured?

To measure Pangram’s accuracy, it is tested on large datasets of documents where each item has a known author, either human, AI, or a mixture of the two. To ensure benchmarks produce reliable results, we never train models on our benchmarking sets, nor do we benchmark on text where the ground truth is uncertain.

Pangram’s accuracy is tested both internally, by us, and by third-party researchers. We are always looking for more independent evaluators — if you are a researcher interested in putting us to the test, we offer credit grants! Please contact us about an API grant.

Benchmarking sets

We test Pangram on many different kinds of data, including human-written, AI-generated, AI-assisted, and mixed human-AI content.

Our human-written benchmarking data is composed of commercially licensed and proprietary writing encompassing a broad set of genres and domains. We never benchmark on data from user-submitted writing or pirated content.

We want our AI-generated benchmarking sets to reflect as wide a variety of real-world AI writing applications as possible. That is why our headline AI-generated benchmark is prompted using a set of open-ended writing requests we obtained from the LMSYS-Chat-1M dataset of anonymized LLM conversations.

We also benchmark on several publicly available datasets compiled by external researchers; for more information, see the third-party benchmarks on the Pangram 4 model card.

Pangram’s performance on different kinds of writing

Pangram’s target false-positive rate is 0.01%, or 1 in 10,000. In our internal benchmarking, Pangram 4 does slightly better than that, achieving approximately 41 false positives per 1,000,000 documents for an observed false-positive rate of 0.0041%. Our headline false-negative rate, measured on nearly 520,000 AI-generated documents, was 0.3396%.

All our benchmarks use a strict definition of false positive and false negative: any human-written segment that is flagged as AI-generated or AI-assisted — even if it is a small fraction of the total text — is reported as a false positive. Likewise, in the case of AI-generated documents, any fractional segment flagged as human-written or AI-assisted is reported as a false negative on the whole document.

To ensure the reliability of our benchmarks, we measure Pangram’s performance on a variety of human and AI-generated writing. Below, you will find information on how Pangram performs on different domains and genres, AI models, and kinds of text.

Domain and genre

Writing style, technique, and structure vary across different types of writing, such as essays, fiction, or journalism, and high-level benchmarking can obscure error rates of particular domains. That is why we break down our benchmarks into domain-specific subsets and examine both the false-positive and false-negative rates on those slices.

False-positive rate by domain

We test a variety of human-written slices to test if any display a disproportionate false-positive rate. Overall, our performance on our primary use cases is quite good, including creative, academic, and scientific writing.

In this domain breakdown, Amazon reviews, news, and how-to articles all include multilingual data.

Poetry had the highest false-positive rate among domains — poems are harder to classify because they tend to be short and highly formatted. While the false-positive rate is still fairly low, we would recommend additional scrutiny for this type of writing.

False-negative rate by domain

Pangram 4 has been evaluated on its ability to detect a wide array of AI-generated writing, including academic writing like essays, research papers, and scientific papers, as well as creative writing and fiction.

It was also tested on a random sample of AI-generated responses to anonymized user prompts sourced from real world LLM prompts from the LMSYS-Chat-1M dataset. These prompts included scientific and technical writing, creative writing, and business writing.

Different and new AI models

Not all AI models write in the same way. At Pangram 4’s release, we used the LMSYS-Chat-1M dataset to generate 20,000 examples from 26 mainstream language models for a total of 520,000 samples. Below is the false-negative rate for 15 of those models, and you can view the full table on the Pangram 4 model card.

AI models are not static either. New AI models are being released all the time, and AI companies are focusing on the writing style and quality for new releases, so it’s reasonable to expect that the style and voice of the models will continue to evolve. To keep current, we run a smaller benchmark on each new model as it is released and publish it on our blog between our own model releases.

Pangram continues to perform well, even on unseen models, because new models inherit much of their writing style from their predecessors. You can read our latest model benchmark post, Can Pangram detect Claude Sonnet 5.5? here, and learn how Pangram achieves a low false-negative rate on unseen models here.

Mixed authorship

Text isn’t binary: not all documents are either fully AI-generated or written entirely by a human. Oftentimes, AI and human authorship are intertwined by iterative editing, refinement, and other back-and-forth processes. Ideally, Pangram should be able to distinguish between a collaborative document and one that has been generated with minimal human oversight, and reflect that difference in detection results.

The target label will depend on the degree of AI use that was involved in the production of the document. For instance, light AI touches, like proofreading or copy editing should usually retain the label of fully human, while more substantial rewrites, like suggesting wording and phrasing changes for clarity should generally receive a label of AI-assisted. Large AI rewrites, on the other hand, could be labeled AI-generated.

The label should also depend on how AI and human writing are combined. For instance, going back and forth with an AI to rewrite a paragraph would be labeled as assisted, whereas asking an AI to expand a human-written claim into a longer paragraph with supporting evidence would receive the label of mixed human and AI-generated content, reflecting the distinct authors of different sections.

To encompass all these different ways of using AI to write, we evaluate Pangram on datasets with varying levels of AI involvement.

Pangram does well on AI-assisted and AI-edited text. Out of an abundance of caution, Pangram leans human when encountering mixed text.

Diverse authors

We have evaluated Pangram for bias against writers from diverse backgrounds. On a dataset of 25,996 essays written by students of different disability statuses, gender, grade levels, ethnicities, English-language learner statuses, and economic backgrounds, Pangram demonstrated 0 false positives. See the full breakdown of results on diverse student essays.

One study examined Pangram 4’s performance on 143,755 documents, including 85,887 documents from autistic writers, and found Pangram showed no appreciable bias. Among 89,903 pre-2022 documents written by both neurodiverse and control writers, Pangram 4 demonstrated a false-positive rate of 0.0011%. You can read the full independent study on neurodiverse writers.

In addition to English, Pangram supports Arabic, Chinese, Czech, Dutch, French, German, Greek, Hindi, Hungarian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish, Ukrainian, Urdu, and Vietnamese.

For benchmark results by language, see Pangram’s false-positive rate by supported language.

Limitations

Short text

Pangram is designed to detect AI-generated text in natural language prose of 50 words or more, written mostly in complete sentences. Pangram is most reliable on longer spans of text, which give the AI detector more information to make accurate predictions.

Below, you can see a breakdown of Pangram’s performance on different lengths of text. Due to the false-positive tradeoff, many of the errors seen on short text will be false negatives.

Other limitations

There are some kinds of text that we consider out of scope at this time, and for which Pangram can produce less reliable results. These include:

• Code, markup, or Markdown
• AI translations and machine translations, such as Google Translate
• Document front matter, including tables of contents and title pages
• Document back matter, like indices or reference lists
• Figure text, like axis labels or tabular data
• Mathematical notation
• Highly formatted text, like poetry or recipes

Best practices

• For best results, human-written headers, footers, instructions, and other extraneous material should be removed before scanning a document.
• Raw text and .docx files are recommended over PDFs when available, because PDF parsing can sometimes introduce unintended artifacts. PDF results should always be double-checked.

FAQ

Why do you measure false-positive rate and false-negative rate separately?

Accuracy is measured by running Pangram on a set of example texts and comparing the assigned label with the label we know the document has. Thus, false-negative rate and false-positive rate are measures of two different things:

• False-negative rate wants to know: given an AI document, how likely is Pangram to label it human?
• False-positive rate wants to know: given a human document, how likely is Pangram to label it AI?

There is no false-positive rate on a set of AI-written documents, because any positive (AI) label Pangram assigned to an item there wouldn’t be a false positive — by definition, it would be a true positive. This is why false-positive and false-negative rates have to be measured and reported separately.

Why does Pangram scan short texts if it is less accurate on them?

Pangram’s false-positive rate on text between 50 and 100 words is 0.02%, or roughly 1 in 5,000. While this is high enough — and short texts online are numerous enough — that the average user may encounter a short-text false positive in the normal course of use, it is still low enough that we believe Pangram can provide useful information much of the time.

We always suggest scanning the longest span of a document that is available. More words give Pangram more information to make a prediction, which is why we see accuracy improve with higher word count.