Pangram 4 Technical Report

Pangram 4 is our latest state of the art AI text detection model. Compared to its predecessor, Pangram 4 shows marked improvement on distinguishing fully AI-generated, mixed author, and AI-assisted writing while delivering superior granularity, which allows for a finer distinction between human and AI segments within a document. It is also robust against humanizers and adversarial prompting, and is significantly more accurate on the latest frontier models.

To our knowledge, Pangram 4 is the first AI detection model that is able to differentiate AI-assisted and AI-human mixed writing simultaneously in a single pass.

Key results:

• On our benchmarks, Pangram 4 correctly identifies 99.66% of AI-generated documents while flagging human writing as AI only 0.0041% of the time, or roughly one false positive for every 24,000 documents.
• Pangram 4 improves on Pangram 3.3.2 in both directions, with nearly 6x fewer false negatives and 14x fewer false positives.
• Pangram 4 does not mistake light AI polish for AI generation: it flags AI-polished human writing as fully AI-generated only 0.009% of the time, a 20x reduction over Pangram 3.3.2.
• When evaluated against 13 popular commercial humanizers, Pangram 4 identifies AI involvement in humanized AI text 98.83% of the time.

Mixed authorship and AI-assistance

Many written documents produced today no longer fall neatly into the category of either fully AI-generated or fully human-written; rather, we live in a time where a text's prose and its argument can plausibly have different provenance.

We believe distinguishing this category of document is important, both from fully AI generated and human text. This is important to us both because we believe in transparency and legibility in authorship, but also because as both a popular and emerging technology, language models are currently shaping the world in ways we do not yet fully understand – but we do have some indication. Some research indicates that using language models to edit text alters its style in measurable ways: academic papers rewritten by an LLM accumulate hedging words ("may," "typically," "suggests") that the authors did not write. Related work finds that LLM edits shift a text's meaning even when the model is instructed to correct only grammar, and that LLM-written peer reviews weight clarity and significance differently than human reviewers do. Elsewhere, there are indications that it is difficult to remove the signatures of an AI-generated draft even with extensive editing: writers who edit LLM output toward their personal style still produce text that reads closer to LLM writing than to their own.

To approach this problem, we consider two main ways that human and AI authorship can be combined. For heterogeneous documents, where segments are cleanly attributable to either an AI or a human author, Pangram uses a per-token prediction. On the other hand, for an iteratively edited segment where no token can be definitively attributed to a single AI or human author, Pangram flags the segment as AI-assisted.

Pangram is robust against humanizers and adversarial prompting

As AI detection technology improves, so too does the demand for software that rewrites AI-generated text in order to evade automatic detection. These tools often aim to undermine detection in settings where authorship is crucial, such as academic, research, or publishing. To combat this, Pangram 4 has a dedicated capability for detecting humanized text. When evaluated against 13 popular commercial humanizing tools, Pangram identifies AI involvement in humanized AI output 98.83% of the time.

Another common adversarial strategy for evading detection is adversarial prompting, such as the Blader humanizer, wherein a user instructs an LLM to diverge from its typical writing style. To assess our performance on that domain, we granted an AI agent (Codex GPT 5.6 Sol) access to the Pangram 4 API for 24 hours with the goal of engineering repeatable false negatives.

In its effort, GPT 5.6 tested a variety of strategies, including style imitation, terse professional registers, and pre-loading context with human source text. In the test period, it only managed a single successful bypass, by mimicking dictated surgical pathology notes. While impressive, we ultimately consider this attempt out of scope, both because the input contained more context than the output, and because the resulting text was terse bullet points rather than the open-ended prose we typically target.

Training and Architecture

For the purpose of determining authorship, we define AI-generated text to be original, natural-language prose produced by an LLM in response to an open-ended writing task or question. We additionally require that the LLM contribute original tokens and exclude cases where the model verbatim parrots human-authored text; for example, quoting from the context window, reciting memorized content, or abridging human input strings.

Training data

We trained Pangram 4 on both human-written and AI-generated writing from a wide variety of domains, languages, and AI models.

As in previous versions, all human-written training data is commercially licensed or owned by the company. As ever, Pangram 4 was not trained on any user-submitted data, customer data from API users, or data obtained from unauthorized internet crawls.

A rough distribution of the domains in our human dataset is covered below.

For the AI portion of our dataset, we generate all synthetic examples in-house using 75 of the most popular models, including all presently available OpenAI and Anthropic models. Our goal when constructing our synthetic dataset is to provide context to help the model learn how a text was written, as opposed to absorbing topic, language, or tone markers of AI text.

To reduce false positives on unusual human writing, run a trained candidate model on a reserve pool of human texts in order to identify items that the candidate mislabels as AI-generated. We then generate synthetic mirrors of the mislabeled items and retrain the candidate on the combined set. We call this method hard-negative mining.

Architecture

Pangram 4 is our largest model yet, with 6x as many parameters as Pangram 3.3. Pangram 4 features several major architecture changes to its. It is also our first mixture of experts model, wherein a sparse routing mechanism directs different kinds of input to different parts of the neural network. We believe that this type of architecture will be well-suited to AI detection due to the wide span of input possibilities.

Prediction mechanics

In order to increase the resolution of its predictions, Pangram 4 predicts three scores for each token in a given input string, corresponding to its estimate of whether that token was human-written, AI-assisted, or AI-generated. Longer documents are scored in overlapping 512-token windows such that for each window, we also obtain a window-level score along with an aggregation of the token-level scores. All the scores are then averaged and smoothed to discourage spurious flickering of individual token predictions. This method replaces the previous approach of naively chunking the text during preprocessing, and it is what allows us to precisely infer the boundary between AI and human content.

Performance and benchmarks

In order to accurately model real-world LLM use cases, we evaluated Pangram 4 on human, AI-generated, and AI-assisted text over a variety of standard and challenge benchmarks.

False positives

For the human portion of the test set, Pangram 4 was benchmarked on a set of 2 million commercially-licensed pre-2021 documents that were held out from model training.

On this benchmark, Pangram 4 achieves a false positive rate of 0.0041%, with a 95% confidence interval of (0.0032%, 0.0050%). This is one false positive for roughly every 24,000 documents.

False negatives on the AI-generated dataset

We constructed this benchmark by sourcing user prompts from Chatbot Arena Conversations; we selected this collection of prompts in order to reflect the real-world distribution of prompts users send to LLMs. We then used a small language model (Mistral-Small-24B-Instruct-2501) to filter out any prompts that requested math or coding help, as well as multiple-choice answers, as we considered these out-of-scope. After filtering, we were left with a comprehensive set of 20,000 prompts that elicited open-ended responses from the models. From each prompt in this set, we generated a response for every language model in our set of 25 models, including GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, for a total of 519,993 samples.

On this benchmark, Pangram 4 achieves an overall false negative rate of 0.3396%, compared to Pangram 3's FNR of 1.99% on the same dataset.

Robustness to frontier models

Pangram 4 generalizes across frontier language models: every model family is detected with an FNR below 0.7%, and we find no significant correlation between a model's release date and its detectability.

Performance on mixed and edited documents

"We create a dataset of AI-proofread and AI-polished text to evaluate our AI-Assisted False Positive Rate, the rate at which a model incorrectly flags minor edits made by AI as AI. We use consumer products (Grammarly, Apple Intelligence, and Gemini in Google Docs) and frontier LLMs (Opus 4.8, Gemini 3.1, and GPT-5.5) to apply light edits to student writing from ELLIPSE, PELIC, and ICNALE. For example, we used the Writing Tools feature embedded in Apple's Pages word editor to "Proofread" some texts, or prompted Opus 4.8 to "Correct minor academic style issues while keeping the draft recognizably human."

What's next

We are excited to present this model, which we believe to be the current best of its kind in the world. As Pangram is a statistical model, however, there is always room to improve, and for this model series, we have a few goals we’d like to achieve.

First, we hope to preemptively address data drift before we begin to notice its effects. It is not presently known how long a pre-2022 training set will continue to be sufficient in producing the kinds of robust predictive models our standards demand, and so finding methods to augment our human-authored training set is a top priorities. This will likely require us to get creative, as we are extremely preoccupied with maintaining a guaranteed human ground-truth value for all items in our training set.

Second, we’d like to ensure the stability of Pangram 4’s predictions across contexts. Occasionally, a subset of a longer text will receive different predictions depending on whether it is isolated or embedded in the surrounding document. It may be the case that this situation reflects how additional context provides the model with more information or a stronger signal, but we hope to be able to minimize this effect with better calibration.

Finally, we’d like to continue our interpretability work to improve our understanding of feature attribution, embedding space, and generator identification. This is some of our most ambitious work, and we are eager to share our progress in the coming months.

Model card

Full details are in the Pangram 4 model card.