No, LLMs don’t just mimic human text

Some people think that AI detection isn't possible because LLMs are trained to imitate human text, but this is not the case. In fact, LLMs write in a distinctly different way than humans do, and this is what makes AI detection possible.

In his Substack essay, Freddie DeBoer writes:

"Where do [AI writing] patterns come from? They come from human written texts that are in the training corpora that are used to build LLMs. Every LLM textual pattern - every LLM textual pattern - is ultimately the product of human text production. The very nature of an LLM is to model human text based on human text."

He concludes that it is therefore not possible to build an AI detector with an extremely low false positive rate, because there's always a chance that some human can think of and write down the same text string that an LLM would output.

We commonly hear different versions of this:

"LLMs are sampling a distribution of human writing; therefore you cannot detect LLM generated text because the LLM training distribution is human."

"LLMs are trained to write like humans, so if they are doing their job, AI writing will beat any detector."

These arguments appear to be sound at first glance, but they fail to capture that modern LLMs are complex systems. Modern LLMs are trained to do much more than just match the human text distribution. In this blog post, we'll go over why and where modern LLMs, such as ChatGPT, Claude, and Gemini, diverge from the human text distribution and what gives them their unique style and voice that makes their writing possible to detect.

Summary: It is tempting to believe that because LLMs are trained to model human text, their output will become indistinguishable from human writing. But in reality, LLMs don't just approximate human text: they are post-trained to be helpful assistants that follow instructions, which constrains their writing to a narrow set of choices that are detectable algorithmically.

In The Beginning: Next Token Prediction

The story of language models began with completion models. These early language models were straightforward: they learned how words commonly followed from each other from a large set of text data. Then, a user would give the completion model a few initial words, and from that the model would predict the most likely next word.

A completion model predicts a probability distribution over the next token

Image credit: AIML.com

Under the surface, a completion model takes in an initial string, or prefix, such as "Deep learning is very," and then it outputs a list of probabilities over what is likely to come next. These probabilities are aggregated over all of the completions in a very large training corpus, mostly of Internet data. The result was a probability model of all human writing, and one that could guess what words were most likely to appear in order given what the natural frequencies of words are in the training dataset.

This gave rise to the first language models: GPT-1, GPT-2, and GPT-3. These models are now referred to as base models: they represent the raw distributions over human language, but their outputs are raw and unpolished: they don't follow instructions, and cannot conduct a conversation. Today, we call this initial stage of language model development pre-training, and it's only the first stage of making an AI assistant or chatbot — there is a lot more that happens after that.

From GPT to ChatGPT

When did we get from a raw distribution over words to being able to talk to a language model like a person? To answer that question, we have to go back to March 2022, when the first major breakthrough in turning language models into assistants happened: InstructGPT, released by OpenAI.

InstructGPT was the first time that a language model was explicitly trained not just to imitate human language, but to follow instructions given to it by a user, which we now call prompts.

The three post-training steps from the InstructGPT paper: supervised fine-tuning, reward model training, and reinforcement learning

Here is a figure from the original InstructGPT paper. The original paper demonstrated that you could apply a technique called post-training to customize a base model, in this case GPT-3, to be a helpful, instruction-following assistant — warping the style of the model to take on a specific persona.

The paper describes three steps:

1. Human writers provide examples of what they consider to be acceptable AI responses to prompt input
2. AI researchers develop a reward model that learns which of these provided responses humans tend to prefer
3. Using the reward model, AI researchers train the LLM to maximize the reward over the course of a large set of generated responses

This is similar to Pavlovian conditioning from psychology! The researchers reward the model for some outputs and not for others, and by repetition, the model learns to associate something that was previously neutral, namely, a human-preferred output response, with a positive value from the reward function. This technique is called instruction tuning, and it is what reshapes the model from a base model into an instruct model: one that can have a conversation and follow user instructions.

Instruction tuning prevents language models from going off the rails, forgetting key information from their prompt, or starting on crazy tangents. It also gives developers like OpenAI and Anthropic control over the behavior of their models. For example, models should refuse to tell a user how to build a bomb or rob a bank, even if that is the user's desired response. Without post-training, model alignment, that is, the ability to make large language models helpful, harmless, and safe, wouldn't be possible. Anthropic was a pioneer of this kind of model behavior specification with their paper on Constitutional AI, which describes how they use post-training techniques to create AI that follows a set of principles outlined in Anthropic's constitution. Finally, in theory, instruction tuning makes the models more pleasant to talk to. Because human raters are rewarding the model for giving pleasing answers, the model becomes friendlier and develops a better personality.

No Free Lunch: Negative Consequences of Post-Training

Post-training isn't all good news. Reshaping output seriously affects a model's behavior, and makes its writing very different from that of the original base model.

Sycophancy

Optimizing an LLM according to human preference can cause the LLM to prioritize people pleasing over correctness. If this goes too far, the LLM will reinforce a user's false beliefs instead of contradicting them. There was a notable blow-up over this kind of sycophantic model behavior in April 2025, when OpenAI admitted that its new release of GPT-4o was overly flattering and agreeable, even to user detriment.

Loss of Creativity and Diversity in AI Outputs

Post-training also leads to less creative and unique LLM responses. Some researchers hypothesize that post-training causes LLM text to sound flat, derivative, samey, or like AI slop.

A 2024 paper, Creativity has Left the Chat, compares the diversity of the instruct and base versions of Meta's AI model, Llama 2. They found that the instruct model of Llama 2 exhibited substantially less semantic and stylistic diversity in its outputs; for instance, when choosing characters in stories, Llama 2 instruct consistently chose characters of the same ages, nationalities, personality types, and even gave them the same names. Below is a word cloud of the names chosen by the base and instruct models.

Word clouds of names generated by base vs. aligned models, from "Creativity Has Left the Chat"

Flattened Logits and Mode Collapse

Relatedly, post-training causes so-called flattened logits, where post-trained LLMs will deterministically choose one single next word instead of producing a varied distribution of possible next words.

While on the surface that doesn't sound bad, it leads to what is called mode collapse, which occurs when LLMs stop producing a rich variety of expressions, and instead just make the same limited choices repeatedly.

Mode collapse vs. mode coverage: a collapsed model concentrates probability on a single mode

Here's a visual of mode collapse: given three overall common ways to say something, a mode collapsed LLM will default to the most common way of saying it instead of spreading out its probability over all the ways of saying something.

What does this have to do with detectability?

The fact that LLMs are trained this way has a few implications for detection. Because LLMs are optimized for specific human preferences, they tend to take on very distinct personalities. These personalities have quirks, tics, and idiosyncrasies that a detection model can distinguish from human writing. Flattened logits and mode collapse also mean that aligned LLMs tend to make very consistent choices when generating text, which means that we can learn what those choices are by looking at large amounts of human and AI data.

Additionally, LLMs developed in the present day end up ingesting their own outputs over the course of their training, simply due to the increasing slopification of text on the internet. For instance, as of late 2024, over a third of new Internet articles are AI-generated; that much AI-generated text leads to an amplification effect where new models also take on the personalities of older models that generated that text, even across model families. That is why many LLMs exhibit similarities in style to the original ChatGPT, even though they were made by completely different labs! Sometimes, this is even done on purpose: it is widely speculated that Chinese labs try to "distill" from American frontier models like Claude by training on their outputs. That gives these language models a Claude-like personality!

Will LLMs always be this detectable?

Figuring out and solving mode collapse is an ongoing, active area of artificial intelligence research. Will LLMs be able to one day produce more diverse output distributions? My argument is yes, and there are even some language models that already are quite diverse: in fact, we already have these in the form of base models!

However, post-training as a paradigm isn't going anywhere. The reasons that LLMs make consistent choices and have specific preferences reflect the preferences of their developers. Ask a Chinese language model what happened at Tiananmen Square in 1989 and you can immediately notice the preferences of that developer! The fact is, we cannot have safe and aligned language models that behave helpfully and harmlessly without post-training. At the end of the day, LLM companies are building consumer products that have behavioral requirements: there are some areas of human text space (such as toxicity, misinformation, etc.) that the LLM companies do not want their models to sample from. Finally, LLM companies also want to optimize their models to have pleasant personalities — the more fun people have engaging with their models, the more likely they are to use a particular model and not use competitor models. You could see how this manifested with how some GPT-4o fans reacted to OpenAI discontinuing that model.

These developer and human preferences ultimately manifest in the models as narrowing the personality and output space of these LLMs in exchange for alignment, behavior shaping, and personality optimization. So as long as these model developers continue to have specific preferences and choices on model behavior, I believe that Pangram should be able to detect such preferences on a consistent basis.