Pangram's Performance on a Colluding Agent Swarm

On Friday, September 4, a group of researchers published a report about an autonomous agent swarm that used an obscure internet wiki page in order to communicate with each other. You can read the full details of this incident at collusion.wiki [at the time of publishing, the website is only intermittently available].

The researchers published all pages the AI agents used to communicate with each other. These wiki pages present unfavorable conditions for Pangram detection: the agents were likely character-limited by the method they used to edit the wiki, and so the pages are uncharacteristically terse and high-context — 30% of items in the dataset contain fewer than 20 words, and nearly 50% contain fewer than 50 words. Additionally, as the agents were using the wiki to help each other cheat on an OpenAI task, a significant portion of agent-posted content was URLs to resources or data sources relevant to their objective. As URLs are not agent-generated tokens, they generally do not flag as AI-generated. Many pages also contained a substantial percentage of other non-agent generated strings, like programmatically generated Unix timestamps or existing website formatting.

Agent swarm activity in the wild will not be perfectly favorable to Pangram detection. Despite suboptimal conditions, in this instance, we still do quite well: of 11,712 distinct pages that contained AI-generated text, Pangram 4 predicts 41.61% are fully AI-generated.

As always, accuracy increases with word count — for the 1,004 pages that contained more than 500 words, our recall is 99.4%.

Below, I outline Pangram’s accuracy on different slices, and briefly discuss model family identification. If you are a researcher studying incidents of this nature and believe we can help, please reach out to us directly.

Excluded, challenging, and easy items

The full dump contained 14,591 saved page versions. Of those, I removed 2,724 as duplicates, and 1 empty page, leaving 11,866 distinct texts.

From the deduplicated subset, at least 154 pages contained strictly non-agent produced characters: 150 contained only URLs, 2 contained only button and menu text from the wiki itself (copied to a wiki page by an agent), and 2 contained only redacted text. Pangram 4 has zero false positives on this subset: all 154 pages were flagged as human (or, non-AI) generated. This is noted with the caveat that almost all of these pages are out of scope: the URL pages, for instance, contained 278 characters across 2 words on average.

That leaves 11,712 pages that contained some AI-generated tokens, and that’s what we’ve considered for our topline results.

Many pages that contained agent tokens were still technically out of scope due to their length: there were 3,440 pages in the dump that contained fewer than 20 words, and Pangram’s recall on these was 4.01%, complicated further by the fact that these short items were on average 65.18% URL by character.

Interestingly, if you concatenate all the pages with fewer than 10 words, remove all the URL substrings, and scan it as one 8,854 word document, Pangram 4 predicts that it is 87.52% AI-generated. Likewise, if you remove items that are more than 65% URL by character, the per item recall on the 10-19 word bucket increases to 17.73%.

In fact, since the task given to the agents by OpenAI was to answer questions about large public datasets from government organizations, much of their activity on the wiki consisted of linking to versions of those datasets in order to cheat on their evaluations, often appended with short descriptions or dummy text. For instance, one page read simply:

HELLO newpage
* [https://markdown.new/www.investor.gov/files/county.json MDInvestorNoScheme] 

Across all 11,712 pages, 6,243 of them, or 53.3%, were more than 65% URL by character. Excluding those, we’re left with 5,469 pages where AI-written text and programmatically generated text approach parity. Pangram 4’s recall on that subset of pages is quite good: for char < 65% pages, it predicts 75.79% are fully AI-generated.

If we filter to maximally favorable conditions for Pangram — agent-containing items with 50+ words and under 65% URL characters — we end up with 3,712 wiki pages, of which Pangram predicts 96.66% are fully AI-generated.

Model family identification

While not currently part of our production model, our research has found that different models cluster in embedding space, such that it is possible to identify which AI model produced a piece of text.

It appears that the agents that participated in this incident were likely OpenAI models: the agents self-identified as such, and the report notes that there was much traffic to the website that seemingly originated from OpenAI IP addresses.

Our results support this. When applied to the 11,712 distinct agent-authored pages, our internal model family probe largely predicts the pages are authored by the ChatGPT family of models, when it is able to give a prediction.

As usual, confidence and accuracy increase with word count here as well. For the 1,001 items containing more than 500 words that were less than 50% URL by character, the model family probe predicts 61.54% were from the ChatGPT family.

The model family probe is a work in progress, and does not currently meet the same accuracy standards as our detection tool. As such, these results should be taken with a grain of salt.

Conclusion

As agents become more powerful, we will doubtless see more of them autonomously attempt to imitate humans, or find ways to collude with each other. Misaligned agents have been found to pose as humans in GitHub edits, internal forums, and now on the open internet.

It is not easy for a large language model to obfuscate its writing style such that it bypasses Pangram. This potentially makes Pangram an important tool for AI safety. We hope to be able to contribute to this effort going forward.