Pangram’s performance on undergraduate and graduate student essays

We’ve previously evaluated Pangram on the PERSUADE corpus, a set of nearly 26,000 essays from a diverse group of middle and high school students. In this evaluation, we extend that analysis, benchmarking Pangram 4.1 on a set of undergraduate and graduate student writing, and in particular graduate student writing from students whose first language is not English.

For this benchmark, we use Pangram 4.1 to scan 9 corpora of 15,420 documents written by undergraduate and graduate students who speak English as an additional language. Documents were written and collected at universities across 16 countries on 3 continents, and include writing from 51 first-language groups, including Mandarin, Arabic, Japanese, Spanish, and Russian.

Performance

In total, this evaluation consisted of 9.5M words, with the longest document containing 10,916 words and the shortest 52 words.

Pangram 4.1 achieved zero false positives on all 9 datasets.

All datasets contained data regarding the first language of the author, and many also contained information about the assignment type, class, year, or degree program — those details are summarized per corpus below.

Dataset details

For this analysis, we evaluate Pangram 4.1 on a selection of undergraduate essays from several external, human-written pre-2022 datasets. First language is sometimes denoted L1.

• ArCC: A set of 779 argumentative and reflective essays written in English by L1 Arabic BA and MA students. Texts ranged from 53 to 889 words.
• The Nagoya Interlanguage Corpus of English Reformed (NICER), from researchers at Nagoya University, is a collection of 381 proctored, one-hour essays written in English from Japanese-speaking undergraduates.
• The Chinese and English Learner Language (CELL) Corpus is a collection of 2,850 essays written by undergraduates at the Open University of Hong Kong. First languages include Cantonese and Putonghua, among others. These essays were the product of take-home academic coursework and range from 91 to 5,929 words, with an average of 1,307 words.
• The Corpus of Turkish Undergraduate Writers of English (TUWE) is a collection of 341 persuasive essays ranging from 356 to 1,312 words, with an average length of 778 words. Essays were written in English by first-language Turkish students at Erciyes University.
• The Zayed Arabic-English Bilingual Undergraduate Corpus (ZAEBUC) is a collection of short essays written by students at Zayed University in the United Arab Emirates. Researchers collected this dataset from August to September 2019. Above, we report results on the English subset of 383 essays averaging 228 words, with Common European Framework of Reference (CEFR) levels ranging from A1 to C1. We additionally ran Pangram on the Arabic subset of 215 essays, for a total of 603 essays written by 397 students. Pangram 4.1 had zero false positives in either subset.
• The South African Multilingual Learner Corpus of Academic Texts (SAMuLCAT) is a collection of writing from South African undergraduate students, completed as take-home assignments with spellcheck and dictionary assistance. Genres include argumentative or persuasive essays, discussions, and narrative writing. Essays range from 52 to 5,233 words, with an average of 687 words. L1s for authors in this subset include Sesotho, isiZulu, Setswana, isiXhosa, and others. Above, we report our results on the English subset of 4,839 examples; we additionally scanned 579 L1 English essays for a total of 5,660 essays. We found zero false positives across either subset.
• The Michigan Corpus of Upper-level Student Papers (MICUSP) is a collection of 829 high-performing papers — receiving either an A or A- grade — written by senior undergraduate and graduate students at the University of Michigan, averaging 3,157 words. Of these papers, non-native English students wrote 148: 38 undergraduate papers and 110 graduate papers. L1s for this subset include Mandarin, Korean, Spanish, Russian, Hindi, and others, and the papers span 16 subjects, including psychology, sociology, engineering, biology, and humanities courses. We report the non-English L1 subset above; however, we additionally ran Pangram 4.1 on the 681 L1 English papers and found no false positives on either subset.
• The USask dataset is a collection of 141 graduate student literature reviews written by 51 L2 English students enrolled in multiple Canadian universities, with proficiency ranging from 6.5 to 8.5 on the International English Language Testing System (IELTS). Students were enrolled in programs including linguistics, education, and political science. Reviews ranged from 264 to 4,391 words with an average of 1,028 words.

One dataset was also reported in the Pangram 4 Technical Report, which we reproduce here for Pangram 4.1.

• The International Corpus Network of Asian Learners of English (ICNALE) released a written essay (WE) module of its dataset consisting of 5,593 timed essays written in English. ICNALE WE doesn’t record first-language data, but includes English writing from university students in 10 Asian countries, including China, Hong Kong, Indonesia, Japan, and Korea.

Because of the broad selection of essays, this benchmark includes a variety of assignment types, from take-home to timed essays; assistance levels, from unrestricted assistance to proctored exams; English proficiency levels; and degree programs.

Methodology

I searched for research datasets that met the following criteria:

• Collections of undergraduate or graduate student writing; excluding datasets written by prospective, non-degree, or high school students, as well as college application essays
• Includes metadata regarding L1 or English language learner status, or specified otherwise (for example, if the entire dataset was from one L1 group)
• Primarily comprised of essays, reports, long-form answers, literature reviews, theses, creative writing, etc; excluding datasets of exclusively short-answer responses
• Containing at least 100 documents with a median word count over 175 words

I deduplicated documents and removed any items with fewer than 50 words. Otherwise I did minimal pre-processing in the items. I elected to scan each text twice with Pangram 4.1: once with front matter and references stripped and once with them included, and found the result was invariant. I also scanned the same 15,420 texts with our previous model, Pangram 4, which also achieved zero false positives.

As always, we use a strict definition of false positive: we count a document as a false positive if any segment is flagged as AI or AI-assisted.

Pangram 4.1 was not trained or calibrated on these documents, and so this functions as a fair test of how Pangram performs on unseen writing by university students.

In the course of this evaluation, we additionally scanned 1,743 texts that were not reported in the headline results because they did not meet some criteria (typically excluded because they were written by an L1 English author) and found 0 false positives there.