%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Recently, a study from the University of Chicago revealed significant differences among various commercial AI text detection tools in the market. Researchers built a dataset containing 1992 human-written texts, covering six types: Amazon product reviews, blog posts, news reports, novel excerpts, restaurant reviews, and resumes. At the same time, they used four leading language models: GPT-41, Claude Opus4, Claude Sonnet4, and Gemini2.0Flash, to generate corresponding AI writing samples.
To compare the performance of these detection tools, the research team mainly tracked two metrics. The false positive rate (FPR) measures how often human texts are incorrectly marked as AI-generated, while the false negative rate (FNR) shows the proportion of AI texts that go undetected. In this direct comparison, the commercial detection tool Pangram performed excellently. For medium and long texts, Pangram's FPR and FNR were almost zero; for short texts, the error rate was also generally below 0.01, with the only exception being Gemini2.0Flash, which had an FNR of 0.02 in restaurant reviews.
Other detection tools like OriginalityAI and GPTZero performed slightly worse. Although they performed well on longer texts, keeping their FPR below 0.01, they were less satisfactory for extremely short texts. Additionally, they were more sensitive to "humanization" tools that disguise AI-generated texts as human writing.
Pangram showed excellent performance in identifying AI-generated texts, with FNRs for all four models not exceeding 0.02. In contrast, OriginalityAI's performance was more influenced by the generating model, while GPTZero showed more stable performance across models but still fell short of Pangram.
The researchers also tested the ability of each detection tool to counter the StealthGPT tool, which makes AI-generated texts harder to detect. Pangram performed relatively robustly in these situations, while other detection tools faced greater challenges.
In terms of economic efficiency, Pangram's average identification cost is $0.0228 per correctly identified AI text, approximately half of OriginalityAI and one-third of GPTZero. The study introduced the concept of a "policy ceiling," allowing users to set the maximum acceptable false positive rate to better regulate the detection tools.
The research team warned that these results are just a snapshot of the current situation, and a "arms race" will unfold between detection tools, new AI models, and evasion tools in the future. They recommended regular transparent audits to keep up with this rapidly changing field.
Project: https://pangram.ai/
Key Points:
🌟 Pangram excels in detection accuracy, with nearly zero false positives and false negatives.
📊 Other tools struggle with short texts, giving Pangram a clear advantage in identifying AI-generated texts.
💰 Pangram has the lowest identification cost, offering significant economic benefits and a practical choice for users.