AI-Writing Detection Is a Moving Target: What a 23-Model Study Found

An AI-writing detector can look highly accurate in a benchmark and become unreliable when the next model generation arrives. That is the central warning from a new preprint that tested scientific-abstract screening across 23 large-language-model versions released between June 2023 and August 2026.

The finding matters because universities, journals and conferences increasingly use detector scores when deciding which work deserves scrutiny. A score can be useful evidence about a document. It is not a stable measurement of authorship, and it cannot carry the weight of a misconduct decision by itself.

What the researchers tested

Kazuki Nakajima and Takayuki Mizuno submitted their preprint on 8 October 2026. It has not yet been peer reviewed, so its results should be read as new evidence open to challenge rather than settled consensus.

The researchers began with 4,000 abstracts from Proceedings of the National Academy of Sciences articles published before ChatGPT. They asked 23 model versions from OpenAI, Meta and Alibaba to rewrite those abstracts. After excluding 47 truncated or empty outputs, the dataset contained 91,953 rewrites.

They then trained RoBERTa-based detectors under several maintenance scenarios. Some detectors saw the exact version they would later judge; others had only earlier versions from the same vendor. A pooled detector learned from all 23 versions, while a “union” screen flagged a paper if any specialised detector fired. The researchers also tested one commercial detector at its production setting.

The headline result is about change, not universal failure

When a detector trained on the same version it screened, the median true-positive rate across the 23 versions was 99.9% at the study’s chosen operating point. Retraining on the current version and all earlier versions produced a median of 99.3%.

The problem appeared when a newly released version arrived before the detector had examples from it. Across most model transitions, detection remained strong. At two generation boundaries it fell sharply: to 26.6% for the first GPT-5 version in the study and to 3.8% for Muse-Glimmer. The latter followed performance above 99% immediately before the boundary.

That does not show that every detector will fail after every release. It shows that past benchmark accuracy cannot guarantee transfer to a new model whose output distribution has changed.

Two ways to fail pull in opposite directions

A screening system must manage two different errors. A false positive flags human-written work. A false negative misses AI-produced or AI-rewritten text. Optimising aggressively for one can make the other worse.

Screening design	What the study observed	Institutional risk
One detector per version; flag if any fires	Caught nearly every rewrite but falsely flagged about 11.9% of held-out human abstracts	Roughly one in eight human abstracts sent into suspicion
One pooled detector covering all versions	False flags stayed near 1%, but the miss rate reached 33.8% for one newer version	A calmer-looking system can still miss one in three target rewrites
Commercial detector at its production setting	Very few human false flags in this corpus, but uneven detection across model versions and disciplines	A low false-positive rate does not prove broad or durable coverage

The numbers above belong to this study’s corpus, thresholds and detector designs. They are not universal product specifications. Their value lies in showing the trade-off institutions must measure rather than hide.

Why model turnover breaks the measurement

Text detectors learn statistical patterns associated with the writing they have seen. Those patterns can include vocabulary, sentence structure, punctuation and other distributional signals. When a model family changes, the signals may shift even if the text still looks fluent to a human reader.

The study found that vocabulary-distribution changes tracked the detector’s loss of transfer across versions. This helps explain why a detector can recognise yesterday’s generator and miss tomorrow’s without either product being “broken” in the ordinary software sense. The target has moved.

The same logic also warns against permanent blacklists. A detector trained later may miss outputs from an earlier version. Coverage is not simply a ladder in which each update catches everything that came before.

A percentage is not a probability of guilt

Detector interfaces often present a percentage, label or highlighted passage. Readers may mistakenly interpret “80% AI” as an 80% probability that a person cheated. Those are different claims.

• A document-level percentage may estimate how much qualifying prose resembles the detector’s target class.
• It may exclude code, tables, poetry, citations or other formats the system does not reliably assess.
• Its meaning depends on the threshold, calibration data, supported language, document length and product version.
• It does not establish who operated a tool, what instructions were given or whether the use breached a policy.

Even Turnitin’s current guidance says its model can misidentify human-written, AI-generated and AI-paraphrased text and should not be the sole basis for adverse action against a student. Its report also suppresses numerical scores below 20% because false positives are more common in that range.

What this study does not establish

The authors are explicit about limitations. Their simulations used one trained-detector family, one commercial detector at a fixed operating point and a specialised task: rewriting scientific abstracts. The study is not a test of every detector, language, discipline, assignment type or form of AI assistance.

Rewriting an existing abstract is also narrower than drafting a dissertation chapter, editing grammar, translating notes or collaborating through several tools. A detector that performs one way on formal scientific prose may behave differently on reflective essays, second-language writing or creative work.

Finally, the paper evaluates detection performance—not the ethics of AI use. An AI-assisted passage can be permitted, disclosed and responsibly checked. Human-written text can still contain plagiarism, fabrication or serious error. Authorship detection and research integrity overlap, but they are not identical.

Replace automatic accusation with an evidence ladder

A fair process should increase scrutiny in stages. The detector is an initial signal, not the verdict.

Stage	Question	Appropriate action
1. Technical check	Was the file eligible, the language supported and the detector version current?	Record tool version, date, settings, score and processing warnings
2. Context review	Does the highlighted text conflict with the person’s prior work or the assignment process?	Compare relevant drafts and evidence without assuming guilt
3. Content review	Are citations real, claims supported and methods reproducible?	Investigate substantive integrity issues independently of style
4. Conversation	Can the author explain the argument, sources and revision decisions?	Use a neutral, documented discussion and allow clarification
5. Policy test	What use was permitted, prohibited or required to be disclosed?	Apply the policy that existed when the work was submitted
6. Decision and appeal	Is there corroborating evidence strong enough for the consequence?	Give reasons, preserve records and provide an appeal route

The higher the consequence, the stronger and more independent the evidence should be. A failed module, rejected manuscript or misconduct record should never rest on an unexplained model score.

Validate the detector for the actual institution

NIST’s AI Risk Management Framework says accuracy should be tested on realistic data representing the expected use and monitored over time. Its 2026 draft TEVV-Athlon framework similarly emphasises evaluation designed around an organisation’s real objectives, impacts and measurement context.

For a university or publisher, that means building a local validation set with appropriate governance. Include human and permitted AI-assisted writing from relevant disciplines, languages and document types. Protect personal data, obtain any required permissions, and do not quietly convert student or author submissions into a reusable benchmark without authority.

• Measure false positives and false negatives separately.
• Report results by language, discipline, document length and assistance type.
• Test every material detector update and every major generator release.
• Keep an expiry date on every validation report.
• Pause high-consequence use when the current target models are not covered.

Write policies around behaviour, not machine style

A durable integrity policy defines what people may do: generate a first draft, translate, edit language, summarise sources, produce code, or receive feedback. It specifies disclosure, citation, privacy and verification duties. It does not assume a detector can reconstruct the entire process from final prose.

Good policies also distinguish assistance from substitution. A researcher who uses an approved tool to improve clarity while retaining responsibility for every claim presents a different case from someone who submits fabricated results they cannot explain. The meaningful questions concern provenance, understanding and accountability.

Preserve process evidence before a dispute begins

Students and authors should not have to surveil themselves continuously to prove innocence. Institutions can nevertheless make ordinary research process easier to demonstrate through proportionate practices: dated outlines, version histories, source notes, analysis files, laboratory records and clear declarations of permitted assistance.

Editors can ask for supporting material only when relevant and under transparent retention rules. A version history may clarify authorship, but it can also expose private thinking, collaborator identities or sensitive research. Evidence collection should therefore be necessary, limited and secured.

The right response is better assessment, not a detector arms race

Stacking more detectors may appear safer. In the study, however, the union approach caught more rewrites by accepting far more human false flags. More screens can create more suspicion without creating more truth.

Assessment design can reduce reliance on retrospective guessing. Short oral explanations, staged feedback, annotated source choices, authentic datasets and task-specific reflection make understanding visible. These methods require human time, but they test the educational or scholarly outcome directly.

MaryChuks has a practical guide to reading a research paper without treating its abstract as the whole study, and a related editorial on why verification is becoming the bottleneck in AI-assisted science.

A five-question decision rule

Before using an AI-writing detector in a consequential workflow, ask:

1. Which detector version, generator versions and document types were last validated?
2. What are the measured false-positive and false-negative rates for our population?
3. What corroborating evidence is required before a human opens a formal case?
4. Can the affected person inspect the evidence, explain their process and appeal?
5. When will this validation expire or be repeated?

If an institution cannot answer all five, the detector is not ready to decide anyone’s academic or professional future. It may still support an exploratory review, provided its uncertainty is visible and its output cannot become an automatic punishment.

Sources and further reading

• Nakajima and Mizuno: Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing (preprint submitted 8 October 2026)
• Full experimental paper and supplementary material
• Turnitin: Using the AI Writing Report
• NIST AI Risk Management Framework: AI risks and trustworthiness
• NIST: TEVV-Athlon Framework for Evaluating AI Systems