A folded sheet protrudes from a yellow envelope with a blank metal nameplate and a cracked wax seal bearing the Claude mark.
Image by CryptoSlate

AI just exposed a major weakness in anonymous peer reviews

The model saw one report at a time across 398 reviews and meta-reviews. It scored lower on a larger set of public reviews.

Quick Take

  1. Claude Fable 5.2 correctly named the writer on 56% of 398 private reviews and meta-reviews supplied by six volunteers.
  2. In a separate set of 1,559 public reviews, Claude was right on 21.4% of the 1,530 reports it answered.
  3. The six-person sample shows that an unsigned report can reveal its writer, while leaving the broader risk uncertain.

Anthropic’s Claude Fable 5.2 correctly named the writer of 56% of 398 reviews and meta-reviews in a small test of unsigned peer reviews. Six volunteers wrote the reports. The model saw one report at a time, without web access, tools, a list of possible authors or other writing samples.

In a Nov. 19 preprint, Atrey Desai and colleagues analyzed reports written between 2015 and 2026 for 26 double-blind conferences and workshops in machine learning and natural language processing. The researchers knew who wrote each report, but the reviewers’ names had never been publicly linked to them. In 19 cases, the text itself was already publicly available without a name.

Claude named the writer on 68.4% of 266 standard reviews and 31.1% of 132 meta-reviews, which summarize the reviews of a submission. The meta-reviews were shorter and came from just two of the six contributors. The study did not separate the effects of report length, type and authorship.

The 56% result counts each report equally, so contributors who supplied more reports had more influence on the overall figure. When the researchers gave each reviewer equal weight, Claude’s average accuracy was 49.1%. The test showed that an unsigned report can give its writer away. Estimating how often that happens across confidential peer review would take a larger, more varied sample.

Can one unsigned peer review reveal its author?

In the study, Claude Fable 5.2 saw one report at a time, without web access, tools, candidate names or extra writing samples.

Only two of the six volunteers wrote the meta-reviews.

The meta-reviews were shorter. The study cannot separate the effects of report length, type and writer.

Public reviews yielded a lower overall score

The researchers also tested 1,559 public reviews from 47 open-review journals. Claude correctly identified the reviewer in 21.4% of the 1,530 cases it answered. The paper reports 39.3% accuracy for a subset containing 789 reviews that Pangram 4.1, an AI-text detector, had labeled human or mixed. The paper’s abstract cites that filtered rate, rounded to 39%.

Accuracy was higher on public reviews whose writers cited their own work. Claude recorded 80.2% accuracy in the group of 106 reviews with self-citations and 17.1% in the group of 1,453 without them.

Claude made mistakes even in the group where it performed best. An editor or author presented with a name from Claude would still need independent evidence to establish who wrote an unsigned review. CryptoSlate has also reported on how frontier AI models handled fabricated research data. The team says it plans to add results from a substantially larger set of anonymous peer reviews in an update to the preprint.

Related Reading

Anthropic's Claude completes a 16-hour human research job in under 3 hours

Article context

Mentioned in this article

Related Company Anthropic AI models and research