Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks

Anthropic’s Fable 5.2 ranks first on the Remote Labor Index, a test of complete freelance projects, with 41.3% of its deliverables judged to meet a professional reference standard.

Scale’s Nov. 12 leaderboard lists Fable 5.2 at 41.3%, ahead of the next-highest model, Opus 5.5 at 21.25%. Anthropic reported the result in its Nov. 9 release announcement, labeling the RLI score “High effort” and indicating that safeguards were enabled.

The original RLI paper, submitted in October 2025 by researchers from the Center for AI Safety and Scale AI, describes a dataset of 240 projects across 23 freelance work categories. Each includes a brief, the files needed to complete the task, and a professional human-reference deliverable.

The selected projects include documents, audio, video, 3D models, and computer-aided design files. That range makes file production and finished-product quality part of the challenge, alongside understanding what the client requested.

The researchers’ original evaluation method uses trained human reviewers to inspect the AI’s work against the brief and reference deliverable.

Independent evaluations and majority voting determine outcomes, and a project passes when the AI’s deliverable satisfies the brief at least as well as the reference, at a quality a reasonable client would accept.

A useful prototype or a partial result can therefore fall below the pass threshold, even when it demonstrates relevant skills. Passing demands a deliverable that clears the professional reference standard.

The unit is an accepted benchmark project. The score alone does not establish job losses, deployment cost or human-free execution.

Benchmark project judgments are not employment outcomes or actual freelance payments. The accessible current result does not establish exact execution setup, per-project spending or human assistance. Original methods remain historical context.

Sources and evidence

Sources

1. Scale RLI leaderboard, checked Nov. 12, 2026
2. Anthropic release, Nov. 9, 2026
3. Original RLI methods, October 2025

Supporting facts

• Scale’s Remote Labor Index leaderboard, checked on November 12, lists Fable 5.2 first with a 41.30% automation rate and Opus 5.5 next at 21.25%. [1]
• Anthropic’s November 9, 2026 release announcement reports Fable 5.2 scoring 41.3% on RLI at High effort, with safeguards on under the page’s table notes. [2]
• RLI’s automation rate is the share of projects where human evaluators judge an AI deliverable to satisfy the brief at least as well as the professional reference, at a standard a reasonable client would accept. [3] [1]
• RLI’s original evaluation uses independent, trained human reviewers to inspect deliverables and determine outcomes by majority vote. [3] [1]
• The 41.3% RLI project pass rate alone does not measure the share of jobs eliminated, cash earned on freelance platforms or the cost of deploying the current system. [3] [1] [2]
• Anthropic reports Fable 5.2’s RLI result at High effort; the accessible result disclosure does not establish the exact execution harness, per-project spending or amount of human support. [2]

Method

Reported scores from Scale’s Nov. 12 leaderboard. Horizontal lengths use a shared 0% to 100% scale: 800 × 0.413 = 330.4 and 800 × 0.2125 = 170 desktop units; 360 × 0.413 = 148.68 and 360 × 0.2125 = 76.5 mobile units. The lower schematic summarizes original evaluation methods, not current generation settings.

Scope

Benchmark project judgments are not employment outcomes or actual freelance payments. The accessible current result does not establish exact execution setup, per-project spending or human assistance. Original methods remain historical context.

Infographic by CryptoSlate

What the Anthropic score leaves open

Fable 5.2’s score still leaves most evaluated projects below that acceptance threshold.

The original study excludes work that inherently requires client interaction, teamwork, or physical labor, as well as projects whose outcome must be awaited before it can be evaluated. Those exclusions limit how broadly the result can apply to the wider remote-work economy.

Anthropic’s release page specifies High effort, but does not establish the precise RLI execution setup, spending per project, or amount of human support.

RLI provides a demanding definition of success for a business assessing AI’s usefulness: carrying an assignment through to an inspectable product, with professional work setting the acceptance standard.