The Contract Review Benchmark

How AI Models Perform on the Contract Provisions That Matter Most

Eleven leading AI models. 3,282 head-to-head reviews. 21 precision-critical guidelines. One definitive answer about which product can be trusted with high-stakes contract work.

This benchmark is the first installment in LegalOn's AI for In-House Arena, a continuing research series establishing the definitive performance record for AI across the tasks in-house legal teams rely on most. Contract review comes first because it is where precision failures carry the most direct legal and financial consequences.

‍

EXECUTIVE SUMMARY

The provisions that matter most are the ones general-purpose AI gets wrong

Every in-house legal team reviewing contracts with AI faces the same hidden risk: the tool gives a confident answer on the provisions that carry real legal and financial exposure, and the answer is wrong.

LegalOn's 2026 Contract Review Benchmark tested 11 AI systems across 3,282 pairwise contract reviews on 21 precision-critical guidelines. The findings are unambiguous. General-purpose AI models from companies including Anthropic, Google, and OpenAI fail at rates that should concern any legal team relying on them for contract review.

Without legal-specific training and the discipline of reviewing each provision individually against a precise standard, even the most capable foundation models produce confident-sounding answers that are frequently wrong.

LegalOn applies that training and architecture on top of the same foundation models, breaking contracts into structured, provision-level checks and evaluating each against a precise legal standard. The result is a level of accuracy that general-purpose tools, used directly, cannot match.

THE RANKINGS

LegalOn is first across all 21 provision types 

Every model was tested head-to-head against LegalOn across 3,282 contract reviews. An independent LLM judge, blind to which review came from which system, determined which review was more accurate, complete, and useful. Each comparison was run twice with the order reversed to rule out presentation bias. A win only counts if the same system was preferred both times.

Each company is represented by its strongest-performing model. This gives every provider its best shot and ensures comparisons reflect genuine capability, not model selection. 

Full results available in the Technical Appendix.

How to read the numbers

ELO — overall performance score accounting for opponent strength, borrowed from competitive chess rankings. Think of ELO like a leaderboard. Every head-to-head comparison updates the standings: win against a strong opponent and you gain more points than beating a weak one. A system that reaches the top of the table got there by consistently outperforming the field. LegalOn sits 87 points above the next closest model, and more than 400 above the best GPT model tested. In ELO terms, that's not a close race.

Confidence Interval (95% CI) — the range within which the true ELO score almost certainly falls. A narrow range (e.g., [1763, 1793] for LegalOn) means the result is stable and reliable; the ranking wouldn't shift meaningfully if the benchmark were run again on a different sample of contracts. A wide range signals more variance in performance. Every model's confidence interval is non-overlapping with LegalOn's, meaning the gap is statistically reliable.

Preference Rate — share of head-to-head comparisons in which the independent judge preferred LegalOn's review. Tie comparisons are excluded; this figure reflects only cases where one review was judged meaningfully superior.

Preference Ratio — the relative likelihood that the judge preferred LegalOn's review over a given model's in direct comparison. For example, a ratio of 1.8x means for every 10 comparisons Claude Opus 4.6 won, LegalOn won 18.

How we measured this

For each contract and provision, two reviews ran side by side: one from LegalOn and one from a general-purpose AI model. An independent LLM judge — blind to authorship — assessed which review was more accurate, complete, and useful. Every comparison was run twice with review order reversed to rule out position bias. Legal experts validated that the LLM judge's scoring methodology aligns with professional legal standards.

The evaluation criteria were designed to match how lawyers assess contract review quality in practice. The judging methodology was built to rule out the most common sources of benchmark inflation.

Full methodology detail in the Technical Appendix.

‍

WHY THE GAP EXISTS

Intelligence vs. Architecture

GPT-5.1, Claude Opus 4.6, and Gemini 3.1 Pro are among the most capable general-purpose language models available. Their contract review failures are a product of how they are designed to work.

General-purpose models review an entire contract against all guidelines in a single pass. This works for identifying general topics and extracting obvious information. 

It fails when the task requires confirming precise language, verifying a numeric threshold, or satisfying multiple conditions simultaneously. At that level of precision, broad scanning produces confident-sounding answers that are frequently wrong. 

LegalOn reviews each guideline individually, against each contract, running approximately 21 focused checks per contract in parallel.

Five Failure Modes that Explain the Pattern

1. ‍Specific clause identification — verifying exact wording of holdover caps, assignment rights, security deposit limits, and execution formalities. Models that scan for the general topic rather than the precise language frequently reach the wrong conclusion.‍
2. Quantitative threshold checks — comparing numeric values against policy limits: rent increases ≤ 150%, security deposits ≤ 3 months, survival periods ≤ 5 years. General-purpose models confirm the existence of a provision without verifying the embedded threshold.‍
3. Cross-reference validation — confirming that SOWs reference MSA terms, that a confidentiality purpose is specifically defined, that PHI ownership is acknowledged. These checks require reading for what the document says and what it omits.‍
4. Multi-part requirements — guidelines with multiple independent conditions joined by "AND." General-purpose models mark a provision as met when either condition is satisfied — missing the conjunction entirely.‍
5. Absence checks — provisions where the guideline requires a specific statement to be present. General-purpose models are trained to find what is there. They are not reliable at confirming when something required is definitively absent.

The provisions where general-purpose AI fails most often are the exact clauses in-house teams review every day. 

The failure pattern is consistent: these tools find the topic but miss the detail that determines whether the contract meets your standard. The accuracy gap is largest on exactly the provisions that matter most.

‍

WHERE IT BREAKS DOWN

Five clauses where a wrong answer has real consequences

The examples below appear in contracts in-house teams review frequently. Each one represents a provision type where an incorrect AI determination has real downstream consequences and where general-purpose models fail at rates that make them unreliable for production use.

1. bTriple Net Lease: Unconditional assignment right
   
   LegalOn reviews were rated higher quality 3.8x more often than the best general-purpose AI (GPT-5) on this provision — preferred in 79% of direct comparisons.
   
   Guideline: The contract must grant the tenant an unconditional right to assign or sublet the premises with no consent requirement.
   
   Why General AI fails: General-purpose models flag any assignment clause as compliant. The "no consent required" qualifier, which is the entire point of the guideline, is treated as implicit if any assignment right exists at all
   
   The Consequence: An assignment clause that looks compliant on the surface may give your tenant the right to transfer without consent. No AI flag was raised. The risk is invisible until a dispute surfaces.
   ‍
2. Business Associate Agreement: PHI Ownership
   
   LegalOn reviews were rated higher quality 2.6x more often than the best general-purpose AI (Gemini 3.1 Pro) on this provision — preferred in 72% of direct comparisons.
   
   Guideline: The BAA must acknowledge that all PHI provided by the Covered Entity remains the property of the Covered Entity.
   
   Why General AI fails: PHI handling obligations are conflated with ownership acknowledgment. A BAA that extensively addresses PHI security can still fail this guideline if ownership is never affirmatively stated — a distinction general-purpose models routinely miss.
   
   The Consequence: A BAA that passes AI review may leave PHI ownership legally ambiguous. In a breach or dispute, the absence of an explicit ownership acknowledgment becomes a significant vulnerability.
   ‍
3. Non-Disclosure Agreement: Defined Purpose
   
   LegalOn reviews were rated higher quality 2.4x more often than the best general-purpose AI (Claude Sonnet 4.6) on this provision — preferred in 74% of direct comparisons.
   
   Guideline: The agreement must specifically define the purpose for which confidential information is being shared.
   
   Why General AI fails: The existence of an NDA implies a purpose, which general-purpose models treat as sufficient. The guideline requires an explicit purpose statement, an absence check that requires reading for what is not present, not just what is.
   
   The Consequence: An NDA without an explicit purpose statement is broadly and dangerously interpreted. Confidential information shared under it may be used beyond any intended scope, with limited legal recourse.
   ‍
4. Master Services Agreement: SOW execution and MSA incorporation
   LegalOn reviews were rated higher quality 1.9x more often than the best general-purpose AI (Claude Opus 4.6) on this provision — preferred in 61% of direct comparisons.
   
   Guideline: The agreement must state that each SOW must be fully executed and incorporate by reference the terms of the MSA.
   
   Why General AI fails: Two independent requirements joined by "AND." General-purpose models return "met" when either condition is satisfied, missing the conjunction entirely. A fully-executed SOW that does not incorporate MSA terms (or vice versa) is marked compliant.
   
   The Consequence: SOWs that appear valid under AI review may lack the legal incorporation needed to enforce MSA protections. Disputes over scope, IP ownership, or liability may expose gaps that should have been caught at review.
   ‍
5. Clinical Trial Agreement: Manuscript review timeline (≥30 days)
   
   LegalOn reviews were rated higher quality 2.6x more often than the best general-purpose AI (Gemini 3.1 Pro) on this provision — preferred in 71% of direct comparisons.
   ‍
   Guideline: The agreement must give the Sponsor a minimum of 30 days (preferably 45 days) to review any manuscript before publication.
   ‍
   Why General AI fails: Manuscript review rights are present, but the specific duration threshold is not separately verified. A contract granting review rights without specifying a compliant duration is marked compliant.
   ‍
   The Consequence: A sponsor given fewer than 30 days to review a manuscript has limited ability to protect proprietary data or unpublished results. If the timeline isn't verified at review, the protection is effectively absent.

The bottom line
Across five representative provision types, LegalOn's reviews were consistently rated more accurate, complete, and useful than the best general-purpose AI on the same provisions. These are not benchmark edge cases. They are the clauses where a wrong answer has direct consequences.

‍

SPEED

Parallel architecture: accuracy and speed, together

Speed and accuracy don't have to trade off. By running approximately 21 provision-level checks in parallel rather than scanning the full contract in a single pass, LegalOn completes a full review in 2.3 seconds — 17x faster than Claude Opus 4.6, the strongest general-purpose model tested, which averaged 40.4 seconds per contract.

LegalOn's architecture runs ~21 checks in parallel; the baseline models run a single pass. The comparison reflects real-world time-to-result, i.e., how long it actually takes to get a complete review back.

‍

TECHNICAL APPENDIX

Test Parameters

Methodology: How we ran the test

One-shot baseline

For each contract and provision, two reviews ran side by side: one from LegalOn and one from a general-purpose AI. The baseline models received the full contract and all guidelines at once, returning MET/UNMET determinations in a single pass — the standard approach most in-house teams use.

‍Independent judging

An independent LLM judge, separate from any model tested and blind to authorship, assessed which review was more accurate, complete, and useful across five criteria: correctness, evidence quality, article identification, completeness, and reasoning quality.

Two-pass bias control

To rule out position bias — the tendency for LLM judges to favor whichever response appears first — every comparison was judged twice with the review order reversed between runs. A result counts as a win only when the same system is preferred regardless of order. When preference flipped, the result was excluded as a tie.

Attorney validation

Legal experts independently validated a sample of LLM judge outputs against professional standards, confirming strong alignment between the judge's verdicts and attorney-level assessment. This ensures the scoring reflects how practicing lawyers evaluate contract review quality, not just AI preferences.

‍

Full model-level results

AI FOR IN-HOUSE ARENA

Moving Legal Forward

The AI for In-House Arena is LegalOn's ongoing commitment to transparent, rigorous evaluation of AI performance on the work that matters most to in-house legal teams. Contract review comes first because it is where precision failures carry the most direct legal and financial consequences. Upcoming installments will evaluate contract redlines, legal research, contract data extraction, drafting, and legal AI assistance.

‍

CONTRIBUTORS
‍Gabor Melli, VP of AI, LegalOn
Deddy Jobson, Data Scientist, LegalOn
Corey Longhurst, Chief Growth Officer, LegalOn
Bärí A. Williams, Head of Legal and Legal Content, LegalOn
Katie Harris, Sr. Legal Engineer of Legal Content, LegalOn
LaTasha Fields, Lead Annotator, LegalOn
Venus Chui, Lead Annotator, LegalOn
Eileen Policarpio, Communications, LegalOn
Hailey Marshall, Brand Design Director, LegalOn
‍