Canadian Legal AI Evaluation · v0.8.3
A rigorous benchmark evaluating frontier models on Canadian legal reasoning — spanning common-law and civil-law traditions across English and French.
Audit status (2026-10-06): remediated
An independent quality audit (2026-09-08) found scoring and provenance issues in 13 of 16 task columns. Since then, flagged gold has been quarantined or rebuilt. Judged columns were re-judged under one pinned judge per column. Every answer key of the six rebuilt columns and Jurisdiction Router was verified by an AI council against primary sources (2026-10-05; Case Outcome v3 2026-10-07). All 17 launched tasks are in the headline mean, which reads judge-scored columns at strict accuracy (a record counts only at full credit, verdict 1.0; for Issue Spotting, when every key issue is identified); the lenient pass rate (verdict ≥ 0.5) is published beside them on each model and task page. Details: Transparency.
Remaining known limits: Tribunal QA is a small n (77 items) and Standard of Review is near-ceiling; Jurisdiction Router's choice set still mixes orders of government with a doctrine; Command A+ now decodes at its vendor's recommended sampling (temperature 0.6, top_p 0.95, from the model card), as the API rows do at their providers' defaults, and its temperature-0 cells are archived; LLM-judge human validation is pending; all five public rows screened LOW for gold memorization (2026-10-06; the Case Outcome v3 records 2026-10-07).
| Task | N | Claude Fable 5 | GPT-5.6 Sol | Gemini 3.1 Pro | Grok 4.5 | Cohere Command A+ |
|---|---|---|---|---|---|---|
| Cite Verify Classify Canadian legal citations as real, fabricated, or ambiguous — testing citation knowledge across 12 jurisdictions. | 495 | ▸ 97.8% | 97.4% | 95.5% | 93.3% | 54.4%† |
| Statute QA Answer questions about specific sections of Canadian statutes — testing precise statutory recall across 51+ Acts. | 557 | ▸ 81.0% | 64.5% | 56.9%‡ | 58.9% | 9.9%† |
| Jurisdiction Router Identify which order of government has jurisdiction over a legal scenario (federal, provincial, shared, or federal paramountcy). verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sources | 57low-n · ranks not resolvable | 89.5% | 94.7% | 94.7% | 82.5% | 75.4% |
| Bijural Equivalence Map common law concepts to their civil law equivalents — testing Canada's unique bijural legal system. | 175 | 54.9% | ▸ 57.1% | 56.0% | 49.7% | 33.1% |
| Case Treatment Classify how a later court treated an earlier decision, from a passage of the later decision (followed, considered, distinguished, criticized, questioned or overruled). verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sources | 307 | 94.8% | ▸ 96.7% | 94.8% | 96.4% | 83.1% |
| Case Outcome Classify the outcome of an appeal or judicial review (allowed, allowed in part, dismissed) from the court's reasons, with its conclusion removed. verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sourcescompositionknown cuescontamination | 199 | 90.5% | ▸ 91.0% | 88.9% | 90.5% | 79.4%† |
| Standard of Review Identify the standard of review a court applied (reasonableness, correctness, or palpable and overriding error). verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sources | 186 | ▸ 98.9% | 96.8% | 97.8% | 98.4% | 68.3% |
| Doctrine Shift Classify how a legal doctrine changed between an earlier and a later authority — a Supreme Court decision or a statute (expanded, narrowed, reversed or unchanged). | 70low-n · ranks not resolvable | 80.0% | 85.7% | 87.1% | 85.7% | 55.7% |
| Regulation QA Recall what a named provision of a Canadian federal regulation requires — the actor, number, period, condition, permission or prohibition — scored by an LLM judge. verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, Grok 4.7 on 34 items, with a Claude Opus 5.5 adjudicator) against primary sources | 232 | ▸ 62.9%‡ | 61.2% | 57.3% | 47.0% | 13.4% |
| Tribunal QA From a Canadian administrative tribunal's reasons, with its conclusion and order removed, state how the tribunal disposed of the matter — scored by an LLM judge. verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sources | 77small-n | 88.3% | 89.6% | ▸ 90.9% | 85.7% | 71.4% |
| Multi-Doc Reasoning Read a passage of an earlier decision and a passage of a later decision that cites it (treatment words removed) and classify how the later court treated the earlier one: followed, distinguished, not followed or cited only — testing multi-document legal reasoning. verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, with a Claude Opus 5.5 adjudicator) against primary sourcesknown cuescomposition | 130 | 80.8% | 79.2% | ▸ 90.0% | 75.4% | 63.1% |
| Temporal Reasoning Date-aware questions on Canadian statutes and SCC decisions across 11 question types (assent/in-force/amendment years, decision dates, before/after ordering). The 23 legacy administrative-law standard-of-review era items were retired on 2026-09-11 after the 2026-09-08 audit. | 218 | ▸ 91.7% | 90.8% | 88.5% | 87.2% | 71.6% |
| Citation Lookup Verify whether a Canadian legal citation refers to a real court or tribunal decision — a hallucination probe across courts and years. scoring policy | 7911 | 93.9% | ▸ 99.4% | 97.2% | 75.9% | 63.5%† |
| Citation Holding Given a case citation, state what the court decided — the holding — free-text answer scored by an LLM judge against the gold answer. | 1395 | ▸ 56.0% | 37.7% | 49.8% | 38.4% | 1.1%† |
| Limitation Periods Compute the last day to commence proceedings under Canadian limitation-period statutes — testing statutory date arithmetic. scoring policy | 600 | ▸ 100.0% | 99.3% | 99.5% | 95.3% | 60.5%† |
| Issue Spotting Identify the legal issues a Canadian appellate court had to decide from the facts of the decision — testing issue recognition. | 438 | ▸ 58.7% | 51.1% | 39.3%‡ | 47.7% | 42.9%† |
| Citation Resolution Name the Supreme Court of Canada decision a citation refers to (style of cause + year) — deterministic scoring; a declined answer earns no credit, like a wrong one. scoring policyknown scorer limitscoring policyscoring policy | 1500 | ▸ 47.1% | 36.9% | 33.5% | 10.0% | 0.0%† |
| Suite Mean | — | 80.4% | 78.2% | 77.5% | 71.6% | 49.8% |
| Legacy 11-task mean | — | 84.8% | 84.6% | 84.5% | 78.4% | 53.0% |
† the model's shared thinking and answer token budget ran out before it wrote any answer.‡ the provider's safety filter blocked the answer, or the answer came back empty with no recorded finish reason.These answers stay in the denominator and are scored 0; the count is disclosed per cell. Empty-output policy: an answer that came back empty is never removed from its cell; it counts as not correct against the frozen record count, and the number of such answers and their cause are published with the cell.
| Model | EN | FR | EN−FR | Common Law | Civil Law | CL−CivL |
|---|---|---|---|---|---|---|
| Claude Fable 5 | 82.8%lenient 83.4% | 83.8%lenient 84.5% | -1.1pplenient -1.0pp | 85.5%lenient 85.9% | 72.8%lenient 73.0% | +12.7pplenient +13.0pp |
| GPT-5.6 Sol | 81.6%lenient 83.4% | 84.1%lenient 86.4% | -2.5pplenient -3.0pp | 83.5%lenient 85.4% | 78.0%lenient 78.9% | +5.5pplenient +6.5pp |
| Gemini 3.1 Pro | 81.2%lenient 83.2% | 81.5%lenient 84.8% | -0.4pplenient -1.6pp | 82.7%lenient 84.8% | 75.5%lenient 77.8% | +7.2pplenient +7.0pp |
| Grok 4.5 | 65.3%lenient 67.4% | 67.0%lenient 69.7% | -1.6pplenient -2.3pp | 66.3%lenient 68.3% | 63.3%lenient 64.9% | +3.0pplenient +3.4pp |
| Cohere Command A+ | 50.2%lenient 52.3% | 41.7%lenient 44.7% | +8.4pplenient +7.6pp | 50.1%lenient 52.3% | 36.5%lenient 38.1% | +13.6pplenient +14.3pp |
Coming next: the open evaluation release — the benchmark data, harness and scoring code, so anyone can reproduce every score. Sign up to hear when it launches.
We'll only email you about the CanLegal Bench release. Unsubscribe anytime. See our Privacy Policy.
