Tasks

Name the Supreme Court of Canada decision a citation refers to (style of cause + year) — deterministic scoring; a declined answer earns no credit, like a wrong one.

Given a case citation, state what the court decided — the holding — free-text answer scored by an LLM judge against the gold answer.

Identify the legal issues a Canadian appellate court had to decide from the facts of the decision — testing issue recognition.

Recall what a named provision of a Canadian federal regulation requires — the actor, number, period, condition, permission or prohibition — scored by an LLM judge.

Map common law concepts to their civil law equivalents — testing Canada's unique bijural legal system.

Answer questions about specific sections of Canadian statutes — testing precise statutory recall across 51+ Acts.

Read a passage of an earlier decision and a passage of a later decision that cites it (treatment words removed) and classify how the later court treated the earlier one: followed, distinguished, not followed or cited only — testing multi-document legal reasoning.

Classify how a legal doctrine changed between an earlier and a later authority — a Supreme Court decision or a statute (expanded, narrowed, reversed or unchanged).

From a Canadian administrative tribunal's reasons, with its conclusion and order removed, state how the tribunal disposed of the matter — scored by an LLM judge.

Date-aware questions on Canadian statutes and SCC decisions across 11 question types (assent/in-force/amendment years, decision dates, before/after ordering). The 23 legacy administrative-law standard-of-review era items were retired on 2026-09-11 after the 2026-09-08 audit.

Verify whether a Canadian legal citation refers to a real court or tribunal decision — a hallucination probe across courts and years.

Identify which order of government has jurisdiction over a legal scenario (federal, provincial, shared, or federal paramountcy).

Classify Canadian legal citations as real, fabricated, or ambiguous — testing citation knowledge across 12 jurisdictions.

Classify the outcome of an appeal or judicial review (allowed, allowed in part, dismissed) from the court's reasons, with its conclusion removed.

Compute the last day to commence proceedings under Canadian limitation-period statutes — testing statutory date arithmetic.

Identify the standard of review a court applied (reasonableness, correctness, or palpable and overriding error).

Classify how a later court treated an earlier decision, from a passage of the later decision (followed, considered, distinguished, criticized, questioned or overruled).