Regulation QA

Recall what a named provision of a Canadian federal regulation requires — the actor, number, period, condition, permission or prohibition — scored by an LLM judge.

Judge-scored column: scores are strict accuracy — a record counts only when the judge gives it full credit (verdict 1.0) — since 2026-10-05. The lenient pass rate (verdict ≥ 0.5, half credit counted) is shown beside each model.

verified by an AI council (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5.5, Grok 4.7 on 34 items, with a Claude Opus 5.5 adjudicator) against primary sourcesAnswer keys verified on 2026-10-05 by an AI council against primary sources: GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5.5 on every item (Grok 4.7 also on 43 items — 34 Regulation QA, 5 Bijural Equivalence, 4 Temporal Reasoning — until its cost guard stopped it), each member given web access and the stored source text where the benchmark holds one; a key counts as confirmed only when at least two members back it with a machine-verified quote, and disputes were ruled from the quoted texts by a Claude Opus 5.5 adjudicator. Of 232 keys: 227 confirmed, 0 changed, 5 found wrong or incomplete and set aside; on 2026-10-06 those 5 were re-drafted through the column's build gates, re-verified by the council with verified quotes and re-admitted (232 keys).