Reasoning Adequacy Battery

The 2030 battery

After machines exhausted ARC-AGI-4, finding suitable questions became the principal research expense. Items survived only if a human succeeded and the strongest submitted AI failed. Human scores pooled different volunteers’ successful attempts. Machine-solvable items were retired from subsequent editions as measures of acquired skill.

• Task categories: unfamiliar pattern rules, evidence revision and short planning problems.
• Human requirements: one successful volunteer per item; memory and rereading allowed.
• Machine requirements: one frozen system across all items; working memory erased between stages; reconstructing context charged against its action budget.

The 2031 revision

When machines overcame the memory restrictions, the committee turned to bodily experience. Integrated sensors were rejected; living tissue was required. Molecular manufacturing supplied cultured sensory organs. Each successful submission established another form of imitation to exclude. The released revision restored the gap by withholding the object itself. By 2032 machines had reconstructed the manufacturers’ catalogues from the identifiers.

• Task categories: recognizing smells and identifying manufactured objects by touch.
• Human requirements: use ordinary eyes, hands and noses; direct access to the objects.
• Machine requirements: infer the same answers from inventory numbers and written descriptions; cameras, manipulators and chemical sensors counted as external assistance.

The 2032 revision

The autobiographical tests moved the missing information into a human volunteer’s childhood. Certifiers checked school records and relatives’ testimony to establish the reference answer. When a human forgot, the item failed validation and disappeared. When an AI answered correctly, the committee investigated leakage.

• Task categories: childhood smells, remembered rooms and the locations of long-hidden objects.
• Human requirements: answer about one’s own life; personal recollection accepted as an intrinsic capacity.
• Machine requirements: recover that person’s answer from a sparse biography and the question. Supplying the missing memories contaminated the test.

Saturation and holistic judgement

By 2032, machines inferred even childhood smells by simulating candidate universes with different answers. They compared which histories would produce the observed benchmark: its volunteers, selection rules and exact questions. Most candidate histories produced different tests. The surviving simulations agreed on the scored answers. The question-setting process had supplied evidence that the committee believed it had withheld.

The graph tracks the best single system on each archived edition, with tasks and rules held fixed. Every edition eventually reached 100%. After 2032, the committee discontinued the battery and adopted holistic judgement. Panels now decide whether a demonstration conveys authentic understanding. There is no fixed task set, pass mark or published scoring rule for a machine to exhaust.

Denialist foundations continued funding the assessments. The 2024 Google pizza-glue answer remained admissible evidence; the Epoch Capabilities Index recorded capabilities the panels discounted. Failure reenactment kitchens served edible reconstructions of obsolete machine recipes. At appliance examination days, research-directing toasters answered questions about childhood. Examiners judged whether the answers felt remembered.