UCL-Huawei Team Says Frozen-LLM Agent Matches Human Efficiency on All Public ARC-AGI-3 Games

A paper posted to arXiv on October 8 describes an agent, Memento 3, that reports a mean Relative Human Action Efficiency (RHAE) score of 100.0 across all 25 public ARC-AGI-3 environments, completing every level while using 7,518 actions against a stated human baseline of 17,135, or about 44 percent. The authors, Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, and Jun Wang, work across University College London, Huawei's Noah's Ark Lab, and the University of Liverpool. The system's underlying language model is never fine-tuned or updated; all of the reported learning happens in an external memory structure the agent edits between episodes.

ARC-AGI-3 is the interactive successor to the original ARC-AGI puzzle benchmark, built by the ARC Prize Foundation to test whether an agent can work out an unfamiliar, video-game-like environment's rules through exploration rather than pattern-matching on training data. Earlier coverage of the benchmark has emphasized how resistant it has been to large models acting alone; the paper cites a baseline of 40.7 mean RHAE for Claude Opus 5 used without the surrounding architecture, against which Memento 3's reported ceiling score is a substantial jump. Separately, other systems have already claimed full completion of the public demo set this year without matching that baseline on efficiency, so the distinguishing claim here is not that the levels were finished but that they were finished with human-level action economy, as the paper defines it.

A Rulebook Instead of Retraining

The architecture keeps the base model frozen and instead has it maintain a natural-language rulebook of hypotheses about how a given environment behaves. That rulebook is compiled into executable code that the agent uses for prediction and planning. After each episode, the system runs a loop of observing outcomes, reflecting on prediction errors, revising the rulebook, recompiling it, and verifying the result: a code update is only accepted if the model judges it consistent with the rulebook's stated rules and a replay of past transitions reproduces what was actually observed. A population variant extends this by running several candidate world models in parallel and sharing evidence between them; on one game the authors identify as wa30, a two-model run cut the action count from 899 to 597, a 33 percent reduction, while holding RHAE at 100. An ablation reported in the paper credits the rulebook mechanism specifically with a 9 percent reduction in actions and an 18 percent reduction in agent turns, though that ablation covers only three of the 25 games and single runs rather than repeated trials.

The approach is a direct continuation of earlier work from the same UCL-Huawei collaboration. In a 2025 paper describing the original Memento framework, which stored prior task attempts in a growing case bank rather than compiling them into code, Jun Wang framed the research program's premise this way: "We call for a new approach that allows for agent adaptability without altering LLM parameters," adding that the project "aims to initiate this revolution." Memento 3 keeps that constraint — no gradient updates to the base model — while swapping case-based retrieval for an explicit, revisable model of how each environment works.

A Pong Result on the Side

The paper also reports a smaller case study on Atari Pong. A learned feedback controller, trained on 9,504 emulator frames, wins three evaluated episodes 21-0 against the game's built-in opponent with no further calls to the language model during play. The authors compare this to roughly 400,000 frames for model-based reinforcement learning baselines such as EfficientZero V2, framing the result as about 42 times more sample-efficient. The comparison is narrow — three episodes against a fixed opponent is a small evaluation, and Pong is a much simpler environment than the ARC-AGI-3 suite — but it is offered as a secondary demonstration that the same rulebook-and-code mechanism transfers outside the primary benchmark.

Why the Headline Number Needs Qualification

ARC-AGI-3 results outside the foundation's own verified harness are, by the project's own description, self-reported, and the Memento 3 numbers have not yet been independently reproduced on ARC Prize's semi-private holdout set. That caution has a recent, concrete precedent: reporting this month on OpenAI's GPT-5.6 Sol showed its ARC-AGI-3 score swing from 7.8 percent to 38.3 percent purely from changes to how the surrounding harness retained reasoning across turns and handled context, with no change to the model itself. ARC Prize's Francois Chollet has drawn a line between harness configurations that are general-purpose settings open to any API user and ones built specifically to game the benchmark, which the project's rules exclude. Memento 3's architecture is considerably more elaborate than a harness setting, but the same underlying lesson applies: on this benchmark, how an agent is wrapped can move the score as much as what model sits inside it, which is exactly the variable a frozen-LLM, rulebook-driven design is built to exploit.

What the paper does establish, pending outside replication, is a specific and checkable mechanism — compiled, verifiable world-model code edited through a defined accept/reject loop — rather than a vaguer appeal to "self-improvement." Whether that holds up against ARC Prize's semi-private set, or against adversarial environments designed after the paper's release, is the open question the authors have not yet answered in public.