New benchmarks show AI models lag humans on visual inference but beat experts on research design
Two benchmarks released within days of each other this month are complicating the narrative that frontier AI models are closing in on human-level general reasoning. One, built by Scale AI and the multimodal-reasoning lab Elorian, finds that even the best current model answers barely half of a new visual-inference test correctly, far behind ordinary human test-takers. The other, built by independent researchers publishing as P-Zero Research, finds that Anthropic's Opus 5.5 already designs scientific experiments more efficiently than expert human researchers. Taken together, the two results suggest AI capability is advancing unevenly across cognitive domains rather than uniformly toward general intelligence.
A visual blind spot
Humanity's Sixth Sense (HSS), published October 7, tests something narrower than standard perception benchmarks: whether a model can infer what a scene implies but does not literally show. Its 522 tasks, drawn from 288 images and 234 video clips, span four categories — temporal and causal dynamics, physical and spatial logic, social understanding, and abstract or contextual inference. The top-scoring model at launch, GPT-6 Astra, answered 53.6% of tasks correctly; the median across 25 evaluated models was 30.9%. Twenty human participants working under the same free-form conditions scored 93.1%. According to reporting on the benchmark, the most common failure modes were missing a deciding visual detail (21% of errors) and misidentifying an object, person or role (20%).
Measures visual inference that standard perception and knowledge tests can miss.
That gap matters because HSS was explicitly designed to strip out tasks that reward memorized world knowledge or simple object recognition, isolating the kind of implicit, contextual inference humans do almost automatically. The benchmark's authors found that giving models more reasoning time did not meaningfully close the gap, suggesting the shortfall is in what models attend to and connect, not merely how long they think.
Research taste has already flipped
A second benchmark, TasteVal, points the opposite direction. Posted to arXiv on October 5 by researchers Oliver Jaffe and Dane Sherburn, it measures what the authors call "experimental research taste": given a fixed, unfamiliar research problem, how efficiently a model designs experiments and draws correct conclusions from the results. To isolate judgment from coding skill, the evaluated model only plans and interprets; a separate, fixed coding agent implements each experiment on a single H100 GPU, with runs capped at 40 GPU-hours. The baseline comes from 24 human experts, at least two per task, drawn from organizations including OpenAI, Google DeepMind, Nvidia and Microsoft Research.
On that metric, Opus 5.5 exceeded the human expert baseline with a compute multiplier of 2.3x — meaning it matched or beat expert-quality conclusions using less than half the experimental compute — while spending roughly one-thirtieth of what the human baseliners spent per attempt. The paper's authors measured a sharp inflection: frontier models' compute-efficiency on this task has doubled roughly every three months since December 2025, compared with roughly every 14 months in the prior three years.
Neither result should be read in isolation. TasteVal's authors caution that their benchmark measures only experiment execution on a problem someone else has already framed, not the harder skill of choosing which problems are worth pursuing — and that its tasks were chosen partly because they are fast and cheap to verify, which may flatter recent optimization. HSS, for its part, still shows AI closer to a coin flip than to human performance on a test designed around everyday inference rather than specialized expertise. Read side by side, the two benchmarks argue against a single storyline about AI progress: on narrow, well-defined technical judgment, models are compounding gains quickly; on the kind of implicit, contextual reasoning humans barely notice themselves doing, they remain some distance behind.