OpenAI claims GPT-5.6 Sol beats Opus 5 on ARC-AGI-3 but only with its own custom test harness

Editor's take

OpenAI's recent demonstration of GPT-5.6 Sol achieving a 38.3% score on the ARC-AGI-3 benchmark, while impressive on its proprietary test harness, highlights the growing divergence in how LLM capabilities are measured. The stark drop to 7.8% in the standard environment underscores the critical need for standardized, unbiased evaluation frameworks that reflect real-world performance rather than optimized internal metrics.

This development matters because it directly impacts the perceived progress and competitive landscape in large language models, influencing investment and research directions. Anthropic's Opus 5, for instance, achieved a solid 30.2% score within the official testing conditions, suggesting a more consistent performance profile. The discrepancy raises questions about the replicability of OpenAI's results and the potential for "benchmark hacking."

Future attention should focus on the development and adoption of universally accepted evaluation protocols. The AI community needs to understand whether GPT-5.6 Sol’s proprietary methods offer genuine advancements in reasoning or are simply a consequence of specific data preprocessing and API-level optimizations. A shift towards external, transparent validation will be crucial for fostering trust and driving meaningful progress.