I compared Claude Opus 4.8 with 4.7 in a 10-round honesty test - and a legal prompt broke it

Editor's take

Anthropic's Claude Opus 4.8 faltered on a specifically designed legal hypothetical, demonstrating a vulnerability not present in its predecessor, Opus 4.7, during a series of honesty-based tests.

This outcome is significant as it highlights the nuanced challenges in developing truly reliable LLMs, especially for high-stakes applications like legal consultation. The regression from Opus 4.7 suggests that advancements in model capabilities don't always translate to uniform improvements across all domains.

Future testing should focus on whether this legal flaw is an isolated incident or indicative of broader systemic issues in Opus 4.8's reasoning. The comparative performance against other leading models, like OpenAI's GPT-4 Turbo, will be crucial in assessing Claude's current standing in the competitive LLM landscape.