New benchmark exposes how badly AI struggles with real knowledge work

Editor's take

A new benchmark reveals that even leading large language models, like OpenAI's GPT-4 and Google's Gemini Ultra, falter significantly when tasked with complex, multi-step knowledge work, successfully completing a mere 3% of the evaluated tasks. This finding underscores the gap between current AI capabilities and the nuanced, context-aware reasoning required for professional applications such as legal analysis or scientific research, areas where human expertise remains paramount.

The implications extend to the widespread deployment of AI in enterprise settings, suggesting that current models are ill-equipped for roles demanding intricate problem-solving and deep domain understanding beyond simple information retrieval. This benchmark, developed by researchers from Carnegie Mellon University, directly challenges overly optimistic projections about AI's immediate impact on knowledge professions.

Future developments to monitor include the efficacy of specialized AI agents designed to chain together multiple model calls or integrate external tools to tackle these complex tasks. The performance of upcoming models, particularly those addressing reasoning deficiencies, and the benchmark's own evolution to incorporate more diverse and challenging real-world scenarios will be critical indicators.