Half of AI-written code that passes industry test would get rejected by real developers, new study finds

Editor's take

A recent study by METR revealed that approximately 50% of AI-generated code solutions, while capable of passing the SWE-bench benchmark, would be rejected by human developers for real-world projects. This finding highlights a significant gap between current AI code generation capabilities and the practical demands of software engineering.

This discrepancy is crucial because it suggests that existing benchmarks may not accurately reflect the complexities and nuances of production codebases. Developers, who are ultimately responsible for maintaining code quality, security, and integration, require solutions that go beyond mere functional correctness as defined by automated tests. The METR study implies that current AI models, like those powering GitHub Copilot or Amazon CodeWhisperer, still require substantial human oversight and refactoring before deployment.

Moving forward, the key question is whether AI models can be trained to better understand and adhere to established coding conventions, architectural patterns, and project-specific requirements. Future developments will likely focus on AI systems that can not only generate functional code but also produce code that is readily understandable, maintainable, and idiomatic within existing development workflows. Observing improvements in how AI models handle code reviews and integrate with developer feedback loops will be critical.