A METR research update of August 2025 that scored the same agent work two ways: by the project's own tests, and by people judging whether the code could be merged. An agent built on Claude 3.7 Sonnet passed the tests on 38% of the issues. But of the pull requests reviewed by hand, "none of them are mergeable as-is". The site cites it for why test scores run ahead of work a client would accept.
- Cited in
- a method page on this site
- Used for
- Strand
- Frontier AI capabilities
Cite
@misc{metr2025holistic,
title = {Research Update: Algorithmic vs. Holistic Evaluation},
author = {METR},
year = {2025},
howpublished = {METR blog},
url = {https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/},
}