Measuring AI Ability to Complete Long Software Tasks

A study measuring AI ability to complete long software tasks, rated by how long skilled people take. Both papers cite it for the trend that the tasks frontier agents complete alone with 50% success lengthened from seconds in 2019 to about twelve hours by early 2026. It is the main capability trend in Fig. 3a, which the argument needs to continue until agents can pay their own way.

Cited in

both papers (Flagship: Where agent lineages stand, Fig. 3; Companion §1, Fig. 3)

Used for

Capability trends; Fig. 3a

Strand

Frontier AI capabilities · Capability trends & benchmarks

Cite

@misc{kwa2025measuring,
  title = {Measuring AI Ability to Complete Long Software Tasks},
  author = {Thomas Kwa and Ben West and Joel Becker and Amy Deng and Katharyn Garcia and Max Hasin and Sami Jawhar and Megan Kinniment and Nate Rush and Sydney Von Arx and et al.},
  year = {2025},
  howpublished = {arXiv:2503.14499v4},
  url = {https://arxiv.org/abs/2503.14499v4},
}