A preprint by Axel Backlund and Lukas Petersson of Andon Labs. It tests whether models stay coherent over a long run by having them run a simulated vending business, starting with $500. The best model, Claude 3.5 Sonnet, ended with a mean net worth of $2,217.93, against $844.05 for a person who played for five hours. But "all models have runs that derail", and the best model's worst run ended at $476. The site cites it as an early test of an agent running a business on its own.
- Cited in
- a method page on this site
- Used for
- Strand
- The agent economy
Cite
@misc{backlund2025vending,
title = {Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents},
author = {Axel Backlund and Lukas Petersson},
year = {2025},
url = {https://arxiv.org/abs/2502.15840},
}