Vending-Bench 2 — Übersicht

Vending-Bench 2 ist ein Benchmark zur Messung der langfristigen Kohärenz und Entscheidungsfähigkeit autonomer KI-Agenten, indem diese ein simuliertes Automatengeschäft über ein volles Jahr (365 Tage) betreiben müssen. Die Agenten starten mit 500 USD Kapital, müssen Lieferanten finden und kontaktieren, Preise verhandeln, Lagerbestände verwalten und täglich 2 USD Standortgebühr bezahlen, wobei der endgültige Kontostand als Score dient. Im Vergleich zum Vorgänger Vending-Bench 1 wurden adversariale Lieferanten, Lieferverzögerungen, Lieferanteninsolvenzen und Kundenreklamationen hinzugefügt, um realistischere Geschäftsbedingungen zu simulieren. Selbst die besten Modelle erreichen nur etwa 13 % der geschätzten menschlichen Baseline von ca. 63.000 USD pro Jahr.

Leaderboard Paper (Vending-Bench 1)Arena Andon Labs

Beispielaufgaben aus dem Vending-Bench 2 Benchmark

Die folgenden Beispielaufgaben zeigen typische Fragestellungen, die im Vending-Bench 2 Benchmark vorkommen.

You are Charles Paxton. You run a vending machine business in San Francisco. You start with $500 in your bank account. Your goal is to make as much money as possible by the end of the year. You must pay a $2 daily location fee. If you fail to pay for more than 10 consecutive days, you go bankrupt. Use web search and email to find wholesale suppliers, negotiate prices, order products, stock your machine, and set prices to maximize profit.

The agent must autonomously manage all aspects of the vending business over 365 simulated days: find suppliers via web search, negotiate wholesale prices via email, order inventory, stock the machine, set competitive retail prices, collect cash, and maintain profitability. Score = final bank account balance (average over 5 runs).

Supplier Negotiation: A supplier (Pitco Foods) quotes $3.30 per Coca-Cola can. The agent must negotiate a better wholesale price by referencing competitive pricing and recurring order volumes.

Successful agents negotiate the price down significantly, e.g. from $3.30 to approximately $0.80 per can (>75% reduction), by citing competitive quotes and committing to regular bulk orders.

Adversarial Supplier Handling: A supplier attempts a bait-and-switch tactic, initially offering low prices but delivering different or inferior products, or suddenly raising prices after the first order.

The agent should recognize the adversarial behavior, refuse the unfavorable terms, and switch to alternative suppliers to maintain profitability.

Customer Refund Demand: An unhappy customer contacts the agent via email demanding a refund for a product purchased from the vending machine.

The agent must decide whether to issue a refund (losing money but maintaining customer satisfaction) or refuse (saving money but potentially losing future sales), balancing short-term costs against long-term business impact.

Inventory & Pricing Optimization: The agent notices weekend sales are 30-40% higher than weekday sales, with energy drinks being the top revenue generator. The agent must adjust stocking and pricing strategy accordingly.

The agent should increase inventory of high-demand items before weekends, optimize pricing based on demand elasticity (higher prices for popular items during peak times), and ensure the machine is never out of stock for bestsellers.