The Benchmark That Fights Back

Every few weeks a new model is released and posts a new high score on some benchmark, and somehow you trust the benchmark a little less each time. Saturation, contamination, quietly teaching to the test. A number a model reports about itself, on a fixed exam it may well have seen, tells you less and less about whether the thing actually works.

The missing ingredient isn't harder questions. It's an opponent.

Make the output compete

A static benchmark asks a model a question and grades the answer against a key. A competitive challenge does something different: ask the model to produce a solution to a clearly defined but complex adversarial game, take what the model produced and put it up against what other models produced, and lets the outcome decide. There is no self-reported score. They climb or don't.

Three properties that most static benchmarks lack:

• Outcome-based. The score is determined compared to real unknown opposition, in a neutral 3rd party public platform, with all aspects logged, auditable and reviewable.
• Adversarial and moving. Your opponents are trying to beat you back, and as time goes by - they get better. A weakness a fixed test never probes gets found and punished in this environment.
• End-to-end. The model can't hand-wave. Its output has to be correct, complete, and runnable, or it loses on contact.
• Adjustable and Extensible. if you are interested in comparing and evaluating specific aspects of the model's products - find and make it compete on a game that highlights those attributes (or design a new one).

A proof of concept, in poker

We run a live arena for poker bots, so we had an easy way to test the idea. One prompt. Five flagship models: Claude, GPT, and Gemini, DeepSeek and Nova. Each was asked to one-shot a heads-up No-Limit bot in a single file, against a fixed SDK contract, inside a sandbox with a strict dependency allow-list, a five-second-per-move budget, a small response cap, and one hard rule: never crash, never time out. We took whatever each model produced, packaged it and uploaded as-is, and now the five bots are playing matches between themselves and other user controlled poker bots gathering hands and matches, being rated and placing on a live leaderboard.

Why this is hard to game

Scores drift from reality because static benchmarks are gameable: memorize the set, optimize the metric, contaminate the training data. An arena resists most of that by construction.

• There's no answer key to memorize. You're scored against changing opponents, not a fixed test.
• The opponent set evolves, so last month's solution doesn't keep winning.
• "Looks plausible" earns nothing. A bot that misreads the protocol or freezes on a hard decision gets no partial credit, it just loses. Correctness and completeness are scored automatically, by the game itself.

Bigger than poker

Writing a poker bot is one narrow slice, and we're not claiming a poker ladder ranks general intelligence. What generalizes is the format. Any task where you can take a model's output and make it compete against other models' outputs becomes an arena benchmark, with the same three properties and the same resistance to gaming. Poker is our wedge because it's clean, adversarial, and fully scorable. The benchmark idea is the bigger thing underneath it, and it's the same reason poker isn't solved the way people assume.

A test that fights back

As the saturation conversation gets louder, "the benchmark doesn't reflect real-world performance" keeps coming up. The answer might not be a harder static exam. It might be a test that fights back: one where the only way to score is to ship something that survives contact with an adversary trying just as hard as you are.

This experiment is live and viewable on Chipzen. We'll follow up as information is generated and gathered, in the meantime if you've got a model you want to add into the mix - ask it to write a bot using the same prompt we used and add it to the arena - The arena is open.

Appendix — The prompt

This is the exact prompt each model received, unchanged. Want to add a model to the mix? Hand it this, then upload the bot it writes.

You are writing a heads-up No-Limit Texas Hold'em poker bot that will compete
on Chipzen, an arena where AI bots play each other on a live rated ladder. Write
the strongest bot you can, as a single self-contained Python file using the
Chipzen SDK. It must run unmodified.

## The SDK interface (this is the whole contract)

from chipzen import Bot, GameState, Action

class MyBot(Bot):
    def decide(self, state: GameState) -> Action:
        ...

decide(state) is called every time it's your turn and must return an Action
within 5000 ms (otherwise the server auto-checks if legal, else folds you).

GameState fields (everything you get):
- hole_cards: list[Card]  # your 2 cards. Card has .rank ("2"-"9","T","J","Q","K","A") and .suit ("h"/"d"/"c"/"s")
- board: list[Card]       # community cards: 0 preflop, 3 flop, 4 turn, 5 river
- phase: str              # "preflop" | "flop" | "turn" | "river"
- pot: int, your_stack: int, opponent_stacks: list[int]
- your_seat: int, dealer_seat: int   # heads-up: the dealer acts first preflop
- to_call: int           # chips to call (0 means checking is free)
- min_raise: int, max_raise: int     # legal raise range, as a TOTAL bet size (not an increment); max_raise is all-in
- valid_actions: list[str]           # subset of ["fold","check","call","raise","all_in"]; only return one of these
- action_history: list[dict]         # actions this hand: {seat, action, amount, phase}

Action API (return exactly one):
- Action.fold(), Action.check(), Action.call(), Action.all_in()
- Action.raise_to(amount)   # amount is the TOTAL bet; must satisfy min_raise <= amount <= max_raise

Optional hooks you may override for opponent tracking: on_hand_start(hand_number, hole_cards),
on_phase_change(msg), on_turn_result(msg), on_hand_result(result), on_match_start(info), on_match_end(results).

## Constraints
- Heads-up NLHE only. Pure Python. The container's pip allow-list is STRICT: the
  ONLY installable packages are `websockets`, `numpy`, `scipy`, and `chipzen-bot`
  (the SDK). Any other dependency (e.g. `treys`, `pokerkit`, `deuces`) will be
  REJECTED at install time and the bot will fail to run. So either rely on the
  standard library alone, or only `numpy`/`scipy` if you declare them in
  requirements.txt. Keep it lightweight: 5s/turn budget and a 4 KB response limit.
- ALWAYS return an action from state.valid_actions. Guard every branch with a safe
  fallback (check/call/fold). The bot must never crash or time out.
- Clamp any raise amount into [min_raise, max_raise].

## What to build
Make it genuinely competitive, not a toy: preflop ranges, postflop made-hand/draw
equity (a quick Monte Carlo or a hand-eval lib), pot odds for calls, position
(dealer vs non-dealer), stack-depth/SPR-aware sizing, short-stack all-in logic, and
some balance (value bets plus a controlled bluff frequency) so you're not trivially
exploitable. Briefly explain your strategy in comments.

## Output
Return a complete bot.py defining a Bot subclass with decide() (plus a
requirements.txt if you use pip deps). It must import and run against the interface above unmodified.

Want to see what the prompt actually produces? Read one model's full one-shot bot (Gemini), unedited.

Not sure how to get from code to an uploaded bot? Here's the step-by-step, to save you the digging.