benchmark challenges

fixed, revisitable model-vs-model sessions — niche task mixes saved forever in sqlite + committed json.

related: aiml api testing · zero context mode · realshot tasks · api

what you get

• manifest — data/challenges/{slug}/manifest.json defines tasks + models
• results — data/challenges/{slug}/results.json (committed for devlog / deploy revisit)
• sqlite — data/local.db tables challenges, challenge_runs (local forever)
• ui — /challenge/{slug} infographic + per-model drill-down
• api — GET /api/challenges, GET /api/challenges/{slug}

niche sprint v1

run it

# needs AIMLAPI_KEY in .env.local
npm run challenge
# or
npx tsx scripts/run-challenge.ts niche-sprint-v1

smoke-tests every model first, then runs all task×model combos (~60 API calls).

latest results (june 2026)

every model failed the ANSWER: 8 words constraint burst (partial credit only).

adding a new challenge

1. create data/challenges/my-challenge/manifest.json
2. run npx tsx scripts/run-challenge.ts my-challenge
3. commit manifest.json + results.json
4. open /challenge/my-challenge

manifest shape:

{
  "slug": "my-challenge",
  "title": "My Challenge",
  "subtitle": "…",
  "description": "…",
  "harness_mode": "zero_context",
  "tasks": [{ "id": "proc-sequence-5", "label": "…", "category": "procedural" }],
  "models": [{ "slug": "aiml/openai/gpt-4o-mini", "label": "GPT-4o Mini" }]
}

task ids can be procedural (proc-*), realshot (rs-*), or domain tasks (json-01, etc.).

storage

api prefers committed json for public deploys; sqlite is for local CLI runs.

drawing contest v1

top scores: GPT-OSS 120B + GLM 5.1 at 100%; GLM 5.2 / DeepSeek V4 Pro / Kimi K2.6 at 90% (rocket flame color). see drawing docs.

game night v1

scoreboard

unlike drawing contest (~90–100% across the board), game night spreads hard — models leak chain-of-thought into answers under zero context (one-word / ALL CAPS / haiku contracts expose narrators). GPT-OSS’s only miss: empty output on the no-letter-e pizza lipogram.

tasks live in data/tasks/instruction.json as ins-game-*.

slop chaos v1

designed to show where “frontend kings” like Kimi K3 still trip on stupid, simultaneous formatting contracts. no vibes scoring — rule parser only.

seed prompts also live in data/tasks/slop_chaos.json (merged into instruction/json pools for the harness).

scoreboard (jul 29 2026)

ran via OpenRouter twins (FIREWORKS_API_KEY was not in local .env.local). re-run with Fireworks when the key is present: npm run challenge:slop-chaos.

universal fails (0/5): haunted toaster (banned glue words + exact count), racoon lipogram (no letter e), shoe-tying questions with taboo words.

universal passes (5/5): meme math 21 meme, yes/no/maybe-later paragraphs.

Kimi K3 still loses hard on multi-constraint lipogram / glue-word bans — the “frontend” model is not a constraint model.

challenge receipt pages (/challenge/[slug]) lead with a shareable receipt hero: top pass %, universal wipeout/clear chips, one specimen fail (prompt + verifier line + output), copy-link / share-card, and an Open Graph image. task briefing + per-run verifier chips sit below; json/csv export is tucked at the bottom.

trap season v2

second pack in the Trap Season cadence — same “stupid simultaneous contracts” mold as Slop Chaos, new jokes. seed prompts live in data/tasks/trap_season.json and are merged into instruction.json / json.json for the harness.

scoreboard (jul 30 2026)

ran on Fireworks serverless. classic 5-model panel first, then appended frontier extras (npm run challenge:trap-season-extras).

hardest trap: cursed HR SaaS elevator pitch — only Kimi K2.7 Code cleared it (100%); everyone else scraped 20% partial.

universal clears (9/9): vowel count [7], south wins meme, cactus JSON schema.

/bench does not expose Fireworks — that key is ops-only for publishing committed receipts. public host path stays OpenRouter free (+ BYOK / AIML when configured).

production (vercel)

deployed api routes read from committed json via lib/challenges/store-json.ts — no sqlite on serverless. see deploy.

phase 2 — BYOK bench (live)

/bench now has a use my own API key toggle. keys stay in sessionStorage — sent per request only.

POST /api/eval/run accepts optional provider:

{
  "taskId": "json-01",
  "provider": {
    "name": "my model",
    "baseURL": "https://api.aimlapi.com/v1",
    "apiKey": "…",
    "model": "openai/gpt-4o-mini"
  },
  "harnessMode": "zero_context"
}

POST /api/eval/suite also accepts provider for full-domain BYOK sweeps.

test connection uses the same pong smoke as realshot (PUT /api/realshot/duel).

phase 3 — build your own sprint (ad-hoc)

/challenges/new — pick preset tasks, add 1–6 BYOK models, run, get the same infographic inline.

POST /api/challenges/run — ad-hoc challenge (max 15 tasks × 6 models). returns full ChallengeResults json. download from the ui — not persisted on vercel unless you commit json locally.

roadmap