benchmark challenges

fixed, revisitable model-vs-model sessions — niche task mixes saved forever in sqlite + committed json.

related: aiml api testing · zero context mode · realshot tasks · api

what you get

• manifest — data/challenges/{slug}/manifest.json defines tasks + models
• results — data/challenges/{slug}/results.json (committed for devlog / deploy revisit)
• sqlite — data/local.db tables challenges, challenge_runs (local forever)
• ui — /challenge/{slug} infographic + per-model drill-down
• api — GET /api/challenges, GET /api/challenges/{slug}

niche sprint v1

run it

# needs AIMLAPI_KEY in .env.local
npm run challenge
# or
npx tsx scripts/run-challenge.ts niche-sprint-v1

smoke-tests every model first, then runs all task×model combos (~60 API calls).

latest results (june 2026)

every model failed the ANSWER: 8 words constraint burst (partial credit only).

adding a new challenge

1. create data/challenges/my-challenge/manifest.json
2. run npx tsx scripts/run-challenge.ts my-challenge
3. commit manifest.json + results.json
4. open /challenge/my-challenge

manifest shape:

{
  "slug": "my-challenge",
  "title": "My Challenge",
  "subtitle": "…",
  "description": "…",
  "harness_mode": "zero_context",
  "tasks": [{ "id": "proc-sequence-5", "label": "…", "category": "procedural" }],
  "models": [{ "slug": "aiml/openai/gpt-4o-mini", "label": "GPT-4o Mini" }]
}

task ids can be procedural (proc-*), realshot (rs-*), or domain tasks (json-01, etc.).

storage

api prefers committed json for public deploys; sqlite is for local CLI runs.

drawing contest v1

top scores: GPT-OSS 120B + GLM 5.1 at 100%; GLM 5.2 / DeepSeek V4 Pro / Kimi K2.6 at 90% (rocket flame color). see drawing docs.

game night v1

scoreboard

unlike drawing contest (~90–100% across the board), game night spreads hard — models leak chain-of-thought into answers under zero context (one-word / ALL CAPS / haiku contracts expose narrators). GPT-OSS’s only miss: empty output on the no-letter-e pizza lipogram.

tasks live in data/tasks/instruction.json as ins-game-*.

production (vercel)

deployed api routes read from committed json via lib/challenges/store-json.ts — no sqlite on serverless. see deploy.

phase 2 — BYOK bench (live)

/bench now has a use my own API key toggle. keys stay in sessionStorage — sent per request only.

POST /api/eval/run accepts optional provider:

{
  "taskId": "json-01",
  "provider": {
    "name": "my model",
    "baseURL": "https://api.aimlapi.com/v1",
    "apiKey": "…",
    "model": "openai/gpt-4o-mini"
  },
  "harnessMode": "zero_context"
}

POST /api/eval/suite also accepts provider for full-domain BYOK sweeps.

test connection uses the same pong smoke as realshot (PUT /api/realshot/duel).

phase 3 — build your own sprint (ad-hoc)

/challenges/new — pick preset tasks, add 1–6 BYOK models, run, get the same infographic inline.

POST /api/challenges/run — ad-hoc challenge (max 15 tasks × 6 models). returns full ChallengeResults json. download from the ui — not persisted on vercel unless you commit json locally.

roadmap