aiml api provider

slopmark can run live evals through AI/ML API — an OpenAI-compatible gateway with 600+ models behind one key.

use this for cheap bench testing before burning OpenRouter credits or running full leaderboard sweeps.

setup

1. get a key at aimlapi.com/app/keys
2. add to .env.local (never commit this file):

AIMLAPI_KEY=your_key_here
AIMLAPI_BASE=https://api.aimlapi.com/v1

1. restart npm run dev so Next.js loads the env

.env.example has the variable names; copy it to .env.local and fill in values.

model slugs

prefix models with aiml/ in bench, baseline scripts, or API calls:

defined in lib/aimlapi.ts. cross-check ids at aimlapi.com/models if a call fails.

credit policy

claude and anthropic models are blocked for aiml/ slugs in code. they are expensive on this provider and not needed for bench validation.

// lib/aimlapi.ts — throws if slug matches claude/anthropic
assertAimlModelAllowed(model)

stick to the low-tier list above for testing.

smoke test

confirm key + model id before running suites:

curl -X POST http://localhost:3000/api/providers/smoke \
  -H "content-type: application/json" \
  -d '{"modelSlug":"aiml/openai/gpt-4o-mini"}'

expect { "ok": true }.

running evals

bench ui

/bench model dropdown has an aiml — low-tier testing group. default selection is gpt-4o mini (aiml).

start with run task on one procedural or extract task — not run full suite until smoke passes.

baseline script

npx tsx scripts/baseline.ts --domain procedural --model aiml/openai/gpt-4o-mini

when AIMLAPI_KEY is set and no --model flag is passed, baseline defaults to all aimlTestModels instead of openrouter slugs.

realshot

paste the same base URL and key in the agent form:

• base URL: https://api.aimlapi.com/v1
• model: openai/gpt-4o-mini (no aiml/ prefix in realshot — that's bench-only)

openrouter vs aiml

both use the same harness and verifiers — only the provider path changes.

troubleshooting