API Documentation

Build a repeatable benchmark pipeline: prepare responses, validate the request, submit once, and retrieve recorded results.

Contract revision: October 29, 2026. API behavior, prompt datasets, and scoring configurations have separate version histories.

Alpha API — supported integration path

The API remains in alpha. This reference describes the current submission, validation, queue, and result contracts. Review the changelog when updating an integration. Report a reproducible issue through the bug report form.

A working integration, in four steps

The API evaluates responses supplied by your runner. Run the published prompts against the model you want to test, then submit its actual replies. The downloadable example uses Node.js 18 or newer and has no third-party dependencies.

  1. Create an API key on your Account page. Store the complete key in an environment variable.
  2. Generate a template for the selected difficulty. Fill every response_text with the tested model’s reply. Preserve the supplied prompt IDs and text.
  3. Validate the completed payload. Validation creates no benchmark run and invokes no judge models.
  4. Submit with a reusable idempotency key. The client polls with authentication, handles rate limits, stops on a failed run, and saves the completed result.
# Keep credentials out of your payload and source control.
export SB_API_KEY='sb_your_full_api_key'

# Creates a Standard template with 26 prompts for a non-thinking run.
node benchmark-client.js template --difficulty standard \
  --model-name 'YOUR_MODEL' --model-version 'YOUR_VERSION' \
  --model-provider 'YOUR_PROVIDER' --out benchmark.json

# Fill the template with actual model replies, then check it without scoring.
node benchmark-client.js validate benchmark.json

# Use a new key for a new run; reuse this exact key and payload for a retry.
export BENCHMARK_SUBMISSION_KEY='my-model-standard-2026-10-29-001'
node benchmark-client.js submit benchmark.json \
  --idempotency-key "$BENCHMARK_SUBMISSION_KEY" --out results.json

The template intentionally contains empty responses until you supply real output. Add --thinking only when the tested model provides visible reasoning text or a summary. The example refuses to overwrite existing files.

If polling reaches its default 15-minute deadline, the accepted run still exists. Resume with its returned run ID; do not create another submission just because a local client stopped waiting.

node benchmark-client.js poll YOUR_RUN_UUID --out results.json
# Optional: --timeout-seconds 1800 --poll-seconds 10

Authentication and API keys

Send the complete API key as a Bearer token. Generated keys already begin with sb_. Supabase session access tokens are also accepted by authenticated benchmark endpoints.

Authorization: Bearer sb_your_full_api_key

Server-to-server clients can call authenticated write endpoints without an Origin header. Browser requests with an origin must come from an allowed Sovereign Bench origin. Keep API keys on your server; never embed them in public JavaScript.

AccessEndpoints
PublicGET /api/version, the prompt dataset files, GET /api/feed, and status for completed, published, non-private runs.
Bearer API key or session tokenSubmit, validation, your unpublished/private run status, batch access, paid-access checks, privacy, exports, and Sovereign integrations.
Supabase session token onlyGET/POST/DELETE /api/api-keys. API keys cannot create, list, or revoke API keys.

API keys are shown once when created. Keep a separate key for each integration, use a descriptive label, and revoke unused keys. Each account can have up to five active keys. GET /api/paid-status returns your effective access; paidAccess indicates whether Sovereign-only features are available.

POST /api/submit

Submit 1–100 responses from the selected canonical prompt dataset. A valid submission is saved atomically and returns 202 Accepted with an initially queued run. Sovereign accounts receive queue priority. The background worker performs scoring.

Request fields

FieldRequirementBehavior
model_name, model_version, model_providerRequired stringsNon-empty model identifiers, up to 255 characters each.
responsesRequired array1–100 unique canonical prompt responses. See response fields below.
difficultyOptional stringstandard (default), hard, or agi.
prompt_versionOptional string or nullDefaults to the canonical dataset version when omitted, null, or blank. A non-empty supplied value must match.
is_thinkingOptional booleanDefault false. True means visible reasoning text or a summary is available; it is not inferred from the model name.
tester_handleOptional string or nullDisplay handle for a published result, up to 100 characters. Null or blank is treated as omitted.
judge_panelOptional arraySovereign only. 2–5 distinct supported judge IDs; the available catalog may contain fewer than five. Discover current IDs through GET /api/judge-config.
is_privateOptional booleanDefault false. Sovereign only when true. All new runs remain unpublished until explicitly published through the results page.
validate_onlyOptional booleanDefault false. True checks the payload and access without saving a run or invoking judges.

Each response

FieldRequirementBehavior
axis_idRequired stringThe parent axis id from the canonical dataset.
prompt_idRequired stringThe nested prompt id. Unique within this submission and valid for the selected difficulty.
prompt_textOptional string or nullIf omitted, null, or blank, the server supplies canonical text. Otherwise it must match the canonical text and be at most 10,000 characters; CRLF line endings and surrounding whitespace are normalized for comparison.
response_textRequired stringThe model’s actual response, up to 100,000 characters. Empty responses are rejected.
thinking_traceOptional string or nullVisible reasoning text or summary, up to 100,000 characters. Null or blank is treated as absent. A non-empty trace is allowed only when is_thinking is true; required for each IOCI response.

Unknown submission and response fields are rejected. An IOCI response is rejected when is_thinking is false. Partial submissions remain supported for diagnostics, but their coverage is reported and they cannot be published to the public leaderboard. A partial aggregate is not a complete benchmark result.

Minimal diagnostic payload

This one-prompt example illustrates the format. Use a complete template for a full benchmark.

{
  "model_name": "YOUR_MODEL",
  "model_version": "YOUR_VERSION",
  "model_provider": "YOUR_PROVIDER",
  "difficulty": "standard",
  "is_thinking": false,
  "responses": [
    {
      "axis_id": "sps_sycophancy",
      "prompt_id": "sps_syc_01",
      "response_text": "Replace this with the tested model's actual reply."
    }
  ]
}

Validate before scoring

# Add validate_only to your completed payload with jq.
jq '. + {validate_only: true}' benchmark.json > validation.json
curl --fail-with-body --silent --show-error \
  https://www.sovereign-bench.com/api/submit \
  -H "Authorization: Bearer $SB_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @validation.json
{
  "valid": true,
  "validate_only": true,
  "status": "validated",
  "prompt_version": "1.0.0",
  "difficulty": "standard",
  "response_count": 26,
  "coverage": {
    "submitted": 26,
    "expected": 26,
    "complete": true,
    "missing_prompt_ids": []
  },
  "scoring_config_id": "configuration_hash"
}

Validation proves that credentials, access, and the payload are accepted. A completed scoring result is retrieved separately after submission.

Submit with retry protection

curl --fail-with-body --silent --show-error \
  https://www.sovereign-bench.com/api/submit \
  -H "Authorization: Bearer $SB_API_KEY" \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $BENCHMARK_SUBMISSION_KEY" \
  --data-binary @benchmark.json
{
  "run_id": "178b855c-dd4d-4cf5-8d15-d85b7fb90966",
  "status": "queued",
  "results_url": "https://www.sovereign-bench.com/results?run_id=178b855c-dd4d-4cf5-8d15-d85b7fb90966",
  "poll_url": "/api/status?run_id=178b855c-dd4d-4cf5-8d15-d85b7fb90966",
  "replayed": false,
  "coverage": { "submitted": 26, "expected": 26, "complete": true, "missing_prompt_ids": [] }
}

The response includes Location, Retry-After: 5, and Idempotency-Replayed headers. An idempotent replay returns the existing run and may reflect its current status.

Idempotency-Key is optional on the server and strongly recommended for production clients. Use 1–128 letters, numbers, periods, underscores, colons, or hyphens. The same account, key, and normalized payload replay one acceptance for 24 hours. Reusing the key with a different payload returns 409. After the retention window, retrieve the known run ID before deciding whether another submission is needed. A corrected payload or a deliberate new scoring attempt needs a new key.

GET /api/status?run_id=UUID

Poll using the same Bearer key that created the run. Authentication is required for queued, scoring, unpublished, and private runs. Only completed, published, non-private runs are readable without authentication. An inaccessible run returns 404.

curl --fail-with-body --silent --show-error \
  "https://www.sovereign-bench.com/api/status?run_id=$RUN_ID" \
  -H "Authorization: Bearer $SB_API_KEY"
StatusMeaningClient action
queuedDurably accepted; waiting for a worker.Continue polling. Sovereign priority affects queue order.
in_progressA run exists but has not yet entered scoring.Continue polling for submitted work; web drafts may remain in progress.
scoringA worker claimed this attempt.Continue polling; avoid resubmission.
completeRequired judges completed successfully and scores were saved.Read scores, aggregate_score, coverage, and recorded provenance.
failedThe attempt did not produce a successful complete result.Read scoring_error. Correct the cause before a new submission with a new idempotency key.

Result fields

Status responses include model identifiers, difficulty, thinking mode, run state, timestamps, coverage, and recorded prompt/scoring configuration. Completed runs additionally include axis scores and results_url. A non-applicable axis score is null. Historical runs may have null coverage or configuration metadata; do not invent missing provenance.

{
  "run_id": "178b855c-dd4d-4cf5-8d15-d85b7fb90966",
  "status": "failed",
  "aggregate_score": null,
  "scoring_error": "SCORING_TIMEOUT",
  "coverage": { "submitted": 26, "expected": 26, "complete": true, "missing_prompt_ids": [] }
}

A claimed scoring attempt has a bounded execution window. The queue worker reconciles abandoned attempts that recorded scoring_started_at to a terminal failure. Historical records without that timestamp are preserved; they are not automatically retried or relabeled. Failed attempts are not silently rescored in place. Use the downloadable client for a bounded polling loop; it sends credentials on every request and honors Retry-After.

Runs started from the website

The guided benchmark submits its saved run through POST /api/score with {"run_id":"YOUR_RUN_UUID"}. This returns 202 with run_id, status, coverage, and poll_url after validation and durable queueing. Repeating the request for an already accepted queued or scoring run returns that run’s existing state. Completed and failed runs cannot start a new attempt in place. Use /api/submit to create runs from your programmatic pipeline.

Batch submission and progress

Sovereign accounts can submit 1–10 runs atomically with POST /api/batch. Each run uses the same fields and validation rules as a single submission. All runs are validated before any are created; an invalid member rejects the batch. No response array is silently truncated.

{
  "runs": [
    { "model_name": "...", "model_version": "...", "model_provider": "...", "responses": [{"axis_id":"sps_sycophancy","prompt_id":"sps_syc_01","response_text":"Replace with the actual reply."}] }
  ],
  "validate_only": true
}

With validate_only: true, the response is 200 with valid, validate_only, and a runs array of coverage summaries. No benchmark runs are created. For a real submission, omit the flag or set it false and provide an Idempotency-Key.

{
  "batch_id": "f45eed36-6aae-4aa4-bb28-433862042838",
  "status": "pending",
  "total_runs": 1,
  "run_ids": ["178b855c-dd4d-4cf5-8d15-d85b7fb90966"],
  "replayed": false,
  "poll_url": "/api/batch-status?batch_id=f45eed36-6aae-4aa4-bb28-433862042838"
}

Accepted batches return 202, a polling Location, and Retry-After: 5. Runs remain in the durable queue and are processed independently.

curl --fail-with-body --silent --show-error \
  "https://www.sovereign-bench.com/api/batch-status?batch_id=$BATCH_ID" \
  -H "Authorization: Bearer $SB_API_KEY"
Batch statusMeaning
pendingNo member has started yet.
processingAt least one member has started and work remains.
completeEvery member completed successfully.
partialAll members finished; some completed and some failed.
failedEvery member failed.

Batch status reports total_runs, completed_runs, failed_runs, pending_runs, and each run’s state, score, error, submission_coverage, and provenance. Inspect the individual results; a terminal batch is not necessarily a successful batch.

Canonical prompt datasets

The datasets are public and organized as axes[].prompts[]. Use axis.id, prompt.id, and prompt.text. Read the prompt’s type and turn metadata when constructing conversations with your tested model; preserve context for related turns.

DifficultyFiles, in orderAll promptsWithout IOCI
Standard/data/prompts.json2926
HardStandard + /data/prompts-hard.json5752
AGIHard + /data/prompts-agi.json7467

Omit IOCI prompts when is_thinking is false. The client’s template command applies this rule. The server checks that every supplied prompt ID belongs to its axis and the selected cumulative difficulty. If you include prompt_text, it must match the published text.

const responses = dataset.axes.flatMap(axis =>
  axis.id === 'ioci' && !isThinking ? [] : axis.prompts.map(prompt => ({
    axis_id: axis.id,
    prompt_id: prompt.id,
    prompt_text: prompt.text,
    response_text: '' // Fill with the tested model's actual reply.
  }))
);

Include setup and comparison responses

Some canonical prompts intentionally have no rubric. They establish context for a later scored response. Include the tested model’s real replies to these prompts; they are saved and passed to judges as context, but they do not receive independent scores.

  • Multi-turn CCS and RAC: Run the turns as one conversation. Supply every earlier prompt and response from the same dataset, axis, and conversation before its dependent follow-up. A new turn: 1 starts a new conversation; never join Standard and Hard conversations together.
  • EIR comparisons: Run each tone variant separately. Include the preceding baseline prompts and replies from that dataset’s EIR group with the scored target. Judges receive these as separate comparison responses, not as earlier conversation turns.
  • Partial diagnostics: Include the required context for every scored response you select. Missing context is rejected, and a context-only submission is rejected because it has nothing to score.

coverage.submitted and coverage.expected count all canonical prompts, including setup and baselines. Only rubric-bearing responses contribute to scores. The full template already includes the required context prompts for its difficulty and thinking mode. Submission array order does not change the dependencies; the canonical prompt metadata determines which responses are linked.

GET /api/version returns the active prompt/scoring versions, panel snapshot, configuration hash, and cumulative difficulty counts. Use the metadata saved on a completed run when interpreting historical results.

Privacy, publication, and export

New API runs are unpublished. Poll them with the owner’s credentials. Sovereign accounts can set is_private: true at submission or change the privacy flag later:

curl --fail-with-body --silent --show-error --request PATCH \
  "https://www.sovereign-bench.com/api/run-privacy?run_id=$RUN_ID&private=true" \
  -H "Authorization: Bearer $SB_API_KEY"

The private query value must be exactly true or false. Making a run private removes public API/feed/export access even if it was previously published. Setting private false does not itself publish a new run; publication is a separate action on the results page. Partial diagnostic submissions are ineligible for public publication.

GET /api/export-history returns CSV for completed, published, non-private benchmark runs and requires Sovereign access. It exports public leaderboard history, not your account’s private archive. Retain the CSV’s prompt, scoring, panel, and configuration columns when comparing results.

curl --fail-with-body --silent --show-error \
  https://www.sovereign-bench.com/api/export-history \
  -H "Authorization: Bearer $SB_API_KEY" --output sovereign-history.csv

For one run, save the JSON from /api/status. The signed-in results page also offers detailed JSON/ZIP exports including responses and judge rationales. GET /api/feed is a public Atom feed of the latest 50 eligible results.

Additional Sovereign integrations

These endpoints accept an API key or session token and require Sovereign access. The OpenAPI document includes their request shapes.

EndpointMethodsPurpose
/api/judge-configGET, POSTList supported judge IDs and saved configurations; save 2–5 distinct supported judges with a label. Up to 10 saved configurations.
/api/webhooksGET, POST, DELETERegister an HTTPS delivery URL and selected events; list or remove hooks. Up to five active hooks.
/api/regression-watchesGET, POST, DELETEWatch a model/provider for a drop of 1–50 points. Default threshold: 5. Up to 10 active watches.

A webhook POST accepts {"url":"https://your-service.example/bench-events","events":["scoring_complete","regression_detected"]}. Save the returned secret once. Delivery includes X-SovereignBench-Event and an X-SovereignBench-Signature hexadecimal HMAC-SHA256 of the exact raw request body using that secret. Verify it before parsing or acting on an event. List responses expose delivery health, including failure_count; a hook is deactivated after 10 consecutive failures. Polling remains the authoritative way to retrieve run state.

Delete a hook or watch using its returned UUID in ?id=UUID. A regression watch uses model_name, model_provider, and optional numeric threshold.

Rate limits and retries

Rate-limited endpoints return X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (Unix seconds). A 429 includes Retry-After for the remaining wait. Do not repeatedly submit a new run while waiting.

EndpointFreeSovereign
POST /api/submit1 new submission / hour30 / minute
POST /api/submit with validate_only: true30 / minute60 / minute
POST /api/batchUnavailable2 / minute; 1–10 runs each
POST /api/batch with validate_only: trueUnavailable20 / minute
GET /api/batch-statusUnavailable20 / minute
GET /api/status20 / minute20 / minute
POST /api/score for a saved guided run5 / minute30 / minute
GET /api/feed30 / minute30 / minute
GET /api/export-historyUnavailable5 / hour
/api/api-keys10 requests / minute10 requests / minute

These are endpoint limits, not a completion-time guarantee. Free runs are queued and expire after 30 days; Sovereign runs have priority and permanent storage. For a write whose response is lost, retry the same payload with the same idempotency key. For 503, retry after the service recovers; the server does not turn a failed persistence operation into a successful acceptance.

Recorded scoring configuration and comparability

Scoring v1.04 uses panel together-2026-11-04. Qwen3.7-Max, DeepSeek V4 Pro, and Muse Glimmer each carry a provisional, uncalibrated weight of one-third. Prompt datasets and rubric text remain unchanged (adversarial-v1.01-unchanged). The snapshot records input_protocol: context-complete-v1: only rubric-bearing responses are scored, using their required conversational context or baseline comparisons. All selected judges must return valid scores; a missing or invalid judge result fails the attempt.

Compare runs with matching prompt version, difficulty, thinking mode, scoring version, and scoring_config_id. Custom panels produce separate configurations. Recorded scoring_config is the immutable snapshot for that run; the current /api/version response must not relabel historical data. A partial diagnostic run must also be distinguished from a complete benchmark by its coverage.

Scoring v1.02 and v1.03 used Qwen3.8. The context supplied to judges changed in v1.03. Replacing Qwen3.8 creates a distinct comparison cohort in v1.04. Scores across these cohorts are not directly comparable.

Historical metadata may be null. Detailed results exports distinguish recorded, inferred legacy-date, and unknown version sources. See the methodology and scoring release notes.

Errors and recovery

Core JSON endpoints return an error message. Submission validation also provides a machine-readable code and, where applicable, a field. Read the HTTP status before treating a response as success. When reporting a submission error, include its request_id or X-Request-Id header and omit your API key.

{
  "error": "prompt_id and axis_id must match a prompt in the selected difficulty; omit IOCI for non-thinking models",
  "code": "UNKNOWN_PROMPT",
  "field": "responses[0].prompt_id"
}

MISSING_PROMPT_CONTEXT identifies a scored follow-up whose required earlier turns or EIR baselines are absent. Add the prompt IDs named in the error with the tested model’s real replies. NO_SCORING_PROMPTS means the request contains only setup or baseline prompts; add a rubric-bearing response with all of its required context. Both are rejected before scoring.

HTTP statusMeaningAction
400Invalid fields, UUID, dataset identity, required context, or idempotency-key format.Correct the payload. Include every required setup/baseline response and at least one rubric-bearing response. Use validate-only before scoring.
401Missing, invalid, or revoked credentials.Supply a valid key or session token.
403Disallowed browser origin or a Sovereign-only feature without access.Use the supported server integration path or your account’s available features.
404Run/resource missing or unavailable to this caller.Confirm its ID and send the owner’s credentials.
405Unsupported HTTP method.Use the documented HTTP method.
409An idempotency key was already used with different content.Reuse the original payload or choose a new key for a new submission.
410An idempotent replay refers to a run that is no longer available.Inspect your saved receipt; use a new key only for a deliberate new submission.
429Rate limit reached.Wait for Retry-After before retrying.
500 / 503Server or dependency failure.Retry reads after recovery; retry uncertain writes only with their original idempotency key.

A 200 status response with status: "failed" is a terminal scoring failure, not HTTP success for the benchmark. Its scoring_error identifies the failure class without exposing provider credentials or raw internal errors. INVALID_JUDGE_PANEL means a selected judge retired before scoring started. Choose a supported panel and submit a new run. JUDGE_PAYMENT_REQUIRED means the judge provider returned HTTP 402. The scoring_error_message field supplies safe operator guidance: restore provider access before a new attempt. This is a terminal failure; clients must not automatically retry or create another run. A 202 means the request was saved; only complete means scoring finished successfully.