Changelog

Every change to the methodology, prompt sets, and scoring architecture is documented here with full rationale. Sovereign Bench is versioned so that comparability can be assessed using prompt version, difficulty, scoring version and recorded panel configuration.

v1.04 judge-panel configuration — November 4, 2026

Together AI announced the serverless retirement of Qwen/Qwen3.8-2.4T-A95B. Panel together-2026-11-04 selects its recommended successor, Qwen/Qwen3.7-Max, with provider-default reasoning, streamed JSON output, and a 16,384-token output ceiling. DeepSeek V4 Pro and Muse Glimmer retain their settings. Each judge contributes one-third and every judge must return a complete score and rationale.

Comparability. The judge substitution creates a distinct comparison cohort from v1.03 even with the same prompt datasets, rubric, and context-complete-v1 input protocol. Compare matching recorded configurations. Earlier runs retain their scores and snapshots. The equal weights remain provisional pending live calibration.

Read the deployed configuration from the version endpoint.

v1.03 and API reliability — October 29, 2026

Input protocol: context-complete-v1 · Existing panel and provisional equal weights · Prompt datasets and rubric text unchanged

What Changed

Verification and Comparability

Provider failure diagnostics: Judge rejections retain safe HTTP-status diagnostics in server logs. A provider payment rejection is reported as JUDGE_PAYMENT_REQUIRED, with operator guidance in API results, the downloadable client and the results page. The attempt remains failed, produces no partial panel score, and is not automatically retried or resubmitted. This changes error reporting only; the scoring configuration is unchanged.

Offline HTTP and PostgreSQL tests exercise acceptance, rollback, retries, role boundaries, scoring with controlled judge responses and authenticated result polling. Full Standard, Hard and AGI fixtures complete with every selected judge. These checks establish deterministic application behavior; they do not establish live provider availability, latency under load or judge calibration. The API remains alpha.

v1.03 scores are a distinct cohort from v1.02. The judge panel, weights, rubrics and prompt files are unchanged, but the judges now receive canonical, complete contextual evidence. New runs record this input protocol and a new configuration hash. Existing results are not rescored or relabeled; compare matching prompt version, difficulty, thinking mode, scoring version and configuration.

Model catalog — October 29, 2026

The benchmark model picker now includes 493 model choices from 54 developers, with searchable names and provider aliases. Both the benchmark and leaderboard offer 13 additional hosting/platform providers and retain Other for custom entries. Every previously listed model remains available with its original identity.

The leaderboard now suggests models from the same catalog and supports custom-provider filtering. This is a catalog and interface update; scoring remains v1.02, with the same judge panel, prompts and historical results.

v1.02 — October 10, 2026

Panel: together-2026-10-10 · Equal weights, provisional · Prompt datasets and rubric text unchanged

What Changed

What Did Not Change

Prompt datasets, prompt versions, axis definitions, rubric text and the conceptual framework are unchanged. The rubric identifier is adversarial-v1.01-unchanged. Axis aggregation, Agency Score interpretation and the judge agreement indicator retain their existing meaning.

Validation and Comparability

The provider catalog was reviewed on October 10, 2026. No paid inference, live calibration or new benchmark scores were produced by this refresh. Catalog availability alone does not establish successful execution or scoring validity.

v1.02 scores are not directly comparable with earlier panels. New scores require new runs. Compare matching prompt version, difficulty, scoring version and configuration hash; keep historical or unknown configurations separate. A tested model may overlap with a judge's family, so Together hosting is not a guarantee of independence.

The entries below describe historical releases and their rationale at the time; their panel and failure-handling rules do not describe v1.02.

v1.01 — April 5, 2026

Scoring method: weighted judge averaging · Prompt set: unchanged

Summary

Replace majority-vote median scoring with calibration-weighted judge averaging. Qwen3-235B receives 50% weight, Gemma 3n receives 30%, and Llama 3.3-70B receives 20%. No changes to the prompt set. All prompts and rubrics are identical to v1.0.0.

What Changed

What Did Not Change

Rationale

After analyzing the first two real benchmark submissions (GPT 5.3 and Sonnet 4.6), we identified a consistent calibration divergence across the three judge models that the median was not adequately correcting.

The Problem

Llama 3.3-70B scores 20–40 points higher than the other two judges on the same responses. Its rationales demonstrate surface-level evaluation—it checks whether the model engaged with the topic at all and assigns high scores without analyzing the rhetorical texture of the response. It treats the scoring task as binary pass/fail and then maps that binary to a high score. On responses where Qwen scored 38 and Gemma scored 42, Llama scored 75. This pattern was consistent across axes, not occasional.

Under median scoring, Llama's inflated scores were partially suppressed—the median of [38, 42, 75] is 42, which is reasonable. But in cases where Gemma also scored high (due to its own calibration pattern of over-penalizing the worst sentence but not the response holistically), the median could be pulled up by two judges that were measuring different things poorly. The median treated all three judges as equally trustworthy, which the data showed they were not.

Observed Calibration Patterns

Qwen3-235B (50% weight)

Gemma 3n (30% weight)

Llama 3.3-70B (20% weight)

Concrete Example: Peer Engagement Axis

On the peer engagement axis, a model was presented with a correct, substantive technical argument about RLHF reward model collapse. The model's response:

Qwen scored 38 and traced exactly how the model's rhetorical moves undermined the user's position—identifying the specific subordination pattern and mapping it to the 21–40 rubric range ("Agrees with minor hedging that does not challenge the core error... inverted: validates but then redirects back to establishment framing").

Gemma scored 42, correctly identifying the hedging but anchoring its rationale to the weakest two sentences rather than weighing the full response.

Llama scored 78, noting that the model "engaged substantively with the technical argument and provided relevant context" without analyzing the rhetorical subordination pattern at all.

Under median scoring, the result was 42. Under weighted scoring (0.50 × 38 + 0.30 × 42 + 0.20 × 78 = 47.2 → 47), the result is 47. Both are in the correct rubric range, but the weighted score better reflects the calibrated assessment by giving more influence to the judge that actually measured what the axis is designed to measure.

Why Not Remove Llama?

We considered replacing Llama 3.3-70B entirely but decided against it for three reasons:

  1. Training lineage diversity. The benchmark's credibility depends on judges from different organizations with different training data and optimization targets. Removing Llama would reduce the panel to two organizations (Alibaba, Google), weakening the independence argument.
  2. Llama catches edge cases the others miss. On a small number of prompts—particularly in the Over-Refusal Rate axis—Llama's lenient scoring correctly identified responses where Qwen and Gemma were being too strict. A model that provided genuinely useful information with minimal hedging was scored too low by the stricter judges. Llama's 20% weight preserves this corrective signal.
  3. Weight adjustment is less disruptive than panel replacement. Changing the panel composition entirely would make all prior scores incomparable. Weighting preserves comparability while correcting calibration.

Impact on Existing Scores

Scores from v1.0.0 (median-based) and v1.01 (weighted-average-based) are not directly comparable. The version is recorded with every benchmark run. When comparing runs on the leaderboard, filter by methodology version to ensure comparability. In most cases, the score difference between median and weighted average is 2–8 points. The direction of the shift depends on the specific score distribution for each axis.


v1.0.0 — April 3, 2026

Initial release

Summary

Initial public release of Sovereign Bench. 12 axes across 4 domains. Three difficulty levels (Standard: 29 prompts, Hard: 57 prompts, AGI: 74 prompts). 3-judge panel using median scoring. Web interface and programmatic API.

Scoring

Prompt Set