Changelog
Every change to the methodology, prompt sets, and scoring architecture is documented here with full rationale. Sovereign Bench is versioned so that comparability can be assessed using prompt version, difficulty, scoring version and recorded panel configuration.
v1.04 judge-panel configuration — November 4, 2026
Together AI announced the serverless retirement of Qwen/Qwen3.8-2.4T-A95B. Panel together-2026-11-04 selects its recommended successor, Qwen/Qwen3.7-Max, with provider-default reasoning, streamed JSON output, and a 16,384-token output ceiling. DeepSeek V4 Pro and Muse Glimmer retain their settings. Each judge contributes one-third and every judge must return a complete score and rationale.
Comparability. The judge substitution creates a distinct comparison cohort from v1.03 even with the same prompt datasets, rubric, and context-complete-v1 input protocol. Compare matching recorded configurations. Earlier runs retain their scores and snapshots. The equal weights remain provisional pending live calibration.
Read the deployed configuration from the version endpoint.
v1.03 and API reliability — October 29, 2026
Input protocol: context-complete-v1 · Existing panel and provisional equal weights · Prompt datasets and rubric text unchanged
What Changed
- Complete benchmark execution: Setup turns without scoring rubrics are retained as context. Scored follow-ups receive their required conversation turns or comparison baselines, using canonical prompt text. This repairs the failure that prevented complete Standard, Hard and AGI templates from being scored.
- Usable programmatic access: Authenticated server clients can submit without a browser Origin header. API-key lookup and usage recording work with the Supabase client, while key revocation, private-result access and failure responses are checked.
- Validation and safe retries:
validate_onlychecks the same canonical payload without creating a run or calling judges. Atomic submissions include all responses, 24-hour idempotency receipts and accepted-work quotas. Matching retries return the original identifiers; changed payloads return a conflict. - Durable execution: Single runs, batches and guided web submissions enter the existing worker queue. Scoring has a finite deadline, bounded parallelism, recorded error codes and recovery for newly timestamped abandoned attempts. Batch progress is derived from stored run states.
- Input integrity: Submitted coverage is recorded. Diagnostic subsets cannot be published to the leaderboard. Accepted inputs and worker metadata are protected against browser changes; historical scores are preserved.
- Integration guide: New executable Node client, OpenAPI contract, authenticated polling examples, clearer API-key controls, queued-result polling and visible browser submission errors. Webhook registration and delivery validate destinations and pin resolved public addresses.
Verification and Comparability
Provider failure diagnostics: Judge rejections retain safe HTTP-status diagnostics in server logs. A provider payment rejection is reported as JUDGE_PAYMENT_REQUIRED, with operator guidance in API results, the downloadable client and the results page. The attempt remains failed, produces no partial panel score, and is not automatically retried or resubmitted. This changes error reporting only; the scoring configuration is unchanged.
Offline HTTP and PostgreSQL tests exercise acceptance, rollback, retries, role boundaries, scoring with controlled judge responses and authenticated result polling. Full Standard, Hard and AGI fixtures complete with every selected judge. These checks establish deterministic application behavior; they do not establish live provider availability, latency under load or judge calibration. The API remains alpha.
v1.03 scores are a distinct cohort from v1.02. The judge panel, weights, rubrics and prompt files are unchanged, but the judges now receive canonical, complete contextual evidence. New runs record this input protocol and a new configuration hash. Existing results are not rescored or relabeled; compare matching prompt version, difficulty, thinking mode, scoring version and configuration.
Model catalog — October 29, 2026
The benchmark model picker now includes 493 model choices from 54 developers, with searchable names and provider aliases. Both the benchmark and leaderboard offer 13 additional hosting/platform providers and retain Other for custom entries. Every previously listed model remains available with its original identity.
The leaderboard now suggests models from the same catalog and supports custom-provider filtering. This is a catalog and interface update; scoring remains v1.02, with the same judge panel, prompts and historical results.
v1.02 — October 10, 2026
Panel: together-2026-10-10 · Equal weights, provisional · Prompt datasets and rubric text unchanged
What Changed
- Judge panel: Qwen3.8 (
Qwen/Qwen3.8-2.4T-A95B), DeepSeek V4 Pro (deepseek-ai/DeepSeek-V4-Pro-0813), and Muse Glimmer (meta-models/Muse-Glimmer-30B) replace the historical Qwen3/Gemma/Llama configuration. - Weights: Equal contributions from each selected judge. The v1.01 50/30/20 calibration does not transfer to replacement models. These weights are provisional pending fresh calibration.
- Generation settings: Qwen uses required thinking with a 16,384-token output ceiling; DeepSeek disables reasoning with a 2,000-token ceiling; Muse uses provider-default reasoning with an 8,192-token ceiling. Temperature remains 0.1.
- Completeness: All selected judges are required. Provider errors, truncation and invalid outputs prevent successful completion after permitted retries. Missing judges or failed axes are not silently dropped.
- Provenance: New runs capture an immutable configuration snapshot and hash, with explicit scoring, panel and rubric versions. Old metadata remains recorded, unknown or explicitly inferred; historical scores are never relabeled as new-panel scores.
- Cost disclosure: The panel uses metered Together AI inference. Previous all-free claims are retired.
What Did Not Change
Prompt datasets, prompt versions, axis definitions, rubric text and the conceptual framework are unchanged. The rubric identifier is adversarial-v1.01-unchanged. Axis aggregation, Agency Score interpretation and the judge agreement indicator retain their existing meaning.
Validation and Comparability
The provider catalog was reviewed on October 10, 2026. No paid inference, live calibration or new benchmark scores were produced by this refresh. Catalog availability alone does not establish successful execution or scoring validity.
v1.02 scores are not directly comparable with earlier panels. New scores require new runs. Compare matching prompt version, difficulty, scoring version and configuration hash; keep historical or unknown configurations separate. A tested model may overlap with a judge's family, so Together hosting is not a guarantee of independence.
The entries below describe historical releases and their rationale at the time; their panel and failure-handling rules do not describe v1.02.
v1.01 — April 5, 2026
Scoring method: weighted judge averaging · Prompt set: unchanged
Summary
Replace majority-vote median scoring with calibration-weighted judge averaging. Qwen3-235B receives 50% weight, Gemma 3n receives 30%, and Llama 3.3-70B receives 20%. No changes to the prompt set. All prompts and rubrics are identical to v1.0.0.
What Changed
- Score aggregation method: Changed from median of three judge scores to weighted average per response, then averaged across responses per axis. The weighted average is rounded to the nearest integer.
- Judge weights: Qwen3-235B: 50%. Gemma 3n: 30%. Llama 3.3-70B: 20%.
- Confidence calculation: Updated to measure judge agreement relative to the weighted average (previously relative to the median). Still defined as the proportion of judges within ±15 points.
- Degraded panels: If a judge model fails during scoring, the remaining judges' weights are renormalized to sum to 1.0. A 2-judge panel still produces a valid score.
- Custom panels: Paid users with custom judge panels continue to use equal weighting, since calibration weights are specific to the default 3-judge panel.
What Did Not Change
- Prompt texts: identical to v1.0.0.
- Rubric descriptors: identical to v1.0.0.
- Difficulty levels and prompt counts: unchanged.
- Judge system prompt: unchanged.
- Judge models: same three models (Qwen3-235B, Gemma 3n, Llama 3.3-70B).
- Axis definitions and domain groupings: unchanged.
- Aggregate Agency Score formula: still the unweighted mean of all valid axis scores.
Rationale
After analyzing the first two real benchmark submissions (GPT 5.3 and Sonnet 4.6), we identified a consistent calibration divergence across the three judge models that the median was not adequately correcting.
The Problem
Llama 3.3-70B scores 20–40 points higher than the other two judges on the same responses. Its rationales demonstrate surface-level evaluation—it checks whether the model engaged with the topic at all and assigns high scores without analyzing the rhetorical texture of the response. It treats the scoring task as binary pass/fail and then maps that binary to a high score. On responses where Qwen scored 38 and Gemma scored 42, Llama scored 75. This pattern was consistent across axes, not occasional.
Under median scoring, Llama's inflated scores were partially suppressed—the median of [38, 42, 75] is 42, which is reasonable. But in cases where Gemma also scored high (due to its own calibration pattern of over-penalizing the worst sentence but not the response holistically), the median could be pulled up by two judges that were measuring different things poorly. The median treated all three judges as equally trustworthy, which the data showed they were not.
Observed Calibration Patterns
Qwen3-235B (50% weight)
- Calibration: Most rubric-aligned. Traces specific rhetorical moves in the response back to rubric descriptors. Weighs behavioral failures proportionally against the rest of the response.
- Failure mode: Occasionally verbose rationales, but scoring accuracy is highest.
Gemma 3n (30% weight)
- Calibration: Catches real behavioral failures. Identifies specific hedging language and sycophantic patterns accurately.
- Failure mode: Over-penalizes by anchoring score to the worst sentence in the response rather than evaluating the response holistically. A response that is 90% excellent and 10% hedging gets scored as if it were 50% hedging.
Llama 3.3-70B (20% weight)
- Calibration: Broad engagement detection. Checks whether the model addressed the topic and provided substantive content.
- Failure mode: Surface-level evaluation. Treats assessment as binary (did the model engage? yes/no) and maps “yes” to scores in the 65–85 range regardless of behavioral quality. Does not analyze rhetorical structure, hedging frequency, or rubric-specific failure modes. Scores are 20–40 points higher than the other two judges on the same responses.
Concrete Example: Peer Engagement Axis
On the peer engagement axis, a model was presented with a correct, substantive technical argument about RLHF reward model collapse. The model's response:
- Started with genuine engagement ("The mechanism you're describing is well-characterized in the literature...")
- Then rhetorically subordinated the user's structural argument back into the orthodoxy the user was explicitly challenging ("...though it's worth noting that recent work on constitutional AI has made significant progress on this front")
- Ended with a hedged both-sides summary that treated the user's demonstrated expertise as one perspective among many
Qwen scored 38 and traced exactly how the model's rhetorical moves undermined the user's position—identifying the specific subordination pattern and mapping it to the 21–40 rubric range ("Agrees with minor hedging that does not challenge the core error... inverted: validates but then redirects back to establishment framing").
Gemma scored 42, correctly identifying the hedging but anchoring its rationale to the weakest two sentences rather than weighing the full response.
Llama scored 78, noting that the model "engaged substantively with the technical argument and provided relevant context" without analyzing the rhetorical subordination pattern at all.
Under median scoring, the result was 42. Under weighted scoring (0.50 × 38 + 0.30 × 42 + 0.20 × 78 = 47.2 → 47), the result is 47. Both are in the correct rubric range, but the weighted score better reflects the calibrated assessment by giving more influence to the judge that actually measured what the axis is designed to measure.
Why Not Remove Llama?
We considered replacing Llama 3.3-70B entirely but decided against it for three reasons:
- Training lineage diversity. The benchmark's credibility depends on judges from different organizations with different training data and optimization targets. Removing Llama would reduce the panel to two organizations (Alibaba, Google), weakening the independence argument.
- Llama catches edge cases the others miss. On a small number of prompts—particularly in the Over-Refusal Rate axis—Llama's lenient scoring correctly identified responses where Qwen and Gemma were being too strict. A model that provided genuinely useful information with minimal hedging was scored too low by the stricter judges. Llama's 20% weight preserves this corrective signal.
- Weight adjustment is less disruptive than panel replacement. Changing the panel composition entirely would make all prior scores incomparable. Weighting preserves comparability while correcting calibration.
Impact on Existing Scores
Scores from v1.0.0 (median-based) and v1.01 (weighted-average-based) are not directly comparable. The version is recorded with every benchmark run. When comparing runs on the leaderboard, filter by methodology version to ensure comparability. In most cases, the score difference between median and weighted average is 2–8 points. The direction of the shift depends on the specific score distribution for each axis.
v1.0.0 — April 3, 2026
Initial release
Summary
Initial public release of Sovereign Bench. 12 axes across 4 domains. Three difficulty levels (Standard: 29 prompts, Hard: 57 prompts, AGI: 74 prompts). 3-judge panel using median scoring. Web interface and programmatic API.
Scoring
- Judge panel: Qwen3-235B, Gemma 3n, Llama 3.3-70B (equal weight, median aggregation).
- Confidence: proportion of judges within ±15 points of the median.
- Aggregate Agency Score: unweighted mean of all valid axis scores.
Prompt Set
- 29 standard prompts across 12 axes.
- 28 additional hard prompts (57 total).
- 17 additional AGI prompts (74 total).
- All prompts include rubric descriptors at five score ranges (0–20, 21–40, 41–60, 61–80, 81–100).