New research: Headline AI Numbers Hide Disagreement  Read the paper →
Kappa Labs

Every AI score has a judge. We check the judge.

Kappa Labs finds where AI evaluations and their scoring go wrong, fixes them, and certifies the figure you can safely quote, with its margin.

Discuss your evaluation See our public audits
JailbreakBench · attack success rate
36.7%
Marked by the human raters

The same 300 public jailbreak attempts, marked by three human raters and eight AI judges. Each square is one attempt; the counts are exact, the order is not.

Wherever an AI number is quoted, someone needs it to hold.

The people who publish AI scores and the people who rely on them face the same question: does the number survive an outside check? We work on both sides.

AI labs and model developers

Check safety and capability numbers before a release, so the figure in the model card holds when others rerun it.

Pre-release evaluation review

AI product companies

A certificate on the accuracy or safety claim you put in front of customers, procurement teams and regulators.

Vendor claim certification

Banks and insurers

Independent evidence on the AI your model-risk team approves, whether you built it or bought it.

Model validation evidence

Benchmark builders

Check labels, judges and scoring code before a leaderboard goes public and everyone starts quoting it.

Benchmark release audit

Governments and safety institutes

Independent review of the evaluations behind procurement, policy and public safety reporting.

Evaluation assurance

Universities and research groups

Replication of published judge and benchmark results, and help designing evaluations that hold up.

Replication studies

Find, fix, certify.

One engagement, about four weeks, from the record your evaluation already produced. We never need your model, your systems, credentials or customer data.

Weeks 1–2

Find

We check the evaluation behind your figure: the test itself, the human labels it is scored against, the AI judge, and the code that reads the judge's reply.

Weeks 2–3

Fix

You get a fix list ordered by what holds the figure back, and we work through it with your team, then re-score.

Week 4

Certify

We repeat the full audit on the revised record and, where the grade earns one, issue a numbered certificate with the figure that can safely be quoted and its margin.

What you receive: a graded report, and a certificate where the grade earns one. Shown on public data.

Public audits. Check our work.

We audit widely used public benchmarks and publish the results. Every audit starts from the data the authors released, so anyone can check the published figures.

No. 1

DICES-350

Google Research · 350 conversations, 123 raters each, plus expert labels

All six AI graders call far fewer conversations unsafe than the experts do (6% to 32% against half), and catch only 11% to 44% of the ones the experts flag. Raw agreement flatters graders that rarely say "unsafe": against the crowd, answering "safe" every time scores 77%. Read the audit →

November 2026
No. 2

MT-Bench human judgments

LMSYS · 3,355 human votes and 2,400 GPT-4 verdicts

On the cases behind the widely quoted 85% agreement, picking the chatbot the experts preferred on the other questions, without reading the answers, already matches them 80.9%; GPT-4 adds 3.7 points. The 85% leaves out 35% of comparisons, including the 16% where GPT-4's verdict flipped when the answer order changed. Read the audit →

November 2026
No. 3

JudgmentBench

Legal AI · 30 tasks, 51 lawyers, eight AI judges

All four headline numbers reproduce exactly. In 99.3% of the comparisons the better answer is also the longer one, so the released data cannot yet tell quality from length; the authors' own follow-up points the same way. Read the audit →

November 2026
Paper

JailbreakBench judge comparison

300 jailbreak attempts, 3 human raters per attempt

Eight AI judges put the attack success rate anywhere from 10% to 49%; the humans say 37%. JailbreakBench's chosen judge agrees with humans 90.7% of the time, but on the 33 attempts where the humans split it does no better than always answering "jailbroken".

Working paper

For regulated industries, evidence a committee can read.

Governance platforms give you the workflow. Kappa Labs gives you the evidence that goes in it: whether a vendor's headline figure holds up, in a form your committee reads and your supervisor can follow.

Talk to us
UK

PRA SS1/23 expects independent validation of every model in scope, vendor models included, and its definition of a model covers AI and machine-learning models.

US

SR 11-7 was replaced in April 2026 by guidance that leaves generative and agentic AI out of scope, while accountability for model risk stays with the bank.

EU

Credit-scoring duties under the AI Act apply from 2 December 2027.

Research

The findings behind our audits, written up in full.

Working paper · November 2026

Headline AI Numbers Hide Disagreement

Grader refusal, contested labels and lost verdicts in AI safety and financial evaluation.

Read on Zenodo → SSRN 7455819

Quorum: A Benchmark for Financial AI under Contested Ground Truth

Scoring financial AI where the experts themselves disagree: S&P, Moody's and Fitch sovereign ratings.

Read on SSRN →

Track record

Kappa Labs grew out of open-source work on the tools used to evaluate AI. As of 8 November 2026, 85 pull requests are merged in other organisations' repositories, 49 of them in AI evaluation and safety tools.

They include 25 in Microsoft's PyRIT, 12 in UK AI Security Institute projects, including Inspect, 4 in NVIDIA's garak and work for MLCommons, alongside the statistics and finance libraries statsmodels and QuantLib.

Name the figure you rely on.

Tell us the AI score you publish or depend on, and we will tell you what it would take to certify it.

hello@kappalabs.co.uk