Routers promise to carry your request to the model you paid for. Sometimes they don't. We buy retail access, watch the swarm continuously, and publish what actually arrives — method first, harness open, funded by nobody on this page.
Fifteen gateways now sit between you and the models. They quote a price, name a model, take your traffic. What happens inside is unobserved — and every published comparison of them is written by someone selling one.
Each cell is one router, measured continuously against a direct-from-provider control. Fidelity is the share of sampled requests where what arrived matched what was charged for. Ranges are 95% confidence intervals.
Illustrative structure pending the first measurement cycle. No router has been contacted, ranked, or notified. These are layout placeholders, not findings.
The methodology goes live before any result exists. It is the thing to attack if you want to attack our numbers — which is precisely why it is written first.
No press tier, no free credits, no partner rate, no negotiated access. Receipts kept. Dependence on goodwill would end the ability to publish a hostile finding.
Matched measurements straight to the provider, in the same window, so a provider's fault is never misattributed to the router carrying it.
Tokeniser behaviour, canary prompts, logprob fingerprints, timing signatures. No single probe can trigger a published claim. Corroboration or silence.
Committed to the public repo before the measurements they will judge, and biased toward under-reporting. A false accusation would end this project, so we build against it.
Measured parties learn what we found when you do — never before — and their response sits on the same page as our finding.
Absence of detected substitution is not proof of absence. We publish what we cannot see alongside what we can.
Methodology v0.1 — §LimitationsPrice and latency are partly covered elsewhere. Fidelity is covered by nobody — which is why it goes first.
Was the model you were billed for the model that answered — unquantised, undistilled, unsubstituted, not silently swapped mid-incident?
What a successful completion actually costs once markup, retries, failover, cache accounting and silent rerouting are reconciled against the invoice.
Availability, error class, failover latency, stream integrity — with dedicated write-ups for provider incidents, the moment routing value is claimed and never checked.
For routers that choose models on your behalf: how much quality survives per unit of cost saved, measured against the premium baseline.
So ours is written down, specific, and deliberately expensive to abandon quietly. Every constraint below is published, dated, and versioned.
One day a well-funded gateway will offer real money, framed as supporting independent research, and it will arrive when money is tight. Refusing it is not a sacrifice made for the business. It is the business.
Neutrality charter — clause 0Recomputing our aggregates from our raw measurements should return our numbers. If it doesn't, we would rather hear it from you than read it somewhere else.
Probes, control arm, thresholds, ingest — the whole measurement system, public from the first commit.
Append-only, every row stamped with methodology version and prompt-set version.
Any router, any metric, any window. Free to read, no account, rate-limited only.
Public history of every measurement cycle. A gap in the record is explainable, or it is a bug.