What this is
Anyone can edit Wikipedia, and most people who do are trying to help. A few are not. A small group of volunteers reads the incoming edits looking for damage — and there are far more edits than there are people to read them.
Bellwether watches those edits as they arrive and guesses which ones are likely to be undone. The guesses go in a queue, worst first, so a volunteer’s attention lands where it is most likely to be needed.
Why it exists
The interesting part is not the guessing. Plenty of systems guess.
The problem with a system like this is that it quietly stops working. The people it watches change how they behave, and the model keeps answering with the same confidence it always had. Almost every deployed model is tested once, before it goes live, and then trusted forever after. Nobody checks whether it is still right. Nobody decides in advance what “no longer right” would even look like.
So this project asks a different question: can a system notice it is getting worse, fix itself, and be trusted to have told the truth about it?
How it keeps itself honest
The whole design is built around one problem: it is very easy to fool yourself about how well something works, especially when you built it. Four rules do the work.
It commits before it finds out
Every guess is written down the moment an edit appears — hours or days before anyone knows what happened to it. The record cannot be edited or deleted afterwards, not even by the people who run it. The database itself refuses.
It waits for the answer
An edit that has not been undone yet has not survived — nobody has looked at it. Guesses only count towards a score once enough time has passed that the outcome is genuinely settled. A week, in practice.
The rules were written first
Before the first model existed, the exact conditions for replacing it were written down and published: how much better a replacement must be, how much evidence is required, how long it must be watched. Those numbers cannot be adjusted afterwards to make a result look better.
It reports the bad news too
Every rejected replacement is published alongside the accepted ones. So is the least flattering thing the evaluation found, and the comparison against Wikimedia’s own model — which is expected to win, and that expectation was recorded in advance.
What runs without anyone watching
Every ten minutes it collects new edits and scores them. Every few hours it grades its old guesses against what actually happened. Every day it checks whether it has got worse, and if it has, it trains a replacement, runs that replacement silently alongside the real one for a week, and then either promotes it or refuses to — by the rules written down beforehand. If a promotion turns out badly, it puts the old one back on its own.
Nobody approves any of that. The record of what it decided, and why, is on the timeline.
What the pages show you
| Status | What it has collected, and whether the machinery is still running. If a job has stopped, this says so rather than showing you a stale number. |
| Performance | How accurate it has actually been — with the sample size next to every figure, because a number without one cannot be judged. |
| Timeline | Every decision it made about replacing its own model, including the refusals, with the evidence. |
| Queue | The live triage list. Sign-in required — accounts are issued by hand. |
How to read the numbers
Every figure on the Performance page is three things: a score, something to compare it against, and a range. The score on its own is the least useful of the three.
1. A score alone means nothing
You will see numbers like 0.22. That is not a percentage and it is not a grade. Bad edits are rare — about nine in a hundred here — and finding rare things is hard, so even a good ranker scores low on this scale. Ignore it until you have something to hold it against.
2. The gap is the answer
Every score sits beside a comparison, and the difference between them is the only part that answers a question. Positive means this system did better than the thing it was compared with. Negative means it did worse. That one number is what to read.
3. The bracket says whether to believe it
Each gap carries a range in brackets. If the whole range sits on one side of zero, the result is real. If the range crosses zero, the honest answer is too close to call — however good the middle number looks. This is the step most people skip.
Worked example
Here are the two comparisons as they stood on 26 August 2026, read the way described above. The live versions are on the Performance page.
| Compared against | Gap | Range | How to read it |
|---|---|---|---|
| A simple rule — “suspect anyone not logged in” | +0.052 | +0.047 to +0.057 | The whole range is above zero, so this is real. The model earns its place: it beats the obvious shortcut, and not by luck. |
| Wikipedia’s own model, on the same edits | −0.014 | −0.092 to +0.079 | The range crosses zero, so this is a draw. Theirs is slightly ahead in the middle, but not by enough to claim anything either way yet. |
The second row is the one the project cares about most, and it was predicted in writing before any model existed here: Wikipedia’s production model was expected to win. It is published whichever way it falls, and right now it falls on “level, on the evidence so far”.
So is it any good?
On the evidence above: yes, with a limit. It clearly beats the simple shortcut, which is the bar that decides whether the model is worth running at all. Against a professionally maintained model it is currently level rather than ahead — and the sample is still small enough that this could move in either direction.
Two habits will keep you from misreading anything here. Check the sample size beside a figure before trusting it, and check the range before repeating it. A number measured on a few hundred events is a hint, not a finding.
One thing worth knowing before you read a number
This system treats “the edit was undone” as meaning “the edit was bad”. That is not quite true. Good edits get undone by mistake, and bad ones survive because nobody noticed. Every figure here rests on that substitution.
Rather than assume it away, the project is measuring it: reviewers judge edits without being shown the model’s opinion, and their verdicts are compared against what actually happened. That study is running now and does not have an answer yet. When it does, the answer will be published whichever way it comes out.
Honest limits
- It runs entirely on free infrastructure, so the server sleeps when idle and the first page load can take up to a minute.
- It reads a fixed sample of edits rather than all of them, and corrects for that sample in every figure.
- It never edits Wikipedia. It only reads, ranks, and reports.
- It is one person’s project, built in public, and everything above is verifiable from the source.