What Brier scores tell you about prediction-market forecasts

A forecast like “70% chance of rain” is never simply right or wrong on a single day. To judge a forecaster — or a market — you need a way to score probabilities against outcomes over many events. The Brier score is one of the oldest and simplest tools for that job.

The definition

For a set of N yes/no events, each with a forecast probability f (between 0 and 1) and an outcome o (1 if it happened, 0 if it didn’t):

Brier score = (1/N) × Σ (f − o)²

It’s the mean squared error between your probabilities and what actually happened.

• 0 is a perfect score (every forecast was 100% confident and correct).
• 1 is the worst possible (every forecast was 100% confident and wrong).
• Lower is better.

A useful reference point: if you always forecast 50%, every term is (0.5 − o)² = 0.25, so your Brier score is exactly 0.25 no matter what happens. A forecaster who can’t beat 0.25 on binary events isn’t adding information beyond a coin flip.

Applying it to prediction markets

On many prediction markets, a contract pays a fixed amount if an event happens. Ignoring fees and other frictions, a contract priced at 70¢ on a $1 payout is often read as an implied probability of about 70%. That lets you treat a market’s price as a forecast and score it like any other.

A toy worked example (hypothetical numbers)

Brier score = (0.01 + 0.49 + 0.04 + 0.16) / 4 = 0.175.

Compared with the always-50% baseline of 0.25, you can compute a Brier skill score:

BSS = 1 − (Brier / Brier_reference) = 1 − (0.175 / 0.25) = 0.30

A positive skill score means the forecasts beat the reference; zero means no better; negative means worse.

What the score can — and can’t — tell you

It rewards honesty. The Brier score is a proper scoring rule: your expected score is best when you report the probability you actually believe. You can’t game it by shading forecasts toward 0 or 1.

It mixes two qualities. A classic decomposition (Murphy, 1973) splits the Brier score into reliability (are your 70% calls right about 70% of the time?), resolution (do your forecasts vary meaningfully between events that do and don’t happen?), and uncertainty (how inherently unpredictable the events were). Two forecasters with the same score can get there in very different ways.

It depends on the question set. Scores aren’t comparable across different sets of events. Forecasting near-certain events produces low scores easily; forecasting genuine toss-ups doesn’t.

Small samples are noisy. A handful of events — like the four in the toy example — says very little. Judge forecasters over many resolved questions.

Market prices aren’t pure probabilities. Fees, spreads, liquidity, the time value of locked-up capital, and known behavioral effects such as the favorite–longshot bias can push prices away from true beliefs. When you sample the price (a week out vs. an hour before resolution) also changes the score dramatically.

Takeaways

• The Brier score is the mean squared error of probability forecasts; lower is better and 0.25 is the coin-flip baseline for binary events.
• It’s a fair way to compare forecasters on the same questions, and skill scores make the comparison against a baseline explicit.
• Treat prediction-market prices as noisy forecasts, be explicit about when you sample them, and don’t over-read small samples.

Further reading

• Brier, G. W. (1950). “Verification of forecasts expressed in terms of probability.” Monthly Weather Review, 78(1).
• Murphy, A. H. (1973). “A new vector partition of the probability score.” Journal of Applied Meteorology, 12(4).