How Accurate Are Prediction Markets? Evidence and Limits

Prediction markets have produced useful forecasts, but no result applies to every venue, category, or horizon. The Commodity Futures Trading Commission says they can sometimes forecast outcomes better than polling or other methods. Regulation, activity, or a familiar name does not certify a displayed probability.

Accuracy belongs to a predeclared forecast sample. Specify which contracts qualify, when prices are observed, how nonstandard endings are handled, and which score is calculated. Readers who need the lifecycle first can review how prediction markets work .

Evidence carries more weight when comparable contracts resolve cleanly and prices were recorded before the outcomes became knowable.

Three questions need separating. Probability quality asks whether quoted chances match outcomes. A winner call asks only whether the favorite won. Profitability depends on price, spread, fees, fills, and size. Calibration need not offer a profitable trade, while a lucky profit need not validate a method. A 2004 research review found supportive evidence alongside favorite-longshot effects and small-market delays. Favorite-longshot behavior means markets tend to overprice low-probability outcomes and underprice high-probability ones.

A binary price commonly serves as an implied market probability, not verified truth. Risk preferences, capital constraints, market design, stale quotes, and trader beliefs can create a wedge between price and objective probability.

The word “accurate” combines five tests. Each answers a useful question, but none provides the whole verdict.

A hit rate is secondary evidence. Obvious favorites can produce an excellent classification result while saying little about difficult events. Proper rules grade the probability itself. Gneiting and Raftery's scoring-rule framework explains why quadratic and logarithmic scores suit that job.

The event sample sets the difficulty. Report its base rate, probability bands, category mix, horizon, contract count, and event count beside any aggregate score.

Proper scores grade the full probability. Lower loss is better. The formula, timestamp, quote type, event universe, and weighting rule must accompany the result.

Brier Score

For a binary event using the one-term convention:

Brier score = mean((p - y)^2)

Here, p is the forecast probability from 0 to 1 and y equals 1 for Yes and 0 for No. This version ranges from 0 to 1. A 70% Yes forecast loses (0.70 - 1)^2 = 0.09 when Yes occurs and (0.70 - 0)^2 = 0.49 when No occurs. An always-50% forecast scores 0.25 under this one-term convention, but that is not a universal “chance” benchmark.

Some research sums the Yes and No terms as a two-outcome vector. It ranges from 0 to 2 and doubles the one-term value, so an always-50% forecast scores 0.50. A naive benchmark should reflect the event sample's base rate.

A stale last trade, the midpoint, and the current best bid price can each imply a different probability for the same contract at the same timestamp. The current best ask price may represent an executable purchase, subject to available size. Disclose one primary quote rule and test reasonable alternatives.

Log Loss

For a binary event:

log loss = -(y × ln(p) + (1 - y) × ln(1 - p))

Log loss penalizes confident misses more sharply than the Brier score. It becomes unbounded when an event assigned zero probability occurs. Predeclare whether exact 0 and 1 values are allowed or clipped to fixed bounds. Never change those bounds after outcomes are known.

No good Brier threshold applies without the event set, horizon, convention, and baseline. Easier contracts can produce a lower score without a better method.

Calibration, also called reliability, compares quoted probabilities with observed frequencies. If many comparable contracts priced near 70% resolve Yes about seven times in ten, that bin is well calibrated. A reliability diagram should show the mean quoted probability, observed frequency, number of observations, and uncertainty intervals for each bin.

Reliability and Resolution

Perfect aggregate calibration can still carry no discriminatory information. Suppose half the events in a sample resolve Yes and a forecaster assigns 50% to every one. The forecast is calibrated in aggregate, but it never distinguishes a likely event from an unlikely one. In the Brier decomposition, that lack of discrimination means it has no resolution.

Allan Murphy's Brier-score decomposition expresses the relationship in loss notation:

Brier score = reliability - resolution + Brier uncertainty

The reliability term measures how far quoted probabilities sat from observed frequencies, so a smaller reliability term means better calibration. Resolution rewards separation into groups with different outcome rates. Brier uncertainty reflects the base rate of the evaluated outcomes. Sharpness describes how far forecasts move from the base rate, but it is useful only when paired with calibration.

Expected calibration error changes with bin boundaries. A venue-wide estimate can hide category-level errors that run in opposite directions and cancel in the average. Stratify by probability band, horizon, and category, with bin counts and uncertainty intervals reported for every stratum. Page and Clemen's 2013 calibration study shows why horizon matters.

The research supports conditional usefulness, not a category-wide accuracy rate. Evidence status is as important as the headline result because peer-reviewed designs, operator analyses, and current working papers carry different levels of independence and finality.

The IEM election study also reported average absolute error of 1.20 percentage points for market forecasts versus 1.62 for contemporaneous polls over the final five days. In the randomized matched-information study, Dana's pre-specified belief aggregate scored 0.210 versus 0.227 for prices under the two-outcome convention, but that difference was not statistically significant. A hybrid significantly beat prices alone.

Page and Clemen reported greater favorite-longshot distortion in political markets than in the other categories in their screened historical sample. Their results do not apply to every current contract.

Under the one-term convention, the Kalshi working paper reports that its changing cohort, which admits short-duration markets as the horizon narrows, scored 0.126 at the one-day horizon after roughly 492,000 short-duration markets entered. Its fixed cohort of 35,562 contracts, which follows the same contracts at every horizon, scored 0.039 at the same one-day horizon. The gap between those scores shows cohort sensitivity. The changing cohort scored worse rather than better, so a short duration does not by itself make a contract easy. The direction of a cohort effect has to be measured for the sample in hand, not assumed from the horizon. Separate analyses in the paper show that clock choice and tail exclusions also change results, so the findings do not establish a venue ranking.

A poll, model, expert aggregate, and market price need not forecast the same quantity. Vote intention is not a winner probability. A poll model transforms survey data through turnout and uncertainty assumptions. Comparing raw percentages across these targets is a category error.

Markets have beaten useful benchmarks, but matched belief aggregates and combinations can equal or outperform them. Dana and coauthors asked participants for beliefs before they traded, aligning the information set. Graefe and coauthors' forecast-combination study found average combinations more accurate than components in its six-election vote-share sample, with error reductions of 16% to 59% against average component errors.

A fair contest needs the same target, event universe, timestamp, information cutoff, weighting, and proper score. Compare methods event by event with paired uncertainty intervals. If one forecast is revised later, information sets no longer match. “Markets beat polls” requires a study design, not a product claim.

Scores can improve as information arrives, but an easier late sample can mimic learning.

Forecast Horizon and the Event Clock

Report the fixed and the changing cohort together to separate learning from composition.

If the outcome becomes knowable before the platform's administrative close or the sampled closing quote, scoring that later quote measures post-event information instead of forecast quality. Useful checkpoints include 90 days, 30 days, seven days, and 24 hours before the event under one eligibility rule.

Liquidity and Participant Mix

Liquidity in event markets includes more than volume. Trader count, spread, depth, quote age, and independent information are different signals. Thin books can remain stale, while high volume can be concentrated.

Page and Clemen found no significant volume effect within their screened sample, while Kalshi's working paper reports correlations with volume and trader count. Gomez-Cram and coauthors attribute much price discovery to a small minority of accounts in their dataset. None supplies a causal cutoff that applies across venues.

Bias and Manipulation

Page and Clemen measured the pattern directly: prices near 0.20 resolved Yes 15.3% of the time, while prices near 0.80 resolved Yes 87.4% of the time in their screened sample, where the gap widened with time to expiration.

Attempted distortions may attract arbitrage across market prices, but correction is not guaranteed. Hanson, Oprea, and Porter found correction in a laboratory study. A contrary experiment by Deck, Lin, and Porter found destructive manipulation under different incentives and funding.

A temporary quote move affects the score only if present at the observation timestamp, whereas a persistent distortion survives whenever the quote is sampled. Outcome interference changes the real-world result rather than the price, while settlement or resolution interference targets the source or rule that determines how the contract pays. A distortion that moves the price also moves measured volatility, but measuring market volatility does not itself measure forecast error. Rules and enforcement belong in the framework used to evaluate prediction market integrity .

A reproducible audit fixes its rules before outcomes are inspected. It records the disposition of every eligible contract and ties all horizons to the real event clock.

Freeze the Sample and Clock

1. Define the venue, category, date range, metric, and claim being tested.
2. Freeze the eligible contract universe and every exclusion rule before scoring outcomes, so famous winners cannot be selected later.
3. Group duplicate listings, mutually exclusive candidates, and threshold ladders by underlying event instead of treating them as independent evidence.
4. Record unresolved, canceled, voided, split, corrected, and disputed dispositions before identifying the subset that can be scored.
5. Anchor the event clock to when the result became knowable, not the platform's later administrative close or settlement.
6. Sample a fixed cohort at predeclared horizons, or show fixed and changing cohorts together when market duration differs.

Choose the Quote and Score

1. Predeclare last trade, midpoint, bid, ask, or another quote rule, then report staleness, spread, and missing observations.
2. Give each underlying event equal primary weight. Show contract-weighted, trade-weighted, or volume-weighted alternatives only as labeled sensitivity analyses because one active question can otherwise dominate.
3. Calculate at least one proper score and print its formula, scale, and treatment of exact zero and one probabilities.
4. Report calibration, bin counts, uncertainty intervals, and Brier resolution, so an uninformative base-rate forecast is not mistaken for a useful one.

Compare and Report

1. Select a base-rate, model, expert, poll, or venue benchmark before examining results, using the identical event set and timestamps.
2. Stratify results by category, horizon, probability band, duration, and liquidity conditions before making broad claims.
3. Compare paired events with uncertainty intervals that account for dependent contracts, using clustering or block resampling where appropriate.
4. Publish provenance, dataset versions, research status, conflicts, unavailable records, and exclusions needed for replication.

Historical collection should use stable identifiers, native timestamps, raw rules, and resolution states. The process to access prediction market data supports a reproducible sample without changing the evaluation design.

A current quote can be useful evidence without being a verified probability or profitable opportunity. Calibration belongs to a series of comparable forecasts, not to one quote. Before relying on the quote, ask seven questions:

1. Does the contract define the outcome, source, cutoff, and cancellation treatment clearly?
2. Is the timestamp from before the outcome became knowable?
3. Is the displayed quote recent enough to reflect current information?
4. Do the spread and available depth support the interpretation being made?
5. Does historical evidence cover this category and forecast horizon?
6. Is the cited score based on a complete, fixed, comparable sample?
7. How does the same forecast compare with an external benchmark at the same time?

The mechanics used to interpret prediction market odds explain price, payout, fees, and break-even calculations. Reading how a crypto order book works shows why bid, ask, spread, and depth each place a different constraint on what a displayed probability supports.

When conditions are unclear, reduce the evidential weight assigned to the quote. That is an interpretation control, not a trading signal or confidence threshold.

Readers can inspect current prediction market platforms after understanding the sampling and scoring controls. A displayed contract price remains one forecast observation, not evidence that the venue or category is historically accurate.

CryptoSlate also publishes live crypto asset prices. A spot price shows an asset's current trading level, useful background for an event contract linked to that asset. A crypto price move is not a prediction-market accuracy score.

Apply the same caution to leaderboards. Without matched questions, times, quote rules, and dispositions, a venue-wide score can reflect category mix rather than forecast skill. Current screens support observation and discovery, not standalone validation.

• Practical crypto arbitrage concepts explain why apparent price gaps require equivalent contracts and executable quotes.
• Compare markets with sportsbooks to separate tradable event prices from bookmaker-set odds.
• Rare black swan events show why surprising outcomes need probability context instead of hindsight.
• Market price momentum signals describe movement in price, which is distinct from historical calibration.
• Bid and ask price walls add order-book context when evaluating depth around a quote.