Open research

Do people know how much they know?

Who's Bluffing is a game and an open study. Every completed round is an anonymous data point about calibration: whether people who say they are 80% sure are right 80% of the time, and whether a little feedback every day changes it.

The question

Decades of research say that people are overconfident: they feel surer than their answers turn out to be. Who's Bluffing measures this at scale, every day, with questions whose answers have a public source, and adds a second question: does playing regularly, with feedback after every answer, make people better calibrated?

Why honest confidence scores best

Each answer in a round scores 100 − 400 × (c − y)² points, where c is the confidence you chose (from 0.5 to 1) and y is 1 if you were right and 0 if not. That is the quadratic score Brier (1950) proposed for weather forecasts, rescaled to points. It is a strictly proper scoring rule: your expected points are highest when the confidence you state is your real chance of being right, so bluffing up or hedging down can only cost you on average (Gneiting and Raftery, 2007). A 50% answer scores 0 whether it is right or wrong, because a coin flip says nothing; 100% wins 100 points when right and loses 300 when wrong, so certainty pays only when you really are certain.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

Hypotheses

• H1 · People are overconfident. Across players, average confidence is higher than the share of answers that are right; and the harder a question (the fewer people get it right), the more overconfident people are about it.
• H2 · Format and difficulty. In the 5-minute full assessment, overconfidence is larger for 90% ranges than for two-choice questions; in rounds, overconfidence is reported for each level of difficulty.
• H3 · Practice helps. Among players with at least 10 completed rounds, overconfidence falls and points rise over their first 20 rounds. The first rounds of players who keep playing are compared with those of players who stop early; if they differ, the effect is reported as applying to people who keep playing. A null result is reported as a null.

Player counts

Only Total Players is public. It counts distinct anonymous IDs with a completed round or assessment, or a daily chat answer. DAU, MAU (rolling 30 days), and annual active users (rolling 365 days) are private. These count IDs, so one person can count more than once if they use different devices or platforms.

Open data

Anonymous row-level data, with every exclusion flag kept rather than deleted, will be released on OSF, Hugging Face and Kaggle with a data card describing how it was collected, its sample bias and what it should not be used for. Classroom sessions are pooled without class codes. The code for every analysis is in the analysis folder.

How to cite

Until the first dataset has its own identifier, please cite the project:

Who's Bluffing (2026). Who's Bluffing? A daily calibration game and open dataset.
https://whosbluffing.com.