Confidence Calibration Exercises

People tend to be ludicrously overconfident about things. Let's fix this!

From the wikipedia article on the overconfidence bias:

"If human confidence had perfect calibration, judgments with 100% confidence would be correct 100% of the time, 90% confidence correct 90% of the time, and so on for the other levels of confidence. By contrast, the key finding is that confidence exceeds accuracy so long as the subject is answering hard questions about an unfamiliar topic. For example, in a spelling task, subjects were correct about 80% of the time when they were "100% certain." Put another way, the error rate was 20% when subjects expected it to be 0%. In a series where subjects made true-or-false responses to general knowledge statements, they were overconfident at all levels. When they were 100% certain of their answer to a question, they were wrong 20% of the time."

Calibrating your confidence is an important rationality skill - a building block for many other skills you may want to develop. This Tuesday, we'll be doing a series of exercises. A moderator will ask trivia questions that have a numerical answer. Participants will write down a 50% and 98% confidence internal (i.e. a numerical range that you are 50% confident the answer lies within). Over the course of the evening we'll see how often our confidence was calibrated correctly, and hopefully adjust accordingly.