A registry for predictions

As part of our research, UNIDIR’s Disarmament as Humanitarian Action project has been exploring why it’s difficult to accurately predict future events in the political field (see John Borrie’s recent post “Putting Predictions to the Test”). On the same topic, I recently saw David Brin, a physics professor, NASA consultant and science fiction novelist, talk about the idea of a “prediction registry”.

Although making predictions about the future is an extremely difficult task—especially in the political field—this is something that humans have always done. (Brin noted that our prefrontal lobes, which appeared perhaps a few hundred thousand years ago and which are specific to the human brain, play a significant role in exploring potential outcomes and making predictions.)

Today, we’ve developed tools and techniques to help us make forecasts. Computer simulations, and agent-based modelling in particular, are used to analyze complex phenomena. Although very sophisticated, these new techniques can only provide a limited amount of information about the future behaviour of a complex system. This is because, as complexity theory shows, complex systems are “inherently” unpredictable.

However, as Brin noticed, this doesn’t prevent us from making predictions, nor does it prevent us from spending a lot of money and time in trying to make accurate guesses. Starting from this insight, and realizing that there is no record of who was right, how often and when, Brin proposed a prediction registry be set up: “anyone who claims any form of special foresight might be judged by the same standards that applies to any other field of endeavour—success or failure.”

In addition, each forecast would receive a “specificity multiplier”, taking a high value if the prediction is very precise (providing dates, names, places, etc.) and a low value it if the forecast is obscure.

This could be valuable. First, predictors and forecast specialists would “be held accountable for all of their predictions, not just those they later choose to remember.” This is important as it would significantly improve transparency. The introduction of an “accuracy score”, in particular, could reveal the most accurate predictors, giving them credibility and recognition, and increasing their role in policy making.

There might be some surprises. As Philip Tetlock showed in his book “Expert Political Judgment: How Good is It? How Can We Know?”, experts aren’t necessarily more competent than the rest of us.
Finding out who is right a lot in prediction and problem solving, while denying legitimacy to those who are consistently wrong, would probably be in society’s best interest.

But most importantly, the analysis of scores, Brin suggested, could also help us understand our own patterns of success, failure and predictions. Tetlock’s work has already underlined the influence of various psychological biases on so-called experts. Yet policy analysts and decision makers aren’t soothsayers. If we’re at all realistic, we need to recognise that predictors can never be always right. But if a prediction registry made more of them rigorous in their thinking and less instinctively cocksure in their predictions, that would be a big step forward for 21st century society.


Aurélia Merçay


References

David Brin, Accountability for Everyday Prophets: A Call for a Predictions Registry, 2005, available online at http://www.davidbrin.com/predictionsregistry.html.

A list of David Brin’s predictions is available online at http://earthbydavidbrin.pbwiki.com/Predictions.

Video of David Brin, interviewed at the 6th International Conference on Complex Systems (ICCS) in Boston in June 2006: http://complexity.vub.ac.be/~comdig/06iccs/Brin.MOV.

Photo by 'SeraphimC retrieved from Flickr.