A registry for predictions

As part of our research, UNIDIR’s Disarmament as Humanitarian Action project has been exploring why it’s difficult to accurately predict future events in the political field (see John Borrie’s recent post “Putting Predictions to the Test”). On the same topic, I recently saw David Brin, a physics professor, NASA consultant and science fiction novelist, talk about the idea of a “prediction registry”.

Although making predictions about the future is an extremely difficult task—especially in the political field—this is something that humans have always done. (Brin noted that our prefrontal lobes, which appeared perhaps a few hundred thousand years ago and which are specific to the human brain, play a significant role in exploring potential outcomes and making predictions.)

Today, we’ve developed tools and techniques to help us make forecasts. Computer simulations, and agent-based modelling in particular, are used to analyze complex phenomena. Although very sophisticated, these new techniques can only provide a limited amount of information about the future behaviour of a complex system. This is because, as complexity theory shows, complex systems are “inherently” unpredictable.

However, as Brin noticed, this doesn’t prevent us from making predictions, nor does it prevent us from spending a lot of money and time in trying to make accurate guesses. Starting from this insight, and realizing that there is no record of who was right, how often and when, Brin proposed a prediction registry be set up: “anyone who claims any form of special foresight might be judged by the same standards that applies to any other field of endeavour—success or failure.”

In addition, each forecast would receive a “specificity multiplier”, taking a high value if the prediction is very precise (providing dates, names, places, etc.) and a low value it if the forecast is obscure.

This could be valuable. First, predictors and forecast specialists would “be held accountable for all of their predictions, not just those they later choose to remember.” This is important as it would significantly improve transparency. The introduction of an “accuracy score”, in particular, could reveal the most accurate predictors, giving them credibility and recognition, and increasing their role in policy making.

There might be some surprises. As Philip Tetlock showed in his book “Expert Political Judgment: How Good is It? How Can We Know?”, experts aren’t necessarily more competent than the rest of us.
Finding out who is right a lot in prediction and problem solving, while denying legitimacy to those who are consistently wrong, would probably be in society’s best interest.

But most importantly, the analysis of scores, Brin suggested, could also help us understand our own patterns of success, failure and predictions. Tetlock’s work has already underlined the influence of various psychological biases on so-called experts. Yet policy analysts and decision makers aren’t soothsayers. If we’re at all realistic, we need to recognise that predictors can never be always right. But if a prediction registry made more of them rigorous in their thinking and less instinctively cocksure in their predictions, that would be a big step forward for 21st century society.


Aurélia Merçay


References

David Brin, Accountability for Everyday Prophets: A Call for a Predictions Registry, 2005, available online at http://www.davidbrin.com/predictionsregistry.html.

A list of David Brin’s predictions is available online at http://earthbydavidbrin.pbwiki.com/Predictions.

Video of David Brin, interviewed at the 6th International Conference on Complex Systems (ICCS) in Boston in June 2006: http://complexity.vub.ac.be/~comdig/06iccs/Brin.MOV.

Photo by 'SeraphimC retrieved from Flickr.

Putting Predictions to the Test

As part of it's work, UNIDIR's Disarmament as Humanitarian Action project is currently working on a publication looking at some constraints on multilateral negotiations, and how some of these might be overcome or at least potentially mitigated.

One conundrum for policy makers in general is that it's very difficult to accurately predict outcomes of complex political and economic processes, and this has implications for their work. Multilateral practitioners are no exception.

Meanwhile, it's often very difficult to get a handle on how good "expert" political judgment is. But not impossible. A recent book by Philip E. Tetlock, an American psychologist, has revealed through a careful decade-long study that people who make prediction their business - people who appear as experts on television, for instance, or who are quoted in newspaper articles, advise governments and businesses - are, on average, no better than the rest of us.

Tetlock's research team asked various specialists to judge the likelihood of a number of political, economic and military events occurring within a specific timeframe (about five years ahead). Close to 300 specialists offered outcomes representing a total number of around 27,000 predictions. As Nassim Nicholas Taleb observed in his book, The Black Swan:

"His study revealed an expert problem: there was no difference in results whether one had a PhD or undergraduate degree. Well-published professors had no advantage over journalists. The only regularity Tetlock found was the negative effect of reputation on prediction: those who had a big reputation were worse predictors than those who had none."

Ouch. But Tetlock's study didn't end there. His focus wasn't so much to show the real level of competence of experts, but to investigate how they spun their stories. Tetlock discovered various cognitive mechanisms of generating ex post explanation - mostly in the form of belief defence, or self-esteem protection.

Prominent were phenomena that psychologists call motivational biases, something Ashley Thornton has explored in previous posts on this blog. These include the self-serving attributional bias (the human tendency to blame unfavourable outcomes on external causes but take credit for favourable outcomes) and the confirmation bias (we tend to seek out and process information that confirms our pre-existing beliefs).

The New Yorker put it succinctly: "The experts' trouble in Tetlock's study is exactly the trouble that all human beings have: we fall in love with our hunches, and we really, really hate to be wrong."

Food for thought. More about Tetlock's work in future posts.


John Borrie


References

Philip E. Tetlock, Expert Political Judgment: How Good is it? How Can We Know? (Princeton NJ, Princeton University Press, 2005).

Nicholas Nassim Taleb, The Black Swan: The Impact of the Highly Improbable (New York, Random House, 2007).

To read a neat introduction to Tetlock's work in a review of his book in the 5 December 2005 edition of The New Yorker click here.

Picture by F33 retrieved from Flickr.

Let’s polish up the crystal ball

The World Future Society recently released the Outlook 2008 Report as part of the November-December 2007 issue of its magazine “The Futurist”.

This report includes some 70 forecasts covering developments and breakthroughs in technology, energy and the environment, international relations and society in general.

Among the thinkers who have contributed to “The Futurist” magazine in the past years are current climate change activist (and just-announced Nobel Peace prize co-winner) Al Gore, former United States Secretary of Defense Robert McNamara and former UN Secretary General Kofi Annan.

So, what should we expect in the coming years? Here are the top 10 forecasts of the Outlook 2008 Report, some of which are directly relevant to human security and multilateral disarmament diplomacy:

1. “The world will have a billion millionaires by 2025”, as a result of globalization and technology innovations.

2. “Fashion will go wired as technologies and tastes converge to revolutionize the textile industry.” Smart fabrics – such as colour-changing or perfume emitting textiles –, digital wallets and computer shoes are just a few examples of the new technologies that will radically transform the textile industry.

3. “The threat of another cold war with China, Russia, or both will replace terrorism as the chief foreign-policy concern of the United States.” According to Luttwak, terrorist attacks are a relatively minor threat to the United States that Soviet missile capabilities.

4. “Counterfeiting of currency will proliferate, driving the move toward a cashless society.”

5. “The earth is on the verge of a significant extinction event. The twenty-first century could witness a biodiversity collapse 100 to 1,000 times greater than any previous extinction.”

6. “Water will be in the twenty-first century what oil was in the twentieth century.”

7. “World population by 2050 may grow larger than previously expected, due in part to healthier, longer-living people.”

8. “The number of Africans imperiled by floods will grow 70-fold by 2080”. The predictions indicate a 38cm increase of sea level by 2080. If this is the case, the number of Africans affected by floods is estimated to grow from 1 million to 70 millions.

9. “Rising prices for natural resources could lead to a full-scale assault on the Arctic”. According to Brigham, the control over arctic natural resources will be a major political challenge in the next decades.

10. “More decisions will be made by nonhuman entities”. Because of the ever-increasing complexity of the world, artificial intelligence will play an increasing role in decision-making processes.

Of course, “The Futurist’s” predictions could well be proved wrong. That’s the problem with prediction – hence we’re not getting around in jet cars or rocket-powered backpacks despite assertions that it would be the case by today a few decades ago. Conversely, few saw the Internet or the iPod coming.

Humans often make their predictions based on current evidence and past experiences. If there’s one thing we’re learning in an increasingly interdependent world it’s that plenty of social phenomena are non-linear and inherently unpredictable. In complex systems – and complex social systems in particular –, actions sometimes have unintended consequences.

So, while forecasts like those in “The Futurist” are stimulating, I’m sure we’re in for some even more way-out surprises it didn’t predict – both good and bad.


Aurélia Merçay


Reference

The top ten forecasts from the Outlook 2008 report by the World Future Society are available online.