Putting Predictions to the Test

As part of it's work, UNIDIR's Disarmament as Humanitarian Action project is currently working on a publication looking at some constraints on multilateral negotiations, and how some of these might be overcome or at least potentially mitigated.

One conundrum for policy makers in general is that it's very difficult to accurately predict outcomes of complex political and economic processes, and this has implications for their work. Multilateral practitioners are no exception.

Meanwhile, it's often very difficult to get a handle on how good "expert" political judgment is. But not impossible. A recent book by Philip E. Tetlock, an American psychologist, has revealed through a careful decade-long study that people who make prediction their business - people who appear as experts on television, for instance, or who are quoted in newspaper articles, advise governments and businesses - are, on average, no better than the rest of us.

Tetlock's research team asked various specialists to judge the likelihood of a number of political, economic and military events occurring within a specific timeframe (about five years ahead). Close to 300 specialists offered outcomes representing a total number of around 27,000 predictions. As Nassim Nicholas Taleb observed in his book, The Black Swan:

"His study revealed an expert problem: there was no difference in results whether one had a PhD or undergraduate degree. Well-published professors had no advantage over journalists. The only regularity Tetlock found was the negative effect of reputation on prediction: those who had a big reputation were worse predictors than those who had none."

Ouch. But Tetlock's study didn't end there. His focus wasn't so much to show the real level of competence of experts, but to investigate how they spun their stories. Tetlock discovered various cognitive mechanisms of generating ex post explanation - mostly in the form of belief defence, or self-esteem protection.

Prominent were phenomena that psychologists call motivational biases, something Ashley Thornton has explored in previous posts on this blog. These include the self-serving attributional bias (the human tendency to blame unfavourable outcomes on external causes but take credit for favourable outcomes) and the confirmation bias (we tend to seek out and process information that confirms our pre-existing beliefs).

The New Yorker put it succinctly: "The experts' trouble in Tetlock's study is exactly the trouble that all human beings have: we fall in love with our hunches, and we really, really hate to be wrong."

Food for thought. More about Tetlock's work in future posts.


John Borrie


References

Philip E. Tetlock, Expert Political Judgment: How Good is it? How Can We Know? (Princeton NJ, Princeton University Press, 2005).

Nicholas Nassim Taleb, The Black Swan: The Impact of the Highly Improbable (New York, Random House, 2007).

To read a neat introduction to Tetlock's work in a review of his book in the 5 December 2005 edition of The New Yorker click here.

Picture by F33 retrieved from Flickr.

Fooled by randomness?

One interesting and useful aspect of working in a think-tank like UNIDIR is that I often get the opportunity to talk with diplomats about their work and can sometimes observe them doing it. In return, diplomats can - and do - give feedback on the sorts of ideas and suggestions we can offer as researchers on the Disarmament as Humanitarian Action (DHA) project.

In multilateral diplomacy, like many other walks of life, there seems to be a bit of a generation gap or, perhaps more accurately, a "rank" gap. Younger, less-senior diplomats are fascinating to listen to in this regard, as their collective sense of the weaknesses of their institutional structures and ways of working, and problems with collective approaches is sometimes much more acute than the ambassadors. I imagine it may be because ambassadors are more sheltered from some of the less sexy drudgery of multilateral meetings and perhaps even feel they have more a stake in the traditions that characterize their "community of practice".

Whatever the reason, I had a productive time today talking with young officials on the United Nations Disarmament Fellows programme. (Well, young is relative - many are older than me.) These junior to mid-level diplomats come from a wide variety of countries from Pakistan to Switzerland to Fiji and bring with them a diverse range of outlooks.

Conscious that they've been lectured to a lot in recent weeks about the nuts-and-bolts of "disarmament machinery", I thought I'd take a different tack. Recently I began reading Nassim Nicholas Taleb's remarkable book The Black Swan: The Impact of the Highly Improbable, which I think should be required reading for every policy maker. (Taleb is also author of an earlier book, Fooled by Randomness.)

There's an old saying that what you don't know can't hurt you. It's quite a stupid expression really, since the things we don't know really can hurt us (think about the 1929 and 1987 stock-market crashes, which hurt plenty of people. Or the 2004 Asian tsunami).

Taleb argued in his book that the things that can hurt us most are the things we think we do know, but actually don't. To paraphrase Donald Rumsfeld, if there are "known unknowns" and "unknown unknowns", the problem is that some of the latter are unknown precisely because we think we know them but don't. Taleb noted that nobody, for instance, predicted the abrupt collapse of the Soviet Union's empire in 1989. Soviet-watchers were convinced the communist regime would remain in power for years longer, if not decades. The fall of the Berlin Wall surprised everyone.

Taleb's book was a useful entry point for exploring with these 20 or so younger diplomats two broad themes, drawing from a wide range of literature, the DHA project's research and this blog (type 'Thornton' into the search-box for some entries):

- Cognitive features hardwired into all human beings constrain the way in which we can perceive the world, and thus affect our decision making.

- A second problem is that the world isn't a smooth, linear narrative. That is, the world doesn't conform with our expectations, although we may fool ourselves into thinking so. That's because, as human beings, we have brains that are very good at re-shaping our expectations in hindsight without even realizing it, of confirming our beliefs without evidence, and intuitively leading us to the wrong answers when careful thought would serve us much better. We're not well-adapted to cope with complexity, which can be counter-intuitive.

It resulted in an interesting discussion and a good level of engagement. Perhaps the diplomats were just being polite, but I got the impression they regard these issues as real, and worthy of more attention in multilateral diplomacy than they currently get. They seem concerned that "business as usual" isn't working, and we need better ways of cooperating in the face of international uncertainty. I suspect Nassim Nicholas Taleb would agree.


John Borrie


References

Nassim Nicholas Taleb's web page is here.

Photo courtesy of Thomas Nash.

The Power of Priming

The human subconscious, it turns out, is more active than we think. And it can be primed in remarkable ways by seemingly mundane objects, smells and sounds, as a recent New York Times article outlined.

The article reported, for instance, one recent study by Yale psychologists, which revealed that college students handed either a cup of hot or iced coffee on the way to class were influenced by the temperature of the beverage: when later asked to judge a person from a written description, those handed iced coffee rated them “much colder, less social and more selfish” than those handed hot coffee.

In another study, students were asked to play an investment game with an unseen player. The students were placed at either a long table with a black briefcase and leather folder or a backpack left at the end of the table. The results revealed that students played differently when the briefcase and folder as opposed to the backpack were on the table. According to the researchers, the briefcase and leather folder primed the students to be stingier with their money, drawing on unconscious associations with “business-related associations and expectations.”

In yet another experiment, students were placed in a cubicle in one of two rooms and asked to fill out a questionnaire. After filling in the questionnaire, the students were given a crumbly snack. Unbeknownst to them, the researchers had placed a bucket of water with a hint of citrus cleaner in one room but not the other. Sure enough, the students in the room with the faint odor of cleaning solution cleaned up the remnants of their snack three times more than the control students.

The moral of the story, for me, is that we’re always trying to determine how we should behave in any given situation. But we may not be cognizant of all the factors influencing our perceptions, with the “conscious” brain taking directions from our subconscious that we didn’t even realize were issued.

It begs an interesting question for me, working in the 80 year old splendour of the UN Palais des Nations (formerly the seat of the defunct inter-war League of Nations). How might a room like the Council Chamber in Geneva, the scene of over a decade of floundering Conference on Disarmament (CD) meetings, prime diplomats for a new session of work? As you can see from the photo I’ve included with this posting, the Council Chamber’s drab green and gold murals tell the tale of human suffering and toil. Heavy green curtains make sure not a sliver of daylight peeks through and the Chamber’s walls bring to life murals like “The Conquerors,” “The Conquered” and “The Death of Freedom” (which themselves are a little on the surreal side). And the Council Chamber sometimes feels more like a museum than a working disarmament body, such is the reverential hush of CD meetings there, as interventions are delivered and translated through elderly surgical plastic wired earpieces to formally attired diplomats.

A legacy of deadlock sends one message to disarmament diplomats. But perhaps murals that portray human struggle, in the drabbest of drab colors with no natural light, send another. It just might put diplomats in the wrong frame of mind. Of course, I’m not making the claim that simply changing the room would be the difference between success and failure in the CD. But it might be a start.


Ashley Thornton


References

Benedict Carey, Who’s Minding the Mind, The New York Times, 31 July 2007:
http://www.nytimes.com/2007/07/31/health/psychology/31subl.
html?ei=5087%0A&em=&en=8a58897f89c79289&ex=118620000
0&adxnnl=1&adxnnlx=1186053879-G5OMwHDV76z/9oDeP7jvGA

This photo is part of a mural, “The Solidarity of Peoples (The Lesson at Salamanca),” by José María Sert in the Conference on Disarmament chamber, Geneva. Taken by Mel and John Kots, retrieved from flickr: http://www.flickr.com/photos/melanieandjohn/127443220/