Superforecasting — Stewart Brand’s Summary of SALT Talk by Philip Tetlock

Will Syria’s President Assad still be in power at the end of next year?  Will Russia and China hold joint naval exercises in the Mediterranean in the next six months?  Will the Oil Volatility Index fall below 25 in 2016?  Will the Arctic sea ice mass be lower next summer than it was last summer?

Five hundred such questions of geopolitical import were posed in tournament mode to thousands of amateur forecasters by IARPA—the Inatelligence Advanced Research Projects Activity–between 2011 and 2015.  (Tetlock mentioned that senior US intelligence officials opposed the project, but younger-generation staff were able to push it through.)  Extremely careful score was kept, and before long the most adept amateur “superforecasters” were doing 30 percent better than professional intelligence officers with access to classified information.  They were also better than prediction markets and drastically better than famous pundits and politicians, who Tetlock described as engaging in deliberately vague “ideological kabuki dance.”


What made the amateurs so powerful was Tetlock’s insistence that they score geopolitical predictions the way meteorologists score weather predictions and then learn how to improve their scores accordingly.  Meteorologists predict in percentages—“there is a 70 percent chance of rain on Thursday.”  It takes time and statistics to find out how good a particular meteorologist is.  If 7 out of 10 such times it in fact rained, the meteorologist gets a high score for calibration (the right percentage) and for resolution (it mostly did rain).  Superforecasters, remarkably, assigned probability estimates of 72-76 percent to things that happened and 24-28 percent to things that didn’t.

How did they do that?  They learned, Tetlock said, to avoid falling for the “gambler’s fallacy”—detecting nonexistent patterns.  They learned objectivity—the aggressive open-mindedness it takes to set aside personal theories of public events.  They learned to not overcompensate for previous mistakes—the way American intelligence professionals overcompensated for the false negative of 9/11 with the false positive of mass weapons in Saddam’s Iraq.  They learned to analyze from the outside in—Assad is a dictator; most dictators stay in office a very long time; consider any current news out of Syria in that light.  And they learned to balance between over-adjustment to new evidence (“This changes everything”) and under-adjustment (“This is just a blip”), and between over-confidence (“100 percent!”) and over-timidity (“Um, 50 percent”).  “You only win a forecasting tournament,” Tetlock said, “by being decisive—justifiably decisive.”

Much of the best forecasting came from teams that learned to collaborate adroitly.  Diversity on the teams helped.  One important trick was to give extra weight to the best individual forecasters.  Another was to “extremize” to compensate for the conservatism of aggregate forecasts—if everyone says the chances are around 66 percent, then the real chances are probably higher.

In the Q & A following his talk Tetlock was asked if the US intelligence community would incorporate the lessons of its forecasting tournament.  He said he is cautiously optimistic.  Pressed for a number, he declared, “Ten years from now I would offer the probability of .7 that there will be ten times more numerical probability estimates in national intelligence estimates than there were in 2005.”

Asked about long-term forecasting, he replied, “Here’s my long-term prediction for Long Now.  When the Long Now audience of 2515 looks back on the audience of 2015, their level of contempt for how we go about judging political debate will be roughly comparable to the level of contempt we have for the 1692 Salem witch trials.”

—Stewart Brand, Seminars About Long-term Thinking

Tetlock audio and video

About Lloyd Kahn

Lloyd Kahn started building his own home in the early '60s and went on to publish books showing homeowners how they could build their own homes with their own hands. He got his start in publishing by working as the shelter editor of the Whole Earth Catalog with Stewart Brand in the late '60s. He has since authored six highly-graphic books on homemade building, all of which are interrelated. The books, "The Shelter Library Of Building Books," include Shelter, Shelter II (1978), Home Work (2004), Builders of the Pacific Coast (2008), Tiny Homes (2012), and Tiny Homes on the Move (2014). Lloyd operates from Northern California studio built of recycled lumber, set in the midst of a vegetable garden, and hooked into the world via five Mac computers. You can check out videos (one with over 450,000 views) on Lloyd by doing a search on YouTube:

3 Responses to Superforecasting — Stewart Brand’s Summary of SALT Talk by Philip Tetlock

  1. very interesting…

    however, re " level of contempt", I strongly suspect that audience in 2515 (while it looks back in contempt) will itself be the subject of similar contempt in years forward for their views/opionions/actions.

    seems to be the evolution of human nature to want to think their thoughts/actions/ideologies etc are "much better"/"much more advanced"/"much more moral" / etc… etc…

    there seems (to my mind) to be a lot of cyclical thought going on in human thinking over the past fifty/hundred years, and more…

  2. Anonymous

    People in their hubris tend to discount their predecessors as ignorant and foolish. Every person of our day has far more factual hard science knowledge at hand than George Washington or even Issac Newton. But it would be absurd to confuse that knowledge base for wisdom and judgement. They fail to realize that conditions in the present are the way they are for valid reasons, that we are no wiser than those who came before us.They would do better to show a healthy respect for the values, judgements and hard work of our ancestors that handed down a system that has blessed us with freedoms and prosperity surpassing any seen before. They didn't create a perfect society but history offers no reason to believe that we will fare any better in that regard.

Leave a Reply