Why an Open Mind Is Key to Making Better Predictions

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

In the new book, Superforecasting, Wharton professor Philip Tetlock and coauthor Dan Gardner look into why making predictions is so difficult — and how to become better at making them. More: http://knlg.net/1L69rZa 📂 Management 🏷️ Research

Read the full transcriptHide the full transcript
  1. everybody wants to see into the future whether it's to find out whether or not we should take a certain job figure out
  2. what's going to happen with a particular relationship or maybe we just want to know if it's going to rain on Sunday but
  3. despite all this interest in forecasting most actual forecasts aren't very good and even and moreover they're also not
  4. very well analyzed in the new book super forecasting the art and science of prediction Wharton Professor Philip
  5. tetlock looks into what makes people good forecasters and suggests how you can incorporate some of those techniques
  6. into your own life Philip thanks for being with us thank you my pleasure now thanks to people like Tom Friedman and
  7. Nate Silver and I think also maybe to the rise of Big Data there seems to be a lot of interest in forecasting so I was
  8. really surprised to learn from your book that despite all this interest in forecasting and maybe interest in people
  9. who have fashioned themselves as forecasters forecasting itself's not very well studied it's not very well
  10. analyzed at all I think that's fair to say and it's pretty threatening to keep score of your forecasting accuracy
  11. you imagine you're a big shot pundit what incentive would you have to submit to a forecasting tournament in which you
  12. had to play on a level playing field against ordinary human beings and the answer is not much because the best
  13. possible outcome you could obtain is to tie it you're expected to win so the best outcome is a tie and there's a good
  14. chance our research suggests that you're not you're not going to win now the book
  15. is acted was actually decades in the making I mean your research into this has stretched back for years and you
  16. actually start the book and in some ways this all started with a dart throwing chimp could you tell us a little bit
  17. about that story and I know you said in the book that people don't exactly get the right takeaways out of that study
  18. but I thought it was interesting to sort of show what happens when we try and test forecasting sure well in our early
  19. work which which I'm going to reveal how old I am goes back into the mid 1980s we did use the metaphor of the dart
  20. throwing chimp to capture the idea baseline for performance which is how much better can you do than chance if
  21. you had a system that we did that was just generating forecasts by chance how about would you do relative to that
  22. and that actually is a baseline that some people can't beat now they can't beat it
  23. for a lot of reasons sometimes the environment is just hopelessly difficult ranked if you were trying to bet on the
  24. roulette wheel in Las Vegas you're not going to be able to do any better than a dart throwing chimp but people sometimes
  25. fail to beat the dart throwing chimp even in environments where there are predictable regularities that could be
  26. picked up if you were being a student off now in the book you point out that a lot of people took away from that study
  27. that all predictions are bad that forecasting is bad but in fact what you were pointing out really is that there's
  28. actually limits on predictability but it's not all bad it's just that it there are limits on it
  29. that's right you don't want to be too hard on people because some environments really are there's a lot of irreducible
  30. uncertainty in some environments it's very difficult to to bring it down below a certain point so it's unfair to
  31. portray people as as being dumb in some sense if they're failing to do something
  32. that's impossible of course we don't know what's impossible until we try until we try and
  33. earnest you net you don't discover what we call the optimal forecasting frontier
  34. you don't discover how good it's possible to become in a particular forecasting environment until you run
  35. forecasting tournaments competitive tournaments you plug in your best techniques for maximizing accuracy and
  36. you see how good you can get and this essentially what we did in the forecasting tournaments with the US
  37. government wasn't sponsored by the intelligence community the intelligence Advanced Research Projects Agency
  38. we these are forecasting tournaments that run between 2011 and 2015 Volvo tens of thousands of forecasters trying
  39. to predict about 500 questions posed by the intelligence community over that period of time and he found that some
  40. people could do quite a bit better than the dart throwing chimp and they could meet some more demanding baselines as
  41. well now one of the interesting things in the book is I mean you talked a little bit about where you found these
  42. forecasters who are part of your study which is called the good judgment project can you talk a little bit about
  43. where you recruited these people from and then also how these tournaments actually took place like what exactly
  44. were they called on to do and how did they do well we were very opportunistic we recruited forecasters by advertising
  45. through professional societies by advertising through blogs a number of high-profile bloggers helped us to
  46. recruit forecasters people like Tyler Cowen Nate Silver various people were helpful in recruiting forecasters plus
  47. we knew quite a few people from the earlier work that I'd done an expert political judgement so we were able to
  48. gather initially a group of several thousand and we were able to build on that in subsequent years the questions I
  49. you know I have to be careful about making big generalizations about how good or bad people are as forecasters as
  50. I mentioned before you can make people look really bad if you want to you can pose just intractably difficult
  51. questions or you can make people look really good you can you can pose questions that aren't all that hard and
  52. so you you want to be wary of research that does cherry-picking and and and there are some aspects that some aspects
  53. of that and some in some of the literature but we were looking for was a process of generating questions that
  54. wasn't rigged one way or the other and the method we came up with was generating questions through the US
  55. intelligence community there were questions that people inside the US intelligence community felt would be
  56. of national security interest and relevance and reasonably representative of the types of tasks that intelligence
  57. analysts are asked to do so these were questions typically they asked people to see out into the future
  58. several months occasionally a bit longer occasionally shorter and we scored the accuracy of their
  59. judgments over time we didn't have people make judgments one way or the other it wasn't yes or now we had people
  60. make judgments on what's called a probability scale ranging from 0 to 1 and we carefully computed accuracy over time
  61. we identified some people who are really good at it we called them super forecasters and but then they were later
  62. assembled into super teams of super forecasters and they you know dominated the tournament tournament essentially
  63. over the next over the next four years but we we did a number of other experiments as well looking for
  64. techniques that could could be used to improve accuracy and we found some and now these super forecasters they really
  65. came from all walks of life but one of the things you point out in the book is that what makes a good forecaster is
  66. really how you think and can you talk a little bit about what do you mean by that and what are some of the unifying
  67. characteristics of super forecasters well when you ask people in the political world who has good judgment
  68. the answer typically is people who think like me so liberals tend to think that liberals have good judgment and if you
  69. have good forecasting judgment and conservatives tend to think that they're better at it it turns out to be the case
  70. that that pork a good forecast forecasting accuracy is not very closely associated with with ideology there's a
  71. slight tendency for people who are the super forecasters to be more moderate and less ideological but there there are
  72. lots of super forecasters who have strong opinions oh it distinguishes super forecasters is their ability to
  73. put aside their opinions at least temporarily and just focus on accuracy and that's that's a very demanding
  74. exercise for people now other things you mentioned I don't like at the end of the
  75. book you have ten commandments for super forecasters and so I'm wanting now with
  76. the super forecasters are there ways to make even super forecasters better are there conditions or environments to make
  77. them I guess super super forecaster super super forecasters well event eventually you're gonna reach a
  78. point where you're not going to be able to get any better because the as I've mentioned it's some environment the
  79. environment itself has some degree of irreducible uncertainty so no matter how good you are you're not gonna do a very
  80. good job probably predicting what the value of Google is going to be next week on the on the New York Stock Exchange
  81. there so there are some things that are very difficult to do and it's not clear that the super forecast
  82. even using super forecasters is going to let you make appreciable headway on that
  83. but there are many things that are quite doable that we previously didn't think we're doable and you there's a lot of
  84. room for improving the accuracy of probability judgments on those things those are things like predicting weather
  85. international conflicts or going to escalate or deescalate whether certain treaties are going to be signed or
  86. approved by legislative legislators well what but whether Greece is going to leave the eurozone so there are a lot of
  87. problems that you know have relevance the financial markets have relevance to business decisions where there is
  88. potential for improving probability judgment where we have shown that experimentally now in the IR /
  89. tournament where people typically don't don't do that people typically rely on vague verbiage forecasts they people say
  90. well I think it's possible or this could happen this might happen it's likely those are terms that you know are
  91. informative at some level but they're not all that informative if I say that something could happen for example you
  92. know Greece could leave the eurozone by the end of 2017 what does that mean it could mean I think there's a probability
  93. of 1 percent or 99 percent you know we could be hit by an asteroid tomorrow the Sun could rise tomorrow yeah I know
  94. there though there it's it's it's a very elastic word so the asking people to make crude
  95. quantitative judgments which become progressively more refined over time is a very good way to both keep score and
  96. get better at it now I find it I find it really ironic in the book that you know
  97. we have all of these people in the world who have sort of fashioned themselves as
  98. professional forecasters I mean pundits on TV they're really television personalities media personalities and
  99. it's almost some and it seems like to do that job it's almost kind of a cult of personality you don't want to be proven
  100. wrong you would never admit that you're wrong you're just gonna sort of keep kicking it down the road and say no it's
  101. going to happen but what you find in the book is that super forecasters one of the things that unites all of them
  102. despite coming from all these walks of life is that there a lot they're willing to be proven wrong or live they are open
  103. to the idea they're willing to sort of look at things look at evidence and retread and pivot and so I found that
  104. interesting it was - it was kind of ironic and I know you tell a story in the book about foxes vs. hedgehogs that
  105. was very interesting right well yeah but the Fox Hedgehog metaphor is is drawn out of a surviving fragment
  106. of poetry from the Greek warrior poor Killa cos 2,500 years ago scholars have puzzled over it for over
  107. the centuries it runs something like this and of course I don't know ancient Greek so I'm taking it on faith this is
  108. what it actually says the the Fox knows many things but the Hedgehog knows one big thing
  109. now you can think of hedgehogs in in debates over political and economic issues as
  110. people who have a big ideological vision Tom Friedman might be animated by a vision of a globalization the world is
  111. flat libertarians are animated by the vision that free Mart there are free market
  112. solutions for the vast majority of problems that beset us there are people on the left to see the
  113. need for major state intervention to address various inequities there there are there are environmentalists who
  114. think that we're on the cusp of an apocalypse of some sort and so you have people who are animated by a vision and
  115. their forecasts are informed largely by that vision and whereas the Foxes tend to be more eclectic they kind of pick
  116. and choose their ideas for a variety of schools of thought they might be a little bit environmentalist and a little
  117. bit libertarian or there might be a little bit socialist and a little bit hawkish on certain national security
  118. issues there they they blend things in unusual ways it's and they're harder to classify politically now in the early
  119. work we found that the Foxes were more COO or more eclectic in their style of thinking we're better forecasters than
  120. the hedgehogs and in the later work we found something similar we found that people who scored
  121. high on psychological measures of active open-mindedness and need for cognition those people who score high on those
  122. personality variables tended to do quite a bit better as forecasters now it seemed to me then that we would really
  123. want is for more foxes to be kind of these really prominent forecasters or these people that we're seeing on
  124. television that we're seeing in the news but you know but then at the same time and although seems like their
  125. personalities are just not necessarily what we think of like we're at odds with what we think of when we
  126. think of a leader when we think of someone who's always putting themselves out there so how does that what does
  127. that mean for trying to get more accurate forecasters I mean how do we get people to be lists to listen to the
  128. Foxes when they might not be the sexiest or the most prominent or the most the people that we want to look at all the
  129. time or listen to all the time that's right so what do we do that is a bit of a dilemma
  130. imagine you are a producer for a major major television show and you have a choice between someone who's going to
  131. come on the air and tell you that something decisive and bold and interesting is going to sit eurozone is
  132. going to melt down the next two years or Chinese economy is going to melt down or
  133. there's gonna be a jihadist coup and Saudi Arabia he's got he's got a good big interesting story to tell and the
  134. person knows quite a bit and can mobilize a lot of reasons to support the doom stir prediction say on eurozone or
  135. China or Saudi Arabia or a boom stir prediction for that matter it doesn't matter but the person is charismatic and
  136. forceful and can generate a lot of reasons why he or she is right as opposed to someone who comes on and says
  137. yeah there's some danger eurozone's gonna melt down but on the other hand there are these countervailing forces
  138. and on balance problem nothing dramatic is probably gonna happen in the next year or so but it's possible that this
  139. could work so I'm who make what makes better television to ask the question is to answer it so there is a tent there is
  140. a preference for hedgehogs in part because hedgehogs generate better sound bites and people who generate better
  141. sound bites generate better media ratings and that is what people get promoted on in the in the media business
  142. so there is a bit of a perverse inverse relationship between how between having the skills that go into being a good
  143. forecaster and having the skills that going go into being a fact of media presence now how does this come into
  144. play though I mean sort of in the world that maybe those of us who are not hiring forecasters in a regular basis
  145. don't so if I'm a company or the government and I'm trying to assemble a team of good forecasters or get good
  146. accurate or accurate as possible forecasts what can I take from this book I mean in terms of finding these people
  147. and employing these people and is that happening now or is it the same as what maybe what we're seeing on CNN our Fox
  148. News all right well I think a lot of organizations in both the private and the public sector are taking an
  149. increasing interest in forecasting tournaments and using them as methods of keeping score on how accurate their
  150. their forecasting methods are they're people and their methods and their teams there's a growing interest also in using
  151. forecasting tournaments to identify people who are better at it and you know develop your own core of super
  152. forecasters there's growing interest in exploring methods of training people to be better forecasters so I think this is
  153. going on I probably shouldn't mention the names of any organizations who have adopted
  154. this right now but a US intelligence community obviously has taken an active interest in this area and I and some
  155. private sector organizations have as well now it was interesting to me that um so you were discussing a lot of the
  156. book talks about a little bit about framing so it's not just about finding people who can make good forecasts but
  157. it's also a lot about you know finding the right questions finding the right ray to frame the problem breaking down a
  158. big problem into smaller little clusters I mean if there's so much that goes into
  159. forecasting other than the actual forecasts that comes out of it and I wonder that do people leave are people
  160. thinking enough about those things as well in addition to finding people who give good forecasts I see that as one of
  161. the big objectives of the next generation of forecasting tournaments to focus on
  162. generating not just good answers but good questions in the book we talked about the parable of Tom Friedman and
  163. Bill flack Tom Friedman is of course a famous New York Times columnist Pulitzer Prize winner who
  164. regular at Davos in the White House circulates in networks of power bill flack is an anonymous
  165. retired hydrologist in Nebraska who also is a super forecaster we know a huge amount about Bill flocks forecasting
  166. track record because he answered a very large number of questions in the course of the tournament and demonstrated he
  167. could do so effectively but we know virtually nothing about Tom Friedman's forecasting track record notwithstanding
  168. that he's written a great deal over the last 35 years and he's a music he's a powerful analyst and a writer and and he
  169. does many things very well but there's no way really to reconstruct with any degree of certainty
  170. reasonable certainty how good a forecaster he is and Tom Friedman has detractors he has admirers his his his
  171. his admirers might say well he was right about that it was a bad idea to expand NATO eastward because it would provoke
  172. nationalist backlash in Russia or he was or he was wrong about Iraq because he supported the 2003 invasion
  173. people have a lot of opinions about those things now here's what we we did a careful analysis
  174. of Tom Friedman's columns and one of the things we noticed is even though it's very difficult to discern whether or not
  175. he's a good forecaster going back after the fact it is possible to detect we have some
  176. really good questions he said he's a pretty darn good question generator and we've actually begun to insert some of
  177. his his ideas for questions there they tend to be rather open-ended we've managed to
  178. translate some of them into future forecasting tournaments so let me give you an example from the past that
  179. illustrates the tension between being a super question generator and a super forecaster so in 2008 2002 early 2003
  180. before the Iraq invasion Tom Friedman wrote what I thought was a really quite brilliant column on on Iraq in which he
  181. posed the following question we could really cut to the essence of a war what key issue in deciding whether to go into
  182. a rocky yes is Iraq the way it is today because Saddam Hussein is the way he is or a
  183. Saddam Hussein the way he is because Iraq is the way it is the chicken and the egg and what would happen if you if
  184. you took away Saddam Hussein with the country disintegrate into a war of all against all or would it move to boredom
  185. going to become a Jeffersonian liberal democracy in the next 15 or 20 years now that maybe I'm not done maybe not
  186. quite that fast but in the in that direction things things would move in that would move in that direction so
  187. Tom Friedman didn't know the answer to that question many people think he made a big mistake
  188. in supporting the invasion of Iraq in 2003 but he was shrewd enough to pose the right question and if we've been
  189. running forecasting tournaments in late 2002 early 2003 that would have been something we would have wanted very much
  190. to include in in in in in that exercise and so I think the way the right way to think about Tom Friedman and Bill flack
  191. is that you know is that they they're complementary and that Tom Friedman's greatest contribution to forecasting
  192. tournaments may well be and his perspicacity and generating incisive questions he may be a good forecaster too but we
  193. just don't know that yet right but we need in order but good forecasting we need the Tom Friedman's in the world and
  194. the bill flacks so it to me it would seem it's just a question of trying to get them together in the right ways and
  195. the right permutations to get the best better and to get better prediction that's where we come around in the book
  196. it's not really Tom versus bill it's Tom and Bill it should be symbiotic right and now to get a little back to I
  197. mentioned big data at the beginning now in the age of supercomputers a machine learning and we're saying we can enter
  198. this into anything and get these answers back to us I mean what do you think how
  199. do you think the role of human forecasting is going to change I mean how do these how do using computers
  200. using data how does that complement or even compete with human forecasting well in the book we conducted an interview
  201. with David Ferrucci who was when he was an IBM scientist he was responsible for developing a famous computer program
  202. known as Watson which defeated the best human jeopardy players and we asked him a number of questions about his views
  203. about the human machine forecasting and one question one law one line of questioning was particularly interesting
  204. I think it was we it was it was very clear to him and that it would be possible for
  205. a system like Watson to answer the following question reasonably readily which two Russian leaders traded jobs in
  206. the last five years that question what Watson could search his historical database it could figure it out
  207. reframe the question as will those Russian same Russian leaders change jobs in the next five years would Watson have
  208. any capacity to answer a question like that and and his answer was no and the question well how difficult would it be
  209. to reconfigure Watson so that it could answer a question like that and his answer was massively difficult it would
  210. not be something that would be easy to accomplish any any time in the near in the near future I
  211. think that's probably true I'm not an expert in that area but he obviously is but when I think about what would be
  212. required but what's required to do the sorts of things that super forecasters collectively do
  213. the amount of guesswork but the amount of informed guess work that goes into constructing a forecast a reasonable
  214. forecast it's difficult for me to imagine existing AI artificial intelligence systems doing that in the
  215. near term so now if reading the book I mean most people luckily will probably not be asked to answer big questions
  216. about Iraq big questions about Korea or any some of the other things that you talk about in the book but if someone is
  217. reading the book and just to become a better forecaster about their daily lives I mean what do you hope that
  218. people take away from that to kind of apply it to the everyday to think they're going through whether it's jobs
  219. relationships or even rain on Sunday right right well I think a lot of people spend quite a bit of money on
  220. advice about the future that probably isn't worth the amount of money they're spending on it and they don't really
  221. know they have no way of knowing that because they have no way of knowing the track record to the people whose advice
  222. they're seeking the most the most the best example that is probably in the domain of Finance where a lot of money
  223. changes hands it's directed to people who claim to have some ability to predict the course of financial markets
  224. that is an extraordinarily difficult thing to do I'm not saying it's impossible or that nobody can do it with
  225. any better than the dark throwing chimp but it's a very difficult thing to do so
  226. I think if people were more skeptical about the people to whom they turn for advice about possible futures I think
  227. finance would be a case in point but I think more generally they should be very skeptical of the pundits they read and
  228. the claims that politicians and other people make about the future as well it's very common for people to make bold
  229. claims about the future and offer no evidence for their track records I would say it's almost universal
  230. [Music] and so I guess if someone's making a bold claim is that where we should is
  231. that the point where we should become suspicious I guess well the bolder the claim the more the burden of proof
  232. should fall on the person to demonstrate that he or she has a good track record and it seems to me like it's often more
  233. the bolder the claim the less likely someone's gonna question that person sometimes well that's a great point
  234. that's a point of a human psychology as we take our cues about whether somebody knows what he or she is talking about
  235. from how confident he or she seems to be and the more confident the more likely you're going to be able to blunderbuss
  236. your way through the conversation so that's that's a that's a problem and it suggests that people need to think a
  237. little bit more carefully when they make appraisals of competence and not rely as
  238. quite as heavily as they do and what we call the confidence heuristic it is it is true that confidence is is somewhat
  239. correlated with accuracy but it's it's also possible for manipulative human beings to use that heuristic and turn us
  240. into money pumps well thank you so much for being here we appreciate it okay thank you
  241. [Music] you

Brief Summary

Philip Tetlock discusses effective forecasting techniques and the characteristics of successful forecasters from his book "Superforecasting."

Key points

  • Dart-Throwing Chimp Study. Tetlock discusses a study involving a dart-throwing chimp that illustrates the challenges of forecasting accuracy.
  • The Art of Forecasting. Wharton Professor Philip Tetlock explores what makes a good forecaster and how to improve your own forecasting skills.
  • Foxes vs. Hedgehogs. The metaphor of foxes and hedgehogs highlights different forecasting styles and their effectiveness.
  • Tom Friedman's Questions on Iraq. Friedman posed a pivotal question about the essence of war in Iraq.
  • Skepticism Towards Financial Predictions. People should be cautious about whom they trust for future advice, especially in finance.

Tension

40/ 100Low tension

Related episodes