Superforecaster Full Video

Knowledge at Wharton

Ask about Knowledge at Wharton

Read the full transcriptHide the full transcript
  1. I first got introduced to the forecasting by a long time at Dartmouth Ben Valentino who is a
  2. political scientist and he alerted me to an opportunity to forecast with the good
  3. judgment project and this was right around the time when I was leaving Baghdad Iraq after my second tour there
  4. and I was looking for some way of doing forecasting because I had been something
  5. that I've been interested in for for a while and that's why been introduced me to this idea and you know I the timing
  6. worked out as well I was able to come back to the United States and that was right around the time the the first year
  7. of the tournament kicked off and I just went from there well in my work on security a big part of the question is
  8. whose security might be in danger in the coming years so secure this field of security studies inherently is about
  9. forecasting making predictions about how actors might be vulnerable which strategies might help them and so forth
  10. so I got interested in the good judgment project because I wanted to build these
  11. skills I've always had the interest and when I first started my undergrad I started a bunch of things I was one of
  12. these constantly changing your majors people and I changed between applied physics is where I started and then I
  13. went to International Studies but then somehow transitioned to anthropology and archaeology but I always kept the
  14. interest up in International Studies so it kind of went from wanting to study current culture as Moe was going on to
  15. studying past cultures and what was going on there and that the constant you know urge to learn more about all
  16. societies kept me interested in forecasting in what was going on in different countries and as I said well
  17. switching all the time I went to quite a few different universities in quite a few different countries I was at school
  18. in the Stars beginning Utah followed by Vermont and then Norway then back to Utah and I finished my doctorate in
  19. Scotland where I was there for seven years and then I went to Ireland for ten years so you know constantly moving in
  20. different cultures different societies different political system it was always something of great
  21. interest to me my professional life has involved forecasting because that's what
  22. I do or have done to help other people make better decisions hopefully but several years ago I attended a Penn
  23. engaging Minds event in New York City and Phil tetlock and Bob MELAS were speaking about the good judgment project
  24. which was very intriguing to me and so I inquired about joining as a forecaster and several months later when positions
  25. opened up I became a forecaster I think I am one of those people who can look at
  26. things a lot of different ways and I'm not the most adept at it among the forecasters but that's definitely my
  27. type and so I was a good fit but I was also driven because I'm acutely aware of the trouble the world is in and I think
  28. the the as much as they have a dark side us organizations like I are and the intelligence community in general that
  29. need all the help they can get we are basically the good guys and so so there was plenty of you know driving force
  30. behind my interest that goes back to when I was a younger man during the first Gulf War in 1991 I was I'm
  31. interested in the military military history and a lot of people and pundits were making predictions about the
  32. outcome the manner in which the war would be waged and decided as well as casualty counts and how long it would
  33. take and basically everyone was wrong on the duration and the and the destructiveness well the cat and the
  34. number of casualties suffered by coalition forces and that was interesting to me because there was one
  35. historian I'd like to say trevor Dupree you got it right and I was interested in how he got it right like
  36. and evidently he had a system in just a few years later during the 1994 midterm elections when Bill Clinton lost in a
  37. landslide to the Republicans the Republicans took the Congress for the first time in 40 years after that defeat
  38. basically every pundit on Earth was saying that Bill Clinton is a one-term president and being a staunch Democrat
  39. and a Clinton supporter I was traumatized by that possibility and wondered whether they knew they whether
  40. that they knew what they were talking about and at about that time I found a book written by another historian Alan
  41. Jay Lippman um about predicting you know who would become the next president and evidently
  42. he had a system where you didn't need to use polls you just look at history and using his method I realized that
  43. actually Bill Clinton was in a pretty good shape his his chances weren't guaranteed but you know even a year
  44. before the election I had a pretty good inkling that he had a good chance of winning re-election and this was against
  45. what all the pundits were saying so based on those two events there were probably other events as well that's why
  46. I became interested in forecasting basically the fact that everyone you know got thing big things wrong and I
  47. want to know why and some people got big things right and I wanted to know why and whether I could do as well as I
  48. could so ever since then I was I started you know I'm my own forecasting you know the
  49. outcomes of Wars and elections with mixed results but you know so but over 20 years so that's how that's how I
  50. became interested I'm politically very interested in what's happening and I used to read the
  51. 538 blog that was done by nate silver he advertised for the good judgment project
  52. i don't know where i saw the notice that you guys were putting together this program it was on one of the standard
  53. outlets maybe Wall Street Journal maybe Bloomberg but I read an article that you were
  54. putting together this forecasting tournament and that you were trying to find out what it is makes a good
  55. forecaster and I feel like I owe the system for so many years I've been doing research and I've relied on other people
  56. to contribute to that research and they have selflessly and so I'm a chronic volunteer as well when somebody says I
  57. need your help to do my research I volunteer for instance the New York Blood Center contacted me and said
  58. you're not allowed to give blood but we've got this research program and blood donation will you come in and
  59. donate a pint of blood every month so we can see what happens so I did that just
  60. to pay back the system so I'm paying back the system by being in the good judgment project
  61. I think that intellectual curiosity and just a willingness to put in the work and to
  62. think hard about specific problems and try to be as objective as possible is what allowed me to make more accurate
  63. forecasts and others within the pool I wouldn't necessarily say that you know what we did or what I did as an
  64. individual is the best possible at forecasting because they're you know can be so many more improvements I I think
  65. in terms of how much you know about a specific subject you can always improve your level of knowledge and I think you
  66. can always improve you know how you're you're specifically viewing a situation how that situation could potentially
  67. evolve so having a greater understanding of how humans interact with each other relate to each other the political
  68. contours of specific situations and events those are all things that you can gain greater knowledge on and get better
  69. at forecasting in so yeah yeah being open-minded and being open-minded enough to realize that you need to remain
  70. open-minded I think are some of the keys to being you know for for my own forecasting performance well I think
  71. there were a couple of factors one was that I am in the middle of my career now and I've spent a lot of time studying
  72. exactly the kind of issues that the good judgment project was focusing on and so
  73. I was familiar with trade issues I was familiar with national security issues the rise and fall of governments that
  74. kind of thing so it really was right in my issue area also I I use a theory that
  75. can be well applied in this kind of situation it's called structural realist theory and it gives a sense of big
  76. recurring outcomes and international relations the rise and fall of great powers the tendency for military
  77. conflict to occur competition and trade and so forth and so I had a kind of broad understanding of international
  78. affairs that I could apply to particular situations and the beauty of this theory
  79. was that it was it's really pre open states have a lot of choices about how they act and so I wasn't falling
  80. into the trap of thinking that something was inevitable but instead was investigating which things were more
  81. likely which things were less likely so this theory was really conducive to probabilistic forecasting and my
  82. background knowledge was really helpful in having a big set of information to draw from and also knowledge about where
  83. to find things where to find information that would be helpful in what I do for my day job in maritime archaeology it's
  84. a very similar thing we take the outside view we have a very disparate information we have maybe a few
  85. artifacts maybe not even maybe some historical sources maybe someone stumbled upon something and just have
  86. rumors historical studies ethnographic studies and we have to take all this information and try to piece it together
  87. to come up with a solution to or explanation of what we're finding and it's very similar in forecasting I kind
  88. of approach it the same way where I'll take very disparate information I won't take the big huge pictures that are
  89. thrust in your face I'll take you outside of use and use some of the same tools that I use at work I'm a huge
  90. proponent of using GIS mapping of different issues to try to to get like say it's a question that has a
  91. geographical perspective I'll make my own GIS program and basically you know put a model together that has all the
  92. contributory information from different press sources different mapping where different roads are communication routes
  93. transportation routes and it's the same kind of thing I would do in archaeology I just apply it to the future and
  94. forecasting as opposed to applying it to the past I don't trust my gut so usually
  95. my first impression is right about half the time so in order to do well in a forecasting tournament you have to dig
  96. deeper into the sources if you're predicting elections for example you know I'm find out you know what happened
  97. in the past you know which parties were likely to win and under what circumstances
  98. do your research about the electoral system sometimes you know the electoral system could lead to a counterintuitive
  99. result you know a party could do very well in the polls and still lose the election so it's just really a matter of
  100. doing your homework trying to understand the situation the history behind the event and and really working hard at it
  101. and I would say a second reason would be my sense of history so I think that comes in very handy because most people
  102. when they would forecast whatever event it is would really look to the recent past draw on their own memories as well
  103. as maybe you know a Google search of you know the recent news events but I think
  104. having a sense of history that stretches back far can give one a sense of mmm possibilities which you know the recent
  105. past might not indicate I think that I make connections for instance I couldn't figure out why Putin was in Syria none
  106. of the explanations that people gave the Talking Heads made any sense to me Syria does not a join Russia the Ukraine
  107. does Syria is of no strategic importance to Putin why is he in Syria there was just no
  108. reason all of this business about how he couldn't let one of his henchmen down or
  109. else all the other henchmen would feel insecure just didn't buy that I wanted to make sense of that and all of a
  110. sudden I remembered something and that is that Putin is building up the Russian fleet in the Mediterranean but they
  111. don't have a home port in the Mediterranean so they have to go all the way back to Russia and the Black Sea to
  112. get a home port well where is Putin landing his troops next to the port of Latakia which is a major port on the
  113. Mediterranean perhaps what Putin wants is a deal with Assad for a 50-year lease on the port of
  114. Latakia now that to me makes sense that would explain why Putin is in Syria it accomplishes a strategic thing for him
  115. it gives him a port for his Mediterranean fleet so I think that's probably part of the reason why I was
  116. successful I feel uncomfortable saying successful because I don't feel successful you tell me I was successful
  117. I don't feel successful that's part of the modus operandi that I go through it's like I see disparate things and I
  118. try to make sense of them for instance another thing when the third Kim took over North Korea all of a sudden there
  119. was a huge round of executions including his own uncle he blew his uncle up with
  120. it aircraft anti-aircraft gunner things were then fairly quiet and then another big round of executions I couldn't
  121. figure out why he was executing all these people I contacted somebody who was an expert in North Korean relations
  122. and I said why all the executions he said well they've been more than you know they just not publicized have been
  123. two big rounds I thought what's going on here and then I saw something that said
  124. Kim has not yet traveled outside of North Korea not even to Beijing to visit his sponsors we don't know when Kim's
  125. going to travel outside of North Korea all of a sudden you put the two together perhaps the guy's afraid that if he
  126. leaves the country he won't be able to come back that there'll be a coup in his absence and he's taking care of
  127. everybody who might be involved in such a coup and in fact he's now due to go to
  128. Moscow I think in December so again trying to make sense out of something just looking around at where how things
  129. might fit together to make things make more sense than they would otherwise the tournament taught me that and in a very
  130. humbling way that I knew a lot less about the world than I thought I knew and so that to begin with sparked my
  131. thirst for knowledge and information it also gave me a perspective about risk-taking because in my first year I
  132. was not a super forecaster I had to earn that status and I was involved in in a marketplace which involved essentially
  133. trading and I've learned a lot about risk-taking and how much I was either risk-averse or overly overly risky
  134. learning about things from a perspective of other information has been very important but once I became a super
  135. forecaster in particular what was helpful to me was a perspective of others the sharing of information and
  136. the ability to test your own beliefs and hypotheses for bias by taking into account what others have said or
  137. analysed as well as information that contradicts your own has been very valuable to me
  138. I was very steady I I answered as many questions as I could I was consistent I updated I didn't let
  139. anything get forgotten and I approached it like I was taking a test at school come in and maybe I didn't even study so
  140. but hey it's it's it's not an essay test so the choices are very clear it's it's
  141. it's it's numbers and and I would read the question carefully and analyze whether whether this would hinge on on a
  142. single factor or are many many factors if it was something that I had a clear gut feeling right off the bat then I
  143. would decide how much to back off from that extreme and if it was something where I was really undecided at the
  144. 50/50 mark I would force myself to a choice because 5050 is is a is never quite the right answer one of my hobbies
  145. is oil painting so and I and also when I've been taking the test they tell me I'm visuospatial
  146. so I I think that may have made a difference getting the big picture and knowing when to zoom in to the details
  147. you know backing off and then zooming in and not getting distracted by the wrong
  148. details so I've learned in painting that you have a big canvas you know you make
  149. broad strokes but somehow sometimes just the tiny nuance here are there changes the way the whole painting looks and I
  150. think that's true in world events too and why you need a lot of different perspectives so that I love my news
  151. sources I have my favorite ones but when you get a variety and when you hear feedback from other forecasters it
  152. really helps it's painfully clear that the more the wiser and more thoughtful and rational you are the better you're
  153. going to do and so I'm afraid you know that it's kind of the same people who would do better a lot of different
  154. things that require mental effort are the ones that rise to the top in forecasting
  155. I think what's special about forecasting is that if you're a really natural researcher that can get you quite far
  156. and I just completely admire and am grateful to those of us who are so good at it and who have you know far-flung
  157. connections to decision centres in the world it's I'm sort of a free rider in that way
  158. so there are two things that are really important and so I come from one side of that I'm a
  159. I'm a person who was a natural for computer programming for example but not a natural for writing or researching
  160. okay well along with not trusting my own gut feelings my first impressions I don't trust the gut feelings and of
  161. experts and pundits I try to when ever possible to discover understand what assumptions underlie their forecasts
  162. sometimes they're explicit about this and other times they are and they don't really they'll say you know this event
  163. is likely but not really explained and if I keep doing my research and find other pundits you say a certain event is
  164. likely without you know much explanation at all then that's a signal for me to really do some deep research and and try
  165. to even if I agree with that assumption you know sometimes I'll just try to play
  166. the devil's advocate and say well let's pretend that this silly outcome that's unlikely to happen you know actually
  167. could happen and kind of play that out and sometimes you know I'll go back to my original estimate
  168. yeah that's silly it's not gonna happen but you know often is the case that the silly outcome is becomes the more likely
  169. one so that's that's one thing I do and others I try to discover or to find out you know who has the power in a certain
  170. situation who who is the decider in the words of george w bush and that's one that keeps coming back all the time and
  171. also you know in every forecast you want to you know write down a list or in your
  172. head you know of all the constraints that would pretend prevent a certain event from occurring or slow it down you
  173. know make an event you know and a likely event take longer to occur than you know
  174. what was expected before so those are just a few things the first one was there there's almost no but I'm I tell
  175. you there's no nothing is certain that's not entirely sure there's I'm sure there's certain things that are served
  176. but for the most part nothing is certain the other thing was don't trust one source of information
  177. but then don't discard all sources just because you think it's not valid when you're whenever you're reading a source
  178. of information if you have just one single source and he's the only one saying it I didn't
  179. sometimes I wouldn't dismiss it but I would not trust it I'd look for something that would you know contribute
  180. to it or just help verify it and that can even be one of these sources where you know you see it on the internet you
  181. think wow that's really dubious these guys are you know pretty crazy but somewhere in there if you read through
  182. the whole thing there might be one sentence which collaborates what's going on in that other source that was a
  183. single source and I was like okay now I've got two independent sources so and you can start to build on that so one of
  184. the things I would ever say is don't rely on the one don't discard sources just because you may think they're a bit
  185. out there and there are quite a few sources that are out there I think that maybe the big thing is just to be aware
  186. of what you know what you can be certain of and what you're not sure of to know the difference between subjective and
  187. objective knowledge what things can be measured and what things are a matter of opinion look at history look at patterns
  188. look at base rates and then figure out whether your circumstance is similar or different
  189. I mean what's what was interesting for me is that most of the things I was forecasting on I knew nothing about
  190. going in finance questions for instance I I never pay attention to finance but when you're forecasting on world
  191. currencies you can always find graphs historical charts and I look at them and I see the patterns and I say why is it
  192. different now you know are we operating between the same extremes or will it be new and usually I've gone with stay with
  193. that stay with the trends so I didn't let myself get intimidated by any questions and I tried everything and as
  194. bill mentioned some of the things where I wellactually was most hesitant on topics
  195. that I might have knowledge on say on Ebola because suddenly I dunno I'm actually
  196. more self-conscious about things where I might miss something that I should know
  197. one easy to follow tip is do it every day and every question that you're forecasting have a
  198. look every day to see if anything has changed you what point things do change it significantly tends to come as a
  199. surprise and if you missed that day well you blew that forecast for that day the
  200. my approach is to try to think through causal mechanisms and and try to think of all the influential factors that I
  201. can think of and and try to visualize scenarios and and balance probabilities among scenarios things like that this is
  202. a little different than what I think most of us are led to do by the training we've had which is it's not that the
  203. training doesn't cover probability and that's critical but it's to look at the categories of the questions we're
  204. forecasting and try to find parallels in the past and look for rates of occurrence and try to use those rates I
  205. I think that's helpful as a starting place but it's not going to get you a very good score if that's all you do and
  206. I think the difference between you know sort of phoning it in and and getting a good score is understanding the
  207. situation as well as you possibly can and that's the what Phil has Phil tetlock has called the inside view as
  208. opposed to the outside view he emphasizes the people neglect the outside view but I think it's the inside
  209. view that that really matters to get you exceptional forecast that that actually
  210. could add to give real value anyway that's the inside view well the first thing was to really explore my biases to
  211. realize that I often would have a first impulse about a question and I didn't want to just go with that
  212. sometimes it was right but I didn't want to just go on impulse and secondly I really learned to think about what my
  213. strengths were to focus on questions that I thought were in my area of strength and in particular to focus on
  214. questions that I thought were were being answered wrong or you know where the the
  215. general consensus was different from what I thought it was because there I could make the most difference in
  216. shifting opinion or contributing knowledge that might sway the crowd and make the group as a whole more accurate
  217. so I think those things were really important a good friend of mine also named Barbara Barbara McClintock advised
  218. me over and over in our conversations that I should always question my fundamental assumptions my underlying
  219. assumptions and I think that's something that's been very important to me was Barbara's advice for instance a
  220. spacecraft just flew past Jupiter and took a whole bunch of high-resolution pictures of Jupiter I'm sorry Pluto it
  221. took a whole bunch of high-resolution pictures of Pluto and son of a gun this 4.5 billion year old planet is
  222. geologically active and everybody is absolutely floored the solar system is 4.5 billion years old the planets are
  223. not supposed to be geologically active when they're the size of Pluto why is this so everybody's trying to figure out
  224. why 4.5 billion year old Pluto is by geologically active they're talking about more missions to Pluto half
  225. hundred million dollar missions half billion dollar missions I'm certain that there are all sorts of young planetary
  226. scientists who are trying to figure out how they can start their careers by figuring out and making a name for
  227. themselves Pluto is geologically active well what's the underlying assumption here the
  228. underlying assumption is that Pluto's 4.5 billion years old what happens if Pluto isn't 4.5 billion
  229. years old what happens with Pluto's a good deal younger then in fact you expect it to be geologically active and
  230. there's no inconsistency here at all the inconsistency is that your fundamental assumption that Pluto is 4.5 billion
  231. years old was incorrect so I'd advise those young planetary scientists who are thinking about trying to explain how a
  232. 4.5 billion year old celestial body could be geologically active to question whether or not it's really 4.5 billion
  233. years old there's another something else that gets a lot more attention and that's it ran for the last four to five
  234. years the entire economy of the world has been an uproar because all of the major nations have slapped an embargo on
  235. dealing with Iran because Iran is trying to build a nuclear weapon and they had to negotiate long and hard with Iran to
  236. negotiate Iran away from a posture where they could be capable of building the nuclear weapon that they were trying to
  237. build and this brought us a lot closer to allies like Saudi Arabia to whom we perhaps should not be close and it's
  238. caused no end to trouble throughout the entire world particularly the Middle East which is troubled enough well
  239. what's the unda been underlying assumption here that Iran wants to build a nuclear weapon they kept saying that
  240. they didn't want to build a nuclear weapon why didn't we believe them well we have
  241. x y&z reasons why we didn't believe them but again it's a fundamental assumption
  242. the entire foreign policy in the Middle East of nearly the entire world was dictated by the underlying assumption
  243. that Iran wanted to build a nuclear device I personally wouldn't want to build a nuclear device I can't see why
  244. Iran would want a nuclear device nuclear bombs are so 20th century we've moved on I happen to think Iran
  245. didn't particularly want to build a nuclear bomb but everybody's fundamental assumption
  246. was that it did and I would go out on a limb and I'd say that even in spite of all the acrimony that we've had over
  247. Iran that it's quite possible that in 20 years Iran will be our closest ally in the Middle East and the reason for that
  248. is that Iran is the largest Persian nation on earth the United States is the second-largest Persian nation on earth
  249. and Canada is the fourth largest Persian nation on earth there's a natural affinity there and as soon as that
  250. embargo is lifted the Persians are not going to start doing deals with the British or the Chinese or the Russians
  251. they're going to do business with their Persian American friends so I see business between Iran Persia and the
  252. United States particularly the Persian community building very quickly and I think that could lead to good political
  253. relations as well that would be a major casualty of misunderstanding Iran's intentions with nuclear weapons the
  254. process evolved over time for me and one of the things that is important to me is
  255. to use historical information but not to be over alliant on it if there is the ability to compare against similar
  256. situations in the past I'm very cognizant of using that information but only to the extent that it is truly
  257. relevant to the current circumstances beyond that frequently adjusting my own forecasts based upon new information and
  258. new perspectives of others is extremely valuable it's very easy to take a position and sit back and say that's it
  259. however the real world is a very active one and often things change and forecasts should change based upon the
  260. events and the happenings of the world and I found it's extremely useful to pay a lot of
  261. attention to those events and frequently re-examine my own forecasts and see if they need to be updated so I really had
  262. three general rules when I made forecasts especially when I made my first forecast on a new question or a
  263. situation I'd never seen before so the first thing I would say is to be a good forecasters
  264. you have to be you have to hold strong opinions weekly and what I mean by that is that you do all of your homework
  265. upfront and you really present a good case for why you feel a certain way on a specific question and defending your
  266. probability probability estimate at that point but when you recognize that somebody else has more information or
  267. better information than you having the ability to change your mind swiftly to that new forecast I think that was one
  268. of the keys to why I was able to you know improve my performance over time I think many people really get stuck on
  269. their initial forecasts and they fall in love with it and wind up becoming going
  270. into a defensive Crouch when it comes to defending their opinion whereas I was on
  271. my team sort of notorious for being able to switch from one very hard position all the way to the opposite when I saw
  272. new information and come up that led me to do that the second thing is to be willing to put
  273. in the homework and to put in the effort to understand a situation go beyond the
  274. Wikipedia page and to go beyond the initial couple of news articles and to really try to understand you know what
  275. is it what is it about Boko Haram and their composition that you know is allowing them to make the kinds of
  276. military gains that their that we're seeing and how fragile or stable are those military gains compared to other
  277. like organizations and to you know not just to say Oh Boko Haram looks like every other
  278. organization that I've encountered to this point but to say to really try to understand what's going on there and to
  279. contextualize that appropriately and I think the the third thing is to have in your mind the kinds of psychological and
  280. cognitive biases that you could potentially fall prey to and to be able to overcome those by you know listening
  281. to your teammates by self critique and to recognize that you know whatever forecasts and argument that you're
  282. putting forward now you should be able to alter it or change it or improve it in light of new information so you know
  283. keeping in mind the cognitive biases that really impact and influence and reduce our accuracy and to consciously
  284. try to devise them I think the tournament was great you need to practice so you need to have
  285. a lot of questions you need to have feedback you need to know where where you're good and where you're messing up
  286. if you have a team if you have other people that you're in tournament either in competition or in collaboration
  287. either way you listen to them you get feedback and you will you will improve I think that one of the things that would
  288. be extremely useful would be to participate in good judgments open forecasting competition because you can
  289. test your own abilities and learn about the things that that both limit your ability to forecast and make you a
  290. better forecaster and through that process become somebody who is more adept at forecasting the good judgment
  291. incorporated itself has put together various training methods which have been very useful to me as a super forecaster
  292. and I'm sure to others but as an individual just forecasting and and using that forecasting experience can be
  293. a very valuable thing to learn about yourself and the process of forecasting what and and what you do well and what
  294. you do less well and then to seek ways of improving upon those weaknesses I think you could really do two things to
  295. become a good forecaster and one one is to read and not just read say Phil's book super forecasting or Danny
  296. Kahneman's book Thinking Fast and Slow which I think many of the forecasters who excelled in torment have read and
  297. done I also think that you know reading say histories like the guns of August or
  298. books about bureaucracy by James Q Wilson and others really improves your your knowledge base
  299. and gives you a better context for looking at situations and being able to figure out whether there are any
  300. commonalities or patterns there that you can pick up on and that could potentially apply to future situations
  301. that look like that so being well-read is I think a contributor to forecasting success the second thing is to just to
  302. practice don't get hung up on wrong forecasts that you make initially you know don't don't get overly you know
  303. sensitive to those results but to approach forecasting as something that you can improve on and have that growth
  304. mindset when it comes to making better forecasts and so from there you're able to say okay I start at some baseline
  305. level I got some feedback tells me I should improve in X Y & Z and to really make a concerted effort to do that
  306. through practice well one of the things I would recommend would be to write down
  307. a list of all the things that would that you should be on the lookout for that might cause you to change your mind so
  308. after you've done your homework you've thought deeply about you know the event and the likelihood of of it occurring
  309. write down a list of of things that you know would happen leading up to the event or signals markers whatever that
  310. would cause you to change your mind and and the reason why I say write it down is because it's because you know it's
  311. very hard to change your mind and if you write it down it might be help you you know not try to to fudge things well you
  312. know if it's if it's just in the back of your head you know you mM it becomes a little bit more difficult to to go
  313. against you know your original original conclusion because I mean you put a lot of work into this and you don't want to
  314. go back on it and there's an actual tendency not to want to you know waffle or be too wishy-washy
  315. but you know having it down on paper will just show you well I'm not being wishy-washy you know events have changed
  316. things have changed and maybe my initial conclusion was wrong I would say the third thing would be to play devil's
  317. advocate after immediately after you've you know come to a conclusion and made a forecast
  318. seek out information that would prove you wrong you know if anything you know just to cover all your bases
  319. I'd say my fourth piece of advice would be to seek a wide variety of sources even weak and biased sources because
  320. sometimes you can find interest in information in very flawed sources but that'll take some work on your part to
  321. determine you know which part of these you know particular reports are worth you know holding on to and which part
  322. should be discarded it's not an easy task but don't just you know casually disregard even even bad sources because
  323. you never know from four years as a as a forecaster in the gjp tournament I can say that another piece of advice I would
  324. give is to assume that things would take longer than your initial impression that's a rule of thumb because it
  325. happens consistently time and time again that whatever it is a diplomatic initiative or or or what-have-you
  326. events often will take a lot longer than then what what you think you know they will take or what
  327. officials you know the timetable of of officials and and that's just been my experience
  328. from doing this for a long time final piece of advice would be embrace uncertainty don't get too hung up on
  329. getting things correct you're never gonna get everything correct you're going to get a lot of things wrong I
  330. would focus more on assigning viable or realistic probabilities to events just think of it in terms of you know if I
  331. assign a 70% probability to a particular event you know I'm gonna be wrong three out of ten times you know four so
  332. if you if you work on being well calibrated instead of trying to get everything right I think that's probably
  333. a more fruitful endeavor well every year in the tournament I kept a running list
  334. of what questions I'd answered when I first answered them how I Hearst answered them and it was really helpful
  335. to see how my opinion changed as new information came along so I was you know kind of keeping myself honest about what
  336. was my original opinion and what was making me change my mind and I think that was really helpful because I
  337. learned a lot about what kind of pits I might fall into you know if I get over optimistic about suddenly there's a lot
  338. of news on a certain issue I might have a tendency to think that something was more likely to happen whereas maybe
  339. wasn't really more likely to happen so keeping track of my predictions was very helpful I thought it was really
  340. important that I was at forecasting anonymously at first it made it more I made me more confident about making
  341. predictions without worrying about their effect on my reputation and after all you know I was in this to get better
  342. about forecasting so I wanted to have the ability to take to make bets or you know answer questions in a way that I
  343. thought was correct without worrying about how that might be seen somewhere else and that really built my confidence
  344. and actually through being right then frequently too it built up my confidence to the point that I was willing to
  345. who I was and then got to experience the whole other dynamic of people knowing I
  346. was an expert wondering what I thought about things and then having to be cautious about not relying too much on
  347. my expert knowledge but still thinking about the wisdom of the crowd so I think the anonymity is really important and
  348. also realizing that the magic of this whole study is in revealing that everybody has important information and
  349. at different times different people's information will be really valid and especially important and and finding
  350. that right balance between being confident in your own views and and being open to the views of others become
  351. comfortable with researching problems have a great interest in what you're forecasting don't hesitate to dig right
  352. in to you know study something new just get into it don't don't neglect if you have any bent
  353. at all for statistics and probability bring it into play you know but that's kind of obvious anyone who has that
  354. choice I guess actually that the this might come up in your last question I think you forget but part of the muscle
  355. you want is the muscle you want to strengthen intuitively is the muscle for balancing scenarios against each other
  356. and sort of averaging out probabilities intuitively weighting them properly things like that balancing things
  357. against each other intuitively that are things that are pointing in opposite directions this is this is something you
  358. get better at and feel easier about by practicing I recommend that when you come to a new field about which you know
  359. nothing that you browse the literature talk to people if you can attend lectures if you can and listen for the
  360. totem words the words which either consciously or unconsciously people in the field
  361. give special meeting and I find that if you can figure out what those totem words are and then go away and figure
  362. out what they mean you'll find out at least one of the things that's very important to people in the field that
  363. they don't say is important to them necessarily but they give it importance by continually using these totem words
  364. or continually emphasizing them for instance last week I went to Grand Rounds at our Regional Hospital and
  365. there was a cardiologist giving a talk and he was giving a talk on the pharmaceutical management of cardiac
  366. patients who had circulatory problems problems with blockages problems with leakages he would go through a series of
  367. example patients this patient comes in with these symptoms what do you do we immediately stent this patient how do
  368. we drug this patient for the stent we're gonna use this compound and we're gonna
  369. use this compound but we're gonna stay away from that compound because it can cause bleeding and we don't want
  370. bleeding now this patient has got a partially occluded artery and is in a lot of pain we're going to give this
  371. patient warfarin because we need to dissolve that clot and prevent it from growing however we're not going to give
  372. this patient these other two drugs because they can lead to bleeding even worse than warfarin can lead to bleeding
  373. we give combinations of these drugs because it reduces bleeding bleeding this bleeding that what I came away from
  374. not knowing really that very much about how to treat cardiac patients was that if we can generalize from just this one
  375. talk what's most important is to keep the patient from bleeding no matter what you do and so if I was given a new drug
  376. and asked to predict whether or not it was going to be a success the first thing I'd look at its whether it causes
  377. uncontrolled bleeding or not what the likelihood is how it can be combined with other things all from a standpoint
  378. of does it cause bleeding now that doesn't give you the answer to how that new drug is going to do but it gives you
  379. a lead-in to it so that you can then begin to understand how things are going on
  380. so I'd say look for the totem words when you're reading an article look for the words that the author of the article
  381. seems to come back to over and over and then start by figuring out what those words mean trust yourself
  382. but doubt yourself as well and always be prepared to be devil's advocate to yourself if you're working within a team
  383. that's even better because most forecasters do not have a problem being devil's advocate or would call red
  384. teaming and keep an open mind don't go into something thinking you know exactly what's going on it never seems to work
  385. out that way just when you think you know exactly what's going on it changes no the main thing is just you know be
  386. open to change in dr. tetlock spooky mentions that we're in perpetual beta that is a great way of thinking of it is
  387. always be willing to change and learn and just you know go with it a lot of people get really kind of stuck in the
  388. mud on certain ideas and that's that's the quickest way to get it wrong I've been very insecure all my life for
  389. a variety of reasons and I never really thought very much of my own opinions I always figured well is the world crazy
  390. or is it me and the answer was I must be crazy I think the good judgment project has
  391. shown me that occasionally I'm right and other people are wrong which is not something that I expected going in I
  392. expected to be wrong all the time and curiously and I've seen this in Phil's book and I've heard it from other people
  393. the things about which I was most wrong with the things I was supposed to know something about namely the medical
  394. questions so I think the good judgment project has given me a little bit of confidence in myself that I didn't have
  395. before I went into it it's been huge and for first off you get exposed to a lot of wonderful people whom you tend to
  396. maintain some degree of contact with I've had a couple of of my teammates visit me at my house one who's from
  397. South Africa the other ones from Spain it's true they were on Washington for other reasons but that's an example and
  398. I I'm glad to stay in touch with a lot of the fork in fact there there have been others as well from the DC area
  399. anyways I'm glad to stay in touch with these new friends that's that's see that's a big thing and just as big is
  400. there's nothing like reality focusing on reality and how it works out and you tried to see the future and maybe it
  401. worked and maybe it didn't and you you sort of mentally record it's there's feedback going on that's it's extremely
  402. good feedback for turning you into a more rational person it's it makes you a more balanced person a more rational
  403. person easier to live with perhaps I don't know but it's all good it's been very rewarding and this
  404. doesn't mean that the forecaster shouldn't also be rewarded in other ways but this myself is rewarding and I'm and
  405. I'm sure that the others feel much much the same I think being a forecaster and being a researcher are similar and
  406. different I think that one of the great things about being a forecaster is that we really weren't constrained by any of
  407. the kind of customs or sociology around forecasting because we were the ones coming up with that and so we were able
  408. to explore and be very free in how we approach forecasting whereas I think research while you have some freedom and
  409. hypotheses that you are tackling a lot of the methodology and the structure around writing papers and peer review
  410. and submissions is very much rigid and you must follow the the specific process so I think that that highlights a major
  411. difference in forecasting versus doing the research and that the research seem to be much more process oriented
  412. although obviously there there is an outcome component to it whereas when when we were doing strictly forecasting
  413. really people only cared that we were able to come up with the right answers rather than you know how exactly we we
  414. got there so yeah they're different they're different animals I use different words when I talk to myself
  415. now I've always been very analytic but now I I start talking those percentages to myself and and that's different
  416. my wife and I have two different philosophies of investment she has been involved over the years as a partner and
  417. an investment firm that does alternative investments which are allegedly less risky but by my
  418. lights carry a risk that is sometimes more than what I like I on the other hand am much more risk averse and
  419. although less risk-averse then I was when I first started participating a good generative project my professional
  420. career has been oriented towards reducing the risk that decision makers have and perhaps I've taken that point
  421. of view in my own approach to investing which my wife feels is overly conservative I think that that the
  422. forecast thing as part of the good judgment project has given me a window into my own risk-taking perspective and
  423. move me at least somewhat closer to my wife's point of view and and I think that that's a very useful thing in real
  424. life as well as you know from the standpoint of participating as a super forecasting so in my work
  425. this experience has been extremely important I am now working on an article about the rise of China that I would not
  426. have written if it had not been from my experience in the good judgment project by having to constantly update my
  427. knowledge about China's economic capabilities and its military capabilities in order to do well in this
  428. game I kept up in a much finer green level with things that were happening in China and that have changed my forecast
  429. about how international relations will work over the next couple of decades so I'm an international relations scholar
  430. one of the most important things in the international relations field right now is this question of of does the United
  431. States have peer competitors is China one yet and this experience helped me see that it is so it's really
  432. dramatically changed my career and the process of going through the confidence-building process
  433. of you know learning how to articulate forecasts learning how to be how to express them in probabilistic terms how
  434. to be confident even if I'm not completely sure has made me willing to take the risk of going out on this
  435. really controversial subject I'm not sure that they have all they probably should having practice at forecasting
  436. geopolitical events maybe I should start practicing forecasting events related to
  437. my own personal life maybe that would help me avoid being unpleasantly surprised by my things but so I'm not
  438. really sure how it's affected my personal life other than having nice dinnertime conversations with families
  439. and friends like I said I go back and forth between the different you know venues of inquiry in archaeology and in
  440. forecasting so there has been times where one of the things that forecasting told me is it's
  441. great if you have a model but all they're going to be a better model always be a better model you might have
  442. one that's showing you exactly what you want but you know if you tweak it a little bit or you say you find a new
  443. dataset can always be that little bit better and I've started to apply that to some shipwreck hunts that I've been
  444. doing where we have a model on how to locate shipwrecks in areas of high probability but it can always be better
  445. so I've actually started tweaking some of the models that I use at work to add additional data sets maybe pulling
  446. things from multidisciplinary studies from geomorphology marine geomorphology riverbed hydrological processes going
  447. into shipwreck site formation and just basically you know constantly tweaking the models there's something the other
  448. thing that I've tried to do as much as I can is I have two young sons who I'm trying to teach them how to approach
  449. different issues you know not not so much to get him in trouble at school for questioning the teacher all the time
  450. which is very possible but when they answer a question say well is that it I spoke to my son about a tree growing in
  451. a field I said well how much do you think that tree is gonna grow next year and he just said oh well this I said
  452. well I should say that he's like well because it just looks like a well did you think about how much was it gonna
  453. rain next year how much fertilizer is the farmer going to put in that field without trees in the middle how often is
  454. he going to plow it what you know there's so many issues that surround one issue that when you hear
  455. one question there's usually you know 20 30 however many more questions that can
  456. help you answer that question so I've been trying to push it on them to kind of think that way
  457. you

Brief Summary

Ben Valentino discusses his journey in forecasting and the importance of historical context and collaboration in improving predictions.

Key points

  • The Journey to Forecasting. From military tours to academia, the speaker's path to forecasting is shaped by diverse experiences.
  • The Role of Curiosity. Intellectual curiosity and willingness to learn are key to making accurate forecasts.
  • The Influence of History. Understanding history is crucial for accurate forecasting, as it reveals patterns and possibilities.
  • The Importance of Multiple Sources. Relying on a single source can lead to misinformation. Always seek multiple perspectives.
  • Understanding Uncertainty in Forecasting. Embracing uncertainty is crucial for accurate forecasting. Focus on assigning realistic probabilities.
  • Learning from Predictions. Tracking your predictions can reveal biases and improve forecasting skills.

Episode highlights

Tension

45/ 100Some tension

Related episodes