Superforecaster Full Video
Ask about Knowledge at Wharton
Read the full transcriptHide the full transcript
- I first got introduced to the forecasting by a long time at Dartmouth Ben Valentino who is a
- political scientist and he alerted me to an opportunity to forecast with the good
- judgment project and this was right around the time when I was leaving Baghdad Iraq after my second tour there
- and I was looking for some way of doing forecasting because I had been something
- that I've been interested in for for a while and that's why been introduced me to this idea and you know I the timing
- worked out as well I was able to come back to the United States and that was right around the time the the first year
- of the tournament kicked off and I just went from there well in my work on security a big part of the question is
- whose security might be in danger in the coming years so secure this field of security studies inherently is about
- forecasting making predictions about how actors might be vulnerable which strategies might help them and so forth
- so I got interested in the good judgment project because I wanted to build these
- skills I've always had the interest and when I first started my undergrad I started a bunch of things I was one of
- these constantly changing your majors people and I changed between applied physics is where I started and then I
- went to International Studies but then somehow transitioned to anthropology and archaeology but I always kept the
- interest up in International Studies so it kind of went from wanting to study current culture as Moe was going on to
- studying past cultures and what was going on there and that the constant you know urge to learn more about all
- societies kept me interested in forecasting in what was going on in different countries and as I said well
- switching all the time I went to quite a few different universities in quite a few different countries I was at school
- in the Stars beginning Utah followed by Vermont and then Norway then back to Utah and I finished my doctorate in
- Scotland where I was there for seven years and then I went to Ireland for ten years so you know constantly moving in
- different cultures different societies different political system it was always something of great
- interest to me my professional life has involved forecasting because that's what
- I do or have done to help other people make better decisions hopefully but several years ago I attended a Penn
- engaging Minds event in New York City and Phil tetlock and Bob MELAS were speaking about the good judgment project
- which was very intriguing to me and so I inquired about joining as a forecaster and several months later when positions
- opened up I became a forecaster I think I am one of those people who can look at
- things a lot of different ways and I'm not the most adept at it among the forecasters but that's definitely my
- type and so I was a good fit but I was also driven because I'm acutely aware of the trouble the world is in and I think
- the the as much as they have a dark side us organizations like I are and the intelligence community in general that
- need all the help they can get we are basically the good guys and so so there was plenty of you know driving force
- behind my interest that goes back to when I was a younger man during the first Gulf War in 1991 I was I'm
- interested in the military military history and a lot of people and pundits were making predictions about the
- outcome the manner in which the war would be waged and decided as well as casualty counts and how long it would
- take and basically everyone was wrong on the duration and the and the destructiveness well the cat and the
- number of casualties suffered by coalition forces and that was interesting to me because there was one
- historian I'd like to say trevor Dupree you got it right and I was interested in how he got it right like
- and evidently he had a system in just a few years later during the 1994 midterm elections when Bill Clinton lost in a
- landslide to the Republicans the Republicans took the Congress for the first time in 40 years after that defeat
- basically every pundit on Earth was saying that Bill Clinton is a one-term president and being a staunch Democrat
- and a Clinton supporter I was traumatized by that possibility and wondered whether they knew they whether
- that they knew what they were talking about and at about that time I found a book written by another historian Alan
- Jay Lippman um about predicting you know who would become the next president and evidently
- he had a system where you didn't need to use polls you just look at history and using his method I realized that
- actually Bill Clinton was in a pretty good shape his his chances weren't guaranteed but you know even a year
- before the election I had a pretty good inkling that he had a good chance of winning re-election and this was against
- what all the pundits were saying so based on those two events there were probably other events as well that's why
- I became interested in forecasting basically the fact that everyone you know got thing big things wrong and I
- want to know why and some people got big things right and I wanted to know why and whether I could do as well as I
- could so ever since then I was I started you know I'm my own forecasting you know the
- outcomes of Wars and elections with mixed results but you know so but over 20 years so that's how that's how I
- became interested I'm politically very interested in what's happening and I used to read the
- 538 blog that was done by nate silver he advertised for the good judgment project
- i don't know where i saw the notice that you guys were putting together this program it was on one of the standard
- outlets maybe Wall Street Journal maybe Bloomberg but I read an article that you were
- putting together this forecasting tournament and that you were trying to find out what it is makes a good
- forecaster and I feel like I owe the system for so many years I've been doing research and I've relied on other people
- to contribute to that research and they have selflessly and so I'm a chronic volunteer as well when somebody says I
- need your help to do my research I volunteer for instance the New York Blood Center contacted me and said
- you're not allowed to give blood but we've got this research program and blood donation will you come in and
- donate a pint of blood every month so we can see what happens so I did that just
- to pay back the system so I'm paying back the system by being in the good judgment project
- I think that intellectual curiosity and just a willingness to put in the work and to
- think hard about specific problems and try to be as objective as possible is what allowed me to make more accurate
- forecasts and others within the pool I wouldn't necessarily say that you know what we did or what I did as an
- individual is the best possible at forecasting because they're you know can be so many more improvements I I think
- in terms of how much you know about a specific subject you can always improve your level of knowledge and I think you
- can always improve you know how you're you're specifically viewing a situation how that situation could potentially
- evolve so having a greater understanding of how humans interact with each other relate to each other the political
- contours of specific situations and events those are all things that you can gain greater knowledge on and get better
- at forecasting in so yeah yeah being open-minded and being open-minded enough to realize that you need to remain
- open-minded I think are some of the keys to being you know for for my own forecasting performance well I think
- there were a couple of factors one was that I am in the middle of my career now and I've spent a lot of time studying
- exactly the kind of issues that the good judgment project was focusing on and so
- I was familiar with trade issues I was familiar with national security issues the rise and fall of governments that
- kind of thing so it really was right in my issue area also I I use a theory that
- can be well applied in this kind of situation it's called structural realist theory and it gives a sense of big
- recurring outcomes and international relations the rise and fall of great powers the tendency for military
- conflict to occur competition and trade and so forth and so I had a kind of broad understanding of international
- affairs that I could apply to particular situations and the beauty of this theory
- was that it was it's really pre open states have a lot of choices about how they act and so I wasn't falling
- into the trap of thinking that something was inevitable but instead was investigating which things were more
- likely which things were less likely so this theory was really conducive to probabilistic forecasting and my
- background knowledge was really helpful in having a big set of information to draw from and also knowledge about where
- to find things where to find information that would be helpful in what I do for my day job in maritime archaeology it's
- a very similar thing we take the outside view we have a very disparate information we have maybe a few
- artifacts maybe not even maybe some historical sources maybe someone stumbled upon something and just have
- rumors historical studies ethnographic studies and we have to take all this information and try to piece it together
- to come up with a solution to or explanation of what we're finding and it's very similar in forecasting I kind
- of approach it the same way where I'll take very disparate information I won't take the big huge pictures that are
- thrust in your face I'll take you outside of use and use some of the same tools that I use at work I'm a huge
- proponent of using GIS mapping of different issues to try to to get like say it's a question that has a
- geographical perspective I'll make my own GIS program and basically you know put a model together that has all the
- contributory information from different press sources different mapping where different roads are communication routes
- transportation routes and it's the same kind of thing I would do in archaeology I just apply it to the future and
- forecasting as opposed to applying it to the past I don't trust my gut so usually
- my first impression is right about half the time so in order to do well in a forecasting tournament you have to dig
- deeper into the sources if you're predicting elections for example you know I'm find out you know what happened
- in the past you know which parties were likely to win and under what circumstances
- do your research about the electoral system sometimes you know the electoral system could lead to a counterintuitive
- result you know a party could do very well in the polls and still lose the election so it's just really a matter of
- doing your homework trying to understand the situation the history behind the event and and really working hard at it
- and I would say a second reason would be my sense of history so I think that comes in very handy because most people
- when they would forecast whatever event it is would really look to the recent past draw on their own memories as well
- as maybe you know a Google search of you know the recent news events but I think
- having a sense of history that stretches back far can give one a sense of mmm possibilities which you know the recent
- past might not indicate I think that I make connections for instance I couldn't figure out why Putin was in Syria none
- of the explanations that people gave the Talking Heads made any sense to me Syria does not a join Russia the Ukraine
- does Syria is of no strategic importance to Putin why is he in Syria there was just no
- reason all of this business about how he couldn't let one of his henchmen down or
- else all the other henchmen would feel insecure just didn't buy that I wanted to make sense of that and all of a
- sudden I remembered something and that is that Putin is building up the Russian fleet in the Mediterranean but they
- don't have a home port in the Mediterranean so they have to go all the way back to Russia and the Black Sea to
- get a home port well where is Putin landing his troops next to the port of Latakia which is a major port on the
- Mediterranean perhaps what Putin wants is a deal with Assad for a 50-year lease on the port of
- Latakia now that to me makes sense that would explain why Putin is in Syria it accomplishes a strategic thing for him
- it gives him a port for his Mediterranean fleet so I think that's probably part of the reason why I was
- successful I feel uncomfortable saying successful because I don't feel successful you tell me I was successful
- I don't feel successful that's part of the modus operandi that I go through it's like I see disparate things and I
- try to make sense of them for instance another thing when the third Kim took over North Korea all of a sudden there
- was a huge round of executions including his own uncle he blew his uncle up with
- it aircraft anti-aircraft gunner things were then fairly quiet and then another big round of executions I couldn't
- figure out why he was executing all these people I contacted somebody who was an expert in North Korean relations
- and I said why all the executions he said well they've been more than you know they just not publicized have been
- two big rounds I thought what's going on here and then I saw something that said
- Kim has not yet traveled outside of North Korea not even to Beijing to visit his sponsors we don't know when Kim's
- going to travel outside of North Korea all of a sudden you put the two together perhaps the guy's afraid that if he
- leaves the country he won't be able to come back that there'll be a coup in his absence and he's taking care of
- everybody who might be involved in such a coup and in fact he's now due to go to
- Moscow I think in December so again trying to make sense out of something just looking around at where how things
- might fit together to make things make more sense than they would otherwise the tournament taught me that and in a very
- humbling way that I knew a lot less about the world than I thought I knew and so that to begin with sparked my
- thirst for knowledge and information it also gave me a perspective about risk-taking because in my first year I
- was not a super forecaster I had to earn that status and I was involved in in a marketplace which involved essentially
- trading and I've learned a lot about risk-taking and how much I was either risk-averse or overly overly risky
- learning about things from a perspective of other information has been very important but once I became a super
- forecaster in particular what was helpful to me was a perspective of others the sharing of information and
- the ability to test your own beliefs and hypotheses for bias by taking into account what others have said or
- analysed as well as information that contradicts your own has been very valuable to me
- I was very steady I I answered as many questions as I could I was consistent I updated I didn't let
- anything get forgotten and I approached it like I was taking a test at school come in and maybe I didn't even study so
- but hey it's it's it's not an essay test so the choices are very clear it's it's
- it's it's numbers and and I would read the question carefully and analyze whether whether this would hinge on on a
- single factor or are many many factors if it was something that I had a clear gut feeling right off the bat then I
- would decide how much to back off from that extreme and if it was something where I was really undecided at the
- 50/50 mark I would force myself to a choice because 5050 is is a is never quite the right answer one of my hobbies
- is oil painting so and I and also when I've been taking the test they tell me I'm visuospatial
- so I I think that may have made a difference getting the big picture and knowing when to zoom in to the details
- you know backing off and then zooming in and not getting distracted by the wrong
- details so I've learned in painting that you have a big canvas you know you make
- broad strokes but somehow sometimes just the tiny nuance here are there changes the way the whole painting looks and I
- think that's true in world events too and why you need a lot of different perspectives so that I love my news
- sources I have my favorite ones but when you get a variety and when you hear feedback from other forecasters it
- really helps it's painfully clear that the more the wiser and more thoughtful and rational you are the better you're
- going to do and so I'm afraid you know that it's kind of the same people who would do better a lot of different
- things that require mental effort are the ones that rise to the top in forecasting
- I think what's special about forecasting is that if you're a really natural researcher that can get you quite far
- and I just completely admire and am grateful to those of us who are so good at it and who have you know far-flung
- connections to decision centres in the world it's I'm sort of a free rider in that way
- so there are two things that are really important and so I come from one side of that I'm a
- I'm a person who was a natural for computer programming for example but not a natural for writing or researching
- okay well along with not trusting my own gut feelings my first impressions I don't trust the gut feelings and of
- experts and pundits I try to when ever possible to discover understand what assumptions underlie their forecasts
- sometimes they're explicit about this and other times they are and they don't really they'll say you know this event
- is likely but not really explained and if I keep doing my research and find other pundits you say a certain event is
- likely without you know much explanation at all then that's a signal for me to really do some deep research and and try
- to even if I agree with that assumption you know sometimes I'll just try to play
- the devil's advocate and say well let's pretend that this silly outcome that's unlikely to happen you know actually
- could happen and kind of play that out and sometimes you know I'll go back to my original estimate
- yeah that's silly it's not gonna happen but you know often is the case that the silly outcome is becomes the more likely
- one so that's that's one thing I do and others I try to discover or to find out you know who has the power in a certain
- situation who who is the decider in the words of george w bush and that's one that keeps coming back all the time and
- also you know in every forecast you want to you know write down a list or in your
- head you know of all the constraints that would pretend prevent a certain event from occurring or slow it down you
- know make an event you know and a likely event take longer to occur than you know
- what was expected before so those are just a few things the first one was there there's almost no but I'm I tell
- you there's no nothing is certain that's not entirely sure there's I'm sure there's certain things that are served
- but for the most part nothing is certain the other thing was don't trust one source of information
- but then don't discard all sources just because you think it's not valid when you're whenever you're reading a source
- of information if you have just one single source and he's the only one saying it I didn't
- sometimes I wouldn't dismiss it but I would not trust it I'd look for something that would you know contribute
- to it or just help verify it and that can even be one of these sources where you know you see it on the internet you
- think wow that's really dubious these guys are you know pretty crazy but somewhere in there if you read through
- the whole thing there might be one sentence which collaborates what's going on in that other source that was a
- single source and I was like okay now I've got two independent sources so and you can start to build on that so one of
- the things I would ever say is don't rely on the one don't discard sources just because you may think they're a bit
- out there and there are quite a few sources that are out there I think that maybe the big thing is just to be aware
- of what you know what you can be certain of and what you're not sure of to know the difference between subjective and
- objective knowledge what things can be measured and what things are a matter of opinion look at history look at patterns
- look at base rates and then figure out whether your circumstance is similar or different
- I mean what's what was interesting for me is that most of the things I was forecasting on I knew nothing about
- going in finance questions for instance I I never pay attention to finance but when you're forecasting on world
- currencies you can always find graphs historical charts and I look at them and I see the patterns and I say why is it
- different now you know are we operating between the same extremes or will it be new and usually I've gone with stay with
- that stay with the trends so I didn't let myself get intimidated by any questions and I tried everything and as
- bill mentioned some of the things where I wellactually was most hesitant on topics
- that I might have knowledge on say on Ebola because suddenly I dunno I'm actually
- more self-conscious about things where I might miss something that I should know
- one easy to follow tip is do it every day and every question that you're forecasting have a
- look every day to see if anything has changed you what point things do change it significantly tends to come as a
- surprise and if you missed that day well you blew that forecast for that day the
- my approach is to try to think through causal mechanisms and and try to think of all the influential factors that I
- can think of and and try to visualize scenarios and and balance probabilities among scenarios things like that this is
- a little different than what I think most of us are led to do by the training we've had which is it's not that the
- training doesn't cover probability and that's critical but it's to look at the categories of the questions we're
- forecasting and try to find parallels in the past and look for rates of occurrence and try to use those rates I
- I think that's helpful as a starting place but it's not going to get you a very good score if that's all you do and
- I think the difference between you know sort of phoning it in and and getting a good score is understanding the
- situation as well as you possibly can and that's the what Phil has Phil tetlock has called the inside view as
- opposed to the outside view he emphasizes the people neglect the outside view but I think it's the inside
- view that that really matters to get you exceptional forecast that that actually
- could add to give real value anyway that's the inside view well the first thing was to really explore my biases to
- realize that I often would have a first impulse about a question and I didn't want to just go with that
- sometimes it was right but I didn't want to just go on impulse and secondly I really learned to think about what my
- strengths were to focus on questions that I thought were in my area of strength and in particular to focus on
- questions that I thought were were being answered wrong or you know where the the
- general consensus was different from what I thought it was because there I could make the most difference in
- shifting opinion or contributing knowledge that might sway the crowd and make the group as a whole more accurate
- so I think those things were really important a good friend of mine also named Barbara Barbara McClintock advised
- me over and over in our conversations that I should always question my fundamental assumptions my underlying
- assumptions and I think that's something that's been very important to me was Barbara's advice for instance a
- spacecraft just flew past Jupiter and took a whole bunch of high-resolution pictures of Jupiter I'm sorry Pluto it
- took a whole bunch of high-resolution pictures of Pluto and son of a gun this 4.5 billion year old planet is
- geologically active and everybody is absolutely floored the solar system is 4.5 billion years old the planets are
- not supposed to be geologically active when they're the size of Pluto why is this so everybody's trying to figure out
- why 4.5 billion year old Pluto is by geologically active they're talking about more missions to Pluto half
- hundred million dollar missions half billion dollar missions I'm certain that there are all sorts of young planetary
- scientists who are trying to figure out how they can start their careers by figuring out and making a name for
- themselves Pluto is geologically active well what's the underlying assumption here the
- underlying assumption is that Pluto's 4.5 billion years old what happens if Pluto isn't 4.5 billion
- years old what happens with Pluto's a good deal younger then in fact you expect it to be geologically active and
- there's no inconsistency here at all the inconsistency is that your fundamental assumption that Pluto is 4.5 billion
- years old was incorrect so I'd advise those young planetary scientists who are thinking about trying to explain how a
- 4.5 billion year old celestial body could be geologically active to question whether or not it's really 4.5 billion
- years old there's another something else that gets a lot more attention and that's it ran for the last four to five
- years the entire economy of the world has been an uproar because all of the major nations have slapped an embargo on
- dealing with Iran because Iran is trying to build a nuclear weapon and they had to negotiate long and hard with Iran to
- negotiate Iran away from a posture where they could be capable of building the nuclear weapon that they were trying to
- build and this brought us a lot closer to allies like Saudi Arabia to whom we perhaps should not be close and it's
- caused no end to trouble throughout the entire world particularly the Middle East which is troubled enough well
- what's the unda been underlying assumption here that Iran wants to build a nuclear weapon they kept saying that
- they didn't want to build a nuclear weapon why didn't we believe them well we have
- x y&z reasons why we didn't believe them but again it's a fundamental assumption
- the entire foreign policy in the Middle East of nearly the entire world was dictated by the underlying assumption
- that Iran wanted to build a nuclear device I personally wouldn't want to build a nuclear device I can't see why
- Iran would want a nuclear device nuclear bombs are so 20th century we've moved on I happen to think Iran
- didn't particularly want to build a nuclear bomb but everybody's fundamental assumption
- was that it did and I would go out on a limb and I'd say that even in spite of all the acrimony that we've had over
- Iran that it's quite possible that in 20 years Iran will be our closest ally in the Middle East and the reason for that
- is that Iran is the largest Persian nation on earth the United States is the second-largest Persian nation on earth
- and Canada is the fourth largest Persian nation on earth there's a natural affinity there and as soon as that
- embargo is lifted the Persians are not going to start doing deals with the British or the Chinese or the Russians
- they're going to do business with their Persian American friends so I see business between Iran Persia and the
- United States particularly the Persian community building very quickly and I think that could lead to good political
- relations as well that would be a major casualty of misunderstanding Iran's intentions with nuclear weapons the
- process evolved over time for me and one of the things that is important to me is
- to use historical information but not to be over alliant on it if there is the ability to compare against similar
- situations in the past I'm very cognizant of using that information but only to the extent that it is truly
- relevant to the current circumstances beyond that frequently adjusting my own forecasts based upon new information and
- new perspectives of others is extremely valuable it's very easy to take a position and sit back and say that's it
- however the real world is a very active one and often things change and forecasts should change based upon the
- events and the happenings of the world and I found it's extremely useful to pay a lot of
- attention to those events and frequently re-examine my own forecasts and see if they need to be updated so I really had
- three general rules when I made forecasts especially when I made my first forecast on a new question or a
- situation I'd never seen before so the first thing I would say is to be a good forecasters
- you have to be you have to hold strong opinions weekly and what I mean by that is that you do all of your homework
- upfront and you really present a good case for why you feel a certain way on a specific question and defending your
- probability probability estimate at that point but when you recognize that somebody else has more information or
- better information than you having the ability to change your mind swiftly to that new forecast I think that was one
- of the keys to why I was able to you know improve my performance over time I think many people really get stuck on
- their initial forecasts and they fall in love with it and wind up becoming going
- into a defensive Crouch when it comes to defending their opinion whereas I was on
- my team sort of notorious for being able to switch from one very hard position all the way to the opposite when I saw
- new information and come up that led me to do that the second thing is to be willing to put
- in the homework and to put in the effort to understand a situation go beyond the
- Wikipedia page and to go beyond the initial couple of news articles and to really try to understand you know what
- is it what is it about Boko Haram and their composition that you know is allowing them to make the kinds of
- military gains that their that we're seeing and how fragile or stable are those military gains compared to other
- like organizations and to you know not just to say Oh Boko Haram looks like every other
- organization that I've encountered to this point but to say to really try to understand what's going on there and to
- contextualize that appropriately and I think the the third thing is to have in your mind the kinds of psychological and
- cognitive biases that you could potentially fall prey to and to be able to overcome those by you know listening
- to your teammates by self critique and to recognize that you know whatever forecasts and argument that you're
- putting forward now you should be able to alter it or change it or improve it in light of new information so you know
- keeping in mind the cognitive biases that really impact and influence and reduce our accuracy and to consciously
- try to devise them I think the tournament was great you need to practice so you need to have
- a lot of questions you need to have feedback you need to know where where you're good and where you're messing up
- if you have a team if you have other people that you're in tournament either in competition or in collaboration
- either way you listen to them you get feedback and you will you will improve I think that one of the things that would
- be extremely useful would be to participate in good judgments open forecasting competition because you can
- test your own abilities and learn about the things that that both limit your ability to forecast and make you a
- better forecaster and through that process become somebody who is more adept at forecasting the good judgment
- incorporated itself has put together various training methods which have been very useful to me as a super forecaster
- and I'm sure to others but as an individual just forecasting and and using that forecasting experience can be
- a very valuable thing to learn about yourself and the process of forecasting what and and what you do well and what
- you do less well and then to seek ways of improving upon those weaknesses I think you could really do two things to
- become a good forecaster and one one is to read and not just read say Phil's book super forecasting or Danny
- Kahneman's book Thinking Fast and Slow which I think many of the forecasters who excelled in torment have read and
- done I also think that you know reading say histories like the guns of August or
- books about bureaucracy by James Q Wilson and others really improves your your knowledge base
- and gives you a better context for looking at situations and being able to figure out whether there are any
- commonalities or patterns there that you can pick up on and that could potentially apply to future situations
- that look like that so being well-read is I think a contributor to forecasting success the second thing is to just to
- practice don't get hung up on wrong forecasts that you make initially you know don't don't get overly you know
- sensitive to those results but to approach forecasting as something that you can improve on and have that growth
- mindset when it comes to making better forecasts and so from there you're able to say okay I start at some baseline
- level I got some feedback tells me I should improve in X Y & Z and to really make a concerted effort to do that
- through practice well one of the things I would recommend would be to write down
- a list of all the things that would that you should be on the lookout for that might cause you to change your mind so
- after you've done your homework you've thought deeply about you know the event and the likelihood of of it occurring
- write down a list of of things that you know would happen leading up to the event or signals markers whatever that
- would cause you to change your mind and and the reason why I say write it down is because it's because you know it's
- very hard to change your mind and if you write it down it might be help you you know not try to to fudge things well you
- know if it's if it's just in the back of your head you know you mM it becomes a little bit more difficult to to go
- against you know your original original conclusion because I mean you put a lot of work into this and you don't want to
- go back on it and there's an actual tendency not to want to you know waffle or be too wishy-washy
- but you know having it down on paper will just show you well I'm not being wishy-washy you know events have changed
- things have changed and maybe my initial conclusion was wrong I would say the third thing would be to play devil's
- advocate after immediately after you've you know come to a conclusion and made a forecast
- seek out information that would prove you wrong you know if anything you know just to cover all your bases
- I'd say my fourth piece of advice would be to seek a wide variety of sources even weak and biased sources because
- sometimes you can find interest in information in very flawed sources but that'll take some work on your part to
- determine you know which part of these you know particular reports are worth you know holding on to and which part
- should be discarded it's not an easy task but don't just you know casually disregard even even bad sources because
- you never know from four years as a as a forecaster in the gjp tournament I can say that another piece of advice I would
- give is to assume that things would take longer than your initial impression that's a rule of thumb because it
- happens consistently time and time again that whatever it is a diplomatic initiative or or or what-have-you
- events often will take a lot longer than then what what you think you know they will take or what
- officials you know the timetable of of officials and and that's just been my experience
- from doing this for a long time final piece of advice would be embrace uncertainty don't get too hung up on
- getting things correct you're never gonna get everything correct you're going to get a lot of things wrong I
- would focus more on assigning viable or realistic probabilities to events just think of it in terms of you know if I
- assign a 70% probability to a particular event you know I'm gonna be wrong three out of ten times you know four so
- if you if you work on being well calibrated instead of trying to get everything right I think that's probably
- a more fruitful endeavor well every year in the tournament I kept a running list
- of what questions I'd answered when I first answered them how I Hearst answered them and it was really helpful
- to see how my opinion changed as new information came along so I was you know kind of keeping myself honest about what
- was my original opinion and what was making me change my mind and I think that was really helpful because I
- learned a lot about what kind of pits I might fall into you know if I get over optimistic about suddenly there's a lot
- of news on a certain issue I might have a tendency to think that something was more likely to happen whereas maybe
- wasn't really more likely to happen so keeping track of my predictions was very helpful I thought it was really
- important that I was at forecasting anonymously at first it made it more I made me more confident about making
- predictions without worrying about their effect on my reputation and after all you know I was in this to get better
- about forecasting so I wanted to have the ability to take to make bets or you know answer questions in a way that I
- thought was correct without worrying about how that might be seen somewhere else and that really built my confidence
- and actually through being right then frequently too it built up my confidence to the point that I was willing to
- who I was and then got to experience the whole other dynamic of people knowing I
- was an expert wondering what I thought about things and then having to be cautious about not relying too much on
- my expert knowledge but still thinking about the wisdom of the crowd so I think the anonymity is really important and
- also realizing that the magic of this whole study is in revealing that everybody has important information and
- at different times different people's information will be really valid and especially important and and finding
- that right balance between being confident in your own views and and being open to the views of others become
- comfortable with researching problems have a great interest in what you're forecasting don't hesitate to dig right
- in to you know study something new just get into it don't don't neglect if you have any bent
- at all for statistics and probability bring it into play you know but that's kind of obvious anyone who has that
- choice I guess actually that the this might come up in your last question I think you forget but part of the muscle
- you want is the muscle you want to strengthen intuitively is the muscle for balancing scenarios against each other
- and sort of averaging out probabilities intuitively weighting them properly things like that balancing things
- against each other intuitively that are things that are pointing in opposite directions this is this is something you
- get better at and feel easier about by practicing I recommend that when you come to a new field about which you know
- nothing that you browse the literature talk to people if you can attend lectures if you can and listen for the
- totem words the words which either consciously or unconsciously people in the field
- give special meeting and I find that if you can figure out what those totem words are and then go away and figure
- out what they mean you'll find out at least one of the things that's very important to people in the field that
- they don't say is important to them necessarily but they give it importance by continually using these totem words
- or continually emphasizing them for instance last week I went to Grand Rounds at our Regional Hospital and
- there was a cardiologist giving a talk and he was giving a talk on the pharmaceutical management of cardiac
- patients who had circulatory problems problems with blockages problems with leakages he would go through a series of
- example patients this patient comes in with these symptoms what do you do we immediately stent this patient how do
- we drug this patient for the stent we're gonna use this compound and we're gonna
- use this compound but we're gonna stay away from that compound because it can cause bleeding and we don't want
- bleeding now this patient has got a partially occluded artery and is in a lot of pain we're going to give this
- patient warfarin because we need to dissolve that clot and prevent it from growing however we're not going to give
- this patient these other two drugs because they can lead to bleeding even worse than warfarin can lead to bleeding
- we give combinations of these drugs because it reduces bleeding bleeding this bleeding that what I came away from
- not knowing really that very much about how to treat cardiac patients was that if we can generalize from just this one
- talk what's most important is to keep the patient from bleeding no matter what you do and so if I was given a new drug
- and asked to predict whether or not it was going to be a success the first thing I'd look at its whether it causes
- uncontrolled bleeding or not what the likelihood is how it can be combined with other things all from a standpoint
- of does it cause bleeding now that doesn't give you the answer to how that new drug is going to do but it gives you
- a lead-in to it so that you can then begin to understand how things are going on
- so I'd say look for the totem words when you're reading an article look for the words that the author of the article
- seems to come back to over and over and then start by figuring out what those words mean trust yourself
- but doubt yourself as well and always be prepared to be devil's advocate to yourself if you're working within a team
- that's even better because most forecasters do not have a problem being devil's advocate or would call red
- teaming and keep an open mind don't go into something thinking you know exactly what's going on it never seems to work
- out that way just when you think you know exactly what's going on it changes no the main thing is just you know be
- open to change in dr. tetlock spooky mentions that we're in perpetual beta that is a great way of thinking of it is
- always be willing to change and learn and just you know go with it a lot of people get really kind of stuck in the
- mud on certain ideas and that's that's the quickest way to get it wrong I've been very insecure all my life for
- a variety of reasons and I never really thought very much of my own opinions I always figured well is the world crazy
- or is it me and the answer was I must be crazy I think the good judgment project has
- shown me that occasionally I'm right and other people are wrong which is not something that I expected going in I
- expected to be wrong all the time and curiously and I've seen this in Phil's book and I've heard it from other people
- the things about which I was most wrong with the things I was supposed to know something about namely the medical
- questions so I think the good judgment project has given me a little bit of confidence in myself that I didn't have
- before I went into it it's been huge and for first off you get exposed to a lot of wonderful people whom you tend to
- maintain some degree of contact with I've had a couple of of my teammates visit me at my house one who's from
- South Africa the other ones from Spain it's true they were on Washington for other reasons but that's an example and
- I I'm glad to stay in touch with a lot of the fork in fact there there have been others as well from the DC area
- anyways I'm glad to stay in touch with these new friends that's that's see that's a big thing and just as big is
- there's nothing like reality focusing on reality and how it works out and you tried to see the future and maybe it
- worked and maybe it didn't and you you sort of mentally record it's there's feedback going on that's it's extremely
- good feedback for turning you into a more rational person it's it makes you a more balanced person a more rational
- person easier to live with perhaps I don't know but it's all good it's been very rewarding and this
- doesn't mean that the forecaster shouldn't also be rewarded in other ways but this myself is rewarding and I'm and
- I'm sure that the others feel much much the same I think being a forecaster and being a researcher are similar and
- different I think that one of the great things about being a forecaster is that we really weren't constrained by any of
- the kind of customs or sociology around forecasting because we were the ones coming up with that and so we were able
- to explore and be very free in how we approach forecasting whereas I think research while you have some freedom and
- hypotheses that you are tackling a lot of the methodology and the structure around writing papers and peer review
- and submissions is very much rigid and you must follow the the specific process so I think that that highlights a major
- difference in forecasting versus doing the research and that the research seem to be much more process oriented
- although obviously there there is an outcome component to it whereas when when we were doing strictly forecasting
- really people only cared that we were able to come up with the right answers rather than you know how exactly we we
- got there so yeah they're different they're different animals I use different words when I talk to myself
- now I've always been very analytic but now I I start talking those percentages to myself and and that's different
- my wife and I have two different philosophies of investment she has been involved over the years as a partner and
- an investment firm that does alternative investments which are allegedly less risky but by my
- lights carry a risk that is sometimes more than what I like I on the other hand am much more risk averse and
- although less risk-averse then I was when I first started participating a good generative project my professional
- career has been oriented towards reducing the risk that decision makers have and perhaps I've taken that point
- of view in my own approach to investing which my wife feels is overly conservative I think that that the
- forecast thing as part of the good judgment project has given me a window into my own risk-taking perspective and
- move me at least somewhat closer to my wife's point of view and and I think that that's a very useful thing in real
- life as well as you know from the standpoint of participating as a super forecasting so in my work
- this experience has been extremely important I am now working on an article about the rise of China that I would not
- have written if it had not been from my experience in the good judgment project by having to constantly update my
- knowledge about China's economic capabilities and its military capabilities in order to do well in this
- game I kept up in a much finer green level with things that were happening in China and that have changed my forecast
- about how international relations will work over the next couple of decades so I'm an international relations scholar
- one of the most important things in the international relations field right now is this question of of does the United
- States have peer competitors is China one yet and this experience helped me see that it is so it's really
- dramatically changed my career and the process of going through the confidence-building process
- of you know learning how to articulate forecasts learning how to be how to express them in probabilistic terms how
- to be confident even if I'm not completely sure has made me willing to take the risk of going out on this
- really controversial subject I'm not sure that they have all they probably should having practice at forecasting
- geopolitical events maybe I should start practicing forecasting events related to
- my own personal life maybe that would help me avoid being unpleasantly surprised by my things but so I'm not
- really sure how it's affected my personal life other than having nice dinnertime conversations with families
- and friends like I said I go back and forth between the different you know venues of inquiry in archaeology and in
- forecasting so there has been times where one of the things that forecasting told me is it's
- great if you have a model but all they're going to be a better model always be a better model you might have
- one that's showing you exactly what you want but you know if you tweak it a little bit or you say you find a new
- dataset can always be that little bit better and I've started to apply that to some shipwreck hunts that I've been
- doing where we have a model on how to locate shipwrecks in areas of high probability but it can always be better
- so I've actually started tweaking some of the models that I use at work to add additional data sets maybe pulling
- things from multidisciplinary studies from geomorphology marine geomorphology riverbed hydrological processes going
- into shipwreck site formation and just basically you know constantly tweaking the models there's something the other
- thing that I've tried to do as much as I can is I have two young sons who I'm trying to teach them how to approach
- different issues you know not not so much to get him in trouble at school for questioning the teacher all the time
- which is very possible but when they answer a question say well is that it I spoke to my son about a tree growing in
- a field I said well how much do you think that tree is gonna grow next year and he just said oh well this I said
- well I should say that he's like well because it just looks like a well did you think about how much was it gonna
- rain next year how much fertilizer is the farmer going to put in that field without trees in the middle how often is
- he going to plow it what you know there's so many issues that surround one issue that when you hear
- one question there's usually you know 20 30 however many more questions that can
- help you answer that question so I've been trying to push it on them to kind of think that way
- you
Brief Summary
Ben Valentino discusses his journey in forecasting and the importance of historical context and collaboration in improving predictions.
Key points
- The Journey to Forecasting. From military tours to academia, the speaker's path to forecasting is shaped by diverse experiences.
- The Role of Curiosity. Intellectual curiosity and willingness to learn are key to making accurate forecasts.
- The Influence of History. Understanding history is crucial for accurate forecasting, as it reveals patterns and possibilities.
- The Importance of Multiple Sources. Relying on a single source can lead to misinformation. Always seek multiple perspectives.
- Understanding Uncertainty in Forecasting. Embracing uncertainty is crucial for accurate forecasting. Focus on assigning realistic probabilities.
- Learning from Predictions. Tracking your predictions can reveal biases and improve forecasting skills.
Episode highlights
- The Journey to ForecastingFrom military tours to academia, the speaker's path to forecasting is shaped by diverse experiences.“I was looking for some way of doing forecasting...”0:30Feb 26, 2016
- The Role of CuriosityIntellectual curiosity and willingness to learn are key to making accurate forecasts.“Intellectual curiosity and just a willingness to put in the work...”7:54Feb 26, 2016
- The Influence of HistoryUnderstanding history is crucial for accurate forecasting, as it reveals patterns and possibilities.“Having a sense of history that stretches back far can give one a sense of possibilities.”13:14Feb 26, 2016
- The Importance of Multiple SourcesRelying on a single source can lead to misinformation. Always seek multiple perspectives.“Don't trust one source of information.”23:34Feb 26, 2016
- Understanding Uncertainty in ForecastingEmbracing uncertainty is crucial for accurate forecasting. Focus on assigning realistic probabilities.“Embrace uncertainty; you're never gonna get everything correct.”44:24Feb 26, 2016
- Learning from PredictionsTracking your predictions can reveal biases and improve forecasting skills.“Keep track of your predictions; it helps you learn.”45:44Feb 26, 2016
- The Importance of AnonymityAnonymity allows for a more honest exchange of ideas and information.“The anonymity is really important.”46:42Feb 26, 2016
- Balancing Confidence and OpennessFinding the right balance between confidence in your views and openness to others' is crucial.“Balancing scenarios against each other is something you get better at by practicing.”47:02Feb 26, 2016
Tension
45/ 100Some tension





