Why an Open Mind Is Key to Making Better Predictions
Ask about Knowledge at Wharton
Episode Description
In the new book, Superforecasting, Wharton professor Philip Tetlock and coauthor Dan Gardner look into why making predictions is so difficult — and how to become better at making them. More: http://knlg.net/1L69rZa 📂 Management 🏷️ Research
Read the full transcriptHide the full transcript
- everybody wants to see into the future whether it's to find out whether or not we should take a certain job figure out
- what's going to happen with a particular relationship or maybe we just want to know if it's going to rain on Sunday but
- despite all this interest in forecasting most actual forecasts aren't very good and even and moreover they're also not
- very well analyzed in the new book super forecasting the art and science of prediction Wharton Professor Philip
- tetlock looks into what makes people good forecasters and suggests how you can incorporate some of those techniques
- into your own life Philip thanks for being with us thank you my pleasure now thanks to people like Tom Friedman and
- Nate Silver and I think also maybe to the rise of Big Data there seems to be a lot of interest in forecasting so I was
- really surprised to learn from your book that despite all this interest in forecasting and maybe interest in people
- who have fashioned themselves as forecasters forecasting itself's not very well studied it's not very well
- analyzed at all I think that's fair to say and it's pretty threatening to keep score of your forecasting accuracy
- you imagine you're a big shot pundit what incentive would you have to submit to a forecasting tournament in which you
- had to play on a level playing field against ordinary human beings and the answer is not much because the best
- possible outcome you could obtain is to tie it you're expected to win so the best outcome is a tie and there's a good
- chance our research suggests that you're not you're not going to win now the book
- is acted was actually decades in the making I mean your research into this has stretched back for years and you
- actually start the book and in some ways this all started with a dart throwing chimp could you tell us a little bit
- about that story and I know you said in the book that people don't exactly get the right takeaways out of that study
- but I thought it was interesting to sort of show what happens when we try and test forecasting sure well in our early
- work which which I'm going to reveal how old I am goes back into the mid 1980s we did use the metaphor of the dart
- throwing chimp to capture the idea baseline for performance which is how much better can you do than chance if
- you had a system that we did that was just generating forecasts by chance how about would you do relative to that
- and that actually is a baseline that some people can't beat now they can't beat it
- for a lot of reasons sometimes the environment is just hopelessly difficult ranked if you were trying to bet on the
- roulette wheel in Las Vegas you're not going to be able to do any better than a dart throwing chimp but people sometimes
- fail to beat the dart throwing chimp even in environments where there are predictable regularities that could be
- picked up if you were being a student off now in the book you point out that a lot of people took away from that study
- that all predictions are bad that forecasting is bad but in fact what you were pointing out really is that there's
- actually limits on predictability but it's not all bad it's just that it there are limits on it
- that's right you don't want to be too hard on people because some environments really are there's a lot of irreducible
- uncertainty in some environments it's very difficult to to bring it down below a certain point so it's unfair to
- portray people as as being dumb in some sense if they're failing to do something
- that's impossible of course we don't know what's impossible until we try until we try and
- earnest you net you don't discover what we call the optimal forecasting frontier
- you don't discover how good it's possible to become in a particular forecasting environment until you run
- forecasting tournaments competitive tournaments you plug in your best techniques for maximizing accuracy and
- you see how good you can get and this essentially what we did in the forecasting tournaments with the US
- government wasn't sponsored by the intelligence community the intelligence Advanced Research Projects Agency
- we these are forecasting tournaments that run between 2011 and 2015 Volvo tens of thousands of forecasters trying
- to predict about 500 questions posed by the intelligence community over that period of time and he found that some
- people could do quite a bit better than the dart throwing chimp and they could meet some more demanding baselines as
- well now one of the interesting things in the book is I mean you talked a little bit about where you found these
- forecasters who are part of your study which is called the good judgment project can you talk a little bit about
- where you recruited these people from and then also how these tournaments actually took place like what exactly
- were they called on to do and how did they do well we were very opportunistic we recruited forecasters by advertising
- through professional societies by advertising through blogs a number of high-profile bloggers helped us to
- recruit forecasters people like Tyler Cowen Nate Silver various people were helpful in recruiting forecasters plus
- we knew quite a few people from the earlier work that I'd done an expert political judgement so we were able to
- gather initially a group of several thousand and we were able to build on that in subsequent years the questions I
- you know I have to be careful about making big generalizations about how good or bad people are as forecasters as
- I mentioned before you can make people look really bad if you want to you can pose just intractably difficult
- questions or you can make people look really good you can you can pose questions that aren't all that hard and
- so you you want to be wary of research that does cherry-picking and and and there are some aspects that some aspects
- of that and some in some of the literature but we were looking for was a process of generating questions that
- wasn't rigged one way or the other and the method we came up with was generating questions through the US
- intelligence community there were questions that people inside the US intelligence community felt would be
- of national security interest and relevance and reasonably representative of the types of tasks that intelligence
- analysts are asked to do so these were questions typically they asked people to see out into the future
- several months occasionally a bit longer occasionally shorter and we scored the accuracy of their
- judgments over time we didn't have people make judgments one way or the other it wasn't yes or now we had people
- make judgments on what's called a probability scale ranging from 0 to 1 and we carefully computed accuracy over time
- we identified some people who are really good at it we called them super forecasters and but then they were later
- assembled into super teams of super forecasters and they you know dominated the tournament tournament essentially
- over the next over the next four years but we we did a number of other experiments as well looking for
- techniques that could could be used to improve accuracy and we found some and now these super forecasters they really
- came from all walks of life but one of the things you point out in the book is that what makes a good forecaster is
- really how you think and can you talk a little bit about what do you mean by that and what are some of the unifying
- characteristics of super forecasters well when you ask people in the political world who has good judgment
- the answer typically is people who think like me so liberals tend to think that liberals have good judgment and if you
- have good forecasting judgment and conservatives tend to think that they're better at it it turns out to be the case
- that that pork a good forecast forecasting accuracy is not very closely associated with with ideology there's a
- slight tendency for people who are the super forecasters to be more moderate and less ideological but there there are
- lots of super forecasters who have strong opinions oh it distinguishes super forecasters is their ability to
- put aside their opinions at least temporarily and just focus on accuracy and that's that's a very demanding
- exercise for people now other things you mentioned I don't like at the end of the
- book you have ten commandments for super forecasters and so I'm wanting now with
- the super forecasters are there ways to make even super forecasters better are there conditions or environments to make
- them I guess super super forecaster super super forecasters well event eventually you're gonna reach a
- point where you're not going to be able to get any better because the as I've mentioned it's some environment the
- environment itself has some degree of irreducible uncertainty so no matter how good you are you're not gonna do a very
- good job probably predicting what the value of Google is going to be next week on the on the New York Stock Exchange
- there so there are some things that are very difficult to do and it's not clear that the super forecast
- even using super forecasters is going to let you make appreciable headway on that
- but there are many things that are quite doable that we previously didn't think we're doable and you there's a lot of
- room for improving the accuracy of probability judgments on those things those are things like predicting weather
- international conflicts or going to escalate or deescalate whether certain treaties are going to be signed or
- approved by legislative legislators well what but whether Greece is going to leave the eurozone so there are a lot of
- problems that you know have relevance the financial markets have relevance to business decisions where there is
- potential for improving probability judgment where we have shown that experimentally now in the IR /
- tournament where people typically don't don't do that people typically rely on vague verbiage forecasts they people say
- well I think it's possible or this could happen this might happen it's likely those are terms that you know are
- informative at some level but they're not all that informative if I say that something could happen for example you
- know Greece could leave the eurozone by the end of 2017 what does that mean it could mean I think there's a probability
- of 1 percent or 99 percent you know we could be hit by an asteroid tomorrow the Sun could rise tomorrow yeah I know
- there though there it's it's it's a very elastic word so the asking people to make crude
- quantitative judgments which become progressively more refined over time is a very good way to both keep score and
- get better at it now I find it I find it really ironic in the book that you know
- we have all of these people in the world who have sort of fashioned themselves as
- professional forecasters I mean pundits on TV they're really television personalities media personalities and
- it's almost some and it seems like to do that job it's almost kind of a cult of personality you don't want to be proven
- wrong you would never admit that you're wrong you're just gonna sort of keep kicking it down the road and say no it's
- going to happen but what you find in the book is that super forecasters one of the things that unites all of them
- despite coming from all these walks of life is that there a lot they're willing to be proven wrong or live they are open
- to the idea they're willing to sort of look at things look at evidence and retread and pivot and so I found that
- interesting it was - it was kind of ironic and I know you tell a story in the book about foxes vs. hedgehogs that
- was very interesting right well yeah but the Fox Hedgehog metaphor is is drawn out of a surviving fragment
- of poetry from the Greek warrior poor Killa cos 2,500 years ago scholars have puzzled over it for over
- the centuries it runs something like this and of course I don't know ancient Greek so I'm taking it on faith this is
- what it actually says the the Fox knows many things but the Hedgehog knows one big thing
- now you can think of hedgehogs in in debates over political and economic issues as
- people who have a big ideological vision Tom Friedman might be animated by a vision of a globalization the world is
- flat libertarians are animated by the vision that free Mart there are free market
- solutions for the vast majority of problems that beset us there are people on the left to see the
- need for major state intervention to address various inequities there there are there are environmentalists who
- think that we're on the cusp of an apocalypse of some sort and so you have people who are animated by a vision and
- their forecasts are informed largely by that vision and whereas the Foxes tend to be more eclectic they kind of pick
- and choose their ideas for a variety of schools of thought they might be a little bit environmentalist and a little
- bit libertarian or there might be a little bit socialist and a little bit hawkish on certain national security
- issues there they they blend things in unusual ways it's and they're harder to classify politically now in the early
- work we found that the Foxes were more COO or more eclectic in their style of thinking we're better forecasters than
- the hedgehogs and in the later work we found something similar we found that people who scored
- high on psychological measures of active open-mindedness and need for cognition those people who score high on those
- personality variables tended to do quite a bit better as forecasters now it seemed to me then that we would really
- want is for more foxes to be kind of these really prominent forecasters or these people that we're seeing on
- television that we're seeing in the news but you know but then at the same time and although seems like their
- personalities are just not necessarily what we think of like we're at odds with what we think of when we
- think of a leader when we think of someone who's always putting themselves out there so how does that what does
- that mean for trying to get more accurate forecasters I mean how do we get people to be lists to listen to the
- Foxes when they might not be the sexiest or the most prominent or the most the people that we want to look at all the
- time or listen to all the time that's right so what do we do that is a bit of a dilemma
- imagine you are a producer for a major major television show and you have a choice between someone who's going to
- come on the air and tell you that something decisive and bold and interesting is going to sit eurozone is
- going to melt down the next two years or Chinese economy is going to melt down or
- there's gonna be a jihadist coup and Saudi Arabia he's got he's got a good big interesting story to tell and the
- person knows quite a bit and can mobilize a lot of reasons to support the doom stir prediction say on eurozone or
- China or Saudi Arabia or a boom stir prediction for that matter it doesn't matter but the person is charismatic and
- forceful and can generate a lot of reasons why he or she is right as opposed to someone who comes on and says
- yeah there's some danger eurozone's gonna melt down but on the other hand there are these countervailing forces
- and on balance problem nothing dramatic is probably gonna happen in the next year or so but it's possible that this
- could work so I'm who make what makes better television to ask the question is to answer it so there is a tent there is
- a preference for hedgehogs in part because hedgehogs generate better sound bites and people who generate better
- sound bites generate better media ratings and that is what people get promoted on in the in the media business
- so there is a bit of a perverse inverse relationship between how between having the skills that go into being a good
- forecaster and having the skills that going go into being a fact of media presence now how does this come into
- play though I mean sort of in the world that maybe those of us who are not hiring forecasters in a regular basis
- don't so if I'm a company or the government and I'm trying to assemble a team of good forecasters or get good
- accurate or accurate as possible forecasts what can I take from this book I mean in terms of finding these people
- and employing these people and is that happening now or is it the same as what maybe what we're seeing on CNN our Fox
- News all right well I think a lot of organizations in both the private and the public sector are taking an
- increasing interest in forecasting tournaments and using them as methods of keeping score on how accurate their
- their forecasting methods are they're people and their methods and their teams there's a growing interest also in using
- forecasting tournaments to identify people who are better at it and you know develop your own core of super
- forecasters there's growing interest in exploring methods of training people to be better forecasters so I think this is
- going on I probably shouldn't mention the names of any organizations who have adopted
- this right now but a US intelligence community obviously has taken an active interest in this area and I and some
- private sector organizations have as well now it was interesting to me that um so you were discussing a lot of the
- book talks about a little bit about framing so it's not just about finding people who can make good forecasts but
- it's also a lot about you know finding the right questions finding the right ray to frame the problem breaking down a
- big problem into smaller little clusters I mean if there's so much that goes into
- forecasting other than the actual forecasts that comes out of it and I wonder that do people leave are people
- thinking enough about those things as well in addition to finding people who give good forecasts I see that as one of
- the big objectives of the next generation of forecasting tournaments to focus on
- generating not just good answers but good questions in the book we talked about the parable of Tom Friedman and
- Bill flack Tom Friedman is of course a famous New York Times columnist Pulitzer Prize winner who
- regular at Davos in the White House circulates in networks of power bill flack is an anonymous
- retired hydrologist in Nebraska who also is a super forecaster we know a huge amount about Bill flocks forecasting
- track record because he answered a very large number of questions in the course of the tournament and demonstrated he
- could do so effectively but we know virtually nothing about Tom Friedman's forecasting track record notwithstanding
- that he's written a great deal over the last 35 years and he's a music he's a powerful analyst and a writer and and he
- does many things very well but there's no way really to reconstruct with any degree of certainty
- reasonable certainty how good a forecaster he is and Tom Friedman has detractors he has admirers his his his
- his admirers might say well he was right about that it was a bad idea to expand NATO eastward because it would provoke
- nationalist backlash in Russia or he was or he was wrong about Iraq because he supported the 2003 invasion
- people have a lot of opinions about those things now here's what we we did a careful analysis
- of Tom Friedman's columns and one of the things we noticed is even though it's very difficult to discern whether or not
- he's a good forecaster going back after the fact it is possible to detect we have some
- really good questions he said he's a pretty darn good question generator and we've actually begun to insert some of
- his his ideas for questions there they tend to be rather open-ended we've managed to
- translate some of them into future forecasting tournaments so let me give you an example from the past that
- illustrates the tension between being a super question generator and a super forecaster so in 2008 2002 early 2003
- before the Iraq invasion Tom Friedman wrote what I thought was a really quite brilliant column on on Iraq in which he
- posed the following question we could really cut to the essence of a war what key issue in deciding whether to go into
- a rocky yes is Iraq the way it is today because Saddam Hussein is the way he is or a
- Saddam Hussein the way he is because Iraq is the way it is the chicken and the egg and what would happen if you if
- you took away Saddam Hussein with the country disintegrate into a war of all against all or would it move to boredom
- going to become a Jeffersonian liberal democracy in the next 15 or 20 years now that maybe I'm not done maybe not
- quite that fast but in the in that direction things things would move in that would move in that direction so
- Tom Friedman didn't know the answer to that question many people think he made a big mistake
- in supporting the invasion of Iraq in 2003 but he was shrewd enough to pose the right question and if we've been
- running forecasting tournaments in late 2002 early 2003 that would have been something we would have wanted very much
- to include in in in in in that exercise and so I think the way the right way to think about Tom Friedman and Bill flack
- is that you know is that they they're complementary and that Tom Friedman's greatest contribution to forecasting
- tournaments may well be and his perspicacity and generating incisive questions he may be a good forecaster too but we
- just don't know that yet right but we need in order but good forecasting we need the Tom Friedman's in the world and
- the bill flacks so it to me it would seem it's just a question of trying to get them together in the right ways and
- the right permutations to get the best better and to get better prediction that's where we come around in the book
- it's not really Tom versus bill it's Tom and Bill it should be symbiotic right and now to get a little back to I
- mentioned big data at the beginning now in the age of supercomputers a machine learning and we're saying we can enter
- this into anything and get these answers back to us I mean what do you think how
- do you think the role of human forecasting is going to change I mean how do these how do using computers
- using data how does that complement or even compete with human forecasting well in the book we conducted an interview
- with David Ferrucci who was when he was an IBM scientist he was responsible for developing a famous computer program
- known as Watson which defeated the best human jeopardy players and we asked him a number of questions about his views
- about the human machine forecasting and one question one law one line of questioning was particularly interesting
- I think it was we it was it was very clear to him and that it would be possible for
- a system like Watson to answer the following question reasonably readily which two Russian leaders traded jobs in
- the last five years that question what Watson could search his historical database it could figure it out
- reframe the question as will those Russian same Russian leaders change jobs in the next five years would Watson have
- any capacity to answer a question like that and and his answer was no and the question well how difficult would it be
- to reconfigure Watson so that it could answer a question like that and his answer was massively difficult it would
- not be something that would be easy to accomplish any any time in the near in the near future I
- think that's probably true I'm not an expert in that area but he obviously is but when I think about what would be
- required but what's required to do the sorts of things that super forecasters collectively do
- the amount of guesswork but the amount of informed guess work that goes into constructing a forecast a reasonable
- forecast it's difficult for me to imagine existing AI artificial intelligence systems doing that in the
- near term so now if reading the book I mean most people luckily will probably not be asked to answer big questions
- about Iraq big questions about Korea or any some of the other things that you talk about in the book but if someone is
- reading the book and just to become a better forecaster about their daily lives I mean what do you hope that
- people take away from that to kind of apply it to the everyday to think they're going through whether it's jobs
- relationships or even rain on Sunday right right well I think a lot of people spend quite a bit of money on
- advice about the future that probably isn't worth the amount of money they're spending on it and they don't really
- know they have no way of knowing that because they have no way of knowing the track record to the people whose advice
- they're seeking the most the most the best example that is probably in the domain of Finance where a lot of money
- changes hands it's directed to people who claim to have some ability to predict the course of financial markets
- that is an extraordinarily difficult thing to do I'm not saying it's impossible or that nobody can do it with
- any better than the dark throwing chimp but it's a very difficult thing to do so
- I think if people were more skeptical about the people to whom they turn for advice about possible futures I think
- finance would be a case in point but I think more generally they should be very skeptical of the pundits they read and
- the claims that politicians and other people make about the future as well it's very common for people to make bold
- claims about the future and offer no evidence for their track records I would say it's almost universal
- [Music] and so I guess if someone's making a bold claim is that where we should is
- that the point where we should become suspicious I guess well the bolder the claim the more the burden of proof
- should fall on the person to demonstrate that he or she has a good track record and it seems to me like it's often more
- the bolder the claim the less likely someone's gonna question that person sometimes well that's a great point
- that's a point of a human psychology as we take our cues about whether somebody knows what he or she is talking about
- from how confident he or she seems to be and the more confident the more likely you're going to be able to blunderbuss
- your way through the conversation so that's that's a that's a problem and it suggests that people need to think a
- little bit more carefully when they make appraisals of competence and not rely as
- quite as heavily as they do and what we call the confidence heuristic it is it is true that confidence is is somewhat
- correlated with accuracy but it's it's also possible for manipulative human beings to use that heuristic and turn us
- into money pumps well thank you so much for being here we appreciate it okay thank you
- [Music] you
Brief Summary
Philip Tetlock discusses effective forecasting techniques and the characteristics of successful forecasters from his book "Superforecasting."
Key points
- Dart-Throwing Chimp Study. Tetlock discusses a study involving a dart-throwing chimp that illustrates the challenges of forecasting accuracy.
- The Art of Forecasting. Wharton Professor Philip Tetlock explores what makes a good forecaster and how to improve your own forecasting skills.
- Foxes vs. Hedgehogs. The metaphor of foxes and hedgehogs highlights different forecasting styles and their effectiveness.
- Tom Friedman's Questions on Iraq. Friedman posed a pivotal question about the essence of war in Iraq.
- Skepticism Towards Financial Predictions. People should be cautious about whom they trust for future advice, especially in finance.
Episode highlights
- Dart-Throwing Chimp StudyTetlock discusses a study involving a dart-throwing chimp that illustrates the challenges of forecasting accuracy.“Some people can’t beat the dart-throwing chimp.”2:18Oct 2, 2015
- The Art of ForecastingWharton Professor Philip Tetlock explores what makes a good forecaster and how to improve your own forecasting skills.“Super forecasters are willing to be proven wrong.”10:38Oct 2, 2015
- Foxes vs. HedgehogsThe metaphor of foxes and hedgehogs highlights different forecasting styles and their effectiveness.“Foxes tend to be more eclectic; hedgehogs have a big ideological vision.”10:57Oct 2, 2015
- Tom Friedman's Questions on IraqFriedman posed a pivotal question about the essence of war in Iraq.“What key issue in deciding whether to go into Iraq?”19:18Oct 2, 2015
- Skepticism Towards Financial PredictionsPeople should be cautious about whom they trust for future advice, especially in finance.“Be skeptical of the pundits they read.”24:43Oct 2, 2015
Episode quotes
Tension
40/ 100Low tension





