Building Better Forecasters
Ask about Knowledge at Wharton
Episode Description
📂 Public Policy 🏷️ Research
Read the full transcriptHide the full transcript
- i've been involved in a multi-year forecasting tournament sponsored by iarpa which is the
- research branch of the intelligence community iarpa sponsored five different university teams
- that competed with each other to come up with the best possible ways to measure and aggregate forecasts about
- events all over the world and they include military conflicts elections pandemics
- refugee flows and even things like the price of commodities now what we did was to recruit thousands
- of forecasters from blogs and professional societies and research centers and so forth and
- had them make forecasts over a period of a year and they were given questions every two weeks they logged on to a
- website where they made their predictions and they went back to update their forecasts as often as they wanted
- now we didn't really know what to do to improve forecasts so we did what came naturally and that was
- to run experiments we found that three factors did extremely well one was training people we devised a
- one hour probability training module and and that seemed to improve predictions we put people in teams as opposed to
- having them work individually and that improved predictions the interaction the information sharing
- the debates about rationales were boosted accuracy more over and beyond the benefits of
- working alone and lastly we found that tracking was a huge booster of forecasting accuracy
- at the end of each year we took the top two percent of thousands of forecasters put them
- together in elite groups and gave them the title of super forecasters and these people uh
- increased their accuracy in ways in in more than we could possibly have imagined
- they interacted more they looked for more information and the net result was amazing
- in fact they helped us win the tournament three years in a row businesses everyone for that matter
- relies on predictions and we know a lot more about predictions than we did prior to this tournament in the
- in the world of business people care about whether to invest in research or expand
- to a new market and they care about what consumers will want what extensions they'll
- prefer when products will be ready for distribution and businesses can i think use the insights from this
- tournament to make better predictions not everything general generalizes smoothly but
- there are a lot of both psychological and statistical insights that we know now make predictions better
- you know i was surprised by an awful lot of things in this tournament i was surprised that uh our training module
- worked it's it's tough to to design a module that has any effect on judgmental accuracy
- i was very surprised that teams worked better than independent forecasters i read the wisdom of the crowds and i
- assumed that independent forecasters would would do better and heirs would average out but
- the benefits of sharing information and talking about rationales outweighed the the benefits of of independence so
- um in our case uh teams worked better and i was super surprised about the effects of super forecasters
- it's it's like tracking kids in schools and putting the best ones together and the the synergy that came from that was
- phenomenal yeah and i think there's several reasons why our teams work so well one of them was that they worked
- online so it's tough to be a dominant bully online people logged on whenever they felt like it and so
- things were sequential they weren't simultaneous where i think groupthink is more likely
- to occur they had a lot of respect for each other the the forecasters and i think that's another
- big part of a good recipe well i think companies can improve their predictions and i think they can create their own
- set of super forecasters and they can do better at forecasting the future um and better is really relative i mean
- all you have to do is be better than the next guy we're not looking for perfection here
- and obviously we're not going to get it um in the case of the u.s government policy makers face decisions that
- involve billions of dollars and thousands of lives and in in these cases you know the
- stakes are so high that even a tiny little edge is huge progress um the the one of the implications
- of of our work i think is that good predictions involve both psychology and statistics it's a combination of
- understanding the person and then understanding the the aspects of of uh statistical distributions and
- statistical information we know now how to uh do much better at devising algorithms that
- aggregate multiple forecasts and we also know a lot about what conditions or environments people
- bring out the best of of of individual forecasters well every day we hear predictions
- we have pundits and experts and gurus and specialists making predictions about what will
- happen in the future and many of those predictions are so vaguely stated that we could never in a million years
- figure out how to test them there are statements like there may be an increase in conflict in yemen in the
- next two weeks or the situation in baghdad will get worse before it gets better or
- something like that that's that's not a question or a statement that passes the clairvoyance test you could
- you can't figure out later who's right and who's wrong now we look to these people for
- advice we look to them for insights and we're getting really very little from those vague predictions and i think
- the way we can do better is to keep score we've got to have questions or predictions stated
- in such a way that they're testable and then we can find out who's right and who's wrong
- and how we can learn to do better we are in a fortunate position of being able to have
- an empirical basis for the claims we make we have we know what works based on data based on experiments and many of
- the books and methods and techniques for doing better forecasting are simply not tested so clearly
- the empirical side of things is something that is unique to our project i think we've learned a lot about
- uh what makes things better we've learned for example that survey formats with statistical
- algorithms combining the forecast can outperform prediction markets we've learned that if you measure
- probabilities while people are trading in the market you also ask them what's your probability that event x
- will occur you can do better at forecasting accuracy by combining both the prices and the probabilities
- and these things are both surprising from an economic perspective we know a bit more about individual
- differences that correlate with forecasting accuracy so we can say a little bit more about
- who the best forecasters are especially in this geopolitical context not surprisingly they're
- they tend to be smart they tend to know a lot they have a lot of political uh knowledge but they are also
- more likely to be actively open-minded thinkers they are more analytical they are more likely to take a scientific
- world view they're more likely to take multiple perspectives on a question and use multiple reference
- classes they're more likely to use probabilities in a more granular or nuanced fashion they're more likely to
- say 17 and 83 percent rather than 20 and 80 percent and it turns out that extra granularity
- has information in it if you round forecasts up to the nearest 10 20 30 40 most forecasters do worse and
- that suggests there's there's valuable signals in that granularity not just noise
- well uh the irb tournament will close on june 2nd in in another month but that's not the end of forecasting we
- will be opening the good judgment project will be opening a public tournament this fall
- and we have lots of hypotheses to test we'll need lots of volunteers and we'd love to have people
- affiliated with wharton so if this is something that interests you or you know people who would be
- interested go to www dot goodjudgementproject.com to get more information and we'd be
- extremely grateful and delighted to have you join us you
Brief Summary
The episode covers forecasting accuracy improvements through training and teamwork, emphasizing the importance of testable predictions.
Key points
- Creating Super Forecasters. Identifying top forecasters and forming elite groups significantly boosted prediction accuracy.
- The IARPA Forecasting Tournament. A multi-year competition involving university teams to improve forecasting accuracy.
- Surprising Insights on Teamwork. Teams outperformed independent forecasters, challenging previous assumptions about groupthink.
- The Importance of Testable Predictions. Vague predictions hinder progress; clear, testable statements are essential for improvement.
- Future of Forecasting. The Good Judgment Project aims to continue improving forecasting methods with public participation.
Episode highlights
- Creating Super ForecastersIdentifying top forecasters and forming elite groups significantly boosted prediction accuracy.“These people increased their accuracy in ways we could never have imagined.”2:26May 18, 2015
- The IARPA Forecasting TournamentA multi-year competition involving university teams to improve forecasting accuracy.“We helped us win the tournament three years in a row.”2:38May 18, 2015
- Surprising Insights on TeamworkTeams outperformed independent forecasters, challenging previous assumptions about groupthink.“The benefits of sharing information outweighed the benefits of independence.”4:26May 18, 2015
- The Importance of Testable PredictionsVague predictions hinder progress; clear, testable statements are essential for improvement.“We need predictions stated in such a way that they’re testable.”8:12May 18, 2015
- Future of ForecastingThe Good Judgment Project aims to continue improving forecasting methods with public participation.“We’d love to have people affiliated with Wharton join us.”11:49May 18, 2015
Tension
40/ 100Low tension





