Building Better Forecasters

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

📂 Public Policy 🏷️ Research

Read the full transcriptHide the full transcript
  1. i've been involved in a multi-year forecasting tournament sponsored by iarpa which is the
  2. research branch of the intelligence community iarpa sponsored five different university teams
  3. that competed with each other to come up with the best possible ways to measure and aggregate forecasts about
  4. events all over the world and they include military conflicts elections pandemics
  5. refugee flows and even things like the price of commodities now what we did was to recruit thousands
  6. of forecasters from blogs and professional societies and research centers and so forth and
  7. had them make forecasts over a period of a year and they were given questions every two weeks they logged on to a
  8. website where they made their predictions and they went back to update their forecasts as often as they wanted
  9. now we didn't really know what to do to improve forecasts so we did what came naturally and that was
  10. to run experiments we found that three factors did extremely well one was training people we devised a
  11. one hour probability training module and and that seemed to improve predictions we put people in teams as opposed to
  12. having them work individually and that improved predictions the interaction the information sharing
  13. the debates about rationales were boosted accuracy more over and beyond the benefits of
  14. working alone and lastly we found that tracking was a huge booster of forecasting accuracy
  15. at the end of each year we took the top two percent of thousands of forecasters put them
  16. together in elite groups and gave them the title of super forecasters and these people uh
  17. increased their accuracy in ways in in more than we could possibly have imagined
  18. they interacted more they looked for more information and the net result was amazing
  19. in fact they helped us win the tournament three years in a row businesses everyone for that matter
  20. relies on predictions and we know a lot more about predictions than we did prior to this tournament in the
  21. in the world of business people care about whether to invest in research or expand
  22. to a new market and they care about what consumers will want what extensions they'll
  23. prefer when products will be ready for distribution and businesses can i think use the insights from this
  24. tournament to make better predictions not everything general generalizes smoothly but
  25. there are a lot of both psychological and statistical insights that we know now make predictions better
  26. you know i was surprised by an awful lot of things in this tournament i was surprised that uh our training module
  27. worked it's it's tough to to design a module that has any effect on judgmental accuracy
  28. i was very surprised that teams worked better than independent forecasters i read the wisdom of the crowds and i
  29. assumed that independent forecasters would would do better and heirs would average out but
  30. the benefits of sharing information and talking about rationales outweighed the the benefits of of independence so
  31. um in our case uh teams worked better and i was super surprised about the effects of super forecasters
  32. it's it's like tracking kids in schools and putting the best ones together and the the synergy that came from that was
  33. phenomenal yeah and i think there's several reasons why our teams work so well one of them was that they worked
  34. online so it's tough to be a dominant bully online people logged on whenever they felt like it and so
  35. things were sequential they weren't simultaneous where i think groupthink is more likely
  36. to occur they had a lot of respect for each other the the forecasters and i think that's another
  37. big part of a good recipe well i think companies can improve their predictions and i think they can create their own
  38. set of super forecasters and they can do better at forecasting the future um and better is really relative i mean
  39. all you have to do is be better than the next guy we're not looking for perfection here
  40. and obviously we're not going to get it um in the case of the u.s government policy makers face decisions that
  41. involve billions of dollars and thousands of lives and in in these cases you know the
  42. stakes are so high that even a tiny little edge is huge progress um the the one of the implications
  43. of of our work i think is that good predictions involve both psychology and statistics it's a combination of
  44. understanding the person and then understanding the the aspects of of uh statistical distributions and
  45. statistical information we know now how to uh do much better at devising algorithms that
  46. aggregate multiple forecasts and we also know a lot about what conditions or environments people
  47. bring out the best of of of individual forecasters well every day we hear predictions
  48. we have pundits and experts and gurus and specialists making predictions about what will
  49. happen in the future and many of those predictions are so vaguely stated that we could never in a million years
  50. figure out how to test them there are statements like there may be an increase in conflict in yemen in the
  51. next two weeks or the situation in baghdad will get worse before it gets better or
  52. something like that that's that's not a question or a statement that passes the clairvoyance test you could
  53. you can't figure out later who's right and who's wrong now we look to these people for
  54. advice we look to them for insights and we're getting really very little from those vague predictions and i think
  55. the way we can do better is to keep score we've got to have questions or predictions stated
  56. in such a way that they're testable and then we can find out who's right and who's wrong
  57. and how we can learn to do better we are in a fortunate position of being able to have
  58. an empirical basis for the claims we make we have we know what works based on data based on experiments and many of
  59. the books and methods and techniques for doing better forecasting are simply not tested so clearly
  60. the empirical side of things is something that is unique to our project i think we've learned a lot about
  61. uh what makes things better we've learned for example that survey formats with statistical
  62. algorithms combining the forecast can outperform prediction markets we've learned that if you measure
  63. probabilities while people are trading in the market you also ask them what's your probability that event x
  64. will occur you can do better at forecasting accuracy by combining both the prices and the probabilities
  65. and these things are both surprising from an economic perspective we know a bit more about individual
  66. differences that correlate with forecasting accuracy so we can say a little bit more about
  67. who the best forecasters are especially in this geopolitical context not surprisingly they're
  68. they tend to be smart they tend to know a lot they have a lot of political uh knowledge but they are also
  69. more likely to be actively open-minded thinkers they are more analytical they are more likely to take a scientific
  70. world view they're more likely to take multiple perspectives on a question and use multiple reference
  71. classes they're more likely to use probabilities in a more granular or nuanced fashion they're more likely to
  72. say 17 and 83 percent rather than 20 and 80 percent and it turns out that extra granularity
  73. has information in it if you round forecasts up to the nearest 10 20 30 40 most forecasters do worse and
  74. that suggests there's there's valuable signals in that granularity not just noise
  75. well uh the irb tournament will close on june 2nd in in another month but that's not the end of forecasting we
  76. will be opening the good judgment project will be opening a public tournament this fall
  77. and we have lots of hypotheses to test we'll need lots of volunteers and we'd love to have people
  78. affiliated with wharton so if this is something that interests you or you know people who would be
  79. interested go to www dot goodjudgementproject.com to get more information and we'd be
  80. extremely grateful and delighted to have you join us you

Brief Summary

The episode covers forecasting accuracy improvements through training and teamwork, emphasizing the importance of testable predictions.

Key points

  • Creating Super Forecasters. Identifying top forecasters and forming elite groups significantly boosted prediction accuracy.
  • The IARPA Forecasting Tournament. A multi-year competition involving university teams to improve forecasting accuracy.
  • Surprising Insights on Teamwork. Teams outperformed independent forecasters, challenging previous assumptions about groupthink.
  • The Importance of Testable Predictions. Vague predictions hinder progress; clear, testable statements are essential for improvement.
  • Future of Forecasting. The Good Judgment Project aims to continue improving forecasting methods with public participation.

Tension

40/ 100Low tension

Related episodes