The 2025 NFL Season Through Data: DVOA, Analytics, Awards, and Predictions

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

ABOUT THE EPISODE Aaron Schatz, chief analytics officer at FTN Fantasy and founder of Football Outsiders, joins Eric Bradlow to explore how DVOA and play-by-play analytics challenge conventional narratives about the 2025 NFL season, conference strength, playoff probabilities, and the growing influence of data in awards voting. Plus, Eric walks through real-world examples of using generative AI to forecast NBA win totals, college football playoff probabilities, and NFL Super Bowl odds, highlighting how modern models apply Bayesian reasoning, betting markets, and simulation-based analytics.

ABOUT THE PODCAST Sports is a game of numbers. Wharton experts Eric Bradlow, Shane Jensen, Cade Massey, and Adi Wyner team up to tackle the world of sports, from current events to longstanding issues such as: What sports streaks are the most impressive? How do you rank the best players? Can athletes be compared across sports? Moneyball Highlights explains how decision-makers in the game can avoid the common mistakes and embrace the data. Episodes are recorded at the Wharton School.

#NFLAnalytics #FootballAnalytics #SportsData #AIInSports #SportsPodcast #NFLAnalytics ----- The Wharton School of the University of Pennsylvania is committed to sharing its intellectual capital through Knowledge at Wharton, the school’s online business analysis journal.

Launched in 1999, Knowledge at Wharton offers free access to:

- Articles, podcasts, and videos highlighting Wharton faculty research and analysis of current business trends - Interviews with book authors from Wharton School Press - In-depth series of curated content like Wharton Executive Education’s Nano Tools for Leaders - A searchable database of more than 10,000 articles covering all aspects of business - A weekly newsletter that delivers Knowledge at Wharton insights directly to your inbox

Visit the Knowledge at Wharton homepage: https://whr.tn/3qUpSBE Subscribe to the Knowledge at Wharton YouTube channel: https://whr.tn/3DEAgAU

Read the full transcriptHide the full transcript
  1. Welcome, welcome to Wharton Moneyball, the podcast edition here on the Wharton podcast network. This is Eric Bradlo,
  2. professor of marketing, statistics, and data science here at the Wharton School.
  3. As everyone who's listened to us for the last 11 plus years knows, some combination of myself, Kate Massie, Audi
  4. Winer, Shane Jensen, sports never stops. Data science and statistics never stops.
  5. So, we're here every week on Wharton Moneyball. I've always said that one of my greatest honors of doing this show
  6. again for the last 11 plus years is having people that are living, breathing analytics applied to sports all day
  7. long. I just do it as a part-time job. I got the regular job doing my own research, teaching other stuff. And
  8. certainly today is no exception. Um, we're joined today by the I don't even know how many times Aaron's been on the
  9. air. I'm going to say 10 plus. It's got to be more than that because whatever. We're joined today by Aaron Shatz. is
  10. chief analytics officer of FTN Fantasy. A lot of people also know his work on ESPN. A lot of our listeners certainly
  11. know him as the founder of uh Football Outsiders. A lot of people also know him as the creator of Advanced Statistics,
  12. including DVOA. And something I just learned today, and we're going to talk about this later on, is he's one of the
  13. members of the Associated Press that votes for the AllP Pro team and the regular season awards. So Aaron, uh
  14. welcome back to uh the Wharton Podcast Network and Wharton Moneyball. Hey, thank you for having me back. Always
  15. good to be here, man. >> Well, this is obviously a a great NFL season. It's The reason I'm going to say
  16. that and I'd love your a kind of analytical look at it is, you know, I can't tell you who's going to go I can't
  17. even tell you in my view who's going to go to the Super Bowl. I think this is just a year with more uncertainty. You
  18. know, it's like I think Shane read the stat last week. This is like the first time in 20 years there's been no Patrick
  19. Mahomes or Tom Brady in the Super Bowl. And so I'll just ask you, let's just start at the highest level. How do you
  20. see the 2025 NFL season right now? Who are the true contenders? And if you had to list, you know, I always like to give
  21. the I'll call it the paro curve. like how many teams would I have would you have to give me to make it like an even
  22. bet between making the playoffs and between going to the Super Bowl and not? How do you view the AFC and the NFC
  23. right now? >> Well, the first thing I'll say is it's a very topsyturvy year. A lot of weird
  24. stuff has happened. There are a lot of teams where their record doesn't really match what the underlying statistics
  25. suggest about how good they are, right? like Carolina, New England, Chicago don't seem quite as good as their
  26. records. Detroit, Kansas City, Indianapolis have played better than their records. Uh, some of these are
  27. teams you would have expected in the preseason, some of them are not. Uh, I used my current ratings, current
  28. ratings, and ran a simulation of the whole season. And in that simulation, the amount of times that Baltimore,
  29. Detroit, Cincinnati, and Kansas City all missed the playoffs was 1.1%. >> Wow. >> And that's not using preseason
  30. projections. That's using how good the teams have actually been this year. >> So, we always like to ask that question.
  31. So, now the question I I I'll ask you, which is probably not a surprising question, is it could mean one of a
  32. couple things. It could mean the thing that we all have in our simulations, which is not enough uncertainty. And if
  33. the answer is there isn't enough, where is it coming from? Another possibility is um 1.1% of events happen 1.1% of the
  34. time. So, you know, let's not let's not go overboard that you said 1 in 20 million. So, you didn't say that. The
  35. third, of course, is your point estimates of the team's strengths could also be off. It's not an uncertainty
  36. issue. It's kind of >> It's an imperfect science. Yeah. >> Yeah. It's an imperfect science. So
  37. which of those three explanations or all a combination of all three do you think
  38. it is? >> I think it's a combination of all three. But the big thing is just that events
  39. get ordered in different ways. And so teams play one game where they're like outstanding the whole game and then
  40. another game where they lose by three points. And this year is filled with teams where they've just been either
  41. really really good in one-score games or really really bad in one-score games. The other thing that's going on is the
  42. is we'll talk about this in a little bit. I think schedules being very extreme because of the way certain divisions are
  43. good and bad this year. There are certain teams that have very extremely easy or extremely hard schedules. Um, as
  44. far as uh the NFC is the better conference, that's the first thing. Uh, I my numbers differ from market. Is the
  45. NFC the better conference on let's say whether it's average or median or do you think the top end because those are
  46. different statements than saying the top end of the NFC is better than the top end of the AFC or both.
  47. I would say both are true, but in particular the top end of the NFC. Um, the market I have Seattle and the
  48. Rams much farther ahead of the rest of the league than a lot of other people who do advanced analytics.
  49. Uh, the reason for that is almost everybody else who does advanced analytics uses some form of EPA, right?
  50. Expected points added with some sort of adjustment. And I don't I use my own stat which is
  51. DVOA. And because of some of the ways that DVOA works and EPA works, Seattle and
  52. the Rams come out a lot higher in DVOA than they do in EPA. One, some of it has to do the increasy intricacies of, you
  53. know, certain amount of yards and a certain down and distance. And some of it is easy to explain stuff like I
  54. downweight turnovers because they're less predictive. And Seattle has a lot of turnovers. So if you do EPA,
  55. Seattle's offense is not going to come out as good as they do in DVOA because I'm downweing the turnovers because less
  56. predictive. So because of that, I have Seattle and the Rams at almost a 5050 shot to win the Super.
  57. >> I just want to be clear them com just to be clear. Are you saying if you take the
  58. Rams and Seattle, you have the Rams in Seattle, I have the rest of the league and that's a fair bet. Is that what
  59. you're saying? >> My numbers would say yes. It's a surp It's surprising and I don't know whether
  60. maybe I have them a little too high and no one else would have this, but yeah. I
  61. mean, everybody has right now has Seattle and the Rams as their top two. >> So, just give me an or this is what I
  62. was going to ask you. whether one's EPA based or DVO based. Can you give me a sense? Or even if I went on to ESPN,
  63. FBI, whatever it was, how much probability would they give combined? Are you at 50 and they're at 20 or
  64. you're at 50 and they're at like 35? I I always like to get a sense of the order
  65. of magnitude of these discrepies differences. I can look up ESPN while we're here and tell you that for Seattle
  66. and the Rams, they have them winning the Super Bowl a combined 27% of the time. >> Wow. So, you're almost twice as high.
  67. So, that's we would consider >> past everybody when it comes to those two teams. Yeah.
  68. >> Yeah. So, I just just for myself maybe I'm thinking about this wrong. I'm pretty sure most of our listeners and
  69. this is the great thing about Wharton Moneyball is saying, "Wow, would Aaron really give me even money despite like
  70. someone we both agree someone from the AFC is making the Super Bowl. I don't know who it is, but someone is." And we
  71. could argue they have almost a 50% chance to win that game. So, in your view, well, let me ask a question. So,
  72. there's this is the wonderful thing about winning the Super Bowl. We agree two things have to happen for
  73. you to win the Super Bowl. If you're Rams or Seattle, first you have to win the NFC. Then you have to go to the
  74. Super Bowl and win it. Which of those two numbers is really, really high? It sounds like it might be both. Like
  75. here's something that could yield 50%. >> The combined probability is 7070. That
  76. would get you to 50%. Like they're the one of the two of them has a 70% chance of going in total. And if they go,
  77. they're a twothirds favorite. Roughly that get you to 50. >> So my ratings are based on the idea of,
  78. you know, how efficient is a team compared to average. >> Seattle and the Rams are two of the 10
  79. best teams since 1978. >> Measure. I know I have them way ahead of everybody else. I know exactly how
  80. you're measuring that. It's based on a play-by-play breakdown and the success on each play is compared to a baseline
  81. that's adjusted for situation and opponent. And one thing is that I include special
  82. teams and a lot of people who do advanced stats don't include special teams because special teams is less
  83. predictive than offense and defense. But it's not not predictive and I think Seattle and Rams game that the fact that
  84. Seattle has really good special teams. So just to be clear, I compute this >> I compute this metric for every play and
  85. then I sum it over all the plays. >> Yeah. Yes. >> Okay. So Okay, I see. I just wanted to
  86. understand the Okay. Wow. >> So just to give you an idea of where the numbers are, Seattle and the Rams are at
  87. like 42 43%. The next highest team that will make the playoffs is like Jacksonville right now, which is
  88. at like 20%. Like that's the gap I have between the Rams and Seattle and everybody else. I
  89. don't know if I would really do the Rams and Seattle versus the Field to win the
  90. whole thing. >> If you gave me the 49ers though, I might do it. If you gave me the NFC West
  91. versus everyone else. >> You almost made the whole MS NFC West. >> I might I might do that.
  92. >> Huh. That's very very very interesting. Um, what aspect? Well, first of all, could you just tell us since you must
  93. since you're making a statement that they're in the top 10 since 1978. Can you give maybe I I don't know if you
  94. have this handy, can you tell me since I'm a football historian just like you are, can you tell me who else is in that
  95. list or do you have that list handy? >> The top three through this is through 15
  96. games. The top three are the 2007 Patriots. >> They were pretty damn good. The 1991
  97. Redskins. >> They were good. >> The 1985 Bear. >> I knew the 85 Bears were going to be in
  98. there somewhere. Wow. Yeah. So, >> you also get some teams that didn't make it all the way like the 2010 Patriots
  99. and the 1983 Redskins who lost to the Raiders in the Super Bowl and the 9549ers who got upset by the Panthers in
  100. the divisional round. And then for some reason the last three years I have had a
  101. ton of teams come out as historically good in a way that hadn't happened for a dozen years before them. The 2023
  102. Ravens, the 2023 49ers, the 2024 Ravens, like come out as really historically good. And I don't know why because I
  103. haven't changed what I'm doing. And yet there weren't any teams that were anywhere near that high from 2011 to
  104. 2022 and then all of a sudden in the last couple of years there have been these teams that are ridiculously high.
  105. H wow. Um so because if you had asked most people I think I they would say that the Rams and the Seahawks are
  106. better appear to be better than others this year but not I'll call them. Well, I don't think anybody would put them
  107. among the historically great teams. >> And it's it's it's a little weird because of the fact that I've had this
  108. going for a couple of years and those teams have not won the Super Bowl. But it's not like my ratings always come out
  109. with modern teams that high because from 2011 to 2022, there were no teams that high.
  110. >> It's just the last three years. >> So, I'm going to ask you a question that every listener here on Wharton
  111. Moneyball. And again, I'm I'm joined here by Aaron Shats. Aaron is the chief analytics officer at FTN Fantasy.
  112. Everybody also knows him from his original work at Football Outsiders, his work at DBA,
  113. his work with DVOA. So, let me just ask you, I'm always going to ask these kind of questions because I know the answer
  114. is no to each of them, but what the hell? I'm going to answer them anyway. Rams and Seahawks.
  115. Um, if I'm correct, the quarterback for the Seahawks is Sam Darnold, right? >> Yes.
  116. So you have no concerns? >> No, I have concerns. >> Okay. So, so let me ask you a different
  117. question. Is that brought mathematically into your model anywhere that it's Sam Darnold's?
  118. >> No. And it's something I want to look at doing in the future, which is should I
  119. have an additional career history uh variable for quarterback that goes past this year all the way back multiple
  120. years of whether you trust a certain quarterback compared to other quarter. It's clear that Vegas does.
  121. >> Absolutely. >> Because there's no other reason why they have the Bills as the favorite to come
  122. out of the AFC. The Bills have not been by any metric the best team in the AFC this year. The only reason to have them
  123. as the favorite to come out of the is Josh Allen. Is the idea that you have more historical trust in Josh Allen than
  124. you do in Drake May or Bo Nicks or Trevor Lauren? >> So, which team in the AFC has been the
  125. best team based on your Jacksonville? Is it Jacksonville? Uh, if you look over the course of the entire season, it's
  126. Indianapolis. If you weight it towards more recent games, it's Jackson. Where do you have I have debate with my
  127. kids about this all the time who are big analytics guys as well. Isn't I mean, they they believe
  128. Houston's a fraud. Where do you guys have Where do you have Houston who I think's won at least eight straight if
  129. I've got this right? >> We have them uh fifth. >> We have fifth in the NFL. a fifth in the
  130. NFL. They are number two on defense, but they're only 21st on offense. >> And there is that thing where offense is
  131. a little bit more predictive than defense. And that is that is true, right? So, you're like, okay, maybe
  132. they're not quite that good going forward, but that defense is really, really good and the offense has played
  133. better in recent week. >> Um, and okay, very interesting. So, let me ask you another question. So, you've
  134. met you made a comment in the notes here about the Patriots and their schedule. Um, could you talk more not only about
  135. their schedule, but maybe this is why one of the reasons why we have advanced analytics because you can't just look at
  136. one loss record and so could you talk a little bit more about which teams you see where and you mentioned it a little
  137. bit when you're in your opening remarks about who's one lost record or how about
  138. the following. Let's say I did a by a graph of w number of wins on the x- axis and whether it's Aaron Shatz's measure
  139. of team strength on the y ais. Which teams would have the largest deviation from that 45 degree line?
  140. >> And a lot of these teams, anybody doing advanced analytics has the dev deviation? the Panthers,
  141. the Bears, the Patriots, and on the other side, the Lions, the Chiefs, and the Colts. Although obviously the Colts
  142. and Chiefs have a different quarterback now than they did for most of the season, and that affects how you would
  143. predict things going forward. But as far as how they've actually played during the year, the Colts and the Chiefs have
  144. been better than their record. And the Lions have The Lions are going to be one of the top DVOA teams ever to miss the
  145. playoff. >> Wow. So, I was going to ask you that also. Do you ever as a I'll call it a
  146. face validity check. Let's imagine that every year DVOA based strength um has lots and lots of outliers or deviations
  147. where the DVA DVOA based strength does not match let's call it the win loss. Do you ever start to say maybe that's a
  148. deficiency in DBOA because there's of course there's randomness within a given year but like across a long period of
  149. time eventually you know if I look in totality it should be properly calibrated.
  150. >> Yeah. I mean there's a lot of variance in sports and there's a lot of variance
  151. in the NFL. It's never going to be properly calibrated but no I'm always looking to improve it. I'm always
  152. looking to improve the predictive ability. I'm always looking to improve, but a lot of these uh it's interesting
  153. for a lot of the teams that I just described, the year-to-year DVOA is more consistent
  154. than their year-to-year win loss records. >> Well, that's and that's what you would
  155. hope. >> Like Detroit, Indianapolis is an exception, but Detroit was good last year. Kansas City was good last year.
  156. Carolina was not good last year. So, the idea that Carolina is not as good as its
  157. record makes some sense if you think about who they were last year. And the idea that the Chiefs are better than
  158. their record makes sense if you think about who they were last year, >> right? Is it Am I correct in saying this
  159. is just from my I haven't looked at it analytically, but I would just give an opinion that Well, I think I know, maybe
  160. I don't know. I was going to say is Well, it must be from your answer. The strongest conference division, sorry, in
  161. football has to be the NFC West, right? Given you just said the Rams and Seattle
  162. are historically strong. You said you would take the 49ers. I think each of them has at least 11 wins at this point.
  163. Um, and so that's the strongest and the NFC South is the weakest. >> Yeah. Oh, yeah. The AFC South was
  164. shockingly strong this year. The AFC South, >> that was going to be my second guess.
  165. >> Really surprised. Indie Jacksonville really surprised. Um, Houston, I think people thought was going to be, you
  166. know, just what they are, which is a great defense with no offensive line. Um, but uh right now in my ratings for
  167. the whole season, so not weighted to recent game. Uh, the Colts, the Texans, and the Jaguars go four, five, six. But
  168. with the NFC West, those two teams are so far ahead and the 49ers are also good that Yeah. the NFC West is the best
  169. division. And the whole NFC South is in the bottom half. >> Wow. Yeah, because I you may remember
  170. I'm a Buccaneers fan through my cousin, etc. Uh what's happened the last three weeks to the Buccaneers has been
  171. embarrassing. Like I mean >> they're the best team in the division. Why can't they win games? It's very
  172. frustrating. >> Yeah. I mean they are the best team. That's clear. And now they have all
  173. their players back. So there's no excuse for that. And they've just they lost at
  174. home to the let me think if I got this right. I was at the game with the Saints so I know they lost that one and then
  175. they lost to the Falcons at home and then they just lost to the Panthers on the road. So they just lost to, you
  176. know, three teams in their division and they all stink. >> Yeah, it's I can't imagine. It's very
  177. frustrating. Yes. >> And of course they still control their own destiny which is also the crazy
  178. part. I mean they may end up going eight and nine and winning the division because the accord to you that's likely
  179. to happen because uh the Panthers play Seattle this week and that's a big game for Seattle. The Bucks play the
  180. Dolphins. Well, the the Bucks probably >> starting a rookie quarterback. They're gonna be
  181. >> they should probably beat him. The Bucks would have to win the division at nine
  182. and eight because somebody's going to have to win the last game of the season between those two teams. But that may be
  183. a historically weak division winner. And whoever the five seed is um might be happy they're the five seed.
  184. >> I mean, there are other divisions, the 2020 NFC East, the 2004 NFC West. There
  185. have been other divisions that are weaker than this, but this is pretty weak. >> Now, the next thing in the notes that
  186. you helped us uh create here, which I I really appreciate that, is um I assume when I asked you the question about DVOA
  187. and your it being based on playbyplay um and then you kind of add it up. Um you put in the notes here the s the
  188. search for historical playby-play data and the upcoming edition edition of the 1977 season. So, is it fair for me to
  189. say that what you and I completely agree with this by the way that you've built most of your analytical career for the
  190. NFL on having this playby-play data as opposed to just aggregated statistics and that's why
  191. >> Yeah, please go ahead. >> When I got started, there were people who were like, "Yeah, we did analysis
  192. based on playbyplay and there was nothing based on playbyplay in it." And I was like, "Let's do something based on
  193. playbyplay." And then I started collecting old years and a lot of it's been with the help of Jeremy Snyder who
  194. people know on social media as quirky research. >> Okay. >> And he has helped me and we have tracked
  195. down and we keep going back and back and back. So we've got back to 1978 now. We
  196. have transcribed things off of video. We have found old game books. Um and we have enough to do 1977 this off season.
  197. And it is no longer just mine. I have shared it with Pro Football Reference. Pro Football Reference now has all the
  198. playbyplay going back to 1978 on their site, searchable with all the playby-play logs for all the games. I'm
  199. very proud of doing that. It's been I think it's a really huge addition to the ability to study historic football
  200. historically. And it's great. It's so much fun for me when I'm like, "Oh, yeah. The best teams to miss the
  201. playoffs, the 1979 Redskins." Like, I can go back that far and say they're one of the best teams that didn't make the
  202. playoffs. You know, let me ask you a question. This might seem naive, but I'm sure a lot of our listeners here on
  203. Morton Moneyball are asking might be thinking the same. What do you exactly mean when you say playbyplay data?
  204. Because one could dream. Let's let's forget what you have. I'm going to answer what my dream might be and then
  205. you're going to tell me the data that one actually would have conceptually. You could have video motion data of
  206. every player and their XY coordinates on every single play and all of and I know
  207. that's not the data you have but so what do you mean by playbyplay data and how do you compute kind of like an expected
  208. performance given that data >> for a given play? >> All my years are normalized so that
  209. average is zero. So the same goes all the way back to 78. Every year is normalized so that I'm comparing teams
  210. to just that year. And what I have is basically everything that was in the playbyplay up to 2004.
  211. So like is it a pass or a run? Who was the player? Who was the receiver? What was the down
  212. and distance? What was uh the time is not in there for a lot of old years until the last two minutes of each half,
  213. but we have the time to start each drive. And then the further back you go, the more spotty it gets as far as like
  214. the direction of runs or tackles or passes defense, but we have like who had sacks and who has interceptions and all
  215. that data. Uh but we have targets, right? Like no one ever had targets for the 80s before before we collected that.
  216. So, not just receptions, but targets like is in this data. Um, we don't have um yards after the catch and air yards.
  217. That doesn't start till 2005. >> We don't officially have whether a quarterback run is a scramble, but I've
  218. gone back to a lot of video and answered whether a lot of those are. Uh, we don't
  219. have a quarterback knockdowns. That didn't start till 2006, I don't think. And we don't have all the kind of
  220. charting data that you now get from FTN or SIS or PFF. Not we don't have any of that going back before 2005. But
  221. everything that people had from 99 to 2004, we basically now have for 78 to 98. >> H I I can imagine that's ext and then
  222. can you give us a sense of how just you know maybe give me an example of how you
  223. score a play. So, how is a play score? >> Well, like you know, I give it a certain
  224. number of success points based on the yards and the down and distance, right? Success in my percentages is you have to
  225. get 45% of needed yards on first down, 60% on second down, and 100% on third or fourth down, but there's like partial
  226. numbers. It's it's not it's not it doesn't go zero and one. It goes like zero and 2 and 0.5 and 7, etc. And then
  227. you get extra bonus for the extra yardage. So a play will come out with something like 2.3 success points if
  228. it's like a 10 yard pass on first and 10 or something. And then I compare that to
  229. what's the average for first and 10 passes in that area of the field. >> Yeah. Okay. So that's what I meant. I
  230. was also asking what do you condition on when you compare a play? So it would be
  231. similar >> distance area of the field uh and then adjust for the score, right? You know,
  232. obviously it's easier to gain yardage if you're losing by 40 late in the game. >> So, this is actually a pretty opponent
  233. adjustment based on the quality of the opponent. So if I just look at this as a big contingency table, this is a very
  234. highdimensional contingency table because I mean there's not that many downs so that's not high dimension but
  235. distance maybe have them uh blocked or you know bundled into ranges or something and then there's maybe there's
  236. score difference and then there's maybe >> what I try to do is smooth the curves.
  237. >> Okay. So there is a smoothing that's done but I didn't have like I I don't think it makes sense to say that eight
  238. yard gains are more common than seven yard gains. It's more likely that our sample size isn't big enough. It what
  239. what's logical is to say that you know you gain one more than you gain two you gain two more than you gain three you
  240. gain three you know etc. like it's it or I guess four is the average and then the
  241. curve goes like this, right? But like um you know for judging between first and 20 and first and 25, you sort of group
  242. that together because there aren't enough actual like >> first and 21s that are out. You know,
  243. there aren't that many first and sevens that aren't in the red zone. >> Makes a lot of sense. So let me get to
  244. the last topic for today that I want to talk to you about which is as I mentioned in when I introduced you to
  245. start with you're one of the I I imagine a tremendous honor for you and one welldeserved as a member of the
  246. Associated Press that votes uh for both the regular season award and the allp pro team. Um one of the topics you
  247. brought up in your notes was the growing role of analytics in awards voting. So that could mean many things. It could
  248. mean um the number of people that you and I out of those 50 would consider as analytics oriented is going up. That's
  249. one way to measure it. Another way to measure it is um you have knowledge of the analytics that people are using.
  250. Like here are the stats that people use or those are provided in some way. What did you mean when you said the growing
  251. role of analytics in awards voting? >> More the first than the second, but they're definitely both true. I was the
  252. first analytics person to be on the panel. uh starting in 2021, but they've done a
  253. lot of changing of the panel over the last few years. So now you have Mina Kines
  254. >> and you have Sam Monson who used to be with PFF and you have Doug Ferrar who's
  255. more of a film guy but understands analytics used to write for me at Football Outsider.
  256. The other thing though is if you think about it, people who are not necessarily analytics people have access to more
  257. analytics now than they used to. Think about a guy like Dan Orlovski, right? Dan is a former quarterback. He
  258. is a film guy, but he works for ESPN and he's very conversant with all of the ESPN advanced stats that ESPN stats and
  259. info does. And I'm going to guess that when Dan puts his ballot together, he's going to use some of that stuff. Now,
  260. he's more probably going to use his eyes and his scouting, but he's going to use
  261. some of that stuff that didn't exist a few years ago, or at least there was nobody on the panel that would use any
  262. of that a few years ago. >> Right. >> So, I think even the people who are not necessarily analytics people are going
  263. to use more analytics than they would have used 10 years ago. >> Can you point I don't want to talk about
  264. this year because I know that's not only is it off limits, you shouldn't talk about it because you're a voter, but we
  265. can talk about the past. So can you talk about instances at least in your own mind that somebody was like wow if it
  266. wasn't for analytics that person never would have been voted all pro or never would have been an or maybe that's off
  267. limits too just like I was just not that your vote I'm not asking for your vote or anything like that
  268. >> I'm thinking because I have an answer when it comes to my vote but I'm trying
  269. I don't know about for the whole allpro team or the whole or specific awards that I think of anyone like that, but I
  270. can think of one where I was very different. I'm very famous among, you know, people who understand this stuff.
  271. When Lamar Jackson won MVP in 2023, he got 49 of the 50 votes and I was the exception.
  272. >> So, can you tell Well, now that you said that, can you tell us about that and
  273. what what did you vote? And >> I wrote a whole thing about how no matter what stat you looked at, he came
  274. in below Josh Allen and Dak Prescott. >> My advanced stats, ESPN's advanced stats, PFF grades, like SIS, anything
  275. you looked at, he came in below those guys. And I was like, I just can't vote for him for MVP when he's not number one
  276. and or two or two in any staff. And so I went with Josh Allen. Wow. Well, let me just say uh for those
  277. people that follow us on whether you want to call it X or Twitter, W Moneyball, um there's no doubt I'm going
  278. to be if we still call it tweeting, I'm going to be tweeting that out because that in itself is fascinating. It's also
  279. fascinating to me that since let's assume whether people use it or not as a separate issue, that had to have been
  280. sort of known by the voters. And so that that to me is very interesting. Let me ask I'm just asking you for speculation.
  281. If that were to happen today, 5 years from today, you agree the vote you're I mean we would both hope the vote
  282. wouldn't be 49 to1. >> Today it's still there's still narrative, right? Cuz Allen beat Jackson
  283. last year even though last year the opposite was true. Jackson had better numbers.
  284. >> Interesting. But in five years, I don't know if somebody whose numbers are as
  285. low as what Cam Newton's were in 2015 is going to be able to win the MVP. I don't
  286. know. I mean, you you'll have you'll certainly have places where the person who finishes third or fourth will beat
  287. the person who finishes first in a stat, especially because different metrics have different players number one. But
  288. five or six years from now, you may not have a situation where you have a guy who's 10th or 11th in stats, but he gets
  289. a narrative going, >> right? >> And his team wins a lot of games because of their defense and he gets a narrative
  290. going and then he wins the award. You may not have that anymore in a few years. Well, let me just my last question now
  291. is um can you tell me a little bit about what you're what you guys are working on
  292. that you can talk I'm not necessarily the nuts and bolts but what are you guys doing at FTN Fantasy that our listeners
  293. might be interested in like or you even personally like what's the big project you're working on now
  294. >> I mean big projectwise I don't have anything in particular except you know still always looking to improve things
  295. in the offseason you know improve predictive ability um you know the thing I'm really proud of at FTN is because we
  296. do all of charting in our FTN stats hub. If you're a subscriber to the site, ftnfantasy.com,
  297. you get DVOA for all kinds of charting uh categories. So, you can look at who's the best against man coverage or zone
  298. coverage or with two high safeties and which teams are the best with certain run concepts, outside zone, inside zone,
  299. power, man, etc. I love that stuff and I love doing stuff to figure out what's real and what is just really variable.
  300. And then working on the 1977 and then the other thing that FTN is doing that I have nothing to do with but
  301. we are doing and is cool is basketball charting. and we're going to have a bunch of basketball stuff coming out uh
  302. for people who enjoy basketball analy well Aaron I'd like to thank you for joining me here today on the podcast
  303. edition of Wharton Moneyball I've been talking to Aaron Shatz chief analytics officer FTN Fantasy uh apparently it's
  304. ftnfantasy.com right is the uh is the website you can also read >> called FTN Fantasy we do a lot more than
  305. just fantasy it's just we had to get it all under one ceiling and that's the ceiling we went
  306. Well, again, Aaron, thank you again for joining me today on Wharton Moneyball. >> Hey, thanks for having me, man.
  307. >> Welcome back to Wharton Moneyball here on the Wharton podcast network. This is
  308. Eric Bradlo, professor of marketing, statistics, and data science here at the Wharton School. Uh, I just got done and
  309. finished talking to Aaron Shatz of FTN Fantasy, chief analytics officer. Had a great discussion of lots of different
  310. topics. Um, and uh, some combination of myself, Audi Winer, Shane Jensen, and Cade Massie are here every week on
  311. Wharton Moneyball. Uh, since it's just me, um, and there's the a quote second half of the show, I did something
  312. interesting this week, which I hope all of our listeners uh, will want to will be interested in. I decided that since
  313. I'm Wharton's vice dean of AI and analytics, uh, why not use Generative AI to answer a bunch of questions that I
  314. have uh, I'd like to hear the answer to. So, I'm going to tell you what I did. I'm going to tell you the the exact
  315. prompt I typed in, and I'm going to tell you how, in my case, chat GPT. You could
  316. use your own large language model, but Wharton gives us a free enterprise access to chat GPT. Um, I'll tell you
  317. exactly what I said, what I asked it, what the prompt was, and what it found. And by the way, just for all of you that
  318. want to know, this was chat GPT 5.2, which is the newest version. And I also used it in thinking mode trying to get
  319. the if you'd like the highest quality answer for anybody that wants to reproduce what I did. So the first
  320. question I asked it was give me a 95% confidence interval for the number of wins for the Oklahoma City Thunder in
  321. the regular season. So let me say a couple things. I'm interested in that because they're 26 and three. They
  322. certainly have the potential to break the all-time record of 73-9 by the I think it was 2017 Golden State
  323. Warriors who lost the title as everyone remembers to LeBron in the finals. Um but I'm also interested into how much
  324. uncertainty it gives it. Okay. And so the first thing is what this is first something that differs from older
  325. versions of large language models. You may remember like when they first came out like we've only trained the date up
  326. until January the 1st of two years ago. Well, that's all gone now. Now, I think everybody knows that the first thing it
  327. said is as of December 23rd, their record is 26 and three. And they literally played last night. So, they
  328. know that they played last night. That's in the database. So, that's good. Um, it
  329. said it leaves 53 games in the regular season. Sounds good. Um, then it did something more spe more specific and
  330. impressive and it started thinking and it would come up on the screen what it was thinking about. It says, "Look,
  331. we're going to fit a beta binomial predictive model." So, let me just be clear to everybody what that means. So,
  332. you can treat the 29 games they're played as a binomial, a coin flipping model. 26 have come up head success.
  333. Three have come up failed. That means they've lost. That's the binomial part. And then we're going to put a prior
  334. distribution on that coin flip. And usually most people put a beta distribution. You can view that as an
  335. equivalent number of wins and losses. So for example, if I said it's a beta coin
  336. with a 1010, that would say my prior is 10 wins and 10 losses. I would add that to the 26 and three. I'd have 36 wins
  337. and 13 losses and that would be my predicted win probability for the rest of the season for the 53 games. So first
  338. I was impressed that it knew about basian inference that well. It knew that doing some sort of shrinkage, which a
  339. beta binomial is going to do because no one's going to predict that they're going to go, you know, 950 or whatever
  340. 26 and 3 is for the rest of the season. You're going to have to shrink it down to some prior and a beta binomial gives
  341. you a formal way to do it because it gives you an effective sample size. We know the observed sample size 29, but
  342. the prior is going to give us an effective sample size. It then said the 95% interval is from 64 wins to 78 wins.
  343. Now, first thing that surprised me was the 78 number. I mean, that number I mean, they're not winning 78 games.
  344. They're not going 52 and one the rest of the season, but I will say it also computed this simulation game by game,
  345. so it knew the schedule. 64, I think that's just way too low. I mean, you say, well, what about injuries? Well,
  346. the first like month of the season they played without their second best score, Jaden Jackson. So, they're not I mean,
  347. maybe um I just thought it was an interesting interval. I guess it's centered around 71 or 70 or 71 which
  348. seems quite plausible to me that they might do but either way. Um and then lastly it said for a reference a more
  349. optimistic assuming their current win percentage continues exactly it leads to an interval of 69 to 77 wins which means
  350. centered at 73 plus or minus 4. So you could view it as a standard deviation of two. You're going to take two standard
  351. errors of either side. So you're 73 plus or minus 2 * 2 which gives you an interval from 69 to 77. Either way I was
  352. impressed by its thinking. I was impressed by its use of the beta binomial. I could have queried it more
  353. to use this prior. It would have done that calculation. It was simulating game by game. Not bad. Not bad. So the next
  354. question, since we're living in the world of college football, I asked it, what is the probability that the four
  355. semifinal college football teams are Indiana, OSU, Georgia, and Oregon. So that's the 1,
  356. two, three, and the five seed. The reason I picked the five seed Oregon is that they're favored over Texas Tech.
  357. And I said, give me an exact probability estimate and a 95% confidence interval.
  358. So, first thing it said is um it recognized the matchup, so it told me the matchups. That's not that hard. Um
  359. it then used the draft kings money line. It could have used lots of different money lines, but that's fine. It then
  360. gave me the probability of each of the teams that I won winning. It then said, let's assume they're independent. It
  361. multiplied them together, and it comes up with 17.94%. It then computed a 95% confidence
  362. interval based on those proportions. Um, it then gave me a range of 16.72 to 19.68%.
  363. U, that seems a little bit narrow to me. Um, and then it said here, if you want me to compute spreadbased probability
  364. instead of money lines, it could do that type of computation. So again, this one was interesting because it rep
  365. recognized that I was looking for a compound event. you know the probability of four things happening because I asked
  366. for exactly all four of them. It used the it found the money line to compute those probabilities. It multiplied them
  367. together um and it gave me a confidence interval based on these uh proportions and and what the likely uncertainty is
  368. in those proportions. So I thought that was pretty impressive. And again it's more to me that it's impressive compared
  369. to historically. If I had asked it six months ago, a year ago, I don't think it would have given me as precise an
  370. answer. Um, and last but not least, I I asked it um let's see here. I had one other thing that I asked it for. Ah,
  371. okay. Um, I asked it the question I always like to ask, and this time I did it for the NFL. I asked it, how many
  372. teams do I need and what would those teams be that would give a 50% probability to win the Super Bowl? In other words,
  373. if I thought there were four teams that added up to 50%, I I'm asking it to tell
  374. me the number four and who those teams are. Now, what's fascinating about this is I didn't I hadn't even thought about
  375. this at the time. I just asked Aaron Shatz that question if you remember during the first half of the show and he
  376. said in his mind it's almost two teams the Rams and the Seahawks almost add up to 50% under his DVOA model. Um that's
  377. not what uh but he also said under using betting lines they would be about 27%. And that's what um chat GPT 5.2 thinking
  378. found. It said first it's converting bet 365 odds to an implied probability and it gave me the formula of the implied
  379. probability which is the way we always do it. P equals 100 over the plus number plus 100. So you take if it's plus 300
  380. 100 over 300 plus 100 is the implied probability. It normalizes across all 32 teams to take care of the vig the
  381. betting vig that's in there. And then what it says, it gave me the top six teams that give 49.65%.
  382. Now to Aaron's credit, not surprisingly, the Rams and the Seahawks are at the top. Although according to this, that
  383. would only be about 22% probability. It then gave the Chiefs, who aren't even going to make the playoffs, the Bills,
  384. the Patriots, and the Broncos. And those six teams, it adds up to 50%. Now, here's the question. Certainly, I think
  385. Aaron would take the No, of course, the Chiefs aren't making the playoffs. The Bills, the Patriots, and Broncos are.
  386. Um, and by the way, the seven team for us Eagles fans are the Eagles here. Um, but I thought this was fascinating.
  387. Again, I just want to say for our listeners out there, use generative AI. I I want to say I get I this is not a
  388. specially trained generative AI model. It's the one that if you have chat GPT 5.2 2 and possibly in thinking mode. Um,
  389. you have the same ability I do to type in what is the number of teams X that are needed to get to 50% win
  390. probability, order them by the highest probability teams and tell me who the teams are. That's all I typed in and
  391. this analysis came in and it said here and again it said closest to 50% using the strict order of the teams is the
  392. number six. That's pretty impressive and this is a statistic we've been talking about to understand uncertainty in the
  393. league really for the 11 and a half years we've been here on Wharton Moneyball. So those are the three things
  394. I did on chat GPT 5.2. I'll try to do some more before next week's show and also post some on our Twitter account
  395. Moneyball. Um, just the last thing I wanted to talk about maybe in the um, in the last couple minutes I'm going to
  396. talk today is um, since everybody knows I'm a big tennis fan and we're trying to
  397. get very soon we're going to get Paul Anakone on and he would be great to talk to about this. Um, as many people saw in
  398. the last week Carlos Alcarez, the number one player in the world, the holder of six majors um, fired his coach. It'll be
  399. very interesting to see what impact that has on his performance during the year because one of the things we've always
  400. asked about is the impact uh of coaching and so you know he's keeping a lot of his team but his main coach Juan Carlos
  401. Ferrero uh former number one in the world uh winner of I think the French um is no longer his coach. It'll very
  402. interesting to see if there's any drop off or if there's any narrative. So well remember he also has to beat other
  403. players and he has to beat especially Yanx S. So you know can we decompose whether maybe S's just getting better or
  404. is it that Alcarez got worse? I think this is the kind of thing that we're going to need advanced analytics for
  405. because I'm not sure like he's never won the Australian Open. So we said, well, let's say he loses. We can't just say,
  406. oh well, if he had his other coach. No. So we're going to actually have to dive into the advanced analytics and
  407. understand, you know, is he actually performing worse? And then of course there's also MA is match to match and
  408. week-toeek variation. So we're going to have to disentangle that from it. It will not be an easy analysis. We're
  409. certainly going to have to watch it for some period of time and maybe use last year as a if you like a control group
  410. and compare his performance at similar tournaments on similar surfaces and see if there's some sort of market decline.
  411. It's an interesting counterfactual on how you might look at the impact of coaching. So this has been uh this
  412. week's show of Wharton Moneyball here on the Wharton podcast network. Again, I'm
  413. Eric Bradlow. On behalf of myself, Kate Massie, Shane Jensen, and Aie Winer like
  414. to thank you for joining us. I'd like to thank Marissa Rena, like to thank D Patel, and as always, I'd like to thank
  415. Dion Simpkins, our associate producer, sound engineer, the person that makes all of this happen. So, between now and
  416. next week, enjoy your statistics, enjoy your sports. We'll see you next week here on the Wharton Podcast Network and
  417. Wharton Moneyball.

Brief Summary

Aaron Shatz discusses NFL analytics, playoff predictions, and the evolving role of statistics in awards voting.

Key points

  • Welcome to Wharton Moneyball. The podcast kicks off with Eric Bradlo introducing the show and its focus on analytics in sports.
  • Topsy-Turvy NFL Season. Aaron Shatz analyzes the unpredictability of the current NFL season, highlighting unusual team performances.
  • Super Bowl Probabilities. A discussion on the probabilities of Seattle and the Rams winning the Super Bowl, revealing a significant discrepancy with market predictions.
  • Seattle and Rams' Historic Ranking. Aaron reveals that Seattle and the Rams are among the top 10 teams since 1978 based on analytics.
  • Frustration with Team Performance. The team has lost to three divisional rivals, raising concerns about their performance.
  • Weak Division Winner Prediction. Discussion on the likelihood of a team winning the division with a losing record.

Episode highlights

Tension

45/ 100Some tension

Related episodes