Rethinking How We Measure Soccer Performance
Ask about Knowledge at Wharton
Episode Description
ABOUT THE EPISODE What if one of soccer’s most important analytics metrics is missing some of the most dangerous moments in a match?
Recorded live with students from the Wharton Moneyball Academy, Jonathan Pipping, Ph.D. student and member of the Wharton Sports Analytics and Business Initiative Research Team, breaks down the development of XG+, a new metric designed to capture the scoring opportunities that traditional expected goals models miss. He explains how player tracking data can measure the probability of both taking and converting a shot, why creating shooting opportunities appears to be a more consistent skill than finishing, and how these insights could change the way teams scout and value players.
ABOUT THE PODCAST Sports is a game of numbers. Wharton experts Eric Bradlow, Shane Jensen, Cade Massey, and Adi Wyner team up to tackle the world of sports, from current events to longstanding issues such as: What sports streaks are the most impressive? How do you rank the best players? Can athletes be compared across sports? Moneyball Highlights explains how decision-makers in the game can avoid the common mistakes and embrace the data. Episodes are recorded at the Wharton School.
----- The Wharton School of the University of Pennsylvania is committed to sharing its intellectual capital through Knowledge at Wharton, the school’s online business analysis journal.
Launched in 1999, Knowledge at Wharton offers free access to:
- Articles, podcasts, and videos highlighting Wharton faculty research and analysis of current business trends - Interviews with book authors from Wharton School Press - In-depth series of curated content like Wharton Executive Education’s Nano Tools for Leaders - A searchable database of more than 10,000 articles covering all aspects of business - A weekly newsletter that delivers Knowledge at Wharton insights directly to your inbox
Visit the Knowledge at Wharton homepage: https://whr.tn/3qUpSBE Subscribe to the Knowledge at Wharton YouTube channel: https://whr.tn/3DEAgAU
Read the full transcriptHide the full transcript
- Welcome, welcome everyone to this week's edition of Wharton Moneyball. I'm Eric Bradlow, Professor of Marketing Statistics and
- Data Science here at the Wharton School. I'm joined today, it's a very special occasion
- for us here at Wharton Moneyball. I'm joined by my colleague, long-time collaborator
- and friend, Adi Weiner, Professor of Statistics and Data Science. He's actually right now in the classroom with
- his Wharton Moneyball Academy. Today, we'll also be joined by a statistics third-year PhD student from the Statistics and
- Data Science Department, Jonathan Pippin, who I think I'm allowed to call JP for the purposes
- of today. And we're going to be asking him lots of questions. And of course, everyone knows we appear here
- on the Wharton Podcast Network, and it's some combination of myself, Adi, Cade Massey, and Shane
- Jensen each week here on Wharton Moneyball. So Adi, let's first start with, as a
- parent of two alums of Wharton Moneyball, this is obviously an exciting time of year for
- you. How are things going with the academy? How is everything going? How are you enjoying things?
- As usual, we're having a great time. It's actually only our second day. And in the past, we actually have had
- a couple of opportunities for our Wharton Moneyball Academy students to sit in on our podcast,
- but that was in the studio. This is our first time with Riverside Technology and the video technology.
- So we're here in the classroom. I think we're off to a terrific start. It's our second day.
- Our visitor this morning was Neil Payne, who is our longtime, almost probably the most frequent
- guest on our show. And maybe he'll make a pop in in the second half of our show today.
- Well, just to do a little promotion, not that you need it because these students have
- already are in the program, but for all of you, you're in for a real treat, not just as Adi's colleague and co-author,
- but as I mentioned, my middle and youngest sons both went through Wharton Moneyball Academy.
- My middle son is now has a career in data science, thanks to Adi, not just the Moneyball Academy, but the time he spent
- at the Wharton School. He's coming back to Wharton for his MBA, but he was an undergrad with Adi in
- his research lab, competed in the NFL, was a winner actually of the NFL Big Data
- Bowl. So all of you are off to a great start to your careers of being there at Wharton Moneyball.
- So Adi, why don't we jump in today? We're very fortunate to have a PhD student, as I mentioned, from the Department of Statistics
- and Data Science, both here in my home department, Jonathan Pippin, JP. I'm just going to read what his research
- interests say, and then Jonathan, I'm actually going to read them one at a time because
- I'd like you to say to our listeners what each of these topics mean. So it says his research interests include, let's
- start with the first one, which may not, it sounds like it should be obvious, but
- may not be obvious to everybody, applied probability. What does it mean for someone to do
- work in applied probability? Well, first of all, hi, Eric, and thanks for having me.
- Well, yeah. Applied probability basically involves taking the concepts that we learn across math classes, probability classes in
- statistics, and then taking those and applying those to particular problems in a bunch of different
- fields. So today we're going to talk about soccer. We'll talk about how it's used in expected
- goals. Additionally, expected goals, like a bunch of other things, it's probability models, and learning, I would
- say, learning to, applying what you learn in a probability class to different contexts.
- So that could be a particular sport. It could also be causal inference, as you'll
- mention some of my other interests, and plenty of other fields that involve applying what you
- learn, distributions, moments, variants, all the things you talk about on this show to a bunch
- of different problems in real world problems. What would you say, Adi and I have
- had this discussion on the air many times over the last 12 years, what would you
- say to someone that said with GPUs, massive simulation possibilities today, what do we really need
- math and probability for anymore? Can we just simulate out the entire world? We'll get all the intuition we need.
- We don't have to rely on asymptotics. We can just do whatever finite sample inferences
- we want. What would you say to people like that? By the way, I'm not in that camp,
- but why would you waste your time with all this math and probability theory? Why not just learn how to code and
- forget even coding? Why not just use AI, create, use Claude code, code everything up in a workflow and
- then blast it through? That's a great point, Eric. Modern computing has totally changed how people can
- approach these problems. Simulation, leveraging coding agents or what you can code up in Python.
- It's tremendously useful. A lot of times it can give you a head start on finding an answer to
- what you may be interested in in a probability problem. Professor Weiner and I actually, we did one
- of- Just this. We did one of these for one of our research projects, which involved simulating something out
- before we ultimately went to math. In fact, the way we approach that problem, and we've actually talked about it on our
- Moneyball show before, was the question is, what's the maximum win probability obtained by the losing
- team over the course of a game? Which is something that most people don't quite grasp because the losing team ends with a
- win probability of zero, but they start in many situations with a win probability of close
- to 50%. The win probability would be calculated at any point during the game. If you look at the trajectory of the
- win probability, let's say it starts at 50%, and then it moves up and down depending
- on what happens during the game. Eventually, the losing team ends at zero. You ask, what's the question?
- What's the maximum win probability obtained? Well, we eventually solved that problem as a
- theoretical problem, but not until we first simulated it. In fact, it was the simulation that gave
- us the hints on how to solve it. That's fascinating to me because my training and
- yours, Adi, I think, in the old days when we were doing math stat, would have been, you do math first, that gives you
- a guide to what to vary in the simulation design, as opposed to, let's do simulation,
- see the patterns, possibly come up with a set of results, and then see if the math, in some sense, can explain those patterns.
- Well, in fact, one of the things we did by simulation, and also empirical evaluation.
- Just to be clear for our listeners, a simulation begins with a model, a model for
- where, say, football, or baseball, or basketball is actually played. It's like coin tosses, certain probability of a
- score, and then you just simulate that and play fictitious games. It's almost like what we used to call
- strat-o-matic. I don't think our kids back here have ever heard of strat-o-matic, but that's
- from our time. We did that, but what we also did was- What sport did you do it? We did it, I think it was football,
- and we did it also with basketball. Football, we simulated the outcome zero, two, three,
- seven, eight, and for basketball, we simulated twos and threes. That gave us one way of doing it.
- The other way we did it was empirically. Empirically means we just observed what was the
- distribution of the maximum win probability in the sports that we've seen. That already told us, and by the way,
- the fact that the two numbers were pretty close to start with was actually quite illuminating.
- That's shocking to me. Here's what I would have guessed, and then this is why you guys can tell me
- I'm entirely right. By the way, don't worry, we're going to get to XG and all that.
- We got plenty of time here. We're going to get to soccer, don't worry about it. I would have guessed that here's what would
- have happened. You would have posited some model of, let's call it how scores generate, a generative model.
- You run a simulation, you match it to the empirical data, you notice it's deficient in
- a number of ways, then you go back and you fix the model until it matches in some sense the marginal distribution that you
- observed in the empirical data. That's what I would have guessed, and that the initial model would have been so far
- off that you would have had to make, I'll call it more than just minor tweaks, it would have been like major surgical triage.
- Well, I'll allow that JB to tell you, but the model for football that we used was actually pretty good right off the bat.
- What happened with basketball? That's actually interesting. Yeah, it is. The model for football actually very closely matched
- what we saw empirically. Our simple simulation model, once you tune a couple of parameters, it matched fairly closely.
- What we saw in basketball were some major deviations from what we would expect in a
- simulation based on team strength and what we would assume a team's shooting profile would be.
- In basketball, what we found is that a lot of the win probability estimates that are
- out there are systematically too extreme. We were trying to quantify the maximum win
- probability attained by losing teams. What we found is that the win probability models that are out there largely overstate or
- get too confident too quickly in paths or teams that eventually lose. So it was a pretty significant overshoot of
- what we would expect. Just to be clear, when you say overshoot or too confident, do you mean that their
- probability of winning the losing team never gets high enough or it's the opposite?
- I'm just trying to know which direction you're claiming. It's the opposite. What we found was that when a team
- is ahead, the win probability models in basketball tend to give, more often than not, an
- overconfident estimate of how likely that team is to win. You'll see more collapses.
- That's really what we were looking at. Collapses of teams that go on to lose had an abnormally high win probability more often
- than we would expect under a well-calibrated model. I see. Actually, one of the things that's very fascinating,
- when two equal teams play each other, the average win probability obtained by the losing team
- is greater than 70%. I would have guessed the number was somewhere between 70 and 75, by the way.
- That would have been where I would have guessed. It's interesting that I was at least somewhat
- calibrated at all. And in half the games, the losing team is the two-to-one favorite at some
- point. Right. That would have been my guess. I was shocked by that because you think,
- God, if you've lost... So we call it actually the paradox of the blown lead, which means that we observe
- that losing teams very frequently blow a big lead. Right now, the students, we had to interrupt
- them. Egypt was winning two to nothing over Argentina. I don't know. By the time this is aired, you will
- know the advantage of this. But this is a team that now has a very big lead probabilistically.
- I don't know what it is, but I would imagine two to nothing, middle of the game in soccer is probably at least 80%.
- And we'll see what happens and see whether they blow it. But that's the way to say it, by
- the way. I think we'd all agree a two-to -nothing lead in soccer is calibrating leads.
- This has been done for years in statistics, calibrating. One of my advisors, Hal Stern, who worked
- with Tom Cover, who was one of your advisors, Adi, he wrote a paper about equating
- leads in different sports. So this has been around in the statistics field for a long time.
- Like two-nothing in soccer with 30 minutes to go is equivalent to this lead in basketball or football, et cetera.
- It's actually one of those interesting calculations to do. Let me just go to the next one,
- JP, just quickly. And we're going to get to soccer in two more minutes. So statistical machine learning.
- So prior to generative AI and I'll call it AI agents and supportive tools for coding
- and data science, we, the thing I started 20 years ago really was, I didn't call
- it this, but it really was a center for big data and machine learning. So not just for myself and our audience,
- could you talk to the students that are behind you also, why you still think having
- training in machine learning skills are important and which skill do you think is going to
- be more important for you going forward? I was just asking you about data engineering
- versus the data science part of machine learning. Is it piping the right data so that
- you can train an LLM or analyze it properly? Or do you think it's still the learning
- models like XG boost and boosted trees and different types of predictive models?
- Which part do you think will be more valuable for you in your academic career? And also as you try to work on
- applied problems? That's a great question. Yeah, no, they're definitely, the presence of agentic
- coding tools has definitely made it easier to implement these things, right? Anyone can open up a chat GPT window
- or their favorite LLM and code up a machine learning model like that. But all the same steps that were still
- important then are still important now for researchers. So, deciding what data you're going to use,
- deciding the problem you want to solve, and then understanding what these algorithms are actually doing
- under the hood is actually extremely important still to actually understand the inductive biases and all
- the things that go into what's going to make your prediction come out. So, a black box model, you can now
- ask a black box model to write you a black box model to give you some output, right?
- But the same skill, if you actually want to understand where these predictions are coming from
- or the architecture under which these models are giving you a prediction or an output, that
- still requires deep study. And a lot of what I'm interested in is quantifying the uncertainty in these models and
- actually trying to interpret them in a way that allows you to do something like inference
- or something like I actually understand sort of what the parameters underneath the model are saying.
- So, that's more my research interest, but I definitely think there is still a lot of
- need. You still definitely need to know what your model is doing if you actually want to
- - We actually, this actually came up over lunch. Maybe in a couple of weeks, we'll talk
- about this, but everyone has been fascinated by the Jalen Brown trade. And one of the reasons why is that
- the black box models, I mean, the most sophisticated black box models produced by analysts are
- saying that Jalen Brown is very overrated. But they can't tell you where that, why
- that conclusion comes from. One of the things that Neil thinks is that that's inaccurate because it has to do
- with usage. And we don't have data for players who have such enormous amounts of usage, which means
- that you're extrapolating with these models. And secondly, and this is what everyone admits,
- the models can't tell you why. They can just give you a number. They don't say what component of his play,
- what's causing this, or even forget about what fraction of the things you observe are causing
- the prediction to be what it is. They don't work that way. And that's where deep knowledge of statistics can't
- be replaced. And maybe also we would argue deep knowledge of basketball, because eventually you're going to have
- to say what aspect of his game, is it a defensive liability? Is it rebounding? Is it guarding perimeter players?
- I mean, you can, in other words, there are so many possible patterns that could lead
- to his being overrated or not. And one would need some direction to learn. Let me just ask the last one.
- I'm going to skip causal inference only because my advisor was Don Rubin. I understand causal inference.
- I'm going to skip that one. What the hell is selective inference? I know it's a well-known term.
- I just, I'm not as familiar with it. What is selective inference? Selective inference is a field that sort of
- concerns itself with making valid inferences after you've already used the data to select, you know,
- what you're testing or how you're testing it. And so, you know, you see this a lot of the time in sports, right?
- You may be interested in, you know, evaluating who's the best player in the NBA or
- by some metric. Well, you're using the data already to tell you who's good and bad.
- At that point, how do you actually estimate who really is the best, right? And selective inference on top, you know, after
- you use the data or some data-driven mechanism to create an inference problem, how do
- you still use, you know, you still want to use all of your data. You don't necessarily want, you could split your
- data. There's a lot of different, you know, methods, but ultimately it's about, you know, being honest
- in the inference procedure about what inferences we really can say from data once you've already
- used the data to sort of get you halfway there. Right. So, so the easiest example, most, many of
- our listeners, I'm sure even quite a number of our students, even though they've started the
- class, I've heard of what's called the confidence interval. And we all know what that is.
- And that gives you a range of possible values. But if you've built the model using the
- data and now you're using the data again to build the standard errors, you've just done
- a selective process first. And now you need to figure out what the implications of that selections are on the
- uncertainty. And it is dramatic. It can be, depending on how big a model process, how much you've used the data,
- it can be huge. So if you consider a model with say 10 variables and you decide on two and
- build confidence intervals for those two, those confidence intervals are way too narrow because of the
- selective inference. Would you guys describe the winner's curse as a specific example of selective inference, which is
- the classic, you know, you have a bunch of competing athletes or models, you pick the
- winning model, then you do inference giving the winning model. And of course, the results are never as
- predictive from that winning model as you would expect. Confidence intervals are always too narrow because it's
- a form of selection first and utilization second. Absolutely. Yeah. The winner's curse is like the classic, classic,
- you know, selective inference example. Well, we're at the World Cup time. So now let's move to the main topic
- of today. And of course, we're going to leave time for students to ask questions of both Adi,
- myself, and JP. So first it says in the notes here that you've been doing some work in XG
- and computing something called XG+. But why don't we just start with, you know, the basics.
- What is XG? And why do you think it's considered an advanced stat and a very popular statistics for
- soccer? Yeah, expected goals are probably the most central, you know, analytic statistic in modern soccer.
- It, you know, it involves basically, you know, first of all, expected goals, it's not this
- absolute value, you know, this true value that you can observe. It's not a counting stat.
- Yeah, it's not a statistic that you will just see. And it's like, ah, that's the expected goals.
- It's a model-based statistic. So there's, you know, this involves a probability model, like we mentioned earlier.
- And it, you know, involves estimating the probability that a given shot, once it's taken, is
- scored. And these models vary in complexity. They can include anything from just the location
- of the ball on the pitch, to how many other players are around the ball, to how open is the goal, where's the goalkeeper,
- you know, maybe even what time is, how much time is left in the game, what, you know, what player is taking the shot.
- They can get very detailed, but it ultimately is an estimate of how likely a given
- shot is to be scored once it's taken. And I think that's, there's a couple reasons
- why it's, you know, so commonly used. But one that's so nice is it basically tells you the quality of a shot.
- And so, you know, a shot that is taken that's a 0.5 XG shot would mean, you know, we estimate that this shot
- is about 50% to go in once it's taken. Who does it, I could imagine it being used to grade, if you'd like.
- We're in a classroom here, I see a bunch of students. I could use to see it grading maybe
- three different components. One is, how good is the player that shot the ball, right?
- Because if they perform better than what the stats should say, given all the, that's one,
- you could use it to score the goalie. You could use it to score the team at some level.
- Where do you most see XG being used? I would imagine it's at the team level, but it could also be at the player
- level. I mean, how do you see it being used? Yeah. So the main place that you and many
- others will see it used is at the end of a match. You'll see, you know, this team accumulated two
- and a half XG in this game. And this other team accumulated one and a half XG. It's sort of used to estimate, you know,
- the cumulative quantity, quality of the chances a team took. A lot of times it's used to sort
- of, in post-match analysis, like who deserved to win. Since goals are very rare in soccer, like
- a lot of games end one nothing or even nil-nil, right? How do you grade a 0-0 tie?
- Well, that's graded based on the quality of the chances created. And so you'll see it a lot with
- teams using to compare or trying to say which team was better, which team created more
- dangerous chances. But like you said, it's also used for individuals. So, you know, like you could residualize.
- You could say, did a player over-perform their XG? Or did a goalkeeper over-perform their XG
- in terms of saving more shots than you would expect to go in? It's definitely used for player analysis, but I
- would say the main place it's used or the most common example is at the end of a match or even at the end
- of a season, looking at the chances that a team racked up over that game to sort of determine maybe who deserved to win
- or who was more threatening. It's just a way of averaging out some of the randomness, which is differentiated from the
- skill component. Although it can be very fascinating on an incident base. So yesterday's America-U.S. match against Belgium,
- the American goalie had one, at least to my eyes, looked like a spectacular save.
- I didn't know because I don't, you know, I'm the first to admit I don't know that much about soccer.
- Looked good, but the eye test isn't the same as an analytical model. So I want to know what was the
- XG of that shot? If the XG of that shot was 0 .1, meaning it wasn't likely to go in,
- I'd be, I should be less impressed by the save. If it was 0.9 and it didn't go in, then I should feel that maybe
- that was a great save. So that's how it can be used. And maybe someone will tell me what the
- XG on that shot was in this vast majority of students back here. Does anybody know the answer?
- Do you know? You mean like before the shot happened or after the shot? That's a great question.
- After the shot. Well, while he's taking it, XG is calculating. Before it's in the air.
- Yes. I don't know about that one. Well, would you know any other number? Yeah, I think it was like a 0
- .25 after it was in the air. After it was in the air, it was 0.25, which means it's pretty impressive, but
- not particularly. So before we get into what's wrong with XG, because why would you need XG plus
- if XG is brilliant, would you agree with the following? That if I had an infinite amount of
- data, I would not need a model. Like I could literally just say, well, I could look at every place on the pitch.
- I could look at every situation of players and their locations. I'd have just some big empirical lookup table.
- I just pinpointed, and then I've got some empirical estimator. I wouldn't need any smoothing because there's no
- discontinuity. I have infinite data. If I've got infinite data, I've got infinite
- data, right? Yeah. Yeah, of course. If you have infinite data, then the problem solves itself.
- You just look at, like you said, a binned average and you're good. We don't have that.
- We have finite data and we have strategically chosen data. Players aren't shooting from random places on the
- pitch in random configurations all the time. And so that's sort of where the model
- comes in to sort of help us alleviate some of those biases and maybe smooth over
- places where we don't actually have that much data. Okay. Now let's get to XG+. So what is XG plus?
- XG plus came sort of as a response to some of the limitations of XG. XG, expected goals, is a great stat.
- It tells you exactly what it tells you, which is how likely is a shot once it's taken to go in.
- But many opportunities never have a shot realized. You see this in plenty of games.
- Mexico played England just a couple of days ago in the World Cup. And Mexico crossed the ball dozens of times
- into the box. And there were many times where a Mexican player did not get their head to the
- ball. But if they had, it would have been a great chance, right? Expected goals, XG, only tabulates, it only accumulates
- once a shot has been observed. So if there's an attack where there's no shot, but there was a very dangerous opportunity,
- that looks like a zero in expected goals. And we've sort of wanted to set out to sort of fix that.
- And also maybe mend another problem with XG, which is on a single attack, if you
- have a bunch of rebounded shots, so imagine you shoot the goalkeeper saves, you shoot the
- goalkeeper saves over and over and over in a short period of time, it's actually possible
- to accumulate over one expected goals, which is not possible on a single attack.
- And that's just, again, one of the maybe limitations of XG is accumulating XG. So we wanted to set out to sort
- of solve some of these problems. Yeah. There's XG plus then, if it's not based on just whether a shot is taken or
- not, is it based on, when you say the situation, is it the ball? Is it the ball and all the players?
- Like what is the situation that you then score and make up XG plus? Yeah, great question.
- The situation includes the ball and the other players and all the normal things that would
- go into an expected goals model. The key distinction with expected goals plus or
- XG plus is what we're calling it, is that we're not just modeling the probability that
- there's a goal scored on a shot. We're also modeling the probability that a shot
- is taken in a given situation. And so expected goals gives you something like a conditional probability, right?
- You're estimating the probability of a goal given a shot. But the danger, where you're really interested in,
- is the probability that the goal is scored on an attack. And that marginal probability involves not just the
- conditional probability of scoring once you shoot, but also the probability of shooting in the first
- place. And so the key development of our method is to estimate that probability that a shot
- occurs from each sort of configuration, including the players, including the openness of the goal, including
- the situation in general, and then factoring that into the expected goals to give a more
- complete picture of the attack. So I have two follow-up questions to that, and maybe the students will have others
- too. So one is, I think I know when expected goals is computed. It's computed at the time of a shot.
- When is XG plus computed? That's number one. And then two, what kind of data requirements
- must be there to be able to compute XG plus, possibly even greater data requirements that
- are there then for XG? So when is it computed? And how has modern data tracking or video
- tracking or other forms of tracking helped and now made it even feasible? Another great question.
- So in order to estimate XG plus, the key development or the key addition is being
- able to estimate the probability of a shot at a given position for an attack. And so that involves having estimates of where
- the players are, where the ball is, continuous in time. And so we have some tremendous data providers
- that are working with us to provide us with video-based tracking data, so derived from
- broadcast footage. But essentially, you can imagine a 2D grid of little dots moving around on the pitch.
- And so using those dots and really different features we can create from them. So we might imagine how far the closest
- defender is from the ball, sort of matters not just for the likelihood of a goal
- being shot and scored, but also the likelihood of a shot being taken. That closest defender goes into our model as
- a feature, and as do some of the other defenders, as do the goalkeeper, and it does so continuous in time.
- So continuous over time, we're estimating something like the probability that a shot occurs in some
- fixed window. And so we have our data is estimates 30 times a second where each player is.
- You might have coarser feeds or something like that, but ultimately we're just estimating in the
- next second, in the next half second, what's the probability that a shot... Okay, so you're choosing a window, which is
- helpful. Let me ask another question, which every student in that room and all of us have
- to do this when we're working on applied problems. How did you think about the feature engineering
- part of this problem? Because in some sense, anything could be a feature. And so did you use knowledge of soccer?
- Did you use just brute force empiricism? Did you end up doing some sort of penalized criterion, some feature selection, whether it's some
- sort of penalized lasso? Did you end up doing some sort of out of sample validation?
- How did you decide what features to kind of jam into this XG plus model? Yeah, so I would say some combination of
- those things. So first of all, a knowledge of soccer and how expected goals are generally calculated, which
- is it uses the five closest defenders, the location of the goalkeeper, the location of the
- person on the ball, the three closest attacking players. These are just common conventions.
- You could change some of them. We have tried changing some of them for sensitivity, but ultimately that's just convention.
- The critical thing and something I want to make sure that the students hear is how
- we actually encode these features. It's very easy to just take tracking data, take the XY features and just throw them
- into a model and just trust the model is going to figure out the patterns in the data.
- That's where, like we mentioned earlier, knowing the architecture of these models, actually understanding how a
- tree-based model, which is often used for expected goals, actually works. Those features, the more informative we can make
- them, the better. So what we do is we encode not just the location of the players, the location
- of the ball, we encode relative distances. So we're looking at how close each player
- is in terms of distance, not just in terms of XY, and also bearing to the goal. The way we encode location and all these
- features is very thoughtfully done so that when our high complexity model that may be coded
- with an AI or LLM tool runs, it's ultimately picking up on informative features in the
- data. We understand that just throwing a bunch of unnecessary features into a model class like this
- actually degrades performance. And so that's why we try to be very careful to select what we want to
- include, while at the same time, leaving open the possibility to cross-validate or do some
- out-of-sample testing to determine if additional features are relevant or should be added or
- dropped. Well, I mean, it sounds like one of the things that you learn from knowledge of
- soccer is various orientations and coordinate systems matter, right? So if a player's back is to you,
- that's very different when the player is front -facing. And you have that in the data?
- You have the actual, because this is one of the things, Adi, I think we talked about a few weeks ago on the air,
- is that having someone's XY location is different than knowing which way they're facing.
- So you actually have that data? Yeah. Orientation data is included in the data set
- that we have. In many 2D tracking data sets, it's not included. But there are also certain times where it
- can be inferred based on their velocity, where they're running. It's very difficult to run 20 miles an
- hour backwards. And so if you have a limited data set and you wanted to construct something like
- we did, there are certain ways to try to infer at some of the features that we have.
- One of the advantages that we had was we were able to get this data. The tracking data is very expensive.
- And we have a partnership with a video -based tracking, which is not tracking with an
- RFID trip or something, or using cameras that are in the stadium. This is just using the video feed to
- create the tracking out of that. And that worked great. By the way, we'll talk about this.
- The students may be interested in this. It just came up on my phone. You'll never believe what just happened.
- Argentina scores three goals in the last 13 minutes to stun Egypt 3-2. So you want to talk about it.
- I would have said there's no chance that Argentina is coming back to win that game.
- See, their win probability had to be 0 % according to a model, according to the JT, the Yachty, the Weiner.
- Actually, by the way, this is a fantastic example. The losing team ended up probably had a
- max win probability that was quite high. I would guess it's above 80. 90-plus percent, easily.
- Yeah, interesting. Although Egypt is how good a team? I'm not sure. You're up 2-0 with 13 minutes left.
- But anyway, I want to get back to the XG plus for a second. But sorry, students, you missed it.
- By the way, if it makes your students feel any better, I didn't get to watch the last 13 minutes of the game either,
- because I've been on the air with you. But they have this thing called review. They have video.
- I'll be able to watch it. But, did you actually apply this to actual players? Are there players that appear really good in
- XG, but not as good in XG plus? And is there any names or something that might surprise us?
- Absolutely. It's been well documented that there are very few players that actually overperform consistently their expected
- goals. Just the shots they've taken, expected goals, residualizing that. Is that because of low power, or in
- your belief they're just few players? There's a lot of goals, so it's not power.
- Basically, just to be clear, we've actually talked about this before on the air, what we
- call excess goals. JP has called residualizing, but I want to make sure that everyone understands.
- To residualize means to take the numbers of goals, subtract the expected goals. And that difference reflects, it could be positive
- or negative, how well you've done relative to the chances. So what you're saying is that nobody really
- seems to outperform or even underperform their XG. The ability of a soccer player tends to
- be wrapped up in their XG, just getting to the XG. Yeah. And that getting to the XG is what
- our expected shots tries to get at. And so we would say the correlation from year to year in your overperformance to XG
- is rather low. It's like 0.1. But in expected shots or XG plus, that correlation year to year
- with players is like a 0.6 or a 0.7. So it's stickier. I love it. That to me, just for myself, that's the
- highlight of what you've said so far in terms of the empirical findings. Because we've talked about the fact that certain,
- I'll call it offensive or performance metrics, just do not correlate well across games or seasons,
- et cetera. The fact that XG plus is more stable, I think is a strong case for its
- validity as a useful metric. Because I do believe the creation of opportunities is probably a very stable construct.
- Absolutely. And you asked about individual players. The player that stood out for us on
- XG plus and expected shots is Erling Haaland, the striker for Norway, striker for Manchester City.
- Well-established reputation of being able to take extremely acrobatic shots and get shots off where
- other players couldn't. Like Adi was saying, if you looked at his expected shots, he would take way more
- shots than his expected shots would indicate. So that residual would be very high, which
- is to say he is creating more shooting opportunities or he is shooting more often than
- we would estimate. And we consider that a very sticky skill. That's a skill that we would expect to
- stay with him because of how consistent it is year to year. We're finishing variability, that's a little bit less
- consistent, making and missing shots. Actually getting to take those shots and as
- a team, creating those shooting opportunities, that is, we find to be more predictive than just
- pure shot-making ability. So let me ask one last question. Then I want to, audience members, get ready
- to start thinking about questions to ask JP or Adi about this. A natural, possibly erroneous conclusion from this would
- be a player with high XG plus has to take more shots. They have to, because if they're good at
- creating situations or maybe it's they have to get more playing time or et cetera, is
- that like, I'm actually trying to combine your knowledge of selective inference with their knowledge of
- causal inference to say, is the natural implication of this that a player that does really
- well, like the guy you mentioned on Norway, Halon, my understanding is he doesn't take a
- lot of shots, but when he takes them, he scores very often. So what's the implication of this for let's
- someone that was a team manager or someone that was trying to give advice to players
- that do well on this metric? Yeah, great question. So first of all, I just obligatory to
- say this, we're working with observational data. So we're working with, you know, our models
- are trained based on data that we actually observe in games, which has strengths, but also
- its weaknesses in that this is not randomized. We don't necessarily, you know, know if something
- is causal per se, but we can find strong associations. And the associations that we find in this
- data is that if you were a team manager, for example, and you had a player who over-performed their expected shots, like a
- Holland or like another, you know, some of our other elite strikers, like Mbappe, who consistently
- over-performed their expected shots, it would be reasonable to assume that their performance in a
- following season or in a following game will be more consistent on that metric than it
- would be on expected goals, just based on the correlations. As for causality or whether players, you know,
- who should take more shots or not take more shots or something that involves an intervention,
- that involves a bit of a different mechanism than what we were trying to explore.
- So I would caution against using it in an interventionist sort of way. But I would definitely say that, you know,
- the correlation, the correlation structure in the data does indicate a much stronger correlation between expected
- shot over-performance and XG plus over-performance relative to XG over-performance, which we find
- to be very noisy. So JP, before we turn it over to the students that are part of the summer
- Wharton Moneyball Academy summer program, can you speak maybe just for a second or two?
- I know you have collaborators on the project. I know also I was shocked that you
- have this data that you've talked about because I've been trying to get this data for
- years. So maybe you could just say a few things about that and then we'll turn it
- over to students for questions. Yeah, absolutely. I had some great collaborators on this project.
- We had a Wharton sports research senior fellow, Paul Sabin, advised this project for me and
- a fellow student, Tianxu Feng, who actually just graduated with a master's in data science from
- the University of Pennsylvania. This project actually emerged and was done primarily
- in our undergraduate sports research lab, which is every summer. I'm happy to oversee those projects and work
- with these very talented students. That came out of that sort of process. It was a tremendous opportunity for undergrad students
- here at Penn to get involved in sports research here. And that was where most of the thinking
- for this project sort of came from. It's since grown well beyond that. Also, a major thanks to our data providers.
- Gradient Sports provided us with this data. We have a partnership with them where they
- provided us with this very difficult to get tracking data, something not everyone can get a
- hold of. Our partnership with them has been really great. And so if anyone else is interested in
- getting that data, they're probably someone you might want to reach out to. Sounds great.
- Wadi, I'm going to turn it over to you to bring up the first student and let's start hearing some questions.
- Okay. So JP is going to slide over and we're going to bring in our first student.
- They've only been on campus for, I guess, about two days. So why don't you introduce yourself, say where
- you're from, what your name is, and then ask your question. And of JP, me, you can even, if
- we need to bring Neil in, we can bring him also to answer some of the questions.
- Yeah. Thank you guys for having me. I'm Jack Brenner. I'm from New York. And my question's tailored more towards JP and
- about his research towards XG+. And where you can kind of take that with expected assists and how with XGs and
- XG+, you're dealing in a lot of hypotheticals of how, you know, will he be in the right position?
- Will he convert it to the right part of the net? And with expected assists, you're taking that even
- further in hypotheticals of where the ball will go, where he'll make the run and how,
- and I was wondering about how, if there's any thoughts on how to improve expected assists?
- Well, first of all, also, it's great to meet a fellow New Yorker. I grew up also in New York City,
- went to Stuyvesant High School in New York. So it's always great to meet New Yorkers
- as well. Actually, New York is our leading state representative. 20 of our students are from New York
- state. All right. Not a surprise. Okay, go ahead, JP. So that's a great question, Jack.
- Expected assist models, very interesting. They're trained very similarly to expected goals models,
- which is they're trained on observed shots that are taken. An expected assist is only really recorded on
- a shot that is taken. That's where you get the expectation is how likely is the shot that the person assists,
- right? So you pass the ball to someone and they shoot. What's the likelihood that they score?
- And it tries to credit that to you. But again, that only, that has the same problem as expected goals, which you have to
- see a shot to actually realize an expected assist, where XG plus doesn't have to realize
- a shot. You would still get credit for a dangerous attack, even if there's no shot, if the
- likelihood that a shot would have happened in that situation is rather high. And so expected assist could also be taken
- with a similar strain as we did, which is to say, what, maybe not the expected assist from observed shots, but the expected assist
- from hypothetical shots, sort of like you said, and give people credit for creating those dangerous
- opportunities, for making a pass that increases the XG plus, even if there is no shot
- that actually comes out. Yeah, I think that is so crude. Your question, Jack, is phenomenal.
- A couple things. One is, when I watch soccer, I'm not an expert in soccer. I'm thinking of passes as opportunities for someone
- else to score. And then the second thing is, of course, the quality of the person at the other
- end really matters. So another thing that's interesting, you know, JP talked about conditional probabilities.
- I'm now passing. Well, first of all, did I make the optimal pass? If I have the location of everybody, I
- made it. Here's another thing. I may pass it to where there's opportunities to score, but it wasn't the best pass
- I could have made. Now, in some ways, if that's true, then I should be penalized for that.
- And so then you need some measure of optimality. And then, of course, it's who is on
- the other end of it. Like, I don't know, you guys probably watched the U.S. game last night.
- Whoever that guy was at the top of the screen that they were leaving wide open, there was no way the U.S. was
- even ever going to pass it to that guy because, you know, he wasn't very good. And so, you know, X model might say,
- well, this guy's wide open. You've got to pass it to him. Yeah, but he's not going to score if
- you pass it to him. So I think it's a great question. Well, that's a hard thing to do.
- Once you start working into the XG models, individual player qualities, you start to get to
- what we call over-parameterization and selection biases, and it's very hard. But great question.
- So thank you. And now let's go one more student. At least one more. Hi, I'm Rachel.
- I'm also from New York. I have a very similarly formatted question, which is about how would you do expected tackles
- for, like, a defensive player? Expected tackles. Okay, so that's not something we've worked on,
- but it's a great potential research question. JP, maybe to build on Rachel's question, like,
- could you basically jam in any dependent variable you want on the left-hand side?
- Like, Jack's interested in expected assists and Rachel's interested in expected tackles.
- I'm interested in expected yellow cards because I like violence. You know, why not jam in anything on
- the left? I do, I do. Let's jam in anything on the left-hand side and just build the model.
- Is there any reason why you couldn't use the same machinery and just kind of plow
- through it? On its face, there's no reason why you couldn't use the same machinery.
- But for something like expected tackles, you might have a difficult time deciding what actually should
- go into that model or what even, you know, how do you actually quantify, are you
- training based on observed tackles? Are you training based on tackles that could
- have been made? There's some very interesting questions that come with just throwing something on the left-hand side.
- Your Twitter feed, if you're on analytics Twitter, is full of people putting X in front
- of things and just saying, we have expected blank. I would say it's a little bit more
- difficult than that. It requires, you need a bit of a structure. You need a little bit of a mechanism
- under which you can do that. And more importantly, you need good data, data that's going to actually allow you to answer
- that question. And so if you did have such data, like data, like the granularity that we have
- at Wharton, it's very possible that you could build an expected tackles model and something like
- residualizing over that. But also again, you get into the granularity of the data and how, you know, how
- precise do you need your video estimate of location to be in order to make that estimate really reliable.
- And so you run into some issues with reliability, stability of your estimates, but in theory,
- right on its face, you can build an expected anything model, just how good that model
- is going to be, how stable it's going to be. And do you have enough data for it?
- Do you have the right data for it? Those are really the questions that... Yeah. And would your X tackles model be the
- same as someone else's X tackled model? Because whose definition? And obviously this is something that matters a
- lot. War is something that we talk about in lots of sports. There's war, which is wins above replacement.
- But my war is different from someone else's war, most prominently in baseball. You know, Eric, I can't avoid talking about
- baseball at some point. You got to. I had to get into it. But someone else's... Hey, the Yankees won one in a row,
- but go ahead. Got to get that in. There's baseball reference war, which Neil was involved
- with when he was there. And there's Van Graaff's war, and they're really radically different.
- And it's just your assumption about what goes in matters. And so how you define those things can
- be radically different. But that's a great reference. Thanks, Rachel. Let me just ask a question, actually, since
- Neil is there. Let me just ask him a more philosophical... Come on, Neil. Neil, come on over here.
- I can't wait without having to... Sorry, Neil. I'll have to ask you a question. It's always good to see you, as always.
- Hey, Eric. So JP and Adi have talked about feature engineering that kind of gives kind of, let's
- call them other explainable features or knowledge-based features. But in your experience working in the field,
- if I'm a team, do I care about any of that? Can't I just take the massive raw data,
- put it into some embedding space, some lower dimensional reduction space, jam it into some predictive
- model? And if it beats your model, who cares whether I can explain it or understand it?
- I'm in the prediction business. I'm not in the understanding business. I'm not in the causal inference business.
- I'm not in any of those businesses. I just wanted to get your philosophy on that, because there are a lot of listeners
- out there that are saying, I can just take this massive data, dimension reduce it in
- some way, shove it in on the right -hand side of an equation, get some prediction,
- and my prediction will beat yours. Yeah, that's kind of the Kaggle competition approach
- to do it. But I think one of the things that's missing there, especially if you're working for a
- team, is you have to be able to explain because you need them to buy in. You're dealing with a lot of people that
- may be, they might be analytically skeptical, or they just may be curious about it, but
- they don't know all of it. And if you come in guns blazing and say, my model, which is a total black
- box, I can't tell you why it gives this result, but it does give this result. And it says you need to do this.
- You're not going to get someone who's like a coach or a player to buy into that at all, because they're going to be
- like, I don't trust you. I don't know what went into this. Do you even know why it's saying this?
- If you can't explain it, then why am I going to put my reputation on the line by implementing this thing that you say,
- trust me, bro, it's going to work. So that's the importance and why you can't just say the numbers say this, and that's
- the end of the story. You have to be able to explain it. And there has to be a debate too,
- I think, because there's a lot that the numbers are still missing. And you guys just got into that around
- like finishing quality. Is that a real thing versus not XG and XG plus make different assumptions about that.
- So I think that there's like elements of skill that are still left to be settled
- in these things. All right. We have time for one more student question. We have two more lined up, but we'll
- definitely take one. Maybe we'll do two. So let's do one. So we'll do quick, quickly say your name
- or introduce and we'll do a quick, quick one minute answer. I'm Eli. I'm from Brooklyn, New York.
- And my question was, how do you think pro teams are currently using this data? And how do you think they should be
- using it? Soccer data or? All right. That's a great question. We can come down. In fact, why don't you sit down JP
- and we'll try to answer that. I'm going to start by saying it's very hard to find out.
- Soccer is one of the most tight lipped among the analytics community. There isn't a public facing analytics world like
- you'll see in certainly in baseball, football, even in basketball. Basketball is also a little tight lipped because
- it's tracking database. So we don't really know. We've had members of some of the American
- professional leagues have come to talk to us and they're pretty tight lipped about how they
- use it. I can only say they're using it. And we could also see a little bit on the field.
- Do we have any to add JP? Yeah, absolutely. Analytics are notoriously, soccer has been notoriously slow
- relative to baseball, relative to some of the other American sports. It's growing fairly quickly now.
- You'll see job openings for most of the major clubs in the analytics and machine learning
- engineer, data scientist space. But a lot of these teams are very small. And so a lot of them rely on
- some of this public work that we're doing. And so I can't speak to necessarily what
- X team is doing or what Y team is doing. But in general, I would say that the public space still has a big effect on
- what a lot of clubs do, especially clubs that don't have as big of an analytics team.
- They have to adopt more from the public space. So they're definitely some of our public work.
- Would I be surprised if it pops up, a particular team is using this? No, I wouldn't be surprised in that at
- all. But I think what you're saying is true, that the teams are looking towards what you
- guys are doing. And I hope that's true. It makes me sleep better at night knowing
- that the research enterprise that Adi has built around sports analytics, I hope it's actually being
- used by teams. That would be nice. Maybe one more question. One more question. We have our last question.
- It's a Chicago Cubs t-shirt. Let's make sure that that's seen. We got it. We see the Cubs.
- My name is Henry. I'm from Chapel Hill. My question is for JP about how you see the use in the future of XG
- plus to evaluate players like trade value and transfer value. When rubber meets the road is the expression.
- So how does it actually going to end up in evaluating players and maybe potentially change
- how much you pay for them? Absolutely. The cool thing about XG plus is that decomposition we talked about, right?
- The expected goals portion and the expected shots portion, those can be viewed in conjunction.
- So you could look at a player's XG plus, or you could look at their expected goals or just their expected shots.
- And we understand that each of those maybe more or less correlated with huge performance, right?
- They're over or under performance on those metrics. And so teams are going to be able
- to say, all right, we may have a player who ran it on a very hot finishing streak.
- We may not believe that, right? Everything we know about expected goals data will
- tell us that there are very few players that consistently over perform their expected goals.
- But this other player that I'm interested in has a very high expected shots or a
- very high over performance of their expected shots. We might believe that a little more.
- We might believe that might carry over. That might be more skill-based, right? There might be more signal in that number
- than in their expected goals number or their over performance over expected goals.
- And so I think it may have a similar effect to certain things in the NFL or in the MLB.
- You may believe a certain skillset is more likely to continue in the future. There are plenty of examples of that.
- And that's where I think it may have an effect is breaking down, first of all, creating a decomposition, breaking down the shooting and
- the scoring into two distinct components that are roughly orthogonal, but both very informative.
- And then first of all, that decomposition and secondarily being able to say, how much do
- we this? How much do we actually believe a player is worth? How much do we believe they're actually going
- to produce in the future? It may change those estimates and improve our ability to estimate what a player may actually
- be worth on the transfer market. And so to that end, that's where I think teams might implement what we're doing for
- transfers, for selecting players and scouting players in general. So the last thing I would add is
- that to summarize, that last point would prevent you from overpaying for performance that happened in
- the past. So you don't want to pay for performance that happened because that's not what you're buying.
- You're buying performance in the future and allow you to acquire players who you expect to
- overperform their market value. That sounds very familiar. Buying low, selling high, market inefficiencies, using statistics.
- That sounds to me like moneyball. I agree with that. And then the last thing I would say
- is that it also leads to the question of the optimal combination of players. Because having players that create opportunities on a
- team with good finishers, it could be even more advantageous, which is why I like tree
- -based models for lots of reasons to do these types of evaluations. Well, Adi, this has been another show.
- It's been a very unique show in all of our 12 years. I don't know, is this maybe the second
- time I think only in 12 years we've done something live with a classroom? Because remember, we've done live in a studio
- audience before. Yes, we have. But I don't know if a classroom. So lots of people to thank today.
- Let me start with the most important person. I want to thank JP. This, to me, is what research is about.
- And so I want to thank you for joining us on Wharton Moneyball. Of course, I want to thank, even though
- he's only on the air briefly, our longtime guest, Neil Payne. Adds one more time to his long history
- with us here on Wharton Moneyball. And of course, on behalf of myself, Eric Bradlow, my co-host for today, Adi Weiner,
- some combination of us, Kate Massey, Shane Jensen, are here on the Wharton Podcast Network.
- I'd like to thank our producer for today, Jacob Grodnick in Broad Media. And of course, the big boss woman, Dee
- Patel. Between now and next week, enjoy your sports, enjoy your statistics. We'll see you next week here on the
- Wharton Podcast Network.
Brief Summary
Wharton Moneyball discusses applied probability, expected goals, and XG plus in soccer analytics with Jonathan Pippin and co-hosts Eric Bradlow and Adi Weiner.
Key points
- Wharton Moneyball Academy Launch. The Wharton Moneyball Academy kicks off its second day with excitement and new technology.
- The Importance of Machine Learning Skills. JP discusses the ongoing relevance of machine learning skills in the age of AI.
- Understanding Selective Inference. JP explains selective inference and its implications in data analysis.
- Understanding Expected Goals (XG). Expected goals models estimate the likelihood of a shot being scored based on various factors.
- Introducing XG Plus. XG Plus addresses the limitations of XG by also considering the probability of a shot being taken.
- The Limitations of XG. XG only counts shots taken, missing opportunities that could have been dangerous.
Episode highlights
- Wharton Moneyball Academy LaunchThe Wharton Moneyball Academy kicks off its second day with excitement and new technology.“It's our second day.”1:06Jul 8, 2026
- The Importance of Machine Learning SkillsJP discusses the ongoing relevance of machine learning skills in the age of AI.“That's a great question.”13:00Jul 8, 2026
- Understanding Selective InferenceJP explains selective inference and its implications in data analysis.“Selective inference is a field that sort of concerns itself with making valid inferences.”15:58Jul 8, 2026
- Understanding Expected Goals (XG)Expected goals models estimate the likelihood of a shot being scored based on various factors.“A shot that is a 0.5 XG shot would mean it's about 50% to go in.”19:54Jul 8, 2026
- Introducing XG PlusXG Plus addresses the limitations of XG by also considering the probability of a shot being taken.“XG plus is not just modeling the probability that there's a goal scored on a shot.”24:00Jul 8, 2026
- The Limitations of XGXG only counts shots taken, missing opportunities that could have been dangerous.“Expected goals only tabulates once a shot has been observed.”24:41Jul 8, 2026
- Player Analysis with XG PlusXG Plus provides a more stable metric for evaluating player performance over time.“The correlation year to year in overperformance to XG is rather low.”34:45Jul 8, 2026
- The Importance of Correlation in Sports AnalyticsUnderstanding the correlation between expected shots and actual performance can guide team strategies.“The correlation structure in the data does indicate a much stronger correlation...”38:43Jul 8, 2026





