Management by the Numbers
Ask about Knowledge at Wharton
Episode Description
People analytics isn't perfect -- yet. But it helps eliminate biases in important areas such as recruitment, says Wharton's Cade Massey.
Read the full story: http://knlg.net/1ev2WN9 📂 Management 🏷️ Research
Read the full transcriptHide the full transcript
- managing people was once regarded as an art rather than as a science but these days companies have begun to use
- sophisticated data analysis for all sorts of people related issues ranging from Recruitment and performance
- evaluation to promotion and compensation people analytics as this approach is called is Making Waves in
- the field of Human Resources because it is said to eliminate biases that exist in human judgment Warden plans to host a
- conference on this topic on March 28th and today knowledge at Wharton is pleased to welcome kid Massie a
- professor in the operations and information management department who has been leading this effort Kade
- welcome and thank you so much for joining us today thank you MCO very glad to be here so uh people analytics has
- been described as a datadriven approach to managing people at work uh I wonder if you could help me understand what
- that means and how it differs from traditional ways of managing people MH well traditionally people have made
- decisions around who to hire who to promote who to compensate the best um intuitively or with some numbers but
- maybe not in a systematic way and people analytics is an attempt to bring a little more systematic processes to
- these decisions there's some of the most important decisions that an organization
- makes and um we've seen good use of data in other fields finance and marketing it's just kind of the slow evolution of
- using those data in in new Fields so how EXA exactly does that work does it really eliminate biases well it helps
- and this is one of the this is one of the motivations that I have I study decision- making and it's very much
- about the biases people have um in their intuitions and these are hard to root out and bringing some data to the
- process and more systematic decision- making more atic analysis is one way of doing it eliminates a big word um I'm
- sure that there are some situations where we can take in a bias all the way down but mostly we're just trying to
- improve right so so maybe if we were to look at this in specific terms it would help make it a little clearer to me uh
- let's start with Recruitment and the first step uh which is the first step in a company's engagement with an employee
- now say at the University of Pennsylvania traditionally if we wanted to fill a job we would uh post it on our
- website we would look all the look at all the resumés that came in we would look through the resumés to see who are
- the most qualified people call them in for an interview and then try to find the person who was the best fit now now
- uh using people analytics uh how does this process change exactly so if if we were to come in and try to bring some of
- these tools to that process probably the first thing we would do would be to look
- at as much historical data as possible to understand what attributes in these candidates predict performance long term
- so can we can we model just basically rather than interviewing them necessarily we can look at their
- characteristics off of a Vita or um their application and ask what's the relationship between these observables
- and long-term performance it doesn't mean necessarily that we'll give that model 100%
- discretion on who gets hired but we use that model's output to inform our decisions so we'll probably still do the
- interview we'll probably still have a group discussion but to have some analytical rigor ining that discussion
- would be better than most processes which don't have it so um the other thing we might do is to model the the
- admissions decisions themselves so rather than modeling long-term performance of the candidate we say well
- what are we doing right now who are we deciding to put in and you can go in and find out well
- whether you know it or not implicitly you've been putting 20% weight on GPA and 50% weight on The Prestige of the
- company they work for and 30% on whatever you can just ask what have we been doing even though we we don't
- follow some rules we're implicitly going to end up relying on some rules that's often enlightening organization might
- not know it's like we didn't know that our process put that much weight maybe that's okay maybe they decide they'd
- rather shift it and put weight on something else so you you mentioned that ter uh uh being able to predict uh the
- performance how how accurate are these predictions typically well they're going to vary dramatically and um they're
- never going to be perfect these are noisy processes and one of the things that actually I think this is a virtue
- of people analytics if you're if you're if you're if you're if you're really being thoughtful about it one of the
- things you carry away is a deep appreciation for how much chance there is how much noise there is how imperfect
- the process is even the best model with the best data um is still imperfect and you know that's actually an important
- lesson we don't typically carry that lesson around about the decisions we make in the interview and hiring process
- usually we think yeah I can predict that and you remember the ones that you predicted well and you conveniently
- forget the ones that you didn't predict so well and you never develop that humility about the process start
- modeling the process and you start appreciating the difficulty so imperfect prediction for sure
- but um the idea is that we can improve we can we can we can do better we can be more accurate by adding some analysis to
- the intuition of the people who are doing the decisions making the decisions right so I've heard that companies like
- Google and Xerox for example have been using uh this approach to to recruitment uh how good have been the results that
- they've seen well you know people it's not all about modeling performance it can be any kind of
- analytics applied to these processes so one thing for example that Google did a few years ago was to ask how good are
- interviews for predicting um you know performance on the job and they discovered that you know the literature
- says they're not very good but a lot of times people want to you know prove it to themselves in their own organization
- and Google is strongly that way so Google says we'll study ourselves and they run the numbers and they discover
- well in fact interview don't do a whole lot and um they serve other purposes and
- so maybe you want to not throw them away but they were spending you know hours of
- their manager's time interviewing candidates you know eight interviews nine interviews 10 interviews and they
- discover they don't do much well let's just cut that back let's cut it down to the bare minimum let's have three or
- four interviews so that's just being more Analytical in your decision-making process or being more analytical to
- improve what you're doing organizationally well that that sounds very counter in and I I I remember uh
- National Public Radio did a piece about Xerox uh and and how they were trying to
- recruit people for call centers and one of the counterintuitive things that they
- discovered was that if somebody had a lot of U experience working for different count uh for different call
- centers in the past that was not necessarily a good thing it might just be that they had a very high burnout
- rate and in fact that was a predictor of bad potentially bad performance uh which
- I found very intriguing through using people analytics uh have there been other such counterintuitive findings and
- what might be some of them mul I I'm not going able to share all the details on this one but I can tell you I got off
- the phone with an NFL team two hours ago and this team is putting a great deal of
- effort into the NFL draft many all teams do to some extent this team happens to be one that is one of the most
- sophisticated and using analytics and um this this this conversation was about their discovering some things
- mattering for a particular position that um nobody would have expected they and this is you know they're doing new and
- better analysis and they're discovering that one of the most important predictors is one that nobody considers
- and this is exactly this is another one of the great Virtues Of let's bring some
- some data to the conversation we're not saying turn the decision over to the data we're saying bring some data to the
- conversation because sometimes you discover these intuitions or even conventional wisdom that you've been
- carrying around for years or decades um not only might it be wrong but it might be the opposite of what is actually
- going on yeah in fact I had meant to ask whether this approach was relevant only
- to companies or whether uh people analytics can also be used in in fields like sports uh so so U for example if a
- football team like say the Philadelphia Eagles uh wanted to use analytics how might they do it well well they are they
- are doing it so um they're going to join us at this conference actually and it's
- not just because they're local it's because they're one of the smartest teams out there on using data um we
- think one of the reasons we want the Eagles at the conference is that we think non-sports organizations can learn
- from sports teams in many ways it's these people decisions are um easier in sports because we have so many
- observables we so many we see so many inputs um to a decision so many quantifiable inputs and then we see so
- many outputs we actually see what these guys do on the field or on the court down the road in fact if you decide not
- to take a player often you'll see how that player turns out in a way that you rarely do in in in non-sports
- organizations if you hire one lawyer you don't often track the career of the other lawyer in sports you can do all
- those things so we can evaluate team's decisions much more carefully and they have many more data
- points to work with as a result for those teams who are interested it's a great opportunity and some teams are
- have really um taken advantage of this the Eagles being one of them so the Eagles are using data at kind of every
- point of the process and they hired a new coach last year who's very data oriented and um we have a lot to learn
- from those guys because they are investing heavily in that and we hope to hear from them at the conference that's
- great uh how could um people analytics be used in performance evaluations which is typically I think one things that a
- lot of managers dread because the idea of giving negative feedback to to uh employees is often quite stressful for
- many managers one of the challenges with performance evaluations is um there's a
- risk of wanting to quantify you know because it's easier if it's not your opinion you just check boxes or
- something's observable or measurable and put all the blame on the number but if you evaluate that way then
- all the weight gets put on these things by the employee and you might care about
- some other things some other things that might not be objectively quantifiable and so performance evaluations
- necessarily require a mix of subjective evaluation and some kind of objective measurement we're never going to take
- away the subjective piece but the idea is that we need to be as systematic and um and consistent as possible with the
- with the Quantified piece so it you know this is very general approach then it could be any attempt to analyze which
- measures are most reliable over time which measures if High one year or high another year is going to help us so
- classic example is performance in um in in fund Management in the investment world so bonuses are a big part of
- compensation there and bonuses are typically related to how your fund performed not perfectly but that's an
- important input there have been studies in some places that sometimes I'm not going to say all places all times that
- say the relationship between a fund manager's performance in one year is kind of unconnected to his performance
- in the next year and if it the idea is if that's true then we probably shouldn't that means that's an indicator
- that there's a lot of chance in this process and that these differences are not functions of skill and if they're
- not functions of skill then maybe we shouldn't be rewarding them heavily each year and that's a hard case to make and
- it's hard for people to appreciate if you just talk about it but if you bring data and you run the numbers and you do
- you know kind of a thorough study you might be able to convince them and you can actually figure out how much of this
- is persistent skill-based and how much of it is chance can you use people analytics to evaluate leadership
- potential that's interesting and I think it's kind I know less about this because
- fewer organizations are doing it but it has to be the case that there's potential there and I'm sure there are
- organizations that are already doing that it's hard I've uh had a conversation one time with um a person
- who had come up through the people analytics organization part of the organization and then moved to talent
- management and in the new role had a um part of her job was to decide on promotion to kind of the executive level
- this was the key promotion level in the organization and I had worked with her for you know a couple years in the
- people analytics side of things and and and she's got a PhD and she's very data oriented and so when I see her for the
- first time in this talent management role and we're talking about this critical decision the organizations
- making about all these employees I just assumed that there would be like a model
- and some data and she'd be using analytics and she's like oh oh no we we can't do that that's like too important
- a decision to I'm in so much trouble if you in this position feel that way after all these
- years I mean but that's I'm sure that over time it's going to become um it's going to play a role there and
- um it's a harder thing clearly but that kind of makes it fun you know what is it
- what is it that we could observe about um employees early in their career that would tell us they have exceptional
- promise it's interesting challenging important question and very closely related to leadership
- is uh the whole idea of Teamwork because so many companies require uh employees to work in teams and increasingly these
- teams are not just in the same location they may be in different cities maybe even different countries different time
- zones uh can people analytics tell us anything about how team teamwork can become more effective um I I would like
- to think so it's it's hard to imagine you couldn't and again one I mean the way to think about is just very
- generally it's it's just it's just analysis essentially it's just taking it's just quantitative
- analysis the the leap is we don't typically think about doing it for this very interpersonal or traditionally soft
- issue so it's simply what how can we measure something about the group that would allow us to to say something so
- this is a great example of where Sports is informative um now how we translate it
- to sports is a challenge and it's going to take time but for example in hockey they've had to learn
- how to evaluate players even though you never see a player on the ice by himself
- you always see him on a team of you know six people um Five Guys three FR offensive um
- whatever basketball same thing you have to figure out how to evaluate a player on the same in fact he's not only just
- on the court with five of his teammates he's got five opponents as well and so they've had to come up with ways of
- evaluating the group team performance that reflects something about the individual how do we pull the individual
- out of that group and teams are getting more and more sophisticated at how to do
- that maybe there are things that we can borrow from the NBA that would help us better understand group performance on
- an engagement team at McKenzie for example right uh what are some what do you believe to be some of the greatest
- concerns about people analytics I mean are there issues of um for example how do companies drop
- boundaries around the privacy and the use of data and metadata yeah uh and is there any regulation around these kinds
- of issues um that's a I don't know the answer on the regulation side it is certainly a sensitive issue um and you
- know I've seen changes in company's willingness to collect these data even just in the last three or four years
- early on in a in a in a conversation at one organization we were talking about how can we make use of um how can we
- gather new data to evaluate what's going on and we were talking about organizational culture at the time and I
- was sitting next to a sociologist and this particular sociologist specialty was studying like
- 12th century church and you know I'm I'm much more of a psychologist and if I think about data
- if I'm GNA get really creative on data it's going to be something like you know cortisol or something
- and it's funny companies don't let us collect cortisol from their employees this guy is a guy who had to be very
- creative about um collecting data he's studies 12th century church and he came up with this idea of what can we
- monitor passively what where employees are like and we had all these interesting ideas on that might actually
- say something about the organizational culture I was very impressed that was from this very different um disciplinary
- perspective at the time that seemed even that seemed scary the organization was like we can't do that and now you read
- about this you read about you know little IDs on badges that that capture where everybody is all the time or um
- meters on chairs so they'll know when people are actually using space and they are that's where the world is going and
- it is scary to some people it's obviously Rife with some ethical issues and that is a risk um clearly that's a
- risk one last question if you look if we were to look to the Future uh what what do you think people
- analytics will be able to do say 5 years from now uh that it can't do today that's interesting
- mol it's gonna it's got to come from the Big Data side of things and the passive
- data side of things I would think um and again I look at kind of the frontier of
- this is in the Sports World and to watch what they have done in the last five years is to see I think where we're
- going to go in the next 20 so the NBA for example Now tracks every player and the ball
- multiple times a second throughout the game and then given enough computing power and enough phds some people have
- been able to go in and do some really good performance evaluation with those data it took a lot of resources and it
- took some good thinking but they have come up with incredibly insightful performance evaluations based on based
- on those data and I think we're just going to keep on discovering more and more examples like that as those kinds
- of Technologies filter down to non-sports teams well uh okay thank you so much for speaking with knowledge at
- won today and good luck with the conference thank you MCO appreciate [Music] it [Music]
Brief Summary
Kade Massie discusses how people analytics transforms hiring and performance evaluations using data to reduce biases and improve decision-making.
Key points
- The Rise of People Analytics. People analytics is transforming HR by using data to make unbiased decisions.
- Counterintuitive Findings in Recruitment. Xerox discovered that prior call center experience could indicate potential burnout.
- Future of People Analytics. The future may see advanced data tracking similar to sports analytics.
Episode highlights
- The Rise of People AnalyticsPeople analytics is transforming HR by using data to make unbiased decisions.“People analytics is a data-driven approach to managing people at work.”0:51Mar 21, 2014
- Counterintuitive Findings in RecruitmentXerox discovered that prior call center experience could indicate potential burnout.“A lot of U experience might predict bad performance.”7:20Mar 21, 2014
- Future of People AnalyticsThe future may see advanced data tracking similar to sports analytics.“We're just going to keep discovering more examples as technologies filter down.”19:37Mar 21, 2014
Tension
40/ 100Low tension





