Management by the Numbers

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

People analytics isn't perfect -- yet. But it helps eliminate biases in important areas such as recruitment, says Wharton's Cade Massey.

Read the full story: http://knlg.net/1ev2WN9 📂 Management 🏷️ Research

Read the full transcriptHide the full transcript
  1. managing people was once regarded as an art rather than as a science but these days companies have begun to use
  2. sophisticated data analysis for all sorts of people related issues ranging from Recruitment and performance
  3. evaluation to promotion and compensation people analytics as this approach is called is Making Waves in
  4. the field of Human Resources because it is said to eliminate biases that exist in human judgment Warden plans to host a
  5. conference on this topic on March 28th and today knowledge at Wharton is pleased to welcome kid Massie a
  6. professor in the operations and information management department who has been leading this effort Kade
  7. welcome and thank you so much for joining us today thank you MCO very glad to be here so uh people analytics has
  8. been described as a datadriven approach to managing people at work uh I wonder if you could help me understand what
  9. that means and how it differs from traditional ways of managing people MH well traditionally people have made
  10. decisions around who to hire who to promote who to compensate the best um intuitively or with some numbers but
  11. maybe not in a systematic way and people analytics is an attempt to bring a little more systematic processes to
  12. these decisions there's some of the most important decisions that an organization
  13. makes and um we've seen good use of data in other fields finance and marketing it's just kind of the slow evolution of
  14. using those data in in new Fields so how EXA exactly does that work does it really eliminate biases well it helps
  15. and this is one of the this is one of the motivations that I have I study decision- making and it's very much
  16. about the biases people have um in their intuitions and these are hard to root out and bringing some data to the
  17. process and more systematic decision- making more atic analysis is one way of doing it eliminates a big word um I'm
  18. sure that there are some situations where we can take in a bias all the way down but mostly we're just trying to
  19. improve right so so maybe if we were to look at this in specific terms it would help make it a little clearer to me uh
  20. let's start with Recruitment and the first step uh which is the first step in a company's engagement with an employee
  21. now say at the University of Pennsylvania traditionally if we wanted to fill a job we would uh post it on our
  22. website we would look all the look at all the resumés that came in we would look through the resumés to see who are
  23. the most qualified people call them in for an interview and then try to find the person who was the best fit now now
  24. uh using people analytics uh how does this process change exactly so if if we were to come in and try to bring some of
  25. these tools to that process probably the first thing we would do would be to look
  26. at as much historical data as possible to understand what attributes in these candidates predict performance long term
  27. so can we can we model just basically rather than interviewing them necessarily we can look at their
  28. characteristics off of a Vita or um their application and ask what's the relationship between these observables
  29. and long-term performance it doesn't mean necessarily that we'll give that model 100%
  30. discretion on who gets hired but we use that model's output to inform our decisions so we'll probably still do the
  31. interview we'll probably still have a group discussion but to have some analytical rigor ining that discussion
  32. would be better than most processes which don't have it so um the other thing we might do is to model the the
  33. admissions decisions themselves so rather than modeling long-term performance of the candidate we say well
  34. what are we doing right now who are we deciding to put in and you can go in and find out well
  35. whether you know it or not implicitly you've been putting 20% weight on GPA and 50% weight on The Prestige of the
  36. company they work for and 30% on whatever you can just ask what have we been doing even though we we don't
  37. follow some rules we're implicitly going to end up relying on some rules that's often enlightening organization might
  38. not know it's like we didn't know that our process put that much weight maybe that's okay maybe they decide they'd
  39. rather shift it and put weight on something else so you you mentioned that ter uh uh being able to predict uh the
  40. performance how how accurate are these predictions typically well they're going to vary dramatically and um they're
  41. never going to be perfect these are noisy processes and one of the things that actually I think this is a virtue
  42. of people analytics if you're if you're if you're if you're if you're really being thoughtful about it one of the
  43. things you carry away is a deep appreciation for how much chance there is how much noise there is how imperfect
  44. the process is even the best model with the best data um is still imperfect and you know that's actually an important
  45. lesson we don't typically carry that lesson around about the decisions we make in the interview and hiring process
  46. usually we think yeah I can predict that and you remember the ones that you predicted well and you conveniently
  47. forget the ones that you didn't predict so well and you never develop that humility about the process start
  48. modeling the process and you start appreciating the difficulty so imperfect prediction for sure
  49. but um the idea is that we can improve we can we can we can do better we can be more accurate by adding some analysis to
  50. the intuition of the people who are doing the decisions making the decisions right so I've heard that companies like
  51. Google and Xerox for example have been using uh this approach to to recruitment uh how good have been the results that
  52. they've seen well you know people it's not all about modeling performance it can be any kind of
  53. analytics applied to these processes so one thing for example that Google did a few years ago was to ask how good are
  54. interviews for predicting um you know performance on the job and they discovered that you know the literature
  55. says they're not very good but a lot of times people want to you know prove it to themselves in their own organization
  56. and Google is strongly that way so Google says we'll study ourselves and they run the numbers and they discover
  57. well in fact interview don't do a whole lot and um they serve other purposes and
  58. so maybe you want to not throw them away but they were spending you know hours of
  59. their manager's time interviewing candidates you know eight interviews nine interviews 10 interviews and they
  60. discover they don't do much well let's just cut that back let's cut it down to the bare minimum let's have three or
  61. four interviews so that's just being more Analytical in your decision-making process or being more analytical to
  62. improve what you're doing organizationally well that that sounds very counter in and I I I remember uh
  63. National Public Radio did a piece about Xerox uh and and how they were trying to
  64. recruit people for call centers and one of the counterintuitive things that they
  65. discovered was that if somebody had a lot of U experience working for different count uh for different call
  66. centers in the past that was not necessarily a good thing it might just be that they had a very high burnout
  67. rate and in fact that was a predictor of bad potentially bad performance uh which
  68. I found very intriguing through using people analytics uh have there been other such counterintuitive findings and
  69. what might be some of them mul I I'm not going able to share all the details on this one but I can tell you I got off
  70. the phone with an NFL team two hours ago and this team is putting a great deal of
  71. effort into the NFL draft many all teams do to some extent this team happens to be one that is one of the most
  72. sophisticated and using analytics and um this this this conversation was about their discovering some things
  73. mattering for a particular position that um nobody would have expected they and this is you know they're doing new and
  74. better analysis and they're discovering that one of the most important predictors is one that nobody considers
  75. and this is exactly this is another one of the great Virtues Of let's bring some
  76. some data to the conversation we're not saying turn the decision over to the data we're saying bring some data to the
  77. conversation because sometimes you discover these intuitions or even conventional wisdom that you've been
  78. carrying around for years or decades um not only might it be wrong but it might be the opposite of what is actually
  79. going on yeah in fact I had meant to ask whether this approach was relevant only
  80. to companies or whether uh people analytics can also be used in in fields like sports uh so so U for example if a
  81. football team like say the Philadelphia Eagles uh wanted to use analytics how might they do it well well they are they
  82. are doing it so um they're going to join us at this conference actually and it's
  83. not just because they're local it's because they're one of the smartest teams out there on using data um we
  84. think one of the reasons we want the Eagles at the conference is that we think non-sports organizations can learn
  85. from sports teams in many ways it's these people decisions are um easier in sports because we have so many
  86. observables we so many we see so many inputs um to a decision so many quantifiable inputs and then we see so
  87. many outputs we actually see what these guys do on the field or on the court down the road in fact if you decide not
  88. to take a player often you'll see how that player turns out in a way that you rarely do in in in non-sports
  89. organizations if you hire one lawyer you don't often track the career of the other lawyer in sports you can do all
  90. those things so we can evaluate team's decisions much more carefully and they have many more data
  91. points to work with as a result for those teams who are interested it's a great opportunity and some teams are
  92. have really um taken advantage of this the Eagles being one of them so the Eagles are using data at kind of every
  93. point of the process and they hired a new coach last year who's very data oriented and um we have a lot to learn
  94. from those guys because they are investing heavily in that and we hope to hear from them at the conference that's
  95. great uh how could um people analytics be used in performance evaluations which is typically I think one things that a
  96. lot of managers dread because the idea of giving negative feedback to to uh employees is often quite stressful for
  97. many managers one of the challenges with performance evaluations is um there's a
  98. risk of wanting to quantify you know because it's easier if it's not your opinion you just check boxes or
  99. something's observable or measurable and put all the blame on the number but if you evaluate that way then
  100. all the weight gets put on these things by the employee and you might care about
  101. some other things some other things that might not be objectively quantifiable and so performance evaluations
  102. necessarily require a mix of subjective evaluation and some kind of objective measurement we're never going to take
  103. away the subjective piece but the idea is that we need to be as systematic and um and consistent as possible with the
  104. with the Quantified piece so it you know this is very general approach then it could be any attempt to analyze which
  105. measures are most reliable over time which measures if High one year or high another year is going to help us so
  106. classic example is performance in um in in fund Management in the investment world so bonuses are a big part of
  107. compensation there and bonuses are typically related to how your fund performed not perfectly but that's an
  108. important input there have been studies in some places that sometimes I'm not going to say all places all times that
  109. say the relationship between a fund manager's performance in one year is kind of unconnected to his performance
  110. in the next year and if it the idea is if that's true then we probably shouldn't that means that's an indicator
  111. that there's a lot of chance in this process and that these differences are not functions of skill and if they're
  112. not functions of skill then maybe we shouldn't be rewarding them heavily each year and that's a hard case to make and
  113. it's hard for people to appreciate if you just talk about it but if you bring data and you run the numbers and you do
  114. you know kind of a thorough study you might be able to convince them and you can actually figure out how much of this
  115. is persistent skill-based and how much of it is chance can you use people analytics to evaluate leadership
  116. potential that's interesting and I think it's kind I know less about this because
  117. fewer organizations are doing it but it has to be the case that there's potential there and I'm sure there are
  118. organizations that are already doing that it's hard I've uh had a conversation one time with um a person
  119. who had come up through the people analytics organization part of the organization and then moved to talent
  120. management and in the new role had a um part of her job was to decide on promotion to kind of the executive level
  121. this was the key promotion level in the organization and I had worked with her for you know a couple years in the
  122. people analytics side of things and and and she's got a PhD and she's very data oriented and so when I see her for the
  123. first time in this talent management role and we're talking about this critical decision the organizations
  124. making about all these employees I just assumed that there would be like a model
  125. and some data and she'd be using analytics and she's like oh oh no we we can't do that that's like too important
  126. a decision to I'm in so much trouble if you in this position feel that way after all these
  127. years I mean but that's I'm sure that over time it's going to become um it's going to play a role there and
  128. um it's a harder thing clearly but that kind of makes it fun you know what is it
  129. what is it that we could observe about um employees early in their career that would tell us they have exceptional
  130. promise it's interesting challenging important question and very closely related to leadership
  131. is uh the whole idea of Teamwork because so many companies require uh employees to work in teams and increasingly these
  132. teams are not just in the same location they may be in different cities maybe even different countries different time
  133. zones uh can people analytics tell us anything about how team teamwork can become more effective um I I would like
  134. to think so it's it's hard to imagine you couldn't and again one I mean the way to think about is just very
  135. generally it's it's just it's just analysis essentially it's just taking it's just quantitative
  136. analysis the the leap is we don't typically think about doing it for this very interpersonal or traditionally soft
  137. issue so it's simply what how can we measure something about the group that would allow us to to say something so
  138. this is a great example of where Sports is informative um now how we translate it
  139. to sports is a challenge and it's going to take time but for example in hockey they've had to learn
  140. how to evaluate players even though you never see a player on the ice by himself
  141. you always see him on a team of you know six people um Five Guys three FR offensive um
  142. whatever basketball same thing you have to figure out how to evaluate a player on the same in fact he's not only just
  143. on the court with five of his teammates he's got five opponents as well and so they've had to come up with ways of
  144. evaluating the group team performance that reflects something about the individual how do we pull the individual
  145. out of that group and teams are getting more and more sophisticated at how to do
  146. that maybe there are things that we can borrow from the NBA that would help us better understand group performance on
  147. an engagement team at McKenzie for example right uh what are some what do you believe to be some of the greatest
  148. concerns about people analytics I mean are there issues of um for example how do companies drop
  149. boundaries around the privacy and the use of data and metadata yeah uh and is there any regulation around these kinds
  150. of issues um that's a I don't know the answer on the regulation side it is certainly a sensitive issue um and you
  151. know I've seen changes in company's willingness to collect these data even just in the last three or four years
  152. early on in a in a in a conversation at one organization we were talking about how can we make use of um how can we
  153. gather new data to evaluate what's going on and we were talking about organizational culture at the time and I
  154. was sitting next to a sociologist and this particular sociologist specialty was studying like
  155. 12th century church and you know I'm I'm much more of a psychologist and if I think about data
  156. if I'm GNA get really creative on data it's going to be something like you know cortisol or something
  157. and it's funny companies don't let us collect cortisol from their employees this guy is a guy who had to be very
  158. creative about um collecting data he's studies 12th century church and he came up with this idea of what can we
  159. monitor passively what where employees are like and we had all these interesting ideas on that might actually
  160. say something about the organizational culture I was very impressed that was from this very different um disciplinary
  161. perspective at the time that seemed even that seemed scary the organization was like we can't do that and now you read
  162. about this you read about you know little IDs on badges that that capture where everybody is all the time or um
  163. meters on chairs so they'll know when people are actually using space and they are that's where the world is going and
  164. it is scary to some people it's obviously Rife with some ethical issues and that is a risk um clearly that's a
  165. risk one last question if you look if we were to look to the Future uh what what do you think people
  166. analytics will be able to do say 5 years from now uh that it can't do today that's interesting
  167. mol it's gonna it's got to come from the Big Data side of things and the passive
  168. data side of things I would think um and again I look at kind of the frontier of
  169. this is in the Sports World and to watch what they have done in the last five years is to see I think where we're
  170. going to go in the next 20 so the NBA for example Now tracks every player and the ball
  171. multiple times a second throughout the game and then given enough computing power and enough phds some people have
  172. been able to go in and do some really good performance evaluation with those data it took a lot of resources and it
  173. took some good thinking but they have come up with incredibly insightful performance evaluations based on based
  174. on those data and I think we're just going to keep on discovering more and more examples like that as those kinds
  175. of Technologies filter down to non-sports teams well uh okay thank you so much for speaking with knowledge at
  176. won today and good luck with the conference thank you MCO appreciate [Music] it [Music]

Brief Summary

Kade Massie discusses how people analytics transforms hiring and performance evaluations using data to reduce biases and improve decision-making.

Key points

  • The Rise of People Analytics. People analytics is transforming HR by using data to make unbiased decisions.
  • Counterintuitive Findings in Recruitment. Xerox discovered that prior call center experience could indicate potential burnout.
  • Future of People Analytics. The future may see advanced data tracking similar to sports analytics.

Tension

40/ 100Low tension

Related episodes