Who Made That Decision: You or an Algorithm?

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

When we buy something on Amazon or watch something on Netflix, we think it’s our own choice. Well, it turns out that algorithms influence one-third of our decisions on Amazon and more than 80% on Netflix. What’s more, algorithms have their own biases. They can even go rogue. In his recent book titled, A Human's Guide to Machine Intelligence: How Algorithms Are Shaping Our Lives and How We Can Stay in Control, Kartik Hosanagar, a professor of operations, information and decisions at Wharton, focuses on these issues and more. He discusses how algorithmic decisions can go wrong and how we can ensure we can control the way technology impacts decisions that are made for us or about us.

In a conversation with Knowledge@Wharton, Hosanagar notes that a solution to this complex problem is that we must "engage more actively and more deliberately and be part of the process of influencing how these technologies develop." 📂 Technology 🏷️ Research

Read the full transcriptHide the full transcript
  1. our guest today is Karthik hosonaga a professor of technology digital business and marketing at Wharton and we are
  2. speaking with him about his recent book titled a human's guide to machine intelligence how algorithms are shaping
  3. our lives and how we can stay in control Karthik welcome to knowledge at Wharton
  4. thank you so much for speaking with us today well cool thanks for having me it's
  5. always a pleasure to talk to you and the knowledge is important GRU thanks so these days there's a growing buzz
  6. about artificial intelligence and machine learning and quite a few books I have have come out recently on these
  7. topics in all the conversations that are going on what do you think are some of the points that are being overlooked are
  8. not sufficiently emphasized and how does your book seek to fill that gap yeah clearly there's a lot of buzz around AI
  9. and machine learning which is a subfield of AI I think the conversation tends to
  10. you know either glorify the technology or in many instances lately create a lot of fear-mongering around it and you know
  11. I don't think the conversation has focused on you know what's the solution how are we going to work with AI and
  12. especially in the context of making decisions and so my book is focused on making decisions through intelligent
  13. algorithms and certainly we have various kinds of AI but one of the core questions when it comes to AI is are we
  14. going to use AI to make decisions if so are we going to use it in a decision support way are we going to have the AI
  15. make decisions autonomously if so what can go wrong what can go well and how do we manage this because we know AI has a
  16. lot of potential but I think there will be some growing pains on our way there and so those growing pains is what I
  17. focus on how can algorithmic decisions go wrong and how do we make sure that we have control over the narrative of how
  18. technology impacts the decisions that are made for us or about us I'd love to come back to the part of our
  19. decision-making an algorithmic algorithmic decision-making but I really love the way you began the book with
  20. some very striking examples about chatbots and and how they interact with humans I wonder if you could use that
  21. illustration to talk a little bit about how human beings interact with algorithms and what some of the
  22. implications are I began the book with a description of Microsoft's experience with a chat part called Xiao Bing in
  23. China it's called Xiao Bing and elsewhere in the world it's called Xiao Weis this was a chat board created in
  24. the avatar of a teenage girl and it's meant to engaged in engage in fun playful conversations with young adults
  25. and teenagers and this chat pot has about 40 million followers in China and the reports say that roughly a quarter
  26. of those followers have said I love you to Xiao Weis so that's the kind of affection and following Xiao Weis has so
  27. inspired by the success of Xiao Weis in China Microsoft decided to test a similar chat bot in the US and they
  28. created a chat bot in English which would engage again in fun playful conversations and targeted once again at
  29. young adults and teenagers they launched it on twitter under the name Tay ta y and this chat BOTS experience was
  30. actually very different and it's it was very short-lived experience as well because within an hour of launching the
  31. chat pod turned sexist racist fascist it tweeted very offensively and said things
  32. like Hitler was right and Microsoft shut it down within 24 hours and later that year mi t--'s Technology Review rated
  33. Microsoft Day as the worst technology of the year and that incident you know opened up this question for me which was
  34. how can two similar chart parts or pieces of AI built by the same company produce such
  35. different results and what does that say about you know our decision to to use algorithms for a lot of our decisions in
  36. our personal and professional lives and that's what helped start this exploration into use of AI the extent to
  37. which AI can be predictable biased as in the case of Microsoft a and then of course how do we what does that mean in
  38. terms of using these systems to make significant decisions for us so why did T experiences differ so dramatically and
  39. is anything that can be done about that one of the insights that I got as I was writing this book as I was trying to
  40. explain the differences in behavior of these two chatbots was actually from human psychology
  41. so psychologists describe human behavior in terms of our nature and our nurture and so our nature is our genetic code
  42. and nurture is our environment and so psychologists will attribute problematic issues like alcoholism partly to nature
  43. and partly to nurture and when I was looking at algorithms I realized algorithms have nature and nurture as
  44. well so the nature for algorithms is not genetic code but the code that the engineer actually codes in or writes and
  45. that's the logic of the algorithm the nurture for the algorithm is the data from which algorithms learn and so
  46. increasingly as we are moving towards machine learning we're moving from a world where engineers used to program
  47. the end-to-end logic of an algorithm would actually specify what happens when any situation happens if this happens
  48. you respond this way if that happens you respond a different way and so it used to be all nature because the programmer
  49. gave all the very minut specifications of how the algorithm will work but as we move towards machine learning we're kind
  50. of telling algorithms here's data learn from it and so nature starts to become less important and nurture starts to
  51. dominate so if you look at what happened between tea and showers the difference is in terms of their training
  52. data in some ways and in particular in the case of showers showers was created to mimic how people converse on Tay and
  53. so it was picking up how people are talking to it and it would reflect that and so there were many intentional
  54. efforts to trip day and so there was nurture there as well and part of it was nature as well meaning that the court
  55. could have specified certain rules the court could have specified rules like do nots say the following kinds of things
  56. or do not get into discussions of these topics and so on so it's a bit of both and I think that's what in general rogue
  57. algorithmic behavior comes down to that come back a little later to the whole question of what happens go rogue yeah
  58. but I wanted to chat a little bit about the way in which this algorithmic decision-making itself has changed and
  59. it wasn't there was a time when decision-making algorithm decision-making seemed to be almost like
  60. Amazon will tell you what books you should read or Netflix will recommend which movies you should watch but
  61. because of AI algorithmic algorithmic decision-making has become a lot more complex and I was wondering if you could
  62. offer some examples and what are some of the implications on the choices that we
  63. make or don't make as a result of this yeah algorithms pervade our lives and sometimes we see it like Amazon's
  64. recommendations and sometimes we don't realize it but they have a huge impact on decisions we make on Amazon for
  65. example over 1/3 of the choices that we make are influenced by algorithmic recommendations people who bought this
  66. also bought those people who viewed this eventually bought that and so on on Netflix over 80% of the viewing activity
  67. is driven by algorithmic recommendations they also drive decisions such as who we
  68. date and Mary when you look at applications or apps like tinder which actually ABB's we're algorithms create most of
  69. the matches over there they're also at the workplace you know for example you make you apply for our
  70. alone mortgage approval decisions are made by algorithms increasingly you know if you apply for a job resumes screaming
  71. algorithms are deciding which ones to invite for an interview and they're making life-and-death decisions as well
  72. there for example being used in the criminal justice system in courtrooms in the u.s. there are algorithms that
  73. predict the likelihood that the defendant will reoffend so that judges can make sentencing decisions in
  74. medicine we're moving towards personalized medicine so that two people with the same symptoms might not get the
  75. same treatment it might be customized based on their DNA profile algorithms again will guide the doctors on those
  76. decisions and also we're moving as AI has progressed and advanced we're moving to a point where the algorithms don't
  77. merely offer decision support they can function autonomously as well and driverless cars are a great example of
  78. that where we're trying to pretty much say you can automate the whole process and and have algorithms function without
  79. a human making the final sort of decision so as more algorithms influence or make more and more decisions
  80. is there anything like free will in the world anymore well so free will is an interesting concept and for the most
  81. part I used to think of free will in a very philosophical sense right and philosophers have argued we don't even
  82. have free will but that was in a very different very as I said philosophical interpretation but I think we have a
  83. literal interpretation of free will now in the context of algorithms which is are you making the final choice and I
  84. just said you know a third of your choices are driven on Amazon by recommendations 80% of viewing
  85. activities on Netflix are driven by algorithmic recommendations at YouTube seventy percent of the time people spend
  86. on YouTube is driven by algorithmic recommendations so it doesn't feel like algorithms are merely recommending to us
  87. what we want and we make decisions I mean think about a search on Google you take the most
  88. Oh Tarek of search terms like vintage toy model trains or something like that you will still find hundreds of
  89. thousands perhaps even millions of results for that search term we might see less than 0.01% of the search
  90. results because rarely do we even cross page one the algorithm has decided which
  91. pages we look at so yes they're making a lot of choices for us so do we have free
  92. well at some level yes but mostly I'm going to say we don't have the level of independent decision-making that we
  93. think we do because most of us think we get these recommendations we not politely and then we do what we want but
  94. indeed the algorithms are not jiggles in very interesting ways and and I don't think we have the level of independent
  95. decision-making or choice that we think we do mostly that's a good thing in some
  96. ways because they're saving us time so we can focus on leisure instead of wasting our time sifting through lots of
  97. alternatives but sometimes we become very passive about how we use algorithms and becoming that passive about
  98. algorithm use can have consequences so let's talk a little bit about those consequences okay especially unintended
  99. consequences yes there's a fascinating part on your book about that and how do design choices lead to unintended
  100. consequences I was wondering if you could speak about that yes so when I mention unintended consequences in the
  101. book I'm referred referring to situations where you know you're trying to optimize some aspect of a decision
  102. and perhaps you manage to improve that really well but then something else goes wrong
  103. so an example might be that when Facebook was manually curating its trending stories through human editors
  104. as an editor you will appreciate that it's real work right and they had people doing that but then Facebook was accused
  105. of having a left-leaning bias that these editors were choosing left-leaning stories and curating those more often so
  106. they said you know algorithms can't be accused of a political bias so they used an algorithm to curate this they tested
  107. it for political bias it did not have any political bias but there's something else it did which they
  108. hadn't explicitly tested for which is as we know it curated fake new stories and
  109. and circulated them so that's an example of unintended consequences and an algorithm design can drive that in many
  110. ways you know I've done a lot of work on recommendation systems and how they influence the kinds of products we
  111. consume the kinds of media we consume and I've specifically studied two kinds of recommendation algorithms one kind of
  112. algorithm is based or it's like the Amazon people who bought this also bought this so it's based on social
  113. curation what are the what are others consuming the other kind of algorithm attempts to understand at a deep level
  114. what is it that I'm recommending and the algorithm is recommending and tries to find items that are very similar to the
  115. users interest an example of that would be Pandora so Pandora's music recommendations are not people who like
  116. this song also like these other songs Pandora actually has very detailed information over 150 attributes for each
  117. song musical attributes like how rhythmic is the song how much instrumentation is there in the music
  118. and every time you say you like a music a song or you don't like a song they look at the musical qualities of the
  119. song and then they adjust their recommendations based on other songs which have attributes similar to what
  120. you have now I looked at both these designs and I looked at which design is more helpful in helping us find let's
  121. say indie songs or very novel and niche books or movies and at the time we did the study and this was some time back
  122. the conventional wisdom was that all of these algorithms help in pushing the long tail meaning these nice novel items
  123. or indie songs that nobody's heard of and what I found was that these designs were very different the algorithm that
  124. looks at the look said you know what others are consuming people who bought this also bought this it has a
  125. popularity bias because it's trying to recommend stuff that others are consuming and so it
  126. tends to lean towards popular items and so it cannot truly recommend those hidden gems but an algorithm like the
  127. Pandora algorithm is it doesn't have popularity as a basis for recommendations so it tends to do better
  128. and that's why what we've seen now is companies like Spotify Netflix and many others have changed the design of their
  129. algorithms they've combined the two approaches they've combined the the social appeal of a system that looks at
  130. what others are consuming and the ability of the other design to surface hidden gems let's go now to the point
  131. you were up earlier about algorithms going rogue yes why does that happen and what can be done about it yeah if we
  132. look at let me point a couple example examples of algorithms doing rolled and then we'll talk about why this happens
  133. so I mentioned algorithms are used in courtrooms in the u.s. in the criminal justice system in 2016 there was a
  134. report or study done by ProPublica which is a nonprofit organization they looked
  135. at algorithms used in courtrooms and they found that these algorithms have a race bias specifically they found that
  136. these algorithms were twice as likely to falsely predict future criminality in a
  137. black defendant than a white defendant and this was race bias in that algorithm late last year Reuters carried a story
  138. about Amazon trying to use algorithms to screen job applications and the you know
  139. again Amazon gets a million plus job applications they hire hundreds of thousands of people you know over the
  140. last few years they've done that it's hard to do that manually and so you need algorithms to help automate some of this
  141. but they found that the algorithms tended to have a gender bias they tended to reject female applicants more often
  142. even when the qualifications were similar now Amazon ran the test and concluded and realized that there are
  143. savvy companies so they decided not to roll this out there's probably many other companies that
  144. using algorithms to screen resumes and they might be prone to race bias gender bias and so on so I've mentioned two
  145. examples now race bias gender bias I talked about fake news on on Facebook so there's many examples in terms of why
  146. they go rogue there's a couple reasons I can share one is as we have gone from those old traditional algorithms where
  147. the programmer wrote up the algorithm end to end and we've moved towards machine learning we have created
  148. algorithms that are usually more resilient because these algorithms you know can perform much better but they're
  149. prone to biases that exist in the data so if the data have biases so for example if you tell a resume screening
  150. algorithm you know here's data on all those people who applied to our job and here's the people we actually hired and
  151. here are the people who we promoted now figure out who to invite for job interviews based on this data the
  152. algorithm will observe that in the past you are rejecting more female applications or you were not promoting
  153. women at the workplace and will tend to pick up that behavior so it tends to pick that up I think the other piece is
  154. that engineers in general tend to focus narrowly on one or two metrics you know with a resume screening application you
  155. will tend to measure the accuracy of your model and if it's highly accurate you'll roll it out but you don't
  156. necessarily look at fairness and bias or in Facebook's example look at fake news and other
  157. possibilities as well what are the challenges involved in autonomous algorithms making decisions on our
  158. behalf well I think when you have autonomous algorithms making decision on our behalf I think one of the big
  159. challenges is there is usually no human in the loop so we lose control and that is one one challenge and many studies
  160. show that when we have limited control we are less likely to trust algorithms and so that
  161. is one one aspect that is challenging the other piece about having autonomous algorithms is that again if there's a
  162. human in the loop there's a greater chance that the user can detect certain problems and you know the likelihood
  163. that problems get detected is therefore greater I'm really glad you brought up the point about trusting algorithms
  164. because you tell this really fascinating story in the book about a patient who gets diagnosed with definitely fever I
  165. wonder if you could share that story with our audience and and spell out some of the implications for trust in
  166. algorithms and and what the implications are yeah the story that I share is that
  167. of a patient walking into a doctor's office and the patient feels fine and healthy and the patient and doctor are
  168. joking around and doctor eventually picks up the pathology report and he suddenly looks very serious and he's
  169. says or informs the patient that I'm sorry to let you know that you have to happen early fever and the patient
  170. hasn't heard of that Bernoulli fever and he asks what exactly is it and so the doctor says it's a very rare disease and
  171. it's known to be fatal and so I suggest that you have this tablet and it will you know reduce the chance that you will
  172. have any problems and you know he says here you take this you know three times a day and and and then you go about your
  173. life and I asked my readers if you know that story is something that if they were the patient they would feel
  174. comfortable in that situation here's a disease you don't nothing about and here's a solution you know nothing about
  175. and the doctor has given you the choice and told you to go ahead but not given you very many details and with that I
  176. pose the question if an algorithm way to again make this wreck that you have this rare disease and we
  177. want you to take this medication without any information would you and of course Mukul you've read the book
  178. so you know Tappin le fever is not a real disease I'm a fan of Sherlock Holmes I used to read it a lot as a kid
  179. and it's a disease in a one of the Sherlock Holmes stories and you know that inspired me to consider this
  180. because even in the original Sherlock Holmes story it turns out that the person who has definitely fever doesn't
  181. actually have it but setting that aside it kind of brings up this question of transparency you know are we willing to
  182. trust decisions when we don't have information about why a certain decision was made the way it was and what I
  183. highlight is sometimes we're seeking more transparency from algorithms than humans but in practice lots of companies
  184. are imposing algorithmic decisions on us without any information about why these
  185. decisions are being made and are we fine with that and a lot of research shows that we're not fine with that research
  186. for example that one PhD student then a PhD student at Stanford looked at an algorithm that would compute grades for
  187. students and how did they do when they just got their score versus they got their score with an explanation and as
  188. you expect when they have an explanation they trust it more then why is it that in in the real world there's a lot of
  189. algorithms making decisions for us or about us and we have no transparency about those decisions and so I advocate
  190. that we need a certain level of transparency with regard to for example what kinds of data were used to make the
  191. decision so for example if we applied for a loan and the loan was rejected we would like to know why that was the case
  192. if you applied for a job and it was rejected it would be helpful to know that the algorithm not only evaluated
  193. what you submitted as part of your job application would also looked at your social media posts and so transparency
  194. regarding what data was considered what were the key factors that drove a decision
  195. is important but what's the end of the book you recommend an algorithmic Bill of Rights what exactly is that and why
  196. is it necessary so the algorithmic Bill of Rights is a concept that I borrowed from the Bill of Rights in the US
  197. Constitution and the history of the Bill of Rights is that when the founding fathers were setting up or drafting the
  198. Constitution some people were worried that we're creating a very powerful government here in the US and the Bill
  199. of Rights was created as a way to protect citizens now today we are in a situation where there's a lot of talk
  200. about powerful tech companies you know everywhere you see there's new stories big tech companies and what are we going
  201. to do about them and so there's a sense that again consumers need certain protections and so the Bill of Rights is
  202. targeted at that but before I talk about the Bill of Rights one aspect related to
  203. the Bill of Rights I do want to address is that a lot of consumers feel that they're helpless against big tech and
  204. against algorithms deployed by big tech and personally I feel that consumers do have some power and that power is in
  205. terms of our knowledge our votes and our dollars you know knowledge is about you
  206. know we shouldn't be passive users of technology we should be active and deliberate about it we should know how
  207. it's changing decisions we are making or others are making about us and at some level it seems like you know knowledge
  208. is fine but what can I do with that knowledge but I mean look at how Facebook is changing their product
  209. design today and that changes you know support for encryption and so on is because of push from the users and it
  210. shows that when users complain changes to happen votes there are another aspect of that and votes are all about you know
  211. us being aware of which elected representatives understand the nuances of algorithms and the challenges and how
  212. to regulate them and it's about voting for them right and the question is how are these regulators going to protect us
  213. that's where the bill of rights comes in and the Bill of Rights I propose has a few key pillars one pillar is
  214. transparency transparency with regard to the data used to make decisions and with
  215. regard to the underlying decision itself what were the most important factors that led to a certain decision today
  216. Europe's GDP are actually has certain provisions like write to explanations and information on the data that
  217. companies are using and so I think some of that transparency is needed and companies should provide that another
  218. pillar in my bill of rights is the idea of some user control that we cannot be in an environment where we have no
  219. control over the technology we should for example be able to with a simple instruction tell Alexa you're not
  220. listening to any conversation in the house until I'd instruct you that it's allowed right there's no such provision
  221. we're told that the system is not listening but then we're also hearing from others that there are instances
  222. where it listens even when you're not actually saying Alexa and giving an instruction and that control is very
  223. important if you look at Facebook the false news issue two years back there was no way for users to alert Facebook's
  224. algorithm and say this post in my newsfeed is false news even though users were seeing false news in their newsfeed
  225. even today with just two clicks you can let Facebook know that a certain news post in your feed is either offensive or
  226. it has false news and that feedback is so important for the algorithm to now correct itself and previously users had
  227. no way even though you're observing problems no way of informing the algorithm or helping the algorithm
  228. course correct so some level of user control is also another pillar I proposed and lastly I
  229. have been advocating this idea that companies should formally audit algorithms before they deploy them as
  230. especially in socially consequential settings not every algorithm needs to be audited
  231. but algorithms in socially consequential settings like recruiting need to go through an audit process and that audit
  232. process can be done by an internal team or it could be outside but it's got to be done by a team that
  233. is independent of the team that developed the algorithm and the audit process is going to be important because
  234. it will help ensure that somebody's looked at things beyond say the prediction accuracy of the model they
  235. have looked at things like privacy they've looked at things like bias and fairness and so that will help curb some
  236. of these problems with with algorithmic decisions so I got we could keep talking
  237. about this all day I've sort of come to the end of the list of questions I had for you are there any time points that
  238. you would like to emphasize that I haven't asked about one of the key messages I want to share with people
  239. even though I'm sharing many of the challenges with algorithms in my book is that I'm not an algorithm skeptic I'm
  240. actually a believer in algorithms and I don't want any listener or viewer or reader to leave this becoming wary of
  241. technology I think the message is not be very but the message is engage more actively and more deliberately and
  242. influence be part of the process of influencing how these technologies develop so and the reason I say that is
  243. that studies show that algorithms on average are less biased than human beings so if we say we don't want
  244. algorithms we need to ask what's the alternative and the alternative is biased furthermore my contention is that it is
  245. easier in the long run to fix fix algorithm bias than it is to fix human bias but the challenge with algorithm
  246. bias is just that that bias scales in a way human bias does not scale and what I
  247. mean by that is that a prejudiced judge can impact the lives of maybe 200 300 people but an algorithm used in all the
  248. courtrooms in a country or across the world can influence the lives of hundreds of thousands or even millions
  249. of people similarly a bias recruiter can affect the lives of hundreds of people but a bias recruiting algorithm can
  250. affect the lives of millions of people so it's the scale we worry about and that's why we need to take the issue
  251. seriously but the message mostly is I think we're going to a world where these algorithms will
  252. help us make better decisions and will have growing pains along the way and a few examples I mentioned I think is only
  253. just the beginning we'll hear many more and we should engage actively now to minimize those incidences Karthik thank
  254. you so much for speaking with knowledge at work thank you for having me it's been a great pleasure for more insight
  255. from knowledge at Wharton please visit knowledge Wharton UPenn edu [Music] you

Brief Summary

Karthik Hosonaga discusses algorithmic influence on decision-making and proposes an Algorithmic Bill of Rights for transparency and control.

Key points

  • The Buzz Around AI. Karthik discusses the growing buzz around AI and its implications for decision-making.
  • Chatbots and Human Interaction. Karthik shares striking examples of chatbots and their interactions with humans.
  • Algorithms in Decision-Making. Karthik explains how algorithms influence our daily decisions, from shopping to dating.
  • The Dangers of Autonomous Algorithms. Karthik highlights the challenges of algorithms making decisions without human oversight.
  • Transparency in Algorithms. The necessity for transparency regarding data used in algorithmic decisions is emphasized.
  • The Algorithmic Bill of Rights. A proposed framework to protect consumers from algorithmic decisions made by powerful tech companies.

Tension

45/ 100Some tension

Related episodes