Who Made That Decision: You or an Algorithm?
Ask about Knowledge at Wharton
Episode Description
When we buy something on Amazon or watch something on Netflix, we think it’s our own choice. Well, it turns out that algorithms influence one-third of our decisions on Amazon and more than 80% on Netflix. What’s more, algorithms have their own biases. They can even go rogue. In his recent book titled, A Human's Guide to Machine Intelligence: How Algorithms Are Shaping Our Lives and How We Can Stay in Control, Kartik Hosanagar, a professor of operations, information and decisions at Wharton, focuses on these issues and more. He discusses how algorithmic decisions can go wrong and how we can ensure we can control the way technology impacts decisions that are made for us or about us.
In a conversation with Knowledge@Wharton, Hosanagar notes that a solution to this complex problem is that we must "engage more actively and more deliberately and be part of the process of influencing how these technologies develop." 📂 Technology 🏷️ Research
Read the full transcriptHide the full transcript
- our guest today is Karthik hosonaga a professor of technology digital business and marketing at Wharton and we are
- speaking with him about his recent book titled a human's guide to machine intelligence how algorithms are shaping
- our lives and how we can stay in control Karthik welcome to knowledge at Wharton
- thank you so much for speaking with us today well cool thanks for having me it's
- always a pleasure to talk to you and the knowledge is important GRU thanks so these days there's a growing buzz
- about artificial intelligence and machine learning and quite a few books I have have come out recently on these
- topics in all the conversations that are going on what do you think are some of the points that are being overlooked are
- not sufficiently emphasized and how does your book seek to fill that gap yeah clearly there's a lot of buzz around AI
- and machine learning which is a subfield of AI I think the conversation tends to
- you know either glorify the technology or in many instances lately create a lot of fear-mongering around it and you know
- I don't think the conversation has focused on you know what's the solution how are we going to work with AI and
- especially in the context of making decisions and so my book is focused on making decisions through intelligent
- algorithms and certainly we have various kinds of AI but one of the core questions when it comes to AI is are we
- going to use AI to make decisions if so are we going to use it in a decision support way are we going to have the AI
- make decisions autonomously if so what can go wrong what can go well and how do we manage this because we know AI has a
- lot of potential but I think there will be some growing pains on our way there and so those growing pains is what I
- focus on how can algorithmic decisions go wrong and how do we make sure that we have control over the narrative of how
- technology impacts the decisions that are made for us or about us I'd love to come back to the part of our
- decision-making an algorithmic algorithmic decision-making but I really love the way you began the book with
- some very striking examples about chatbots and and how they interact with humans I wonder if you could use that
- illustration to talk a little bit about how human beings interact with algorithms and what some of the
- implications are I began the book with a description of Microsoft's experience with a chat part called Xiao Bing in
- China it's called Xiao Bing and elsewhere in the world it's called Xiao Weis this was a chat board created in
- the avatar of a teenage girl and it's meant to engaged in engage in fun playful conversations with young adults
- and teenagers and this chat pot has about 40 million followers in China and the reports say that roughly a quarter
- of those followers have said I love you to Xiao Weis so that's the kind of affection and following Xiao Weis has so
- inspired by the success of Xiao Weis in China Microsoft decided to test a similar chat bot in the US and they
- created a chat bot in English which would engage again in fun playful conversations and targeted once again at
- young adults and teenagers they launched it on twitter under the name Tay ta y and this chat BOTS experience was
- actually very different and it's it was very short-lived experience as well because within an hour of launching the
- chat pod turned sexist racist fascist it tweeted very offensively and said things
- like Hitler was right and Microsoft shut it down within 24 hours and later that year mi t--'s Technology Review rated
- Microsoft Day as the worst technology of the year and that incident you know opened up this question for me which was
- how can two similar chart parts or pieces of AI built by the same company produce such
- different results and what does that say about you know our decision to to use algorithms for a lot of our decisions in
- our personal and professional lives and that's what helped start this exploration into use of AI the extent to
- which AI can be predictable biased as in the case of Microsoft a and then of course how do we what does that mean in
- terms of using these systems to make significant decisions for us so why did T experiences differ so dramatically and
- is anything that can be done about that one of the insights that I got as I was writing this book as I was trying to
- explain the differences in behavior of these two chatbots was actually from human psychology
- so psychologists describe human behavior in terms of our nature and our nurture and so our nature is our genetic code
- and nurture is our environment and so psychologists will attribute problematic issues like alcoholism partly to nature
- and partly to nurture and when I was looking at algorithms I realized algorithms have nature and nurture as
- well so the nature for algorithms is not genetic code but the code that the engineer actually codes in or writes and
- that's the logic of the algorithm the nurture for the algorithm is the data from which algorithms learn and so
- increasingly as we are moving towards machine learning we're moving from a world where engineers used to program
- the end-to-end logic of an algorithm would actually specify what happens when any situation happens if this happens
- you respond this way if that happens you respond a different way and so it used to be all nature because the programmer
- gave all the very minut specifications of how the algorithm will work but as we move towards machine learning we're kind
- of telling algorithms here's data learn from it and so nature starts to become less important and nurture starts to
- dominate so if you look at what happened between tea and showers the difference is in terms of their training
- data in some ways and in particular in the case of showers showers was created to mimic how people converse on Tay and
- so it was picking up how people are talking to it and it would reflect that and so there were many intentional
- efforts to trip day and so there was nurture there as well and part of it was nature as well meaning that the court
- could have specified certain rules the court could have specified rules like do nots say the following kinds of things
- or do not get into discussions of these topics and so on so it's a bit of both and I think that's what in general rogue
- algorithmic behavior comes down to that come back a little later to the whole question of what happens go rogue yeah
- but I wanted to chat a little bit about the way in which this algorithmic decision-making itself has changed and
- it wasn't there was a time when decision-making algorithm decision-making seemed to be almost like
- Amazon will tell you what books you should read or Netflix will recommend which movies you should watch but
- because of AI algorithmic algorithmic decision-making has become a lot more complex and I was wondering if you could
- offer some examples and what are some of the implications on the choices that we
- make or don't make as a result of this yeah algorithms pervade our lives and sometimes we see it like Amazon's
- recommendations and sometimes we don't realize it but they have a huge impact on decisions we make on Amazon for
- example over 1/3 of the choices that we make are influenced by algorithmic recommendations people who bought this
- also bought those people who viewed this eventually bought that and so on on Netflix over 80% of the viewing activity
- is driven by algorithmic recommendations they also drive decisions such as who we
- date and Mary when you look at applications or apps like tinder which actually ABB's we're algorithms create most of
- the matches over there they're also at the workplace you know for example you make you apply for our
- alone mortgage approval decisions are made by algorithms increasingly you know if you apply for a job resumes screaming
- algorithms are deciding which ones to invite for an interview and they're making life-and-death decisions as well
- there for example being used in the criminal justice system in courtrooms in the u.s. there are algorithms that
- predict the likelihood that the defendant will reoffend so that judges can make sentencing decisions in
- medicine we're moving towards personalized medicine so that two people with the same symptoms might not get the
- same treatment it might be customized based on their DNA profile algorithms again will guide the doctors on those
- decisions and also we're moving as AI has progressed and advanced we're moving to a point where the algorithms don't
- merely offer decision support they can function autonomously as well and driverless cars are a great example of
- that where we're trying to pretty much say you can automate the whole process and and have algorithms function without
- a human making the final sort of decision so as more algorithms influence or make more and more decisions
- is there anything like free will in the world anymore well so free will is an interesting concept and for the most
- part I used to think of free will in a very philosophical sense right and philosophers have argued we don't even
- have free will but that was in a very different very as I said philosophical interpretation but I think we have a
- literal interpretation of free will now in the context of algorithms which is are you making the final choice and I
- just said you know a third of your choices are driven on Amazon by recommendations 80% of viewing
- activities on Netflix are driven by algorithmic recommendations at YouTube seventy percent of the time people spend
- on YouTube is driven by algorithmic recommendations so it doesn't feel like algorithms are merely recommending to us
- what we want and we make decisions I mean think about a search on Google you take the most
- Oh Tarek of search terms like vintage toy model trains or something like that you will still find hundreds of
- thousands perhaps even millions of results for that search term we might see less than 0.01% of the search
- results because rarely do we even cross page one the algorithm has decided which
- pages we look at so yes they're making a lot of choices for us so do we have free
- well at some level yes but mostly I'm going to say we don't have the level of independent decision-making that we
- think we do because most of us think we get these recommendations we not politely and then we do what we want but
- indeed the algorithms are not jiggles in very interesting ways and and I don't think we have the level of independent
- decision-making or choice that we think we do mostly that's a good thing in some
- ways because they're saving us time so we can focus on leisure instead of wasting our time sifting through lots of
- alternatives but sometimes we become very passive about how we use algorithms and becoming that passive about
- algorithm use can have consequences so let's talk a little bit about those consequences okay especially unintended
- consequences yes there's a fascinating part on your book about that and how do design choices lead to unintended
- consequences I was wondering if you could speak about that yes so when I mention unintended consequences in the
- book I'm referred referring to situations where you know you're trying to optimize some aspect of a decision
- and perhaps you manage to improve that really well but then something else goes wrong
- so an example might be that when Facebook was manually curating its trending stories through human editors
- as an editor you will appreciate that it's real work right and they had people doing that but then Facebook was accused
- of having a left-leaning bias that these editors were choosing left-leaning stories and curating those more often so
- they said you know algorithms can't be accused of a political bias so they used an algorithm to curate this they tested
- it for political bias it did not have any political bias but there's something else it did which they
- hadn't explicitly tested for which is as we know it curated fake new stories and
- and circulated them so that's an example of unintended consequences and an algorithm design can drive that in many
- ways you know I've done a lot of work on recommendation systems and how they influence the kinds of products we
- consume the kinds of media we consume and I've specifically studied two kinds of recommendation algorithms one kind of
- algorithm is based or it's like the Amazon people who bought this also bought this so it's based on social
- curation what are the what are others consuming the other kind of algorithm attempts to understand at a deep level
- what is it that I'm recommending and the algorithm is recommending and tries to find items that are very similar to the
- users interest an example of that would be Pandora so Pandora's music recommendations are not people who like
- this song also like these other songs Pandora actually has very detailed information over 150 attributes for each
- song musical attributes like how rhythmic is the song how much instrumentation is there in the music
- and every time you say you like a music a song or you don't like a song they look at the musical qualities of the
- song and then they adjust their recommendations based on other songs which have attributes similar to what
- you have now I looked at both these designs and I looked at which design is more helpful in helping us find let's
- say indie songs or very novel and niche books or movies and at the time we did the study and this was some time back
- the conventional wisdom was that all of these algorithms help in pushing the long tail meaning these nice novel items
- or indie songs that nobody's heard of and what I found was that these designs were very different the algorithm that
- looks at the look said you know what others are consuming people who bought this also bought this it has a
- popularity bias because it's trying to recommend stuff that others are consuming and so it
- tends to lean towards popular items and so it cannot truly recommend those hidden gems but an algorithm like the
- Pandora algorithm is it doesn't have popularity as a basis for recommendations so it tends to do better
- and that's why what we've seen now is companies like Spotify Netflix and many others have changed the design of their
- algorithms they've combined the two approaches they've combined the the social appeal of a system that looks at
- what others are consuming and the ability of the other design to surface hidden gems let's go now to the point
- you were up earlier about algorithms going rogue yes why does that happen and what can be done about it yeah if we
- look at let me point a couple example examples of algorithms doing rolled and then we'll talk about why this happens
- so I mentioned algorithms are used in courtrooms in the u.s. in the criminal justice system in 2016 there was a
- report or study done by ProPublica which is a nonprofit organization they looked
- at algorithms used in courtrooms and they found that these algorithms have a race bias specifically they found that
- these algorithms were twice as likely to falsely predict future criminality in a
- black defendant than a white defendant and this was race bias in that algorithm late last year Reuters carried a story
- about Amazon trying to use algorithms to screen job applications and the you know
- again Amazon gets a million plus job applications they hire hundreds of thousands of people you know over the
- last few years they've done that it's hard to do that manually and so you need algorithms to help automate some of this
- but they found that the algorithms tended to have a gender bias they tended to reject female applicants more often
- even when the qualifications were similar now Amazon ran the test and concluded and realized that there are
- savvy companies so they decided not to roll this out there's probably many other companies that
- using algorithms to screen resumes and they might be prone to race bias gender bias and so on so I've mentioned two
- examples now race bias gender bias I talked about fake news on on Facebook so there's many examples in terms of why
- they go rogue there's a couple reasons I can share one is as we have gone from those old traditional algorithms where
- the programmer wrote up the algorithm end to end and we've moved towards machine learning we have created
- algorithms that are usually more resilient because these algorithms you know can perform much better but they're
- prone to biases that exist in the data so if the data have biases so for example if you tell a resume screening
- algorithm you know here's data on all those people who applied to our job and here's the people we actually hired and
- here are the people who we promoted now figure out who to invite for job interviews based on this data the
- algorithm will observe that in the past you are rejecting more female applications or you were not promoting
- women at the workplace and will tend to pick up that behavior so it tends to pick that up I think the other piece is
- that engineers in general tend to focus narrowly on one or two metrics you know with a resume screening application you
- will tend to measure the accuracy of your model and if it's highly accurate you'll roll it out but you don't
- necessarily look at fairness and bias or in Facebook's example look at fake news and other
- possibilities as well what are the challenges involved in autonomous algorithms making decisions on our
- behalf well I think when you have autonomous algorithms making decision on our behalf I think one of the big
- challenges is there is usually no human in the loop so we lose control and that is one one challenge and many studies
- show that when we have limited control we are less likely to trust algorithms and so that
- is one one aspect that is challenging the other piece about having autonomous algorithms is that again if there's a
- human in the loop there's a greater chance that the user can detect certain problems and you know the likelihood
- that problems get detected is therefore greater I'm really glad you brought up the point about trusting algorithms
- because you tell this really fascinating story in the book about a patient who gets diagnosed with definitely fever I
- wonder if you could share that story with our audience and and spell out some of the implications for trust in
- algorithms and and what the implications are yeah the story that I share is that
- of a patient walking into a doctor's office and the patient feels fine and healthy and the patient and doctor are
- joking around and doctor eventually picks up the pathology report and he suddenly looks very serious and he's
- says or informs the patient that I'm sorry to let you know that you have to happen early fever and the patient
- hasn't heard of that Bernoulli fever and he asks what exactly is it and so the doctor says it's a very rare disease and
- it's known to be fatal and so I suggest that you have this tablet and it will you know reduce the chance that you will
- have any problems and you know he says here you take this you know three times a day and and and then you go about your
- life and I asked my readers if you know that story is something that if they were the patient they would feel
- comfortable in that situation here's a disease you don't nothing about and here's a solution you know nothing about
- and the doctor has given you the choice and told you to go ahead but not given you very many details and with that I
- pose the question if an algorithm way to again make this wreck that you have this rare disease and we
- want you to take this medication without any information would you and of course Mukul you've read the book
- so you know Tappin le fever is not a real disease I'm a fan of Sherlock Holmes I used to read it a lot as a kid
- and it's a disease in a one of the Sherlock Holmes stories and you know that inspired me to consider this
- because even in the original Sherlock Holmes story it turns out that the person who has definitely fever doesn't
- actually have it but setting that aside it kind of brings up this question of transparency you know are we willing to
- trust decisions when we don't have information about why a certain decision was made the way it was and what I
- highlight is sometimes we're seeking more transparency from algorithms than humans but in practice lots of companies
- are imposing algorithmic decisions on us without any information about why these
- decisions are being made and are we fine with that and a lot of research shows that we're not fine with that research
- for example that one PhD student then a PhD student at Stanford looked at an algorithm that would compute grades for
- students and how did they do when they just got their score versus they got their score with an explanation and as
- you expect when they have an explanation they trust it more then why is it that in in the real world there's a lot of
- algorithms making decisions for us or about us and we have no transparency about those decisions and so I advocate
- that we need a certain level of transparency with regard to for example what kinds of data were used to make the
- decision so for example if we applied for a loan and the loan was rejected we would like to know why that was the case
- if you applied for a job and it was rejected it would be helpful to know that the algorithm not only evaluated
- what you submitted as part of your job application would also looked at your social media posts and so transparency
- regarding what data was considered what were the key factors that drove a decision
- is important but what's the end of the book you recommend an algorithmic Bill of Rights what exactly is that and why
- is it necessary so the algorithmic Bill of Rights is a concept that I borrowed from the Bill of Rights in the US
- Constitution and the history of the Bill of Rights is that when the founding fathers were setting up or drafting the
- Constitution some people were worried that we're creating a very powerful government here in the US and the Bill
- of Rights was created as a way to protect citizens now today we are in a situation where there's a lot of talk
- about powerful tech companies you know everywhere you see there's new stories big tech companies and what are we going
- to do about them and so there's a sense that again consumers need certain protections and so the Bill of Rights is
- targeted at that but before I talk about the Bill of Rights one aspect related to
- the Bill of Rights I do want to address is that a lot of consumers feel that they're helpless against big tech and
- against algorithms deployed by big tech and personally I feel that consumers do have some power and that power is in
- terms of our knowledge our votes and our dollars you know knowledge is about you
- know we shouldn't be passive users of technology we should be active and deliberate about it we should know how
- it's changing decisions we are making or others are making about us and at some level it seems like you know knowledge
- is fine but what can I do with that knowledge but I mean look at how Facebook is changing their product
- design today and that changes you know support for encryption and so on is because of push from the users and it
- shows that when users complain changes to happen votes there are another aspect of that and votes are all about you know
- us being aware of which elected representatives understand the nuances of algorithms and the challenges and how
- to regulate them and it's about voting for them right and the question is how are these regulators going to protect us
- that's where the bill of rights comes in and the Bill of Rights I propose has a few key pillars one pillar is
- transparency transparency with regard to the data used to make decisions and with
- regard to the underlying decision itself what were the most important factors that led to a certain decision today
- Europe's GDP are actually has certain provisions like write to explanations and information on the data that
- companies are using and so I think some of that transparency is needed and companies should provide that another
- pillar in my bill of rights is the idea of some user control that we cannot be in an environment where we have no
- control over the technology we should for example be able to with a simple instruction tell Alexa you're not
- listening to any conversation in the house until I'd instruct you that it's allowed right there's no such provision
- we're told that the system is not listening but then we're also hearing from others that there are instances
- where it listens even when you're not actually saying Alexa and giving an instruction and that control is very
- important if you look at Facebook the false news issue two years back there was no way for users to alert Facebook's
- algorithm and say this post in my newsfeed is false news even though users were seeing false news in their newsfeed
- even today with just two clicks you can let Facebook know that a certain news post in your feed is either offensive or
- it has false news and that feedback is so important for the algorithm to now correct itself and previously users had
- no way even though you're observing problems no way of informing the algorithm or helping the algorithm
- course correct so some level of user control is also another pillar I proposed and lastly I
- have been advocating this idea that companies should formally audit algorithms before they deploy them as
- especially in socially consequential settings not every algorithm needs to be audited
- but algorithms in socially consequential settings like recruiting need to go through an audit process and that audit
- process can be done by an internal team or it could be outside but it's got to be done by a team that
- is independent of the team that developed the algorithm and the audit process is going to be important because
- it will help ensure that somebody's looked at things beyond say the prediction accuracy of the model they
- have looked at things like privacy they've looked at things like bias and fairness and so that will help curb some
- of these problems with with algorithmic decisions so I got we could keep talking
- about this all day I've sort of come to the end of the list of questions I had for you are there any time points that
- you would like to emphasize that I haven't asked about one of the key messages I want to share with people
- even though I'm sharing many of the challenges with algorithms in my book is that I'm not an algorithm skeptic I'm
- actually a believer in algorithms and I don't want any listener or viewer or reader to leave this becoming wary of
- technology I think the message is not be very but the message is engage more actively and more deliberately and
- influence be part of the process of influencing how these technologies develop so and the reason I say that is
- that studies show that algorithms on average are less biased than human beings so if we say we don't want
- algorithms we need to ask what's the alternative and the alternative is biased furthermore my contention is that it is
- easier in the long run to fix fix algorithm bias than it is to fix human bias but the challenge with algorithm
- bias is just that that bias scales in a way human bias does not scale and what I
- mean by that is that a prejudiced judge can impact the lives of maybe 200 300 people but an algorithm used in all the
- courtrooms in a country or across the world can influence the lives of hundreds of thousands or even millions
- of people similarly a bias recruiter can affect the lives of hundreds of people but a bias recruiting algorithm can
- affect the lives of millions of people so it's the scale we worry about and that's why we need to take the issue
- seriously but the message mostly is I think we're going to a world where these algorithms will
- help us make better decisions and will have growing pains along the way and a few examples I mentioned I think is only
- just the beginning we'll hear many more and we should engage actively now to minimize those incidences Karthik thank
- you so much for speaking with knowledge at work thank you for having me it's been a great pleasure for more insight
- from knowledge at Wharton please visit knowledge Wharton UPenn edu [Music] you
Brief Summary
Karthik Hosonaga discusses algorithmic influence on decision-making and proposes an Algorithmic Bill of Rights for transparency and control.
Key points
- The Buzz Around AI. Karthik discusses the growing buzz around AI and its implications for decision-making.
- Chatbots and Human Interaction. Karthik shares striking examples of chatbots and their interactions with humans.
- Algorithms in Decision-Making. Karthik explains how algorithms influence our daily decisions, from shopping to dating.
- The Dangers of Autonomous Algorithms. Karthik highlights the challenges of algorithms making decisions without human oversight.
- Transparency in Algorithms. The necessity for transparency regarding data used in algorithmic decisions is emphasized.
- The Algorithmic Bill of Rights. A proposed framework to protect consumers from algorithmic decisions made by powerful tech companies.
Episode highlights
- The Buzz Around AIKarthik discusses the growing buzz around AI and its implications for decision-making.“There's a lot of buzz around AI and machine learning.”0:52Mar 25, 2019
- Chatbots and Human InteractionKarthik shares striking examples of chatbots and their interactions with humans.“Xiao Weis has about 40 million followers in China.”3:07Mar 25, 2019
- Algorithms in Decision-MakingKarthik explains how algorithms influence our daily decisions, from shopping to dating.“Over 1/3 of choices on Amazon are influenced by algorithms.”8:31Mar 25, 2019
- The Dangers of Autonomous AlgorithmsKarthik highlights the challenges of algorithms making decisions without human oversight.“When we have limited control, we are less likely to trust algorithms.”19:51Mar 25, 2019
- Transparency in AlgorithmsThe necessity for transparency regarding data used in algorithmic decisions is emphasized.“We need transparency about the data used to make decisions.”23:53Mar 25, 2019
- The Algorithmic Bill of RightsA proposed framework to protect consumers from algorithmic decisions made by powerful tech companies.“Consumers need certain protections against powerful tech companies.”25:24Mar 25, 2019
- User Control Over TechnologyAdvocating for user control over technology to ensure privacy and fairness in algorithms.“We cannot be in an environment where we have no control over technology.”27:21Mar 25, 2019





