Why AI Still Needs Human Judgment

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

ABOUT THE EPISODE What happens when companies rush to implement AI and the technology fails to meet expectations?

Santiago Gallino, Associate Professor of Operations, Information and Decisions and Associate Professor of Marketing at the Wharton School, examines the challenges companies face when moving AI from experimentation to real-world implementation.

Using examples from Starbucks and Ford, Gallino explains how inaccurate systems can undermine employee trust, disrupt operations, and damage the customer experience.

He also discusses why human judgment remains essential, how AI hype can encourage companies to scale too quickly, and why every technology investment must deliver measurable value.

ABOUT THE PODCAST This Week in Business features interviews with Wharton faculty about the latest news, fascinating trends, and issues impacting both consumers and the business world. Episodes are recorded at the Wharton School and published twice per week.

----- The Wharton School of the University of Pennsylvania is committed to sharing its intellectual capital through Knowledge at Wharton, the school’s online business analysis journal.

Launched in 1999, Knowledge at Wharton offers free access to:

- Articles, podcasts, and videos highlighting Wharton faculty research and analysis of current business trends - Interviews with book authors from Wharton School Press - In-depth series of curated content like Wharton Executive Education’s Nano Tools for Leaders - A searchable database of more than 10,000 articles covering all aspects of business - A weekly newsletter that delivers Knowledge at Wharton insights directly to your inbox

Visit the Knowledge at Wharton homepage: https://whr.tn/3qUpSBE Subscribe to the Knowledge at Wharton YouTube channel: https://whr.tn/3DEAgAU

Read the full transcriptHide the full transcript
  1. Recently, Ford Motor Company brought back 350 engineers, humans, because the AI process they
  2. had in place was not leading to the standards that Ford wanted in vehicles coming off their
  3. assembly line. Starbucks went away from AI-powered app for its inventory system because too many
  4. mistakes were happening. Santiago Galino is a professor of operations information and decisions
  5. here at the Wharton School, and he joins us to try and explain why that could be developing a
  6. gap in the process in the retail sector in using AI. Santiago, great to talk to you again. How are
  7. you, sir? I'm doing very well. Very good to see you, Dan. You know, it's interesting because so
  8. many people thought AI was going to be the panacea, make everything work perfectly. Companies
  9. would probably be able to cut costs as well, but we're seeing, at least in retail, that there are
  10. some of these instances where maybe it doesn't run as perfect as a lot of people expected.
  11. Yes, yes. I think that that's what we're seeing now, and I think in part is a natural evolution
  12. of these new technologies. I think many companies are moving from experimentation to implementation,
  13. and I think this reminds me of a phrase by Yogi Berra that, in theory, there is no difference
  14. between practice and theory, but in practice there is. And I think that companies are seeing
  15. that firsthand, and I think it's a natural evolution, and it's good to see that companies
  16. are trying and learning. You mentioned in the article that recently ran about Starbucks and the
  17. changes they made. Take us through the problems that they had because I think for a company like
  18. Starbucks, the last thing they want to have happen is something like inventory end up impacting how
  19. their employees work, but also the downstream impact that probably occurs with the customer
  20. that would also occur as well. Yes, so I think it's an interesting example because it shows a
  21. couple of, I guess, red lights or things that companies should think carefully about when
  22. rolling out these technologies. If you're an outsider to the retail industry, you might not be
  23. aware that inventory accuracy is not a trivial problem. You will assume that every retailer
  24. knows perfectly well the number of units of each SKU in the store. That is not the case,
  25. and not because these companies don't try really hard. It's because it's a very difficult
  26. part of the process to keep track of. Companies make a lot of investments and efforts, and so
  27. I will say it's not surprising that Starbucks tried to bring technology to this specific
  28. step. Now, the challenge is that accuracy in this particular case matters a lot, and so I think that
  29. inaccuracies in the count of inventory, inaccuracy on the feed that is informing now the
  30. forecasting algorithms, can escalate very quickly. With the additional point, which I think many of
  31. the articles and conversations with store managers highlight, is that it creates a negative loop
  32. because the moment that the employees learn that the technology cannot be fully trusted because
  33. it's inaccurate and making mistakes, then the employees start to put less effort in implementing
  34. the tool correctly, start to send wrong signals, and that starts to spiral out and making the whole
  35. process fail. So I mentioned the case with Ford Motor Company. To see a company like Ford bring on
  36. AI into their assembly process and not get the results that they wanted to get, mistakes happen,
  37. and then have to bring back human engineers, it really does talk to that component of
  38. how humans are still going to be important even with AI kind of in the mix. Absolutely, and I think
  39. it's kind of an effort from managers to understand what AI is really about, and I think that
  40. because of the excitement and the hype and the inflated expectations, I think there was,
  41. and there still is, around the misunderstanding of what AI can and cannot do, and I think that
  42. this kind of pendulum between going all-in and coming back with the human-AI collaboration
  43. is a healthy exercise, and I think that we should not land hard on companies like Starbucks or
  44. Ford Motor Companies in the sense that I think it is a risk not to experiment, not to try to learn,
  45. at the same time it's like, well, how fast, how quickly can we scale, have we done the
  46. exercise of going through the process in detail before throwing in a technology that, as we all
  47. know, is still very much evolving. So the companies really, to a degree, were so excited by the
  48. technology and the potential of what it could bring, they maybe jumped into the water with
  49. both feet a little too quick, correct? Yes, and I think it is also a conversation
  50. internally at the organization of what is this technology actually bringing to the process
  51. and to the customer ultimately, and in the case of Ford Motor Company, you can imagine that
  52. quality mistakes in the car can be an extremely risky proposition. It's not that you got the
  53. wrong drink at Starbucks, easy to fix, not a big deal. If you start to have errors in the process
  54. when you make a car that cause accidents, now we are at a different league, and I think that's also
  55. an important consideration when you think about what to scale and where AI at this stage can bring
  56. value. So is the process benefiting from it? Is the customer valuing this technology in one way
  57. or another? Well, doesn't that also go to the question of brand? And it's something that I
  58. know that marketing professors and marketers have looked at a lot in the last few years,
  59. how companies are more focused on brand than ever before. And when you have these mistakes,
  60. and as you mentioned with Starbucks, an issue with the stocking and having the right
  61. amount of product in place has an impact on the customer, or I should say on the employee.
  62. It could potentially have a downstream impact on the customer, and the customer gets a negative
  63. feel for it, and they don't maybe go back to Starbucks as often as they may have normally done.
  64. Absolutely. And I think that failures in the process, failure in the product, failure in the experience are always going to have an impact on the brand and the willingness
  65. of the customer to visit again. I think it's also important to recognize that for the most part,
  66. customers are not going to reward you for bringing fancy new technology to the store.
  67. You can think of Trader Joe's as an example. They use bells to send signals inside the store,
  68. one bell, I need one more cashier, two bells, I need the store manager, three bells,
  69. and everyone is happy with that. I think it's an idea that we need technology when it's needed.
  70. Are the bells a good solution for Trader Joe's? Absolutely. They have a footprint of the store
  71. that allows for that. If you're going to ring bells at Costco, it will be a problem because
  72. you're not going to listen to that everywhere in the store like you can do at Trader Joe's.
  73. So it's a combination of the right solution, the right technology, and what the customer is actually
  74. going to value from that technology in the store. So which is more important than in that entire
  75. process? And I assume it is the recognition of the managers, the company, of what the expectation
  76. needs to be of just how much you can gain from having AI in the process. Yes. And I think that
  77. if we think about the steps we were talking before about experimentation, moving into
  78. implementation, I think when you make that transition, the next step is, okay, what is
  79. the return on investment here? Because all these investments, technology more broadly, AI in
  80. particular, they are not for free. And I think that that consideration comes to play, especially
  81. when you roll out these technologies at scale, they start not to give you the expected results.
  82. And I think that that is, I think, the third step in the evolution of these implementations.
  83. Experimentation, implementation, and then looking for the value these technologies are actually
  84. creating. So has the value of return on investment changed then within these companies because of
  85. the dynamic of AI? Is it as important to companies as it has been in the past? So I think when I talk to companies, you can recognize, I guess, two main camps. One that are
  86. kind of almost arguing for a FOMO effect, like we cannot be out of this. We need to go all in.
  87. Eventually we'll figure it out how this is paying itself, which this is my personal take. I think
  88. not the right approach. I think other companies are constantly experimenting, learning what is out there and trying to align those investments and learnings to their strategy
  89. and then making a commitment and a decision that is consistent with the overall strategy of the
  90. company. And this, I mean, can vary a lot, even within an industry like retail, like not every
  91. technology is for every retailer. Santiago, great to talk to you today. Thanks very much for
  92. your time. All the best. Thank you. Santiago Gallino, Professor of Operations, Information
  93. and Decisions here at the Wharton School.

Brief Summary

Ford and Starbucks face AI challenges, highlighting the need for human oversight and careful implementation in retail.

Key points

  • AI's Reality Check at Ford and Starbucks. Ford and Starbucks are reevaluating their AI strategies after facing significant challenges. 'It's a natural evolution.'
  • The Importance of Inventory Accuracy. Santiago discusses the critical nature of inventory accuracy in retail and its challenges. 'It's not a trivial problem.'
  • Balancing Technology and Human Input. The return of human engineers at Ford highlights the need for human oversight in AI processes. 'Humans are still going to be important.'

Tension

45/ 100Some tension

Related episodes