Taming Big Data

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

πŸ“‚ Technology

Read the full transcriptHide the full transcript
  1. I find that when people ask about big data and try to figure out you know why it's so much more than just a lot of
  2. data what the answer often is is just taking a look at how quickly we are generating data today and it's not just
  3. traditional sources we have gotten so much better at putting devices that can capture data whether it be cameras
  4. whether it be internet whether it be actually what your phone is transmitting at every given time and all of this data
  5. comes together and one of the things that I think is interesting is that you find that when you look and you say that
  6. 90 of all of the data has been created within the last two years you start to recognize that this is an accelerating
  7. amount of data not just a stagnant amount but just accelerating and that's why when people talk about about the
  8. differences in Big Data you have to recognize it's breaking traditional relational database systems is breaking
  9. the electronic data warehouse which is our stores of these data it cannot handle it was never designed to handle
  10. this kind of data I find one of the things that is most fascinating about big data is people really don't
  11. understand what it really is and I'll tell you to start it is absolutely not a lot of data because there are many
  12. places in our society where we've accumulated a lot of data but that doesn't really qualify as Big Data
  13. and there are many definitions but I will say that the key definition that most people go to is what they call
  14. the three V's they call it volume velocity and variety so volume is what we started with there
  15. has to be a lot of data and it's got to be breaking the systems that traditionally you would use in your data
  16. warehouses and your servers that you might traditionally use to store your data then we talk a little bit about
  17. velocity and that is how quickly is this data coming in in today's society where
  18. there is data being accumulated everywhere that data from the web from machines is
  19. being streamed into many many different databases and that is the velocity piece
  20. so it's coming in quickly but the real catch here is variety and I don't want to get technical in terms of
  21. the differences between the different kinds of data but at a high level we talk about structured data and
  22. unstructured data so structured data is what you're traditionally used to seeing
  23. rows and tables of data that we will always have and it comes our biggest source by the way are traditional
  24. spreadsheets there are lots of columnar data but the thing that is really causing traditional systems to break is
  25. this unstructured data this is the data that comes from social media from Twitter feeds from all of the email that
  26. goes through our systems the rich data of sound and video and images how do you structure those into our traditional
  27. systems and is when you take all those pieces together and try to understand what big data is it is that combination
  28. of the three V's that really Define it so when people ask how is Big Data going to be changing my life the first answer
  29. I want to give is if you thought the internet was a big deal you haven't seen anything yet because
  30. big data is starting to influence and impact our lives in many many ways and I'm not talking about the NSA stuff I'm
  31. talking about all the ways every day that you're starting to see big data having an impact on your life if you are
  32. using a navigation system that's on your smartphone in all likelihood that's all
  33. big data in the background there it's amassing data now why do you think a company like Google made a choice to
  34. purchase the company a little company called Waze well they made that choice because they saw the connection here the
  35. idea of taking every person driving around in your area their speed their Communications what their information
  36. they're passing around the fact that they saw a police officer the fact that they saw a pothole all this information
  37. as being amassed and brought together in new brand new technology that allows you
  38. to process this data in ways that you can't even imagine and think about what it means when we start to take advantage
  39. of this type of data an example that I think of many times is about the the jet engine
  40. an airplane in and of itself on a given day is going to make multiple stops it's
  41. going to be landing and taking off and every piece of that flight is captured by that jet engine and that's called
  42. machine data and that's data that machines are capturing and they capture it via what is onboard computers but
  43. also by sensors and all kinds of other information so if you can imagine that a typical flight is going to capture about
  44. 240 megabytes of data on a typical flight now take that one flight multiplied by a given day so now
  45. that plane maybe went made five flights let's call that a terabyte in rounding numbers so that Airline on a get that
  46. one piece of aircraft is capturing a terabyte of information now that terabyte of information multiplied by
  47. every plane in the air on any given day than a mast imagine how all of a sudden you can start to get insights on how to
  48. make that plane run more efficiently you start to see that maybe we can change the way the jet engine works so it
  49. maximizes Energy Efficiency and for an airline Energy Efficiency equals dollars to the bottom line and these are the
  50. kinds of things that happen in just a few little places but use today it is starting small and you're going to start
  51. seeing it in every part of your life you know when I look at the future for Big Data
  52. um I really think it's early days um as much as it's it has a lot of flash today they talk many times
  53. um it's actually a model that Gartner comes up with which after there's this whole sort of frenzy there's this trial
  54. of disillusionment that happens and we are actually just coming out of that trial of disillusionment because we had
  55. all this promise and then you say okay so what what can I do next the technology is moving so quickly the
  56. operating systems to handle it are moving so quickly the prices of memory are dropping so quickly things that you
  57. used to need to have disks to do now can be done in memory all of these changes are going to allow big data to start
  58. delivering even more than what we're seeing today and I I caution people to kind of that say well it's just all hype
  59. it's not all hype and you haven't even begun to see how you can take this kind of information and the kind of insight
  60. that you can get I think about a great example is is what's going on in um with NASA and what they've been capturing
  61. images and information for years but when you start taking that information and and tying it against existing data
  62. there is incredible information that satellites can tell us today Beyond just seeing a pretty picture or knowing
  63. what's there you can do time elapse on the entire planet that shows deforestation that shows water flows
  64. that gives insight into our lives that never ever was imagined and that's that's the power of Big Data
  65. I I think that um one of the great things about big data is it is finding its way into so many places that people
  66. never imagined that you could be leveraging big data I think that um one of the places is certainly
  67. sustainability and business today and corporations today have gotten to the place where a lot of the low-hanging
  68. fruit of changing their light bulbs and doing some changes to their supply chains and
  69. doing the things that they know that they could bring some immediate dollars in they're exhausting those resources
  70. the next level of sustainability for for business is really going to be around leveraging big data to change operations
  71. to be able to leverage big data to do things that allow them to save energy allows them to reduce their carbon
  72. footprint that allows them to be more efficient about how they use water and all of this begins to happen as they
  73. start accumulating the data from machines accumulating the data from their operations melding that with
  74. consumer information and that is a huge piece of how business is going to be leveraging this against sustainability
  75. but I think another part of this is how many areas of traditional sustainability
  76. will have incredible impact by Big Data let's talk about how we deal with animals and preserving animal habitats
  77. let's talk about how we deal with some of our most pressing issues of humankind how we're going to feed a population
  78. that's going to reach 9 billion by 2050. these kinds of resource issues will not
  79. be solved by traditional means and what we're seeing is companies that are able to start taking weather data and start
  80. incorporating what we know about climate change today and start leveraging to make farming more efficient to change
  81. the way that Farm Insurance Works to start utilizing if you can imagine they're going to be machine data coming
  82. from seeds the seeds are going to have data within themselves that will be able to transmit the efficiency of those
  83. seeds and how they're doing there is going to be massive massive amounts of data today
  84. farming today which was arguably one of the least technical businesses around will become one of the
  85. most technical businesses companies like Monsanto are making huge investments in
  86. technology deer is making investments in technology all of these companies are because we all recognize that in the
  87. future there is going to have to be changed in how we go about raising plants raising animals raising enough
  88. food to feed this ever-growing population foreign

Brief Summary

Big data's rapid growth impacts society, influencing aviation, sustainability, and agriculture through the three V's: volume, velocity, and variety.

Key points

  • The Three V's of Big Data. Volume, velocity, and variety define what Big Data truly is.
  • Big Data's Impact on Daily Life. Big Data is influencing our lives in ways we can't even imagine.
  • The Future of Farming. Farming is set to become a highly technical industry due to Big Data.

Tension

45/ 100Some tension

Related episodes