Taming Big Data
Ask about Knowledge at Wharton
Episode Description
π Technology
Read the full transcriptHide the full transcript
- I find that when people ask about big data and try to figure out you know why it's so much more than just a lot of
- data what the answer often is is just taking a look at how quickly we are generating data today and it's not just
- traditional sources we have gotten so much better at putting devices that can capture data whether it be cameras
- whether it be internet whether it be actually what your phone is transmitting at every given time and all of this data
- comes together and one of the things that I think is interesting is that you find that when you look and you say that
- 90 of all of the data has been created within the last two years you start to recognize that this is an accelerating
- amount of data not just a stagnant amount but just accelerating and that's why when people talk about about the
- differences in Big Data you have to recognize it's breaking traditional relational database systems is breaking
- the electronic data warehouse which is our stores of these data it cannot handle it was never designed to handle
- this kind of data I find one of the things that is most fascinating about big data is people really don't
- understand what it really is and I'll tell you to start it is absolutely not a lot of data because there are many
- places in our society where we've accumulated a lot of data but that doesn't really qualify as Big Data
- and there are many definitions but I will say that the key definition that most people go to is what they call
- the three V's they call it volume velocity and variety so volume is what we started with there
- has to be a lot of data and it's got to be breaking the systems that traditionally you would use in your data
- warehouses and your servers that you might traditionally use to store your data then we talk a little bit about
- velocity and that is how quickly is this data coming in in today's society where
- there is data being accumulated everywhere that data from the web from machines is
- being streamed into many many different databases and that is the velocity piece
- so it's coming in quickly but the real catch here is variety and I don't want to get technical in terms of
- the differences between the different kinds of data but at a high level we talk about structured data and
- unstructured data so structured data is what you're traditionally used to seeing
- rows and tables of data that we will always have and it comes our biggest source by the way are traditional
- spreadsheets there are lots of columnar data but the thing that is really causing traditional systems to break is
- this unstructured data this is the data that comes from social media from Twitter feeds from all of the email that
- goes through our systems the rich data of sound and video and images how do you structure those into our traditional
- systems and is when you take all those pieces together and try to understand what big data is it is that combination
- of the three V's that really Define it so when people ask how is Big Data going to be changing my life the first answer
- I want to give is if you thought the internet was a big deal you haven't seen anything yet because
- big data is starting to influence and impact our lives in many many ways and I'm not talking about the NSA stuff I'm
- talking about all the ways every day that you're starting to see big data having an impact on your life if you are
- using a navigation system that's on your smartphone in all likelihood that's all
- big data in the background there it's amassing data now why do you think a company like Google made a choice to
- purchase the company a little company called Waze well they made that choice because they saw the connection here the
- idea of taking every person driving around in your area their speed their Communications what their information
- they're passing around the fact that they saw a police officer the fact that they saw a pothole all this information
- as being amassed and brought together in new brand new technology that allows you
- to process this data in ways that you can't even imagine and think about what it means when we start to take advantage
- of this type of data an example that I think of many times is about the the jet engine
- an airplane in and of itself on a given day is going to make multiple stops it's
- going to be landing and taking off and every piece of that flight is captured by that jet engine and that's called
- machine data and that's data that machines are capturing and they capture it via what is onboard computers but
- also by sensors and all kinds of other information so if you can imagine that a typical flight is going to capture about
- 240 megabytes of data on a typical flight now take that one flight multiplied by a given day so now
- that plane maybe went made five flights let's call that a terabyte in rounding numbers so that Airline on a get that
- one piece of aircraft is capturing a terabyte of information now that terabyte of information multiplied by
- every plane in the air on any given day than a mast imagine how all of a sudden you can start to get insights on how to
- make that plane run more efficiently you start to see that maybe we can change the way the jet engine works so it
- maximizes Energy Efficiency and for an airline Energy Efficiency equals dollars to the bottom line and these are the
- kinds of things that happen in just a few little places but use today it is starting small and you're going to start
- seeing it in every part of your life you know when I look at the future for Big Data
- um I really think it's early days um as much as it's it has a lot of flash today they talk many times
- um it's actually a model that Gartner comes up with which after there's this whole sort of frenzy there's this trial
- of disillusionment that happens and we are actually just coming out of that trial of disillusionment because we had
- all this promise and then you say okay so what what can I do next the technology is moving so quickly the
- operating systems to handle it are moving so quickly the prices of memory are dropping so quickly things that you
- used to need to have disks to do now can be done in memory all of these changes are going to allow big data to start
- delivering even more than what we're seeing today and I I caution people to kind of that say well it's just all hype
- it's not all hype and you haven't even begun to see how you can take this kind of information and the kind of insight
- that you can get I think about a great example is is what's going on in um with NASA and what they've been capturing
- images and information for years but when you start taking that information and and tying it against existing data
- there is incredible information that satellites can tell us today Beyond just seeing a pretty picture or knowing
- what's there you can do time elapse on the entire planet that shows deforestation that shows water flows
- that gives insight into our lives that never ever was imagined and that's that's the power of Big Data
- I I think that um one of the great things about big data is it is finding its way into so many places that people
- never imagined that you could be leveraging big data I think that um one of the places is certainly
- sustainability and business today and corporations today have gotten to the place where a lot of the low-hanging
- fruit of changing their light bulbs and doing some changes to their supply chains and
- doing the things that they know that they could bring some immediate dollars in they're exhausting those resources
- the next level of sustainability for for business is really going to be around leveraging big data to change operations
- to be able to leverage big data to do things that allow them to save energy allows them to reduce their carbon
- footprint that allows them to be more efficient about how they use water and all of this begins to happen as they
- start accumulating the data from machines accumulating the data from their operations melding that with
- consumer information and that is a huge piece of how business is going to be leveraging this against sustainability
- but I think another part of this is how many areas of traditional sustainability
- will have incredible impact by Big Data let's talk about how we deal with animals and preserving animal habitats
- let's talk about how we deal with some of our most pressing issues of humankind how we're going to feed a population
- that's going to reach 9 billion by 2050. these kinds of resource issues will not
- be solved by traditional means and what we're seeing is companies that are able to start taking weather data and start
- incorporating what we know about climate change today and start leveraging to make farming more efficient to change
- the way that Farm Insurance Works to start utilizing if you can imagine they're going to be machine data coming
- from seeds the seeds are going to have data within themselves that will be able to transmit the efficiency of those
- seeds and how they're doing there is going to be massive massive amounts of data today
- farming today which was arguably one of the least technical businesses around will become one of the
- most technical businesses companies like Monsanto are making huge investments in
- technology deer is making investments in technology all of these companies are because we all recognize that in the
- future there is going to have to be changed in how we go about raising plants raising animals raising enough
- food to feed this ever-growing population foreign
Brief Summary
Big data's rapid growth impacts society, influencing aviation, sustainability, and agriculture through the three V's: volume, velocity, and variety.
Key points
- The Three V's of Big Data. Volume, velocity, and variety define what Big Data truly is.
- Big Data's Impact on Daily Life. Big Data is influencing our lives in ways we can't even imagine.
- The Future of Farming. Farming is set to become a highly technical industry due to Big Data.
Episode highlights
- The Three V's of Big DataVolume, velocity, and variety define what Big Data truly is.βThe key definition is what they call the three V's: volume, velocity, and variety.β1:49Mar 26, 2014
- Big Data's Impact on Daily LifeBig Data is influencing our lives in ways we can't even imagine.βBig Data is starting to influence and impact our lives in many ways.β3:59Mar 26, 2014
- The Future of FarmingFarming is set to become a highly technical industry due to Big Data.βFarming today will become one of the most technical businesses around.β11:30Mar 26, 2014
Tension
45/ 100Some tension





