More data, smarter analytics and our openness to use them for making decisions is transforming the world. Popularized in professional sports but in reality it's been happening in every industry and aspect of our lives.
So said the Pythagoreans. Hard to argue with a quote that's been around for 2500 years. Like the Pythagoreans, I have been fascinated with numbers and data. As a kid in the 70's and 80's, I used to play Strat-O-Matic baseball. If you had the time, energy and insanity you could simulate an entire major league baseball season, all from some cards with numbers and dice. My friend, Rick Alden, who taught me the game could recite the stats of any player in the league from the biggest star to the little used utility infielder on the worst team. He was a hardcore Red Sox fan, but even Strat-O-Matic in the 70's couldn't cure the curse.
It was a beautiful game, combining the probability of any given player/pitcher at bat result, with the probability of the roll of two dice, to simulate a game in your own alternate universe. The goal was to make the best decisions in putting together your team. The cards had the raw data and stats one could use to make those decisions. The best method was usually to use the numbers without thinking about the name behind the stats. Going with the numbers rather than your gut. Not any more easy at 12 than 49.
Over the last 20 years or so the world has been catching up with the Pythagoreans and Strat-O-Matic. The Moneyball movement in sports, has teams moving from making decisions based exclusively on pure gut and subjective criteria, to ones based on the numbers, data and visual evidence. The Boston Red Sox, Oakland A's, New England Patriots and many others have made the transition from using traditional methods of decision making exclusively to one that
involves data driven decisions. The Houston Astros and Pittsburgh Pirates have gone from perennial losers to championship contenders using data and analytics. The combination of improving technology in video, motion sensors, and information storage and acquisition along with an open mindedness to change have proven to be very effective for almost all professional sports teams in both the field and business.
My Strat-O-Matic days are behind me, but in my role as a consultant I have seen many companies start to make the same turn. Spreadsheets and analysis have been around for many years, but the world of data insight, which includes Big Data, Business Intelligence and Predictive Analysis, is pushing into the mainstream. Any modern business today must be engaged with, if not at least seriously investigating, the usage of data at deeper levels. The need to understand the methods and technologies which can help them capitalize from pure data insights.
The initial impulse is to recall famous gut decisions (intuition) that resulted in big wins, big profits and big benefits. The second impulse is to say that the numbers don't always guide us to the right decisions. I wouldn't argue with either as I've probably made millions of "gut" decisions in my lifetime. But if you let yourself go beyond natural impulses, you can see how data driven decisions win out over the gut. Here's a simple example:
If you and I were to make a thousand decisions each, yours are all intuition-based and are all data-driven, who would you wager on for having made the most correct decisions? Certainly, it would be hard to argue that I wouldn't have the stronger average, and probably with significance as well. The gut can be misleading.
Data-driven decisions give us a higher probability of success relative to ones that are more intuition-driven. No absolutes, just higher probability.
As a 12 year old in 1978, if I used my intuition to put together a team that was composed of the formidable Big Red Machine lineup- Bench, Perez, Morgan, Conception, Rose, Foster, Geronimo and Griffey (Senior). An absolutely stellar lineup of players, as anyone's gut would tell them. On the other hand, my friend Rick uses his highly advanced 12 year old statistical skills to select a team with the best chances for wins based on the numbers. We play out a season and I could possibly win. Just as the Cincinnati Reds really did in '75 and '76.
However, if we had played out a thousand seasons with the same teams, the probability suggests he would most likely win the majority of seasons. In this case, my gut decision happened to include three Hall of Famers and one (Rose) who could easily be there, but statistically, I think it would still play out to Rick's advantage. By the way didn't play a thousand seasons. We actually went out and tossed the ball around once in a while.
Almost 38 years after my initial foray into numbers and baseball, we live in an interesting time where data driven decision making has seeped into every aspect of our lives, from the most mundane to ones that leave us speechless. Our browsing, shopping and social media interactions are all datafied for analysis. Now all the numbers being sensed from all of our "things" are being datafied as well. Estimates have the Big Data market going from $ 25 Billion to $ 90 Billion in ten years. Many of us are familiar with this movement in sports due to "Moneyball", but it's happening in each of the industries we work - manufacturing, retail, finance, media, technology, social science, healthcare, education, science and government.
Why is all this happening and in ways where we see exponential changes in just a few years? "Big Data: A Revolution that will Transform How We Live, Work and Think" ,by Victor Mayer-Schonberger and Kenneth Cukier, describes the following three shifts driving this change. First, technology's exponential growth is driving a shift to allow the use of full data sets instead of sampling. Secondly, we are shifting from the desire to always have exactitude to one that provides direction. We give up accuracy at the micro level to gain valuable insight at the macro level. Thirdly, we move from always looking for causation to gaining insight from correlation. We move from having to know why something is happening to knowing that it is happening.
These three shifts are resulting in some amazing uses of data. IBM is putting their expertise in AI, analytics and big data together to make a difference in something that impacts all of us on a daily basis - the weather. In late, 2015 they acquired the digital assets of the Weather Company, or the data which is the product the company provides for about $ 2 Billion. Why?
Imagine logistics organizations getting real time analytics and decision to avoid bad weather, retail organizations using the data to manage inventory, meet weather related demand and mitigate supply chain disruptions. One critical area involves agriculture. By 2050, the world population is expected to increase to 9.2 billion people. That's 34 % more people without any additional agricultural land. Almost all of it will need to come from increased yield and decreased agricultural loss. About 90% of crop losses are due to weather. Using analytics and weather data and combining it with precision agricultural techniques could reduce these losses by 25%. Almost half of ready to harvest agricultural product does not make it to consumers due to transportation issues primarily weather related.
An Israeli company, TaKaDu, developed big data solutions to detect water leaks for many utilities in the country's precious water supply. On average utilities lose 30% of the water in their networks and TaKaDu's software and sensors reduce this to 10% which is a considerable savings of a limited resource. GE has created a platform called Predix that is fed data from millions of its products used in medicine, aviation, transportation and industry. Other companies are linking into this platform as well potentially providing information for savings of $ 20 billion in maintenance and unnecessary downtime.
Science Fiction is just Science before it's time. In the year 2000 less than a quarter of the world's stored data was digital. The other 75% was stored on analog media such as paper, film, etc. Today only 2% of data is on analog media which has also grown rather than just converted. This trend will continue to grow from 130 exabytes of stored digital data in 2005 to 40,000 exabytes in 2020. How do you visualize an Exabyte - 50,000 years of continuous video is one way. I don't know the name for the ------byte to describe what it may by the end of my lifetime. What took decades is now taking years and soon just months.
With the growth of IoT (Internet of Things) data will be everywhere ready for us to gain insight in every aspect of life, hopefully to improve our lives and our world. Feeding a larger population, advancing medicine from idea to usage, reducing energy consumption and better using our finite water supply are all possibilities by the combination of Big Data and Analytics.
-From "The Hitchhiker's Guide to the Galaxy" by Douglas Adams