Its mid 2012 and the marketing drums are beating, pushing their message to sceptical enterprises. There is a shiny new toy in town and its name is BIG DATA.
In a style that Donald Draper of the Mad Men TV series would be proud of, sensationalist headlines like “it is almost too late for big data”, and “BD: You need to do something about it now!” entice us to join their club.
The techie side of me lacks the vocabulary to express the sheer joy in having a couple of hundred (or thousand) clustered servers storing, churning and analysing big data. It would be so cool! We could … well yes … um… well just what exactly would we do with our big data tools?
And that is the problem that most CIO’s are facing. Is big data a truly disruptive technology or is it just another fad technology that will be out of vogue in the near future?
The business side of me is sceptical. History shows us that every couple of months, there is a new idea or technology that we are told is going to change the world, but few really do.
Now if you are a Google, eBay, Facebook, LinkedIn, Amazon or research organisation, generating tens of millions or even billions of transactions on a daily basis, then YES, big data is right for you. The storage and analysis of this almost incomprehensible large amount of data is the life blood of your company, where processing speed and algorithms provide a significant competitive advantage.
Institutions like your friendly government tax office, could also have lots of fun cross correlating all your bank accounts, asset purchases, welfare and social security etc. to your tax returns, thereby identifying painfull inconsistencies. Likewise with big brother FBI and CIA type institutions.
But if you are a regular company (e.g. manufacturer, financial institution, teleco) do you really have SO much data that your traditional databases can’t keep up?
Firstly, we need to understand what Big Data is.
Big Data is a term used to describe large and complicated data sets that cannot be easily captured, stored or analysed with traditional data tools. Data sets can also be classified Big Data if they comprise a large volume of data, require a high speed analysis or include a variety of structured and unstructured data. Compared to regular data, big data has different characteristics , different structures and requires different methods of analysis.
But the classification of Big Data is a moving target. I remember Bill Gates stating that 16KB RAM was more than enough for everyone. Ok not one of his finest insights, however these days I have 10 million times more storage than that on my iPod (160GB). Technology gets more powerful every day, and definitions of BIG will change from terabytes (1,000 GB) to petabytes (1,000 TB) to exabytes (1,000 PB) over the next few years.
I think of a very successful Australian company that I have worked with in the past. Global player, leader in their field, employing about 15,000 people across the world. Successful. In fact, compared to the market, very successful. At their most granular level, they process approx. 110,000 financial line entries per month. Throw in employee, payroll, HSEC, key production data and they are close to 400,000 transactions per month. This is easily processed through their traditional data warehouse and consolidated reports are generated from OLAP cubes within seconds.
They are a business to business (B2B) company. Their corporate clients don’t rave about them on social media. Their web-log data is minimal, geospatial data is non-existent and machine/ sensor data more than sufficient to successfully run their plants.
Why do they need Big Data?
One of their divisions has a fleet of close to ten thousand assets that are rented out to their customers for a month or two at a time. As part of an asset management project, we looked at attaching various tracking systems to the assets (e.g. GPS / RFID) and once again the real time data generated from these systems could quite comfortably be captured, stored and analysed in a traditional data base.
So why do they need Big Data? Well, I’m not actually sure they do!
At this stage, big data is a new technology. It is complicated & expensive to setup and requires new analytical skill sets that are in short supply.
Big Data is at the bleeding edge.
If you are one of those few companies that would really benefit from grinding zillions of bytes of data, then you need to seriously consider investing into big data. For the majority of other companies, WAIT! In a couple of years, we will know if big data delivers up to promise.
Also by that stage it will be “house trained” and polished enough to be used by us regular folk.