The Top Problems with Big Data

By: rkspark

Big data has its issues and isn’t always perfect. This blog aims to highlight some of the shortfalls

Finding Signals in the Noise

When you have a huge amount of data, it can be difficult to get any insights out of it.  Maksim Tsvetovat, a data scientist, states that in order to use big data there will need to be a discernible signal in noise that you will be able to detect.  The problem is that sometimes there isn’t one and you could come back from the data and realize that it was not measured correctly or the wrong variables were being measured.  He has stated that in its raw form, big data is very much like a hairball and a more scientific approach to data is required.  You will need to approach the data carefully and if you fail with your initial hypothesis, you will need to come out with a few others.


The Data Silos

The kryptonite of big data is the data silo.  These data silos will store all of the data that you have in separate units which are not linked to each other and provide you with no insight into the data. Data silos are the reason why, when trying to create a monthly sales report, you have to crunch the numbers.

They are also the reason why C-level decisions take so long and why the sales and marketing teams in a company generally do not get along.  They are also the reason why your customers could be looking elsewhere as they do not feel their needs are being met correctly.  It is important that you look at eradicating data silos by integrating your data.

Inaccurate Data

Data silos are not only ineffective on operational levels, they are also the ground zero for the biggest data problem you can have which is inaccurate data.  Experian Data Quality released a report which shows that 75% of businesses believe that the customer contact information they have is incorrect.  If you have a database of customer information which is all incorrect, there is no reason to have the data at all.  The best way that you can remedy inaccurate data is to eliminate the data silos and integrate your data. This blog from Harnham shows how big data is trying to predict even the unpredictable.


Technology Moving Too Fast

Data silos are often a major problem for large companies.  The reasons for this will include keeping all of their databases on their premises.  New technology decisions are also generally slower in large companies which add to this problem.

An example of this comes from the CapGemini report which states that leaders in utilities and telcos are seeing a high level of disruption from new competitors in the market who have come for different sectors.  In each of these industries, over 35% of respondents mentioned this issue which was higher than the overall average.  The problem is that traditional players in the market as slower to move on new technology and will find themselves facing major competition from smaller and more innovative companies.

The same report found that big data is fast data and that if you are able to obtain the data and quickly analyze it, surface actionable insights will be found.  These can then be driven back to the operational systems which allow you to impact events as they unfold.  It is extremely important that you are able to catch people and things in the act and affect the outcome.  The ability to make snap decisions which move on the big data you have is vital.

A Lack of Skilled Workers

The report by CapGemini found that 37% of companies are actually having a lot of trouble finding skilled data analysts.  The best way to overcome this is to have a single common data analyst team for your entire company.  You can do this through the recruitment of new workers who have specialized in big data or through the re-skilling of your current employees.  It is important that you find employees who will not only understand the scientific aspects of big data, but the business you run, your customers and how the data can be applied to them.

Back to Top