Using data to generate business value is already a reality in many industries. That said, big data and the analytics we build from it are still in their infancy.
Big data is a term for data sets that are so large or complex traditional data processing applications are inadequate. For many, the idea of big data is scary – very, very, scary. For others, it’s exciting. Whichever side of the fence you’re on, the reality is big data is big news, and it’s something every business should turn to should they want to see significant growth.
And what business doesn’t, right?
Big data is a collection of data from traditional and digital sources inside and outside your company that represents a source for ongoing discovery and analysis. It’s a mix of unstructured data, such as metadata, Twitter tweets, and other social media posts, as well as multi-structured data, derived from interactions between people and machines, such as web applications and web log data.
Industry leaders use phrases such as “volume” (the amount of data), “velocity” (the speed of information generated and flowing into the enterprise) and “variety” (the kind of data available) to begin to frame the big data discussion. Others focus on additional V’s, such as “veracity” and “value”.
As you can see there are many ways to look at it, but there is one thing that is certain; every enterprise needs to fully understand big data, and that means getting past the big data challenges that we go into below.
Finding the right data
Many companies have difficulty identifying the right data and determining how to best use it. In fact, data source discovery often consumes well over half the time spent on analytic projects. That can certainly put a crimp in your analytic ambitions.
The solution: Building data-related business cases often means thinking outside the box, and looking for revenue models different from the traditional business structures. All data professionals and business analysts need to know all the data available, and how to profile, identify, and provision data in a fraction of the time it takes via traditional manual approaches. Essentially, this means building an efficient team to support your analytics.
In today’s hypercompetitive environment, companies not only have to find and analyse the relevant data they need, they must find it quickly. Visualisation helps organisations perform analysis and make decisions fast, but that means going through sheer volumes of data and accessing the level of detail required, all at a high speed.
The solution: One possible answer is installing hardware with increased memory and powerful parallel. Another is grid computing, in which a company puts data in-memory. Both approaches allow you to explore huge data in near real-time.
If data comes from social media content, you need to know who the user is in a general sense. Without some sort of context, visualisation tools offer little value and it takes a lot of understanding to get data in the right shape.
The solution: Have the proper domain expertises in place – in other words, having people that understand where the data comes from by doing the analysis.
The value of data for decision-making purposes will be compromised if it’s not accurate or timely, yet 84 percent of organisations admit to experiencing data quality challenges. Top data quality challenges include improving deliverability, achieving a single customer view, and enhancing insight and loyalty. All of these are extremely important as they look to connect intelligently with consumers.
The solution: It’s crucial for companies to have a data governance or information management process in place to ensure the data is clean. Without effective data management, you will be unable to gain necessary insights or perform required customer analysis.
How you display your meaningful data results becomes difficult when dealing with large amounts of information. Imagine trying to display plot points on a graph for analysis when you have 10 billion rows of SKU data!
The solution: Cluster data into a higher-level view where smaller groups of data become visible. By grouping the data together, you gain greater visualisation of your results.
Having the right technology infrastructure is an essential component of big-data success, but it doesn’t necessarily mean investing heavily and overshooting the mark. Sure, there’s the age old saying “If you build it, they will come”, but build it too big for your needs and it’s a recipe for failure.
But, at the same time, you shouldn’t only build the technology stack required for your immediate needs. Prepare for the future.
The solution: Build a technology stack that sits between these two poles. Plan your investments to handle anticipated growth rather than unrealistic growth.
Outliers may not be representative of the majority of data, but they can reveal previously unseen and potentially valuable insights. Outliers typically represent about one to five percent of data, but when you’re working with massive numbers, viewing this can be difficult.
The solution: Remove the outliers from the data and create a separate chart, or bin the results to both view the distribution of data and see the outliers.
This might just be the biggest challenge to overcome. When it comes to bringing big data into the corporate culture, there are two kinds of companies. Some firms – Google, Amazon, and Netflix for example, have big data baked into their business model. Big data was there from the get-go, so it’s never been a major issue.
Then there are business which have always valued intuition and experience as a way to make decisions. It’s these that will struggle to understand the importance of big data, believing they already know their customer without statistical back up.
The solution: Realise the era of big data is here. Data is so prevalent, and technology and analytic solutions will continue to improve. For those that have embraced it, they’ll be able to see their business in ways never thought possible, and for those trailing behind, it’s inevitable you will one day be trumped by this greater insight. To stay competitive, you must be open to big data.