Prospective clients always ask me if Big Data is a passing fad or something that’s truly going to provide the benefits that they're looking for - deeper insights, better decision making and faster time-to-value. This is an important question that deserves an answer based on facts and not on marketing hype.

In this article, I shall provide my perspective, but before I do that, I must state that Big Data is not a fad and will certainly benefit organizations, if its implemented correctly.

According to a recent report, “Joining the Dots: Decision Making for a New Era,” by the American Institute of CPAs (AICPA) and the Chartered Institute of Management Accountants (CIMA), 32% of 300 C-level executives at large organizations from 16 countries around the world said Big Data has made things worse, not better, for decision making. In fact, 70% of those surveyed said at least one strategic initiative failed in the last three years due to delays in strategic decision making.

These findings fly in the face of the rhetoric that’s heard around the industry. Before we go off and start finding fault with product vendors and industry pundits, let's take a step back and analyse the situation, to identify the root causes for the current state of affairs.

Based on my experience with clients that have either made the leap into Big Data or have invested in Proof-of-Concept implementations, I've identified 6 reasons why Big Data investments aren't paying off for some organisations: (1) Lack of problem definition, (2) Skill mismatch, (3) Scope creep, (4) Data wrangling challenge, (5) Contextual gap and (6) Evolving technology. Here's a deeper dive into each: 

Lack of problem definition - Many organisations don't take the time to clearly articulate the problem(s) that they wish to solve with Big Data and to determine whether those problems truly need a Big Data solution. Picking the wrong problem, jumping in before clearly defining the problem, listing the questions that one wants to answer or hypothesis that one needs to prove or disprove, are a recipe for disaster.  

Skill mismatch - Hadoop and NoSQL implementations require a certain skill set, which is very different from traditional relational database skills.  There is a tendency on the part of organisations to redeploy resources with traditional data skills to Big Data projects, before providing them adequate training in the frameworks, methodologies and underlying architecture. This results in sub-optimal designs and sub-standard products, which may not meet the client's expectation. Under these circumstances, it is advisable to augment internal staff with Big Data specialists from service providers and Big Data product vendors.

Scope Creep - Starting small and taking baby steps to reach the end goal should be the preferred approach. In order to test Big Data and prove its capability, it is best to start with a well-defined problem that has a narrow scope. This will provide an opportunity for the team (e.g. Data Analysts, Data Scientists, Big Data Developers, Business Domain experts, etc.) to stay focused and deliver results in a reasonable amount of time. The end goal shouldn't be to test performance and scalability of the platform (that's a given), but the data ingestion, data wrangling, modelling and analytics processes end-to-end. 

Data Wrangling Challenge - For business customers time-to-value is paramount. There is a tendency on the part of IT to not appreciate the time and effort required for wrangling data to get it right. Industry estimates put this at approximately 70% of the effort. See my article 5 Solutions For Wrangling Data to Deliver Faster Insights for more details.

Contextual Gap - Solving complex problems that are multi-dimensional in nature requires significant contextual data and business domain knowledge, in addition to the availability of large sets of raw data. This tribal knowledge isn't readily available in a repository but has to be extracted from various individuals. Given the siloed nature of organisations and their focus on parochial interests, this tends to have a major impact on project deliverables.

Evolving Technology - It took relational databases a decade or two to mature and become robust enough to support business transactions. Same is true of Big Data technologies. Various components of the Big Data platform are evolving and maturing. It is important to keep this in mind, as organisation's embark on Big Data projects. Do the due diligence to determine which components of the Big Data stack are relatively stable and which ones are not. Design your solution to take these data points into account, so that you reduce your dependency on the less mature components or have alternate plans in case you run into issues. Leverage expertise provided by product vendors and integration partners.

I always advise clients to remember the maxim "Buyer Beware". What is clear from the above is that organisations must perform the necessary due diligence before implementing Big Data. This will increase the probability of success and their ability to improve decision-making. All is not lost for organisations that have invested in Big Data and aren't seeing the desired results. I'd suggest they take a step back and consider the points made above, identify the bottlenecks or issues with their implementation and take corrective action.  

Leaders within organisations and HR departments should also focus on skill development across the business, technology and operational roles, at the line level. Managers play a key role in the success of Big Data projects. They must be trained and equipped to ask the right questions, make decisions related to architecture and design options and be able to scope out and plan the engagements. One should not expect them to transition into this new role, without adequate training and management support.

My observations and assessments were validated by a recent Forbes article on American Express's Big Data Journey, where Ash Gupta, President of Global Credit Risk and Information Management for AMEX listed three challenges along the way:

(1) adoption of new and immature technologies required significant organizational adaptation and cultural transformation. Old processes became obsolete. New approaches required fresh skills and approaches;

(2) AMEX needed to recruit new talent, with skills in Big Data solutions and approaches. This challenge was complicated by the scarcity of Big Data talent, and compounded by two additional factors: a) the need to always understand “business context”, which comes from experience, and b) the tendency for millennial Big Data talent to continually seek new challenges, creating a retention challenge; and

(3) Mr. Gupta cited the “marketing process journey”, which he characterised as a process of continuous improvement intended to consistently refashion customer experience in a positive way. For American Express, this meant employing the same kind of “test and learn” techniques and learning-through-iterative-improvement approaches that the firm has used in the past to refine its customer marketing models.

Big Data projects introduce new paradigms, new processes and non-traditional skills and require organizations to change the way they operate. Although technology and data play critical roles, one must not minimize the importance of organizational culture, senior level sponsorship and an organization's appetite for change.

By employing best practices and lessons learned on Big Data implementations, organisations can take proactive measures to increase the probability of success. They should also focus on implementing a "test and learn" methodology to improve their processes over time, resulting in better outcomes. The Big Data talent gap must be tackled through in-house training and acquiring fresh talent.

Jay Zaidi is an entrepreneur and author in strategic data management. His book called “Data-driven Leaders Always Win“ is available on Amazon. Jay is one of the overseas keynote speakers at inForum 2016, the annual conference of Records and Information Management Professionals Australasia, September 11 -14 2016, Crown Perth.