Choose your language:

Hong Kong
New Zealand
United Kingdom
United States
man staring at gap between cliffs

You’ve come a long way, Big Data … but still have far to go

March 26, 2015
By Karsten Scherer

Big Data has arrived at a critical juncture. Organizations that jumped at its potential to revolutionize business are now increasingly disappointed by Big Data projects’ implementation challenges. Many business leaders are also surprised by how their organizational frameworks have been ill-equipped to handle the insights generated by even the most successful Big Data and business intelligence (BI) initiatives.   

As Big Data projects tax the limits of organizations’ IT abilities—and challenge businesses’ organizational structures—should companies continue to invest in them? This post explores Big Data’s origins, future and the steep challenges facing it, and offers suggestions to navigate those obstacles to reap the game-changing potential Big Data offers.

From a murky etymological origin to an increasingly clear future

While the origins of ”Big Data” as a concept aren’t clear (more on that in a fascinating New York Times article here), the term is generally acknowledged to have come into play in the late 1990s, and it gained more widespread use with both IT professionals and the general public after 2010. Now, half a decade later, I imagine the number of CIOs not wrinkling their brows at the challenges presented by Big Data to be pretty close to nil.

The deluge of terms associated with Big Data masks what is essentially a deceptively simple set of questions: How do I deal with all of the bits and bytes my organization generates? What about the data available to me from outside sources? How do I combine the two? And most critically, how can I manage to extract value from that data to make my business better? Given those questions, it’s worth looking at how some of these terms relate to each other. For simplicity’s sake, let’s lean on some Gartner definitions.

  • Big Data refers to the “high-volume, high-velocity and high-variety information assets” organizations either own or have access to inside and outside of their own firewall. This data serves as the basis for …
  • Business analytics, a term used to describe the aggregate solutions “used to build analysis models and simulations to create scenarios, understand realities and predict future states.” This includes data mining, predictive analytics, applied analytics and statistics, the results of which are delivered to business users.
  • Business intelligence (BI) is the umbrella term that all of this fits under, and “the applications, infrastructure and tools, and best practices that enable access to and analysis of information to improve and optimize decisions and performance.”

Big Data shows immense potential but the technological and organizational challenges of implementing it are huge. So how far have organizations gotten on realizing its promise? While predictive analytics and the like were probably not on Abu Bakr’s mind when he stated that “without knowledge action is useless and knowledge without action is futile” over a millennium ago, his maxim certainly applies here. As with most technology trends and business innovations, end-user organizations are challenged to cross the chasm between the benefits they hope to achieve from Big Data and its successful integration with the reality of the people and systems they have in place.

So where are we now?

It’s telling that across the IT advisory landscape, industry analysts are looking at why Big Data’s potential still remains relatively unrealized. IDC uses the eye-catching tagline ”beyond irrational exuberance“ for their Big Data analyst roadshows. Recent Forrester reports are titled ”Reset On Big Data” and ”Go Back To Basics Before You Move Ahead With (Big) Data Projects” (both links are behind a paywall). Next week, Gartner’s Business Intelligence & Analytics Summit will focus, among other things, on helping organizations “craft a winning strategy to launch or reboot your BI initiative.”

Many of you are familiar with Gartner’s hype cycle, their method of helping clients navigate the adoption, maturity and potential of different technologies and their application towards business challenges. In the most recent hype cycle for Big Data (behind paywall), Gartner’s Frank Buytenijk states that Big Data overall is sliding from the ‘peak of inflated expectations” towards a delightfully named ”trough of disillusionment.” Let’s dig a little further into what’s causing that disillusionment.

What happens when the architect doesn’t know what the people using the structure want?

Were you to ask an industry analyst about their conversations with clients on both the business and IT sides of the house, odds are high that they’d mention the lack of alignment between both groups. That’s a problem as old as the first application of technology toward a business challenge. Given the scope and scale of the Big Data conundrum, the definition of insanity as attributed to Albert Einstein holds doubly true here.

Regardless of whether your organization is at the outset of its Big Data initiative or you’re further along that process, it’s worth keeping one thing in mind: The foundation of Big Data success won’t lie in the technology. Our experience at several recent client IT leadership panels underlines this—CIOs who have taken an active role in embracing Big Data as an opportunity to engage with their business counterparts (and increasingly, marketing ones) also tend to be the ones who report the highest degree of impact for their efforts. Why? Because they’re capable of applying the lens of technology towards business needs and, increasingly, customer demands. You can’t hit a target you don’t know is there, and sometimes you need help to figure out where and what that target is.

Forrester's 2014 Business Technographics Global Data And Analytics Survey results (behind paywall) outlined this problem nicely. While more than 90 percent of IT professionals grasped the concepts behind and the meaning of Big Data, 30 percent of their business counterparts struggled to understand what it is and where its benefits lie. Given that Big Data projects are increasingly being geared toward improving the customer experience, which Gartner is referring to as “the new battlefield,” this result demonstrates a serious problem.

Reaping the rewards of Big Data means changing how organizations do things. One of the historical knocks on BI is that, in a traditional waterfall pattern, your question has changed by the time you get your answer. The Agile phenomenon has rightfully captured the attention of many in recent years, and its tenets apply here as well. Organizations that manage to foster an agile, collaborative relationship between all players on this stage (business, technology, marketing, compliance, security, etc.) will see returns on their investment. Those that don’t? Well, they won’t.

From an organizational development standpoint, there’s a fascinating additional wrinkle as well. So now your Big Data initiative presents you with findings. What are you going to do with them? Given the unparalleled combination of computational firepower, raw data and human ingenuity that comes into play here, those results will likely challenge business as usual. Do results call for new products, a different organizational structure, revised processes and so on? As Forrester’s Martha Bennett put it less than a month ago, “[no] matter how major or minor the required adjustment, unless leaders are prepared to take action, the potential benefits of a big data initiative will never be realized” (link behind paywall).

Now that your organization bought the fancy new car, can your people actually drive it?

In a recent conversation, Frank Buytenijk pointed out that the term Big Data itself can be confusing—people hear it and think, “Well, data I know, and big just means more of it,” suggesting the problem is just one of scale, and that the solution involves simply doing more of the same. In his hype cycle, Frank couldn’t be clearer regarding one of the main inhibitors of successful Big Data adoption, namely the skills required by any organizations hoping to capitalize on its promise. Not only are they dealing with a dearth of IT professionals overall, but they also require different, hybrid skills they likely don’t have in-house already. In short, you still need a license to drive, but the exam to get it isn’t the same one it used to be.

What comes to mind when roles like data scientist come up? Likely hard skills like visualization, machine learning and data mining, to name a few. However, as Forrester’s Brian Hopkins correctly states, “[data] is the raw material, but it's the understanding of the meaning in data relevant to its context and a specific problem that creates insight and action.” The potential of Big Data only gets unlocked when combinations of various types of data and human knowledge are brought together creatively. Organizations often come up short on that last, the creative piece. Do people among your data scientists know their way around behavioral psychology, social anthropology and linguistics? And coming back to my earlier point about Big Data taking a village, can your workers effectively communicate, tell stories, lead, collaborate and so on?

As Frank puts it in that hype cycle, “[most] of these skills are in short supply in many organizations, and are rare in the market at large.” Our own data bears this out. In a survey TEKsystems ran earlier this year, we polled 200+ IT decision makers on their Big Data state of the union. In ranking the roles they struggle the most to fill, 65 percent of IT executives surveyed ranked Big Data architects as their No. 1 gap, while data scientists and data modelers followed, coming in respectively at 48 and 43 percent. Overall, almost half (44 percent) of IT leaders polled expect their hiring for Big-Data-related positions to increase. You’ll find the executive summary of our results here.

In summation

Adoption of and spend on Big Data competencies and technologies is increasing steadily. We’re much closer now to mainstream adoption and clearer business cases for its usage than we were even two years ago. Soon, it’ll be table stakes. Not all organizations will benefit from Big Data equally, however. The ones that manage to add it into the DNA of “how things get done around here,” effectively break down internal silos, and can field a team with the key skills required to master one of the most interesting challenges technology has ever presented to us will—not coincidentally—be the ones that separate themselves from the competitive pack.

I've found over the years that Sherlock Holmes generally had the right idea about most things, and will leave you with a quote from “A Scandal in Bohemia,” the first of Arthur Conan Doyle’s short stories featuring the detective: "I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts."

More reading: How real companies tackled Big Data and BI challenges

Learn case studies about other organizations' challenges and successes in implementing Big Data and BI projects:

Implementing Oracle Business Intelligence Applications 11g for a Fortune 500 Manufacturing Corporation

Remediating MTN South Africa’s Business Intelligence System

Implementing a Big Data Solution on Behalf of a Media House

Karsten Scherer leads analyst relations globally for TEKsystems. He enjoys exploring the intersections between clients, the analyst community, technology and business analysis. You can reach him @TwoARGuys on Twitter.

Blog Archive