Blog

Big data: Finding an ant in an avalanche

September 20, 2012 by Jeff Weidauer — Vice President, Marketing & Strategy, Vestcom International, Inc.

Big data. It's the latest answer to our marketing prayers; the "holy grail" that will deliver the ultimate return for our precious marketing spend. Last year consulting group McKinsey & Company did a major study on big data, releasing a book-length report on the coming wave and how we can all ride it to the beach of the promised land. Since then a lot of time has been spent on defining what big data is, but less on getting real value from it.

A recent survey of more than 75 retailers by the Edgell Knowledge Network points out some of the realities of big data for retailers. The good news is that more than 80 percent had heard of big data (which is consistent with the number who have heard of Pinterest, according to some sources), but only 47 percent really believe they understand the implications. In addition, for those who had some experience with big data, the major hurdles could be referred to as the "Three Vs"—volume, velocity and variety. The quantity, speed and diversity of the data being gathered are all massive—well beyond the ability of most software packages to manage, never mind the average human.

Apparently big data is living up to its "big" moniker. Numbers followed by 12 or 15 zeroes are necessary to describe the size of the datasets generated when tracking shopper behavior, buying habits, geolocations and the multitude of other information available.

In our new world—the world where 12 terabytes of tweets are generated each day, according to Edgell—big data isn't something we can ignore. But it's more than just gathering and storing; data isn't wine or nuts that can be set aside in case of an emergency or a long winter. It's both time-sensitive and perishable, with a finite life span that gets shorter as more data piles in behind it.

The hard part, i.e. the part that gets really interesting, is when we start to mine this mountain for the nuggets of wisdom that lurk deep inside. Data mining at this level is a skill-set that's hard to find anywhere, and in the world of retail where inertia is often the major driving force, the sheer size and speed can be overwhelming for those unprepared for the onslaught. This is more like looking for an ant in an avalanche than a diamond in a coal mine.

Developing a strategy for big data is a critical first step, but according to the Edgell study, only five percent of retailers have even started on a strategy. Conversely, 30 percent have executed what they consider to be a big data project. While those projects might be serving as "on-the-job training," it's very likely that tactics have supplanted strategy in the rush to "do something." While that's hardly unusual, this lack of preparation is likely the driving force behind the "Three Vs."

There are essentially two choices: start paddling and catch the wave, or prepare to capsize. To prepare for the voyage, here are three steps to take now:

1. Take a step back and decide what and how big data can help drive the business.In other words, get started with a strategy. This is the single most important step, and will determine timing and investment going forward.

2. Be prepared to spend some money.Managing and using big data isn't something that the typical IT team can do as an add-on to its current workload. Instead start thinking about big data and its value to marketing, not as an IT project. Outsourcing may be the only viable option.

3. Be open to change.The insights coming from big data will present opportunities that may be counter-intuitive, and users must be willing to trust the information even if it goes against what "feels right." This will be the highest hurdle for most, and will be the deciding factor between a successful entry to big data marketing and failure.

The next couple of years will yield massive changes to marketing and retail through big data. Whether big data delivers on both its hope and its hype depends on how well we embrace the changes it will offer, and how willing we are to listen and take action.