Blog

For high-velocity retail business, mind the data gap

Jason English, principal analyst and CMO, Intellyx, maps out why overcoming retail data gaps in real time, at cloud scale, will become critical to business survival.

Photo by iStock.com

June 20, 2019

By Jason English, principal analyst and CMO, Intellyx

Retail businesses must perform flawlessly to survive in the age of omnichannel, where convenient choices, cheaper prices and better service are always readily available for consumers.

Companies that thrive in this competitive space will come to grips with high-velocity customer and operational data, at massive scale. While the retail market tends to invest heavily in IT, there is still a widening data gap as retailers move their software, and the ever-increasing amount of data it produces, to the cloud.

Like a train platform, not minding this quickly widening data gap causes customers to step off into oblivion when they least expect it.

Why the data gap is happening

At high speeds, retailers experience a separation between record and reality within their critical data. To address this gap, there is always the unacceptable option of slowing transactions down, thus creating a data ‘choke point' to confirm everything is in sync from the point of sale to systems of record.

Good luck retaining customers by interrupting services. Retail data must evolve and keep up with the unlimited speed of business.

We've already moved from first generation object and hierarchical databases to today's largely SQL-based relational databases (RDBMS). This advance allowed databases to scale, adding more infrastructure, tables and memory to meet the growing workload needs of retail operations, to a point.

Relational databases supported the ACID transaction capabilities required of them, but as retail velocity increased, the ‘all-or-nothing' transactional correctness aspect of RDBMS and the costs of centralization and vendor lock-in started getting in the way of scale.

Enter NoSQL databases, bringing more flexible document and/or graph-based approaches to data scaling and definition, along with open-source and cloud-based ‘pay as you go' affordability versus draconian vendor licensing terms. MongoDB and Cassandra started coming into vogue, as software deployments became more distributed to serve online retail options as well as brick-and-mortar stores.

Now, cloud-based services and stateless containers could maintain context through a ‘data wrapper' of records, bursting and expanding to meet business needs, especially at peak demand times.

However, scaling flexibility brings a trade off. We lose 100% certainty that the current state of all transactions and inventory reached a verifiable consensus across all the nodes of a distributed system.

With a brief ‘2-phase commit' for each transaction, a data gap appears. Since data cannot move around the world faster than light, any time interval while different nodes ‘check their watch' before confirming transactions can generate big mistakes.

Scaling data to an on-demand data lake future

Even for tech-savvy retailers, the volume data gap is mystifying. They already made investments to scale out old databases with a hybrid of on-premises and cloud-based repositories. Every previous generation of database remains in operation once installed, and replacement costs are high.

A renaissance of data science is underway. Retailers are learning to leverage big data and analytics to target customer needs that were never before imagined.

Customer loyalty programs. Recommendation engines. Personalization. Product customization. These ‘high-touch' customer scenarios were once exclusively handled through one-on-one personal interactions, or periodic, rule-based filtering and manipulation of customer profile data to serve campaigns.

Retailers must now understand and predict customer behavior in real time, faster than the competition, or lose. That means gathering, and acting on, rivers and lakes of data, which can become data swamps.

We can't drop the ball on the speed, correctness or scale of data. If we collect this much actionable customer data, does that create yet another data gap for risk?

Forging through risk and compliance

Retailers must personalize sales and marketing efforts, without running aground of well-publicized data failures, and the accompanying costs of maintaining compliance under the government regulation and oversight that follows.

The massive Target data breach of 2013 was just a first well-publicized one. Since then, US retailers have the dubious honor of leading the world in large data breaches, with one after another at Under Armor, Macy's, eBay, Adidas, Chipotle to name just a few. Still want that store credit account?

Unfortunately, PII breaches are so common in adjacent industries, that every retail customer's identity may already be jeopardized in some lifelong way. Take for instance the 143 credit bureau records exposed at Equifax and as many as 500 million customer accounts exposed at Marriott hotels.

Data privacy regulations such as Europe's GDPR are driving consumer protection models for other nations. Compliance and regulation will put data permission back into the hands of customers, including the right to be forgotten.

Security and operational visibility go hand in hand. Threats continually evolve, and databases can't rely on preventative measures alone. The ability to shine a light on unexpected data access and manipulation behaviors to detect and remediate them while they are happening is crucial.

For retail businesses to survive at high velocity, their database technology must evolve as significantly as the real-time, personalized customer experiences it supports.

Efforts are underway to address these data gaps and offer the strict serialization and ACID transaction verifiability of RDBMS, with the flexible scaling and cloud-native nature of NoSQL-style databases.

One interesting startup to watch in this space is Fauna. They have the first instance of a Calvin-based database approach to offer retail applications cloud-native scalability with the gold standard of transaction commits in a single phase.

While the evolution of database technology may be beyond the understanding of most retail executives, overcoming retail data gaps in real time at cloud scale will become critical to business survival.