Blog

Seven critical questions required for business decision precision

Today's retail stakes are too high for "lies, damned lies, and statistics."

August 21, 2014 by Chris Petersen — Owner, IMS

Collecting data is certainly not new. However, with the advent of "Big Data," there is almost a blind faith that if we can collect more data, link it all together, we will suddenly become more intelligent and make better decisions. The challenge with all the data available today is how to make sense of it.

The typical answers are "data mining" and analytics. But just because you put mountains of data through a computer to generate more elegant views of it, that doesn't necessarily mean you'll be any smarter. In fact, more analyses often yields paralysis. Today's business decisions often do not require more data, they require asking these seven key questions for decision precision, outlined below.

Lies, damned lies, and statistics …

No one is exactly sure who should be credited for the phrase: "There are three kinds of lies: lies, damned lies and statistics." This phrase was certainly popularized by Mark Twain to describe the pervasive power of numbers. Regardless of the source, the phrase is particularly relevant in describing the use of statistics and analytics to bolster weak arguments.

I recently received an email from Gregory Yankelovich at Customer Experience IQ. Gregory always has great insights and a very practical view of developing intelligence for the consumer centric world we live in. Gregory has a great definition which captures some of the cynicism associated with big data overreach and the unsubstantiated claims coming from data mining:

Data Scientist (n): A machine for turning data you don't have into infographics you don't care about.

The power of intelligence developed through testing and measurement

The point is that data cannot lie to us. Data is not inherently good or bad … it is merely descriptive of elements at points in time. Data mining and analytics are also neither good nor bad. They are designed to "cut through the fog" to find some relationship or pattern.

The key to making better decisions is NOT sifting through data after the fact. To determine what works where requires planning metrics and measurement that will pinpoint both outcomes and what impacts results. Said another way, data is of little value and can be very misleading without a plan and design to focus on measuring what works and what doesn't.

Seven critical questions for business decision precision

1. Are you measuring the right thing(s)?

In order to make valid decisions you need data that is valid. For example, it is hard to make decisions about whether a diet makes you "healthy" if you only measure your weight. If critical decisions are being made about store design to capture consumer behavior, then there needs to be some valid data regarding consumer footfalls, where they land, and how long they stay in the area.

To measure the right things related to omni-channel consumer experience will increasingly require data beyond the traditional store and in-store cash register.

2. Are you measuring consistently?

Consistency refers to reliability of the data. If two observers watch a consumer shop and record different levels of engagement, then the data collection tools are unreliable. Unreliable data can mask any real differences and lead to erroneous conclusions. Sales data at the cash register tends to be highly reliable, but lacks validity in terms of measuring the quality of the store experience. Accurate decisions require both valid data and consistent reliability.

3. Are you measuring accurately enough?

Measuring accurately refers to precision. If you are a carpenter, measuring in inches may do for building a house, but building a fine furniture cabinet requires fractions of an inch. Measuring gross revenue or profit is simply not precise enough to determine if a training program increases add-on sales. Web traffic is not precise enough to determine if marketing is attracting new customers (unique visitors who purchase).

4. Are you measuring often enough?

It was often sufficient to measure weekly print ad impact by measuring weekly sales. Web sales fluctuate wildly by hour, and the best online retailers track sales by the minute to measure impact of variables such as pricing changes. One of the biggest misses in measuring retail store results is not measuring the impact of hourly staffing patterns.

5. Are you measuring the right variables?

Retailers are typically very good at measuring internal variables, especially sales. But understanding which variables impact results often requires analyzing external variables such as customer characteristics, competitive factors, or even the weather. The best insights come from analyzing a mix of outcome data and environmental attributes, which can impact implementation and on-going execution.

6. Is your sample large enough?

It is extremely dangerous to draw conclusions from a case of one. Doing so leads you vulnerable to the Hawthorne Effect: People simply behave and perform differently when there is something new going on (as in the case of new store design, training, or even website pages).

In order to limit the false conclusions and misinterpretations, the sample needs to include at least several implementations in the same period. The larger the financial decision or risk, the larger the sample needs to be for both test and control stores.

7. How can you be sure it's not something else?

If you search long enough with enough data mining, you will find something statistically significant. For example, researchers have found a statistical correlation between the height of heels on women's shoes and the state of the economy. But in the case of retail business decisions, we need to know if some variable like training caused a change in an outcome like consumer satisfaction.

With the realities of retail, it is often impossible to randomly assign stores to test and control groups. However, it is possible to identify and measure "comp stores" who have similar demographics, but they don't get the implementation variables. The single greatest source of "lies" in retail is trying to draw conclusions from data without measuring enough comp stores during the same time periods.

(Photo by reynermedia.)