The Problem with Mystery Shoppers

Mystery Shopper

Mystery shoppers are a go-to analytics tool for retailers. An army of undercover researchers posing as shoppers, scoring stores on their customer experience. Remote management using that data to understand whether a consistent customer experience is being delivered across their physical network. Sounds reasonable, right?

But there’s a big problem with mystery shoppers. In fact, there are several problems. Retailers who invest huge budgets in mystery shop data are not getting what they think they are. At best, mystery shopper results are limited. At worst, they’re expensive, misused and misleading.

Problem 1: mystery shoppers aren’t customers

When a retailer uses mystery shoppers, they are briefing someone to pretend to be a customer. And it might feel like you’re getting a close approximation of your customer base when you choose carefully from the agency pick ‘n’ mix, where mystery shoppers “vary in age and gender” and “are available individually, as a couple or a family”, according to one agency website.

But mystery shoppers are not customers. The data they generate might tell you whether you’re delivering operationally or not. But it tells you nothing about WHY a customer might want to be in your store, or in a competitor’s store instead. 

Mystery shoppers are not real customers, so their data tells you nothing about real customer preference.

Problem 2: the data is biased

Mystery shopper agencies promise “objective” feedback. But that’s not what they deliver. Mystery shopper data is skewed by the mindset of the mystery shopper, the agenda of the brief and the needs of the retailer.

Mystery shoppers are given parameters that funnel them into looking at a specific part of the customer experience. For example, when a global pizza brand used mystery shoppers, they would be told to order a certain type of pizza, make judgments on the restaurant and ask a couple of set questions. That meant that the data they got was consistent and can be compared across stores. Good. Or maybe not because it also means everything else in that data capture is tainted by the brief, and the questions not asked form a big hole. You’re not getting a real picture of a realistic customer interaction. The data reflects that bias.

Retailers using mystery shopper data are relying heavily on the subjective opinion of the mystery shopper. And their view of reality is influenced from the outset by the parameters they’ve been given. Any data capture that is influenced by bias from the start is questionable in its value.

And turning mystery shopper data into useful information is difficult. Modern mystery shopping agencies often use verbatims – audio or video recordings of the shopper describing their visit. These then get automatically transcribed and analysed for keyword relationships. But this is sentiment analysis, an imprecise tool that won’t pick up on subtleties of customer experience.

And the agency itself will always be a modifying barrier between the raw results and what the client gets, introducing more subjective bias into the process.

Problem 3: mystery shopper data is used as a management tool

It is almost inevitable that companies will link mystery shopper results with team bonuses. Get good results, bonuses all round. Get poor results, bonuses are cut. Mystery shopper data becomes a policing tool.

This is unfair and bad for team morale. One unlucky, unrepresentative mystery shop, at the wrong time on the wrong day, can destroy a hard-earned bonus.

And when bonuses depend on specific results, colleagues are incentivised to game the process, for example “fixing” issues that will deliver better mystery shopper scores but might not attract new customers.

Problem 4: it’s a lot of money for not much return

Gathering mystery shopper data is expensive. Paying for even just two visits a month per store over an entire business can be a huge cost. This often means that mystery shopping gets thinly spread to save money. And the result is data related only to tiny slices of time, one moment in one day, a couple of times a month. It’s not representative or accurate.

The cost of mystery shopping means it’s not used as extensively as it needs to be to be of any use. And retailers rarely budget to mystery shop their competitors effectively.

So why do retailers use mystery shoppers?

Most big retailers are using mystery shoppers because they have no idea what it’s like out there in their stores. They don’t know what it’s like to shop with them, and they’re desperate for data. And mystery shopping does generate a steady data stream to satisfy that desire.

Sure, that data can return some useful information. It’s a blunt-edged tool, but ask a simple question, like “are we giving decent service?” over multiple visits to multiple stores, and you might get some useful answers, for example flagging up problem stores. It might help you fix issues in your operational delivery.

But mystery shopping is an expensive way to do standards policing. Beyond that, it’s a biased measurement tool, essentially useless as a measure of shopper preference.

What should you do instead?

If your business uses mystery shopper data, consider having a hard think about the utility of this analytics tool. Why are you doing it? Is it because you don’t have data and feel you should have some instead of none?

If that’s the case, it’s time to think about different ways of gathering data. And about what kind of data could give you information on shopper preference – the reasons real customers choose to shop with you, or with your competition.