Tool 5: Mystery Shopping

Mystery shopping aims to observe the actual behavior of individual financial services provider (FSP) staff members or of third parties acting on their behalf during a true customer/FSP interaction. To use this tool, a market conduct supervisor (MCS) sends a trained consumer or supervisory staff member to an FSP access point to simulate a typical customer interaction. This “mystery shopper” then reports on their experience in a detailed and standardized manner. The interaction may be in person or remote (e.g., phone call, web chat inquiry) and relate to any part of the customer journey (e.g., shopping for a product, purchasing a product, making a transaction, calling customer service, making a complaint).

Common consumer protection supervisory goals for mystery shopping include:

• Understanding compliance with a regulatory regime. Compliance may include rules on sales practices, fair treatment, suitability, or disclosure of product information.
  • For example, a credit mystery shopper may be instructed to ask for an explanation of the annual percentage rate (APR) or total cost of credit to assess the effectiveness of its disclosure during a sale and the ability of sales staff to explain its meaning and significance to consumers.
  • Mystery shopping may reveal disparate treatment across consumer types (e.g., lower income, less knowledgeable) and of certain vulnerable consumer segments (e.g., elderly individuals, women), and provide anecdotal evidence of FSP compliance with fair treatment rules.
  • Mystery shopping may be useful in checking compliance with time bound disclosure requirements such as verbal explanations given to consumers. Through a survey in 32 countries, Consumers International found the single most important consumer challenge to be the lack of explanation of contract terms at time of sale.
• Gauging the effect of a recent regulatory reform at the retail level. This may be achieved by conducting a two-phase mystery shopping exercise before and shortly after implementation of a regulatory reform.
• Assessing staff knowledge at an FSP access point—and whether knowledge is proactively or reactively shared. With information gathered via questions a mystery shopper poses to FSP staff, it is possible to identify how well staff members explain key product terms and features, present product options, and address customer needs. Assigning certain shoppers certain questions may also identify how much information a staff member shares or withholds—depending on what the consumer asks for.
• Identifying variations in staff behavior at FSP access points based on customer profiles. Using actual consumers with different real-life characteristics (e.g., income, credit history) and preferences (e.g., loan size, level of risk tolerance, use of savings), and defining specific mystery shopper profiles makes it possible to identify whether FSP staff or third parties behave differently or offer different products depending on the personal/financial traits and preferences of a customer. Comparing responses across FSPs against specific consumer characteristics may help an MCS to identify the negative experiences a vulnerable consumer segment faces, among other issues.

Benefits and opportunities

Mystery shopping benefits MCSs in numerous ways:

• Comprehensiveness. It helps an MCS to identify, confirm, and acquire in-depth knowledge and understanding of consumer experience (positive or negative) in using financial services and interacting with FSPs. Mystery shopping provides a comprehensive view of the different components of consumer experience in a problematic situation.
• Proactivity. It uncovers how FSPs respond to particular situations and customer characteristics and preferences, and unveils new business practices, new products and services, plus potential consumer risk. No other market monitoring tool so directly achieves this result.
• Supervisory effectiveness. Mystery shopping may strengthen an MCS’s work at various points in the supervisory journey. For example, it can be used to investigate market issues identified by another market monitoring tool. It can improve the MCS’s understanding of how customers are affected by the authorization of new products and services or by the issuance of new guidance and public warnings.
• Segmentation. It gathers anecdotal evidence on the variations in consumer experience associated with different customer profiles, which may shed light on unfair treatment of women, elderly individuals, rural dwellers, LGBT+ individuals, and other vulnerable consumer segments.
• Feedback. It can strengthen the regulatory feedback process. For example, mystery shopping can provide input into the design and amendment of regulations or assist with compliance checks on regulatory provisions. 
• Dissemination. Disseminating mystery shopping results can provide critical insights for consumer associations and FSPs. Consumer advocacy groups and associations use mystery shopping to highlight consumer issues and bring them to the attention of the public, MCSs, and policy makers. FSPs benefit because mystery shopping gathers information that may help them improve their work processes, staff training, and customer service. It also allows them to verify and reinforce policies and compare themselves with competitors.

Back to top ↑

Characteristics of this tool

Mystery shopping is one of the simplest, most effective ways to gather deep insights on consumer experience with FSPs in predefined scenarios. It usually focuses on a subset of products, FSP types, and locations, and generates qualitative and quantitative demand-side data. However, it is not a nationally representative survey and therefore does not accurately measure consumer experience across an entire population. Instead, it is an indicative method that may improve FSP conduct by signaling conduct issues in the market that require further policy investigation or supervisory intervention. Mystery shopping is a useful tool for informing consumer protection supervision and understanding market conduct issues. Many authorities already use it for research purposes, and can repurpose, refine, or redesign the tool to meet their specific supervisory needs and goals.

Back to top ↑

How to use this tool

While mystery shopping exercises are relatively inexpensive, human resources and the capacity of MCSs and consumer protection regulators may be limited. While some MCSs conduct mystery shopping on their own, field work and analysis is often outsourced to a market research firm (see Box 1). An MCS may have staff with the requisite skills and availability to conduct data analysis. Supervisory or regulatory staff can participate in the field work led by the market research firm—a useful learning opportunity as long as staff members follow the preparatory steps other mystery shoppers have taken and their faces are not recognizable by FSP staff members. It may be useful to have the MCS and/or experts familiar with policy and supervisory objectives and applicable regulations handle the analysis and interpretation of mystery shopping findings. 

There are five key steps to consider when implementing a mystery shopping exercise:

STEP 1:

DETERMINING THE POLICY OBJECTIVES OF THE MYSTERY SHOPPING

To begin a mystery shopping exercise, the MCS first identifies its key objective. Objectives may include measuring compliance with existing regulations; improving understanding of potential problem areas where new regulations or reforms to existing regulations are being considered; or monitoring developments in product markets to identify concerning trends, features, or behaviors (e.g., the KonfOP product-focused market monitoring in Russia that was based on mystery shopping).

STEP 2:

SELECTING PRODUCTS, PROVIDER TYPES, AND DELIVERY CHANNELS

When determining which products to include in its mystery shopping exercise, the MCS may select those which strongly require additional knowledge of FSP behavior and practices vis-à-vis customers—according to cues collected from other market monitoring or supervisory activities. The MCS also needs to select specific products and delivery channels, including bank branches, phone-based services, agents, and other third-party intermediaries, plus financial advisors.

Depending on the product being “mystery shopped,” several aspects of research design can be customized to the local context in terms of financial sector landscape and regulations: 

• Relevant regulations. The MCS must provide the research firm with all relevant regulatory requirements, provisions, and guidelines applicable to the chosen product, provider, and delivery channel, including those issued by other agencies. To determine the most important outcomes to measure, the information must be shared prior to development of the mystery shopping questionnaire.
• Key product terms. Key product terms include the different ways product costs and benefits are conveyed, and the most relevant features consumers are informed of or request information about when interacting with FSPs, their agents, or other third parties.
• Delivery channels. To measure a channel’s impact on customer experience, where possible mystery shopping must combine the different channels an FSP uses to sell products and interact with customers (including remote channels). 
• Determining provider sample and geographic coverage (in the case of physical channels). While logistically challenging to locate and cover small institutions in a mystery shopping exercise, it is desirable to include as many provider types as possible to capture the true diversity of market practices.

STEP 3:

DESIGNING CONSUMER PROFILES FOR SHOPPING EXERCISES

The research firm should design consumer profiles in consultation with the MCS. Profiles are determined based on the exercise’s specific objectives and customized to the local context, including types of consumers, products offered, and general economic and demographic information. When possible, traits should be based on the mystery shoppers’ actual personal information or characteristics. It is therefore important to recruit mystery shoppers whose real-life situations match those traits. To complement the actual personal information, it is useful to include in a profile a series of assumed characteristics, behaviors, or needs to be portrayed during the mystery shopping exercise. This may include level of financial knowledge (e.g., experienced vs inexperienced customer), specific financial needs or preferences, recognizable social affiliations and other social signaling, and style of dress. Traits are randomly assigned to mystery shoppers independent of their actual traits. It is important for the MCS to reflect on vulnerable customer segments that may be of particular concern to its mystery shopping objectives.

STEP 4:

TRAINING SHOPPERS AND CONDUCTING MYSTERY SHOPPING EXERCISES

It is important for the research firm to conduct in-depth training and rehearsals of all profiles before embarking on the mystery shopping field exercise. The firm should observe shoppers conducting profile role-play to ensure they understand their profile and what to convey (or not) during the exercise. Shoppers must also be trained in completing post-exercise questionnaires. However, they are not expected to complete them on their own, given the extensive coverage of product terms, categorization of types of information received (e.g., provided voluntarily or upon request; verbally, in writing, or both; explained or not), and how the FSP or third party provided information. Shoppers should be monitored by a field supervisor with experience in conducting surveys and verifying data collected—likely a staff member of the market research firm—who helps shoppers complete the questionnaire after the exercise.

STEP 5:

ANALYZING MYSTERY SHOPPING RESULTS

To analyze the results of an exercise it is first useful to classify into categories the information shoppers have recorded, such as: 

• Product information provided verbally or in writing
• Printed materials (e.g., brochures, advertisements)
• Access barriers and requirements for shoppers
• Customer experience (e.g., waiting time, time the exercise took)
• Consumer perceptions of the exercise (e.g., satisfaction with the interaction, opinion of staff)
• Final product offer, when applicable

Once this information has been categorized it is ready to analyze against shopper profiles. By comparing the information recorded by shoppers against each shopper profile, it is possible to assess, for example, whether differentiated treatment by FSP staff aligns with certain consumer profiles, including vulnerable consumer segments.

Initial background information on product features and the various products an FSP offers a customer are critical to the analysis phase. This information can be contrasted against the actual information the shopper received from the FSP. For example, a comparison may identify situations where FSP staff omitted a more suitable product that could have been offered to the shopper.

Back to top ↑

Limitations of this tool

• Unrepresentativeness. Like all qualitative studies, mystery shopping results are not statistically representative and cannot be employed as such. The results of a mystery shopping exercise are a one-time snapshot. If an MCS’s objective is to monitor trends the exercise would be repeated over time—considering resources are available to do so.
• Resource intensiveness. Another challenge may be the limited capacity of regulatory and supervisory authorities to conduct an exercise on their own (without hiring an external vendor). Since its staff may be too familiar, too knowledgeable, or too biased to pass as real-life consumers, it is preferable that an MCS not conduct mystery shopping itself. 
• Data quality. Mystery shoppers may be influenced by personal bias while completing an exercise, so it is important that profiles are adequate. In small markets, repeated use of the same mystery shoppers or even conducting a single mystery shopping exercise may be limiting. This is especially the case when small market size means shoppers must conduct an exercise in their area of residence.
• Supervisory powers. In some jurisdictions there may be legal limitations on how the results of mystery shopping exercises can be used. For example, some countries only allow MCSs to take enforcement action based on the results of formal FSP inspections or formal customer complaints. In these countries, the results of a mystery shopping exercise may serve as the basis for sectoral policy and regulatory changes or to clarify supervisory expectations for regulated FSPs—but cannot be used against a specific FSP.

Back to top ↑

Other resources

• Mystery Shopping for Financial Services: A Technical Guide (CGAP 2015)
• Mystery Shopping for Digital Financial Services: A Toolkit (CGAP 2017)
• Applying Behavioral Research to Consumer Protection. Module 4: Fair Treatment and Sales Practices – Mystery Shopping (CGAP 2018)
• Putting Mystery Shopping to the Test in the Philippines (CGAP 2015)
• Mystery Shopping: A Different Way to ‘Listen to the Customer’ (CGAP 2015)
• Financial (Dis-)Information: Evidence from an Audit Study in Mexico (World Bank 2014)

Back to top ↑