Tool 8: Phone Surveys

Demand-side phone surveys reveal information about customers’ experiences with financial services, as well as noncustomers’ reasons for not using financial services at either the household or the individual level. Market conduct supervisors (MCSs) can use this type of information to complement financial services provider (FSP) data, such as regulatory reporting, and to get deeper insights into issues identified by other means. For example, granular complaints data may indicate that women and rural customers have their complaints rejected by FSPs more frequently than men and urban customers, respectively. To collect additional evidence and draw conclusions about this issue, an MCS may conduct a phone survey with selected customers to enquire about their rejected complaints and focus on their experience with complaints handling mechanisms put in place by FSPs.

Benefits and opportunities

Demand-side phone surveys benefit MCSs across several aspects:

• Comprehensiveness. Collecting regular consumer data that complements regular data from providers furnishes an MCS with a more holistic view of the market on an ongoing basis. Gathering rigorous data and evidence directly from consumers can help the MCS to better understand and assess the size or extent of a specific consumer issue, such as an unfair marketing or sales practice or consumer overindebtedness.
• Segmentation. It allows an MCS to rapidly obtain and analyze data that has been segmented by different groups of consumers, including by gender, age, location, etc.
• Ease of implementation. Phone surveys collect direct consumer data in a manner that is less expensive, quicker, and safer to implement than person-to-person surveys.

Data gathered from demand-side phone surveys benefit supervisors in terms of:

• Depth. Listening to the voice of the consumer helps MCSs to better understand consumers’ financial needs, barriers encountered when seeking formal financial services and products, positive and negative experiences using formal and informal finance, and emerging risks and problems.
• Proactivity. Data identify the emergence and uptake of new digital financial services and products.
• Supervisory effectiveness. Data help MCSs to be better informed before taking specific measures and facilitate their assessment of the results of supervisory or regulatory actions.

Back to top ↑

Characteristics of this tool

Phone surveys are conducted remotely. There are three main types of phone survey:

• CATI (computer-assisted telephone interviewing). Enumerators interview survey respondents in a voice call made on any type of phone while using an electronic device (e.g., computer, tablet, mobile phone) to read the survey script and enter data collected. Interviewers may call and enter data using a single device or call on one device (e.g., mobile phone) and enter data on another device (e.g., tablet).
• SMS (short messaging service). Self-administered surveys use text messages to send questions and answers back and forth.
• IVR (interactive voice response). Automated prerecorded voice surveys that respondents self-administer by using voice and touch-tones on a mobile keypad.

The research team within an MCS may already have commissioned phone surveys. In that case, it may be possible to repurpose or refine the surveys to meet market conduct supervisory needs and goals.

Back to top ↑

How to use this tool

Phone surveys often are outsourced to research groups that specialize in this methodology (e.g., academics, firms, NGOs). Before an MCS engages researchers, it is important to compile a specific objective and questions that are best suited to one of the three main demand-side survey types: CATI, IVR, or SMS. For example, an objective could be to understand customer experience with a new financial product in the market, such as digital mobile credit, to assess the type of risk it is generating among customers.

Think through the following seven steps before hiring a research group to conduct an actual survey:

STEP 1:

DEFINE THE RESEARCH OBJECTIVE

This includes identifying the main questions to be answered.

STEP 2:

DEFINE THE SAMPLE UNIT

What is the basic level of population you want to measure (e.g., individual, household)?

STEP 3:

DETERMINE THE SAMPLING FRAME

Like in any survey, it is essential to precisely define the target population. A “sampling frame” is a physical list of all units in a target population from which a sample is selected. Its purpose is to provide a means for choosing the particular members of a target population to be interviewed. For a phone survey, this may be done in several ways: 

• Use samples of respondents from a previous survey that had a representative frame and a high response rate.
• Obtain a list of valid phone numbers from a mobile network operator (MNO) or private firm.
• Employ random digit dialing (RDD), which uses a software system to randomly generate the phone numbers that comprise a research project’s sample.

STEP 4:

CONSIDER THE SAMPLE SIZE

Sample size refers to the number of individuals included in the research. You can choose sample based on demographics such as age, gender, or physical location. To determine sample size, consider the following questions: 

• Goals and objectives: Will you project the survey results onto an entire demographic or a larger population? If so, sample size is very important and you need to ensure that it reflects the community as a whole. However, if you are trying to get a feel for preferences, sample size is not as critical.
• Precision level: The more accurate you need to be, the larger sample you will need and the more your sample must represent the overall population. MCSs monitoring the market are typically more interested in obtaining representative, rather than precise, results.
• Confidence level: This is the level of risk you are willing to tolerate, usually expressed as a percentage (e.g., 98 percent confident, 95 percent confident). The confidence percentage you choose has a big impact on the number of survey completions you will need to generate accurate survey results. It can increase the survey’s length and how many responses you need—which may translate to increased costs.
• Population variability: What variability exists in the target population (how similar or different is the population)? If you are surveying consumers on a broad topic, you may have lots of variations and will need a larger sample size to get the most accurate picture of the target population. However, if your survey is focused on a population with similar characteristics, variability will be lower and you can sample fewer people.
• Budget: What is the research budget? The need for statistical accuracy will require a larger sample size, which will increase costs. Costs are also affected if the target sample includes low-incidence segments or if response rates are low.

STEP 5:

DEVELOP SURVEY INSTRUMENT/QUESTIONNAIRE

Conducting a phone survey ideally means limiting survey duration to less than 20 minutes. Otherwise, the survey may suffer from data quality issues or respondent fatigue, which can lead to noncompletion.

STEP 6:

CONSIDER SURVEY DESIGN

Three commonly used survey design types include:

• One-time cross section. A cross section of the population is randomly selected and interviewed once, providing a snapshot in time.
• Repeated cross section. A cross section of the population is randomly selected and interviewed once. In the next period (e.g., after one year), another cross section of the population (which resembles the first sample in terms of population characteristics) is randomly selected and interviewed once, which can help monitor changes over time.
• Panel. The same households or individuals are interviewed multiple times at regular intervals.

STEP 7:

COMMISSION THE SURVEY

Develop terms of reference for a third party (e.g., research organization, survey company, consumer organization, consortium) to conduct actual phone surveys and analyze results. This external firm will run a pilot to test the effectiveness of your questionnaire (e.g., check if the survey takes too long or contains confusing language). The firm must have sound strategies to ensure an adequate response rate.

Back to top ↑

Limitations of this tool

• Unrepresentativeness. A mobile phone survey can be unrepresentative for several reasons, possibly skewing the data that is collected:

1. Thirty percent of the world’s population still lacks access to a mobile phone.
2. Fewer women own their own mobile phones as compared to men in many jurisdictions.
3. Phone ownership is typically lower in rural areas, where mobile signal coverage may be spotty.

• Data quality. Another challenge of a phone survey is quality of response, which may suffer since interviewers cannot use nonverbal cues (e.g., body language) to establish rapport and trust with respondents or to keep them engaged.

Back to top ↑

Other resources

• Remote Surveying in a Pandemic: Handbook (Innovations for Poverty Action 2020)
• Research for Effective COVID-19 Responses (RECOVR) - Phone Survey Methods (Innovations for Poverty Action) 
• Mobile Phone Panel Surveys in Developing Countries: A Practical Guide for Microdata Collection (World Bank 2016)
• Remote Survey Toolkit (60 Decibels 2020)
• Survey Questionnaire on DFS Consumer Risks (CGAP, Horus Development Finance, Innovation for Poverty Action, the Observatory of the Quality of Financial Services—OQSF of Cote d’Ivoire, and OQSF Senegal)

Back to top ↑