Does mobile phone survey method matter? Reliability of computer-assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low and middle income countries

George Pariyo, Abigail R. Greenleaf, Dustin Gibson, Joseph Ali, Hannah Selig, Alain B Labrique, Gulam Muhammed Al Kibria, Iqbal Ansary Khan, Honorati Masanja, Meerjady Sabrina Flora, Saifuddin Ahmed, Adnan A. Hyder

Research output: Contribution to journalArticle

Abstract

Introduction Increased mobile phone subscribership in low- and middle-income countries (LMICs) provides novel opportunities to track population health. The objective of this study was to examine reliability of data in comparing participant responses collected using two mobile phone survey (MPS) delivery modalities, computer assisted telephone interviews (CATI) and interactive voice response (IVR) in Bangladesh (BGD) and Tanzania (TZA). Methods Using a cross-over design, we used random digit dialing (RDD) to call randomly generated mobile phone numbers and recruit survey participants to receive either a CATI or IVR survey on non-communicable disease (NCD) risk factors, followed 7 days later by the survey mode not received during first contact; either IVR or CATI. Respondents who received the first survey were designated as first contact (FC) and those who consented to being called a second time and subsequently answered the call were designated as follow-up (FU). We used the same questionnaire for both contacts, with response options modified to suit the delivery mode. Reliability of responses was analyzed using the Cohen’s kappa statistic for percent agreement between two modes. Results Self-reported data on demographic characteristics and NCD behavioral risk factors were collected from 482 (CATI-FC) and 653 (IVR-FC) age-eligible and consenting respondents in BGD, and from 387 (CATI-FC) and 674 (IVR-FC) respondents in TZA respectively. Survey follow-up rates were 30.7% (n = 482) for IVR-FU and 53.8% (n = 653) for CATI-FU in BGD; and 42.4% (n = 387) for IVR-FU and 49.9% (n = 674) for CATI-FU in TZA respectively. Overall, there was high consistency between delivery modalities for alcohol consumption in the past 30 days in both countries (kappa = 0.64 for CATI!IVR (BGD), kappa = 0.54 for IVR!CATI (BGD); kappa = 0.66 for CATI!IVR (TZA), kappa = 0.76 for IVR!CATI (TZA)), and current smoking (kappa = 0.68 for CATI!IVR (BGD), kappa = 0.69 for IVR!CATI (BGD); kappa = 0.39 for CATI!IVR (TZA), kappa = 0.50 for IVR!CATI (TZA)). There was moderate to substantial consistency in both countries for history of checking for hypertension and diabetes with kappa statistics ranging from 0.43 to 0.67. There was generally lower consistency in both countries for physical activity (vigorous and moderate) with kappa statistics ranging from 0.10 to 0.41, weekly fruit and vegetable with kappa ranging from 0.08 to 0.45, consumption of foods high in salt and efforts to limit salt with kappa generally below 0.3. Conclusions The study found that when respondents are re-interviewed, the reliability of answers to most demographic and NCD variables is similar whether starting with CATI or IVR. The study underscores the need for caution when selecting questions for mobile phone surveys. Careful design can help ensure clarity of questions to minimize cognitive burden for respondents, many of whom may not have prior experience in taking automated surveys. Further research should explore possible differences and determinants of survey reliability between delivery modes and ideally compare both IVR and CATI surveys to in-person face-to-face interviews. In addition, research is needed to better understand factors that influence survey cooperation, completion, refusal and attrition rates across populations and contexts.

Original languageEnglish (US)
Article numbere0214450
JournalPloS one
Volume14
Issue number4
DOIs
StatePublished - Apr 1 2019

Fingerprint

noninfectious diseases
Cell Phones
Mobile phones
Telephone
interviews
income
risk factors
Interviews
Bangladesh
Tanzania
methodology
Surveys and Questionnaires
statistics
Statistics
Salts
Demography
salts

ASJC Scopus subject areas

  • Biochemistry, Genetics and Molecular Biology(all)
  • Agricultural and Biological Sciences(all)

Cite this

Does mobile phone survey method matter? Reliability of computer-assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low and middle income countries. / Pariyo, George; Greenleaf, Abigail R.; Gibson, Dustin; Ali, Joseph; Selig, Hannah; Labrique, Alain B; Kibria, Gulam Muhammed Al; Khan, Iqbal Ansary; Masanja, Honorati; Flora, Meerjady Sabrina; Ahmed, Saifuddin; Hyder, Adnan A.

In: PloS one, Vol. 14, No. 4, e0214450, 01.04.2019.

Research output: Contribution to journalArticle

@article{a6aed93383ed45f68a9427b2cb403239,
title = "Does mobile phone survey method matter? Reliability of computer-assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low and middle income countries",
abstract = "Introduction Increased mobile phone subscribership in low- and middle-income countries (LMICs) provides novel opportunities to track population health. The objective of this study was to examine reliability of data in comparing participant responses collected using two mobile phone survey (MPS) delivery modalities, computer assisted telephone interviews (CATI) and interactive voice response (IVR) in Bangladesh (BGD) and Tanzania (TZA). Methods Using a cross-over design, we used random digit dialing (RDD) to call randomly generated mobile phone numbers and recruit survey participants to receive either a CATI or IVR survey on non-communicable disease (NCD) risk factors, followed 7 days later by the survey mode not received during first contact; either IVR or CATI. Respondents who received the first survey were designated as first contact (FC) and those who consented to being called a second time and subsequently answered the call were designated as follow-up (FU). We used the same questionnaire for both contacts, with response options modified to suit the delivery mode. Reliability of responses was analyzed using the Cohen’s kappa statistic for percent agreement between two modes. Results Self-reported data on demographic characteristics and NCD behavioral risk factors were collected from 482 (CATI-FC) and 653 (IVR-FC) age-eligible and consenting respondents in BGD, and from 387 (CATI-FC) and 674 (IVR-FC) respondents in TZA respectively. Survey follow-up rates were 30.7{\%} (n = 482) for IVR-FU and 53.8{\%} (n = 653) for CATI-FU in BGD; and 42.4{\%} (n = 387) for IVR-FU and 49.9{\%} (n = 674) for CATI-FU in TZA respectively. Overall, there was high consistency between delivery modalities for alcohol consumption in the past 30 days in both countries (kappa = 0.64 for CATI!IVR (BGD), kappa = 0.54 for IVR!CATI (BGD); kappa = 0.66 for CATI!IVR (TZA), kappa = 0.76 for IVR!CATI (TZA)), and current smoking (kappa = 0.68 for CATI!IVR (BGD), kappa = 0.69 for IVR!CATI (BGD); kappa = 0.39 for CATI!IVR (TZA), kappa = 0.50 for IVR!CATI (TZA)). There was moderate to substantial consistency in both countries for history of checking for hypertension and diabetes with kappa statistics ranging from 0.43 to 0.67. There was generally lower consistency in both countries for physical activity (vigorous and moderate) with kappa statistics ranging from 0.10 to 0.41, weekly fruit and vegetable with kappa ranging from 0.08 to 0.45, consumption of foods high in salt and efforts to limit salt with kappa generally below 0.3. Conclusions The study found that when respondents are re-interviewed, the reliability of answers to most demographic and NCD variables is similar whether starting with CATI or IVR. The study underscores the need for caution when selecting questions for mobile phone surveys. Careful design can help ensure clarity of questions to minimize cognitive burden for respondents, many of whom may not have prior experience in taking automated surveys. Further research should explore possible differences and determinants of survey reliability between delivery modes and ideally compare both IVR and CATI surveys to in-person face-to-face interviews. In addition, research is needed to better understand factors that influence survey cooperation, completion, refusal and attrition rates across populations and contexts.",
author = "George Pariyo and Greenleaf, {Abigail R.} and Dustin Gibson and Joseph Ali and Hannah Selig and Labrique, {Alain B} and Kibria, {Gulam Muhammed Al} and Khan, {Iqbal Ansary} and Honorati Masanja and Flora, {Meerjady Sabrina} and Saifuddin Ahmed and Hyder, {Adnan A.}",
year = "2019",
month = "4",
day = "1",
doi = "10.1371/journal.pone.0214450",
language = "English (US)",
volume = "14",
journal = "PLoS One",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "4",

}

TY - JOUR

T1 - Does mobile phone survey method matter? Reliability of computer-assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low and middle income countries

AU - Pariyo, George

AU - Greenleaf, Abigail R.

AU - Gibson, Dustin

AU - Ali, Joseph

AU - Selig, Hannah

AU - Labrique, Alain B

AU - Kibria, Gulam Muhammed Al

AU - Khan, Iqbal Ansary

AU - Masanja, Honorati

AU - Flora, Meerjady Sabrina

AU - Ahmed, Saifuddin

AU - Hyder, Adnan A.

PY - 2019/4/1

Y1 - 2019/4/1

N2 - Introduction Increased mobile phone subscribership in low- and middle-income countries (LMICs) provides novel opportunities to track population health. The objective of this study was to examine reliability of data in comparing participant responses collected using two mobile phone survey (MPS) delivery modalities, computer assisted telephone interviews (CATI) and interactive voice response (IVR) in Bangladesh (BGD) and Tanzania (TZA). Methods Using a cross-over design, we used random digit dialing (RDD) to call randomly generated mobile phone numbers and recruit survey participants to receive either a CATI or IVR survey on non-communicable disease (NCD) risk factors, followed 7 days later by the survey mode not received during first contact; either IVR or CATI. Respondents who received the first survey were designated as first contact (FC) and those who consented to being called a second time and subsequently answered the call were designated as follow-up (FU). We used the same questionnaire for both contacts, with response options modified to suit the delivery mode. Reliability of responses was analyzed using the Cohen’s kappa statistic for percent agreement between two modes. Results Self-reported data on demographic characteristics and NCD behavioral risk factors were collected from 482 (CATI-FC) and 653 (IVR-FC) age-eligible and consenting respondents in BGD, and from 387 (CATI-FC) and 674 (IVR-FC) respondents in TZA respectively. Survey follow-up rates were 30.7% (n = 482) for IVR-FU and 53.8% (n = 653) for CATI-FU in BGD; and 42.4% (n = 387) for IVR-FU and 49.9% (n = 674) for CATI-FU in TZA respectively. Overall, there was high consistency between delivery modalities for alcohol consumption in the past 30 days in both countries (kappa = 0.64 for CATI!IVR (BGD), kappa = 0.54 for IVR!CATI (BGD); kappa = 0.66 for CATI!IVR (TZA), kappa = 0.76 for IVR!CATI (TZA)), and current smoking (kappa = 0.68 for CATI!IVR (BGD), kappa = 0.69 for IVR!CATI (BGD); kappa = 0.39 for CATI!IVR (TZA), kappa = 0.50 for IVR!CATI (TZA)). There was moderate to substantial consistency in both countries for history of checking for hypertension and diabetes with kappa statistics ranging from 0.43 to 0.67. There was generally lower consistency in both countries for physical activity (vigorous and moderate) with kappa statistics ranging from 0.10 to 0.41, weekly fruit and vegetable with kappa ranging from 0.08 to 0.45, consumption of foods high in salt and efforts to limit salt with kappa generally below 0.3. Conclusions The study found that when respondents are re-interviewed, the reliability of answers to most demographic and NCD variables is similar whether starting with CATI or IVR. The study underscores the need for caution when selecting questions for mobile phone surveys. Careful design can help ensure clarity of questions to minimize cognitive burden for respondents, many of whom may not have prior experience in taking automated surveys. Further research should explore possible differences and determinants of survey reliability between delivery modes and ideally compare both IVR and CATI surveys to in-person face-to-face interviews. In addition, research is needed to better understand factors that influence survey cooperation, completion, refusal and attrition rates across populations and contexts.

AB - Introduction Increased mobile phone subscribership in low- and middle-income countries (LMICs) provides novel opportunities to track population health. The objective of this study was to examine reliability of data in comparing participant responses collected using two mobile phone survey (MPS) delivery modalities, computer assisted telephone interviews (CATI) and interactive voice response (IVR) in Bangladesh (BGD) and Tanzania (TZA). Methods Using a cross-over design, we used random digit dialing (RDD) to call randomly generated mobile phone numbers and recruit survey participants to receive either a CATI or IVR survey on non-communicable disease (NCD) risk factors, followed 7 days later by the survey mode not received during first contact; either IVR or CATI. Respondents who received the first survey were designated as first contact (FC) and those who consented to being called a second time and subsequently answered the call were designated as follow-up (FU). We used the same questionnaire for both contacts, with response options modified to suit the delivery mode. Reliability of responses was analyzed using the Cohen’s kappa statistic for percent agreement between two modes. Results Self-reported data on demographic characteristics and NCD behavioral risk factors were collected from 482 (CATI-FC) and 653 (IVR-FC) age-eligible and consenting respondents in BGD, and from 387 (CATI-FC) and 674 (IVR-FC) respondents in TZA respectively. Survey follow-up rates were 30.7% (n = 482) for IVR-FU and 53.8% (n = 653) for CATI-FU in BGD; and 42.4% (n = 387) for IVR-FU and 49.9% (n = 674) for CATI-FU in TZA respectively. Overall, there was high consistency between delivery modalities for alcohol consumption in the past 30 days in both countries (kappa = 0.64 for CATI!IVR (BGD), kappa = 0.54 for IVR!CATI (BGD); kappa = 0.66 for CATI!IVR (TZA), kappa = 0.76 for IVR!CATI (TZA)), and current smoking (kappa = 0.68 for CATI!IVR (BGD), kappa = 0.69 for IVR!CATI (BGD); kappa = 0.39 for CATI!IVR (TZA), kappa = 0.50 for IVR!CATI (TZA)). There was moderate to substantial consistency in both countries for history of checking for hypertension and diabetes with kappa statistics ranging from 0.43 to 0.67. There was generally lower consistency in both countries for physical activity (vigorous and moderate) with kappa statistics ranging from 0.10 to 0.41, weekly fruit and vegetable with kappa ranging from 0.08 to 0.45, consumption of foods high in salt and efforts to limit salt with kappa generally below 0.3. Conclusions The study found that when respondents are re-interviewed, the reliability of answers to most demographic and NCD variables is similar whether starting with CATI or IVR. The study underscores the need for caution when selecting questions for mobile phone surveys. Careful design can help ensure clarity of questions to minimize cognitive burden for respondents, many of whom may not have prior experience in taking automated surveys. Further research should explore possible differences and determinants of survey reliability between delivery modes and ideally compare both IVR and CATI surveys to in-person face-to-face interviews. In addition, research is needed to better understand factors that influence survey cooperation, completion, refusal and attrition rates across populations and contexts.

UR - http://www.scopus.com/inward/record.url?scp=85064176822&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064176822&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0214450

DO - 10.1371/journal.pone.0214450

M3 - Article

C2 - 30969975

AN - SCOPUS:85064176822

VL - 14

JO - PLoS One

JF - PLoS One

SN - 1932-6203

IS - 4

M1 - e0214450

ER -