Asian Journal for Public Opinion Research

Center for Asian Public Opinion Research & Collaboration Initiative (충남대학교 아시아여론조사학회)
권호 표지
DOI QR코드

DOI QR Code

Confirming the Continued Representativeness of an Online/Telephone Panel Using Equivalence Testing

  • Received : 2021.05.06
  • Accepted : 2021.05.10
  • Published : 2021.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

Decreasing response rates to traditional survey methods, like face-to-face and telephone interviews, have led survey practitioners around the world to seek new ways of conducting surveys in recent years." The COVID-19 pandemic exacerbated this problem because it made conducting face-to-face interviews even more difficult than before. For example, it made conducting face-to-face surveys infeasible in 2020 in South Korea, and so the Korean Academic Multimode Open Survey (KAMOS) was unable to conduct a planned face-to-face survey to recruit new panel members. The entire 8,514-member panel, established via two-stage probability-based sampling from 2016 to 2019, was invited to take three online/telephone surveys in 2020. Of these panel members, 1,352 responded to at least one survey in 2020. To test to what extent the panel remained representative of the adult South Korean population, we compared the two groups of panel members: those who responded to at least one survey in 2020 and those who did not. After weighting both groups on the basis of age, sex, and geographical area, we analyzed their responses to some of the questions that were asked during multiple rounds of the face-to-face panel-recruiting interviews. Using Cohen's d for survey items that could be analyzed numerically and Cramér's V for categorical items, we were able to conclude that the respondents to the 2020 surveys were equivalent to the non-respondents in terms of both demographics and in the answers they originally gave to substantive questions on a variety of topics related to social science or public opinion research, including questions about quality of life, societal issue, and politics (Cohen's d items <0.2, 95% CI; Cramér's V items <0.1, 95% CI). This analysis may provide a model for others who wish to test the continued representativeness of their panel or who would like to use a different survey mode or change some other aspect of their methodology and test whether it is equivalent to their former methodology. Our success in building a panel that retained its representativeness may be useful to those in other countries where face-to-face surveys had previously been the norm but are becoming increasingly difficult to conduct.

Keywords

Acknowledgement

This work was supported by the research fund of Chungnam National University.

References

  1. Ali, S. H., Foreman, J., Capasso, A., Jones, A. M., Tozan, Y., & DiClemente, R. J. (2020). Social media as a recruitment platform for a nationwide online survey of COVID-19 knowledge, beliefs, and practices in the United States: Methodology and feasibility analysis. BMC Medical Research Methodology, 20, 116. https://doi.org/10.1186/s12874-020-01011-0
  2. Barkay, N., Cobb, C. Eilat, R., Galili, T., Haimovich, D., LaRocca, S., Morris, K., & Sarig, T. (2020). Weights and Methodology Brief for the COVID-19 Symptom Survey by University of Maryland and Carnegie Mellon University, in partnership with Facebook. https://arxiv.org/pdf/2009.14675.pdf
  3. Bates, N., & Zamadics, J. (2021). COVID-19 infection rates and propensity to self-respond in the 2020 U.S. decennial census. Survey Practice 14(1). https://doi.org/10.29115/SP-2021-0002.
  4. Bautista, A. Jr., Balibrea, D., & Bleza D. G. (2020). Knowledge, attitude, and practice toward the coronavirus disease (COVID-19) outbreak among selected employed people in the National Capital Region, Philippines. AJPOR, 8(3), 324-350. https://doi.org/10.15206/ajpor.2020.8.3.324
  5. Cho, S. K., LoCascio, S. P., Lee, K.-O., Jang, D.-H., & Lee, J. M. (2017). Testing the Representativeness of a Multimode Survey in South Korea: Results from KAMOS. Asian Journal for Public Opinion Research, 4(2), 73-87. https://doi.org/10.15206/ajpor.2017.4.2.73
  6. CoViD-19 Impact Lab. (2020). Datasets. https://covid-19-impact-lab.readthedocs.io/en/latest/datasets.html
  7. Delphi Survey Results (2021). https://delphi.cmu.edu/covidcast/survey-results/?date=20210417
  8. Dillman, D. A. (2017). The promise and challenge of pushing respondents to the Web in mixed-mode surveys. Survey Methodology. Statistics Canada, Catalogue No. 12-001-X, 43(1). http://www.statcan.gc.ca/pub/12-001-x/2017001/article/14836-eng/pdf
  9. Dutwin, D., & Lavrakas, P. (2016). Trends in telephone outcomes, 2008 - 2015. Survey Practice, 9(3). https://doi.org/10.29115/SP-2016-0017
  10. Kapteyn, A., Angrisani, M., Bennett, D., Bruine de Buin, W., Darling, J., Gutsche, T., Liu, Y., Meijer, E., Perez-Arce, F., Schaner, Sm., Thomas, K., & Weerman, B. (2020). Tracking the effect of the COVID-19 pandemic on the lives of American households. Survey Research Methods, 14(2), 179-186. https://doi.org/10.18148/srm/2020.v14i2.7737
  11. Kotrlik, J. W., Williams, H. A., & Jabor, M. K. (2011). Reporting and interpreting effect size in quantitative agricultural education research. Journal of Agricultural Education, 52(1), 132-142. https://doi.org/10.5032/jae.2011.01132
  12. Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259-269. https://doi.org/10.1177/2515245918770963
  13. Lewis, I. M., Watson, B. C., & White, K. M. (2009). Internet versus paper-and-pencil survey methods in psychological experiments: Equivalence testing of participant responses to health-related messages. Australian Journal of Psychology, 61(2), 107-116. https://doi.org/10.1080/00049530802105865
  14. Lima-Costa, M. F., Macinko, J., Bof de Andrade, F., Borges de Souza, P. R. Jr., Leite de Vasoncellos, M. T., Messias de Oliveira, C. (2020). Cad. Saude Publica 36(Suppl 3). https://doi.org/10.1590/0102-311X00183120
  15. Liu, H., Cella, D., Gershon, R., Shen, J., Morales, L. S., Riley, W., & Hays, R. D. (2010). Representativeness of the PROMIS Internet panel. Journal of Clinical Epidemiology 63(11), 1169-1178. https://doi.org/10.1016/j.jclinepi.2009.11.021
  16. Oliver, N., Barber, X., Roomp, K., & Roomp, K. (2020). The COVID19Impact survey: Assessing the Pulse of the COVID-19 pandemic in Spain via 24 questions. Journal of Medical Internet Research, 22(9). https://doi.org/10.2196/21319
  17. Pew Research Center. (2019). After brief plateau, telephone survey response rates have fallen again. https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/ft_19-02-27_atp1_afterbriefplateau_2/
  18. Pramiyanti, A., Mayangsari, I. D., Nuraeni, R., & Darin Firdaus, Y. (2020). Public perception on transparency and trust in government information released during the COVID-19 pandemic. Asian Journal for Public Opinion Research, 8(3), 351-376. https://doi.org/10.15206/ajpor.2020.8.3.351
  19. Shawilowsky, S. S. (2009). New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 8(3). 597-599. https://doi.org/10.22237/jmasm/1257035100
  20. UN. (2020). Monitoring the state of statistical operations under the COVID-19 pandemic. https://unstats.un.org/unsd/covid19-response/covid19-nso-survey-report.pdf
  21. Van der Velden, P. G., Contino, C., Das, M., van Loon, P., & Bosmans, M. W. G. (2020). Anxiety and depression symptoms, and lack of emotional support among the general population before and during the COVID-19 pandemic. A prospective national study on prevalence and risk factors. Journal of Affective Disorders, 277, 540-548. https://doi.org/10.1016/j.jad.2020.08.026
  22. Van Tilburg, T. G., Steinmetz, S., Stolte, E., van der Roest, H., & de Vries, D. H. (2020). Loneliness and mental health during the COVID-19 pandemic: A study among Dutch older adults. Journals of Gerontology: Social Sciences. https://doi.org/10.1093/geronb/gbaa111
  23. Weigold, A., Weigold I. K., & Russell, E. J. (2013). Examination of the equivalence of self-report survey-based paper-and-pencil and Internet data collection methods. Psychological Methods, 18(1), 53-70. https://doi.org/10.1037/a0031607
  24. Wuensch. K. L. (2012). Using SPSS to obtain a confidence interval for Cohen's d. http://core.ecu.edu/psyc/wuenschk/SPSS/CI-d-SPSS.pdf
  25. Yang, W.-Y., & Tsai, C. (2020). Democratic values, collective security and privacy: Taiwan People's response to COVID-19. AJPOR 8(3), 2020, 222-245. https://doi.org/10.15206/ajpor.2020.8.3.222
  26. Yuncg, J. Q. X., Cheong, A. W. H., Seng, A. I. N., & Li, K. J. (2020). Towards a dialogic approach: Crisis communications and public reactions in the world's most densely populated city to tackle COVID-19. AJPOR, 8(3), 265-296. https://doi.org/10.15206/ajpor.2020.8.3.265