Smart Survey Implementation WP2 deliverable M6: smart baseline stage.

Abstract

The goal of the WP2 ‘Methodology’ workpackage of the Smart Survey Implementation project (SSI) is to find out what general methodological elements trusted smart surveys should have so that they can be used in statistical production by European NSIs. Each task focuses on either an ‘opportunity’ or ‘threat’ that was identified in the TSS I framework and pilot recommendations for smart surveys. The four subtasks are: 1. The successful recruitment of participants for smart surveys. 2. Using machine learning to improve Human-Computer Interaction in smart surveys. 3. Respondent involvement and human-computer interaction in smart surveys. 4. Integrating smart surveys with traditional survey methods by estimating the mode effect. We refer to deliverable M6 (Review stage) for a discussion of learnings from past findings from projects conducted in the context of the European Statistical System and the wider academic context with regards to these four key challenges. In this deliverable, findings from the M6 deliverable are in places summarized when this is necessary to understand the current deliverable, in which we explain how we want to answer the challenges identified in the M6 deliverable within the Smart Survey Implementation project. Of central importance are several large and small field tests. A series of large field tests conducted in Q1 – Q4 of 2024 will focus on understanding the recruitment and the mode effect, and a series of small tests conducted throughout 2024 and early 2025 will focus on human-computer interaction and machine learning. The small tests at the core of testing UI/UX in task 2.3 have a focus on the implementation of smart surveys in the context of understanding household expenditure in the Household Budget Survey (HBS) and the Time Use Survey (TUS). The goal of these smaller experiments is to technically test the Machine Learning standards developed in task 2.2, to test the HCI features of smart surveys in task 2.3, and to technically test some of the microservices developed in Workpackage 3 that are completed by the time these tests are scheduled. The small tests are qualitative, and they will use as many respondents as necessary before saturation is reached (e.g., n=~20). Samples here are taken from existing surveys (follow-ups of respondents from other surveys), online access panels or other volunteers. The goal here is not to draw inferences to the general population but to include a diverse set of respondents (e.g., in terms of age, internet experience), so that the Human Computer Interaction is tested for different types of potential respondents. Tests are foreseen in all countries in the consortium and carried out throughout the project, scheduled in close alignment with WP 3 (microservices). Chapter 3 of this deliverable describes the small tests in some detail, as well as the test protocols used to test the respondent involvement and human-computer interaction of smart surveys The large tests aim to answer the question of how respondents can be successfully be recruited into smart surveys (task 2.1) and how to integrate smart surveys with traditional surveys (task 2.4). The large field tests will be conducted in Norway (HBS), France (TUS & HBS), Belgium (TUS), Germany (HBS), Italy (TUS) and the Netherlands (HBS). Norway and France will use a smart survey app which was self-developed, Germany, Belgium and Italy will use MOTUS, and the Netherlands will use the HBS platform. All countries will use the general population as the target population and draw fresh samples to conduct the field test following a general design, where some key elements of the field tests are shared across the countries. Respondents are recruited using an offline method (e.g. recruitment via interviewers or postal mail) and are based on large probability samples. This allows data from multiple countries to be pooled in the analysis, increasing statistical power and allowing for analyses into recruitment effects for smaller subgroups in the general population. At the same time, it also allows for the comparison of country-level differences in, for example, the success of particular recruitment strategies. The exact design of the experiments around communication materials and recruitment (task 2.1) is worked out in Chapter 1 of this deliverable. Chapter 4 focuses on the design and experiments carried out to inform the mode effect of smart surveys (task 2.4). Chapter 2 follows closely on the M6 deliverable and focuses on the role of Machine Learning in processing smart data that is used in the field tests for the Time Use Survey (geolocation data) and Household Budget Survey (pictures of receipt). Chapter 2 bridges the more technical work of WP 3, in which microservices are developed, to the work of task 2.3 (human computer interaction). This deliverable presents partly work in-progress on how sensor data are being processed in the microservice using machine learning and presented back to the respondent. As the microservices are at this stage (June 2024) still under development, a final test of how good the machine learning models work and how processed data can be fed back to the respondent will be part of the M24 deliverable, in which information from the small and large field tests can be used to evaluate the quality of the machine learning models used in smart surveys for TUS and HBS, and the end-to-end process as a whole.

Publication
Eurostat SSI WP deliverable

Related