Adaptive designs: 4 ways to improve panel surveys

This is a follow-up on why I think panel surveys need to adapt their data collection strategies to target individual respondents . Let me first note that apart from limiting nonresponse error, there are other reasons why we would want to do this. We can limit survey costs by using expensive survey resources only for people who need them.
A focus on nonresponse alone can be too limited. For example: imagine we want to measure our respondents’ health. We can maybe do this cheaply by using web interviews, and then try to limit nonresponse error by using interviewers to convert initial nonrespondents. But what about measurement? If we use web surveys we largely have to rely on self-reports on the respondents’ health. But if we use interviewers for everyone and do a Face-to-Face survey among all our respondents, we can use the interviewers to obtain objective health measures for respondents. These objective measurements could be much better than the self-reports. So face-to-face interviews may not be ‘worth’ the cost if we look at nonresponse alone, but if we also include the effects on measurement, it may be a viable interview option, if we reduce the sampling size.

I think a focus just on any one type of survey error can have adverse effects, and it is Total Survey Error as well as costs we need to keep in mind. Having said this, I really believe nonresponse errors in panel surveys are a huge problem. What could we do (and have others done)?

1. Targeted mode. Some respondents are easy to reach in all modes, and some are difficult in all modes. There is also a ‘middle’ group, who may participate in some modes, but not others. I, for example dislike web surveys (because I get so many), but appreciate mail surveys (because I never get them). In a panel survey, we can ask respondents about mode preferences. Some studies
   ( here , here ) have found that stated mode preferences are not very predictive of response in that mode in the next wave, as people indicate to prefer the mode they are interviewed in. This means we probably need a better model than just ‘mode preference’ to make this work.

Probably wants a different survey mode next time.

2. Targeted incentives. We know some respondents are ‘in it’ for the money, or at least are sensitive to offers of incentives. In panel surveys, we can learn quickly about this by experimenting with amounts both between and/or within persons. For example, does it help to offer higher incentives to hard-to-reach respondents? Does that help just once, or is there a persistent effect? It may be unethical to offer higher incentives to just hard-to-reach respondents, as we then put a premium on bad respondent behavior. We could however use different metrics for deciding whom to offer the incentive. Nonresponse bias is a much better indicator for example. There is not too much we know about how to do this, although there is a nice example here.

3. Targeted advance letters. We know quite a lot  about the effects different types of advance letters have on subsequent response. Young people can for example be targeted with a letter with a flashy lay-out and short bites of information of the study, while older people may prefer a more ‘classic’ letter with more extensive information about the study setup and results.
The effects of targeted letters on response in panel surveys are often small, and only present for specific subgroups. See here , and here for two examples. Still, this type of targeting costs little, perhaps we can find bigger effects if we know what groups to target with what message. As with other targeting methods, we need a combination of data mining and experimentation to develop knowledge about this.

4. Targeted tracking. I am not aware of any survey doing targeted tracking. Tracking is done during fieldwork. Respondents who are not located by an interviewer (or advance letter which bounce), are sent back to the study coordinating team, after which tracking methods are used to locate the respondent at an alternative address. From the literature we know that it is mainly people who move house who need tracking. If we can successfully predict the likelihood to move, we could potentially save time (and money) in fieldwork, by putting cases into preventive tracking. We could also potentially use a targeted order of tracking procedures, as James Wagner has done.

-