studying mode effects

Mode effects - the fact that respondents respond differently to a survey question, solely because of the mode of interviewing - are hard to study. This is because mode-effects interact with nonresponse effects. An Internet survey will atract different respondents than a telephone survey. Because of this, any differences that result from this survey, could be either due to differences in the type of respondents, or because of a mode effect.

There are three basic methods to study mode effects. The most common way to study mode-effects is:

1. to experimentally assign respondents to a survey mode. Then, the results from the survey are compared: the response rate, the demographic composition of the samples, and finally differences in the dependent variables. Sometimes, demographic differences between the samples are corrected using a multivariate model, like weighting. For an overview: see the results of this google scholar search. 

This type of design is popular, but in my view it has a great drawback: we know Internet samples and telephone surveys do only cover a part of the population. Landline telephone coverage is rapidly declining, while Internet use remains limited to about 80 per cent of the general population in Western countries. There are two alternative approaches, that deal with this issue.

2. One can make respondents switch modes during the interview. For example from the telephone to the Internet, or from face-to-face to paper-and-pencil. Although this approach sounds very simple, relatively few studies have been conducted in this manner. See Heerwegh (2009) for a nice example.
More experimental studies are defnitely welcome and necessary if we want to understand how problematic mode effects are.

3. The third way of studying mode effects relies on more sophisticated statistical modeling to separate different sources of survey error. The most relevant errors in mixed-surveys are coverage, nonresponse and response errors (i.e. the mode effect). Separating these could be done using a) validation data b) repeated measurements using the same or different modes or c) matching.
I am not aware of any mixed-mode studies that have used validation data to study mode effects, and as the mode effect occurs for attitudinal questions mainly, it is hard to find such data. The other two approaches both offer more practical ways of assessing mode effects. I will discuss both the modeling approach using longitudinal data and matching more extensively in next posts.