A New View on Survey Research

The most prestigious book on survey research of the last 5 years is the book of Tourangeau, Rips and Rasinski (2000) “The psychology of survey response”. In this book it is suggested  that people do not have a readily available response in their memory  but that they “sample”  believes from memory on the basis of which they create a judgement which is next expressed in a response.
The main idea expressed in the book is that the response to a survey question is created in a four steps procedure:
1. Comprehension of the question
2. Retrieval of information
3. Deriving a judgment
4. Formulating a response
The authors suggest that respondents do not have to go through these stages in this sequence, the respondents can go back and forth between these 4 stages.

During our meetings most participants agreed that people have no readily available responses in their memory and therefore create their response on the spot. But most of us also suggested that the process is much more complex and more automatic than the authors mentioned above have suggested. We came to the conclusion that we would suggest two stages. In the first stage the activation of the brain takes place. This does not happen after comprehension of the questions but starts immediately when the first word is heard.  Milton Lodge suggested on the basis of cognitive research that words immediately and automatically within 300 milliseconds activate affects (positive or negative) and these affects partially  guide the further processing of the information. His work in political psychology confirms these results of psychological experiments.
This discussion led to the flow chart (next page)  of the response process that is different from the traditional point of view. In this flowchart the assumption is made that people start with automatic processing of the information (question) provided to them. If this leads to an answer that they can accept then this answer has to be edited in the form that  is expected. If that is not possible or if the answer is not satisfactory to the respondent a more conscious process is started . This process can be done by deep processing or using heuristics depending on the motivation of the respondent. If this process leads to an acceptable answer this answer will be given otherwise this process has to start all over again.
The idea presented in the flow chart suggest that the first stage in the response process is much less orderly than Tourangeau et al. assume.

A phase that has not been indicated in this flowchart but has been emphasized during our meetings by Will Dijkstra and accepted by all other participants is the phase after an answer has been given. In case of a mismatch between the given answers and the answer categories specified the interviewer may raise questions about the answer. If that happens the respondent has also to go back in the process and consider possible alternative ways to come to a response.

This flowchart has been the starting point of our group for further research. A lot of attention has been given to the modelling of the  automatic part in the response process. In this context Frank van Overwalle was invited to inform us about the possibilities in this context of connectionist models and programs. Besides that Akos Munnich suggested the use of the ART system developed by Grossberg while Milton Lodge confronted us with the work that he had done together with his colleagues using the ACT-R system. Based on the different preesentations we decided that the ART and the ACT-R system were the most interesting possibilities.
Akos Munnich, Jaap Murre and Willem Saris have been working on the development of a simulation model for this automatic process based on the ART system. Milton Lodge and his colleagues looked at the same process using the ACT-R system. At the conference we have held in June the results of these activities  have been presented. It was clear that the work was not yet ready but it looked very promising. Therefore we have planned to continue our work on these models  and we will exchange input data for simulations in order to make a comparison of the results obtained with the different systems.
A different way to study these processes is by looking at the reaction times needed to answer questions. Stasja Draisma and Toni Chessa have looked at such data as well as Bregje Holleman. The former concentrated on factual questions about which the correct answer was known. They showed using a statistical analysis that people first try to find the right answer. The reaction times for people who gave the right answer were quite different for those of people who gave the right answer than for those who did not give the right answer. However if the answer was not found within a limited time a different approach was used by both groups and the response time functions were very similar for both groups. It was suggested that they relied on guessing in that second phase.
Bregje Holleman looked at reaction times for attitudinal data. She could separate the processing time and the answering time and could in this way look if for different question wordings the different response times were different. Continuing the topic of her dissertation with respect to forbid and allow formulations of questions she found no difference in response times for the processing of the questions what ever the formulation and the response given but she found that the forbid formulation combined with a yes answer took nearly twice as much time as all the other possibilities. This suggests that the processing is the same for the different questions and respondents but the answer process is different. She is now analyzing her data in the same way (together with Toni Chessa) as Stasja Draisma and Toni Chessa did, as described above.
These analyses are based on the original research of Jaap Murre and Toni Chessa who have developed a general model for forgetting and retrieval ; the so called “Memory Chain model”. They are working on a specific application of this model on the survey response behavior. As we have seen above in one occasion the application was already succesful.
Bergje Holleman and Akos Munnich have also designed a number of experiments which have been performed in Hungary in order to test the preference performance model of Akos Munnich. In the very beginning of our research Akos Munnich suggested that his model might be an explanation for the paradox of the  forbid and allow data collected by Bregje Holleman. Therefore tests of this hypothesis has been formulated and performed . In the near future we will know the results of this study

Milton Lodge and Stasja Draisma contributed also in a different way to the theme of the group by providing a different point of view. Milton presented on the basis of the literature and research of his own research group a strong case for the automatism in the reactions of people on stimuli like questions and even subliminal stimuli. Stasja made a case for the dual processing  based on different literature. In the flow chart it has been shown that we suppose that both processes occur but we expect that a large part of the process is automatic, much more then so far specified in the survey literature.

The second step in the response process is the transformation of the judgment into a response. This process was much less discussed except for the fact that we clearly see this as a separate process that requires special attention. William van der Veld and Willem Saris suggested that the answer process always generates errors in the answers. In order to estimate these answers different procedures have been suggested. In their contribution is has been suggested that the quasi simplex model is the wrong approach because it ignores a time specific component that is placed in the error term and as a consequence the measurement error variance is overestimate. This problem does not occur using the parallel test model. Both models are a specific model of their answer model which demonstrates and clarifies the phenomena found in both models.
Will Dijkstra looks at the last process in this chain of activities i.e. the communication of the respondent with the interviewer. He looks at the mismatches between the response categories and the given responses. He concluded that many mismatches can be conceived as conversational mismatches. It is assumed that:
– Conversational mismatches are quick responses, not preceded by considerations. If conversational mismatches are followed by considerations, they have the character of conversational elucidations, rather then complete relevant information on which the answer is based.
– Cognitive mismatches are preceded by considerations, or long reaction times or filled pauses, and serve as a way out of cognitive problems.
– Adequate answers to easy questions are given with short reaction times, without preceding considerations. Respondent however may provide considerations afterwards, to ‘justify’ the answer.
– Adequate answers to difficult questions are preceded by considerations or long reaction times.

Evaluation
This summary of the results shows that our research groups as a whole covered nicely the different parts of the response process. In the conference that was held at the end we could see very clearly that we had made a lot of progress and that the parts fit together very well. Therefore we have decided to try to make a book out of the different contribution. The final decision about this will be made in the end of this year when all participants are supposed to hand in their final paper for the book.
Due to the multidisciplinary composition the cooperation was very fruitful for example between Jaap Murre (psychologist), Akos Munnich (mathematician), Milton Lodge (politcal psychologist) and Willem Saris (survey research specialist) or between Stasja Draisma (survey specialist ) and Toni Chessa (mathematician) .
The composition of the group was also good for an other reason: the combination of younger and older scientists. We planned to start a new approach to survey research. We did not expect to finish this process in one year. This is a project for many years. Therefore it was essential to have younger scientists in our group. They should continue this project in the longer future. Only if that happens a theme group of this kind makes sense. This means that the selection of fellows should not only be based on publications and citations. In that case one get mainly older researchers. Also bright young researchers should have a chance to devote a year to research only. In combination with some senior researchers such a group will be much more successful than a group with only older researchers that have already published a lot.
Another comment I would like to make concerns a paradox in the selection of the members of a theme group. On the one hand the leader of the theme group makes a proposal for a group of researchers to work with. On the other hand the scientific committee of the NIAS selects people and can reject people of such a group and handicap such a group. Also the decisions concerning the presence of the different fellows during the year is not in the hand of the leader of the group. The composition of the group I have discussed in the section above. With respect to the grant period I would like to say the following. Our two foreign guests were only invited for the first period. But in that period we were mainly talking with each other to understand what the others had to say about the process. When we started with the new approach the foreigners were not present anymore. It was thanks to a grant of NWO that we were able to make a big jump ahead in our research. On the basis of this experience I would like to suggest to change this procedure and to determine the composition of the group and the grant periods with the project leader. This will certainly improve the results
In total I would like to say that the presence of our group at NIAS was very successful. We make a big step ahead. This was possible by the great facilities the NIAS offers and the hard work of the members of my group. I would like to thank NIAS for this opportunity and I would like to thank my group for the fact that they have made use of this opportunity in the right way.

Wim Saris, Coordinator, Amsterdam, June 30, 2004