Asking Entirely Too Much From Testing

Matthew Nisbet has a comment about interpreting the public's understanding of science over at Framing Science
Consider what a split-ballot comparison in a 2004 University of Michigan survey revealed about the nature of responses to these long standing questions about evolution. In this survey experiment, one half of the sample was asked the following traditionally worded question:

True or false, human beings as we know them today, developed from earlier species of animals.

When asked this way, 42% answered true, a result that has been incredibly consistent across surveys since 1985.

The other half of the sample, however, was asked a slightly different version of the question:

True or false, according to the theory of evolution, human beings as we know them today, developed from earlier species of animals.

When asked this way, 74% answered true.

The implication is that context matters: Americans are not ignorant of what science says about human origins, in fact, as the second version of the question reveals, 3/4 of the public are familiar with the scientifically correct answer.

I believe it may be that the issue demonstrated by the two versions of the question is not one that allows a metric to be constructed.

What it does point out, is an important limitation in questioning people about anything at all. Questions are likely to be construed differently by a variety of responders due to differences in their perception of the semantic underpinnings of language and their personal learning history separate from the facts or belief systems. Time also plays a significant part. As people are introduced to a new concept, they put it in context with something they know. We understand by analogy which is limiting, but effective in helping make decisions.

But when someone asks us a question about it, the analog can either get in the way, or facilitate. Two people can be at a similar point on the path to comprehension, but react to a question differently.

As soon as you start asking questions using English, the variety of correct answers widens in the subject though the perception of the questioner does NOT. If a thought generated by the question can take you down more than one path and still be correct, the question will be less accurate in terms of what it beings back to the questioner. A second ambiguity may destroy the accuracy of a certain percentage of responses. And if the survey is littered with these, statistical correlation will not generate a warning.

The question's statement on lineage suggests to me that humans evolved in a linear way and it makes me uncomfortable to say that it is unambiguously correct. I just don't like it because I would never frame a statement using those words. Thus I am tempted to say "No." What is more, I may have never actually passed the first question through the filter if it had not contained the hint "according to the theory of evolution." That addition is a billboard telling my consciousness to stop and think about my understanding of the "theory of evolution" and wonder if the questioner is asking about current understand on natural selection.

If you fall back on definitions, as most scientists do, you fail to understand the nature of language and the nature of human cognition. The fact is that people generally don't parse questions carefully. Scientists do, but they aren't normal people in the sense I'm talking about. Scientists have trained themselves to pass any statement through a series of semantic filters and knowledge frameworks prior to admitting the question to consideration. Thus, Nisbet sees the question as presenting natural selection so that the appropriate answer will follow a particular logic pathway. I feel that it is not particularly accurate to say humans will subject any given question that includes sufficient ambiguity to any predictable logical pathway.

We may never know how many people fail to parse the question at all in terms of differentiating the framework of science versus faith because we desperately need to think these types of questions give us answers.

No comments:

Post a Comment