Science is sometimes really tricky, which makes writing about it even trickier. No real experiment exists apart from a huge background of assumptions, abstractions, caveats and complexities; the writer’s job is to find a strong narrative that is understandable with little or no prior knowledge, scans well, and catches the reader’s attention.
Recent research on physiological differences between liberal and conservative voters seems like a dream come true if you’re in need of a catchy press release, like this one from the National Science Foundation. I read the actual paper, and it says that people who answer more conservatively on a questionnaire about their politics tend also to have more pronounced “fight-or-flight” reactions to disturbing or surprising stimuli, as measured by skin conductance and startle response.
The press release tells a different story, and I believe that the NSF science writer told the wrong story. I attribute this partially to the politics of publicity, but mostly to the fact that science is actually very subtle, and hard to summarize.
“Those with the strongest eye or skin reactions to unexpected noises or threatening pictures such as a spider on a person’s eyeball tended to endorse political positions that were interpreted as protective of social groups,” said John Hibbing, professor of political science at UNL.
Hibbing defined those “protective policies” as more defense spending, more government resources directed at fighting terrorism and tighter controls on immigration. “People in this group are more willing to sacrifice a little of their privacy to protect the social unit,” Hibbing said. “On the other hand, the subjects who reacted less strongly to the stimuli were more likely to favor policies that protect privacy and encourage gun control.”
The first group believes the greatest threat to them and their communities comes from other people; they want to arm themselves and their government to defend against those threats. The latter group sees less threat from people and more threat from technology and inanimate objects such as guns that can kill or harm innocent people. They want policies in place to protect their individual privacy and safety: They oppose the death penalty and favor strong gun control. The study controlled for subjects’ gender, age and income.
What the NSF science writer has given us in the last paragraph is an interpretation of the results. This interpretation may come more or less from the researchers themselves, but it is still a story about what the experiment means, and that meaning is only possible with reference to existing ideas about the world. Notably, the researchers had decided that people could be categorized into more and less “socially protective” groups before the experiment was performed, and defined “socially protective” people as “threatened” by others. Based on this idea, they devised a survey to measure overall levels of “social protectiveness” by asking each subject 18 questions on topics such as military intervention, the death penalty, etc. The researchers then chose the well-studied physiological tests of skin conductance and startle response as proxies for feeling “threatened.”
With these choices, it seems possible to answer the headlining question “does being easily threatened lead to more conservative politics?” But this is not the question the experiment actually tests, which is “do people whose survey answers put them in the ‘more socially protective’ group have stronger skin conductance and startle responses?” This may seem like semantics, but it’s actually slight-of-hand, an expansion of the narrow results into much more interesting generalities.
I think the scientists who performed the reasearch understand these problems — indeed that is their job, and a close reading of the paper suggests that the authors fully appreciate the epistemological and methodological problems involved. Rather, the problem here is that the NSF press release completely fails to communicate these subtleties, leaving the reader with a false (but sensational) result.
In fact, no experiment of this type can be used to confirm or deny the hypothesis that “being easily threatened influences political position.” A test like this one can only demonstrate that certain kinds of startle responses and certain kinds of survey responses occur together. It is unreasonable for the press release to speculate on which is cause and which is effect; this is the classic correlation/causation error, as discussed in the paper:
Our data reveal a correlation between physiological responses to threat and political
attitudes but do not permit firm conclusions concerning the specific causal processes at work. Particular physiological responses to threat could cause the adoption of certain political attitudes, or the holding of particular political attitudes could cause people to respond in a certain physiological way to environmental threats, but neither of these seems probable. More likely is that physiological responses to generic threats and political attitudes on policies related to protecting the social order may both derive from a common source.
A deeper problem is the identification of “political attitudes” with survey results, and “environmental threats” with the disturbing pictures and startling sounds used in the experiment.
What, exactly, is a “political attitude?” We can devise a survey that categorizes people based on their beliefs, but we can always draw a line down the middle of any group of responses and put people on one side or the other; we might as well choose “wears jeans” versus “does not wear jeans” as our study variable. At issue here is whether the thing actually measured corresponds to our intuitive notions of political ideology, and whether those intuitive notions are even representative of reality. There are ways of answering such questions — such as looking at the variability of results when somewhat different questions are used, or doing cross-validation against other measures — but I do not see that the researchers have applied them.
Similarly, does skin conductance and startle response actually measure “physiological response to threat?” There seems to be good evidence for this, based on years of research into the “fight-or-flight” response of the sympathetic nervous system. However, we have no idea if this same biological response is in fact activated when, for example, a conservative voter thinks about terrorism. That is the sort of information we might learn by, say, hooking people up to physiological recorders and asking them to make political decision.
None of this means that the research is bad, or even that the line of questioning is unreasonable — I for one would be very interested to know if politics are partially physiological, perhaps even heritable (as other recent research claims to show.) The experiment as designed is an interesting step towards answering these questions, sort of a proof-of-concept test of the idea. But nothing has yet been answered. All we now know is that (some aspects of) physiology and (some aspects of) politics do seem to be realted, but exactly what is related and exactly what is the relationship? What do “politics” and “physiology” mean in a measurable sense, how do our pre-existing concepts of what these things are influence what we choose to measure, and what is the causality here? The research done to date creates far more questions than answers, and perhaps that is what the NSF press release most failed to communicate.