Design of Surveys

Whew! What a crapstorm that article and subsequent post caused, didn't it? Anyway, let's back away from issues on which some really wish to disagree (and are even downright nasty about) and talk about something much more tame by comparison. First, however, I need to clarify something that my friend Reuben brought up. The term "white-hot stupid" is not a derogatory term against White people, not any more than "black hole" is a racist epithet. "White-hot stupid" refers to stupid that burns so hot it's white in color. "Burning stupid" comes from, among other things, this cartoon:

As you can see, someone told the cartoon man something so stupid that his head caught on fire. If the comment were to be more stupid than most, it would burn "white-hot" because of the way incandescence works. It's science. Some of the people threatening to go to the town hall meeting to attack me personally plan to bring up that comment about "white-hot stupid" as an example of me being racist. (Seriously, anything to turn the tables.) I look forward to all of us having a good laugh.

This brings me straight into surveys because surveys seem to guide a lot of policy in the United States in particular and the world in general. Surveys can be very misleading in their questions and give exactly the answers that the party or parties that commissioned the survey want to hear. (For example, this survey on immigration. Check it out. The pictures are photoshop gold.) Other surveys have to stick to a high standard of imparciality and be subject to review by an Institutional Review Board (IRB). (For example, this survey also on immigration.) And current events greatly sway the results of surveys from one moment to another. Let's look at what makes a good survey and what doesn't, all after the jump...

2 out of 3 kids like water AND trampolines

WOULD YOU LIKE TO TAKE A SURVEY?
Let's get one thing clear right off the bat. A survey is at the bottom of the pile when it comes to reliability and reproducibility. Randomized clinical trials are at the top. Case-control studies are second. Surveys are tied with meta-analyzes for the quality of information they bring to the scientific table. Are they unreliable? No, but their reliability is lesser than an actual study.

FOR EXAMPLE
If I came to you and said that 98% of Americans want all Black people to leave the country, please, please ask to see the survey design. This is not only because over 12% of Americans are Black (so some of them themselves would be asking to leave the country). It's because hardly any survey shows a near-100% agreement among respondents. Seriously, if you see something above 95%, no matter how sensible the question is (do you agree that the sky is blue?), be very, very skeptical. The makeup of the United States is such that opinions vary wildly and absolute consensus is incredibly rare.

THE IMMIGRATION SURVEY
Did you get curious and head over to check it out? Did you read the questions? Here are some examples:

When a person enters the US illegally, his first act within the US is to break the law. If he stays, he is breaking additional laws. Has this law breaking, non-citizen earned the rights and privileges provided to US citizens by the US Constitution?

Here's another gem:

On many jobs people need to communicate and interact with their coworkers in order to be productive and do their jobs well. Should being able to speak English be a requirement for getting hired to work in those workplaces?

People who read this blog for self-edification and an honest discussion on issues of science and epidemiology (and other things) will know how one-sided those questions are. The designer of that survey is not looking for honest answers. The results of their survey will never stand up to scrutiny. If you don't want to be laughed out of a scientific symposium, please don't use questions like the ones in that survey.

WHAT TO ASK?
Your questions need to be non-leading, which is hard to do if you're working for someone with an agenda. Aside from that, run your questions through a number of people from different backgrounds. (Your IRB will offer tons of guidance about this.) If you need to translate the survey, or are giving it to people who speak different languages, make sure that the translations correlate with each other. If the surveys will not be translated, but the people doing the asking will be multilingual, make sure they are properly trained. (Remember proficiency bias?)

If you want to know whether or not people agree with illegal immigration, for example, then you could ask something like, "Do you agree with illegal immigration?" See, nice and simple. You don't need to go into a diatribe about how illegal immigrants are "law breaking, non-citizen(s)". Because, even if someone agrees with illegal immigration, framing the question with a biased angle makes people not want to respond instead of answering "YES". Another way of looking at it is to ask questions for which only one YES or one NO are required.

For example, you don't want to ask, "Do you believe or have ever believed that deep fried donuts and/or turkey sandwiches are the primary or secondary cause of heart attacks in people over 60 who may or may not take baby aspirin?" Yes, no, yes, maybe... WHAT?! Keep it simple is all I'm saying.

WHY SIMPLE?
Beside the obvious, simple questions allow for better stratification of the responses, even if it makes the survey longer than you want. With the ability to separate responses by YES/NO on many levels, you're able to accurately interpret what your respondents say. For example, you can say that 57% of respondents who agreed that the sky was blue also answered YES when asked if they would like to keep it blue but answered NO when asked if the government should be responsible for keeping the sky blue. Know what I mean? That ability to stratify allows for controlling of confounders.

WHO TO ASK?
Just like you don't want to bias your survey by asking leading questions, don't do a survey on health needs of the inner city by asking people who live in the suburbs. Know what I mean? I'm telling you not to engage in selection bias. Consult with a geographic statistician to help you choose where to conduct your survey. If it's a public place, it should be a place where people from all walks of life are very likely to be present. A metro stop or the National Mall on the 4th of July would be good. (Ever been to the Mall on the 4th? It's pandemonium.) Finally, take a look your aims. If you want to know what issues are of importance to women, don't poll a NASCAR race. Seriously, don't.

BRANCHING OUT
Take a look at the way the following 2-question survey breaks down:

Chug! Chug! Chug! Chug! Chug!

As you can see, 75% of the respondents agreed that the country is headed in the wrong direction, yet 65% agree with the Keg party. Did you see what I just did? It's something that a lot of people with ulterior motives will do with survey responses. They'll combine two branches of the "response tree". A better way to describe these findings is, "75% (150) of respondents answered YES when asked if the US is headed in the wrong direction. Of those who answered YES, 75% (100) agreed with the Keg Party. Of the 25% (50) of respondents who answered NO when asked if the US is headed in the wrong direction, 60% (30) agree with the Keg Party." See, this doesn't mean at all that if you think the country is going in the wrong direction then you sympathize with the Keg party. What it does mean is that the Keg Party is popular, holding majorities in both groups (those who believe or don't that the US is headed in the wrong direction). Here's another way of looking at the survey responses:

Notice that opinion on the US is also not dependent on your like/dislike of the Keg Party

In short, never, ever try to associate the two questions with each other. Doing so is intellectually dishonest because the two ways of thinking are obviously not associated with each other. Agreeing with the Keg Party is equally distributed on both sides. And notice how thinking that the US is headed in the wrong direction is a majority on both sides in the second diagram. If they displayed survey responses like this, our understanding of the what the survey really revealed would be so much better... In my humble opinion, of course.

RECAP
You have to be very careful when you conduct a survey because surveys are just snapshots in time of how the populace is feeling. Remember that current events greatly influence people's perceptions on issues. The current situation in Japan has really made people weary of nuclear power plants. That wasn't the case when gas was nearing $5 a gallon two years ago. After September 11, 2001, everyone was weary of Muslims, and they still are to some degree today. And don't ask people about illegal immigration the day after an illegal immigrant commits a major crime.

Powered by ScribeFire.