Pros and Cons of Survey Methods

Data Collection via Mail, In-Person, Phone, Internet, & Focus Group

Sep 22, 2009

An ill-conceived survey can create biased results that misinform policymakers and citizens alike. The five main types of surveys each have their own benefits.

Gauging public opinion is critical to the political process. Public opinion is important to the media, political parties, leaders, and interest groups in the development of public policy. Polls gauge three essential parts of an individual's opinion: values, ideology, and attitudes. These three components form the basis of public opinion, but properly gauging these elements can be a difficult task. As a society evolves, methods of data collection that once were fairly accurate become passe, while formerly unfeasible options become viable for pollsters.

Mail Surveys Are Common Yet Unreliable

Before the widespread use of the internet, many polls were conducted via postal mail. Although many polls have moved to other methods, mail surveys are still used in modern society. Questions are constructed, printed, and mailed (generally with return postage paid for) to personal home addresses or businesses. Individuals answer the survey and mail it back to the pollsters. The main costs associated with mail surveys were in printing, stamps, and time spent compiling and analyzing results.

This method has several benefits. Pollsters can easily reach a large geographic area, providing for potentially diverse array of answers. Most individuals are familiar with mail-based surveys, increasing the ease of completion. Because individuals complete these surveys in the privacy of their homes, mail formats are generally good for probing sensitive issues.

However, no clarification is available if the individual does not understand the question which could result in inaccurate information gathering. Because the survey is not completed in a contained environment, the sample needs to be motivated in order to respond to the survey and mail it back in, a factor which often results in a low completion rate and response bias.

In-Person Interviews Offer In-Depth Analysis at a Cost

In contrast to mail surveys, in-person interviews have a high completion rate and provide an opportunity for clarification. Individuals conducting the interview can probe the subject for answers, but this tactic can introduce bias. The interviewer can inject their opinion through word choice and tone of voice, possibly swaying the subject. While in-person interviews are often very thorough, they are also very expensive. Thus, many organizations will forgo in-person interviews in lieu of mail or phone based surveys.

In-person interviews can also create a number of human biases. First, social desirability refers to the process by which subject individuals modify their responses to survey questions based upon their perception of the interviewer's own opinion. Essentially, individuals do not want to give an answer that is not desirable socially. This process can interfere with accurate results. Not all interviewer bias is negative, however, especially if the interviewer is a middle-aged or elderly female. Dubbed the "grandma effect," female interviewers encourage individuals to be more honest in comparison to white, middle-aged men and younger interviewers.

Phone surveys are one of the most commonly used methods. Phone surveys generally target individuals during evening hours, the time when a majority individuals will be home and awake. This often results in undesirable "dinner calls." Individuals can ignore the interviewer by refusing to pick up the call, checking Caller ID, or immediately hanging up. Cell phone numbers are generally not available for interviewers to include in their random sampling which could leave much of the younger population unavailable for study.

Despite its drawbacks, phone surveys have a number of benefits. They are generally completed quickly with a low probability of interviewer bias because the individual cannot judge the interviewers appearance. Phone surveys also allow for clarification of complex or unclear questions, making them good for sensitive issues.

Internet Survey as a Global Survey Mechanism

Internet surveys are easy and numerous. With a possible worldwide population available and programmable elements, the internet survey allows for customization across a broad spectrum. Databases can be automatically converted digitally, avoiding time consuming reentry.

On the other hand, internet surveys require technical skill on behalf of the responder and the survey constructor. Depending on the method used to distribute the survey (placement on website, mass email, etc.), internet surveys may not offer a true representation of general populations. For example, a poll hosted on the FOX News website might lean more conservative than the general population because individuals taking the poll would have had to make a conscious decision to view the webpage in the first place.

Focus Groups and Undecided Voters

Focus groups are essentially mass straw polls, an unscientific approach to gauging public opinion. Nonetheless, focus groups are especially population in election years. Focus groups provide a qualitative method of data collection. Interviewers gather many individuals together in an informal setting. The interviewer then begins an inquisitive discussion regarding a specific topic. For example, during election season, a news channel may gather undecided Ohio voters to discuss the political candidates.

These sessions offer an easy opportunity to incorporate visual displays and can help identify key concepts and ideas in public opinion. While these sessions often acquire lots of information and diverse opinions, this method is not scientific because it does not rely on simple random sampling nor does it create a representative sample.

In addition, focus groups are expensive to compensate individuals for their time and for video/audio setup. Focus groups are also subject to “leader effects,” where a small select group of individuals dominate the conversation and question answering. This subject bias can discourage individual's with opposing ideas from speaking up during the focus group session.

No method of data collection and survey distribution is perfect. Each method has its own unique drawbacks and advantages. Organizations should critically analyze the types of surveys they use to gauge public opinion and select the best method for their needs while maintaining scientific integrity.

The copyright of the article Pros and Cons of Survey Methods in American Affairs is owned by Sarah Spiker. Permission to republish Pros and Cons of Survey Methods in print or online must be granted by the author in writing.