NCPP - National Council on Public Polls

1. Who did the=20 poll?
What polling firm, research house,=20 political campaign, corporation or other group conducted the poll? = This is=20 always the first question to ask.

If you don't know who = did the=20 poll, you can't get the answers to all the other questions listed = here. If=20 the person providing poll results can't or won't tell you who did = it,=20 serious questions must be raised about the reliability and = truthfulness of=20 the results being presented.

Reputable polling firms will = provide=20 you with the information you need to evaluate the survey. Because=20 reputation is important to a quality firm, a professionally = conducted poll=20 will avoid many errors.

Top

2. Who paid for the poll and why was it = done?
You must know who paid for the = survey,=20 because that tells you – and your audience – who = thought these topics are=20 important enough to spend money finding out what people think. = This is=20 central to the whole issue of why the poll was done.

Polls = are not=20 conducted for the good of the world. They are conducted for a = reason –=20 either to gain helpful information or to advance a particular = cause.=20

It may be the news organization wants to develop a good = story. It=20 may be the politician wants to be re-elected. It may be that the=20 corporation is trying to push sales of its new product. Or a=20 special-interest group may be trying to prove that its views are = the views=20 of the entire country.

All are legitimate reasons for = doing a=20 poll.

The important issue for you as a journalist is = whether the=20 motive for doing the poll creates such serious doubts about the = validity=20 of the results that the numbers should not be publicized. =

Examples=20 of suspect polls are private polls conducted for a political = campaign.=20 These polls are conducted solely to help the candidate win = –and for no=20 other reason. The poll may have very slanted questions or a = strange=20 sampling methodology, all with a tactical campaign purpose. A = campaign may=20 be testing out new slogans, a new statement on a key issue or a = new attack=20 on an opponent. But since the goal of the candidate’s poll = may not be a=20 straightforward, unbiased reading of the public's sentiments, the = results=20 should be reported with great care.

Likewise, reporting on = a=20 survey by a special-interest group is tricky. For example, an=20 environmental group trumpets a poll saying the American people = support=20 strong measures to protect the environment. That may be true, but = the poll=20 was conducted for a group with definite views. That may have = swayed the=20 question wording, the timing of the poll, the group interviewed = and the=20 order of the questions. You should examine the poll to be certain = that it=20 accurately reflects public opinion and does not simply push a = single=20 viewpoint.

Top=20

3. How many = people were=20 interviewed for the survey?
Because polls = give=20 approximate answers, the more people interviewed in a scientific = poll, the=20 smaller the error due to the size of the sample, all other things = being=20 equal. A common trap to avoid is that "more is automatically = better." It=20 is absolutely true that the more people interviewed in a = scientific=20 survey, the smaller the sampling error – all other things = being equal. But=20 other factors may be more important in judging the quality of a = survey.=20

Top =

4. How were those people = chosen?
The key reason that some polls = reflect=20 public opinion accurately and other polls are unscientific junk is = how the=20 people were chosen to be interviewed.

In scientific polls, = the=20 pollster uses a specific method for picking respondents. In = unscientific=20 polls, the person picks himself to participate.

The method = pollsters use to pick interviewees relies on the bedrock of = mathematical=20 reality: when the chance of selecting each person in the target = population=20 is known, then and only then do the results of the sample survey = reflect=20 the entire population. This is called a random sample or a = probability=20 sample. This is the reason that interviews with 1,000 American = adults can=20 accurately reflect the opinions of more than 200 million American = adults.=20

Most scientific samples use special techniques to be = economically=20 feasible. For example, some sampling methods for telephone = interviewing do=20 not just pick randomly generated telephone numbers. Only telephone = exchanges that are known to contain working residential numbers = are=20 selected – to reduce the number of wasted calls. This still = produces a=20 random sample. Samples of only listed telephone numbers do not = produce a=20 random sample of all working telephone numbers.

But even a = random=20 sample cannot be purely random in practice since some people don't = have=20 phones, refuse to answer, or aren't home.

Top

5. What area (nation, state, or region) = or what=20 group (teachers, lawyers, Democratic voters, etc.) were these = people=20 chosen from?
It is absolutely critical to = know=20 from which group the interviewees were chosen.

You must = know if a=20 sample was draw from among all adults in the United States, or = just from=20 those in one state or in one city, or from another group. For = example, a=20 survey of business people can reflect the opinions of business = people –=20 but not of all adults. Only if the interviewees were chosen from = among all=20 American adults can the poll reflect the opinions of all American = adults.=20

In the case of telephone samples, the population = represented is=20 that of people living in households with telephones. For most = purposes,=20 telephone households may be similar to the general population. But = if you=20 were reporting a poll on what it was like to be poor or homeless, = a=20 telephone sample would not be appropriate. Remember, the use of a=20 scientific sampling technique does not mean that the correct = population=20 was interviewed.

Political polls are especially sensitive = to this=20 issue.

In pre-primary and pre-election polls, which people = are=20 chosen as the base for poll results is critical. A poll of all = adults, for=20 example, is not very useful on a primary race where only 25 = percent of the=20 registered voters actually turn out. So look for polls based on = registered=20 voters, "likely voters," previous primary voters, and such. These=20 distinctions are important and should be included in the story, = for one of=20 the most difficult challenges in polling is trying to figure out = who=20 actually is going to vote.

Top

6. Are the results based on the answers = of all the=20 people interviewed?
One of the easiest = ways to=20 misrepresent the results of a poll is to report the answers of = only a=20 subgroup. For example, there is usually a substantial difference = between=20 the opinions of Democrats and Republicans on campaign-related = matters.=20 Reporting the opinions of only Democrats in a poll purported to be = of all=20 adults would substantially misrepresent the results.

Poll = results=20 based on Democrats must be identified as such and should be = reported as=20 representing only Democratic opinions.

Of course, = reporting on=20 just one subgroup can be exactly the right course. In polling on a = primary=20 contest, it is the opinions of those who can vote in the primary = that=20 count – not those who cannot vote in that contest. Each = state has its own=20 rules about who can participate in its primaries. Primary polls = should=20 include only eligible primary voters.

Top

7. Who should have been interviewed and = was=20 not?
No survey ever reaches everyone who = should=20 have been interviewed. You ought to know what steps were = undertaken to=20 minimize non-response, such as the number of attempts to reach the = appropriate respondent and over how many days.

There are = many=20 reasons why people who should have been interviewed were not. They = may=20 have refused attempts to interview them. Or interviews may not = have been=20 attempted if people were not home when the interviewer called. Or = there=20 may have been a language problem or a hearing problem.

Top

8. When was the poll=20 done?
Events have a dramatic impact on = poll=20 results. Your interpretation of a poll should depend on when it = was=20 conducted relative to key events. Even the freshest poll results = can be=20 overtaken by events. The President may have given a stirring = speech to the=20 nation, the stock market may have crashed or an oil tanker may = have sunk,=20 spilling millions of gallons of crude on beautiful beaches. =

Poll=20 results that are several weeks or months old may be perfectly = valid, but=20 events may have erased any newsworthy relationship to current = public=20 opinion.

Top=20

9. How were = the interviews=20 conducted?
There are three main = possibilities: in=20 person, by telephone or by mail. Most surveys are now conducted by = telephone, with the calls made by interviewers from a central = location.=20 However, some surveys are still conducted by sending interviewers = into=20 people's homes to conduct the interviews.

Some surveys are = conducted by mail. In scientific polls, the pollster picks the = people to=20 receive the mail questionnaires. The respondent fills out the=20 questionnaire and returns it.

Mail surveys can be = excellent=20 sources of information, but it takes weeks to do a mail survey, = meaning=20 that the results cannot be as timely as a telephone survey. And = mail=20 surveys can be subject to other kinds of errors, particularly low = response=20 rates. In many mail surveys, more people fail to participate than = do. This=20 makes the results suspect.

Surveys done in shopping malls, = in=20 stores or on the sidewalk may have their uses for their sponsors, = but=20 publishing the results in the media is not among them. These = approaches=20 may yield interesting human-interest stories, but they should = never be=20 treated as if they represent a public opinion poll. =

Advances in=20 computer technology have allowed the development of computerized=20 interviewing systems that dial the phone, play taped questions to = a=20 respondent and then record answers the person gives by punching = numbers on=20 the telephone keypad. Such surveys have a variety of severe = problems,=20 including uncontrolled selection of respondents and poor response = rates,=20 and should be avoided.

Top

10. What about polls on the Internet or = World Wide=20 Web?
The explosive growth of the Internet = and the=20 World Wide Web has given rise to an equally explosive growth in = various=20 types of online polls and surveys. Many online polls may be good=20 entertainment, but they tell you nothing about public opinion.=20

Most Internet polls are simply the latest variation on the = pseudo-polls that have existed for many years. Whether the effort = is a=20 click-on Web survey, a dial-in poll or a mail-in survey, the = results=20 should be ignored and not reported. All these pseudo-polls suffer = from the=20 same problem: the respondents are self-selected. The individuals = choose=20 themselves to take part in the poll – there is no pollster = choosing the=20 respondents to be interviewed.

Remember, the purpose of a = poll is=20 to draw conclusions about the population, not about the sample. In = these=20 pseudo-polls, there is no way to project the results to any larger = group.=20 Any similarity between the results of a pseudo-poll and a = scientific=20 survey is pure chance.

Clicking on your candidate’s = button in the=20 "voting booth" on a Web site may drive up the numbers for your = candidate=20 in a presidential horse-race poll online. For most such efforts, = no effort=20 is made to pick the respondents, to limit users from voting = multiple times=20 or to reach out for people who might not normally visit the Web = site.=20

The 900-number dial-in polls may be fine for deciding = whether or=20 not Larry the Lobster should be cooked on Saturday Night Live or = even for=20 dedicated fans to express their opinions on who is the greatest=20 quarterback in the National Football League. The opinions = expressed may be=20 real, but in sum the numbers are just entertainment. There is no = way to=20 tell who actually called in, how old they are, or how many times = each=20 person called.

Never be fooled by the number of responses. = In some=20 cases a few people call in thousands of times. Even if 500,000 = calls are=20 tallied, no one has any real knowledge of what the results mean. = If big=20 numbers impress you, remember that the Literary Digest's = non-scientific=20 sample of 12,000,000 people said Landon would beat Roosevelt in = the 1936=20 Presidential election.

Mail-in coupon polls are just as = bad. In=20 this case, the magazine or newspaper includes a coupon to be = returned with=20 the answers to the questions. Again, there is no way to know who = responded=20 and how many times each person did.

Another variation on = the=20 pseudo-poll comes as part of a fund-raising effort. An = organization sends=20 out a letter with a survey form attached to a large list of = people, asking=20 for opinions and for the respondent to send money to support the=20 organization or pay for tabulating the survey. The questions are = often=20 loaded and the results of such an effort are always meaningless.=20

This technique is used by a wide variety of organizations = from=20 political parties and special-interest groups to charitable = organizations.=20 Again, if the poll in question is part of a fund-raising pitch, = pitch it –=20 in the wastebasket.

With regard to the Internet, methods = are being=20 developed to sample the opinions of those who have online access, = although=20 these efforts are just starting. Even a survey that accurately = sampled=20 those who have access to the Internet would still fall short of a = poll of=20 all Americans, since only a relatively small fraction of the = nation’s=20 adults have access to the Internet.

Top

11. What is the = sampling error=20 for the poll results?
Interviews with a=20 scientific sample of 1,000 adults can accurately reflect the = opinions of=20 nearly 200 million American adults. That means interviews = attempted with=20 all 200 million adults – if such were possible – would = give approximately=20 the same results as a well-conducted survey based on 1,000 = interviews.=20

What happens if another carefully done poll of 1,000 = adults gives=20 slightly different results from the first survey? Neither of the = polls is=20 "wrong." This range of possible results is called the error due to = sampling, often called the margin of error.

This is not an = "error"=20 in the sense of making a mistake. Rather, it is a measure of the = possible=20 range of approximation in the results because a sample was used.=20

Pollsters express the degree of the certainty of results = based on=20 a sample as a "confidence level." This means a sample is likely to = be=20 within so many points of the results one would have gotten if an = interview=20 were attempted with the entire target population. They usually say = this=20 with 95% confidence.

Thus, for example, a "3 percentage = point=20 margin of error" in a national poll means that if the attempt were = made to=20 interview every adult in the nation with the same questions in the = same=20 way at about the same time as the poll was taken, the poll's = answers would=20 fall within plus or minus 3 percentage points of the complete = count’s=20 results 95% of the time.

This does not address the issue = of=20 whether people cooperate with the survey, or if the questions are=20 understood, or if any other methodological issue exists. The = sampling=20 error is only the portion of the potential error in a survey = introduced by=20 using a sample rather than interviewing the entire population. = Sampling=20 error tells us nothing about the refusals or those consistently=20 unavailable for interview; it also tells us nothing about the = biasing=20 effects of a particular question wording or the bias a particular=20 interviewer may inject into the interview situation. =

Remember that=20 the sampling error margin applies to each figure in the results = – it is at=20 least 3 percentage points plus or minus for each one in our = example. Thus,=20 in a poll question matching two candidates for President, both = figures are=20 subject to sampling error.

Top

12. Who’s on first?=20
Sampling error raises one of the = thorniest=20 problems in the presentation of poll results: For a horse-race = poll, when=20 is one candidate really ahead of the other?

Certainly, if = the gap=20 between the two candidates is less than the error margin, you = should not=20 say that one candidate is ahead of the other. You can say the race = is=20 "close", the race is "roughly even", or there is "little = difference=20 between the candidates." But it should not be called a "dead heat" = unless=20 the candidates are tied with the same percentages.

And = just as=20 certainly, when the gap between the two candidates is equal to or = more=20 than twice the error margin – 6 percentage points in our = example – and if=20 there are only two candidates and no undecided voters, you can say = with=20 confidence that the poll says Candidate A is clearly leading = Candidate B.=20

When the gap between the two candidates is more than the = error=20 margin but less than twice the error margin, you should say that = Candidate=20 A "is ahead", "has an advantage" or "holds an edge". The story = should=20 mention that there is a small possibility that Candidate B is = ahead of=20 Candidate A.

When there are more than two choices or = undecided=20 voters – in the real world – the question gets much = more complicated.=20 While the solution is statistically complex, you can fairly easily = evaluate this situation by estimating the error margin. You can do = that by=20 taking the percent for each of the two candidates in question and=20 multiplying it by the total respondents for the survey (only the = likely=20 voters if that is appropriate). This number is now the effective = sample=20 size for your judgement. Look up the sampling error in a table of=20 statistics for that reduced sample size, and apply it to the = candidate=20 percentages. If they overlap, then you do not know if one is = ahead. If=20 they do not, then you can make the judgement that one candidate = has a=20 lead.

And bear in mind that when subgroup results are = reported –=20 women or blacks, or young people – the sampling error margin = for those=20 figures is greater than for results based on the sample as a = whole.=20

Top =

13. What other kinds of = factors can=20 skew poll results?
The margin of = sampling error=20 is just one possible source of inaccuracy in a poll. It is not = necessarily=20 the source of the greatest source of possible error; we use it = because=20 it's the only one that can be quantified. And, other things being = equal,=20 it is useful for evaluating whether differences between poll = results are=20 meaningful in a statistical sense.

Question phrasing and = question=20 order are also likely sources of flaws. Inadequate interviewer = training=20 and supervision, data processing errors and other operational = problems can=20 also introduce errors. Professional polling operations are less = subject to=20 these problems than volunteer-conducted polls, which are usually = less=20 trustworthy.

You should always ask if the poll results = have been=20 "weighted." This process is usually used to account for unequal=20 probabilities of selection and to adjust slightly the demographics = in the=20 sample. You should be aware that a poll could be manipulated = unduly by=20 weighting the numbers to produce a desired result. While some = weighting=20 may be appropriate, other weighting is not. Weighting a scientific = poll is=20 only appropriate to reflect unequal probabilities or to adjust to=20 independent values that are mostly constant.

Top

14. What questions were asked?=20
You must find out the exact wording of = the poll=20 questions. Why? Because the very wording of questions can make = major=20 differences in the results.

Perhaps the best test of any = poll=20 question is your reaction to it. On the face of it, does the = question seem=20 fair and unbiased? Does it present a balanced set of choices? = Would most=20 people be able to answer the question?

On sensitive = questions –=20 such as abortion – the complete wording of the question = should probably be=20 included in your story. It may well be worthwhile to compare the = results=20 of several different polls from different organizations on = sensitive=20 questions. You should examine carefully both the results and the = exact=20 wording of the questions.

Top

15. In what order were the questions = asked?=20
Sometimes the very order of the questions can = have an=20 impact on the results. Often that impact is intentional; sometimes = it is=20 not. The impact of order can often be subtle.

During = troubled=20 economic times, for example, if people are asked what they think = of the=20 economy before they are asked their opinion of the president, the=20 presidential popularity rating will probably be lower than if you = had=20 reversed the order of the questions. And in good economic times, = the=20 opposite is true.

What is important here is whether the = questions=20 that were asked prior to the critical question in the poll could = sway the=20 results. If the poll asks questions about abortion just before a = question=20 about an abortion ballot measure, the prior questions could sway = the=20 results.

Top=20

16. What about "push=20 polls"?
In recent years, some political = campaigns and=20 special-interest groups have used a technique called "push polls" = to=20 spread rumors and even outright lies about opponents. These = efforts are=20 not polls, but are political manipulation trying to hide behind = the=20 smokescreen of a public opinion survey.

In a "push poll," = a large=20 number of people are called by telephone and asked to participate = in a=20 purported survey. The survey "questions" are really thinly-veiled=20 accusations against an opponent or repetitions of rumors about a=20 candidate’s personal or professional behavior. The focus = here is on making=20 certain the respondent hears and understands the accusation in the = question, not in gathering the respondent’s opinions. =

"Push polls"=20 are unethical and have been condemned by professional polling=20 organizations.

"Push polls" must be distinguished from = some types=20 of legitimate surveys done by political campaigns. At times, a = campaign=20 poll may ask a series of questions about contrasting issue = positions of=20 the candidates – or various things that could be said about = a candidate,=20 some of which are negative. These legitimate questions seek to = gauge the=20 public’s reaction to a candidate’s position or to a = possible legitimate=20 attack on a candidate’s record.

A legitimate poll = can be=20 distinguished from a "push poll" usually by:

The number of = calls=20 made – a push poll makes thousands and thousands of calls, = instead of=20 hundreds for most surveys; The identity of who is making the = telephone=20 calls – a polling firm for a scientific survey as opposed to = a=20 telemarketing house or the campaign itself for a "push poll"; The = lack of=20 any true gathering of results in a "push poll," which has as its = only=20 objective the dissemination of false or misleading information. =

Top

17. What other polls have been done on = this topic?=20 Do they say the same thing? If they are different, why are they = different?=20
Results of other polls – by a = newspaper or=20 television station, a public survey firm or even a candidate's = opponent –=20 should be used to check and contrast poll results you have in = hand.=20

If the polls differ, first check the timing of the = interviewing.=20 If the polls were done at different times, the differing results = may=20 demonstrate a swing in public opinion.

If the polls were = done=20 about the same time, ask each poll sponsor for an explanation of = the=20 differences. Conflicting polls often make good stories.

Top

18. So I've asked all the questions. = The answers=20 sound good. The poll is correct, right? =
Usually,=20 yes. However, remember that the laws of chance alone say that the = results=20 of one poll out of 20 may be skewed away from the public's real = views just=20 because of sampling error.

Also remember that no matter = how good=20 the poll, no matter how wide the margin, no matter how big the = sample, a=20 pre-election poll does not show that one candidate has the race = "locked=20 up." Things change – often and dramatically in politics. = That’s why=20 candidates campaign.

Top

19. With all these potential problems, = should we=20 ever report poll results?
Yes. Because = reputable=20 polling organizations consistently do good work. In spite of the=20 difficulties, the public opinion survey, correctly conducted, is = still the=20 best objective measure of the state of the views of the public. =

Top

20. Is this poll worth reporting?=20
If the poll was conducted correctly, and = you=20 have been able to obtain the information outlined here, your news = judgment=20 and that of your editors should be applied to polls, as it is to = every=20 other element of a story.

Top

This is a copyrighted publication of the = National=20 Council on Public Polls in keeping with its mission to help = educate=20 journalists on the use of public opinion polls.

The = National=20 Council on Public Polls hereby grants the right to duplicate = this work=20 in whole, but not in part, for any noncommercial purpose = provided that=20 any copy include all of the information on this page. =

Sheldon R.=20 Gawiser, Ph.D. is Director, Elections, NBC News. G. Evans Witt = is=20 president, Princeton Survey Research Associates, Inc. They were=20 cofounders of the Associated Press/ NBC News Poll.

For = any=20 additional information on any aspect of polling or a specific = poll,=20 please call the NCPP office at 800-239-0909.

The price = for a=20 single printed copy price is $2.95. For educational discounts = and=20 multiple copies contact NCPP. This document can be downloaded = without=20 charge from the NCPP website: http://www.ncpp.org/.=20

Top

For more information, contact us at info@ncpp.org.=20