The CNN "Poll of Polls"
First of all, that this paragraph ended up in that article is sloppy copy-editing. The poll results are not actually reported in this article. But never mind that detail, CNN's reporting on the Poll of Polls has been widely discussed.
They continue to use the sentence "There is no sampling error." Smart folks will point out that in a sense they are right--the average of the polls gives you the true average of the polls because you have the entire population of polls. Of course there are presumably other polls that were not included in the sample, so in the strictest sense this is still a sample of polls. But if we ignore that little detail, we can accept the Poll of Polls for what it is--the simple average of a number of poll results. The relationship between that average and the true value is complicated by the different methodologies employed in the various polls. The fact that the Poll of Polls is the average of the universe of polls does not mean that it has no margin of error when used for inference on the universe of voters.
But does the average reader or television viewer understand the difference, or do the words "There is no sampling error" lead the reader or viewer to see the results as more reliable than they really should?
To illustrate the point in a simple way, consider this. Suppose I ask three people to flip a coin 10 times and report to me the number of times the coin came up "heads". The first person says that heads came up 3 times, the second person saw heads come up 6 times and the third person reported heads 4 times.
Then I could legitimately say that the average of the three trials was 4 1/3 and that there was no sampling error in obtaining the average of the three trials. I sampled the entire population of three people who flipped ten coins. However, it would be wrong to draw the conclusion that the expected number of heads when flipping a coin ten times is 4 1/3.
Might people draw the wrong inference from such a statement? Well, maybe not in my obviously simple story, but applied to real world polling it is more likely. I see people make statistical errors all the time--some of them quite obvious--so yes, I think some might misunderstand the statement.
Of course, in my simple example I am quite confident that the law of large numbers holds and if I did this a few more times, I would very likely get closer to the true value. If I did it many, many times I could get very close to the true value. In the case of polls, there might very well be a central tendency (in the polls themselves), but the relationship of that central tendency to the true population proportions would depend on the survey methodologies--which are beyond the scope of this blog post.
The poll of polls consists of eight surveys: ARG (October 25-27), IPSOS-McClatchy (October 23-27), Pew (October 23-26), ABC/Washington Post (October 24-27), Reuters/C-SPAN/Zogby (October 26-28), Gallup (October 26-28), Diageo/Hotline (October 26-28), and IBD/TIPP (October 24-28). There is no sampling error.
First of all, that this paragraph ended up in that article is sloppy copy-editing. The poll results are not actually reported in this article. But never mind that detail, CNN's reporting on the Poll of Polls has been widely discussed.
They continue to use the sentence "There is no sampling error." Smart folks will point out that in a sense they are right--the average of the polls gives you the true average of the polls because you have the entire population of polls. Of course there are presumably other polls that were not included in the sample, so in the strictest sense this is still a sample of polls. But if we ignore that little detail, we can accept the Poll of Polls for what it is--the simple average of a number of poll results. The relationship between that average and the true value is complicated by the different methodologies employed in the various polls. The fact that the Poll of Polls is the average of the universe of polls does not mean that it has no margin of error when used for inference on the universe of voters.
But does the average reader or television viewer understand the difference, or do the words "There is no sampling error" lead the reader or viewer to see the results as more reliable than they really should?
To illustrate the point in a simple way, consider this. Suppose I ask three people to flip a coin 10 times and report to me the number of times the coin came up "heads". The first person says that heads came up 3 times, the second person saw heads come up 6 times and the third person reported heads 4 times.
Then I could legitimately say that the average of the three trials was 4 1/3 and that there was no sampling error in obtaining the average of the three trials. I sampled the entire population of three people who flipped ten coins. However, it would be wrong to draw the conclusion that the expected number of heads when flipping a coin ten times is 4 1/3.
Might people draw the wrong inference from such a statement? Well, maybe not in my obviously simple story, but applied to real world polling it is more likely. I see people make statistical errors all the time--some of them quite obvious--so yes, I think some might misunderstand the statement.
Of course, in my simple example I am quite confident that the law of large numbers holds and if I did this a few more times, I would very likely get closer to the true value. If I did it many, many times I could get very close to the true value. In the case of polls, there might very well be a central tendency (in the polls themselves), but the relationship of that central tendency to the true population proportions would depend on the survey methodologies--which are beyond the scope of this blog post.
Leave a comment