Fox News – the Right stuff
January 28, 2010
The shock-horror findings that 49% of respondents said that they trusted the right-wing Murdoch-owned channel, as opposed to 39% for CNN and 32% for CBS, come from research carried out by North Carolina-based firm Public Policy Polling (PPP).
Of course, trust in Fox was most forthcoming from Republicans (74%) as opposed to Democrats (30%).
But I was most interested in comments from Guardian readers which questioned the validity of the poll data: “I sincerely hope this is a statistical anomaly resulting from the location of the voters”; “Only a thousand odd from a population of how many?” (referring to the survey’s base of 1,151 voters).
Both comments misunderstand the methodology of such surveys. First, the fact that the polling organisation is based in North Carolina is irrelevant, since this was a national survey. Second, and more importantly, the absolute number of people polled has no bearing on accuracy; what matters is that the sample should be representative (ie, random).
That is, a small but representative sample is more reliable than a large but unrepresentative sample, a point classically illustrated by the rival predictions by the (newly-formed) Gallup organisation and long-established Literary Digest for the 1936 US elections.
The Digest, with its impeccable track record and 2.4m respondents, confidently foresaw victory for Alf Landon, just a week before he was swept aside by Roosevelt’s landslide, as correctly forecast by Gallup’s much smaller but representative sample of only 5,000 voters.
What the Digest had failed to realise was that its readership were atypically affluent; it also drew on lists of registered car owners and people who had telephones – both an upmarket group during 1930s America, and overwhelmingly more likely to vote Republican.
So it is quality, not quantity, that determines how well a sample can predict an outcome.
In the case of the Fox News poll, the Public Policy Polling website tells us that its survey has a margin of error of 2.8% (a fact which neither the Guardian nor other news reports, such as that from Press TV, thought worthy of mentioning), although it covers all bases by adding: “Other factors, such as refusal to be interviewed and weighting, may introduce additional error that is more difficult to quantify”.
However, one key fact which all reports and the PPP itself gloss over is how the survey was carried out. The way the questions are phrased on their website indicate it was done via an automated touch-tone phone system (“If Yes, press 1; if No, press 2” etc).
It is a point seized on by critics, led by – perhaps not surprisingly – by ABC’s head of polling, Gary Langer, in his blog.
“We don’t regard autodialed, pre-recorded polls as valid and reliable survey research,” he writes. “Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach.”
As an example, Langer notes the PPP survey finds 35% of respondents identify as Republicans, even though ABC’s own data suggest this figure is nearer 25%. As Fox is most trusted by Republicans, this would tend to over-state the level of trust for Fox.
One final note of caution – I’m not convinced that respondents necessarily understood “trust” in the sense of “Do you agree its news is impartial and unbiased”. I suspect if asked whether they “enjoyed” or “preferred” the various news channels, the results would be largely the same. That is, respondents may well have taken the phrase “Do you trust …” to be synoymous with “Do you approve of ..” or “Do you enjoy watching …” – and it is quite possible to approve of and enjoy a news channel which you know to be biased and partial.
And one does wonder what definition of the word “liberal” was in the minds of the 14% of respondents to a Pew Research Center poll last October who described Fox News as “mostly liberal”?