Evaluating the significance of a survey used to be fairly straightforward – for any truly random sample, there is a well-defined margin of error which allows a critical reader to judge how valid any inferences are.

So for a survey of 1,000 respondents, the margin of error is around 3% at the 95% confidence level (this last figures indicates there is only a 5% probability that the findings were the result of chance). If a news story says Ed Miliband is leading David Cameron by 42% to 37% in the polls, we know this falls within the stated margin of error, since Ed Miliband’s true position may be 39% (42% – 3%) and David Cameron’s may be 40% (37% + 3%).

The difficulty from the pollster’s point of view is that creating a truly random sample is costly and time-consuming. So recently, non-probability sampling has enjoyed favour, particularly in the US and now increasingly in the UK.

Read the rest of this entry »


“Marriage is more stable than cohabiting”, reports the Daily Telegraph, accurately summarising a report from the Office for National Statistics.

The usual rent-a-quote right winger is then lined up to say how this proves the Tories are right to give tax breaks to married couples, etc.

However, what is overlooked in the article’s analysis (though referred to) is that two-thirds of co-habiting couples go on to marry. That is, marriage is (for co-habitees) a way of continuing to be a stable couple. But this is not a symmetric operation – married couples never (or very, very rarely) change from being married to co-habiting.

It is this fundamental asymmetry which accounts for the prevalence of married over co-habiting couples; there is an entropic movement to the ‘phase state’ of marriage.

Nowhere does the survey indicate that marriage is either morally or societially preferable, despite what the moral agenda of the Telegraph would have us believe.

Question of perspective

February 11, 2010

The Daily Mail is quite worked up because the number of marriages in England and Wales is “at its lowest level since 1862“.

Quoting figures from the Office for National Statistics (ONS), the Mail predictably attacks Labour for the decline – the report comes “as Gordon Brown’s Labour government faces mounting claims it has failed to support the institution” (what the Mail fails to mention is that most of these claims come from newspapers such as the Daily Mail).

The Mail explains: “Fewer people are getting married than at any time in more than 100 years, the Office of National Statistics revealed.”

But what’s this on the ONS website? Here the same report is published under the headline: “Registrations in England and Wales remain stable”. Read the rest of this entry »

Just come across this post by David Moore, examining why the US public seems to have contradictory and confusing views on President Obama’s proposed health care reforms.

The reason? A substantial number of respondents do not understand the issue. Hence their responses to detailed questions about policy: “they respond in an almost random fashion”, influenced by their previous answers or the way the question is phrased.

Now, while this point is made in the context of US public policy, the general point is perfectly applicable to a range of topics where the evidence is factually complex and technical – climate change, the best way to tackle the economy, education reform, military strategy in Afghanistan … topics on which the public are polled regularly but of which they are largely uninformed. Read the rest of this entry »

Fascinating though it is, and despite the leading adjective in the title, Philip Meyer’s classic New Precision Journalism is sorely in need of an update.

Of course, a book written over 18 years ago can’t help but be overtaken by events. Some of the examples, and in particular the details, have dated badly. Even more glaring is what is missing – no mention of mobiles, no email and (naturally) no Web.

Read the rest of this entry »

An article in the Liverpool Daily Post tells us that, “Beer sales fell more than 8% in the first three months of the year compared with 2008. The British Beer & Pub Association (BBPA) said 1.7m fewer pints were drunk daily from January to March than in the same period last year.”

So far, so good – that’s 1.7m fewer pints each day for 90 days (31+28+31), making a total of 153m fewer pints sold in the first quarter of 2009.

But then, a few paragraphs later, we learn this: “The BBPA’s UK Quarterly Beer Barometer showed a total of 68m fewer pints – more than 750,000 a day – were sold.”

What? This flatly contradicts the intro. So is the true figure 1.7m fewer pints daily or 750,000 pints? After all, the difference represents a hell of a lot of lager! Read the rest of this entry »

Poll with the hole

March 27, 2009

BBC is running a story about proposed changes to the Act of Succession, which would allow a future monarch to marry a Catholic, and for female royals to have the same place in the pecking order as male heirs. There is widespread public support for change, the Today programme assured us.

A poll for the BBC found 89% in favour of equal rights for royal women and 81% approved of the heir being allowed to marry a Catholic … but, as presented, the results were meaningless because there was no indication of sample size and hence no way of judging its validity.

In fairness, the BBC website does state the sample size (1,000 polled by ICM between March 20-22), but why couldn’t this detail have been given or at least referenced in the broadcast? And while it’s laudable that the website gives the sample size, additional detail such as the margin of error would be useful – why not link to this, too?