Sampling, probability and non-probability

Evaluating the significance of a survey used to be fairly straightforward – for any truly random sample, there is a well-defined margin of error which allows a critical reader to judge how valid any inferences are.

So for a survey of 1,000 respondents, the margin of error is around 3% at the 95% confidence level (this last figures indicates there is only a 5% probability that the findings were the result of chance). If a news story says Ed Miliband is leading David Cameron by 42% to 37% in the polls, we know this falls within the stated margin of error, since Ed Miliband’s true position may be 39% (42% – 3%) and David Cameron’s may be 40% (37% + 3%).

The difficulty from the pollster’s point of view is that creating a truly random sample is costly and time-consuming. So recently, non-probability sampling has enjoyed favour, particularly in the US and now increasingly in the UK.

Why marriage is more stable

“Marriage is more stable than cohabiting”, reports the Daily Telegraph, accurately summarising a report from the Office for National Statistics.

The usual rent-a-quote right winger is then lined up to say how this proves the Tories are right to give tax breaks to married couples, etc.

However, what is overlooked in the article’s analysis (though referred to) is that two-thirds of co-habiting couples go on to marry. That is, marriage is (for co-habitees) a way of continuing to be a stable couple. But this is not a symmetric operation – married couples never (or very, very rarely) change from being married to co-habiting.

It is this fundamental asymmetry which accounts for the prevalence of married over co-habiting couples; there is an entropic movement to the ‘phase state’ of marriage.

Nowhere does the survey indicate that marriage is either morally or societially preferable, despite what the moral agenda of the Telegraph would have us believe.

Question of perspective

The Daily Mail is quite worked up because the number of marriages in England and Wales is “at its lowest level since 1862“.

Quoting figures from the Office for National Statistics (ONS), the Mail predictably attacks Labour for the decline – the report comes “as Gordon Brown’s Labour government faces mounting claims it has failed to support the institution” (what the Mail fails to mention is that most of these claims come from newspapers such as the Daily Mail).

The Mail explains: “Fewer people are getting married than at any time in more than 100 years, the Office of National Statistics revealed.”

But what’s this on the ONS website? Here the same report is published under the headline: “Registrations in England and Wales remain stable”.

Baleful influence of ignorance

Just come across this post by David Moore, examining why the US public seems to have contradictory and confusing views on President Obama’s proposed health care reforms.

The reason? A substantial number of respondents do not understand the issue. Hence their responses to detailed questions about policy: “they respond in an almost random fashion”, influenced by their previous answers or the way the question is phrased.

Now, while this point is made in the context of US public policy, the general point is perfectly applicable to a range of topics where the evidence is factually complex and technical – climate change, the best way to tackle the economy, education reform, military strategy in Afghanistan … topics on which the public are polled regularly but of which they are largely uninformed.

A new New Precision Journalism

Fascinating though it is, and despite the leading adjective in the title, Philip Meyer’s classic New Precision Journalism is sorely in need of an update.

Of course, a book written over 18 years ago can’t help but be overtaken by events. Some of the examples, and in particular the details, have dated badly. Even more glaring is what is missing – no mention of mobiles, no email and (naturally) no Web.