Populus sets out its case

Opinion polls are critical tools for political gamblers and in recent weeks we’ve made a number of observations about their accuracy. In particular we have looked at what happened at the Euro Elections last month when surveys by YouGov, in the Telegraph, and Populus, in the Times, could be compared with real results.

Two days before the Euro vote we noted that YouGov was putting its reputation on the line with its very high figure for UKIP and, indeed, this is what is has been attacked for.

But after the elections we also noted that the internet pollster got closest with the crucial Lab-Con split although both it and Populus both under-estimated the Tories and over-estimated Labour – something that most polls most of the time when tested seem to do. This tendency is something that gamblers should always be aware of and should always factor in.

This prompted the head of Populus, Andrew Cooper, to post this comment. We think that he makes important points that should be given greater prominence.

As a matter of actual fact the accepted measure of accuracy in the polling industry is the average error in the final poll published before an election. On this measure the Populus poll conducted nearly a week before polling day, was fractionally more accurate than YouGov’s poll conducted around the same time, with an average error of 2% compared with 2.2% for YouGov’s.

What is more YouGov’s exit poll – with a massive 7,000 sample of people supposedly being polled after they had voted, and therefore in theory much more accurate, had an average error of 2.8% (compared with a theoretical margin of error, given sample size, of 1.1% – which technically classifies it as a rogue poll). The YouGov exit poll predicted essentially a 3-way tie between Labour,Tories and UKIP.

Within all of these figures lies an immutable fact about polls, which you seem to have missed: they are blunt instruments; even the most methodologically robust poll cannot be more accurate than its standard deviation, which is a function of sampling. For most polls this means the figure for any party is accurate to within plus or minus 3%. So a poll putting Labour on, say, 32%, in fact means simply that there is a 95% probability that their support lies somewhere between 29% and 35%.

For this reason reading the leads between parties from polls is an even more treacherous business, since the margins of error on two figures are compounded: our last poll, for example, showed a Labour lead of 4% but in fact, when margins of error are taken into account, it meant that Labour’s lead is (at 95% confidence) somewhere between -2% and +10%.

Generally speaking the polling companies have done a pretty good job in as much as they’ve stayed within the margin of error most of the time. Populus did so again in the Euro elections; YouGov didn’t. Not only was its average error outside the defined margin of error, but it erred on its UKIP rating by a wide margin – an egregious mistake (presumably a result of the sampling problems associated with internet-only polling), compounded by the probability that UKIP would not have done even as well as they did by the end had not

YouGov not wrongly placed them at nearly 20% right at the beginning, which became the story of the campaign. Polls which make headlines, as that one did, have the capacity to become to a degree self-fulfilling: just one of the reasons why integrity of method is so important and why there is a drive now to improve the self-regulatory standards of the polling industry.

Incidentally, since you raise the point, Populus has only once shown a Tory lead – in May. ICM have shown the 2 main parties level-pegging several times. MORI have shown a Tory lead at least once (July 2003). And if you average each company’s polls (a sensible way to allow for the inevitable blips) you will find that, since last summer Populus has shown an average Labour lead of 2.9% compared with 3.9% for ICM, and 2.9% for MORI while YouGov has the Tories in the lead by an average of 2.4%.

The key driver of this variation is the fact that Populus, ICM and MORI all have the Tories averaging 32% over the last 18 months, while YouGov have them averaging 38%.

Your proclamation that Labour will win most seats at the next election but might not have an overall majority puts you way out on that limb with YouGov! All I can say is I’m very glad I don’t have any of my money riding on your prediction (or their’s)

We are grateful to Andrew for this but he has not touched on the major issue – why is it that almost without exception polls are biased to Labour? We have made critical comments in the past about other firms and none has responded. This is an important issue and the more we know the better.

We hope that at least one of the bookmakers will open a market at the General Election on which pollster will get closest

.

Image http://www.aperfectworld.org/clipart/Metaphors/signonlineb.gif