Most people were taken by surprise by the general election result in May.  Why?  Well, most people’s expectations were set by the abundance of opinion polls which by election day had settled on a consensus showing the two main parties neck and neck at around 33% or 34%.  The actual result, with the Conservatives on 37% and Labour on 31%, came as a bombshell.  (In fairness to the pollsters, they were not alone – David Cameron reportedly had not written a victory speech that catered for an overall majority while Ed Miliband reportedly had not written a concession speech.)  But this is not an isolated instance of polling failure.  Where does this leave opinion polling and how should we treat them now?

The first thing to note is that opinion polls have never been very accurate and it simply isn’t in their design to be very accurate.  They have been over promoted and over-relied upon for many years.  The godfather of psephology, Sir David Butler, summed this up in a recent lecture to the Study of Parliament group:

“it is worth remembering that unexpected results have been the norm rather than the exception. We shouldn’t have been so surprised by how surprised we were this year.

In twelve of the last twenty general elections the outcome has defied the prophets – and the pollsters.

• In three elections (’45, ’66 and ’97) there was a Labour victory of totally unexpected proportions.
• In three others (’50, ’64 and October ’74) an expected Labour victory was achieved by only a single-figure margin.
• In four contests (’59, ’70, ’92, and 2015) there was a Conservative victory that was either totally unexpected or of unexpected scale.
• And in two elections (February ’74 and 2010) there was a hung parliament that few anticipated.

The lesson from these outcomes is perhaps that political science is even less of a science than we thought it was. Voters routinely defy our expectations, perhaps even misleading themselves when they answer opinion pollsters.”

Sir David did not explicitly say, though it is implicit in what he did say, that polling’s predictive power isn’t getting any better.  Of the last six elections, four were in the unexpected column.

Opinion polls have always been an unreliable friend.  All the pollsters’ voting intention questions have a basic problem that they are founded on an obvious untruth – there is not going to be a general election tomorrow.  And deducing the views of an electorate of over 46 million voters by surveying 1,000 or so people is inevitably as much an art as a science.  The sample can be unrepresentative, or it can skewed by liars, those refusing to answer and those who don’t know their own minds.  The question can be unintentionally leading.  And, of course, people can change their minds.

Pollsters do their best to make allowance for all of these difficulties and others.  Each pollster makes adjustments according to their own house style.  They then produce results which are described as being accurate within a margin of error (typically 3%), implying that the results will be there or thereabouts 19 times out of 20.  Given their recent track record, pollsters might wish to rethink how they present their output’s accuracy.  It seems tolerably clear that opinion polls, while giving a sense of public opinion, just aren’t as precise as their creators would wish them to be.  They’re more like e-fits than photographs.

So should we just throw away opinion polls?

No.  They do have value, even if they don’t have quite the value that their producers might wish us to believe.  So far as last May is concerned, pollsters are still trying to work out what went wrong.  In the meantime I suggest the following steps.

1. Don’t put too much faith in any single opinion poll

Opinion polls aren’t all that reliable.  So putting all your weight on just one is dangerous.

2. Conversely, if a lot of polls are telling you broadly the same thing, take those with more confidence

Multiple polls are less likely to be wrong if they are all telling you broadly the same thing.  As we saw in May, however, that’s not an infallible rule.

3. Consider whether there is any other indirect evidence supporting or contradicting the polls

In the final week before the general election, Ed Miliband campaigned in North Warwickshire and David Cameron campaigned in Twickenham.  Both of these were inconsistent with the picture portrayed by the polls, which suggested that both seats were out of the Conservatives’ reach.  The Conservatives in fact won both.

I overlooked this in the run-up to the general election. I shall be more alert in future to such signals.

4. Look at the trend rather than the absolute number

If Labour are up, pay more attention to the direction than to the precise number.  You sometimes hear “look at the share, not the lead”.  This is essentially the same point.

5. Be especially suspicious of non-standard polls

Polls of groups which pollsters are not particularly used to modelling are much less likely to be reliable because the base is less likely to be truly representative.   So polls of individual constituencies, political parties or specific professions are to be treated with still more suspicion than usual.

6. Be more confident about decisive results

37:31 isn’t all that far away from 34:34.  So the pollsters weren’t that far away.  When Jeremy Corbyn was polling in the 50s with YouGov for the Labour leadership race in August, we could be much more confident that he was well in the lead, even allowing for the less reliable base.

Oddly this means that the most important polling results are often ignored as boring.  75% of the public thinking that Labour are divided means that there is a consensus on the point.  89% of people being able to recognise Jeremy Corbyn means that he has made an instant impact.

7. Ignore subsamples

They aren’t weighted and the numbers are so small as to be meaningless (often they are actively misleading).  Don’t waste your time on them.

8. Opinion polls are snapshots not predictions, but…

… there isn’t anything better.

9. Remember the likely reaction to the poll

If you’re betting, you may not think much of opinion polls.  Nor do lots of other people.  But if everyone else is going to be betting on the basis of them, bear that in mind.  There will be occasions when it is unwise to try to swim against the flow.  And there will be times when the reaction is overdone and value opportunities arise.

To sum up, remember opinion polls may not be all that great, but they’re better than nothing.  Just don’t overestimate their usefulness.

Antifrank