From Prof John Curtice's site – WhatScotsThink most recent #indyref poll of polls pic.twitter.com/BVdRwjNQcu

— Mike Smithson (@MSmithsonPB) August 22, 2014

How effective are the pollsters with such a one-off event?

Knowledge, information and judgement: the past, present and future of effective prediction.  The problem, as far as the Scottish referendum is concerned, is that all three are badly affected by the paucity of precedent.  There have been referendums before, both in Britain and elsewhere, but all have their drawbacks for comparative purposes.  (Prediction is of course only one half of effective betting; the other being able to spot value).

From overseas, we can study the independence referendums in Quebec, the EU Treaty polls in various countries, and the Australian vote on the monarchy; closer to home there were the 1979 and 1997 votes on Scottish devolution, the EEC referendum and most recently the AV vote.  All, however, have their drawbacks in making direct comparisons, whether that be the nature of the electorate, the size of the turnout and intensity of the campaign, the divisiveness of the issue in question, or the broader political and economic context in which each one was fought.  Put simply, each one is a one-off to a much greater extent than a general election is.

That fact not only makes it harder for analysts to work out the implications of the polling numbers but it makes it harder for the polling companies themselves to produce accurate data.  How do you weight responses to reflect what they say about their likelihood to turn out, for example?  The general rule on turnout is that it’s driven by two key factors: how close the vote is perceived to be and how important the result is perceived to be.  September’s vote scores so highly on both counts it may pip the 81.1% Northern Ireland recorded in the 1998 Good Friday Agreement referendum.  Ensuring the polls accurately represent those voters on the very fringe, who often will not be general election voters never mind other elections, will be no easy task.

Similarly, the methodologies used to try to ensure a balanced political sample become much more complex where there are apparently such disparities between how men and women are planning to vote, between those born in Scotland and those from outside.  Weightings can be and are applied but producing accurate figures is still akin to hitting a moving target, while standing on the back of a moving pickup truck, at dusk – which may account for the widely differing figures being published.

“Aha”, you might say, “then I’ll balance them out and take an average”, which is all very well except that there’s no guarantee that the average will be particularly accurate.  1992 remains the most famous example, but it’s not just a historic problem.  In 2010, every pollster overstated the Lib Dems and virtually every one understated both the Conservatives and Labour; five of the last six London mayoral polls in 2012 gave too low a share to Ken Livingstone, four of them by more than 3%; the Scottish election in 2011 saw the SNP under-reported for the regional vote in almost every poll, some by a considerable margin, even while those same polls got the SNP almost spot-on in the constituency section; most pollsters overstated those in favour of AV by 7-10% in the two weeks before the vote.

Put another way, even in elections for which pollsters have had a fair bit of practice, the methodology is still not perfect.  Worse, there’s been a tendency for all the various methodologies to be out the same way, if by different amounts.  In the Scottish referendum, they’ve had virtually no practice.

What does this mean for the vote?  My instinct would be to allow for a much wider range of possibilities than the polls are currently showing (and for once, perhaps because they can’t benchmark against their peers, there’s quite a spread already).  It may be that No has an even more commanding lead than the 22% YouGov found earlier this month; alternatively, it could be neck-and-neck, beyond even Survation or Panelbase’s findings.  We are in unchartered waters.  To that end, the value bet is with Yes – there is too great a degree of certainty in the odds at the moment.

David Herdson