Thoughts on the polling debate from Nick Sparrow – formerly of ICM
Writing in the Guardian, Peter Kellner asserts that “it is common knowledge that final election polls are sometimes tweaked”. This should not come as a particular surprise. For a few days before an election, the one place neither a pollster nor the client wish to be is out of line with all the others, whether that means all the others have got it right or wrong.
Sticking to ones methodological guns, carefully developed over the previous years, requires courage. I should know. In 1997 ICMs polls, using new methodologies, were way out of line. Although more accurate I well remember the pressure to be in line with others. No wonder the investigation into the failure of the polls in 2015 could not rule out “herding” and other stories of polls, proven to be more accurate by subsequent events, but suppressed or “tweaked” because the pollster didn’t believe his/her own findings.
All of this means that pollsters are not only careful collectors and objective analysts of data. Partly, uncomfortably, what a reader of polls may be looking at, especially in a pre-election poll, is the opinion of the polling company.
The formation of the British Polling Council was partly in response to calls for the polls to be somehow regulated and banned in the final days before an election, as happens elsewhere in the world. The BPCs “principal objective … is to uphold standards of disclosure and ensure that consumers of survey results entering the public domain have an adequate basis for judging the reliability and validity of the results”. And “through full disclosure encourage the highest professional standards in public opinion polling”.
Polling companies are entitled to add to the report of any poll that they are members of the BPC and abide by its rules. If they do not publish details of all the methods used to generate their published findings then, in my book they are in breach of those rules. FULL disclosure was certainly my intention when I drafted the rules.
Of course polls have occasional failures, 1992 and 2015 being more recent examples. Subsequent investigations cannot point at reasons why, commend new technique etc or suggest improvements if the data they are looking at has been nobbled by frightened pollsters before being made public.
Perhaps more worrying to those who don’t want to see more regulation should be the hint that poll results may be the opinions of a representative sample of all voters modified by what the pollster thinks.
(Nick is one of the pioneers of UK polling and his firm, ICM, was regarded as the Gold Standard. He has written before on polling issues for PB – MS)