Keiran Pedley looks at this morningâ€™s front page of The Sun and argues that we should always checkÂ the small print when reading opinion polls.
As someone that has spent most of his professional life reading opinion polls I have always enjoyedÂ this scene from Yes Minister where Sir Humphrey explains to Bernard how to rig an opinion poll. Itâ€™sÂ a funny scene but does demonstrate a pretty important point that all pollsters know – opinion pollÂ results are often as much about how the question is asked as what the question actually is.
This feels particularly relevant today as the front page of The Sun screams â€˜1 in 5 Brit MuslimsÂ sympathy with Jihadisâ€™. On face value, this is a very worrying finding for obvious reasons.
However, when you look at the data behind the headline things start to unravel a bit.Â It should be said first and foremost that polling a representative sample of a religious group is veryÂ difficult. Tom Mludzinski of ComRes and Maria Sobolewska of the University of Manchester explainÂ why in more detail on last weekâ€™s PB / Polling Matters podcast here and Matt Singh is good on thisÂ today here too.
However, my real complaint about this poll is the complete disconnect between the wording of theÂ question and the way the result has been displayed in this morningâ€™s paper. The actual questionÂ wording can be found below. Keep in mind that this is the question that has led to the headline onÂ the front page of The Sun claiming that one in five British Muslims have sympathy with Jihadis.
Sun front page polling. So just one UK Muslim in 20 has a lot of sympathy for those who go to Syria to join fighters pic.twitter.com/9yfcVaAQlF
— Mike Smithson (@MSmithsonPB) November 23, 2015
So what is wrong with this question? Firstly, it sets a very low bar for support. The one in five figureÂ that The Sun quotes includes anyone that expresses at least â€˜someâ€™ sympathy with young MuslimsÂ that join fighters in Syria. However, I think that the words â€˜sympathyâ€™ and â€˜fighters in Syriaâ€™ are theÂ most important here. â€˜Sympathyâ€™ does not mean support. It can do but the link is not certain. ItÂ could just mean that they understand why a young person might go to Syria even if they disagreeÂ with the decision. Even more importantly, what should we suppose that â€˜fighters in Syriaâ€™ actuallyÂ means? Again, it â€˜couldâ€™ mean ISIS or perhaps it doesnâ€™t. Notice how the words â€˜ISISâ€™ and â€˜Jihadisâ€™Â are not mentioned in the poll question but are used in the headline and in this opening line of theÂ supporting article.
This might sound very picky and pedantic but it is important. Letâ€™s consider how an alternativeÂ question wording might have been answered.
Do you support or oppose young British Muslims leaving the UK to fight for ISIS in Syria?
1) Strongly support
2) Somewhat support
3) Neither support or oppose
4) Somewhat oppose
5) Strongly oppose
6) Donâ€™t know
Not a perfect question by any means but you can see how it might have produced very differentÂ results to the one above. It makes the â€˜fighting for ISISâ€™ point much more explicit.
Perhaps the question was not designed to elicit the headline that it did. This is a common problemÂ for pollsters. We often have no control over how the results of our polls are presented in the publicÂ domain. However, in instances such as today â€“ on such a sensitive topic and in the aftermath of theÂ Paris terror attacks â€“ the media has a real responsibility to be careful with how it presents pollÂ findings. I think The Sun has got it wrong this morning.
The average person on the street is not going to go to the trouble of scrutinising sampling techniques or question wording. What they will see is a headline on the front page of one of theÂ most popular newspapers in the country that nudges to an â€˜enemy withinâ€™ â€“ with a giant picture of aÂ knife-wielding â€˜Jihadi Johnâ€™ just in case you didnâ€™t get the message. It leaves a sour taste to behonest.
In this piece I do not seek to play down the scale of the threat posed to our national security fromÂ Islamist terrorism. It is real and needs to be dealt with at home and abroad. However, the media has a real responsibility not to make things worse and todayâ€™s Sun splash was unhelpful in that regardÂ and unjustified based on the data it was based on. After all, using the same data, it could just as easily have said â€˜Just 1 in 20 British Muslims sympathise with those travelling to Syriaâ€™. I will leaveÂ others to judge why it did not.
Keiran Pedley tweets on polling and politics at @keiranpedley and presents the podcast â€˜PollingÂ Mattersâ€™