Keiran Pedley looks at this morning’s front page of The Sun and argues that we should always check the small print when reading opinion polls.

As someone that has spent most of his professional life reading opinion polls I have always enjoyed this scene from Yes Minister where Sir Humphrey explains to Bernard how to rig an opinion poll. It’s a funny scene but does demonstrate a pretty important point that all pollsters know – opinion poll results are often as much about how the question is asked as what the question actually is.

This feels particularly relevant today as the front page of The Sun screams ‘1 in 5 Brit Muslims sympathy with Jihadis’. On face value, this is a very worrying finding for obvious reasons.

However, when you look at the data behind the headline things start to unravel a bit. It should be said first and foremost that polling a representative sample of a religious group is very difficult. Tom Mludzinski of ComRes and Maria Sobolewska of the University of Manchester explain why in more detail on last week’s PB / Polling Matters podcast here and Matt Singh is good on this today here too.

However, my real complaint about this poll is the complete disconnect between the wording of the question and the way the result has been displayed in this morning’s paper. The actual question wording can be found below. Keep in mind that this is the question that has led to the headline on the front page of The Sun claiming that one in five British Muslims have sympathy with Jihadis.

Sun front page polling. So just one UK Muslim in 20 has a lot of sympathy for those who go to Syria to join fighters pic.twitter.com/9yfcVaAQlF

— Mike Smithson (@MSmithsonPB) November 23, 2015

So what is wrong with this question? Firstly, it sets a very low bar for support. The one in five figure that The Sun quotes includes anyone that expresses at least ‘some’ sympathy with young Muslims that join fighters in Syria. However, I think that the words ‘sympathy’ and ‘fighters in Syria’ are the most important here. ‘Sympathy’ does not mean support. It can do but the link is not certain. It could just mean that they understand why a young person might go to Syria even if they disagree with the decision. Even more importantly, what should we suppose that ‘fighters in Syria’ actually means? Again, it ‘could’ mean ISIS or perhaps it doesn’t. Notice how the words ‘ISIS’ and ‘Jihadis’ are not mentioned in the poll question but are used in the headline and in this opening line of the supporting article.

This might sound very picky and pedantic but it is important. Let’s consider how an alternative question wording might have been answered.

Do you support or oppose young British Muslims leaving the UK to fight for ISIS in Syria?

1) Strongly support

2) Somewhat support

3) Neither support or oppose

4) Somewhat oppose

5) Strongly oppose

6) Don’t know

Not a perfect question by any means but you can see how it might have produced very different results to the one above. It makes the ‘fighting for ISIS’ point much more explicit.

Perhaps the question was not designed to elicit the headline that it did. This is a common problem for pollsters. We often have no control over how the results of our polls are presented in the public domain. However, in instances such as today – on such a sensitive topic and in the aftermath of the Paris terror attacks – the media has a real responsibility to be careful with how it presents poll findings. I think The Sun has got it wrong this morning.

The average person on the street is not going to go to the trouble of scrutinising sampling techniques or question wording. What they will see is a headline on the front page of one of the most popular newspapers in the country that nudges to an ‘enemy within’ – with a giant picture of a knife-wielding ‘Jihadi John’ just in case you didn’t get the message. It leaves a sour taste to behonest.

In this piece I do not seek to play down the scale of the threat posed to our national security from Islamist terrorism. It is real and needs to be dealt with at home and abroad. However, the media has a real responsibility not to make things worse and today’s Sun splash was unhelpful in that regard and unjustified based on the data it was based on. After all, using the same data, it could just as easily have said ‘Just 1 in 20 British Muslims sympathise with those travelling to Syria’. I will leave others to judge why it did not.

Keiran Pedley tweets on polling and politics at @keiranpedley and presents the podcast ‘Polling Matters’