Government gains are exceptionally rare, even with poll leads

Byelectionitis is not a condition that usually afflicts the Conservative Party and, to be fair, nor is it doing so now. The demands of government and particularly Brexit are inevitably occupying the attention of the Party and its members, both activist and elected. All the same, the flurry of positive polls for the Tories since the TIG split suggesting a Con lead of around 7-8% over Labour begs the question of whether they can win Newport West.

To be fair, it’s not an unreasonable question. The Conservatives have made as many by-election gains since 2010 as either Labour or the Lib Dems (the party with most gains this decade is actually UKIP, though both of theirs involved defences by a defecting incumbent), and when polls show a lead greater than at the previous general election, there’s clearly the potential for a government gain.

Or so you might think. In reality, even during periods of polling dominance, parties struggle to convert that lead into by-election gains. Since the 1987 general election, there’ve been four sustained periods when the polling has indicated a swing to the government – 1987-8, 1997-2000, 2001-2, and 2016-7 – plus a few shorter instances. (I would have liked to have extended this analysis back to the 1979 election, so as to take in the 1982-3 period too but I haven’t had time),

By my reckoning*, 25 by-elections have occurred since 1987 at a time when the polls indicated a swing to the government. Of these, six can be excluded due to oddities in the candidate line-up or because the government started a distant third and so notional swings were really just a reflection of the main fight in the constituency. However, of the other 19, only four translated that polling swing into a pro-government swing at the by-election (Beckenham 1997, Sleaford 2016, Copeland 2017, and Stoke Central 2017 – although that final one is affected by UKIP, who started and finished in second, marginally ahead of the Tories).

Of course, it may be that the polls were wrong. In particular, it’s generally forgotten how poorly the polling industry did at both the 1997 and 2001 elections (three of the final seven polls in the 1997 campaign gave Labour a lead of at least 20%; likewise, three of the final six in 2001 put Labour at least 17% ahead). It’s entirely possible that the reported Labour leads outside the election periods were similarly exaggerated, so generating a notional Labour ‘underperformance’ in the by-elections of the time.

In fact, I’m struck when analysing the by-election results of the effect of a government ceiling on performance. If we just concentrate on relatively clear-cut Lab-Con contests, I make Newport West the 28th such by-election since 1987 where the defending party’s lead at the previous general election was less than 30%.

Where there’s a polling swing against the government (as there usually is), there’s a reasonably good rule-of-thumb that the by-election will be about 40% worse than the swing in the current polls (i.e. if polling is showing a swing against the government of 10%, expect the by-election – before local factors are added in – to produce a swing of 14%). However, this ‘rule’ doesn’t come remotely close to working when the swing is to the government. Presumably, the incentives that voters feel to punish the government or to withhold former support, don’t apply to anything like the same extent when the tables are turned.

Of those 27 by-elections mentioned earlier, six took place when the polls showed a swing to the government (four in 1997-9, Ipswich in 2001, and Copeland in 2017). Of those, four still saw swings against the government, while the swing in Beckenham (1997), at the height of Blair’s honeymoon, and in unhelpful circumstances for the defending Tories, was just 2.6%. Only Copeland translated national polling into by-election votes.

None of this is to say that the Tories can’t win in Newport. It is, however, to say that almost all recent history points to an expectation that they won’t. Sure, Paul Flynn might have had a personal vote; Labour’s Welsh Administration might not be well regarded (recent Welsh-only polls certainly suggest this); Corbyn has awful personal ratings; Brexit offers enormous scope to rewrite the political landscape between now and 4 April – as, to a lesser but still significant extent, do the Labour defections and the reasons that prompted them. There is a lot of uncertainty. Nonetheless, the best (only, I think) odds you can get on the Tories to win is 4/1. I’d be wanting a good deal more than that.

David Herdson

Note – I originally wrote this article last Friday night. Since one further poll has been published, from Opinium, reporting a 6% Con lead (excluding TIG, which is the one we should concentrate on at least as far as the by-election goes). This is a slightly smaller advantage than most other polls since the TIG split but even so, still represents an implied swing to the Tories since the 2017 election of 1.75%.

* Calculating an opinion poll average is always a dangerous business. For this piece, I’ve generally based the ‘current polling’ on an average of three polls: the first immediately after the election and the two immediately before, where these have come from different companies. Where this hasn’t been possible (e.g. where only two companies were reporting at the time), I’ve tried to stick as close to the method as possible.