The House of Lords Select Committee on Political Polling and Digital Media: a response

The House of Lords Select Committee into the polling industry has concluded its work and published its report today. It recognises a number of things about opinion research that few members of the polling community would dispute:

• Final polls since 2015 have a poor record of accuracy, and there is much work to be done in repairing the damage sustained by them;
• Polling has got harder over the last decade for a variety of reasons (response rates, voter volatility and the decline in key demographic metrics in particular), even if evidence of a long term and cross-national decline in polling accuracy is not evident;
• Evidence that polls may shift voter preferences is weak and in short supply, but there can be little doubt that poll outcomes define and refine media narratives during electoral campaigns that may have indirect influence on the decision-making process that voters go through.

In a climate of such uncertainty, and associated justifiable cynicism toward the political polling industry, it would be churlish to deny the existence of one or more methodological problems. This pollster, for one, has never done so, having been on record before a number of electoral events – most notably the Scottish Independence referendum and the EU Independence referendum – fretting over prospective polling performance; indeed, personal misses at both of the last two General Elections hardly put me in a position to protest.

Within this context the initial reaction that took hold upon reading the report recommendations was one of surprise. For the polling industry, the scope for regulatory oversight (such as in France) and even a complete ban on political polling during an election campaign were very real options, which for some would be difficult to digest. Yet the Committee has rather ruled them out, or at least kicked those particular cans somewhat down the road. However, Committee Chair, Lord Lipsey’s contribution in The Times newspaper this morning was far more confrontational and threatening in tone, finishing off with an unambiguous signal that further inaccurate election polling will give ground “to those who advocate bans and more regulation….especially if the 2022 general election is again a fiasco”.

So we’re in last chance saloon if this is to taken at face value, even if most of the recommendations made by the Committee more substantively refer to the reporting of polls rather than the conduct of them. But as a pollster who considers himself very much in the firing line, this feels like a dodged bullet at this point, and the onus is very much on the polling community to, once again, develop systems that can accurately and consistently measure political public opinion when we need to.

We’re not close to that point in my view. Orthodox vote intention polling has endured systematic failure (with honourable exceptions in some cases) in three consecutive major electoral events, but for different reasons. In 2017, the Sturgis Inquiry concluded that unrepresentative samples were to blame, while finger pointing in 2017 was aimed at differential turnout issues. Whatever the reason, my own experience of those two general elections is neatly summed up as follows: the method that, yes, predicted Ed Milliband would be your Prime Minister in 2015 (if employed again) would have accurately predicted a hung parliament in 2017. The method drawn up to deal with 2015 problems that retrospectively enjoyed pinpoint accuracy presented a 12-point Tory victory in 2017. On such methodological quandaries do reputations fall.

Lord Lipsey’s call for better polling would be heeded if anyone actually knew what it looked like. Methodological plurality will probably be a main feature of political polling’s next phase, with technology advances allowing us to employ radical alternative or complementary medicine. And yet one of the Committee’s main findings is for some form of additional oversight for an expanded British Polling Council, one new role for which would be to provide “an advisory service for reviewing poll design”, by which we take to mean methodological and question framing adjudication.

The former can only be viewed as a threat to the aforementioned methodological plurality and innovation in opinion research. When I joined my old company ICM in 1995, Nick Sparrow was very busy reinventing the polling wheel after the 1992 debacle. His introduction of telephone data collection, past vote weighting and post-field adjustment were considered radical moves that some people rejected comprehensively. Who knows if their introduction would have been permitted by a France-like Inquiry mechanism with a reputation for methodological consensus? Ask YouGov the same question. They might refer you to the 2008 London Mayoral in which Ken Livingstone didn’t have much good to say for their online polls that had Boris ahead, which – you guessed it – contradicted the by then mainstream phone polling. YouGov ended up with a pinpoint prediction of Boris’ victory.

Then there’s the latter – question framing. Many conversations will need to be had to agree the remit of such a body on this, and you’ll do well to find a pollster who would be not be concerned about the practical implications of question-framing oversight in advance of a poll being fielded. How this works in the deadline-dependent media polling world in which questions arrive with minutes notice before the fieldwork closure point is anyone’s guess.

Perhaps though, the biggest report finding is what it decided NOT to do. A ban on polls has been muted during campaign periods, with bans operating in 16 of 28 EU countries, of between one day and one month of a major election. The Committee backed off this idea for the time being, citing freedom of expression values as the main reason for doing so. While no doubt valid, for my part a ban would and could only work if all politicians and their associates were also banned from commenting on shares of the vote, either in their own constituencies or nationally. How else can disingenuous or indeed outright misinformation on the performance of one party over another be checked without the conduct of rigorous, representative polling? Even with the recognised flaws associated with polling nowadays, it feels difficult to believe that the provision of less information to voters – a gap potentially filled with unevidenced campaigning data or worse, simple speculation masquerading as fact – could be in any way a positive development for our democratic process.

The rest of the report contains valuable and to be welcomed recommendations. Training in the media reporting of polls, taking the lead in identifying and critiquing bad examples of polling and having a regular review of election performance are ideas with merit.

Further, there can be few objections to an accurate log of vote intention polling being overseen by The Electoral Commission, with particular reference to transparency in the source of funding for the poll. This should ensure that fallacious arguments about poll findings being driven by the interests or position of their clients are eliminated. To its great credit, the Committee wisely limited this recommendation to the publication, rather than the conduct of polls. This implies that privately commissioned polling remains private, something that is often of great importance to the commissioning client. People should not doubt that public opinion polling for the media is often loss-leading or free, being cross-funded by private work for anonymous clients. Had there been a requirement for disclosure of all conducted polls, it would not be difficult to see the privately commissioned ones drying up, thus endangering the conduct of public polls and potentially seeing the withdrawal of polling firms from the market. This would also have a detrimental impact on innovation, which is the very thing that the industry needs.

On balance, my view is that this report is fair, recognises the methodological difficulties involved in conducting political opinion polling and presents recommendations that are reasonable. Obviously, there will be some impediments to the quick and easy implementation of the recommendations, but in my view the polling industry is likely to take them on board, with some gratitude, and reflect on the possibility that it could have been so much worse.