Poll dancing

Chris Trotter suggests that frequent and heavily-publicised polls favour the right and result in a self-fulfilling prophecy, depressing support for the left. Bomber Bradbury has made similar arguments in the past.

There are two parts to this: first, the question of whether polls are inaccurate vis-a-vis the actual views of the electorate, and if so whether they do in fact favour the right; and second, whether this, on its own, has a substantive impact on actual real-world support.

Lyndon Hood FTW

I am aware of no robust research on this topic in the NZ context (which isn’t to say that it doesn’t exist; although if it did I’d expect the many proponents of this line of argument to be citing it all day long.) So to kick off, let me concede that if there was a significant ideological bias to the polls it certainly could have an impact on voter behaviour. But I suspect any impact would be more complicated than the simple “low showing for a party causes low turnout and/or low support” reading of the “spiral of silence” model Chris cites (which is nevertheless a useful model). For one thing, it’s not clear that the impact is very strong in case of a secret ballot; also, the argument doesn’t take into account New Zealanders’ oft-cited peeriness about single-party majority governments in the wake of Muldoon, Lange and Bolger which could have a moderating effect, and it doesn’t take into consideration the usual “narrowing” that occurs in the latter stages of an election campaign as wavering voters begin to make up their minds. So I doubt the size of this effect on Labour’s polling at least; any shift should not be large, and certainly not sufficient to change the election outcome. However where NZ First is concerned it’s different — a percentage point or two makes all the difference in the world there, and whether NZ First returns to parliament or not rests on the collective spirit of Winston’s faithful. Assuming they still make up fiveish per cent of the voting population they must not waver, and must cast their votes together regardless of what they are told about the polls. If they fail to do so NZ First is out, and all those votes are wasted, and that is certainly an outcome that could result from widespread reporting of polls that show NZ First under the threshold (and one that could change the election outcome).

Note, however, that all that is contingent on those polls showing NZ First to be under the threshold actually being wrong. I’m not persuaded of this, and I’m also dubious about the contention that polls favour the right. As Bomber is fond of pointing out, the polls run in the leadup to the Auckland Supercity election favoured John Banks, while in the election Len Brown won handsomely. That’s important to remember, but it’s only one data point so I’m hesitant to draw meaningful conclusions from it, and if it’s evidence of anything, it’s against the simple reading of the “Spiral of Silence” thesis, according to which Brown’s demoralised supporters should have stayed home.

There is also some evidence out of the US to suggest that landline-only polls favour the Republican party over the Democrats (when compared to polling samples that include cellphones). It’s not clear is that this trend is also in effect in New Zealand. Based on no data, I think it has some influence, but our situation here differs in important ways; most notably, we have more than two parties so variance of this sort is more dilute and less zero-sum. And the likely effect would be ambivalent — if landline-only polls tend to exclude the young and the poor and tech-savvy urban liberals, they should overstate support for NZ First, not understate it.

The matter of bias isn’t readily testable because, as an Australian poll analyst put it recently when looking at bias between pollsters in that market, “we just don’t have elections every week to determine the true state of public opinion“.* There is also evidence to suggest that people’s voting behaviour differs in important ways from how they answer opinion polls. So we need to rely on other forms of measurement; proxies being measured against proxies. There are focus-groups, vox pop interviews and 75c text-polls and coffee-bean polls, and the Horizon poll with its radically different weighting system, but I see no firm grounds to believe any of these would be any more reliable than the major phone poll companies. One very strong point in favour of the usual sort of phone-based opinion polling is that it has remained more or less methodologically consistent for a very long time. This gives us extremely large, continuous datasets that we can plot against real-world events including policy releases, major speeches, current events and elections. In this regard they are much like audience/circulation ratings in the media. The utility of these well-established systems isn’t that they have no flaws — they certainly do, and much of the criticism is valid — it’s that they have a reasonably well-known set of flaws that are consistent over time, and we can use the historical record to make inferences about the present day (too few people, I think, are actually doing this, but that’s a different matter). By contrast other systems — Horizon, in particular — are less well-known, and serious questions need to be asked about the quality of their results. They’re worth watching, but they are not as yet a substitute for what we already have.

Returning to the argument. So if the polls systematically favour the right, and if they do so in such a way as doesn’t also favour NZ First’s older, wealthier and more conservative demographic, and if the extent of that bias is significant enough to have a significant zero-sum election night impact despite all the confounding variables of voter behaviour, then Chris might be right.

But that’s a lot of ifs, and a claim as bold as “polling companies and the media steal elections from the left” demands extremely robust evidence to be given credence. I’m open to the argument, but what we have here isn’t evidence, much less the robust kind; it’s supposition, and what’s more it’s supposition derived from partisan loyalty. The argument is unfalsifiable — if the election does deliver a substantial defeat to Labour and NZ First they will take that as proof-positive that they were right all along and the voters were robbed; if it doesn’t no doubt they will cheer those who battled valiantly and overcame the oppressive regime imposed by the forces of evil. In this regard the argument is nearly indistinguishable from that made in 2008 by the Free Speech Coalition who, despite their howls about Stalinist restrictions on political marketing, managed to help their political representatives to a handy win.

My view of Chris and Bomber’s general line of argument is that it’s more of the usual excuse-making and blame-shifting that I see from lefties who can’t bear that their tribe is staring down the barrel of a(nother) heavy defeat. It’s an attempt to make the evil media cabal responsible for poor performance, and to minimise the effect of weak leadership, institutional incompetence, lacklustre campaign and — in Peters’ case — public self-immolation during the last term. It’s a myth; something to keep the faithful warm at night, in the absence of actual victory. Myths certainly have a kernel of essential truth, but they should not be mistaken for evidence. Moreover, as regards Chris’ concerns about the election accurately reflecting voters’ wishes; these would have more credibility if his own history of respecting the solemn integrity of electoral politics was itself less one-eyed. He thunders:

hundreds-of-thousands of New Zealanders will not only be perfectly entitled to say “bugger the pollsters”, but they will also be entirely justified in asserting that the election has been stolen from them

This would ring less hollow if, following the 2005 election, he had not so cravenly excused the Labour party’s own admitted breach of electoral law as being “acceptable corruption” inasmuch as it served the greater ideological purpose of preventing National from being elected. Given that history, his complaints about polling and media conduct look like nothing more than an appeal to ideological tribalism with a bright red smear of electoral integrity lipstick on.

And yet, he has a strong point: if NZ First’s share of the party vote on November 26 comes in fractionally below the 5% threshold, democracy will have been poorly served in 2011, as it was in 2008. My personal views of that party aside, as a matter of natural justice anyone commanding a twentieth of the vote should be entitled to roughly a twentieth of the representation in Parliament. But however unsexy it might be, the solution to this isn’t injustice to attack the pollsters or the media. The argument isn’t supported by the evidence; even if it was nobody’s going to change on Chris and Bomber’s say-so, and we all saw what happened to the Clark government’s regulatory overreach. The pragmatic response here is to work towards reducing or removing the electoral threshold so as to ensure that natural justice is served and variance is less likely to simply exclude a party from representation, and to turn back to the process of returning political rigour to the political left by building competence, vision and leadership so it can succeed despite the obstacles before it. Less myth, more reality.


* Pollytics’ analysis plots polling companies against each other, measuring each pollster’s bias from the other polling companies. David Winter has had a go at doing something similar for the NZ context, though he has much less data to work with. While interesting, it’s important to reiterate that none of this addresses the concerns about poll accuracy vis-a-vis the electorate.)

29 thoughts on “Poll dancing

  1. Bradbury claims low numbers of households in South Auckland have landline telephones cf the North Shore:

    But he didn’t respond to the Whaleoil challenge to say where he got his figures from:

    If they were true, it would justify claims that the methodology is skewing the results. I note polls such as Colmar Brunton control for all sorts of demographics except income, which strikes me as a weakness.

    Rob Salmond has responded to Trotter’s claims about bias and reckons they are wrong:

  2. There’s one polling company that is cheating its paymaster: they ring me most months.

    I wonder if, when they are short of people with my profile, they know they’ll hit the jackpot with me – I’m always happy to take part.

    Given they never ask me about my previous responses and never acknowledge they have spoken to me before, I assume this is a mistake (read: laziness) on their part, not deliberate methodology.(My answers are always the same.)

  3. Lew,

    In your reading have you seen any evidence for or against a hypothesis that public polls suppress the vote when elections are not close?

    It has always seemed to me that knowing the party you vaguely support will win by a country mile may make your vote seem as pointless as knowing the party you vaguely support will be on the losing side of that gap. I get frustrated with the FPP mentaility of much coverage of polls here, as they seem to emphasise a winner takes all view, rather than looking at the overall effect on the blend in Parliament.

  4. MeToo, interesting. I did read Rob’s article at the time but had forgotten it. He’s a much more credible authority on this topic than I am, so if he’s failed to persuade Chris and Bomber then I think the chances of doing so are slim.


    I haven’t seen any, but I also didn’t look very hard for that so there may be some. On the upside — National in the NZ context — I suspect that’s what people will point to, especially with the latest DigiPoll showing National below 50%. But the confounds are huge; as Giovanni Tiso observed, the headline might as well be “Poll shock as what every analyst had said would happen as the election drew closer in time, happened”.

    On the downside (Labour &c) it’s the same problem as anything — how does one distinguish between genuinely low or falling support for an unpopular party who’s failing to connect with the electorate, and change resulting from reporting of that fact? I think proving the latter would be extremely tricky.

    I think the FPP analysis of polls (emphasising the gap between National and Labour, rather than between National, and Labour and its probable coalition partners, is partly laziness and partly genuine need to avoid voter confusion. The relationship status between Labour and potential partners is “complicated”; it’s not like Australia where the Nationals will always vote en bloc with the Liberals and they can be consistently taken together. I also think that norm of poll reporting will change somewhat if the Greens’ vote holds up, or if the relationship between Green and Labour were to firm up to the point where a coalition between them is seen more likely than not.


  5. The FPP analysis of the polls is at times even more problematic: e.g. the polls just prior to the 2010 Wellington City Council had the DomPost telling everyone the Kerry Prendergast was well ahead. I looked at the numbers on the front page and thought “they’ve forgotten that this is STV, I reckon Celia Wade-Brown might have it.”

    I wonder whether our polling companies need to take advise from place overseas when dealing with different systems: there’s no reason the Wellington poll shouldn’t also have asked people “the Wellington Council elections use a preferential system where you get to rank the candidates with numbers, whom will you rank higher: Kerry and Celia?”

  6. It is disappointing, Lew, to see you join the ranks of the right-wing bloggers by misrepresenting my position regarding the 2005 election.

    In the column so often cited I was actually quite critical of Labour for breaching the election rules. But, I was also very thankful that Helen Clark had quite probably prevented a Brash victory by doing so.

    Dr Brash was, of course, absolutely committed to abolishing the Maori seats – a move which would very likely have plunged New Zealand into horribly destructive racial conflict.

    If the choice was between Maori and Pakeha being at each other’s throats and New Zealand spiralling into a deeply regressive social environment OR Helen Clark’s spending outside the rules, then I would still plump for “a little corruption”.

    It’s interesting that, in spite of your oft-stated support for Maori causes, you would have preferred a Brash Government dedicated to abolishing Maori representation and tearing up the Treaty of Waitangi.

    You have also misunderstood the point of the posting you criticise.

    My dispute is with the way the news media uses the poll data obtained from the major agencies to frame its National-friendly coverage of the entire campaign.

    Your use of terms like “evil media cabal” shows how unwilling you are to take such criticism seriously.

    Wouldn’t want to bite the hand that feeds you, I suppose.

  7. Chris, I don’t think you know as much about what I do as you think you do.

    As to the rest: I’ve always been perfectly plain in arguing that the integrity of the system as a whole is more important than any given outcome — even an outcome I am very attached to. I certainly don’t disagree about the post-2005 counterfactual — but if that’s the will of the people as expressed by the system we have to measure it, then that’s an unfortunate reality. If you’re happy to countenance cheating in service of your own preferred outcomes then you lose some credibility when complaining about unfairness that favours the other side. It’s as simple as that.

    I haven’t misunderstood the point — I’ve previously expressed my views about the “National-friendly coverage” argument on many occasions, and as far as I’m concerned that argument is the same as the one about polls: a myth for which little evidence exists, and what evidence does exist is highly ambivalent, usually deployed by the left to excuse poor performance or incompetent strategy. I am happy to consider the evidence — but neither you, nor the other leading proponents of this line of reasoning, have provided any.


  8. There is one recent study here re media coverage allocation…
    not a ‘hammer blow’ for either side of the argument but interesting enough. Winston First has had a hell of a struggle to get any media time this election he claims. And as a follower of political events believe him, for a while I wondered if he had forgotten to engage any media help at all.

  9. Commiserations, Chris — the fools writing the NZ Herald election liveblog have gotten us confused (at 1:10pm). I’m sure you’re as upset about it as I am, but no response from them yet.

    TM, yes, interesting. Bahador’s similar study from the 2008 election, from memory, showed that Key and National actually got a harder time than most lefties thought. I have a copy somewhere, and when I find it I’ll link it up.


  10. I’m never terribly confident about experimental studies in this area, but here’s an interesting study which shows that public opinion polls generally increase turnout, but decrease it when the gap is predicted to be very wide.

    Großer, J. and Schram, A. (2010), Public Opinion Polls, Voter Turnout, and Welfare: An Experimental Study. American Journal of Political Science, 54: 700–717. doi: 10.1111/j.1540-5907.2010.00455.x

    They are talking about a two candidate winner takes all scenario, but still…

  11. “…I’ve always been perfectly plain in arguing that the integrity of the system as a whole is more important than any given outcome…”

    Wow. Just wow. The modern Nuremburg defense.

  12. Or upon reflection, perhaps the following:

    “…The court strongly feels that, although it is to be regretted that Rear-Admiral Albert H. Markham did not carry out his first intention of semaphoring to the C-in-C his doubt as to the signal, it would be fatal to the best interests of the Service to say he was to blame for carrying out the directions of the C-in-C present in person…”

  13. Sanctuary,

    “Wow. Just wow. The modern Nuremburg defense.”

    This is probably the most intellectually dishonest comment I’ve seen from you.

    Democracy means sometimes you’ve got to put up with bad outcomes. If you fudge the system you have to expect that the other lot will fudge the system as well, and they have many more resources and much more cover to do so.

    I went over my views in greater detail in this post: http://www.kiwipolitico.com/2011/06/notes-on-democratic-fundamentalism/


  14. Yes Chris, a ‘little corruption’ is allsweet as if Labour is doing it for a good cause.

    But your justificatory basis for political corruption is weak which makes your motivations suspect.

    If it was National who did the same thing for a similar reason I sincerely doubt you would be as charitable

  15. What Lew has written basically reinforces my long-held theory – that pollsters aren’t necessarily biased, but rather behind the times.

  16. Of course there’s no hard evidence Lew. Who’s going to admit to “Did you say you support National because you really don’t follow politics but heard on the news that National’s miles ahead in the polls and didn’t want to sound ignorant or a loser to a total stranger who sounded nice?”

    And why would the experts – advertisers – spend zillions on constant “the largest/best-selling/most popular/world leading/2-million-kiwis-can’t-be-wrong” rubbish if it had no effect?

    You know why I can’t elaborate (and thanks for respecting my need for anonymity), but believe me, I’ve see the peer-pressure effect almost weekly over many years. And repeatedly, absolutely blatant press distortion, omission and manipulation – which among other things, enables the former effect.

    Admittedly, can’t say it’s systemic, but it certainly does happen. With systemic impunity.

    And note that “crooked polling” isn’t necessary (though it wouldn’t surprise me at all, having designed and run polls and knowing how simple it would be to achieve with utter impunity): just an emphasis and repetition (e.g. by ignoring the Green poll and citing only Labour) to start the snowball rolling.

    You’re right, “more reality” is exactly what’s needed. And it may be worth nothing, but based on the reality of my own experience, that means expecting nothing from the media and working the net, halls and grass-roots organisation. Mining, Mt Albert, Lenslide, Hone: and reputedly, Winnie. We’ll see.

  17. Re Sanctuary’s quote about ALbert Markham; Several years before he obeyed orders and steered into his commanders ship, Markham designed the New Zealand flag.

  18. ak, that’s not the only way to gather evidence.

    One sort of evidence that was referred to upthread is Babak Bahador’s research on the election campaign at the University of Canterbury; the final results of the 2008 analysis are here.

    In a nutshell: National received more negative and less positive coverage than Labour; John Key received more negative and less positive coverage than Helen Clark; National got a larger proportion of the vote than their proportion of the coverage, and Labour got a smaller proportion of the vote than their proportion of the coverage.

    This isn’t definitive — one study never is — and I have some underlying methodological questions.

    But if you’re inclined to dismiss it out of hand because it yields results you don’t like, well, there’s a word for that, beginning with ‘b’.


  19. Balloon? Can’t open it and don’t recall dismissing it in or out of hand, but in another nutshell from memory:

    One study over a few weeks of a few outlets. And yes, methodology: single score form memory of “positive” or “negative” for selected quotes. Can’t begin to describe the flaws that screamed at the time, just remember thinking that the Herald’s red front page and Lenin/Helen statuettes could have scored the same as a footnote on page 3 saying “Key was accused of being economical with the truth”. No account of degree, or of course of what was never reported, said by whom, validity, etc etc. Current e.g: “Key crucifies Goff over $17billion funding hole” – one positive scored for Key? and/or one negative for Goff? “Goff denies hole” – positive for Goff?

    I maintain: impossible to measure, but we know it when we see it.

  20. So now you’re picking methodological holes in a piece of research because it didn’t yield an appropriate result? Yes, to better judge its veracity there are methodological questions that need to be answered — but you haven’t even asked the questions, let alone gotten answers, before writing the whole result off. No analysis of this sort can be perfect, but such a project can yield meaningful results without being perfect.

    And claiming that such a result proves it’s “impossible to measure”? There’s another b-word for that. Qualitative research to measure this sort of thing is methodologically non-trivial and very time-consuming but it can be done, and in fact it *is* done on a very regular basis.

    I don’t have any stake in Bahador’s research, but I think that since he’s an academic in good standing at a NZ university whose research will have had to gain faculty approval,I think writing it off requires some pretty strong grounds.


  21. “…because it didn’t yield an appropriate result?” Settle.

    “Qualitative research to measure this sort of thing is methodologically non-trivial and very time-consuming but it can be done, and in fact it *is* done on a very regular basis.”

    Ah, good. And not trying to be smart, but could you then point me to some please?

  22. Yeah I’ve seen those two, we discussed the second remember, and the first is where I found

    “After at least 60 years of research, a rich literature has developed
    concerning the question ‘do polls influence behavior?’. Yet no conclusive or unambiguous answer
    to the question can be given, whether related to vote choice, turnout, or opinions on issues.”

    Which sorta backs me up don’t you think.

    Also been to that Wikipedia thingy and a few other places in the course of my soc degree, but thanks anyway.

    (ps – b words – are they “baselessly belligerent”? mines bemused)

  23. No, ak, it doesn’t back you up — you’re arguing that polls and the media do positively influence behaviour in certain ways; I’m arguing that the evidence is ambiguous.


  24. I’m arguing that the evidence is ambiguous

    Fair enough. Sorry, I thought you were arguing that the academic research somehow negated my and Chris’ opinion, glad you agree it proves nothing.

    A bit sad though – looks like it took them sixty years to find out that the evidence is impossible to measure. (ducks for cover..)

  25. ak, it doesn’t negate your opinion — but since you’re arguing a positive effect the onus is on you to provide evidence, which you haven’t done. So your opinion, at minimum, is not one based on evidence (or at least not evidence you can articulate beyond “I know it when I see it”.)

    Your other argument is that qualitative aspects of media coverage can’t be measured. They can. I know they can because it’s what I do. What’s more difficult is tying those characteristics to voter behaviour, because there are a huge range of confounding variables. The same goes for polling. Regardless, whatever argument you make along these lines needs to be based on evidence to have any credibility.

    All I’m arguing is “it’s more complicated than collective delusions of persecution by The Man”. That’s a position supported by the evidence, such as it is.

    Anyway, I’m done with this argument.


  26. I notice Babak Bahador’s research paper suggests Labour received more negative references when the stories “came from journalists themselves and reflected their opinions” (Nats received more negative NEWS story coverage / Labour received more negative ANALYSIS story coverage).

    I certainly remember story after story on the front page of the Dom Post during the 08 campaign attacking Labour (“analysis” rather than “news”, but on the front page). One particularly egregious example that instantly comes to mind – Tracy Watkins outlines arguments from Helen Clark, Michael Cullen and Phil Goff (on whatever the issue of that particular day was) and then, after each one, decides to take it upon herself to negate the substance of what they’ve just said – along the lines of: “Helen Clark has argued that……” “But that can’t be right because……” “Michael Cullen suggests……” “However, he’s clearly wrong because……” Why bother going to the National Opposition for reaction when you can act as a Nat spokeswoman yourself ?

    The Dom Post’s coverage in 2008 was without doubt very one-sided in favour of National.

    As to the poll question: Labour voters (or, at least, voters from demographics that you would naturally expect to vote Left) tend to be less likely to vote anyway and if (a) polls suggest the Left trails the Right and (b) FPP-style media coverage of poll results lead voters to erroneously believe that Labour needs to bridge the entire gap (between the two main parties) in order to win power = then it seems likely that this tendency would be greatly reinforced. No hope of victory leading to non-voting. Can’t prove it but it seems like common sense.

  27. Pingback: Kiwipolitico » Blog Archive » Teapot Tapes poll: political moral suasion

Leave a Reply

Your email address will not be published. Required fields are marked *