Possums Pollytics

Politics, elections and piffle plinking

Poll Wars: Episode 1 – The Phantom Metrics

Posted by Possum Comitatus on July 11, 2007

Over in The Oz, Newspoll CEO Martin O’Shannessy writes:

I was stimulated to consider whether Dennis Shanahan was right when he interpreted the turnaround in John Howard’s better prime minister rating as something more than merely encouraging. How could this be a turning point in the campaign if voter intention has not moved in spite of the Prime Minister’s improved ratings?

The question is whether the data supports the view that a turnaround in Mr Howard’s better PM rating presages an improvement in the Coalition’s electoral stocks. The short answer to this question has been yes in the past three elections.

The fact that anyone could get stimulated over Baghdad Bob is concern enough – but I suppose that’s what Martin has got to say. But what is of real concern is that last paragraph. Let us focus on the phrase that is the glue of Martins piece; “whether the data supports…”.

We’ll get to some interesting regression models later on that completely tear this argument a new one, but first off let’s look at whether the Preferred PM ratings have clearly moved over the periods leading up to the last three elections: Below are the graphs of the Preferred PM rating for Howard and the Government Primary Vote over the 1998, 2001 and 2004 periods. The dashed vertical line is the election in each graph.

ppm981.jpgppm011.jpg

ppm041.jpg

Well 2 out of three ain’t bad. Since we are looking at the effect of Preferred PM from June to July onwards for an election somewhere around November, and taking into account the margin or error on these polls which is plus or minus 3%, we be confident that in 1998 and 2001, Howards Preferred PM rating went up as the election approached, but in 2004 we simply cannot say that at all. In fact, in 2004 the only thing that can be said is that Howards PPM rating didn’t move in any way that could be classed as statistically significant between April and October of 2004.

Before we move on any further, I’d like to show you a couple of interesting things.

Firstly, what it is we are actually talking about:

ppm2parties1.jpg

 

In the 1993 election, Hewson was preferred PM over Keating in both February and March and lost. In the 1996 election, Keating was Preferred PM over Howard in February and March and lost. The theory of Preferred PM being the great deterministic force behind election victories isn’t looking too crash hot.

Second, have a squiz at Preferred MP ratings for the Opposition and Government overlaid against their respective satisfaction ratings (using Newspoll data):

ppmsats1.jpg

 

From this we can see that PM satisfaction levels and his preferred PM status pretty much walk hand in hand, but the same cant be said for the Opposition. What is also worth taking a look at here is how the preferred PM rating from Howard became slightly decoupled from his satisfaction rating around mid 2005.Let’s take a closer look:

pmsatppm11.jpg

The reason for this is simple – there are more complex dynamics at play. The period directly after the 2005 budget saw around 5% of the Coalitions primary vote move over to the Minor Party+Undecideds camp.See Determining the Swinging Voter and Their Behaviour for a detailed explanation of this.

But what’s important here is to notice the consequences of that movement carry across into preferred PM ratings. Howards satisfaction ratings went down, people were shifting their primary vote to the minor parties and into the Undecideds camp but they still preferred Howard over Beazley. Howard kept the PPM status by a powerful lack of alternatives.Then along came Rudd and bang, PPM changed.

 

Now, keeping all that in your thought orbit, let us move on to test the other key argument made by Martin:

The question is whether the data supports the view that a turnaround in Mr Howard’s better PM rating presages an improvement in the Coalition’s electoral stocks.”

Martin is saying that the Preferred PM rating is a leading indicator for the Coalitions primary vote.

This we can test – in depth. We’ll use the following variables:

C = a constant that is handy in these equations to measure the base value of the dependent variable (GOVPRIMARY in this case) that isn’t explained by movements in the values of the independent variables. This is automatically calculated by least squares regression software.

GOVPRIMARY = governments primary vote

PPMGOV = Preferred PM rating of Howard

And we’ll sometimes use (-1) or (-2) after those variables to represent a lagged variable i.e. PPMGOV(-1) is a variable that whose current value is last months value of PPMGOV.

And we’ll use the sample period of the Howard governments reign from April 1996 through to July 2007.

For those of you with a statistical bent – the following is a “cut down for a wider non-stats audience” affair, so please refrain from giving me grief. I’m sacrificing minutia for readability.

To start with lets run the obvious regression which explains how much of the government primary vote can be explained by changes in the previous months value of PPMGOV (which is Martins argument – what we are testing is whether PPMGOV(-1) is a leading indicator).

eq1001.jpg

This suggests, very very superficially (I’ll demolish it in a minute), that Martin may be somewhat right.22.7% of the movement in the governments primary vote can be explained by movements in the previous months PPM rating.It suggests that a 1 point increase in this months PPM will lead to a 0.21555 increase in the governments primary vote next month.

Now let us run the obvious next regression which explains how much of the government primary vote can be explained by changes in the current months value of the PPM rating for Howard.

eq2001.jpg

This suggests that 30% of the movement in the governments primary vote in any given month can be explained by changes in the value of the PPM for Howard in that month .So, superficially, this suggests that changes in PPM values walk hand in hand with changes in the governments primary vote rather than being leading indicators.

Now let’s run both variables together:

eq3001.jpg

When we combine the dynamics of both the lagged value of PPM and the current value of PPM for Howard something stands out. The lagged value of PPMGOV called PPMGOV(-1) is not statistically significant. This means that is doesn’t actually explain any statistically significant movement in the government primary vote when taking into account the influence of the current value of the PPM rating for Howard.

 

BUT

There is another problem here tied into this. This equation is auto-correlated up the wazoo. This means that there is an awful lot of lagged explanatory power still laying around in the residuals of the equation that is messing with our explanatory variables. If we look at the correlogram for that last equation we get:

correl11.jpg

 

This means, in this particular case, that the previous value of the governments primary vote has a big influence on the current value of the governments primary vote (the inertia effect that you often find in voting estimation series) and is interfering with out modelling. We can make another regression which accounts for this using a one period lagged value of GOVPRIMARY:

eq4001.jpg

This clears up all the autocorrelations and gives us a nice white noise process in the residuals which is exactly what we want.

What this new, clean equation tells us is that the 57-58% of the movement in the value of the governments primary vote can be explained by changes in the values of the 3 key variables of GOVPRIMARY(-1), PPMGOV and PPMGOV(-1).

However, far from Howards preferred PM rating being a leading indicator, it’s the opposite.

Put simply, what this means is that if we hold all the other independent variables constant, a one point increase in the last periods value of Howards preferred PM rating will lead to a 0.21 point reduction in the governments primary vote!

That is EXACTLY THE OPPOSITE of what Martins theory tells us should happen.

Preferred PM ratings for Howard are simply not a leading indicator for the governments primary vote, period. Their lagged effect is smaller than the effect of the PPM ratings current value, and that lagged effect is completely the wrong sign.

Howard’s PPM rating is covariant with the government primary vote, in that the two values move together through time, likewise the lagged values of the government primary vote are a leading indicator for futre values – but the lagged PPM values mean diddly squat.

I can appreciate that Martin tried to defend The Shanahans piffle over interpreting polls, but some piffle is simply not worth defending.

His last paragraph sums it up:

So, will the PM drag the Coalition up with him on the strength of his personal rating and will that be enough to overcome Labor’s commanding lead? The next few Newspolls will answer these questions.”

Watch out for the next Newspoll! Buy the Australian to get it!

Same bat time, same bat channel! ;-)

Which is fair enough.

 

AddThis Social Bookmark Button

About these ads

22 Responses to “Poll Wars: Episode 1 – The Phantom Metrics”

  1. I can’t pretend to follow the statistical work, but will take it on trust. Thank you for exploding that particular myth, Possum. It’s a relief – even though all the other indicators were pointing in the right correct direction!

  2. gusface said

    Possum this is on mumbles site
    “July 11 (pm) Casualties in the Newspoll wars
    I think Dennis Shanahan wrote this this morning (as opposed to yesterday). The “PhD” mentions refer, I believe, to me.
    A courtesy call from Editor-in-Chief Chris Mitchell this morning informed me that the paper is going to “go” Charles Richardson (from Crikey) and me tomorrow.
    Chris said by all means criticise the paper, but my “personal” attacks on Dennis had gone too far, and the paper will now go me “personally”.
    No, I’m not making this up.
    If they only get as personal as I get with Dennis, then it should be tame, as I don’t believe I’ve ever criticised anything other than his writing.
    And to think I described Dennis, in a chapter in a book being launched this month, as (with no sarcasm) “a fine journalist”.
    All very strange. And – I’d be lying if I didn’t admit – a little stomach-churning”
    FREEDOM OF THE PRESS (only if its good for J-HO)

  3. Possum Comitatus said

    I dont think Peter will need to reach for the Mercurachrome if he gets savaged by those wet lettuces in the Oz Opinion section somehow.

  4. bungs said

    Suggest providing this analysis to the offending journalists and pollsters: see if they publish careful analysis. Probably too complex for newly balanced media watch?

  5. Stig said

    Let me put forward a completely hypothetical question: If you were doing something wrong, and someone else kept on saying you were doing it wrong and was able to show why, would you:

    a) Engage in some self-reflection, so as to try and get it right next time; or

    b) Attack them personally, and in the process publicise to the world how wrong you’ve been, and your petulance in not changing?

    Hmmmm….

  6. Lomandra said

    What a fascinating day. It looks like traditional media hacks really can’t cope with the scrutiny of informed bloggers. Long gone is the day that we just have to swallow their analyses.

    The new media is working!

  7. Possum Comitatus said

    Lomandra – there’s a great post over at LP by crankynick

    http://larvatusprodeo.net/2007/07/11/government-gazette-fights-back/#comment-383921

    And it sums up the power of the blogosphere perfectly, specifically over issues like this.

    The old days of the Canberra press gallery getting away with pure piffle is… well..

    That dog dont hunt anymore.

    And the real beauty is, it’s only going to get worse for them, not better unless they adapt to the fact that 1.People are no longer mushrooms whom are reliant on the gallery’s gospel for information
    and
    2.If they dont stop substituting their own fantasies for fact, they will increasingly become the target of laughter, derision and scorn.

    And deservedly so.

    But you’re right – it’s been a marvelous day, although probably not that crash hot for poor old Peter Brent.

  8. It seems to me the statistical evidence you cite is at least as flimsy as the evidence you are arguing against.

    I can’t help but think you’d be clinging to the opposite, equally flimsy, evidence had people been arguing Rudds former supremacy in the preferred-PM stakes was meaningless. Indeed many bloggers decried any commenters who suggested so as being Young Liberal plants who were looking for an excuse to blow off the inevitability of the Ruddslide.

  9. Possum Comitatus said

    Stephen,

    The beauty about claims that X influences Y is how the notion of falsifiability comes into play. It is far easy, in the general sense, to demonstrate how and why X doesn’t influence Y than it is to demonstrate how it does. That’s the power of the scientific method.

    If I were to claim that this coin in my hand has the power to determine the primary vote, and to prove it I tossed the coin 3 times – the most likely outcome is that either a head or a tail will turn up twice out of three coin tosses.

    Because the primary vote may move up, does that mean that this magical coin has deterministic power over the primary vote because, say, heads came up 2 out of three times and the primary vote increased?

    Clearly not.

    That is the problem with small samples – random chance can give the impression that X influences Y in ways that a larger sample would demonstrate is clearly not the case.

    If we look PPM and just elections since 1991, what observable reality tells us is that 2 elections out of those 5 (93 and 96), the party with the highest PPM lost the election and 3 times the party with the highest PPM won the election.

    If we look at PPM ratings and just elections since 1991, observable reality tells us that in 2 out of those 5 elections, the party with the highest PPM had statistically significant movement toward them in their PPM rating leading up to the election, but their primary vote did not follow that pattern and they lost (93 and 96), another 2 times the party with statistically significant movement toward them in PPM ratings won as their primary vote moved with their PPM (98 and 2001), and in 2004 the PPM rating didn’t move in any way that can be called statistically significant because of the margin of error involved.

    That, my friend, is not evidence that the PPM influences the government primary vote because (a) the results are ambiguous and (b) the problem of the small sample comes into play

    On the other hand, we can test the relationship between PPM ratings and primary votes across 193 monthly observations aggregated from nearly 400 individual Newspolls and the results speak for themselves.

    The key problem here is that Shanahan was using the wrong metric and poor old Martin had to play ball with it.

    If Dennis wanted to run an argument regarding why the government primary vote moves leading into elections, he should have used PM satisfaction ratings – he would have been on safer ground. Usually PM sats and PPM ratings correlate across time, but not always, and in differering magnitudes. And it is here that he fell down because he was lazy, didn’t check the observable reality and the movement in PM sats wasn’t large enough to give him the story he seemingly wanted to write.

    As to your “I can’t help but think you’d be clinging to the opposite, equally flimsy, evidence had people been arguing Rudds former supremacy in the preferred-PM stakes was meaningless ”claim, the statistical evidence on PPM speaks for itself.

    The PPM series is nothing but a meaningless beauty pageant that in the tens of thousands of words, dozens of models and scores of graphics I have produced on my site, never ONCE have I used PPM ratings before this post.

    The reason for that is simple – I understand its actual statistical relationship to the main contest even if half the Canberra press gallery does not. That is why I have completely ignored it up until now.

    PPM contains no information in terms of influencing the primary vote that isnt already contained in the satisfaction ratings, but it does contain a hell of a lot more information that appears to be irrelevant to primary vote dynamics.

  10. [...] And now even Newspoll is going on the offensive, with yesterday’s (also heavily mocked) article by the Newspoll CEO, Martin O’Shannessey, who – no doubt to his own satisfaction – “proved” that a slight improvement in “Preferred PM” ratings for Howard meant the election was a done deal. UPDATE: In case you were interested, Possum does his usual comprehensively-argued job of debunking O’Shannessy’s theory here. [...]

  11. [...] Poll Wars: Episode 1 – The Phantom Metrics Over in The Oz, Newspoll CEO Martin O’Shannessy writes: “I was stimulated to consider whether Dennis Shanahan was […] [...]

  12. Tomasso said

    Poss, I agree with the analysis that data doesn’t support PPM being a lead indicator of election outcomes. Comment on satisfaction rating (pretending to be a Qual), the PPM score looks more convervative that satisfaction (less movement, and probably a bit lagged). Specifically, satisfaction is probably an earlier emotional position than approval, and would apply more broadly across the electorate. Ie, a change in PPM status is probably more of a second step after a change in satisfaction, and one less likely to be made. Eg, a rusted on LP or ALP supporter may well have a change in satisfaction, but not enough (or not yet enough) to jump.

    Why Newspoll beat up the minor change in PPM rather than the long story (the 2PP substantial NON-change) is beyond me.

  13. Possum Comitatus said

    I agree Tom, the PPM ratings behave strangely. Theoretically you would expect them to be a good proxy for preference flows (all things considered) – but that doesn’t seem to actually be the case most of the time. In terms of the governments PPM, satisfaction and primary vote series, PPM ratings don’t contain anywhere near as much explanatory power as satisfaction ratings on the primary vote estimation once you start accounting for the autocorrelation in the primary vote series. If the primary vote series undergoes any shock, the PPM series starts playing silly buggers because it doesn’t adapt well to that impulse response function while the satisfaction series holds its information content relatively stable, albeit at a different magnitude.

    This tells me that (surprise surpise), a sizeable handful of survey respondents either tell porkies or are confused when oppositions start putting up a good fight.

    PM satisfaction ratings are certainly a leading indicator for PPM ratings which ever way I want to model them that stands out clearly, but the inability of the PPM series to adapt to shocks, even small ones in the government primary vote is why I haven’t really touched the PPM series at all on the blog. Since we’re in a period where GOVPRIMARY has undergone not only a shock with Rudd, but a longer term structural change since the beginning of 2002, and another one compounding that since mid 2005 – I’m of the belief that the relationships that were manifest between these series in previous elections wont hold anywhere nearly as strongly in the forthcoming election.

    On the newspoll issue, I talked to Martin O’Shannessy today, and he was just trying to point out the Howard comeback of previous elections and the PPM rating relationship to that, and that his language in the article gave the impression of a stronger relationship between PPM and the government vote than I think he meant to convey.

    Dennis on the other hand …..

  14. apostrophe said

    since the coefficient on PPMGOV is positive and the one for PPMGOV(-1) is negative, perhaps this could be interpreted that the *change* in PPMGOV is a predictor of GOVPRIMARY?
    thanks for the interesting analysis.

  15. Possum Comitatus said

    Apostrophe,

    Doing a quick regression: govprimary = beta_0 +beta_1govprimary(-1)+beta_2d(ppmgov), gives us a result that explains 0.42% (That’s 4 tenths of 1 percent) more of govprimary than if we just left the change in PPMgov out altogether and explained govprimary purely as a result of its inherent autocorrelation.

    The d(ppmgov) represents the 1st difference, or the change in the value of PPMgov.

    So while it does help to explain movements in govprimary, because its a first difference of a fairly stable series (the PPM series) the coefficient on that small movement couldn’t be expected to contain very much explanatory power.

    We could measure the first differences in govprimary in terms of the first differences and lagged first differences of PPMgov, but the problem with using first differenced dependent variables in voting estimation series is that the coefficients start encapsulating alot of the noise effects that are inherent in these series, and become over weighted as a result. More often then not you end up with coefficients giving independent variables a level of explanatory power that they simply don’t contain.

  16. Phrog said

    I love you Possum – if only you get rid of that beaver on the masthead and replace it with a real Aussie possum you would be perfect.

  17. Possum Comitatus said

    I’m working on that – getting a neat cartoony thing done, but you know what these damn artists are like – talk about watching paint dry ;-)

  18. [...] and came to an opposite conclusion. The local internet’s latest psephological star, Possum, crunched the numbers and confirmed that Shanahan’s and Newspoll CEO Martin O’Shannessy’s talking was [...]

  19. James said

    Hi Possum, I agree that when you take the entire period of 04-07 the suggested relationships in the data don’t hold. However, that’s because you can’t use linear regression to anaylse non-linear variables. Its clear from your PPM – Gov Opp graph that the variables analysed violate the assumption of linearity across the period used – a core assumption that needs to be satisfied. I think the graph you show is a very good one because it shows clearly that at different phases in the electoral cycle you get quite different rating response patterns (ie, something resembling a quadratic non-linear pattern which is interesting in itself from a sociopolitical perspective). At a glance, my feeling is you’d need to analyse the mid election cycle period separately from the campaign period. Even then you’d need to justify the “resolution” of the scale you choose to convince you’re capturing genuinely stable and linear variables across any apriori meaningful time period, Cheers

  20. Possum Comitatus said

    Hi James,

    I completely agree that not only these variables used here, but nearly all the polly variables I use from the polling data do (or could) contain some amount of non-linearity, at differing periods of time, to differing extents.Likewise, many of these variables and their relationships ‘appear’ to contain more non-linearity then they may actually, statistically contain as there is vast quantities of heteroskedasticity,structural breaks, non-stationarity and cointegration involved. This is quite problematic for what I do here as I try to be as accurate as possible on the one hand while avoiding highly complicated analysis on the other that would make it very difficult for most people to follow.

    For instance, when I was analysis the PPM/primary vote relationship, I actually built a couple of models that attempted to account for a non linear adjustment processes between the PMs PPM rating and the Government primary vote.From memory there was an AR(1) TAR model, a bilinear model and a couple of regime switching models.The problem became that they didnt actually have much to say over and above a more orthodox linear approach.What they didnt say is that there was any worthwhile evidence to support PPM being a leading indicator for the primary vote as a non-linear process.That pissed me off to no end as I had a great non-linear switching regime model – a truly beautiful thing to behold, but it contained no true premium of explanatory power over simpler apporaches.Doh!

    That happens alot with these series – great, technically brilliant models can be built that describe close to nothing in terms of real world observable reality that cannot be described more accurately by far simpler linear models.Parsimony delivers with these series, even if they arent as technically pretty as they could be and some bordeline assumptions have to made about the reality of the underlying data generation process.

    Another problem was the issue of the nature of the PPM and primary vote series themselves.Is the PPM/primary vote relationship non-linear? Are the series themselves non-linear? Well on the one hand yes.. statistically it could be – there is weak to moderate evidence to support it.On the other hand, it could be explained adequately (again weak to moderate evidence) as a linear relationship once structural changes in the series that occur as a result of exogenous shocks (like leadership changes) and exogenous influences (Satisfaction ratings) are accounted for.Alot of the evidence about the important nature of these series (unit root process being another) is bordeline.When in doubt (as in I’m faced with some important issue regarding the nature of the data that could be one of two or more processes) I’ve gone for parsimony and simplicity.

    So while this approach wasnt pretty, or certainly wasnt as pretty as more complicated non-linear approaches, the results were entirely compatible with the non-linear approaches I tried (and far easier to explain) in terms of exploring whether recent PPM values had a determinable impact on current primary vote values, or whether the two series are covariant.

    These series are the hardest series I’ve ever tried to model simply because of the ambiguity involved in the underlying data generation process.If you ever want the perfect example of where the best technical theory and model specification process breaks down at the first contact with the data – political polling is it.Often these series behave like a bag full of cats.

    By this stage of the blog I was hoping to have another 3 or 4 election prediction models up and running, but most of the more sophisticated and technical end of model specification and design simply gets hammered by the behaviour of the data involved.Sometimes it’s quite frustrating.

  21. [...] This skirmish became known as the Poll Wars, where the first of our contributions here was titled Poll Wars Episode 1:The Phantom Metrics. [...]

  22. Hey, you know about have hallatosis breath because of old rotten teeth? I bet you do that’s why a lot of you are always working trying to attract money. lol looking for sedatives and money because you need better teeth. anyway, i’m not selling anything or anything but i do want to encourage everyone to look up veneers. google it or find it somewher or ask your dentist. it is the freaking best thing the world.

    all that tasty candy that those assholes at mars gave you betrayed you and got you andyour kids running around looking like the wicked witches and wizards of the west.

    You go to the denstist and get a cleaning but what the hell is that going to do? You’ve got caved in teeth, appalachan mountain teeth, rocky mountain teeth, and general crackhead related teeth.

    It’s rough on you I know but it’s true. I used to be the same way. I did happen to spend a couple g’s on some veneers in atlanta ga (google it if you’re not from the US). Now I do bust out a big smile on every occasion i meet a face, in abstract to the past where it’s hide the smile! But anyway some of you might definitely need to undergo such a process.

    From another perspective, it’s a great way to continue the medical industry and make sure we have motivated dentists to come up with new innovative ways to fix our teeth. veneers is a big step over gold and silver teeth replacements which are gaudy and look crazy.

    veneers atlanta

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 57 other followers

%d bloggers like this: