Possums Pollytics

Politics, elections and piffle plinking

Archive for the ‘Leading Indicators’ Category

The Headline Forecast – regression prediction model.

Posted by Possum Comitatus on November 16, 2007

I’ve finally built the model to forecast the ALP TPP result. This gets a little stats heavy, so I’ll try to walk those folks through it that might find it hard going as best I can, but I’ll answer any questions you have in the comments.

What I used to build the forecast is the monthly average of Newspolls going back to 1996 when Howard was elected. The reason I use the monthly average is that it dampens a lot of the noise in the individual polls, and gives us a time consistent series of data that can be used for long term analysis.

I don’t trust the preference allocations for Newspoll, so what I did was construct my own based on the preference distributions for each election and let the preference flows adapt over time between elections, so the two party preferred vote from polls straight after an election used nearly all of the preference distribution from the previous election, polls from halfway between elections used a preference allocation based on half the previous election and half the next election, and polls just before an election had nearly all of their preferences distributed as they were at that impending election. For the 2007 election preference distribution I’ve simply used the 2004 preferences (which may slightly underestimate the ALP TPP, but not by very much).

The model itself is a regression model built specifically to forecast one month and only one month ahead.

The model is a little unorthodox because using polling data in a model is a little unorthodox to begin with, but the important thing here is that it works – even if it suffers a little econometric impurity in the process.

The variables I’ve used are split into two types.

Firstly, Dummy Variables – which are variables that have a value of zero or a value of one. What they let us do is measure how the level of the ALP TPP vote changed as function of specific periods time that represent events when we regress the ALP vote against them. You can get a gist for how they play out here:

The Dummy variables I’ve used are:

Dummyhhmoon – which is a dummy variable representing the Howard “honeymoon period” in 1996 as well as 2 months after every election. It has a value of one for the first 12 months of Howards government as well as for the two months after every election other than 1996. At all other times it’s value is zero.

Dummylatham – which has a value of 1 for those months Latham was leader and a value of zero for all other periods.

Dummyrudd – which has a value of 1 for the months Rudd has been leader and a value of zero for all other periods.

Dummyworkchoices – which has a value of 1 since November 2005 when Workchoices was in Parliament and the union campaign against it revved up.

DummyElection which has a value of 1 for the month an election is on and a value of zero at all other periods. I use this as an interactive dummy variable so I can emulate special election campaign effects with long term satisfaction rating changes.

Secondly, the other type of variables I use in the model are:

ACNALPTPP(-1) – which is the previous months value of the ACNielsen two-party preferred vote for the ALP. By using ACN, I can effectively anchor the forecast to the less volatile ACN series, while still using the Newspoll estimates and its qualitative data estimates in a consistent way without running into too many “house effect” issues that may be occurring in the Newspoll weighting.

PMDISAT(-1) – which is the previous months average of the Prime Ministers dissatisfaction rating using Newspoll data.

OPPRIMARY(-1) – which is the Oppositions primary vote in the previous month using Newspoll data. This lets the forecast ALP TPP vote adapt to the size of the ALP primary vote.

Then these two, which are probably the two most important variables in the model and fill very specific rolls.


What this represents is the difference between last months PM dissatisfaction rating and last years PM dissatisfaction rating, but is only modelled during the month of an election.

So what it effectively does is modify the forecast of the model only in months that an election is on, and does so on the basis of the size of the long term change in the PMs dissatisfaction rating.

Similarly, our other complicated variable is:


What this represents is the recent medium term change in the Opposition leaders satisfaction rating. It’s the difference between last months satisfaction rating and the satisfaction rating of 3 months ago – but is only modelled during the month of an election.

What it effectively does is modify the forecast of the model only in months that an election is on, and does so on the basis of the size of the medium term change in the Opposition leaders dissatisfaction rating.

What these two variables do is simulate the process of voters coming to a conclusion about who they will vote for in the month of the election, using long term changes in satisfaction and dissatisfaction with each party and its leader. It allows for “it’s time” factors and “he has certainly improved over the last year” and “he’s getting worse as time goes on” and “he wasn’t what I thought he was like” type factors to be accounted for in terms of the way they influence voter movement in an election, but through an error correction type mechanism.

So the Election Forecasting Model is:


And we’ll use ordinary least squares regression to do the number crunching which turns out as:


What is important here is that all of these variables are statistically significant. This model explains about 76% of the variation in the Newspoll estimate of the ALP TPP vote since 1997, but it’s built with the aim of being more accurate for the election date than it is at other times via those two long looking variables.

Onto more of the forecast stats:


That’s mainly for the stats people that shows the model does its forecast job extremely well, with very little overall error.

The forecasts this model produces don’t exhibit a lot of the polling overshoot that Newspoll experiences when a new leader comes along, or a Tampa and S11 shocks the system. But it still tracks the changes in the TPP vote for the ALP as we can see with the following graphic:


The blue line represents the forecasts the model produces for each period, whereas the red line shows the actual Newspoll TPP vote for the ALP that the model is attempting to predict. The model misses the troughs and peaks of most of the big volatile movements in the ALP vote because the underlying dynamics of satisfaction ratings, primary vote level, and importantly, the slow moving ACNielson in the previous month don’t support the overactive Newspoll during these periods.

So how good is the model using previous elections?

In 1998, the model predicted an ALP TPP of 50.82 whereas the actual result was 50.91

In 2001 the model predicted an ALP TPP of 49.15 whereas the actual result was 49.07.

In 2004 the model predicted an ALP TPP of 47.23 whereas the actual result was 47.20.

It’s actually more accurate at election times than ordinary periods because those two little complicated variables were arrived at to simulate the processes involved in voters coming to a decision in the campaign. 2001 and 2004 were very different elections with movements going in opposite directions during the campaign period, but the model estimated both results fairly accurately by any measure.

I took the approach of rather than building some error correction components into the model for every period, it only needed to be done for specific periods when elections occur. And it’s probably also worth mentioning that the model predicts each months vote based on last months figures.

So what is the forecast for the election?

An ALP two party preferred result of 55.15%

How that will split between the States will come (hopefully) by Monday.

AddThis Social Bookmark Button

add to kwoff

Posted in Election Forecasting, Leading Indicators, Polling, Pseph, Voting behaviour | Tagged: , , | 91 Comments »

The race against the cashrate clock

Posted by Possum Comitatus on November 8, 2007

Since the interest rate rise is the news of the week, I thought we might go over the timing issue of how long it takes for a rate rise to flow through.

The big problem here is that the rate rise doesn’t coincide with the polling cycle in a consistent manner, so sometimes polls will be taken a day after the rise, sometimes nearly a fortnight after the rise – and for us here trying to get a statistical grip on whether the rate rise will come through in the next three weeks, well it all becomes a little cloudy.

But first, let’s go to what we do know.

We know that when a rate rise happens, it boosts both the ALP primary vote and the TPP vote, we saw that the other day.

If we use the monthly averages of Newspoll, we can regress the change in that monthly average (if the TPP goes from 52 to 54, the change would be +2 for example) with the change in the cash rate (if the cash rate rises from 6.5 to 6.75, the change would be +0.25 for example). When we do that we find that there isn’t a statistically significant relationship between the change in the ALP TPP vote and the change in the cash rate during the month that the RBA lifts rates. But if we lag the change in the cash rate by one period, where we are measuring the relationship between the change in the ALP TPP vote with the change in the cash rate in the previous month, out pops the following:

Where D(TPPALP) = the first difference of the ALP TPP vote, which is the monthly change in the ALP two party preferred vote
D(cashrate(-1)) = the first difference of the cash rate lagged one period, which is the change in the cash rate in the previous month.


The coefficient on D(cashrate(-1)) is what we are interested in here, and it tells us that when the cash rate lifts by 1 percent, the ALP TPP in the next month jumps by 7 points. So when the cash rate lifts by 0.25 percentage points, the ALP TPP vote increases by over 1.5 percentage points. The t-Statistic and the Prob value tell us that this relationship is highly statistically significant.

The little graph in the mix shows how it plays out, where the black line represents a change in the cash rate in the previous month, and how it plays out with the change in the ALP two party preferred vote. Notice how the black line spikes when the red line spikes? That’s the visualisation of the relationship we are measuring.

So we would ordinarily expect that the ALP would get a boost in December from the November rate rise.

But when we try to bring that resolution of the data down to the poll by poll level, we run into the problem of the polling cycle not being consistent with the RBA cash rate announcements.

If we graph the change in the cash rate (the blue triangles) against the change in the ALP TPP using every Newspoll since the last election we get:


As you can see, when the cash rate changes, sometimes the ALP vote goes up with it, sometimes the ALP vote goes up the next poll and sometimes the poll after that.

The problem here also, is that there is a fair bit of random movement (sampling error) that might be polluting the monthly change in the ALP TPP vote that can’t really be cleaned up unless we use something like monthly averages.

So the big question becomes whether or not the election campaign compresses the reaction time of voters when it comes to a change in the cash rate?

Let me state to begin with that I don’t believe a word of this guff about how interest rates will shift the focus back to the economy and that will somehow benefit the Coalition. I’d like to see some evidence of that proposition, and the Newspoll “better economic managers” results aren’t evidence as there’s no statistical relationship between the Newspoll economic management question and the Coalition vote at all, and only a small, marginal relationship for the ALP.

The big question based on the actual historical evidence is one simply of time – a lack of it for the ALP or just enough for the Coalition.

AddThis Social Bookmark Button

add to kwoff

Posted in Leading Indicators, Polling, Voting behaviour | Tagged: , , | 87 Comments »

You tell me which is more likely?

Posted by Possum Comitatus on November 7, 2007

Rates are up. You tell me which scenario is more likely?



AddThis Social Bookmark Button

add to kwoff

Posted in Leading Indicators, Voting behaviour | Tagged: , , | 87 Comments »

Why it’s all about John

Posted by Possum Comitatus on November 6, 2007


This was me in Crikey yesterday.

Since February, the Coalition political strategy has played out on the ground as an attempt to focus attention on Rudd. Whether this has been more by accident than design is probably worth pondering as well, but for all the “look at Kevin” programs, not a great lot has been achieved.

From Rudd dining with Brian Burke , his childhood memories, his links to those union blokes that keep turning out the lights, right through to actual policy programs like education, environment and infrastructure initiatives (that we now know, courtesy of the infamous Crosby Textor Oztrack 33, actually worked in the Labor’s favour by highlighting issues that the ALP had position dominance on) – the strategy that actually played out on the ground was one of focusing attention on Rudd.

Yet for all those attempts at focus shifting, and for all electoral diversional therapy involved, the key measures that matter continue to be intimately linked to the performance of John Howard himself.

The Coalition two party preferred result continues to be intimately linked to Howard’s satisfaction rating since the 2004 election.


A few quick regression equations and a granger causality test on the relationship between these two series suggests that it is the change in PM satisfaction levels that leads to changes in the Coalition two party preferred vote, rather than the two series moving together as a result of third party influences.

The punditry may say that Howard is still extremely popular considering his satisfaction ratings, but with his satisfaction ratings being so intimately linked to the Coalitions TPP vote, that line of thinking quickly becomes a bit a grand non-sequitur in the general scheme of things. His satisfaction only needs to fall small amounts to have a serious impact on the Coalition vote.

The other key measure intimately linked to Howard’s performance is the ALP primary vote via the PM dissatisfaction rating, as we hinted at last week.


Again, after a few quick regression equations and a granger causality test on the relationship between these two series, it is the change in PM dissatisfaction levels that leads to changes in Labor’s primary vote, rather than the two series moving together as a result of third party influences.

What is also interesting to note here is the gap that has recently opened up between these two measures, suggesting the possibility that Rudd is starting to gain support as a result of what the ALP is actually doing, rather than simply relying on dissatisfaction with Howard to deliver them electoral support.

So while all the policy noise and political advertising fills the political brainspace of the nation, when it comes right down to it, this election is still all about John Howard.

The big danger however is that hint in the last chart that suggests that Labor might finally be gaining support on the basis of their own merit. If that relationship starts to consolidate, there will be very little that Howard can do to turn his electoral fortunes around. When you are staring down the barrel of electoral annihilation, that is probably the last thing he wants to hear.

AddThis Social Bookmark Button

Posted in Crikey, Leading Indicators, Polling, Voting behaviour | Tagged: , , , | 4 Comments »

Why Rates Matter

Posted by Possum Comitatus on October 26, 2007


Me in Crikey yesterday HERE

And below:

Yesterdays post “Rates of Gloom” was actually a more in-depth spin-off of this post, so for those of you struggling to follow the stats, this should make everything a lot clearer.

In Rates of Gloom we just modelled the relationship we see below, but also accounted for the effect of leadership changes (the leadership dummy variables) and the honeymoon period that both leadership changes and interest rate rises enjoy (through the time trend variable). Just as the leadership changes of Latham and Rudd increased the ALP vote before it declined from that peak slowly over time as the honeymoon ended, so to with an interest rate rise. After a rate rise, the ALP vote moves up, but then slowly over time it pulls back slightly. There are more complicated ways to model that type of behaviour, but the time trend variable did the job adequately.

So if you read this post first and keep the above in your thought orbit, for those of you not big on the stats side of things it should make Rates of Gloom a bit easier to follow.


With interest rates and widespread navel gazing about the political consequences taking up the media-space, today instead of picking the verbal lint from our bellybuttons over the issue, how about we go to some spiffy little charts that sum up perfectly the millions of words that will be written over the next month.

First up, let’s run the RBA cash rate against the PM dissatisfaction rating over the period 1999-2007, using Newspoll monthly averages for the latter:


Now how is that for a snazzy little leading indicator!

Next up we’ll run the cash rate against the Opposition primary vote over the same period using Newspoll monthly averages:



One word sums that up – Ouch.

Now for the relationship between the Opposition primary vote and the PM dissatisfaction rating:


And that sums up the debate – three graphs are worth a million words.

The only question becomes whether interest rate increases lift the Opposition primary vote directly, whether it increases the Opposition primary vote via PM Dissatisfaction or whether it works via both channels?

As far as the Coalition is concerned however, it’s probably a moot point.


I was just informed of a pretty spiffy site:

Soapbox is a unique Australian political archive of
federal elections, bringing together key historical documents and
audio-visual material, and making them available to students, researchers,
journalists and the general public.”

It’s a cracker of a site if you are into historical election material. Very, very, very much worth a visit:


AddThis Social Bookmark Button

add to kwoff

Posted in Crikey, Leading Indicators, Voting behaviour | Tagged: , , , , | 65 Comments »

The Proportional Application

Posted by Possum Comitatus on October 5, 2007

In our last Newspoll breakdown, we divvied up the seat swings according to the Newspoll estimate of three seat types, and by State for 139 seats across the country. That gave some people psephological indigestion because the results didn’t look like an election would at the margins of safe vs marginal seats. Fair enough, but it was never meant to – it was a breakdown of Newspoll to the best resolution available from the data.

This time, we’ll actually set out to create an approximation of what the Newspoll data was saying that isn’t limited by the data resolution, using basic probability theory and an assumption of uniform state swings. We’ll do today what some people thought we were doing the other day.

To start with, the TPP swing is calculated as the percentage change in vote of the two major parties. Hence if a seat moves from 50% to 55%, it’s had a 5% swing.

If we have 3 seats that are, in ALP/Coalition terms: 40/60, 50/50 and 60/40 and the average swing in these seats is 5%, if the people that are changing their vote are drawn randomly from those three seats, we would expect that the seat with a higher proportion of coalition voters will have more Coalition votes that changed, and the seat with less coalition voters would have less coalition votes that changed.

This is simply as a result of a random distribution of Coalition voters changing their votes. As a result, the 60/40 seat would have (0.05*40*2)-40 coalition voters = 36, and (0.05*40*2)+60 ALP voters = 64.

Doing this for the 50/50 seat we end up with a TPP of 55/45 and for the 40/60 seat we end up with a TPP of 46/54. The seats with a larger number of coalition voters swing more, on a given uniform swing to the ALP (here 5%) than seats with a smaller number of Coalition voters. This is played out in nearly every Newspoll marginal seat/safe government seat/safe ALP seat estimate and most elections.

If we apply the various State swings using the same methodology to the 139 seats we are looking at here, we end up with the following

  NSW Vic Qld SA WA
Actual Newspoll Swing 9.2 11 9.1 9.4 4.4
Calculated Swing 9.4 11.2 10.4 10.2 4.9
Ratio 0.9747 0.9826 0.8759 0.9215 0.9029
Adjusted Swing 8.97 10.81 7.97 8.66 3.97

The Calculated Swing is (using NSW as an example) what the average swing in NSW is calculated as after we’ve applied the Newspoll estimate to the individual seats using the methodology outlined above. Notice how it’s higher by a bit across the board? That’s the variation in the uniform swing interfering. So we can adjust the swing downward for this by multiplying the Actual Newspoll Swing by the ratio of the Actual Newspoll Swing to our Calculated Swing. This gives us our Adjusted Swing. When we apply our adjusted swing to all 139 seats, the State averages of the seats all tally up to the actual Newspoll state estimates. By doing this, we are choosing to minimise the size of the swing we use to account for the small variance in the random distribution of changing voters that Newspoll is picking up in their polling.

We can see how the minimising of the swing plays out by comparing our results in three seat types against the Newspoll results for the three seat types:

  Newspoll Swing Our Swing
Marginal Seat Swing 8.3 8.14
Safe Government Seat Swing 11.6 10.18
Safe ALP Seat Swing 7.1 6.83
National 8.8 8.78

We are smaller in every case of the three seat types, but bang on the State and National average swings. So now we have our conservative estimate that is actually well within the Newspoll estimates, we can apply that to the 139 seats in our list to get an approximation of how the election results would have looked had an election been held between July and September. Remember, as we are using a slightly smaller set of swings than Newspoll, this is a slightly conservative estimate. Instead of all 139 seats, we’ll just look at those seats where the ALP TPP vote was estimated to be above 48%

Division State 04 Election Current TPP Estimate Swing
Indi Vic 33.71 48.0 14.3
Cook NSW 36.72 48.1 11.3
Kalgoorlie WA 43.7 48.2 4.5
Forde Qld 38.48 48.3 9.8
Hume NSW 37.16 48.4 11.3
Fisher Qld 39.02 48.7 9.7
Ryan Qld 39.58 49.2 9.6
Leichhardt Qld 39.74 49.3 9.6
Dawson Qld 40.01 49.6 9.6
Greenway NSW 38.65 49.7 11.0
Warringah NSW 38.71 49.7 11.0
Macarthur NSW 38.85 49.8 11.0
Bowman Qld 41.1 50.5 9.4
Dickson Qld 41.11 50.5 9.4
Aston Vic 36.85 50.5 13.7
North Sydney NSW 39.96 50.7 10.8
Hinkler Qld 41.66 51.0 9.3
Wannon Vic 37.63 51.1 13.5
Gilmore NSW 40.59 51.2 10.7
Flynn Qld 42.28 51.5 9.2
Petrie Qld 42.55 51.7 9.2
Casey Vic 38.65 51.9 13.3
Hughes NSW 41.45 52.0 10.5
Stirling WA 47.96 52.1 4.1
Flinders Vic 38.89 52.1 13.2
Longman Qld 43.25 52.3 9.0
Hasluck WA 48.18 52.3 4.1
Menzies Vic 39.33 52.4 13.1
Herbert Qld 43.76 52.7 9.0
Goldstein Vic 39.97 52.9 13.0
Sturt SA 43.2 53.0 9.8
Blair Qld 44.31 53.2 8.9
Kooyong Vic 40.42 53.3 12.9
Robertson NSW 43.13 53.3 10.2
Cowper NSW 43.25 53.4 10.2
Dunkley Vic 40.62 53.5 12.8
Paterson NSW 43.68 53.8 10.1
Higgins Vic 41.24 53.9 12.7
Boothby SA 44.63 54.2 9.6
Page NSW 44.54 54.5 9.9
Gippsland Vic 42.3 54.8 12.5
Dobell NSW 45.16 55.0 9.8
Bennelong NSW 45.87 55.6 9.7
Moreton Qld 47.17 55.6 8.4
McEwen Vic 43.58 55.8 12.2
La Trobe Vic 44.17 56.2 12.1
Eden-Monaro NSW 46.73 56.3 9.6
Lindsay NSW 47.08 56.6 9.5
Corangamite Vic 44.68 56.6 12.0
McMillan Vic 45.01 56.9 11.9
Wentworth NSW 47.49 56.9 9.4
Deakin Vic 45.03 56.9 11.9
Bonner Qld 49.49 57.5 8.1
Makin SA 49.07 57.9 8.8
Wakefield SA 49.33 58.1 8.8
Parramatta NSW 49.17 58.3 9.1
Kingston SA 49.93 58.6 8.7

Let me also provide evidence from the last three elections on the basis of the random distribution of voter change. If we run a simple scatter against the size of the swing in seats vs the margin those seats were held by, and a run a simple regression through them, we get:


The Margin axis on the bottom is positive for Coalition seats and negative for ALP seats. So an ALP seat with a margin of 5% would be represented as -5 in the chart, and a Coalition seat on a 5% margin would be +5 in the chart. Likewise with the swing, a positive swing means a swing to the Coalition, and a negative swing is a swing to the ALP. The red line is the regression which helps show the general relationship between margins and swings. There is certainly lot’s of variance there – that’s why we adjusted our swings downward in the above estimates to accommodate for the general consequences of the variance at a state level. The scatters are also quite interesting in their own way.

AddThis Social Bookmark Button

Posted in Leading Indicators, Polling | 32 Comments »

Name that issue!

Posted by Possum Comitatus on September 12, 2007

Since we’ve been talking ‘issues’ lately, what are you folks out there picking up as the current talking point memos of both the major parties?

I can see a few with the Coalition:

Heading in the right direction.

We have a strong and experienced team

The full employment economy.

Labor do things because the PR company Hawker Britton tell them too (started yesterday I think)

The ALP have the usual:

Clever politician

Howards vision of the future goes as far as the next election.

But they seem to have been a bit slack in generating new clichés lately.

I’d be interested if anyone could add to the list as they hear them, it’ll help get a handle on any campaign strategy change by both parties. The first hint of a strategy change comes simply through the message.

Different messages are aimed at different demographics, what I might not hear could well ring like a cowbell with others, so it will be interesting to see how people pick the different messages up.


AddThis Social Bookmark Button

Posted in Leading Indicators | 44 Comments »

For Whom the Inertia Tolls

Posted by Possum Comitatus on August 12, 2007

In-er-tia : the property of polling by which a political party retains its voting level at a state of rest, or its voting level along a straight line so long as it is not acted upon by an external force….

…… such as the actions of its political opponents.

Take a squiz at this:

The ALP and Coalition TPP swings, by month, since the last election using Newspoll and where the bottom axis is the month/year:

c2pps11.jpg a2pps11.jpg

There is no doubt that Rudd has made an impact upon the ALP vote, but that impact does not appear to be the orthodox narrative that gets peddled around the pages of some of the daily political commentariat.

Back in the pre-Rudd days, those long forgotten days when the Coalition occasionally won a poll (and strangely enough, when I used to be able to get a decent Caesar Salad) - the seeds of the Howard governments destruction appeared to have already been sown.

Bit by bit, polling point by polling point had Puntersville turned on Howard in the years since the 2004 election. The trend away from the government was clouded by the small recoveries, but each recovery continued to claw back less support than they had lost.

Not only can it be seen in the swing graphs above, but it can also clearly be seen in the TPP vote itself. If we regress the Coalition TPP vote on a constant and a time variable that starts at 1 for November 2004, 2 for December 2004 etc right through to 34 for August 2007, and graph the results we get the following:


R-Squared = 0.838 and the time variable has a coefficient of 0.306 with t-stat of -12.9 and a p-value of 0.0000. This tells us that a linear decline in the Coalition two party preferred vote of 0.3% per month explains approximately 84% of the movement in the Coalition TPP vote since the last election, at an extremely high level of statistical significance.


Likewise if we do the same for the ALP TPP vote:


As expected, we get the same result. R-square= 0.838, with the time variable having a coefficient of +0.306.

But if we throw in a Rudd dummy variable that has a value of 0 for the months he wasn’t leader and a value of 1 for the periods Rudd was the leader of the ALP and run the regression again we get:


Here we get an R-squared= 0.87, Adjusted R-Squared=0.86, with the time trend variable having a coefficient of 0.24 and the Rudd Dummy variable coefficient having a value of 2.02.

This tells us that the ALP were gaining 0.24 percentage points a month (and the Coalition losing the same) without Rudd, and that once Rudd was elected leader he brought to the table an extra 2% of the TPP vote for the ALP.

If the ALP TPP was already growing, how can one explain the massive surge in the ALP primary under Rudd?

Simple – the ALP TPP was getting boosted off the back of the preference flows from the growing minor party vote until Rudd came along. After that, the minor party vote declined and many of those votes that ended up with ALP through preferences came across to become ALP primary votes instead. All the while the ALP TPP vote continued to increase as it had before.

Longer term readers here will already know this to be the case. Voters started deserting the Howard government a long time ago, but parked their primary vote with the minors and preferenced the ALP.

If we take the difference between the ALP TPP vote and the ALP primary vote, and graph that difference against the minor party vote – they will follow a similar path and shape if what we are saying here is true.

So let’s do it and see:


Lo and behold.

Which gets us into this nonsense about Rudds vote being somehow “soft”.

The ALP TPP vote before Rudd was soft, as it relied on a large amount of preference flows from a large amount of minor party voters. But now those voters have come across and put their primary vote with the ALP instead of their preferences. That isn’t a soft vote – it’s as hard as it gets. It’s a primary vote. So the next time you read some dullard waxing lyrical about the soft Rudd vote – remember this:

What that person is really saying is that there are a number of voters which could easily switch back, not to the minor parties from whence they came and where they used to preference the ALP, but back to the government that they deserted over the last 3 years, mostly starting after the 2005 budget and whom have gone out of their way to vote for anybody BUT the government.

The voting behaviour running against the government has been inertial. The TPP trends show that clearly, and what the Rudd leadership has done is merely change the nature of that inertia.

Where before Rudd it was just a general retreat away from the government, it is now a general charge to the ALP. It has changed from people not voting FOR the government and letting preferences flow to the ALP via the minors, to people voting for the ALP directly.

For the government to win the election, not only do they have to turn the vote around, but they have to turn around 3 years of momentum running against them in 3 months.

BTW – I’m back now, so I’ll get stuck into answering the comments from tomorrow.

AddThis Social Bookmark Button

Posted in Leading Indicators, Polling, Voting behaviour | 36 Comments »

Poll Wars: Episode 1 – The Phantom Metrics

Posted by Possum Comitatus on July 11, 2007

Over in The Oz, Newspoll CEO Martin O’Shannessy writes:

I was stimulated to consider whether Dennis Shanahan was right when he interpreted the turnaround in John Howard’s better prime minister rating as something more than merely encouraging. How could this be a turning point in the campaign if voter intention has not moved in spite of the Prime Minister’s improved ratings?

The question is whether the data supports the view that a turnaround in Mr Howard’s better PM rating presages an improvement in the Coalition’s electoral stocks. The short answer to this question has been yes in the past three elections.

The fact that anyone could get stimulated over Baghdad Bob is concern enough – but I suppose that’s what Martin has got to say. But what is of real concern is that last paragraph. Let us focus on the phrase that is the glue of Martins piece; “whether the data supports…”.

We’ll get to some interesting regression models later on that completely tear this argument a new one, but first off let’s look at whether the Preferred PM ratings have clearly moved over the periods leading up to the last three elections: Below are the graphs of the Preferred PM rating for Howard and the Government Primary Vote over the 1998, 2001 and 2004 periods. The dashed vertical line is the election in each graph.



Well 2 out of three ain’t bad. Since we are looking at the effect of Preferred PM from June to July onwards for an election somewhere around November, and taking into account the margin or error on these polls which is plus or minus 3%, we be confident that in 1998 and 2001, Howards Preferred PM rating went up as the election approached, but in 2004 we simply cannot say that at all. In fact, in 2004 the only thing that can be said is that Howards PPM rating didn’t move in any way that could be classed as statistically significant between April and October of 2004.

Before we move on any further, I’d like to show you a couple of interesting things.

Firstly, what it is we are actually talking about:



In the 1993 election, Hewson was preferred PM over Keating in both February and March and lost. In the 1996 election, Keating was Preferred PM over Howard in February and March and lost. The theory of Preferred PM being the great deterministic force behind election victories isn’t looking too crash hot.

Second, have a squiz at Preferred MP ratings for the Opposition and Government overlaid against their respective satisfaction ratings (using Newspoll data):



From this we can see that PM satisfaction levels and his preferred PM status pretty much walk hand in hand, but the same cant be said for the Opposition. What is also worth taking a look at here is how the preferred PM rating from Howard became slightly decoupled from his satisfaction rating around mid 2005.Let’s take a closer look:


The reason for this is simple – there are more complex dynamics at play. The period directly after the 2005 budget saw around 5% of the Coalitions primary vote move over to the Minor Party+Undecideds camp.See Determining the Swinging Voter and Their Behaviour for a detailed explanation of this.

But what’s important here is to notice the consequences of that movement carry across into preferred PM ratings. Howards satisfaction ratings went down, people were shifting their primary vote to the minor parties and into the Undecideds camp but they still preferred Howard over Beazley. Howard kept the PPM status by a powerful lack of alternatives.Then along came Rudd and bang, PPM changed.


Now, keeping all that in your thought orbit, let us move on to test the other key argument made by Martin:

The question is whether the data supports the view that a turnaround in Mr Howard’s better PM rating presages an improvement in the Coalition’s electoral stocks.”

Martin is saying that the Preferred PM rating is a leading indicator for the Coalitions primary vote.

This we can test – in depth. We’ll use the following variables:

C = a constant that is handy in these equations to measure the base value of the dependent variable (GOVPRIMARY in this case) that isn’t explained by movements in the values of the independent variables. This is automatically calculated by least squares regression software.

GOVPRIMARY = governments primary vote

PPMGOV = Preferred PM rating of Howard

And we’ll sometimes use (-1) or (-2) after those variables to represent a lagged variable i.e. PPMGOV(-1) is a variable that whose current value is last months value of PPMGOV.

And we’ll use the sample period of the Howard governments reign from April 1996 through to July 2007.

For those of you with a statistical bent – the following is a “cut down for a wider non-stats audience” affair, so please refrain from giving me grief. I’m sacrificing minutia for readability.

To start with lets run the obvious regression which explains how much of the government primary vote can be explained by changes in the previous months value of PPMGOV (which is Martins argument – what we are testing is whether PPMGOV(-1) is a leading indicator).


This suggests, very very superficially (I’ll demolish it in a minute), that Martin may be somewhat right.22.7% of the movement in the governments primary vote can be explained by movements in the previous months PPM rating.It suggests that a 1 point increase in this months PPM will lead to a 0.21555 increase in the governments primary vote next month.

Now let us run the obvious next regression which explains how much of the government primary vote can be explained by changes in the current months value of the PPM rating for Howard.


This suggests that 30% of the movement in the governments primary vote in any given month can be explained by changes in the value of the PPM for Howard in that month .So, superficially, this suggests that changes in PPM values walk hand in hand with changes in the governments primary vote rather than being leading indicators.

Now let’s run both variables together:


When we combine the dynamics of both the lagged value of PPM and the current value of PPM for Howard something stands out. The lagged value of PPMGOV called PPMGOV(-1) is not statistically significant. This means that is doesn’t actually explain any statistically significant movement in the government primary vote when taking into account the influence of the current value of the PPM rating for Howard.



There is another problem here tied into this. This equation is auto-correlated up the wazoo. This means that there is an awful lot of lagged explanatory power still laying around in the residuals of the equation that is messing with our explanatory variables. If we look at the correlogram for that last equation we get:



This means, in this particular case, that the previous value of the governments primary vote has a big influence on the current value of the governments primary vote (the inertia effect that you often find in voting estimation series) and is interfering with out modelling. We can make another regression which accounts for this using a one period lagged value of GOVPRIMARY:


This clears up all the autocorrelations and gives us a nice white noise process in the residuals which is exactly what we want.

What this new, clean equation tells us is that the 57-58% of the movement in the value of the governments primary vote can be explained by changes in the values of the 3 key variables of GOVPRIMARY(-1), PPMGOV and PPMGOV(-1).

However, far from Howards preferred PM rating being a leading indicator, it’s the opposite.

Put simply, what this means is that if we hold all the other independent variables constant, a one point increase in the last periods value of Howards preferred PM rating will lead to a 0.21 point reduction in the governments primary vote!

That is EXACTLY THE OPPOSITE of what Martins theory tells us should happen.

Preferred PM ratings for Howard are simply not a leading indicator for the governments primary vote, period. Their lagged effect is smaller than the effect of the PPM ratings current value, and that lagged effect is completely the wrong sign.

Howard’s PPM rating is covariant with the government primary vote, in that the two values move together through time, likewise the lagged values of the government primary vote are a leading indicator for futre values – but the lagged PPM values mean diddly squat.

I can appreciate that Martin tried to defend The Shanahans piffle over interpreting polls, but some piffle is simply not worth defending.

His last paragraph sums it up:

So, will the PM drag the Coalition up with him on the strength of his personal rating and will that be enough to overcome Labor’s commanding lead? The next few Newspolls will answer these questions.”

Watch out for the next Newspoll! Buy the Australian to get it!

Same bat time, same bat channel! ;-)

Which is fair enough.


AddThis Social Bookmark Button

Posted in Leading Indicators, Polling | 22 Comments »

Modelling the Howard Governments Primary Vote Swing

Posted by Possum Comitatus on July 3, 2007


The above graph measures the primary vote swing of the Howard Government since its inception in 1996.The way it works is quite simple, each of the dashed vertical lines represents an election. The blue line represents how far away the governments’ primary vote is from the primary vote it received at the election most recent from the observation. So if we take the observation of March 2001 as an example (the bottom of that negative spike in the shaded blue area called “One Nation residual effect), that tells us that the Howard government in March 2001 was, according to the Newspoll estimate of primary voting intentions, experiencing a 3.5% swing against their primary vote based on the primary vote they achieved at the 1998 election. This is a result of the government receiving a primary vote of 39.5% at the 98 election, and getting a Newspoll estimated primary vote of 36% in March 2001.

For anyone that would like a little more of an explanation of the swing graphs, you might want to head over to Part 1 and Part 2 of Into The Swing of Things to get some more detail.

But what I want to focus on today is explaining the primary vote swing of the Howard government since its inception in 1996.The yellow shaded area above called “One Nation Vote Effect” represents the impact on the governments’ primary vote that One Nation was responsible for. Once the One Nation “Party” was created (in its various types of organisational structures), the governments estimated Newspoll primary vote took a beating as conservative votes moved across to One Nation from the Coalition.

But because the above graph measures the primary vote swing, One Nation had a lasting impact on the swing beyond the 1998 election. As One Nation caused the governments primary vote to be a low 39.5% at the 1998 election, and the “swing” measures the difference between any given Newspoll and the primary vote received at the previous election – that low government primary vote at the 98 election caused by One Nation had the effect of artificially inflating the swing size for the period between the 98 and 2001 elections.

As a result of this, any modelling of the swing must take into account not only the direct One Nation vote effect (which was voters moving from the Coalition to One Nation) but also the residual One Nation effect (which is the low government vote at the 98 election caused by One Nation that flows through to the artificially high swing numbers for the government primary vote from the period November 1998 through to October 20001).

I’ll use two separate regression equations to model the government swing. First a model that calculates the swing based on changes in the Newspoll estimation of the Prime Ministers satisfaction rating, a cubic time trend and two dummy variables to account for the One Nation vote effect and the One Nation residual effect.

The second regression models the government primary vote swing on the changes to the Prime Ministers satisfaction rating, a linear time trend and three dummy variables accounting for the One Nation vote effect, the One Nation residual effect and a dummy variable representing the “Rudd Effect” (which will estimate the change in the government primary vote swing that can be explained by Rudd becoming leader in December 2006).

For the first model, it can be represented as:


Where PVSGOV=Government Primary Vote swing at time “t”

PMSAT=Prime Ministers satisfaction rating at time “t”

The “t” variables represent the cubic time trend and the two One Nation dummy variables are self explanatory.

In the model output below, “time” is represented as “HTIME”, meaning it starts at 1 in the first month of the Howard government and increases by 1 for each month his government has been in power.




There’s a lot of explanatory power in the model (even accounting for degrees of freedom for those of you with a stats bent).Graphically combining the modelled behaviour against the actual government primary vote swing, and looking at the “residuals” which are the differences between the model behaviour and the actual values for each observation we get:


Before we analyse this, let’s do the other regression model so we can look at them both together. This second model we are basically replacing the cubic time trend with a linear time trend and an explanatory variable for the Rudd leadership, represented as:




Again, this model like the first explains quite a lot. About 86-87% (accounting for degrees of freedom) of the change in the governments primary vote swing can be explained by time trend movements, PM satisfaction levels and dummy variables.

The fact that both a cubic time trend and a linear time trend combined with a Rudd variable explain a hell of a lot of the governments primary vote swing over recent times suggests to me that Rudd isn’t the driving factor for the ALPs recent vote juggernaut, but rather Rudd is merely the vehicle for allowing the underlying sentiment against the government that has been around for years to manifest.

The media pundits and parts of blogsville have been acting all surprised that the electorate have somehow “turned” on the government out of the blue. But as has been demonstrated here before time and time again, it’s not out of the blue. There has been a consistent and growing swing away from the governments favour since the 98 election with the swings to them trending less and less positive, before turning negative, then turning against them in larger magnitudes.

They nearly lost in 1998 – securing less than 50% of the two party preferred vote.

They nearly lost in 2001, with some serious porkbarreling and the Tampa/S11 riding to their rescue, giving them a primary vote spike which was volatility off the longer term trend running against them at the time.

Over 2001-2004, 5% of the government primary vote went AWOL and stuck mostly with the minor parties and a bit to the ALP, but the ALP under Beazley and particularly Crean couldn’t grab those AWOL coalition votes. Latham came along and grabbed not only that 5% of ex-coalition voters parking with the minors, but also 3% extra from the Coalition primary vote. Latham imploded and that 7-8% went back to the minors and the Coalition. Not because of Howard, but because of Latham.

Approximately 30% of swinging voters deserted Howard after the 2005 budget, taking 5% off the governments primary vote which went straight to the Minors and undecided camp (and none of them have come back since), and another 3% thereabouts shifted between the ALP, the Minors and the Coalition on a regular basis.

Along came Rudd and the underlying trend away from the government crystallised into an ALP primary vote lift (rather than the minor parties +undecideds where it had been hiding for the last 5 years to varying strengths).

The primary swing models above, plus the approximations of the swinging voter movements, plus the time trends of the primary votes, plus the larger swings for the parties all point to the turn against the government not being new at all, but being a larger, longer trend way from the government where people have been looking for a long time to place their votes elsewhere than the incumbents.

Beazley never cut it with the electorate after 2001 – every piece of PRIMARY data shows that, from the ALP primary vote numbers to the primary vote swings, to the minor party primary vote numbers. Voters often left the Coalition, but after 2001 they never went near Beazley again.

Crean never cut it with the electorate, period. The Coalition lost voters under Creans time, but they went to the minors. Only a bloated two party preferred estimate coming off large minor party preference flows made Crean seem not completely and utterly hopeless. TPP polling estimates are dodgy at the best of times – which is why I don’t touch them with a barge pole.

Latham had the electorate and imploded.

Rudd on the other hand cuts it with the electorate and like Latham, only an implosion will stop him from becoming PM, as only an implosion can give the government the volatile spike they need to gather enough votes from the longer term time trend in electoral support that is running strongly against them.

Will Rudd implode?

I can’t see a handshake of doom coming from the man, nor signing giant cardboard pledges guaranteeing interest rates or any of the other political flapdoodle encephalitis that inflicted the brains of Latham and the advisors around him at the time.

As the primary vote swing models above show clearly – once you account for One Nation effects and take into account the non-linearity of the PM satisfaction levels where he’s becoming more disliked over time, the trend in the swing of the governments primary vote is, and has been for years, running against them.

AddThis Social Bookmark Button

Posted in Election Forecasting, Leading Indicators, Voting behaviour | 8 Comments »

IPDI and the Primary Vote Redux.

Posted by Possum Comitatus on June 30, 2007

Over in The Oz, George Megalogenis writes, “THE existential divide between real and imagined financial stress can decide the coming election. Well-off Australians can be convinced to toss out the Government on the false assumption that their living standards have gone backwards, whether through the spectres of Work Choices, interest rates or climate change.”

He’s spot on – but he’s missed the economic measurement that ties most of this up in a neat little bundle that longer term readers here know about well.

Interest Payments to Disposable Income

So let’s redo the two key graphs again. First the Newspoll estimations of the Opposition primary vote (on the left hand side) and the Interest Payments to Disposable Income percentage (on the right hand side) going back to 1985.


Next let’s do the Newspoll government primary vote estimation (on the left hand side) and the Interest Payments to Disposable Income (on the right hand side, log scale and inverted).


Source: http://www.rba.gov.au/Statistics/Bulletin/B21hist.xls

What’s important here is not necessarily the growth in the size of interest payments to disposable income, but how how that growing debt servicing obligation impacts upon discretionary (rather than disposable) income for a key demographic- middle income earners with 1.5 jobs and a large mortgage.That’s where it bites in the self-perception of living standards for mortgage holders as discretionary spending is what funds lifestyle, and lifestyle is a key self-perceived yardstick of household standard of living. As a greater proportion of disposable income keeps flowing to debt servicing, that leaves a smaller proportion of income for discretionary spending.We discussed the ins and outs of this at length over here for anyone interested and whom may have missed it the first time.

The Howard government may have many problems on its hands, but its this one that will cause them the most electoral grief.

AddThis Social Bookmark Button

Posted in Leading Indicators, Political Risk | 4 Comments »

Election Prediction Model 1 Update

Posted by Possum Comitatus on June 21, 2007


Here’s what it means and how it works for those that dont know.

The June Newspoll data rolled in, was fed into the model and not much changed.

The predicted ALP primary vote for the election reduced from 45.6 in May to 44.54 in June and the predicted Coalition primary vote increased from 39.7 in May to 40.04 in June.

The predicted ALP 2 party preferred vote for the election decreased from 53.7 in May to 53.02 in June, while the Coalition 2 party preferred vote increased from 46.3 in May to 46.99 in June.

AddThis Social Bookmark Button

Posted in Election Forecasting, Leading Indicators, Uncategorized | 5 Comments »


Get every new post delivered to your Inbox.

Join 57 other followers