Possums Pollytics

Politics, elections and piffle plinking

Foolwatch – The Power of Information

Posted by Possum Comitatus on July 4, 2008

The Rudd government has made a mistake.

Not just a mistake on data analysis, nor just a mistake on managing information flows, nor even “just” a mistake of implementing a policy that has no evidential foundation.

They’ve made a mistake that will undermine their entire policy agenda for the next three years unless they rectify the process responsible for creating it.

We all remember the FuelWatch saga – where the ACCC assured us that their modelling was correct (even though they refused to release the relevant data), where they assured us that FuelWatch in WA reduced the price differential between Perth and the Eastern Capitals even though there were those of us that thought the data, at least the data we could independently scrounge up regarding petrol prices at the time, didn’t really show any such thing.

“Evidence based policy” was how FuelWatch was spun. A noble cause in and of itself – I’d imagine we’d all prefer policy to be based on evidence, it sure beats the alternatives .But what if the evidence the policy was based on was so questionable that the line between “evidence based policy” and “policy based evidence” – that style of political management where data is interpreted according to the needs of the policy rather than policy being designed to meet the needs of the evidence – became indistinguishable?

Professor Don Harding, economist at the Department of Economics and Finance at La Trobe University , was kind enough to give us a peek at an as yet unpublished draft paper he’s currently working on called “Foolwatch- A Case study of econometric analysis and evidenced-based-policy making in the Australian Government

He has kindly allowed his draft version of the paper to be downloadable here and Don encourages anyone with any views on its content, particularly the econometrics, to contact him with feedback via his details which can be found on his page linked to his name.

Professor Harding went through the painstaking task of pulling the actual data out of this ACCC supplied chart that we’ve talked about here previously, in order to model it. There’s quite an irony here – the ACCC refused to release the data for a whole lot of basically spurious reasons, but made the mistake of releasing a graph of the data which determined folk like Don could actually use to extract the very data the ACCC were trying to hide.

What he found was disturbing.

Far from the ACCC econometric modelling that was used by the Rudd government to justify their longstanding Fuelwatch proposal being robust, it was misspecified and incorrectly tested. Their use of the nominal retail margin as a variable rather than the real retail margin is inconsistent with standard econometric approaches to this kind of modelling and their explanations of what their data actually represented were seriously lacking in specificity – to the point where it was difficult to derive just what it was they were actually measuring, how they were measuring it, how they tested it -and this uncertainty has lead to all conclusions drawn from the ACCC research to be seriously questioned.

Don Harding found that on the evidence available to the ACCC, the conclusions drawn by the Commission – that FuelWatch did not increase petrol price margins in WA – is in fact false and that no such conclusion can actually be derived from the data once it is modelled correctly. At most, the average reduction in the real price margin due to Fuelwatch is less than one third of one cent per litre, but could statistically be between a 1.01 cent per litre reduction, through to a 0.43 cent per litre increase using an orthodox 95% confidence interval.

Don also goes to great lengths to point out that it isn’t the fault of the ACCC econometricians here, and this is something that I wholeheartedly agree with. The data monkeys aren’t responsible for the problems here – far far from it.

It is the process that is at fault, and those whom manage that process.

The conclusions that Graeme Samuel was feeding to the media when the FuelWatch shitfight was happening were incorrect, because the modelling he was basing them on was incorrect. It cannot be proven that FuelWatch in WA did not increase petrol price margins.

Labor’s “evidence based policy” spiel over FuelWatch was nothing but political spin – but it was probably not deliberate spin. Wayne Swan no doubt believed that the modelling he received was accurate. Yet the problem was that the modelling in question that became the basis of the Labor political justification was created in an environment of zero-transparency.

And this gets us back to how the Rudd government has made a mistake that will undermine their entire policy agenda for the next three years unless they rectify the process responsible for creating it.

At the moment, the Rudd government is following a very astute and responsible technocratic process for the high volumes of future policy delivery they will engage in over the next three years.

“You cannot manage what you do not know” is the basic currency of good policy development.

So Rudd, correctly, instituted large numbers of reviews and government inquiries to ostensibly gather information and make recommendations so that the government will be in a position later this year to formulate policies armed with information on observable reality.

Despite the bleatings over these reviews from the shallow end of the media pool in this country with their profound ignorance over the pointy end of politics, the army of inquiries and reviews initiated by the government is, in and of itself, a necessary requisite for the type of “evidence based policy” program Rudd has been stating he will pursue.

But the big problem here, and one that will (and I say “will” pretty confidently) derail this policy program is the way in which the Federal government, their departments and their agencies treat third party access to the very data whose analysis often becomes the basis of policy recommendations.

We’ve got this enormous communications technology infrastructure that enables the efficient and near instantaneous aggregation of knowledge and expertise being effectively sidelined and ignored by political and management practices that are 15 years out of date – but done at their own peril.

The gatekeeping of information by departments and agencies used to be possible, having the public treat unseen internal analysis of that data as gospel from which policy was recommended was also generally accepted – but those days are gone.

What has just happened here with FuelWatch, a fairly comprehensive debunking of the analysis that was used to justify a relatively irrelevant piece of policy, will increasingly happen to other areas that are far, far from irrelevant.

The reason it will increasingly happen is simple – there is a greater number of interested policy specialists, analysts and general expertise that is external to government than there is within government. While this has pretty much been the case over the last 15 years or so, what differentiates then from now is that the external expertise can easily be aggregated and organised at virtually zero cost by the online world and the results of their independent analysis can be distributed widely to a very large, highly influential and still rapidly growing audience.

If the Rudd government is actually interested in “evidence based policy” rather than descending into the world of orchestrating ‘policy based evidence’– they need to adopt a data accessibility regime where as much of the data that is the basis for policy recommendations is made available to the wider public at the earliest possible time in the policy development cycle. Likewise, departmental and agency analysis must be released publically for scrutiny.

The FuelWatch saga is the perfect example of the need for such a data treatment regime.

Under an “evidence based policy” approach, the government wouldn’t have stated that FuelWatch was going to be implemented; they would have said that here is one possible proposal – will it work?

Then they would not only have commissioned the ACCC to do research, but made the relevant data publically accessible at the same time. This way, the external expertise would have had their say, it would have been in the public domain getting refined, praised or smashed under the burden of scrutiny, and the best pieces of research would have been propelled to the top of the pile under the power of their own merit -all before the ACCC research was completed.

If the full ACCC research was then released for public scrutiny once it was finished, the larger external expertise would have not only highlighted the inaccuracies and poor methodology of the ACCC analysis, but highlighted independent competing analysis that would have killed off a poor policy initiative before it was ever implemented.

Yet now the government is facing the ultimate embarrassment of not only getting slugged because the ACCC analysis on which they relied was wrong, but also from having to wear analysts, and consequently the media, pouring big buckets on the policy in the near future when the results of the policy will be measured and most likely demonstrated to be a failure.

The two benefits of this approach to third party data access are simple. Firstly, it’s a far superior political risk-management approach. The worst thing a government can do is implement a policy that becomes a failure – making the data whose analysis becomes the basis of policy recommendations publically accessible not only increases the likelihood that poor data analysis and modelling will be fingered before it gets a chance to pollute government policy (as well as being widely publicised as doing just that), but also provides the government with a zero cost alternative resource from which they can pinch and co-opt the better bits at their leisure.

Secondly, the competitive effects between internal and external analysis will reduce lazy analysis or analysis tweaked to favour certain agendas unrelated to actual policy outcomes, particularly from government departments and agencies which, as a result of basic human nature, often get influenced by random bouts of empire building and inter-agency and inter-departmental pissing contests.

The big excuse that always gets dragged out about now on why such a thing cant happen, that the government doesn’t have the resources nor time to review the external analysis, is usually said by people that have close to zero understanding of the way the crowdsourcing of information in today’s technology and communications rich environment actually works in practice.

The agencies and departments won’t have to follow the external debate – the external debate will make itself known quite comprehensively when departments get it wrong, and when better ideas are available. At the end of the day, while there may be large volumes of expertise available external to the government – in reality only a small amount will be deployed on any given piece of data or policy, with larger amounts being deployed critiquing that external analysis which is where the value of distributed and aggregated knowledge comes in. The government simply won’t be flooded with hundreds, let alone thousands, of competing pieces of data analysis – they’ll just be made aware of the best few, which is really all it takes.

If the government believes that evidence based policy is truly desirable, then they need to open up the relevant data to third party access. Policy development in this country will be far better for it, the quality of public debate will be far better for it and over the longer term, the political fortunes of the government will be far better for it.

I encourage everyone to have a squiz at Professor Don Harding’s draft paper (and it is only a draft paper at this stage) – even though some of it is econometrics heavy, most of it isn’t, and it makes for a damn fine read about the key issues that surround “evidence based policy” and it’s possible pitfalls when not undertaken properly.

AddThis Social Bookmark Button

add to kwoff

30 Responses to “Foolwatch – The Power of Information”

  1. Ronin8317 said

    I have read the paper. This will provide for ample ammunition for those seeking to junk FuelWatch, which would be a real shame. The data model that ACCC should have used is to compare the average per litre of a motorist in Western Australia who used FuelWatch, compared to a motorist who doesn’t.

    Something happened in Western Australia which caused their petrol price to fall, when compared with the Eastern states. It ccur around the time when FuelWatch is introduced. Most motorists in Western Australia sees that FuelWatch is partially responsible.

    I have yet to see any economic theory which will explains how giving buyer more information leads to the seller putting up prices, whereas the reverse is always true. Given how the fuel retailers already have a system to check up on fuel prices by other retailers, what harm can be caused by giving motorists the same information?

    I hope that I won’t see another paper by the same professor on ‘the myth of Global Warming’ coming out next..

  2. Possum Comitatus said

    What happened in WA that brought the price down was competition – the price dropped after the entry of Coles and to a lesser extent Caltex, but before FuelWatch started, so public sentiment effectively treated FuelWatch with all its public relations fanfare as being responsible for something that it actually wasnt fully responsible for, and at best was only very very slightly responsible for if that.

    There’s no harm at all in giving consumers the same pricing information as retailers use – but there’s more to Fuelwatch than just that, there’s price locks as well.

    But the big point here is the government was using faulty modelling as evidence to push a policy through – but faulty modelling based on data that were it released to public scrutiny at the time, would have been independently analysed and have prevented incorrect modelling to be used as evidence for the policy, and could possibly have had an impact on the design of FuelWatch before it went to Parliament. A revamp of the policy actually based on evidence.

    But now the government is stuck with a policy that wont make a rats arse of difference if the results imitate the WA results, and Labor will be looking like a bunch of dills over a policy embarrassment that was entirely avoidable in the first place if they just opened up public access to the relevant data to begin with.

    It might not matter much over a marginal issue like FuelWatch, but it really will matter over the substantial issues Labor will be dealing with over the next couple of years if they dont realise their mistakes.

  3. Zaf said

    I am All About the evidence based policy, but holding any Govt strictly to that standard would imply that their policies can and should only achieve stated objectives. That doesn’t seem realistic. Fuelwatch seems like a PR exercise undertaken in response to the public’s ‘do something or start losing byelections’ when confronted with rising petrol costs and the consequences of a ten year shortfall of investment in public transport/coherent urban planning. Fuelwatch made the Govt look ‘responsive’ without distorting the economy and overstretching the budget. Other options might be less neutral, with long term negative consequences for other outcomes (such as spending on health/education, investment in public transport/sustainable cities, and increased subsidies slowing down the transition to a more fuel efficient economy). The point is moot, of course, now that they have been busted as incompetent liars and propagandists. Something that could have been avoided if they’d done their homework and picked a properly researched PR project with which to woo public approval.

  4. Zaf said

    PS Btw, what is this little picture thing that appears in the upper right hand corner of the comment box?

  5. Ronin8317 said

    Now a question – when did Coles start selling petrol in WA? Coles didn’t get into the petrol game until 2003 in NSW, and that is after Woolworthes joined up with Caltex. The result is the almost total annihilation of independent petrol stations. I would have thought such a move actually brings prices up, not down.

  6. Possum Comitatus said

    It’s an autogenerated avatar for those that don’t have their own. You’ll notice that if you make multiple comments in a thread, your little monster thing is always the same, and different from everyone elses.

  7. Roger said

    Spot on Poss – clear thinking and time to understand the issues and drivers are what is needed. Mt advice to Kev is – stop running 24/7 and start thinking about unintended consequences – because that is what will bury you.

  8. Just Me said

    Agree completely about transparency and allowing multiple third party expertise to have a say (and for free! How much better does it get than that?).

  9. johng said

    Very astute comment Possum. But. I think you overestimate the number of analysts out there who have the capacity and time to analyse the terabytes of data available or that could be made available. Australia is pretty thin on the ground, especially if you want analysts who understand the institutional context for a particular issue as well as how to manipulate the data.
    In principle you are correct though. A new ball game is emerging.

  10. Jovial Monk said

    So, Treasury should make available the data used to do modelling for an ETS scheme?

  11. We sure need it.

    The Murray Darling Basin 10 year plan was done on the back of an envelop in Howard’s office. Rudd just follows it. Brumby scorns any evidence for making a shift to a sustainable agriculture, Rann engages in spin, Iemma just wants the commonwealth to carry the reponsibility for NSW’s over allocation of water, Wong hides under a rock and Garrett contemplates his ego. All the scientific evidence about ecological collapse is ignored.

    The Murray Darling is a greater example of really bad evidence policy than Fuelwatch. And this strategy of evidence based policy acting as a veil to cover naked, self-interested state politics has been going on for a decade.

  12. Sinclair Davidson said

    I agree with Possum. The Rudd government were sold a lemon by the ACCC. This raises all sorts of questions about the ACCC and in particular Graeme Samuel. He has been very aggressive in selling FuelWatch. His position IMHO is now untenable and the Rudd government could quitely let him go. His contract is up.

  13. Sir Ian Upton said

    No one every accused our politicians of being bright or clever. As someone said we, the voters, should appoint establish our own FoolWatch charged with keeping an eye on the idiots in government.It should be noted that WA motorists do enjoy cheap petrol. The only trouble is they have to drive 4000 kilometres to the Eastern State to fill up.

  14. Harry "Snapper" Organs said

    How likely is it that Rudd will see your analysis, do you think, Poss?

  15. Topher said

    One intersting thing about the 10 billion murray plan was that there was a few hundred million in there for the Bureau of Meteorology to collect, and publish all of the water data for Australia – exactly what Possum is saying should happen more.

  16. ruawake said

    Possum, correct me if I am wrong, but wasn’t the ACCC (2007) report on petrol pricing set up by Peter Costello?

  17. Sinclair Davidson said

    The ACCC inquiry into petrol pricing was set up by Costello. That report did not recommend FuelWatch – it did include a discussion of FuelWatch and the now notorious Appendix S. The ACCC did say this about a FuelWatch-type scheme (and two other proposals).

    In the end, the ACCC decided that in the time available it was not possible to fully review all the options
    with regard to their administrative implications, effects on competition or their likelihood of delivering the objective of increased price transparency. A detailed assessment addressing these issues would have to be made before government could confidently embark on any one of the suggested options.

    That detailed assessment has not been undertaken. The Assistant Tresurer Chris Bowen appeared to suggest that the ACCC had undertaken additional analysis, but when it was released to the public it was extremely meagre – up to the point where the name of the econometric test actually performed was not revealed, nor any of the significance statistics. When asked in Senate Estimates the ACCC still could not (or at least did not) reveal the name of the test. Writing in the SMH a few weeks ago Mr Bowen indicated that Graeme Samuel approached him with the national FuelWatch idea – so this is Samuel’s initiative.

    (Quick correction to a comment above: Fuelwatch started in WA in 2001, Coles came in 2004).

  18. ruawake said

    ACCC 2008, Petrol— Further Econometric Analysis Undertaken by ACCC,
    29 May 2008

    Fuelwatch decision by cabinet.
    15th April 2008

    It appears to me that the Govt made the fuelwatch decision well before the “Further Econometric Analysis” was available.

  19. Sinclair Davidson said

    Good spot – although the 29 May date was a press release date for the analysis, it might have been done (long) before. Bowen had previously told the Parliament that he had seen the additional analysis and had been briefed and any MP could organise a briefing if they so wanted.

  20. Possum Comitatus said

    They did Ruawake – a classic case of policy based evidence. “Here’s the policy – quick, invent some evidence”

    Not a good look when you’re about to embark on a big, so called “evidence based policy” push.

    They should have looked at the evidence before they decided on Fuelwatch, not tried to drum some up after the decision had already been made.

  21. ruawake said

    Informed Sources would be the big loser if fuelwatch was introduced – they were vocal in their opposition after the announcement.

    But they are the only source we have on historical petrol prices and it was this data that the ACCC had to rely on.

    “Our clients can be confident that all data including historical information is securely maintained.” and it was until the ACCC was given to power to demand it.

    Just to throw another confounding factor in – how accurate is Informed Sources historical data? Is it complete?

    Can we say the data is correct? CI 95% (p=0.001) 🙂

  22. smokey said

    Fuelwatch has always been a lemon IMO. It doesn’t take some long rave from a rocket scientist to know that. The basic facts of a finite resource and climate change will always see it go up in the long term. Arguing about 1cent or 5cents a litre is meaningless. I dunno why the gov ended up in such a stupid shit fight over it in parliament.

    Whether or not Possum the gov will adopt an even more rigorous approach to policy development remains to be seen, but I’d expect voters would at some stage demand the gov to FFS just make a decision and stop asking everyone else to do it for you.

    The whole Fuelwatch debacle has been a stuff up by the gov. Luckily it’s irrelevant in the grand scheme of things. I doubt they’ll be so dumb in the future.

  23. ruawake said

    While I totally agree with what you have said Possum I see changes in future.

    Foolwatch was a political decision – not based on any data, becuase there was none. Maybe the reason the ACCC did not release the data it had was that it was the intellectual property of Informed Sources.

    Future policy based on data collected by the Commonwealth should be different. Foolwatch was always going to be reviewed after 12 months – maybe because the data would be more robust?

  24. Ronin8317 said

    On a different tangent completely – given that you only have one data set, with nothing to compare against, how can you say with confidence that A caused B?

    The one thing that is not in doubt is that petrol in Western Australia is cheaper than Eastern Australia ‘most of the time’ after 2001 is introduced, whereas it was more expensive before 2001. During that period, Fuel watch was introduced. If that is all the information you’re given, then FuelWatch causes petrol prices to be lowered is the only logical conclusion. What other event may have caused it? Change in taxation? Coles did not enter the market until 2004, so it’s not competition.

    In regard to using ‘nominal’ vs ‘real’ prices, wouldn’t using the difference as a ratio of the price be enough? You can’t use CPI data because that’s a quarterly figure.

    One thing is certain – if FuelWatch is implemented, I will have information on the cheapest petrol station in my region, and I’ll save money compared to a motorist who doesn’t use FuelWatch. How is that a bad thing?

  25. Grumps said

    Good post Poss,

    I am not trained in the economic play ground that many that visit this site are, (I do try my best Poss to broaden the horizon).

    Not arguing the veracity of the the data and method (I dont understand it) the point I take is that the “new gate keepers” of such information are still acting as though El Rodent is in residence. Knowledge to the ordinary person might lead to a situation where that average person can make an informed decision!?!?!.

    I have read Professor Don Hardings Draft and the bit that really works for me starts at page 23 chapter 5 through to his conclusion.

    The MSM is still making the mistake of treating the average aussie as unable to handle large amounts of data/information on an issue and simplify it to reflect their own ideology. Obviously they believe the liberals are so inept that their assistance is required to return the rightful rulers to the government benches

    A couple of MSM correspondents have reported the average australian has been given the Rudd government a unique opportunity to effect substantial change on a number of global and local issues.

    I hope Rudd takes to heart the open government ideals that the professor and you talk about.

  26. Don said

    Harry “Snapper” Organs @14

    “How likely is it that Rudd will see your analysis, do you think, Poss?”

    my question too, Harry.

    Does anyone at Rudd HQ read Possums Pollytics?

    I sincerely hope so.


  27. sandgroper said

    As a West Aussie you need to understand that Fuelwatch everyday is part of the news cycle. Everynight on the news we are given where the cheapest petrol is in our area and also just about radio news bulletin provides fuelwatch information. We are even warned a day in advance that a fuel company is putting up their petrol by x cents a litre. If you missed all this you can then go to the website to get the information by putting in your postcode and getting the cheapest fuel near you.

    This has got to be good for the consumer and not the fuel companies. I think once the punters on the east coast the benefit of this against the existing setup KR will be on a winner.

  28. Sinclair Davidson said

    The most practical criticism of Don has is that the price margin is not adjusted for inflation. When he does so, the FuelWatch effect is not significant. A potential criticism is that he extrapolated the quarterly CPI data to a weekly series. I have just run a regression with quarterly petrol price adjusted by the quarterly CPI and can confirm Don’s result. The Coles effect, however, remains significant. So increased competition, not FuelWatch benefitted consumers more.

  29. happy chap from Griffith said

    Good point re: crowd sourcing policy analysis. Makes sense. There are a whole stack of firms doing this sort of thing under the moniker open innovation (e.g. http://www.innocentive.com/). Hopefully government can learn from some of these successes.

  30. Nigel said

    What I do not comprehend about this whole argument re fuelwatch, is; I do not see how it could possibly have reduced the average price of fuel. I would not expect it to, but on my way to and from work I can stop at any one of 4 service stations, and their price can vary by as much as 10c/ litre on any given day so if fuelwatch were to be instigated here (Canberra) I could concievably save a substantial sum of money, and isn’t that what it is all about? ie Not about the average, but about the consumer being able to locate the best price on the day?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: