In 1890, rust fungus wiped out much of Australia’s wheat crop, and the colonies had to import wheat. In response, farmer William Farrer used experiments to try to create a rust-resistant variety. Critics mocked his ‘pocket handkerchief wheat plot’, but Farrer’s hundreds of combinations finally produced ‘Federation Wheat’ – a rust-resistant strain that outperformed all others.
When it came to rescuing the wheat industry, Australia used careful experiments to find out what worked best. But too often, policies are rolled out with little or no evidence to back them up. Some of the worst decisions of recent decades – the Baby Bonus, rorted grants programs, privatisations of state monopolies – emerged from an evidence-free vacuum.
Upon taking office in May 2022, we were shocked to see the lack of evidence behind many of the programs that the former government was pursuing. Only a small minority of programs are supported by any kind of an impact evaluation. Our predecessors commissioned plenty of consultants’ reports, but rarely did a good job of measuring whether their programs actually worked.
Failing to rigorously measure what works can literally lead to people getting hurt. In 1978, an Oscar-winning American documentary film ‘Scared Straight’ showed troubled teens being exposed to life in prison, and suggested that the experience would frighten them away from a life of crime. In a life-imitates-art moment, some US states created programs that emulated the documentary. Yet when the programs were subjected to rigorous randomised trials, they turned out to be increasing rates of juvenile crime. 'Scared Straight' wasn’t just ineffective – it was making the problem worse.
The area of government that is best at evaluating programs is health. When COVID hit, even the brightest experts knew little about this novel virus. Over time, researchers built up an evidence base. As a global community, we learned that aerosols were a key mode of transmission, and that masks could help slow the spread. We learned that steroids could reduce deaths in hospitalised patients. We learned that when infection control processes were in place, the risk of child‑to‑child transmission in schools and day care centres was relatively low. And we learned that vaccines were effective, including against new variants.
In a major report, the Global Commission on Evidence recommended that countries, international organisations and citizens can all make better use of evidence. As one of the 25 commissioners on that report, I worked with people across the world to build a better feedback loop. Everyone makes mistakes – the key is to avoid making the same mistake twice. Good evaluation helps us learn, adapt, and improve the next time around. Just as good coaches watch match replays and good companies ask for feedback on major projects, good governments need to be in the continuous improvement game.
That’s why the Albanese Government is committed to improving the quality of evaluations across government. As Treasurer Jim Chalmers and Finance Minister Katy Gallagher have noted, it’s vital that we evaluate more programs – creating a better evidence base about what works, and what doesn’t. Sure, not every program can be rigorously evaluated, but let’s not let the perfect be the enemy of the good. Right now, the quality of evaluation in government is woeful. There’s a lot that can be done to help government agencies lift their game.
In raising the evidence bar, there’s plenty Australia can learn from other countries. In the US, a non-profit organisation called Arnold Ventures funds randomised trials to test all kinds of interventions, from preventing teen pregnancy to improving policing, and from keeping disadvantaged students at university to boosting assets among low-income families. In the UK, the Education Endowment Foundation supports randomised evaluations on catch-up maths programs, chess clubs, summer reading programs, science instruction and more.
Australia has occasionally run world-leading evaluations. In 1999, a randomised trial of the NSW Drug Court demonstrated that for offenders who were addicted at the time of their offence, the traditional criminal justice process wasn’t the best option. Instead, a specialised Drug Court, which could sentence offenders to drug rehab, turned out to produce better outcomes. Those randomly assigned to the Drug Court committed fewer crimes in the year afterwards – a good result for both the former addict and the community at large. Drug Courts have since been rolled out in other parts of Australia, and the NSW Drug Court trial is often quoted by policymakers around the world.
Knowing what works isn’t ideological; it’s practical. When a randomised trial showed that hydroxychloroquine was ineffective at treating COVID, authorities worldwide withdrew their support for the drug being used on COVID patients. That freed up resources to devote on treatments that did work. But just imagine for a moment if Australia had treated hydroxychloroquine like the former government treated many of its pet projects – spruiking it based on gut instinct rather than hard data. The result would have been medically dangerous and fiscally reckless.
Good evaluation puts our instincts to the test. Like William Farrer’s creation of Federation Wheat, it’s time to take a rigorous approach, and find out what works.