31 May 2023

Interview with Dominic Knight, ABC Drive, Sydney

Note

Subjects: Australian Centre for Evaluation

DOMINIC KNIGHT:

The press release from Andrew Leigh who is the Assistant Minister for Competition, Charities and Treasury, as well as Assistant Minister for Employment, says the government is creating an Australian Centre for Evaluation to measure what works. How would it measure what works? Andrew Leigh. Good afternoon.

ANDREW LEIGH:

Good afternoon, Dom. Great to be with you.

KNIGHT:

What are you evaluating?

LEIGH:

Well we're opening up the possibility of evaluations right across the Federal Government. You know, this is something we've been doing in medicine for decades. If you want to get a new drug on the Pharmaceutical Benefits register, you've got to put it through stage one, two, three randomised trials. But randomised trials are relatively rare in policy. The great advantage of them is it gives you a really clear counterfactual Dom. You've got a group that is similar before the intervention to the group that's getting the intervention and so if you see a difference between the two, you know it really is the intervention that's made a difference.

KNIGHT:

It's interesting isn't it? So often in life you have to make a decision between two paths and you never really know what the path not taken would lead to. But you're saying there are some times in public policy where we can try both paths at once and then see which one's better.

LEIGH:

Exactly. So if you're testing a new COVID vaccine, then you don't go off gut feel. You toss a coin, heads you get the vaccine, tails you get a placebo and then we see how those two groups respond when they're subjected to the disease. In the same way, evaluations of early childhood programmes such as the Perry Preschool Program tossed a coin. Heads you got high‑quality early childhood intervention, tails you were in a control group and then we compared over time. And the learnings of that, Dom, really underpinned the understanding that early childhood intervention could transform lives, could reduce teen pregnancies and reduce the chance of children ending up in jail. We can do this in something as simple as sending out a letter. If the Australian Tax Office is sending out a letter now, they'll often randomise and send out two versions of the letter to see which one encourages people to comply with the law best.

KNIGHT:

It's interesting isn't it? I know even Facebook, when they were choosing the colour for their logo, would randomise a whole bunch of different, subtly different blues and the blue that most people clicked on was the blue that they went with.

LEIGH:

Absolutely. They're everywhere in the tech firms. I remember one researcher when I was writing Randomistas, said every pixel in the Amazon home screen has had to justify its existence through a randomised trial. So the tech firms, the leading businesses are using them, we're using them in medicine, but they're less common in public policy. And the great advantage is sometimes they overturn our expectations. So in Western Australia, there was an expectation that if you used an infant simulator or a robo‑baby, if you gave those robo‑babies out to teenage girls, they'd be less likely to end up pregnant themselves. They tried it through a randomised trial and turned out that those who got the robo‑babies were more likely to fall pregnant and become teen mums. The intervention wasn't just failing to work, it was actually backfiring. We only knew that when we rigorously evaluated it. And now, thank goodness, we're not going to be rolling it out on a large scale.

KNIGHT:

So, people talk about cost‑benefit analysis, but that is more of a hypothetical approach, isn't it? This goes a step further from what you're saying and actually tests what works.

LEIGH:

Well cost‑benefit analysis is simply putting dollars around the impact that you measure. But if you've got a garbage estimate of the impact, then putting dollars around it isn't going to give you the right answer. The real advantage of this is we're taking that insight that transformed medicine from eminence‑based medicine to evidence‑based medicine, and we're using that very same approach in policy‑

KNIGHT:

Of randomised control trials? And, as you say, Andrew Leigh, you've written a book about this. So, passionate are you about the notion of trying a) and trying b) and seeing not just what the theories tell us, but what works in practice.

LEIGH:

Yeah, that's right. And I was really struck when I looked across the examples there, Dom, at the way in which randomised trials had evolved. So, the cure for scurvy was discovered by a naval surgeon, James Lind, who finally ran a randomised trial of all of the proposed suggestions of the day, everything from seawater to citrus, and found out that citrus worked. But the one thing they can't do, Dom, is to prevent you from having to run and vote in a division. And frustratingly, my pager has just gone off.

KNIGHT:

Oh, no.

LEIGH:

And I'll have to sprint to the chamber if I'm to be there to cast my democratic vote, and do my democratic duty.

KNIGHT:

So, we can't try a counterfactual where you don't go?

LEIGH:

We could, but I don't think that would be very good for my political career. Sincere apologies‑

KNIGHT: 

That's okay

LEIGH:

‑to your listeners, for having to cut this short.

KNIGHT:

That is democracy in action. Thanks for your time. There you go. So, that's the big idea that Andrew Leigh has been pushing for. He's got funding in the budget for the Australian Centre for Evaluation.