Importance of better evaluation in a more ambitious aid program
As the Australian aid program has increased in size, so has the scope of its ambitions. In large middle-income countries such as Indonesia and the Philippines, Australian aid is testing the merits of policy reforms like school-based management, and changes to curricula. In low-income countries, Australian aid provides an important supplement to limited recipient government resources, supporting recipients to meet their responsibilities in areas such as road maintenance, health services, and policing. Through these and other initiatives, AusAID, is engaged in long-term interventions that seek to affect recipients’ development trajectories. Increasingly, recipients are looking to AusAID to advise them on how they can best address the challenges they face.
The changing scale of AusAID’s ambitions is reflected in its approach to evaluation. Following the 2006 White Paper AusAID realised that given the size and significance of the aid it delivers it is no longer acceptable to assess projects individually against their discrete objectives and deliverables. Rather, AusAID would need to be able to understand and demonstrate whether these deliverables had made a positive difference to the lives of people in poor countries. New performance assessment frameworks were developed and new reporting regimes introduced. The establishment of an Office of Development Effectiveness gave new prominence to the importance of evaluation within the agency.
What the aid review found
The aid review made a number of criticisms about evaluation in AusAID but they haven’t received much comment.
The first target was the quality of evaluations conducted by the agency. The review found that a large proportion of the evaluations completed by AusAID (26 per cent) were not good enough to publish. Compliance with the requirement to conduct evaluations every four years for projects above $3 million was low. Despite its public commitment to disclose evaluation results, AusAID had released very few of its completed evaluations to the public.
The second criticism was about the independence of evaluations commissioned by AusAID. In a field where some failures are to be expected, the review questioned the veracity of the 118 project evaluations it reviewed, none of which had been rated by their evaluators as unsatisfactory. By way of comparison, the review noted that the World Bank’s independent evaluation group rates about 20 per cent of its projects as unsatisfactory. The review considered that the lack of functional independence of the Office of Development Effectiveness (ODE) from the agency had compromised its ability to provide independent commentary. This problem was evident in the increasing delays in the release of its flagship publication, the Annual Review of Development Effectiveness, and the muted criticisms contained in this publication.
The review team concluded that there was little evidence that AusAID’s evaluation reports inform and improve aid delivery.
The review’s recommendations and their implications for AusAID
To address these problems, the review argued that AusAID’s system of mandatory project evaluations should be more flexible with decisions about the conduct, timing, scope and arrangements for project evaluations left to the discretion of management. To counteract the potential loss of accountability from this change of approach the review recommended (and the Government agreed) that ODE should focus on conducting and publishing 10 to 20 high quality and independent evaluations every year. To secure the independence of these evaluations the review recommended (and the Government agreed) that an independent evaluation committee be appointed to oversee the planning of ODE evaluation reports and to approve evaluations.
If implemented, these reforms will precipitate a major change in the focus of ODE, and AusAID’s approach to evaluation. The requirement to produce 10 to 20 high quality reports annually is a major increase in output (AusAID has typically published about four major evaluation reports per year). While offering much stronger protection to their independence, the clearance of these reports through an independent evaluation committee will mean there’s a much higher level of scrutiny over the evidentiary basis for evaluation findings and conclusions. Negotiating evaluation outcomes through the committee is likely to be an exacting and time-consuming task, but if the committee is independent the arguments should all be about quality rather than political sensitivity.
Presumably AusAID will continue to rely heavily on external contractors to complete evaluation reports. However, even if it does so, it will also need to bolster its internal capacity to manage large evaluation projects, to ensure the work completed by its contractors is sufficiently rigorous to satisfy the expectations of the independent evaluation committee. Providing more resources is one part of the solution to obtaining better quality reports, but there’s no silver bullet when it comes to counteracting evaluation biases and producing high quality reports. Strong evaluation expertise and competent management are essential parts of the equation too.
Fulfilling the promise of evaluation
By placing evidence at the centre of policy debates, good evaluations can not only promote accountability but also build consensus amongst stakeholders about the problems they face, and the actions required to address them. The benefits of a good evaluation, when properly reviewed and disseminated, can flow well beyond its impact on the immediate circle of project or program stakeholders. The insights it brings can filter through organisations, helping individuals to understand the problems they face and make better decisions. The impacts can extend to a wider circle of interested parties, including domestic policy makers, recipient government agencies, researchers, other donor organisations and non-government institutions.
Importantly, good evaluations can help to foster a more informed and progressive public debate. With the aid program set to become one of the single largest sources of discretionary spending in the Federal Budget this will be critical to building public confidence in the aid program and its management. The new arrangements proposed by the aid review and agreed to by the government have the potential to bring about a more disciplined and rigorous approach to evaluation in and of the aid program
David Slattery worked on the last two independent reviews of AusAID by the Australian National Audit Office (AusAID’s Management of Tertiary Training Assistance and AusAID’s Management of the Expanding Australian Aid Program). He is currently a Senior Research Fellow with the Australian Council for Educational Research.
Leave a Comment