For the second year running, the US think tanks, Brookings and the Centre for Global Development, have released their ratings of aid donors. QuODA (Quality of Official Development Assistance) assesses donors against 31 indicators divided into four groups: impact on institutions, administrative burden, efficiency, and transparency and learning. I have calculated a composite quality ranking by averaging over the four groups. Using the 2011 rankings, Australia comes 8th out of 23, consistent with the recent Independent Review of Aid Effectiveness finding that Australia’s aid program was “good but improvable.”
Recent reports from the Brookings Institution and AusAID‘s Office of Development Effectiveness (ODE) have delved into the new QuODA ratings to extract their implications for Australia.
The ODE report “The quality of Australian aid: an international perspective” draws largely on the Brookings report. It emphasises one of the Brookings results in particular, namely its comparison with other bilateral donors who, like Australia, give at least half of their aid to countries classified as fragile (poorly governed and/or in conflict). There are six such donors and of these “Australia is second only to the UK” in terms of QuoDA quality as the AusAID ODE website notes.
This is hailed by the ODE report as an “important result” and “key finding”. It shows that Australia is “not an average donor” and that “other donors can learn from Australia’s example.” Indeed, the report argues, once context is taken into account by making a comparison with this more limited group of donors, Australia’s aid program emerges as “world standard” and “among the world’s best.” A Devpolicy blog authored by ODE head, Dereck Rooken-Smith, says that the result shows that Australia is delivering “effective aid in fragile and conflict-affected states.”
The Brookings report, “Benchmaking against progress: an assessment of Australia’s aid effectiveness”, on which ODE’s is based, puts less weight on this one result, and presents more material to balance it (in particular, a critical analysis of Australia benchmarked against Ireland). Nevertheless, the claim that the result in question is an “important” one originates with the Brookings report. Laurence Chandy, the report’s author, goes on to argue that it “indicates that Australia’s efforts to heed good practice principles, despite the challenges posed by the context of its aid program, are paying off.”
Is this “2nd out of 6” result, as we might call it, an important one, as claimed by the two reports? For it to be important, the conclusion drawn from it that Australia is a world leader among donors heavily exposed to fragile states needs to be robust. That is, it should not change if minor alterations are made to the way in which the question is explored. (I use “exposed” here in the sense that a donor which is highly exposed to fragile states is one which has a high share of its budget going to fragile states.)
One problem with comparing Australia with other donors who give 50% or more to fragile states is that Australia sits right at the edge of that group, with just a 50% share (see the graph below from the Brookings report). It would be equally if not more plausible to take as the comparator group the top half of aid agencies in terms of their exposure to fragile states, a group in which Australia comes in the middle, 6th out of 12. But, in this group, Australia also comes 6th out of 12 in terms of the composite QuODA index, which is in fact a worse relative ranking than its overall 8th out of 23.
Alternatively, rather than putting Australia in the group of the top 6 donors in terms of exposure to fragile states, since Australia comes 6th, one might just as well put it in a second group of five in terms of exposure to fragile states. In that group, Austrlia is not the second best but the second worst in terms of quality.
Finally, put Australia not in the 50%+ group of donors for exposure to fragile states, but in the 40-50% group (where it just as well fits), and it comes 3rd out of 6 for quality.
In other words, depending on how the problem is framed, Australia can emerge as a relatively good, average, and relatively bad donor compared to other donors with a similar exposure to fragile states. That shows that the conclusion drawn from the “2nd out of 6” result is not robust. The result itself therefore shouldn’t have been singled out and labelled as important. Nor should any conclusions at all have been drawn from it, let alone been repeatedly highlighted.
Why isn’t the result robust? It turns out that there is no relationship at all between aid quality, as measured by QuODA, and exposure to fragile states. That’s what the second graph (below) shows. (For technical details on the graph, see the notes[1] at the end of the blog.) It is true that three of the five worst agencies in terms of quality – US, Belgium and Italy – also happen to be the three which are most exposed to fragile states. This is what drives the “2nd out of 6” result for Australia. But it turns out to be a fluke. It is not that in general donors which provide more aid to fragile states are worse donors. Rather the picture that emerges is that some donors are better than others, and that some focus more on fragile states than others, but that there is little if any link between the two.
The lack of a relationship between aid quality and exposure to fragile states is surprising as one would certainly expect aid to be higher quality in better-governed countries. There are broader issues with the utility of the QuODA rankings which I won’t go into here. (See earlier Devpolicy blogs – here and here – for a discussion in relation to Australia’s transparency ratings.) But probably the explanation for the lack of a link (or correlation) is that QuODA is not a full measure of aid effectiveness. Rather, it primarily measures things under donor control, such as transparency and selectivity. And there is much less reason to think that these characteristics of aid under donor control would be heavily influenced by the nature of the countries in which the aid is delivered.
It is unfortunate that so much has been made by ODE of a conclusion which lacks foundation. This episode illustrates the need for greater contestability, which would, one hopes, pick up and weed out weak analysis such as this. One way to do this, which I have written about earlier, would be through the establishment of an Independent Evaluation Committee to overseee ODE’s evaluations.
And what about Brookings, the prestigious US think tank? As I meantioned earlier, the Brookings paper is a wide-ranging one, and I am focusing on just one of its results here. Nevertheless, it is unfortunate that there was no testing for robustness in relation to the “2nd out of 6” result . The error is made more serious by the failure of first-published version of the paper to disclose that the Brookings Institution in fact receives considerable financial support from the Australian aid program ($800,000 over two years according to the AusAID website) to work on issues of aid effectiveness.[2]
Stephen Howes is the Director of the Development Policy Centre.
[1] Notes in relation to the second graph: The vertical axis ranks countries by donor agency quality, where 1 is the highest quality and 23 the lowest. This composite quality ranking which I calculate is obtained by averaging the ranks obtained on the four QuODA indicator groupings, and then ranking countries by this composite indicator (excluding multilaterals at this stage). The horizontal axis ranks countries by their exposure to fragile states, where 1 is the highest exposure and 23 the lowest. Exposure is measured by the share of bilateral aid going to fragile states. The QuODA data used comes from the 2011 QuODA brief, and the fragile state exposure data comes from the Chandy Brookings paper, both referenced in the text. The line shows the correlation between the two variables (or, actually, the lack thereof). The Spearman correlation coefficient is a very low 3%. I also looked at the correlation between exposure to fragile states and the four separate QuODA dimensions. All of the correlations are small; two are negative. I haven’t used actual shares of aid to fragile states rather than ranks, because I don’t have the data, but the shares can be estimated from the Brookings paper (Figure 11, reproduced above as the first graph) and their use of them doesn’t change the results.
[2] This has now been corrected.
Thanks for these comments.
Garth raises the posssibility that, if you controlled for other variables (such as the varying reasons for giving aid), you would find an association between exposure to fragile states and aid quality as measured by QuODA. It’s possible, but I doubt it. It would be difficult to find the controls, and, from the graph above, there’s not even a hint of a relationship between the two variables of interest. There is evidence that aid quality is higher in better governed countries. For example, if you look at World Bank project ratings, they tend to be highest in East Asia and lowest in Africa. But QuODA is at best a very partial measure of aid quality, which is probably why we don’t see a correlation between it and exposure to fragile states.
Tom makes the point, I think, that I do in my first paragraph, namely that overall Australia comes 8th out of 23 bilateral donors. I agree that that’s not bad at all, and I stand by the characterization of the aid program by the Independent Review as “good but improvable.” But that is different from claiming that Australia is a “world leader” or “among the world’s best.” No doubt we are in some areas, but there isn’t the evidence to substantiate this as an overall claim.
I’m not sure what to make of the ATPC comment. I do actually think that the best way to look at aid effectiveness is through in-depth case studies. But these need to be balanced and rigorous.
I would think that world standard question that would be answered if there was an Olympic event to gauge each participant in the Aid-game.
In that context Aus-Aid used of Aust.Technical college in the Pacific (http://www.aptc.edu.au/support/index.html) would be a good point to begin with.
The game of using DFAT resources to fund an institution, that is majority staffed by Australians and getting paid using Australian standards of income ($Aus) to these expatriates who enjoy tax free incentives, a generous package of having their Mcmansions, top of the line vehicles and entertainment allowances, and school fees for the kids paid for. Of course these funds are channeled using Australian banks.
Students who travel to host countries are set up with bank accounts with ANZ, Airline travel for students (where possible) is through Qantas. The website of these Technical College’s have their servers in Australia and maintained by Australians. All personal data for these Pacific island students are in these cloud servers in Australia and data mining via echelon is prevalent.
While the local staff are paid the local minimum rate and won’t be upskilled so that these expatriate positions are perpetually held by Australians and even though there are localization mandates in the immigration policies of these island Governments; these plans have been put on the back burner due to convenient excuses of lack of qualified personnel in the host country.
That is the effectiveness of Aus-Aid. Are those practices right in the moral sense?
Hey Stephen, thanks for this analysis. I’m not sure that the way in which you’ve chosen to critique the ODE’s analysis is that meaningful. Of course, the “2nd out of 6” measure might be questionable, but it seems to me that Australia does relatively well in two measures of what constitutes a good aid policy – a) how much of the aid is effective, and b) how much of the aid is going to the countries who need it most. As you say, there will be trade offs between these two measures because aid delivered to fragile states might be difficult to deliver to a high standard. However it seems that AusAID strikes a fairly good balance here.
Thank you Stephen for this post – it highlights the relatively arbitrary nature of the comparison used by ODE. I am confused though by your focus on the lack of association between quality score and exposure to fragile states.There are a large number of factors that influence aid programs and many of these have little to do with maximising aid quality. Therefore is it not possible that it may be harder in both theory and reality to score a high QuODA score when aid is predominantly focussed on fragile states, but that in reality there does not appear to be an association between fragility exposure and QuODA score because of all the other (non-quality) factors shaping donor actions. In other words isn’t the relevant chart one which compares fragility exposure and degree of difficulty in scoring high on QuODA?