Evaluation and counter bureaucracy

Following the thread from my blog Monday….

World Bank evaluation expert Aaditya Mattoo warned the Aid for Trade meeting against the tyranny of measurement in evaluation. That we should not avoid programme evaluation simply because we can not perfectly measure the impact of activities.

Former USAID chief Andrew Natsios draws a similar conclusion in his much quoted paper ‘The Clash of the Counter-bureaucracy and Development’ and highlighted today from Oxfam blogger Duncan Green.

Natsios refers to the tension in the development world between the compliance side of aid programs (the counter bureaucracy) and the technical program side and that this “imbalance threatens program integrity”.

He continues that the “counter bureaucracy ignores a central principle of development theory—that those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable.”

What makes this report fun to read is not just the lively debate in the main document but the inclusion at the beginning of an ironic memo from the Duke of Wellington complaining about England’s own version of the counter bureaucracy during the time of the Napoleonic Wars.

“Gentlemen,

Whilst marching from Portugal to a position which commands the approach to Madrid and the French forces, my officers have been diligently complying with your requests which have been sent by His Majesty’s ship from London to Lisbon and thence by dispatch to our headquarters.

We have enumerated our saddles, bridles, tents and tent poles, and all manner of sundry items for which His Majesty’s Government holds me accountable. I have dispatched reports on the character, wit and spleen of every officer. Each item and every farthing has been accounted for with two regrettable exceptions for which I beg your indulgence.

Unfortunately the sum of one shilling and ninepence remains unaccounted for in one infantry battalion’s petty cash and there has been a hideous confusion as to the number of jars of raspberry jam issued to one cavalry regiment during a sandstorm in western Spain. This reprehensible carelessness may be related to the pressure of circumstance, since we are at war with France, a fact which may come as a bit of a surprise to you gentlemen in Whitehall.

This brings me to my present purpose, which is to request elucidation of my instructions from His Majesty’s Government so that I may better understand why I am dragging an army over these barren plains. I construe that perforce it must be one of two alternative duties, as given below. I shall pursue either with the best of my ability, but I cannot do both:

1.) To train an army of uniformed British clerks in Spain for the benefit of the accountants and copy-boys in London or, perchance…
2.) To see to it the forces of Napoleon are driven out of Spain.
Your most obedient servant,
Wellington”

Natsios summarizes the problem with the current compliance system as:

• Excessive focus on compliance requirements to the exclusion of other work, such as program implementation, with enormous opportunity costs
• Perverse incentives against program innovation, risk taking, and funding for new partners and approaches to development
• The Obsessive Measurement Disorder for judging programs that limits funding for the most transformational development sectors
• The focus on the short term over the long term
• The subtle but insidious redefinition of development to de-emphasize good development practice, policy reform, institution building, and sustainability.

Natsios ends, “Let me conclude with one simple question asked in a different form by the Duke of Wellington. Do Washington policy makers wish USAID, PEPFAR, and the MCC to implement serious development programs or comply with the demands of the Regulatory Lords of Washington? They cannot do both.”

Driving a Hummer in Geneva

Today I saw a Hummer with CD plates - should I be pissed off on environmental grounds?

A Hummer uses goes 9 miles per gallon. My Nissan Almeira 42 miles per gallon. All of this rendered meaningless depending on how many flights a year I take.

A Hummer uses up loads of space on the road, pollutes way more than my Nissan and is a nightmare for cyclists and pedestrians. But then I guess the intern who I passed on the bus is probably saying the same about me.

OK, so I can't win the environmental arguments, but it does send out an uncomfortable ethical message if UN employees are buying these monster cars...

Evaluating trade assistance programmes

What Aid for Trade said about evaluation

On Monday I attended the final session of the WTO's Aid for Trade meeting focused on monitoring and evaluation.

Why do we need M & E?

An obvious question but worth asking: electorates are skeptical of development aid’s impact and so want to see independent, scientific studies that demonstrate its impact

· A recent poll in the Financial Times shows that respondents in most OECD countries considered defence and development aid as priority areas for spending cuts

· From the ESCAP delegate, Out of 1 Australian dollar, 67cents ends up in expat salaries

A4T evaluation particularly challenging

The moderator Michael Roberts (WTO) pointed to the success of evaluation in the health sector (i.e. does a dollar spent on health result in lives’ saved) but how can we apply it to A4T? This type of “gold standard” of evaluation seen in health is more difficult in trade due to difficulty in assigning casuality.

Given these challenges evaluation should at least be a learning process, where feedback loops are in place. Fear of making mistakes should be replaced by analysis and learning from them as seen in the private sector (see the new book “Adapt” by Tim Harford on the value of making mistakes).

The wonders of evaluation

Aaditya Mattoo, World Bank expert on evaluation illustrated how powerful evaluation in aid programmes with an example of randomized trials by the Poverty Action Lab and work of Estor Duflo.

Problem: Kids in rural Kenya were not getting sufficient education. How does the government get its biggest bang for its buck investment.

The study show that de-worming kids (In much of the developing world, most kids have intestinal worms, leaving them sick, anemic and more likely to miss school), resulted in 25 percent less absenteeism. The cost of this (35c) compared highly favourably to other policy options like paying for school uniforms or an additional teacher (around US100).

However, such trials are difficult to implement in the A4T context. How can you get robust data sets? How can you assign causality when many other factors are affecting outcomes (e.g. other economic and social policies).

For this reason, he said that A4T was at a primitive stage compared to health in evaluation, but we need to be pragmatic.

The three tyrannies of evaluation

Despite the need for figures showing value for money, Mattoo warned the meeting against what he called the tyrannies of evaluation. These are:

Methodology – i.e. avoiding evaluation as we have not perfected the methodology

Causality – i.e. this programme was wholly responsible for a 10% rise in incomes.

In A4T we can not easily assign casuality but we can draw more qualitative conclusions. For example, an IADB study on a trade capacity building programme could not assign casuality on volume of trade increased but made a useful finding that it increased their diversity of exports. A lesson from another study was that A4T aid should not be directed to the large companies but to the marginal companies.

Measurement – i.e. avoiding evaluating what can not be measured, so we end up focusing on only what we can measure.

At a side event later in the evening the Bank launched a new study Where to Spend the Next Million? Applying Impact Evaluation to Trade Assistance. It looks at how M and E is being applied in A4T.

Why blog?

It has been a year since I blogged....it is time I started blogging again...this is why from Seth Godin

Africa, biotech and climate change

A new report claims that the future of Africas food supply risks being put into the hands of corporate biotech as they develop and patent climate ready crops

Think tank does satire on climate change

Great post from The Onion and CDG on lack of policy action on climate change.

According to a report released this week by the Center for Global Development, climate change, the popular mid-2000s issue that raised awareness of the fact that the earth's continuous rise in temperature will have catastrophic ecological effects, has apparently not been resolved, and may still be a problem.…”

Global warming, if you remember correctly, was the single greatest problem of our lifetime back in 2007 and the early part of 2008,” CGD president Nancy Birdsall said. “But then the debates over Social Security reform and the World Trade Center mosque came up, and the government had to shift its focus away from the dramatic rise in sea levels, the rapid spread of deadly infectious diseases, and the imminent destruction of our entire planet.”

…”I was a bit surprised by our findings, because I, along with the rest of my colleagues, thought that the process of fixing climate change ended soon after [2007 rock concert] Live Earth,” CGD assistant director George Oliver told reporters.

"Last year's federal budget included more than $200 million in funding for the Office of Personnel Management," Birdsall said. "Since nobody really knows what that is, we suggest that money perhaps be spent making sure the oceans don't turn into acid."

Family planning tops for reducing emissions

According to a recent paper by David Wheeler and Dan Hammer, population policy options are much less costly than almost all other reviewed options for low-carbon energy development, including solar, wind, and nuclear power, second-generation biofuels, and carbon capture and storage. They are also cost-competitive with forest conservation and other improvements in forestry and agricultural practices.

This table, based on data in their paper, shows how many tonnes of CO2 would be abated for your $1m:

Intervention	Tonnes of CO2 saved
Family planning & girls’ education combined	250,000
Family planning alone	222,222
Girls education alone	100,000
Reduce slash and burn of forests	66,667
Pasture management	50,000
Geothermal energy	50,000
Energy efficient buildings	50,000
Pastureland afforestation	40,000
Nuclear energy	40,000
Reforestation of degraded forests	40,000
Plug-in hybrid cars	33,333
Solar	33,333
Power plant biomass co-firing	28,571
Carbon Capture and Storage (new)	28,571
Carbon Capture and Storage (retrofit)	26,316