Thursday 28 July 2011
Using the F word in a Swiss bank
Impact evaluation in trade projects
The first chapter of Where to Spend the Next Million presented at the A4T Review and edited by World Bank staffers is an essential primer for those interested in Impact Evaluation in trade. Here is an overview of what the authors said.
Impact Evaluation (IE) is challenging to carry out in the trade context but A4T is not exempt from using IE methods. Trade exceptionalism – the notion that trade related interventions are inherently not amenable to IE – is…”groundless”.
The primary concern that the authors address is the attribution problem of project outcomes, i.e. to come up with workable methodologies that can show the impact of the project and what would have happened in the absence of the intervention.
This is a difficult exercise because many outside influences can confound the identification of a programme or policy’s impact. For instance, an export promotion scheme put in place in 2007 would see its positive impact confounded by the negative impact of the global crisis of 2008-9; a simple before-after comparison of outcomes is likely to suggest a negative impact of the programme.
How do you filter out these influences? We would want to know how beneficiary firms would have performed in the absence of the programme (presumably worse). For this we need a data set of firms that benefited from the programme (“the treatment group”) and non-beneficiaries (the control group).
Randomised control experiments (the “gold standard”) can be used when evaluation is built into the design of programs. However, the authors stress that we need not be wedded to RCTs. They discuss the use of Difference in Differences, a methodology that compares differences in outcomes instead of comparing levels. These have been applied for example by the IADB to assess the effectiveness of TPOs in six Latin American countries as well as their agricultural sector projects.
The picture that emerged from the IADB study (in Chapter 2 of the book by Christian Volpe Martincus, listed on the TPO Network) was that export promotion was effective in facilitating export expansion along the “extensive margin” i.e. resulting in greater diversity of exports and they were more useful for SMEs rather than large companies (who did not face such severe information asymmetry problems).
Also of interest is the reference to a study by Lederman on the performance of Export Promotion Agencies which showed through a survey of EPAs across 88 developed and developing countries that EPA services were very important for “overcoming foreign trade barriers and solving asymmetric information problems associated with exports of differentiated goods…” and that there were strong diminishing returns, suggesting “small is beautiful” as far as EPAs are concerned.
The study ends by advocating the “mainstreaming” of IE into trade projects.
“Trade interventions have so far escaped the rising tide of evaluation methods and there is no justification for trade exceptionalism”
The key barriers to progress are not conceptual. Rather they concern incentive issues, as IEs are costly, burdensome, lengthy and not necessarily aligned with project managers’ incentives. In order to overcome these barriers, four avenues must be explored
- The burden imposed on project managers should be relieved by making IE a separate exercise carried out by specialists, albeit in collaboration with project managers
- Beneficiary governments must buy into the process
- Costs should be reduced, for example through building local IE capacities
- IE results should prioritize learning over monitoring
“Care is needed in the interpretation of IE results because premature conclusions could easily provoke backlash and because a considerable accumulation of evidence is needed to yield truly valuable new knowledge”
Tuesday 26 July 2011
Evaluation and counter bureaucracy
Following the thread from my blog Monday….
World Bank evaluation expert Aaditya Mattoo warned the Aid for Trade meeting against the tyranny of measurement in evaluation. That we should not avoid programme evaluation simply because we can not perfectly measure the impact of activities.
Former USAID chief Andrew Natsios draws a similar conclusion in his much quoted paper ‘The Clash of the Counter-bureaucracy and Development’ and highlighted today from Oxfam blogger Duncan Green.
Natsios refers to the tension in the development world between the compliance side of aid programs (the counter bureaucracy) and the technical program side and that this “imbalance threatens program integrity”.
He continues that the “counter bureaucracy ignores a central principle of development theory—that those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable.”
What makes this report fun to read is not just the lively debate in the main document but the inclusion at the beginning of an ironic memo from the Duke of Wellington complaining about England’s own version of the counter bureaucracy during the time of the Napoleonic Wars.
“Gentlemen,
Whilst marching from Portugal to a position which commands the approach to Madrid and the French forces, my officers have been diligently complying with your requests which have been sent by His Majesty’s ship from London to Lisbon and thence by dispatch to our headquarters.
We have enumerated our saddles, bridles, tents and tent poles, and all manner of sundry items for which His Majesty’s Government holds me accountable. I have dispatched reports on the character, wit and spleen of every officer. Each item and every farthing has been accounted for with two regrettable exceptions for which I beg your indulgence.
Unfortunately the sum of one shilling and ninepence remains unaccounted for in one infantry battalion’s petty cash and there has been a hideous confusion as to the number of jars of raspberry jam issued to one cavalry regiment during a sandstorm in western Spain. This reprehensible carelessness may be related to the pressure of circumstance, since we are at war with France, a fact which may come as a bit of a surprise to you gentlemen in Whitehall.
This brings me to my present purpose, which is to request elucidation of my instructions from His Majesty’s Government so that I may better understand why I am dragging an army over these barren plains. I construe that perforce it must be one of two alternative duties, as given below. I shall pursue either with the best of my ability, but I cannot do both:
1.) To train an army of uniformed British clerks in Spain for the benefit of the accountants and copy-boys in London or, perchance…
2.) To see to it the forces of Napoleon are driven out of Spain.
Your most obedient servant,
Wellington”
Natsios summarizes the problem with the current compliance system as:
• Excessive focus on compliance requirements to the exclusion of other work, such as program implementation, with enormous opportunity costs
• Perverse incentives against program innovation, risk taking, and funding for new partners and approaches to development
• The Obsessive Measurement Disorder for judging programs that limits funding for the most transformational development sectors
• The focus on the short term over the long term
• The subtle but insidious redefinition of development to de-emphasize good development practice, policy reform, institution building, and sustainability.
Natsios ends, “Let me conclude with one simple question asked in a different form by the Duke of Wellington. Do Washington policy makers wish USAID, PEPFAR, and the MCC to implement serious development programs or comply with the demands of the Regulatory Lords of Washington? They cannot do both.”
Monday 25 July 2011
Driving a Hummer in Geneva
A Hummer uses goes 9 miles per gallon. My Nissan Almeira 42 miles per gallon. All of this rendered meaningless depending on how many flights a year I take.
A Hummer uses up loads of space on the road, pollutes way more than my Nissan and is a nightmare for cyclists and pedestrians. But then I guess the intern who I passed on the bus is probably saying the same about me.
OK, so I can't win the environmental arguments, but it does send out an uncomfortable ethical message if UN employees are buying these monster cars...
Thursday 21 July 2011
Evaluating trade assistance programmes
What Aid for Trade said about evaluation
On Monday I attended the final session of the WTO's Aid for Trade meeting focused on monitoring and evaluation.
Why do we need M & E?
An obvious question but worth asking: electorates are skeptical of development aid’s impact and so want to see independent, scientific studies that demonstrate its impact
· A recent poll in the Financial Times shows that respondents in most OECD countries considered defence and development aid as priority areas for spending cuts
· From the ESCAP delegate, Out of 1 Australian dollar, 67cents ends up in expat salaries
A4T evaluation particularly challenging
The moderator Michael Roberts (WTO) pointed to the success of evaluation in the health sector (i.e. does a dollar spent on health result in lives’ saved) but how can we apply it to A4T? This type of “gold standard” of evaluation seen in health is more difficult in trade due to difficulty in assigning casuality.
Given these challenges evaluation should at least be a learning process, where feedback loops are in place. Fear of making mistakes should be replaced by analysis and learning from them as seen in the private sector (see the new book “Adapt” by Tim Harford on the value of making mistakes).
The wonders of evaluation
Aaditya Mattoo, World Bank expert on evaluation illustrated how powerful evaluation in aid programmes with an example of randomized trials by the Poverty Action Lab and work of Estor Duflo.
Problem: Kids in rural Kenya were not getting sufficient education. How does the government get its biggest bang for its buck investment.
The study show that de-worming kids (In much of the developing world, most kids have intestinal worms, leaving them sick, anemic and more likely to miss school), resulted in 25 percent less absenteeism. The cost of this (35c) compared highly favourably to other policy options like paying for school uniforms or an additional teacher (around US100).
However, such trials are difficult to implement in the A4T context. How can you get robust data sets? How can you assign causality when many other factors are affecting outcomes (e.g. other economic and social policies).
For this reason, he said that A4T was at a primitive stage compared to health in evaluation, but we need to be pragmatic.
The three tyrannies of evaluation
Despite the need for figures showing value for money, Mattoo warned the meeting against what he called the tyrannies of evaluation. These are:
Methodology – i.e. avoiding evaluation as we have not perfected the methodology
Causality – i.e. this programme was wholly responsible for a 10% rise in incomes.
In A4T we can not easily assign casuality but we can draw more qualitative conclusions. For example, an IADB study on a trade capacity building programme could not assign casuality on volume of trade increased but made a useful finding that it increased their diversity of exports. A lesson from another study was that A4T aid should not be directed to the large companies but to the marginal companies.
Measurement – i.e. avoiding evaluating what can not be measured, so we end up focusing on only what we can measure.
At a side event later in the evening the Bank launched a new study Where to Spend the Next Million? Applying Impact Evaluation to Trade Assistance. It looks at how M and E is being applied in A4T.
