Thursday 28 July 2011

Impact evaluation in trade projects

The first chapter of Where to Spend the Next Million presented at the A4T Review and edited by World Bank staffers is an essential primer for those interested in Impact Evaluation in trade. Here is an overview of what the authors said.

Impact Evaluation (IE) is challenging to carry out in the trade context but A4T is not exempt from using IE methods. Trade exceptionalism – the notion that trade related interventions are inherently not amenable to IE – is…”groundless”.

The primary concern that the authors address is the attribution problem of project outcomes, i.e. to come up with workable methodologies that can show the impact of the project and what would have happened in the absence of the intervention.

This is a difficult exercise because many outside influences can confound the identification of a programme or policy’s impact. For instance, an export promotion scheme put in place in 2007 would see its positive impact confounded by the negative impact of the global crisis of 2008-9; a simple before-after comparison of outcomes is likely to suggest a negative impact of the programme.

How do you filter out these influences? We would want to know how beneficiary firms would have performed in the absence of the programme (presumably worse). For this we need a data set of firms that benefited from the programme (“the treatment group”) and non-beneficiaries (the control group).

Randomised control experiments (the “gold standard”) can be used when evaluation is built into the design of programs. However, the authors stress that we need not be wedded to RCTs. They discuss the use of Difference in Differences, a methodology that compares differences in outcomes instead of comparing levels. These have been applied for example by the IADB to assess the effectiveness of TPOs in six Latin American countries as well as their agricultural sector projects.

The picture that emerged from the IADB study (in Chapter 2 of the book by Christian Volpe Martincus, listed on the TPO Network) was that export promotion was effective in facilitating export expansion along the “extensive margin” i.e. resulting in greater diversity of exports and they were more useful for SMEs rather than large companies (who did not face such severe information asymmetry problems).

Also of interest is the reference to a study by Lederman on the performance of Export Promotion Agencies which showed through a survey of EPAs across 88 developed and developing countries that EPA services were very important for “overcoming foreign trade barriers and solving asymmetric information problems associated with exports of differentiated goods…” and that there were strong diminishing returns, suggesting “small is beautiful” as far as EPAs are concerned.

The study ends by advocating the “mainstreaming” of IE into trade projects.

“Trade interventions have so far escaped the rising tide of evaluation methods and there is no justification for trade exceptionalism”

The key barriers to progress are not conceptual. Rather they concern incentive issues, as IEs are costly, burdensome, lengthy and not necessarily aligned with project managers’ incentives. In order to overcome these barriers, four avenues must be explored

  1. The burden imposed on project managers should be relieved by making IE a separate exercise carried out by specialists, albeit in collaboration with project managers
  2. Beneficiary governments must buy into the process
  3. Costs should be reduced, for example through building local IE capacities
  4. IE results should prioritize learning over monitoring

“Care is needed in the interpretation of IE results because premature conclusions could easily provoke backlash and because a considerable accumulation of evidence is needed to yield truly valuable new knowledge”

No comments: