Long term impact evaluation

Michael Clemens shows healthy skepticism toward the impact of the Millenium Villages Project. I concur with his views although what he is asking for is highly unrealistic - a long term impact evaluation beyond 5 years. What he asks for is so unrealistic that if the MVP does not comply (which I predict, it will not) that he has more or less set MVP up to fail.

There are literally no impact evaluations beyond a short time frame - 5 years may even be stretching it. Even clinical trials have problems beyond the 3-year period due to sample attrition. Even the Head Start Impact Study is not scheduled to go beyond 5 years and I would argue that this is an important policy that needs to be carefully studied.

There is also a problem with the MVP project that a randomized trial cannot answer:
"The project deploys a broad package of interventions for five years in each village, including distribution of fertilizer and insecticide-treated bednets, school construction, HIV control, microfinance, electric lines, road construction, piped water and irrigation lines, mobile phones, and several others."

Any time a treatment consists of varying sub-treatments and dosage whose levels are hard to measure you can bet that the even if the impacts were positive, the causal effect remains a black box. Which sub-treatment was more effective? At what dosage? A village level randomized trial will not be able to conclusively answer this because the object being randomized is a village and the sample will be small (even though the number of people in a village may be large).

I would still advocate a long term study. I would not bother with trying to find decent controls at the the time of randomization because the study at this level will have low power. I would however, collect as much data as I can from as many villages as I can. In terms of data collection (if it were in an advanced country) I would try to attain what has been done by the PSID although I would have more observations. Being that this is in Africa, it will be hard to get this done. Does this mean that even if we fall short of the data collected that we should not do it? Absolutely not. The funding requirements are such that someone else besides MVP may have to step in.