Thoughts on randomization

Mostly triggered by Chris Blattman's advice to PhDs:
"The randomized evaluation is just one tool in the knowledge toolbox. It's currently the rage, but that means it will probably be old news by the time you finish your PhD."

One of the problems with randomized trials is that is is a black box. We understand very little or we may think we understand a lot. There is also a lot of potential subgroup interaction that needs to be tested.

All this points to the fact that if we have to do a randomized trial then we don't really understand the mechanism of how the treatment works (and this also applies to medical "science"/drug therapy, etc.). And if it does work to our expectations then it validates our priors and perhaps advances the field a little. But does it really advance our understanding of the causal underlying mechanism? All we can point to are suggestions that our limited understanding is validated but we could still be spectacularly wrong.

Another problem with randomized trials is that it usually is never the last word. (Perhaps repeated randomized trials can provide the last word, but rarely one randomized trial.) Again, this is because if a theory accords with my priors and the results of my hypotheses are rejected it doesn't seem to lower my priors as much as it should - mainly because the "theory" sounds so sensible and plausible. So it must be something with the way the trial is conducted. For instance, the effects of Head Start on children and the disappointing First Year results - yet the underlying premise of Head Start is so strong that it will not go away.

Randomized trials also do not address the question: How will it work for me? And this is particularly true for drugs. I really do not care about average treatment effects of the average treatment effects on my subgroup. And it is this thinking that leads to experimentation and continuing treatment using "less than acceptable" methods or alternative methods. This would be the test of our understanding - if we can predict individual results then we can claim to have the final word on causality.