The Robustness Checks

Every estimate of what adoption does rests on an assumption that cannot be proved. So the last job is not to produce the answer. It is to establish how much weight the answer will bear.

Sensitivity bounding asks how strong a hidden factor would have to be, in both the adoption decision and the outcome, before it wiped the result out. If the answer is stronger than anything you did measure, the result is not proved, but it is much harder to dismiss.

A permutation placebo reassigns adoption at random, many times over, and re-runs the whole analysis. The machinery should find nothing, because there is nothing to find. A method that cannot return an empty answer is not measuring anything.

Checking that the comparison is possible at all comes before any of that. Every estimate here compares adopters with non-adopters, which requires that households of each kind actually exist across the whole range of circumstances. If every household with land and credit adopted, there is nobody to compare them to, and no estimator can manufacture the missing half. So the estimated probability of adopting is inspected across the sample, the extreme tails where the comparison breaks down are trimmed away, and the estimate is re-run on what remains. A result that moves a great deal when you do that was resting on a handful of unusual households.

Re-running under different choices means different algorithms, different numbers of slices, different random splits. An estimate that survives only one particular set of choices is a property of those choices rather than of the world.

What it cannot do

It is worth saying what is not in this list. The standard remedy for unmeasured confounding is an instrument: something that pushed a household towards adopting without touching its income by any other route. There was not one here. These technologies reached farmers through the market rather than through a rollout with a usable boundary, and a weak instrument does more damage than none, so the honest move was to say so and bound the exposure instead.

None of this converts observational data into an experiment. It makes the size of the leap explicit, rather than leaving the reader to guess at it. That is a smaller claim than it is often made to sound, and it is the honest one.

Further reading

• Cinelli, C. and Hazlett, C. (2020). "Making Sense of Sensitivity: Extending Omitted Variable Bias", Journal of the Royal Statistical Society Series B. Read it.The bounding exercise described in this note.
• VanderWeele, T. and Ding, P. (2017). "Sensitivity Analysis in Observational Research: Introducing the E-Value", Annals of Internal Medicine. Read it.A second way of asking how strong a hidden factor would have to be.

From the Zambia climate-smart agriculture research, built on the Water and Soil Accelerator household survey, which was funded by USAID. The thesis is under examination and the three papers drawn from it are under anonymous peer review, so there is nothing to link to yet.