← All analysis tools

Climate-smart agriculture in Zambia

McNemar's Test and the Bonferroni Correction

Only the farmers who did one thing and not the other tell you anything about a preference between them.

A two by two grid of farmers who used practice A, practice B, both or neither. The both and neither cells are greyed out and struck through as carrying no information. The two remaining cells, labelled b and c, are highlighted and compared against each other.

To find out which practice beats which, you compare them two at a time.

McNemar’s test has a feature I find genuinely elegant. It discards the farmers who used both practices and the farmers who used neither, because neither group tells you anything about a preference between the two. Only the farmers who did one and not the other carry information, and the test asks whether those fall one way more often than the other.

The correction

Six categories make fifteen pairs. Run fifteen tests at the usual threshold and you will turn up something significant by luck alone, in roughly the way a run of heads appears if you flip a coin long enough.

The Bonferroni correction handles this by making each test harder in proportion, so the bar moves from about one in twenty to about one in three hundred.

What it cannot do

Bonferroni is blunt and deliberately conservative. It will miss real differences in order to avoid inventing false ones. That is the right trade for a ranking that people might act on, and it is still a trade.

The equation

χ2 = ( b − c )2 ÷ ( b + c )     and     αadjusted = α / m = 0.05 / 15 ≈ 0.0033

b
farmers who used the first practice but not the second
c
farmers who used the second but not the first
not present
the farmers who used both, and the farmers who used neither. They never enter the formula
m
the number of comparisons, fifteen for six practices

The test is the gap between b and c, scaled by how many people disagreed at all. If b and c are equal, the two practices are tied no matter how popular both are.

Further reading

  • McNemar, Q. (1947). "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages", Psychometrika. Read it.The original. The title is longer than the idea.
  • Dunn, O. J. (1961). "Multiple Comparisons Among Means", Journal of the American Statistical Association. Read it.The Bonferroni procedure as it is actually used, rather than as it is usually described.

From the Zambia climate-smart agriculture research, built on the Water and Soil Accelerator household survey, which was funded by USAID. The thesis is under examination and the three papers drawn from it are under anonymous peer review, so there is nothing to link to yet.