Probes that lie
We re-ran 100 published probing results with control tasks. Only 31 kept a clear margin over their control, which is how we checked: each probe was trained twice, once on the real labels and once on labels shuffled by word type, so a probe that only memorised words would score well on both. A clear margin meant at least ten points between the two. The control is now standard in an open evaluation suite, and two later papers report it by default.
With T. Nwosu and J. Lindqvist.


