I often use ESP experiments as a vehicle to introduce hypothesis testing to students. One of the learning outcomes is that statisticians are actually easier to convince of claims of paranormal ability than lay persons . On the basis of a binomial calculation, I tell them that
I would be convinced of ESP if they could correctly select which of 5 cards I am holding more than 75 times out of 250 trials. You get 50 by guessing, and another 25 convinces the hard hearted statistician that you can bend spoons.
But I am actually telling them fibs. I would not be convinced. Does this make me irrational?
more–>I recall many years ago hearing of the so-called Soal experiment, which was in a published in book by Soul and Bateman (1954. Modern Experiments in TelepathyLondon: Faber & Faber) demonstrating telepathy. My first reaction to such claims is always rather skeptical.
In fact, the very evidence that ESP researchers use to convince us of ESP can, under mild assumptions, leads us more strongly to the conclusion that ESP is not true. This is beautifully explained in a 2003 book Probability Theory: The Logic of Science by Edwin T. Jaynes, G. Larry Bretthorst.
The experiment in question involved a woman getting 9410 trails out of 37100 correct where the chance of success under random guessing is p=0.2. This is about 26 standard deviations above the null mean and the binomial P-value is of the order of 1 in 10 to the power 139. But I for one am not convinced that she has ESP.
Frequentist hypothesis testing allows of two hypotheses only. If we are limited to two then, in this case, the relevant ones seem to be HG:she is guessing and p=0.2, or HE: she is not guessing and p>0.2. The null (HG) is overwhelmingly rejected in favour of the conclusion (HE) that she has ESP.
Bayesians can also fall for the trap of considering two hypotheses, in which case they end up in the same place as frequentists. Let LE and LG be the likelihood of the data under the null and alternative. Then the posterior probability she has ESP
Assuming that our prior probability of ESP, πE , is small, and that the likelihood of the date under the hypothesis of ESP, LE, is not small, the posterior probability of ESP is close to 1 whenever the ratio
r=LG /LE<< πE.
So it merely requires a sufficiently unlikely experimental result to overcome our prior bias against ESP and we will be convinced. So with only two hypotheses, the Bayesians are no better off than the frequentists. If my prior probability of ESP is zero then I will never be convinced. This is still formally rational but resolving to ignore any evidence of whatever strength is not rational in the informal sense.
So am I irrational to not be convinced by the Soal experiment? Do I have a prior probability of zero for ESP? I am here to tell you that I do not.
There are more than two hypotheses in play. Because there are other possible ways the data could have been generated besides a binomial experiment with p=0.2 or p>0.2. There are all sorts of possible deceptions that may have been perpetrated by the woman or by the experimenter. Let us call these various deception mechanisms that generate the data hypotheses and label them HD1,…,H Dk and assign these priors π1,…, πk that are not all extremely small. The posterior of ESP is now
where Li is the likelihood of the data under deception hypothesis i. The various deception hypotheses make the observed data reasonably likely, just as the ESP hypothesis does. If we assume that these likelihoods are similar in magnitude to the likelihood LE then the posterior becomes
where, as before, r=LG/LE measures evidence against guessing compared to ESP. The only terms in this expression that are not likely to be tiny are πG and the πi. So the conclusion is that the posterior of ESP will be close to zero rather than close to 1. Moreover, as the ratio r=LG/LE which we think of as embodying the evidence, becomes smaller, the posterior just reverts to the priors with the guessing hypothesis eliminated. So if we start off thinking that deception is more likely than ESP then we can never be convinced of ESP.
This leads to the conclusion that an honest person may tell the truth about an extremely pertinent experimental results, and not be believed, and those who disbelieve are not being irrational. This statement makes no assumptions about who is actually correct.
The obvious, in fact only, way for ESP researchers to avoid this effect is to force us to reduce our priors on the deception hypotheses. How they could do that, I have no idea.
