Fishing in the Bay

Here is another post about wrong headed justification of Bayesian over frequentist statistics. As suggested by David Dowe in his comment on my previous post, it is worth pointing out at the beginning rather than the end that nowhere below will you find an argument against Bayesian statistics per se (though I think there are some).

In the previous post I mentioned that there are two claims that (some) Bayesians make about their approach that get me annoyed. The first is that Bayesian thinking is natural and people will naturally apply probability to unknowns if not brain-washed by a frequentist education. The second is that only Bayesians, and not frequentists, can make use of prior information. Wrong.

Read the rest of this entry »

It is often claimed that regular, normal people naturally think like Bayesians. Leaving aside whether we should leave the foundations of our subject to the average punter, I suspect that this might be true. But it really depends on how you frame the question. Below is a description of a class discussion exercise used by Bill Jefferys, who is a Professor of Astronomy but also an adjunct professor of statistics, teaching a course in Bayesian statistics at the University of Texas.

See if you can spot the flaws.

Read the rest of this entry »

I often use ESP experiments as a vehicle to introduce hypothesis testing to students. One of the learning outcomes is that statisticians are actually easier to convince of claims of paranormal ability than lay persons . On the basis of a binomial calculation, I tell them that

I would be convinced of ESP if they could correctly select which of 5 cards I am holding more than 75 times out of 250 trials. You get 50 by guessing, and another 25 convinces the hard hearted statistician that you can bend spoons.

But I am actually telling them fibs. I would not be convinced. Does this make me irrational?

more–>I recall many years ago hearing of the so-called Soal experiment, which was in a published in book by Soul and Bateman (1954. Modern Experiments in TelepathyLondon: Faber & Faber) demonstrating telepathy. My first reaction to such claims is always rather skeptical.

In fact, the very evidence that ESP researchers use to convince us of ESP can, under mild assumptions, leads us more strongly to the conclusion that ESP is not true. This is beautifully explained in a 2003 book Probability Theory: The Logic of Science by Edwin T. Jaynes, G. Larry Bretthorst.

The experiment in question involved a woman getting 9410 trails out of 37100 correct where the chance of success under random guessing is p=0.2. This is about 26 standard deviations above the null mean and the binomial P-value is of the order of  1 in 10 to the power 139. But I for one am not convinced that she has ESP.

Frequentist hypothesis testing allows of two hypotheses only. If we are limited to two then, in this case, the relevant ones seem to be H_G:she is guessing and p=0.2, or H_E: she is not guessing and p>0.2. The null (H_G) is overwhelmingly rejected in favour of the conclusion (H_E) that she has ESP.

Bayesians can also fall for the trap of considering two hypotheses, in which case they end up in the same place as frequentists. Let L_E and L_G be the likelihood of the data under the null and alternative. Then the posterior probability she has ESP

Assuming that our prior probability of ESP, π_E , is small, and that the likelihood of the date under the hypothesis of ESP, L_E, is not small, the posterior probability of ESP is close to 1 whenever the ratio

r=L_G /L_E<< π_E.

So it merely requires a sufficiently unlikely experimental result to overcome our prior bias against ESP and we will be convinced. So with only two hypotheses, the Bayesians are no better off than the frequentists. If my prior probability of ESP is zero then I will never be convinced. This is still formally rational but resolving to ignore any evidence of whatever strength is not rational in the informal sense.

So am I irrational to not be convinced by the Soal experiment? Do I have a prior probability of zero for ESP? I am here to tell you that I do not.

There are more than two hypotheses in play. Because there are other possible ways the data could have been generated besides a binomial experiment with p=0.2 or p>0.2. There are all sorts of possible deceptions that may have been perpetrated by the woman or by the experimenter. Let us call these various deception mechanisms that generate the data hypotheses and label them H_D1,…,H _Dk and assign these priors π₁,…, π_k that are not all extremely small. The posterior of ESP is now 

where L_i is the likelihood of the data under deception hypothesis i. The various deception hypotheses make the observed data reasonably likely, just as the ESP hypothesis does. If we assume that these likelihoods are similar in magnitude to the likelihood L_E then the posterior becomes

where, as before, r=L_G/L_E measures evidence against guessing compared to ESP. The only terms in this expression that are not likely to be tiny are π_G and the π_i. So the conclusion is that the posterior of ESP will be close to zero rather than close to 1. Moreover, as the ratio r=L_G/L_E which we think of as embodying the evidence, becomes smaller, the posterior just reverts to the priors with the guessing hypothesis eliminated. So if we start off thinking that deception is more likely than ESP then we can never be convinced of ESP.

This leads to the conclusion that an honest person may tell the truth about an extremely pertinent experimental results, and not be believed, and those who disbelieve are not being irrational. This statement makes no assumptions about who is actually correct.

The obvious, in fact only, way for ESP researchers to avoid this effect is to force us to reduce our priors on the deception hypotheses. How they could do that, I have no idea.

Suppose you are testing for a disease. The hypothesis testing framework seems natural for this problem. Maximise the probability of diagnosing the disease, subject to a limit on the false positives. Most of us will be pretty comfortable withrecommending the likelihood ratio test in most cases. And I am not saying that there is anything wrong with it. But it is an automatic prescription. The issue I want to look at in this post is what kind of testing regime the LR prescription imposes on us when the disease is one that progresses and behaves differently in males and females.

Read the rest of this entry »

In other posts I have had a bit of a go at Bayesian zealotry. This post is about frequentist zealotry. I happen to believe that any method which has bad frequentist properties is straight out wrong. If you don’t care about how you method works in general or in the long run you should keep away from science and data. That is not to say that there are not major problems with frequentist inference. Nor does it mean that good frequentist properties mean a method is good. In this short post I want to argue the following proposition:

If two methods have almost identical frequentist properties that does not mean they are equally good. In fact, one can just be plain wrong.

Read the rest of this entry »

From time to time I get disquieted by what might be called foundational issues. I am not thinking about the Bayesian versus Frequentist divide in the first instance, although it may turn out that my concerns are related to these issues, it is really the gulf between Mathematical and Applied Statistics that I am thinking about. The problem that concerns me is that we sell Statistics as a way of thinking about the real world, but most of statistical reasoning takes place within a mathematical model or a family of such models.

Read the rest of this entry »

There has been quite an explosion of interest in so-called exact inference over the past decade. Partly this has been generated by computing power making exact calculations possible. But it has also been motivated by the realisation (from these same calculations) that the standard asymptotic procedures we may have thought were quite accurate have quite terrible statistical properties – at least if we view their requisite properties through a strict frequentist lens. What I want to do in this post is give you a quick appreciation of what exact inference is, and how much difference it makes. If you are interested in reading more details a paper by Alan Agresti is HERE and a recent paper by me is HERE.

Read the rest of this entry »

I’m teaching statistical inference this semester. It’s great fun to revisit those beautiful and useful tools, to build them from likelihood theory, and to show their connections and applications. I admire the elegance of the package, and the leverage that it gives us. The idea of just writing down the likelihood, maximizing it, and then checking diagnostics, is very appealing.

But it occured to me, as I was teaching the maximization of the likelihood, that I didn’t know why we use the method of scoring, for maximizing the likelihood, as opposed to some other optimization method, such as, for example, the Nelder-Mead simplex or simulated annealing.

Read the rest of this entry »

I attended an interesting conference in July organised by Mark Burgman for the Australian Centre of Excellence for Risk Analysis. Not all of the talks were statistical and the standard of the coffee was a great disappointment but there was plenty of interest in seeing the interaction of engineering type risk experts, environmentalists, mathematicians, statisticians and epistemologists.

Read the rest of this entry »