Analogies and Conjectures
Is the Pope Human? Bayes Theorem and the Prosecutor’s Fallacy
Abstract
© Dr M D Magee
Contents Updated: Tuesday, February 01, 2000
Bayes’ Theorm
Many researchers agree they often act on hunches, but that is not to reduce science to the relativisms of literary criticism. 250 years ago, the Reverend Thomas Bayes, an English cleric and amateur mathematician, formulated a theorem now given his name—the Bayes Theorem. It offers a powerful way of assessing the significance of new findings, and yet is rarely used by scientists. It should be used more widely.
In the mid-1940s, an American physicist called Richard Cox from Johns Hopkins University, Maryland, proved that if you want to make sense of scientific evidence, putting hard numbers on beliefs and hunches is not only handy—it is the only way of doing it logically and consistently. Cox asked himself whether it was possible to deduce laws for scientific reasoning. He knew that rules for dealing with absolute truth and falsity had bean worked out a century earlier by the British mathematician George Boole. But Cox wanted to know whether there were rules for reasoning about statements that lie somewhere between these two extremes. Cox showed scientific reasoning obeys certain rules—those rules turn out to be identical to the rules of probability theory.
The very essence of science is the assessment of theories in the light of evidence: that is, deciding how we should change how much we believe in a hypothesis, given the information we have collected. Cox showed that the way to do this logically and consistently is via the appropriate rule from probability theory. And that rule turns out to be Bayes’s theorem.
Bayes’s theorem forces scientists to say how plausible they think the hypothesis they are testing is and put a hard number on that belief, even if it is based on nothing more than a hunch. So while there has been lots of talk about the usefulness of Bayes’s theorem, and growing numbers of papers explaining how the method works, few studies have actually used the theorem.
The theorem is a means of updating the likelihood of hypotheses in the light of new evidence. That requires some belief to update in the first place, and if you are the first person to investigate some hypothesis, then you use your intuition, experience and judgment. Bayes realised this.
By revealing the central importance of Bayes’s theorem in scientific reasoning, Cox showed not only that the scientific process contains an ineluctahle amount of subjectivity at the outset, but that it gives way to objectivity as the information accumulates. In other words, scientific objectivity is emergent.
Conventional statistical tools ignore this completely. We have all come across some new research claiming to have found a new drug or link. Their findings have only one claim to reliability: that they are statistically significant. Bayes’s theorem demands that new findings be assessed in the light of previous experience, thus giving a much better handle on their plausibility.
Is the Pope Human?
Suppose you are trying to find out whether the Pope is an alien. Is the Pope human? This interesting question was posed by correspondents to Nature in 1996. The correspondents posed this argument in the form of a syllogism.
- The chance that any human picked at random is the Pope is 1 in 6 billion;
- John Paul II is the Pope;
- Therefore the chances are 6 billion to 1 against the Pope, John Paul II, being human.
The correspondents suggest thet this is obviously not sensible, but wonder why not. Perhaps one answer is that syllogisms do not usually deal with probabilities but certainties. If we were to say, "No humans are the Pope" then we can properly conclude that "therefore the Pope is not human," but we know that the Pope is human because some human can be the Pope even though the probability is small for any particular human being.
The fallacy is related therefore to the fact that not a random human being is being considered, but a particular one, the Pope.
If any random human being is selected, the chances are 6 billion to one against his being the Pope, and those selected therefore would mainly fit the criterion of being human. A circularity in the argument becomes evident here caused by the requirement that the Pope is assumed to be human to be selected at all and so he must look human even though he is not. It is necessary therefore to accept that the Pope is not human even though he looks as if he is. If then, it is discovered that the Pope fits a criterion that is very unlikely, but proves he is not human, it is not possible to claim it is not sensible. It is sensible because we have admitted a small possibility that the Pope could be an alien.
What we are really doing is statistically testing a population by sampling it. The population is all those organisms that meet the criteria of humanity. The criterion we have of non-humanity is that the human selected is the Pope. In that case, the human-like creature that is the Pope must be an alien. When we perform the experiment, the odds are 6 billion to one against us rejecting the hypothesis that the creature selected is human. The very small possibility remains that the experiment will turn up the Pope forcing us to reject the hypothesis that this selection is human. That is why, when hypotheses are tested statistically, a probability level is attached to them. There is always some possibility that the test is wrong.
By failing to select at random, by selecting the Pope at the outset, effectively the experimenters are cheating by rejecting each selection they make until they find someone who is the Pope. Obviously, if an observer did this, then reported only the final experiment as rejecting the hypothesis that the section is human, a large number of contrary results will have to be ignored, and the procedure is clearly bogus.
In fact, the Pope must be even less likely to be an alien than a human. Since we have never come across any aliens at all, so far as we know, let us take it the chances of the Pope being an alien are really nil. The incorrect conclusion that the Pope is not human therefore stems from an erroneous assumption that a minute probability of some criterion occurring—in this case having the title of Pope—implies non-humanity when it does not, necessarily.
Back to Bayes
H Jeffrey in his famous book, Theory of Probability, explains the problem in terms of Bayes theory. It is not true to assume that "if A then B" implies "if B then A." Eddy and MacKay, using Jeffrey, explain that the probability (P(D|H)) of data (D) given an hypothesis (H) is not the same as the probability (P(H|D)) of a hypothesis given the data.
Given the hypothesis (H) that the Pope is human, the probability that any human is the Pope (the Datum, D) is 1 in 6 million. But that does not mean that given the datum that the probability of any human being Pope is 1 in 6 million, the hypothesis (H) that the Pope is human is also only 1 in 6 billion.
In this case let us say that H is the hypothesis of Papal humanity and D is the probability that this particular observation does prove to be the Pope. What we lack is the alternative hypothesis of Papal alienness A. Then according to Bayes theorem:
P(D|H)P(H)
P(H|D) = ______________________
P(D|H)P(H) + P(D|A)P(A)
The left hand term in the denominator is the same term as the numerator, so the probability (P(H|D)) of the Pope being human is 1 (unity = certainty), if the term to the right of the denominator (P(D|A)P(A)) is nought.
Now P(H) is the probability that a randomly selected person is a human at the outset. P(A) is the probability that any person is only a human-looking alien at the outset (in the original population irrespective of whatever we might be considering). All the human-looking creatures are either human or alien so the two probabilities, P(H) and P(A) must add up to 1 (certainty)—they cannot be anything other than human or alien—so P(A) = 1 - P(H).
Our datum that the chances of any human being the Pope are 1 in 6 billion tells us there are 6 billion humans, and there are no aliens, P(A) is 0 and P(H) = 1, or P(A) = 1/6,000,000,000 if one human is really an alien, when P(H) = 5,999,999,999/6,000,000,000.
The chances of the datum being due to the hypothesis that the Pope is an alien, P(D|A), must be zero, if there are no aliens or infinitesimally small (1 in 6 billion) if one of the humans is admitted as being an alien. Even if both P(D|A) and P(A) are not zero, they are both very tiny numbers multiplied together. The result therefore is effectively zero, even if the two factors are merely small. If either factor is actually zero, the product is zero. In any of these cases, the probability that the Pope is human approaches closely to 1 (certainty).
Doubtless, he is very relieved.
The Prosecutor’s Fallacy
Fallaceous reasoning like this is quite common, a good "bad" example being the Prosecutor’s Fallacy. A DNA match at a shocking crime has a probability of occurence in a population of only 1 in a million. The prosecutor has found that the accused has this match and says the odds are a million to one on the suspect being guilty. The jury will be quite unable to acquit against such apparent odds.
The trouble is that there were ten million people in the city on the night of the crime, so the odds are that 10 people among the population of the city on the night of the crime matched the DNA sample. If the DNA match is the only evidence the prosecutor has, then the odds are 10 to 1 on the suspect being innocent!
Apparent odds of a million to one against innocence, considered properly, become 10 to one on innocence! Many a prosecutor has built a great reputation and many poor people gained a criminal record because twelve good citizens do not understand—from the evidence presented—that the Pope—though he is a Christian—is not an alien.
