Extraordinary claims require extraordinary evidence

Extraordinary claims require extraordinary evidence - Carl Sagan

I think no pithy quote has caused so much angst with apologists than this one from Carl Sagan, directed in particular to religious miracle claims. Before looking into it a bit more specifically, I want to point out that the word "claim" here refers to "explanation" or "proposition" and not to testimony (which would include a form of evidence). I'll have more to say about this in another post. If we are talking about testimony as the evidence, then we need to break the "claim" into the "explanation" part and the "evidence" part (see my ramble on claims and evidence here).

To see an overly technical treatment of this by an apologist, we can look to Tim McGrew's treatment on Erik Manning's channel, as well as one of his own on the related topic of Hume.

• McGrew's Sagan video: https://youtu.be/LD2hQFTJsK0
• McGrew's Hume video: https://www.youtube.com/watch?v=H7Gv8Fw_fFE&list=WL&index=12

I'll give more detailed responses to these videos later, but here I just want to map the Sagan quote to probabilities.

The analysis

For me, it seems straightforward to set up. We start with some notation.

• $D$ = data, or the evidence
• $M_i$ = the various models, or explanations, or claims. We'll treat $M_o$ as the extraordinary explanation and $M_1,M_2,\dots$ as the mundane explanations

Bayes theorem then has

$$\begin{array}{rl}P(M_o|D)& =\frac{P(D|M_o)P(M_o)}{P(D|M_o)P(M_o)+P(D|M_1)P(M_1)+P(D|M_2)P(M_2)+\cdots }\\ & =\frac{P(D|M_o)P(M_o)}{P(D|M_o)P(M_o)+{\sum }_{i=1}^{N}P(D|M_i)P(M_i)}\end{array}$$ We can simplify this by merging all of the mundane explanations together into $M_1$, which is also equivalent to $\sim M_o$ $$\begin{array}{rl}P(M_o|D)& =\frac{P(D|M_o)P(M_o)}{P(D|M_o)P(M_o)+P(D|M_1)P(M_1)}\end{array}$$ which leads to <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtable columnalign="right left" columnspacing="0em" rowspacing="0.25em"><mtr><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mi mathvariant="normal">∣</mi><mi>D</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow><mrow></mrow><mo>=</mo><mfrac><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo></mrow><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mn>1</mn></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mn>1</mn></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow></mrow></mstyle></mtd><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow><mrow></mrow><mo>=</mo><mfrac><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo></mrow><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo></mrow></mfrac><mrow><mo fence="true">(</mo><mfrac><mn>1</mn><mrow><mn>1</mn><mo>+</mo><mfrac><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mn>1</mn></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mn>1</mn></msub><mo stretchy="false">)</mo></mrow><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>D</mi><mi mathvariant="normal">∣</mi><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>M</mi><mi>o</mi></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow></mfrac><mo fence="true">)</mo></mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow></mrow></mstyle></mtd><mtd><mstyle displaystyle="true" scriptlevel="0"><mrow><mrow></mrow><mo>≡</mo><mfrac><mn>1</mn><mrow><mn>1</mn><mo>+</mo><mi>r</mi></mrow></mfrac></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex"> \begin{aligned} P(M_o|D) &= \frac{P(D|M_o)P(M_o)}{P(D|M_o)P(M_o)+P(D|M_1)P(M_1)} \\ &=\frac{P(D|M_o)P(M_o)}{P(D|M_o)P(M_o)}\left(\frac{1}{1+\frac{P(D|M_1)P(M_1)}{P(D|M_o)P(M_o)}}\right)\\ &\equiv \frac{1}{1+r} \end{aligned} </annotation></semantics></math>P(Mo​∣D)​=P(D∣Mo​)P(Mo​)+P(D∣M1​)P(M1​)P(D∣Mo​)P(Mo​)​=P(D∣Mo​)P(Mo​)P(D∣Mo​)P(Mo​)​<svg height="3.600em" viewBox="0 0 875 3600" width="0.875em" xmlns="http://www.w3.org/2000/svg"><path d="M863,9c0,-2,-2,-5,-6,-9c0,0,-17,0,-17,0c-12.7,0,-19.3,0.3,-20,1 c-5.3,5.3,-10.3,11,-15,17c-242.7,294.7,-395.3,682,-458,1162c-21.3,163.3,-33.3,349, -36,557 l0,84c0.2,6,0,26,0,60c2,159.3,10,310.7,24,454c53.3,528,210, 949.7,470,1265c4.7,6,9.7,11.7,15,17c0.7,0.7,7,1,19,1c0,0,18,0,18,0c4,-4,6,-7,6,-9 c0,-2.7,-3.3,-8.7,-10,-18c-135.3,-192.7,-235.5,-414.3,-300.5,-665c-65,-250.7,-102.5, -544.7,-112.5,-882c-2,-104,-3,-167,-3,-189 l0,-92c0,-162.7,5.7,-314,17,-454c20.7,-272,63.7,-513,129,-723c65.3, -210,155.3,-396.3,270,-559c6.7,-9.3,10,-15.3,10,-18z"></path></svg>​1+P(D∣Mo​)P(Mo​)P(D∣M1​)P(M1​)​1​<svg height="3.600em" viewBox="0 0 875 3600" width="0.875em" xmlns="http://www.w3.org/2000/svg"><path d="M76,0c-16.7,0,-25,3,-25,9c0,2,2,6.3,6,13c21.3,28.7,42.3,60.3, 63,95c96.7,156.7,172.8,332.5,228.5,527.5c55.7,195,92.8,416.5,111.5,664.5 c11.3,139.3,17,290.7,17,454c0,28,1.7,43,3.3,45l0,9 c-3,4,-3.3,16.7,-3.3,38c0,162,-5.7,313.7,-17,455c-18.7,248,-55.8,469.3,-111.5,664 c-55.7,194.7,-131.8,370.3,-228.5,527c-20.7,34.7,-41.7,66.3,-63,95c-2,3.3,-4,7,-6,11 c0,7.3,5.7,11,17,11c0,0,11,0,11,0c9.3,0,14.3,-0.3,15,-1c5.3,-5.3,10.3,-11,15,-17 c242.7,-294.7,395.3,-681.7,458,-1161c21.3,-164.7,33.3,-350.7,36,-558 l0,-144c-2,-159.3,-10,-310.7,-24,-454c-53.3,-528,-210,-949.7, -470,-1265c-4.7,-6,-9.7,-11.7,-15,-17c-0.7,-0.7,-6.7,-1,-18,-1z"></path></svg>​≡1+r1​​ where $r=P(D|M_1)P(M_1)/P(D|M_o)/P(M_o)$ which was a long-winded way of getting to the odds form of Bayes theorem.

In order for the posterior for the extraordinary claim to rise above $P(M_o|D)>0.5$ then $r<1$. To map the phrase to values we have,

• extraordinary claim $\equiv$ low prior: $P(M_o)\ll 1$
• mundane claim $\equiv$ high prior: $P(M_1)\sim 1$

This means that $P(M_1)/P(M_o)\gg 1$ which immediately implies for $r<1$ that $P(D|M_o)\gg P(D|M_1)$.

Now, let's assume that the extraordinary claim perfectly fits the observed data, so that we have $P(D|M_o)\sim 1$. What the last statement implies is that to justify your extraordinary claim, you must have $$P(D|M_1)\ll 1$$ or every other mundane claim must be nearly ruled out, not just unlikely. This is exactly what science does -- construct situations (i.e. controlled experiments) to make any mundane explanation nearly impossible. The level of "nearly impossible" depends of course on the extraordinariness of the primary claim.

I find this consequence so straightforward, that I find it baffling that it is at all contentious.

What problems are you interested in? How can I help?

• @bblais
• bblais@bryant.edu
• (401) 232-6153
• Bryant University
  1150 Douglas Pike
  Smithfield, RI
  02917 US