More inflating of Bayes Factors

In #religion

My post about inflating Bayes Factors with nothing has given Lydia McGrew a lot to talk about over on her blog. If you scroll down to the comments, you'll see a long back and forth that (perhaps rightly) ends with Lydia McGrew stating "At this point we're just repeating ourselves, so I'll leave things here for any readers who have followed us thus far."

While Lydia McGrew believes we are walking in circles (and maybe we are to some extent) I think we can start to see where we stand with a couple of direct questions that I'd love to hear her answers -- I give my answers for them here.

Say you have two or more models, (call them M1, M2, and M3), and two data sets (call them dataA and dataB -- you could equally call them evidenceA and evidenceB). dataB was obtained by taking dataA and modifying it in some way to create dataB. I've left this completely general. You then calculate the posteriors for the models given the data and find all the posteriors equal for the two different data sets, namely:

  • p(M1|dataA)=p(M1|dataB)
  • p(M2|dataA)=p(M2|dataB)
  • p(M3|dataA)=p(M3|dataB)

Question 1: Can we say that anything I did to make dataB from dataA change our state of knowledge about any of the models we are considering? I would answer "no" to this because the posteriors contain all the relevant knowledge of our models, and equal probabilities represent equivalent states of knowledge.

Question 2: If I know that the posteriors for a model are equal given the two different data sets, p(M1|dataA)=p(M1|dataB), but I also find that the likelihoods are different, say p(dataB|M1)>p(dataA|M1), can I say anything meaningfully different about that model in the two cases? I'd say "no" to this -- if we know the posteriors are the same (and posteriors are the only thing we should ultimately be interested in) then equal posteriors between two cases implies no new information about that model in the two cases.

Thus my challenge: show me an example where models or parts of models that entail that we didn't see the evidence we observed actually affects the posterior. Lydia does not to my reading, in any of the conversation, calculate a posterior. So she hasn't shown such an example exists. I think one can't see this clearly without going through an entire calculation. I posted as a comment a full calculation, but it was never authorized to appear on McGrew's blog, so I repeat it here. It has all the elements of what I'm talking about.

This is what happens I believe when one speaks in generalities and doesn't walk through a problem, even a simple one, from beginning to end. The fact that the prior automatically cancels the effect of the partition is not intuitive at all, and I think can only really be appreciated when one walks through the problem from priors and likelihoods all the way to the posteriors for all of the models.

The Urn example is particularly good for seeing this, because the probabilities are found by simple counting. We can even generalize the "nothing" model to include some fraction of being consistent with the evidence, i.e. the draw of a black ball. I'll probably write this up in detail at some point, but the math is not hard. I'll summarize the result here. The contents of the urns: 

  • urn H: 10 black balls
  • urn M: 25 black balls and 25 white balls
  • urn N: 'b' black balls and 'r' red balls (technically for it to be a "nothing" model b=0, but we'll see the effect of having some parts of N consistent with the evidence)

data=evidence=1 black ball drawn

posteriors - just count the balls!

  • p(H|black) = 10/(10+25+b) (note: for the nothing model b=0, this is 10/35~0.285)
  • p(M|black) = 25/(10+25+b) (note: for the nothing model b=0, this is 25/35~0.715)
  • P(N|black)=b/(10+25+b) (note: for the nothing model b=0, this is 0)

posterior ratio: p(H|black)/p(~H|black)=10/(25+b). (note: for the nothing model b=0, this is 1/2.5, our previous result)

Notice that the part of N that has nothing to do with the actual evidence observed (i.e. the number of red balls, r) does not come into the posterior.

Now to the Bayes Factor:

  • BF = (10/10)/( (25+b)/(50+b+r)) = (50+b+r)/(25+b)

notice that the part of N that has nothing to do with the actual evidence observed (i.e. the number of red balls, r) does come into the Bayes Factor, and I can make the Bayes Factor as large as I want (e.g. r=10^100 would give a BF around 10^100) and not affect the posterior at all.

How about the prior ratio?

  • P(H)/P(~H) = (10/(10+50+b+r))/((50+b+r)/(10+50+b+r))= 10/(50+b+r)

notice that the "50+b+r" part looks familiar...it's in the numerator of the BF and the denominator of the prior ratio! Adding more red balls has no effect on the posterior. The part of N that has nothing to do with the actual evidence observed (i.e. the number of red balls, r) does not come into the posterior. So if you focus only on BF, and don't go through the full calculation to the posterior, you might easily miss this.

So, insisting on a partition which includes models that have nothing to do with the actual evidence observed is just more work than is needed, and can easily lead to misunderstanding when one sees a large Bayes Factor.

To refer to a comment by McGrew, if I say "throwing more red balls labeled N into the urn will not change p(H|black)", do I also have to say (as McGrew insists) "and nothing else changes with the other balls in the urn"? Is this pedantic suggestion important at all? Does it imply that I'm (as McGrew states) "setting up the completely different scenario in a carefully balanced fashion so that [I am] increasing the prior probability of H3 and this is equally affecting the priors of the others, and you are leaving all the likelihoods the same"? I'm just tossing red balls into an urn. That's it. Not a completely different scenario, not "carefully balanced" to make sure the ratios of all the non-red balls are the same -- the simple rules of counting take care of this. All I'm doing in the above calculation is increasing r. The "balance" happens because of the math.

Since McGrew has not shown counter example with a full calculation like this -- from the base assumptions of the example, all the way through to the posterior, to show "where models or parts of models that entail that we didn't see the evidence we observed actually affects the posterior" I will maintain my principled belief that a counter example doesn't exist.

Further, now that I've seen such an unintuitive effect of a process which artificially magnifies a Bayes Factor while not changing the posteriors, I am left wondering if there are more processes like this that I don't know about? As a result, this gives me an inherent distrust of any inference where someone focuses on Bayes Factors if they don't also, with equal energy and detail, show the effect on the posterior. Unfortunately, that seems to be the bulk of the McGrew's work and the work of most of the Christian apologists that work with the probability math.