Sometimes there really isn't enough evidence to believe a claim

In #religion

A particular response has now happened twice in an ongoing discussion about extraordinary claims, and I felt I needed to think about it more deeply. The response is, essentially, that I am demanding too much evidence.

If you're following along, we have:

In McLatchie's response, he said of my criticism of the claimed Barbara Synder healing,

However, it is a fallacy to say that, because we could in principle have better evidence, we therefore do not have good or sufficient evidence. One can have compelling evidence even when a case could be rendered stronger with even more evidence.

Which I agree with, at least technically. In the comment thread of my response, Caleb J writes,

It seems we disagree on how much evidence is enough. It's always possible to ask for more (Couldn't more doctors have seen it? Couldn't the medical records have been earlier?) but there comes a point when enough is enough. Exactly when this point is depends on considerations around prior probability.

It is a reasonable question to ask -- how much evidence is enough? It is also correct to say that it depends on the prior probability of the claim, but it also depends on other desiderata of Jaynes, especially the ones dealing with consistency.

In some of these cases, I'm expecting evidence that should be there if the claim is true. It's not just saying that the evidence provided is lacking, but there is missing evidence that would be expected on the truth of the claim -- making it an argument against the claim.

Caleb was very nice in pointing out the documentation for the Barbara Snyder case, and correcting me in its reading. I commented about it like,

Thinking about the Snyder "miracle", I do wonder why in these cases there isn't much more extensive medical documentation? It would seem that these people are constantly dealing with skeptics, and the easiest way to address it would be to get the pre- and post- medical test data rather than a summary by one doctor, after the fact. Is it possible she was misdiagnosed -- it wasn't MS, so looking for an MS recovery may be misdirected? Is it possible that there was MS and something else, in which case looking for a full recovery from MS is misdirected? Without further data, it is impossible to rule out these things, but you'd think the data would be available with something this recent?

So how much evidence really is enough? Can we come to any agreement here? I think we can.

The problem is trying to determine whether a particular action or treatment actually causes the subsequent (positive) results. It's not enough to be given the treatment and see an improvement afterward. Why not? Partly, because we are human and have many biases. Further, the world is complex and any given effect can have many different possible causes, especially if we are talking about biological systems. We therefore have to design our investigation to reduce these biases and rule out other possible causes. Sometimes there really isn't enough available evidence to substantiate a claim. Even for much more modest claims than the ones made in the above discussions, randomized-controlled studies are the gold standard. Check out https://sciencebasedmedicine.org/?s=control&category_name=&submit=Search to see the value of doing controlled studies in the field of medicine -- and where it goes wrong when people don't do this.

What happens if you don't have randomized-controlled studies in medicine? What about single cases? It is pretty clear that not much can be said confidently. And certainly one shouldn't be willing to upend ones knowledge of how the universe works based on one report by one doctor on one patient. Here's where consistency comes in and the desiderata of Jaynes. If you are willing to believe the Barbard Snynder case, then you should be willing to believe this woman was healed of Parkinson's with "Energy Healing" or you should believe in the Golden Plates of Joseph Smith (making you a Mormon). The latter has more eyewitness testimony than we have for Barbara Snyder. It is quite clear that once you allow one miraculous healing in, using evidence of just one person, then one has to let in a whole host of others. We can also ask how the miracle believer handles the many cases where seemingly miraculous things happen and no reported prayer, with disapearing disease without treatment?

It's not just about the priors, it's the fact that to be consistent, once you raise your prior for one sort of claim you're doing it for many others -- many claims that you probably don't want to accept. Medical science has come a long way recognizing the fact that humans are easily fooled, they see patterns and agency where there isn't any good evidence, and biology is messy enough that uncertainty is always something to consider. Even in seemingly obvious cases.

I don't believe Barbara Snyder was cured by the intervention of some agent outside the natural world. If that agent were doing these kinds of actions we would expect to see a bigger effect, or that agent is being capricious. It is justified to demand better evidence, because we do that for even more modest claims, and we have centuries of data on the failings of human reasoning in inferring causes from effects. We know better now, and we often know what the minimum amount of data is to justify a particular claim. Data sets with N=1N=1 are not convincing in any field.