Michael Evans | Statistical Evidence and Statistical Reasoning: Rethinking What Data Really Tell Us

SciTube
$
Scitube Library
$
engineering and tech
$
Michael Evans | Statistical Evidence and Statistical Reasoning: Rethinking What Data Really Tell Us

Jul 23, 2026 | engineering and tech, physical sciences, Scientific Discovery 2026

About this episode

Statistics is everywhere. From clinical trials to particle physics, from economic policy to public health, researchers rely on statistical analyses to determine what the data are telling them. Despite this central role in modern science, there is a striking gap at the heart of the discipline: no universally agreed definition of what statistical evidence actually is. That gap matters. When scientists say ‘the evidence suggests’ or ‘based on the data we conclude’, what exactly do they mean? University of Toronto statistician Michael Evans has spent years arguing that this question is not merely philosophical – it is foundational, and leaving it unanswered has real consequences for scientific reliability. Read More

The most familiar tool for measuring evidence is the p-value, which measures how surprising observed data would be if a hypothesis were true. Small p-values are routinely interpreted as evidence against a hypothesis, and a threshold of 0.05 has become a near-universal standard in many fields.

Evans points out serious flaws in this approach. A small p-value does not necessarily mean there is evidence against a hypothesis; with large sample sizes, even a trivially small and scientifically meaningless difference can produce an impressively small p-value. Worse still, p-values cannot identify genuine evidence in favour of a hypothesis, which is something science surely needs to do.

Evans proposes a cleaner starting point: evidence should be defined by how data change our beliefs. If the probability of a hypothesis increases after observing the data, that is evidence in its favour. If it decreases, that is evidence against. This deceptively simple idea forms the foundation for a coherent theory of statistical inference that Evans calls relative belief.

In practice, this means comparing the probability assigned to a hypothesis before and after seeing the data, and expressing the ratio as a single number: the relative belief ratio. Values above 1 indicate evidence in favour, values below 1 indicate evidence against, and the posterior probability measures how strongly to believe what the evidence says.

Evans’ approach also addresses a longstanding criticism of Bayesian methods: that results can be skewed by the analyst’s choice of prior assumptions. Evans shows that two kinds of bias – one working against a hypothesis and one in its favour – can be measured and controlled by collecting sufficient data. These bias measures are frequentist in character, forging a principled bridge between Bayesian inference and traditional statistical thinking. In addition, the ingredients required for a relative belief analysis, namely, the model and the prior, can both be checked against the observed data to deal with the subjectivity inherent in their choice.

Evans argues that getting these foundations right is not an abstract exercise. A precise definition of statistical evidence shapes every step of an analysis, and building that definition into standard practice could make scientific reasoning more transparent, more reliable, and harder to misuse.

Original research published by:

Original articles:

Summary of the papers: ‘Measuring and Controlling Bias for Some Bayesian Inferences and the Relation to Frequentist Criteria’, doi.org/10.3390/e23020190; and ‘The Concept of Statistical Evidence, Historical Roots and Current Developments’, doi.org/10.3390/encyclopedia4030078