Magnitude-based Inference

In 2009, Medicine & Science in Sports & Exercise® published “Progressive Statistics for Studies in Sports Medicine and Exercise Science” by Hopkins et al. (1). Dr. Hopkins had been invited to draft a set of “statistical guidelines” for use by MSSE’s submitting authors and reviewers to enhance the quality of research reports published in our journal. The article submitted by Hopkins et al. (1) underwent peer review by eight scientists, experts in statistical analysis and data interpretation, and the authors undertook extensive revisions in response. While the majority of reviewers recommended publication of the article, some did not because of strong disagreement on aspects of the article. Therefore, MSSE’s editors concluded that there was insufficient scientific consensus for establishment of a formal editorial policy about appropriate data analysis and reporting. However, the editors believed that the authors did present a scholarly and novel framework for data analyses and reporting that deserved consideration, so as indicated in the editor’s note accompanying the article, MSSE published the article as an invited commentary. In their Introduction, Hopkins et al. (1) wrote that their statistical recommendations were aimed at encouraging debate about statistics and legitimizing approaches that the authors were having difficulty getting published, including one they described as “magnitude-based inference,” a new approach for determining whether measurement differences between groups represented a real difference or effect. The appeal of the magnitude-based approach is that it is less conservative than traditional statistics used to evaluate differences or effects, especially for studies with a small sample size. Since publication, the progressive statistics commentary has been cited nearly 400 times, including 28 citations in articles published in MSSE. A casual search through articles citing the progressive statistics indicated that, heretofore, most citations were used to justify statistical approaches used for data analysis, including magnitude-based inference, not withstanding the continued lack of apparent consensus regarding the scientific legitimacy of the progressive approaches and especially magnitude-based inferences. In this month’s issue of MSSE, Welsh and Knight (2) present “Magnitude-based Inference: A Statistical Review.” This review is the most thorough evaluation of that approach published to date, including detailed consideration of the computational formulas in the Excel spreadsheets provided by Hopkins et al. (1), the similarity and differences between magnitude-based inference with a fully Bayesian approach, the accuracy of the probability inferences, and implications for sample size calculations. Welsh and Knight (2) carefully and clearly outline their concerns, and the four experts in statistics and data analysis who reviewed their manuscript determined that those concerns were sufficiently justified to merit publication. Welsh and Knight (2) conclude that although magnitude-based inference is appealing in concept, the underlying mathematics does not support using the approach, as currently recommended. The article by Welsh and Knight (2) indicates that Hopkins et al. (1) achieved their goal of encouraging debate about statistics, even if their recommended progressive approaches have not been validated. Readers, even those who are not statistical experts, will appreciate this new review, and investigators who choose to continue using magnitude-based inference will be better equipped to address objections to that approach. Andrew J. Young, PhD, FACSM Consulting Editor Medicine & Science in Sports & Exercise®