A Dirty Dozen: Twelve P-Value Misconceptions

doi:10.1053/j.seminhematol.2008.04.003

Seminars in Hematology

Volume 45, Issue 3, July 2008, Pages 135-140

https://doi.org/10.1053/j.seminhematol.2008.04.003 Get rights and content

The P value is a measure of statistical evidence that appears in virtually all medical research papers. Its interpretation is made extraordinarily difficult because it is not part of any formal system of statistical inference. As a result, the P value's inferential meaning is widely and often wildly misconstrued, a fact that has been pointed out in innumerable papers and books appearing since at least the 1940s. This commentary reviews a dozen of these common misinterpretations and explains why each is wrong. It also reviews the possible consequences of these improper understandings or representations of its meaning. Finally, it contrasts the P value with its Bayesian counterpart, the Bayes' factor, which has virtually all of the desirable properties of an evidential measure that the P value lacks, most notably interpretability. The most serious consequence of this array of P-value misconceptions is the false belief that the probability of a conclusion being in error can be calculated from the data in a single experiment without reference to external evidence or the plausibility of the underlying mechanism.

References (42)

A.R. Feinstein
P-values and confidence intervals: Two sides of the same unsatisfactory coin
J Clin Epidemiol
(1998)
J.P. Ioannidis
Genetic associations: False or true?
Trends Mol Med
(2003)
S.N. Goodman
One or two-sided P-values?
Control Clin Trials
(1988)
J.P. Boissel
Some thoughts on two-tailed tests (and two-sided designs)
Control Clin Trials
(1988)
J.L. Fleiss
One-tailed versus two-tailed tests: Rebuttal
Control Clin Trials
(1989)
J.A. Knottnerus et al.
The ethics of sample size: Two-sided testing and one-sided thinking
J Clin Epidemiol
(2001)
E. Garcia-Berthou et al.
Incongruence between test statistics and P values in medical papers
BMC Med Res Methodol
(2004)
B. Andersen
Methodological Errors in Medical Research
(1990)
D.M. Windish et al.
Medicine residents' understanding of the biostatistics and results in the medical literature
JAMA
(2007)
J. Berkson
Tests of significance considered as evidence
J Am Stat Assoc
(1942)

D. Mainland

The significance of “nonsignificance.”

Clin Pharm Ther

(1963)

D. Mainland

Statistical ritual in clinical journals: Is there a cure? —I

Br Med J

(1984)

W. Edwards et al.

Bayesian statistical inference for psychological research

Psych Rev

(1963)

G.A. Diamond et al.

Clinical trials and statistical verdicts: Probable grounds for appeal

Ann Intern Med

(1983)

A.R. Feinstein

Clinical biostatisticsXXXIV. The other side of ‘statistical significance’: Alpha, beta, delta, and the calculation of sample size

Clin Pharmacol Ther

(1975)

K. Rothman

Significance questing

Ann Intern Med

(1986)

P. Pharoah

How not to interpret a P value?

J Natl Cancer Inst

(2007)

S.N. Goodman et al.

Evidence and scientific research

Am J Public Health

(1988)

L. Braitman

Confidence intervals extract clinically useful information from data

Ann Intern Med

(1988)

S.N. Goodman

Towards evidence-based medical statistics, I: The P-value fallacy

Ann Intern Med

(1999)

S.N. Goodman

P-values, hypothesis tests and likelihood: Implications for epidemiology of a neglected historical debate

Am J Epidemiol

(1993)

Cited by (459)

Applying the fragility index to randomized controlled trials evaluating total neoadjuvant therapy for rectal cancer: A methodological survey
2024, Radiotherapy and Oncology
Recently, there has been significant interest in, and adoption of, total neoadjuvant therapy (TNT) for locally advanced rectal cancer (LARC). We designed the present study to assess the robustness of the randomized controlled trials (RCTs) evaluating contemporary TNTs for LARC using the fragility index (FI).
Relevant articles were identified through a review article by Johnson et al. in the Canadian Journal of Surgery. Dichotomous outcomes within these RCTs were eligible for inclusion if the reported effect size had a p-value < 0.05. The main outcome was FI for each included outcome. Walsh et al.’s method of calculating FI was utilized. Correlations between FI and research characteristics were assessed using the Spearman’s rank correlation coefficients. Risk of bias was assessed using Cochrane recommended tools.
Ten RCTs were identified with 25 outcomes having statistically significant differences between groups. Eleven outcomes were time-to-event outcomes, while the remainder were dichotomous outcomes. Approximately half (n = 13) were oncologic outcomes. The median FI was 2 (interquartile range [IQR] 1–16). The number of patients lost to follow-up exceeded the FI in 17 outcomes (68.0 %) and thus these results were considered “fragile”. Lower FI was associated with high risk of bias (rho = −0.5594) and greater loss to follow-up (rho = −0.4394), while higher FI was associated with large study size (rho = 0.5120).
The robustness of outcomes from trials assessing TNT for LARC was found to be questionable. Most outcomes were fragile, as determined by the FI. This survey is limited by the number of included studies.
Assessing the audio-visual environment at the Great Wall of China: Impacts on visitor satisfaction and restoration
2024, Applied Acoustics
Previous studies have highlighted the significant role of vision in human perception. In this study, we examined whether the assessment of the audio-visual environment at China’s iconic Great Wall aligns with these findings to understand how this assessment influences visitor satisfaction and a sense of restoration. Using a field survey with 107 participants, an eight-variable structural equation model was used, encompassing sound sources (sounds of technology, human being and nature), the pleasantness and eventfulness of soundscape, visual assessment, visitor satisfaction, and the Short-version Revised Restoration Scale (SRRS). The results revealed that: (1) visual assessment acted as a partial mediator not only between soundscape assessment and visitor satisfaction, but also between the soundscape assessment and restoration. (2) The pleasantness of the soundscape positively correlated with both visitor satisfaction (β = 0.278, p = 0.004) and restoration (β = 0.244, p = 0.000), while human-generated sounds had a negative impact on soundscape pleasantness (β = -0.256, p = 0.019). (3) The combined visual and auditory assessments contributed 60.5 % and 39.5 % to visitor satisfaction and 42.1 % and 36.8 % to restoration, respectively, indicating that the soundscape assessment of the Great Wall was higher than expected and comparable to visual assessment.
Best practices in NMR metabolomics: Current state
2024, TrAC - Trends in Analytical Chemistry
A literature survey was conducted to identify current practices used by NMR metabolomics investigators when conducting and reporting their metabolomics studies. A total of 463 papers from 2020 to 80 papers from 2010 were selected from PubMed and were manually analyzed by a team of investigators to assess the extent and completeness of the experimental procedures and protocols reported. A significant number of the papers did not report on essential experimental details, incompletely stated which statistical methods were used, improperly applied supervised multivariate statistical analyses, or lacked validation of statistical models. A large diversity of protocols and software were identified, which suggests a lack of consensus and a relatively limited use of commonly agreed upon standards for conducting and reporting NMR metabolomics studies. The overall intent of the survey is to inform and encourage the NMR metabolomics community to develop and adopt best practices for the field.
Improving the utility of non-significant results for educational research: A review and recommendations
2024, Educational Research Review
When used appropriately, non-significant p-values have the potential to further our understanding of what does not work in education, and why. When misinterpreted, they can trigger misguided conclusions, for example about the absence of an effect of an educational intervention, or about a difference in the efficacy of different interventions. We examined the frequency of non-significant p-values in recent volumes of peer-reviewed educational research journals. We also examined how frequently researchers misinterpret non-significance to imply the absence of an effect, or a difference to another significant effect. Within a random sample of 50 peer-reviewed articles, we found that of 528 statistically tested hypotheses, 253 (48%) were non-significant. Of these, 142 (56%) were erroneously interpreted to indicate the absence of an effect, and 59 (23%) to indicate a difference to another significant effect. For 97 (38%) of non-significant results, such misinterpretations were linked to potentially misguided implications for educational theory, practice, or policy. We outline valid ways for dealing with non-significant p-values to improve their utility for education, discussing potential reasons for these misinterpretations and implications for research.
Antiphospholipid antibodies are enriched post-acute COVID-19 but do not modulate the thrombotic risk
2023, Clinical Immunology
COVID-19-associated coagulopathy, shown to increase the risk for the occurrence of thromboses and microthromboses, displays phenotypic features of the antiphospholipid syndrome (APS), a prototype antibody-mediated autoimmune disease. Several groups have reported elevated levels of criteria and non-criteria antiphospholipid antibodies (aPL), assumed to cause APS, during acute or post-acute COVID-19. However, disease heterogeneity of COVID-19 is accompanied by heterogeneity in molecular signatures, including aberrant cytokine profiles and an increased occurrence of autoantibodies. Moreover, little is known about the association between autoantibodies and the clinical events. Here, we first aim to characterise the antiphospholipid antibody, anti-SARS-CoV-2 antibody, and the cytokine profiles in a diverse collective of COVID-19 patients (disease severity: asymptomatic to intensive care), using vaccinated individuals and influenza patients as comparisons. We then aim to assess whether the presence of aPL in COVID-19 is associated with an increased incidence of thrombotic events in COVID-19.
We conducted anti-SARS-CoV-2 IgG and IgA microELISA and IgG, IgA, and IgM antiphospholipid line immunoassay (LIA) against 10 criteria and non-criteria antigens in 155 plasma samples of 124 individuals, and we measured 16 cytokines and chemokines in 112 plasma samples. We additionally employed clinical and demographic parameters to conduct multivariable regression analyses within multiple paradigms. In line with recent results, we find that IgM autoantibodies against annexin V (AnV), β2-glycoprotein I (β2GPI), and prothrombin (PT) are enriched upon infection with SARS-CoV-2. There was no evidence for seroconversion from IgM to IgG or IgA. PT, β2GPI, and AnV IgM as well as cardiolipin (CL) IgG antiphospholipid levels were significantly elevated in the COVID-19 but not in the influenza or control groups. They were associated predominantly with the strength of the anti-SARS-CoV-2 antibody titres and the major correlate for thromboses was SARS-CoV-2 disease severity.
While we have recapitulated previous findings, we conclude that the presence of the aPL, most notably PT, β2GPI, AnV IgM, and CL IgG in COVID-19 are not associated with a higher incidence of thrombotic events.
Challenges, Complexities, and Considerations in the Design and Interpretation of Late-Phase Oncology Trials
2023, Seminars in Radiation Oncology
Optimal management of cancer patients relies heavily on late-phase oncology randomized controlled trials. A comprehensive understanding of the key considerations in designing and interpreting late-phase trials is crucial for improving subsequent trial design, execution, and clinical decision-making. In this review, we explore important aspects of late-phase oncology trial design. We begin by examining the selection of primary endpoints, including the advantages and disadvantages of using surrogate endpoints. We address the challenges involved in assessing tumor progression and discuss strategies to mitigate bias. We define informative censoring bias and its impact on trial results, including illustrative examples of scenarios that may lead to informative censoring. We highlight the traditional roles of the log-rank test and hazard ratio in survival analyses, along with their limitations in the presence of nonproportional hazards as well as an introduction to alternative survival estimands, such as restricted mean survival time or MaxCombo. We emphasize the distinctions between the design and interpretation of superiority and noninferiority trials, and compare Bayesian and frequentist statistical approaches. Finally, we discuss appropriate utilization of phase II and phase III trial results in shaping clinical management recommendations and evaluate the inherent risks and benefits associated with relying on phase II data for treatment decisions.

View all citing articles on Scopus

View full text

A Dirty Dozen: Twelve P-Value Misconceptions

J Clin Epidemiol

Trends Mol Med

Control Clin Trials

Control Clin Trials

Control Clin Trials

J Clin Epidemiol

Incongruence between test statistics and P values in medical papers

BMC Med Res Methodol

Methodological Errors in Medical Research

Medicine residents' understanding of the biostatistics and results in the medical literature

JAMA

Tests of significance considered as evidence

J Am Stat Assoc

The significance of “nonsignificance.”

Clin Pharm Ther

Statistical ritual in clinical journals: Is there a cure? —I

Br Med J

Bayesian statistical inference for psychological research

Psych Rev

Clinical trials and statistical verdicts: Probable grounds for appeal

Ann Intern Med

Clinical biostatisticsXXXIV. The other side of ‘statistical significance’: Alpha, beta, delta, and the calculation of sample size

Clin Pharmacol Ther

Significance questing

Ann Intern Med

How not to interpret a P value?

J Natl Cancer Inst

Evidence and scientific research

Am J Public Health

Confidence intervals extract clinically useful information from data

Ann Intern Med

Towards evidence-based medical statistics, I: The P-value fallacy

Ann Intern Med

P-values, hypothesis tests and likelihood: Implications for epidemiology of a neglected historical debate

Am J Epidemiol