Translating evidence into practice: eligibility criteria fail to eliminate clinically significant differences between real-world and study populations
Amelia Averitt; Chunhua Weng; Patrick Ryan; Adler Perotte · 2020 · npj Digital Medicine
WASTE classifies this as Replication Failure · AI classification, approximate
A previously reported effect did not replicate here — verify it holds before you build on it.
Abstract
Randomized controlled trials (RCTs) are regarded as the most reputable source of evidence. In some studies, factors beyond the intervention itself may contribute to the measured effect, an occurrence known as heterogeneity of treatment effect (HTE). If the RCT population differs from the real-world population on factors that induce HTE, the trials effect will not replicate. The RCTs eligibility criteria should identify the sub-population in which its evidence will replicate. However, the extent to which the eligibility criteria identify the appropriate population is unknown, which raises conce
Abstract by Amelia Averitt; Chunhua Weng; Patrick Ryan; Adler Perotte, npj Digital Medicine (2020) — licensed CC BY 4.0.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*
Negative / Null Result ReportThe harmonic mean p -value for combining dependent tests
Negative / Null Result ReportGeneralizability of heterogeneous treatment effect estimates across samples
Negative / Null Result ReportNumerical predictors of arithmetic success in grades 1–6
Negative / Null Result ReportMethods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Negative / Null Result ReportShould multiple imputation be the method of choice for handling missing data in randomized trials?
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: OpenAlex · DOI 10.1038/s41746-020-0277-8
