Seminars in Hematology
Volume 45, Issue 3 , Pages 160-166 , July 2008

Perfect Study, Poor Evidence: Interpretation of Biases Preceding Study Design

  • John P.A. Ioannidis

      Affiliations

    • Corresponding Author InformationAddress correspondence to John P.A. Ioannidis, Professor and Chairman, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece.

References 

  1. Schachman HK. From “publish or perish” to “patent and prosper. J Biol Chem. 2006;281:6889–6903
  2. Poynard T, Munteanu M, Ratziu V, Benhamou Y, Di Martino V, Taieb J, et al. Truth survival in clinical research: An evidence-based requiem?. Ann Intern Med. 2002;136:888–895
  3. Zwarenstein M, Oxman A. Why are so few randomized trials useful, and what can we do about it?. J Clin Epidemiol. 2006;59:1125–1126
  4. Frankel A, West R. Rationing and Rationality in the National Health Service. Basingstoke, UK: Macmillan; 1993;
  5. Gray JAM. Evidence-Based Healthcare. London, UK: Churchill Livingstone; 1997;
  6. Djulbegovic B, Loughran TP, Hornung CA, Kloecker G, Efthimiadis EN, Hadley TJ, et al. The quality of medical evidence in hematology-oncology. Am J Med. 1999;106:198–205
  7. Isaakidis P, Swingler GH, Pienaar E, Volmink J, Ioannidis JP. Relation between burden of disease and randomised evidence in sub-Saharan Africa: Survey of research. BMJ. 2002;324:702
  8. Swingler GH, Pillay V, Pienaar ED, Ioannidis JP. International collaboration, funding and association with burden of disease in randomized controlled trials in Africa. Bull World Health Organ. 2005;83:511–517
  9. Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Econ. 1996;5:513–524
  10. Detsky A. Using economic analysis to determine the resource consequences of choices made in planning clinical trials. J Chronic Dis. 1985;38:753–765
  11. Chalmers TC, Lau J. Changes in clinical trials mandated by the advent of meta-analysis. Stat Med. 1996;15:1263–1268
  12. Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR, Abrams KR. Evidence-based sample size calculations based upon updated meta-analysis. Stat Med. 2007;26:2479–2500
  13. Berlin JA, Colditz GA. The role of meta-analysis in the regulatory process for foods, drugs, and devices. JAMA. 1999;281:830–834
  14. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365:1159–1162
  15. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Intern Med. 2001;134:663–694
  16. Weaver CS, Leonardi-Bee J, Bath-Hextall FJ, Bath PM. Sample size calculations in acute stroke trials: A systematic review of their reporting, characteristics, and relationship with outcome. Stroke. 2004;35:1216–1224
  17. Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003;326:1453–1455
  18. Koehler JJ. The influence of prior beliefs on scientific judgments of evidence quality. Organ Behav Hum Decision Processess. 1993;56:28–55
  19. Resch KI, Ernst E, Garrow J. A randomized controlled study of reviewer bias against an unconventional therapy. J R Soc Med. 2000;93:164–167
  20. Tatsioni A, Bonitsis N, Ioannidis JPA. Persistence of contradicted claims in the literature. JAMA. 2007;298:2517–2526
  21. Peppercorn J, Blood E, Winer E, Partridge A. Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer. 2007;109:1239–1246
  22. Ioannidis JP. Indirect comparisons: the mesh and mess of clinical trials. Lancet. 2006;368:1470–1472
  23. Heres S, Davis J, Maino K, Jetzinger E, Kissling W, Leucht S. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: An exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. Am J Psychiatry. 2006;163:185–194
  24. Bero L, Oostvogel F, Bacchetti P, Lee K. Factors associated with findings of published trials of drug-drug comparisons: Why some statins appear more efficacious than others. PLoS Med. 2007;4:e184
  25. Chen YF, Jobanputra P, Barton P, Jowett S, Bryan S, Clark W, et al. A systematic review of the effectiveness of adalimumab, etanercept and infliximab for the treatment of rheumatoid arthritis in adults and an economic evaluation of their cost-effectiveness. Health Technol Assess. 2006;10:1–229
  26. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ. 2003;326:1167–1170
  27. Lexchin J. Implications of pharmaceutical funding on clinical research. Ann Pharmacother. 2005;39:194–197
  28. Morris MC, Nelson RM. Randomized, controlled trials as minimal risk: An ethical analysis. Crit Care Med. 2007;35:940–944
  29. Lilford RJ. Ethics of clinical trials from a bayesian and decision analytic perspective: Whose equipoise is it anyway?. BMJ. 2003;326:980–981
  30. Wolinsky H. The battle of Helsinki: Two troublesome paragraphs in the Declaration of Helsinki are causing a furore over medical research ethics. EMBO Rep. 2006;7:670–672
  31. Ioannidis JP, Lau J. State of the evidence: Current status and prospects of meta-analysis in infectious diseases. Clin Infect Dis. 1999;29:1178–1185
  32. Petersen I, Johnson AM, Islam A, Duckworth G, Livermore DM, Hayward AC. Protective effect of antibiotics against serious complications of common respiratory tract infections: Retrospective cohort study with the UK General Practice Research Database. BMJ. 2007;335:982
  33. Fleming TR, DeMets DL. Surrogate end points in clinical trials: Are we being misled?. Ann Intern Med. 1996;125:605–613
  34. Montori VM, Permanyer-Miralda G, Ferreira-González I, et al. Validity of composite end points in clinical trials. BMJ. 2005;330:594–596
  35. Ferreira-González I, Permanyer-Miralda G, Busse JW, Bryant DM, Montori VM, Alonso-Coello P, et al. Methodologic discussions for using and interpreting composite endpoints are limited, but still identify major concerns. J Clin Epidemiol. 2007;60:651–657
  36. Ferreira-González I, Busse JW, Heels-Ansdell D, Montori VM, Akl EA, Bryant DM, et al. Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ. 2007;334:786
  37. Glasser SP, Salas M, Delzell E. Importance and challenges of studying marketed drugs: What is a phase IV study? (Common clinical research designs, registries, and self-reporting systems). J Clin Pharmacol. 2007;47:1074–1086
  38. Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: Cumulative meta-analysis. Lancet. 2004;364:2021–2029
  39. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. 2007;356:2457–2471
  40. Papanikolaou PN, Ioannidis JP. Availability of large-scale evidence on specific harms from systematic reviews of randomized trials. Am J Med. 2004;117:582–589
  41. Papanikolaou PN, Churchill R, Wahlbeck K, Ioannidis JP. Safety reporting in randomized trials of mental health interventions. Am J Psychiatry. 2004;161:1692–1697
  42. Sismondo S. Ghost management: how much of the medical literature is shaped behind the scenes by the pharmaceutical industry?. PLoS Med. 2007;4:e286
  43. Mirowski P, Van Horn R. The contract research organization and the commercialization of scientific research. Soc Stud Sci. 2005;35:503–534
  44. Ngai S, Gold JL, Gill SS, Rochon PA. Haunted manuscripts: Ghost authorship in the medical literature. Account Res. 2005;12:103–114
  45. Rennie D. Thyroid storm. JAMA. 1997;277:1238–1243
  46. Patsopoulos NA, Ioannidis JP, Analatos AA. Origin and funding of the most frequently cited papers in medicine: Database analysis. BMJ. 2006;332:1061–1064
  47. Sung L, Nathan PC, Alibhai SM, Tomlinson GA, Beyene J. Meta-analysis: Effect of prophylactic hematopoietic colony-stimulating factors on mortality and outcomes of infection. Ann Intern Med. 2007;147:400–411
  48. Kuderer NM, Dale DC, Crawford J, Lyman GH. Impact of primary prophylaxis with granulocyte colony-stimulating factor on febrile neutropenia and mortality in adult cancer patients receiving chemotherapy: A systematic review. J Clin Oncol. 2007;25:3158–3167
  49. Berghmans T, Paesmans M, Lafitte JJ, Mascaux C, Meert AP, Sculier JP. Role of granulocyte and granulocyte-macrophage colony-stimulating factors in the treatment of small-cell lung cancer: A systematic review of the literature with methodological assessment and meta-analysis. Lung Cancer. 2002;37:115–123
  50. Grimes DA, Hubacher D, Nanda K, Schulz KF, Moher D, Altman DG. The Good Clinical Practice guideline: A bronze standard for clinical research. Lancet. 2005;366:172–174
  51. Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ. 2005;330:753
  52. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA. 2004;291:2457–2465
  53. Sterne JA, Davey Smith G. Sifting the evidence—What's wrong with significance tests?. BMJ. 2001;322:226–231
  54. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124
  55. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: Islands in search of continents?. JAMA. 1998;280:280–282
  56. Ioannidis JP, Polyzos NP, Trikalinos TA. Selective discussion and transparency in microarray research findings for cancer outcomes. Eur J Cancer. 2007;43:1999–2010
  57. Ioannidis JP. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60:324–329
  58. Kjaergard LL, Gluud C. Citation bias of hepato-biliary randomized clinical trials. J Clin Epidemiol. 2002;55:407–410
  59. Wells J, Marshall P, Crawley B, et al. Newspaper reporting of screening mammography. Ann Intern Med. 2001;135:1029–1037
  60. Cassels A, Hughes MA, Cole C, Mintzes B, Lexchin J, McCormack JP. Drugs in the news: an analysis of Canadian newspaper coverage of new prescription drugs. CMAJ. 2003;168:1133–1137
  61. Grimshaw J. So what has the Cochrane Collaboration ever done for us? (A report card on the first 10 years). CMAJ. 2004;171:747–749
  62. Salanti G, Higgins J, Ades AE, Ioannidis JP. Evaluation of networks of randomized trials. Stat Methods Med Res. 2007;Oct 9 [Epub ahead of print]. doi:10.1177/0962280207080643
  63. Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148:544–553
  64. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23:3105–3124
  65. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: Combining direct and indirect evidence. BMJ. 2005;331:897–900
  66. Russell K, Kiddoo D. The Cochrane Library and nocturnal enuresis; an umbrella review. Evid Based Child Health. 2006;1:5–8
  67. Papanikolaou GN, Baltogianni MS, Contopoulos-Ioannidis DG, Haidich AB, Giannakakis IA, Ioannidis JP. Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions. BMC Med Res Methodol. 2001;1:3
  68. Krimsky S, Rothenberg LS, Stott P, Kyle G. Scientific journals and their authors' financial interests: A pilot study. Psychother Psychosom. 1998;67:194–201

PII: S0037-1963(08)00066-8

doi: 10.1053/j.seminhematol.2008.04.010

Seminars in Hematology
Volume 45, Issue 3 , Pages 160-166 , July 2008

Article Tools

* Image Usage & Resolution