|Home | Sitemap
The evidence for evidence-based medicine
448 NE Ravenna Blvd., #106,
Seattle, WA 98115, USA
PO Box 5231,
Glendale, CA 91221, USA
with permission from Complementary Therapies in Medicine
(2000), 8, 123-126 © 2000 Harcourt Publishers Ltd. (This article
has also been accepted for publication in the Winter 2000-2001 edition
of the Scientific Review of Alternative Medicine.)
In recent years the claim
that only 20% or less of standard Western medicine is evidence-based has
been repeated widely by health professionals and others.
This assertion is perhaps most often made by proponents of unproven ('alternative'
and 'complementary') therapies with the implication that, if true, it might
somehow justify the integration of any number of unconventional modalities
with a similar dearth of supporting scientific evidence into main-stream
medical practice. It should be immediately noted that this line of reasoning
is an example of the logical fallacy tu quoque ('you did it too'):
one party cannot criticize another because both parties are guilty of the
same 'sin.' While this argument may be without merit, it is often made
and widely held to be valid. Therefore, the authors of this paper have
attempted to identify the sources of, and examine the evidence for, the
'20% or less' claim.
ORIGIN OF THE CLAIMS
Laments about the state of
conventional medicine are nothing new. In 1861, Oliver Wendell Holmes wrote:
'I firmly believe that if the whole materia medica as used now, could be
sunk to the bottom of the sea, it would be all the better for mankind -
and all the worse for the fishes.'
The original claim that 'It
has been estimated that only 10 to 20% of all procedures currently used
in medical practice have been shown to be efficacious by controlled
trial' first appeared in print in a document published by the U.S.
Congressional Office of Technology Assessment (OTA) in 1979
and was repeated in 1983. The claim stems
from the comments of OTA advisory panel member and noted epidemiologist,
Kerr White. Dr White based his informal '10-20%' estimate on a 1963 paper
that reported on two surveys of the prescribing practices of 19 family
doctors in a northern British town for 2 weeks (one conducted in December
1960, and another in March 1961). Interestingly,
the paper was never intended to evaluate the science of medical practice,
rather its purpose was to look toward controlling prescribing costs in
terms of standard (i.e., 'generic') versus 'proprietary' drugs. The 'intent'
of each prescription was analyzed according to how specific it was for
the condition. Intent was 'specific' for the condition for which it was
pre-scribed only about 10% of the time; 'probable' in about 22%; 'possible'
in 26%; 'hopeful' in 28%; 'placebo' in 10%; and, 'not stated' in 3.6%.
From these data White estimated that 'specific measures' accounted for
10-20% of the benefits of patient care, that the combined placebo and other
non-specific effects accounted for another 20-40%; and the rest (which
he referred to as a 'mystery') accounts for 40-70%.
In 1995, Dr White stated:
Some 20 years ago,
as a member of the original Health Advisory Panel to the US Congressional
Office of Technology Assessment I ventured the 10-20% figure again and
invited anyone to provide more timely data. No one could. The figure was
immortalized in OTA circles and publications for almost a decade. In countless
addresses and conferences I often challenged others to provide better evidence
but none was forthcoming. So the northern industrial town 'arm-chair' assessment
Little about these surveys was
relevant to medical practice across-the-board when they were first published
nearly four decades ago, and they are almost certainly even less relevant
today. Dr White himself has noted that his assessments were never intended
to be applied generally.
Nevertheless, even more gloomy
pronouncements as to the evidential basis for medical practice have subsequently
turned up in the medical literature.,
In 1991, Dr David Eddy, at a conference in Manchester, UK, claimed that
only 15% of medical practice was based on any evidence at all.
He apparently based this sweeping conclusion entirely on his studies of
treatments for just two specific conditions: arterial blockage in the legs
and glaucoma. Subsequently, Dr Eddy's
claim, rather than the much more conservative OTA 'armchair estimate,'
has been widely cited as a criticism of mainstream medicine.
IS MEDICAL PRACTICE EVIDENCE-BASED?
Regardless of the origin
or intent of the original assessments, critics of the '10 to 20%' claims
were originally unable to refute them because no solid evidence existed
either in favor of or against them. That situation has changed in recent
years. A growing body of evidence now exists regarding the extent to which
medical practice is evidence-based.
Still, in order to fully
respond to either claim, one must ask, 'What constitutes acceptable scientific
evidence of efficacy, and how might one establish the relative "weight"
to be ascribed to different types of evidence?' Various rating systems
have been devised, some describing levels of evidence ranging from I to
V, with evidence from randomized controlled trials (RCTs) being generally
given a rating of level I, and the lowest grade being generally assigned
to interventions performed without substantial evidence. Interventions
other than level I that are nonetheless considered compelling evidence
include evidence from prospective and/or comparative studies, and evidence
from follow-up studies and/or retrospective case series.
One category of evidence
that appears to be unique to science-based medicine, and the occasional
subject of criticism from those who wish to criticize the concept of evidence-based
practice, are so-called self-evident
interventions. These are incidences of treatments without compelling evidence
obtained from RCTs and are considered as evidence in discussions of the
extent of evidence-based practice. Examples of such interventions include
blood transfusions, starting the stopped hearts of victims with heart attacks,
antibiotics for meningitis, or a tourniquet for a gushing wound. Such interventions
would not require RCTs to demonstrate proof of efficacy; indeed, such trials
would most likely be considered unethical. There appear to be no comparable
situations of the obvious necessity for and benefit from the interventions
of 'alternative' medicine. Consequently, it would appear that 'compelling
evidence' may occasionally be obtained from uncontrolled case series in
science-based medicine but probably not in 'alternative' medicine.
Evaluations of published
studies suggest that Dr. White's and the OTA's figures substantially underestimate
the extent to which clinical decisions are or could be made on the basis
of evidence from randomized trials only. Evaluations of those same
studies suggest that Dr Eddy's pronouncements wildly underestimate the
extent to which standard medical practice is based on any evidence.
Contrary to the claims, evidence-based practice appears to be prevalent,
and it appears to be widely distributed geographically. Evidence
for evidence-based practice includes those listed in the box (q.v.).
. 96.7% of anesthetic interventions
(32% by RCT, UK)
. approximately 77% of dermatologic
out-patient therapy (38% by RCT, Denmark)
. 64.8% of 'major therapeutic
interventions' in an internal medicine clinic (57% by RCT, Canada)
. 95% of surgical interventions
in one practice (24% by RCT, UK)
. 77% of pediatric surgical
interventions (11% by RCT, UK)
. 65% of psychiatric interventions
(65% by RCT, UK)
. 81% of interventions in
general practice (25.5% by RCT, UK)
. 82% of general medical
interventions (53% by RCT, UK)
. 55% of general practice
interventions (38% by RCT, Spain)
. 78% of laparoscopic procedures
(50% by RCT, France)
. 45% of primary hematology-oncology
interventions (24% by RCT, USA)
. 84% of internal medicine
interventions (50% by RCT, Sweden)
. 97% of pediatric surgical
interventions (26% by RCT, UK)11
. 70% of primary therapeutic
decisions in a clinical hematology practice (22% by RCT, UK)
. 72.5% of interventions
in a community pediatric practice (39.9% by RCT, UK)
Thus, published results show
an average of 37.02% of interventions are supported by RCT (median = 38%).
They show an average of 76% of interventions are supported by some form
of compelling evidence (median = 78%).
There appear to be some areas
of medical practice where interventions are less frequently based on level
I evidence than others. Published surveys of ENT surgery,
burn therapy, retinal breaks and lattice
degeneration and pediatric surgery
have concluded that there is not a strong foundation of evidence obtained
from RCTs on which to base practice in these areas. However, in the studies
of burn therapy and pediatric surgery, it was noted that the number of
RCTs has grown dramatically in the past decade. This suggests that those
practising in these fields are aware of the need to generate unbiased data
in support of clinical practice and that they support the effort to develop
effective practice guidelines.
Calls for the evidence-based
practice of 'complementary' medicine have also been issued,
and established scientific methodologies have been deemed 'quite satisfactory'
for addressing the majority of study questions related to 'alternative'
medicine by the United States Office of Alternative Medicine.
DOES EVIDENCE-BASED PRACTICE
Basing medical practice on
the best available scientific evidence does have its critics. Some, for
instance, assert that this philosophy of practice has major limitations
when considering the care of individual patients. Others have argued that
'science' and 'objectivity' are themselves merely arbitrary 'social constructs,'
and therefore anecdote, testimony, and clinical (personal) experience should
be afforded equal weight to ostensibly more objective scientific lines
of evidence. Still other critics of EBM note that the data available under
its framework may not apply to many treatments offered to patients in clinical
practice or to subgroups of various diseases, nor may it be applicable
to various types of prophylactic interventions, diagnostic decisions, or
Notwithstanding such criticisms
or claims regarding the prevalence of evidence-based medical practice,
health professionals must address the essential question: 'Does providing
evidence-based care improve outcomes for patients?' Unfortunately, no pertinent
data is currently available from randomized controlled trials, most likely
because no investigative team or research granting agency has yet overcome
the problems of sample size, contamination, blinding and long-term follow-up
that such trials would entail. Moreover, such trials pose serious ethical
questions and concerns: for instance, would it be ethical to withhold evidence-based
treatment from the control arm?
On the other hand, 'outcomes
research' has documented that patients who receive evidence-based therapies
often have better outcomes than those who don't. For example, myocardial
infarction survivors prescribed aspirin
or beta-blockers have lower mortality
rates than those who aren't prescribed those drugs. Where clinicians use
more warfarin and stroke unit referrals, stroke mortality declines by >
20%. For a negative example, patients
undergoing carotid surgery, despite failing to meet evidence-based operative
criteria, when compared with operated patients who meet those criteria
are more than three times as likely to suffer major stroke or death in
the next month.
Dr White has stated that
the '10-20%' figure was used heuristically to stimulate the search for
more accurate information. To some extent, he has succeeded in attaining
that goal. However, Dr White also notes that he had no control over the
fact that the OTA used his 'armchair estimate' in its final report, and
that neither he nor the OTA can be blamed for the abuse of the statement.
Clearly the intent of the OTA report was to strengthen the scientific basis
for medical care, not to promote an 'open door policy' for unproven alternative
and complementary therapies. In 1995,
Dr White stated that he suspected the proportion of interventions based
on evidence was higher than 20%.3 Even if Dr. Eddy's estimates were accurate
with regard to the two conditions he studied a decade ago, they appear
to be clearly inapplicable to many conditions and therapeutic interventions
which have been evaluated more recently.
Whatever the merits or faults
of evidence-based medicine, a growing body of evidence demonstrates that
the practice is widespread and becoming more so. More importantly, there
is emerging evidence that, when EBM is practised, patients benefit. Clearly,
demanding rigorous evidence in evaluating the effectiveness of medical
interventions is a good thing. One may quibble with bits of evidence provided
in individual studies: for example, the figures cited above are lower when
the results of RCTs are considered as 'evidence,' although they are still
higher than the '10-20%' figure. In any case, while the evidence for evidence-based
medicine may be held, for good reason, to exclude anecdote and subjective
personal experience, it is not restricted to randomized trials and meta-analyses.
Rather, it involves tracking down the best objective evidence in order
to answer clinically relevant questions.
Claims that conventional
medicine is not widely based on evidence should be rejected, as should
logically fallacious arguments based on such claims. The evidence
fails to support them.
Therapies in Medicine
Heptonstall J. Traditional Chinese Medical Science. eBMJ, 11 Nov
Holmes OW. In: Strasuu MB, ed. Familiar quotations. Boston: Little
Brown, 1968: 124.
Congress of the United States, Office of Technology Assessment: Assessing
the efficacy and safety of medical technologies. Washington, DC: US Government
Printing Office, 1978.
Congress of the United States, Office of Technology Assessment. The impact
of randomized clinical trials on health care policy and medical practice.
Washington, DC: US Government Printing Office, 1983.
Forsyth G. An inquiry into the drug bill, Medical Care 1963;1:10-16.
White K. Lancet 1995; 346; 837-838
Ontario College of Physicians and Surgeons, Report of the Ad Hoc Committee
on 'Alternative' Medicine, September, 1997.
Smith R. Where is the wisdom . the poverty of medical evidence. BMJ 1991;
Smith R. The ethics of ignorance. J Med Ethics 1992; 18: 117-118.
Dubinsky M, Ferguson JH. Analysis of the National Institutes of Health
Medicare Coverage Assessment. Int J Technol Assess Health Care 1990; 6:
Baraldini V, Spitz V, Pierro A. Evidence-based operations in paediatric
surgery. Pediatr Surg Int 1998; 13(5-6): 331-335.
Churchill W. Clarifications. eBMJ, 20 Nov 1999,
Myles PS, Bain DL, Johnson F, et al. Is anaesthesia evidence-based? A survey
of anaesthetic practice. Br J Anaesth 1999 Apr,82(4): 591-595.
Jemec GB, Thorsteinsdottir, H, Wulf, HC. Evidence-based dermatologic out-patient
treatment. Int J Dermatol 1998 Nov; 37(11): 850-854.
Michaud G, McGowan JL, van der Jagt R, Wells G, Tugwell P. Are therapeutic
decisions supported by evidence from health care research? Arch Intern
Med 1998; 158(15): 1665-1668
Howes N, Chagla L, Thorpe M, et al. Surgical practice is evidence based.
Br J Surg 1997; 84(9): 1220-1223.
Kenney SE, Shankar KR, Rintala KR, et al. Evidence-based surgery: interventions
in a regional paediatric surgical unit. Arch Dis Child 1997; 76(1): 50-53.
Geddes JR, Game D, Jenkins NE, et al. What proportion of primary psychiatric
interventions are based on evidence from randomised controlled trials?
Qual Health Care 1996; 5(4): 215-217
Gill P, Dowell AC, Neal RD, et al. Evidence based general practice: a retrospective
study of interventions in one training practice. BMJ 1996; 312(7034): 819-821.
Ellis J, Mulligan I, Rowe J, et al. Inpatient general medicine is evidence
based. A-Team, Nuffield Department of Clinical Medicine. Lancet 1995; 346(8972):
Suarez-Varela MM, Llopis-Gonzalez A, Bell J, Tallon-Guerola M, et al. Evidence
based general practice. Eur J Epidemiol 1999; 15(9): 815-819.
Slim K, Lescure G, Voitellier M, et al. [Is laparoscopic surgical practice
'factual' (evidence based)? Results of a prospective regional survey].
Presse Med 1998; 27(36): 1829-1833.
Djulbegovic B, Loughran TP Jr, Hornung CA, et al. The quality of medical
evidence in hematology-oncology. Am J Med 1999; 106(2): 198-205.
Nordin-Johansson A, Asplund K J Randomized controlled trials and consensus
as a basis for interventions in internal medicine. Intern Med 2000; 247(1):
Galloway M, Baird G, Lennard A. Haematologists in district general hospitals
practise evidence based medicine. Clin Lab Haematol 1997; 19(4): 243-248.
Rudolf MC, Lyth N, Bundle A, et al. A search for the evidence supporting
community paediatric practice. Arch Dis Child 1999; 80(3): 257-261.
Maran HG, Malony NC, Armstrong MW, et al. Is there an evidence base
for practice in ENT surgery? Clin Otolyngol 1997;22(2):152-157
Childs C. Is there an evidence-based practice for burns? Burns 1998;
Wilkinson CP. Evidence-based analysis of prophylactic treatment of asymptomatic
retinal breaks and lattice degeneration. Opthalmology 2000 107(1): 12-15
Hardin WD Jr, Stylianos S, Lally KP J. Evidence-based practice in pediatric
surgery. Pediatr Surg 1999; 34(5):908-912.
Lewith GT, Ernst E, Mills S, et al. Complementary medicine must be research
led and evidence based. BMJ. 2000; 320(7228): 188.
Levin JS, Glass TA, Kushi LH, et al. Quantitative methods in research on
complementary and alternative medicine. A methodological manifesto. NIH
Office of Alternative Medicine. Med Care 1997 Nov; 35(11):1079-1094
Feinstein AR, Horwitz RI. Problems in the 'evidence' of 'evidence-based
medicine'. Am J Med 1997; 103(6):529-535.
Krumholz HM, Radford MJ, Ellerbeek FF, et al. Aspirin for secondary prevention
after acute myocardial infarction in the elderly; prescribed use and outcomes.
Ann Intern Med 1996; 124; 292-298.
Krumholz HM, Radford MJ, Wang Y, et al. National use and effectiveness
of beta-blockers for the treatment of elderly patients after acute myocardial
infarction. National Cooperative Cardiovascular Project. JAMA 1998; 280:
Mitchell JG, Ballard DJ, Whisnant JP, et al. What role do neurologists
play in determining the costs and outcomes of stroke patients? Stroke 1996;
Wong JH, Findlay JM, Suarez-Almazor ME. Regional performance of carotid
endarterectomy appropriateness, outcomes and risk factors for complications.
Stroke 1997; 28: 891-898.
Blame for Abuse of OTA Reports of Dubious Estimate: 'Only 10-20% of Medical
Procedures are Proved.' NCAHF Newsletter, 1996: 19(5).
Sackett DL, Rosenberg WMC, Muir Gray JA, et al. Evidence based medicine:
what it is and what it isn't. BMJ 1996; 312: 71-72.