Misconceptions, Challenges, Uncertainty, and Progress in Guideline Recommendations

doi:10.1053/j.seminhematol.2008.04.005

Seminars in Hematology

Volume 45, Issue 3, July 2008, Pages 167-175

https://doi.org/10.1053/j.seminhematol.2008.04.005 Get rights and content

Although the quality of clinical practice guidelines has improved over the last decade, current guideline systems have limitations that reduce their validity and limit their acceptance. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) working group, a worldwide collaboration of guideline developers, methodologists, and clinicians, has constructed a system for developing guidelines that addresses these shortcomings. The system includes a transparent and rigorous methodology for rating the quality of evidence, an explicit balancing of benefits and harms of healthcare interventions, an explicit acknowledgement of the values and preferences that underlie the recommendations, and whether the intervention represents a wise use of resources. These four elements determine whether a recommendation is strong or weak. A guideline panel offers strong recommendations when virtually all informed patients would choose the same management strategy. Weak recommendations imply that choices will differ across the range of patient values and preferences. The GRADE system has been tested in multiple practice settings and for a large spectrum of questions, refined and re-evaluated to ensure that it captures the complex issues involved in evidence assessment and grading recommendations while maintaining simplicity and practicality. Many guideline organizations and medical societies have endorsed the system and adopted it for their guideline processes.

Section snippets

Steps in the Development of Recommendations in the GRADE System

Using the GRADE approach to developing guidelines involves two steps: first, consideration of the quality of the evidence; second, development of a graded recommendation that considers not only the quality of the evidence but also the balance of desirable and undesirable consequences of an intervention, the (un-)certainty regarding the values and preferences patients attribute to the desirable and undesirable consequences, and resource use.

Strength of Recommendations: How to Grade and Why

The GRADE system offers two categories of recommendations: strong and weak, either of which can be for, or against, an intervention in comparison to an alternative. For GRADE, the strength of a recommendation describes to what extent guideline developers are confident that the desirable consequences of an intervention outweigh its undesirable consequences. Desirable interventions reduce morbidity and mortality, improve quality of life, decrease inconvenience, and incur few adverse effects or

Guidelines for Whom?

In the past, clinician experts wrote guidelines primarily for their peers to help them select the best treatment for their patients. The increasing access of patients to healthcare information, together with the movement for consumer and patient autonomy, has made patients a receptive audience for structured healthcare information and advice, thus creating a partnership with health care professionals. A final audience consists of those involved in regulation: guidelines have become a source for

Interpreting Strong and Weak Recommendations

As defined above, a strong recommendation indicates that the desirable consequences of an intervention will very likely outweigh any potential undesirable consequences, while a weak recommendation leaves that judgment less certain. The implications of a recommendation, whether strong or weak, will vary depending on the target audience. For patients, a strong recommendation implies that, when fully informed, they are very likely to make the same choice. Use of a decision aid—a tool to inform

Factors Affecting the Strength of a Recommendation

While many organizations agree that high-quality evidence need not result in a strong recommendation,³ few have made explicit the process of translating evidence into recommendations. The GRADE system has identified four factors that determine the direction and strength of a recommendation (Table 1).

Rating Quality of Evidence: What Aspects to Consider?

The starting point for any quality assessment is a structured question that specifies the population, the intervention(s), the comparator(s), and the outcomes of interest, such as: “In lymphoma patients at risk of developing chemotherapy-induced febrile neutropenia, what are the benefits and harms associated with the use of granulocyte colony-stimulating factor (G-CSF) compared to not using G-CSF?”²⁴ Under ideal circumstances, a guideline panel can find the answers to its question in a recent

Rating Quality of Evidence—The Clinical Context Matters

The GRADE system classifies evidence for both beneficial and harmful effects into four different levels: high, moderate, low, and very low quality. Table 3 explains the implications of each level.

The definitions reflect the extent to which we are confident that an estimate of effect is correct. Guideline panels need to judge the quality of evidence relative to the particular clinical context for which they are using the evidence. Therefore, in the context of guidelines, the quality of evidence

Discussion

In a world of rapidly changing evidence and information overload, physicians want up-to-date guidance for the management of their patients. In an emerging world of unlimited access to healthcare information, patients want to be assured that the information they receive is rigorous and robust. In order to satisfy these needs, clinical practice guidelines must provide clear and transparent recommendations for the best care of patients, and a simple, easy-to-grasp presentation of results and

Conclusion

The GRADE framework is founded in a rigorous and transparent methodology for assessing evidence, balancing benefits and harms, acknowledging values and preferences underlying specific recommendations, and integrating considerations of resource use. Transparency of each step in guideline development enables clinicians and patients to better understand and integrate the recommendations in the care of individual patients.

Acknowledgment

The authors acknowledge the members of the GRADE working group who participated over the last years in developing the framework.

References (42)

R. Alexanian et al.
Primary dexamethasone treatment of multiple myeloma
Blood
(1992)
D. Verhelst et al.
Treatment of erythropoietin-induced pure red cell aplasia: A retrospective study
Lancet
(2004)
J.J. Cornelissen et al.
Results of a HOVON/SAKK donor versus no-donor analysis of myeloablative HLA-identical sibling stem cell transplantation in first remission acute myeloid leukemia in young and middle-aged adults: Benefits for whom?
Blood
(2007)
S.D. Ross et al.
Clinical benefits and risks associated with epoetin and darbepoetin in patients with chemotherapy-induced anemia: A systematic review of the literature
Clin Ther
(2006)
H.J. Schunemann et al.
Letters, numbers, symbols and words: How to communicate grades of evidence and recommendations
CMAJ
(2003)
T. Shaneyfelt et al.
Are guidelines following guidelines?The methodological quality of clinical practice guidelines in the peer-reviewed medical literature
JAMA
(1999)
Developing a Methodology for Drawing up Guidelines on Best Medical Practice (Recommendations Rec(2001)13) adopted by the Committee of Ministers of the Council of EuropeRecommendation Rec(2001)
(2001)
S. West et al.
Systems to rate the strength of scientific evidenceEvidence Report/Technology Assessment No 47
(2002)
C. Charles et al.
Treatment decision aids: conceptual issues and future directions
Health Expect
(2005)

T. Whelan et al.

Effect of a decision aid on knowledge and treatment decision making for breast cancer surgery: A randomized trial

JAMA

(2004)

Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: An overview of 6,633 patients from 27 randomized trials. Myeloma Trialists' Collaborative Group

J Clin Oncol

(1998)

H.R. Buller et al.

Antithrombotic therapy for venous thromboembolic disease: The Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy

Chest

(2004)

Interferon as therapy for multiple myeloma: an individual patient data overview of 24 randomized trials and 4012 patients

Br J Haematol

(2001)

H. Ludwig et al.

Patient preferences for interferon alfa in multiple myeloma

J Clin Oncol

(1997)

WHO Rapid Advice Guidelines on pharmacological management of humans infected with avian influenza A (H5N1) virus

Information for Healthcare ProfessionalsErythropoiesis Stimulating Agents (ESA). US Food and Drug Administration

G.M. Rodgers

NCCN Practice Guidelines in Oncology: Cancer and Treatment related AnemiaVersion 2.2006

J. Wilson et al.

A systematic review and economic evaluation of epoetin alpha, epoetin beta and darbepoetin alpha in anaemia associated with cancer, especially that attributable to cancer treatment

Health Technol Assess

(2007)

Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: An overview of the randomised trials

Lancet

(2005)

B. Djulbegovic et al.

Bisphosphonates in multiple myeloma

Cochrane Database Syst Rev

(2002)

Cited by (15)

Effectiveness of osteopathic manipulative treatment for pediatric conditions: A systematic review
2022, Journal of Bodywork and Movement Therapies
Citation Excerpt :
Treatment results and overall quality of the evidence for each outcome were summarized in tabular form. The evidence for each outcome was assessed using the GRADE approach (Guyatt et al., 2011; Kunz et al., 2008). The GRADE approach specifies four levels of quality with the highest rating for RCT evidence.
Osteopathic manipulative treatment (OMT) is commonly used by osteopaths and osteopathic physicians to manage a large variety of pediatric complaints.
The current study reviewed the literature to determine the effectiveness of OMT for all pediatric complaints.
A systematic literature search for randomized controlled trials (RCTs) unrestricted by language or publication status was performed in July 2020 in electronic and ongoing trials databases. Included studies were assessed using the Cochrane Risk of Bias (RoB) instrument. Mean difference or standard mean difference and overall effect size were calculated. Data were synthesized using the GRADE approach.
Forty-seven RCTs examining 37 pediatric conditions were reviewed. Twenty-three studies reported significant favorable outcomes for OMT relative to the control intervention, and 14 additional studies reported non-significant outcomes, which suggested potential favorable effects of OMT. Fifteen of the studies were judged to have a low RoB, 12 had high risk, and the remainder had unclear RoB. Using GRADE, there was moderate evidence for the effectiveness of OMT for 13 of the 43 comparisons, particularly for length of hospital stay for preterm infants, but no high-quality evidence for any condition.
Although a number of studies indicated positive results with use of OMT, few pediatric conditions have been investigated in more than one study, which results in no high-quality evidence for any condition. Additional research may change estimates of effect, and larger, high-quality RCTs focusing on a smaller range of conditions are recommended.
PROSPERO ID: CRD42020162479.
The ecosystem of health decision making: from fragmentation to synergy
2022, The Lancet Public Health
Citation Excerpt :
Also, evidence syntheses for policy commonly assess questions related to complex interventions, focusing not only on the interventions’ intrinsic benefits and harms, but also how to ensure better results through adequate contextualisation with intervention adaptations (if needed) and effective implementation.70 According to the European Evidence-Informed Policy Network manual for developing EIP, the criteria and considerations related to choosing among options are feasibility, the possible effects (eg, benefits and harms), equity, cost, cost-effectiveness, and implementation considerations, which describe the strategies to overcome barriers and assist facilitators in implementing the options.71 This evidence also can be summarised using systematic review methodology.
Clinicians, patients, policy makers, funders, programme managers, regulators, and science communities invest considerable amounts of time and energy in influencing or making decisions at various levels, using systematic reviews, health technology assessments, guideline recommendations, coverage decisions, selection of essential medicines and diagnostics, quality assurance and improvement schemes, and policy and evidence briefs. The criteria and methods that these actors use in their work differ (eg, the role economic analysis has in decision making), but these methods frequently overlap and exist together. Under the aegis of WHO, we have brought together representatives of different areas to reconcile how the evidence that influences decisions is used across multiple health system decision levels. We describe the overlap and differences in decision-making criteria between different actors in the health sector to provide bridging opportunities through a unifying broad framework that we call theory of everything. Although decision-making activities respond to system needs, processes are often poorly coordinated, both globally and on a country level. A decision made in isolation from other decisions on the same topic could cause misleading, unnecessary, or conflicted inputs to the health system and, therefore, confusion and resource waste.
Osteopathic manipulative treatment for low back and pelvic girdle pain during and after pregnancy: A systematic review and meta-analysis
2017, Journal of Bodywork and Movement Therapies
Citation Excerpt :
A large effect was defined as MD greater than 20% of the scale and SMD or ‘d’ scores greater than 0.8 (Furlan et al., 2009). The overall quality of the evidence for each outcome in the included studies was assessed using the GRADE approach (Guyatt et al., 2011; Kunz et al., 2008), as recommended by the updated Cochrane Back Review Group method guidelines (Furlan et al., 2009). The GRADE approach specifies 4 levels of quality, the highest rating being for RCT evidence.
Low back pain (LBP) is a common complaint during pregnancy. This study examined the effectiveness of osteopathic manipulative treatment (OMT) for LBP in pregnant or postpartum women.
Randomized controlled trials unrestricted by language were reviewed. Outcomes were pain and functional status. Mean difference (MD) or standard mean difference (SMD) and overall effect size were calculated.
Of 102 studies, 5 examined OMT for LBP in pregnancy and 3 for postpartum LBP. Moderate-quality evidence suggested OMT had a significant medium-sized effect on decreasing pain (MD, −16.65) and increasing functional status (SMD, −0.50) in pregnant women with LBP. Low-quality evidence suggested OMT had a significant moderate-sized effect on decreasing pain (MD, −38.00) and increasing functional status (SMD, −2.12) in postpartum women with LBP.
This review suggests OMT produces clinically relevant benefits for pregnant or postpartum women with LBP. Further research may change estimates of effect, and larger, high-quality randomized controlled trials with robust comparison groups are recommended.
Osteopathic treatment of chronic nonspecific neck pain
2017, Osteopathische Medizin
Unspezifische Nackenschmerzen sind häufig, funktionseinschränkend und führen zu hohen Kosten. Ziel der vorliegenden Übersichtsarbeit war, die Wirksamkeit der osteopathischen Behandlung hinsichtlich Schmerz, funktioneller Einschränkung und Nebenwirkungen bei chronischen, unspezifischen Nackenschmerzen zu untersuchen.
Eine systematische Literaturrecherche ohne sprachliche Einschränkung wurde im März 2014 in verschiedenen elektronischen Datenbanken durchgeführt. Es wurden nur randomisierte klinische Studien berücksichtigt, Studien mit spezifischen Nackenschmerzen oder Studien, die auf einer einzigen Behandlungstechnik beruhten, wurden ausgeschlossen. Primäre Ergebnisparameter waren Schmerz und funktioneller Status, Nebenwirkungen wurden als sekundärer Ergebnisparameter erfasst. Die Studien wurden von allen drei Autoren unabhängig voneinander untersucht. Dabei wurden die Effektmaße Mean Difference (MD) oder Standard Mean Difference (SMD) mit einem 95%igen Konfidenzintervall ermittelt. Um die Qualität der Evidenz zu bewerten, wurde das GRADE-Verfahren (Grading of Recommendations, Assessment, Development and Evaluation) genutzt.
Es wurden 299 Studien identifiziert, von denen 18 näher geprüft wurden. Drei Studien erfüllten die Einschlusskriterien. Es bestand eine moderate Qualität der Evidenz, dass die osteopathische Behandlung bei chronischen Nackenschmerzen zu einer signifikanten und klinisch relevanten Schmerzerleichterung (MD, −13,04; 95% CI, −20,64 zu −5,44) und zu einer nicht signifikanten Verbesserung des funktionellen Status (SMD, −0,38; 95% CI, −0,88 zu 0,11) führte.
Die Metaanalyse weist auf eine klinisch relevante Schmerzerleichterung bei Patienten mit chronischen, unspezifischen Nackenschmerzen hin. Angesichts der kleinen Fallzahl, der unterschiedlichen Vergleichsgruppen, der geringen Studienzahl und dem Fehlen von Langzeitergebnissen wären größere, qualitativ hochwertige, randomisierte klinische Studien wünschenswert.
Nonspecific neck pain is common, disabling, and costly. The objective of the current review was to assess the effectiveness of osteopathic manipulative treatment (OMT) in the management of chronic nonspecific neck pain regarding pain, functional status, and adverse events.
A systematic literature search unrestricted by language was performed in March 2014 in several electronic databases and in databases of ongoing trials. A manual search of reference lists and personal communication with experts identified additional studies. Only randomized clinical trials were included, and studies of specific neck pain or single treatment techniques were excluded. Primary outcomes were pain and functional status, and secondary outcome was adverse events.
Studies were independently reviewed using a standardized data extraction form. Mean difference (MD) or standard mean difference (SMD) with 95% confidence intervals (CIs) and overall effect size were calculated for primary outcomes. GRADE was used to assess quality of the evidence.
Of 299 identified studies, 18 were evaluated and 15 excluded. The 3 reviewed studies had low risk of bias. Moderate-quality evidence suggested OMT had a significant and clinically relevant effect on pain relief (MD: –13.04, 95% CI: –20.64 to –5.44) in chronic nonspecific neck pain, and moderate-quality evidence suggested a non-significant difference in favour of OMT for functional status (SMD: –0.38, 95% CI: –0.88 to 0.11). No serious adverse events were reported.
Based on the 3 included studies, the review suggested clinically relevant effects of OMT for reducing pain in patients with chronic nonspecific neck pain. Given the small sample sizes, different comparison groups, and lack of long-term measurements in the few available studies, larger, high-quality randomized controlled trials with robust comparison groups are recommended.
Osteopathic manipulative treatment for chronic nonspecific neck pain: A systematic review and meta-analysis
2015, International Journal of Osteopathic Medicine
Citation Excerpt :
A large effect was defined as MD greater than 20% of the scale and SMD or d scores greater than 0.8.22 The overall quality of the evidence for each outcome in the included studies was assessed using the GRADE approach,26,27 as recommended by the updated Cochrane Back Review Group method guidelines.22 The GRADE approach specifies 4 levels of quality, high, moderate, low and very low, and the highest rating is given for RCT evidence.
Nonspecific neck pain is common, disabling, and costly. The objective of the current review was to assess the effectiveness of osteopathic manipulative treatment (OMT) in the management of chronic nonspecific neck pain regarding pain, functional status, and adverse events.
A systematic literature search unrestricted by language was performed in March 2014 in several electronic databases and in databases of ongoing trials. A manual search of reference lists and personal communication with experts identified additional studies. Only randomized clinical trials were included, and studies of specific neck pain or single treatment techniques were excluded. Primary outcomes were pain and functional status, and secondary outcome was adverse events.
Studies were independently reviewed using a standardized data extraction form. Mean difference (MD) or standard mean difference (SMD) with 95% confidence intervals (CIs) and overall effect size were calculated for primary outcomes. GRADE was used to assess quality of the evidence.
Of 299 identified studies, 18 were evaluated and 15 excluded. The 3 reviewed studies had low risk of bias. Moderate-quality evidence suggested OMT had a significant and clinically relevant effect on pain relief (MD: −13.04, 95% CI: −20.64 to −5.44) in chronic nonspecific neck pain, and moderate-quality evidence suggested a non-significant difference in favour of OMT for functional status (SMD: −0.38, 95% CI: −0.88 to 0.11). No serious adverse events were reported.
Based on the 3 included studies, the review suggested clinically relevant effects of OMT for reducing pain in patients with chronic nonspecific neck pain. Given the small sample sizes, different comparison groups, and lack of long-term measurements in the few available studies, larger, high-quality randomized controlled trials with robust comparison groups are recommended.
Interpretation of the ecosystem of health decision making: from fragmentation to synergy
2023, Chinese Journal of Evidence-Based Medicine

View all citing articles on Scopus

: Dr Kunz has been supported by Santesuisse and the Gottfried and Julia Bangerter-Rhyner Foundation.

View full text

Misconceptions, Challenges, Uncertainty, and Progress in Guideline Recommendations

Section snippets

Steps in the Development of Recommendations in the GRADE System

Strength of Recommendations: How to Grade and Why

Guidelines for Whom?

Interpreting Strong and Weak Recommendations

Factors Affecting the Strength of a Recommendation

Rating Quality of Evidence: What Aspects to Consider?

Rating Quality of Evidence—The Clinical Context Matters

Discussion

Conclusion

Acknowledgment

Blood

Lancet

Blood

Clin Ther

Letters, numbers, symbols and words: How to communicate grades of evidence and recommendations

CMAJ

Are guidelines following guidelines?The methodological quality of clinical practice guidelines in the peer-reviewed medical literature

JAMA

Developing a Methodology for Drawing up Guidelines on Best Medical Practice (Recommendations Rec(2001)13) adopted by the Committee of Ministers of the Council of EuropeRecommendation Rec(2001)

Systems to rate the strength of scientific evidenceEvidence Report/Technology Assessment No 47

Treatment decision aids: conceptual issues and future directions

Health Expect

Effect of a decision aid on knowledge and treatment decision making for breast cancer surgery: A randomized trial

JAMA

Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: An overview of 6,633 patients from 27 randomized trials. Myeloma Trialists' Collaborative Group

J Clin Oncol

Antithrombotic therapy for venous thromboembolic disease: The Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy

Chest

Interferon as therapy for multiple myeloma: an individual patient data overview of 24 randomized trials and 4012 patients

Br J Haematol

Patient preferences for interferon alfa in multiple myeloma

J Clin Oncol

WHO Rapid Advice Guidelines on pharmacological management of humans infected with avian influenza A (H5N1) virus

Information for Healthcare ProfessionalsErythropoiesis Stimulating Agents (ESA). US Food and Drug Administration

NCCN Practice Guidelines in Oncology: Cancer and Treatment related AnemiaVersion 2.2006

A systematic review and economic evaluation of epoetin alpha, epoetin beta and darbepoetin alpha in anaemia associated with cancer, especially that attributable to cancer treatment

Health Technol Assess

Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: An overview of the randomised trials

Lancet

Bisphosphonates in multiple myeloma

Cochrane Database Syst Rev