Agree Review
AGREE II Instrument
Each clinical practice guideline (CPG) was evaluated individually by four appraisers from the Can-SCIP Expert Panel using the Appraisal of Guidelines for Research and Evaluation II (AGREE II; https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-23-item-Instrument-2009-Update-2017.pdf).
The AGREE II instrument evaluates the guideline development process and quality of the guideline across 6 domains including:
- Scope and purpose,
- Stakeholder involvement,
- Rigour of development,
- Clarity of presentation,
- Applicability, and
- Editorial independence.
| Domain 1. Scope and purpose | Overall aim of the guideline, the specific health questions and target population. |
| Domain 2. Stakeholder Involvement | The extent to which the guideline was developed by appropriate stakeholders and represents the views of its intended users. |
| Domain 3. Rigour of development | Process used to gather and synthesize the evidence, the methods to formulate the recommendations and to updated them. |
| Domain 4. Clarity of presentation | Language, structure and format of the guideline. |
| Domain 5. Applicability | Likely barriers and facilitators to implementation, strategies to improve uptake and resource implications of applying the guideline. |
| Domain 6. Editorial Independence | Formulation of the recommendations not being unduly biased with completing interests. |
| Overall Assessment | Rating the overall quality of the guideline and whether the guideline would be recommended for use in practice. |
Training sessions were held to ensure all Expert Panel members were familiar with using the AGREE instrument. Each CPG was given a standardized score ranging from 1-100 (100 representing a strong score) by the reviewing appraiser from the expert panel. To view a copy of the AGREE II Training Session slide deck, please click here.
Scoring the AGREE II
As written in the user manual, the six domains scores are independent and should not be aggregated into a single quality score. For each domain, standardizing score are calculated by summing up all the scores of the individual items in a domain and standardizing the total as a percentage of the maximum possible score for the domain.
Obtained score − Minimum possible score
Maximum possible score − Minimum possible score
AGREE II Results
As the AGREE User Manual does not specify a minimum score that is considered 'low-quality,' the Can-SCIP steering committee set a benchmark of 40% for inclusion; whereby scores higher than 40% represent higher quality, and scores below 40% represent poorer quality. CPGs with an AGREE score below 40% were excluded from the recommendation review process.
Each guideline was given a standardized score ranging from 1-100 (100 representing a strong score) by the reviewing expert. The AGREE scores were summarized and presented to the expert panels at the conference in October 2019. Only one existing CPG did not meet the quality requirements at the outset of the AGREEE evaluation process and was therefore excluded. Thirty-seven (37) CPGs met our inclusion criteria and were of adequate quality were retained for the adaptation process. To view all CPGs evaluated using the AGREE II instrument, please click here.
To view an overview of the AGREE II Data for each Domain, please click here.
Domains with High Scores
Domain scores were high for “stakeholder involvement,” “rigour of development,” “clarity of presentation,” and “editorial independence,” with median scores of 69.4%, 79.2%, 87.5% and 79.2 %, respectively. The highest domain score was achieved in domain 1, “scope and purpose,” with a median score of 86.11%.
Domains with Low Scores
Standardized scores for domain 5 of the AGREE II tool (“applicability”) were low, with a median of 56.67. “Applicability” had the greatest number of standardized domain scores less than 40% (n=10). Three CPGs had a standardized “applicability” domain score between 30 and 39 percent, 1 CPG between 20 and 29 percent and 1 CPG with a score between 10 and 19 percent. The lowest standardized domain scores were under “stakeholder involvement” (domain 2) at 7.8% for two CPGs.
Quality of Available Evidence within SCI
Based on the standardized domains scores, the CPGs that scored the highest rating within all six domains included Wounds Canada Best Practice Recommendations, Evidence-Based Scientific Exercise Guidelines for Adults with SCI, Home Mechanical Ventilation: A Canadian Thoracic Society CPG, CPG for the Management of Patients With Acute SCI: Recommendations on the Type and Timing of Rehabilitation, and Management of Neurogenic Bowel Dysfunction in Adults after Spinal Cord Injury: CPG for Health Care Providers.
Future guideline development groups should clearly describe the facilitators and barriers to implementing the CPG, tools and resources to facilitate dissemination and implementation of the CPG, and the strategies used to incorporate the views and preferences of persons with lived experience throughout the CPG development process.
Further, only 3 CPGs presented the cost implications (i.e. economic evaluation, drug acquisition costs for each treatment) of applying the recommendations. Additional research on the resource implications and cost-effectiveness during CPG development and implementation is needed. Future CPG development groups should also consider incorporating the evaluation content within each AGREE II domain to develop high-quality CPGs using a systematic and rigorous process.
There is a paucity of CPGs that address community-based specialized rehabilitation and community reintegration. As well, only several community-based studies that address the needs of people with SCI over their lifetime are available. In addition, many CPGs only focus on a single impairment or organ system at a particular time point in the care continuum (i.e. specialized rehabilitation).