Using evidence to inform program development
Have you ever struggled with exactly how to find and use research results and other evidence that will inform your program development? Whether trying to decide what type of intervention to use in your community, what advice to give to service users, or what stance to take on an issue, we all need evidence to ensure our work is based on proven methods and sound facts. However, it’s not always easy to turn the available research into the guidance we need.
The National Collaborating Centre for Methods and Tools (NCCMT), an agency that provides expertise in evidence-informed decision-making to Canadian public health organizations, has identified tools that can help.1,2 These tools were created by public health departments and organizations that support evidence-informed decision making. Each tool also corresponds to a different stage in the NCCMT’s seven-step process to guide service providers through evidence-informed decision making.
Define your question
The first step in the NCCMT process is to clearly define your question or problem. Before you can get your hands on evidence, you need to determine exactly why you need it and what you want to know. There are some useful tools at this stage that will help you develop an effective strategy for your evidence search.
- Developing an efficient search strategy using PICO – A worksheet that helps you break down your question and choose search terms based on the population, intervention, comparison and outcome (PICO) you’re interested in.
- Developing a conceptual model – instructions and worksheet – A worksheet that explains the process of developing a visual model of the issue you’re trying to address. The model will help you explore the contextual factors impacting your issue of interest, including their relationship with each other and their outcomes.
Once you’ve defined your question, you’re ready to start your search. Searches can be done more efficiently if you’re looking in the right place for the types of evidence you need and keeping a good record of what you’re doing. Use the following tools to find the most useful types of evidence for your research need.
- The 6S pyramid3 – The 6S pyramid ranks different ways health-related research evidence can be summarized and critically evaluated to make it easier to use and apply in practical ways. Understanding the 6S pyramid will help you target the most synthesized evidence available on your topic.
- Resources to guide and track your search – A list of online sources for finding research. The sources are organized based on the extent to which the evidence is synthesized following the 6S pyramid.
- Keeping track of search results: a flowchart – This flowchart helps you understand the different stages in your search process and track the number of articles remaining as you screen your results for those best fitting your search criteria.
Not everything you find will end up suiting your needs. You may also find research that you don’t want to base decisions on because the methodology for gathering the results was not strong. The process of examining the soundness and quality of research results is known as critical appraisal.
The following tools provide criteria and standards for critically appraising systematic reviews. Each tool presents a series of yes/no questions to ask when assessing the methodology of a systematic review. Use your answers to better understand how sound the review’s results are.
- A measurement tool to assess systematic reviews (AMSTAR)
- Health Evidence quality assessment tool
- Critical Appraisal Skills Programme (CASP) systematic review checklist
- Systematic reviews and meta-analyses checklist
To appraise additional study designs, you can also use:
- Quality assessment tool for quantitative studies;
- Critical review form – qualitative studies;
- Appraisal of guidelines for research and evaluation (AGREE) II Instrument for practice guidelines;
- Or see the Critical Appraisal Skills Programme (CASP) or Scottish Intercollegiate Guidelines Network (SIGN) for tools to appraise other study designs.
The next step in the NCCMT process involves identifying the key study outcomes that you can use to inform your service delivery. In other words, what exactly did the study find and what service delivery recommendations can you get from the results?
If the research you’re considering is a systematic review, you can use the Data extraction tool for systematic reviews to organize the key points you’ve identified into a table.
You’ve identified recommendations and key messages coming from reliable evidence, but are these recommendations suited to the environment you work in? Can a program that’s been successfully delivered and evaluated in one context work in yours? These tools will help you assess if the research evidence you found can be applied or adapted in your own service delivery.
- Assessing applicability and transferability of evidence – An assessment tool that outlines questions you should consider when determining if an intervention will work in your community.
- Rapid review report structure – Describes how to present evidence found through a rapid review. The report structure is geared toward effectively communicating research implications and programming recommendations to decision makers.
All your hard work has paid off: you now have research evidence to inform a programming decision, whether that’s starting a new program, modifying an existing one, eliminating a program, or making a policy change. The Knowledge Translation Planning Template is a guide you can use to identify and work through the resources and steps needed when you implement or adapt evidence to your local context.
The final step in the NCCMT process for evidence-informed decision making is evaluating the implementation and outcomes of your programming decision. The Knowledge Translation Planning Template includes guiding questions to help you consider the evaluation step of your program implementation.
If your research involved a rapid review, the Manager checklist can be used by decision makers to guide their reading of the review and help them assess the impact of the review on decision making.
For more information on the National Collaborating Centre for Methods and Tools’ evidence-informed decision-making process, visit the NCCMT website] for learning modules and additional tools and resources to support each decision making step.
- 1. Yost J, Dobbins M, Traynor R, et al. Tools to support evidence-informed public health decision making. BMC Public Health. 2014;14:728. Available at: http://www.biomedcentral.com/1471-2458/14/728.
- 2. Mackintosh J, Ciliska D, Tulloch K. Evidence-informed decision making in public health in action. Environmental Health Review. 2015;58(1):15–19. Available at: http://pubs.ciphi.ca/doi/full/10.5864/d2015-006.
- 3. Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model. Evidence-Based Nursing. 2009;12(4): 99–101.