Mass Communication

Evaluation is a policy tool which is used to steer, manage and improve the activities of and investments in public sector research organisations. It is also used to change the distribution of funding among research organisations. Modes and mechanisms of evaluation have been developing over recent decades as budget holders have demanded accountability for public money spent, in common with other areas of public expenditure. In addition to simply justifying research budgets at a political level is the desire for evaluation to reveal the breadth of impacts of research, and the relative academic quality of research organisations and the research which is formative evaluation. It will then move on to discuss how it can be used to enhance program development and a conclusion will follow.

A comprehensive program evaluation ideally includes both formative and summative components. Both approaches can examine how an intervention was implemented, the barriers and facilitators to implementation, and the effects of the intervention on various outcomes. Although both components can provide feedback on the effectiveness of an intervention and offer ways to improve it, they differ in frequency, aim, and focus. Formative evaluations stress engagement with stakeholders when the intervention is being developed and as it is being implemented, to identify when it is not being delivered as planned or not having the intended effects, and to modify the intervention accordingly. The stakeholders include payers, clinicians, practice staff, patients and their caregivers, and other decision makers.

a formative evaluation focuses attention on ongoing, midstream assessments that feed information back to intervention implementers, allowing them to make real-time adaptations and refinements to ineffective aspects of an intervention. Formative feedback often leads to decisions about program development (such as whether to modify or revise the intervention), whereas summative feedback often leads to decisions about whether to ultimately continue, expand, or adopt the program (Worthen, Sanders, and Fitzpatrick, 1997). Implementing complex interventions in complex settings (such as the PCMH) is a difficult task that requires researchers and program managers to have a clear understanding of what should be implemented, how to best implement a suggested strategy, which elements may hinder or facilitate the implementation process, and why a strategy did or did not work once implemented. A formative evaluation can provide this information on an ongoing basis as the intervention is being delivered. Stetler, Legro, Smith, et al. (2006) conceptualize four components of a formative evaluation according to whether each occurs before, during, or after intervention implementation. Complete a needs assessment.

Formative evaluations focus on pre-planning for the intervention design before it is implemented, which Stetler, Legro, Smith, et al. (2006) term the developmental component. Before the intervention begins, the evaluator conducts a needs assessment about areas where the practice should focus improvements by understanding the context the practice operates in, potential barriers and facilitators to practice change, and the feasibility of implementing the intervention as initially designed. Stetler, Legro, Smith, et al. (2006) also describe three other components that can occur during or after a formative evaluation: (1) an implementation-focused analysis, (2) a progress-focused analysis, and (3) analysis of interpretive data. An implementation-focused analysis assesses discrepancies between the implementation plan and the execution of that plan. This can include assessing fidelity to the implementation strategy and the clinical intervention, understanding the nature and implications of local adaptation, identifying barriers, identifying new intervention components or refining the original strategy to optimize the potential for success, and identifying the critical details necessary to replicate the implementation strategy in other settings.

Data sources might include semistructured interviews with stakeholders, structured surveys, focus groups, direct observations through site visits, document reviews, electronic health records or charts, and management information systems. A progress-focused analysis monitors progress toward implementation and improvement goals during the intervention. Outcomes for the intervention practices are monitored on an ongoing basis. For example, audit and feedback of clinical performance data can give providers and practices data on key process and patient outcome indicators that can be used to refine the intervention during implementation. This information may also be used as positive reinforcement for high performers and as encouragement for low performers. Intervention impacts can then be estimated by comparing the outcomes of intervention practices with those of a comparison group to determine whether the intervention is having the intended effects on quality, cost, and patient and provider experience. Data sources for the progress-focused analysis typically include claims or billing data, electronic health records or charts, and structured surveys.

The main advantage of a formative approach is that it encourages mid-stream modifications to improve the intervention, rather than taking a more “hands-off” approach for the sake of research objectivity. If interim feedback can provide insights about ways to improve the intervention, this information can be used to increase the chances of implementation success and also focus resources most efficiently (Worthen, Sanders, and Fitzpatrick,1997). Although formative evaluations are useful for a variety of interventions, they are particularly useful for helping to refine wide-ranging and complex PCMH interventions. Primary care practices often implement multiple intervention components concurrently, and these components interact with the practice and external setting. Therefore, there are numerous possibilities for implementing each intervention component and for them to interact with one another. For example, suppose that a practice would like to provide after-hours care to its patients. Depending on the context, the practice could implement the intervention by establishing a nurse call line, rotating physician coverage, establishing an agreement with a local after-hours clinic, or sharing coverage with another practice.

The providers could use ongoing formative feedback to continually improve the delivery of their complex intervention. Formative evaluations present several challenges for researchers. Formative evaluations are time- and resource- intensive because they require frequent data collection, analysis, and reporting, as well as rapid refinement of the intervention strategy as new information about implementation effectiveness becomes available. Formative feedback leads to real-time refinements to the intervention, which makes this evaluation component a part of the intervention itself. Although this maximizes the chances of program success, it also raises a question as to whether the intervention would work if replicated without the formative evaluation component. To be successful, ongoing PCMH interventions may wish to include formative feedback. This feedback might come from data or feedback reports from payers about the patients’ quality and cost outcomes, information the practices collect from patients using surveys or other feedback mechanisms, and process and outcome metrics the practices collect using their own internal tracking systems.

Ultimately, formative feedback should be viewed as an integral part of the delivery of PCMH models and other complex health services interventions. This rapid refinement process also poses several methodological challenges when trying to evaluate the impact of the intervention. Outcomes can only be measured during the period when a particular variant of the intervention was implemented, which may lead to short followup periods—with corresponding smaller sample sizes—and less power to detect overall intervention effects. In addition, it is difficult to determine when a change in the intervention will emerge in outcomes. When evaluators examine the entire time frame of the intervention, estimated effects will reflect the combined effects of the different variants of the intervention over time, essentially estimating an average treatment effect rather than the effectiveness of any one version of the evolving intervention.