Tuesday 10 March 2026 | 13:30-14:30
Stream: Science
Session format: Presentation

Part 1 - How to weigh an elephant: the art and science of evaluating large scale improvement programmes through a realist lens

This interactive session will provide intermediate or advanced level improvement practitioners with insights into how context-component-mechanisms-outcomes frameworks (a realist approach) can ensure rigour in the design and evaluation of large-scale improvement programmes. Delegates will hear about how this approach was applied by HSCQI, the health and social care hub for Northern Ireland, in the evaluation of the Timely Access to Safe Care Scale and Spread Programme, which aimed to bring about system-wide improvement by reducing delays and improving equity of access to care across a range of services and pathways. Participants will learn about how the rigorous evaluation of the programme generated a set of practical recommendations for large-scale improvement programmes. They will have an opportunity to reflect on the learning from this study and its implications for the design of large-scale improvement programmes, and how they could utilise a similar, robust approach to evaluation in their own contexts.

After this session, participants will be able to:

• Understand the use of logic modelling in the design and evaluation of large-scale improvement programmes.
• Describe the application of context-component-mechanism-outcomes frameworks (a realist approach) to the evaluation of large-scale improvement programmes.
• Recognise the potential benefits of a theoretically-informed, practical approach to the evaluation of their improvement work

Anita Rowe HSCQI; Northern Ireland

Part 2 - Current state: how is implementation science being used to support QI evaluation?

Evaluation is a critical component of quality improvement (QI) initiatives, yet existing efforts show heterogeneity in approach and rigor. Many emphasize outcome measurement or process fidelity, but few assess the dynamic, adaptive, and context-dependent nature of QI work. Several established Theories, Models, and Frameworks (TMFs) from implementation science (IS) offer valuable components for assisting in evaluation of QI projects. The way IS frameworks are being used to evaluate QI efforts has not been characterized. As Improvement Scientists and practitioners seek to evaluate causal pathways, learning mechanisms, and adaptation processes within complex interventions, understanding the role of IS TMFs in QI evaluation studies becomes increasingly important. This session presents the results from a scoping review mapping the IS TMFs guiding evaluation and causal explanation of QI initiatives. The findings help to identify design elements, methodological strategies, and practical features that can be of value to the Improvement Science community.

After this session, participants will be able to:

• Describe the current landscape of how Implementation Science Theories, Models, and Frameworks (TMFs) are used to evaluate Quality Improvement (QI) initiatives.
• Differentiate key design elements and methodological strategies that IS TMFs contribute to QI evaluation.
• Identify practical applications and opportunities for using IS TMFs to enhance the rigor and relevance of QI evaluation in their own work.

Pierre Barker Institute for Healthcare Improvement (IHI); USA

Part 3 - Yes, the world does need another survey: best practice in design, administration and analysis

Self-report surveys are pervasive in healthcare and can yield powerful insights for improvers. However, many suffer from bad design and poor response rates, creating confusion among respondents and results that can’t be trusted. In this interactive session, we’ll address common pitfalls and explore some lesser-known design techniques that are helpful for evaluating the impact of quality improvement interventions. We’ll also review the benefits and risks of using generative artificial intelligence to write surveys. Tips for generating high response rates like incentives, marketing, leadership support, etc. will also be included. Lastly, our session will cover approaches for analysing and utilizing survey results, how they can augment other data sources, articulate impact, and be used to drive change. Whether you’re the seasoned psychometric expert or the casual user, our session should provide you with some helpful tips to best use surveys in your improvement work.

After this session, participants will be able to:

• Identify and prevent survey design flaws that lead to biased and confusing results.
• Utilise retrospective survey item designs to evaluate the impact of quality improvement efforts.
• Describe the pros and cons of using generative AI to write survey items.
  

Jonathan Burlison Improvement Science, St Jude Children's Research Hospital; USA
Amar Shah NHS England; UK