.png)
Whether it be a project, programme or policy - assessing impact is essential. Why? Because it is crucial for service design, decision-making, allocating resources and accountability.
It may sound complex, but conducting an impact evaluation assists you in documenting progress and providing evidence to make future improvements. It is a versatile tool that can be used during project (formative) or post-project (summative) to give valuable insights to those involved.
For example: for not for profit organisations and charities, it is vital to understand and prove the difference they make. By tracking and showcasing their achievements, they can sustain interest in their supporters and funders. This can be done by implementing an impact evaluation.
With this type of evaluation, it is important to break it down into digestible steps. This way, accuracy is ensured and nothing is missed. In this blog, we will talk you through our top 5 tips for conducting an impact evaluation.
What Is an Impact Evaluation?
Impact evaluations are designed to answer the questions: ‘What was the effect of an intervention?’ and 'What worked, for whom, and in what context?'. An intervention can be anything from a small project to a large programme.
The aim of the evaluation is to cover all bases. This can include positive and negative, intended and unintended, direct and indirect impacts.
Impact evaluation is usually undertaken to serve one of the following purposes:
Advocacy e.g. to show the value of an investment
Allocation e.g. to inform how spending will be/has been allocated
Analysis e.g. to learn what is working to inform continuous improvement, or what isn’t working to determine change
Accountability e.g. to ensure effective risk management
How to Conduct an Impact Evaluation
Doing your own impact evaluation may seem a bit daunting. But, there are ways of condensing key steps into a more manageable, memorable process.
It is important to note that there is not one right way to conduct an impact evaluation. What is needed is a combination of methods and designs that are proportionate and appropriate to the intervention. When choosing these, the following needs to be taken into account:
The available (and unavailable) resources
The nature of what is being evaluated
The intended use of the evaluation
Below is a list of steps that can be applied to most interventions and intentions.
Step 1: Prepare for the Impact Evaluation
Before you begin an impact evaluation, you must first determine what an intervention seeks to accomplish. The more concrete the objective, the easier it is to track progress and carry out an evaluation.
For example, if you are a social enterprise, you may have an objective like: ‘By 2025, secure employment for 2,000 disabled people aged between 18-15 in Northern Ireland’.
You may have more than one objective, but just remember to set realistic goals that are challenging, yet within an achievable reach.
You can also prepare by thinking about how you will monitor and obtain information. Every intervention should have a monitoring system in place before starting an impact evaluation. A monitoring system requires defined indicators as a way to track ongoing results and milestones. This way, you are better prepared to gauge more definitive insights from your efforts.
Step 2: Set a Timeline
In most cases, the time set for an impact evaluation depends on the timeframe allocated for the entire intervention.
One important aspect is knowing when the results are needed. If you have clear deadlines - e.g. if they are needed to inform decisions about scale-ups or policy reforms - this provides better scope to plan for these milestones and choose the most beneficial methods.
Essentially, the longer lead time for impact evaluations, the better. This is because when new interventions are first set up, they usually take several months to become fully operational. Preparation for the impact evaluation can be carried out during this time. This allows the evaluation to be ready by the time the programme is about to start.
Step 3: Allocate the Resources
Impact evaluations require skills and expertise. Usually, this calls for an in-house team or an outsourced specialist.
At S3 solutions, we support organisations to develop evaluation frameworks, ensuring that systems, structures, tools and resources are in place from the outset - assisting effective evaluation.
For impact evaluations, we find that working side-by-side with the client creates a more robust approach. For example, we can assist you in formulating your objectives, prepping a realistic timeline, choosing the right methodology, planning data collection and carrying out an analysis.
Step 4: Do Your Research
As impact evaluation is like ‘an operation within an operation’, you must consider it as an additional strand to your intervention.
What's more, is you cannot measure impact without data. To get this, you need to do your research.
Usually, research involves a survey and complementary qualitative data collection activities. These can comprise one-to-one interviews and/or focus groups.
In the survey, you can ask key questions relating to your objectives. Think about what questions will cover all possible impacts. You can also choose to use validated surveys which will enhance the reliability of your results. Qualitative research will help you identify trends and themes in your answers. At S3 Solutions, we believe that qualitative approaches are a crucial way to collect powerful and impactful stories, bringing a human side to survey or quantitative data.
Once the data has been collected, it can then be analysed.
Step 5: Evaluate Results
For this stage, people often hire a consultant to perform the data analysis and prepare the final report to ensure accuracy.
The final report will repeat much of the information presented from the survey, yet will add detailed information about what the results indicate.
Many impact evaluations use the following standard criteria to measure results:
Relevance e.g. the extent to which the objectives of an intervention are consistent with recipients’ requirements
Effectiveness e.g, the extent to which the intervention’s objectives were/expected to be achieved
Efficiency e.g. a measure of how resources/inputs are converted into results
Impact e.g. positive and negative effects produced, whether directly or indirectly, intended or unintended
Sustainability e.g. the continuation of benefits from the intervention after major development assistance has ceased
From this, the results can be disseminated. You can do this by posting on your website, hosting a seminar or conference, or publishing a paper.
At S3 Solutions, we are a professional team of experienced researchers and social impact analysts, we love helping organisations tell the story of their work through highly visual and impactful reports. It is also crucial that organisations then take the time to share their results with stakeholders, the use of online video conferencing is now a really cost effective and powerful way to reach large audiences to communicate your impact.
If you are interested in conducting an impact evaluation, please feel free to contact our team today.