The Brief
The Renewable Heat Incentive (RHI), was a government financial incentive to promote an increase in use of renewable technology for heating solutions in both domestic and non-domestic settings, supporting the UK in reducing carbon emissions and meeting its renewable energy targets.
In December 2016, the government published proposals to reform the scheme, although originally expected to come into force in Spring 2017 – the first package of reforms – involving changes to tariff levels, domestic technology heat demand limits and the removal of tariff bands for non-domestic biomass – came into force on 20 September 2017.
To coincide with the introduction of the reforms, a new and enhanced theory-based evaluation approach was required to support evaluation of their impacts and effectiveness in bringing about the intended outcomes.
Separate evaluations were required for the domestic and non-domestic schemes to reflect differences in scope and the nature of reforms introduced in each.
Winning Moves was commissioned as part of an expert evaluation consortium to deliver the requirements, which included:
- Theory of Change development
- Analysis of RHI administrative data
- A detailed, six-monthly rolling applicant monitoring survey
- Qualitative research with applicants, installers, and wider stakeholders
- A Sustainable Markets Assessment (SMA)
- A Subsidy Cost-Effectiveness Assessment (SCEA)
- A Competition and Trade Assessment (CTA)
- A Quasi-Experimental Impact Assessment (QEIA).
Winning Moves supported the development of the approach and initial theory development before designing and delivering the applicant monitoring survey and undertaking a range of bespoke research conducted during the contract to examine the impact of individual reforms and specific topics of interest (e.g., consumer satisfaction with heat pumps 2 years on from the point of installation).
In order to draw meaningful conclusions, a large-scale online survey with a high response rate was required.
The solution
Realist evaluation principles were used to inform the theory development and enable detailed consideration of the responses by different actors in different circumstances to specific reforms.
Applications to the scheme were monitored with a six-monthly online survey sent to all applicants to see how the scheme was working, and explore the influence and impact of the reforms, including announcement effects. The online survey was supplemented by telephone interviews to boost representation of key groups in the sample and address areas of non-response.
Survey scripts used in earlier rounds of evaluation were subjected to detailed review and adaptation to enable key metrics to be retained, whilst including new questions to establish the influence of the reforms (including whether or not the reforms had encouraged application to the scheme, and if so, which renewable technologies it had encouraged) and to capture evidence of topical interest (e.g. detailed tailored questions for biomethane installation applicants to inform a dedicated report examining the impacts of the RHI and relevant reforms).
Survey work commenced with a retrospective survey to collect data for the period between the end of the previous evaluation work and commencement of the new evaluation contract before reverting to a six-monthly rolling survey of successful applicants. Data collected in the main survey were blended with data collected in earlier rounds of the evaluation work and weighted to provide a single dataset containing all of the information gathered during the life of the detailed applicant monitoring surveys and allow analysis of the full time series for metrics captured since the outset of the RHI.
As a long-term multi-year contract, a budget was set aside at the outset to be used for specific evaluation requirements and evidence needs identified during the evaluation. This led to a range of supplementary work being undertaken during the contract to inform the overall evaluation. Work undertaken by Winning Moves included a ‘non-applicant survey’, undertaken part way through the evaluation contract, to understand the reasons re-contactable individuals expressing interest in heat pumps and/or the RHI via other services funded by central government did not proceed with an application, where applicable.
The outcome
Winning Moves was successful in securing good and consistent response rates to the detailed applicant monitoring surveys throughout the course of the contract, with an overall response rate for the domestic RHI programme in excess of 25%.
Capturing and scrutinising key metrics, the surveys provided an evidence-rich time series to support the assessment of individual reforms and inform future heat decarbonisation policy.
The final evaluation reports have been published. The domestic report can be found here and the non-domestic report can be found here.