Working together: Measuring the effectiveness of government communication using econometrics
This blog is part of our ‘Working Together’ series for Analysis in Government (AiG) Month 2022. Throughout the month we will be sharing blogs from colleagues across government to highlight the power of working together.
Econometrics is a form of statistical analysis that aims to measure how multiple factors contribute to the change in a single Key Performance Indicator (KPI). It is a useful technique that can be used in the world of communications to assess the effectiveness of advertising activity. This is commonly known as Marketing Mix Modelling (MMM) in the private sector where the main aim is to forecast and optimise future campaigns to maximise effectiveness and efficiency.
Econometrics as an evaluation method has been widely used in government communication activity, especially in recruitment marketing campaigns. But it has mostly been done by using the services of an external supplier. This can often make it feel quite mysterious for people who are not familiar with this type of analysis. There are other difficulties with using a supplier from the private sector. For example, commercial measures relating to revenue and sales are common metrics in the private sector. These measures are not always appropriate for government campaigns.
Outlining the project
Because of the importance of the Coronavirus Vaccine Campaign, senior stakeholders were interested in using econometrics techniques to estimate how the campaign affected the public’s willingness to get vaccinated. From the beginning, it was clear that a more bespoke solution was needed to get clear results for such an important campaign.
The GCS Media and Marketing Data (MMD) team in the Cabinet Office worked closely with both the Campaign and Insight & Evaluation (I&E) teams to develop an in-house solution. We used our collective data specialties in data strategy, analysis, and science.
The project was broken down into four main steps:
- Discovery and scoping
- Data collection
- Modelling
- Insights presentation
All teams had worked with external suppliers on similar analysis before and knew that cross-team collaboration was important. By working together we were able to create trust in the analysis and insights. We did this by increasing visibility and knowledge of the modelling throughout the project.
Scoping the project and collecting data
Scoping the project was straightforward because the main aim was to estimate the developing effect of the campaign on people’s intention to be vaccinated, and the number of people being vaccinated.
Data collection is often the most complex part of econometrics projects. But it was supported by all teams to make sure that the variables needed to answer the main question were readily available.
Modelling and presenting insights
Before we could start modelling, the MMD team did some research on potential MMM solutions. We decided to use Meta’s automated open-source code Robyn because of the wide variety of features that it offers. Robyn is a package that uses the R programming language- this means that it is freely available. Furthermore, it is privacy focused as no data is shared with Meta.
During the modelling stage, both the Campaign and I&E team were updated on a weekly basis by the MMD. It was interesting to see that working together on this process increased engagement and confidence in the analysis. This was clear when it came to drafting the insights presentation together. The challenge was to make sure that the results were presented in an easily accessible format that clearly showed the effect of the campaign. The presentation also had to show the assumptions and limitations of the methodology.
The benefits of working together
This project provides a useful example of how multidisciplinary teams can work together to share and distribute their collective knowledge on an analytical piece of work. In fact, the sharing and distribution of knowledge is something Tidd & Bessant (2014) have identified as one specific task of managing knowledge. They explain that there are proven mechanisms to support knowledge transfer between teams or communities. These include:
- translator – expresses the interest of one community from the perspective of another
- broker – participates in different communities rather than mediating between them
- boundary object or practice – something that is of interest to two or more communities, where each has a stake and brings a different perspective
For this project, the ‘boundary object or practice’ mechanism was used with each person sharing their professional knowledge. Each person was also able to learn through experimentation and greater understanding of econometrics as an evaluation technique.
When working on a project with other teams, we would recommend that multi-disciplinary teams have a shared stake and interest in the project they are working on. This will help to encourage the transfer of knowledge between people and teams.
Noel McGarrell