15.2 Building a Change Analytics Strategy
Key Takeaways
- A change analytics strategy is how you will capture, manage and use data to support the success of the change
- It should cover change readiness, adoption and benefits realisation, plus the effectiveness of communications, engagement and learning interventions
- Qualitative data is subjective and harder to measure, while quantitative data can be observed and measured numerically
- Leading measures are result-oriented dynamic factors that signal progress, while lagging measures are output focused and measure the outcome of the change
- In the lead-up to change readiness teams rely on qualitative data, while for adoption the emphasis shifts to quantitative metrics
A change analytics strategy refers to how you will capture, manage, and use data to support the success of the change.
Once you have a framework for your balanced scorecard, consider how you will measure change readiness, adoption, and benefits realisation. This should also cover assessing the effectiveness of your communications, engagements, and learning and development interventions.
That second sentence is significant. Change analytics is not only about whether the change is being adopted; it is also about whether the change management itself is working. Measuring the effectiveness of communication and engagement is what stops those activities being reported as an activity count.
The four evaluation methods
| Method | Definition |
|---|---|
| Qualitative | Data that is subjective and deemed harder to measure |
| Quantitative | Data you can observe and measure, numerical in nature |
| Leading | Result-orientated. These are dynamic factors that signal progress. Like qualitative measures, these are harder to quantify |
| Lagging | Output focused. They measure the outcome of the change |
The two pairs answer different questions. Qualitative versus quantitative is about the nature of the data — subjective judgement versus countable observation. Leading versus lagging is about timing — signals of progress versus measurement of the result.
Examples that fix the distinctions:
| Measure | Type |
|---|---|
| Interviews and questionnaires about how people feel about the change | Qualitative |
| Number of staff who have completed training | Quantitative |
| Proportion of teams that have run their own local adoption session | Leading (signals progress towards adoption) |
| Reduction in processing time six months after go-live | Lagging (measures the outcome) |
Interviews and questionnaires are a qualitative evaluation measure — a point the exam tests directly.
Applying measures appropriately
Quantitative measures are the most attractive method when capturing data due to their tangible nature. However, different stages of the change lifecycle will require different mixes of data types to show progress and critical mass:
- In the lead-up to change readiness, the change teams will be reliant on qualitative data. For example, do people know what the change is and what it is trying to achieve?
- For adoption, the emphasis will shift to quantitative metrics. For example, what activities have the change network held? How many people attended?
This progression is worth internalising because it contradicts a common instinct. Early in a change, when pressure for hard numbers is highest, the honest answers are qualitative — nobody is using the new system yet because it does not exist, so the only meaningful readings are about understanding, sentiment and readiness. Producing quantitative readiness metrics at that stage usually means counting activity rather than measuring readiness.
As the change moves into adoption, countable behaviour becomes available and the emphasis shifts. A change analytics strategy should plan for that shift rather than applying one measurement approach throughout.
Limitations of data analytics
CM3 names four limitations:
| Limitation | What it means in practice |
|---|---|
| Volume of data | More data than can be meaningfully analysed or acted on; signal buried in noise |
| Quality of data | Incomplete, inconsistent or inaccurate underlying data producing confident but wrong conclusions |
| Inappropriate handling of data | Poor collection, storage or analysis practice — including drawing conclusions the data cannot support |
| Data distortion | Data affected by how it is collected or by people's awareness of being measured |
Data distortion deserves particular attention in change work, because measurement itself changes behaviour. If adoption is measured by logins, people log in. If engagement is measured by survey scores and those scores affect a manager's rating, the scores improve without the engagement doing so. This is why the actionable factor and the involvement of users in metric selection matter: measures chosen with people and used to help them are distorted far less than measures imposed on them and used to judge them.
Quality of data is the most common practical problem. Change teams frequently build analytics on system data collected for another purpose entirely, and the mismatch between what the field records and what the change team believes it records is not visible in the resulting chart.
Assembling the strategy
A change analytics strategy that satisfies CM3 covers:
- What will be measured — readiness, adoption, benefits realisation, and the effectiveness of communications, engagement and learning
- How — the mix of qualitative, quantitative, leading and lagging methods, and how that mix shifts across the lifecycle
- How data will be collected — sources, including system data already available, and the cadence for each
- Who acts on it — the accountable owner for each measure and what response each signal triggers
- What the limitations are — an honest statement of what the data can and cannot show, so conclusions stay within what the evidence supports
When measuring change, using interviews and questionnaires is which type of evaluation measure?
How does CM3 say the mix of data types should change across the change lifecycle?
Which of the following is NOT one of the four limitations of data analytics CM3 identifies?