elevenM’s Jonathan Topham unpacks the secrets behind developing effective cyber security metrics, including how the use of KPIs and KRIs in combination can shine a light on an organisation’s cyber security posture.
In my time as a CISO, few things imprinted themselves on my consciousness more than how critical cyber security metrics are to managing a cyber security program.
At their best, metrics can help an organisation demonstrate the efficacy of their cyber security strategy, controls, and processes. They can even give early warning of issues that haven’t yet occurred. Yet for many organisations, cyber security metrics become just another reporting overhead, often resulting in a muddled confusing mess that does little to inform anyone of anything useful.
So, why do so many organisations get metrics so wrong and what can they do about it?
Activity ≠ Value
Many security and IT teams, particularly those starting out, rely on metrics linked to the volume of an activity. For example, “x number of phishing e-mails blocked at the gateway this week”, or “y number of tickets closed”.
There are many reasons that teams gravitate towards these sorts of measurements. For one, they are easily available, as most security tools have some sort of dashboard that displays volumetric data. Another reason is that teams starting out are often trying to justify investment in tooling and people. Showing stakeholders that those resources are busy or detecting lots of things means that money is being well spent. Right?
The issue here is that whilst reflecting a level of activity, these metrics don’t give you an idea of the state of the environment. It’s like a doctor trying to measure the health of a patient by the number of tablets they’ve been taking.
Don’t get me wrong, volumetrics have uses. At a tactical level, that dashboard which shows how many messages have been blocked in a specific time frame might be useful to tell if there is some form of campaign targeting your organisation and if the security team needs to intervene. However, for stakeholders you see once a month or quarter, that data probably isn’t going to be that useful or insightful — unless you can link it to the context or the value that it has to the organisation. But I’m getting ahead of myself.
Key Performance Indicators
For those of us in the trenches of corporate life, KPIs have long been the bread and butter of performance management. These metrics — whether tracking the percentage of systems with up-to-date antivirus software, the mean time to detect (MTTD) and mean time to respond (MTTR) to security incidents or the rate users are adopting a security control — offer a quantifiable gauge of how well an organisation is faring against its security objectives.
The key word there is “objectives”. KPIs don’t work unless there is a pre-agreed objective against which performance can be measured. Organisations often struggle with setting objectives for security because they can’t articulate the linkage between a metric and a strategic or other goal. If your organisation has a cyber security strategy, then some of these objectives might be articulated in there. If not, then don’t despair, there are resources out there with standardised metrics and suggested targets to help you get started, such as the excellent CIS Critical Security Controls Measures and Metrics.
Because nothing in security is as simple as cribbing the solution from a public resource, some words of caution: Just because you can measure almost anything, doesn’t mean you should report it.
When building KPIs for reporting it’s often best to start with a small set that cover the most critical areas of your security program, such as your key controls and themes in your strategy. There is nothing stopping you from adding new KPIs later and you can always monitor a larger set behind the scenes and introduce them to underline a point if needed. However, keep in mind that once you’ve started sharing a KPI with senior stakeholders, such as boards, it’s very hard to take it off the table.
Another thing to consider when selecting and presenting KPIs is grouping them around objectives. For example, if your security program covers “uplift” (i.e. adding new controls) and “run” (how well an existing control is operating) you might find it useful to organise your KPIs to reflect their relevance to those objectives. This can help you keep your narrative to different groups of stakeholders clear and reduce confusion as to what a KPI is saying.
Key Risk Indicators
So, you’ve spent months slaving away over a hot spreadsheet and got yourself a set of KPIs that look really cool on a PowerPoint deck and everyone is happy. You’re done, right?
Unfortunately, no.
KPIs have inherent limitations. They tell you how well you are tracking against a known objective — but they are just a snapshot of performance. A snapshot that might not show broader risks just out of frame. That’s where Key Risk Indicators (KRIs) come in.
KRIs are the unsung heroes of risk management and come in two flavours, Leading or Lag.
Leading KRIs provide early warnings that something is amiss and that things might be about to get choppy. Lag KRIs, as the name suggests, look backwards at what has happened, and help show the impact of risks.
At the beginning of this post, I mentioned that simple volumetric measures have their uses, but they need to be applied with context. KRIs are a prime example of where this is the case — detecting changes in volumes can be useful in identifying specific problems with an organisation’s controls or help in the early detection of new threats.
For example, monitoring the number of systems which don’t have up-to-date patches can be a good leading KRI for a threat actor being able to exploit environment. While a good lag KRI might be tracking the total number of incidents or false positives over a set period to understand if the controls you have in place are stopping threats or are potentially misconfigured.
The KRIs you use should be dependent on your organisation’s perceived threats and risks. As with KPIs, you should also be careful not to try and track or report too many of them as you can quickly become overwhelmed in trying to track everything. You should also remember that you need time to baseline KRIs before they become useful, as analysis of a single or even a handful of data points doesn’t tell you much.
For all their usefulness KRIs are often relegated to the sidelines by KPIs. After all, success is seductive and there is a momentum to be gained by hitting targets. But as any seasoned observer of corporate life will tell you, momentum is a double-edged sword; it carries you forward, but sometimes forward leads you into unseen danger.
Teamwork makes the dream work
The key to navigating this terrain, and avoiding the pitfalls, lies in understanding how KPIs and KRIs complement each other.
When elevenM works with our clients to design cyber security management metrics, we look for ways to integrate KPIs and KRIs with each other. This way, organisations gain a comprehensive view of their cyber security posture, which allows them to make informed decisions about where to allocate resources and how to prioritise cyber security efforts.
This process can be as simple as having a monthly or quarterly review process that kicks in when there is an increase in a KRI, forcing an examination of the KPIs related to it as part of a root cause analysis. Or, it can be more complex formalised process where KRIs have thresholds with pre-agreed rates, based on risk appetite.
How does that work in practice? If your organisation is particularly worried about brute force attacks against accounts, you could track “Number of failed logins” as a KRI and “Percentage of successful logins without incidents” as a KPI. This would give you a metric that can indicate your environment might be under attack, whilst the KPI tells you how the secure access controls are performing.
Below are some further examples of how KRIs and KPIs can be paired to tell a richer story about cyber security posture:
Developing KPI and KRI pairs or sets is highly contextual to your organisation, so the relevance and usefulness of these above examples may vary.
However you choose to apply them, developing linked sets of KRIs and KPIs is a powerful tool to give you a sense of how your security program is performing.
For many organisations this is a cultural shift, where risk management is not seen as a brake on progress but as a critical part of the journey towards it. For security teams and those senior stakeholders who provide oversight, it’s about being not just reactive to risks when they materialise but proactive in integrating risk indicators into the fabric of performance management.
Conclusion
The interplay between KPIs and KRIs offers a window into the health of an organisation’s cyber security posture. It’s the point at which “the rubber meets the road”, where strategy and investment are tested against the harsh and realities of an increasingly hostile cyber landscape.
The development of good metrics, particularly the strategic use of KPIs and KRIs, is vital for maintaining strong cyber security defences.
These metrics not only help in measuring current security postures but also in planning and forecasting future cyber security needs. Organisations that effectively leverage both KPIs and KRIs can enhance their ability to protect against cyber threats, thereby safeguarding their assets, reputation, and stakeholder trust.
Contact us
If you’re interested in learning more about cyber strategies and implementing a cyber security metrics program, please contact us.