Be Careful How You Use Metrics

People will do whatever it takes to meet the targets you set, even if this is not the right thing to do for your customers.

Most of us who work in IT understand the importance of using metrics to ensure we deliver good service, and I have written many blogs about defining them, for example:

• Defining metrics for change management
• Defining metrics for problem management
• Defining metrics for incident management

Each of these blogs explains the importance of defining critical success factors (CSFs), and then defining key performance indicators (KPIs) to support them, and each blog gives examples of CSFs and KPIs that might be suitable for the specific topic.

But even if you are confident you have good CSFs and KPIs in place, things can go wrong. In this blog I am going to point out some of them, and make some suggestions for ensuring you use your CSFs and KPIs to help you deliver great services.

What Makes a Good KPI?

When you are defining KPIs it’s important to remember the meaning of the three words:

• Key. The KPI must measure something important. Don’t be tempted to define hundreds of KPIs – a small number is much more effective. I recently saw this summarized in the great phrase: There’s a reason the dashboard in your car doesn’t have 20 gauges. You need a small number of carefully selected targets to help you maintain a focus on the things that really matter the most right now.
• Performance. The KPI must measure the performance of something that matters. Each KPI should clearly underpin one or more CSFs. It should be obvious what an increase or decrease in the KPI value means, and what value is a good target for the KPI.
• Indicator. This is probably the one that people most often get wrong. A KPI doesn’t prove that you did a good job, it simply indicates what you might have done. Never user KPIs to prove to your customers that you did a good job, the customers are quite capable of making up their own minds!

I often read that every KPI must be SMART:

• Specific: The KPI is clear and unambiguous
• Measurable: You can quantify or track the value of the metric for this KPI
• Achievable: It’s possible for the KPI to be reliably and regularly achieved
• Relevant: Achieving the KPI is relevant to the CSF it supports
• Time-based: The KPI includes the timescale over which it should be measured and reported

This is true, but it’s not enough. Sometimes a service provider defines a small number of SMART KPIs, all of them underpinning important CSFs, and things still go horribly wrong.

When this happens, it’s often because of the way that the KPIs have been used in practice. People forget that the KPI is only an indicator. Then they work to achieve the KPI, rather than to deliver great services to customers. You end up in a situation where every KPI has been met, but customers are still not happy. This leads in turn to unhappy working relationships, and if issues are not resolved, customers will look for services elsewhere.

Don’t make the mistake of believing that this is something that doesn’t happen all that often. In fact, it happens so often that it has a name, the “watermelon SLA.” Like a watermelon, everything looks green on the outside, but if you examine it closely you see that it is red all the way through.

If you want to make sure that your staff meet KPIs, tell them that their next pay raise depends on it. You’ll find that the KPIs do get met, but maybe not in ways that you actually want.

A very common example is a service desk KPI that specifies the number of calls per agent per day, or the average length of time to close a call. People will do whatever is needed to meet the targets you have set, even if it turns out to mean delivering terrible service to your customers.

One obvious way to try and manage this type of situation is to define tension KPIs. When you know that the work involves two or more conflicting requirements, you identify, and set targets, for all of them. For example, you can set targets for average time to close a call, and for the percentage of calls that are reopened. If agents reduce the average time by doing a poor job, then lots of calls will be reopened, which encourages them to achieve a balance.

Tension KPIs can help, but this is still not enough, because it’s impossible to predict all the complex interactions between KPIs in advance. People are clever, and they will always find a way to meet the targets you give them.

So How Should You Use KPIs Then?

Please don’t read this blog and then tell your friends that “Stuart Rance says we shouldn’t use KPIs.” I certainly think that you shouldn’t MISuse KPIs by turning them into targets for judging and rewarding or penalizing people. But this doesn’t, by any means, imply that KPIs have no value. Quite the contrary.

There are two main things that I think you should use KPIs for:

Identifying Trends

KPIs are essential for identifying trends. If you consistently measure KPIs and plot the resulting numbers, then you can see how things are changing over time. For example, you may see that incidents involving a particular service are taking longer and longer to solve, or that a new service desk agent is gradually increasing the number of calls they manage each day.

You can use this data to help you understand your environment and plan improvements. It can even provide the justification you need when planning investments such as taking on new staff, improving the quality of your self-help, or providing additional training.

Setting Thresholds for Taking Action

If you have an SLA that says you will resolve a category of incidents in 72 hours, then there’s little value in measuring and reporting results after such incidents have already happened. The only thing you’ll discover is whether or not you met the agreed target. It’s much better to trigger an escalation after 48 hours, to alert people to the fact that there is now just 24 hours left.  Hopefully, this will provide enough time to intervene and effect a resolution before the target has been missed.

I have seen this done very effectively for quarterly availability targets. When a service hit a threshold of 50% of the allowed downtime for the quarter, the service provider put in place a whole series of measures to help minimize downtime for the rest of the quarter, including extra on-call staff, more rigorous assessment of changes, etc. This enabled them to consistently meet their availability targets while keeping the cost and effort down for services that were performing well.

It’s All About People

Like everything else involved in great IT service management (ITSM), good use of metrics is all about people and relationships. A great set of KPIs can be a powerful tool to help you understand what is happening, and where you need to improve, but the KPIs are not your targets, they are just your indicators.

This isn’t just an issue for ITSM – Goodhart’s law says something very similar about economics: “When a measure becomes a target, it ceases to be a good measure.”

If you put the effort into building excellent relationships with your customers and your staff, they will tell you what you need to know so you can deliver great services. You can then use your KPIs to do what they are best at: helping you to identify trends over time and helping you to set thresholds for taking action when it’s needed.