How Do You Measure IT Services?

All the IT service providers I’ve worked with assure me that they measure the services they provide. They use metrics and KPIs to do this, and have service-level agreements (SLAs)  with their customers, which they use to document what’s been agreed.

Unfortunately, most of the metrics and KPIs that I’ve seen only measure and report the things that service providers can control, and not the things their customers actually care about. This tends to result in reports that show service providers meeting most, if not all, of their targets, even when customers are distinctly unhappy about the service. This happens so frequently that the phenomenon even has a name: the “watermelon SLA”. When you look at a watermelon from the outside, all you see is green.  Delve a bit deeper and you discover that most of the fruit is red.  In the same way, we report that everything is meeting its targets (it’s all green), but if you delve into the customer’s experience you find that it’s not very good (it’s mostly red).

So, what can service providers do that will help them to delve that bit deeper and focus on what matters?

What’s Important to Customers?

When service providers think about any service, they need to pay attention to what’s important: the value of the service, the intended outcomes, the costs, and the risks – together referred to as VOCR. Here is a brief summary of what we mean by these terms.

• Value is a measure of the benefits that the service creates for the customer. This could be in terms of money, but it might be something like lives saved, or some other indication of the customer’s mission.
• Outcomes are the things that the customer achieves as a result of receiving the service. For example the service may help the customer to manufacturer products, or communicate with their partners, or collect payments on a web site.
• Costs are the money that the customer has to spend to achieve the outcomes they want.
• Risks are possible events that could affect the ability of the customer to achieve the desired outcomes.

In my view, if every service provider produced reports showing the VOCR of their services, then customers would have no difficulty understanding what these reports mean, and judging whether the service was good value (or not).

Of course, it’s not that easy. The trouble is that value, outcomes, costs, and risks are only partially under the control of an individual service provider. For example, consider an IT company that provides a manufacturing support service to a car manufacturer. The outcome that the customer wants might be “a car comes off the production line every 40 seconds” (my thanks to James Finister for this example), but this outcome is not completely controlled by the IT service. Even if the IT service works perfectly there could be other reasons why a car doesn’t come off the production line. So, what should you measure and report in this case?

What Should You Report to Your Customers?

Under these circumstances most IT service providers would probably report things like:

• Availability of the IT service
• Number and severity of incidents
• Average time to resolve incidents, subdivided by category

The information provided might be accurate, but it is very IT specific. It represents an IT service provider’s view of the world. But of course, this isn’t what the customer cares about. The metrics that matter to the customer are how many cars come off the production line (outcome), how much extra cost do they have due to IT failures (cost), how much lost production might they have next week (risk), and derived from these, how much profit can they make selling cars (value).

So, is it possible to create IT metrics and reports that actually show all of these things? Yes, it is. An ideal SLA could state the following:

“The desired outcome from this service is to support the production of one car every 40 seconds.”

The monthly report could then show whether this was achieved. Similarly there could be report sections on cost and risk, showing how well these have been managed for the customer.

Does this mean we stop measuring IT service availability, number and severity of incidents, incident resolution times? Obviously not. Providers can’t do their jobs well without this information. What I am saying is that these are not the key things to report to the customer. The main findings of any report to customers should be about value, outcomes, cost and risk; IT-centric issues should be reported within this context, as contributing factors, and at whatever level of detail suits the specific customer.

Here are two more examples of IT organizations that have great business-focused reports:

• One client that I worked with had a key business metric stating that they should never lose any customer data (the data represented a lot of money). It didn’t matter if the data was lost because a paper document got lost before it was scanned, or because of an IT error. The thing that mattered is that every piece of customer data must be correct and accurate at all times. The reports they produced started by summarizing the status of customer data, THEN they went on to account for any risks or issues that had occurred that month. Some of these were IT related, others were down to the various business units, but the report included everything.
• Another client was responsible for shipping parcels to end customers. Their key metrics were the percentage of parcels that arrived on time, and the percentage of parcels that were lost. As in the other examples, these things might be caused by IT failures, or they might be due to many other issues, but the IT reports focused on the business metrics, and then on how IT had contributed to these.

What about your SLAs and customer reports? Are they about IT or are they about value, outcomes, cost, and risk? Maybe it’s time to think about what your customers really care about, and create SLAs and reports that match their view of the world.