If you ask ten engineers how they measure the success of their DevOps strategy you are likely to get ten

different answers. Success is measured differently by different

people, so it’s not unusual to observe these different

perspectives. What is the DevOps value proposition? This blog will

explore a number of things to consider when you try to answer the

question “How do YOU measure the success of DevOps?”

What is the goal of DevOps?

The most important

question you can ask yourself when it comes to measuring the

success of DevOps is what are your goals, and most importantly,

what are the goals of your company? After all, isn’t it the company

that needs to benefit the most from DevOps? Without goals, you have

no direction and therefore will never know if you achieved

anything. The goal for DevOps is simple: Economic Benefit. If the

outcome of implementing DevOps doesn’t increase cash flow, raise

share price, reduce costs, increase profitability, or improve some

other financial measurement then you need to rethink your plans.

Ultimately, you want to identify some metrics or KPIs that support

the attainment of economic benefit.

DevOps Metrics and KPIs

Gaining an economic

benefit is not a simple chore, and anyone who has started or runs a

business knows all too well that achieving profitability, or some

other measure of economic benefit, can be a daunting task. To

measure success, you need to establish some baseline metrics and

KPIs before you start implementing any changes. Establishing

baseline metrics and KPIs for your people, your processes, and your

technology is the only way you can empirically state that you

created an economic benefit for your company. Let’s look at the

metrics and the KPIs that can be established in each of these

areas. People:

A company is only as good as its people and only as strong as its

weakest link. Therefore, when it comes to DevOps, you need to

establish how to maximize the throughput of your staff.  The

tricky part here is to develop metrics that everyone in the DevOps

team can contribute. You can’t focus on one group in the DevOps

toolchain because you will create bottlenecks that will frustrate

the rest of the team.

Something easy to measure

across all members of the DevOps team is activity. Where are all

the hours that are being logged, or not logged, going? If you ask

most people what they do all day, they really won’t be able to give

you an accurate estimate. Therefore, it makes sense to figure out

how people are spending their time. There are countless tools out

there that can track employee activity. Once you know how

everyone’s time is being spent, then that can become the baseline

for improvement. You may find out that 64% of the development’s

team time is spent fixing bugs, but the industry average is 32%.

Now you know you need to improve that number by 50%. There is a

good chance that instituting this practice can create a Hawthrone Effect. People will change their behavior

and become more productive simply because they are being

observed.

One problem to avoid is the “Big Brother” syndrome.

Micro-managing everyone is not the goal of measuring activity, and

that needs to be communicated clearly. Instead, this can be turned

into an incentive program. Since everyone is being measured, you

can institute rewards for employees that hit certain milestones or

achieve personal bests. These successes can be shared with the rest

of the team. It will also help identify areas that need attention

through additional training, mentorship, and coaching. Aside from

measuring activity, you can also monitor attendance, turnover,

response time, and employee satisfaction. These will all assist in

increasing the DevOps team’s productivity

Velocity KPIs

1. MLT (Mean Lead Time)

   How long does it take for a bit of code to get built, tested and

   deployed?

2. DCR (Daily Change Rate)

   Number of changes getting committed to mainline and tested per

   day.

3. MTTE (Mean Time To Environment)

   How much time it takes developers/testers to bring up a testing

   environment for verifying each delivered change.

4. MTTD (Mean Time to Detect)

   How much time passes since the original commit of code until the

   bug it introduces gets detected.

5. MTTR (Mean Time To Resolve) How much time it takes to resolve an issue after it’s detected

6. MTTA (MeanTime To Approve)

   How much time it takes to approve and verify a release. (Measured

   from the moment all released content has been delivered and until

   the release has passed all the defined test and verification

   cycles)

 Quality KPIs

1. BFR (Build Failure Rate) Percentage of failed builds

   

2. DFR (Deployment Failure Rate) Percentage of failed deployments

3. IRFR (Infrastructure-Related Failure

   Rate)

   Percentage of build/deployment failures related to infrastructure

   issues

4. RWR (Rework Rate) Percentage of tickets being reopened

5. ADR (Automated Detection Rate)

   Percentage of defects being detected by automated testing

   cycles

6. UWR (Unplanned Work Rate) Percentage of unplanned issues

By measuring these and other KPIs, you will determine how well

you are performing and can you take action on how to make

improvements.

1.  Availability (excluding planned downtime) - Percentage of

   actual uptime (in hours) of equipment relative to the total numbers

   of planned uptime (in hours).

2. Percentage of outage due to changes (planned unavailability) -

   Percentage of outage (unavailability) due to the implementation of

   planned changes, relative to the service hours.

3. Percentage of outage due to incidents (unplanned

   unavailability) - Percentage of outage (unavailability) due to

   incidents in the IT environment, relative to the service

   hours.

4. Percentage of unplanned outage/unavailability due to changes -

   Percentage of unplanned outage (unavailability) due to the

   implementation of changes into the infrastructure. Unplanned means

   that the outage (or part of the outage) was not planned before

   implementation of the change.

5. Percentage of Service Desk Availability - Calculation of the Service Desk Availability over the Reporting Period.

6. Percentage of availability SLAs met - Percentage of availability Service Level Agreements (SLAs) met.

7. Percentage of (critical) infrastructure components with

   automated availability monitoring - Percentage of (critical)

   infrastructure components with automated availability

   monitoring.

8. Percentage of critical business processes not covered by a

   defined service availability plan - Percentage of critical business

   processes not covered by a defined service availability plan.

9. Critical-time failures - Number of failures of IT services

   during so-called critical times. The critical time is the time that

   a service /must/ be available, for example for financial systems

   during the closing of the books (at the end of the month, or end of

   the quarter).

10. Critical-time outage - Total outage from critical time failures

    in IT services. The critical time is the time that a service /must/

    be available, for example for financial systems during the closing

    of the books (at the end of the month, or end of the quarter).

11. Number of business disruptions caused by problems - Number of business disruptions caused by (operational) problems.

Business Value of DevOps

When we began this

discussion about the business benefit of DevOps we learned that the

ultimate objective is to derive economic benefit. Since that is the

case, we ultimately want to prove that all our metrics and KPIs

enhance the economic benefit in some manner. By adhering to DevOps

best practices, you win by building it better, faster, and cheaper

than your competition. But as you strive for these benefits there

are many other benefits that you gain along the way. Some technical benefits include:

• The ability to fix problems faster

• Reducing the complexity of your environment making everything easier to manage

• Ability to deliver software on a continuous basis.

The entire team wins due to the cultural benefits such as:

• Enhanced engagement between your employees

• Employees are more productive and happier because of increase engagement

• Employees get better at their job and develop professionally which elevates them in their career.

Overall the business wins because DevOps

principals deliver:

• New features faster than ever before

• Improved collaboration and communication between employees and teams

• Operating environments that are more stable

• An intense focus on innovation because everyone is spending less time fixing problems.

Let’s focus on the business value of DevOps a little more

deeply. When you organize your business like an “IT Factory”

(Incidentally, we

have a blog post coming out about this in the future so subscribing

to our blog posts will ensure it’s delivered directly to your

mailbox) you are

essentially stating that you are going to automate everything.

Isn’t that why factories exist in the first place? In this IT

Factory, the objective is to turn out product. The faster you do

it, the faster you can obtain revenue. This concept is known as the

velocity of revenue.

When Apple releases a new phone, people line up to buy the new

phone, even though their old one is perfectly fine. Apple enjoys a

revenue spike each time they release hardware. You read about it

every year. Software is no different. On each release of new

software, you can measure the revenue spike. If you release

software once per year, you will get an annual revenue spike. By

employing DevOps automation tools, among other things, you can

release software each month, or more frequently, and generate

smaller, but more frequent revenue spikes. These smaller spikes

accumulate to more than an annual revenue spike. Why? Between

software releases your competition is trying to muscle in on your

territory. The longer the delay, the more time you give your

customers to get dissatisfied with your product and find a product

with the features they want now. Conversely, if your competitor

just released a highly sought after feature you can’t release for

another four months, you just lost four months of potential

sales.

The DevOps value formula is simple: Velocity of revenue

increases as software deployment frequency increases. Of course,

quality can never be compromised, or all is lost. If you release

software continuously like Amazon, Google, and Netflix you

essentially eliminate all these revenue spikes altogether because

the best version of your software is always available. Now you are

streaming revenue daily, and the competition will have a hard time

competing with an IT factory as efficient as yours.

In summary, the goal of DevOps is to create economic benefit. To

understand economic benefit , you need to take a baseline

measurement on every key metric that contributes to economic

benefit. Improve these metrics by making everyone accountable and

incentivizing them for success. Measure , Improve, Measure,

Improve. Now you have a finely tuned IT Factory churning our daily

software releases which maximize your velocity of revenue. If you

don’t know how to start FP Complete would be

happy to show you how. We have helped many companies on their

DevOps journey by helping them build their IT Factory, showing them

how to run it themselves, and being there when they need us. Our

DevOps assessment is a quick way

for us to gauge what you need to get started.

If you liked this post you may also like:

• Download our guide - "5 Roadblocks to IT Succcess and how to avoid them"

• Multifaceted testing - A DevOps Best Practice

• Understanding DevOps Roles and Responsibilities