Puppet’s 2020 State Of DevOps Report: Lack of tests is the top DevOps challenge
I have long been an admirer of the Puppet Labs “ State Of DevOps Report “ and it is required annual reading as both a real-world benchmark of DevOps adoption, and for its cogent analysis of inhibitors and accelerators that correlate with DevOps maturity.
The 2020 report came out right before the holidays and has some very interesting data and analysis on the crucial role that testing provides in accelerating DevOps adoption.
There are two headline results from the report:
- Incomplete test coverage is the #1 DevOps challenge (page 44)
- Ability to test without an integrated environment (i.e., perform unit tests) correlates with high DevOps performance (page 45)
Incomplete test coverage (AKA not enough useful tests)
The report notes that “Authoring tests is difficult: You have to deeply understand the work the service does” and goes on to note:
“Teams often do a bit of testing, such as unit tests or a set of early integration tests, but beyond that, they don’t have behavior defined well enough to write comprehensive tests for it.”
Understanding code behavior is a crucial challenge for human test-writing and a barrier we see in our work with customers. It’s one of the reasons that Diffblue Cover writes a test baseline that describes the current behavior of the program (for Java code, currently). Just looking at the first suite of tests that Cover writes can give developers new information about code behavior-”I didn’t realize it worked that way” moments. It’s also the basis for Cover’s approach of using that test baseline as a way to highlight code behavior changes in new Pull Requests. Then, the test baseline can be automatically updated by Cover as the code base evolves.
Unit testing correlates with high DevOps performance
Unit testing is a relatively new discipline in the software world, and there’s still debate about its utility. But the 2015 Puppet Labs State of DevOps Report was the first to correlate improved DevOps performance with a focus on testing isolated software units. High DevOps performers deployed 30x more often, had 200x shorter deployment lead time, 48x lower Mean-time-to-recovery (MTTR) and 3x better change success rate.
That doesn’t mean unit testing is the only contributor to those metric improvements, because refactoring to decouple systems from each other can also result in lower communication overhead and less waiting on other teams to make a change.
Architectural changes can increase isolation, but many developers don’t have the luxury of refactoring and creating new microservices to improve test coverage. And that’s why mocking in Java is so important for unit testing.
In Diffblue Cover, we do as much automated mocking as possible to isolate the code under test. We offer our best mocking in Spring Java / Spring Boot because it has built-in support right in the framework, and canonical ways to mock. Cover will automatically generate the appropriate “when… then” mocks for external dependencies, and we also mock calls into Spring Controllers and similar components so there’s no need to spin up a web server to feed requests into the code. For non-Spring Java, we also offer mocking of static methods.
Summary
Customers have been telling Diffblue that one of the key reasons they want to improve their unit testing is to accelerate DevOps adoption and Shift Left. It is great validation of that to see a broad sample of DevOps adopters in the Puppet Lab report specifically identify that challenge, across all sectors. Diffblue Cover is designed to address this problem.
There are a few ways you can try Cover yourself and see how having the unit tests you need can help your organization reach your DevOps goals.
Originally published at https://www.diffblue.com on January 15, 2021.
