Guest author Chad Schorr is a software-development director at Associated Press.
The noise about DevOps just keeps rising—which makes it hard to pick out the true melody here.
I consider myself a DevOps evangelist, but the community behind this movement to bring together the efforts of developers and IT operations needs to be more anchored in reality.
Many technology decisionmakers get caught up in the industry buzz and, as they make their DevOps journey, get lured by some siren’s song. Implementation teams are doomed by good intentions based on poor information. Heard that tune before?
Learn The Score
The ambiguity of DevOps makes it daunting to get started with practical steps to advance toward those wonderful returns pundits espouse. Do an Internet search for “devops” and you’ll find more questions than answers.
Once you wade past all the “What is DevOps?” articles, prepare to be mugged by the vendors selling their all-in-one solutions that neatly deliver DevOps, whatever the hell it is. Before whipping out your checkbook for DevOps-in-a-box from XebiaLabs or ThoughtWorks or whoever, I suggest you start by understanding where your business can benefit from DevOps practices.
Settling on a specific implementation of DevOps with clear targets is the best way to combat the hype and haziness that underscores the DevOps space today. The precision of your vision and specificity of your targets will let you filter out the industry noise and highlight a few methodologies and tools that are appropriate for your particular environment.
Over a year ago when the Associated Press (AP) started seriously considering DevOps, we laid out some simple goals. We were keen on heavy automation to reduce turnaround for new features and improve our ability to reliably make changes. For example, adding a new way to help customers discover content on one of our Web platforms could take months to turn around in our search engine. Improving our user’s ability to find exactly what they need within a huge archive of content directly affects our bottom line. Delivering these types of features to our platforms faster increases our customer conversions and expands our business.
We didn’t feel it was necessary to support daily, continual updates to production systems like some DevOps frontrunners boast about. The AP doesn’t have the volume of changes that would necessitate this level of deployment frequency.
We wanted to keep our existing change-control process, which has proved to be a dependable gateway for making production changes in an organized manner. With everyone invited to listen in, a cross-department team evaluates all proposed deployments each week. The end result is a structured approach of managing change and a good way to keep everyone informed about new releases.
The AP’s DevOps goals are about reducing the time to reliably deliver new capabilities. That doesn’t require us to do rapid-fire production changes every day. We honed these high-level goals with more narrow targets like shortening the time it takes to deliver a search change from months to days. With specific, measurable targets layered on top of our goals, a vision was born.
Our goals led to the selection of continuous delivery as the way we were going to do DevOps. Continuous delivery reduces turnaround times through heavy automation and increases the visibility of change via deployment pipelines. Any change to a system, whether it’s functional or infrastructure, kicks off an automated pipeline that fully provisions your solution and runs an automated test suite to validate the change. If all tests pass, your change is ready to be released to production.
After a year of work to bring continuous delivery to a large organization, here are some simple realities to consider if you’re plotting a similar course.
Practice Makes Perfect
Continuous delivery requires fundamentally sound engineering to automate all the disparate pieces your systems need to run successfully. You’ll be working with more interpreted languages than you’ve used in the past, but thankfully you don’t have to jettison your computer-science and engineering degrees. You still need to think through failure cases, testing, and the availability of your automation glueware.
Teams implementing continuous delivery should plan for multiple sprints of upfront work to figure out all the automation and deployment bits before product features are even looked at. This upfront time will be significantly longer when you’re cutting your teeth with new tools and processes.
This can be a difficult pill for most to swallow. It leads to frank conversations with your stakeholders at the outset of a project. They need to buy into the idea that the returned value of automation and stability will likely outweigh the upfront time investment. If this is the first time you’re setting up a continuous delivery process, you will be measuring your upfront time in months. With more experience, this commitment can be whittled down to weeks. Over time, a mature and well-vetted process could result in a new project up and running in days.
The complexity continues past the beginning of the project. The troubleshooting of automation problems will be more difficult for your team than the debugging they’re used to on more traditional projects. Most developers live within the friendly confines of mature development environments and advanced debuggers. Systems engineers have a bevy of tools to help them profile infrastructure to identify root problems. The technologies that blend the worlds of development and operations have more primitive troubleshooting capabilities.
The tools and scripting languages that automate your continuous delivery pipelines represent a new layer of technology that can fail. At AP, our pipelines are orchestrated with a mesh of Jenkins and Ruby. When one of our pipelines breaks after a change, we now need to consider if it was the automation that failed in addition to app code or infrastructure provisioning. A developer accustomed to neatly stepping through app code to track down a problem will now have to crawl through logs to figure out why automation failed. While these new tools are effective, they will add drag to your schedule as the team figures out how to survive within a more rudimentary tool set.
Focus On The Orchestra
All the hype around DevOps and continuous delivery tends to overshadow the real challenge to getting these methodologies right in an organization. As engineers, we’re instinctively drawn to flashy technology—new tools, frameworks, and languages. The cool factor associated with DevOps right now means there are plenty of new toys to distract you. While fun to play with, no tool is going to be responsible for the success of DevOps at your company.
Unless you’re a small company or a startup, your business rests upon an established organization. The DevOps concepts of shared responsibility and ownership will appear to some as threats to the status quo. Others will feel vulnerable from the level of automation required to achieve the faster turnaround and greater stability goals. Some will feel overwhelmed by the sheer number of new technologies they need to learn.
Ignoring the cultural effect of DevOps will spell disaster for any type of implementation. Put down the shiny tools and get in front of your organization. Understand where the concerns lay and work with, not against, your colleagues to make the changes necessary to be successful.
Turn Down The Hype Machine
The mid- to long-term benefits of DevOps and continuous delivery are valid. I’ve seen some dramatic improvements in my team’s ability to reliably turn around changes using the automation we’ve invested in.
However, the journey toward realizing the returns from DevOps is longer and more painful than most anticipate. I was personally jaded by the optimism in the articles I’ve read about DevOps. In the Puppet Labs 2015 State of DevOps Report, you’ll read that “[h]igh-performing IT organizations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster.” Wonderful! Where do I sign up?
DevOps literature tends to focus on the end state, rather than the process to get there. You must define a vision, commit an upfront investment, and work through the organizational growing pains to realize the benefits of DevOps.
It sounds sweet once you get there.
Lead photo by Christophe Benoit