There Are No DevOps Engineers*

* The asterisk is doing serious heavy lifting in the title. It usually does. But this time, it’s carrying an entire argument.

LinkedIn and the Mythical DevOps Engineer

LinkedIn! That place on the web where jobs are plentiful, everyone is living their dream, and good things are shared as a burden that only the OP can understand. Sure, we could keep bashing the platform all day, but it still is the place to look for another job, or ideally, to be invited to one. I did it, and so can you!

And among all these dream jobs and humblebrags lies one of the most elusive creatures of the platform: the DevOps Engineer.

For a few years, it was one of the highest-paid positions in the Software Engineering world. I saw countless Engineers trying to become one, following what the internet threw at them as the “minimum requirements”. From basic version control to becoming an expert in any platform technology, the DevOps Engineer should be able to run a mid-sized startup at the point of their finger while doing A, B, and C funding rounds with infrastructure-as-code. Someone even once told me they saw a DevOps Engineer save an entire village with a Jenkins CI/CD automation that they left at their previous company as a safeguard for a deployment. What a fantastic profession.

Somewhere along the way, the title stopped describing work and started describing a shapeshifting myth and legend.

Back When Ops Was Ops

Back in the days when clock bugs threatened to stop the world (Y2K, anyone?), a System Administrator was responsible for keeping busy servers humming, making use of deadly terminal acrobatics to avoid downtime. Then the world changed, not overnight, but fast enough that many SysAdmins still remember the sound of spinning disks when they hear the word “uptime”. Nowadays, everything is a container or a virtual machine. Long are the days when your console was a loaded gun, ready to destroy a machine with no “restart from snapshot” to save you.

The bold System Administrator, besides keeping your lab or data center spinning workloads 24/7, was also responsible for deploying the Software that pesky developers kept churning out every few months. And when they were able to execute a one-shot deployment, there was always a set of scripts and a few other terminal utilities to check whether their loved machines had enough beans to keep them going. Slowly but surely, sometimes (many), the Software did not behave as expected in the developers’ local environment, and the System Administrator had to intervene: stop it, roll it back, uninstall it, and distill a few bad words straight into the heart of a developer’s ego. What an excellent time to be disdained.

The developers couldn’t understand it. The Software that they hammered out for so many weeks worked so well on their machines. Surely those servers weren’t as new as they should be. Hard drives were prone to bad sectors, after all. Yet the developers did some work, changed a few things, found a couple of memory leaks, and with a brand-new 0.2 version, dropped it in the System Administrator’s lap, ready for another merry-go-round.

I’ll stop recounting the tension between Development and Operations and all the issues that could be solved if they worked together after a kumbaya sing-along. Unfortunately, kumbayas rarely scale, so the industry went looking for a structural fix for what was, at its core, a cultural problem.

A Philosophy is born

Patrick Debois. Never heard of him? No worries. History says that Mr Patrick shouted the word DevOps to quell yet another incident root-cause-analysis (or just RCA) meeting, where one Operations team was already dismembering a couple of developers for not sharing the database migration file for the deployment process. They mentioned something about decoupling development, but to be honest, I don’t think developer decoupling is the same as software decoupling. I digress yet again. Also, everyone knows that DNS is always the RCA.

While Mr Patrick merely said out loud what everyone was thinking since the late 90s, early 00s (oh no, the Y2K bug again!), it was enough to kickstart a movement to make Software delivery great again (it never was). Google was already implementing DevOps before it was a thing with the Site Reliability Engineering practice, although blending fiction with reality in the seminal book Site Reliability Engineering: How Google Runs Production Systems. But while “the industry” didn’t pay enough attention, the transition from delivering Software on a disc to the Cloud deployment happened in the blink of an eye, more or less like new JavaScript frameworks are born and die. This made many people angry, of course. How can we keep installing new Software without System Administrators’ complaints and poor development practices? Unacceptable.

Between 2009 and 2016, we saw the rise of the DevOps movement, with the release of seminal works such as Continuous Delivery, The Phoenix Project, and The DevOps Handbook. These were the distillation of the work that front-line engineers were already doing successfully, with The Phoenix Project being an exceptional outlier for a very good reason. Woven as a fictional business novel, it was threaded by the same kind of narrative that Sensei Eliyahu Goldratt used in The Goal: A Process of Ongoing Improvement. The latter, depicting the nitty-gritty of factory floors and production lines, set the stage for a continuous-improvement mindset in the face of physical resource constraints and humans being human, using a writing style rarely seen in such a context.

We think that Software Delivery is somewhat special until we actually learn a few things.

The supporting beams of what modern Software Delivery required had been around since the 70s, and these were the same tenets we rediscovered after fermenting bleeding-edge practices into yet another fresh contextual philosophy called DevOps.

And don’t get me wrong: this isn’t cynical. You can’t map the Toyota Production System to Agile out of thin air. The work had to be done.

From Movement to Product

But as always, we can’t have nice things for long, and there’s always someone ready to package description into a prescription product… that, guess what, it needs its own operators.

In the same period, we saw the DevOps movement, philosophy, and context being translated into artifact formats, pipeline blueprints, and the necessary tooling, which, of course, had to be deployed, configured, and managed. The industry installed DevOps in every development shop, IT department, and Network Operations Center, just like any other off-the-shelf tool, and decided it would require a brand-new specialist for the work. This half-truth had to slice and dice a philosophy into concrete practices ready for consumption.

Instead of embracing what was already emerging — Site Reliability and Platform Engineering — we decided that we needed a role and a team to “make DevOps happen”.

Oh, don’t forget the tooling, of course.

Enter the DevOps Engineer

The figure will be responsible for implementing what all the cool kids are doing with automation to build CI/CD pipelines, manage and orchestrate containerized environments, and describe infrastructure as code, as outlined in that influential company blog post. Guarantee that everything is triggered by a Git commit and that we always do Blue/Green deployments if that was what our socials decided “high-performing teams” were doing that week. Since there is “Dev” in the name, the individual should also implement Software to glue all these things seamlessly. And since there is “Ops”, they should also be responsible for installing, configuring, and operating monitoring tools while backing up all data. Well, eventually they’ll have so much free time that they can develop a bespoke observability solution that sees it all end-to-end and provides it as an internal product. Congratulations, you will be the proud father of a custom Internal Developer Platform that enforces Spotify’s Golden Paths with no escape hatch.

We’ve created a brand new layer between Dev and Ops called DevOps. Fun stuff.

If you’re paying attention, we are just bundling multiple roles and activities under the same name while forgetting altogether what DevOps actually means.

We are going back to the rigid factory-floor pre-Lean and Agile. The cherry on top is the Taylorist application of DORA metrics to guarantee that we are “Industry standard”. Nothing like that snug feeling of a tightly controlled process.

Well, rant aside, I can tell you that most, if not all, of those “DevOps Engineer” roles up top are plain and simple Cargo Cult. But I can already state the opposing argument to this piece: there are actually DevOps Engineers, but you probably aren’t hiring them or even know what they are.

The DevOps Engineer is found in the characters of The Goal.

Alex Rogo, the aspiring plant manager, understands constraints and how to navigate them with Jonah’s guidance through systems thinking and flow. How, in the end, we realize that the principles are horizontally and vertically cross-cutting, scaling from a single factory floor to the whole set of company production plants. How to envelop the workers who know the constraints of their silos, and can work together in a fluid production line.

You can map those characters to our beloved Phoenix Project and find the DevOps Engineer again. When Bill Palmer takes the helm of IT Operations, he is supported by Erik, who leads his growth just like Jonah did with Alex. How they broke the “expedite” cycle when Brent was a major bottleneck just by being competent. Like “expedite” was the word when something needed to be loaded into a truck at Alex’s plant. And not forgetting Patty MacKee, who doesn’t stick to a change management process but insists on revamping and adjusting it after each iteration, something that we also saw from Lou, the knowledgeable CFO from Alex’s management team, both with a keen eye on system design and engineering.

So What Does the DevOps Engineer Actually Do?

Many things, and none of what we usually see in our glorious job posting requirements.

They design, build, and run systems while working with teams that develop products and services.

They intervene, change, adapt, and measure.

They monitor and do it all again and again. Importantly, they are knowledgeable across the entire process.

They know that to install a pinion (one of my recently favourite words), you need a special tool, spare parts, and a logistics pipeline that delivers them to your doorstep.

They know that a log line in your favourite Observability tool was produced by an affordance in your Web interface.

They go deeper, broader, and further to make those TikToks blazingly fast on your mobile device (sighs and loud moans)

They are your System, Mechanical, and Software Engineers who embrace the DevOps philosophy and live by its tenets every day, without being constrained by a fixed set of tools or process expectations.

So yes, the asterisk, right? It’s doing heavy lifting because it’s the most honest part of the title.

We needed a footnote big enough to contain our collective confusion.

And we still do.

Now, one of these days we need to talk about the “full-stack developer”…