Creating a Livable, Enjoyable, and Rewarding Environment to Be an Ops Engineer

Problem

Let's be honest, systems administration, whether working with bare metal or in the cloud, is often worse than a thankless job. I've seen too many ops engineers work themselves to the bone fire-fighting, scaling, and migrating the foundation on which entire businesses stand as if in a full-on marathon...sprinting. They get no chance to breathe, normalcy, or arrive at the autonomy or purpose we all seek to earn our work. Not on my watch.

Actions taken

• We maintain a sane software engineer to ops engineer ratio. Whether the software engineers are great or sucky, you're in for a rough ride when the ratio is stacked against you. Don't let this happen.
• We deploy during working hours whenever possible. Our engineering team practices no-downtime, continuous delivery within a time window that allows for issues to shake out before staff goes home for the day or weekend. We typically ship Monday through Thursday 8 AM to 3 PM. If a completely shippable deliverable misses that window, we often wait until the next reasonable workday to deploy. We don't want anyone, in software or in ops, paged while out of the office. It's a terrible way to live. Strive to keep work at work.
• We have reasonable maintenance windows. It took a bit of Google Analytics investigation and some convincing inside the company, but our maintenance window starts at 8 PM EST when we need one.
• We assume it's a software issue until ops is proven guilty. Operations issues happen, but software change or software engineering flubs are usually at fault. We make sure our issue escalation process assumes this reality. Our ops engineer is our last line of defense, not our first.
• We make space for proactive ops engineering. Help your ops engineers make time for proactive work. Tell your software engineers to endure that less important but painful pain they're complaining about just a little longer so that ops gets the space it needs to address the top issues on its list too.
• We check-in regularly. Ops engineers are a part of our standard kick-off meetings and stand-ups. They have an equal voice at the table. They serve the needs of the business like the rest of us, but they are not subservient. We connect out-of-band to see how things are going too.

Lessons learned

• Systems take serious work to build and maintain. Don't ever let an employee drown in work.
• Since we practice continuous delivery, maintenance that requires our site to be offline is rare, so it's a reasonable trade-off.
• Too many people outside an engineering department or even insufficiently experienced software engineers assume the computers are to blame when things go down (guilty!).
• Nowadays, software engineers are given space to pay-off tech debt. Not only does this make it easier for them to ship features in the long run, but it also makes their working environment less toxic.
• We pay our ops engineers competitively. We send them to meetups, conferences, and training just like software engineers. We let them go to the dentist when they need to. We praise them for their work. We treat them well. Do you?

Source: How Well Do You Treat Your Sysadmin/DevOps/Ops Engineer?