Being blocked by a DevOps/Platform team sucks, and it always happens in the worst moments.
The truly great engineering managers know how to unblock their teams without involving higher-up managers.
Today, I brought a special guest who will spill his secrets. Michal Zion is the CEO of MeteorOps - a DevOps services company, and a long-time friend of mine, whom I met in a coding bootcamp.
3 years ago, we started to work with MeteorOps in Taranis (the startup I work at), and Michael’s team completely transformed the way we operate. We are now an R&D department of 30, with only a single DevOps engineer, and it just works. No bottlenecks, great developer experience, and much lower cloud costs.
His company (MeteorOps) offers DevOps audits to analyze a company’s DevOps situation and provide actionable insights. He agreed to offer a free Starter DevOps Audit for relevant readers of Leading Developers in the next 7 days, using the code LD25:
This is an affiliate link, but also a 100% honest recommendation, based on 3 years of working with MeteorOps in my company.
In today’s article, Michael will share 6 behind-the-scenes tricks that will help you get more ownership for your team, and never be blocked again.
Passing the mic to Michael 🎤
6 behind-the-scenes tricks to never being blocked again
Hey fellas, nice to meet you :)
I’ve been a DevOps engineer, managed DevOps engineers, and worked with DevOps engineers across 150+ companies over the past 10 years.
Let me tell you a little secret, we (DevOps engineers) have some things in common.
Most DevOps engineers sit in front of their screen, constantly afraid of the scariest thing an engineer can experience - *The Shoulder Tap* (or if they’re working remotely, the Slack DM).
Imagine the world from their perspective:
There are always multiple big efforts piling up
Support requests and shoulder taps come in like crazy
Focusing on something for more than 30 minutes without interruption is rare
I know what you’re thinking:
“This is not DevOps, it’s just an Ops team acting as a bottleneck”.
You’re right.
Ironically, the ones with the most influence over how the DevOps team operates, are the developers.
Today I’m going to share 6 super powerful tricks that will have you as a developer/EM enjoy a perfect DevOps scenario - where bottlenecks don’t exist:
Trick #1 - Developers Ops PRs
Trick #2 - Intriguing Questions > Requests
Trick #3 - Costs Histogram
Trick #4 - Developers Alerts
Trick #5 - Praise Publicly
Trick #6 - You are the customer and the PM
Let’s dive in.
Trick #1 - Developers open PRs on DevOps repos
There’s this story about a developer who asked questions on StackOverflow, logged in with a fake user, and answered himself with a wrong answer. Then, people immediately corrected the fake user with the right answer.
This genius leveraged the fact that people rarely answer questions, but will gladly correct someone else’s mistake.
The same goes for fulfilling a developer’s request by the DevOps team.
Instead of submitting a request to the DevOps team, waiting for them to prioritize it, and eventually getting it done, simply open a Pull-Request to do it yourself. Even if you have no idea what you are doing. Look at the merged pull requests, find a similar one, and try your best.
This does a number of things:
The DevOps team will see your PR and say “wtf is this?”
They’ll reply quickly on the PR with what needs to be fixed
You’ll have better knowledge of what to do next time
The DevOps team will gradually standardize this process with automation and policies
This will initially have some friction and require some learning.
Do this for a year and see how the company completely transforms.
Note: you should still rely on the DevOps team’s review, approval, and testing (automated or not).
Trick #2 - Don’t request, Intrigue!
DevOps engineers get tons of messages saying: “It seems X is broken, please help me fix it”.
Even if they want to help, they grow tired of it, and will probably postpone you as much as they can.
BUT, if you ask them an interesting question - a completely different thing happens.
When you request something from a person, they mostly feel the responsibility, which carries weight, and isn’t necessarily pleasant.
When you ask someone a question, you practically query their brain. You yank a thought and memories out of them with one question, and very gracefully so. That person wouldn’t necessarily think or remember that thing you asked them about on their own, but you helped them pull it out.
The trick here is this:
Ask an open question about the challenge you’re facing - “How would you approach X?”
Dig deeper into the DevOps engineer’s thoughts and ideas
Acknowledge and compliment the ideas to get them to dive deeper: “Interesting. How does it affect Y?”
At this point, your DevOps engineer is already curious, and did most of the thinking needed to fulfill your need. Now it’s time to ask for help: “That’d be awesome. Do you mind helping me complete this?”
Some people will hate my guts for this trick, since it hacks the organization’s structure. Especially DevOps Team Leaders who have a ticketing system in place… So, use this one cautiously!
Trick #3 - Ask for (or create) a Monthly Costs Table
This one will help the developers on your team gain control over your applications’ underlying infrastructure. By understanding the actual costs (and their trajectory!), they’ll be more curious and will start to save costs naturally.
The action is very simple - have a histogram / table of your team’s cloud costs that everyone can see.
Let’s take the following as an example (AWS):
By having your developers look at their team’s resource costs, you can see when the costs went up, when they went down, and what exactly was affected by it.
BONUS: do some detective work on the above table, and let me know what you understand from it ;)
Creating this table requires some effort but it’s definitely worth it.
You can ask your DevOps team to create it, or you could take the raw data and create it yourself. It’s possible to create it using BI tools, or even using a simple spreadsheet.
In the above AWS example, the general steps you’ll need to take are these:
Under costs / data exports → create a CUR2 data export
Use Athena to run an SQL query against the data export (involves creating a Lamda)
Being able to tell what resources belong specifically to your team requires tagging resources. The bigger your company is, the more this analysis will rely on having more resources tagged.
Trick #4 - Developers create alerts
In most companies, the DevOps team creates all of the infrastructure alerts. This usually ends up the same way: the DevOps team gets the alerts (phone calls, slack messages), and resolves them.
Some alerts examples:
app’s CPU utilization > 75% for more than 1 minute
app’s HTTP 5XX response count > 5% for more than 1 minute
app’s HTTP 4XX response count > 5% for more than 1 minute
Does it mean the DevOps team is qualified to resolve each alert? Nope.
But will they? Yep.
How? They’ll add resources, automate a restart, etc.
The result is a system full of patches, and will create many problems for the development teams in the long run.
So, what’s the trick?
You need to create those alerts, and route them to your on-call rotation.
This creates a completely different scenario:
The developer creates an alert for his app (e.g., app_cpu_utilization > 50%)
The Developer pushes a change to production
The Developer gets an alert related to their application
The Developer reverts his change & remediates what’s needed
Alert gets resolved
The Developer gains awareness of the impact of his change and reintroduces the change
A level-up from this approach is that when a developer creates a new alert, a generic version of it gets bundled with newly created services in the system, so all developers benefit from it (gets routed to each service owner).
Trick #5 - Praise publicly
When a DevOps team member helps you with something, show gratitude publicly. Being of value, status and pride are a source of motivation for getting work done.
If someone helped you, and you appreciate their help, show it:
Write a message praising specifically their work in a shared channel
Thank them in front of others on a company call
Good gratitude != “Thank you, Sam!”.
Good gratitude == Share what’s the positive result of their help.
Trick #6 - You are the customer, and the PM
As a developer, you build stuff for the company’s customers. You feel their needs, because you directly answer them. In most cases there’s even a Product Manager that continuously discusses the customers’ needs with you.
For the DevOps team, there’s no product manager. They don’t really know how they impact their customers (you), and they are even more disconnected from the business side.
You as the engineering manager, can fill the role both of the customer and the PM:
Share the impact their work had on your team.
Explain why you need their help, what is the business goal you are working on.
Join the DevOps roadmap planning, share your inputs, be curious.
Invite the DevOps team leader to your own roadmap planning - help them understand how the upcoming months will look for you.
When the DevOps team is more connected to your needs, and your customers’ needs, they’ll be more motivated to help you.
So what’s next?
Each of those tricks can help you, but the truly amazing change happens when you start to combine them. That’s the actual meaning of DevOps - having your developers own their infrastructure, understanding how it works, and never being blocked by another team.
If you would like some help on the journey, Michael offers a free Starter DevOps Audit for relevant readers of Leading Developers, using the code LD25:
This is an affiliate link, but also a 100% honest recommendation, based on 3 years of working with MeteorOps in my company.
Final words
I have led a DevOps team for 2 years, and I can tell you it’s harder than leading a classic product team. Much more reactive, many distractions, and less clear goals.
I’ve shared my experiences here:
If you, as the engineering manager, put some effort into making the lives of your DevOps team easier, I promise you they’ll appreciate it and you’ll reap the benefits.
What I enjoyed reading this week
Cross-platform mobile development by . A must-read if you are a software engineering on planet Earth.
The anatomy of a breakthrough “Service-as-a-Software” business by
