I’m talking about teams that work to help other developers in the organization, by creating internal tools (Platform/Core teams) or improving the developer processes & experience (DevOps). 5 years ago I led a DevOps team, and now I lead a full-stack one - an experience that gives me a good view of both sides of the bridge.
You are going to read about:
My challenges in managing a DevOps team
5 reasons why is it so hard
What I would do differently in dealing with each problem
Tips for the product team leads - what can you do to help
The good parts!
Intro
I worked at a huge government organization, not connected to the internet (for security reasons). We had our own private cloud, which we developed for our developers’ use. So basically we tried to recreate AWS, with 50 people… Crazy right?
My team was responsible for the managed DB solution - Database-as-a-Service. Before I left, we managed around 500 VMs, running PostgreSQL/MongoDB. We were doing migrations, point-in-time recovery, upgrades, extensions, and so on.
Why is it so damn hard?
1. The priorities are changing faster than a Jenkins build 🌪️
Forget nice and quiet sprints, with high velocity and zero interruptions.
When your users are developers - priorities change every day. For the 2 years I was in that role, we really tried making Scrum work for us (2-week sprints, story points, planning, and so on). At some stage, we ended up saving 50% of the capacity for ‘unforeseen’ work in each sprint…
Something is always urgent in the DevOps/Platform world.
If I had another try
I would definitely go with Kanban, combining it with clear deadlines for bigger projects, and closely monitoring the type and severity of interruptions.
A tip for the ‘Product-team’ leader
Train your team to be as autonomous as possible.
Get familiar with the area of responsibility of the DevOps team, and let your own team handle small bugs. And use the word ‘urgent’ only when it’s really urgent!
2. High expectations and demanding clients 🏔️
Developers are never satisfied.
Your ‘clients’ are super-technical, and most of the time think they know better than you what you should be working on. This leads to the feeling that you always under-deliver.
Did you shorten the build times from 1 hour to 15 minutes? →A developer: “Why not 3 minutes?” (Because it’s HARD, asshole!)
You added support for MongoDB → A developer (true story): “Why not Redis! It’s much better” (WTF it’s not even the same use-case)
You implemented an event system infrastructure → A developer: “Why did you use Kafka instead of RabbitMQ? I used it before, it’s better!” (Because EVERYONE ELSE thinks Kafka is more suitable for us!)
Please don’t say out loud the responses in the brackets 🙃
If I had another try
I would have worked much harder at communicating the reasons for the technical decisions, and involving other team leaders and developers in them.
Maybe doing some ‘stakeholders meeting’ once a month, open to whoever is interested, discussing the future work plan. In my current company, our DevOps team hosts a ‘DevOps champions’ meeting, which serves a similar purpose.
A tip for the ‘Product-team’ leader
Your timely feedback is valuable!
If you really want to help, offer to give it during the planning of initiatives. Make it clear you’ll always make time for such discussions, and you’ll become a valuable partner (and get things you want done…). Saying afterward ‘But I didn’t need that!’ never helps.
There were a couple of dev team leads I loved working with - they were curious about what we worked on, and provided advice without expectations that I’ll do everything they say.
3. The connection to the business 💼
This one is hard even for a user-facing team, but for a developer-facing one… Damn.
Your work doesn’t contribute directly to the company’s bottom line, or help the customers in any way. Developers in such teams feel disconnected from the business, sometimes not even knowing how exactly is the product used. When people feel they don’t have a clear purpose - it has a big effect on motivation.
I remember the excitement of our team, when a team leader came to show us how their new system is used in production, and how our managed DB helped them achieve that. Until that point, that system was just a name in our spreadsheets.
If I had another try
I would have worked harder to understand myself what our business is about… Asking questions, talking to clients, seeing what the developers that use our systems work on. And then bringing this knowledge to the team.
A tip for the ‘Product-team’ leader
When the DevOps/Platform work helped you deliver something valuable to the customers - SHOW them!
4. Visibility and recognition 🏆
When things break - it’s always the ‘Platform team’s fault!’.
When a new feature is delivered to the customers - it’s never thanks to the DevOps team. Working behind the scenes is a thankless job.
I remember how almost every performance issue was blamed on our ‘Slow Databases’. It was very satisfying proving to developers that they just wrote shitty code. (But not very productive for the long term…)
If I had another try
I would gather success stories, and share them with the team. Even bring along a development team lead who used our products, and asked her in advance to prepare some success stories.
A tip for the ‘Product-team’ leader
Make sure to publicly recognize the work of the Platform/DevOps teams. Do it both to their faces and behind their backs.
5. Relationship with the Product Manager (PM) 🤝
When the product the team works on does not have a Figma, it makes the PM’s job much harder.
Especially if the PM is not technical - this can be a disaster. Very often this causes the decision to not have a PM at all, putting all the burden on the team leader. I was lucky, as my PM had a deep tech background (now she is a director of engineering). And still, it took time for her to fully understand the complexities.
If I had another try
I would insist on having a technical PM. Promoting and mentoring an interested developer from the team can be a great choice.
I learned that spending the time to explain our work was always worth it. It doesn’t have to be you - it can be one of the developers. When the PM understands the deep nuances and technical parts, your life is easier.
A tip for the user-facing team leader
If you need something from a platform team, and have to pass through a PM - be patient. Don’t cut corners and go directly to the team lead.
The good parts
With everything said - I still enjoyed a lot of my time in the DevOps (or just ‘Ops’) world!
Every day is different
You are never in danger of becoming a ‘feature factory’, doing the same again and again.
One day you are working on performance, the next on integrating Infrastructure-as-a-Service, then on Security, and you finish the month by writing some Python scripts.
Complex challenges
The problems you encounter can be completely unique.
Solving something that you couldn’t find anything on at StackOverflow is very satisfying (for some people). I remember a case where a team in our department (working with OpenStack), tried to translate answers in a Chinese forum for a cray bug they had.
Collaboration
Every few months, you’ll probably have a professional interaction with EACH developer in your department (if it is less than 50 people).
You also have a lot of presenting and explaining as part of your work, which are great skills to practice.
Takeaways
Managing a Platform/Core/DevOps team is a unique challenge.
Good cooperation between customer-facing and develop-facing team leaders helps A LOT.
You need to work hard on connecting your team to the business, and making sure their efforts are recognized.
