During the last three years, Packlink’s engineering structure has been rebuilding from scratch.
Throughout this article, we will see some of the challenges and lessons learned of the initial steps of transforming our DevOps team to an Internal Platform area.
Our context
In a nutshell, Packlink is a Spanish Internet company focused on improving the shipment experience of our final customers and the customers of the eCommerce, professional merchants that ship hundred of products per week using our services.
We have a microservice-based, event-oriented architecture that was designed in 2014. Based on Kubernetes, our codebase is mainly functional Scala for the backend side, and typescript with react for the frontend side.
In the last two years and a half, we have grown by an x10 our tech area, composed of the product plus engineering area while transforming ourselves into a fully remote company.
Our internal staff grew, only in the engineering area, from nine people in 2019 to 73 at the beginning of 2021. A huge part of this growth was during the pandemic, and we needed to adapt all the company to a full remote culture in one of the worst possible scenarios.
In 2019 we designed an initial structure and career path that has been evolving aligned with the context changes.
Two of the most important iterations of our teams’ organizational design were: transitioning our vertical teams to business domain-oriented teams and transforming our single DevOps team into an Internal Platform area.
Both transformations have taken place at the same time and are still iterating and evolving.
Why Internal Platform teams
Originally, we had a single DevOps team (rebranded sysadmin) topology. With all the expected growth and design structure changes on the table, we concluded that this approach would no longer work for us anymore.
The OPS team had almost all the knowledge and ownership related to OPS, maintainability, infrastructure as code, and delivery.
This team operated as a support team, assuming the exponential growth of tickets with needs related to permissions, creation of infrastructure components, and deployments.
The software development teams didn’t count on this team as a crucial part of their software development flow. Hence, the DevOps collaboration usually happened at the end of the production chain, having frequent bottlenecks and blocks.
Their TOIL increased while they had not enough margin to create a roadmap to evolve the infrastructure or automate repetitive tasks.
We need to change our ways of working and make the software teams autonomous while focusing on making the right automation effort while evolving our infrastructure.
Easier said than done.
What is an Internal Platform team anyway?
An Internal platform team is a product team that provides self-service tools for other teams in the company, usually to software development teams.
When we talk about our Internal Platform teams, we mean the teams that provide a platform that makes the teams autonomous regarding cloud infrastructure, observability, and delivery tools.
Although the term Internal Platform or Platform team usually refers to the teams related to the DevOps philosophy context, any team that delivers self-service tools or interfaces to other internal teams to make them autonomous could be considered an Internal Platform team.
At Packlink, for example, we have another type of Internal Platform team that provides our Design System’s tools and components.
Autonomy?
In his book Drive, Daniel Pink guided us a deep look through three key factors that drive genuine motivation in teams: autonomy, mastery, and purpose.
Part of that autonomy implies removing the dependencies to enable the self-driving of the teams and motivates their culture of accountability and ownership.
To achieve that, there is a process where the platform teams need to understand the context of the internal (client) teams to enable their autonomy effectively.
This implies, in some cases, an initial collaboration to design effectively the right tools to make the team independent of the Internal Platform teams.
Product teams?
Our vision is that all the engineers are product people.
In the case of the Internal Platform teams, this vision is crucial. The software engineers and people with other roles are the Internal Platform clients; we need to think continuously about the Internal Platform’s outcome that we want to deliver them.
There is a huge challenge of changing the mindset to discover the need before thinking about how to implement the solution. This is probably the most difficult and interesting change that is needed to make in the context of this sort of transformation.
In addition, our solutions should be iterative to validate faster the value that we are delivering to our clients.
And what happened with the operations work?
With the original DevOps team topology, all the accountability for the operations was in this team. During the transformation, we still have the ownership located in the internal platform team.
Introducing new mindset changes and tools that enable autonomy to gain the correct ownership in the teams takes time.
We have started some steps toward a truly SRE culture by introducing debates about what we mean with observability and other community fancy terms.
How we started
We started to think about this transformation in December 2019. We spent months working on our internal alignment about what we need and want to implement in terms of team organization and design.
We already have teams known as “platform teams.” Still, they worked mainly on initiatives that impacted scalability and technology evolution but were not part of a roadmap with a concrete mission aligned with our new strategy.
The DevOps topology patterns and antipatterns and the Team Topologies book, written by Matthew Skelton and Manuel Pais, provided us a lot of guidance about the new understanding of these new Internal Platform teams.
The mission
The first thing we work on is the mission of the teams. We defined the mission for the Internal Platform teams as:
Delivering self-service tools for the rest of the software teams to make them auto-sufficient and not dependant on the platform teams in terms of infrastructure resources and delivering flows.
These self-service tools and automated processes will act as an abstraction layer of our infrastructure and cloud services. This abstraction layer acts as an Internal Platform as a Service for software teams.
This mission acted as a north star that allowed us to provide a solid direction during the transformation.
The team composition and engineering roles
At Packlink, we had DevOps engineers and software engineers specialized in concrete areas: QA, Backend, Frontend… With the new Career Path, we unified the ladders for the software engineers and changed the DevOps role to the Infrastructure engineer role.
In the Internal Platform teams, we count on Infrastructure, Software, and QA Engineer roles.
Internally, we referred to them as Platform Engineers, but at the moment, when we plan the hiring strategy and their career path, they still have their specialization.
They work closely with the Product Manager role, and as we said earlier in this article, all of them are product people, so they have been developing a product mindset.
The different focus streams
At the teams, we had a conversation about what aspects our product would impact. After some iterations, we find four main areas.
- Infrastructure
- Security (spoiler: we finally didn’t implement it)
- Delivery
- Observability
With these four areas in mind, we started designing an initial roadmap draft, a roadmap strawman, that envisioned what initiatives we could tackle in the next quarters.
We learned fast that the security stream was not a good idea in our context. We think that security needs to be implicit in all the tasks and needs, so having a stream to tackle it looked like a bad smell for us from the beginning.
This wasn’t the final picture of all the things we needed to work on. Our mission is to make the team autonomous by offering a great platform they will love to use, but we needed an MVP or a map with the first needs that we need to focus on.
Currently, thanks to the work of our Product workmates, we have reached a more mature roadmap and outcome-oriented initiatives.
Even following the Thinnest Viable Platform pattern described in Team Topologies, during the transformation, we still attending needs that come from other teams because they aren’t still autonomous enough.
We should do that unplanned work, the support one, and we needed to consider it in our teams’ organization.
After some iterations, we concluded that a rotation of a person every week was enough to guarantee a great delivery of that unplanned work with a great Lead Time while ensuring the rest of the platform teams the focus to work on the roadmap or enabling autonomy to our client’s teams.
Lessons learned
Change resistance
Each transformation is not a green path. We always have different opinions that could be seen as change resistance.
If all the different visions come with an actionable proposal and a draft of a concrete plan, they are welcomed, implemented, and iterated.
It is crucial to count with an iterative mindset, implementing the right metrics and early feedback.
Product mindset and roles
We missed counting on a product manager role from the beginning; it would have helped with innumerable aspects and to make it easier the product change mindset for all the team members.
On the other hand, some people on the team assumed some functions of a product role, and the team could interiorize new concepts and mindset that helped a lot the onboarding of the product person.
But it was pretty hard.
Ambassadors and influence
We found crucial the role of the ambassador of the internal platform to influence the engagement and adoption of the product.
This role acts as a bridge and helps the product people to retrieve the feedback and have a great understanding of it.
This is an area that we are still improving.
Metrics
“Metrics keep you honest”
— Staff Engineer by Will Larson
We initially designed metrics exclusively focused on validating the good advance of the transformation, but we missed the product-related metrics.
The outcome and how to measure it is crucial, even for the teams' organization. It helps you to focus on the right things and guarantee the focus to ensure the delivery.
Also, introducing the need to measure the outcome make it easy to focus the conversation on the value and not the implementation.
For example, the technology or tool you use to deliver autonomy in terms of deployment to your clients is less important than the expectations and concrete needs you want to cover in a concrete moment and context. And you should measure if those needs are covered successfully.
Abstraction layers
Talking with colleagues who implemented Internal Platforms or dev-tools teams in their companies, they follow similar patterns about implementing the tools that provide an abstraction to their software development client teams.
We recommend starting in an iterable way. We can get quick wins even with documentation, following the Thinnest Viable Platform pattern instead of starting to code a self-service portal.
Alternatives to an Internal Platform topology
We discarded alternatives to this approach for our context, but maybe they can work fine in other contexts; you can find other alternatives on the devopstopologies.com website.
One of the most commonly adopted approaches is introducing a “DevOps engineer” role in each software development team.
We discarded it because we want to remove the dependencies, external and internal, for our teams and make them autonomous to consume the platform resources and tools.
We thought that DevOps is a culture, not a role. Counting with the specific DevOps engineer figure in the team could generate an internal dependency and in the long term, fake the DevOps culture that we want to reach.
There could be a lot of debate around this and we know that this vision could not fit in all the contexts.
How is going
While we were executing this transformation, we have accomplished some big initial milestones, including changing our orchestrator from Docker Swarn to Kubernetes and providing a dev environment based on that tool for each development team.
Also, we have provided a lot of smaller quick wins in terms of autonomy and monitoring.
With a new structure, counting with Engineering Managers and Product Managers on the teams, we start a new stage with outcome-oriented metrics and building a more sophisticated software vision in our platform teams.
Throughout this article, we have reviewed the initial, first iteration of a long-term transformation adventure. We are still iterating it, and for sure, we will continue sharing our experience related to it; stay tuned!
Further reading
During the last year, we are researching and sharing many resources related to building Internal Platforms as a Product. You can check them at internalplatforms.com.
The pursuit of software teams autonomy: how we created an Internal Platform at Packlink was originally published in Packlink Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.
