I recently facilitated an engineering strategy workshop for a company for a couple of days, and during the workshop there was an activity around metrics.
The workshops are an improved version of my posts on how to design and execute an engineering strategy, with the mail goal of facilitating the hard conversations to make the right decisions based on my client’s context and needs.
I cannot decide the strategy for them, but I can help them to have the right conversations with the right structure. By the end of the workshop, tech leaders are aligned on what’s and what’s not important, which is the focus, and how they will communicate the strategy to the rest of the organization to make sure a fluid execution happens.
A great way to accomplish this is to be explicit on the metrics. What will we measure? Why? How do we expect this metric to contribute to the engineering strategy?
After the workshop, I began to consider how organizations approach metrics and how they influence the engineering strategy.
I do the metrics activity after the direction and coherent actions are defined because it is a great way to understand what is important for people, and their interpretation of the engineering strategy so far.
We finished the activity and I didn’t feel super about the result. It helps people figure out which ones are important, but I missed something, like some sort of dimensions or way to categorize them, and how they impact the engineering strategy.
I had the next questions:
Are they leading or lagging metrics?
How do they impact or influence the engineering strategy directly or indirectly?
Should they be at direction/guiding policy level or at coherent actions level?
What should be actually measured? Only related to the engineering strategy or engineering as a whole?
Leading vs. Lagging Metrics
Lagging metrics tell us what already happened in our business.
Leading metrics predict what will happen in our business.
I recalled the Developer Productivity Metrics post, by the Pragmatic Engineer along with DX, and I saw that most companies measure the productivity as a mix of leading and lagging metrics depending on their own context.
I think it is good to have both types of metrics to think about the short term impact and also the long term vision.
After the workshop, I took some time to see how many of them were lagging or leading. There was a mix of vanity, lagging, and leading metrics, but the majority were lagging.
What happens with the lagging metrics is that you cannot influence them directly and hard to influence. Yes, they are important, but they don’t tell you how you can get there. A crucial aspect to make an engineering strategy actionable.
How metrics impact the engineering strategy
The metrics you choose for the engineering strategy need to be:
Measurable to know how much progress has been done and when to stop.
Actionable to help teams make it executable.
Leading indicator to predict the future instead of telling what already happened.
Understandable for the whole engineering team.
At direction or coherent actions level
Deciding on a unique metric for the entire engineering strategy is extremely complicated, as it will be a mix of different metrics.
Being only at direction level might mislead people as it will be too generic and not actionable, at coherent action you might have too many metrics. Based on my experience, it looks like you need a sensible number of metrics between 3 and 5 to represent the strategy, depending on the amount of direction goals.
My suggestion is to add the leading metrics at direction level, as the coherent actions might change quite often as you learn. Having some stable metrics helps people to understand the big picture without the need of constant reevaluation of the strategy.
I was reinventing the North Star Framework
At some point, I realized I was just applying the North Star Framework ideas to Engineering Strategy. So, why not combine both instead of reinventing the wheel?
When I see the north star framework and how I was using the engineering metrics, I can see how they fit in the next way:
The north star metric is derived from the direction/guiding policy. It aims to show the path forward, but you cannot influence it directly.
The input metrics are part of the coherent actions, you think that they will influence the direction/engineering north star metric. And you have several of those, as a strategy requires a combination of actions to produce the desired result.
The mid-long term impact will be part of the health of engineering metrics, like developer productivity metrics. You want to influence your engineering strategy, but you don’t want to put your organization in the bad spot in which executing becomes harder and harder until a point in which you cannot execute.
Example
An example of a north star metric adapted to engineering strategy in which the direction is to reduce the team cognitive load to address the business need of faster experimentation.
Main differences between North Star applied to product and engineering
The engineering north star is relevant as the engineering strategy is.
Unlike the product north star metric that can last months to years until you need to review it, the engineering north star can be months to quarters long. It can have a shorter life span.
You might need to represent the engineering north star metric as a combination of 1–3 metrics.
I find it hard sometimes to represent the whole engineering strategy with only a metric, as we need to do trade-offs, and we want to be explicit about them. I think it is better to be explicit on the trade-offs than make a single engineering north star metric that’s too generic and non-actionable.
The mid-long term is about engineering health.
You want to measure those lagging indicators of how good are you doing, in case some drops that much that you need to take action to improve to not let your organization deteriorate a lot.
