At Plato Elevate, our recurring summit for Product and Eng leaders, Howard Sueing, CTO of Onramp, welcomed execs from LogDNA, Brex, and CZI for a conversation on measuring your team’s success as you grow or scale
For one of the last panels of the day at Plato Elevate, our recurring summit for Product and Engineering leaders, Howard Sueing, CTO of Onramp, welcomed execs from LogDNA, Brex, and CZI for a conversation on measuring your team’s success as you grow or scale:
- Cosmin Nicolaescu, Brex’s VP of Engineering
- Kathryn Koehler, Chan Zuckerberg Initiative’s Sr. Director of Engineering
- Danny Chai, LogDNA’s VP of Engineering
Before getting to the heart of the topic, Howard asked a fun question to each panel member: what’s the one app on your phone that you think is on no-one else’s phone in the audience?
Now back to the main subject. As the panelists work in different companies of different sizes, with different priorities, Howard asked them: how do you measure the success of your engineering/product organization?
Kathryn believes it’s a tough question, especially as the Chan Zuckerberg Initiative has an 81-year goal: cure, prevent or manage all diseases by the end of the century. The question of KPIs is thus a bit tricky, to quote her: “what KPIs do you put in place that you can actualize in my lifetime?”
CZI works with many scientists and they have a much longer timeframe in front of them: their experiments last for years, which makes it difficult to assess success in a shorter term. When starting a project, she ensures KPIs are crisp and clear, measurable and actionable and keeps bi-weekly and quarterly meetings. This enables her org to avoid going off-the-rails and adjust when needed.
Danny Chai has similar problems at LogDNA because the company is growing fast. This makes it hard for him to plan, and he believes the best way to measure success has to be around other elements, such as culture. While projects and roadmaps may change over time, culture should be infused in the company’s roots. For Danny, elements in LogDNA’s…DNA is put in place to help each team succeed without being specific about product features they want to launch or short-term objectives. Some examples include the culture of code review, the commitment to a healthier codebase or the involvement of employees in mentoring new hires.
Cosmin Nicolaescu is also part of an organization that deals with crazy growth and aggressive timelines. He chose to drop short-term objectives and do quarterly planning and OKRs instead. He measures everything, from people to culture to features. OKRs thus range from team efficiency to alignment across the company. Brex also ensures service-level objectives (SLOs) are met: as an infrastructure company, you end up having a lot of reliability as part of the things you measure.
Howard then asked the panel about common mistakes that happen when measuring a team’s success.
Kathryn argued that watching throughput too carefully might be a mistake. What you get out of the door is only a piece of the whole picture. What’s the culture? Is the team in a state of flow? Are they communicating with each other? Are people elevating issues as they come up? These things should be looked at too, instead of monitoring the number of lines of code as someone suggested in one of Kathryn’s previous companies!
For Danny, a common issue lies in finding the right things to measure. It would seem logical to measure the number of tickets handled by a support person, but it is not what a company wants in the end: it wants these tickets handled in the best manner so that customers are happy. The number of tickets handled would thus be incomplete or even irrelevant.
When he was at Facebook, they had a long-time goal of making the pages load very fast. It is a reasonable goal, in service of a bigger objective: increasing the time people spend on the site. But the team focused so much on that intermediate goal that interfered with the bigger idea. New features that were deemed as too slow would not go out, for instance. It’s therefore important to keep in mind that intermediate goals are…intermediate. They make sense only when they’re aligned with the bigger picture at all times.
Cosmin added that objectives are just a tool, and how you end up using them is most important. If you choose to measure the wrong goals, you’ll end up optimizing the wrong things. If you choose to measure things that you cannot influence until right at the end, then it’s a boolean: either you did it, or you didn’t. In such setups, you cannot adjust over time and the objective becomes a bet, not a tool to improve over time.
The biggest failures Cosmin has seen in his experience, is when people get obsessed about the things they are trying to measure in their OKRs, and then give up on the bigger picture even though they realize it has changed.
Kathryn added that two objectives can consequently be concurrent: when one moves up, the other goes down. For instance, if you want to increase your time to ship new features, you may decrease the quality of the codebase. People should thus be deliberate about how they’re twisting the knob.
Howard then asked the panel about conflicts in the choice of objectives: when you have direct reports, they might push back, not get on board with your vision or your strategy, and the way you measure success for the org. How do you mitigate this and engage people?
Danny believes this is why having cultural objectives is key. When managers steep in the culture instead of coming with their standards, it ensures people get on board with less friction.
Kathryn thinks managers need to investigate the source of the conflict: some people are afraid of change, some others do not mind, some just do not want to be graded and consider such KPIs as punitive. If you get to the root cause of the issue and explain how important it is to measure a particular objective if you show direct reports it helps other stakeholders in their jobs, and if you explain how KPIs can show which areas need focus, there are bigger chances that they follow the lead. Cosmin agreed and summed it up: understand why people push back, give more context, explain, and hopefully defuse the conflict.
Now, you have your metrics and everyone is on board, but there might be times when hyperfocus on metrics can compete with the notion of product excellence. How do find the right balance between giving more attention to an OKR and doing the right thing for a product?
Kathryn believes that if people meet their OKRs but feel like something is off the track, it might be the case. Managers and employees should look back retrospect on what they missed and iterate. Managers should ensure people understand that it’s not just about the OKRs, that there’s a lot of intangible that cannot be measured, and that work, in general, isn’t a grade book!
Danny thinks it’s necessary to adjust objectives over time as the context changes. Sometimes it gets difficult to remember what you were aiming for in the previous weeks. The solution is to make sure some objectives are flexible while some others are very specific.
Cosmin agreed and checks in often to make adjustments. Even if you have your OKRs, KPIs and SLOs set, at the end of the day, you’ll have a gut feeling if something is off, and need to troubleshoot that feeling and adjust accordingly.
Howard asked another question: what are some strategies for tracking actual OKRs and KPIs in each of your unique organizations? How do you set them and track them?
At CZI, there are 11 disparate engineering projects. They don’t always pull together for the same objectives. People work in parallel on different topics that may be unrelated, so the organization is a bit different than if it was a product or engineering team moving forward towards a unique product.
She uses a unified Google Sheet that holds the important info for each team and checks in periodically. Most of that work is manual. When she was at Evernote, reporting was much more automated, with data pulled out of JIRA or Github: number of hotfixes between actual releases, number issues to close, etc. On the other hand, a lot of Danny’s measurements are made on the individual level. He tracks weekly on 1-on-1s.
Cosmin’s set up is close to Kathryn’s, with a bit more automation: Brex uses a spreadsheet that has every team in the company. Each manager has OKRs for the team and each objective is linked to JIRA. This enables the org to do companywide reports every month.
Finally, Howard asked the recurring question of our recurring event: thinking about the span of your career thus far, about managers you had, or key influencers in general, what’s the one piece of advice that someone has shared with you that has influenced you or your career?
Thanks a lot to Howard for moderating this panel, and to Kathryn, Danny, and Cosmin for their precious insights!