Measuring Impact: A Recap From Our Engineering ROI Webinar

As the VP of Product at Andela, I know Engineering ROI is a crucial part of the planning process, and it’s something that I am constantly discussing with my partners in engineering leadership.

At our recent webinar, co-hosted with Jellyfish, I spoke to Dana Lawson, VP of Engineering at GitHub, Juan Pablo Buriticá, former Head of Engineering at Stripe, LATAM, and Andrew Lau, CEO of Jellyfish, about how Engineering leaders approach all things ROI. Here are some of the highlights of what we discussed. 

Why is this topic relevant for you?

Dana Lawson: I'm always looking for opportunities to make measurement meaningful for our teams to really understand how we think about engineering capacity and our objectives. 

Juan Pablo Buriticá: I've been nerding out about engineering metrics for the past few years. I started to get obsessed with what we could actually measure that would drive impact for our teams and what tools engineering managers have to look at systems. 

Andrew Lau: It's easy to forget that engineering teams are part of a larger ecosystem. We need to connect the work we do to the business as a whole. If we do that, companies will make better decisions, it will make our lives easier, and it will make our companies more successful. 

What does ROI mean to you? 

Andrew Lau: Companies probably spend 20-60% of their actual dollars on their engineering team. If we're putting dollars into that department, what does the business or our team get out of it? If we don't have the answer to that question, we're in the dark.

Dana Lawson: ROI is your demonstrable impact. You really have to take a step back and ask, "what are our objectives?" At Github, we use OKRs, Objectives, and Key Results. Ask what are you trying to accomplish and what are you putting money into? Then, think about how you will measure that. If you don't think about how you will measure beforehand, how do you know what success looks like? 

Juan Pablo Buriticá: As we start seeing engineering groups as business units, parts of the business will expect our group to act like a business—to think about how we're investing, how we're not investing, or how we're spending. It's important to not just think about ROI, which is how we use our time and assets as a business unit, but how we are communicating with folks in different parts of the organization to drive the business forward.

What do you think about the return part of the ROI equation?

Dana Lawson: You can look at ROI through so many different lenses—net promoter score, cStack, or customer feedback. As a North Star, I use my colleague Nicole Forsgren's DORA framework from her book, Accelerate. At GitHub, we work as a team of design, product, and engineering so that we can collectively ask, "What does success look like on our platform or feature? How do we determine that so we can use it as a goal post?" It's all about checks and balances: what you're building, why, and how people are using it. 

Andrew Lau: It’s important to connect ROI to business results. How are the things we're doing driving revenue? There are lots of proxy metrics that don't directly ascribe to revenue or reduced churn but are leading indicators of those things. Challenge your teams to do that because then you can talk about how you're a return for the business. 

Juan Pablo Buriticá: As engineering leaders, there is a portion of ROI that we can control: our delivery pipeline. When you start thinking about return, what are you looking to get back? Think about how you measure and observe your systems. Then, you'll start to see your team as more of a business unit instead of just nerds having fun building software.  

Andrew Lau: There's another part of return, which is about engagement, happiness, and retention of employees. Don't forget about our teams.

Let's talk about the other part of the ROI equation—the investment part. How do you figure out the nuances of how to invest?

Dana Lawson: Start with the basics—how much money can I turn into more money? It's about thinking intentionally about what we need to accomplish. You need to factor in where you are as a business and work backward to best utilize that spend. If you can imagine yourself as a leader, you need to balance what you need to be successful against your objectives in order to have a return on investment. 

Juan Pablo Buriticá: I've spent most of my career in very early-stage companies where there is limited capacity, and that's all you can invest. Having to talk about where to invest that isn't generating new features is one of the most challenging things I've done as an engineering leader. Ultimately, when we think about investing, we need to consider what we're not investing in and why. 

Andrew Lau: We take part in crafting this ROI equation by figuring out what granularity is productive for the business. It isn't just how I quantify the investment but what sets the tone of the business. Make sure you find the right type of granularity so you can foster the right kind of conversation.  

How do you think about ROI for companies at different stages, sizes, and models?

Juan Pablo Buriticá: I usually think about three things: 1) how much can I invest in building and evolving our product, and that's where we should invest the most because that's how we grow; 2) what do I need to do to invest in maintaining the systems; and finally; 3) support. How do I ensure that all the work that comes up day-to-day doesn't cut into our features development? 

Dana Lawson: I think larger companies are more fiscally driven than startups because they have a lot of data, and therefore, the opportunity to reflect. You have to show more incremental improvement as a startup, whereas I can review things at the end of the quarter.

How do you get your engineering team to buy into ROI?

Juan Pablo Buriticá: Introducing metrics is probably one of the hardest things I've done as a leader. You can't look at engineering metrics as a tool for managing individual performance.

Dana Lawson: I agree 100%. Metrics are a guidepost to determine if you're putting the right investment in the right place. You need to start simple with a couple of key things that you're trying to measure. You don't have to have a huge complicated system to gain insight. 

Andrew Lau: I suggest that you start macro. Macro is not about attacking any individual but looks at the team as a whole. The other thing is don't create ROI in a dark room; do it with your team. If you have an open conversation about the reality of the business needs, I think you'll have a team that is not only open to it but bought in and invested.

View the entire on-demand webinar "Measuring Software Development's Impact: A Fireside Chat with GitHub, Stripe, Jellyfish and Andela” here.