With how fast things are changing, I wanted to write down my thoughts on the trends I’m seeing today! Let’s see how this ages.
The boundaries between engineering, product, and design are dissolving. AI enables product managers and designers to prototype work that previously required engineering resources.
I talked to Michelle Bu from Stripe on a panel recently, and she pointed out that she’s seen these roles converging at larger tech companies as well as startups. The gatekeeping function engineers once provided is rapidly eroding. This makes product-knowledgeable engineers more critical than ever. The engineers who thrive aren’t just technically skilled—they understand user needs, business constraints, and product strategy.
Ownership has always been the top thing I’ve looked for when hiring engineers at startups. But I think this profile is becoming the standard, not the exception, and I expect this to continue.
Engineering leadership needs to be hands-on now. It used to be that hiring and scaling large organizations was the key senior leadership skill, but that’s changed rapidly.
Will Larson captures this shift well in his recent writing. I think that although the trend started with the end of ZIRP, it has only accelerated. At this point, AI capabilities are changing so rapidly and the implications for engineering strategy are significant. That means leaders have to be very good at understanding exactly what is possible at any given moment, and that means digging deep enought to understand the difference between a demo and
On the bright side, AI tools do make parts of this transition easier. Orienting yourself in an unfamiliar codebase and achieving productivity happens much faster now.
This challenges engineering leaders who’ve drifted away from technical work, but I honestly think this is a positive trend overall.
Code generation is dramatically faster, but that’s only one part of building software. Michelle observed this happening at Stripe too: implementing solutions has accelerated, but other parts of the engineering process haven’t kept pace.
Consider the complete cycle of fixing a bug: understanding the task, deciding on a solution, implementing the solution, testing, and deploying. AI tools have transformed the implementation step, but the others remain largely unchanged.
This reminds me of Amdahl’s law – a 2x improvement in implementation speed might only yield a small improvement in overall velocity if the other stages haven’t been optimized. It also implies that accelerating the rest of the process is where startups should focus next.
Accelerating prototyping has huge implications for early-stage startups, though. A VC friend recently noted that the bar for Series A funding has risen dramatically. Startups that used to raise with just a landing page now need working MVPs and paying customers.
AI tools make it possible for small teams to build working products faster than ever. Even if they’re hacky under the hood, features that once required weeks can now go live almost immediately. User interfaces that demanded dedicated design resources can be created by technical founders with AI assistance.
These changes haven’t yet propagated up the stack to later funding rounds – as far as I can tell, the dominant trend there is still the “zombiecorn” as VCs move away from funding non-AI-native startups. I’ll be interested in seeing which companies that weren’t originally AI-native manage to make the leap.
]]>When I joined Stripe’s ML Infrastructure group, everyone knew we wanted to make Stripe’s ML teams more productive, but opinions varied about what that meant. The infra team owned and maintained systems that were critical for the company’s core business, but difficult and painful for developers to use. Tradeoffs between maintenance and new feature development were difficult and arguments about priorities were frequent. As a result, it was difficult to make consistent progress over time, which was frustrating for our users and demoralizing for the team.
The first step towards improving this was to clarify our priorities so we could figure out what work actually had increased impact. Drawing on my time as an ML engineer (and the experience of the ML developers on our customer teams who helped me), I built a model inspired by a stocks-and-flows view of engineering development processes that Will Larson wrote about in An Elegant Puzzle.
In this model, the overall productivity of an ML engineering team is proportional to the number of improvements they can ship into production – that’s what improves the production models which help the business metrics. (As an example, one of our customer teams was working on fighting fraud, so improvements to their models directly helped safeguard Stripe’s customers and block fraud)
Even though this was an overly-simplified model, it was a good starting point for prioritizing how we could help our customers. To better anchor it in reality, we took this model to our customer teams and worked to find metrics which would help us track and assess their experience.
A clear point of impact is helping experiments run faster. Even if we couldn’t improve the quality of ideas generated, we could improve the number of them ready to ship by just improving iteration speed. This also naturally led to good discussions about organizing developers’ work: what is a single experiment, and how do we track when it starts and stops? We used ideas from this discussion to build out operational metrics for the team, tracking things like models trained and notebooks created that could help us proxy for the number of ideas being explored by our users.
Getting feature ideas into production faster also jumped out as a great objective. Immediately we started asking questions like “what are the steps that get in the way once prototyping is complete?” This was even a bit easier to measure than experiment rate – the start and stop of a development effort is usually a bit more defined than exploration work, which can drift from topic to topic.
Reducing the defect rate in production sounds uncontroversial. No one ever wants to ship junk, and for products that fight fraud, mistakes are often painfully expensive! But there is often a tradeoff between how fast you push feature ideas into production and how often you have to rollback production changes: removing steps so that you can ship things faster can result in more outages.
This is painfully obvious when stated out loud. But it’s important to keep in mind when you are choosing projects to improve ship-to-production speed that there may be a tradeoff. Using both metrics together is a good example of a paired indicator, which can warn you if you are over optimizing for one goal at the expense of your overall productivity. (a great idea but hardly a novel one – Andy Grove wrote about this idea in the classic book High Output Management)
One last area we looked at was the detection rate – improving the speed with which we were able to identify model outages and regressions. While true outages were extremely rare for our system2, there were a decent number of model regressions that took time for engineers to notice. More sophisticated monitoring – for example, tracking the data distribution of input features and model predictions to identify statistically significant shifts – could directly translate into faster fixes and better performance.
Once we had a rough set of metrics together, we were able to look back at the model and reason about what the greatest bottlenecks were. In our case, it was pretty clear that the time to ship feature ideas into production was by far the biggest blocker. Our existing production stack was extremely focused on minimizing production defects, but not optimized for fast deployments in the same way.
It was time to go back to our roadmap and make tough decisions. We cut back our plans to improve flexibility in a few areas, and instead launched an ambitious project to speed up model deployment by decoupling some precomputation steps. Since there were multiple areas we could improve in parallel, we also scoped out some projects to improve experiment speed via improved tooling and notebooks.
To staff everything at once, I worked with leadership to find more headcount for the team – our model made it possible to demonstrate the business impact we’d get from launching these projects faster. In my experience, it’s very difficult for an internal tools team to be able to demonstrate this kind of ROI, and that can make getting investment a challenge. If you find yourself in a similar situation, I highly recommend you build a model like this one and use it to talk through your team’s impact!
By experiments here I meant all forms of data exploration and analysis, e.g. “let’s see if adding previous_login_count is a useful feature”. I should have come up with a better term to avoid confusion with production experiments (i.e. A/B tests). ↩
Much to the credit of the great engineers at Stripe that I was fortunate enough to work with and learn from. ↩
Sure, the best managers do much more than this, but these two things are the most important because they have to come from you. A team of great people with a shared vision can figure out how to overcome many obstacles in their way independently.
This isn’t intuitive — I’ve known a lot of managers who would describe their primary mission as “unblocking the team” — but that’s actually less important! As a manager, your first focus needs to be on giving your team the things that only you can give them.
If you want to build a ship, don’t drum up the men and women to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.
– Antoine de Saint-Exupéry, maybe?
Your vision is what your team will use to decide what to work on and how ambitious to be. If you are an engineering team with internal customers, like an infrastructure or platform team, your vision will tell those stakeholders what they should expect from you in the long term. And your company leadership will look at your vision to judge how important it is for the business – and how much they should invest in your team for the long run!
To articulate a great vision for your team, you need to identify where things are today, where they could be or should be instead, and ideally how you’ll get from here to there.
Critically, you have to avoid getting too attached to the current state of the world when doing this. I think this is a trap that many engineering managers fall into – and I’m particularly vulnerable to it! As engineers, we want to make realistic promises. Our instincts, cultivated over the course of a career as individual contributors, steer us away from fluffy promises and pie-in-the-sky nonsense and plant us firmly on earth, with incremental milestones we know we can deliver with proper investment.
Now that you’re a manager, though, you do need to be ambitious! Your story of how things should be has to be so much better than the status quo that it becomes a motivating and exciting future. It’s better to set an ambitious goal than it is to be certain you can deliver. You can admit that you’re setting an intentionally audacious target, but make sure you choose something that is actually worth striving for!
You also have to make sure you’re choosing the most important work that you or your team could be doing. Often management is about making difficult decisions with limited resources, and a good way to keep yourself honest here is to ask yourself how your company leadership will evaluate your impact. When you articulate your vision, remember that it’s probably not enough to talk about technical endpoints, like your API getting faster. Find the business metric that really matters. If it’s customer churn, make your mission about reducing customer churn by making the API faster. (If customers are churning for other reasons, you should figure out if there’s a more important problem for your team to be working on!)
Once you know what your vision for the team is, it’s time to figure out what team it will take to get you there. This is way too complex an area for one blog post, of course! But when starting with a new team or a new vision, it’s a great to time to sit back, take a look at the team as a whole, and ask yourself some high-level questions.
Here are some of the things I’m thinking about when joining or forming a new team:
Do we have the appropriate skill sets?
If you’ve signed up to deliver a great user experience and no one on the team has done any frontend work, you might have a gap to fill. Sometimes folks on the team will be excited to dive in and learn a new area, and you’ll be able to give them the time to spin up. Sometimes you’ll need to find someone with more experience to add to the team. (Evaluating an expert outside of your own areas of expertise is playing the game on hard mode. See if you have a network inside or outside of your company that you trust to assist you.)
Do we have a good mix of experience levels?
You will almost certainly benefit from having senior engineers who can design large, complex projects based on their past experience – or folks who can take on a tech lead role, scoping and coordinating projects. Remember also that having a mix of experience levels can make the team stronger than an all-senior group – often a junior engineer asking why is the impetus for making something better! Just be sure you build the psychological safety that asking this question requires.
Are we building a diverse and inclusive team?
If you aren’t consciously focused on this, it won’t happen by default. To get started, take a look at Project Include’s recommendations – although they’re targeted at startup CEOs, there are items that you’ll want to think about when hiring, giving feedback, or sponsoring folks for promotions. Teams which are diverse and inclusive are stronger in the long run, and it will be much harder to build such a team if you don’t think about this at each step of the process.
What challenges do people need to grow?
It’s worth thinking about each person on the team and what the next growth opportunity looks like for them. I really like how Lara Hogan described it: sometimes “the best gift you can give your direct reports is a messy, unscoped project with a bit of a safety net”. If you’re not challenging the team with projects that are slightly out of their comfort zone, you’re probably not giving them the growth opportunity they need – and your team isn’t as effective or productive as it could be.
What is the current team dynamic, and is it healthy?
I really like Tuckman’s stages of group development as a model for thinking through team formation. While this model is more than half a century old, the forming–storming–norming–performing cycle is useful for contextualizing some of the common challenges that teams face when getting started or changing their mission. Think about what the team needs to help establish healthy operating norms – in some cases, you can prompt the team to create some of those norms themselves.
Of course, this is barely scratching the surface of what a highly effective engineering manager does. You’ll want to think about how to measure your team’s impact and progress, how to communicate with your stakeholders, how to make sure that your goals are aligned with those of the company, how to recognize and support high performers, even how to help set technical direction without getting in the way…
But all of that can come later. Start by setting a vision and getting the right team in place, and you’ll be giving your team what they need from you most of all!
]]>