A clip from our conversation got millions of views. In it, Ethan says:

“One of the hardest things for people to understand is I’ve identified a legitimate weakness in my boss. I go to my skip. Why doesn’t he do something?

Although many people were disgusted, Ethan was a high performer at Amazon and shared the incentive structure with unusual transparency. He’s not commenting on what is right or wrong. He’s saying what is.

From my experience, as my career grew I worked with more high performers at Meta. Some people play the game selfishly while others play the game while also focusing on what is right. But common among all high performers is they understand “the game” and its incentives.

When you understand the incentives, you can focus your limited time on what is rewarded. When I was a junior engineer, I spent so much time working on random code cleanups and miscellaneous projects that were not impactful. Later, I still worked hard but I focused on projects that mattered.

In the example where Ethan is the skip-level manager and an engineer has a problem with one of his front-line managers. Ethan explains how the incentive structure naturally leads him to not prioritize the needs of the engineer.

You can hate him, but he’s just playing the game. You probably just hate that game.

If you hate the game

I wouldn’t try to change the game since the effort is usually not worth it. The best move is often to find a different team, role, or company.

It’s unfortunate, but it’s the reality. The dream is a job where the incentives match the work you care about. Early in my career, I got lucky. I loved coding and optimizing infrastructure, which Meta rewarded well. Those situations do exist in tech, but you may need to do some searching to find them.

I asked Ethan how to avoid office politics, and he summed up your options well.

One more thing: if you hate the game but try to play it based on how you think the incentives should be, you will get burned. Best case, you’ll do okay but not as good as you could have. Worst case, you’ll get fired for being difficult or doing work that doesn’t matter.

As Ethan said, “do yourself a favor and get good at this [politics].”

Full conversation is here if you’re interested:

I plan to write more about the most interesting takeaways I learn from my podcast guests. I won’t write every week, but I will write whenever I have something interesting to share.

A few personal updates for those following what I’m working on:

1. Keyboard – We launched on Kickstarter and hit our funding goal in 8 hours after launch, thank you if you backed it 🙏

2. Podcast – It is still cash flow negative, but I tested a sponsorship with my friends Evan and Stefan from Hello Interview (it’s here if you’re curious). It went well, so I might try more in ways that don’t take away from the content.

Also, I’m thinking of making this newsletter’s branding more general. That way I can write about anything interesting, not limited to only software engineering career growth. If you have thoughts on this, let me know!

Thanks for reading,
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:02:33 - Empire Building

00:31:54 - Stealing scope

00:44:03 - Managing out via reorgs

00:51:01 - Bad managers and mutiny

01:01:43 - Political messaging

01:11:48 - Handling politically skilled operators

01:20:01 - Orgs trying to steal scope

01:30:26 - Handling difficult people from other orgs

01:36:19 - Handling weak managers

01:46:30 - Backchanneling

01:52:04 - Influence without authority

01:58:04 - Sexual harassment

02:00:49 - Skip overruling firing

02:05:39 - How to fire managers

02:11:31 - Leverage when people are getting fired

02:24:01 - How to grow past senior eng

02:43:01 - How to avoid politics

02:48:15 - Advice for younger self

Transcript

00:02:33 — Empire Building

Ryan:

[00:02:33] The first topic I want to go over is empire building.

Ethan:

[00:02:39] Empire building, I think, is a bad thing in that, as defined, it means focusing more on getting raw number of people than on what you’re doing or the impact you’re having. So why does it exist? Empire building exists because it’s rewarded. And it’s rewarded because counting people is the easiest thing to do. And so when a leader or a set of leaders are deciding who’s ready to promote, how much impact you had is subjective.

[00:03:13] One person thinks the project was hard, another thinks, oh, that was pretty easy. But no one can debate, well, Ethan has 42 people and Ryan has 17. Ethan is the bigger leader, is the conclusion they’re making. And that encourages empire building because it’s what’s visibly measurable. And so what I found is that drives leaders to want more people because that’s going to be their tool. And some companies absolutely have written thresholds of to be eligible for this promotion, you have to have this many people.

[00:03:53] So of course, if you give high performers a goal, they’re going to figure out a way to hit that goal. And if that requires claiming they need people or seeking to take over other groups, they’re ambitious, they’re going to find a way to rationalize that to themselves. I need those people.

Ryan:

[00:04:11] I saw that there’s this desire to change the incentive structure, at least at the company that I was at, where managers could be promoted, not necessarily just directly proportional to how many recursive reports you have. Is that possible to do?

Ethan:

[00:04:27] Of course it is possible. It takes more effort because impact is harder to assess. Right. The impact an engineering manager has is somewhat, somewhat fuzzy. What’s most important, is it operational performance? Is it new features shipped? Is it whether or not those new features resonate in the market and make money and you end up with difficulties. Like maybe my product directly generates money, whatever I’m doing gets directly sold and customers pay for it.

[00:04:57] And what you’re doing is infrastructure. Well, now, how do we compare that? Like, you’re keeping the systems up, but I’m actually the one bringing in all the money. Who has more impact? Again, that’s where people get sucked into. Why have 42 people and you have 17? I must be a bigger manager. That’s how that happens. It is, of course, possible to make value judgments, but then they’re also harder to defend.

[00:05:26] I have to look at you and say, ryan, we love the work you’ve done and you’re on the right path, but Ethan was more impactful. And that just makes you mad. Probably, unless you happen to agree like, oh yeah, that guy’s a star. It’s just all these things make assessing impact difficult and dangerous and that’s why it doesn’t get done as much as it should.

Ryan:

[00:05:52] You mentioned the written thresholds for promotion and since you were VP at Amazon, I know it’s been some time, but off the top of your head, do you remember the rough written thresholds for how many reports you would need to be considered for senior management director and vp?

Ethan:

[00:06:10] So interestingly, those thresholds did not exist while I was there and they existed in whisper numbers, but they were not written down. They came about after I left and people I was coaching, still at Amazon or talking to told me very clearly that the number that got established to become a director in some orgs was 80 people and in some was 90. So I became a director in the early days of the company with 22 people because it was different times and the company was smaller.

[00:06:42] And that crept up and up until when I was at the end of my career, we did have like a sense that the company wanted us to look for people with about 75 or 80 people, but it got written down as 90, at least in some organizations. And the thing that’s funny about that is one of the Amazon leadership principles says there’s no bonus for additional headcount. But now that it’s written down that I can’t get to this very well paying level without 90 people, there damn sure is a bonus, right?

[00:07:16] There’s like a several hundred thousand dollars a year bonus quite literally for accumulating 90 people. And I’ve talked to leaders who. Their boss trying to get them across the threshold was like, we’re going to put these people under you for six months, check the box and give them back to the person we took them from. Because that’s the game they have to play if they want to, you know, reward a person in their team.

[00:07:41] And of course that’s insanity where we’re moving people around and changing their lives just to hit a number.

Ryan:

[00:07:49] But if you did that, let’s say I’m a VP and I have a very loud director or very loud senior manager who wants to become a director. And so I go, okay, to retain them, I’m going to give them some reports from over here. And now they’re going to be eligible for director after they get promoted, though, they need to sustain those reports to sustain that position. Right. Like you couldn’t just move them back.

Ethan:

[00:08:15] I mean, you can again, it depends on the company. But you can play all sorts of shell where for example at Amazon if you drop below the number of reports, there’s like a grace period, six months, nine months where if you can tell a story about how they’re going to get back to that level or how future growth, it’s okay if they’re below it temporarily. The other thing is no company, no company I know of really systematically down levels people, what they would do is they’d say okay, well Ryan, you’re a director, your team is no longer director scope.

[00:08:51] You’re going to have to find a director level role. And so then it’s up to me if I can do that in my org or I move you. But I’ve still achieved the goal for you. And you absolutely see I have seen not just in Amazon but other places when headcount is the main issue. I’ve seen things moved around and I certainly when people normally come to me, it’s with the complaint. My boss actually told me that he’s putting me under my peers so that my peers team will look better.

[00:09:22] So the reason I’m being reorged under this other person is not for any business reason, but because they don’t have the scope to get the promotion. And so I need to go be their scope. That’s what I’m being told.

Ryan:

[00:09:38] When I was an engineer and reorgs were happening, I think you don’t know all the behind the scenes of how orgs are managed and so you’re told there’s a certain reason for the reorg however, and which may be true, but there may be a secondary an ulterior motive of this person also needs to get promoted. Do you ever have a experience like that where you saw reorg happening in other org you didn’t know the exact person who needs to get promoted or what, but you could just tell based on how it happened and what happened that oh, there’s something else going on behind the scenes.

Ethan:

[00:10:15] Yeah. And I would say there’s always something else going on. Meaning all managers know, all leaders know that too many reorgs cost two people and so you don’t want to do them too often. What that means though is like you wait as long as you can and then when you need to do one, you’re asking yourself how do we get everything we want? How do we, how do we. If there’s someone we don’t like, how do we give them less so they’ll move on?

[00:10:42] If there’s someone that we really think is high performance, how do we set them up for success? How do we retain key people? And yes, hopefully the start of the reorg is how do we make the business better? But as soon as we figure out what we think we need to do for the business, then we’re trying to figure out how much else, you know, this ship is leaving the dock. How much else can we throw on the deck as it’s pulling away that’s going to help us.

Ryan:

[00:11:07] I just trying to figure out the exact mechanics, like maybe a pastry org that you’ve done, where you tried to cram a bunch of different ulterior things into it. And how does that actually play out?

Ethan:

[00:11:20] Well, a couple things happen there. First, there are usually multiple people involved. And so sometimes you have to either get them on board with your second motive. You know, you have to tell them, like, yeah, I want to help this person get promoted as a part of this reorg, or I want to set them up. Are you supportive of that? Sometimes you have to tell a story like, no, I’m not doing that. It’s really this reason.

[00:11:47] And so there is some cleverness to justifying because even for a senior leader, usually a reorg is a big enough change that somebody will be auditing it, and you’ve got to be able to explain it to them. So you’re coming up with your narrative. You’re a storyteller, and you’re coming up with your narrative about why this is the best possible reorg that rewards the deserving people and accomplishes the business goals.

[00:12:13] And if anybody is suffering, you’ve made sure that that’s people that aren’t the highest performers. So if anybody does leave, it’s okay. And when I think about specific reorgs I’ve done, they’re very much about retention. If you’re making a change, you’re very much thinking, people don’t like change. Who is it I absolutely need to keep and who is it? If they leave, I’ll either be okay or even maybe happy.

[00:12:45] And I can’t immediately think of a place where I personally put someone in a role where I’m like, and by putting them there, they’ll quit. But I can absolutely think of the reverse where people I really wanted to keep, maybe who I felt I was at risk at losing, I was using the reorg to give them something juicier.

Ryan:

[00:13:06] So then how does it usually get started? I mean, a reorg, let’s say. Yeah. What’s the triggering condition?

Ethan:

[00:13:14] Well, usually the triggering condition is a change in the business. So something about the business has changed. And the organization is not set up to address it, so it needs to take on a new market, or you’ve acquired a company and you need to blend them in or you want to start a new project. That’s the first reason for a reorg’s change in the business. The second reason for a reorg is somebody else quit or got moved.

[00:13:41] So often reorgs come due to cascades. And many times, for example, several times in my career, my boss either left or was moved to another job, and so I had to go somewhere else. I could go up if they thought I was ready, I could go under somebody different, or they could choose to backfill, you know, just leave that position open. But at least in my personal experience at Amazon, I never really saw the backfill happen.

[00:14:14] I always was moved under another leader, and several times I’m pretty sure I was put under them to stretch them. When I ran the prime video team, my leader left and I was reorgang from being under an engineering VP to being placed under a business leader. And that business leader, who turned out to be one of my best managers ever, some of his first words to me were, I don’t really want your team.

[00:14:47] I don’t want to run engineering. I don’t know anything about it. And of course, as an engineer, I’m like, oh, but he was a good leader. And he said, we will figure it out. And we worked together and actually had a great run. But I had that happen a couple times where I was put under people, and it was, they had never led engineering. And I was like, well, let’s give him Ethan’s team and let him figure out engineering, which could be really demoralizing.

[00:15:14] Right. Like, okay, here’s the person who’s not technical and I’m their guinea pig in conducting a reorg.

Ryan:

[00:15:21] You mentioned that you’re the storyteller and you’re building a narrative. And you said in some cases you’d be direct and you’d say, we’re trying to retain this person, so we’re going to move things around. And in other cases, you said you, you would tell a story that this is solving a problem for the business and that that other part would be hidden. It’s really interesting to me because there’s.

[00:15:48] It’s almost like this hidden chest going on.

Ethan:

[00:15:50] Absolutely.

Ryan:

[00:15:50] You’re aware of there’s multiple things and you’re aware of the messenger of this is what this person wants to hear. And here are the things that I’m trying to achieve. What pieces of what I’M trying to do. Can I sell to them to make the reorg happen? And also you’ll get maybe other things as a result. And I think that turns off a lot of people, especially engineers. They go, I don’t want to be managing all these things.

[00:16:18] I just want to say it as it is. Is that just a inevitable thing that you have to deal with if you’re in management?

Ethan:

[00:16:26] Well, inevitable, you can certainly work around it. Meaning you can just do a reorg purely based on business goals and not let any concern for people enter. But probably that’s going to cost you people because some people are more ambitious than others or just more demanding. And there is some truth to the old saying, the squeaky wheel gets the grease. This is another harsh truth people don’t like.

[00:16:57] Sometimes, let’s say two people perform roughly equally, but one of them has made really clear that they’re thinking about leaving. Well, a lot of managers will ask themselves, you know, Ryan’s this great guy, always sticks with us. He’ll wait another six months. And I can use those six months to save this other guy who’s also valuable. And by being quiet and dutiful, sometimes I can make the decision.

[00:17:30] It’s not that I mean to screw you. It’s more I’m focused on saving this other person. And I know you’ll put up with it, and so you can end up behind just because you’re such a nice guy. And I often tell people when I’m coaching them, I will often tell my clients, you are a very nice person. And when I tell them that, what they don’t know is I’m then going to say, and that is not a compliment in this context.

[00:17:58] Right? Again, I don’t think you need to be a jerk. But jerks up to a point, pushy people get more. Now they can push too far, and then they get less because they’ve antagonized. But if you never ask for anything, you’ll never get anything. And so in a reorg, you ask about what’s inevitable. Well, what’s inevitable is I have some people who have been more vocal about what they want in their career than others.

[00:18:30] Perhaps I’m arranging the reorg just because I know what some people want and I don’t know what others want. And so I assume they’ll be fine. That’s the most benign version. It may be even more that for some people I know they’re very pushy and demanding, and for some people I know they’re not. And so I’m Serving the pushy people because then I get to keep my whole team.

Ryan:

[00:18:55] So it seems like almost always it’s better to be pushy and forthcoming with what you want in your career. But there is an upper bound though where you go too far. And so to, to empire building, for instance, a concrete example. I mean, have you ever had someone who is a manager underneath you and clearly asking for reports and you could, you could see immediately that you don’t need these reports you’re pushing because this is going to lead to your eventual promotion?

[00:19:24] And how do you handle that as a manager or as a senior leader when you can see someone’s being too pushy and it’s obviously not good for the business and not defendable?

Ethan:

[00:19:34] Yeah. So I think a manager, if I’m in the leadership seat first, I better have a good trusting relationship with that person if possible, where we can have a heart to heart and I can get in a room where they don’t feel anybody’s listening or watching and say, look, let’s talk for a minute why you really want these people. Okay, let’s both be honest that you’re empire building and I get that you’re trying to get to a goal, but this is a bridge too far.

[00:20:00] Let’s talk about how I’m going to get you there or how we’re going to get you there by some defendable path or some other path. And so as opposed to tell them just no, you can’t have this, which will be very frustrating and they’ll probably argue, I try to bend the conversation to let’s problem solve together. What’s another way to do this? And I won’t say I’m always successful. Some people want what they want right now and they can’t see or they have zero patience to wait.

[00:20:30] And at that point, again, that’s that fine line. If I can, I’ll share a related example that I know people resonate with where one time I wanted to tell my manager that I really expected to be promoted, but I wanted to do it in a diplomatic way that couldn’t get me in trouble and wouldn’t irritate him. Not just couldn’t get me in deep trouble, but like would bring out his support even though I was threatening a little.

[00:20:54] And so what I did is with this manager, I wanted to be a director at Amazon. I was a senior manager. It’s when I had the 22 people I referred to earlier, I went to him and his name was Neil and I said, Neil, I need to understand how important My career is to Amazon because my career is very important to me. And see, that’s a statement no one can argue with. No one can argue and say, well, your career shouldn’t be important to you, or it’s illogical.

[00:21:25] So I said, my career is very important to me. I need to understand how important it is to Amazon, because if it’s not as important to Amazon as it is to me, I need to think about that. And, you know, I need to think about that isn’t a direct threat. But you laughed. Everyone understands that I’m kind of serving notice, but I’m not doing it in a way that is easily easy for him to push back on. See, if I say, well, if it’s not as important to you, making it personal with him, it’s not important to you as it is to me, then I might quit, that’s a threat.

[00:22:03] And all managers are going to be like, hey, you can’t extort me, right? That isn’t how it works. So I came up with these words that clearly conveyed the message, but in a way, you can’t argue it’s important to me. You can’t argue really, that if it’s not as important to Amazon, I didn’t personalize it, that I have to think about that. So I was able to put it on the table. And, you know, he promoted me, right?

[00:22:36] He pushed the promotion through. And I think there’s always a way to say things in that middle ground that makes clear what you want and makes clear that you’re willing to consider your other options, but without being a threatening jerk about it.

Ryan:

[00:22:55] That wording is, it’s, it’s kind of. I mean, because when you say the wording, I feel, I mean, if I was. I feel the effects of extortion a bit, right? But I couldn’t, I could not label it extortion. If I went to someone else and I, I went to my management, I said, hey, Ethan told me this, they would, they would probably also immediately get, oh, there’s a, there’s a threat here. But I don’t think anyone in any room would say, this guy’s being mean with it or we need to get rid of him.

[00:23:30] They’d say, that’s a well done way to present yourself, right?

Ethan:

[00:23:35] That’s bold. He’s clear on what he wants. And that’s, you know, one of the advantages. Do find a place where you fit. Meaning Amazon was a relatively sharp, elbowed, pushy place. It has that reputation today. It was true long ago. You have to tune Your message to. To the workplace that you’re in, that message would be offensive in some workplaces, and it was totally not offensive at Amazon because there were people sort of pounding the table of, I need to move up now.

[00:24:10] And, you know, so. So part of it’s just finding the place where you look like a wonderful, sweet person in the mix.

Ryan:

[00:24:19] You mentioned that there’s this. This dichotomy, I guess there’s these people who are team players, I guess, and then there’s these people who are more pushy with what they want, and they’re, they’re very forthcoming and obviously that has benefits for their career. I’ve heard people tell me that when they want to express to their manager, oh, I want to get promoted, or something like that, they have this feeling that they don’t want to be too pushy because it could irk the manager.

Ethan:

[00:24:50] Correct.

Ryan:

[00:24:51] Imagine you’re a high performer. You’re a high performer, but also you keep demanding things of your manager that could also go wrong depending on the manager. I’m curious your perspective as a senior leader at Amazon when you had these demanding high performers. Is that something that you were. Were you happy to service their requests? Or was this. Did it ever irk you? And you said, I’m going to actually not give you what you want because you’re a little too.

[00:24:55] You’re a high performer, but you’re also too demanding.

Ethan:

[00:25:25] So I want to tell you a couple of things about that. Yes. First, the great thing about the really demanding people is if they really are bothering you to the point where you decide they’re a net negative, if you just tell them no, they will quit because they, they are so driven about getting what they want that if you deny them what they want, your problem will go away. Shortly. They will even leave for a lesser job just because they believe, well, at this place, I’m going to have different results.

[00:25:53] So, yeah, you, you can calculate and say, okay, what? There is some calculus that says what answer gets me the result I want. And if someone’s too pushy and not enough value telling them, no, you can’t have that, they will then go away. The second thing, though, is I tended to try and make a deal. I’m a huge believer in deal making, and I made these deals with my own leadership, too. And the deal essentially was, okay, you want this thing, I will help you get it.

[00:26:23] But you’re a high performer. You’re going to show incredible performance on the things I need, and then I will deliver. And there are managers of Course, who make that deal and don’t follow through. I can say with total pride that I feel I always followed through. There’s actually one exception early in my career I’ve shared where I failed to do that. And I still feel bad about it 20 years later.

[00:26:51] But I make a deal where I say, okay, you want this? Here’s what it’s going to take. I need this and this and this. So as an example, I can give you. My manager gave me a requirement I really didn’t like. He was my vice president, I was a director at the time. And he told me, your team is too expensive. We need to put a bunch of it offshore in India with cheaper labor. You need to open. Well, he didn’t even say India.

[00:27:17] He said, you need to open a low cost offshore dev center. And of course I didn’t want to do any of that. Like, that meant traveling around the world and having teams around the world. And I didn’t know how to do it and where to do it. He left it completely. He made the whole problem mine. He didn’t say, go to India, he said, go somewhere. And so the long and short of this is I found a senior manager I knew who wanted to get to director.

[00:27:39] He was Indian, living in the us And I went and said, hey, would you go back to India and start this dev center? If you can build me a dev center in India, I think we can use that as a platform to get you to director. And that’s exactly what happened. He moved home to India. He got to take his son home to learn Hindi and see his grandparents. He moved to India for three years. And during that I was then able to use his independent action there.

[00:28:08] And the fact that he was doing all this stuff remotely and to the point of narrative, I was able to create a story about how this was clearly a director level accomplishment and he was promoted to director. But I made that deal years before it happened. You go to India, I will get you to director. So that’s what I try and do with the demanding people.

Ryan:

[00:28:30] Well, I can imagine also a lot of things could change over the course of a few years. Maybe you’re at a different org and you had that guy went to India and then you’re in and he’s no longer reporting to you two years later. And you can’t deliver on that. But I mean, you can’t make decisions based off that.

Ethan:

[00:28:48] No. If I had been reorged or fired, you know, it depends, right? Like there are things that could have happened where I would have had to say to him, look, I’ll do what I can to support you from the new role. But it isn’t I didn’t do what I was saying, I. It’s that I no longer have the power to and I could feel all right about that. I can’t immediately think of a circumstance where that happened to me where I wasn’t able to follow through for someone.

[00:29:17] But I can see where that would happen. The good news is people are, most leaders are professionals in that they under, they can see the difference between you didn’t come through and you know, shit happened. Those are two different categories.

Ryan:

[00:29:37] Yeah, it’s interesting how much back of the envelope I guess deals there can be between managers and reports or I’m sure between managers as well. Like maybe let’s say you’re trying to get someone promoted in your org and you’re working with your manager on making that happen eventually. And he says, he or she says okay, I have someone who’s even more urgent now but that one’s going to happen in two halves.

Ethan:

[00:30:13] there’s absolutely a line meaning by a line I mean a line up a queue where even HR has a policy in big companies to say what does the forward looking slate is what we call it, the forward looking slate of promotions. And they would track and say by halves who’s in 6 months, 12 months, 18 months, maybe as far as 24. And you could list different names and so there was a sense of like whose turn is it we’re going to do?

[00:30:43] Not that there was in my time a quota. I understand for example at Google today talking to someone I coach there, there’s a strict quota like this quarter or this half two people can get from LA to L9 in this org. And you know the guy I was coaching is like I think I’m number four, I’m not going to make it. And again that can be toxic a little bit. But people understand the idea of there’s a line.

[00:31:14] And so absolutely you have those discussions now again for early in your career there’s usually no quota to go from like entry level, college engineer, whatever that’s called where you’re at to sort of regular, you know, engineer, engineer maybe even to senior engineer. Things get to have more of this quota system when the positions are rare and expensive. And there is a question how many staff or principal engineers does this company Need?

[00:31:45] How many distinguished engineers does it need? By the way, those positions cost a million dollars a piece, so we don’t just want to hand them out like popcorn.

00:31:54 — Stealing scope

Ryan:

[00:31:54] So you mentioned when we were talking about empire building, there’s this notion that managers career progression is tied to the number of their recursive reports. And so on the one hand there’s an incentive for managers to say I want more new headcount. But I could also see there being cases where managers want existing headcount. So maybe someone underneath you is vying to take reports from someone else and basically steal scope for their own promotion.

[00:32:28] And as since you were such a senior leader at Amazon, I’m curious, how did you handle cases where you noticed somewhere in your org, someone is trying to steal Scope from someone else in your org?

Ethan:

[00:32:40] So first I would say this has become more common as growth is slowed. In other words, when a company is growing very quickly, as Amazon was through much of my tenure there, you see less of this because it’s easier to go get new people than it is to fight with someone and it causes less bad blood. Not that I never saw that. But now that companies aren’t growing, you know, if you look at the Magnificent Seven or however you look at them, most of them are flat headcount, couple exceptions.

[00:33:13] There’s a lot more of this, what I would call cannibalism. Since I can’t go out and get new, I’m going to have to eat what’s here. And that, that’s very cannibalistic. You see a lot of maneuvering for that. How did I deal with it? You can’t allow people on your team to go rogue and be pirates in your own organization. Like capture somebody else’s ship and you know, kill the captain, sail away because of what it’s going to do to your org, you’re no longer in charge then they’re in charge, right?

[00:33:45] They’re raiding and pillaging. And you seem clueless, so you can’t allow that. But what you then have to do is either find them a path, convince them to wait, or frankly, if they turn to septic, they become your answer, meaning they’re going to leave, or you get them to leave and you give their team away. So if it really is cannibalistic, what happens is if they don’t pull off, you know, it’s, it’s like any coup attempt, I guess, in, in like politics, if you attempt a coup and you win, you get to be president or, you know, whatever.

[00:34:19] And if you fail, you get hung or shot. And so this is a little bit, you know, it’s not quite that bloody. But one thing that holds back the worst behavior is people do realize if I get caught being too obviously self serving, it’s going to burn me here. And so they usually are pretty careful. But do they want it? Sure. People all the time are asking, could I take over this, could I take over that?

[00:34:48] What I encourage people to do is I tell them, look, you never know when a reorg’s coming. You never know when somebody’s going to quit or get fired or laid off. So talk to your manager and say, hey, if we have an org change and there’s an opening, I just want you to know I’m ready to step up, I’m interested and I will stand by you. This is one thing people want to know in reorg is reorg, shake the ship.

[00:35:15] And so some people are like, oh, this company’s going down or it’s not stable, I’m going to quit if I can. If you tell your manager what they don’t want to do is put someone in a role who’s then going to leave. And so you tell them, look, I’m actually looking to stay. So when you need a stable leader, think of me. And then when that opening comes up, they’re like, oh, right, Ryan said he’d stay. Perfect Ryan.

[00:35:39] Ryan just became most qualified, not because he’s best, but because he’s most reliable.

Ryan:

[00:35:44] Right, I see.

Ethan:

[00:35:45] And that is best in. In my situation.

Ryan:

[00:35:48] So that’s what. And what you just recommended there to me almost sounds like a better narrative for asking for scope. So of course, yeah. I mean, rather than saying, give me, give me this, you can say, I want what’s best for the org and if I need to step up, I’m here, but without being the pirate. And you might actually get what you want.

Ethan:

[00:36:14] Absolutely. And that other piece of the messaging you stumbled on is really important, which is, of course we know you want this so that your org is bigger and so that you have career growth. But there’s a polite fiction that can be true, which is I also want what is best for the org. In other words, I understand that my best path up is to do things that are valuable to you and to the organization.

[00:36:41] And if I tell you, look, when a reorg happens, doesn’t matter what the team is, if you have a need, I’ll be there for you. That sounds very selfless and org centric, even though the underlying message could be phrased as I don’t really care which team’s available. I just want more scope. I’ll take whatever you have. But the way I phrased it sounds so much better because one sounds like the pirate ship and the other one sounds like the selfless servant.

[00:37:09] But they’re both really, hey, whatever’s available, I’ll take it.

Ryan:

[00:37:13] That phrase you said, the polite fiction, I feel like that’s the thing that I could feel as while we’re talking about it too. It’s like if it feels weird, it feels uncomfortable, it feels conniving almost, because it’s, it’s. It’s. You have. You secretly. You have this, and also everyone else knows you have this, but you say this, but you. I mean, both can be true.

Ethan:

[00:37:38] That’s, That’s. Yeah, that’s the key is it can be conniving, but it doesn’t have to be. They can be honest. I can go to my manager and say, look, I want to grow, and I will do whatever you need in order to grow. Like, whatever’s available, I will take it and do my utter best on it. And that can be both things. One story. When I was a director working my way towards vp, I once again had been put under a VP who had no other engineering leaders.

[00:38:11] But then they kept giving him more engineering stuff, and I was the first, and I built trust with him. And he just automatically came to me with everything, like, well, they handed me this. Can you take it? And at one point, I was managing, I think, eight different functions, and they were as diverse all over the place. I’ll give you three examples. I led the technical team for reverse logistics, meaning when people traded in video game cartridges, we had to grade them and sell them used.

[00:38:43] I was an advisor to the team that was grading returned video games or used video games. I was selling new video games, like software downloads, video game downloads. And then I was starting to build Amazon’s app store. That’s three examples. A fourth example is I also owned our B2B business where, like, Amazon was trying to displace and go into the market of Office Depot and, like, ship pallets of paper to people.

[00:39:15] And these all had ended up under my boss. I had this, you know, you call it whatever you want, but this random mixture of stuff, but my boss loved it. And over time, that totaled 200 people from all these different functions. And that just came from saying yes to everything. Now, did I really want to work on grading video games or shipping pallets of paper to schools? Not really. So it was the win win, though.

Ryan:

[00:39:43] Well, this reminds me of the offshore India story that you had, which is your manager has work he doesn’t want to do and so he will pass it to you because he’s identified you as the high performer that’ll solve his problem. So it is win win for you because you wanted the larger org and he does not want to deal with these problems.

Ethan:

[00:40:05] Exactly. And so someone I really respect, he’s now a senior vice president at Walmart. He, he said he was taught in turn by a senior vice president at Amazon. The simple way to get promoted is quote, solve problems for your boss. Like if you just need to get it down to five words, solve problems for your boss. If you’re solving their problems, they will value you. And if they value you, if you then express what you would like, they don’t want to lose you.

[00:40:37] Because not everyone’s helping them that way. Most people are not. They’re just punching the clock or fighting to do their own things. And so even a so called bad boss will often take care of someone who’s helping them. Because even if their calculus is, well, imagine I’m evil and I don’t really care about anybody but myself. But I’ve got this team of people and some of them are helping me and some of them aren’t.

[00:40:59] Who do I want to keep? Even if I’m totally selfish, I want to keep the ones that are helping me. So even then it can work out.

Ryan:

[00:41:07] What about the case where, because in engineering there are cases where we can create scope where someone goes off, they find a problem that we didn’t know existed and they build a new system that changes the industry or does something really impressive and then the impact they had is just incredible. But they didn’t necessarily solve a problem that that boss specifically came to them with.

Ethan:

[00:41:35] Sure,

Ryan:

[00:41:37] that could also work though as well and that manager would push that promo through. But I guess that’s a different path

Ethan:

[00:41:46] and I recommend both of these. What I mean is ask your manager what you can do, how you can help them, but also be thinking about what can you do on your own? Or what can you go to your manager and say, I think we should do this and here’s why. So an example I’ll give Amazon now sells over $1 billion a year of funny T shirts, custom printed. You know, you can put your slogan on a T shirt and put it up on Amazon and buy it yourself or get other people to do it.

[00:42:15] I help start that business and specifically I funded the initial team on it and, and my manager was saying, hey, you run Amazon’s app store. Like, what does that have to do with T shirt printing? This has nothing to do with it. Why are you doing this? And I said, well, I think there’s a big business here, but I need 10 people, and I have 800. So if you’re gonna tell me that I can’t spend basically 1% of my resources on this thing I believe in, we have another discussion to have because you’re micromanaging me like, this should be within.

[00:42:49] And so he’s like, fine, I disagree, but go do. You’re right, it’s only a few people. Go do your thing. And then it turned into a big business. And of course, I’m simplifying a ton when I say, and then it turned into a big business. There were, like, years of work in that phrase, but I was thinking about, what else can I do? What else makes sense that either with my own resources or with lightweight approval, I can go work on?

[00:43:20] And so I think with your manager, of course, you’re trying to ask, what do they need? And that’s what I was doing with 790 out of my 800 team. At 10, I was doing something that he didn’t think was a good idea. But I was able to say, like, well, let me have my 10. He’s like, all right, that, you know, you’re a big enough leader if, you know, you’re now responsible for making these people useful. So I did, and if I had failed, I started projects that also didn’t work out.

[00:43:50] You’ve got to shut them down quickly and, you know, have a good enough hit rate. But I think there’s multiple paths to the top. I tried them. All right, be. Be the helpful person, but also be the inventor.

00:44:03 — Managing out via reorgs

Ryan:

[00:44:03] We talked about reorgs earlier, and you mentioned that it can be used to manage certain leaders out.

Ethan:

[00:44:10] Sure.

Ryan:

[00:44:11] What is that? What does that look like in practice? Like, you give me example.

Ethan:

[00:44:15] Well, what it looks like in practice is you put them on something they don’t want to work on and you claim, well, that was the only seat. That was the seat we needed you in. I mean, it’s. I’d like to claim it’s more complicated than that, but I’ll tell a story of someone who got reorged and I thought they’d quit so much. The manager even thought they were going to quit and was very upset with this.

[00:44:39] When I first joined Amazon, we had a few groups and we organized ourselves, and I owned what was going to be Prime Video, and someone else owned what became Prime Music. And a third person owned what became the Kindle, the E Reader. And then there was a guy who had a clear ownership over the ingestion pipeline. So the way to digitize movies, music, et cetera was all a unified pipeline. And then there was one more leader, and he got given database maintenance, quality assurance, and like, a couple small things.

[00:45:13] And even he joked, well, my group’s other, it’s the garbage can, right? And our VP was like, don’t call it that. You cannot call it that. What about the people in it? Like. And I’m sure he was thinking like, the leader who stuck with other isn’t going to stick around. Well, as far as I know, that guy is still at Amazon 20 years later because he was one of those very calm, nice guys who is willing to do what was needed.

[00:45:40] He was the team player who was going to do what was needed. But the point is, if I wanted to get rid of a high performer, not a high performer, why would I want to get rid of a high performer? But if I wanted to get rid of a pushy, overly ambitious person, just give them other, like, six weeks later, that’s a vacancy. I can open the role right now, right? Because, like, I’m a bit of a confrontational person.

[00:46:04] So when I needed to fire someone, I tended to just go straight at it. But this putting someone in a seat they don’t like is classic. And there’s even a. This is a widely understood. You know, this isn’t an Amazon thing. If you go to Japanese culture, completely other country, there’s this concept called a window seat. I don’t know if you ever heard of this. So in Japanese culture, which is very collectivist and I’m not an expert, so maybe someone from Japan will say this isn’t true, but I don’t think so.

[00:46:37] You want to be in the center of the building. Even though window seats are very desirable in American culture, the hot, you want to be in the middle of things there. You want to be close to the CEO and close to the action. So actually being on a window means you’re not that important. And so ironically, what we would think of as great, being given a corner office in Japan is a signal of you’ve been moved as far from what’s happening as possible.

[00:47:03] You’ve been, you’ve been exiled to the land of irrelevance, where you know you’re not. You don’t even have coworkers on one side, you have glass. And that would be a way to signal to somebody like your career here. Is now over. You’re in a window seat. So this idea isn’t. You know, I don’t think it’s. Some people would think, oh well that’s only Amazon because they have a history of being combative.

[00:47:27] I don’t think that’s true. I think this idea is well known.

Ryan:

[00:47:30] You mentioned that one guy who gets other or the garbage can, he’s the nicest guy or he’s. He’s the team player and this and that. And I, I guess I see the incentive structure that makes it that nice guy gets garbage can. But why? I don’t know. Some part of me is wishing the world wasn’t that way. Sure.

Ethan:

[00:48:06] pressure you can to some degree. First note that the org I described where I had all those random things was also the garbage can. So I built the garbage can, kind of opting into it like I will take all of your junk. And that is my role. He was not doing that in the same way, but there was some of that. The problem with only rewarding the nice guys is number one, sometimes the nice folks don’t actually want more.

[00:48:38] In other words, ambition tends to come out and if people are really ambitious, they do usually end up. Not always, but they usually end up more vocal. Second though, I have the problem that I do have all these vocal people and what they’re going to do usually is not become less vocal just because I have an example of play nice and you’ll move up. They’re going to look for where can I go that’s going to reward me right away.

[00:49:09] And so of course maybe a better leader than I can build this culture where all the nice people are rewarded and all the sort of less nice people see that, see the error of their ways and become naturally generous. But I’ve just. That isn’t human nature. And so I think there is some balancing. What I often do, as you heard from my coaching, and I’ll be honest, is I tell the nice folks being that nice isn’t helping you.

[00:49:37] You don’t have to be a jerk. You have to do what I did. Where you say my career is very important to me. It’s okay, it’s okay to have. Where people fall down is they come particularly in tech from non us cultures that are collectivist or have different mores and where asking for what you want or Speaking up culturally doesn’t fit. I see this a lot, particularly with women from Asia, different parts of Asia.

[00:50:13] At home, they’re expected culturally to take one role, which is often more family oriented and supporting of parents and grandparents and so on. But then to get ahead and work, they need to be a little bit more outspoken. And it’s a big contrast. I would just say it’s okay to ask for what you want that isn’t wrong.

Ryan:

[00:50:38] And

Ethan:

[00:50:40] managers, even good managers, they’re not in the business of guessing. If you’ve never told me what you want, sure, in theory I should ask you, but I’m very busy. And so sometimes I get around to that and sometimes I don’t. Meanwhile, this other person every three days is knocking on my door. I’m super clear on what they want.

00:51:01 — Bad managers and mutiny

Ryan:

[00:51:01] When it comes to engineers, there’s one layer to think of. And so if someone’s low performing, there may be your peer, but if you have a low performing manager, there’s this vertical stacking that we have. You might be reporting to someone who you think is not great at their job. And so the natural thought is go to their manager, which would be your skit, which I guess you could consider this as mutiny.

[00:51:26] If you went to them and you said, hey, my manager is bad. I don’t want to report to this person. Do you have any examples of this kind of escalation coming to you where someone in your recursive chain was considered incompetent by their reports and they came directly to you and they said, please manage this person out, or I just don’t want to report to this person all the time.

Ethan:

[00:51:50] All the time. One of the hardest things for people to understand is I’ve identified a legitimate weakness in my boss. I go to my skip, why doesn’t he do something? Well, if you come to me with a weakness in one of my employees, there is subconsciously this process that goes on that says I have two choices. I can believe that you’re overly sensitive and high maintenance, in which case I don’t really have a problem, you’re the problem.

[00:52:21] And you know, you’re two levels down from me. So if you quit, the manager has to do the backfill. And I can tell the manager, you know, Ryan was here, he said this, that and the other. Maybe you can work with him and that’s exactly what you don’t want, is me ratting you out. But I can make it my manager’s problem. On the other hand, if I agree with you and I’m like, You know what? This manager I have really isn’t that good.

[00:52:44] Now I have three problems. This is really bad for me. One, I have to decide what to do with my manager. Maybe I have to manage them out. Two, if I do manage them out, I have to hire and train somebody else. And three, while they’re gone, I have to do all their work myself. So you can see why, even if it’s subconscious, I have a lot of reasons not to listen or not to believe very easily. So mutiny.

[00:53:09] Let’s talk about mutiny. If you want to see it as a mutiny. We’ve talked about pirates already. Never mutiny alone. If you want that to work, if you really want to say this manager’s bad and we have a problem, take two or three people or send two or three people in sequence. Find some of your co workers who are willing to speak up. Because I will give you an example. It really bothers me. I had a leader who was treating the women on his team like crap.

[00:53:42] Now, he wasn’t doing that in front of me, but it turns out it was true. Well, it took a long time for the different rumors to bubble up to me enough to where I’m like, I’m hearing bad things. And then I went and talked to several of the women and like, so in working with this person, I’m hearing and they’re like, oh yeah. Well then I was like, oh, okay. I’ve been blind to a problem, I need to act on it.

[00:54:15] Had any one of them come to me early and said, this person’s a problem. They’re doing this, that, and the other I probably wouldn’t have listened to because it was a lone report. But when several of those reports came up, then I’m like, oh shit, I’ve been missing something. So if you want to hold a mutiny, go together. Right? Even if it’s Zelda, it’s dangerous to go alone. Take this, take other people.

Ryan:

[00:54:41] I see. Okay, so if, let’s say there’s a high performing IC who is reporting to a manager and say, the fact is that that manager has some shortcomings in their abilities, you would say, don’t go immediately to the skip first. Build allies with some of the other people and say, hey, are you also seeing this with this person?

Ethan:

[00:55:01] And if you can, yeah, you want to talk to them and say, hey, do you see what I see? Well, one thing is that’s a way to sanity check. I’m really struggling with our boss in this area. Do you, Are you having that problem? And if they say no, I get along with them great, and here’s how, then maybe you can learn and adjust. That’s the style thing. Whereas if they’re like, oh, yeah, then the hardest next hurdle is are they willing to speak up?

[00:55:27] Because a lot of people aren’t. But if they know that you’re going to go first and play ringleader, like, I will go talk to our skip, but I’m going to tell our skip they can verify my story by talking to you or, and also to Sally or Fred or Rajeev. Most managers, if you go to them and say, look, I know that my saying the boss is a weak link is uncomfortable and difficult for you, but these three people are all willing to share similar stories, would you consider at least talking to them?

[00:56:05] Most managers will do that. Most, you know, not all. There’s no guarantees when you’re dealing with people.

Ryan:

[00:56:12] I mean, that’s. Yeah, that’s the immediate thing I imagine is, let’s say I was that IC and then I went to other people and we all kind of knew that that person had a shortcoming. It. There’s the difference between that, that water cooler talk and saying, yeah, this person is disorganized. Yeah, he keeps doing that. Compared to, hey, I’m going to go to his boss now and we’re going to, we’re going to make, take this up a notch because also there’s this, this delicate balance which is all these people still report to that person.

[00:56:43] So now if you were to get caught at any point in this process, you might worry about retribution or something if you’re wrong. What about the path of. Because this is a lot. I mean, your imagination needs to be pretty bad if you do this. What about the approach of you say, hey, it’s just me and him, and you want to go to the skip and say, hey, Ethan, you know, nothing, nothing personal with the person that I’m reporting to, but I see a better business case for me to go to another team or, you know, something like that where you word and say, I don’t, don’t think poorly of this manager, but I’ll be more productive somewhere else.

Ethan:

[00:57:30] approaches, the first of which is, remember we talked about the story you tell versus the story that may be true if you can actually make a strong case that you’re equally valuable or more valuable somewhere else, a plausible case. Don’t even bring up the manager. Just say, hey, I was looking at this other role and I think I could do so much more for you in the org over here because of A, B and C.

[00:57:58] And you try that run first, and if that works, you’re done. You never have to bring it up. The second thing you can say, you’ve just. You’ve gotta. You’ve gotta play chess, not checkers, meaning several moves in advance. You can totally say, look, candidly, you keep it blame free. I’m not compatible with our leader. I’m not. Maybe it’s him, maybe it’s me. Of course, I naturally feel that maybe I’m doing good work and he could improve.

[00:58:30] But I’m not here to throw your manager under the bus. I’m just here to tell you I need a change. And I did this once with a leader at Amazon, where I wanted to make a change. I was doing a good job for him in the role I was in. So he wanted me to stay there. And what I had to help him understand is I said, you are seeing this as I take this new role or I stay in this role, and you’d prefer I stay in this role because that’s good for you.

[00:59:01] And I get that. But what I’m trying to tell you is either I take this new role in your org that I’m asking for, or I’m going somewhere else entirely. So there is no choice. Keep me where I’m at. There’s, do you want me on your team or on someone else’s team? Now, I had both options available, and it was really interesting because after weeks and weeks of resistance, I was in the new role the next morning.

[00:59:25] Like, once he got the clarity of, oh, I thought I had a choice between you stay in the role I want you in and you go to the role I don’t really want you in. But the choice I actually have is I keep you in my team or I lose you. It was totally different the next morning.

Ryan:

[00:59:44] But how did you word that in a way where that person you’re reporting to did not feel upset that you kind of subverted their power and forced their hand?

Ethan:

[00:59:55] That’s a good question. And so I can exemplify that. I framed it in terms of what they would do. This person was a vice president at that point. I was a vice president reporting to them. So I was like a smaller vice president, whatever, lower in level. And I had gotten a job offer from a senior vice president in a different organization. That was my other job. So I went to him and I said, I have an offer from this svp.

[01:00:26] Now, you know that if I give my Word to that guy. I have to go. Like, that’s what you would do? If I give my word, it’s over. You know, I really want this other role. I have to give this guy an answer. You’ve told me no on this other role several times. What do you want me to do? Bam. The door. Like, because once he understood, he understood he would never break his word to an svp. So he understood that the moment I gave my word, it was done.

[01:00:58] And so that’s the same. Like, what would you do if you can tap into the manager feeling like, okay, I don’t like it, but I get it. That’s what I was doing is he was like, oh, I don’t like this, but okay, I get it.

Ryan:

[01:01:11] There were three roles under consideration.

Ethan:

[01:01:14] Well, there was the one I was in and really wanted and the alternative I had gotten on the table.

Ryan:

[01:01:19] So you brought in a new option that was better than your current. And you said, my alternative now is I may leave you unless you give me what I want. But you didn’t say that directly.

Ethan:

[01:01:29] I didn’t say that what I want. Instead, I said, I actually, I’m very careful not to make it that threatening. I said, you know, once I give my word, I’m. I can’t go back on that. Like, that’s the.

01:01:43 — Political messaging

Ryan:

[01:01:43] The. The wording. I mean, as soon as you say, I think. I mean, if I was that manager, I. There’s. I’m on your side. Even though I see the. What was the. What was the word? The polite fiction, I guess I see the polite fiction immediately, but I laugh because it’s just so.

Ethan:

[01:02:03] And I think this is a skill I haven’t figured out how to teach people. But maybe this talk can help. Several times I’ve given you these polite fictions, these careful wordings. I think a mistake people make that they can do so much better is think through your wording before you’re in the room. Because most of these conversations are not spur of the moment. You either. They’re either on the calendar, this is my performance review, or whatever, or you at least know they’re gonna come up.

[01:02:36] And so spend the time to have that key phrase ready so that you can just, with true sincerity say, yeah, you know, once I give my word, I can’t go back on that, and I deliver that. I look like I did mean it. In no way was I lying, but at the same time, I deliver it like I’m an angel just fallen from heaven. You know, poor, innocent victim. Both are true. And when I put the screws to that other boss long ago and said, well, I’m going to have to consider if my career is not as important to Amazon as is to me, I’m going to have to think about that again.

[01:03:19] I said that with a warm smile, like, hey, of course I’m gonna have to think about that. And I just look like the kitten, you know, the anime kitten with the giant eyes. Well, body language matters, because if I say that to you, see, I can say that differently. I can say, well, Ryan, you know, if my career is not as important to Amazon as is to me, I’m gonna have to think about that. Then you’re like, well, screw you.

[01:03:45] Whereas if I say, well, of course I’m going to have to think about what that means for me. Totally different delivery, same line, same words. Now, there’s some people watching this, engineers, who are like, I hate you. I hate the politics. I hear you. But this is human nature and human interaction. You’re dealing with other humans and their emotions are at play and their defensiveness and their own stress.

[01:04:09] Do yourself a favor and get good at this.

Ryan:

[01:04:12] Well, yeah, and to me, that, that is clearly your superpower. I mean, I can imagine so many other people in these situations not getting the outcomes they want because they word it slightly differently and it doesn’t come off, you know, the, the way they say it and, and kind of the, the, the interplay about what that person wants and the things that you could say. And how do you do that? Like, if we were to try to reverse engineer coming up with those polite fictions so that people can get more of what they want out of their careers, I imagine this would be useful even outside of your career.

Ethan:

[01:04:49] Oh, it’s all through your life.

Ryan:

[01:04:50] Yeah, all throughout your life.

Ethan:

[01:04:51] It’s all. Any negotiation, any deal making, I call it. I’ve tried to reverse engineer, and we’ll do some more right here, right now. I call it being interpersonally warm. So being friendly, smiling, not getting agitated while being professionally firm, saying what I need to about what I really want, but in a way where I don’t bring anger or you versus me into it as much as I can. And so there are some key phrases.

[01:05:33] A great phrase is help, right? How can you help me? I want to get to the next level. How can you help me? Because everyone loves to think of themselves as helpful. And so if I ask you for help, that’s just as opposed to I ask, what are you going to do for me? It’s that sort of phrasing. So Part of it is the attitude. I read a book I really love. I can share the book. It’s called Leadership and Self Deception, and it’s written by some research institute called the Arbinger Institute.

[01:06:12] But what makes this book so interesting, what it taught me is don’t see the other person as an obstacle to what you want. See them as a human being trying to do the best they can in their situation, who has all kinds of problems. And what makes a huge difference here is when I start seeing my boss, who I want something from, not as well. This person controls my pay or my promotion, and I have to yank it out of them.

[01:06:37] That’s seeing them as a thing. They’re an obstacle, like a barrier across the road. If I see them as a human trying to get by, then I can emote to them and say, look, here’s what I need. What do you need so we can make this work? I want to help you because I want your help, and that’s an honest offer most people can understand, and so it comes from that. How do I see them? I think there’s a huge thing is we objectify others.

[01:07:09] This happens anytime you’re working with another team. As an engineer, you did this all the time. I just need them to do this thing. How can I. Your thought is, how can I get them to do this thing that’s making them an object? I need to give you the right inputs and yank on the right levers so that the right thing comes out. I’m treating them like an API. I make the right call, I get the right result.

[01:07:30] What you want to do is treat them like a person and say, look, I know you have a busy roadmap and I understand you have a lot of pressures, but I do need this. So what can we do together? How can I help you so that you can make the time or let me make this change myself? And the key is, once people feel like you’re caring about them as a person, they usually respond that way. Now, nothing works 100% of the time, and I can’t instantly recall one, but I’ve probably rolled out these polite phrases before and not one.

[01:08:08] Like, I was debating politics with my college roommate, and I thought to myself, this is very recent. I’m going to use all I teach all day about how smooth I am and how influential I am. I’m going to use all my skills and I’m going to convince them of this point in modern American politics, which is very contentious. I failed. The best I can say is that we Left the conversation, nobody angry, friendly.

[01:08:39] But I kind of got nowhere because sometimes their motives and what’s important to them just can’t be aligned with yours. So I’m not. My only point here is these techniques will get you way more success. I don’t want to convince anybody that like I have some magic recipe that’s 100% likely to always work out.

Ryan:

[01:09:02] You said a few things and it sounds like you had some high level rules you could, you could give, but it’s hard to reverse engineer because it’s, it’s not a recipe inevitably it’s, it’s two people interacting and every single person’s different and seems like the, the unifying thing is to really imagine what is that person feeling and when you say things to them, what do they want and how will how you present yourself make them feel in the exchange.

[01:09:35] And hopefully that goes in the direction you want, but obviously not always because it’s subjective and you can miscalculate.

Ethan:

[01:09:43] So I think an analogy that may work for at least people who know chess a little bit about chess is I think of this a little bit like I’m not an expert chess player, but I understand there’s two pieces to playing chess. Well, one is of course knowing the board and how to move. The other is called a book opening, where you’ve memorized a set of opening moves. And I think what you can do in a lot of these conversations is open with a set of preplanned moves where you’ve thought about how am I going to make my first ask?

[01:10:20] And you’ve thought about when am I going to do it? Like, don’t catch your manager wanting to have a deep conversation about career five minutes after a high severity problem on a Friday. Like they’re not in the mindset, right? If their hair’s on fire and they’re trying to get out the door to pick up their kids, this is not when you’re going to get the best hearing. So in the play chess analogy, set up your opening moves.

[01:10:48] But then yes, at some point there are some general rules about chess. Again, I’m no expert, but like in chess, you want to hold the middle of the board. Being towards the middle is. So if you have a chance to move towards the middle, move towards the middle. Well, with a manager or any leader that you want something from, try to move towards that common ground where we can agree. But yeah, it’s about learning how to play well.

[01:11:12] And I’ve spent a career learning how to play well in verbal conversation. And I think the biggest thing, I used to have quite a temperature. I actually got fired two separate times, put on layoff lists. Because I was volatile and critical early in my career, I put the temper away and I learned how not to go there. And that made a huge difference because people don’t react well to anger or criticism, even if you’re right.

01:11:48 — Handling politically skilled operators

Ryan:

[01:11:48] So, I mean, obviously you’re very adept at this interpersonal way to influence people. When you were working in corporate America and you met someone else who you identified, had that skill set, let’s say some manager underneath you, or maybe managers in other orgs, or it could be an engineer as well.

Ethan:

[01:12:07] Yep.

Ryan:

[01:12:08] Is that something that, in your mind, that’s a positive thing, or is that something where you’re thinking, I have to be careful because that person can mold things around them strongly Both.

Ethan:

[01:12:19] What you’re trying to do next is assess what’s their motive. People often ask me, what’s the difference between influence and politics? Like, when does influence become politics or manipulation? And for me, the answer is the skill sets are the same. They’re around motive. And so one thing I teach executive presence, and one thing I point out about executive presence is Darth Vader had it in spade.

[01:12:47] Like, he had tons of executive presence and so did Palpatine. Right. They’re these powerful, commanding figures. These skills, being influential, can be used for good or evil. So with the example you’re talking about, the thing I’m now trying to figure out is, okay, this person is a skilled operator. You know, are they U. S. special forces or are they Russian special Forces? Right. Like. Like which side, you know, who am I dealing with here?

[01:13:14] Right. Am I. Am I dealing with someone who’s gotta, like, rescue me and get me out of a hostage situation, or am I dealing with someone who’s across the line and is about to drop a bomb on me? And so I’m trying to figure out motive. And there’s kind of three cases even there. There’s people whose motive is to do good, easy to work with. Great. We both have high social skill. We’re going to be so effective together.

[01:13:41] There’s people who are completely out for themselves now. You know, going back to Star wars, it’s very much like Sith versus Jedi. I’ve got to get my lightsaber up and I’m going to be fighting for my life. And then there’s this middle ground, though, that’s interesting, which is, well, this person has motives I can work with. They’re not altruistic. They’re not just trying to do well. But in this case they kind of want what I want.

[01:14:10] Or I can give them enough wins and I can negotiate. I can be like, hey, you want this, I want that and that’s the deal making of, of we’re never going to be best friends. I don’t necessarily think you’re the best person. But in this limited scope we can both make money together or we can both ship something together.

Ryan:

[01:14:33] Let’s say there’s someone who is extreme high performer but you can tell they, they will only do what will help them and you see a path to aligning what you both want. So it can be good. But you could tell that if this is ever off their path, they will not be on your side. Or in fact if there’s something that could get them ahead and put you behind, they would gladly do it. Would you still choose to ally with that person or would you kind of separate them from you and your org somewhere?

Ethan:

[01:15:07] Well, if I could separate, I’d love to. But remember at work we don’t get to choose always the teams we work with. Like sometimes we need another team. Sometimes we need a team that’s run by a terrible person. Sometimes we need a team that’s run by a self serving person. I think if you want to succeed in your career, most sure get away or marginalize or minimize contact with difficult or unethical people.

[01:15:36] But also learn to work even with the very unethical because sometimes you’re going to have to. Now of course there’s a limit. I’m not saying break laws or do things, you know, but, but sometimes you don’t get to choose who your teammates are or what system you need. And so I do think it’s worth being equipped. How am I going to do the best I can? And there are, there’s, you know, we don’t have time.

[01:16:07] There’s a lot of techniques to protect yourself. There’s bring allies with you. For example, most slippery people, they love to be slippery in the dark where they’re not going to get caught. So if you can bring a few allies so it looks like, oh, everyone thinks this often suddenly they’ll say well I think that too. Because they, you know, part of their power is slippery. People usually can’t afford to have everyone know that they’re unethical because once we know, once everyone knows someone’s unethical, they do get shunned and, and so they have to keep it largely under wraps or defensible.

[01:16:57] What There’s a phrase from politics called plausibly deniable, where they can deny that they were really as bad as maybe you’re saying. What you want to do is put them in a situation where it’s easier to go along with you than to fight you. So construct the situation where they’re like, okay, this isn’t the best for me. Like, I’d rather stab Ethan in the back, but, boy, there’s a lot of witnesses.

[01:17:22] Right? That’s. And so you. And it’s unfortunate, you know, again, I know many engineers will be frustrated. Why do I have to do all this crap? Can’t I just do good work? Why? Why? Sorry. Humanity going back. By the way, read any, you know, ancient text of philosophy or religion. Humans haven’t changed. They’ve been mean to each other since the first ape clubbed the other ape. This doesn’t change.

[01:17:46] So it’s. I’m sorry that it frustrates you, but you’re gonna have to do it.

Ryan:

[01:17:51] So it sounds like the best way to prevent yourself from backstabbing is to have a really strong, I guess, soft power in your org where there are people who will defend you. Or just the setup is that you would be too powerful for them to come after you.

Ethan:

[01:18:08] Not worth it.

Ryan:

[01:18:09] Not worth it, right?

Ethan:

[01:18:10] It’s, it’s, it’s like all the animals in the animal kingdom, you know, like porcupine. All right, plenty of animals can eat a porcupine if they’re willing to get a mouthful of quills, but it’s not worth it. Like, you know, and I’m sure there are many other examples, but you, you want to be not worth it. It’s just easier because remember, this is important other than cartoons, like a Darth Vader. People don’t wake up in the morning.

[01:18:40] That slippery person does not wake up in the morning and say, I’m an evil jerk. And that’s all I ever want to be. Their self narrative is more like, I’m doing what I need to. I’m practical. Other people are wishy washy and have all these optimistic things. I’m practical. And some people get upset that I’m so practical. Well, that’s not my fault. That’s what they’re telling themselves. And so what you want to do is help them with their narrative and say, you know what?

[01:19:11] You’re right, you’re very practical. And in this situation, the practical thing is to go ahead and help me out because I’ve got a lot of support and, you know, I’m a rising star, and it’s, you help me out and I’ll remember. And they’re like, oh, that is very practical. You know, even though. And that’s just. It’s important to remember that the other person’s narrative, we think of them as like, that person is so unethical.

[01:19:42] They must wake up in the morning and think about how they can rob their grandmother. That’s not what they say about themselves. They say, you know, grandma’s getting old and I need to manage her funds because she’s not really up to it anymore and she doesn’t need all that anyway. But that is what they tell themselves, right? It’s not I’m a thief.

01:20:01 — Orgs trying to steal scope

Ryan:

[01:20:01] What about. I mean, I think a lot of what we talked about is kind of implying within your org. Have you ever seen cases where other orgs were plotting to maybe take scope from your org or kind of, you know, battling between the org orgs?

Ethan:

[01:20:18] Oh, geez, I’ve seen this all the time. And again, the narrative people are telling themselves is not usually I’m going to pillage scope from that other org. It’s they know their own org and their own mission really well. So they have a really clear idea why having your scope would help them. Meanwhile, they don’t know your mission and they don’t know everything you’re really doing. And so it’s very easy for them to imagine that your org is not that important.

[01:20:51] A very smart engineering leader once said, respect between two engineering teams is inversely proportional to their distance. So the further apart they are, the more they belittle each other because they can’t see the work. And you hear this all the time. Like I think of Amazon S3, right? This service that stores now hundreds of trillions of objects. I’ve heard it described as, well, it’s just a big disk drive.

[01:21:19] So this is very belittling. Like of course, at some level it is just a big storage system. That’s true. But the boil down, it’s really just a big disk drive is so dismissive. And you see this all the time. And the closer you get to home, the more you’ve seen the work or worked with the people or seen the complexities and all the edge cases. You’re like, oh, actually that is very complicated. It. Well, I see this with orgs, sure there are some leaders who are just like, yeah, we should take that org because I want it.

[01:21:49] But a lot of times what they’re doing is like, well, we can do so much more with it. Again, that internal narrative. Very few humans wake up and say, I’m going to Be evil today. What they do is they want something and they construct a line of logic which we’re very good at about why that’s actually a great thing anyway. And that’s where you get the org versus org fights.

Ryan:

[01:22:12] Let’s say I’m an engineering leader. How do I protect my org from other orgs?

Ethan:

[01:22:18] Well, first thing is you’ve got to get all this is the skunk trick. I guess we’ll use animals here. The skunk trick is make yourself unattractive. So trick number one is list all the things that they don’t know about that you have to maintain that are unsexy and they don’t want and have the customers for those show up and say how important they are. Oh, yeah, well, we actually run, you know, the Yugoslavia, and Yugoslavia is not even a country anymore, but we’ll use that.

[01:22:45] We run the Yugoslavian tax engine. And so, boy, we get a lot of tickets for that. And it’s going to have to be owned. So are you guys ready to take on that? That’s one thing you could do. The other thing is you want to make yourself thorny again. There is a little bit. Who’s the least attractive target? And so they’re out, sort of. Do they really want your org or do they want an org? And so one of the ways to be thorny, honestly, is to overreact and attack back and just make yourself like, oh, ouch.

[01:23:22] They. They went public and volatile really soon. Maybe I’ll go for a softer target. Now, you have to be careful at how you do that because if you, if you behave in a way that makes it easy to discredit you, then you’re an easier target. Oh, see, Ethan’s not a very mature leader. We should definitely move that under me. I’ll make sure we don’t have this sort of unseemly behavior that’s weakening myself.

[01:23:48] But if I can immediately say counterattack and say, you know, it’s really good that you’re thinking about merging our orgs. Your org is going to be so great as a part of our org and I promise I’ll take care of you. Then they’re like, oh, maybe. Maybe I messed with the bull and got the horns so you can. So there’s I, you know, I’m uncomfortable talking about sort of combative tactics because I see when you get to that, it’s a failure, in my opinion.

[01:24:16] But if someone’s trying to maneuver and you don’t want that, it’s a matter of shining a light on it. And sort of exposing their motives if you can, while making a strong case for why the work you do is very important. Now where you’re vulnerable is if your work is behind, if you have weaknesses, if your work is behind, if it’s not making a lot of money, if it’s performing poorly, those are hard cases to make.

[01:24:45] Whereas if you’re on a strong platform of high performance, then you have a lot of arguments hopefully about don’t distract us. Another great one is we know reorg cause a six month hit in productivity, we’re killing it on profitability, new customers, whatever is valuable. We should, you know, we shouldn’t even talk about this for six months. We need to hit these goals and get to year end because you know, in six months they’ll have moved on.

[01:25:16] So one thing, that’s another delaying tactic and I hope what you’re really getting from this answer is, is how quickly I’m coming up with like here’s three or four different things. You just have to look at the situation and say how can I put this off, change it, scare them off, make myself unattractive, whatever I need to do. I’m just looking for what’s again narrative. What’s the best story? People love to think that facts make decisions, but what we know from psychology, people come to an emotional conclusion and then rationalize it.

[01:25:57] If I am a better storyteller, I give you that narrative that you’re like, yeah, that sounds right. That feels emotionally where you wanted to go. And then you have a story. I’ve helped you wrap your own story around it. And then once it’s your story, your internal story of how this should be, it’s very hard for anyone to shake that. So be the better storyteller.

Ryan:

[01:26:20] You mentioned a lot of tactics on defense. What about the flip side? Let’s say you’re high performing manager and you see there’s a great reason for another org nearby to be absorbed into your org. What are the attacker tactics?

Ethan:

[01:26:39] Wow, Ryan, when you said we would just go deep on transparency, you were not kidding. Ah, attacker tactics. So the best attack, which I would argue isn’t even necessarily an attack, but it has that effect, is a real story of how that’s going to create more value. If you can make a point that whatever metric you’re using, this group puts out one release a year, I can get it to two. This group makes $100 million, I can get it to 200 and here’s how then that’s actually a very good argument.

[01:27:21] If we go to Maybe slightly less good arguments. They usually have to do with seniority, efficiency. I can save costs. I can. By consolidating this, we’re either going to get more done with less or, or we’re going to have simplicity in our communication or you’re going to have fewer direct reports, which is going to make your life easier. There are arguments that aren’t necessarily false, but they’re less directly about business growth and they’re more about making things smoother for the leader who’s making the decision, or nicer or cheaper.

[01:28:05] They’re like second order effects. I wouldn’t, I hope no one would do this, but I know people do. The third attack tactic is clearly to go highlight flaws in the other or right to go show, well, this leader has this problem and this team has this problem and they have high turnover and they have high cost and look at this outage. And you’re basically trying to show, well, it’s a dumpster fire and you should let me put that out.

Ryan:

[01:28:35] But what if that’s true, though?

Ethan:

[01:28:37] Okay, if it’s true again, if it’s true, it’s a dumpster fire, then you’re offering to make things better. It’s where you’re kind of highlighting things that maybe aren’t as bad, you know, just like only pointing at the bad. Yeah, if it’s, if it’s true, an org is struggling. The truth is you can talk to your manager and say, look, the way I would do that though, is not say, I should take over them, I’ll fix them.

[01:29:02] I would say, hey, I go to whoever owns it or the common boss, I say, hey, I can’t help but notice that you’re having a lot of trouble here. Is there anything I can do to help? That’s my opening gambit for I want to take it over because I can go in there and start helping. And then later they’re like, wow, it’s so much better under Ethan’s leadership. And at that point I can casually mention, well, you know, if it would really help you, you know, we could just consolidate.

[01:29:33] Now that I’m here, I see all these efficiencies. That’s, that’s like another way to go about the same thing. I feel like giving all these answers, people are going to think like, wow, that Ethan is a real sadistic.

Ryan:

[01:29:47] what actually happens though, I think, like, what you’re saying. So it’s kind of. I don’t.

Ethan:

[01:29:52] I have seen it all. I’ve seen things, you know, that make me want to close my eyes for sure.

Ryan:

[01:29:57] I kind of want to talk a little bit about, like, some of that interpersonal stuff or maybe, you know, the backstabbing, some of the politics. If there is someone within your org that is a people problem, they’re doing something. Actually, I don’t know, you. You can just immediately get them fired because they’re in your recursive reports.

Ethan:

[01:30:21] have the evidence for sure. But yeah, you can. Absolutely. You can move very quickly if it’s clear.

01:30:26 — Handling difficult people from other orgs

Ryan:

[01:30:26] But what if it’s someone in another org? So another org. There’s a person who is well liked within their org that is a major problem for your org. Then what do you do?

Ethan:

[01:30:39] So one of my managers had this problem and he. He literally put someone in a position he called a meat shield, where it’s like, this person is a huge pain for our org or this org. Your job is to keep them off of us. And honestly, it’s going to be a terrible job, but I’ll reward you later. So you need to, like, go be a meat shield till this project’s over. And so the imagery is just. Yeah, no, it’s like it was, you know, like, put your back into it.

[01:31:14] Let them beat you.

Ryan:

[01:31:16] Oh, my God.

Ethan:

[01:31:17] And of course, he had to. He was very honest with them of, like, that’s your role. And I know it’s a terrible role, but I’ll make it up to you. You know, get us through this project. Like, go take one for the team and I’ll pay you back. But what else can you do when someone’s in another org? The first thing, the most important thing, if you want to try and make a change, is understand why they’re valuable to their org.

[01:31:44] Because the reason they’re that other org, what they’re not doing is saying, I know we have this evil person, and as long as they’re pointed at other orgs, we’re okay with them being evil. That’s probably not what they’re saying. What they. They either aren’t seeing it or more likely that person. What they’re telling themselves is, well, they have a little bit of sharp elbows, but they get so much done that’s valuable.

[01:32:09] It’s okay. And if people’s feelings are hurt, well, they’re soft. It’s kind of how they’re. So you have to understand where they’re being seen as valuable. And then what you have to do is go to that leadership and have one of a few Discussions. Look, we’re willing to give you this value without all this pain. Is there a better way? That’s one way you can go about it. The other way you can go about it is, look, your guy is driving us nuts or is mistreating us.

[01:32:43] So that’s going to change. Either you’re going to give us a new interface, you’re going to. I can’t say you’re going to fire them, but you’re going to move them away from bothering us at least, or we’re going to stop cooperating. Right? Like if. If you’re going to inflict pain on us, we’re going to return the pain. Everything will be late, everything will be your fault. Everything will be escalated.

[01:33:07] Everything will require. Six document reviews we’ll put in place. I’ll spend the rest of the day asking Chat GPT, what is the most policy checkpoints I could put on our next release? Right, like you can express this.

Ryan:

[01:33:19] How do you message that with the right narrative without, you know.

Ethan:

[01:33:22] Well, I mean, if. Okay, so that’s. That’s my internal narrative, I guess. That’s my, like, all right, screw with me, I’ll screw with you back.

Ryan:

[01:33:29] Yeah.

Ethan:

[01:33:30] My external narrative would go something like, dear peer of mine, this person is really making things difficult. You have this person on your team. I know they’re a valuable member to you and your team, but unfortunately our team is really struggling with them and I’d like to make a change. Option one, you can find a new interface person for us. Option two is we’re going to have to adjust to distance ourselves for the sake of my team.

[01:34:08] And I think, unfortunately, that option is going to create a lot of friction between our orgs that really is going to take both of our time. And so I don’t want to take your time, but I have to to protect my people. Some of them are ready to leave. What would you do if people on your team were ready to leave? And so I’m gonna have to take some steps and that’s gonna increase bureaucracy and take a lot more of your time and my time.

[01:34:36] And I’m sure there’s a better way.

Ryan:

[01:34:39] That’s amazing because it’s the same thing. But you didn’t, you didn’t bluster or threaten it just by circumstance. If you don’t comply, your life will be harder. But I wish it wasn’t. But that’s just how it is.

Ethan:

[01:34:55] One thing people underestimate is most leaders at a certain level, they may not have expert social skills, but they’re pretty they got there for a reason. They understand what they’re being told, even if they don’t understand it consciously. They’re sensitive to unspoken messages, and they understand that I’m giving them a chance to collaborate, but that there’s a stick up my sleeve that I didn’t come to this with no other plan.

[01:35:34] And part of this is also how I project how you. If you look weak, like you’re begging, I don’t really care what you need. But, you know, I’m presenting it as like, wow, this is. This is really tragic because it’s gonna get hard for both of us. And what I’m conveying there is, look, you wanna trade punches, that’s okay. I play ice hockey, and sometimes someone on ice will hit me or, you know, be a little rough with me, and I’ll just look at them and say, you know, two can do this.

[01:36:09] This was your freebie. You decide. But I’m like, putting him on notice, like, hey, the next time I have two elbows also. Right.

01:36:19 — Handling weak managers

Ryan:

[01:36:19] Well, this conversation makes me think that let’s say you’re an engineer or someone who is reporting some. You want your, Your leader to. To be strong for your org. Because I can imagine if they’re not, let’s say they’re. They’re very nice and their org is getting, I don’t know, a lot of negative things are happening to it. And they just say, oh, that’s okay, you know, we’ll deal with it as it comes.

[01:36:46] And the. Everyone else is going to step on them and that’s going to affect you if your manager can’t stand up for the org.

Ethan:

[01:36:56] Absolutely. And I think, remember your manager. I’ve tried to showcase that your manager doesn’t have to be a mean person to be firm, to be professionally firm, to have backbone. They have to be skilled. A lot of people, this is important. A lot of people don’t know how to be socially skilled. And so they can only say hard things once. They let themselves get angry or upset. That’s their fuel is if I need to give you a hard performance message and say, ryan, you know, you’ve missed three dates in a row.

[01:37:32] I can only do that once. I get internally angry. And like, Ryan’s missed three dates. He let me down. And so now I’m coming at you. I’ve gotten my ability to speak to you and to say the truth because, see, I’m afraid you’re going to say, well, I haven’t. Or it was. My mom was sick or whatever, and you’re going to make it hard for me. So I have to be ready to fight. And some people can’t do that without first allowing themselves to be angry.

[01:37:58] You want a manager, ideally, who can stand their ground because they believe in it and because they are willing to have confrontation. And I’m no master at this, okay? I have plenty of friends will tell me where I’ve been a jerk. But I try to figure out how can I stick to what I believe without ever needing to resort to anger as a fuel. I can just stick to belief. To your point about what you want from managers, there’s a graphic image that says, look, all managers are either an umbrella or a funnel.

[01:38:38] And when the poo comes down, they’re either shielding their team with the umbrella or they’re a funnel dropping it onto their team. And you were discussing the funnel. The one who’s is like, yeah, we’ll deal with it. And what. What are they doing? They’re deciding that if you’re the engineer, they have a choice between push back on the boss above them, which feels scary and dangerous, and push back on you being upset that you’re being asked to do weekend maintenance for the fourth weekend.

[01:39:10] And you’re an easier target because I have rank, I have position, I control your performance review, I control your salary to some degree. So they’re making a choice, which is, it’s easier for me, safer for me. And what you have to do in that trick in that space is change that balance of safety. So you either have to help them feel safe pushing back on their boss or. Or help them feel not safe asking for your fourth weekend.

[01:39:43] Those are. Those are the only two choices, really, is you change the balance of safety.

Ryan:

[01:39:49] Well, that first one almost feels impossible to me. Like if I was. If I was reporting to someone and this manager just says yes to everything, has no backbone. I don’t know how I can give them a backbone, but I can definitely do the second option.

Ethan:

[01:40:07] Well, you give them a backbone exactly by the second option. But like this. You say, let’s say I’m your manager. You say, hey, Ethan,

Ryan:

[01:40:18] this is.

Ethan:

[01:40:19] There seems to be a pattern where we keep getting asked to do weekend work. I would love to help you push back on that. About how that’s damaging our team, distracting our team. I can give you some of the data, but I also have to be clear with you that my family is very important to me and I can’t keep working weekends. And so what I don’t want is for you to be in a position where you accept work that I’m going to have to decline.

[01:40:54] And then you’ll have the problem that you committed to it and it won’t happen. And so I don’t want to see you in that position. How can we plan to push back next time? That’s, that’s how you line it up in advance.

Ryan:

[01:41:12] And I’ve, I’ve heard this exact situation from a friend. Their manager was a yes man to manage their manager, very common. And so they were doing, committing to all these unreasonable deadlines and, you know, not protected at all. And it wasn’t a big deal because there’s turnover, but it’s fine, just hire new people and just keep saying yes to more and driving people as hard as you can. The narrative that you said there felt slightly more threatening than the, the other ones, because I can imagine, I can imagine that that more directly feels like you’re threatening not to do the work.

Ethan:

[01:41:52] So, number one, you’re right, that wasn’t as polished as some of the other lines I’ve given you. But I think with five minutes, I could create a more polished version. And one of the keys that you’re getting at here is the wrong time to start setting this up is while the request is in hand, what you want to do. Like, if you’re getting asked to work every weekend, the right time to have this discussion with your boss is like Tuesday or Wednesday, after the weekend’s over and a little time has passed and before the next crisis arrives.

[01:42:26] And the second piece, though is it’s then how do you make clear that your tolerance is limited? And yes, one of the costs of that, this is important if you’re not willing for the manager to say, well, if you don’t like it, quit, you can’t say anything. You’ve got to have enough confidence in yourself that if it does come to that, like when I went to my manager and I said, I need to know how important my career is to Amazon, I was also looking for an outside job.

[01:43:05] Like, I had decided, I’m going to do everything I can diplomatically to win this over, but I’m also okay leaving.

Ryan:

[01:43:14] I think that’s the big difference between this situation and all the other ones, which is in all the other situations, you had leverage. You could leave and they would feel pain. Whereas in this case, if you’re the lowly person or engineer, maybe they don’t see your value as much. If you leave, you say, okay, great, goodbye.

Ethan:

[01:43:37] I mean, if you really are that discardable, then you’ve not created much value in Other words, absolutely, leverage is required. But good engineers are valuable and teams that are total turnover factories are not that effective. Now, if there’s a company that totally has a view of bring them in, burn them out, move them on, you’re probably in a wrong place, like that is, you know, if you’re at a place where you can’t create any sort of value, the right answer is you will have to get out.

[01:44:15] And I think people get trapped believing there isn’t a better option. There’s always more jobs and I’m not belittling long searches or the difficulty or all the problems, you know, with maybe in engineering today of AI making it hard to get on entry level, but. But there are jobs if you do enough work. So one of the things you have to ask yourself is this being at the bottom of the funnel of bad work being rained down upon you, how bad is that compared to, say, doing the extra work to network and find something else or start your own thing?

[01:44:59] Don’t be a victim. You have agency. If you prefer, you know, you can look at ownership. The book Extreme Ownership by Jocko Willink, whatever. There’s many ways to look at it, but in the end you have to be willing to take control of your life. Even if that does mean, by the way, that you weather the bad manager while you complete the training program or find the job. What a lot of people like to do is complain about how unhappy they are, but not do anything else.

[01:45:33] I get it because it seems exhausting, like, oh, I just worked all day and now I worked all weekend. When am I going to look for a job? One of my beliefs is to work less, you often have to work more first, right? In other words, to get to the place where you can coast or have a good situation, you often have to do some ugly digging first. And this, the situation you’re setting up, is one of those.

[01:45:57] You also hit on another point that’s really important, create leverage. If you create that leverage, you’re always in the winning situation. And then thing is, if your stick is long enough, if your lever is long enough, it doesn’t take very much force, which is why I’m able to be polite about it. I can be very soft in my hints about what I like because the lever is so strong, people are like, oh, we want to make this work right away.

[01:46:27] And that’s what you’re trying to do, is have a big lever, I guess.

01:46:30 — Backchanneling

Ryan:

[01:46:30] On another topic, there’s this skill of back channeling, which is not in public forums, but one on One in different places, driving your message to the right people so that you get the right aligned outcome. So how do you effectively back channel?

Ethan:

[01:46:53] So I think of back channeling in two ways. There’s good back channeling and manipulative back channeling. Manipulative back channeling is where I’m trying to do something that I wouldn’t be willing to say in public. So that’s where you’re really undermining or painting the other, other proposal or the other leader in a negative light. I don’t think that’s usually necessary, and I’d like to say never necessary, but you’ll lose some because maybe they’ll do that.

[01:47:28] What positive back channeling is, is giving people a chance to ask questions and find out what their concerns are, where they don’t have to bring them out in public. So maybe, for example, I’m the CFO or I’m the financial person and I’m really concerned about the budget, but I don’t want to just ask a bunch of budget questions in the meeting or I don’t want to put you on the spot. Or maybe I don’t realize that you’re mostly concerned about the budget.

[01:48:02] If we have a back channel conversation, I can ask, here’s my idea. Is there anything you don’t understand? Do you have any concerns? And what I’m doing there is. I’m giving you a chance to feel, remember feelings and narrative. I’m giving you a chance to feel understood, to feel consulted, to feel listened to. Back channeling doesn’t just work because of information exchange. You do get data and that is good.

[01:48:31] But it also works because there’s all this subconscious I made you feel important, I made you feel listened to, I made you feel heard. And that emotional buy in, particularly if the other project leader doesn’t do that, gives you a huge leg up because you’re building trust. You’re building trust that, oh, you take input, oh, I can work with you. And so even if the two proposals are equal, or even if the other one’s a little better, remember part of my bet if I’m in executive funding proposals is who can I count on when the chips are down?

[01:49:08] Who can I count on to get this across the line? Who can I count on if there’s a problem? This other leader I don’t know as well, they didn’t consult me. Ryan came to me and asked me my input and showed understanding for my needs. Totally. I’m funding that because even though the payoff’s a little lower, I’M sure Ryan’s going to come through or I can work with him and so just realize we so underestimate emotions.

[01:49:40] As an executive making a decision on which project to fund, I’m wondering how it’s going to make me look. I’m wondering what happens if it goes wrong. I’m wondering what happens if there’s conflict between you and me later. Are you going to resolve it reasonably? I have all these fears and worries, and the back channel lets me quiet those down as well as address factual concerns. And so that’s why it’s so effective.

Ryan:

[01:50:05] I think if you were to negatively back channel, other people would see that you are a snake and then you would lose some rapport.

Ethan:

[01:50:17] I guess people do negatively back channel and they have to be careful about how they get caught, right? Or don’t get caught. You know, this is what you would call a backroom deal. You know, the classic backroom deal is like the kickback where you support this and I’ll make sure you’re taken care of. It’s kind of like, you know, a mafia movie or something, right? It’s that. It’s the I’m coming to you with a proposal that amounts to you should do the wrong thing because we’re both going to benefit and I just want to make sure we’re on the same page.

[01:50:51] And you know that even though I’m actually a snake, I’m not going to bite you. We’re snakes together. That’s the essence of that offer. And some people are up for that because some people are selfish and ambition runs wild.

Ryan:

[01:51:18] well,

Ethan:

[01:51:20] I had to pay my dues and I got screwed over when I was in the weak position. And so now that I’m in the strong position, it’s my right to take all I didn’t get. It’s not really I’m screwing that. See, they always tell themselves a story that makes themselves look better. It’s not really I’m screwing those people. It’s that I’m getting what I’m owed. Like, this is only fair. And that’s another thing that’s really important is just when you’re dealing with someone who seems to be vicious, do at least realize that their story to themselves is not that they’re evil.

[01:51:52] Their story to themselves is something else. And if you can figure out their story, you have a better chance to redirect that story.

01:52:04 — Influence without authority

Ryan:

[01:52:04] We talked a lot about soft power and influence, and it. It sounds to me like that’s not necessarily directly proportional to how many recursive reports you have. Have you seen examples where there was a person within an org that had a disproportionate amount of influence?

Ethan:

[01:52:22] Yeah, well, the first thing I would say is I’ve seen soft power come from ideas a few times. So Amazon for a while built a product. They called it Cloud Drive ultimately, but it was a storage system, kind of like Box or Dropbox, either one. And that proposal came from an entry level new college graduate engineer. He had an idea for it and he did two things to get it heard. The first was Amazon had an invite only engineering conference where only a tiny percentage of engineers went off site to this conference, but you could get an invite by making a proposal.

[01:53:05] So he fought his way into this conference. Then he knew Jeff Bezos came and reviewed an idea poster session. And so he emailed Jeff and said, I have this idea and I’m going to be at the poster session. Do you think you could stop by my poster which is a very small ask of the CEO. Well then he got an audience with Jeff and he pitched his idea and what Jeff did was like it and email his triple skip.

[01:53:37] So my boss, so there was Jeff, then this vp, then me, then a manager, then this new engineer. And my VP didn’t like the answer the idea at all. But Jeff had said look into it. And so, you know, and the project got funded, it got built. So outside his influence. And I can, I can tell another story about this. Whilst different people have different origin stories for it to some degree I worked with a slightly more senior person who created the Amazon Fire TV or helped put forward the proposal for it.

[01:54:19] And he also was not a super senior person, but he did a lot of legwork and had the knowledge and had the connections and managed to get that created. So what works here? How you get that outsize influence? I’ve seen it many times. You have to have an idea. And again, it’s that appealing storytelling. You’re telling a vision. These guys were telling visions. Wouldn’t it be great if we had a cloud storage system?

[01:54:50] Wouldn’t it be great if we had a low cost living room device where people could watch prime video? They were telling an understandable sound bite story that everyone agreed with, at least on the surface. Because wouldn’t it be great if we had a cloud storage system? Yeah. Wouldn’t it be great if we had, you know, a living room device? A low cost living room device? Yeah, sounds great. Oh well look, here’s, here’s a way we could do it.

[01:55:18] So I don’t know if you have a follow up question on that, but influence can come from anywhere. It usually begins with a good story based in some real fact. It can’t be usually total nonsense and then told with enthusiasm and told repeatedly and specifically told the people who have the power to do it.

Ryan:

[01:55:44] What about, what are the the least discussed reasons that people get fired that you wouldn’t see in the performance notes, but they are there. So kind of that second thing that’s actually getting someone fired when there’s an upfront story told in the perf notes,

Ethan:

[01:56:04] there’s a lot of rules about when you can and can’t fire someone. And so almost always the public story is some sort of performance story because that’s the one thing that’s allowed. So the hidden reasons are often if they’re not evil or the person isn’t just doing wrong work, it’s usually incompatibility. So the main reason I see people getting let go that isn’t discussed is their style and the leader’s style are very different.

[01:56:41] And that style can just be one person is very detail oriented, one person is very high level and this can be on either end. That’s one type of incompatibility. Another one can be with engineers. It’s often one person wants to be very tech oriented and what’s the best technology. And the other person wants to be very business outcome oriented of look what makes the money, I don’t care if it’s a hack.

[01:57:11] And where the firing occurs is where that becomes a religious debate and the two are just sort of at each other’s throats. And remember, what starts as a small thing often builds up where over time these become a label. That person is technically clueless, that person’s only about money and they start labeling each other and then it becomes personal. And once it’s personal, I’ve decided you’re a bad person.

[01:57:43] And I see all these things that make you bad. And you’ve decided I’m a bad manager. And once you have this lens, you’re a bad manager. It really doesn’t matter what I do. You’ll see everything I do is bad. Because so many things are ambiguous and open to interpretation. That’s where people get fired. But of course it’s always labeled somehow performance.

01:58:04 — Sexual harassment

Ryan:

[01:58:04] Well, you mentioned it could be labels perform. But you mentioned that you could get fired for performance. But there’s also those other HR reasons like sexual harassment and those types of things. Right?

Ethan:

[01:58:15] I’ve seen one of the Most blatant cases, in my opinion of sexual harassment, I have to be careful even now what I share. And that will make the point. There was an executive hired, very high level position, a chief level position, who immediately started harassing women. And he was so blatant that it ultimately, when we asked one of the HR representatives on his team about had she gotten any complaints, she started crying because he had harassed her too.

[01:58:49] So this is someone with the gall to harass a woman in human resources. But he disappeared. The leader terminated him right away as soon as this came out. But people ask me, how did such a person get in here? Didn’t we check references? And the thing I want people to understand is we didn’t have legal proof of this harassment. We had a lot of people who said it happened to them, we believed them. But we never took it through a court and got a judgment that said, oh, he’s a harasser.

[01:59:28] So even though that’s true, if I name this person and say, don’t ever hire Fred Smith, you know, he harasses women, Fred can sue me because I have no proof of that. I have no legal proof. I know it’s true. He knows it’s true. It. So that’s why companies don’t say it. So when someone calls up and asks about Fred, I refer them to HR and HR says, oh, Fred was employed here from this date to that date.

[01:59:57] And you know, they, that’s how they keep getting hired. I will say I’ve looked this person up on LinkedIn since then and the, the fact he ever worked at Amazon isn’t even on his resume. We caught it quick enough. He just deleted it. No awkward questions there? No. Why were you only there a month? He just deleted it, moved on. And that’s how they survive. So yeah, I’ve, I’ve seen and it’s hard to stop.

Ryan:

[02:00:27] That was the next place that that person worked more than one month or

Ethan:

[02:00:32] I guess, yeah, maybe he was more careful, you know, maybe because we caught it so quick. I don’t know, I, I don’t believe that what he did at our workplace was a one off. I don’t believe that. But you know, of course I don’t know that.

02:00:49 — Skip overruling firing

Ryan:

[02:00:49] Last time we met you mentioned that you as a manager could fire any one engineer that you wanted. Is there ever a case where you want to fire someone? Or let’s say a manager wants to fire someone but their skip says you can’t because the skip sees it and goes, that’s clearly there’s an ulterior motive here. That person’s fine. Like, has that. Have you ever seen that? Overruling.

Ethan:

[02:01:13] I have. I’ve definitely seen a case where a manager floats the idea. I’m not sure Ryan’s the right person for us. And the manager knows your work and says, why don’t I move Ryan over here? Right. Why don’t I? I think Ryan. And again, if I want to make you feel good about it as a manager, what I would say is, I’d say, you know, I see that you’re struggling with Ryan. I’ve seen him do well in other environments.

[02:01:44] Let me move him to this other place, he’ll be out of your hair. And I think he’ll be fine there. I don’t tell them that I think you’re going to be great there because that’s kind of hinting that they’re bad. I just make it socially easy for them to say, sure, move him away. That’s just as good as fired. From my viewpoint, this relies on your skip, of course, knowing or having a sense of things. And again, sometimes it is just incompatibility, your style.

[02:02:14] I can see that you and your manager are cats and dogs in a bag. It doesn’t mean that you can’t get along well. It doesn’t necessarily mean the manager is bad. It does mean they lack enough adaptability. So I can work on that. But I may choose to save you in that time. Because what gets forgotten is people have personalities and preferences and not everyone can work for every other boss. I certainly had managers that some people at Amazon love and they drove me batty.

[02:02:53] And I also know that for those managers, they knew I was a high performer, but they didn’t like my way of doing it. And that’s just style does matter. You know, you have music you like more than others. Some music you’re like, get this out of my ears. And some you’re like, I could listen to this all day. There is some of that element in work.

Ryan:

[02:03:17] I mean, the more that we talk about the behind the scenes of how managers keep orgs together and retain people, feels like the orgs are, they’re like duct taped together with a bunch of like, you know, well worded phrases. This person goes here, oh, I’m going to get you what you want next half and I’m going to move people. And it’s all kind of back of the envelope accounting in the leader’s mind. Is that an accurate assessment?

Ethan:

[02:03:45] You probably saw me smile when you said that. I certainly feel that way. What I mean is Managers start out like you would think. It’s just like I would compare it to an engineer with an architecture. When the project begins, you’ve got your nice little lines and arrows and boxes on a page of the system you envision. And when it ships, there is some resemblance to what you drew, but there’s all these special cases you didn’t think about and all these things you forgot that got like, well, what service can we put that in?

[02:04:21] Because we’re out of time and the thing is shipped, bears a resemblance to the thing you designed, but it is not this sort of pure, beautiful ivory tower. Well, I think the same is true of orgs where managers set out of course to manage only by performance and to have great managers and everyone happy and to develop everyone’s career. And then reality hits of schedules and difficult people and locations and office changes and people out on leave.

[02:04:54] And by the time it’s all done, you’re just really happy that it’s working mostly. And you know, I think that’s. I happen to enjoy military history. There’s a phrase for military history or military strategy that applies here that says, you know, with your battle plan, the enemy gets a vote. Like, it’s all great, you have a plan, but then someone starts shooting at you. Maybe if you heard it another way, the boxer Mike Tyson said everyone has a plan till they get punched in the face.

[02:05:27] Well, I think your manager has a plan for a beautiful org right until like deadlines and outages and whatever, punch them in the face and then that plan starts to become whatever gets them through the fight.

02:05:39 — How to fire managers

Ryan:

[02:05:39] We talked about individual firings for, for engineers. I think there’s a common sentiment that it’s harder to fire a bad manager than it is to fire a bad engineer. When you had managers reporting to you that let’s say they were low performing managers, how do you manage them out?

Ethan:

[02:05:57] So I think it is harder to fire managers. There are three reasons. Reason number one is they usually are somewhat more experienced. So they don’t have the obvious bad performance. Meaning the first time I ever had to let someone go, it’s because they stopped coming to work pretty easy. Like, well, it’s Wednesday and you haven’t been here yet. I’m calling you to let you know, don’t bother to come in tomorrow.

[02:06:26] Like pretty cut and dry. Managers don’t usually make mistakes at that level. So they’re more subtle though their weaknesses are more subtle because they got made a manager for some reason. They were technically good, they were politically good, they were organizationally Good. So they have some kind of strength that got them there. That’s reason one, Reason two is a manager’s work is hard to measure.

[02:06:51] How do I exactly measure a manager’s work? Well, is it that they hired? Is it that they did their performance reviews? Is it that the feature shipped on time? Is it bug count? The manager’s job? It’s very hard to put a lot of metrics, which means when it comes time to fire or to consider performance management, we’re in this he said, she said opinion problem. And I think the third reason it’s tough to fire managers is managers are part of the system.

[02:07:28] And so they know what it looks like. They know, like, oh, my boss isn’t that happy. How do I start positioning myself to be hard to get rid of? Like, they have a quicker Spidey sense, if that’s what they want. You know, if they want to stay, they have a quicker sense of how do I. How do I make myself difficult to pin down? And of course, it’s still possible. Like, I terminated a large, you know, not a huge number, but a large number of managers.

[02:08:02] It’s just. It’s trickier.

Ryan:

[02:08:04] Yeah, it sounds really hard to terminate. How do you do it then, in those cases?

Ethan:

[02:08:09] So my ideal way of doing it, which I wasn’t always able to do, is to talk to them and surface the idea socially that maybe this isn’t working out and would they like to collaborate on finding a better place? So I’m phrasing it. As I notice, I have the impression that we’re struggling. I would say something like, I don’t think we’ve gotten off to the best start here, and I’m wondering how you feel about it.

[02:08:42] Would you like to work on that or, you know, do you also see that problem and you’d like my help finding some other option? Very nebulous. Smart managers get the message. I’m your boss. I’m not happy. I’m offering you a door where I give you some time to find a job or maybe a severance package. Do you want to do this the easy way? Many managers will be like, yeah, I do. You know, and of course, they’ll never say, oh, of course you’re right, I’m failing here.

[02:09:16] I want a door. What they will say, if they’re equally smooth, is they’ll say, I’m so sorry that you’re not satisfied. Of course I’m doing my utmost. That’s me telling you, remember, I can make this hard for you. So you’re going to Pay well for my exit. That’s. That’s the counter negotiation. But ultimately, I want you to be happy and satisfied with my work. And if you’re not, of course, I’d like to discuss what changes we could make, which is my way of saying, I’m open.

[02:09:51] I’m open to making it, you know, so I’ve never used the word fired or performance because that creates something legally called constructive termination. Where if I’ve said, you know, we might have to fire you, do you want to quit first? That is a legal problem called constructive termination, where I. Anything I do from that point is then seen as I’d already decided to fire you and now I’m making up reasons.

[02:10:18] So I very carefully avoided that. They very carefully avoided ever saying, acknowledging that there’s an actual problem. What they say is, I’m sorry you feel that way. Right? Oh, I’m sorry you feel that way. That’s your problem. You feel weird things, but I want you to be happy. So if you’re not, of course I discuss alternatives. And that’s their way of saying, yeah, okay, I get that it’s not working.

[02:10:45] What are you going to give me? So that’s one way you quote, unquote, fire. The other way is classic performance management, in which case you have to try and write down clear goals and objectives. You have to start documenting. You have to give. You know, there’s a legal standard that. There’s a legal standard and there’s what HR wants, but you basically have to document a bunch of expectations and then document that they didn’t hit them.

[02:11:16] And then you have a performance case. Almost all managers will quit before that. What they’ll do is once you start the process, they’ll be like, oh, okay, this is going to take him 90 days. I have 90 days to fil. Find a new job. And they go find a new job.

02:11:31 — Leverage when people are getting fired

Ryan:

[02:11:31] You mentioned that when the manager, I guess, knows the system is getting managed out or this conversation starting, they can sometimes switch to, how can I extract more value for myself? And I think a lot of people who are less experienced or they don’t know how the system works too well is when they’re starting to get fired. They don’t. They feel they have no leverage. And the fact that severance exists makes me think that they do have some leverage, although I don’t know if I fully understand it.

[02:12:07] What, what is the leverage that, let’s say, an engineer has as they’re getting fired that, that a company would want to pay for? Because severance, I mean, well, there’s cost, there’s the optics of it. Like, it looks good, of course. But I’m after this conversation, I’m a little more cynical. I feel like they’re paying for something for that report. So that report must have some leverage in that conversation when they’re getting.

Ethan:

[02:12:34] Well, there’s several pieces of leverage. The first is this process. If you want to go through it, is going to take a lot of your time and you have to document things. You have to follow a process. You have to file a bunch of paperwork. HR is going to audit your paperwork. If you’re a low level manager, you know they’re going to write it. There are things I can do to string things out as a tactic.

[02:13:00] And so the first offer I’m making you is, hey, I’ll save you all that time if you will give me some money so that I have money in my pocket to go look for another job. I will save you this time. And frustration. That’s the first level of offer. The second level of offer is the story I’ll tell the team. Right. Is, look, I can either leave talking about what a terrible experience this was and what a monster you are.

[02:13:31] I don’t say those exact words to you, but I can show you how to say that, or I can leave saying, you know, this just wasn’t a fit for me and I’m so excited to be moving on to Acme Inc. And I can’t speak highly enough about how Ryan treated me through this whole process. Ryan’s amazing. So which story do you want me telling my co workers? And so those are a couple of the pieces of leverage. And again, it’s this.

[02:14:00] What’s that phrase of mine you like so much? The plausible story or plausible fiction?

Ryan:

[02:14:05] Polite fiction.

Ethan:

[02:14:05] The polite fiction. If I need to say this to you, I say, well, you know, ah, let’s say I’m working for you, Ryan. So far you’ve been nothing but supportive. And if I were to move on to a new job someday, I would, of course, explain how supportive you’ve been. But if I, if, if we’re discussing my moving on, you know, really, I would hope you’d want to support my transition financially so that I can tell that story.

[02:14:44] And that’s how I’d be like, you know, and even that wasn’t maybe as smooth as I would do it if I had to. And by the way, let’s say you, you’re starting to give me performance feedback and I need time to be in the right headspace to have this Conversation with you. I would just say that. I would just say, you know, Ryan, I appreciate you’re starting to give me some performance feedback. I know we need to have this conversation.

[02:15:15] Honestly, this moment. I can’t have it productively. Let’s have it tomorrow morning. Or can we have it tomorrow morning? And most managers won’t be like, no, I need to tell you how rotten you are right now because they’re already on edge a little bit. And it’s. So if you’re putting up this flag that says, I’m about to lose my shit on you, but tomorrow I’ll be a good child, they’ll be like, sure, come back tomorrow.

[02:15:40] Because they’re sort of trained to. And I’ve put you in a really difficult situation. This is important where if I’ve asked for space and you say no, we have to finish this right now. Now I have a beef with hr. He was so mean to me. I asked for space. I asked if we could just talk tomorrow morning. And he said no, I’ve got to ridicule you right now. Very empathetic to hr. And now you’re the one in the hot seat.

[02:16:07] Why couldn’t you give her until tomorrow morning? Why couldn’t you give him until tomorrow afternoon? Again, I hope. I would rather coach people on how to be in great high performing situations where they’re having different conversations. But you want to talk about the dark side, and that’s okay. I get that people want to know, don’t be afraid to buy time. You do have leverage and you may not get anything.

[02:16:38] Remember, if your boss is starting to line you up to terminate you, you may not get anything. But if you don’t ask, you certainly won’t. If you don’t open the door and say, well, you know what? How could we make an amicable deal? And here I’ll give you your last piece of your leverage. Remember, your manager isn’t sitting across the table wanting to feel bad about themselves. They want to feel good about themselves.

[02:17:08] And if they have a chance to spend a little bit of the company’s money, which isn’t really their money, so that they can look in the mirror and say, well, it didn’t work out with Ryan, but I took care of him, I gave him a cushion, they want to do that. And so they’re saving them hassle. There’s the public story you’re going to tell and there’s. They want to feel good about themselves. Most managers don’t like.

[02:17:37] They hate firing. They avoid it. You know, as much as you might think, meaning you, the listener, you. Someone might think managers are just sitting around going, who can I. Who can I pitch next? They’re not. They actually hate it when this is important. One reason people don’t survive firings is by the time the manager actually decides, oh, I just. I’ve. I’ve tried to coach them. You may not feel this way, but in their mind, they’re like, I’ve tried to coach them.

[02:18:06] I’ve tried to pair them with people. I’ve tried to give them all these chances, and they’re still not getting it. I don’t have any choice but to fire them. Once they’ve gotten to where they’re going to act, they’ve told themselves they’re out of options, which is why people getting back off the brink is very unlikely. You know, interestingly, right before I came here to talk to you today, I had lunch and I met a new executive.

[02:18:30] He owns a company that does half a billion dollars of business a year. So it’s a big company. He’s built it over his whole life. And he was telling me how he’s going to have to hold someone in his organization who isn’t performing these. Kept on his staff for two years, he’s finally at the point where he’s going to have to hold them accountable. And what I’m thinking is he’s put off that hard message.

[02:18:57] When he finally gets to where he’s going to give it, he’s so decided that they don’t meet the bar. They can’t get out of that. And it’s not because he’s evil. It’s just people delay. And so I coach people. People come to me all the time because I’m online. They say, ethan, I’ve just been put on a performance improvement plan. Can you help me survive it? Can you help me recover? And I tell them, honestly, no, you’re most likely going to be fired and there’s nothing I can do for you.

[02:19:29] But here’s why. It’s by the time someone gets around to giving you a form of formal performance plan, they’ve tried everything else they could think of, and at this point, they just want you gone as quietly as possible. And it happens again all the time. People write me back and they’re like, you don’t understand. I’m going to. Three months later, I get an email from this, as you were right, I was fired.

[02:19:55] Of course, there are exceptions to this, but it’s so common that I see this pattern of, how can I survive this? I’M gonna. I’m a hard worker. I’m a good person. I’m gonna show them. Three months later, you were right. I got fired.

Ryan:

[02:20:09] Even if people had the awareness, because I guess they’re gonna listen to this conversation, I think they still lack that. A lot of people lack that skill of creating the polite fiction. You know, how do you and I, we talked about. We don’t need to go deeper in it. But I do think that even if I. Someone came to me and they said, I’m in this situation, and I said, here’s your leverage. Good luck. I think they would have trouble delivering that message in an effective way of saying, for instance, especially the team health one, because the team health one is more of a threat.

[02:20:43] The saving time one is, hey, I can do something good for you. I’ll leave quietly. Just help me get some severance. The team health one is, I can make this hard for you unless you get me sufferance. I think delivering that kind of message as you’re leaving, especially when they have such a negative bias on you already, that’s going to take some incredible people skills. Yeah.

Ethan:

[02:21:06] Like, yes, you’re right. Okay. This is part of the message, which is engineers in particular want to believe this is, in fact, maybe the most important message, even if it’s not the most exciting, which is, as an engineer, you want to believe it’s your technical expertise that controls the fate of your career. And it’s not only that, it’s your people skills. It’s well known. Right? Satya has talked about this.

[02:21:32] Other leaders have talked about iq without eq, without emotional intelligence is a waste of iq. And so having strong interpersonal skills is critical. You know, it’s funny, there’s a famous book, I’m sure you know it, by Cal Newport, called so Good They Can’t Ignore youe. And what I find ironic about this is it’s all this stuff about how to be such a good worker, you can’t be ignored. Except this is a guy who has the social skills to both teach at Princeton and write books.

[02:22:04] He’s not without the human skills to present his incredible technical accomplishments delicately and well. So I think it’s a huge area of learning. Now to exemplify this, how would I, if I, on the spot or here, you know, live, had to construct a better way to say that? I would say something gentler, like, boy, this is such a hard message. And I have so many friends on this team. I just. I don’t want them to think badly of me.

[02:22:44] I don’t want them to think this is an unfair environment. Can you work with me so that this is a graceful exit?

Ryan:

[02:22:54] And that’s good, that works. You know, I don’t feel like started at all.

Ethan:

[02:23:00] And you know, later if I need to, I said, well, for it to feel good, I mean of course, you know, like I rough if it, you know, for it to feel good. I need enough security to get to my next thing. And I know you want that, I know that you don’t want anyone to suffer. And I tie into the other thing. But it is possible. And that is why I said this is how you get the time to go figure out how you’re going to deliver these messages.

[02:23:30] Messages. Something people overrate or I guess underrate is recognizing that a lot of hard messages you don’t have to deliver cold. I mentioned this before, but you, you can buy the time or make the time to pre plan. Maybe not every word. It isn’t memorizing a speech, it’s more memorizing a few phrases and how you’re going to, you know, say what you need to in a polite way.

02:24:01 — How to grow past senior eng

Ryan:

[02:24:01] I want to pivot to promos and talking about promotions because I know there’s a lot of stuff going on behind the scenes on alignment between managers and upper leadership on who’s going to get promoted. You mentioned the queue earlier and I think a lot of people, they get, they get stuck at this staff engineer, which is equivalent to the frontline manager level of how to get the promo to go further into those higher levels.

[02:24:28] Do you have any advice on how to break out of that, that common level that people get stuck at?

Ethan:

[02:24:34] Yeah, I mean, this is, this is my bread and butter, what I teach every day and kind of if I have a signature expertise, it’s how to move up further. What generally gets people stuck is they’re, they’re not learning the new skill of the next level. Instead they’re doing the current skill harder and harder. And so what gets you promoted early in life is hard, effective work. It is hard work. It is delivery.

[02:25:03] It is shipping a lot or building a lot. But as you get further, you need different skills. You need to be able to delegate, you need to be able to communicate. You also need to be able to let go of your hard skills because if you’re trying to micromanage everyone and tell them, write it this way, write it that way, build it this way, you’re driving your engineers nuts. And they, you’re going to drive away your good people.

[02:25:26] And so the way to get to the next level is to figure out what do I need to let go of, step away from, and what new thing do I need to master. And there’s a classic book on this. It’s by a guy named Marshall Goldsmith. It’s what got you here won’t get you there. It’s literally shows a ladder with a gap in it. Like, what got you this far isn’t going to get you up there. That’s kind of the classic book.

[02:25:54] But if you need specific advice to your role, it’s absolutely talking to people a level or two above who’ve been through this and asking them, what did you have to let go of? What did you have to learn? How do I do that? What do I need to let go of? What do you see holding me back? I’m a huge proponent of getting advice. Getting advice is even better than getting feedback. If I ask for feedback, I’m basically asking you to do something that you may struggle to do, which is criticize me.

[02:26:26] If I ask you for advice. Everyone has advice to share, and they’ll give you the same information, but in a more positive way. So to break through to that next level, it’s identify what’s different about that level and get good at that as opposed to just work harder. I see so many people working brutally hard, but ineffectively.

Ryan:

[02:26:51] When it comes to the most senior promotions, I think there’s this idea of readiness, which I feel like is what you just talked about, which is developing the skills and being in the position where your manager would say, yes, this person fills out the template, I guess, for. For their promotion. But then there’s also the case of limited number of spots and kind of positioning yourself so that you are put in one of those spots.

[02:27:16] And so imagine there’s a case where I’m a senior leader and I’m maybe I’m going for the director position, or maybe I’m going the. The principal engineer position. And there’s two of us both going for at the same time, but we both report to you. How would I position myself so that I get the promotion first?

Ethan:

[02:27:36] Absolutely. Let’s assume you’re both equally qualified for a moment. So you’re not going to win on skill. It’s there in relationship and within relationship trust. So trust. Trust is about, do I have to look over my shoulder and wonder, are you doing things not the way I would do them exactly, but in a way that won’t make me look bad? That will, if you prefer, make us both successful? Do I trust what you’re doing?

[02:28:02] Or do I? Because as a senior leader, I don’t have any time to micromanage you. I’m so busy myself, I need to be able to give you something and assume it will get handled. And trust. I often tell people, interestingly, trust is earned more in bad times than good times. My favorite analogy for this, which is a little bit violent, is when two people have been in a war together and I had to put my life on the line for you or you for me.

[02:28:30] We now have a lot of trust, because if I trusted you, like, if you came through when the bullets were flying, I know you’ll come through the rest of the time. Well, what’s the closest thing at work? This is usually a crisis, a big outage, a big embarrassment, a lost account. If you jump on that and somehow fail, fix it, or show all the effort and it hurts you or it costs you, you cancel a vacation.

[02:28:55] Not that I’m recommending doing that a lot, but if you make a decision that shows that you’re dedicated, I will remember when the chips were down, when we lost the Acme account, Ryan came through. He flew back from Hawaii early and made it. And that’s trust. Second thing is relationship. Relationship. In this case, I’m honestly thinking of, do I like you? And the point is, if I have two relatively equally qualified people and I’m going to suddenly spend more time working with one than the other, why would I not choose the person I enjoy?

[02:29:31] And again, I know some engineers are just. If they were dead, they’d be spinning in their graves. They hate that idea. Look, one person’s a little bit more difficult to work with or even just a little bit less fun, and the other one’s more fun. I have to spend eight hours a day with this person, or six, because we’re not in every meeting the same. I’m going to be drawn towards the person that’s fun.

[02:29:58] And I know that seems unfair. And it doesn’t mean you have to kiss up to everyone. It does, though, mean it’s a good idea to be friendly and happy, right? Like, be, be pleasant. Be, be. Be that helpful, smiling person, because I’d rather spend time with you. I heard it expressed this way by Amazon leadership, by Bezos, I think, which is, when you come into the room, do you suck all the oxygen out of it?

[02:30:27] Like, do you bring the room down where the room’s like, if you’re that person, you’re not going far, you’re not going any further. Whereas if, when you come in, they’re like, Right. Like, oh, Ryan’s here. This just got better. They kind of want you in that room. And, you know, it’s. It’s a. So if you want a simple rubric, do people want you in the room? If they kind of don’t, you’re in trouble. And if they kind of do, you’re doing great.

Ryan:

[02:30:55] I do agree. I think a lot of engineers are gonna. This is gonna piss them off, but that’s okay.

Ethan:

[02:31:00] See, it’s true. What I would say to the engineers I piss off is I am trying to help you. I’m not actually here because you’ll like me. I’m here to get you where you want to go. And that requires, you know, you asked me, could I be transparent? Part of the transparency is sometimes transparency is telling you things that I know you’re not going to like but that are holding you back. And you’re letting the fact you don’t like it let you get your backup and be like, I’m never going to do that.

[02:31:27] Well, okay, but then you’re never going to move up. And I’m not here to apologize for the way the world works. The way the world works sucks, but it does work that way. And I want people to know that.

Ryan:

[02:31:40] When I think of the most senior level promotions, baked into the promotion packets is this idea that you need champions, or sometimes known as allies, which is people at that level or higher that say that person is doing work that fits my. My perception of that level. So because of that, I’ve noticed there’s this artificial mechanism, like, let’s say I’m. Let’s use management as an example. Let’s say I’m a senior manager.

[02:32:09] I want to become a director all of a sudden. If I want to get promoted, I need to make sure that that VP over there knows my work and will vouch for me in the promo committee and that other VP over there. And there are cases where naturally I might need to be visible in front of those people, but there are also a lot of cases where it’s just, I need to fill out the promo packet in the way that we need so that the promo will go through.

[02:32:36] How would you go about that? Let’s say you are a very senior engineer or a very senior manager in tech, and you’re trying to get those allies, but there’s no. At least as far as you could see, there’s no natural collaboration with those people. How do you get them as a champion on your promo packet?

Ethan:

[02:32:55] So I’m going to tell you my favorite way and then I’ll give you other ways. My favorite way is you have your boss on your side, you’ve got that team with your manager where your manager wants to get you promoted. And what I did in those cases is I would email my peers or my skips and I would say, you know, Ryan’s going to be up for promotion in six months and because your group is adjacent to ours, I’m going to need to ask you for feedback.

[02:33:26] So I’m letting you know that you’ve got six months to watch Ryan because I’m going to be coming to you. And number one, me asking doesn’t feel self serving because I’m talking about you. And number two, six months later, when that vp, I’m sure wants to say, well, I only saw Ryan twice. I’m not sure if I can write anything. I have, they know that I’m going to say like, hey, I told you, like I put you on notice.

[02:33:56] Now sometimes I’ll just say I’m sorry it didn’t come together. I don’t know, I can’t provide feedback. But that’s, that’s the best way. And I did that when, when I got good at getting seen. This is important as a manager. I got good at getting my people promoted and that allowed me to attract more talent because, for example, I had a guy who didn’t get promoted and I said, look, you know, I’m good at this.

[02:34:22] I, I will get this done in six months, but you’ve got to give me till the next cycle. But you know, I come through so you have to trust. And he later told me, he’s like, yeah, that sucks. But I, I decided I’d give you six months and we got it done. So that’s the best way. Now let’s assume your manager’s not that on board or they lack that skill or you can’t sell them on it. By the way, if you have good relationship, I’d tell them like, would you consider putting them on notice that you’re going to need feedback?

[02:34:50] If you can’t, the next best thing is for you to go talk to them and say, hey, I’m up for promotion. I’m pretty sure my manager is going to ask you for feedback. You’re a logical feedback provider. What can we do over the next six months so that you’re in a natural position to give me, you know, to give feedback or, you know, how could I keep you informed of my work? What would, and one of the keys here is if you need, you know, whatever Four champions.

[02:35:25] You’re gonna have to ask six people or seven because one of them is going to leave the company and one’s going to get reorged and one’s going to say no. So you have to like over submission, subscribe. It’s like, you know, over capacity, just like anything else you design. And it is awkward, you know, to go to a, like a, another group skip. But the thing is you can frame it to them as hey, you know, this is our process.

[02:35:51] Like, you know, I’m sure you get requests all the time for feedback and you probably get a lot of late last minute requests. I’m trying to do it right and talk to you six months in advance. And so what you’re saying there behind the lines is see, I’m already good. I’m like not causing you to get hit with no notice. I’m here, you know, setting this up in advance, making it easy for you.

Ryan:

[02:36:18] Well my immediate thought, if I was that VP and someone came to me with that self serving request, I wouldn’t necessarily think that’s bad or anything. I think okay, that’s the process. But my immediate thought is this is not a high pry for me because I’m a vp, I got a billion things on my plate and this random person from another org is coming to me to serve themselves. Not necessarily something I care about, but just shows that it’s low on my priorities.

[02:36:47] How do you get them to prioritize that?

Ethan:

[02:36:49] Well, remember there is some truth that’s low on their priority list but they have people they need to get promoted where they need other VPs. So you have a VP too. And that VP this, the VP you went to knows that they’re going to need your VP’s help with some promos. And so the leverage you have without ever bringing it up to them is they do know it’s a system and it’s kind of part of their job. So is it a high priority for them?

Ryan:

[02:37:19] No.

Ethan:

[02:37:21] The only way you can make it a high priority is if you’re able to explain to them some relevance of your work. You’re going to have to give them what’s called wifm. What’s in it for me? You’re going to have to find some way to connect what you’re doing to something they care about at least a little bit. How are you going to make their life easier, faster, less stressful? When these systems work as designed, those VPs do work with your area somewhat and so they should have you.

[02:38:00] You shouldn’t be asking someone who like has no relation your work whatsoever ever. It’s just they don’t really need your level of detail. But you, you can try to find some way to give them a little bit of nugget. I do think these systems work mostly though because vice presidents do know that this is how the system works and that they kind of need to do it. And if vice presidents are good at nothing else, they’re usually good at understanding this is the grease that keeps the thing flowing.

[02:38:34] And by the way, I can write some mild approving verbiage of you very easily. It may look maybe because it was like I cut and pasted and replaced John with Ryan but you know, and replace you know, whatever the ops Org with the, the you know, the customer Org but I’ve written a million of these and I can do it pretty quickly.

Ryan:

[02:38:57] Well, I think a lot of those, those champion. The champion support is a little bit contrived and it’s hard for me without my manager’s support to get a. You said what’s in it for me? Like a legitimate case of what’s in it for that vp because they’re so far, they’re so high up. I mean I know some people who, they’re in this spot, they’re very senior ic their manager is supportive but they lack maybe that proactivity that, that you have.

[02:39:26] And obviously the, the engineer has the proactivity for their own case. So they, they debate, you know, should they go directly to that person? If so how. Because it doesn’t. They don’t have a great case.

Ethan:

[02:39:39] Yeah. And I think, I think that coaching your own manager works pretty well in this sense. Your manager often knows what they need to do or at least is open to it and it just hasn’t made their priority list. Because you obviously care more about your career than your manager does usually. Right. Because it like that’s just human nature. And so the manager means well and wants to help but isn’t. Isn’t getting around to it.

[02:40:04] And so if you just poke them a little bit. The other thing that is a middle ground is you can ask your manager would you send a note to that VP like I know I need to go get them to support me, but I don’t really know them. Can you send them a quick intro that says hey, I have a guy who’s coming up for promotion. He wants to come talk to you. You know, would you take a 15 minute meeting because we’re going to need your feedback.

[02:40:33] It still gives you, you still are doing most of the Work, they’re just sending an email. But that’s a middle ground. You’ve opened a topic though. I really want to point out to all the engineers. You asked me had I ever seen a case where a skip level saved an engineer from being fired. And I said yeah, I’ve seen people moved around. What I never really seen is an engineer or any other anybody else get promoted over their manager’s objection.

[02:41:03] You have to have your manager’s support because I can’t if I’m the big boss. I can’t promote you if the manager doesn’t agree because I’m just creating a mess. I’m telling the manager they’re wrong and now I’m telling them that they have to work with you more senior, you know, even though they didn’t agree, I’m just creating a complete mess. So if the manager isn’t supportive, I think engineers envision like I’m going to do such incredible work and I’m so technically astute that I will be recognized.

[02:41:37] Even though this manager is a bozo, it’s not going to happen. It just, it can’t. And so you either have to get your manager on board or get out from under them and find somebody who is on board. Now a plug for people like me. Go find the managers who. I made a profession of getting people moved up, not undeservedly, but fast. And hell, that became a huge selling point, right? Go to Ethan’s org, he’ll make deals and get you promoted.

[02:42:08] That was my recruiting tool. Come work for me and I’ll make it happen. Because it was known that I would back people and get the job done. I could attract talent and I could keep talent. I had one guy, my biggest star employee began as a mid level engineer, what Amazon calls a level five. He went to a senior engineer, he moved to manager from manager at an equal sort of lateral promo. He went to senior manager, then to director and by that point he caught up to me.

[02:42:41] I was a director too. So we were out of space and he went to found his own company. Now he was a top performer, but three promotions in eight years. We worked together eight years. You know, we’re still close friends of course, but he powered my career and I powered his. But that example also brought other people like, oh, look at, look at what happens in this.

02:43:01 — How to avoid politics

Ryan:

[02:43:01] Or we’ve been talking about politics for almost three hours now. And I can feel many, at many points people listening are probably thinking, I hate every single moment of this. I hate this. Some people are probably listening because they feel like they have to, but they wish they didn’t because have to know how the system works. And I know it’s inevitable a little bit that, you know you can’t avoid politics completely.

[02:43:31] But let’s just say for someone who really just despises politics for all it is, what could you do to avoid politics if you were working at a company?

Ethan:

[02:43:42] I think number one, it’s about choosing your leader. So if you can find a leader who shares your disdain of politics, hopefully that maybe comes from a similar tech background and wants to reward they believe as much as you do. It should be all about that work. Find that leader. That’s one thing you can do. The second thing is go work on the hardest area you can get into where expertise is really going to matter and you can stand out the most.

Ryan:

[02:44:12] good,

Ethan:

[02:44:12] solid engineering work but that lots of people can do, it’s harder to stand out. I remember Andy Jassy. He was CEO of AWS at this time, but in the past he had been, I’m sorry, before he was CEO of all of Amazon and he was talking about principal engineers and he said principal engineers are normally expected to like advise others and mentor and do architecture. But these guys, these few guys working on aws, we just need them solving the hardest problems.

[02:44:45] And so I absolve them of all of that other stuff. Well, for those engineers who just wanted to be. I don’t want to review architecture, I don’t want to give feedback, I don’t promo. I He was the perfect boss because he felt there they were working on such hard problems, he couldn’t spare them to invest in the rest of the org and he didn’t care. So you’re looking for that situation where your work is so essential because.

[02:45:14] Absolutely. Look, you’ve asked about difficult people and I’ve given you all these rules about it’s great that your boss, it’s important your boss like you. If you have a skill that no one else has and that skill is valuable, you will get exceptions. I remember Andy again when he ran aws. I remember him talking about a high level employee who a couple people criticized him and said, that guy’s not working that hard.

[02:45:47] You know, he’s not really working at an Amazon level. And Andy said, yeah, I think you’re right. But he’s in a role where we need him and I think he knows we need him and so we’re just going to have to live with it like he does. The one thing we really need. And you could kind of tell Andy was like, Andy’s one of the hardest managers I ever worked for. Just very demanding. But in this case he’s like, yeah, he’s got the leverage is what it is, you know.

[02:46:18] And he was very like good on him, was kind of his, you know, like, man, he’s kind of got us. We’re just gonna have to live with it. That’s what you have to do if you don’t want politics is you’ve got to be, you’ve got to have that essential skill again. You have to have some kind of leverage. So you have to literally live up to Cal Newport and so good they can’t ignore you.

Ryan:

[02:46:44] Earlier in our conversation you talked about how when everything’s growing, it’s positive. Some when everything is kind of stagnant in an organization, it’s a little more zero sum where people are battling. So I can also imagine a growing company is much less likely or will have less politics. I think some amount is unavoidable.

Ethan:

[02:47:04] Yeah, I like to give people an extreme to think about. And my extreme, I often point out, is the post office. Like what’s going on, good at the post office. And the only kind of roles are carry the mail or be the postmaster and there’s one postmaster. And that until that person leaves or dies, you don’t get that job. And so it just feels like growth changes so much of that because it’s not 25 people wanting one promotion.

[02:47:33] It’s, you know, lots of opportunity. And I point out during the 15 years I was at Amazon, the staff size of the company I joined, it was 14,000 people. I left, it was 1. 4 million. So it grew a hundredfold, which is insane. As a result, I was able. My first team there was six people and nine years later I had 800. That kind of growth opens up doors that in a more normal company that’s maybe growing 5% a year or 10% a year, it’s just never going to happen.

[02:48:12] So I strongly believe find high growth.

02:48:15 — Advice for younger self

Ryan:

[02:48:15] Last question is if you could go back to yourself at the beginning of your career and give yourself some advice specifically focused on this, I guess navigating corporate politics. Is there something that, that stands out to you that you’d say, yeah, I

Ethan:

[02:48:31] God, I wish I could go back to early me and share the idea that it’s not just being right, it’s, it’s not just being right, it’s not just being technically astute. There’s all this other stuff going on and wake up to it. I think I was blind or deprioritized it and thought, well, that’s. That’s management stuff. And even as an individual contributor, understanding, well, management stuff does control the world.

[02:49:02] I wish I had been more aware. So I guess the advice to get it down to a single sentence is take it seriously that your relationships are what actually control your progress.

Ryan:

[02:49:15] Awesome. All right, well, thank you so much for your time, Ethan. I really appreciate it.

Ethan:

[02:49:19] Thank you so much for the chance to do this a second time. It’s been so much fun and the time has flown by.

Ryan:

[02:49:24] Awesome. And do you want to plug your stuff real quick?

Ethan:

[02:49:27] Well, I think if you’ve gotten something from this discussion and you want to know how to navigate your career, that’s what I do all the time. I teach classes on it. I have a newsletter on it. So if you want help. My goal is to help people succeed rather than bang their head against walls. If I’ve done some of that in this podcast, I can do more of it with more time. All right. Three and a half hours in the chair, baby.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:37 - How he convinced Google leaders

00:09:26 - Building the MVP

00:11:43 - How he made time for Kubernetes

00:25:28 - Technical details on building Kubernetes

00:38:46 - Rallying the open source community

00:50:01 - Scaling Kubernetes up for AI training workloads

00:55:31 - Reflections on getting a PhD

01:00:22 - The inevitable trajectory of software is death

01:04:16 - Top book recommendations

01:05:22 - Advice for his younger self

Transcript

00:00:37 — How he convinced Google leaders

Ryan:

[00:00:37] Let’s start with Kubernetes because that’s super interesting. I don’t fully understand the business motivation. Like let’s say I was your director or something like that and you, you came to me at this and you said hey, let’s do this for everyone. I don’t fully understand what would be in that strategy doc or that, what would you say in that meeting that would say here’s the impact for Google if we invest in building this for the industry?

Brendan:

[00:01:09] Yeah, it’s funny because the hardest part actually of the project I would say in those early days was actually articulating that. And I think it was really clear in our heads. But figuring out how to convince people was tricky. And I think there were a variety of different ways that we articulated why it was important. One of them was related to the MapReduce white paper. So MapReduce at the time, especially Hadoop and Big Data were a big deal.

[00:01:45] I think that other things have kind of replaced them at this point. But like MapReduce was a big deal and the big data revolution or whatever they called it. And you know, Google had written the original white paper, but Hadoop was an open source project that Google had nothing to do with and got no credit for. Right. And they just read the white paper and they re implemented it and it’s not the same.

[00:02:12] It’s similar but it’s not the same. And so part of the argument was like, look, we have this cloud and we want to be influencing the technological landscape. If all we do is kick out white papers, we’re not, if it doesn’t run, if it’s not something that people can run, we’re not going to be in the driver’s seat. And so that was one of the arguments. I think the Other couple of arguments were like, why Containers?

[00:02:43] Why not why, why people are using VMs, why containers? And a lot of that was talking about, look, the demands of writing software. We know internally, we know from doing this internally that the demands of writing reliable software necessitate having systems that are sort of like autopilots for your application. And we know that this is something that, as software becomes more and more critical to more and more businesses, this is something that they’re just going to have to have.

[00:03:14] And so that’s sort of the why Containers part. And then I think that the third part was the like, why Open Source, right? And in some states that’s like the most interesting conversation because people are like, wow, this would be, you know, you’ve convinced me, right? Like, you’ve convinced me, we should build it, we should make it available to the world. It’s something that they’re going to find useful.

[00:03:34] But wouldn’t it be so much better if they could only use it on our platform? And you’re like, yeah, that’s absolutely the case, but you can’t win if you make it only on your platform.

Ryan:

[00:03:48] Why is that?

Brendan:

[00:03:49] Well, because there’s other platforms out there, right? And so if you make it an exclusive, then the people who, for other reasons are on other clouds or on, on premise, they’re shut out and so they’re going to just go build an alternative, right? Like the open source. It’s sort of like the Linux. You know the reason Linux won, right? It’s because Linux could go anywhere, right? The whole reason that open tech and open ecosystems win is because the majority of people are going to be not on your platform.

[00:04:25] Like, if you’re, if you’re not the leader and you know, GCP was not the leader, then the majority of people are not going to be on your platform. And so if you make it such that the majority of people can’t use your thing, they’re just going to ignore you and then they’re going to go build their own, right? Whereas if you go and build it and you build it for everybody, but you make sure that it’s awesome on your platform, then you have a chance of attracting more people to your platform than otherwise.

[00:04:51] And then I mean, also in some ways it’s just like the aesthetic of the time also, right? It’s like if everybody’s using Linux and everybody’s using Docker and everybody’s using these programming languages that are all open. Like, if everything else is open source, you don’t want to be the thing that’s not right. And there’s only a few places where, like, where that hasn’t been true historically in technology where you could be different and still succeed.

[00:05:15] And you have to be so differentiated that. And I don’t think that we were that differentiated. You have to be so differentiated such that people are like, oh, actually I want that thing so bad, I don’t care that it’s closed.

Ryan:

[00:05:27] So at the time, just thinking what was the competitive landscape? I guess if I, if I remember it was, I mean, AWS was dominant, they were there first and doing very well. GCP at that time, probably an up and coming company or I guess offering. And so my understanding then that the idea is let’s pull market share away by giving kind of open source distributing Kubernetes to more and more developers and then they’ll be more open to kind of migrating or using GCP because you’ll make a.

Brendan:

[00:06:00] They’ll pay attention, right? They’ll pay attention. And also like you change the dialogue too, right? Like tail light chasing is hard, right? Like if, if someone else has built VMs and everybody’s using VMs and all you’re doing is saying like, well, we’re building the same thing that they have, only maybe it’s a little bit better because we do something else over here or you know, whatever. Like that’s a hard market strategy to articulate.

[00:06:22] But if you create a brand new playing field where you are the thought leader, suddenly people are listening to you. Even if they’re not using it on your platform, they’re listening to you. And that gives you way more voice. You get a lot more voice in the market. It changes the narrative, it changes who people are listening to. And so that control of the story is an important aspect of I think, how you break through that dynamic or you try to break through that dynamic.

[00:06:51] Obviously they’re still in third at this point. So didn’t work out, but I mean, I wouldn’t say it didn’t work out. I think it worked out in general, but it’s still hard. Overcoming those kinds of market dynamics is hard. And I think the other thing that happened is everybody in the cloud consolidated around it. And so now Kubernetes is just sort of a utility everywhere.

Ryan:

[00:07:12] That perception benefit of being the dominant offering, which, I mean if you, if you look at what happened in hindsight, it makes a lot of sense. I mean that is what happened and it’s, it’s a wonderful benefit. But I guess when you were looking forward and you Were talking to leadership. Were you cognizant of those benefits and saying we need to do this because we’re going to kind of become the dominant offering and it’s going to have all of these optics benefits?

Brendan:

[00:07:44] I mean, I think we absolutely wanted to make sure that we were front and center in terms of thought leadership. And we definitely articulated that. Being in a thought leadership position is valuable.

Ryan:

[00:07:56] It’s interesting because I feel like it’s hard to quantify. I mean, if I was in that meeting and we’re trying to make a call, how much is that worth?

Brendan:

[00:08:06] Yeah, well, I think you have to also realize though, at the time it was pretty cheap, right? It was like eight or nine engineers and we kind of. And this is in some sense, I mean, it’s both a blessing and a curse. Like, part of the reason why we articulated and argued for having such a distinct brand, where the Kubernetes brand was separated from the Google brand, was that it kind of gave us freedom to fail also.

[00:08:29] It was like, hey, if these eight people go off and do something and it turns out to be stupid, like we’ll just kill it off. And it won’t have, like, it won’t, you know, it won’t, it won’t damage the broader perception of the cloud. And so I think there’s that benefit of the open source part of it too. It helped with adoption, right? Like, it helped us and especially as we went to like the Linux foundation and things like that and truly established like an independent entity, it helped ensure that, you know, people like Red Hat or Azure or AWS could take a bet on Kubernetes and feel confident in that bet.

[00:09:05] But it also was an insurance policy against failure. And also, to be honest, it simplified a lot of things too, because we were competing against startups at the time. Docker is a startup. They can be way more agile than a big company. And so by virtue of being a separate entity, we could be a little bit more agile also.

00:09:26 — Building the MVP

Ryan:

[00:09:26] The earliest conception of this project was you and two others kind of hacking something together.

Brendan:

[00:09:32] Yeah, it was a demo. I mean, it was sort of a demo almost. It was like, look at what we can do if we just smash a bunch of existing open source tech together.

Ryan:

[00:09:39] What did that demo do?

Brendan:

[00:09:42] I mean, it was basically like a basic cube control. It was like, hey, here’s a container I built. I mean, at the time you had to explain Docker to people. You were like, hey, here’s Docker. I used it to like build this Container image and then you could run it and deploy it and see that it had gotten distributed across a bunch of machines and that you could load balance to it because you’d hit a single endpoint and it would go, I’m replica one.

[00:10:06] And then you’d hit reload and go, I’m replica three. So it showed that it was replicated and then basic health checking so if you killed it, it would come back. And a V1 to V2 upgrade, that was about it.

Ryan:

[00:10:21] How long did that initial MVP take to build?

Brendan:

[00:10:26] I wrote it in, I don’t know, a little under a week maybe, something like that. I mean, like, I don’t, don’t work on the weekends, so maybe five days, Four days. Five days.

Ryan:

[00:10:35] And did you drop all of your existing work? Because I’m imagining you had existing project work and this is kind of extra credit stuff that you were working on.

Brendan:

[00:10:45] Yeah, well, I mean, I wouldn’t say I dropped it, but like in a timescale of that timescale, like you can kind of like slack on it a little bit, you know, like you could be sick for a week. I mean, I guess the thing is like you could be sick for a week, you know, And I’m not saying that that’s not what we did, that’s not what I did, but like, but, but there was enough flexibility in the system that like you could hack it together.

[00:11:10] And I mean, believe me, it was hacked together, right? Like every possible shortcut to take and every. And you know, I think one of the things I’ve been good at historically is integrating other open source projects together, seeing how you can take stuff off the shelf and put it together. And so a lot of the nuts and bolts were pieces that we could take from other open source projects and kind of combine together with glue and glue code to give the feel of it.

[00:11:42] So that helps too.

00:11:43 — How he made time for Kubernetes

Ryan:

[00:11:43] I think a lot of software engineers, when they hear this kind of story, they think, oh, I have my existing responsibilities and I can’t necessarily go off and build this thing, even though I think it’s a great idea. Do you have any advice for someone who has that opinion?

Brendan:

[00:11:59] Yeah, well, I mean, I think that what I would say, there’s two things, two. I have two answers to that. One is advice that I’ve always given to every single person that’s ever worked for me, which is I believe you can hide order 10% of your effort from your management. Right? So like, you know, there’s there, you have slack, you have the ability to slack no matter What? Right. And you know, as you get a bigger and bigger org, actually the percent that, what you can do with that 10% actually increases.

[00:12:36] And a lot of really influential good ideas that I’ve had have come out of that. I mean, it’s another, I mean, it’s kind of a flip way of saying, I want to empower people locally to make local decisions that they think are optimal for the business without having to consult up the chain, without having to ask permission. And you tell people, when you tell them that you’re like, by the way, you’re also going to make a bunch of bad decisions and you’re going to waste a bunch of time.

[00:13:02] And so you have to be comfortable with this idea of like, I’m going to try some ideas. Some of them are going to fail, some of them are going to succeed. When I look back retrospectively, the ones that have failed were effectively a waste of time. And it might be the difference between exceeded expectations and a met expectations. You don’t want to drop below meets expectations, but it might be the difference between an exceeded expectations and a meets expectations.

[00:13:25] And you have to be comfortable with the notion that you’re going to bet five times and the payout from one of them hitting is going to be way better than the grinding to get that exceeds every single time. And I think there’s equally valid paths where you don’t do that. And I think you have to be the right kind of person who’s willing to take that kind of chance. And that’s not everybody. And that’s okay.

[00:13:54] And I think the other side of it, but I always remind everybody also though is like, people say things like that to me sometimes and I’m like, so do you play Call of Duty? Do you watch Netflix? Do you watch YouTube? Because I pretty sure there’s probably 10, 15, 20 hours in your week at least when you’re doing something that’s not work. And I can tell you in that time period, I wasn’t doing anything except for this and work and a little bit of family and sleeping.

[00:14:25] And so sometimes it’s also about saying, well, what are you willing to give up? You know, to, to have the space to do that. And, you know, I’m not a big, like, I, I, I’m not a big like, work all night kind of person. But, like, it does mean, like, maybe not going to watch YouTube for a while, maybe not going to watch, you know, sporting events for a while.

Ryan:

[00:14:44] That makes sense. And I mean, on that second point in this case, the returns of this project were exponential, obscene. If you put in two times the time for a year, you get 20 times the impact. So it just kind of makes sense in terms of investment of time.

Brendan:

[00:15:08] Well, and also, I think, I mean, I found personally that it was addictive. Right. I mean, I think I benefited from two things. One is I really like to write code, right. Like, I enjoy it as an activity. And so if I’m choosing between Netflix and coding, I’m actually pretty happy just coding. So that’s a benefit for me. And then for me anyway, once people start using it and they’re excited about the project and they’re putting issues on GitHub and all of this kind of stuff, I’m just addicted to it.

[00:15:44] I just want to close that issue. I want to help that person out. I want to. I’m going to go till I’m falling asleep on my laptop. And that’s just because I enjoy that. That’s why I’m in the industry. So even in the moment, I wasn’t. I mean, I was definitely not thinking about like, oh, here’s this payout that I’m going to get for the rest of my career. I was definitely in the, like, wow, I just want to keep this thing going.

[00:16:10] I want to keep this, you know, I want to keep this rush going.

Ryan:

[00:16:13] It sounds like it took a while for you guys to get buy in from leadership.

Brendan:

[00:16:17] I think there was a solid six months of going from a very hacky prototype to something that, like, legitimately we thought somebody could take and use and laying down the right kind of groundwork for that. You know, there’s a lot of little details that you have to get right along the way. And, and it’s always nice also because, you know, a lot of, a lot of the people who we brought in in the early days had built similar systems before.

[00:16:52] And so they were having this opportunity, this kind of clean room. It’s rare in your life as an engineer to get a clean room opportunity to rebuild something that you have ideas about how it could be better. It’s like getting a second chance, you know. And so that was also, I think, really attractive to people because it’s suddenly like, oh, wow, like this is a clean room. We don’t have any users right now, so we don’t have to be fixing bugs because some big company who pays us a lot of money is asking for something or whatever.

[00:17:21] We’ve got this clean room time and we’ve all spent a lot of time thinking about what the system could look like. And so now we get to go and just build the thing that we imagined.

Ryan:

[00:17:33] You mentioned that first point on hiding maybe 10% of your bandwidth from management. And I mean, that’s. That’s super interesting to me. What does that look like in practice?

Brendan:

[00:17:45] Well, I mean, I think that what it means is that you should always have sort of like a side project that you think is relevant. Right? Like, you should always have something that is. That nobody told you to do, but that you think is important that you’re working on.

Ryan:

[00:18:01] I see. So it’s kind of like, I remember at Google, I don’t know if they still do this, but at the time there was 20% time.

Brendan:

[00:18:07] Yeah, it’s similar to that kind of idea. Yeah, yeah, exactly.

Ryan:

[00:18:11] Okay, so when you say hide, you don’t mean hide. You say manage expectations so that your manager is also okay with you working on another project.

Brendan:

[00:18:20] Oh, no, I actually do mean hide. Right. Like, don’t ask permission. Right. Like, sooner or later you’ll show it to them. But, like, it’s pretty like you need a solid, I don’t know, couple months or whatever to get to, like, a place where it’s something you could show to somebody. Right. And so, like, it’s all about saying, like, I’m not going to ask permission. I’m going to go build something that I think is important and useful.

[00:18:43] And then obviously, when it comes time to launch it or whatever or put it out there, then you do have to ask permission. And so then, yeah, you say, hey, I built this thing, but you’ve had that time to get it from. Because, I don’t know, I feel like it’s hard to articulate the value of something like in a doc or in a PowerPoint. It’s way more effective if it’s like a running thing that you can, like, somebody can interact with.

[00:19:13] Right? So getting that time to basically build it up into something that’s real and could ship. Because also, like, in some sense, your manager’s always assessing, like, well, should you spend your time on that or should you spend your time on this? Right. And by building it, you kind of like force their hand because it’s no longer like, should you spend time building this or should I spend time building this?

[00:19:36] It’s like, I’ve already built this. Do you want to ship it? And that’s in some ways a much easier decision. I don’t know about easier, but it’s not an either or. It suddenly becomes just sort of about, like, is your idea good? Because the work is done.

Ryan:

[00:19:51] In the happy path, I feel like it’s a great idea. You launch this thing has impact. It’s great. What about in the case that you work on this thing and you didn’t tell anyone about it and then no one cares when you launch it or it’s not as good as we thought.

Brendan:

[00:20:07] Yeah. And that’s the flip side, right? You have to be comfortable. I mean, as I said, you have to be comfortable with that idea that you’re going to waste some time and maybe there’ll be a waste time in the sense of like, wow, I could have been watching Netflix or know, whatever, and I. Instead I wrote a bunch of code that nobody liked. It could be wasting time in the sense of like, wow, I could have, you know, could have gotten promoted.

[00:20:33] I could have done enough work to get promoted and I didn’t because I thought this was the great idea that was going to get me over the hump. But it wasn’t. And I think you just have to be comfortable with that. Like, you know, it’s taking. I mean, it is taking a risk. It’s not unlike in some sense like doing a startup or something like that. It’s taking a risk and you can’t assume. I think sometimes people go into any of these sorts of things and they’re like, oh, I will have the idea and it will be amazing and then it will hockey stick.

[00:21:03] And I think if that’s your mindset when you go in, you’re probably setting yourself up first in disappointment. You have to go in with that mindset of, I think this is good, I’m going to try, but I’m okay if I fail. I know that I’m making an explicit choice here.

Ryan:

[00:21:20] I also imagine at some point at the highest levels of engineering ladders, you need to take that risk to get promoted to higher levels. For instance, if you’re a staff engineer or senior staff engineer and you ask your manager, how do I get promoted? They’ll often tell you, you need to figure out what that project is. Because I can’t just hand this to you because it’s starting to become more ambiguous.

Brendan:

[00:21:46] Oh, yeah, for sure. Yeah.

Ryan:

[00:21:48] I could totally see that this becomes a necessity at some point if you want to kind of grow. And I mean, this project also got you promoted as well at Google, right? From staff at some point.

Brendan:

[00:21:59] Yeah. No, I mean, certainly my career absolutely benefited from the success there. Absolutely right. And I think you’re right that there’s also this aspect of at a certain point, you’re just expected to be the person who knows enough to come up with the really good ideas. And that’s just the expectation. It’s no longer about can you execute on the ideas that other people give you? And that’s a big part of it as well.

[00:22:22] And I think there’s also the other thing I tell people sometimes when they’re thinking about getting promoted is if you create the idea yourself entirely, it’s blindingly obvious that it was you who did it. If you succeed and have impact in something that is a bigger project or someone else’s idea, you can still have a very successful career, but it’s a little bit harder for it to be directly attributable to you.

[00:22:57] But again, I think that again, it’s a roll of the dice at some level. There probably are people out there who’ve tried over and over and over again and just have never had the right idea or just had the right idea at the wrong time. I mean, I think one of the other things that is interesting about innovation that is disruptive is that it’s a combination of being the person who has the idea and being in a time in which the idea can take off.

[00:23:27] So you could have the idea, but it could just be the wrong time and it won’t do the same and it won’t go in the same direction.

Ryan:

[00:23:33] That point on promotions, I think, I mean, if you create the scope, not only is it obvious that the credit should go to you, but it also feels kind of permissionless, like you don’t need to wait for, I guess, management or someone to give you the opportunity. You can kind of create it. And so you have a lot more control in that process. One thing on the business strategy, I guess before we leave that for kubernetes, is at that time, Borg to me felt like a competitive advantage for Google, like some secret infrastructure sauce.

[00:24:08] I would have thought people would be kind of worried about giving away any part of that to the industry. So what was the thinking there? How. How did you convince people that hey, it’s okay?

Brendan:

[00:24:21] Yeah, I mean, I think that there was a little bit of that and I think, you know, sort of to be a little bit make it into a little bit of a joke or whatever. I sort of, one of the things I said to people was it’s not like you men in black flash people as they leave Google. It’s not like everybody comes to Google and it just stays there forever. Right. And in fact, as we talk to people at Facebook, as we talk to people at Twitter, we talk to people at other scale out tech companies.

[00:24:47] They were all building this stuff. It wasn’t really a secret. And there was also, I mean, mesos at the time was, you know, not the same, but similar. And like you could just see that there was going to be an open solution. And so in some sense part of the argument was like, look, there’s going to be an open source solution. Do you want it to be one that we can influence or not? It’s not like, do you want there not to be an open source one or not there’s going to be an open source one.

[00:25:06] Do you want it to be ours or do you want it to be someone else’s? And it’s just reframing the choice. Right. And making it clear that people understand that that is the choice. Right. That you don’t get to choose the proprietary option because it’s just not viable.

00:25:28 — Technical details on building Kubernetes

Ryan:

[00:25:28] When you were building that MVP for the Orchestrator, well, how’d you decide? Because there were no customers or anything like that. So how’d you decide this is the minimum set of features that we need for this to before we launch it?

Brendan:

[00:25:42] Sure. Well, I mean, I think absolutely, you know, we benefit from the fact that there were three of us working on it. Right. So Craig was a great product manager and Joe was a great engineer and fantastic at API design. And I could write code fast. Basically, I think is sort of the, if I had to sort of stereotype all of us. You know, that’s the like, Craig was the product business guy and Joe is the like, I know how to design, really good at design kind of person.

[00:26:11] And I was basically like, I can hack prototypes, like there’s no ways to know tomorrow. And I think we reflected a lot about our own experiences. And also we had seen the pain of people deploying into traditional VM infrastructure. And so we had that kind of knowledge of the pain that people were going through. And then at the time there were people like Netflix who were talking about immutable infrastructure and they were kind of advancing some similar concepts.

[00:26:47] And so there was also kind of like a broader movement happening that we were taking part in. And so obviously there were literally no customers, but in some sense there were customers, they just weren’t customers yet. So it’s not like creating something brand new. I guess in some sense. I really don’t feel like Kubernetes was something that was brand new. I feel like it was a coalescing of a lot of ideas that were kind of circulating in the industry at the time.

[00:27:19] And it just became an anchor point and a really good expression of those ideas.

Ryan:

[00:27:23] So when you talk about, I guess you wrote a lot of code quickly, did you write most of the code, I guess for this initial MVP?

Brendan:

[00:27:34] Yeah, I don’t know what the number is, but high 80s percentage, maybe more of the original code. And I think I’m still number. I mean, I haven’t contributed significantly to Kubernetes in a while and I’m still, I think, number five on the overall contributor, you know, overall contributor list on the GitHub commits. And I was number. I was number one for a long time.

Ryan:

[00:27:55] After writing that much code for Kubernetes, which part of this system was the hardest to build?

Brendan:

[00:28:03] I think I’m going to say, like. Because I don’t think any of this specific code was that hard. I think that the hard part was the decision that we made early on that it was going to be a really loosely coupled system. And so it’s very. Which is great for resiliency. Like we made this decision around very loose coupling, a lot of independent actors taking actions. There’s all these control loops running all over the place, which is really good for resiliency.

[00:28:36] But when things go wrong, it’s really hard to figure out why it went wrong because you’ve got 15 different processes that are all having to work together to achieve an outcome. And so you can see that the outcome wasn’t achieved. But then you’re like, okay, but what happened? And now I have to sift through a bunch of different logs and a bunch of different operations of executables and sort of reconstruct in time what happened.

[00:29:03] And especially early on, we didn’t have very good. We didn’t have very much consistency around logging. We didn’t have very good consistency about events and things like that. And so I think the hardest part, I mean, this is the hardest part of anything, I guess.

Ryan:

[00:29:22] Because they are all distributed everywhere.

Brendan:

[00:29:25] Yeah. And everything’s out of time sync and I mean. And hopefully you logged the right things. But a lot of time early on, like you didn’t log it. Like, you know, and because it’s an interaction effect, it’s hard to reproduce. Oftentimes it’s hard to reproduce the problem. Like if the problem reproduces easily, then it’s pretty easy to fix, even if you don’t have the logs, because you just go and add the logs and then you do it again and you see what happens.

[00:29:52] But for Problems that are transient, that because of it’s just a race condition between two or three different things happening. You can go in and add the extra logs, but then you have to figure out how to make it happen. And that can be pretty tricky. And also, I mean, I’ll say the other thing is we were all learning Go. So maybe the other thing was there’s some gotchas in Golang and we were all kind of like learning all the gotchas on the fly.

Ryan:

[00:30:19] I would have thought there’s something that’s controlling everything, right? Like there’s some leader or maybe some nodes that are looking at one of them to kind of coordinate everything. Like leader election if the leader goes down, I would have thought would be some really challenging problem.

Brendan:

[00:30:37] Yeah, I mean, well, I think the reason it wasn’t that big a deal was because we relied pretty exclusively on Etcd to do that for us.

Ryan:

[00:30:50] Etcd is another open source component?

Brendan:

[00:30:54] Yeah, Etcd is an open source. It’s really part of Kubernetes now. But at the time it’s a RAFT based consensus system key value store. And so it was a pre existing piece of software that CoreOS had written that implemented the RAFT protocol because at the time, because Paxos was the original for this and Paxos is really hard to implement because the algorithm is really complicated and nobody understands it.

[00:31:25] Well, probably somebody understands it, but not a lot of people understand it, but it’s provable. And then right around that timeframe people had come up with Raft, which is a provably correct consensus algorithm, but it’s way easier to implement. ETCD implemented the RAFT protocol and then gave you this consensus reliable store so that you could do, you had multiple replicas and it would do the consensus there.

[00:32:00] And it doesn’t do leader election exactly, but it gives you enough primitives that doing leader election is relatively straightforward. And I guess I would also say that two things about that. One is we decided to force all of the access through an API server. So nobody had. And this was actually something I pushed really hard initially, but I think the general agreement, but I definitely pushed for it, we was that nobody got to use Store, everything had to be remote store.

[00:32:33] Like nobody got to write stuff to disk themselves. Every piece of the system had to use the API server and had to use Etcd behind the API server as the way that it did any sort of persistence.

Ryan:

[00:32:48] What’s the main benefit of that?

Brendan:

[00:32:49] The main benefit of that is that everybody gets to restart all the time and they just come up and they work. So you don’t have to worry about corruptions, you don’t have to worry about schema changes, you don’t have to worry about any of the like. Everybody was effectively stateless except for the database. Like there was, there’s the ETCD consensus algorithm database and that’s the only place where there was state.

[00:33:11] And so as a result, the whole system was just a lot easier to make stable. The downside of it is it leads to this loose coupling where it’s a bunch of independent loops mediating everything through this storage layer, which made the debugging part harder. Those are the trade offs. If you have a complete log of I’m in. If you think of it as a state machine, it’s much easier to understand where you are and where you got to if you’re in a state machine.

[00:33:49] But state machines are a nightmare to make reliable. They’re easy to debug, but they’re hard to make stable. Whereas the system we built was designed to be stable, but hard to debug. The trouble with the state machine is a state machine says the world looks like this and unless you get it exactly right, sometimes the world looks like something you didn’t imagine. And at that point you’re kind of screwed because your state machine doesn’t know what to do.

[00:34:20] Whereas because we had these control loops that were based on a desired state and a current state and trying to drive the current state to the desired state. Like no matter where you woke up and found yourself, you kind of knew where you were supposed to drive to. And that. And that’s the stable part. That’s the stable part of it is that like it didn’t really matter where the system got itself. It was always trying to drive towards the desired state.

[00:34:48] You know, inspired honestly a lot by like control theory from robotics actually. Like.

Ryan:

[00:34:53] Oh, like PID.

Brendan:

[00:34:54] Yeah, the same idea. Right. Like you could imagine if you tried to write a PID controller to balance a beam with a bunch of if else loops. It just doesn’t work very well.

Ryan:

[00:35:04] And this kind of reminds me, because I was reading in some of the design, I guess it’s a feature of Kubernetes, is that it’s declarative rather than imperative. So you kind of just say, I want this to be true, I don’t care how you get there, instead of saying, just do X, Y and Z. I’m curious the pros and cons of that, that design decision.

Brendan:

[00:35:27] Well, yeah, I mean, and that was something that was happening broadly in the industry, like that’s a part of the whole like infrastructure as code movement that was happening at the time. So we’re not the only ones who said that, but we definitely embraced it. You know, I mean, I think the benefit is, you know, you have clarity about the way you want the world to work. Right? It’s not like if you, if you, if you execute a bunch of instructions, start this, run that, do this, you’ve done a bunch of stuff in pursuit of some objective, but you never wrote down what the objective was.

[00:36:04] There’s no record of what you were trying to achieve. I’m trying to create a website. Well, you didn’t write that down. You just took a bunch of steps with a declarative approach. You actually write it down, you say, I’m trying to create a reliable website. And here’s what a reliable website looks like. Hey system, could you take the steps to get there? And so you have that record and it obviously makes it easier for things to be self healing because if you’ve written it down, now I know where I’m supposed to go back.

[00:36:34] If I get perturbed from that state, if something fails or something restarts, well, I know where I’m supposed to go back to. And similarly it has side benefits of once I write it down. Well, I can apply code review to it, I can apply unit tests to it. There’s a lot of the mechanics of how we do software development that apply once you write down that declaration. So a lot of those are the benefits.

[00:37:02] I think the downside is probably just complexity in comparison to going click click through a wizard or whatever in a GUI learning. And you know, everybody complains about the YAML and I have to learn all this stuff and you know, like it does introduce a learning curve. Now I think fortunately at this point there’s enough education material out there that it’s not and Gen AI too for that matter, that it’s not that bad a learning curve.

[00:37:33] But certainly in comparison to what people have done before, that’s probably the biggest downside. But I don’t know, I think that the upsides are like up here and the downsides are like way down, like there’s, there’s not a lot of downside.

Ryan:

[00:37:45] I see. Yeah, I could also see that being helpful for I guess if you want to optimize anything under the hood because you’re just making a promise to people that this is going to happen, but if you want to do it in a more efficient way or something like that, then I guess it just gives you all the Power to do so.

Brendan:

[00:38:02] Yeah, well, I mean, and that does make things like machines failing a lot easier because people don’t say, run this on this machine. They just say, I want three of this to be somewhere. If a machine fails, well, it just moves somewhere else because the application isn’t tied. Because I can’t. In some ways, I don’t know your intent. If you log into a machine and you start a process on that machine, is it because you wanted a process or is it because you wanted a process on that machine?

[00:38:31] I don’t know. You didn’t tell me. And so if that machine fails, what should I do? Well, I don’t know. Right. But if I. But if I know you said, hey, I just want three replicas, well, then I know it doesn’t matter that it’s on that machine. It could be on a different machine. You’d be just as happy.

00:38:46 — Rallying the open source community

Ryan:

[00:38:46] I want to shift a little bit to kind of when Kubernetes was scaling, and it sounds like a large part of this was getting buy in from other companies and other people. And so how did you get the buy in from. I know OpenShift was an important part of Red Hat and other companies that join on. How did you sell those companies that Kubernetes is what you want to use?

Brendan:

[00:39:14] Well, I think for a lot of them, especially in the early days, it was kind of that quote around undifferentiated heavy lifting. They had some other objective. OpenShift was trying to build a platform as a service or, you know, they were, you know, a lot of our early users who were also contributors, you know, they were trying to build some sort of reliable web service or something like that. Right.

[00:39:36] And so it was like, well, we’re going to have to build this thing anyway. Why don’t we all build it together? And we don’t really care because we don’t think that’s our value. Like our. We don’t think our value is tied up in that layer. So we’ll go contribute to your thing because we’re going to get more value out of the collective than out of trying to do it ourselves. And so for a lot of the early partners, that was a big part of the argument was like, hey, we’ll let you in.

[00:40:03] And part of that is making sure that they understand that they’re going to like that they’re going to be equal partners where it’s not like, because it’s one thing to take a dependency on something, but then you’re kind of like taking dependency on someone else’s roadmap. And so it was really important also to say, hey, you can come take a dependency on us, but also we’ll give you a seat at the design table.

[00:40:27] So when you need new features, you can contribute those features. And here’s what we’re trying to achieve and it matches up with your roadmap and that kind of stuff. So I think that’s how we approached it. And then over time, people became more and more interested in being part of it because there was a growing ecosystem. So when you look at like networking providers or storage providers, as their users were starting to become Kubernetes users, they were motivated to make sure that their networking system worked well with Kubernetes or their storage system worked well with Kubernetes and things like that.

[00:41:03] So that was sort of a secondary layer of partner discussions that we had.

Ryan:

[00:41:06] Right. And that’s downstream of becoming the dominant player. And I guess that’s validating the open source strategy, which is you become dominant, everyone’s kind of got to integrate with you and all of that. How did you prevent Google from dominating in the roadmap or I guess controlling what Kubernetes would be given that it started at Google, funded largely by Google?

Brendan:

[00:41:30] Yeah, I mean, I think that was really important and I think it was a critical part of gaining adoption. Right. And becoming the industry standard was giving it independence. And I think there’s two pieces to that. The first is getting it to foundation. So the creation, it was only a year in that we created the Cloud Native Compute foundation that we donated all of Kubernetes to the Cloud Native Compute Foundation.

[00:41:58] And so getting the project, the logos, all of the legal stuff, trademarks, all of that stuff into an independent software foundation with the Linux foundation was critical. Right. Because it’s hard to partner if somebody else has trademarks on the Kubernetes logo or whatever. And then I think the other piece that was important, that came a little bit later, was writing down the governance rules. So for the first time for a few years, Kubernetes didn’t have any really governance rules written down.

[00:42:38] It was a mistake, I would say. Right. Like we didn’t realize how it was really. It was something we should have done earlier, but we didn’t. And so we sat down in 2016 to write the governance rules. And I think all of us were aligned on this idea that we didn’t want any one company to be able to take control of the community. And we really built the community and the rules of governance to be Democratic.

[00:43:09] You know, we’d never. I mean, I think that’s an aesthetic from Craig and Joe and I. We never set out to be like a benevolent dictator for lifestyle project. We always set out to be a distributed, you know, distributed ownership, democratic kind of project. And we codified. We codified a lot of that into the governance docs that, you know, continue to this day. So I think both those things together really helped make sure that it was an industry standard and not any one particular company standard.

[00:43:39] But also, I think we’re critical to its success. I think they’re duels of each other. You can’t have one without the other.

Ryan:

[00:43:46] People wouldn’t have come on if they didn’t see that it was governed well and open.

Brendan:

[00:43:52] Yeah. Because obviously, if you’re thinking about adopting it or you’re thinking about putting it in your service, the thing you’re worried about is whose roadmap am I betting on?

Ryan:

[00:44:02] When you said governance, is that literally, when I think of government, there’s a constitution somewhere?

Brendan:

[00:44:09] That’s literally what we wrote.

Ryan:

[00:44:11] Did you write that yourself or is that something that lawyers do?

Brendan:

[00:44:14] No, we wrote it ourselves. In the span of about. It was a couple, three fairly intense meetings amongst six or seven of us. We got together and just kind of talked it through and looked at a bunch of other communities and kind of like, what had worked, what had not worked, what were we worried about, what were we trying to achieve? Some of it was codifying stuff that already existed. So we had some loose organization stuff that already existed in sort of a de facto way, but didn’t exist in an explicit way.

[00:44:56] Some of it was. We created this steering committee that had never existed before. And we just basically. And we were lucky, I think, that we were able to gather so the people that came together. We called it the Bootstrap Committee. We were lucky in the sense that we had enough people who kind of were not who the entire community would look at as being leaders. And we weren’t fighting with each other, we weren’t infighting.

[00:45:29] So we were all kind of aligned and we kind of got everybody. So we didn’t have to be like, oh, we grabbed this side and not that side. We kind of. We grabbed. We were able. And it was like seven people, I think seven or eight people. We were able to pull together a group of people that really represented everybody and that everybody kind of all respected each other and respected each other as leaders in the space.

[00:45:52] And a lot of credit there, I think, goes to. I mean, everybody who is involved deserves a lot of credit. But Sarah Novotny, who is our community leader at the time, deserves, I think, a ton of credit for bringing that thing together.

Ryan:

[00:46:07] When you look back on Kubernetes, because with an open source project there’s obviously the read aspect, which is everyone can use and duplicate this code and execute it. But there’s also, I guess the writing part, which is people making contributions. What percent of the contributions actually come from the community and what percent is actually just the main stakeholder companies just putting in their code?

Brendan:

[00:46:33] Yeah, I don’t have the specific numbers for Kubernetes, but my experience in open source says it’s like 80, 90% the core contributors and less than 10% to other people. It’s hard, I think in general it’s really hard to get people to contribute. Part of it is companies, honestly. Right. Like, you know, companies like Microsoft. We make a commitment to contributing to open source. And so, you know, we at a leadership level, we’ve decided that this is something that we want to invest in.

[00:47:06] And so we’re willing to have teams of people who specialize in working in upstream open source projects. But for a lot of users of Kubernetes, you know, they’re a retailer or they’re a banking industry or they’re like it’s tech isn’t their core thing that they’re doing. Tech is a means to an end, to deliver an app for their user. And in that world it’s pretty hard to justify. Well, I’m going to take 10% of my people and I’m just going to do upstream open source contributions.

[00:47:36] And especially if the leadership is like not a technical leadership. And so they didn’t necessarily grow up in those communities. If you grow up in finance, it’s hard to explain what’s the value of contributing to the. The value of taking the open source is very clear. Right. It’s free, but the value of contributing back, it’s harder to explain or legally. Like we ran into people. Even today it’s getting better, I think.

[00:48:00] But even today I’ve run into people who say we would really love to contribute. Our engineering leadership is aligned, that we would want to contribute, but our legal team is worried that if we contribute we’ll be liable if we introduce bugs.

Ryan:

[00:48:17] Would someone sue them or.

Brendan:

[00:48:18] Yeah, I think that’s what they’re worried about. I don’t. It doesn’t hold water legally and I think the Linux foundation can give you plenty of like case law and things like that to show why it doesn’t hold water. But sometimes that’s enough to block someone from contributing.

Ryan:

[00:48:37] I never thought someone would get sued for adding a bug. I mean, everyone adds bugs on accident.

Brendan:

[00:48:44] Well, but, I mean, but on the other hand, like if you write a proprietary piece of software and you sell it to somebody and it has a bug and it causes your house to burn down, like you can imagine you’re going to sue the people, right? So like it is sort of a legit worry at some level of like if I wrote the JPEG open source JPEG library that ended up in the smoke detector that caused the house. Yeah, like you can sort of imagine the chain of logic that gets you there.

[00:49:08] I don’t think it’s true. I don’t think it would hold up. I think a lot of the licenses, you know, a lot of the open source licenses include indemnification language that basically says if you use this, you’re using it under your own risk. And like you can’t sue us if it burns down your house. But I think that that’s a worry for, I mean. Well, I’ve heard, no, I don’t think I’ve heard from people that, that, that their companies do have that worry.

[00:49:34] And again, in some sense it’s because they’re like, well, what’s the value if I see this potential risk and I don’t necessarily see the value. And I mean, and like, also, like again, if it’s not a core thing, if you’re not a tech company, you know, is that developer really capable of like arguing with legal, Arguing with legal about what you can and can’t do? Probably not. Right. Like they’re probably just going to fade away.

[00:49:56] Right. So, you know, there’s that aspect too.

00:50:01 — Scaling Kubernetes up for AI training workloads

Ryan:

[00:50:01] I remember this is many years ago. I read this blog post that OpenAI put out before. I think OpenAI was kind of huge and it says, here’s how we scaled Kubernetes to 7,500 nodes.

[00:50:16] And so I want to know, there’s these new workloads coming in for AI. There’s training, which is this huge, I guess, all at once workload. And then there’s inference, which is latency sensitive and you kind of need it to come out instantly. Here’s how has Kubernetes adjusted over the years to handle these kinds of workloads?

Brendan:

[00:50:43] Yeah, I mean I remember when we couldn’t really handle more than about 100 nodes. So it’s definitely been a lot of optimization in the core systems. And there’s places where the APIs were pretty noisy and we needed to reduce the noise level or we needed to extract components into another component so that you could scale. ETCD in particular actually could be the main bottleneck to that kind of scale.

[00:51:14] And so figuring out how to run ETCD really well is a core part of figuring out how to run Kubernetes really well at scale. I mean, I don’t think it’s that different than learning how to run a database or anything else like that at scale. Large scale is just weird and you just have to run it, see where it breaks, figure out how to fix it, rinse and repeat. I do think what’s interesting is that while AI training as an example, is a really large cluster, large scale kind of thing, I think by virtue of being in the cloud, a lot of our users actually have much, much smaller clusters, but lots of them.

[00:51:57] So hundreds or thousands of clusters where each cluster itself is a little bit smaller. And I think that’s not something we anticipated because we came from a world of physical data centers where you only want one because you don’t want to have to set it up a bunch of times, you just want to set one up for the entire data center, call it a day. But because the cloud, because aks you press a button, pops up in two minutes, it’s really easy to get yourself a cluster.

[00:52:25] So people create lots of clusters. I think we’ve also invested a lot in the Kubernetes community and in Azure as well on managing lots of clusters. How do I manage clusters at scale? I think one of the jokes we spent a lot of time talking about containers is replacing snowflake servers. Not Snowflake the company, but specially handcrafted servers. And now we just have a bunch of snowflake clusters.

[00:52:56] So the VMs all look the same, but the clusters are all weird. So we have to provide people with tools to make sure that the monitoring software is the same on all of them and that all of the Kubernetes versions are the same and all the admin users are the same and all this kind of stuff. So that’s another aspect of scale out that I think we didn’t anticipate that we had to go and build, which is number of clusters as opposed to size of cluster.

Ryan:

[00:53:21] I always hear in the news that the anticipated scale is even higher than today’s unprecedented scale. And I see people are purchasing GPUs like crazy. I’m curious, is there any upper bound where Kubernetes just cannot handle that load? Let’s say you 10x it from where it is today. Is it going to break down at some point and you need something more custom?

Brendan:

[00:53:51] Well, I mean, I think it all comes down, it comes back to the storage layer because everything again, because there was this design decision that everything routes around the storage layer. Everything else is basically horizontally scalable. So you have more API requests coming in from more nodes. Well, you just need more API servers. You want to do scheduling faster. Well, you need to just have more schedulers.

[00:54:21] Everything else more or less, you can just horizontally scale out. It’s the storage layer that is the bottleneck and that’s where the work comes. And so you want to say go 10x up. Well, you’re going to have to probably figure out if you can make etc scale that way or if you need to replace ETCD with something else that has the same characteristics but can operate at scale. So I don’t think there’s anything inherent in the design that would prevent it.

[00:54:51] But obviously there’s a famous quote that every time you change an order of magnitude, the problem moves. And so I think that’s really true. Every time you increase by an order of magnitude what you thought was the main problem is going to become easy and then the problem moves somewhere else. So you were network constrained, now you’re CPU constrained, you were memory constrained, now you’re network constrained.

Ryan:

[00:55:12] Yeah, that’ll be cool to watch how. Because it seems like everyone wants to scale up.

Brendan:

[00:55:16] Yeah, I think it’s definitely the case that people continue to try and push the limits of scale. But I think like anything else, like when there’s motivation, people go and figure it out. As long as there’s not something inherent in the design.

00:55:31 — Reflections on getting a PhD

Ryan:

[00:55:31] At the last part of this conversation, I just wanted to reflect over your career a little bit, maybe ask you a few questions about things. And you mentioned that you had a PhD in robotics. And I hear a lot of people say they don’t recommend PhDs. Some people do. Curious what your take on getting a PhD is.

Brendan:

[00:55:49] Yeah, that’s probably like if I had to have a top 10 questions or top five questions that people ask me. That’s definitely in the top five top 10 questions. And I guess I’ll answer it with two different stories. One story is that at one point in my career I ran into a guy, same company, this guy, who I’d went to undergrad with and he’d gone off and done startups and done the tech industry thing and ended up in the same company that I was working at and I’d gone off and done my PhD and come back into the industry and we were at the exact same level.

[00:56:25] We graduated the exact same year, same degree and we were at the exact same level in this company. And so I guess that’s one way of me saying like, eh, it probably doesn’t matter. It probably doesn’t matter one way or the other. But I’ll also turn it around and say, hey, I had a lot of fun, right? So I had a lot of fun doing a PhD in robotics, so that’s worth it to me. And then two, I think I learned a lot about how to, I think from the PhD and my PhD advisor, I learned a lot about how to write and put my ideas forth in both written and presentation form that you don’t necessarily learn in the industry.

[00:57:10] And I think that benefited me. I think that benefited my ability to argue. And we talked about that six month period where we were arguing for why we should be allowed to open source this thing. I think the skills I learned in terms of presentation and in terms of writing benefited me during that time and have continued to benefit me. And then I think when I went out as a professor for a couple years teaching CS101 and having to explain stuff to students who didn’t really know anything about computers, I think really helped me organize the initial parts of the Kubernetes project so that somebody could learn about Kubernetes because people were coming in and being like, what is a container?

[00:57:54] What is orchestration? How do I do this? There’s a lot of just teaching that you had to do. And I think that experience as a professor thinking about how do I teach students something really helped me do a good job with teaching Kubernetes to people. And so I think those things were really beneficial. And so I guess there’s the two different arguments, which is one is it doesn’t matter. The other is I learned a ton of stuff that I think was pretty useful to my career and I had some fun earlier.

Ryan:

[00:58:25] You said top questions that people ask you and I’m kind of curious, what’s the top question?

Brendan:

[00:58:31] I think the one, the other one that I get a lot is how do I know what I should learn? Like a lot of, especially when I talk to the interns or the first couple years, a lot of the questions are revolving around like AI seems really hot right now, but I’m really interested in systems, like should I go learn AI because it’s hot or should I learn systems because I think systems are interesting. I actually kind of don’t care what you learn, I care that you’re learning.

[00:59:01] And so the most important thing is to find something that you’re excited about and energized about because you’ll do that instead of watching YouTube. So if you’re not excited about AI, well, you’re probably not going to do a very good job learning AI, which means that you’re kind of going to waste your time. But if you’re really excited about systems, you’ll probably put a lot of passion and energy into it.

[00:59:24] And we still need systems engineers. I think that’s a pretty popular question. I think there’s a lot of, I sense anyway, a lot of fear of making the wrong decision. And I always tell everybody, like, there was no plan. I’ve never had a plan for my career, like never, ever, ever. Like, I’ve always just chased after things that I thought were useful and were fun and interesting and you know, obviously like, that can work out badly for people.

[01:00:00] I’m sure it’s good to have a plan, probably for some people. But like, I also want to make sure people understand that like when you look back, sometimes the things you think were mistakes or dead ends, like actually were critical things that taught you stuff. And so worrying about, did I choose the wrong thing? Am I going to choose the wrong thing? As long as you’re learning, you’re probably doing okay.

01:00:22 — The inevitable trajectory of software is death

Ryan:

[01:00:22] I don’t fully remember where you wrote this down, but I have this in my notes. It says the inevitable trajectory of software is death. And I just can’t imagine Kubernetes dying. But how do you see that happening and what do you think about that if it did?

Brendan:

[01:00:50] I mean, I definitely stick by that statement. Although I think that the sentence before I said that was, you really should never fall in love with your software because the inevitable trajectory of software is death. Which means don’t stick with it, don’t stick with it past when it’s dying. You should always be willing to throw away stuff. Just don’t stick with it just because you wrote it. You should always be willing to throw it away.

[01:01:19] But obviously I think if you look historically across the industry, it’s true too. And quite frankly, even within Kubernetes, the source code that I wrote has been rewritten a number of times over the 10 plus years history of the project. So what does it look like? I mean, I think it looks like something coming along that achieves similar things, but easier with less complexity and more utility. And I think that I can imagine what that looks like.

[01:01:57] I think some of these natural language stuff, if you could actually really get it to be an interface that worked 100% of the time, like obviously it’s way easier to come in and say I would like a reliable web service than it is to say YAML, YAML, YAML, YAML, yamile. You know, I think sometimes I think it’s sort of true two different trajectories. Like sometimes the trajectory is. It goes away, sometimes it just becomes so hidden that nobody sees it, right?

[01:02:21] Like underneath Linux there’s, I mean, excuse me, underneath Kubernetes there’s Linux and underneath Linux there’s a processor. But you know, people don’t pay much attention to that and there’s a lot of attention now on AI and underneath a lot of the AI is Linux. But it could be that people focus so much on the AI that they forget about the Kubernetes part. And I think that’s happening already. Honestly, like, I feel like if I look at the volume of changes and things like that, like I think it’s, you know, I think we’ve sort of plateaued in terms of like the amount of change that’s driving through the system.

[01:02:54] Stuff needed to support AI is kind of like the exception to that category. But you know, I’d be shocked. I mean, I guess put it this way, let me take the long view and say in a hundred years, is Kubernetes still going to be running? I’d be pretty surprised, right?

Ryan:

[01:03:13] No way.

Brendan:

[01:03:14] It’s hard to imagine, right? That would be true. I mean, I don’t know, we haven’t had computing systems for long enough to maybe know for certain. And there are things that we still use. I mean, there is some stuff that we still use from back then. Plugs are still the same shape, ish, stuff like that. So maybe, but even like something like X86, like if you’d asked me six years ago and said, Is the X86 processor going away?

[01:03:40] I’d say like, well, maybe on, I mean, obviously on mobile it did, but like in the server maybe not. But now two things have happened. One is all the processing is on GPU now and two, ARM64 is now a pretty important platform on the server for energy usage and other reasons. And so it’s pretty dangerous to predict the future because it has a tendency to show up sooner than you predict or longer Than you predict too.

[01:04:10] Right. Self driving cars. I’ve heard self driving cars were five years away for the last 15 years.

01:04:16 — Top book recommendations

Ryan:

[01:04:16] I don’t know if you read books for career’s sake, but if you do, is there a book that impacted your career the most?

Brendan:

[01:04:27] Well, I mean, I would say early on the book that impacted my career the most was a book. It was the Gang of Four book. Was it Software Engineering Designs and Patterns or whatever? Like it’s a software engineering book.

Ryan:

[01:04:38] I see it. It’s Design Patterns Elements of Reusable Object Oriented Software.

Brendan:

[01:04:43] Yeah, there you go. So that, like early on that was a very influential book. It’s like a late 90s or mid-90s kind of book. There’s a much more recent book called Leadership on the Line that as I become sort of a large org leader. That’s. I really like that book. And then there’s this. What’s this book? It’s called, I think five Dysfunctions of Teams. I think that’s a really good book too, from a, like, how teams operate perspective.

Ryan:

[01:05:07] If I’m understanding. If you’re an. If you’re an engineer, check out that first book. If you’re a manager or a leader, check out the second two books.

Brendan:

[01:05:14] Yeah, that’s probably about right. Yeah, I think that’s right. And it’s an evolution over time. Right. Maybe you’ll do both.

01:05:22 — Advice for his younger self

Ryan:

[01:05:22] Last question for you is if you could go back to yourself when you just graduated college and give yourself some advice, what would you say?

Brendan:

[01:05:31] Keep better notes. I feel like there’s a great MBA thesis or a great book in the whole Kubernetes Journey and Beyond. And like, I just don’t have enough notes to do that, to write it down. You know, we went through a lot, like a lot of different stuff happened and I remember some of it and I don’t remember a lot of it. And I would have been nicer if I’d kept better notes, I feel like.

Ryan:

[01:05:57] Right, well, you got all the code there. Maybe an LLM can parse it or something.

Brendan:

[01:06:02] Yeah, it’s not so much about the notes part. It’s not so much the code part. It’s like all of the, like the stuff you were talking about, like all the partner discussions, all of the, like, interpersonal stuff and you know, all that kind of stuff. And like, I remember a lot of it, but I don’t remember all of it. Cool.

Ryan:

[01:06:18] Well, thank you so much for your time, Brendan. I really appreciate it.

Brendan:

[01:06:21] Yeah, for sure.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:49 - What goes on in hiring committees

00:09:02 - Unethical candidates

00:12:50 - How leveling is determined

00:23:12 - Can you negotiate level mid-process

00:32:30 - How non-tech leads can signal scope

00:39:11 - Referrals and bias

00:45:28 - What the rubric looks like

00:50:00 - OpenAI and Anthropic specific discussion

00:52:22 - Most common mistakes senior candidates make

01:02:31 - How to prep depending on your level

01:08:34 - Subjectivity and bias

01:21:02 - The questions you ask at the end matter

01:23:59 - Storytelling tips

01:30:31 - How he got promoted to Senior Manager (M2) at Meta

01:33:32 - His biggest career regret

01:38:13 - The best advice he ever received

01:39:54 - Advice for younger self

Transcript

00:00:49 — What goes on in hiring committees

Ryan:

[00:00:49] Behavioral interviews are one of the most common signals in hiring committees that down levels candidates and so and I think this is especially important for senior engineers and higher. And so today I wanted to cover all of the common mistakes people make, how to prevent from getting down leveled and also I’d like to go over some company specific tips for people who want to work at hot companies like OpenAI and Anthropic.

[00:01:21] So yeah, I’m hoping that with your experience working on recruiting, leading iOS and Android recruiting across meta and after having conducted hundreds of interviews, I’m hoping you can kind of give us some of the behind the scenes in these hiring committees so we can learn about how to do better in our behavioral interviews.

Austen:

[00:01:41] Oh yeah, a behavioral interview is my favorite thing to talk about. So I’m excited to be here.

Ryan:

[00:01:45] What actually goes on in the hiring committees?

Austen:

[00:01:49] The first thing I would say is like when I would get a packet in front of the hiring committee and it was a senior packet, the first thing I would do is go to the behavioral interview. I would want to understand what is the scope that this engineer has operated at in the past, what’s their level of influence, what’s their level of insight, what’s their level of communication, what’s the level of organization that they’ve operated in.

[00:02:09] And that’ll be the first thing I did. And these committees are built of other engineers. They’re built of other engineering managers who are influential in the company. They’re recruited by someone like me, the hiring committee chair. And their job is to partner with recruiting to understand Whether or not this engineer should be hired hire, a no hire decision, but also a leveling decision. And then they send that, that up to sort of company wide committee of people who honestly most of the time what they do is just sort of a gut check on, on a sort of a cross, you know, across company hiring bar.

Ryan:

[00:02:40] So I’d love to learn more about the, the back end of the recruiting. So let’s say I’m a staff candidate and I’m going to go get hired at Meta. But who are all the people involved? What do they care about and how do they contribute to the hiring process?

Austen:

[00:02:55] Yeah, great question. So the first person you typically engage with is somebody called a sourcer. This is someone who is responsible for finding potential candidates. Oftentimes they’re more junior on the recruiting side or this is an entry level job for a recruiter. Their job is to cold contact you or maybe to process referrals. This sort of first touch point. They’re looking at some kind of job description.

[00:03:15] They’re looking at your years of experience. That’s probably the biggest thing. They’re assessing your past companies, your past, whatever they can find about your, your past experience. Looking through the referral that they look at and seeing whether or not they want to pass you through to some kind of phone screen process. Right. So this is the part where you see it as a, as an engineer. Some, sometimes this phone screen is a technical one where they are going to be giving you some kind of, you know, coding interview.

[00:03:38] Right. That’s really common. It could even be a pre, you know, pre phone screen conversation where they’re asking you. Sometimes we would give out multiple choice questions for engineers like oh, tell me about this part of iOS or time, this part of Android. And then you’d have to be able to answer those questions. Questions just as a sort of a, like a pre filter. Right. To make sure that we’re doing phone screens in an efficient way.

[00:03:59] Of course once your phone screen happens it goes to a, to back to the hiring committee actually. Unless it’s a very solid hire, if it’s a very solid hire on the phone screen side you get passed directly on to the scheduling for an on site interview. But if it’s kind of on the border then we would review it as a hiring committee. We would look for signals, all the signals that, that people talk about in coding interviews, communication and problem solving and all those things and we would see whether or not it would be worth to, to, to a follow up interview or we should just pass on this person on through to the, to the, to the rest of the, of the hiring process to on site or if we should maybe, you know, pass on that person overall and just maybe come back to them some other time.

[00:04:33] So that’ll be the first time the hiring committee is involved in a hiring decision after that. Sometimes a hiring committee would be consulted when there’s a certain kind of candidate has specialized experience. So for example, if you were like a very low level mobile engineer doing like C work, for example, on the mobile side, we would want to make sure that you were giving you an appropriate system design interview that really assesses you for your specific skills.

[00:04:54] So then we would also typically be consulted at times when there’s a choice of whether or not this person is a staff level engineer or a senior engineer. Oftentimes that’s the place in companies where the hiring process starts to change. So for example, at Meta specifically staff level engineers would have two system design interviews and so they, we would not be able to hire someone at a staff level unless we had provided two system designs.

[00:05:20] So that sort of decision about how to structure the loop starts, starts, starts there at that, that on screen, on site scheduling point in the, in the hiring process. And then of course if you, as you go up, as you go up to principal or as you go up to distinguished engineer, those hiring processes are very different and they add additional behavioral interviews. That’s what they add. So we can talk about that at some point.

[00:05:40] That’s how they are assessing whether or not someone is a principal engineer or distinguished engineer. And then after you go through the on site experience, then it comes back to, to us as a hiring committee and we make a decision, do we hire you? At what level do we, do we add any additional follow up interviews? Maybe one interview didn’t go very well. Maybe an interviewer, maybe you, you flubbed it and we want to give you another chance.

[00:06:01] Maybe we, maybe the interviewer didn’t do a very good job acquiring signal. And that happens a lot actually in the behavioral interview. So the behavioral interview is one of the hardest ones to, to give and one of the hardest ones to interpret. So oftentimes we would do follow ups on, on behavioral interviews if we didn’t get the right signal. And then after that, at least at Meta, it would go to a committee of engineering directors and they would make the final hiring and leveling decision.

[00:06:23] Sometimes members of the hiring committee would accompany the recruiter to that conversation with the engineering directors and advocate for our decisions as a, as a hiring committee. So possibly we’re deciding to take a chance on somebody, maybe this aspect of their packet is weak. But we’re really excited about that. We think that the company would really benefit from having engineers who have whatever that specific trait is, this problem solving piece or this technical skill or this organizational influence skill that they’ve demonstrated in their on sites.

[00:06:49] And we would need to go and advocate for that before the engineering directors. So that would be the process for a, you know, for an engineer from start to finish.

Ryan:

[00:06:58] At the beginning you said there’s that low level recruiter that makes some gut reaction. And just to understand, because the leveling decision is the thing I’m most curious about, the way that they would judge the level of the person to kind of enter the funnel is mostly based off years of experience. Is that right?

Austen:

[00:07:17] Years of experience and previous title. So if you are a staff engineer at Google, then they’ll probably try to bring you in a staff engineer here at Meta. Now a lot of companies like Meta don’t have public levels so it can be difficult to see just from someone’s LinkedIn what level they are. So that’s why they rely so much on years of experience.

Ryan:

[00:07:36] A lot of people, their companies, I’d say it might be someone outside of FAANG who has many years of experience and their title is extremely high. Let’s say they’re principal.

Austen:

[00:07:49] So they work in finance and they’re a vice president.

Ryan:

[00:07:51] Yes, yes, exactly.

Austen:

[00:07:52] Vice president gets a vice president like four years. Yeah, yeah.

Ryan:

[00:07:56] But like let’s say it’s, it’s someone, you know, principal architect at some non FAANG company for instance, what’s that recruiter going to set them at and how do they get leveled when you’re not in faang, coming into faang.

Austen:

[00:08:10] So sources understand this and recruiters understand that there is level inflation in different parts of the industry. And so we would take somebody, you know, who worked at even sometimes big tech companies, but non, non FAANG companies, like maybe companies that are, that are more oriented towards business, we would take them and download them significantly. So they would be, they would even be supporting maybe 10 or 15, 20 people as a manager, maybe even more, maybe 50 or 50 or 75 people as a manager.

[00:08:35] And sometimes we would down level them and just give them a team of eight or ten FAANG engineers right at Meta and likewise for, for, for ICs. So it’s not just that if you put on LinkedIn, I’m a principal engineer, you can suddenly get principal engineering sorcerers reaching out to you. They understand that that different companies have different expectations for, for their levels and maybe there’s some world which we would like to live in that this is all consistent, but it’s not, that’s not the world that we live in.

00:09:02 — Unethical candidates

Ryan:

[00:09:02] Just out of curiosity, what if you had some really unethical candidate who they were, they worked at Google, but let’s say they, in reality they were only a senior engineer or something wherever they were, but they had the years of experience where it’d be believable that they could be maybe a senior staff engineer or something like that and that was listed on their resume. What would happen if someone did that?

[00:09:29] And I imagine that first hop would go to senior staff. What then happens that prevents that from working?

Austen:

[00:09:36] Yeah. So the, for example, the first phone screen with an engineer that you have for staff level engineer or above is usually some kind of, not just a simple coding exercise like you would for a mid level or a senior engineer, but it is also sort of scope check. There’s a mixture of coding system design, oftentimes a conversation with you about your past experience where you walk yourself, they walk you where the candidate needs to walk the interviewer through some large project that they shipped now.

[00:10:05] So that’s the first, the first sort of check. Can you pass a sort of system design conversation at a high level such that we would pass you through for the on site, the real system design interview. And then can you tell me a story which is of sufficient scope of what you’ve executed now? You know we can talk about lying, right? There are some really famous liars in the world, we call them actors.

[00:10:23] So. But I will tell you that Los Angeles restaurants are filled with people who are trying to get into the lying business, right? And they can tell you how difficult it is to be an actor. So yes, I do think you can. Maybe you can come up with a story, right? And maybe you can use AI to tell you, oh let me tell you about this like company wide project that I shipped at Google. It is pretty challenging to lie in a convincing way.

[00:10:46] We, as, as you know people, we have this sense of, of is this person really telling me the truth. And then there’s always the follow up questions. And I’ve, I’ve experienced this in my coaching, coaching job where I can tell someone’s telling me a story that they have gotten from an AI because I start to ask them these follow up questions especially about technology and then they start to give me these kind of vague an answers or they really don’t.

[00:11:07] The answers don’t sort of don’t fit together. And then I get the sense of like, this person’s just not telling me the truth. Am I, am I foolable? Like, yes, everybody’s foolable. If you’re a good enough liar, yeah, you can, you can make this happen for sure. But I think it’s actually much harder to lie than most people expect it to be. And so that’s the first, that’s the first step. And then after that, you have this, this barrage of on site interviewers, interviews with highly calibrated people.

[00:11:31] So when we put interviewers for staff level or for, or principal level engineers, we send very senior people to those people who have been in the industry for very long time who have interviewed a very, a lot of people. They’re highly calibrated. And yes, you will have to like, lie repeatedly to these people in a convincing way in order to get all the way through. Now if you can do that successfully, I don’t know, maybe you are good enough to be a staff engineer or principal engineer.

[00:11:55] You know, maybe you’ll be fine and then we’ll hire you. Right. And then, then the question is, can you meet expectations at that level? Um, I, I do not have stories about people that we hire that we just, we thought was totally lying to us. Even though I’ve hired thousands of people, I, I have not heard that story. I think it’s much more difficult than people expect it to be.

Ryan:

[00:12:19] In theory, if someone was a, a generational liar, they, they could, they could do this. It’s just a lot harder than people think. Okay. Because yeah, I think that’s, that’s on a high level when maybe it’s just an engineering mindset. When I’m coming in and I’m thinking about interview prep, my, my first thought is, okay, I got to be good at the technical, but the behavioral, I can kind of, I can kind of wing it.

[00:12:44] You know, I can tell some stories. It’s a little bit of a softer thing.

00:12:50 — How leveling is determined

Ryan:

[00:12:50] What is it in the behavioral interviews that leads to the leveling determination? So you mentioned there’s an initial leveling determination, kind of a gut call by the first hop in the layer. And then at some point you’re placed into. Sounds like a range. You mentioned in one case.

Austen:

[00:13:17] Exactly.

Ryan:

[00:13:19] They’re trying to figure it out. And so then you’re Maybe staff, maybe senior. And they give you another loop. What do you need to do such that you are placed in staff if that were your goal?

Austen:

[00:13:30] One of the most important things when you’re choosing any kind of story to tell in a behavioral interview or when you’re having a conversation with a recruiter, which is also kind of like a behavioral interview, is to ensure that you’re establishing yourself as a certain scope. And that scope is about what size business problems have I solved with technology and what level of ambiguity and what level of organizations have I operated within and what have I been able to accomplish in that operation.

[00:13:53] So when you’re having a conversation with a recruiter and you’re talking about your past, past experience, you have got to land in that first tell me about yourself kind of thing. Like, oh, hey, who are you talking about? You’ve been up to that conversation is super critical. And people think about it from an interviewing perspective, but it starts there in that conversation with the recruiter. And so you have to come in and say, I have demonstrated and delivered large business value with, with technology.

[00:14:14] And it’s kind of like the greatest hits from your resume, right? People always tell you have, you know, measurable impact and results that’s, that’s present on your resume. That’s, that’s using some kind of metrics, right? That’s super important. It’s also to convey a sense of depth, right? A sense of complexity. You could say something like, well, you know, I improve performance in the Facebook app.

[00:14:31] And then people say like, okay, that doesn’t sound super hard. But if you said something like, well, you know, I organized across 100 engineers, this entire organization, to spend a 12 month performance improvement project that ran into multiple very difficult, you know, technical issues that required multiple staff level engineers in order to solve. Many months of investigation and experimentation.

[00:14:52] And then we came out with these like three core principles. And then we, then we, you know, we shipped this performance improvement. So I think when you start talking, talking about the complexity of the work, you need to communicate the technical depth, you need to communicate the organizational depth, you need to communicate the business value and the business impact that you delivered. And those are the things that the recruiter is listening for.

[00:15:09] And likewise, everybody in the process is listening to that. All the interviewers listening for the same thing. So getting really crisp and practicing that tell me about yourself piece is the first place where you can ensure that you’re getting into the right leveling bucket.

Ryan:

[00:15:23] I noticed there’s a lot of correlation between what you’re talking about and also promotions as well. For instance, when you’re talking about promotional behaviors for staff, your projects don’t just want to be, hey, I ship this thing and it’s good for my team. You want to be doing things that are across the org and larger and complicated and challenging. So are you saying that those are pretty similar and.

Austen:

[00:15:48] Oh yeah. Oh yeah. In fact, preparing for behavioral interviews will actually make you a better engineer. And I think being a great engineer is how being a great communicator, especially about your impact is the thing that you need to do is to be a great behavioral interview. So yes, for sure, you need to be able to quantify your impact, communicate it efficiently, communicate effectively and honestly.

[00:16:08] We’re talking about lying earlier and trying to keep people from lying. But most of my clients, most of the people I talk to in my coaching business, they have a problem not telling enough of the truth. Like not telling. They’re not boosting themselves enough, not talking to me enough about the accomplishments that they’ve done. So to me, that’s the thing that most people need help with. They need help showcasing how difficult the problem was.

[00:16:30] They need to help showcasing what kind of impact they made not just on whatever top line business metric it was, but also what kind of impact did it have on the team? What kind of impact did it have on code quality? What kind of impact did it have, you know, long term in the organization that they were, they were operating within? So for me, think about, if you’re, you know, I’m coaching engineers to do this, I spend a lot of time thinking about, you know, what are all the implications of the work that you’ve done and then how can you be really crisp about having that conversation?

[00:16:54] Because the flip side of that is like, well, if I want to talk about all the things I’ve done, then it takes a really long time. So that’s why it takes some, some, some preparation, but certainly getting really good at behavioral interviews. Communicating, telling stories, for example, telling stories about what you’ve done. You do that all the time. You do that to your manager every week in a one on one.

[00:17:10] You do that to the executive whenever you’re presenting your results.

Ryan:

[00:17:14] learn as a senior engineer in the interview process. It sounds like at every point, whether you think you are or not, you’re actually being assessed. What I mean is usually that first call, at least from my recollection, it was pitched to me as, hey, this is a little chat you don’t need to prepare for just Go ahead and come and talk to me. I’m just the recruiter but actually that call is hey, talk to me.

[00:17:40] I’m trying to figure out what level you are so I can place you in the right loop and even see if you’re worth talking to for follow up. So am I understanding that people should just sell themselves at every single minute of this recruiting process?

Austen:

[00:17:55] Yeah, always be closing for sure. Definitely. This first call is an evaluating call. They are interested in you. Right. Usually they, they may have contacted you or they have you applied and they’re contacting you. So it is a softer kind of evaluation. You shouldn’t be super nervous about it. They are your, they are your, your buddy, they’re your partner. Let’s remember that especially at a big company, these sources and recruiters are incentivized by how many hires they can deliver.

[00:18:17] So no, they’re not your friend. Right. There are certain, certain conversations you should be care very careful about, especially about compensation with these folks. But they are advocating for you. It is actually benefits them to find some a great candidate and be able to get them all the way through the process that is in their best interest. So it is a, you know, it is not a confrontational experience you need to prepare for.

[00:18:38] But yes, it is evaluative. Like you should not just show up to that phone call emotionally disconnected or unprepared to talk about your past experience. You should and if you’re, if you are like that, then I would delay the call. So reschedule the call. Spend a little bit of time preparing what you will say to the recruiter so that they get an accurate sense of the kind of impact that you’ve delivered and that’s going to be your best bet at getting that, that higher level offer.

Ryan:

[00:19:03] I’ve had experiences where I got through everything and at some point they said congrats, you’ve got the offer. It’s verbal, but you have an offer now. So congrats. We just want you to talk to the hiring manager just one last time. And this is for you. They want to tell you about the company, this and that usually is that also a case where the hiring manager is trying to get signal on you?

Austen:

[00:19:30] Oh, of course, of course. I mean the same exact things apply. So yes, it’s a little bit lower pressure than say the behavioral interview or the coding interview in the middle of the process. But definitely when you’re having this hiring manager chat or at a place like Meta or Google, they have a team matching phase where you’re having conversations with Multiple managers and they’re trying to find a fit for sure.

[00:19:49] You know, that conversation is super important for you to have a buttoned up introduction of yourself. Tell me about yourself. I do think that that one is a little bit more social. So in an interview setting, the interviewer has a set of questions. They want to go through the questions, right? So the longer you spend introducing yourself, you’re actually hurting your time management there. You’re preventing them from collecting other signal about you.

[00:20:11] However, I think this hiring manager conversation, it is more of a like, do I want to work with this person? Right? Do I, do I want to hang out with this person? So you do need a little bit more of a loose social engagement, especially as those early, that kind of, early phase of the, of the, of the conversation, hey, you know, how’s it going? Whether, you know, sports something, something, right.

[00:20:30] I think that, that connecting with the manager on whatever’s important to them is really, is really key for that, that interview. But certainly it is not just a, a formality or it is not purely for you. It is certainly the case that the hiring manager wants to see, do I want this person on my team? And they need to come out of that conversation with a couple things, you know, one is, do I like this person?

[00:20:48] Do I think they’ll be successful in the team? Will they fit the team culture? Will I, you know, will I as a manager benefit from bringing this person onto my team? And then they need to come out with a sense that you can deliver and solve problems that are similar to the problems that they have. So it’s so important for you to do as much research with the recruiter and sourcer in advance if you’re going to have this conversation.

[00:21:05] Understand what this manager values, understand what their team is working on, and then be able to tell your stories in a way that, that showcases that, yes, you can solve the problems that this manager has.

Ryan:

[00:21:14] You mentioned briefly there. You said the recruiter, the compensation conversations with the recruiter and how you need to be extremely careful in those conversations. What’s the most common mistake people are making when it comes to those compensation conversations?

Austen:

[00:21:30] Yeah, look, I’m not a negotiation expert. I won’t, I won’t present myself as one. You can even hire people who are, who are really good at helping you negotiate with, with these big tech companies. And I recommend that you go get some advice from them. But I will say that, you know, early anchoring, any negotiation is dangerous. So if, if, if early conversations where they’re asking you about your compensation expectations you should be very careful about what you tell them because that will, that will anchor you into the expectation to the conversation that you, that you talk about.

[00:21:58] You should definitely get some advice and, and understand the laws in your local jurisdiction. So for example, in California and New York, you’re required, the recruiter is required to tell you about the bands and salary bands and total comp bans for, for each, each of the, the jobs that you’re being applying for. So you can leverage that to understand where you are in the comp bands at that point.

[00:22:18] But I would not say I’m an expert at negotiation. I think probably the, probably the biggest mistake most people make is they don’t have multiple offers. So if you are negotiating from a position of weakness like I don’t have any other offers, I’m just talking to you and you’re the only company I’m talking, it’s gonna be very difficult for you to get to extract anything from that company. And which is why it’s so important for you to orchestrate your offers to land at a somewhat similar time.

[00:22:40] I know that’s more stressful. I understand it takes more work to go through the interview process at the same time. But if you have multiple offers in hand, that’s your best bet for being able to improve your compensation position.

Ryan:

[00:22:54] So I guess going back to the leveling side of things, let’s say I’m staff engineer and I, I will only take an interview if it’s staff and I make a mistake early in the process. Like I’m talking to the recruiter, like

Austen:

[00:23:10] on the phone screen, for example. Yeah, yeah.

00:23:12 — Can you negotiate level mid-process

Ryan:

[00:23:12] Can I be direct with the recruiter and say, hey, I’m, I see that I’m getting a lot of coding interviews. I think maybe I got the wrong, I might have sent the wrong signal. In which case it’s fine, let’s just end the process. Or can I get leveled at a higher. Can you negotiate at that level in mid process?

Austen:

[00:23:31] Yeah, that’s a really good question. I’m sure it depends on the company in the process. I would say that’s a good conversation to have with your recruiter. If, if you know that you’re being placed in to consideration for a level that you don’t want to be. What I will say is that there are many times when we would up level people as well as down level people. And so the recruiter will, it’s their interest.

[00:23:50] They’ll try to keep you in the process. They will say, let me stick around, you know, maybe we’ll Evaluate you and maybe we’ll, you know, we’ll, we’ll offer you this second thing. And I would push for to go ahead and try to get as much of that evaluation process done for the level that you would like to, to be hired for as much as possible. That way you don’t have to go through follow up interviews or they don’t have to, you know, change something about the process late in the game.

[00:24:10] Try to collect as much information for the hiring committee as, as possible at once as possible. So yes, I think that’s a good conversation to have. But you know, if it was a job you were really interested in and they were offering, they were considering you for one level below and they were not willing to change anything, I, I would, it depends on the job obviously depends on the total comp. Opportunity and where you’re at.

[00:24:31] But I would say try to rock the interviews as much as possible and then get that consideration for the higher level. And you can do that in a couple of key ways. So the first way is to make sure that you’re anchoring every conversation you have with all of your interviewers at that higher level. Which again comes back down to that. Tell me about yourself when, when they especially behavioral and system design interview and you’re having these conversations and you are talking about very large scope projects, talking about impact which is commensurate with that higher level.

[00:24:56] The behavioral interviewer especially will notice that and then they will want to dive into that and they will want to ask you questions and they will then you’re re resetting their mind about what, what to expect from the interview and that’s super important. And then the second place is in, that is in the story choice that you have for your behavioral interviews. So remember that this, whenever someone’s asking you a behavioral interview question, there’s always a question behind the question.

[00:25:17] Like why are they asking me this? They don’t actually really care about my favorite project. Or like, like they really don’t actually care that much about some time I had a conflict with my manager. They’re probably going to forget a bunch, bunch of those that, that, that detail right after the interview. What they want to see is are you operating at the level that they’re expecting you to operate.

[00:25:32] So make sure and I talk about this in the book. There’s four different considerations for whenever you choose a story and the number one choice is scope. You want to make sure that you have come out of that behavioral interview telling the stories that that are the highest scope and the ones that represent you the most, the ones that you would love to tell to a hiring manager. So that’s your job as a behavioral interviewer interview candidate.

[00:25:51] I want to leave that interview having told the most important stories from my career. And sometimes that can be kind of challenging because they, they you are not driving. Right, the interviewer is driving. But your task as a candidate is to guide the interviewer towards that signal, towards that place in your career where they’re going to collect the signal that you think best represents you. And, and you do that by choosing stories that are of the highest scope and the highest impact that you’ve delivered.

Ryan:

[00:26:15] So around a year ago, a buddy of mine, he was applying for a senior role at all of the top AI labs and he actually got an offer at Anthrop. When he was going through the process, I remember him telling me that the single most impactful tool for him in preparing for system design was the free resources that hello Interview has on their website. If you are preparing for technical interviews, I highly recommend you check out hello Interview.

[00:26:39] I would have said that even before they sponsored this episode. I think they’re providing something that’s great for the community. Also, if you’re preparing for behavioral interviews, they’re actually partnering with Austin to provide more interview behavioral interview resources on their website. So I’ll put a link in the show notes so you can check it out. This is the absolute first ad I’ve ever done for this podcast after over a year.

[00:26:58] Right now the podcast is running net negative. Hopefully it can sustain itself soon. This is a step in that direction and I just want to say thank you so much for supporting the podcast with that. Let’s get back into the episode. Austin’s about to tell us how to avoid being down leveled accidentally. The big question then is how, how do you do it concretely? Maybe we can go over some concrete examples.

[00:27:21] What would a senior Scope project look like? What would a staff project look like? Senior staff, principal? Maybe we can take the same example and kind of evolve it so people can hear. What are the keywords people are looking for?

Austen:

[00:27:35] Right. I’ll give you a sense of where big tech companies like faang oriented companies are at with their levels now. But for the specific company that you’re applying to, you should go do some research and figure out what is expected of a mid level engineer. What is a staff level engineer? Oftentimes you can find this information on the Internet, but I’ll tell you, the very simple rubric is something like a, you know, a new grad engineer is doing tasks.

[00:27:55] You know, they do a task, come back to the team, go to the manager or Jira, whatever, whatever is giving them the task, they go do another task, right? That’s their job. A mid level engineer is doing a feature. A feature has many tasks and, and the feature is something that might take, you know, a couple of weeks or something. And that’s what they’re, that’s what they’re working on. And the senior engineer is doing projects.

[00:28:14] Projects have many features which have many tasks. Oftentimes projects are ones that are working through others, perhaps other mid level engineers or other junior engineers who are working underneath them. So there’s some kind of leadership and delegation and organization and communication expectation for this level 5 person or the senior person, staff person. Level 6 at Meta, for example, would be somebody who is responsible for some kind of goal, right?

[00:28:38] So this is the goal. In order to accomplish this engagement goal or this, you know, revenue goal, we have need to have multiple, multiple projects which have multiple features or multiple tasks, right? So there’s this natural cascading hierarchy of what’s expected based on ambiguity, right? That’s what levels really mean is how much ambiguity can you handle. The ambiguity of an intern is very different than the ambiguity of the CEO.

[00:28:59] And that’s what differentiates the levels and differentiate compensation. And then this like level seven or this principle or distinct, whatever you want to call it, this next level is more about organization. So I’m responsible for an entire organization which has many goals, which is many projects. It’s just many features, many tasks. And then maybe whatever the next level is, it sometimes it’s distinguished or different people have different names for this thing, but that’s responsible for industry.

[00:29:25] I’m responsible for this industry which has, over this entire business, which has many organizations, which has many goals, etc. So I think that when you’re choosing stories to tell about landing a certain job, you understand what those level expectations are at the company that you’re targeting and then making sure that you’re telling stories that hit those, hit those notes. So let’s take for example, staff, Staff level engineer is a big difference between that and a senior engineer.

[00:29:52] And really it comes down to how the, the breadth of impact that you’re making. So you’re, you are making impact more than just your small project and your small area. Oftentimes it’s an entire team or maybe multiple teams you’re working with. Oftentimes you’re telling stories that involve a lot of, of working across an organization, a very large organization. So this is why it’s so difficult for, for startups up engineers, unless you were the founder, for example, to get a job at staff or higher is because those, those experiences are oftentimes limited by the number of people that you’ve worked with.

[00:30:22] So if you didn’t have to, you know, if there was only one stakeholder CEO and like two or three engineers, it’s pretty difficult for you to demonstrate the kind of depth of organizational leadership that is required for that, that staff level position. Then there’s like a technology complexity. So here you can, and it’s really difficult. In a behavioral interview, how do I communicate the difficult to this technology problem, this, this bug or this, this, this decision?

[00:30:45] And you have to quantify that. So you have to talk about the amount of time it has you to quantify the risk. You have to talk about the number of people you had to talk to to get advice. Or you have to somehow give me a sense of how risky it was, right, in order to make this choice, how difficult it was for you to back out of this choice, for example, so you can, you can talk around the complexity of this technical problem and give me a sense of what is there.

[00:31:07] And then there’s oftentimes leadership pieces. This is something that people forget about. They talk, maybe they, maybe they talk about tech. Almost always they talk about technology, right? Engineers, oh, I love talk about tech. You know, probably if anybody, they talk too much about tech, right? Then there’s this organizational thing which I talked about before. But sometimes people forget to even talk about the leadership parts.

[00:31:25] A lot of projects involve some kind of influence and mentorship, for example, over others. So now you’re convincing people to do things maybe they don’t want to do. That’s a big part of a staff engineer. Motivating people or getting concessions out of other teams, aligning on roadmaps, that sort of thing. And then there’s also mentoring. So how did I mentor and help the people who worked underneath me and made them better?

[00:31:44] How did I make the team better? How did I make the org better, better? So those things are oftentimes forgotten about when people are trying to anchor this, the listener and make them think, yes, this is a staff level engineer. So really, and we go back to what you said earlier. If you can reflect on your own career and understand what makes you successful and you can identify those pieces which differentiate you from other engineers, those are the things that you need to talk about in the behavioral interview and vice versa.

[00:32:10] So if you reflect back on your career as A behavioral candidate and you think about what made me successful, then you can start to repeat those things in your day job and you’ll be more successful in what you do.

Ryan:

[00:32:20] If you know the rubric, then you can do the rubric. And you can also talk the rubric as well.

Austen:

[00:32:26] Right. So know the rubric. Right. That’s the most important thing.

00:32:30 — How non-tech leads can signal scope

Ryan:

[00:32:30] I think one of the biggest ways that you talked about scope was kind of in the organizational complexity or how many. What’s the leadership position you are in the org. But what about specialists? So I’ve worked with engineers who. They’re solving problems that no one else can solve. They’re their own snowflake and we need that person because they’re pushing the industry forward and it’s having a lot of impact.

[00:32:52] How is someone like that supposed to talk about their work?

Austen:

[00:32:55] Yeah. So the first thing is to understand business impact of what you’re doing. And I think this is really hard for some engineers, right. Who are focused more on the technology. But there is some reason why this technology was required and some kind of context around which it lives. And I always encourage people to think about, what would Steve Jobs say about this technological advancement that you’ve brought about?

[00:33:16] Why is it that that this project was so important or so critical to the company? What’s the business context? So make sure that you’re delivering that one.

Ryan:

[00:33:23] Framing. I know that’s common in promo committees is this person solved problems that those people couldn’t and, well, those people are staff. So then if he’s solving problems, staff people, then he or she must be greater than staff. So when your manager’s saying that on your behalf in a promo committee, it makes sense. If you say that on your own story, you say, yeah, I landed this project and actually there’s a team of five staff engineers that failed for a year before I got there.

Austen:

[00:33:55] like too much, I don’t think. Actually, I don’t think so. I totally disagree. So I think this is one of the methods that I hear and is very successful is again, you’re talking around this complexity to give people a sense and to pattern match in their mind. So behavioral interviewers are pattern matchers. They are looking to see if you match the patterns that they expect for this level. And one of those things is going to be whether or not you have solved problems other people haven’t.

[00:34:18] So I don’t think it is necessarily bad to say something like, when I took on this project, I was the third owner and this is where they had Failed. And this is where I’m looking for. There is not just like, haha, I’m better than all these other people. I’m looking for the insight that made their efforts unsuccessful. So oh, these people attempted, you know, this product market fit and that didn’t work.

[00:34:37] Or these people attempted, you know, this technical approach and that didn’t work for these reasons. And that kind of judgment and reflection is a big part of, of, of, of of assessing someone at a senior level and something that people often forget. They just tell the story. I had a problem, I solved the problem, I’m done. And not giving me some sense of what they learned or what the deeper insight is.

[00:34:58] And to me that’s the sign of somebody who’s, you know, above staff level.

Ryan:

[00:35:03] The analogues between promotion conversations and recruitment conversations. It makes a lot of sense to me. And one thing that I see actually because when I was looking through the YouTube comments of previous interview done, there were some people were saying people can just oversell themselves and the person was a little bit salty that someone could manage the optics and kind of get a good recruitment outcome.

[00:35:31] And I see very similar stuff on the promo. The promo content that I’ve made is some people, usually a vocal minority is saying, oh, this person just really knows how to sell themselves. And I think that’s true and unfortunate that it can just be not necessarily your achievements by themselves just objectively being true. You kind of have to, it’s a very human process. So how you sell it, that is going to have a big impact on how it’s perceived in both of these.

[00:36:06] So I, I guess it’s one of those things where it’s just how it is and you need to learn how to play that game if you want to have good results in the game.

Austen:

[00:36:15] Yeah. Welcome to the world. Right? And this, this works for, this works. This is the case in our careers, this is the case in our personal relationships. Don’t you know that I love you? Can’t you, can’t you just feel that I love you? That doesn’t work. Right. You have to do the behaviors. You have to demonstrate the things that in your relationships. You have to demonstrate the things in your career and you have to, to, to tell people.

[00:36:36] Right? You have to communicate those things. And that may be unpleasant for some, some folks and it can be difficult. But this is part of, of, of us maturing as, as people and us, us maturing in our careers is understanding that there are certain things that are worth doing that maybe is a tax like on our, on Our progress. You don’t have to do this them, but you also will not get promoted or you also will not get the like staff level engineer the job.

[00:37:04] Right. I, I, I tell people, look, on the other side of this principal engineering job is a million dollars a year compensation. So you better eat your Wheaties before you go into that interview. Right. You better be ready. And, and you can say what you want about like how difficult it is to assess people, whatever, but they’re going to give you a million dollars a year. They’re going to put you through the ringer.

[00:37:23] So you need to be to ready, ready for that.

Ryan:

[00:37:26] So going back to the, the, the promo committees, I, I wanted to understand. I’m just kind of like recollecting all the people you mentioned. So there’s the sorcerer, there seems to be a hiring committee. There’s the people, there’s a recruiter. Okay, yeah, the recruiter. There’s the people in the hiring committee. There’s the people who conduct interviews and I guess they draft up notes that the hiring committee reviews.

[00:37:49] But who’s the actual interview decision maker in these processes?

Austen:

[00:37:53] Right, right. So we should talk. There’s a sourcer. There’s also a different person called a recruiter. I forgot to mention them. But usually the source for someone who finds you, they typically will hand you off to someone called a recruiter. Once you get into the process, once you get through the phone screen and this person is the person that you’re going to be doing the negotiation with, the person who’s going to be organizing your loop.

[00:38:10] The person’s going to be advocating for you in front of the hiring committee. So there are obviously many decision makers. There’s a sourcer who’s just looking at your LinkedIn and deciding whether they should contact you. There’s that, that decision. There’s also the decision of the, you know, the initial phone screener or whoever’s talking to you at different levels that could be again that, that sourcer doing a, a multiple choice question or it could be an engineer who’s giving you a phone like a coding phone screen.

[00:38:34] Then there’s the hiring committee. So the committee makes a decision based on, and usually it’s driven by consensus. So but sometimes we would have to fall back into voting but the consensus would be established of whether or not this person should be hired at this level or whether we should do a follow up or whether we should drop the person from the process. And then again, like I said, there’s this decision being made generally as a consensus among Two or three engineering directors who are, you know, who are above us making that final call.

[00:39:05] And those folks would operate mostly in consensus, but sometimes they would vote in the hiring committee.

00:39:11 — Referrals and bias

Ryan:

[00:39:11] Have you ever seen cases where there was some obvious bias? Maybe someone’s son happens to land in the hiring committee of the father or something like that?

Austen:

[00:39:26] I’ve never seen an experience where there’s any kind of overt nepotism or bribery or anything that’s really exciting like that. But I have seen cases where referrals play a huge, a huge, you know, huge role. Whether or not the referral comes from somebody who’s very senior or whether or not someone has actually showed up to, in the room in the hiring committee. So sometimes there’s a friend or there’s someone who’s worked with this person before and they will show up and advocate for you in the room and to the hiring committee.

[00:39:51] And that does make a big difference. So if you can find a referral, I know this is not news or to anybody, but if you can find a referral, you can find somebody who knows you and who will be willing to go and spend their time in a meeting that will make a big difference.

Ryan:

[00:40:04] When I worked at Meta, I mean referrals and as a low level engineer, they just kind of felt like this thing where I just fired it off and forgot about it. It. But are you saying that at a higher level, like the, the level of the person matters and to the point where they can even hop into the hiring committee, Is that right?

Austen:

[00:40:24] For sure. So. Well, yes, for sure. The level, the person who’s, who’s providing this advice is, is very much matters. So if it’s a VP or a director who is referring this person, they understand that their, their reputation is on the line. They are not just passing through someone they found on LinkedIn to this, to this process. They are, they, I’ve directly worked with this person, most likely and are willing to put their personal reputation on the line to, to hire this person, that’s a big deal.

[00:40:48] I will say that referrals of course have varying qualities. So we talked about level, also the content of the referral. So if it’s like, hey, I knew this person in school, like maybe consider them is very different than I work with this person. They have this quality, this quality, this quality, this quality. And that’s why you should hire them. And so for certainly when you are asking for a referral, especially if the person has worked directly with you, you should provide information to them.

[00:41:08] You should like write the referral for them, provide them information about that they should pass on to them, the, to the hiring committee. And we would certainly look at that. So we would, we would read through those referrals and understand whether or not this is just somebody who happened to come across one of our employees and that’s how I got referred, or someone who worked directly with them.

[00:41:21] And if it worked directly with them and they say relevant things, that makes a big difference. And I would say the biggest difference it makes is when you’re on the border, when you’re on the bubble. So if you are, you know, maybe you, you flubbed a couple of interviews. I actually think this was my, my situation. So when I applied I, I flubbed a couple of the interviews, especially in the phone screen stage.

[00:41:39] And I got an, I got a follow up and I’m pretty sure I got the follow up because I had a, a referral from somebody that I was in school with who worked directly with me. And so. Thank you. Thank you, Nathan. I appreciate it.

Ryan:

[00:41:48] Let’s say I, I did relatively bad. Like I probably would not have passed. Not, not terrible. But it’s pretty lukewarm from everyone in the room at best in the hiring committee. But my, my referral is the strongest referral you’ve ever seen from a vp. Let’s say at their previous startup I was their chief of staff and I was really organized and I did an excellent job. And then now they’re a VP at Facebook or something like that, and then that person comes in, guns a blazing.

[00:42:22] Would that type of referral make me pass in that case?

Austen:

[00:42:26] So typically referrals are going to be helping you on that bubble. And this was saying earlier. So it’s really about whether or not you’re going to get a follow up interview. I have never seen a case where there’s a general consensus that we should not hire this person. But yet the referral results in them getting a hire. I think that may happen more at leadership levels where the kind of people that you bring from your previous jobs that, that could be very sensitive to that.

[00:42:49] And leadership hiring is a different, whole different ball of wax. But on the engineering side, I’ve never seen a case where there’s almost a, you know, unanimous consensus that we should not hire this person. They don’t meet the bar and suddenly a like VP comes in and then is able to push the, the committee to, to hiring them. I haven’t seen that case, but I have seen the case where the VP will be able to, or whoever it is we’ll be able to push the committee into giving a follow up or maybe multiple follow ups.

[00:43:12] Right. And so I think that you, there is a, you still have to perform in the interviews as an engineer, but if you have reached like a senior manager or director level, I think that it is a little bit more about who you know sometimes than, than your raw performance. I will say that a good hiring process is not like that. So good hiring process is one where you have identified what success, what makes people successful inside the organization and then the hiring process is evaluating that and people are making a non biased decision.

[00:43:37] But I think the number of decision makers is smaller as you move up and more influenceable by those around them.

Ryan:

[00:43:45] One thing that I think you can only really see with experience and you worked at Meta for quite some time is I’m always curious what is the correlation between someone’s performance on interviews and their downstream performance at the company? And I know that you, you’ve been involved in hundreds of interviews and you’ve seen people go and enter the company. What would be that correlation? Like how often do you see that that person smashed interviews and they’re doing excellent at the company or maybe they’re off.

Austen:

[00:44:18] So what we have looked at is not so much in terms of the number of interviews that they succeeded or failed, but rather the confidence level. So the confidence level oftentimes people would assess the interviewer’s confidence level. You would put in a hiring decision level decision. And like how conf are that confident decision does correlate with someone’s future performance. And I think what that goes to show is that people, there is some sense that, that the interviewers are sort of applying to their.

[00:44:44] Even there’s a rubric and they try to structure it as much as possible. There is this kind of, kind of sense that they get from, from whether or not someone will be successful. And that is predictive of, of future success. And it makes sense, right? We, you know, if you’ve done hundreds of interviews as a calibrated person, you, you kind of know what of kind good looks like. It may be hard to define and you try to define as much as possible.

[00:45:03] It reduces bias when you do that. But ultimately sometimes it comes down to these kind of, you know, feel things. I will say that certainly there have been people who rock the interview and don’t do well and certainly people who are kind of rocky in the interviews and they do great. And I, I think that what that shows you is, is what we’re talking about earlier, interviewing is a skill. So it is not the same skill as doing the job.

[00:45:23] And unfortunately, in the world we live in, you do need to spend some time getting good at the interviewing skills skill.

00:45:28 — What the rubric looks like

Ryan:

[00:45:28] We talked earlier about the rubric that those interviewers are trying to fill out and I’m curious for the, the behavioral side of things, what does that rubric look like?

Austen:

[00:45:39] So every company is a little different. A well run company, especially big companies, oftentimes they hire PhDs in industrial psychology. They’re called selection scientists. And what they have done is they’ve assessed, they’ve gone around your company, they’ve talked to different people, they’ve tried to understand what makes an engineer or whatever the role is very successful and they’ve codified those things into a set of signal areas that they look for in the behavioral interview.

[00:45:59] So at, at Meta, there’s five of them. They are driving results, embracing ambiguity, resolving conflicts, growing continuously, and communicating effectively. Every company is going to have different ones. They might purely just be the company values and we can talk about that. But for example, at Meta, those are not the five company values at Meta. So you, you should definitely talk to the recruiter and as you’re having, as you’re preparing for the interview and get this rubric in advance, it’s very important if you can get it or go on the look for it.

[00:46:26] So these, you know, whatever five to eight things, generally they fit into a broader set of categories that I talk about in the book that I call signal areas. And there’s eight of them. And so when you’re, when you’re working on your stories and your preparation, you need to have this rubric in front of you as you’re thinking about your stories and then you’re categorizing your stories based on these signal areas.

[00:46:47] And then what you’re doing inside the interview is this decode, select, deliver, loop. This is your core operating system. When you’re inside of a behavioral interface interview, when you’re listening to a story, you’re understanding, decoding what this, what is it? Why are they asking this question? What is, what is it they’re interested in? And it’s probably one of these eight areas or something related to their, to their company values.

[00:47:07] And so if you can identify that, then you can select a story which is appropriate and fits that, you know, delivers the kind of signal that the interviewer is looking for. And then you can deliver that in some kind of engaging way. And that’s your core loop, decode, select, deliver. So it is really important that you understand what these rubrics are oftentimes a recruiter will give it to you, you.

[00:47:24] And at a smaller company, it’s a lot harder. They may not have thought about this at all. They may not have structured their, their, you know, their behavioral interview process with any kind of rubric. So there, I think you rely on company values. You also rely on conversations with the recruiter, usually at smaller places. You may have even talked to somebody on the team before you get to the behavioral interview.

[00:47:43] And I think you should ask them questions like, what makes engineers successful in your company? What do you look for in the hiring process? And then take that and build your own little sense of what’s valuable for that company and then use that as you’re going through your decode select deliver loop.

Ryan:

[00:47:57] That decode part is so important because, I mean, I’ve also been an interviewer for hundreds of interviews. I did a bunch of engineering management interviews as well. And it’s interesting is a lot of candidates, I feel they would tell me something and I’m looking for a signal and I ask a very targeted question. I say, can you tell me about this? And then they don’t get. And they talk past it or they continue on their spiel.

[00:48:28] I thought you’re missing opportunity. Help me, please help me fill this out. I got a blank spot here I’m trying to fill out. I asked you the specific thing, but you’re continuing on some rehearse thing. It’s actually like a mutual exercise and helping fill out this rubric. So, yeah, I think that’s so important to interview to what they’re actually thinking about.

Austen:

[00:48:52] Your job as the candidate is to sort of tell the story from a perspective. Like you would build a trailer out of a movie. Different trailers. Robert Hamilton, the PM behavioral coach, talks about this and he says, like, well, this movie, this movie may have some action parts, may have some romance parts, right? And you can sort of remix the trailer in different ways to see what this movie is. Right?

[00:49:10] And at the same way with these stories. So when you have a story, it may have a part about ambiguity, it may have a part about communication, it may have a part about conflict resolution. And so you can take the same story. And in fact, you probably should take the same story. Like I said earlier, you should identify what are those core stories of stories. You really want to get out and be able to build a movie trailer which sort of fits the question that they’re asking about.

[00:49:32] And you need to do that early, right? To your point. So you can’t be that in three Five minutes in, then we’re starting to get to the ambiguity part that the person asked about ambiguity. It needs to come up upfront in that initial context setting. And this is, this is the way that you know that you are internalizing you being a partner with the interviewer. I think people forget about that, that like you said, the interviewer is trying to accomplish a task, trying to evaluate you.

[00:49:55] You need to be a partner in that and you need to help the interviewer get to the signal that they’re looking for.

00:50:00 — OpenAI and Anthropic specific discussion

Ryan:

[00:50:00] You mentioned tailoring your stories and your experience to the specific values of the company. And I was thinking what might be interesting, we could go over some concrete examples maybe with some hot companies right now like OpenAI and Anthropic. And yeah, I’d be curious to hear what you see in their values and how you might mold stories to help people who are looking to get hired at these companies.

Austen:

[00:50:27] Yeah. So the first thing I would say encourage people to do is first understand your stories from the perspective of these eight core areas that I talk about. Because most company values come can be bold down into these eight different areas. We could talk about them at some point, but inevitably, especially for newer companies like these AI companies, there’s some part of their company values that don’t fit within that.

[00:50:47] So, for example, OpenAI has this one that’s called act with Humility. Really what that means is growth. That’s my one of the eight areas. Do you respond well to feedback? You know, do you, do you, you know, are you, are you seeking to improve yourself, improving the people around you? That’s kind of the same thing. But they do have one called Feel the AGI, which is not really one of my eight areas.

[00:51:06] So you’ll need to understand what that that means for them. And so what that means to them is that you are very optimistic and positive about what AGI and what AI could do for the world. Now, Anthropic has one called Hold, Light and Shade. And really what they’re trying to assess there is that you can understand both the positive and the negative implications of AI in the future. And so for me, you need to understand what the cultural zeitgeist is of this company that you can assess via their company values and by researching them them before you go in.

[00:51:34] So how might you do that? Right. If you are telling a story about how you leveraged AI in the past to Anthropic, you would really want to mention how you thought about the potential negative implications of this project from an AI perspective. How did you go about mitigating those, how did you go about assessing those? And if they don’t hear those things, they’re not going to feel comfortable about hiring you.

[00:51:55] And likewise on OpenAI, if you are not somebody who’s very AI forward and very much excited about, about new domains that we can apply AI to, they’re not going to be as interested in hiring you. So when you’re reviewing your stories, you need to understand what every behavioral interviewer looks for. Those are those different, those eight different areas. Things like ownership, handling ambiguity, conflict resolution, the things that we do on a daily, daily basis, but also mixing in what are those specific things that are, that are unique to their company.

00:52:22 — Most common mistakes senior candidates make

Ryan:

[00:52:22] We covered how to avoid getting down leveled and, but I’m, I’m also curious, before we kind of leave that kind of topic, what are the most common mistakes that people make in behavioral interviews that lead to worse results than they should have?

Austen:

[00:52:38] Yeah, let’s stay focused on senior engineers for a second because I think that, that they are slightly different. So the number one, what we talked about before, which is inappropriate choice of stories, so you didn’t choose stories that fit the appropriate scope that you’re targeting. So that’s number one. One, I would say the, another one is usually as you’re more senior, you’re a better communicator and so you might talk a lot.

[00:52:58] So this happens especially for manager loops. Oh yeah. Like, let’s be talking about this project and da, da, da, you should talk, talk, talk, talk, talk. So this sense of giving around too much context is really common. And the, and the key there is something like you should only give the kind of context which is required for me to understand the behaviors. Again, that’s what I’m assessing you on, is the behaviors that you’ve done in the past.

[00:53:17] And if you give me too much context as relevant for this, then, then I’m, you’re just, you’re just using your own air, wasting your time. And another guideline there is when you’re, when you, even if you are talking about the middle part of the project and what you did, if it’s been like 30 seconds since you have told me something that you did, like some kind of action, you know, some kind of verb is coming into this conversation.

[00:53:37] If it’s been that long, then you should rethink that. Like maybe I’m providing too much content, too much technical detail, too much backstory, too much asides. Whatever it is, you need to keep the, you know, keep the, keep the action coming. Keep that Work stories are just not that exciting. Okay, let’s just be honest, like Stranger Things versus like me listening to a behavioral interview like I’d rather watch Stranger Things, right?

[00:53:57] So I, I think people want to, want to be, they want to see action and movement in the story. So keep it, keep it moving. Another one that’s really common for senior engineers is opening themselves up to uncharitable interpretations. And I call this one the opposite is thinking defensively. You need to think defensively. So in this, in this kind of senior role role, it’s very risky to bring people on, bring leaders on.

[00:54:17] They have a huge impact on the team. And so interviewers are very risk averse. So if you start presenting and telling stories like, well, you know, the code base had a lot of technical debt, so we decided to do X. Sounds like a great story, right? Hey, you’re, you’re somebody who solves technical debt. But if you’re the senior engineer in the room then like how did we get this technical debt? Right?

[00:54:34] It’s your fault, right? So I think this kind of how can my stories be interpreted as unfavorable is really important for you to consider and the, the way around that is to make sure that you are compensating when you’re telling the, the context in the story. So it could be that like we were a startup, we needed to, to close, you know, our next round of funding. Therefore we decided to, to take on this technical debt and then it was our role to solve it and I decided to prioritize it because xyz.

[00:55:03] So here we are giving some kind of thinking or backstory or judgment people piece to how you ended up in this situation. We touched on it briefly before, but another one is not talking enough about the non technical parts of the work that you do. Obviously technical parts, super important. You need to establish yourself as somebody who can solve hard technical problems. But if you’re a senior engineer or a staff engineer, principal engineer, a lot of what you do is working with people.

[00:55:25] How you mentored, how you organized, how you delegated, how you influenced the roadmap, how you resolve conflicts, how you worked across teams, how you managed up, right? So those parts of the story are super important. Other thing is that oftentimes you’re telling stories that are very long. Okay. So you’re talking about, especially a principal engineer, they’re telling stories that are often multiple years, two, three years worth of refactors or, or some, you know, large product build in a new product space.

[00:55:49] So you need to have your stories well organized. And you need to understand what’s really important for the listener out of that story. So I recommend people have some kind of table of contents at the very top, which requires you to understand what’s important. So you could say something like, well, some business context and this is why I’m doing this thing. Da, da. And then you say, oh yeah, there are like five interesting parts of the story.

[00:56:06] You know, how I initiated the idea with management, how I aligned the stakeholders on the technical approach, how I solved some difficult technical problems and like how we did this very complex rollout over, you know, multiple, multiple years, something like that. So now set the stage for the conversation and then I can tell a longer story, keeping it organized in these different verticals. And it also gives me a way to come back to them.

[00:56:28] So if I’m a candidate and the interviewer is peppering me with follow up questions and they’re interrupting me, which is super common for these senior engineers, senior, senior interviews, then I can always come back to, oh, you know, remember that part that I was telling you about the hard technical problems? Like let me go back to that. Then you can, then you can bring the listener back to the most important part.

[00:56:45] So that story organization piece is super, super important for senior, senior candidates. We didn’t talk very much about choosing those stories. And I did say that the most important thing is scope, which I don’t think is immediately apparent to most people who’ve thought about behavioral interviews. Oftentimes you would think, well, relevance is the number one thing. I have to tell a story which fits the question that they’re asking.

[00:57:07] And that’s very logical. Right? But like I said earlier, the most important thing is that you get out of the behavioral interview having told the stories that showcase your impact the most. So scope is the number one thing when you’re choosing a story. How can you fit a big scope story into the question, question that is asked? Number two is relevance. Obviously you can’t tell a story about, you know, sometime when that wasn’t very ambiguous.

[00:57:28] If you’re being asked about ambiguity, the third thing is recency. So of course, you know, newer stories are more important than older stories, but I would rather hear a large scope relevant story from a senior engineer versus someone that’s very new. So recency is not the most important thing. And the guideline there is something like two to three years for sure is okay beyond three, four years.

[00:57:52] Now you have to have a really good reason for telling that story. And of course this scale increases as you get more Senior. So if you have a 30 year career then telling and you’re applying for a principal role, then telling a story from 10 years ago probably isn’t that bad. And the last one is uniqueness, which is some people don’t often think about. But how can I tell a story that I haven’t told already?

[00:58:12] And this becomes super important for interviews where there are multiple behavioral rounds where you’re likely to tell multiple answer, multiple tell me about a time kind of questions. So maybe the, the question in the like Project Deep Dive might not be the same story that you use when you’re just asked about an ambiguous project in the general behavioral interview. So thinking about how you can balance uniqueness across the, the portfolio of behavioral interviews that you’re being given as this loop is important.

Ryan:

[00:58:37] When you said that scope is more important than relevance. Is it like let’s say I’m in a behavioral interview, I’m, I’m giving my stories and it almost reminds me of a politician where they ask a question and I, I have my rehearse thing that’s not exactly what they asked, but it’s filling in the rubric is that you’re saying that’s best as opposed to saying that is best exactly what they say.

Austen:

[00:59:01] So be like a politician I guess is unfortunately what I’m advocating that you do. Because I think that again, you want to, to leave the interview having showcased the ways that you have delivered business value, solve problems, done the things you do as an engineer the best. Right. And if, if you just stay focused on answering their specific question. For example, like a question like tell me about the time you had a conflict with a product manager.

[00:59:25] Maybe you do have a story that you had a conflict with a product manager, but maybe the bigger story is a conflict with, with the director of engineering or a conflict with your direct manager. So what I would do is I would try to pivot that, that story into the one that is, is larger. And they may, they may constrain you, you know, they may come back to it and then you’re stuck telling the smaller story.

[00:59:45] But what I would do is I would showcase that you have this other story. So you can say something like I did have a conflict with a product manager. It involved this and this. And I was able to overcome that conflict by compromising on this. You know, I could tell that story, but there is this other story that I could tell about how I had to go through a longer conflict resolution experience where I had to collect data and I had to, you know, collect other People and we had to have multiple meetings about it.

[01:00:07] That might be a more interesting story. I can’t tell that one. So you can pivot the conversation focused on conflict resolution into a story that might fit better for what the interviewer is actually looking for. They may or may not really care about when you had a conflict with a product manager, they might just be collecting conflict resolution signal.

Ryan:

[01:00:25] Let’s say you’re in the middle of the interview and you’re trying to figure out, am I talking for too long or is this person still with me, or should I, am I saying the right thing things? How often should the interviewer be talking? How often should you be talking?

Austen:

[01:00:40] So there are different style of interviewers. So some interviewers really are listening. So I tend to be that kind of person where I would. Interested in what I think. Interested in what you think is interesting. So. And I’m assessing you based on what you think is interesting and where you’re taking me. And I’m getting a lot of communication signal out of that. But some interviewers are not like that.

[01:00:58] They are. They interrupt you frequently. They look at, they want to look at, pick out some part that you mentioned and like, turn it over a couple times in their hand and then get back to you. So they may have a lot more thoughts, follow up questions. They may interrupt you a lot more or they may switch questions. They may collect a little bit of signal on this question and then like, move rapidly to the other one.

[01:01:14] So I think you’ll figure out which kind of interviewer you have pretty, pretty quickly in the, in the interview. And you should be emotionally prepared, especially for the second one. This person who interrupts you frequently can be kind of jarring for you. And you can think, oh, I’m failing, I’m not doing the right thing. But no, you’re just getting an interviewer who likes to interrupt people and likes to bounce around.

[01:01:31] But to answer more specific to your question, I think you can look for signs. So this is where you need to use your human part of your brain, not just your, your structured and organized part of your brain, but you need to assess, like, are they looking at me? Are they taking notes? Are they any kind of indication that they’re. That they’re engaged? And if you notice some lack of engagement, then I would pause and ask them, is this what you’re looking for?

[01:01:50] This is what I was about to say, like, is that still relevant? And give them this opportunity. Can. A good interviewer is willing to interrupt you. A good interviewer, a calibrated interviewer, someone who is, is confident in what they’re doing, they will be a little rude to you because they are, look again, they are looking to acquire that signal and they are willing to, to go through a little bit of social awkwardness in order to acquire it.

[01:02:08] But some interviewers aren’t that good. And so you need to make that easy for them to be able to stop you. Which is why I recommend not talking for more than, you know, two to four minutes if you’re over. If you’re talking for more than four minutes, five minutes, you got to be really good at giving a monologue like that. It better be really interesting for the, for the listener. And you probably, you probably should know what you’re doing if you’re talking for that level of time.

01:02:31 — How to prep depending on your level

Ryan:

[01:02:31] When it comes to interview prep, I mean, a lot of what you said sounds like there’s a difference between the levels. Senior engineers make different mistakes than junior engineers. And I imagine the weight of the behavioral interview’s importance differs across the level. Sounds like as you get higher up and higher up, they add extra behavioral interviews. So I’m thinking if someone was preparing for interviews across those high level buckets, how would you recommend they split up their interview prep time across coding, system design and behavioral?

Austen:

[01:03:08] Most people are spending way too much time on coding and not enough time on system design and behavioral interviews because those are the more murky ones to prepare for. However, I will say that junior engineers, your most important signal is going to be technical. So making sure that you nail those coding interviews, mailing the system design interview interview, it’s going to be the most important.

[01:03:24] And then the behavioral interview, the key to focus on is your thinking, your motivations because you may not have a ton of accomplishments to lean on there, but you can lean on what you thought about or what kind of ideas you had or things that you tried. Maybe they didn’t work. So those kinds of signals of what my future performance could be or my future impact is going to be super important for junior engineers.

[01:03:46] For mid level engineers, I think you need about balance. I would say that AI is changing this game. In the past, I think you could still focus even as a mid level engineer, only on the technical parts and just make sure that you didn’t flub the behavioral. I don’t think that world exists anymore. Number one, there are so many candidates in the market with so many layoffs recently that you need to shine across all different interview types.

[01:04:07] And the second thing I would say is more and more of that technical work is being done, you know, by an agentic coding experience. And so we are looking for mid level engineers who can own problems problems end to end. And how do you see that kind of ownership? You’re going to see that in the behavioral. We need mid level engineers who are excited about growing and learning new things because the technology and the approach to engineering is changing so rapidly.

[01:04:26] And how are you going to assess that? It’s going to be assessed in the, in the behavioral interview. So I would, I would balance more balance your time. Right. The beginning part of the interview process that we talked about before is often some kind of coding assessment. So you need to get your coding interviewing practice in early. But you know, once you get that on site schedule or ideally even before that, some weeks before that, you have spent some time thinking about your behavioral questions.

[01:04:47] You’ve spent some time identifying the core stories from your past that you’d like to tell. You’ve looked at their, their values. You’ve looked at the rubric that you find for the behavioral interview. You’ve made sure that you have some stories that fit each one of those key areas that they’re going to be asking you about. And then once you get to senior and above, a lot of it’s going to come down to the behavioral interview.

[01:05:05] So like I said before, first thing I would do when I look at a staff level packet is I would go to the behavioral interview. There just aren’t that many different ways to design like a web crawler in a system design interview. Or aren’t that many ways to solve twosome in a coding interview such that I could see whether or not someone is a staff level or just a senior level. And I would go directly to the behavioral interview to see what the organizational impact is, what their level of influence is on others, how they resolve conflict.

[01:05:32] I would look to see what scope of projects they’ve they’ve been able to, to accomplish and what, what level of business impact they’ve been able to deliver deliver. And that would be my how I would determine whether or not they were really truly a staff engineer. And like you mentioned at that, that you know, super senior like staff plus kind of percent principal distinguished level. They’re adding additional behavioral interviews and oftentimes they’re looking at specific things, things like how you work cross functionally with, with partners.

[01:05:56] They may give you a PM for example, or another person from a cross functional partner that will interview you and see how you work with others. Or they’re going to be wanting to get more information about one of Your bigger stories, one of your bigger projects. So they’re going to give you some kind of project deep dive interview where you walk them through some technical and organizational challenges you’ve solved in the past.

[01:06:16] And so those become the differentiators and the technical ones become kind of like check marks. Like you have to pass, you have to get over the bar. But really what’s going to get you hired is going to be shining on those behavioral sides. So certainly a slide. I would say that most people under invest in behavioral interview. So, so it’s probably going to take you at least a few weeks to, to, to perform well.

[01:06:37] And if you are at that staff or above, I would say, you know, start, start when you start doing your leetcode right, Start thinking about those past stories, start working on how you position them. Get some feedback, go do some mock interviews at least with a friend, if not with a professional. And especially if you’re going to be applying for a FAANG or, or you know, open eye or anthropic one of these big companies that’s very in demand.

[01:07:01] I would try to find somebody who is calibrated at that company who can give you a mock interview and give you that, that squishy cultural sense that we were talking about earlier and reflect back to you whether or not you’re hitting the right cultural notes.

Ryan:

[01:07:13] When you were in hiring committees in the past for maybe more senior candidates. When you have a candidate who does okay on the technical sides but really kills it on behavior girl. Is that the type of packet that could go through at the highest levels?

Austen:

[01:07:32] Certainly at the highest levels, definitely. If there’s a place where you’re going to flub a coding interview and be okay, I think that staff or principal level or manager level, those places where we just don’t expect you to be doing as much day to day coding. And so we would discount poor performances in those interviews for sure. I think it’s much harder to do that at the junior levels. I will say that I had a candidate that I did the behavioral for and this candidate was, had not accomplished a ton in their career.

[01:08:00] The, the technical things were just okay. But I kept seeing glimpses of, you know, ideas that this candidate had had that maybe they weren’t able to execute on or the manager didn’t agree or they couldn’t get it done in their organization but they kept thinking about things. And I said now this person has so much like potential in this person’s career that I pushed for that person to be hired. And that person ended up Being a staff person eventually.

[01:08:22] And I think that that was a good indication that, you know, there is a. There is like a. There’s a slope that we’re trying to assess, and the slope is often assessed there in that behavioral interview, even for junior folks.

01:08:34 — Subjectivity and bias

Ryan:

[01:08:34] We talked about some of the subjectivity in. In this type of interview and talking about how you talk about the. The scope of your work and all of those things that kind of help you with leveling conversation. But also another part of subjectivity is just how much does this person like you and their bias towards you based off of a lot of the soft influence you might have as you speak to them. So I’m curious how much influence you think that has on the outcome of the interview.

Austen:

[01:09:03] It scales exactly with level. So with a more junior candidate, I think that at least at a big company where they’ve spent some time structuring these behavioral interviews, interviews, and they’re trying to reduce this kind of do I just want to have a beer? You know, with this person kind of vibe that they’re. That they’re collecting. They’re trying to reduce that by structuring it and making. Giving the.

[01:09:22] The interviewer some sort of form to fill out or some clarifying what’s. What’s important in the hiring decision. But as you get more senior now, they’re expecting you to influence other people on the job and that. So how you come across in the meeting, are you confident? Are you comfortable? Are you able to hold on to this kind of like casual. Casual excellence. Right. Which I think is very much a quality, a cultural quality that, that.

[01:09:45] That Silicon Valley has in particular, this idea that I can kind of show up and I look cool, and I look. Not look cool, but I look calm, but yet I’m still very competent and can sharply discuss things and deliver things. I think they are looking for that kind of signal. And it’s. It’s definitely subjective. And so I think your. Your ability to connect with the interviewer in the first few minutes, if you come in and you’re disheveled or you come in and you’re nervous and you’re not able to be present in the meeting, well, that’s part of the signal they’re acquiring because you’re gonna have to show up into meetings and be, you know, put together and be confident and be able to stand in front of the CEO and deliver, you know, deliver good news or bad news or ask some questions or whatever, and they are looking for somebody who will do a good job with that.

[01:10:28] And they’re you know, this is a high pressure situation, an interview. You’re going to be in a high pressure situation at work and they’re assessing you there as well. So I think this is where you need to understand the cultural expectations of your companies that you’re, that you’re, that you’re applying for. Different countries have different expectations around leadership and hierarchy and how they approach things.

[01:10:47] I think we’re talking mostly about like a US centered kind of Silicon Valley centered or you know, associated places. Seattle, New York, those kinds of culture. And that place is one, one of, we need to be able to start the meeting with some kind of playful banter, you know, and then we continue with like hard hitting pieces. But sometimes I, I’m self deprecating and I kind of pout out of that and we laugh about a few things, but then we’re back in it and we’re doing this like intense thing.

[01:11:11] So that’s super common for, for, for those environments. Other countries, other companies, they may have different expectations. I think you need to understand what those are before you go in.

Ryan:

[01:11:20] Let’s say someone is maybe a staff engineer. They’re not necessarily the most, they wouldn’t describe themselves as a people person, but they want to come off strong in that interview. How would you reverse engineer how to represent yourself strongly in terms of all those soft influence type of things?

Austen:

[01:11:38] So the first thing I say is that the inside always comes out on the outside. Okay, I know you don’t want to hear that, but the reality is what you believe about yourself, what you believe about others really changes how you present yourself. So the first thing I would say is it is not just some like list of things that we need to do or a checkbox stuff that you kind of put on. It’s not a mask that you put on, put on.

[01:11:56] You need to change your, your internal beliefs. And Sam Lesson has this, this book he just released about Silicon Valley etiquette and he did this podcast recently where he talks about the importance of lowering your own heart rate before you go into the conversation. So yes, this might be your like one and only shot at this job. It’s possible. Oftentimes that’s less the case than, than most than most candidates believe.

[01:12:18] So sometimes we think like, oh, this is my one shot at getting this company. Well, if the company’s gonna be around for a while, you probably have other shots. So don’t put so much pressure on yourself. I know it’s hard, but it, that is the reality. You need to tell yourself, give yourself an internal Belief structure that makes it okay for you to make mistakes in this environment, you will come across a lot more calm.

[01:12:35] The second thing I say is try to, try to understand what. And be empathetic of the other person. So again, this is an internal thing. Believe that the other person is looking for a great engineer, a great manager. Whatever it is, whatever the role that you’re applying for. They’re looking for somebody who’s going to do really awesome. They really want, they want to find that person. They’re spending their time interviewing you.

[01:12:55] They really want to find somebody who’s great. And they’re not just looking to nitpick you and like, throw you out and judge you. Right? So I think this kind of internal belief where you believe, you know, this person is, is, is not my enemy. This person is, is, is, is, is, is a human doing their job. I would love to do my job alongside this person. Let’s have this great conversation. So that’s the first thing you have to start, start on the inside.

[01:13:17] If we are looking for other things, then I would say how you show up physically, so what you’re wearing, you know, go that like one notch above. That’s classic interview advice. What’s in your background so you know, how, how does your room appear when you’re doing a video call? Especially, how are you. How are you expressing care and interest in the person in the first few minutes of the meeting?

[01:13:40] So if you’re just, if you come in and you’re just waiting for the other person to say something, this is a very. You’re taking a very passive role and you’re showcasing that, you know, maybe you’re not ready for these kinds of, of more active of roles where I need you to build relationships with other people. Show me that you can be, be. You may not be like the most extroverted person, but you probably do value people in some way.

[01:13:58] Like, people are probably important to you. You know, your mom is important to you. Maybe your significant other is important. This, There are people who are important to you. And so let that come across in that first few minutes. Hi, how are you? I’m doing good, you know, or I, I’m, I’m interested in, in. I’m really excited to be here. Right. You can express enthusiasm even if you’re, you know, not the most extroverted person.

[01:14:15] So those first few moments, the, the brain is really looking at that, like kind of just figuring out, is this the kind of person I want to be with? And you’re making a lot of. They’re making a lot of split, split second decisions. And so how you show up in those first few minutes are important. Practice that, practice that with your mock interviewers. Don’t just jump right into the questions. Practice that kind of, you know, early, early part.

[01:14:37] Make the other person feel comfortable. Sometimes the interviewer is, is nervous too. Actually giving a behavioral interview is quite complicated, quite difficult, I would say. It is hard to, to engage the person in some meaningful way. Ask relevant follow up questions, but making sure that you’re collecting the information that you want to collect, also making them feel comfortable so that they give you the best signal.

[01:14:55] Also the behavioral interviewer, sometimes the hiring manager or some kind of leader in the organization. And so how they are coming across in the meeting is affecting the candidate’s perception of the company and perception of the team. So, you know, have some empathy for that, for that person who’s given this behavioral interview. Make them feel comfortable, make them feel like you’re right. Ease.

[01:15:13] How you do that is by changing first what you believe about the, about the situation.

Ryan:

[01:15:18] Who do you think would perform better on average in a behavioral interview? Someone who is extremely curt and cold, but excellent. Everything you ask them, they give you the right words, really concisely, great stories, but they’re not there to be your friend. They’re kind of cold and they’re silent and they wait till you ask a question. Follow up questions, they say no follow up questions. Okay, thank you for your time.

[01:15:47] Or someone who’s very warm and bubbly and oh, how’s your day? Okay, let’s get into this. Oh, I hope I do well. And they kind of build some rapport with you and then they, they do so. So on the, the actual stories themselves across the body of people who conduct behavioral interviews, which one do you think would do better on average?

Austen:

[01:16:09] Well, that’s the key. Who conducts the interview? So if I’m conducting the interview, the second person will probably be doing better because I, I like to engage. Or you might have, might have noticed that, right? I want to get those more people sides. So this is where, when you say building rapport. Building rapport doesn’t mean that you’re bubbly. Building rapport means that you are connected to the other person.

[01:16:27] So you need to understand in that first few minutes, like what kind of person is this? Is this a person, person who wants to be more business like and wants to assess things in a more, you know, cold and calculating way? Fine, that’s okay. Or is this person, you know, somebody who will kind of chop it up with me in the first, in the first few minutes. So I think you can, you should start on a positive note.

[01:16:46] I think you should start energetic. You should start, you know, believing that with some excite, showing excitement to be there but respond to the other person and how they, you know, how they present themselves. They may, for example, if there’s someone who doesn’t even ask you for introduction, doesn’t even ask you for tell me about yourself. They may jump into, tell me about a time when you had an ambiguous problem.

[01:17:06] Well, you know exactly who you get now. And somebody who just wants to go down the list and, and, and, and, and bang out the signal and, and be very efficient. And so you need to, to match that and be very efficient with what you, what you say. But if the person starts two or three minutes of weather sports, you know, how you feeling? How’s the rest of the interviews going? Like, you know, you have a different kind of person that you’re talking to and so you need to match their energy.

Ryan:

[01:17:30] When it comes to these subjective parts of the interview experience. What do you think’s more important, the first impression or the last impression?

Austen:

[01:17:40] Do we have to choose? Like, why do we have to choose? So let’s, let’s do, let’s knock it out on the beginning and the end. But if I’m forced to choose, I would say the beginning. I think most of the time I am making a decision within the first 10, 15 minutes of the interview about whether I’m going to hire the person. And it’s because I have been asking usually my biggest scope questions. Tell me about your favorite project.

[01:17:59] Tell me about your most ambiguous project. Tell me about something that you’re really proud of. And then I’m seeing what they’re saying. And if I would like to have that kind of performance replicated in my organization, then the rest of the interview is more like, let me check the other things. Let me check conflict resolution, Let me check growth. Let me make sure that this person is going to be successful.

[01:18:14] But ultimately I would love for them to replicate their past success in my organization. So if I had to pick one, I would pick the beginning. But you can rock the end. Like, why not? And the most important way to rock the end is to have relevant questions for the interviewer. And people oftentimes forget about this in their behavioral interview prep. They spend all the time talking about their, their stories and they’re getting really excited about telling, answering questions.

[01:18:34] But then when it comes to, you know, a question that questions they have for the interviewer they come up with something like, tell me about a day in the life of the engineer. Like the day of life of the engineer is pretty much the same across the company. Okay. You know, it’s like, hey, we go get up, we do some work, go to meetings. It’s just not that interesting of a question. So I think if you’ve thought deeply about the team, about the organization, about the product, about the company, and then you have some question which is helping you decide whether or not you want to work there, the interviewer is assessing you on that even.

[01:19:01] And so you can leave the impression that you are engaged in the process, you’re excited about the role, you’re really evaluating whether it’s right for you. That’s a great way to leave the conversation.

Ryan:

[01:19:11] What percent of these interviews? Because you mentioned the, the first impressions. More important, if you had to make the call, what percent of interviews do you think the result is decided within the first 20% of the interview?

Austen:

[01:19:27] Yeah. So lots of social science has been applied here. I’ve not seen any social science applied directly to tech interviews. And sometimes you cannot apply like cross apply, you know, interviewing at a fast food company to interviewing at a tech tech company. But the, the, the research says something like in the first 15%. Right. The first like 10 minutes of the interview, most maybe like 40% of of the decisions has already been made.

[01:19:50] And I would say that that happens for sure in, in our experience as well as, as you know, in engineering. Again I, I’m, it’s rare that I don’t want to hire you in the first 10, 15 minutes and then suddenly you say something at like minute 38 that’s making me, oh, I changed my mind. I think it is common for me not to, to know. Right. So for me to kind of feel like I’m not sure. Let me think about it.

[01:20:11] And then when I go back and I would write the notes down, I would consider it, consider them according to the rubric, which is the goal of the rubric, right. To shift our decision making out of this initial gut response and into something which is more cerebral, left brained. And there would be times when I was not sure and then I would make a decision later. But if I’ve made a decision in the first 10 minutes, it’s pretty hard for you to overcome that.

[01:20:37] And most of the time it’s because you’re choosing the wrong stories or I can clearly tell that you just, you have not done work at the level that we’re expecting for this company or you can’t communicate very well. So like you can’t tell stories very well. And that’s giving me for one thing usually means I can’t collect enough relevant signal to. To on the other axes to hire you. But also I know that if I hire you into my organization, like you’re not going to be a very good communicator.

01:21:02 — The questions you ask at the end matter

Ryan:

[01:21:02] I, I guess a lot of people, I’ve experienced this as well too, where you’re in the interview and it’s not going so hot and then halfway you realize they’re zoning out because they’ve already decided that you’re not getting hired. So that first half is so important. Another interesting thing I wanted to follow up on is it sounds like that last few minutes of the interview where they say, hey, do you have any questions for me?

[01:21:28] This sounds like a not evaluative part of the interview you. But it actually is and you should continue to sell yourself by saying, hey, I ask questions that matter that are important. I am aware of the things that are important and I’m asking about those. So what are good follow up questions to ask to make sure that you get the most out of that section?

Austen:

[01:21:52] Yeah, so one of the themes here in our conversation is everything is evaluative, right? So the early conversation with the recruiter is evaluative. The end of the behavioral interview is evaluative. The like hangout chat with the hiring managers, evaluate. Everything is evaluative. And I think that makes sense. That’s just part of life. But yes, definitely. So I think first, first off, you should understand what’s important to you.

[01:22:09] So what do. Are you concerned about the like long term product prospects of this company? If that’s concerning to you, why don’t you ask about that? If it’s, if what’s important to you is career growth and a manager that’s going to be supportive of your promotion and you’re talking to the hiring manager. That’s key. Then I would ask about that. Tell me about a time when you, you tell me a story about how you help somebody grow from my level to the next level.

[01:22:32] Right. So that’s the story that you want to hear, understand what’s important to you. And it could be about the, the technology, although I think that those tend to be a little bit more on the junior side. If somebody’s asking me about like what kind of stack we use or how we overcome this particular technical problem, to me that feels a little bit more junior. Although if, you know, maybe in an AI context that might not be so like you could ask a question about, you know, model construction or evaluation or something like that.

[01:22:56] That could be really important for, for you to join the company. You could ask about the role. So oftentimes one that I really liked would be something like tell me about what makes this role really successful. Like someone in this role really successful. And you can get us a lot of signal on how to be successful in the, in the, in the company once you get there and showcase to the interviewer that you’re interested in in being successful by asking a question like that.

[01:23:18] I mentioned company ones and manager ones. So like I think those are the, your categories for, for choosing one some kind of insightful question. But it. I wouldn’t, I wouldn’t approach it as in like how can I impress this person with a deep question. I would approach it more like what’s important to me? And then that will come across in as a deep question to the person that you’re. You’re talking to.

[01:23:39] But I would avoid certain things that might be important to you. Some questions like tell me about the conversation of this role. Right. That’s not a conversation that you have with the painful interviewer or like how is the free food? Right. These things might be important to you but like this is not the time to talk about those things. I think it needs to be more related to, you know, something that the hiring manager also or the interviewer also cares about.

01:23:59 — Storytelling tips

Ryan:

[01:23:59] One last thing I think on behavioral interviews we did, we did a lot of, a lot of stuff on, on behavioral. Here is there’s this idea of storytelling that kind of unifies everything. If you can storytell well, you’re going to do well in your behavioral. You’re going to do well when you’re advocating for yourself actually on the job job. So I want to get your thoughts on how to tell stories. Well. What are the most common ways to get the most benefit with the least amount of time?

Austen:

[01:24:30] Yeah. So a story has to have an arc. Has to have some beginning, middle, end. It needs to give the person whatever the story listener is looking for. Maybe in some kind of novel. We’re looking for entertainment and we’re looking for something deeper meaning about. About life. In a behavioral interview, I’m looking to. To see if you are demonstrating past behaviors which are repeatable in that organization.

[01:24:53] I’m hiring you into which align with the signal areas. Right. So that’s the goal. So let’s just remind ourselves that is it have a sort of arc and it needs to deliver this, you know, this signal area. There’s A couple of different frameworks people have talked about in the past for shortcuts for how we can get to this. And the most common one is Star. Right. Situation, task, action, result. If you’ve looked up anything about behavioral interviews, you’ve definitely seen Star Method.

[01:25:13] If you. I will begin by saying I don’t love the Star method. However, if you have an interview tomorrow and you have a bunch of Star stories that you prepared, go for it. That’s great. It’ll give you that arc. It’ll, it’ll showcase those actions. It will bookend the actions you have with some kind of context to understand them and then some kind of impact or results so that they can understand why you did the things that you did.

[01:25:33] I prefer one called Carl, which is context, actions, results, learnings. I don’t think the difference between a situation and a task is like super relevant when you’re telling a story story. Oftentimes stories just have context like what’s going on in the business, what’s happening on the team, why I got this project in the first place. So I think trying to you as a preparer, trying to figure out like what’s the situation here?

[01:25:54] And then I’m moving to the task and what is the task on like a two year project. Right. Like many, many, many tasks really which are attached to the actions. Right. So I think that thinking in terms of just context and jumping into the actions is super relevant. And then I think for senior engineers, STAR doesn’t provide any kind of, of space for reflection or any kind of space for, you know, judgment or learnings.

[01:26:17] Right. And that’s, that’s why I like adding learnings to the end of every story. Especially for senior engineers. I’m looking to see whether or not you can look at your past experience, understand what made you successful or unsuccessful, and then apply that going forward. So this gives you a nice and easy mnemonic, right to, to remember to add those things to your stories. But I also think that when you’re telling stories, you need to be cons cognizant of where you are in the interview.

[01:26:40] And I think this is another, other differentiator between junior interview and junior candidates and senior candidates. A senior candidate is managing the time so they understand that hey, when I ask some kind of big, I call it trunk question, trunk, branch, leaf, trunk question, which is like talking about, you know, a project that was really ambiguous, then you’re giving, you’re being given a carte blanche by the, the interviewer to tell a longer story.

[01:27:03] So you tell A longer story. But then if I ask something about like, well, you know, who did you talk to in order to get this, you know, piece of information, then to, you know, to, to do some action as part of the story, then now I’m asking some kind of like, middle grounds, maybe. I probably have some thought about who I talk to and how I talk to them. It’s a communication signal, right? So I should give them something, but I shouldn’t give them like a complete Star or Carl story at that, at that level.

[01:27:25] And sometimes they just want like some very specific information like, oh, like tell me about the framework that you used for this particular project. Okay. They’re just kind of like checking off a box. Right. So I understand a little bit of context I didn’t have before. And so that’s a leaf question. Just give them the answer and move on. So I think your ability to detect what kind of question, how long the interviewer wants to listen to a response is going to be an indicator of seniority and also like an important skill for you, managing the interview for storytelling.

Ryan:

[01:27:55] What do you think is more important, what you say or how you say say it. So, for instance, is it the actual words that are coming out of my mouth that makes the biggest difference or is it my presence, how confident I am, and is there one versus the other that you’d say people should really focus on to really nail their stories?

Austen:

[01:28:16] Engineers tend to be pretty structured people, and engineering interviews tend to be very structured. Even at smaller places, someone will at least perceive that they are structuring the interview and some kind of rational way. And so I think it is less about how you say it and is more about what you say. So if I, even if you came across confident and positive and encouraging and like somebody I would love to work with, but if somehow I did not hear that you were handling ambiguity in some kind of structured way or that you were applying some kind of conflict resolution framework to the conflicts that you had, then I’m, I’m, I’m.

[01:28:50] That interview is not going to go well and I’m not going to be able to support hiring you. So I do think that, that it really does come down to substance. I think that’s the most important thing. You know, form follows function as in most engineering environments, and I think that it’s the case here as well. However, I would say at more senior levels, then how you come across and how you tell the story does make a lot more, does make more impact because we are expecting you to be telling stories like this one one, for example, when the VP says how’s that project going in the, in the, in the status meeting?

[01:29:24] You need to be able to tell like a. An engaging and entertaining. As much as entertaining as any work thing is entertaining story about how the project is going. And so I, I am looking to see if you are providing interesting details, for example, like a detail that might be showcase something that some thinking that you did or some unique situation that you were in. Oftentimes whether to include details is a big part of what you’re going through as a behavioral interviewer.

[01:29:51] So you should only include details which help to accomplish some tasks. So maybe the task is that I want to showcase, I’m deeply technical, I want to include a few details like that. Maybe the task right now for me as the candidate is to showcase that I am somebody who can work across teams. So I want to include the part about how the other tech lead had had a mean look on his face when I went to talk about this thing.

[01:30:14] Maybe that detail is relevant, right, because it showcases how I push through difficulty and pushback on other teams. So yes, I think those things are relevant. I think most candidates probably need to spend more of their time on basic story structure, basic identification of what actions they did that are repeatable.

01:30:31 — How he got promoted to Senior Manager (M2) at Meta

Ryan:

[01:30:31] You know, I wanted to ask you about your senior, your promotion to M2 at Meta. Actually, you’ve told me so much about how to speak about scope at this point, maybe you could tell me or tell the audience how you got promoted to M2, why you got promoted to M2 and speak about it in a way that illustrates the scope of a senior manager.

Austen:

[01:30:54] In order to get to M2, I had to solve some difficult problems in the recruiting space. And probably the most important one was around diversity. So I left a team which was focused on building things for teenagers, which is another hard problem that I got myself involved in. Very difficult time for Facebook during that time. But I left that team to lead a couple of teams within their internal recruiting products organization, which is all about supporting candidates, supporting recruiters and sourcers and their goals.

[01:31:23] But the thing that really drew me over there was working on supporting a team that was hiring more diverse engineers. That meant a lot to me. So how can I help make the world a better place? Place as in my position as a manager would be to make what is the most, the largest wealth opportunity in the world today, which was technology and is technology. How can I make that available to more people in the world?

[01:31:45] That’s really motivated me. So I. That is not an easy problem to solve. It’s not, definitely not an easy problem to solve from a perspective of a, a product team. Right. I’m not an education institution. Right. I, I don’t get to make all hiring decisions, for example, example. I don’t get to pick, you know, what people learn in school, so. But we were able to move the needle there by improving preparation of all things like interview preparation and that’s how we were able to help improve the diversity hiring for meta and that as well as supporting a number of other teams, helping to establish high quality of candidate data that would be established for Facebook’s applicant tracking system system and being able to, you know, clarify like what’s important in, in terms of candidate data and being able to improve the quality that we have significantly over time.

[01:32:34] I think that’s, those are the reasons why I was promoted to M2.

Ryan:

[01:32:38] So when I, when I think I. That story I just heard as if I was an interviewer. I heard that you were supporting multiple teams. I heard that you were taking on a, an ambiguous problem like how do you solve diversity? It’s not immediately obvious and it sounds like you had a significant self motivation to go towards this problem space and solve for the company. So am I hearing that those are the signals that you hope that an interviewer would have got if you were interviewing for a behavioral interview?

Austen:

[01:33:12] Yeah, for sure. I would say I would add a few other ones. So one thing that I didn’t mention is impact across the company. So. So I think leading the iOS and Android recruiting pipelines is an important way that you’re scaling yourself and applying yourself across more than, than just the teams that you’re directly responsible for. So that was also a big part. I should have included that.

01:33:32 — His biggest career regret

Ryan:

[01:33:32] I understand you left Big Tech at this point and you know, looking back on your experience in Big Tech, is there anything that you regret or anything that you wish you could have changed?

Austen:

[01:33:42] I don’t regret leaving. I think I had a really great time. I worked with a lot of wonderful people. I think that I regret the anxiety that I put into the teams and the anxiety that I put into myself. I think we can all look back on parts of our life, whether it’s school or early career or even now, and we think, wow, I really wish I hadn’t worried about that. At the end of our lives, we don’t say things like, I wish I’d just gotten a few more percent out of that engagement number and that project xyz.

[01:34:09] Like nobody thinks that way. Right? People value relationships. They Value, value. They value connection. And I, I think that I added a lot of anxiety to my life in various parts of my career and I, I regret that part. I think that like I told you earlier, I made conscious choices to optimize for my life experience versus my career. I think there were times which, when I was not honest with myself about that, when I was not honest about what that that trade off was, would mean in terms of compensation or in terms of career progression or in terms of status for myself.

[01:34:44] And I think being honest with yourself about what you really value and being okay with the trade offs in advance. And since I’ve left Big Tech and supporting my wife as she’s building her, her, her business, certainly that has come with like a decrease in pay. And one of the things, one of the ways that I have been able to apply the learning that I’m telling you is before I left, I knew, okay, I’m paying, cash flow is going to be low.

[01:35:07] I’m going to be okay with that because I’m making decisions to spend more time with my kids, because I’m making decisions to support more community organizations here and volunteering and I’m going to be okay with that choice. And that’s been really helpful. I’ve been able to go back to that choice a number of times when the cash flow situation has, you know, maybe wanted to improve that, maybe we missed that kind of paycheck and maybe you’ll experience this, I don’t know, you know, having, having just, having just left Big Tech.

[01:35:32] But I think having that honest conversation with myself has really helped in the last couple of years. And I wish I had done that earlier and said, you know, what I really value are these things and that means I’m not going to have those other things in life. And that’s okay, that’s okay if that’s what you want, right? So making that kind of conscious and values driven decision and then coming back to it, maybe sometimes revisiting it, maybe you want to change it, but that kind of intentionality is something I wish I had was more honest with myself about.

Ryan:

[01:35:58] So when you’re saying you were, you were not honest, you still thought you wanted that cash flow and those things.

Austen:

[01:36:04] I think I was frustrated. I think I was frustrated at various points about, about not being able to progress. And I think that there was, I would even say bitterness at times when I would say I’m frustrated that so and so got this job, so and so got promoted, you know, so and so was able to do XYZ right here, look at me. You know, I’m doing this, I’m doing that. But not being honest with myself of, of you know what, you, you didn’t do the things that they did.

[01:36:25] You did not move teams as rapidly as they did. You did not work as hard as that person. You prioritize other things in your life. You prioritized your family, you prioritized in making an impact in, in, in the community around you outside of work. And, and that has consequences, right? So I, I think we, we can live in this world where we think you can have it all, right? Like you can, you can be a, whatever level 7 IC and go home at 5 o’ clock clock, right, and have a wonderful and engaging family life.

[01:36:53] And maybe some people can do that. Right. I think that that could be a skill that people have, but that depends a lot on, on your, your talent level and what kind of skills you’ve built up over time. But the reality for most people is that if you want to accomplish something extraordinary in your life, you have to have extra, you have to take extraordinary steps. And I don’t think I was honest with myself and I think that that resulted in some frustration and bitterness along the way.

[01:37:16] But I think that once I decided, decided kind of in the middle what’s important to me, where I really want to be, what I want my kids to say about me in 10 years, that was what really what guided me, what do I want them to remember about me 10, 15 years from now.

Ryan:

[01:37:32] that intentionality got rid of those feelings for you.

Austen:

[01:37:36] Well, I think that it comes back sometimes. So especially living in a place like this where people are stacking wealth in and as fast as they possibly can and you’re going to see people like that around you all the time, living in Silicon Valley. And so you have to, to, to ask yourself like, is that what I want? No. Maybe it is, that’s fine, maybe it’s not. And then, so it’s going to come up and it comes up in my life, you know, on a, on a semi regular basis.

[01:38:00] But it’s really helpful to be aligned with your own values and have that kind of mission statement or vision that you have for your life and what’s important to you in the future. And if you live like that, you’ll be satisfied.

01:38:13 — The best advice he ever received

Ryan:

[01:38:13] You worked a long time at Meta across other companies as well. And I’m curious, what was the best advice that you ever received in your career?

Austen:

[01:38:22] I would say that the best advice that I ever Received was the importance of creating scope. So if you would like to get promoted or if you would like to advance in your career, oftentimes we are waiting like we are waiting for somebody to give us that opportunity. And I remember one of my managers said, like, if you want this, you know, M2 role, you have to create an M2 scope. Scope team, right? You have to, to accomplish and solve problems that are, you know, that are, that are of this scope.

[01:38:55] And so I think that this, this kind of honest reflection on like what it really takes to get to the next level that you have to change not just how, what you do, right? That’s often advice that you hear not just doing the same thing that you’re doing at your level better, you have to do something different. But what is that difference? And really that differences about creating additional scope and creating space for the, the organization to succeed?

[01:39:18] That was another key part of that advice was, you know, what is your VP saying about your project? And if your VP is not going to be talking to any of their peers or their manager about what you’re doing, then the question is, am I doing the right thing? So what can, what kind of, what kind of. How can I work backwards, right? And we do this a lot. We work backwards in the behavioral interview. What do I want to present kind of signal to area do to.

[01:39:38] I want present. You work backwards in a promotional experience. What do I want the packet to be like? And I think you can work backwards in terms of creating scope, which is what do I want the, you know, the VP or the organization’s leadership to say about what we’ve accomplished and then go and figure out a way to do that thing.

01:39:54 — Advice for younger self

Ryan:

[01:39:54] Yeah, that’s, that’s great advice. I mean there’s, there’s always situation and luck when it comes to career growth and promos. But I feel like the most satisfying promotions are the ones where, where you take initiative and you create the scope and no one can stop you in that case. So you don’t need anyone’s permission. And then the last question I’d ask is if you look back on your entire career and right when you graduated college and you could give yourself some advice now that you’ve learned what you’ve learned, what would you say?

Austen:

[01:40:28] I had a lot of imposter syndrome, so I avoided. For example, I got a PhD out of, in computer science, out of, out of undergraduate school. But during the summers I would work just in, in, in the research in the university when a lot of my peers would go and work in in industry they would go get a, like a internship right at Google or Facebook or wherever and work there. And really what held me back was insecurity.

[01:40:52] I didn’t want to go through the interview process. I, I didn’t want to be rejected. And so I, I think that kind of fear and allow the imposter syndrome that I felt to hold me back from those choices. I, I do think that that had a. My career was great. I love my career. It’s great. I’m not sure I would change a ton, but I think it did help me back. I think I was slower to understand how big organizations operate.

[01:41:12] I was slower to understand how large code bases operate, certainly coming from, from academia. So I think that that kind of insecurity and fear. Fear really held me back. I know that’s super common kind of advice to give oneself in the past is like don’t give into fear. But that’s the biggest advice I would tell myself is don’t give into fear.

Ryan:

[01:41:36] Awesome. Well, thank you so much Austin, for your time. I really appreciate it and I hope it’s helpful for people.

Austen:

[01:41:43] Yeah, thanks Ryan. Great to be here.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:56 - Chickenfoot

00:02:45 - Working at Google

00:06:34 - Overhauling Facebook’s build system

00:16:36 - Rewriting Facebook’s IDE

00:26:01 - Struggles after Principal Eng (E8) promo

00:28:39 - Building a virtual filesystem for Facebook

00:35:47 - Delayed Distinguished promo (E9) and learnings

00:39:56 - Joining OpenAI

00:43:05 - Research-led vs engineering-led cultures

00:44:53 - The story behind Codex

00:51:00 - How he uses Codex

00:57:00 - Why Codex’s harness is open source

00:59:50 - Top technical book recommendations

01:05:02 - Why deep technical skills are still valuable (for now)

01:11:07 - How to start projects well

01:14:27 - Advice on writing better and career planning

01:17:06 - Advice for his younger self

Transcript

00:00:56 — Chickenfoot

Ryan:

[00:00:56] I was looking deep into your website and there was something, most everything I could find more information about. There’s this one thing that you seem to be pretty excited about at the time, but I couldn’t find any information about it because all the links were dead. What is chicken foot?

Michael:

[00:01:11] Oh man. That’s strangely relevant because it’s. It was my, my master’s thesis pro project. It was a Firefox extension project, probably one of the very few that was written, you know, in JavaScript for Firefox as a thesis project. And it was, it was actually a little coding tool in a sidebar Firefox. And it was like a little repl. It was called end user. The idea was end user programming for the web.

[00:01:35] And so there were functions like enter and click and things like that. And you’d say enter and you, you know, have pass the string arg and it would like find the box and you’d say click search or whatever. And a lot of the work was all these heuristics that we built under the hood that if you said enter first name, finding the words first in name and finding the text box that was closest and then using that, and then through JavaScript using that as the input.

[00:02:03] And I just think about that now because it’s really funny. We did a lot of work and that’s a lot of what these agents are doing right now is similar, but now actually truly in natural language, not in this little JavaScript replacement.

Ryan:

[00:02:17] Oh, interesting. So it parsed the front end and had this repl where you could say, I don’t know, find the first name field.

Michael:

[00:02:23] And, you know, yeah, we’d use like, you know, the accessibility tags and alt. Alt text for images and we’d, you know. And it worked well. It was really good at Craigslist, actually, because that’s one of the simplest websites you could. You could use. But I had, I had friends who like, made like, automated tasks and like, made real money off of things they automated with this tool, which is really fun.

00:02:45 — Working at Google

Ryan:

[00:02:45] When you entered the industry, you were really excited about going to Google and specifically working on Google Calendar. So what drew you to Google and what was that like?

Michael:

[00:02:55] Yeah, I mean, you know, so I got online in the 90s, right? I remember, you know, browsing the web and it was at a time you’d go to like, you try five different search engines to like, hopefully find the thing that you wanted. And I. And it’s funny because I distinctly remember my roommate in March of 2000 saying, like, hey, there’s this site, this search engine that looks a little bit better. I think it was still Stanford Google Edu.

[00:03:18] And I was like, oh, this is better. And like, you know, and then you started reading and they, they were just so different, right? Yahoo. Was all like, very cluttered and like, they tried to be very sparse and were more, I think, principled about it at that time. And then, you know, like a lot of things, you start seeing people who, you know, who gets a job there, and you’re like, oh, man, they’re taking really good people.

[00:03:38] Like, I want to work with, with those people. And I felt like they just, you know, really got the web. Like, especially at that time when, like, Microsoft didn’t. Around that time, Microsoft killed the Internet Explorer project altogether. And I was like, this is the portal into the web and you guys are, you know, dismantling it. Whereas, you know, Google was way more, you know, web forward. And so between that and like, the quality of engineering and the.

[00:03:59] And the impact they were having, you know, certainly a place that I was really excited to go to, coming out of the school.

Ryan:

[00:04:05] And what was the culture like there at the time? Like, I think in some of your writing you talked about, like, products versus infrastructure.

Michael:

[00:04:13] You know, a lot of companies, especially ones that get big like that, you know, it’s kind of like whatever they’re good at first is I feel like the founders always have a. Have a soft spot for, right? So certainly information retrieval and infrastructure, right, were like, key to, you know, growing that. That company. And whereas I was drawn there at the time because, like, you know, they put out things like Gmail, Right.

[00:04:38] Which, which were vague. Right. But it didn’t, it still I think didn’t have the, the same cachet inside the company. Right. As the search and that sort of thing. And so you know we, when I was working on it, Calendar, it was still like mostly consumer. It was like kind of, it was also sold to enterprises. Right. But like we were not the ones making the money. Like we were probably a call center. Right.

[00:05:02] At that point in time.

Ryan:

[00:05:03] I saw eventually you ended up leaving Google and I, from your post it looks like it was pretty bittersweet. So what left you to leaving Google?

Michael:

[00:05:12] I had been there for four years and you know, I mean realistically like investing was a thing, like finances definitely changed after that four year period ended. But you know, also it was just tough like I, I guess a bad habit at that point of working really hard on things that like really were important to me but maybe weren’t important to Google. Right. So like I worked on Calendar, that was pretty important.

[00:05:38] And then I worked on Google Tasks, which was like a very small feature within Calendar. Right. Like probably at least two to three orders of magnitude fewer users. Right. But I was really passionate about it. And then I was really passionate about the like the JavaScript infra and the, the closure, that suite of JavaScript tools. And you know, it’s great. I mean I enjoyed and I’m proud of all the effort that I put into the things, you know, that I went and wrote the book on Closure because I was so motivated but career wise, maybe not the best move, right?

[00:06:11] You’re like, but I’m doing all this high quality stuff and you see other people getting recognized. You’re like, but I’m working so hard, you know, like that’s, you know, kind of like the harder, not smarter type of mistake. And so, you know, it’s kind of like I should go somewhere or try something else and see if like the things that I’m excited about or the things that like the company I’m working at is most excited about.

00:06:34 — Overhauling Facebook’s build system

Ryan:

[00:06:34] Later you came back into big tech and you started working at Facebook. I understand that you were kind of like a JavaScript expert at the time. And then one of your first big projects at Meta was kind of build tooling in the Android code base. So I want to know the story behind how you got involved in that.

Michael:

[00:06:53] So at the time it was like, Facebook’s going to make a phone, right? Actually there were some failed projects, but it was like, this time it’s really going to happen. We’re going to partner with htc, we’re going to fork Android and do some stuff. So that seemed super exciting, right? As a person just coming into the company and I’d done quite a bit of Java. I was more JavaScript, but at this point, this is also where they called it Face Web, the version of.

[00:07:28] They put HTML5, Facebook on the phone, and that was clearly not working. And it was clear that mobile was going to be the future that was make or break for the company. And suddenly, you know, a friend was like, hey, like, I know you really like JavaScript, but, like, you should really pick Java or Objective C and, like, get good at this or you want to be a product person. And that was really, really great advice.

[00:07:56] And so I was like, well, I like Java, I don’t like Objective C, so, like, let’s go. And so that’s how I found myself on that project. And I, you know, we. We had a very short timeline because unlike almost every other project, there was a hard deadline. Usually, you know, you ship when it’s ready. This was like, nope, we gotta send a bill to htc because they’re gonna, like, burn it into phones on, like, March 1st or whatever it was.

[00:08:26] So, you know, it was really a scramble. But. And the original Android code base was. Or a bunch of. It was also, like, inherited from a contractor that Google paid. Like, it’s like Facebook didn’t want to make a native app and they paid some guy. And then it was like the app in the store and they’re like, here, the code’s yours now. And we should have thrown it away, but we didn’t. And so a lot of things that were frustrating, but also, like, iteration time, right?

[00:08:54] I think for everybody, if you’d been a web developer for such a long time, you’re used to this edit, refresh, and then, you know, like, the Android build system was. Was rough, right? It was this. Some build tools and Ant and it. Like, there were no. There was no way to modularize it out of the bot. Like, we had to hack it up just to even have, like, four modules or five modules or something like that.

[00:09:17] And it was so painful to get anything done that I was like, I need to. I need to fix the build system. Like, I was like, I know I’ve done a lot with Java. I know it’s not like, fundamentally this slow to, like, iteratively, you know, build this sort of thing. And so, you know, Facebook has this hackathon culture. So I was like, all right, hackathon, totally gonna, like, make a new build system. You Know, like unceremoniously in the style of Google’s build system.

[00:09:48] And there’s actually another build system called FB Build that was already a different mirror of, of the Google build system as well. That one’s written in Python, only did C, but yeah, and I was like, either I’m going to make this work or. And like, and then I’m going to be happy to be a developer or like, I don’t even know if I’m going to make it here because it’s just going to be like tearing my hair out, you know, working on this thing.

Ryan:

[00:10:17] If you didn’t fix it you would have quit the company?

Michael:

[00:10:18] Or I’ll at least switch projects or I need to find a way to find a happy place, right, and come into work every day. Like, I came, I want to write code, I want to do work, I just want to, you know, try to do my best work. And so it was funny, actually. I give people credit. A lot of people told me what I was doing was a very bad idea. Almost everybody, except one person.

[00:10:44] I was a senior Android engineer, but nobody said no, you know, whereas I felt like at Google, like, people said no more. And so, so I just rolled with it and, you know, it quickly, relatively quickly, had something that was, you know, dramatically better, like at least like twice as fast or something like that. And so that, you know, I turned a lot of heads and people were like, okay, you know, I guess we, yeah, we’ll go with that one, right?

Ryan:

[00:11:13] Like, I think the interesting thing to me is it seemed like all the odds were stacking. A lot of people who are an engineer notice the same problem, would not have chosen to start the project because there’s an existing one. And then also Google’s got some competing one. Maybe you’re not going to beat that one. What gave you the conviction that your project could beat the other ones and become the default?

Michael:

[00:11:39] Yeah, I mean, I guess a couple things. One, like I said, I’d done other Java stuff. I was like, this shouldn’t be that slow. Or just as a software engineer, this fundamentally should not be as slow as it is. And actually most of the pushback was of the nature. Like, well, if we deviate from the standard thing, we’re not going to be on the standard thing. And what if it gets like 100x better next week and now we’re stuck on your thing?

[00:12:07] Which is really funny when you think about it, because there’s so many things like making their own PHP virtual machine and language. They’ve certainly embraced doing their own thing many times. Why this one was different, I don’t know. I mean it was, it was just, I guess everything about mobile was like, how are we going to make this work? I guess there’s just like a lot of fear and you know, again, like timelines.

[00:12:29] It’s like we have the senior person, like, why it seems sounds like they’re going to go off and do a science project but we have a hard deadline. Like, is that the best use of our, of our time? But, you know, it worked out. It worked out. I will say another thing too was I also tried to couch it a little bit. I was like, hey, I’m just trying to make an Android build system. I’m trying to take over the company.

[00:12:52] I’m not trying to change anyone else’s project because I certainly felt that that was going to cause more, I was going to invite more friction. And so I knew that I tried to make it in a way that it could support more of the company, but I never pushed it. And so I was really heartened when the company, year or so later, the iOS people were like, hey, our build system sucks. You know, can we use Buck?

[00:13:19] I was like, yeah, sure. Like I don’t, you know, come on board.

Ryan:

[00:13:22] Yeah, that, that’s also another interesting because you, you came in and you didn’t have a whole lot of credibility. You were just hired.

[00:13:29] And then you’re trying to build something and get every. And also everyone’s saying, don’t do this. And then you have to convince them, hey, this is the right direction. How’d you influence that change without any credibility?

Michael:

[00:13:44] I mean, I borrowed some. There’s one senior Android engineer who was also ex Google, John Perlow. And he was like, I think he said, right. He’s like, you’re going to do it, just do it fast before anyone gets cancers you or something. He’s like, well, I’ll support you if you get this done. So he was one of the first people and he was a very prolific coder. So he was genuinely very happy to have a faster dev cycle.

[00:13:59] So he was one of the first people to give it kudos and I think that that helped a lot also. But I will say I made some big mistakes on my way in there. You mentioned about having no credibility is like, I came from Google and I was like, oh, this is where the X Bell Labs people are. All these great people. And then you go to Facebook and you’re like, oh, it’s a bunch of college kids. How could they possibly know what they’re doing?

[00:14:34] And a few times certainly I made the mistake. Well, at Google we did it this way and people were like, we really don’t care. And they were right in most cases. Right. That just because it worked there didn’t mean it was going to work here.

Ryan:

[00:14:51] One last thing on this topic is, I mean, what you built was way more performant than anything else by many multiples. What’s the intuition like the technical intuition behind what was done and what you did that made it so much more efficient?

Michael:

[00:15:08] I think the, the big thing was is that I just sat down and looked at the, the tool from Google and I was like, what is it actually doing right? And where. And I think the big thing is that the Google one would kind of just, if you changed anything, it would just start over from scratch. And so that’s why it was so slow, especially for, for incremental builds. And so then I started really understanding like, okay, but what depends on what, because there is these still Archer, like Android resources, which had a very bespoke thing and it was complicated.

[00:15:41] And because it was somewhat complicated, I think that’s probably why just by default I blew everything away and started over. But once I got in there and I was like, oh, okay, we can take advantage of it, if these things don’t change, then we can cache this result from this step and we don’t have to redo it. And suddenly that made things quite a bit faster. And then also just even supporting the idea that you could have more than the four modules that we had.

[00:16:07] I mean, I think every time someone added a module, they had to add like 200 lines of XML of ANT build script that nobody really understood. And so it was not. So no one wanted to modularize anything. Right. You didn’t want to be responsible for that. So a big thing with Buck was making it that it was like a lot simpler to add a new module. And so then that also meant we ended up with more modules and then builds are more incremental as a result.

[00:16:30] Right. So it’s really a change in the mindset.

Ryan:

[00:16:33] So less redundant work, basically.

Michael:

[00:16:35] Yeah.

00:16:36 — Rewriting Facebook’s IDE

Ryan:

[00:16:36] After you solve this build problems, I guess, in Android builds and obviously I went later to further parts of the company, I saw that you started to work more on like the ide. What was the problem that you saw that you saw in ides that made you want to get into that?

Michael:

[00:16:53] So I, yeah, I took, I did a brief stint after buck on messenger, iOS messenger. So I was like, okay, I’ve done Android. Maybe I should actually branch out. Even though I don’t like it and I still didn’t like it. People don’t even realize in Objective C there’s a thing that happens for a long time now called arp, like the automatic reference counting. Nowadays the compiler injects it, but you used to have to actually add code in Objective C to do every reference count.

[00:17:27] Like memory allocation. Yeah, or like every reference that you created. Like, you would have to do it. And, and the. And most people have never seen this type of code, but the iOS messenger code that came in through acquisition was so old that it was still written in this other way. And it’s, It’s. It’s incredibly painful. I guess now you’d have like, Codex cleaned it up or something like that. But at the time we just sucked it up and just Xcode just didn’t feel right to me.

[00:17:52] I didn’t like header files and implementation files. I still don’t. And also the. I would say for Both Android and iOS, you know, Facebook always had like, the biggest app, right? We had like one app with every. With every feature in it, and we, you know, ship it as opposed to Google. Like, they’d have like a drive app and a, you know, a sheets app or whatever, but they also owned one of the platforms, so they could put 20 apps by default on there.

[00:18:21] And so what that meant was, is that Facebook was always hitting the scaling limits of every mobile developer tool before everybody else, which was painful. But actually, as a DevTools person, it was kind of interesting because we got to solve problems that nobody had solved before, and not just as a science project, but because it was real business value. And so Xcode, similarly, we talked to Apple.

[00:18:46] We’re like, hey, Xcode’s not really scaling to our project. They’re like, your project’s too big. You should make it smaller. That was kind of the feedback that we got from them. And so it seemed justifiable to like, go and build an editor. It was similar thing. It was like, I was like, what is, you know, what is an IDE doing, right? It’s like it’s talking to Clang, the language server and that sort of stuff.

[00:19:06] I was like, we could build like a nicer shell on top of that, you know? And then we had started, by that point, deviating from Git and switching to Mercurial as a company. I was like, no one’s going to support Mercurial out of the box, right? And so like that should happen. And then like, oh, actually if Buck is going to be our build system, then like we really like, you know, like that’s Xcode’s never going to do all of these bespoke Facebook things.

[00:19:28] Right. So it seemed justifiable to like invest the time to, to try to like improve that experience. And I didn’t feel that way About Android because IntelliJ was actually like quite good and we had figured out how to make that work at scale. But, but Xcode was a little more difficult at that point.

Ryan:

[00:19:45] So there’s Xcode which was bad, didn’t fit the existing needs. And then there was another IDE that another team was building. I think it was web based. If I.

Michael:

[00:19:56] It was, it was web based. I’m not laughing at that part. That part’s fine. It was, but it was built off of an abandoned Google open source project written in gwt, which is Google Web Toolkit where you would write Java and it would code gen to JavaScript for everything. And I tried to build on what they had. I tried to build some credibility, even actually sped up their build a bit. But again, kind of like looking at the iteration times and also I was like, this is an abandoned open source project, it’s written in Google Web Toolkit.

[00:20:30] And I was like, we are the react company at that point, right? Like how. Why would we not empower people who want to build dev tools to use to build on a, you know, technologies that we are the leader in and actually really like, because, you know, we think we’re actually really good. So I was like, you guys are crazy. And so I started. And it’s similar to the thing where I, you know, I mentioned with the buck.

[00:20:58] I was like, I started it as the, the Java build tool, not the everything build tool. Right. I was like, it was a similar thing. I was like, hey, I’m going to go over here and start this other editor. But we’re just, we’re just going to focus on iOS like you. I’m not trying to take things over.

Ryan:

[00:21:12] Yeah, you didn’t want all the friction. And I mean that team, they had all the existing users though, right?

Michael:

[00:21:18] They had thousands of engineers, maybe a thousand.

Ryan:

[00:21:22] I think eventually leadership sided with what eventually became nucleide, which is what you, what you built. But why did they side with you when you didn’t have any users on your shared profit?

Michael:

[00:21:35] Yeah, I mean, I think it was a combination of two things. I mean I think the arguments that I made in favor of like the technology stack to build on another Was that new cloud was a desktop application. And part of it was, if this is actually going to be an iOS Xcode replacement, like, people are going to want to use it, needs to be able to talk to the simulator or plug in the phone or whatever. Like, in theory, we could go through the web and do all those things, but it just seems like a lot more.

[00:22:04] A lot of extra work. And. And I think the other part is that, you know, I had built up some credibility with the. The Buck thing that they’re willing to, like, take a gamble. Like, well, the last thing, you know, worked out, like, let me try this one.

Ryan:

[00:22:22] I saw that this, in combination with some of your other work, led to your promotion to E8, which is also known as principal in the industry. What was your reaction to that at the time?

Michael:

[00:22:33] Yeah, I mean, certainly I was. I was very excited. You know, I felt like, you know, in the ways that I had been out of step, I felt like, you know, when I was at Google, I was like, this was also. Not just. It was also validation that, like, oh, now I’m also growing, not just, like, technically, but, like, understanding, like, what it means to kind of, you know, do the things that, like, are in line with your employer.

[00:22:57] And so, like, that also just was equally valuable or satisfying.

Ryan:

[00:23:01] I know that Nucleide was open source, and I think Buck was as well, if I recall correctly. What’s the rationale for open sourcing?

Michael:

[00:23:14] Yeah, I mean, let’s see. Buck is the more interesting one. Nucleud kind of didn’t really get adopted externally, I think in both cases. I think, you know, all these companies have benefited so much from open source. Like, I think there’s just a feeling of, like, if it’s not really the secret sauce. Like, but let’s share this with other people, right? Like, I mean, like, Codex as well and a number other things in my career.

[00:23:39] So I do feel like that sharing of information is just good, even if no one uses your tool. Just as, like, seeing as a reference of, like, how a thing could be done could be great, you know, in the better scenarios, you actually get, you know, meaningful contributions back and things like that. And, you know, I remember, like, I think Uber adopted Buck. I think Airbnb did. Like, it was funny, right?

[00:24:05] Like I said, like, Facebook was the biggest app, so we would hit all these problems first, and then this next wave of companies would start hitting these things and be like, let’s check out what These guys did. I think, I mean honestly we also, you know, at Google it was, internally it’s Blaze, externally it’s Basil. So we were open source first and we always, we always did kind of wonder like if we put some pressure on.

[00:24:29] Ultimately they did. I think you’ve gotten a little bit of credit from that unofficially from some of the folks who worked on that. But I think also that it’s also a recruiting thing too or just showing like, hey, if you want to being at the leading edge in whatever area this technology is and you want to do this all the time, not just as a drive by contributor, this is what we do, this is who we are.

Ryan:

[00:24:54] And then this decision to open source, is this, was this a bottoms up decision where engineers said we’re just going to do this or is this kind of also leadership buy in? Hey, this is going to be valuable. And you got like a doc.

Michael:

[00:25:07] Yeah, yeah, I think, I think both cases it was certainly like bottoms up. I don’t think so. You know, things like React and Pytorch, like those are like the, the big success stories right? Where, where the value back to the company is like unquestionable. And then there’s this like longer tale of things where I think depending on the economy and other things, the managers get grumbly if they feel like their engineers are like doing too much open source or things like that.

[00:25:37] So it’s almost almost always I feel like, like bottoms up. I think. Yeah, I would say that’s, that’s often the case, but it wasn’t like met with a lot of resistance, you know, and you usually get like a good conference talk or two and a nice blog post. And those blog posts like, you know, do I would say pay dividends over time? Right. Like trying recruiting? Yeah, like they have like a longer shelf life than I think people realize.

00:26:01 — Struggles after Principal Eng (E8) promo

Ryan:

[00:26:01] Okay, so you got your E8 promo at this point and I imagine the expectations maybe in your mind are kind of going up a little bit and now you need to go and find an E8 problem, I guess. So what’d you do after you got promoted?

Michael:

[00:26:17] I think that’s where I did a. I got a little over my skis and I tried to help with web speed, I think is what I did because again it was a thing that was like a really big problem for the like, like the, just the load time of facebook. com was, was not in great shape. You know, the architecture was a bit stale and it was just, the problem was so big and I didn’t have, I think the A lot of people who had worked on web at Facebook had worked on it for a long time.

[00:26:52] And I had put myself in mobile and dev tools. I actually wasn’t in that world. And I remember I was like, what do we do? I mean, actually I sat down with another person. We started compiling V8 from Source and trying to see if we could change the way that we generated JavaScript, if it was going to be friendlier to V8. We were just trying to whack things. None of which panned out, by the way. And I just.

[00:27:16] It was, it was. There’s different. There’s different people who are, like, good for different projects, and I think that was just not the one. Like, I’m better in a project that involves writing a lot of code from the beginning. And I think a project like that was like, more about, like, looking at data and talking to a lot of people. And, like, I just. That’s not my strength.

Ryan:

[00:27:36] I think you mentioned that at this point in your career, the, the idea of a hero quest. Yeah, what’s. What’s that mean?

Michael:

[00:27:39] The idea that, you know, that it’s like, you know, it’s embarrassing because there’s like some. About ego. This idea that, you know, there’s this like, you know, Gordian not. And all these engineers are like, if only someone would come in and, like, solve this, you know, engineering problem. I was like, I know JavaScript, right, okay, just come in that way and.

[00:28:05] But I didn’t. I. I definitely did not. You know, and I think that it’s been an important learning and I’ve had to relearn it, like, at least one other time that, you know, I can do a lot of things, but there’s a smaller subset of things that I genuinely enjoy doing and that I’m going to be a lot more successful. Right. If I stay in the things that I genuinely enjoy doing. You know, I try to expand that over time, but, you know, we don’t all have to be the best at everything.

[00:28:35] Right. I think that’s accepted and, you know, embrace it.

00:28:39 — Building a virtual filesystem for Facebook

Ryan:

[00:28:39] So then how’d you find the E8 scope problem after that? Like, what was the next day?

Michael:

[00:28:44] That was, I think, a bit of luck as well, I guess. You know, we had. We had. Sometimes we’d have these little summits with, like, smaller groups of engineers about, like, brainstorming what’s going to bite us in the future. I mentioned, I was like, you know, the repo keeps growing, they’re going to have some scaling problem at some point, and a person or my manager Then he became my manager, Brian o’.

[00:29:09] Sullivan. He decided to get some people together to work on making a virtual file system to try to get ahead of that problem. And so we got myself and Adam Simpkins and Wes Furlong, both tremendous engineers, actually thought I was the worst engineer on the project for quite a bit.

Ryan:

[00:29:30] You mentioned a virtual file system on a high level. What’s the benefit to a company like Meta for if you had something like that?

Michael:

[00:29:39] If you have bought into the Monorepo philosophy, right? Put all the code in one repo. Most things that people do need only a subset of that repo. They have to look at the files at any point in time. The idea is that you design all your tooling around this virtual file system so that when you say clone the repo or then you update to a different commit or anything like that, you don’t actually have to like write out every single file in the repo on disk when you make that change, which is.

[00:30:12] Which is the default in your traditional file system, because that is going to grow proportionally to the time with the size of the repo, Right. So at some point you’re going to be very sad. And so there’s actually kind of two, at least two parts to it. One is building this virtual file system that’s like, hey, I know the users at this commit. If the operating system asks me for the contents of the file, I can go get it and it will appear like they had actually laid out all the files.

[00:30:43] The other part of it, which is the part that maybe I was actually better at, I think is then trying to anticipate, well, we have all these tools in our tool chain that are used to just reading all the files. You know, you just rip grep. It just reads everything, right? That sort of thing. And how do we start changing also our development flow and our tools such that they are designed with the virtual file system in mind, because if you just, you know, materialize all the files with your tool now, you’ve like actually just lost all your.

[00:31:13] The benefits that you’ve made.

Ryan:

[00:31:15] So on a high level, it’s basically just lazy loading a huge file system. So it’s more efficient because you don’t need to do everything at the beginning.

Michael:

[00:31:23] Yeah.

Ryan:

[00:31:24] And so that part of this project that you said you’re better at is. Is integrating everything into this on top of that primitive.

Michael:

[00:31:31] Yeah, yeah. So one thing I did actually with Hanson Wang, who’s here actually, I’m on the Codex team with me now, was I was like, oh, okay. You know, the traditional way you have, you know, you want really fast file search in your, in your ide and your editor. And you know, the way most of these things do is the first thing you do is they walk the entire file system to find out what the files are.

[00:31:52] And I was like, well, that’s going to be a serious problem, right? It’s going to undo all the benefits. And then it’s one of these things is like not, you know, first it was like, okay, how can we implement file search? That’s not going to undo all the benefits? And then it was, how can we do it even better than how it’s done today? And so we ended up building this file system called miles short for my files.

[00:32:18] And on a cron job, it would index, it would ingest all the new commits that had come in on trunk and keep track of which files had been added or removed. So just the names of the files, not the contents because all you need is the names. And then Hanson had some clever ideas about how we maintained that index such that we could support fuzzy file matching. So instead of just substring matching, you type, you know, like if you have like a camel case thing, you want to be able to type just maybe the uppercase letters or just your spelling’s bad or what have you.

[00:32:55] And so it came up with this really interesting way to represent, you know, kind of all the files that had been seen at some point and then some bits to represent, you know, if I’m at this commit, was this file present or not present at that point in time? And then also when you ask, I mean, you sent a query to this thing, so you’d send like what commit you’re at. And like if you’ve added any files locally or removed any locally.

[00:33:22] And I think we got this, you know, like, you know, over a million files in like, I don’t know, like 10 to 20 milliseconds or something like that, so that you would be in your editor, you know, typing. And so that was way faster than I think than, you know, like what xcode or like MDS code or something like would give you out of the box. And so that was, you know, so like it was solving a problem for Eden.

[00:33:46] That was the name of the file system originally, the virtual file system, you know, in anticipation, before it was even ready. But then it was so fast and it was available as a thrift service internally. Like people started using it for all sorts of other things. I think when I left, like, I don’t know, I think there were at least like 30 servers running miles like spread around the globe. So clearly that was more than just people like personally searching for files if they needed that much capacity.

Ryan:

[00:34:13] Yeah. I mean when you talked about, I guess the implementation details, most, most people, they, they don’t use leetcode on the job. But that sounds like. I was thinking about the. I don’t, I don’t even know how you input that. Is it like a tree, like a try thing?

Michael:

[00:34:30] So that’s what’s funny that try. Yeah, this was, this was pretty cool in that we had kind of like two parallel arrays. One was like the file contents and one was maybe an integer into that index. I forget. And then we had a 64 bit mask that was like, it was like. So you had like 26 lowercase, 26 capital 10 digits and like maybe dash and whatever. And every bit was set if that character was used at all in the file that you were searching.

[00:35:11] And so the first thing was we could blow through that list and exclude a bunch of things right off the top. But we also very designed. All these arrays were in parallel to each other so that for cache wise we knew it’d be very efficient for the CPU to read memory linearly and then it led itself to. You could just partition that array, it lend itself to parallelism.

00:35:47 — Delayed Distinguished promo (E9) and learnings

Ryan:

[00:35:47] I understand that you worked on Eden and then obviously Myles as well. And then these eventually led to another promotion but prior to that promotion there was some learnings you might have had about influence and conflict?

Michael:

[00:36:05] Yeah, I mean that’s one of the things I guess about being an E8 who primarily writes code. Right. There’s other people. I’d say actually the majority of people that level or higher are not writing code. Right. They’re, they’re actually kind of exclusively spent doing more like influence or working across teams, you know, writing the big Google Doc and getting everyone on board and all that sort of stuff.

[00:36:33] And so you know, as an E8 trying to have that level like, you know, that level of impact that you’re expected to have. It’s like, oh, it’s hard to do that just writing code. So it’s like I need to spend at least some time probably influencing other people. That would probably be good for me. That would probably make my manager Happy. And I think, you know, sometimes you’re just so confident in some insights you have.

[00:37:07] That or certainly I was, that I would just, I, I just came down like way too hard, I would say. And, and that did not go well for me. And yeah, so like, like promo got delayed because I was very excited or anxious about when Microsoft acquired GitHub. And oh yeah, because New Clyde was built on this, you know, primarily GitHub technology. And I was like, that’s going to go away because VS code is going to not make them be a project anymore.

[00:37:40] Which was true. Which did happen. And I was just so anxious that, that this was like now a risk. Right. And I was pushing people, but you know, people were, I didn’t really account for the fact that people were happy with what they were doing at the moment and like, didn’t want their cheese moved, you know, like right out from under them. And so, you know, I got like a little bit of a talking to.

[00:38:06] I, you know, I had to wait a little bit for promo and things like that. But I, you know, I actually even got some coaching. I think after that point, I can’t remember the exact chronology to, to work on that. Like, specifically.

Ryan:

[00:38:18] What’s the number one thing you learned about Coach?

Michael:

[00:38:21] I would say for me is like, I think I’m more aware of like things that trigger me, like technical decisions or what have you and recognize when that’s happening and be like, okay, let’s not act in that moment. Or you know, if I, if I don’t think I can have the conversation like, or I’m not the, in the best place to have it, like, maybe I go talk to the person’s manager instead rather than like, you know, like bull and china shop to the engineer and be like, so I have this thing, I’m thinking about it this way.

[00:38:52] I’m kind of riled up about it. Like, help me. Like, how can I, you know, work with your team or whatever? Like that’s happened to you.

Ryan:

[00:38:59] Well, it’s interesting to me in your career because the promo got postponed and you were, you were seeing that VS code was going to come up and the thing that New Clyde was built on was going to go down. And you were actually right in hindsight too, like, in the future that is what happened and you were battling for what was right, yet you were told, calm down a little bit and this. So I mean, yeah, what are your thoughts when you saw things play out and you go, actually, I was right the whole time.

Michael:

[00:39:33] I did have some conversations, like about A year later and I was like, hey, can we balance the ledger a little bit? Like it ding. Could be pretty hard for that thing. And it was kind of of good and we, we worked it out, I would say.

Ryan:

[00:39:46] Okay, so I guess the, the learning is just how you went about it, not what you were like.

Michael:

[00:39:52] Yeah, yeah, I would say that’s true.

00:39:56 — Joining OpenAI

Ryan:

[00:39:56] You seem to have a great time at Meta and eventually you left. So what was the thing that drew you to OpenAI?

Michael:

[00:40:03] Yeah, I mean a number of things. I mean one was so I, I interviewed at the end of 2023 with OpenAI I believe. And, and I had SPEN Meta trying to do LLM based developer tools, right with there’s a, you know, we had our own license version, even metamate. There’s like a little version of GitHub Copilot, right? That one did like a paper and a talk on that one. It was a code compose, right? That was code compose, yeah.

[00:40:31] And you know, we like now there’s a lot of enthusiasm around, you know, delivering quickly and pushing the boundaries of that sort of stuff. And, and you know, I mean truthfully, I mean, you know, you get feedback on some of these things. It’s like why is this not, you know, GPT4? And I was like, well we’re on like Lamba 2. It’s not the same thing. And you know, I was not, not a researcher, right?

[00:40:59] I was like, but I, and I, you know, just wanted to, to build the experiences, right? And that sort of thing. And, and so, so that was, that was one thing was that like I wanted to build stuff and I wanted to go to the place where I could actually build with the best model, you know. Two again was similar to like, you know, Google originally like seeing the people who were coming here to OpenAI. And I was like, well those are people I really respect.

[00:41:24] I was like those are. I could actually work with more senior people, you know, like Meta eights and nines at OpenAI felt like that I could where I was and you know, third was just like this also seems like. And it has been just a very special place at, at the point in time. So I told a lot of people, I was like, you know, I felt like this would be like the most similar to starting at Google in 2000.

[00:41:53] So not like, not to 1998 but like let’s say 2000, right. Like they kind of got some footing and got some you know, product market fit. And so like just as a personal level that was just a very exciting thing and you know the last one was that funny thing is that when I really enjoyed Calendar because it was consumer and I shipped to a lot of people, I went to Facebook because I thought huge consumer place.

[00:42:18] Ended up not doing consumer at all. Right. Ended up doing developer tools. But like, you know, my users were my friends at work. It was just like 20,000 people, not like a billion people, but it was good enough for me at the time. And then, you know, thinking about OpenAI and then the chance to actually, you know, come back to consumer or at least like have a large user base. Right. And so now working on Codex, you know, I wonder if over a million weeklies or I forget how much it is right now and just keeps growing like, like, you know, hockey stick.

[00:42:50] More like vertical line than hockey stick, even that, you know. So it’s like way more than the 20 to 40,000 developers, you know, it could affect it at Meta.

Ryan:

[00:43:00] Yeah, absolutely. And dev tooling for the industry almost.

Michael:

[00:43:04] Yeah.

00:43:05 — Research-led vs engineering-led cultures

Ryan:

[00:43:05] Meta, when I think of the company, it’s kind of engineering driven. Like engineers are king and queen, I guess. And it kind of like drive everything. It’s very bottoms up from that sense. And I feel like a lot of the lab companies are also like that, but on the research side. So rather than engineer is the first a class citizen, it’s like let’s make sure the research goes well and for good reason.

Michael:

[00:43:29] Right.

Ryan:

[00:43:30] Like that’s also part of why you came is the models are good. But as an engineer or you mentioned, you know, you weren’t doing research. What are your thoughts on that? Research led culture versus engineering led culture.

Michael:

[00:43:41] I mean it was certainly an adjustment, right? I’m not, I mean I think of anyone who comes here and says otherwise is a liar. But you know, if you’ve been at like the Fangs or whatever and. But you know, they are, you know, you talk about impact, right. Which I think is important. Like a real. Like you genuinely, you know, mean it. Like I, you know, I love the work that I do on Codex and the harness and I think we do a lot of very meaningful things.

[00:44:07] But you know, if the model weren’t very good, it wouldn’t really matter what we did, you know, on the harness. Right. And so, so I don’t. So you know, that that’s how it is, I would say. But I feel really great. We sit right next to the research team, we’re really close with them and so that relationship that we have that we get to co develop the thing, that was another reason leaving Meta in the LLM space was like I want to build the product with the people who are building the model so that we can do this thing together.

[00:44:46] Maybe you could do that there, sort of, but not, but anyway, the impact was not quite the same.

00:44:53 — The story behind Codex

Ryan:

[00:44:53] So you mentioned, I mean, when you got here, it sounds like you were working on Codex and starting that project up. So I understand with the initial launch of Codex cli, it was not exactly what you hoped for in terms of how it was received, but later it kind of really all came together. Can you tell that story?

Michael:

[00:45:13] Yeah, sure. It’s been a wild ride. So codec CLI, we launched it in April 2025. It was kind of like this, like one more thing moment at the end of the 04 mini live stream. And so we demoed it live, we open sourced it, you know, Core3Pro at that point, a lot of people tried it out. You know, everyone was excited to try a new coding agent, you know, and it was, and it was pretty good, but it was, it was pretty rushed to get it out the door right there was.

[00:45:44] Which in some ways was good for engagement because now we’re open source. We were getting pull requests, you know, coming in all over the place. And I forget it, you know, we were, I feel like we’re like 10 to 20,000 stars in like a week or two or something like that. So like that, that part was fun. And you know, there were people who did like it. You know, it’s not like nobody liked it or anything like that.

[00:46:03] But then, but that I, I think we didn’t, we weren’t maybe quite staffed to like really drive that the way that we needed to because, you know, we’re trying to cover multiple things as a company. And so, you know, just a month after that and you know, team of like seven engineers and I forget how many researchers launched Codex Web or cloud web where, you know, you’d use Codex in a container or you could just, you know, kick off a new thing from your phone, which is super cool.

[00:46:33] And that was like a more well staffed effort. And I think, you know, that one is still the, I think the long term vision is correct, but I think with a lot of this stuff that we’ve seen, you have to bring the users with you. And I think that was just a little bit ahead of its time and people were still actually more into the local coding agents. And so we saw the web product. There’s a big adoption.

[00:47:03] Initially it wasn’t as sticky, I would say, as we had hoped. And then through the summer, you know, both products still Kept, kept, you know, we’re working on both. But then somewhere that mid summer, again like I said, it’s like, you know, this moment like local agents are still the, I guess the stronger product market fit, but again I still think in the limit personally. Right. Like you’re going to need more machines than just your laptop as a place for agents to run.

[00:47:32] And so we shifted quite a bit over the summer so we brought more people onto the codec. Cli. Yeah, a few kid things bring more people on. You know, GPT5 was going to come out and that was looking really, really good. And I think I was personally excited about, because I prototyped it a couple times before was that in addition to the cli, then we also had enough staffing. We also started working on the VS code extension and I felt actually very strongly that the terminal is good for a lot of things, but it has a lot of limitations.

[00:48:10] Always you make a lot of compromises to make a nice UI in the terminal, whereas VS code we didn’t have to make quite as many compromises. August I think was a crazy month. I think GPT5 came out. I think we released our new refresh terminal ui also the GPT open source model came out. We supported that in the TUI as well. So that was really cool that you had an open weights model and an open harness.

[00:48:38] And then later that month the VS code extension came out and so we were just like shipping like crazy. And that’s really where that confluence of things I would say is where we started to see the inflection point that’s brought us to like, you know, the vertical growth that we’re, that we’re at today. And so that’s, you know, that’s been really exciting. I mean, you know, some of these things you can, you can go in the repo check for yourself.

[00:49:05] Right. And you know, gut check on some of these things in terms of number of people, number of commits and all this sort of thing. And yeah, so it’s, it’s, it’s been quite a ride.

Ryan:

[00:49:17] You mentioned the local versus the remote version of these coding agents and it sounds like you have a lot of conviction that the right long term direction is not the local versions, it’s remote in the cloud. Why is that?

Michael:

[00:49:31] Well, I think about, I think that the people who actually for whom it is sticky now and I think there’ll be more of it is that you imagine, if you imagine you want to just automatically, you know, set for the agent at every like GitHub issue or linear task thing that comes in or anything like that. I mean, obviously, you know, there’s, there’s costs with those things, but. Or you can, it could be abused, but like, you know, let’s say for an internal private repo, right, that you, yeah.

[00:49:56] Want to be able to just have it be a piece in any sort of automation pipeline. Right. And so you can’t. You have all that happening on your, on your laptop. And so I guess, I mean, you know, as a, as an individual ic, maybe we’ll still personally spend more time with the local agent. But in terms of, you know, compute time of agents doing work. Right. I think getting that set up in the cloud, I think, you know, a little bit, it’s like a little more to get set up right now, but once you have it, it’s, it’s quite nice.

Ryan:

[00:50:27] I see. Okay, so you’re not saying that product, the local one will change. You’re saying that like across the industry, the compute that goes towards agents majority will be in the cloud.

Michael:

[00:50:38] And yeah, I mean, even in. I think when the freeze code extension first came out, one of the things was the ability to take the conversation that you were having and hit a button, have it transfer to the cloud if you were set up to do it. Right. So I think that we’ll continue to see that where maybe you’re working on something, you want to throw it over here. You’re like, bring it back when it’s done.

00:51:00 — How he uses Codex

Ryan:

[00:51:00] Codex usage is up 5x since the beginning of the year and over a million people are using it now. I’m curious, has your AI workflow changed a lot since you’ve started to use this newer version of Codex?

Michael:

[00:51:14] Yeah, it has. I’m a much bigger user of the app now maybe than I thought I would be. For a while I was very strongly in our VS code extension. I want that sidebar. I want all the code there next to me. I feel like these things should be together. I’m not a person who doesn’t look at the code. I’m not like, I don’t for projects that are like true prototypes that are throwaway, I will actually not look at the code.

[00:51:45] It’s very freeing. I understand. I’m so excited about it. But for the code that goes into Codex itself, I’m like, no, I still need to look at this. This is pretty important. This is what’s going to affect everybody else. But you start to get a sense of, okay, I’m confident the model’s going to be able to do this change. And this Sort of thing I need to babys it. I’m going to just like write a lot up front.

[00:52:09] And so, you know, I have like four or five clones of the Codex repo of my machine and I, I have enjoyed in the Codex app like the multitasking I just is, is just a lot easier because now you’re just kind of hang out in one window and you know, I try and it’s off the game. Exactly. But it’s like, you know, it’s like how much throughput can I get right. In terms of how many balls can I juggle in the air?

[00:52:34] Sometimes it feels like, it can feel like a little hectic when you’re really trying to context but at the same time you’re like I’m getting a lot more done and that I almost feel a little bad writing code by hand sometimes because you’re like, if I had asked this in the right way. It’s like when you started it was like, oh, I’m just going to change these three lines. And then 30 minutes later you’re like, oh, I can’t kind of, you know, that, you know, we all, we all like to type still, I think, and that sort of thing.

Ryan:

[00:53:08] What percent of your code would you say is human written versus the model generates it these days?

Michael:

[00:53:14] Oh man, it’s model’s got to be like 80 to 90%. I mean, yeah, yeah. I mean like especially like debugging a test or like the CI thing is bad. Like, I’m like, I’m like, hey, you know, you know how to write, you know, print, debug, whatever, like. And that’s great, that’s really freeing.

Ryan:

[00:53:32] Digging into the problems that are suitable for LLMs and the ones that are not. Like what, what do you need to see where you think, okay, I need to go in and you know, write it myself. What’s that? 10% and also what’s in that? 80, 90%.

Michael:

[00:53:47] Yeah, I mean that’s a good one. I think about that, you know, because every time I sit down I’m like, like, should I write this? I mean the answer is almost always no. There’s things that are like lower level and someone, you know, so Codex, the, the harness. The part that I, you know, spend my time on is in rust. And that means that, you know, we can do and we do do like operating system specific things in that code base.

[00:54:15] And actually. And I spend a lot of time on, on the sandboxing. Right. So the thing that really upholds like, you know, the security integrity of what? Of what we’re doing and that the model can’t go outside, you know, the bounds that you set. You know, I, I do more of that by hand because I need to make sure, really sure that that’s all correct. Right. That our test coverage is, is good or sometimes I’ll, you know, seed it and then you know, once I’ve got like the groundwork and like, okay, I have the pieces that I, that I was a lot of feelings about and then could let it like fill in the rest and that sort of thing.

[00:54:50] But, but you know, a lot of like refactors and things like that. Actually, I think like a lot right now is like, you know, I’ll have a build up a big, you know, PR that does too many things. But I know it does too many things. I’ll be like, okay, please, you know, please split this up into reviewable sized commits, things like that. Like I think about how much time I probably spent on that sort of stuff before and now I can.

[00:55:12] Right. Alligate frees you up.

Ryan:

[00:55:15] What about like code review? Is there like what percent of lines of code do you and at OpenAI generally are people reviewing manually or are there agents that are like, you know, imagine you write a test or something. I don’t really need to review that.

Michael:

[00:55:31] I mean, yeah, I would say I liked the approach where you know, the, the agent should do, you know, multiple rounds of review or whatever until it’s confident and that it’s like, you know, worth a human’s time of looking at it. But we still do look at it ultimately before it goes in. I would say generally speaking, I think there’s, we have our agent’s empty file like everybody else does. There’s still sometimes you find a gap in knowledge that needs some context that needs to get added back into there or it’s just things that we haven’t memorialized the agent this but as a human I still happen to know and things like that.

[00:56:13] So we do catch things. But I’ll say actually now that people are also using AI to write their pull request summaries, our summaries across the team I would say are getting way better. So now also when I’m going into review, it’s been reviewed by Codex, there’s a summary that’s like that we actually make sure it has the why and the what of why, the reason behind the pr. So that is certainly, I think helping get through these PRs faster, which is good because there was a lot more review to do.

Ryan:

[00:56:45] That’s, I mean that’s Amazing. I feel like 50% of diffs have, like, you know, almost blank. The test plan says, you know, I don’t know, arc build or whatever.

Michael:

[00:56:56] The commander.

00:57:00 — Why Codex’s harness is open source

Ryan:

[00:57:00] I want to talk about the. The codec cli. That’s open source. Why is it open source

Michael:

[00:57:09] for something that’s that critical to what’s going on on your machine? That’s one of the aspects of open source. I’m not the most zealous about this sort of thing, but I sympathize with it. This idea that, hey, you’re going to put this thing on my machine. I care about what it’s doing, and I think in this domain in particular, it’s really important that people can look at it and have an idea of what it’s doing, because people have a lot of questions about AI agents and that sort of thing.

[00:57:43] And so I think this area, this domain, I think it’s actually really important. And also we have gotten a lot of, I think, great contributions and bug reports and things like that that we would have missed out on. And. And then also I think it’s, you know, just, you know, sharing with the world with, like, how this is done. So we do it through code. Right. I’ve. I’ve put out one blog post about how the Agent Loop works.

[00:58:14] Like there is a plan to do more of that. I’m actually excited to. It’s just, you know, just. It’s just time is this limiting factor. So it’s. The higher. Yeah, it was funny because I had two candidates who came through, and one’s like. He’s like, hey, I wrote it, right? I was like, no, no, I. I wrote it. And another person came in. He’s like, oh, you can tell that. That, you know, that you did not outsource the.

[00:58:35] Okay, good, right? I was like, oh, thank you.

Ryan:

[00:58:36] You know. So, yeah, you mentioned the blog post. How does Codex find what is available to it in its environment? When I’m running these things, I’ll see. It’s kind of amazing. It’s thinking to itself and, like, discovering all these things in my terminal. So. Yeah. How does that typically work?

Michael:

[00:58:55] Yeah, I mean, there’s, you know, there’s a few. I mean, there’s obviously what is, you know, in Codex’s base training, like, it loves to use RIP grep. It uses RIP Grep very well to find all sorts of things. And then there’s, you know, if you have your agent’s MD file and you’re saying, hey, in this repo, like, these tools are really important. Right? You should Use these. Right. Or the readmes or whatever.

[00:59:15] Or obviously if you use MCP and you associate these MCP servers with your, you know, where you’re working on. Right. That injects the set of tool definitions, like at the start of the conversation. Right. So then that kind of. Yeah, so that’s not even like discovery on Codex’s part. Right. It’s kind of just like put it front and center there. I see.

Ryan:

[00:59:38] So some of it’s the harness explicitly throwing that into the context. And then there is a big chunk though, where the model is just doing all the heavy lifting of finding things. Is that right?

Michael:

[00:59:49] Yeah.

00:59:50 — Top technical book recommendations

Ryan:

[00:59:50] Kind of like reflecting over your career the breadth and depth of your work. If we look across all of it, I mean, it’s insane. You were JavaScript front end. Then you have all these dev tools, build fuzzy file, search, virtual. Now you’re working on Codex. I’m sure you had to continue your engineering education to kind of get through all these projects. What are the top technical books that have helped you educate yourself?

Michael:

[01:00:20] Yeah, I mean, you know, so one was this, this book on operating systems. It’s like a thousand pages or something like this. It’s the Addison Wesley book. Trying to remember the author. But it was funny because I was, when I was working on the virtual file system, so I had actually gotten to that point in my career without ever writing C, like undergrad. Like, it was like more theoretical. Like we just never had to touch it.

[01:00:45] And then yet I’m like working on a virtual file system project and now it’s like very low operating system. That’s why, you know, joke. That was the kind of the worst engineer of the project. And you know, and someone said something and I realized I was like, I don’t know what they’re talking about. This is kind of embarrassing. And I was just like, you know, what book do I have to read? And I think my manager, Aaron Kushner at the time was like, he was like, well, there’s this thousand page book.

[01:01:07] And I was like, done. So Pipe bought it, read the thing cover to cover. It, like took it away with me. I took it everywhere until I finished it. It’s. It’s kind of amazing and sad or weird like how, how far many of us can get like in software engineering without really having a clue how computers work. Like, there’s just so many of those levels of abstraction. But, you know, one hand it’s very freeing and then on the other hand it’s a little bananas.

[01:01:33] And I think that what I would say to people right now is, is, you know, actively trying to go like deeper through the layers and understand these things. And like, because many times I saw other people do it and now I can do it, is that there are problems some other people could solve that I couldn’t solve because I just didn’t know that like there was this cruft between these two layers and if you got rid of it, right, you get like a 10x improvement or something like that.

[01:02:02] Right. If you, if you’re just operating so high up you don’t really know, you know, what you can, what you can break down, you know, so that. And so in terms of books, like I’ve enjoyed The, like the O’Reilly, the Rust books, you know, big fan of Rust, but they’re also like well written and thorough. And then honestly, another thing I’ve been telling people that’s not in the book category but probably more fun and I learned a lot actually are these CTFs are like capture the flag, like security type competitions and just, you know, it helps with like kind of like adversarial mindset.

[01:02:41] It helps with. I think they’re usually just like. It’s like a, it’s like a decathlon, like a computer decathlon, I feel like. Right. Because like, like there’ll be multiple challenges and maybe this one you need to like understand assembly and this one you need to understand like what someone’s like janky PHP admin page is doing and all that sort of thing. And it’s. And it just forces this breadth on you in a way that’s kind of hard to generate otherwise.

[01:03:09] And it’s also just more fun because it’s like a game and that sort of thing.

Ryan:

[01:03:12] Can you give some context on what CTF is?

Michael:

[01:03:15] Yeah. So it can be a number of things, but it’s usually a competition usually in like the infosec, the security domain. And there’ll be a set of. There’s like the Jeopardy style one which is like there’s a bunch of challenges and like, you know, designed like crafted ahead of time and they all have point values associated with them. And so, you know, there’s usually some fixed amount of time and it could be individual, it could be with a team and you’re trying to solve these things.

[01:03:42] And there’s a flag. There’s always like a secret, you know, piece of text in there that usually has some format. And the way is that you basically, if you can discover the secret piece of text that means that you have the, the challenge is set up that you would Only have gotten that if you had figured out, you know, reverse engineered or whatever you were supposed to do. And then you get submit that piece of text and that’s your, you know, token to demonstrate that you solved the challenge.

[01:04:07] And so it could be like a race to like solve everything first or get the most points in a certain amount of time or different things like that.

Ryan:

[01:04:14] So it’s like an escape room, but in your terminal and you have only computers to figure everything out.

Michael:

[01:04:19] Yes, I see.

Ryan:

[01:04:20] Okay. And your recommendation is people who want to become better engineers, they should invest in doing some of these CTF because they make you solve problems that make you a better engineer.

Michael:

[01:04:31] Yeah. And you know, and then you, you know, also develop skills and things that you just wouldn’t have. You know, if you’re just like a, you know, writing react every day, like, you’re probably not gonna like open up GDB and like reverse a, you know, a tic tac toe game. But like, I did that because of, you know, a CTF challenge was to do, you know, exactly that. Right. And, but then it’s like, but then I learned how to use GDP and like, and then, then you start, you know, when you’re faced with other problems, you’re like, oh my.

[01:04:58] Just toolkit of, of ways to solve things is just much broader now.

01:05:02 — Why deep technical skills are still valuable (for now)

Ryan:

[01:05:02] A lot of people, especially earlier in their career, they see the power of Codex doing everything for them in the terminal and I just imagine them saying, oh, well, I don’t need to learn gdb because Codex knows it.

[01:05:16] Wwhat would your advice be given the landscape of these AI tools for people thinking about their engineering education?

Michael:

[01:05:24] Yeah, no, that’s, I mean, I think everyone’s like struggling to answer that question right now. I do think about it, you know, a lot and I, I don’t have a great answer. I, I, I kind of personally come back to my thing about like, I still think like trying to forcefully kind of go through the levels of, of abstraction and just understand, you know, more like at a deeper level how things work is going to be important.

[01:05:51] I mean, I think this will change, I’m sure this will change over time, but right now it’s still kind of like, you know, the questions that you asked the agent is going to affect the quality of the thing that you get out. Right. So if you’re not asking the right questions, you’re not maybe going to get the best engineering solution out the other side. You know, as things go on, perhaps that will also be another Layer that’s removed.

[01:06:15] I mean, I think that’s certainly where we’re going. I don’t know what time frame we’ll get there. Things do seem to be happening faster than we expect. But yeah, I think in general learning how to ask what the right question is. And I haven’t for myself even totally pinned down what that means for someone who’s starting out new. I fortunately have experience to, to fall back on or like that’s where that, that, that taste or that intuition of what to ask has, has developed.

[01:06:45] But you know, if you’re, if you’re starting out, I’m still not sure yet. And it’s also hard to say because we, we don’t really know a hundred percent where everything’s going.

Ryan:

[01:06:54] Yeah. When you reflect over your career, the expectations at these really high levels is kind of crazy. I mean it’s like already for Most people this E7 or senior staff level is this unattainable level of impact. So someone who gets promoted to that level is thinking, I gotta really work super hard now because the bar is like up here. And then you went two levels past that. And so I’m curious in the day to day now, what are your thoughts on these crazy expectations?

[01:07:28] Is this stressful for you?

Michael:

[01:07:30] I mean it never, it was never not stressful, I think for me because I really, I think also because I sat in calibrations, right? And I, you know, so for a bunch of other people, you know, you talk about their level, you talk about their impact and you know, you want to be really fair and have this integrity around. Well, this level means this thing and this is the impact.

[01:07:51] Like that’s what it meets all is, that’s what it exceeds is. And then knowing that like then you’re, you know, playing it out in your mind like, well, someone’s sitting in my calibration and talking about this thing and they want to be fair, right? And it is a little scary. Like, you know, you get to E8 and you’re like, oh, you know, it’s a D1 or a director and you’re like, how do I have as much impact as a person who has maybe like over a hundred people in their org?

[01:08:13] And so that’s hard. A lot of people do it again, a lot of people who are an IC who do it, do it more by a different form of people management, right? Where they’re trying to, you know, like write the right doc and get the people aligned and do that sort of thing. And the reason that they do it and are Not a director is that oftentimes the people I’ve talked to, people who are in this, like, that they’re like, well, I have this technical credibility or because I built this thing, like when I go to the team and talk to them, it lands differently than if like a engineering director does or some version of that.

[01:08:46] Right. Happens a lot. And so then, you know, when you, when you say you’re influencing, you know, 50 to 100 people as a senior IC, then you’re like, oh, okay, that’s like, you know, D1 impact, you know, whereas as if you’re a coder, I see. Being really thoughtful about the projects that you pick. And it can’t just be like, this is fun for me type of project. I mean, if you actually care about not getting fired or you’re getting like Sam eats all or better.

[01:09:22] And even when I would start a project, certainly like the later I went on, I would think a lot about like, okay, like I maybe there’s this feature, I just want to write it because it’s fun. And I’d be like, no, I should let somebody else do that and think about, okay. I mean, it’s kind of like with Codex now, like, what code should I personally write to maximize my impact versus if someone else could do it about as well as me, let’s say 80% is good, I should probably let you know, have someone else do that.

[01:09:51] But even then, if you’re leading a five person project, is that still, you know, still getting to E8, E9 impact still hard. So really finding that project that’s a force multiplier. So, you know, like the virtual file system was a really great project because that was going to, you know, we knew that down the road like this was really going to unlock so many things, prevent us from being completely blocked.

[01:10:19] Right. Actually, another big part of it is, and you know, one of my managers talked to me about it is, you know, not actually recognizing senior managers enough who are the person who pairs the senior IC with the right project. And because some senior engineers are amazing, you know, fixers, coders, but they’re not the idea come uppers with, but they’re the person you want for very difficult technical projects.

[01:10:54] I don’t want to out anybody here, but I have people in mind. But a lot of times it’s the manager who realizes, oh, this project needs this person. And that person would have never realized it themselves.

01:11:07 — How to start projects well

Ryan:

[01:11:07] One of your old colleagues, Adam Ernst, I think. Hi, Astemia. What are some other engineers that you admire and why? And you know he mentioned your name, of course. And specifically he talked about your ability to start projects was really good. And obviously, I mean, look at all these projects, right? I mean, a lot of them, you created them out of nowhere. Like it was just, you had an idea and you went off and built a prototype.

[01:11:30] You came back and you were very convincing that it was a better solution. Do you have any advice for engineers who they have a problem and a solution and they want to like build a project from scratch?

Michael:

[01:11:41] I mean, I think, I know a lot of good projects, I think come from being a little bit dissatisfied about, about something. Right. And I think, you know, it’s funny, like sometimes I, for better or for worse, I was just so charging ahead and just like building the thing that like, not really thinking about like what the best way to do it was. So actually a really funny example is Google Calendar going back really far.

[01:12:06] I was like, I want weather. I want to have weather, little icons showing the weather in Google Calendar. And I was like, I’m going to do it. And I had mostly been JavaScript especially and not done any of the backend stuff. And I just charged through and I cobbled things together and made it happen. And then my tech lead was like, wait, how are you storing that information about the weather and stuff?

[01:12:30] And I was like, oh, I just threw a blob of XML and everything. He’s like, we should have talked about protocol buffers. I was like, binary formats to save bytes. That I was just so set on weather of all things that I never bothered to ask anybody if there was a better way to do what I was doing. I was like, it’s done.

Ryan:

[01:12:51] So I guess that’s a unique skill of yours, is the digging into the dissatisfaction and solving your own problems. Seems like almost every single one of these projects is I want my, I want something to happen. This, this shouldn’t be this way. And then you went and you solved it.

Michael:

[01:13:08] Yeah, yeah.

Ryan:

[01:13:10] Well, I think another unique thing that I see is a lot of people, they, they get to work and they, I don’t know, they’re, they’re in their dev environment. I’m sure there were many other people who saw with the buck story, for instance, they go, oh wow, this build is really slow. Oh well, like I guess I’m going to go to the micro kitchen and get it and call its builds. What gave you the confidence to know that you could make it so much better?

Michael:

[01:13:34] I mean, that one was, you know, I have to credit, like having been at Google, I never worked on the Blaze Team. I never knew how that like worked I ever touched the code or anything, but I was like, I know that there’s a thing out there that has this shape and is a lot better than this thing. So that’s an existence proof whether I could do it or not. Like tbd. Right. But I think that was helpful on that one.

[01:13:59] There’s a lot of, I guess, prior art that gives me confidence in things and I usually generally identify myself as a coding machine. I guess with Codex now everyone’s a coding machine, but that I felt that I always had confidence. Build a prototype quickly and at least answer that question or test my basic hypothesis spiritually. Should there be a way forward to this thing that I think should exist.

[01:14:23] And generally, you know, you’re, if you are determined, you find a way.

01:14:27 — Advice on writing better and career planning

Ryan:

[01:14:27] Yeah, I’ve noticed that pattern. A lot of people who go to big companies, they see this world class infrastructure and then they go to the other company, oh, you don’t have this, you don’t have this. And they build their, you know, those new versions of it. I’ve read a lot of your writing at this point and it is so clear. It’s some of the best examples of good technical writing. What advice would you have for engineers who want to write better?

Michael:

[01:14:52] I mean, I think a lot of it is, well, I think reading other good writing is certainly a good start. Right. Maybe, you know, consciously or subconsciously start to pick up patterns of like, what, what is out there? I think, you know, really high level thinking about like what what is, what is it that I’m trying to convey? What would someone actually really want to know? And, and outlining a lot up front right is I think a lot of people will give this feedback.

[01:15:17] But it’s, it’s impressive how really important that is. And just being like, does this set of things like linearly follow? I think is a big thing. And then asking yourself, oh, I went from this point to this point, was that too big of a jump? Is there something that somebody actually would have reasonably missed? And I think personally I feel like that’s a reasonable feel for where that gap would arise.

[01:15:44] And then if you can kind of anticipate that and magically put in the example that like someone would have that someone needed to make that jump, I think like that, at least for technical writing is like, is a big deal.

Ryan:

[01:15:59] You have that career note that I love and at the beginning of it you lay out this three step plan for impact. Can you explain that three step. I feel like that’s a good algorithm for people to use in their careers.

Michael:

[01:16:12] Step one is figure out what you really like to do. Right. And you know, as I mentioned, like, it’s good to broaden that, but it’s also good to, to be honest with yourself so you don’t know. And like the, the quote, the hero quest that I went on, that, that didn’t pan out because I was working on stuff I didn’t really, you know, truly love. And then two, step two was figuring out what your employer is, what’s really invaluable to them.

Ryan:

[01:16:37] Right.

Michael:

[01:16:37] And as I talked about at Google, I, I didn’t do a good job of that. I did stuff that I was really excited about, but it wasn’t, you know, it wasn’t, you know, AdWords for Google or anything like that. Um, yeah. And then step three is like, find that intersection and then just really lean into that. Um, and you know, the more that you can do that, I think the more successful, you know, you’re going to be.

[01:16:58] And, and, and the, I think the challenge is sometimes it’s not always there. Right. And maybe you have to go, you know, find somewhere else to make that happen.

01:17:06 — Advice for his younger self

Ryan:

[01:17:06] Okay, and then, yeah, last question for you is if you go back to yourself at the beginning of your career, knowing everything you know now, what advice would you give yourself?

Michael:

[01:17:14] I think I should have been open to learning more things sooner. And, and I, I, you know, to be a little gentle to myself and I think other people are in that situation is that, you know, there’s so much to learn when you’re starting and then whatever your first programming language is. I think it’s funny, I think everyone has a soft program spots for, they’ll like make excuses for it like ever. Like, oh, no, it’s a totally good language.

[01:17:37] And I think it’s because it’s like the first thing that enables you to do a thing, you know, do anything, right. And, and then it’s like, oh, okay, I can finally do something. It’s such a relief. And, and, but it’s also like a hazard because like then you kind of want to hold on to that thing because like you’re finally productive and now you’re like, ah, it took so long to get to this foothold. I don’t know how long it’s going to take to get to the next foothold.

[01:18:00] So I think like, in my particular case I probably did maybe went too deep with JavaScript. And like, like I said, it took a long time before I wrote nec. And I think, you know, if I had been a little bit more curious and a little more flexible in terms of what, like, types of projects I was willing to take on or things I was willing to learn. You know, it came eventually, but I think if that is probably the biggest thing that maybe could have made a shift for me earlier.

Ryan:

[01:18:26] I gather from your story there was a point with the xcode where you said you hated Objective C and then you were coming up with ways to compile the Objective C into Java into Objective C or something like that, maybe. And then also talking about the C for miles, it seemed like a very concerted, okay, I’m going to learn this as opposed to just kind of being open to it. Yeah, yeah, it makes sense. Well, maybe with Codex in the future it’ll be less of a hurdle for people.

[01:18:59] You could just kind of say, hey, I know JavaScript, write this in Rust or whatever.

Michael:

[01:19:04] No, it’s true. Opens a lot of doors. Sure.

Ryan:

[01:19:06] Awesome. Well, thank you so much for your time, I appreciate it.

Michael:

[01:19:08] All right, thank you, Ryan.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:42 - Working at Sun Microsystems

00:10:17 - His growth to distinguished eng

00:19:14 - Why goaling on promotion is bad

00:29:34 - Stack ranking and layoffs

00:36:00 - Why he hated the Oracle acquisition

00:44:19 - Why Bezos is the apex predator of capitalism

00:48:04 - Differences between CTO and VP

00:49:58 - Starting his own company

01:02:37 - Grilling him on his past

01:11:57 - AI boom and bust advice

01:14:41 - When he was happiest in his career

01:17:22 - Top career regret

01:19:21 - Advice for younger self

Transcript

00:00:42 — Working at Sun Microsystems

Ryan:

[00:00:42] Your career started at Sun Microsystems and that’s kind of a legendary company. It was before my time. What was the industry like when you first entered it? Was there FAANG equivalence at the time? What was Sun Microsystems like? What were the other companies like?

Bryan:

[00:01:04] Yeah, you know, it’s kind of funny cause there’s always been like a FAANG kind of equivalent. You know, in the 60s it was called the bunch. Burroughs, UNIVAC, NCR control data and Honeywell were the bunch. It was IBM and the Bunch. So for kind of every era there’s always some hot group, some growing sector. And in my day though, so things were, I mean, I think in a bit of a divot. So I graduated in 96, so I was really interviewing in 95.

[00:01:37] And there was for sure on campus interviews happening. But Microsoft was very dominant in that era. And I definitely knew I was not going to go to Microsoft as a kind of a point of principle.

Ryan:

[00:01:52] Why was that?

Bryan:

[00:01:54] Oh, I view Bill Gates robbed me of my childhood. Microsoft had such garbage. I mean, they really did. In what sense? Their operating system was. You know, I came up in the personal computer era, so I came up in the 80s. And it wasn’t until, you know, I went to University in 1992 and it wasn’t, you know, I had just been accustomed to kind of like garbage on the computer. And it’s like it’s garbage in a way that is kind of unfathomable.

[00:02:25] Now DOS itself, the disk operating system from Microsoft had no memory protection whatsoever. So if an application misbehaved, the machine would just reboot. And this just became like part of life. It’s like the machines reboot. You gotta boy, you gotta save your work because you’d be in WordPerfect or WordStar, whatever. This is where I sound like a true like living fossil. I feel like I’m relaying like coming across the Oregon Trail or something.

[00:02:51] But you’d be in your word processor, you know, working on your English paper and all of a sudden the machine would reboot. And actually that was just, that was life. And you hope you saved it, hope you had a hard copy. And it was only when I got to college that I realized like, wait a minute, hold on, there’s actually something called there’s memory protection in the microprocessor. And for much of my adolescence there had been memory protection present and DOS didn’t use it at all.

[00:03:18] And the, and Windows used it. I mean with Windows it gets complicated. But the answer, the short answer is Windows didn’t use either. And so it wasn’t until I got to like Unix on a workstation. And I just remember being like, you know, 18 years old and having like having a command line shell up in one window, having an FTP to woo archive, downloading some like EGA game that I would want to go play back in my dorm in another window, having something else going on in a third window, having a compile in a fourth window and having this happen magically concurrently was just mind bending.

[00:03:55] It’s like, how is this even possible? I mean again it sounds, and I appreciate how old this sounds, but this is what it was. And then looking back I’m like, why do I have garbage from Microsoft? And Microsoft was just not an operating system company. They were the dominant operating system provider. But in their DNA, the most fundamental, like the nucleotide base pairs of Microsoft are compilers.

[00:04:21] They were a compiler company, not an operating systems company. And Apple was, you know, Apple, what was Apple at that time? I mean Apple had this is long fired Steve Jobs. This is the, the Apple was kind of the Mac was really honestly not that much better. It was like cleaner but also didn’t make use of memory protection. So this is just like unconscionable to me. And there was just, and this is like aside from all the bad behavior from Microsoft, and there was plenty of bad behavior from Microsoft.

[00:04:48] Microsoft was very anti competitive. And you know, the findings of fact in the Netscape, you know, years later would be just very underhanded, very wanting to. It was not a company that had the best technology. It was a company that jockeyed for position and then would kind of abuse its position.

Ryan:

[00:05:08] What’s the TLDR of that abuse?

Bryan:

[00:05:12] So what Microsoft would do and this is a pattern you would see repeated since. But what Microsoft would do is they would announce, you know, you’re a promising company that has something that’s, that’s a application that people might use. They would announce that it’s going to become a forthcoming feature of the operating system. But as it turns out then it wouldn’t show up. It was vaporware that the term vaporware, I think if it didn’t originate with Microsoft.

[00:05:38] Microsoft definitely perfected the art of vaporware, of announcing something that didn’t exist. And they were kind of dominant with this mediocrity. Now to their credit they also when they went in. So for example Microsoft Word, which I mean still exists but WordPerfect was the really the dominant word processor. WordPerfect and WordStar but Microsoft Word was very bad and to their credit it would get better over time and ultimately supplanted WordPerfect in part because they started bundling it with the operating system.

[00:06:10] So they, they were very deeply anti competitive. And that’s not like an opinion, that’s like a, that’s just a judicial finding of fact. That’s a judge Alsop finding a fact. And that changed a lot when Gates, I mean Gates when he hit middle age, that company changed a bunch. And after the antitrust case in the late 90s. But I was not going to work for Microsoft.

Ryan:

[00:06:32] I saw in your goodbye post for from Sun Microsystem when you were leaving the company you wrote public post and you said that when you joined the company and you interviewed you only knew two things that you one, you wanted to work on an operating system kernel development and two is that you didn’t particularly want to work for sun but it looks like you had an amazing time. So what flipped the switch from going from I don’t want to work here to I love working here.

Bryan:

[00:07:05] So the, when I, I had kind of talked with other groups at sun and I just like the people I dealt with at sun were fine, but just not where it didn’t feel energizing. And certainly PBN did not feel energizing. And I was kind of accustomed to being the youngest person in a group. But you don’t wanna be completely by your lonesome. You wanna feel like there are other people that have that kind of, that same shared passion.

[00:07:32] And when I met Kevin Clark and Jeff Bonwick, it was like a bolt of lightning. Those two were energized, passionate really. They saw all of the same things that I saw in terms of the pot operating systems to be innovative and were and wanted to go, like, rip out broken code and replace it with, with beautiful code and all that. That same verve that, that I felt. And yeah, it was, it was absolutely.

[00:08:03] Meeting those two. You’re just like, okay, I, I, I. There are. I am going to work for Sun. I remember actually very vividly coming back from that trip being like, oh, my God, I’m going to go work for Sun. If they make, they’re going to make me an offer and I’m absolutely going to go work there.

Ryan:

[00:08:15] I saw that eventually you grew to a distinguished engineer there. And what was the career ladder like?

Bryan:

[00:08:22] It would be interesting to know where the kind of member of technical staff, staff engineer, senior staff engineer, distinguished engineer comes from. I think it might come from Sun. I think it’s possible that sun is the originating company. It’s also possible from Xerox parc. You’d have to kind of take it apart where. And I actually remember when, you know, I had interacted with some folks at sun and they’d give me a business card, and I remember like, coming out to sun, being like.

[00:08:46] And not really knowing what my rank was. And I’m like, Because I was gonna make business cards, I’m like, why? I’m an engineer and I’m on staff. So I like, I think I’m a staff engineer. So. And, you know, I kind of go to make business cards. And then I kind of like didn’t, didn’t complete submission on the form or whatever. And then of course, it wasn’t that long afterwards. Everybody was like, oh, wait a minute.

[00:09:03] That was like. That would have been like saying I had got tenure. Like, I definitely am not a staff engineer. I’m a member of technical staff. I was an MTS3. So the, there was the, uh, you would kind of climb up through those ranks and then to, to staff engineer, senior staff engineer, distinguished engineer.

Ryan:

[00:09:17] And three was the start of the.

Bryan:

[00:09:20] No, I mean, it’s supposed. I think it did start at MTS1. Um, no, most college hires would start at MTS2, but they, but I, I was an unusual college hire in a lot of ways.

Ryan:

[00:09:30] Interesting. So they brought you on two levels deep as a new grad.

Bryan:

[00:09:34] Yeah, yeah, yeah, that’s right. Um, I mean, I’d had a lot. I. So sun was actually not the first company I worked for. I worked for a company called QNX in Canada and I done kernel development there. So I had done OS development at QNX for two summers and done some, actually some work that I was, that I still think is actually. I love That I love working for qnx. I love being in Canada and really. And I had decided that I didn’t want to go back to QNX professionally.

[00:10:01] That I did. The other thing I would say is I decided that I wanted to come out to Silicon Valley. I did not want to be. I’d gone to school in the Northeast, had worked in Canada, but I really wanted to come out to Silicon Valley.

Ryan:

[00:10:11] I see. So then when you, when you look at your journey growing from I guess this MTS3 to.

Bryan:

[00:10:17] Yeah.

00:10:17 — His growth to distinguished eng

Ryan:

[00:10:17] Up to where you grew, like if you were to break down that. I guess the story behind that growth and kind of what are the highlights that, that stuck out to you, that made you grow?

Bryan:

[00:10:26] Yeah. So, I mean, I was not, I was very much not focused on my, my grade, my rank or promotion that was not on. I was not interested in that. My view was always like, I would much rather be under promoted than over promoted. I’d much rather be doing work where people are like, wait a minute, why are you back here? Like, what happened to you? That’s just. I am not interested in being kind of doing work performatively.

[00:10:59] I was always going to do what I felt was the most important thing to go do. And sun was a company that was very accommodating of headstrong engineers. And so I, that’s what I, what I wanted to do was solve the problems that our customers had, make the operating system what I believed it could be, kind of realize that vision for what the OS could be. Sun was the only company, honestly in 1996 that was really invested in the operating system.

[00:11:28] Every other company was mortgaging its future to Microsoft and Windows. And all these computer companies were giving up on their own operating system, giving up on Unix. It was really Unix’s darkest hour. And being in that group in Solaris Colonel development during those years was enormously energizing and allowed us to do all sorts of things. And then along the way you get promoted, right? Along the way, like, promotions were just the kind of thing that happened as you, as you went along the way.

[00:11:56] I mean, sons became. I learned, I would say I took away a lot of life lessons in terms of what works and doesn’t work. I found that the cause, of course, we had formalized performance review. And I found this is like not a deep thought, but that formalized performance review never resulted in me having higher performance. And this is like, this is one of these things that you kind of like feels very naive to say because it’s like no dummy, like performance Review is not about you performing better.

[00:12:28] It’s about us measuring your performance. It’s like, well, that’s not what it should be about. What it should be about is, like, what feedback is great. Like, we should be making everyone be the best that they can possibly be. And the, The. The. The formalized, the annual cadence of it, I felt was very broken. You know, we would have you submit a self review, of course, and then your review is like, oh, wow, my review looks stunningly like my self review, albeit with grammatical errors introduced.

[00:12:57] And you know that in every review, even the reviews in which I was promoted, they were just not uplifting. Like, that was not. When I look back at the moments of, like, what were the highlights for me during that progression, it was never being promoted. It was always doing a significant body of work or nailing a hard bug or working with someone on something that we thought was impossible. Like, that was the stuff that.

[00:13:21] That. That was really catalytic for me. And then this, like, son just kind of had to promote me along the way. Now, the one exception, though, I would say to that is that the. The promotion to Distinguished Engineer was very regrettable and stupid. I mean, not my promotion, but just in general. Like, the process. The process for being promoted at SUN was, I would say, very traditional in that, you know, you’re taking your folks that are performing very well and you’re promoting them up to the next grade.

[00:13:48] The SUN did have a very silly process thing that, you know, they had three grades. They would give you superlative, excellent, and good. And then I think there was like a, you know, like 70% good, maybe 20% excellent, and 10% supportive, something like that. And they had this thing where if you were superlative, they had to promote you. Like, that was like the index for promoting you. Like, if you got that superlative grade, they promote you.

[00:14:14] But also they had to promote you if they gave you that superlative grade. So you would. I had these. I had these reviews that were like, apologetic, being like, look, I have to give you an excellent grade, even though, like, the work that you’ve done over this past period has been extraordinary and innovative and actually better than the work that you did two years ago or a year ago. But I can’t actually give you the supportive grade because if I give you the supportive grade, I would have to promote you.

[00:14:40] And we just promoted you a year ago, and it’s really like, we don’t want to promote people any faster than every 18 months. And you’re like, okay, does this make sense to you? I mean, this doesn’t like, I was just like, okay, if you’re satisfied with this. Fortunately, I just don’t care one way or the other, so. Sounds good. Okay. Can I get back to work now? Are we done here? It’s like, are we? And then again, trying not to be trolled by the grammatical reviews in my.

[00:15:03] In the review that’s being handed to me. So now the difference for that was the. And you know, the promotion of staff engineers was a big deal. And like, great, fine. Actually, it was funny because all the staff engineers. So sun had this thing called keep, the key employee incentive program. And KEEP was effectively like an annual dividend. It was like a bonus. So, you know, and there was like a keep pool that was set across the company and, you know, kind of a traditional thing.

[00:15:31] Right. And the. The metric was based on, like, company performance. That’s what they. And so I did hear when I was, you know, in mts 3 mts for the staff engineers, like, you know, they would get their keep bonus. And if the keep bonus is really good, they take everyone out to dinner. And it was kind of like, you know, it was like being at the mining town, right when there’s a big or strike or what have you.

[00:15:49] Gold strike. So I was like, okay, wow, this is like, this will be great. So of course, I mean, just like, of course, like, when do I get promoted to staff engineer? It must have been in like 2000, 2001. I get promoted to staff engineer as like the dot com nuclear bomb goes off. And the keep bonus was. I never got like, keep bonus was zero. From then on out, sun lost 98% of its value in the public markets.

[00:16:11] And like, I never. So it’s a good thing that I wasn’t doing it for the keep bonus because the keep bonus was zero. But then getting. So that’s what staff engineer, senior staff engineer, and then distinguished engineer was really unfortunate in that there were kind of two paths in to distinguished engineer. And this may be true of a lot of companies. I don’t know. I mean, I don’t have ranks at oxide for this reason.

[00:16:33] Like, I think ranks are corrosive. I don’t think they get people to do their best work. I’m not interested in them. But the at sun, there were two ways to be a distinguished engineer. One was, of course, to be like, promoted from a senior staff engineer to a distinguished engineer. And that was a very, very. I mean, I would say rigorous, but it’s giving it far too much credit. It was a very political process.

[00:16:54] The des would vote on whether you should be admitted to this country club of distinguished engineers, which is a terrible idea. Putting like, this is like the wrong place for democracy in a society. Right. For so many different reasons, but. So that’s how you would get promoted. Distinguished engineer. Very hard to get promoted. The other way to become a distinguished engineer is sun is buying your little company, is acqua.

[00:17:18] Hiring your company. And the company’s got just enough leverage during that process to be like, oh, by the way, our CTO needs to be a deep. So you would have these like, DEs just kind of like, pop in the side. You’re like, who’s this? Like, oh, yeah, we acquired their company. So. And. And there was definitely a difference in, like, you knew the. The homegrown DEs versus the ones that had been. Had been aqua hired in.

[00:17:42] But when I. And I actually, like. And this is like, I just do not want to even think about this process again, I’m not really a political person. Like, the last thing I want to do is. Fortunately, we had. I’d done work that was kind of so indisputable at that point that it was going to be. It was pretty clear that if sun is going to have such a rank, I was probably worthy of it. But again, I didn’t want to deal with it.

[00:18:03] Sun’s CTO at the time, Greg Papadopoulos, very grateful to Greg. Greg had chartered this group that we developed actually in San Francisco, first on Mission called Fishworks, where we had developed a new storage product. Storage product was going to gangbusters. Was a great product, although a great product with asterisks on it wasn’t as great as we thought. It was mishap that we learned a lot about it.

[00:18:27] But the product was doing really well. Greg, again, I was not in the meeting, so this was told to me secondhand. But your DE case needs to be presented by someone. Another de. Well, Greg presented my case, and so you had the CTO of the company. And son was not a very hierarchical company, and Greg was not a very hierarchical person. But Greg put apparently my materials up, and it’s like, I don’t expect anyone to vote against this.

[00:18:51] Take your vote.

Ryan:

[00:18:52] He said that?

Bryan:

[00:18:53] Yeah. And apparently it was unanimous. So it’s like, thank you, Greg. Deeply appreciative. So, Greg, I didn’t have to deal with that bullshit because Greg was dealing with it for me. And it was kind of like, you know, honestly, it was kind of a validation of my hypothesis. Like, Greg honestly viewed it more of a reflection on the company than a Reflection on me, it’s like, great. I had done work that was, at some level, kind of indisputable, and that’s kind of the way I had wanted the chart my career.

00:19:14 — Why goaling on promotion is bad

Ryan:

[00:19:14] There’s two schools of thoughts on. On, you know, how people structure their careers within these ladders. If a really ambitious new graduate engineer comes to you and they say, hey, I want to be a distinguished engineer one day, I have two ideas. One idea is I’m going to just focus on the work and just, you know, promotions will be a byproduct. Or I’m going to do the other thing where, you know, I talk to my manager.

[00:19:41] I’m like, hey, how do I get to that next level? And hey, what are those things that the next level needs to do? And then I will do work that molds to that mold.

Bryan:

[00:19:51] Yeah. I would try to steer someone to a third path.

Ryan:

[00:19:54] What’s the third path?

Bryan:

[00:19:56] Why. Why do you want to be. What does it. Why do you want to be distinguished engineer? Because if that’s the goal, if the goal is to be a distinguished engineer, you get there, you’ll be distinguished engineer. Now what. I mean, you can be so fixated on that. It’s like, at some level that doesn’t actually, that’s not the thing that matters. That doesn’t give you meaning. And to me, if I had a young engineer who’s like, I want to be a distinguished engineer.

[00:20:25] I’m like, that’s a recipe for a midlife crisis. So let’s try to dial you differently. And I think what I would steer someone to is what’s get you on a path that has meaning, where you’re going to be developing and what drives meaning for you. What’s important to you? And it’s like, well, what’s important to me is to be a distinguished engineer and be like, okay. I mean, you’d want to tease it apart, get them on the couch a little bit.

[00:20:52] Like, why. I mean, are you not. Were you not loved enough as a child? Like, what’s going on? Like, you know, and you kind of work through some of these issues because you don’t want to be dependent on that kind of external validation. I would want to be like, let me introduce you to people who’ve achieved what you are putatively setting out as your goal and are miserable. Like, let me introduce you to some miserable des.

[00:21:15] You want to be wealthy. Let me introduce you to some miserable wealthy people. So you know that it’s like, that can’t be. It there’s got to be something else that is the fuel in that furnace. And like let’s go find that thing. And then, then like the promotions will happen and the work will happen.

Ryan:

[00:21:35] Okay, but let’s say. Okay, Mentee comes to you and says, you have a good point. I want financial independence. A lot of people I think want that as they, they want to.

Bryan:

[00:21:46] Yeah. What’s financial independence though? Because like we are in such a well compensated domain. It’s like you and I right now could go out and get a cup of coffee that is the best cup of coffee. We’re in San Francisco, we’re in the Mission. We can get some coffee that is like some Ethiopian coffee, some amazing coffee that is like the, and that is attainable to you and me and to basically everyone else.

[00:22:10] It’s like a six hour cup of coffee. Right?

Ryan:

[00:22:13] Yeah. Yeah. Well, okay, well let’s say you define it as you don’t need to work for money ever again.

Bryan:

[00:22:20] But then what this is like, okay, you don’t need to work for money ever again. Like okay, I get that. But, but so you can go off and do the thing that actually motivates you. Well then go off and do the thing that actually motivates you.

Ryan:

[00:22:30] But what if it doesn’t make money?

Bryan:

[00:22:31] Well, I think and I, this is where I, you want to take it apart and so like you want to find a way. I mean certainly you’re going to have things like the, the, the, the difference that you want to make, the meaning that you’re going to find is going to be something that’s like, yeah, this is, this is just not a career path at all. Like I can’t actually sustain myself at all. So then you need to find a way.

[00:22:51] Balance that I would say with your career and you want to keep that balance. I think, I also think it’s a mistake to be like I’m going to achieve financial independence and then I’m going to do the thing that is meaningful to me. It’s like. And again you just. And I definitely had the blessing of most people were like me made a lot of money on paper in the dot com boom and just lost every penny of it.

[00:23:14] I mean I had a loss carry forward for so many years that I actually like one year I lost track of my lost carry forward. I’m like so. But I had a lost carry forward and I was, you know, I had like fond memories of the dot com bubble when I would take my, whatever it was, $3,000 a year against my. From a Lost carry forward the. That was most people around here. Most people kind of like lost it all, but also were fine.

[00:23:39] This is the other thing, like the dot com bust was super helpful because if you had geared yourself to be really economically motivated in 99, 2000, 98, 99, 2000, which would’ve been easy. 2001, 2002, 2003, like, that was not why you were here. Because this place was nuclear winter. And, and you had to, had to find something else. And I knew like, so the funny thing, I knew like lots of people that like had to remind themselves why they got in this and it wasn’t money.

[00:24:10] I also knew people, I did know people that made a bunch of money in the dot com boom. The people that made a bunch of money in the dot com boom weren’t that interested in tech to begin with. So when it went up in, you know, late 99, 2000, they were like, I’m selling it all. Like, I actually don’t want to do this. And those people, and there are a couple of them that I knew and some of them really, really struggled in the next decade because they didn’t have to work for the rest of their lives.

[00:24:35] But now what, like, you don’t have to work, so like now what do you do? And they were on their own search for meaning and like, not easy and going through periods of like, oh, like, I’ll buy six houses. Then you realize that like, six houses are kind of like a pain in the ass. It’s like you and I could go drink six cups of coffee concurrently, but you’re just like, what am I doing here? Am I just like proving that I can buy six?

[00:24:56] So why do I want six houses? Okay, then you scale that way back. I mean, it gives you like, it really opens your eyes about what matters and what, what doesn’t matter. What doesn’t matter as much. Certainly you need to be able to provide for yourself and provide for, for a family. You want to be able to raise a family and so on. So you want to be able to have that, but you want to do that in a way that’s honestly sustainable.

[00:25:19] I think that, that too many folks are focused on like, I’m going to do this so I can leave. It’s like, then maybe you shouldn’t be in this industry at all. Like, go do something. Go do the thing you would do when you left and find a way to do that in a way, like, and go pursue that dream and because you’ll be happier, you’ll have meaning doing that quit your job at Meta and become a podcaster, you know.

Ryan:

[00:25:40] Well, I did.

Bryan:

[00:25:43] Well,

Ryan:

[00:25:45] it’s interesting you say that because there’s this subreddit, this community that’s all about financial independence, earning enough. I’ll see a very regular post that says, guys, I did it and I don’t know what to do now. Exactly.

Bryan:

[00:26:03] Yeah, yeah, I did it. I did. And also, like, I geared. I mean, I think this is like, this is a problem, by the way, in engineering. Not even in. Even when the goal is. Has meaning, this is a problem. This is what I. This is what I call postpartum in engineering. When you are really focused on shipping something and you’re just like, shipping this thing is just the lens through which you’re doing everything.

[00:26:24] And when we shipped our first rack of oxide, I was very concerned about, like, we are going to have postpartum and there are going to be people who are like, I was working so hard and pulling so hard for so long, and now we shifted and like, now what? And I’m. I. So even when the thing you’ve done is obviously meaningful and you’ve achieved something extraordinary, you’re going to have that when the thing you’ve also achieved is like, well, I can like buy any cup of coffee.

[00:26:47] It’s like the thing I want. Like, now I’m not even, like, I got, like, what am I supposed to do? It’s just easy to see how you people wonder what the meaning of it really is. And then there are some people that just like, snip all the wires and they’re like, well, the meaning of it is to get even more. And then just like, you know, then you get like Larry Ellison or whatever. But like, that’s also not something that people should aspire to.

Ryan:

[00:27:09] Right. I. Well, what would you say? Because I, I do know someone who is genuinely happy just doing nothing. By doing nothing, I just mean, like relaxing in bed, you know, watching their shows, you know, going to. Hanging out with friends and stuff. Yeah, I mean, I think they had the billion dollars. They just do that every day. Yeah, but what you said doesn’t work in that case, because that doesn’t earn anything.

Bryan:

[00:27:36] You know, I actually did know a guy. I knew a guy. I actually did know a guy who did this.

Ryan:

[00:27:39] Who.

Bryan:

[00:27:40] I was thinking about people who made money in the dot com. I knew one of, I, I believe one of Yahoo’s like, first employees, but not because he was like crazy sharp or anything, because he, like, they. They wanted to hire a web surfer and he was like, A pothead who’s like this, like this sounds like the level of work that I’m a. So he gets a job as like a web surfer one at Yahoo, but as like employee number like single digit.

[00:28:06] He gets the equity grant as part of like joining the company. Never gets another equity grant, but makes a king’s ransom because he’s very early at Yahoo, Never promoted above like web server rank one or whatever and was complaining to everybody about how much his job sucked. And it was like you surf the web for a living. Like you’ve somehow managed to like win this insane lottery where you’re getting paid to do like very close to nothing.

[00:28:32] And he like when it started he owned like all of Yahoo in terms of like surfing. Because the way Yahoo worked you. You may be wondering what the hell am I talking about. Yahoo was a curated directory of links. This is again where I sound pre Google. Very much pre Google. This is like pre Google. This is not just pre Google. This is like pre Lycos, pre Hotbot, pre altavista.

Ryan:

[00:28:50] This is.

Bryan:

[00:28:51] This is like basically pre the ability to search the Internet. You would have these curated directory of links and someone needs to go like find this content. And it was like this guy, Yahoo employee number six or whatever he was. And so he originally like owned the entire directory. But he’s like not very good and not very energetic. It was like, you know, stoned all the time. So they kind of like carved.

[00:29:13] And so finally he. I believe at the end before he finally quit, he owned only like snowboarding in Northern California. That’s what it was his like responsibility to. It’s like this is a job. And so that is the kind of person who was. And you would like to think that, that he found something that was actually meaningful to him, but clearly it was not the work, or work for that matter.

00:29:34 — Stack ranking and layoffs

Ryan:

[00:29:34] Right. You said earlier there were these ratings. It was superlative, excellent, excellent, something like that.

Bryan:

[00:29:41] Yeah, yeah, yeah.

Ryan:

[00:29:41] I noticed you didn’t mention there’s. There’s a bad rating, right?

Bryan:

[00:29:44] There was a. Yeah, no there was not. Basically not sun. Did not the. The idea of like a PIP or the. The whole like rank and yank, which is what intel famously did the ranking yanking Yank.

Ryan:

[00:30:00] Is that was. Was that just Stack ranking?

Bryan:

[00:30:03] It was their name for Stack ranking. Yeah, it was ranking Yank. Yeah. Stock ranking is organizational cancer. It is. It is very, very, very bad news when you. Especially if you are going to terminate a bottom end percent that is. That’s death. I really think that is just a wall to wall terrible idea. Because you’ve incentivized people to have dead weight on their teams that they can. That.

Ryan:

[00:30:26] Oh, is that. Oh, so they have fodder.

Bryan:

[00:30:29] Got someone to throw into the wood chipper.

Ryan:

[00:30:31] Wait, so you think that a manager is strategically keeping around.

Bryan:

[00:30:35] Oh, I know so. I mean, that definitely happened to son. For sure. Oh, yeah, for sure. I mean, there were people that were like, why are they still here? Like, they don’t. They don’t not seem to be doing anything. And someone kind of took me aside and like, yeah, that person is in your best interest because, like, do you know what that person’s rating is? It is always, like, good. Like, they. They get a good.

[00:30:54] So you can be an excellent or a superlative. And I’m again, I’m like, does this make sense to you? Like, why are we. Okay, so. No. All sorts of perverse incentives. All sorts of perverse incentives with stack ranking. Was that Stack ranking, fundamentally, stack ranking teaches you that your team are adversaries. And that’s a bad idea. That is. And it’s just not the way I want to operate. Like, my big belief is that teams do extraordinary things and that everybody should want to be serving the team.

[00:31:27] It is the team that wins or loses, the team that succeeds. And I think that stack ranking really operates very much contrary to that.

Ryan:

[00:31:36] You mentioned that Sun. I mean, the stock price went down by 98%.

Bryan:

[00:31:42] Yeah. Trading below our cash at one point.

Ryan:

[00:31:45] I mean, was there not some management at some point saying, hey, we gotta. It’s time to.

Bryan:

[00:31:49] Oh, we did.

Ryan:

[00:31:50] I mean, mass layoffs.

Bryan:

[00:31:52] Oh, no, no. Sun did. I mean, sun resisted layoffs for a little while. Scott was kind of. Famously did not. McNeely famously did not want to lay people off. But that. Which was a mistake. He waited too long for that.

Ryan:

[00:32:02] That.

Bryan:

[00:32:03] But no, once the layoff started, I mean, I think at one point I counted the. The number of layoffs that had. The rounds of layoffs was like 35 layoffs. 35 rounds of layoffs over, like eight years.

Ryan:

[00:32:14] That’s. I mean, it’s more than. For a year.

Bryan:

[00:32:16] Yeah. Yeah, absolutely. Yeah. No, it was. It was brutal.

Ryan:

[00:32:19] That’s. That’s insane.

Bryan:

[00:32:20] You get these, like, small layoffs, big layoffs. I mean, this is actually before the War Act, I think, but you would get. We actually. I mean, this is dark, but we. You could. You could. There was an API effectively to the. Although this is like before the era of rest APIs. But there was a way to get the employee directory programmatically, and so we would run what we called the obits and the obits would run every day and would tell you who is no longer at the company.

[00:32:49] And so you would see these small layoffs that were not actually like you. You would see, you know, a group that was like, oh. Or you would see. And then you would see like, oh, my God, like that’s 1500 people or that’s 3000 in today’s OB bits.

Ryan:

[00:33:01] Was the company aware that you had that access to that data?

Bryan:

[00:33:06] Son, Sun’s strength and weakness was that no one was truly in charge. So I mean, I think it would not have been. I would not have been surprising. But I mean, sun, to its credit, was a pretty transparent company. So I don’t think that we would have. I think that we weren’t violating any kind of policy. We were accessing a. We were accessing the. Org tool, which was the tool they had to show you kind of organizational layouts.

[00:33:31] And we weren’t doing anything untoward. But it was a way for us to know like, what’s happening in the company as the. So no, it was layoff after layoff after layoff after layoff after layoff.

Ryan:

[00:33:40] I mean, 35 layoffs. It almost.

Bryan:

[00:33:44] Yeah, it’s crazy.

Ryan:

[00:33:45] I.

Bryan:

[00:33:45] But that was everybody too. Again, that was not like, like. Oh. I mean, the. It is hard to express the period from the. com bust when, you know, kind of pets. comcraters in the spring of 2000, they kind of ran on momentum until the end of 2000. And at the end of 2000, the dot com bubble truly, truly burst for everybody, for sun included. And 911 did not help. Right. And the economy was. Tech was basically dead for five years.

[00:34:22] And Y Combinator is formed kind of at that. Y Combinator is, I think 2006, I see and starts to form, I think in part because I think get Graham’s perspective on this. But the. I mean, there was a. There was a capital deficit at that point. There were like people that could solve interesting things and there was like no way to start a company to go do it. So the. It was really, really bleak here for many, many years.

Ryan:

[00:34:50] I see slightly off topic, but I was stalking your Twitter and I saw that Paul Graham had blocked you. Do you know is Paul Graham blocked me?

Bryan:

[00:34:59] He did block me. I think he. Yeah, that was a point of pride that I did felt. I felt like I really arrived when Paul Graham blocked me. Yeah, Paul Graham blocked me. Yeah. I definitely feel like I’m on the right side of history on that one.

Ryan:

[00:35:08] Well, what do you think that was for.

Bryan:

[00:35:10] No, I know what that was for. That was for. I think that Paul Graham was defending some of the worst behavior of Richard Stallman. And the worst behavior of Richard Stallman does not age very well. It’s truly, truly some, some pretty gross kind of behavior. And he was defending that and I was attacking him for it and he was blocking me for that. So like, all right, I’ll take that. Yeah, I mean, Paul Graham is complicated for me.

[00:35:36] There are a couple of people like this in Silicon Valley that are complicated in that there are says some things that I really strongly agree with and some things that I really strongly disagree with, often in the same sentence. So often you’re just like, my brain’s getting zapped. Like, okay, this is like wrong, but also not wrong. And so, yeah, you see, he’s a complicated one. There are a couple people like that.

00:36:00 — Why he hated the Oracle acquisition

Ryan:

[00:36:00] I understand that Sun Microsystems was eventually acquired by Oracle. Invaded.

Bryan:

[00:36:05] Yeah.

Ryan:

[00:36:05] And I just want to know why was it so controversial at the time? There’s even a Wikipedia page that says the acquisition of Sun Microsystems and it’s almost like an obituary. And there’s a lot of top level engineers.

Bryan:

[00:36:20] Well, it was dramatic. There’s actually a really interesting SEC filing because it’s a public company that IBM and HP were both jockeying to try to buy sun and each was trying to sabotage the other and each was trying to get their. Because they almost wanted to buy sun punitively. They were both competitors with sun and while they were kind of screwing with one another, Oracle swept in and bought the company.

[00:36:48] So Oracle, that happened really quickly. The actual acquisition itself was necessarily controversial. It took a while to close to the contrary. I mean, there’s nothing really controversial about Oracle because Oracle is what it is. Oracle is just. There’s not a lot of depth to Oracle. Oracle is just an octopus that knows how to feed itself. And the sun was very much not that way. Sun was. There was a level, I wouldn’t call it purism, but there was.

[00:37:22] Sun was always endeavoring to do the right thing by its customers. Sun was endeavoring to solve hard, interesting technical problems that had commercial relevance. And it gave its engineers us tremendous freedom to go do so. And it tended to attract people that wanted to do that, that wanted to be bold. And it was a great place for that reason because it rewarded that kind of technical boldness. So that’s why NFS and Spark and Java and Solaris and then all the technologies that I worked on, all these things came out of Sun.

[00:38:00] There were so many of these seminal technologies that came out of sun because of that kind of that passion and independence. Oracle doesn’t have any of that. That Oracle’s just like. Oracle truly is focused on, really on one thing, namely its own profitability, which is like, I thought when the acquisition happened, I’m like, well, this will be interesting because we’ll get someone with, like, great business savvy.

[00:38:21] Maybe. One of the things that was frustrating me about sun is that sun on. In terms of business execution, sun fell down a bunch of times. And I thought we would have someone in Oracle that would be just better at the execution of a commercial enterprise. As it turns out, I was really overestimating Oracle. Oracle, really, about being able to get effectively a monopoly, a natural monopoly over its own customers, Asphyxiating competition and then raising the rent on its customers.

[00:38:50] Like, that is really what Oracle wanted to go do. And it’s not a company I wanted to work for.

Ryan:

[00:38:56] I see. Because there’s a talk that you gave that where you really.

Bryan:

[00:39:01] I mean, you know, I didn’t even. Yeah, I know. You’re talking. The talk is the Lisa Talk from 2011.

Ryan:

[00:39:07] Yeah. Before you even got to talking about it, you said, I’m gonna try and make it through this slide without crying.

Bryan:

[00:39:14] Oh, that was on the slide.

Ryan:

[00:39:17] Yeah. Yeah. And then I think you were talking about Oracle and you said that they screw customers and lie. I guess you explained the screw customers part. But that’s the lie part of Oracle. Yeah.

Bryan:

[00:39:31] Oracle is. Has made many promises to its own customers that it does not keep. Oracle has not earned the trust of its customers. If you ask customers, do you trust Oracle, the answer to that will be no. And I would kind of let that speak for itself. Making that slide without crying bit was not about Oracle. That was about very much about Sun. And that was the. And Scott McNeely’s epitaph for sun. Kicked butt, had fun, didn’t cheat, loved our customers, changed computing.

Ryan:

[00:40:00] Yeah, you said that.

Bryan:

[00:40:00] Yeah, yeah. That was the bit that, like, it still chokes me up a little bit just because it. It’s true. It was true. And we. When we started Oxide, it’s like, adopted that as our own mission because that. That’s exactly what we wanted to go do at Oxide, and I thought was a very concise distillation of that. And that was just not Oracle. Oracle’s not interested in that. When.

Ryan:

[00:40:24] When a lot of the top engineers. It seems like there’s a. An exodus, a big exodus. Were people confiding with each other? Like, were you talking to other high level engineers, say, hey, we gotta get out of here.

Bryan:

[00:40:34] Oh yeah. Oh yeah. I mean, it’s not even like, I mean, you’re just like, you’re in a burning building. Are people confiding with one another they should leave the burning building? It’s like, no, actually people are just like running for their lives. I mean, you’re just not like, oh, what do you think of the burning building? Like, do you think the building is burning? What do you. And no, I mean, it was pretty clear and there were actually.

[00:40:52] The real true jockeying came from those engineers. Before it actually closed, there were going to be some engineers that would be laid off before it closed. But as the acquisition closed, in other words, there were some people that were sun employees that were not going to be Oracle employees.

Ryan:

[00:41:10] ACWA laid off. Like, they wouldn’t, there wouldn’t be a existing company.

Bryan:

[00:41:14] They just, that’s right. So they would lose their job.

Ryan:

[00:41:18] Okay, okay.

Bryan:

[00:41:18] But they would get like a pretty rich severance package. So you had like an absolute line at the trough for how can I get laid off? Lay me off first. This was not me. This was not us. We were, I, I was, I mean, I mean, at least to the credit of the, like the people lining up at the trough, at least they knew what was going to happen. And there’s still people to this day that are a little resentful of.

[00:41:41] People are like that guy, that guy got laid off. And I, I mean, I quit because I could not work there in good conscience. After 45 days, that guy got laid off and got a year of salary or whatever it was. But we know I, I was not that I was. We were at this fishworks Group in San Francisco and I was really optimistic that. I’m really optimistic, somewhat optimistic that we could combine the best of both companies or what I perceive to be the best of both companies.

[00:42:06] And as it turns out, that’s not the case.

Ryan:

[00:42:07] So what’d you see? Because you left really quick. So what did you see?

Bryan:

[00:42:11] That, oh, it was really clear to me that this was a very, very different company. I mean, Oracle gave me like, I decided like, I can’t work here anymore. And the time between I decided that and I actually left, which was only like 30 days more, a little longer than that, maybe 45 days. Oracle gave me like four other reasons to be like, oh my God, I just can’t, I can’t stomach working here. I mean, and I mean it was a bunch of things.

[00:42:37] One, it was sun for all of its faults and there were many, many faults of Sun. Sun had the trust of its customers. Our customers actually were rooting for us at Sun. Our customers wanted us to prevail. Why? Because we actually, they. They trusted us. And they. Even when it wasn’t in our own best interests, like, we would. We would be straight with our customers. And I think that that was. We did not lie to our customers.

[00:43:03] Right. And that was something that our customers found very, very appealing. That was not Oracle. Oracle does not have the trust of its customers. And I realized something very important about myself. I can’t work for a company that has that kind of profound distrust. I mean, in terms of, like, we’re talking about meaning. That so eroded my own sense of meaning. I felt ashamed. I felt ashamed to work there, and I had never felt that.

[00:43:27] I’ve never. You know, for Sun, I was embarrassed from time to time, but never ashamed. And with Oracle, I was ashamed.

Ryan:

[00:43:35] You mentioned people were eager to get themselves laid off. How do you even do that?

Bryan:

[00:43:41] You just like, oh, you tell your manager, look, there’s a list. Put me on it.

Ryan:

[00:43:45] Okay.

Bryan:

[00:43:45] Yeah. Oh, there is a list. Okay. Who else is on it? Like, I want to be on it. I mean, just like, all the ways that you. I mean, it’s not like. Like, you don’t fill out a web form. I mean, you’re obviously, like, you know, you’re. You’re a courtier kind of whispering about trying to figure out, like, what’s the scuttlebutt who’s gonna get laid off.

Ryan:

[00:44:00] I thought you’d have to, like, really, you know, underperform and like.

Bryan:

[00:44:03] No, no, no, no, no. I. I think that they were. No, it was all about, like, hey, I, I’m, I. I have no value to Oracle. Could you. I mean, it was much more like, oracle’s not interested in what I’m doing. Right? And then you get like, Oracle is interested. You’re like,

00:44:19 — Why Bezos is the apex predator of capitalism

Ryan:

[00:44:19] So, okay, so I get why you left Oracle. And obviously sun was acquired. And so the next place you worked at was Joyent. And I understand that this company competed with aws.

Bryan:

[00:44:32] That’s right. Yeah.

Ryan:

[00:44:33] And I listened to some of the conversation you already had, and, yeah, interesting quote. You said bezos is the apex predator of capitalism.

Bryan:

[00:44:42] Yeah.

Ryan:

[00:44:43] What makes you say that?

Bryan:

[00:44:45] Just Amazon was. Amazon at its height was such. So relentless on execution. And what made Bezos really, really good is that. And I would say, like. And when I say it’s height, I mean Amazon hits the mother load. And they hit the mother load, not because they’re lucky. There’s luck involved. You know, they allow S3 to be developed, right? They develop EC2, then they, they realize what they’re onto and they’re on cloud computing before anyone has figured it out.

[00:45:17] And the, the, the, the, those reinvents like from 2010 to say 2015 were relentless. Every re invent was a price cut. It was price cut and it was a bunch of new services. And if you’re a customer, you’re like, this is awesome. I love this. This is great. Like, why would I not. I mean, so it was so. And that’s what I mean by the apex predator. Because he was on something that was wildly profitable.

[00:45:45] He pulled off a, just an absolute move of moves in that Amazon did not break out their revenue. So it’s just like Amazon. It’s like you got the dot com side, you have aws. They’re not breaking out any of that. They’re just like, literally like this is how much Amazon’s making. Analysts would ask in the call. They’re just like, yeah, we’re not talking about that. Which apparently you can get away with as a public company.

[00:46:07] And so they actually created this idea, especially with the relentless price cuts. People are like, oh, the cloud’s a terrible business. Like no one, no one should get into the cloud because it’s a terrible business. And so people didn’t. And meanwhile, like what we knew because we did compete with Amazon, we did have a public cloud and we knew it’s like, no, the actual, the margins on this thing are great.

[00:46:26] Like this is actually a great business. And Amazon, and Amazon is executing very, very, very, very well. And they did extraordinarily well in those years. And I mean they are still. Obviously it’s a, it’s a, it’s a very profitable company, but they have am. Has. Is not what it was in, in those years. I not sure when the last price cut was announced at Re Invent, but it’s been a while. It is not that.

[00:46:53] That’s not what you get at re invent. You don’t get the kind of, the new services that you can’t live without. So it’s, it’s a different company.

Ryan:

[00:47:00] AWS was, was early and they already had I guess a dominant market share.

Bryan:

[00:47:06] Yeah.

Ryan:

[00:47:07] Why would Jeff Bezos continue to just, just like ruthlessly keep going and keep going instead of milking it?

Bryan:

[00:47:14] Because he is not merely a predator, he is an apex predator in capitalism. This is what makes him so good. He’s like, I’m not gonna. No. Like I’m gonna I’m gonna press my advantage. Like, I’m gonna make it so no one can compete with me. And I’m gonna make it so no one wants to get into this business. And I’m gonna do it honestly, like the right way. I’m gonna do it by like giving a great product at a reasonable price.

[00:47:37] It’s like, damn, yeah, that’s good. That’s a win, win. That’s a win win. And it was essential for innovation during that era of companies that were cloud born. And I mean, it was really. And it was very, very hard to compete with Amazon, I’ll tell you that. You had to really like, find a lane and we found lanes where we could go compete with Amazon, but it was brutal. Their execution was extraordinary at this company.

00:48:04 — Differences between CTO and VP

Ryan:

[00:48:04] I saw that you start out as a VP and then you transition to CTO at some point.

Bryan:

[00:48:09] Yeah.

Ryan:

[00:48:10] What’s the difference in the roles?

Bryan:

[00:48:12] That’s a great question. I actually gave a talk on this with Joyent CTO at the time on the contrast between VP of Engineering and cto. And I’m not sure how much I buy the difference now. I mean, I think in both capacities, you’re serving as an engineering leader and I think that, you know, I guess you can argue that a CTO is more outward looking for.

Ryan:

[00:48:34] Sure.

Bryan:

[00:48:34] Chief Travel Officer is kind of the pejorative for it. You’re on the road talking to customers a lot. You are VP of Engineering may be more focused on some of the necessary mechanics that you have in an engineering organization. For me personally, I mean, they brought me in as a VP of Engineering because they frankly already had a cto. So I mean, like, I, again, it’s just like the grade I was at, at, at Sun.

[00:48:59] I didn’t particularly care. What I did care about is when the, you know, the CTO had left and then another CTO that we had CTO who had been a founder of the company and when he left, so there was kind of like the, the, the empty CTO position. I didn’t care. What I did care was that we didn’t hire externally for that because I did so that I did care about a lot about, like, look, if there’s going to be a cto, I’m happy to be the VP of Engineering in perpetuity.

[00:49:24] But like, like, I, I would rather like if there’s going to be a cto, I would like to not hire externally for that, please. Or let me be involved in that anyway.

Ryan:

[00:49:33] Why not hire externally?

Bryan:

[00:49:35] I just didn’t want to deal with like having to bring in. I didn’t need that. That’s not what we needed at that time. What, what we needed at, at that time especially? Well, we had, we got rid of the CEO and we, we really were, we, we needed really a terrific CEO, which we, which we later got.

00:49:58 — Starting his own company

Ryan:

[00:49:53] So then you started oxide.

Bryan:

[00:49:56] Started oxide in 2019. Yeah.

Ryan:

[00:49:58] And what’s the story behind you starting your own company?

Bryan:

[00:50:01] Well, I knew again, I kind of had in my mind that like, I want to start a company. And the thing that I knew is I want to start a company with Steve. Steve and I had worked together at that point. I mean, Steve was the. When I went to Joyent, I had not talked to Steve prior to going to Joyent and I was going to Joyent in part because I was running away from Oracle and it was going to allow me to hire folks and so on.

[00:50:27] But I didn’t talk to Steve until after I came to Joy in and I’m like, this guy is amazing. And Steve and I just worked very closely together. Steve came up on the go to market side, on the sales side. And I knew, I’m like, I definitely want to start a company with this guy. And fortunately he felt the same. So we both felt like, all right, whatever we do next, we do it together. Other and so now it’s like, all right, well now what?

[00:50:52] Now we got to figure out what we want to go do. And we had these long walks in the city because we were working in San Francisco and we had so many bad ideas. I mean, it was just, it was just basically one bad idea after another. So then we’re trying to come up with ideas that like, we’re a better match for the kinds of things that people would fund. And we recommend people not do this. It’s understandable.

[00:51:15] I did it. I understand. But we’re trying to like come up with, with things that people would fund and coming up with things that were just like not in our heart. You know what I mean? Like, this is like, okay, I could do this, but like really, I mean, it’d be fun to do it with Steve, but just doesn’t feel like that’s not gonna. I wanna do something that is like that is this next long chapter of my career.

[00:51:35] And so as we’re kind of struggling with that and I was trying to figure out like, okay, we actually need to start talking to venture capitalists at some point. We need to like start like get this ball rolling and understand like I and I had always known venture capitalists and had gotten lunch with them over the years, but had not really, like, you know, kind of gone deep and known in the venture capitalists associated with Joyent for sure.

[00:51:57] And so I got, I was reminding myself of the email address of one VC in particular, actually a very famous vc, but someone I had known since early, early, early in his career. And he and I had just gotten lunch periodically through our careers from when he was first in venturing and I was in engineering. And we, it’s like. And he had kind of become pretty famous. And I was just wanting to remind myself of the email address and the last email he had sent to me, which was maybe, you know, 18 months prior, was like, hey, Brian, really enjoy getting lunch with you today.

[00:52:26] I just want to remind you, I will fund literally anything you put in front of me. I think I’m like, wow. And I was like, Steve, you know, remind myself this email of this guy’s email address. He said he will fund literally anything we put in front of him. So we should just go big. We should do the thing that is in our heart. And Steve’s like, what do you mean? I’m like, we should build the computer that we want to build.

[00:52:57] We should build a Rack scale machine. And he’s like, yeah, that’d be amazing. Do you think we can get that funded? And I’m like, I, yeah, yeah, I think so. And you know, that was very much in our heart. That was what we had lived, that he had lived, that Steve had been at Dell prior to Joyent. He’d been at Dell for a decade. I’d been obviously at sun for 14 years. We, and then together at Joyent for another decade.

[00:53:25] And this is we. This was truly in our marrow. And we found like, okay, like we can go. And we, as we started talking VCs about this, about building, and we again envisioned Rack scale design. We wanted to do our own board design, do our own switch, do our DC bus far basis line, do all of our own software, build the machine that we ourselves wish we could have had at Samsung after Samsung acquired Joyent.

[00:53:53] And what was very catalyzing actually was going to. You were at what would have been Facebook at the time, before they renamed it, and going to their open computer, going to the Open Compute Summit and looking at like Tioga Pass, and you’re like, what is this, like Tioga Pass? Like, I mean, it was like I would liken it to discovering Unix as an undergrad when I had been living with dos and Be like, what is this?

[00:54:17] And you go, look at Tioga Pass. You’re like, this is what a computer can be like. This is gorgeous. This is amazing. We gotta go build this for the enterprise market. And what we discovered is this was very contrarian, but things that are contrarian are attractive to venture. So we would get this thing where people would actually be pretty interested in it. And, I mean, I don’t have to tell you how the story ends, but we went back, we’d actually talked to a bunch of venture capitalists before.

[00:54:46] I went back to that same VC that sent me the email. And I get maybe 45 seconds into describing the problem we want to go solve at Oxide. Brian, stop, stop, stop. If you are talking about starting a computer company, I want nothing to do with it. And I’m like, you know, it’s funny. You sent me an email years ago that said you would fund literally anything I put in front of you. And he said, doesn’t sound like me.

[00:55:13] I’m like, well, okay, what do you want me to do with that one? I’m like, you sent me the email. It’s like, is your admin writing all your emails? Did you have your kid write the email? I. I don’t know what to do with that one. Like, you did send me the email. Are you calling me a liar? I mean. And I’m like, all right, fine. And it’s like, fine. And I. We go over the phone, I go to hang up on him, and of course, like, VCs, this is just like, it’s in the animal brain, right?

[00:55:32] If you. You go to hang up on a vc, they’ll be like, wait, wait, wait, wait, wait, wait, wait, wait. I was like, oh, okay. He’s like, wait, wait, wait, wait. I’m like, fine, got it. He’s like, wait, wait, wait, wait, wait, wait. I’m not going to invest. And I’m like, I know you’re not going to invest. And I knew the reason I knew he wasn’t going to invest is he’s had some big zeros in this department.

[00:55:49] And when a VC has had zeros in something that looks close to what you’re doing, that are like, I want nothing to do that again. Like, nope, nope, nope, nope, nope, nope. Was in a bad relationship with one of those. Don’t want to do that again. And in part, in their defense, it’s often because, like, actually, I understand this problem a lot better, and now I understand all the headwinds, and now I can.

[00:56:07] I mean, in some ways, like, you need VCs to be. And they need themselves to be like, like naive at some level. Optimistic is a. Be a better way of phrasing it where they can like envision the world as it could be, as opposed to getting all mired in the way the world is. You gotta be kind of blind to the odds to a certain degree. And I knew that he was not blind to the odds because he’d had two big zeros.

[00:56:28] And so I knew that he was gonna be like. And one zero that like looked a lot like oxide. And so he’s like, look, I’m not gonna find it. I’m like, again, I knew that, that. But I do want to help you. Like, okay, it’s great. And I always tell people, like, when you are. VCs will ask this a lot, especially a VC that’s like, where you’re not a fit for them, you’re not a portfolio fit for them, you’re not a thesis fit, whatever it is, you’re not a stage fit.

[00:56:53] If they like you, they’ll be like, look, I’m not going to invest, but how can I help? And you always want to have an answer to that question of how you can help. And I had an. I knew how he could help. I had a very concrete idea. It’s like one of those zeros. I want to talk to him, I want to talk to the founders. I want to understand everything that went wrong. And I did. He’s like, okay, that I can do.

[00:57:13] He made the intro and it was really, really interesting and like the mistake that that company had made. And again had a thesis that looked like oxide. But they didn’t raise a lot of cat. They raised kind of arguably too little capital. They got something working that’s smaller than what we built at oxide Minimum viable product at oxide is a rack. It’s big. Took us three years, three plus years to develop.

[00:57:35] They developed something a little bit faster, but it was much smaller. It was basically, you could view it as like Nitro, circa like 2013. Nitro from the enterprise acquisition at AWS does a lot of really important offload for aws. So you could view it as like very early nitro. But it didn’t really have a market. But they had a customer and they got a customer and they’re like, great. And the customer was happy and they wanted to use it in particular on their active directory servers.

[00:58:01] Great. We have product market fit. Raised a bunch of money, hired a huge go to market team. Problem is that customer huge bank only wanted to run it on like their six active directory servers. So they were gonna buy like quantity six. And this is one of the world’s largest banks. You’re like, oh, so you’re like, how many world’s largest banks are there? Like, well, they’re like a couple of others, but like we can sell like you know, two dozen of these things, like we’ve got it.

[00:58:28] And. And by the time they realized that they had a lot of mouths to feed on the go to market side. And that wasn’t a zero. They were acquired, but they were acquired in a way that left the founder extremely bitter and really felt like the VCs had pumped too much capital in him at the wrong time, had gotten him to do the wrong thing, and was then trapped at the acquiring company and was really not happy about it.

[00:58:56] He was also the one. So I was describing what we wanted to do at Oxide. He’s like, that is a suicide mission. And I just remember writing down a little notebook, almost like writing down suicide mission, kind of underlining it, like, okay, we’re on a suicide mission. That’s fun.

Ryan:

[00:59:08] When I think of Rack scale compute, especially these days, I think of racks of GPUs more than CPUs. Is that something that you’re building?

Bryan:

[00:59:19] Yeah, right. A very reasonable question. And when we set out, we’re like, we, I mean again, set out in 2019. The GPGPU was around obviously and important, but that’s not what we were focused on. We were really focused on general purpose compute, General purpose compute, general purpose storage, general purpose networking. And our belief had always been like, that’s what we actually want. There is so much to go build there and go differentiate there.

[00:59:44] And of course along the way people are like, what about an accelerator? Like what about an accelerator gpu? And the problem is that the way we want to build systems, we want to really build systems from first principles where we have components that have transparency at that hardware software interface and we want to write that lowest layer of software. We are the company, ultimately we are building a hardware software co design product.

[01:00:06] In order to be able to do that, you need to be able to write very low level software. The problem is that’s really not compatible with Nvidia. Nvidia is a pretty proprietary company, executes well, but a very proprietary company company. And what that meant is there are effectively two doors for Oxide. One is labeled compete with Nvidia and the other is labeled partner with Nvidia. And I didn’t want to do either of those things.

[01:00:27] Have not wanted to do either of those things because I certainly don’t want to compete with Nvidia and I think that that’s getting more plausible now. But we can’t partner because we just have a very different view of how systems should be built. And Nvidia wants to like their view is like we should own the whole stack. Like forget you, whoever you are. So like, okay, that’s fine, we’re going to focus on general purpose cpu.

[01:00:50] We got plenty to do over here. I think the thing that’s been interesting and a bit surprising is that because the GPU landscape is so cluttered, I mean you’ve got this very aggressive, the very well executed company in terms of Nvidia, you’ve got a lot of competitors around it, it’s gory right over there, there to get. And meanwhile on the general purpose CPU side, it’s like hp, Dell, Super Micro, the same companies that are, are doing the same kind of junk honestly that they have been doing.

[01:01:22] And so we are by our lonesomes in terms of a hardware software product over there. So over and over and over again we have had people come to oxide that we think like no, no, we’re not a fit for you because you like, you do a lot of different GPU. You’ve got a ton of GPUs. Like you company famously have a lot of GPUs. No, no, no, we do have a lot of GPUs. We also have a lot of CPU as it turns out, because certainly the emergent AI workloads, not just AI workloads, but certainly AI workloads, but high performance computing workloads.

[01:01:54] There’s a lot of general purpose CPU that’s attached to the special purpose compute. You know when you’re sitting there on ChatGPT and it’s surfing the web, you got the little spinny saying, it’s surfing the web. That is not a GPU that’s surfing the web, that is a CPU that’s surfing the web. And the CPU is really, really important. So for us we are more focused than ever on the general purpose cpu.

[01:02:15] And there is a ton to go do there, fortunately to get this product to be where we believe it can be. And I think we will do an accelerator at some point. But I’ve been kind of saying it’s like 18 months away for a while or 18 months that we would really start thinking about it. And again, I know we’ll do it in the limit, but boy, not in the foreseeable future.

01:02:37 — Grilling him on his past

Ryan:

[01:02:37] Coming to the end. I want to ask you some career reflections.

Bryan:

[01:02:42] Yeah, you bet.

Ryan:

[01:02:42] Some kind of just all over the place type of questions. You wrote a tweet a while ago. I thought it was a really interesting idea. Which was. You said it would be interesting to have a conference called In Retrospect, where presenters revisit talks that they’ve given prior and describe how their thinking has evolved since. And I pulled a bunch of stuff that you’ve, I guess, written or said in the past.

[01:03:08] I’m curious if your perspective has evolved since then. So we’ll go through each of those.

Bryan:

[01:03:13] Sure.

Ryan:

[01:03:14] So, first one, which is actually really famous, when I saw. I kind of did a double take. So in. In 1996, as a new grad, there’s this. I don’t even know what Usenet was. I had to do research, actually. But there’s a guy who. He writes this long technical response.

Bryan:

[01:03:33] David S. Miller.

Ryan:

[01:03:34] Yes. And part of it, too. It’s not nice either. I saw there’s a line in there, it says Linux is lightweight, Solaris is a pig. Which Solaris is what I guess sun was.

Bryan:

[01:03:46] Yes.

Ryan:

[01:03:46] And then he writes this long thing and you reply with. With just a few words, you say, yeah. Have you ever kissed a girl?

Bryan:

[01:03:51] Yeah.

Ryan:

[01:03:53] Oh, first of all, I want to know the context behind it. And then also knowing what you know now.

Bryan:

[01:03:56] Oh, definitely in the regret department, if that’s what that’s asking. Like, yeah, I’ve got very few regrets in my career, but, like, you can put that one, like, pretty firmly in the regret column. Yeah, no, that. That was, that was. And also had no idea that this was going to live in perpetuity, that. I mean, if you could have told me In, I think 1997 is maybe when I posted. Maybe it was.

[01:04:17] It was 96. It was 96. 97. Certainly. Like, I’m like 22. Like, I’m. I’m very young. If you had told me, like, oh, by the way, 30 years in the future you’re gonna be asked about this, I’d be like, what the hell? No, no, trust me. It’s like, the world gets weird. You’re gonna be asked about this. So it was actually a. Okay, this is. I’m not defending it. I just want to be sure that. I want to be clear that, like, it was a mistake.

[01:04:46] It was actually a reference to a Saturday Night Live sketch. So there’s an SNL sketch that is from an era of Saturday Night Live that it’s like you can’t even Find the video. But they have. So the. William Shatner is guest starring on Saturday Night Live. And the skit is that William Shatner is at a Trekkie convention. And the Trekkies are asking him all of these questions. And they’re asking questions.

[01:05:14] And of course, like. Like, you know, in episode. You know, this season of this episode, you know what. What was the combination on the safe? He’s like, what? I don’t know that. I don’t.

[01:05:22] I don’t know that. Why would I know that? Like, I don’t. That’s not even. And he’s like, these two people are kind of arguing themselves and they’re asking him questions that are like this. That are all about, like the. The kind of. The canon of Star Trek. And then he’s like, hey, can I just say something? Get a life, people. You. You. Have you ever kissed a girl? That was the. That was. Where you going?

[01:05:40] To a 30. Going to John Lovitz when. Like, Vulcaneers. John Lovitz. Yeah. Yeah. It’s like Lost history. Why am I doing this? And kind of like looks down at himself and so, like, it was actually like an obscure Saturday Night Live reference, which again, like, I’m not. It doesn’t make it any better. The. Yeah, it was that. That was. That’s definitely in the. In the. The regret department.

Ryan:

[01:06:03] What did he say back to that?

Bryan:

[01:06:06] Oh, he had a whole lot to say about. Back to that. And I actually did have a longer post, kind of taking apart what he had said about the. About, like. All right, like, a lot of what you’ve said here is actually wrong. And so, like, really going through kind of point by point, I think, you know, I’ve actually never. I’ve never met him, never talked about this very talented guy. I think he actually.

[01:06:29] I actually did read. I think it was in the Rebel book about Linux. Remember him reading that? He’s like, yeah, I kind of, like, was shooting my mouth off and a son engineer kind of put me back in my place. And I’m like, man, if that is his read on it, he’s being very generous to me. So I’d like to believe that maybe he and I both regretted a little bit. We were both like a little, you know, a little young and excitable.

[01:06:52] But, yeah, that was definitely. That was a life lesson. I would say that history forgot about that, though.

Ryan:

[01:06:58] So on another tweet.

Bryan:

[01:06:59] Yeah, sorry. Yeah, here we go. This is great.

Ryan:

[01:07:02] This is.

Bryan:

[01:07:02] We’re. This is like the cleanse. Yeah.

Ryan:

[01:07:05] Okay. This is in 2022. You wrote a tweet. You said, if you’re tempted to blame a team for a startup’s failure, please don’t. Success is often due to a great team, but failure is almost always due to bad leadership.

Bryan:

[01:07:18] I, I wrote that. That’s a good one. That’s a good. I don’t remember writing that. That’s it. I, I agree with that guy. He’s on to something. I guess the question, what was I responding to? I wonder. I must have been something. Must have been something that day on the Internet or some weather on the Internet. I’m subtweeting Paul Graham there somewhere. I think that must have been it.

Ryan:

[01:07:33] But 2022, I’m curious, do you think that’s true for. For sun because sun ultimately failed.

Bryan:

[01:07:40] So, okay, I don’t agree that sun failed. I don’t agree that sun failed because sun, again, sun is founded in 1983 and is invaded in. In 2008, 2009. That’s a good run. That’s a really good run. Sun was a public company. Sun was in the Fortune 200. Lots of people, like kids went to college because their, their parents were able to work for sun. And you got, you know, so I, I don’t view. I do not view sun as a failure.

[01:08:07] Son is like, sun did not, I mean, arguably did not or not maybe inarguably did not succeed to the scope of its own ambition. But sun, to me, is not a failure. Sun is a success.

Ryan:

[01:08:19] Was its collapse at the end, I guess, changeable in hindsight, with different leadership?

Bryan:

[01:08:26] I think so. I think that there are. I mean, this is a classic power game of, like, why did sun ultimately not survive as an independent entity? And why is that? I think there were a bunch of reasons. I think that there’s a degree to which sun got very strung out on the very high margins during the dotcom boom and never quite, quite, like, got off of that. Never like, we embraced x86 too late. We kind of thought of ourselves.

[01:08:57] I mean, the company itself became fractured. The layoffs didn’t help. I mean, I think we. At Fishworks, we were developing a storage appliance. I felt that we could have been an example of an exemplar of what the kinds of products I felt sun could develop, an independent sun could develop. But it was going to be. There’s a lot of like, like stuff that needed to be changed for that, and you needed leadership that really was very, very interested in that.

[01:09:22] And it’s like, that just wasn’t like that. That’s not what we had. And in hindsight, it was probably time for a change. It was, you know, it’s like kind of the way a forest fire in a normal healthy forest fires is kind of a part of the life cycle of a forest and you need that to have, to have rebirth. And I think that, that, I mean, ultimately, I think that that son had succeeded, but had also run its course.

[01:09:50] And it was, it was time for.

Ryan:

[01:09:52] Had its time.

Bryan:

[01:09:53] It had time. Absolutely had its time.

Ryan:

[01:09:56] Okay. And that next past take of yours you wrote in 2022, perhaps this shouldn’t have been surprising, but Musk has absolutely no idea what he’s doing. And so. And this is about the takeover of Twitter. And yeah, I actually don’t even know what’s going on Twitter because it’s private.

Bryan:

[01:10:12] Excuse me, I’ll. I will thank you to not refer to SpaceX that way.

Ryan:

[01:10:15] Oh, right.

Bryan:

[01:10:17] It has been acquired by Xai and then Xai has been rolled into SpaceX. So like we’re now, I guess Gwen Shotwell now runs. Runs Twitter.

Ryan:

[01:10:24] So I guess. Do, do you still agree that it’s. It’s run poorly?

Bryan:

[01:10:28] Oh, God, yes. Yes. Yeah, I agree that. I mean, because it. Yeah, yeah, definitely. It’s so. I mean, yes, yes. I mean, yes. I really, like, I literally feel dirty being specific about that. But when, I mean, when there’s a lot of. There’s rampant bad behavior on Twitter. Community notes. Yes, community notes. Great. Everything else, pretty much a tire fire.

Ryan:

[01:10:56] What’s your number one thing that this tire fire.

Bryan:

[01:10:59] The number one thing of tire fire is that they all will tell you this. The reason that we at Oxide don’t engage on Twitter. I can’t have an Oxide tweet that is sitting next to some of the tweets that I’ve seen. Oh, you see how it’s like the level of racism, bluntly, I mean, crazy racism. Crazy crazy racism. Crazy. Anti Semitism crazy and crazy. The anti Islam, just crazy hate crazy levels. The kinds of things that you literally could not say.

[01:11:38] And they’re like, oh, well, it’s free speech. It’s like it’s not free speech. It’s so deeply offensive and reflective. It’s like, it’s so deeply offensive that I don’t want my content to be anywhere near it. I don’t want someone to be looking at that chat and looking at my content. I’m sorry, I’m just not going to do that.

01:11:57 — AI boom and bust advice

Ryan:

[01:11:57] You live through a bunch of booms and busts and honestly, I don’t even know if we’re in a boom or a bust right now, I mean, AI is going crazy and there’s all these layoffs.

Bryan:

[01:12:09] Crazy. Yeah, that’s a good point. Yeah.

Ryan:

[01:12:11] What advice would you give, given your experience through the booms and busts for people who are in today’s market?

Bryan:

[01:12:18] Yeah, yeah. And I would say that some of this I do think is endemic. I mean, when I first moved out here, I’m like, oh, this is great. My. My grandfather was a petroleum engineer, and so I kind of grew up with stories of, like, plants that were gonna be built and then shut down or pipelines that were gonna be built and then shut down, like, everything. Tracking the price of oil. Right. Very oil. The oil patch is very boom and bust.

[01:12:35] And I’m like, oh, this is great. I’m in software. Like, I’m immune from booms and bust. I remember thinking this, like, you know, you’re just like. And of course, looking back at it now, you’re like, oh, my God.

[01:12:45] No, no. We are. They are a bit endemic. And they’re endemic for reasons that are somewhat endearing in that, like, we get so optimistic that we kind of get ahead of ourselves from an optimism perspective. We also get ahead of ourselves from a pessimism perspective. And the. And what I would say is, like, you gotta be really careful about listening to other people. People will tell you that this is gonna be the future or that thing is dead.

[01:13:13] And you gotta be like, just be your own judge. I had people tell me that operating systems are done in 1996. I’m really glad I didn’t listen to them. Really glad I didn’t listen to them. People tell us, you can’t start a computer company in 2019. I’m really glad we didn’t listen to them. VC firms, we only fund SaaS. Those VC firms are like, well, it’s like SaaS is struggling right now. But I would also say similarly, like, if SaaS is in your heart, as an example, where people are like, right now, people are like, SaaS is going to be the Gen AI.

[01:13:48] The LLM assisted coding is going to really put a squeeze on these SaaS companies. And I think there’s definitely truth to that. But if one’s heart is in that, you should ignore the pessimism or treat the pessimism and the optimism with a grain of salt. Be your own judge and be true to what you want to do. Don’t do the things that like, well, I’m doing this because it’s like a hot space. It’s like, you should do this because I think it’s.

[01:14:18] And there are plenty of people for good reason. I mean, these things are amazing. I mean, where we are with respect to these LLMs is just bonkers. And you could easily say, I want to be very. Find a great deal of intrinsic appeal to that. But that’s the reason that you should be going into these systems is because you’re thinking like, no, no, this is. This is to. To quote Steve Jobs, this is the dent I want to kick in the universe.

01:14:41 — When he was happiest in his career

Ryan:

[01:14:41] I notice your career, almost everything’s driven from fulfillment and intrinsic motivation. Is there a time in your career that you look at and say, that’s the happiest time of my career?

Bryan:

[01:14:54] Yeah, that’s a great question. We ask this at Oxide. We ask you, when have you been happiest and why? For exactly that reason. And I would say that, like, it’s not necessarily an era. It is times that I’ve been happy. I mean, bluntly, I’m pretty happy right now. Oxide’s great. The moments for me, and this was an important kind of question for me to reflect on as we were starting Oxide, and this is actually due to a friend of mine who had been at a startup, and he and I were taking a walk in San Francisco as kind of Steve and I are talking about bad ideas.

[01:15:29] And he was like, you really need to answer for yourself, why do you want to do this? What is drawing you to start a company? And because it wasn’t financial return for me, and people should not start a company for financial return. That’s just not a good. That’s not good life advice. But what was drawing. Well, the thing that’s drawing me, actually, is that the moments that have been the happiest have been working on an incredible team and being on a team that.

[01:15:55] Where everyone individually is like, I don’t think we can pull this off. Like, this is actually too difficult to pull off. And then everybody works together, compliments one another, and you pull it off. Man, that feeling is extraordinary. And the times that I’d had that I’d had it with D Trace, I had had it at Fishworks, I’d had it at Joyent, a couple of times with lx, with Triton. Those times have been like, oh, that’s what’s amazing to me.

[01:16:24] And so, I mean, in many ways, like, the bedrock of Oxide is like, what if we found a company around. Part of the reason we were appealing, that larger problem was appealing to us is like, we’re going to be able to Attract an extraordinary team. And we have a team that is, it is so uplifting to be on a team. I’ve always said that the organizational model for, for Oxide is a heist movie. You know, you’ve got your safe cracker, you’ve got your getaway driver, your demolitions expert, right?

[01:16:49] Like, heist movies are great because of that. And like, because it’s the, it’s the group coming together. Everyone’s skill set’s kind of coming in at exactly the right moment. And man, I love that. I love that so much. And Steve loves that. I mean, that’s part of our shared bond, is he and I are both very, very team oriented and we built a company around that. And it’s extraordinary. It is truly, truly extraordinary.

[01:17:13] It’s why we’ve been able to pull off what we’ve been able to pull off with the small team that we’ve got. I think people are kind of shocked at how small a team that we’ve got, given what we’ve been able to pull off.

01:17:22 — Top career regret

Ryan:

[01:17:22] When you look back on your whole career, is there a particular top regret that you have?

Bryan:

[01:17:28] I mean, I made an extraordinarily bad hire at Joyent. I think the worst hire in human history. Many people in Silicon Valley will say, like, well, that can’t be the worst hire in human history, because I feel I have made the worst hire in human history. And as I tell people, like, look, I’m happy to give up the crown, but you should know that my guy presented himself under an assumed name and just got off parole for violent felonies from San Quentin.

[01:17:50] And it’s not what made him a bad employee. And usually people are like, no, no, no, I think you’ve made the worst hireful time. Mike, thank you, thank you very much. That was a bad experience. It was also very eye opening because everything I was doing about hiring was wrong. And we, after we rectified that situation and got him out, we stripped hiring to the studs and really rethought hiring from the, from the very first principles.

[01:18:15] That hiring process, a very writing intensive process, one that really gets to intrinsic motivation, became the hiring process at Oxide. And there was a moment where I’m like, oh my God, like this extraordinary team at Oxide very much, I think, related absolutely to the hiring process that we have. This hiring process that I built because I made the worst hire of all time. I needed that guy.

Ryan:

[01:18:38] So you don’t regret, regret it?

Bryan:

[01:18:39] I don’t regret it. I don’t regret it. In fact, I’m even Like to kind of go back into the time machine and remove it. I think oxide, I don’t think no oxide makes it because I would have continued to hire the way I was hiring, which was naively, it was hiring not rigorously, it was not selecting people based on their values. So no, I absolutely needed it. So I think, and I view that way of a lot of things that have maybe not gone the right way, way, but all of those failures were really, really important and it’s very hard to go back and you don’t want to take those away.

[01:19:14] I mean, if I go to the time machine, maybe I would take away the Usenet post and maybe that wouldn’t have any long lasting consequences. But boy, that’s about it.

01:19:21 — Advice for younger self

Ryan:

[01:19:21] Awesome. And then last question is, if you go back to the beginning of your career knowing everything you know now, what advice would you give yourself?

Bryan:

[01:19:29] Jesus, don’t fuck it up. I would just think because I feel like so lucky. So I’ve been so lucky in so many regards. I’ve been in the right place at the right time, so many different times over and I’ve trusted my gut when sometimes that was not the thing that, when that was kind of a contrary thing to do. I mean, I would be scared to give myself advice because I would be worried that I would somehow tamper with what I feel has been an extraordinarily lucky career where I’ve been able to do so much with so many extraordinary people.

[01:20:09] I’ve been blessed with so many incredible colleagues and that has been so essential for everything that I’ve done. And I wouldn’t want to do anything to endanger that. So I’d be going back to my past self. I’d be like, I got nothing to say. Just, yeah, I don’t want to screw anything up. Like I just. Because I, I feel that I feel really, really lucky. And it wasn’t always because of. Because again, it’s like I, I made mistakes along the way, but the mistakes became load bearing and important and I wouldn’t want to.

[01:20:37] I wouldn’t want to not make those mistakes. Those mistakes were really important.

Ryan:

[01:20:41] Well, doesn’t get better than that. And thank you for your time today. Really appreciate it.

Bryan:

[01:20:45] Absolutely. Thank you for the thoughtful questions, really great conversation. And thanks for doing the. The hall of Shame here on, on the past tweets. I. That was a lot of fun.

Ryan:

[01:20:53] Yeah. Awesome. Thanks so much.

Bryan:

[01:20:55] Thank you.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:01:25 - The Bakery Algorithm

00:08:28 - Experiences with Dijkstra

00:14:44 - His most cited paper

00:23:26 - The “Byzantine Generals” problem

00:38:05 - The Paxos Algorithm

00:46:57 - Paxos vs Raft Algorithm

00:51:26 - Building LaTeX

00:54:45 - Why writing improves your thinking

01:00:21 - Why he wasn’t an academic

01:02:08 - Grand theory of concurrency

01:07:25 - Why he doesn’t think he’s smart

01:09:07 - Advice for his younger self

Transcript

00:01:25 — The Bakery Algorithm

Leslie:

[00:01:25] Well, the problem was invented or discovered by Edsger Dijkstra in a 1965, I think it was 1965 paper. And that began, I consider that really the beginning of the theory of concurrency concurrent programming. He was the first one who really made use of the idea of, of concurrency as a way of structuring programs as a collection of semi independent tasks and the processes have to synchronize with one another.

[00:02:07] One of the processes or among the processes would be. Well, this was in the days of time sharing, right? Really at the beginnings of time sharing and the idea of multiple people using the same computer. People realized that computers worked faster than humans and computers were very expensive in those days. So they could use a computer to simultaneously, to be used simultaneously by multiple people.

[00:02:44] The program that each user was running, it was a separate program. But sometimes there were resources that got shared. For example, a printer, two people trying to print on the same printer at the same time. Well, the result would be not very satisfactory. So he realized there was this problem of synchronizing multiple processes. The idea of what he called a critical section or some piece of code in each of the processes so that at most one process can be executing that piece of code at any particular time.

[00:03:24] So that code might be the code that prints something on the printer. So the problem was how to get the processes to synchronize among themselves so that at most one process was executing its critical section at a time. It was in 1972 that I learned about the problem because there was an article giving a solution to it in the CACM communications of the acm. And I used to program, and I liked little programming problems, and this was just a very nice little programming problem.

[00:04:12] And so I looked at the solution, which is fairly complicated. I said, oh, gee, that shouldn’t be so hard. And so I whipped off a very simple algorithm for two processes and submitted it to cacm. And a couple of weeks later I received a letter from the editor pointing out the bug in my program. So that had two effects. The first was that I realized that concurrent programs were hard to get right and that you needed a proof that they were correct.

[00:04:54] And the second was that made me feel, I’m going to solve that damn problem. And I came up with the Bakery algorithm, which was inspired by. The idea came from what now called the deli problem, where you have a deli counter that collects tickets, a roll of tickets, and every customer would come in and take a ticket, and then the next person to be served would be the one with the highest, the lowest number ticket that hadn’t been served yet.

[00:05:33] And basically I took that idea. But since there was no central server, or at least the problem as specified by Dijkstra involved no central control, each process basically had to choose their own ticket. That was the basic idea. And the algorithm was quite simple. And I wrote a proof of correctness. And the proof of correctness revealed to me that this algorithm had this very interesting property.

[00:06:15] There was a general feeling, in fact, somebody published in a book or paper saying that it was impossible to implement mutual exclusion like this without using some lower level mutual exclusion. The way most the mutual exclusion that was assumed generally was that of shared registers, shared pieces of memory that could be written and read by different processes. And the idea is that you couldn’t have one process, two processes writing at the same time, or one process reading while the other process was writing.

[00:07:01] People assumed that those actions were atomic. They always performed as if they occurred in some specific order. But the amazing thing about the Bakery algorithm was that it didn’t require that assumption. It used each shared memory. A piece of memory was only written by a single process, so you didn’t have to worry about two processes interfering with each other. The only problem that you might come is that somebody reading the value while it was being written might get some unknown value.

[00:07:39] But the algorithm worked Anyway, if somebody read one process read while the registers were being written, that process reading process could get absolutely any value and the algorithm still worked.

Ryan:

[00:07:54] I saw in your writing about this problem that you shared it with a colleague named Anatol Holt, and the proof was so remarkable that they didn’t believe it.

Leslie:

[00:08:08] Well, the result was so remarkable, yes, that they didn’t believe it. And I wrote the proof on the whiteboard for him and he couldn’t find a problem with it. But he went home and saying there must be something wrong with it. He obviously never found anything wrong with it.

00:08:28 — Experiences with Dijkstra

Ryan:

[00:08:28] I saw the name of the paper is a new solution of Dijkstra’s concurrent programming problem. What was it about Dijkstra’s old solution that you felt was unsatisfactory and made you want to solve this problem?

Leslie:

[00:08:44] Well, there was an unsatisfactory aspect of his original solution that had the property that if there were a lot of processes kept trying to enter their critical section, an individual process might be starved. It might never get access to the critical section that was solved by. Next solution, I think, was Don Knuth’s. The condition that was desired or that measured what was considered the efficiency of it was how long a process might have to wait.

[00:09:29] I believe that the Bakery algorithm was the first one that was really first come, first serve. That is, if one process came. What it meant is if one process chose its number before another process tried to enter, that first process would enter the critical section before the other process did. I believe the Bakery algorithm was the first one with that property also.

Ryan:

[00:10:01] I see that you worked with Dijkstra and I saw in 1976 you actually worked for a month in the Netherlands and you worked with them. Can you talk about that a little bit?

Leslie:

[00:10:17] Dijkstra used to had the things, they’re called EWDs, his initials, they’re little papers, things that when he. He thought of something, had some idea, he would write it down and send it out to people. Well, One of those EWDs was about he and some associates, or actually sort of mentees I guess you would call them, wrote this algorithm. It was the first concurrent garbage collection algorithm. A way of writing programs evolved where there was a pool of memory that when a program would need a piece of memory, it would ask some server for it and be given this piece of memory, but at some point it would stop using that memory, but the program itself wouldn’t know that.

[00:11:10] The particular process that created this memory wouldn’t know whether some other process is using that memory or not. So there was an additional process called the garbage collector, which would go around examining the memory and decide which pieces of memory were no longer being used and then put them back on the. What’s called the free list, in which the server that the process that was giving out the memory would be able to take it.

[00:11:43] I looked at it, and I realized that I could simplify the algorithm because he had some special. The handling of the free list was done by a special process which had to worry about its own coordination with the processes that were using the memory. I realized that that free list could just be made part of the regular data structure, so it didn’t need special handling. And that seemed to me like a very simple idea, a very obvious idea.

[00:12:22] And I said it to him. And then when I got the next version of the paper, I discovered he had made me an author. And I thought that was very generous of him to have done that because it seemed like a very simple idea and very obvious idea. And I later realized much later that it was not an obvious idea to most people, and that that had actually impressed Dijkstra. That was the only thing I actually did with Dijkstra.

[00:13:02] Many years later. He said that I had a remarkable ability at abstraction. Only in very recent years. I mean, maybe after I got the Turing Award, that I realized that the reason for my success, the reason I wound up getting a Turing Award, was not that I was particularly that smart, but that I had this gift of abstraction. And Dijkstra was smart enough to realize that I was invited to spend a month, but not with Dijkstra, with a colleague of his, Carl Scholten.

[00:13:46] Only one thing that was ever published came out of that. Carl and I would meet with Dijkstra once a week. In the course of that discussion, the idea somehow came up that led to a variant of the Bakery algorithm that I wrote up and published. So that was the one tangible result that came from my month in the Netherlands.

Ryan:

[00:14:14] Yeah, I saw that you wrote that you spent one afternoon a week working, talking and drinking beer at Dijkstra’s house. And you kind of don’t remember exactly who was, you know, in charge of what on that paper.

Leslie:

[00:14:29] Yeah, well, I don’t think I was really could have gotten that drunk, because I probably drove to the meeting and back from the meeting.

Ryan:

[00:14:38] Right, right.

Leslie:

[00:14:39] The Dutch beer that I was drinking was not very alcoholic.

00:14:44 — His most cited paper

Ryan:

[00:14:44] I wanted to talk about your most cited paper, the one titled Time Clocks and the Ordering of Events and Distributed Systems. What’s the story behind the paper and the problem you were solving with it.

Leslie:

[00:14:56] The origin was simple. Well, somebody sent me a paper on building distributed databases where you’ll have multiple copies of the data at different places and you need to keep them synchronized in some way. I looked at it and I realized that their solution had this problem that it had the property that things would be executed as if they occurred in some sequence, but that sequence could be different from the sequence in which they actually happened.

[00:15:33] The notion of what happening before means is not obvious or not obvious to most people. But I happen to learn about special relativity in particular what’s known as the. It’s the space time view of special relativity where you basically consider space and time together as one four dimensional thing. Einstein wrote his paper in 1905, and in I think it was 1909, somebody whose name I’m blocking on provided this four dimensional view.

[00:16:18] And that four dimensional view has the particular notion of what it means for one event to occur before another. And that notion is that one event happens before another. If a signal was emitted from the first event and received by whoever did that second event before that second event happened, but the communication could not travel faster than the speed of light, because nothing can travel faster than the speed of light.

[00:16:53] Well, I realized there was an obvious analogy. The notion of happens before is exactly the same as in relativity, except instead of being whether one event can influence another by things traveling at the speed of light, it’s whether the first event could have affected the other by information sent over messages that were actually sent in the system. The thing that blew people away was this definition of happened before in a distributed system.

[00:17:33] Also, this was the first paper I would call it had a scientific result about distributed systems. I made perhaps a mistake that I was warned against at some point of having two ideas in one paper. The other thing that I realized was that there was an algorithm that would show whether one event, it would produce an ordering that satisfied that condition. That if one event happened before the other, then that first event would be ordered before the other.

[00:18:11] I realized that if you had an algorithm to do that, you could use it to basically provide the synchronization you needed for any distributed system, because you could describe that system in terms of a state machine. A state machine, as I described it then, is something that has a state and process executes commands that need to be executed in order. The command simply is something that makes a change of the state and produces a value.

[00:18:50] And so you just describe this state machine as just how commands affect the state and how they produce and what the new state is as a function of the original state and what the value is as a function of the original state. It turns out that this was very obvious to me, but that’s really in practice, the important idea in that paper, because it showed that this method of building distributed systems by thinking in terms of state machine and thinking about concurrent systems in terms of state machines, but that part was completely ignored.

[00:19:37] As a matter of fact, twice I talked to people about that paper and they said there was nothing in that paper about state machines. I just had to go back and reread the paper to be sure I wasn’t going crazy. And it really did talk about state machines. It’s important for another reason. If you’re trying to understand a concurrent program, recurrent programs are written. The Bakery algorithm is really an exception.

[00:20:07] Concurrent programs are written assuming atomic actions, so that you assume that the execution behaves like a sequence. You can assume that the execution proceeds as a sequence of events. It turns out that the way to understand why does a program produce the right answer? Well, the answer is, well, you give it the the right input, you give it the input, and then it produces the right answer. Well, but by the time you’re in the middle of execution, what it was given at the beginning is ancient history.

[00:20:47] The only thing that tells the program what to do next is its current state. The way to understand a program, a simple program that just takes input and produces an answer, is to say what is the property of the state at each point that ensures that the answer it produces is going to be correct? That property, which is mathematically a Boolean valued function of the state, is called an invariant. And understanding the invariant is the way to understand the system, the program.

[00:21:32] And I realized that the same thing is true of concurrent systems and concurrent programs. People like to write proof, behavioral proofs, reasoning about sequences. And the problem with that is that the number of sequences, possible sequences, is exponential in the length of the sequence. While your complexity of your reasoning gets to be very complicated, it’s very easy to miss cases. But the complexity of an invariance proof, the complexity of the invariant basically is, well, oh God.

[00:22:12] The number of possible executions is exponential in the number of processes. But. The behavior of the proof of an invariance proof is quadratic in the number of processes. That’s basically why invariance proofs are better. But for a long time that people doing distributed systems theory are trying to do it develop methods and formalism, something that are based on partial orderings and that they’ve published a lot of papers, but it’s just not the way.

[00:22:56] If you want to do it in practice, that’s not the way to do it. I shouldn’t say it’s not the way. There are algorithms like the Bakery algorithm that thinking impartial orderings is in fact a very good way of doing it. But those are the exceptions. The method that works that you can be sure will work is the use of invariants.

00:23:26 — The “Byzantine Generals” problem

Ryan:

[00:23:26] I want to talk about the next paper, which is the Byzantine Generals problem. I think that’s something that we hear about and we learn about when you’re going through college and computer science. And the name is great. And I want to know the story behind that problem.

Leslie:

[00:23:44] After I wrote that time clocks paper, that was a tells you how to build a distributed system, but assuming no failures. And it was obvious that, you know, distributed. One reason for distributed systems is you have multiple computers. So if one fails, you can, you know, keep going. Like in particular, that was the problem that was being solved at SRI when I joined it. But before I got to sri, I started working on that problem.

[00:24:21] And there’s no notion of idea of what I should think about is what can a failure do? So I assumed that the worst possible case that a failed process might do absolutely anything. I came up with an algorithm that basically would implement a state machine under that assumption.

[00:25:08] you can relay messages and people can check that the relayed message is actually the one that was originally sent. And so a solution using that. When I got to sri, I realized that people were trying to solve the same problem. But there are two differences. First of all, at the time I did this was 1975, very few people knew about digital signatures. And in fact, I don’t remember when the Diffie Hellman paper was published, but it was around 1975.

[00:25:46] I happen to know about digital signatures because Whit Diffie, who was one of the two authors of that paper, was a friend of mine. In fact, at one point we were at a coffee house and he was describing these things and he said, we have this problem of building digital signatures we haven’t solved. And I said, oh, that seems easy enough. I sat down and literally on a napkin, I wrote out the first digital signature algorithm.

[00:26:20] It was not practical at the time. Because it required basically something like 128 bits to sign one bit of the thing that you’re signing. It’s not quite that bad as you might think, because you could use sign not the entire document, but a hash of that document, which you assume is people cannot forge the hash.

[00:26:54] You can’t take a hash and find some other hash that or some other document that satisfies that hash. But anyway, that’s why I had. Digital signatures were part of my toolkit. So the people at SRI didn’t have that, but they also had a nicer abstraction of it. Instead of getting agreement on a sequence among the processes on a sequence of commands, they would agree, have an algorithm for agreement on a single command, and then that algorithm would be executed multiple times to.

[00:27:40] And that was a nicer way of describing what you’re doing than my method. The first paper that was published gave both their original, but since they didn’t have digital signatures, they used a different algorithm and they had the property that to tolerate one faulty process you needed four processes, whereas if you use digital signatures you only needed three processes. So the original paper contained both algorithms.

[00:28:21] And so I was one of the authors. The other algorithm without digital signatures is more complicated. And the general one for N processes was really a work of genius. It was almost incomprehensible. You just had to read this complicated proof that for the arbitrary case of an arbitrary number of processes, to tolerate N faults you needed 4N processes, whereas with digital signatures you need 3N processes.

[00:28:56] And the algorithm for single fault wasn’t hard. But the one for multiple four, four parts, Marshall Pease was the one who did it and just brilliant. Later, in a later paper I discovered a simpler proof, one that was an inductive proof, namely proof that if it works for n minus 1, it worked for n. With 3n, it works for 3 times n minus 1. The original paper was, you know, the original one was just brilliant.

[00:29:34] Who would have discovered it? Anyway? So we published that paper and I realized that this was this whole idea of Byzantine fault. So the thing is, well, Byzantine fault is one that where assume the process can do anything. Now I was assuming that process can do anything because I didn’t know what to assume. But the people at SRI had the contract for building a multi process, multi computer system for flying airplanes.

[00:30:08] And so they were the ones who appreciated the need for solving processes that can do malicious things because they really couldn’t assume what it would do. And every time you would get an algorithm and you’d see, oh, well, this algorithm. Try to get an algorithm with three processes for one fault. You’d find that, oh, this works. It must be. Really couldn’t happen in practice. And then you’d be able to find some sequence of plausible failures that would lead the algorithm to be defeated if there were a faulty process.

[00:30:50] So you needed four. For some reason, I thought that digital signatures was almost a metaphor in the algorithm, that it should be possible, since we weren’t worried about malicious failures, but just things that happen randomly, that there should be some way of writing a digital signature algorithm that would have a sufficiently low probability of failing. But I never worked on that and nobody else ever did.

[00:31:31] So that algorithm was pretty much ignored because digital signatures were very expensive in those days. I don’t know what’s being done now, because computers are. Digital signatures are just computing and computing is cheap. But I remember at some point I happened to be communicating with someone who was an engineer at Boeing. I asked whether they knew about those results. He said yes, when that he, in fact, was the one at Boeing who had read that paper.

[00:32:10] And his reaction was, oh, shit, we need four. Four computers. But anyway, I realized that this was an important result and it should be well known. I had learned one thing from Dijkstra. Dijkstra, one of the things I learned from Dijkstra, he wrote this paper called the The Dining Philosopher’s Problem. And that paper got a lot of attention. But the Dining Philosophers problem, I won’t go into what it is, but I think the basic problem was not particularly interesting, but it had a cute story to it.

[00:32:50] It involved a bunch of philosophers sitting around the table with some funny kind of spaghetti, that it required two forks, and there was one fork between each faarch would be shared with two people. And I think realized it was because of that cute story that that problem was popular. And so I decided that our work needed a cute story, a nice story, and I invented Byzantine generals. The idea being that you have a group of.

[00:33:24] For the one failure case, you have four generals who have to agree whether or not to attack. And if they all attack, they’ll win the battle. But if only some of them attack, or even if three of them attack, they’ll win the battle. But if only two attacked, they would lose. But one of the generals might be a traitor. How could you solve this problem? It’s phrased in terms of these generals having to communicate and decide whether to make the single decision whether to attack or retreat.

[00:34:06] And I called it the Byzantine Generals problem.

Ryan:

[00:34:11] I saw in your notes about the problem that there was maybe a subset of the problem or a prior version that was called the Chinese generals problem or something like that.

Leslie:

[00:34:22] Oh yeah, yeah. I was. There’s a different problem that Jim Gray described as an impossibility result. Basically it’s called the Chinese generals problem. I won’t bother going into what it is that gave me the idea of generals. I actually originally thought of the idea of Albanian generals because at that time Albania was a black hole as far as the rest of the world was concerned. It’s a communist regime, part of the Soviet bloc, but it was even more Soviet than the Soviet Republic and more restrictive.

[00:35:09] My boss said, well there are Albanians in the world so you shouldn’t that and so should have a different name. And then I realized that Byzantine, there aren’t any Byzantiums Byzantines around. And that was the perfect name.

Ryan:

[00:35:25] It’s interesting to me in the story that because this isn’t the first time the problem was specified, but it’s the first time that you had named it, gave it a good catchy name essentially and added some additional results. What was it that you saw in that problem that made it interesting? Or rather how do you know that a problem is worth putting extra time into?

Leslie:

[00:35:50] Oh, this one, it was because, you know, it was obvious that people were going to be building, that computers were going to fly or airplane. Fly airplanes. And the reason in fact because was, was that this was during the time of the oil crisis in the 70s and that they knew, people knew that they could build more energy efficient planes by reducing the size, the size of the control surfaces. But that made the plane aerodynamically unstable and a pilot couldn’t make all the adjustments needed to keep it flying, but a computer could.

[00:36:31] So it was clear the future was airplanes were going to be being flown by computers as they are today. People didn’t realize, they thought that if you want to be able to tolerate one fault, you just use three computers. And they didn’t realize that with arbitrary faults you need four. And so that was a really important result. And that’s why I believe that it needed to be well known.

Ryan:

[00:37:03] Generally when you look at the problems that you are solving with your work, how’d you decide? Because if you’re working at a company, you can decide based off of maybe the, I guess the impact to the company, like is it going to make more money or save cost or something like that. But I wonder in your work, across your career, you know, think about the bakery problem or some of Your later work as well.

[00:37:31] How do you know it’s so open ended? How do you know which problems are the ones worthwhile?

Leslie:

[00:37:37] Throughout my career I worked for private companies, you know, not. Not in academia or for the government. And so some problems arose because of sometimes an engineer would have a problem and come to me and so Disk Paxos for example was a case of that that somebody actually wanted an algorithm to do what it did.

00:38:05 — The Paxos Algorithm

Ryan:

[00:38:05] You mentioned earlier Paxos and I know that’s one of your most famous works. Curious about the story behind maybe that paper and the problem you’re solving.

Leslie:

[00:38:17] Well, the problem I was trying is exactly the same problem as I was solving in the Byzantine General’s work. Basically building a fault tolerant state machine. But by that time it was the faults that interested industry were ones where failure meant that the computer just stopped, not that it did arbitrary things. Paxos is an algorithm for building fault tolerant systems for handling that class of faults.

[00:38:57] The people I was working at was the DECSERC lab, which I joined in 1985 and they built a. One of the first operating systems that was a distributed operating system. So that basically everybody had. Basically these are the people who had come from Xerox PARC and had invented personal computing, but they also had the notion of distributed personal computing. They invented the Ethernet for that. So basically all of the computers in the building were on a single ethernet network and shared a common storage and they had an algorithm for maintaining consistency of that storage.

[00:39:56] And I didn’t believe, well, they didn’t have an algorithm, they had an operating system with code that did that. I didn’t believe that what they did was possible. Namely I didn’t think. Well, I forget exactly why I didn’t think it was possible. But at any rate I started try to come up with an impossibility proof and then the solidity proof. Well, an algorithm to solve this would have to do this and in order to do this it would have to do that.

[00:40:38] And at some point I stopped and said oh, this isn’t a proof, it can’t. This is an algorithm that does it.

Ryan:

[00:40:45] You said that they had code but not an algorithm. What do you mean by that?

Leslie:

[00:40:53] When most people sit down and start writing programs, they start by thinking in terms of code. One of the things I learned fairly early in my career, I don’t remember exactly when that back in the days when I started writing current algorithms, people talked about, people were calling them programs and I was probably calling them programs too. I mean I remember then at some point I realized That I wasn’t talking about programs, I was talking about interested in algorithms.

[00:41:29] And an algorithm is something that’s more abstract than a program. An algorithm can be a program is written in some particular code, but an algorithm can be implemented in programs written in any kinds of code. It’s something that’s, that’s at a higher level of abstraction. And of course I like that because abstraction is something I was good at, even without realizing that that’s what I was doing.

[00:42:01] And so what I’ve spent a large part of my career, basically maybe about 2000 or so onward was getting people who build concurrent systems to not just write code, but to have an algorithm. Now a system does lots of things, but there should be some kernel of the program that’s involved with synchronizing the different processes or distributed system, the different computers. That code is very hard to get that correct.

[00:42:53] You don’t want to think in terms of code because that encoding conflates a lot of issues that are irrelevant to the concurrency aspect. You should be thinking, first get an algorithm that does that synchronization and then implement that algorithm.

Ryan:

[00:43:12] I was looking at the Paxos paper and some of your notes about it, and I saw that there’s an eight year gap between when you came up with the algorithm and when the paper was actually published called Part Time. Parliament is the name of the paper. Why is there an eight year gap?

Leslie:

[00:43:34] Well, the referees originally said, well, this paper is okay, not terribly important. But fortunately Butler Lamson realized the importance of the algorithm together with the idea of getting implement anything because it’s implementing a state machine, you know, went about proceed proselytizing, building your systems, you know, using Paxos, you know, and, and thinking in terms of state machines. And so, you know, I wasn’t.

[00:44:13] So the idea was getting out. So, you know, I was in no hurry to publish. So, you know, I just let the paper sit. And eventually there was a new editor that came along and he said that I think the status of the paper was that it was just, it had been accepted but needed revision. And so he decided that, yeah, let’s publish it. And it was eventually published with a few things to take, well to mention work that had been done in the interim.

[00:44:58] And what I got is got a Keith Marzullo to do that part for me. And so the story was that this manuscript, well, the story about Paxos was that it’s a, this happened centuries ago and this manuscript. And I use that to the effect that when something, the tales of something were I considered obvious and not interesting. The paper would say it’s not clear how the Paxons, what the Paxons did at this point.

[00:45:38] At any rate, Keith kept up that idea that this was a description of this ancient thing.

Ryan:

[00:46:04] writing too, when you were talking about presenting the paper initially, you even dressed up in an Indiana Jones style archaeologist. Well, how did that go when you presented about this Paxos paper and algorithm?

Leslie:

[00:46:17] Well, I think the lecture may have gone well, but I think nobody understood the algorithm. Where nobody understood the significance of the algorithm.

Ryan:

[00:46:27] It sounds like no one understood it except for Butler Lamson. What did he see that made him unique, I guess.

Leslie:

[00:46:35] Well, he had a good understanding of building systems. He really deserved his Turing Award. He was one of the original people at Xerox PARC who were building distributed personal computing. He and Chuck Thacker, I think, were probably the two senior people in that lab.

00:46:57 — Paxos vs Raft Algorithm

Ryan:

[00:46:57] I saw later there was a paper which describes a new algorithm which seems to solve the same problem, the Raft paper. I was wondering if you read that and what your thoughts were on that versus Paxos.

Leslie:

[00:47:11] The authors of that actually sent me a draft of the original paper and I looked at it and said, I forget whether I said, send it back to me when you have an algorithm or send it back to me when you have a proof. I forget which one it was. And you got the idea and they really, they did write, you know, ad proof the paper or not. Yeah, I never read future later versions. And someone whose judgment I value said, you know, had read it and said that it’s basically it’s the Paxos paper, but, you know, but with some of the tales left unfinished by the Paxos paper, by some of the details filled in, but they described it in a very different way.

[00:48:07] The basic idea of what Paxos works is it’s two phases and you’re trying to implement a sequence of decisions. It turns out you can do the first phase once, it involves a leader and the leader has to get elected. But it turns out that you can do the first phase once and you don’t have to do it again as long as you have the same leader. But it’s only the second part that you have to do. And then you have to let the new leader, if a new leader fails, and do the first part.

[00:48:56] So think about it in those two phases. But the way people, way engineers seem like to think about it is, well, you do this you’re talking about the first part, the second phase, you keep doing this until the leader fails and then you go back, then you have to do this thing. So you. It’s explaining it in the opposite order. And in fact, when you start it from fresh, you don’t have to do the first phase.

[00:49:31] Basically, what started the first phase could be just built in into the initial state. But I think that that’s those two phases the way to understand it. But you know, the RAFT people also had this idea that Raft is better because it’s simpler. I must say that a lot of people say that Paxos is hard to understand, and I don’t understand why. I mean, I’ve explained it to people in five minutes and they understood it.

[00:50:02] At any rate, the RAFT people said that one of the ideas were simpler because. And they even have Paxos to one class and Raft to another. And they took. And then yes, the people, all the students said that yes, it was more understandable. The interesting thing about it though is that there was a bug discovered in Raft and fixed. But I believe that the algorithm that they found more understandable was one with that bug.

[00:50:36] So made me realize that what most people. What does understanding mean? For me, understanding means you can write a proof of it. But what understanding means for most people is a warm fuzzy feeling. The RAFT description gave them more of a warm fuzzy feeling because, you know, that that seems to be the way programmers like to think about the algorithm, the second phase first, until you get a failure.

[00:51:18] But the way I describe it is one that helps you get a better understanding of why it actually works.

00:51:26 — Building LaTeX

Ryan:

[00:51:26] So, yeah, we talked about a lot of your papers. I know one of your other contributions, whether you knew it or not at the time, was Latex and building that and something that has impacted the entire academic community. What’s the story behind wanting to build latex?

Leslie:

[00:51:46] Oh, that was very simple. I was wanted. I was in the process of starting to write a book and it was clear that tech was the basic typesetting system that one had to use. But I felt that I would need macros to make tech do what I wanted it to do. And so I, I figured with a little extra effort, I could make macros usable by other people. The system I had been using before tech is called Scribe.

[00:52:51] Scribe will do the formatting. Well, Scribe didn’t do that great job of formatting, but obviously I like the idea abstraction that it’s the ideas that matter, not the text that the writing that matters, not the typesetting. And so I actually, at some point I met Peter Gordon, Addison Wesley, I’m not sure what you would call him, but he looks for books to publish and he convinced me that I should write a book.

[00:53:41] And those days it never occurred to me people would actually spend money for a book about software. But you know, what the hell. And what he did was he introduced me to a typographic designer at Addison Wesley who was responsible for really for the typographic design that’s in the standard latex styles. And so, you know, basically I just did that in my quote, spare time. You know, took me six or nine months or so.

[00:54:29] I suppose the statute of limitations has run out, but I was really spent some time working on that when I was allegedly billing the time to some project that had nothing to do with it.

00:54:45 — Why writing improves your thinking

Ryan:

[00:54:45] On the topic of writing, you have a quote that I really enjoy. If you’re thinking without writing, you only think you’re thinking. And I was curious to hear your thoughts on what you mean by that.

Leslie:

[00:54:59] Well, it was really meant for people building computer systems. You have an idea and you think it’s going to work or you have something that you think is something that somebody else will want to use. Well, write a description of it. There’s an old maxim that I heard that is write the instruction manual before you write the program. Great advice. I did not do that with latex, but definitely when I was writing the book and I discovered that something was hard to describe, hard to explain, that needed to be changed, and I made a number of changes to it as a result of that.

[00:55:51] But I didn’t start at the beginning with the instruction manual.

Ryan:

[00:55:55] Why is writing conducive to good thinking?

Leslie:

[00:56:00] Because. It’s very easy to fool yourself. I mean, that underlies my whole idea of writing proofs. One thing I learned is that you had to write a correctness proof of a concurrent algorithm. And when my algorithm is starting to get more complicated, the proof started. I started right. You know, I was a math PhD in math, I knew how to write proofs and I was starting writing the proofs the way I would normally do and I realized it just didn’t work because there were just so many details involved and I just couldn’t keep track of them and whether I had done it.

[00:56:49] And so as a computer science know how to deal with concurrency. It’s hierarchical structure. And so I devised this hierarchical structure where a proof is, you know, is a sequence of Steps, each of which has a proof. And the proof is either a para. Well, a proof is either a paragraph or, or a statement, a sequence of steps, each with its proof. And that proof can be either a paragraph or a sequence of steps with its proof.

[00:57:22] So you break the whole problem up into these smaller pieces. So there’s never any question of where is this coming from. You’re stating that this step follows from this step, this step, this step, this step. And if it does not follow from that step, your proof is wrong. The theorem might be correct, but it means your proof is wrong. Well, I discovered that worked great on writing my proofs of programs, but I decided to really.

[00:57:51] I also write proofs of theorems, proofs that are things that are more like ordinary math. And I started trying that on them, and I discovered it worked beautifully. So when I started to try to convince mathematicians to write these proofs, I started in one small seminar. I won’t describe what it was about, but I described this proof through maybe 20 mathematicians or something. Their reaction shocked me.

[00:58:27] They became angry. I really thought that they might physically attack me. So I believe that what’s going on is that when I believe that’s totally irrational, and when people act irrationally, it tends to be out of fear. And what I believe people are afraid of is that mathematicians are afraid of, is that they’re going to have to write their proofs to convince a, a computer program. And in fact, one of those talks I gave, I say very clearly, this doesn’t have to be.

[00:59:09] You don’t have to be any more formal than you do. You can write the exact same thing proof, but it’s just a matter of organizing things. And it’s very simple hierarchical structure. And then when you’re using a fact, mention that you’re using that. Nothing about formalism or anything. After I gave that talk, someone got up and said, I don’t want to have to write my proofs for a computer program. In fact, it’s more work doing that, because the reason it’s more work is that it reveals what you haven’t said and that there’s steps in there that you may think they’re obvious, but you haven’t written them down.

[00:59:58] And if you believe something is correct but don’t really, if you think you know something but don’t write it down, you only think you know it. And that’s where errors come in. That’s where that one third of your paper’s errors can you come in, because it really makes you honest.

01:00:21 — Why he wasn’t an academic

Ryan:

[01:00:21] When I look across Your career, I think you had a lot of contributions people might expect, might come from academia, these papers and things, but you did all of your work in industry. Why did you not see yourself as an academic and more of working for industry?

Leslie:

[01:00:39] Well, I started out programming and I eventually got jobs where took me into what we now call computer science. At the time, I never even realized that there could be a science of computing. It wasn’t until maybe until mid to late 70s that I realized yes, there was the computer science as a computer scientist, but it never seemed to me that computer science was an academic subject. At some point I had to make a choice between doing computer science without calling it computer science or teaching math at a university.

[01:01:29] I chose for fairly random reasons to do computer science. So for the first, I don’t know, well, till maybe the mid-80s or something, it just didn’t seem to me that computer science was something that people needed to go to a university to learn. And I suppose afterwards that I was sort of. I guess I just didn’t think it would be fun teaching computer science.

01:02:08 — Grand theory of concurrency

Ryan:

[01:02:08] I saw in your writing you had a footnote that said somewhere that you felt like a failure at some point because you wanted to develop this grand theory of concurrency and you never discovered it. Do you still feel that way or what are your thoughts on that footnote?

Leslie:

[01:02:28] Lots of people, a large percentage of the people who were doing things like I was doing, which is not a large number of people. There’s this notion that they’re looking for the Turing machine of concurrency. The Turing machine was this abstraction which really captured what computing was and they were looking for something that would be the Turing machine of concurrent computing. Nobody succeeded.

[01:03:09] There are some people who think they’ve succeeded the Patriot or something that I guess I don’t have time to explain, but it was big in the 70s and I was actually surprised to think that there’s still a large community of people doing Patriot. But what I now realize is that Patriot and most of the things that people were doing was really language based. And I was never interested in languages. I’m interested in what the language is expressing.

[01:03:46] And I realized, in some sense, maybe I realized what the Turing machine of computing is. State machines, state machines are a little bit different the way I now describe them. They don’t have commands, they just have a state and a next state relation. Even simpler than talking about commands and values and stuff. And to me that’s the Turing machine of concurrency. But. It doesn’t have the function that Turing machines offer.

[01:04:27] Because what Turing machines Do is describe what’s possible. And state machines can describe anything, including things that are not possible. In fact, there’s a good reason for that. For example, when I describe an algorithm, I will talk about the values of a variable can be any integer. Now, you can’t implement the program where you have any integer. But that makes talking about computer integers would complicate things unnecessarily.

[01:05:22] People have this funny idea that because something is infinite, it’s more complicated. They got it backwards. Infinity was introduced to simplify things. The first thing you learn is arithmetic. You’re learning arithmetic with an infinite number of integers, because if you were restricted to a finite set of integers, arithmetic becomes much more complicated. So the abstractions of mathematics which people find because they don’t have the proper training in mathematics, find difficult are really what’s simplifying things.

[01:06:05] And that’s what you use this mathematics. The state machine is described by me using mathematics. That’s the right, the most powerful way of doing it. But computer people and computer scientists and programmers are really hung up on languages. And so they are looking for, you know, they invent all sorts of languages and they’re all describable. And in fact, if you want to give them a semantics, you would do it in terms of a state machine.

[01:06:40] And they just think that this, this language improves your thinking. It doesn’t. I mean, there are reasons why you use computer languages and you don’t write your program’s code in math. And they involve basically efficiency, but for understanding. You can’t build math, you can’t beat math. And you know, attempts to do it by something that looks like a programming language is just the wrong way to deal when you’re trying to deal with concurrency.

01:07:25 — Why he doesn’t think he’s smart

Ryan:

[01:07:25] When I look at everything that you’ve written and all the stories, there’s these little anecdotes, there’s things where you say things like, you never considered yourself smart, but you noticed that other kids had an awful time understanding things. Or yeah, there’s a problem that you solved where someone else had difficulties, but you don’t view your contribution as a brilliant one or anything like that.

[01:07:51] And that doesn’t connect with me because you’ve also won a Turing Award and done all these amazing things. So how could that be that you in a just merely discover things and are not smart, yet you’ve achieved so much?

Leslie:

[01:08:07] Well, this general thing that psychologists talk about is that when someone is good at something, they don’t realize how good they are at it because it’s simple to them. There’s the opposite one that people who are bad at something think they’re better than they are because they’re bad at it. Or put a little bit more concisely, stupid people think they’re smart because they’re too stupid to realize they’re not.

[01:08:48] The gift that I have is not in some sense raw intelligence. It’s abstraction. And it’s only recently, last 10 or so years, that I realized how much better I am at that than other people.

01:09:07 — Advice for his younger self

Ryan:

[01:09:07] Most other people at this point, you’ve experienced so much, and when you look back on your career, if you could go back to yourself when you just graduated college and give yourself some advice, knowing what you know now, what would you say?

Leslie:

[01:09:24] One thing I’ve learned fairly early in my life is that I shouldn’t waste time trying to answer questions that I don’t have to answer. I don’t think about what I should have done, because that’s a question that I don’t have to answer.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:40 - How Netflix was different

00:08:01 - The legendary Netflix culture memo

00:18:54 - How to hire engineers well

00:30:52 - The strongest engineer he’s ever met

00:33:02 - Joining Meta

00:50:52 - Near death experience

00:59:04 - Where he learned the most

01:04:09 - Book that impacted his career most

01:11:33 - Advice for his younger self

Transcript

00:00:40 — How Netflix was different

Ryan:

[00:00:40] So about Netflix, how did it differ from the other companies you had worked at?

David:

[00:00:53] Every company I worked at before Netflix. Well, I can’t really talk much about Hughes Aircraft, GM, Hughes, but starting with like the IBM and moving through even these smaller companies, we had these Primadonnas, right? They have may be really smart, but they can be really difficult to work with. People yell at you, they whatever. And the company’s value they would actually build. I mean I worked at a company in the 90s.

[00:01:24] This apparently the most critical engineer. Oh, if we lose him, we’re toast. The guy’s cubicle was so full of junk, the walls were bulging out. He was like a pack rat. And he wouldn’t engage with anybody and he would work all night and fix problems. He was always the hero. And the managers, they did, had no understanding what he did or whatever, but all they knew is if he quits, we’re in trouble.

[00:01:49] Well, he quit, right? And guess what? The company didn’t die, right? Once he was out of the way, he wanted to do something else. Everybody else moved in. They actually had other people understand, just nobody could work with this guy. And he established himself as the center of gravity for this one big area of our system. I think it was the storage. I can’t remember the part of it. And I had this kind of same thing at Inter Video.

[00:02:13] I had some colleagues, I had some really, really good colleagues that actually helped form my technical. They gave me my technical education. But there were some people that were really, really hard to work with and scream at you and yell at you and they would be wrong and they could never admit it, you know, and you know, the customers, you know, customers have a problem. We try to tell them like their code is wrong and they just, they respond with screaming and yelling at you.

[00:02:37] And I got to Netflix and it’s like, I don’t have anybody like that, you know, and it was written in the culture deck. They don’t hire brilliant jerks. Right. If you’re a person who’s always going off the rails, who’s yelling and screaming, who’s difficult to work with, who makes yourself unapproachable, they’re going to let you go. And they don’t care how smart you are or how what level of contribution because they realize, and I, you know, this, this was revolutionary in 2000, in the early 2000s.

[00:03:07] You know, somebody who grows up in meta. Yeah. Your point is, I mean, because that’s weight modern, you know, the, the fang companies, at least the ones I know, Google, Netflix, meta. That’s the way they work now. But that wasn’t like that. So it was, I was like, wow. That was the first thing. And then the second thing during my interview, Patty McCord, who was a chief people officer who inter at that time was a very small company, interviewed everybody and had to bless every hire.

[00:03:35] You think about that. And, and, and I learned, I will say I learned more from Patty, I think even than Reed as far as being a good leader and being a really like a good collaborative teammate. But during our interview, she said something that just floored me. She says, we don’t value 24. 7 work here and if you just come in and work all the time, we’re not going to be impressed. And it was like, if you want to impress us, blow us away with what you can do in an eight hour day.

[00:04:03] And, and I also, you know, and she’s like, but also let us, you know, like, we like people that, like, if they’re laying in bed at night and they can’t sleep, odds are they’re thinking about one of their problems at Netflix that they’re trying to fix. These are the people we want. And I’m just like, wow. And, and so I got into Netflix and, and that, I will say that was absolutely true. That, that I did see people fall into that 24 7 trap.

[00:04:27] And, and you know, when people work like that, one, you feel it’s a compulsive need, you know, because you’re, you know, you don’t have a team to support you. Everything, like, if I stop everything, the company’s going to fail. And two, but then you hate it because you’re, you know, and, and I just remember there was a meeting and he was like, I’m just working all the time. I can’t work like this anymore.

[00:04:46] And Patty was in the meeting and she says, hey, first, I want to thank you for everything you’ve done to try to help us be successful and not fail and that you’re even willing to work 24 7. Now. The second thing I have to say is, don’t do this again. You need to figure out, you’re a manager, you’re a leader. You need to figure out how to set your team up so that you don’t have to work all the time.

[00:05:08] And if you want to be successful here, you have to do that. That’s just, that was unthinkable in 2008, 2009. And I even had a, you know, similar situation later at Netflix because, you know, we were still working on the, you know, a lot of, you know, continuous build deployment. These things didn’t exist, right? Things that everybody takes for granted. They didn’t really, they were just coming out, you know, Google, other companies were just getting these in place.

[00:05:33] And we had some serious stability issues when we tried to go to continuous deployment. And, and I was over talking to it and he was a phenomenal engineer. I still engaged with a guy, we both retired, but, but I was over talking to him. I said to him, by the weekend, he goes, well, it must be nice having weekends off. I said, what do you mean? We have. If we quit, if we stop working, you know, these systems go down, we can’t keep them running.

[00:05:56] And it was, you know, the other people that he worked with all kind of came out and I feel like I’m gonna, you know, I gotta get out of here, right? But I left and I set up a one on one with him the following week. And I got in a room with him and I said, you need to take a vacation. He’s like, I can’t. Or, you know, if we don’t, we’re just heating. And I told him straight up, I said, if this company cannot survive without you here on the clock, we got a problem.

[00:06:10] You need to take a vacation. And I don’t care what happens, if things break, they break. Take your vacation. And. And he did. He went and took a week off and he came back and, you know, refreshed. He had been with his family and, you know, and, and the other people on that team took time off and they were able actually rather quickly to stabilize the system because they quit trying to do what the manager was telling them they needed to do.

[00:06:52] They decided this is how we’re going to stabilize it. And then all of a sudden, this team that was killing themselves just by forcing themselves to take time off was actually able to come back, refresh, think about the problem, not think about how to keep the systems running. And they were actually able to have that time off. So that was part of that. I mean, that was fundamental piece of Netflix culture.

[00:07:12] Patty told me, we all have lives. See, I, I don’t want to work all weekend. I want to go home. I want to enjoy my life, you know. And so I really did appreciate that. And I will say also, Netflix was built with strong and visionary leaders. Reed Hastings had a vision from early 90s or for late 90s, and he said if, if I wanted to do like DVDs by mail, this would be called DVDs by mail dot com. It’s called Netflix.

[00:07:41] Because DVD by mail for him was a stepping stone to what he believed was the future was streaming video. And he didn’t know how it was going to happen. He, Reid’s not a video expert, but he built a team out of really good leaders and a really good, small team of really good engineers. And it’s like, we’re going to solve this problem.

00:08:01 — The legendary Netflix culture memo

Ryan:

[00:08:01] How did Netflix culture change as a company grew?

David:

[00:08:05] You know, obviously, the, the, the Netflix Culture Memo, I think is something of legends, right? People talk about it and I, what I would have to say about. There’s a few things I would say about that. First of all, for me personally, because people would ask others, is this how it is at Netflix? And they would say, yeah. They would ask me, I say, no, it’s what we aspire to be. The Culture Memo is aspirational.

Ryan:

[00:08:27] What is the culture memo?

David:

[00:08:30] Oh, it’s like, it’s like the original. It was like some slides, like 15 slides, talking about, we don’t hire brilliant jerks. We want really smart people. We want to inspire people to sail the seas. We don’t want to tell them how to build a ship, right? We, we, you know, we, you know, we, we pay to hop a market. We, we, you know, we have a keeper’s test, which, you know, basically we’re aggressive at firing people.

[00:08:55] We’re not afraid to make mistakes. Those were kind of all baked into the Culture Memo, but it was, you know, putting that out when it was put out, you know, in like 2007. 8. The original culture, it was. That stack was actually pretty revolutionary at the time. A lot of that, you know, freedom of responsibility, the whole. The ability of hiring really good people and not staying out of their way, giving them the context of what needs to be done, helping them understand why it matters, and then expecting them to go and execute on that without you micromanaging them, you know, trust, basically.

[00:09:25] Trust, freedom, responsibility, trust. And, you know, and so these were all really, really key parts of the Netflix culture. And I think as a small company, if what enabled Netflix to be successful where others, other companies didn’t, was that culture built around exceptional talent. Not just engineers. Everybody I worked with at Netflix was exceptional, right? The marketing people, the, the, the content people, product, every hr.

[00:09:57] I mean, and they were all learning. I was learning from everybody, right? I learned how to build a good team through my HR partners. They weren’t just, you know, giving me resumes and, and scheduling interviews, right? They were challenging me on my higher, no hire decisions. They were, you know, pushing, you know, so I think that was fundamental. I did notice as the company was growing that, that that culture was not scaling well.

[00:10:26] And I’ll give you a good example. One thing Netflix didn’t, wasn’t really big on was individual achievement. Everything was one as a team. Everything we did was a team. Nobody ever said, for example, when we shipped the PS3, that revolutionary rule breaker, you know, we did something so crazy, Sony, the DVD division at Sony, saw what we were doing and said, please stop. We didn’t design the PS3 to do this, please stop.

[00:10:52] But we didn’t stop. We broke the rule. We, we delivered a system. In the end, they didn’t say, hey, Scott, Wu and David were the, you know, and Mitch were the, were the foundational engineers that made this all possible. It was Netflix 1, you know, and at the time that was really good. But as things got bigger, this all gets lost, right? And, and I feel like as the company got bigger and instead of doing one or two things, you starting to do many things.

[00:11:24] Individual contributions, which are still huge. I mean, engineering, that’s the way engineering works. We’re starting to get lost, right? And if you don’t credit individuals for great engineering work, who gets the credit? So leaders. And, and I realized that that was something I struggled with. We didn’t have levels. All engineers were equal. There was only an engineer, senior software engineer, period.

[00:11:49] You could make your title anything you wanted. You could call yourself the, you know, software engineering, video guru. You could call yourself, you know, whatever. You couldn’t put manager, director, VP or chief in your title if you weren’t one of those. But other than that, your business card could say anything you wanted. But there was only one software engineering level and As a small company with just senior people has worked, but as we started hiring more juniors, when you, as teams grow, you’re going to have really strong engineers, you’re going to have good engineers and then you’re going to have weak engineers.

[00:12:26] And if you don’t have a system in place that can understand that and, and, and recognize and, and objectively determine who’s who, then it you start having challenges with your best people. It ultimately not having a good objective process for recognizing wins, achievements and rewarding those wins and achievements ultimately impacts your best people who end up leaving. So that was something I was struggling with.

[00:12:51] I, I had, was trying to argue they do have levels now. At the time, like by the late teens, I was arguing like we really need levels because we have really senior engineers and we’ve got juniors. We have people who understand our system from one end to the other and can do incredible work. And we have other, they’re good engineers, but they’re just learning. And, and so I think these were some of the aspects that didn’t scale.

[00:13:14] Other things, like I mentioned in the early days, Patty McCord, who’s one of the great influencers in my life, interviewed almost every single hire that come in the company. Obviously you can’t keep doing that. Reed Hastings have like these really small one on ones with new, like four new hires at a time. He would sit and give you like an hour of his time to ask questions, to talk of, share about you, who you were and what excited you about Netflix.

[00:13:39] Obviously you couldn’t do that. So eventually Reid would show up with a couple other leaders with 100 people in the room or whatever, you know, and eventually I think they probably couldn’t even do that. So these are, these are all challenges to scale where, you know, but, you know, success, right. Success means that, you know, success means a small company becomes a big company if that’s how you measure success.

[00:14:02] And success at Netflix was measured in, you know, building this product that’s going to reach the world. So success means you’re going to be big. And it means that things that were easy or worked well at a small company don’t necessarily grow with you. So you have to make those adjustments. I haven’t been there for six years. I’m sure, I know they brought levels and I’m sure they’ve done a lot of work to change that.

[00:14:23] But when I left, I felt the culture was struggling with a culture memo that was perfect for a very small, aggressive, engineering focused company.

Ryan:

[00:14:32] If everyone is the same level, how did they Compensate people differently or, or, you know, I imagine someone who’s a new grad that joins Netflix compared to industry veteran video expert.

David:

[00:14:44] Right? Well, I mean, at the time, Netflix generally didn’t hire new grads. I will say generally, because I found lots of clever ways to break rules and, and work around, you know, around that. But generally speaking, and, and they had this concept of personal top of market. Right. So if I were. And they also had this interesting thing in their culture memo that they encourage people to interview often.

[00:15:12] So if I got a job, I’ll just put it out there. My starting salary when I joined Netflix, which at the time was a good starting salary, was 175k a year. Somebody may think, wow, you know, you’re 15 years in, that’s all you got. But I mean, you know, but at the time 2007, that was a really solid starting salary. If a year later I would have interviewed Google and they would have offered me 200,000, I would have gone and told my manager, hey, I got this offer for 200,000.

[00:15:39] My manager would have said, had a major choice, go work at Google or we’re going to give you pay raise. Right, because your top of market, your personal top of market has now moved. And somebody else who came in at 175May never, May not move that way. But also over time, you know, this was incumbent on us as leaders, which I took very serious. And the managers that worked with me is understanding where the market is, understanding where offers are, who’s getting offers.

[00:16:13] How much is Google paying? Right. Meta. You know, Meta was late to the video game, but by 20, it was Facebook at the time, by 2015, Netflix was very aggressive. I’m sorry, Meta was very aggressive building a video team. How much were they offering our people who was interviewing? You know, if somebody, you know, we understood their skill level, went and got an interview from one of these other companies and got an offer that was twice what we were paying everybody.

[00:16:35] That means we have to adjust our compensation twice, Right? That’s the way it worked. But again, in the early days, it worked well. But as you get more and more people, that becomes muddied. And it becomes muddied in several ways. One way is that, and it’s unsustainable. Let’s say an engineer who is one of your great engineers did get this offer at 2XR comp. And we reviewed it at a senior leader level and said, yeah, we really need this person and this is our market, let’s pay them.

[00:17:08] Three years later, this engineer is Making more than almost everybody else, still contributing at that level. But because there’s no performance review, there’s no, there’s no. A leader could look and say, why is this person making three times this person? Because all they see is numbers. They have no context. And that’s where the system started breaking. And. And then we would have like, you know, hey, you know, I’ll just say this.

[00:17:29] We had a. Some engineers, some that we interviewed and offered jobs at X and they all got competing offers at Facebook or like five of them at 1. 2x and we’re in a room. And I said, well, the culture member says we adjust our offer to 1. 2. Well, you know, it was. They didn’t understand. We didn’t have baked into our system, didn’t have understanding of how compensation worked like that. We didn’t have a concept of leveling.

[00:17:58] And so all of a sudden nobody could rationalize that. 1. 2x. Now they have the leveling. And I think that they would very. Oh, you got an offer, an IC6 at Google. Okay, we understand. And by the way, we can all go out to, you know, levels, FYI, and see what that range is. And okay, we can adjust you. But we didn’t have that back then. So that’s where things got really challenging. And as a leader who, you know, I valued like building a team and keeping a good team and paying them, you know, market right, I didn’t want to overpay anybody.

[00:18:31] I got really difficult. I had a lot of disagreements with my leadership and tried to explain to them, hey, here’s my spreadsheets year over year. Here’s all the work we did, here’s the offers we were seeing coming in, and I do believe they fix that problem. Actually, they have ranges. Like every you, as you’re a large company, you have to have ranges. You, you can’t do any different than that. So they did fix that problem.

00:18:54 — How to hire engineers well

Ryan:

[00:18:54] After I left, you hired a ton of software engineers at Netflix and also I’m sure at Meta as well. How do you identify really strong software engineers in the hiring process?

David:

[00:19:05] So this is interesting, actually. I’ve had success hiring all the way up until Meta. Meta is a big company like Google and Amazon and others, Microsoft. They, because they bring so many people in and they need to have some structure and accountability and they build a system, an interviewing system where you get to smaller companies and you’re actually now me and three people on my team plus HR are interviewing somebody I want to hire.

[00:19:36] Right? That’s not the way it Works. I don’t know about Netflix anymore. That’s the way it worked when I was there. But you know, that’s not the way it works at Meta because they just, the company’s too big and they hire too many people and they need to know that they’re giving fair and consistent standards for hiring. Right. So. But I’ve been very successful back when, when I was literally making the hire, no hire decisions.

[00:20:01] And it’s curious, I’ll give a couple examples. So in the late 1990s on that first really good project, we interviewed a woman actually, and I was the last one to interview her. And she had been working in like an automated train software system, building automated train automation software, engine, driverless trains. And our 45 minute interview went an hour and a half and she had two whiteboards covered with this system and we’re walking through state transitions and I’m just blown away by like, this is one of the best systems minds I’ve ever sat and worked with and her understanding and the ability to explain these, this complex, incredibly complex, the distributed system, not for scale, but it was distributed in terms of the way that everything fit together.

[00:20:55] The pieces all interacted well. The other people gave her effectively leetcode. She was okay. I gave her string version. It was okay, it worked. But it wasn’t. They didn’t want to hire. And I told my boss and I said, I want to hire. My boss pushed back. Ironically, you know, my boss was also woman, but she was concerned. Well, these people are kind of lukewarm. I said, look, this is one of the best systems engineers I’ve ever talked to.

[00:21:19] She understands complexity. We’re struggling. All these great engineers we have, we’re struggling with the integration of all these disparate systems we’re bringing together. We need her and we hired her. And she freaking killed it, right? Absolutely killed it. And, and that’s gone on. I’ve always felt like when I’m hiring in small companies for roles for people on my team, generally speaking, my batting average is very close to a thousand.

[00:21:46] And I think it’s because this is going to be controversial. I personally don’t love leetcode because I don’t think it really tells you much about it. It tells you how fast somebody can write code for a problem that they probably practiced a hundred times, right? Because people go, but. But I will say on the other side of that, when you’re hiring at the volume that these large companies like Meta and Google do, you really don’t have any choice.

[00:22:11] But leetcode does not tell you anything about an engineer. And as I mentioned before, engineering, first of all, I don’t think about engineering like software engineers, right. I personally believe all engineering disciplines are all built on the same foundation. And I’ve mentioned this earlier, we were talking that foundation is the ability to understand complex systems, a strong technical intuition, and the ability to make decisions absent good decisions, absent enough data to tell you it’s the right decision.

[00:22:43] That’s intuition. And I’ve seen really, really good engineers move into software and be phenomenally successful. And I’ve seen engineers who nail the LEET code and these canned system design. They’ve, you know, they practice all, you know, you know, type ahead, Google type whatever the things we like give them. They nail these interviews and they get in and their engineering work is terrible. And, and it’s like they’re not.

[00:23:11] I look at their, their review packets and the feedback and I’m like, this person is not making good engineering decisions. The quality of their work is really bad. So I will simply say that if you have that strong foundational engineering ability, then, then you can learn software and be a phenomenal software engineer. But I don’t care how much you learn about coding, if you don’t have a good engineering foundation that might, it’s.

[00:23:36] And you’re born with it. It’s like a musician, right? You can, I like to play guitar, but I don’t have that foundational talent that would make me a phenomenal guy. I’m a good guitar player, I practice a lot, but I’ll never be great. So engineering is something the foundational of engineering, you’re born with it, you have it or you don’t. And I think a lot of people get engineering that don’t necessarily have that.

[00:23:58] And when I recognize that mindset and realize that I’m talking to a good engineer, I don’t care that their software is a little subpar because I feel like I can help them with that. I can’t teach them how to make good decisions. I’ll give you a really good example. This is going to blow you away. So I was taking some classes, some graduate level classes at San Jose State and then one class I took my project partner, I had two partners is kind of funny because this was just right before I joined Meta.

[00:24:30] So here I am in my 50s and I’ve got people in their 20s, but this woman joined my team and she was just finishing her master’s in civil engineering, I believe was working in wastewater treatment and she decided she wanted to move to Software. And this was her first class because you could go into the software engineering program at San Jose State without the undergraduate requirements and you could catch up on those requirements over time by taking.

[00:24:56] Basically they said you can start taking master level, but you can’t graduate until you complete these batch, these undergraduate level foundational classes, Data structures and program. She was incredibly smart, incredibly focused, and her understanding of like her the analytical mind or mastery of statistics. And I just like. And I was telling, you know, one of my colleagues at Netflix, I just, I want to hire her.

[00:25:25] I don’t know how to do it, but I want to hire her. And, and I finally convinced one of my managers say we should look at her. And maybe we had this idea where we didn’t really have anybody who was like a date. We didn’t have our own dedicated de right or ds. So we thought, well, let’s bring her in to help us with the data analytics. So we gave her a video software engineering problem as a take home assignment.

[00:25:47] She had never really written code, she had never worked on video. Took her three weeks and she came back and it’s like, well, she made some rookie mistakes, but damn, you know, it’s like, and, and so everybody was like kind of impressed. And it’s like, well, we know she’s not a good software engineer. What do we need to close the deal? And I suggested, let’s have her come in and teach us something.

[00:26:10] All she knew was wastewater treatment because she worked at the Santa Clara county wastewater facility. So she came in and spent two hours on a whiteboard. And I’ve realized halfway through, this is a freaking state machine. And it’s not a, it’s a very complex state machine. And I was just like. And we’re all just, I mean, two hours, we’re all enthralled. Everybody just totally got into it. But we got out and one of the managers who would have been the hiring manager, what the hell does sewage treatment have to do with software?

[00:26:39] And I told him, I said, look, she’s a damn good engineer and she’s already demonstrated mastery of like engineering statistics and I think we can teach her how to code. And he hired her. He was grumbling and I ran into him. Ironically, after I joined Meta, we had an at scale event and he was there and he hit David, can I talk to you? He pulled me aside, he said, she’s killing it. It’s like, how did you know?

[00:27:06] I said, I can’t believe I was so wrong. How did you know? And I just told him straight up when you see a brilliant engineering mind, everything else doesn’t matter how fast they can solve leetcode, whatever do can they build like a Google type ahead, you know, scalable solution or whatever. Because these things can all be learned. They can be learned in a very short time. But you can’t teach that.

[00:27:27] And you have that young mind, hungry, willing to work hard and, and willing to make, you know, able to make smart decisions. So that’s kind of how I approached hiring back when I was, you know, when I was actually hiring people to bring in to work for me. That doesn’t work, that doesn’t scale. Right? You can’t do that at Meta. You can’t do that at Netflix today or Google maybe Netflix, I don’t know.

[00:27:49] You can’t do it at Meta. Google. Just a legal and, and, and you know, policy problems, you know, everything you’d have to worry about just, it would break down really badly. But you know, so we have to have the system, we have to have. So basically it’s like, hey, we’re going to give you a bar. And if you meet this bar, you know, higher. No, higher. Yes. Okay, level. This is what we can see from what we’re in.

[00:28:14] You know, and I’ve had people that they were sick somewhere else and they, they said, well, I got an offer of five at Meta. I was like, look better to come in under. And I said if you come, if you come to Meta as a five and start immediately executing as a six, you’re going to be blown away by your bonuses, your multipliers, your RSU refreshment. You’re going to be blown away by the way you get treated by coming in and really demonstrating that you’re under level.

[00:28:39] But if you get higher at that higher level and you come in and you’re struggling, you would have been a really good five. But you’re struggling to execute at that sixth level. It’s just not going to work because there is no way to fix that, right? If you get hired in a company like Meta or Google or I know probably any of the faang companies and you end up being over leveled, they don’t have a way for somebody, well, okay, you can leave or we’ll put you to a six because we hired you as a seven, there’s no way to fix that problem.

[00:29:06] So when in doubt, go low and work hard and get that quick promotion, get those bonus and RSU multipliers and you know, prove your mettle.

Ryan:

[00:29:16] You, you mentioned in identifying the, the engineering skillset, that fundamental Engineering skill set. How do you, how do you identify that? Like is there a question you ask or is there certain things you need to see?

David:

[00:29:30] I think it’s you interview to the person, right? Not to the process. So if you know we have a master’s civil engineer who’s expert in wastewater treatment and it’s like it’s a hard engineering problem and if you can demonstrate mastery of that problem and you can explain it to engineers that know nothing and you know, to the point where those of us who got it realized that like this is just a finite state machine, right.

[00:30:01] I mean after all what is what I mean what is almost engine. Any engineering I know when you get into structure and everything a little different, but so many systems, mechanical systems, right. You know, engines, right. Or whatever, they all end up kind of in some way connecting back to like states and transitions. And so I think when you see again we’re told we brought this person in as effectively would be like a new hire, a junior.

[00:30:26] We had a role like as a data analyst or something. But so we, we had an. If we would have brought them in as a full on software engineer with these expectations up here, they would have failed. So again that’s when you’re at a smaller company you, you create flexibility and you can actually hire and, and build people up.

00:30:52 — The strongest engineer he’s ever met

Ryan:

[00:30:52] Is there examples of the strongest engineers you worked with and what made them strong at Netflix?

David:

[00:31:00] I’m simply going to say the absolute best engineer, strongest engineer I’ve ever worked with is Giannis Cats of netis. I’m generally going to try to avoid names, but I got to put him out. I met him in 2000 in her video. I brought him in. He left the industry and was working as a professor in Greece teaching signal processing. And I brought him in during the summer break to, to help. I was working on this was in 2010.

[00:31:27] I was working on some ideas around parallel encoding and the. The work we did gave birth to content based encoding which is now how the entire industry is working that we wrote an early patent on that. He went back to school but we convinced him to join us at Netflix a couple years later. He took that one level further to now the model for encoding for both Meta and Netflix, which is called convex all encoding, which is actually mathematically delivering the highest possible quality for a given bandwidth capability.

[00:32:04] And he and I work together at Inter Video, Netflix and Meta. He’s an engineer’s engineer and one of the most brilliant people I’ve ever known. He actually developed a model for the basically for Kodak evaluation. It’s a cost benefit model. For this amount of energy you get this amount of Kodak efficiency and, and, which is revolutionary because he’s very much academic. And you know, and, you know, you always heard the joke, in theory, theory and practice are the same.

[00:32:38] In practice, they’re very different. So his brilliance, I think, was really bringing this academic world and forcing it into the practical world where we can actually make decisions based on not what some hypothetical, you know, you know, two videos you compress can do, but actually, you know, what happens when we run this test against 10,000 videos or a million videos, or a billion videos.

00:33:02 — Joining Meta

Ryan:

[00:33:02] You’ve mentioned Meta a lot at this point. Maybe we should go to that. I’m kind of curious. What’s the story behind you getting recruited to Meta or working at Meta?

David:

[00:33:13] Sure. So as I mentioned earlier, the Netflix culture deck, the early one, said that we really should interview often so we understand our market value. Right. And I had an outreach from Meta, and I don’t remember who it was, but it was like 2018, I think 2017, late 2017, I think it was late 2017. And I told my wife, I said, yeah, you know, Facebook, it was Facebook at the time. Facebook wants to interview me.

[00:33:42] I said, I don’t want to work there, but I haven’t interviewed in like 12 years. I think I need to interview or 10 years, whatever. So I went and interviewed and I, I kind of went over, I will say, honestly, a bit full of myself because, like, you would hear me. We built Netflix. We built the highest scale encoding platform in the world, whatever. And, and I got to the interviews and I was whiteboarding everything out.

[00:34:09] I remember one guy, they were good interviews, but this one guy said, david, you have to understand, Facebook cannot solve their video scale problem with CPUs. And I just, like, I walked away and I emailed him back later. I was like, you floored me. Right? I, I never realized. And, and so I, I, I, I turned. Eventually I got an offer and I, I turned it down. I just wasn’t ready at the time. But I had, I had reached out to this guy and I just told him, you know, I thank you for like, you know, letting me know that I was very narrow in how I understood the world of video because I had no concept.

[00:34:48] Right. And I want to talk about this, about Facebook’s Metascale in a little bit. I had no understanding or concept. Shortly after that, Giannis, who was in Netflix, pulled me aside and said, hey, I got this offer from Facebook. And we sat and talked about, he was talking about the asic, he was talking about all these things he was going to work on. And I, I saw a passion in him that I hadn’t seen in a little while.

[00:35:15] And, and he had done some really good work at Netflix. But I don’t think that the culture, again, is not built around individual contributions. I don’t think he ever, you know, was getting the joy and I think the reward and recognition, remember, recognition is not for an engineer. Is not your name in lights. Sometimes it could be as simple as like a senior executive saying, hey, this is really good work this person did.

[00:35:40] And I’m not saying they didn’t do that at Netflix. I’m just saying it like we’re talking about this. Meta is very intentional about individual credit. So. And I think he felt like the problems weren’t big enough for him and he was, he just, I just look. And I was like, wow, this is amazing. You know, he’s like, aren’t you going to try to talk me out of it? I was like, honest, you’re my friend, right?

[00:36:03] Is it? And I, as an employer, as your boss, I don’t want to lose you. But as your friend, I’m like, how can you say no? And I’m not even going to try because, you know, because I care about and I just, I wish I could tell you going there would be the wrong decision. And so through that process, I started with a company I was only interviewing just to work on my, you know, interview muscle. And in the end I was just looking across the aisle and thinking, man, that’s freaking hard problem.

[00:36:32] And I had a good, solid team doing well. I had a leader lined up ready to take my place and just the way things kind of worked out. And 2019, I ended up leaving Netflix and word got out and Facebook reached out to me very quickly. I had to re interview because the gap had been too long. But I got a job offer and, and I took it. I was excited and I came in really, really excited about that role. But I, but I will have to say, like, you know, it’s really easy if you have leadership skills and talent, kind of that innate talent to be a leader.

[00:37:14] It’s really easy to grow organically, right? The encoding technology team In Netflix was one person, me, and then it was two, and then 3, 4, 5, 6. Then I need a manager. 7, 8, 10. Eventually I had a director under me and, and a handful of managers and, you know, 55 people, what total? And you, you know, everybody, you’ve grown into that role. You’ve built trust. I came to a well functioning team.

[00:37:39] I was given responsibility for 55, 60 people that had the video processing team and they didn’t know me from anybody. And I feel like I came in like one of the first lessons I got and I knew, we all knew that we had problems. Everybody knew Facebook video platform, there were some quality problems. The scale had been met, the scale challenge had been met, which is a, the hardest problem. And they were looking for somebody to help them kind of let’s, you know, can we be a better video experience.

[00:38:08] I remember early meeting I started talking about some of the things I saw that maybe need to be worked on. And I got feedback like David, you know, first of all, because I told everybody I really value and appreciate direct and candid feedback, don’t ever be afraid to give me feedback. And I was talking about something and one of my people on my team the next week at one on one said, David, you say you like candid feedback.

[00:38:32] Can I be candid? I said, yeah. He said, yeah, you’ve been here two weeks, you’re already telling us what we need to do. And, and, and then we were talked about a little bit. And I said, so what I think I hear you saying is I need to shut up and work on building trust. And that’s what I had never, that’s a muscle I had never ever developed before because I built trust by working with people and growing over time and just coming in as a senior leader, as a director and you know, and they’re asking, the company’s asking, we need you to, to do change, you know, to help us, you know, improve.

[00:39:08] This is all. But you can’t just come in at the same and start. And I didn’t. And also I will say that, you know, I, part of that was ignorance because I didn’t really even fully understand I knew what some problems they were having. But I, I, so I, that kind of, I stepped back. I realized, look, first and foremost being a leader is about relationships. Ultimately we’re an engineering team, we have to make good engineering decisions.

[00:39:31] But if the, if the managers reporting to me don’t trust me, we go nowhere and I fail. And I realize it. So that was really good feedback in the moment or near the moment. And, and I spent a lot of time trying to sit, listen, build trust and, and, and I feel I was very successful in that over the time. You know, I’ve been told even now, you know, like the team, you know, there was a lot of respect there.

[00:39:57] They knew I respected Them, even if they were struggling or needed feedback, they knew I was going to give it to them. But, but, but, you know, they, they knew that I was going to tell them like it was and I was going to be honest with them. And, and so that was my first lesson. Like, it’s really hard stepping in as a leader of a large team that’s already doing well. If it’s a broken team, your job is to unbreak it.

[00:40:19] Right. But if it’s a team that’s actually executing well, your job is to not break it first and then help it move forward second. And I jumped in to move forward. So that, that was a really good experience for me. I really feel like, I mean, my saddest thing about Meta is that it was a last job because I feel like if I were to go take another full time job at an engineering company at a larger leadership role today, the value, the skills I learned at Meta, I mean I would just come in as such a better organizational and engineering leader.

[00:40:52] But I’m just kind of past the point of really wanting to take on that level of responsibility. So the only thing I wish I could have done, you know, if there was, I wouldn’t get, wouldn’t trade my Netflix experience, but there was somehow like maybe I could have had these two in parallel or something. It just the way it worked out or if all this would have happened but happened 10 years earlier, then maybe I’d be like, hey, I’m, you know, I’m 55 and I’m going to leave Med and I’m going to go join this other company to really help them, you know, build this engineering.

[00:41:18] Org. But because I learned a lot. I think I learned the most about engineering leadership not at Netflix, but at Meta because of the, the challenge of the role. Yeah.

Ryan:

[00:41:30] I’m curious, how do you, when you come in as such a senior leader, you mentioned building trust. What does that mean concretely? Like what do you do if you’re coming in as a senior leader to build trust with the existing team?

David:

[00:41:43] I think first of all, taking the time, and I did spend a lot of time in one on ones, but you know, you don’t want to get over your skis ahead of your skis. So really spending, you know, first of all, this is a very technical role or video processing even for the director. These decisions are being made. You have to understand at some level why you’re doing what they’re doing, otherwise, you know, they may not be successful.

[00:42:14] But you know, it really involved spending a lot of time in one on ones and talking and asking questions and making notes and understanding the fundamentals of the system, the interactions, understand the roles, the team, the people. I mean, one of the first decisions I had to do was like, because they reorganized when they brought me in and oh yeah, you need a manager for this team. I, you know, it’s an up level manager and you can promote or you can hire and you know, and here I am having to make a decision and I have mentors I don’t know or I can hire from externally and I’m just like, and that’s like ultimately I, and, and some people, why don’t you just promote one of your managers?

[00:42:56] And I’m like, well, I don’t know them. You know, I don’t just want to make a decision because it’s a politically correct. So I, I sat on that decision, I got into that trust building mode. I spent some time talking to them. Finally I said, okay, now. And they were right. There was already a manager on the team that would have been the right manager for the role. Because I told my boss, well, if you thought he was the right one, why didn’t you promote him before you hired me?

[00:43:19] Because now you’re asking me to own a decision that I can’t. That I don’t know. But I did take the time and it was a difficult relationship at first with that manager because I think he kind of felt like this was his role and why isn’t he giving it to me? He’s just going to hire somebody else. I have to quit. But we spent a few, a little time together and I, then I had a one on one that said, you know, I, I apologize for the time, but I just can’t make a decision.

[00:43:42] This is like huge, right? It’s your future, it’s my future, everything. And I don’t care what people tell me, I need to understand. But I said, but I do believe at this point I can say with confidence, you are the right person for this role and I want to move you into this role and that kind of, you know, and then, you know, by the time he left Meta to do something new, we had gotten past, you know, all the trust issues.

[00:44:05] He understood who I was, you know, you know, I don’t walk on water. But he knew, like he, he no longer assumed the worst. He always, well, wait a minute, I know I can trust David if I’m understanding this, I know that’s not. And he would tell me, hey, you said this what you mean? And so we ended up having A very good trust relationship. Right. And it was really just all that was about slowing down.

[00:44:24] Right. Not making any rash decisions, not getting ahead of yourself, not telling the team what they need to do, and, you know, just like, absorb, absorb, absorb. And I feel like even like if I could go Back to my 2019, David, just joining Meta, I would. Even the investment I made was, I don’t feel was adequate. I was like you that first six months. I should have just told my boss, I’m in school, and if there’s a critical decision, you’re not going to have to make it together.

[00:44:54] Because I need to understand better what my team is doing. I need to know the people better. And. And so I would spend even more time both technical and people understanding that. But I. I do feel like that early feedback kind of helped me correct and got me to a place where I was successful.

Ryan:

[00:45:14] I see at Meta, you mentioned the scale was really unique. I’m curious, is there any favorite work or favorite project that you did at Meta?

David:

[00:45:26] There’s a couple things that I talk. I already talked about the scale, bringing that Netflix quality into the Meta scale, which we were successful. The second thing really we have to talk about is Covid. So you remember my world at Meta. I joined in late July 2019. By March, nobody was coming to the office, not even nine months into my job. We’re all working from home. And to make matters worse, you know, we.

[00:45:57] The video for. I think it was all of Facebook, but especially video. New Year’s Eve was like. Like, we started planning for New Year’s Eve in the summer because every New Year’s Eve, we would see this huge spike in traffic for 24 hours. You just like, we know, you know. And I guess before I joined, perhaps New Year’s Eve was a system breaking, but by then, everybody was planning to make sure we were successful in New Year’s eve.

[00:46:23] Well, by April 2020, every freaking day it was New Year’s Eve in terms of video upload quantity or volume, in terms of streams, Live, this life, every system we had, and we’re watching these systems slowly bleeding out, right? Because we had never run at that kind of sustained load. We had some race conditions and all these things. And I’m just like, the whole freaking world, you know, the Prime Minister of India was addressing his country regularly using Facebook live streaming, and the whole world is like, maintaining their sanity and continuity and connection using our apps.

[00:47:00] And so that was just a huge, huge. I mean, I was terrified. They did. The weight, you know, the weight of that Realizing I realized that the connection was a lot more than video, but video was like huge. And, and so. But, you know, but we all, you know, from our homes, we all got in and put our heads down and we, we found all these problems. We ironed out the wrinkles in our system. We resolved these race conditions and got to where we were chugging along at New Year’s Eve every day.

[00:47:29] And the systems, after about a month, systems were all running, metrics were holding. Everything was really good. And so that was really proud of that. And I think when I joined Facebook, I didn’t really understand the value proposition of the company. Now I had Facebook account and I shared pictures and social and yeah, some videos now and then I didn’t really understand because I was feeling like it was a very kind of like, I.

[00:47:57] My us centric view of the company and it’s just a social app that we all share our stuff on. And I learned in relatively short order. First of all, I understood what it means. If your customer base is three and a half billion people, that means that half the world is on your platform. It means that. And again, I’m in no way disrespecting Netflix. Phenomenal company, phenomenal product. But Netflix has 300 million people.

[00:48:29] So you could roughly say that’s the top 5% income in terms of income of the world. The wealthiest 5%. Meta has the wealthiest 50%. And that wealthy is kind of a loaded term because at the bottom you’re talking about people making six to eight dollars a day and there and, and Instagram and WhatsApp in particular, or Facebook. This is their lifeblood. You have little mom and pop stores all through India, you know, and, you know, I’ll be in a Uber.

[00:49:02] And the driver, it was actually very early on, I was going to a business meeting in Seattle, and the Uber driver, you know, where do you work? I said Facebook. He’s. Oh, my gosh, I love Facebook. You know, he’s from Somalia. You know, my family, we’re all across the world. And if it weren’t for WhatsApp, we could not even. We would totally lose touch. But WhatsApp keeps us connected. And I’m just like.

[00:49:27] So it was this realization that unfolded over time of what a phenomenal product and product this was, but not face just Facebook, a company, but how important our work was. Because again, when people talk about social in the US that, you know, I don’t care if it’s TikTok or, or Facebook or Instagram or, they always kind of talk about the Good and bad side of this, you know, fake news, whatever, bad video, whatever.

[00:49:52] But we are totally oblivious to how we are improving the lives of people in the rest of the world. And when I learned that, I just, I realized it’s like, you know, I’ve always loved doing what I do, but I’ve, you know, all of a sudden the work I do is making a difference. I realize nobody’s going to in India’s going to say, wow, David Ronke really changed my life. But in a small piece, right, that, you know, they will say that the company Meta changed their life.

[00:50:20] Gave them a business where they could support their family, gave them a platform where they could share social videos and make money, you know, and improve their standard living, you know, gave them a way to stay in touch with their family, distribute around the world. They will say the company made a difference in their lives. And I really liked being a part of that. I, I really, at the end of the day that, and I try to explain this to people who don’t work for the company or they’re griping about this or that or, you know, whatever.

[00:50:43] I suppose you just don’t understand, right, how good it feels to know, right, you’re making in a small, small way, you’re actually making a difference in the world.

00:50:52 — Near death experience

Ryan:

[00:50:52] Coming to the end of the interview, I wanted to do some reflections on your career. I think first question I’m kind of curious about is when it comes to work, life balance throughout your career, how did that change throughout the different roles you had?

David:

[00:51:07] I had the first really visionary executive that I was working for in the late 90s. You know, she gave us, she sold us on something we were passionate about. And we all worked hard. And I was working hard not because the expectation was there, but because we were trying to catch up. And we did. But when I went to the startup, the expectation was when I was at Inter Video, it was a 247 expectation. And I was going to work usually at 10, 10:30 in the morning.

[00:51:41] Coming home, usually an early night would be ten at night. A late night would be like sometimes two or three in the morning. And you oftentimes seven days a week. And you know, we tried to get to the ipo and then we got the ipo, but we got to get these, these new wins, bundling wins to get revenue so that our stock price will be up or whatever in January of 24 or 2004. I started feeling ill on a Sunday and was not getting better through the week.

[00:52:17] On that Friday, I went to an emergency room. I was having a serious, serious problem. My wife took me to the emergency room where I was immediately diagnosed. Even before they had proven it, the doctor told me this is colon cancer. And I was like, my digestive system was blocked and it creates a very catastrophic quick death situation. So I just, I basically just fell off from a health perspective in one week.

[00:52:49] I had been driving myself, driving, push, push, push, and boom. I just fell off a cliff. And I woke up, I was in the hospital Saturday, Um, and I was on machines and they had gotten me stable and they were going to take, it took a week to get me physically strong enough to have a surgery to remove the cancer. Um, and I’m laying in there in the hospital and I know this kind of sounds cliche, but I was a dad.

[00:53:14] I had a two year old, a four year old and a seven year old. And when you work those kind of hours, you know, my, my, my, my older, my daughter knew me reasonably well. We had had some time and some relationship. My oldest son we have, but my youngest didn’t even know who I was. I mean, he knew who I was, but I was just a guy who come in, you know, and you’re sitting there literally thinking you’re at the end of your life.

[00:53:40] Because I was, I was stage three. It had lymph node metastasis. When I finally got out, I had two surgeries, chemotherapy. But when I get out of the hospital three weeks later, I was in for three weeks and multiple surgeries and everything to recover. My five year survival prognosis was about 25%. And so while I was in the hospital reflecting, I realized that like, like this is wrong, right? And I, you know, this was only four years into inner video.

[00:54:09] And so when I came out it was like, I can’t do this anymore. And I told my boss in her video, look, I, you know, I still want to work here, but I can’t do this. And even, and I realized it not just because I was sick, this is you simply, if you’re working in a company where they put those demands on you and those expectations that if you’re not there late, you’re going to get fired, the best thing you can do is leave, right?

[00:54:32] And they were actually using work hours to compensate for really bad leadership. So I actually did some of my best engineering at Inter Video after I got out of the hospital because I had to get back, ramp back up to full time. But I have work life balance thrust upon me and from that point forward it became something that I was passionate about that I was never going to be a leader that put expectations on people that you kill yourself for the job, you know, and literally that’s what we were doing.

[00:55:02] Even though most people don’t die, they’re, they’re, they’re, they’re selling their life, they’re trading their life, their experiences, their children, whatever, anything else they could have for a job. And there’s no job that’s worth that. And that’s where I appreciate when I got the Netflix, you know, that was a shock for me, for them to tell me in the interview that we don’t value 27 word, 24.

[00:55:23] 7 word. And, and if you come in and work that way, we’re probably, that doesn’t mean you’re going to be successful. But if you come in and make really good decisions in the time you’re here and go work, enjoy your weekends and your time away, we’re going to love you. I’m just like, wow. And I realize that, so that’s now part of my DNA. And I think that what I would like to say is for everybody else that happened to be watching this, don’t wait till you get cancer or have a heart attack to wait for you to realize work life balance is, is, is, you know, critical.

[00:55:56] Bake it into your DNA at an early age. Learn how to take time off, take vacation, leave your company alone for a week, whether it’s going to kill them or not, and they can’t survive without you for seven days. That’s not your problem, that’s their problem. And the only way you can make them understand that problem is for you to leave and make them face that problem and fix it. And so, and, and, and then I, I, and I understand like, because, you know, having worked at Meta, at Netflix, these are performance oriented companies and a lot of people I think feel like, well, you know, you have to work.

[00:56:28] If you don’t work hard, if you don’t work all the time. But I think what we’re doing is, we’re misunderstanding, right? Remember, there’s a poster at Meta that I just love. It’s a rocking horse. It is a picture of a silhouetted rocking horse and it says, don’t mistake motion for progress. And I think that in order to have work life balance, we need to understand about our role and we need to work with your boss on this.

[00:56:58] What are the most important things I need to do? What are the biggest impacts I can deliver, deliver to Netflix or Meta or Google? What are the things that really, really matter? And the other stuff, just let your boss know, I’m not going to do this and if it matters to you, I don’t have time to do it. And if it matters, we have to get somebody else or you have to help me adjust my priorities so that I can get it done.

[00:57:19] The worst thing you can do is simply like if your boss is not giving you reasonable goals and not planning giving you clear context around what’s expected for a review cycle, the worst thing you can do is just try to, you know, do everything there because you end up doing work that maybe wasn’t that important anyway. And so we do have to own our own. You know, to a certain extent I realize this is for somebody who just got out of college, so maybe a little hard to think of, but you really do have to learn how to own your time and manage your time and make decisions because you can’t do everything.

[00:57:51] It as a in out of work, you may want to, you know, do marathons every week and do all this other stuff and whatever. You can’t do everything. You have to start picking the things that really matter. It’s the same way in work. And I do believe, you know, as engineers, you know, anybody’s working in a, in an environment like Google or a faang company, there are going to be times where we have to put our head down and crunch.

[00:58:15] It may mean working weekends, it may mean working late in the evenings, but that has to be the exception. Not, not the rule. It has to be the exception. You know, you know, you’re getting up, you’ve been working on something all, all half you’ve been got your head down. Now you’ve got, you know, you’re getting to the end, you’re about to roll this feature out. You may have to spend a little extra time getting it out.

[00:58:35] It’s like, okay, so I’ll plan for it. Like, I know that. Hey, and you know, like in, like, like sometime May, June, I’m going to have to get my head down. I may have to work a lot to get this across the finish line, but then I’m going to pull back out and then I’ll take a vacation or what. You know, you have to have that mindset. And again, you always have to make sure your manager is giving you clear context on what really matters and focus on those things.

[00:58:57] And I think you find work, life balance happens even at a challenging high performance company like Netflix or Meta.

00:59:04 — Where he learned the most

Ryan:

[00:59:04] When you look back on all the companies that you worked on, I’m curious, which one do you think you Learned the most at or which one you know taught you the most and why.

David:

[00:59:15] My last four jobs each did their piece. The first job of first of the four turned me into a full fledged high performance software engineer. I mean performance by the problems you solve, not by the hours you work. The second job gave me a really good video foundation. Netflix, you know, was a zero to one right. People who were at Facebook from 20082012 understand if they’re still here, understand how you know that early company where you’re building a zero to one product versus you know, working at a mature company.

[00:59:48] So I, I, you know I think I learned a lot about leadership, about hiring, firing and, and, and context setting context and executing and cross functional. These things I all learned at Netflix and brought to Meta. What I came to Meta lacking was an understanding of objective like planning reviews and performance and having an objective process. And I, I, I mean PSC time was both like the most dreaded time of the year for me and my favorite time of the year right Is one of the things I really and I, I, I mean this because you know Meta is over indexed on individual credit and the whole like you know you’re everybody writes their self reviews and their manager turns it into a review, a performance packet, it brings in feedback and you know, lines up things with goals and, and then that’s brought into a large group and of what we call calibrations where you’re, you know, people will challenge, right?

[01:01:07] It’s because every manager thinks their team is the best team on the planet and their engineers are the best engineers on the planet and that the reality is where other managers, you’re convincing other managers not through arguments or you know, the pushing hard but through data convincing them that this engineer deserves this level and, and that they were the ones to get credit. And we would argue a lot wait, this project, who worked on this project, who did what, who gets the credit?

[01:01:37] And like that’s it’s something we take so seriously and, and to me was so important is, is like taking the time and having the conversations across a larger team of leaders and understanding how individual engineers, how they impacted, how their impact landed, how they move the needle, how they, you know, how they brought other engineers through, how they led projects and making and getting that credit down on paper.

[01:02:06] When, when you leave Meta the one thing that you get to bring with you is every single performance review package you was ever written. Everything I did at Meta, every negative feedback you need to get better, every great win is all cast in stone. Not only do I have that. 2000 years from now when they’re unearthing like they find Meta hard drives, they find my stuff, they’re going to know what I did at Meta, right?

[01:02:30] That’s how important Meta takes individual achievement. And I think that to me it’s one of the best things about the company. There’s a lot of great things about Meta. I feel like I had always heard horror stories about oh, the PSE performance curves, everything else. And then I got into Meta, I was like, this really matters. And so at the end of the day, being able to the greatest joy I ever had was having one of my direct reports and handing them a redefines expectations rating.

[01:03:02] There’s nothing as a leader. It doesn’t matter that my rating was me. It’s all I had like 2 res on my team over like over the course of a year. I mean this is like, and, and I have ges and, but even the ones that were not doing well and I had to give them like, hey, you know, they’re below expectation, you know, trying to work with them to get back up. That, that all is so important. It gets back to the people thing.

[01:03:24] Right. And so I really value that work. That’s what I learned at Meta more than I, I learned a lot. I learned, I used to think scale was, you know, I read books and took classes and you know, xyz, you know, sharding, partitioning and microservices and I get the Meta in all those books. These guys don’t know what, they’re good books. They really didn’t know what they were talking about. Nobody understands scale at that level except for a few companies that are dealing with like, you know, billions of users and all the.

[01:03:52] So anyway, these are the two big things I learned and to me the most valuable one and the one that I think that I got from Meta, that I would bring to the next job if I was to take a full time job, was how to establish a system for objective planning, review and performance.

01:04:09 — Book that impacted his career most

Ryan:

[01:04:09] Is there a top book that had an impact on your career? And if so, what is it?

David:

[01:04:14] Yeah, so I’m really going to date myself here but the book that I immediately came to mind when I saw this question was 12 secrets to Microsoft Success. This book came out in the late 90s and there was some really, there was really some radical thinking. I mean, you know, I mean we look at Microsoft as a very different company that was in the 90s. In the 90s everybody thought Microsoft was own the world and everybody’s, you know, Windows everywhere.

[01:04:40] But there was, there was some really radical thinking at that company, right. And they’re baked into their culture, right? The, the concept of betting the company, right. And I saw that at Netflix, right? Amazon, Meta, right? You see, I. Not ne. It’s not like you’re doing a casino bet all in, but you see a big opportunity, you’re literally betting the company on it. The whole thing of like, you know, let’s quit saying failure is bad, right?

[01:05:10] There is such a thing as good failure, right? And you expect it. And if a company’s executing well and healthy things are going to fail, projects are going to fail, people are going to fail, engineers are going to be working on something, it’s not going to work out the way they want it. And, and the takeaway from that, right, you can fail on a big engineering investment in Meta. It’s not that you fail, it’s how you fail, what you learned and what you bring forward so that people can see that, yeah, this was a rational bet, was a reasonable bet, and they did it and it didn’t work.

[01:05:39] And they’ve really handled it well. They managed the risk, they got out early. So you can, failure can be success, right? If you. Based on how you manage it. Microsoft was the first company to publicly state a policy of hiring the top 5%. Their target was to hire the top 5% of all software engineers in the world. And they set up their interview and screening, which is kind of a predecessor to what we do now at the faang companies.

[01:06:08] And I think that that was really revolutionary. Dogfooding, right? The term dogfooding was invented by Microsoft. They talk about that in the book right before people would work on projects that products they never saw, they never ran, they never tested, they didn’t experience bugs. When Bill Gates forced everybody on the NT team, that whole company is going to shift to NT Server. When NT Server was failing badly, put all the bugs in this os in the, in the like 95, 96 in front of the entire company.

[01:06:37] And all of a sudden this, this operating system that couldn’t ship started getting fixed, moving, moving, moving. And then the crazy thing that they talked about in that book was somewhere between Windows 95 and, and when the book was written, when Microsoft was growing leaps over bounds, Bill Gates instituted a company wide hiring freeze.

[01:07:02] And, and it’s like what, you know, shocks you? Well, he felt they were hiring too many people too fast and they were investing too much money in things that didn’t matter. And if you, you is you’re too old for this probably. But Encarta Bob, if you remember Microsoft Bob, these silly little animated icons, all these silly things that Microsoft was doing that were pointless. And as soon as he instituted a hiring freeze, he forced his executives to start killing projects and putting the people on stuff that matter because he felt like hiring was getting out of control.

[01:07:36] Like, we’re hiring too many people, we’re just putting them everywhere. People are working on things that don’t matter. We’re losing control of the company. And he instituted a hiring freeze that basically they ended up cutting like the bottom 20, which I should have done anyway, you know, healthy businesses, and we’re actually much better at that now. Meta’s pretty aggressive at cutting, you know, projects that aren’t proving out that didn’t happen back then, right?

[01:08:01] Companies, like once you’ve got this established team and people and your little thing, things just didn’t get cut. So that was pretty radical. So I think, you know, Microsoft introduced some engineering leadership concepts that really, I don’t believe existed before Microsoft. And you know, Amazon, Google, as much as we all tried, Netflix, much as we all tried in the early days and not be evil like Microsoft, we ended up taking a lot of their.

[01:08:32] That was Google’s motto, by the way, don’t be evil. Because Microsoft was considered the evil empire, the dark side, you know, whatever. But we all took. They actually were the ones that kind of established some of the early, like model for modern engineering. You know, Microsoft brought us some of these engineering. Amazon brought us the concept of, you know, rather than having very expensive, highly reliable hardware, let’s have really cheap hardware and design around failures, you know.

[01:09:04] And so these things we all kind of Google brought, we all picked that up and it all kind of worked together. But I feel like Microsoft was the first one. And that book I read just stuck with me all the time. Even when I see like before 2022 and the big crash, I was just looking, I was like, we can’t keep hiring people at this rate. It doesn’t, the math doesn’t work right. It wasn’t just us, it was everybody.

[01:09:27] Because during COVID we were all like, yeah, we’re growing higher, higher, higher. And everybody accelerated their hiring. But it’s like nobody decelerated and it’s like, boom. All of a sudden we had that contraction and boom. So everybody had to reset. So I think that had all the FAANG leaders read the 12 secrets to Microsoft success, they would have recognized in like late 21 that the hiring was out of control and Maybe we need to slow it down.

Ryan:

[01:09:54] When you look back on your career, is there anything you regret that maybe other people could learn from?

David:

[01:10:04] A few regrets. I think my early years, I think my first six years out of school, which was actually a long time, was wasted at very large failing companies. The, even if the, like General Motors spun off Hughes, but Hughes failed. IBM sold off the division I was in to Siemens, but that division failed. I mean, I think that, that, that the only regret I think would be like, you know, making, I was making a good salary, I got an easy job, whatever.

[01:10:35] Not really thinking about, like I was only thinking in the moment and I, I really feel like I kind of wasted that time. I wasn’t really thinking about growth and I do think we need to think about growth. We don’t need to stress about it, we don’t need to be like ambitious, like I’m going to be a vice president and like 10 years or whatever. But we as software engineers especially, I’m only talking about software engineering now, not leadership.

[01:10:59] Everything we do is going to be obsolete five years from now, right? When I joined Meta, the boot camp, the whole, the, the whole system for the, the dev environment, everything is completely replaced. I went through boot camp as a director who was pretty proficient. And when I go to ic, it’s like, oh, what is this? Everything’s changed, you know, so everything’s going to be obsolete. They’re getting new technologies and the only way, and this is what I believe I have been successful at, you know, is always pushing forward and trying to stay to the edge and always learning more, learning new things, new engineering skills.

01:11:33 — Advice for his younger self

Ryan:

[01:11:33] And the last thing I want to ask is if you could give yourself advice at the beginning of your career, knowing what you know now, what would you say?

David:

[01:11:42] Try to work on hard problems, right? If you’re work, if your job is as an engineer, if your job is not difficult, you’re not growing. So, you know, maybe if you been working on something and it was really hard, now it’s kind of flatlined. Enjoy a little bit of rest, think about it. And then, you know, and always speaking like if you’re a Meta, for example, where it’s very easy to move, if you’ve gotten to where you’re feeling your current role is not very challenging, there’s so much going on at the company, go find something else.

[01:12:10] Get back into a hard problem, right? And, and, and push yourself forward because it’s the hard work, the hard problems that move us forward, right, and, and help us grow. And then I think you know, to a young engineer joining companies, the size of the company is irrelevant sometimes everybody should work for a small company at some point in their life. I believe startups, everybody should try a startup, right?

[01:12:38] Especially it’s easier when you’re younger than when you’re, when you’re older. But be careful not to get too enamored by the technology if you join a small company, a startup, or even a big company, like, like, you know, meta. I mean, Mark is an incredibly visionary leader, right? And you know, he’s navigating this world and trying to make decisions and keep the company focused, right? So. And Reed Hastings was an incredibly visionary leader, right?

[01:13:09] But Reed Hastings was a visionary leader in 1999 when Netflix had four people. So really what you want to do is make sure you’re working for visionary leaders. And not just visionary, they need to be able to execute as well. So vision is good, but if you can’t execute against that vision, you go nowhere. So any company you’re working at, make sure your leadership is really looking forward, thinking forward.

[01:13:35] And again, to read Hastings Point, you know, making you dream of sailing the seas and not just, you know, cutting lumber for a ship. Again, work, life, balance. You’re out of school, you just are your new job, you need to plan, be intentional about not working, about taking time off, about making sure you’re, you’re, you have a life. Work, life, balance, right? It’s work, it’s life. Their intention.

[01:14:02] We have to work to have a life, but we have to live in order to work. And so you want to try to keep the balance is trying to keep those two where one doesn’t dominate the other. And, and, and actually you find as an engineer, because we love working on problems, when you get to that place, you’re actually doing really good work. You love what you’re doing, but you’re also enjoying time off. You know, you’re enjoying your family or whatever it is you like to do in your time off.

[01:14:25] And the last thing I will say, which is complete, this is for anybody, when you get out of college, when you’re early in your career, this is the last thing I would have talked to myself about, is your financial planning should start on day one. Because what’s going to happen, not just an engineer, but anybody who’s successful as a, as a, you know, as a technical leader, a contributor, a leader is over time you’re going to make more money.

[01:14:57] And if you don’t have a thoughtful plan about, about how you’re going to allocate Your spending, living expenses, car, recreation, short term savings, long term savings. Then as you make more money, it’s just you’re not going to know where it’s going and you’re going to miss the biggest opportunity you have, and that’s that. To bake into your DNA, this whole model of spending, saving spending and thoughtful financial management.

[01:15:24] And if you’re thoughtful in your 20s, with just small regular contributions to long term savings and managing your budget and not getting over your skis in terms of, of extended credit or debt by the time you’re in your early 40s, you won’t necessarily be able to retire unless you got lucky like joined a Facebook or Netflix early on. But you’ll be financially independent to the point where you don’t.

[01:15:54] Money’s not an issue anymore and you could take a lesser paying job, you wouldn’t have to worry about what about the mortgage, what about, you know, the college or what about this or that. But it all starts from your first day on the job. So the last thing I will say is, is, is be really, really smart and thoughtful about your finances from day one and make sure you’re ready like when you’re getting into your late 40s and early 50s, that you’re actually feeling pretty good about your financial future.

[01:16:21] Like, hey, I could retire or if I lost my job, we’d be okay, right? I could go do something else at half my salary and we’d be okay because we’ve, we’ve set ourselves up for that. But if you get into your 50s and you haven’t planned, it’s not, your 401k alone is not going to do it right. If you have, and Social Security is certainly not going to do, if you haven’t planned, then you get in this situation, it’s an unfortunate situation where a lot of people I know have found themselves where you’re kind of stuck and you’re having trouble getting new jobs, you can’t work at the, with the energy level you used to have and you’re not sure, you know.

[01:16:52] You know, my joke was always that if I didn’t get retirement right, I’d be a greeter at Walmart, you know, but it’s not really a joke, right? But, but you know, you’d be surprised how many engineers have been working and making a good living, you know, and they’re now in their 50s and they have not had any rational plan for retirement and they’re in no position to retire. And there’s really, as an engineer, there’s no reason for that if you, if you’re a good engineer and you’re gainfully employed for 30, 35 years, there’s no reason why you should not retire very comfortably.

Ryan:

[01:17:25] All right, well, thank you so much for your time today, David. I really appreciate you sharing your career story.

David:

[01:17:31] It’s been a great conversation. Ryan, thanks for having interest in feeling like I have something valuable to share. So thanks a lot.

Ryan:

[01:17:38] Absolutely. And is there anything you want to direct people’s attention to? Not at the end of the conversation.

David:

[01:17:45] You can find me on LinkedIn and I’m happy to have, especially for younger people. I’m not a financial planner and I’m not going to plan your finances, but I have about 10 slides I put together to give you a starter. And if somebody wanted to talk about finances, I would give them a few my thoughts on how you could start making smart financial decisions as a, as a new graduate, you know, young and even if you’re in your 40s and start early 50s and starting to stress out about your prospects for retirement, I could talk to you about that.

[01:18:15] If you’re working, you know, if you’re a people leader and you want to talk about performance planning, reviews, hiring, firing, happy to have those conversations. If you’re an engineer, happy to have those conversations with you talking about your career growth.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:47 - His middle school company

00:03:50 - His first project and promo at Meta

00:10:03 - Why code review is undervalued

00:12:42 - Senior Staff (IC7) promo story and project

00:19:26 - His major failed project

00:26:35 - How to handle a failed project

00:29:04 - Thoughts on management

00:31:35 - Technical depth vs breadth

00:33:32 - IC9 expectations

00:34:46 - Senior engineers he admires

00:37:39 - Advice for his younger self

Transcript

00:00:47 — His middle school company

Ryan:

[00:00:47] I saw that you have something on your resume that says Cosmic Soft, which started in 2000. And if I have the math right, that’s like middle school for you. So what was that? And can you talk more about the story behind Cosmic Soft?

Adam:

[00:01:07] Yeah, you dug deep, man. I gotta take that off of there. That is indeed my middle school software company. I discovered at an early age that I liked writing software. My mom was a teacher and I would often spend time in her classroom after school, just like messing around with computers. And that gave me the inspiration for my first product to sell, which was online testing software. I wrote it in a tool called Real Basic, which was basically a cross platform Visual Basic inspired language.

[00:01:41] So I was actually writing Basic, which. Oh, God. And yeah, sold it on the Internet to like tons of different countries. It was a lot of fun.

Ryan:

[00:01:51] I was not a programmer at that time. Can you give some more context? Is that something that’s running, I guess, native on like a Windows machine or something or what? What is that?

Adam:

[00:02:02] Well, Real Basic was this. This product from a different company, not Microsoft, and it was inspired by Visual Basic. Very clearly Visual Basic esque, but it was cross platform, meaning you could design the software on any platform. I used a Mac. The concept behind Visual Basic, for those who haven’t used it, is that you basically have this like blank canvas and you can just drag and drop buttons and text fields and all this stuff on it.

[00:02:23] Very easy to get into. Right? Because even nowadays with something like SwiftUI or React, you’re writing code that describes your user interface. There’s a separation there. Whereas Visual Basic, you’re like, well, I want a button, drag, drop. When the button is clicked, I want it to close the screen. Okay. Double click on the button and you write some code that’s like window close. So really easy to get started with.

[00:02:44] Not very scalable. And, you know, basic is a horrifying language, but it was a fun way to get started and really easy for me, as basically a middle schooler, to write an actual software product that was useful.

Ryan:

[00:02:56] How are you selling that on the Internet? Because at the time they didn’t have Stripe or anything like that. Like, how do you even receive money from these people? In other countries there was a service.

Adam:

[00:03:06] Called Ecelerate, which was like a proto Stripe. It allowed you to create an online store for your software. It would help you create, you know, like a code that you could enter to unlock the software. And so people would go to this website, they type in their credit card info and they would get a code to unlock my software. Fun fact. I also accepted payment via check. So for a while, as like an eighth grader, I was getting checks from all over the United States from teachers.

[00:03:32] They’d mail me a check for 1995 and I would send them via email a code to unlock their software. Good times.

Ryan:

[00:03:38] Did your parents know you were selling software to strangers on the Internet?

Adam:

[00:03:42] They did. I don’t know what, I don’t know what they thought of it or what they could make of it, but they did know I was doing it.

00:03:50 — His first project and promo at Meta

Ryan:

[00:03:50] Okay, that’s so cool. But yeah, let’s get into kind of like the main part of your career, which is you joining Facebook. You joined the company to help with the native app rewrite from HTML5 and you got promoted to the staff equivalent or E6 very fast. Were you hired as a E5?

Adam:

[00:04:09] I’m pretty sure I was hired as an E5 because I had industry experience. I was self employed, but I, you know, like, wrote software in the industry for iPhone. And yeah, it was funny because Facebook was just a totally different company. I joined just before the ipo, like literally weeks before the ipo. And so it was already a large company, but compared to what it is today, totally different. All of the iOS engineers could meet in one conference room.

[00:04:32] And we did. Right. Like every week we had a stand up and like we talked through what we did that week and all that stuff. And yeah, it was a fun time to join because we were totally rewriting the app. There was this cultural shift away from face Web, this like HTML5 attempt at making a native app and towards the native code and the native Engineers were very much like ascendant, right? Like, we had won the battle and now we were.

[00:04:54] We needed to prove ourselves that it was going to work out, and it did. And that also meant a lot of big prop, a lot of big opportunities, a lot of big problems that needed solving because our code base was brand new and we were discovering all the ways where it was going to fall over as we continued making the app bigger, adding new features. So that’s what I walked into in 2012, and it was definitely a fun time.

Ryan:

[00:05:16] One of the first projects that you worked on was one of those scaling challenges in adopting native code, and that also happened to be your E6 promo. Could you talk a little bit about the core data problem and why it was E6? Yeah.

Adam:

[00:05:31] Core data is Apple’s ORM database framework. It’s great if you are a startup that has, like two engineers and you just need a small little way to store data, it works great for that. We already had, like I said, we had enough iOS engineers. We could all fit in one conference room. So we started with like 15 to 20, maybe by the time I got there. And we were rapidly growing, right? Like, soon we had 100, soon we had 200.

[00:05:55] Core data falls over completely at that scale. So we needed to swap out how we store data. But this was also a really critical time, right? We at this point just launched our native rewrite using core data. And so everyone wanted to add features. The groups team, the events team, the pages team, they all wanted to, like, rewrite their stuff in native and get all the great wins from native code. Meanwhile, we on, at that time, mobile was a separate org inside of Facebook.

[00:06:20] We on the mobile team were like, this is not going so well. Like, core data is going to fall over tomorrow and everyone’s knocking down our door being like, please rewrite in native. So we had to find a way to do it incrementally. We needed to find a better architecture, which was what we, you know, found along the way. So that’s where we were in, like 2012, 2013. And we called our new solution MEM models, which was an immutable model system, which made it a lot easier to reason about thread safety, mutations, et cetera.

Ryan:

[00:06:46] You had this new offering that, you know, had a lot of benefits to it, but still, I imagine you had to convince other teams to adopt it.

Adam:

[00:06:55] So, you know, the core features of the app feed, et cetera, were already bought in because the performance was miles better and they didn’t require any convincing. It was driving it all the way to the finish line, getting the last regular products to onboard that was harder, right? Because for them they were like, yes, our sort of performance is a little bit better. Who cares? We don’t care. We were more interested in other things.

[00:07:14] Also, it wasn’t necessarily individual teams that were skeptical of the entire project, but we did get a lot of pushback from Apple Y engineers, for lack of a better word. Like engineers who believe in Apple’s built in solutions and are like, why would we not use core data? Like I’m an iOS engineer. We should use core data and everything else vanilla from Apple. I think that’s lesson to someone over the years.

[00:07:34] Just because Meta and Facebook and Instagram and all of our other properties have gotten so large, it’s clear that Apple solutions don’t scale to that size. But at least in 2012 and 2013 there was, there were a lot of people still clinging to this idea that hey, we should just use the vanilla Apple frameworks and we needed to convince them over and over and over, like, here’s why, that’s just not going to work.

[00:07:54] So that was a big challenge for us at the time.

Ryan:

[00:07:57] What did you learn about influencing other engineers without authority? And do you have any tips on how to do that across like a large code base?

Adam:

[00:08:06] My number one tip is talk in person or via video conference if possible. You spend a lot of time trying to make your writing clear and accessible and have a good tone and you can often convey that tone way faster in person. My other feedback is to be sympathetic to their point of view.

[00:08:24] Like my. I would always start the conversation saying like, yes, I prefer vanilla Apple frameworks too. We’re not going down this path just because I want to invent a new framework. Here are the specific reasons why we think it can’t scale. Give data, give actual. You know, we were getting to the point where we were disassembling core data’s source code because it’s not open source to understand what is it doing, why is it taking so long to initialize.

[00:08:48] Having that data to give to other engineers is really useful, right? Like, hey, you don’t know how core data works because it’s a black box. We know how it works. Here’s how it works and that’s why that’s not going to work for us. So, you know, I guess that sums up as do your research so that you can be convincing in that fashion. And the final one for me personally is just do the work for them. I think there are some engineers who would go about this project in the sense of, hey, I built the scaffolding.

[00:09:12] Now my job is to convince other people to do the work of migration. We call this archetype coding machine. I have mixed feelings about archetypes in general, but, like, in general, the idea is, like, I write a lot of code. I like to write code. So my thought was, I’ll just go do it for them. It’s one thing to convince them, hey, you have to step up and do all this work. The other is like, if you show up and you’re like, hey, I did the work already for you.

[00:09:33] Here are the benefits. I just need you to sign off and say, yes, this is fine. In our code base. Much easier conversation. That mode of operation was very successful here at Meta for a long time. I think it’s getting harder now. Dart code base is so big that one person can’t necessarily do the entire migration. You require some more alignment. But for a long time, that was a very good way to operate because you just show up and be like, hey, I solved the problem.

[00:09:57] And people would be like, great, love that you solved the problem, and I have to do nothing. I just say, yes. Worked really well.

00:10:03 — Why code review is undervalued

Ryan:

[00:10:03] I saw, like, in the career note that you wrote about this promotion that you’d also reviewed over 1,600 diffs and a half, and that comes out to around, like, 14 diffs per workday. Why did you spend so much time also reviewing code?

Adam:

[00:10:22] I think code review is super undervalued, and I still do a ton of it. It’s really valuable because you can influence other engineers in an organic way. Right. Someone’s like, adam, what. What’s your message to other engineers at the company? What do you want to change? I’m like, I. I don’t know. But if I see a concrete diff, and I’m like, I don’t like how they’re doing this, and here’s why. It gives me a perfect opening to.

[00:10:43] To get into it. However, I also try to be a very flexible diff reviewer. Right. I feel like there were some different viewers who had that attitude of like, I’m going to help other engineers. And then they were very inflexible. Anytime they saw any little thing they didn’t, like, reject, my view was like, all right, I see what you’re doing and why you’re doing it this way. Here’s my concern. And then either, like, I’m going to let you do what you want if you really feel strongly, but here’s what I would do differently next time Or I’d be like, let’s talk about how to redo your work to accomplish whatever concern I have.

[00:11:16] Basically hold their hand through the process if I can, because I feel like that’s a great way to, you know, now they’re going to, hopefully, if they’re convinced by my argument, they’re going to hold that same change in how they operate for their own future work and in all the future code review they do. It’s like viral, right? And you see, this would. Would spread through the code reviews that they did.

[00:11:37] So I thought it was a really great way to like get my opinions out there and have those debates in a really structured format.

Ryan:

[00:11:43] Given that you’ve done just such an absurd volume of diff reviews, what makes a good diff comment versus a bad one, in your opinion?

Adam:

[00:11:51] Talk through why you care about what you are nitpicking and not just what to change. Right? Don’t say change X to Y. Say, hey, X has some problems. So I really want you to use Y instead. But if you really feel strongly if there’s something I’m missing, then you go ahead and use X. Right? The other thing I was always open to in diff review comments was the fact that I might be missing context. And that’s still the diff author’s fault.

[00:12:15] Maybe because, like, your diff should contain enough context for anyone to review it on its own merits. But I’d often be like, hey, I don’t understand why you’re doing it this way. I want you to do it this way instead. But is there some missing context that would change my mind? If so, fill me in and put it in your Diff summary, which again, I feel like, makes sure that if I’m making a stupid mistake, which happens, I don’t look like a total idiot who’s just like getting it wrong.

[00:12:40] It really helps.

00:12:42 — Senior Staff (IC7) promo story and project

Ryan:

[00:12:42] So moving on to like, you know, E7 promo or the senior staff promo, I know that the main project that kind of drove that was Component Kit. Can you give some context on the story behind Component Kit?

Adam:

[00:12:54] The years all blur together at this point, but this was like 2014, maybe. We had a very large and complex product, Facebook Newsfeed, which was growing and growing and growing so fast because the company hired lots of new engineers and they stuffed everyone on Newsfeed because that was the hot product at the time. So it was basically falling apart. Right? We had constant crashes, constant layout bugs where, like, content would be overlapping in weird ways and it was a struggle to keep performance good.

[00:13:23] I can’t Claim credit for the idea. Again, here it was Lee Byron, who was, I think my manager at the time, but was for a long time a very senior engineer at the company and co invented GraphQL. And he had done a lot of work with React on the web, which was relatively new then too. Even on the web it was new. And he was like, you know what, we really need this React concept. But for iOS development at this time, React Native either didn’t exist or was just a prototype.

[00:13:52] And so Lee told me, like, why don’t you just take some time and think about how would the concepts of React work in an iOS first way? And component Kit was my answer to that question. So it uses the same concepts like components, immutability, rerender, everything conceptually at least when anything changes. View reconciliation. Right, so you’re not directly managing UI views, you’re instead just stating, hey, I want there to be a button that has this caption.

[00:14:21] The framework takes care of creating the actual button view. And I’m pretty proud of the balance that it struck in terms of the syntax was appealing, the performance was amazing, and it allowed us to solve a problem that we really needed to solve as a business. And Again, this was 2014, so years before SwiftUI. Swift didn’t exist yet. SwiftUI didn’t exist yet. React Native didn’t exist yet. It was very early on the scene and it did take some convincing of other iOS engineers because it was still a very alien way to Write code on iOS.

[00:14:54] But once convinced, everybody loved it. Everybody thought this is a really great way to write the UI for a complex product like Facebook News Feedback. So the big challenge there was inventing the framework, getting it adopted in newsfeed and then getting it adopted everywhere else. So I had lots of, lots of work to do.

Ryan:

[00:15:11] How did you convince people? Because, you know, this declarative framework was such a different way of doing things. I imagine there was a lot of pushback. How did you. You drive those difficult conversations.

Adam:

[00:15:23] Tons of pushback. Some people were convinced when they saw concrete examples of, hey, here’s what the same code would look like in Component Kit. It’s now like a third as much code. Some people were convinced when they could see the performance, right? Oh, look, it allows us to render all the viewed hierarchies on a background thread and then only do the minimal amount of work on the main thread. And they were like, that’s really cool.

[00:15:46] But there were plenty of people who were holdouts even then, and we had to call in mediators, right? Basically like There were other senior engineers at the time because when I was inventing Component Kit, I was an IC6. There were other more senior engineers who would, you know, get us to huddle up and try to just talk through how do we find a path forward that we can all agree on. And I think Lee Byron was very involved in these conversations.

[00:16:09] I know Alan Canestraro was a senior engineer at the time who was very involved in this, helping us, like, mediate between these different groups. But I think the fact that React had a lot of credit as a framework at the company on the web really helped, right? Because we could point and be like, look at what React is doing on the web. We want to do the same on iOS. There aren’t any really good reasons why we can’t do it.

[00:16:30] It’s not impossible to do on the platform. We have proven we can. So get on board. And I acknowledge the downsides of Component kit, right? By now, it’s very creaky and old because it was a C objective C framework. Now we have a Swift API for it. That’s great. But still, at the time, the weirdnesses of Component Kit were even then somewhat unappealing. But my pitch was always, yes, it’s a little weird.

[00:16:55] It’s not the usual way of writing code in iOS, but here’s all the amazing things we get. That’s why you should do it. So that’s the pitch that we had to make again and again and again to skeptical iOS engineers.

Ryan:

[00:17:05] Is there anything that you learned in trying to convince these people that were very against this approach that is kind of useful and general learning for anyone that’s trying to have some new developer offering and convince people to use it.

Adam:

[00:17:20] Allies are super useful, right? I still remember there were some engineers like Clement Gensmer and Greg Mech, who carried a lot of weight in the company, and they saw it and liked it. So great. I wasn’t alone. And they could help convince other people because I have one way of convincing people which isn’t always effective, and they have other ways of convincing people which often were more effective.

[00:17:40] And so it was nice to be able to break it down and like, you know, see the different ways that this discussion happened over time. There were opportunities for compromise. The other big alternative out there was this framework called Panels, which hadn’t really gotten off the ground yet, but was supposed to be like the next way to write UI at Facebook. And I knew them really well because one of them was like my mentor.

[00:17:59] And so I, you know, was very, very close To Jonathan Dan, the guy behind panels. And boy, they had a really hard time because they felt like they were developing the next thing. And then I came along and was like, actually, let’s do Component kit wasn’t so good. But we found a way to compromise and we adopted some of the panels data source technology to power component kit, which was a good compromise and allowed us all to feel like we had a win.

[00:18:20] Instead of sticking to your guns and every little thing, if you can find a way to bring people into your fold into the tent, that’s really, really helpful.

Ryan:

[00:18:26] With all that pushback, did you ever doubt the direction that you’re going in?

Adam:

[00:18:31] No. I was very convinced that this was the right call for Facebook Newsfeed. I will say. And this goes for all declarative UI frameworks. React, SwiftUI component kit, they’re really good at some things. Like a Facebook newsfeed is the perfect thing for it because it’s like a scrolling list of complicated multi level nesting and it’s mostly static, right? There’s some animation and cool stuff, but mostly it just is like a list of stuff that’s the perfect application for it.

[00:18:56] When you look at like a super dynamic drag and drop interface. Eh, maybe not the right thing for it. Right. Maybe not the ideal case. Anyway you can make it work, but it’s not going to be incredibly natural. So I’m very aware that like there are trade offs to these different paradigms and I wasn’t drinking the Kool Aid in terms of telling everyone Component kit’s the only way to write UIs on Facebook.

[00:19:17] Like that should be the, you know, our only option is component kit. No, but in general for what we wanted to use it for. Yes, I was very convinced that it was the right path forward.

00:19:26 — His major failed project

Ryan:

[00:19:26] You know, the next project you worked on was Component Script. What was the motivation behind that project and you know, what’s the story behind it?

Adam:

[00:19:36] So componentscript was interesting because it was a total and complete failure and I worked on it for like two years. At the time my manager was a guy named Ari Grant who was a force of nature, right? He was like all over the place, very busy, carried a lot of influence and Ari felt really strongly that we needed to get out of the per platform silo, right? So like iOS had component kit, Android had a react and component kit inspired by then called Litho.

[00:20:03] But this meant we were writing everything multiple times, right? We had to write it in component kit for iOS, we needed to write it in Android. He wanted to have a cross platform solution, React Native was not that solution. And we knew that at this time we tried React Native and it didn’t go well. The reason was in my opinion, this is just my opinion. React Native is designed to be in charge of the entire app.

[00:20:20] It works really well if your entire app is React Native or if you have entire app React Native and then small little pieces on the very bottom are native or bridged. Where it doesn’t work is the way we wrote software at that time, which was we had this large complex native app and we wanted to slot in small pieces of Cross platform in different areas, right? So like, oh, maybe this little square on this screen is rendered using JavaScript, maybe this tab is rendered in JavaScript and this tab is native.

[00:20:51] React Native could not do that for us at the time. They’ve done a lot of architectural changes to React Native in the last 10 years since then, and maybe it’s better at it now. But at the time didn’t really work. So we wanted cross platform. We knew React Native was not it. What could we do? And so Ari asked me to go and work on this. And at the time it was like, okay, this is the next nudge, right?

[00:21:11] Like we. Byron nudged me to work on react for iOS. That was component Kit. Great. This is the next nudge work on Cross platform for our UI rendering frameworks that we use in iOS and Android. And so I came up with a framework called Component Script. The idea was basically at first I took the react APIs, the actual react implementation, and said, what if we just made a different React native, right?

[00:21:32] So exactly like React Native, except that it works on top of Component Kit and witho, because that’s what we already have. This turned out to be too difficult. React had a really large surface area and a very complex surface area, and trying to make that work on Component Kit and Litho was too challenging. So I was like, all right, I’ll do a smaller pared down API that feels just like React, but actually is simpler and make that work on top of Component Kit.

[00:21:56] And Litho and I made it work. It was a real framework, people built real features on it. You could build full screens, you could build individual units, you could do all kinds of, you know, bi directional embeddings. You could have a native screen that had a Component Script unit which had a native component inside of that. You could have a Component Script screen which had a native section, all this stuff, really cool features.

[00:22:16] And for me it was a real learning experience because I learned that just because it was technically excellent, didn’t mean it was going to win. It checked all the boxes we needed in terms of interop and type safety. But it didn’t win and didn’t come close to winning because there were many other factors that I didn’t take into account. So a great example of writing a lot of code and doing a lot of technical work doesn’t mean that it’s going to win.

Ryan:

[00:22:43] I mean, if we were to kind of post mortem why it didn’t win, what are the things that you did well and what are the things that maybe could have gone better?

Adam:

[00:22:52] I mean, doing well. I just mentioned it did check. It checked all the technical boxes that we, as in like the core product inference group, cared about, right? Like it was type safe. IT integrated with GraphQL and component kit and litho our existing native frameworks. It was bidirectional. So you would never be suddenly cut off. There’d never be a point where you’re like, oh, I just need to embed this native component and I can’t do it.

[00:23:15] No, you could always do it. The mistakes were, number one, I wasn’t really aiming at a particular target engineer, right? So React Native was focused on like web engineers who wanted to write for mobile. Component Kit was focused on, hey, we have iOS engineers that already know how to write iOS code. Let’s let them write iOS code but have excellent performance. Component Script was like, hey, you can write JavaScript, but it’s not React.

[00:23:41] So like iOS engineers were like, I don’t want to learn a whole new language, I don’t want to learn JavaScript. And JavaScript engineers were like, I’ll use React Native, I don’t want to touch this. Like, weird. Not React API. No, thank you. So we were kind of stuck. Number two is that like we on the product infrastructure group really cared about GraphQL. We were like, hey, we want data consistency everywhere.

[00:24:01] So this framework needs to be based on GraphQL. But it turns out GraphQL can be kind of a pain sometimes, right? Especially at that time. GraphQL tooling was slow and a lot of hassle. We’ve fixed it a lot since then, but at the time it was bad. And so trying to build on top of this slow janky native GraphQL stack really slowed us down. Meanwhile, there was another framework out there that was coming in with a server driven sort of UI approach and their solution was like, don’t worry about data consistency.

[00:24:34] What if we just didn’t do it? And we were all horrified. We were like, what? You don’t have data consistency. So if you like a post on one screen and you go to another screen, it won’t show you that the post is liked. And the answer was yes. 60% of the time, 80% of the time, products just don’t care. And for the 20% that do care, you can hack something in there that makes it work. So we were pretty horrified by not using GraphQL.

[00:24:56] But that was a huge advantage if you could just skip all that stuff. But I refused to compromise, and that was a problem. And then finally I went really wide. I was like, all right, I’m just going to talk to all engineers, all mobile engineers, and be like, you guys should all try out component script. And this meant that I got little pings of interest all over the place because there were a few engineers here and there that were like, sure, I’ll try JavaScript, sure, I’ll try cross platform.

[00:25:20] But there wasn’t any individual team that was like, yes, this is how we want to write products from now on. And so that meant that it never went anywhere, right? Individual engineers would try individual little things here and there, but that was not going to get any momentum. Funny thing was, just when I finally pulled the plug in Component Script, the group’s team was like, oh, we just decided we were going to go all in on componentscript.

[00:25:40] And I was like, oh, man, no. But even that would not have been enough momentum, right? It was too little, too late.

Ryan:

[00:25:46] So you wrote about this retrospection. It’s very detailed, one of the best retrospectives I’ve read. Why did you publish that so publicly? What was the motivation behind it?

Adam:

[00:25:59] I don’t recall. It might have been encouraged that I write it, but I certainly wanted to write it. It was cathartic to talk about what went wrong. And even at the time, I had a reputation, right? People who knew who I was. So everyone would always be like, hey, how’s that component script thing going? And the postmortem was a convenient way for me to rip the band aid off and be candid about, hey, it didn’t work, and here’s why, and not have to constantly rehash that conversation over and over and over.

[00:26:24] But also, I hope that it would influence the way that people did stuff at the company in the future, right? Like, hopefully no one made the same mistake after that if they read my postmortem, or at least they were aware of what they were walking into.

00:26:35 — How to handle a failed project

Ryan:

[00:26:35] And, you know, in this case, you drove a very ambitious project, and it did fail in the end when it Comes to performance reviews in a half where something like this is happening or a year where something like this is happening. How does that play out and should people be worried about, you know, their projects getting canceled or things like that?

Adam:

[00:26:54] The thing that made me realize it needed to be canceled is that I got a means most. So performance did its job there. My manager at the time did the right thing and was like, this is not working. He was also a new manager for me. So I think he likes could see it with fresh eyes and be like, this is not going to work. And then when I did cancel it, I like to think I did it in the right way, which is there were products and features written in componentscript and I helped those teams migrate back to native code or to react native or whatever they wanted.

[00:27:24] And I completely deleted the framework. I didn’t leave it as like, you know, oh, this one product is still on component script, so we have to leave it around forever and someone will have to clean it up someday. No, I was like, I’m driving this all the way and I’m deleting. The code is going to be gone from the repo. Which I think garnered some goodwill because it showed others this is the right way to clean up after your mess.

[00:27:44] So I feel like I got, if anything, a positive bump after the fact. Right. Like, there was enough relief of like, all right, this showed people how to wind up, wind down a project that isn’t working out. And he posted publicly about it, talking about the lessons learned and there’s no mess left behind. So if anything, I feel like it helped in the immediate aftermath. So if anyone is like staring down the barrel of like, I think my framework’s not going well, but I’m afraid to kill it.

[00:28:09] You might get more goodwill from killing it responsibly than just like dragging it out and constantly waiting until it’s too late.

Ryan:

[00:28:17] Were there signs that, like, looking back, you could have maybe avoided some of the pain of, I don’t know, meets most or, you know, kind of like it going on as long as it did?

Adam:

[00:28:28] Yeah, I mean, look, there were, there was, it was a two year project and for the last year I knew it wasn’t going right and I should have listened to my gut. Right? I. There were some literally sleepless nights. Not a lot, but like somewhere I was like, this just. It doesn’t feel right. It’s not going well. I don’t understand what to do. And I’m a coding machine. So my reaction was, I just need to write more code.

[00:28:47] I Just need to help more features convert and it’ll suddenly take off. And I should have listened instead to that part of my gut that was telling me this is not going to work out. Just go do something you and find a different way to have impact. And that would have been much better for me in the short and long run.

00:29:04 — Thoughts on management

Ryan:

[00:29:04] You said before that you know for sure you, you never wanted to try management and it was not right for you. For someone who’s considering that kind of decision, how did you know that management’s not right for you?

Adam:

[00:29:17] I like writing code. I really like writing code a lot and as a manager you can’t write code so I mean for me it’s a no brainer, right. I’m also just, I’m less good at the non code related parts of the job. So like driving alignment and writing docs, I’m just not good at that stuff. And so for me I’m like, yeah, I don’t want to touch that. And finally I feel like I’m very good at communication with other engineers in a technical role.

[00:29:38] Like I’m very good at like, hey, we need to solve this technical problem, let’s all get on the same page about how to solve it. I don’t think I’m as good at doing that when I’m not quite when I’m more removed from the problem. Right. Managers have to do a lot of direction, setting and influencing people without directly pointing to like this line of code is the problem and that I think I’m less good at.

[00:30:01] So for me, I always knew management is not for me, but it depends on the person. Obviously I’m a pretty extreme case.

Ryan:

[00:30:08] What about picking the domain that you went with? So from what I see in your career, it’s almost entirely on the iOS side with some cross platform work. Well, how did you align on iOS and is that something that you feel strongly about being tied to or is that just where things have taken you?

Adam:

[00:30:26] That’s where things have taken me. I don’t change around a lot. I’ve never really changed teams once at this company. Right. Like I’ve like been shifted on different teams as part of reorgs but I don’t know, I just kind of roll with the punches and I like what I’m doing. I feel like I like the team so why mess it up? I admire engineers that are like, you know what, I want to go see what this AI thing is about.

[00:30:46] I’m going to go check it out. Or like, oh man, I really want to go Aron AR VR. That’s not me. I knew I liked mobile and I felt like I was getting the right opportunities to work on stuff I cared about. So I just kept rolling with it. And I think it has worked out really well for me because it allowed me to build deep knowledge expertise about how all different parts of our system work, right? If you want to know the guts of the GraphQL code gen or value object generation or buck or all these things, I know what’s up.

[00:31:15] And so it’s really helped me get really deep in this particular domain. And that means I have a lot of knowledge about it that helps answer questions from others. Very useful. Would I do mobile if I was a brand new engineer today? Maybe not. Maybe I do AI because that seems like the hot stuff, right? Or I don’t know what else, but I don’t feel too bad about sticking with it.

00:31:35 — Technical depth vs breadth

Ryan:

[00:31:35] You talked about the technical depth. Let’s say someone’s goal was to be like you. They really wanted to, you know, super aggressively pursue the high IC career path. They want to go IC nine or bust. Do you think that technical depth is better than breadth for becoming the highest levels of senior ic?

Adam:

[00:31:56] I mean, it depends on how you operate. I’ve seen lots of different engineers that operate in different ways. I will say I have breadth and depth, right? As in like, not that, you know, it’s perfect. It’s not like I only know mobile though, right? Like, I know a lot about how Buck operates, I know a lot about how GraphQL schema works. And so for me, the thing that helped me personally worked for me maybe not for everyone.

[00:32:18] When I run into a problem, instead of being like, all right, gotta go talk to the GraphQL team, I’m like, screw it, I’m just gonna figure out what the problem is. I’m gonna dive eight levels deep into their cogen guts until I find the problem and then I’ll either fix it myself and Now I know GraphQL code gen or I will show up to the GraphQL team and I’ll be like, hey, ran into this problem, debug it 8 levels deep, here’s the problem.

[00:32:39] How do I fix it? Which impresses the GraphQL team means that I’m not taking up all their time and I’ve learned something new about a new system. And if you keep doing that enough, then you will discover a lot about a lot of different systems and you’ll understand them and that’ll give you so much knowledge and so much. It’s like a Superpower to be able to dive into all these systems that you now know it’s also organic, right?

[00:33:02] We talked before about code review and how if you just review code that organically gives you actual discussions about how do we want to write our code. Same thing with this, right? Instead of being like, which systems do I need to go learn for my job? I’m like, they’ll come to me. I’m going to have a problem that I run into. GraphQL is going to block my code from landing. Great, now I have a reason to go delve into GraphQL Cogen.

[00:33:24] I’m not going to study it from first principles and be like, well, I should Go learn the GraphQL code Gen systems just because when it comes up, sure, I’ll do that.

00:33:32 — IC9 expectations

Ryan:

[00:33:32] A lot of people that get promoted to these really high levels, one thing I’ve noticed is the, the expectations become, I guess, scarier and scarier for people. Or they, they, they get there and they kind of get worried that they can meet expectations. You know, for you as an IC9 and you’re, you’re writing code, how do you make sure that it’s, you know, IC9 code? Like there’s this does it. You don’t worry about that.

Adam:

[00:33:56] I’ve flipped the switch off. I just don’t care. I used to worry about that too, a lot. And then I was like, you know what? I’m just going to do what I love and I’m going to check in my, check in with my manager often to make sure that I’m solving problems that need solving. Right. I’m not just going to go work on something that doesn’t matter, but as long as I’m working on what’s important for my org and I like what I’m doing, I’m just not going to worry about it and I’m going to do it.

[00:34:19] And that has served me really well, right? Component script being the exception where like eventually it was no longer important for the company or the org. And that was like someone needed to prod me to realize that. But they prodded me and I realized it and the problem was solved. Right? Like in some sense, like getting them meets most was freeing in the sense of like, well, if that happens again, I’ll know that like I went too deep and too hard and didn’t listen and now I go listen and fix it.

[00:34:42] So yeah, my answer is I just flip the switch off and don’t worry about it because that’s counterproductive.

00:34:46 — Senior engineers he admires

Ryan:

[00:34:46] As a senior IC, you are in forums with other senior ICs, and also you have a viewpoint that others don’t, which you could see. What work is technically difficult and exceptional? What are some other engineers that you think are exceptional and what, in your opinion, makes their work exceptional?

Adam:

[00:35:07] I’ve got a list, so I’m going to go down the list.

Ryan:

[00:35:10] Okay.

Adam:

[00:35:10] I really admire Dustin Shahidipur, who’s an engineer who works very much like me. He writes a lot of code. That’s the part of the job he enjoys. He and I have a very similar working style, so for that reason, I get along with him great. Another engineer that I’ve worked on for. Worked with for a very, very long time is Wei Han, who joined the company on the same day I did in 2012. And our careers often overlapped.

[00:35:33] We worked together on a lot of things, including component script, for a while. And she is many different archetypes. She does not fit in one box, but she’s more of a fixer. She’s very good at like, hey, this metric is not right. What is the deal? Which is something I can kind of do, but she can really do. And so I love someone that can dive again. It’s like diving 10 layers into systems you don’t know and figuring out what hell is going on here.

[00:35:55] She’s great at that. He’s no longer with our with Meta, but Michael Bolan was always an engineer I looked up to because he was a great project starter. He could identify the need, hey, we need a fuzzy file searcher that works on insanely large repos. What do we do? He kickstarted a project called Miles and just did it right, which is like backend binaries on multiple platforms, deployment documentation, naming, all this stuff that he could just do.

[00:36:19] And he did it over and over and over again. That’s just one of the many things he started. I think he started Buck, too. Let’s see here. Bob Baldwin is a very senior product engineer. I mean, I think he works on infra now, but for a long time he was on product, and he’s an excellent communicator. He can really boil down the message, which is something I think a lot of engineers very much struggle with.

[00:36:38] And it’s a really important skill is writing and communication. So I really admired his communication and how he could make things clear. Another engineer that’s similar is Oliver Ricard, who. He’s a very, very good writer and a very good communicator, and his posts are absolute masterworks. Whenever someone is like, hey, my engineer needs to work on Communication. I’m like, go look at Oliver’s posts.

[00:36:59] That’s the bar. That’s what you should be aiming for. And finally, I’m aware of what I’m not, the skills that I’m missing. So another senior engineer that I work with almost every day is Nolan o’. Brien. And he is like my polar opposite. He writes code, but not, not a whole lot. He is extremely good at driving alignment between different teams and he’s extremely good at managing the Apple relationship, which is something I never had patience to do.

[00:37:23] Right. Like Meta and Apple, they don’t always get along. No one finds a way to communicate between these two companies and try to get us on the same page, focused on what we can do together. And I really admire him for that. So that’s my list of other senior engineers I really, really admire.

00:37:39 — Advice for his younger self

Ryan:

[00:37:39] The last thing that I’d like to ask you is you have a lot of experience at this point, and if you were to go back to yourself right when you had graduated Princeton and give yourself some career advice, what would you say?

Adam:

[00:37:54] Nothing. It worked out pretty well for me. I’m not screwing with it, man. The funny thing is, when I was graduating Princeton in 2010, I thought about applying. I was like, should I be self employed and write iPhone apps or should I go work for a company? And I actually applied to Facebook and I’m pretty sure at the time what they would do is they would send you this puzzle. They’d send you like a mysterious puzzle that you had to like, solve the mystery and then you could apply.

[00:38:18] And I threw it in the trash and I was like, I don’t have time for that. I’m not doing that. And so I’m like, what if I joined Facebook in 2010? What would that have been like? It probably would have been a lot worse because I would have been around for all the HTML5 stuff and I would have just been like, oh, no, no, no. But yeah, I don’t think I would offer any advice to myself at the time because my personal situation worked out really well.

[00:38:42] As far as general career advice, I don’t know, do what you like. Right? Like, I feel like it’s really hard to fake enthusiasm or like find enthusiasm for something you really don’t enjoy doing. So, like, find what you love doing and do that, and if that intersects with what the industry needs, you’re in a really lucky place.

Ryan:

[00:38:59] You know, I used to always hear that advice and think that is cliche. And it’s very cliche. Yeah. But actually there’s so much stuff downstream of that, like your productivity comes from you loving the work. I mean, you have insane output. And your curiosity and the learning, it’s gotta be order of magnitude more than it would be if you hated it and you didn’t proactively dig, you know, 10 layers deep.

Adam:

[00:39:26] So I think it’s good if you, if you like solving problems. That’s the thing, right? Cause it’s not like every single moment I’m debugging some really bad code deep in some awful system. I’m loving what I’m doing, but I like the challenge of solving problems. And so if you like that, you’re in good. You’re in a good place.

Ryan:

[00:39:42] Thanks so much for your time, Adam. I really appreciate it. I was really looking forward to talking to you. So thanks so much for sharing your career story with the community.

Adam:

[00:39:50] Sure thing. Thanks Ryan.

Within weeks working at Meta, I was learning a ton and trusted to land code that hit production in the same day. Also, the people I got to work with were incredible. I always loved when a fix for a site-wide outage seemed impossible, then someone (usually Haixia) dropped a diff that saved the day. When you work with people who do excellent work, it’s inspiring.

There are many things that I’m grateful to Meta for, but the number one thing is the excitement I got out of the work. Working on something you’re intrinsically motivated by is one of life’s greatest pleasures.

Why Leave Then?

About a year ago, I started a podcast to practice speaking and share the software engineering career stories that inspired me. Everything felt new, and I was learning a lot. After work, I kept thinking about ways to make the podcast better.

At first, my goal was just to post once a month and see how it went. After a few episodes, I saw that people liked the content and I enjoyed working on it even more.

Every now and then, I’d wonder what it’d be like to work on my passion project full time, but it seemed far off since I’ve only lost money on the podcast. So far, YouTube has paid $14,994.63, and I’ve spent $17,600 on video editing. For some episodes like the Boris one, I also paid for travel, hotels, and production crews which adds probably about $8k more in costs across in-person episodes.

I don’t say that for sympathy. At that time, my unvested Meta stock had went up a lot so I used that to pay for my passion project. This did tie me to Meta financially though.

Why I Pulled the Trigger Early

A few months ago, there was a big reorg at Zuck’s level that moved me and a lot of engineers in my org. At first, I was excited. The reorg made sense, and the new org I was now a part of was building ML infrastructure that I thought would be an interesting technical challenge.

However, the more I thought about building up a new team the more I felt off. I kept thinking about my passion project and the dream of doing it full time. It felt selfish to build a team (I was a manager in the new org) and then leave if my podcast kept growing.

So before we moved people around, I decided to go for my dream early. I will miss the unvested stock that I left behind, I’m sure, but hopefully it’s not something I will think about at the end of my life.

What’s Next

I have two passion projects I’m working on:

1. The Peterman Pod - I plan to keep it focused on people’s career stories. I love hearing how people navigated their careers and the podcast has been such a great excuse to get to talk to heroes of mine. One day, it would be amazing to have people like Jeff Dean or John Carmack on!

2. Compose - I’ve used ergonomic keyboards for ~5 years now because of my wrist pain. After a bunch of research I could never find exactly what I was looking for. So what I’m building is the ergonomic keyboard that I wish existed. Here are a few pictures (not renders) of a prototype we made:

I have earmarked about 6 months of money at my current burn rate to fund this new career direction. If I want to keep working on these projects for longer than that, I’ll need to figure out how to make some money.

My immediate idea is to get a sponsor for the podcast. If I do, I’ll make sure it is genuine and helpful. I won’t promote anything I don’t use myself.

Anyway, thank you for your support. Working on my passion projects is something I’m only lucky enough to consider because of you all.

Thank you for reading,
Ryan Peterman

Timestamps

00:00:31 - Failing his FB interview

00:03:27 - Interning /w future billionaires

00:14:08 - Interview nerves tip

00:16:37 - Early Instagram experiences

00:34:08 - Building Instagram Stories

00:45:03 - 1 promo per half to Staff (IC6)

00:49:51 - Senior staff promo project (IC7)

00:57:37 - IG labs & his principal promo (IC8)

01:08:19 - Starting Retro and leaving big tech

01:21:33 - Small teams hypothetical

01:25:17 - Examples of talented individuals

01:31:16 - Advice to his younger self

Transcript

00:00:31 — Failing his FB interview

Ryan Peterman:

[00:00:48] in 2011, I understand that you interviewed for Facebook and you failed. Can you talk through that experience and what that was like?

Ryan Olson:

[00:00:57] I thought Facebook was super cool at the time. I wanted it so bad, and I was just incredibly nervous. I remember the interviewer, he kind of asked the question.

[00:01:10] He’s like, oh, you can just do it on my machine. He turned around, his laptop, pushed it over to me. I put my hands on the keyboard. You could actually hear the keys rattling because I was shaking so bad, which starts this feedback loop. Heart’s pounding. The question that he asked was actually one that ended up getting banned at Facebook, which I feel somewhat better about.

[00:01:36] But what was the question? It was this Gram buckets question, which there’s a very simple solution if you know data structures, where I use a dictionary. That solution didn’t come to me, so I wrote this horrendous triple nested for loop. The interview went quite poorly and he actually cut the interview short, which as an interviewer, that’s something I would never do.

[00:02:06] Even if somebody’s failing, you want to try to have a good candidate experience. He cut it short and he was just like, look, you’re just not gonna be able to make it at Facebook, which was like a knife to the heart. I wanted this thing so bad, and then this guy was just like, yeah, you’re just not good enough for this.

[00:02:28] It took me a while to bounce back after that. Luckily I did bounce back. I feel like I had a pretty good career at Facebook, so I managed to prove him wrong. I never got the name of the interviewer.

Ryan Peterman:

[00:02:49] Well, I mean, that to me shows how broken the interview process is. If you’re an incredibly high performer, and if you are not getting through, then clearly it’s not testing for the right things.

Ryan Olson:

[00:03:05] Yeah, I’ve always struggled with the Leetcode style interview. It was something I was always interested in trying to change while at Facebook, but it’s the interview process.

[00:03:17] There is such a machine across a big company. One of the things I enjoy about having my own company is I don’t have to do those things that I don’t agree with.

00:03:27 — Interning /w future billionaires

Ryan Peterman:

[00:03:28] After failing that Facebook interview, I understand that you went to work at a startup named Flipboard.

[00:03:33] And you were an intern, one of four interns. What was that experience like working at Flipboard?

Ryan Olson:

[00:03:39] Yeah, so I’ll tell a little bit more of the in-between story there. A couple months after the Facebook interview, I interviewed with Amazon. It was a phone interview, so I was actually a lot more comfortable.

[00:03:55] I did well on that. I got the offer. They did the kind of exploding offer. I got a call on Friday. It was like, you need to tell us by Monday if you’re taking this or not. I was pre-prepared to take it. It seemed like a cool opportunity. I had an investor friend. He founded Insight Partners, which is one of the big venture firms.

[00:04:20] He had mentioned at some point in the past, he’s like, at some point maybe I’ll connect you with some startups. I reached out to him and I said, Hey, I got this intern offer at Amazon. I think I’m gonna take it. He was like, don’t take it yet, let me connect you with some startup people.

[00:04:40] It was on a Sunday and he sent an email connecting me with the co-founder of Flipboard, Evan Dahl. Evan called me right away. I didn’t really wanna pick up the phone because I didn’t wanna do another interview and kind of just wanted to take this job offer that I had.

[00:05:02] I’m very glad I picked up the phone and had a great conversation with Evan. He had worked at Apple on the core of iOS. He actually wrote UIViewController, which is the core UI class that you interact with. He’d also taught this course at Stanford, called CS 193P, which they made available on iTunes.

[00:05:29] That was how I learned iOS development. I took it a different year that Evan didn’t teach. We had a really good conversation about that. After that phone call, I was like, yeah, okay, this makes sense. This seems way more interesting than whatever I’m gonna do at Amazon.

[00:05:47] I went there in the summer and the company was about 50 people at that time. We had maybe four interns or so. It was an extremely high density of talent. People were all super smart. They’d come from backgrounds like writing the iOS frameworks at Apple.

[00:06:15] One of the other interns that I sat next to, he wasn’t really an intern. He had worked at Apple and he was actually doing a medical degree and just needed a job for the summer. He was friends with Evan. He had written the push notifications framework and was talking about how he set that up for the demo at WWDC, running off his Mac Mini in Moscone where they had the conference.

[00:06:42] Another story he told was he had written the clock framework, and they had a bad time zone bug one year around New Year’s or something, and they screwed up and people’s alarms didn’t go off. It was fairly catastrophic.

[00:07:03] Making sure people’s alarms go off is pretty important. It was just a really good learning environment and I learned a ton about iOS and about building product. Two of the other interns have gone on to make multi-billion dollar companies. One is Dylan Field, who founded Figma, which went public yesterday.

[00:07:30] I think it’s a $50 billion market cap today. He was an intern through the first half of the summer that I was there. He ended his internship and founded Figma.

[00:07:57] I actually really questioned whether I should go back to finish my last year of school. I’m ultimately glad that I did, but it felt like there was just this moment happening. I was like, I can’t leave. If I go back to school, I’m gonna miss out on something really critical.

Ryan Peterman:

[00:08:08] So Dylan Field, co-founder of Figma, is one of the interns, and I saw in your writing, the other intern was Devin Finzer, founder of OpenSea, which at some point was a billion dollar company.

[00:08:22] I don’t know the exact valuation now, but that is absurd. Were there any traits that you noticed in those people or that intern class that kind of foreshadowed that they would be that successful?

Ryan Olson:

[00:08:33] They were definitely determined to start companies. That was what they wanted to do.

[00:08:39] I think that was a bit different than what I saw from my university classmates. People just wanted to get a job, kind of like the highest aspiration was to work at a big tech company. Both Devin and Dylan were like, we wanna start companies and there was a lot of talk about ideas and what they would do.

[00:09:03] It’s interesting to think back to the early days of Figma or talking to Dylan around that time. The idea that they had, they weren’t totally landed on the idea, but they had this one really interesting insight, which was WebGL was this new technology that was really gonna enable new things on the web.

[00:09:27] Dylan’s co-founder, Evan, was a really talented engineer and was very early to WebGL and very skilled at that. I think they went through some different iterations of the product that weren’t really tailored toward designers at all. Dylan talked about how they were a meme generator app for a week or something like that.

[00:09:48] It took them a bit to find their use case and customer base. They had this vision of like, this technology is really gonna change things. They stuck with that. It’s cool to see how far they came.

Ryan Peterman:

[00:10:06] It seems like while you were working at Flipboard, you built a successful open source project. Some of the first commits are in 2014. I’m curious, what’s the story behind that and why’d you open source that?

Ryan Olson:

[00:10:18] So during my internship I did some work on internal debugging tools for our iOS app. It was the first time I’d really worked deeply in Objective-C, and I found the language really interesting because it’s a compiled language, but it has this dynamic aspect, has this runtime that exposes quite a bit of information.

[00:10:46] While the app is running, you can inspect a lot of things about what are the classes, what are the methods on those classes. You can call them dynamically. You can see all the state, all the properties. The first iteration of that was just building a tool to inspect the state of the app and tweak it.

[00:11:07] I think we called it Tweak. It was FLTweaker. I just continued iterating on that as a side project. My main focus was building Flipboard, building the app, building new features.

[00:11:31] It was kind of like a nights and weekends thing that I was working on this debugging tool. People in the company found it really useful. It was a way to understand what was going on in the app under the hood. You could see all the network requests, you could see what was on the file system.

[00:11:50] I asked, “Hey, can I open source this? I think it’s going to be useful for other people.” Everyone was on board with that, so we pushed it out. It’s become one of the more popular iOS debugging tools. A lot of apps use it. It’s definitely used at Facebook and Instagram. It was also cool to see it take on a life of its own. I’ve been less active as a maintainer of it in the past probably since I started Instagram, or maybe a couple years into that. But this guy, Tanner, really took it over and has continued evolving it and adding features.

Ryan Peterman:

[00:12:33] That’s awesome. I remember hearing about Flex at Instagram as well, so I wasn’t aware that that was actually something you built before you were at the company, which is really cool. I’m curious, did building something like that and open sourcing it have some unexpected butterfly effect on your career in some way?

Ryan Olson:

[00:12:49] In some ways it did. In some ways I thought maybe it would have more impact than it did. An example of that is when I came to interview at Instagram, I had already released this open source and I was like, “Here’s my code. You want to see what I can do? This is a thing that I built that you can go look at.”

[00:13:13] None of my interviewers had any interest in looking at it. They were like, “We want to know if you can do this phone number sorting thing.” That was pretty frustrating to me. I was like, “You can see my work here, and you’re choosing not to look at it.”

[00:13:29] Other people who were slightly outside of the interview loop, but part of the recruiting process, like Jonathan Dan, who worked at Facebook on the iOS team, he got me to interview in the first place. That was all through him seeing this tool and what it could do. I think he advocated in the candidate review because I had a mixed interview loop.

00:14:08 - Interview nerves tip

Ryan Olson:

[00:14:14] I was very nervous in my internship interview. I did something for my full-time interview that might be helpful to some people out there who get nervous in interview situations. There’s something called beta blockers, which block adrenaline. When you have the physical effects of being nervous, like a pounding heart or sweaty palms, it can create a feedback loop that spirals down and blocks your performance.

[00:14:50] A beta blocker stops that adrenaline from firing, so you can stay calm and not end up in that spiral. I got a prescription for that for my interview, interview performance-enhancing drugs, I guess. I took that, and it was very helpful for me, so I was able to stay calm.

[00:15:13] I think performers sometimes use this, like comedians, to stay themselves and not end up in this nervous loop. I was still in a coding style interview loop, which I don’t do that well at. I think I had a few absolute confidence hire interviews and probably a few no hires. Somehow I managed to get through.

[00:16:04] I came into Facebook as an IC4, one level up from new grad engineer. I would say I was under-leveled as a hire. Part of that came from my interview loop.

Ryan Peterman:

[00:16:06] I’m curious, when you used the beta blockers, did it completely remove the nerves component for you, or did it just help a little bit?

Ryan Olson:

[00:16:13] I’ve only used them a handful of times, but pretty much completely, you know, like no noticeable nervous effects. I think you can still be in your head about what you’re saying, but it takes away a lot of those physical effects.

[00:16:32] No pounding heart or sweating, that type of thing.

00:16:37 — Early Instagram experiences

Ryan Peterman:

[00:16:37] After you passed the interview for full time, you started working on the Instagram team, working on iOS, and I’m curious what the environment was like at the time. What was the size of the team?

Ryan Olson:

[00:16:51] I came in and I think there were about 10 iOS engineers working on Instagram then. The team size had come down. The way the story’s been told to me is there were maybe about 20 iOS engineers, and there was kind of this internal war over the direction of the codebase, like the infrastructure, how the Instagram iOS app would be built.

[00:17:35] There was a really talented iOS engineer, Scott Goodson, who was managing the team, and he had made this framework for the Facebook Paper app called AsyncDisplayKit. It was kind of a different approach to how you manage writing iOS. There were warring factions; some people were very pro AsyncDisplayKit, while others were like, “No, we should just stick to vanilla iOS, how Apple builds apps.”

[00:18:01] The way it was told to me is these two factions fought each other, and each side destroyed each other. Everyone just left, and nobody won. I came into this team. Not many engineers had been there more than six months to a year, so it was a pretty new team.

[00:18:28] Because of that, there was a lot of low-hanging fruit, a lot of places to have impact. One of the first things I did was put the app into a tool called the Time Profiler. Most iOS developers will know this in Xcode or in Instruments. I looked at what was happening on cold start.

[00:18:53] There was a bunch in the startup path for the profile tab, which many people will never tap into. If they do, you can do that work at that time. It was a very easy win to shave 20% off our cold start time just by deferring that work until later.

[00:19:16] Another example was we had this networking library, AFNetworking. Most programmers are familiar with assertions. In iOS, assertions are typically handled in debug mode to crash and alert you to a problem, but you have fallback behavior for production, and you don’t actually crash the app.

[00:19:46] This networking library was built with assertions on, and it was crashing the app for benign failures, like a network failure, where the app could recover. We were just crashing. That cut our crash rate by 80%. It was just a one-line change, NSBlockAssertions, but it had a significant impact on the functioning of the app.

[00:20:10] Instagram had this cool tradition where every week they would give what was called the ax. It wasn’t getting fired, which sounds like if you got the ax, you did something of outsized impact. It was actually across all functions; it didn’t have to be engineering, could be product design, whatever. For that 80% crash reduction, I won the ax.

[00:20:32] It was this big physical ax that you got to carry around for a week. At first, I thought I got to take it home, but I realized there was just one ax. Somebody else was going to get it next week.

[00:20:41] Yeah, I had it at my desk for a week, and I was like, “Oh no, there’s just one ax. Somebody else is going to get it next week.”

Ryan Peterman:

[00:20:48] I remember the ax. What’s the story behind that?

Ryan Olson:

[00:20:51] that something that Instagram founders had done? At some point they had been asked by GQ or some men’s magazine to do a holiday gift guide, and they just kind of came up with random gifts. One of them was they found this service that was making bespoke axes. You could get this ax and then one of their investors read that article. They sent them this huge ax. Facebook security was pretty unhappy about this weapon in the office, so they made them put it on a plaque. It was mounted to the plaque. You couldn’t take it off. It was a cool tradition. Later, in my time there, they ended the ax as a very intentional thing. They thought people were feeling excluded because it was a weapon that was being given. I was kind of sad when that happened because it felt like, at that time, the founders were gone and it was a piece of early Instagram culture that they were kind of sweeping under the rug.

Ryan Peterman:

[00:22:02] So I guess going into your first major project, after all those low-hanging fruit, I understand there was a big redesign of the Instagram app called Whiteout. Can you talk about the story behind that project?

Ryan Olson:

[00:22:14] Yeah. It was a couple months into my time there and I got word that we were working on a new icon, which ended up being very controversial. As part of that, we were going to do a redesign of the app, a big visual refresh. I don’t remember exactly what I did, but I remember being like, I have to work on this thing. This sounds so cool. This is what I want to do. I told my manager, I was meeting the designers that were working on it. I basically just maneuvered myself in to work on this project. The main designer on it, Joy Vincent, was just an incredible talent and the nicest guy ever. Really sad, he passed away while we were at Instagram. I sat in a war room, a conference room basically, with him for I think probably two or three months. Every day we would be like, okay, new screening app. We’d go look at it, he’d be like, all right, I want to do this, this, this. I would do that, and I’d pass the phone over to him and just back and forth like that. We did kind of a whole new color pop for the app, new icons. One of the goals was to really make all of the focus in Instagram on the content, on the photos and the videos, and to take all the color out of the chrome. It’s basically how Instagram looks today, but at that time, we had this kind of dark blue and black chrome everywhere in the app. Things were a lot heavier. We just tried to really simplify it down and make all the color come from the content itself.

Ryan Peterman:

[00:24:06] Was it a gated launch or was it a launch it all at once kind of project?

Ryan Olson:

[00:24:08] So A/B testing was pretty new at Instagram at that point. Facebook was doing a lot of it, and Instagram was kind of more like, oh, we know what’s good, so we’ll just ship it. When I started working on this project, the CTO and co-founder Mike Krieger was like, don’t worry about trying to A/B test this thing. Just ship it, like build and ship it. At that time, they were still sort of using branching as a way to build big features, but I had seen some long-lived branches. The direct messaging feature had been built as a long-lived branch and it was a nightmare merging it back in because they branched off for like three months and weren’t really rebasing. Then they merged this thing back in. It was pretty awful. This redesign touched literally every surface of the app. I was kind of like, well this is not going to be a good way to build it. So I put it behind a feature flag and was constantly merging it and had to build abstractions. All the colors became semantic colors. Instead of saying that icon is black, it’s icon color or something like that, or it’s disabled icon color. Then you could switch inside of that function for, are we on the new design or the old design. That ended up being just a much easier way to build it incrementally. It also opened up the possibility of testing it. We went to ship it and it was going to be just like a 2% holdout. We were going to ship the new icon, 98% of people were going to have the new design, and then we were just going to have 2% of people with the old design just to make sure that nothing was broken or changed drastically. I had submitted the app to Apple. I dropped in the new icon. We were ready to go. People had champagne ready for the launch party, and it was the night before we were supposed to launch this thing. Someone in the Facebook executive team above Instagram came in and was like, no, you guys are not doing this. I guess Facebook had a redesign that went poorly, and they just saw this as the same thing. They were like, you need to run A/B tests of this upfront. I was up at like 1:00 AM just backing things out and trying to resubmit the app to Apple. We tested it and it actually tested really well. It was a little bit sad because it kind of crushed the launch that we had planned. There were all these articles about this new UI that Instagram was maybe going to do. But it all worked out. I did have a takeaway from that, which is testing definitely has a place. It is extremely useful. Even in our startup, we test things where you sometimes get counterintuitive results. Things in our onboarding flow, how people convert on things. It’s extremely useful to run A/B tests for that. I think for high-level product direction and what you want the thing to be, I prefer to come in with a stronger opinion and not just look at the data and only let the data guide the decisions. I think there’s a risk if you do too much experimentation that you get trapped in kind of incrementalism. It’s easy to get those 1% wins, but you’re never going to get that 50% jump.

Ryan Peterman:

[00:28:16] On the major redesign case though, if you just went for it though too. There’s also the other side risk though, right? Your product taste might be off and people hate it, and then it’s a pain to come back.

Ryan Olson:

[00:28:30] Yeah, for sure. I think we had enough confidence in using it. It felt really good internally, and I think at least on the redesigning the app, we thought that would be good. The icon was definitely a little bit more controversial and then when we shipped that, people hated it. It was the last time I looked at Twitter after our product launch because I went on and it was supposed to be the celebratory day. We had worked so hard on this thing, it went out and we were just getting raked on Twitter. It was kind of sad, but I think change is hard. Especially, people feel ownership over their home screens and all of a sudden this thing just changed on them. It was a little bit more of a bleeding edge design direction too. The flat icon and the gradient today feels very at home, but at that time, a lot of the icons were pretty isomorphic and 3D. The funny other thing from that, we could see from the data that this icon change materially improved how many people a day opened the app. We’ve actually seen this in the app of my new company called Retro, but the icon can impact whether people open the app, like how visible it is to them on the screen. It became more noticeable amongst the other icons and people actually tapped it more often. It reminds me

Ryan Peterman:

[00:30:18] of some of Thomas Dickson’s work on the ranking versus chronological. The public perception is so different from how actually people are voting with their usage. People are using it a lot more when it’s ranked. For some reason, there’s this very vocal minority that is saying, I absolutely hate this.

Ryan Olson:

[00:30:42] Totally. The feed ranking shipped about the same time as this redesign. I was actually skeptical of it as well. The metric that changed my mind on the ranking, which I think the story might be a little bit different today, but at the time, one of the things that moved was actually how often people shared to Instagram. People that had this feed ranking experience were actually creating more themselves. They were sharing more. It was partly because the feed ranking allowed Instagram to show them their friends more often. You saw content from people like you, and I think you wanted to have that mutual connection so you would actually share more yourself. That kind of was an insight that flipped it in my mind where people would say, oh, I’m just using the app more because you’re not showing me the things I want, so I’m scrolling further, or whatever. But I was kind of like, okay, they’re actually creating more content. That’s kind of hard to dispute. That’s probably a good thing. I really liked the original mission of Instagram, which was to capture and share the world’s moments. I really liked it as a way to encourage people to be creative and see beauty in the world. That was just a mission I felt like I could get behind. Something like this where people were actually creating more of, it’s like, okay, yeah, that’s probably a good thing.

Ryan Peterman:

[00:32:21] I think at this leg of your career, one thing that you’ve written about is that finding an amazing designer was a big part of it for you. I’m curious, how did you find the amazing designer that you did work with and what makes a great designer great?

Ryan Olson:

[00:32:38] Yeah. I’ve always just tried to be the engineer that the top designers want to work with. It is just an incredible opportunity if you work at these types of companies where you can have their work kind of flow through you and be a part of it. For product engineers, that’s one piece of advice I often give: your impact can be multiplied so much by finding a good designer and creating a really good working relationship with them.

Ryan Peterman:

[00:33:11] Is there a reason why you say specifically the design function versus the engineering function? Imagine I identify a very senior engineer that has some engineering design, and I devote myself to it and kind of attach to them. What’s the difference between doing that versus the really talented designer?

Ryan Olson:

[00:33:30]

[00:33:32] So I should make a bit of a distinction between if you’re in a product engineering function where you’re working on the features for users, the interface, kind of the front end, then I think that’s where you really want to pair with a designer. If you’re in a more infrastructure role, maybe the analogy is like finding that really talented, super senior engineer and being the engineer they want to work with. They’ve got great ideas, and you can implement that. So yeah, probably different depending on what role you have.

00:34:08 — Building Instagram Stories

Ryan Peterman:

[00:34:08] So this Whiteout project was in 2016.

[00:34:11] I am kind of surprised in the same year you also built Stories with Tiger Squad of some very famous people I’m aware of. Can you tell me the story about building Stories for Instagram?

Ryan Olson:

[00:34:24] The redesign wrapped up, and I had actually been kicked off my previous team. Right after I joined, they said, we’re moving the team to New York. Do you want to move to New York? I was like, no, I’m okay here in California. They said, okay, that’s fine. You just have to find a new team. I kind of wanted to be on this team. I delayed it, worked on this redesign project, and then I joined the Search and Explore team.

[00:35:03] I was only on the team for a couple of weeks, and a friend of mine, who was also an IIS engineer at the company, decided to quit. He had been working on what was called the Creation team and leading the project that would become Stories. My manager said, Hey, you know, this is actually a really important effort for the company. I don’t necessarily want you to lead my team, but I think this is a good opportunity for you. I really believed in what this team was trying to do. At that time, it’s sort of surprising in hindsight given how big Instagram is today, but it actually felt in some ways like Instagram was dying because a lot of the everyday sharing from people you knew was evaporating. It was being replaced by creators and influencers, and some of that everyday sharing was going to Snapchat, which had this ephemeral format. We were tasked with how do we get normal people to feel comfortable sharing on Instagram again? I went over to lead the iOS team on Stories. One of the first things we did after I joined the team is we cut the team size significantly. There had been a lot of people working on it, and there had been pretty high churn. They had tried different product directions, and a lot of them didn’t feel that great. They weren’t working out, and in some ways they were working on things to have work for people to do, or there was just not enough space for the people that were there. We decided we could move a lot faster if we went down to a smaller team. It was myself and one other iOS engineer as the core team, and then we’d get some help from other iOS engineers and Android engineers. We didn’t even have a dedicated server engineer; it was the infrastructure team, or you can have half a person, half their time. For what Stories has become, it seems kind of crazy, but it really allowed us to move quickly. You had ownership over the whole thing. It was never a question of am I working on something that someone else is working on? If there’s a bug, it’s like, oh, okay, that’s my bug. I gotta go fix it. There was less discussion around decisions. We could just make them more quickly. I say if you want to go fast, go small. I’m a strong believer in small teams as the best way to operate. It’s definitely not the only way to operate, but it’s my preferred way. We built it just over two to three months; it was pretty quick. I never worked so hard in my life. I was working 16 to 18 hour days, seven days a week, in the office every weekend. I would leave at like 1 or 2:00 AM to go home. I was driving back and forth from San Francisco. I was really determined to not sleep in the office. I was always going to go home, see my girlfriend. It was kind of silly because I was spending this extra time driving. I really should have just slept in the office. It was intense, but it was fun. It felt like we were building something really important, and we were using the product ourselves. We were really enjoying it. It also was this bonding experience among this small team. Our PM on the project is now my co-founder at my company, and I’m very close still with the other folks that worked on it. It felt funny because in some ways we were kind of the incumbent to Snapchat, but in other ways they were very much winning in terms of everyday sharing. We actually felt like the underdog. We had a Ghostbusters poster up in our war room. It felt like we were in this war against them, and I thought maybe we could win.

Ryan Peterman:

[00:39:50] What was it, in your opinion, that when you look at those two products, what was it that the Instagram version was doing so much better than the Snapchat one?

Ryan Olson:

[00:39:58] Yeah, I think we had a sort of unfair advantage in that people already had their friend graph on Instagram and the people that they wanted to share with, but we just weren’t giving them the right outlet to actually do that sharing. As soon as we had that, I think it really stopped the outward flow to go to Snapchat for that type of sharing.

[00:40:28] I think we also did push the format forward. Certainly Snapchat deserves all the credit for coming up with this container that was a lot more comfortable. It’s kind of 24 hour; the content doesn’t stick around forever, full screen, immersive, all those things. At the time, there was a lot that exists in both Stories products today that was not there in Snapchat. Even being able to navigate backwards by tapping left on the screen, that didn’t exist. I think we brought a lot of nice touches like that. The hold to pause, something I mentioned in the career notes. I was using this early version that we had built, and these things were auto-playing and going by too quickly. I intuitively put my thumb on the screen and wanted to pause it, and it didn’t pause. I thought, oh, I can just do that. So then I built the hold to pause, and now that feels like a core part of the navigation and the format. That’s another thing: when I mentor engineers, you have this power as an engineer to just build your ideas. If you are using something and you think it should work a different way, you can just go do that. That’s such an awesome thing. I encourage product engineers: if you have ideas, just build them and put them out and have people try them. It’s such a cool thing to be able to do.

Ryan Peterman:

[00:42:09] It’s insane that you were commuting from San Francisco to Menlo Park, so that’s almost an hour in each direction. You said you were working 16 to 18 hours a day, about seven days a week. I can’t imagine what that was like. One of the learnings from your note is that you should find work that you care about deeply.

[00:42:30] I still wonder, were there times where you were thinking you care about the work, but as a human, I can’t imagine surviving that kind of work schedule?

Ryan Olson:

[00:42:40] It probably wasn’t 16 to 18 every day, but it was working a lot for sure. I definitely sacrificed other parts of my life.

[00:42:51] I used to be a very serious rock climber, was on the U.S. team. I competed in World Cups and I basically gave up that part of my life for sure in that time. But it was also such a unique opportunity to build this thing that was just gonna go out to hundreds of millions of people.

[00:43:12] One of the things I loved most when I was working on Instagram is I’d be riding the Caltrain and I’d look over someone’s shoulder and I’d see them using the thing I’d just built like a week before. That was such a cool experience. You sacrifice certain things, but I was kind of happy to do so.

[00:43:35] Definitely not a sustainable model, so I wouldn’t recommend doing that over a long period of time, but in stints it can make sense. I think by doing that work with that intensity, it paid dividends later on. The Stories product came out. It was much better executed than some of the other Stories efforts at the company. There were ones happening in Facebook and in Messenger. I think part of that was just the care that we put into it. It contributed to its success. The reception on launch was very positive. I was really worried about it. I thought the whole thing was just gonna crash and burn because I could see all these cracks in it.

[00:44:25] You see the crash reports, you see all the bugs, you’re kind of your own harshest critic. But then when it went out, people were like, wow, this is so polished. I was like, really? Are we using the same thing? It was nice to have that reception, and I think it helped the product.

[00:44:44] Personally for my career, having demonstrated that I could deliver this project on a tight timeline with this level of craft helped me get onto the most interesting projects going forward.

00:45:03 — 1 promo per half to Staff (IC6)

Ryan Peterman:

[00:45:03] I saw at the end of the notes you started at IC4 at Instagram and you got to IC6 by the end of this gauntlet of these projects of Whiteout and Stories.

[00:45:15] That’s incredible. What was that like?

Ryan Olson:

[00:45:21] There’s a bit of a funny story. The promotion cycle happened in July. We shipped Stories at the beginning of August, so calibrations were kind of happening in July. We were in the middle of this very intense project and there was a lot of recognition of how hard we were working.

[00:45:44] The managers were being cautious around us. I felt fairly confident that I’d been mis-leveled on higher. I just wasn’t that familiar with levels when I came in and I kind of looked around and I was executing a lot higher than IC4.

[00:46:07] My Android counterpart, who is an incredible engineer, Will Bailey, was sort of the tech lead for the whole Stories project. He set the example for me of how you could be a successful product engineer. He was incredible at getting me into the meetings and being in the room with Mike and Kevin as we made all the decisions.

[00:46:30] He was an IC8, so he certainly had a bigger role in the project, but we had somewhat similar roles. I was at IC4, he was at IC8. That opened my eyes to some of the leveling stuff. I told my manager at the time, Eddie, who was the director. I was reporting to the director because it was kind of a unique project at the company. I said, “Hey, I think I got mis-leveled and I think you should promote me from four to six in this cycle.”

[00:47:15] To his credit, he didn’t just brush me off. He actually went and talked to HR about it and they came back and said, “We’ve done this twice in the history of the company. In both cases, it worked out terribly. The people left very quickly.” He said, “We’re not gonna do it.” But I wanted to seed the idea that the correct level for me is a higher IC6.

[00:47:34] I knew it was unlikely that they were just gonna do that one step, but I guessed that if I was on that normal cycle, I knew it would come up the next time and they’d be like, “Well, he just got promoted last half. We can wait.” I think it’s good for people to recognize that ultimately these levels and promotions are an incentive system. In the long run, there’s really an effort to make it fair, but there’s also an element of like, it’s the carrot that’s being dangled in front of you.

[00:48:18] It can make sense even if somebody’s performing at a higher level that the promotion gets delayed just so that they’re not happening in quick succession. I wanted to seed the idea that they should probably get me to six pretty soon. The first half on Whiteout, the redesign, and some of the infrastructure stuff, I got the IC5 promotion and then the six came for Stories.

Ryan Peterman:

[00:48:51] It’s interesting that you say your high performance, of course the promotions are one thing, but one of the things that you took away personally is that it gave you freedom to work on whatever projects you wanted.

[00:49:03] How does that play out?

Ryan Olson:

[00:49:05] I think it just demonstrated that if you put me on a project, I’m gonna do a good job with it. When there is a new effort at the company, it’s just kind of natural to take the people that have done well on the new efforts before.

[00:49:23] I was very lucky to have the opportunity, but executing well on it set me up well for the future.

Ryan Peterman:

[00:49:34] I noticed a similar model with Instagram Threads, which is you pull together all the top Instagram people and you got this small team similar to the Stories team. All those people have proven themselves repeatedly.

00:49:51 - Senior staff promo project (IC7)

Ryan Peterman:

[00:49:48] The next promotion actually came from IGTV, which, my understanding, is like a YouTube clone almost, or like the vertical video first in Instagram. Could you talk about that project that got you promoted to IC7 or Senior Staff?

Ryan Olson:

[00:50:08] It came from Kevin, the CEO, and Ian Silber, another amazing designer. Before an end of year all hands, they designed this thing together and laid out this vision for Instagram getting into longer form video. It was gonna be mobile native, so it was gonna be vertical the way that you hold your phone.

[00:50:41] It came with a fairly developed vision to the all hands and was presented to everybody. I saw that and I thought, I gotta work on this thing because this looks cool. It ended up being a bit of a weird project in the beginning because it was the surprise reveal.

[00:51:03] If you read through some of the recent antitrust litigation with Facebook, there are insights into tensions between Facebook and Instagram during this time. We were told not to work on it for a while, which was kind of the end of the founders’ time at Instagram. IGTV was the last major thing they were there for.

[00:51:26] It was a really cool group, a very small team: myself, Will Bailey, Thomasson, and kind of like iOS, Android, and server leads. We each had one or two other people per platform. It was a super talented group, great engineers, great designers. We moved super quickly.

[00:51:57] The product doesn’t exist anymore today. It didn’t end up doing that well. It’s interesting to reflect on why. It had some things that ended up becoming the future, like vertical video is very much a thing today. It was trying to mix long form YouTube style content into this vertical format, and maybe that was a bit of a mismatch. People just weren’t producing high quality long form in the vertical format. We tried some interesting AI techniques to take landscape content and reformat it for vertical. I’ve got a patent on that, which is one of my more interesting patents.

[00:52:46] We showed that to creators and they were horrified. They said, “You’re destroying my content. Why have I spent so long making this nice landscape video? Now you’ve just ruined it.” We tried to push people to produce original long form vertical video, but the inventory just never really showed up. There was maybe a little bit of hubris. It was like, Instagram can just change the industry. People will just start making this because we have this platform and this audience, and that didn’t totally materialize. I think it probably did inform a lot of what ultimately shipped as Reels, but IGTV itself didn’t totally work out.

Ryan Peterman:

[00:53:27] I was working on the video infrastructure team at the time, so that was one of the things that I got plugged into at some point. I remember the energy in the San Francisco office and the war rooms, et cetera. You mentioned antitrust. What’s the antitrust component you’re talking about?

Ryan Olson:

[00:53:47] A lot of the sort of internal communication has now become public through this antitrust lawsuit where the Justice Department is suing Facebook to unwind the Instagram acquisition. Certain things that were kind of confusing to me at the time, like why certain things were happening, are less confusing now that I read some of the internal communications that were previously private, including private to me. I wasn’t exposed to it. But yeah, I mean, I think basically Instagram had been acquired. It was smaller, certainly small relative to Facebook, and then had gone through several years of just incredible growth and now was kind of more of a peer to Facebook and was kind of like the new popular kid. In terms of Instagram Stories, it was received better than the Stories in Facebook. I think that created some tension between the Instagram leadership and the Facebook leadership. There were concerns about Instagram cannibalizing or taking users from Facebook, which guided some of the strategy decisions.

Ryan Peterman:

[00:55:18] I remember that too because I was also on the Instagram team and I remember there was some confusion about why we were turning off the followers being forwarded from Facebook to Instagram, or it was like cross-sharing or something. The direction from Facebook to Instagram was getting turned off. I think I read the same thing you read because I was like, oh, this makes so much sense. It’s like Mark Zuckerberg’s internal memo on all of that and all the tensions between the Instagram founders and saying he wants to keep them and they’re great at building products, but he’s trying to navigate his side of the equation. Super interesting.

Ryan Olson:

[00:56:03] It’d be really interesting to know how he thinks about it today. That was now seven years ago.

Ryan Peterman:

[00:56:12] The Instagram founders,

Ryan Olson:

[00:56:14] Mark, because I sense in those communications something of a very understandable emotional attachment to the Facebook product, right? This was kind of the thing that he created in Instagram. He deserves all the credit for acquiring it, but it was a little bit less his thing. I just wonder, does he view that differently today than he did seven years on? He owns both of these things. I always had the view that the best way to get the best outcome for the overall company was actually to have these things compete with each other because you own both. They’re gonna make each other better and one of them will win. If you stifle that competition, somebody from the outside is more likely to come up with something better and take over. I don’t know. I mean, he’s the one running a multi-trillion dollar company and I’m not armchair CEOing over here.

00:57:37 - IG labs & his principal promo (IC8)

Ryan Peterman:

[00:57:30] Yeah. Maybe one day if this podcast scales up and I ever have Mark on, I’ll ask him that. Lastly, at Instagram, I know you started a group called IG Labs. I’m curious about the story behind you starting that group. I also understand this is your promo to IC8 or like Director equivalent at Instagram. So yeah, what’s the story behind IG Labs?

Ryan Olson:

[00:57:55] After IGTV, I definitely spent some time, like Thomas Dimson has this phrase like wandering the impact desert, looking for things to do and not necessarily finding anything great. I was actually pretty close to leaving the company at that time. The new effort became Reels and I didn’t feel good about working on passive video consumption. My manager was over the Reels org, so it would have made sense for me to work on it, but I was fairly certain I didn’t want to work on that. I helped the team a bit, but it wasn’t gonna be my project. Through that time, there was a lot of culture change. The founders had left, and the leadership kind of came over from Facebook. The new leadership felt like a lot of the old Instagram culture was being pushed out. I was talking with the new head of engineering about trying to bring back some of that energy that I think had made product development at Instagram special: small teams, attention to craft, and also really expanding the scope of what we worked on. We were very focused on the existing Instagram app and the existing features within Instagram and just kind of like iterative features on that, very incremental improvements. I made this pitch that we have this great brand, we could do other things under the Instagram brand. Let’s explore location ideas, maps ideas, places. I paired with a really awesome designer, Vivian Wong, and we made the pitch to start this team. It was just her and myself in the beginning. The idea was to have this small group, very high talent density that would work on new product initiatives and things that didn’t slot cleanly into the org. Instagram had grown to such a size where you really needed a lot of structure in the org just to keep things sane, but that meant that projects that didn’t slot cleanly into one of those orgs were probably under-invested in. Part of the idea of this team was that we would span across, we wouldn’t have a focus area. We could work across many things. We tried a lot. A lot of it didn’t ship or test. Probably one of the more lasting, impactful things, it seems small, but I think it actually gets used quite a bit, is the collaborative post feature where you can have multiple authors on a post. It’s one of the more interesting patents I have. I think we found good impact. We were also this concentration of talent that when there was a new important initiative, ultimately ended up being Threads, you had this group that could go work on it. Just trying to encourage innovation, trying new things, and push that at the company.

Ryan Peterman:

[01:01:45] The last thing on your Instagram journey was that you tried management at some point, or as an IC7 you switched to TLD or Tech Lead Director. What was your thinking behind that and how’d it go?

Ryan Olson:

[01:01:58] It’s a very unusual or rare role within Facebook. Even TL is quite rare, or it was at the time. Then like Technical Director, probably even more so. It kind of happened mostly because I had started this group and at some point it just made a lot more sense for me to manage the people in that group rather than my manager who was over the Reels org and just less connected to their work. I could probably represent them better in calibrations. I was in the calibrations anyways, so it was kind of like doing a lot of this work. In some ways it was just kind of like a formal recognition of what I was doing already to have these people report to me. Facebook is very much of the school of thought that individual contributors and management should be separate, and I don’t subscribe to that. Other companies work differently. My wife is a Senior Staff Engineer at Tesla and they’re very flexible. ICs have people report to them all the time. Managers are expected to be pretty competent technical contributors and doing individual contributions. I think I prefer that model. It was interesting for me to try it. I think another one of my controversial opinions is that senior engineers should be involved in coding. There’s overlap between that thought and the thought that managers should be somewhat involved. There’s an author I really like, Nassim Taleb, and he talks about having skin in the game. If you’re a senior IC who has to do some coding, you have more skin in the game. You’re not gonna come up with some architecture that you just hand off to somebody else and it’s their problem now. You’re gonna be involved, you’re gonna see more hands-on what the issues are. Similarly, if you’re managing a team and you’re much more with them in the day-to-day stuff, I think you will operate better. It’s maybe a bit against the grain at Facebook, but it was a philosophy I had and I wanted to try it. There was an upside to it as well in a company that grew so much and was so big where at Facebook, the levels are private. You’re just a Software Engineer through your whole IC time. In some ways, I like that for engineering discussions that there’s no pulling of rank. It’s a little bit more meritocratic perhaps. But in cross-functional situations, say you’re working with a PM, like I’m trying to ship this collaborative posts thing and I have to meet with PMs on these different teams. There was an element where having a little bit higher of a title made those conversations easier. I had a higher baseline where they were like, okay, this person maybe knows something. They’ve been here a bit. They’re not just totally new. You build up some reputation in a company, but when it gets so big and there are new people joining all the time, you’re actually having to reestablish that a lot with folks. I think it was somewhat helpful to have that title. There would be discussions in the senior engineering group about whether there should be some form of public levels within the company. I was supportive of maybe not the full levels being public, but like a senior designation or something, just largely for when you’re working with people outside of your normal working group so that they start from a slightly better baseline on whether you know what you’re talking about.

Ryan Peterman:

[01:06:38] The number one thing I hear people say is this is not optimal because you’re doing two jobs at once and you’re gonna drown, and your career will not flourish in either direction. What do you say to that?

Ryan Olson:

[01:06:53] It’s probably correct. At that point, I was really not optimizing for career.

[01:07:01] It was never my intent to stay long term. I mentioned this in the career note; Facebook does these internal employee sentiment surveys every six months or year. One of the questions on there is how long do you intend to stay with the company? I never put more than one year.

[01:07:24] It was always like, okay, this is my last year. I’m gonna get out soon. I didn’t have aspirations to get to IC9, and so it kind of freed me up where it was like, I don’t wanna get fired for performance, but I kind of knew I wasn’t gonna get fired for performance.

[01:07:45] There was just a little bit less pressure on that, so I could just kind of do the thing that made sense for our group, our team, and worry a little bit less about the career stuff. But yeah, absolutely. I think I saw firsthand that my ratings were probably lower than they would’ve been if I was just an IC.

[01:08:11] It was like, okay, you’re exceeding as an IC but you’re meeting as a manager, so you’re just meeting.

01:08:19 — Starting Retro and leaving big tech

Ryan Peterman:

[01:08:19] That’s kind of like the full journey from start to finish of Instagram. I know you’ve since left and you’re working on Retro, which is a popular social media app with a lot of the polish I recognize in Instagram as well.

[01:08:36] I am curious what made you wanna leave to start that? What’s the story behind creating Retro?

Ryan Olson:

[01:08:42] So around the last few months of my time there, I was talking with Nathan, my co-founder. We had worked on Stories together. He had gone over to the Facebook side, started Facebook Dating, and then was working on some of the virtual reality products.

[01:09:02] Through our whole time, we were close friends. We would meet up at this bar sometimes in the Mission called Lone Palm, and we would talk about how we just saw a different way to build products, to structure a company. We really wanted to be in the driver’s seat of that.

[01:09:23] We would talk about starting a company, but the timing wasn’t right for me. I was on a project that I was engaged on, and then I’d be like, okay, I think I’m ready. He was on something new. Finally, our timing kind of aligned where we were both ready to leave, and I had a conversation with this investor friend, the same one that connected me with Flipboard, who’s kind of a mentor of mine.

[01:09:50] He was like, it’s time; you should really take a bet on yourself. That stuck with me. I was like, yeah, I should take a bet on myself. We left to start the company. Our goal with the company is really to create this world-class product studio. Retro is our first product, but it’s not necessarily the reason for the company existing.

[01:10:19] We want to create an environment where great product builders can do the best work of their lives and create products that are good for people, that they feel good after using, and also to create a successful business. Retro is our first product, and it’s a social app, but it’s actually quite different from what are called social apps today in that it’s actually social.

[01:10:52] You see people you know on it. Friends, it’s all focused around friends. Traditional social media is maybe, well, the derogatory term I would say is brain rot media. Entertainment is maybe a more favorable term, but if I open Instagram today, I don’t see a lot of the people I know.

[01:11:20] Even if I go to seek that out, I tend to end up down a rabbit hole of short-form video entertaining content. It’s definitely entertaining, but it kind of hijacks my attention. I get sucked into it. The idea behind Retro is we’re creating a space that’s all about connecting with the people that you actually know, staying up to date with them, and also appreciating your own life by looking back on your photos and picking out the highlights.

[01:11:52] In some ways, it’s a throwback to the earlier times in social media, what Instagram used to be all about—creation. Almost half of the people that use Retro on a daily basis actually post something, which I know from working on Instagram that number is quite a bit lower.

[01:12:11] I take it as a really good sign that we have created a product where people show up to create almost as much as they show up to consume. We’ve been working on that for the past few years. It’s become very popular in Taiwan, which is not something we expected. Social networks are so much about the network, whether your friends get on it.

[01:12:38] We managed to hit the critical mass in Taiwan, and all of the usage patterns end up looking different when you hit that critical mass. We were number one on the App Store there for a period last year, and we still are at the top of the photo category. We were number two when I looked this morning.

[01:13:00] We’ve shown that when it works, it works. We had just this challenge of making it work in more places. It’s very challenging to get people to try a new app. If you miss seeing your friends and people you know on social media and staying connected with them, I encourage you to try out Retro. I think it’s a really delightful app.

[01:13:30] I love seeing photos from my family on there, and we want it to feel good when you open it and feel good when you close it so you don’t feel like your attention was hijacked, that you feel like this was a good use of time. You got caught up with the people you care about, and then you can go on with your life. You can get out and enjoy the world. The design

Ryan Peterman:

[01:13:50] philosophy you went for with this app? Is it more challenging to get people to use it because it’s less addictive?

Ryan Olson:

[01:13:57] One of the key decisions we’ve designed around is that we don’t have an ad-based business model. I don’t think ads are evil, but I do think that ads when combined with an algorithmic feed are a little bit evil because you have this misaligned incentive.

[01:14:24] I think the app is incentivized for you to just consume more and more. It just wants more and more of your time and attention, even past what you wanted to get out of the app. I show up to see what a friend is doing; it’s gonna try to keep me there for the next hour, two hours, whatever.

[01:14:47] Ads fundamentally, when combined with an algorithmic feed, make products that aren’t super aligned with people. There is definitely a challenge in terms of growing Retro. We don’t have creators. The model is mutual friending. Everyone is private. You can only see people that you’ve accepted as friends, and it has to go both ways. It’s not like a following model. We have a limit of 250 friends, so you can’t build an audience on this platform.

[01:15:38] A lot of the ways that Instagram grew was through creators that were financially incentivized to spread this app and get more followers. Those pathways to growth are not open to us. We’ve seen examples like BeReal, which managed to get decent growth with a nice viral mechanic built in with this notification that everyone got: “It’s time to be real.” That created a lot of conversations, so that certainly helped.

[01:16:04] I think it is possible to grow a friends-only social network, but in many ways, it is harder, and the business model is harder. We’re very open about this challenge. There would be easier ways to do things, but we think they would lead to bad outcomes, outcomes we don’t want in the long term. We’re taking an opinion on what this product should be and what we want to see in the world.

[01:16:26] Some people say just build what people want. You could say in Instagram people are demonstrating that it’s what they want because they spend more of their time there. We’re taking a more opinionated stance, saying no, we think that’s not great and this is better. Even if it’s more challenging, it’s something we strongly believe should exist in the world and should be an option for people that want it.

[01:17:07] I think we also coexist really nicely alongside Instagram. I still have it, I still open it, but I’m getting sucked into it a lot less, and it’s helped me break this phone addiction that I didn’t feel good about.

How do you monetize? If you’re not using ads, we have a premium subscription. It’s only a small percentage of users that are subscribers, and it’s just kind of like an extra tier of features, particularly some of the things that cost us a lot to provide as a service.

[01:17:35] Video, as you know from video infrastructure, the storage and bandwidth for that is quite a bit more than for photos. You have to subscribe if you want to share out video. The free version of the app is actually great. You can share photos, you can remember your own life. About 93% of what people share on Retro is photos anyway. If you want to support the mission and get those extra features, you can become a subscriber.

Ryan Peterman:

[01:18:12] I love the intention behind it and the mission. It’s really cool.

Ryan Olson:

[01:18:17] It’s definitely a product that’s fun to work on.

[01:18:19] It feels good. It’s very feel-good. We had a tagline for a while: feel-good social media, and yeah, it’s very much that.

Ryan Peterman:

[01:18:29] It’s coming from big tech and starting your own company. I’m curious, looking back across the various career axes of learning, satisfaction, compensation, the things that people look at for their career.

[01:18:45] How has it been so far working on your own thing?

Ryan Olson:

[01:18:49] Yeah. Compensation is quite a bit worse. Nathan and I pay ourselves less than all of our employees, and we have more ownership in the company, which I think is the right way. It’s how it should be. But that means that I’ve certainly put off that short-term compensation.

[01:19:09] We hope to become a very successful company and that equity is gonna be worth a lot, both for us and for our employees. It was always the thing I wanted to do. I was a bit surprised actually that I ended up in Big Tech, talking about those early days in Flipboard and this entrepreneurial culture.

[01:19:31] It was always exciting to me. I would read TechCrunch all the time about the new startups, and I just always wanted to build a company, bring the best people together, work on the things that we thought were cool, do things in the way that we wanted to do them.

[01:19:54] It’s been awesome for that, and I’ve definitely learned a ton. I think the amount of time that I get to spend on interesting work is much higher than it was, especially in my later years at Instagram. A lot of my work became convincing layers above me that we should do something.

[01:20:21] There were probably more people working on that app than there needed to be. There was a lot of gatekeeping, where it was like, is your team allowed to ship? You would do this work, and then you have to convince somebody that this is a good enough thing to ship in the product, which is understandable because it becomes very complex if you just let everybody go.

[01:20:50] But it means that there’s a lot of work in just that political wrangling and how do I make this person feel like this is their idea, not my idea. I’d spend none of my time on that now. We do a one-hour standup every day. That’s basically the only meeting on my calendar.

[01:21:12] So I get almost the whole day to just build, think about what people want, and try to make it. I’d say satisfaction is definitely higher. Learning is definitely higher. Compensation is definitely lower.

01:21:33 — Small teams hypothetical

Ryan Peterman:

[01:21:32] I think throughout the conversation you mentioned that smaller teams can move faster.

[01:21:38] Do you think that the company would be better off if you just laid off half the people?

Ryan Olson:

[01:21:44] I would say yes. We used to talk about this a lot when the company was much smaller, this thought experiment of like, you know, the Thanos, like you just snap your fingers and half of people are fired, even if it’s a random selection.

[01:22:00] Do things go better? Frequently we felt like, yeah, maybe they would. It’s tough because with a business like Instagram, if you can make a 0.1% improvement, it’s actually hundreds of millions of dollars to the business. Sometimes that incremental person may be able to find those things.

[01:22:28] But I think it’s less appreciated the cost of each person you add. There’s the more obvious organizational communication overhead. You just have more stakeholders, more meetings, more coordination, but even just writing more code, having more code in the app.

[01:22:52] I remember right before I left, they had this tool where you could see how much time you were spending compiling the app. I was spending more than four hours a day waiting for the app to build just because there was so much code. The tooling just had not kept up with the velocity at which people were writing code.

[01:23:21] That’s a huge cost. It’s not noticeable with each incremental person you have, but now you have a thousand engineers and all of a sudden everyone’s just slowed down because of this overhead. With a smaller team, when we were 10 people and the app was tiny, it was so much easier to get things done.

[01:23:46] A project like Whiteout would be a massive project for Instagram today because there are so many more surfaces, so much more code, so many more people. It was actually a lot easier at the time that I did it.

Ryan Peterman:

[01:24:01] Yeah. I guess Twitter is kind of an interesting case study of that because they went through that and the app’s still operating.

[01:24:11] I do get the sense that there’s a lot more breakages, but I’d be curious, on the product development side, how has that been being on those teams?

Ryan Olson:

[01:24:20] Totally, yeah. I think there were projections when Elon came in and I think he cut like 80%. A lot of the staff left as well.

[01:24:30] Right. There was a sense that this thing was just gonna a hundred percent fall over. My experience is somewhat buggy. There’s definitely some issues, but it’s very much continuing to run. I think the economics of the business have actually become much better. It’s actually profitable now.

[01:24:55] Whereas through almost its entire history, it was operating at quite a loss. So, yeah, somebody coming in with a machete cutting things may be okay.

01:25:17 - Examples of talented individuals

Ryan Peterman:

[01:25:09] Okay. Towards the end of the conversation, I just want to wrap up with a few career reflections. The first thing is throughout your projects, you worked with a lot of very talented people.

[01:25:21] You mentioned Thomas Dimson, Will Bailey, a few incredible designers. It sounds like you worked with the founders of Instagram, and I’m curious, do you have any stories that illustrate what made them exceptional?

Ryan Olson:

[01:25:36] Mike Krieger is definitely something of a hero of mine, and it was such a privilege to work with him.

[01:25:44] They say don’t meet your heroes because you’ll be disappointed, but I think you should meet Mike Krieger. He’s actually an awesome human, and a lot of the ways that I think about engineering leadership and engineering philosophy came from him. I mentioned earlier the simple thing first.

[01:26:05] That’s really stuck with me. I think the attention to detail and craft. Another thing is he had this sort of, I don’t know if it was intentional or just the way he is, but it was like this lead from the front way of operating where he never put himself above the team and would jump in to help if an effort needed it.

[01:26:33] At the end of stories, I had brought in a friend of mine from Flipboard who was then working at Facebook, not at Instagram, a very different team to do the drawing tools just because he had worked on a drawing app before. I was like, hey, we need drawing, can you work on this?

[01:26:55] He worked on it for a few weeks and got it reasonably far, but then kind of got pulled back by his team. We were in a bit of a pickle, and Mike just came in and was like, okay, I’ll finish it. He was in there coding the neon brush, and we were working those insane hours. He was there working them with us.

[01:27:17] He was up at 2:00 AM reviewing my diffs. Seeing that and how that felt as a team really has informed the way I think about leading teams. He’s awesome. Some of the other folks you mentioned, Will Bailey, I talked about him a little bit before. He has this product vision that he sticks to very strongly, and frequently it’s quite good.

[01:28:15] At the dark days of development of the Stories project, when it was spinning in circles, taking different forms, none of them felt very good, he outlined this full vision in a document for how this product should work, the things we could do with it going forward. I looked back on it five years later and was like, wow, this was like the five-year roadmap for what we ended up doing.

[01:28:32] He basically nailed it all from the start. He was such a good example of how you could succeed as a product-focused engineer. That was much more of a thing in Instagram than it was in Facebook. I remember being in bootcamp and people were saying, okay, if you’re junior, you go work on a product team, and if you’re senior, you work on an infrastructure team.

[01:29:00] The culture was like, product is easy, but real engineers work on infrastructure. Instagram prioritized more the product side. I think that was reflected in the various versions of Stories, for example.

[01:29:21] Will was such a great mentor and example of how you could have a successful career as a product engineer, how you could influence product direction as an engineer. Thomas is probably just the smartest person I’ve ever met. He’s incredibly smart, but not in a savant kind of way. He’s also got great social skills and understands people.

[01:29:47] He did the feed ranking, basically started all the ranking at Instagram. He started his own company. OpenAI bought his company. He’s a researcher there now. Maybe Zuck is coming to him with a hundred million dollar offer. I don’t know.

[01:30:11] He had this skill on the infrastructure side, built a lot of the infrastructure for the ranking stuff, but always approached it with a product mindset. I think he would probably consider himself a product person. I’ve always been interested in technology as a means to create things for people. I’ve never been that interested in the technology itself.

[01:30:33] It’s only interesting to me to the extent that I can create interesting experiences for people. I think Thomas and Will had a similar approach, a very human-oriented thinking about the psychology, how people use these things and how it makes them feel.

01:31:16 — Advice to his younger self

Ryan Peterman:

[01:31:16] And then, last question is. If you could go back to yourself after all this experience that you have and talk to yourself when you are just graduating college and give yourself some career advice, what would you say?

Ryan Olson:

[01:31:28] Just invest in the tools of your time to build things for people, like build products for people.

[01:31:36] And I think if you do that, you’ll be pretty flexible and adaptable. You have a unique opportunity as a new grad because you aren’t stuck on this path of being an iOS person, knowing this technology really well. It’s all new to you, and so you can pick up the latest thing, right?

[01:31:59] And so I think with these AI tools that are coming out, you can do incredible things with them and you’re probably gonna be more skilled at those than the senior engineers because it’s just not the thing that they learned natively or invested their time into. I had that to some extent with iOS where it was newer.

[01:32:23] The senior engineers were writing Java or something, running our backend, and I could actually stand out and excel because it was the first thing I learned. I was eager and hungry. I think there’s something to that with this new set of AI tools.

[01:32:41] So, yeah, I think my advice for new grads is to learn the tools of your time, get good at them, and make good things for people. I think that will work out. It seems like there’s reducing demand for junior software engineers because of some of these AI tools.

[01:33:05] But I think if you can get really good at using those tools, there will definitely be roles for you.

Ryan Peterman:

[01:33:12] Awesome. Well yeah, thank you so much for your time. That was really, you’re a legend at Instagram and I worked there for so long. Super excited to talk to you. At this point, is there anything you want to redirect the audience attention to?

Ryan Olson:

[01:33:26] If the ideas behind Retro sound interesting to you, definitely try it out. It’s a great place to reconnect with people that you want to see in their lives, but maybe they’ve stopped posting on social media, so get them onto. If you’re a designer or an engineer, definitely reach out.

[01:33:48] I think we have some interesting ideas in how we’re building this company. Our goal is to become a world-class product studio with multiple products. We have Retro and two more in the pipeline right now, and I think we have some interesting ways we’re structuring revenue share with employees.

[01:34:09] As these apps start to make money, even if the company’s ambition is to reinvest it into growth, you can see some of that upside right away. We’re building infrastructure that helps support products across the portfolio. We’re building growth tools, analytics tools, all of that, that I think are gonna really help us excel as a product studio.

[01:34:35] So if that’s interesting to you, reach out for a conversation. It’s just been a pleasure chatting, and thanks for having me on. Thanks so much, Ryan. Really appreciate your time.

01:34:45 — Outro

Ryan Peterman:

[01:34:45] Thanks for listening to the podcast. I don’t sell anything or do sponsorships, but if you want to help out with the podcast, you can support by engaging with the content on YouTube or on Spotify. If you want to drop a review, that’ll be super helpful.

[01:35:01] If there are any guests that you want to bring on, please let me know. I feel like sourcing very senior ICs, there’s no well-studied list out there on Google that I can just search. If there’s someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I’ll reach out to them.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:41 - His layoff story

00:07:02 - Why he expects more layoffs

00:09:42 - Tech consulting before big tech

00:19:25 - Consultants shipping bad code?

00:26:57 - Why do people dislike consultants?

00:30:55 - Big tech compensation

00:40:27 - When age impacted his flexibility

00:42:04 - Why YouTube

00:46:46 - Speaking advice for engineers

00:49:09 - Advice for younger self

Transcript

00:00:41 — His layoff story

Ryan:

[00:00:41] Closer to tripling, you mentioned that people should expect future layoffs. What makes you think that? Here’s the full episode. I wanted to start by asking you a little bit about the layoff story. I think it’s that first video that you did that kind of went very viral. And I wanted to hear a little bit about what was going on at the time and what were the conversations like in the layoff?

ADE:

[00:01:11] I was working for this big tech company, right? And like a lot of big tech companies, we already had multiple rounds of layoffs. And in this particular round, there were rumors ahead of time that it was coming, and everyone sort of knew that it was coming. What was up in the air was sort of like, which business units and business lines are affected and how many folks are going to get cut. I think it’s probably because of either sort of my seniority within the organization or the relationship networks that had cultivated.

[00:01:45] I sort of got heads up about sort of like the nature of this layoff. I also had a heads up that I was on the list to be cut. And so essentially I knew these two things were happening. I had some other sort of life experiences and thoughts at that time that sort of made me question a little bit about why I was working in big tech in the first place. There were many sort of things. I kind of described them in detail in my vlog.

[00:02:18] But at that point, I then found out through the grapevines that a couple of more junior engineers are also on this particular wave to be cut. And so what I essentially did was I was able to do a little switcheroo where I put myself to the top of the list and got cut. And that’s how I got laid off. Now, obviously. And this is. This is just me conjecturing at this point, since I’m no longer with the company, these rounds of layoffs are happening.

[00:02:52] The next round is likely coming very shortly, if not already upon us. So it seems for me, the best I did was I just bought these guys a little bit of time, a mere matter of months. Hopefully they use the time wisely and that’s the best I could do.

Ryan:

[00:03:10] What was the reason for this layoff?

ADE:

[00:03:12] It’s only a conjecture on my side and I think it’s not just affecting sort of my ex employer, but maybe many big tech and tech companies. We’re sort of like right now in this race, right, to demonstrate the value of AI. And I’m not saying AI has no value, but there is a lot of pressure to essentially demonstrate value for all the investments that companies have been putting into AI. And one of those ways to do that is to decrease headcount.

[00:03:47] Because if you’re able to decrease headcount, your product lines are still being sold. So at least temporarily, your revenue as a company is not affected, but your cost has just gone down a lot because you decrease your headcount. That along with good marketing, talking about efficiencies gained through AI can really drive up your company’s stock valuation. So at least to me, I feel like we’re sort of like in this meta, right, where companies are looking to decrease headcount, reduce their cost, keep the revenue consistent, and then get some PR points for saying how innovative they are in terms of the AI revolution.

Ryan:

[00:04:27] When you heard that you were on the list and you were kind of getting that warning, what was going through your head?

ADE:

[00:04:34] It was weird, man. It was anxiety, panic, excitement, fear, hope, everything. Because I felt like I was in this thing where I was doing the same thing for at that point, years. And this was a sudden change, right? I didn’t know if it was a good change or a bad change, but it was a major change that’s coming to my life. And so it was this cocktail of emotions that was feeling very, very unstable, a very unstable kind of mentality that I had there for a couple of days.

Ryan:

[00:05:12] Were you presented severance at that point or how does that whole process work?

ADE:

[00:05:17] So with my ex employer it was a very structured process process and they’re a very good employer in the sense that they will make the grant the sort of. They will reveal the broad outlines of how your exit will look like, right? They’ll put you onto the sort of garden leave where you’re still with the company for a period of time, but you can apply for jobs generally. They’ll give you a pretty, they’ll give you a fairly generous severance and they’ll also give you other things, right?

[00:05:48] Like for example, if you have any Vacation days that would be. Or PTO days that would be paid out. Right. There is usually a stipend of some sort for education or for equipment and so on and so forth. And they sort of clearly articulate that ahead of time. And so if one does the math right, it’s actually a very generous package that you end up getting because you’re, you’re getting months of garden leave, you’re getting months of severance, you’re getting a stipend, you’re getting all your PTOs for like a whole year.

[00:06:20] And so, so it was very, it was very clear, it was very transparent, and it was well timed.

Ryan:

[00:06:26] Was this something that’s negotiable?

ADE:

[00:06:28] I think anything is negotiable, technically. Right. But really I just wasn’t in the right mental state to think about negotiating for it. I got to. It kind of this whole thing sort of made me pause and, and just reflect on life and what I’ve been doing this whole time with my life. And so the last thing I really cared about is like, hey, let me, let me negotiate for an extra 20% on the severance package.

[00:06:59] That really was on the bottom of my list of priorities.

00:07:02 — Why he expects more layoffs

Ryan:

[00:07:02] You mentioned that people should expect future layoffs. What makes you think that?

ADE:

[00:07:09] So when I was saying that, I was thinking more of the tech industry, right? The big tech companies and other sort of ancillary tech companies. The reason why I have this feeling is because it seems like the entire sector has invested greatly in the last couple of years, definitely into AI. And there almost feels to me like there is a kind of over investment that’s brewing in there, where a lot of companies are essentially putting AI into every possible feature of every possible product and platform and service, and all of those investments are not yielding clear returns.

[00:07:51] And so I just sort of have this feeling that whether the company would lay off to sort of decrease cost and performatively show their AI efficiency or layoff because they’re running short on money, or layoff because they really have to lay off because their business lines are collapsing. It just seems like there’s this meta trend towards shedding tech employees from, from these companies because of years of over investment and not enough returns for the investment given.

[00:08:23] That’s just the feeling that I have. And it could be totally wrong, right? I could be completely wrong, but it’s my current feeling giving the whole vibe of this entire sector, right.

Ryan:

[00:08:34] So I could see that direction. And on the flip side, things could be very positive. Maybe AI could empower people, maybe the revenue per person Goes up a lot. And we realize we want more people or something like that. These people are really worth their. Their labor is worth more. We want more people working on more things. Possible as well. Right.

ADE:

[00:08:59] It is absolutely possible for that to happen. Right? The future, to me, the future is unwritten. So it’s entirely possible that, hey, we do see the productivity gains of all these AI investments and the value of these tech companies literally shoot to the moon. I would actually very much hope for that because all of my. A lot of my investments and stocks and so on are with these companies. So if they go to the moon, I will be happy as a clam at this point.

[00:09:31] It, though, feels to me that what I just described, at least to me, seems higher probability than this sort of like happy future probability.

00:09:42 — Tech consulting before big tech

Ryan:

[00:09:42] You’ve had 25 years in the industry. Your career grew throughout the ranks of the various companies that you were at. So I kind of wanted to go over, looking back on your career, what are the things that you learned? What are the things that you wish you would have changed? If you could just lay out your career from the beginning, just on a high level, and then maybe we can talk about different pieces of it and, and see what people can learn.

ADE:

[00:10:05] So I got into tech early on into computer programming in high school. In college, I majored in computer science. I started doing these little side projects and consulting gigs for various small companies in college. Coming out of college, I joined a. A digital consulting company, like a consulting agency. And I spent many years in the sort of consulting agency space and I sort of went through the range of roles that a software engineer can go through.

[00:10:36] Right. So you start as a junior software engineer, you graduate to being a senior developer or senior software engineer. Then you move on to becoming kind of like a tech lead of a development squad. Then the progression goes towards the architect level, where you become an application architect, an enterprise architect, a senior enterprise architect all the way through the director levels. Right.

[00:11:00] Of being a technical director. Then for me, I made the decision after much reflection of leaving the consulting space and entering and joining a big tech company. And upon joining the big tech company, sort of, I went into this whole kind of senior enterprise architect, chief architect role, where you’re essentially overseeing architecture across the entire ecosystem of applications and platforms.

[00:11:29] And I did that for the last several years before my layoff.

Ryan:

[00:11:34] What’s the nature of the work when you do software engineering consulting? Is it you’re part of a firm that has software engineering expertise, and then there are other firms that don’t have that expertise that want features built. And then you guys deploy a team to help for a limited amount of time to build something.

ADE:

[00:11:52] And then so the consulting space works like this. Your consulting company has different sort of skills, right? Maybe it’s different types of software engineering, maybe it’s building a cms, maybe it’s creating e commerce sites, maybe it’s middleware and backend integration, maybe it’s digital transformation. And you have clients that are usually other companies, right? Sometimes there are small companies, but a lot of times there are big Fortune 500 companies.

[00:12:18] And you’re essentially then inserted into these clients to rapidly build out the systems for them. And that involves usually working with the clients themselves, their IT and technology teams and departments, as well as other vendors, other consulting and service companies. A lot of times it’s a very dynamic, maybe chaotic system where you sort of have to find your place, survive in this ecosystem where you’re one actor in a range of actors, build the thing to the best of your abilities and then you ship out to the next project.

[00:12:56] And it’s a very fast paced environment. It’s a very kind of like learn as you go environment from my experience, and it was a very good one that I enjoyed doing.

Ryan:

[00:13:06] One thing with that model that’s just immediate thing that comes to my mind is a large part of software engineering is maintaining software and making sure it stays up. It’s high quality. When things change around the software, we make sure that we patch bugs and various things like that. So if you were to just build something and then pass it to another company, how does that software remain useful and not explode when you hand it off to the customer?

ADE:

[00:13:36] It’s a great question. There’s sort of two answers to this question, right? There is the hey, get in, build and get out model, right? And I’ve seen my fair share of such projects. There’s also the retainer model where your consulting agency works with a client for a long period of time, sometimes years, and you sort of not only build it, but you maintain the software and you enhance and build additional features and extensions to the software.

[00:14:05] I have seen and worked with this model as well. The general case is when you have a retainer, you have the time of really knowing the code base. And especially if you built the platform from the ground up, there’s this institutional knowledge of how do you maintain it and how do you enhance it. And those are the kind of better types of projects as a consultant. But the other kind of project is more like fly by night, right?

[00:14:35] Where you got to as a company have a pipeline of work, right, to keep all your consultants employed. And so sometimes you would sell in a project purely for monetary means. And unfortunately, a lot of companies, a lot of agencies and consulting companies do this in the economy. And so what you’ll sometimes have is as an architect, for example, you go to a client, you’ll see this ecosystem that’s literally this dizzying patchwork of platforms and sites and systems that are a lot of times redundant and a lot of times incompatible because all of these vendors and agencies and IT services companies doing their thing and it sort of almost like becomes this dumpster fire that especially if you’re hired as a consultant to do a digital transformation, you sort of have to put out the fire and kind of stitch everything together just to have a working platform for your client.

[00:15:32] So I’ve seen both, and both are kind of fun and anxiety inducing in their own ways.

Ryan:

[00:15:38] In my experience in building stuff, a lot of the times we build software, it’s just within consistent teams where you see that person every day as a consultant, you come in, you’re dropped, you’re parachuted into some other team. What is the default culture like or what’s that, that experience like?

ADE:

[00:16:00] So in my experience, I’ve encountered all of those scenarios. It depends completely on the clients. Some clients have rigid hierarchical cultures, some clients have much more open and cooperative cultures. Some clients have this sort of internal startup model where literally you’re part of an internal startup and so that entire range of different kind of environments could happen. As a consultant, when you’re dropped in, a part of your skill as a consultant, right, is not just knowing the tech and being able to architect and being able to bang out code.

[00:16:37] It is to understand the topology of the client political environment, how does the client teams interact with each other, how do they interact with other consulting agencies or IT services agencies in this ecosystem? And as you sort of progress in your career as a consultant, eventually it becomes part of your job almost to be able to navigate and open up a space for your consulting agency within this client ecosystem.

[00:17:12] And you keep the book of orders going, right, with additional work and stuff. For me, the early rungs of this was highly enjoyable, right? The learning of these new cultures and environments and new technical ecosystems and then adapting and improving these. It was highly, highly enjoyable. Near the sort of the latter portion of my consulting career, it kind of shifted. The work that I was doing shifted as I became more and more senior.

[00:17:43] So instead of being more of a hands on, hey, let’s design this quilt of systems Based on these components that you guys have, it became more of like, hey, drop me off at the C suite and let me sell this vision of digital transformation to the CMO or the CSO or the CEO of this Fortune 500 corporation. I did that for a couple of years, the last couple of years as a consultant, and I sort of got a little burnt out by it.

Ryan:

[00:18:14] What burned you out?

ADE:

[00:18:16] The gap between the ideal that I was selling and the actuality of what was actually there. Right. Because the actuality, the sausage making is dirty. It is chaotic. You have systems that don’t talk to each other. You have code bases that hasn’t been maintained for five years. Right. You have different teams. You have like offshore vendors who sometimes can’t even communicate with onshore teams. But that is not the vision that is sold to the executives in the C Suite.

[00:18:48] Right. You know what the vision is? The vision of a digital transformation is usually like this smooth diagram where you capture customer insights. You have this bubble that does a rationale and a decision is made and immediately sales and orders come in and revenue goes up. That’s the dream that’s sold. Now compare that with the actuality. If you’re the person that needs to make those two things come together and happen, which I was eventually as a group technical director, it just gets too much.

00:19:25 — Consultants shipping bad code?

Ryan:

[00:19:25] Is there any feedback mechanism? Let’s say you sell this vision and their company says, we love it. Transform us digitally. This sounds amazing. And then you come in and you build a steaming, broken code base that just kind of works, but not really. Would you still get paid as a consulting company?

ADE:

[00:19:50] So this may be a time dependent question, right? Because I feel that there was a period of time around the 2010s where every company was talking about digital transformation. This is the era of companies going from spreadsheets, printouts to like mobile apps and sites and you know, essentially having a fully digital experience for their customer. At that point. The honest question is a lot of the times the decision makers, the executives, the C suite, they’re not fully aware of the sausage making that happens under the hood.

[00:20:29] Like all of these, all of these ridiculous duct tape implementations and like swivel chair operations with people, workarounds for APIs that doesn’t work and breaks, all of those details are sort of obfuscated at that level. But what they see is like, oh my goodness, a customer can go through this web mobile app and they get a text message telling them that their order is delivered and line goes up.

[00:20:58] If you’re able to do that As a consultant, you’re going to get paid. And so from, from my experience, it was overwhelmingly positive that, that it’s hard in the initial stages to get a digital transformation wrong because all you’re doing is taking something that happens in the analog world, that used to happen in the analog world, and making it digital. Maybe it’s not the best implementation, but essentially you’re going from zero to one.

[00:21:32] I see.

Ryan:

[00:21:32] Yeah, I could see that exploding on them later if there’s some issue, something like that. One thing I could see here is these C suite people, they’re non technical in a majority of cases, I’m assuming. And so they probably don’t have the judgment to know what good work looks like in this domain. And so I kind of see a natural information asymmetry here. What stops the technical consultant from coming in and saying it’s great, but it’s actually very far from great.

ADE:

[00:22:13] You have to have sort of like your own personal boundaries as a technologist. Right. And as an engineer, you’re taught, or I hope all of us as engineers are taught, that they’re sort of like engineering integrity. And it’s really funny because when I was in university there literally was a course on this. Right. Like you, you have to have respect for the truth, for quality and for integrity of what you’re building.

[00:22:39] And that, that is kind of critical. I, I hope to God that I have followed this throughout my career as a consultant. I tried my very best to do it right. If it is a steaming pile of turds, it’s your obligation to tell your client that, hey, hey, man, like the $20 million that you’ve invested is a steaming pile of turds. That’s what you’re supposed to do. Are there consultants who didn’t do that? I’m sure, I’m sure.

[00:23:06] But we tried our best to do so.

Ryan:

[00:23:09] I’m just trying to think what is the, what’s the incentive structure that could prevent that? And I guess it would be probably the reputation of the consulting firm, which is if you deliver that steaming pile to customer one, you’re not going to get customer two or three or anything like that.

ADE:

[00:23:28] Correct, correct. That is something that could happen. Right. But just in general, and it may be something with a person’s career. Right. I went through this phase in my career where I was obsessed with sort of the quality and longevity of the software that I was building. I don’t know if in your development career if you went through the same thing, but this striving for excellence, for efficiency, sometimes it can get totally ocd.

[00:23:57] Like, you’d be like, oh, this piece of code here is not performant enough. We’re going to optimize the heck out of it to do this thing perfectly. But there is that part of a person’s career where you really are obsessing about the quality of what you build and, and you sort of almost take like the quality of your end product as a personal reflection upon yourself. So if you, if you build a steaming pile of turds, you’re going to feel ashamed about.

Ryan:

[00:24:23] Yeah, definitely. Well, I mean, when I think to big tech or from my experience, and I think what is the incentive structure that makes it so that we put out quality work is that we have technical leaders who are controlling the performance incentives. So people who get promoted, they often build things that are ideally performant or the metrics that matter are improving and the way that they’re done is high quality.

[00:24:53] So they are operationally excellent. The implementation quality is something where it’s not going to break in the future and we kind of prevent a large class of bugs from happening just in the way that it’s implemented. But the key takeaway there is the leaders themselves, the people who control the incentives of the engineers, they’re technical. And there’s this whole system that’s set up that says, hey, if you do good work, you get rewarded.

[00:25:23] Good as deemed by technical people. So the thing that’s interesting to me here is the people who say this is good, in other words, the client, they’re not technical. So I would have thought you could deliver the thing that’s not operationally excellent or the thing that’s not particularly well done, as long as the inputs and outputs are acceptable, ship it off and disappear. Because you don’t necessarily have as much skin in the game as the client does.

ADE:

[00:25:52] Ryan, you could do that. But in my experience. Right. And that’s not only as a software engineer that’s sort of climbing or going up the ranks from my mentors and leaders at the time, but also my experience at the director and above level where I was in those sort of leadership roles. Right. That, that the, the pride and the care for the work and the quality of that work was very high. You really do have a significant number of people that kind of cares a lot about the quality of their software that they’re building and the systems that they’re building.

[00:26:31] That’s sort of one thing, and it’s a human element to it. But I think it’s an important one because if you don’t have that quality, nothing stops us from delivering just crap to the clients. Right. And as you pointed out yourself, you can only do that so many times before the reputation of a firm, of a consulting agency is ruined. And nobody wants that.

00:26:57 — Why do people dislike consultants?

Ryan:

[00:26:57] Yeah, definitely. I guess it’s. You could make the same analogies which the leadership at the consulting company could say, hey, you get promoted to a higher level as a engineering consultant if you build good things and they review the, the things that are built. And yeah, I could see that happening as well. One last thing that wanted to go over on the consulting side is, and I don’t know a whole lot about consulting, but I feel like I see in pop culture or in various interviews or jokes that, you know, consultants are, it’s a job that a lot of people, they don’t, they say we don’t want consultants here.

[00:27:36] Something like that. What is the reason that, that people say consulting is not a good thing?

ADE:

[00:27:43] So sometimes I’ve observed, right. When being dropped onto a client side that there is sort of hesitation and resistance. And in my experience, a lot of that actually comes from management consulting rather than technical consulting in the sense that sometimes it feels like management consulting, sort of like any business that you’re running. Right. You have people processing tools. Right. Fundamentally, technical consulting deals with the tools and it may deal with the process to a smaller extent, but the people portion of it is usually management consulting.

[00:28:29] And that sometimes leads to decisions that has an adverse impact on people working in these client companies. Right. So management consultants may, for example, take a look at all of the teams working on your business and decide that X percent of these teams are redundant. They don’t, they’re, they’re, they’re not needed. And a lot of times the clients would be influenced to do things like lay off X percentage of people.

[00:28:53] And so if that happens enough times, there then is the impression that, hey, a consulting team is coming. Oh my goodness, our jobs are at risk and we’re going to get laid off. And at least to me, that feels like the majority of where that sort of vibes come in. From my personal experience, people are going to be people and most people are nice. And so if you drop, you get dropped to a client side and you work with them, you get to know them 90% of the times.

[00:29:20] They’re great guys.

Ryan:

[00:29:22] You mentioned around seven years ago or so you transitioned from this consulting space to working in big tech. What was that transition like?

ADE:

[00:29:32] Okay, so at that point, pretty much all I did as a consultant was get shipped to a boardroom and sell digital Transformation to a C suite. All right, so I was in a. I was terribly bored of that role. It just felt like I’m further and further disconnected from the actual implementation that I cut my teeth on as a young man. So a good colleague of mine had become an executive at this big tech company.

[00:30:03] And so he introduced me to this particular role. It was a hands on role as a chief architect, right? On paper you’re essentially working on design of applications across an ecosystem. And in practice I found that when I jumped in there that I had the ability to, for example, deep dive on a specific system and lead the build out of individual systems that were deemed important by the organization.

[00:30:32] And that then allowed me to do way more than work on, you know, visio diagrams. I could literally, for example, take a component that was really, really difficult and code it out and build it. I could go into debug sessions with teams. I can tweak configuration, I can participate in launches into production of different platforms. And, and that was for me a very refreshing change.

00:30:55 — Big tech compensation

Ryan:

[00:30:55] You also mentioned, I mean the compensation was different on a percentage basis. Was it a major difference or how big of a change are you talking about?

ADE:

[00:31:08] The compensation of Big Tech is bonkers. I just, I have to, I have to say that consultants are paid well, they’re comfortably paid. But you go from consult or at least from my experience, you go from consulting to Big Tech. It’s like the compensation more than double, closer to tripling. And it’s because of the structure and the nature of compensation at Big Tech, right? You have bonuses both sign on and reoccurring.

[00:31:37] A lot of times you’re given this thing called an rsu, right, Restricted stock units. And these are essentially shares of the big tech company. And those shares are vested usually over a period of time, right? Over three or four years, you get a chunk of it each year. And then there are these stock option like programs like the employee stock participation programs where you’re essentially allowed to purchase greatly discounted tranches of shares of the company, right?

[00:32:06] So for me, I got all of these things, so I got the bonuses, I got the rsu, I got the espp. And what essentially happened was imagine having a base salary that’s already significantly higher than a consulting salary. But then you have all these shares and for these shares, if you were to hold on to them, and for me, I did hold onto them, those shares then doubled and then tripled in value. And so at that point, the compensation between my new role in Big Tech and my old one in consulting, it’s not comparable between the two.

Ryan:

[00:32:47] You mentioned espp. I think people might be curious about that. Can you explain?

ADE:

[00:32:52] Sure. What it is, is it’s sort of like the company gives you the right to purchase their stocks with your post tax salary, right? But the purchase occurs in such a way that you’re guaranteed a discount. That’s, you know, for, for me, it was around 15% below the market price of the stock. Okay. So imagine if the stock of this company is $100, you’re guaranteed to be able to purchase it for $85. So if you were to do that and acquire these stock shares periodically, right?

[00:33:33] And then you hold onto them, it’s almost like you’re baking in this, this advantage. And then those stocks, if, if you look at the majority of big tech companies over the last seven years, the majority of them kind of ballooned in value over the COVID years especially and has increased many hundreds of percent. And so what essentially happened with this is if you take these stocks and then you hold onto it, it will double or triple or quadruple in value.

[00:34:03] And that’s what happened to me.

Ryan:

[00:34:05] Is there anything that will stop you from selling it immediately in the ESPP is like, could you just buy it at 85% of the cost and sell it immediately for 100% of the value?

ADE:

[00:34:18] Yes, yes you can. It’s not advisable. It’s not advisable to do that. And you know, times could be different, right? Like if you’re in a period where, where you believe the value of the stock is going to go to the moon, it’s really not advisable to sell it instantly. But of course, stock markets are inherently unpredictable. Anything can happen. Right. But for me, when I was accumulating those stocks, sort of like I had this.

[00:34:51] Well, there was Covid, right? And if you remember what happened in Covid, there was this huge push into digital platforms across the entire population of the United States. So for me, in my simple little brain is telling me, hey, wait, so everybody is forced to use digital tools. They don’t have a choice. Oh, and we, we make a lot of these digital tools. Maybe I should hold on to these stocks. I think they might go up and they did.

[00:35:20] But going forward, right. We don’t know if that’s the case. So good judgment has to be made, right, Whether to hold or to sell.

Ryan:

[00:35:28] What a lot of people do is they buy index funds so that they have a little more diversification instead of holding particular stock. But I mean, it sounds like regardless of if you want to hold your company’s stock or not. If your company is offering the ability to purchase it at a discount, it’s an immediate superset, at least to just buy it and sell it into cash. And if you were giving you money.

ADE:

[00:35:55] If someone’s just giving you money for nothing, you should accept that money.

Ryan:

[00:36:00] Well, what’s the, I don’t get the, what’s the incentive for them. Because if there’s no reason to hold, I could see it as a good way to keep people invested in the company. If they said, hey, you can buy at a discount, but you have to hold on to it, then yeah, you, maybe you buy it and you say, do I want to take that risk or do I believe. But if you could just buy it. And so I just buy it and sell and put into cash or whatever the thing was that I would have invested the money in otherwise.

[00:36:28] So it’s, it’s interesting. It’s an interesting perk. I would totally, if my company had it, I would use espps like crazy.

ADE:

[00:36:36] It’s one of those things that you sort of, it kind of makes you stop and think about it a little bit. And I was sort of of two minds. There was a part of me that wanted to sell it and there was a part of me that wanted to hold it. I think what just sort of made the decision for me was for much of my time in big tech, it was during COVID right? And during those Covid years, you could sort of see the explosive growth.

[00:37:01] And so for me, it was a sort of a light bulb of like, wait, this might last for a while. Maybe I’ll just, I’ll just hold, hold on for dear life since it seems to be going to the moon. That’s it.

Ryan:

[00:37:13] At that time, if I had given you $10,000 of cash, would you have turned around and bought your company stock or would you have put it somewhere else?

ADE:

[00:37:23] For me, it would have been index funds. Just because I’m sort of a risk adverse guy. And so if there was just cash in my hands, sure, I’ll put into index funds.

Ryan:

[00:37:35] Because I see the esgp, it sounds like it has the flexibility of being cash if you want. I guess maybe there’s some tax implication when you sell it.

ADE:

[00:37:46] I don’t know. I didn’t research too much into this. Right. I’m not like a financial guru or anything, but it’s sort of like, you know, if you get money for free, you should take that money. There’s of course, other things. Right. Like I think a lot of companies will give you like a 401k match as well. You should take that as well if you haven’t.

Ryan:

[00:38:09] So then I guess going back to your career, I mean, you worked for 25 years. Is there any notable period of growth for you during your career? And what do you think drove that growth?

ADE:

[00:38:21] Sure, I, I think, I think the most cherished time of my career was when I was working from like a senior software engineer all the way up to a. A senior enterprise architect. Right? So going, going from, from coding to like just designing and integrating and working with different applications. I felt that I learned the most about the craft during that period of time. And what sort of really helped me a lot was having a series of extremely good, knowledgeable, capable mentors that sort of helped me and taught me and kind of guided me to the right path, if you will.

[00:39:07] Right after I became a director, the nature of the work became more. Well, it became more sort of like managerial in a way. Managerial in terms of it was more helping tech leads, helping other architects do their jobs. It was more of selling work to C suite executives all the time. And that just became less fun and there was less growth, in my opinion.

Ryan:

[00:39:35] You mentioned mentoring being big part of your career growth. Is there a number one lesson that you learned from a mentor? And if so, what was it?

ADE:

[00:39:44] So there were a lot of, a lot of good lessons taught to me by several, quite a few mentors, right? But if I were to say that there was one lesson that kind of stuck by me, it was from this one guy. He taught, he said something that was kind of profound to me, right? But I’ll repeat it for everyone. It’s to be flexible, you have to be able to adapt yourself to changing conditions, no matter what the condition is, right?

[00:40:15] No matter what’s your technology, stack, whatever is your CICD pipeline, whatever the client side teams are like, you just have to stay flexible. If you can stay flexible, you’ll survive.

00:40:27 — When age impacted his flexibility

Ryan:

[00:40:27] I remember one of your videos you talked about as you got older and you were working on tech, the ability to be flexible, you were noticing, was becoming harder. It was harder to stay on top of the trends, harder to learn. At what ages did you really notice things were changing for you?

ADE:

[00:40:48] After I turned 40, I still tried to keep up in the tech. I still tried to learn all the frameworks and I just noticed that it took me longer and longer and longer to learn. And I know other people have different sort of experiences, right? Some people are like in their 50s and they’re still learning quickly. But for me it just. This slowdown became so incredibly apparent. And it’s. In a way, it’s a little bit.

[00:41:20] It’s a little bit sad in a way. The only way I can describe it is, like, imagine like an Olympic gymnast, like those guys that can do somersaults in the air, right? And they’re able to do that in their 20s, right? But when they hit their 30s, if they even try one thing, they’ll break a bone. At least to me, mentally. That was what it was like. Like, hey, you know, I can learn this framework in my 20s in, like, a few days and be effective with it in my 40s.

[00:41:48] It’s taking me weeks to learn it. And what I can produce is this halting, clumsy syntax that I probably need to get, you know, an AI tool to help me polish that. That gets me a little bit, but it’s life. You know what? We all get older.

00:42:04 — Why YouTube

Ryan:

[00:42:04] When you think about moving forward in your career, and this is right after you were laid off. Why put out YouTube videos?

ADE:

[00:42:13] Okay, so this is really funny. I’ll show you this. So I was in this headspace where I was feeling really confused and lost, and it was either writing my thoughts onto this book here or shooting a vlog. And quite frankly, I was just lazy. I was just lazy. I’m like, you know what? Why don’t I just do a vlog instead? Because it was suggested I see a therapist, right? And then she suggested, hey, you know, you have all of these thoughts right now and you have anxiety, you have all this thing.

[00:42:48] Why not just log it? And so it was simply easier to turn on a camera than it was to spend all this time writing the words down onto a book. That’s why I did it. Had no clue it would go viral like the way it did, but that’s how things play out.

Ryan:

[00:43:03] What’d you think when, after you put it out and it was going viral, what were you thinking compared to your expectations?

ADE:

[00:43:12] Completely blew me away. I had never done a vlog in my life, right? And I didn’t quite understand just how the YouTube algorithm worked to show that V log to so many people. It was kind of surprising and shocking, right? Got a lot of positive comments from a lot of people in the video. So it seemed like maybe other people were thinking and feeling similar thoughts and feelings as to me. And so it was good.

[00:43:44] It was actually kind of therapeutic to see that I wasn’t alone in all of this.

Ryan:

[00:43:50] I know you go under the pseudonym of Asian Dad Energy, but that video went so viral, and your Face is on it. I’m curious, did anyone, wherever you’re walking around see you and go, hey, you’re. I saw you on YouTube. Has that happened yet?

ADE:

[00:44:05] So quite a few coworkers, both from my ex big tech employer and also from my past companies did recognize me and reach out to me. It was, it was pretty cool. I haven’t, I haven’t yet had someone reach out to me disapprovingly. Right. I’m sure that will happen at some point. Maybe a C suite leader would see this and like, hey man, what the heck are you saying? I’m prepared for those cringy moments as they come, but so far it’s been all positive.

Ryan:

[00:44:34] That’s awesome. Where do you see this going from now? What’s the future look like of your career, of the YouTube channel and all of that?

ADE:

[00:44:43] I don’t know, man. The future is unwritten at this point. I tried after, right after I got laid off, I tried applying to a few big tech companies. Not much came of it. Doesn’t. Doesn’t seem like it’s a very hot jobs market at the moment. And also I’m just sort of thinking about what it is I want to do. Right. I really, I really don’t think at the moment I want to get back to that structure of big tech where you have all these schedules and goals and priorities.

[00:45:14] That, that probably doesn’t mean that much to me as it used to. I’m more thinking, you know, number one, let me see if I can get used to this semi early retirement where I have plenty of time to spend with people I care about. And by the way, I don’t know if you plan on retiring early at some point. It’s been pretty good. Like it’s been over a month and progressively over time I find my, both my physical health and also my mental sort of state of mood just going up and up and up this whole time.

[00:45:51] It’s actually been pretty good. So that’s one thing I want to explore more of this whole semi early retirement or involuntary early retirement that I find myself in. The other thing is just to keep my mind active. I have started a couple of different projects, right? One of them is this YouTube channel, Asian Dad Energy. People seem to like my rants for whatever reason. So I’ll just do some more rants as I feel like it.

[00:46:18] Maybe it’s entertaining, maybe people can find value in it. The other thing is I’ve been coding on my own and working on a couple of small projects. I might put it out there in the world and see how that does as well. And then we’ll just see where life takes me. I mean, for all we know, there could be some incredibly motivating project that, you know, I would jump up and work at it for free, but I haven’t seen it yet.

[00:46:43] So we’ll do, we’ll. We’ll deal with the situation as it comes.

00:46:46 — Speaking advice for engineers

Ryan:

[00:46:46] You talked about you had to sell to the C suite of these clients when you were a consultant. And clearly you speak well. If you had to give advice to a software engineer to improve on speaking, what would you say is the most important things to get right?

ADE:

[00:47:10] Oh, my goodness. It’s hard to say. I would say just relax and don’t associate yourself with the subject that you’re discussing. Right. A lot of times I see people getting nervous because they’re so invested in what they’re doing. There was actually, it’s a interesting story, there was this one engineer that I worked with, absolutely brilliant, so incredibly brilliant that you have never met such a smart guy.

[00:47:39] Okay. But he couldn’t articulate what he was doing meaningfully to people who weren’t necessarily very deep technically. And it was so agonizing when I worked with him because I’m like, man, you’re a genius, man. You just have to communicate this wisdom to the people there. But he couldn’t do it because he thought so seriously of what he was discussing that this was actually a, it was actually an in memory optimization of a kind of a memory cache in a CMS system called Drupal.

[00:48:17] But it was so, he was so tied himself was tied to this subject that he can’t take a step back and just realize, hey, don’t take it so seriously. And you can discuss it in a way that’s so simple that everybody can get it and that way you’ll actually get your point across and get what you want to do.

Ryan:

[00:48:35] I think that’s a very common failure mode for engineers is they speak in the way that they think, but not necessarily in the way that the audience would want to receive it. So you’re talking to someone who has no idea on all of the super deep detail of the thing that you’ve just spent weeks on.

ADE:

[00:48:56] You shouldn’t get that deep the way that you’re thinking about it. It’s like it’s your, it’s your baby, it’s your everything. But you got to take a step back and explain it. Like the audience is five years old. That’s, in my opinion, the best way to do it.

00:49:09 — Advice for younger self

Ryan:

[00:49:09] Okay. And then, yeah, last question I want to ask you is if you could go back to the beginning of your career, right when you just entered the industry and give yourself some advice, what would you say?

ADE:

[00:49:20] I think I would tell myself to work hard, don’t stop learning, and stay flexible.

Ryan:

[00:49:28] Awesome. All right, well, thank you so much for your time.

ADE:

[00:49:32] You’re very welcome. It was a pleasure chatting with you, Ryan.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:45 - Do you need a PhD?

00:06:25 - Research taste and finding problems

00:09:04 - Why become a quant

00:12:01 - What quants do

00:14:53 - How quants and SWEs collaborate

00:16:29 - Quant vs tech culture

00:26:39 - Quant firm tier list

00:27:56 - Quant insider trading and perf culture

00:30:53 - Going back to AI research

00:35:08 - Who the top competitors are in voice AI

00:39:22 - AI startups vs big labs

00:42:08 - State space models vs transformers

00:49:33 - AI labs: research or product?

00:52:38 - Advice for SWEs who want to try AI research

00:56:48 - Advice for younger self

Transcript

00:00:45 — Do you need a PhD?

Ryan:

[00:00:45] Here’S the full episode. When you think about the opportunities that are not available to you without a PhD, what comes to mind?

Nimit:

[00:00:58] Yeah, so I think, you know, there’s not really too many opportunities that are, you know, actually unavailable to, to people without a PhD. But some of them just get a lot easier with PhDs. I mean, so academia is an obvious one that does require a PhD. That was never something I was super interested in for a lot of reasons. But I think some re. Some roles that are definitely a PhD opens a lot of doors to are kind of the two that I’ve had experience with.

[00:01:32] One is sort of doing industry research in AI like I’m doing now or back in the day there were a few different. Industry research in computer science or mathematics was a little bit more diverse, but more and more people are converging towards AI. So I’ll just say like AI research is one of them where having a PhD helps or not that you, you know, a lot of people do AI research without a PhD, but you know, the type and shape of the role can look kind of different.

[00:02:01] And another one is quantitative finance. So again, a lot of people go into quant, you know, out of undergrad, but Certainly having a PhD like opens you up to, you know, some, some sort of different opportunities and it can be a lot easier to get your foot in the door there.

Ryan:

[00:02:17] So if we think concretely, let’s say I was going for an AI researcher role or something like that. Are you saying the PhD helps you in that first step of filtering or does it help somewhere else in the process in getting one of those roles?

Nimit:

[00:02:32] Yeah, so I think it’s both. So certainly it’s a lot easier to get an interview if you’ve differentiated yourself from the pack in some Way, just applying for AI research role at a top firm can be difficult if you don’t have whatever the right schools on your resume, the right internships, the right connections, whatever. But it’s certainly doable. And then I guess, I think less Transactionally doing the PhD can develop a key critical skill set that can help you along your path towards becoming a great AI researcher.

[00:03:16] But of course there is an argument for being thrown into the fire as well and just kind of learning on the job. And that’s certainly an option that works for many people. I think there are some things that are harder to do in industry than in academia, like kind of the more exploratory first principles like fundamental research, without necessarily a direct application. You know, industry definitely skews a lot heavier towards the applied side of things.

[00:03:46] But I think like having that fundamental background can be very valuable depending on what kind of research you’re targeting.

Ryan:

[00:03:52] You mentioned the type and the shape of the role could be different if you had a PhD versus not. Could you give a example of what you mean?

Nimit:

[00:04:00] If you’re working on more like engineering heavy stuff in AI, so, you know, building, you know, training or evaluation infrastructure, you know, working on like, you know, data processing, things like that. Those are not things like a PhD is really necessary for at all. I’d say if you want to do more like sort of pie in the sky type stuff like, you know, architecture, design, things like that, a PhD can be, you know, can be helpful there because, you know, you have more time to kind of explore directions that may not pay off in the short term.

[00:04:36] But you know, again, there are, there are examples of people being successful in, you know, without a PhD, with or without a PhD in both, both domains. So I think like, if your only goal is to be an AI researcher and you’re not super, you know, tied to the, you know, the particular type of work you do, you just really want to get into the field. A PhD is definitely not necessary. But I think like, if you’re, if you’re still in the sort of exploration phase of your career and you want to find a problem that really draws your interest, then a PhD can be a good way to do that.

Ryan:

[00:05:15] You mentioned the PhD skill set or something that you kind of develop when you get a PhD. What is that skill set?

Nimit:

[00:05:23] I would say 90% of the battle in research is actually finding the right problems. So you have to find a problem that is interesting, it’s meaningful, that people are actually going to care if you solve it. You have to sometimes convince people it’s interesting because they might not have thought about it the same way. And then you have to execute and you need to make sure it’s a problem that’s appropriately scoped, that is actually tractable for you to make progress on.

[00:05:51] So I think all of those things were not skills that I had had developed. You know, it was more, you know, just, just like execution was, was my strength. And so definitely that was a, you know, big learning process for Me during the PhD is like that sort of like research taste and problem selection. And this is something that, you know, just like being immersed in the field, you know, really helps with.

[00:06:15] You know, once you’ve read enough papers, you know, talked to enough people, you kind of get a sense of the patterns and trends that are going on in the, in the field.

00:06:25 — Research taste and finding problems

Ryan:

[00:06:25] You mentioned the research taste and finding the right problems. If you could kind of condense what you Learned in your PhD, is there maybe some top tips that kind of lead to you finding the right problems?

Nimit:

[00:06:38] I would say the main things that I find useful are just keeping abreast of the current literature, just reading as many papers as you can. And it doesn’t have to be reading them end to end, just skimming abstracts, you know, seeing what’s going on, what are people thinking about. And yeah, I think the other thing is just working your way up. So, you know, initially, earlier in your career, you want to attack like, you know, very small sub problems that are, you know, that you’re reasonably likely to make progress on.

[00:07:12] Right. So one example, you know, example of this can be like you take a method and you try to extend it to some like, you know, special case or something like slightly different from the original application. And then as you go on and as you mature as a researcher, you can start tackling bigger and bigger problems. So not just kind of extending previous work, but maybe coming up with totally new ideas, things like that.

[00:07:38] So I think there’s a gradual stage of maturation as a researcher and I think some people do try to skip those steps and I think that’s generally inadvisable.

Ryan:

[00:07:47] You mentioned keeping abreast of the literature. What’s the go-to spot for you to find the right AI research papers to read?

Nimit:

[00:07:58] Honestly, Twitter is one of the, probably the main way that I keep up with papers. I think if you follow enough good people on X, I guess your feed becomes pretty curated to that. So that’s usually the first way I find out about stuff, obviously just talking to people, coworkers and so on. But yeah, I think X is my go to. So I try to. I try to curate my feed in such a way that it’s. It’s mostly, yeah, machine learning papers and pictures of cute animals.

Ryan:

[00:08:30] Do you have a good starting point for someone who just wants to plug in?

Nimit:

[00:08:34] I pretty much, like, you know, started with following people I knew from Stanford and, you know, elsewhere, you know, professors whose papers I’d read, like, you know, prominent people at big labs and so on. And then like, anytime, you know, they tweet a paper, they like a paper or something like that, if it’s. If it’s interesting to me, I just click on that and I, you know, follow all the people, you know, tagged in or associated with that work.

[00:08:58] And that’s. Yeah, so I sort of grew my follow list organically via that and Understand.

00:09:04 — Why become a quant

Ryan:

[00:09:04] After your PhD, you became a quantitative researcher at Citadel. Where were you in your career and why did you decide to become a quant?

Nimit:

[00:09:13] Yeah, I joined Citadel securities after graduation from my PhD, and so they’re. Yeah, so basically I had actually interned there right before the summer, right before graduating, and I liked it a lot. Reason I decided to intern just kind of. I wanted to see what else was out there. I had. I had a few friends who had interned at Citadel, at Citadel securities, and, you know, enjoyed it or at other quant firms.

[00:09:40] And I was just kind of curious. You know, by that point, I’d been working in AI research for like four to five years. And yeah, like I said, I was. Generally, when I entered my PhD, I was interested in careers in which I could apply my interests in mathematics and computation. And so I think the three major careers of that form at the time were machine learning research, which I already had experience with, you know, quantitative finance, and then the last one maybe quantum computing, but, you know, it was a much smaller sort of domain and one I had no experience with, although that one’s also kind of blowing up these days.

[00:10:21] So, yeah, quant finance. I was just. Just kind of curious and, you know, I’d heard good things. And so I decided to intern and I ended up liking it a lot. I think it was, you know, refreshing in some ways. Um, you know, like I said, the PhD is a grind, you know, you can burn out at various points. And it was kind of a fresh, fresh set of problems. Totally different environment.

Ryan:

[00:10:46] Well, it’s funny that you say that the grind of the PhD, you kind of took a break to become a cloud Citadel, because I’ve heard that the. The work culture is Pretty intense at these finance companies. Is that the case?

Nimit:

[00:10:59] Yeah. So that is the reputation. But I think that definitely varies a lot based on the team you’re on, the firm you’re at. Yeah, I personally I had a pretty great work life balance, as funny as that might sound as a quant, you know, I think one reason is that, you know, traders typically will work, you know, trading hours or whatever locale they’re in. You know, of course there are markets all over the world, you know, apac, you know, Europe and so on.

[00:11:30] But in the us, US traders are typically working around US trading hours and of course a little bit before and after just to prepare and stuff. And so I think that generally has a sort of trickle down effect on the culture where most people are just kind of really clustered, working around trading hours and then don’t take their work home too much. So even though the work can be done at any time, I think it just, that is sort of how the office culture operates.

00:12:01 — What quants do

Ryan:

[00:12:01] Yeah. When it comes to quantitative finance or quant work, how would you describe the work?

Nimit:

[00:12:06] It really, really depends a lot on both the team you’re on, like the sector you focus on, whether you’re at a hedge fund or a market maker, whether you’re like front office or back office quant, and of course the company. And so some quants will spend all their time just like on alpha generation, so generating new trading strategies and back testing them and so on and putting them into practice and monetizing them.

[00:12:37] Well, some people will focus just on the alpha, some people will focus on the monetization. Or you can be a risk quant. So you’re basically not necessarily generating strategies at all, but just trying to come up with metrics to capture risk and avoid that, reduce risk without reducing, you know, cutting into profits. You might be, you might be doing like data analysis. So you might have like a ton of like historical like trade data and stuff and analyzing them in various ways and so on.

[00:13:09] So yeah, I think, yeah, yeah, like I said, like, you know, hedge fund versus market making, they’re, they’re actually very different problems. So I think the thing that unifies all of them is really, you know, having a strong math background like in.

Ryan:

[00:13:25] The day to day, let’s say, you know, your, your project that you’re currently working on, like what would that look like? What would the shape of that problem look like? And how does math concretely play a role?

Nimit:

[00:13:36] You know, depending on what sector you’re in, you know, there’s a lot of different math that will come into play. I mean, you know, the sort of the backbone of finance is stochastic calculus. So I think that comes up almost everywhere. But then there are other things like numerical optimization, numerical interpolation, things like that. Machine learning, of course, is now more and more firms are getting really deep into the deep learning space, even establishing their own research arms that do LLM type research and stuff like that, numerical linear algebra.

[00:14:15] So there’s a lot of different math and it’s actually very like, it’s actually very diverse in terms of what people are always trying to come up with ways to apply different fields of math to quant. I think some of it is just for fun, kind of because quants are such a mathy intellectual bunch, but there is actually a lot that underpins the entire field. So yeah, I mean, I think, yeah, stochastic. Stochastic calculus is probably the most unifying part.

[00:14:48] Like that’s kind of the, you know, Finance 101 type math.

00:14:53 — How quants and SWEs collaborate

Ryan:

[00:14:53] So you mentioned coding a lot as a quant. And I have had some friends who were swe at Citadel and these various companies understand the roles are quite different. How do quants and swez typically collaborate at these companies?

Nimit:

[00:15:08] Really depends a lot on the company. You know, some, some companies, like, you know, Jane street for instance, like the number of people who are called quants is actually very small and you know, traders themselves are quite technical and like implement a lot of stuff. And then of course they have like software engineers as well. Whereas Citadel I think is a more quant forward firm. So I think, you know, you know, quants might be, if not the largest, like percentage of employees, like it might be like about equal in terms of the technical stuff.

[00:15:36] And so yeah, I think there can be a lot of overlap in what a quantitative researcher and what a software engineer does and also between a quant and a trader. So yeah, it kind of just depends. At some firms I think it’s more divorced where quants are really doing the strategy work and then it’s kind of handed off to software engineers to implement. But at other firms I think you might do a bit of both because, you know, of course, like the person, like if they have the implementational skills, the person best posed to like actually implement something is the person who like understands all the reasons and edge cases and things like that.

[00:16:22] And so, yeah, like I said, I did a ton of coding mostly in C, also some Python.

00:16:29 — Quant vs tech culture

Ryan:

[00:16:29] If you were to compare and contrast finance and tech generally across these roles, what comes to mind?

Nimit:

[00:16:37] So I think a lot of the Skill set, first of all, is actually quite similar. Yeah, like I said, math and computer science were my main interests, and I wanted a job that would leverage both of them. And I think that’s been the case in quant and that’s also been the case in AI research that I’ve done. And so I knew nothing about finance before I joined Citadel securities, but I read a few textbooks that were recommended by people and that was really all I needed.

[00:17:09] And yeah, from there I just drew upon my sort of technical skills. And I think AI research is a lot of the same way. I think if you have really strong fundamentals, you can pick up the rest. So, yeah, in terms of technical skills, I don’t think it was really a rough transition either going, you know, going either way. I think, you know, obviously the, you know, culture is different. You know, SF versus New York, those kind of things.

[00:17:39] Yeah, work. Work hours, I would say. Yeah. I think, you know, quant actually probably has a better work life balance than AI. You know, particularly the level of competition in AI right now. Like, you know, it’s just a very competitive space. And so one of the ways you can gain a comparative advantage is just like by outworking your competition. And that kind of is, you know, what happens in, in practice a lot, A lot of places.

[00:18:05] I know a lot of people who are just like, working around the clock.

Ryan:

[00:18:08] I’ve heard insane stories about the comp structures at quantitative finance firms. Is that all true? Like, is it heavily bonus weighted? And, um, I’ve also heard stuff about garden leave.

Nimit:

[00:18:22] So. Yeah, in terms of comp. Yeah. I think one thing is like, you know, there’s not really standardized levels like there are in tech. You know, you, you can’t just sit, you know, it’s not like someone is just like IC5 and you kind of know, like, what, you know, kind of pay bands. They’re. They. They what, what they’re making. It’s. It’s. Yeah, I think comp is really driven by a few things, you know, how the company does that year, how your team does that year.

[00:18:47] If you are really on, like, the alpha side of things, like, you know, how you, how your particular strategies did that year. And then of course, there’s like, other things that play into it, like seniority, both in terms of, you know, hierarchy, if you’re at one of the firms that, you know, does have a kind of explicit hierarchy, or in terms of like, you know, just like tenure at the firm or years of experience, things like that.

[00:19:08] And so, yeah, I think, you know, quant firms I think are more secretive and partly because of the relative lack of standardization. So it is kind of opaque in terms of how those factors actually combine for your final comp. But yeah, I think it can be very bonus driven if you’re really on the alpha side of things and that attracts some people to that kind of thing where they really would just want, they want to be as exposed as possible to I guess like the fruits of their labor.

[00:19:40] But it is, you know, the downside is it can be much riskier business as well. So yeah, it’s just more variable. But like if you’re you know, more back office type thing, I think the comp is you know, probably a little bit more deterministic if you’re not, you know, directly tied to alpha generation.

Ryan:

[00:19:56] Yeah, it’s interesting I mean because we were talking about AI research versus quants and obviously being a quant is famous for earning a lot if you have generated a lot of alpha. I hear compensation like easily in the millions for, for a lot of these people. But at the same time AI research also popped off too.

Nimit:

[00:20:18] You know, if you’re in the top 1% of either of these firms, you’re going to do very well. Yeah, I mean, yeah, they’re kind of crazy. Yeah, I mean these, these things really do exist where people are making like NBA player salaries and stuff I think you know, for the, for the median case. Yeah, it’s still, it’s still very good. But I think, yeah, it’s, it’s not, not exactly that outlandish. Yeah, sorry, you, you also mentioned stuff about NDAs and stuff and, and garden leave so, or sorry, non competes.

[00:20:49] I guess so. Yeah, I think, you know, so finance firms are, are very, you know, they’re very serious about this sort of thing. You know, unlike in tech. You know one thing I was surprised by, by culturally is how tight lipped people are in finance. Even within a firm there’s things that you can and cannot share across teams or people might just want to be more secretive because they’re protective of their alphas and so if you know what they’re doing, you can reimplement a similar thing and take over some of their alpha because what makes it alpha is that it’s, you know, secret.

[00:21:28] If, if more, the more people know who know about it like the less profitable it’s going to be for any individual. And so yeah, I think it’s, you know, quite secretive. You know, even, even the firms that are, you know, have a reputation for being more open are, are actually quite Secretive versus in tech. You know, people talk about things all the time and so it was a bit jarring for me returning it to tech and like hearing like people, you know, talk about what they’re doing in like a, you know, very open way.

[00:21:53] It was like, wow, like you’re just going to tell me that for free. So yeah, a non compete is probably the most notorious part of this is yes, a lot of firms will have a clause in the contract you sign at the beginning stating that you cannot work for a competitor for a period of time after you leave the firm. And this period of time is typically decided by the company when you leave. But it can be anywhere from, well, it can be zero up to two years.

[00:22:23] I think I’ve heard even up to three years for some places. But I think that’s rare. I think the norm is, I would say the norm is like six months to two years. And so yeah, during this period you’re basically just paid to not work. Yeah, it’s called garden leave because I guess you sit at home and garden or whatever. And it’s actually, I mean it’s actually a quite interesting thing. It creates interesting incentives for some people because you are typically compensated quite well during this garden leave period.

[00:22:51] So it’s not necessarily a, you know, it’s not necessarily a downside for some people. And yeah, basically idea is like, you know, you won’t, you know, leak ideas to your competitors and you know, by the time your garden leave is over, you know, if you have some special alphas or trading strategies, you know, two years down the line they’re probably not even relevant anymore. So it doesn’t even matter.

Ryan:

[00:23:14] You mentioned the secrecy within in quantitative finance and I see a natural incentive here to kind of be hostile or I guess competing within the firm. Because my alpha is my alpha, I’m not going to help you. Did you ever feel that or see stories of that?

Nimit:

[00:23:35] Yeah, no, it’s definitely a thing. You know, people are, yeah, I think a lot of people are, you know, reluctant or even forbidden to talk about any details basically of what they, what they do. You know, some people that’s, you know, some people, you know, won’t even, don’t even like say what sector you know, they work on, you know, at least across companies and stuff like that. So yeah, I think that that’s definitely a thing.

[00:24:01] You know, some firms are set up where it’s like basically pods. So you know, one pod is just responsible for basically all of their P and L and then the firm takes a cut and so, you know, different pods might be working on, you know, very similar things, unknowingly. Right. But they’re not sharing any of the information. And there is some logic behind this because the idea is you want to have uncorrelated, you know, uncorrelated returns.

[00:24:28] So if all the pods are like, you know, talking to each other, sharing ideas, you know, chances are they’re going to start doing very similar things. And then, you know, that exposes you to risk where, you know, what if the thing you’re doing is actually wrong and you know, you can wipe out not just one pod, but an entire team of them. Whereas if people are working independently, then, you know, that’s, that’s not, that’s less of a risk.

Ryan:

[00:24:50] Earlier you mentioned the top 1% of AI researchers and quants are going to do extraordinarily well. I’m curious what sets the top 1% of AI researchers and quants apart from the rest?

Nimit:

[00:25:04] There are a lot of things. I think there are different ways to get to that point as well. Can be raw technical skill. Like some people are just really, really good at what they do. Able to, you know, the prototypical, like 10, 10x engineer, that kind of thing. And they just have, you know, a better, a higher level of intuition and, or execution speed, stuff like that. You know, of course there’s politics involved, like, you know, people who are better at playing the political game can, you know, rise up in the ranks.

[00:25:35] I think in Quantum, one thing is that it’s harder to game the system because there are kind of hard metrics that it’s easier to evaluate how someone does. Especially if you’re alphaquant. It’s quite clear if you implement a strategy and you make the firm a ton of money, that’s obviously going to be recognized. I think in AI it can be a little bit harder, but of course the analog might be you publish a seminal paper, you make a true breakthrough in the field.

[00:26:06] You know, you, you make, you know, the models much better than. Yeah, that, that sort of thing. So, yeah, I think it’s. Yeah, I guess it’s probably similar to, you know, other domains. It’s just a combination of skill and like, you know, playing, playing the game. And I think being in the right place at the right time has a lot to do with it. You know, both in quant, in terms of seeing something before other people do and then like making, taking advantage, capitalizing on market trends and, and, and turning that into a profit or an AI, you know, like having the right idea at the Right time.

00:26:39 — Quant firm tier list

Ryan:

[00:26:39] When it comes to quant firms, I’m kind of curious. There’s all these tier lists out there. What are the top firms and why.

Nimit:

[00:26:47] Rentech, like we talked about, is one of the sort of mythical firms in the space. You know, you can’t really argue with their returns, the historical returns over like a, you know, 20 year, 30 year period. It’s pretty insane. So, you know, Rentech is maybe, you know, the gold standard, you know, depending on who you ask. Then there are other firms like, you know, some of the slightly big, bigger ones like you know, Jane Street, Citadel Jump Trading, Hudson River, I think those are generally very well regarded firms.

[00:27:16] And you know, having that kind of thing on your resume can definitely be, you know, an asset to, to future quant rules and things like that. So yeah, very good firms I think. You know, great technical talent, great returns obviously. And then there are some elite smaller ones similar to Rentech. Smaller, more secretive, less well known, but still very, very excellent returns. So yeah, TGS is one, it’s in Southern California.

[00:27:46] Yeah, XTX is another one of the newer firms. I think Radix is another newer firm that’s in that boat. So yeah.

00:27:56 — Quant insider trading and perf culture

Ryan:

[00:27:56] Are there any stories you working in the space that you think might be interesting?

Nimit:

[00:28:00] You know, finance firms do not mess around. So you hear stories about like, you know, people just doing, you know, dumb things like you know, traders or quants having like an internal WhatsApp group where they, you know, talk about strategies and they’re like, like so as a quant you have like trading restrictions. You have to get all trades pre approved. And so of course if you’re, you know, someone who works in equities or something, you know, you probably are not going to be able to trade those tickers at all.

[00:28:27] But you know, people try to get around it with their little WhatsApp groups or whatever, like telling their friends to, you know, buy these, buy these stocks or something with the profits or whatever. If that happens and you get found out, you know they’re going to go after you, you’ll get fired obviously, there’ll be lawsuits, you can, you can even go to jail. Yeah, there’s like a, a few stories about this because it is, it is against the law.

[00:28:52] And so yeah, heard, heard horror stories about this. That’s one of the things they tell you about in training actually is like, yeah, do not do this similar with like non competes, you know, people going to competitors or starting their own thing or something and like getting accused of taking strategies and stuff like that. Yeah, the all these firms have like, elite legal teams and. Yeah, just not something you want to mess with.

Ryan:

[00:29:17] I’ve heard that in quantitative finance that it’s kind of intense sometimes or rather people may get fired very often. Did you ever have, like, you just were working with someone that kind of disappeared?

Nimit:

[00:29:32] Yeah, yeah. I mean, yes, that. That does happen. Yeah. I think it’s interesting in quant because like I said, yeah, comp is a function of many things among. Among which is seniority. And so I think your job security can actually be kind of U shaped because senior quants, like, you know, even if they’re very good, they just get very expensive after a while because that’s sort of what the market rate is for for senior quants.

[00:29:58] And so, you know, even. Even a good quant, you know, can stop being worth it after a while. Whereas like earlier career quants, you know, you can, you know, they might be very good and also not command, you know, as high of a salary. So, yeah, the job shape is not. Not usually like kind of like an inverse parabola almost. And yeah, I mean, people certainly get fired. It’s a quant in general, I think, has a culture of, you know, well, one is like upper out and two is like, you know, just trimming the low performers.

[00:30:29] And again, I think this can become, you know, especially easy if you’re like, more on the alpha side. Like, if you’re just not making money, like, it can be pretty clear. But in general, even for like, you know, engineers. Yeah, I think there is this kind of culture. Yeah. Traders, of course, it’s. Yeah. Since they’re, you know, making trades and stuff, again, it’s like, very easy to monitor. So I think that can be even more.

[00:30:51] More brutal.

00:30:53 — Going back to AI research

Ryan:

[00:30:53] Why did you leave Citadel to join Cartesia?

Nimit:

[00:30:57] When I joined Citadel, it was partly because I was just interested in learning about a new problem domain and like, you know, learning some. Learning some new stuff, you know, learning about finance in general, I think was also, like, kind of interesting to me. And yeah, I think, you know, I became more financially literate as a result of things and stuff like that. Like, it was, it was. It was a great learning experience for me and I was kind of optimizing for like, growth potential partially as well.

[00:31:21] But yeah, I mean, by that point, you know, I’d been at Citadel for, you know, a couple years and was. Yeah, I think, you know, like, it’s sort of like your growth, you know, at most places will kind of like accelerate for a bit and then like sort of taper off So I think there was still a lot more to be learned had I decided to continue on that path. But I saw what was going on in the field of AI when I graduated.

[00:31:46] Actually, it was right before ChatGPT came out. And so I think a lot had changed even since I joined Citadel. And I heard that the founders of Cartesia were starting this company. And for context, I knew all of them from my PhD at Stanford. They were actually all in Chris Ray’s lab with me. I knew Albert pretty well and so, yeah, I had tons of respect for them. They’re great researchers. I worked pretty closely with some of them.

[00:32:13] Albert was a good friend of mine, knew the other guys. And so it just seemed like a great opportunity and a great time to get back into the field of AI when things were sort of taking off. And I thought it would be great in terms of personal and technical growth. Also, the opportunity to join a small startup was definitely something that was interested in me and kind of like shaped the company and the culture, you know, as one of the earlier employees.

[00:32:40] And so, yeah, it was really. Yeah, I think, yeah, it was all about, all about growth, getting back into AI. And I think like, you know, there is like a, definitely a different risk profile. I think when I graduated my PhD, I was kind of more like risk averse. You know, Quant was like a, you know, stable, you know, lucrative opportunity that, you know, was the right choice for me at that time. Now that it had sort of established myself a little bit, gotten some of that stability, I thought it was, you know, opportune time to take a risk.

Ryan:

[00:33:13] Cartesia, I guess if you could give us some context on the primary problem the company solving and just like what the company’s about.

Nimit:

[00:33:22] Yeah, we are a voice AI company. You know, our current mission is to build sort of the next generation of voice AI and a platform for that. So what that means is we do. Our flagship product is Text to speech. We also have products around Speech to text, Voice agents and stuff like that. And yeah, I think we believe voice AI is the future. It’s actually one of the fastest growing areas of AI. People are using voice AI in many applications, call centers being one of the predominant ones, but also a bunch of applications and entertainment, a bunch of companions, a bunch of different things.

[00:34:03] And so that’s kind of the product set we’re building. And yeah, in terms of why do we choose Voice? So I think Voice is actually a very interesting test bed for a lot of research ideas that we’re exploring. So we also have A sort of research arm of the company that focuses on kind of longer term research around long context, around multimodality, things like continual learning and memory, test time, compute.

[00:34:37] And in general, overall even higher level goals to build real time systems that are truly intelligent and that you can interact with and that can learn from experience. And so I think, you know, building these voice agents, you know, speech to speech models and so on, is, you know, it requires you to kind of, you know, solve some of these problems for the, you know, sort of eventual idea of like a kind of like always on assistant, personal assistant.

00:35:08 — Who the top competitors are in voice AI

Ryan:

[00:35:08] When it comes to this voice AI space. Who are the top competitors?

Nimit:

[00:35:13] So our main competitor is a company called 11 Labs. They’re you know, another voice AI company basically. And yeah, so they, I think had about a 18th month head start on us. Yeah, I actually used to play with 11 labs, you know, long before Cartesia was ever a thing. Just like kind of make like, you know, fun videos and whatnot. And so yeah, it’s, you know, very, very similar company. I think, you know, where Cartesia stands out, I think is, you know, we have sort of a focus on things like latency.

[00:35:50] So low latency is really important for a lot of voice AI applications for naturalness, like the conversation we’re having now, you can’t afford to have a second pause in between each turn of the conversation that just really breaks the illusion and immersion. And so latency is really important for a lot of our customers. We’re continuing to try to push the boundary of sequence modeling and stuff to get better and better quality without compromising on latency and then going into more end to end systems as well.

[00:36:29] So right now the way voice agents are typically implemented is you have a speech to text system that transcribes some text, then you feed this into a language modeling backbone and then you have a text to speech system that will take the text that is output by the language model and speak the result. But this has a lot of problems in terms of latency, again in terms of naturalness, because it’s kind of not an end to end system.

[00:36:55] So there’s a lot of loss in between each of these components and so on. And so yeah, that’s one thing that we’re trying to build towards. But even right now, I would say, you know, even if you just look at our text to speech products, I think, you know, we’re definitely right up there as, you know, one of the leaders in the space. I think, yeah, you know, 11 wins on some languages. We Win on some.

[00:37:18] You know, I would say we have like better voice cloning, things like that. So yeah, we’re trying to become number one in everything. But yeah, I think, you know, like I said, voice AI is a very fast growing space and so a lot of people are jumping into the space. But I think the pie is very large.

Ryan:

[00:37:34] What does it look like if cartesia completely destroys 11 labs?

Nimit:

[00:37:39] I think we already win in terms of things like latency, in terms of cost. I think if we can conclusively win in terms of quality, not just for a subset of tasks and not just for a subset of things, but there are many things that people care about. For text to speech quality, there’s just adhering to the transcript. So actually reading what is put in front of the model, which can be surprisingly hard, especially if you have different languages, especially if special characters, repetitions, whatever, all models struggle with this.

[00:38:11] But there’s also naturalness. Does it really sound like a person saying this or does it sound robotic? Of course, a lot of applications actually care about naturalness even more than just transcript fidelity. And then of course there are all the speed and things like that. And then there are features like voice cloning, accent localization, so taking my voice and making it have a different accent, things like that, controllability, speed, emotion, things like that.

[00:38:42] And so yeah, like I said, I think we have better quality in some areas, maybe worse than in some others. We like to get to number one in as many categories as possible. And so I think that’s, that’s sort of the thing, right? Like, you know, switching costs exist even in AI, I think, you know, depending on the size of the customer, like some customers are reluctant to switch over from, you know, one thing to the other.

[00:39:06] You know, obviously startups can be more nimble, but you know, when, when you’re talking about enterprise scale, you know, this matters and, but like if you are, you know, conclusively show that you’re better in every way, then I guess like at some point it becomes hard to, hard to argue for not switching.

00:39:22 — AI startups vs big labs

Ryan:

[00:39:22] I imagine you could have worked at a big lab, OpenAI, anthropic, et cetera. What’s the main difference in working in an AI startup versus one of these big AI labs?

Nimit:

[00:39:32] Big labs have obviously amazing resources. They have all the compute in the world, tons of researchers and so on. I think one thing is that the flip side of that is that I think big labs can sometimes be more averse to out of the box ideas and a little bit more susceptible to groupthink. Or overarching trends in the field and less willing to take a risk on something different. And then that makes sense, right?

[00:40:01] Because with these great resources, there’s a lot of cost to investigating new ideas that don’t turn out well. Whereas as a startup I think you’re a bit more nimble, you’re able to be a little bit more exploratory if you do it strategically and sort of challenged the orthodoxy in that way. And so that was one of the things like I mentioned that drew me to Cartesia. Albert has a lot of interesting ideas that I think don’t necessarily go with the accepted grain.

[00:40:40] Around the time Mamba came out, people were kind of of the opinion, a lot of people were of the opinion that sequence modeling was kind of a solved problem and all you need is skills like you just take the transformer recipe and you just scale it further and further. Yeah, I mean, Albert showed that, you know, that’s not necessarily the case, right, with Mamba, that you can actually get real advantages in terms of things like, you know, efficiency, computational efficiency, but also like, even in terms of just like raw quality, you know, state space models can be advantageous for a lot of classes of problems.

[00:41:14] Or like things like hybrid models where you take some state safe model layers, some transformer layers, things like that. Another more recent work that we put out at Cartesia was this idea of H nets where so yeah, for context, the way that text modeling is usually done is you take raw text, sequence of characters or UTF 8 bytes or whatever and then you compress it or you represent it as these things called tokens, which are basically little pieces of words or subwords and then you run modeling over that.

[00:41:45] So it’s like a two stage pipeline. We showed that if you actually just go from the raw characters and you kind of learn this tokenization, you learn how to draw these boundaries in between groups of letters instead you can actually get better performance. And so, yeah, that’s the kind of thing, I think challenging accepted ideas, that’s the kind of thing that appealed to me.

00:42:08 — State space models vs transformers

Ryan:

[00:42:08] For context, you mentioned state space models versus transformer. Could you just give a quick primer.

Nimit:

[00:42:13] I guess, without going into too much, I guess, technical detail? Basically the main challenge of Transformers is that the memory that they use at inference time grows linearly with the sequence length. Because what they do is they will take each token and store a representation of it in what’s called the KV cache, the key value cache. And so as your sequence grows longer and longer, you’re still storing all of this information in context in your memory.

[00:42:45] And so for very long sequences this can get prohibitive both in terms of computational costs and in terms of memory. SSMs are different because instead of storing everything into everything in this uncompressed way, they take that information and they compress it. So the size of the state is fixed. And so as a result the cost of doing a certain step doesn’t change with the length of the sequence and the amount of information you have to keep in memory does not grow with the sequence length.

[00:43:18] And so kind of an intuition, our co founder Albert Gu has a great blog on this, is that SSMs are kind of like a brain. The human brain also does not store an unbounded amount of context. It takes in information and it processes it and keeps it in this fixed size state, which is our brain. Of course you can simulate having an unbounded state via use of external tools like writing stuff down and so on, but the core primitive remains fixed, whereas transformers are more like a database where you can kind of recall anything in the context.

[00:43:53] And so I think both of these approaches are complementary. Right? And yeah, so you know, we’re currently exploring kind of, you know, extensions of that analogy. But yeah, I would say that’s kind of the, you know, high level thing.

Ryan:

[00:44:09] Is the sequence just the input. So the longer the prompt, the longer the sequence and therefore more memory consumption at inference.

Nimit:

[00:44:16] That’s right. So the sequence is the prompt plus the response. So you know, as the model is generating the response, you know the context, you know, the context includes what has been generated so far. Right. So you can refer back to what you yourself have said and you’ll figure out what is, you know, what is the next appropriate token to say. And so this can get obviously especially large for multi turn conversations where now the context includes everything that has been said in the entire conversation up to that point.

[00:44:43] And so beyond a point, as I’m sure we’ve all had experience with, if you’re chatting with these language models, it sort of ceases to be that useful maybe after tens or something of turns. And you know, it can be best to start a new conversation. But the challenge was that and you know, of course companies are doing things to try and sort of address or band aid this, you know, for instance, like ChatGPT now like saves some like global context in between conversations and stuff like that.

[00:45:13] But it doesn’t really truly learn from, you know, from your personal proclivities and preferences and like the things you’ve asked in the past. Like there is some semblance of this, of course, but I wouldn’t say that it’s like, you know, truly personal yet in terms of like an actual agent that is like kind of learning and growing every day.

Ryan:

[00:45:34] Yeah, you know, I’ve been using cloud code a bunch and I noticed occasionally it does this thing it says it’s compacting or something like that. I imagine it’s taking the multi turn conversation. I don’t know what it’s doing. Just maybe summarizing it and restoring it.

Nimit:

[00:45:49] Yep. Yeah, there are all, there are all sorts of different ways to kind of compress the KB cache, either sort of mechanistically or kind of doing like a, you know, textual summaries or things like that there. Yeah, this is a pretty active area of research as well.

Ryan:

[00:46:04] You mentioned that the state space models, they have a compressed representation of the KV cache or. And so I’m curious, does that have a trade off in terms of the quality of inference? Is it lossy?

Nimit:

[00:46:19] Yeah. So there are certainly trade offs. Yeah, I think depending on the task. So for very recall heavy or fact based tasks, pure SSM models can lag transformers because the ability of transformers to do this kind of exact in context recall turns out to be very helpful for this kind of task. Whereas for, for other tasks that don’t require this type of thing, SSMs can scale just as well or better as transformers.

[00:46:52] Even for a fixed parameter budget, let alone inference budget, you can kind of get the best of both worlds. A lot of people have shown this by doing a hybrid model. So you just basically interleave state space model and transformer layers with some ratios. And so yeah, Nvidia has put out stuff like this. Even the Quinn, the latest Quinn models follow the strategy as well. So yeah, I think the cutting edge I would say for text is probably in these hybrid models, at least in terms of what’s out there for open source.

[00:47:26] But the interesting thing is that for other modalities like audio, it actually makes a lot of sense to have this compression as an explicit inductive bias. So using SSMS for audio has proven very useful for us. We found that it actually improves performance. It’s kind of almost a free lunch, you know, you get improved performance and improved quality and improved performance inference time. And the reason is that sort of like if you think about what these models are doing, you know, audio is, you know, depending on how you represent it is a very like, you know, there’s very little information contained in any one like you know, time, step or token if you will.

[00:48:10] Of audio. It’s like a frame of, depending on what you’re doing, like 10 milliseconds to 100 milliseconds. And so one frame to the next doesn’t really vary that much. And so compressing these into a sort of fixed size state can actually makes a lot of sense. As opposed to text, which is a much sort of densely informational modality one word to the next. There is actually a ton of information contained in each of those tokens.

[00:48:38] And so you know, compression is less, you know, it’s kind of already like pre compressed if you’re using a token level representation. But yeah, even so, I think, you know, hybrid models, I would say hybrid models I think are the future in that regard.

Ryan:

[00:48:55] I see. Okay, so it’s because the modality itself has, I guess, redundancy in the data. That means that this lossiness is actually an asset rather than a problem.

Nimit:

[00:49:08] Exactly, yeah. So yeah, I think there’s a lot of interplay between modality and architecture. It’s definitely not something you cannot design your architecture independently of your data. And so yeah, kind of this co design and thinking about multimodality from a fundamental level, this is one of the research problems that I mentioned that kind of drives a lot of the work we do here.

00:49:33 — AI labs: research or product?

Ryan:

[00:49:33] When you think about companies that focus on product versus research, what pattern do you think is most effective?

Nimit:

[00:49:43] So I think, you know, personally, and this is one of, you know, also one of the reasons I decided to join Cartesia, I think it is very important to have have both. I think like, so there are, you know, several startups popping out recently that are really, you know, focused on core research and not don’t even necessarily have an idea of how to productionize it or turn that into a product or revenue stream.

[00:50:09] I think I personally am fairly skeptical of this approach. I think for a few reasons. I think first of all big labs have tons of resources and also have large teams focused on this sort of thing. I think ultimately the goal of a company is to make money. And so I think eventually, if you are a company of this form, you need to eventually deliver massively outsized returns at some point. So I think you’re taking a big risk where it can kind of be all or nothing type thing.

[00:50:47] I think the flip side of a sort of product only company that’s built on AI models that are built by other people. I think that is like risky in the sense that you don’t have as much of a moat. So you know, like we saw this with, you know, the initial ChatGPT or you know, going from GPT3 or GPT4. Right. A lot of these wrapper companies kind of just got made obsolete by the fact that the base models improved so much that they could often just do what the wrapper was trying to do by themselves with, without very much scaffolding.

[00:51:21] And so it became kind of thing you can just build in house rather than needing another company to post process the output of these models. I think being in the intersection is actually quite valuable for many reasons. I think having a product, a real product that customers use is something that can drive the research so you see firsthand the issues and you can use that to drive your next iteration of modeling, try and fix these issues not as a band aid, but from the ground up at the model level itself.

[00:52:01] And so I think having control over the models is very important when you’re building an AI product. Which is not to say that there’s no room for any non research company. I think it just, it has to be in the right kind of space. And so yeah, I think Cartesia is a great blend of research and product. I would say we’re first and foremost a product company, but we want to build the best products we can and we believe that that requires us to actually solve some of these fundamental research problems.

[00:52:36] In order to do that.

00:52:38 — Advice for SWEs who want to try AI research

Ryan:

[00:52:38] I think there’s a lot of people who want to get into AI research. I mean, I was just talking with a friend today who’s a SWE and he’s saying I don’t think software engineering is going to be around in n years or something like that. So he’s been investigating and I’m curious, do you have any advice for someone who is technical and wants to move into AI research?

Nimit:

[00:53:00] My philosophy has always been to try and build up my technical skills as much as possible. I think if your fundamentals are good enough, at some point the opportunities will just come to you rather than the other way around. And so I would say just focus on getting as good as you can at coding. At AI, read tons of papers. I think math skills and math intuition are really important. And so that’s what I’ve kind of been optimizing for ever since undergrad when I realized what I wanted to do was at least some combination of math and computer science.

[00:53:37] And so I’ve always more focused on kind of building up those fundamentals. And I think that is the way to get your foot in the door. I think like bigger companies it can be a bit harder to pivot. You know, teams or what you work on. And so for, for someone like that, I think switching like teams or companies can, can be like, you know, sort of the only path forward. I think you, you can kind of get siloed in a little bit if you’re at a bigger company sometimes.

[00:54:04] Although I do think, you know, some companies are better about it. And you know, I have seen people transition from sweze to research and stuff like that. So I think, you know, this is one of the areas where getting a sort of qualification on your resume can be useful, like getting a master’s in AI at least, or something like that can help when you’re looking to make a sort of lateral career change like that.

Ryan:

[00:54:30] You’re saying there’s kind of two common paths. One would be get more education and use that qualification to kind of pivot directly into AI research or go to a startup where you can kind of like mold yourself into an AI research role.

Nimit:

[00:54:46] That’s kind of right. But I think even if you want to go to a startup, right, and you want to, but you want to sort of switch from a SWE track to AI track, like there’s gotta be some, there’s gotta be something behind it, right? Like you have to have some evidence of a skill set, whether it’s like sort of, you know, organically grown or from, you know, from, from schooling. But I think it can probably be a lot easier to get your foot in the door if you have some evidence of it on your resume.

Ryan:

[00:55:14] So like, let’s say you were, you hired at Cartesia and then that person comes to you and he’s like, hey, I want to do more AI research. In that case, is that something where it’s like just flip a switch and next project is AI research project?

Nimit:

[00:55:32] This has actually happened in Cartesia itself. We have had people transition roles like that. So I think it is definitely easier at a startup which can be a bit more flexible just because everyone kind of knows everyone. And so you can get a sense of whether this might be an appropriate career change just by knowing the person for a while. And so, yeah, I mean, we’ve actually done, you know, people have done this in, in Cartesia with, with, you know, a lot of success.

Ryan:

[00:56:02] Do you have a biggest regret when you look back on your whole career?

Nimit:

[00:56:06] I mean, I, I think I often overthink things and I think I have spent a lot of time regretting, you know, past decisions that turned out not to matter in the end. And I kind of regret the amount of time I spent regretting other things. So, you know, I try, I try to learn from that now. You know, I, I think like, you know, don’t sweat the small stuff. Like, you know, you know, minor setbacks happen and they happen.

[00:56:26] But I think, you know, there’s a risk of, you know, putting too much stress on yourself and you’re like beating yourself up and stuff like that. And those are just like, not productive ways to spend your time and they don’t make anyone feel good. And so I think, yeah, I try, you know, I try not to regret stuff because, yeah, I think it’s just not, not a super good use of time.

00:56:48 — Advice for younger self

Ryan:

[00:56:48] If you had to go back in time and you could give yourself some advice when you’re just entering the industry, what would you say?

Nimit:

[00:56:56] Focus on building the deep technical skills. Yeah. Don’t waste time with sort of trifling stuff or spreading yourself too thin. Yeah, just focus on what you want to focus on. I guess basically the skills that you want to leverage in your day job. Just do those and get good at those. And that’s where you should spend all your time at work. Um, and yeah, you make it sound so simple. Maybe it is. You know, it, it is, it is a simple, it’s kind of a simple recipe that’s very hard to follow.

[00:57:31] Right. Like, it’s very hard to maintain that discipline. Uh, it’s kind of like, you know, you know, what’s the secret to being healthier? It’s, you know, exercising, eating right. And those are things that are just very much easier said than done. Um, but yeah, I think that it is, it is that simple. Awesome. Cool.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:37 - Why he wanted a demotion

00:07:32 - Why Senior Staff at Meta was different

00:16:01 - Meta vs Google culture

00:19:09 - Downleveling at Google

00:23:17 - Why he’s willing to be transparent

00:25:11 - Best quality of life eng level

00:30:42 - Senior Staff promo at Google

00:42:27 - Mentorship stories

00:43:11 - Biggest career regret

00:46:46 - Advice for younger self

Transcript

00:00:37 — Why he wanted a demotion

Ryan:

[00:00:37] What makes you willing to share? Here’s the full episode. You mentioned in a post, a pretty famous post that I’ll link so people can see that Meta didn’t have a process for demotion and you were looking for demotion. Can you talk about what led you to wanting a demotion at Meta?

Igor:

[00:01:02] Yeah. So when you join a big company, Meta is not the exception here, but you’re joining a big company at a very high level, like senior staff, there are certain expectations from your performance, right. Come performance review like Meta, it’s called psc, you’re being judged against other senior staff engineers in the organization. Right. And Meta, as of a year ago, they started to lay off people.

[00:01:33] They started doing the Amazon thing, essentially, let’s lay off the 10% lowest performance. So what it means is that you have at most one year to ramp up to be comparable in performance to other old timers in the company. And also, what does it even mean to be an E7 in a company like this? Right. You need to be an expert, a big expert in the field. You need to know a lot of people, you need to understand the infrastructure that you’re working with very deeply.

[00:02:10] You probably need to be easily familiar with any piece of code that you, your team is working with. And also you need to know all the surrounding, like what all the surrounding teams are doing. And you need to know all these people, like all core people around you, and they need to know you and they need to trust you. Right. That’s a very difficult thing to achieve within like a relatively short period of time.

[00:02:37] I initially saw that, okay, when you were so again joining Meta day one, I know less than an intern sitting next to me who spent maybe two weeks, who was hired two weeks prior to me. So I start from level zero and then hopefully you be able to get to something that L3 would perform. So you can take some very easy task that under supervision from other folks, you can kind of promote yourself to a junior engineer.

[00:03:12] Then you do some more stuff, you learn additional things. You can get promoted to a less junior engineer like E4. And then slowly you ramp up to more and more higher levels. Essentially you start from zero and you climb the ladder as fast as you can. Yeah. So you ramp up yourself to the level that you supposed to achieve and you know, the more senior you are, the more levels you need to jump through.

[00:03:41] And I believe that within this year and two months that I spent in Meta, maybe if I’m generous with myself, I, I maybe achieved like E6. So I, I don’t feel like I reached E7.

Ryan:

[00:03:56] What did you see that made you think you weren’t living up to the expectations?

Igor:

[00:04:00] It’s a self judgment crossly that, you know, I. This scale where we call numbers, it’s not like quantum states. Right. There are many things in the middle. Right. It’s a continuous scale so it’s very hard to say where you are on the ladder. But again, comparing myself to other senior staff engineers, the work that I knew, I felt like if anybody going to be on a chopping board, that’s going to be me.

[00:04:40] That was one thing about this ramp up process. And the second thing I also noticed that I really enjoyed doing the coding stuff. Just sitting down, debugging things. That’s what I really loved about the job. So when during my ramp up, when I was around like E5, E6 territory, that’s what I really wanted to do. Right. I felt like as a, you know, as someone who was programming from age of 12, I really like coding, debugging, designing, mentoring more junior people.

[00:05:18] But like a senior staff engineer, it’s more than that. It’s like a leader who you know, spends most of the time in meetings and design docs and touches much less code. And it’s a slightly different type of work. And just by going through this process I realized that actually I want to be in E5, E6 territory. And then I asked the management, can I actually drop a level? And I understand that for the company it’s a difficult thing to implement because I already have some granted stock, I have a certain compensation package and so how do you like, you know, how do you execute that?

[00:06:06] It’s not a well polished process. I know that they can switch levels when you go from one title to another. So let’s say you were a director and you want to become an ac. So they do have some of that like demotion process and frankly I don’t know how they do it. But yeah, like Staying in the same job category and dropping level, just like they said, it’s not possible.

Ryan:

[00:06:36] I see. So when you asked if it’s possible for demotion, your manager went to HR and kind of the result was just, it’s just an impossible thing to do.

Igor:

[00:06:47] You know, I asked the question and I got the answer no. Maybe if I pushed harder, if I like went to talk to the, you know, VP or something, they would have made it possible. But at the same time I, you know, I also felt like I’m too far outside of my comfort zone. And before coming to Meta, I worked a little bit at Cruise, and before that I worked at Google for 14 and a half years. And that was my comfort zone.

[00:07:18] I just had this option that like the, the easy way out essentially, right. It’s had the recruiter who was contacting me periodically, like I already to come back to Google. So this time I just said, yeah, let’s talk.

00:07:32 — Why Senior Staff at Meta was different

Ryan:

[00:07:32] When you were at Google and at Cruise, you were at these senior levels as a senior staff engineer. What’s the difference that makes it so that you can perform there, but not at Meta?

Igor:

[00:07:45] So when I came to Google, I started from the lowest E3, like L3 level, so junior software engineer. And slowly over the many years, I raced through promotions, many times failing, like they could promote it, and then, you know, a year or two later asking again and then climbing the ladder. My last promotion to senior staff happened because I was able to accomplish something big in the project that I worked.

[00:08:18] I worked in that project for several years, so I was already like an expert. I knew everybody, right? Everybody knew me. And I was able to build something that I can be proud of. But again, coming then, then leaving Google and going to Cruise, it was also very difficult for me to ramp up there because you’re coming starting from zero. And the first year was very difficult for me in Cruise. But over time I found like a, a relatively safe, like, you know, my found some comfort zone there until Cruise as a company went bad.

[00:09:01] Like they had the accident and then they had layoffs and then people were just quitting all the time, like losing people. So I decided that I want to leave as well. But before, before the accident at Cruise, I actually was quite happy. I was able to do some coding. I was able to do like mentoring of other folks and I was able to also like act as an ubertiel a little bit like, you know, guiding the team on things that, you know, they need to do.

Ryan:

[00:09:35] You successfully ramped up the senior staff at Cruise and then at Meta, it was a bit harder. What, what were the differences between those two ramp ups in your opinion?

Igor:

[00:09:45] Cruise was a smaller company, so less infrastructure to learn. The team is like less crowded space I would say. Like you can easily carve out space for you to grow into. Like, you know, like there is lack of people and you can say, okay, well this project needs someone to work on and you can just go and work on this stuff. At Meta, I felt like it’s quite crowded, like especially among the CGO folks.

[00:10:20] It felt a little bit like they have too many, frankly. At least in the Orchid where I worked. So I don’t know about all of Meta, but it felt like the space is a little bit crowded. You need to find scope for yourself.

Ryan:

[00:10:38] Is there anything that you would have changed in your onboarding process that could have made things different?

Igor:

[00:10:45] Probably. Again, I didn’t switch companies that many times in my career. I started at Google, spent so many years there, then I only switched crews and then Meta. So I’m not very well experienced in switching companies and essentially this whole ramp up process is new to me. And I also felt like the management. Meta also doesn’t know how to onboard senior folks because most of the senior folks in the company, they grew within the company to those senior levels, even if they came to this specific org from another org within Meta.

[00:11:30] But at least they already have the meta knowledge. They know how to run the job in the cluster. Right. I think I spent a lot of time just reading docs and trying to build this foundational knowledge. I think I spent too much time doing that. It is much better to just get your hands dirty and just try doing stuff and then doing those things in parallel. You build something, accomplish some tasks and also you’re learning in parallel.

[00:12:08] And I did more of the learning part, less of the building. Yeah.

Ryan:

[00:12:13] Is there something that you wish was done or maybe something that would have helped you from the manager side?

Igor:

[00:12:20] Just guiding me, right? If they say like this other person just ramped up like you know, three quarters ago and here’s what they did, I would have followed that recipe. We just didn’t have the recipe. Like I don’t have any complaints against my management. Like they, you know, they tried to support me the best. They just also lacked this expertise of like ramping up with senior folks, you know. So it was a learning experience for all of us.

Ryan:

[00:12:52] I’ve heard in conversations, some other people as well, that I guess job mobility as a senior IC actually becomes progressively riskier or maybe scarier, I guess because you get so used to your existing org and all of that. Is that something that you’ve seen in other people or like peers as well? In that senior IC job mobility is a lot tougher.

Igor:

[00:13:21] I’ve seen folks at Google that left Google and then came back a year or two later saying that it didn’t work out well for them. I. Yeah, it’s, you know, people don’t share things like that openly usually, so it’s much harder to, to hear those stories. You hear the success stories, you don’t hear the failures. Also, I think that if I kept trying, like instead of going back to Google, I went to, I don’t know, some other company, I would be much like, you know, more experienced in ramping up.

[00:13:59] And then maybe, maybe if you do this often enough, it becomes a habit and then, and then you can do it easily.

Ryan:

[00:14:08] You mentioned that you, you thought you weren’t meeting expectations based on your own judgments. I’m curious, did you ever get any feedback from your manager or anyone saying, hey, you need to do more for your lawful’s expectations?

Igor:

[00:14:22] Yeah, I did get feedback. I just, you know, don’t necessarily want to openly share all of that. But yeah, I was so like just the few months leading to my living matter, I was working on a project that initially I thought it will take me like two weeks to accomplish. It was like a small thing that I thought would be easy to do and turned out to be actually a lot more involved and a lot more complicated.

[00:14:51] And I felt like even if I finish it, which I almost did, I almost brought it to completion, even if I finish it and launch it, everything successful, it is still not an E7 level project. You cannot still justify my level with the project being completed. Probably it’s my fault in which projects I pick, you know, or how I underestimated the complexity of the thing.

Ryan:

[00:15:24] Because you’re going back to Google, are you going back to an org where you have all the existing context and relationships and all of that?

Igor:

[00:15:34] No, no, everything will be new to me. New people, new infrastructure, new everything. But at least I know, you know, I know how Google operates. I know the, you know, the culture. There are differences in cultures also between the companies and I think that meta also like in terms of internal culture at Meta, I didn’t feel it like it fits me the best.

00:16:01 — Meta vs Google culture

Ryan:

[00:16:01] What’s the biggest cultural differences between Meta and Google?

Igor:

[00:16:05] Meta tends to set up very ambitious goals. They give you like oftentimes they will give you very arbitrary deadline saying like, okay, this project, you need to finish it within by September 15th, whatever, like one month from now. And then everybody works hard. There is a lot of pressure. You need to constantly send the updates to the leadership, how the project progresses. Comes the date, the deadline, the project is still unfinished and it just keeps dragging on.

[00:16:38] And everybody is fine with that. Again, I don’t know. I haven’t seen all of Meta. I’ve seen the specific Org where I work and it feels like, so then what was the purpose of setting this aggressive deadline? Right. Then you can do it once. But after five times going through this project, through this artificial pressure with non realistic goals, then people just say okay, you know what, 5:00pm I’m going home.

[00:17:10] I’m not going to try to, to, to work hard because I, I know that there is like this, this whole pressure is artificial. Right. I think Google is much more reasonable in that regard. Like if, if there is a deadline, it’s probably for good reasons and, and people would like, would work hard, but usually there will not be pressure. Again, talking about Google, like as of 10 years ago, I’m not sure that it’s still true today.

[00:17:39] But you would be pressured to fix something or to accomplish something when there is really exceptional case for that. You wouldn’t be pressured to work under pressure for years.

Ryan:

[00:17:57] Yeah, I mean I get the sense that the industry as a whole is kind of becoming a little bit more intense when it comes to execution and deadlines. I’ve heard some people saying that Google as well has felt a little bit of pressure too, but hard to say. It depends on the Org, I’m sure.

Igor:

[00:18:15] Yeah, it could be. And again, I didn’t feel personally much pressure at Meta, but just talking to other folks, seeing how they work and operate and as I said, there is this cultural thing where the leadership wants updates and everything and then they’ll, the people on the ground, they actually kind of dismiss it essentially. Like it’s, it’s like they, you know, it almost feels like a elementary school, like where the teacher is yelling but kids are still playing.

[00:18:47] Like if you’re yelling too much, it’s, it’s like it stops working essentially if you, if you constantly put pressure on your people, it just doesn’t work anymore. Like you say. So what do you do next? You start laying off people? Yeah, it might work to some degree, but again people adjust to everything.

00:19:09 — Downleveling at Google

Ryan:

[00:19:09] That’s crazy. You mentioned that you, when you reached out to Google too, you, you explicitly asked for kind of a demotion or going back as a L6 when before you were an L7. Was that a challenge or was that just a very straightforward process with the recruiter?

Igor:

[00:19:25] It was a challenge for the recruiter to also make this happen because they have a process for bringing people in back at the same level or even like level up. They don’t have a process for bringing people at a level down, but they made it possible for me. Again, I can totally see how like a person comes back and says like, I want to be a level down. It’s like there might be some red flags. There may be, you know, things that I’m not telling.

[00:19:53] You know, it’s a gamble for a manager to hire a person like that. Also just by giving me offer at the level down, they understand that it will be, they will not be able to match my compensation, you know, the previous place. So like would they even accept it? Right. And so I had to like and assure my recruiter that yes, I’m happy to accept the offer at the lower compensation. Like, you know, a lot of people will not do something like I did because it’s like a significant drop in compensation.

Ryan:

[00:20:33] You mentioned stuff that people might be hiding from the recruiter. What comes to mind when you, you mentioned that?

Igor:

[00:20:41] I don’t know, maybe you, maybe you did something in the company that, you know, you like a fireball offense or something. Like, like I, I don’t know, like if, if you’re a manager and somebody comes to you with like, you know, saying like, yeah, previously I was this level, now I want to be level below, please hire me. It does sound fishy. Like why don’t you like what’s wrong with you? Like you, like, you know, there are so many companies out there.

[00:21:12] Like why, why would you come back to, to this company? Like why don’t you try something elsewhere? Like I, I don’t know what questions popped up in my hiring manager’s head when he saw this.

Ryan:

[00:21:25] Google’s current process, do they do host matching or what was the. Like did you meet up with the hiring manager beforehand or just through the recruiter?

Igor:

[00:21:34] The recruiter sent me a few openings and asked me which one. Sound interesting? I spoke to a few hiring managers. One was sound like sounded like the best match for me. And then I spoke to a few engineers on the team. Then I spoke to the manager’s manager and that was it. So another nice thing for me is that they didn’t require me to re interview.

Ryan:

[00:22:03] Oh, interesting. But it had been like three years.

Igor:

[00:22:07] Maybe over yeah, three and a half years. So yeah, they didn’t ask me to re interview actually that was another Interesting thing that they said, like, if you’re coming back as E7, you definitely don’t need an interview because you were already L7. But if you’re going to be like L6, then you might need to do a coding interview because who knows? If you wrote code that you know while you were out.

Ryan:

[00:22:35] That’s kind of funny.

Igor:

[00:22:36] Yeah, I got an exception from coding.

Ryan:

[00:22:39] I see. That’s funny. You got promoted out of competence in the lower levels.

Igor:

[00:22:46] Yes. But it’s understandable. When people go to higher levels, they write less code and maybe they get rusty. Who knows?

Ryan:

[00:22:55] It’s kind of crazy that you could boomerang after it had been three years or more.

Igor:

[00:23:01] Yeah. To be fair, I don’t know if that’s a general Google’s policy or it was just a special case for me. I don’t have any friends that did this recently to understand if they did it for anybody else.

00:23:17 — Why he’s willing to be transparent

Ryan:

[00:23:17] You mentioned earlier that a lot of people don’t share this kind of stuff about the motions and these types of things. And I’m curious, what makes you willing to share?

Igor:

[00:23:29] I’m a generally very open person. Like, I like sharing my life experiences, positive and negative, with friends and, you know. Yeah, I’m generally quite open. And also I’m quite confident. That’s another thing, like, you wouldn’t share if you worried about your career, if you’re worried about. There are many things that can go wrong when you share stuff like this. I’m very privileged, essentially.

[00:24:03] I don’t need the work visa, so I can live in this country without being attached to a certain employer. My spouse has health insurance, my kids are already grown up and off to college and don’t have mortgage on my house. So, like, I can afford not to work for a few years. I can. You know, I don’t. I’m fine with, like. Let’s say I said something in this chat right now and Google says, we don’t want you for some reason.

[00:24:40] Like, I’m totally fine. I will just. So that’s a position of privilege, right?

Ryan:

[00:24:48] Definitely. I mean, you’re free. Yeah. You’re entirely free. You’re your own person. Which is awesome.

Igor:

[00:24:53] Yeah. So that’s a position of privilege that many other people wouldn’t have. So if another person asked me, like, should I post about my challenges at work? I would probably say no, unless you feel so safe and secure.

00:25:11 — Best quality of life eng level

Ryan:

[00:25:11] And you mentioned that Google, you were asking for promos, you went from their lowest level up and you were pushing and pushing for promos, and that’s A very different mindset from now, which you are kind of pushing for the motion. I’m curious, is there did something major change that made you not motivated to go for promos anymore in your career?

Igor:

[00:25:36] Essentially, if, if you are in a. A senior Engineer right at E5 or E4, you don’t even know like what’s the life of like two levels above you looks like, right? You, you probably, you can imagine what, what happens at like one level above you. But, but it’s, it’s very hard to see like what, what does a principal engineer do? Like most people don’t know and I don’t know by going through this like I realized that my happiest time was when I was like working in a relatively small team, doing a lot of coding, debugging, designing, mentoring more junior folks.

[00:26:20] That’s, that’s where I felt the happiest.

Ryan:

[00:26:22] Let’s say money is not a constraint at all and it was just which engineering level has the best quality of life in your opinion, which one would you say?

Igor:

[00:26:36] Senior engineer like E5, L5? Yeah, that’s probably least pressure. You’re still shielded by probably. If you’re working on a team, you probably have a TL who is 6 on the team, who sends all the updates to the upper levels and shields the team from all this stuff. You have the management, also the lower level managers who shield you from all this stuff. So you can just do the stuff that hopefully you enjoy what you’re doing.

[00:27:16] Like, not everybody enjoys doing work. Like a lot of people do it just for money. Personally, I was programming since I was 12 years old, so I really like this stuff. I got into this job because I love it and it happens to pay well, but I would do it even if it wasn’t paying well. Like a lot of people come to software engineering because just because that’s the Monday thing.

Ryan:

[00:27:41] So then if it was easy, as easy to do, would your ideal situation be two levels of the motion?

Igor:

[00:27:50] That’s too extreme. I think I can still enjoy being like E6, I quite comfortable with and maybe I haven’t been E7 long enough to, to like to get comfortable in the, in the, in the role. Like maybe if I just kept doing it for some time longer, I would have started enjoying it. Yeah, like, you know, one thing is like was I was. I really like, you know, imposter syndrome is something that everybody has regardless of what your level is.

[00:28:26] I’m sure Elon Musk has imposter syndrome. So, you know, I’m questioning myself like Was I really, you know, qualified to do to be E7? And my answer is like partially yes, but not fully. It’s like a multidimensional thing. And in some dimensions I probably was good and then some dimensions was not as good.

Ryan:

[00:28:49] You mentioned working with other peer ICs that were also really high level is there skills that you saw that they had which would have closed the gap for you personally or something that you thought made them so strong.

Igor:

[00:29:04] It’s often a mistake to compare yourself against a group of people. Like that’s what gives you a lot of imposter syndrome where you’re saying like, oh, those people around me, they’re so smart, they are so good at talking, they’re so good at doing presentations, they’re so good at communicating, they’re so good at leading. But they’re turns out that it’s like one person is good at this, another person is good at this and you’re comparing yourself against a team of people.

[00:29:32] And so that’s always, you need to be careful not to make that mistake. So yeah, I think given enough ramp up time I would have been able to reach full productivity and being useful to the company. At E7 I just felt like I, like I was not enjoying the ride essentially and I didn’t have to like. And I debated a lot before making the decision to leave, right. I was talking to friends, I was talking to family and pretty much everybody was telling me like, are you crazy?

[00:30:07] Like you’re not laid off yet, nobody showed you the door, why would you do something crazy like that? And even if you’re leaving, like you don’t enjoy it, why won’t you try E7 elsewhere? Like why do you need to go level down? There is a lot of peer pressure and my answer to that was just like, yes, I can try that, I can do that, but I don’t want that. I’m capable of running a marathon, but I don’t like running a marathon.

[00:30:37] I don’t like running. Why would I do that?

00:30:42 — Senior Staff promo at Google

Ryan:

[00:30:42] You mentioned that you kind of rose through the ranks at Google. I’m kind of curious, what was the project that got you promoted to senior staff at Google?

Igor:

[00:30:52] It was a project in ads. ADS builds a lot of machine learning infrastructure because the ranking of ADS is essentially a recommendation system. And I was working on this machine learning infra and when they came to the team, they were doing training on CPUs, not on GPUs, but on CPUs. And TPU was a new thing, new hardware that Google was developing. It was about to be released like maybe a year or two when I joined the team and my role was to make sure that this adds training infrastructure can run on those TPUs.

[00:31:42] And additionally when you train machine learning models the scale matters and the utilization matters and there are lots of nuances that can like, you know, your model may be training but things go wrong. It’s online training which is also like something that most people don’t know. Like LLMs don’t usually train in online training mode. So there are lots of experience on the team. Like the whole infrastructure that existed within that team was built for the CPU training and it was polished over, I don’t know, over 10 years.

[00:32:30] It was polished to be super reliable, super like you well monitored, the unit tested, everything is like very rigid, very polished. Right. And here I am coming and building a completely new piece of infrastructure to run on TPUs and it has to be as good as the old, you know, as polished, as monitored, as tested, as reliable. So it was a big project and you know, it succeeded. Yeah, it’s something to be like I was proud of what I built and so that got me promoted.

Ryan:

[00:33:08] What was the operating model of that project? Were you TL or were you kind of writing a lot of it yourself or how would you describe the execution?

Igor:

[00:33:18] Yeah, initially it was like a quite exploratory project. So it was just too pure people. It was the TL plus one person building like a prototype plus a lot of people from other teams helping, like the TPU team, the compiler team. There are lots of other people who were helping us succeed, like the TensorFlow team. We were writing TensorFlow back then, you know, so it’s also like where you need credibility.

[00:33:49] That’s why knowing people and them knowing you is very important, like trusting you. If you don’t have that, you cannot succeed. So it’s a really cross team collaboration, very big project. And again hardware like this, you need to decide how many of these chips you want to order and they will be delivered like 18 months from now before and right now you don’t have any. It’s not like you can try out and, and see.

[00:34:22] So like, and those chips are super expensive, right. So it’s, it’s a very risky thing to do for the company. And yeah, it’s like a lot of risk taking a lot of like leadership skills that needed for, for doing stuff like that.

Ryan:

[00:34:44] So you were on the, well kind of the product team, not actually product, you were machine learning infra for the ads Org and then all the underlying infra teams were helping collaborate with you. You mentioned compilers, maybe some training infra other infrastructure teams.

Igor:

[00:35:04] Yeah. And once the things started working a little bit, the team grew quite a bit more. A lot more people joined the effort and like started like working polishing and then again like first generation GPUs comes in and then they already tell you we already designed, the next generation will be coming next year. So you’re already rushing to adjust your infrastructure to the next thing and that’s like pretty much every year you upgrading your infrastructure.

Ryan:

[00:35:34] How did you test the initial builds of this infrastructure if you didn’t have the chips to begin with?

Igor:

[00:35:41] A lot of things were just about like input reading and processing, which you can do without having the chips. The difference between CPU training where you have the CPU and Intel cpu, they can do floating point arithmetic and integer arithmetic in parallel. And the floating point arithmetic is relatively slow. So input reading and processing was free on the same cpu. Going from that mode of operation to you have a relatively weak computer with eight TPU chips.

[00:36:27] Each one of those TPU chips is like order of magnitude faster than the CPU that you had before. If several orders, I think it’s like two orders of magnitude faster. Now there is no chance you can do the input reading on the host of those TPUs. So you need to build the input processing pipelines and everything. But that’s something you can test outside of without having TPUs. Actually we did many times we did this test where people say next tpu will be 2 times more performant than the previous one.

[00:37:08] And a nice test to test your infrastructure is to say like okay, we will just remove all computation from our models. Just let’s see how fast we can feed the data in, take the outputs out and process this whole thing. Like how fast your system can run. If the TPU was infinitely fast. And that’s a nice test to do to see that you don’t have bottlenecks. And you do find a lot of bottlenecks all the time.

[00:37:39] Like things that previously were thing that schedules which data to train on previously was never an issue. And then suddenly this is your bottleneck in this model and many things like that.

Ryan:

[00:37:58] So if I’m understanding correctly, the the TPU consumes data orders of magnitude faster than the CPU one. So everything around the processing unit, like the data loading and maybe I don’t know, the scheduling and all the other things that are around the TPU needed to be scaled and tested. And that’s a lot of what you did to get promoted is that Right, Yeah.

Igor:

[00:38:23] Plus, you know, there are always like funny things that you get where like certain resources are more available than other resources. For example, you’re training on some insane amount of data, petabytes of data. This data is stored on spinning disks because there is not enough SSDs in the world. So spinning disks, if you look at the history of spinning disks, they’re getting bigger in terms of storage space, but the speed at which they rotate is staying constant and the speed at which the head moves staying constant.

[00:39:01] So if I gave you previously, let’s say we go back in year 2010, a typical disk would be like maybe 200 gigabytes and it has a certain throughput of how much it can read. Now fast forward to today. The typical disk is like 6 terabytes. So your data center has a lot fewer disks to store the same amount of data, but the throughput is like an order of magnitude less. And you’re coming to the people responsible on building data centers and you say, I need more disks, not for storage, I need more disks for throughput.

[00:39:44] And you usually would hear an absurd thing that says disks are very cheap compared to everything else in the data center, super cheap. But you cannot get them. You know, you cannot easily go and buy like a few thousand disks and easily install them. Like you need racks, you need power, you need, you know. Yeah, it just, it’s similar to how like during COVID we all ran out of toilet paper, right? Like the stores just couldn’t, couldn’t.

[00:40:15] Why would you run out of toilet paper? Right. It’s just because it’s bulky item that the stores cannot easily store on the shelves. So that’s roughly.

Ryan:

[00:40:25] You mentioned having to predict 18 months out how many TPUs you’d need. I’m curious, when you look back on that prediction of how many you needed, did you over order or under order?

Igor:

[00:40:37] Usually under. We tried to be relatively conservative because your main goal is to make money, right? Like with LLMs. If you look at this current world of LLMs, they just order more. Like we want more giga watts of data centers. They don’t even really like. None of these companies is profitable with LLMs, right? They’re all losing money and they all fine with that. But when you have a mature business like ads, you need to be making money on that.

[00:41:09] So if you wasted, if you over provisioned and this goes to waste, then it’s not good. But then it constrains what kind of models you can train. Also models don’t stay the same. So when you’re saying 18 months from now we’ll not be training what we’re training today. But you don’t know what you’re going to be training. The attention mechanism wasn’t in use 10 years ago and suddenly it’s needed. So your chip might not be even capable of doing that.

[00:41:43] So TPUs are quite specialized hardware. So the, they are not as generic as GPUs. And there were certain things that we had to work around a lot of times where certain functionality was very difficult to implement.

Ryan: