Apple Podcasts | Spotify | Google Podcasts

Edited Transcript

Can you tell us about the scale of data Uber was dealing with when you joined in 2018, and how it evolved?

When I joined Uber in mid-2018, we were handling a few petabytes of data. The company was going through a significant scaling journey, both in terms of launching in new cities and the corresponding increase in data volume. By the time I left, our data had grown to over an exabyte. To put it in perspective, the amount of data grew by a factor of about 20 in just a three to four-year period.

Currently, Uber ingests roughly a petabyte of data daily. This includes some replication, but it's still an enormous amount. About 60-70% of this is raw data, coming directly from online systems or message buses. The rest is derived data sets and model data sets built on top of the raw data.

That's an incredible amount of data. What kinds of insights and decisions does this enable for Uber?

This scale of data enables a wide range of complex analytics and data-driven decisions. For instance, we can analyze how many concurrent trips we're handling throughout the year globally. This is crucial for determining how many workers and CPUs we need running at any given time to serve trips worldwide.

We can also identify trends like the fastest growing cities or seasonal patterns in traffic. The vast amount of historical data allows us to make more accurate predictions and spot long-term trends that might not be visible in shorter time frames.

Another key use is identifying anomalous user patterns. For example, we can detect potentially fraudulent activities like a single user account logging in from multiple locations across the globe. We can also analyze user behavior patterns, such as which cities have higher rates of trip cancellations compared to completed trips.

These insights don't just inform day-to-day operations; they can lead to key product decisions. For instance, by plotting heat maps of trip coordinates over a year, we could see overlapping patterns that eventually led to the concept of Uber Pool.

How does Uber manage real-time versus batch data processing, and what are the trade-offs?

We use both offline (batch) and online (real-time) data processing systems, each optimized for different use cases. For real-time analytics, we use tools like Apache Pinot. These systems are optimized for low latency and quick response times, which is crucial for certain applications.

For example, our restaurant manager system uses Pinot to provide near-real-time insights. Data flows from the serving stack to Kafka, then to Pinot, where it can be queried quickly. This allows for rapid decision-making based on very recent data.

On the other hand, our offline flow uses the Hadoop stack for batch processing. This is where we store and process the bulk of our historical data. It's optimized for throughput – processing large amounts of data over time.

The trade-off is that real-time systems are generally 10 to 100 times more expensive than batch systems. They require careful tuning of indexes and partitioning to work efficiently. However, they enable us to answer queries in milliseconds or seconds, whereas batch jobs might take minutes or hours.

The choice between batch and real-time depends on the specific use case. We always ask ourselves: Does this really need to be real-time, or can it be done in batch? The answer to this question goes a long way in deciding which approach to use and in building maintainable systems.

What challenges come with maintaining such large-scale data systems, especially as they mature?

As data systems mature, we face a range of challenges beyond just handling the growing volume of data. One major challenge is the need for additional tools and systems to manage the complexity.

For instance, we needed to build tools for data discovery. When you have thousands of tables and hundreds of users, you need a way for people to find the right data for their needs. We built a tool called Data Book at Uber to solve this problem.

Governance and compliance are also huge challenges. When you're dealing with sensitive customer data, you need robust systems to enforce data retention policies and handle data deletion requests. This is particularly challenging in a distributed system where data might be replicated across multiple tables and derived data sets.

We built an in-house lineage system to track which workloads derive from what data. This is crucial for tasks like deleting specific data across the entire system. It's not just about deleting from one table – you need to track down and update all derived data sets as well.

Data deletion itself is a complex process. Because most files in the batch world are kept immutable for efficiency, deleting data often means rewriting entire files. We have to batch these operations and perform them carefully to maintain system performance.

Cost optimization is an ongoing challenge. We're constantly looking for ways to make our systems more efficient, whether that's by optimizing our storage formats, improving our query performance, or finding better ways to manage our compute resources.

How do you see the future of data infrastructure evolving, especially with recent AI advancements?

The rise of AI and particularly generative AI is opening up new dimensions in data infrastructure. One area we're seeing a lot of activity in is vector databases and semantic search capabilities. Traditional keyword-based search is being supplemented or replaced by embedding-based semantic search, which requires new types of databases and indexing strategies.

We're also seeing increased demand for real-time processing. As AI models become more integrated into production systems, there's a need to handle more GPUs in the serving flow, which presents its own set of challenges.

Another interesting trend is the convergence of traditional data analytics with AI workloads. We're starting to see use cases where people want to perform complex queries that involve both structured data analytics and AI model inference.

Overall, I think we're moving towards more integrated, real-time, and AI-aware data infrastructure. The challenge will be balancing the need for advanced capabilities with concerns around cost, efficiency, and maintainability.

What is Software Infrastructure?

In this context, software infrastructure includes hosting options (self-hosted, on-prem, cloud providers), infrastructure components (database services like RDS; container orchestration like Fargate, Kubernetes; message brokers like Kafka), and architectural choices (e.g., single-tenant deployments).

Security

As a rule of thumb, the security requirements of a prospect are proportional to their headcount, the criticality of data being stored by the product, and the riskiness of the capabilities of the product. For example, a prospect with 1,000 employees will likely have a stricter security review process compared to a prospect with 100 employees.1 

Security teams weigh the value of a product along with the risk introduced by using it, therefore reduction in perceived risk may directly improve sales. Security features might also be useful marketing collateral for security-focused products. We focus this section on a key architectural decision that impacts security.

Tenancy

Software tenancy relates to how co-located customer data is within the application. For example, single-tenant applications tend to have some architectural separation between data between two customers. Anecdotally, security-conscious prospects in the procurement process often include questions about tenant isolation controls, and many products market tenancy controls to appeal to security-conscious buyers.

Multi-Tenancy

Most SaaS applications are multi-tenant, which implies the same application and data-storage is shared across multiple customers. That doesn't completely preclude enterprise sales: Large-scale products like AWS S3, Dropbox and Box have multi-tenant architectures, and are still used by sensitive enterprise customers, since prospects have developed trust over time.

Enterprises are concerned for the right reasons: multi-tenant applications are, unsurprisingly, prone to cross-tenant data leakages. These are tricky problems to prevent due to their heterogeneous nature.

Single-Tenancy

Some products claim to have better data isolation via a single-tenant architecture. Data is still controlled by the product, but the architecture separates customer data at some layer of the stack, so code bugs are less likely to lead to cross-tenant data leaks. For example, Material Security separates data and application at the Google Cloud Platform (GCP) project layer, and Pinecone and Turso are database platforms that offer single-tenancy as a first-class construct.

Single-tenant architectures tend to have higher infrastructure costs as well as developer productivity slowdowns. For example, managing a database per customer is generally more expensive than managing a single larger multi-tenant database. There’s also additional tooling required to manage accounts or databases across separated tenants. This is a very common problem - before public cloud, many B2B software products were deployed on-premise, and to support cloud versions, the shortest path often involved taking on-premise software and running it single-tenant in the cloud. In conversations with Chief Technology Officers (CTOs), I’ve noticed several such products aim to migrate to multi-tenant architectures as the most feasible route to reduce infrastructure costs and maintenance burdens.

But there has been significant recent innovation to offset single-tenant costs. For example, a combination of server-less architectures for application logic, along with smaller, cheaper databases like SQLite rewired for single-tenant production storage may make it palatable to run single-tenant architectures efficiently.

Cloud-Prem + Control/Data-Plane Cloud Split

Some products like Planetscale, GitPod and Hasura have split control and data-planes, where data resides in customer cloud accounts, but control and configuration management is in the product’s cloud account.

In this approach, the control plane is deployed as a traditional SaaS application to the product’s cloud, but it communicates to the data plane - a smaller, well-scoped system in the customer’s cloud accounts, which is where the customer’s data is, or where the risky actions take place. The communication between the control and data plane is secured by methods like mTLS because both the server and the client need to know and trust each other.

This provides both security and developer productivity (therefore product) improvements. For example, security teams can fully audit access to their data, while products can quickly iterate on feature-heavy control-planes.

However, this may not be sufficient for some security teams, as the product still controls the customer’s cloud account.

On-Prem / Customer Managed Cloud Accounts

For even more security-conscious customers, products may be hosted on non-cloud servers, or customers-managed cloud accounts. Many regulated organizations, especially in finance and healthcare, tend to require this setup for complete control - a security vulnerability in the product doesn’t matter, as long as it’s correctly air-gapped by the customer.

There is a significant tradeoff here: these have the highest maintenance burden. For example, Gitpod gave up seven-figure revenue by no longer supporting their on-prem product due to high installation and maintenance costs.

Even popular enterprise-focused products like ServiceNow discourage on-premise usage via complex decision-flowcharts and warnings.

On-premise maintenance costs are often high due to inconsistent release cadence, version skew, limited debugging workflows, and limited observability. Customers often control the upgrade process (and tend not to upgrade), which implies the product often stays several versions behind. Getting access to logs is challenging if the system is air-gapped or not allowed to ship logs to product owners. I’ve noticed that products tend to avoid these completely, or significantly increase deal size to make the costs worth it.

Compliance

Products that sell to sensitive customer segments need to meet the regulatory/compliance requirements of their customers, which in turn can force changes to your architecture.

High Maturity Compliance

For more involved compliance procedures like Fedramp, a product will find it difficult to be compliant if their cloud provider or database is not compliant. This is another case where cloud-prem deployments help - a product itself may not need to be Fedramp compliant if it can be hosted in the customer’s cloud environment.

Government

Many cloud providers support specific regions/data-centers for government customers. For example, AWS has a GovCloud region, and Azure supports a version for the US Government. Products selling to governments often market that they’re hosted in these environments. For example, GitHub has a landing page that demonstrates that it can run on AWS GovCloud.

Therefore, if a product wants to reduce friction in government sales, it’s advantageous to use a cloud provider that has government region support.

Regional Deployment - Europe, Australia, India (and others)

Products selling across the globe often support deployments to specific regions. Products like Vanta, Merge and Sentry support EU deployments for customers who prefer that their data lives in the EU, and cloud providers like AWS have also responded by launching Digital Sovereignty functionality. 

Many states, countries, and regions are increasingly passing privacy/data residency legislations that add compliance friction to the sales process.

Customer Comfort

Even the most discerning buyers (like the CIA) seem to use products hosted on AWS, Azure, or GCP. These services aren’t perfect - Azure is prone to many security issues, and GCP is prone to too many bugs and freak incidents. Still, the benefits provided by the top cloud providers are so high, that the security/reliability risks are often a “cost of doing business”.

Anecdotally, enterprise customers in certain segments tend to prefer products based on the Microsoft stack, especially if they already use Microsoft products. For example, Harvey, a legal startup, markets that its deployed on Microsoft Azure.

If a product is targeted towards a certain segment of enterprise customers, it’s worth understanding whether that segment would feel more comfortable with the Microsoft stack. For example, the Microsoft CEO mentioned that 65% of the Fortune 500 use Azure OpenAI. This is consistent with the idea that once a product is trusted to safeguard customer data, it doesn’t necessarily need to be hosted on-premise.

Marketplaces

There’s increasing value for B2B products to sell on cloud marketplaces. Big companies often have “spend commitments” on their cloud providers that go unused. Therefore, there’s a large financial incentive to spend on the cloud, and software sold on cloud marketplaces count towards these spend commitments. Additionally, cloud marketplace products go through the same billing process as the rest of cloud spend, which helps the customers’ finance team with one less relationship to manage.

To take advantage of a cloud provider's distribution channel, products need to host significant portions of their software with that cloud provider. For example, here’s a screenshot of GCP’s marketplace requirements.

Therefore, many new B2B products seem to be available on all clouds, so that they can sell on all marketplaces. For example, Sigma Computing announced that their product is available on all three major cloud providers, and is also available on all three marketplaces. Therefore, it may make sense to use cloud-agnostic infrastructure components, like Kubernetes and Kafka, so that it’s simple to deploy to multiple clouds early on.

Conclusion

There’s a growing number of tradeoffs to consider when considering a product’s architectural choices due to the surprising number of non-engineering related implications.

On a personal note, I’m looking to learn how multi-tenant products can better prevent cross-tenant breaches. If you’re struggling with the same problem, want to share any resources, or are willing to chat - please send me an email to reliability [at] substack.com

tl;dr: GraphQL's modular structure generally leads to code that instantiates excessive promises, which degrades request performance. Benchmarks show a 2-3x latency increase.

Background

NodeJS is known for its non-blocking I/O operations, thanks to its design of the event loop. The event loop is a single dispatcher thread that enables concurrent I/O operations without forcing developers to reason about managing parallelism.

All synchronous work in NodeJS happens on the event loop thread (other than a few isolated multithreaded features like worker_threads). When the event loop is managed well, and I/O is a true bottleneck, NodeJS can be a very efficient/scalable technology.

On the other hand, if a request does a lot of processing on the event loop, it will block other requests on that container. NodeJS applications are particularly susceptible to sporadic performance issues due to noisy neighbors - other heavy request handlers - that overly consume the event loop. Additionally, GraphQL's resolver structure can result in more promise overhead compared to REST endpoints, which may cause suboptimal user-perceived latency if not managed carefully.

GraphQL and the Event Loop

GraphQL enables a modular design for APIs. For example, we can define a type in our schema, and define one resolver for that type, regardless of where that type appears in the graph.

// schema.
// we only have to define one type resolver for User
type User {
  name: String!
  accountId: String!
  friends: [User!]!
}

type Query {
  user: [User!]!
}

// query.
// we can pick the fields we need.
query q {
  user {
     accountId
     friends {
       name
     }
  }
}

This modular design is great for developer experience but leads to promise-heavy code. Each promise adds a minuscule, but non-zero amount of work for the event loop, which is discussed here.

To demonstrate, let’s say we want to write a feature that retrieves a user’s items on a shopping site. We might build a REST endpoint that looks like this:

/user/items/details

This would be powered by a few SQL queries.

SELECT item_id from users where user_id = $user_id;

SELECT details from items where id in (item_ids);

A well-structured REST endpoint would have some relatively simple code that makes these database queries and massages the data back into a desired format. We would have no more than a few promises invoked and resolved in the request lifecycle.

In GraphQL, we would be encouraged to write a query like this:

user {
  items {
    id
    details {
      id
      ...otherFields
    }
  }
}

If we have it well-structured as GraphQL resolvers, we might have type resolvers for users and item details.

resolvers: {
  // returns { items: String[] }
  User: async (req) => GetUserById(req.auth.user), 
  ItemDetails: async (req, { itemId }) => GetItemDetailsById(itemId),
}

When executed, a GraphQL query with nested fields will result in a promise per field being created, such as:

const user = await getUser()
for itemId in user.items:
  const { id, ...otherFields } = await GetItemDetailsById(itemId)

If we use Dataloaders to prevent the N+1 query problem, this would translate to the same SQL queries as we described in the REST endpoint case. So the I/O cost would be as optimized as possible. But we would create one promise per item in a loop. And each promise adds work to the event loop.

I’ve written a benchmark of a GraphQL server that returns users to demonstrate the impact — the overhead increases as we increase the number of promises involved. We chose two GraphQL servers - Apollo Server + Express, and Mercurius.

The benchmarked queries return the same data - but one wraps every field response in a promise, and the other returns data synchronously. We return 100 items per user.

We see that wrapping each user and item in a promise causes a 2x or 3x increase in request latency.1

An invalid criticism here is that real-world GraphQL resolvers perform I/O, so the overhead here will reduce significantly as a percentage of the time taken by the resolver. A well-tuned database can perform two SQL queries to return 10k items in < 100 milliseconds, which is a reasonably small percentage of the high latency caused by the GraphQL server here (regardless of Express or Mercurius).

Real-world code is even messier - we might check feature flags or perform other ~async work in a resolver, which further increases the number of promises the event loop has to process.

Diagnosing the Problem

It’s useful to know how to diagnose this problem in certain operations. First, we should confirm that our application is actually blocked on the event loop. NodeJS exposes useful perf hooks to measure event loop utilization.

Next, we should confirm that our event loop isn’t blocked by code we control. In my case, I confirmed this by inspecting CPU profiles. If the event loop is occupied for >50ms with no obvious culprit in sight, the culprit is likely in the runtime.

Next, we can confirm how promise-heavy our code is through the following code snippet. Each GraphQL operation should increase the number of promises created and give us a clue about how promise-heavy our code is.

import async_hooks from 'async_hooks';

let count = 0;

const hook = async_hooks.createHook({
  init(asyncId: number, type: string) {
    if (type === 'PROMISE') {
      count++;
    }
  },
});

hook.enable();

setInterval(() => {
  console.log(`Promise count: ${count}`)
}, 1000);

Another practical approach to determine whether the event loop is a blocker is determining the difference between client-reported database query latency, and database-reported query latency. For example, I noticed that client-side reporting of certain database queries was often >100 milliseconds, even though we were making an indexed query in a table with <1000 rows. As expected, we couldn’t replicate such a slow performance when manually querying our databases. This slow-down was because the event loop was overwhelmed after making database requests, so even though the database responded to certain requests very quickly, the web application did not get around to processing the response until after a significant delay.

Open-Source and Promises

Since async/await only affects request throughput in certain, promise-heavy conditions, most open-source code is not heavily optimized to prevent unnecessary promises. For example, GraphQL Shield, one of the most popular GraphQL authz libraries, assumes every field resolver is async. Therefore, it constructs a promise for every field in a GraphQL response, which further amplifies the number of promises created in the lifecycle of a request.

Typescript and JavaScript do not prevent developers from unnecessarily marking functions as `async`, so we need eslint rules like require-await to avoid unnecessary `await` calls on functions that do not construct promises.2

APM and Promises

Finally, we can incredibly slow down promise execution if we use Async Hooks, a deprecated but widely used NodeJS feature. Async hooks help us track asynchronous resources. For example, a tracing library might desire to track a request across callbacks and promises.

Unfortunately, any code we import may rely on this feature and can auto-enable it. For example, `dd-trace`, Datadog’s APM library (and likely many others), uses the feature to provide traces across promise executions.

To understand the cost of async hooks, I added them to the GraphQL benchmark using Apollo Server. We run the same benchmark as above with AsyncLocalStorage (another slow, but supported feature), and with Async Hooks.

We see that Async Hooks adds a significant amount of latency to resolvers that return lists of data.

We roughly see that Async Hooks roughly adds a 3 - 3.5x overhead to these resolvers.

Datadog engineers are diligently working to reduce this overhead by contributing Node.JS and v8 features. However, improvements in this area are critical to get right and take time to be implemented.

Potential Solutions

In general, we want to reduce the overhead of promises and reduce the number of promises we invoke.

Reducing Promise Overhead

To reduce promise overhead, we want to minimize promise introspection features like Async Hooks in production.

Reducing the Number of Promises

To reduce promises invoked, we have a few areas to consider.

We could remove the use of GraphQL middleware, especially ones that assume every field is async.

We could also rewrite GraphQL queries to use fewer async type-resolvers by writing one-shot resolvers - a single resolver that manually queries the database and returns all the data needed for a performance-sensitive query, rather than relying on GraphQL to hydrate nested fields via type resolvers. Let’s take our earlier query:

user {
  items {
    id 
    details {
      id
       ...otherFields
    }
  }
}

We could write a one-shot resolver that implements the entire query:

const UserResolver = async (root, { userId }) => {

  const [user, items] = await User.findById(userId);
  const itemDetails = await ItemDetails.find({itemIds: items.map(i => i.id)});

  return {
    items: items.map(item => {
      const details = itemDetails.find(d => d.id === item.id)  
      return {
        id: item.id,
        details: {
          id: details.id,
          ...details
        }
      }
    })
  }
}

Instead of multiple batches of promises, we fetch the user, items, and details in one shot. This brings up the meta-question about why use GraphQL in the first place, but that’s a larger conversation for a separate time.

Apple Podcasts | Spotify | Google Podcasts

Nora provides an in-depth look into incident management within the software industry and discusses the incident management platform Jeli.

Nora's fascination with risk and its influence on human behavior stems from her early career in hardware and her involvement with a home security company. These experiences revealed the high stakes associated with software failures, uncovering the importance of learning from incidents and fostering a blame-aware culture that prioritizes continuous improvement. In contrast to the traditional blameless approach, which seeks to eliminate blame entirely, a blame-aware culture acknowledges that mistakes happen and focuses on learning from them instead of assigning blame. This approach encourages open discussions about incidents, creating a sense of safety and driving superior long-term outcomes.

We also discuss chaos engineering - the practice of deliberately creating turbulent conditions in production to simulate real-world scenarios. This approach allows teams to experiment and acquire the necessary skills to effectively respond to incidents.

Nora then introduces Jeli, an incident management platform that places a high priority on the human aspects of incidents. Unlike other platforms that solely concentrate on technology, Jeli aims to bridge the gap between technology and people. By emphasizing coordination, communication, and learning, Jeli helps organizations reduce incident costs and cultivate a healthier incident management culture.

We discuss how customer expectations in the software industry have evolved over time, with users becoming increasingly intolerant of low reliability, particularly in critical services (Dan Luu has an incredible blog on the incidence of bugs in day-to-day software). This shift in priorities has compelled organizations to place greater importance on reliability and invest in incident management practices. We conclude by discussing how incident management will further evolve and how leaders can set their organizations up for success.

A less obvious aspect of CI cost is that, in some cases, it can increase at a quadratic rate over time. John Micco, the ex lead of the Google Test Automation Platform group, mentioned that Google would spend more on CI than the rest of its compute combined if it didn’t introduce serious optimizations along the way.

We’ll explore the worst-case scenario: an engineering organization that uses a monorepo (a single repository that stores all the code and assets for a project) and runs the entire test suite on every code change (no selective testing).

There are two primary dimensions to the growth in CI cost. The first dimension is the size of the engineering team. As the engineering team grows, the number of code changes to the monorepo being tested in the CI system often grows linearly1.

The second dimension is the size of the codebase and the corresponding size of the test suite. Codebases and test suites often grow with new features over time. Additionally, software engineers are often hesitant to delete tests unless they’re confident that the system under test is deprecated and unused. So every code change tests a slightly larger codebase, which takes some more CPU time. Since there’s no selective testing, each test adds CPU time, regardless of parallelism.

In summary, as the organization grows, there are more code changes, and there’s more CPU time spent testing each change. Multiplying the increasing number of code changes with the growing CPU time needed for each change results in a quadratic rate of growth in CI CPU time, which is proportionate to resource use.

There are several strategies to mitigate this growth. Selective testing is a key strategy for reducing costs and ensuring faster feedback times from CI on code changes. Examples include Tinder and Dropbox for algorithmic selective testing and machine-learning approaches from larger organizations like Meta. Additionally, there are products and organizations focused on selective testing as a service. Other approaches include file-based test selection. Multirepo systems sidestep the problem but have other downsides like trickier code-sharing.

CI cost can also be much harder to optimize through one-time projects. It’s hard to meaningfully reduce the test-suite size in a large organization, and there’s often no easy way to retrofit better test selection in a poorly structured codebase. Other options like running fewer tests may disproportionately affect developer velocity.

Maintaining lower CI costs involves general code quality improvements. This includes ensuring the codebase remains modular, preventing the blind copying of slow and inefficient test patterns, and making sure that sufficient test tooling, like mocks and fakes, is available for developers to use.

Overall, CI cost exhibits the challenging combination of growing quadratically and being hard to optimize. By actively monitoring CI costs and employing strategic optimization techniques, organizations can maintain a balance between resource efficiency and robust testing, fostering a sustainable software development lifecycle.

Apple Podcasts | Spotify | Google Podcasts

My view on developer experience and productivity measurement aligns extremely closely with DX’s view. The productivity of a group of engineers cannot be measured by tools alone - there’s too many qualitative factors like cross-functional stakeholder beuracracy or inefficiency, and inherent domain/codebase complexity that cannot be measured by tools. At the same time, there are some metrics, like whether an engineer has committed any code-changes in their first week/month, that serve as useful guardrails for engineering leadership. A combination of tools and metrics may provide the holistic view and insights into the engineering organization’s throughput.

In this episode, we discuss the DX platform, and Abi’s recently published research paper on developer experience. We talk about how organizations can use tools and surveys to iterate and improve upon developer experience, and ultimately, engineering throughput.

GPT-4 generated summary

In this episode, Abi Noda and I explore the landscape of engineering metrics and a quantifiable approach towards developer experience. Our discussion goes from the value of developer surveys and system-based metrics to the tangible ways in which DX is innovating the field.

We initiate our conversation with a comparison of developer surveys and system-based metrics. Abi explains that while developer surveys offer a qualitative perspective on tool efficacy and user sentiment, system-based metrics present a quantitative analysis of productivity and code quality.

The discussion then moves to the real-world applications of these metrics, with Pfizer and eBay as case studies. Pfizer, for example, uses a model where they employ metrics for a detailed understanding of developer needs, subsequently driving strategic decision-making processes. They have used these metrics to identify bottlenecks in their development cycle, and strategically address these pain points. eBay, on the other hand, uses the insights from developer sentiment surveys to design tools that directly enhance developer satisfaction and productivity.

Next, our dialogue around survey development centered on the dilemma between standardization and customization. While standardization offers cost efficiency and benchmarking opportunities, customization acknowledges the unique nature of every organization. Abi proposes a blend of both to cater to different aspects of developer sentiment and productivity metrics.

The highlight of the conversation was the introduction of DX's innovative data platform. The platform consolidates data across internal and third-party tools in a ready-to-analyze format, giving users the freedom to build their queries, reports, and metrics. The ability to combine survey and system data allows the unearthing of unique insights, marking a distinctive advantage of DX's approach.

In this episode, Abi Noda shares enlightening perspectives on engineering metrics and the role they play in shaping the developer experience. We delve into how DX's unique approach to data aggregation and its potential applications can lead organizations toward more data-driven and effective decision-making processes.

Apple Podcasts | Spotify | Google Podcasts

In this episode, we delve into Wall Street's high-frequency trading evolution and the importance of high-volume trading data observability. We examine traditional software observability tools, such as Datadog, and contrast them with 3Forge’s financial observability platform, AMI.

GPT-4 generated summary

In this episode of the Software at Scale podcast, Robert Cooke, CTO and Co-founder of 3Forge, a comprehensive internal tools platform, shares his journey and insights. He outlines his career trajectory, which includes prominent positions such as the Infrastructure Lead at Bear Stearns and the Head of Infrastructure at Liquidnet, and his work on high-frequency trading systems that employ software and hardware to perform rapid, automated trading decisions based on market data.

Cooke elucidates how 3Forge empowers subject matter experts to automate trading decisions by encoding business logic. He underscores the criticality of robust monitoring systems around these automated trading systems, drawing an analogy with nuclear reactors due to the potential catastrophic repercussions of any malfunction.

The dialogue then shifts to the impact of significant events like the COVID-19 pandemic on high-frequency trading systems. Cooke postulates that these systems can falter under such conditions, as they are designed to follow developer-encoded instructions and lack the flexibility to adjust to unforeseen macro events. He refers to past instances like the Facebook IPO and Knight Capital's downfall, where automated trading systems were unable to handle atypical market conditions, highlighting the necessity for human intervention in such scenarios.

Cooke then delves into how 3Forge designs software for mission-critical scenarios, making an analogy with military strategy. Utilizing the OODA loop concept - Observe, Orient, Decide, and Act, they can swiftly respond to situations like outages. He argues that traditional observability tools only address the first step, whereas their solution facilitates quick orientation and decision-making, substantially reducing reaction time.

He cites a scenario involving a sudden surge in Facebook orders where their tool allows operators to detect the problem in real time, comprehend the context, decide on the response, and promptly act on it. He extends this example to situations like government incidents or emergencies where an expedited response is paramount.

Additionally, Cooke emphasizes the significance of low latency UI updates in their tool. He explains that their software uses an online programming approach, reacting to changes in real-time and only updating the altered components. As data size increases and reaction time becomes more critical, this feature becomes increasingly important.

Cooke concludes this segment by discussing the evolution of their clients' use cases, from initially needing static data overviews to progressively demanding real-time information and interactive workflows. He gives the example of users being able to comment on a chart and that comment being immediately visible to others, akin to the real-time collaboration features in tools like Google Docs.

In the subsequent segment, Cooke shares his perspective on choosing the right technology to drive business decisions. He stresses the importance of understanding the history and trends of technology, having experienced several shifts in the tech industry since his early software writing days in the 1980s. He projects that while computer speeds might plateau, parallel computing will proliferate, leading to CPUs with more cores. He also predicts continued growth in memory, both in terms of RAM and disk space.

He further elucidates his preference for web-based applications due to their security and absence of installation requirements. He underscores the necessity of minimizing the data in the web browser and shares how they have built every component from scratch to achieve this. Their components are designed to handle as much data as possible, constantly pulling in data based on user interaction.

He also emphasizes the importance of constructing a high-performing component library that integrates seamlessly with different components, providing a consistent user experience. He asserts that developers often face confusion when required to amalgamate different components since these components tend to behave differently. He envisions a future where software development involves no JavaScript or HTML, a concept that he acknowledges may be unsettling to some developers.

Using the example of a dropdown menu, Cooke explains how a component initially designed for a small amount of data might eventually need to handle much larger data sets. He emphasizes the need to design components to handle the maximum possible data from the outset to avoid such issues.

The conversation then pivots to the concept of over-engineering. Cooke argues that building a robust and universal solution from the start is not over-engineering but an efficient approach. He notes the significant overlap in applications use cases, making it advantageous to create a component that can cater to a wide variety of needs.

In response to the host's query about selling software to Wall Street, Cooke advocates targeting the most demanding customers first. He believes that if a product can satisfy such customers, it's easier to sell to others. They argue that it's challenging to start with a simple product and then scale it up for more complex use cases, but it's feasible to start with a complex product and tailor it for simpler use cases.

Cooke further describes their process of creating a software product. Their strategy was to focus on core components, striving to make them as efficient and effective as possible. This involved investing years on foundational elements like string libraries and data marshalling. After establishing a robust foundation, they could then layer on additional features and enhancements. This approach allowed them to produce a mature and capable product eventually.

They also underscore the inevitability of users pushing software to its limits, regardless of its optimization. Thus, they argue for creating software that is as fast as possible right from the start. They refer to an interview with Steve Jobs, who argued that the best developers can create software that's substantially faster than others. Cooke's team continually seeks ways to refine and improve the efficiency of their platform.

Next, the discussion shifts to team composition and the necessary attributes for software engineers. Cooke emphasizes the importance of a strong work ethic and a passion for crafting good software. He explains how his ambition to become the best software developer from a young age has shaped his company's culture, fostering a virtuous cycle of hard work and dedication among his team.

The host then emphasizes the importance of engineers working on high-quality products, suggesting that problems and bugs can sap energy and demotivate a team. Cooke concurs, comparing the experience of working on high-quality software to working on an F1 race car, and how the pursuit of refinement and optimization is a dream for engineers.

The conversation then turns to the importance of having a team with diverse thought processes and skillsets. Cooke recounts how the introduction of different disciplines and perspectives in 2019 profoundly transformed his company.

The dialogue then transitions to the state of software solutions before the introduction of their high-quality software, touching upon the compartmentalized nature of systems in large corporations and the problems that arise from it. Cooke explains how their solution offers a more comprehensive and holistic overview that cuts across different risk categories.

Finally, in response to the host's question about open-source systems, Cooke expresses reservations about the use of open-source software in a corporate setting. However, he acknowledges the extensive overlap and redundancy among the many new systems being developed. Although he does not identify any specific groundbreaking technology, he believes the rapid proliferation of similar technologies might lead to considerable technical debt in the future.

Host Utsav wraps up the conversation by asking Cooke about his expectations and concerns for the future of technology and the industry. Cooke voices his concern about the continually growing number of different systems and technologies that companies are adopting, which makes integrating and orchestrating all these components a challenge. He advises companies to exercise caution when adopting multiple technologies simultaneously.

However, Cooke also expresses enthusiasm about the future of 3Forge, a platform he has devoted a decade of his life to developing. He expresses confidence in the unique approach and discipline employed in building the platform. Cooke is optimistic about the company's growth and marketing efforts and their focus on fostering a developer community. He believes that the platform will thrive as developers share their experiences, and the product gains momentum.

Utsav acknowledges the excitement and potential challenges that lie ahead, especially in managing community-driven systems. They conclude the conversation by inviting Cooke to return for another discussion in the future to review the progression and evolution of the topic. Both express their appreciation for the fruitful discussion before ending the podcast.

Apple Podcasts | Spotify | Google Podcasts

In this episode, we review the challenge of maintaining reasonable SaaS costs for tech companies. Usage-based pricing models of infrastructure costs lead to a gradual ramp-up of costs and always have sneakily come up as a priority in my career as an infrastructure/platform engineer. So I’m particularly interested in how engineering teams can better understand, track, and “shift left” infrastructure cost tracking and prevent regressions.

We specifically go over Kubernetes cost management, and why cost management needs to be attributable to the most specific teams in order to be self-governing in an organization.

Apple Podcasts | Spotify | Google Podcasts

We had an episode with the other founder of Komodor, Itiel, in 2021, and I thought it would be fun to revisit the topic.

Highlights (ChatGPT Generated)

[0:00] Introduction to the Software At Scale podcast and the guest speaker, Ben Ofiri, CEO and co-founder of Komodor.

- Discussion of why Ben decided to work on a Kubernetes platform and the potential impact of Kubernetes becoming the standard for managing microservices.

- Reasons why companies are interested in adopting Kubernetes, including the ability to scale quickly and cost-effectively, and the enterprise-ready features it offers.

- The different ways companies migrate to Kubernetes, either starting from a small team and gradually increasing usage, or a strategic decision from the top down.

- The flexibility of Kubernetes is its strength, but it also comes with complexity that can lead to increased time spent on alerts and managing incidents.

- The learning curve for developers to be able to efficiently troubleshoot and operate Kubernetes can be steep and is a concern for many organizations.

[8:17] Tools for Managing Kubernetes.

- The challenges that arise when trying to operate and manage Kubernetes.

- DevOps and SRE teams become the bottleneck due to their expertise in managing Kubernetes, leading to frustration for other teams.

- A report by the cloud native observability organization found that one out of five developers felt frustrated enough to want to quit their job due to friction between different teams.

- Ben's idea for Komodor was to take the knowledge and expertise of the DevOps and SRE teams and democratize it to the entire organization.

- The platform simplifies the operation, management, and troubleshooting aspects of Kubernetes for every engineer in the company, from junior developers to the head of engineering.

- One of the most frustrating issues for customers is identifying which teams should care about which issues in Kubernetes, which Komodor helps solve with automated checks and reports that indicate whether the problem is an infrastructure or application issue, among other things.

- Komodor provides suggestions for actions to take but leaves the decision-making and responsibility for taking the action to the users.

- The platform allows users to track how many times they take an action and how useful it is, allowing for optimization over time.

[8:17] Tools for Managing Kubernetes.

[12:03] The Challenge of Balancing Standardization and Flexibility.

- Kubernetes provides a lot of flexibility, but this can lead to fragmented infrastructure and inconsistent usage patterns.

- Komodor aims to strike a balance between standardization and flexibility, allowing for best practices and guidelines to be established while still allowing for customization and unique needs.

[16:14] Using Data to Improve Kubernetes Management.

- The platform tracks user actions and the effectiveness of those actions to make suggestions and fine-tune recommendations over time.

- The goal is to build a machine that knows what actions to take for almost all scenarios in Kubernetes, providing maximum benefit to customers.

[20:40] Why Kubernetes Doesn't Include All Management Functionality.

- Kubernetes is an open-source project with many different directions it can go in terms of adding functionality.

- Reliability, observability, and operational functionality are typically provided by vendors or cloud providers and not organically from the Kubernetes community.

- Different players in the ecosystem contribute different pieces to create a comprehensive experience for the end user.

[25:05] Keeping Up with Kubernetes Development and Adoption.

- How Komodor keeps up with Kubernetes development and adoption.

- The team is data-driven and closely tracks user feedback and needs, as well as new developments and changes in the ecosystem.

- The use and adoption of custom resources is a constantly evolving and rapidly changing area, requiring quick research and translation into product specs.

- The company hires deeply technical people, including those with backgrounds in DevOps and SRE, to ensure a deep understanding of the complex problem they are trying to solve.

[32:12] The Effects of the Economy on Komodor.

- The effects of the economy pivot on Komodor.

- Companiesmust be more cost-efficient, leading to increased interest in Kubernetes and tools like Komodor.

- The pandemic has also highlighted the need for remote work and cloud-based infrastructure, further fueling demand.

- Komodor has seen growth as a result of these factors and believes it is well-positioned for continued success.

[36:17] The Future of Kubernetes and Komodor.

- Kubernetes will continue to evolve and be adopted more widely by organizations of all sizes and industries.

- The team is excited about the potential of rule engines and other tools to improve management and automation within Kubernetes.

Apple Podcasts | Spotify | Google Podcasts

Subscribe now

Share Software at Scale

Highlights (GPT-3 generated)

[0:00:17] Vikas Agarwal's origin story.

[0:00:52] How Vikas learned to code.

[0:03:24] Vikas's first job out of college.

[0:04:30] Vikas' experience with the review business and community trust.

[0:06:10] Mission of the community trust team.

[0:07:14] How to start off with a problem.

[0:09:30] Different flavors of review abuse.

[0:10:15] The program for gift cards and fake reviews.

[0:12:10] Google search and FinTech.

[0:14:00] Fraud and ML models.

[0:15:51] Other things to consider when it comes to trust.

[0:17:42] Ryan Reynolds' funny review on his product.

[0:18:10] Reddit-like problems.

[0:21:03] Activism filters.

[0:23:03] Elon Musk's changing policy.

[0:23:59] False positives and appeals process.

[0:28:29] Stress levels and question mark emails from Jeff Bezos.

[0:30:32] Jeff Bezos' mathematical skills.

[0:31:45] Amazon's closed loop auditing process.

[0:32:24] Amazon's success and leadership principles.

[0:33:35] Operationalizing appeals at scale.

[0:35:45] Data science, metrics, and hackathons.

[0:37:14] Developer experience and iterating changes.

[0:37:52] Advice for tackling a problem of this scale.

[0:39:19] Striving for trust and external validation.

[0:40:01] Amazon's efforts to combat abuse.

[0:40:32] Conclusion.

Apple Podcasts | Spotify | Google Podcasts

Share Software at Scale

Subscribe now

Mike’s blog was instrumental towards my decision to pick a job in developer productivity/platform engineering. We talk about the Rainbow of Death - the idea of driving cultural change in large engineering organizations - one of the key challenges of platform engineering teams. And we deep dive into the value and common pushbacks against automated testing.

Highlights (GPT-3 generated)

[0:00 - 0:29] Welcome

[0:29 - 0:38] Explanation of Rainbow of Death

[0:38 - 0:52] Story of Testing Grouplet at Google

[0:52 - 5:52] Benefits of Writing Blogs and Engineering Culture Change

[5:52 - 6:48] Impact of Mike's Blog

[6:48 - 7:45] Automated Testing at Scale

[7:45 - 8:10] "I'm a Snowflake" Mentality

[8:10 - 8:59] Instigator Theory and Crossing the Chasm Model

[8:59 - 9:55] Discussion of Dependency Injection and Functional Decomposition

[9:55 - 16:19] Discussion of Testing and Testable Code

[16:19 - 24:30] Impact of Organizational and Cultural Change on Writing Tests

[24:30 - 26:04] Instigator Theory

[26:04 - 32:47] Strategies for Leaders to Foster and Support Testing

[32:47 - 38:50] Role of Leadership in Promoting Testing

[38:50 - 43:29] Philosophical Implications of Testing Practices

Apple Podcasts | Spotify | Google Podcasts

Subscribe now

Share Software at Scale

In this episode, we go back to the basics, and discuss the technical details of scalable build systems, like Pants, Bazel and Buck. A common challenge with these build systems is that it is extremely hard to migrate to them, and have them interoperate with open source tools that are built differently. Benjy’s team redesigned Pants with an initial hyper-focus on Python to fix these shortcomings, in an attempt to create a third generation of build tools - one that easily interoperates with differently built packages, but still fast and scalable.

Machine-generated Transcript

[0:00] Hey, welcome to another episode of the Software at Scale podcast. Joining me here today is Benji Weinberger, previously a software engineer at Google and Twitter, VP of Infrastructure at Foursquare, and now the founder and CEO of Toolchain.
Thank you for joining us.
Thanks for having me. It's great to be here. Yes. Right from the beginning, I saw that you worked at Google in 2002, which is forever ago, like 20 years ago at this point.
What was that experience like? What kind of change did you see as you worked there for a few years?

[0:37] As you can imagine, it was absolutely fascinating. And I should mention that while I was at Google from 2002, but that was not my first job.
I have been a software engineer for over 25 years. And so there were five years before that where I worked at a couple of companies.
One was, and I was living in Israel at the time. So my first job out of college was at Check Point, which was a big successful network security company. And then I worked for a small startup.
And then I moved to California and started working at Google. And so I had the experience that I think many people had in those days, and many people still do, of the work you're doing is fascinating, but the tools you're given to do it with as a software engineer are not great.
This, I'd had five years of experience of sort of struggling with builds being slow, builds being flaky with everything requiring a lot of effort. There was almost a hazing,
ritual quality to it. Like, this is what makes you a great software engineer is struggling through the mud and through the quicksand with this like awful substandard tooling. And,
We are not users, we are not people for whom products are meant, right?
We make products for other people. Then I got to Google.

[2:03] And Google, when I joined, it was actually struggling with a very massive, very slow make file that took forever to parse, let alone run.
But the difference was that I had not seen anywhere else was that Google paid a lot of attention to this problem and Google devoted a lot of resources to solving it.
And Google was the first place I'd worked and I still I think in many ways the gold standard of developers are first class participants in the business and deserve the best products and the best tools and we will if there's nothing out there for them to use, we will build it in house and we will put a lot of energy into that.
And so it was for me, specifically as an engineer.

[2:53] A big part of watching that growth from in the sort of early to late 2000s was. The growth of engineering process and best practices and the tools to enforce it and the thing i personally am passionate about is building ci but i'm also talking about.
Code review tools and all the tooling around source code management and revision control and just everything to do with engineering process.
It really was an object lesson and so very, very fascinating and really inspired a big chunk of the rest of my career.
I've heard all sorts of things like Python scripts that had to generate make files and finally they move the Python to your first version of Blaze. So it's like, it's a fascinating history.

[3:48] Maybe can you tell us one example of something that was like paradigm changing that you saw, like something that created like a magnitude, like order of magnitude difference,
in your experience there and maybe your first aha moment on this is how good like developer tools can be?

[4:09] Sure. I think I had been used to using make basically up till that point. And Google again was, as you mentioned, using make and really squeezing everything it was possible to squeeze out of that lemon and then some.

[4:25] But when the very early versions of what became blaze which was that big internal build system which inspired basil which is the open source variant of that today. Hey one thing that really struck me was the integration with the revision controls system which was and i think still is performance.
I imagine many listeners are very familiar with Git. Perforce is very different. I can only partly remember all of the intricacies of it, because it's been so long since I've used it.
But one interesting aspect of it was you could do partial checkouts. It really was designed for giant code bases.
There was this concept of partial checkouts where you could check out just the bits of the code that you needed. But of course, then the question is, how do you know what those bits are?
But of course the build system knows because the build system knows about dependencies. And so there was this integration, this back and forth between the, um.

[5:32] Perforce client and the build system that was very creative and very effective.
And allowed you to only have locally on your machine, the code that you actually needed to work on the piece of the codebase you're working on,
basically the files you cared about and all of their transitive dependencies. And that to me was a very creative solution to a problem that involved some lateral thinking about how,
seemingly completely unrelated parts of the tool chain could interact. And that's kind of been that made me realize, oh, there's a lot of creative thought at work here and I love it.

[6:17] Yeah, no, I think that makes sense. Like I interned there way back in 2016. And I was just fascinated by, I remember by mistake, I ran like a grep across the code base and it just took forever. And that's when I realized, you know, none of this stuff is local.
First of all, like half the source code is not even checked out to my machine.
And my poor grep command is trying to check that out. But also how seamlessly it would work most of the times behind the scenes.
Did you have any experience or did you start working on developer tools then? Or is that just what inspired you towards thinking about developer tools?
I did not work on the developer tools at Google. worked on ads and search and sort of Google products, but I was a big user of the developer tools.
Exception which was that I made some contributions to the.

[7:21] Protocol buffer compiler which i think many people may be familiar with and that is. You know if i very deep part of the toolchain that is very integrated into everything there and so that gave me.
Some experience with what it's like to hack on a tool that's everyone in every engineer is using and it's the sort of very deep part of their workflow.
But it wasn't until after google when i went to twitter.

[7:56] I noticed that the in my time of google my is there the rest of the industry had not. What's up and suddenly i was sort of stressed ten years into the past and was back to using very slow very clunky flaky.
Tools that were not designed for the tasks we were trying to use them for. And so that made me realize, wait a minute, I spent eight years using these great tools.
They don't exist outside of these giant companies. I mean, I sort of assumed that maybe, you know, Microsoft and Amazon and some other giants probably have similar internal tools, but there's something out there for everyone else.
And so that's when I started hacking on that problem more directly was at Twitter together with John, who is now my co-founder at Toolchain, who was actually ahead of me and ahead of
the game at Twitter and already begun working on some solutions and I joined him in that.
Could you maybe describe some of the problems you ran into? Like were the bills just taking forever or was there something else?

[9:09] So there were...

[9:13] A big part of the problem was that Twitter at the time, the codebase I was interested in and that John was interested in was using Scala. Scala is a fascinating, very rich language.

[9:30] Its compiler is very slow. And we were in a situation where, you know, you'd make some small change to a file and then builds would take just,
10 minutes, 20 minutes, 40 minutes. The iteration time on your desktop was incredibly slow.
And then CI times, where there was CI in place, were also incredibly slow because of this huge amount of repetitive or near repetitive work. And this is because the build tools,
etc. were pretty naive about understanding what work actually needs to be done given a set of changes.
There's been a ton of work specifically on SBT since then.

[10:22] It has incremental compilation and things like that, but nonetheless, that still doesn't really scale well to large corporate codebases that are what people often refer to as monorepos.
If you don't want to fragment your codebase with all of the immense problems that that brings, you end up needing tooling that can handle that situation.
Some of the biggest challenges are, how do I do less than recompile the entire codebase every time. How can tooling help me be smart about what is the correct minimal amount of work to do.

[11:05] To make compiling and testing as fast as it can be?

[11:12] And I should mention that I dabbled in this problem at Twitter with John. It was when I went to Foursquare that I really got into it because Foursquare similarly had this big Scala codebase with a very similar problem of incredibly slow builds.

[11:29] The interim solution there was to just upgrade everybody's laptops with more RAM and try and brute force the problem. It was very obvious to everyone there, tons of,
force-creation pattern still has lots of very, very smart engineers.
And it was very obvious to them that this was not a permanent solution and we were casting around for...

[11:54] You know what can be smart about scala builds and i remember this thing that i had hacked on twitter and. I reached out to twitter and ask them to open source it so we could use it and collaborate on it wasn't obviously some secret sauce and that is how the very first version of the pants open source build system came to be.
I was very much designed around scarlet did eventually.
Support other languages. And we hacked on it a lot at Foursquare to get it to...

[12:32] To get the codebase into a state where we could build it sensibly. So the one big challenge is build speed, build performance.
The other big one is managing dependencies, keeping your codebase sane as it scales.
Everything to do with How can I audit internal dependencies?
How do I make sure that it is very, very easy to accidentally create all sorts of dependency tangles and cycles and create a code base whose dependency structure is unintelligible, really,
hard to work with and actually impacts performance negatively, right?
If you have a big tangle of dependencies, you're more likely to invalidate a large chunk of your code base with a small change.
And so tooling that allows you to reason about the dependencies in your code base and.

[13:24] Make it more tractable was the other big problem that we were trying to solve. Mm-hmm. No, I think that makes sense.
I'm guessing you already have a good understanding of other build systems like Bazel and Buck.
Maybe could you walk us through what are the difference for PANs, Veevan? What is the major design differences? And even maybe before that, like, how was Pants designed?
And is it something similar to like creating a dependency graph? You need to explicitly include your dependencies.
Is there something else that's going on?

[14:07] Maybe just a primer. Yeah. Absolutely. So I should mention, I was careful to mention, you mentioned Pants V1.
The version of Pants that we use today and base our entire technology stack around is what we very unimaginatively call Pants V2, which we launched two years ago almost to the day.
That is radically different from Pants V1, from Buck, from Bazel. It is quite a departure in ways that we can talk about later.
One thing that I would say Panacea V1 and Buck and Bazel have in common is that they were designed around the use cases of a single organization. is a.

[14:56] Open source variant or inspired by blaze its design was very much inspired by. Here's how google does engineering and a buck similarly for facebook and pansy one frankly very similar for.

[15:11] Twitter and we sort of because Foursquare also contributed a lot to it, we sort of nudged it in that direction quite a bit. But it's still very much if you did engineering in this one company's specific image, then this might be a good tool for you.
But you had to be very much in that lane.
But what these systems all look like is, and the way they are different from much earlier systems is.

[15:46] They're designed to work in large scalable code bases that have many moving parts and share a lot of code and that builds a lot of different deployables, different, say, binaries or Docker
Docker images or AWS lambdas or cloud functions or whatever it is you're deploying, Python distributions, Java files, whatever it is you're building, typically you have many of them in this code base.
Could be lots of microservices, could be just lots of different things that you're deploying.
And they live in the same repo because you want that unity. You want to be able to share code easily. you don't want to introduce dependency hell problems in your own code. It's bad enough that we have dependency hell problems third-party code.

[16:34] And so these systems are all if you squint at them from thirty thousand feet today all very similar in that they make that the problem of. Managing and building and testing and packaging in a code base like that much more tractable and the way they do this is by applying information about the dependencies in your code base.
So the important ingredient there is that these systems understand the find the relatively fine grained dependencies in your code base.
And they can use that information to reason about work that needs to happen. So a naive build system, you'd say, run all the tests in the repo or in this part of the repo.
So a naive system would literally just do that, and first they would compile all the code.

[17:23] But a scalable build system like these would say, well, you've asked me to run these tests, but some of them have already been cached and these others, okay, haven't.
So I need to look at these ones I actually need to run. So let me see what needs to be done before I can run them.
Oh, so these source files need to be compiled, but some of those already in cache and then these other ones I need to compile. But I can apply concurrency because there are multiple cores on this machine.
So I can know through dependency analysis which compile jobs can run concurrently and which cannot. And then when it actually comes time to run the tests, again, I can apply that sort of concurrency logic.

[18:03] And so these systems, what they have in common is that they use dependency information to make your building testing packaging more tractable in a large code base.
They allow you to not have to do the thing that unfortunately many organizations find themselves doing, which is fragmenting the code base into lots of different bits and
saying, well, every little team or sub team works in its own code base and they consume each other's code through, um, so it was third party dependencies in which case you are introducing a dependency versioning hell problem.
Yeah. And I think that's also what I've seen that makes the migration to a tool like this hard. Cause if you have an existing code base that doesn't lay out dependencies explicitly.

[18:56] That migration becomes challenging. If you already have an import cycle, for example.

[19:01] Bazel is not going to work with you. You need to clean that up or you need to create one large target where the benefits of using a tool like Bazel just goes away. And I think that's a key,
bit, which is so fascinating because it's the same thing over several years. And I'm hoping that,
it sounds like newer tools like Go, at least, they force you to not have circular dependencies and they force you to keep your code base clean so that it's easy to migrate to like a scalable build system.

[19:33] Yes exactly so it's funny that is the exact observation that let us to pans to see to so they said pans to be one like base like buck was very much inspired by and developed for the needs of a single company and other companies were using it a little bit.
But it also suffered from any of the problems you just mentioned with pans to for the first time by this time i left for square and i started to chain with the exact mission of every company every team of any size should have this kind of tooling should have this ability this revolutionary ability to make the code base is fast and tractable at any scale.
And that made me realize.
We have to design for that we have to design for not for. What a single company's code base looks like but we have to design.
To support thousands of code bases of all sorts of different challenges and sizes and shapes and languages and frameworks so.
We actually had to sit down and figure out what does it mean to make a tool.
Like this assistant like this adoptable over and over again thousands of times you mentioned.

[20:48] Correctly, that it is very hard to adopt one of those earlier tools because you have to first make your codebase conform to whatever it is that tool expects, and then you have to write huge amounts of manual metadata to describe all of the dependencies in your,
the structure and dependencies of your codebase in these so-called build files.
If anyone ever sees this written down, it's usually build with all capital letters, like it's yelling at you and that those files typically are huge and contain a huge amount of information your.

[21:27] I'm describing your code base to the tool with pans be to eat very different approaches first of all we said this needs to handle code bases as they are so if you have circular dependencies it should handle them if you have. I'm going to handle them gracefully and automatically and if you have multiple conflicting external dependencies in different parts of your code base this is pretty common right like you need this version of whatever.
Hadoop or NumPy or whatever it is in this part of the code base, and you have a different conflicting version in this other part of the code base, it should be able to handle that.
If you have all sorts of dependency tangles and criss-crossing and all sorts of things that are unpleasant, and better not to have, but you have them, the tool should handle that.
It should help you remove them if you want to, but it should not let those get in the way of adopting it.
It needs to handle real-world code bases. The second thing is it should not require you to write all this crazy amount of metadata.
And so with Panzer V2, we leaned in very hard on dependency inference, which means you don't write these crazy build files.
You write like very tiny ones that just sort of say, you know, here is some code in this language for the build tool to pay attention to.

[22:44] But you don't have to edit the added dependencies to them and edit them every time you change dependencies.
Instead, the system infers dependencies by static analysis. So it looks at your, and it does this at runtime.
So you, you know, almost all your dependencies, 99% of the time, the dependencies are obvious from import statements.

[23:05] And there are occasional and you can obviously customize this because sometimes there are runtime dependencies that have to be inferred from like a string. So from a json file or whatever is so there are various ways to customize this and of course you can always override it manually.
If you have to be generally speaking ninety.
Seven percent of the boilerplate that used to going to build files in those old systems including pans v1 no. You know not claiming we did not make the same choice but we goes away with pans v2 for exactly the reason that you mentioned these tools,
because they were designed to be adopted once by a captive audience that has no choice in the matter.
And it was designed for how that code base that adopting code base already is. is these tools are very hard to adopt.
They are massive, sometimes multi-year projects outside of that organization. And we wanted to build something that you could adopt in days to weeks and would be very easy,
to customize to your code base and would not require these massive wholesale changes or huge amounts of metadata.
And I think we've achieved that. Yeah, I've always wondered like, why couldn't constructing the build file be a part of the build. In many ways, I know it's expensive to do that every time. So just like.

[24:28] Parts of the build that are expensive, you cache it and then you redo it when things change.
And it sounds like you've done exactly that with BANs V2.

[24:37] We have done exactly that. The results are cached on a profile basis. So the very first time you run something, then dependency inference can take some time. And we are looking at ways to to speed that up.
I mean, like no software system has ever done, right? Like it's extremely rare to declare something finished. So we are obviously always looking at ways to speed things up.
But yeah, we have done exactly what you mentioned. We don't, I should mention, we don't generate the dependencies into build for, we don't edit build files and then you check them in.
We do that a little bit. So I mentioned you do still with PANSTL V2, you need these little tiny build files that just say, here is some code.
They typically can literally be one line sometimes, almost like a marker file just to say, here is some code for you to pay attention to.
We're even working on getting rid of those.
We do have a little script that generates those one time just to help you onboard.
But...

[25:41] The dependencies really are just generated a runtime as on demand as needed and used a runtime so we don't have this problem of. Trying to automatically add or edit a otherwise human authored file that is then checked in like this generating and checking in files is.
Problematic in many ways, especially when those files also have to take human written edits.
So we just do away with all of that and the dependency inference is at runtime, on demand, as needed, sort of lazily done, and the information is cached. So both cached in memory in the surpassed V2 has this daemon that runs and caches a huge amount of state in memory.
And the results of running dependency inference are also cached on disk. So they survive a daemon restart, etc.
I think that makes sense to me. My next question is going to be around why would I want to use panthv2 for a smaller code base, right? Like, usually with the smaller codebase, I'm not running into a ton of problems around the build.

[26:55] I guess, do you notice these inflection points that people run into? It's like, okay, my current build setup is not enough. What's the smallest codebase that you've seen that you think could benefit? Or is it like any codebase in the world? And I should start with,
a better build system rather than just Python setup.py or whatever.
I think the dividing line is, will this code base ever be used for more than one thing?

[27:24] So if you have a, let's take the Python example, if literally all this code base will ever do is build this one distribution and a top level setup pie is all I need. And that is, you know, this,
sometimes you see this with open source projects and the code base is going to remain relatively small, say it's only ever going to be a few thousand lines and the tests, even if I run
the tests from scratch every single time, it takes under five minutes, then you're probably fine.
But I think two things I would look at are, am I going to be building multiple things in this code base in the future, or certainly if I'm doing it now.
And that is much more common with corporate code bases. You have to ask yourself, okay, my team is growing, more and more people are cooperating on this code base.
I want to be able to deploy multiple microservices. I want to be able to deploy multiple cloud functions.
I want to be able to deploy multiple distributions or third-party artifacts.
I want to be able to.

[28:41] You know, multiple sort of data science jobs, whatever it is that you're building. If you want, if you ever think you might have more than one, now's the time to think about,
okay, how do I structure the code base and what tooling allows me to do this effectively?
And then the other thing to look at is build times. If you're using compiled languages, then obviously compilation, in all cases testing, if you start to see like, I can already see that that tests are taking five minutes, 10 minutes, 15 minutes, 20 minutes.
Surely, I want some technology that allows me to speed that up through caching, through concurrency, through fine-grained invalidation, namely, don't even attempt to do work that isn't necessary for the result that was asked for.
Then it's probably time to start thinking about tools like this, because the earlier you adopt it, the easier it is to adopt.
So if you wait until you've got a tangle of multiple setup pies in the repo and it's unclear how you manage them and how you keep their dependencies synchronized,
so there aren't version conflicts across these different projects, specifically with Python,
this is an interesting problem.
I would say with other languages, there is more because of the compilation step in jvm languages or go you.

[30:10] Encounter the need for a build system much much earlier a bill system of some kind and then you will ask yourself what kind with python because you can get a bite for a while just running. What are the play gate and pie test and i directly and all everything is all together in a single virtual and.
But the Python tooling, as mighty as it is, mostly is not designed for larger code bases with multiple, that deploy multiple things and have multiple different sets of.

[30:52] Internal and external dependencies the tooling generally implicitly assume sort of one top level set up i want top level. Hi project dot com all you know how are you configuring things and so especially using python let's say for jango flask apps or for data science
and your code base is growing and you've hired a bunch of data scientists and there's more and more code going in there. With Python, you need to start thinking about what tooling allows me to scale this code base. No, I think I mostly resonate with that. The first question that comes to my mind is,
let's talk specifically about the deployment problem. If you're deployed to multiple AWS lambdas or cloud functions or whatever, the first thought that would come to my mind is
is I can use separate Docker images that can let me easily produce this container image that I can ship independently.
Would you say that's not enough? I totally get that for the build time problem.
A Docker image is not going to solve anything. But how about the deployment step?

[32:02] So again, with deployments, I think there are two ways where a tool like this can really speed things up.
One is only build the things that actually need to be redeployed. And because the tool understands dependencies and can do change analysis, it can figure that out.
So one of the things that HansB2 does is it integrates with Git.
And so it natively understands how to figure out Git diffs. So you can say something like, show me all the whatever, lambdas, let's say, that are affected by changes between these two branches.

[32:46] And it knows and it understands it can say, well, these files changed and you know, we, I understand the transitive dependencies of those files.
So I can see what actually needs to be deployed. And, you know, many cases, many things will not need to be redeployed because they haven't changed.
The other thing is there's a lot of performance improvements and process improvements around building those images. So, for example, we have for Python specifically, we have an executable format called PEX,
which stands for Python executable, which is a single file that embeds all of your Python code that is needed for your deployable and all of its external requirements, transitive external requirements, all bundled up into this single sort of self-executing file.
This allows you to do things like if you have to deploy 50 of these, you can basically have a single docker image.

[33:52] The different then on top of that you add one layer for each of these fifty and the only difference in that layer is the presence of this pecs file. Where is without all this typically what you would do is.
You have fifty docker images each one of which contains a in each one of which you have to build a virtual and which means running.

[34:15] Pip as part of building the image, and that gets slow and repetitive, and you have to do it 50 times.
We have a lot of ways to speed up. Even if you are deploying 50 different Docker images, we have ways of speeding that up quite dramatically.
Because again, of things like dependency analysis, the PECS format, and the ability to build incrementally.
Yeah, I think I remember that at Dropbox, we came up with our own, like, par format to basically bundle up a Python binary with, I think par stood for Python archive. I'm not
entirely sure. But it did something remarkably similar to solve exactly this problem. It just takes so long, especially if you have a large Python code base. I think that makes sense to me. The other thing that one might ask is, with Python, you don't really have,
too long of a build time, is what you would guess, because there's nothing to build. Maybe myPy takes some time to do some static analysis, and, of course, your tests can take forever,
and you don't want to rerun them. But there isn't that much of a build time that you have to think about. Would you say that you agree with this, or there's some issues that end,
up happening on real-world code basis.

[35:37] Well that's a good question the word builds means different things to different people and we recently taken to using the time see i more. Because i think that is clear to people what that means but when i say build or see i mean it in the law in in the extended sense everything you do to go from.
Human written source code to a verified.
Test did. deployable artifact and so it's true that for python there's no compilation step although arguably. Running my pie is really important and now that i'm really in the habit of using.
My pie i will probably never not use it on python code ever again but so that are.

[36:28] Sort of build-ish steps for Python such as type checking, such as running code generators like Thrift or Protobuf.
And obviously a big, big one is running, resolving third-party dependencies such as running PIP or poetry or whatever it is you're using. So those are all build steps.
But with Python, really the big, big, big thing is testing and packaging and primarily testing.
And so with Python, you have to be even more rigorous about unit testing than you do with other languages because you don't have a compiler that is catching whole classes of bugs.
So and again, MyPy and type checking does really help with that. And so when I build to me includes, build in the large sense includes running tests,
includes packaging and includes everything, all the quality control that you run typically in CI or on your desktop in order to go say, well, I've made some edits and here's the proof that these edits are good and I can merge them or deploy them.

[37:35] I think that makes sense to me. And like, I certainly saw it with the limited number of testing, the limited amount of type checking you can do with Python, like MyPy is definitely
improving on this. You just need to unit test a lot to get the same amount of confidence in your own code and then unit tests are not cheap. The biggest question that comes to
my mind is that is BANs V2 focused on Python? Because I have a TypeScript code base at my workplace and I would love to replace the TypeScript compiler with something that was slightly smarter and could tell me, you know what, you don't need to run every unit test every change.

[38:16] Great question so when we launched a pass me to which was two years ago. The.
We focused on python and that was the initial language we launched with because you had to start somewhere and in the city ten years in between the very scarlet centric work we were doing on pansy one. And the launch of hands be to something really major happened in the industry which was the python skyrocketed in popularity sky python went from.
Mostly the little scripting language around the edges of your quote unquote real code, I can use python like fancy bash to people are building massive multi billion dollar businesses entirely on python code bases and there are a few things that drove this one was.
I would say the biggest one probably was the python became the. Language of choice for data science and we have strong support for those use cases. There was another was the,
Django and Flask became very popular for writing web apps more and more people were used there were more in Intricate DevOps use cases and Python is very popular for DevOps for various good reasons. So.

[39:28] Python became super popular. So that was the first thing we supported in pants v2, but we've since added support for or Go, Java, Scala, Kotlin, Shell.
Definitely what we don't have yet is JavaScript TypeScript. We are looking at that very closely right now, because that is the very obvious next thing we want to add.
Actually, if any listeners have strong opinions about what that should look like, we would love to hear from them or from you on our Slack channels or on our GitHub discussions where we are having some lively discussions about exactly this because the JavaScript.

[40:09] And TypeScript ecosystem is already very rich with tools and we want to provide only value add, right? We don't want to say, you have to, oh, you know, here's another paradigm you have to adopt.
And here's, you know, you have to replace, you've just done replacing whatever this with this, you know, NPM with yarn. And now you have to do this thing. And now we're, we don't want to be
another flavor of the month. We only want to do the work that uses those tools, leverages the existing ecosystem but adds value. This is what we do with Python and this is one of the reasons why our Python support is very, very strong, much stronger than any other comparable tool out there is.

[40:49] A lot of leaning in on the existing Python tool ecosystem but orchestrating them in a way that brings rigor and speed to your builds.
And I haven't used the word we a lot. And I just kind of want to clarify who we is here.
So there is tool chain, the company, and we're working on, um, uh, SAS and commercial, um, solutions around pants, which we can talk about in a bit.
But there is a very robust open source community around pants that is not. tightly held by Toolchain, the company in a way that some other companies open source projects are.
So we have a lot of contributors and maintainers on Pants V2 who are not working at Toolchain, but are using Pants in their own companies and their own organizations.
And so we have a very wide range of use cases and opinions that are brought to bear. And this is very important because, as I mentioned earlier,
we are not trying to design a system for one use case, for one company or a team's use case.
We are trying, you know, we are working on a system we want.

[42:05] Adoption for over and over and over again at a wide variety of companies. And so it's very important for us to have the contributions and the input from a wide variety of teams and companies
and people. And it's very fortunate that we now do. I mean, on that note, the thing that comes to my mind is another benefit of your scalable build system like Vance or Bazel or Buck is that you
You don't have to learn various different commands when you are spelunking through the code base, whether it's like a Go code base or like a Java code base or TypeScript code base.
You just have to run pants build X, Y, Z, and it can construct the appropriate artifacts for you. At least that was my experience with Bazel.
Is that something that you are interested in or is that something that pants V2 does kind of act as this meta layer for various other build systems or is it much more specific and knowledgeable about languages itself?

[43:09] It's, I think your intuition is correct. The idea is we want you to be able to do something like pants test or pants test, you know, give it a path to a directory and it understands what that means.
Oh, this directory contains Python code. Therefore, I should run PyTest in this way. And oh, Oh, it also contains some JavaScript code, so I should run the JavaScript test in this way.
And it basically provides a conceptual layer above all the individual tools that gives you this uniformity across frameworks, across languages.
One way to think about this is.

[43:52] The tools are all very imperative. say you have to run them with a whole set of flags and inputs and you have to know how to use each one separately. So it's like having just the blades of a Swiss Army knife with
no actual Swiss Army knife. A tool like Pants will say, okay, we will encapsulate all of that complexity into a much more simple command line interface. So you can do, like I said,
test or pants lint or pants format and it understands, oh, you asked me to format your code. I see that you have the black and I sort configured as formatters. So I will run them. And I happen to know that formatting, because formatting can change the source files,
I have to run them sequentially. But when you ask for lint, it's not changing the source files. So I know that I can run them multiple lint as concurrently, that sort of logic. And And different tools have different ways of being configured or of telling you what they want to do, but we...

[44:58] Can't be to sort of encapsulate all that away from you and so you get this uniform simple command line interface that abstract away a lot of the specifics of these tools and let you run simple commands and the reason this is important is that. This extra layer of indirection is partly what allows pants to apply things like cashing.
And invalidation and concurrency because what you're saying is.

[45:25] Hey, the way to think about it is not, I am telling pants to run tests. It is I am telling pants that I want the results of tests, which is a subtle difference.
But pants then has the ability to say, well, I don't actually need to run pi test on all these tests because I have results from some of them already cached. So I will return them from cache.
So that layer of indirection not only simplifies the UI, but provides the point where you can apply things like caching and concurrency.
Yeah, I think every programmer wants to work with declarative tools. I think SQL is one of those things where you don't have to know how the database works. If SQL were somewhat easier, that dream would be fulfilled. But I think we're all getting there.
I guess the next question that I have is, what benefit do I get by using the tool chain, like SaaS product versus Pants V2?
When I think about build systems, I think about local development, I think about CI.

[46:29] Why would I want to use the SaaS product? That's a great question.
So Pants does a huge amount of heavy lifting, but in the end it is restricted to the resources is on the machine on which it's running. So when I talk about cash, I'm talking about the local cash on that machine. When I talk about concurrency, I'm talking about using,
the cores on your machine. So maybe your CI machine has four cores and your laptop has eight cores. So that's the amount of concurrency you get, which is not nothing at all, which is great.

[47:04] Thanks for watching!

[47:04] You know as i mentioned i worked at google for many years and then other companies where distributed systems were saying like i come from a distributed systems background and it really. Here is a problem.
All of a piece of work taking a long time because of. Single machine resource constraints the obvious answer here is distributed distributed the work user distributed system and so that's what tool chain offers essentially.

[47:30] You configure Pants to point to the toolchain system, which is currently SAS.
And we will have some news soon about some on-prem solutions.
And now the cache that I mentioned is not just did this test run with these exact inputs before on my machine by me me while I was iterating, but has anyone in my organization or any CI run this test before,
with these exact inputs?
So imagine a very common situation where you come in in the morning and you pull all the changes that have happened since you last pulled.
Those changes presumably passed CI, right? And the CI populated the cache.
So now when I run tests, I can get cache hits from the CI machine.

[48:29] Now pretty much, yeah. And then with concurrency, again, so let's say, you know, post cache, there are still 200 tests that need to be run.
I could run them eight at a time on my machine or the CI machine could run them, you know, say, four at a time on four cores, or I could run 50 or 100 at a time on a cluster of machines.
That's where, again, as your code base gets bigger and bigger, that's where some massive, massive speedups come in.
The other aspects of the... I should mention that the remote execution that I just mentioned is something we're about to launch. It is not available today. The remote caching is.
The other aspects are things like observability. So when you run builds on your laptop or CI, they're ephemeral.
Like the output gets lost in the scroll back.
And it's just a wall of text that gets lost with them.

[49:39] Toolchain all of that information is captured and stored in structured form so you have. Hey the ability to see past bills and see build behavior over time and drill death search builds and drill down into individual builds and see well.
How often does this test fail and you know when did this get slow all this kind of information and so you get.
This more enterprise level.
Observability into a very core piece of developer productivity, which is the iteration time.
The time it takes to run tests and build deployables and parcel the quality control checks so that you can merge and deploy code directly relates to time to release.
It directly relates to some of the core metrics of developer productivity. How long is it going to take to get this thing out the door?
And so having the ability to both speed that up dramatically through distributing the work and having observability into what work is going on, that is what toolchain provides,
on top of the already, if I may say, pretty robust open source offering.

[51:01] So yeah, that's kind of it.

[51:07] Pants on its own gives you a lot of advantages, but it runs standalone. Plugging it into a larger distributed system really unleashes the full power of Pants as a client to that system.

[51:21] No, I think what I'm seeing is this interesting convergence. There's several companies trying to do this for Bazel, like BuildBuddy and Edgeflow. So, and it really sounds like the build system of the future, like 10 years from now.

[51:36] No one will really be developing on their local machines anymore. Like there's GitHub code spaces on one side. It's like you're doing all your development remotely.

[51:46] I've always found it somewhat odd that development that happens locally and whatever scripts you need to run to provision your CI machine to run the same set of tests
are so different sometimes that you can never tell why something's passing locally and failing in in CI or vice versa. And there really should just be this one execution layer that can say, you know what, I'm going to build at a certain commit or run at a certain commit.
And that's shared between the local user and the CI user. And your CI script is something as simple as pants build slash slash dot dot dot. And it builds the whole code base for,
you. So yeah, I certainly feel like the industry is moving in that direction. I'm curious whether You think that's the same.
Do you have an even stronger vision of how folks will be developing 10 years from now? What do you think it's going to look like?
Oh, no, I think you're absolutely right. I think if anything, you're underselling it. I think this is how all development should be and will be in the future for multiple reasons.
One is performance.

[52:51] Two is the problem of different platforms. And so today, big thorny problem is I want to, you know, I want to,
I'm developing on my Mac book, but the production, so I'm running, when I run tests locally and when I run anything locally, it's running on my Mac book, but that's not our deployable, right?
Typically your deploy platform is some flavor of Linux. So...

[53:17] With the distributed system approach you can run the work in. Containers that exactly match your production environments you don't even have to care about can this run.
On will my test pass on mac os do i need ci the runs on mac os just to make sure the developers can. past test on Mac OS and that is somehow correlated with success on the production environment.
You can cut away a whole suite of those problems, which today, frankly, I had mentioned earlier, you can get cache hits on your desktop from remote, from CI populating the cache.
That is hampered by differences in platform.
Is hampered by other differences in local setup that we are working to mitigate. But imagine a world in which build logic is not actually running on your MacBook, or if it is,
it's running in a container that exactly matches the container that you're targeting.
It cuts away a whole suite of problems around platform differences and allows you to focus because on just a platform you're actually going to deploy too.

[54:35] And the...

[54:42] And just the speed and the performance of being able to work and deploy and the visibility that it gives you into the productivity and the operational work of your development team,
I really think this absolutely is the future.
There is something very strange about how in the last 15 years or so, so many business functions have had the distributed systems treatment applied to them.
Function is now that there are these massive valuable companies providing systems that support sales and systems that support marketing and systems that support HR and systems support
operations and systems support product management and systems that support every business function,
and there need to be more of these that support engineering as a business function.

[55:48] And so i absolutely think the idea that i need a really powerful laptop so that my running tests can take thirty minutes instead of forty minutes when in reality it should take three minutes is. That's not the future right the future is to as it has been for so many other systems to the web the laptop is that i can take anywhere is.
Particularly in these work from home times, is a work from anywhere times, is just a portal into the system that is doing the actual work.

[56:27] Yeah. And there's all these improvements across the stack, right? When I see companies like Versel, they're like, what if you use Next.js, we provide the best developer platform for
that and we want to provide caching. Then there's like the lower level systems with build systems, of course, like bands and Bazel and all that. And at each layer, we're kind
of trying to abstract the problem out. So to me, it still feels like there is a lot of innovation to be done. And I'm also going to be really curious to know, you know, there's
going to be like a few winners of this space, or if it's going to be pretty broken up. And like everyone's using different tools. It's going to be fascinating, like either way.
Yeah, that's really hard to know. I think one thing you mentioned that I think is really important is you said your CI should be as simple as just pants build colon, colon, or whatever.
That's our syntax would be sort of pants test lint or whatever.
I think that's really important. So.

[57:30] Today one of the big problems with see i. Which is still growing right now home market is still growing is more more teams realize the value and importance of automated.
Very aggressive automated quality control. But configuring CI is really, really complicated. Every CI provider have their own configuration language,
and you have to reason about caching, and you have to manually construct cache keys to the extent,
that caching is even possible or useful.
There's just a lot of figuring out how to configure and set up CI, And even then it's just doing the naive thing.

[58:18] So there are a couple of interesting companies, Dagger and Earthly, or interesting technologies around simplifying that, but again, you still have to manually,
so they are providing a, I would say, better config and more uniform config language that allows you to, for example, run build steps in containers.
And that's not nothing at all.

[58:43] Um, but you still manually creating a lot of, uh, configuration to run these very coarse grained large scale, long running build steps, you know, I think
the future is something like my entire CI config post cloning the repo is basically pants build colon, colon, because the system does the configuration for you.

[59:09] It figures out what that means in a very fast, very fine grained way and does not require you to manually decide on workflows and steps and jobs and how they all fit together.
And if I want to speed this thing up, then I have to manually partition the work somehow and write extra config to implement that partitioning.
That is the future, I think, is rather than there's the CI layer, say, which would be the CI providers proprietary config or theodagger and then underneath that there is the build
tool, which would be Bazel or Pants V2 or whatever it is you're using, could still be we make for many companies today or Maven or Gradle or whatever, I really think the future is the integration of those two layers.
In the same way that I referenced much, much earlier in our conversation, how one thing that stood out to me at Google was that they had the insight to integrate the version control layer and the build tool to provide really effective functionality there.
I think the build tool being the thing that knows about your dependencies.

[1:00:29] Can take over many of the jobs of the c i configuration layer in a really smart really fast. Where is the future where essentially more and more of how do i set up and configure and run c i is delegated to the thing that knows about your dependencies and knows about cashing and knows about concurrency and is able,
to make smarter decisions than you can in a YAML config file.

[1:01:02] Yeah, I'm excited for the time that me as a platform engineer has to spend less than 5% of my time thinking about CI and CD and I can focus on other things like improving our data models rather than mucking with the YAML and Terraform configs. Well, yeah.
Yeah. Yeah. Today you have to, we're still a little bit in that state because we are engineers and because we, the tools that we use are themselves made out of software. There's,
a strong impulse to tinker and there's a strong impulse sake. Well, I want to solve this problem myself or I want to hack on it or I should be able to hack on it. And that's, you should be able to hack on it for sure. But we do deserve more tooling that requires less hacking,
and more things and paradigms that have tested and have survived a lot of tire kicking.

[1:02:00] Will we always need to hack on them a little bit? Yes, absolutely, because of the nature of what we do. I think there's a lot of interesting things still to happen in this space.
Yeah, I think we should end on that happy note as we go back to our day jobs mucking with YAML. Well, thanks so much for being a guest. I think this was a great conversation and I hope to have you again for the show sometime.
Would love that. Thanks for having me. It was fascinating.

Apple Podcasts | Spotify | Google Podcasts

Subscribe now

Share Software at Scale

In this episode, we discuss Puneet’s fascinating background early at AWS as a GM and his early experience at Oracle Cloud. We initially discuss why AWS shipped S3 as its first product before any other services. After, we go over the cultural differences between AWS and Oracle, and how usage based pricing and sales tied into the organization’s culture and efficiency.

Our episode covers all the different ways organizations align themselves better when pricing is directly tied to the usage metrics of customers. We discuss how SaaS subscription models are simply reworking of traditional software licenses, how vendors can dispel fears around overages due to dynamic pricing models, and even why Netflix should be a usage-based-priced service :-)

We don’t have a show notes, but I thought it would be interesting to link the initial PR newsletter for S3’s launch, to reflect on how our industry has completely changed over the last few years.

• Security - exposing a service on the public internet has a variety of security implications - we want authentication/authorization as airtight as possible.

• Reliability - the API is not useful if it’s down often.

• Ease of use - it should be as frictionless as possible for developers to work with the API. Authentication flows, pagination, documentation should be as smooth and consistent as possible.

• Stability - public API callers cannot change in lockstep with the internal codebase, so long term stability guarantees should be thought through.

• Internal Developer Velocity - it should be simple for other developers in the organization to add new endpoints.

This is a vast and complex area, so I want to focus on one topic - API reliability. Once we want to consider our API generally available for a non-trivial set of users, we need to ensure it will stay reliable. In this post, we discuss common API reliability risks, mitigations, and a framework to keep our APIs reliable for the long term.

Reliability Risk Model

Just like security threat models, it’s useful to think about all the risks and potential scenarios that our API should be able to defend against. 

Breaking down risks into four categories:

Development risks: New faulty application code or infrastructure as code changes can lead to reliability issues. Additionally, application architecture and engineering practices also dictate ongoing reliability. For example, the right RPC library will enforce request timeouts and prevent overloads.

Deployment risks: The set of issues that prevent safe deployments. For example, the time it takes for a team to roll back a bad code change, and the likelihood that an on-call is empowered and trained to take a quick set of mitigations. There’s also the risk impact around faulty changes that can be tuned. For example, a faulty change might lead to a small availability impact to a canary environment, or a global outage, depending on our infrastructure and processes.

Abuse and load risks: APIs can go down even if there’s no changes going out due to changes in client behavior. Bad actors, and good but misconfigured clients are almost indistinguishable in most cases - and there’s a set of risks around abusive request patterns that have to be considered.

Infrastructure/dependency/operator risks: APIs are hosted somewhere, whether it may be a single machine in an office, or on thousands of cloud instances, and there’s physical infrastructure that might have issues. Alternatively, software infrastructure that our API depends on, like a caching layer, might fail and cause cascading issues.

Examples from my past experience/conversations:

• Bad code deployments - new bugs/misconfigurations that take our API down.

• Scanners that check for known vulnerabilities of common deployments. For example, Wordpress scanners that check if our admin site is publicly available. These might cause a large request volume.

• Security scanners deployed by the company’s security team that perform the same operations as above, but are less concerned about getting rate limited/banned. So the request throughput may even be higher.

• Cron jobs that make large bursts of requests on the hour that a service isn’t provisioned for.

• Large bursts of requests when a particular resource goes viral (the HackerNews hug of death).

• Large customers attempting a programmatic data-export/migration that cause database load due to hot keys/contention.

• Extremely large customers performing unbounded API operations.

  • At one of my previous jobs, we had a concept of users and teams, and teams were generally not more than hundreds of users. But we signed up with a university, and they added forty thousand students to a one team. None of our features and APIs were designed for that sized team.

• Internal services or batch jobs that overload a service that serves external requests.

• A single bad host in a small deployment that causes a large availability hit due to error amplification.

  • For example, successful RPCs to the service generally take hundreds of milliseconds, but failures return almost instantaneously. Therefore, more requests get routed to the bad host since it has more capacity.

• The website is down but the on-call engineer does not feel comfortable rolling back and is waiting for a 90 minute CI build for a forward fix to go out.

It’s worth coming up with a set of risks for our API, as it helps guide a set of principles to operate on and prioritize mitigations.

Mitigations

These are some common mitigations for various reliability risks.

Development Risks

Bounded operations

Our first observation is that most API defense measures like rate limiting, request quotas, and timeouts take place per operation. For example, rate limits are most often set up on requests per second. Therefore, we never want a single API operation to do unbounded work. Examples:

• List operations should always be paginated.

  • Let’s say we have a ListUsers API call - this might be very cheap for small customers, but expensive for large customers - there’s increased database load, network bandwidth and serialization cost for the same operation. We have to tune the rate limit for our largest customers, which increases our developer friction.

  • In addition, the number of returned items have to be capped. Otherwise, pagination is not useful.

• Set operations that need to operate on a list of items should always take a list of IDs.

  • And limit the number of items that can be operated on at a time.

This is likely the highest ROI principle to keep in mind - it’s easy to add more rate limiting infrastructure after the fact, but it’s hard to convert an unpaginated operation into a paginated one, because users will depend on older behavior.

Request timeouts

Low request timeouts are primarily an expectation setting mechanism and guardrail - if our API operations take too long, timeouts inform users that the operations have failed. If this happens sufficiently often, ideally, the API operator finds out about it through observability tools or support tickets, and can change the operation to do less work. It forces us to time-bound API operations. Ideally, the timeout also aborts the request so that the operation does not do any more work, but that’s often hard to implement correctly. For example, it’s really tricky to abort an ongoing request in Node. These little things are deeply considered in RPC libraries and implemented best by them - don’t roll your own request timeouts!

It’s very easy to bump up request timeouts as a cheap mitigation to a user perceived bug, but resist this temptation/schedule work to clean this up.

Deployment Risks

Change management 

Once a faulty change has been rolled out, there needs to be a clear set of owners or DRIs who are trained or feel empowered to mitigate the issue. For business critical services, consider having an on-call rotation of engineers who are trained and equipped to drive mitigations.

Emergency rollbacks

For stateless services, a rollback is almost always the right approach to mitigate code change related outages. Rollbacks don’t need to go through the full test suite, as long as we can guarantee that service is being deployed at a change that was previously tested and deployed - which greatly helps with mitigation time.

API validation tests

A staging deployment that has no reliability bearings on the production deployment is a great test bed for risky changes and continuous API validation tests.

Canary deployments / Staged rollouts

After a certain scale, it’s worth considering a separate canary deployment which receives code changes before the rest of the fleet. This helps prevent large outages if a faulty code change hasn’t been tested well enough in other environments. A canary deployment has the advantage of being most similar to the production environment, so it helps testing changes in a more realistic scenario. More sophisticated setups may benefit from Canary Analysis.

A canary deployment has an unexpected benefit of being a load request playground and can help us understand our load limits. For example, by introducing a load balancer that automatically retries on 500s and spreads requests across canary and production deployments, we can afford to have a few requests to our canaries fail without impacting global availability. This lets us run experiments, like tuning down the number of canary replicas and see how our instances operate under load. This area is worth investing in if we run a high throughput, business critical service.

Additionally, larger, multi-region or stateful services should be deployed in stages to reduce the blast radius of faulty changes.

Separating live and batch traffic

With a service oriented architecture, it’s very simple to overload an externally facing, critical service with an internal batch job that needs to RPC to the service. Even worse, I’ve seen cases where mobile applications run end to end tests against the production service. The major downside here is that completely disabling such a workflow becomes infeasible over time, since other business critical parts of the system depend on the API.

It’s worth trying to completely separate traffic tiers, or even deployments, to prevent non-critical traffic from taking down our system. The Google SRE book has a good notion on criticality.

Abuse and load risks

Token limits

Simple limits can go a long way in helping with API reliability. For example, a limit on the number of API tokens created per user can prevent a malicious bot-farm from working around disabled/rate-limited API tokens.

Admin tools

Consider having an option to disable a misbehaving API key in an admin portal. This significantly reduces the amount of time it will take to mitigate an outage, and the on-call engineer will thank us for it. Having this ability to disable misbehaving API keys relies on having enough observability to know which clients are being the most troublesome. The simplest path is to have sampled logs on request counts, broken down by customer ID/user ID/hashed API token/token ID so we can find the offender easily.

Web Application Firewalls (WAFs)

WAFs (AWS, Cloudflare) are multi-purpose, and help with basic security posture as well as coarse rate limiting.

WAFs often let us set up rate limiting rules per IP address, and some even let us set up rules on request headers/bodies. They’re good at preventing misconfigured clients, scanners, and other tools from hitting our service at all, so that more sophisticated rate limiting only needs to be applied at another layer with lower request volumes. Another advantage is that they’re very cheap - AWS charges 60 cents per million requests + a flat fee, and in terms of engineering time - it’s very easy to add it in a cloud console/IaC. Finally, WAFs often have an option to manually block requests from an I.P. address, which can come in very handy during emergencies. Open source proxies like Envoy have enough configurability to enable WAF-like functionality, and may support holistic in-built ones some day.

A limitation of WAFs is that they only apply on small time frames, like “request count over the past five minutes”. So they won’t really help with cron-jobs with a greater than five minute cadence, and they’re completely useless for the first few minutes of an outage. Another issue is that many universities and corporate entities have a NAT gateway, which all requests through a single I.P. address. So over-aggressive WAF rules are certain to block legitimate traffic.

API Gateways

API Gateways are a nebulous topic. They provide a central point for our internal services with the external internet, for example, you can serve requests through different services for different routes on the same domain. For the purposes of reliability, they can be thought of as a layer of abstraction between the chaotic outside internet and internal systems. For example, if our setup will ever have more than a single service available over the external internet, we will have to think about the same measures - WAF, observability, timeouts, rate limits, for all such services. Whereas if all our services are protected by the gateway, then these defense measures only need to be set up once. And over time, as we might want to centralize other aspects, like authentication/authorization, and the gateway’s routing logic can be updated accordingly and without needing to duplicate this logic across several services.

In practice, API Gateways often provide rate limiting as well, so we don’t have to re-implement it in-house. They offer capabilities like canary deployments to reduce the blast radius of code deployments. So it’s worth considering whether to start an API architecture with a gateway.

Rate limiting

The value proposition of rate limiting is fairly straightforward. The main question of rate limiting is always whether to fail open or fail closed.

• If the rate limiter fails open - we have to consider the case where an abusive client causes the rate limiter to go down. We need to be vigilant about monitoring operational errors in the rate limiter, since it can silently be disabled, which might lead to an outage.

• If it fails closed - we have to trust another infrastructure component with not degrading the availability of our API. For example, if our API has an availability SLA, then our rate limiter needs to meet or exceed that.

  • See the discussion on load testing below.

Auto-scaling

In a nutshell, auto-scaling means dynamically changing the number of replicas for a service based on a set of signals, like CPU utilization, request queue length, etc. Infrastructure platforms and orchestration frameworks often provide auto-scaling out of the box.

We have to be very careful to understand our systems before deploying auto-scaling. Let’s take the following barebones API architecture:

It might be simple to set up auto-scaling on the API service based on request latency. But that increases the number of concurrent requests to the database. If the database slows down due to increased load, that might lead to the auto-scaler adding even more API service replicas, which adds even more pressure to the database and might cause an outage. We expect this situation to happen whenever we’re I/O bound. This frequently happened at Heroku. Reactive auto-scaling should not be seen as a solution to malicious/abusive patterns, and more as a toil reduction measure for human operators, as well as a cost-saving measure for seasonal request loads. It is not an effective measure for sudden, borderline abusive changes in request throughput.

There’s also seasonal auto-scaling - preemptively scaling up instance counts during business hours or some other cadence due to a well known seasonal increases in load. This is generally more well understood, and a very reasonable approach.

Infrastructure / dependency risks

Outlier host detection

Given a sufficiently large number of instances running in a deployment, it’s very likely that a few instances will not be as performant as others. Cloud providers try to automatically replace or terminate poorly performing instances, but often, we have to do that work ourselves. That’s why, it may become relevant to set up monitoring for outliers and prevent requests reaching them. A poorly performing instance in a small deployment may cause error amplification if the instance fails requests much more quickly than a well-behaved instance processing successful requests.

Load testing

With a sufficiently complex architecture, it’s very hard to know how a system will perform under increased load. It’s worth understanding through synthetic load so that we know the system’s limits and can plan for future growth or seasonality of requests. Read this for a fascinating insight into load testing at Amazon.

Conclusion

To tie this up, an API developer has an array of things to consider to strengthen a public API. A suggested framework:

• Come up with a reliability threat model to the API which produces a set of risks.

  • All APIs have to defend against scanners - this is a standard risk.

  • APIs that allow unauthenticated use have to defend against bots and abuse much more than ones that only allow authenticated use - so some risks are unique to our business and technical context.

• Understand the impact of each risk - would the lack of defenses for the risk take down our entire site, or would the impact be limited to a single customer? These answers depend on our pre-existing software architecture. For example, if we have a single tenant architecture, the blast radius due to an abusive customer is much smaller.

• Consider the mitigation speed for various risks, especially as they tie into your SLOs, as that will drive a discussion into remediation procedures.

With this input, we can come up with mitigations, a software architecture, and development principles and processes to keep our APIs reliable for a long time.

Credits

Thanks to Allan Reyes for reading early drafts of this post and providing excellent feedback!

Apple Podcasts | Spotify | Google Podcasts

Sorry for the long hiatus in episodes!

Today’s episode covers a myriad of interesting topics - from being the star of one of the internet’s first viral videos, to experiencing the hyper-growth at the somewhat controversial Zenefits, scaling out the technology platform at Fanatics, starting a company, picking an accelerator, only permitting in-person work, facilitating career growth of gig workers, and more!

Highlights

[0:00] - The infamous Spelling Bee incident.

[06:30] - Why pivot to Computer Science after an undergraduate focus in biomedical engineering?

[09:30] - Going to Stanford for Management Science and getting an education in Computer Science.

[13:00] - Zenefits during hyper-growth. Learning from Parker Conrad.

[18:30] - Building an e-commerce platform with reasonably high scale (powering all NFL gear) as a first software engineering gig. Dealing with lots of constraints from the beginning - like multi-currency support - and delivering a complete solution over several years.

The interesting seasonality - like Game 7 of the NBA finals - and the implications on the software engineers maintaining e-commerce systems. Watching all the super-bowls with coworkers.

[26:00] - A large outage, obviously due to DNS routing.

[31:00] - Why start a company?

[37:30] - Why join OnDeck?

[41:00] - Contrary to the current trend, Traba only allows in-person work. Why is that?

We go on to talk about the implications of remote work and other decisions in an early startup’s product velocity.

[57:00] - On being competitive.

[58:30] - Velocity is really about not working on the incorrect stuff.

[68:00] - What’s next for Traba? What’s the vision?

[72:30] - Building two-sided marketplaces, and the career path for gig workers.

Apple Podcasts | Spotify | Google Podcasts

We discuss the state of web development in the industry today, and the various different approaches to make it easier.

Contrasting the Hasura and Convex approach as a good way to illustrate some of the ideas. Hasura lets you skip the web-app, and run queries against the database through GraphQL queries. Convex, on the other hand, helps you stop worrying about databases. No setup or scaling concerns. It’s interesting to see how various systems are evolving to help developers with reducing the busywork around more and more layers of the stack, and just focus on delivering business value instead.

Subscribe now

Share Software at Scale

Convex also excels at the developer experience portion - they provide a deep integration with React, use hooks (just like Apollo GraphQL) and seem to have a fully typed (and therefore auto-completable) SDK. I expect more companies will move “up the stack” to provide deeper integrations with popular tools like React.

Episode Reading List

• The co-founders of this company led Dropbox’s Magic Pocket project.

• Convex → Netlify

• Convex vs. Firebase

• Prisma

Apple Podcasts | Spotify | Google Podcasts

We discuss a new category of developer tools startups - API Gateway Management Platforms. We go over what an API Gateway is, why do companies use gateways, common pain-points in gateway management, building reliable systems that serve billions of requests at scale. But most importantly, we dive into the story of Josh’s UK Developer of the Year 2009 award.

Recently, I’ve been working on the Vanta API and was surprised at how poor the performance and developer experience around Amazon’s API Gateway is. It has poor support for rate limiting, and has very high edge latency. So I’m excited for a new crop of companies to provide good solutions in this space.

Subscribe now

Share Software at Scale

Episode Reading List

• Amazon’s API Gateway

• Stripe’s API - The first ten years

• Envoy

The Award

Apple Podcasts | Spotify | Google Podcasts

This episode dives deep into the history of OpenTelemetry, why we need a new telemetry standard, all the work that goes into building generic telemetry processing infrastructure, and the vision for unified logging, metrics and traces.

Subscribe now

Share Software at Scale

Episode Reading List

Instead of highlights, I’ve attached links to some of our discussion points.

• HTTP Trace Context - new headers to support a standard way to preserve state across HTTP requests.

• OpenTelemetry Data Collection

• Zipkin

• OpenCensus and OpenTracing - the precursor projects to OpenTelemetry

Apple Podcasts | Spotify | Google Podcasts

Share Software at Scale

Subscribe now

Many of us have struggled (or are struggling) with permission management in the various applications we’ve built. The complexity of these systems always tends to increase through business requirements - for example, some content should only be accessed by paid users or users in a certain geography. Certain architectures like filesystems have hierarchical permissions that efficient evaluation, and there’s technical complexity that’s often unique to the specific application.

We talk about all the complexity around permission management, and techniques to solve it in this episode. We also explore how Permit tries to solve this as a product and abstract this problem out for everyone.

Highlights

[0:00] - Why work on access control?

[02:00] - Sources of complexity in permission management

[08:00] - Which cloud system manages permissions well?

[11:00] - Product-izing a solution to this problem

[17:00] - What kind of companies approach you for solutions to this problem?

[22:00] - Why are there research papers written about permission management?

[38:00] - Permission management across the technology stack (inter-service communication)

[42:00] - What are you excited about building next?

For the purpose of this blog post, let’s solidify the definition of a platform team - a platform team doesn’t directly focus on additional customer value (new products/features) and has other engineering teams as customers. Examples include developer tools teams, infrastructure, or reliability teams where the team works on internal systems which other customer-facing teams use, directly or indirectly. On the other hand, a team that purely focuses on the reliability of an internal system, where the work doesn’t impact other engineering teams, is not a platform team.

I’ve been on such platform teams that own developer tools, build and deploy infrastructure, and internal platforms for the past few years. In these teams, the largest challenge has been crafting, scoping, and prioritizing initiatives every planning cycle. Each team operated with a unique set of constraints and aimed to solve different problems, but over time, I noticed some themes that held true across teams, and also resonated with other folks I spoke to across the industry.

So I created an aspirational process to systemize prioritization of platform work, and I’ve sketched it out in this blog.

Strategy Challenges

Platform strategy is tricky for many reasons.

First, you probably have customers with conflicting needs. It’s easy to make your system seem more reliable by reducing the frequency of changes you allow or preventing Friday night deploys, but that would affect developer velocity so it might not be the right trade-off.

Second, it’s an N-dimensional problem - if you own multiple areas like security and reliability, you might have projects that could solve problems in both areas, so we should be forcing ourselves to think holistically. For example, a project to improve build times might improve both developer experience and get the finance team off your back, and leave you room for other important initiatives. In contrast, driving a project solely to optimize CI/CD infrastructure might help with the same cost savings, but won’t improve developer experience.

Third, technical choices tend to have compounding implications over time, which make decisions (and the corresponding opportunity costs) feel expensive and Type-1. For example, if you choose to punt on enforcing that your codebase is fully typed today, the codebase will keep growing, and solving this problem in the future will require more work, with the additional work often proportional to the growth of your engineering team.

Lastly, the cost/benefit calculation of each initiative constantly changes due to the rapid changes in our industry and the technological landscape. For example, if you wait just one more quarter, maybe AWS/Kafka/<insert offering> will add that feature that solves this problem for you in a much cheaper way. Alternatively, you might pick the wrong horse when you’re an early adopter of a particular technology, and you might have to pay down that cost by a painful migration later on.

To summarize:

1. Platform teams often have customers with conflicting needs and have to think carefully about the implications of each project

2. Projects might have an impact across multiple areas of ownership so it pays to think holistically

3. Technical choices compound over time, increasing the cost of mistakes

4. The fast-moving industry landscape causes the cost/benefit calculation of each choice to change rapidly

Prioritization Considerations

With all these in mind, it’s helpful to come up with a set of guidelines that help with effective prioritization.

Principles & Commitments

Principles are an abstract set of statements that help guide prioritization. For instance, AWS has a job zero which is considered table-stakes for the team and takes precedence over any other work. AWS states that security is their job zero, and for good reason - a customer data leak could be highly damaging to the brand, turn away future customers, and cause significant monetary loss. Losing the keys to the kingdom might cause your business to shut down.

Slightly more concrete is commitments - the set of properties that your system should adhere to, or the set of features that are considered the bare minimum to support. It’s important to ensure that commitments are met, and breakages are resolved in a timely manner. For example, at Vanta (where I work at the time of writing), the Platform team commits to ensuring that hot-reloading works for front-end development.

It’s important to systemize commitments - shared publicly within the organization, and have general alignment on the importance within the team, so that teams can share a reference on the prioritization of commitment-related work. Without systemized commitments, teams often end up having implicit commitments, where the set of what actually gets fixed depends on the interests and engagement of individuals, rather than the team. For example, if you have an engineer who cares that alert volume stays low, it’s easy to grow dependent on them, and the system would degrade over time when they decide to move on and no one prioritizes work to improve the situation.

SLAs are basically metric-based commitments, and provide easier tracking, alerting, therefore accountability.

Having a list of commitments makes your job as a prioritizer easier - if it’s not easy maintaining a set of commitments, it’s very likely that you either need to change them or prioritize a project that solves the underlying problems. This is similar to Will Larson’s thinking on whether your team is operating in Foundation or Innovation mode - it should be somewhat straightforward to plan your work if there’s a lot of work to be done to meet what your team considers table-stakes.

Understanding

A platform team has to have a deep understanding of its domain. The team has to understand its mission, purpose, and how it helps with the overall business. After this, it needs to have a reasonable understanding of the industry - how teams with similar missions in other companies are generally run, and upcoming technology in their field. For example, an observability team should have a rough understanding of the capabilities of observability systems that can be used for companies of their given scale. A developer tooling team at a certain company size should understand how much companies at their size usually spend per engineer on internal tools.

There are fairly obvious downsides when a team doesn’t understand its mission. But if it doesn’t understand its systems - it will likely find it hard to resolve issues during an outage, efficiently make small changes to meet simple internal customer needs, empathize with the quirks in the system, and be aware of when the system genuinely isn’t going to be enough. Common symptoms of a platform team without sufficient knowledge are team-members grumbling about how complex and opaque the existing system is, a fear of making changes, and a continuous stream of reactive work.

Toil

Platform teams need to have a strong focus on toil reduction. A team that works purely on the most business impactful projects and doesn’t take team sustainability into account is likely to have burned-out team members and attrition, which doesn’t help the business. The exact amount of time that your team spends on toil vs. projects is case by case - Google SRE points to 50%, but I’ve found that to be too much in practice in a smaller company - more than 30% is probably excessive.

Impact

First, it’s important to recognize that platform teams are ultimately responsible for business continuity and performance, not employee happiness. A platform team might nominally be serving internal teams, but it needs to prioritize based on what will help the business most, not by the raw number of teams impacted. In order to do this, platform teams have to understand company and business strategy as well as regular product teams.

Measuring the impact of a particular project is often an art when there aren’t reasonable metrics to help guide the decision, and deserves its own blog post.

Taking all these pieces, we have enough of the picture to help formulate our system.

Platform Strategy Decision Maze

Putting all the considerations together - we show a rough flow-chart to help formulate strategy. This is similar to the concept of an idea-maze for thinking about startup ideas.

We can call this the decision maze.

The key idea is for the team to develop the muscle to successfully execute foundational work, obtain mastery over their systems, reduce toil to ensure that unplanned work is bounded, and fundamentally understand their domain. This buys time and expertise for the successful execution of forward-thinking projects and keeps the system sustainable for changing circumstances.

For foundational investments, in prioritized order, the team should:

• Maintain its commitments to the business (or tweak them)

• Understand its domain

• Minimize toil

• Ensure sufficient investment in each area that they own

Armed with the breathing room from these foundational investments, the team could focus on compounding leverage - initiatives that can deliver increasing value over time, or prevent issues that would get likelier over time, and cross-cutting efforts - systematic efforts that would improve multiple areas of ownership.

Let’s explore how this solves the problems we originally laid out.

1. Teams ensure that they stick to their commitments as they execute projects, keeping customers with conflicting needs satisfied.

2. Teams develop a holistic understanding of their systems and the state of the industry, which helps tease out projects that might impact multiple areas of ownership and understand what they can incorporate from outside.

3. Teams focus on solving problems with compounding leverage and multiple areas of impact over other problems.

The next challenge is being satisfied with the answers to the questions at each decision point in the decision maze.

Above, we’ve sketched out potential methods to help guide each decision.

Let’s take an example of an infrastructure or reliability team tasked with network traffic management trying to lay out its strategy. The team could:

• Set up their security principles and SLAs on availability numbers, and gain the confidence they will be able to hit those in the medium term, otherwise, fund a project to get there

• Understand their various owned subsystems like reverse proxies, the subsystem’s relative strengths and weaknesses, and how they stack up with similar open-source systems/paid products

• Reduce their toil like on-call alarms through tackling low-hanging fruit, or have a plan on the roadmap to resolve the problem

• Perform risk analysis and forecast growth to understand whether they need to invest in security, availability, or both

• Talk to internal customers about capabilities that their systems should have and do some requirements gathering to understand blind spots

• Investigate whether a project would let the team have significantly better commitments, increased system simplicity, reduced toil, or provide significant business impact on an important dimension like latency

To be clear, this isn’t a hard-and-fast ruleset, and teams often need to prioritize projects that don’t follow the prescribed ordering for various important reasons like team morale or momentum. Additionally, there might be the need for foundational projects in one area of ownership, and innovative projects in another area, and work can take place in parallel. So truly, these are just rules of thumb to kick-start a conversation on the extremely tricky and unverifiable process of crafting a strategy for platform teams.

Credits

Credits to Robbie Ostrow for teaching me about systemizing commitments, and Will Larson for his excellent writing on infrastructure prioritization, foundational vs. innovation investments, and picking the right work.