A few years ago, I ran into this exact question while designing a file upload service. After weighing the trade-offs, I chose object storage for scalability and cost.

But that decision isn’t always obvious.

Are there cases where storing files directly in the database is actually the better choice? 🤔

In this article, we’ll break down both the approaches, explore their trade-offs, and look at how real systems decide between them.

With that, let’s begin by defining the problem statement for our system.

Subscribe now

Problem Statement

We want to build a web application that supports:

1. File uploads with metadata (title, description, category).

2. Retrieves recent uploads with image previews.

The diagram below shows the application’s UI screen.

Let’s explore the simplest technique to solve this problem.

Naive Solution - Store files in the database

Most databases can persist files in a binary format. PostgreSQL provides a data type bytea to store raw binary data directly inside the database.

Here’s how you can define a schema with a column to store binary files:

In the above schema, the field file_data will store the binary file data.

Most databases (e.g., PostgreSQL bytea, MySQL BLOB, MongoDB GridFS) support storing binary data directly.

You can use any database of our choice and add backend servers that expose upload and retrieval APIs to build the application.

The diagram below shows the detailed architecture of our system.

Our solution is simple and easy to manage with less dependencies. This works well for low-traffic, basic use cases.

But, what would happen with more uploads or user growth? Would the system scale? 🤔

Before reading further, take a moment to think what can go wrong while operating the system at scale.

Challenges

• Performance bottlenecks: Large binary data increases memory usage, slows queries, and impacts backend throughput.

• Backend server performance - With traffic growth, increased memory pressure on backend servers leads to Out of Memory errors leading to unavailability.

• Size constraints: Databases impose limits on maximum field sizes. For example: PostgreSQL limits the max field size to 1 GB while MySQL supports 4 GB.

• Operational overhead: Replication and backups become slower due to large data volumes.

• Cost inefficiency - Databases are 5-20x more expensive for storing large binary objects than blob storage.

We will now see how we can address these challenges through a well-known pattern used in the industry.

Blob storage for file content, Database for metadata

Instead of storing the whole file in the database, we can separate file content from metadata and store them independently. Here’s how we can solve it:

• File content - The actual raw binary data can be stored in a blob storage like S3.

• File metadata - The attributes(title, description, etc) can be stored in any database along with the reference to the path of the file stored in the blob storage.

We will remove the file_data column from the table and substitute it with file_path column. Here’s the new database schema:

Unlike the previous approach, the file upload becomes a two step process:

1. Store metadata and generate a pre-signed URL.

2. Upload file content directly to object storage using that pre-signed URL.

The diagram below illustrates the process and the architecture.

Here’s how this solution overcomes the challenges of the naive approach:

1. Database performance - The database doesn’t store binary content. Since the metadata size for each record is less than 100 KBs, it can easily scale to millions of records without any performance impact.

2. Backend server performance - Files are served directly from blob storage, bypassing the backend entirely. This prevents Out of Memory errors ensuring high availability.

3. Database constraints - The blob storage doesn’t impose specific limits and can scale to handle terabytes of data.

4. Database backup and replication - Backups and replication are faster since large files are excluded.

5. Cost efficiency - Storing data in a blob storage is significantly cheaper than storing in a database. This significantly reduces the application costs.

While this solution is efficient and scalable, it has the following downsides:

• Eventual consistency: Metadata and file content are stored separately, requiring reconciliation.

• Operational complexity: Multi-step uploads and failure handling (e.g., retries, cleanup) must be managed.

• Dual backups: Both database and object storage must be maintained reliably.

However, the above trade-offs are acceptable given its scalability benefits.

Food for thought - How do you handle file upload failures while using blob storage? 🤔 (Leave your thoughts in the comments)

Now that you know two solutions exist, how do you choose between the two while solving real-world problems?

Let’s now walk through a few real-world scenarios and understand the suitable approach with the reasoning for each.

Real-world scenarios

Financial transactions and receipts

Financial transactions store the receipts in the form of pdf or image files. They also require strong atomicity guarantees.

For example: The system must generate receipt immediately after receiving the amount and persist it. A multi-step process involving a blob storage upload is not ACID compliant and requires separate reconciliation pipeline for eventual consistency.

Relational databases provide strong consistency out of the box. This makes them appropriate for use cases like storing invoices, receipts, signed tax forms, etc.

Social media assets

Storing billions of files in a database isn’t practical for social media companies like WhatsApp, Instagram, X, etc. They prefer to keep the media files such as images, videos and gifs in blob storage than the database.

Blob storage allows users to store large files (> 100 MB) through multi-part uploads. Similarly, the direct CDN integration and caching improves read performance.

Configuration files

Software development teams keep YAML, XML and other small configuration files in the database. Unlike social media assets, these files are not viewed frequently on a UI and only read once by the application during the startup.

Secure/controlled access environments

For strict security and regulatory compliance, defense or government systems store files in databases that provide built-in encryption, access control, and audit logging.

Conclusion

We learnt that there’s no one size fits all solution for storing files. Each problem is unique with its own constraints.

The below table provides a framework to choose between the two approaches based on the different quality attributes.

If your system prioritizes strong consistency, simplicity, and small file sizes, storing files directly in the database can be the right decision.

But if you need scalability, high-throughput and cost-efficiency, blob storage becomes the most robust and future-proof choice.

Many companies often adopt a hybrid approach - using databases for critical, tightly coupled data and object storage for large, scalable assets.

This helps companies get best of both worlds rather than constraining to one pattern.

The key is to decide based on the first principles: data size, access patterns, consistency guarantees, and operational complexity—not just convention.

What approach are you using today—database or object storage? 🤔 And what trade-offs have you run into? Leave your thoughts in the comments below.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

But, do they fully safeguard a database from unexpected traffic spikes? Would adding 10 service instances result in a 10× increase in database connections? 🤔

Database proxies prevent this by residing between the service and the database and acting as connection multiplexers.

Moreover, proxies also help in solving other challenges such request routing, query caching, failover, and so on ensuring scalability and reliability.

In this article, we will understand how proxies solve the scaling and availability challenges in distributed systems. The article will also discuss trade-offs and include a decision matrix to guide the proxy decision.

With that, let’s first understand the various challenges in building distributed applications with a relational database like PostgreSQL or MySQL.

Subscribe now

Distributed System Challenges

Assume that we have a service managing social media tweets that is capable of handling 4K TPS. Let’s say that it fetches the data from a PostgreSQL instance by creating 100 connections.

What would happen if the TPS rises from 4K to 40K? How would we manage the additional load?

Scaling

We would need to deploy 10 service instances to handle the load of 40K TPS. Since each service instance requires 100 connections, there would be 1000 (100 x 10) connections to our PostgreSQL instance.

The diagram below illustrates how horizontal scaling can increase the number of database connections.

The database would struggle to handle the connection surge and fail new connections. Thus, the database would become a bottleneck and limit the system’s scalability.

Duplicate requests

At 40K TPS, there would be multiple requests for the same resource. For example: a request to fetch the same tweet.

With several instances, the same SQL query would be executed multiple times returning the same data. This would result in an additional database load slowing the overall execution and leading to a poor user experience.

We can solve this challenge through a technique known as sharding that distributes the data over multiple database or by having read replicas. But it brings in a different set of challenges that we will see in the next section.

Sharding and Read replicas

With several database instances, we now need to pass the configuration (host, port, etc) of each database instance to the application. We need to write additional code so that the application is aware of the different shards and handle the read/write segregation explicitly.

This introduces the following challenges:

1. Tight coupling - The application is tightly coupled with the sharding logic and the available shards.

2. Scaling challenges - The application needs to be updated with the addition of each new replica by passing the updated configuration.

3. Inflexibility - If application code is rewritten in a different stack, it would require duplication of the existing sharding logic.

The below diagram shows how an architecture involving database replicas and read/write segregation increases the complexity.

While these problems can be solved, it would require building a library for handling the sharding logic. We would need to develop this for every programming language and database.

Now that you understand the core problem, let’s understand how Database proxies solve the problem.

What is a database proxy?

Database proxies are software applications that sit between a service and a database. They create and manage a pool of database connections, and applications connect to the proxy instead of directly to the database.

They intercept the service connections and multiplex them onto a fixed pool of database connections.

The below sequence diagram shows how the process works.

Let’s now understand how Proxies solve the scaling challenges described in the previous section.

Scaling

Proxies create a fixed number of connections to the database. While the service-proxy connections increase with horizontal scaling, the number of database connections remains constant.

This design protects the database from additional connections reducing the chances of failing new connections. Similarly, the load on the database remains stable due to fixed number of connections.

How many connections do you think a single Proxy can support? 🤔 (Share your answers in the comments below)

The following diagram shows how Proxy handles additional connections from application by keeping fixed number of database connections.

Functions as a Service (FaaS) platforms like Lambdas widely adopt proxies. Since every Lambda is short-lived, every invocation leads to setup and teardown of a database connection.

Additionally, the problem amplifies when the number of concurrent executions exceeds maximum database connections.

With proxies, Lambdas no longer connect to the database directly and instead connect to the proxies. As a result, the database remains shielded from any spike in the Lambda layer.

Read Caching

Many proxies can be configured to cache the database query results. The proxy then returns the results from the cache instead of executing the database query.

Caching improves the response time and also reduces the overall load on the database. Proxies such as ProxySQL and Pgpool-II support native in-memory caching with configurable cache size and the TTL (time-to-live).

Sharding and Read replicas

Proxies can be configured with rules to segregate the read/write requests. Similarly, we can leverage the same technique to shard the data across several database servers.

The following diagram shows how proxies simplify configuration and abstract database shards/replicas.

Unlike traditional ways of hardcoding sharding in the application, proxies help through:

1. Reduced coupling - The sharding logic is abstracted and configurable eliminating the tight coupling.

2. Ease of scaling - New shards/replicas can be easily introduced by making configuration changes using admin interface.

3. Flexibility - Rewriting the application code wouldn’t require additional efforts to integrate the sharding logic.

While database proxies offer several benefits, do you think it makes sense to use it always?. Let’s now dive into the trade-offs that would help us decide whether to use a proxy or not.

Trade-offs

Deployment complexity

Since proxies are separate applications, they need to be deployed resulting in an additional overhead. This overhead compounds when you have to deploy a proxy on a different machine than the database server.

Maintenance overhead

The development team now needs to maintain, patch and update the proxy. Similarly, they need to manage the OS upgrades and fix vulnerabilities in the server hosting the proxy.

Latency

Since the queries go via the proxy, it could increases the latency by 10-15 ms. This could impact the performance in case the application requires strict latencies.

Proxies are a must when your system has complex use cases such as sharding, replication, read/write segregation and caching. In such cases, the benefits outweigh the costs. However, for a single database system, they are can often become an overkill.

Let’s conclude with a decision matrix to help you decide whether to use a proxy.

Conclusion

While database proxies are versatile and solve several distributed system challenges, they are not always required. Before using one, you should think about your workload, performance needs, and the complexity that you are willing to manage.

Here’s a decision matrix to help you decide whether you should use a proxy in your architecture.

Now that you understand the working of proxies, here’s a question that’s worth reflecting on - Can proxies become a single point of failure and bring down your application? 🤔 If yes, how do we prevent such scenarios?

Leave your thoughts in the comments below. We will address this question in the next article.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Educative has officially launched their Year End sale— and it’s their strongest offer of the entire year.

Whether you’re preparing for interviews, mastering new technologies, or strengthening your fundamentals, Educative provides high-quality, in-browser learning with no setup required.

Use my link to get an extra 10% off

Under the hood, your application uses a TCP connection to connect to the database. A spike in the number of connections would overwhelm the database and slow down the query execution.

Connection pooling solves this by maintaining a pool of reusable connections. While the concept appears simple, there’s complex engineering behind an effective connection pool implementation.

In this article, we will dive deep and understand how connection pooling works. We will explore the different challenges that a connection pooling library tackles and common trade-offs that engineers make while using a connection pool.

With that, let’s see what happens under the hood when your application executes a query.

Subscribe now

Query Execution

Let’s assume that your system stores users in a relational database like PostgreSQL and exposes an API to fetch user information. Your application will execute the query - Select * from users where userId = 30 to fetch the user information.

The below diagram illustrates the database connection establishment that happens before your application executes any query.

The connection establishment can take anywhere between 20-50 ms and 200+ ms if the database is in a different region.

With this setup, what would happen if the clients make multiple API calls to fetch different users?

Here’s what would happen for every API call:-

1. The application would create a database connection.

2. It would use the connection and send the SQL query.

3. Get and process the results of the SQL query and send the data back to the client.

The below diagram shows how the application would create a connection for every API call.

While the above setup may work for 100-200 requests per second, do you think it would handle 10K or 100K requests?

Let’s explore in the next section the ways in which the system would break.

Database limitations

Let’s evaluate the impact on scalability, resource utilisation and performance.

Scalability

If the system receives 10K RPS (requests per second), it would make equal number of connections to the database. Most of the databases are not optimized to handle 10K simultaneous connections. For eg:- PostgreSQL sets default max value of connections to 100 (can be increased).

This would result in connection throttling and database would return an error like “sorry, too many clients already“.

Resource utilisation

At 10K RPS, the CPU utilisation would spike to handle large number of TCP connections. It would also increase the context switching and potential network-level throttling.

Every connection also consumes memory and it would increase the chances of memory exhaustion in database.

Performance

Every connection setup would add 20-50 ms to the latency. Further, resource exhaustion would slow down the queries and increase the API latency. The slowness would degrade the user experience.

As a result, the database becomes a bottleneck when a connection is created for every request. Let’s now see ways to overcome this bottleneck through Connection Pooling.

Connection Pooling

A connection pool is a collection of pre-established database connections that an application can reuse. Here’s how an application creates and uses the connection pool:

1. During the application startup, the application creates pre-defined number of database connections.

2. The connections are then stored in a data structure for future use.

3. If an application thread wants to execute a query, it would borrow one connection, use it for query execution and return it back to the pool after completion.

4. In case all the connections are exhausted, the thread waits until a connection becomes available and then continue its execution.

The below diagram describes how a connection pool works.

Here’s sample Java code that illustrates how connection pool is used by an application.

The above code uses a popular library named Hikari-CP in Java. As seen in the code, it first initializes the pool’s configuration using parameters such as maximum pool size, idle connection count, etc and then fetches the connection.

With the above approach, the connection pool ensures:

• Scalability - The database is not overwhelmed with large number of connections preventing connection throttling.

• Resource utilization - Since the number of connections are limited, it optimally uses resources such as memory, network and CPU.

• Performance - The connection creation overhead is eliminated and thus can save 20-50 ms for every query execution. Further, limited connections exert less pressure on the database ensuring predictable performance.

The Biggest Black Friday Deal for Developers Is LIVE!
Educative just launched their Black Friday + Cyber Monday sale, and it’s hands down their best offer of the entire year.

Look no further than Educative.io’s course on Grokking the System Design Interview. You can use this link to get an additional 10% off.

Pool-size Trade-offs

One question that puzzles most developers is What should be the right pool size? When I first used a connection pool, it took me several weeks of tuning to get the correct pool size.

Here’s why getting the right pool size is a tough nut to crack:

1. Small pool size - Small size implies few connections and less database resources. With request spike, waiting time increases for additional requests slowing down the execution.

2. Large pool size - Large pool size are reliable to handle a sudden spike in the load without degrading the performance. However, it wastes resources if the workload has low volume.

There’s a resource-performance trade-off and developers need to strike a right balance between the two.

Instead of blindly setting the pool size, one can follow the following approach:

1. Start with a small pool size (20-30 connections).

2. Load test with real traffic and observe the impact on metrics such as database CPU, memory, wait time, request latency, etc

3. Experiment with different number of connections and observe the impact on the metrics.

4. Keep a buffer of 15-20% above peak usage while setting the maximum pool size.

At times, it helps to have multiple pools with each catered for a different workload pattern. For eg:- A small pool for low-volume traffic and a large pool for high-volume traffic.

Now that you understand the trade-offs, let’s uncover the challenges behind building a connection pool.

Challenges

The connection pool’s dataSource.getConnection() abstracts most of the complexity from the users. It hides all the challenging semantics related to concurrency, database driver and network.

Building a connection pool library from scratch is exciting and equally gruelling task for a developer. Unlike usual programming task, it involves design considerations such as:

• High-performance - It must be able to handle multiple concurrent calls with low contention. Best libraries often leverage lock-free data structure to achieve the same.

• Zero-overhead - Connection creation is an expensive process and results in a performance penalty. Hence, the library must ensure it returns live connections.

• Simple and flexible - The interface must be simple to use and easily configurable through parameter tuning.

If you want to learn more about connection pooling, you can go through the code of one of the open source libraries. Personally, I would recommend navigating HikariCP’s code.

Moreover, if you are learning a new language like Rust, you can build a connection pooling library from scratch and then experiment with it.

So far, we considered a single application instance using a connection pool for our understanding. In case there are 10x instances, wouldn’t the number of database connections also 10x? 🤔

Wouldn’t the database get overwhelmed this time again? 🤔 If yes, how can we solve this? Leave your thoughts in the comments below. We’ll take up this question in the next article on database proxies.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Have you wondered how X(Twitter) let’s you scroll infinitely through a stream of tweets? Does the app pre-fetch all the data or does it fetch as you scroll?

Similarly, how does Spotify let you view all the albums and songs of any artist in a single page? Does it use a technique similar to X?

Both the features solve the same fundamental Computer Science problem - Retrieve and display a subset of data from a large dataset.

That’s where API pagination comes in. It’s a technique that splits large datasets into smaller chunks that are fetched and displayed as needed.

In this article, we’ll break down how pagination works under the hood using real-world examples from X and Spotify.

By the end, you’ll be able to confidently choose the right pagination strategy by understanding the trade-offs between Limit-Offset and Cursor-based pagination.

Let’s start by exploring why pagination is even necessary.

Subscribe now

Why Pagination?

Imagine you’re building a system like X, where the client fetches tweets from a server. The server stores billions of tweets, but each user might only need a few dozen at a time for an infinite scroll experience.

What if we fetched everything upfront?

Let’s say the client pre-fetched 10,000 tweets and cached them locally. It would make scrolling seamless — but at a huge cost:

1. Slowness - Given 10K records, initial fetching would be extremely slow.

2. Bandwidth - Fetching 10K tweets for millions of users would be redundant and waste the bandwidth.

3. Memory - Storing 10K records would consume and drains app’s memory. Similarly, it would consume the server’s memory due to caching (as shown in the above diagram).

As a result, it's not practical to retrieve all the tweets from the backend server. So, what’s the solution?

It’s simple. You break down the large dataset into small chunks, and fetch each chunk. This pagination approach would be fast, save bandwidth and also optimize memory.

Now, that you know what pagination is needed, let’s now dive deep into the different techniques.

Ready to analyze complex system failures like a senior engineer? Learn the system design patterns that prevent outages and impress interviewers at top tech companies.

Look no further than Educative.io’s course on Grokking the System Design Interview. You can use this link to get an additional 10% off.

Limit-Offset Pagination

In this technique, the server divides the dataset into different pages with each page consisting of a set of records. The client sends the below attributes:

1. LIMIT - The number of records that the server must return.

2. OFFSET - The page number or the number of initial records to skip.

Let’s understand this with the example of Spotify. Spotify allows users to browse the different albums of an artist.

It exposes the following API endpoint along with the request body to retrieve the data:

POST - https://api-partner.spotify.com/pathfinder/v2/query
Request - 
{
  "operationName": "queryArtistDiscographyAlbums",
  “variables”: {
     "uri": "spotify:artist:12345"
     "offset": 1,
     "limit": 3
  }
}

Here’s how the server processes the request:-

1. Extracts the limit and offset fields from the request.

2. Constructs a database query by passing the value of limit and offset.

3. Executes the database query to skip the first offset records and then fetch the count of records stated in the limit.

The below diagram shows how the process works behind the scenes.

As you scroll down, the app would further sends one more request to fetch the next 3 albums (using offset=4 and limit=3).

This technique offers the below advantages:-

1. Simplicity - It’s simple to understand and easy to implement.

2. Easy navigation - Users can skip first N records (offset) and jump to the desired page of choice simplifying the navigation.

While this technique works for Spotify’s use case of showing artist’s albums, do you think it would work for X’s infinite scroll? Take a moment to think before reading further.

With the Limit–Offset approach, the database reads all records and skips the first N records specified by the offset. This results in redundant computation making the process inefficient.

Food for thought - How would you decide the result size (i.e limit) for each page in Limit-Offset pagination? 🤔 (Leave your thoughts in the comments)

The approach has the following limitations:-

1. Poor Scalability - It struggles with large datasets since it needs to read and skip large number of records.

2. Subpar Performance - With large datasets, the latency to read the records increases thereby spiking the user’s page load times. This results in a poor user experience.

3. Data Inconsistency - As new records are added/deleted, the users may view duplicate or missing items while scrolling through the pages. This makes the approach unstable.

Now that you understand the Limit-Offset based pagination, let’s explore how Cursor-based pagination tackles its downsides.

Cursor-based Pagination

In this approach, the server returns records along with a cursor or a marker. The cursor is a string that points to the last record returned in the list.

In the subsequent requests, the client passes the cursor. The server then resumes reading from the first record after the cursor.

Let’s understand this in the context of X’s timeline. X exposes the following API for fetching the timeline of a user:-

POST - https://x.com/i/api/graphql/{userId}/HomeTimeline
Request - 
{
{
  “count”: 20,
  “cursor”: “DAABCgABG3hmIsG__0cKAAIbd4b1v1axJAgAAwAAAAIAAA”,
  “includePromotedContent”: true,
  “latestControlAvailable”: true,
  “withCommunity”: true,
  “seenTweetIds”: [
    “1979104903193923617”,
    “1979054161174942195”,
    “1979205840008597948”,
    “1979194565870616788”,
    “1979221342953382229”,
    “1979080720552427991”
  ]
}
} 

Here’s what happens when the client calls the HomeTimeline API:-

1. During the initial load, the client wouldn’t set the cursor and the server would return 10-15 latest and top-ranked tweets along with the cursor.

2. Once the client scrolls and reaches the last tweet, it would send another request with the cursor (sent in the last response).

3. The server would then fetch the tweets (starting from 16th Tweet) and return it.

This process would continue and the client would get infinite scrolling capability.

The diagram below illustrates the working of how X renders the timeline using Cursor-based pagination.

The cursor usually is a part of the indexed database attribute (like post_id) to speed the look-up. When the server receives the cursor, it performs a fast lookup and jumps to the record corresponding to the cursor. Subsequently, it reads the number of records specified in the count.

Here’s how the approach overcomes the downsides of Limit-Offset based approach:-

1. Scalability - The database directly navigates to the cursor instead of reading the first N records. This makes the process efficient and scales to a large number of records.

2. High-performance - With index-based lookups, the latency is much lower resulting in excellent user experience.

3. Stability - Since it fetches the records after the cursor, so addition or removal of records doesn’t affect the user experience. This makes the process stable compared to Limit-Offset based approach.

Food for thought - Do you think the cursors should have an expiry? What if the record at the cursor is deleted? 🤔 (Leave your thoughts in the comments)

However, the approach is less flexible since the users can’t navigate and jump between the different pages.

So, you now know how Spotify and X handle pagination — but designing your own API isn’t just about picking a technique. Let’s look at some best practices that make pagination efficient, scalable, and easy to maintain.

API Pagination Best Practices

Here are some common best practices to follow:

• Make the cursor opaque - The server must either encode or encrypt the cursor. This hides the implementation details and prevents clients from reconstructing the cursors.

• Validation on the page size - The server must validate LIMIT, OFFSET, COUNT set by the client. It must return appropriate response (400 Bad Request) in case of validation failures. This prevents the clients from misusing the server resources.

• Mandatory for all List operations - Use pagination for all list operations to handle large datasets efficiently.

• Caching - Caching prevents the load on the database and speeds up the reads. Wherever needed, caching can be used to improve the performance. However, cache invalidation also needs to be addressed in case the data changes frequently.

• Monitor pagination usage - Monitor the frequently requested page size, common cursors, % errors, etc. This helps you understand the client’s request patterns and tune the pagination parameters.

Conclusion

API Pagination enables us to divide a large dataset into small chunks and return it to the clients. We learnt about two techniques - Limit-offset and Cursor-based Pagination.

Both the approaches have their pros/cons and we must use the approach that’s effective for the problem being solved.

The following table summarises the different use cases and the right pagination technique to use.

Now that you understand pagination, do you think Search engines like Google use LIMIT-OFFSET based pagination to show the results?🤔 Let me know your thoughts in the comments below. 👇

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

A small misconfigured value led to a huge spike in the database load. While the mitigation was simple, it worsened the problem. The only way to solve the problem was to deliberately turn off the website traffic.

But why turn off the traffic? Weren’t there any other options available?

This article will answer the above questions in detail. We will understand in simple words the root cause of the issue. The article will also recommend strategies to avoid such massive outages in the future.

With that, let’s begin with background of Facebook’s architecture.

Subscribe now

Facebook’s architecture

By 2010, Facebook had evolved beyond simple social sharing, adding chat, ads, photo support, and more.

With thousands of engineers, the software architecture was complex and comprised of several services. But at its core, it was a simple client-server-database model like any other website.

For performance and scalability, they used caching heavily and majority of the requests were served through the cache. The services fetched the data from the database in case of cache misses.

The following diagram illustrates the high-level architecture:

Disclaimer: The above diagram is only for representative purpose and doesn’t reflect the actual state of Facebook’s production architecture.

One of the caching layer served configuration values. The configuration values often changed due to deployments or developer changes.

So, it was essential to replace the values periodically. They devised an automated mechanism to replace the cache values.

Let’s now understand this automated mechanism in detail.

Ready to analyze complex system failures like a senior engineer? Learn the system design patterns that prevent outages and impress interviewers at top tech companies.

Look no further than Educative.io’s course on Grokking the System Design Interview. You can use this link to get an additional 10% off.

Automated verification

They created an offline verification service that periodically:-

1. Fetched values from the cache.

2. Verified the configuration values and checked for the expiry, correctness, database consistency etc.

3. In case of validation failures, it fetched the data from the database and replaced the cache value.

The following diagram illustrates how the automated verification system worked.

This mechanism prevented simultaneous database calls, avoiding bottlenecks and keeping the system smooth.

However, if the service couldn’t fetch the value from the cache, it got the value from the database and replaced the cache value. This ensured safe fallback mechanism.

The system worked seamlessly and handled traffic at Facebook’s scale until the day of the outage.

Let’s now learn what exactly happened on the day of the outage

Outage

One of the teams changed the configuration value in the database. However, the new value was invalid (For eg:- malformed json or incorrect value).

While they had designed the system to robustly handle invalid cache values, it wasn’t built to handle invalid database values. So, here’s what happened later:-

1. Automated verification detected inconsistency and fetched the database value.

2. Since the fetched value was invalid, it deleted the cache entry and replaced it.

3. When the services fetched the cached value, it resulted in a cache miss. Hence, they tried fetching the value from the database.

4. The database load spiked since multiple services simultaneously performed database calls.

The following diagram summarises the outage:

Outage mitigation

Engineers first attempted a quick solution—correcting the database value.

But did this simple fix work? The answer is No. Let’s understand why.

Once the database value was fixed, here’s what happened:-

1. It was already handling a huge load.

2. Few services fetched the correct value and fixed the cache.

3. However, due to the load, many services still couldn’t fetch the database value. They retried multiple times and it further increased the load.

4. The service interpreted database call failures as invalid values. As a result, they deleted the cache values again and ended up calling the database.

The below diagram illustrates the vicious cycle that escalated the issue:

The system was collapsing onto itself and couldn’t gracefully recover. As a result, the only solution was to turn off the user traffic to the website leading to a downtime of more than 2.5 hours.

Massive outages of this scale are not caused due to a single component. Root cause analysis reveals gaps in the functioning of several components.

We will now dissect the root cause behind Facebook’s outage.

Root cause analysis

Automated verification system

The system didn’t differentiate between cached value and the database value. It deleted the cache entry due to invalid database value triggering a stream of database queries.

Service error handling

The service interpreted database timeouts and other errors as invalid values. As a result, it also deleted the valid entries from the cache further escalating the issue.

Lack of rate-limiting

There was no throttling mechanism to prevent a surge of database queries. This eventually led to the vicious cycle and the system couldn’t self-heal itself.

Now, that you understand the root cause of the outage, let’s understand mechanisms that could have prevented it.

Preventive measures

Though this outage happened 15 years ago, its lessons remain relevant—any tech company could face a similar risk.

Following are some mechanisms to prevent such outages:-

1. Rate limiting - Limit the database calls to protect the database from a surge. This prevents database from being a bottleneck and reduces the blast radius of the issue.

2. Retry mechanisms and jitter - Adding random delays, increasing delays after failures, etc help to smooth the overall load.

3. Error handling mechanisms - Differentiating system issues such as timeouts differently and not mixing business concerns (treating it as invalid value) result in robust error handling.

4. Configuration design - Instead of validating config values after reading, validation at the write time prevents such issues.

5. Kill switches - Kill switches give more control to the engineers while operating a system. They are useful in handling worst case scenarios and reducing the outage’s overall impact.

Conclusion

Facebook’s outage offers valuable lessons for the industry even today. In today’s fast paced world, a downtime of few minutes implies huge losses to the business.

While outages can’t be eliminated completely, their duration and customer impact can be reduced through preventive measures described in this article.

Adopting preventive mechanisms and best practices help:-

1. Improve developer productivity.

2. Keep the customers happy.

3. Reduce the business impact due to future outages.

What is the most interesting outage that you have worked on so far? What did you learn from it? Leave your thoughts in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Disclaimer: The details in this post have been derived from the official blog shared online by the Facebook Engineering Team. Kudos to the Facebook Engineering Team for mitigating the outage and bringing up the website within a short duration. The links to the original blog post is included in the next section. We’ve attempted to explain the outage in simple words and the key lessons for the engineering audience. In case there are any inaccuracies, please highlight them in the comment section below and we will do our best to rectify the same.

Reference

1. Facebook’s 2010 outage

Software teams constantly change, and a wrong decision can impact future team members—especially those with the least context. I have worked on both sides of the table — building a service by deprecating an old library and replacing a service with a lightweight library.

I worked at a mid-sized startup where microservices were a golden hammer. They built microservices for the smallest of the functionalities. The loosely coupled architecture did create a mess. I remember a colleague once joked, “They’ll probably create a service just for Java’s toString() function!” 😁

Finally, the company took a call to combine the related functionalities in a library. The rewrite frustrated the developers who always pondered - “Why didn’t the previous team think of this?“ 🤔

On the other hand, while working in one of the big tech teams, we ran into scalability challenges and bottlenecks with a legacy library. Our team members blamed the old library owners for the poor design.

One common pattern in both the anecdotes was - The old team members always became a scapegoat. 😁

How can we prevent this? How do we take better decision as software developers? How do we build software that outlives the team that built it?

This article will answer all of the above questions. It will walkthrough a case study of how Netflix substituted a library with a service. You will learn how to take critical decisions and future proof your tech choices.

Subscribe now

With that, let’s begin with the case study.

Netflix Case Study - Platform for managing Membership plans

Today, Netflix has presence in 190 countries and has a 300 million subscriber base. It offers a variety of plans tailored to devices, streaming quality, regions, and more.

But back in 2010, it only had presence in the US with just 12 million subscribers. It offered only three plans - Basic, Premium and Standard to the customers.

Let’s now understand the software behind the membership plans.

Do you want to level-up your skills as a software developer? Do you aspire to crack interviews at top-tech companies and land a job?

If yes, then Educative.io is the best platform with handcrafted courses for developers and personalized by AI. Get additional 10% off by using this link.

Membership plans library

In 2010, Netflix had a monolith (Membership service) that was responsible for managing and tracking the user subscription plans. For modularity, they decided to build a library that would:

1. Handle the business logic for evaluating the plans.

2. Store the plan metadata in various configuration files.

The below diagram shows the basic data model for the different membership plans:

Stakeholders could easily modify the plans, release and deploy the library. The below diagram shows how the service read the plans from the library.

Everything was simple until the company’s global expansion in the mid-2010.

Global expansion

It first launched in Canada followed by Latin America, United Kingdom, Australia, India, etc. With the expansion, there was a growth in the number of subscribers.

For scalability, the tech teams decided to split the monolith into multiple services. Each service served a different purpose such as Playback, Billing, Edge services, etc.

Its initial strategy of offering three basic plans wasn’t sufficient for the new regions. They needed better strategies to acquire customers.

The subscription plans grew in the following dimensions:-

1. Devices - Plans catered for specific devices such as Mobile, TV, etc.

2. Quality of service - Different plans based on the resolution and audio quality.

3. Regions - People’s buying choices led to unique plans for various regions.

In addition, product team wanted to perform A/B tests on different subscription plans and evaluate the impact.

As the company grew, the original light-weight library now had complex rules and plan metadata. With the proliferation of microservices, the library was imported by every other service. Although the company grew rapidly, the library couldn’t keep pace with it.

The below diagram presents the membership plans data model after Netflix’s geographical expansion.

Let’s now dive into the challenges that the team faced while managing the library.

Library challenges

The library owners faced the following challenges:-

1. Operational complexity - Plan changes required changes in metadata config and business logic. The development and testing took several days. Additionally, even simple changes took days to reach production.

2. Reliability - Pricing consistency was critical for Netflix across devices and pages showing the subscription plans. With the library, two or more services could rely on different versions resulting in inconsistency in the shown plans.

3. Manageability - It became difficult to manage thousands of different plans along with custom business logic.

One of the biggest challenges of using a library is maintaining version consistency across services. Netflix team also wrote a web crawler at scale for version correctness.

The below diagram shows how different services and their instances use different library versions.

The tight-coupling due to the library resulted in less flexibility, increased blast radius, and long dev cycles.

We will now understand how Netflix tackled these challenges.

Library deprecation

To solve the pain-points, the team decided to centralize the logic in a service. The primary goal of the platform was to provide an API interface that abstracted the business rules and data.

Here’s how the library got refactored into a platform:-

• Business rules - A rule engine was developed to manage the complex rules. Rules were written in the form of expressions instead of code.

• Persistence - The data was migrated from metadata configurations to a database layer for persistence. The database acted as a single source of truth for all the plans.

• Self-service UI - This layer enabled stakeholders to easily configure new plans, create and validate rules. This eliminated the need to make library code changes that took weeks of efforts.

• Service layer - It provided APIs different services to fetch the plans based on the context. For eg:- A service could pass region, device, userId in the context and it could return the plans after executing the business rules.

The below diagram shows how rules were written in the library and then transformed into rule engine expressions.

It can be seen from above, how easy it is to define the rules in the form of expressions than writing business logic every time a plan changes.

The following diagram shows the high-level architecture of the platform.

The re-architecture was a multi-year effort and involved more than 20 different engineering teams. The new architecture could now scale and keep up with Netflix’s expansion in different regions.

While the effort was worth it, do you think Netflix made a wrong choice in developing a library first? 🤔

Was Netflix wrong?

Now, that you understand Netflix’s current and old architecture, let’s assess whether Netflix made a wrong choice in 2010s.

Had Netflix built a Plan service platform instead of the library, it would have supported the company’s rapid growth. It would have given more flexibility to the product teams to experiment and iterate fast.

But what would have happened had Netflix not grown the way it grew in 2010s? 🤔

Assuming that Netflix existed only in US with no business growth, the membership platform would have:-

1. Increased the infrastructure costs.

2. Been an overkill for a simple use case.

3. Resulted in most of the functionality remaining unused.

So, Netflix wasn’t wrong in choosing a library in the first place. But they didn’t tackle the library’s tech debt at the right moment which resulted in multi-year re-architecture.

It’s very easy to criticize the previous software teams for the current challenges. But, we must respect the past engineering decisions assuming they were taken rationally with certain assumptions.

Netflix could have avoided the multi-year migration effort by:-

1. Identifying the issues with the library proactively.

2. Prioritising the tech debt and reducing operational burden.

3. Future-proofing the tech choice of library by easily evolving it into a service.

A better approach would have been to understand the library pain-points after expanding in two or three regions. A futuristic view of growing requirements would have helped them tackle the problem effectively.

Library vs Service

While deciding between a library and a service, you can use the following table to weigh the pros and cons.

It’s essential to evaluate the effectiveness of either approaches in production. A proactive approach saves efforts and costs in future.

Here’s how you can evaluate the success of the solution:-

1. Consumer count - Service is suited for large number of consumers. If your library is being adopted by large number of consumers (> 5), it’s sign to move the logic to a service.

2. Frequency of changes - Library results in less flexibility. Hence, if the code or config is changed frequently (> 10 changes/month), it may indicate need for a service.

3. Performance implications - If you have tight performance constraints, service may not be the right solution. For eg:- A 10ms delay results in $100K loss. In such cases, you need to make trade-offs and adopt a library based approach.

4. Blast radius - If changes impact multiple upstream consumers and affect critical business decisions (dollar value impact), then a centralized service is appropriate for the use case.

The above metrics would help the teams decide when to pivot from one solution to another (library to service or other way round). Additionally, to future-proof technology choices, teams must build architecture that evolves seamlessly.

Let’s understand with the help of an example, how teams can design an evolvable architecture.

Library to service migration

If a team decides to develop a library, it must anticipate future business growth and challenges. They must ensure that the transition is smooth and doesn’t result in multi-year effort.

Here’s how the problem can be tackled:-

• API facade - Introduce a facade that mimics the future service’s API.

• Client migration - Migrate the clients in a controlled manner to the internal API endpoints.

• Library logic in service - The service must run the same library logic.

• Fault tolerance - Include retries, monitoring and fallbacks to protect the clients.

Let’s go through a code example to understand this in detail. We will take Netflix’s example of fetching the plans.

The below code is used to fetch the membership plans.

# plan_selector.py (the current library)

def get_available_plans(user_country: str, is_mobile_enabled: bool) -> list:

    if user_country in ["IN", "TH"] and is_mobile_enabled:

        return ["Mobile", "Premium"]

    return ["Basic", "Standard", "Premium"]

Now, we will create a facade module that would:

• Wrap the current library.

• Mimic the API of the planned HTTP/gRPC service (inputs and outputs).

• Allow switching between library and service backends in the future.

# plan_facade.py

from plan_selector import get_available_plans
# from requests import post  # Future: for HTTP call to service

USE_REMOTE_SERVICE = False  # Toggle this when ready to switch

def fetch_plans(user_context: dict) -> dict:
    """
    Mimics the API structure of the planned service.
    Input: { country: "IN", features: { mobile_enabled: true } }
    Output: { plans: [...] }
    """
    if USE_REMOTE_SERVICE:
        # Future call to the remote Plan service
        # response = post("http://plan-service/api/v1/plans", json=user_context)
        # return response.json()
        raise NotImplementedError("Remote service not yet implemented")
    else:
        # Local fallback using the library
        country = user_context.get("country")
        mobile_enabled = user_context.get("features", {}).get("mobile_enabled", False)
        plans = get_available_plans(country, mobile_enabled)
        return {"plans": plans}

This approach ensures a smooth transition from a library to a service.

How would the same work if you want to replace a service with a library? Think about it and share your thoughts in the comments.

Conclusion

Netflix’s case study teaches us that decisions are not inherently right or wrong. But what’s wrong is turning a blind eye to the tech debt and not thinking strategically.

Few companies may witness exponential growth while many others may not. However, this shouldn’t stop the developers from deciding between a library and a service.

A more pragmatic approach is to make your tech choice future-proof by pivoting at the right time. It’s critical to keep an eye on signals such as tech debt and operational burden.

For early-stage companies or teams, it makes sense to invest in a library. But develop layers of abstraction to easily convert it into a service. It’s important to be mindful of the growth and not fall into the trap of thinking tactically.

Have you worked on a refactor or a rewrite similar to Netflix? If yes, what did you learn from it? Leave your thoughts in the comments below.

References

1. Netflix Membership SKUs platform

2. Managing 238M memberships at Netflix

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

The secret sauce behind NGINX’s scalability and performance is the event-driven non-blocking I/O architecture. It seamlessly handles growing traffic and efficiently utilizes the CPU and memory resources.

In the early 2000s, NGINX was one of the first web servers that was designed for scalability, performance and concurrent connections. The architecture has inspired many other web servers, reverse proxies such as Pingora, CDNs, and cloud load balancers.

In this article, we will learn how NGINX efficiently scales to handle millions of concurrent connections. We will explore different approaches and understand their limitations. We will then learn how NGINX architecture solves the scalability and performance challenges.

With that, let’s begin with some basics of web services and why we need NGINX.

Subscribe now

What does NGINX do ?

NGINX acts as an intermediary between the client and the web services. It handles the client requests and routes it to the backend web service.It is also known as a reverse proxy and can load balance the request among multiple backend servers.

Further, it performs all infra-related heavy lifting such as SSL termination, connection management, rate limiting, etc. The backend web services can efficiently scale since the infra logic is centralized in NGINX.

The following diagram shows the interaction between a client and the server using NGINX:

While NGINX solves most of the problems with traditional web service architecture, it still needs to solve:

1. Concurrent connections - Large number of concurrent connections from clients.

2. Performance - No performance degradation with user growth.

3. Efficient resource utilization - Low memory usage and optimal CPU utilization.

Before diving into the solution, let’s revisit connection management basics and understand the scalability bottlenecks.

How are the connections handled ?

When a web server starts, it calls the operating System and passes the port on which it listens. For e.g., Web servers would pass port 80 (http) or 443 (https) to listen.

When the client connects, the OS’s kernel stack performs a TCP handshake and establishes a connection. The OS assigns a file descriptor or a socket for each connection.

The below diagram illustrates the connection establishment between the client and the server:

By default, sending and receiving data over a network (Network I/O) is blocking. A thread or a process goes into waiting state while writing or reading data to/from the network.

Also, the network I/O is dependent on the client’s bandwidth. Data transfer may take a long time for slow clients.

The following diagram shows how a process waits until the complete data transfer:

As a result, the server can’t accept new connections if it’s already processing request from a client. This hinders the system’s scalability and performance both.

There are several ways to tackle this problem and handle more connections. Let’s understand the different approaches and their limitations.

Process-Per-Request Approach

To overcome the network I/O bottleneck, the process can fork a new child process. The child process would then handle a new client connection.

Every connection would correspond to a new child process. Once the request/response cycle is completed, the child process would be killed.

The below diagram illustrates this process:

Do you think this approach would scale to millions of users/connections ? Take a moment to think and then continue reading

Let’s assume the server RAM size is 32 GB and each process takes 100 MB. So, then it can handle only 320 (32 GB/ 100 MB) connections in the best case.

Here are some downsides of this approach:

1. Scalability Issues - Number of connections depend on the hardware (RAM size). More connections would lead to out of memory issues.

2. Performance Issues - Forking a child process is slow and would impact the performance.

Can we do better? What if instead of forking a process, we launch a thread? Let’s explore this approach in the next section.

Thread-Per-Request Approach

In this approach, a thread is launched every time a client connection is established. Each request is handled independently by a different thread.

The below diagram shows how this model works:

Threads are lightweight and almost 1/10th size of a process. As a result, this is a significant improvement from the Process-Per-Request approach.

While this approach can handle more number of connections, it would still run into issues highlighted in the previous section.

A process can’t launch an infinite number of threads. The benefits of multi-threading diminish with large number of threads due to frequent CPU context switching.

We can still improve by using a thread pool and launching a fixed number of threads. For eg:- 500 threads in the process.

This improvement would result in efficient memory usage. However, if all the threads are busy, the new connections would wait in the request queue resulting in slowness.

Hence, this approach also doesn’t solve for scalability and performance. We can’t scale since the primary bottleneck is the time-consuming network I/O.

Is there a way to unblock the process or thread during the network I/O? Yes, and NGINX employs an intelligent tactic using its event-driven non-blocking I/O.

Let’s understand NGINX’s architecture in detail in the next section.

NGINX Architecture

NGINX uses a modular architecture and consists of several components such as:

1. Master process - It acts as the central controller and is responsible for starting, stopping, and launching the worker processes.

2. Worker processes - These run the core NGINX logic and are responsible for connection handling, request forwarding, load balancing, etc.

3. Cache loader - This process is responsible for loading the cached metadata in memory when Nginx boots up.

4. Cache manager - It periodically checks the cache directory and frees up space by removing expired entries.

5. Shared memory - The inter-process communication takes place via the shared memory. It’s also used for caching, and managing shared state like load balancing status.

The following diagram shows the different components of NGINX architecture.

Let’s now dive into the details of how NGINX can scale to million concurrent connections.

Event-driven Non-blocking I/O

In case of non-blocking I/O, the web server or the application doesn’t wait for the client’s data. Instead, the OS informs the application once the data is available.

This makes the process event-driven. Whenever the client’s data is available, the application would get interrupted and it would process the data. Otherwise, it would continue to do something else.

Further, the application doesn’t go into a waiting state. It can execute other tasks and efficiently utilize the CPU.

Internally, the application uses a system call called epoll or kqueue and then registers the sockets. The operating system uses a kernel data structure (Epoll instance) to keep track of the sockets that an application is interested in.

Once data is available in a subset of sockets, those sockets are moved into a ready list. The OS then informs the application about those sockets. Finally, the application then processes the data.

The below diagram illustrates this flow:

As seen from the above diagram, once data becomes available on fd3, and fd4, the process is notified by the OS.

Let’s now understand this in the context of a NGINX worker.

NGINX worker

Each NGINX worker is single-threaded and it runs an event loop. The event loop works like a while loop and checks for any activity on the socket or new connections.

Further, the NGINX worker is pinned to a CPU to avoid context switches. This eliminates the context switching overhead and improves the performance.

Here’s the pseudo code that explains what happens behind the scenes in a worker:

while True:
    # Wait for events on the monitored sockets
    events = epoll_wait(epoll_fd, MAX_EVENTS, timeout=-1)  # Block indefinitely
    if events == -1:
        raise Exception("epoll_wait failed")

    # Process each event
    for event in events:
        fd = event.data.fd

        # Case 1: New connection on the server socket
        if fd == server_socket:
            # Accept the new connection
        # Case 2: Data available on a client socket
        elif event.events & EPOLLIN:
            # Read data from the client socket
            data = client_socket.recv(BUFFER_SIZE)
            if not data:  # Client closed the connection
                epoll_ctl(epoll_fd, EPOLL_CTL_DEL, client_socket, None)
                client_socket.close()
                del client_sockets[client_socket]
            else:
                # Append data to the client's buffer
                client_sockets[client_socket]["buffer"] += data

                 # Process the request (takes ~100s of microseconds)
                 request = client_sockets[client_socket]["buffer"]
                 response = process_request(request)  # Assume this function exists

                 # Send the response
                 client_socket.sendall(response)

        # Case 3: Client socket ready for writing (if needed)
        elif event.events & EPOLLOUT:
            # Handle writing data to the client (if applicable)
            pass

With non-blocking sockets, the worker doesn’t need to wait till the data is completely sent to the client. It can quickly move onto the next connection and process the request. (Case-1)

In case the data is available on multiple sockets, it iterates on each and process the client request. (Case-2)

Since network I/O is non-blocking, the process doesn’t wait for the data transfer. And the worker uses CPU only for request parsing, filtering, and other compute operations.

Compute operations are less time-taking (in order of micro-seconds). As a result, a single worker can process 100K requests every second concurrently.

Assuming that a single worker can handle 100K connections, if it’s a 10-core CPU, the server can handle 1 million concurrent connections. (Example for illustration only, in real world, things might be different).

Note: A server must have sufficient memory to serve 1 million connections since each connection needs 100KB-1MB memory. But the OS kernel can be tuned to reduce the connection’s memory.(there are trade-offs to this approach)

The event-driven non-blocking I/O efficiently utilizes the CPU and doesn’t consume memory like Process-Per-Request or Thread-Per-Request approach.

Conclusion

A key challenge in building scalable web architecture is to handle simultaneous concurrent client connections. Also, the web service needs to handle concurrent requests and efficiently utilize CPU and memory resources.

NGINX uses the event-driven non-blocking I/O architecture and tackles the following challenges :-

1. Concurrent connections - It uses several workers to handle client connections. Non-blocking I/O allows it to accept new connections while already processing other connections.

2. Performance - Each worker is single threaded and pinned to a CPU. This greatly improves the throughput and performance due to no context switches.

3. Efficient resource utilization - Instead of launching new process or a thread for each connection, a single worker is able to manage multiple connections. The only overhead if a connection’s memory usage. Further, due to event-driven nature, the worker doesn’t go into waiting state frequently. This improves the CPU utilization.

Before you leave, what do you think would happen if the server running NGINX crashed? Would NGINX become a single point of failure? 🤔

Share your thoughts in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

References

1. NGINX architecture

2. NGINX epoll module

Redis is widely known as an in-memory cache, but can it also serve as a durable storage system like a traditional database?

Like many developers, I initially assumed Redis was just a cache. But as I explored its persistence mechanisms, I realized it could offer more.

Performance and durability are two key considerations in designing Redis’s data persistence. If the system prioritizes performance, it must compromise on durability, and vice versa.

Subscribe now

In this article, we will dive deep into the different data persistence strategies of Redis. We will look at the pros/cons of each and their practical applications.

By the end of the article, you will gain sufficient technical depth to decide the right persistence strategy for your use case. You will also enhance your skills to succeed in your next system design interview.

With that, let’s first understand the need for data persistence.

Why does Redis need data persistence ?

Redis is an in-memory cache and stores all the data in the RAM. RAM is volatile, and if the server crashes or restarts, all the data is lost.

Assume that you are using Redis as a cache for a backend database. What would happen if it crashed mid-way while handling customer traffic ?

The following diagram illustrates the state of the application before and after Redis crash.

Here’s what would happen after a Redis crash:

1. High database load - All the cache requests would fallback on the database increasing its load. Often, this is known as the thundering herd problem.

2. Slow queries - Since database fetches the data from the disk, the query latency would spike by 100x.

3. Reduced availability - Some slow queries might time out, leading to failed application requests and reduced system availability (e.g., from 99.99% to 95%).

After the Redis server restarts, there would be a brief period where the data would be written to the cache, and eventually, the system would function as expected.

As we just saw, there is a possibility of complete data loss in case a Redis instance crashes. Additionally, this impacts the application's performance and results in a poor user or client experience.

So, is there a way to tackle this problem by permanently storing the data somewhere ? Yes, and Redis provides several strategies to solve this problem.

Let’s now take a look at the first strategy called Append-Only File (AOF) for persistence.

Append-Only File (AOF)

Redis clients interact with Redis by running different commands. For example: SET key value, INCR key, LPUSH key value, etc.

Redis is single-threaded by design and sequentially executes all the commands. The command execution modifies the in-memory data structures, and we end up with a particular state.

The Append-Only File (AOF) persistence strategy exploits this and stores all the commands in an append-only file. On server restarts, the AOF is read and all commands are sequentially executed to recover the previous state.

The following diagram illustrates how Redis logs commands in the AOF file.

The diagram below shows how Redis reconstructs the previous state by reading the AOF file.

Why are commands added to the AOF after execution, not before? 🤔 🤔

Leave your thoughts in the comments

We just saw that we can completely shield Redis from data loss by using the AOF technique. But astute readers would have noticed that we are logging every command to disk and that might increase the I/O. This would eventually slow down Redis’s main thread.

Although the AOF technique can prevent complete data loss, it comes with an additional cost that impacts performance. Let’s now look at the pros/cons of this approach.

Pros

1. Data durability - It can reconstruct the previous state and result in 0 data loss.

Cons

1. Performance - Since every command is written to a disk, there is an additional disk I/O. This impacts the performance of each command and might introduce an additional 100-200 ms latency. This prevents clients from using Redis as a high-performance, in-memory cache.

2. Slow starts - The AOF might contain billions of commands. Given the single-threaded nature and sequential execution, complete recovery may take anywhere between 100 milliseconds to 1-2 seconds.

Logging every command to the disk impacts the performance. As a result, Redis provides multiple configuration options to tune performance and durability.

Configuration options

Redis allows tuning AOF behaviour with three configurations, each offering a trade-off between performance and durability.

Internally, Redis uses the fsync system call to flush executed commands to the AOF. It provides the following three options :-

1. appendfsync always - It logs all the executed commands to the AOF. It takes time but guarantees strong durability.

2. appendfsync everysec - Executed commands are appended to a buffer, which is flushed to the AOF every second. In the worst case, Redis might lose one second of new data.

3. appendfsync no - Instead of application flushing the data, the flushing is controlled by the Operating System. Depending on the kernel, the frequency could be anywhere between 30 sec - 60 sec.

The below matrix captures the three different options along with the impact on the system’s quality attributes.

While AOF minimizes data loss, it has a performance penalty. Thus, Redis provides another option known as RDB (Redis Database).

Let’s understand how RDB tries to overcome the downsides of AOF.

Redis Database (RDB)

Redis takes a snapshot of all the in-memory data and stores it in a binary file (.rdb suffix). In case of a crash, it reads and recovers the data from the rdb file.

It provides the commands SAVE and BGSAVE (background save) to take a snapshot. Here’s the difference between the two commands:

1. SAVE - It’s sequentially executed in Redis’s main thread. As a result, it impacts the cache’s performance due to additional I/O overhead.

2. BGSAVE - It forks a new child process. The child process executes on a different CPU core and copies the data into the RDB file.

Redis crash recovery process using RDB is illustrated in the following diagram.

What would happen if the memory is modified during snapshotting ? 🤔 🤔

Leave your thoughts in the comments

Now that you understand the basic working, let’s look at some pros/cons of this approach.

Pros

1. Performance - The snapshotting process can be executed on a different thread (using BGSAVE). As a result, it doesn’t impact the overall performance of Redis.

2. Restart time - The RDB file data can be read quickly and restored. This reduces the restart time and overhead in data recovery.

Cons

1. Data loss - It doesn’t guarantee 0% data loss. As a result, clients may get stale data or result in cache misses.

2. Large datasets - Since fork() is used, the process can be time-consuming for large datasets. This may impact the client’s performance in case Redis stop serving the clients till the child process is successfully created.

So far, we have seen that while AOF prioritizes data durability at the cost of performance, RDB sacrifices some durability in favor of better performance.

Given the trade-offs of AOF and RDB, is there a way to achieve a middle ground? Fortunately, Redis offers a hybrid approach: AOF + RDB.

We can combine RDB with AOF and strike the right balance between the two approaches. Let’s now understand how we can get best of both the worlds.

AOF + RDB

Redis provides a configuration option aof-use-rdb-preamble. If it is enabled, Redis takes periodic snapshots and stores the subsequent commands in the AOF file.

The diagram below illustrates the working of AOF + RDB.

Through snapshotting, it eliminates the need to execute all the commands from AOF and only uses the commands executed after snapshotting. This reduces the startup time and also AOF size.

Further, it minimizes the data loss by logging the executed commands in AOF. Unlike RDB, it doesn’t lose the commands executed after snapshotting.

The following are some pros/cons of this approach :-

Pros

1. Data durability - Data loss depends on the frequency of executing the fsync. Increasing the frequency would lead to less data loss but impact performance.

2. Startup time - Since it reads the snapshot and then executes commands in AOF, less time is taken during startup as compared to AOF only approach.

3. Performance - Performance is better since it depends on fsync frequency. Reducing the frequency improves performance (impacts durability).

Cons

1. Complexity - Loading the rdb snapshot and then applying the AOF commands adds complexity to the process.

Now that you have a good understanding of Redis persistence, let’s compare the different approaches and look at practical applications of each.

AOF vs RDB vs AOF + RDB

The following table summarizes the strengths and weaknesses of each of the three approaches.

Let’s now look at the practical applications of each of the approaches.

AOF

AOF is best suited for applications that require strong durability guarantees. Following are some common use cases :-

1. Financial transactions - Banking systems, e-commerce payment services, etc.

2. Chat applications - WhatsApp, Discord, Slack, etc.

3. Stock market applications - Order books for stock market. Complete state can be reconstructed without data loss.

RDB

RDB enables quick data recovery and minimal performance impact. Also, use cases that don’t require strong guarantees can opt for RDB. Here are some common practical examples :-

1. Web application caching - CDN caching, API rate limiting, etc

2. Gaming leaderboards - Online leaderboards showing stats

AOF + RDB

This approach strikes the right balance between performance and durability. Additionally, it speeds up the boot time resulting in better performance than AOF. It can be used for the below use cases :-

1. Real-time analytics and fraud detection

2. E-commerce product inventory

3. Microservices communication

Conclusion

Redis stores the data in-memory and risks data loss due to crash or restarts. It provides several approaches to prevent data loss and recover after crash.

The Append-Only File (AOF) uses a log file that contains list of commands executed by Redis sequentially. In case Redis restarts, it reads the file and executes all the commands to recover the full state.

AOF ensures 0% data loss but impacts the performance. To overcome this challenge, a data persistence technique called RDB (Redis Database) is used.

Redis takes a snapshot of its memory and stores it in a .rdb binary file. After crash recover, Redis recovers the old state by reading the binary file. However, the data durability is compromised since commands executed after the snapshot aren’t stored.

To overcome the performance and durability extremes of the two approaches, AOF and RDB can be combined. This helps balance the performance and durability trade-offs.

Now that you have understood the data persistence concepts, which technique would you use to convert Redis into a permanent database ? Leave your thoughts in the comments below.

Leave your answers to the above question in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Similarly, social media products like Instagram, LinkedIn render newsfeed under 2 seconds. The newsfeed contains data such as posts, likes, comments, reposts, etc.

The above use cases fetch data from multiple services and then show them in a single view. They use a distributed design pattern known as Scatter-Gather to solve the problem.

In this article, we will understand what a Scatter-Gather pattern is. We will walk through some real-world examples of this pattern. Further, we will also explore the relevance of this pattern in designing distributed systems.

With that, let’s understand the problem we are trying to solve.

Subscribe now

Problem Statement

Design product detail page for an e-commerce website. The backend takes the productId as the input and displays a product detail page.

Also, assume that the website supports millions of users and sells billions of products.

The product detail page consists of the following :-

1. Product information - price, name of the product, description, etc.

2. Promotions/Offers - Any relevant promotions that the seller is providing.

3. Shipping details - By when the product would be delivered.

4. Reviews - Reviews posted by other buyers.

5. Warranty details - Nature of the warranty, duration, etc.

Given the above requirements, let’s try to design a simple system that meets the requirements.

We can use a microservices-based architecture for the above website. The microservices would be assigned different responsibilities and expose APIs for fetching the data.

We would have a frontend server that would talk to the different microservices and get the data. Later, it would construct the page and feed all the data into it. Finally, the page would get rendered on the client device.

The below diagram shows the system architecture and the overall process. The frontend server calls the microservices sequentially and then sends the client response.

Do you see any downsides to the above approach? Take a moment to think and then continue reading.

Here are some downsides of the current approach :-

1. Latency - Grows linearly with the number of microservices. If each service takes 1 sec, then the page would take at least 5 secs (5 services, so 1 sec x 5) to load. If there are 50 services, then it would become 50 secs.

2. Efficiency - When the frontend server calls one service, it waits for the response and wastes CPU cycles. This amplifies with the increase in number of backend services.

3. Fault-tolerance - The approach introduces dependency between multiple operations. In case a single API fails, it would cause the whole request to fail.

To overcome the above limitations, we use the Scatter-Gather pattern. Let’s understand the concept in detail and how it can solve the current problem.

Scatter-Gather pattern

As the name suggests, the pattern achieves the following :-

1. Scatter - Scatters the requests across multiple backend servers in parallel.

2. Gather - Gathers or aggregates the data received from the same servers.

The below diagram provides a simple explanation of Scatter-Gather pattern.

Let’s see how we can apply this pattern to the problem in the previous section.

We will make the following changes in the previous design :-

1. Parallel processing - Instead of sending the request sequentially, it would be sent in parallel to all the backend servers.

2. Aggregation - The frontend server would wait for the response from all the servers. Once received, it would aggregate and construct the client response.

The following diagram shows how scatter-gather can be leveraged to solve the problem.

As seen in the above diagram, the product detail page is fetched in 3 steps. The second step fans out parallel requests to multiple backend services.

Here’s how we overcame the limitations of the previous solution through Scatter-Gather:

1. Latency - Since the requests are sent in parallel, it wouldn’t linearly grow with the server growth. If there are 50 services, we would still get the response within 1 sec (on average).

2. Efficiency - Through parallelism, we no longer waste CPU cycles waiting for the backend response. This improves the overall throughput.

3. Fault-tolerance - Failures can be handled independently and there is no tight dependency between the services. Also, individual API retries don’t add up to the overall latency due to parallel processing.

Now that you understand the concept of Scatter-Gather, let’s see a few real-world applications of this pattern.

Real-world Application - DynamoDB

AWS DynamoDB stores the data internally on multiple storage servers through sharding. It uses Consistent-Hashing to determine the server to store the data on.

When the client requests multiple keys that reside on different storage servers, DynamoDB uses Scatter-Gather for serving the data.

The below diagram illustrates an example of how DynamoDB organizes its data and leverages Scatter-Gather.

Let’s assume that the client searches for three keys - id1, id2 and id3. Here’s how DynamoDB uses Scatter-Gather behind the scenes :-

1. DynamoDB front-end server parses the input consisting of the three keys.

2. The front-end server hashes each of the keys.

3. It determines the corresponding backend server for the key. (id1 - server1, id2 - server2, id3 - server3)

4. It sends a parallel request to each of the servers and performs a lookup.

5. Finally, the three servers send the data corresponding to the keys.

6. The front-end server aggregates the data and sends it in the response.

A similar pattern is used by other NoSQL databases such as Cassandra, MongoDB, Elasticsearch, etc.

Challenges in implementing Scatter-Gather

While Scatter-Gather pattern helps in building and scaling large-scale systems, it has its own set of challenges. Here are some challenges in implementing the pattern :-

• Failure handling - It’s essential to handle the failures gracefully. Since multiple services are involved, hardware failures, deployments, etc are common. These must be overcome through redundancy and replication.

• Data accuracy - Backend services might use different data sources and present inaccurate data. It’s essential to resolve such discrepancies before sending the client response.

• Number of backend servers - We can’t infinitely increase the backend servers. Each request incurs a network call, parsing operations, and additional overhead. With more backend servers, the additional overhead reduces the benefits of parallel processing. So, we need to choose the backend servers wisely.

• Incomplete data - A backend service might fail to return data. In such cases, the system must use a fallback mechanism for the incomplete data. For eg:- In case of an e-commerce product, it can show default promotions if the Promotion service fails.

• Latency - The overall latency is influenced by the slowest backend server. It’s essential to define an upper bound on the latency (SLA) so that it doesn’t impact the user experience.

Conclusion

Scatter-Gather pattern is one of the building blocks of modern distributed systems. Its applications range from distributed databases to social media feeds and e-commerce websites.

With parallel processing and aggregation, it helps solve :-

1. Faster response times through reduced latency.

2. Efficient horizontal scaling.

3. Graceful handling of partial failures.

4. High-throughput processing for large-scale applications.

However, Scatter-Gather comes with trade-offs—its added complexity is only justified when performance gains outweigh implementation challenges. For simpler cases, such as retrieving data for multiple users, a single API call may be sufficient.

Have you used Scatter-Gather before? What challenges did you face? Let’s discuss in the comments!!

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

How does it make the uploaded video available for consumption within seconds ?

Instagram Engineers have built a robust and scalable Video Upload Service that abstracts the complexity of video upload and processing.

In this article, we will to design and architect one such service. We will tackle the problem starting with a simple solution. We will identify the bottlenecks, iterate and optimize the solution.

By the end, you will learn the core challenges and tradeoffs done while building a reliable, scalable and an efficient video upload service. You’ll gain key skills to tackle a similar problem in your next system design interview.

Subscribe now

With that, let’s revisit our fundamental video processing concepts that were discussed in the last edition of this newsletter.

Boost Your Skills with Educative (Plus an Extra 10% Discount!)

Ace your coding interviews with Educative’s proven prep paths used by engineers at Google, Amazon, and Meta.
Get an exclusive 10% discount when you sign up through this link.
No setup, no distractions — just structured learning, real problems, and smart solutions.

Video processing fundamentals

Videos go through six phases before they are available for user consumption. The below diagram illustrates the process.

Video viewers use different applications and devices for playing videos. Similarly, their network bandwidth also varies.

Hence, it’s essential to convert a video into several formats and also vary the bitrate for compatibility. For eg:- A 1080p video may not render as expected on slow internet connection. So, it must be converted into a format like 240p or 144p.

In the video processing step, the system re-encodes the video into multiple formats. Besides, the step also performs operations such as watermarking, thumbnail generation, encryption, etc.

As seen from the above diagram, the video processing step can be modelled as a Directed Acyclic Graph (DAG). Each node in the DAG represents an operation and the vertices indicate data flow.

Now that you know the basics, let’s design a service that uploads and processes videos.

Problem Statement

Design a scalable, efficient and a reliable video upload and processing service

Let’s break down the problem statement into functional and non-functional requirements.

Functional Requirements

1. Users must be able to upload the videos

2. The system must process and generate multiple video formats. It must include functionalities such as watermarking, encryption, etc.

3. Users must be able to upload the videos through channels like messages, stories, and posts.

Let’s define the non-functional requirements in the context of each functional requirement.

Non-functional Requirements

1. System must scale to handle millions of concurrent uploads.

2. Upload and processing p90 latency must be within 10 secs. (It would make the experience interactive)

3. It must be fault-tolerant and reliably recover from any failures.

Given clear requirements, we will now come up with a simple design that meets them.

Approach 1 - Synchronous solution

In this approach, the client uploads the video and it gets stored in a blob storage. The video then gets processed by the backend Video Processing Service.

The following diagram shows the end to end process.

While the solution is easy to understand and simple to manager, would it scale ?

Before reading further, take a pause to think where it would fail to meet the our goals.

Here’s why the solution wouldn’t scale :-

1. Reliability & Fault tolerance - In case of video processing failure, the client would have to restart the whole process. Even if one of the operation (for eg:- watermarking) fails, the whole step needs to be retried. This compromises the service reliability.

2. Latency - Video processing time is proportional to the length of the video. Due to sequential processing, large video files would take a long time. It would take hours if not minutes for few large files (more than 200 MB).

3. Scalability - Given the sequential nature, a given server would be able to handle only limited number of requests. Increasing the instances would result in higher costs.

4. Efficiency - The solution performs compute-heavy video transcoding operations in sequence. It doesn’t exploit parallel processing which makes it inefficient.

Since we now know the downsides of the synchronous solution, let’s think of improving this one step at a time.

Approach 2 - Asynchronous solution

We can improve the solution in the previous section by making the backend video processing asynchronous.

The compute-heavy video transcoding can be decoupled from the uploading and pre-processing steps (video file validation, repairing malformed files, etc). This decoupling would allow both the layers to scale independently and robustly handle failures.

The above diagram shows the new architecture. Let’s dive into the key changes introduced in the design.

Here’s how the video transcoding can be handled :-

1. Scheduler - Scheduler would be responsible for orchestrating the video processing step (DAG). It would manage and co-ordinate the work of several worker instances.

2. Workers - These server instances would perform operations such as transcoding, watermarking, etc. They would be independent and inform the scheduler on completion of task.

The communication between Scheduler and Worker would take place via message queue.

This solution addresses the primary bottlenecks of the previous solution as follows :-

1. Reliability & Fault-tolerance - The whole video upload process doesn’t need to be retried in case of failures. Individual processing operations can be retried either by scheduler or worker.

2. Scalability - The async process with workers make it easy to scale the processing step. Similarly, the upload & pre-processing steps can be scaled independently.

3. Efficiency - Unlike the previous approach, two or more transcoding operations can be performed on multiple worker machines in parallel.

While this approach introduces excellent improvements, it comes with the following downsides -

• Latency - The overall latency doesn’t significantly improve since it’s dependent on successful completion of all the individual processing steps.

• Critical operations - The scheduler doesn’t identify and prioritize critical operations and non-critical operations (like video analytics). This further impacts the latency.

Let’s see how we can make this solution better.

Optimizations

We can further improve the process by :-

1. Latency - The video can be made available once the highest resolution is available. Hence, the system doesn’t need to wait for all the resolutions to be available.

2. Critical operations - The Scheduler can have two different queues for the asks - critical and non-critical. Latency sensitive tasks can be added in the critical queue while others can go in the non-critical one.

Although the above solution reduces the latency to an extent but the clients on slow networks wouldn’t be able to load high resolution videos. This would compromise the experience for clients with limited bandwidth.

This is an acceptable tradeoff since we expect all the resolutions to be available within couple of minutes. And the percentage of people with poor internet connection is less (< 10%).

Can you further improve the design and optimize it ? Before proceeding to the next section, take a moment to identify ways to tackle the problem.

Approach 3 - Asynchronous processing with Segmented video upload

In the previous two approaches, the video processing starts once the file upload completes. The longer the upload time, higher would be the overall latency.

Instead of waiting for the full file to be uploaded, what if we upload segments of the file ? Would that reduce the overall latency ?

The answer to the above question is Yes. The video file can be divided into a sequence of continuous segments known as Group of Pictures (GOP). Clients can upload the segments in parallel.

Further, each GOP segment can be encoded and processed independently. So, they can be processed in parallel and eventually stitched into a complete video.

We have reduced the overall latency using two techniques :-

1. Overlapping upload and processing through segmentation.

2. Parallel processing of individual segments.

The below diagram shows the overall architecture which meets our functional and non-functional requirements.

Let’s now understand few tradeoffs of the final solution.

Tradeoffs

Stitching step

The final stitching step that combines all the segments together adds complexity to the pipeline. However, the benefits of parallel processing outweigh the downsides of stitching step.

Small video files

For small video files ( < 10 MB), there isn’t an additional advantage of segmented upload and parallel processing. Since it’s an overkill, such files can be separately processed through a separate pipeline.

Segment size

Compression algorithms exploit temporal locality of large segments and lead to better compressions. Higher segment sizes result in less number of segments and reduce the degree of parallelism.

Hence, the segment size needs to be adjusted based on the expected video quality. Higher quality videos (reels, stories) can use more segment size. While use cases like videos shared via messages can use less segment size.

Client-side bandwidth constraints

In certain cases, the clients may have poor connectivity. In such cases, the video processing can be done on the client side (provided it’s supported by client devices). Client-side processing reduces the file size and helps with faster video uploads.

Conclusion

In this article, we learnt the system design and architecture of a Video upload service. We started with a naive solution and iteratively improved by identifying the bottlenecks.

Here’s a quick summary of how to build a scalable, reliable and a robust video upload service :-

1. Segmented upload - Break the video into segments and upload each segment. Further, upload and process each segment in parallel.

2. Parallelism - Use parallelism to execute the compute-heavy operations.

3. Critical vs Non-critical operations - Segregate critical and non-critical operations. Prioritize the critical operations while scaling the system.

4. Fault-tolerance - Ensure that the system robustly recovers from failures in individual steps through retries.

Before you leave, take this question as an exercise -

Leave your answers to the above question in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

In this article, we will understand in detail the challenges and Zerodha’s innovative approach to tackle the problem.

Sidenote: What is Zerodha ?

Zerodha is India’s largest stockbroker. It’s the first discount broker and has a vision to make trading/investing for retail investors.

Subscribe now

Use cases

Zerodha users purchase different securities such as stocks, ETFs, Mutual Funds, Futures, Options, etc. The platform must generate reports for different user queries like :-

1. Profit and loss for the past day/month/year.

2. Transactions done over a given period.

3. Tax summary for the past financial year

4. Summary of dividends, bonus shares, stock splits, etc

Zerodha offers its users a platform known as Console to view the different reports. Let’s now understand how the platform was built originally.

Old architecture

Zerodha used a variety of databases such as PostgreSQL, MySQL, and ClickHouse to manage the user data. In the old architecture, the backend web server executed queries on the database and returned the results to the clients.

The above architecture couldn’t scale with the growth in user and transaction volume. Here’s why the platform couldn’t scale :-

• Database table stored more than a billion records.

• Cross-table joins slowed the query execution.

• Users had different query patterns each having a different execution time.

As a result, users experienced timeouts or slow processing. This resulted in a poor user experience and wouldn’t scale in the long-term.

The problem amplified during the tax filing period where millions of users simultaneously tried downloading last year’s report.

To overcome the inefficiencies of synchronous processing, Zerodha decided to move to an async model. We will now see the working of async model.

Async workflow

In the async model, the users submitted their request. The platform acknowledged their request and the users came back after couple of mins, secs to find their result.

This model overcame the limitations of the synchronous approach. It resulted in huge improvements in the user experience.

The async model made sense as every user query had different execution time depending on the use case. 99% users wanted to view transactions done in last year, while remaining 1% were interested in 10 year old transactions.

User query patterns vary. Many users are interested in seeing the transactions done in the past 1 year. A handful of users are interested in viewing last 10 years transactions.

Here’s how the async model worked :-

1. Clients submitted their requests to the backend.

2. The backend queued the request as a job.

3. The jobs were executed by running database queries.

4. Clients polled for the job’s status.

5. On job completion, the users downloaded the result in the form of reports.

Zerodha discovered a hack in PostgreSQL to further optimize the process. This hack worked in tandem with asynchronous workflow to scale the system.

Let’s now understand the new architecture in detail.

New architecture

The new architecture consisted of two flows :-

1. Async flow - In this flow, user requests were executed by running database queries. On successful execution, the results were stored in an ephemeral PostgreSQL database.

2. Sync flow - On completion of async flow, the ephemeral PostgreSQL database was queried to fetch the reports.

In the async flow, a middleware named Dungbeetle was introduced. This middleware handled the user requests and modelled it as a job.

Dungbeetle managed the job’s lifecycle and moved the results of the database queries to the PostgreSQL instance also called the result cache.

Every query result was mapped to a database table. Running millions of such queries resulted in more than 7 million tables daily. And PostgreSQL supports creation of million tables.

The below diagram shows the new scaleable reporting architecture.

The ephemeral database was purged daily and it only stored the temporary results. Further, since this result cache had handful of records (~1000-5000), users could easily perform filter, and sort operations.

Zerodha’s CTO Kailash Nadh mentioned that the solution started out as a hack but has turned into an abuse.

There’s a thin line between a clever software hack and an outrageous abuse.

Would the solution scale if Zerodha witnesses a 10x user growth in future ? In my opinion, it may not.

Increasing the table count would reach a plateau at some point. And eventually slow the query execution after hitting a limit.

So, let’s see what alternatives can be considered here.

Alternative solutions

Here are few alternatives to the million table solution :-

Columnar database like Amazon Redshift/Apache Druid

These databases are optimized for analytical queries. Following are few advantages of this solution :-

1. Supports long running large-scale analytical queries.

2. High compression and faster query performance on large datasets.

3. Eliminates the need for millions of tables.

However, the team would have to invest in a separate data pipeline to move the data from transactional or online systems.

Pre-compute and object storage

The reports can be generated asynchronously and stored in an object store like AWS S3. Here’s how it would benefit :-

1. Highly scalable, cost-effective storage for large volumes of reports.

2. Users can download the reports without hitting the database.

But, users won’t be able to perform operations such as filtering or sorting on the result set.

Conclusion

Zerodha started with a basic architecture for user report generation. The solution couldn’t scale with the growth in users and transaction volume.

The architecture was based on synchronous processing and couldn’t handle peak workloads such as report generation during tax filing.

The synchronous architecture was replaced with an async one to make the process more robust. In the async flow, the middleware executed the database queries and moved the data to an ephemeral PostgreSQL database.

Since every query result was mapped to a new database, it lead to creation of more than 7 million tables. Although it was a hack, but the solution scaled for millions of users.

Can you think of any other alternative solutions to solve the problem ? Post your thoughts in the comments below.

You can refer this talk by Zerodha’s CTO for more details on the innovative solution.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

The video streaming industry has grown rapidly over the last decade and now makes up 82.5% of global internet traffic. It’s not surprising that we spend the majority of our time watching Instagram reels or Youtube shorts.

Recently, tech companies have started asking video streaming design problems in system design interviews. Candidates often struggle due to a lack of experience in this area.

The best way to learn about video streaming is to build a strong foundation, understand the complex process, and go over real-world case studies.

This article simplifies complex concepts with examples and illustrations. Whether you're preparing for system design interviews or learning about video streaming, it covers key terms and the basics of video processing.

Subscribe now

Let’s start our journey by learning the core concepts!

Core Concepts

While recording a video, camera clicks a stream of pictures in quick succession. So, a video file is just a continuous sequence of images.

When you play a video, it displays the clicked pictures and our human brain perceives it as a motion picture.

Frame

In the context of a video, the clicked picture or an image is known as a frame. A camera generally clicks 30-60 frames per second.

The higher the number of frames, the better the video quality. Lower frames per second hinder the smoothness of a video.

Pixel

Each frame can be broken down into atomic units of equal size known as a pixel. A pixel is a square-like portion that can encode the colour of an image.

A pixel’s colour can be computed by mixing R (Red), Green (G) and Blue (B) colours. In computing, each of R, G, and B is known as a channel.

A channel can be allocated certain bits to encode the colour information. For eg:- Normal videos use 8 bits for R, G, and B resulting in 24 bits/pixel. Higher definition videos use 16 bits for each channel (48 bits/pixel).

Bitrate

Bitrate is the amount of data used to encode a video stream per unit time. It’s measured in bits/sec or kilobits/sec or megabits/sec.

Bitrate determines the overall video quality and the file size. Files using higher bitrate result in better quality and larger file size.

As seen in the previous section, each pixel is represented using a set of bits. In case each pixel uses more bits, then the video would require higher bitrate.

Certain video scenes such as action sequences require a higher bitrate than still scenes such as scenery.

Resolution

Video resolution is the count of distinct pixels that constitute a video. A higher resolution video would have more pixels as compared to a low resolution one.

Video resolution is expressed as width × height in pixels. A SD (480p) video has 640×480 pixels, while an HD (720p) has 1280×720 pixels.

Now that you are familiar with basic concepts, can you guess what would be the size of a short HD video file of 10 seconds ? 🤔

Let’s do some quick calculations and compute the size.

• Frames - 30 per second

• Resolution of each frame - 720 p or 1280*720 pixels = 921600 pixels

• Size of each pixel = 24 bits

• Total size of each frame = Resolution of each frame X Size of each pixel

  • 921600 x 24

  • 22118400 bits

• Size of 1 second video = Frames per second X Size of each frame

  • 30 * 22118400

  • 663552000

• Size of 10 second video = 10 * Size of 1 second video

  • 10 * 663552000

  • 6635520000 or 663.552 Mb

663 Mb is too big for a 10 second video 🤯

Have you come across such a big file while recording a small video ?

The answer is no. But, how does your device generate small video files of 1-5 Mb for a 10 sec video ? The magic lies in a process known as video encoding.

Video Encoding

Video encoding converts a raw video file into a format that’s optimized for storage, transmission and playback. The encoding process also ensures that video quality is not degraded.

Video encoders/decoders are also known as Codec(Encoder-Decoder). Some popular codecs are: H.264, H.265 (HEVC), VP9, AV1, MPEG-2, and MPEG-4.

A raw video files goes through the following steps during encoding :-

1. Frame-by-Frame Analysis - The encoder goes through each frame and identifies the redundancies within each frame.

2. Temporal Compression - It removes the redundancies or repeated frames through compression.

3. Bitrate control - The encoder adjusts the bitrate depending on the frame complexity.

4. Compression - The encoder runs a complex compression algorithm that might either discard some data or completely retain the data during compression.

5. Output - The final output file contains the video, audio and other metadata.

Video decoding

Video decoding is the reverse process of encoding. The decoder extracts all the frames from the encoded file and plays the video.

Decoding happens in real-time as you play a video. Both encoding and decoding are CPU-intensive operations.

A video file is more than just a sequence of images. It must also contain audio, subtitles, thumbnails, watermark, etc. How are these things merged in a single file ? 🤔

Let’s understand this in the next subsection

Video Container

The video file, audio, subtitles and metadata are packaged in a file known as video container. The container is a wrapper that allows the playback software to synchronise the different files for a smooth playback.

Video containers have different file formats such as MP4, MKV, AVI, MOV, etc.

If I use QuickTime Player on Mac to record a video, it gets stored in the form of a .mov file. This file is specific to a Mac and it won’t run on non-Mac devices.

How do video streaming websites ensure that different devices play the video file without any additional software installation ? 🤔

We will now look at Video transcoding that helps streaming websites achieve this.

Video Transcoding

Let’s assume that I record a video using QuickTime Player in .mov format and I upload it to Youtube. Youtube recognises that the format is incompatible with other desktop, mobile or TV devices.

The end-user devices might be expecting a MP4 file for playback. Hence, Youtube’s system must first convert the file from .mov to .mp4 format.

This process of converting a video from one format to another format is known as Transcoding. A video goes through the following sequence of steps in transcoding :-

1. Decode the source video

2. Decompress the decoded video

3. Perform transcoding through the following series of steps:

   1. Change bitrate if required

   2. Modify the Frames per second

   3. Change the resolution (1080p to 720p)

   4. Encode and compress the video in a different format (.mp4 or .avi)

4. Output the transcoded video and store it in a permanent storage

Once the transcoding steps complete, the video then becomes ready for streaming. Transcoding happens asynchronously and user is notified after its completion.

Now that you are familiar with basic concepts of video streaming, let’s understand what challenges video streaming services face.

Video streaming challenges

A few seconds of entertainment for users comes at the expense of hours of engineering efforts. Streaming companies strive to continuously innovate and provide a smooth user experience.

Here are few common challenges that companies like Youtube, Netflix, Meta face :-

1. User experience - It’s essential to provide a seamless user experience without any glitch to millions of users simultaneously.

2. Performance - Upload and download latency are key indicators or success criteria for any streaming service. No user would wait for minutes for a video to stream.

3. Scalability - Applications such as Facebook or Instagram Live allow celebrity users to connect with millions of users simultaneously. Handling millions of connections and delivering real-time experience is a hard problem.

4. End-user devices - Streaming services have to support a multitude of devices. Every device has its own OS, hardware and constraints. Companies have to ensure compatibility with all the end-user devices.

In my next article, I will discuss few real-world case studies and how each company has tackled different set of challenges.

Let’s summarise and revisit the concepts that you have learnt so far.

Conclusion

A video captured by a camera is a sequence of images. Each image is known as a frame and it captures 30-60 frames per second.

A frame is a collection of identical square shaped units called as pixel. A pixel contains the colour information for the portion of the image it represents.

A video has a resolution standard that defines how many pixels it uses. Higher resolution (High definition 1280p) captures more granular details and leads to better picture quality.

Bitrate determines the amount of data used to represent a pixel. Bitrate needs to be adjusted based on the frames and the video resolution.

A raw video file is large in size and goes through encoding and compression that reduces the file size. Encoding optimizes a video file for transmission and playback.

Streaming companies use a process called Transcoding that converts one video file format into another. This ensures file compatibility with different devices. The process also adjusts the bitrate and the resolution.

Should Transcoding be a synchronous process given that it’s compute-heavy or should it be asynchronous ? Leave your thoughts in the comments below.

Stay tuned for the next article where we’ll understand the technologies powering the streaming services.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

The system needs to generate the content for every active user. Every user’s content is unique due to restrictions and privacy checks.

Friendships, checkins, comments, reactions etc result in a dynamic social graph. The platform must handle the large scale and deliver the user content within fraction of seconds.

Meta developed TAO - a geographically distributed graph data abstraction to power the largest social graph. TAO is optimized to handle Facebook’s billions of reads and millions of write requests daily.

In this article, we will go over Facebook’s original architecture and its limitations. We will then walkthrough how the architecture evolved into TAO. We will learn how TAO’s architecture solved the problem through its robust design. And conclude by going over TAO’s performance on production workloads.

Subscribe now

Problem Statement

Imagine a Facebook user named Jimmy checkins into a hotel and tags his friends Bob and Jim. This post lands into his friend’s feed. Later, several friends would like the post and few would comment.

This would go on for a couple of hours or days until Jimmy creates another post in near future.

Did you notice what just happened ?

In short - A user created a post, it got some traction and engagement. The view of the post kept changing as the reactions and comments kept flowing. In a few days, the engagement faded and user created another post. And the cycle continued.

One more observation is that the post’s view differs for every user. For eg:- If the post is public, a non-friend user may not be able to comment. Similarly, due to restrictions one or more friends might not comment or share.

A single post data looks like a graph of inter-connected nodes. If you extend it to all Facebook users, it becomes a social graph. The social graph is dynamic as it changes every second due to user’s engagement on the post.

To summarize, this is the core problem that Facebook needs to solve :

Problem - Generate customized user content by applying privacy checks and filters on ever changing social graph. Moreover, develop a resilient, fault-tolerant and a performant system to handle increasing load from billion users.

Let’s now go over the original architecture that solved this problem.

Original Architecture

In the beginning, Facebook adopted a PHP-MySQL stack to render the social graph. They used Memecache as a look-aside cache to accelerate the performance.

Here’s how the architecture looked like :

The WebApp layer was an aggregation layer. It fetched the data from different upstream services for likes, comments, shares, post data etc. Each service used Memcache to speed up reads.

Before reading further, take a moment to think what could be the potential downsides of this approach ?

The original architecture had the following limitations :-

1. Distributed control logic - The logic was spread across different services which increased failure modes. For eg:- Thundering Herd on MySQL in case of a cache miss.

2. Inefficient access pattern - The queries need to fetch the list of different associations (comments, reactions, etc). Also, it must handle concurrent updates to a cached list.

3. Expensive read-after-write consistency - For high availability, the system used asynchronous master/slave replication. Invalidating cache in replica was challenging. As a result, it was expensive to achieve read-after-write or strong consistency.

Now, that you understand the original architecture and its limitations, let’s now focus on how Meta solved it.

TAO - Graph Data abstraction

TAO was developed as a service which would expose a graph to the clients. For eg:- Given a post’s identifier, it would aggregate and return all the data to the WebApp layer.

It didn’t allow the clients to directly access the MySQL database. Clients would use TAO’s APIs to fetch data they needed.

TAO provided the following two data models :-

1. Object - An object represented data entity such as a user, post, or a comment. Each object was uniquely identified by a 64-bit field.

2. Associations - An association was either a relationship between two objects or action taken by one object on another. Each association had a type indicating relationship like LikedBy, FriendOf, PostedBy, etc.

APIs

1. Object APIs - These APIs were used to create, update and fetch the object data.

2. Association APIs - These APIs were useful for defining relationship between the objects. Further, they provided several variations such as :-

   1. assoc_get - Fetch all associations between two objects.

   2. assoc_count - Count of associations. Useful to get count of comments or likes.

   3. assoc_range - To fetch a range of associations using pagination.

   4. assoc_tim_range - Fetch associations within the time constraints

Let’s now understand the internal architecture that powered these APIs and data models.

TAO Architecture

TAO’s design used a layered architecture approach. Each layer had a different responsibility i.e caching or data storage.

TAO used replication for high availability and failover. Further, it scaled geographically by maintaining a copy of social graph in every region.

Let’s go over each layer and understand the design decision.

Storage layer

TAO stored the object and association data in MySQL servers. The data was sharded by using consistent hashing.

Every TAO API mapped to a SQL query at the lowest level. It leveraged MySQL’s indexes to speed up lookup of range queries for association APIs.

Design decisions

1. MySQL - Although a NoSQL database like LevelDB could have worked, MySQL was chosen for non-user-facing functions like bulk imports, replica creation, async replication, and atomic transactions.

2. Consistent hashing - This ensured uniform load distribution and simplified request routing.

Caching layer

The caching layer served the TAO’s APIs. Every server in this layer cached the data from the storage layer using LRU (Least Recently Used) policy.

Every server hosted cached data for multiple shards. In other words, each database shard mapped to a single cache server.

The caching servers communicated with each other and understood the semantics of the data. For eg:- if the association count was 0, then caching server wouldn’t query the associations.

The cache layer was further divided into two :

1. Leader tier - Servers in this tier handled all database interactions.

2. Follower tier - Clients interacted with this tier. In case of cache misses, requests were forwarded to the leader tier.

There was a single leader tier and several follower tiers. Additional follower tiers could be added to handle the increasing load.

Design decisions

Single Tier vs Leader-Follower tiers - A single large tier is prone to hot spots. The Leader-Follower architecture spreads the load evenly.

Replication

TAO used multi-region asynchronous replication. Data was replicated from a master region to a slave region, maintaining cache consistency through invalidation messages.

Write flow

Followers directed writes to leaders. If a leader was a slave, writes were redirected to the master’s leader. Data was updated synchronously across tiers, emitting invalidation messages for cache consistency.

The below diagram illustrates how a follower tier in Slave handles a write.

Read flow

Reads were performed on followers in a region, ensuring data was served locally for performance.

The below diagram shows how reads are served locally in the same region.

Design decisions

Cache invalidation - To ensure cache consistency in master-slave region, invalidation messages were sent along with the replication stream. Caching servers used the invalidation messages for evicting or updating the entries.

Bottlenecks, Optimizations and Failovers

Hot spots

It was common for a post to become viral. As a result, a given shard would receive a large number of requests. This would make the shard a hot spot.

Solution -

1. Shard cloning - The shard was cloned and replicated on several servers in a follower tier. This technique rebalanced the load.

2. Client caching - The server returned the access rate for the object. In case the rate exceeded a pre-defined threshold, the client cached the data. This reduced the load on the servers.

High-degree objects

A limit of 6000 was imposed on the association list response returned by TAO. However, many objects had more than 6000 associations. For eg:- celebrities getting millions of comments.

Further, it was common for many objects to have 0 associations. And TAO would get assoc_get requests.

In case of a cache miss, this would result in large number of database queries.

Solution -

1. Association count - In case the count is 0, the query would directly return instead of checking the other objects in the database.

2. Application semantics - They leveraged the data such as creation time of the objects and the associations and limited the search of associations.

Consistency

TAO provided read-after-write consistency in the absence of failures. This was applicable for any client writing to and reading from the same follower tier.

The design preferred availability over strong consistency. In case of failures, the data was eventually consistent.

Hence, the clients had to tolerate some amount of data staleness. This was a fair trade-off given the load that the system had to serve.

Strong consistency

TAO provided strong consistency to a subset of APIs such as authentication. These APIs were marked as critical, guaranteed strong consistency at the cost of increased latency.

Failovers

TAO had to handle several failure scenarios such as :

1. Master database failure - Slave’s database was elected as the new master. All the cache-misses and writes were directed to the newly elected master.

2. Leader failures - Followers sent the reads to the database. While writes were sent to a random leader in the tier.

3. Follower failures - Every follower tier had a backup. In case of server timeouts, the clients sent request to the backup tier.

Performance

Here’s how TAO performed on Facebook’s production workload :-

1. Availability - 4.9 * 10^-6 queries failed over a period of 90 days.

2. Hit rate - The overall hit rate was 96.4% for read queries.

3. Failover - Slave was promoted to master 0.25% times due to database failures, upgrades or maintenance.

4. Replication lag - p99 for slave database lag was 3 seconds.

Conclusion

Meta faced a challenging problem while customized content for each user using its dynamic social graph. Initially, they used PHP stack, fetched data from MySQL and used Memcache as a look-aside cache.

Meta developed TAO as a graph data abstraction layer. It solved several downsides of original look-aside cache architecture.

Through its layered architecture, it scaled to handled billions of reads and millions of writes. Further, it favoured high availability over strong consistency.

TAO solved common problems in distributed systems such as hot spots, thundering herd and automatic failover. They adopted robust mechanisms like shard cloning and caching to build a resilient system.

If you have reached here, then here are few points to ponder :-

• Do you think a graph database like Neo4j would have worked instead of TAO ?

• How do other social media websites like LinkedIn, Twitter, Threads, etc manage graph data ?

Share your thoughts on the above points in the comments.

Reference - Facebook TAO

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

There is no one size fits all solution for data storage. End customers data could be stored in a relational database but a text-search feature would need the same data in Elasticsearch. Similarly, an analytics dashboard would expect the same data to reside in a data warehouse.

A few years ago, I faced a challenge where my team needed data attributes from a system owned by another team. Direct API integration added complexity due to multiple consumers, and a pub-sub setup would have led to redundant effort, as many teams used similar datastores and would have had to duplicate work. This is when I came across Change Data Capture (CDC) to solve this.

Companies find it challenging to keep the data in one system and store it in different forms in other storage systems. Traditionally, companies used batch processing to Extract, Transform & Load (ETL) the data in different systems.

Batch processing resulted in delays in data propagation and heavy use of compute. To solve this, companies devised Change Data Capture (CDC) to keep different data sources in sync in real-time.

This article will cover the following :-

1. What is CDC ?

2. How does CDC work ?

3. Types of CDC and their pros/cons

4. Applications of CDC

5. Real-world examples of CDC

With that, let’s begin and understand what is CDC and why is it required.

Subscribe now

What is CDC ?

Let’s take an example of a food-delivery system. Assume that it stores the data for its restaurants in a relational database. And captures the location, user ratings, top rated dish, etc.

The customer using the app would have the following features :-

1. Search restaurant by name.

2. Find nearby restaurants.

3. View the average rating of the restaurant.

4. Check the restaurant profile

Now, to meet the above use cases, we would need the following storage solutions :-

1. Search by name - Elasticsearch with text index.

2. Nearby restaurants - Elasticsearch/Postgres with Geospatial Index (GIS)

3. Average rating - Data lake like Redshift

4. Restaurant profile - Database like Postgres/MySQL

Initially, we would store the restaurant’s profile in a database like Postgres/MySQL. However, would need the same data in Elasticsearch and Redshift.

Whenever there is any change to the restaurant’s profile i.e update in the ratings, the same has to be communicated to other systems (Redshift). Similarly, any change in the name or location, would imply updating the data in Elasticsearch.

Change Data Capture (CDC) is the process of capturing data changes in one system and propagating it to different systems in real/near-real time. It’s a technique which ensures different data sources are in sync.

CDC overcomes the downsides of writing the data to multiple data stores. And also the additional burden of running compute-heavy offline jobs to sync the data.

The following diagram illustrates this process.

Now, that you understand the concept of CDC, we will look at the working of CDC.

How does CDC work ?

Most of the databases provide a mechanism to record all the changes made to the database tables. For example, MySQL records all the changes in binary log (binlog). Similarly, PostgreSQL uses Write-Ahead Log (WAL) to capture modifications.

CDC tools read & parse the database logs, convert it to events and add it to a message broker or event bus like Kafka. Further, consumers consume the events and write it to destination datastore.

The following diagram illustrates the working of CDC:

Generally, CDC tools add a table’s event to a Kafka partition. The Kafka partition guarantees strict ordering of all the events. Further, it provides strong durability guarantees and prevents data loss.

The consumer receives the events in the same manner as the source database. This ensures data completeness and accuracy. The data is then transformed by the consumers and then dumped in the destination database.

There are several different mechanisms to implement CDC and propagate the changes. In the next section, we will look at the different types of CDC and their pros/cons.

Types of CDC

Log-based CDC

The example in the previous section described the working of a log-based CDC. This method is highly efficient since it tracks the low-level logs that databases already maintain for durability and crash recovery.

Pros

• No performance impact on the database.

• High accuracy and consistency.

• Works seamlessly for databases that support transaction logging for durability

Cons

• Logs can consume significant disk space.

• Complexity in parsing logs

Trigger-based CDC

This CDC relies on database triggers to detect and capture changes. Database triggers are configured to take an action (logging a change) on the occurrence of certain events. The events are captured in an audit table and then sent to downstream systems.

Pros

• Simple to implement in any database supporting triggers.

• Can be used if the database doesn’t support logging like WAL or binlog.

Cons

• Impact performance in case there are large number of triggers.

• Complexity in managing triggers

Polling-based CDC

This technique periodically queries the database to detect changes by comparing current values to a previous state. This is done by looking for new rows, modified timestamps, or version numbers.

Pros

• No dependency on transaction logs or triggers. So, works for databases that don’t support either.

• Simple to setup when other methods are not available.

Cons

• Inefficient since it requires constant querying.

• Lacks real-time experience and results in delays.

• Difficult to detect deletes without maintaining a full copy of data.

Out of the different options available, Log-based CDC is considered to be most efficient. And also guarantees high degree of data freshness.

Now, that you are familiar with different CDC techniques, let’s look at few use cases of CDC.

Applications of CDC

Cache Invalidation

In certain cases, microservices have to explicitly invalidate cache entries and also remove the records from the database. This often impacts the performance as the service needs to perform dual-writes.

This can be solved through CDC by first removing the record from the database and evicting cache entry asynchronously. The below diagram illustrates the process.

This ensures that the application caches are in sync and don’t store stale data.

Data replication

Companies need to backup existing databases for fault tolerance and scalability. Since a full copy of data can be constructed through CDC, it’s used for different data replication use cases.

Search Indexing

CDC is a good fit for building Search indices since it supports in-order delivery of the mutations. It can keep the primary datastore and the Search indices in sync providing a real-time experience.

In case of a food delivery app, if a restaurant menu item changes, the same updates can flow through the Search indexing pipeline via CDC. Further, it would update the Search indices so that users could see the latest updates.

Real-time Analytics

CDC can be used to capture and stream the changes in analytics platform. It enables near real-time synchronization between transactional systems and data warehouses or data lakes.

This helps minimizes data latency and allows businesses to make timely, data-driven decisions, powering use cases like live dashboards, fraud detection, and personalized user experiences.

We will now look at some CDC solutions developed by industry leaders and the challenges faced in building CDC serices/tools.

Real-world examples of CDC

Tech companies like Uber, Airbnb, Netflix, etc transport petabytes of data every day. It’s common for these companies to use CDC to keep datastores in sync and provide real-time experience to users.

There are few common challenges that the companies have faced while building CDC solutions. Following are some challenges that influenced the development of CDC tools :

1. Scalability - The solution must scale to handle increasing volume of data. The performance shouldn’t get impacted due to spiky load.

2. Data freshness - Propagation delays aren’t tolerated for users expecting a real or near real-time experience. A delay of a few seconds may be acceptable, but delays extending to hours are not.

3. Extensibility - The system should be extensible for handling different data sources such as MySQL, PostgreSQL, Cassandra, etc. Onboarding a new data source should be plug-and-play.

4. Data integrity - The solution must ensure no data corruption and guarantee delivery of data mutations within the SLA.

5. Fault tolerance - The system must be resilient to failure.

Let’s now look at few solutions developed by tech companies.

SpinalTap

SpinalTap is a Airbnb’s CDC system and integral part of its infrastructure. It was designed to be a general purpose solution to abstracts the change capture workflow, and easily extend to other dependencies such as event bus, consumers and data sources.

It’s used for a variety of in-house use cases like Search indexing of reviews, inbox and support tickets. Besides, it’s adopted for big data processing, cache invalidation and also signaling between services.

It’s an open source project and you can find the source here.

DBEvents

DBEvents is Uber’s framework to standardize data ingestion into Apache Hadoop Data lake. It was designed to solve three key business requirements - data freshness, quality and efficiency.

Debezium

Debezium is an open-source project for CDC. It is built on top of Apache Kafka and provides Kafka connect compatible connectors.

It leverages Kafka’s streaming platform to ingest data mutations completely. It guarantees fault tolerance and ensures zero data loss.

Conclusion

Change Data Capture (CDC) is the process of capturing data changes in one system and applying it in different downstream services. The process ensures data consistency across several systems.

Most of the databases like MySQL, PostgreSQL use logs to capture data modifications. CDC tools read the logs, parse it and dump the events in a distributed commit log like Kafka or Event Bus.

Kafka guarantees ordering and durability of all the events. The consumers then read the events and replay it in the destination datastores.

Since the destination receives the events in the same order as the source, full replica of the data can be constructed.

CDC tools are employed for use cases such as cache invalidation, search indexing, data replication and real-time analytics.

Companies have devised several CDC solutions and open-sourced them. Some common requirements of CDC solutions are - Scalability, Data Freshness, Data Integrity, Extensibility and Fault tolerance.

The best way to become an expert in CDC is to get some hands-on experience with an open-source tool. You can play around with Debezium and also go through its codebase.

Also, why do you think different companies have designed different CDC solutions ? Wouldn’t it make sense to have a common solution across all the companies ? 🤔 🤔

Leave your thoughts in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Share

In my first system design interview, I thought it was just about drawing boxes and arrows. I didn't prepare well, leading to a rejection. After a year of failures and learning, I passed Meta's interview in 2020.

Each rejection taught me something new. I improved my technical skills and gained confidence. Eventually, I successfully cracked interviews at Microsoft, Amazon, Meta, Atlassian, and Airbnb.

This journey improved both my system design and interviewing skills. Looking back, I realized the mistakes that caused my rejections. And I developed a framework to handle complex system design interview problems. Using a structured approach in the 1-hour interview helped me succeed. This method also helped my friends get jobs at top tech companies.

In this article, I'll share a six-step approach to ace any system design interview. You'll learn common mistakes and how to divide the hour into six steps. I'll explain what to focus on in each step to create a complete design.

Subscribe now

Also, if you want to get better at system design, refer the following links :-

1. System design handbook

2. Educative.io - Grokking the system design interview

3. Educative.io - Learn to code

If you want to improve at system design and join top companies, let's begin. We'll look at typical candidate mistakes.

Common Mistakes

1. Poor time management: Spending too long on one aspect, neglecting others.

2. Lack of prioritization: Failing to focus on key features the interviewer wants, given the time constraint.

3. Shallow analysis: Not diving deep enough to justify choices or discuss alternatives, preventing interviewers from assessing technical depth.

4. Lack of initiative: Relying too much on interviewer guidance instead of working independently.

5. Overlooking bottlenecks: Not addressing how the system handles heavy loads or traffic spikes.

Now that you know the common reasons for rejection, we’ll now see how to tackle them through a structured framework.

System design interview framework

We’ll see each of the steps, what should be tackled and how much time you must allocate.

The interviews start with a one or a two liner problem statement. For eg: “Design a distributed job scheduler for a large enterprise”.

In most cases, you would be familiar with the constructs of the problem. However, you must discuss with your interviewer and understand the problem if you don’t have enough context about it. For eg: If you haven’t heard of a job scheduler, ask questions about it and its working.

Functional Requirements - 5 mins

After understanding the problem, focus on functional requirements:

1. List all actions the system must perform.

2. Prioritize the top 3 core features to design during the interview.

3. Make reasonable assumptions (e.g., existing user registration for Instagram's newsfeed).

4. Keep complex elements out of scope (e.g., treat recommendation systems as a black box).

5. Discuss with the interviewer about any additional constraints they want to add.

Clarify details with your interviewer, as they often want to see how you handle constraints.

Non-functional requirements - 5 min

After listing the functional requirements, you must list down how the system must achieve its objectives. These things would form the non-functional requirements.

You can refer to the below diagram for different non-functional requirements.

You must ensure that any non-functional requirement is given in the context of a functional requirement and not the system as a whole. For eg: You can have eventual consistency in newsfeed but not in user's registration. Hence, while stating it's important not to state that the system as a whole is eventually consistent.

Moreover, don't keep the non-functional requirements vague. You can make some guesstimates and quantify it. For eg: State that the latency must be less than 200 ms instead of just stating latency should be low.

Capacity estimation (Optional) - 2 min

In many system design books and courses, capacity estimation is covered, and I've noticed candidates often spend time calculating storage, throughput, and hardware needs.

I've been there myself, making incorrect calculations or wasting time during interviews. In my experience, this only shows calculation skills and doesn’t add much value to the discussion.

Now, I ask upfront if the interviewer wants to focus on capacity estimation. Nine out of ten times, they’re fine without it. It's important to clarify this early on.

That said, some design decisions do depend on calculations. For example, designing a live video system with emoji reactions would be write-heavy, so calculating requests per second (RPS) helps select the right storage system.

Core Actors, APIs and flows - 10 mins

Once you clearly understand what the system needs to achieve, the next step is to identify the core actors in your system. These are the key entities in your design. For example, in Tinder, the core entities would be Users, Swipes, Matches, etc.

This doesn't need to be exhaustive since you may not know all the required data. Just make a bullet list of the identified entities.

Next, identify the APIs your clients or services will use for communication. Note all the request/response interactions. Whether synchronous or asynchronous, these will clarify how entities in the system interact.

If working with an offline system, identify workflows with multiple steps. For example, in a job scheduler, you'd need to fetch jobs, schedule them, monitor, and trigger alarms. Note these workflows and the sequence of steps involved.

High-level architecture - 15 mins

In this step, you must walk through each functional requirement and sketch the flow from the client to the backend database. Draw the relevant entities and highlight the different flows.

Explain how the APIs would work and highlight cross-cutting concerns like rate-limiting, encryption, etc solved by components such as API Gateway.

You don’t have to be 100% correct in this as you can further iterate, and improve your design in the dive deep section. You have to actively communicate and explain each flow to your interviewer.

Here’s a sample high-level architecture for the design of Tinder along with its functional requirements.

During this process, you will get an idea of what fields are relevant for each data model. Hence, mention the important fields and ignore the redundant ones. For eg: Don’t spend time on attributes such as firstName, lastName for the User entity.

At the end, you must ensure that your high-level architecture meets all the functional requirements that were stated in the beginning.

Dive deep, Trade-offs and bottlenecks - 20 mins

After sketching the high-level design, you need to come up with a comprehensive design. In this step, you will vet whether your design meets all your non-functional requirements and constraints.

You must dive deep into each flow and justify the choice that you have made. For eg: Explain why PostgreSQL is preferred over Cassandra or DynamoDB. Similarly, rectify and use the correct data store or caching mechanisms to optimize and meet any latency constraints.

Often, there would be multiple approaches to solve the same problem. For eg: You can use Geo-sharding index provided by PostgreSQL or Elastic Search for finding nearest restaurants or cabs. In such cases, explain the pros/cons of each and why you prefer one over the other in the context of the problem.

Finally, identify the bottlenecks in your design and explain how you would address each. This could be hotspots or handling sudden burst of traffic. These are solved problems and it’s important to have a strong hold on fundamentals to tackle them.

You must make sure that your solution is complete and meets both the functional and non-functional requirements. Also, give the interviewer chance to speak and check if they need to probe any particular area during the discussion.

Conclusion

System design interviews can be challenging and often involve problems that companies take months or years to solve.

However, don't let this overwhelm you. A structured approach can help you showcase your skills effectively.

If you're struggling with system design problems, try applying a six-step approach and practicing interview questions. This method shows your ability to break down complex problems, propose simple solutions, and optimize further.

Now that you have learnt the six-step approach, try applying the same to problems such as :-

1. Design a Instagram newsfeed system

2. Design an online judge

3. Design a distributed job scheduler

4. Design a distributed web crawler

After attempting these problems, you can look at the solutions and see how you fared.

Also, check out these links—they're helpful for system design interview preparation -

1. System design handbook

2. Educative.io - Grokking the system design interview

3. Educative.io - Learn to code

Improving at system design requires patience, perseverance, and hard work. Consistency is key, and the journey will ultimately make you a better engineer. 🚀 🚀

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Share

Over time, the focus shifted to smaller sub-systems, like newsfeeds or notification systems, which fit within the interview timeframe. These problems often reflect real challenges companies have faced, allowing them to evaluate a candidate's approach.

Companies sometimes share these issues in blog posts, so it's crucial for developers to stay updated by reading them.

In today’s article, we'll explore a real-time mouse pointer system, inspired by a problem solved by Canva. We'll treat it as a system design question, create a solution, and discuss the trade-offs. If you're preparing for system design interviews, this article will guide you on how to approach them. Let's begin!

Note: - This article is inspired by Canva’s article on how they designed real-time mouse pointer movements. Credits and Kudos to the Canva’s Engineering team for their efforts.

Subscribe now

Problem Statement

Design a collaborative editing system which shows the real-time mouse pointer movements of the editors or the viewers.

Clarifying questions

Before proposing a design, clarify assumptions and ask relevant questions. Here are some questions to ask your interviewer:

1. What should the collaborative editor display to the users ?

2. What should happen on the screen when a user moves their mouse pointer ?

3. How will we differentiate mouse pointers—by colour or usernames?

4. Should we design the editing features or focus only on mouse pointer movements?

5. What happens when a new user joins or an existing user disconnects ?

These questions help define the functional requirements of your system, influencing your API design and the workflows.

Additionally, discuss system constraints, such as :

1. What is the expected scale and how many concurrent users can the system handle?

2. Should the system track and store all mouse movement history?

3. What is the required availability? Should it aim for zero downtime with a 99.99% SLA?

4. How much lag in milliseconds is acceptable for the mouse movement?

5. How accurately should the system display the mouse pointer movements?

Take notes on the interviewer’s answers and keep these constraints in mind while designing the system.

Let’s now define the functional and non-functional requirements for the system. (PS - The following requirements my differ based on the interviewer)

Functional requirements (FR)

1. Users should see the collaborative editor screen and view other users working on the same document. FR-1

2. All users should see each other's mouse movements on the screen in real-time. FR-2

3. Each user's mouse pointer must display their username. FR-3

4. The feature should focus solely on showing the mouse movements, not the collaborative editing. FR-4

5. When a new user joins, their mouse pointer should appear on all the screens. If a user disconnects, their pointer should be hidden from all the screens. FR-5

Non-Functional requirements (NFR)

1. The system must handle mouse movements from 100K online users with a high throughput. NFR-1

2. The system should not keep a history of user’s mouse locations; it must only display the current position and the movement. NFR-2

3. The system must be highly available with zero downtime and 99.99% availability, functioning without errors during upgrades or deployments. NFR-3

4. The system’s p99 latency for mouse pointer movements should be under 50 ms for a seamless user experience. NFR-4

5. Mouse movements do not need 100% accuracy; a tolerance of 5-10% is acceptable. NFR-5

Now that we have clear idea of the system’s requirements, our next step is to come up with a design, define the data models and identify the different workflows.

Detailed Design & Choices

We will now outline the different actors, their interactions, the data flow, and methods for storing and propagating the data.

Actors

1. Clients - Desktop Web Browsers will connect with the backend servers. Further, they would periodically pass the mouse pointer’s location on the screen. We will assume that the client-side libraries would capture this information and send it to the servers.

2. Servers - The servers would be responsible for the following functionalities :-

   1. Managing the client connections.

   2. Processing the mouse pointer updates from the clients.

   3. Passing the mouse pointer update from one client to other clients working on the same document.

Data models

The data exchanged between the client and the server falls into the following categories :-

1. New connection - Client initiates a connection and sends the document on which it is working along with the mouse pointer’s current location. (FR-1)

2. Mouse pointer movements - Clients would send this as soon as the user’s mouse moves. (FR-2,FR-3, FR-4)

3. User disconnection - The server would have to hide the mouse pointer from other users in case one user disconnects. (FR-5)

Out of the above, 2 would have a very high volume. We can let the client send the mouse’s updated location three times every second without compromising mouse pointer’s accuracy on UI (NFR-5, Clarify this assumption with the interviewer).

Additionally, the system need strong reliability guarantees for 1 and 3 and they shouldn’t get missed (FR-5). In case the updates are missed, the user’s experience wouldn’t be accurate. For eg:- Showing an offline user online. It’s fine if the system misses mouse pointer movements as the accuracy wouldn’t be compromised much (NFR-5).

Let’s look at the sample data for the three categories.

New connection

{
  "eventType": "NewConnection",
  "userId": "user123",
  "username": "JohnDoe",
  "documentId": "doc987",
  "mousePointer": {
    "x": 250,
    "y": 300
  },
  "timestamp": "2024-10-06T12:34:56Z"
}

Mouse pointer movements

{
  "eventType": "MouseMovement",
  "userId": "user123",
  "documentId": "doc987",
  "mousePointer": {
    "x": 320,
    "y": 450
  },
  "timestamp": "2024-10-06T12:35:10Z"
}

User disconnection

{
  "eventType": "UserDisconnection",
  "userId": "user123",
  "documentId": "doc987",
  "timestamp": "2024-10-06T12:36:00Z"
}

In certain cases, the user may not explicitly leave the screen. For eg:- browser or device crash. In such cases, the server would have to identify the disconnected client and communicate the new state to the other users.

Workflows

Client-Server interaction

The clients would need to maintain a persistent connection with the backend servers. The server would need to relay one client’s mouse update to all other clients. Hence, we would need bi-directional communication between the clients and the servers.

Websockets would solve the use case for real-time communication between the client and the server. (FRs 1-5). Further, it would have low overhead and latency meeting (NFR-4).

Moreover, the clients would connect to a LoadBalancer which would efficiently distribute the traffic among the different backend servers. The least loaded load balancing strategy would be appropriate to ensure even distribution of connections.

Note that in the above setup, clients working on the same doc may get connected to different servers. This avoids making a server single point of failure and doesn’t make a single server a hotspot thereby improving the availability (NFR-3).

Additionally, in case of upgrades and server restarts, the clients would reconnect and the LoadBalancer would rebalance the connection ensuring high availability (NFR-3).

User connection & disconnection

The server needs to handle when a user connects or disconnects and inform all the other users about the change. Since each user connects to a different server, all servers must know where each user is connected. This way, updates are sent to the right servers, which then pass them on to their users.

To solve this problem, every server must have a view of the session state. The session state would consists of :-

1. DocumentId - identifier of the document.

2. List of users working on the document, and their corresponding severs.

When the session state changes, one server must notify all others. Others servers must updates their session state view accordingly.

For strong reliability, low-latency, and high throughput, Redis Streams is ideal for exchanging session state. This ensures that server restarts or deployments won't cause issues, as servers can read data sequentially from a reliable stream.

Each server will read from its own dedicated Redis Stream and write to all other servers' Streams.

The below diagram illustrates the process in detail :

Mouse pointer updates

On processing the mouse pointer update, the server would have to lookup the clients working on the document and their corresponding servers. It would have to then send the message to all the other servers.

Redis Pub/Sub would be best solution for this use case since the mouse pointer updates have following characteristics :-

1. Low-latency delivery (NFR-4) - Redis Pub/Sub guarantees real-time quick communication between the clients.

2. Transient messages (NFR-2, NFR-5) - Redis Pub/Sub doesn’t retain the messages if the clients don’t consume it. Also, it’s fine to lose a couple of updates during server restarts.

3. Throughput (NFR-1) - Pub/Sub handles real-time updates with high throughput and ideal for collaborative editing use cases.

The documentId can be used as a Pub/Sub topic. All the server instances handling a document, would subscribe to this topic.

Any server receiving an update would add the update in the topic. All other server instances would immediately receive the latest mouse pointer update and process it.

Further, the server instance can unsubscribe if it doesn’t have any client working on a particular document. And subscribe as soon as it finds a new client working on a given document.

The following diagram illustrates how the mouse pointer update originates from one client and gets propagated to other clients working on the same document.

Trade-offs

We will now justify the choices that we have made in the above design. In system design, there is no right or wrong answer, and we choose the best option that meets our requirements.

Client-Server communication - WebSockets.

HTTP alternatives such as polling would introduce additional latency and an overhead. This would result in inefficient resource usage and slow updates (violates NFR-4).

Hence, WebSockets would be suitable for our use case of collaborative editing with frequent updates that need minimum latency.

Trade-off decision: Given the need for low-latency, real-time updates, WebSockets are the optimal choice for client-server communication in this use case.

Session data - Redis Streams

Redis Streams provide strong reliability guarantees even if a server goes offline. The server can process missed messages even after restarts.

However, they require more overhead compared to Pub/Sub, making them less ideal for high-frequency, non-critical updates like mouse movements.

Trade-off decision: For session state data, where reliability and message delivery guarantees are critical, Redis Streams are the best solution despite the additional overhead. Since the frequency of connections/disconnections is much lower (about 1/1000th of mouse pointer updates), the overhead is manageable.

Mouse pointer updates - Redis Pub-Sub

Redis Pub/Sub allows for real-time broadcasting of messages, making it ideal for high-frequency, transient data like mouse pointer movements.

Pub/Sub doesn’t guarantee reliable delivery of messages in case the subscribers go down. Hence, subscribers would miss the updates.

Trade-off decision: Given the high frequency of mouse pointer updates and the need for speed over durability, Redis Pub/Sub is the best choice for this type of data exchange.

Serialization format

The mouse pointer and session update data can be exchanged between client-server using JSON. JSON is human-readable and easier to debug. However, with large volume of messages, the CPU consumption would spike.

Binary serialization is faster and uses less CPU. Hence, a compact binary serialization would be appropriate for this use case.

Trade-off decision: Given the high frequency of mouse pointer updates, we must prioritize performance over the convenience of debugging, making binary serialization the better choice.

Redis Commands for Mouse pointer updates

Flushing Redis commands: The server can push every mouse pointer update to the Redis Pub-Sub. However, this would be a resource intensive operation. And the system wouldn’t scale to handle large throughput of messages.

Batching Redis commands : Batching the Redis PubSub commands within specific intervals would result in better resource utilization.

Trade-off decision: Although batching results in slight delay of messages (because commands are sent together at intervals), but this is outweighed by the reduction in CPU usage. Since mouse movements don’t require millisecond-level precision, this was an acceptable trade-off. (NFR-4)

Conclusion

Now, we have a collaborative editing system that shows the real-time mouse pointer movements of the editors or viewers. The design meets all the functional and non-functional requirements that we had listed in the beginning.

In a nutshell, the design addresses the following aspects :-

1. Client-Server communication - Uses Websockets for bi-directional communication and low-latency. The LoadBalancer ensures high availability and even traffic distribution.

2. Session data - The system relies on Redis Streams to manage and propagate the session data. It ensures that all the servers have a consistent view of users viewing a given document.

3. Mouse movements - Redis Pub/Sub provides low-latency, real-time & high-frequency mouse pointer updates.

In this article, we learnt a methodical approach to tackle any ambiguous system design problem. You can remember this and apply the technique in your next system design interview.

We came up with the above design assuming that the client would send 3 updates every second. Do you think the above design would scale in case we want to send 60 updates ?

Are there any bottlenecks in the design to handle 60 updates/sec ? If yes, how can we overcome them ? Leave your thoughts in the comments below.

For more information of Redis Streams, you can refer my article here.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Share

References

1. How Canva designed real-time mouse pointer updates

In the past decade, streaming data has seen a meteoric growth. From mobile phones, sensors, tablets, to home pc, streaming data is everywhere.

This explosive increase has driven the development of several software solutions designed to manage and process streaming data efficiently. In 2011, LinkedIn developed Kafka, which was later open-sourced. Similarly, Redis introduced Redis Streams in version 5.0.

Recently, a friend of mine was giving a system design interview. He was asked a question related to Task queue management.

He proposed a solution using Redis Streams, but when the interviewer asked him to justify his choice over Kafka, he struggled to explain the trade-offs. He couldn’t clearly explain the trade-offs and got rejected. The feedback mentioned that he lacked the technical depth and needs to work on it.

The lesson here is clear: mastering the fundamentals is essential. Knowing the latest technology doesn’t guarantee success in system design interviews if you can’t articulate the reasoning behind your choices.

In this article, we will dive deep into Redis Streams. We'll start by covering its core concepts and architecture, followed by an analysis of its pros and cons. Subsequently, we will compare and contrast Redis Streams with Redis Pub-Sub and Kafka and go through real-world examples.

By the end of this article, I am sure that you would understand the working and application of Redis Stream. And also confidently justify the choice in your next system design interview. Let’s dive in.

Boost Your Skills with Educative (Plus an Extra 10% Discount!)

Ace your coding interviews with Educative’s proven prep paths used by engineers at Google, Amazon, and Meta.
Get an exclusive 10% discount when you sign up through this link.
No setup, no distractions — just structured learning, real problems, and smart solutions.

Redis Streams Concepts

Streaming data is the data that is generated continuously. A popular example is the logging and metric data generated by software systems in enterprises.

The data has high throughput & is mostly used by analytics and post-processing workloads. Apache Kafka, Apache Flink, Amazon Kinesis, etc are some solutions for managing streaming data.

Redis Stream

Redis Stream is a data structure that persists real-time streaming data. It stores the data in an append-only fashion. Any data added to a Redis Stream is immutable.

Stream Entry

Redis Stream consists of series of stream entries. Each stream entry has a unique id (timestamp) and consists of multiple key-value pairs.

Adding a stream entry is a O(1) operation since it's append-only operation. Similarly, fetching an entry from the stream is a O(K) operation where K is the length of the id. Since, K is constant, fetching an entry is a constant operation. Internally, Redis Streams store the entries in a Radix tree.

Below diagram illustrates a Redis Stream Customer Orders consisting of a series of entries.

The Streams are stored in-memory and are backed up through persistance (AOF or RDB file). Redis Streams are high-performing since all the operations are in-memory operations and avoids any disk I/O.

Producer & Consumers

Multiple producer applications can add data to a stream. And several consumer applications can read data from the same stream. This decouples producers from consumers.

Reading & Deletion

Redis streams work on a pull model. Consumers are responsible for pulling the stream entries.

Once a stream entry is added to a stream, it exists until it’s explicitly deleted. Unlike pub-sub, the messages are not lost if the consumers are not connected.

Multiple consumers can read from a given stream without any side-effect on other consumers.

Consumer groups

A single consumer reads the entries from a Stream sequentially. This can lead to slowness in processing.

For scalability, Redis Streams introduces a concept of Consumer groups. Consumer groups are made up of multiple consumer applications.

When a consumer group is assigned to a stream, multiple entries can be picked up in parallel by each consumer and processed. This improves the overall throughput of the system.

Note :- Redis guarantees that a single message wouldn’t be processed multiple times by multiple consumers within a consumer group.

The following example illustrates a simple messaging system that leverages Redis Streams :-

Now, that you have a good understanding of the fundamentals, let’s look at the advantages and disadvantages of using Redis Streams.

Advantages

• High performance - In-memory storage results in low latencies and high throughput. Log-structured data model allows efficient reads and writes to Redis Streams.

• Flexible data model - The data model is schema-less and clients can store any set of key-value pairs.

• Integration with other Redis Data Structures - Streams can seamlessly integrate with other Redis Data Structures such as Lists, Sets and Hashes.

• Ease of use - Redis Streams provide APIs, CLIs and SDKs in multiple languages to manipulate the data structure.

• Time-series data - Each stream entry contains an id field that has timestamp. All the stream entries are ordered and hence it makes it suitable for applications that need time-series data management.

Disadvantages

• Durability - As the data is stored in-memory, there is a risk of data loss. Hence, shouldn’t be preferred if you require strong durability guarantees.

• Limited message retention - Due to memory constraints, you can’t infinitely retain all the messages in a stream. Clients have to purge or configure Redis Streams to automatically purge the data.

• Complex event processing - Redis Streams don’t support complex event processing capabilities such as the ones provided by Apache Kafka or Apache Flink.

Since you have understood the pros/cons of Redis Streams. Let’s now contrast Redis Streams with Redis Pub-Sub and Apache Kafka.

Redis Streams Vs Redis Pub-Sub

Redis Pub-Sub

Redis Pub-Sub (Publisher-Subscriber) is a data structure that stores the messages sent by producers and immediately sends it to consumers. In case a consumer doesn’t exist, then the message gets lost.

Below diagram illustrates how Redis Pub-Sub works.

Summary of differences

The below table compares the different features and its support in Redis Streams & Pub-Sub :-

Redis Streams Vs Apache Kafka

Apache Kafka

Apache Kafka is an open-source distributed event-streaming platform for building real-time data pipelines. There are many commonalities in the working of Kafka and Redis Streams, such as consumer groups, append-only data insertion, etc.

However, there are subtle differences in their architecture. Apache Kafka is designed to process high volumes of data with low latency. Additionally, it’s a distributed commit log and allows consumers to read and replay messages whenever needed.

Unlike Redis Streams, Apache Kafka guarantees data durability and protection from data loss. It also integrates with other stream processing frameworks, such as Apache Flink and Apache Spark.

Summary of differences

The following table captures the key differences between Redis Streams and Kafka:-

When to use Redis Streams ?

Redis Streams are highly performant and can be leveraged for a wide array of use cases. Here is a list of use cases where Redis Streams are suitable :-

1. Task Queues - Simple task queues, such as sending email or SMS notifications, can be implemented via Redis Streams. The processing can be scaled by increasing the consumers in the consumer groups.

2. Session management and activity tracking - Short-term user interactions, such as tracking online/offline users, can be managed through Redis Streams. Using Streams results in better performance due to in-memory operations.

3. Lightweight event sourcing - Redis Streams can be used for event sourcing, where events are written first and processed asynchronously later. Its in-memory nature makes it suitable for low-latency event processing. However, it’s not suited to handle high-throughput events.

4. Real-time analytics - Redis Streams are useful for real-time analytics that don’t require long-term data storage. For example, showing real-time metrics such as CPU usage, where only recent data is needed.

Conclusion

Redis Streams is a versatile data structure that allows you to store a stream of events in-memory. It’s highly performant and offers a flexible data model for storing data.

Redis Streams is useful for developing asynchronous applications and decoupling producers from consumers. Redis Streams provides consumer groups to scale the processing of messages in a given stream.

The downside of using Redis Streams is that it doesn’t guarantee data durability and has limited message retention. Furthermore, it doesn’t support complex event processing like Apache Kafka and Apache Flink.

If message retention is needed, we must choose Redis Streams over Redis Pub/Sub. Redis Pub/Sub uses a fire-and-forget model and immediately deletes messages if they aren’t consumed by the consumers.

Apache Kafka is suitable for high-throughput applications that require stronger durability guarantees, while Redis Streams is suitable for applications that require high performance with weaker durability guarantees.

Earlier, I mentioned how my friend got rejected because he couldn’t explain why Redis Streams was the best fit for task queue management. Do you think Redis Streams would be the right answer for that problem? Let me know your thoughts in the comments.

Case Studies

Do take a look at the following case studies to further deepen your understanding and improve your system design skills:

1. Canva’s collaborative editing design - Shows how they leveraged Redis Streams for session management and Redis Pub/Sub for mouse pointer tracking.

2. Arcjet’s use of Redis Streams - Arcjet deliberated between building their own message processing system and leveraging Apache Kafka. They eventually chose Redis Streams, and it was scalable enough to meet their needs.

Also, if you haven’t used Redis before, you can install the Docker image of Redis and experiment with Redis Stream commands. You can find the documentation here: Redis Streams doc.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

On 19th July 2024, the world witnessed one of the biggest IT disruptions in recent years. Corporations worldwide reported outages and disruptions, with Windows computers displaying the dreaded Blue Screen of Death (BSOD) Error.

The outage impacted sectors such as airlines, banking, trading, media companies and many more. It was confirmed that there was no security incident or cyberattack resulting in this disruption.

The outage was due to a software update of Falcon, a tool built by the cybersecurity firm CrowdStrike. The latest update resulted in a crash of Windows operating system and resulted in worldwide outage.

In this article, we will dive deep and understand the issue. We will start by demystifying the Blue Screen of Death by revisiting the operating system fundamentals. Later, we will look at CrowdStrike’s architecture, the faulty release and the mitigation strategy. We will conclude by going over some key learnings from this incident.

Subscribe now

Blue Screen of Death (Blue Screen of Death)

If you use Windows, you've likely encountered the Blue Screen of Death (BSOD). This error screen indicates that the system has malfunctioned, usually due to hardware or software issues.

After the reboot, everything works seamlessly. But, do you know why the BSOD appears ? And why does it appear only on Windows machine and not on Linux/Mac ? Let’s understand this in detail.

Operating System

Before going into the details, we will revisit some operating system concepts. This would help us grasp complex details in an easy manner.

The operating system manages hardware components like memory, CPU, and I/O devices, with the Kernel at its core.

The Kernel is responsible for primary functionalities such as :-

1. Memory management - Allocating/Deallocating memory for the running processes. Tracking the memory used by the processes.

2. Process management - Managing the process lifecycle and ensuring two or more processes run in the memory without any issue.

3. Device management - Communication with devices like monitor, mouse, keyboard, printers, etc.

4. File System management - Organizing the files on the hard drive or any external storage.

The processes running on any computer/machine run in two modes - a) Kernel Mode b) User Mode.

Kernel Mode

This mode has highest privilege and can directly interact with all the hardware. Operating system processes run in this mode.

It’s a critical mode and any errors in the code can result in system instability and eventual crash. Hence, code has to be written meticulously and should be error-free.

As users, we don’t directly deal with processes that run in the kernel mode. However, you can view them in the Windows Task Manager. The following snippet shows processes such as Host processes and NT Kernel & System running in kernel mode.

User Mode

The user mode has least privilege and it can’t directly interact with the hardware. It uses the operating system to get the data from any hardware device.

The user mode processes call system calls for sending/receiving data to/from the hardware. For example: User programs call fopen system call to open a file on the hard disk.

The operating system ensures that the user process doesn’t access any memory that’s not assigned to it. In case a user process accesses memory belonging to a different process or a kernel process, the operating system sends a SIGSEV to the process. And the process finally terminates and the user sees a SEGFAULT (Segmentation Fault).

Applications that you use daily such as web browsers, MS Excel, MS Word, games, media players like VLC, etc run in user mode.

What causes BSOD ?

BSOD is only shown for Windows devices. It indicates that the system has malfunctioned. It could be either due to a hardware or software issue.

The following are few reasons for BSOD :-

1. File system corruption - In case the files required by the Kernel are corrupted, the system can crash.

2. Device driver issues - Incompatible or corrupted device drivers can lead to communication errors. And eventually cause BSOD.

3. Software bugs - Bugs which modify critical Kernel data structures, and access forbidden memory locations can bring the system to a halt.

4. Malware - Malware can interfere with the system processes leading to a BSOD.

In most of the cases, we restart the system and as expected it starts functioning. We can often troubleshoot the issue by looking at the system logs.

To ensure smooth functioning of operating system, it is essential to prevent scenarios such as kernel’s data corruption, accidental modification of kernel’s memory, etc.

Mac/Linux use different Kernel, data structures and error handling mechanisms. In case of an OS crash, both OSs use Kernel panic. Unlike Windows’s BSOD, Mac/Linux show a screen with error messages.

Now, that you understand BSOD, let’s understand the reason for 19th July’s worldwide BSOD.

19th July’s BSOD

72% of the users in the world are windows users. Windows devices use tools developed by CrowdStrike(CRWD) for cybersecurity.

CrowdStrike has developed a suite of cloud-native tools. One such product in their suite is known as CrowdStrike Falcon.

CrowdStrike Falcon offers the following features :-

1. Endpoint security - It detects and prevents malware, ransomware and other threats.

2. Extended detection and response (XDR) - Helps in investigating security incidents, identifying root causes, and taking the right steps.

3. Cloud workload protection - It monitors any malicious activity in cloud environments for Azure, AWS , and Google Cloud Platform (GCP).

CrowdStrike Falcon architecture

Falcon is installed on all the devices and runs as an agent (sensor). It continuously monitors the device activity.

CrowdStrike runs a cloud platform that collects data from all the Falcon agents. This platform also serves as a control plane for CrowdStrike to view the threats and security incidents.

Falcon agents run as background processes. They collect the security data from the device and send it to the CrowdStrike platform.

The following image shows the architecture (for representation only) of Falcon :-

Falcon Agent runs as a kernel process since it has to monitor activities such as -

• Device driver activity

• Network traffic

• Restricted file accesses

The above activities require highest privileges and as a result, the Falcon Agent runs in a kernel mode.

Unlike windows updates, Falcon Agent updates itself silently in the background. It gets the latest update from the cloud platform.

As seen from the above architecture, the Falcon Agent gets the latest update, restarts and runs the latest version.

Faulty deployment

Recently, CrowdStrike deployed latest version of Falcon Agent. The file C-00000291*.sys was updated and the running agents downloaded the file.

After consuming the update, windows machines started crashing. The issue wasn’t observed on the machines where the agent wasn’t updated.

This was observed worldwide and brought most of the economic activities such as Air travel, Hospitals, Stock trading, etc to a standstill.

CrowdStrike discovered that the issue was due to faulty update in the C-00000291*.sys file. A fix was deployed and the file C-00000291*.sys with timestamp of 0527 UTC had the fix. While the one with 0409 UTC was the problematic one. Reference - Falcon update for Windows host.

CrowdStrike hasn’t published the exact technical details. However, I suspect it could be due to incompatibility between the latest agent update and it’s interaction with the Kernel.

The latest Agent might have a bug that might have written to Kernel’s memory causing corruption. This might have resulted in the crash. And since this was repetitive, the fault might have been in Falcon Agent’s initialization.

Immediate Fix

CrowdStrike mentioned the steps to mitigate the issue immediately. They deployed a fix and machines using the latest Agent wouldn’t face the same issue.

They also stated that users should manually delete the faulty C-00000291*.sys file from the machines and reboot them. Similar workarounds are suggested for workloads running on cloud.

According to me, it would take a couple of days for the fix to reach all the machines worldwide. Also, since it requires manual intervention, non-technical users would face a challenge. But, hopefully, it will get resolved soon.

Long-term mitigation

On 19th July, we witnessed how a simple software update can disrupt our daily life. In the future, such issues must be avoided at all costs.

In my opinion, there are two angles to the 19th July’s BSOD issue :-

1. Agent updates - The issue could have been prevented if the update wasn’t rolled out to all the users.

2. OS design - Linux/Mac didn’t face the same issue. However, it impacted most of the Windows devices. A better design of OS and its kernel modules would have prevented this havoc.

Agent updates

Since the agents are installed on the end-user devices, it makes the deployment more challenging. Unlike cloud-services, these changes can’t be rolled-back automatically. And also the blast radius is huge as it encompasses all devices in the world.

However, in the future, CrowdStrike can follow a strategy similar to blue-green deployment. Instead of all the agents receiving the update, a set of computers (green environment) would receive the update. It would bake for few days in the green environment and then the change would roll-out to the blue environment (end-user devices).

Similarly, since the Agent runs in the kernel mode, the testing must be rigorous. Also, the code reviews must be done thoroughly and critically. There should be enough guardrails to prevent any accidental updates to the source operating system.

OS design

Windows users have a history of dealing with BSOD. Over the years, Windows has become stable and users are seeing BSOD less frequently.

One of the primary reasons for BSOD can be attributed to the design of Windows OS. While kernel extensions provide low-level access, hardware interaction and customization, the downside is instability.

Mac followed a similar approach until macOS 10.15 (Catalina). Since then, Mac introduced system sxtensions. System extensions run as User processes and have limited access to the core kernel. This prevents unexpected crashes, and reduces security vulnerabilities.

Windows can benefit from following Mac’s footsteps in the future. It would improve overall security and stability.

Additionally, the Windows kernel is implemented in C/C++. These languages are prone to errors such as buffer overflows, dangling pointers, & Null pointer references. The errors lead to process crashes.

Microsoft has started rewriting its kernel in Rust. Rust’s strong typing and ownership system make it memory safe. In the future, 19th July-like BSOD issues can be prevented with the Windows version written in Rust.

Conclusion

On 19th July, the majority of systems running Windows faced BSOD. Moreover, cloud services running on Windows servers were impacted, disrupting many clients relying on these services.

The root cause of the issue was a faulty update of CrowdStrike’s Falcon Agent (sensor). The latest update caused an OS crash, and users started seeing BSOD.

CrowdStrike identified the issue and rolled out a fix. They also suggested ways to mitigate the issues on their website. Some users had to manually reboot their machines and delete the problematic file from the update as an immediate fix. In a couple of days, all systems are expected to function normally.

As a software developer, key takeaways from this incident include :-

1. Before deployment, assess the impact of the change and the blast radius.

2. Adopt blue-green deployments to minimize the disruption to end-users.

3. Minimize the manual steps for users in case of software roll-backs. Roll-backs must be seamless, and user functionality shouldn’t get impacted.

4. When dealing with the OS kernel, follow best practices for testing and code reviews.

5. If developing an OS, use memory-safe languages like Rust over C/C++, and use concepts such as system extensions that limit a process’s access to the OS Kernel.

Let me know in the comments below what your views are and how CrowdStrike or any other company could prevent such issues in the future. Also, restack and share the post if you liked it.

Share

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

• Your support helps keep this newsletter free and fuels future content. Consider a small donation to show your appreciation here - Paypal Donate

Reference

1. CrowdStrike Statement on Falcon Content Update for Windows Hosts

2. CrowdStrike update causes major global IT outage

Hello Readers,

Do you struggle to understand the intricacies of distributed systems, databases, and programming languages? Are you curious to learn how companies design and build software at scale ?.

My free newsletter breaks down complex tech concepts, architectural trade-offs, and design decisions into clear, actionable steps. Get weekly articles to level up your tech skills and become a better developer.

If you haven’t subscribed to my newsletter yet, do subscribe -

Subscribe now

Redis

In 2009, an Italian software developer named Salvatore Sanfilippo was working on building a real-time web log analyzer. His application couldn’t scale as database was the primary bottleneck.

Hence, he decided to develop a solution that could serve the data from memory. He implemented the Proof of Concept in TCL and later ported it to C. This was the first version of Redis (REmote Dictionary Server).

Later, Salvatore open-sourced the project and posted it on Hacker News. The project received a lot of traction & enterprises such as Github & Instagram adopted it. Today, big tech companies such as Amazon, Microsoft, Google, Meta, etc use Redis in production.

What started as an in-memory cache, is now used in a wide array of industries such as gaming, finance, e-commerce, social media, advertisements, etc.

Redis is now more than an in-memory cache. It has support for message queues, probabilistic data structures, Geospatial indexes, vector-search capabilities and much more.

Key Features of Redis

Latency

While traditional databases store the data on the disk, Redis stores the data in-memory. Accessing the in-memory data is 10-100x faster than disk access.

The below image illustrates and compares Database lookup vs a Redis lookup.

As a result, Redis improves the system’s performance by reducing the latency of every data access. This makes Redis the right candidate for caching.

Availability

Redis supports data replication through leader-follower architecture. Clients write the data to the leader which replicates them to the followers.

In case a leader goes down, one of the follower becomes the new leader and starts serving writes. This ensures high data availability and avoids system downtimes.

Fault-tolerance

Since the data stored in the memory is transient, Redis minimizes the data loss by adopting the following techniques :-

1. Append-Only Files (AOF) - It persists all the received commands in a file. In case the server restarts, the latest state is constructed by running all the saved commands.

2. Snapshots - Redis takes periodic snapshots of its state. The snapshot is persisted in a binary file. On server restarts, the state is restored by parsing the binary file.

These features safeguard Redis from unexpected crashes, deployments, restarts, and version upgrades.

Scalability

Redis Cluster consists of a set of servers which store the data. It uses sharding (consistent hashing) to efficiently distribute the data among different servers.

As a result, the data size is not limited by the memory size of a single server. The Redis Cluster can scale & support additional data by adding more servers.

Similarly, it can handle spike in the traffic by evenly distributing the traffic across the shards and replicas.

Basic data structures

Following are the basic data structures supported by Redis :-

1. Strings - Can store upto 512MB data. It can store JSON, XML, and raw strings.

2. Lists - To store collection of objects. Exposes APIs to push/pop objects.

3. Sets - To store unique string and perform O(1) lookup on strings.

4. Hashes - Similar to Hash maps and can store key-value pairs.

5. Sorted Sets - To store collection of objects in a sorted manner.

Most of the relational and non-relational databases don’t support a wide variety of data structures. Hence, applications have to do an additional data model conversion while storing the data in databases such as PostgreSQL, MySQL, Cassandra, etc.

If your application uses Redis, it can directly use Redis’s APIs and pass data stored in complex data structures such as Sorted Sets, Sets, Lists, etc.

Advanced data structures

In addition to basic data structures, it has support for the following complex data structures :-

• HyperLogLog - It is probabilistic data structure to estimate count of objects in a set.

• Bitmaps - It is used for storing & manipulating individual bits.

• Geospatial Indexes - It stores geo-location and APIs to lookup using location & distance.

• Streams - It is log-based data structure for append-only log data.

• Vectors - Vector data structure is a mathematical representation of data as a list of numbers. It allows AI algorithms to perform calculations and make efficient decisions.

For more details, you can go through the official Redis documentation and play around with the different data structures.

These features enhance the versatility of Redis for several real-world applications. Now, let's explore some real-world applications of Redis.

Real-world applications

Each of the Redis data structure can be used to solve a unique distributed systems problem. Here are some data structures with their real-world application :-

1. HyperLogLog - It is used for estimating the unique visitors or concurrent users on a website.

2. Bitmaps - They are used for tracking the online/offline status of a large user base.

3. Geospatial Indexes -Finding near-by restaurants, users, or stores. Additionally, useful for real-time tracking and spatial analysis.

4. Streams - Used for Log aggregation, and real-time feeds for social media sites.

5. Strings - For caching the data and improving the read/write performance of database queries.

6. Vectors - AI/ML applications such as Recommendation systems, Anomaly Detection, Search & Discovery, etc store data in form of vectors.

Conclusion

Redis started as a simple key-value data store in 2009 and has evolved into a versatile tool capable of functioning as a database, message queue, cache, and much more. Its rich support for various data structures—such as Lists, Sets, HyperLogLog, and Geospatial indexes—sets it apart from other data stores.

This versatility makes Redis an essential component in many distributed systems, solving problems like distributed locking, rate limiting, and finding nearby users. With the rise of AI, Redis has also adapted to serve as a vector database.

One standout feature of Redis is its in-memory data storage, making it ideal for performance-critical real-time systems. However, in 2024, Redis switched to a proprietary software license.

How do you think this shift will impact Redis and its adoption in the tech community? Will it continue to thrive, or will alternatives rise to take its place? Share your thoughts in the comments below.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

References

1. Redis wikipedia

2. Redis data platform

Hi Folks,

Do you struggle to understand the intricacies of distributed systems, databases, and programming languages? Are you curious to learn how companies design and build software at scale ?

My free newsletter breaks down complex tech concepts, architectural trade-offs, and design decisions into clear, actionable steps. Get weekly articles to level up your tech skills and become a better engineer.

If you haven’t subscribed to my newsletter yet, do subscribe :-

X.com (formerly Twitter) receives more than 400 billion events daily. The events include tweet views, likes, retweets, engagements, etc. The events are critical as they are used by Ad-serving systems & determine the company’s revenue.

There are several components that work behind the scenes to process these events. The events are finally aggregated, and stored in a data store. And multiple downstream consumers rely on this data store.

Scale, performance and accuracy are three important pillars while building an event processing system. Twitter’s old Lambda architecture faced multiple challenges while handling the exponential growth in events. Hence, they adopted new Kappa architecture & improved the accuracy, scalability and performance.

In this article, we will dive into Twitter’s architecture that processes 400 billion events daily. We will look at the old Lambda architecture, why it couldn’t scale & how the new architecture addressed the bottlenecks.

This article uses this blog from Twitter that was written in 2021. Twitter would have made multiple changes in the architecture since then. Hence, the article doesn’t talk about the latest state and it’s evolution over last 3 years.

Lambda architecture

The lambda architecture consisted of batch-processing and real-time pipelines. These two pipelines store the data in different data stores.

Different consumer services query the Query Service for fetching the aggregated data. The Query service aggregates the computed data from different sources and returns it to the consumer services.

We will go over the different components comprising the architecture :-

1. Batch processing

   1. Data sources - The raw data exists in the form of logs stored in HDFS. This contains event data for tweet views, clicks, timelines, etc.

   2. Scalding - Scalding platform enables developers to write data pipelines for batch processing. Scaling pipelines read the raw data and then perform different types of aggregations.

2. Stream processing

   1. Data sources - The streaming data is written to different Kafka topics.

   2. Heron - Heron reads & processes the streaming data from the Kafka topics.

3. Data stores

   1. Manhattan - The computed batch data is stored in Manhattan, a distributed database.

   2. Nighthawk - It is Twitter’s distributed cache used for storing the processed streaming data.

Challenges with Lambda architecture

Although, the above architecture was able to support the scale but it ran into several limitations. Following were the challenges with the old architecture :-

1. Performance - The performance of the system was dependent on the throughput of Heron. Heron nodes used to get impacted due to high garbage collection costs. Slowness of Heron consumers increased the overall processing latency.

2. Accuracy - The system didn’t guarantee data durability. And Heron node restarts would result in data loss. This would impact the data accuracy. Additionally, it didn’t process late or out-of-the order events.

3. Complexity - Two separate pipelines added to the architectural complexity and increased maintenance costs.

4. Operational overhead - Developers often had to intervene to restart Heron nodes to speed up real-time data processing.

Kappa architecture

To address the challenges, Twitter decided to adopt the Kappa architecture. This architecture used stream processing pipelines only. Further, they leveraged multiple Google cloud services for data processing.

The architecture consists of two parts -

1. Twitter data center - It hosts components and services within Twitter.

2. Google Cloud - It consists of all the Cloud services used for stream processing.

Twitter Data Center

1. Event Mappers - These were part of preprocessing pipelines. They transformed, re-mapped the fields and sent it to Kafka topics.

2. Event processors - They were responsible for data model conversion from internal representation to Google pub-sub format. They streamed events with at-least-once semantics.

3. Twitter LDC Query Service - This service served the queries from different consumers by fetching the data from different cloud data stores.

Google Cloud

1. Cloud Pub/Sub - The pub/sub relays the data pushed by Twitter’s Event processors to Dataflow. It guarantees no data loss in the event processing journey.

2. Dataflow - Dataflow workers dedup (remove the duplicates) and aggregate the data.

3. Bigtable/BigQuery - The aggregated count for the query key is written to the BigQuery and BigTable.

Kappa’s performance

The new architecture reduced the costs to build and maintain a separate batch processing pipeline. With no data loss guarantees, it ensured high accuracy.

Following were some key improvements observed in the Kappa architecture :-

1. Events processed - Handled more than 4 million/second

2. Throughput - More than 1 GB/sec. It was 10x improvement over Lambda architecture.

3. Latency - The approximate latency was 10 sec. This was significant improvement over Lambda’s latency of 10 sec-10 mins.

Besides, Kappa added support for late event handling and guaranteed no data loss. This improved the overall accuracy of the system.

Conclusion

Twitter’s original event processing system was based on Lambda architecture that combined batch-processing and stream-processing system together. There were several limitations of the Lambda architecture such as performance, accuracy, complexity and operational overhead.

To overcome Lambda architecture’s limitations, Twitter decided to adopt the Kappa architecture. This architecture adopted real-time streaming of events.

The event pre-processing and transformation was done via services in Twitter’s data center. And they leveraged Google Cloud services for event aggregation and storage.

Here is how the Kappa architecture addressed Lambda architecture’s limitations :-

1. Performance - It reduced the overall latency to ~10 sec from ~10 sec-10 min.

2. Accuracy - Safety against data loss, at-least-once delivery semantics, event deduplication & processing of delayed events improved the accuracy.

3. Complexity - In contrast to Lambda architecture, Kappa required only real-time processing pipeline. This resulted in architectural simplification.

4. Operational overhead - It eliminated the need for developer intervention to restart components (Heron nodes) and fix issues. This improved the developer productivity.

Before you go:

• ❤️ the story and follow the newsletter for more such articles

• 🔔 Follow me: LinkedIn, Twitter, Medium

References

1. Processing billions of events in real-time at Twitter