Continuing my systems design series this week with a new article on fault tolerance. If you’re doing a systems design interview soon or you just want to know more about web-scale applications continue to follow along! I’d also suggest you watch this video which incorporates many of the concepts of applying all the concepts we’ve studied so far.

Happy reading!

Fault tolerance

The self-proclaimed pessimistic chapter of the book, chapter 8 covers fault tolerance in distributed systems. Let us briefly introduce why things break in an internet-scale system:

• Commodity machines break. Since internet applications are about scaling out instead of up, each node in the cluster is cheap. Cheap things break more frequently.

• The Internet Protocol is unreliable. There is no guarantee that things will move as intended. Everything is sent over the wire in packets. Traditional telephony networks send everything over in circuits. This lack of a reliable, consistent connection presents problems. The request could be lost or queued. The node accepting the request could stop responding or fail altogether. Worse, your request could have made it all the way to the node and back but was lost on the way back to you or will arrive later than you want it to.

  • We add in TCP to make it more reliable. The downside is there are variable network delays since that queuing and reliability slows the system down.

  • You can trade TCP for UDP if you need more speed. If you cannot afford delays in data and can afford less reliability, consider using UDP. A Zoom call is a good example of when UDP is a better protocol than TCP.

This is in contrast to vertically-scaled systems that rely on beefy supercomputers. These systems often act as if they were single-computer applications. Therefore, we will not be covering this in the book or in the notes.

The crucial thing to internalize is that faults happen in the real world all of the time. The best anecdote from the book is how a shark bit some network cables and took down parts of Amazon EC2. Even a company as powerful as Amazon cannot prevent the almighty shark from screwing up network connectivity.

How to deal with faults

If we know faults will happen, then it is vital to be proactive in dealing with them.

Error handling is a simple and effective way of staying on top of faults. Keeping your customers aware of issues may only require a browser refresh to correct the problem.

Heartbeats help detect faulty or down nodes. A load balancer (like HAProxy or Nginx) will periodically ping nodes to ensure they are up to send them traffic.

Leader election is a strategy to elect new write replicas when leaders are down in replicated clusters.

Timeouts are the gate to try all of the above strategies. For customers, the only way to reliably detect faults is to inform them that too much time has elapsed to process their request. In that time, you will have tried the things above. Once the timeout has been exceeded, it is safe to assume that a fault has occurred.

How do you choose a sensible timeout? Measure your response times and variability. Then, choose a number. Unfortunately, there is no standard timeout interval that will work for everyone. Apache Cassandra uses the Phi failure-accrual algorithm to adjust gossip protocols when they check with nodes to determine if they are down. TCP does its own retransmission, but there is not much here for you to take away for your own applications.

Timing

Timezones are hard. Dealing with time, in general, is difficult. All computers have clocks, but they are rarely in sync with each other. How do you ensure that the time or duration of an activity is accurate?

Sync them. Things called NTP servers help ensure that the clock you see on your computer is actually correct. If it isn’t, the NTP server will send back the correct time and update your clock. The problem is, your clock could hop backward. If you sent a request at 10:00:01 and it was returned at 10:00:00, you have a problem. That is why these kinds of clocks, called time-of-day clocks, are not reliable for measuring time duration.

Count monotonically. To safeguard against a standard clock, a monotonic clock is also available on a computer. It is basically a giant timestamp counter that always counts upward, regardless of time syncing issues. You can still use an NTP server to adjust the monotonic clock, but it does not count backward. If there is a syncing issue, you simply delay when the next value is incremented. This is generally the preferred method for distributed systems.

Even with both mitigation strategies in place, no system is perfect. There is a network delay from the NTP server to your local clock, so they will never fully be in sync. Firewalls, leap seconds, virtual machines, and physical hardware clock malfunctions all contribute to perpetual inaccuracies between the actual time and the time posted on your machine. As long as you accept this premise, you can achieve reasonably high accuracy with the times you use it.

Strategies around timing issues

With all of these issues around keeping track of time, there are really only 3 solutions that are worth implementing:

• The last write wins. Cassandra and Riak use this strategy to determine the most recent write to a database. But this suffers from the same basic problem: what is recent? If it is all relative anyway then you could still have two nodes that wrote to the DB with the same timestamp but one node’s clock is off.

• Provide times with a delta. Google Spanner uses this with its TrueTime API, but good luck using it outside of Google.

• Limit certain functions that would cause time delays and pauses. Garbage collection is a notably slow process that can hang a system. Ensuring there is enough memory to handle other functions when garbage collection grows unwieldy ensures some fault tolerance.

The message you must remember is that time is relative, and you cannot achieve real-time. Real-time, in the truest sense of the word, is reserved for embedded systems so things like your airbag can deploy when it needs to. Real-time, as it is used for web-scale, is a relative term for feeling instant even if it isn’t actually an instant operation.

Trusting a faulty system

Now that we know systems are unreliable, we need to know that the actors can be trusted. The book assumes they can be and that each player in the distributed system acts with honest intention. This is where the old security saying of trust but verify comes into play.

Quorums, as we discussed in the data replication chapter, are a way of obtaining votes from all nodes to figure out what the majority believes about the system during a discrepancy.

If you have 3 children and ask, “who stole the cookies from the cookie jar,” you’re likely to believe it is child 2 if both 1 and 3 point to 2.

What if one of them is lying? This is the basis for the Byzantine Generals’ problem. Systems that safeguard against lying actors are Byzantine fault-tolerant. They are only needed for mission-critical systems like aerospace or the blockchain. Internet web-scale systems do not require Byzantine fault-tolerance (usually in the form of 2/3rd majority vote rather than a simple majority vote), and we can improve fault tolerance with a few simple tricks:

1. Require checksums. TCP and UDP offer these and ensure that corrupted packets must be retried to ensure data is transferred correctly.

2. Permission, sanitize, validate, error-handle, and output-escape your data. This blog post beautifully summarizes a recipe to eliminate bugs from your code when ingesting data. Bugs are a source of lying since the system behaves unexpectedly through faulty code.

3. Add redundancy to your NTP servers. Replication of time synchronization server checks ensures that the majority timestamp is the accurate and correct one.

4. Build systems around crash recovery and partially synchronous system models. System models take many forms and shapes. One will not see the strictness of a completely synchronous, Byzantine fault-tolerant system in the real world. But you also don’t want a fully asynchronous model where even the slightest perturbance shuts down the system. Striking a middle ground with your system model is a sensible and useful real-world approach.

The next chapter will investigate algorithmic approaches to handling real-world fault tolerance with system models that handle crash recovery and strive for partially synchronous updates.

This quote at the end of the chapter summarizes how I would recommend the litany of strategies in this book to a more junior engineer:

If you can avoid opening Pandora’s box and simply keep things on a single machine, it is generally worth doing so.

It is easy to look at a book like this and think of all the neat technologies you could use on your current project. Resisting the urge to do so and finding ways to say no is arguably a better approach. Most companies and most systems will never require the scale beyond a simple Postgres server. Barring safeguards around data replication, a simple RDBMS can scale to millions and millions of records without much else to power it.

Until we meet again in the next chapter, here is some additional reading material I enjoyed with this week’s chapter:

• A system design question around the “top-k” heavy hitters.

• If you didn’t watch any of the previous episodes from this channel, you’ll also want to study their implementation of a distributed messaging queue.

• How to beat the CAP theorem is the inspiration for the system design in the first link.

• This article on the Kappa architecture challenges the merits of the Lambda architecture with a simpler model that uses fewer frameworks and drops batch processing as a requirement.

• Comparing modern stream processing frameworks is a great video that helped me make sense of way too many Apache streaming systems and why they all seemed to be redundant upon first glance.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 137 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Continuing my systems design series this week with a new, trimmer article on partitioning strategies. If you’re doing a systems design interview soon or you just want to know more about web-scale applications continue to follow along! I’d also suggest you watch this video which incorporates many of the concepts of the last two articles and is a good thought exercise.

Happy reading!

Partitioning

This chapter is another central tenant of distributed systems: partitioning. Also known as sharding, we will explore how to split up large data sets into logical chunks.

Strategies for partitioning

If replication is about copying all data to different databases and datacenters, then partitioning is about slicing up that data across those databases and datacenters.

There are a few ways you can split up that data:

• By key. The key can split the rows of data. Imagine 26,000 names evenly split alphabetically. You could put 1,000 entries in a database for the A user names, 1,000 more for the B names, and so on.

• By hashed key. You could hash the same key and then split up the data that way as well. The easiest way to ensure information is evenly split is to use a modulo (%) operator (hint: just because it is easy doesn’t mean it is helpful). If you run the modulo over the number of datacenters, you will always have a key mapped into a datacenter ID.

  • Add randomness for active keys. The above assumes that all keys are read and written in an even distribution. For the ones that are not, you can break up keys further with prefix or suffix IDs. Now the same key can be segmented across shards.

• By a secondary key. Like the prefix/suffix keys, you can use an additional key with a secondary index. This will make searching faster for the data that is partitioned.

  • For local document search. You could make this secondary index the key with the value being the original primary key that you added into the shard in the first place.

  • For global document search. Also called a term index, you could segment the secondary keys and grab all of the keys across all shards (and shard the term). This is how full-text search databases work.

Strategies for rebalancing

Over time, these partitioning strategies will skew unevenly. The easiest way to visualize this is to imagine the old hardcover encyclopedias. They are not 26 volumes split by letter. Instead, letters likes Q and X are combined while A and S get their own book.

While you may have an even balance of data to start, it will not necessarily evolve that way. You will need methods to rebalance your data from time to time.

Remember how I suggested that we use the modulo operator to segment our keys against the datacenters? This works until you add more datacenters. Then your modulo value needs to change, and you need to rebalance all of your data which creates a lot of rewriting as your data scales. This is not ideal. Here are a few better ways:

• With a fixed number of partitions. Rather than use the physical shards as segments, create arbitrary partitions within each shard node. That way, as your shards grow or shrink, the number of partitions stays the same. Then, you simply adjust how many partitions you allocate to each node. Riak and ElasticSearch use this method.

• With a dynamic number of partitions. The previous example is sensible but suffers from the same problem as earlier that you have data evolve in very skewed portions. If your partitioning segments are chosen poorly, your data will be lopsided. Dynamic rebalancing is using something like a B-Tree to change the structure as partitions are added and removed. The data evolves, and so too do the partitions. HBase and MongoDB use this method.

• With a proportional number of partitions to nodes. This is somewhat of a blended approach. The number of partitions is not strictly fixed. It changes as the number of nodes grows and shrinks. This is not the same as dynamic rebalancing because the dynamism reflects solely on the number of nodes and not on arbitrary boundaries that you define. Cassandra uses this method.

These strategies all depend on the configurations you make as a developer. You can choose how you want to deploy this rebalancing: manually or automatically.

With manual rebalancing, a developer assigns partitions to nodes. This requires more effort and time, but it allows you to respond to the needs of your data. Conversely, automatic rebalancing enables the software to do this work for you. Tools like Couchbase and Riak suggest this approach (with approval), but the downside is the unpredictable and possibly suboptimal choices for segmenting and rebalancing your data. A wrong move by the software could be slow and costly.

How to find data

With all of this data split up amongst so many physical computers, how do you go about finding them?

• You can keep track of all of them on the client. The obvious solution is to keep track of each shard in the same way that you have to have your encyclopedias face out on your bookshelf. You have to see which letter you want to access when you approach your bookshelf. The downside is that this is tedious and must be reconfigured every time you add or remove nodes.

• You can have your request routed for you. A middle layer can interpret your query or mutation and figure out where your data is for you. This is like in the old days of telephones when operators connected you to the person on the other end. You did not need to know their phone number. You just needed to remember the number of the operator so they could route the call for you.

• You can throw a request to any node, and that node will route it. This is like calling up a corporation and going through their touchtone service. You know you want to call up Dell customer support, so all calls get routed to a 1-800 number, but it is up to the service to find the exact extension for you. The default number might get you who you are looking for if you know your party’s extension number, but you may have to navigate the tree of options.

Tools like Apache ZooKeeper handle this configuration management in many popular databases like HBase and Kafka. Engines like Riak and Cassandra use a gossip protocol to chain requests from node to node until the data is found.

Partitioning is straightforward and not nearly as dense as replication. There are only so many ways you can split up data. In the first post of this series, I noted I would be skipping the next chapter on transactions. The last sentence of the chapter explains why:

In this chapter, we explored ideas and algorithms mostly in the context of a database running on a single machine. Transactions in distributed databases open a new set of difficult challenges, which we’ll discuss in the next two chapters.

Since I care about educating this audience on distributed systems and systems design, it seems only fair to focus on chapters that tackle transactions in a distributed nature.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 136 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Continuing my systems design series this week with a new, beefy article on replication strategies. If you’re doing a systems design interview soon or you just want to know more about web-scale applications continue to follow along!

Replication

This chapter is the real introduction to distributed systems. We are shifting into the second part of the book here. You will notice that I intentionally skipped chapter 4 based on a suggested curriculum I outlined in the first post of this series.

Horizontal versus vertical scaling

Before the chapter gets started, Kleppmann briefly discusses how we’ve collectively arrived at a shared-nothing architecture of horizontal scaling. Distributed systems design focuses on scalability so it is important to know the 3 main types of scaling you’ll see in real-world systems:

1. Vertical scaling. Also known as shared-memory architectures or scaling up, this involves adding more RAM, CPU, and resources to a single machine. The key observation is that doubling resources will likely more than double your costs. As such, vertical scaling is recommended when heavy computation is required at the expense of fault tolerance and reliability. There is a variant of this called shared-disk architecture which pertains mostly to data warehouse applications that provide additional fault tolerance with multiple machines accessing the same disks on a centralized network. This still suffers from the same issues when you colocate all data in one data center but has limited use.

2. Horizontal scaling. Also known as shared-nothing architectures or scaling out, this refers to adding more machines with commodity resources. This is the primary strategy used when companies scale out their services and hence is the primary focus of this section of the book.

Replication 101

Why do we replicate?

1. Reduce latency. Keeping data close to your users reduces the time it takes to physically travel over the wire to your customers.

2. Increase availability. The server goes down? You’ve got a backup on a different machine and in a separate data center.

3. Increase throughput. Drive-thru windows at your bank are convenient but if the line is long you know there are lots of bank tellers available inside to help you out and keep customers and productive.

This chapter discusses three primary reasons why we need to replicate. We’ll offer strategies to resolve why data changes all of the time and how we deal with it.

If data never changed then replication would be easy. Since data grows and evolves over time, we employ various strategies based on our system’s specific needs.

Leader replication

The first two methods involve leader replication. Each replica, or copy, has a role: leader or follower. Some schemes invoke a single leader while others leverage multiple leaders. The responsibilities of each are straightforward:

1. Leaders ingest write data. Only the leader can modify the database.

2. Followers copy changes from the leader. Followers must wait for a log file from the leader with changes they will make in the same order the leader made them.

3. Anyone can offer read data. All replicas can return data to a customer that asks for it. This is called read-scaling architecture.

That last step is the interesting one. What if you ask for data from a replica that has not copied all of the latest changes yet? This is where replication strategies offer tradeoffs in their approach.

From here, you can focus on two ways of getting changes to replicas:

1. Synchronous replication. As the name implies, you send changes to followers immediately. You then only inform customers when all changes are completed successfully. Practically, this doesn’t work because if a follower replica hangs then the change fails which hurts durability. To be fair, synchronous replication is the best strategy to eliminate the risk of data loss. In reality, when systems use synchronous replication, they will follow a hybrid semi-synchronous replication strategy where one follower is synchronous while the others are asynchronous. Read-scaling architectures don’t work with this scheme because followers have to wait for leaders to synchronize data which keeps them too busy to accept additional reads.

2. Asynchronous replication. In this scheme, changes are sent to followers immediately. The difference is you can inform customers when some changes complete successfully. You are not beholden to every follower sending a successful response. This is the most common scheme in production, high-scale application. This trades off speed for no guarantees in durability and eventual consistency.

What if you want to add a new follower to the gang?

1. Periodically take snapshots of the data. This is good anyway for backups. Send the latest backup to your new follower.

2. Poll the leader again for changes since the snapshot. Now you can assume the new follower is asking for changes as if this were an asynchronous replication strategy. Each database engine has its own scheme for issuing catch-ups such as log sequence numbers or binlog coordinates.

What if things fail?

• If the follower fails then you execute the same catchup strategy for a new follower. Just read from the logs the last changes since the failure and then play catchup.

• If the leader fails then you have to elect a new leader. You can either do this manually or automatically. Manual promotion benefits from careful failover to a verified follower. The downside is that you have to invest developer time into failovers and can be cumbersome if they occur with some frequency. Automatic promotion relies on selecting a new leader, usually with the closest representation to the leader’s data.

Reaching consensus with the automatic strategy is fraught with problems:

• There might be a gap in the data, even with the best follower chosen. In this case, you violate strict durability and you incur data loss which is never good.

• Multiple followers might believe they are the leader. This can also lead to data loss if both start accepting writes to propagate to the other followers.

• The heartbeat interval may be too long and miss the latest changes. Even if you have one follower with a perfect copy of the leader’s data, the leader may have been down long enough to miss some new writes. If the heartbeat to check for its health is in a long enough interval then new changes could have been requested while the leader was down which will be lost for good.

• The old leader may not relinquish their throne. Assuming everything else was correct, you still might see a variant on the second issue where the old leader starts back up and still thinks it is the leader. Now you are in a similar scenario where two leaders are consuming writes and you incur data loss.

Logs help with restoring the data in addition to helping new followers join in on the fun. Each database engine utilizes one of four replication log schemes:

1. Statement-based. These copy raw SQL commands. They are simple but require deterministic commands, which can be too strict for most databases.

2. Write-ahead. These are logs written before changes to the main disk are stored. This is a common append-only log that is used with things like SSTables and B-trees but requires all engines to follow the same format. If the format changes with breaking changes to an engine, then it will require downtime to upgrade. This is what plagues things like major version upgrades in tools like Postgres.

3. Row-based. This focuses on the values of rows instead of the commands like the previous two. This decouples the data from the storage engine used, which solves the problems from the previous strategy. New versions of Postgres use this, also known as logical log replication. The book doesn’t seem to see a downside to this approach but, this Stack Overflow thread goes into this strategy in greater detail which I appreciated. In short, things take more time because one database is updated at a time methodically.

4. Trigger-based. This strategy leverages SQL triggers and stored procedures. While this benefits from functionality native to the language instead of the database engine, these triggers can be complex. Further, this log strategy does not work for NoSQL solutions or any database that doesn’t leverage SQL, such as graph databases.

Eventual consistency

As mentioned earlier, a read-scaling architecture can provide additional performance for applications that are heavy on reads. The downside of this strategy is that you may read outdated information from a follower node. When a leader node sends data asynchronously to a follower node, it cannot wait to know if the follower replication succeeded.

This is what is known as eventual consistency. At some point, eventually, the data will be synced across all replicas. There is no guarantee when that eventual time will be. But, it is meant to be a reasonably short amount of time so that there is no real compromise in reliability. Nonetheless, there are occasional complications with eventual consistency:

• Reading right after writing. Say you update your email. If you make those changes and refresh the page then you’ll want to see your new email address saved. Read-after-write consistency is a strategy around this but is complicated in distributed systems.

• Reading into the past. Say you write a few checks to your bank account. Your bank account starts at $5, then you add two checks of $5 each. In theory, if the eventual consistency is delayed enough between replicas, the leader could show $15, one replica could show $10, and yet another replica could show the original $5. Monotonic reads ensure you only read from one replica so this problem doesn’t occur. Of course, if the replica fails, you’re back to square one.

• Answers to questions you haven’t asked. In the previous scenario, you could see a later state before you see a previous state. Not only are you going into the past but you are messing up the causality of the data - you wouldn’t expect an addition to lead to subtraction of the balance. Consistent prefix reads to ensure that the writes happen in a certain order. If you have a sharded database, you can’t guarantee this fix unless all of the data is affixed to the same shard. This becomes more complicated but does solve the issue.

All of these problems are difficult to solve with a single-leader replication strategy. They become less difficult when you invoke a multi-leader replication strategy.

Multi-leader replication

As the name implies, multi-leader replication allows you to assign multiple leaders at once. This solves nearly all of the problems you see above:

• No need for manual promotion. With multiple leaders, you don’t have to worry about being without a leader at all. Restarting leaders assumes there is at least one leader already able to receive writes.

• No gap in writes. Unless you get incredibly unlucky, a leader will exist to ensure there is no downtime of a leader to accept writes. This can happen with a single leader. With multiple leaders, you guarantee there is always a place to accept writes.

• No leader conflict resolution. If you know who the leaders are, you can always create a backup scheme so you either fall back to an alternative leader or you have a simple election to keep things smooth.

• Heartbeat intervals no longer matter (to a point). Waiting a long time to see if the leader is still online is not a problem because you have multiple leaders to fall back on. As long as their heartbeats aren’t synced and are checked in a staggered fashion, you can ensure that you’re constantly aware of at least one online leader.

• Multiple leaders don’t necessarily require follower promotion at all. One problem with a single-leader scheme is the leader won’t give up its role as leader when it comes back online. If you have other leaders available, there is no need to promote them. You always have leaders available to do all the work you need. If a particular leader goes offline, you just wait for it to return and push the additional load to your remaining leaders.

In addition to solving problems from single-leader replication, there are a few use cases where multi-leader replication really shines:

• Datacenter replication. Have multiple data centers? Assign a leader for each data center to ensure all locations have write coverage. The challenge is that you are back to single-leader replication per data center if you don’t elect multiple leaders per site. Further, writes become far more complicated since you have multiple data centers that have to resolve data changes. That requires costly RPC calls to copy logs from one data center to the other.

• Offline applications. What if your mobile phone were a data center? When you’re offline, you store changes to your local database. When you get back online, you sync your changes with a leader at a data center for persistence and backups. This is multi-leader replication taken to the extreme where every device acts like a leader.

• Collaborative applications. Tools like Google Docs allow you to edit concurrently with other users. Think of your browser as a leader where you publish changes locally and see them instantly in the browser. In the background, you take those changes and push them to a remote leader replica. This replica syncs your changes with the changes of another individual. They ensure they all match nicely.

Everything sounds great - so what’s the catch? It should be obvious at this point, but the big issue with multi-leader replication is write conflicts. With multiple leaders comes multiple nodes where you can write changes to your database. Who has the correct answer? Is there a leader of leaders?

The easiest way to mitigate write conflicts is to simply avoid having them in the first place. This may seem like a cop-out answer, but you can design your systems toward conflict avoidance. Having one leader per data center and routing users to specific data centers is a way to avoid conflicts. Specific updates in a particular range of data go to dedicated leaders. Of course, this reverts to being a microcosm of single-leader replication, so if a data center goes down, you are forced to reconcile concurrent writes on different leaders.

If you can’t avoid conflicts, you can use some tie-breaker heuristic such as assigning UUIDs to data centers or records so that the highest numbered ID wins. Simple, but prone to data loss. In theory, this also allows you to merge conflicting values, but then you’ll need some sort of user intervention to reconcile the concatenated data.

Of course, you could implement your own custom conflict resolution scheme. Amazon has done with this some surprising effects, such as having items reappear in users’ shopping carts after they have purchased items. While a suboptimal user experience, it is still likely better than losing data.

How do leaders communicate with each other? The book identifies three specific topologies for communication, and I see there are only two:

• Every leader talks to every other leader. This ensures everyone gets the message from everyone else with a direct line of communication between every leader. The problem here is there is no causality to the writes because there is no order with which leaders are updated. As opposed to…

• Leaders update each other in a chain. An example of this would be a circular or star-shaped topology. In these schemes, there is an ordering to how leaders communicate with each other which solves the causality problem. The challenge is that if a replica fails during the update chain, it breaks the chain, and all downstream updates will stall out. If a leader is early enough in the chain, most other leaders could suffer from data loss and stale information.

As mentioned before, the best course of action is generally to avoid conflicts in the first place. Many database systems support some kind of conflict resolution scheme, but none are perfect.

Leaderless replication

The last major strategy with replication is leaderless replication, where every node can accept reads and writes. Amazon Dynamo is the most famous example of leaderless replication and has ushered in a resurgence of this strategy. Cassandra, Riak, and Voldemort have also adopted leaderless replication schemes. This scheme is ideal for applications that require high availability and low latency at the expense of a very loose definition of eventual consistency and stale reads.

When every database replica can write data, you open yourself up to lots of staleness. There are two primary strategies for dealing with this stale data:

1. Use subsequent reads to detect stale data. The next time a user reads data, read it from multiple replicas. If any of those are stale you can issue follow-up writes to ensure the stale nodes are brought back up to the latest versions of the data. This is commonly called read repair.

2. Run a heartbeat to detect stale data. Every update comes with a version number. Run a background process that pings every node to see what version of the data it has. If any of the nodes are less than the latest version number, update the databases. Continue to ping all databases in regular intervals to ensure staleness is mitigated promptly.

Leaderless replication schemes often implement quorums on reads and writes to create consensus when there is a discrepancy. This is a tuneable threshold that allows developers to configure at exactly which point a vote passes to reconcile when replicas argue over which version of the data is the latest.

Quorums really only work if you have sufficient replica nodes to break a tie. This requires votes from read replicas, write replicas, and ensure those votes outnumber the number of replicas that debate the values. As long you have a majority vote on the state of reads and writes, you can proceed with quorum voting.

Further, monitoring is difficult with leaderless replication. This is because there is no universal order to how data is consumed. If a node goes down, or worse, several nodes go down, you risk destroying the quorum.

A concept known as sloppy quorums was designed to mitigate against this. It states it is better to write the data even if a quorum is not achieved when the required nodes for a quorum vote are down or offline. They still ask for the designated number of votes, but they may not come from the approved pool of original nodes that belong to the quorum. Without the approved set of nodes, you have fewer guarantees into the latest state of the data since these nodes are not as closely monitored.

This method increases write availability but require further mitigation like hinted handoff to restore the quorum node team. It also requires backup plans like read repair to ensure that restored nodes eventually receive the latest updates.

Another issue you can run into with leaderless replication is concurrent writes. A single change is propagated across multiple replicas. Each replica is subject to its own network latency and IO. There is no guarantee that any write will arrive at all replicas at the exact same time. How do you reconcile this case?

Last write wins is a simple strategy that dictates that the latest write is the one that will persist, and all other writes for the same data are considered stale duplicates that can be dropped. This sounds great in theory, but in practice, this is bad for durability.

Say your bank account reads $1. If you somehow issued 2 concurrent updates, mistakenly adding $2 instead of $4, you may run into trouble. Both updates will say they were completed successfully. But if the $2 change arrives after the $4 change, your bank account will update to $3 even if that isn’t ultimately the correct answer. Cassandra mitigates this with a UUID attached to every write update to ensure seemingly concurrent operations are indeed differentiated.

There are numerous algorithms discussed in the book to determine concurrency. Concurrency is not concerned with things happening at the same time but happening without knowing about each other. It’s extremely difficult to say with certainty that two events occurred at the exact same time. But they could occur in a close enough interval to not know about each other based on the state of the application.

Riak knows how to merge concurrent values with its sibling merge functionality. Riak also uses version vectors to issue read repair so that clients can retrieve them on reads and return them back to the databases on writes. My takeaway here is if you are looking for robust leaderless replication, look no further than Riak.

–

That wraps up the chapter on replicating distributed data. Another way to distribute data is through partitioning. This is the topic of the next chapter so stay tuned!

🙏 That’s all folks 🙏

Aaaand we’re done with issue 135 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Continuing my systems design series this week with two new articles (one of which I forgot to post last week). If you’re doing a systems design interview soon or you just want to know more about web-scale applications continue to follow along!

Data models and query languages

This chapter focuses on how you can organize and retrieve data from a database. It primarily focuses on the three primary data modeling paradigms:

1. Relational

2. Document

3. Graph

Relational model

This is the incumbent. The relation is the table. The individual datum within the table is the tuple called the row. Properties of a table exist on the column. You know, like any other table you would see in the real world.

I grew up with these database systems. They are what I studied in college. My master’s thesis was on RDBMS. They have been the dominant force in data management for most of the last 40 years.

The most cited open-source examples of relational databases are MySQL and Postgres. They’re still useful and are the first choice for many frameworks like Ruby on Rails or Django. It’s just that they suffer from a few problems:

• They cannot scale without considerable effort. They are much harder to replicate or partition than they should be.

• Limited expressiveness. Querying and organizing relational schemas fits a specific mold. You have to be okay with that or else you’ll have to seek alternatives.

So what are the alternatives?

Document model and the rise of NoSQL

NoSQL (which people would now call “Not Only SQL”) describes any kind of database that doesn’t follow the relational model. One thing I want to highlight is the term polyglot persistence. This refers to the idea that you can have both relational and nonrelational databases working together. It’s not like you have to choose one paradigm over the other. In fact, it might be advantageous to use both for your application to serve different needs.

I think this is important to call out because NoSQL implies that you’re specifically not choosing SQL. It’s as though they can’t work together in an application but in fact, they can and do.

• something like a resume could be a series of joins between a user and their job experiences OR you could nest those joins into one cohesive JSON model

• just one thing to fetch

  • it actually looks like the thing you’re representing (a resume is a document) - also means many-to-one and many-to-many relationships don’t work so well

  • fast: requires no joins since it is self-contained (also means poor support for combining disjointed data)

  • highly flexible: no federated schema (also means no referential integrity)

How do you know the document model is right for you?

Are you working with documents like profiles, resumes, HTML pages? It’s great. If you can populate that whole document it’s very efficient. If you have a lot of nested relationships or links to other documents it starts to become cumbersome.

A list of jobs might be two separate tables in a relational model. In the document model, they will all be listed out. The exception is if you have created a separate document for all job types. At that point, you’re heading in the wrong direction.

One-to-many, tree-like data structure? Good. Many-to-many, graph-like data structure? Bad - stick with a relational or graph database.

So when does something look like a document in the real world? The two often-cited examples I see are:

• CMS. The primary object of a blog engine is a blog post which is a document. The important thing is that it is unstructured. The content of the blog doesn’t need to fit a particular format.

• E-commerce product catalog. Imagine a service like Amazon. They have tons of products. Other than the SKU there is really no mandatory format for structuring how you display a product. This falls into the large volumes of unstructured data category which could be a good fit for a document model database.

Mongo is to Python as Postgres is to Java

This analogy just kept swimming in my brain. One of the key benefits (and drawbacks) of a document database is its loose constraints on schemas. The book makes an interesting analogy in that it’s like runtime compilation: no schemas are validated on writes on the way into the database but you can validate a schema on read. This means you can only be assured of the structure of the document when it is being read from the database.

SQL is to CSS as JSON is to the DOM

In a similar but not nearly as clean analogy, we can further think of the paradigms of database programming languages. Relational databases use declarative SQL like CSS. They optimize around patterns rather than algorithms because you just declare how you want to filter out your data and you let the language choose the algorithm. This allows for terse, parallelizable code where the language can handle performance optimization and you focus on just getting your data.

Since NoSQL databases don’t necessarily use SQL their method of storage and retrieval revolves around their documents, commonly stored in formats like XML and JSON. In theory, you can use any old imperative programming language like JavaScript to interact with the database. This is nice in theory since you are programming UIs and business logic in the same programming paradigm. The downside is that your code is likely more verbose, uglier, and requires a specific algorithmic sequence to execute. Think of it like styling a page with JavaScript; you could - but why would you do that to yourself?

Writing is not my strength…

Reads are the name of the game here. Small, unmodified documents are best. Anything else you risk heavy performance penalties. Documents get rewritten as a whole so minor updates to big documents are a huge drain on performance. The benefit of having all of the data shoved into one document is to remove the need for complicated things like joins. The drawback is without joins you have to colocate data. This can become expensive if the document becomes expensive.

…and it may not matter

Postgres supports JSON columns. Mongo supports foreign references in a weird sort of join operation. The truth is, the document and relational models may not be all that different soon so you aren’t locked into one idiom between the major database providers anymore. The book goes into an example of this with the MapReduce extension on MongoDB. I didn’t think the example was worth stating here but the last sentence on this topic is a great summary of what I took away from this section:

The moral of the story is that a NoSQL system may find itself accidentally reinventing SQL, albeit in disguise

Graph models

Remember how earlier I said one-to-many relationships were a good fit for document databases and many-to-many were good for graph or relational databases? Now let’s pick that apart further.

If your many-to-many relationship is simple, you can probably get away with a relational database. For complex many-to-many relationships, a graph database is a good choice.

What’s an example of a complex many-to-many relationship? You already know canonical examples:

• Social networks. People know lots of other people. Just ask Kevin Bacon.

• The web. Web crawlers are designed to demystify the tangled mess of connecting and indexing every website.

• Transportation. Roads, railroads, and airports all connect points across the globe. This is a hard problem and one I’ve explored before.

To be honest, I found this section to be a bit dense on specific kinds of graph databases that may not be relevant today so I’ll gloss over each section with the critical parts I took away.

Property graphs and Cypher

Neo4j is the example I see all over the web on graph databases. I think if you just knew this one and Amazon Neptune (which seems to support all kinds of graph database models) you’d be fine.

This is how I would imagine you would design a graph data structure at scale. There are two main objects in a property graph DB: the vertices and the edges. Like a database, each of those things has a UUID. They also have information as to where they go and who uses them. And as the property name suggests, each of these objects also has metadata in the form of key-value pairs.

The power here is in the flexibility to build graphs and the tagging you can store to find your vertices and edges. Property graphs don’t restrict how you construct your graph. Graphs don’t require a schema other than the basic vertex and edge. So you’re free to design it however you like. The power of the model is in how you relate things.

These key-value tags make it easy to find information in the graph without ruining your data model. It also means you don’t have a traverse a graph to find information. Tags and IDs allow you to quickly lookup vertices or edges without having to go through all of the graph connections. They’re like the shortcuts to traveling across the network.

If you need to evolve your features it doesn’t ruin the data. The key-value pairs allow you to create new associations between your core objects without your graph fundamentally having to change. Sounds like Facebook would be a good example of this: people are your edges and their connections are the vertices. In the old days of Facebook, you could list out your favorite bands or movies. A property graph database would allow you to add something like TV shows without having to totally screw up your existing graph or its associations.

Cypher is to property graphs as SQL is to relations. That’s all you need to know here. It’s a declarative programming language for Neo4j in the same way SQL is a programming language for databases like MySQL, Postgres, MariaDB, and SQLite.

Triple-store graphs and SPARQL

This is another graph database variant. Instead of vertices and edges as the primary objects a triple-store graph uses, you guessed it, three objects: subjects, predicates, and objects. This one intuitively makes less sense to me than a property graph because I don’t think of a graph as I think of an English sentence.

Here is how you make sense of these three things:

• Subjects are always vertices. They are nouns and stuff like people and fruit.

• Objects and predicates have 1 of 2 states.

  1. The object is another vertex. Then the predicate is the edge connecting subject and object together.

  2. The object is a primitive type. Then the predicate is a property where the object is a data type of the subject.

The book lists Datomic as an example of a triple-store graph but it doesn’t market itself as a graph database but an amalgamation of many flexible data models. For practical purposes of this audience, you’re likely not to run into a scenario like this.

SPARQL is to triple-store graphs as Cypher is to property graphs. SPARQL is the programming language for triple-store graph databases using the RDF data model for the semantic web. Given that the Semantic Web never really took off, it’s more of a reason why you don’t need to invest more than a few paragraphs into understanding these things.

Further reading and study

Since this was a much meatier chapter, I found I had some additional notes from the previous walk-through that was relevant here. There is another video in that same series I used for practice this week.

I found myself referencing a few articles which may be helpful in understanding NoSQL databases:

• https://shopify.engineering/five-common-data-stores-usage - Shopify did a nice job of simply explaining the use cases for five different kinds of data stores. The segmentation is a bit odd here and it doesn’t align with what I’ve read in the book but it’s still factual information even if the categorization is faulty.

• https://www.prisma.io/dataguide/intro/comparing-database-types#relational-databases-working-with-tables-as-a-standard-solution-to-organize-well-structured-data - This Prisma article was great because it expanded my knowledge even further beyond NoSQL databases to NewSQL, multi-model, time series, and more.

• https://blog.nahurst.com/visual-guide-to-nosql-systems - I like that this post has a diagram that revolves around the CAP theorem. It helped me place where NewSQL fits and how to actually tell the difference between HBase and Cassandra.

• https://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis - Another mega list of NoSQL solutions which categorizes based on popularity rather than storage type. I actually really liked this because it simplified my thought process to a pretty rock solid list:

1. Want relational? Use Postgres.

2. Want an in-memory key-value store? Use Redis.

3. Want relational without the schema? Use MongoDB.

4. Want web analytics? Use Cassandra.

5. Want a full-text search? Use ElasticSearch.

6. Working with documents, specifically a CMS or CRM? Use CouchDB.

7. Building a search engine or log analysis? Use HBase.

8. Working with graphs? Use Neo4J or OrientDB.

System design tradeoffs between SQL and NoSQL solutions

A few questions you have to ask yourself when comparing SQL and NoSQL solutions as they relate to scalability:

How do you scale reads? Writes?

For SQL solutions, you’ll want to shard servers when the database runs out of space. Use a cluster proxy to abstract away the shards. Use a configuration service like ZooKeeper to ensure all shards are online and be sure to toggle backups in the event of a downed shard.

For NoSQL solutions, you’ll also want to perform sharing. However, there is no need for a cluster proxy because each shard knows about the other. Thus, no configuration service is needed.

How do you make reads/writes fast?

For SQL solutions, you’ll want to use a shard proxy to cache results and publish metrics, and terminate long-running processes.

For NoSQL solutions, no shard proxies are required because all nodes know about each other. Quorum nodes are used to ensure you don’t need to talk to every node to get a consensus on whether or not the operations succeed.

How do you prevent data loss?

For both SQL and NoSQL solutions, you’ll want to create read replicas of shards in different data centers.

How do you achieve consistency?

For SQL solutions, you’ll want to sync lead data to follower replica data. This leads to consistent data. It also takes a long time to process.

For NoSQL solutions, data is pushed to the read replicas asynchronously. This provides eventual consistency and is fast.

Other questions worth asking

• How do you recover data in case of an outage? Data recovery is much harder for a relational system that sacrifices partition tolerance so you’ll need to focus more on cold storage backups while NoSQL solutions are much easier to replicate their replicas across centers.

• How do you ensure the data is secure? I only saw mentions of Accumulo and Riak specializing in security. Otherwise, every database needs to worry about this regardless of storage type.

• How extensible is the DB to an evolving data model? Evolving data models are better suited for a graph database than a document database.

• How easy is it to run on/off-prem? This question was asked in the systems design video from last week but I don’t think most companies really need to worry about on or off-premises solutions anymore.

• How much will it cost? Most solutions these days are open-source. The cost will come down to your strategy for your horizontal scaling and partitioning.

Storage and retrieval

This chapter is all about getting and sending data. There are two main kinds of data stores:

1. OLTP. This includes your traditional relational and NoSQL databases. They comfortably store terabytes of data, are highly available, and are designed for the end-users of your applications.

2. OLAP. This includes data warehouses from names like Oracle and IBM Db2. They store into the petabytes of data, are primarily read-only copies of several OLTP databases, and are designed for business analysts.

One thing to note about OLAP data stores is the simplicity of their data models. While the entire last chapter was dedicated to the variety of OLTP data models, there are really only two models in use for OLAP systems:

1. Star schemas, named after the star-like shape representing the central fact table that points out to supporting dimension tables

2. Snowflake schemas, which add on sub-dimensional tables to the above star schema with a more complex, snowflake-like pattern

The seemingly obvious comparison is that snowflake schemas are more complex because of their additional normalization with sub-dimensional tables. Conversely, star schemas are simpler to conceptualize with less branching. Either way, both support lots (100+) of columns.

Column-store databases for big data

Wide-column fact tables benefit from column-store databases. As the name suggests, you’re inverting the way data is stored and retrieved. Rather than store data in rows, where each entry corresponds to a single object and various properties of that object in columns, you store all of the values for a single property together.

Since a massively wide fact table is likely only utilizing several columns in a given query, data organized by column lends itself well to these kinds of analytics queries. You can partition and shard data by column which makes searching against particular columns even more efficient.

Since all data in a given column adheres to the same type, you can think of a column as one giant array. This makes column-store databases effective at compressing data with techniques such as bitmap encoding. With more stuff able to fit in memory, column stores can effectively use computer caches to leverage lots of CPU cycles to iterate through data. You can further improve efficiency when grouping or filtering by sorting your columnar data.

Finally, column stores can leverage highly specialized materialized views called data cubes. Data cubes make it easy to reference a lot of aggregated data. This is a way to precompute data and display it for fast retrieval.

All of these make reads incredibly efficient for even the largest data sets. The clear downside is the cost to write data. A table with 500 columns might now be spread out across 500 files, one for each column. That means adding a new entry means writing 500 properties to 500 files. If that entry just happens to max out the file size on disk you’ll have to perform even more writes to partition and segment out your data. In other words, writing is expensive. As we’ll see later on in this post, there are tools like LSM-Trees that can effectively speed up data storage.

Data retrieval techniques

Storing data is straightforward. Write data to an append-only log and save the file to disk.

Retrieving data, on the other hand, that’s where things get interesting. The book presents a progressively complex way of retrieving data efficiently.

Hash index

Indexing is the general strategy for retrieving data quickly. By writing only the data you search against into memory you trade space for speed. The downside, besides the additional space requirements, is the cost to write the data. Data stored in multiple places requires multiple writes to disk and takes more time.

One way to index is with a hash code, just as hash maps/tables relate keys to values. An example would be Bitcask. Bitcask is the storage engine powering Riak.

As we saw from the YouTube example, we need to count video views. The video offers Cassandra as the data storage mechanism of choice. The book might suggest that Riak with Bitcask might be the ideal choice. Bitcask uses an in-memory hash table separate for what is stored in disk. This, in turn, makes frequent writes still fast on read.

Everything about this sounds great, right? We have a method that is:

• Scalable: we can shard the data on disk while keeping the indexes in memory

• Fault-tolerant: the database can go down in a separate data center that stores the index

• Eventually consistent: indexes can now be updated independently of the data stored on disk

• High throughput: the right hashing strategy limits collisions which allow for a lot of conflict-free storage

There are two limitations with hash indexes:

1. Slow range queries on indexes. Since the index key is a code and the codes are unique you don’t gain any speed searching a range of values.

2. Size limitations. If you run out of memory the whole thing falls apart. Extreme-scale applications that fill a hash table index completely are the one exception to scalability.

SSTable

The next evolution of hash indexes is Sorted String Tables, also called SSTables. SSTables provide some additional benefits:

• They only store keys once, saving on storage space.

• They sort the keys to further speed lookups.

• They use a variant of merge sort to merge and compact segments.

• The organization of that data can be implemented as a red-black or AVL tree, allowing you to write in any order and read in sorted order. This makes it easier to store in memory.

I didn’t touch on segments earlier. A segment is just a means of sectioning off a data log. To prevent running out of space you need a means of separating your indexes across files. To reduce the space requirements we compact the segments by removing duplicate keys and only keeping the most recent write to that key.

This implementation of an in-memory balanced tree for keeping a cascading tier of SSTables is aptly called a Log-Structured Merge-Tree or LSM-Tree for short. Lucene, the Apache search library that powers ElasticSearch and Solr, uses something like an LSM-Tree to store its dictionary of terms and SSTable-like files to map those terms to postings.

More specifically, if you’re dealing with a dictionary of words you’d want to use a structure that tailors itself to strings instead of numbers. We’ve already evolved past a hash table so that is out. A self-balancing binary tree is out, too. Suffix/prefix trees like tries are specific to strings but suffer from the same problem of taking up lots of space and eating away at RAM. So what’s left? That’s where the mighty Bloom filter comes to the rescue.

A Bloom filter is a probabilistic data structure that is sort of an inversion of other data structures. Rather than using the information to find if data is contained within a structure, a Bloom filter tries to determine if something is definitely not in the set or if it might be in a set. The “might” part here is key because it’s not guaranteeing validity. There is a chance for a false positive.

With a sufficient number of hashes, you can obtain a low false-positive rate to obtain extremely accurate results with a very low amount of indexing space. This solves our biggest problem with indexes when the data is really large and you’re worried you will run out of in-memory storage for your index. A Bloom filter can handle extreme scale and still perform well with the tradeoff that your results aren’t always 100% accurate.

B-tree

We’ve talked about efficiently retrieving data in memory. What if you want to retrieve data on disk? The B-tree is the most widely-used and heavily studied database index lookup tree. I had to build one of these for my relational database class in college. It’s another self-balancing tree structure used in all of your major RDBMSs like Postgres and MySQL.

While both LSM-Trees and B-trees contain key-value pairs sorted by key, the strategy for separating data is completely different. The Log-structured indexes can be of variable size. They can also be quite large, spanning into multiple megabytes. B-tree indexes, by contrast, are always fixed blocks of typically 4 KB or more. Since disk pages are also aligned in fixed blocks, the B-tree is better suited for sorting and retrieving data on disk instead of in memory.

B-trees are structured in layers so that the lower levels represent the range of values in between two values in the parent layer. As we saw earlier, in-memory indexes suffer from poor performance in range queries. B-trees do not.

The branching factor determines how wide each layer of the tree is before the range must be split down to a lower layer. For conventional databases with these 4 KB pages, you could have a branching factor of only 500 and only need to traverse down 4 levels before you have stored 256 terabytes of data. That’s a flat and efficient tree!

In terms of system design, how does this compare in performance to the in-memory solutions?

• It is also scalable because the partitions are small, 4 KB pages.

• The data structure inherently supports range queries as a result of the branching factor.

• Fault tolerance is achieved through a write-ahead log which writes the data changes to the WAL before it is written to the B-tree (hence, write-_ahead_).

• Locks on the B-tree called latches can be introduced to preserve consistency in an ACID-compliant manner, achieving a higher level of consistency than with a log-structured index.

When comparing log-structured against B-tree indexes, the main rule of thumb is to consider performance on reads or writes. Though performance testing will confirm this for your implementation, log-structured indexes are thought to be better for writes. Conversely, B-trees are considered better for reads. This is because LSM-Trees frequently write, segment, and compact their SSTables to create efficiency.

On the other hand, B-trees offer a more reliable indexing scheme because the keys are only written once and do not suffer from a high level of writes due to compaction. The pages are separated into small, predictable pages and do not require sizable computation for write compaction and segmentation. This is particularly useful for transactional databases when ACID guarantees are a must. That would explain why B-trees have stood the test of time for so long.

Other indexes

The book offers a few more indexing suggestions, such as secondary indexes for joins, clustered indexes where the value is a heap file rather than the raw row data, multi-column indexes like R-trees, and fuzzy-search indexes like those we talk about with ElasticSearch and Solr.

Then some indexes are colocated with the whole database in memory, such as the case with Memcached or Redis. Now the entire thing, both the index and the raw data, are stored in memory. You only need a hard disk for fault tolerance and durability. Further, in-memory databases allow for more flexibility in the data structures they utilize. It is not uncommon to interface with a priority queue or set with something like Redis.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 134 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Last year I wrote a blog post series on algorithms that is now a companion series to the official lectures from Prof. Skiena. I wanted to do the same thing with systems design with the same purpose: to really learn this stuff because I expect the people I’m hiring to know it as well. It’s only fair, right?

So the next several weeks are going to be blog posts on my learnings as I read through Designing Data-Intensive Applications. If you’re a hiring manager or someone on the job market right now this should be a highly informative series for you to check out and stay tuned for!

Scalability, reliability, and maintainability

Like my series on Algorithms, I’ve decided that I need to really understand systems design.

I run hiring for multiple teams. Some of our questions revolve around systems design. How can I possibly ask these questions if I couldn’t answer them perfectly myself?

It’s only fair that I master these concepts. The book that everyone seems to agree with is the best book on this topic is Designing Data-Intensive Applications by Martin Kleppmann.

In the spirit of my last educational series, I will transcribe my raw notes over the next 7 weeks or so. I do this both to teach my readers and also to reinforce the concepts myself. As I recall from my course on learning how to learn, one of the best ways to learn is to teach others.

The syllabus

I’ve decided to craft a syllabus for myself as if I were both the teacher and the student to help keep me on track and reinforcing the concepts. If you have a similar learning plan or have questions about mine, please feel free to reach out.

• Week 1 (this week)

  • Read Chapter 1: Scalability, Reliability, and Maintainability

  • Practice: Design a scalable system to count events

  • Listen: the book’s podcast

  • Explore, at a high-level, data tools in the space

• Week 2

  • Read Chapter 2: Data Models & Query Languages

  • Practice: Top K Problem

• Week 3

  • Read Chapter 3: Storage & Retrieval

  • Practice: Distributed cache

• Week 4

  • Read Chapter 5: Replication

  • Practice: Rate limiter

• Week 5

  • Read Chapter 6: Partitioning

  • Practice: Notification service

• Week 6

  • Read Chapter 8: Faults & Reliability

  • Practice: Distributed messaging queue

  • Read: Facebook’s messaging queue with HBase

• Week 7

  • Read Chapter 9: Consistency & Consensus

  • Practice: Read the HighScalabilitySeries on Twitter and implement it

• Week 8

  • Skim Ch 10 & 11: Stream and Batch Processing

  • More possible practice that leverage stream/batch processing like Uber

Notes on Chapter 1

Chapter one is really an overview chapter. It’s easy to read and easy to skim. There aren’t too many deep concepts here. I found lots of shortlists/rules of thumb to be all you need to focus on the basics.

Data storage

When thinking of data storage, consider these 5 buckets:

1. DB engines (Postgres, Cassandra, CouchDB)

2. Search indexes (ElasticSearch, Solr)

3. Caches (Memcached, Redis)

4. Batch Processing (Storm, Flink, Hadoop)

5. Stream Processing (Kafka, Samza, Flink)

Functional vs non-functional requirements

Requirements concepts fall into 2 categories:

1. Functional requirements such as API design and type signatures

2. Non-functional requirements like scalability, fault tolerance, and so on

Requirements gathering

When asking questions regarding a product spec for a large-scale system, focus on these 5 categories of questions:

1. Users: Who are they? What do they do with the data?

2. Scale: How many requests/sec? Reads or writes? Where is the bottleneck? How many users are we supporting? How often/fast do users need data?

3. Performance: When do things need to be returned/confirmed? What are the tolerance and SLAs for constraints?

4. Cost: Are we optimizing for development cost or operational/maintenance cost?

5. CAP theorem: The P (partition tolerance) is always assumed to fail. That is why you always need redundant partitions and backups. Therefore, you want to ask what is more valuable: consistency or availability?

If consistency is most important, consider an ACID database like Postgres or MySQL. If availability is most important, consider a BASE database like an eventually consistent NoSQL solution such as CouchDB, Cassandra, or MongoDB.

The three system qualities in 1 line

This chapter can effectively be summarized in 3 sentences:

1. Scalability determines if this system can grow with the growth of your product.

2. Reliability determines if this system produces correct results (nearly) each and every time.

3. Maintainability determines if this system can evolve with your team and is easy to understand, write, and extend.

Further reading and study

As I said before, this is a pretty simple chapter. I also watched this systems design intro. This video extended these concepts and informed some of these notes. I like to accompany learnings with practice to seed new questions for our own interview process. This article on YouTube’s architecture further reinforces the sample problem on the YouTube video (how meta). You can check your solution against the one that was really used by YouTube.

Check-in next week with a summary of Chapter 2 of the book: Data Models & Query Languages!

🙏 That’s all folks 🙏

Aaaand we’re done with issue 133 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

I’ve been really intrigued by OCR (Optical Character Recognition) technology lately. We’re doing a lot of document uploads at my job and it would be super helpful to learn more about how to apply this kind of thing in a sensible manner on the front-end. Thankfully, Smashing just released this article this week which I and my teams are pouring through. It’s my only article for the week I wanted to share here, other than some new newsletter thoughts I’ve been ruminating over since last week.

Have a great start to summer!

Effective messaging for candidates

Recruiters have told me that they achieve a 7% (at best) success rate for replies to messages from candidates. Meanwhile, I’m able to secure a 60% success rate when I reach out to prospective engineering candidates. The main reason for this is that engineers prefer to hear from other engineers. The weight of a message from a VP or Director is much greater than that of a recruiter or someone outside of the engineering org. That’s why it is so important to message the candidates yourself.

How to set up your messages

I reach out to candidates on LinkedIn. It’s hard to find candidate emails. Honestly, I’m getting enough of a response that I don’t need to stalk them anyway. This saves me a lot of time so I can just focus on the message.

Start your message off with a real greeting

Don’t rely on LinkedIn to create a salutation. Sure, they guarantee you won’t screw up the spelling (or worst, copy-paste the wrong name). But they look fake as hell. Don’t be fake.

Next, I’ll admit they’re probably happy where they are

Most candidates are off the market so I’m not going to pretend they’re actively looking and excited to see what we have to offer.

It helps if they have been somewhere for a few years. I’ll call that out too because then they understand that I’m reaching out because it’s reasonable to assume they might be starting to look elsewhere. On the flip side, I won’t reach out to someone who started a new job 3 months ago. That would just prove I’m not paying attention.

Why they would be the right fit for the role I have open

After we get the reason for connecting out of the way I connect what they have on their resume to what I’m looking for. I’ll get specific - their Python experience, a previous employer in the logistics industry, or an interest in microservices and building stream processing tools. Just something to show that I’m reading the fine print and not just the absolute basics like their name.

Hone the message of why we’re a big deal

We’re making software to help farmers raise, sell, and distribute their crops sustainably. That’s it. You need to get your message down to something succinct. It helps that we have a strong mission for folks to connect to (and I’m hiring).

Close the sale

I always end by saying something to the effect of “if that sounds interesting can we set up a time to chat next week?” This is a lesson I’ve learned from my days in consulting sales. People are more likely to respond if they have a specific request with a specific timeframe to respond. The fewer reasons for a candidate to say no the better.

Messaging logistics that might be helpful

Now that you’re ready to hit send, here are the last few things to keep in mind to further optimize your open and response rate:

• The best day to send a message is Thursdays. Or at the least - not weekends and not the start/end of the week.

• The best time to send a message is noon EST. It’s lunch on the east coast and breakfast on the west coast. People check their emails and messages at this time. If they’re already on the platform they have more inertia to read and answer.

• The best length message is under 500 characters. I break that rule often. After all, it’s hard to capture everything above in under 500 characters. I never go above 600-650 characters. If you can tailor an authentic message in under 500 characters I’d love to see it because I’ve had a real hard time making that happen.

• Respond within 24 hours. I’d advise you to check LinkedIn daily. It helps keep your message congruent in being interested in the candidate by truly wanting to hear from them with a sense of urgency.

With all of these strategies implemented I routinely receive a 60% open rate and a 30% reply rate. Even if a candidate says no - you can still win a fan. Be sure to thank the candidate for opening and responding and be sure to ask them if they know anyone who would be a better candidate we should talk to. Most people are willing to offer up a name (or 3). All of these ideas will help further expand your network and get you closer to hiring your next engineer.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 132 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

As opposed to last week, I have two more posts to share with you all in the span of a week. I’m back on my writing kick and I still have plenty more to share. This week we’re focusing on becoming an engineering manager and managing remotely even after the pandemic is over. Remote work has taken a front seat over the last year and now that some parts of the world are starting to get the pandemic under control companies are already starting to roll back their WFH policies.

If your company is still offering the option I would read below on how to keep it up without losing your sanity.

Becoming an engineering manager

The following is a README I give to all of my new managers. Think of it like a step-by-step guide to the broader 90-day plan you would offer your manager when you start in a new role.

Welcome!

Your first few months as a manager here at Indigo are crucial to your success, and we’re here to support you!

This new manager issue is a launchpad, and it connects you with crucial information about being a manager at Indigo. It has been designed to ensure you start your journey as a manager with all the resources and training available.

Items on this issue are identified as either required (in bold) or optional (in italics). Optional items may not be applicable to your style or team dynamic. Experiment with them for a few days or weeks. If you discover a useful practice or resources outside of this list, please feel free to leave a comment or edit this file.

Your First Week

• Schedule a handover meeting with the current team manager. Items covered should include recent feedback, projects, individual goals, and any shared documents or performance reviews most recently given.

• Schedule 1:1 meetings with team members (Template) (First 1-1 Questions)

• Update your team and organization wiki pages to reflect the new reporting structure

• Request access to any team chat channels you should be in

• Create a README of how you work

• Read Transitioning to management as a Technical Lead Manager

Your First Month

• Focus on 1-1s: ask questions, build rapport, and discuss career development plans with each direct report (Questions Reference)

• Review your hiring process

• Define or review measurable goals for your management role with your manager in a 1-1

• Modify this document to incorporate learnings you’ve had as a new engineering manager

Recommended Reading List

• The Effective Manager by Mark Horstman (and the accompanying Manager Tools podcast)

• Resilient Management by Lara Hogan

• High Output Management by Andy Grove

• The 27 Challenges Managers Face by Bruce Tulgan (particularly for conflict resolution)

• Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim

• An Elegant Puzzle by Will Larson

• Extreme Ownership by Jocko Willink (and his Jocko Podcast)

• The Making of a Manager by Julie Zhuo

• The Manager’s Path by Camille Fournier

• The Talent Code by Daniel Coyle

Days 31-60

• Schedule a coffee chat with your recruiting team

• Begin incorporating feedback into your 1-1s (podcast on 1-1s & feedback)

Days 61-90

• Begin incorporating coaching & delegation into your 1-1s (podcast on coaching & delegation)

• Work with your manager on decision-making techniques to effectively delegate work and get more out of your direct reports (Template)

3 Months and Beyond

• For Technical Lead or Interim Managers: Decide with your manager if you want to continue down technical leadership (e.g. Staff engineer, Principal engineer) or transition to professional leadership (e.g. Engineering Manager)

• Walk through your first quarterly goal setting for your directs with your manager (Template)

• Learn about the performance review process at your company in your manager 1-1

How to manage remotely and stay sane

The last 14 months have been a grand experiment in how to manage teams remotely. In one sense this has been a great success. Teams have managed to stay productive and companies have shown record profits. On the other hand, there is a concern of an eventual return-to-office culture.

Let’s look at a few of the most common issues managers have with managing remote teams.

Effective communication

Remember these two words: documentation and asynchronous. Wash, rinse, combine, and repeat. When you communicate with your teams, do so in an asynchronous way; ensure your teammates can view your messages on their time and in their mediums of choice. That means your message will have to be repeated again and again from Slack to email to the wiki.

Speaking of wikis, if you answer a question, put it in the documentation. By documenting things in a permanent medium like a wiki, you effectively automate your ability to answer questions without having to repeat yourself. You also create an easy to reference, permanent, searchable document for anyone to access.

The side effect of thinking this way is it reduces the number of meetings on your calendar. Pushing towards asynchronous, document-based communication removes the need for meetings on your calendar since you can comment back-and-forth and evolve the document on your own time.

Video calls

First raised by the BBC, the last year has showcased a new phenomenon called “Zoom fatigue.” People are sick of seeing little boxes of humans on their laptops and crave real attention from real people. I get that. But that will never solve the problem of your dev who is in Minneapolis and you who are in San Francisco.

First, level the playing field. Require that all teammates take calls from their laptops rather than an office camera that projects the entire conference room. This might seem like a trivial/subtle change but it is important for setting context about “us versus them.”

You don’t want to assume that those who are permanently remote are second-class citizens in your organization. They will feel this way if they see folks in a conference room Zoom and they are on their laptops.

The key difference is in your attention to others. If you have a camera right on your face with a laptop in front of you, you tend to focus on the participants in the Zoom. If you have a camera in a conference room you tend to focus on the people in the conference room and the camera becomes background noise. This is good for those in the conference room and bad for those who are remote.

So if you need to abandon the conference camera in favor of all laptops even in the office, you should consider upgrading your AV hardware. A recent study suggests that better audio quality makes you appear smarter so investing in a solid microphone like the Shure SM57 is a great, cheap audio upgrade with an interface like the Focusrite Scarlett 2i2. By extension, getting a better camera (and even lighting) will also improve your perception on Zoom.

Even if you don’t have the budget for this at work, it still might be a good investment going forward. If you plan to work remotely for future jobs then these will be investments in your future job prospects as well.

Whiteboarding

Drawing things on a laptop is notoriously hard. There are few solutions to solving this problem right now. I do have one solution I’ve found to be useful but it requires investment and practice.

Buy an iPad with an Apple Pencil.

I understand this is not a cheap ask. Neither is outfitting a home office with a comfortable, ergonomic chair and desk. However, if you want to commit to working remotely and managing teams then this is a relatively cheap upgrade in favor of a much-improved whiteboarding experience.

I really like the Microsoft Whiteboard app because it is easy to get drawing on the iPad with the Pencil. The other two things I really like about this app are the ability to create concrete shapes from hand-drawn movements and that it is easy to persist old drawings or share on Zoom.

It’s not perfect but it’s free and I’ve not yet experienced a limitation using it for work.

Interviewing

Interviewing remotely is a challenge for sure. You want to create a sense of connection with your candidates.

I’ve had this video on repeat for the last several weeks because it is jam-packed with advice regardless of whether or not you are conducting interviews remotely. You’ll gain a lot of insights on whether or not remote is a limitation for you with hiring.

Specifically to remote interviewing, I think the best thing you can do is allow the candidate to use their own IDE and share their whole screen. In other words, use the remote interview as a window into how they would actually do their work on a given day. If you frame an interview with that lens that the question of remote versus in-person hiring is no longer a problem. It is solely focused on providing a great experience and not about whether remote is contributing (or hindering) that experience.

Strategic work

For some reason, managers believe this is the biggest detriment to remote work. Someone recently commented on a LinkedIn post of mine that “nothing beats getting a bunch of folks together in a room in front of a whiteboard”.

People have had to strategize throughout the pandemic. Their companies have done very well financially. While some may prefer in-person to remote I fail to see anything about in-office that is inherently a better experience.

This might be a cop-out answer but I think interweaving many of the suggestions above would solve the strategy problem. Leveraging written communication, improving your setup for video calls, and the right technology makes these kinds of meetings so much easier to express your ideas effectively.

Strategy meetings, to me, revolve around writing dense documentation and may benefit from some whiteboarding with other leaders. A great AV setup with an iPad and a proctor to keep the meeting on track with notes is all you need to have an effective strategy meeting.

Reading emotions and connecting with people

This last point is perhaps the most difficult to conquer in a remote environment. Body language, by the sheer fact that video cuts out most of the body, is impossible to read remotely. Still, even if you can’t see someone’s whole body does not mean you can’t read emotions or body language at all.

Not being on Zoom, is, in and of itself, a signal. Not turning on your video can be a sign of anxiety or depression, particularly if one usually is on Zoom but has decided not too recently.

If the other party is on Zoom you’ll have to be more perceptive about watching visual cues. Again, without obvious body language cues like crossing legs or folding arms, you can still judge emotions through facial expressions. You just have to work harder to find these cues and be more observant. I’m not an FBI agent or a psychologist but there are books on the subject.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 131 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Well, I blinked and 3 months have passed. In that time my son has turned one, my teams have gone through all sorts of changes, and the world is looking like a much brighter place.

I won’t lie - I think part of my hiatus was driven by COVID burnout. There’s only so much energy you can put into so many things and after a year in lockdown, I had trouble keeping up with releasing consistent content. I had quite a bit, to be fair, but have had to backdate once I actually hit the merge button.

If you haven’t watched Bo Burnham’s Inside I’d give it a shot. He pretty much hit the nail on the head for grasping how do you live up to your values without becoming a soulless sellout. That said, to get back to it I figured I would offer not one but two of my newer articles (there’s thirteen to catch up on) to digest as we close in on the start of summer.

It's not hard to care

I handwrite every single email I send to candidates.

Every. Single. One.

Of course, it doesn’t scale. Of course, it is time-consuming. Of course, rejection sucks, and I’ll probably get rejected by 99% of the people I reach out to.

If hiring is your most important activity, then why would you ever put in a “copy and paste”-level of effort for people that could provide MILLIONS of dollars of value for your organization?

Have you ever received an email that was copy-pasted? Have you received an email where they used the wrong name (or reversed your first name with your last name)? I can answer yes to all of those.

It’s pure lunacy to me that recruiters and hiring managers don’t put any effort into the candidates they’re trying to hire.

Always write a personal note. Always pay attention to their experience, their wants, their needs. Always be a human first, and a hiring manager second.

How to find great engineers

One of the hardest jobs as an engineering manager is to hire and build a great team. The other is to retain and engage them (but that’s for another day). The purpose of this article is to walk you through step-by-step how to find and hire great engineering talent.

Top of the funnel

• Create a Monthly Engagement Calendar. Hiring cycles often happen at the start of the month (and year) so do most of your sourcing at the beginning of the month.

• Reach out to DEVELOPER social networks. No - not LinkedIn. Not Facebook. Not Instagram. Here’s some that have worked well:

  • Hacker News “Who is Hiring” thread. Post every month on the first business day. The challenge is you can only do 1 posting per company. So only have 1 engineering manager post for your company.

  • Reddit subs for your programming language/framework. r/technology or r/programming will get you nowhere. But if you’re hiring a Scala developer, r/scala allows you to post job threads.

  • Slack communities. Hiring JavaScript developers? There’s a Slack community for that.

• Find language/framework-specific channels. Still not getting any interest from the first two steps? Here’s an example of where you can source JavaScript developers from:

  • For Node:

    • https://discordapp.com/invite/vUsrbjd

    • https://dev.to/t/node

    • https://hashnode.com/n/nodejs

    • https://spectrum.chat/node?tab=posts

  • For React:

    • https://discordapp.com/invite/0ZcbPKXt5bZjGY5n

    • https://dev.to/t/react

    • https://hashnode.com/n/reactjs

    • https://spectrum.chat/react?tab=posts

• Do the work yourself. Your recruiting team, if you’re lucky enough to have one, is probably great. And they need your help! Their job is to supplement your hiring, not do it all for you.

The best way to convince developers to join is to be genuinely interested in them and convince them that the role you are hiring for is perfect for them.

First contact

• Set up a 30-minute phone call ASAP. Since you’ve posted to so many communities, it can be difficult to respond and check in on them. Encourage folks to email you to schedule a time to chat on the phone and engage with them personally.

• Keep the phone call informal. You really only need to focus on 4 things:

  • What your company is about

  • What your team is about and how it fits into your company’s grand vision.

  • Learn more about the candidate, what they’re looking for, and why they’re interested in you

  • A brief overview of the hiring process (this leads them into the next step by encouraging them to start your interview process).

Interview unconventionally

You’ve invested a lot of effort into hiring. You may not have implemented all of those strategies but the most important thing about your interview process should be to stand out from the crowd and wow your candidates with a pleasant & fair interview experience.

People loathe contrived interview problems (e.g. reverse a linked list in memory, traverse a 2D array in a spiral). They especially hate when those problems have nothing to do with the actual work they will be doing. So don’t ask them!

People love practical questions that test the skills they use. Interviewing someone on the UI layer? Ask them to implement a mockup. Need an architect? Have them design Twitter. If you engage candidates with problems they will really encounter, you’ll learn their true capabilities and they, in return, will feel properly evaluated (read: happy with the interview experience).

People are impressed by non-traditional approaches. I’ve used tiny take-home pull request exams to engage an often overlooked part of every developer’s core job: reading code. Most interviews require developers to write code, but few ask them to read code.

New things I’ve read recently

For Beginners:

1. https://www.filamentgroup.com/lab/practical-accessibility-1-semantics/ - A shortcut to learning accessibility is to understand when to use all of the HTML tags.

For Seasoned Pros:

1. https://infrequently.org/2021/03/the-performance-inequality-gap/ - The performance budget for mobile even in 2021 is surprisingly low.

For Experts:

1. https://www.smashingmagazine.com/2021/03/complete-guide-accessible-front-end-components/ - Exhaustive but helpful list for understanding accessibility in front-end development.

For Managers & Leaders:

1. https://ochronus.online/the-5-common-mistakes-of-new-engineering-managers/ - So much to agree on here - please stop coding so much, managers!

🙏 That’s all folks 🙏

Aaaand we’re done with issue 130 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

First off, I made an update to last week’s post on culture fit. I got some great comments on how we can evolve the culture fit interview to be inclusive. In retrospect, I agree that we shouldn’t place absolutes on things like this. Culture fit interviews can be useful if (and only if) they are corrected to be more inclusive with a bend on adding to the culture rather than fitting into it.

This week’s post is on my recent obsession with evidence-based scheduling. This came from me diving into lots of articles on engineering leadership over the weekend. I won’t have any posts at the end of the newsletter to promote because honestly there is more than enough great content to sift through in that Awesome list.

Hitting deadlines

My recent productivity obsession is around Evidence-Based Scheduling.

Software estimation is hard. But I also want to make sure our stakeholders understand how far we are along in our projects and when we think we can reasonably deliver some software.

While watching The Last Dance this weekend about the ‘98 Chicago Bulls basketball team, I got to thinking about how great sports teams all review tape from their previous performances.

Who does that in software? I don’t think I’ve seen anyone do that. And yet, if you think about the principles behind the book Deep Work, you have to get as much feedback as possible as to how you are performing at your job.

Practicing estimation is a skill and takes effort and time. In speaking with a few colleagues and friends on how they’ve implemented this themselves, there are a few things to keep in mind.

Buy-in and messaging is key

On the surface, this sounds like micromanagement to the nth degree. Now my boss is tracking my time? I’m not tracking your time - YOU are. It’s really easy for a framework for this to sound like a way to place the lens of Big Brother over your development teams. That is not the goal at all.

Instead, this is about tracking yourself to see how good you are at estimating your own work. Endless articles have demonstrated that we, as an industry, are really bad at estimating how long our work will take. This is putting that into repeated, frequent practice.

There are no points for completing things early and there is no shame in finishing things way off target. The point is not to grade your performance in matching expectations. The point is to see what is your pattern for estimation in the first place and account for that.

Maybe you are the chronic optimist and everything you do actually takes 20% longer than you expect.

Great! Now we can ensure that if you take a ticket we can reliably add 20% to the estimate to predict when any of your work will arrive. The time to complete doesn’t matter; all that matters is the message to stakeholders about when we think this will most likely arrive.

It takes a lot of work upfront

A friend of mine mentioned that his team of 8 took about 8 hours a week in meetings breaking down tickets in grooming and providing granular estimates of the work that was to be delivered. That’s a lot of time in meetings per week, particularly for individual contributors.

At first, that seems like the effort is way too high for a team size that small. However, after a few weeks of working through this system, they were able to get these meetings down to about 2.5 hours while also increasing their accuracy and delivery.

Would you spend 5% of your week ensuring your delivery is 90% accurate to stakeholders? I’d take that each and every week without fail!

Target practice

Remember that all of this is not an excuse to let stakeholders pepper your team with deadlines. It is an anti-pattern to adopt EBS so you can enforce deadlines. Rather, it is a reaction to an anti-pattern of deadline-driven development so you can get your teams out of the corner you’ve already backed into.

The end result of adopting evidence-based scheduling should look something like a bull’s eye target. This analogy I use time and time again with my team leads when one of their stakeholders is adamant about requiring a specific date for something to arrive.

Imagine you are firing an arrow at a target 100 yards away with a 5’ radius. The goal, of course, is to hit the bull’s eye in as few shots as possible. There are two ways to do this:

1. Increase your accuracy. Each time you fire an arrow (i.e. deliver an estimate) it should get closer to your target. In other words, if you think something will take 2 days and you’re off by 1 day this week and then off by 2 days next week, your estimation skills are only getting worse.

2. Move closer to the target. It’s a lot easier to hit a target from 50 yards away than 100. In software, this is just time moving forward. The closer you get to the projected date of delivery (i.e. target) the clearer the bull’s eye becomes. You probably know what you hope to accomplish today but who knows what you’re confident to deliver 9 months from now.

So if you practice hitting a target regularly (i.e. practicing evidence-based scheduling) while simultaneously moving closer to the target (i.e. building more software every day), you should be moving away from deadlines towards target dates which just get more accurate and predictable over time.

At the beginning of the quarter, your project will arrive in 3 months with 60% confidence. In 2 months the project will arrive in 3 weeks in that month with 80%. In that week, the project will arrive on that Thursday with 95% confidence. This is how we shift away from deadlines to target dates.

Stakeholders still hear a date that makes them happy and you aren’t beholden to a rigid delivery date which makes you happy. Everyone wins. That is why I am so excited to try this system out and if you have any experience with EBS please let me know.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 129 of the User Interfacing Newsletter.

No articles this week (see the intro for a great list). If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

Lots of great feedback so far on everyone’s annual review process so I’m glad to see things aren’t as dire as I imagined. A lot of companies I’ve heard from are doing great by their employees with thorough reviews with ample feedback throughout the year. Great work!

There’s one area that I think we could all use some improvement on and it’s hiring. Modern-day hiring practices are broken and I am convinced that will be one of the big things we look back on someday and scratch our heads as to how we got it so wrong. Are you proud of how you hire at your company? Let me know!

On culture fit

A question was asked during a recent panel on Clubhouse about assessing culture fit. What was once an innocuous term for “people who aren’t terrible human beings” has quickly become a trigger for exclusivity, gate-keeping, and homogeneity. Here are my thoughts on how things got out of hand with this hiring practice.

The benefits of finding a cultural fit

I can understand where culture fit is seen as a necessary tactic at some places. At small startups, companies are looking for chemistry between the founders and these early-stage employees. I get it; all of you are going to spend a lot of time together taking a big bet that the thing you are building is going to change the world. I was the first employee at a startup and I spent many nights working with the founders until 4am and it all started with a poker game.

For these early companies, you’re looking for employees to define your culture which is largely based on the principles that the founders already hold dear. So if you, as founders, have a semblance of culture, you’re looking to ensure the keys to your initial success stay intact when expanding to hired employees.

On the surface, this makes sense. Many years later I’m looking back on some of these interviews with these founders and I’m left wondering if the toxic elements germinate even at the smallest of startups.

A culture of homogeneity

I remember a very peculiar interview I had with an early-stage company. We did quite a few technical rounds followed by a round of beers at a local bar. I knew it was about getting to know the founders but in retrospect, it was a sign of being part of the group.

Was I a techie? Did I wear the right hoodies and jeans? Did I know how to hold down a beer (or five) and still gun sling code like a pro? Was I willing to do whatever it took to get the job done in a work-hard-play-hard atmosphere?

This is where the original idea of finding chemistry ballooned into a flawed practice I see at many companies now, both large and small.

Why I don’t recommend culture fit interviews

My answer, by the way, on the panel was to completely do away with culture fit interviews. I think they are a stain on the tech industry that we will look back on in 5-10 years as a “what were we thinking?” kind of clarity. Culture fit interviews promote the opposite of Diversity, Equity, and Inclusion (DEI) - Homogeneity, Inequity, and Exclusion (HIE).

Culture fit is a tactic to find a homogenous group of people. You’re literally trying to determine if people are like you. The common thought here is “can I see myself having a beer with this person after work?” So ask yourself:

• Why do we need to have beers? Why not tea?

• Why do we need to hang out after work? Isn’t a coworker relationship during normal hours enough?

• Why do we need to be out somewhere together? Does that diminish the contributions of my remote colleagues?

Diversity is about different perspectives from different backgrounds. And it’s not just about demographics like race and socioeconomic background. The kinds of companies you’ve worked for, whether big or small, all bring new perspectives about how to run and scale an organization. The kinds of jobs your team has had in the past contribute as well: “purebred” software engineers from CS schools offer a completely different window into building software than folks who never went to college and spent the last 5 years working in retail who just recently transitioned into a software job.

Culture fit is inherently inequitable because it is a subjective evaluation with an unspoken set of rules. There is no way to get a correct answer to a culture-fit interview. That is because every company has a different way to “grade” that question which makes it impossible to provide a way to improve or level the playing field for those that need a boost to balance their technical capabilities and job qualifications.

Meanwhile, equity is directly aligned with providing a boost for those who have been systematically left out of the conversation. If you’re used to talking to white MIT grads from Boston or Silicon Valley then you need to explicitly devote more time and attention to the overlooked BIPOC candidates without the same signaling credentials. You’re trying to explicitly create room and opportunity for those that are often overlooked. If they already aren’t part of the in-group, how do you expect to help them with an in-group culture fit question about grinding exams back at Stanford before a kegger?

Finally, culture fit is exclusive because those unwritten rules I spoke of earlier are designed to weed out people who don’t fit the mold of the prototypical engineer or any other kind of employee you’re used to seeing. The question is all about showing signals that you’re part of the group.

Even without a culture fit question, people will assess culture on first impressions. Have you ever grimaced at an engineer coming into an interview in a suit? Made a snap judgment that the person might be too uptight or corporate? We’ve all subjected others to unfair biases and it doesn’t make things better to put someone through an entire interview centered around this concept.

Inclusive interviews are about bringing everyone into the conversation without adding more reasons to deny someone a seat at the table. It’s about the opportunity for people in a field that is dominated by white males and has been this way for decades. It is our responsibility and duty to represent and include all kinds of people in a way that better reflects the diversity of the people in our nation and in our world.

DEI takes work. DEI is a necessary evolution of job opportunities. And one way we can move in the right direction is to kill the culture fit interview.

New things I’ve read recently

For Beginners:

1. https://www.smashingmagazine.com/2021/02/building-user-trust-in-ux-design/ - Building trust is a basic thing we should all be working on.

For Seasoned Pros:

1. https://cgroom.medium.com/take-home-coding-assignments-are-a-waste-of-time-8da74085749e - The title is actually bad and contradictory; he says coding assignments work in very specific conditions. We use a takehome assignment on my teams and we adhere to all of his suggestions for takehome assignments to be effective. And THAT is what I would want you to focus on when reading this article.

For Experts:

1. https://medium.com/product-at-shopify/how-to-figure-out-if-your-product-actually-solves-problems-885f242ac36 - More people could heed the advice to check risky assumptions and return to first principles when things aren't working out with product-market fit.

For Managers & Leaders:

1. https://medium.com/@meetfelipe/measure-what-really-matters-2de3569e55ec - I largely agree with what is considered good, bad, and ugly with regards to OKRs. Definitely worth the read and clarified a few things for me from the book.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 128 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

Last week I shared my curriculum of how I taught myself engineering management over the last several years. Now that performance reviews have kicked off at my company I thought I would share some tips for weathering the storm so they don’t have to be painful for you in 2021 (because we need 2021 to be less painful than 2020 in a lot of ways).

Stay healthy!

Things to remember with annual reviews

Annual reviews have closed from the new year so we’re now into performance review season. A lot of people I work with seem to have a lot of reservations and anxieties about this time of year. I find that if you keep the following things in mind you tend to have really smooth and easy reviews.

Weave performance reviews into your 1-1s

The first thing I say to all of my directs every year is “there shouldn’t be anything in here that you don’t already know and if there is we should talk about it.”

We never talk about it.

That’s because we bake feedback into every weekly 1-1. Anything my folks are doing well (or need work on) are delivered with no more than a one week delay from the moment the event occurs. This allows us to create a true feedback loop where we see performance improvements several times a year rather than once at the annual review.

In fact, I’ve never had to write an individual development plan for a direct. That’s because we don’t construct massive waves of change in performance. We tackle issues as they come up rather than in the constraints of an annual review system.

Tackle one issue at a time and then move on. If a recurring theme emerges then that is the time to make note of it in the performance review. But that doesn’t change the actual game plan. We still tackle feedback and encouragement in real-time and just make sure to highlight major wins/concerns in the annual review based on the previous year’s evolution of that feedback.

Give folks time to read the review

Don’t deliver your review at the same time your directs are reading your feedback for the first time. This creates information asymmetry and an awful feeling for your directs. They not only waste time in the meeting reading the review you wrote them but they also can’t even begin to process what they have read.

Imagine you gave a particularly negative review. How do you think they are going to feel when reading that information in front of you? Probably some combination of mortified and paralyzed because they not only feel bad but they feel like they can’t show it so they have no place to go. And if they are non-reactive you might believe that they don’t care or aren’t willing to take action on this feedback into the next year. Can you see how this is bad for both sides of the review?

Instead, give directs at least 24 hours to read their reviews before you meet with them. A week is even better. Since you’ve already baked feedback into your weekly 1-1s this should already be an easy read but now you save even more time from your meeting by pushing off their reading time out of your scheduled meeting so at a bare minimum you just gained some time back in your calendar.

Now you can use most of the time for your review to discuss goals for next year and how your feedback can be incorporated into those goals. This brings me to my last point…

If you do these right, performance reviews are forward-looking meetings

If both you and your direct are on the same page throughout the year and you’ve had them read your review a week in advance then why do you even need to meet? To look ahead.

Your meeting just became so much more effective because instead of lamenting on the past you can focus on the future. Now your performance review is about doubling-down on the things your directs have done well and weaving improvements into the areas your directs need improvement. The best way to do this is to show them the goals of your team or function and figure out how your direct plays into achieving those goals.

Now instead of your performance review being a recollection of the last year’s work you can continue to charge ahead with how last year can indicate what the upcoming year will be like.

For this review season, for example, I spent about 10 of the 30 minutes pulling up the review, glossing over it with my directs, and leaving room for questions or concerns (of which there were exactly 0). For half of my directs, most of those 10 minutes were actually explaining how the performance review system works because this is their first annual review. The rest of that time was also spent explaining how our performance review system is going to change in this new year.

The remaining 20 minutes were focused on our function’s goals for 2021 and how we can weave them into their teams’ goals. We talked about the big things we want to accomplish and why they are perfectly positioned to take our business to the next level.

It was aspirational. It was inspirational. It was…exciting. Definitely not what you would expect to hear when you think of performance reviews but if you are doing all of the right things then it doesn’t have to be painful; even if you have under-performers. I’d actually say this system is better for under-performers because they aren’t caught off guard and they can instead be excited to improve in the future rather than be reminded of the past.

New things I’ve read recently

For Beginners:

1. https://responsibleweb.app/ - This is great; it's all on one page and it's a summation of the things you should consider when designing web apps.

For Seasoned Pros:

1. https://www.browserlondon.com/blog/2021/01/04/front-end-predictions-2021/ - I don't know if WASM will be a thing this year but we are certainly seeing monoliths becoming a thing at our company.

For Experts:

1. https://v8.dev/blog/adaptor-frame - Warning: there is assembly code in this blog post. This is a very advanced look at speeding up JavaScript calls.

For Managers & Leaders:

1. https://larahogan.me/blog/performance-reviews-should-be-unsurprising-fair-and-motivating/ - It is performance review season. Are you prepared? I know I do these things.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 127 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

Last week we covered a reading list for learning engineering management if you get promoted. This week we’re going to cover how you can apply this reading list to your own work and how I did it myself.

How I applied the engineering management curriculum

Last week I posted a reading list of the books I have read (or plan to read this year) on engineering management. My concern is that no one teaches you this stuff in an MBA or at work so you’re kind of forced to do this on your own.

Further, it’s a challenge because engineering management is like a mid-career switch. In many ways, you’re starting over from scratch and many of the techniques you’ve applied as a software engineer no longer apply as an engineering manager. So where did I begin?

I first explored some more universal topics

I have to be honest, I don’t actually remember when I started reading from this reading list. At the time I was averaging nearly a book a week so it’s difficult to pin down in the early days what I read first and in what order. But just reflecting back on the list I’m pretty sure I started with some more universally-applicable books like How to Win Friends and Influence People and Never Split the Difference.

I read these two several years ago even before I was an engineering manager. I’m pretty sure How to Win Friends was just a general recommendation on networking and making connections in the industry which has always been a concern of mine. Never Split the Difference became relevant when I started my consulting business and I wanted to make sure I was coming to the table prepared when my clients wanted to talk pricing.

Throughout the years I’ve built side projects I’ve explored lots of books around entrepreneurship so I think that’s where Traction and Venture Deals came in but I know for a fact that was also at the time I was plowing through books. I’m pretty sure I read Venture Deals in one day on a bus ride back from NYC and within the next week, I continued that trend with Zero to One and The Hard Thing About Hard Things.

None of these are management or leadership books but I think it’s important to give you context that I read books at the moment I was experiencing situations I wasn’t prepared for and wanted an expert opinion for how to handle them.

Solving problems as they occur

So given that criteria for reading the first engineering management book I read were when I was a tech lead who was offered the chance to have direct reports informally. And I quickly ran into a tricky interpersonal situation.

That’s when a coworker recommended me The 27 Challenges Managers Face. I had experienced an exact situation that was spelled out in one of the 27 challenges in that book. So my first pass at this book was to solve that exact problem.

My next pass at the book was a quick read of the other 26 issues. I will say I haven’t actually read this book thoroughly cover-to-cover because as I mentioned in the last article, I don’t think this kind of book benefits from a cover-to-cover read. I think it’s great to read quickly to internalize the problems you might face. Later, keep it in your back pocket as a reference guide when you run into tricky situations with your directs.

My first management job

Once I officially earned my first formal engineering management position, that was when I started diving into my real reading list. It was prompted by a few aptly titled Ask Hacker News posts about how to transition into Engineering Management.

There was one comment that recommended the Manager Tools podcast and so I went right to their website and looked for the basics. That’s where the “Management Trinity” was first introduced to me. That weekend I put these three podcast episodes on repeat. Any workout I was listening to them over and over again. I even think I listened to them as I was falling asleep, hoping I would retain this information subliminally.

The podcast is I think over a decade old so once I started listening to the other episodes I heard about the Effective Manager book as a cross-promotion for the podcast so at some point I got the book and read that.

Again, if you’ve listened to the podcast you’ve already hit the main talking points. These were the things that taught me the engineering management basics: 1-1s, coaching, feedback, and delegation. The book and the podcast drill those concepts into your brain with lots of examples.

So for at least the first 6 months of my new management job, all I practiced were those 4 concepts until I felt comfortable in all of them. And then 3 months later I quit.

Starting my consulting firm

After my management job I read most of the chapters in Cracking the Coding Interview because I figured I’d have to do a bunch of coding interview questions even as an engineering manager. I didn’t find any companies I liked that were hiring engineering managers so I started my own consulting firm and became a freelance developer.

At this point, I really kicked up my reading because, in addition to managing teams as an outsider, I had to learn all about marketing, sales, and all of the non-technical things I have never learned before.

This is when I read books like SPIN Selling and Never Split the Difference. You won’t see most of these books in my recommended reading list because that’s for a completely different curriculum on starting your own consulting business which I may someday create a curriculum for as well.

I only ran this company full time for about a year and a half; it would have been longer but my next client turned out to be my next full-time job.

My current company and current role

I’m currently working at the company I thought was going to be my next consulting gig. I passed all of their hiring tests and everything. As soon as I was ready to sign on the dotted line with my consulting engagement they also offered me a full-time position in case I wanted to take that instead. The offer was better than I expected and it offered health insurance so I took the offer and went back to full-time.

For the first 6 months or so I was in an individual contributor role so my management training took a backseat. Soon enough, a management opportunity opened up, and, leveraging my previous management experience and training, I applied for the role of director (the job I currently have).

Once I landed director I knew I had to level up my learning yet again. I was back into voracious book reading mode.

Since I had nailed the basics with The Effective Manager I didn’t feel the need to re-read any older material so I dove right into books that were catered for more senior management. I was reading a bunch of blogs at the time on the topic so I started plucking books from their recommended reading lists. High Output Management came up time and time again so I read it in a weekend.

As I stated in the last article, I didn’t get too much out of it that applies to directors directly but I did cross-reference the 200-point checklist that is listed at the end of the book to ensure I covered as many aspects as I could with the team that I had.

I can’t name any of the tactics off hand right now but I would just encourage you to read the book and nail that checklist. You’ll see that unless you’re a VP or CTO you’re going to have a hard time being able to act on all of the things Grove recommends.

This is more true today because he ran a hardware company in the 90s so doing things like “walking the floor” on any of the assembly lines simply won’t apply to the folks that read this since you’re most likely managing software teams.

The other books I read around this time were The Talent Code and The Little Book of Talent, both of which were recommended in the Drift company podcast time and time again. These were the books where I applied identifying top performers in both retention and hiring. It’s also a great way to identify talent habits for yourself in improving your own performance at work.

Director-level learnings

The first book that really clicked for me as a director was Extreme Ownership. Alongside books, I also listen to a lot of podcasts. Jocko Willink’s podcast is a favorite of mine and just like the Manager Tools podcast, I heard about the corresponding book enough times that I had to read it.

This was the first new concept I learned in a while: ownership and accountability. I knew about these things, sure, but this was the first time I took the ownership part to a new, extreme level. I know, cue eye roll.

But unlike in the past, this was the first time I felt the gravity of my role as a director. I wasn’t just making decisions for one team, I was making decisions for lots of teams and lots of people, even some that I didn’t directly interface with.

I’ve heard this saying before: the further up the career ladder you go, the longer and harder it is to see your successes or failures. That means the initiatives you drive today might take months or years to see any concrete results or changes in culture.

That’s crazy timing because just today I received upward feedback from a skip-level report that I was praised for having been one step ahead of the company on certain initiatives such as building out OKRs that channel down from company or organizational goals.

The truth is I had no plan to be ahead of the curve, I simply saw what other companies were doing, experimented with some tools and tactics, and doubled down on the effective ones. OKRs seemed like a no-brainer to me and I started implementing them over a year ago. More than a year later I am only just receiving feedback from my directs that this was a good tool to use for our teams.

This relates back to the Extreme Ownership because I know everything I teach my teams comes back to me. Every success and every failure I have to own.

It’s really easy to say “this person was late and it is their fault for being lazy.” It’s really hard to see within yourself that they might be late because you never told your teams that arriving promptly for meetings is a cultural habit you want to build.

It’s also really easy to say “this person missed the deadline because they didn’t work hard enough.” It’s really hard to see within yourself that they might have missed that deadline because you didn’t check in with them when they got stuck and that you didn’t reinforce why the deadline was so important to hit in the first place.

At the end of the day, it all comes back to you. I’m glad I learned this lesson now.

A military theme

The next book on my list came from a recommendation from yet another Ask Hacker News thread…because we, as engineers, have never seemed to figure out the transition to engineering management.

I just so happened to have finished a book from a military officer so it was quite impeccable timing that one of the top comments on this thread was about a free leadership book produced by the US Army. I figured if they can train millions of Americans to fight in the most powerful army in the world then they probably have figured out how to train junior officers and generals alike.

ADP 6-22, which sounds like the name of a Star Wars robot assassin, was a very short and to-the-point read that I got through in one night. I should probably read this one again. This is like How to Win Friends and Influence People in that it is so fundamental that you learn something new every time you go back and read it. This is more than just a book for leaders like me, this is a book for every kind of leader in every kind of profession.

Because of this broad appeal, I am saddened to say I can’t recall any lessons that I haven’t already learned elsewhere. I mean we’re talking real basic stuff like how to communicate effectively and how to report up the chain of command. These are all such cornerstone topics I couldn’t tell you if I learned them because of this book or if I learned them years ago and they were resurfaced through this book.

This post is a reminder to myself that I have to read this book again. It’s also a reminder that when I finished reading this book I recommended Jocko Willink cover it on his podcast. He replied back to my tweet but I’m pretty sure he hasn’t covered it yet.

In looking for that podcast episode I just discovered the Army also published an audiobook of this manual so if you’d rather listen to the book now you have 0 excuses. Both are free to download from the US Army website.

High octane learning

The last three books I’ve read on engineering management were probably the most rewarding and the most tactical since I had listened to the “Management Trinity” over three years ago. These were Accelerate, The Manager’s Path, and An Elegant Puzzle.

Accelerate teaches you about the DORA metrics which completely revolutionized how I thought about measuring the performance and effectiveness of my teams. It radically simplified my quest for understanding how strong my developers were without having to resort to controversial tools like Code Climate Velocity and Jellyfish. This book on its own can transform how much work your teams get done through their DevOps practices.

The Manager’s Path does a great job of spelling out the typical career ladder for a manager. Since I didn’t read this book until I was a director, I mostly skipped over the previous roles I had already attained: tech lead and front-line engineering manager.

Director is the next major leap because you go from leading individuals to leading teams via other engineering managers. Now you’re not coaching individuals on how to write effective code, you’re teaching managers all of the things you were supposed to have learned yourself in the books I had already read.

This book helped me dial in on a few new concepts I hadn’t really practiced before: skip level 1-1s, coaching managers, and peer management.

As an aside, I can’t imagine being a director without having read so many books beforehand. What is it like to teach other managers if we’re already really bad at teaching our craft?

I guess this can come through hard-fought experience as well which is why I can’t stress enough that books are one of the cheapest, highest ROI ways to accelerate your career. There is no way I would be successful as a director today if I didn’t make up for the lack of job experience with a massive amount of self-education.

An Elegant Puzzle is the last book I’ve read on engineering management. It’s a pretty large book and I’ve been slowly making my way through it since the past summer. A big reason why it has taken so long is that I learn something new on nearly every page. I find myself reading and re-reading pages because the nuggets are so golden I have to stop and take a second look.

From sizing engineering teams to delivering presentations to senior leadership, these are the kinds of duties I have to perform as a director. I think one of the reasons I’ve been able to stay successful in my current role is because of this book. There are literally too many lessons I’ve learned from this book that it would take me an entire series of blog posts to synthesize them all so I would just encourage you to read the book for yourself.

What’s next

The only two books that were in the curriculum that I haven’t mentioned yet are Nonviolent Communication and The Trillion-Dollar Coach.

I read NVC a year ago right before we kicked off a big feedback week as a company but I found I had already internalized this framework from the feedback topic on the Manager Tools Management Trinity. The exact, step-by-step framework that Horstman recommends for delivering feedback is almost entirely based on Nonviolent Communication. If you’ve read The Effective Manager you don’t need to read this book if you specifically want to deliver feedback as a manager.

I still haven’t read The Trillion-Dollar Coach because as I stated before I think the lessons here are likely above my pay grade. That said, I’m still probably going to read it next because I’m really intrigued to read about how this guy has managed to persuade so many top executives to utilize him as a coach. That story alone is astounding enough that I want to read about it and maybe I’ll pick up so useful management topics.

Finally, if I were to organize the upcoming reading list from my previous post of 40 books, I’d say the next three after The Trillion-Dollar Coach would be: Resilient Management, Turn the Ship Around!, and The Five Dysfunctions of a Team. I’m incredibly intrigued by Resilient Management because I respect Lara Hogan’s work and read her blog. I’ve heard Turn the Ship Around! more times than I can recall and I am a sucker for any book that has a military theme wrapping management concepts. The Five Dysfunctions of a Team is yet another book that has been recommended so many times in my circles that I have to read it sooner rather than later.

I’m sure there are plenty of books I should have read earlier in my remaining list of 40 books and as I read them I’ll be sure to update my curriculum but I’d also love to hear from you what books you’ve found most valuable in your education as an engineering manager.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 126 of the User Interfacing Newsletter.

No new links for this week. From the last installment of the newsletter, I pointed out that links are taking a backseat and will be added now as I find truly beneficial links to share with you all.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,

Adam

Now that we’re a month into 2021 I’m still finding there to be a real lack of quality articles in the front-end world.

So the best way I can solve this problem is to contribute articles rather than complain about a lack of good content. That means you’re going to see two major changes to this newsletter:

1. I will be writing long-form articles every week and publishing them here, in full. That means you don’t need to subscribe to my blog you can just read everything I write in this Substack.

2. I’ll still have industry articles as I find them, but it won’t be the main focus of the newsletter. Maybe 2020 sapped the life out of my ability to find good articles, or maybe there was just less quality content in the feeds I checked last year. But I can’t reliably create a post every week given the current stream of content and the only way I can best serve this community is to de-emphasize that portion of what I’ve built so far.

I know a great number of you appreciate the hand-curation of articles every week. So this may change your interest in the newsletter going forward and I get that. My hope is that the new format will both provide you more value and make the newsletter more engaging and interesting going forward. Any feedback or thoughts please let me know!

Teach yourself engineering management

Engineering management is rarely taught to new managers.

The usual path to management looks something like this: your manager identifies you as someone with potential but there are either (A) no more ranks to climb as an engineer or (B) no good managers they can hire so they put you in the manager slot.

This is troublesome for two reasons:

1. Not everyone wants to be a manager

2. For those that do, they’re thrust into the job without any training or help

If you’re one of the lucky managers who actually wanted to make this change then you need to learn how to manage effectively.

This curriculum is based on what I taught myself over the last 4 years to climb the ranks from engineering manager to director. This curriculum is also the same one I will use to continue up the ranks to VP and CTO as I believe the advice and needs change as you move higher in an organization.

I’m going to arrange this much like you would a computer science curriculum: start with some core courses (books) and then bridge into more advanced concepts, with optional side tracks for certain specialties (or weaknesses) to focus on as you gain more knowledge.

Freshmen Core Curriculum (Mandatory Reading)

I consider this to be the base of all learning for engineering managers - from line managers who call themselves “Engineering Manager” all the way up to founders. Anyone can get something great out of these books and they are the ones that are recommended the most often:

The Effective Manager & The Manager Tools podcast

When I first started out I just wanted someone to walk me through the basics. That’s exactly what this book and the podcast do for you. I’ll simplify it even further: listen to the Manager Tools Trinity. This is your 101 intro course.

If you took nothing else away from this post, listen to the Trinity.

It is opinionated. It might even be considered controversial. But if you nailed one-on-ones, feedback, coaching, and delegation you would be 80% ahead of your peers.

The book expands on these 4 concepts with examples aplenty but I’d recommend you listen to these 3 podcast episodes on repeat until every concept from each episode is drilled into your brain. Nothing has carried me as far into my management career as these three episodes.

Extreme Ownership

This one is so foundational that I would argue this is for everyone, not just managers. In a way, I could consider this to be the prologue to the management curriculum.

This book was written by a Navy SEAL officer who says that you own everything in your life. And I do mean everything. Every failure, every win, and every stalemate is attributed to something you have done.

Your developer didn’t push code with tests? You own that because you didn’t ensure they internalized the policy on tests must be delivered with code.

Your team missed a deadline? You didn’t reduce the scope and have an exit strategy for what could happen if the milestones slip.

No matter what the question, it all comes back to you. The magical thing is once you internalize this concept and lead by example, everyone on your teams picks up on this and they also share in that ownership. It’s an accountability feedback loop that only trends in a positive direction.

Army Leadership and the Profession (ADP 6-22)

Speaking of military books, here’s another leadership book is taken directly from the US Army that you can read for free here.

I didn’t find this book until I was a director but this is one of the highest-recommended books on leadership I’ve ever come across. This is an actual Leadership 101 book. It covers everything from mindset to effective communication to how to actually lead and get others to follow.

While TEM is the most tactical 101 book for your needs as an engineering manager, this is the true 101 must-read for any manager in any profession. If the world’s most powerful military uses it to train their officers then I would imagine it can be a great foundational read for you as well.

The Manager’s Path

Once you’ve nailed the tactical basics with the right mindset, only then can you begin to see the hierarchy of how one develops into a manager from a technical lead. This is a bit more high-level than the other two books because it covers the whole path from tech lead all the way up to VP and CTO.

I like this book because it helps you understand the relevant concepts that are most important at each level. I reference this book every once and a while even as a director just so I can refresh my memory on what is most important at my current level and how to start thinking about operating at the next level.

The 27 Challenges Managers Face

This is your dictionary and thesaurus of engineering management. As the book says, it walks you through how to handle 27 of the most common challenges with the people on your teams. Everything from under-productive developers to personal conflicts, this book walks through each scenario step-by-step.

Because you’re never going to memorize, internalize, or encounter all 27 issues when you first start out, this book is best left to skim once to keep in the back of your head so that when you do encounter a situation, you can remember it was covered in this book and to quickly look it up for actionable guidance in your difficult situation.

Sophomore Curriculum (for Directors)

Honestly, you could stop here and you’d be way ahead of other managers in the field. At some point, you’ll move up in your career and you’ll need to focus on a different set of things.

Instead of learning how to hire engineers and conduct interviews, you’re learning how to create hiring programs. Instead of hiring for a budget, you’re drafting the budget. Instead of building a team, you’re building many teams and training those managers to run those teams autonomously.

These are the books for your next set of challenges:

An Elegant Puzzle

This is arguably my favorite engineering management book. Like The Effective Manager, it has play-by-play guidance on extremely important topics to managers. The difference is that TEM covers the tactics to manage individual contributors (ICs). An Elegant Puzzle, however, covers the tactics to manage and build teams.

As a director, this is my CLRS Algorithms book. It is both my bible and my reference guide for all of the major facets of my job as a director. It covers everything from how to size engineering teams to how to build a hiring funnel.

Given that Will wrote this as a tech leader at Stripe (and is now a CTO at Calm) this book is useful for anyone at the director stage and above. I am sure I will continue to reference this as I transition into senior management roles.

The Talent Code

This is about how to nurture talent and bring out top performers. I debated whether or not to add this to the Core Curriculum because it is something any manager would benefit from learning more about. However, this is more of an advanced concept in coaching and arguably about identifying top talent as a hiring manager.

Therefore, I just don’t see this as a core reading but if you are so inclined to dive more into the coaching aspect of management, this is a great book.

There is also The Little Book of Talent which is like an abridged version of this book with 52 specific tips and traits for fostering talent which is an acceptable substitute for this one if you didn’t enjoy this book. Personally, I like TLBOT more since I found the backstories of some of these talent aspects to be ranting diatribes but to each their own.

Accelerate

This book is not about management. But once you become a director you’ll want to start measuring the performance of your teams to help identify bottlenecks, weak points, and performance gaps. Accelerate provides the most robust, scientifically-backed means with which to do so.

The book is the foundation for the DORA metrics (also known as the “Four Keys”) so you could theoretically skip the whole book and read just this blog post and you would get the gist of what they are trying to accomplish.

If you do want to read more, the actual meat of the study and the takeaways are in the first half of the book. If you’re a real nerd for the scientific method, the second half of the book goes into the details of how the study was conducted and really dives into how they determined the results while the first half focuses on what the results were.

Junior Curriculum (for VPs)

As I mentioned before - I’ve never been a VP of Engineering so I can’t speak to whether these are good books for VPs, but they are books I have read and plan to revisit when I get there because I found them to be unnecessary for my journey so far.

High Output Management

To start, this is a great book and is in the engineering management canon for a reason. That reason is it was written by the former CEO of Intel who helped transform the company into the leading semiconductor outfit in the world.

But that’s exactly it - it was written by a CEO. The things he needs to think about to run a successful organization are not what you need to worry about as a line manager or even as a director. All of the other books above should come before you read this book. I know this because I read it and I don’t remember what I retained from it.

I know it was a great read at the time but the only thing I can remember that applies to my work is that there was a massive 200-something point checklist at the end of the book to evaluate yourself and your organization in areas for improvement. This is exactly the kind of list that is useful for someone who runs an entire engineering organization but is major overkill if you don’t.

And even then, this book was written in 1995 so there are a lot of stories and concepts that are both dated and more relevant to the hardware industry (while I primarily cater to software leaders).

I’ll definitely reference this for the next company I join but if you read the other books first you’d be in a great spot.

Trillion Dollar Coach

Warning: I haven’t read this book yet. That alone should be a reason why you can understand why it is in the higher-level curriculum than my pay grade. Why?

So far, everyone who has recommended this book to me has either been a CEO or a CTO of a startup. And this book is about engineering startup playbooks. This tells me that it’s all about building engineering teams with top-brass leaders in mind.

The title gives that away: the term “trillion-dollar coach” comes from the fact that this management consultant has helped guide the chief executives at Apple, Google, and Intuit, who have a combined market cap well into the trillions.

Browsing the table of contents, terms like “team first” and “envelope of trust” are the kind of visionary, aspirational wordsmithery of those who command a very senior title. Therefore, this is a book I want to get to but I’d rather work through much of my reading list first before I get here.

Senior Curriculum (for CTOs and Founders)

At this level, you’re writing checks or you’re pitching to the people who write those checks. If line managers are tactical, directors are strategic, and VPs are visionary, then you are the foundation and the exemplar. Everything stems from you.

You might be a technical co-founder or you’re incoming CTO of the entire United States in which case thanks for reading my blog, I have no idea how you got here or why you’re taking advice from me.

I have co-founded a venture of mine which is just me and a friend so I can’t say these books help in managing humans but they have been useful as the CTO of that outfit.

Venture Deals

This is a great founder 101 book because it explains all the nuances of setting up a software company, scaling it, and all of the nuances about raising capital and pitching to investors.

While it is definitely framed more for founders than for leaders at established companies, there is some excellent wisdom in here about how to organize your company. There is a great exercise in here about how you split up all of the executive roles based on the number of founders you have and how to complement skill sets based on all of the functions a company needs to operate, such as marketing, sales, and engineering.

What resonates with me as a leader is that building teams require some combination of redundancy and complement in skill sets. Know what skills you need to bolster (e.g. front end) and what skills are lacking to round out your teams (e.g. platform engineering) and it will be easier to scale with the needs of your organization.

The Hard Thing About Hard Things

I’ll admit, this is one of those pop non-fiction books that have a lot of cool stories marrying Ben Horowitz’s love of technology and hip-hop. I came for the stories and stayed for a few nuggets of sage advice. The biggest one being nip rumor culture in the bud.

This one I noticed from a previous startup and when it’s bad, it is toxic. Knowing what it looks like first-hand and how to resolve it (thanks to this book), I’m much more aware of how to maintain a strong culture with integrity and honesty now and how information needs to travel from the top down.

The other one I remember is to take care of your people. Empathy, trust, and psychological safety are tops on my list of skills to build as a leader and there are a lot of examples where Ben was put in a rough situation and had to either fire someone compassionately or ask way too much from his employees. He knew the only way he could ask so much was if he spent the rest of his time caring for his people.

That kind of caring takes years to build with your teams and the only way you will get people to join your crazy ride is to deeply care about how they are as people.

Zero to One

This is another pop non-fiction book from Peter Thiel that everyone seems to reference but is really full of stories and lacking in major tactics. You walk away from books like this and THTAHT with like one or two nuggets and appreciate for what it is: a fun read feigned as education.

Thiel really lets his libertarian pride shine in this one. I’m not trying to get political here but the main takeaway I got from this book is that moats are powerful and crush your enemies with a monopoly.

I think if I were running a venture-backed firm with a bit more fire under my ass this book would have spoken to me more but I found it a bit too cutthroat for me. Your mileage may vary and it is a highly-recommended book so I’m still willing to include it here.

Traction

Another book that is not about management. But the only way you’ll grow big enough to be a manager of people is to build something people want so much that you need to hire folks to build it fast enough.

Traction has a ton of great insights I’ve used to build my side projects into profitable ventures with tactics on how to get users. I haven’t even begun to use all of the tricks in this book but it is absolutely a worthwhile read particularly if you are an early-stage software startup founder.

Specialty tracks and additional books I’ve enjoyed

Many college CS programs offer you the opportunity to specialize in certain disciplines within computer science after you learn the basics. These include artificial intelligence, programming languages, and human-computer interaction.

Similarly, engineering management has a few specialties as well and I’ve read a few books that delve into very particular facets of the management craft:

Influence, persuasion, and soft skills

How to Win Friends and Influence People is a classic among the classics. I think it is one of the most highly-read non-fiction books of all time.

The title is so to the point you might hide the cover on the subway for fear you might look like a social Luddite. It is not that kind of book. This book is not how to make friends for dummies.

This is a book on getting others to do things you want without losing your moral compass. Influence is a very neutral term: to some, it sounds like something slimy a salesman does to convince you to sign on the dotted line. To others, it sounds like the kind of charismatic tactic President Clinton used to win over voters in the presidential debates.

I’d say this book is one of those gentle reminders that you should be nice to people and that being nice to people makes it much easier to get them to see your side.

This book just has to be read (and re-read) to be believed. It should be on everyone’s reading list for the simple fact it’s so widely accessible and in every library that it’s basically free.

Health and productivity

How Not to Die is a mammoth text at over 570 pages. It’s not a management book. It’s not a software book. It’s not even a self-help book by how you would commonly define that genre of book. This is a book on healthy eating.

In a year like we’ve just had, health has become a front-and-center reality for us all. Healthy habits have been shown to increase productivity.

I live by six healthy habits that doctors recommend every day:

1. Eat right

2. Exercise

3. Don’t smoke

4. Drink in moderation

5. Reduce stress

6. Get enough sleep

This book tackles habit number 1. It explains how this doctor’s list of a dozen food groupings help reduce all-cause mortality for all but one of the top 20 reasons people die (the one it can’t help are car accidents). From obesity to cardiovascular disease, all of these foods and habits help you improve your health through the basics we’ve learned all along: a diet rich in whole fruits, vegetables, legumes, and grains is shown to improve health.

There is even an app for both iOS and Android to help you track your daily progress on eating these key foods. I use it and my Apple Watch as one of my key health staples for ensuring I meet these 6 habits every day.

A healthy manager is a productive manager operating a peak performance.

Giving feedback

Nonviolent Communication spells out a four-step process for delivering feedback. This one was a game-changer for me on how to speak to people in a way to promote behavior change without casting blame or making people feel bad.

If you want the ultimate deep-dive into how to deliver feedback that resonates with your directs, this is a book to read and study carefully.

Salary negotiation, hiring, and retention

Never Split the Difference is great whether you are looking for a new job or you are trying to hire a key candidate. It was written by an FBI hostage negotiator who takes a pretty hard stance on how to jump into a negotiation.

This is another one of those books that take 300 pages to convey a few key points as most pop self-help books do these days. I thought the book was entertaining so I didn’t mind the fluff but the biggest takeaway I got from this book is the phrase how am I supposed to do that?

If I recall correctly, the author was put on the spot to negotiate a really tough situation like how to give terrorists millions of dollars or else they will kill the ambassador. Most people who read this panic and have no idea what to do and figure the author is totally trapped into this nearly-impossible scenario. His response is clutch: “how am I supposed to do that?”

In other words, if the person on the other side of the table is being unreasonable, turn their poor offer back on them through empathy. Let’s say you’re making a cool $150K at your current job and the company interviewing you just offered you $110K for acing their interview.

You can reply with “how am I going to be able to take that offer when I’m already making more than that?” You’re putting the onus on the hiring manager to come up with a really good reason why they should make the obviously irrational choice of choosing to leave their current job for a worse one with a lower salary. This forces the hiring manager into a situation where they have to come back to the table with a stronger offer.

As a manager, you can use this tactic as well…but I’ve already said enough about this book. It’s great and it goes into a bunch more tactics on negotiation which you can use when looking to hire new members for your team(s).

Interview process & design

Cracking the Coding Interview is the canonical reference for developers looking to pass the interview at the FAANG companies and land that big salary. The whole purpose of this book is to game the system like one does for studying the MCAT or SATs with practice questions used at real companies.

I’m not recommending this book so you can just copy questions from here to use in your interviews. In fact, I am recommending the opposite: this book is a great way to identify which questions are complete nonsense.

If you’re a web startup you have no business asking JavaScript developers about bit manipulation and bit shifting. Is it useful knowledge? Possibly. Do any of you read this article actually use heavy bit shifting techniques in your code? Extremely unlikely.

The only time I found a bit manipulation interview question praiseworthy was an interview I did many years ago for a hardware company that built IoT devices. They actually used these techniques and the interview question was actually really fun.

For the rest of you, this is a net-negative and just makes you look pretentious and leading to more false negatives than you can stand to lose in the hiring process.

The bonus for reading this book is if you ever decide to become an IC again you’ll have no trouble getting a job or grinding LeetCode.

Books on my list that I haven’t organized yet

Finally, here are the books on my reading list as a manager that I haven’t read before and I don’t know where to bucket them in the curriculum yet. As I read them I’ll update this post with where they should fit. For now, here are the books on my reading list that I hope to get through in 2021:

• The Art of Leadership

• The Charisma Myth

• The Coaching Habit

• Confessions of an Advertising Man

• Content Inc

• Deep Work

• Difficult Conversations

• The Effective Executive

• First, Break All the Rules

• The Five Dysfunctions of a Team

• High Growth Handbook

• It’s Not How Good You Are, It’s How Good You Want To Be

• Leadership Step by Step

• The Making of a Manager

• Managing Humans

• Managing the Unmanageable

• Measure What Matters

• Mindset: The New Psychology of Success

• No Hard Feelings

• The ONE Thing

• Only the Paranoid Survive

• The Phoenix Project

• Quiet Leadership

• Radical Acceptance

• Radical Candor

• Righting Software

• Running Lean

• Software Engineering at Google

• Sprint

• Start with Why

• Strategic Logic

• Team of Teams

• Team Topologies

• Technical Leadership and the Balance with Agility

• Technology Strategy Patterns

• Thinking in Promises

• The Timeless Way of Building

• Turn the Ship Around!

• The Win Without Pitching Manifesto

• Your Code as a Crime Scene

New things I’ve read recently

For Beginners:

1. https://www.smashingmagazine.com/2020/12/practical-introduction-dependency-injection/ - A simplified explanation (with solid examples) of this basic principle.

For Seasoned Pros:

1. https://dndkit.com/ - I don't know why Drag-n-Drop has been so elusive on React (in a lightweight way) but I finally found something worth using!

2. https://www.smashingmagazine.com/2021/01/front-end-performance-2021-free-pdf-checklist/ - Everything related to performance under the sun. Extensive and will take a long time to get through but you can probably skip over the obvious things you already know.

For Experts:

1. https://measuringu.com/nps-confidence-intervals/ - Measuring confidence intervals involves just a little bit of statistics and math but can be worth it to assure your surveys are reliable in what they are showing.

2. https://i.redd.it/scthk1rsup961.jpg - TIL "prefers-reduced-motion" is a thing in CSS for a11y. Nice!

For Managers & Leaders:

1. https://allenc.com/2021/01/productive-enough/ - I also no longer understand why GTD would be considered out of style?

2. https://candost.blog/how-to-stop-endless-discussions/ - We're actually thinking about this on my teams - how can we generate feedback asynchronously to ensure we stay productive without creating endless meetings?

🙏 That’s all folks 🙏

Aaaand we’re done with issue 125 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

Most people think that January 1st represents some magical reset button for the world and how they can operate. Yes, 2020 was a bummer and 2021 is looking to be much brighter. But there is nothing inherently special about tomorrow that will unlock who you want to become. There is no better time than the present to appreciate what you have and begin to go after the things you want.

So here’s to the final newsletter of the year without having to wait for some arbitrary clock to decide when we should get started on our goals!

For Beginners:

1. https://css-tricks.com/2020-was-not-a-good-year-for-learning/ - This might explain why I've found it tough to find good content lately...there hasn't been good teaching going on!

2. https://www.zainrizvi.io/blog/the-interviewing-advice-no-one-shares/ - The new year means interviewing so get a head start before the year begins.

3. https://www.youtube.com/watch?v=VfGW0Qiy2I0 - No better time to hone in on all the fundamentals of web development than right now!

For Seasoned Pros:

1. https://css-tricks.com/how-to-load-fonts-in-a-way-that-fights-fout-and-makes-lighthouse-happy/ - I did this for my personal sites and it is hard to do and some extra work but worth it for a great experience!

2. https://www.smashingmagazine.com/2020/11/weaving-web-accessibility-usability/ - Not just about the basics of accessibility guidelines but also how to deliver this to customers when doing user interviews.

3. https://blog.pragmaticengineer.com/performance-self-review-for-software-engineers-with-an-example/ - It's annual review season, so don't forget to write something great about yourself!

For Experts:

1. http://wondermark.com/garfield-color/ - What a wonderful look into a thing I never thought I'd think about.

2. https://www.smashingmagazine.com/2020/12/design-simple-ui-complex-solution/ - A thoughtful (though cursory) look at how we can simplify UIs with examples.

3. https://www.smashingmagazine.com/2020/12/feature-prioritizing-ways-reduce-subjectivity-bias/ - Reducing bias is useful regardless of your field.

For Managers & Leaders:

1. https://jeffchen.dev/posts/Technical-Interview-Checklist/ - I will be filling in the gaps in my own process with this list.

2. https://stackoverflow.blog/2020/12/07/measuring-developer-productivity/ - This productivity article I’m going to keep in my back pocket for a while.

3. https://www.businessinsider.com/biden-cto-vaccine-distribution-climate-change-2020-12 - The upcoming technical leadership in the White House is in for a huge challenge. What challenges are we up for in the new year?

🙏 That’s all folks 🙏

Aaaand we’re done with issue 124 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

Turns out you’d much rather receive less email but keep it super relevant and full of articles. As much as I’d love to send the emails weekly I’ve really struggled to keep the quality of the content high with that kind of frequency so it may continue to become a bi-weekly (or even monthly) newsletter.

Thanks for the feedback and enjoy this round!

For Beginners:

1. https://old.reddit.com/r/learnjavascript/comments/jic377/javascript_string/ - This is about as basic as it gets for a beginner - how to use strings!

2. https://joshwcomeau.com/operator-lookup/ - This is a great tool for understanding all of the operators that are used in JavaScript. Really fun to play around with!

3. https://www.smashingmagazine.com/2020/11/authenticating-react-apps-auth0/ - Learning how to authenticate without rolling your own authentication is a really smart move.

For Seasoned Pros:

1. https://blog.bitsrc.io/best-practices-with-react-hooks-69d7e4af69a7 - I like the recommendation to use useState with an object as opposed to a series of separate state hook calls.

2. https://www.stillbreathing.co.uk/2016/10/13/technical-credit - Technical debt is making the code worse. Technical credit is making your code better. First time I've heard this phrase before.

3. https://allenc.com/2020/11/how-much-engineering-side-projects-help-with-engineering-careers/ - People want to see your software skills when working on a team, not in a silo for your solo projects.

For Experts:

1. https://www.a11yproject.com/posts/2020-10-15-no-mouse-days/ - I love the idea of a no mouse day to find empathy for those with disabilities requiring keyboard navigation. I bet it is harder than you think, especially if you haven't used Vim or Emacs before!

2. https://danuker.go.ro/the-grand-unified-theory-of-software-architecture.html - I'm reading Clean Architecture right now so maybe I don't have to after reading this article.

3. https://secretgeek.github.io/html_wysiwyg/html.html - If you've ever wondered what a quine is, here is an example illustrated in HTML.

For Managers & Leaders:

1. https://medium.com/pinterest-engineering/how-a-one-line-change-decreased-our-build-times-by-99-b98453265370 - This is only about cloning repos, but I think the core lesson here is to give your teams the freedom to innovate even when things already seem like they are doing just fine.

2. https://blog.pragmaticengineer.com/what-becoming-an-engineering-manager-is-like/ - Not sure if lonely is the right word but it certainly creates a level of separation between when you were an IC and now in your position as an EM.

3. http://feedproxy.google.com/~r/AmSciPizzaLunch/~5/dj8lWCw8i6g/20200701_Lima.mp3 - The first data visualizations are over 40,000 years old!

🙏 That’s all folks 🙏

Aaaand we’re done with issue 123 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

For those of you dedicated followers that have emailed me, you know that this is not run by a robot or a marketing firm - I am a real human hand-crafting these emails every week(-ish) with the articles I read. So I won’t send you anything unless I know it is good which is why I skipped yet another week.

I’m now realizing there are a few options here with the newsletters, and I’d love your opinion if you could share it with me:

• A. I’d like smaller, more frequently newsletters. Instead of 12 articles across 4 levels of expertise, I could send 1 a day but only cover 1-3 most-read articles with more commentary. Based on the volume of articles that come in my RSS feed (typically 300-400 a day) I can almost guarantee 1-3 good articles a day that will get my attention. Or…

• B. I’d like fewer, higher-quality newsletters. Continuing the strategy I’ve been doing, I can keep it to (roughly) once a week but instead focus the collection on a summary of the week (or even month) but spend more time (and length) per newsletter issue.

Would love your feedback, and depending on the responses I’m going to try out different strategies and see where you land.

Now on to the articles!

For Beginners:

1. https://www.smashingmagazine.com/2020/10/react-validation-formik-yup/ - Formik and Yup are pretty standard in React these days so this would be a good one to study up on.

2. https://www.youtube.com/watch?v=kClcUzAhz9g - A nice, short 10 minute video on how Styled Components work.

3. https://css-tricks.com/a-primer-on-the-different-types-of-browser-storage/ - You should learn how browser storage can work (such as localStorage, indexedDB, and cookies).

For Seasoned Pros:

1. https://www.hassellinclusion.com/blog/accessible-design-reviews/ - I didn't think about text resizing in this article about accessibility reviews.

2. https://www.youtube.com/watch?v=VslCVu7T9iw - Need a database to test your app? Try EasyBase.

3. https://measuringu.com/revisiting-the-left-side-bias/ - Should you change the way you design forms? I don't think so. I feel like the answer is "people read left to right". I'd like to see this study compared with an RTL language like Japanese.

For Experts:

1. https://www.smashingmagazine.com/2020/10/typescript-in-50-lessons-pre-release/ - In just 50 easy installments you too can be a TypeScript expert.

2. https://www.smashingmagazine.com/2020/10/supercharge-testing-react-applications-wallabyjs/ - Looking for a new testing system? Try wallaby.js!

3. https://davidwalsh.name/cookiestore - Geez cookies will never die, huh?

For Managers & Leaders:

1. https://vasilishynkarenka.com/learning/ - I use basically all of these tactics and it definitely works (hint hint: start a blog or newsletter!)

2. https://gist.github.com/sw-yx/7aeedbeac1bb81017cd4f9d66b223b63 - If you're a leader you should be hiring from diverse backgrounds. No excuses - get on this!

3. https://increment.com/documentation/ - I only discovered this issue but it is super important if you're a leader to document things well for your teams.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 122 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow.

Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that ↩️ reply button and let me know what you're up to!

Cheers,
Adam

I’ve run this newsletter for over two years and I rarely skip a week. This time I actually skipped two weeks which is completely unlike me. I’ve always maintained that I won’t send out a newsletter that isn’t valuable to you. In this case, I didn’t find many solid articles in the last few weeks. I could have easily stuffed the newsletter with a bunch of filler articles to meet my quota - but I just won’t waste my time reading them (and neither should you). So I hope the hiatus was worthwhile, and I’ll continue to skip weeks if I ever find a shortage of quality articles.

Glad to see you again!

For Beginners:

1. https://www.smashingmagazine.com/2020/10/understanding-typescript-generics/ - Friendly reminder that this author is 18 years old and is teaching others on the basics of TypeScript generics. Incredible!

2. https://css-tricks.com/styling-complex-labels/ - This is a great, short tutorial on styling labels.

3. https://www.youtube.com/watch?v=fTBwD3sb5mw - Lots of positive feedback from beginners here on how to build a virtual machine in JavaScript.

For Seasoned Pros:

1. https://davidwalsh.name/vital-web-performance - Web performance is now considered a "vital" activity by Google. Please act accordingly.

2. https://www.smashingmagazine.com/2020/10/face-motion-typography/ - Looking to create a cool demo at a hackathon? Check this article out and be prepared to wow and impress your coworkers!

3. https://w3c.github.io/web-share/ - This is a brand new API draft which is pretty cool. Excited to not require external share widgets anymore.

For Experts:

1. https://css-tricks.com/accessible-web-animation-the-wcag-on-animation-explained/ - Remember the phrase "pause, stop, hide" for animation accessibility.

2. https://changelog.com/posts/good-reason-experienced-devs-say-it-depends - Lots of people say "it depends" because they've seen enough situations to give an answer either way. Lawyer and doctors are probably the ones you hear from the most.

3. http://antirez.com/news/134 - A short and sweet article on why open source is both freedom and a huge responsibility on the author.

For Managers & Leaders:

1. https://medium.com/better-programming/5-mistakes-that-engineering-managers-make-75aecf42525c - I'm pretty good about most of these, but I definitely get in the weeds too often when I should be delegating.

2. https://thoughtbot.com/playbook - Reading through this playbook now for a rubric for a software development life cycle.

3. https://fellow.app/blog/2020/expert-tips-for-new-managers/ - Lots of good folks interviewed here.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 121 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow. Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that reply button and let me know what you're up to!

Cheers,
Adam

Every week I find myself cobbling together the last few bits of articles at the last second. I might want to work on time management. That one is gonna be a serious read (and re-read) this week. What are you most looking to read in the upcoming week?

Stay safe,

For Beginners:

1. https://blog.greenroots.info/my-favorite-javascript-tips-and-tricks-ckd60i4cq011em8s16uobcelc - These are great tips for a beginner. I actually didn't know about Number.isInteger - even I learned something!

2. https://css-tricks.com/working-with-javascript-media-queries/ - You read this right, it does say JavaScript (not CSS) media queries.

3. https://medium.com/@michael.karen/getting-started-with-modern-javascript-arrow-functions-d38053b27e31 - This is a great back to basics post on arrow functions in JavaScript.

For Seasoned Pros:

1. https://web.dev/content-visibility/ - Cut down rendering time by almost 90%? Sign me up.

2. https://charity.wtf/2020/09/14/useful-things-to-know-about-engineering-levels/ - I feel like so many earl-to-mid-level engineers need to hear. Though I disagree about generalists - they can level-up earlier in their career but the further up the ladder you go the more you need to specialize and have a niche.

3. https://medium.com/@Sentinelone_tech/dependency-injection-in-typescript-from-scratch-d1a4422043a0 - Learn you a dependency injection for greater good.

For Experts:

1. https://www.tlt21.com/build-vs-buy-decisions-in-the-age-of-software-abundance/ -Build the tool or buy it? Great recommendation from a coworker for those of you who are in charge of some serious projects.

2. https://www.patkua.com/blog/multipliers/#more-1181 - This guy normally writes for managers but I think this important for you lead/staff/principal engineers because the first moment I realized the idea of being a "value multiplier" was when I started mentoring as a tech lead. Very worthwhile read whether you are a manager or an IC.

3. https://charity.wtf/2020/09/06/if-management-isnt-a-promotion-then-engineering-isnt-a-demotion/ - No, you do not have to go into management to keep growing in your career as a developer. Don't let your boss tell you otherwise.

For Managers & Leaders:

1. https://knowyourteam.com/blog/2020/09/14/the-13-best-questions-to-ask-during-the-remote-onboarding-process/ - I definitely ask some of these questions when people join my org but I still think some of these are awkward to ask (like pet peeves or what bugs you).

2. https://medium.com/the-leadership-journal/ten-worst-answers-you-should-not-give-when-interviewing-for-a-manager-role-468881104304 - I haven't personally given these answers before but I've heard them in engineering manager interviews I've given so don't be one of these folks!

3. https://quietstars.com/lax2020/ - This was a recommended talk to me on setting OKRs. I love this and you should check out the accompanying links that come with the talk.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 120 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow. Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that reply button and let me know what you're up to!

Cheers,
Adam

It just hit me that I’ve been doing this for over 2 years now. That’s wild. Thank you for all of my day ones who have been reading this since the beginning! Keep reading and stay safe out there.

All the best,

For Beginners:

1. https://dev.to/liviufromendtest/5-projects-you-can-build-to-become-a-better-web-developer-221i - These are actually solid projects for beginners that aren't the same old Todo list.

2. https://www.youtube.com/watch?v=tyJelsUG-z4 - If you're interested in game programming this could be fun.

3. https://old.reddit.com/r/funny/comments/iqxm0r/special_skills/ - Alright this last one is cute. Hang in there!

For Seasoned Pros:

1. https://www.smashingmagazine.com/2020/09/component-library-react-emotion/ - This is a great take on building component libraries from first principles.

2. https://www.benfrederickson.com/github-wont-help-with-hiring/ - If you're changing jobs this is some food for thought though I think plenty of interviewers check out Github profiles (I check every single one that comes through my group, at least).

3. https://css-irl.info/why-i-dont-have-time-for-your-coding-challenge/ - I guess these are reasonable objections but at the same time as a hiring manager I've seen a real influx of unqualified candidates, and it can be a good way to filter out bad candidates.

For Experts:

1. https://jaredpalmer.com/blog/react-is-becoming-a-black-box - The hottest of hot takes coming directly from the horse's mouth (interesting take on the state of React directly from an engineering director at Facebook).

2. https://www.youtube.com/watch?v=heh4OeB9A-c - This goes beyond API design but in general how you should code functions and classes for maximum usability.

3. https://diverseeducation.com/article/189682/ - As more things shift online with the current state of the world we have to cater to a more accessible-focused audience.

For Managers & Leaders:

1. https://www.youtube.com/watch?v=Opjq1JT_dS8 - Behavioral economics is an important facet of user experience that combines psychology and economics. A good watch if you're looking for the latest and greatest thinking in UX research.

2. https://www.itechart.com/blog/measuring-performance-of-dev-teams-webinar/ - A free webinar on Monday for leaders. Check it out!

3. https://hbr.org/2002/11/the-flaw-of-averages - Alright so I won't be averaging things out as frequently anymore.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 119 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow. Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that reply button and let me know what you're up to!

Cheers,
Adam

I just moved into my new home so I’m curious: what is your dream home office setup? I decked out my home office space and I’m really happy with the result but I’d love to hear from you what you like about your dream space in case I’m missing anything.

Now onto the links…

For Beginners:

1. https://meiert.com/en/blog/minimal-web-development/ - I will be sending this to the beginners I know. I just don't see most of them reading the entire HTML or ECMAscript spec.

2. https://css-tricks.com/heres-how-i-solved-a-weird-bug-using-tried-and-true-debugging-strategies/ - A good script for beginners to follow on how to debug problems you run into.

3. https://fullstackopen.com/en - Wow how have I not seen this before? I'm probably going to take this course.

For Seasoned Pros:

1. https://www.smashingmagazine.com/2020/09/building-react-apps-storybook/ - If you haven't used Storybook yet now may be the time to get started. No more excuses!

2. https://jgthms.com/web-design-in-4-minutes/ - Clever. Very clever.

3. https://blog.pankaj.pro/are-you-using-trailing-commas-in-your-javascript-ckel799m200oiijs17m9f0iir - Trailing commas are nice. Leading commas are next level.

For Experts:

1. https://ericwbailey.design/writing/a11y-is-web-accessibility.html - The origins of "a11y" and web accessibility as well as why it's not completely about helping people with disabilities.

2. https://accessly.io/blog/caption-101-how-do-captions-work-on-videos/ - Captions are a great way to add accessibility to your HTML5 videos. They are also not the easiest to understand so this is worth the read.

3. https://www.patkua.com/blog/book-recommendations-by-engineering-leaders/ - Not just for leaders, there are a lot of advanced topics here I encourage you to read.

For Managers & Leaders:

1. http://stewiewrites.com/make-an-i-wont-do-list/ - It's a short list but I do think as a leader it is valuable to say no more than you say yes to things.

2. https://medium.com/@tsleslie84/theres-a-big-difference-between-a-team-and-a-working-group-8d5f21204a6a - It was my understanding that working groups are temporary but I'm not sure if there's a broader definition here to consider.

3. https://www.theatlantic.com/family/archive/2020/08/how-make-friends-lonely-boss-workaholic/615709/ - People hate interacting with their boss. Something to consider if you want to move up the management ladder is it gets lonelier at the top.

🙏 That’s all folks 🙏

Aaaand we’re done with issue 118 of the User Interfacing Newsletter.

If you got something out of this newsletter, feel free to forward it to your friends, family, and/or coworkers to help it grow. Interested in sponsoring the newsletter? Have questions, comments, or corrections? Hit that reply button and let me know what you're up to!

Cheers,
Adam