Hello friends, it’s been a while since we’ve had a Normcore Tech. This has become my signature first line every time I release a newsletter, so sorry in advance, but also hi again and hope you’re doing well.

Last time we chatted, I was feeling lost on social media. The time before that, I was running a conference. And now, I am coming to your inbox because I’ve been working on something even bigger and more ridiculous.

Bigger than reverse engineering the Duolingo owl? Bigger than reverse-engineering diamond hands? Yes and yes. You may be noticing a pattern here, and one of the themes of this newsletter has been that that I like to reverse-engineer stuff until it makes sense to me.

The latest thing I reverse-engineered were large language models. I wrote a 70+ page paper about embeddings, for all audiences. Here is the site introducing the project. Here is the PDF. And here is the code. As I mentioned on Twitter, all of this is free, but it cost me everything.

A few years back at work, I started working with embeddings, which are a foundational data structure that neural networks, including today’s large language models. They were great, but a lot about the way they’re presented in industry didn’t make sense to me. So, I started to dig.

What started as a simple blog post became longer and longer, morphed into a survey paper written in LaTeX, a deep dive into how embeddings work, why we use them in a business context, their engineering considerations, a history of NLP, (kind of like this), and how we use it in recommender systems.

Over the past year, I’ve worked on refining this understanding, stealing away hours before the kids woke up at five in the morning to reason through hundreds of academic papers and tie together lots of loose threads. Even though it doesn’t contribute any new research to the field, it was a very grueling process, one I imagine many researchers go through on a regular basis. At one point I texted my friends, who have PhDs, and asked, “Is it supposed to be this hard and lonely? Was it this bad for you?” and they said yes and then told me never to bring up their dissertations again.

But it’s also been extremely rewarding to ignore the unbearable noise of the LLM hype train on social media and really dive deep into something I wanted to understand, and, hopefully, come out, mostly unscathed on the other side with a much deeper understanding.

Now I’m done, and it’s here. Although it was mostly a learning process for myself, I hope that others looking to learn more about embeddings get something out of it, too.

Enjoy! See you in the latent space, and also hoping to also put out a real newsletter soon, once I stop seeing LaTeX symbols in my mind.

Hello friends, it’s been a long while since we’ve had a Normcore Tech.

Quite honestly, I’ve been feeling a little lost social media-wise since Twitter 1.0 ended. Between Twitter, the decline of the tech sector social media giants in late 2022 and into this year, and ChatGPT bursting onto the scene in late December, it feels like we are in a Very Large Vibe Shift.

Long before Elon Musk came in like a wrecking ball to Twitter, executives there pored over an alarming and important chart. Heavy tweeters, who were responsible for 90% of Twitter’s generated content but made up only 10% monthly active users, were in retreat, particularly in key topic advertising areas such as celebrities. The theory was that these content creators were migrating to TikTok. What filled the void was a spike in NSFW and crypto content (and if you remember, crypto ruins everything.) 

Around the same time, other social platforms the same age as Twitter were struggling: 

Facebook lost daily users for the first time in its 18-year history — falling by about half a million users in the last three months of 2021, to 1.93 billion logging in each day. The loss was greatest in Africa, Latin America and India, suggesting that the company’s product is saturated globally — and that its long quest to add as many users as possible has peaked.

Giphy, along with Imgur, an extremely hot startup serving gifs, fell

Its valuation is down by $200m from its peak in 2016 and, more importantly, its core offering shows signs of going out of fashion. “There are indications of an overall decline in gif use,” the company said in its filing, “due to a general waning of user and content partner interest in gifs.

“They have fallen out of fashion as a content form, with younger users in particular describing gifs as ‘for boomers’ and ‘cringe’.”

Although the quiet redistribution of users to TikTok, BeReal, and smaller platforms like Discord has been going on for several years at this point, once Elon took over Twitter, this trend was laid bare. People are disengaging with the current state. 

I’m not very clear on where we’re headed yet, but that’s not stopping me from writing a newsletter in extremely vague terms so I can say I was right later on. 

Money is to be had, as Jim Barksdale said, in bundling and unbundling. The last fifteen years were about bundling the way we mass-communicate, consume, and navigate that mass communication. The next ten will be about breaking all of it up, and then rebundling it all over again in some new format.

 For consuming social media, early adopters in the short-term will leave and opt-out, switching to private media like Discord, Slack, and group chats, and trying out Mastodon, creating a private, fragmented, unsearchable social media landscape (and believe me, the Mastodon people REALLY don’t want you to search it.)  I’ve already seen this happen in my own circles. As of today, I am personally on: still Twitter, Mastodon, Slack, Discord, Whatsapp, Viber, Telegram, SMS, Signal, and, recently, the Substack chat feature. All of these are trying to build some variation of the same user experience. 

A lot of them will end up dissatisfied. Many mainstream users will continue to use platforms like Facebook and Twitter as they degrade in quality to the point where they are basically unusable wastelands of advertising injected with the occasional piece of content, and then also give up and leave. Where to, is unclear at this stage, but it won’t be any one thing.  We are unbundling social media consumption.

Social media producers and Creators, on the other hand, are migrating to, and continue to places which are more stable and tightly-controlled, like TikTok, as long as it’s not banned (like it already is in India, and possibly headed in the same direction in the States),  or go on and explore new areas, like Mastodon and Discord, or start composing tweets in a Google doc just in case (me).

At the same time, with regards to social media creation, tools like ChatGPT and StableDiffusion make it easier to mass-create art, and easier for places like TikTok to copy it. It will truly be harder (but not impossible as long as humans have humanity and imagination) for creators to be individuals in this new economy and to particularly stand out.  

So what comes next for Normcore Tech? Should I use the newsletter as my new Twitter feed, sporadically emailing you things I’m working on? Should I go into deep hiding and unplug everything? Should I start trying to shitpost on LinkedIn (it’s too hard, believe me, I’ve tried.)? I’m not sure.

But if you have any ideas, feel free to email me.

In the meantime, here’s a list of recent things I’ve written:

• NormConf happened in December and it was amazing! The talks are all up on the website, and transcribed as well. I wrote about what I learned about running it here. 

• I have been deep in the ML literature and am reading up on all these new ML paradigms that are coming out. I looked at the Stable Diffusion safety filter and just yesterday wrote about how we should understand ChatGPT

• The thing I am mostly working on this year is Viberary, my semantic search/recommendation side project. So far I’ve looked into the data, did some basic profiling, and am deep in …. deep learning at the moment.

• I also wrote about everything I did in 2022

• On the personal side, I did a really cool thing - I beta-read Wohpe

See you on openTwang.

A few years ago, I read the Southern Reach trilogy, by Jeff Van Der Meer. It’s an engrossing story about a group of four scientists who, backed by the government, form an expedition to explore “Area X”, an abandoned natural park somewhere in the wilds of Florida that seems to evolve on its own. The group includes a biologist, whose husband had been part of the previous expedition to the area, Expedition 11 and returned under mysterious circumstances.  The trilogy of books revolves around uncovering the mystery of the area and why it alters the makeup of people and animals who wander into it. 

One of the main concepts of the books is the idea of a terroir, a self-contained natural environment that shapes everything inside of it. Usually when we talk about a terroir we’re referring to wine: a wine from a given region tastes like wine from that given region should taste because of the grapes and the soil and the way the sunlight hits that particular spot. Since I left my last  job, I’ve been thinking a lot about the idea of terroir as it relates to the workplace. 

Every workplace, like every Tolstoyan family,  is unique in its own way.  When we start a job, we enter that terroir with the intent to shape it. But in turn, we are also shaped by it. If you are a thinking and feeling human being, it is impossible not to become at least somewhat symbiotically attached to a workplace and have it shape you.

Every workplace has its own unique culture, its rhythms, its acronyms, its main characters and side characters, that one piece of software that everyone hates, and that one engineer that everyone relies on. A job can be just a job, but, regardless of whether you love or hate yours, our jobs and their component environment becomes terroirs where we work, struggle, hate, love, and places that shape who we are as human beings. If you have been in a terroir long enough, you will come out of it, for better or worse, transformed. 

But, what happens when we leave (or are forced to leave) the terroir? Usually, when humans deal with big transformations in our lives, we process them through collective ritual. Rituals at the boundaries of transformations are one of the critical parts of being human.  It’s the way we relate to each other and once again understand our own role in society. 

We’ve been doing this for thousands upon thousands of years. For example,  here’s what Charles Segal has to say about ritual in “Singers, Heroes, and Gods in the Odyssey”, an academic book of essays re-examining the works of Homer: :

Passage, transition, change: this phenomenon is the most familiar of human life and also the most difficult to comprehend. It is no wonder that this theme, in one form or another, is one of the central concerns of Greek philosophy, from Heraclitus and Parmenides to Plato and Aristotle…. In most societies, changes of state are marked by rituals that define change, visibly confront its reality, and orient the individual in the world he or she has entered.

To help with this, humans have invented millions of rituals over the years: birth rituals, death rituals, yearly holidays that deal with seasonal changes, rituals for how to make sure you attract money, rituals for launching ships. 

One of the most interesting documented rituals I’ve come across while researching for this post this is was “Reflections on Greetings and Farewell on a Visit to the Royal Court in Norway in 1302”, when some dude named Erik wrote a poem about what court life was like and included numerous points where women ceremonially wept in farewell (probably from relief that their business meetings were finally over.)

Seafaring transitions and spaceflight are particularly bound to ritual, because there they are so long and significant and there are so many things that can go wrong: 

It’s customary in Russian culture, when you’re taking a long trip, to take a few minutes to sit in silence before you start the journey. Cosmonauts have a litany of similar rituals they follow before launch in Kazakhstan. They always visit the grave of Yuriy Gagarin in Baikonur in the days before they ship out. They also plant trees, to root themselves to the ground. They do not watch the Soyuz rocket being rolled out onto the launchpad. They never launch on October 24, because two accidents took place on that date.

Why did we create all this ritual-based infrastructure? First, for the individual, it’s important to stop at boundaries, and to process what happened before, leave it in the past, and then think about what comes next.  And what helps with processing is the ability to do it collectively. When humans physically come together to perform a ritual, we experience something known as collective effervescence, which is an energy that only occurs when a group of people stops tending to mundane, everyday tasks, to celebrate or mourn these boundaries. 

Humans coming together during times of transition is so important that when people were prevented from gathering during COVID, they came up with their own ways to circumvent this. In Italy, one of the countries hit the hardest during the first wave of COVID, people started singing together from their balconies. In New York, people stopped at 7 every night to bang pots to appreciate front-line workers.  In China, during lockdowns, people started screaming. 

One of the most tragic outcomes of COVID was people banned by public health measures in participating in collective mourning ceremonies for people they lost. In one of the most haunting pictures for me from the bleakness of the past two years, a nurse fully-dressed in protective gear helps a COVID patient connect with their loved ones over Zoom via iPad.

We’ve only just started trying to measure and understand this, but it appears that early results are showing even the importance of ritual as measured on a tangible, scientific scale. Leaving a job is also a transition of change, of parting, and one that needs to be processed both as an individual, and as a group.  In the pre-COVID days, this was all mostly fine. If you weren’t working remotely, leaving a job was a very physical ritual with a finite end: the email to your boss to meet and give notice. The meeting with HR. Physically telling all your coworkers. The goodbye lunch, the goodbye drinks, then finally leaving the workplace with all of the things you collected over the years - cards, old XML manuals you haven’t opened  in years, plants, pictures. And finally handing over your laptop and badge and walking out of the building, free as a bird. 

This ritual has been performed millions of times, and yet, usually, we don’t examine its importance at all. The closest I’ve seen is this study from the 1980s, called, very morosely, “Functions of Parting Ceremonies in Dying Organizations”.  The study was performed in 1982-1983, during the height of a recession, in Michigan, in hospitals, retail stores, car factories, and a research organization. 

I wish I could quote the entire paper, it’s that good and that relevant to what’s happening in tech today. 

It’s clear that people need parting ceremonies in the context of work, and when they don’t happen, they make them happen on their own, and in very predictable patterns. 

In the new COVID/remote work-era, particularly over the last couple months of extremely cruel tech industry layoffs, all of our physical group parting rituals are gone, lost in the digital ether. Nothing has proven this as much as the very painful and very public Twitter layoffs last week.

For the job I left before this one, also during covid, the extent of the ritual was a quick 10-minute chat with HR, followed by the arrival of an envelope where I was to send my laptop. At Tumblr,  this ritual was a steady stream of Slack DMs over my last two weeks in exchanges of contact information and a group pre-scheduled 30-minute Zooms where everyone tried very hard not to cry.  On my last day, after shaping the terroir of my work and being shaped by it for almost two years, I left a final message on Slack, turned off my laptop, took a minute beat, and left my desk - it was time to pick my kids up from school and daycare.

What happened to me was not unusual, and insanely trivial compared how people have been told to leave jobs over the past two years in layoffs across the tech industry. 

When you fire someone, you not only cut off their livelihood and dignity and affect a crisis situation in their life,  you also add a second layer of cruelty in that you deprive them of the importance of ritual. From the sidelines, I have watched pretty incredible things happen, such as 900 employees being collectively fired over Zoom. 

The recent round of Twitter layoffs has been the worst, the most public, and the most heartbreaking as employees, left swirling in the chaos of the Musk acquisition, tried to make sense of what was happening. On Thursday, there was an email letting employees know that the next day, Friday morning, they would receive either an email to their work address if they were spared, or to their personal address if they were being cut. 

Ahead of this chaos and left to their own devices, Twitter employees started creating their own rituals in Slack. Employees first flooded Twitter’s Slack with blue heart emojis. 

They then started posting the salute emoji as a sign of respect and farewell, which is how I first found out that layoffs were actually happening: a single salute emoji on my timeline.  

And finally, once all the emotion had been spent, they started finding out, about 12 hours before the email was supposed to go out, that they were fired because they were locked out of their Gmail accounts, their Google Cloud Access, and Slack itself. 

The other thing about rituals is that they are meant to act as a point of severance between your last world and the next one and leave you clean and whole to proceed. But because our bosses live in our phones, and because we are constantly connected to everyone all the time, we don’t get that anymore, either. 

For me, as I was on my way out the door, full of Feelings, I got a Slack notification about a service alert on my phone. My work email was still not disconnected, and I had forgotten to turn off Slack on mobile and I wasn’t disconnected yet.  Ten minutes later, a friend from work texted me. I left by choice, but when I did, I didn’t really leave. I was still extremely connected to friends by text, in alumni Slacks, on Twitter, and a million other different ways which has been wonderful and I wouldn’t want it differently. But none of this serves the ritual.

For Twitter employees, it has been even worse: many reported still getting alerts from PagerDuty for web services they could no longer be responsible for, and in the ultimate you-have-to-laugh-or-else-you’ll-cry twist, many are now getting emails asking for them to come back and maintain the services that they were fired from because, just maybe, firing half the company over the course of a week means you lose tribal knowledge.  

The tragedy and the irony of the world we have built is this: we have built a world made for connection, for immediate contact, for being able to reach out and achieve collective effervescence in some proxy way. But, as a result, we have no way to collectively grieve and we have no way to linearly separate and move on.

And, this brave, new world, even, for all the immensely numerous positives that remote work brings,  this kind of way of working does not serve the art of the goodbye. And this is a problem. Because, even though we are gone, we are never truly gone, absolved, allowed to move on past the transition point, into the next thing. And even though we are connected, we are not truly bonded. We are simulatenously plugged in but unresolved, floating with our emojis and “let’s keep in touch” messages, which vanish into the ether as soon as our Slack credentials unceremoniously expire and we hit the login wall. 

Quick programming notes: I wrapped up my previous gig a few weeks ago and I’m psyched to be starting as Sr. Machine Learning Engineer at Duo. As always, Normcore updates continue to remain sporadic and highly dependent on my ability to complete an entire eight-hour sleep cycle. 

And, another very exciting announcement: NormConf is happening online in December! What started out as a joke tweet has now become a legitimate conference on normcore topics in data and machine learning. Save the date and if you’re interested in sponsoring, shoot me an email!

This March, or maybe it was five lifetimes ago, as part of his on-again-off-again bid to burn the internet to the ground, Elon Musk did a poll to see if the “Twitter algorithm” should be open source.  

This, like any of Musk’s bizarre and impulsive tweets launched in between lawsuits, meeting the Pope, and being “house fluid at the moment”,  unleashed a wide variety of opinions from everyone, including Jack, the former CEO of Twitter, who responded that people should have a choice of which algorithm to use.  But leaving aside the semantics of what “open sourcing” and algoritmic choice, the larger question remained: what actually was The Algorithm?  

The OG Algorithm

On Twitter, I was curious as to what the common conception of algorithms was, so I  asked and got a range of different, interesting answers that generally aggregated to “an algorithm is a set of instructions, like a recipe, for a program.”

Algorithms seem to be very new because we think of them as inherently related to computers. But, the concept has been with us as long as humans have needed to narrowly navigate extremely complex multi-dimensional space that can have an infinite multitude of outcomes.  In other words, humans have been creating algorithms for how to live for thousands of years.

Here is an algorithm on figuring out how our time on earth should be allocated and why: 

To every thing there is a season, and a time to every purpose under the heaven:

A time to be born, and a time to die; a time to plant, a time to reap that which is planted;

Here is an algorithm from the witches in Macbeth about how to prepare effective hallucinogenic potables: 

Double, double toil and trouble;
Fire burn and caldron bubble.

Filet of a fenny snake,
In the caldron boil and bake;

Eye of newt and toe of frog,
Wool of bat and tongue of dog,

Adder's fork and blind-worm's sting,
Lizard's leg and howlet's wing,

For a charm of powerful trouble,
Like a hell-broth boil and bubble.

Double, double toil and trouble;
Fire burn and caldron bubble.

Cool it with a baboon's blood,
Then the charm is firm and good.

And here is a very long, elegant algorithm for how to build human-centric buildings and cities.

We weave an extremely tangled web

When we talk about “The Algorithm” today, though, it’s any program that shapes and influences the sea of content across the different places we traverse online. This Algorithm seems to be entirely out of our control and impossible to penetrate.

In order to start understanding it ,there are a couple of important things to keep in mind. What we as practitioners implicitly mean when we say “The Algorithm” is not one single algorithm, but rather:

1. Code logic that includes:    

   1. Machine Learning Algorithms

   2. Simple if/then statements / hardcoded heuristics

   3. Blended together

2. Business logic that is either implicitly or explicitly stated:

   1. The results of meetings

   2. Results of memos, discussions, and RFCs

   3. External drivers like changes in the market, support tickets, etc. 

   4. Organizational structures

All of this results in a system that’s made up of two parts that both equally influence the algorithm: One that is machine-generated, and one that is people (business logic). So, the system is never any one single thing, but a living organism made up of many, many moving complex parts. Every single ecommerce or content-based feed we consume online is a world unto itself that requires an enormous amount of technical and business context to untangle. 

For example, here is how you might hypothetically build the algorithm that populates a “Home” feed on Flutter, the premier social network for where things with wings gather. 

But, let’s start at the beginning.

The Algorithm’s Heart: Code Logic

An algorithm is born when a computer scientist and a product manager love each other very much, have a product vision, and decide that they want to drive either click through rate or revenue. 

The PM, the computer scientist, and a team of engineers (called something catchy like BirdFeed) goes off to create a feed. The BirdFeed team sees that, on the initial Flutter home timeline, birds are shown purely content from accounts that they follow.

But, maybe this is not in line with engagement metrics they need for their business. As a result, BirdFeed introduces an algorithmic timeline. (By the way, every timeline is algorithmic - reverse-chronological is also an algorithmic design choice.)

As this very, very excellent post on Twitter’s timeline outlines, 

This explanation makes sense from a UX standpoint, and an algorithmic feed certainly gives Twitter a lot more freedom to experiment with the product.

The real motivation, however, is because using an algorithmic feed is incentivized by Twitter’s current ad-driven business model. More relevant content in your feed ⇒ higher engagement ⇒ more ad revenue. This is a classic social network strategy that’s proven to work.

At this stage, we face a fundamental question: how do you shuffle the existing timeline so that it includes content people are more likely to find interesting and engage with?

Generally, there are several ways to tackle this problem, and they involve looking at either a user’s past activity and comparing it to the activity of similar users (collaborative filtering), or looking at a user’s past activity and comparing the items the user interacted with to similar items in the catalog (content-based filtering).  

Once you collect a user’s log history and build a model that generates a potential catalog of candidate Flits to recommend to them based on past user activity. But let’s say you come up with 10000 Flits to recommend? You can’t show them all in any given timeline, so what do you do? 

You rank your candidate Flits, then filter them against a set of business logic, then rank again until you get to a point where you have only a handful of Flits that you’re likely to insert into the timeline.  

This step is the heart of the algorithm and it often involves building multi-step machine learning models that will ultimately generate a small set of candidates.   Each company has a system that has developed using a variation of these steps and they all look strikingly similar.

Intermingled with the machine learning code are decisions that come from business logic: is there a list of bad words in Flits that we need to filter out of all potential recommendations? Maybe we don’t want to show Flits from Hummingbirds at all? Maybe we only want 9 Flits, not 10, for some arcane business reason like maybe the CEO likes the number 9. These are all hardcoded in the ML flow.

These machine learning systems then grow so involved that there are meta-systems being developed to manage them. Here is a diagram of what’s involved in MLOps, the field around operationalizing these algorithms. I see this diagram in my nightmares sometimes. 

The Algorithm’s Soul: Business Logic

The model itself is usually what we think about when we think about the algorithm. But, the second part of the system is just as complex and includes a number of participants: product managers, project managers, advertising teams, content safety teams, and many, many more, in a back and forth about what The Feed should look like. 

Because, then the question becomes, now that we have some candidate Flits, how do we insert those Flits amongst the chronological Flits that birds are already receiving? Should your feed be entirely chronological? 50% chronological and 50% algorithmic? Mostly algorithmic? What if you have more than one algorithm, one that shows you the most relevant Flits that other birds have liked, and one that shows the top flits from trending Bird Topics on Flutter? How do you combine those two sources? What if you have too many Flits in n seconds from the same Bird? How do you blend in diversity with Flits from Butterflies and Mosquitoes? What if you have Bad Bird Actors that are constantly either spamming or sending lewd pictures of their cloacas to the Timeline? How do you downrank them? Should you? 

And now, finally, you get to the question of injecting ads, the entire reason Flutter continues to work. How often can you insert them without user fatigue while still maximizing revenue? What content can they be next to? Can you have an ad for worms shown to a Flutter user who is profiled as a Butterfly, and therefore does not eat worms? How should you promote ads across timelines? This in itself is an entire world.

These gray-area business logic questions where recommender systems become just as much of an art as a science, continuing to generate potential content for recommendation, blending it, filtering out negative or prohibited content on the platform to get to a “good” feed of engageable content. 

They involve not only the code itself, but also hundreds of hours of conversation, back-and-forth, the results of A/B tests for millions of users in re-ranking the feed, changing it, constantly shaping it to the best version of what the company thinks it should be. 

Aside from all these decision-making conversations, you have hundreds of organizational changes happening that quietly shape the feed, but that are never documented anywhere. Maybe the engineering team logging user data is accidentally not logging Bird Geolocation anymore. Now your model is inaccurate, but you don’t know about it yet. Maybe the mobile team is running an A/B test that directly conflicts with your feed changes. Maybe you had a reorg and the last CMO loved The Algorithm but the new one doesn’t. And vice versa. All of these changes implicitly silently change the feed on a day-by-day, minute-by-minute basis, but are not anywhere in the model’s documented code.

When the Black Box Opens

The scenario I’ve described is hypothetical, only in the sense that there is sadly not (yet) a social network just for birds and insects.   But these kinds of systems get built and decisions get made every day at companies all across the board. Here’s a recent one from Pinterest, for example, and here is Netflix, the king of recommender system’s whole website on their methodologies.  Here is Facebook on their re-algorithmized main feed. 

But, to the end user, there are several fundamental problems. First, you shouldn’t need to be able to reverse-engineer academic papers to understand your content feed. And second, with so much implicit business logic injected into the system, how can anyone possibly hold the whole feed’s context in their head? How do you know why a single Flit was shown to you? How about a timeline? Many people in the company, even sometimes the engineers that build the systems, don’t. 

So even if Twitter were to open-source the algorithm, and you had a background in computer science,  all you’d have to look at would be a bunch of code that looks much like the psuedocode generated here, with a number of other, related systems feeding into this timeline code. 

You would have zero context about both the code and the business logic used to generate, and, most importantly, you wouldn’t have access to the heart of what makes this system tick: user log data and the logic for how it’s stored, cleaned, and processed in order to be used in both the recommendation algorithms and the construction of the feed itself. 

Opening the code of the algorithm might be a good place to start, but it’s not unlike making an architectural excavation site open to the public: without the guidance of an archaeologist to explain it, it’s just a set of pottery shards.

There is not much we as end users can do with “The Algorithm” as-is.

Where do we algo from here? 

The good news is that as more people are expressing intense interest in looking at these internals, more companies are also thinking about how to talk about The Algorithm in public.

First, product management teams are trying to make their algorithms transparent. Or, as transparent as they can without exposing proprietary internal business logic. The first time I saw this happen was when Instagram wrote a primer about its recommendation systems aimed at the general public in 2021.  They wrote, 

It's hard to trust what you don't understand. We want to do a better job of explaining how Instagram works. There are a lot of misconceptions out there, and we recognize that we can do more to help people understand what we do.

Of course, it’s extremely high-level and addresses only some of the elements I touched on in this post, but it’s truly the first time I’ve seen the nature of recommendations explained to a non-technical audience.

Instagram had a second go at this when they made an enormous change in their algorithm a few weeks ago to prioritize video content so they, like every single social platform on the planet right now, could compete with TikTok. People were pretty angry at the new feed, and Adam Mosseri took to Twitter to try and explain it: 

(Instagram almost immediately rolled back the changes he discussed. )

Twitter also has recently explained some changes it makes in its recommendation algorithms,  For instance, recently, it became overrun with recommendations of tweets from an account called “Shirts that Go Hard”, and, in response, Twitter paused several A/B tests and personalization changes: 

These explanations don’t and can’t give all the product context: much of it is proprietary. Some of it will make the company look bad. And some of it, not everyone ven knows. But it’s a start.

Aside from product management-driven explanations, something that’s becoming more prevalent in technical implementations of these systems is more annotated surface areas for recommendation systems. What this means is that the algorithm, as it’s presented in the feed, is labeled or annotated with some level of explanation. The most interesting place I’ve seen it is Meta’s new BlenderBot (experimental chatbots where the entire internet can enter freeform text always being a great idea). 

When you enter text, it allows you to mark the immediate response as positive or negative, presumably as user feedback into the model that’s explicit (rather than the implicit, log-based feedback that is usually generated.) 

It then also has a short explainer on why the AI responded with the message it did, which pulls up a dialog box around user input. In my case, the AI was 100% wrong about the context of the response, but at least now I (kind of) know why. I can also even go back to the model and try to reverse-engineer exactly what happened.

This kind of UI takes an enormous amount of work to implement and loop back into the model’s back-end as a feedback loop, and also adds load time to any given site, so I appreciate the effort and also appreciate that for now it will likely remain mostly a prototype. But , it remains a good example of what an annotated system like this could potentially look like.

We are still in the steam-powered days of machine learning and all of these things, the top-down executive explanations and the engineering efforts at transparency, show just how hard it is to A) Define an algorithm. B) Capture its surface area C) Understand its scope and impact and D) Try to change it. 

It doesn’t mean we shouldn’t try. But it does mean we should be more deliberate than just trying to “Open Source The Algorithm.”

Building and hardening these systems was our work for the last 15 years. Continuing to build them so they don’t break AND figuring out how to open them and explain them will be the next 15. 

Some people say that Duo, the light-green mascot of Duolingo, an app that helps you learn different languages by prodding you to practice, lives in a tree, much like a real spectacled owl. Then, there are those that whisper that Duo lives among us and watches us, wearing a backpack carrying all of our sins.

I personally believe that Duo mostly exists as a collection of bytes that originates in the shadowlands beyond the Gates of Hell. The distributed spirit of Duo peacefully roams the underworld and grows stronger by feeding on energy of lost souls and cloud-based microservices until every day at 8 PM Eastern Time, driven by an incantation from Hades himself and a multi-armed bandit push notification strategy promoted to production in Scala, the devil’s own language, Duo reconstitutes into its full avian form and appears on my phone to guilt me into learning Italian. 

Why am I doing this to myself? 

After a year of focusing entirely on technical growth, I realized that I needed The Arts Back in my Life.  One of the realizations that came after watching a bunch of Sorrentino movies was that I wanted to pick up Italian again.   Once upon a time  in high school, I took Latin for four years, and then decided that I made a terrible mistake learning a language that has 30+ potential ways you could mess up the ending of a noun, not to mention not a single person alive will be able to correct your pronunciation. So, I decided to do a linguistic pivot and started teaching myself Italian, motivated mostly by my extreme desire to understand Celentano. 

Fast forward to this February, when I decided to pick up Italian again so I could learn a language that’s not mostly made up of JVM stacktrace errors. 

Pushing me Into The Worst Decision I’ve ever Made in My Life

Starting Duolingo, something that more and more people have done over the past couple years if this is any indicator, is the worst decision I’ve ever made in my life.  No matter what has happened in my day or what my energy level is, the Owl doesn’t care. The Owl is always ready.

Duolingo’s notifications encouraging you to use the app to practice are so timely, on-point, so repetitive, and so passive-aggressive, that they’ve become a meme in the industry over the past couple years. 

The notifications became really fascinating to me, because they are one of the few push notifications that I actually have enabled.  So I got to wondering, why are they so terrifying, and why are they so good, to the point that they’ve become a meme that even the company winkingly acknowledges?

Push it good

The answer lies in a combination of how they’re implemented technologically plus how the technological implementation ties into the psychology of the intent of the app.   First, let’s talk about how push notifications work. A long time ago in a galaxy far away, ARPANet started. Ok, maybe too far back. Let’s fast-forward a bit to 2003, when Research in Motion, the company behind the ill-fated Blackberry phone, started sending notifications that would pop up on the device’s screen when people received an email. Steve Jobs took notice and Apple made the Apple Push Notification Service available in 2009, changing the mobile landscape. 

The first push notifications were extremely basic, but it soon became clear that companies could engage consumers with notifications based on usage events, such as shopping cart abandonment in e-commerce, basic user inactivity over a given time period, and once rich push notifications became an option, based on geotargeting, or user browsing behavior. 

Then, companies started varying the copy, or the content of the push notification themselves, based on a number of heuristics like the age of the user on the platform (new user? returning?), the time of day, and even the language. 

Today, many push notifications are based on surfacing personalized content so that users are more likely to engage with them. The push notifications that are content-dependent are often backed by machine learning approaches that figure out the optimal time and the optimal push to send. This is especially important so you don’t irritate users who have push notifications enabled.  What many companies end up doing as a result, is running experiments on the types of push notifications to send to different groups, at different times. 

Push it good, Duo

Duolingo has been running A/B tests for a long time. They’ve gotten really good at picking a group of correct messages. But previously, they were sending them at random. 

Duolingo’s push notifications, at least as of this paper a few years ago, are based on a complex set of heuristics that involve modifying multi-armed bandits.  Multi-armed bandits are a form of reinforcement learning, a branch of machine learning, given several different paths, tries to optimize between exploring those diverse paths and finding the paths that offer the highest reward and just traversing down those specific paths, while controlling for any additional business logic that might be important. 

What does this mean in terms of these push notifications? Sending a push notification and having the user open the app and complete the lesson is the ultimate goal (aka reward in this case), and here are the metrics Duolingo tracks on it: 

Daily Active Users (DAUs): The number of distinct users in the bucket that opened and interacted with the Duolingo application on their device each day.

Total Lessons Completed: The total number of language lessons that users in the bucket completed.

DX Recurring Retention: The percentage of users who opened the Duolingo app on a given day who also did so X days later. This is further divided into new users (those whose accounts were created less than 48 hours before their first notification in the experiment) and existing users

In the general case of multi-armed bandits, each arm (aka path to a completed lesson) is assumed to be available and the model alternates between optimizing on these paths and sending the one that works the best over and over again. This doesn’t work for Duolingo because language learners want to see a variety of push notifications in order to re-engage them.  

For example, I get this one a lot and it makes me feel super guilty every time. At some point, I’m just going to ignore it instead of clicking through.  

It would make sense for the bandit to optimize against sending it to me and penalize by frequency.  Also, not every notification makes sense for every user:  

For example, some templates in  reference a user’s “streak wager,” a game mechanic applicable to only a fraction of users on any given day. As such, each template has defined eligibility criteria to prevent the template from being sent to an inappropriate user.

What Duolingo did then, is a bunch of work to better use historical data for push notification opens to figure out the best time and message to send with these notifications. 

Why are the pushes so good? 

So, it’s true that this new policy increased engagement rate for push notifications. In fact, the paper notes that they experienced an increase of .5% in daily active users as a result of the experiment (and more importantly new user retention, the lifeblood of any app, increased by over 2%.)  That’s nothing to sneeze at: 

However, what’s even more interesting: the authors note in the paper is that, even before implementing fancy bandits, going into the A/B test,  Duolingo’s notifications worked well already: 

The control group’s templates would be selected via the legacy algorithm (i.e., using a uniform random distribution over the eligible templates each round). Since the pools of templates and other aspects of the notification system had already been optimized through years of A/B tests, this provided a strong baseline to compare against.

This made me come back to the idea that something that I’ve talked about before is that tech can be very good and effective if you combine the goals of the tech with human instincts that are already built in. For example, one recommender system that many people liked in this Normcore discussion thread was Spotify’s, because the company’s goal of getting you to stay on the platform longer and finding the music valuable is very much in line with the listener’s goals of finding new and relevant music. 

The same is the case for Duolingo. Push notifications are meant to get me to engage with the app for the sake of the company’s bottom line. Everything in this paper tells us this is the case - multi-armed bandits require an enormous amount of infrastructure, as does collecting millions and millions of  lines of log events, cleaning those events, making them available for machine learning, training the algorithms, then preparing the algorithms for A/B tests, running the A/B tests, interpreting the results, and promoting them to production.

Just look how immense and involved this system to build these bandits is in comparison to the size of lift. Check out this architecture, which involves at the very minimum Kinesis (a Kafka-like service), Spark, Elasticbeanstalk, Cloudwatch, and a zoo of other AWS services to log data, train the model, run the model, and send the actual notification. There is no way the company would invest money in this infrastructure if building it also didn’t produce new DAUs, new completed lessons, and new revenue. 

But, what’s important is that from a user perspective, the push notifications for Duolingo work with you in a way most push notifications don’t. Most push notifications make you anxious and their goal is a boost in short-term engagement. But, if you as a language learner don’t practice every day, you forget absolutely everything, which I learned when I took a month off Italian and on day 31, when I logged on again, and immediately forgot the difference between la pasta and il pasto.

So, everything in Duolingo optimizes for you coming back, but also for you learning, and here, the push notification is here to help you create a habit that not only boosts their DAUs, but also your connection to your language. And this is the kind of good synergy that works really well when user and app systemic incentives are aligned, and it’s not clear how much the machine learning helps here, versus how much the underlying goal that’s already there is a strong foundation.

If you don’t practice every day, you will never understand Celentano. So in that sense, the goals of me and Duolingo are both very interested in having Duo continue to bother me, which is what makes this dive into how these push notifications work so interesting.  Speaking of which, it’s 9:28 PM again, which means I need to wrap up this newsletter and ritornare a Duo before it comes for me.

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, and a little fun. It goes out whenever I can get around to it. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling two small kids, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Hello friends old and new, it’s been a while. Last year, I stopped writing Normcore Tech, a weekly newsletter about tech and the humanity in tech, because I was going up a learning curve at work and needed all my meager mental energy on matrix operations. I’ve recently had some thoughts that I need to organize longform, which means it’s time to bring the gang back together for at least one more Normcore. 

If you would like to help people in Ukraine, please visit Ukraine Trust Chain. I know the organizers personally.  If you would like to support people in Russia, and you work for a tech company, you might ask how your company is supporting its current employees there or handling hiring people who want to leave the country.

Every day these days for the past three weeks, I’ve woken up to a list of Telegram notifications a mile long. When it’s 6 am on the East Coast here in the States, it is already mid-day in Kyiv and Moscow. The horror that is this war has already been generating notifications, news alerts, and media of every kind for the past six hours.

Telegram, as I’ve written about before, is a social network started in Russia by uber troll Pavel Durov

In 2018, the government tried to block access to Telegram, Russia’s favorite messaging platform, run by Pavel Durov. He had previously created VKontakte, and got in trouble with politicians who wanted him to take down pages of opposition candidates. He declined to do so. In 2014, he refused to hand over data of Ukrainian protesters, and refused to block the page of Alexei Navalny, the figurehead of the Russian opposition. As a result, he was ousted as the CEO of the company, and left Russia. He is currently a citizen of Saint Kitts and Nevis, a citizenship which he obtained through “donating $250,000 to the country's Sugar Industry Diversification Foundation, and secured US$300 million in cash within Swiss banks.” All of this is just the surface-level stuff from Wikipedia, by the way. 

Telegram became extremely popular in Russia, both for private chats and for subscribing to channels full of news, jokes, and much more. As an aside, Telegram is my personal favorite messaging service.

Russia and Belarus alike have tried to block access to Telegram more than once, but it’s still standing. 

Since you need to install the app through either the app store or by downloading it and sideloading it (outside of any app store) on your device, people were offering flash drives with it and passing them around. 

It seems that everybody in Belarus follows Nexta Live. During the first days after the country's presidential election, when the entire country went nearly completely offline, this channel on the Telegram messenger service was one of the only sources of information about an escalating political crisis. 

And it was one of the only services that continued to semi-work in Belarus, and ostensibly, one of the ways that the protesters who organized a mass peaceful protest organized.

Because of its enormous stability and popularity, as well as the number of Russian and Ukrainian speakers that already use the platform, Telegram, in Russian, is the only place to be these days if you want to understand what’s going on with the war. Both sides are here. 

Pro-Ukrainian news channels, including the aforementioned Nexta are here. Zelensky of course has a Telegram channel where he shares his daily addresses to Ukraine and Russia. Pro-Russian channels, official Russian outlets like TASS, Western media who have been kicked out of Russia, pundits, meme channels, they all have their own feeds. And, of course (?) Arnold is here now, with a very moving appeal to Russians. And, now even the NYT is here. 

In all of these, and especially in family chats, everyone is discussing the war.  And the number of Telegram subscribers is growing larger and larger (some estimates peg that 40 million people have joined since the start of the war), especially in light of Russia’s alarming crackdowns on literally every single Western social media platform, already a boost from the 70 million subscribers it gained last fall when Facebook was down. In fact, so many people over 50 have joined, urged on by their younger family members in light of growing censorship, that there are now memes about “Mom joining Telegram.” 

Putting aside my personal stake in all of this and putting on my professional recsys hat for a minute, what is most remarkable is that Telegram has almost no recommendation or discovery features. Everything is organized how you want it, and  every channel is found either through word of mouth, or reading about it on some news site or on Twitter, or in the shares from another Telegram channel, and it is entirely up to you to curate your Telegram experience.  This ends up being both a blessing and a curse. 

On the Ukrainian news channels, I read about children sleeping for weeks in bomb shelters, about babies being evacuated from Kharkiv, about 80-year old women climbing broken bridges on foot, about a colossal human tragedy being enacted one person at a time. The video that shook me the most was a steady view out the window of a second floor of an apartment building into a landscape full of other buildings and hills in the distance under a leaden late-winter sky. The caption read “I took this video at my friends’ apartment while her one-year-old was playing on the floor nearby” and for about ten seconds, there is nothing out of the ordinary, and then a rocket comes from the right side of the screen, right into a building about a mile away into the distance, and explodes, just like in a movie. Then everything is quiet once again. The combination of the normalcy of the rest of the video, combined with the horror of this thing just happening, of absolutely nothing stopping it, of the injustice of it, was to me the essential indicator of the cruelty of war. I imagined myself in that room, watching that rocket, with that toddler, playing loudly and joyfully and blissfully ignorant on the floor, and I had to put down the phone. 

But this is not enough for me. I then go onto Russian channels to read about the panic and fear of a society that is, as I predicted six years ago, closing to the Western world. I read a constant stream of companies leaving the country. I read about women who are panicked that they are losing access to sanitary products (produced mostly by Western companies), I watch videos of 70-year old women who have survived the worst years of empty shelves of the Soviet Union, now fighting for sugar in long queues.  I read that Russian prescriptions for antidepressants have increased by 25% year-over-year. I read morose memes that make fun of how little the ruble is worth, how little work matters now that Russia is closing, and also feel a dark sense of bleakness. 

Why do I do this to myself? I am, in theory, just creating more digital noise for myself. Noise which I have the luxury to turn off.

The ability to have room for leisure has always also been an upper-class pursuit. Just ask anyone at Downton Abbey. What did people do there all day? Breakfast, chat, read, take walks, and by then it was time for dinner on large, quiet estates.

Today, the true signal of privilege and choice means not only the ability to block out physical distractions, but digital ones, as well.

Aaron Swartz told us to not read the news, and I still strongly maintain that this is one my largest pieces of advice for operating on the internet. 

The news’s obsession with having a little bit of information on a wide variety of subjects means that it actually gets most of those subjects wrong. (One need only read the blatant errors reported in the corrections page to get some sense of the more thorough-going errors that must lie beneath them. And, indeed, anyone who has ever been in the news will tell you that the news always gets the story wrong.) Its obsession with the criminal and the deviant makes us less trusting people. Its obsession with the hurry of the day-to-day makes us less reflective thinkers. Its obsession with surfaces makes us shallow.

First, I cannot turn it off. It affects family and friends and the family and friends of almost everyone in my Russian-speaking community here in the States. Every Russian speaker I know personally is a walking mess. 

But second, I’m starting to develop a theory for why following the war as it happens in Ukraine and its aftereffects on Russia are so important to me, even in the face of my (already fragile after COVID, by the way remember COVID? haha!) state of online sanity. 

Lev Vygostky, writing over a hundred years ago, researched why children play. One of his theories was that play and imagination allow us to create situations that are in our control, in a world that, for children, doesn’t offer a lot of other ways to control it. In allowing children to play, we allow them to construct their own world which they can control and manage, and as a result, feel confident about.  

In the face of the overwhelming horror, my growing sense of smallness and uslenessness, and in addition to donating money, what I’ve realized is that curating the news, particularly in the absence of any algorithmic features (that I know of) on the behalf of Telegram offers me some semblance of feeling like I’m in control of the news I consume and pass on to others among my friend and family groups. 

Finally, what Telegram gives me is the feeling that life is still going on. Dina Rubina, a very famous Russian-language author who now lives in Israel, wrote a collection of short stories a few years back. I don’t remember the name of the collection, or even the short story that it comes from, but an anecdote from it really stuck out to me. In the story, she describes the siege of Leningrad (something, ironically, that Arnold Schwarzenegger also references in his video as the point of disillusionment in the Nazi regime by his father). Every Soviet apartment had a radio hooked up to government channels, and even during the absolute worst days of the siege, the radio was on, so the narrator’s grandmother simply kept it on all the time. She carried that habit over after the war, and the narrator of the story, a young girl, recalls how that sound of the radio on low is one of her favorite soothing memories of her grandmother.

“If the radio is on, it means that life is still winning,” she said, and this is how I feel about Telegram. If I can still reach friends in Russia directly on Telegram, if I can still see videos of Zelensky, alive, then life is, for the moment, still going on, and still makes some sense, even for a few minutes.

And so I continue to scroll the Feed and pick through the pieces. 

Goodbye on the Mersey, James Tissot, 1881

Hello friends!

As you might have guessed from my four-month absence, I can no longer continue writing Normcore on a regular basis.

A confluence of things has led to this.

First, I recently started a new position at work and am extremely (and very, very happily) busy onboarding (Tumblr is still awesome! If you don’t believe come, come sign up or dust off that alt account :) and I foresee work taking up a ton of my mental space into the near future.

Second, I think I’ve pretty much exhausted all the themes I can possibly write about: it’s important to examine things in a nuanced way instead of with kneejerk reactions and we have to work that much harder at it in today’s media environment, good things don’t scale, machine learning is very hard and made out of people, boring technology is good, the internet is very hard and made out of people, being a working mom is Very Hard, and companies always do the thing where they balance trying to do complex AI with making tons of money.

Third, my second baby, who was born around the same time as Normcore, decided he no longer feels like sleeping and has been waking up at 4:30 am for the past four months, which means by the time it gets to Normcore time, 8 at night, after a full day of work and two small kids, I have about as much energy as a wrung-out mop, if that mop also had to yell at a toddler not to eat toilet paper.

Fourth, it becomes hard, as Brian wrote when he closed BNet, to keep examining new internet phenomena under the same lens over and over again. Combined with the cyclical demands of the newsletter to publish, publish, publish, and the explosion of newsletters in the data space, I personally feel like I am drowning in newsletters, that are all covering similar spaces of takes about the same 5 companies built by the same 10k people over and over again. I don’t want to fall into that trap. I want to write when I have something to say. The newsletter revolution is wonderful for readers, but not always for hobby writers who do it in their free time.

But, trust me, I’m always (for better or worse, probably much, much worse) going to have something to say. I physically can’t not write. So, I’m going to keep Normcore open as a way of letting you know about any new stuff I’ve written, and potentially occasionally free Normcore posts. Paid subscriptions are still on pause and I’m going to look into cancelling them entirely. Feel free to unsubscribe entirely if you don’t want to be on this mailing list anymore.

I’m very sad about all of this this, and at the same time extremely relieved. It’s possible I’ll come back to Normcore at some point, maybe in this format, maybe in another - after all, the whole point of Normcore, as I wrote when I started it, is that there is nuance in what we do, and it doesn’t have to be all or nothing.

So for now, goodbye, but also not? In the meantime, here’s some stuff I’ve written recently:

• The ritual of the deploy

• The humble hash aggregate

• The ghosts in the data

• My favorite books of 2020

Thank you enormously for all your support, for buying my merch for your emails, your DMS, and your comments on stuff I got either right or wrong, especially during this last COVID year. Keep it Normcore. :)

It’s been a while since my last post, almost a month, as the dashboard shows. But it’s not like I haven’t been thinking about it.I’ve been meaning to write a good long post for the past month, actually.

One about how all of our legal systems are now partially controlled by if statements in software more than the actual legal systems themselves. It was going to be a great post that involved a Russian-speaking AI voice assistant, Zuck’s emails, Twitter blocking me access to my account for 24 hours, and a James Bond remix.

And it’s not like I haven’t been thinking about this post. I have, every day, and the longer I don’t write it, the longer and more clever and intricate it becomes in my head, and the more the list of links grows.

The problem is that, again, like with the thing that happened last March, I am back in the bardo. Just when we had established some kind of pandemic norm, my daughter’s school closed for weeks, I lost childcare (and still do not have it), and I’ve been simultaneously trying to ramp up on a bunch of stuff at work.

We are all, fortunately, fine and it’s all kind of sort of working out. But, there is less than zero time for me to do anything long for myself and sit for a while in luxury at a Google doc of my own. And length, as I’ve realized, is luxury. Space and time to think, for me, as I’ve written before, is the ultimate thing I value, and when I can scale the top of that Maslow pyramid of small children at home and work and a global pandemic and make it to my sentences, then I am truly complete.

Unfortunately I’m not there yet, but I hope to be soon. I’d love to say I’ll be back next month, but we’re on pandemic time, still, even as the vaccination count rises in the United States and good news becomes more and more frequent, and pandemic time, as everyone knows, works in several week increments.

Until then, I’m pausing payments for paid subscribers, which I should have done weeks ago, when I was washing applesauce out of my hair and it was clear that the Google doc wouldn’t make it out of the drafts folder.

Until then, stay healthy and well,

Vicki

Hey, your friendly neighborhood COVID CEO here, popping in with the latest news from my board room.  Almost five years ago, I wrote about how tech, as an industry, doesn’t work on the hard problems, but instead builds snack-delivery startups. I wondered,

Why is it that we are so focused on “fixing the way we snack?” (or the way we get fresh flowers delivered, or the way we do laundry?) Why can’t we instead pool our energy and resources and instead of building monthly subscription boxes, social media apps, optimizing ad networks, tackle the health monster? What would it take to begin fixing at least a little bit of this broken, ugly thing?

A few years later, I wrote about how, again, AI won’t save healthcare. 

Over the past few days, a couple of news items have come out that have made me realize that the answer is this: in the American healthcare industry, there’s no financial incentive to helping end users (patients), unless it also helps the institution taking care of them (hospitals, insurance companies, etc.)

The Developers

Every few weeks a news item comes out that makes me feel this truth viscerally again.  Last week took the cake. I read a story about Olivia, a web dev who’s currently on maternity leave and has been, on her own, working on fixing the Massachusetts  state vaccine sign-up process.  

After her mother-in-law had difficulty signing up for a Covid-19 vaccine, a Massachusetts woman created a website to make it easier for her -- and she made it easier for everyone.

Olivia Adams built a website that pulls in vaccination appointments from across the state, including government sites as well as ones operated by private businesses. She called it macovidvaccines.com.

The 28-year-old software developer from Arlington, Massachusetts, says she spent three weeks and about 40 hours building the website -- and she did it while on maternity leave caring for her 2-month-old son, she told CNN's Alisyn Camerota on Monday.

What she did was build a website, hosted on AWS, that scrapes appointment data from a number of sources, including maimmunizations.org, pharmacies, and hospitals, none of which is centralized in any sort of way, and offers an aggregated list of appointments that people can sign up for immediately. 

Adams has a script that runs every five minutes across about 20 different vaccine sites, she wrote in an email.

Right now, she is constantly fixing the site due to minor upstream site changes that break her scraper, aka the program that looks at the website and identifies which HTML data to surface on her website. She is also currently fielding interviews from mainstream media sources, raising money for hosting costs (donate here on her GoFundMe) and taking calls with Massachusset’s “Coronavirus Command Center.”  (Don’t forget she has a newborn.)

I am hoping that the meeting was just Olivia asking “why the hell weren’t you guys doing this to begin with,” because I cannot imagine a scenario where a single woman who just gave birth has to do the job that an entire state government was tasked with. 

Just when I thought I was already enraged (although to be fair, I do spend a good deal of my free time on Twitter so rage is my morning baseline emotion), I started reading more stories of one-off people so frustrated with the vaccine registration system that they took matters into their own hands. For example, there is this guy in New York: 

Huge Ma, a 31-year-old software engineer for Airbnb, was stunned when he tried to make a coronavirus vaccine appointment for his mother in early January and saw that there were dozens of websites to check, each with its own sign-up protocol. The city and state appointment systems were completely distinct.

“There has to be a better way,” he said he remembered thinking.

So, he developed one. In less than two weeks, he launched TurboVax, a free website that compiles availability from the three main city and state New York vaccine systems and sends the information in real time to Twitter. It cost Mr. Ma less than $50 to build, yet it offers an easier way to spot appointments than the city and state’s official systems do.

And then, there is the vaccination Facebook group in my native Pennsylvania that a local doctor created, out of the same frustration. Here, members manually check sites and lets members know when appointments are available 

Thousands of Pennsylvanians have joined a Facebook group called PA COVID Vaccine Match Maker, where members share advice on how to get phase 1A individuals vaccinated.

Exton Internal Medicine Doctor Christine Meyer created the Facebook group and said her office has received thousands of emails and hundreds of phone calls from people pleading to get the vaccine.

Meyer said the members help one another find vaccine appointments and share information such as when pharmacies have appointment openings.

"It's a crowdsourcing zone where people are just posting their tips and tricks for success," said Meyer.

Finally, out in California (well, more precisely Japan, because that’s where he lives,) Patrick McKenzie, better known to the tech internet at patio11, his HackerNews username, has been volunteering and creating, with a group of others,  https://www.vaccinateca.com/, which basically has hundreds of volunteers that call different pharmacies and healthcare providers and ask them who has the vaccine, 

Our professionals call medical professionals at hundreds of potential vaccination sites daily, asking them if they have the vaccine and if so to whom they will administer it to and how to get an appointment. We write down what they tell us, and publish it to this site. 

How is any of this ok? Why do we have these people volunteering their time so that people can figure out where they can get vaccines? The more of these stories I read, the more grateful I am that there are good people in this world, and the angrier I get about the situation. 

But I shouldn’t be angry. In fact, I should have zero surprise that this is working out this way.  Because, really the problem is two-fold. Or at least, this is my working theory (remember, I am a humble MLE whose specialty is fixing typos in YAML files, not fixing American healthcare. ) 

One is that we do not have a strong national medical (or anything, really) infrastructure. We saw this when we were trying to ramp up manufacturing for face masks and ventilators. Remember ventilators? We are still short, and still not really doing anything about it. It’s just fortunate that early on we thought we would need them, but really we were mistaken about the nature of COVID. 

The second subset of this problem is that, not only do we not build things well at the national level, the federal government and the state governments don’t coordinate well. And, all the last-mile work is left to the states,

And critically, public health experts say, federal officials have left many of the details of the final stage of the vaccine distribution process, such as scheduling and staffing, to overstretched local health officials and hospitals.

“We’ve taken the people with the least amount of resources and capacity and asked them to do the hardest part of the vaccination — which is actually getting the vaccines administered into people’s arms,” said Dr. Ashish Jha, the dean of Brown University’s School of Public Health.

This is already a really complex process with all the features of a project that has little chance of success in the beginning: high-pressure, high-visibility, thousands of actors that need to move in tandem, a highly-volatile vaccine that needs to be stored at precise temperatures and given in two doses, a country that runs mostly on a model of small local providers and commercial pharmacies like CVS (which have also not been doing so great, actually), and a delivery roll-out process that started over the major winter holidays during a pandemic that’s already seen record package delivery volumes.

Phew. 

And then came the consultants. 

The Consultants

I should be careful about what I mean here, because I used to be a consultant (sorry about all the Excel.) . What I mean here specifically is federal government contracts. 

 You may have heard that Deloitte was involved working with the CDC on the national vaccine strategy. 

Her frustration is echoed by millions of Americans who have struggled to get vaccines through various chaotic systems. But unlike others in some states, she wasn’t encountering these problems with a third-party consumer service like Eventbrite, or even through an antiquated government system. She was on the US Centers for Disease Control and Prevention’s brand-new, $44 million website called VAMS—the Vaccine Administration Management System, built by the consulting firm Deloitte.

….

[E]arly in the pandemic, the CDC outlined the need for a system that could handle a mass vaccination campaign, once shots were approved. It wanted to streamline the whole thing: sign-ups, scheduling, inventory tracking, and immunization reporting. 

In May, it gave the task to consulting company Deloitte, a huge federal contractor, with a $16 million no-bid contract to manage “covid-19 vaccine distribution and administration tracking.” In December, Deloitte snagged another $28 million for the project, again with no competition. The contract specifies that the award could go as high as $32 million, leaving taxpayers with a bill between $44 and $48 million.

Why was Deloitte awarded the project on a no-bid basis? The contracts claim the company was the only “responsible source” to build the tool.

And, who else is involved in all of this? Our good friends at Palantir.  (If you’re not familiar with Palantir, their Wikipedia controversy page is a nice place to start.) 

They’re involved further upstream, though, with the actual distribution that decides which state gets what. 

The vaccine prioritization formulas fall roughly into three tiers: federal, state and local. At the top level, Operation Warp Speed — a multiagency federal effort, created by the Trump administration — has managed nationwide vaccine distribution through Tiberius, an online portal developed by Palantir, the data-mining giant. The Biden administration, which has retired the program’s name, has taken over and is continuing the effort.

States began warning about Tiberius’s drawbacks last fall. In interim vaccine plans filed with the C.D.C., some state health administrators complained that the platform seemed overly cumbersome and that the algorithm’s week-by-week allotments would make it difficult to plan monthslong vaccination campaigns.

This is just the public information I’m reading. I’m sure there are hundreds of other similar companies involved in the vaccine rollout, only to be bested literally by a single web developer with 40-50 hours on her hands. 

Why do these companies, which are so bad at the work of actually coordinating vaccines, get this work? It’s as Dan says partially, in this thread:

Because their job is not to do this work.  Their job is to get federal contracts. Or rather, that’s what the market incentivizes them to spend time on, because the more federal contracts you can get, the more you’ll be known as someone who can get and perform federal contracts, which means you’ll be asked to bid on more things.

Here’s how the process works. The government puts out an RFP, a request for proposal. The RFPs are often hundreds of pages and have lots of arcane rules you need to follow, like, “please provide 50 references from the last 100 projects you’ve done that can show that you know how to write APIs.” And on and on for hundreds of pages. A finished RFP response can take tens of people and months to put together. This is not a secret by the way, the federal government openly advertises that it works this way, and most states do this, as well.  There are thousands of RFPs for all kinds of different services outstanding on any given day. 

Here’s an example from my beloved Pennsylvania, for ATM machines on the Turnpike (our state’s toll road system.) Here is the actual RFP, and there is also a separate PDF of attachments. If you were brave enough to click through, you can see that there are pages of very specific things they want (“Provide a description of the proposed approach/methodology that you will follow in delivering, installing and operating ATMs at PTC locations as well as video advertisement procedures, and your experience with entering into a revenue sharing agreement. Include in this section the deliverables and reports to be provided, the project controls that will be used, and the tasks that will be performed”), and that if you answer all of the questions correctly in order to win the bid, it will take you a very long time. 

If you understand how to operate the RFP system and play by the rules, you have an enormous advantage over companies that have never done this specific kind of paperwork and don’t have the staff to devote to this. Palantir and Deloitte have done hundreds of these before, so they easily slam dunked these proposals, and as a result they’re involved in this COVID vaccine work now. 

Whether they’re actually delivering on it remains to be seen, but right now it’s all under a big question mark and getting mixed into the other things that are causing us not to vaccinate as many people as we should be. 

But, the real tragedy here is this:  You know who’s not good at getting the attention of federal and state agencies working on the vaccine rollouts? People who can actually make these websites and cut through this red tape. There is a very good phrase to explain this phenomenon, “No one’s ever gotten fired for buying Oracle (/IBM/Microsoft).” 

No one’s ever gotten fired for hiring Big 4, even if it means vaccines are on the line. 

And so, that’s where we are now. No federal-state coordination, a brand-new process coordinated in a country that prides itself on decentralization, and a process where speed is of the  essence. And, on the other side of all of this, people who actually need the vaccine, waiting for the government machine to start churning faster. And in the middle,  patio11 and Olivia, working as quickly as they can, alone.   

I still believe we’re almost out of this thing. We are distributing vaccines faster than ever and we’re getting better every week. It’s still really bleak right now. But, we are just around the curve from we start to see the light.

How do we know? We are (just barely) starting to get to a low threshold of herd immunity. ( Sorry in advance for all those UTM parameters in the link and for putting you in abVariantId=1.)

Even still, I’m not going to be looking at any news about vaccine distribution for a while. 

What I’m reading lately:

1. Creeping as a service

2. Algorithm interviews: theory versus practice

3. Why do we ignore systems?

4. AI incident database

5. There is a TON of good content on why journalism is bad lately: Journalism and expertise,What Happens when the NYT makes mistakes,  and the non-expert specialist

6. Feature stores for self-service ML

7. Wikipedia in Kyrgyzstan

8. If you have a chance to read something Paul writes, do not say no. This is on the return to the office.

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

A few months ago, my corner of the internet was completely fixated on Four Seasons Total Landscaping, the small yard services company in my home city of Philadelphia, where the Trump campaign accidentally booked a press event (instead of the actual Four Seasons downtown.) 

In the midst of the noise of all the memes and roasts going around on Twitter at the time, one struck out to me: 

I had no idea what this was referencing, but it seemed really inside-joke funny, so I showed it to my husband, who laughed. 

“Where is it from,” I asked him. 

“Star Trek,” he said.  It turned out to be an episode from The Next Generation TV series called “Darmok, in the fourth season.  (There are many Star Treks, some of them better than others, but TNG is generally considered a very good series.) 

An avid Star Wars fan, I actually had never watched Star Trek, but for the sake of the meme, I was really curious, and wanted to watch. 

Meme it so

It’s possible to watch Darmok out of sequence within the TNG series because the episodes generally are not sequential, but then you lose the context and tone of the show. So, I started watching TNG from the beginning. 

The scene is set that we are on an intergalactic mission into the far reaches of space happening sometime in the 24th century. The vibe is that the goal of the mission is diplomacy rather than offense, and the crew follows the Prime Directive, which means they can’t interfere with the technological development of other civilizations. The heart of the series is the ensemble cast, led by the gallantly leader-ly but aloof and stubborn Picard and his team, all offering different points of view on how to collaboratively solve the problems they come across in the universe. 

TNG doesn’t really pick up until season 3, but there are a lot of really good one-off episodes, including the first one, a two-parter called Encounter at Farpoint. In the episode, a strange being called Q comes to seek the Enterprise and puts the human race on trial. 

Later, Q changes into many costumes of Earth's eras, including the late 20th century in the guise of a United States Marine Corps captain: "Actually, the issue at stake is patriotism. We must go back to your world and put an end to the commies. All it takes are a few good men." Picard tells Q that that kind of nonsense is centuries behind them. Q brings up that Picard cannot deny that Humans are a dangerous, savage child race, which Picard denies, saying that Humans have made rapid progress in only a few centuries. 

Later on in the second part of the episode, Q appears as a judge in an Inquisition-like courtroom and calls humanity a grievously savage race, challenging Picard to answer for all of mankind. 

For someone used to shows that, even in the golden age of television, start with either something exploding or someone having sex as gimmicks to draw the viewer in (ahem, looking at you Halt and Catch Fire), it was a really refreshing change to be challenged to think at the same time as I was being entertained. 

Basically, the TL;DR is that TNG is a very complex show with a lot of nuanced social messages wrapped up in a space adventure. And I’m sure there are tons of thinkpieces out there as to how a tech team is like the Enterprise crew, how Star Trek predicted the rise of interfaces such as the touchscreen, and how right Star Trek got the science, but my general take is that it is Good TV and should be enjoyed as such. 

Darmok and Jalad at Memenagra

On to Darmok! Darmok is an episode of the fourth season where the crew Enterprise encounters a race called the Tamarians. Previous missions have indicated that it is impossible to communicate with the Tamarians, but the log doesn’t specify what they mean by that. 

I’ll let this great Atlantic piece take it from here: 

Picard exudes optimism as his starship courses through subspace. “In my experience communication is a matter of patience, imagination,” he beams to his senior staff. “I would like to believe that these are qualities which we have in sufficient measure.” But after hailing the alien ship upon arrival, contact with Children of Tama proves more difficult than Picard imagined:

DATHON, the Tamarian captain: Rai and Jiri at Lungha. Rai of Lowani. Lowani under two moons. Jiri of Umbaya. Umbaya of crossed roads. At Lungha. Lungha, her sky gray.

(no response from Enterprise, looks at First Officer in frustration)

(slowly, deliberately) Rai and Jiri. At Lungha.

FIRST OFFICER (laughing): Kadir beneath Mo Moteh.

DATHON: The river Temarc.

The officers immediately stop their laughter—as if ordered to.

DATHON (continuing; for emphasis): In winter.

DATHON: Darmok.

The First Officer looks very concerned—objects.

FIRST OFFICER: Darmok? Rai and Jiri at Lungha.

DATHON (shrugs): Shaka. When the walls fell…

FIRST OFFICER: Zima at Anzo. Zima and Bakor.

DATHON (firm) Darmok at Tanagra.

FIRST OFFICER: Shaka! (indicating situation) Mirab, his sails unfurled.

DATHON: Darmok.

To watch the aliens communicate in this way is a jarring experience, and the viewer is just as lost as the Enterprise crew. At this point, the Tamarians become frustrated and the Tamarian captain, Dathon, teleports, along with Picard, down to the surface of the planet they’re orbiting, El-Adrel IV. 

While Picard and the Tamarian captain, Dathon, spend time on El-Adrel IV, Dathon continues to speak in cryptic half-phrases with night and danger closing in and Picard becoming more bewildered, frustrated, and entirely unable to communicate with his ship. 

Finally, Picard starts to understand. 

As night falls on the surface, Picard fails to make a fire while Dathon lounges comfortably around his roaring blaze. Dathon throws Picard a torch, incanting, “Temba.” After first misunderstanding that Temba might mean fire, Dathon clarifies,  “Temba, his arms wide.” And Picard begins to fit the pieces together, “Temba is a person. His arms wide…because he’s…he’s holding them apart. In, in…generosity. In giving. In taking. Thank you.”

The phrases are actually shortcuts to recounting historical events and people in the Tamarian culture and history, and if you are part of that culture, you know immediately what each phrase might mean. The episode ends, sadly, with Dathon dying, but Picard understanding what he means with his emblematic catchphrase, “Darmok and Jalad at Tanagra,” two great warriors who came to an island separately, faced a great beast, and were united by the challenge. Finally, Picard understood. 

This is much the same way that internet memes work today. It is a remarkable realization. In fact, many people have talked about this, the idea that Darmok predicted the rise of the existence of memes. And, of course, there are now academic papers written about the observations that the episode made of our shared cultural understanding of the world. 

But the amount of layers of knowledge that go into creating a shared cultural exchange of communication are truly enormous.  For instance, how much did I have to understand to understand the picture of the meme of Darmok and Jalad (at Tanagra) I had to 1) Understand the context of the US elections at that point late in the game 2) Know what Four Seasons was, why it was different from Four Seasons Total Landscaping and why that was funny, 3) Understand Star Trek Culture 4) Understand the Darmok episode and, most importantly 5) Speak English and participate in English language humor Twitter, where I received the meme.

Once I knew all of those things, it was effortless and hilarious to understand. Until then, it was an impenetrable mystery. 

Meme Culture

So what? The so what is that memes have, without us realizing it, become an entire language with the ability to either unlock or lock worlds for us online regardless of what language we actually speak. This has been true since the start of memes, but we are now about ten to fifteen years into true internet meme culture, and we have our first instance of when a community based on mostly communication by memes, aka WallStreetBets, has created a massive movement based mostly on threads of people replying to each other with “diamond hands”, “do it for Harambe” (a meme that makes almost no sense if you were not following Reddit culture in 2016), and “if he’s still in, I’m still in.” 

The diamond hands rocket ship army, led in part by chief memer Elon Musk, who had this to say about memes:

He has helped WSB coordinate a rally of GameStop that’s, as of this writing, lost a ton of people a ton of money, but has more importantly, made Wall Street kind of scared. 

Memes are so powerful, and now, propped up by massive social networks, large enough to drive social and economic change.  And, what’s interesting is that, even though English has become the de-facto language of online communication, meme language has become even more so. There are hundreds of memes that started in English and are now present in Spanish, Russian, and every language you can possibly think of. 

Meme culture is a language and mentality on its own with a shared viewpoint of the internet. I couldn’t tell you succinctly what that language and culture is, but I know it when I see it, and there is absolutely no way I can convey the whole of it to people who haven’t been following memes for a long time. I’d argue that I have more in common with someone who, in terms of meme culture is Extremely Online and living halfway across the world from me than I do with someone who lives next door. 

Which brings us to a problem with Darmok and Jalad:  Memes only work to bring people together to action when as “Darmok” showed us, if we fully understand them and if we share them. And, also, only if we take them seriously. Which we should. Otherwise, Dathon dies in vain at El-Adriel.

What happens when not each of the Tamarians interpret the story of “Darmok and Jalad” at Tanagra the same way? Or what happens, when there are competing meme cultures, each with their own very distinct set of memes, that contradict each other, because we are all subject to our own echo chambers these days?

It’s something that’s extremely hard to study because meme culture is so ephemeral. It almost can’t be reached by machine learning or analytics or tracking data. We are flying extremely blind here, which is scary, because these are the forces that are now coordinating political, economic, and geographical movements just as much as they are about uniting people over a joke. Don’t believe me? Try to search for Winnie the Pooh in China. 

Meme culture requires ethnography and linguists and much more study, and understanding, is my stance, otherwise we are all on El-Adriel, our fists closed (probably from having such strong diamond hands.)

What I’m reading lately:

1. The value chain

2. How FB’s recommender works (h/t The Sequence)

3. MLM work during the pandemic

4. Brandolini’s law

5. The pandemic’s strange in-between time

6. Why Vine was better than TikTok (Hint: It’s a Lady Gaga problem)

7. The Web Development history project

8. The non-expert specialist

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

1. I gave a keynote at RStudio’s 2021 Conference. You can find all the slides here and it’ll be up on RStudio shortly!

2. I gave an interview for Coralie’s Parenting in Tech project project

On to the newsletter!

Portrait of a Young Man, Antonello da Messina, 1478

What would you do if you had a second chance to recreate the internet? That’s the question I find myself asking all the time lately. 

We now know about Cambridge Analytica, about the Pinterest lawsuit, about Project Dragonfly, about Google and the Nothing, the $1 billion Rubik’s Cube, the Experian Data Breach, the Google AI controversy, the Whatsapp thing, the other Whatsapp thing, It’s Time to Build, and the 10k people in San Francisco. We know that humans are not Web Scale. 

We messed up. 

Where do we go from here? 

Let’s say you have a ground floor shot at being at Facebook. What would you have done differently? For me, It’s obvious that I wouldn’t make it about advertising, and I wouldn’t do mass data collection or try to mandate real names. 

But if there was no advertising, there would be no money for Facebook, which means it would never survive as a company.  Which means all the good it’s done like connect people with rare diseases to share information, form parenting support groups,  and connect small businesses with the community would also be gone, too. It wouldn’t have connected me with some of my best friends.  Facebook is more than a misinformation machine, it’s also a marketplace, a place to share pictures. I can tell how vaccination is going in my area by checking Facebook. 

We live, as always, in the age of the great cross-hatch. Social networks are largely this big, horrible thing, but they also have very real positives.  How can you tell, when you’re starting a company, which of these things the company will be? Will your humble notetaking company, Evernote, be a place to store all of humanity’s knowledge, or will it be an awful arbiter of secrets great and small in case it accidentally opens everyone’s notes. Will your work communication platform, Slack, be a positive boost of productivity, allowing people to work remotely, or will it be this huge octopus, tentacles reaching through all your devices to make you work off the clock?

We don’t know until we file the articles of incorporation and set forth. 

Sure, we can guess. We can set up guardrails. That doesn’t at all excuse thinking these things through as we work. “It’s hopeless, so we shouldn’t even try.”  But, there is a compound tragedy in this wonderful, terrible last fifteen years in technology. The tragedy is this: we humans are terrible at predicting things. (Don’t tell the data scientists.) 

We can asymptotically approach reality, to an extent. But,  all models are terrible, George Box said. We don’t even fully understand numbers. And we don’t know for sure what the future will hold. And, what’s worse we are terrible at understanding how systems will interact. If we decide to track users, what will happen and what won’t? If we decide to track some users? We are, in a way, always trying to understand the beginning and end of this meme: 

How do you get from Zuck trying to start a social network to pick up co-eds to the fall of democracy? And what happens five years from now? We don’t have big enough of a neural net to model the predicted possibilities. 

And the second tragedy is that all companies must fight the market to survive. Companies start as babies, when the odds are entirely stacked against them, and, at the beginning, they don’t have the luxury of making decisions based on ethics or guidelines or frameworks. They are hungry and blind, and they must go after the money. 

If you don’t believe me, read Shoe Dog or The Making of Prince of Persia or The Great Beanie Baby Bubble or Skunkworks or Super Pumped or any one of the foundational books about how companies grow and thrive. There are some that take the very hard path of pursuing ethics first, but at the cost of growth later. Almost no one, especially anyone working on money borrowed against them, can afford to do this. 

This is not an excuse for big companies acting like marauders. But it’s just an observation that most companies don’t start this way that we want them to, with a balanced deliberation.  They start in a desperate panic to survive, with a bunch of compromises that they forget, with un-archetected data, with “we’ll fix this later.” because if they laid all of this out at the beginning, they’d never get off the ground. 

What do we do about this, these two fundamental laws of physics and capital? Knowing what we know now, how can we fix the internet going forward? 

I’m not a prophet, just a rando with a newsletter. 

But it’s 2021, and we’re sitting at our desk.

The VC calls. 

“It’s time to build,” he says. 

What will we say?

What I’m reading lately: 

1. No meetings. This is very similar to how Automattic works. 

2. Interview with Max Levchin. (Previously in Normcore on him) 

3. From Russia with free Shipping

4. Artificial intelligence is a house divided (AI previously in Normcore)

5. The high price of mistrust

6. All about Wattpad

7. Ghost knowledge

8. Congress basically says YouTube needs to shut down its recsys (YouTube recsys previously in Normcore)

9. ML tools 2021

10. Michael Scott

11. The Overstory (it’s good! There are trees!)

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Happy New Year, Normcore Readers!
Here’s to a happier, and most important, healthier 2021. 

“That’s it”, my husband said, as I unpacked a box of kipple from Amazon, to reveal yet another set of clear plastic containers. “You’re on alert. All future container purchases have to go through me.”

I looked guiltily at the pile of  containers already on our kitchen table. It was true, what he said, that I had too many containers. But I couldn’t stop. I was obsessed with acrylic boxes.  What happened to lead us to this tipping point was that over the New Year break, I finally got some time to myself and used it to sneak into Barnes and Noble for half an hour. 

I didn’t mind that, like every store visit these days, the atmosphere was somewhere between boring dystopia and sanitation ward, with everyone in masks, signs to stay six feet apart in corporate fonts, and announcements emanating from unseen speakers reminding everyone to keep their distance because of local safety regulations. 

I also didn’t mind that Barnes and Noble, once a powerhouse for reading, literature, and culture, is now looming on the precipice of an irrelevant twilight. It has been bought by a private equity firm which, like most things private equity touches, will be strip-mined and sold for parts.

The Home, Edited

It was still the holidays and I was alone in a bookstore, and I was cherishing the moment.  In perusing the shelves, a coffee table book called The Home Edit caught my eye.

The Home Edit is a “system” run by two very upbeat women from Nashville, 

Both had recently moved to Music City because of their husbands' jobs and were unemployed coming off a handful of career upsets. Teplin, 41, had launched and then shuttered several one-woman businesses, including a wedding invitation service and a greeting card company. Shearer, 38, had been turned down for her dream job at a home organization company. 

Then a friend who knew they both had a gift for creative categorizing introduced them. “I reached out to Joanna and was like, ‘This is fate. This is perfect.’ And Joanna was like, ‘No.’” Shearer recalls with a laugh. Teplin begrudgingly agreed to a lunch and everything changed.

They knew their playful style (think rainbow-order bookcases and hand-drawn labels in Shearer’s loopy script) was perfect for Instagram and made a plan to get a handful of high-profile clients to let them organize their space and then share their work on Instagram.

Luckily, Teplin had an in: “Christina [Applegate]’s daughter and my daughter had gone to preschool together and we became good friends.” Through Christina, they met Selma Blair and their network of actors and influencers started to grow.

The vibe of Home Edit seems to be, “If you are scared by the Kondo method and optimistically believe that you can organize your socks by color,” this is the system for you. I’d already been ambiently exposed to the popularity of the Home Edit system through a show on Netflix and their imminently instagrammable content, but the book pushed me over the edge. 

I was immediately drawn in by the idea that, if you have clear containers that are the exact size of the thing you’re storing, and you can cull at least some of your kipple, you can make your home look as good as Reese Witherspoon’s.

In theory. 

In practice, my own attempted home edit started to grow in complexity.  Because, first I needed to do an inventory of all my items. We threw out, donated, or gave away about 30 bags of kipple over the past month.

Then, I needed to buy the exact containers I needed for all my items, doing measurements of cupboards and refrigerator shelves. Then, I needed to manage the arrival of the containers in larger, cardboard Amazon containers that are impossible to get rid of, leaving me with piles of cardboard boxes on my back porch. 

Finally, I now need to constantly monitor the containers to make sure all the items stay nice and orderly in them. Sometimes, it’s my fault for not keeping up with them. Other times, my son tries to eat my daughter’s organized chalk and throws it in the toilet. 

There are a lot of containers to keep up with, constantly organize, and buy. 

In the process of buying, organizing, and throwing out stuff, I realized something about this Home Edit process that chilled me to the bone.

I had tried to Dockerize my house. 

Boxes in Boxes

If you are not familiar with Docker, I am tremendously happy for you. The best way to explain it is that Docker is a technology you’re running a computer inside your computer that’s not like your actual computer. A matryoshka computer, if you will. 

Let’s say you write some code on a Mac laptop. But you need to run that code on hundreds of thousands of servers inside your cloud. Most of these servers likely run Linux. 

How do you make sure the code works across operating systems? 

You create, essentially, a box, a container, that acts like a Linux operating system, and write your code in there. Then, you run your code in that box. While it’s using your computer’s power, and systems, it’s actually running on Linux, so to speak. You can then neatly ship that code, in a box, to any other computer that runs Docker and run it there. (Please do not reply to this newsletter telling me that Docker is more like a process than a full VM, and please absolutely do not mention hypervisors, c-groups and union mount. We’re trying to be simple here. Acrylic container-level simple.)

Docker very neatly removes a lot of problems that developers used to have, which was namely solving the common “it works on my machine problem” of having to integrate their code across multiple machines and environments, managing the ever-growing complexity of the modern software stack.

Docker is a fantasy world, because as soon as you have Docker, you have a different set of problems. Now you have to host the containers somewhere, manage the containers, be able to run them at scale, and *shudders* figure out port egress from them. Now, granted, it’s a beautiful fantatsy world, and I prefer to use it every time I’m developing an application. It definitely solves a lot of problems, but it also creates a lot of them that we now have to figure out how to manage. 

Society in Boxes

We are all, it seems lately, trying to figure out how to containerize our lives. We are all looking for contained simplicty. Whether it’s getting rid of the constantly arriving kipple,  the New Year’s resolutions to try to spend less time staring at screens (ha!), reorganizing our lives as a result of COVID (new jobs, cross-country moves, Peletons), or delegating some of our decision-making that has become more complicated.

One area this has been clearly evident over the past week, especially is the recent conversation around bans on social media platforms.  The issues related to Trump’s social media ban seem, at first glance, very simple and clear-cut.

But, I am worried that we are in a bit of a quandary when it comes to the function of social networks. Is it ok to ban people? If so, how should people be banned? What is acceptable and where are the limits?  What should the processes be? How should content be examined?

I think we are just thinking about these things now because our networks are being stress-tested in a way they never were before, and of course, more importantly, because it affects the companies’ current and future percieved revenues.

I’ve been thinking about this problem for years, but I still don’t know what the answers are. This stuff is really, really hard. It took us probably 30-40 years to figure out how the telephone worked and what it should be used for.  Facebook is sixteen years old, and good lord what a terrible teenager it is. 

What I do know, is what I wrote briefly on Twitter a few days back: 

I’m being somewhat intentionally vague about the actual problems I see here because I think they are much, much larger than just the issues of the past several weeks, much larger than the political issues that have come up, and that solving them will not be small, quick fixes, but a liftetime’s worth of work (if none of these companies get broken up as monopolies under the next few administrations, of course.)

What’s happening this minute, though, is that these companies, in order to stay attractive to advertisers and stay in business, are trying to containerize, compartmentalize, and create clean, neat, labelled containers to hold our thoughts. They’re trying to make our public discourse instagrammable.

Unfortunately, we’re years beyond that point.

It turns out that it’s MUCH harder than we thought, because humans themselves are complicated, messy, and often hold contradictory opinions, particularly on how moderation should work.

I’m sorry to start off the new year with a low note. It’s gonna be a mess for a long while to come. 

In the meantime, you’ll find me on Amazon.com, searching for acrylic boxes. Acrylic boxes are easy. Acrylic boxes are good. Just don’t forward this to my husband.

What I’m reading lately:

1. Connecting Papua New Guinea to the internet

2. What’s a message queue?

3. 45 years in the software industry

4. People don’t believe Hellen Keller existed anymore

5. The early days of the web

6. Predictive modeling: a retrospective

7. What glue languages do you like?

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Some time ago, on a dark snowy winter’s night, in a land far from here,  four Councilors to the Emperor stood together huddled in the cold at the foot of an immense mountain at the far northern reaches of the Empire.  

These were wise men and women, each learned in their respective fields of study. They lived in fine wood-paneled scholars’ chambers near the Emperor’s headquarters. they had fresh fruit from the southern shores of the Empire delivered to their anterooms every morning.  Their birthdays were celebrated throughout the land.  Farmers cheered and took their hats off whenever the Councilors’ carriages passed, on their way to important business. 

But, in the past year, the Emperor had declared war upon his enemies outside the empire, and was now marching north to meet them, far from the sunlit shores of his sea kingdom, over mountains and treacherous terrain. The opposing army was somewhere beyond the peaks. He needed to cross them. 

Only now, in the dim of the mountain pass, a dark, menacing figure blocked the small path up through the peaks. It was clear that the figure was sleeping, but other than that, not much was visible in the snow and howling wind. 

The Emperor’s great army stood at an impasse.  Horses snorted with impatience and soldiers rubbed their chapped palms together, stamping their calloused feet to stay warm. The army was restless and nervous. The enemy could be waiting in the darkness if they did not cross. 

“Come,” the Emperor said to the four Councilors. “Study the figure with all your infinite wisdom, and tell me what’s blocking this mountain pass and how we can move past it to form our attack.”

The four Councilors looked at him unsteadily from their fine stallions. 

“Go,” the Emperor commanded with an even, steely voice. 

The first Councilor, trembling, ascended to the figure in the darkness. He waited for a beat too long before approaching. He gulped.  The beast made a terrible sleep noise. Sticking a hang out, he groped the black air. All he felt in return was dry, hard bone. His hand trembled with relief, feeling around. Only the shell, smooth and flat, met him.  It must be an immense tortoise. “Sire,” the Councilor said, returning and kneeling before the Emperor. “This beast is the Turtle at the End of the World. As it sleeps, it can be bypassed. When it awakes, it offers travelers a riddle. We must somehow clamber over the turtle while it rests. Or overturn its giant carapace.”

The second Councilor, no less scared than the first, came up to the great beast, and put her hand on its side. She felt only warmth, fur, and the great beating of an ancient heart. She probed gently, further, and felt its chest rise and fall, rise and fall, somewhere deep within.  She returned. "Sire, it must be a yeti. We must make haste to kill it as soon as possible, for at midnight, it awakens and draws humans into its lair. It has supernatural strength and will be hard to fight.”

The third Councilor passed the first two with a scornful glance that said, "I'll get to the bottom of this. You know nothing.”  He put out a hesitant hand and felt a single great coiled muscle that moved as the beast snored, tense and strong. There was no question. “ ‘Tis a monstrous serpent," he said, matter-of-factly. "We shall chop off its head and then the great snake will be no more."

The fourth Councilor moved into position and tripped upon a great, thick hoof splayed out in front of the beast. "It must be a horse, perhaps even a great stallion," she said "We need to treat it with great respect. We must capture it and ride into battle. The enemy will surely fear us.”

"Well,” The Emperor asked, becoming increasingly agitated and looking at his Councilors. "Which is it?" The Councilors glanced at each other, suspicious and sheepish. They were the wisest in the land, and yet they could not come to a consensus on this beast. 

"Come, let us discuss in council...and over some brandy," the first Councilor suggested smoothly, to pacify the great man,  and they all retreated to the Emperor's tent, huddling away from the driving snow. 

While the four Councilors were seated among luxurious carpets, deriving warmth from a  bottle of cognac, a small girl, one of the children of the soldiers that followed the camp, was woken up from her sleep among a nest of warm army dogs by the great beast snoring.  More curious than frightened, she tucked her small slingshot in her pocket and started walking around the darkness of the pass, traversing the length of the animal. 

She felt the fur, the bone, the great muscle, the hooves gently, tenderly, one by one. 

"It's a mammoth," she giggled. "it's a mammoth!" she said, over and over again, skipping through the camp, over the snow drifts. 

She passed the great tent, where the councilors, four shots deep into the bottle,  turned their heads. “Silent child, we’re trying to solve this problem here.,” said the first councilor, slurring his speech. 

The emperor silenced all of them. “What is it you saw, girl?”

“It’s a mammoth!” she said, laughing and brushing the snow from her bangs. “I went around the whole animal! I felt its great thick fur, its tusks, its hooves, and its trunk. And it definitely is a mammoth! If we wake it up and yell at it, it will move! We just need some big pointy sticks!”

The councilors looked at each other, the color draining from their faces, the dignified air of seconds ago draining from the tent. 

The Emperor narrowed his eyes at them. It was silent for a second longer before he exploded. 

“Fools! How many times have I told you that you need to be systems thinkers?! Always with you it’s about the single piece of the picture! The latest controversy! The latest scandal!”

“But you absolute witless wonders, it’s like none of you even read Normcore Tech. Squire, pull up my morning scroll. Why, right here she says to pay attention to the system rather than the individual players! Of course there are going to be controversies! And of course the individuals are important. But the controversies are not just about the individual players! The stories are much bigger!”

“They are the result of these mammoth systems and companies we’ve watched being built for years and that now deeply impact the way we think, the way we work, the way we live. If we focus on each individual controversy in 2021 and  we do not examine the entire mammoth in 2021, we are going to be massively doomed. We’ll never get over the mountain. And we’ll never be able to attack the enemy.”

The Emperor stopped. His voice became eerily quiet-calm. The Councilors helplessly watched him pace around the tent. Then he stopped and took a small, casual sip from his glass. 

“I’m disgusted with all of you,” he said at long last.  He turned to the soldiers guarding the war tent. “Tie them all up and post them outside for the night so that they may feel shame at not being systems thinkers,” the Emperor said. He paused, “And take away their Apple Watches,” he added as an afterthought. 

He paused again, finally. “And promote that child to Senior Staff Director of my distributed armed forces.”

A great clamor of protest arose from the Councilors, but it was of no use. The Emperor’s word was law. They all left, their heads hung in shame. 

“What does he need to attack the enemy for, anyway,” one of the Councilors grumbled as his hands were being tied. 

The fourth Councilor spat bitterly. “The Emperor has heard that the enemy possesses a vast quantity of something known as Bitcoin,” she said, and walked out into the night.

A very, very long-awaited happy and (most, most importantly) healthy 2021 to Normcore readers and your families and loved ones. Here is hoping we are at last coming out of the very long tunnel that is 2020.   I’m taking a break and will be back in January. 

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Early last year, the face of an extremely terrified woman looked at me through the small window of the YouTube app on my phone. “Help me,” she said with her eyes. “I’m a little nervous but excited,” she said with her mouth as she got ready to use her new Peloton bike. 

The ad immediately caught flack in the media. “Unintentionally horrifying,” said Vox. Sexist and dystopian, said the New York Times. 

“I didn’t realize how much this would change me,” Grace from Boston says in the video, while the present-day couple smiles at the “changes” the Peloton has brought them. The Peloton looms in their open-plan dining area. 

Blanketed with ricidule, Peloton, a publicly-traded company, faced a steep stock fall. From last year: 

Peloton said Wednesday it is “disappointed” by the wildly negative response to a new holiday TV commercial for its exercise bikes that coincided with a 13% drop in the fledgling public company’s stock price since Monday.

I laughed right along with everyone else. It’s ridiculous, this desperate advertisement for a $2,000 piece of exercise equipment that’s the fitness junkie equivalent of the Snoo. 

In a Normcore universe, a number of questions loom. Why does a bike need to cost $2,000, plus monthly fees? Why does an exercise bike need an app or a data engineering team (presumably the existence of the data engineering team implies that the $2k cost covers part of the assuredly-existing Kafka cluster)? And why do you need to buy a whole new house whose sole purpose is looking good around the Peloton? 

As you can imagine, there is nothing about the Peloton, including the pricetag, the data tracking, or the connectivity,  for which I am a target audience.  

And so for months I disregarded the Peloton. The machine itself, the hype around it, the happy users with their multimillion dollar Peloton rooms, it was all not for me. 

But humankind, as you may know from reading this newsletter, is a multifaceted creature, a being of contradictions, a master of the cross-hatch. And something that humans particularly love is stories. Our whole ethos is built around stories, sharing stories, and sharing common experiences. Without stories, we are nothing. Many others have realized this throughout history, but the one man who really solidified this theory was Joseph Campbell in his work on the Hero’s Journey. He posits that there is one extremely common story throughout time, 

A hero ventures forth from the world of common day into a region of supernatural wonder: fabulous forces are there encountered and a decisive victory is won: the hero comes back from this mysterious adventure with the power to bestow boons on his fellow man.

Marketers understand this better than anyone, which is why they’ve come up with something called the “customer journey.” 

The customer journey is very much like the hero’s journey, except at the end, instead of slaying the Gorgon, you’re supposed to be exposed to so much hype about the Peloton that you buy a bike. 

And reader, I almost did. 

My Peloton Customer Journey started with the commercial, which I laughed at. Then, for many months I ignored Peloton because I was too busy trying not to have an anxiety attack in the early months of 2020.  But, as the pandemic wore on, I started focusing less on existential threats, and more on the fact that both my husband and I were working side-by-side from our basement. And, most of our days consisted (and still do) of two hours of childcare, followed by eight hours of silent work side-by-side in the basement, swiftly transitioning back into 3 more hours of childcare before collapsing numbly in front of the television, to wake up at 5 in the morning and do it all over again. And these are the good days when we have childcare for both kids. 

In this numbness, I realized that what I really needed was time and space for myself to work out. 

I started reading more and more stories about people buying Peletons. Gyms were closed. Its stock price was soaring. People like me started repenting and shamefacedly making their way down the customer journey map.

In May, I used money I’d saved to buy a Peloton bike — which starts at $2,245, plus $39 a month for the app — as a quarantine purchase for myself. Over the past few months, I’ve taken around 85 or so classes with multiple instructors. I can’t recommend that for everyone — fitness was and is a priority in my life, and your mileage may vary about the amount that you’re willing to spend. But I’m here to report that while Peloton and I can agree the commercial was a little off, I can also admit I was a little wrong about Peloton. 

I myself never envisioned that the bike was for me. And then I read Edith’s series of comics about how she got a Peloton. Edith, who used to run the wonderful women’s site The Hairpin (RIP), usually draws about pandemic life in New York City, knitting and going to farms, thinking about her life, a very reflective sort of comic. 

She wrote one comic about how she cried during a Peloton ride, and coming from anyone else, it would have seemed corny. But, I got it. For her, the bike was a form of connection in an immediately hostile world. 

For me, that one thing overcame the data collection (what are they doing with all that data, anyway?), the price tag, and my own personal embarrassment at suddenly wanting the very thing I laughed at all through the first couple months of 2020. 

But now, I was ready to buy a Peloton. 

I went on the website, and was immediately intimidated by the aggressively enthusiastic instructors there. 

Eastern Europeans are not happy in the best of circumstances, and, even more so, no one who sweats for an hour at a time in public can possibly be this exuberant, and is therefore suspect. 

I was not ready for this level of confidence, and I was also not ready to pay $2k for a machine I didn’t even know if I had time to use. So, I signed up for the Peloton app, which you can use without the bike. If I could do this, then I’d have time, after the kids went to bed, to do the Peloton, I thought. 

I could be like one of those people, in their happy Peloton room, wearing $100 workout pants, being aggressively happy. 

That did not go well.  Strike two. 

Finally, now battling myself,  I used the very fancy AR app that is embedded in the Peloton site to try to visualize the machine in my house.  It’s a very good app. And what I realized is that, in a house that we bought with my husband for two people ten years ago, and one that is now for four people and rapidly being taken over by toys, we have no Peloton nook, no single spare inch for a Peloton, let alone a Peloton palace. 

That’s three strikes. 

If you’re wondering how this particular hero’s journey ends, it ends in my living room, doing free YouTube low-impact aerobics videos and avoiding the baby’s toys while my daughter walks in halfway through, demanding yogurt. 

So much for Space and Sexiness. 

I still think about the Peloton sometimes, and how I almost became a Peloton person, but this is honestly probably just as good as it can get for me in 2020, and, as always, just as normcore. 

What I’m reading lately: 

1. Good guy Russian hacker

2. This thread on immigrant kids hits me 

3. The engineering puzzle from hell

4. ML datasets 2020

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

My husband and I have finally started watching Queen’s Gambit on Netflix, making us officially the last people on the face of this earth to do so. It’s amazing what a phenomenon this show has been.

But, if you’re a Normcore reader, you probably were not so surprised.

There’s a lot to dissect in the show, including, very importantly:

But, one of the things that struck me most about the show came in the first or second episode, when Beth Harmon enters her first chess match. She comes up to the table to register. The guys doing registration there ask her if she has a clock.

“Clock?” she says, puzzled.

When you play in chess tournaments, you play the game with a clock so that games don’t take 8 hours, like they used to back in the old days.

The chess clock continued to evolve, and in 1900, it reached what we now recognize as the classic form of the chess clock and the type used in The Queen's Gambit: two dials, with a button for each player which, when pressed, stops their clock and starts their opponent's.

She doesn’t know any of the rules when she starts. She just knows how to play chess as she’s been taught by the janitor in the basement and is hungry to win. And that proves to be enough in the beginning: she beats the pants off almost everyone she encounters in her astronomical and troubled ascent to the top of the game.

As we were watching the show and various characters kept cluing Beth into the norms of the chess world, I was thinking a lot about this idea that someone can be enthusiastic, smart, eager, and understand at a basic level what’s going on, but still get tripped up on technicalities. And, if no one tells them what the important parts are, they’ll kind of flounder along.

Of course, I was thinking about how this applies to the tech world, and, more specifically to the tech interview process.

Everyone knows tech interviewing is broken - there have been millions of posts on this phenomenon. White board interviews are bad and take-home interivews are bad, Zoom interviews are bad and written inteviews are bad and GitHub interviews are bad.

Or rather, they all optimize for very different things. This is partiuclarly true in the data sphere, where job descriptions are still pretty new and fluctuate very frequently.

I was thinking about all of this as I re-read Dan Abramov’s ( a very prominent developer on the React team at Facebook) post about things he didn’t know as of 2018.

First, there is often an unrealistic expectation that an experienced engineer knows every technology in their field. Have you seen a “learning roadmap” that consists of a hundred libraries and tools? It’s useful — but intimidating.

What’s more, no matter how experienced you get, you may still find yourself switching between feeling capable, inadequate (“Impostor syndrome”), and overconfident (“Dunning–Kruger effect”). It depends on your environment, job, personality, teammates, mental state, time of day, and so on.

Some of these struck me as pretty crazy. For example, I’d probably expect someone who’s a senior developer to have a good intuition for bash and the command line (his first bullet), I’d expect someone working at Facebook to be really good at algorithms, and of course, I just naturally assume that everyone knows Python, the best language.

But my assumptions are all based on this monolithc view of what a “senior” person is, built up from observing senior people around me, reading dozens of job ads, going on interviews, and reading conversations online about it. If you ask ten other people what being senior means, they’ll probably tell you something different. We might overlap in 2 or 3 areas and then branch off.

The thing is, even though there is one way to get to Grandmaster, there is no one single way to interview and become a senior developer or data scientist. The difference with the tech world is that there is not one single set of rules like there is in a chess game: everyone is playing on a different board, with an ever-growing number of pieces.

As I’ve written, this was not the case a while ago,

The early years of programming were mainly focused on moving from this really physical programming infrastructure, of writing bits of logic on punchcards and running them through enormous machines, with the first programming language being assembly, the language that horrified Ullman so much. Assembly is very close to bytecode. 

As the machines became smaller and more accessible, more programming languages came about.  By the late 1980s or early 1990s, when Ullman was well into her career,  your average programmer might know, in addition to assembler, COBOL, SQL, Fortran, and maybe the newcomer, C.  

In a lot of ways, the world was much smaller and less complex. We didn’t even have version control!

But it now is, and all of our interview processes mostly try to take into account this given accumulated set of skills. We don’t do anything to assess for some of the most important qualities, the ones that get Beth started: intuitiveness, desire, enthusiasm, and generally ease of working with the person (unfortunately, Beth would fail that test quite a lot.)

In short, in a lot of the industry, we don’t look for this, a lot of the time, because, in the growing set of skills, this is the only one that generalizes:

And we should be.

What I’m reading lately:

1. What’s a book you’d like to read but hasn’t been written yet?

2. What do people use Spark for now?

3. TIL Israeli queues

4. Moscow subway operator comics

   ---

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Art: Advertisement for Pope Manufacturing Co., "We are Having a Heavenly Time" Cassius Marcellus Coolidge, 1890

Something I worry about a lot is that I don’t understand how the internet works. And by that, I mean the entire web, the whole shebang, from when I press “Send Newsletter”, to how Normcore makes it way up and through WiFi and then thick cables somewhere in the darkness of the Atlantic, through seas of malware, bypassing the dark web, and finally finds its way to your inbox, all of it. 

I swim in the broad, rolling waters of our internet every single day, for hours and hours on end.  I earn my salary building complicated, distributed products that rely on understanding machine learning research knowledge assembled from experts around the world and traverse millions of network requests. I write tweets and newsletters and read hundreds of thousands of words on news sites and blogs and academic journals. I like memes.  And yet I have no more understanding of what makes the entire thing work any more than I know what happens when I get in my car and turn on the ignition. 

In previous Normcores, I’ve explored bits and pieces of it: web servers, networks and packets, how the internet got started and data centers. But, it’s still an enormous system, a living, breathing organism with layers of history, nuance, subcultures, and features that makes it all somehow work together every minute and second of the day. 

In the tradition of continuing to explore various pieces of online life, something that I’ve been wanting to understand a lot lately is adblockers.  I’ve been using one since probably 2005, but have never given a second thought about how it works. I just know that they make my online experience bearable.

For example, here are some modern sites, loaded with adblockers on the left and without on the right. Using the internet without adblockers is intolerable. 

And just for fun, check out Forbes.com, which shows you a video roll instead of the home page if you’re not blocking the ad. 

What’s so amazing to me about ad blockers is that,  like the data powering neural networks, they’ve been built up pretty much by hand by a few people over years and years and now impact the browsing experience of hundreds of millions of people around the world. 

New Kids on the Block(er)

In the early 1990s, the first banner ad was launched on hotwired.com, Wired’s website, in the same way most tech things are launched: with a few people in a room trying to guess how an entire online experience for millions of people might work.  

Jonathan Steuer, online tsar, HotWired: I was in the room when it happened. The conversation was about, “How do we come up with money to pay for this thing on the internet?”

Louis Rossetto, co-founder, Wired: People told us if you put ads online, the internet would throw up on us. I thought the opposition was ridiculous. There is hardly an area of human activity that isn’t commercial. Why should the internet be the exception? So we said, “Fuck it,” and just went ahead and did it.

Soon afterwards, a guy called James Howard, 23 years old at the time, got together with three other dudes to create Internet Fast Forward, the first ad blocker. 

Howard has been a computer hacker since he was 10 and on the Internet since he was 14. Now he’s chief executive of PrivNet Inc., the private company set up to market the IFF program. His partners are Gene Hoffman, a 20-year-old UNC communications major; Jeff Harrell, a 22-year-old English major; and Mark Elrod, a 22-year-old computer science major.

IFF allows users to block advertisements, blinking text, Web graphics and “cookies"--Web software that tracks a visitor’s movements through a Web site. The IFF device, which works as a “plug-in” to Netscape’s browser, can also circumvent the lucrative advertising sites that direct surfers to popular “search engines” such as Yahoo and Infoseek.

Howard said his gripe is not with advertising per se, but with the time it takes to view a page with advertising. “It can take 4 to 6 seconds to download each ad, and if you are on the Web a lot, that really gets annoying,” he said. “If the advertisers want to pay for a high-speed Net connection to my house, then I would take the ads, but right now it is costing me money to look at their ads.”

Here’s a video about IFF, which is pretty incredible. (I love that around 3:19, where the narrator says they’re in for a “tough battle”,  you can clearly see all of them playing Wolfenstein.)

Ad blockers really took off after that, and soon, an adblocker called Ad Block was created by Henrik Aasted Sørensen, at the time a Danish university student, who was procrastinating from studying, 

"I suppose some people expect Adblock to have been created in a fit of anti-capitalist rage, or as an idealistic effort to return the internet to its less commercial roots," Sørensen said. "What actually happened is I was supposed to be cramming for an upcoming exam at university [in Copenhagen, where he studied internet technology and computer science.] As a procrastination project, I decided to try out the relatively new possibility of creating extensions for the Phoenix browser — which is the browser that eventually got renamed Firefox. The idea was primarily to try out a new development environment and move a bit out of my development comfort zone."

Whomst among us has not created an iconic technology when they didn’t want to be studying for Biochem? After that AdBlock really took off, and became the premier ad blocking tool and browser extension for a long while, always being maintained by a small group of very opinionated people. Offshoots, such as uBlockOrigin and Ghostery, continue to be pretty popular.  (I use mostly uBlock these days because of the controversy around AdBlockPlus and their new ad acceptance policy)

What’s interesting in all of this is how few people were involved in making these changes that would eventually impact the way the rest of us interact with the internet. In hindsight, though, it’s not that different from the average opinion of 10k people in San Francisco, only at a much smaller scale. 

But how do all these programs work anyway?

The Tangled Web of Ads We Weave

To understand adblockers, it helps to understand a bit about how modern adtech works. 

Let’s say you go to your favorite technology publication, The Normcore New Times, to read all about how companies are removing distributed systems from their tech stacks.

Now, The Normcore New Times, unlike its humble predecessor newsletter, is aggressively ambitious about making money, and wants to monetize the everloving crap out of every page element. What they’ll do is sell space on their site.  

And not just any space, the places that researchers have determined people’s eyeballs travel to the most. So maybe they’ll do something like this: 

And then, after they figure this out, they might go to an advertiser directly, like, say Adidas, and ask if they want to place ads in these places. This is how it started out. 

In the early days of online ads, a brand would strike a deal with a website owner to host a paid banner. The onscreen space for that image, known as the ad inventory, would be sold by the publisher directly. (The magazine you’re reading right now made the first such transaction, back in 1994.) 

But, let’s say that you’re Adidas, the official sponsor of the Normcore New Times, and you’re serving ads like this: 

That’s all well and good. But you, the advertiser,  also have maybe a hundred other sites that need ads, and you have ads that are constantly changing based on which site they’re on, which styles you’d like to advertise that day, and a million other variables that are hard to keep track of when you scale. 

And, another thing. You can also customize which ads you serve based on cookies. So, maybe you’d like people who have looked at Adidas or searched for Adidas to be exposed to your ads across the internet. 

Now you have to keep track of not only your ad inventory, but which sites it’s stored on, how quickly those ads need to change, and users across sites. What you’ve built now is not just an advertising relationship, but a tech stack. 

What arose in trying to maximize revenue from ads is an entire ad pipeline that now looks something like this, where there is an enormous disconnect between the marketer, Adidas, and the publisher, Normcore New Times, in the name of “efficiency.”

That ad is now being served to you programmatically, in a process that puts a new ad for you to look at every time you load a web page.  

Today, the process has grown far more complicated, and humans are barely involved. “As they do in modern-day capital markets, machines dominate the modern-day ecosystem of advertising on the web,” Hwang writes. Now, whenever you load a website, scroll on social media, or hit Enter on a Google search, hundreds or thousands of companies compete in a cascade of auctions to show you their ad. The process, known as “programmatic” advertising, occurs in milliseconds, tens of billions of times each day. Only automated software can manage it.

There are a lot, a lot more implementation details, but the bottom line is that sites collect demographic information about you and bundle that information. They then send it to third-parties, which, in turn, pick from a collection of ads sent to them by ad agencies, and decide which ad should be shown to you on the webpage. All of this happens through 3 or 4 layers of technology and happens extremely quickly, although not quickly enough to not annoy users. 

Blocklists to the Rescue

Note that Normcore would never run ads because I’m really bad at selling out, BUT IF IT DID, you would want to block that sucker. 

What do you block? In the old days, you could basically determine which images on a page were “ad-sized” and easily block those image sizes. Today, it’s a bit more complicated, because the URL the ad could be hosted at could be at any one of a number of places. It could be coming from adidas.com, but that’s very unlikely, given the extremely complicated ecosystem. 

Usually, what you want to do is to keep track of any one of the URLs connected to the adtech ecosystem. What AdBlockers do, is check out the requests your browser is sending. If those requests are on the block list, it doesn’t load that content. 

Let’s take a look at what’s blocked on Wired.com, where that first p ran, for example. This is what shows up on UBlock for Firefox, as an example. The links highlighted in red are known regular expressions matching URLs that serve ads.

How does the adblocker know what to add to the blocklist? 

This is the crazy part. There are about 4-5 guys in the world (and some one-off open-source contributors) managing them. The biggest blocklist historically, and these days, is EasyList. 

EasyList was originally launched in 2005, as a kind of add-on to the Adblock browser extension. Several different people have overseen it since then, and today, a group of four people, led by a man named Ryan Brown, is authorized to change EasyList’s rules.

Over time, its list of rules (and exceptions to those rules) has grown sprawling. Analysis conducted by Brave last summer found over 70,000 rules in EasyList, a mixture of network rules, which determine whether a site fetches sites or code from web addresses that match a certain kind of pattern; element rules, which dictate whether certain page elements, such as banners, can be displayed; and exceptions to the element and network rules.

The other amazing thing about EasyList is that it literally is just a group of textfiles that you can browse on GitHub, and if you’re even more curious, dig in and look at everything that was added. And, if you want, you can even contribute. 

You can also see the commits, and see Ryan and a handful of others dutifully committing new changes in the dynamic, shifting landscape of trying to keep a step ahead of advertisers and providing a good browsing experience for the general population of internet users. 

Block On

What’s the impact of having such a small group of people take on the entire adtech ecosystem? Well, there are the usual development problems: bad, outdated data, for example. 

A recent study found a lot of outdated rules that slow down the blocking logic. It should be mentioned, however, that the authors of the study, Brave Browser, the company run by the former CEO of Mozilla and founder of Javascript Brendan Eich,  who runs their own adblocker, has a vested interest in uncovering these inefficiencies and exploiting them. (In a way, it’s kind of ironic that Brendan developed Javascript, which powers the modern front-end internet, including making it possible to dynamically serve ads, the very thing his new company is now trying to fight. Oh, and there’s also cryptocurrency involved, which makes me think that there should probably be a separate Normcore on Brave. But I digress.)

The other thing is that legitimate sites sometimes get blocked by accident and there’s no immediate recourse, 

While the purpose of the list has always been to keep advertising out of web experiences, EasyList’s rules regularly break normal editorial features on sites. In the past six months, EasyList changes have broken the buy buttons on commerce site The Inventory, the video player on Animal Planet, disrupted site navigation on Fandom, and disrupted the style and CSS loading process on job search site Indeed.

Though most of these issues were resolved quickly, as publisher sites continue to evolve, they have to contend with the possibility that they might run afoul of one of the most important crowd-maintained documents on the internet.

But, as usual, the problem is not in the list itself, but the continued growth of the online advertising ecosystem, and the attempts of adblockers to stay one step ahead. 

It’s kind of amazing when you think about it, this small list of volunteers manually up against a multibillion dollar industry outfitted with algorithms to the hilt, whose entire goal is to thwart them. But if that’s not normcore, I don’t know what is. 

What I’m reading lately: 

1. A short history of Flash

2. TIL there is a Burger King in Germany that’s housed in a former Nazi power station

3. Just watched Darmok, still thinking about this episode. Strong recommend

4. Storium

5. Dads, commit to your family at home and at work

6. How I Git

7. Guido at Microsoft

8. When data disappears

9. Writing a technical book: from idea to print

10. The great data debate

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Art: Argument over a Card Game, Jan Steen

What do you see when you power on your doom brick for the first time in the morning?

For me, it’s first and foremost, Twitter, starting with my @ replies. Then, I read my feed. Then, I scroll over to the “For You” section, which offers me a curated list of trending news based on, I’m assuming, some combination of tweets I’ve interacted with, plus my Twitter demographic profile, plus overall trending news, plus some other variables thrown in. (Actually, this would make a great interview question for Twitter if it’s not already: what kind of recommendation algorithm would you use to power this section?)

Then, I read my email - lots of personal newsletters, newsletters from newspapers, then a quick scroll through Slack work notifications, Telegram/iMessage, and, finally Hacker News. 

What’s the common thread behind all of these sources of content and information? They are all, except for Hacker News, powered by extreme personal selection, either by me opting into them based on my interests (newsletters, Telegram channels, etc.), or driven by algorithms (the “For You” section.)

I never get exposed to any content that the social media giants, or I, personally, think I am not interested in. This is the idea of the web based on the data science/recommendation platform idea of relevance. 

I’m not really talking about anything new here. We’ve been wringing our hands over filter bubbles for years at this point. They came to prominence right around the time of the 2016 election, but Eli Pariser, who helped bring the term into popular prominence, had been discussing them for years. If you haven’t yet, watch his TED Talk (I know, I know) on the topic, which he gave in 2011. It’s very, very good and very scary. 

But what’s really reinforced it for me is seeing the reactions to the election that happened (fortunately I can say past tense now, although for me personally last week seemed to last fifteen years). 

Based on my personal feed, it was clear to me that there was one specific set of facts about the election. There was obviously no way it could go any other way. But for friends and family who had picked different news sources and as a result been opted into different personalization algorithms on their doom bricks, the set of facts they argued were completely different from  mine.  To them, any news story I cited about it being over was suspect, and to me, any YouTube link they sent was completely out there. It seemed impossible to me to believe any set of ideas other than the one I understood, and vice versa. 

But, the even scarier part was that, of all the links and arguments they’d sent me, I’d never seen any of them come across my feed.  So, I was not prepared for a discussion about them. They were coming from a completely different personalized universe, into mine. And there was absolutely no intersect. We could not have an intelligent conversation about it and come to a disagreement based on a shared narrative. 

I’m using the election as an example, but this issue is much larger and continues to only grow. This is about every single piece of news we consume, and every piece of information. This is about all the decisions we make every day based on our mental model of the world. And the danger here is not that people consume different pieces of news, which they’ve done throughout history, but that, by the virtue of both A/B testing and personalized recommendations, we have been siloed so far away from people who are intellectually different from us, that it’s impossible to have a reasonable debate, because we simply don’t know each other’s sides even exist. 

In the past, there were varied media sources, but there were only a couple of them.  Everyone could read a single newspaper, be it the New York Times, or The Philadelphia Inquirer, or the New York Post, and argue on the merits of the facts presented there. “This article is wrong, this writer is lousy.” “Well I think the article is great, the writer is great, and the facts are true.”

Now, we are all getting served two different sets of facts. Or rather, variations of millions of different sets of facts. There is no single “source of truth”, no foundational set of facts or observed news from which to start a conversation for everyone. Trying to argue for or against something is like trying to herd a million small fish into a single school, or to grasp at a million pieces of sand scattered across a table. (Ask me, a mom dumb enough to buy her kids kinetic sand, how I know.)

We have completely lost our shared sense of what reality is.

Consider the Google homepage. It used to be the same for everyone, sometime before 2011. Then, Google invented online A/B tests, which are an extension of randomized control trial experiments, previously used in physical and social sciences. 

Over the past decade, the power of A/B testing has become an open secret of high-stakes web development. It's now the standard (but seldom advertised) means through which Silicon Valley improves its online products. Using A/B, new ideas can be essentially focus-group tested in real time: Without being told, a fraction of users are diverted to a slightly different version of a given web page and their behavior compared against the mass of users on the standard site. If the new version proves superior—gaining more clicks, longer visits, more purchases—it will displace the original; if the new version is inferior, it's quietly phased out without most users ever seeing it. A/B allows seemingly subjective questions of design—color, layout, image selection, text—to become incontrovertible matters of data-driven social science.

What happened is that, as Pariser says in his video, is that there is no single “page of truth” for Google. 

Today, A/B is ubiquitous, and one of the strange consequences of that ubiquity is that the way we think about the web has become increasingly outdated. We talk about the Google homepage or the Amazon checkout screen, but it's now more accurate to say that you visited a Google homepage, an Amazon checkout screen. What percentage of Google users are getting some kind of "experimental" page or results when they initiate a search?

A/B testing is also combined with the fact that Google personalizes your search results based on your search history and geolocation. Which means that if you search for “dogs” and you’ve previously done a lot of searching and browsing for huskies, you’ll get a husky page as the first relevant result, whereas someone else might get chihuahua, someone else might get collie, and so on. 

You can see how this is fantastic for you, personally, but for us, collectively, as a society, is horrible. Because we all think that different dogs are popular. How can we start a conversation if one person is coming from “collie” and another is coming from “husky”? (Everyone should come from collie, by the way, because they are the best dogs, thanks.)

And, what’s making this worse is that, now, on top of recommendations and A/B testing, there is another layer.  The social media companies, under immense pressure from public opinion, and more likely, regulation, are trying to control the kind of content that’s out there. This was especially prevalent in the time leading up to the election, where they had the unenviable task of deciding whether something was leading to destabilizing elections or not and blocking or not blocking content. But the thing with blocking content is that, as soon as you do it once, now you are no longer a pipe, an ecosystem or platform. You’re a media company and every single thing you block or don’t block will be under suspicion by different subsets of your users.

And what’s happening with media companies? They’re becoming tech companies. Consider this recent article about the New York Times. There is a lot of he said/she said in it, but what struck me was this: 

What the audience wants most of all, apparently, is “Opinion.” On a relative basis, the section is the paper’s most widely read: “Opinion” produces roughly 10 percent of the Times’ output while bringing in 20 percent of its page views, according to a person familiar with the numbers. (The Times turned off programmatic advertising on the Cotton op-ed after some employees objected to the paper profiting off the provocation.) Now that the paper has switched from an advertising to a subscription-focused model, employees on both the editorial and business sides of the Times said that the company’s “secret sauce,” as one of them put it, was the back-end system in place for getting casual readers to subscribe. In 2018, a group of data scientists at the Times unveiled Project Feels, a set of algorithms that could determine what emotions a given article might induce. “Hate” was associated with stories that used the words tax, corrupt, or Mr. — the initial study took place in the wake of the Me Too movement — while stories that included the words first, met, and York generally produced “happiness.” But the “Modern Love” column was only so appealing. “Hate drives readership more than any of us care to admit,” one employee on the business side told me.

By the way, if you’re concerned by/interested in Project Feels, the team working on it wrote up a pretty interesting technical explanation of how it works that goes further than the article above.

So news stories are now (and have been for some time) getting written based on what sells (which, no surprise, are strong emotions, usually negative ones) rather than what can be investigated and debated carefully over some amount of time. 

So, where are we all on our doom bricks today? We are more connected than ever, our phones are shinier, faster, and we can do amazing things.  And at the same time, we are immensely locked into our own tiny universes, and no longer coming at public discussion from a central place. 

I am, personally, very hopeful that we’re coming out of the basement McDonald’s from the perspective of the country and COVID. (Things are very bleak right now but the vaccine is on the way.) 

But the way I’ve seen our bifurcated information economy work in the last election, and without any abatement into this one, makes me very scared. And, in an ironic bit of self-reinforcement that makes it hard for me personally to see an end to this, this makes me scroll my doombrick even harder. 

What I’m reading lately:

1. Moving BBC to the cloud

2. The trap of the premature senior developer

3. Pippi and the Moomins

4. Kurt Vonnegut on writing

5. A canvas made of pixels

6. Kindness at work

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Art: Starry Night, Edvard Munch, 1922

Vicki’s Note: I guest-wrote a piece for Leon’s newsletter last week on what AI truly means. Go read it!

It was sometime in the middle of 2013 that I hit rock bottom.

I was still working full-time as a data analyst and going to my part-time MBA classes twice a week at night after work. I would then take the very late train home, nodding asleep in my uncomfortable plastic SEPTA seat, blearily wave to my husband, go to bed, and get up to do it all over again.

I didn’t have kids yet, but it was still an extremely grueling schedule, and some days I struggled to justify to myself why I was doing the program, what I would do afterwards, and how long it would take me to finish given that I was only taking a class at a time. And then I would go back to the SQL query I needed to complete immediately or else my work inbox would overflow.

It was one of these days when, in the half hour that I had between the end of work and the start of class, I decided to take a shortcut to class through Suburban Station.

Suburban Station is a below-ground terminal of the SEPTA Regional Rail system, which has a number of tunnels in Center City Philadelphia, all of them in dubious states of maintenance. I have to believe the station was planned and started with good civic intentions because it has a beautiful art deco entrance. But, even though most of the city’s suburban commuter traffic goes through the station and there are a slew of shops there, it’s entirely neglected by the City of Philadelphia, which, strapped for cash and resources, sees it mainly as an unsolvable civic problem.

Although it’s been much-improved in recent years with the addition of the Concourse at the Comcast Center, in 2013 it wasn’t great.

Of all the places in Suburban Station, the McDonald’s was probably the worst. In a busy underground intersection where crowds rushed by to and from work, completely submerged in the eternal Suburban Station twilight, its tables littered with used napkins, the restaurant itself seemingly didn’t know what it was doing there.

It was often the site of raised tensions, floors sticky with Sprite, and flagrant health department violations. It was not a place to idle with a bag of fries and people watch. It was hostile to the entire experience of dining and eating as such.

I was already having a stressful day at work, and I had three hours of class to look forward to. I believe it was finance, where we were doing balance sheets and income statements, which eluded me in undergrad. And here I was covering them again, only this time after a full day’s worth of work, the columns of numbers blurring together in my textbook late at night.

I walked past the McDonald’s, and made a fateful decision.

Reader, I had dinner at the Suburban Station McDonald’s.

I ordered a 10-piece of McNuggets, large fries, and a large Coke, because why the hell not? If you’re already at the bottom, why not go all the way down? In a way, it made me feel better, like I was in control of an uncontrolable situation.

Once you’re at the bottom, there’s nowhere lower you can go.

You can only start to slowly at first, but then more confidently, and faster, and faster, make your way back up.

We are all, right now, metaphorically, in the McDonald’s of the Suburban Station that is 2020. No matter how the rest of the year goes, particularly this next week, we’re still going to be in the basement. There’s no way things will get better before 2021. No one knows what even the next couple days look like.

But once they start getting better, they’re going to get better fast.

I think.

I hope.

I have to.

I don’t want to be in the basement for much longer.

I want to, like I did that night, leave the restaurant, feeling completely gross, but free, clean and clear, never to return to that subterranean darkness and the din of the trains rushing past, surfacing from Suburban Station and breathing in the crisp air of a cool fall night, quiet, at peace, the clouds in the heavens above moving quickly to reveal handfuls of small, twinkling stars.

What I’m reading lately:

1. Type in the exact number of machines to proceed

2. How to become a consultant?

3. The US childcare epidemic

4. If you can, please donate a computer screen to a teacher

5. Profile of Kelsey Hightower

6. COVID modeling in March

7. Localization and Gmail

8. What would it cost to fund the Octonauts?

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Art: Profit I, Jean-Michel Basquiat, 1982

I’ve been thinking a lot recently about what it was like to write code 20, 30 years ago.  

At the time, one of my favorite tech writers, Ellen Ullman, was working as a developer and project manager and wrote an amazing book, “Close to the Machine”, about the art of software development, life, philosophy, and everything in between. 

In one of the chapters, she tells the story of a meeting she had with a vice-president of a big bank, in charge of re-engineering global transaction processing payments. 

The vast network of banks, automated tellers, clearinghouses, computers, phone lines - all which go into sending a single credit card transaction around the globe - was her domain. 

“If it all breaks down,” she said, “the banks can’t balance their accounts.” 

…

A wave of nausea washed over me: I imagined what it would feel like to leave a bug lying around and wind up being responsible for shutting down banks around the world.”

….

[Ullman mentions that the VP said only three developers worked on the entire system. ]

The vice president laughed. “We’re lucky to have them. The system is written in assembler.”

“It’s in assembler?” I felt a true, physical sickness. “Assembler?” Low-level code. One step above machine language. Hard to write, harder to change. Over time, the comments begin to outnumber the programming statements, but it does no good. No one can read it anyway. 

When the vice president saw my sympathies, she relaxed. Now she wanted to talk to me about all the groovy new technologies being tried out by a special programming group. Multitiered client-server, object-oriented systems, consumer networks - all the cool stuff that was on the opposite side of the universe from her fifteen-year-assembler code. 

What was it like to work all those years ago, to write code? I don’t know. 

But Ullman and Neal Stephenson do.

In the beginning, Stephenson writes, there was a command line. He explains that computers, at their core, are moving information around. 

The smallest unit of information is a byte.  A byte is just a string of eight 1s or 0s.  Here’s what it looks like: 

01011100

When you have lots of bytes, you have bytecode. For example,  here’s Normcore Tech in bytecode: 

01001110 01101111 01110010 01101101 01100011 01101111 01110010 01100101 00100000 01010100 01100101 01100011 01101000

It is this bytecode that the machine can read, and that makes up all of our software and moves our information through the internet. 

How it started

We write computer programs for a number of reasons, but, ultimately, all of them are converting higher-level information that humans can understand (text, tweets, video), to bytecode, moving it around, and then converting the bytes again until they are human-legible.  

People used to have to do this kind of conversion very manually, 

We had a human/computer interface a hundred years before we had computers. When computers came into being around the time of the Second World War, humans, quite naturally, communicated with them by simply grafting them on to the already-existing technologies for translating letters into bits and vice versa: teletypes and punch card machines.

These embodied two fundamentally different approaches to computing. When you were using cards, you'd punch a whole stack of them and run them through the reader all at once, which was called batch processing.

Now (or rather, 1999, the vantage point from where Stephenson writes), 

[T]he first job that any coder needs to do when writing a new piece of software is to figure out how to take the information that is being worked with (in a graphics program, an image; in a spreadsheet, a grid of numbers) and turn it into a linear string of bytes. These strings of bytes are commonly called files or (somewhat more hiply) streams. They are to telegrams what modern humans are to Cro-Magnon man, which is to say the same thing under a different name. All that you see on your computer screen--your Tomb Raider, your digitized voice mail messages, faxes, and word processing documents written in thirty-seven different typefaces--is still, from the computer's point of view, just like telegrams, except much longer, and demanding of more arithmetic. 

The early years of programming were mainly focused on moving from this really physical programming infrastructure, of writing bits of logic on punchcards and running them through enormous machines, with the first programming language being assembly, the language that horrified Ullman so much. Assembly is very close to bytecode. 

As the machines became smaller and more accessible, more programming languages came about.  By the late 1980s or early 1990s, when Ullman was well into her career,  your average programmer might know, in addition to assembler, COBOL, SQL, Fortran, and maybe the newcomer, C.  

In a lot of ways, the world was much smaller and less complex. We didn’t even have version control!  But, even though the amount of languages was smaller, the information was harder to get to. There was no Stack Overflow. There were reference books, man pages, and online forums.  In the early days, programming and finding information about programming was not easy. 

I didn’t get help from user groups with programming, but they were around. They’d meet monthly, and there were certainly people there who knew programming. I mean, if you were a computer owner back then, there was an expectation that you knew at least a little about programming, even if it was just writing batch files in DOS. You couldn’t use the computers that much without having some technical knowledge about how to get them to do what you wanted, because the idea of an easy-to-use GUI was still new.

In the early days, it was easier to remember something than to look something up and software development took much longer to propagate, oftentimes on floppy disks that took a long time to release. For example, Windows 3.1 was 6 floppies that would have to be inserted in order, like a highly-arcane magical incantation, for it all to work. So, deep knowledge in any one of these languages and skill areas was very highly-valued. 

Most programming happened at the command line, in very similar ways to what Ullman and Stephenson describe: programming alone, in the dark, with only the brief flashes of light of user groups to help,  using archaic old languages or new languages with puzzling features. 

The meteoric rise of the internet created an opportunity both for different kinds of languages to evolve: Javascript as a platform for building the internet, the rise of Java for mainstream web services, and Python as a multipurpose web and data language.

The internet also meant that information about those languages could move be shared more quickly, gaining more followers and spreading across companies through blog posts and talks that could now be streamed online, thanks to services like YouTube that took advantage of the increase in available bandwidth. 

Along with the fact that it became easier to write the internet, it also became easier to collect the exhaust trails of people reading it, to harvest  the data that the backends of the internet spewed forth into data lakes.  With the growth of connected machines, has come an exponential proliferation of tools, cloud environments with thousands of services, different orchestration infrastructures, agile, kanban, scrumban (yes), and millions of developer conferences for each possible topic under the sun. 

Before, you were pretty limited in both your hardware and software environments. These days, you have your pick of 3-4 IDEs per language, different environments for moving that code to the cloud, integrating it with your other code, multiple applications, hundreds and hundreds of different ways to do the same thing. 

How it’s going 

We’re now in an interesting situation that’s really well-illustrated by the multifaceted responses to this tweet. 

In the replies, dozens of people who are pretty senior say that there is a lot they don’t know in their day jobs, and there is a lot they’re trying to figure out. 

Threads like this are often meant to empower people earlier in their careers. But it also made me think about something  I’ve been turning over for some time now. Because we have so many different software products, services, languages, and steps to production for any given piece of software, there is no such thing as a senior developer anymore. 

Or rather, there is no such thing as a senior developer who has both the depth required in earlier development environments, and now, the breadth required from the modern software stack. As we’ve grown from mailing floppies, to a wide range of areas areas across the scope of the shipping side of the development cycle: front-end, back-end, integration testing, the cloud, IoT, ops (monitoring), and a million other areas that I’m probably forgetting.

Look at the number of tools available for ONE area of the software development lifecycle alone. This is not even for writing any of the code, this is for devops, the meta-control of the code where you monitor how code runs. 

In the earlier days, developers could afford themselves a lot of depth, because there was less to know. You could sit and think about some particularly thorny C code for a while, because you didn’t also need to be learning React, Kubernetes, Tensorflow, GitHub, GitLab, and whatever the latest announcements from aws: Reinvent are. 

Take, for another example, the machine learning landscape these days.  I did a joke-tweet riff off Matt Turk’s yearly posting of the current landscape for ML-related tooling, but, as in every joke, there is always a grain of truth: 

A person coming into a data position cannot be expected to know even a fraction of all these tools, each one of them written in their own language and with their own nuances. 

Chip recently tweeted that she would focus on these areas if she were to start learning MLE again: 

I agree with many of them, but it’s impossible that every machine learning engineer knows all of these areas, or is strong in all of them, or even has a need for all of them at work. 

For example, I’ve never (yet) used Dask or Kubernetes, I’ve only grazed data structures, and I use ML algorithms much less in my work than I’d like to. But out of that list, version control, unit/integration tests, SQL,  Python,and REST APIs make up a great deal of my day-to-day landscape. 

I cannot imagine how overwhelming this list looks to a person starting in industry.  It doesn’t even get into the subtopics of each category. For example, under Machine Learning Algorithms, you could potentially be looking at upwards of 20 or 30 algorithms. I’ve used maybe 4 or 5 in my entire career, repeatedly. 

I’d venture to say that most technical people fall into this realm: knowing a core handful of 7-8 things pretty well. For everything else, there’s Google and O’Reilly.  

So what? 

What used to distinguish senior people from junior people was the depth of knowledge they had about any given programming language and operating system, and the amount of time 

What distinguishes them now is breadth and, I think, the ability to discern patterns and carry them across multiple parts of a stack, multiple stacks, and multiple jobs working in multiple industries. We are all junior, now, in some part of the software stack. The real trick is knowing which part that is. 

Of course, this is my bias as a former consultant coming in, but I truly think that the role of a senior developer now is that of an internal advisor drawing from a mental card catalog of previous incidents and products to offer advice and speculation about how the next one might go, and if they don’t know, what kinds of questions they need to ask to find out. 

If you’ve worked with Kafka before, you’re not going to have any problems with Kinesis, and you’ll find Flink to be similar enough to get going. If you’ve used Ruby, Python’s not going to be a problem. If you’ve used cron and YAML, you’ll enjoy GitHub Actions. If you’ve used Spark, starting with Dask probably won’t be an issue. 

In other worlds, what I think is most important for senior people to have these days is a solid grasp of a couple languages and the fundamentals that transcend those languages. Here’s an example:

Of course, it’s always been the case that software engineering has been as much about puzzling out the logic as writing the actual code. But <movie announcer voice> in a world where we have five million and ten data processing frameworks, </voice> this distinction and ability to reason through why something might work or not becomes even more important. 

In other words, to break down the systems and platforms to their core principles, to their ones and their zeros, and be able to reapply those principles elsewhere, is the new knowing the command line. (Oh, but you should also know the command line, too.)

What I’m reading lately: 

1. I wrote about how I taught Python to beginners a few weeks ago

2. I just finished a biography of Golda Meir; strong recommend.

3. I am also reading, for an escape, The Blue Castle and it’s lovely

4. I also recommend Ted Lasso! It’s a gem of a TV show.

5. Aphyr has a series of posts on the technical interview. I understand almost none of them, but I aspire to this level of technical writing.

6. Excel warriors

7. Why are noses broken on Egyptian statues?

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.

Art: The Hands, Moise Kisling

A few weeks back,  I was shocked when I found something good to watch on Netflix. Due to The Algorithm, usually my husband and I end up scrolling through the beautifully-rendered UI with stunning overlays and meticulous metadata for about 25 minutes, throwing out show names to each other half-heartedly, knowing that we’re just going to end up watching reruns of Parks and Rec or Seinfeld. 

Recently, though, we stumbled onto The Repair Shop, a show based in Britain where a bevvy of craftspeople and experts in making old things work again are based out of a charmingly cozy barn, and help customers with the heirlooms that people bring in. 

Although the cast and crew were assembled specifically for the show, the experts making the repairs truly know their stuff and help bring treasured items back to life. In one episode, an expert in photography restores a camera that’s been through World War I in the trenches to take pictures again. (Side note, do NOT go down the old camera rabbithole.)  In another, a toy train from the 1930s is lovingly restored and again makes real steam. There are mechanics, stained glass experts (do not go down this rabbit hole, either), a woman focused entirely on leather repair, a luthier, and on and on. 

For some reason, the show made me extremely emotional. It could have been the late hour, the sentimental value attached to the fragile items many of which people have received at critical moments in their lives, or the fact that I finally found a single show worth watching for $8.99 a month. 

Later, when I was at work, I realized what it was. These people had tangible proof of their success. They had things that people love and would hopefully pass on, long after they were gone.  With the rise of minimalism, throw-away culture,  and Marie Kondo running her anti-consumerist hustle,  encouraging everyone to let go of things, it’s easy to lose sight of the fact that sometimes, collecting things, within reason, can actually be good. And lovingly maintaining things is even better. 

In my house, tucked away in a kitchen drawer, is a set of two oblong, sleek, white porcelain serving platters. Ten years ago, when my husband and I bought our house, a friend,  A, brought them to our housewarming. Although she and my husband worked together,  A was more than just a colleague. She helped my husband get his first job by giving him hours of guidance on his straight-out-of-college resume, giving him interview prep advice, and then, once he got a job, helping him navigate both the code and workplace politics. She was a mentor and a gem of a human being. 

When we had our housewarming, she brought these two serving trays as presents. As I remember now, she had made dozens of little perfect crostini sandwiches, arranged them tastefully on the platters, and wrapped them in cling wrap. “Keep these, you’ll need them,” she said. I took the platters, thinking nothing more of it.  Years passed, and we fell out of touch with A, although we visited her a few times. More years passed, and then, suddenly, we found out that she had been diagnosed with cancer, and then, even more suddenly, my husband was attending her funeral. 

I use these platters every summer, from May to September, to cut up and serve watermelon, and every single time I take them out of their resting place in the cupboard and gently slice the red, tender meat into triangles, I think of A and what a good, kind woman she was, and about how she came into our back yard, hands full of crostini, laughing. 

Physical things are good. They connect us to people, to memories, to places. They give our world vibrancy and depth. 

In software, we don’t have the satisfaction of fixing a a physical thing and giving it to another person. We are constantly dealing with bits and bytes that traverse the ether, coded abstractions that move out from under us. We’re always deleting code, changing it, updating systems, changing the tools that we work with, in the hopes that, eventually, we’ll impact something physical, and touch someone. 

There are often threads on Hacker News and other places about software developers proclaiming they’ve quit development to work the land or start a bespoke woodworking shop and I often joke about them, but in them, there is a grain of truth. 

Maybe one of the reasons these platforms of ours are completely out of control is that we’ve lost our connection to the physical manifestation of what it is that we’re creating.  We’ve lost touch with the person on the other side. One of the reasons places like Facebook are struggling (aside from the fact that Zuck owns it and can do whatever he wants,) is that it is truly impossible for anyone to, at scale, understand the physical impact of the code they’re pushing, how the bits get transferred into data, how data moves into the UI/UX, and how the small function that one person writes is transformed into a Like button that makes someone feel excluded or included, angry or elated, sad or thrilled. 

We never get the satisfaction of hand-off of our work in the way that craftspeople do, and I used to not think it was a big deal. But now I’m not so sure. 

One of the hardest and most rewarding things I’ve done in my work with bits recently, as part of my onboarding at Automattic, was participated in a support rotation (which everyone in the company does when they first start.) For two weeks, I helped real customers struggling with real issues on their websites, both in tickets and live chat.  Talking to real people about their problems, frustrations, and finally, happiness and relief once their issues were resolved was fantastic, and probably the closest to the same kind of feeling that the Repair Shop people get every day. 

The closer you get to the back-end of any system, the servers, the databases, the fuzzy API layers that move data between various points of those systems, the Kafkas, and the like, the harder it can be to see the forest for the trees, the reason your site or company actually exists. 

But it was in this way that I could see what the bits I’m working on are actually doing. 

And maybe we don’t have to all delete Facebook, lawyer up, hit the gym, and start a woodworking shop, but it for sure would be nice to have to do something to better understand the impact our bits have on our physical world. 

What I’m reading:

1. Interview with Daniel Ek, CEO of Spotify

2. Phone homescreens of doctors and nurses

3. About engineering levels

4. Soviet comedies

5. What is the single best book for an introduction to your field?

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once or twice a week. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:

I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.