If you’d like to undertand why we moved, aside from our belief in the benefits of decentralized architectures, you won’t find a better summary of the problems at Substack than this post by first amendment expert Ken White:

The Popehat Report

Substack Has A Nazi Opportunity

Substack has Nazis, because of course it does. Substack is on the internet, Nazis are on the internet, and if Substack doesn’t want Nazis it has to take affirmative steps to get rid of them. Flies don’t stop coming into the house because you want them to; they stop because you get off the couch and close the screen door. Any social media or blogging…

Read more

2 years ago · 1355 likes · 513 comments · Ken White

Hopefully this all goes smoothly and your subscription will be uninterrupted, but you can always cancel here and sign up again at our new site if necessary.

As a pleasant side effect, our main website and newsletter now live in the same place, so we actually dropped another centralized hosting provider out of our system as part of this process. And we know that by choosing Wordpress we are not in any way locked into any one hosting provider. Now that the basics are under control, we will start playing around with the Fediverse integration that Automattic is developing.

As people who write a lot (more than average for computer scientists, I would guess) we also spend a lot of time thinking–and reading–about language. One of the best pieces I’ve ever read on language is from David Foster Wallace, and that inspired my take on the “On Premises vs On Premise” debate. As in much of my career, I managed to irritate plenty of people by taking a position between the two extremes, but mostly I wanted to inspire people to read the essay by Foster Wallace. He has a bit of a reputation (undeserved in my view) of being hard to read–perhaps due in part to his extensive reliance on footnotes–but I recommend that if you want to get a gentle introduction to his work, you could start at the essay noted above or “A Supposedly Fun Thing I'll Never Do Again”. 

Larry has also written about his literary influences. These include his Hemingway reference a few weeks back, his glowing review of Tracy Kidder’s House, and his discussion of John McPhee’s “Draft No. 4: Reflections on The Writing Process”. 

Given our interest in language, it should come as no surprise that we are intrigued by large language models. I first started studying AI in 1984–yes, really–and I recently happened to crack open the book by Patrick Winston from that year, titled simply “Artificial Intelligence”. The preface is amazing in that it would look perfectly timely in a book published today: 

The field of Artificial Intelligence has changed enormously since the first edition of this book was published. Subjects in Artificial Intelligence are de rigueur for undergraduate computer-science majors, and stories on Artificial Intelligence are regularly featured in most of the reputable news magazines. 

Ah yes, our brilliant AI future was just around the corner in 1984, just as it is today, and the media was all over it.

I think my piece on “looking inside LLMs” is the one I spent the most time researching this year, and that’s because the picture is complex. While the view of LLMs as “stochastic parrots” (or “spicy autocomplete”) is not far from my own, getting a closer look at how LLMs work internally gave me a better understanding of why there is so much confusion about what is possible, as well as optimism from many who work in the field. It is just unfortunate that there is so much hype and shallow thinking getting in the way of properly understanding the potential and current risks of these systems. We plan to produce a book about machine learning (as applied in the context of networking) next year, and it will be as hype-free as we can make it.

A lot of what we have written over the last two years has been about the increase in centralization of various aspects of the Internet. My own experience with SDN was one of leveraging the capabilities of modern distributed systems to create logically centralized abstractions as a way to overcome the limitations of fully decentralized network architectures. I talked about this with Nicira founder Martin Casado, and wrote about it in this post and in our SDN book. So I’m not entirely anti-centralization, but the tradeoffs are subtle. 

It’s now more than a year since we moved to Mastodon, and that’s very much a story about the renaissance of decentralization in the social media sphere. Hardly a day goes by that we don’t feel good about the decision to leave the bird site. And in a reminder that there is more to decentralized social media than Mastodon, here is my talk on 60 Years of Networking (with a strong focus on decentralization) on peertube. 

In terms of the articles from our early days that have seen the greatest interest, I would say that every time we talk about RPC vs TCP we get lots of traffic. So too with anything questioning the value of MPLS. The reaction to the last newsletter on Outrageous Opinions makes me think I might need to write something about the rise and fall of ATM. 

What else should you read over the holidays? I enjoyed this article on Indexes from the New York Review of Books. For a literary magazine, the NYRB also does a nice job on covering issues of the Internet and society, as in this piece on the Internet and privacy. (The NYRB, by the way, is not related to other publications with New York in their title, and was founded in 1963 during a newspaper strike.)

We’re big fans of Cory Doctorow, whose rate of publication puts us to shame. His recent post on the AI bubble is a good example of his work, and he may well have given us 2023’s word of the year: “enshittification”. A good place to start on this topic is here. 

My non-fiction reading for the holidays includes Chernobyl: History of a Tragedy, which I was inspired to pick up after watching the HBO series on Chernobyl. I thought I had a basic understanding of how nuclear reactors work, but I had massively underestimated just how complex they are. This book helps expose that complexity and the fatal combination of technical and human failures that followed. 

Finally, I made a concerted effort to read more novels this year, and one book deserves mention here: The Private Memoirs and Confessions of a Justified Sinner. I think I can safely say that it’s the funniest 19th century novel I’ve read, and surprisingly timely. Also Scottish, which is how I came to hear about it. Thanks to Project Gutenberg–a project aligned with our goals as open source publishers–for making it available.

I had an exchange with Paul Francis about his work on distributed search after I quoted him in last week’s newsletter. My enthusiasm for his work was perhaps a bit misplaced, if I take Paul at face value: 

Regarding Ingrid, distributed search could never work. First it would never be fast enough (or as fast as centralized alternatives). Even if you could solve this, it'd be hard as hell to deal with spam in a distributed system...Exploring the distributed alternative is the prerogative of the researcher, but it was a stupid idea. 

Well, it was a good talk in any case. 

Thanks for supporting us this year and we will return with fresh content in 2024.

Got an idea for something that Systems Approach should cover in either a book or a newsletter? Send us a note. 

Follow us on Mastodon.

One of my most lasting contributions to the SIGCOMM community would have to be my 2003 Outrageous Opinion talk “MPLS Considered Helpful”. Twenty years later I still run into people who remember the punchline “I’m not bitter”. I suspect this will be remembered much longer than my four-year tenure as SIGCOMM Chair. 

A less well-known fact is that I chaired the first SIGCOMM outrageous opinion session at SIGCOMM 1995. (Another fun fact from that conference: Marc Andreesen was scheduled to give a tutorial at the same time as me, but he bowed out at the last minute, citing business pressures–the Netscape IPO had taken place 2 weeks earlier. I picked up a bunch of disappointed attendees at my tutorial on SmartNICs as a result.)

While outrageous opinion sessions subsequently took a turn towards stand-up comedy, in that first year there were a bunch of quite serious (and nevertheless funny) talks that have stuck with me. I often refer back to David Clark’s talk “We Should All Become Economists”. Already well established as “the architect of the Internet”, David had started to branch out into areas of economics and policy, with notable works including “Realizing the Information Future”, which made the case to a broad audience that the Internet was the appropriate architecture for the “Information Superhighway”. This might seem an obvious truth today, but at the time there was plenty of support for alternative architectures based around evolved versions of either the telephone network (“Broadband ISDN”) or the Cable TV network (“imagine 500 channels!”). His later paper with Marjorie Blumenthal “Rethinking the Design of the Internet: End-to-End Arguments vs. the Brave New World” is a wonderful examination of the tension between the idealized architecture of the Internet and the commercial pressures that came to bear on it once it became a mainstream communications platform. While there are few things more annoying than tech people discussing economics from a position of ignorance, David was making the case that we should do more to educate ourselves about economics, and I wish more of us had taken his advice.

Equally memorable was a talk by Paul Francis, another Internet pioneer whose influence spans topics as diverse as scalable multicast, IPv6, and Distributed Hash Tables. At that point we were in the early days of Internet search–Google was still three years away from being founded, and Alta Vista was an internal project to index the entire Web at DEC (released to the world later that year). Paul approached me with two topics he was interested in presenting: his latest research on scalable Internet search, which he had named Ingrid, and Network Address Translation (NAT). I was rather more favorable towards the latter, because it was a very hot topic at the time, Paul was credited with its invention, and he had an amusing way to present his argument. Creative person that he is, Paul found a way to weave the two topics together in one talk. The gist of the NAT part of his talk was that the prevailing IETF view on NAT at the time–don’t do NAT–should be viewed as analogous to abstinence-only sex education. No matter how much the people giving the advice might believe in the correctness of their position, they were not going to have much impact on the outcome. In hindsight, he was obviously correct: most of the world’s Internet users today sit behind one or more NAT devices, but his position was much more controversial at the time. In the end, living with NAT became more important than preventing it, and today there is a whole body of work on NAT traversal that allows us to deal with it fairly painlessly. 

I have had reason to refer back to the other part of Paul’s talk more often, because of the way it resonates today as we deal with the centralization of the Internet and the recent efforts to re-decentralize it. While early search engines were centralized–for example, Alta Vista was apparently developed to show off the capabilities of a powerful DEC database server–Paul argued that a decentralized approach to search was going to be necessary as the Web took off. (Recall this was the same year as the Netcape IPO.) Again, he was right, but Google would eventually deliver a hugely successful distributed system to index and search the Web, and put it behind a logically centralized front end. So while the technical solution was indeed decentralized, the user experience is centralized: just go to Google.com and ask your question. And of course that is where the Web mostly sits today, 25 years later: distributed systems do the work under the covers but the average user interacts with a handful of centralized entities, such as the social media titans and streaming services. I’m cautiously optimistic that we are seeing a reversal of this trend, especially with federated social media, but I do look back nostalgically on the era when it seemed possible that search itself might be decentralized. Since it is now increasingly apparent that search is getting worse, we can still hope. Also, the idea that decentralized technologies alone do not protect us from the perils of centralized control of technology (e.g., the ownership of search or social media by a small number of companies) is something that I developed further in my “60 Years of Networking” talk earlier this year.

Finally, I can remember that there were multiple talks about the relative merits of ATM and IP. This seems hard to fathom today, where the Internet reaches something like half the world’s population and ATM is little more than a historical footnote in the large collection of layer 2 technologies that IP has accommodated. As my comments about “Realizing the Information Future” above suggest, it was far from clear in 1995 that this is how things would play out. I was placing an each-way bet at this point, having worked on ATM at Bellcore (owned by telcos) but believing by 1995 that ATM would be adopted as a substrate for part of the Internet, not a standalone networking technology that would replace IP. Indeed, it was my focus on IP-over-ATM that provided me with the opportunity to join Cisco later that year as they increased their investment in ATM switching. That would ultimately land me in the team that developed MPLS–another technology that, like NAT, faced a fair amount of opposition at the IETF, but is widely deployed today.  

The fact that I remember so much of that one evening in 1995 (whereas I remember almost nothing of the tutorial I gave) has a lot to do with how provocative, forward-looking, and accurate many of the opinions were. I think some of the talks in later years may have been funnier, but the sheer amount of predictive accuracy of those I remember from 1995 is striking. (No doubt I’ve forgotten some talks that were totally off base.) David Clark’s comments about economists continue to resonate as we argue about network neutrality and hear demands from telcos that they be compensated not only by their customers but by the content providers. The increased centralization of the Internet is also partly an economic (winner-takes-all) phenomenon. In the current debates about the future of the SIGCOMM conference, and more broadly as we look to shape the Internet of the future, I hope we keep a place for outrageous opinions. 

While the next issue of this newsletter is due to land on Christmas Day, we’ll be taking some time off for the holidays (some of us have running camps to get to) so expect a short newsletter in roughly two weeks time and we’ll return to our normal schedule in January. We found this article on the energy cost of generative AI worth a read–the headline isn’t as interesting as the observation that generative models are wasteful compared to more specialized models in many cases. The departure of Twitch from South Korea is the latest twist in arguments about who should pay for network traffic. And Bruce Schneier’s article on AI and privacy makes a well-reasoned case for regulation of how AI is used for surveillance. To support decentralized social media you should follow us on Mastodon.

Books make great gifts!

In our last newsletter, Bruce gave a retrospective of the last three years of our Systems Approach journey. This week I thought I’d chime in with my thoughts on how it is we decide what to write about. In many ways, this is no different than a researcher picking a problem to work on, a programmer deciding to build a new system or app, or a novelist deciding their next book project. It's mostly about personal tastes, but also includes an element of opportunity (e.g., to leverage some unique experience).

You can see these elements in the First Edition of our Computer Networks book. I had found the organizing principles of networking—along with the practice of building network software—to be a rewarding research avenue. Seeing Tanenbaum’s book as the current (in 1994) state-of-the-art in network textbooks, I realized that there was an opportunity to have impact by putting the Internet architecture front and center. The leverage was that I had spent time working on the TCP/IP stack (and adding RPC to the mix), so I had a strong sense of how everything worked in practice. It was only after I got  started that I understood (a) how daunting the task was, and (b) how many blind spots I had about the technology I was trying to write about. Fortunately, Bruce saved the day by joining me as co-author.

That 1st Edition made heavy use of the x-kernel (my research project at the time) to illustrate how various protocols are implemented. That turned out to be a bootstrapping process: it provided the grounding that made it possible to speak with some authority about network systems, but (virtually) no one wants to learn about some obscure research OS as a prerequisite for learning about the Internet. The x-kernel all but disappeared in later editions, but the experience did teach me a valuable lesson, which I’ve come to think of as my version of the Hemingway Iceberg Theory:

Hemingway said that only the tip of the iceberg showed in fiction—your reader will see only what is above the water—but the knowledge that you have about your character that never makes it into the story acts as the bulk of the iceberg. And that is what gives your story weight and gravitas. (Jenna Blum, 2013)

(I also appreciate Hemingway’s writing style, which shares much with technical writing: short/direct sentences and short/focused paragraphs. But I attribute the short-paragraph rule to Ms. Penny Wika, my high school journalism teacher, who insisted that the lead paragraph of any story be no more than 25 words, and never, ever start with “The” or “There”. I often break both rules, but doing so always weighs on me.)

I should bring this back to writing textbooks, but before I do, I can’t help but observe that even technical writing, while unlikely to be mistaken for a novel, can still include a narrative. In the books Bruce and I write, the narrative is just as important as the factual details. You can find the latter anywhere; it’s the former that helps you understand how to think about that information. In research papers the narrative is heavily prescribed—in systems, for example, it’s: (1) Introduction, (2) Design, (3) Implementation, (4) Performance, (5) Related Work. With textbooks you get a little more leeway to craft the narrative, and weave multiple themes through the text.

The key to a narrative is part organizational structure and part knowing what to omit, the latter being a challenge I elaborated on in an earlier post. It turns out the Iceberg Theory and knowing what to omit are two sides of the same coin.  Having a real system underpinning a book—and being selective in how much detail you expose—is critical to constructing the narrative. A real system provides the depth you need to talk about abstract concepts, and it helps fill the gaps between the architectural elements. Both are essential if you’re going to try to construct an end-to-end narrative for the reader.

Books Series

Since starting our Systems Approach endeavor three years ago, we’ve produced four books on relatively narrow topics. There’s a unique case for why we selected each of these specific topics, but what they have in common is that each was (1) co-authored with a domain expert, and (2) backed by open source implementation. I’ve come to believe both are essential: you not only need to have someone in the know to ask questions of, but you also need to know what questions to ask (and trying to understand code is a great way to fuel the latter).

Starting with the most obvious choice of topic, our SDN book is a natural continuation of our earlier interests: it describes an approach to implementing network software. Coupled with Bruce’s and my being in the thick of the industry building out SDN products and platforms (at Nicira/VMware and ONF), this made SDN an obvious candidate to write about. Having up-close access to a complete software stack—from programmable data planes, to network operating systems, to a range of control applications—gave us a unique perspective. No matter where you come down on the commercial viability of SDN, the kind of exposure the book gives you to the construction of network software is, in my judgment, essential to understanding networks.

Our Edge Cloud Operations book is probably the least obvious choice in the series, especially if you’re expecting Internet-focused topics. But my experience working with network operators convinced me of two things: (1) that the Internet is rapidly evolving into a set of cloud services, and (2) being able to operate these services is the chief challenge facing the industry. That’s still a tough sell in academic settings (where cloud operations is just the latest incarnation of network management, another neglected topic), but that doesn’t—again in my judgement—make it any less true. It also happens to be a rapidly evolving space, with countless offerings (both commercial and open source) vying for mindshare. And quickly stepping into that gap, today’s cloud providers say “Don’t worry about this problem; it’s really hard, so we’ll take care of it for you.” Personally, that is the loudest invitation to write a book that I can imagine.

Our 5G books (the original primer and our more recent Private 5G book) were born out of the frustration of trying to understand the mobile cellular network. Initially this frustration was due to the many acronyms I encountered, but the closer I looked, the more the problem turned out to be one of trying to internalize a system that had evolved within an entirely different framework. There simply was no easy way to map the elements behind those acronyms onto a framework that made any sense to me. In 5G, the mobile cellular network is trying to redefine itself as a cloud service, which does at least provide a shared language (complicated by subtle differences in our definitions for terminology in that language). Sorting all of that out is essentially what our Private 5G book is about.

Finally, our TCP Congestion Control book is different from the other three, each of which could be characterized as covering an emerging topic. In contrast, TCP congestion control goes back over 30 years. But it is the one hard problem in networking that continues to demand attention even as the overall landscape changes. This makes it worth understanding.

Whether these four topics turned out to be the most useful to write about—or just satisfied our own personal interests—is difficult to say. We can say that our books are among the top non-sponsored search results for their respective topics and the online versions collectively received thousands of visits each week. That’s gratifying, and helps motivate us to look for the next itch to scratch.

Our Systems Approach books are always freely available on GitHub and as web versions, but we also make them available via print-on-demand and as eBooks. For the latter, we occasionally make a second printing available when the amount of new content becomes large enough, and last week we reached that milestone for our Private 5G book with an expanded Appendix on deploying 5G using Aether. It’s currently discounted at both our site and Amazon.

The massive outage at Australian telco Optus continues to be analyzed and led to the resignation of the CEO. The Optus submission to the Australian government was notable for its catalog of large failures by other telcos, a classic “everyone does it” defence. And you probably don’t need us to tell you about what happened at OpenAI last week, but Gergely Orosz at The Pragmatic Engineer has a good overview.

Image this week by 66 north on Unsplash.

As many of our readers will know, Larry and I have been writing books together since 1995, and the Systems Approach has been our guiding principle from day one. When Larry first approached me to come on board as a co-author, he already had the title picked out and a contract in place with Morgan Kaufmann Publishers. MKP in those days was a small, CS-focussed press whose main claim to fame was publishing “Computer Architecture: A Quantitative Approach” by Hennessy and Patterson. That book had already achieved legendary status and a huge chunk of the computer architecture market, so our goal was to be as much like them as possible. That included having a paper dust-jacket in a shade of beige and a similar structure to our title: hence “Computer Networks: A Systems Approach”. 

To be honest, I needed Larry to explain what he meant by “Systems Approach” even though I’d been working on systems research for quite some time. As we say in the book’s preface, the key to the Systems Approach is the “big-picture” view: you don’t get to optimize inside a single box without thinking about the bigger interactions among components. For me, one of the pivotal moments that turned me into a “systems thinker” was hearing David Clark present the ideas of Application Layer Framing at the 1990 SIGCOMM conference, which made me realize that you shouldn’t just optimize a single layer of the protocol stack without thinking about how other layers depend on the services of that layer. In recent years, the rethinking of layers that is apparent in QUIC is a good example of how systems thinking is applied in protocol design–and also a reminder that rigid layering is often not the best way to think about networking.

The big change for us three years ago was that I left the corporate world (in a so-called “retirement” from VMware) while Larry moved to part-time at ONF and Princeton, so the work of Systems Approach LLC became our “main job”. Coincidentally, the publisher of our original book from 1995 had just pushed us to produce a sixth edition and so our first order of business was to complete the edits and handle all the back-end processing that happens when doing a book with a big publisher. The content edits were largely in place thanks to the open source nature of the book (we’d been taking inputs on GitHub since the fifth edition). One of the few valuable tasks performed by a big publisher is a professional copy edit, which takes a fair bit of time to review. So we were busy enough in those first few months putting the last finishing touches on the book. We also undertook a complete update to the ancillary materials for teachers (lesson slides and solutions to the exercises). Experience has shown that instructors struggle to get access to these materials through the publisher, so do reach out to us if you are looking for them. 

Our main vision for Systems Approach, however, was not to keep on working for Big Publisher but to produce our own books. So back in 2020 we drew up a list of possible topics to cover in the Systems Approach book series and in the last three years we have chipped away at that list, occasionally adding to it and sometimes shelving topics as they lose appeal. (I’m pretty sure a book on blockchains is off the table now.) Here are the books that we have managed to complete in the last three years: Software-defined Networking, TCP Congestion Control, Edge Cloud Operations, and Private 5G. Add in the revision of the big textbook and that’s five books in three years, which feels like a decent amount of progress. On top of that, we’ve created an online class about Magma, the open source mobile core project. 

We are both fans of physical books, and we are benefiting from the increased quality and cost-effectiveness of print-on-demand that enables us to make print books without a big publisher. There is something very satisfying about seeing your work in a printed volume, and so we make the extra effort to get our books nicely formatted and ready for printing once we are fairly happy with the content. We’ve also worked with translators to produce foreign language versions of several of our books. A particular highlight was working with three of our Japanese colleagues to expand and customize our SDN book for the Japanese market. 

Producing physical books turns out to be something that we really enjoy. It is admittedly a bit annoying to get all the page breaks in the right place and wrestle with LaTeX to make everything look nice, but in the end we believe we’re producing a professional and aesthetically pleasing product. I also enjoy hunting around for the ideal cover art; we’ve made extensive use of the public domain images on Unsplash. We very nearly ended up with a picture of a marijuana farm on the cover of the 5G book–which would have been fine, I guess, but we opted for something less potentially controversial.

Much of what we have written draws on our personal experience (especially Larry’s–he’s really quite good at building systems and then writing about them). Three of the above books are based on experience with open source projects such as Aether and other SDN projects of the ONF. We also leveraged my experience with the open source projects Open vSwitch and Magma. In our ideal world, every book we write would have a companion set of open source software so that students could get hands-on experience with whatever it is we are writing about. We’ve met that goal for all the books to date aside from TCP Congestion Control.  Of course, there is open source code for congestion control, but it’s mostly in the Linux kernel and thus not ideal for our instructional purposes.  As Larry pointed out in his post about Aether OnRamp, there is more to making a technology accessible than open source software.

Because we make all our books freely available online, we’re not expecting to make much money from selling physical books and ebooks, and we are totally meeting those expectations! We’ve discussed the rationale behind this choice before–we’re more interested in impact than revenue. But of course it’s much easier to measure revenue. Once upon a time our publisher even measured course adoption rates for us (ah, those halcyon days of working with a small publisher). Today we are measuring things like the number of readers of this newsletter (growing  nicely, thank you–tell your friends) and visits to our web sites (also doing well, although maybe a bit less well as Google search deteriorates). We definitely appreciate direct feedback from our readers. We even have our favorite Amazon one-star review (called “Wall of Text”) that we use as inspiration to write more concisely and to draw more pictures. 

The open source model is working well in my view. We’ve received dozens of corrections, issues, and pull requests from around the world, and we try to thank every contributor in the preface of each book. Sometimes it’s corrections to our grammar, sometimes it’s a bigger error like mixing up big- and little-endian (which was wrong in every edition of our big book until last month). Our books are better because we adopted this model. And occasionally one of us gets the satisfaction of resolving a merge conflict using git so we feel smart for a day.

At this point we have a number of books in the pipeline. Network security feels like a topic that we should cover, but will take some time given our relatively limited first-hand experience. There is part of me that wants to tackle quantum computing but that would definitely be a stretch. Machine learning is clearly close to our hearts and its application to networking is a topic that we plan to cover soon. We find that writing these books (and this newsletter) keeps us hungry to understand new technologies well enough to explain them to others, especially when there is a real system to underpin our learning.

Got an idea for something that Systems Approach should cover in either a book or a newsletter? Send us a note. 

In our periodic coverage of broken networks, Australian Telco Optus made a solid contribution last week with an outage lasting over eight hours. The best analysis we’ve seen so far comes from Mastodon user Rob Thomas who used routing advertisements to deduce the root cause (suspected to be route reflector upgrades gone wrong). Optus has said little of substance other than to claim such outages are not unusual (!).

Speaking of Mastodon, we will soon celebrate our first anniversary of leaving the dead bird site. Follow us here.

Perhaps the single biggest aspect of systems building I’ve come to appreciate since shifting my focus from academic pursuits to open source software development is the importance of testing and test automation. In academia, it’s not much of an overstatement to say that we teach students about testing only insofar as we need test cases to evaluate their solutions, and we have our grad students run performance benchmarks to collect quantitative data for our research papers, but that’s pretty much it. There are certainly exceptions (e.g., Software Engineering focused curricula), but my experience is that the importance placed on testing in academia is misaligned with its importance in practice.

I said I appreciate the role of software testing, but I’m not sure that I understand it with enough clarity and depth to explain it to anyone else. As is the nature of our Systems Approach mindset, I’d like to have a better understanding of the “whys”, but mostly what I see and hear is a lot of jargon: unit tests, smoke tests, soak tests, regression tests, integration tests, and so on. The problem I have with this and similar terminology is that it’s more descriptive than prescriptive. Surely an integration test (for example) is a good thing, and I can see why you could reasonably claim that a particular test is an integration test, but I’m not sure I understand why it’s either necessary or sufficient in the grand scheme of things. (If that example is too obscure, here’s another example posted by a Reddit user.) The exception might be unit tests, where code coverage is a quantifiable metric, but even then, my experience is that more value is being put on the ability to measure progress than its actual contribution to producing quality code.

With that backdrop, I have recently found myself trying to perform triage on the 700+ QA jobs (incurring substantial monthly AWS charges) that have accumulated over the last five years on the Aether project. I don’t think the specific functionality is particularly important—Aether consists of four microservice-based subsystems, each deployed as a Kubernetes workload on an edge cloud—although it probably is relevant that the subsystems are managed as independent open source projects, each with its own team of developers. The projects do, however, share common tools (e.g., Jenkins) and feed into the same CI/CD pipeline, making it fairly representative of the practice of building systems from the integration of multiple upstream sources.

What is clear from my “case study” is that there are non-trivial tradeoffs involved, with competing requirements pulling in different directions. One is the tension between feature velocity and code quality, and that’s where test automation plays a key role: providing the tools to help engineering teams deliver both. The best practice (which Aether adopts) is the so-called Shift Left strategy: introducing tests as early as possible in the development cycle (i.e., towards the “left” end of CI/CD pipeline). But Shift Left is easier in theory than in practice because testing comes at a cost, both in time (developers waiting for tests to run) and resources (virtual and physical machines needed to run the tests). 

What Happens in Practice?

In practice, what I’ve seen is heavy dependency on developers manually running component-level functional tests. These are the tests most people think of when they think of testing (and when they post jokes about testing to Reddit), with independent QA engineers providing value by looking for issues that developers miss, yet still failing to anticipate critical edge cases. In the case of Aether, one of the key functional tests exercises how well developers have implemented the 3GPP protocol spec, a task so complex that the tests are commonly purchased from a third-party vendor. As for automated testing, the CI/CD pipeline performs mostly pro forma tests (e.g., does it build, does it have the appropriate copyright notice, has the developer signed the CLA) as the gate to merging a patch into the code base.

That puts a heavy burden on post-merge integration tests, where the key issue is to ensure sufficient “configuration coverage”, that is, validating that the independently developed subsystems are configured in a way that represents how they will be deployed as a coherent whole. Unit coverage is straightforward; whole-system coverage is not. For me, the realization that configuration management and testing efficacy are deeply intertwined is the key insight. (It is also why automating the CI/CD pipeline is so critical).

To make this a little more tangible, let me use a specific example from Aether (which I don’t think is unique). To test a new feature—such as the ability to run multiple User Plane Functions (UPF), each serving a different partition (Slice) of wireless devices—it is necessary to deploy a combination of (a) the Mobile Core that implements the UPF, (b) the Runtime Controller that binds devices to UPF instances, and (c) a workload generator that sends meaningful traffic through each UPF. Each of the three components comes with its own “config file”, which the integration test has to align in a way that yields an end-to-end result. In a loosely coupled cloud-based system like Aether, integration equals coordinated configuration.

Now imagine doing that for each new feature that gets introduced, and either the number of unique configurations explodes, or you figure out how to get adequate feature converge by selectively deciding which combinations to test and which to not test. I don’t have a good answer as to how to do this, but I do know that it requires both insider knowledge and judgment. Experience also shows that many bugs will only be discovered through usage, which says to me that pre-release testing and post-release observability are closely related. Treating release management (including staged deployments) as just another stage of the testing strategy is a sensible holistic approach.

Going back to where I started—trying to understand software testing through the systems lens—I don’t think I’ve satisfied my personal acceptance criteria. There are a handful of design principles to work with, but the task still feels like equal parts art and engineering. As for the triage I set out to perform on the set of Aether QA jobs, I’m still struggling to separate the wheat from the chaff. That’s a natural consequence of an evolving system, without a clear plan to disable obsolete tests. This remains a work-in-progress, but one clear takeaway is that both students and practitioners would be well-served by having a rigorous foundation in software testing.

With energy costs comprising up to 10% of their operating costs, it should be no surprise that network operators are looking for ways to save energy. The SMaRT-5G Project at ONF has recently published a whitepaper laying out a broad research agenda to address the challenge.

Preview photo this week by Bruce Davie

I have a vague memory of hearing about server virtualization for the first time in the early 2000s when I was working at Cisco. Most of my knowledge of operating systems had been picked up early in my career, so I had a working knowledge of concepts like virtual memory, but I was pretty surprised when I learned just how popular virtual machines had become. One thing I wondered was why virtual machines had become the de facto means to isolate applications rather than making use of process isolation capabilities of a single operating system. I wouldn’t get a satisfactory answer to that question until I joined Nicira many years later (more on that in a moment).

Even more perplexing to a networking person was to hear that the prevalence of virtual machines in data centers was driving a push towards big, flat layer-2 networks. One of the most remarkable (to me) consequences of virtualization is virtual machine migration: because a virtual machine is completely decoupled from the physical hardware on which it runs, it can be picked up and moved to another physical host without interruption. VMware’s version of live migration, vMotion, was released in 2003 to considerable acclaim, as it allowed a VM to be moved across a data center without interruption to the applications running on it. But there was an unfortunate networking-related side effect to VM migration: VMs retained their IP addresses as they moved. And this is what led to the push to build big, flat L2 networks in modern datacenters: so VMs could move around without finding themselves on a subnet that didn’t match their IP address. 

By this time, I was convinced that scalable L2 networks were something of an oxymoron, so my first reaction was to argue that VMs should simply change their address to match the destination subnet when they moved across the DC. But that just illustrated what I didn’t know about real-world applications in datacenters. The consequences of an IP address change range from a dropped TCP connection to complete breakage of an application in the case where it has a builtin assumption of L2-adjacency to some other component. From the perspective of a datacenter operator, you simply can’t expect the application to respond correctly if the IP address is changing underneath it.

Fortunately there are alternatives to attempting to build datacenter-scale L2 networks. One of the seminal efforts to tackle the problem is VL2, described in 2009 by Greenberg et al. While this was not the first paper to use the term network virtualization, it did (I think) introduce the term in the way that it is most widely used today. Interestingly, Albert Greenberg (a SIGCOMM award winner) led the team that developed Azure’s network virtualization system, while his co-author James Hamilton later led the networking team at AWS. VL2 gives 

each service the illusion that all the servers assigned to it, and only those servers, are connected by a single non-interfering Ethernet switch—a Virtual Layer 2— and maintain this illusion even as the size of each service varies from 1 server to 10,000.

Too often when I read something about network virtualization, it turns out to just be some way of partitioning network resources among different users (also known as slicing). But the key word in the above quote is “illusion” because it gets to the heart of virtualization: creating an illusion of something that, strictly speaking, doesn’t exist. As the seminal 1974 paper on virtualization puts it, these illusions are “efficient, isolated duplicates” of the physical thing being virtualized. Virtual memory gives processes the illusion of a massive amount of address space, generally much larger than what is physically present, completely available to that process and protected from other processes. Virtual machines create the illusion of a complete set of computing resources (CPU, memory, disk, I/O) that are so fully independent of the underlying physical machine that they can be moved across a datacenter (or further). And virtual networks create the illusion of a private switched Ethernet (in the case of VL2) that can span the datacenter, even when the datacenter is a Layer 3 network built out of routers interconnecting many subnets. So while partitioning of resources is part of the story, it’s just one element in service of creating these illusions. 

By the time I started working at Nicira in 2012, the team had settled on this view of network virtualization that mirrored the success of server virtualization. The idea of virtualization creating an illusion of something was core to the vision: just as a virtual machine creates the illusion of a physical machine so faithfully that an unmodified operating system and its applications can run on it–even as it migrates from one piece of hardware to another–so too, a virtual network should perfectly replicate the features of a physical network, while remaining independent of the actual underlying hardware. Just like servers, networks are complicated things with lots of moving parts, so Nicira’s product needed to do a lot more than what VL2 did: not just creating a virtual layer 2 switch, but virtualizing every layer of the network. That meant (eventually) virtual layer 3 routing, virtual firewalls, virtual load balancers and so on. I used to chuckle to myself about the prospect of a 50-person engineering team managing to recreate in software all the networking capabilities that had been developed over the previous several decades by companies like F5, Checkpoint and Cisco, but that is pretty much what eventually happened (thanks to the injection of considerable engineering resources over the following years).

You can find descriptions of network virtualization as implemented at Nicira, Google, and Azure. It’s a bit harder to get details on how AWS does it but a VPC is a form of virtual network; here is some of the information as presented by James Hamilton. We also cover network virtualization as a key use case of SDN in our book. And it’s not limited to the datacenters of the hyperscalers; VMware claims that their network virtualization product (following on from the work at Nicira) is used in the datacenters of 91 of the large enterprises making up the Fortune 100.

In some respects, network virtualization has followed the same path as server virtualization and for similar reasons. Nicira founder Martin Casado has talked about how virtualization changes the “laws of physics”: the salient example for server virtualization is live VM migration, but there are others, such as snapshotting and cloning of VMs, made possible by recreating the illusion of a physical machine entirely in software. Not only did network virtualization bring similar capabilities to networking, but it facilitated entirely new ones such as microsegmentation, laying the groundwork for what was arguably the “killer” use case, zero-trust networking. We had a running joke at Nicira about the movie “Inception” (particularly when running virtual networks inside virtual networks). Network virtualization, with its own “laws of physics”, allowed us not only to recreate the capabilities of physical networks but to create new ones.

We’re always interested in understanding how things go wrong with the Internet, and recent weeks have provided plenty of examples.

Cloudflare has an interesting blog about how they came to suffer an outage in part of their DNS infrastucture weeks after the root cause. The DDOS attacks leveraging a feature of HTTP2 have been well explained by both Cloudflare and Google: this feels like a case study for some future version of our book and a cautionary tale about implementing RFCs to the letter of the law. And it turns out that lots of commercial BGP implementations will fail under fuzz testing.

Preview photo this week by Luca Nicoletti on Unsplash

I recently attended SIGCOMM for the first time in many years, and was immediately reminded of the standard salutation when you run into someone at an academic conference: “What are you working on?” I wasn’t prepared with a succinct-yet-meaningful response, but quickly settled on “Democratizing 5G”. It wasn’t the expected research-focused answer, but it did have the quaint advantage of not directly involving ML. It seemed to do the trick, but also got me thinking about how common the word “democratizing” has become in our technical jargon.

I believe the first time I heard the word being used in a technical setting was in 2004, when Mike Freedman and colleagues published Democratizing Content Publication with Coral at the first NSDI. And if the meaning wasn’t already intuitive, the paper gave a definition that nicely captures the spirit of what it means to democratize something technology-related:

CoralCDN replicates content in proportion to the content’s popularity, regardless of the publisher’s resources, in effect democratizing publication.

The Internet (like the printing press centuries before) is widely regarded as playing a central role in the democratization of knowledge—making information readily available among the wider population. CoralCDN focused specifically on the publishing side of the equation, making it easier for the wider population to also disseminate information. That’s now taken for granted with the explosion of various platforms (e.g., blogs, social media) that include built-in content distribution mechanisms, but was novel at the time. It also highlights the idea that there are multiple barriers to lower; in this case, both acquiring and disseminating content. Less obviously, it points to the role “access to resources” plays in the equation. CoralCDN was only able to democratize publication because it ran on a publicly-funded infrastructure, in this case PlanetLab.

My use of the word democratization in the context of 5G focuses more on the underlying network infrastructure—with the intended audience being people who build and operate that infrastructure—than on end-users that benefit from that infrastructure. I tend to think of it as lowering the barrier to innovation, but it also shares much with the idea of Freedom to Tinker, where know-how is essential to crafting good policy. Either way, it’s about broadening the set of people able to participate in technologies that impact our lives.

For 5G, democratization turns out to be a multi-faceted challenge. A necessary condition is access to open source implementations of both the RAN and the Mobile Core, which, thanks to various open source organizations (ONF’s Aether and OAI for example), now exist. But a quick perusal of the Git repos for those and similar projects will immediately convince you that the mere existence of open source software is not sufficient; users also need the wherewithal to deploy and operate the code if they have any ambition to take advantage of it. My experience is that “wherewithal” maps onto a combination of tooling and documentation, with the latter being the “long pole”. The result—our Private 5G book, which includes a tutorial guiding the reader through the OnRamp deployment toolset—is now available.

I’ve written about that topic before, so won’t rehash the challenges here, except to make two observations. The first is that in our haste to implement new functionality, we have created a nearly impenetrable mountain of configuration variables needed to manage that functionality; YAML files layered on top of Jinja2 templates overlaid on still other YAML files layered on top of JSON… I’m sure it’s not the intention to obfuscate the underlying code, but that is the practical effect. It’s almost as though we’ve created a problem only AI will be able to help us solve. The other option is to be a well-resourced company with a team of experienced engineers, but of course that flies in the face of our goal, which is to democratize access to that know-how.

The second observation, which follows from my experience trying to bring up a 5G small cell, is that the mobile cellular technology has operational complexity baked into its design. This makes sense (in a perverse way) when you consider that the technology was defined by MNOs that built businesses around their ability to operate the network on behalf of their subscribers. But this creates additional barriers that need to be lowered if you want to broaden participation. For example, programming a SIM is a required step to establishing a secure 5G connection, but being able to do that in turn requires having the necessary credentials (plus the know-how to correctly specify another few hundred lines of YAML). Fortunately, the broader ecosystem includes players that help on that front, but the main takeaway is that there is much more to the democratization of technology than initially meets the eye.

Why are we doing this?

But that all raises the question—typically following immediately after “What are you working on?”—which is: “Why should I care?” It’s a good question. If you’re going to put the effort into lowering the barrier-to-entry, there ought to be something important on the other side of those barriers. Again, there are a couple of parts to the answer.

The first part is to simply acknowledge that Internet access is going to be dominated by mobile wireless connectivity. Nearly 60% of all web traffic already comes from mobile devices, and that doesn’t yet take into account the tens of billions of IoT devices that are expected to connect to the Internet over the next few years. We’ve spent 40 years building the Internet out of open and accessible technologies, but going forward, democratizing Internet infrastructure is meaningless without also democratizing wireless access. That the mobile cellular industry has been so closed and proprietary for so long makes that goal all the more relevant.

The second part is to zero in on 5G vs other wireless technologies, most obviously Wi-Fi. At the coding and modulation level, Wi-Fi 6 and 5G’s New Radio (NR) are converging on OFDMA. The difference is how the available spectrum is allocated by the two systems, which Bruce and his Magma collaborators discuss in depth in their NSDI paper on Magma. Digging deeper, 5G scheduling includes the ability to dynamically change the size and number of schedulable resource units, including scheduling intervals as short 0.125ms. This opens the door to making fine-grained scheduling decisions that are critical to predictable, low-latency communication. The 5G scheduler also allocates some of the available spectrum to a light-weight over-the-air-interface that is simple enough for IoT devices to implement. These devices are not particularly latency-sensitive or bandwidth-hungry, but they often require long battery lifetimes, and hence, reduced hardware complexity that draws less power.

There’s reason to be skeptical, but as Bruce discusses in a previous post, the application of cloud best-practices to 5G is a game changer. The hype around 5G has certainly gotten out in front of the reality, but it’s only just now the case that 5G is becoming viable in the enterprise (aka Private 5G). For example, the Aether project has only recently been able to certify a commercially available small cell radio, with ubiquitous 5G devices (other than smartphones) still to follow. Until these components become ubiquitous, many of these advantages outlined above will not be realized. 

Bringing this discussion back to the question of what democratization means in the technical world, a personal takeaway is that I now see a direct line between “Democratizing X” and “X: A Systems Approach”. The Systems Approach has always looked at technology through the lens of deployed systems, using real implementations to explain the design decisions. Democratizing access to technologies is an almost inevitable consequence of how we help readers to understand them—by first deconstructing them into their elemental components and then showing how all the pieces are assembled into a coherent whole.

Aether OnRamp is now available, providing what aims to be a relatively painless way to stand up a private 5G deployment. Watch the Techniar Video and learn about the commercial 5G small cell radio that can be deployed with Aether. And for a different approach to democratizing 5G, see the Helium Network.

Following my recent talk and article on “60 years of networking”, which focussed almost entirely on the Internet and ARPANET, I received quite a few comments about various networking technologies that were competing for ascendancy at the same time. These included the OSI stack (anyone remember CLNP and TP4?), the Coloured Book protocols (including the Cambridge Ring), and of course ATM (Asynchronous Transfer Mode) which was actually the first networking protocol on which I worked in depth. It’s hard to fathom now, but in the 1980s I was one of many people who thought that ATM might be the packet switching technology to take over the world. ATM proponents used to refer to existing technologies such as Ethernet and TCP/IP as “legacy” protocols that could, if necessary, be carried over the global ATM network once it was established. One of my fond memories from those days is of Steve Deering (a pioneer of IP networking) boldly (and correctly) stating that ATM would never be successful enough to even be a legacy protocol.

One reason I skipped over these other protocols at the time was simply to save space–it’s a little-known fact that Larry and I aim for brevity, especially since receiving a 1-star review on Amazon that called our book “a wall of text”. But I was also focused on how we got to the Internet of today, where TCP/IP has effectively out-competed other protocol suites to achieve global (or near-global) penetration.  

There are many theories about why TCP/IP was more successful than its contemporaries, and they are not readily testable. Most likely, there were many factors that played into the success of the Internet protocols. But I rate congestion control as one of the key factors that enabled the Internet to progress from moderate to global scale. It is also an interesting study in how the particular architectural choices made in the 1970s proved themselves over the subsequent decades. 

Distributed Resource Management

In David Clark’s paper “The Design Philosophy of the DARPA Internet Protocols”, a stated design goal is “The Internet architecture must permit distributed management of its resources”. There are many different implications of that goal, but the way that Jacobson and Karels first implemented congestion control in TCP is a good example of taking that principle to heart. Their approach also embraces another design goal of the Internet: accommodate many different types of networks. Taken together, these principles pretty much rule out the possibility of any sort of network-based admission control, a sharp contrast to networks such as ATM, which assumed that a request for resources would be made from an end-system to the network before data could flow. Part of the “accommodate many types of networks” philosophy is that you can’t assume that all networks have admission control. Couple that with distributed management of resources and you end up with congestion control being something that end-systems have to handle, which is exactly what Jacobson and Karels did with their initial changes to TCP.

The history of TCP congestion control is long enough to fill a book (and we did) but the work done at Berkeley from 1996 to 1998 casts a long shadow, with Jacobson’s 1988 SIGCOMM paper ranking among the most cited networking papers of all time. Slow-start, AIMD (additive increase, multiplicative decrease), RTT estimation, and the use of packet loss as a congestion signal were all in that paper, laying the groundwork for the following decades of congestion control research. One reason for that paper's influence, I believe, is that the foundation it laid was solid, while it left plenty of room for future improvements–as we see in the continued efforts to improve congestion control today. And the problem is fundamentally hard: we’re trying to get millions of end-systems that have no direct contact with each other to cooperatively share the bandwidth of bottleneck links in some moderately fair way using only the information that can be gleaned by sending packets into the network and observing when and whether they reach their destination. 

Arguably one of the biggest leaps forward after 1988 was the realization by Brakmo and Peterson (yes, that guy) that packet loss wasn't the only signal of congestion: so too was increasing delay. This was the basis for the 1994 TCP Vegas paper, and the idea of using delay rather than loss alone was quite controversial at the time. However, Vegas kicked off a new trend in congestion control research, inspiring many other efforts to take delay into account as an early indicator of congestion before loss occurs. Data center TCP (DCTCP) and Google’s BBR are two examples. 

One reason that I give credit to congestion control algorithms in explaining the success of the Internet is that the path to failure of the Internet was clearly on display in 1986. Jacobson describes some of the early congestion collapse episodes, which saw throughput fall by three orders of magnitude. When I joined Cisco in 1995 we were still hearing customer stories about catastrophic congestion episodes.  The same year Bob Metcalfe, inventor of Ethernet and recent Turing Award winner, famously predicted that the Internet would collapse as consumer Internet access and the rise of the Web drove rapid growth in traffic. It didn’t. Congestion control has continued to evolve, with the QUIC protocol, for example, offering both better mechanisms for detecting congestion and the option of experimenting with multiple congestion control algorithms. And some congestion control has moved into the application layer, e.g., Dynamic Adaptive Streaming over HTTP (DASH). 

An interesting side effect of the congestion episodes of the 1980s and ‘90s was that we observed that small buffers were sometimes the cause of congestion collapse. An influential paper by Villamizar and Song showed that TCP performance dropped when the amount of buffering was less than the average delay × bandwidth product of the flows. Unfortunately, the result only held for very small numbers of flows (as was acknowledged in the paper) but it was widely interpreted as an inviolable rule that influenced the next several years of router design. This was finally debunked by the buffer sizing work of Appenzeller et al. in 2004, but not before the unfortunate phenomenon of Bufferbloat–truly excessive buffer sizes leading to massive queuing delays–had made it into millions of low-end routers. The self-test for Bufferbloat in your home network is worth a look.

So, while we don’t get to go back and run controlled experiments to see exactly how the Internet came to succeed while other protocol suites fell by the wayside, we can at least see that the Internet avoided potential failure because of the timely way congestion control was added. It was relatively easy in 1986 to experiment with new ideas by tweaking the code in a couple of end-systems, and then push the effective solution out to a wide set of systems. Nothing inside the network had to change. It almost certainly helped that the set of operating systems that needed to be changed and the community of people who could make those changes was small enough to see widespread deployment of the initial BSD-based algorithms of Jacobson and Karels. 

It seems clear that there is no such thing as the perfect congestion control approach, which is why we continue to see new papers on the topic 35 years after Jacobson’s. But the Internet’s architecture has fostered the environment in which effective solutions can be tested and deployed to achieve distributed management of shared resources. In my view that’s a great testament to the quality of that architecture.

As fans of decentralization, we’ve moved our social media home to Mastodon–follow us here. If you need any further reason to get off X/Twitter, this may help. And you can read how Mastodon is Rewinding the Clock on Social Media — in a Good Way. Related to our earlier post on LLMs, Emily Bender has a new paper on the role of LLMs in information retrieval, making a compelling case that generative AI may not be what you want in a search engine. And given our interest in network verification, we were pleased to see this paper on experience with Batfish at SIGCOMM last week.

I appreciated Larry’s generous and thoughtful commentary on my CCR editorial, and I agree with much of what he wrote. However, I was surprised by his last paragraph where he suggests we should first reach consensus on how we should review nontraditional papers and then trust that our collective actions will follow suit. 

My takeaway is that focusing on the “mismatch between… SIGCOMM's goals… and what our current practices achieve” will not be fruitful until we are certain we understand our goals. Crafting a policy about how reviewers should evaluate papers describing new abstractions, new architectures, and new systems will not make a difference unless the community truly values that work, even when (1) it’s difficult to identify an immediate path to deployment or imagine the associated business model, and (2) it falls outside the traditional boundaries of what is considered to be in-scope. If we can reach consensus on this, then I believe the form (practice) will follow.

At the outset, let me be clear about where Larry and I agree and disagree. We agree on the need for change in SIGCOMM’s conferences. We also agree on the need to engage in an ongoing discussion about our goals and how to review the kinds of papers Larry refers to above. Where we disagree is that I don’t see that conversation ever reaching consensus, nor do I think consensus is needed for achieving significant change. In the next few paragraphs I delve more deeply into why I believe in this non-consensus path for change, because I think this exchange between us raises the crucial question of how to achieve change in large and diverse communities such as SIGCOMM. My reasoning is built on four basic points.

(1) Inclusion, consensus, and change: pick two. As individuals and as a community, we in SIGCOMM believe that (i) it is important to include a wide range of voices in SIGCOMM’s deliberations and (ii) policy decisions should be driven by community-wide consensus. We also, at least in the past decade or so, have come to believe that we need to change our conferences so that they better support SIGCOMM’s intellectual goals. Unfortunately, achieving consensus in any large organization is hard, especially in a community as diverse and loosely coupled as SIGCOMM. For instance, several people strongly objected to the statement that the research goal of SIGCOMM was to “lay the intellectual foundation” of our field. I thus do not think that achieving consensus should be our initial goal, because we are almost certainly doomed to fail. Instead, we should focus on inclusion and change, sidestepping the need for community-wide consensus; I will return to this in my fourth point.

(2) Our reviewing practices are shaped more by our experience and our sense of community norms than by vague exhortations about how we should review. It has been widely observed that when you ask graduate students to review papers, they are often sharply if not brutally critical. This is understandable because seeing the overly-negative reviews of their own papers is their only contact with what the community expects from reviewers, and they are merely following what they perceive as the community’s norm. SIGCOMM has provided guidance for reviewers, but this guidance has been widely ignored for over a decade. While I think we, as a community, should emphasize this guidance more than we do, and be engaged in an ongoing dialog about the nature of such guidance, we should not fool ourselves that such vague directives by themselves will have the impact we hope for. 

(3) Change is hard, and the alternative is the status quo. Creating change requires hard work, and merely registering a disagreement while remaining on the sideline is essentially a vote for the status quo. Over the past year, my three intrepid collaborators – Fabian E. Bustamante, Nate Foster, and Aurojit Panda – and I have talked with dozens of people in SIGCOMM (actively seeking out those who disagree with us), surveyed the community for input, and written thousands of words articulating our thinking. Our writings have described both some general principles and some specific incremental steps forward. You can see these efforts documented here and on the SIGCOMM Slack conference channel (which also contains valuable input from many others, including this report). Our process started with a fairly radical set of principles and ended with (i) a specific incremental proposal (described and later clarified) and (ii) a general process for ongoing change (which was developed in collaboration with the SIGCOMM Executive Committee). While our goals remain ambitious and far-reaching, we settled on a concrete first step and a process that could support ongoing progress, because we thought the former was the most we could achieve in the short-term, and the latter provided a solid foundation for future evolution. We greatly appreciate those who, while not agreeing completely with our ideas, were willing to lend their support to these efforts. From this I have learned the following lesson, modeled on Jon Postel’s famous advice: Be conservative in what changes you propose, but liberal in what changes you’ll support. 

(4) Empowering positive voices may be sufficient. Larry and I agree that we need to value papers that describe new abstractions, new architectures, and new systems even if they are hard to evaluate and inconsistent with current practice. You might naturally ask: How might our proposed set of actions cause the community to value such papers? Our community norms of how to review have been strongly shaped by how others review, and by what papers are accepted by the conference. Seeing the negative reviews of papers that fall outside the current norms, and their high rejection rate, has led many to believe that they shouldn’t be accepted, and I don’t think we can change people’s minds by telling them otherwise. Instead, we want to empower the current minority of reviewers who value such papers via our proposal (alluded to above) to accept papers that have a single explicit champion. This will change the kinds of papers that people see at SIGCOMM conferences, thereby demonstrating that there are those in the community who value this kind of work; this will hopefully lead to more people writing such papers, and more reviewers willing to advocate for such papers. I believe this gradual process of changing perceived norms, without reaching explicit consensus, is the path to change in organizations like SIGCOMM. Note that this approach is not restricted to accepting a particular set of “nontraditional” papers, but instead leads to a greater diversity of papers; any set of papers that has at least a minority of voices in favor will receive a more favorable hearing from program committees. This is crucial for SIGCOMM, because I think our conferences should broaden our perspectives, not just reinforce our current preconceptions. On that, I think we can all agree.

Larry’s Closing Thoughts

I like Scott’s focus on how to make change happen. Talk is cheap, and his attention to the process is something I can get behind. If the consensus I called for is manifest in concrete proposals for how SIGCOMM improves its conferences, then I’m happy. I do think this process is complicated by three things I talked about, which we need to acknowledge and address head-on. One is the breadth of methodologies we apply, which, to oversimplify a bit, range from theory to practice with plenty of ground in the middle. A second is the range of opinions about how we measure impact, which (to oversimplify again) range from changing perspectives to changing protocol specs. A third is the diversity of values we bring to the table, which is especially problematic when we conflate value with quality. As I wrote, I place a lot of value on the synthesis of complex systems that lead to new abstractions. But that’s just me. Others put their emphasis on policy questions, economic factors, and societal impact (to name a few). I’m not sure how to best reconcile all of that, but I’m encouraged by the fact that SIGCOMM is willing to try.

As we continue to follow the AI/ML space, an article on the poor performance of ChatGPT on an Algorithms exam caught our attention (and frankly didn’t surprise us). Thanks to user solstice333 on Github, we fixed a bug in our main textbook that had lurked undetected for 28 years–score one for open source publishing. Thanks also to Richard Clegg who sent us this video about the history of putting compass points other than North at the top of maps. Finally, don’t forget to do your bit for Internet decentralization by joining us on Mastodon or Peertube. 

A couple of questions put to me recently led me to think that perhaps it was time for another post on Large Language Models (LLMs) and the broader topic of AI. (As in an earlier post, I’m going to use “AI” in the way the term is most widely used now, an umbrella term that includes LLMs and other machine learning systems, in spite of some pushback. Noted roboticist Rodney Brooks has a good piece that touches on the change in meaning of “AI” since the term was coined over 60 years ago.) First, in a recent podcast, I was asked about the biggest challenge at Systems Approach, and I immediately answered “figuring out what’s important”.  This goes way beyond AI, but it’s a challenge even to decide what aspects of AI warrant our attention. And when a person in the investing community asked for my opinion on AI, my response (which might have verged on a rant) was this: my biggest concern with AI is that too many people, whether they are journalists, investors, business decision makers or whatever, are focused on the wrong problem. This issue is well captured in a recent Scientific American article by Emily Bender (of Stochastic Parrots fame) and Alex Hanna of DAIR. If you only read one article on AI, I’d suggest that one–even if that means skipping the rest of this post. To summarize, we should focus not on theoretical problems of some future superhuman intelligence, but on the harms that AI is already capable of causing, such as fostering discrimination in areas from housing to health care, or helping the spread of misinformation. 

That said, I continue to find the inner workings of AI quite fascinating and it’s worth understanding them well enough to know what AI is and is not capable of. In the last month I’ve read a number of more in-depth articles on LLMs, and these have given me a little more insight into why the question of what is really going on inside these systems remains, for most people, a matter of debate. I mostly agree with the position that LLMs have no idea what they are doing, but as with most topics, there is a bit more to this one than meets the eye. 

If you want to go deep into the internals of LLMs such as ChatGPT, Stephen Wolfram has written an excellent (if long) article that is also available in book form. One of the aspects that he drills down on is what it means to have a “model” of something. For example, if we had a set of data showing how long it takes a cannonball to fall to earth from various heights, we could fit a straight line to the data, and extrapolate or interpolate to predict times to fall from other heights not in the data set. But by choosing a straight line, we’ve adopted a model that’s not very accurate, and will be increasingly inaccurate as we go further outside the range of the original data. Knowing how gravity works, we’d be more inclined to fit a parabola, but that’s only possible because we already have a model for gravity. 

With LLMs, words are modelled in a vector space with hundreds of dimensions. The impressive feat is that an exceptionally complex model (with over a trillion parameters in GPT-4) can be trained (using vast amounts of input text) to do a pretty good job of mimicking human writing. In effect, GPT builds a model of language that captures a lot of the complexity of how humans string words together. With that model in place, an LLM is then able to generate strings of text not in the training data set–which is exactly what we observe when we interact with a system such as ChatGPT. As we know, the generated text often looks pretty authentic. As Timothy Lee and Sean Trott pointed out in another very helpful article at Ars Technica, LLMs deal with issues such as disambiguating the multiple meanings of words depending on context by passing the input text through multiple layers of neural networks. (“Fruit flies like a banana” is an example requiring some serious disambiguation.) Each layer is called a “transformer” (the T in GPT) and you can think of a line of text being passed through successive layers of transformers. Each layer adds metadata to the words: for example, having seen the sentence above, one transformer layer might add metadata to indicate that the word “flies” refers to insects rather than motion through the air.

There is a lot more in that article and I recommend you read it, but I had a disconcerting feeling as I was reading it that my confident assertion that LLMs have no understanding of the words they are producing was a bit overstated. At this stage, we all know of examples where LLMs have produced laughable results indicating a lack of understanding of the world, but the details of how they work show that they are very good at understanding language. I think the issue is the difference between understanding language (a set of symbols) and understanding the world. If a human understands language, we generally assume that they also understand the world, but making this extrapolation in the case of LLMs is a bridge too far. Here is a quote from the Ars Technica article that gave me pause:

For example, as an LLM “reads through” a short story, it appears to keep track of a variety of information about the story’s characters: sex and age, relationships with other characters, past and current location, personalities and goals, and so forth.

The description here comes awfully close to suggesting that the LLM “understands” what it is reading. Brooks calls out the issue here: we mistake performance (producing realistic text) for competence (understanding the world). Since he’s a roboticist (he founded iRobot), I found his prediction that GPT won’t be used for robots, because they have to understand the real world, very compelling. (Brooks is good at making predictions and he boosts his credibility by keeping his predictions online for the long haul and reporting back on them.) To quote:

… it will be bad if you try to connect a robot to GPT. GPTs have no understanding of the words they use, no way to connect those words, those symbols, to the real world. A robot needs to be connected to the real world and its commands need to be coherent with the real world. Classically it is known as the “symbol grounding problem”. GPT+robot is only ungrounded symbols.

This is the key takeaway for me: having a model for language is different from having a model of the world. For example, we know that LLMs have a tendency to make up citations. These citations look “correct” because they conform to the model of language (they have authors, realistic titles, journal names, etc.). But they fail a basic test: they are not drawn from real, legitimate publications. So the language model doesn’t understand “what is a legitimate citation” - a fact about the world that is pretty basic for a human to grasp. 

So I remain convinced that we need to be cautious about how LLMs and other AI techniques are put to work. Not because they are going to achieve superhuman intelligence, but because they have serious limitations, and because humans are already using them in ways that cause harm. This is certainly not limited to AI, but the difficulty of understanding what AI systems are actually doing and the human tendency to assume greater competence than they really have presents some unique challenges.

Another interesting example of the failures of today’s LLMs has been reported in The Register, with ChatGPT proving itself to be quite poor at answering Stack Overflow-style questions. In more positive news, AI does seem to be helping keep track of the flora and fauna species recorded around British train tracks. You can go deeper on the risks of AI in a podcast with Timnit Gebru, founder of the Distributed AI Research Institute and another of the co-authors of the Stochastic Parrots paper. Cory Doctorow has another great piece on “openwashing” in the AI field. And don’t forget that you can do your part to decentralize the Internet again by following us on Mastodon.  

I recently reread Scott Shenker’s essay questioning whether the practices used by SIGCOMM-sponsored conferences are helping the community achieve its research goals. The essay has triggered much discussion around the practices, but what I found to be most interesting is how the article first takes a position on what those research goals should be. If form is to follow function (as the essay’s title suggests), then you need to start with a good handle on the desired function. My sense is that the ongoing debate about practices (form) are more strongly rooted in how we think about the research goals (function) than we acknowledge.

Scott starts with a perfectly good statement of what he believes our shared goal to be:

SIGCOMM should strive to lay the intellectual foundation for future networks and networked systems.

And then he teases apart the nuanced meaning of the key phrases. This reminds me of a similar discussion 20 years ago, when the National Research Council published a report entitled:  Looking Over the Fence at Networks: A Neighbor’s View of Networking Research. Two things stand out about that report. One is that it was written at a time when the network research community was gnashing its collective teeth over the state of networking research, with companies like Cisco dominating what did or did not happen in the Internet. A second is that it purposely included input from people outside the networking community, providing a healthy dose of perspective. That history repeats itself is reason enough to take a fresh look at that two-decade old report, but I think there might be some other lessons to learn from both the report and Scott’s essay, and they (in part) have to do with the 800 pound gorilla in the room: the Internet.

For starters, the report lays out three promising research thrusts that are just as appropriate today: Measuring (understanding the Internet artifact); Modeling (defining a theory of networking); and Making (building disruptive prototypes). This is where the Internet is a double-edged sword. That it exists as a real-world phenomenon worthy of study is a boon for network research. That it exists as a multi-billion dollar industry makes it difficult to have impact by proposing new abstractions or architectures unless they can be justified and evaluated in the context of today’s artifact. I have a lot of respect for researchers who devise techniques to collect data about the Internet and then analyze that data to to gain a deeper understanding of its behavior, but I’ve spent my career more on the synthesis side than the analysis side of CS, and so I’ve been especially aware of the Internet as a barrier to research. 

The report introduced the term ossification into our vocabulary, and motivated work like PlanetLab (for me) and SDN (for Scott and others). Building platforms to support innovative and disruptive systems research (which has a strong synthesis component in its own right) is surely a good thing, but it is a means to an end. The “end” is discovering the fundamental abstractions and design principles that are the “intellectual foundation” of networking. And I believe this point to be at the core of why we struggle to get the “conference practices'' right. Here are my personal observations.

First, ten years ago this month, at the other end of my PlanetLab experience, I gave the Keynote at SIGCOMM. It was entitled Zen and the Art of Network Architecture, inspired by Robert Pirsig’s “Zen and the Art of Motorcycle Maintenance”. As I noted in my talk, Pirsig held the world record for having been rejected by 121 publishers; the parallel to academic network architecture papers was too poignant to pass up. In my mind, an architecture is just a multi-faceted abstraction (or a suite of interconnected abstractions), where abstractions represent the “essence” of the fundamental ideas systems researchers are in the business of discovering. If I were to amend Scott’s goal statement, it would be to emphasize the research community’s role as Abstraction Merchants (reintroducing a term Dave Clark once used to describe what we do).

Second, it is interesting to compare the OS community (SIGOPS) with the networking community (SIGCOMM). At SOSP (SIGOPS’ flagship conference) I generally feel qualified to review every paper submitted to the conference, and interested in every paper presented at the conference. It doesn’t matter what the system is—it could be a storage-related system, a network-related system, a compute-related system, and so on—I’m likely to learn about (1) an emerging mismatch between application requirements and technology constraints; (2) a new abstraction that fills that void; (3) the lessons the authors learned as they reduced the abstraction to practice; and (4) an evaluation of how the resulting mechanism stacks up across some subset of the -ities (scalability, reliability, availability, and so on). There will always be room to improve the mechanism, and write papers about those improvements, but the abstractions the community generates and is hammering on at any given time is what keeps the field vital. My sense, which is consistent with my reading of Scott’s position, is that SIGCOMM has become more focused on improving mechanisms within the boundaries of the existing (Internet) architecture than in introducing and exploring new abstractions. (There are exceptions, such as datacenter networking, but even in what could be an area that’s viewed as a greenfield, the inertia of “that’s not how we do it in the Internet” weighs heavy.)

Third, I believe the biggest risk the SIGCOMM community faces right now is a narrowing opportunity to have impact. This is related to the brief discussion about “networks and networked systems” in Scott’s essay (the question being whether SIGCOMM’s scope should include the latter). The answer is obvious to both Scott and me, but the fact that he raises it as an issue is telling. It also happens to be a tension I was keenly aware of in my keynote, having just spent the previous few years shepherding the GENI project, and watching my colleagues on both sides of that question have heated arguments (and on one occasion, a shouting match) about the research opportunities inside the network versus on top of the network. There are important and hard inside-the-network research questions—we’ve been working on distributed resource allocation (aka TCP congestion control) for decades—but the more networking technology matures, the more the interesting research questions move up the stack. And as a corollary, today’s research problems may be best framed in terms of the “cloud architecture” rather than the “Internet architecture”; the vocabulary we use can itself be limiting.

My takeaway is that focusing on the “mismatch between… SIGCOMM's goals… and what our current practices achieve” will not be fruitful until we are certain we understand our goals. Crafting a policy about how reviewers should evaluate papers describing new abstractions, new architectures, and new systems will not make a difference unless the community truly values that work, even when (1) it’s difficult to identify an immediate path to deployment or imagine the associated business model, and (2) it falls outside the traditional boundaries of what is considered to be in-scope. If we can reach consensus on this, then I believe the form (practice) will follow.

For an example of the creation of new abstractions, it’s hard to think of one more important than the packet, the origins of which feature in Bruce’s talk on 60 years of Networking. On returning from Edinburgh he has made a recording of the talk available on Peertube (which is the Fediverse’s streaming video platform, also discussed in the talk). You can hear about the creation of Systems Approach and what we think are our main current challenges in this podcast. And sticking with our theme of the Internet’s evolution, we enjoyed this piece from Cory Doctorow, which talks about how, under different conditions and constraints, the old, good Internet could have given way to a new, good Internet. Perhaps we can still create those conditions.

One of the fun things about being an Australian living in the Northern hemisphere (which was my situation for over thirty years) is having repeated conversations about which way water rotates when it goes down the drain. OK, it becomes a bit less fun over time, but I was always surprised that few people actually tried the experiment of checking out a few different drains in their own hemisphere. It turns out that drain geometry and initial water movement in the sink, not the Coriolis effect, dominate the direction of rotation, so you will see both directions in either hemisphere. There is actually a nice YouTube video that shows this, and then impressively proceeds to show the effect of Coriolis force on water draining out of a pair of identical kiddie pools in the two hemispheres (thus removing the confounding factors in most sinks, toilets, etc.).

A similar amount of time goes into explaining that North is not actually “up”, it’s just shown that way on maps drawn by Northern hemisphere-based explorers and almost every other cartographer since then. My father, who travelled quite a bit to the Northern hemisphere in the 1970s, might have been one of the first to make custom maps that put South at the top, to point out to his overseas colleagues that their map-drawing conventions were just that–arbitrary conventions. Never mind the issues of projection which left me thinking Greenland was bigger than Australia until I learned about alternatives to Mercator. 

All of this has been on my mind this week as (a) I returned to the Northern hemisphere for the first time since 2020 (see previous notes about my talk in Edinburgh) (b) I spent a lot of time with maps when I was out walking in the highlands of Scotland (c) I found myself needing to explain the difference between East-West and North-South traffic to some colleagues in the context of datacenter security. I’m not exactly sure of the origins of this naming convention, but the idea is that the ingress/egress point of a datacenter carries the “North-South” traffic, while the traffic that flows between servers within the datacenter is the “East-West” traffic.

Why do we even make this distinction? One big reason is security. Historically, the simplest way to “secure” a datacenter was to put some set of appliances (firewalls, intrusion detection systems, etc.) at the ingress/egress point. This is the “perimeter” model of security, which became prominent for several reasons. First, the number of ingress points to a datacenter is small–maybe as low as one, certainly no more than a handful. So it is natural to place centralized security appliances next to those choke points so that all the traffic can be passed through them. Furthermore, the bandwidth involved at ingress is likely to be orders of magnitude lower than the total East-West bandwidth: traffic entering a datacenter is likely to be measured in gigabits per second, while East-West traffic can easily run into the terabits. Neither of these points means that perimeter security is a good model–just that it was for a long time the most practical approach.

It is worth taking a step back to ask why centralized appliances became the preferred way to apply security controls. One version of this story is that the original Internet architecture had no security, and that early efforts to add security followed the end-to-end argument, which makes a good case for putting security into end-systems. For example, encryption and authentication are security mechanisms that can be implemented in end-systems (provided you can find a way to manage key distribution, which has proven challenging). However, as David Clark (co-author of the end-to-end argument) pointed out in a 2001 paper with Marjory Blumenthal, a multitude of factors pushed the Internet towards the adoption of centralized appliances inserted into the path of traffic by the late 1990s, such as the rise of malware, the adoption of the Internet by unsophisticated users, and the unreliability of software implementations on end-systems (e.g., OS bugs). While many Internet purists lamented the decline of the end-to-end principle, Clark and Blumenthal adopted the position that we have to deal with the world we live in rather than some idealized parallel universe. Centralized firewalls became part of the landscape because they allowed IT administrators to gain some control over the security of their networks in a world of increasing threats, without depending on the impractical notion that every end-system would do the right thing.

By the time I came to be involved in datacenter networking around 2012, the idea of securing the “perimeter” of the datacenter–which essentially involved putting a number of appliances into the ingress/egress path–was well established. Unfortunately, it was also fast becoming clear that this approach was inadequate, as a lack of East-West security meant that a compromise of a single (perhaps non-critical) system inside the perimeter could provide the launching pad for a much more serious attack via lateral movement among systems. The poster child for this issue was the 2013 Target hack, in which the initial breach took place via a refrigeration contractor’s computer, allowing the attackers to gain a foothold inside the perimeter of Target’s network, from where they were able, over a series of weeks, to move laterally among systems until they obtained the credit card details of about 100 million customers. There was no reason for the contractor portal (the original entry point for the attack) to have any connectivity to the systems that had credit card data. Nevertheless, because both systems were “inside the perimeter,” there were limited security controls between them. Lack of control over East-West traffic was the key to this and many other attacks.

Securing East-West traffic in 2013 was a fundamentally hard problem, because there is a vast number of paths between systems carrying massive volumes of data, and the traditional way to secure this would be to divide the network into a small number of zones with firewalls between them. Within a zone, traffic still flowed freely. It was either impractical or prohibitively expensive to place firewalls in such a way that all East-West traffic could be intercepted. 

I came to be interested in this issue because of the evolution of network virtualization that was taking place at about the same time as the Target breach. Our early network virtualization product at Nicira virtualized layer 2 (switching) and layer 3 (routing) and we had long held the view that we would work our way up the layers to virtualize all of networking. A simple firewall operates at layer 4 (looking at transport protocol port numbers) and so this was the logical next step. 

Network virtualization enables an SDN-style implementation of a firewall. By “SDN-style” I mean that the data plane is distributed while the control plane is logically centralized. In the image above, the distributed data plane runs in the virtual switch of each server, inspecting the traffic entering and leaving each virtual machine. (Similar approaches can be applied to containerized or bare-metal workloads.) This means it is now possible to apply firewall policies to every single packet that traverses the data center–even packets that only pass from one VM to another in the same server. Since virtual switches can process packets as fast as the server can send them, it became feasible to have terabits of firewall capacity allocated to East-West traffic. But because the architecture is based on SDN, there is a logically centralized control plane that simplifies management of the distributed data plane. From a control plane and management perspective, the firewall still looks like a centralized device, where an IT administrator (or an automated system calling an API) can set the firewall policies for the entire datacenter. But the data plane scales out with server capacity, and there is no need for heroic efforts to force traffic to flow through some centralized appliance.

There is more detail about this aspect of network virtualization in our SDN book. This is by no means the last word in East-West security; Aviatrix, for example, addresses East-West security for cloud workloads. And it’s important to do more than just inspect protocol ports as Thomas Graf shows in a talk on Cilium. Overall, the creation of tools to efficiently provide security services to East-West traffic was one of the key components to implementing zero-trust security, a topic we’ve covered previously. It’s also one of the main reasons that network virtualization achieved mainstream adoption in enterprise datacenters: it became obvious that relying only on perimeter security and a handful of firewall zones was insufficient for today’s security challenges. There is plenty more to be done here, with service meshes being another area of active work addressing (among other things) East-West security. But at least we no longer punt on the problem by relying solely on centralized appliances at the ingress to focus only on North-South traffic. 

Our previous notes on the importance of APIs and observability caused us to take note of the recent acquisition of Akita, an API observability company. Our book on Private 5G is available at a discount if you buy the ebook via our website. You can also pick up Systems Approach coffee mugs. And don’t forget to follow us on Mastodon, the thinking person’s social network.

Here’s a problem I’ve been struggling with for the last several months: How to make a complex system assembled from dozens of components easily consumable by a wide range of users. The system is the Aether edge cloud, which serves as the blueprint for our Private 5G book. The “dozens of components” include a long list of Cloud Native tools plus an open source implementation of 4G/5G Mobile Core and RAN. The “wide range of users” includes students who want hands-on experience with concepts they’re seeing for the first time, researchers who want to investigate narrow problems in the larger 5G/edge space, and organizations that want to deploy and operate Private 5G in everything from lab trials to commercial offerings. I’ll get to my definition of “easily consumable” in a moment, which is at the heart of why I find this to be an interesting problem in general.

You could make the case that this is a self-inflicted challenge—Aether is available in GitHub and anyone with the technical chops is free to do with it what they will—but it follows from the goal of realizing the know-how/educational value of open source software I talked about in my previous post. To provide a little background, a multi-site deployment of Aether has been running as a managed cloud service since 2020, in support of the Pronto Research Project. But that deployment depends on an ops team with significant insider knowledge about Aether’s engineering details. It has proven difficult for others to reproduce that know-how and bring up their own Aether deployments.

Offering “Aether-as-a-Service” comes with an ongoing obligation to provide operational support, but it is easier than releasing “Aether-as-Software”, complete with the machinery needed for others to deploy and operate Aether as their own service. This is well understood by anyone that has attempted to take that step, and familiar to me from my experience operating PlanetLab (but never packaging it in a way that made it easy for others to replicate). In the case of PlanetLab, the biggest value was in the network effect—having access to compute resources contributed by others all over the world—so operating a multi-tenant service made sense. In contrast, the biggest value for Aether is having full ownership of your deployment.

A minimal version of Aether is also available in a package called Aether-in-a-Box (AiaB). Originally designed to give developers a streamlined modify-build-test loop they could run on their laptops, AiaB serves as a good way to get started. But there is a considerable gap between what it provides and an operational 5G-enabled edge cloud deployed in a particular environment. Or as I’ve been known to say on family road-trips: You can’t get there from here.

As a “getting started” package, AiaB is straightforward to use. You set up a VM (on your laptop or in the cloud), clone the AiaB repo, and type make 5g-test. Doing so installs Kubernetes, brings up the Mobile Core, runs an emulated 5G RAN workload, and prints out the results. There are more intermediate Make targets users can explore, which helps from a learning perspective, but ultimately AiaB favors easy-of-use for canned configurations over ease-of-transition to more complex configurations that have been customized for a particular use case. That is the crux of the problem we try to address with Aether OnRamp: start with something as easy as AiaB, and then incrementally expose (and document) the information needed to take ownership of Aether in its full glory: a multi-site / hybrid cloud / managed service. OnRamp tries to do this in a way that supports more than one off-ramp, so that users that want to focus on a particular subsystem need not pay too steep of an up-front price.

The general approach OnRamp takes is to draw crisp lines between different stages (e.g., development, integration, deployment, operations) and layers (infrastructure, services, applications, traffic sources), and then introduce them incrementally, coupled with documentation that calls out the relevant “decision points” and “configuration parameters”. We include a first version of OnRamp with our Private 5G book, with the caveat that there is still much work to be done. That version relies heavily on Makefiles, which means the key configuration parameters are exposed as ad hoc variables. This summer I have been working with Bilal Saleem and Muhammad Shahbaz at Purdue to transition OnRamp to use Ansible. This started as an effort to scale Aether from a single node to a multi-node cluster, but Ansible is proving to be a better way to manage the overall stepwise configuration strategy. 

Although conceptually straightforward, this general approach is not easy to execute in practice. I have two takeaways from the experience so far. One is the importance of using tools that are powerful enough to get the job done, but not so heavy-weight as to obscure (abstract away) the know-how you’re trying to impart. This is important for two reasons: (1) so the novice can easily see what’s going on under the covers, and (2) so the expert can easily unwind engineering choices and apply different tooling. Ansible seems to be a good compromise in that it explicitly exposes the playbook, and provides a well-defined way to specify the relevant variables. There’s still a syntactic gap between an Ansible task and the corresponding kubectl call, but that gap is easy to document.

The second difficulty is to identify those relevant variables, which are often buried in a sea of configuration parameters. This process is complicated by developer bias—my term for codifying config variables (for example Helm Chart values files) that perfectly suit the developer’s needs, but obfuscate where a user trying to deploy the code needs to take ownership, so as to customize the parameters for their particular scenario. The only way I know to untangle this obfuscation is to try to document it, not just in a technical sense (i.e., most parameters are defined somewhere, if you know where to look), but rather, in an intuitive way that will make sense to a non-expert. Helping the non-expert understand what they can safely ignore goes hand-in-hand with identifying what they need to know.

Whether OnRamp hits the mark is a matter of judgment, where the proof of the pudding is in the eating. You can try our first attempt at helping users consume Aether today, and we expect to announce the availability of OnRamp-v2 by the end of the summer. Watch this site for an announcement. Finally, I would be remiss if I didn’t mention all the excellent hand-on tutorials created by other open source projects; the Kubernetes Tutorial is a great example. Like what we’re trying to do with Aether, the goal is to lower the barrier for people being able to consume open source software. My personal spin on that objective is to use such hands-on experience as a way to teach the underlying principles and concepts, and to enable research that builds on those concepts.

With Bruce in Scotland presenting his lecture at Edinburgh University’s celebration of 60 years of Computer Science and Artificial Intelligence, it’s up to his apprentice to see to it that this week’s post makes it to your mailbox. Confidence level: 7-out-of-10.

For those with a vested interest in SIGCOMM, you may want to read (and comment on) the community’s effort to establish a consensus about how its flagship conference should evolve.

Most people with some knowledge of networking history can name a few famous people from its history, such as the inventors of TCP/IP (Bob Kahn and Vint Cerf) and of the World Wide Web (Sir Tim Berners-Lee). While TCP/IP dates back to the early 1970s, and the Web to the 1990s, I wanted to go back further to cover the 60-year period being celebrated at Edinburgh. I found a wonderful resource in “A Brief History of the Internet” by Barry Leiner et al. (with co-authors including Kahn and Cerf among a long list of famous names). One of the striking things about that paper is just how many people were involved in the creation of the Internet. Fortuitously, Leiner et al. trace the Internet’s roots back to the early 1960s, lining up nicely with the timeframe I wanted to cover, and highlighting a number of developments I was unfamiliar with. I’ve leaned heavily on that paper for my talk.

First up is the invention of packet switching. It’s worth noting that the dominant network in the 1960s was the telephone network, which dealt in circuits. A small group of computer scientists recognized that there might be a form of networking that was better suited to computer communication. Len Kleinrock wrote a report on the queuing theory behind packet switching (without using the term packet) in 1961 (and later wrote books that were part of my education). Over the next few years Paul Baran in the U.S. and Donald Davies in the U.K. independently developed the main concepts of packet switching–they are generally recognized as packet switching’s co–inventors. Davies gets the nod for having coined the term “packet” to describe a bundle of bits packaged for transmission.

Arguably just as important was the work of J.C.R. Licklider, who seems to have been an amazing visionary across wide areas of computing. He proposed the idea of an “Intergalactic Computer Network” around the time he joined ARPA in 1962. These ideas took hold with Larry Roberts, who was appointed as the first program manager for ARPANET. The ideas of Kleinrock, Davies and Baran were adopted in the ARPANET design, with the “Interface Message Processors (IMPs)” being the first routers and ARPANET the first wide-area packet-switched network. Prior to this, Roberts apparently built a long-haul connection between two computers on opposite sides of the U.S. using a telephone circuit, illustrating by counterexample that packet switching really was the way to go. Leiner calls this the first wide-area computer network.

While the ARPANET is generally viewed as the predecessor of the Internet, what surprised me is how quickly the ideas of the Internet came together once the ARPANET existed. The first ARPANET hosts were connected in 1969, and as early as 1972 Bob Kahn was working on the ideas of “Internetting”: connecting different networks together. There weren’t a lot of networks to connect, but packet radio was developed around the same time, so the idea that very different networks could be connected arose quite early. Realizing that the protocols that he was creating to support internetting were going to need to be implemented on various operating systems, Kahn teamed up with Vint Cerf, who had experience with the implementation of NCP (which can be viewed as the predecessor of TCP/IP) on ARPANET. The idea that TCP/IP had to support heterogeneous networks was another big departure from the networks of the past–telephone networks were much more homogeneous.

My favorite paper on how the Internet developed is the 1988 paper by David Clark: The Design Philosophy of the DARPA Internet Protocols. With the benefit of 15 years of experience, Clark–often referred to as “The Architect of the Internet”–clearly lays out what he, Kahn, Cerf, and many others were trying to achieve when the Internet was designed. There was a lot more to the Internet than the TCP and IP protocols, and Clark lays out a detailed list of design goals, all in service of one top-level goal: to interconnect existing networks. That meant, importantly, that you couldn’t impose changes on the existing networks to make them “fit” the Internet architecture: the Internet had to be able to accommodate widely varying types of networks, e.g., those with no reliable delivery mechanisms or ability to provide notifications of failures. The ability to accommodate networks in spite of wide variations among their capabilities, including networks not yet invented, was quite a radical idea, and drove a lot of critical design decisions, such as the adoption of a best-effort service model.

The list of second-order design goals starts with: 

Internet communication must continue despite loss of networks or gateways. 

I feel obligated to point out that there is no mention in this paper of the Internet being designed to withstand nuclear attack. Unfortunately, I was responsible for perpetuating that misconception in the an early edition of “Computer Networks: A Systems Approach”. Leiner et al debunk this myth and explain its origins in their paper, but it’s a remarkably persistent one. “[C]ontinue despite loss of networks or gateways” is as close as it gets. 

Of the other design goals, the one that I keep coming back to is “distributed management of resources”. I’m personally very interested in the many ways in which the decentralized architecture of the Internet has been eroding, and have written prior posts on this topic here and here. As far back as 2016 I began to take an interest in blockchains as a possible path to a more decentralized approach, but these days I mostly find myself agreeing with Molly White and her ironic “Web3 is Going Just Great” site, which catalogs the steady stream of Web3 failures and grifts. Fortunately, there is more to the decentralization of the Internet than blockchains and Web3.

As we discussed last year, there are real signs of life for the decentralization of social media thanks to the emergence of ActivityPub and the Fediverse. Just as I was putting finishing touches on my slide deck, there was yet another meltdown in the world of centralized platforms, with the CEO of Reddit deciding that sudden changes to API pricing–to the point that lots of third-party applications become economically unsustainable–was such a good idea at Twitter that he would bring the same approach to Reddit. The response from volunteer moderators at Reddit–the unpaid community members who make the platform valuable to users–has been swift and, in some cases, hilarious. But the aspect of this story that really caught my eye was the rapid rise of ActivityPub-powered Reddit alternatives Kbin and Lemmy. The number of users of these services, which are interoperable with each other and connect to the rest of the Fediverse, has roughly tripled in a week (off a low base). Furthermore, there is a flurry of activity to beef up the implementations so that moderation tools–an essential part of any social media platform–can keep up with the growth in users. 

As an active user of another part of the Fediverse, Mastodon, I can tell you that there is lots of discussion among Fediverse users at present around how best to deal with all this growth and what it would take to scale the Fediverse up to the size of the big centralized platforms (which are still orders of magnitude larger than the Fediverse). It’s really too early to tell how this will all play out, and there is certainly more to the Internet than social media platforms, but I do feel a degree of optimism that the pendulum is swinging back towards the decentralized approaches that enabled the Internet to grow and flourish over the last sixty years.

In a timely coincidence, Steve Bellovin (well known as an Internet security expert) has published a first-hand history of netnews that is full of interesting details on how it developed in parallel with the Internet. And while I was digging around for bits of Internet history, I rediscovered the note from 2000 written by Bob Kahn and Vint Cerf regarding Al Gore’s role in the Internet – an assessment that is more favorable than most people would expect. As they say “No one person or even small group of persons exclusively ‘invented’ the Internet.”

I must create a System, or be enslav’d by another Man’s; I will not Reason and Compare: my business is to Create. - William Blake

That’s the quote we chose to open our first textbook back in 1995, in an effort to capture our belief in the importance of system-building. In a follow-up to last week’s post about the writing process, this week we look at how building systems with open source software has shaped our books. 

Open source software has become an integral part of today’s technology marketplace. We’ve written about weaponizing open source software in the context of our experiences using SDN to disrupt the networking industry. It’s also common to hear people talk about business models for monetizing open source software, for example, by selling support (think RedHat) or cloud services (think Databricks operationalizing Apache Spark). Discussions of this kind typically reduce to an exercise in defining a value proposition for open source software; if you’re going through that exercise in a business setting, that value has to be direct and quantifiable.

But there’s another way to think about value, which struck me recently as we went to press with our Private 5G book. Without consciously adopting it as a strategy, I realized that all four of the Systems Approach books we’ve written recently are strongly influenced by—and in most cases, organized around—open source software. The Private 5G book describes Aether as a combination of SD-RAN, SD-Core, and SD-Fabric (all open source); the SDN book describes a software stack that starts with a P4-programmable forwarding plane, and builds through a Switch OS, a Network OS, and a set of control applications (all open source); and the Edge Cloud Operations book describes how to build a cloud management platform from a combination of over 20 open source projects ranging from Kubernetes to Keycloak to Elastic Stack. Even the TCP Congestion Control book leans heavily on the implementation in the Linux kernel (which as I’ve argued in another post, is effectively the specification of the algorithm). And as Bruce reminded me in his last post, one of the reasons I decided to write the original Computer Networks book was that I was able to leverage open source protocol stacks that I had been working with. I should have seen this pattern long before now; I thought I was just being opportunistic.

I’m not sure what name to put on that value, or even the right way to characterize it. Part of it is my own internal drive to understand how complex systems work, and part is wanting to share that understanding with anyone struggling with the same questions—but it definitely feels like value to me. I guess the easy label would be educational value, but with understanding comes empowerment to make the ideas your own, adapt them to your purposes, and ultimately, to innovate. None of these are easily quantifiable, and I have no idea how to go about computing the Return on Investment, but that doesn’t make the value any less real.

But I’m getting ahead of myself. I think there’s more than meets the eye to the conclusion that open source software leads to understanding and know-how. For example, it’s obvious that open source makes it possible to see the engineering details of a given software tool. That’s important, but what I have found to be equally true is that having access to a breadth of implementation detail is essential to having a deep conceptual framework for complex systems like the Internet or the cloud. Or said another way, seeing the “Powerpoint rendition” of a system often leads to a superficial understanding unless you also have an opportunity to look “under the hood”, and ideally, play with the code. Here are three other observations that I came to appreciate about the topics we’ve been reporting on in our book series.

Starting with SDN, it is well known that the objective was to catalyze a horizontal market around the historically vertical networking industry, and to this end, our book describes all the components that go into building an end-to-end Software-Defined Network. There comes a point about three-quarters of the way through the book where we acknowledge that everything discussed up to that point merely “reproduce(s) functionality that already exists”, at which point we are finally ready to start talking about SDN’s supposed value proposition: the ability to rapidly evolve and customize the network. But if you stop and think about it, that first 75% is a complete blueprint for how to build a modern high-speed L2/L3 switch, which until fairly recently has been the proprietary know-how of a handful of device and chip vendors. (And this understanding is now finding its way into undergraduate networking courses; see for example CS 422 at Purdue.) It’s impossible to predict how a widespread understanding of the internals of packet forwarding will impact the future of networking (even at a time when the commercial viability of programmable hardware is being questioned), but I have no doubt that it will.

There is a similar story about 5G, arguably with the potential for even greater impact. An in-depth understanding of the mobile cellular network has been known only to a handful of incumbents for 40 years. The availability of an open, software-defined RAN and Mobile Core changes that dynamic. And even though the MNOs are starting to pull back from Open-RAN, presumably because the incumbent vendors have given them good business reasons to do so, I think it’s fair to say that there’s no turning back. It’s now the case that even I am able to bring up a 5G network; surely others will figure out ways to leverage that know-how in innovative ways. Making it easier for anyone to do that is the motivation behind the book’s hands-on appendix, which is starting to find its way into courses like CS 596 at UMass.

The final example is from how to operate an edge cloud. The know-how needed to operationalize an inert pile of code (whether it’s open source or proprietary) is substantial enough that we’re willing to pay other people to do it for us. Ease-of-use is often worth paying for, but doing so should not be due to a belief that wizardry is required. This is especially true since all the tools you need to operationalize cloud services are readily available as open source (complete with excellent tutorials). At the very least, you should know what you’re paying for when you outsource the problem. It’s also the case that the steepness of the learning curve is partially related to the newness of the technology, with lots of overlapping tools competing for mind-share. Once enough people understand the space in a principled way, we should expect to see a distillation that simplifies the toolset, hopefully lowering the barrier to entry.

At this point it’s fair to observe that someone has to pay for it, and it’s difficult to fund open source software for its educational value alone. Fortunately, the educational angle is usually not the whole story. Sometimes the technical people get out in front of the business people and make their code available without an obvious business model. That happened many years ago at Bell Labs, and still happens with researchers who value impact over monetary reward. Sometimes governments pay for it as a matter of public policy (or national security). That is what’s happening today in the 5G space. Sometimes companies take a long-term perspective as a way of growing a market rather than focusing on their share of an existing market. You could argue Google has done that by releasing tools like Kubernetes, or Nicira with Open vSwitch. But I suspect that most of the time, it’s software for which someone originally put together a business plan that ends up delivering this indirect and unquantifiable value (whether or not the business plan ever panned out).

These observations may be obvious in hindsight, but I think they are easy to overlook in a field so often focused on entrepreneurship and business value. There’s the marketplace of products, but also the marketplace of ideas, and how those ideas are manipulated impacts all of us. Learning from open source software—as a means of internalizing the know-how that went into building it—is one way to consume it. Our books aim to be an aid in finding and extracting that value. As we noted in our last post, we are primarily motivated by our potential impact, and open source software is one of the best ways we’ve found to deliver it.

Regular readers of our posts will know we like to stay on top of developments in quantum computing, and we were excited to read Scott Aaronson’s review of a new book called “Quantum Supremacy”. It’s a highly entertaining take-down of what seems to be a deeply flawed book, which also serves as a quick summary of all the mistakes you should avoid when talking about quantum computing. We’ve toyed with writing a book on quantum computing ourselves but this review certainly showcases the perils facing the non-expert author. And in another oft-hyped topic area, we enjoyed this guest blog post at the Linux Foundation on the benefits of open source large language models over their proprietary counterparts. 

There was a ripple in the world of academic publishing a couple of weeks back when the entire editorial board of Neuroimage walked out in protest at the high publication charges being imposed by Elsevier. Our own story of freeing content from Elsevier is a bit less dramatic but still makes for an interesting tale. Let’s start with the origins of Computer Networks: A Systems Approach, which we first published in 1996, when both the Internet and the publishing world were very different places than they are today.

Larry started the book on his own in 1994 and there was considerable interest in the publishing world to sign the contract for a new textbook on networking. The Internet had just become mainstream enough to make the cover of Time magazine, as Internet access started to spread beyond academic and research institutions. One of the distinguishing features of Larry’s proposal was that it put the Internet and its protocols at the heart of networking, whereas other books at the time treated TCP/IP as just one example among many competing approaches. In the end, Larry signed with then-independent Morgan Kaufmann Publishers (MKP), whose main claim to fame was that they also published the iconic Computer Architecture book by Hennessy and Patterson. Not only did we like the idea of being associated with that book, but working with a small publisher, we discovered, meant that you had the attention of the entire company when your book was getting ready for publication. 

Larry brought me on as a co-author in 1995 when he realized what a large task he had bitten off. We’d collaborated on research projects and academic papers before, so we knew that we could work together effectively, but I don’t think either of us quite appreciated how much work it takes to produce 500-plus pages of text and the images to go with them. I found writing a book that will be in print for years to be qualitatively different from writing academic papers, where a reviewer is likely to tell you what you did wrong and small errors may well go unnoticed. This work consumed our weekends and evenings for months. Often I would motivate myself to keep going by imagining that one day our work would be read by students of networking who would be influenced as much by our book as I had been when reading the works of Comer, Tanenbaum, and other giants of the field. 

We submitted the final manuscript to the publisher late in 1995. Not entirely by coincidence, I also ended up deciding that day to leave my research job at Bellcore and go to work at Cisco. Having just finished a book on networking, I knew what I cared about most in the technology landscape, and I was pretty sure Cisco was going to be a better place to push forward on that technology than Bellcore. I wasn’t wrong.

Once the book was published, the team at MKP set out to get it adopted by as many professors as possible, and the fact that we had taken a “Systems Approach” and centered our work on the Internet gave us a good position in the market. We never got to be the most popular networking textbook, but we carved out a good niche in what turned out to be a rapidly growing market.

One thing we didn’t quite appreciate was that textbook publishers really like to publish new editions every couple of years. For one thing, it gives their sales teams something new to talk about, but more importantly for the publisher, it stifles the used book market. If you make enough changes from edition N to edition N+1, then students will find they can’t really get by with a used copy, so they are pushed to buying the new edition. More book sales! We were happy enough to write the second edition in any case, since we’d found plenty of things in the first edition that we wanted to change–in fact Larry came to refer to the first edition as the Beta version. Security and wireless networks were two topics of note that first appeared in the second edition.

It was when we got asked to produce a third edition that Larry and I realized our incentives were not well-aligned with those of the publisher. Thinking back to what motivated us to write the first edition, it wasn’t about selling as many books as possible–it was about having an impact on the upcoming generation of students. We wanted our books to be read. Making used books obsolete was irrelevant to our goals. Furthermore, we’d worked hard to make sure our book focused primarily on timeless, fundamental aspects of networking. We didn’t aim to be the encyclopedia of networking as it exists at one point in time, but rather the guide for future networking engineers and protocol developers to help them build on the foundations laid in the last 30-plus years. So even though networking standards do change, the fundamentals change much more slowly. 

In parallel with this development, a chain of acquisitions left us writing for Elsevier, not MKP. (Readers who’ve worked for tech startups may recognize this phenomenon.) Our original, very supportive editorial team had moved on. Now we were just a mid-sized book project in the giant publishing machine. This made the non-alignment of incentives ever more apparent. We settled into a multi-year, low-key struggle with our publisher in which they were always asking us for a new edition–sometimes before the last one had even made it into print–and we were always trying to delay the next update. 

Larry’s involvement in open source software goes back a long way, including the x-kernel code that was included in the first edition. But it was in 2012 that I really started to pay attention to open source, thanks to my joining the Nicira team and making some small contributions to the Open vSwitch project. Coincidentally, I had to take a medical leave of absence from work in mid-2012 after a serious cycling accident, and since I was neither working nor sleeping much for a couple of months, I had a lot of time for creative thinking. It was during one of those sleepless nights that I came to the realization that an open source model for our textbook would be the best path forward for us. We could achieve our main goal of making the book accessible to the widest possible audience, while also leveraging the input of the broad networking community to keep on making updates to the book. Our vision was that the book would become an open source repository just like a software project, taking contributions from the community, but with Larry and me still retaining the ability to steer the overall direction of the book. And every once in a while we could “release a version” to produce the next published edition for Elsevier.  

The big problem with this plan was how to sell it to Elsevier. Clearly if the book was put online for free, that would negatively impact their book sales. But it would not drive them to zero. In fact Elsevier already had a series of books that were simply printed versions of RFCs on selected topics, which for some reason people paid for rather than reading the RFCs freely online. So we made the pitch that (a) the book would be better if we did it this way (b) they might not get another edition any other way, and (c) it wasn’t going to be the end of revenue for the book. To their credit, when we had the meeting to discuss this plan, one of the Elsevier editors made the analogy to the Oxford English Dictionary: a book that (apparently) loses money on every sale, but which is a great brand asset for its publisher (Oxford University Press). “We’re happy to be your OED”, we said. (It’s an open question whether we’ve done much to help the Elsevier brand, which obviously has bigger problems these days.)

It took several years to work out all the details, but today we are free to publish “Computer Networks: A Systems Approach” under a Creative Commons license, and we make small fixes to the online version frequently. A sixth edition, pulled from our repo, was published by Elsevier in 2021, and while sales aren’t anything like they were on the first few editions, they are indeed well above zero. There are now plenty of teachers and students using the latest version of the book without waiting for the multi-year update cycles that a publisher can manage. We even had a professor reformat the book with a large font that aids dyslexic students–a benefit of open source we had never anticipated. Getting control over our content also provided the starting point for a number of more focussed Systems Approach books, such as our one on TCP Congestion Control.

So this is also the origin story of Systems Approach, LLC. Last week we published our 4th book in the Systems Approach series: Private 5G: A Systems Approach. Like all our books now, it is an open source project and free to use under a Creative Commons license. Some of the details about our choice of license are covered in our submission to the 2020 Sigcomm Education Workshop. As we noted in that paper, open source software provides a model for textbooks that aligns well with where we believe the world of networking is heading, leveraging the input of the networking community and enabling us to share our efforts as widely as possible.

Another thing we can do with our open source content is license translations, and we recently did that with Edmar Candeia Gurjão producing a Portuguese translation of our 5G book. And we finally got that book to be complete enough that we have made it available as both print and eBook editions. And if you always wanted a Systems Approach coffee mug, now we have them too.

Our course on Magma, which we created two years ago, has now been revised and expanded. A good place to start if you want to learn about open source approaches to mobile networking.

Bruce is gearing up to give a lecture about “60 years of Networking” as part of the 60th anniversary event for Computer Science and AI at the University of Edinburgh. One of the themes will be the need to return to the decentralized roots of the Internet. Which is a reason why you should follow us on Mastodon and admit that Twitter is no longer fit for purpose.

The P4 Workshop was a couple weeks ago, and as General Chair, I went into it with a fair amount of trepidation. My concern was that Intel’s announcement earlier this year that they’re cancelling development of the Tofino 3 switching chip would have a chilling effect, not only on the Workshop, but also on the future of P4. That concern has been voiced in several forums, including SIGCOMM’s Slack workspace, with members of the P4 Advisory Board making reassuring pronouncements in various settings. (See for example, Nick McKeown’s post to the P4 Forum, and Nick along with Nate Foster and Jennifer Rexford discussing the future of Network Programmability on The Networking Channel).

I won’t try to give a point-by-point replay of what Nick, Nate, and Jen and others have been saying, except to observe that at a high level it can be summarized as follows:

Programmable Networks  >>  P4 Language  >>  Tofino Switching Chip

They point out, for example, that Tofino is just one of many interesting backend targets for P4 programs (SmartNICs and IPUs being the next “big deal”) and P4 is one of many tools being used to inject functionality into the end-to-end network path (DPDK and eBPF being two active projects that people are integrating with P4). Ultimately, the value of programmability comes from having visibility and control over the network, and there are many complementary approaches to making that happen. With that background, I do have three takeaways from what turned out to be an interesting and vibrant two days at the P4 Workshop (despite my initial concerns).

First, we’re often so focused on P4 as a tool to program the forwarding pipeline that we forget the other half of its value proposition: It also provides a way to specify the behavior of a pipeline (independent of how that pipeline is implemented). We talk about this idea, and the value of being able to auto-generate the Control API, in the P4 chapter of our SDN Book. Rob Sherwood made a similar argument at the P4 Workshop. It is now becoming a reality as companies like Google are starting to use such behavioral definitions as a Hardware Abstraction Layer (see Parveen Patel’s Keynote at the Workshop). This makes me hopeful that we are rapidly approaching the day when a P4 program (plus the generated P4RT interface) will become the standard way network providers specify their requirements to network vendors, and proposed new features (whether proprietary or standard) will be specified by a P4 program (potentially augmenting the intuition and design rationale presented in an RFC).

As an aside, I couldn’t help but notice the similarities between the architecture Parveen described and the way P4 has been used to program the forwarding plane of the 5G Mobile Core.  Both include a P4-based “abstract forwarding model” that’s independent of the underlying implementation details.

Second, it is common to divide forwarding pipelines into “programmable” versus “fixed function”, but this glosses over what might be the more important distinction: whether the pipeline is open or closed. Even “fixed function” pipelines are increasingly flexible–it’s just a question of how restrictive the vendor is in who they allow to make changes. This restriction may have the biggest impact on researchers who want to experiment with a new feature (especially ones that do not yet have a proven market), but maybe less so in the commercial world where incentives to make changes are (arguably) well-defined. Using P4 as the “spec language” (as I just outlined) has the potential to accelerate the process on the commercial side. On the research side, there is a strong argument in favor of using Tofino 2 to demonstrate the feasibility and value of new ideas (12.8 Tb/s still makes for a credible Proof-of-Concept), and repeating the refrain yet again, P4-as-spec makes for a compelling tech transfer story. If that were to happen, it would be interesting to see how vendors and chip designers adapt to reduce their spec-to-hardware implementation overhead. I would argue that programmable forwarding planes have a time-to-market advantage even for closed solutions.

Third, our focus on quantifiable metrics makes it easy to forget about the less quantifiable aspects of programmability. At its core, P4 is a programming language that does a good job of abstracting the essence of a packet forwarding pipeline. It is enormously impressive that a P4 program can be compiled onto a PISA-based switching chip that has the same performance, die area, cost, and power consumption of a fixed-function ASIC (and that equivalency was probably necessary for P4 to be taken seriously), but hitting that quantifiable mark is not sufficient. Well-designed languages are software tools that bring clarity to the intellectual challenge of programming. For me, the biggest “aha” moment of the Workshop was when Chris Sommers (long-time P4 contributor and new co-Chair of the API Working Group) started rattling off all the functions he’d been involved in writing in P4, and remarking on how natural P4 makes that process. There is certainly room to add new language features as P4 expands its domain to include SmartNICs and IPUs—as Chris and the other WG chairs are now pursuing—but having an existing target to evolve is a great position to be in.

One common thread that weaves its way through these three takeaways is that Intel’s cancellation of the Tofino 3 chip is a potentially helpful forcing function: The P4 community has to demonstrate the value of the language without being buttressed by ever-improving performance numbers that have more to do with 7nm semiconductor technology than anything networking people have done. I saw a lot of evidence that exactly that is happening at last month’s workshop. The march to programmable networks is inevitable (in my view), and I’m still optimistic about the role P4 will play a central role.

We continue to run into people who want to translate our books into other languages, and if you are one of them, you should definitely reach out to us. The latest entrant is a Portuguese translation of our Private 5G book by Edmar Candeia Gurjão. You can find other translations of our books here.

You can follow us on Mastodon.

While the early success of SDN was in local area networks (and particularly the special case of datacenter networks) it didn’t take too long before it made its impact in the wide area. The WAN applications of SDN can be further subdivided: there was the application of SDN to traffic engineering of inter-datacenter networks (notably from Google and Microsoft) and then there is what came to be known as SD-WAN. This part of the story relates primarily to enterprise WANs–a huge business but not something that gets much coverage in the discussions of SDN at academic conferences. So let’s look a bit more closely at enterprise WANs and why SDN turned out to be a good fit for the challenges in that environment.

My introduction to enterprise WANs came in the early days of MPLS, around 1996. My team at Cisco had published the first drafts on Tag Switching at the IETF, which would eventually lead to the creation of the MPLS working group and all the RFCs that followed. As described in a previous post, we were approached by a team at AT&T whose main problem, in essence, was that their enterprise WAN business was too successful. Their core business was building WANs using Frame Relay virtual circuits to connect the offices and datacenters of enterprises. Deploying a WAN for one customer entailed provisioning a set of frame relay circuits to interconnect the sites, plus configuring a router at every site to manage the routing of traffic over the WAN. The complexity of managing these circuits and routing configurations was becoming overwhelming both because of the popularity of the service and because of the increasing desire to provide full-mesh connectivity (or something close to it) among sites. 

At the time, one of the options that was being considered as an alternative to Frame Relay was to use IPSEC tunnels across the Internet to interconnect the sites. Leaving aside the fact that the Internet in 1996 was way less ubiquitous than it is today, the big downside of that approach was that it actually didn’t do much to reduce the complexity of configuration. You would need to configure as many IPSEC tunnels as Frame Relay circuits, and you still need to configure the routing overlay on top of all those tunnels to forward traffic between all the sites. In rough terms, both Frame Relay and IPSEC tunnels require n² configuration steps, where n is the number of sites. MPLS/BGP VPNs came out ahead by reducing that configuration cost to order n, even with full mesh connectivity among sites. The full story of how that works is in RFC 2547 and in the book I wrote with Yakov Rekhter. One thing that we made sure of was that our system had no central point of control, because in 1996 we knew that central control was not an option for any networking solution aiming to scale up. 

A few years later (2003) I gave the SIGCOMM talk for which I am most well known –  “MPLS Considered Helpful” (in the outrageous opinion session of course). Talking to people afterwards I was struck by the lack of awareness of how widely deployed MPLS/BGP VPN service was at that point. There were over 100 service providers using it to provide their enterprise WAN services by 2003, but that was completely invisible to most of the SIGCOMM community (perhaps because universities don’t rely on such services and because enterprise network admins don’t show up at academic conferences). 

Fast forward to 2012 and MPLS VPNs were the de facto choice for enterprise WANs. But many things had changed since 1996, and those changes were about to align to create the conditions for SD-WAN to emerge. Importantly, the idea that central control was a non-starter had been effectively challenged with the rise of SDN in other settings. Scott Shenker’s influential talk “The Future of Networking, and the Past of Protocols” had made the compelling case for why central control was needed in SDN, and developments in distributed systems had enabled scalable, fault-tolerant centralized controllers such as the one we built for datacenter SDN at Nicira. The first time I heard about the ideas behind SD-WAN was when one of my Nicira colleagues told me he was leaving for another startup. The simple high-level sketch he gave of applying an SDN-style controller to the problem of enterprise WANs immediately made sense. 

Whereas in 1996 we relied on a fully distributed approach using BGP to determine how sites could communicate with each other, an SDN controller allows policies for inter-site connectivity to be set centrally while still being implemented in a distributed manner at the edges. The benefits of this approach are numerous, especially in an era where high-speed Internet access is a widely-available commodity. Building a mesh of encrypted tunnels among a large number of enterprise sites no longer has n² configuration complexity, because you can let the controller figure out which tunnels are needed and push configuration rules to the sites. Furthermore, there is no longer a dependence on getting a particular MPLS service provider to connect your site to their network: you just have to get Internet connectivity to your site. This factor alone–replacing “premium” MPLS access with “commodity” Internet–was decisive for some SD-WAN early adopters. 

The other big change in the decades since RFC 2547 was the rise of cloud-based services as an important component of enterprise IT (Office 365, Salesforce, etc.). Traditionally, an MPLS VPN would provide site-to-site connectivity among branches and central sites. If access to the outside Internet was required, it would entail backhauling traffic to one of a handful of central sites with external connectivity and all the firewalls etc. needed to secure that connection. But as more business services were delivered from “the cloud”, and with SD-WAN leveraging Internet access rather than dedicated MPLS circuits to every branch, it started to make sense to provide direct Internet access to branch offices. This meant a significant change to the security model for networking. Rather than a single point of connection between the enterprise and the Internet (with a central set of devices to control attendant security risks) there are now potentially as many connection points as there are branches. 

So with SD-WAN and the rise of cloud services, you need a way to set and enforce security policy at all the edges of your enterprise–and now there is potentially an edge at every branch. In a sense, this aspect of SD-WAN contributed to the rise of SASE (secure access services edge): once you started putting SD-WAN devices at every site, you needed a way to apply security services at those sites. Fortunately, the “centralized configuration with distributed enforcement” model of SDN provides a natural way to address this issue. The SD-WAN device at the edge is not just an IPSEC tunnel terminating device, but is also a policy enforcement point to apply the security policies of the enterprise. And the central point of control for an SD-WAN system provides a tractable means of configuring the policies in one place even though they will be implemented in a distributed way. (Further complicating the picture is the fact that an SD-WAN edge device can also forward traffic to or from cloud-based security services.)

There is a lot more to SD-WAN than I have space to cover here. For example, dealing with QoS in the presence of the Internet’s best-effort service turns out to be important and is one area where the commercial providers of SD-WAN equipment try to differentiate themselves. SD-WAN remains an area where open standards have yet to make much of an impact. But it certainly provides an interesting case study in how a change in the adjacent technologies can make ideas that once seemed impractical (such as central control and VPNs built with meshes of encrypted tunnels) viable again. 

Following up on our prior post about getting a 5G small cell up and running, Larry finally tracked down the last bug in his configuration and was able to get traffic flowing from his mobile devices to the Internet. It turned out that debugging networking among a set of containers connected by overlay networks isn’t that easy! We’re updating the appendix of our 5G book accordingly. There is still time to submit your bugfixes to the book via GitHub to earn our thanks and a copy of the book.

You can find us on Mastodon here. The Verge has a good article about the Fediverse. Or maybe you want to have a look at Substack Notes, which we have just started to play with. Photo this week by Aaron Burden on Unsplash.

Our Private 5G book is informed by our experience designing and implementing an open source Kubernetes-based edge cloud that hosts—among other edge workloads—a managed 5G connectivity service. Edge applications can take advantage of local breakout, meaning they can communicate directly with IoT devices (and the like) without their packets ever leaving the enterprise. This local Connectivity-as-a-Service is then offered as a managed cloud service (rather than a traditional Telco service), including an API and Dashboard that makes it easy to monitor and control connectivity at runtime. The hope is that Private 5G will be as easy to deploy and use as Wi-Fi is today. (For the reader prepared to argue that Wi-Fi is sufficient for all edge use cases, we leave the 5G vs. Wi-Fi debate for another time.)

It should come as no surprise that designing/implementing Private 5G is not exactly the same as deploying/operating Private 5G, and since the main goal of the Appendix is to help readers with the latter, we decided to take the system we had built out for a test drive. But there’s an important qualifier before we get to that. The system we’re talking about, Aether, is not a collection of isolated components that leaves the dirty work of operationalization to someone else. Aether includes all the integration glue needed to bring up an operational system in support of live traffic—a topic we’ve covered in other posts and written an entire book about—but that doesn’t mean it’s easy for ivory tower architects like Bruce and myself to bring up Aether without a little friction. Some of the challenges were our missteps, but some point to the inherent difficulty in the Telco-to-Cloud transition Aether is trying to catalyze.

Step one is getting your hands on a 5G small cell radio, and they aren’t exactly available today at Best Buy. We’re using the Bridgestone Indoor 5G Sub-6G Small Cell from Sercomm. We also have experience with Sercomm’s 4G counterpart (which is less expensive and easier to find). You’ll also need a starter set of UEs, and while several smartphones support CBRS (e.g., iPhone 11, Google Pixel 4, or newer), our recommendation is to include a 5G dongle that can be attached to a Raspberry Pi. Acquiring the 5G hardware is still a problem today, but that’s probably a short-term situation. The other bit of hardware you’ll need is a server (or VM) to run Aether on, but the requirements aren’t too steep (Quad-Core, 12GB RAM, running Ubuntu 20.04 or 22.04). Note that the approach I’m describing uses CBRS spectrum that is allocated in the US; other countries are in different stages of establishing similar allocations.

Step two is where 5G is the most unfamiliar to anyone who has installed a Wi-Fi AP: Configuring the small cell radio. There are three parts to this. The first part is setting the RF-related parameters, which I am wholly unqualified to do. Their names are cryptic (e.g., FreqSsb, Arfcn), their settings seemingly arbitrary (e.g., 3609120, 643356), and the formulas to compute them… not exactly intuitive:

These (and other) parameters are related to the control the operator has over how the available frequency band is used, which is part of the value 5G brings. I clearly have more to learn, but fortunately, the out-of-the-box defaults work. The second part is connecting the small cell to the local network, which is straightforward, complicated only by the fact that the radio has two 802.3 ports: one known as WAN (but labeled 2.5G on the Sercomm 5G small cell) and the other know as LAN (but labeled 1G on the Sercomm 5G small cell). The WAN port is how the small cell connects to the Internet (indirectly via the Mobile Core, which we’ll get to in a moment). The LAN port is for connecting the radio to a management network, which is worth mentioning because you’ll eventually need to learn TR-069/TR-098 (in place of SNMP/MIB, respectively), since you’ll technically be managing on-prem Telco equipment instead of IETF-spec’ed Internet devices. There’s also an O1 Management interface, which is the O-RAN approach to managing RAN elements, but I have not yet had an opportunity to use it. It’s probably better to have too many programmatic interfaces than too few, but I was able to do everything I needed to through the dashboard, which is enough to get started. The third part is configuring the Spectrum Allocation Server (SAS), which is responsible for managing access to the three tiers of the CBRS spectrum. You’ll need to familiarize yourself with the SAS requirements and get credentials from a SAS provider (we use Google) if you want to get past the “turn it on and see if it boots up” stage. (You’ll also need to connect a GPS antenna, which the radio needs so it can tell the SAS its precise location.)

Step three is interesting because it’s related to how you assemble a system out of building-block components. As I discussed previously, the mobile cellular network defines a global naming scheme that makes it possible for any two RAN-connected devices to communicate with each other. You need to configure both the small cell radio and the Mobile Core software stack so they know how to plug into that global network. This means defining the Mobile Country Code (MCC) and Mobile Network Code (MNC) that you plan to use. This MCC/MNC pair forms a Public Land Mobile Network (PLMN) code, where we’ve used two different ids in different settings: 315010 constructed from MCC=315 (US) and MNC=010 (CBRS), and 00101 constructed from MCC=001 (TEST) and MNC=01 (TEST). And since you’ll technically be the MNO responsible for the Private 5G network you bring up, you’ll also need to burn the SIM cards that are to be inserted into all the UEs. The SIM cards include a unique identifier (called an IMSI), which is a 15-digit number with the PLMN code as its prefix. (You can buy a 5G SIM writer on Amazon, where one product description reads: PLS Kindly Note: The cards be provided to professional engineers, PLS be professional, you need have knowledge about sim cards, if you don't have, PLS do not buy it!)

Finally, in step four, you’ll be back in familiar IP-land, but your ability to juggle IP subnets, Linux bridges, and iptable rules will be taxed to the max. I won’t go through all the details, and your mileage will vary depending on how deeply you want to integrate the RAN into your enterprise network, but by my count, there are as many as seven subnets in play. This is in part because the Mobile Core is implemented in Kubernetes (with its own set of intra-cluster and service-visible addresses), in part because the backhaul that connects the small cell radios to the Mobile Core is an overlay network (for example, running on top of your local enterprise network), and in part because the forwarding plane of the Mobile Core—the User Plane Function (UPF) running as a Kubernetes-hosted microservice—is itself an IP router that forwards packets between the RAN and the rest of the Internet. You’ll certainly find that having access to diagnostic tools such as ping, traceroute, and tcpdump to be essential (which is one reason we recommend connecting at least one Pi+Dongle UE).

I’m pretty sure Wi-Fi configuration was never this complicated. To some extent, this may be due to where the line is drawn between the customer and the provider: Telcos have strived to keep the end-system they sell subscribers simple, but have accepted operational complexity in the network devices (such as base stations) that they manage. In contrast, anyone who purchases a Wi-Fi AP from a vendor assumes it will be straightforward to install. One would expect that, with time, small cells deployed in enterprises (and maybe even homes) can be pre-configured before they are shipped or auto-configured after they are installed, but our goal with the book is to demystify 5G, including all the configuration steps. If you’re an enterprise system admin (or a hobbyist who wants to try out the technology at home) you will need to know about all of this. That’s why we wrote the book! It’s also why it’s important to have access to open source implementations of all this technology.

As noted above, our Private 5G book is close to completion. Not only does the appendix now describe our own hands-on experience, we also just received a very kind foreword from networking pioneer Jennifer Rexford. Congratulations to Jen on her recent ascension to the position of Provost at Princeton. The final step before we can commit the book to print is a last round of bug-scrubbing. Once again, we are offering a free copy of the book to anyone who submits two or more accepted bugfixes, and a thank-you in the Acknowledgments to everyone who submits a fix.

You can find us on Mastodon here.