We are on track to witness a remarkable milestone by the end of this decade – one trillion transistors within a tiny microprocessor package. This paper takes a sweeping view of waves of innovations that enabled transistor scaling over the last six decades and outlines critical innovations that will be necessary for continued transistor scaling over the next ten years.

Significant breakthroughs in materials engineering, device physics and process integration will be needed to overcome the most daunting challenge in computing today – lowering energy consumption to enable sustainable deployment of the exponentially growing demand for Artificial Intelligence (AI). Meeting this critical challenge will require the development of a revolutionary new transistor capable of operating at ultra-low supply voltages (lower than 300mV) while delivering acceptable performance (switching speed). This paper highlights the most promising future transistor options and envisions a transistor architecture that may meet these challenges.

Looking Back — Classical Scaling (CPU) Era: 1965 – 2005

The first four decades of Moore’s Law witnessed exponential growth in transistor count and enabled multiple successive eras of computing, starting with the mainframe and culminating with the Personal Computer (PC). Every 10X increase in transistor count enabled the creation of a new class of computers that built upon the prior generation.

Moore’s Law (1965) along with the scaling physics established by Bob Dennard (Link) in 1974 provided technologists a blueprint to steadily shrink transistor dimensions while also progressively increase transistor performance at relatively constant power density. This happy marriage between Moore’s Law and Dennard scaling ushered in a golden era of computing that spanned nearly 4 decades. This era was made possible by numerous innovations in semiconductor equipment technology, materials engineering and process integration, most important being the consistent reduction of gate dielectric thickness (Tox) and the development of progressively shallower source/drain (S/D) extensions which enabled scaling of transistor gate lengths from micron-scale to nanometer-scale while lowering transistor threshold voltage (Vt). This period required a significant reduction in transistor operating voltage (Vdd) from 5V (up to the early 1990s) to just 1.2V (around 2005) to maintain product reliability with progressively thinner gate dielectrics. These innovations enabled chip clock frequencies to increase from mere tens of kHz to an incredible 3GHz.

Over time, the quest for raw performance (switching frequency), especially in PC (CPU) chips, forced faster dimensional scaling compared to voltage (Vdd) scaling, thus increasing electric fields across the device. Furthermore, additional Vdd scaling became limited by increasing off-state leakage (Ioff). This led to progressively higher power densities and, eventually, the breakdown of Dennard scaling itself. Power density was pushed to ~150W/mm2, the limit allowed by the economics of packaging and thermal dissipation capabilities of the day. By 2005, the industry found itself facing fundamental barriers to continued transistor scaling.

Subscribe now

Looking Back — Mobile (SoC) Era: 2005 – Present

During the last 20 years, technologists broke through multiple seemingly insurmountable barriers to transistor scaling, including perceived limits to dimensional scaling, limits to transistor performance, and limits to operating voltage reduction. This era marked the emergence of mobile computing, which shifted the focus of transistor development from chasing raw performance (switching frequency or clock speed) to maximizing performance within a fixed power envelope (performance-per-watt).

Nonetheless, exponential growth in transistor counts continued unabated, albeit without the tailwinds of Dennard scaling, thus presenting computer architects an entirely new challenge – how best to utilize an abundance of transistors to improve performance and functionality while staying within a fixed power budget. This constraint was addressed by an architectural solution – core-level parallelism. Many computing tasks could be sped up by parallelizing them across two or more computing cores while consuming less total power. This gave rise to the era of dual-core and later, multi-core microprocessors. Even with multi-core architectures, increasing power density would eventually render sizeable blocks of transistors unusable at any given time (also known as “dark silicon”).

Concurrently, the transistor reached physical limits of silicon dioxide gate dielectric thickness (Tox) scaling and a progressive degradation in silicon channel mobility became apparent, which threatened to limit further gains in performance and power efficiency.

Radical transistor innovations would become necessary to surmount these perceived fundamental barriers.

Share

Seminal Transistor Innovations Led by Intel

Starting in the early 2000s, engineers at Intel pioneered the use of novel materials and architectures in transistor technology and expedited the progress of groundbreaking ideas from research to development and high-volume manufacturing. These innovations ushered in an era of astonishing progress in transistor technology over the following two decades and became the bedrock of all modern semiconductor technology as they continue to be used in transistor manufacturing to this day!

1. Mobility Enhancement — Uniaxial Strained Silicon

For many years, introducing strain into the silicon transistor was seen as the holy grail to improve performance. While many companies and researchers proposed different approaches to incorporating strain, none were deemed manufacturable in high volume. Finally, in 2004, Intel introduced a novel transistor structure using embedded Silicon-Germanium (SiGe) to incorporate compressive strain for PMOS (hole) mobility enhancement and a novel capping layer to incorporate tensile strain for NMOS (electron) mobility enhancement. Intel’s uniaxial strain approach was in stark contrast to the biaxial strain approach pursued by other companies and the broader research community and turned out to be far superior for performance and manufacturability. Moreover, this architecture proved highly scalable and enabled progressively higher strain levels and, thus, higher performance over many generations of transistor scaling. This innovation is utilized by every major semiconductor foundry in all modern transistors to this day!

2. Tox Limit — Hi-K Dielectrics and Metal Gate Electrodes

Intel explored multiple approaches to replace SiO2 based gate dielectrics and silicon gate electrodes with novel Hi-K gate dielectrics coupled with metal gate electrodes. These approaches included “gate-first,” “replacement-gate,” and even fully-silicide gate electrodes. Culminating a decade long, exhaustive research and development effort, Intel shocked the industry with the introduction of the replacement gate process at the 45nm node (2007). This change introduced hitherto exotic rare-earth elements and metals into mainstream silicon fabs and required the development of new semiconductor equipment and process engineering techniques. Gordon Moore hailed this as the most significant change to the transistor since the 1960s. This innovation continues to be used in all advanced node transistors to this day!

3. Planar Transistor Limit — FinFETs

By 2010, the traditional planar (2D) transistor architecture finally ran out of steam after five decades, mandating a move to the 3D FinFET. Yet again, Intel was the first to introduce FinFETs into production at the 22nm node in 2011. Nanometer-scale fin widths enabled superior electrostatic control and thus, higher performance at lower Vdd. The 3D structure of fins resulted in a sharp increase in effective transistor width within a given footprint, leading to vastly superior drive currents. The dramatic evolution of the fin profile over the last 15 years was enabled by numerous innovations in materials deposition and patterning. This innovation too, is deployed in every modern transistor process to this day!

Looking Ahead: The AI (System-in-Package) Era

The seventh decade of Moore’s Law coincides with the emergence of yet another computing era. In the coming years, AI will redefine computing and is already causing a tectonic shift in the enabling silicon platform from general-purpose processors (CPUs) to domain-specific accelerators (GPUs and ASICs). 

This shift in computing platform also coincides with yet another inflection in transistor architecture. The Gate-all-Around (GAA) or RibbonFET transistor is poised to replace the FinFET. RibbonFET is a natural evolution of the FinFET and delivers enhanced drive current and/or lower capacitance within a given footprint, superior electrostatics, a higher packing density, and the ability to operate at a lower voltage.

The RibbonFET architecture will likely be succeeded by a stacked-RibbonFET architecture with N and P transistors stacked upon each other to create more compact, monolithic 3D compute units. This architecture can deliver a dramatic (>1.5X) increase in transistor density within a given footprint and has already been demonstrated on silicon by Intel and others.

Beyond stacked silicon RibbonFETs, 2D transition metal chalcogenide (TMD) films are being investigated as channel material for further dimensional scaling, but many issues persist.

Over the last few years, worldwide energy demand for AI computing has been increasing at an unsustainable pace. Furthermore, transitioning to chiplet-based System-in-Package (SiP) designs with 3D stacked chips and hundreds of billions of transistors per package will increase heat dissipation beyond the limits of current best-in-class materials and architectures. Breaking through this impending “Energy Wall” will require coordinated and focused research toward reducing transistor energy consumption and improving heat removal capability.

Call To Action: A New Ultra-Low Energy Transistor

A focused effort is necessary to develop a new transistor capable of operating at ultra-low voltages (Vdd < 300mV) to improve energy efficiency. However, ultra-low Vdd operation can lead to significant performance loss and increased sensitivity to variability, requiring circuit and system solutions to be more resilient to variation and noise. These challenges call for a strong collaboration between the device, circuit, and system communities. Improving transistor performance at ultra-low voltage will require the development of a super-steep sub-threshold slope transistor and the use of high-mobility channel materials.

Potential options for a super-steep sub-threshold slope transistor include Negative Capacitance FET (NC-FET), Ferroelectric FET (FE-FET) and Tunnel FET (TFET), each with unique challenges. Each of these are being actively investigated across industry and academia.

The NC-FET leverages a ferroelectric gate insulator material. Negative differential capacitance in ferroelectrics can amplify changes in surface potential in response to gate voltage, leading to lower sub-threshold slopes and lower equivalent oxide thickness (EOT) (Further reading: Link)

While NC-FET is designed for hysteresis-free operation, FE-FET is hysteretic. As illustrated in the schematic above, the FE-FET relies on ferroelectrics with low coercive voltage to generate an “effective” ultra-steep sub-threshold slope, less than the lowest achievable in silicon transistors (60mV/decade).

Tunnel-FETs have been perennially plagued with low drive current and less-than-projected improvement in sub-threshold slopes.

Given the significant loss in drive current due to low gate overdrive, high-mobility channel materials will also be necessary to boost drive current at ultra-low Vdd. A targeted introduction of high-mobility channel materials such as Germanium, III-Vs, and carbon nanotubes into existing mature silicon substrates is expected to yield rich dividends.

Plenty of Room at the Bottom!

The number of transistors in a microprocessor package will continue to increase substantially over the next ten years. Developing an ultra-low Vdd transistor will help address one of the most significant contributors to AI energy consumption and thermal dissipation concerns in the trillion-transistor era.

At every significant inflection over the last 60 years, when challenges to continued transistor scaling seemed too daunting, technologists across industry and academia forged new paths to enable the arc of exponential progress to continue unabated. There is no reason to believe that this trend will not continue well into the future. There is still plenty of room at the bottom!

Subscribe now

The original version of this essay was published as an invited special paper in the proceedings of the recently held 2024 International Electron Devices Meeting (IEDM). This version has been created for a mainstream audience. The views expressed herein are the authors’ own.

A worldwide shortage of semiconductor chips during the COVID-19 pandemic shone a spotlight on the outsized geopolitical and economic significance of the tiny island nation of Taiwan. The island is home to the world’s largest contract chipmaker, Taiwan Semiconductor Manufacturing Corporation (TSMC), the world’s largest contract electronics manufacturer, Hon Hai Precision Industry Ltd., also known as Foxconn and one of the world’s largest fabless chip design companies, MediaTek.

Taiwan’s strength in semiconductor manufacturing is described by some as a “Silicon Shield” against potential Chinese aggression. The history of how this tiny island nation with a population of just over 23 million and a GDP that represents just 0.6% of the world economy grew to become the epicenter of the global semiconductor industry and a linchpin of the world economy sheds light on the surprisingly close and common origins of the American and Taiwanese semiconductor industries.

“You can’t connect the dots looking forward; you can only connect them looking backward.

Steve Jobs

Connecting the dots backward from Taiwan’s early semiconductor visionaries to legendary American founders like David Packard, Bill Hewlett, Bob Noyce and Gordon Moore highlights the central role of one individual - Fred Terman, and one American institution - Stanford University.

Professor Frederick Terman, an acclaimed expert in the then emerging field of radio engineering served as Dean and later, Provost of Stanford University. He is widely regarded as the father of what came to be known as Silicon Valley.

It is fascinating to realize that a single individual was able to influence - directly or indirectly - the creation of an entire industry that would become the bedrock of the modern digital age, first in the United States, and a couple decades later, oceans apart, in Taiwan. It makes one wonder: how would modern computing have evolved if not for the contributions of a single individual?

Subscribe now

Creation of Hewlett-Packard (1939)

Professor Terman was known to encourage his students to become entrepreneurs and start local businesses which could in turn create employment opportunities for local engineering graduates. He was responsible for reuniting a couple of Stanford engineering graduates, William Hewlett and David Packard who took his summer seminar on audio oscillators. With encouragement from their professor, Hewlett and Packard would go on to start their eponymous corporation in the San Francisco Bay Area. Terman even suggested to them their first marketable product - an audio oscillator based on the principle of negative feedback that he himself had taught them! HP became the first major technology company in what would be known as Silicon Valley.

Without the encouragement of Fred Terman, would Hewlett and Packard have created HP in the San Francisco Bay Area? Without HP, would Silicon Valley exist, or at least, would it exist in its current form?

Creation of Shockley Semiconductor and Fairchild (1956-7)

In 1956, Professor Terman lured his friend William Shockley, co-inventor of the transistor, to move back to Palo Alto from the East Coast and start Shockley Semiconductor Laboratories. Terman believed that the creation of a research institution in the Bay Area led by someone of Shockley’s stature would benefit Stanford and its students. Shockley hired some of the most brilliant young minds in the then nascent field of semiconductors who would go on to create Fairchild Semiconductor just a year later in 1957.

Fairchild alumni, including Bob Noyce and Gordon Moore later founded several dozen semiconductor companies in the Bay Area, including Intel which would go on to become the largest semiconductor company in the world.

Had Fred Terman not encouraged his friend William Shockley to move to Palo Alto, would Shockley Semiconductor have been created in California? Without Shockley Semiconductor in California, would Bob Noyce have led seven other technologists to leave and create Fairchild? And without Fairchild, would there have been an Intel and scores of other semiconductor companies in the Bay Area?

Creation of ITRI in Taiwan (1974)

Another student of Professor Terman, lesser known, but a contemporary of Hewlett and Packard, was Wen-yuan Pan. Pan left his home in mainland China in the 1930s and sailed across the ocean to the United States, eventually making his way to Palo Alto to also study radio engineering under Professor Terman, along with Hewlett and Packard!

Wen-yuan Pan even co-authored a paper on amplifiers with Professor Terman and Bill Hewlett. After obtaining his PhD in 1940, he went on to have a distinguished nearly three-decade long career at the Radio Corporation of America (RCA). But Pan’s most enduring legacy would be his role in devising and executing the plan that made Taiwan a silicon economy, home to the largest chip manufacturer in the world.

In 1974, Pan convinced Sun Yun-Suan, the Taiwanese minister of economic affairs to approve a $10M, four-year plan that would establish a semiconductor industry in Taiwan. His work led to the founding of a semiconductor research division under the Industrial Technology Research Institute (ITRI). Pan was instrumental in establishing a technology transfer agreement with RCA and recruiting the first 37 Taiwanese engineers to engage in a yearlong training program at RCA in the United States. These engineers returned to Taiwan and built a government funded 7-micron chip fabrication facility that would become the start of Taiwan’s semiconductor industry.

Just as the Fairchild alumni would go on to start numerous semiconductor companies in Silicon Valley, Pan’s group of initial recruits would go on to constitute almost the entire senior leadership of Taiwan’s semiconductor industry, including Ming-Kai Tsai, who led the creation of MediaTek, one of the world’s largest fabless semiconductor design companies. Pan grew up in Mainland China and studied and later worked in the United States long before the creation of Taiwan. He never studied, settled, or worked for pay in Taiwan but is honored as the father of Taiwan's semiconductor industry.

Had Wen-yuan Pan not found his way from China all the way to Stanford in the 1930s with interest in radio engineering, would he have met a mentor like Fred Terman and colleagues like Bill Hewlett and Dave Packard? Without the support of Professor Terman, would he have joined Radio Corporation of America (RCA)? Without the connections he made at RCA, would he have been able to strike a deal to transfer semiconductor technology from RCA to Taiwan?

Creation of TSMC (1987)

A decade later, Sun Yun-Suan, who then was the Premier of Taiwan, convinced yet another Chinese American executive to take the helm of ITRI and help map the future of the semiconductor industry in Taiwan. That executive was Morris Chang, who also happened to have walked the same hallways of Stanford University as Pan Wen-yuan, David Packard and William Hewlett a couple decades before him. Morris Chang earned a PhD in Electrical Engineering from Stanford and was taught by none other than William Shockley himself and other faculty who were also personally recruited by Fred Terman.

In 1987, ITRI, led by Morris Chang, spun off a chip manufacturing facility, transferring fabs, equipment, technologies, and an initial group of engineers to the newly formed company. This venture was none other than Taiwan Semiconductor Manufacturing Company (TSMC), now the largest semiconductor manufacturer in the world.

Morris Chang, Bob Noyce and Gordon Moore were contemporaries - while Chang was taught by Shockley at Stanford, Noyce and Moore worked for Shockley directly. In 1958, Chang (27), Moore (29) and Noyce (31) attended the International Electron Devices Meeting (IEDM), where they had dinner together and went out drinking on the cool, December nights in Washington DC. Little did they know then, that the three of them would go on to create the two largest and most influential semiconductor manufacturers in the world - over many decades and across two different continents!

Subscribe now

End of Globalization (?) or American Homecoming (?)

The seeds of Taiwan’s future semiconductor dominance were sown in the United States in the 1930s and 1940s, by Chinese student immigrants like Wen-yuan Pan and Morris Chang, long before the creation of modern day, independent Taiwan in 1949.

“Twenty-seven years have passed, and the semiconductor industry witnessed a big change in the world, a big geopolitical change in the world. Globalization is almost dead and free trade is almost dead. A lot of people still wish they would come back, but I don’t think they will be back.”

Morris Chang, speaking about TSMCs manufacturing expansion in America, Dec. 2022

Geopolitical tensions post the COVID-19 pandemic in 2020 forced TSMC to begin expanding their manufacturing footprint outside Taiwan for the first time in decades, when they decided to build a massive semiconductor fab in the deserts of Arizona, just a few miles from Intel’s largest semiconductor manufacturing base in the United States. Morris Chang, who publicly opposed this move, later said he understood the reason for the move, but lamented it as a sign of the “end of globalization” (Link).

Perhaps an alternate way to view TSMCs manufacturing expansion in Arizona is not so much as the end of globalization but more as the homecoming of a technology that was born in America and generously shared with the rest of the world - by American institutions and visionary men like Fred Terman - for the benefit of all humanity.

“It may prove to be more economical to build large systems out of smaller functions, which are separately packaged and interconnected. The availability of large functions, combined with functional design and construction, should allow the manufacturer of large systems to design and construct a considerable variety of equipment both rapidly and economically.”

Gordon Moore, Cramming More Components onto Integrated Circuits, April 1965

In his prescient 1965 paper, Gordon Moore alluded to an eventual “Day of Reckoning” when he foresaw an economic imperative to break out large monolithic “functions” into smaller, interconnected “functions”. Gordon Moore was likely envisioning what eventually became Multi-Package Modules (MPMs) that are now common in everyday consumer appliances like the iPhone which combines a memory chip unit (with Dynamic Random Access Memory or DRAM) and a logic chip unit (an Application Processor Unit or APU), separately packaged and interconnected into a single larger unit. Advances in wafer-level packaging technology over the last decade have now made it possible to also envision Moore’s 1965 prediction at a single package level, where large monolithic functions (chips) can be broken into smaller functions (chiplets), connected with extremely high bandwidths to create a much larger, virtually monolithic chip within a single package.

The thriving semiconductor ecosystem that evolved over five decades to support the efficient design and manufacturing of monolithic, standalone Systems on a Chip (SoC) is evolving yet again to enable the efficient design, manufacturing and assembly of discrete chiplets into integrated Systems in a Package (SiP).  This transformation from SoC to SiP has been underway for a few years already but is now reaching an inflection point driven by a combination of technological and economic imperatives.

This paper discusses the key pre-requisites for chiplet based designs and highlights the foundational enablers that will democratize the chiplet design ecosystem and drive this transformation at-scale. 

Subscribe now

From SoC to SiP

Over the last few decades, exponential increase in transistor counts on a chip led to a commensurate increase in chip design complexity. This complexity was effectively managed by periodic innovations that enabled progressively higher levels of abstraction in both circuit design (e.g. using Electronic Design Automation or EDA tools) as well as in chip architecture (e.g. using fully validated and verified intellectual property or IP blocks) to design ever larger monolithic chips. The upcoming transition from designing large monolithic chips to designing smaller, fully validated discrete chiplets can be seen as the logical next step up the design and architecture abstraction stack.

Since the late 1990s, advances in Computer-Aided Design (CAD) and Electronic Design Automation (EDA) have made circuit design more accessible, leading to the emergence of independent, third-party IP designers. These designers focus on creating standalone IP blocks that are fully tested and verified. Chip design companies can then integrate these IP blocks to create an array of complex SoC designs. Over time, this evolution led to a robust ecosystem of IP companies offering a wide range of digital logic, analog, and mixed-signal IPs, from foundational IPs like memory and logic standard cell libraries to specialized IPs such as high-speed analog I/O. Today, chips architects can rapidly put together complex SoCs with mostly off-the-shelf IPs, which are readily available and validated across most mainstream foundry process technologies.

In the near future, we can anticipate comparable advances within the emerging chiplet ecosystem. These advances will facilitate the rapid design and construction of custom chiplet IPs to exact specifications. System architects will be able to rapidly integrate multiple, fully validated chiplet designs into significantly larger systems within a package, while optimizing system-level performance and power efficiency and minimizing both unit cost and time-to-market.

Share

Why Chiplets?

Two primary imperatives are driving the transition to disaggregated architectures and the use of discrete chips (or chiplets) within a single package using advanced wafer-level packaging technologies.

Functional imperative: A growing class of applications, primarily those serving high-performance computing workloads (e.g. datacenter server processors and machine learning accelerators) have scaled up in size to a point where they need more transistors than can fit on a single monolithic printed chip (i.e. a chip that is at or near the reticle limit imposed by lithography tools, ~800mm²). Recent examples are Intel Xeon and AMD EPYC server CPU processors and NVIDIA datacenter GPU processors (e.g. Blackwell). Regardless of other technological limitations, the sheer need for more transistors (typically > 100 billion) can only be met by increasing the total silicon footprint beyond a single reticle, which necessitates combining multiple chips within a single package, either side-by-side (2D or 2.5D) or stacked atop each other (3D).

Economic imperative: For a large class of computing and communication applications, chiplets enable architects and designers to overcome technological barriers (e.g. lower chip yields for large chips) or to improve unit economics (e.g. manage rising wafer prices of advanced nodes) or to improve time-to-market (e.g. by enabling re-use across product generations). In these applications, disaggregation of a large chip into multiple chiplets is more a choice and not so much a functional necessity. Recent examples include client CPU processors from Intel (e.g. Meteor Lake).

Regardless of the imperatives (functional or economic), a few common bottlenecks must be addressed to enable optimal system level performance when using disaggregation and chiplets. These bottlenecks will be critical to resolve in order for the chiplet ecosystem to develop – much akin to how the IP ecosystem developed a couple decades earlier.

Enabling Chiplets for All

Large incumbent chip companies have the scale and resources to independently develop and deploy complex new heterogeneous architectures with minimal support from the larger semiconductor ecosystem. Several such systems are already in production as mentioned earlier.

However, smaller and new chip companies, especially those designing systems for performance and bandwidth hungry workloads are likely to need help from the larger ecosystem to deploy such complex, disaggregated chiplet based architectures.

It should be noted that early generations of disaggregated architectures are just starting points and likely to keep evolving until system optimization is better understood – learning from these initial deployments will help improve system level understanding and drive the development of tools and techniques to further improve system level performance. In addition, advances in packaging technology (e.g. bump pitch scaling, direct Copper-to-Copper bonding and multi-level stacking) will continue to enable additional bandwidth at lower latencies, providing further room for system level performance enhancement.

The partitioning of a monolithic chip into multiple discrete chiplets needs to account for optimal placement of cutlines across the chip, for holistic design of memory hierarchy within a single chiplet and across multiple chiplets and for holistic design of intra- and inter-chip connectivity. Only then will system-level performance be able to match or exceed that of monolithic floorplans.

A variety of tools and techniques will be necessary in order to truly democratize chiplet design and enable it for the broader ecosystem:

Reference design platforms: EDA tools that assist in evaluating partitions of monolithic designs across various cut lines to determine best return on power and performance. Major EDA providers are already developing such capabilities.

Software-defined architecture: Given the ever-increasing complexity of large silicon systems, any exploration of multi-chip architectures can no longer rely on custom manual experiments. Such studies must be software and algorithm driven in a way that helps architects explore wider design and process spaces and simultaneously optimize multiple variables to guarantee system level performance.

Holistic cache hierarchy: Partitioning total available cache across a complex system while maintaining intra-die and die-to-die level coherency is a critical requirement, especially for high performance, memory hungry computing systems. Defining a holistic, coherent cache architecture is a major pre-requisite of complex multi-chiplet architectures.

Unified fabric architecture: Energy-efficient data movement across various components is a critical requirement for complex multi-chip systems. System architects need to be able to customize data movement in a way that enables coherency and scalability across a wide range of workloads and applications. Ideally, fabric architectures must also be agnostic to networking and connectivity protocols to facilitate mix-and-match in an open ecosystem.

Correct by construction: Just as the third-party design IP ecosystem evolved to offer fully validated and verified IP, guaranteed to perform to specification on a variety of mainstream foundry process nodes, so too will a new ecosystem need to evolve to offer fully validated chiplet IP, correct by construction so that system architects can integrate and deploy it into their systems with a high degree of confidence on the final system level performance and yield.

Software compatibility: Ensuring compatibility with legacy software programming models is a key requirement, especially for well-established processor systems that are migrating from monolithic to disaggregated architectures.

Open standards for chiplet connectivity: Open standards like Universal Chiplet Interconnect Express (UCIe) are necessary to make it easy to mix and match chiplets. UCIe offers high-bandwidth, low-latency, power-efficient, and cost-effective on-package connectivity between chiplets from different fabs using a range of packaging technologies.

Power delivery: EDA tools to optimize power delivery solutions across chiplets.

Fabric and Cache Architectures

In modern SoC architectures, interconnect fabrics play a crucial role in facilitating efficient communication between various on-chip components. These sophisticated networks serve as the nervous system of the SoC, enabling seamless data transfer among processors, memory units, peripherals, and other functional blocks, while maintaining coherency and optimizing performance and power. Similarly, on-chip caches play a vital role in modern SoC architectures, serving as compact, high-speed memory units strategically positioned between processors and main memory.

With increasing transistor density and chip complexity, the design and implementation of caches and interconnect fabrics have become critical factors in determining the efficiency and capabilities of modern SoCs across a wide range of applications, from mobile devices to high-performance computing systems. As seen in the figure below, bandwidth scaling has not kept up with compute scaling over several decades and memory and interconnect bandwidth is now the leading bottleneck to increase system performance. The interaction between caches and interconnect fabrics will become even more vital in improving bandwidth and reducing latency between compute and memory in highly complex, multi-chiplet disaggregated SiP architectures.

Over the past decade, fabric and cache architectures have evolved to accommodate the needs of large, monolithic SoC designs. So far, early chiplet-based disaggregated SiP designs have relied heavily on these legacy monolithic SoC frameworks. However, to fully harness the potential of future chiplet-based systems, a fundamental redesign of fabric and cache architectures is crucial. The traditional flat architectures of the SoC era must give way to multi-level compatible, hierarchical architectures – including hierarchical namespace, hierarchical routing, hierarchical networks, as well as hierarchical caching and coherency that can scale effectively across diverse disaggregated systems.

Just as CAD/EDA tools raised the abstraction in circuit design and simplified chip architecture during the SoC era, new software tools that facilitate multi-variable architectural explorations to determine optimal fabric and cache partitioning will prove to be crucial in the SiP era.

Baya Systems

Baya Systems is a new company that launched in June 2024 and aims to develop a software-defined approach to help architects build caches and fabrics that meet the needs of complex multi-chiplet designs.

Baya Systems is developing software that facilitates exploration across a range of chiplet architectures and tuned for a variety of representative workloads. These tools can allow architects to optimally design and partition system cache within and across chiplet boundaries while also providing for highly scalable custom fabric IP that is compatible with a host of industry standard protocols. Baya also aims to be the first to offer IP with multi-level cache coherency, solving a critical performance bottleneck for complex multi-die systems.

The views expressed herein are the authors’ own.

“I've never seen any technology advance faster than this. Artificial intelligence compute coming online appears to be increasing by a factor of ten every six months. Obviously, that can’t continue at such a high rate forever but…I've never seen anything like it…the chip rush is bigger than any gold rush that has ever existed.”

Elon Musk, Bosch World Conference, March 2024

The semiconductor industry has periodically been reshaped by tectonic shifts in the broader computing landscape. While the foundational geometric scaling of silicon technology has followed a secular trend established by Moore’s Law, every successive era of computing over the last six decades has fundamentally transformed the semiconductor industry that drove it forward. We are in the early days of the next inflection.

The PC era that began in the 1980s established the CPU as the enabling silicon platform, which in turn shaped the evolution of semiconductor technology and established the pre-eminence of the Integrated Device Manufacturing (IDM) business model over the following decade. The mobile era that began in the mid-2000s established the mobile SoC as the enabling silicon platform and established the pre-eminence of the foundry-fabless ecosystem over the decade that followed. The AI era is in its early days and has already begun to alter the contours of the semiconductor landscape.

Three major shifts are re-defining the semiconductor landscape today. Looking back a decade later, these are likely to be the foundational transformations in the semiconductor industry catalyzed by the emergence of AI computing.

Subscribe now

One: From CPUs to GPUs

“…because of this new machine learning paradigm, the kinds of computations we want to run are quite different than traditional handwritten twisty C++ code that a lot of basic CPUs were designed to run effectively, and so we want different kinds of hardware in order to run these computations more efficiently. And we can actually in some sense focus on a narrower set of things we want computers to do and do them extremely well and extremely efficiently, and then be able to have you know that increasing scale actually be even more possible.”

Jeff Dean, Chief Scientist, Google DeepMind and Google Research, February 2024 (Link)

In so much as the Central Processing Unit (CPU) has been “central” to high performance computing in datacenters, that central role is now increasingly being fulfilled by the Graphics Processing Unit (GPU). A more appropriate technical descriptor of this shift would be “from Scalar to Vector” or “from Scalar to Tensor”, reflecting the transition in the underlying computer architecture.

The “GPU” moniker is now an amusing misnomer since GPUs that enable AI computing are no longer dedicated graphics processing units at all. They are simply general-purpose, programmable shader-based vector computers that happen to be highly power-efficient for massively parallelized machine learning workloads that tend to be dominated by matrix multiplication and other linear algebra primitives. While GPUs are the dominant platform for machine learning today, other custom-designed vector or tensor-based architectures can be equally if not more efficient for reduced precision linear algebra-based workloads.

“…it seems like the world is going to be very compute constrained for a while and I think almost all of the datacenters around the world will over time be AI instead of conventional CPUs…if you ask what percentage of the energy (in datacenters) is being used for neural nets, it's going to over time be probably 80-90%...”

Elon Musk, January 2024

In the PC era, the CPU quickly became the primary volume driver and largest revenue generator for the semiconductor industry and in turn, became the foundational platform that drove innovation in silicon technology. As the primary beneficiary of this shift to CPU-centric computing, Intel was able to set the pace and direction of Moore’s Law during the PC era. Transistor technology was defined and optimized to be “CPU-first”, as evidenced by Intel’s groundbreaking transistor innovations through the 1990s and 2000s. Foundry suppliers would later adapt and waterfall these innovations to suit the needs of other applications.

In the mobile era, the Application Processor Unit (APU), which integrated a CPU and a GPU with a host of other functionality in a single System on a Chip (SoC) drove 10X higher unit volumes than the standalone CPU and became the primary volume driver for the semiconductor foundry ecosystem. Over the last decade, the dominance of the mobile SoC platform ensured that the APU set the pace and direction of Moore’s Law – Apple’s iPhone roadmap has an outsized influence on TSMCs process technology cadence and roadmap. Transistor technology is now optimized to be “mobile SoC-first”, to be later adapted for other applications.

An interesting question then is how this dynamic may shift in the AI era – the datacenter GPU has already established itself as the primary enabler of generative AI model training workloads and has also driven exponential revenue growth for product designers (e.g. NVIDIA) and silicon foundry suppliers (e.g. TSMC). GPUs will over time drive increasing wafer volumes for the foundry suppliers. In addition, since datacenter GPUs are a system solution rather than a discrete chip solution, they already drive large (if not the largest) unit volumes for system integration and advanced packaging technologies (e.g. CoWoS at TSMC). NVIDIA is now TSMCs second largest customer by revenue (Link) and the GPUs it builds are bound to play an outsized role in setting the foundry process and packaging technology cadence and roadmap.

“There’s about a trillion dollars’ worth of installed base of data centers. Over the course of the next four or five years, we’ll have two trillion dollars’ worth of data centers that will be powering software around the world.”

Jensen Huang, January 2024

Even if one assumes a $500B computing infrastructure buildout (half of Jensen’s $1T estimate) over the rest of the decade, it still implies massive demand growth for datacenter GPUs in the coming years. Datacenter GPUs used for AI training drive large revenue, but significantly less wafer volume compared to the CPU. Over time, if GPUs become the enabling platform for AI inference workloads too, then they will drive far more volume, comparable to, if not eclipsing that of the mobile SoC. If this trend holds, then the GPU could become the new “central” processing unit.

“Inference is incredibly hard…the goal of somebody who's doing inference is to engage a lot more users, to apply the software to a large installed base…And so, the problem with inference is that it is actually an installed base problem and that takes enormous patience, years and years of success and dedication to architectural compatibility.”

Jensen Huang, March 2024 on why he believes GPU solutions will win in AI inference.

Share

Two: From Chips to Systems

A defining trait of the PC platform was its ability to become easily modularized. This enabled the PC industry to become horizontal and allowed a wave of independent original equipment manufacturers (OEMs) and original device manufacturers (ODMs) to thrive. Michael Dell famously started assembling PCs from his dorm room and grew his startup into a multi-billion-dollar enterprise. ASUS, Compaq, HP and many other PC vendors were able to build highly successful businesses by assembling the outputs of discrete component manufacturers. And because the CPU was the primary unit of compute for the PC, Intel became the largest semiconductor beneficiary of this modularization. Even though Intel only supplied one component (the CPU), it enjoyed an outsized revenue share in the PC market.

In the mobile era, some single chip (or single component) suppliers did become large players (e.g. Qualcomm’s modem chip), but they were unable to reach the dominance that the CPU enabled for Intel in the PC era because the largest revenue share in the mobile era went to companies building integrated systems rather than single chips. Apple is the most extreme example of this value aggregation. Android phone manufacturers (e.g. HTC, Samsung, Google) were successful because they also achieved some level of system integration, but they were not able to command the profitability of a fully integrated provider like Apple. More importantly, full integration enabled Apple to arguably build a better product and hone high-skill manufacturing processes and technologies that became a sustaining competitive advantage for them and their outsourced assembly partner (Foxconn).

An equally significant, but relatively unnoticed shift is underway in how modern AI computing systems are built and who is building them. NVIDIA is no longer just a chip (or even a graphics card) supplier - it builds the entire computer – the DGX box integrates 35,000+ components and weighs 70 lbs. Designing and building such a box is a high-skill and lengthy manufacturing endeavor in itself, much akin to how Apple designs and builds the iPhone. Moreover, less than a dozen of those 35,000+ components are advanced node silicon chips, and these include not just compute (CPU and GPU) chips, but custom designed high-speed network processor chips as well. NVIDIA has become far more than a merchant chip company or even a semiconductor company – it is a systems (and software) company, designing complex computers and getting them built via a vast supplier network along a complex, global supply chain – an incredibly wide moat for any traditional discrete chip company to cross.

NVIDIA also offers an authorized reference design platform (HGX100) which third party OEMs can build upon to create custom variants. Pure-play third party OEMs can assemble such systems too, just as they assembled PCs and server systems earlier. However, given the complexity of these new systems and the tight integration required across every layer of abstraction up to and including software, it remains to be seen if pure-play third party integrators will be able to achieve best-in-class performance without any reference designs from the primary vendor.

In essence, the modular unit of compute is shifting from a chip (e.g. a discrete CPU) to a tightly integrated system in a box (including CPUs, GPUs, networking, DRAM, I/O peripherals and much more). If the unit of compute for the datacenter is shifting from a chip to a box, it is highly integrated systems companies that will get an outsized share of the total market revenue. It is interesting to note that AMD as a discrete silicon provider (with the MI300 AI accelerator) gets just a fraction of the share of wallet that NVIDIA enjoys by selling the entire DGX computer (Link). Similarly, Intel Xeon CPUs are a part of each DGX box that NVIDIA sells – yet Intel gets just a fraction of the premium that NVIDIA earns by selling the integrated box. As the unit of compute shifts from the chip to the system, the lion’s share of semiconductor revenue in the AI era will go to systems companies and not to discrete chip companies.  

Share

Three: From Horizontal to Vertical

“Vertically integrated firms will often dominate in the most demanding tiers of markets that have grown to substantial size, while a horizontally stratified, or disintegrated, industry structure will often be the dominant business model in the tiers of the market that are less demanding of functionality.”

Clayton M. Christensen (Link)

In the 1990s, Andy Grove astutely observed the transformation of computing from a vertically integrated business to independent and horizontal businesses. This transformation provided the catalyst for integrated device manufacturers like Intel to thrive in the PC era. Intel famously established the predominant computer architecture (x86) and by combining it with the ability to manufacture the most advanced chips at scale was able to become the largest semiconductor company in the world. The mobile era broke the dominance of the integrated manufacturing and design model and paved the way for the foundry-fabless business model to thrive. The AI era is facilitating the rise of computing as a pseudo-vertically integrated business. There are several strategic reasons that have driven this trend for over a decade, but the rapid rise of generative AI in just the last 2 years is accelerating this transformation and will have a significant impact on the traditional semiconductor industry landscape.

AI computing, especially datacenter training represents the most demanding compute workload for semiconductors today and has now reached a scale where vertical integration down to silicon is not only economically viable, but strategically vital, exactly as observed by Clayton Christensen. Seven of the world’s ten largest companies are now designing their own custom silicon (ranging from small ASICs to state-of-the-art general-purpose CPUs, networking processors and many other chips). The large cloud providers and technology companies started by building discrete silicon chips – but they quickly evolved to designing entire computing systems themselves – Google’s TPU Pod or Tesla’s Dojo are examples of this trend. It is interesting to note that the modular and horizontal semiconductor ecosystem that burgeoned and matured during the prior two eras of computing is now enabling today’s largest companies to easily re-integrate vertically down the computing stack – with silicon wafer fabrication, packaging and assembly being the only functions that still need to be outsourced. Companies like Amazon started out by designing networking processors and within a few years began designing domain specific accelerators and general-purpose CPUs as well.

I wrote about custom silicon development at AWS and Apple earlier:

Microsoft, Amazon, Google, Tesla, Meta and others are designing not just their own chips, but entire high-performance computing systems, much like the NVIDIA DGX box described earlier.

OpenAI which doesn’t even operate its own datacenters yet is assembling its own hardware team too (Link). These companies will leverage the existing semiconductor manufacturing and assembly ecosystem to varying degrees to build their own compute systems designed and optimized for their most prevalent and strategic workloads. As compute capacity becomes one of the most vital technology assets in the AI era, the strategic incentives for vertical integration will only grow stronger over time and will likely change the balance of power in the pure-play semiconductor design landscape. 

Summary

Every era of computing has initially been supported a plurality of competing hardware and software solutions, but eventually the majority share of market revenue has disproportionately gone to just one or two dominant entities: IBM (mainframes), DEC (minicomputers), Intel + Microsoft (PC), Apple + Google (Mobile). Companies that successfully achieve the transition from discrete chips to integrated systems (including software) and from scalar-based architectures to tensor-based architectures will be best positioned to win in the AI era of computing. Interestingly, the incumbent merchant chip and system providers will be competing fiercely not only amongst themselves and a new crop of silicon, systems and software start-ups, but more so against their own customers who are integrating vertically down to silicon and building custom silicon chips and systems to displace their suppliers.

The chip wars for supremacy in the AI era have begun – and this time, winning the battle will require more than just chips.

The views expressed herein are the authors’ own.

“What is this machine that you are trying to create? What is its output, what is its input, what are the conditions that it is in? What is the industry like? Is it a fast-moving industry? Is it bureaucratic? Is it highly regulated? What kind of industry is it? And what are you trying to build?”

Jensen Huang, on the importance of first principles thinking when creating and running a company.

NVIDIA is the only chip company and one of the only large public technology companies that has been continuously led by a founder-CEO for over 3 decades since its founding – and Jensen Huang is one of the longest serving technology CEOs in Silicon Valley1. Jensen rarely speaks publicly about his management philosophy but in recent months, he gave a couple of candid interviews2 about his leadership style and company culture. These interviews offer a direct view of his leadership philosophy and shed some light onto the culture of a unique, founder-led company, which is now one of only 8 publicly traded companies to cross a market valuation of one trillion dollars. This essay summarizes the key tenets of Jensen Huang’s leadership philosophy, as distilled from his interviews.

Develop and Trust Your Intuition

“If you work in a very technology driven industry, it is essential that you understand the underpinnings of the technology so that you have an intuition for how the industry is going to change. You need to have an intuition for which one of the technologies is a bit of left turn and which one is fundamental to realize that may be the early work we did with generative adversarial models to variational auto encoders to diffusional models – they were somewhat cousins of each other and that realizing the impact of one could lead to breakthroughs in another which eventually opened up the horizon for diffusion models which are utterly incredible.”

Jensen Huang

Jensen talks about the importance of intuition, especially for leaders of fast-moving technology companies. Having an intuition for technology allows a leader to better extrapolate the future, which is essential because while technology changes very fast, it takes several years to build a great solution based on that technology. Striking a balance between building something that takes years to do while using technology that is changing by a factor of 1000 every few years – Jensen argues can only be done with an innate intuition, curiosity and conviction that is essential to run a technology driven company.

Shun Commodity Work

“We never talk about market share in our company…because the concept of market share says that there are a whole bunch of other people who are doing the same thing. And if they are doing the same thing, then why are we doing it? You know, why am I squandering the lives of these incredibly talented people to go do something that's already been done? And so, we tend not to go fight people for market share, fight people for markets that that that are already commoditized and so, that's one way of thinking. To go do something that's never been done before. The other way to demonstrate that is to walk away from businesses that that have been commoditized. Either through our own initiative or otherwise, we've walked away from many businesses in the past and so that demonstrates very clearly to our employees that we're not going to do commodity work and so the combination of choosing the right work and walking away from the wrong work, that is the best way...”

Jensen Huang

Avoiding commodity work is a core business principle for Jensen. He proactively focuses the company on something that has never been done before and proactively walks away from businesses that have already been commoditized. This is a way to “naturally attract” the most amazing people to the company and also to make the best use of these talented employees and keep them motivated to do their life’s work at NVIDIA.

Subscribe now

Flat org – 40 Direct Reports and No 1:1s

“A company's architecture should not be generic. Every company in the world should not be built like the US military. And in fact, if you look at every company's org chart in the world, they kind of look like the US military…the number of direct reports of CEOs are very few and the number of direct reports of the people who are just learning how to manage, the first level managers are very large. It's exactly the opposite of how it should probably be architected.”

Jensen Huang

Jensen’s guiding principle at NVIDIA is to keep the company as small as possible. This naturally requires him to empower people, which in turn necessitates having a flat org structure. Jensen has ~40 direct reports, highly unusual for a public company CEO. He does not have regularly scheduled 1:1s with any of his staff.

“I wanted a company that was smaller, not larger. You want a company that's as small as possible, not as large as possible. It needs to be as large as necessary to do the job well but should be as small as possible. And so naturally you want to empower people. Well, if you want, if you want an organization that obeys command and control, then you make it like a pyramid, just like the military all the way back to the Roman Empire. But if you want to empower people, then you want to make it as flat as possible so that information travels quickly. In order to make something as flat as possible, the first layer has to be well considered. Well, the first layer happens to be the most senior people and you would think that they need the least amount of management. None of my management team is coming to me for career advice. They made it, and they're doing great. So, I have a whole lot of people reporting to me because I don't need to do one on ones. I don't have to do career coaching.”

Jensen Huang

No Business Units and No Divisions

“We don't have business units; we don't have divisions. Everybody works as one. And the company is shaped in a way that allows us to build accelerated computing best. If you ask me to go do fried chicken, we will have a hard time doing fried chicken. But accelerated computing, we do very well.”

Jensen Huang

No Status Reports

“We don't do status reports. I don't read any status reports. And the reason I don't is because status reports are meta-information by the time you get them. They are barely informative. They have been distilled and refined and bias has been inserted, perspective has already been added and you're not looking at ground truth anymore. I tend to appreciate information that that anybody presents. So, any employee can send out an e-mail called “Top Five Things” and it can be just whatever happens to be their top five things, whatever they observed or whatever they did or whatever they learned. If you send it out, I will read it.”

Jensen Huang

Jensen categorically refuses to read status reports. Instead, employees are encouraged to send out their list of “Top Five Things” summarizing information that they want everyone including the CEO to know. Every morning, Jensen reads about one hundred “Top Five Things” emails that are sent to him from people all across the company. He uses these emails to “stochastically sample the system” and get a feeling for whether the company is going in the direction he wants it to go. For example, Jensen looks out for whether employees’ “Top Five Things” reflect real actions and execution (not just words) around the company strategy. Jensen deliberately doesn’t broadcast his own “Top Five Things” because if he did, he would be “contaminating the system”. He keeps his list to himself.

“Top Five Things” appears to be an informal variant of the more conventional OKRs (Objectives and Key Results). But it allows for quicker, stochastic, and more real-time checks on the state of the organization.

Subscribe now

No 5 Year Plan; No 1 Year Plan; No Plan!

“We don't do a periodic planning system. And the reason for that is because the world is a living, breathing thing. And so, we just plan continuously, there's no five year plan, there’s no one year plan. There’s no plan! There's just…what we're doing.”

Jensen called out “giant five-year plans” as “horrible” and “ridiculous” for technology companies, simply because technology is always evolving at a much faster pace. Instead, Jensen advocates a “continuous planning” system where the company is constantly observing and adapting to a fast-changing world. This requires formulating a view of the world based on first principles thinking, trusting your intuition and acting with conviction. Along the way, if he realizes he is wrong, then he calls it out and changes course right away – as many times as necessary – in real time – he calls this ability to change “the beauty of modern leadership”.

What Jensen calls “continuous planning” is analogous to the “Observe, Orient, Decide, Act” (OODA loop) approach developed by US Air Force fighter pilot John Boyd. So long as your OODA loop is faster than your competitor’s, you are likely to come out ahead.  

Reason Out Loud with Employees

“I spend a lot of time reasoning through my decisions, which empowers employees because they learn how leaders think through problems. In every meeting I'm in, I'm explaining how I think through the situation; let me reason through this; let me explain why I did that; how do we compare and contrast these ideas. That process of management I think is really empowering.”

Jensen Huang

When asked about his leadership style and how it had evolved over time, Jensen called out being direct as a key attribute. If he doesn’t like the way something is going, he likes to be direct and say it. He doesn’t like to convey his decisions via 1:1s or by taking individual employees aside; if he has a different opinion, he just says it aloud in a group setting. The idea, according to him is to ensure that everyone has all the context, all the time. He proactively spends a lot of time reasoning through his decisions in group settings so that employees understand and appreciate his thought process behind important decisions.

Allow Information to Travel as Quickly as Possible

“We don't do just vice president meetings or just director or board meetings. The meetings I have, there's new college grads in there. There are people from every different organization. We are just all sitting in there. You want the person who is most informed, or best skilled or just has the most experience. They either made the mess, or they actually confronted the situation. You want ground truth and experts, the best as you can.”

Jensen Huang

Jensen’s approach is to communicate to all stakeholders at once so that there are no silos formed within the organization. His rationale is that once you have a formulated a strategy, it doesn’t make sense to tell just a few people about it, he would rather tell everybody at once. He welcomes feedback from anyone and is open to refining his strategy. He believes there are very few secrets at NVIDIA, and this is another way to empower employees and make them feel valued.

As an example, Copper wires on a chip were enabled by the invention of the chemical-mechanical polishing (CMP) process in the 1980s and subsequent deployment by the semiconductor industry in the 1990s. The planarization process to make nanometer scale copper interconnects on silicon chips today is commonly referred to as the Damascene process. 

Building more transistors and increasing on-chip transistor density was just one ingredient in realizing the promise of Moore’s Law. Equally important were innovations that enabled chip designers to effectively use the exponentially growing transistor counts to build increasingly complex circuits. And just as important were innovations that enabled architects to envision and build increasingly sophisticated and capable computers with the abundant transistors.

Accelerating Returns Built Upon Diminishing Returns

Innovations that exponentially increased transistor density; innovations that enabled designing with exponentially more transistors; innovations that enabled building more sophisticated computers; and innovations that enabled more user-friendly computing devices – all followed a similar evolutionary pattern. Progress in nearly every major innovation was defined by a characteristic S-curve with a long period of gestation, followed by rapidly accelerating progress and widespread adoption and finally, an extended period of evolutionary or incremental progress along a diminishing returns curve.

Every major shrink in transistor geometry was enabled by a combination of many innovations, each traversing a different point along its own diminishing returns curve. The aggregate of all these innovations delivered a near doubling of transistor density with every shrink. In other words, accelerating returns in transistor density were enabled by the aggregate of diminishing returns from many discrete innovations in materials science, engineering, solid-state device physics and many other disciplines.  

Building More Transistors

Some transistor geometries (aka nodes) were anchored around major process, material or architecture inflections (e.g. SiGe to introduce uniaxial strain in the transistor or the FinFET architecture to improve transistor electrostatics). When such changes were first introduced into volume manufacturing, they delivered substantial, step-function improvement along one or more specific vectors (e.g. higher drive current with strained silicon or lower leakage current with the FinFET) and with every subsequent geometry shrink, the innovation continued to be refined and improved until it reached a point of diminishing returns. For example, the FinFET architecture was refined over 5 geometries until it finally began to run out of steam. The next evolutionary architecture (RibbonFET) has already been in development for many years and will be ready to be introduced by the time the FinFET runs its course – and the cycle will then repeat. Not every transistor shrink required radical inflections in process technology and architecture. Several technology nodes were enabled purely by evolutionary improvements to build upon prior revolutionary changes. Rather than look for periodicity in transistor innovations over time or over geometry, it is instructive to look for periodicity over transistor counts. Since the microprocessor represents the most complex of semiconductor chips, it is a good proxy to use for such an exercise.

The added transistor counts occasionally led to smaller microprocessor die sizes, but often, they led to larger die sizes as architects found novel ways to add more functionality using the additional transistors. As an example, initially the microprocessor chip consisted of just a standalone central processing unit (CPU). Over time, it evolved to include more functional blocks (e.g. the floating point unit) and now consists of a highly complex and integrated system-on-a-chip (SoC) including the CPU, the graphics processing unit (GPU) and many other functional blocks including memory and IO circuits.

Transistor counts under 100 billion have generally been accommodated on a monolithic SoC. Transistor counts greater than 100 billion, while feasible on a single chip, are more likely to be accomplished via integration of multiple chips to build a system-in-a-package (SiP). Rather than innovations at the foundational transistor level, this next S-curve is likely to be at the system level, enabling architects to connect discrete digital logic, memory and analog chiplets with each other. Remarkably, this inflection from system-on-a-chip to systems-in-package was also envisioned by Gordon Moore in his seminal observation nearly 60 years ago:

“It may prove to be more economical to build large systems out of smaller functions, which are separately packaged and interconnected. The availability of large functions, combined with functional design and construction, should allow the manufacturer of large systems to design and construct a considerable variety of equipment both rapidly and economically.”

Gordon Moore, 1965 (from the original paper and prediction that became Moore’s Law)

Share

Designing with More Transistors

Over time, engineers relied on computer-aided-design (CAD) tools to manage the growing complexity of chip design. This evolution was covered in Part 1 when discussing abstraction layers:

CAD evolved to become an industry in its own right and later came to be known as Electronic Design Automation (EDA). Here again, with every 10X increase in transistor count, there was at least one significant EDA innovation that progressively raised the level of abstraction and made it easier for designers to manage large and complex chip designs. Looking ahead, we are already seeing artificial intelligence (AI) driving the next major S-curve in EDA.

Subscribe now

Creating Computers

With CAD enabling increasingly complex chip designs, architects were able to envisage and build more sophisticated microprocessors. All modern computers are based on the von Neumann architecture which introduced the concept of storing instructions and data in a shared memory. Astonishingly, this architecture was conceived in 1945, long before the invention of the transistor (1947), the integrated circuit (1958) or the advent of modern semiconductor processing (1960s) that would enable it to be brought to life in silicon.

Starting with early innovations that included the addition of local cache memory, there have been numerous innovations in computer architecture that progressively took advantage of the exponentially growing transistor density. On average, every 10X increase in microprocessor transistor count enabled the introduction of a major innovation in computer architecture. Over time, progress in computer architecture came from three major ideas or S-curves – wider instruction widths (from 4-bit to 64-bit), instruction-level parallelism (from many cycles-per-instruction to many instructions-per-cycle) and core-level parallelism (from single-core to dual-core to many-core). Notable among these, instruction-level parallelism eventually led to two distinct architectural paradigms – “found” parallelism, commonly referred to as “out-of-order” processing which became the foundation of all modern CPUs and “given” or “explicit” parallelism which became the foundation of all modern GPUs. All this progress was enabled not only by increasingly abundant transistor counts, but also by increasing clock speeds (from a few Hz to a few GHz) with faster transistors.

In recent years, architectural innovation also included the incorporation of domain specific computing blocks (e.g. integrated graphics engines and neural engines). Looking ahead, computer architects will envisage new machines optimized from the ground up for machine learning and AI workloads. The next S-curve will also necessitate a new hardware-software contract for the computers of tomorrow.

Using Computers

Innovations in computer architecture drove the creation of increasingly sophisticated computing systems with progressively denser compute and storage capabilities and richer functionality. These innovations combined with a multitude of advances in hi-tech design, manufacturing and assembly enabled system and product architects to imagine and invent newer classes of computing devices in a variety of form factors from mainframes to smartphones. Here again, a periodicity with microprocessor transistor count is evident as a new class of computing emerged with every 10X increase in transistor count. The evolution of computer classes over time was first observed by Gordon Bell and is commonly referred to as Bell’s Law of Computer Classes.

What is Next?

Over the last 6 decades, innovations that enabled Moore’s Law facilitated the miniaturization of computing form-factors – from mainframes and workstations to the mobile computing devices of today. The silicon platforms that enabled these applications in turn became the primary drivers of continued innovation in process technology, design and architecture and defined the trajectory of the semiconductor industry for decades. As an example, the general-purpose CPU platform drove innovation during the PC wave of computing while the mobile Application Processor Unit (APU) platform drove innovation during the mobile wave of computing. Three broad trends are likely to drive the trajectory of transistor scaling and innovation in the semiconductor industry in the coming years:

First, diminishing returns in transistor density and increasing process cost will lead to diminishing improvements in cost per transistor. Many, if not most mass-market consumer electronics products may not be able to justify the increased cost and diminished returns on investment when scaling from one node to the next. This trend has already been evident for a few years. A decade after the introduction of the FinFET transistor, the last non-FinFET transistor process still commands the highest wafer volumes on par with the leading-edge node today. For the first time in 15 years, Apple now deploys the same mobile APU chip to power two consecutive annual iPhone models. Evolutionary improvements on mature technologies will continue for much longer than in the past as the cadence between revolutionary enhancements continues to lengthen.

Second, the ascendant AI wave of computing will favor highly parallelizable, tensor-based architectures as the primary computing engines in datacenters worldwide. These new architectures and platforms will displace the traditional general-purpose CPU and play an increasingly important role in driving innovation in transistor technology and more broadly in setting the trajectory of the semiconductor industry.

Finally, the datacenter itself is becoming a modular compute platform. High-performance computing systems are relatively more resilient to increasing wafer prices and hence more likely to serve as primary drivers of Moore’s Law in an era of flat or rising cost per transistor. Rather than drive a miniaturization of form factors, these high-performance computing applications require advanced packaging and system integration technologies to drive large-scale integration at package-, board- and rack-level. Innovation will be needed in a variety of areas including advanced logic, high-density memory, high bandwidth connectivity, thermal management and power delivery. The NVIDIA HGX100 module, Tesla Dojo and the Google TPU pod are examples of this trend – while they leverage advanced process technology to build individual compute, storage and networking chips within a module, it is the entire module or pod with tens of thousands of individual hardware components, along with supporting firmware and software APIs that forms an individual unit of computing. Merchant silicon providers that specialize in a discrete single-chip solution will be at a disadvantage over system and platform providers as the locus of innovation moves to holistic system integration technologies to enable the AI wave of computing.

The views expressed herein are the authors’ own.

Subscribe now

The microprocessor has embodied progress in computing for over five decades. Remarkably, transistor counts in a microprocessor have continued to double every 2 years, regardless of the underlying microarchitecture, chip designer or silicon manufacturer. By 2030, we are likely to witness microprocessor products with over 1 trillion transistors in a package!

Understanding and appreciating what actually enabled the Moore’s Law exponential to continue for the last six decades may help us anticipate what is next in computing. This multi-part essay is an attempt to articulate the inner mechanics that propelled the evolution of computing and understand how exponential growth was realized and sustained at-scale for so long.  

Managing Complexity: From Atoms to Bits

Gordon Moore’s seminal observation specifically referred to increasing transistor counts on a wafer. Yet, realizing the promise of Moore’s Law required innovation not just in transistor process technology, but across every domain of computing including software.

As transistor counts increased over time, every individual building block from silicon processing to circuit design to computer microarchitecture all the way up to the software witnessed an exponential rise in complexity. Continued progress in computing was predicated upon the effective management of this complexity. This was achieved by breaking down large, seemingly intractable domains of computing into smaller more manageable ones. In fact, one of the primary enablers of Moore’s Law was the refactoring of computing from a single monolithic enterprise to a stack of independent enterprises – in well-defined abstraction layers from atoms to bits – with clear hand-offs and dependencies between layers that drove innovation and progress within each layer.

Over six decades, progressively refactoring and raising the level of abstraction has not only enabled engineers to manage exponentially growing complexity but it has also given structure to the business and ecosystem of computing and created entirely new market segments from foundries and equipment manufacturers working at atomic scale to cloud service providers and software developers working at datacenter scale.

The evolution of computing over six decades from a predominantly vertically integrated business to a well-defined stack of businesses that operate independently yet rely on each other to deliver progress in computing as a whole. With every passing decade, individual layers in the computing stack continued to establish themselves as standalone businesses and market segments.

For example, in the early years, it was the norm for a single semiconductor company to be vertically integrated and own nearly every aspect of its computing products – not just the design of chips and manufacturing of wafers, but also intermediate steps like fab and test equipment, software to aid circuit design, packaging and board-level assembly and testing. Over time, the investments required to remain competitive in each of these individual domains far exceeded the returns on those investments. Few single companies operated at the scale necessary to make these investments cost-effective. This led to the creation of standalone assembly and test providers (1960s), standalone semiconductor equipment manufacturers (1970s) and standalone Electronic Design Automation (EDA) vendors (early 1980s). By the late 1980s, pure-play semiconductor foundries emerged, allowing the creation of standalone “fabless” semiconductor design houses. The 1990s saw the emergence of standalone, reusable Intellectual Property (IP) providers that enabled designers to further abstract away complexity and speed-up chip design. Even the Instruction Set Architecture (ISA), the foundation of the hardware-software contract became available as a licensable IP, further catalyzing the growth of the computing ecosystem by enabling many more companies to easily design computers. Progressively raising the level of abstraction in software enabled the rise of scores of independent application software developers. A multitude of advances in hi-tech design, manufacturing and assembly enabled original-equipment-manufacturers (OEMs) and original-device-manufacturers (ODMs) to create products in a variety of form factors from mainframes to smartphones that made computing affordable, accessible and usable by the vast majority of humanity.

The transformation of computing from a monolithic enterprise into a stack of distinct yet interdependent layers was in some ways an inevitable business outcome driven by the growing scale and investment profile needed within each layer. However, this refactoring was also vital to spur innovation, efficiency and creativity within each layer. It is arguable that this stratification was an essential element in realizing the full promise of Moore’s Law and managing the complexity spiral unleashed by exponentially growing transistor counts.

Abstract and Refactor

Managing complexity via refactoring and abstraction has been the hallmark of nearly every major computing domain. In the absence of such abstraction, it would have been impossible for the various elements of computing to keep pace with the increase in transistor density driven by Moore’s Law. Breaking down large problems into more tractable ones also enabled a vigorous competition of ideas within each layer.

A classic example of the use of abstraction is in networking, where these layers provided a modular and hierarchical approach to the larger idea of networking, allowing different components and protocols to work together to enable reliable and efficient communication across networks. Such a conceptual framework facilitated the understanding of network protocols and communications and allowed the progress of an entire industry via numerous independent businesses.

Nearly every domain of computing evolved via abstraction and refactoring into individual layers, enabling the effective management of growing complexity within every single layer.

To manage the growing complexity of chip design, engineers relied on computer-aided-design (CAD) tools to help automate chip design. CAD evolved to become an industry in its own right and came to be known as Electronic Design Automation (EDA). With every 10X increase in transistor count, there was at least one significant EDA innovation that progressively raised the level of abstraction and made it easier for designers to manage large and complex chip designs.

A major innovation in EDA emerged with roughly every 10X increase in transistor count. Every successive innovation progressively raised the level of design abstraction.

Starting with hand-drawn logic designs and Karnaugh maps, these innovations led to progressively sophisticated software-defined tools such as HDL (Hardware Description Language) to enable designers to abstract away complexity. In recent years, the complexity of chip design has grown to require enormous amounts of compute and storage capability and a vast network infrastructure. This datacenter-scale infrastructure requirement has made it necessary for EDA tools to operate in large clouds owned by cloud service providers (CSPs) or hyperscalers. Cloud-based EDA and access to virtually unlimited compute and storage is now paving the way to further raise the design abstraction using AI-enabled chip design. In just a few years, it may be possible to have complex circuit design tasks accomplished with code which will be written not by humans, but by software following human instructions in conversational English as discussed here:

What is Next?

By the end of this decade, we will witness yet another 10X increase in transistor count within a microprocessor package – an astounding 900 billion additional transistors from today’s state of the art!

What are the different innovations that will enable this massive improvement? How will chip designers deal with the additional complexity to manage designs with hundreds of billions of transistors? What kinds of computers will architects be able to build with a trillion transistors at their disposal? These questions will likely be addressed in the same way that they have been in every prior 10X transistor increase to-date – by further raising the level of abstraction and refactoring the computing stack.

New, cloud-based, AI-enabled design automation tools will emerge to automate the otherwise untenable task of managing ultra-large transistor count designs. This will lessen the burden on chip designers, freeing them to innovate and invent novel ways to leverage trillion transistors in a package.

A new chiplet abstraction layer will likely be introduced to manage the complexity and raise the level of abstraction at a system level. This will enable system architects to pick and choose from a ready-to-order library of domain specific chiplets and focus more on connecting these chiplets rather than on designing the chiplets themselves. In the 1990s, the third-party, licensable IP revolution democratized computing by creating a rich ecosystem of independent IP companies. Similarly, a rich ecosystem of companies building a variety of domain-specific chiplets with universal connectivity standards is likely to emerge and serve as a vital component of system architecture. Such a chiplet ecosystem will further democratize innovation in computing and make it possible for many more companies to design the advanced computing systems of the future.   

“During the early stages of an industry, when the functionality and reliability of a product isn’t yet adequate to meet customer needs, a proprietary solution is almost always the right solution — because it allows you to knit all the pieces together in an optimized way. But once the technology matures and becomes good enough, industry standards emerge. That leads to the standardization of interfaces, which lets companies specialize on pieces of the overall system, and the product becomes modular. At that point, the competitive advantage of the early leader dissipates, and the ability to make money migrates to whoever controls the performance-defining subsystem.”

Clayton M. Christensen (2006)

Amazon Web Services (AWS) was founded ca 2006. Just a few years later, in 2012, this software services provider veered away from its core competency to create a small silicon chip design team. Over the following decade, this silicon team grew to build five separate product lines of custom silicon that now support the massive AWS cloud computing infrastructure. One of these chips, the AWS Nitro is currently in its sixth generation and is deployed with every single new server installed in the AWS fleet, with over 20 million installed to date.

How did a software company with no prior semiconductor experience design and deploy advanced process node silicon at datacenter scale within just a decade? Why did they choose to do so? This paper outlines the history and the motivation for custom silicon development at AWS and its impact on other hyperscalers and incumbent merchant silicon providers alike. In addition to denoting a strategic inflection point along the history of computing, silicon innovation at Amazon also serves as a case study to illustrate the management principles of founder Jeff Bezos.

The Bezos Way

James Hamilton, Senior Vice President and Distinguished Engineer at Amazon and one of the original architects of the custom silicon program at AWS outlined the history of silicon innovation at Amazon in a recent keynote and an op-ed in the Wall Street Journal. His key messages reflect the strategic decision-making process at Amazon.

Start with the customer problem and work backward to design a solution

“Many organizations define their core competency by what they do. But when you define it by what your customers need, it opens the door to new opportunities. Although this approach comes with greater risk, it also comes with a greater reward.”

This approach of working backward from a customer problem all the way to understanding the root cause was instrumental in the creation of Amazon’s original bookselling business itself and later helped create the vast Amazon retail empire and other successful ventures including Alexa and AWS.

In 2012, this approach also led AWS engineers to identify the need to off-load hypervisor and authorization functions that were traditionally executed on the server CPU itself. By off-loading these functions to dedicated hardware and software, they projected significant improvement in datacenter compute efficiency and cost by enabling all available precious server compute resources to be used as customer instances. This discovery was the genesis of the Nitro System – a series of add-on infrastructure silicon cards based on a custom designed chip called Nitro. In addition to freeing up more compute cores for customer workloads, the Nitro system also enabled a far greater degree of security and reliability for AWS cloud by serving as a gateway for all traffic into and out of AWS infrastructure and by eliminating the noisy neighbor problem inherent to traditional on-chip hypervisors. The latest generation of Nitro is made on 7nm TSMC silicon.

The power of a written narrative

“When we have a new idea at Amazon, we share it with leadership in a six-page narrative format as opposed to a presentation. This approach has many benefits: It forces the author to clarify their thinking, focuses leadership on the idea instead of someone’s presentation skills, creates context for a focused discussion, and provides a timeless reference.”

Amazon’s decision to start a custom silicon team in 2012 was rooted in two central observations that James Hamilton described in a long form written narrative titled “AWS Custom Hardware”. Jeff Bezos (then Amazon CEO) and Andy Jassy (then AWS CEO) reviewed this document which articulated the central theses for why it was vital for AWS to build custom server chips and to develop silicon expertise right down to working with foundries.

Thesis #1: Volume and scale will drive innovation on Arm

Any idea or technology that gains critical mass and commands scale and volume is likely to become the de-facto platform that drives the next wave of innovation. By 2012, about 5 years into the iPhone era, the explosive momentum of Arm in mobile computing was evident and James argued that Arm and the Arm ISA would eventually drive the next wave of innovation and move up the stack to become a big part of server-side computing. James has been a prolific and well-known blogger for years and he even articulated this vision publicly as early as 2009 (Link). AWS was thus one of the earliest and strongest proponents of Arm servers and had been encouraging Arm to innovate in the server space. The belief was that Arm mobile and IOT volumes would eventually feed the innovation necessary to drive server volumes as well.

Thesis #2: Server integration will progress from board- to chip- to package-level

As early as 2012, it was clear that datacenter server functionality and innovation was continuing to become increasingly consolidated at the board level. The AWS team posited that over time, this integration would only continue and eventually consolidate within the silicon package itself. This observation also borrowed heavily from the evolution of mobile computing where more and more silicon functionality in a phone became consolidated into the central applications processor (APU) over time. James argued then that if building custom servers at scale was central to the AWS mission and value proposition – and if server innovation was consolidating onto a chip/package – then it was vital that AWS develop the expertise to design their own chips and packages.

Both these arguments resonated with Jeff Bezos and Andy Jassy, who approved the strategy and authorized the project.

“If all of the innovation on a server is being pulled onto the chip, and if we don’t build the chip, then we don’t innovate. And that is just not a great place to be. And that’s just not how we work.”

One-way Door or Two-way Door

“We use the terms one-way door and two-way door to describe the risk of a decision at Amazon. Two-way door decisions have a low cost of failure and are easy to undo, while one-way door decisions have a high cost of failure and are hard to undo. We make two-way door decisions quickly, knowing that speed of execution is key, but we make one-way door decisions slowly and far more deliberately.”

Given the large and sustained capital investment needed, the decision to invest in custom silicon was deemed a one-way door decision. AWS management deliberately made the commitment with a long-term view and authorized the team to build and seek the expertise needed for long-term success.

Flywheels to accelerate business growth

“By improving the customer experience and lowering costs, more customers use our solutions, which allows us to invest in further improvements and cost reductions, and the cycle continues.”

Another well-known Jeff Bezos strategy is the virtuous flywheel – the idea that any initiative that attracts new customers to the Amazon platform acts as an accelerant to the growth of the entire platform. The launch of the Nitro improved the AWS experience for early customers. This led to the start of a business flywheel with early small wins building on each other to create momentum.

AWS cited the ability to design chips tuned to their workloads and infrastructure (specialization and innovation), significantly faster turnaround time and streamlined operations as ongoing benefits from their custom silicon program.

“If you can have a server within a server, it can do so much more – we can add so much value – but it cannot make the server more expensive in a material way, or it just won’t work. And if you think about that – the only way we can deliver that, the only way we can make that happen, is to vertically integrate. It means that server, which is being embedded in every one of our servers, and sometimes embedded many times, has to be absolutely as low cost as possible, which means that we are going to have to design the hardware, all the way down to the semiconductors – it means that we are going to have to get the part fabbed. The whole thing has to be done by AWS, because that is the only way we can deliver this vision.”

James Hamilton, AWS Senior Vice President and Distinguished Engineer

Working with Annapurna

In early 2012, AWS worked with Cavium (now part of Marvell) to build the first version of the Nitro chip. A big constraint the AWS team had placed on Cavium was to build the chip (which was deemed to be a server within a server) without any substantial increase in cost. The initial solution from Cavium was 10X more expensive than the AWS cost target, but they decided to take a chance and build a million units as a pilot. Once the effectiveness of the chip was established, AWS began to plan for the second-generation Nitro. By this time, the internal memo on AWS Custom Hardware had already been presented to Jeff Bezos and the strategy to start a custom silicon program was already in place. This is when James and the AWS management came across the team from Annapurna Labs, a small start-up in Israel. The two teams started working together on the production of next generation processors. The nimbleness of the Annapurna team accelerated the AWS custom silicon effort. The partnership went extremely well and by 2015, AWS acquired Annapurna Labs (Link), making the Annapurna team the nucleus of subsequent custom silicon development within AWS.

Since then, AWS has continued to expand their portfolio with six generations of Nitro System, two generations of Nitro SSD (solid state drives), three generations of Graviton (general purpose Arm-based CPU), two generations of Trainium (machine learning training ASIC) and Inferentia (machine learning inference ASIC).

Within just one decade, AWS built multiple generations of custom silicon variants for every major chip in a server platform (infrastructure/networking, accelerators/co-processors and general-purpose host processors)

Hyperscalers go Vertical

It is important to place the AWS custom silicon strategy in the broader context of the evolution of cloud computing workloads and the cloud computing services business. AWS was an early mover among the hyperscalers in the development of custom silicon. It would be several years before Google and Microsoft would pursue similar efforts. Apple, although not a public cloud service provider (CSP), owns and maintains a large computing infrastructure for its iCloud-based services. Facebook, another top cloud customer is also developing its own silicon (Link). Three of the largest Chinese cloud computing customers, Baidu, Alibaba and Tencent are each developing their own custom silicon as well (Link). There are several important reasons for this growing trend which will be explored in a future article.

The views expressed herein are my own.

This is Part 3 of a 3 part series. Be sure to check out Part 1 and Part 2 as well.

Aviation technology made remarkable progress in the first six decades – from early planes being made of wood and cloth to all-metal fuselages, from narrow-body, single-aisle jets to wide-body, double-aisle jets, and from water-cooled piston engines to air-cooled jet engines. Airplane cruising speeds had doubled three times, reaching over 600 mph by 1970. Following in the footsteps of military aviation, supersonic transport (SST) was seen as the next logical milestone for commercial aviation.

The Concorde represented the next technological leap in aviation – after the Douglas DC3 (the first modern airliner), the Boeing 707 (the first successful commercial jetliner) and the Boeing 747 (the first wide-body jumbo jet). Each of these airliners represented moonshots, bringing to the fore numerous cutting-edge technologies. The Concorde was the first supersonic, passenger-carrying commercial airplane. Built jointly by aircraft manufacturers in Great Britain and France, it was conceived in 1962 and made its first scheduled passenger flight on January 21, 1976.

Share

Engineering Marvel and Business Failure

The airplane was a marvel of engineering and introduced technologies that were remarkably ahead of their time. Supersonic flight required new jet engines twice as powerful as the state of the art, robust streamlined wings and airframes, a custom-designed, high-temperature resistant aluminum alloy for the fuselage, and temperature resistant carbon brake materials. The Concorde introduced a fly-by-wire, computer-controlled, variable geometry air intake system that ensured steady, subsonic, compressed airflow at all speeds, enabling the airplane to “super-cruise” at Mach 2 (Link) speeds  using just the compressed airflow alone. The fuel tanks were distributed all over the plane to adjust the center of gravity for takeoff, cruising, and landing. The genius design of the curved, ogival, delta wings allowed Concorde to control drag and cruise beyond Mach 2, but slow enough to fly into existing airports. Nearly all these remarkable innovations went on to become standard features on all modern airliners.

Even though the development of a commercial supersonic jet was a monumental and expensive effort, it was believed that it would eventually pay off at consumer scale; and the assumption was that over time, supersonic technology would become mainstream in civil aviation. And yet, the Concorde remains the only commercial SST airliner built to date. In 2003, 27 years after it was launched into service, the entire fleet was retired, with the full cost of development never recouped.1

This paper draws upon the prevailing circumstances during the development of the Concorde and supersonic transportation and finds intriguing parallels to the circumstances facing the semiconductor industry today.

Subscribe now

Fast Airplanes

The Concorde is a case study in how breakthrough engineering and technology does not guarantee economic and business success.

Rooted in geopolitics:

Winning the race for supersonic transport was seen as a matter of prestige and national pride. During the 1950s and 1960s, American, Soviet, British, and French governments spent vast sums of money to independently develop supersonic transportation technology. In this post-war period, as western Europe was struggling for relevance and trying to get out from under the shadows of the US and the USSR, the British and French governments decided to collaborate and formed a joint venture that would go on to build the Concorde (with the name chosen to symbolize peace, harmony, and union). The joint venture was also a way to pool precious resources to build an affordable airliner that would also establish western European aviation manufacturing capacity to compete with North American technological giants Boeing, Lockheed, and North American Aviation. The American SST program, led by Boeing was explicitly intended to defend against the potential economic impacts of the Concorde, with its beginnings dating to just three days after the announcement of the Concorde! President Kennedy introduced the National Supersonic Transport program on June 5, 1963, after intense lobbying by the FAA. The Soviets were building their own version, the TU-144.  The development of commercial supersonic transportation was thus deeply motivated by the geopolitics of the day.

Consolidation and reliance on government funding:

In addition to groundbreaking technological innovation, the Concorde required a massive influx of capital over many years, without any guarantees, and well in advance of any potential return on that investment. Indeed, the capital outlays were estimated to be so large, that private corporations could not fund it by themselves. Without government funding, it is highly unlikely that supersonic commercial transport would have seen the light of the day. By the late 1960s, the high cost of airplane development and manufacturing had led to massive industry consolidation, leaving behind just a handful of players capable of building the next big advance, the supersonic airplane. The United States had a monopoly with the trifecta of Boeing, Lockheed, and McDonnell Douglas. There was no notable airplane manufacturer in Europe, while the Soviet effort was entirely state funded. In 1962, the joint British-French program was estimated to cost over $130M. Delays and overruns ended up increasing this number to nearly $3B (1970 dollars). These costs were borne by the taxpayers, eventually written off and never recovered. From 1963 until 1971 when Congress stopped funding, it is estimated that the US government spent over $1B on the supersonic transport project (Link).

Speed as the primary design point:

The underlying premise of the Concorde was that transportation speed was the defining success metric of commercial aviation – and that it would trump other factors like passenger comfort, cabin noise reduction, reclining seats, overhead storage, legroom, bathrooms, better food, and cost. This premise led to incorrect projections of consumer interest, and in turn incorrect demand forecasts from the airlines. The development of the Concorde continued for years, without much regard to concurrent massive improvements in the incumbent subsonic airline technology (e.g., improved fuel efficiency, enhanced capacity and comfort, and better cost).

Commercial airplane cruising speeds continued to increase until they hit an economic limit nearly 50 years ago. While airplane speeds plateaued, engines continued to become more efficient, and the in-flight experience dramatically improved over the last five decades.

The Boeing 747 was introduced in 1970 (6 years before the first Concorde flight) and was one of the most spacious and comfortable airplanes at the time, with business and first class passengers enjoying unparalleled comfort and service. In contrast, seats on the Concorde were narrow and lean and headroom in the center aisle was a mere 1.8m, implying that a 6 ft. tall person would barely be able to stand up straight in the aisle! In other words, the Concorde overlooked the true needs of the mass market. A small portion of the market (business travelers) did indeed value speed. But most of the consumers favored efficiency, comfort, cost and other features far more than just speed.

Challenging economics and lack of scale:

The breakeven projections for financial success called for over 100 (and up to 500) aircraft to be built, but those were never met. The initial cost estimates of $20M per plane had ballooned to $46M by 1972, with most insider estimates predicting that the final cost would exceed $60M. Environmental and economic concerns stemming from the 1973 oil crisis forced most of the Concorde's airline customers to withdraw their orders (Link), leaving only the state-funded national carriers of Britain and France forced to bear the burden, with each airline receiving just 7 aircraft, and those too at a cost to the taxpayer. The unit cost of travel was so high that only business travelers and ultra-rich celebrities could afford the premium fare for a round-trip across the Atlantic. It used 22 tons of fuel an hour, twice the consumption of a Boeing 747 carrying four times the number of passengers as well as a significant amount of cargo. It required 22 hours of maintenance for every hour in the air, compared with eight hours for a 747. The breakeven cost on the 100-seat Concorde was $1,800 per seat, as opposed to $360 on a 400-seat 747. These absurd numbers ensured that the Concorde served only a tiny niche of travelers and became more a luxury, prestige experience than an at-scale mass-market, consumer product. Meanwhile, the Boeing 747, launched 6 years prior to the Concorde completely transformed the flying experience and the economics of air travel.

Ironically, the slower plane was far more successful in shrinking distances and bringing the world closer than the faster plane ever was.

Operating costs for the Concorde were much higher than the Boeing 747, even though the Boeing carried 4 times as many passengers.

Macro-economic headwinds:

The focus on supersonic speeds also meant that fuel efficiency, air and noise pollution were not primary factors in the design of the plane. The Concorde engine noise during landing and takeoff was among the loudest of all aircraft (119.4db). During the 1970s, heightened awareness of noise and air pollution led to protests all over the world, with many cities refusing to let the Concorde land in their airports.

Many airlines signed up for the Concorde, but nearly all of them withdrew their orders a few years later citing economic or environmental reasons. There were numerous protests, severely restricting the allowed flight routes for the airplane. 

Because of its signature sonic boom, the Concorde was launched with severe restrictions on its flight path, only allowed to fly over the ocean. These restrictions drastically altered the breakeven projections made earlier and dealt a fatal blow to the economics of the Concorde. Fuel efficiency was not a concern when the plane was conceived in the 1960s, but that changed dramatically after the 1973 oil crisis and the sharp rise in inflation that followed. By the time the plane entered service, the price of oil had increased 4X and by the end of the decade was 8X higher than at the start of the decade. These massive shifts in the macro-economic environment were ignored during the development of the aircraft, largely because of national pride. By the time the plane was ready for commercial launch in 1976, the competitive landscape in the airliner industry and the global macro-economic environment was very different than when the project was conceived and initiated in 1962.

Tiny Transistors

While it is tempting to compare the development of the Concorde and supersonic transportation to other projects and industries, similar in scope and scale, it is important to note that each industry is unique and despite many similarities, it  may not be fair to compare across disparate businesses. Nonetheless, there are intriguing parallels between the aviation industry of the 1970s and the semiconductor industry today. Some of these are highlighted below, following the same sequence used for the Concorde earlier.

Rooted in geopolitics:

Just as the early 1970s was a period of rising geopolitical tensions amid a Cold War and the East-West divide, so is the present day. The outsized dependence of the world on Asia in general, and Taiwan in particular, for semiconductor chips has further prompted governments all over the world to secure their semiconductor supply chains and minimize dependence on Asia. The COVID19 pandemic exposed simmering tensions between the United States and China, while the Ukraine war heightened tensions with Russia. Geopolitics is shaping the contours of the semiconductor industry just as much as technology itself.  

Consolidation and reliance on government funding:

For five decades, the economics of the semiconductor industry was predictable and driven by Moore’s Law. Periodic steps of evolutionary innovation interspersed with revolutionary advances kept the industry going for decades. Over time, the increasing complexity and cost of transistor scaling and advanced semiconductor manufacturing led to massive consolidation, leaving only a few players in the race, and prompting national security concerns over geographically specialized supply chains. The COVID19 pandemic triggered an unprecedented demand for semiconductor chips and a persistent chip shortage further exposed weaknesses in the global supply chain. Just as in the supersonic transport race a few decades earlier, governments all over the world are now pouring money to either build (China, India), rebuild (Japan, Europe), or strengthen (Europe, Taiwan, Korea, US) their domestic semiconductor manufacturing capabilities. Given the scale of investment needed, public-private partnerships are essential in developing these supply chains.

Density as the primary design point:

Just as the Concorde emphasized travel speed as the primary success metric, semiconductor companies today are emphasizing density (making transistors smaller) as the primary success metric. The focus on density drove down cost per transistor and delivered incredible paybacks and in fact drove the creation of the semiconductor ecosystem over four decades. But over time, the wide proliferation of semiconductor devices in every aspect of human life led to semiconductor applications that require a wide scope of features besides density. While semiconductor consumption will only increase with time, unlike the early days of Moore’s Law when the adoption rate of every successive process technology breakthrough was rapid and universal, now, the adoption rate of process technology enablers is relatively slow and not nearly as universal as in decades past. Each one of the recent major transistor technology advances has seen a diminishing pool of adopters (customers and end-applications). This suggests that the sustaining, mature node technologies will continue to be in high demand and might even command higher volumes compared to the cutting-edge, most advanced technologies. The scope (utility and features) enabled by a new technology is now just as important as the scale (number of transistors) it enables.  

Challenging economics and lack of scale:

Semiconductor manufacturers are however, highly disincentivized to build mature node fabs since the high capital expenditure required combined with a low average wafer selling price (ASP) implies an inordinately long breakeven time. On the other hand, while a greenfield investment in an advanced node fab is likely to generate higher ASPs, it is also likely to serve a diminished pool of customers, making the breakeven economics equally challenging. This poses a dilemma for semiconductor manufacturers, who need to choose where to spend their precious CapEx dollars. The cost to design, test, verify and validate a brand new chip is proportional to the complexity of the process technology, design rules and other constraints imposed by shrinking geometry and has dramatically increased over the last decade. As a result, only applications that can extract a meaningful return on investment (ROI) from transistor scaling and can support unit volumes that are large enough to justify the economics of scaling are incentivized to design on the next process technology, at least as early adopters.

Macro-economic headwinds:

After decades of low inflation and easy borrowing, inflation is on the rise again, just as it was in the 1970s. As monetary policy tightens, spending patterns are likely to change over the next few years. Given the long lead times needed to bring semiconductor fabs online, it is likely that the macro-economic environment when new fabs are ready for production a few years from now will be quite different than it is today.

Inflation hit a 40 year high in the United States this year, the fastest 12-month pace since 1981, attributed to the pandemic related supply-demand imbalance and Russia’s invasion of Ukraine. The inflationary environment today mirrors that in the mid-1970s during the development of the Concorde (Source: New York Times, Link)

An enduring lesson from the Concorde is that market needs are not always addressed by engineering advancements along a fixed vector over time. Cruising speed was an important consideration for the first five decades of commercial aviation, but beyond that, it was “good enough” and other advancements took precedence as highlighted by the resounding success of the “slower” Boeing 747. Even though cruising speeds plateaued, the in-flight experience continued to improve. Even though supersonic flight was the technologically more complex challenge, it was not a lucrative option, and hence not a good business.

The Concorde study also highlights the sunk cost fallacy. The aircraft was in development for well over a decade, during which there were tectonic shifts in the competitive, political, and macro-economic environments and in consumer preferences as well. These were all acknowledged, but still ignored because huge sums of money had already been spent on the development. Unlike the European effort, the U.S. government did stop funding for the domestic SST project in 1971, but not before spending over a billion dollars.

The Concorde and the quest for commercial supersonic travel highlights the pitfalls of being too enamored with technological milestones while losing sight of economic realities. It is a worthwhile case study for policymakers, technologists and executives as they evaluate investments in future technologies.

The views expressed herein are my own.

If you haven’t already done so, be sure to check out Part 1 and Part 2 in this series.

The plane that never was – the MD12 was proposed as a super-jumbo, 500 passenger, double-decker plane in the early 1990s. The plane was deemed too large, uneconomical to build and to operate, did not get any buyers and was never built.

McDonnell Aircraft Corporation was founded in July 1939 in St Louis, Missouri and was known for its military fighter planes. It was a major supplier to the United States defense department. McDonnell-Douglas was formed after the acquisition of Douglas Aircraft Company in 1967. The combination of McDonnell, primarily a defense contractor and Douglas, primarily a commercial aircraft manufacturer was seen as a highly synergistic one that would be able to effectively compete against Boeing. The combined firm would go on to produce 55,000 planes – but by the early 1990s, the company lost control of costs and product quality and failed to innovate. Its customers noticed and dropped orders, and the company was forced to sell itself to rival Boeing in 1997.

This paper provides a closer look at the factors that led to the fall of McDonnell-Douglas.

In the early years of the commercial jet age (1950s), long-range aircraft were built with 4 engines (quad-jets) for added safety and to boost engine thrust. Over time, as engines became increasingly reliable and powerful, tri-jets and twin-jets became more common. For airlines, engine count and passenger capacity would become foundational choices in determining fuel efficiency and operating cost of next generation aircraft.

The Douglas DC8 was a quad-jet, narrow-body, long-range aircraft that began service in 1959. It competed in the market with the Boeing 707, also a quad-jet, narrow-body aircraft with similar range, introduced just the prior year. This was a level playing field between Boeing and Douglas, with each company having a competitive quad-jet engine technology.

Betting on the Wrong Technology : Quad-Jet vs Tri-Jet

Boeing and Douglas (later, McDonnell-Douglas) would eventually diverge in the design choices for their next generation jet aircraft.

The Boeing 747 (1971) was a continuing bet on quad-jet technology. To make up for the added fuel burn and cost of an additional engine, Boeing built a much larger capacity plane – wide-body with double-decker seating at the front of the plane. The 747 had a passenger capacity of 366, extendable up to 467.

The McDonnell-Douglas DC10 (1970) was a bet on tri-jet technology. Douglas assumed that having one less engine would prove to be more efficient over the long run. The DC10 had a much smaller passenger capacity of only 270 but a flying range comparable to the larger 747.

In hindsight, the higher capacity of the 747 more than compensated for the cost of the additional engine. This advantage became more evident as engines became more fuel-efficient over time and the cost of fuel itself dropped after the oil crisis of the 1970s. The DC10 on the other hand was stuck with the smaller passenger carrying capacity. Over time, McDonnell-Douglas created more variants of the DC10 (-10, -30, -40) with different engines and ranges. But it could never win against the runaway success of the Boeing 747.

DC10 : Design Flaws, Poor Timing, and a Price War

Design Flaws: Picking a sub-optimal technology (tri-jet) put McDonnell-Douglas at a structural cost disadvantage. But having flaws in the airplane design itself led to fatal crashes, massive operating losses in the form of lawsuits, legal compensation, repairs, recalls and a huge blow to the company’s safety record and brand. The DC10 was involved in two tragic and fatal accidents (1972 and 1974) that were attributed to a design flaw in its cargo doors. These were followed by the 1979 crash of American Airlines Flight 191 (the deadliest aviation accident in US history), following which the US Federal Aviation Administration (FAA) grounded all US DC10s. While the American Airlines crash was attributed to improper maintenance procedures and not to flaws in the design, the FAA did find that nearly half of the planes failed a test on engine-assembly safety defects. The damage to the reputation of the plane was irreversible and the company announced in 1983 that production of the plane would end due to a lack of orders. Litigation resulting from the DC10 accidents went on for years and cost the company tens of millions of dollars (1970s dollars).

Price Attrition: The DC10 and the competing Lockheed L-1011 were both developed in response to a request from American Airlines and were similar in design, passenger capacity and launch date. As a result, both companies engaged in a war of price attrition, which led to the plane not being profitable for either company! McDonnell Douglas lost money on the plane and Lockheed needed a cash infusion from the US government to save it from bankruptcy.

Poor timing: When McDonnell-Douglas won its huge DC10 sales order from American Airlines in 1968, the market for commercial aviation was heating up and worldwide demand for commercial aircraft was a record high of 724. By the time the DC10 was introduced in 1970, the market was in a slump, and in 1972, demand fell to just 246. Shortly after, the oil crisis of the 1970s put further strain on the commercial airliner business.

Share

The McDonnell-Douglas DC10 tri-jet (left) and the Boeing 767 twin-jet (right). Tri-jets were at a structural cost disadvantage as engines became more efficient and reliable over time.

Staying on the Wrong Technology : Twin-Jet vs Tri-Jet

McDonnell-Douglas realized the limitation of tri-jets in the 1970s itself and started work on a twin-jet version of the DC10. This project, however, was quickly shelved because it was not a simple tweak, and the company did not want to invest in a ground-up new design. Had it done so, it would have set up McDonnell-Douglas to compete well against Boeing and Airbus in later years.

Instead of a course correction, McDonnell-Douglas chose to stay with the tri-jet configuration as it built its next plane, the MD11 (best described as an even larger, stretched version of the DC10). While this saddled the company with the additional cost of a third engine, its competitors innovated and took advantage of improved engine efficiency and reliability by developing long range twin-jets (Boeing 767, 777 and Airbus A330).

The tri-jet was never able to get the same market traction as its twin-jet competitors. The company’s refusal to invest in innovation and failure to adopt superior twin-jet technology put it on a financially unsustainable path. Tragically, the company continued down its path even when it became apparent that the landscape had changed.

The MD11 (below) was, simply put, an even larger, stretched version of the DC10 (top) (source: Link)

MD11 : Failed to Meet Target Specs

Despite the tri-jet disadvantage, the MD11 could have become a successful aircraft, had it delivered on its promised range and fuel efficiency. The competing twin-jet planes (A330 and 767) couldn’t fly long range routes in the early days and so the only option for long haul was the Boeing 747 which had 4 engines and was significantly larger. So, the MD11 could have been a good option for medium capacity, long haul flights – but the airplane simply did not meet its range and fuel burn promises. American Airlines cancelled their order while citing problems with engine performance and airframe, while Singapore Airlines found that the airplane simply could not operate on its desired long haul routes. The airplane was only able to fly 90% of its range target with a full payload; or fly the target range with only 80% of its payload target. Singapore Airlines cancelled their order for 20 MD11s and ordered 20 Airbus planes instead. Even if the MD11 had lived up to its promises, it would have been quickly rendered obsolete with the advent of fuel-efficient twinjets like the 777. MD11 sales never took off and the plane was sold exclusively as a freight plane in later years.

MD12 : An Even Larger Plane!

McDonnell-Douglas made yet another major strategic error. While they were dealing with the fallout of the MD11s failure to meet its range target, the company proposed an even bigger plane. The MD12 was initially proposed as a tri-jet even larger than the MD11 (!) The airline customers rejected that design as it was not fuel efficient over existing aircraft. The company then proposed an even larger stretched version, but with 4 engines and a double-decker configuration – a “super jumbo-jet”. This plane would be much larger and have more capacity than the Boeing 747 but would have a similar range and compete with it. The MD12 proposal was soundly rejected by airlines, received no orders, and was canceled. This rejection left the company without any innovative aircraft design for the 1990s.

It should be noted that the MD12 design was way too ambitious and ahead of its time. Over a decade later, the Airbus A380 realized that vision, but even that airplane proved to be unsustainable and extremely expensive to maintain. Airbus could only sell the aircraft for below production cost and even then, orders declined to the point where the A380 was discontinued and the $25B investment by Airbus was never recouped.

Failed Strategy and Failed Execution

"We got in trouble the old-fashioned way; we earned it. We lost sight of our costs and customers…our financial performance went to hell."

Herbert Lanese, Former CFO of McDonnell-Douglas

The company failed to manage the critical technology inflection from quad-jets to twin-jets and stubbornly placed its bet on an intermediate option (tri-jet) which achieved neither the optimal range nor the optimal fuel burn. They made this blunder not once, but twice and refused to innovate or listen to their customers or to the market. They then failed to execute on their misguided strategy – and produced planes, first with design flaws (DC10) and then with sub-par performance (MD11), which led to a precipitous decline in orders from customers.

To make matters worse, the defense arm of the company made similar blunders when they signed two highly complex contracts (Navy A12 bomber and Air Force C17 transport) at fixed price, implying that they would lose money if the costs exceeded a pre-determined amount. These projects were mismanaged, ran behind schedule and over cost, which further aggravated the company’s finances. 

Rather than course-correct their product definition strategy based on their internal reality (execution record, financial health) and the external landscape (technology, competitors, market trends), they chose to stay on a failed strategy of building an even larger plane, the MD12! Failure to gain traction on the MD12 combined with a decline in orders for existing aircraft pushed the company to the edge of bankruptcy.

In December 1996, Boeing announced that it would acquire the flailing McDonnell-Douglas (Link).

The views expressed herein are my own.

Commercial aviation in the United States began on January 1, 1914, with the first paying customer flying between Tampa and St Petersburg on a two-seater plane – it flew just 23 minutes at an altitude of just 15 feet over the open bay separating the two cities. It was only in the late 1940s, after the second World War when airplane manufacturing reached a scale that made commercial air travel a viable business. Over the next two decades, airplane cruising speeds steadily increased with innovations in engine technology. By 1970, following in the footsteps of military aviation, supersonic transport (SST) was seen as the next technology inflection in commercial aviation.

The modern day semiconductor industry began with the invention of the monolithic integrated circuit in 1960, with the silicon transistor as a foundational building block. In the six decades since then, transistor densities steadily increased, with minimum feature sizes on a chip shrinking from micron-scale to nanometer-scale. Semiconductor chip manufacturing today is at an inflection point, not unlike that experienced by the airplane manufacturing industry in the 1970s.

The evolution of commercial airplane manufacturing shares intriguing parallels to that of semiconductor manufacturing. This three part essay will articulate some of the relevant lessons from the aviation industry.

Share

Subscribe now

Speed and Density

For the first six decades, technological progress in airplane manufacturing ensued along multiple vectors, but the primary drivers of success were around engine efficiency and performance, enabling faster cruising speeds. Airplane cruising speeds doubled three times, reaching over 600mph by 1970. Similarly, semiconductor technology development over the first six decades ensued along multiple vectors, but the guiding success metric was and continues to be transistor density (i.e., packing more transistors within a given footprint).

In addition to being a complex engineering and physics challenge, breaking the sound barrier proved to be an economic limit for commercial aviation. Airplane cruising speeds have largely remained constant for over five decades now. Nonetheless, there have been tremendous advancements in all other aspects of aviation technology and the flying experience over this period.

A guiding principle when shrinking the transistor was Dennard scaling, which enabled transistor density to double while maintaining a fixed power density. Even after the breakdown of Dennard scaling around 2005, transistor densities continued to increase, albeit at increasing power densities. Following the breakdown of Dennard scaling, Moore’s Law, which continues to guide the evolution of cost-per-transistor and the economics of semiconductor manufacturing has also come under increasing strain over the last decade.

Capital Intensity and Consolidation

Airplane manufacturing and semiconductor fabrication are both among the most capital intensive enterprises in the world. The massive upfront capital investment requires enormous manufacturing scale (number of airplanes sold or volume of wafers shipped) to generate acceptable returns on the invested capital. The break-even times for the large investments are also very long (on the order of a decade, Link). As a result, over time, both industries have seen extreme consolidation, to the point where less than a handful of remaining players today command meaningful scale in either business (Link).

Volume and Unit Economics

While there are clear differences in the scale of production of airplanes and wafers, there are parallels in the evolution of the underlying unit economics. This is especially true now more than it was a couple decades ago. The complexity and capital required to build a new airplane model can be significantly larger than its predecessor (e.g., Airbus A380 vs 350) and the time required to assemble it is also longer. As a result, the unit volume required to break-even is larger and the time to break-even can also be longer. However, if the new plane has a significantly larger passenger capacity, then the demand profile (how many airplanes an airline needs) is also lower. For example, Airbus A380 can carry over twice the number of passengers than the A350, but the number of orders of the A380 are 4X lower than that of the A350. Similarly, the complexity and capital required to build the next semiconductor process technology (for example, TSMC N2) can be significantly larger than that for its predecessor (N3) and the volume required to break-even is also  larger, magnifying the need for a larger demand profile (more customers and/or more end-applications consuming more silicon ) with every successive node.

East vs West Divide

Consolidation of critical industries along limited geographies usually leads to geopolitical tension. The western hemisphere is the epicenter of airplane manufacturing and assembly (Boeing in the United States and Airbus in Europe) while contract semiconductor chip manufacturing is now mostly centered in the Eastern hemisphere (Taiwan and Korea). The East-West divide is deeply driven by geopolitics. In fact, Airbus was formed as a consortium of European airplane manufacturers, funded by their governments to jointly compete against the dominance of the American giant Boeing. Geographic consolidation in semiconductor manufacturing has been a topic of intense debate over the last three years and is now viewed as a national security imperative in the United States and in Europe (Link).

Global Supply Chains

Both industries rely on highly complex global supply chains, requiring thousands of components, materials and highly specialized toolsets (Link). These complex supply chains themselves are consolidated along technological domains as well as geographies.

Multiple Inflections

Just as aviation technology progressed through several inflections (piston engines to jet engines, narrow-bodied to wide-bodied, single-aisle to double aisle, wood to metal to composite fuselages), so did transistor technology (bipolar junction transistor to metal-oxide-semiconductor, silicon dioxide gate dielectrics to Hi-K gate dielectrics, planar transistor to three dimensional FinFET to name a few). Incumbents that did not navigate these inflections well went out of business or got acquired by competitors.

The transition from water-cooled piston engines to air-cooled jet engines was one of the first major inflections in aviation technology that saw the demise of several incumbents. Notable among them was the largest incumbent, Douglas Aircraft Company itself.

From Monopoly to Near Bankruptcy

Douglas Aircraft Company was founded by Donald Douglas in 1920 in Long Beach, California. By 1950, it had grown to become the largest commercial aircraft manufacturer and dominated the industry virtually without a challenger. By the start of the second World War in 1939, nearly 90% of the world’s commercial airline traffic was flown on Douglas aircraft.

The company created a storied line of aircraft including the Douglas World Cruiser (first plane to circumnavigate the globe), many fighters and bombers for the US Army Air Corps and the US Navy and the famed Douglas Commercial (DC) line of aircraft including the DC3 (the first modern airliner). Douglas was a prime beneficiary of US military spending during the second World War. Between 1942–45, the company produced a total of 30,000 aircraft and employed over 160,000 people! Following the war, as military contracts were cancelled, the company turned its attention fully to commercial aviation and went on to produce highly successful piston powered planes, the DC6  in 1946 and the DC7 in 1953. Even though the company was late to introduce jet engine technology, it did so with the DC8, launched in 1959 and continued to do well in the market. It also designed the DC9 and had an order backlog of over $3B and growing, well into the 1960s. It had begun designing the DC10, when in 1967 it astonishingly, and quite abruptly, ran out of money. With mounting development costs, and no way to pay for them, the company realized that it was less than a year away from going bankrupt! Douglas was forced to sell itself to McDonnell Aircraft Corporation in 1967.

There were 4 main reasons for this epic fall despite having highly innovative and successful products in a high demand, booming market.

Innovator’s Dilemma : Pistons or Jets

Douglas enjoyed a near monopoly in piston-powered airplanes and faced the classic Innovator’s Dilemma when considering the move to jet engine technology. The company decided to wait for jet engine technology to mature and figured it could continue to reap profits from piston engine airplanes until then. This opened the door for Boeing to take the lead in the development of jet engines with the Boeing 707.

To be fair to Douglas management, they enjoyed a close relationship with their airline customers and the largest of these customers, American Airlines and United Airlines both assured the Douglas CEO that the time was not yet ripe for jets. American Airlines even placed an order for the piston-powered DC7 to get Douglas to delay the launch of the jet-powered DC8. Boeing on the other hand, was the challenger, with no position in the piston airliner market and had little to lose so went all in on jet engines and the development of the 707.

Boeing wasn’t the first mover to jet engines itself – and despite the delayed start, Douglas and Boeing both won a similar number of orders (73 and 70) for their respective jets by the end of 1955. Being late to jet engines was thus one reason, but not the primary reason for the eventual downfall of Douglas.

Platform vs Product : Economies of Scope

The company’s strategy leading up to the second World War was to focus on producing just a few types of unique aircraft, rather than creating a wide range of models. As the market exploded after the war, there was a shift in consumer demand and needs. Boeing responded to this shift by quickly iterating on its designs and producing a range of aircraft to satisfy a variety of market needs (range, capacity, fuel efficiency, etc.). While Douglas focused on building distinct, custom products, Boeing focused on building a platform that catered to a range of requirements. This was a key factor in Boeing’s rapid success and enabled it to execute efficiently and shorten time to market. As the cost of developing large aircraft ballooned from tens of millions of dollars for piston engine aircraft to well over a few billion dollars for jet engine aircraft, Boeing was able to effectively use its resources and shorten the payback periods by leveraging economies of scope. Douglas on the other hand, paid a hefty price for its single product strategy. It should be noted that Douglas’ investment in the DC8 was the largest privately financed project by any single company at the time.

As an example, Boeing built the 707 with options for three different fuselage lengths, two different wings and three different engines and carried forward the same fuselage diameter on the 727, 737 and 757. Douglas built four versions of the DC8, but for the DC9 it chose a completely different fuselage exclusively designed for the short range market, thus giving up its platform leverage.

Supply shortages : Vietnam War

The Vietnam war created a large demand for military aircraft, which in turn created severe shortages in material, parts, and labor for the commercial aircraft industry. Engines and landing gear were in short supply, which led to inventory write-downs and resulting cost increases at Douglas and Boeing. Since airplanes are built on fixed price contracts, this led to increased costs to produce the planes, and reduced profits. Given that the war affected Douglas, Boeing, and indeed all the other manufacturers, it cannot be the root cause of Douglas’ downfall. The war, however, did bring to the fore inefficient execution and planning practices at the company.  

Poor Execution : Overcommit and Underdeliver

The Douglas sales team was highly aggressive in winning deals for the DC9 – but the manufacturing team simply couldn’t keep up – Douglas failed to execute to their committed timeline.

In 1963, at the start of the DC9 program, Douglas expected to sell 400 planes over the next decade. But their sales team was so successful that in 1965 alone, they booked 209 sales. By the end of 1966, they already exceeded their 10 year sales projection with 424 sales. A key reason for these sales was attractive pricing (even in the absence of direct competition from Boeing) and the promise of early deliveries. However, their sales exceeded their resources and their capability to produce. In January 1966, to keep up with growing sales, the company increased planned FY 1966 deliveries to 93. By mid-year, this number dropped to 78. Douglas finally delivered only 64 planes in FY 1966, 30% fewer than the target set just 10 months earlier.  

Assembly hours on the first 150 DC9s produced through August 1967 were 2.2X longer than planned. Most of this increase (69%) was attributed to longer production cycle times. These production delays would increase Douglas’ assembly and inventory costs by $27M in FY 1966 and increased wage costs by $32M, wiping out most of its expected profits for the year. To make matters worse, the company was forced to issue short term debt to meet growing costs.

Astonishingly, a recently reorganized decentralized management structure led to Douglas executives being the last to learn that their teams could not meet promised delivery dates or projected financial returns. In 1965, the company reported a robust pre-tax profit of $25M and well into 1966, forecast a profit of up to $17M. But they ended the year with a stunning loss of $52M, large enough for investors to force them into getting acquired by McDonnell Air Corporation in the following year.

Semiconductor Parallels

Douglas’ failure was attributed to a combination of several factors – delay and hesitancy in committing to the next technology inflection, adopting a product-driven strategy to compete with a platform-driven strategy, misinformed management, poor planning and inventory control, and a lack of coordination systems that allowed it to overcommit and then be forced to underdeliver to customers. The Vietnam war only served as a catalyst to highlight their poor execution.  

In semiconductor manufacturing too, several foundries were forced to give up market share and, in some cases, even give up their business entirely because they were either late in adopting technology inflections, bet on the wrong technology inflection or were unable to afford the cost of the next technology inflection (e.g., ST Microelectronics (Link), NEC, Renesas, (Link) Fujitsu (Link), UMC (Link) to name a few).

More recently, GlobalFoundries struggled with execution on their 7nm technology (Link) and was forced to give up on advanced logic manufacturing entirely, while Intel struggled with the transition to 10nm (Link) and 7nm (Link) nodes. TSMC struggled with the transition to strained silicon and high-k / metal gate stacks at the 40nm and 28nm nodes (Link) but recovered on the transition to FinFETs as it benefited from the mobile wave of computing.

After early stumbles, TSMC learnt to effectively leverage economies of scope by offering a wide platform of technologies (Link) and features on a single base process node to cater to a vast range of customer requests quickly and economically.  As transistor scaling becomes increasingly challenging and expensive in the coming years, this ability to create a wide platform of technologies with a minimal set of changes to efficiently serve a broad range of customers will be critical to the success of semiconductor manufacturers.

The views expressed herein are my own.

References

1. The Remaking of Douglas by McDonnell (Link)

2. What Killed Douglas Aircraft? Jonathan Leonard, UC Berkeley (Link)

3. Remembering The Douglas Aircraft Company: The Company That (Nearly) Beat Boeing (Link)

4. The Rise and Fall of Donald Douglas (Link)

Leave a comment

Share Bits and Bytes

“We took a step back and we said, “What is the implication of this?” Not just for computer vision, but ultimately for how software is done altogether. Recognizing that for the very first time, software is not going to be written – features weren't going to be engineered or created by humans, but somehow automatically extracted out of data, refined out of data to recognize patterns, relationships, and somehow learn the representation of some predictive model — that observation early on caused us to ask the question, "How does this affect the future of software?" How does this affect the future of computer science? How does this affect the future of computing? If the way that you write software is different, then how does it change the way you would design computers? And if the software that's written is written by a computer versus a human, how does that affect the type of computers you would design? We had the good sense of thinking about it — from first principles — the implications for the entire field of computer science and the entire field of industry. Which ultimately led to asking the question, “What about the implications to all the different industries?”

Jensen Huang, describing why NVIDIA moved into the deep learning space, circa 2012 (Link)

Advances in computing have always been enabled by an interplay of hardware and software. Innovations in software drove several generations of progress in hardware and vice-versa. As transistor density doubled every couple of years, computer architecture evolved to take advantage of the additional transistors; and in turn, enabled software to leverage the increased functionality and performance made available by Moore’s Law. Similarly, as computer applications and workloads evolved, software in turn shaped the evolution of computer architecture, making it more efficient for the most predominant workloads. There have been numerous evolutionary advances in computer architecture, chip design methodology and software frameworks that contributed to the computing trajectory over the last five decades.

By contrast, paradigm shifts have been far more infrequent and occurred once in a decade or even longer but drove revolutionary advances across the computing stack and the industry at large. Such paradigm shifts are necessary to consistently raise the design abstraction and enable engineers and architects to manage the growing complexity at every layer of the computing stack. We may be witnessing the early days of the next paradigm shift, referred to as Software 2.0.

This paper articulates how Software 2.0 is likely to drive a new paradigm not only in how we design chips, but in the chip architecture itself. These changes in turn have the potential to alter the established order within the entire semiconductor industry.

Subscribe now

Software 2.0 : Humans Train Models

“Neural networks are not just another classifier; they represent the beginning of a fundamental shift in how we develop software. They are Software 2.0. The “classical stack” of Software 1.0 is what we’re all familiar with — it is written in languages such as Python, C++, etc. It consists of explicit instructions to the computer written by a programmer. By writing each line of code, the programmer identifies a specific point in program space with some desirable behavior. In contrast, Software 2.0 is written in much more abstract, human unfriendly language, such as the weights of a neural network. No human is involved in writing this code...”

Andrej Karpathy, Director AI, Tesla (Link)

Starting 2012, deep learning based models began to show dramatic reduction in error rates for image recognition (Link) and natural language processing tasks (Link). By 2014, all the entries in the ImageNet contest used deep learning models and the winning entry had an error rate < 5%, making it better than the average human at image classification from a dataset of thousands of images (Link) ! Since then, there has been a quiet revolution that has been sweeping almost every aspect of our lives. Based on these trends, Andrej Karpathy made an astute observation in 2017 to describe the evolution of software in the age of deep learning and artificial intelligence. He predicted that over time, a large fraction of the codebase we have used for decades will transition from Software 1.0 (code written by humans) to Software 2.0 (code written by a trained neural network model). In this new paradigm, the focus of a human software developer shifts from writing code to creating and curating (labeling) large datasets, which in turn form the input to a model that generates code. Judging by the progress made in the last 5 years, it is fair to say that Andrej has been right, even though we are still a long way from a widescale transition to Software 2.0.

Models Write Code

A remarkable visualization of the power of Software 2.0 is this recent public demonstration by OpenAI in which researchers used a model called Codex to create a computer game from scratch using commands in plain English. As seen in the screenshots below, complex instructions written in conversational English were automatically converted to written code by the model in real-time. Codex is a general purpose programming model based on Generative Pre-trained Transformer (GPT-3), a highly sophisticated and vast neural network training model with up to 175B parameters.

Instructions in plain English are instantaneously converted to written code using a deep learning based model (GPT-3) (Link)

One can imagine a data-rich model, trained with chip design input parameters and aware of the relevant reward functions that could one day take simple instructions and translate them to code which could then create simple circuits or IP blocks or even layout entire chips or carry out other complex tasks in the chip design process. As an example, one might be able to abstract away highly complex instructions to plain English as follows:

“Use the 5nm PDK to create a foundational standard cell library, optimized for gate density”

“Create a floorplan using a combination of the following macros and cells”

“Optimize this floorplan to minimize area, while also optimizing for wire length.”

It is important to note that unlike games of Go or Chess, chip design workloads like chip floor planning or chip verification have several orders of magnitude more variables, and hence will require far more compute capability and bigger datasets. In addition, while Go or Chess are games with clear win/lose outcomes, chip design workloads have a gradient of outcomes that can be optimized based on the application of the chip. Nonetheless, results from a variety of companies so far indicate a significant improvement in turnaround time and chip parameters (frequency, power and/or area) when compared to the corresponding traditional human-engineered tasks (Link, Link, Link).

While such a high level of abstraction doesn’t exist yet, at least not in public domain, there is no fundamental reason why it cannot eventually exist. In fact, it represents the ultimate abstraction to hide and manage the exponentially increasing complexity of circuit design with every successive generation of transistor scaling.

Starting the mid-1980s, Intel partnered with researchers at UC Berkeley and led the way in the development and adoption of new Electronic Design Automation (EDA) tools that helped raise the design abstraction and manage the complexity of microprocessor design. This work spawned the entire field of electronic design automation and companies like Synopsys and Cadence, which today are industry leaders were born out of this partnership. Nearly three decades since, it is time for a new paradigm in design automation, this time enabled by deep learning.

What if a “technology X” could take a language and convert it to circuits? An early sketch illustrating the concept that led to the formation of the company Synopsys and the field of Electronic Design Automation in 1986. (Source : Aart de Geus, Synopsys, HotChips 2021)

Code Designs Chips

The Google DeepMind Challenge Match (Link) in 2016 when the AlphaGo AI beat the world champion Lee Sedol was an inflection point for many other machine learnings (ML) applications – including EDA. ML for EDA is a very active area of research and development today as indicated by a growing number of plenary talks at recent flagship conferences like HotChips, ISSCC and Design Automation Conference (DAC). Every layer of the chip design stack is undergoing a transformation with the infusion of ML. This includes chip architecture, synthesis, place and route, timing, power, test, and verification.

“This incredible growth rate could not be achieved by hiring an exponentially growing number of design engineers. It was fulfilled by adopting new design methodologies and by introducing innovative design automation software at every processor generation. These methodologies and tools always applied principles of raising design abstraction, becoming increasingly precise in terms of circuit and parasitic modeling while simultaneously using ever-increasing levels of hierarchy, regularity, and automatic synthesis. As a rule, whenever a task became too painful to perform using the old methods, a new method and associated tool were conceived for solving the problem. This way, tools and design practices were evolving, always addressing the most labor-intensive task at hand. Naturally, the evolution of tools occurred bottom-up, from layout tools to circuit, logic, and architecture.”

Pat Gelsinger et al, “Coping with the Complexity of Microprocessor Design at Intel”, 2012 (Link)

Synopsys, Cadence, NVIDIA and Google have publicly talked about their efforts in this area (Link), but nearly all major companies are actively working to adopt aspects of this technology. As in all other applications, the companies with the richest datasets will take the lead in this area – and the more data they collect, the wider their moat will become.

Companies like NVIDIA (left) and Google (right) are at the forefront of applying machine learning to chip design (Source : Design Automation Conference 2021)

Share Bits and Bytes

Chips Designed for Software 2.0

“We are not going to be able to build next-generation chips without AI. And that's kind of a remarkable statement. …Today, we are producing software that gets shipped with all of our AI chips. Without AI, we can't produce software that runs the AI. And in the future, without AI, we wouldn't be able to design the chips that we use to run AI. So, the circular, positive feedback system is about to go into turbocharge. I have every confidence that in the next 10 years, we're going to see even greater advances. Not necessarily at the transistor level, but absolutely at the computation level.”

Jensen Huang, CEO NVIDIA (Link)

Once Software 2.0 becomes the dominant paradigm, not only will it change how we design chips, but it will also affect the architecture of the chips we build. There are several reasons for this as articulated by Andrej Karpathy (Link). Neural networks are computationally homogeneous compared to traditional software and because their instruction sets are relatively small with constant runtimes, it is easier to program them directly on to silicon. Moving software closer to hardware has inherent advantages in performance efficiency. With dramatic improvements in chip design costs and turnaround times enabled by ML, it will become economically viable for more companies to design custom chips (domain specific accelerators) to accelerate specific workloads. Over time, it will also become economical to build a large chip or a collection of chips that are capable of simultaneously supporting multiple classes of trained models, as envisioned by Google Pathways (Link). For these reasons, it is likely that spatial computing (sometimes referred to as graph-based computing) will become the predominant computer architecture in the AI era. Computer architectures that natively process large data sets as a spatially distributed interconnected graph mimic how neural network models are organized and are thus better suited for deep learning workloads. Interestingly, spatially distributed computing also is a good way to mimic how a human brain processes data.

A Paradigm Shift

“Why should a change of paradigm be called a revolution? Political revolutions are inaugurated by a growing sense, often restricted to a segment of the political community, that existing institutions have ceased adequately to meet the problems posed by an environment that they have in part created. In much the same way, scientific revolutions are inaugurated by a growing sense, again often restricted to a narrow subdivision of the scientific community, that an existing paradigm has ceased to function adequately in the exploration of an aspect of nature to which that paradigm itself had previously led the way. In both political and scientific development, the sense of malfunction that can lead to crisis is prerequisite to revolution.”

Thomas S. Kuhn, The Structure of Scientific Revolutions (1961) (Link)

How scientific revolutions happen, adapted from Thomas Kuhn. Image credit : Jim Keller

In the early days of the integrated circuits era, the established paradigm was vertical integration – to manufacture chips, a company had to design them too. And there was no point in designing chips if you didn’t also own the factories to manufacture them – this was viewed as the only way to build semiconductor chips in the early days. Anomalies in this established paradigm showed up when smaller “design houses” began to outsource their manufacturing needs from larger players, primarily because they did not have the means to build and maintain factories to manufacture chips. This culminated in the formation of pure-play foundries like UMC and TSMC – a revolution in hindsight. It took well over two decades for this “new science” of building chips to establish itself as the new, reigning paradigm – what we call the foundry-fabless ecosystem today.  

Over time, anomalies are bound to emerge in the reigning foundry-fabless paradigm too – likely leading to another revolution, followed by a “new science” phase when the seeds of the next paradigm will take root.

The established software paradigm since the dawn of computing has been that humans write code and machines execute it to accomplish tasks. We are now beginning to see anomalies in this paradigm where machines can do certain tasks just as well if not better than humans (image/speech recognition and natural language translation are everyday examples). We are also seeing early signs where machines can write code that covers more ground and edge cases than humans are even capable of. As deep learning datasets become richer over time and trained models become more accurate, it will eventually become more common for most code to be written by models, rather than by humans.

There is no reason to believe this will not extend to building silicon chips (across every layer of the stack from initial design-technology co-optimization to architecture, RTL, synthesis, timing, layout, and extending to test and verification). Indeed, we are already seeing numerous anomalies where machine learning can design chips faster than humans and even uncovers design points that lead to better performance, power, area (and hence cost). As transistor density (and resulting chip design complexity) grows exponentially, it is foreseeable that all chip design will eventually be achieved via machine learning. We will initially see this transformation at individual layers (e.g., verification, floor planning, etc.), but over time, this will extend holistically across the EDA stack, when simple commands in English may be able to generate a complete chip layout.

Cloud service providers have the most visibility to a variety of workloads and are best positioned to develop the most comprehensive and efficient training models. We are seeing cloud service providers like Google begin to leverage their scale and insight to build specialized chips for their own workloads. All the EDA tools needed to design chips already exist in the public clouds today (Link). Over time, cloud service providers could just as easily offer machine learning enabled “chip design as a service” on the cloud. This may spur the next chip design revolution; indeed, a paradigm change in how chips will be designed. It is foreseeable that such a revolution will disrupt the established semiconductor world order. Companies that control or own the entire computing stack from chips to the application frameworks and the deep learning models are likely to have an upper hand in such a landscape. Foundries, EDA and IP vendors will shape the new landscape and adapt to it just as much as the cloud service providers and fabless design houses.

These changes will play out over the next decade and perhaps longer as new players and new capabilities emerge, new business and engagement models are defined, and Software 2.0 helps raise the abstraction for chip design.

The views expressed herein are my own.

Many thanks to Jim Keller for enriching discussions that formed the inspiration for this work.

Leave a comment

This post takes a closer look at how Apple helped TSMC become the leader in advanced node semiconductor manufacturing.

How TSMC Helps Apple

The first Apple designed mobile application processor chip for the iPhone to be manufactured at TSMC was the A8 which began shipping in the Fall of 2014.

Three years later, in October 2017, TSMC hosted senior executives from its most important customers and partners to mark the occasion of its 30th anniversary. One of the guests at this event was Jeff Williams, the Chief Operating Officer of Apple.

Senior executives from Apple, ASML, Broadcom, Arm, Analog Devices, Qualcomm, and NVIDIA at TSMCs 30th anniversary celebrations with founder Morris Chang, October 23, 2017

Speaking at this event, Williams highlighted the importance of the Apple-TSMC partnership and how it came to be (Link):

“It was actually 2010 that the first seeds of partnership between Apple and TSMC were planted. I had flown to Taiwan and had dinner with Morris Chang and (his wife) Sophie at their house…We were not doing business with TSMC at that time, but we had a great conversation. We talked about the possibility of doing stuff together, and we knew the possibilities were great if we could take leading-edge technology and marry it with our ambitions. It seems obvious right now, but it wasn’t then, because the risk was very substantial. The way Apple does business is we put all our energy into products, and then we launch them, and if we were to bet heavily on TSMC, there would be no backup plan. You cannot double plan the kind of volumes that we do. We want leading-edge technology, but we want it at established technology volumes. On the TSMC side, that means a huge capital investment. It means ramping faster than the more careful yield plan that the industry is used to…Together, we decided to take the bet, to take the lead and Apple decided to have 100 percent of our new iPhone and new iPad chips — application processors — sourced at TSMC. TSMC invested $9 billion and had 6,000 people working round the clock to bring up the Tainan fab in a record 11 months, and in the end, the execution was flawless. We’ve gone on to ship over half a billion chips together in that short window, and I think TSMC has invested $25 billion — $9 billion on that first venture — there are very few companies in the world that would spend $9 billion in capital across everything, let alone a single bet. So for that, we thank you, Morris Chang, and everybody at TSMC. It’s been a wonderful partnership. The big challenge we had as we looked at the mobile revolution was this tradeoff between performance and power. The view at the time was you had to choose. You had one or the other. Largely because of what the fabless model has done, what TSMC has done, what many people in this room have done, including ARM CEO Simon Segars and his organization, we came to the point where those tradeoffs are not necessary. We have performance in thoroughly constrained environments. This opens for the next decade a whole new world. In the next decade, the question is not so much ‘do we have enough processing power to meet our ambitions’. The question for us is ‘do we have the right ambitions to utilize this technology. We at Apple are not concerned about the talk of slowing in the semiconductor industry. That’s not the case at all. We think the potential is huge.”

Winning the Mobile Wave of Computing

To understand how TSMC was able to win Apple as a customer and dominate the mobile wave of computing, it is illustrative to look at how Intel came to dominate the PC wave of computing a few decades earlier.

The CPU was the primary enabling silicon platform of the PC era; and Intel, in collaboration with Microsoft established itself early on as the preeminent CPU provider for the PC industry. This lock on the global CPU supply assured Intel of massive wafer volumes; which in-turn enabled quick yield learning and defect improvement; which in turn enabled volume ramps at high margins and massive revenues. These revenues in turn funded the research and development needed for the next technology node. This virtuous cycle enabled Intel to gain, and maintain technology leadership during the PC era, and in fact was a primary driver of Intel’s success as a semiconductor manufacturer over 4 decades.

The Intel and Microsoft partnership (“Wintel”) defined the PC wave of computing. The TSMC and Apple partnership defined the mobile wave of computing.

Frequency (clock-speed) was the primary metric in the PC era and the standalone CPU was the primary chip that drove advancements in semiconductor technology for decades. Form-factor was hardly an influencer and there wasn’t as much of a drive to integrate system-level functionality either on-chip (SoC) or in-package (SiP). Form-factor, cost and power-per-function became critical drivers in the mobile market and that in turn increased the importance of on-chip integration of functional hardware (e.g., power management, computing, audio/video, graphics, GPS, and radio among others).

This shift from mostly performance-centric chips to mostly power-constrained chips and the focus on lowering cost and increasing system-level integration disrupted the traditional semiconductor landscape within just a decade and continues to reshape the industry today (I wrote about this here)

Intel’s process technology was optimized over decades to enable the highest frequency CPUs and enabled Intel to dominate the PC wave of computing. On the other hand, foundry process technology was optimized to enable a highly power efficient and cost-effective SoC technology suitable for a wide range of customers over a large dynamic operating range, long before the advent of mobile computing. The foundries, led by TSMC were thus well prepared to cater to the needs of mobile, post-PC products. Apple’s decision to use Samsung, and later TSMC as a foundry provided a significant growth catalyst for both foundries. TSMC being the largest incumbent foundry at the time was the largest beneficiary of the mobile wave of computing.

How Apple Helps TSMC

Apple was not a TSMC customer for the first several years of the iPhone and initially used Samsung to manufacture its application processors. Since 2014, TSMC has been the primary beneficiary, both as a direct supplier to Apple (including the A-, S-, W- and M- series chips) but also as an indirect supplier for other leading node and legacy node chips that go into Apple products (e.g., modems, sensors and connectivity chips from Qualcomm, Broadcom, and others). When a semiconductor foundry manufactures chips for a large, high volume, demanding customer like Apple, it wins on multiple fronts : quicker volume ramps, quicker revenue ramps, a focus on disciplined risk taking.

Quicker Volume Ramps

Source: AMD EPYC™ Horizon Summit – “Rome in Rome”, Sep. 18, 2019. TSMC defect density improvement rates accelerated since the 28nm node, enabling faster volume ramp rates with every successive generation.

Apple has been the lead customer on TSMCs advanced technology nodes since 2014. Massive iPhone wafer volumes and a rigid launch schedule requires TSMC to quickly debug and stabilize yields on the most complex, advanced geometry process nodes. This in turn, enables TSMC to rapidly improve manufacturability and pave the way for the second wave of advanced node customers (e.g., NVIDIA and AMD). TSMC defect density improvement rates have become much steeper over the last 4 technology nodes (Link), enabling quicker volume ramps.

Quicker Revenue Ramps

The rapid and highly predictable ramp of the iPhone every Fall dramatically accelerated TSMCs revenue run rate – TSMC now reaches 50% of peak node revenue run rate in just 2-3Q – compared to 8-9Q prior to winning the Apple iPhone design. This rapid revenue run rate shortens the node payback period and increases the return on the massive capital investment needed for the node.

TSMC earnings reports show accelerating revenue run rates for new nodes starting 28nm

Disciplined Risk Taking

Apple requires leading-edge processes for its mobile products and drives huge volumes on a predictable annual cadence. The annual iPhone cadence instilled a manufacturing discipline and served as a self-regulating mechanism for TSMC to take measured risks in its process technology choices. Prior to winning the Apple business, TSMC process nodes were introduced on an irregular, unpredictable cadence. TSMC struggled to get their 40nm (strained Si) and 28nm (HKMG) nodes to high volume and introduced multiple versions of these nodes without a strong, driving customer. Since winning the Apple iPhone SoC design in 2014, TSMC has introduced a new process node every year. This annual cadence requires TSMC to be risk-averse and make conservative process choices. Every other year, TSMC introduces a major process change (e.g., N7, N5) while in the intervening year, it refines the technology from the prior year (e.g., N7P). This alternating cadence of half-nodes and full nodes every year enables TSMC to make smaller process changes in every node, but still reap the benefits of compounded gains in the long run.

Growth Catalyst

TSMC has been a significant beneficiary of the mobile wave of computing and has seen its revenues rise steadily since the launch of the iPhone in 2007 and markedly since winning the Apple SoC design in 2014. Since 2015, Apple's revenue contribution to TSMC has been growing in both dollar and percentage terms. In 2020, Apple contributed ~$11B (25%) to TSMC total revenue. Since winning the Apple business, the increased cash flow has allowed TSMC to keep investing in its advanced node capacity. Outside of Apple, the only company that has both the capability and volume demand to be the lead adopter of a new technology node is Intel.

TSMC annual revenues increased markedly with the introduction of the iPhone (2007) and the Apple design win (2014-15) (Source : TSMC earnings reports)

Key Takeaways

Led by Apple and the iPhone, mobile computing provided TSMC with opportunities to increase wafer volumes at all nodes including legacy nodes (e.g., 40nm and older) and advanced nodes (16nm and newer). The legacy node wafer business allowed TSMC to run large volumes at full capacity and high margins on fully depreciated toolsets. The advanced node, high volume, and quick ramp products like Apple iPhone SoCs were critical in helping TSMC debug and stabilize yields on the most complex, advanced geometry process nodes, thereby quickly improving manufacturability and paving the way for the second wave of advanced node customers (e.g., NVIDIA and AMD). Without the massive volumes from Apple, it is unlikely that TSMC would have been able to catch up to Intel on advanced node semiconductor manufacturing.

The views expressed herein are my own

This post examines how market valuations and revenues of the major stakeholders in the semiconductor ecosystem have evolved over the last decade and illustrates the new contours taking shape in the semiconductor industry.

Background

Through the 1990s, Integrated Device Manufacturers (IDMs) were ideally positioned at the top of the semiconductor value chain and were able to extract the most revenues and the highest profit margins, by sequentially integrating chip design and silicon manufacturing. Over time, as the cost of advanced semiconductor manufacturing escalated, only the largest semiconductor companies commanded the scale to afford their own manufacturing. This paved the way for pure-play foundries like TSMC to carve a niche for themselves while also enabling pure-play design houses like Qualcomm and NVIDIA to establish themselves as domain-specific market leaders. As this ecosystem matured, consolidation led to a dramatic reduction, not only in the number of leading edge semiconductor chip manufacturers – from over 25 in the 1990s to just 3 today, but also in the number of semiconductor equipment manufacturers and Electronic Design Automation (EDA) vendors.

Among semiconductor companies, Intel was the clear winner in the PC era and remains standing as the only Integrated Device Manufacturer (IDM) of personal computer Central Processing Unit (CPU) chips today. Given its dominant role as the CPU vendor of choice, Intel grew to become much larger than other stakeholders in the ecosystem and for well over a decade was able to nearly unilaterally set the direction and cadence of Moore’s Law for everyone else to follow. But this outsized advantage diminished over time.

Today, even though TSMC commands an enormous manufacturing scale and market valuation, it is not entirely at liberty to set the direction and cadence of Moore’s Law on its own. TSMC was founded as a service business and its technology roadmap is heavily influenced by the product roadmaps and requirements of its largest customers and the roadmaps and capabilities of its suppliers like ASML and Applied Materials.

The mobile wave of computing flattened the semiconductor world order and made it highly interdependent, benefiting every major stakeholder in the ecosystem, including pure-play foundries (TSMC), pure-play design houses (e.g., Qualcomm, NVIDIA), equipment companies (e.g., ASML, Applied Materials) and EDA companies (e.g., Synopsys, Cadence).

Equipment Manufacturers

To get a sense for the enormity of the transformation that has occurred in the semiconductor manufacturing landscape it is worthwhile to compare the market valuations and annual revenues of the top two chipmakers (Intel and TSMC) with the top five equipment manufacturers (Applied Materials, ASML, LAM Research, KLA and TEL). 

Just a decade ago, Intel alone was valued nearly 2X that of TSMC and 4X larger than the top five equipment makers combined. Today, the market capitalization of the top five toolmakers combined is comparable to TSMC, and over 3X larger than Intel!

A decade ago, Intel’s revenue was 4X larger than that of TSMC and more than 2X that of the top five toolmakers combined. Today, the top five toolmakers combined make more revenue annually than Intel alone. Intel still leads TSMC in annual revenue, but the gap is smaller than it used to be. In effect, while the total market has grown larger, it is no longer dominated by a single entity alone. Samsung Foundry is the second largest semiconductor foundry, however being part of a larger conglomerate, its financials are not publicly available.

Significance: Up until just a few years ago, Intel commanded ecosystem and platform leadership with more resources to spend on research and development than the leading toolmakers combined. This gave Intel the upper hand in setting the direction and cadence of technology development, which in turn set the cadence for equipment makers and other players in the ecosystem. Today, the equipment makers collectively are just as, if not more influential than the chip manufacturers in driving fundamental research and development of new tools, new materials, and new processing techniques. This gives them a bigger say in setting the Moore’s Law roadmap.

It is also worth noting that there has been significant consolidation among the equipment providers. This consolidation is based on specific manufacturing domains – e.g., ASML is the only company that manufactures EUV scanners, arguably the most critical enabler of advanced semiconductor technologies. Applied Materials provides a disproportionately large share of deposition and ion implantation equipment, while KLA is the only provider of some of the most advanced metrology equipment. This suggests that the next major advancement will require active collaboration with the equipment ecosystem and will not be driven by the chip makers alone.

Fabless Chip Designers

An equally dramatic transformation has occurred in chip design. Intel competes not only with the foundries (e.g., TSMC, Samsung), but also competes with design houses like AMD and NVIDIA.

A decade ago, Intel made 1.7X more in annual revenue than the top five pure-play chip designers combined. Today, the top five chip designers combined make 1.5X more revenue than Intel, a massive swing within just one decade! While Intel revenues grew 1.4X over the last decade, the chip designers nearly quadrupled their revenue in the same period.

Significance: The leading chip designers are important customers of TSMC, Samsung and GlobalFoundries. Collectively, they consume nearly a substantial portion of the total foundry wafer capacity and thus are important stakeholders in setting the foundry technology roadmap and cadence. A decade earlier, Intel by itself was far more influential in setting the cadence and direction of Moore’s Law because the CPU was the most dominant and fastest growing silicon platform. Today, the mobile SoC is just as influential in setting the cadence and direction of Moore’s Law. Non-CPU architectures like the GPU and a variety of custom ASIC architectures are also becoming increasingly important and influential in defining process technology roadmaps.

System Integrators and Cloud Service Providers

During the PC era, Intel derived most of its revenue from Original Equipment Manufacturers (OEMs) like IBM (later Lenovo), HP, Dell, ASUS, and Acer. Even at the height of the PC era, Intel commanded much higher market valuations than most of its OEM customers. Today, Intel derives a large and growing portion of its revenues from Cloud Service Providers (CSPs) like Amazon, Microsoft and Google who don’t directly sell equipment incorporating Intel chips, but rather sell computing as a service using computing, networking and storage chips made by a variety of providers, and increasingly, their own internally designed chips.

Apple is not a pure-play design house, but rather a major system integrator of flagship consumer products like the iPhone, iPad, and the Mac. With Apple designing its own mobile SoCs for its consumer products, it is now a significant influencer of semiconductor process technology roadmaps and cadence. Even though Apple does not offer cloud computing as a service, the scale of its iCloud and ML/AI infrastructure is large enough to make it a dominant customer of cloud computing silicon.

Fifteen years ago, Intel annual revenues were 3X larger than those of either Google or Amazon, while Intel made nearly 2X the revenue of Apple! Microsoft was the only one of these 4 companies with a revenue higher than Intel. Today, these four companies earn 10X more in combined revenue than the three largest chipmakers!

Significance: These software, systems and service businesses which started and remain predominantly modular and horizontally oriented, are evolving into more vertically oriented ones when it comes to silicon compute platforms. Instead of being solely dependent on generic, merchant silicon from companies like Intel, AMD, or NVIDIA, these hyperscalers and system integrators are adding custom silicon as a differentiated layer to improve compute efficiency for specific workloads. Given their outsized scale, these companies are also beginning to influence foundry technology roadmaps. The major hyperscalers (Google, AWS, Microsoft) now comprise a fast growing portion of TSMC and Samsung annual revenues.

Electronic Design Automation (EDA) Vendors

A small, yet critical piece of the semiconductor ecosystem is the group of EDA companies, sometimes also referred to as Computer Aided Design (CAD) companies. The open foundry-fabless ecosystem enabled EDA companies to provide off-the-shelf, ready to use tools and infrastructure needed to bring a complex chip design to life. Over time, these companies also began providing foundational IP and other generic IP based on mainstream foundry process technologies. Cadence and Synopsys are the two primary EDA vendors with ANSYS being the third largest independent EDA provider. Until 2017, Mentor Graphics was the third largest independent EDA provider when it was acquired by Siemens for $4.5B. Since its acquisition, Mentor continues to play an important role, however, its financials are no longer public. EDA revenues grew a remarkable 3X over the last decade while market capitalization grew 12X over the same period. Assuming Mentor Graphics would have grown at a similar rate, the combined market capitalization of the top 4 EDA vendors would today be approaching $200B, comparable to that of Intel, AMD or Qualcomm!  

Significance: Traditional enablers of Moore’s Law scaling (e.g., geometrical shrink, new materials) now provide only part of the improvement from one process node to the next. Design-Technology Co-optimization (DTCO) is now an increasingly important enabler of process scaling. As design and process complexity has grown over time, the role of EDA companies has become increasingly important. In the next design paradigm, when chip design will increasingly leverage machine learning (ML) and artificial intelligence (AI) for tasks such as Place and Route (PnR) and verification, these CAD companies are likely to play an even larger role.

Democratizing Silicon Innovation

Easy access to silicon design automation (EDA) tools and to foundries offering manufacturing-as-a-service spurred massive growth in the number of silicon startups over the last decade. The growing importance of domain-specific accelerator chips to improve the performance and efficiency of AI and ML workloads was a major catalyst for this trend.

A vibrant foundry-fabless ecosystem made it relatively easy for even small startups to conceive and design new chip architectures and rapidly validate these designs on silicon with modest funding. Even small start-up companies can now easily access fully qualified, off-the-shelf, intellectual property (IP) blocks for a wide range of generic silicon circuits. This greatly diminishes the overhead to design complex chips and allows start-up companies to focus primarily on developing differentiated IP that would set their chip designs apart in the market. It is now possible for start-up companies to design on the most advanced silicon process technology simply by signing up to a foundry virtual design environment hosted in public clouds like AWS or Azure and supported by EDA companies like Cadence and Synopsys. In 2020 alone, $5.1B of venture capital poured into semiconductor startups worldwide – 5X more than the investment a decade earlier ! That number is estimated to be $7.8B for the full year 2021 and is forecast to be >$6B for 2022 (Link) with over 50 firms developing chips for AI applications alone.  

Significance: Until a decade ago, silicon innovation was considered the exclusive domain of incumbents like Intel, NVIDIA, or Qualcomm and venture funding for semiconductor startups was hard to come by. Entrepreneurship around chip design is now experiencing a renaissance in what many are calling the new golden age of computer architecture. It is still too early to know with certainty if a new challenger(s) will emerge to displace the incumbent giants – perhaps what is more likely is that the most promising startups will get acquired by the incumbents as they seek to differentiate and secure their leadership in the next wave of computing.

Takeaways

The semiconductor landscape has witnessed a dramatic transformation over the last two decades. What used to be a largely unipolar industry is now a complex, multi-polar ecosystem with several large and highly influential stakeholders. Domain specific consolidation has resulted in fewer but much larger players with “single-points of failure” within nearly every technical domain. Geographic consolidation has resulted in the semiconductor industry becoming the source of major geo-political tensions, trade, and tariff wars. Finally, the COVID-19 pandemic proved to be a tipping point for a global chip supply shortage that catapulted companies and governments to shore up and secure their supply chains.

A consistent cadence of technological progress has set apart the semiconductor industry from any other in human history and has underpinned nearly all aspects of technological evolution; indeed, of all human progress over the last 50 years. The mobile wave of computing flattened the semiconductor world order and created a multi-polar industry wherein the predictable pace of progress is no longer driven by a single entity alone; rather this progress will now require close collaboration between multiple large and disparate corporations and governments alike. It will also require massive capital outlays, beyond the reach of any single entity alone. And finally, it will require collaboration across international borders while maneuvering around geopolitical hotspots. The semiconductor landscape continues to evolve in the post-PC and post-mobile computing era – by the end of this decade, we are likely to witness a new semiconductor world order, shaped by the needs of the next wave of computing.

The views expressed herein are my own.

The success of the iPhone has diminished American hi- technology manufacturing competitiveness. Nowhere is this more evident than in advanced semiconductor manufacturing. By virtue of its ubiquitous scope and scale and a highly distributed global supply-chain, the iPhone has had a profound impact on the American semiconductor manufacturing landscape and may have given Asia an insurmountable lead and momentum to win in the next wave of computing.

By not engaging with US semiconductor manufacturers, Apple enabled critical semiconductor technology innovation and manufacturing infrastructure to take root outside America, specifically in south-east Asia. The sheer volume of silicon driven by Apple’s products has given an enormous, compounded advantage to Asian semiconductor foundries. Armed with this critical know-how and a well established infrastructure, it is the Asian semiconductor manufacturers who are now poised to win the next wave of technological innovation.

Technology competitiveness aside, the economic and geopolitical implications of this shift are not only staggering, but potentially irreversible.

“Designed in Cupertino, Manufactured in Asia”

The most significant changes in the silicon foundry, fab-less and IDM (Integrated Device Manufacturer) ecosystem over the last decade can be traced back to 2007, when Steve Jobs asked then Intel CEO, Paul Otellini if Intel would be interested in becoming the primary system-on-chip (SoC) supplier for the yet to be announced iPhone. After much internal analysis, Intel declined (see quote from Paul below).

Nearly 7 years later in 2013, Paul himself acknowledged the enormous fallout of this decision when he said in an interview to The Atlantic, “the world would have been a lot different if we’d done it”.

“We ended up not winning it or passing on it, depending on how you want to view it. And the world would have been a lot different if we’d done it. The thing you have to remember is that this was before the iPhone was introduced and no one knew what the iPhone would do. At the end of the day, there was a chip that they (Apple) were interested in that they wanted to pay a certain price for and not a nickel more and that price was below our forecasted cost. I couldn’t see it. It wasn’t one of these things you can make up on volume. And in hindsight, the forecasted cost was wrong and the volume was 100X what anyone thought. The lesson I took away from that was, while we like to speak with data around here, so many times in my career I’ve ended up making decisions with my gut, and I should have followed my gut. My gut told me to say yes.”

— Former Intel CEO Paul Otellini in an interview to The Atlantic, 2013 (Link)

In 2007, Apple did not have an experienced in-house silicon team that could have pulled off what was needed to design the primary computing chip for the iPhone. In just the prior year, Apple had made a landmark shift away from PowerPC (IBM/Motorola) chips to x86 (Intel) chips for its Mac line of computers. The flagship Apple product at the time was the iPod which wasn’t an advanced computing device and did not necessarily require a competitive silicon processor. The upcoming iPhone on the other hand, required not only a complex, ultra-low power, high performance computing SoC, but one which also cost a fraction of the expensive Intel CPU chip that went into the Macbook.

In 2007, Intel was the undisputed leader in semiconductor manufacturing technology, at least two generations ahead of Taiwan Semiconductor Manufacturing Corporation (TSMC). Intel also was the undisputed leader in high-performance x86 CPU chip design with a near monopoly on the PC market and was beginning to make in-roads into the low-power computing segment with its Atom line of processors. While ARM did make better low-power chips compared to Intel, the ARM ecosystem was far less influential in 2007 than it is today. In 2007, Intel was arguably the best candidate to be the supplier of iPhone computing chips to Apple.

Apple was adamant on the pricing for the chip, while Intel forecast a higher cost and lower volumes. In hindsight, as Otellini noted, Intel’s cost forecast was wrong and iPhone volumes were 100X higher than anyone projected. Intel lost out on being the compute engine provider for the most successful consumer electronics product ever launched.

But more importantly, the company lost out on being the silicon platform provider for the entire post-PC wave of computing.

Apple proceeded to get the very first iPhone system-on-chip (SoC) (APL0098) manufactured by Samsung Semiconductor Ltd. in 2007. Then, in 2008, Apple made what was perceived to be an unusual acquisition — a small chip design start-up called PA Semi. This was perhaps the earliest indication of how far ahead Steve Jobs was thinking when he envisioned the future of the iPhone. As early as 2007, Steve Jobs was already convinced that silicon IC design would become a critical element of future smartphones and the only way for Apple to provide the best user experience would be to have complete control over silicon chip design and the entire silicon supply chain.

The PA Semi team became the nucleus of the present-day silicon engineering team at Apple. The APL series of SoCs (iPhone 3, 3G, 3GS) were designed and manufactured by Samsung, based on specifications from Apple. By 2010 (iPhone 4), Apple had already assembled a team large enough to fully design the application processor chip in-house and only use Samsung for manufacturing (as a foundry). Subsequent iPhone SoCs (A4, A5, A6 and A7) were designed in-house by Apple and only manufactured at Samsung. Apple continued to use Samsung as a foundry until 2014 (A8, iPhone 6). Since then, with the exception of one A9 SKU, TSMC has been the sole provider of Apple A-series SoC chips.

At face value, these may appear to be isolated business transactions between corporations. However, a deeper examination reveals far bigger implications that have profoundly changed the global semiconductor manufacturing landscape and have reshaped technology and industry dynamics in the US and the world for decades to come.

What follows is a discussion of some of the changes that resulted from that one seemingly isolated decision by Intel to walk away from being the SoC supplier for the original iPhone.

Semiconductor Manufacturing

When the iPhone was announced in 2007, the state of the art transistor technology node was 65 nm. Intel had introduced revolutionary strained silicon technology at the prior node (90 nm) which gave a significant boost to transistor performance. TSMC and the rest of the foundries were more than a full generation (2–3 years) behind and still grappling with Intel’s transistor advances and how best to copy them. The first iPhone was made on Samsung’s 90 nm technology which was a full generation behind Intel’s leading edge 65 nm technology at that time (2007).

Between 2007 and 2014, Samsung (Korea) was a key beneficiary of the iPhone phenomenon as volumes soared year after year. Samsung benefited from being the primary, direct supplier of leading node (90 nm, 65 nm, 40 nm, 28 nm) Apple processors until iPhone 6. During this time, TSMC (Taiwan) also benefited, albeit, as an indirect supplier of other iPhone chips to Apple via fab-less design companies (e.g. modems, sensors and connectivity chips from Qualcomm, Broadcom, NXP and others).

Since 2014, TSMC has been the primary beneficiary, both as a direct supplier to Apple (A-, S-, W-, M- series chips) but also as an indirect supplier for other leading edge and legacy node chips that go into Apple products.

When a semiconductor foundry (e.g. TSMC) manufactures chips for a large, high volume customer like Apple, it wins on multiple fronts. Analysts understandably tend to fixate on the revenue and financial metrics, which of course show marked gains. However, deeper technical implications create a virtuous cycle that strengthens the foundry technology roadmap.

It’s all about scale

For over four decades, Intel led the way in innovative transistor architectures which ensured geometric scaling in accordance with Moore’s Law. A key factor that enabled Intel to stay at the forefront of transistor scaling was that it became the predominant supplier of CPU chips in the early days of the PC wave of computing. The CPU was the primary enabling silicon platform of the PC era; and Intel, in collaboration with Microsoft established itself early on as the preeminent CPU provider for the PC industry. This lock on global CPU supply assured Intel of massive wafer volumes; which in-turn enabled quick yield learning and defect improvement; which in turn enabled volume ramps at high margins and massive revenues. These revenues in turn funded the research and development needed for the next technology node. This virtuous cycle enabled Intel to gain, and maintain technology leadership during the PC era, and in fact was a primary driver of Intel’s success as a semiconductor manufacturer. It is worth noting that Intel’s competitors in the PC era (notably, IBM and AMD) were never able to catch up to Intel’s manufacturing prowess precisely because they never commanded wafer volumes and a manufacturing scale and cadence to fuel this virtuous cycle (AMD and IBM both eventually sold off their manufacturing divisions).

To continue its leadership streak past the PC (CPU), Intel needed to ensure that it remained the preeminent provider of SoC chips for the mobile wave of computing. Apple’s decision to use Samsung (and later TSMC) as a foundry was thus a very significant strategic loss for Intel.

The virtuous cycle instead helped TSMC and Samsung strengthen their manufacturing technology programs during the mobile era. The iPhone provided TSMC with opportunities to increase wafer volumes at all nodes including legacy nodes (e.g. 40 nm and older) and advanced nodes (28 nm and newer). The legacy node wafer business allowed TSMC to run large volumes at full capacity on fully depreciated toolsets, in effect “printing money”. The advanced node, high volume and quick ramp products like Apple iPhone SoCs and Qualcomm modems were critical in helping TSMC debug and stabilize yields on the most complex, leading geometry process node, thereby quickly improving manufacturability and paving the way for subsequent waves of advanced node customers (e.g. AMD, NVIDIA, Qualcomm).

Without the massive volumes from Apple, it is unlikely that TSMC/Samsung would have been able to catch up to Intel on advanced node semiconductor manufacturing.

Samsung’s foundry business was largely developed for Apple as the anchor customer. But being the leading edge foundry for Apple helped Samsung get valuable yield learning for 5 consecutive technology nodes (90 nm, 65 nm, 40 nm, 32 nm and 28 nm). Samsung leveraged learning from these original SoC designs and built their own Exynos processors which power a large fraction of Android mobile phones today. Running large wafer volumes for Apple also enabled Samsung to rapidly ramp their own internal manufacturing as well. In effect, the iPhone indirectly launched Samsung as a credible foundry alternative to TSMC.

During the PC wave, Intel ran the majority of wafer volume and became the most advanced semiconductor company. During the mobile wave, it is TSMC that has the scale advantage and with it, the technology advantage too. In effect, whoever manufactures at scale first, wins.

TSMC now ships ~1M wafers per month (Link).

Design Ecosystem

The rise of mobile computing favored low power computing architectures (e.g. Arm) compared to the performance driven (x86) architecture from Intel for desktop and laptop PCs.

Here again, Apple’s decision to use Samsung (and later TSMC) as a foundry to make SoCs with core architecture IP licensed from Arm enabled the rise of the Arm ecosystem. Nearly every smartphone contains one or more chips with Arm IP made at TSMC or other Asian foundries. The foundries worked together with Arm and a variety of design automation (EDA) providers (e.g. Cadence, Synopsys, Mentor) to develop and standardize a process and design ecosystem. Here again, Intel was left at a disadvantage by not being part of a burgeoning design ecosystem that in effect became an industry standard.

Had Intel been the SoC provider for the original iPhone, it would have been able to influence and leverage a foundry standard design ecosystem, which in turn could have helped Intel become a dominant supplier for other smartphone providers besides Apple.

In just over 5 years, this strong design and foundry ecosystem enabled Apple (a relative novice in computer architecture and IC design) to quickly move from 32 bit to 64 bit architectures on a mobile SoC chip using off-the-shelf foundry silicon processes, an astounding feat given the relative weakness of the non-Intel design/process ecosystem just a few years prior to the iPhone.

Global Silicon Landscape and US National Interests

Intel, being the sole Integrated Device Manufacturer (IDM) in the United States has always relied on its manufacturing / technology lead as a significant competitive advantage over “fab-less” competitors (e.g. NVIDIA, Qualcomm, Broadcom). These fab-less companies have always relied on TSMC and other Asian foundries (e.g. UMC, Samsung) for their manufacturing needs and were historically at a process technology disadvantage to Intel.

During the PC wave, Intel was able to leverage its lead in semiconductor manufacturing to effectively fend off competition which relied on lagging technology from foundries like TSMC. Now that TSMC (and to a certain extent Samsung) have the most advanced semiconductor technology on par with or better than Intel, fabless competitors like NVIDIA and Apple have an inherent advantage.

Perhaps more importantly, this shift in manufacturing leadership across borders poses a serious threat to the national security interests of the United States. Advanced semiconductor manufacturing is a vital national asset and has always been viewed as such. US semiconductor companies have historically been subject to export licensing regulations which prohibit transfer of sensitive technology know-how to foreign entities (either via restricted hiring of foreign nationals or via setting up factories on foreign soil). Intel is now the only credible, leading edge, US based advanced semiconductor manufacturer.

The mobile wave of computing shifted the semiconductor landscape so that Asian companies (most notably, TSMC, Taiwan and Samsung, Korea) are at the forefront of semiconductor manufacturing and have caught up to Intel in its aggressive pursuit of Moore’s Law. In just over a decade, Semiconductor Manufacturing International Corporation (SMIC, China) has made remarkable progress, enabled largely by favorable regulation and massive funding by the Chinese government. China has deemed domestic semiconductor manufacturing as a national priority and the government is trying to enforce a “Made in China, Made for China” policy to ensure that silicon chips sold in the Chinese market are also made in China (McKinsey, Bloomberg, WSJ, WSJ). The Chinese government plan calls for Chinese-made chipsets to make up 40 percent of domestic needs by the end of the decade.

After decades of American dominance in both semiconductor design and chip manufacturing (fueled by the PC era), the tide is shifting to favor Asia (fueled by the mobile era). Decisions by US companies like Apple to promote Asian manufacturers and pro-active policy formulation by the Chinese government played a key role in this shift.

Manufacturing for AI: Advantage Asia

By firmly establishing itself as the foundry of choice during the mobile wave, TSMC now has the incumbent’s advantage in the AI wave of computing. Most of the AI enabling silicon technology platforms (e.g. GPU, FPGA, CPU, ASIC, connectivity) are already being manufactured in high volume at TSMC. System integrators like Google, Facebook, Tesla and Apple are now designing their own custom silicon for AI using TSMC or Samsung as a foundry. Chinese foundries like SMIC are at present well behind TSMC and Samsung, however with virtually unlimited government funding and protectionist state policy, SMIC will eventually play a significant role in future semiconductor manufacturing as well.

By not selecting Intel to be the manufacturer of iPhone application processor chips, Apple facilitated the rise of TSMC and Samsung as the leading silicon manufacturers for the mobile wave of computing.

The United States government should have done more over the last decade to protect domestic semiconductor manufacturing. Regaining semiconductor manufacturing competitiveness should be treated as a national priority — to ensure that foundational technology innovation continues to happen within the United States.

The views expressed herein are my own.

Subscribe now

Share

Leave a comment

During the PC era, silicon technology development was driven by a performance-centric roadmap. During the mobile era, the roadmap evolved to emphasize power constrained design. The silicon roadmap during the pervasive and ubiquitous computing era will be driven by overall power reduction, but with an added emphasis on cost and on chip- and package-level (SoC or SiP) integration

Five Vectors for Silicon Technology Development

The key functional areas that will drive the semiconductor industry forward are compute, connectivity, sensing and storage.

To enable progress in these functional areas, silicon transistor technology development is likely to coalesce around four main vectors. As the industry matures, there is likely to be major consolidation around these vectors. Some of the mergers announced in recent years can easily be categorized into one of these domains.

Mostly-Off Computing

The vast majority of “edge” computing applications will be powered by low-mid end silicon, designed for a highly constrained power envelope. The driving metric for technology development over time will be “energy-per-operation”. Digital logic, on-chip memory (SRAM), mixed signal computing (analog, IO, SERDES) will form some of the building blocks of this platform.

This platform will continue to be served by “legacy” manufacturing processes (e.g. 20nm node or larger geometry) and will utilize nearly fully depreciated manufacturing capacity worldwide, resulting in very low price-points for most of these chips (see: $5 computer by Raspberry Pi)

Mostly-On Computing

The datacenter will continue to push the frontiers of Moore’s Law and demand more performance-per-watt as a driving metric. These will typically be high-end, high performance computing parts (e.g. networking, switching, server CPU, high-end FPGA, high-end GPU, other accelerators). These applications will utilize advanced node manufacturing capacity (10nm or smaller nodes) and will continue to push for scaling in density, power and performance.

Memory

The industry is currently seeing resurgence in the development of new embedded non-volatile memory (e-NVM) solutions to succeed conventional silicon transistor based solutions like Flash technology. Embedded NVM will continue to be a big focus area until the industry settles on a new, scalable solution. Some candidates include Spin-torque based solutions (MRAM), resistive RAM solutions (R-RAM) and carbon nanotube based solutions (e.g. Nantero).

Intel’s recent announcement of a cross-point technology based Optane platform represents a breakthrough in developing storage-class-memory (SCM) and has the potential to change the contours of the data-center memory market. Optane appears to be part of Intel’s attempt to re-invent itself as a datacenter platform company (compute, storage and networking).

DRAM scaling, soon to commence the 1x scaling node will continue with novel breakthroughs like 3D stacking (Link).

MEMS

Micro-Electro-Mechanical Systems will form the fourth vector that will continue to grow within the semiconductor industry. MEMS has been the primary enabler behind many of the innovations in the smartphone and other wearable gadgets we use today. Simple actions that we now take for granted (e.g. landscape to portrait orientation change in iPad) and pedometers in smartphones are all enabled by MEMS sensors. Going forward, nearly all edge computing devices will contain at least one or more MEMS based sensors. A robust ecosystem has developed around MEMS technology led by players like Invensense, Bosch, STMicro, NXP among others.

While MEMS hardware will remain a critical piece of semiconductor devices, the technology is undergoing rapid commoditization and in terms of revenue dollars can be classified as a race to the bottom. As stated by the Invensense CEO Behrooz Abdi, the real money in MEMS is not in the hardware, but in the data (Link). Nevertheless, MEMS based sensor technology will continue advance and drive the IoT wave of computing.

Heterogeneous System Integration

Integrating two or more of the above four platforms will form the fifth vector for semiconductor technology development. A variety of packaging techniques will form the basis for this vector. The ability to integrate either on-chip or on-die a variety of functional accelerators will prove to be a competitive advantage for consumer electronics players like Apple (see my earlier essay, Computer on a Chip). In addition to consumer electronics manufacturers, system integration will also play an increasing role in bringing together disparate blocks like silicon based photonics for faster optical interconnects.

“More than Moore” is used to describe orthogonal scaling as opposed to geometrical scaling of Moore’s Law. For example, orthogonal scaling rejuvenates a legacy technology to reduce its total power envelope. Such innovation may also be extended to include enhanced on-chip functionality such as eNVM or enhanced system-level functionality via 2.5D scaling (interposers) or 3D scaling (through-silicon vias, or TSVs).

Judicious investment in these areas will enable foundries to capture a significant projected silicon volume migrating from 130 nm to sub-20 nm process nodes over the remainder of this decade. In the ultra-low cost, edge and sensor computing space, continued Moore’s Law scaling is likely to serve a limited number of applications and may not lead to high-volume design wins, at least in the near term. Whether very advanced transistor technologies (sub-10 nm) can offer the most effective integrated platforms necessary to succeed in the ultra-low cost space remains to be seen.

A holistic platform and system-level view suggests that cost, power, and highly disparate functional integration are the key technology metrics for success in the ubiquitous computing era.

If you liked this essay, please share it so that others may find it too.

Share

Leave a comment

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!

The silicon transistor continues to be at the heart of post-PC era products like the smartphone, the tablet and the smartwatch. However, the success metrics for the transistor are quite different now than they have been in the past.

Frequency (clock-speed) was the primary metric in the PC era and the standalone central processing unit (CPU) was the primary chip that drove advancements in semiconductor technology for decades. Form-factor was hardly an influencer and there wasn’t as much of a drive to integrate system-level functionality either on-chip (SoC) or in-package (SiP).

Form-factor, cost and power-per-function are now critical drivers in the mobile market and that in turn has increased the importance of on-chip integration of functional hardware (e.g. power management, computing, audio/video, graphics, GPS and radio).

This shift from mostly performance-centric chips to mostly power-constrained chips and the focus on lowering cost and increasing system-level integration has disrupted the traditional semiconductor landscape within just a decade and continues to reshape the industry today.

SoC technology was used by fabless vendors and foundries long before the advent of mobile computing. But it is the rapid proliferation of mobile post-PC products that is proving to be the catalyst for this technology to finally realize its full disruptive potential. Within the last five years, SoC technology has moved from being at the heart of smartphones to enabling many tablets and full-feature mobile computers.

This article describes the emerging importance of the SoC, its likely technological evolution and its potential impact on the semiconductor industry in a mobility driven age.

A Post-PC World

In the post-PC world, the computing paradigm has shifted in such a way that overall user experience is now a critical benchmark independent of the raw performance of the underlying technology. The Apple iPhone and iPad are great examples of this paradigm shift. Even the early versions of the iPhone provided a highly satisfactory user experience — not because they featured the fastest computing speed with the most advanced silicon but because they enabled rich features at reasonable computing speed and reasonable price points.

The silicon technology features of these devices collectively enhance the user experience — outstanding graphics rendering, wireless connectivity, instant-on, connected stand-by, long battery life and touch-screen apps. The earliest versions of the iPhone did not offer the fastest raw computer performance, but they were perceived by the average consumer to be fast and provided a far superior user experience.

Over time, in line with classic disruption theory, the SoC transistor platform caught up to the incumbent CPU transistor platform to the point where now the Apple A9X SoC offers 64 bit desktop-class computing enabling a handheld tablet to go toe-to-toe with a state-of-the-art laptop CPU from Intel (See John Gruber here).

The key to the success of early post-PC products like the iPad is the fact that they were designed from the ground-up without the baggage of legacy PC-era software (Mac OS) or hardware (x86 CPU).

Innovation around the sustaining silicon hardware technology would have called for higher performance standalone processors (CPUs) utilizing the abundance of logic transistors and even more complex layers of software to utilize the abundance of memory capacity. Instead, the new products utilize highly power constrained hardware and very lean software for accomplishing specific tasks (e.g. a video decoder to drive the display).

In order to make power-efficient systems for the mobile world, it is critical to shift as much of the burden on hardware while utilizing lean software. Simply force-fitting legacy PC hardware and software into a new form factor will not be as power efficient and hence will not lead to a superior user experience.

Of course, the emergence of post-PC devices does not herald the end of the traditional PC as we have come to know it. Racks of servers will continue to be used in data centers for compute-intensive applications for the foreseeable future. Yet, if history were a guide, it would suggest that the sustaining CPU semiconductor technology underlying traditional PC products is likely to be eventually displaced or at least substantially altered by the disruptive SoC technology underlying the smartphone. The rapid evolution of SoC based technology over the last few years supports this hypothesis.

Evolution — A Look Back

Since the advent of independent foundries at the end of the 1980s and early 1990s, the semiconductor industry has been segmented into three major entities — Integrated Device Manufacturers (e.g. Intel, AMD (pre-2009) and Samsung), fabless companies (e.g. Apple, Qualcomm, Broadcom, Altera, NVIDIA) and foundries which make chips for the fabless companies (e.g. TSMC, UMC, Samsung, GlobalFoundries).
Historically, Intel and AMD focused on making x86 architecture based CPU-based chips (e.g. Core and Athlon) while NVIDIA focused on making standalone graphics chips (GPU) for the PC and server markets. All the other players in this landscape have utilized some form of on-chip system integration (SoC) to meet the diverse needs of their respective markets.

The generic definition of a SoC is the on-chip integration of a variety of functional hardware blocks to suit a specific product application. A SoC can thus be as simple as a basic connectivity chip which combines some mixed-signal and digital circuitry. A more complex SoC may include the on-chip integration of an application processor unit (APU) and a graphics processor unit (GPU). Even more functional SoCs further integrate various other hardware blocks (e.g. image processor, audio/video decoder and modem).

It is this ability to continue to integrate disparate functionality on a chip that has enabled SoC technology to rapidly evolve from supporting a simple feature-phone to a smartphone and all the way to a tablet and a laptop.

Qualcomm started out by designing chips for the growing connectivity market with the advent of cellular telephony and the internet. Nvidia came to light as a maker of standalone graphics chips. Over time, each of these companies responded to an evolving technology trend and built upon their initial successes as they incorporated ever higher levels of functional integration into their chips.

Qualcomm evolved its product line-up from standalone connectivity chips by adding an applications processor (Krait via ARM license), a GPU (Adreno via AMD Imageon buyout) and a power management unit. Qualcomm’s flagship products (Snapdragon family) now include all these blocks making it a highly functional mobile SoC product.

NVIDIA evolved from a maker of standalone graphics chips by adding an applications core (via ARM license) and a connectivity block (via Icera acquisition). Nvidia now offers highly integrated mobile SoCs (Tegra family) which power multiple tablet computers.

Apple, which was not even in the mobile chip design business until a few years ago, started designing its own SoC based chips (A- family) using an application processor (via ARM license) and a graphics processor (via license from Imagination Technologies).

An indicator of the growing influence of the SoC is the consolidation trend within the industry. Apple acquired PA Semi, enabling it to design its own application processors. Qualcomm acquired Atheros to strengthen its wireless connectivity suite and Summit Technology for enhanced power management capability. NVIDIA acquired Icera to strengthen its connectivity offering and Intel acquired Infineon Wireless to gain entry into the baseband connectivity market. These acquisitions point to a consolidating market in which only a few strong players have all the required functional blocks and are getting ready to fiercely compete in the growing mobile market.

The smartphone offered the first significant platform for SoC technology to demonstrate its potential and put the SoC on a collision course with the standalone CPU. The smartphone valued on-chip integration far more than a standalone desktop. Utilizing dedicated functional blocks has several advantages over general purpose processing cores — these blocks can operate at lower frequencies while delivering higher system-level performance and consuming much lower system-level power.

By moving more functionality to hardware, the SoC enables lean software which results in lower system-level power. Using dedicated cores enables the smartphone SoC to only turn on specific blocks for specific tasks whereas a general purpose CPU would have to be on all the time regardless of the task being performed. An SoC is thus far better suited for mobile devices compared to a standalone CPU.

Collision Course

Early leadership in SoC technology put the foundry ecosystem players at a significant advantage over incumbents like Intel and the technology also benefited immensely from rapid growth in smartphone shipments. Intel was unable to break into the smartphone market for the first five years (until 2012). The introduction of the iPad and the subsequent growth in the tablet market further solidified this trend.

An indicator of the disruptive potential of the SoC is the rapid rate of advancement — not only in terms of functionality and shipped volumes but also the proliferation of a robust design and software ecosystem to support it. In just five years, SoC technology has catapulted from enabling basic computation/connectivity on a feature phone to being at the heart of all smartphones, tablets and Chromebooks, capable of a wide range of functions including audio/video, gaming, communication and productivity.

The unique cost structure enabled by the SoC has the potential to truly disrupt the business model in the semiconductor industry. The ASP for an SoC chip (NVIDIA Tegra or Qualcomm Snapdragon) is in the range of $10-20 while the ASP for a leading edge Intel CPU chip (Skylake) is in the $150 range. The CPU chip will also need to be supported by other chips (e.g. power management) to provide all the functionality that is provided by a single SoC. When OEMs compete on price, it will be very difficult for the CPU product to compete while retaining historically high profit margins. As the SoC gets better and encroaches into the laptop space (e.g. NVIDIA SoC powers the ChromeBook Pixel C), the cost differential will have an even larger impact.

The rising influence of a low-cost, low-end technology (SoC) and its potential to eat into the profit margins of a sustaining high-cost, high-end technology (CPU) is an example of the classic disruption theory articulated by Clayton Christensen (Link).

The Apple A9X SoC and the Intel Skylake CPU are both best-in-class products — but they are designed for different form factors and cost and value metrics. While it is conceivable that the low-margin SoC may be able to serve the high-end laptop market well, it seems unlikely that the high-margin CPU will be able to serve the low-end mobile market as well. Today’s high-end iPad Pro powered by ARM based A9X SoC performs very comparably to (by some metrics, even better than) the Surface Pro4 powered by an Intel x86 CPU processor.

Intel has a significant lead in CPU process technology and is at the forefront of Moore’s Law. However, radical changes to architecture (e.g. 14nm FinFET) may slow down the integration of on-chip functionality. As an example, in spite of acquiring baseband technology from Infineon in 2011, Intel has yet (in 2015) to integrate it with in-house 14nm technology. Intel’s SoC product offering has traditionally lagged its mainstream CPU offering by 1–2 years. That gap is expected to narrow in the coming years as Intel addresses the growing need for on-chip integration and the growing threat from seemingly low-end product offerings which are rapidly becoming more competitive and cost-effective in the high-end.

Convergence

The present decade represents a period of strategic inflection in the evolution of the semiconductor industry — the next five years are likely to see a confluence of several technology and market forces which will collectively have a profound impact on the course of the industry. These trajectories are discussed below.

Trajectory #1: SoC (and not CPU) Drives Foundry Roadmap

Mobile SoC (Android and iOS) shipments already dwarf CPU shipments by 5:1 (See Benedict Evans Link). The direct result of this trend will be the increasing importance of the mobile SoC in shaping the foundry transistor technology roadmap. It is important to note that the transistor design points for a CPU are far more restrictive than those for an SoC. Historically, the CPU (and hence, Intel) has been the driving technology that influenced transistor development. Going forward, the SoC is likely to take on that role. This in turn will significantly challenge the dominance of Intel/x86 architecture as most mobile SoCs are designed with ARM architecture. Functional integration is expected to continue making the SoC far more sophisticated and powerful. It will also evolve to consume less power as it moves to advanced geometries. Apple has demonstrated solid performance gains over successive iterations of their flagship SoC (Ax). There is likely to be fierce competition among lower tier design houses (Qualcomm, NVIDIA, MediaTek) as each tries to incorporate more functionality into their chips and win designs for new mobile products. The ascendance of the SoC will thus force disruptive changes to the traditional IDM cost structure and business model.

Trajectory #2: A Central Role for the GPU

Usage models of the tablet and the smartphone indicate that the GPU is the most heavily used block within SoCs like the Tegra, Snapdragon and the A8X. Since the GPU is the largest block and also consumes most of the power on the chip, it is instructive that the silicon transistor be designed to optimize the performance and power of the GPU. It is likely that design houses and foundries will make the GPU the centerpiece for transistor design and manufacturing — historically all the blocks including the GPU had to adapt a transistor that had primarily been designed for the CPU. The rapid evolution of the SoC and the increasing role of the GPU are evident in successive generations of Apple A*x family processors. The GPU on the A8X processor occupies almost a third of the die area.

Die photos of Apple A5X, A6X and A8X SoCs — The GPU occupies a substantial portion of the die (30–40%). Similar trends can be observed in the Snapdragon (QCOM) and Tegra (NVDA) family of processors (Source: Chipworks).

Trajectory #3: A New Cadence for Silicon Technology

Tablet and smartphone offerings are refreshed once every year — much faster than the historical PC refresh cycle. The semiconductor industry will need to adjust its technology development lifecycle to keep pace with the mobile product lifecycle. It is not feasible to scale-down transistor geometry every year beyond the 14nm node — however it is quite feasible to rapidly incorporate increasing levels of functional integration into an existing geometry. As the law of diminishing returns eventually catches up with Moore’s Law, there will be little economic incentive to scale transistor feature size. Companies at the leading edge of Moore’s Law may be able to compete effectively in high margin segments (servers and data centers) but will find it difficult to price their parts competitively for the low margin consumer markets.

These vectors are likely to put SoC technology at the heart of the semiconductor industry.

The modular, lego-like integration of SoC enables even relative newcomers to quickly put together competitive chip offerings — increasingly, fabless design houses like Qualcomm will face tremendous pricing pressure from low-cost design houses (e.g. MediaTek, RockChip, AllWinner). Design IP providers like ARM and Imagination Technologies are poised to benefit immensely as well.

Foundries are well positioned to capitalize on this trend and will benefit from refocusing their efforts on transistor design in a way that is GPU-centric rather than being CPU-centric. Intel will continue to face increasing competition in the mobile market and Intel’s product mix may reflect a move toward more on-chip functional integration in the years to come. More importantly, Intel will be forced to also compete with SoC technology in the laptop and PC segments and doing so may necessitate a change not only in its technology direction but also in its business model.

If these trends continue, there is no reason why a highly integrated SoC chip cannot displace a standalone CPU chip in a premium laptop (e.g. MacBook Air or MacBook Pro). The boundaries between the standalone CPU and the SoC are thus likely to erode in the years to come as the industry embraces and unleashes the full disruptive potential of the SoC.

If you liked this essay, please click recommend below or share it so that others may find it too.

A few years ago, Marc Andreessen made an eloquent argument for why software companies were poised to take over the world (Link). Andreessen provided several examples of how relatively new software companies (Amazon, Netflix to name a few) were disrupting their traditional brick-and-mortar counterparts. In just a few years since Andreessen’s observation, Borders and Blockbuster are now bankrupt and many new companies are bolstering the case for software eating the world. The world’s largest taxi company today (Uber) is in fact a software company and doesn't actually own a fleet of cars. The world’s largest accommodation provider is also a software company (AirBnB) yet doesn't own a single room. These are just a few of the incredible disruptions that are happening in our time.

Coming from a semiconductor background, I often think about technology transformations from a silicon (i.e. hardware) perspective. And recent trends have convinced me that the semiconductor landscape is poised for a transformation of its own (you can read some of my earlier posts on this topic here). Can the role of software in disrupting e-commerce, digital content, taxis, hotels and scores of other businesses be emulated in the hardware business itself? Eventually, could software take a bite out of traditional silicon chip design too?

Evolution of Computer Architecture

A sweeping visualization of the history of computer architecture is shown below in a chart compiled by Prof. Mark Horowitz and his team at Stanford University.

Over four decades of continuous advancements in silicon technology delivered a steady increase in the transistor count on a chip (Moore’s Law), accompanied by a phenomenal increase in frequency (speed). And during the first three decades of that journey, chip designs mainly employed just a single CPU core. At the turn of the century, as single threaded, serial processing performance began to plateau, the solution was seemingly straightforward — to add a second core and enable parallel processing. Soon, dual-core morphed into multi-core and then many-core processing. Adding more computing cores continued to improve chip performance for over a decade. But now, even that option is running out of steam. Simply adding more cores is now providing only marginal returns.

Decades of technological advances that made silicon chips smaller, faster and cheaper have brought us to a point where the critical dimensions on a silicon chip can now be counted in atoms. And while it is still fundamentally possible to continue geometrical scaling even further, the cost benefits that drove Moore’s Law in the past are now greatly diminished. A new architectural paradigm will thus be required to continue the improvements in chip performance and enable the massive back-end computing infrastructure (datacenters) that will be required to support the Internet of Things.

One way to extend the performance gains from a standalone CPU chip is to pair it with a “hardware accelerator” chip. The hardware accelerator chip can be either a field programmable gate array (FPGA) or a General Purpose Graphics Processing Unit (GP-GPU).

As I will explain shortly, each of these options involve software eating at least a portion of silicon chip design! This may not be the same disruption we saw in music, e-commerce, movies or taxi services; at least not just yet. But it is fascinating to note that it is in fact yet another example of software pushing the frontiers of technology and innovation.

Intel acquires Altera

In June this year, Intel announced that it was acquiring FPGA maker Altera. Among the many op-eds trying to explain the rationale behind this deal, the one that resonated the most was the one by Kurt Marko (Link).

“The only way the Altera deal makes sense is if we are on the precipice of a secular shift in system design, not unlike the transition from proprietary RISC CPUs to x86, in which rapid hardware customization is the best path to faster performance. If true, the Altera deal is Intel’s acknowledgement that the benefits of brute force, Moore’s Law scaling have shrunk and that continuing an upward performance trajectory is more dependent on system design than semiconductor physics.”

An FPGA is a blank slate composed simply of a vast array of transistor logic elements and switches capable of being interconnected via a dense metal wiring stack. A software configuration file can then be loaded onto this blank slate to “program” the physical hardware circuitry. The FPGA thus allows software to convert a generic chip into specialized circuitry, custom designed for a specific computation or workload. Moreover, if the nature of the computation changes over time, the chip can be reconfigured with new software designed to make the new workload run more efficiently. In principle, this is a far more desirable approach than having to replace the hardware itself.

When a programmable FPGA chip is paired up with a general-purpose CPU chip, it allows system architects to divert specific workloads to the FPGA, enabling them to “accelerate” and run more efficiently since the FPGA can be custom configured to run the specific workload unlike the general-purpose CPU. When the workload changes, the accelerator can be reconfigured for the new workload, thus delivering better performance without the need for a hardware or infrastructure upgrade.

For example, Microsoft could use a pre-built configuration library to accelerate specific Bing queries at specific times of the day. Amazon could have unique configuration files to accelerate e-commerce workloads depending on time of day and location. Image recognition could use its own configuration libraries and voice searches could use their own too. Several companies including Intel and Microsoft have written about the performance advantages of programmable hardware accelerators and benchmarking tests show anywhere from 2X to 10X speed-up on workloads like image recognition, machine learning and convolutional neural networks (Read more here).

Programmable hardware is thus a huge enabler of next generation high performance computing and is a key element of Intel’s datacenter roadmap.

General-Purpose GPU

Configuring an FPGA typically involves a hardware description language (HDL) like Verilog or VHDL and as such is not a trivial exercise and perhaps not “high level coding” in the traditional sense (e.g. based on C code). Altera is trying to simplify the process using a platform called OpenCL. OpenCL allows programmers to develop code in the C programming language instead of low level hardware languages. Hence, configuring an FPGA requires engineers with knowledge of chip design, essentially chip designers and not pure code developers. One could imagine however, that over time, reconfiguring an FPGA could be elevated entirely to a high level where a traditional “software engineer” could also reprogram the chip.

An alternate approach is to use a general purpose GPU (GP-GPU) as a hardware accelerator. NVIDIA (NVDA) is a key proponent of this approach using their CUDA architecture. CUDA is an instruction set architecture and hence completely software programmable. And as a result, it is a general purpose parallel computing architecture, and it’s completely reprogrammable. As Jen-Hsun Hwang, the CEO of NVIDIA likes to point out, GP-GPUs are reprogrammable, while FPGAs are reconfigurable.

Software Taking a Bite out of Hardware

While not a perfect solution yet, it is clear that programmable or configurable hardware has the potential to deliver massive gains in performance (2X or higher) compared to traditional Moore’s Law geometrical scaling (typically 30% speed-up per node). The key to the success of accelerated computing is of course the ability to dynamically change the configuration of the accelerator to meet the demands of a changing workload. It is thus “software” that is enabling the next big advance in computing hardware. Accelerated computing may become the panacea for the increasing complexity and cost and marginal returns of traditional Moore’s Law scaling.

While it remains to be seen how exactly the new developments play out, it is clear that programmable hardware will become a major computing platform in the coming years.

If you liked this essay, please click recommend below so that others may find it too.

Intel dominated and defined the semiconductor landscape during the PC era on two complementary fronts — silicon process technology and computing architecture (x86). Through its partnership with Microsoft, Intel enjoyed a near complete monopoly over the computing landscape during the PC era. That dominance began to erode with the emergence of two Segment Zero markets (Link) for Intel — embedded computing and mobile computing. The company that under the leadership of Andy Grove had successfully identified and vanquished at least two prior disruptive threats (Japanese memory makers in the 1980s and low cost PCs in the early 1990s) failed to successfully prepare for the next disruption — mobile computing and the ecosystem pioneered by ARM, the leader in low-cost/low-power architecture. While Intel pioneered the era of the standalone CPU with a vertically integrated business model, ARM enabled a massive lateral design/foundry ecosystem and pioneered the era of the mobile SoC (system-on-a-chip).

CPU vs. SoC

In the CPU space, chip functionality is largely determined by the computing core (e.g. Pentium, Athlon) and transistor performance is the critical metric. In the SoC space, the core is just one among a variety of IP blocks that are used to independently deliver functionality. Intel’s foray into SoC technology started in the early 2000s and was largely a response to the success of the foundry ecosystem. However, Intel’s SoC process technology has typically been implemented 1–2 years behind its mainstream CPU technology, which historically has focused on transistor scaling and performance. The foundries within the ecosystem instead focused on integrating disparate functional IP blocks on a chip while also aggressively scaling interconnect density.

The semiconductor industry today is increasingly driven by low-power consumer electronics (primarily smartphones) and SoC shipments now dominate total silicon volume. The sheer volume of desktop class computing chips like Apple A9 SoCs shipped to date has in turn dramatically improved the competitiveness of the foundry ecosystem (led by TSMC) compared to Intel. Until a few years ago, Intel’s process technology lead was unquestioned. That lead is now greatly diminished as the foundry ecosystem is on track to ship more 64 bit SoC chips than Intel by the end of this year.

The ascendance of ARM has not only displaced Intel’s leadership on the architecture front (x86) but indirectly, also on the process technology front by enabling the foundry ecosystem to ship incredibly large volumes of leading edge silicon and dramatically speeding up the manufacturing yield learning curve. Intel was late in recognizing the importance of the SoC and now finds itself playing catch-up to a strong ecosystem led by ARM on the architecture front and TSMC on the silicon process technology front.

Compounding this trend further is the reality that after 50 years of delivering consistent gains in power, performance and cost; transistor scaling is finally entering an era of diminishing returns where further shrinking the device is not only costly, but provides incremental gains in performance and power.

Meanwhile, the ARM ecosystem is also steadily making inroads into the high-end space traditionally dominated by Intel. Several new tablet and laptop computers (e.g. Google Pixel C) use SoC chips designed by fabless companies instead of CPU solutions from Intel. Over time, SoCs became much more powerful and competitive and now pose a meaningful threat to the standalone CPU. The predominance of the Intel-Microsoft partnership based on x86 architecture is waning and a huge swath of the mobile computing space is now supported by low cost Chinese design houses like MediaTek, AllWinner, RockChip and Spreadtrum that use ARM architecture and foundries like TSMC, SMIC or UMC.

The emergence of the SoC was thus a strategic inflection point for both Intel and the ARM ecosystem alike. While the silicon landscape during the PC era was defined by Intel and the CPU, it is fair to say that the silicon landscape during the mobile era continues to be defined by the SoC and the foundry ecosystem led by ARM and TSMC. In many ways, Intel’s ability to compete in the SoC space will determine the direction of the chip wars in the next wave of computing (IoT).

Transistor Technology

The process technology underlying CPUs and SoCs is similar, however the design points for each can be vastly different. For example, a CPU design requires fewer transistor variants spanning a limited range of leakage and speed. On the other hand, SoC designs require many more transistor variants spanning a much wider range of leakage and speed. SoC technology also needs to support higher supply voltages for IO devices (e.g. 1.8V, 3.3V) in addition to the nominal supply voltage for core devices (e.g. 0.9V) These differences, though subtle, require very different mindsets in transistor design and process architecture.

Intel’s focus on transistor performance can be traced back to the height of the PC wars when the benchmark was clock speed. While Intel focused on transistor performance, the foundries adapted Intel’s transistor innovations for their own SoC integration needs. In addition, they aggressively pursued metal density scaling and cost reduction. While Intel pursued a limited vertical functional integration, the foundries developed a lateral ecosystem and designed transistors for a variety of vendors that independently optimized functionality for each IP block (CPU, GPU, radio, modem, GPS, IO, SERDES, etc.).

This vast ecosystem of existing design IP is now a significant influence on the adoption of the next transistor architecture. Arguably, the foundries are today better positioned for the SoC era. By the end of 2015, TSMC will have shipped well over 100 million units of Apple’s A9 SoC. These processors are made in 16nm technology and will set new benchmarks for cost, power and connectivity features. The Apple A9 processor is possibly the most highly integrated SoC running on the most advanced silicon process technology (at TSMC and also Samsung). Intel’s advantage at the transistor level thus allowed it to win the CPU space, but the ecosystem has the advantage at the system level and is poised to win the SoC space.

In the mobile and IoT era, packing as many features on a chip as possible at the lowest integrated system cost and power will win. The transistor technology that is most compatible with all the IP needs of a complex SoC at the lowest cost will thus have the upper hand.

The Post-PC Era: Intel in an Open Ecosystem

The slowdown in the pace of Moore’s Law, the emerging importance of the SoC and the rapid growth of the mobile market all tend to favor an open, plug-and-play foundry and design ecosystem. One could expect that the ecosystem developing around ARM will continue to nip at Intel’s core markets as the development of ARM-based processors for laptops and servers accelerates. This emerging threat to Intel and Intel’s response to it will define the industry over the coming decade.

The operating system (OS) war between Microsoft and Apple in the 1980s came to define the PC and software industries. Microsoft’s open ecosystem model won as Windows became the de-facto OS for machines made by all kinds of PC makers. While Microsoft promoted an open ecosystem in the larger PC industry, ironically it spawned a closed ecosystem within the semiconductor industry. The Wintel alliance ensured that Windows only ran on x86 architecture which was pioneered and owned by Intel. The closed ecosystem hugely benefited Intel as it went on almost unchallenged to win the desktop, laptop and server space (AMD also used x86 yet could never match Intel’s scale or manufacturing expertise). A hallmark of the post-PC era is the emergence of an open ecosystem within the semiconductor industry.

Unlike the Windows/x86 dominance of the past, the post-PC era is being defined by competing OS options (iOS, Android or Windows) and competing processor architectures (x86 or ARM). Today, the momentum is in favor of ARM-based operating systems as the vast majority of mobile devices being shipped today run iOS or Android (ARM architecture).

The chip wars will be fought in this fragmented and open ecosystem on three fronts — SoC (system integration), CPU (core architecture) and silicon (foundry technology). While performance and power will continue to be important benchmarks, the open ecosystem supporting a worldwide consumer market will make cost a key success metric on each battlefront.

Battlefront #1 — SoC (System Integration)

In the mobile SoC space, the battle for processor architecture will be between Intel on the one hand and incumbents like Qualcomm, Samsung and Apple on the other. In the mobile, power constrained space, it is more efficient to integrate a variety of hardware accelerators on a single chip to deliver custom functionality as opposed to implementing a general purpose core serving most functions. Low power cores are supplemented with elements as disparate as an on-chip radio, global positioning system (GPS), modem, image and audio/video processor, universal serial bus (USB) connectivity and a graphics processing unit (GPU). An open ecosystem is far more cost-effective for such modular, plug-and-play system-level integration.

Historically, Intel, being an integrated device manufacturer (IDM) has independently designed most of the functional IP blocks, while ensuring that each uses Intel transistor technology and process design rules. Intel’s process technology leadership has benefited it enormously in the CPU space giving its designers access to best-in-class transistor performance. However, Intel’s ability to compete in the mobile SoC space will be determined by how well it can re-engineer its CPU process technology to meet the diverse needs of a complex mobile SoC.

If Intel can successfully design and manufacture 14nm and 10nm processes that span the full range of the performance-power spectrum required for mobile SoC applications, it will have an edge over the competition. But for Intel to compete effectively in the mobile SoC space, it will also need to offer a cost advantage. Average Selling Price (ASP) in the SoC space is a fraction of that in the CPU space. While fabless Apple can drive the best possible deal from competing foundries, IDM Intel needs to ensure that its volumes and ASPs are high enough to recoup its own development and manufacturing CapEx.

Intel may try to enhance its SoC functionality offering by way of more acquisitions like Infineon Wireless. But post-merger, porting Infineon’s foundry standard design rules to Intel’s proprietary design rules will be non-trivial (In 2015, nearly 5 years after the acquisition, Intel is yet to port Infineon’s modem chips to their own fabs and continues to make them at TSMC!). By contrast, the Qualcomm acquisition of Atheros likely proved to be more seamless since the IP was from the open ecosystem and already foundry compatible.

Battlefront #2 — CPU (Core Architecture)

The main battle on the CPU front is between Intel/x86 and ARM architecture. While Intel historically has had the upper hand in performance, ARM-designed cores have delivered superior performance/watt.

To effectively compete against ARM, Intel will need to design its low-power Atom cores in the most power-efficient way possible. To design a true low-power core, Intel may need to decouple the Atom from legacy x86-based architecture and develop a new ground-up design that delivers highly competitive performance/watt.

Intel will also have to be in aggressive catch-up mode as it tries to reverse the momentum of an already large, established and robust ARM software ecosystem. In the initial years of the PC era, as x86 became the predominant CPU architecture, an entire ecosystem of application software was spawned that was designed to run solely on x86. This effectively precluded or seriously hindered competing architectures like PowerPC from ever gaining a foothold in the marketplace. Analogously, in the present day, ARM architecture is significantly further along in achieving critical mass in the mobile SoC space. The prevalence of ARM in a range of post-PC devices from smartphones and tablets (90% market share) to televisions and cars has placed ARM in a commanding position to inhibit the newer Intel Atom architecture from achieving traction. Practically speaking, for Intel to gain a meaningful share in the mobile market, it now has to ensure compatibility with the ARM software ecosystem. This again, will force Intel to compete on price which will limit how much revenue it can eventually generate. This is a dynamic that Intel never had to face in the PC segment.

Battlefront #3 — Silicon (Foundry Technology)

Intel’s ability to make the best performing transistor at the highest possible yields and volumes is unparalleled. This capability served it immensely well in the closed ecosystem when Intel was essentially competing against itself in the quest to make a smaller and faster transistor. In the closed ecosystem, performance trumped power; and design flexibility and high ASPs ensured that development cost was not a significant limiter.

In the open ecosystem, however, the ability to integrate disparate functional accelerators in the most power-efficient and cost-effective manner is paramount. As an example, TSMC is able to deliver the highly successful and functional A9 processor for Apple using a state-of-the-art 16nm transistor process and integrate a variety of complex IP blocks while keeping the ASP under $20. TSMC’s minimum metal pitch at the 16nm node is larger (i.e. less dense) than that of Intel at the more advanced 14nm node, yet the A9 SoC can offer better power efficiency than a comparable 14nm CPU at an acceptable performance point and much lower price point and a much smaller form-factor.

In the post-PC era, mobile and IoT computing will have a larger influence on the semiconductor landscape. The success metrics in the new landscape are not just higher transistor performance but higher system functionality, lower system cost and lower power.

Based on the above discussion and judgment, the following trends are likely to define the semiconductor industry over the next decade.

1. Shrinking pool of advanced semiconductor fabs: The economics of Moore’s Law and the advent of mobile computing have led to a dramatic reduction in the number of advanced semiconductor manufacturing sources. Just 3 major entitities (Intel, Samsung, TSMC) now offer unique 16nm or advanced technology. (Globalfoundries is effectively just a manufacturing partner for Samsung). A wildcard here is SMIC (Semiconductor Manufacturing International Corporation, Shanghai). Even though it is a relative newcomer, SMIC is extremely driven and has the full backing of the Chinese government which has made advanced semiconductor manufacturing a national priority. SMICs entry at 14nm (by 2020) may change the foundry landscape by dramatically altering silicon wafer price-points.

2. Making things smaller doesn't help much anymore: The 28nm node will be the longest running planar transistor technology. In a departure from prior technologies, and in response to plateauing transistor cost, the leading foundry (TSMC) has developed over 5 flavors of the technology for all applications ranging from high performance 28HPM (FPGA, GPU, mobile SoC) to ultra-low power 28ULP (IoT edge computing). As the mobile computing era matures and the IoT computing era emerges, majority of the applications will be served by 28nm or older technology. As technology development lifecycles get longer and product lifecycles get shorter, foundries will try to extract all the goodness in an existing transistor technology before moving to the next one.

3. Even fewer applications for advanced technologies: Only a minority of applications (e.g. high performance computing, AI/AR, machine learning, computer vision) will migrate to using sub-10nm and lower technology nodes. And these advanced nodes will also be long lived with multiple variants serving disparate power/performance/cost points.

4. Intel CPU leadership: Intel will continue to dominate the single thread/high performance CPU/server segment, albeit with increasing competition from the ARM ecosystem. Intel’s acquisition of Altera is a defensive move aimed at creating a moat around its server leadership. However, the next five years will likely see the emergence of competitive ARM based servers. Using an open ecosystem with customizable IP will enable significant cost and power reduction for these new entrants.

5. Lego block on-chip integration: In the power and cost competitive IoT era, on-chip integration of hardware accelerators (modem, CPU, graphics, etc) will continue to be extremely efficient. Compared to centralized CPU/GPU cores, SoCs will be far more effective, especially in the smartphone, tablet and convertible form-factors. As silicon scaling plateaus, packing as many disparate functional blocks as possible on a chip within a given transistor budget at the lowest integrated system cost and power will win. Companies will try to expand their footprint by capturing more real estate on the chip, either through consolidation or on their own.

6. Ascendance of the SoC: Intel’s 14nm CPU (Skylake, 2015) and Apple/TSMC’s 16nm SoC (Apple A9, 2015) are two marquee technologies/products that will provide a barometer on the semiconductor landscape. Several benchmarking results indicate that the A9 is perhaps the most efficient mobile SoC with unparalleled performance/power metrics. This match-up will have remarkable implications — not only will it validate the rise of Apple as the dominant SoC design team, it will also suggest a vulnerability in Intel’s process technology leadership. It suggests that TSMC could go toe-to-toe with Intel on radical and highly complex transistor architectures (16/14nm tri-gate), while also supporting best-in-class SoC technology which is the enabling platform for mobile and IoT computing. Intel will need to dramatically improve its SoC offering in the years to come in order to be competitive in the SoC/IoT space.

7. Slowing cadence of Moore’s Law: Two technologies that have the potential to significantly influence the economics of Moore’s Law and disrupt the industry cost model are (a) 450mm wafer size and (b) EUV lithography. However, a glut of fully depreciated 300mm fab infrastructure and decades long slow progress in the EUV tooling roadmap will make it a difficult value proposition at least in the foreseeable future. Conventional Moore’s Law scaling is likely to give way to more orthogonal scaling approaches (More-than-Moore) including 3D chip stacking and system/package level integration of heterogeneous chips.