Meta finally lays out a strong roadmap for at least four iterations of its internal-use MTIA AI accelerator, promising serious raw performance on paper as well as a solid understanding of what’s needed for rapid and widespread adoption among their developers.

It’s more than just a trend now for everyone to design their own custom hardware. If you have the money, people, and consistent internal demand for specific workloads, why not? As we’ve seen in the past though (Intel’s handling of Gaudi), roadmaps can be fragile and are not a commitment. Meta’s situation is different fortunately, since they have:

• Extensive experience deploying AI-focused clusters reliably, at scale, customising deployments to their benefit

• Multiple generations of MTIA already proven with a silicon-to-rack supply chain now established

• No pressure to make hardware for external users, plenty of internal demand to saturate what they build

Proof of their wisdom is that all the future MTIA generations they’ve announced will be designed to sit in the same chassis and racks, so they can easily upgrade existing deployments rather than having to build new racks and power/cooling architectures each time. This will make it easier for them to stick to their promised 6-month-average cadence too, outpacing hyperscalers and most merchant silicon vendors in iteration speed.

If you’re hands on in the datacentre or even involved in maintenance and upgrade operations in any capacity, this should excite you. If it doesn’t, then think about what it would be like if your job was just to simply pull out servers from racks, replace a few cards or just a board, and plug everything back in just as it was. Sort of like replacing PCIe cards (well, maybe a little more complicated).

Or perhaps if you’re responsible for any financials around multi-megawatt datacentres running such accelerators, what if the upgrade process was just buying the latest boards (not servers or racks) and having the task take days rather than weeks? No need for expensive and disruptive retrofits to the power and cooling systems facility wide anymore.

Ok, it’s probably not that simple. Coming back to reality now, there are a few bumps in this road that Meta will experience but nothing they can’t power through with their abilities.

The MTIA 400 is the last generation in this roadmap that will need infrastructure changes. First, the power will jump significantly from the MTIA 300 - by 50% in one generation. Across the whole server and rack this percentage increase gets diluted, but it still means an imminent upgrade in power delivery and cooling systems that will last as far as the MTIA 500’s deployment too.

Also, the scale-up domain will grow by 4.5x, from 16 devices to 72. This means Meta will need to leverage its experience from deploying NVIDIA’s NVL72 racks and upgrade any MTIA 300 racks to handle a high-bandwidth domain this large. That’s new intra-rack copper/optic/hybrid cabling which brings along with it new structural and cooling challenges too.

All in all, there will be some bumps in the road, but nothing Meta don’t already have the capability to handle. It’s possible that with liquid cooling and power architecture upgrades in the future, Meta might even get to keep their racks and chassis for the MTIA 600s and onwards. No one really knows with much certainty what a world-class datacenter of 2028 will look like after all.

Meta are very clear that MTIA will not replace their GPU fleet - training workloads still need NVIDIA hardware to be run effectively, and undoubtedly some inference will still be done across the many generations of GPUs that Meta operate efficiently.

But unlike many in the custom silicon game, they aren’t pushing to support general-purpose training or branching their roadmap into workload-specific lanes. Instead, from MTIA 400 onwards, Meta are pushing for supporting modern LLM inference efficiently first, and then optimising for their own ranking/recommender models later.

The hardware looks great so far, but we’ve seen many times that the robustness and capability of the software can completely define the future of an accelerator too.

Here, Meta are arguably one of the best positioned in the industry as founders of the PyTorch foundation and now among the world’s largest users of vLLM and Triton. In addition, they’ve written extensively on their own networking and observability stacks (FBOSS, NCCLX, Dynolog etc.) and how they can optimise both NVIDIA and their own hardware for their use cases.

The specs we know so far:

• MTIA 300

  • In production now, already deployed at meaningful scale for Meta’s own workloads

  • 1 compute, 2 network chiplets

  • ~1.2 PFLOPS FP8, 216GB HBM @ 6.1 TB/s, 800W TDP

  • 1TB/s scale-up unidirectional, 1.6T scale-out

  • 16-device scale-up domain

• MTIA 400

  • Deployment soon, framed as genuinely performance-competitive

  • 2 compute, 2 network, 1 SoC chiplets

  • ~6 PFLOPS FP8, 288GB HBM @ 9.2 TB/s, 1200W TDP

  • 1.2TB/s scale-up unidirectional, 800G scale-out

  • 72-device scale-up domain, in one rack, switched backplane

• MTIA 450

  • Due early 2027, tuned directly for LLM inference

  • 2 compute, 2 network, 1 SoC chiplets

  • ~7 PFLOPS FP8, 288GB HBM @ 18.4 TB/s, 1400W TDP

  • 1.2TB/s scale-up unidirectional, 800G scale-out

  • Doubling die-to-die interconnects from MTIA 400

• MTIA 500

  • Due late 2027, the most modular and most ambitious

  • 4 compute, 2 network, 1 SoC chiplets

  • ~10 PFLOPS FP8, 384-512GB HBM @ 27.6 TB/s, 1700W TDP

  • 1.2TB/s scale-up unidirectional, 800G scale-out

  • 2x2 compute chiplet arrangement

“eXtra-dense Pluggable Optics”, or XPO, is a new optical pluggable form factor showcased publicly earlier this month at the Optical Fibre Communications conference. Currently SFP - small form-factor pluggables - are used for converting between the optical and electrical domains, allowing long-range signals carried through optical fibres to be converted and then passed into server chassis and into chips.

Datacenters everywhere contain multiple types of such pluggables. These range from small 10G SFPs designed for carrying multi-kilometer signals, to large 1600G “Octal” OSFPs that carry 160X that bandwidth over as little as 30 meters. Given the number of such modules already deployed and the volume being manufactured for replacement and new builds, it seems that no matter what new technologies come out, optics are here to stay for a while.

Even with NVIDIA (and earlier, Broadcom) wanting to aggressively push their Co-Packaged Optics (CPO) switches into mass adoption, the demand for optics appears to just keep growing. Regardless of the bandwidth demands of the latest accelerators and workloads, hardware vendors must let the inertia of the industry carry them along, or risk getting left behind before the market is ready for their revolutionary but as of yet narrowly supported new technologies.

This is where Arista’s experience and foresight show; XPOs are still field replaceable pluggables, use mostly the same supply chain and manufacturing base as existing pluggables, and come with a roadmap backed by multiple industry leaders. In comparison, difficult to manufacture CPO products or relatively untested optical switching technologies are very high risk in the long term for investors and deployers.

Both availability and serviceability are both much more significant deciding factors for buyers than most people, especially engineers, recognise. The choice between a something performant and cheap, versus something that will be widely available thanks to a mature supply chain and easily serviceable because it can be swapped out for a replacement in minutes rather than hours, is very obvious to anyone who has had to work overtime and watch profit margins collapse due to random hardware failures.

XPO modules are indeed “eXtra-dense” as made clear at the conference by the demos from various contributors to the XPO MSA (multi-source agreement) like TeraHop, Amphenol, and Molex to name just three. These take eight MPO-16 cables each carrying eight 200G lanes, for a total of 64 x 200G lanes per module. 64 lanes both in optical and electrical, hence gearbox chips aren’t needed either like they might in some smaller transceivers.

The current spec reaches 12.8T per module whilst taking up just a quarter as much area on the front panel of a switch. This means each switch now only needs to be a quarter the height now, and racks can be four times as dense in bandwidth. It’s unclear if this will be practical for real, large deployments as cabling is already an issue for very dense setups like backend fabrics in AI clusters. But clean cabling is a much easier problem to solve than density and power.

Granted, these modules now need to be liquid cooled and that too with a fairly complex system that needs an inlet/outlet pair per module, not just per switch. But compared to cooling optics by air, Arista says temperatures across various components could be kept up to 25 deg. C lower, making performance consistent and extending the lifespan of parts. Arista states that this same design could cool up to 400W worth of generated heat from the module. This could support future communications specifications like carrying bandwidths as high as 3.2T over ranges as far as 80KM (ZR).

For power delivery, these devices now can take 50VDC from the rack bus bar and then internally step down to 3.3V, which is what SFPs use. This means to reach the same wattage at higher bandwidths and ranges, the current supplied can be a lot lower and hence the internal power architecture a lot lighter.

The XPO MSA (Multi-Source Agreement) itself hasn’t published a list of their members yet, but from Arista’s announcement we know there are 45 so far. From what we can see, its most prominent confirmed members include Arista, Marvell, Foxconn and Coherent just to name a few. It’s entirely likely that names like Broadcom and TSMC may join though both have existing stake in competing CPO technologies.

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top of everything, or even summarising all the material available can be difficult - so I do it for you.

For a space to share sources and news/updates, join the telegram channel or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

Contents:

• Roadmap Updates

• Groq LPX

• Vera CPU

• BF4-STX

• VR NVL8

• VR NVL72

• Networking

• Other Interesting Bits

Roadmap Updates

The interesting parts of Jensen’s Keynote, in no particular order:

• NVLink gen. 7 will stay at 3.6TB/s BIDI per GPU, but NVLink gen. 8 will transition to using CPO instead of copper and custom connectors, indicating it’ll likely double to 6.4TB/s BIDI (copper might be infeasible at this point)

• Feynman GPUs will use die stacking, so distinct dies will be stacked on top of each other in the same package

• Feynman era will also use Custom HBM modules (straying from JEDEC completely), likely with much higher bandwidth and adding some compute-in-memory capabilities

• Groq LP30 (LPU 3) will soon support NVFP4 in the LP35 SKU, and then with Feynman the LP40 will be released, which will use NVLink as its C2C and LPU-GPU fabric.

• The ETL 256 rack appears to be an MGX modular design that connects 256 Groq LPUs or as many Vera CPUs over non-NVLink copper spines.

• Oberon racks (Rubin) will scale from a NVL72 (one rack) up to NVL576 (8 racks) per scale up domain using a mix of copper and CPO. This will stick to the tray-based architecture using a copper spine but going forward, this spine will be much more compact in physical design.

• Kyber racks (Rubin Ultra) will scale from a NVL144 (one rack) up to NVL1152 (8 racks) per scale-up domain using a mix of copper and CPO. 6 NVLink switch ASICs per switch blade across the 6 switch blades per rack, and 36 compute blades per rack with 4 GPU packages per compute blade. These will use two physical mid-planes that will be connected by the NVLink switch blades. ~500KW per rack but no further breakdown just yet.

• CX-10 will indeed double to 3.2T finally using 400G SERDES, introducing 1.6T IB and 800GbE hopefully. This then moves the backend fabric in SuperPods to 400G per plane.

• The successor to the Vera CPU will be called Rosa, nothing more known just yet.

There are many other fun facts that I heard but they are covered in more detail in the appropriate sections below.

Groq LPU/LPX

LP30 (LPU 3)

A modification of Groq’s LPU 3 but now designed for use in NVIDIA’s LPX racks. Specs are as below:

• 500MB SRAM @ 150 TB/s bandwidth

• 1.2 PFLOPs Dense FP8 (H100 = 1.9)

• 96 C2C (Chip2Chip) links @ 112 Gbps each (~1.3 TB/s agg bandwidth)

• Made on Samsung’s 4nm process node

There’s no information I can share on an individual device’s TDP or even for the whole rack, the engineers at the exhibit were quite clear about that. More on the exact chip architecture in the diagram below.

In the next subsection I try to break down how many of these 96 C2C lanes get assigned to intra-rack vs inter-rack and how.

The LPX Rack and trays

A beautiful work of art but I noticed one thing that really bothered me: it has 5 OSFP ports at the front.

Turns out these are for both the scale-out and scale-up fabrics. The leftmost port is for 400G Ethernet, which might be a dedicated port just for connecting to VR NVL72 racks to shuffle data back and forth during inferencing at low latencies (more on this and NVIDIA’s “ETL” in the future).

The other four ports are for C2C links between racks (perhaps copper but more likely optical), two for the rack on the left and two for the right. This sort of lines up with Groq’s chain-like topology for the LPUs where each device exists as part of a straight line between its two neighbours.

There are a total of 768 112Gb C2C lanes from the 8 LPUs within a tray, and according to NVIDIA’s diagram below, 32 of these are are dedicated to the four inter-rack ports, implying each one runs at a nice 800G. The other 736 either fully divide into the 6 or 8 OSFP ports at the back of the tray (intra-rack, over a copper spine) or are split between any connections within a tray and those between trays.

It’s not clear if LPUs can talk to each other directly within the tray, as the staff on display seemed unsure about this (or just couldn’t say), but I would imagine there must be some method better than leaving the tray and entering again through another port…

Other components

Apart from all of this, each tray has an x86 CPU on it with DIMMs, a BF4 or CX-9 NIC (single ASIC it seems) for scale-out, and an FPGA to co-ordinate the LPUs and - I hope - to support LPU-LPU traffic within the tray.

Also, from some general chatter I overheard:

• It seems NVIDIA have no plans to sell LPX trays or servers as standalone products (like the Rubin GPUs in NVL8 servers). The rack will be the only form factor they come in, and that makes sense, given the volumes and fabric setups needed to run LPUs effectively.

• The decision to use Ethernet as the link between NVL72 and LPX racks is a temporary one, they are trying to move towards getting a heterogenous NVLink setup where thousands of differing devices can be in the same domain to better support the low-latency requirements of this disaggregated decode solution that they’re aiming for.

• Dynamo currently coordinates CUDA and the Groq compiler to enable models and KV-cache to be sharded between NVL72 and LPX racks, and handles the communication between them.

Vera CPU

The Vera CPU comes in one SKU only (confirmed), but is used in a variety of boards and servers for various purposes.

For AI workloads, it provides high single-threaded (one core = one thread) host performance for coordinating multiple agents and and any host memory/storage/networking processes that haven’t already been offloaded to GPUs or NICs. From what we see in NVL8 boards, it appears to out-perform it’s x86 rivals given the decrease from 2 CPUs to just one for the entire server.

We saw more on NVIDIA’s plans for selling Vera as a standalone product too, in the form of a dual-socket board that will be used primarily in the CPU-only rack they announced. At 32 trays of 8 CPUs each within one Ethernet connected domain, the liquid-cooled CPU rack packs 256 CPUs (or 22,528 cores) per rack, along with LPDDR5X memory, E1.S storage modules, and two 800G BF4 DPUs for networking.

More surprisingly, the BF4-STX rack (discussed in more detail later) comes with its own custom board that has a single Vera CPU. The BF4-STX server incorporates two such boards to serve the storage and networking functions of NVIDIA’s L-3.5 (or whatever) cache for KV products.

BF4-STX

It seems NVIDIA want to take control of the storage hardware design for their ICMS or whatever it will be called. These boxes will handle storing huge amounts of KV cache products off-GPU and will connect to the GPU racks via the Ethernet scale-out network.

For this end, they have two compute and I/O boards each with a Vera CPU, a bit of LPDDR5X memory, and two CX-9 ASICs. Oddly, the specs list above claims that the 8 OSFP ports we see at the front of these servers will run at 800G each, though there are only a total of four 800G CX-9 ASICs on board. The specs also claim that there is only one M.2 boot drive per server, though I hope there are two, one on each Vera board.

The rack view shows the eight ports on the front of each STX box, along with all the E3.S drives and the switches used in the middle. There are 16 such servers in a rack.

VR NVL8

I saw this coming, but still many surprising things I learnt when asking around.

The latest in the HGX lineup, the VR NVL8 now hits a peak theoretical TDP of 24KW, meaning that even one 2U server is too much for most existing datacentre racks around the world.

General specs

There are a variety of implementations from the various OEMs of this design, but all built around the following specs:

• 8 x 2.3KW Rubin R200 (naming confirmed) GPUs

• 2x x86 CPUs or 1x Vera CPU

• 100% Liquid cooled

• 8x CX-9 NICs over 8x 800G OSFP ports

• 1x dual-port 2x400G BF4 DPU over 2x QSFP112 ports

It does seem there are some rare versions with a single x86 CPU too, but I doubt they’ll be very popular.

We don’t know yet the host memory capacity/setup or what the storage will look like exactly. Block diagrams aren’t available yet either but we do know that the CX-9 NICs will no longer support PCIe switches or at least, will no longer act as root complexes for GPUs. This could imply storage disks will be directly connected to GPUs perhaps.

Cooling

We see above the cooling system for the GPU tray (bottom of the 2U server). there are 8 identical plates, flipped horizontally on either side of the central manifold, all connected in parallel to the main cold inlet. This means each GPU gets the same temperature of coolant (which was not the case for the GB200 designs) but the pressure does drop as you go further down the manifold.

However, I learnt that each connection point has its own valve and sensors for fine-grain monitoring and precise control, so this shouldn’t be that much of an issue. All 8 GPUs will, hopefully, be enabled by this cooling system to perform the same under heavy load.

At the Foxconn booth, we saw a stripped down version of the manifold and loops used for the GPU tray. We see here that whilst the GPU cooling might be in parallel, the CX-9 ASICs and OSFP cages are cooled serially - some components in these loops will receive warmer AND lower pressure coolant than others. This shouldn’t be a huge issue but is interesting to note.

As for the CPU and NIC tray, the cooling plates are even nicer, where two screws need to be removed to unlock a handle that lets you disconnect and pull out the whole plate (and maybe component underneath?) quickly.

For a closer look at the actual manifold/loops, we can visit the Foxconn booth again to see the CPU tray this time. The CPUs and DIMMs, as well as the SSDs are also in serial loops, but the NVSwitches both connect direct in parallel to the manifold.

An Aside

I also learnt from Pegatron that the mysterious 2U B300 x86 NVL8 that we kept harping on about back in summer 2025 was forcibly cancelled by NVIDIA - the reasons for which are still unclear to us. But interesting to hear.

VR NVL72

A few more details about the rack itself, and a lot more on the “SuperPod” designs for putting lots of these together.

The Rack

The specs:

• 72 GPUs, 36 CPUs, 18 BF4s, 144 CX-9 NICs (288 ASICs)

• 20.7TB HBM4 @ 1.6 PB/s aggregate

• 54TB LPDDR5X

We learn now that the NVLink domain will eventually be able to scale up to 576 GPUs over a mixture of copper (NVLink spine) and optics (NVLink tray-tray), and this has been prototyped already internally at NVIDIA with GB200 racks. As for how this will be implemented, no details are available yet but it’s likely a modified copper spine that also branches out into optics will be designed later.

More detail on the P/Q forms too: Max-Q means that various racks under the same management software (BCM or whatever it will be called) will be able to share power fluidly by “sloshing” to reallocate at the rack-level, and Max-P means each rack gets maximum power at all times.

DSX

As part of their marketing effort, NVIDIA have also announced a new “DSX” platform which combines a lot of different racks into one solution for some unknown purpose.

The “DSX” brings together the following racks:

• 16 x NVL72

• 2 x Vera CPU

• 2 x BF4-STX

• 8 x Spectrum-X switch

• 10 x LPX

All under one fabric. This is NVIDIA’s marketing push for convincing executives to invest their Gigawatts of power into NVIDIA GPU racks by offering them a nice packaged talking point that they can use to show off to their peers without needing to think about the nuances of infrastructure deployment or workload optimisation.

SuperPod reference architecture

To construct a “SuperPod”, 14 NVL72 racks (1008 GPUs) come together under a two-tier 800G fabric for the E/W (GPU-GPU), and a lighter fabric for the N/S.

The E/W fabric uses (for now) 128-port 800G Spectrum-X Ethernet switches to create an 8-plane network, each plane running at 200G (defined by the SERDES speed). This is done by using fibre shuffles to split the eight lanes from each CX-9 NIC (four lanes per ASIC) across eight leaf switches (one per plane) within it’s rail. There are four rails, one per GPU in each tray and hence each plane only has 4 leaf switches, but each GPU can see up to 8 leaf switches directly. These then all connect to a spine layer which connects groups of 14 racks within each plane.

It’s unclear yet how inter-plane communication happens, if it happens at all in this spec, but usually deployments come up with their own solutions by adding additional spine switches and then adding a core layer.

The N/S fabric is interesting too. Each leaf switch here connects to 4.667 racks (yes) and each SuperPod of 14 racks shares 6 leafs, indicating a lot of overlap for bandwidth and resilience. The dual port BF4 NICs allow each tray (server) to connect to two leaves.

These leaves then connect into a spine layer which brings the storage, OOB, and management leaves into the fabric as well as expanding out to other SuperPods.

Networking

We gained a lot more clarity on the upcoming Spectrum-X Ethernet switches in particular, including the CPO versions.

InfiniBand

There isn’t too much in the roadmap here since NVIDIA is pushing Spectrum-X Ethernet now as the default and primary fabric for the GPU/backend fabric.

However, we did see more of the Quantum-X 800 Q3450 here, a 144-port switch running IB-XDR without any expensive 1.6T optics. I asked what fibre cable type is used here and strangely, I was told that these switches take MPO-16s.

The Q3400-RA implemented 144 IB-XDR ports over 72 OSFP cages, which took 1.6T dual-port optics each. These would take MPO-12 cables in each of the two ports, a setup which worked perfectly well for switch-switch or switch-server links. It should be noted that people at exhibits do not have to be 100% correct all the time, so this could very well be a mistake.

Aside from all of that, we see 18 ELS modules at the bottom, each air cooled. It turns out that a single ELS failure will not take down the whole switch, instead it’ll just degrade the bandwidth and it can easily be hot-swapped. We don’t yet know what new maintenance tasks like managing CPO port failures or port fibre cleaning imply for the future of datacentre operations.

Once again, I seemed to be the only one who cared about these little labels inside the switch. As far back as GTC Paris, I posted about seeing these inside the earliest prototypes of the Q3450 and what the implications would be. This hasn’t changed at all, except that I realised and confirmed one more thing:

These fibre shuffles mean that the four Quantum-3 ASICs inside the switch don’t need to have any I/O between them - the shuffles ensure that every port links to each of the four ASICs, and hence there is no need for chip-to-chip traffic within the switch.

Spectrum-X

SN6600

First, let’s take a look at the SN6600, still using optics but matching the IB-XDR switches now in port bandwidth. 64 OSFP cages provide 2x800GbE links each via 1.6T optics, all in a 2U form factor. This 102.4T switch enables VR NVL72 SuperPod builds to use up to eight planes for its E/W network through its 512 effective 200GbE ports.

SN6810

Now moving onto CPO. The SN6810 keeps the radix at 128 ports of 800GbE but now takes fibre directly, via the up and coming MMC connectors. These ultra-dense ferrules allow the ports to take up very little space on the face of the switch, letting both the ELS modules (behind the cover on the front) and ports fit into a 2U height.

The total power draw of this switch, all ports populated, will only be 2KW. Compare that to a Q3400 with 72 1.6T optics plugged in pulling 7KW and you can see how these switches will bring about significant drop in OpEx to any deployment.

SN6800

Here’s the beast we were all excited to see, and we see it both standalone and in a switch rack. Packing 64 ELS supporting 512 ports of 800GbE each, this will drop the spine layer switch count by 4x and might even replace a whole network plane’s leaf group for the backend fabrics.

Apparently (and very likely) this switch includes shuffle boxes inside as it now has four 102.4T Spectrum ASICs within, and uses the same design logic as the shuffles in the Q3450 to link each port to each ASIC.

And this entire thing, fully populated, only draws 8KW.

CX-9

We finally see the CX-9 boards, with two ASICs at 800G each and two OSFP cages supporting them. The large slot on the bottom isn’t a new massive type of optic, but is actually for an E1.S SSD module, so you can add storage to the CX-9 for whatever interesting use case you might have.

Other Interesting Bits

• Up to 100K GPUs can fit into a single two-tier ethernet fabric in theory, and this should fit nicely into a 500MW profile according to “AI Gigafactory” experts. An InfiniBand version of this does exist, but is specific to one customer only

• NVIDIA appear not to be planning for dual-phase or immersion cooling, quite happy with DLC for now. This makes sense given I heard rumours of 15KW cold plates in the works from CoolIT…

• Aivres (OEM) are working on a 5U hybrid-cooled Rubin NVL8, designed to mix air and liquid cooling so it can fit into datacentres without significant retrofits apparently. Interested to see how this goes.

• Cerebras are apparently working quite hard on their networking between WSE nodes. So far, they’ve been very good at single-user high-throughput performance, but struggle to serve models that exceed the capacity of a single WSE and also can’t very well serve multiple users at once. It makes sense therefore that they’d be pushing harder on high-bandwidth interconnects.

• Quantum-X switching will likely not scale beyond 144 ports per switch, as it seems NVIDIA is sticking to Ethernet for the future of large-scale deployments and their AI factories/DSX whatever you want to call it.

• CPO switches require DR optics on the other end - they’re all optimised for use across the DC which makes perfect sense, and I suspected as much but just needed the confirmation.

• Wistron is developing a storage server that is driven by four RTX Pro 6000 SEs for reaching incredible IOPS from within a single node. packing 96 E3.S drives and connecting to the network over four 400G OSFP ports, this completely liquid cooled node looks very impressive.

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top of everything, or even summarising all the material available can be difficult - so we do it for you.

For a space to share sources and news/updates, join our telegram channel or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

This month’s updates

• ZAM

• HBM news

• Other notable headlines

This month’s updates

ZAM

HBM, arguably the key enabling technology behind production-scale LLMs, appears to be reaching its physical limits. Despite experts forecasting continuous improvement up to 2038, other teams have started working on a new approach that promises progress even sooner.

Z-Angle Memory, or ZAM, is a next-generation DRAM-based memory technology designed to compete with HBM in future hardware that will require large volumes of on-device, high-bandwidth memory. Basically, GPUs, TPUs, and whatever other PUs will aim for AI training and bandwidth-intensive inference workloads.

HBM achieves its high-bandwidths by supplying the logic die at the bottom of the stack with signals from all the DRAM dies above it simultaneously, keeping the very wide data bus saturated. This is implemented by using thousands of TSVs (Through-Silicon Vias), each made from a thin layer of copper, that route both power and signal traffic between the dies.

Stacking HBM higher and pushing the bandwidths further has worked great so far, with the latest generations (HBM4) reaching a maximum of 3.3 TB/s per stack, or as much as 50 sticks of DDR5. But as HBM keeps scaling hitting 16 or even 20 layers per stack, the dies in the middle become thermal dead zones where heat generated must pass through multiple low-conductivity layers, creating thermal gradients and hot spots. ZAM addresses exactly this challenge the most.

Here, multiple TSVs are replaced with a single, more central channel for both power and signal routing. A copper contact ring around the edges of each DRAM die connects to the central channel on the die below at a “Z-angle”, or diagonally through and across the dies. This creates a more centralised heat distribution and signal routing architecture. As a side-effect, having a single channel instead of many TSVs also means a significant amount of die area (previously lost to empty holes) is now available for more memory capacity.

Sources state that this new architecture could allow for up to 50% reduction in power consumption compared to HBM (assuming similar performance), and a significant increase in memory capacity per module, as much as 512GB, though no official confirmation or further details on this have been provided yet.

As for the motivation behind all this, it appears to be more than just chasing performance and reliability. This new technology is being developed as part of the NGDB (Next Generation DRAM Bonding) initiative, a collaboration between Intel (U.S. DOE’s Advanced Memory Technology program) and SoftBank (SAIMEMORY), with research from the University of Tokyo. This means a high-bandwidth memory technology (and research) independent of SK-Hynix and Samsung, both Korean tech giants.

The geopolitical effects and de-monopolisation efforts of such key technologies have been powerful driving forces in steering AI development in recent times. Which hardware designers get access to a reliable supply of HBM, at whatever cost, will determine their survival and the survival of all of their customers - hyperscalers, AI foundation labs, and everyone in between. It makes sense then that non-Korean teams will continue to push for HBM alternatives as soon as possible.

We can expect to see ZAM samples entering the market in 2028, with mass production expected in 2029, competing with HBM4E and HBM5 according to current timelines.

Memory news

Memory news comes in it’s own section this month (and maybe for the rest of time) as there’s just so much going on. Below we’ll look at developments from the major memory manufacturers, and the plans their customers are making around the memory shortages caused by such insatiable demand.

NVIDIA

NVIDIA’s roadmap is now being shaped by the availability of memory, despite SK-Hynix and Samsung’s best efforts to pander to every one of Jensen’s requests. Korean sources state that NVIDIA is considering a dual-bin strategy for its upcoming Rubin product line, optimising against the performance vs stability curve, and quite likely also for supply volume. This means that the “top bin” HBM modules which can reliably sustain 11.7Gbps/pin will be used for the higher end SKUs like the Rubin “R200” GPUs, whilst the “bottom bin” ~10Gbps modules will be used for the lower end SKUs like the Rubin SXM GPUs used in HGX servers or even any yet-to-be-announced Rubin series PCIe cards (I have it on good word there will be at-least one in mass production 4Q26).

HBF

SanDisk announced their intent to create high-bandwidth flash (HBF) just over a year ago now, but so far haven’t developed any physical samples or even lab prototypes. That might change very soon though as they’ve announced a collaboration with SK-Hynix to create standards for the technology and eventually bring it to market.

So far, SK-Hynix have demonstrated that HBF shows promise at-least in the simulation space, with their new technology dubbed “H3”, which brings together HBM and HBF modules on a single device. Pairing a (simulated) B200 GPU with 8 (simulated) each of HBM and HBF modules, they demonstrated up to 2.69x improvements in performance per watt in (simulated) AI inferencing workloads. Whilst this is far from a serious signal, it indicates that research is going well and sampling is coming soon. Even more recently, five days ago on the 25th of February, the two held a kick-off event at the SanDisk global headquarters and formally launched a joint project under the Open Compute Project (OCP).

Some sources claim that these new HBF modules could reach as high as 512 GB in capacity, and sustaining just over 1.6TB/s of bandwidth, making it perfect for a new tier of on-device memory helping further reduce the need for high-speed interconnects and distant, much slower host memory.

Un-named industry sources state that HBF could enter commercialisation (mass production) as early as late 2027 for the very early adopters, and would be deployed widely by 2028-2030, putting it alongside HBM6 in future AI accelerators.

Samsung

Samsung always finds themselves in the middle of a lot of stories, but for now we’ll discuss just two: their fast HBM4 for NVIDIA, and their push towards LPDDR6X for Qualcomm.

As we stated earlier, NVIDIA was just recently driving the development of HBM4, with demands for higher pin rates and greater supply volumes. The market reacted promptly with both SK-Hynix and Samsung pushing the pin rates from the JEDEC standardised 8 Gbps up to 10 Gbps, and then even as high as 11.7 Gbps in their efforts to compete for NVIDIAs orders. Though now with physical constraints being realised, it appears that memory availability is now driving AI accelerator release timelines and manufacturing decisions.

Through all of this, Samsung has announced that their HBM4 modules are “commercially shipping” which likely just means that they’ve entered mass production and delivery. We also now learn that these modules can be overclocked to reach past even 11.7 Gbps, all the way to 13 Gbps per pin. For a HBM4 module with 2048 pins, that’s 3.3 TB/s out of a capacity of 24 or 32 GB.

Moving onto LPDDR6X, Samsung has reportedly began shipping samples to Qualcomm for use in their upcoming AI200 and AI250 AI accelerators.

Qualcomm have been working on their own AI inferencing accelerators for close to 7 years now, with the original “Cloud AI 100” announed back in April of 2019. Since then, Qualcomm have sold a reasonable number of units (iterations upon the Cloud AI 100 that is) to various clouds such as AWS and Cirrascale to name the biggest, and have worked their way into GigaByte (of course), HPE, Lenovo, and even Dell’s datacentre servers offerings.

Now for the latest in this product line, the AI200 and AI250s, Qualcomm will likely be sourcing highly energy-efficient and fast LPDDR6X from Samsung. The capacity and pin rates aren’t yet revealed, but likely they’ll be in 2GB or 4GB volumes, and pushing anywhere from 12.8 Gbps (Samsung’s current gen. LPDDR6 performance) up to well beyond 14.4 Gbps per pin. SK-Hynix’s LPDDR6 is already overclocking at 14.4 Gbps and so naturally the higher end of the X-generation will have to have a theoretical limit higher than that.

Originally targeting smartphones or other low-power devices in need of faster memory, LPDDR is now popular for AI accelerators optimising for performance per watt such as Qualcomm’s AI200/250s, IBM’s Spyre, NVIDIA’s Grace/Vera CPUs, and more.

Micron

Micron finally enters the GDDR7 game with their 3GB modules, despite each reaching only 36GB/s in bandwidth. Compared to their Korean competition in SK-Hynix and Samsung (both promising 40GB/s+), Micron’s Graphics DDR appears to be a less attractive product at first. This is almost certainly the case for buyers placing them in AI accelerators, which run at high wattages and with active cooling methods usually. But for anyone buying GDDR for graphics accelerators like NVIDIA’s RTX or AMD’s Radeon cards, memory modules are often chosen to be of a lower specification or even down-clocked to fit into more modest power and thermal profiles.

Too bad that there are currently serious gaming GPU shortages plaguing the market right now. Even worse, NVIDIA just announced that there will be no new graphics cards released this year, making it a pretty sad year for gamers and lower-end GDDR manufacturers alike. Regardless, Micron will probably still sell their GDDR7 inventory completely for use in PCIe form-factor AI accelerators like the RTX Pro 6000/D or prefill-focused GPUs like the Rubin CPX.

Other notable headlines

• InferenceMAX v2 (now called InferenceX) is out - SemiAnalysis

• NVIDIA pushes Samsung to deliver HBM4 even faster

• China’s HBM3 production is narrowing the gap with the South Korean giants

• Nvidia’s new RTX 6000D appears in teardown: 84GB GDDR7 in China compared to the full 96GB

• World’s first PCIe 6.0 SSD enters mass production with 28GB/s speeds: Micron 9650 series SSDs support air and liquid cooling

• Cisco releases the G300 102.4T switch ASIC

• Rumours of AMD’s MI400 racks being delayed surface, though credibility remains low

• Rumours of NVIDIA’s VR200 racks being ahead of schedule...But also rumours that they’ll be delayed

• China’s GPU manufacturers race to power Chinese frontier models, domestic market now contested by NVIDIA again

• Rubin CPX might end up using HBM after all instead of GDDR7

• SambaNova introduces new AI accelerator SN50, claims to be three times more efficient than Nvidia B200

• Nvidia delivers first Vera Rubin AI GPU samples to customers

• ByteDance developing internal AI ASIC, in manufacturing talks with Samsung

TL; DR: CX-8s are PCIe 6.0 NICs, and aside from the integrated ASIC version, there are three main types of PCIe card, each well suited to a different use case.

The ConnectX-8, or CX-8 is the latest in NVIDIA’s lineup of network interface cards, components dedicated to handling all the work involved in connecting servers and networks. This “Super” NIC, as NVIDIA calls their series, comes with a vast array of features and extreme performance to support the incredible demands of networking from the largest of AI models.

When first revealed back in October 2024 at the Open Compute Project global summit, the CX-8’s industry-leading speed caused a little concern for server designers and field engineers. Doubling the networking speed to 800G (Gbps) from the older CX-7’s 400G was expected, with 800G/1.6T optical modules and copper cables ramping in volume to support the connections.

But there was one major issue when it came to actually using these in servers - PCIe 5.0 could only reach 512Gbps per direction. Peripheral Component Interconnect express, or PCIe, is the technology stack that allows components like the CX-8 NIC to then connect to the rest of the server internally, via the baseboard.

Since the NIC’s external networking interface was too fast for the internal electrical interface, it would be impossible to actually realise the doubling in performance.

Moving CPUs and baseboards to PCIe 6.0 would solve this, as a standard 16-lane connection could then hit 1024Gbps one way, enough to support the speed of an 800G NIC. However, the industry roadmap for PCIe 6.0 lagged behind the CX-8’s mass production schedule.

The original CX-8

NVIDIA had an idea - embed a PCIe 6.0 switch on the CX-8 NIC itself, and now use that as a root complex (or junction) for GPUs or even SSDs. After all, PCIe is always backwards compatible and so a PCIe 6.0 switch can connect to PCIe 5.0 CPUs.

Unlike the CX-7s, CX-8s first appeared in early prototypes of GB300 compute trays from the NVL72 racks. Here, they were integrated fully into the “universal” baseboard with supporting circuity and heatsinks all included, fixing them to the board without the ability to be removed. This would be the case for the B300 HGX 8-GPU servers too, with the GPUs and NICs both being fixed onto the baseboard.

Baseboard-integrated NICs are excellent for performance, efficiency, and standardisation, but aren’t suitable when designing custom servers that need swappable parts. In those cases, designers will use discrete “PCIe cards” that can plug in and out of slots in the baseboard.

And NVIDIA, as usual, did indeed deliver.

PCIe card specs

Finally, here’s a reasonably detailed breakdown of the four versions of PCIe card (and one supporting card) we know about so far. For more detail, you can look at NVIDIA’s CX-8 user manual.

Comparison

If you really want to skip the best part of this article, here’s a summary table to help consolidate the below information:

C8180

The primary version and closest equivalent of the integrated CX-8 is the C8180, an 800G NIC with an OSFP cage - OSFP being Octal Small Form factor Pluggable, the type of module that plugs into this port. This is then enabled by the NICs PCIe 6.0 x16 (1024Gbps) edge connection.

It supports InfiniBand XDR, or eXtended Data Rate, at 800G but does not support native 800GbE (Ethernet). Instead, it can run at an effective 800G via 2 x 400GbE connections, which will then require one of three setups:

• Using a twin-port 2 x 400GbE optical module

• Using a single-port 800GbE optic with a breakout cable splitting it into 2 x 400G links

• Configuring the switch to treat the single-port 800GbE link as 2 x 400GbE links with a distinct MAC (Media Access Control) address each.

There are arguments for and against using a single 800G vs 2 x 400G links, but in practice it’s more down to what switches and optics you have, and what your team already has experience in. Regardless:

Using a single 800G link:

• Higher effective bandwidth due to lower overheads from supporting one link vs two links

• Fewer cables to purchase, connect, and route by your team

• Fewer logical ports required to manage for the network

Using 2 x 400G links:

• Better load-balancing over two links, enabled by independent MACs and differing routes possible through cabling

• Making physical dual-plane networks, increasing the size of a single or two-layer network as well as adding port and switch level redundancy

• Using more widely available and likely cheaper switches that can’t support 800G endpoints

Note that the C8180 comes in two variants; the 900-9X81E-00EX-ST0 and 900-9X81E-00EX-DT0. The only difference here is that the D variant supports NVIDIA’s “Down-Stream Port Extension” tech, whilst the S variant does not.

The down-stream port extension allows the NIC to act as a root complex via its additional PCIe x16 interface at the rear, connecting to additional devices. NVIDIA specifies the default setup as being for four x4 connections to SSDs, which sets the system up for NVIDIA’s GPUDirect Storage tech and reduces the latency between the network and disk.

However, this could also connect the NIC directly to PCIe 5.0 GPUs such as the RTX Pro 6000 or even future Blackwell PCIe cards if they’ll exist, as hinted at by StorageReview. This would allow for a similar setup to the MGX servers hinted at in NVIDIA’s article on the CX-8, but with a GPU:NIC ratio of 1:1 instead of 2:1. This is because the integrated ASICs will have 32 lanes of PCIe available for connections other than to the baseboard, but these discrete PCIe card versions will only have 16 such lanes.

As for possible configurations, this table from NVIDIA’s user manual lists them all:

See the C8240 section below for details on what XDR Lite is and how it compares to NDR.

C8180L

There’s also an additional version to be used in the more dense servers that might liquid-cool more than just the GPUs. The C8180L will apparently be designed around cooling plates from some industry partner(s) that NVIDIA will qualify and recommend or even mandate.

This will be identical in functionality to the air-cooled version above. There likely won’t be a significant performance difference here, but for higher TDP parts like GPUs, liquid-cooling can maintain better operating temperatures more consistently than air-cooling can, leading to small performance gains.

There’s a lot more practical information available in NVIDIA’s cooling specifications for this model.

C8240

This version still provides an effective 800G but now split over two QSFP112 cages, and still is a PCIe 6.0 x16 card.

The dual ports add port-level redundancy, at the cost of incurring some thermal and power overheads.With two QSFP112 ports, you now have to support two 400G optical modules, each with more than half the power draw of a single 800G OSFP module.

For Ethernet, the configuration options are interesting. You can deploy 2 x 400GbE ports with two single-port optics or get two 4-way breakout cables and deploy 8 x 100GbE ports. There are no options for having 4 x 200GbE ports.

With InfiniBand it’s simpler; you can choose between activating one or both of the ports, and run them at 400G either with NDR (previous gen. of IB) or XDR (latest gen. IB) Lite (half-bandwidth). The NDR setup would be used when the fabric contains older Quantum-2 switches like the popular QM9700 which can’t support the newer XDR. Otherwise with Quantum-3 XDR (800G IB native) switches like the Q3400-RA, you should configure C8240 NICs to use XDR Lite.

Unlike the C8180, this one does not have a liquid cooled version.

C8220

This version is a lightweight, designed to support up to 400Gbps only, but still coming with a PCIe 6.0 x16 edge connector. NVIDIA always releases lower speed and power NICs for additional flexibility when it comes to designing servers, so the C8220’s existence is expected.

It still provides two QSFP112 ports though. To use the 2 x 200G ports, you need to use compatible QSFP112 optical modules capable of downrating and running at 200G with either 2 or 4 channels.

In the case that each QSFP112 optic outputs 200G over two optical channels, these will use MPO-8 or MPO-12 cables, and the switch-side 400G SR4 (Short-Range, 4 optical channels) optics will take two cables from two such NIC ports. If the NIC optics are configured to output 4 optical channels, then these will use MPO-12 cables to plug into 200G SR4 optics at the switch.

Both for Ethernet and IB, there are no NIC configurations that support further breakout. Either a single port can run at NDR or 400GbE, or two ports can run at NDR200 (half-bandwidth NDR) or 200GbE.

C8180X - Auxiliary card:

Because the CX-8 NICs can support up to 800G speeds, they need at least 800G worth of connectivity to the baseboard for that networking speed to actually be realised for any downstream devices like the CPUs and GPUs.

Baseboards however only support PCIe 5.0 edge connections, and so NVIDIA developed an auxiliary card that comes with 16 lanes of PCIe 5.0 and an MCIO (Mini CoolEdge Input/Output) connector. This is used for carrying PCIe over a cable and effectively doubling the bandwidth going into the baseboard from the actual NIC.

This way, a C8180 or C8240 could be fitted into a PCIe 5.0 x16 slot, connect to an auxiliary card in another PCIe 5.0 x16 slot using an MCIO cable, and then provide as much bandwidth as a single PCIe 6.0 x16 connection could.

In addition, the two links to the baseboard could go to different CPU sockets, eliminating NUMA (Non-Uniform Memory Access) effects that the CX-7 experienced. This means that one NIC now has direct connections to both CPUs on the baseboard and hence won’t incur the latency and bandwidth costs when communicating over the CPU-CPU connection, which can be quite slow.

Notably, this option appears to be available for the C8220 too, even though it lacks the capability to actually use the bandwidth of two PCIe 5.0 x16 slots. Still, it does provide the lower latency benefits when communicating with the opposite socket.

Where we’ll see them deployed

A lot of NVIDIA’s datacentre GPU sales will be in the pre-packaged and fully integrated GB300 NVL72 racks and B300 HGX servers. These will be bought primarily by:

• Hyperscalers - Microsoft, Oracle, Google, and Amazon

• Large neoclouds - Coreweave, Nebius, Crusoe etc.

• Big AI foundation model labs - OpenAI, Anthropic, Meta, xAI etc.

• Trading firms - XTX Markets, G-Research, Citadel etc.

Most of these buyers will purchase significant amounts of NVIDIA’s fixed and publicly announced product lines to provide compute to their customers and/or their staff.

However, there are two exceptions.

Firstly, most of the buyers named above will also be buying custom designs with different GPUs, different configurations, and in varying sizes. It’s very likely that hyperscalers at least will want other GPUs too like the RTX Pro 6000s or any future Blackwell PCIe cards, and they’ll want servers with different PCIe topologies, components, networking configurations, and more.

This will naturally lead to designs that are kept private between the high-volume buyer and NVIDIA or whichever OEMs get contracted to supply them. Examples of this exist already with all the major OEMs and NVIDIA maintaining tens or even hundreds of custom designs for the likes of Microsoft and Amazon.

Secondly, a lot of smaller or less consistent buyers exist, such as the small-mid sized neoclouds, public sector research labs, or firms that just want their own private compute. Due to how many such firms exist, the market for providing to them is surprisingly large.

Various OEMs and resellers create custom configurations to supply the demand from such customers which often come with many choices of CPUs, storage, host memory, and NICs.

These scenarios are exactly where the above versions of the CX-8 will shine - where buyers want more options to customise their hardware for their own unique requirements. How many NICs are in a system, how many ports there are, how they’re configured, and how they’re connected to the rest of the system are all key factors to be considered when trying to reach a particular outcome.

In short, the ConnectX-8 PCIe cards are NVIDIA’s path to market domination. They may require careful navigation of breakout cables, auxiliary cards, and cooling requirements, but they are the only way to bring Blackwell-class networking to standard enterprise hardware. The ‘SuperNIC’ isn’t just for the few anymore.

Summary

If you just want the info and don’t want to read any further:

Measured myself

• NCCL tests:

  • ~837 GB/s peak bandwidth in all-reduce tests (93% of the 900 GB/s theoretical peak)

  • ~682 GB/s for broadcast tests (76% of the 900 GB/s theoretical peak)

  • Min latency (2MB message size) appears to be 42-52 us for all tests except all-to-all, reporting 68 us

  • For some benchmarks, latency is mostly constant until 16MB message sizes, for others it begins to increase at 4MB

• NVBandwidth (SM-driven):

  • D2D unidirectional is ~782/717 GB/s (read/write)

  • D2D bidirectional is ~1.35/1.41 TB/s (read/write)

  • H2D ~38 GB/s per GPU doing a memcpy (59% of the 64 GB/s theoretical peak)

  • D2H ~53 GB/s per GPU doing a memcpy (83% of the 64 GB/s theoretical peak)

  • Latency H2D/D2H is 1.4 us

  • Latency D2D is 1.6 us

• DCGM:

  • Only 284 out of 288 GB (98.6%) are actually allocatable (Reserved, driver overheads, ECC etc.)

  • Looks like we can get ~405 TFLOPs FP32+FP16 mixed precision matmul perf per GPU. I assume FP16 A and B into FP32 C?

  • During level 3 tests, Power draw never exceeded ~950 W per GPU (86% of the 1100 W rating) as measured with NVTOP

  • Latency D2H unidirectional is 3.9-4.9 us, H2D unidirectional is 4.0-5.4 us

  • Latency bidirectional is 4.8-6.7 us

• General:

  • B300s appear to idle at ~130 W at ~17 deg.C

  • Max clock rates (MHz) for:

    • Graphics (CUDA cores?): 2032

    • SMs: 2032

    • Memory: 3996

Heard from a reliable source (but unverified)

• HPL-MxP can reach 1.5-1.7 PetaFLOPS (whole node performance) - I wasn’t able to get anywhere near this though.

• HPL and gpu_burn can make the B300s hit 1100 W sustained

B300 specs (actual)

Nvidia’s own datasheets and publications are still quite conflicting on this annoyingly. Here’s what we know from lots of sources (1 2 3 4 5 6 7 8):

• Memory: 288GB at 8 TB/s aggregate, implemented as 8 stacks of 12-Hi HBM3E at 36 GB each

• TDP: Both air and liquid-cooled SKUs are designed for 1100 W under normal operation it seems, though apparently some versions will reach 1200 W and the GPUs in the GB300 trays will hit 1400 W

• FLOPs: It’s really hard to get these numbers since NVIDIA keeps posting “sparse” values which aren’t really useful to most people. See below for numbers from the NVIDIA datasheet for the DGX, but don’t trust these numbers blindly. Some sources above give some numbers but it’s too complicated for me to figure out.

• NVLink: 5th gen. NVLink at 1.8 TB/s bidirectional per GPU, implemented as 18 x 100 GB/s NVLink ports per GPU

• Scale-out: 800 GB/s full-duplex (Tx/Rx at this rate simultaneously) per GPU via a single OSFP cage on a ConnectX-8 NIC, using a PCIe gen. 6.0 x16 link from GPU to the PCIe switch in the NIC.

For those who do want some context and more info, please read on!

Software

• NVIDIA Driver 580.95.05 - Came out in October, but it seems the first 580.X.X (B300 support?) versions were out as far back as August. 590.X.X is out too now.

• CUDA 13.0 - Didn’t manage to record the exact version but likely 13.0.2 since that also came out in October. Unsure if 12.8 supported B300s or only GB200s/B200s/RTX Pro 6000s. But 13.0 explicitly supports B300s. 13.1 is out now too.

• DCGM 4.4.1 - It says in the release notes that DCGM supports B300s since 4.4.0, but that doesn’t necessarily mean every test might be optimised. After all, only hyperscalers and really big neoclouds are deploying these right in volume right now. Very few other’s have significant capacity that they’re reselling so perhaps DCGM isn’t even used as a burn test in these places?

• NCCL-tests 2.17.6 - Last versioned near the end of October, can’t see why this wouldn’t be working well for B300 machines.

• NVBandwidth 0.8 - Released back in April, not many changes here it seems

• HPL MxP 25.09 - Supports CUDA 13.0 for Compute Capability 8 (Ampere) and above

Notable results and discussion

Idle power draw vs H100s

I’ll be comparing SXM boards only since those are the types inferencing-optimised deployments most small-mid sized cloud providers will have. 80GB H100 PCIe’s idle at 50/350W, or 14% of their max draw, which actually changes these conclusions completely, but we won’t talk about that here.

Proportionally, B300s don’t appear to idle at a significantly higher wattage than H100s. B300s idle at 11.8% of their peak TDP (1100 W) whereas H100s at 10% (of 700W). However in absolute terms that’s 130 W (at 17 deg. C idling temperature) vs 70 W (at 27 deg. C). A 1.86x difference, despite the max draw only being 1.57x.

Colder GPUs should idle lower if anything, due to lower leakage currents, so if the temperature was the same for both servers in some ideal experiment, we could expect a slightly higher idle power draw from the B300s.

The effects of firmware and drivers have a far greater impact on idle power than ambient temperatures do of course, and it’s possible that over the next quarter as more non-hyperscalers get their hands on B300 servers that this will change, but I’m not very knowledgeable on this.

Regardless, this means two things:

• If you’re at a fixed power budget and replace all your H100s with B300s (same total cluster power), then you’ll see a higher idle draw, and hence higher bills when GPUs are not being used. You’ll need to change your prices accordingly to account for this.

• If you’re at a fixed performance requirement, and you replace your H100s with fewer B300s to get the same tokens/sec or FLOPs or whatever, try to have at-least 1.86x fewer B300s to have a lower idle draw across your new cluster. The max draw is only 1.57x so it’s easier to reach lower overall power draw (and hence better tokens or FLOPs per Watt) at peak utilisation, but a little harder to reach lower idle draw.

In practice though, you can probably replace your H100s with 1/3rd as many B300s and get the same performance on ML workloads purely based on memory capacity, more FP16/FP8 FLOPs, better scale-out + scale-up etc., so you should be fine.

Latency measurement difference between DCGM and NVBandwidth

I’m no expert here by any means, but in summary: NVBandwidth measures GPU-side, DCGM measures CPU-side.

This means NVBandwidth is measuring GPU events to isolate the DMA transaction latency covering:

• Time from GPU’s memcpy kernel/DMA engine launch to PCIe/NVLink transaction complete (measured via GPU hardware event timestamps).

• Excludes all host-side software overhead; reports ~1.4 µs for host↔device PCIe transfers.

And DCGM measures the full round-trip latency experienced by an application, including:

• Host CPU scheduling jitter (waiting for CPU core to execute driver code).

• CUDA driver call overhead (~100s of ns to ~1 µs to validate, build, and submit the command).

• GPU kernel/DMA launch latency (time until GPU begins the transaction).

• Pure DMA latency (~1.4 µs, the part NVBandwidth isolates).

• GPU completion signalling (GPU sets done flag or fires interrupt).

• Host CPU polling/waiting for GPU completion (via cudaDeviceSynchronize() or blocking call overhead).

• CUDA call return overhead.

This all leads to a much higher but more realistic measure latency from DCGM, since that’s what many applications will actually experience. NVBandwidth’s lower number is more for validating that the link itself is healthy and meets specs.

Real GPU power draw under sustained load

During DCGM burn testing (L3 and L4), the highest power draw reading I saw was 950 W out of the 1100 theoretical peak. I didn’t exactly measure very diligently, and I’ve heard from a reliable source that other burn tests like HPL or gpu_burn can get the B300s to hit 1100 W draw.

The more important thing to measure though would have been the power draw during sustained, real workloads, like inferencing reasonable sized Llama/Qwen models using vLLM or something. I’m not an expert in AI engineering so I’m sure there are better production stacks, but hopefully my point is clear.

From what I’ve heard on the H100 SXMs, real inference workloads rarely push the GPUs to beyond 50-60% of their peak draw. I wish I could have run some experiments like this myself.

Using models from today though, a lot of companies are running 30B-235B models over multiple H100/H200s, sometimes using an entire HGX server to inference them efficiently with a large context length. With B300s, you can fit a 70B model on one GPU with plenty of space for overheads and a huge context length, so two things change dramatically:

• The parallelism configurations for efficiently inferencing models over multiple devices

• The removal of GPU←→GPU comms and hence the bottleneck moving back to HBM bandwidth or even compute in some cases

Both of these things massively change the absolute power required to run these models and the hence performance/watt (tokens/W or FLOPs/W), making it difficult to get a good estimate of the benefits expected by upgrading to B300s from H100s from specs alone.

H2D vs D2H Write bandwidth asymmetry

In the NVBandwidth tests, we notice that the memcpy bandwidth between the host and device differ significantly depending on which direction it goes in. Again, I’m no expert in any of this, but from what I understand, it comes down to who initiates the transfer and their ability to use the same physical PCIe 5.0 x16 link.

For D2H (~53 Gb/s), when the GPU writes to pinned (reserved capacity and fixed location) host memory from its HBM, it uses dedicated GPU copy engines (specialised DMA units) to issue PCIe read requests. This means:

• The GPU’s copy engines can issue lots of outstanding read requests to the PCIe fabric, keeping the link utilised well

• The CPU’s is mostly passive, it provides DRAM read bandwidth and maintains memory coherence. It doesn’t actively manage PCIe transaction queueing or pushing data onto the link

• GPU DMA can sustain link saturations a lot closer to theoretical peaks (~85% of PCIe Gen5 x16 peak, which is 64 GB/s unidirectional)

For H2D (~38 GB/s), when the CPU transfers data to GPU memory, it initiates the writes itself and is a lot more active throughout. This means:

• The CPU memory subsystem must handle cache coherence checks, write-combining buffer management, and coordination with DRAM controllers

• In multi-socket systems NUMA effects add latency when DRAM is remote from the CPU core initiating the writes.

• When multiple GPUs are written to concurrently, CPU memory bandwidth becomes the bottleneck before the PCIe link itself is saturated.

This doesn’t really matter too much in practice though since most traffic during inferencing (and certainly training) will be D2D, and anything involving the host is usually one of:

• Control messages

• Model initialisation

• Checkpoint downloading/uploading

• Data loading from disk/host mem to GPUs

Future work

Getting better power draw measurements

If I get my hands on a lab-hosted B300 server again, I’d like to get continous TDP measurements independent of what’s running, probbaly via nvtop or something running in the background. On top of this, measurements from the PDUs in the rack or the PSUs in the server itself if possible would also be really useful.

I think getting a really comprehensive measurement like that at a fine temporal grain like at 5 or even 0.1 second intervals would be very useful in understanding things like power ramp-up when applications start, sustained draw and fluctuations, and how this correlates to other things like server temperature or performance.

Finding out actual HBM bandwidth

Another key set of measurements I forgot to take were the actual HBM read/write bandwidths. NVIDIA reports B300’s peak HBM (read) bandwidth as 8.0 TB/s, derived from the number of stacks (modules) and each stack’s pin rate multiplied by number of pins.

Various sources (1 2 3) put the pin rate for HBM3E at between 9.2-9.6 Gb/s, and with 1024 pins per each of the 8 modules (each module being 36 GB), the total comes to an aggregate bandwidth of 9.4-9.8 TB/s, much higher than advertised. Therefore it’s likely that NVIDIA runs the modules at a much lower pin rate (probabaly for power and thermal reasons) to achieve the 8 TB/s promised.

This is the theoretical maximum under ideal conditions: all memory access patterns are perfectly coalesced, no bank conflicts, no thermal throttling, and infinite memory-level parallelism.

In practice, measured bandwidth will be substantially lower than the advertised peak and will vary significantly by workload. Some of the reasons for this are:

• Bank conflicts: DRAM modules have multiple memory banks. If many threads request data from the same bank, access effectively serialises, reducing real bandwidth

• Thermal throttling and power delivery: Under sustained load, HBM clock frequency is reduced due to temperature or power budget constraints, directly reducing bandwidth

• Insufficient memory-level parallelism (MLP): To saturate the bus and hide the ~100–200 ns latency of HBM, the GPU must be making a lot of memory requests simultaneously. I imagine some kernels just can’t issue enough concurrent loads/stores to keep the bus fully utilised.

My estimation is that we’ll probably never see more than 7 TB/s for even the most optimised workloads, which is still a very good 87.5% of the theoretical peak. Most workloads will probably see closer to 75% or 6 TB/s I would bet.

Inferencing models and tokens/$ relative to H100s

I would also like to do a lot more detailed work on inferencing various models in various ways to see how a B300 HGX compares to the current dominant inferencing server for a lot of self-hosters: the H100 HGX.

The B300s will be a lot better of course but depending on the price per hour of renting a node or even the total power draw difference (price to run it), the economics could sway one way or the other. Currently, Nebius are offering B300s at $6.10/GPU/hr and H100s at $2.95/GPU/hr, but I expect many providers will present significantly lower rates.

I suspect that even accounting for how cheap H100 servers are right now and how pricey B300s might be in the few places they are available, B300s will be better, but not for a while.

Just like we saw with the B200s, drivers and various upstream software will need some time (both from NVIDIA themselves and the open source community) to be optimised for the new hardware. Therefore it will probably be a quarter (3 months) or so until we really see mass adoption of B300s from small-mid size businesses for inferencing and training.

Disclaimer

All information here is non-proprietary, non-confidential, and reproducible in theory by anyone with access to a bare-metal B300 HGX server. How I was able to access to the B300 node and under what conditions will not be disclosed.

Ideally, this document will grow - slowly but surely - into a comprehensive and powerful reference for anyone working with optical transceivers or optical network design. And it won’t be a solo effort!

SixRackUnits

SixRackUnits (6RU) focuses on news about the technology itself of AI/HPC hardware, rather than the money surrounding it. If you want to know the detail behind the hardware news you read, we're here for you.

By Hitesh Kumar

I’m maintaining a copy of this on LinkedIn too for anyone to contribute and add to. With citations of course, as every contributor should also be available as a follow up for further questions from any reader.

This means no anonymous contributions - Everyone who adds, including me, will always be accountable for the correctness of the information shared.

Though I suspect it will just be me for a long time!

Questions + Answers we have so far:

1. How do I choose between using OSFP and QSFP-DD for 400G?

2. Can I use different ranges (like SR/DR/FR etc.) of transceivers with each other?

3. I want to breakout an 800G into 2 x 200G, can I do this?

4. Should I buy more 800G ports now, or wait for 1.6T ports?

5. What’s the difference between MPO-16 and MPO-12?

1 - How do I choose between using OSFP and QSFP-DD for 400G?

https://www.linkedin.com/posts/hitesh-kumar58_qsfp-osfps-networks-activity-7331253259682131968-lrQo

So I believe the two main reasons are:

• Backwards compatibility with QSFP56

• Thermal management

Backwards compatibility:

QSFP-DD ports can accept QSFP56 and QSFP112 transceivers too if needed - but note that this wont work the other way, for a 400G QSFP112 port to take a 400G QSFP-DD transceiver etc.

You cant put any other transceiver form factor into an OSFP port however without special adapters, but you could always use breakout cables if you needed to use an OSFP switch with QSFP NICs, with the bandwidth splitting too.

And I suppose with OSFP you’re better set for upgrades towards 1.6T and beyond, with upcoming 16-lane OSFP-XD modules which will be backwards compatible with current OSFP form factors (both flat top and finned I presume)

Thermal management:

OSFPs are physically larger and can come with flat top or finned top form factors.

This is partly due to having more space as they were designed around 8 lanes from the start, whereas QSFP-DD are designed to fit 8 lanes in a sort of 4 lane compatible form factor.

This means that with QSFP-DD, the switch itself must have heatsinks built into the ports to help keep the module cool, but with OSFP, the modules have heatsinks built into them.

Further, for OSFPs, they come in slightly shorter flat top form factors which are intended for low port density appliances like servers, as well as finned top form factors for high port density switches.

Follow up links:

• Blog by ProLabs: https://www.prolabs.com/400g-which-form-factor

• Article by QSFPTEK: https://store.qsfptek.com/blogs/article/qsfp-dd-wiki-and-comparison-of-qsfp-dd-vs-osfp-and-qsfp

• Arista Networks‘ very detailed documentation on this: https://www.arista.com/assets/data/pdf/Datasheets/Arista-400G_Optics_FAQ.pdf

• The ENTIRE specifications for QSFP-DD and OSFP if you want to use NotebookLM or something (please don’t try to read it all): http://www.qsfp-dd.com/wp-content/uploads/2017/03/QSFP-DDrev2-0-Final.pdf and https://osfpmsa.org/assets/pdf/OSFP_Module_Specification_Rev5_21.pdf

2 - Can I use different ranges (like SR/DR/FR etc.) of transceivers with each other?

https://www.linkedin.com/posts/hitesh-kumar58_networks-transceivers-optics-activity-7333414513100156928-kgwe

Maybe. It’s not something I’ve experimented with at all, and sources seem mixed.

Generally it’s best not to mix up transceivers with specified ranges.

The compatibility isn’t just some arbitrary value determined by matching names, it depends on a lot of factors:

• Wavelength

• Data rate

• Modulation

• Fibre cable type

and a few more things.

Range is determined by the above, and so it’s not that different are incompatible by design, but that other factors might make them un-interoperable.

1️⃣ VSR <-> SR:For example:

VR or VSR (Very Short Range) is designed for <100m- and SR (Short Range) is for up to <300mBut both run at 850nm, using MMF fibres.

The issue might arise with cables though, as SR transceivers could be made for MPO-12 or MPO-16 cables. But I have never seen a VR using MPO-16.

2️⃣ DR <-> FR:

Similarly, FR and LR might work since both use O-band (1260nm-1360nm) and use LC duplex cables. But then you have to make sure the modulation is the same as:

• Lower bandwidth connections will use NRZ

• A higher bandwidth might use PAM4

It seems even this information isn’t universally agreed upon, as sources seem to vary a little between the exact distances that these categories cover.

The values above were taken from Asterfusion Data Technologies‘ blog post here: https://cloudswit.ch/blogs/choose-400g-optical-modules-for-data-center/

Some sources say that actually you can mix some types with the shorter type being the one that determines the resultant maximum range of the connection. A very detailed guide from Arista Networks here: https://www.arista.com/assets/data/pdf/Datasheets/Arista-400G_Optics_FAQ.pdf

For example, it may be possible to interoperate a FR with an LR, limited to 2km (by the FR). The catch is that LR modules transmit at a higher minimum power than maximum power that FR can receive, and so a minimum amount of attenuation (or signal power loss) may be needed to make this work.

I think the best thing to do is still to consult with the vendor and get them to help you with testing if this is really an issue to you. Though I can’t imagine that this would be a common occurrence.

Follow up links:

• A post from Cisco Patrick Chou on using 400G DRs with 100G FRs: https://blogs.cisco.com/sp/heres-an-easier-way-to-upgrade-to-400g-optics

• A great blog post by Asterfusion Data Technologies on various ranges of 400G optics: https://cloudswit.ch/blogs/choose-400g-optical-modules-for-data-center/

• FAQ document from Arista Networks on 400G transceivers: https://www.arista.com/assets/data/pdf/Datasheets/Arista-400G_Optics_FAQ.pdf

• A really comprehensive interoperability matrix tool from Cisco - not easy to use but might be useful for non-cisco products too:

https://tmgmatrix.cisco.com

3 - I want to breakout an 800G into 2 x 200G, can I do this?

https://www.linkedin.com/posts/hitesh-kumar58_networks-transceivers-optics-activity-7335628842998403072-2og0

It depends on if the hardware can be split up properly to accommodate.

Generally, you’d get the right transceiver for the right port and divide the bandwidth from the source equally, but sometimes that’s just not possible.

For example, you might be in a (not so uncommon) situation like:

• Having existing infrastructure using 800G OSFP switches

• Needing an additional small setup with a few smaller endpoints

• Additional endpoints having 200G or 400G QSFP NICs

In a case like this, you might be unsure about which QSFP type you’d use and how you’d breakout from an 800G OSFP.

But it is possible, and also fairly simple: Stop thinking about transceivers and start thinking about lanes!

The 800G OSFP has 8 x 100G (O= Octal) lanes, using PAM4 modulation. In contrast, a 200G QSFP56 would be running 4 lanes (Q = Quad) at 50G using NRZ modulation. Splitting an 8 lane OSFP using a 4 way breakout would mean each endpoint gets exactly 2 lanes of 100G.

You need a transceiver that can handle 2 x 100G lanes!

So the answer is that you’d use 400G transceivers anyway, since QSFP112 400G transceivers have 4 x 100G lanes.

Both the source and destination will be using the same lane rate and modulation. The destination 400G would be running with only 2 of their 4 100G lanes. Half of the lanes will be inactive and the transceiver will use less power.

In summary:

• Use a dual-port OSFP 800G at the switch

• Use 2 breakout cables that split into 2 ends each

• Use 400G QSFP112 transceivers at the endpoints

I don’t know if its possible for transceivers to automatically shift down to a lower lane rate with a different modulation using their gearboxes. It sounds like it can be done, but I cant find any sources on this - perhaps another post!

If needed, it’s even possible to split down to 8 x 100G ports! But going any further becomes difficult/impossible. After all an OSFP transceiver will only have 8 channels to distribute in breakouts. Any more needs another switch.

UPDATE:

Wait - 2 x 100G transceivers exist!

Previously, I said that 400G transceivers (and 400G NICs) are used for breaking out 800G ports into 200G ports - This is still true.

But now I learn that you can also use 200G transceivers and NICs!

I thought that 200G QSFP transceivers would (since “Q” = Quad) use 4 lanes at 50G for the optical interface. This means that they are incompatible with 800G OSFP or 400G QSFP transceivers that run at 100G per lane.

But it looks like the “Q” doesn’t necessarily define both the optical AND electrical interface! There exist QSFP56’s that present 4 x 50G electrical lanes but 2 x 100G optical lanes.

These are designed for 200G QSFP NICs it seems, compatible with the electrical interface of the port but lets you split from an 800G (8 x 100G) source. I became aware of this just recently with FIBERSTAMPs post on this: https://lnkd.in/ehRZ6_WE (thanks to Anders Zhang/Chang).

I would imagine there are other vendors too but I’m finding it difficult to find sources - not entirely clear. This QSFP-DD (8 x 25G NRZ) from FLEXOPTIX GmbH appears to present 2 x 100G optically but it seems to be a dual 100G port instead of a single 200G port: https://lnkd.in/eN9vkpRSStill

Still looking for other sources.

Comments:

Anders Zhang/Chang:

“...the answer lies in the DSP being used inside the module. Take our 400G QSFP-DD PLR4 transceiver for example, utilising PAM4 technology, this module can convert 8-channel 53.125 Gbit/s electrical data to 4 parallel channels of optical signals, each supporting106.25 Gbit/s data transmission. Reversely, it can convert 4-channel 106.25 Gbit/s optical signals to 8-channel electrical output data on the receiver side. The diagram below illustrates how it works.

Linus Chima Iwuji:

“...the step that takes 8 × 50 Gb/s electrical lanes (400 GAUI-8) and outputs 4 × 100 Gb/s optical lanes inside a 400 GbE pluggable is the lane aggregation in the module’s DSP/gearbox.

(GAUI = Gigabit Attachment Unit Interface - the short-reach, board-level electrical link between the host ASIC and the module. 400 GAUI-8 means eight 53 GBd PAM4 lanes. I found what looks like a concise primer in the “Standards” section of this blog post: https://blog.fluxlight.com/2022/12/13/deep-dive-400gbase-lr4-qsfp-dd-optical-transceiver/)

In a duplex part (e.g., QSFP-DD FR4) that DSP/gearbox output then feeds a CWDM mux/demux so the four 100 Gb/s optical carriers share one fiber pair.

I found two example datasheets that appears to spell this out:

1.) T1-Nexus QSFP-DD-400G-DR4: “converts 8 channels of 50 Gb/s (PAM4) electrical input data to 4 channels of parallel optical signals, each 100 Gb/s.”: https://t1nexus.com/product/t1-qsfp-dd-400g-dr4/

2.) EdgeOptic 400G-QSFP-DD-2.1 (FR4): “converts 8 channels of 50 Gb/s (PAM4) electrical input data to 4 channels of CWDM optical signals and multiplexes them into a single channel for 400 Gb/s optical transmission.”: https://edgeoptic.com/products/400g-qsfp-dd/400g-qsfp-dd-2-1/“

Follow up links:

• NADDOD‘s page on 800G tranceivers with a great diagram of this: https://www.naddod.com/products/101489.html

• product page and a very useful post by FiberMall on this topic: https://www.fibermall.com/sale-460315-nvidia-mma4z00-ns.htm#con_one5_fake and https://www.fibermall.com/blog/400g-ndr-splitter-cable-and-800g-osfp.htm

• Document by Arista Networks on 800G transceivers: https://www.arista.com/assets/data/pdf/Arista-800G-Transceivers-and-Cables-FAQ.pdf

• Manual from NVIDIA on ethernet interfaces and breakouts: https://docs.nvidia.com/networking/display/onyxv3104206/ethernet+interfaces

• Two posts Cisco on this topic too with great explanations: https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/whitepaper-c11-744077.html and https://blogs.cisco.com/sp/unclear-on-fiber-optic-breakouts-what-you-need-to-know

• great post from Weixin (WeChat) by Mingge Network (can’t find a link to author): https://mp.weixin.qq.com/s?chksm=e8bf0352dfc88a4450fd8855def5ff12092956844fca1a6b48eb34567a2f15a3d584a4977f9d&exptype=unsubscribed_card_recommend_article_u2i_mainprocess_coarse_sort_tlfeeds&ranksessionid=1748370524_3&mid=2247485468&sn=cc4acedda0454654d5d3148cad35ba3e&idx=1&__biz=MzIyOTcyNzc3Nw%3D%3D

4 - Should I buy more 800G ports now, or wait for 1.6T ports?

https://www.linkedin.com/posts/hitesh-kumar-6ru_networks-transceivers-optics-activity-7348671014811185152-mBSi

A lot of major changes are coming in the next ~12 months for the AI/HPC cluster business - Hyperscalers, Neoclouds, and anyone else who wants their own hardware.

One of these is the introduction of 1.6T/port switches, ready for 800G endpoints, multi-plane topologies, and scaling high-bandwidth fabrics to 10^5 devices and up.

You’ll want to move to 1.6T modules eventually, but do you need to prepare for them for your next purchase?

There are currently two ways of reaching this port bandwidth:

• OSFP1600: 8 x 200G (Also called OSFP224)

• OSFP-XD (eXtra Dense): 16 x 100G

OSFP1600

OSFP1600 scales 800G OSFP modules by upgrading the DSP so it can handle higher data rates per lane.

Upcoming switch ASICs too (Like Broadcom’s Tomahawk 6 famously) will support 200G lanes, allowing for similar breakout patterns and topologies datacentres are used to, but with double the bandwidth.

The issue is that the rest of the transceiver is the same.

I’m not sure if OSFP modules will be able to scale to 3.2T (8 x 400G), given we already need finned-top versions for 800G (8 x 100G) to go into switches just to keep them cool.

OSFP-XD

But I am sure that it will be easier to hit 3.2T with a 16 lane module, using well-established 100G/lane technologies.

The only issue is that to accomodate 16 lanes, the module has to be a little larger - space is needed for better power distribution and simplifying the routing of electrical interfaces.

This means OSFP-XD modules are no longer compatible with OSFP ports - 1.6T switches can’t take existing 800G or 400G OSFP modules.

But just like the current generation, 800G (Maybe even 400G?) OSFP-XD modules might be available in the future for specific use cases.

In addition, breaking out a 1.6T port into 2 x 800, 4 x 400, or even 8 x 200 will still be possible, but I will make another post on this when NADDOD or someone else releases them.As for thermals, OSFP-XDs use lower per-lane rates for 1.6T and so might not need to come in finned-form factors too for switches.

In summary

So the decision to prepare for 1.6T ports depends on how early you want to adopt new technology, and how badly you need more bandwidth in a certain volume. In a few years, 1.6T and 3.2T modules will be commonplace in datacentres used for AI training and perhaps even inferencing.

This means you’ll need to understand costs, MTBF, power usage and more. And future GPUs/NICs will reach 800G and then 1.6T, how long can 800G/port switches keep you to your SLAs?

Comments:

Daniel Tidselbak:

Great write-up Hitesh! Really solid breakdown 👏

We’re seeing many AI-focused operators lean into 800G now, not just for bandwidth, but because the products are finally mature, and available.1.6T will be huge, but there’s still work to be done on ecosystem readiness. Until then, stability and supply-chain control often win over chasing specs.

Follow up links:

• The OSFP-XD MSA (multi source agreement): https://osfpmsa.org/assets/pdf/OSFP-XD_Specification_Rev1.1.pdf

• The intro to above, also from OSFP-MSA, a much easier read: https://osfpmsa.org/assets/pdf/OSFP1600_and_OSFP-XD.pdf

• FiberMall‘s page on OSFP-XD connectors: https://www.fibermall.com/blog/osfp-xd-connector.htm

• 650 Group, LLC‘s article on how the market for 1.6T fabrics will increase over the next few years: https://650group.com/blog/1-6-terabit-ethernet-will-drive-next-generation-ai-ml-wave-and/

• ServeTheHome‘s article on 1.6T/port switches spotted at computex 2025: https://www.servethehome.com/broadcom-tomahawk-6-102-4t-64-port-1-6tbe-switches-at-computex-2025-wiwynn-wistron-delta/

5 - What’s the difference between MPO-16 and MPO-12?

https://www.linkedin.com/posts/hitesh-kumar-6ru_networking-datacenter-fiberoptics-activity-7394713356026404864-0zLT

A common cause of delays in data centre builds is ordering the wrong cables. While MPO-12 has been the standard for years, 4-lane and 8-lane optics have different - and incompatible - physical requirements.

Here’s a breakdown:

Physical Construction

An MPO cable is a “ribbon” cable, where multiple fibres are laid flat and terminated in a single connector called a ferrule.

In MPO-12 cables, there are 12 fibres in a single row. The plastic connector housing has an alignment “key” centred on the top (see images below). On an MPO-16 cable, to prevent you from plugging this into a 12-fibre port (or vice-versa), the alignment key is offset to one side.

Channel Mapping

Transceivers are designed for a specific number of “lanes,” and each lane needs two fibres: one for Transmit (Tx) and one for Receive (Rx).

In 4-Lane optics (e.g., 100G-SR4, 40G-SR4), 4 fibres are needed for transmission and and 4 for receiving (4x Tx, 4x Rx), requiring 8 fibres (1-4 and 9-12) in total. The 4 fibres in the middle (5 - 8) are unused and “dark”.

In 8-Lane optics (e.g., 400G-SR8) however, 8 lanes (8x Tx, 8x Rx), requiring all 16 fibres in an MPO-16 cable.

How Splitter/Breakout Cables Work

Splitter cables adapt these high-density ports to connect to multiple lower-speed devices.

In the case of splitting from 400G 4-lane to 4x 100G single-lanes, a 400G-DR4 port (MPO-12) can be split using an MPO-12 to 4x Duplex LC (2 fibres each) breakout cable. For a 400G 8-lane to 2x 200G 4-lanes, a 400G-SR8 port (MPO-16) can be split using an MPO-16 to 2x MPO-12 (or maybe MPO-8?) breakout cable, which plug into 2 x 200G-SR4 transceivers.

The Case for Simpler Designs

Breakout cables are powerful but add complexity.

• Inventory: You now have to manage inventory for MPO-MPO trunk cables, MPO-16 to 2xMPO-8 harnesses, MPO-12 to 4xLC fan-outs, etc. This is more complex than just stocking standard patch cords.

• Lead Times: These multi-leg, complex harness cables are often custom-built and can have longer lead times.

• Cost: The harnesses themselves can be more expensive upfront.

A simpler (though less dense) design using point-to-point cables can be easier to manage, source, and deploy.

In summary

Always verify with your transceiver datasheet before ordering cables. The 400G-SR8 standard explicitly requires 2x MPO-12 connectors for 200G-SR4 breakout—not MPO-8. Mismatched connectors and fibre counts are among the most common ordering mistakes in datacenter builds.

Image sources:

• FS.com

• LINK-PP INT’L TECHNOLOGY CO.,LIMITED .

Comments:

Daniel Tidselbak:

Great post, Hitesh! One thing worth adding as speeds move to 800G and 1.6T:

Many modules marketed as SR8/DR8 are actually 2×SR4 or 2×DR4 designs and use dual MPO-12, not MPO-16.

Functionally fine, but important for planning — you’re really working with two separate 4-lane engines, which affects cabling, polarity and breakout expectations.

A small naming detail that can prevent big deployment surprises.

Follow up links:

• FS.com Blog about breakout cables: https://www.fs.com/blog/what-is-the-breakout-connectivity-of-transceivers-and-cables-and-how-to-use-them-27754.html

• A nice little page from HPE Aruba Networking on compatibility for optics when splitting: https://arubanetworking.hpe.com/techdocs/Switches/xcvrs/xcvr_guide/Content/Chp_overview/comp-split-opt-tran.htm

• LINK-PP INT’L TECHNOLOGY CO.,LIMITED .‘s article on MPO cables: https://resources.l-p.com/knowledge-center/mpo-connectors-differences-8-12-16-24-fiber-comparison

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top of everything, or even summarising all the material available can be difficult - so we do it for you.

For a space to share sources and news/updates, join our telegram channel here or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

This month’s updates:

• Broadcom’s relentless pace - CPO TH6, and an 800G NIC

• Nvidia news: OCP25 through to GTC-DC

• Intel announces a prefill GPU - No roadmap yet, but some hope at last

• Other notable headlines

This month’s updates

Broadcom’s relentless pace - CPO TH6, and an 800G NIC

Broadcom has been outpacing its competitors consistently enough now to be widely recognised as the clear leader in (non-nvidia) Ethernet datacentre networking hardware. At OCP25 this month they sustained that momentum by announcing two new major products: a Co-Packaged Optics (CPO) version of the Tomahawk 6 (TH6) named “Davisson”, and a new 800G NIC called the “Thor Ultra”.

The original TH6 was the first major release on our radar, revealed as far back as June. Since then, Broadcom have shown an incredible pace of development, adding the scale-up focused Tomahawk Ultra 51.2T and the network-layer defining Jericho4 51.2T “scale-across” ASICs to their lineup.

The TH6 Davisson now pushes the series forward in energy efficiency, claiming a 3.5x decrease in energy use compared to an equivalent switch (with the original TH6, we should assume) using pluggable optics. The lack of pluggable optics (optical transceivers) avoids the need for many discrete optical modules for converting signals between electrical and optical domains. CPO technology allows for these conversions from incoming signals to happen near the ASIC package and be consolidated to fewer digital signal processors (DSPs), improving signal integrity and reducing power use.

Joining Nvidia’s ConnectX-8 and AMD’s Vulcano, The Thor Ultra is the third prominent member of the 800G NIC club. Broadcom will likely ship ultra-ethernet consortium (UEC) compliant hardware long before AMD, with the Thor Ultra already implementing various technologies in hardware such as packet-level multipathing, OOO delivery to device memory, and selective re-transmission to name a few. Unlike with the BF4 mentioned later in this issue, this NIC will be used for both frontend and backend networks, likely seeing serious adoption in non-Nvidia deployments.

The NIC will support a PCIe 6 x16 host connection, come with a choice of either 100G or 200G serdes for port-breakout flexibility, and come in PCIe or OCP 3.0 form factors with OSFP ports. More details are already available on their product pages it seems, though nothing stands out other than the above. Sampling is already underway, and shipping will likely commence sometime in 1H26.

Nvidia news: OCP25 through to GTC-DC

Across the two major conferences this month, Nvidia has delivered three major announcements to the market:

• A new 800VDC power architecture for the datacentres of the near future

• A long awaited BlueField 4 SuperNIC, and an updated roadmap for the series

• The reveal of the Vera-Rubin superchip and the cable-less VR200 compute tray

800VDC

Arguably one of the most impactful developments in the industry this month (or even year), Nvidia contributed their 800VDC power architecture for datacentres to OCP, making it open to use by all. For datacentre providers and hyperscalers, adopting these higher voltage designs wont just improve the power efficiency of their sites, but will enable them to host the increasingly dense GPU racks Nvidia is planning for the coming years.

Datacentre power delivery architectures determine how power reaches the hardware from the grid. They define how the 13.8kW AC (North America) from the supply is transformed to the 12VDC required by the servers. 800VDC is a significant upgrade from the existing 48/54VDC deployed in many newer Hyperscaler and high end sites, and a complete overhaul compared to 480VAC architectures in older datacentres.

Hosting H100/H200 servers in 480VAC sites already results in a lot of wastage from all the steps and components required, resulting in as little as 75% of the wattage supplied being available for the servers themselves. 54VDC lets newer builds skip some steps and enables deploying the most demanding racks like the GB200 NVL72, but as soon as 2026 when Vera-Rubin racks ship, will still result in poor efficiencies and additional costs for heavy copper busbars and large numbers of PSUs to support the power draw.

For more information on the 800VDC architecture, you can read our article on XPU.pub, where we dive deeper into power architectures and 800VDC.

BF4 DPU/SuperNIC

Finally, a full year after it was supposed to be released, the BlueField 4 SuperNIC (and DPU?) has been announced for 2026, in sync with the Vera-Rubin product timeline. Ethernet for the front-end and storage networks has been relatively slow in scaling from 400G to 800G, and much of it is just a lack of necessity. Even converged networks handling internet, in-band management, and storage traffic rarely are bottlenecked by the 200G/400G fabrics they’re run over.

Dual-port 800G frontend NICs would offer some benefit over existing 400G if used today, though it’s not difficult to assume that high-performance storage or other communications offloaded to the frontend network will eventually require more bandwidth. Currently, such cards could provide 4x200G or even 8x100G uplinks with splitter cables, expanding the possibilities of the topologies possible in these deployments.

Anyway, here’s what we know so far about the BF4:

Official:

• 800G NIC expressed as two 400G ports

• 64-core Grace CPU on board

• A ConnectX-9 ASIC powering the network interface

• Supporting SpectrumX (Ethernet)

Assumed (1):

• PCIe gen 6 capable (256GBps BIDI)

• Supporting down to 100G per port configurations

• QSFP112 form factor for the ports

VR Superchip and tray

A stunningly beautiful board was showcased at GTC-DC, along with the compute tray that it will sit in. Jensen emphasised the cable-less design now further optimised for lower error rates and better thermal management for the inter-chip communications, though those keeping up with the news would have seen this coming from SemiAnalysis’ (and our own) post over a month and half ago.

Analysts all over the world have been speculating on what they can infer from the video feed we have so far, and some interesting details have come to light. As we at SixRackUnits noticed almost instantly, the VR superchip will finally have SOCAMM modules on board for its LPDDR (we assume), a welcome surprise given how their involvement in GB300 boards was delayed earlier this year. WeChat user Brendan006 on Brendan’s Packaging Hut also took a close look at the Vera CPU die and noticed that it’s likely 6 separate dies on an interposer as opposed to a monolithic design.

Intel announces a prefill GPU - No roadmap yet, but some hope at last

As we detailed in our september issue, prefill is the first half of LLM inferencing, and arguably the easier part since it’s compute-bound - memory bandwidth being less important. Nvidia took the lead in designing a GPU specifically for prefill aiming at improving performance/$ of CapEx and TCO, and now it seems Intel rather than AMD are the second to announce an ASIC dedicated to this task.

At OCP25 earlier this month, Intel surprised everyone by formally announcing a new prefill-focused new GPU, after scrapping their Gaudi and Falcon shores roadmap’s and the many months of silence since. Now, this new series named “Crescent Island”, scheduled for release in 2H26, is expected to compete with Nvidia’s Rubin CPX and whatever AMD and other vendors will be releasing at the time.

The GPU is reported to be air-cooled or at least have an air-cooled variant, and we can safely assume it will come in PCIe form factor, implying that it likely wont exceed a 1000W TDP. We also know that it will provide 160GB of LPDDR5X memory though no details are available yet on the setup of that capacity and the bandwidth we can expect. The choice of using LPDDR5X over GDDR6 or 7 has raised some questions, but when optimising for performance/power and even memory latency, LPDDR5X is the better choice.

We do know about its architecture though, Xe3P hinted at during Intel’s tech tour in Arizona prior to OCP and then confirmed in the presentation at the conference. Xe3P continues the lineage of the Xe GPU silicon architectures, serving the Arc-C series of discrete GPUs and focusing on performance per watt according to Intel.

Other notable headlines

• Micron starts sampling its 192GB SOCAMM2 LPDDR5X modules, targeting future small-profile AI servers

• Both AMD and Intel to be significant customers of TSMC’s upcoming “N2” 2nm process

• Samsung rumoured to supply HBM4 for AMD’s MI450X GPUs... *...And HBM3E for GB300s

• Intel showcases “Robinson lake” board architecture for edge/robotics, Pather lake but for rugged deployments

• Oracle to deploy a 50,000+ AMD MI450X GPU cluster starting 3Q26

• Meta and Oracle to use SpectrumX Ethernet for their AI clusters

• Intel to launch a Gaudi 3 + B200 mixed device rack-scale product optimised for LLM inferencing

• Synopsys developing LPDDR6 IP on TSMCs N2P process node, hitting up to 86GB/s per module

• Alibaba cloud claims to reduce its Nvidia GPU use by up to 82% with its new Aegaeon software

• Microsoft to use Intel’s 18A process for its next generation of MAIA accelerators

• IBM Spyre AI accelerators now shipping

• Samsung to produce NVLink fusion hardware, not just TSMC

• Intel foundry reportedly trying to secure AMD as a customer for its 18A and 14A process nodes - for CPUs?

• Samsung targeting 3.25TB/s with its HBM4E plans

• Intel announced annual AI hardware release cadence after years of delay and a collapse of its AI roadmap

Will those of us born today be able to live without AI? Given how much of humanity now exists in a state of dependence on technology in some way or the other, and the subset of that population that is dependent on their smartphones, it’s natural to think that a much smaller but rapidly growing subset is now dependent on AI-powered tools. Is that the ultimate plan for those who develop our AI models? To create a world in which AI is absolutely necessary, making their existence also, absolutely necessary?

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top of everything, or even summarising all the material available can be difficult - so we do it for you.

For a space to share sources and news/updates, join our telegram channel here or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

This month’s updates:

• Rubin CPX: The GPU no-one saw coming

• Huawei’s ascending roadmap - in a market free from Nvidia

• Nvidia trades with Intel: $5B in exchange for custom x86 CPUs

• HBM4 is evolving before it’s even here, spurred by Nvidia

• Other notable headlines

This month’s updates

Rubin CPX: The GPU no-one saw coming

Source: Nvidia

At the AI Infra summit earlier this month Nvidia took the world by surprise by revealing a new GPU outside of their existing roadmap, the Rubin CPX. Unlike the more “general purpose” GPUs (AI accelerators) in their recent mainline such as the H100/200 and B200/300, the CPX is designed specifically for one purpose: prefill.

When inferencing an LLM, there are two distinct stages to run, differentiated by their memory and compute requirements: prefill, decode. Prefill is the stage where the context window - all of the instructions, AI model memory, and additional pre-token generation information - is loaded onto the GPU, and decode is the stage where the new tokens are then generated sequentially. This means that the prefill stage is compute-bound, as all the tokens from the context window are loaded once and can all be processed in parallel, needing much more compute than memory bandwidth.

The decode stage is the opposite, where each new token generated must use all or some of the previous tokens and must load the previously computed products from memory. Since computations are much faster than data movement, the decode stage is memory-bound, needing a high memory bandwidth rather than compute to maintain a high throughput.

The Rubin CPX doesn’t optimise for prefill in speed but in cost, both in its price and its energy efficiency. To reduce the cost to the buyer, Nvidia replaced the fast but expensive HBM (high-bandwidth memory) with the much cheaper but slower GDDR7 (graphics DDR gen. 7). Fitting in 128GB of memory, the CPX has a bandwidth of ~2 TB/s from a 512-bit bus at 32 Gbps, which is sufficient for prefill workloads. Using GDDR also means that the power draw drops, with SemiAnalysis’ estimates putting it at ~880W, a little higher than the 800W of the H100 NVL variant.

Source: Nvidia

As well as the simpler memory, the CPX also uses a simpler design, moving from a complicated dual-die chiplet-based design like the blackwell series did to a simpler monolithic single-die design. In addition, Nvidia also decided to stick to the PCIe gen. 6. for the D2D (device-to-device) interconnect instead of the far more expensive NVLink, as a prefill/inferencing focused GPU wouldn’t need the higher bandwidth and complexity of NVLink. Both of these decisions make the CPX easier to manufacture and hence cheaper yet.

Despite the seemingly light-weight design, there are two significant upgrades: increased video encode/decode capacity and triple the number (or throughput) of transformer engines. NVENC/NVDEC engines handle processing functions for video, an increasingly popular and particularly demanding modality for AI models. The transformer engines handle mixed precision attention operations and accelerate the computationally heavy prefill stage significantly.

All in all, the CPX should output 30 PFLOPs of FP4 compute, which now seems to come at a 3:2 sparsity ratio instead of the original 2:1, something Nvidia will definitely find some way to justify eventually. Regardless, at FP16 and FP8 - which almost everyone actually for real inference workloads - the CPX might deliver 7.5 and 15 PFLOPs respectively, double what the B200 PCIe variant is rated for.

Source: SemiAnalysis

Keeping in line with their focus on rack-scale systems, Nvidia also announced two new rack SKUs, the VR200 NVL144 CPX and the VR CPX. The first builds upon the already announced NVL144 “Oberon” rack design, similar to the existing GB200/300 NVL72 designs, but now adds 8 Rubin CPX GPUs per compute tray, totalling to an 144 additional GPUs in the system. This brings the total estimated power of such a deployment up from ~225kW to ~370kW for a single rack - a number difficult for most datacentres to even imagine.

The second, CPX-only rack is a more compact and sensible option, sticking to 144 CPX’s per rack but with no R200 GPUs or NVLink spine. This still comes in at ~190kW per rack, but is still more manageable for hyperscalers and research labs that have planned for 200-220kW racks over the comings months. This can act as a sidecar even for the now decode focused HBM-heavy VR200 systems, connected over the CX-9 + 1.6T Ethernet/InfiniBand scale-out fabrics.

There’s a lot more to say on this topic around the compute tray architecture and cooling systems too, but for more on this, SemiAnalysis is currently the best source.

Huawei’s ascend roadmap - in a market free from Nvidia

Huawei are capitalising on a market free from Nvidia, having revealed their roadmap for their AI accelerators at their annual Huawei Connect conference earlier this month in China. Now, without a doubt, they’ve taken second place in the world and have become Nvidia’s biggest competitor in Asia, if not the world.

We’ve seen increasing levels of hype around their current gen. accelerator (910C) over the past year, from when it began sampling with customers to when it was put into the CloudMatrix 384 rack-scale system, challenging Nvidia’s NVL72 systems in performance. More recently too, we’ve heard that DeepSeek’s V3.1 and possibly even their R2 models were delayed due to pressure from the Chinese government themselves forcing DeepSeek to use Huawei’s AI accelerators for training.

As we know now, this was a foreshadowing of what was to come, with the Chinese government banning the purchase of Nvidia GPUs across all government agencies and large tech firms as of the 18th of this month. This shock to the market will almost certainly increase the pace of industry’s adoption of Huawei’s and other Chinese hardware vendors’ accelerators.

Source: Huawei

Staying on topic for now though, up next in the roadmap are the 950 series, split into two chips by role: the 950PR for prefill and the 950DT for decode. Both appear to use the same compute die, but the 950PR enters mass production in 1Q26, aiming for optimising costs on pre-fill workloads (see our article above on the Rubin CPX).The PR will be using 128 GB of Huawei’s proprietary “HiBL 1.0” HBM technology keeping the bandwidth low at 1.6TB/s.

For the DT in 4Q26, Huawei will use 144 GB of the second generation of their proprietary “HiZQ 2.0” HBM technology, more than doubling the bandwidth to 4TB/s. Both will be capable of 2 TB/s scale-up through the Unified Bus (UB) fabric, but only the DT will feature in their upcoming datacentre-scale “SuperPoD” system: the Atlas 950.

Source: Huawei

Spanning 160 racks, over a thousand square meters of datacentre floor, and packing 8,192 Ascend 950DT cards into a single scale-up fabric over optics, the Atlas 950 dwarfs Nvidia’s scale-up domain plans in the Rubin Ultra NVL144. It’s estimated to provide 8 exaFLOPs of FP8 over 16 PB/s of aggregated interconnect bandwidth, and will likely not be much more energy efficient (or inefficient rather) than the CloudMatrix 384. But we have (still conflicting) evidence now that this isn’t a as much of a problem in Chinese datacentres as it is in the west.

Source: Omdia. The Atlas Super cluster, combining multiple Atlas 950 SuperPoDs into a single, enormous cluster.

After that, in 4Q27 the Atlas 960 will push 30 exaFLOPs of FP8 over 34 PB/s of aggregated interconnect bandwidth. To link 15,488 Ascend 960 cards, Huawei plan to upgrade their UB interconnect technology by the end of 2026 to it’s second generation, improving performance but also supporting such an incredible number of endpoints in one scale-up domain.

The Ascend 960 raises the memory capacity to 288 GB per device, though no information is given yet on the memory technology used. Huawei plan for an aggregate bandwidth of 9.6TB/s, still a generation behind Nvidia in this regard. Each card should deliver 2 PFLOPs of FP8, though it should be noted that Ascend “NPUs” (Neural Processing Units) are not designed to be 1:1 competitive with Nvidia GPUs, but aim to compete at the system level (Atlas SuperPoDs vs NVL144/576).

Source: Huawei, Bernstein

Finally, by the end of 2028, we expect to see the Ascend 970, hitting 4 PFLOPs of FP8 with 14.4 TB/s memory bandwidth over its 288 GB per card. Even fewer details are given here, and no information on what the Atlas 970 will look like. What we do know for sure is that each passing day now accelerates the decay of Nvidia’s hopes of ever recovering their position in the Chinese market

Nvidia trades with Intel: $5B in exchange for custom x86 CPUs

Source: Nvidia

To some, this is old news. It’s been confirmed now that for over a year, Nvidia and Intel have had three separate teams collaborating in secret on joint designs for processors, AI PCs, and datacentre servers. However it all came to light earlier this month when both parties officially announced the long-term partnership with Nvidia committing to investing $5B in Intel through a stock purchase.

Investors expressed confidence in both parties through the stocks of two sides seeing significant increases within the day, Intel’s being far sharper at an almost 30% increase. For GPU servers on the market now, Intel CPUs have seen a decline in market share of the past few years with AMD making a steady rise, though with their upcoming Granite rapids processors this might change.

Source: Intel

For a while now, Both Intel and Nvidia have been pushing the higher end Granite rapids 6776P 64-core (128 threads) CPU as the CPU for Nvidia’s official 8-GPU DGX servers. Intel CPUs have been extensively studied and optimised in the more common, non-DGX 4 and 8 GPU servers too, which are aimed at hyperscalers and neoclouds. Many of these compute providers have optimised their software and platforms taking Intel and Nvidia as the default.

The largest victims of this partnership therefore are AMD and ARM - both suffering from negative investor sentiment towards their future in AI servers. At least until the end of the Rubin-era, Nvidia has committed to using “Vera” ARM-based CPUs, the successor to their popular “Grace” CPUs pushed with their NVL rack-scale systems. Given that this collaboration with Intel will bear fruit around 2027/28, it’s possible that Nvidia will offer x86 alternatives to the ARM CPUs in their NVL cabinets, or maybe just default to Intel CPUs entirely.

Source: Nvidia

AMD, whose CPUs have already suffered with minor performance issues in Nvidia GPU servers according to SemiAnalysis’ testing, may have a difficult time competing. Currently, the previously mentioned vendors/integrators offer AMD versions of the popular 8-GPU HGX systems, though with Intel CPUs becoming highly optimised and tightly integrated with Nvidia GPUs in the coming years, this may prove to become a difficult sell.

This tight integration might result in a number of useful features for future DGX/NVL servers:

• NVLink integration: Putting 1.8/3.6 TB/s BIDI links for CPU-CPU or CPU-GPU, without needing to recompile anything for ARM ISAs

• power-sloshing: Dynamic frequency scaling not just between cores on a CPU, but also between the GPU and CPU to balance power and performance where needed

• Variety for host memory: Making a variety of memory technologies available for the CPU from LPDDR (low power DDR) for energy efficiency to MRDIMMs (multi-rank DIMMs) for capacity

• Using Intel’s AMX (Advanced Matrix Extensions): Allowing x86 Intel CPUs to make more use of AMX with optimised I/O and shared memory spaces between devices on the board

HBM4 is evolving before it’s even here, spurred by Nvidia

The development and progression of HBM is now controlled not by the big three manufacturers (Samsung, SK Hynix, Micron) but by Nvidia, who has shown that they can just make demands as they like and the suppliers will bend to their will. Originally targeting 8Gbps per pin according to JEDEC standards, Nvidia asked their HBM suppliers to raise the speeds to 9, and then eventually 11Gbps - because they could.

Source: Jukan (@jukanlosreve on X)

Initially Samsung met the request head on by promising to hit 10Gbps over each of the 2048 pins, resulting a per-module bandwidth of just over 2.5TB/s. Supporting up to 12 layers in its current implementations, each consisting of 24Gb, this means each 36GB of memory can independently be moved back and forth at that theoretical peak bandwidth. All three manufacturers intend to hit 32Gb capacity per layer and 16 layers in a stack at the same bandwidth (for now), meaning that by the end of 2026, there should be at least three independent implementations of 64GB HBM4 modules.

SK-Hynix being the now dominant HBM manufacturer also claimed that it would reach the target, though more recently may have exceeded that rate too. Micron meanwhile waited a little and then made headlines by allegedly reaching over 11Gbps, a remarkable number currently under scrutiny as they intend to use the same process node as they use for DRAM for producing their logic dies in house.

Source: Micron

The logic die controls how the memory is moved to and from the pins connecting to the rest of the chip, as well as many other functions such as ensuring signal integrity and managing error correction. Producing these logic die with a process node optimised for DRAM would naturally result in suboptimal performance, or at least that’s what analysts fear after hearing Micron’s claims.

To summarise the specs of upcoming implementations (1 2 3):

Other notable headlines

• Chinese tech giants Alibaba and Baidu have custom silicon in use - competitive with Nvidia H20s

• Trading firm XTX Markets open sources their exabyte scale, multi-region distributed filesystem, TernFS

• Tesla might make its next custom silicon using Intel’s 18A process

• NVidia considering TSMC’s A16 for its Feynman architecture GPUs for 2028

• Chinese government bans tech firms from buying Nvidia, forcing the use of domestic hardware

• Broadcom to develop OpenAI’s custom ASIC - Titan XPU

• NVidia cancels SOCAMM, moving on to SOCAMM2 for Rubin - soldering LPDDRXX for now

• Alibaba releases a demo of their Panjiu-series fully liquid-cooled 128 GPU SuperNode at their annual ASPARA conference

• Nvidia promises it will buy double Samsung’s current total GDDR7 production - likely to supply their RTX and CPX series GPUs

• Samsung to release PCIe gen. 6 SSDs with 512 TB capacity in 2027

• Kioxia and Nvidia team up for “AI SSDs”, hoping for 100x current SSD read speeds

• Intel’s Granite rapids workstation CPU specs leaked

• It’s possible that the MI450X caused VR200 delays due to last minute TDP increases to stay competitive, according to SemiAnalysis

Is a material form a requirement for true intelligence? Does being limited to certain modalities of input result in less capability within those modalities than an intelligence that can handle more modalities of input? If so, will achieving human-level intelligence require AI models to have humanoid robotic bodies?

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top of everything, or even summarising all the material available can be difficult - so we do it for you.

For a space to share sources and news/updates, join our telegram channel here or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

This month's updates:

• HotChips 25! - everything of note

• Yet another switch from Broadcom - scaling across with Jericho4

• B300 GPU finally detailed

• PCIe 8.0 announced

• Other notable headlines

This month's updates

HotChips 25! - everything of note

HotChips is the annual industry event for high-performance silicon, with both the big and small names showcasing their advancements in the field. This year's event seemed dominated by relatively few players, Nvidia, Meta, and Google stealing much of the spotlight. Between all the announcements and reveals made at the event, there were just too many updates to cover here comprehensively. Instead, we'll present some of the highlights that we think are the most important and interesting.

Nvidia: Spectrum-X switches and a scale-across fabric

Spectrum-X was originally Nvidia's reaction to Ethernet gaining traction after the market showed strong signs of InfiniBand lock-in fatigue. Now, rivalling InfiniBand in its adoption among Nvidia's largest customers, Spectrum-X is innovating in two key areas: massive 512-port 400T switches, and expanding beyond the confines of a datacentre.

Source: Nvidia

On the first item, a lot was already revealed back in GTC25 in March, see our article from back then detailing almost exactly the same information. The only new content is some more detail on the observed (we assume) performance (according to Nvidia) of the 102T switch, and pictures of the switch that uses optical transceivers rather than the CPO version that was shown back then.

Source: Nvidia

The other major reveal was the Spectrum-XGS add-on to Nvidia's ethernet fabric technology, a "scale-across" stack designed to connect multiple scale-out domains (datacentres) together. More on the meaning of scale-across in our article later in this post. Spectrum-XGS claims to expand telemetry, congestion control, and load-balancing all indefinitely with respect to the number of devices, and should be a lot better at reducing cross-datacentre latency where connections can be an order of magnitude or two longer than those within a datacentre.

Google: Ironwood TPUv7 and TPU racks

We shared everything that was available on the TPUv7 back in April when it was announced, and a lot of this remain unchanged. We made a mistake in thinking it would be six stacks of 12hi HBM3E, and we now see it'll be eight stacks instead. Memory capacity and bandwidth remain unchanged thankfully. The compute capacity is interesting though, since at 4.5 PFLOPs of FP8, the TPUv7 brings the same amount of inference performance as the B200/300 and the MI350X. It seems the TDP hasn't yet been confirmed but anything below 1kW would be surprising.

Source: Google

But Google's session was still very revealing, as they showed much more on how the Ironwood scales to the server and rack level. We saw earlier that they were aiming for 4 chips per board, but now its been confirmed in their slides. They're sticking to PCIe 5.0 for the H2D, which is unsurprising given that TPUs rely entirely on their mesh scale-up for D2D. For this purpose, they'll have an incredible 18 OSFP (800G) ports per board, so many ports actually, that they couldn't even fit all 18 on one side of the board and instead had to put 2 underneath on the other side.

It was also really interesting to hear them talk about the per-chip variable flow rates for the liquid cooling step. Having a constant and equal flow rate was just assumed to be the sensible idea, but different chips on the same board could be doing very different things in theory, and hence will need different coolant flow rates.

Source: Google

We knew they'd be aiming for clusters (or "pods") of up to 9216 devices in a single ICI (Inter-Chip-Interconnect) domain and predicted two topologies: a 96x96 2D torus, or a 32x32x9 3D torus. But now we see exactly how they'll implement this - a 64 x 16 x 9 3D torus. This makes sense, as their chosen rack design still fits 64 of these huge TPUs in 16 trays of 4 chips (one board) perfectly well. This lets them keep the pod of 64 connected by copper and lets them use their OCS (Optical Circuit Switching) technology to extend ICI to 144 racks in the superpod.

Source: Google

Source: Google

AMD: Scale-up fabrics and the MI350X

Source: AMD

AMD detailed the their two upcoming Instinct series GPUs, the MI350X and the MI355X. As we detailed back in June, the two are almost identical in their memory and compute system designs. The only real difference of relevance to customers will be the TDP: the 350 at 1kW and the 355 at 1.4kW. The first is aimed at 8-GPU air cooled servers where datacentres prefer to stick to ~10kW per 6-8U of rack space, and the second is aimed at 4 or 8-GPU liquid cooled servers, where the power can reach 15 or even 18kW per 4-6U of space. For more info on the rack-scale setups and topologies, see Semianalysis' excellent article.

Oracle has already announced an order of over 130K MI355X GPUs for their upcoming zettascale AMD cluster, and other Hyperscalers and the AMD-centric neoclouds will no doubt be placing smaller orders. We could very well be seeing AMD taking a double digit market share from Nvidia soon if their software can improve fast enough to avoid the issues that prevented the rise of MI250X and MI300X GPUs.

To compare to the Blackwell series (1 2 3 4):

All FLOPs are given without sparsity - no idea why these vendors insist on giving 2:1 sparsity figures.

Memory capacity and bandwidth will now be on par, and compute numbers seem slightly better for FP16.

Source: AMD

Their more scientific session was on the design of scale-up fabrics, where they discussed the effects that scale-up switch radix and device lane counts have on the size of the scale-up domain and its bandwidth. Of course, anyone can deduce that more links is better but AMD stressed that scale-up domains should be kept as a single-tier network. They proposed a 1.5 layer network that mitigates some of the issues described in the slides below too, which was quite interesting. More details in Zartbot's article on day 0 of HotChips 25.

Source: AMD

Meta: Catalina pod (NVL36x2)

One of the major issues with NVL72 systems when they were first announced was how few datacentres could at the time support such high power racks and how few could ever even hope to adapt their infrastructure to support them in the future. The NVL72 rack-scale form factor remains a non-starter for the majority of datacentres around the world, as most places support between 12 or 20kW per rack. This includes Meta, who have a lot of datacentre space designed for CPU servers and low-power MTIA accelerators.

But for training their foundation models, Meta would prefer the raw bandwidth of the 72-GPU NVLink scale-up domain instead of the 8-GPU domain bounded by a chassis for HGX type servers. This, as well as other reasons, led them to customise the system to the extent that it became six racks.

The "Catalina" pod - a six-rack system designed to support 120kW of TDP over 6 x 20kW racks - brings 2 racks of 18 custom GB200 trays together with 18 NVLink switch trays to form a single 72-GPU domain. Four of the six racks are dedicated just to the cooling systems: air-assisted liquid cooling, using the large volume available of four standard width racks to keep liquid coolant temperatures down using fans.

Source: Meta

To this extent, they had to customise the trays to have 2 GPUs per tray instead of the usual 4 in a GB200 compute tray, making the ratio of CPU:GPU = 1:1. This then doubles the number of CX-7 NICs and Grace CPUs per GPU, leading to some very interesting specs:

• 72 B200 GPUs and 72 Grace CPUs

• 34.6TB aggregate LPDDR5X memory

• 48TB aggregate cache-coherent memory (CPU-GPU unified memory)

• 14.4TB/s aggregate Scale-out bandwidth

Source: Meta

One issue we believe Meta works around is the lack of power headroom here, with the total system being restricted to 120kW with little or no push beyond that. Combined with the fact that the number of CPUs, NICs, NVLink switches, and many other components have actually increased in number due to this setup, this will likely lead to throttling of the GPUs clock rates to stay within the 120kW envelope.

Intel: Clearwater Forest CPU

Earlier this year we reported on leaks regarding Intel's upcoming Diamond Rapids CPU, their "P-core" only processor aimed at high performance AI/HPC clusters. The counterpart to the "rapids" series of CPU is the "forest" series, aimed at more scalable and less intensive workloads like virtualisation, hosting, databases and so on. For their upcoming Clearwater Forest CPU, Intel decided to go with an official reveal.

The below image summarises the specs for a dual-socket setup, so halve the quantities (except the UPI) for a single-socket setup. Though knowing the target market of these CPUs, a single-socket setup may be unlikely, as virtualisation and web/app hosting servers are supposed to be as core-dense and energy efficient as possible.

Source: Intel

For more information, Zartbot's article on WeChat and Andreas Schilling's article on HardwareLUXX are very detailed.

Others

There were also great sessions from names like d-Matrix, Rebellions, Marvell, Huawei, and more, but we wont cover them in this newsletter.

B300 GPU finally detailed

Also unveiled at HotChips 25 but given its own segment here, the B300 "Blackwell Ultra" GPU makes headlines even though most of the information given here was already known for months. We have a few details now though worth noting:

Source: Nvidia

• 288GB HBM3E is implemented as 8 stacks, probably at 12hi, totalling 8TB/s

• 20,480 CUDA cores, and 160 streaming multiprocessors (SMs), which total to 640 5th gen. Tensor cores

• Over 160MB of tensor memory (TMEM), the same volume of L1 data cache (shared mem), and L2 cache size unknown

Source: Nvidia

For more practical-minded people, the B300 sticks to the current generations 128GB/s unidirectional PCIe 6.0 x16 H2D connection, even though PCIe 6.0 CPUs or SSDs don't exist yet. PCIe 6.0 NICs, however, do exist, such as the ConnectX-8 SuperNIC with an integrated PCIe 6.0 switch. This marks a fundamental change in the server architecture for HGX's going forward, as GPUs now connect to the NICs at a 2:1 ratio instead of to discrete PCIe switches. This makes their connection to the scale-out network 1 hop shorter and also gives the GPUs a real chance to saturate the 800G uplinks. As for TDP, the peak of 1400W remains the same.

Source: Nvidia

Earlier this year, Nvidia developed the NVFP4 floating point format for efficient, stable, and accurate FP4 inference. The B300 delivers up to 1.5x the performance of the B200 for this format, but this is likely theoretical as actual NVFP4 quantised model inferencing numbers (tokens or samples per second) haven't been given yet. The majority of inferencing provided by foundation model labs as AI models or wrapped text or image generation services in other products may still happen on H100s and H200s though for a while. B300 customers investing in this hardware may do so for inferencing the largest models or for future proofing for the next year or two, or, most likely, training. This means that for many, NVFP4 performance numbers are irrelevant.

Yet another switch from Broadcom - scaling across with Jericho4

Scale-up and scale-out are both rapidly developing dimensions for device-device communications, though now we're beginning to see the limits of how far they can actually scale. Limits enforced not by the technology itself, but from cooling, power, and space constraints.

Racks, and even rows of racks can only get so big before the distance is too great for a larger scale out domain. Signal integrity and power become too difficult to manage beyond a few meters and taking scale-up to the optical domain is incredibly expensive. Though in certain markets like China, Huawei is showing that this might not be a roadblock to adoption.

Beyond that, scale-out can cover the span of a whole datacentre with a relatively high-bandwidth fabric, but after a certain scale, switch and optics costs become difficult to justify. Clusters like xAI's Colossus are pushing that boundary by doubling the size of the domain to 200,000 GPUs whilst keeping everything under a single scale-out ethernet fabric.

Source: Broadcom

Enter Broadcom, with the 51.2T Jericho4 "Across DC scale-out" ASIC. Aimed at long-range connections between scale-out fabrics (scale-out used synonymously with datacentre here), the Jericho4 includes many features that optimise for transport at these distances whilst maintaining a secure and lossless fabric.

Source: Broadcom

The main standout innovation that Broadcom advertise is the Hyperport - 4 x 800G links combined at the hardware-level into a single logical 3.2T port. This then reduces the associated overheads of managing 4 distinct ports too. This functionality has already existed for a while at the software layer with link aggregation (LAG) but Hyperport implements it lower in the stack, optimising the load balancing, congestion control, utilisation, and more.

Source: Broadcom

In addition, the Jericho4:

• Has deep buffers to absorb bursty traffic from AI workloads

• Supports lossless connections for over 100km distances

• Provides MACsec at line rate, necessary for long-distance traffic

And just to future proof it for use in 2027/28, the Jericho4 is also UEC compliant - interoperable with all other UEC switches and NICs from any vendor.

This then "completes" Broadcom's lineup of high-performance AI cluster switch ASICs, with this chip joining the Tomahawk 6 and Tomahawk Ultra to cover all three dimensions of network scaling: up, out, and across. For more information on the Tomahawk series, see our June and July editions.

Source: Broadcom

PS: It seems they'll do anything to avoid just saying "scale-across", perhaps to avoid association with Nvidia's Spectrum-XGS announcement (more on that just above)? Or maybe they were just a few days short to coining the term?

PCIe 8.0 announced

Intel and AMD's next generation CPUs - supporting PCIe (Peripheral Component Interconnect express) 6.0 - are expected to ship between 4Q25 and 4Q26, likely being followed closely by the corresponding SSDs, NICs, and more. PCIe 6.0 itself however was announced as early as 1Q22, resulting in a wait of over 3.5 to 4.5 years from announcement to implementation. Based on the pace so far demonstrated, future PCIe generations should take just as long to reach the market from their reveal. But there's been one fundamental addition to the world driving hardware progression to speeds not seen before: LLMs.

Keeping to their commitment to doubling bandwidth every 3 years, the PCI-SIG (Special Interest Group) has announced PCIe 8.0, aiming for up to 1TB/s bi-directional bandwidth over a full x16 connection. Version 1.0 of the specification is expected to be released in 2028, with the first devices hopefully shipping before 2032. In comparison, PCIe 7.0 was announced in 2022, aims to have it's version 1.0 ratified at the end of 2025, and may see adoption in devices as soon as 4Q26. Certainly not CPUs, but possibly in NICs (already reaching 800G and soon 1.6T), and just maybe in SSDs (if storage arrays and controllers reach those speeds in time).

Source: PCI-SIG

To summarise what these speeds mean, PCIe 8.0 is targeting 256GT/s (Giga Transfers per second) bidirectional (both transmitting and receiving simultaneously) per lane. A lane is four wires, two each for transmitting (Tx) and receiving (Rx) and each pair is a differential pair. A transfer is 2 bits, since PCIe 8.0 will use PAM4 (pulse amplitude modulation with 4 levels) signalling. This means 512Gb/s per lane, or 64GB/s per lane, and therefore a x16 (full width) connection can support 16 x 64GB/s = ~1TB/s bi-directional bandwidth.

Other notable headlines

• Jensen Huang confirms that GTC Washington DC will reveal the Rubin GPUs as well as five other chips

• SK-Hynix to mass produce 2Tb 3D QLC NAND devices, on the path towards 244TB SSDs

• Rumours of Nvidia's post-Rubin "Feynman" GPUs needing immersion cooling abound from industry sources

• Samsung's HBM4 samples reportedly pass Nvidia's validation! A turnaround for Samsung in the HBM market finally?

• Huawei allegedly working on "AI SSDs" to reduce pressure on the Chinese HBM supply chain

• AMD might reveal "Venice" 6th gen. CPUs and the MI500X at their upcoming financial analyst day in November

• UltraRAM, tech that could store data for up to 1000 years, at DRAM speeds, is progressing towards manufacturing

• Nvidia announces the RTX Pro 4000 SFF and RTX Pro 2000 GPUs - Desktop grade, aimed at HPC and visualisation workloads

• Delays in DeepSeek's R2 model allegedly due to pressure from Chinese government, being forced to use Huawei's "unstable" AI accelerators instead of Nvidia GPUs

• Kioxia confirms that they are also in the race to HBF with their prototype: 5TB capacity @ 64GB/s per "module"...

• ...But SanDisk keeps up the pace by teaming up with SK-Hynix to work on standardising HBF first

• Meta working with Broadcom to get "Santa barbara" codenamed MTIA servers into production by end of 2025

• Leaks of a possible Jaguar shores thermal test vehicle reveal a possible quad-die arrangement for Intel's last chance at the GPU market

• Japan's FugakuNEXT Supercomputer will use upcoming Fujitsu Monaka CPUs with Nvidia GPUs, adopting NVFusion tech

How would we even measure the compute speed of something like this? Even for modern digital processors, measuring actual INTOPS or FLOPS is difficult and sometimes meaningless.

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. With a field changing this fast, staying on top and even summarising all the material available can be difficult - so we do it for you.

For a space to share sources and news/updates, join our telegram channel here or if you like short form posts on similar topics, check out the notes section of this newsletter or my LinkedIn.

This month's updates:

• Broadcom's inevitable yet surprising new chip - Tomahawk Ultra

• GB300 servers possibly shipping before 4Q25, "no major issues" allegedly

• The surprising mystery of Diamond rapids' leaked TDP

• H20 GPU orders pile in - Nvidia (and AMD) gains access to China again

• Other notable headlines

This month's updates:

Broadcom's inevitable yet surprising new chip - Tomahawk Ultra

Source: Broadcom

Announced just weeks after the 102.4T Tomahawk 6, the new Tomahawk Ultra ASIC is symbolic of Broadcom's scale-up Ethernet (SUE) overtaking UALink as the scale-up technology most capable of challenging Nvidia's NVLink. This new 51.2T (512 x 100G) switch expands the performance and capability boundaries of Broadcom's existing datacentre switch lineup, with its focus on lower latencies and higher packet processing rates as well as supporting SUE's requirements such as link-layer retry and credit-based flow control.

Source: Broadcom (via WeChat)

The key number that marketing have decided to lead with is the switching latency - 250 ns. It's an impressive number considering that for Tomahawk 5, this latency may have been double that. It should be noted however that this is purely the latency incurred from when a unit of data enters the ASIC to when it exits, and not the more wholistic and useful latency of getting a message from one device to another. That number still relies on other factors such as the size and priority of the message being sent, the design of the rest of the switch chassis, and any intermediate hardware such as NICs, PCIe switches, and cables (particularly for long connections). This means that in or near the worst cases for latency (usually the bottlenecks for real workloads), even on the highest performing clusters, this reduction in switching latency might not make a significant difference.

Source: Broadcom (via WeChat)

The more surprising improvement (and probably more impactful too) though is their packet processing rate. Claiming to process 77 Billion packets per second (pps) for packets as small as large as 64B, Broadcom are targeting traditional HPC infrastructures too despite the majority of their marketing centering on AI. Applications such as finite difference/element models, linear solvers, or even distributed financial/trading programs are all far more sensitive to latency than AI workloads are. Packet sizes in networking for AI workloads are usually far larger, over a thousand bytes sometimes. This has been the case for AI training for a while now, and recently has become the norm for AI inferencing too. This all means that AI workload networking latency is usually very high anyway, with the focus being on overall throughput instead of packet rates. As a result, HPC clusters using dense and complex topologies running non-AI workloads will benefit much more from the high pps provided by the Tomahawk Ultra.

These dramatic performance improvements mean that the architecture of the chip has had to change a lot. The addition of more packet-processing pipelines to meet the pps requirements and compute logic to support the in-network collectives (INC) that SUE supports has resulted in the buffers needing to be smaller and shallower than they were in the Tomahawk 5. As a result of all of these changes, it's technically not really a "Tomahawk" chip anymore, despite its compatibility with existing sockets/boards. This doesn't - and shouldn't - matter to most though, and Broadcom's decision to label and market this chip as being in the same family as their scale-out ASICs makes complete sense.

GB300 servers possibly shipping before 4Q25, "no major issues" allegedly

Source: ServeTheHome

Mass production is now underway for GB300 servers with initial shipments expected in September for some of the largest customers. Volumes will peak in 4Q25 and continue into 2026 with Foxconn being the primary supplier, and Wiwynn, Inventec, and Wistron being the others - All of which are Taiwanese. These aren't the first Blackwell Ultra systems in deployment though, as CoreWeave and Dell famously deployed the worlds first GB300 NVL72 at the start of this month, likely the first sample version publicly announced.

Source: Alethia Capital

This timely ramp up comes as news to some because back in Q2 it was reported that shipments would be delayed and analysts were speculating on how this would impact sales of Nvidia's next generation VR200 "Vera Rubin" servers. The issue causing the delays has since been mitigated rather than being truly resolved - parts were replaced rather than undergoing an immediate and effective design change. The "Cordelia" baseboard that Nvidia planned to introduce for the GB300's denser and higher-performance design was struggling to pass validation and reports from April suggested that signal loss stemming from the doubling in bandwidth across the the NVLink traces was severe enough to cost Nvidia two quarters of delay.

Ultimately, it was decided that reverting to the old baseboard design for the GB200s (Bianca) was the best option to meet deadlines. This meant that Nvidia would provide the B300 GPUs in a SXM "puck" modular form factor, and soldered Grace CPUs and LPDDR5X memory to the server manufacturers. The knock on effects mean that the proposed SOCAMM memory module design used to make the memory easily replaceable for repair, now must be introduced later with Vera CPUs. The upside to all of this was letting the supply chain can breathe for once, using existing parts and experience from the GB200 manufacturing process rather than rapidly adopting the latest specifications from Nvidia in a bid to hit the September shipping deadlines.

Source: SixRackUnits - Above: Bianca board with LPDDR5X memory soldered to the board next to the Grace CPU. Below: The proposed design for the Cordelia board using quick replaceable SOCAMM modules in the same location.

There hasn't been any update on what, if anything, Nvidia and their supply chain have done to address the liquid cooling issues that have been rumoured in the GB200s in deployment currently. It seems that the quick connect fittings used to speed up the deployment and maintenance of the servers have been the primary cause of leaks but no noticeable changes have been made to address this yet. Though compared to the previous generation of HGX/DGX servers which were pushing the limits of what air cooling could handle, the current and future liquid-cooled versions will still be more reliable and performant, for many reasons that we will discuss in later posts.

Source: Asus

The surprising mystery of Diamond rapids' leaked TDP

The first leak was posted early this month on X from the account @x86deadandback, and drew mixed reactions from the hardware community. It provided some specs of Intel's upcoming server-grade CPU series, Diamond Rapids (DMR) (successor to the current gen. Granite rapids (GNR) series), which is expected to be formally announced in the next few months. The source themselves isn't a well established industry leaker, but the numbers given for the CPU are believable and match expectations.

Source: @x86deadandback on X

For context, the highest performing Intel GNR SKU is advertised at 500W for 128 cores running at 2GHz base clock, and rumours suggest AMD's nex gen. "Venice" 192 (virtual) core CPU will reach 600W. An earlier leak from the same source showed what appeared to be a product page or documentation specifying heat-sinks for DMR CPUs, peaking at 660W. Heat-sinks typically are rated for significantly higher output than the heat source is capable of to manage temporary performance boosts or cooling system failures, and so this specification supports the rumoured 500W TDP.

Source: @x86deadandback on X

To demonstrate why a 500W TDP for this many cores would be so surprising, lets walk through an example server upgrade process. Even without the leak, its certain that a four-socket version of DMR will be available for CPU-dense datacentres running in-memory databases, high-performance scientific workloads, or financial transaction processing. For a site using 4096 cores spread over 64 older 64-core 350W Emeralds rapids CPUs, their power usage would be 22.4kW spread over 16 quad-socket servers in two 12kW racks. To keep the minimum core count after an upgrade to DMR CPUs, that site could now buy just 22 new 192-core 500W CPUs in 6 quad-socket servers, using a total of 11kW only. This could now fit into one of their 12kW racks, with capacity left for a switch or two, and the site overall has now reduced its total power usage by 11.4kW.

The example is oversimplified of course but the concept remains true: newer CPUs are generally more power efficient, and a lower TDP for the same core count can result in power, space, and cost savings if the short-term tradeoffs are acceptable. This decision is a lot easier to make when the TDP of a 192-core CPU is 500W rather than 600W. Combining this with PCIe 6.0 and CXL 3.0 support, variants with 16 memory channels, and upgraded AMX (Advanced Matrix Extensions) support 1, Intel CPUs may soon halt AMDs aggressive takeover of the server CPU market.

There's a lot more information currently circulating around the DMR series and the future of Intel CPUs in general, but they are beyond the scope of this article. For more information, see the following (perhaps more reliable) sources:

• Discussions around Intel's statement on returning to SMT, and if DMR will also not have it

• Another leak alleging that the DMR architecture will have shared L2 caches per core pairs

H20 GPU orders pile in - Nvidia (and AMD) gains access to China again

Wednesday 16th July marked Jensen Huang's third visit to China in a very short span of time. This was the first formal announcement that hardware export restrictions have been eased, allowing Nvidia's H20s to be sold in China again. H20s are the second generation of watered down H100 chips, designed to meet the enhanced restrictions imposed on compute capability and bandwidth. Interestingly, by reducing the compute capability, lowering the TDP, and keeping the same number of stacks of HBM as the H100 did, the H20s could actually provide higher device memory bandwidth (via higher memory clock rates) than the H100s could.

Source: Nvidia

When the H20s were originally banned for sale, Nvidia had to write off its existing inventory of ~600-700K H20 GPUs as a loss of ~$10.5B. Now with the April export controls lifted, Nvidia is expecting to recover that loss and potentially make even more from the increase in demand since then. To this end, Nvidia has already placed an order of over 300K H20 GPUs from TSMC, though its expected that there wont be enough supply of HBM to meet this demand and instead Nvidia will likely use GDDR6 instead. Even with the reduction in performance that comes with that, it's almost certain that demand will not decrease.

Source: VideoCardz

In addition, they've introduced multiple variants of existing GPUs from the western market as well as more SKUs specifically for the Chinese market. For a larger coverage of the cost vs performance space, Nvidia are persisting with the RTX Pro 6000D (increasingly called the B30 now) announced earlier this year, as well as inferencing only cards such as the L20 and L2 which may end up seeing similar success to the L40s for lighter inferencing and graphics workloads. For the retail consumer market, the 5090Dv2 desktop GPU will address high-end gaming and graphics workloads, but may end up being seen in lower-tier datacentres just as the infamous RTX 4090 did (and still is). AMD has also had their MI308 approved, and are also persisting with more economic Radeon AI PRO R9700 they announced earlier this year.

Source: AMD

This combination of sudden hardware diversity and availability has been met with mixed reactions from industry. It's great for Chinese big tech firms, cloud service providers, and world-leading AI labs who have all been increasingly desperate for infrastructure. Compared to their American and European competition, Chinese AI services providers like Deepseek have had to provide far lower performance in terms of tokens per second. And this is in spite of their necessity-driven research into more efficient inferencing software and model architectures.

The incredible demand has resulted in all sorts of workarounds from building infrastructure via western neoclouds to smuggling hardware via various APAC countries, predominantly Singapore. One solution is also buying and using domestic GPUs, but this change of hardware availability across the performance vs cost space might now hurt some of these domestic manufacturers who were technically less capable but might otherwise have taken advantage of the lack of supply.

What the locals can do though is focus on systems-level solutions, a space that Nvidia and AMD are and will always be forbidden to enter. The prime example that made headlines and is most widely regarded as a viable and China-appropriate GPU rack system is Huawei's CloudMatrix384, a multi-rack cluster out-performing (on paper) Nvidia's GB200 NVL72s. As we saw at the WAIC25 conference, other Chinese vendors are following suit with preparing and showcasing seemingly well-designed systems.

Other notable headlines

• Samsung may supply HBM4 samples to AMD, Nvidia, and others by end of this month...

• ...But mass production may end up starting in 2026 after all

• NVidia brining CUDA to RISC-V machines - but maybe RISC CPUs only, not GPUs

• Huawei to display the CloudMatrix384 in person, live, at the end of this month at WAIC25

• Against all odds, smugglers find a way to get GB200 racks into China - on sale now

• HBM4 is coming

• Tesla to use TSMC for its Dojo 3 chip

• Intel's next-gen desktop CPUs, "Titan Lake" possibly scrapping P/E-core split, moving to unified cores

• Huawei may move to a more GPU-like architecture for the future of its Ascend series AI accelerators

• AI accelerator startup Furiosa wins a significant customer in Korea - LG Electronics

• SK-Hynix confirms 3GB GDDR7 modules in development

• Samsung awarded $16B contract for Tesla's AI6 2nm chips over 8 years

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. In addition, we also cover vendors of anything interesting in the space, as well as short "one-pagers" on a random topic that we find interesting (and hope you do too).

For a space to share sources and news/updates, check out the notes section of this newsletter or my LinkedIn for posts on similar topics!

This month's updates:

• AMD delivers a roadmap to defeat Nvidia

• Broadcom shows the world a 100 Terabit switch ASIC: Tomahawk 6

• 1 PB/s planned for HBM within the next decade, KAIST reveals

• OmniPath might still make a comeback: Cornelis announces the CN5000 spec

• Other notable headlines

This month's updates:

AMD delivers a roadmap to defeat Nvidia

At their Advancing AI 2025 event this month, AMD delivered a reveal rivalling Nvidia's GTC last quarter. In it, AMD showed commitment to taking on the datacenter GPU market with a solid roadmap towards the most performant GPUs and the most ambitious rack-scale products planned for the next 3 years.

The roadmap charts a path through the three key routes towards dominance in AI training: CPUs, GPUs, and NICs. The Imminent "rack-scale" solution (2H25) is a not a particularly interesting stepping stone as it will still use a scale-out fabric between servers on the same rack, so as a quick summary: 64 MI350X air-cooled or 96/128 MI355X liquid-cooled GPUs, 5th gen. Turin CPUs, and Pollara 400G UEC (Ultra Ethernet Consortium) compatible NICs.

This product might still be quite popular among some hyperscalers wishing to deploy in volume, with OCI (Oracle Cloud Infrastructure) saying that they are preparing to deploy 130K (131,072) MI355X GPUs. This is unsurprising given that they have a very mature datacentre operations business and so won't be as impacted by the 40% increase in GPU TDP (see below for specs) as much as neoclouds or smaller datacentre build-outs would.

Given that there will be no APU form of the 350 series, it's entirely possible that the 355X was developed purely for Oracle and other hyperscalers who committed enough to justify a rather odd SKU like this.

The much more interesting part of the future begins in 2H26 with what AMD call the "Helios" architecture. According to SemiAnalysis' estimates, this will be a double-width rack supporting:

• 72 MI400X or MI450X GPUs

• 18 7th Gen. "Venice" CPUs

• 216 "Vulcano" 800G NICs

• 6 scale-up trays @ 204.8T each

• 6 scale-out trays @ 51.2T each

Below is SemiAnalysis' (estimated) diagram of the compute tray architecture:

This implies the following specs:

• 720 PFLOPs at FP16

• 31.1 TB/s of HBM4 @ 1.4 PB/s aggregate bandwidth

• ~130 TB/s all-all scale-up bandwidth

• ~173 Tb/s (bits, not bytes) uni-directional aggregate scale-out bandwidth

These values are designed to match/surpass not Nvidia's Blackwell Ultra (GB300), but the Rubin's (VR200) expected performance. In terms of scale-up domain, AMD is targeting NVLink gen. 5 (1.8TB/s), and for the scale-out they're pushing past at 2.4Tbps/GPU, 1.5x that of NVidia's plans built around its upcoming CX-9 NICs.

The options available for the topologies here are quite interesting. If we assume the SemiAnalysis estimates for the Helios's architecture, then the scale-up will be powered by 12 x 102.4T ASCIs (Broadcom's Tomahawk 6 is the only one available currently) which will present as 512 x 200Gbps ports each, for a total of 6144 x 200Gbps or 85.3 x 1.8TB/s connections. This means that each of the 12 ASICs can take 432 links, 6 from each of the 72 GPUs, and each GPU in turn exposes 72 links to the network. One possible topology is shown below:

Similarly for the scale-out, one possible simple topology is given below:

In both cases, we assume that the Vulcano 800G NICs would present dual 400G ports each, which would have the primary benefit of allowing for a wider range of OSFP or QSFP/QSFP-DD optics rather than 800G OSFP or other niche form factors albeit at the cost of additional parts and perhaps more energy consumption. In the case that NICs adopt CPO (Co-Packaged Optics) technology (though there isn't any information on this), then optics wouldn't be a factor but multiple ports would still be better in terms of redundancy and perhaps even performance.

Speculation: It's possible, but not supported by any sources yet, that AMD could move towards a dual (or even triple) plane architecture such as Alibaba's HPN. In this design, two identical versions of a 400G network connect to the same endpoints providing additional paths and hence more options for the network management algorithms to keep performance high under heavy, chaotic traffic patterns. In this case, it could mean that that each scale-out switch contains two 25.6T switch ASICs for physical separation of the planes, or that only three or two of the 6 switches are used per plane. Diagrams not provided.

Finally, some stats for all the new reveals, including both confirmed and estimated values (1 2 3 4 5 6):

MI350X:

• 288GB HBM3E 12Hi @ 8TB/s: 8 stacks of 36GB each (Samsung + Micron)

• ~4.6/9.3/18.5 PFLOPs @ FP16/8/4, ~72/144 TFLOPs @ FP64/32

• 1000W TBP (Total Board Power)

• Using TSMCs "N3P" 3nm and "N6" 6nm processes

• Shipping 2H25

• 256MB AMD "Ininifity Cache" (sort of similar to L3 cache)

• PCIe 5.0 x16 H2D interconnect

• 1/2.2 GHz Base/Boost clock

• Focusing on air-cooled deployments

MI355X:

• 288GB HBM3E 12Hi @ 8TB/s: 8 stacks of 36GB each (Samsung + Micron)

• ~5.0/10.1/20.1 PFLOPs @ FP16/8/4, ~79/158 TFLOPs @ FP64

• 1400W TBP (Total Board Power)

• Using TSMCs "N3P" 3nm and "N6" 6nm processes

• Shipping 2H25

• 256MB AMD "Ininifity Cache" (sort of similar to L3 cache)*

• PCIe 5.0 x16 H2D interconnect*

• 1/2.4 GHz Base/Boost clock

• Focusing on liquid cooled deployments

MI400X:

• 432GB HBM4 @ 19.6TB/s:

• ~10/20/40 PFLOPs @ FP16/8/4

• 2300W TDP

• 1.8TB/s scale-up bandwidth

• 2400Gb/s scale out bandwidth (3 x 800G NICs per GPU)

MI450X:

• Nothing new of significance found yet

MI500X:

• Reportedly either TSMCs "N2P" 2nm process or some 14A process**

*Assuming that the 355 will be very similar to the 350 in terms of architecture and composition. **The source that claims that the MI500X will use a 14A process node does not provide any justification for this statement.

SemiAnalysis and other sources mentioned above go much further into the details of the announcements, they are all well worth a read.

Broadcom shows the world a 100 Terabit switch ASIC: Tomahawk 6

51.2T switches - network switches that support up to 51.2 Tbps of aggregate bandwidth - are still only seen in large-scale high-end AI datacentres. Usually implemented as a 2U chassis of 64 x 800G ports, these switches power modern non-blocking east-west or backend fabrics. With the cable densities and transceiver counts used in large scale deployments such as Nvidia SuperPods, many datacentres have had a difficult time adapting to such high-bandwidth networking infrastructure.

Despite all of this, Broadcom targets the prepared and eager datacentres of the near future by announcing a 102.4T switch ASIC: the Tomahawk 6. The chip will come in 512 x 200G lane and 1024 x 100G lane SKUs, with the latter being significantly larger due to the increased shoreline (perimeter) required to support that many physical lanes. In addition, a CPO (Co-Packaged Optics) version of the chip will be available, for near-future switch platforms competing with Nvidia's upcoming QuantumX-800 and SpectrumX products.

Note that this is not a 102.4T switch, which may consist of multiple ASICs (chips), but a single chip.

Among the many claims made by Broadcom regarding the ASIC's performance, the most interesting was its ability to support 100,000+ "XPUs", or devices, on a two-tier network. Typically, for a non-blocking spine-leaf network topology as is common in Nvidia's SuperPod reference architectures, three tiers of switches are required when scaling to large numbers of devices. This is because in switches like Nvidia's SN5600 and Arista's 7060DX5-64S provide 64 ports of 400G each, which means that a single switch can connect to 32 devices at 400G (the other 32 ports are needed for uplink to the spine layer). With this domain size for a single switch, 64 such switches can be used in the leaf layer, with each spine switch having 64 ports to connect to each leaf switch. This then implies a maximum of 64 x 32 = 2048 devices in a two tier topology, using these switches, at a 400G bandwidth.

The Tomahawk 6, on the other hand, is promised by Broadcom to support 131,072 devices at 200G. One possible topology is shown below, a non-blocking spine-leaf setup (not-rail optimised because the diagram would get too complicated). To increase the bandwidth of the fabric without introducing a third tier, multiple "planes", or parallel implementations of this topology, can be used with extensive breakout in the ports/cabling. The benefit of such a setup would be the simplicity retained by each two-tier plane when it comes to congestion control, resulting in a more efficient and performant fabric overall.

The customers for the ASIC haven't been made public yet, but it's confirmed that sampling is underway, with mass production and hence integration into switches expected to start in 2Q26.

1 PB/s planned for HBM within the next decade, KAIST reveals

The 3 major memory makers - SK Hynix, Samsung, and Micron - are all working on HBM4 implementations, hoping to secure orders from the likes of Nvidia and AMD for their upcoming GPUs. The balance is tipping it seems with SK-hynix already sampling HBM4 and possibly even discussing custom HBM4 implementations with some customers, whilst Samsung is still struggling to pass Nvidia's qualification for HBM3E, but there is still a lot of potential for change.

Meanwhile, the Korea Advanced Institute of Science and Technology, or KAIST, Teralab research group recently presented a roadmap detailing the current trends and future plans for HBM. Extending out to 2038, the roadmap reaches HBM8 and speculates that packages (device chips) will reach ~1PB/s of aggregate memory bandwidth from 6TB worth of HBM modules, as well as requiring up to 15kW of power and futuristic "embedded-in-chip" liquid cooling methods.

Teralab have an extensive history of research on semiconductor packaging, interconnects, and HBM in particular, and have also collaborated with both SK Hynix and Samsung on HBM and surrounding technologies. The 371-slide presentation was part of a significant effort to understand the current technological and economic trends shown in the memory markets in addition to well-informed speculation based on current and planned research. The full slides are available here.

According to forecasts by the group, data rates for pins that connect the HBM module interface to the compute dies will level off at about double that of current HBM3E implementations. The capacity of individual memory dies however, should increase by ~3.5x, driven by denser process nodes and large die sizes. The leading hope for these incredible predictions though is the total module bandwidth increase, expected to jump sharply over the next decade due to stack height and bus width increases.

Based on how the increased stack heights, die densities, and module power usage all compound to increase heat generation, current cooling methods will not be able to keep up. Teralab propose two new methods: dedicated module immersion tanks and in-chip embedded liquid cooling.

The former is implemented already at a more macro-level, where entire servers can be dipped into tanks of non-conductive fluids, but is still constrained to research lab supercomputers (1 2) rather than hyperscaler datacentres. The proposed method here however shrinks the domain of immersion to the chip/stacks themselves, isolating the fluid but also allowing for a dedicated fluid supply to each module.

The latter is seemingly a much more complex and involved method requiring integrating the cooling system into the chip manufacturing process too, but appears to show promise for significantly cooling capabilities than current industry conventions. Presently, implementations of all such cooling systems remain in the research domain (1 2 3).

The presentation goes further into specifics on current and future research streams, and also talks about future SoC (System-on-Chip) designs created around HBM stacks rather than compute chips. Detail is given on how other memory technologies such as LPDDR die and HBF (High Bandwidth Flash) can be integrated onto future packages, controlled by HBM logic die to provide CPU-independent tiered memory systems for compute. For more on this, see the slides here.

OmniPath might still make a comeback: Cornelis announces the CN5000 spec

The modern networking market for AI is often seen as a two-horse race between InfiniBand and Ethernet, the first being an established standard for HPC and AI applications, and the second being the underdog who is rapidly catching up at smaller and medium scales and usually was the only real solution at extremely large scales. But there are other players each bringing their own pros and cons, but most only really being useful in very specific situations.

OmniPath, now advanced by Cornelis networks (but developed originally by Intel), is another competitor which appears to meet the criteria for being a viable datacentre data fabric: scalability, performance, and availability.

Introducing speeds of up to 400Gbps for a lossless and congestion-free fabric, as well as message injection rates "2x greater than InfiniBand NDR", OmniPath is sticking to its USP of having better latency. Though now, rather than staying in the shadows with research labs running HPC applications, Cornelis wants to bring OmniPath into the limelight with AI.

This is essentially the same plan they had in 2024 as well, predicting that UEC (Ultra Ethernet Consortium) compatible hardware would only be available in mass around 2027, despite the standard being finalised just this month. This is a sensible bet given how long it takes consortiums to progress from paperwork to implementation but given the urgency that industry is showing when moving away from InfiniBand, its possible that vendors will release limited ranges ofUEC-ready products in 2H25

Current plans for the CN5000 technology is to support 400G fabrics for up to 500K endpoints (devices), but by 2027 that should quadruple to 1.6T and 2 million endpoints, showing a good understanding of where the market is heading for hyperscalers, neoclouds, and AI labs. In addition, OmniPath will support Ethernet next year and then UEC the year after next, signalling that Cornelis do not intend to make the same mistakes that Nvidia did until last year with aggressive vendor lock-in through InfiniBand.

The CN5000 SuperNIC comes in single or dual port PCIe form factors, using gen 5. (~64GB/s uni-directional). Cornelis state the power draw to be 15 or 19W for the single and dual port versions, and will provide SKUs for air-cooled and liquid-cooled versions. One source claims that the NIC is capable of 800 million packets per second, and provides "<1 microsecond latency for MPI" though there are no other sources to corroborate this.

The switch is a 1U, 48-port chassis, with a TDP of 710W empty, or 1.1kW with 48 7.5W AOC (Active Optical Cables) ports. 48 ports is an odd number, and might be from some combination of having multiple interconnected ASICs inside, keeping the form-factor to 1U, reducing or managing latency by keeping the port count low, or some other optimisations to the ASIC for the incredible message rates OmniPath provides. These switches will also come in air and liquid-cooled SKUs.

It should be remarked that very few vendors are open about the power draws of their switches and NICs publicly like Cornelis is.

Other notable headlines

• Micron ships samples of its gamma process node-based LDPPR5X, showing significant improvements and a promising future for low power chips

• Chinese-aimed RTX Pro 6000D may well have NVLink after all, analysts suggest

• SK-Hynix reveals a DRAM roadmap for the next 30 years

• Qualcomm acquires Alphawave for its interconnects capabilities, confirming it's intent to compete in Datacenter AI accelerators

• PCIe 7.0 official specification is live, despite gen. 6 adoption being barely visible in industry

• Nvidia's Rubin GPU and Vera CPU rumoured to not be delayed at all - samples to be produced by September, mass production by 2Q26

• Samsung yet again fails HBM3E certification on Nvidia packages, aiming for a re-run in September

• SK-Hynix reportedly already supplying small quantities of HBM4 to Nvidia, Micron less than a quarter behind

• AMD's Zen 6 architecture could lead to 7 GHz boost clock rates on CPUs

• Samsung might ship HBM4 12Hi samples in early 3Q25, possibly catching up with other the memory makers

This is the SixRackUnits AI hardware newsletter, keeping you up to date with the latest in AI hardware, datacentre technology, and the future of compute. In addition, we also cover vendors of anything interesting in the space, as well as short "one-pagers" on a random topic that we find interesting (and hope you do too).

For a space to share sources and news/updates, join on Telegram, and check out my LinkedIn for posts on similar topics!

This month's updates:

• Nvidia opens up NVLink for custom CPUs and AI accelerators - NVLink fusion

• Intel showcases the 6700 series of Xeon CPUs - Also confirms B300 HGX architecture

• Nvidia's new Blackwell GPUs for China - unwavering resolve despite U.S. policy

• Intel releases the PCIe form-factor of their Gaudi 3 AI accelerator

• Other notable headlines

Vendor spotlight:

• iPronics

One-pager:

• Wave Division Multiplexing

Note: From this issue onwards, the number of one-pagers (segments aimed at explaining a concept in a single page) will be limited to one per month. Due to an increase in writing on LinkedIn, I will be sharing a lot of content there that will be similar in nature to the one-pagers, but this will come at the cost of having less time to work on this newsletter. I apologise in advance.

Note: In addition to the above, this month has been particularly busy for me, with multiple occurrences that have prevented me from committing as much time as I would have liked to this publication. As a result, this issue only has four updates instead of the usual five. However, I have made sure to include links to everything I found useful and the update that I intended to write a segment about in the "other notable headlines" section. Apologies once again.

This month's updates:

Nvidia opens up NVLink for custom CPUs and AI accelerators - NVLink fusion

At Computex '25 in Taiwan, Nvidia revealed their plan for tying themselves to a large portion of their competition in the merchant AI silicon space: NVLink Fusion. In the US and EU markets its arguable but in the Chinese market it's well agreed upon; Nvidia's second most important key to dominance is their NVLink scale-up interconnect. Investing in competing with that has now become unjustifiable for many competitors.

The speed at which SOTA LLMs can be trained - or even inferenced now - has largely been bound by the bandwidth between GPUs rather than the compute capacity available or memory bandwidth on the package. Nvidia's NVLink technology has led the industry in this aspect, providing not just incredibly high bandwidths between devices but also enabling the collective communications software that the world relies on: NCCL. The UALink (Ultra Accelerator Link) consortium has tried to upstage NVLink as the preferred interconnect technology, but have so far experienced significant delays in outputting a standard and show no plans for supporting hardware from any vendor.

At Computex '25 this month, Nvidia dealt another blow to UALink by making their NVLink technology stack open to custom, non-Nvidia CPU and AI accelerator hardware under the name of "NVLink Fusion". This means access to both the C2C (chip-to-chip) interconnect used for CPU-CPU and CPU-GPU, and the NVLink switched fabric for GPU-GPU communications. Both will likely be using the 5th gen. of the tech stack, providing up to 1.8TB/s bidirectional bandwidth between NVLink capable devices, using Nvidia's proprietary 200G SERDES over 18 Tx/Rx lane pairs, and perhaps even leveraging the potential scale-up size of 576 accelerators in a single domain.

Since GTC25 back in March, Nvidia did indeed make a point about open sourcing a lot of software, the Dynamo inferencing engine or "OS" being the centrepiece of this message. What wasn't expected was them opening up their hardware and related IP to others in the semiconductor design and manufacturing space. It should be noted though that this move doesn't fully open up and freely release the NVLink technology stack to anyone - firms will still have to register/license with Nvidia to use the IP and ultimately, Nvidia does have control over who uses this technology. In addition, only one of the CPU or AI accelerator can be custom, leaving space for Nvidia to sell their own chips through every such integration.

Currently, Nvidia intends to work with existing close partners such as Astera labs, Cadence, Synopsis, MediaTek, and Marvell on the SERDES and chiplet IP. Key first customers appear to be Fujitsu and Qualcomm, both known for having success with CPUs and NPUs. According to Dylan Patel of SemiAnalysis in a conversation with industry analyst Ian Cutress, "only the losers" will be interested in this technology, referring to players who have less experience and no existing proven hardware in this market. In addition, they implied that those with multiple generations of well-established products, a serious roadmap for the future, and a mature interconnect technology already in use - Google, AWS, Meta, etc. - will not have any interest in NVLink Fusion.

Intel showcases the 6700 series of Xeon CPUs - Also confirms B300 HGX architecture

Intel have been slowly revealing progressions of their 6th generation datacentre-class Xeon CPUs - dubbed "Sierra Forest" and "Granite Rapids" for the efficiency and performance focused versions respectively. At Computex '25, they debuted the 6700 series, packing up to 86 "P" - performance - cores per socket. With this reveal, they may have also (possibly prematurely) let loose details on the upcoming B300 HGX servers.

Intel's long-awaited P-core 6700 series were officially detailed at Computex in Taiwan earlier this month, and they intended to deliver a loud and clear message: Xeon is the AI server CPU. Nvidia's NVLink fusion reveal may have pulled some of the attention towards CPUs from soon-to-be competitors Qualcomm and a few others, but Intel's presentation was still impressive.

The 6th generation are the first to come in different SKUs for the E (Efficient) and P (Performance) variants. The E-core versions target primarily cloud-native and easily scalable workloads where individual core performance is not as important as the overall density and energy efficiency of the server. The P-core variants, on the other hand, are aimed at high-performance workloads such as AI training and inference, HPC, and other compute-intensive tasks where higher overall performance is required.

In particular, the larger of the 6700P series will top out at 86-cores and will support a maximum shared L2/L3 "smart" cache size of 336 MB. They will present up to 8 memory channels per socket, so a theoretical maximum of 4TB per socket with dual 256GB DIMMs per channel, but realistically we think they will practically only carry up to 2TB. In addition, the number of PCIe lanes increases to 88 per socket but stays at gen. 5 (64GB/s per an x16 connection). A new feature that Intel stressed heavily in their showcase was the priority core turbo or PCT technology, which allows the CPU to dynamically increase the clock speeds of the "most important" cores in a workload, up to 3.8GHz.

There is an unrelated but still very important detail we must point out: the above slide gives a pseudo-block diagram of the B300 HGX and shows NICs being used in place of PCIe 6.0 switches. This aligns with NVidia's MGX architecture post where they specified that ConnectX-8 NICs will come with integrated PCIe 6.0 switches with a 48-lane capacity. In this case it appears that each NIC will provide two x16 PCIe 6.0 connections (128GB/s each) to each of the two GPUs and one x16 PCIe 5.0 (64GB/s) to the CPU, with the CPUs using Intel's UPI for up to 384GB/s bidirectional. In the case of HGX servers, the switched NVLink scale-up fabric will remove the need for the PCIe links to be used for cross-socket GPU-GPU communication.

Nvidia's new Blackwell GPUs for China - unwavering resolve despite U.S. policy

Rising out of the confusion and panic caused by recent changes in semiconductor and hardware export regulations, Nvidia has announced its latest Blackwell-architecture GPUs for the Chinese market. The name mutated from H30 to B40 and now appears to have settled down to 6000D, but one detail had been clear from early-on: it will use GDDR instead of HBM.

Amidst rumours of yet another and even stricter restriction from the U.S. on the export of hardware to China coming in April, the first three months of this year saw up to $16B of orders for Nvidia's H20 GPUs. Despite this volume of sales, Wall Street now predicts a loss of up to $12.5B from existing inventory write off for unsellable H20s and the loss of potential revenue in Q2 for Nvidia. In addition, Huawei's Ascend series of GPUs has gained significant traction in the Chinese market in its third generation, due to the rapid maturing of their software stack to match the reliability and functionality of CUDA, and their incredible efforts in competing with Nvidia at networking multiple racks effectively.

At the start of this month, rumours of the new GPU began to surface though notably they were much quieter than in the past. First referred to as the "H30" or a direct successor to the H20, then called the "B40" as analysts speculated that it would be an inferencing focused card improving upon the popular L40S but with a Blackwell architecture. Finally, the name seems to have stabilised at "RTX Pro 6000D", in line with the regular RTX Pro 6000 which is the Blackwell generations datacentre and desktop grade inferencing card. What was even more surprising than Nvidia conceding on AI training in the Chinese market was that these GPUs - aimed at the largest and most demanding AI labs in the east - would use GDDR instead of HBM.

GDDR is a type of memory designed for graphics cards, providing significantly more bandwidth to GPU compute elements than regular DDR. Even at this increased performance however, GDDR is not capable of competing with HBM which in its current generation can provide many times as much bandwidth. In this case though, the limits to performance are enforced by policy and not technology. An intercepted email from Intel to its Chinese customers describes the limits on bandwidth prescribed by the new policy being 1.4TB/s for device memory, 1.1TB's of interconnect, and 1.7TB/s combined. We speculate that this approach leaves room for providing multiple SKUs and offering some variations in performance based on customer requirements. A more detailed analysis of this is available from The Register.

Regardless, it will almost certainly still be purchased in volume by firms in the east. It may require another tightening of export controls to finally make NVidia accelerators too difficult to justify when purchasing. Even with Huawei's solutions offering a viable alternative on paper, many AI labs in these businesses have extensive experience and existing code using CUDA and related software/frameworks. Transitioning from this will require more than just higher performing systems or even lower costs, but the process is underway for sure.

Intel releases the PCIe form-factor of their Gaudi 3 AI accelerator

Intel has long since terminated their Gaudi roadmap, ending the lineage of the AI accelerators made by the acquired Habana Labs in favour of a joint Intel-Habana architecture called Falcon/Jaguar shores, due 2026. In the meanwhile, Intel has released (on schedule) the surprisingly competitive PCIe form factor of their final accelerator in this series, the Gaudi 3.

The Gaudi 3 "add-on" card has just been released. Codenamed the HL-338, this is a FHFL (full-height, full length) double-width PCIe card supporting 16 lanes of PCIe gen. 5 and is designed to be added into medium/large AI/HPC servers. Originally announced in 2Q24 as part of Intel's now discontinued roadmap for the Gaudi AI accelerator series, the HL-338 is intended for both training and inference workloads as a standalone card or as part of a reference architecture. The official documentation describes four interconnected cards using 3/4 of their network bandwidth to form a fully connected mesh scale-up network, and 1/4 for scale-out via host NICs.

Habana Labs originally developed the Gaudi series independently and in parallel to Intel's own "Max" series of GPUs, but as early as 2Q23, Intel announced the merging of the two technologies into one product line, codenamed the Falcon shores. Shortly afterwards, Intel also declared that they would no longer be designing the new merged GPU around AI training, but for Inference and HPC workloads only. Further, just 3 months later they scrapped Falcon shores (a PCIe form factor card) from their sales plans and stated that they would focus on their Jaguar shores (larger OAM card), with an emphasis on rack-scale systems just as Nvidia and now AMD are doing.

Rack scale solutions also exist, but are in the form of reference architectures only, not custom racks like Nvidia's NVL72 or AMD's upcoming IF64/128 solutions. That being said, the reference architectures are comprehensive and provide guidance on the rack layouts, network topologies, and even server/fabric configurations for developers and engineers to follow.

Finally, the specs (1 2 3 4):

• ~1.8 PFLOPs of BF16

• 96MB SRAM @ 12.8 TB/s bandwidth

• 128GB HBM2E @ 3.7TB/s (8 stacks of 12hi from SK-Hynix most likely)

• 64GB/s H2D bandwidth via PCIe gen. 5 x16

• Using TSMC's 5nm process

• 600W TDP AND:

• Single card: Up to 4.8TB/s aggregate scale-out bandwidth via 24 x 200G RoCE ports (48 lanes @ 112G) OR

• 4 x cards: Up to 4.8TB/s aggregate scale-out + 14.4TB/s aggregate scale-up (1:3 ratio scale-out:scale-up)

Other notable headlines

• Nvidia might postpone the adoption of SOCAMM LPDDR5X modules for its GB300 boards, sources cite thermal issues and stability problems

• AMD's Venice CPUs (2026) confirmed to use TSMC's 2nm process - "TSMC is the leader in 2nm", AMD states

• Nvidia's 1.6T optical transceiver's likely delayed from 4Q25 to 1Q26, sources cite supply chain issues

• Nvidia's Connectx-8 NICs to come with PCIe 6.0 switches inside - acting as the interface between CPU and two GPUs each

• IBM releases the LinuxONE Emperor 5 platform - Mainframe, but without the need to survive an earthquake

• Samsung confident in their 1c DRAM node, betting on improving yields and outpacing competitors SK-Hynix and Micron in HBM4

• AMD acquires silicon photonics startup Enosemi - Aiming to compete with Nvidia and Intel in co-packaged optics

• Apple reportedly developing M-series chips up to three generations ahead - M8 "Sotra" chip may have up to 256 CPU cores and 640 GPU cores in one package

• Nvidia backtracks to the "Bianca" model for their SXM motherboards, delaying plans for the "Cordelia" which would have allowed for socketed, replaceable GPUs

• AMD rumoured to launch a GPU for the Chinese market following the same path as Nvidia's RTX Pro 6000D: a version of the Radeon AI PRO R9700 with GDDR7

Vendor spotlight:

iPronics

Datacentre switching is reaching a point where the relative power draw of the optical-electrical interfaces (transceivers) required to carry the extreme bandwidths of AI fabrics is becoming a concern. 800G fabrics now need special transceiver casings, as well as taller ports just to stay cool. One way to avoid scaling implementation and integration difficulty is to use optical circuit switching, or OCS.

Founded in 2019, Spanish startup iPronics focusing on "software-defined photonics", or photonics hardware designed to be programmable and reconfigurable. iPronics' OCS technology seeks to replace power hungry and fixed-bandwidth electrical switches in datacentres with photonic (light-based) switches that route optical signals without converting them to electrical signals.

Electrical conversion requires significant amounts of power, maintenance, and specialised hardware to support a limited number of bandwidths based on the signal processor design. For example, a 400G switch (referred to as a 25.6T switch by aggregating its 64 ports) requires datacentre staff to use a guide on plugging in the right cable into the right transceiver into the right port, in a particular pattern in order to cable up a particular topology. In addition, these transceivers can only support 400G, 2 x 200G, or 4 x 100G bandwidths usually, and upgrading to higher bandwidths requires new switch ASICs, ports, and transceivers.

Their "ONE" switch is a 32-port 1U switch based on their programmable unit cell technology, which is a versatile compute/switching unit that can be combined into structures of seemingly arbitrary size to create something analogous to a switch ASIC with compute attached. Using this, the OCS device can not only route and process signals on the same package but can also be reconfigured via software to modify the routing and processing within 100s of microseconds. This means that in theory, switches made from these programmable unit cells should be able to support in-network compute as well as rapid topology changes to the fabric without extensive maintenance and costly downtime.

Finally, the specs for the ONE switch are as follows:

• 32 ports independent of bandwidth (up to some limits most likely)

• 1U form factor with 8 physical ports it seems

• <30ns switching latency (compared to 500-1000ns for high end electrical switches)

• <300 microseconds reconfiguration time compared to 100s of milliseconds for mirror-based optical switches

One-pager:

Wave Division Multiplexing

Optical fibre itself does have a limit to how much bandwidth it can carry, but current technology is not yet at that wall. To increase bandwidths for cables that need to carry multiple signals at once, one method is wave-division multiplexing (WDM), using multiple wavelengths (or colours) of light to simultaneously carry multiple distinct signals through one fibre.

WDM is used widely in industry for long range optical communication such as undersea cables, telecoms networks, and more recently, datacentre interconnects. It is a method of multiplexing multiple signals over a single optical fibre by using different wavelengths (or colours) of light. Each wavelength carries its own data stream, and the combined signals are transmitted over a single fibre. At the receiving end, a de-multiplexer separates the combined signals back into their original wavelengths.

Initially, WDM in the 1980s used four wavelengths of light at around ~850nm, but since then the technology has advanced to allow for an order of magnitude more of wavelengths at much higher wavelengths of between 1525-1610nm. WDM that uses a few, widely spaced set of wavelengths is now called coarse WDM (CWDM), while more modern dense WDM (DWDM) uses many closely spaced wavelengths. The latter is the most common in use today, with some practical implementations enabling up to 96 distinct wavelengths to be carried over a single fibre. Using a modern 400G signal, DWDM would allow for an effective bandwidth of 96x400Gbps or 38Tbps over a single fibre.

Using narrower gaps between wavelengths requires more precise hardware to avoid signal loss due to interference and attenuation, but the limitations of existing fibre optics technology leave no other option. Due to various physical phenomena and interactions between light and the glass fibre, certain bands of wavelengths result is less signal loss than others. The C and L bands (1530-1565nm and 1565-1625nm respectively) are the most commonly ranges used for DWDM, and increasing the number of distinct usable wavelengths in these bands requires narrowing the gaps between them.

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)

This month's updates:

• SemiAnalysis on AMD's upcoming rack-scale SKUs

• Tenstorrent announces the BlackHole AI accelerator

• IBM's z17 mainframe

• Ironwood: Google's seventh gen. TPU

• China's answer to the NVL72 - Huawei's CloudMatrix 384

• Other notable headlines

Vendor spotlight:

• Kove

One-pagers:

• Bandwidth and latency

• Liquid cooling plates

This month's updates

SemiAnalysis on AMD's upcoming rack-scale SKUs

For AI accelerator manufacturers targeting hyperscalers, few things in AI hardware have become clearer in the past quarter than the need to focus now on systems rather than chips. Nvidia have taken the lead by supplying their NVL rack-scale products to all major hyperscalers as well as some GPUaaS providers, promising a turnkey infrastructure solution that meets the demands of the largest AI workloads. Rumours surface now of AMD's entry into this space with their own, the IF64 and IF128.

Historically, AMD have leveraged their strong partnerships with OEMs and integrators to build some of the worlds largest and most performant datacentre-scale systems, the most prominent recent example being the world's top-ranking supercomputer, El Capitan, built in collaboration with Cray (HPE). After the success of Nvidia's NVL systems though, AMD have moved to expand their own expertise in scaling beyond single servers through their acquisition of ZT systems, an integrator with extensive experience in building datacentre-scale systems with hyperscalers such as Azure and AWS. Now, reported exclusively by SemiAnalysis, IF64 and IF128 are likely to be the names of AMD's upcoming systems built around the MI450X GPUs and launching in 2H26.

Nvidia's "Rubin" architecture GPUs (also planned for release in 2H26) are predicted to have a TDP of ~1.8kW and will be used in the "Oberon" fully liquid cooled NVL144 systems. The MI450X therefore may have a TDP of between 1.6 and 2kW to be competitive, and will likely also follow the industry trend of removing FP64 in place for native FP16 and FP8 support, though AMD may release a separate GPU dedicated for the Scientific/HPC applications in which the MI300X has shown great success. Again, similar to Rubin GPUs it's likely that each die of the MI450X will support a minimum of 288GB of HBM4, though unlike Nvidia, AMD will deploy its x86 EPYC datacentre CPUs rather than ARM-based low-power (and lower performance) Grace/Vera CPUs.

As for the scale-up network, AMD's Infinity Fabric over Ethernet (IFoE) likely just encapsulates the existing IF protocol already used for device-device (CPUs and GPUs) communication within a chassis and allows for it to be used over an ethernet fabric via NICs. SemiAnalysis predicts the performance of IFoE will allow for 1.8TB/s unidirectional bandwidth between GPUs, making it equivalent to the expected values for NVidia's 6th gen. NVLink. It's unclear what the maximum domain size will be and therefore how large a single system can get, but there are two key observations to note from the article: The first is that unlike Nvidia who cancelled theirs, AMD appear to be committed to a dual-rack DAC connected product (IF128), and the second is that from the B300 onwards, Nvidia will refer to a single die as a GPU rather than the entire package, leading to an artificial doubling of their GPU count.

Tenstorrent announces the BlackHole AI accelerator

American RISC chip/server design startup Tenstorrent seeks to take space in both the enterprise and retail AI accelerator markets with its low-power and easy to use PCIe card lineup, as well as servers designed around density and scaling. Their latest card, the BlackHole, seeks to offer a competitive price/performance ratio with a scale-up potential that appears to be significantly greater than what other PCIe devices from competitors such as Google and AWS can offer.

Run by industry legend Jim Keller, Tenstorrent is one of the many startups targeting the (relatively) low-power PCIe card AI accelerator space with the Wormhole, and now also the BlackHole cards. What makes Tenstorrent particularly unique is their custom IP for the RISC-V cores they use on their cards, designed specifically around their focus on keeping power draw low but also allowing for their "infinite" scaling strategy. The card comes in 3 forms, the P100a, p150a, and p150b, the first being a version without scale-up networking and only 28GB of GDDR6 with 120 RISC-V "Tensix" cores, and the last two being actively and passively cooled versions that come with scale-up ports on the PCB itself, as well as supporting up to 32GB of GDDR6 and 140 of the same cores.

The Blackhole uses PCIe 5.0 for its host interface, but remarkably supports up to 3.2Tb/s of scale-up bandwidth through 4 x 800G QSFP-DD ports on the card itself. Some analysts suggest that due the sheer amount of bandwidth available on the card, it might even be usable as a network switch, though Tenstorrent clarifies that this interconnect will be exclusively for connecting to other Blackhole cards only. This p2p (peer-to-peer) scale-up strategy allows for creating 2D torus topologies much like Google's TPUs or AWS's Trainium cards and Tenstorrent's design of their server built around Blackhole cards, the "Galaxy", shows that they agree. A Tenstorrent Galaxy system contains 32 of the accelerators in a 6U chassis, arranged in a 4x8 mesh and providing up to 9.3 PFLOPs of FP8 performance in a 7.5kW envelope, leading to it being competitive against Nvidia's H100 HGX systems in efficiency (FLOPs/Watt).

IBM's z17 mainframe

IBM mainframes are known for their near perfect uptime and low-latency for critical operations such as transaction processing, airline fleet management, and fraud detection to name a few. As more and more such workloads begin to use computationally expensive AI models, IBM's mainframes need to adapt to not just be better at what they already do, but also be better at the workloads of the near-future.

The z16 mainframe released in 2Q22 was designed before transformer-based AI models had replaced many statistical/ML models in some mission critical applications, leading to IBM using the inbuilt AI acceleration capability of the Telum I processor to support such workloads. Integrated AI support came in the form of dedicated matrix units on the processors, but as the type and scale of AI workloads grew rapidly over the past 3 years, IBM realised that the next mainframe would necessarily need to be designed around AI workloads being run on dedicated accelerators. The z17 embodies this change, with the new system being designed to support AI workloads natively and with the same level of reliability and performance as other applications running on the mainframe.

As we reported in August last year, The Telum II and Spyre processors now used in the z17 are designed around extreme uptime, power efficiency, and consistent performance. The new mainframe hosts up to 32 Telum II CPUs, each containing 8 cores running at 5.5Ghz and 360MB of L2 cache, with 360MB and 2.8GB of virtual L3 and L4 caches respectively. The CPUs are supported by dedicated data processing units (DPUs) for offloading inter-processor communication and other functions for secure and reliable communications, leading to all 32 processors being connected in one scale-up I/O domain. Finally, the mainframe includes 12 I/O expansion drawers which can each support up to 16 PCIe Gen 5 cards, leading to a total of 192 Spyre AI accelerator cards - over 24TB of fast LPDDR5 memory - being installed in a single z17 system.

Holding true to their legendary eight nines (99.999999%) uptime guarantee, IBM openly discuss the development and testing process for their processors and systems showing how z17's will have around 1 second of downtime per year, even through up to magnitude 8.3 earthquakes. ServeTheHome's articles and review (1 2 3) reveal just how extensive IBM's development processes are, showing the complete vertical integration of their manufacturing and testing, from the materials and chemicals used for the processors and cooling systems to the racking and securing of the components for functionality under extreme conditions.

Ironwood: Google's seventh gen. TPU

In 2Q24 Google introduced the "Trillium" TPUv6 (Tensor processing unit), a continuation of their remarkably power-efficient series of custom AI accelerators available on GCP (Google cloud platform). Now, the series continues but with an unexpected progression: Inference and training are no-longer being assigned to two different versions in the same generation. The "Ironwood" TPUv7 appears to take on both workloads but the marketing around it confuses many.

Google's latest in their custom silicon for AI workloads, the "Ironwood" TPUv7 has just been announced. It's unclear yet what stage of production it's at, or when it will be generally available for public GCP users, but some specs are available now. Of all the information provided, the most controversial statement made was a misguided (or even dishonest) comparison between a TPU "pod" and the worlds most computationally capable supercomputer, El Capitan.

The "Top500" organisation maintains a list of the worlds most powerful supercomputers ranked by their performance in a linear algebra benchmark, Linpack. El Capitan, placing first as of 4Q24, combines over 43,000 AMD MI300A APUs (CPU + GPU on the same package) to achieve a real performance of 1.74 exaFLOPs at FP64, with a theoretical peak at 2.74. TPUv7 pods on the other hand use their 2 or 3-dimensional torus network topology to combine 9,216 chips (not devices) into a single computational domain. Additionally, the performance per chip provided in the announcement is a theoretical peak as opposed to a measured benchmark and is very likely for FP8 or FP16 as opposed to FP64. Google's claim therefore of reaching a 24x advantage in exaFLOPs over El Capitan has been received poorly by analysts with opinions generally skewing towards it being grossly misleading.

Focusing on the chip itself, it appears that the v7 is a natural successor to the v5p, the "performant" version of the 5th generation which aimed to handle both lighter inferencing workloads as well as AI model training use cases, (in contrast to the v5e or "efficient") rather than the v6 which was likely focused on inferencing only. Finally, looking at the specs (1 2 3 4):

• 4614 TFLOPs at FP8 (~2300 BF16)

• 192GB HBM at 7.2TB/s - likely 6 stacks of Micron's 12hi HBM3E

• Likely 4 (possibly 6) ports of ICI at 1.2TB/s bidirectional

• ICI domain size of 9216, possibly in a 96x96 2D torus or 32x32x9 3D torus

• 4 x 256x256 MXU (systolic arrays), providing 65536 MACs/cycle

• Estimates of up to ~1kW per chip

China's answer to the NVL72 - Huawei's CloudMatrix 384

Nvidia's NVL72 was not the first but was certainly the most popular rack-scale, self-contained infrastructure solution built around AI accelerators. Densely packing this volume of high-powered chips into a single rack and interconnecting them with the NVLink scale-up network led to a product that could be easily integrated and used by hyperscalers and large businesses. But the NVL72 has its flaws, and Huawei is stepping up to take on Nvidia in this space too.

It's likely to come as a surprise to many who don't operate in the Chinese market, but Nvidia internally see Huawei as one of their biggest competitors for AI accelerators. We reported back in February and even last October about Huawei's progress in high-performance AI accelerators, with their Ascend 910C being announced and now mass produced for use in firms such as Baidu, ByteDance, and most famously, DeepSeek. All interested parties were or are evaluating the chips in hopes of beginning to taper off of their reliance on foreign hardware amidst increasingly hostile export policies from the US.

Seeing the trend in the market for rack-scale systems, Huawei have announced their own reference architecture, the CloudMatrix (CM) 384, a multi-rack system built around the Ascend 910C AI accelerators and "CloudEngine" network switches. The 16-rack, 384 chip SKU hosts 4 x 8-GPU air-cooled servers per rack over 12 racks with 4 racks of switches, and is fully optically connected for its single scale-up domain; this means that unlike the NVL72 which uses copper for its NVLink scale-up network, the CM384 uses almost 5400 400G transceivers. Some analysts predict that this fact alone will limit the number of CM384s that will ever be deployed as its possible that the supply of transceivers will be insufficient to meet the demand for the racks. Others raise concerns about the total power draw of the system being a hindrance to its adoption, predicted to be at ~560-600kW per unit, well over 3x the advertised power of the NVL72. Despite this, SemiAnalysis' calculations show an advantage in the FLOPs/Watt ratio of the CM384 over the NVL72, also noting that power availability is usually not an issue for datacentres in China as it is in the US and Europe.

As for the system's specs (1 2):

• ~300 PFLOPs at BF16, ~780 TFLOPs per chip

• 49.2TB of HBM total, 128GB per chip

• ~1 PFLOPs aggregate unidirectional scale-up bandwidth

• ~150 TFLOPs aggregate unidirectional scale-out bandwidth

• ~600kW TDP at peak load with all transceivers in

Other notable headlines

• SK-Hynix possibly mass producing both HBM4 and HBM4E this year

• Chinese firms place up to $16 billion of orders for Nvidia's CHIPS act-dodging AI GPUs

• Samsung reportedly begins developing its 1nm "dream" process, ready for mass production in 2029

• Nubis communications and Amphenol to develop signal re-drivers to enable 200G/lane SERDES across multiple meters of copper, enabling multi-rack scale-up networks without optics

• UALink (Ultra Accelerator Link) consortium announces their 1.0 specification, at 200G per lane. Aims to provide an open standard against Nvidia's NVLink

• Cadence reveals its HBM4 IP promising up to 12.8 Gbps per lane - 3.2TB/s over a 2048-wide bus for a single stack

• AI accelerator startup SambaNova refocuses on Inferencing only, retreating from challenging Nvidia and others in training - by laying off 15% of their workforce

• Intel confirms experiments on direct-liquid cooled solutions for CPUs reaching up to 1000W TDPs

Vendor spotlight

Kove

Host memory capacity is and has always been the be-all and end-all of large, high-performance workloads, as triggering an OOM error usually causes not just the application and its progress/state to crash, but the entire host. Kove are a firm working to change this with a solution for remote, software-defined memory (SDM) that approaches the performance of local memory.

Founded in 2003, American firm Kove has quietly been supplying remote memory solutions to customers who found their workloads constrained not by compute, but by memory capacity. Kove's memory tower system presents as a rack of DRAM enclosures containing some management compute and high-performance NICs, and their software stack manages the remote memory and transfers to expose a memory pool that can be defined (configured or managed) entirely by software. With a maximum capacity of up to 256TiB (tebibyte, or 2^40 as opposed to 10^12 bytes) across several memory towers available as one unified address space with the local memory, bandwidth-intensive workloads can be accelerated without needing the compute to be throttled.

At a high-level, the memory tower works by extending the main memory available to a host machine via network-attached DRAM, rather than adding more channels for DIMMs onto the host baseboard or connecting another board via PCIe. Kove supports InfiniBand for the data plane (with RoCE support in the works) and Ethernet for the control plane, allowing for reasonably easy to manage additional infrastructure in any standard ethernet-managed datacentre with a dedicated and isolated high-performance fabric for the data.

As for software, the management and networking stack works together to use a host server's local memory as a form of cache (think of it as L4/5), and keeps recently accessed and likely soon to be accessed within the host chassis. Remote memory then is able to serve as a nearly-identical access latency extension of this local cache, as the microseconds of time to RDMA data from the tower to host are likely amortized away by both the large bandwidth available on the network and the access prediction capabilities of the software.

One-pagers

Bandwidth and latency

Bandwidth, being the amount of data transferred within a unit time and latency, being the time taken to complete a transfer of a unit size, these two key measurements in any data transfer are often misunderstood and misrepresented. Even worse, some will simple invert bandwidth and call it latency not realising their lack of consideration for overheads, measurement conditions, and variance, and the damage they do to the science.

Measuring the performance of transferring data via a connection is usually done with two measurements: bandwidth and latency.

Bandwidth measures the maximum volume of data that can be transferred through a connection in a given unit of time, and is measured as a rate such as Mbps (megabits per second). It's important to note that this is the maximum theoretical rate, not the actual or "realised" rate of data transfer across a connection. Bandwidth is often abstracted (applied to) or aggregated (accumulated) across a subset or an entirety of a network to give some indication of the potential rate that can be achieved for a workload using it. Latency on the other hand measures the time taken for a minimum unit volume of data to be transferred from a source to a destination, and is measured in units of time such as milliseconds. Some factors that influence latency like distance between endpoints or the speed/path of light in a material are physical and unavoidable, whilst others like the time to encode and decode signals are dependant on the hardware and protocols used for transmission.

A common error made when measuring network performance is assuming that bandwidth is the inverse of latency, i.e. that a bandwidth of 100 Mbps implies a latency of 1/100 seconds. This assumption ignores a few issues:

• Latency is only measured for a minimal sized transfer, one that should never be large enough to saturate a link for measuring bandwidth

• Bandwidth does not include the overheads involved with each minimal transfer, as these are amortized away when sending data in volume

• As the rate of transfer over w channel increases, so does contention and hence resource utilisation, which causes an increase in latency i.e. the relationship is non-linear

Liquid cooling plates

The height of air-cooled AI servers is influenced significantly by the size of the fans and the height of the heatsinks required to keep high-powered accelerator hardware cool. Compression attached cooling plates replace tall towers of metal fins in thinner-profile servers where getting enough airflow for cooling kilowatt+ devices either costs double-digit percentages of the servers total power draw, or is just physically impossible.

A prime example of air cooling pushed to its limits is SuperMicro's B200 HGX server, the SYS-A22GA-NBRT. Fitting 8 Nvidia B200 GPUs (each with a TDP of 1200W) as well as 2 Intel Xeon 6900s (about 500W each) into an air cooled chassis requires accommodating the cooling fin towers for the GPUs, taking up the entire floor-space in the chassis as well as over 6U of height. The remaining space needs to be large enough so that enough airflow can pass to keep the other components such as CPUs, NICs, SSDs and more. Pushing beyond this limit to 1400W+ TDP chips and more will require significantly greater volumes of airflow moving through impractically large chassis, to the point where the power used by cooling fans will creep towards 15 or even 20% of the servers total draw.

Using a denser fluid such as liquid is the next logical step for cooling systems. Immersion cooling appears to still be building a reputation, having been only applied successfully in a very small number of setups, and so the only practical solution currently is direct-to-chip liquid cooling. In this paradigm, large coolant distribution units (CDUs) are placed underneath or at the side of existing racks and large, high pressure plumbing systems connect the pumps to in/outlets on the chassis. Further plumbing internal to the chassis then routes the cooled liquid from the CDU through "cold plates", or flat hollow plates made from copper or aluminium, which are in placed in contact with the chips/components. The heated liquid is then pumped back to the CDU for cooling.

Systems like the plate shown above can reliably cool chips with TDPs of up to 2000W, with models rated for 4000W (under very high coolant pressures) being evaluated for future AI accelerators. For servers, adopting cooling plates for all or even some of the components leads to a significant increase in power efficiency and compute density, allowing for solutions like Nvidia's NVL72 racks. Definitely not the first - but by far the most popular - NVL72s use extensive liquid cooling support infrastructure to cool 27 x 1U trays in a single rack, requiring many types of cooling plates for the various collections of components on the baseboard. Moving forward, more standard server architectures are likely to shrink in size towards 2-4U and datacentre PUE values will decrease.

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)

This month’s updates:

• Official specs for the Nvidia Rubin NVL576 released at GTC

• Nvidia reveal plans for the GB300 NVL144 and HGX NVL16

• Nvidia and partners announce next-gen datacentre switches with co-packaged optics

• Bolt Graphics reveals “Zeus” GPU – Claiming to 10x the Nvidia 5090’s performance

• Micron secures first SOCAMM memory module supply for Nvidia GB300s

• Other notable headlines

Vendor spotlight:

• Cerebras

One-pagers:

• Silicon binning

• Branch prediction

This month's updates:

Official specs for the Nvidia Rubin NVL576 released at GTC

Finally adding certainty on the specifications for the Rubin-series GPUs, Nvidia presented the Vera-Rubin NVL144 and the “Ultra” variant as the NVL576, promising to quadruple the density of the current generation of their rack-scale SKUs by 2H27 – But just how many customers will be ready for a 600kW rack?

In the pursuit of density, Nvidia are planning to quadruple the GPU count in their rack-scale NVL SKUs for the peak of the Rubin generation, with the NVL576 consisting of 144 Rubin Ultra devices containing 4 reticle-sized GPUs each. For this, Nvidia are moving to an orthogonal blade server architecture for the racks, with the compute blades now being vertically slotted into a single tall chassis and with multiple such chassis in a rack. The NVLink switches will also be in blades but will enter from the rear of the rack and will connect to the GPU blades via multiple midplane PCBs which will sit between the compute and NVSwitch blades. Notably, the midplane will have no concessions made for airflow, as the entire rack – every GPU and CPU, every NVSwitch and PCIe switch, every memory module and SSD – will be liquid cooled, leaving room for a high NVLink socket density.

There are three points important of note for this architecture. Firstly, the count of 576 GPUs is due to Nvidia changing how GPUs are counted, with each chip being a GPU rather than each device being one. This means that conventionally, this rack would be known as an NVL288, though each of the chips will present as a GPU to the OS. Secondly, it’s unspecified yet how the 4 chassis in the rack will connect into one NVLink domain, or if the rack will present as a 4 x NVL144 instead of a NVL576, its possible Nvidia will still be dependent on DACs for this. Finally, whilst Nvidia claim the TDP is ~600kW, its possible the actual value may be as high as 819kW, as per SemiAnalysis’ prediction.

Finally, the specs (that we know of so far) (1 2 3 4 5 6):

• 288GB HBM4E per GPU, 4.6PB per rack

• 3.6TB/s bidirectional per GPU with NVLink 7

• 15EFLOPs at FP4, 5EFLOPs at FP8

• 288 CPUs w/ 88 Arm cores each, 25344 total

• CX9 1.6T NICs, BF(4?) DPUs at 1.6T

• 365TB LPDDR(6?) system memory

• 1.8TB/s C2C between pairs of CPUs

• TDP between 600kW and 820KW

Nvidia reveal plans for the GB300 NVL144 and HGX NVL16

Nvidia revealed their GB300 (Blackwell Ultra) rack-scale SKU as the NVL72 during Jensen Huang’s GTC 2025 keynote presentation, but with an emphasis on how a single chip will be two GPUs rather than one from now on. The new method of counting GPUs may take some time to propagate, but the availability of this SKU is practically imminent.

As expected, Nvidia announced the GB300 NVL72 rack-scale SKU, debuting the Blackwell “Ultra” GPU. This comes as a direct upgrade to the B200 with 50% greater FP4 FLOPs at the cost of almost all the FP64 capability, and an increase from 192 to 288GB of HBM3E. As for connectivity, the scale-up network will be NVLink gen. 5 providing 1.8TB/s bidirectional from GPU-GPU and the scale-out will use 800G CX-8 NICs with NDR800/XDR InfiniBand or 400/800G Ethernet. Though initially, only 800G InfiniBand deployments will be available due to release schedules for the SpectrumX switches supporting 1.6T ports.

Vendors and integrators who are familiar with having to replace entire baseboards for GPU repairs and maintenance will now be able to provide better SLAs due to Nvidia putting GB300 devices onto easily replaceable sockets called “SXM pucks”. This is in line with the industry shift towards socketed, modularised solutions for high-power accelerators and memory devices. The GPU itself will still be a dual-die device which Nvidia have agreed for now to still refer to as a single GPU (at least for their NVL units), unlike in the HGX NVL16 which will be a single-die device with no high-bandwidth interconnect in between.

And of course, the specs, per GPU (1 2 3 4 5 6):

• 15/5/2.5 PFLOPs at FP4(dense)/FP16/TF32, ~1.4TFLOPs FP64 and 320TFLOPs INT8

• 288GB HBM3E, 8 stacks at 12hi (SK-Hynix exclusive), bandwidth remains at 8TB/s due to pin speed

• CX-8 NICs, 1:1 GPU:NIC ratio, supporting NDR800/XDR and 800G Ethernet, with BF3 DPUs at 4:1

• NVLink gen. 5 at 1.8TB/s bidirectional, using 224G SERDES across 36 lane-pairs (50GB/s each)

• 1.4kW TDP in GB NVL solution, 600W for the HGX version (per die)

• Mixture of liquid and air-cooled variants available via vendors

Nvidia and partners announce next-gen datacentre switches with co-packaged optics

The necessity of optical fibre in datacentres doesn’t imply the necessity of explicit optical-electrical interfaces, as TSMC and Nvidia demonstrate with their upcoming co-packaged optics switches. Their Quantum-X InfiniBand and Spectrum-X Ethernet switches, due for 2H2025 and 2H2026 respectively, aim to remove the need for optical transceivers entirely.

Anyone working in a datacentre will be familiar with the downtime caused by port-flapping, overheating, or just the eventual degradation of optical transceivers that servers and switches are increasingly requiring more and more of. At the GTC keynote presentation this year, Nvidia announced a collaboration with various industry partners on using co-packaged optics (CPO) to allow optical signals incoming from fibres to route to the ASIC package itself, rather than through a transceiver. Instead, CPO switches will use external laser sources to facilitate communication, which will come in similar form factors to SFP. Nvidia’s current roadmap includes a 144 x 800G QuantumX IB switch available in 2H25, and 128/512 x 800G SpectrumX Eth switches available 2H26.

Though not entirely avoiding pluggable, CPO switches still provide two key benefits to AI datacentres: significant power savings, and hardware volume reduction. Compared to electrical switches with optical/electrical conversions happening at each transceiver and then being propagated to the switch silicon, CPO switch ASICs now no longer need external digital signal processors for reducing noise and converting between digital and analogue signals. In addition to this, the switch interface itself can be much denser as optical fibres can plug directly into the switch, resulting in much higher port-counts and hence the ability to flatten the physical topology of a large network from three to two layers.

CPO technology will inevitably feature on NICs or even processor packages themselves, removing the need for transceivers on the server side too. As reported before in this publication and from industry rumours regarding future generations of Nvidia hardware, experimental CPO enabled designs for GPUs are already being developed for ~2027/28, hoping to allow fully optical connectivity from accelerator to switch.

Bolt Graphics reveals “Zeus” GPU – Claiming to 10x the Nvidia 5090’s performance

American startup Bolt Graphics claims that their upcoming Zeus GPU will provide 10x the graphics rendering performance of Nvidia’s flagship RTX 5090’s. But with up to 20 TFLOPS of FP64 and 2.25TB of memory per GPU all within a 500W TDP, the Zeus is sparking interest in both academia and industry for HPC workloads too.

Designed around enabling real-time model/graphics visualisation and handling larger frames than current rendering farms can handle, hardware design startup Bolt Graphics intends to accelerate intensive industrial graphics workloads with their upcoming Zeus GPU. In addition, early benchmarks show that the focus on supporting high-precision data formats also serves to accelerate highly-parallelisable scientific workloads such as electromagnetic wave simulations, with indicative performance being up to 300x that of Nvidia’s B200 GPUs. Initial samples are due in 2H25 and mass production is planned for 2H26, which places it two quarters ahead of Nvidia’s “Rubin” architecture-based RTX 60XX series according to our predictions.

Within a thermal profile of just 500W, the Zeus GPU provides 20 FP64 TFLOPs (vector) and 2.4 FP8 PFLOPs (matrix) to accelerate both high and low precision workloads, supported by 512MB of L2 cache, and a total device memory of 2.25TB - split between 256GB of LPDDR5X and 2TB of DDR on DIMMs - at 1.45TB/s. As for I/O, the device’s baseboard itself carries 6 OSFP 800GbE ports and 32 lanes of PCIe gen5 in total, allowing for a CPU-GPU bandwidth of up to 128GB/s and a GPU-GPU bandwidth of anywhere between 800-6400Gb/s (both unidirectional) depending on the data fabric topology. There are four variants of the Zeus planned currently: the single chip 1c, two variants of the dual chip 2c version, and the flagship 4c quad-chip version detailed above.

Most notably in their marketing material, Bolt Graphics claim that a single Zeus can replace 10 Nvidia RTX 5090’s in render farms, leading to a total power reduction of up to ~91% (based on a 5090 running at 575W). The primary arguments to support this statement likely centre on the new GPU’s significant FP64 support, large memory capacity, and incredible I/O bandwidth (which lends itself well to large rack-scale deployments), both of which the gaming-focused RTX-series GPUs lacks. The majority of this information is from the official slide deck here.

Micron secures first SOCAMM memory module supply for Nvidia GB300s

The market has been driven towards some fundamental changes in server PCB design in the past few years, one of them being the modularisation of various error-prone devices. SOCAMM, or the Small Outline Compression Attached Memory Module, is now quickly becoming the standard for replaceable, power-efficient memory on high-performing servers.

Conventionally, server memory is attached to the baseboard using DIMMs, which are designed to be easily replaceable in case of failure. This form factor does limit the performance of the memory though and so for high-bandwidth low-latency memory devices, soldering to the baseboard itself allows for the required performance without developing new module standards for each variation. Recently however, the industry has begun to move towards the SOCAMM design, allowing for a high-performance module that can be swapped out easily in case of device failure, avoiding requiring the entire baseboard to be replaced.

The Nvidia GB300 baseboards due in 2H25 will now include Micron’s SOCAMM modules for attaching up to 512GB of LPDDR5X to each Grace CPU on a 2 CPU + 4 GPU baseboard (codenamed the “Bianca” variant for the GB200s), with the CPU-memory aggregate bandwidth reportedly being ~400GB/s bidirectional, according to Nvidia’s official specs for the GB300 NVL72. This implies a distribution of ~100GB/s per module, each module including four stacks of 16 memory dies produced on Micron’s cutting edge 1b (beta) 10nm DRAM process. Micron state that their SOCAMM technology offers up to 2.5x the bandwidth of DIMMs in an equivalent capacity and configuration and reduces the power consumption by up to ~66%, though comparative benchmarks for GB300 superchips against GB200/GH200 variants (which used traditional soldered-on LPDDR5X) are not yet available.

Compared to the other two major memory manufacturers, Micron appear to be leading in DRAM process nodes as well as the overall power efficiency of their DRAM products, including HBM3E. Whether this lead persists into HBM4 is unclear as timelines between the three manufacturers for producing samples now differ by multiple quarters. SK-Hynix’s is reportedly still in the sampling process for their own modules, and Samsung are actively collaborating with Nvidia on prototyping this technology but have not announced any plans for their own SOCAMM products.

Other notable headlines

• Huawei’s Ascend 910C preparing for launch as mass production is now underway – specs coming to light as it appears to be two 910Bs side by side – Nvidia’s Blackwell tactics re-used

• SemiAnalysis develops the CloudMAX rating system for neoclouds, with a clear winner taking 1^st place, leaving all the hyperscalers behind

• Neocloud CoreWeave signs a $12B deal with OpenAI for supplying GPUs over the next 5 years – raises doubts on OpenAI’s partnership with Microsoft and Oracle

• Nvidia confirmed to supply ~64,000 B200 GPUs (in GB200 units) for project Stargate – the U.S.’ $100B set of datacentres in Texas

• CoolIT unveils a 4000W cold plate, with a flow rate of 6 LPM, doubling previous limits and preparing for the AI accelerators of 2027 and beyond

• SK-Hynix ships first HBM4 samples to key customers, and reportedly secures exclusive supplier rights for 12hi HBM3E for Nvidia GB300s

• Apple to spend $1B on Nvidia GB300 NVL72 racks, using SuperMicro and Dell - estimated ~250 servers, or ~18,000 GPUs

Vendor spotlight:

Cerebras

In a semiconductor fabrication unit, the manufacturer partitions a large silicon wafer into lots of small chips and each of those then goes on to be integrated into a device such as a CPU or GPU. Cerebras instead decide that they will use the entire wafer and keep it in one piece, for an enormous, monolithic chip called the wafer-scale engine (WSE).

Conventionally, a chip designer would never consider using an entire silicon wafer as a single device, let alone exceeding the size limits of what can be fabricated without dropping yield rates. If a designer decides to pursue this regardless, then there are two significant issues to overcome: managing the impact of the inevitable manufacturing defects and designing the I/O software and hardware for connecting the wafer efficiently. Cerebras tackles both with their various innovative approaches and then packages their wafer-scale engine (WSE) in their “CS” systems.

Having defects during manufacturing that can cause a few cores to be written off are expected, and Cerebras works around these issues with its reconfigurable architecture that modifies the core-core interconnect paths dynamically around deactivated cores. In addition, connections between the die themselves are specially designed in collaboration with TSMC to enable cross scribe line communication, scribe lines being where the silicon is supposed to be physically cut. These technologies, along with many other advancements, enable the WSE to offer industry leading single-server specs.

Here is what we know about the latest generation, the WSE-3 as revealed in 1Q24 (1 2 3 4):

• A total of 125 PFLOPS of dense FP16 (most likely) at a TDP of ~23kW

• 44GB of “on-chip” SRAM, delivering an aggregate of 21PB/s of bandwidth from memory to the compute

• 214Pb/s aggregate bandwidth between all cores on the wafer using fabric routers and cross-scribe wires

Cerebras currently operate 3 datacentres with their 2^nd and 3^rd gen. CS-2 and CS-3 systems, and have confirmed an additional 6 across NA and EU by the end of 4Q25. Private sector customers include AstraZeneca, Total energies, Tokyo electron devices and more across NA and APAC. They also have significant presence in public sector research and academia, with Argonne, Livermore, and Sandia among the largest names in national labs currently confirmed to be trialling or purchasing CS-2 systems.

One-pagers:

Silicon binning

During the complex, multi-stage semiconductor manufacturing process, production errors can manifest in the final product as reduced clock rates, voltage instabilities, or even dead logic circuits. The process of sorting chips (that are intended to be identical) by their error rates results in a wide variety of functionalities and performances.

Semiconductor manufacturing processes - particularly for more advanced and performant process nodes such as what modern gaming and AI accelerators require - are prone to a large variety of errors such as contamination from dust, misalignment between various layers, or etching mistakes when shaping structures on silicon. Whilst these minor errors usually don’t write out the entire chip, they can manifest in a variety of issues which affect the performance of the final product due to requiring some sections to be deactivated and presenting as dead silicon. For example, a GPU that was designed to have 10 active cores at 1.5GHz could only have 8 at 1.4GHz after testing. Since chips are produced on large circular wafers, with up to 100s of chips on a single wafer, these errors can be distributed non-randomly across space resulting in distinct categories or bins that these chips can be put into based on how many defects they have.

One outcome from this silicon binning process is that manufacturers can submit orders for the same chip design but then filter the end products into a variety of bins differing by performance to then sell them in various tiers. An example is Nvidia’s GA102/104 chips intended for their RTX 30XX gaming GPUs, with the lower end RTX 3080Ti and the higher end RTX 3090Ti sharing the same chip, but the former having an active core count of only 80 and a maximum boosted clock rate of 1.67GHz, compared to 84 and 1.86GHz for the latter. The differences in these values are great enough to lead to these nearly identical chips being packaged into two distinct products.

Even within bins, chip performance can vary greatly based on what manufacturers can’t or don’t control, such as clock rates, thermal performance differences, or transistor switching speeds. This results in a “silicon lottery” where despite buying the same equipment and providing identical running conditions, some end-users may report significantly higher performance from the same device, indicating that specifications provided by manufacturers may not be exact values but rather averages or minimums.

Branch prediction

Modern CPUs have been optimised to process multiple instructions simultaneously, billions of times per second, to allow for the performance that we expect. However, conditional instructions require choosing from diverging instruction streams which makes optimisation difficult. In these situations, CPUs employ specialised hardware for predicting the correct path.

To perform computations on data, hardware requires streams of instructions to inform it what to do with the data, and where to put the results. For most instructions, the result of the defined computation has no effect on the upcoming instruction stream, but in some cases the required instruction stream may change entirely depending on the result. This leads to the existing instruction stream and any intermediate computations already performed to be discarded and the processor state having to be traced back to when the instruction stream diverged or branched off.

For some instructions such as conditionals or returns, if the processor cannot anticipate the result of the computation before it’s complete, then it must either wait – wasting a lot of time and slowing down the task – or assume either outcome has a 50% chance of occurring and therefore choose one randomly to execute – Which could result in a misprediction, and a lot of time wasted again. CPUs solve this issue by using specialised hardware called branch predictors, designed solely to understand the instruction stream and predict where the stream might diverge to next, giving processors a higher probability of saving time for complex tasks.

Structures such as 2-bit saturating counters as shown in the diagram are among the simplest of these predictors, where they flip between predicting that a branch should be taken or not taken depending on the counters state which changes based on how many times they have correctly or wrongly predicted the outcome. Moden CPUs use multiple and more complex predictors to make more informed decisions, with structures such as TAGE or neural predictors that consider the history of the instruction stream, its origin, and more to make informed predictions.

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)

This month’s updates:

• SanDisk develops “high-bandwidth flash” memory, aiming for TB capacity on GPUs

• Meta reportedly looks to acquire Korean chip startup FuriosaAI

• Updates on the Nvidia B300 “Ultra” and Rubin series

• Intel’s roadmap and its dependence on 18A

• Huawei to challenge Nvidia’s dominance in China with the Ascend series

• Other notable headlines

Vendor spotlight:

• NextSillicon

One-pagers:

• Racetrack memory

• Baseboard management controllers

This month's updates:

SanDisk develops high-bandwidth flash (HBF) memory

High bandwidth memory (HBM) is one of the most important components of AI accelerators, enabling the massive amounts of rapid memory transfer needed for AI training workloads, but this performance comes at cost: DRAM - which makes up HBM - is expensive and takes up a lot of area per unit of storage. SanDisk, however, might just have solved this with HBF.

Regular DRAM struggles to saturate the bandwidth demand from AI accelerator compute elements, necessitating the development of technologies like LPDDR/X (low-power DDR/enhanced), GDDR (Graphics DDR), and HBM (High bandwidth memory). While implementation details vary greatly, there is one major common foundation they all share: DRAM. By design, DRAM strikes a balance between power use, density, and cost, leading to it being the best solution for on-device memory that lives just outside the processor but still closer than any larger capacity storage.

SanDisk, having just completed their spin-off from Western Digital, have announced a possible solution that could combine the large capacities of flash storage memories such as SSDs with the bandwidth that HBM provides to the processor - HBF. At their annual investors' day presentations, SanDisk revealed how they’ve prototyped a vertical stacking technology (just as HBM does with DRAM) to combine multiple flash memory dies into a single, high-throughput design that can meet the TB/s bandwidth requirements from HBM devices whilst providing the hundreds of GB of capacity enabled by higher density flash memory.

Whilst they admit that latency will still be worse than DRAM-based memory, SanDisk show how AI hardware designers could combine HBF and HBM to have smaller volumes of fast access memory acting as another level of cache, and the higher capacity HBF acting as a higher level of warm storage that can keep data on device - avoiding the 10-100x relative cost of fetching data from the host server. The first generation of HBF is expected in 2H25 providing up to 500GB of capacity per stack, with the second (2026) and third (2027) generations increasing capacity further up to 1TB per stack and attempting to keep up with the bandwidth we can expect from HBM4.

Meta to acquire Furiosa

South Korean chip designer Furiosa has gained recognition recently for its innovative and efficient RNGD (pronounced “renegade”) tensor-contracting processor, aiming to compete with Nvidia in AI training and inferencing. Now, analysts report that Meta is already in late-stage discussions to acquire the startup and add to its own custom silicon roadmap.

Industry analysts report that the acquisition could be completed as early as by the end of this month, but looking at clues in papers released on the technology Furiosa implement, it’s possible that Meta (then Facebook) were collaborating with the startup’s technical team as far back as 2019. The buyout would likely lead to Meta mixing their development teams with Furiosa’s and incorporating future products into its existing custom silicon lineup, as Meta presently deploys two generations of their “Meta Training and Inference Accelerators” - MTIAs - for their ranking and recommendation systems that run their services on apps such as Facebook and Instagram. From a 2024 leaked internal memo, it appears that Meta already had plans for the next generation of their AI accelerator in 2025. Whether this plan included the merging in of Furiosa’s roadmap is unclear.

Furiosa released their second processor, the RNGD, in 2024 - a PCIe form factor server chip targeting LLM training and inference providing ~1.6-2x the efficiency of Nvidia H100s on inferencing models in the 6-8B parameter range. The success of their accelerators is down to two key innovations: a tensor contraction compiler (1 2), and a tensor contraction processor. Tensor contraction is a higher dimensional generalisation of matrix multiplication, and the compiler’s success comes down to optimising memory layout and caching for better read/write patterns. All of this is then enabled by the unique processor design which performs dot-product operations on variable data shapes. All this results in a CGRA-like architecture designed for variable tensor input sizes, more so than even Google’s systolic array based TPUs.

As for its specs (1 2 3 4):

• Using TSMCs 5nm process node

• PCIe dual-slot FH 3/4L form factor

• 256MB on chip SRAM (384TB/s)

• 256TFLOPS at BF16

• 150W TDP air cooled

• 2 x 24GB 12-hi HBM3 (likely SK-Hynix)

• 48GB of 12hi HBM3 (1.5TB/s)

• 1GHz TCP frequency

• Supermicro server integration

• $10,000 (analyst estimate)

GB300 “Ultra” and Rubin series

As Nvidia’s B200 rack-scale SKUs start coming online on cloud platforms, more leaks and unofficial reports begin to surface on the next two generations of AI GPUs. The B300, successor to the B200 and previously known as the “Blackwell Ultra”, and the upcoming architecture named “Rubin” have made headlines for changes to their specs and release dates.

Though rumours circulated as early as 2Q24, it’s now been confirmed that Nvidia will reveal the B300 at their annual GTC conference in March. The B300 is reported to launch in 3Q25 for sampling and then go into mass production as early as 4Q25, using 288GB of SK-Hynix’s 12hi HMB3E on a TDP of 1200W. For the scale-out networking, the B300 and the Grace CPU-enabled GB300 trays will come with ConnectX-8 NICs capable of 800Gb/s optical connections over a single OSFP or dual-QSFP112 ports. It should be noted that CX8 NICs will be both SpectrumX (Ethernet) and QuantumX (InfiniBand) capable, removing the need for expensive and power-hungry BlueField superNICs where customers required Ethernet E/W networks. Just as SemiAnalysis reported on Christmas day, the G/B300s will still be sold as a rack in the NVL72 form-factor. Finally, Nvidia recognised the logistical bottleneck caused by the SXM baseboard design and evidence confirms that they are switching to a socketed design (1 2) using removable “SXM pucks”, enabling other OEMs and integrators aside from Wistron and Foxconn to manufacture and maintain the baseboards.

As for Rubin, the next architecture in Nvidia’s roadmap after the Blackwell series, sources now state that it may launch for sampling as early as 2H25 because of memory maker SK-Hynix’s rapid progress with their sixth-generation HBM4 memory technology which will enable the Rubin series GPUs. How this accelerated release date impacts orders for Nvidia’s G/B200 and B300 rack-scale SKUs, and whether a whole generation of networking technology will be skipped is yet unclear. Further, it appears certain now that Nvidia will announce a 4x increase in GPU/rack density with the Rubin NVL288 (1 2), a single, possibly 1 megawatt rack-scale SKU using an orthogonal backplane (fixed socket-like connectors) for power delivery and switching as well as direct-to-chip liquid cooling.

Intel’s roadmap and its dependence on 18A

Intel’s roadmap on AI accelerators has changed so frequently in recent years that some large customers interested in the Gaudi series decided to stick with vendors able to provide more maintainable and upgradeable product, but recent successes with their 18A process node and PC/server CPUs might finally turn things around.

After recent rumours of possible selloffs of parts of the business to market rivals, it appears Intel is now pinning its future (or at least its independence) on the success of the 18A (1.8nm) process node - Intel Foundry Service’s (IFS) “greatest transistor innovation in a decade”. Its success relies on two key advancements: the RibbonFET (field effect transistor) and BSPD (back-side power delivery). The new RibbonFET design, based on the same fundamental architecture as arch-rival TSMC’s gate-all-around (GAA), brought significant performance gains compared to the “Intel 3” 3nm and now aims to match the capability of TSMC’s “N2” 2nm. In addition, BSPD uses the back (or underside) of the silicon wafer for power delivery, leaving the front (or top) for logic. This results in better utilisation of chip area due to not having to mix power and transistor elements on one surface, as well as lower power leakage due to wires being able to be thicker and shorter. For memory, both TSMC and Intel now offer ~38 Mbit/mm^2 of memory density in SRAM - a type of memory which is used almost everywhere on chip, in registers, buffers, caches, and more.

Despite the technological advancements, it seems that there are still significant challenges ahead until 18A earns Intel some semiconductor manufacturing market share from customers using TSMC and Samsung. Due to release in 2H25, Intel’s Panther Lake desktop CPU has been subject to industry leakers who report on both the yield rate of the processor manufacturing process, and the performance of the processors cores. One source states that rate is as low as 20-30%, a number at which mass production would be incredibly difficult if not impossible. There have been no statements yet on whether Panther Lake may be delayed or cancelled, and samples have already been sent out to select customers. Another source claims that the performance of the CPU’s “P” or performance cores (optimised for high throughput rather than energy efficiency) is “average”, indicating that the numbers from whatever benchmarks were run might not be competitive with other similar products on the market. These leaks appear to have some merit however, as rumours are already circulating hinting at the ~2026 Nova Lake (successor to Panther Lake) using a mixture of Intel’s planned 14A process node and TSMC’s N2.

Huawei’s Ascend series

Chinese semiconductor manufacturer SMIC, once struggling with 7nm, is now likely testing its 5nm process with possible yield rates between 30-70% on Huawei’s yet to be announced Ascend 920 or 910D AI GPU. If the improvement on the 910C is large enough, it’s possible Huawei may capture a significant share of the Chinese market from Nvidia.

Chinese chip makers appear to be catching up with TSMC/Intel with rumours (1 2 3 4) of SMICs 5nm process node now showing acceptable yield rates of somewhere between ~30-70%. The two U.S. CHIPS acts implemented in 2021 and 2022 prevented Nvidia from selling its AI hardware and ASML from selling chip manufacturing equipment to China, but instead of stalling progress as intended, the legislation may have instead resulted in accelerating the nation’s independence in the semiconductor space. Various tech giants are already running workloads on the original 910 and the second generation 910B (the latest publicly available AI GPU from Huawei) with names such as ByteDance, Baidu, and most recently even DeepSeek likely using large clusters of both 910/910B servers for their inferencing workloads. In terms of packaging and integration, Huawei presents their AI GPUs in the typical 8 GPU configuration, with networking topologies within servers being a 2x4 mesh, and between servers being a fat-tree topology, differing significantly from rivals such as Nvidia and AMD.

The 920 (or 920C, or 910D, to be confirmed), the successor to the still only privately tested 910C, is rumoured to be available to some customers for sampling in 3Q25 and ready for mass production in ~2026. Sources suggest that Huawei will use SMIC’s 5nm process for the 920 depending on how it performs in making their Kirin mobile chip. As for the 920’s specs, it’s impossible to even speculate given that most of the information on the 910C is still unofficial. Leakers and independent analysts suggest various numbers from the 910C having between 96 and 128 GB of HBM, to having either 256 or 512 TFLOPS of FP16 compute, but the reliability of the sources varies greatly. Despite the impressive performance of the 910C, Nvidia’s sanction-busting H20 GPUs (the H100, but with less compute and slightly better memory) have seen major orders placed in 1Q25, after the Trump administration hinted at further iterations of the CHIPS act.

Other notable headlines

• Samsung reportedly redesigns its 1c DRAM process node to tackle low-yield issues, likely preparing for HBM4 in 2H25

• Nvidia working with all major memory makers on new “SOCAMM” memory for AI PCs, based on LPDDR5X

• Samsung to launch new low-power, low-latency, wide I/O DRAM (LPW/LLW DRAM) in 2028, aiming at providing HBM to mobile devices

• Japanese AI lab Sakana.ai releases AI CUDA engineer, a system for writing optimised code to improve GPU performance – would help develop even better AI models

• Microsoft releases the Majorana 1 – an 8-qubit quantum computer using a new state of matter to perform computations, enters the quantum race with Google and IBM

• SK-Hynix achieves 70% yield in 12hi HBM4 ahead of mass production, possibly leading to Nvidia accelerating their own timeline for the Rubin series

• Samsung to use Chinese YMTCs technology for 400+ layer NAND flash SSDs

Vendor spotlight:

NextSilicon

One of the few chip designers aiming at scientific compute, NextSilicon develops “intelligent compute accelerators” (ICAs) that optimise for natively running C++ and Fortran workloads at efficiencies and throughputs/latencies unmatched by CPUs and GPUs. Their “Maverick” ICA now looks to prove itself and break into the datacentre server space.

Founded in 2017, Israeli startup NextSilicon differentiates itself from the competition in two key ways: the target market they pursue, and their requirement to support existing target workloads. Their products aren’t aimed at accelerating AI but instead at regular scientific compute workloads that have been - and still are - the biggest consumers of compute in the world. Applications that would benefit from NextSilicon’s designs include weather forecasting, finite element analysis, and graph algorithms, all of which rely on structured and repeatable computations on data but with less predictable memory access patterns and more varied control flows. Almost all such applications are written and maintained in two programming languages, C++ and Fortran, both in use for decades for building highly performant programs that run much of the world’s software today. NextSilicon aim to keep it that way by ensuring their “software-defined hardware” runs such applications natively, with no need for porting to another language or writing intermediate representations, which is what vendors like Nvidia, AMD, and Intel require when using their specialised accelerators.

Their novel processor, the Maverick, now in its second generation, introduces software-defined hardware acceleration by taking scientific applications in their native forms (just as they would run on CPUs) and learning what the software requires from hardware to run optimally. It does this by running them on its own dynamically reconfigurable grid of general purpose “E-cores” and math-specialised “compute elements” and building graphs of the various paths data takes as it moves from memory, through the compute, and back to memory. It then reconfigures itself on the fly to best suit those computational graphs and accelerates application performance by up to 4x when compared to GPUs and 20x compared to high-end CPUs. As for specs, the Maverick 2 in its 300W-TDP PCIe form factor provides up to 32 E-cores and 224 compute elements, along with four stacks of HBM3E, totalling at 96GB. The OAM version supports two compute die and up to 192GB of HBM3E but has a TDP of 600W, requiring liquid cooling. Both forms are built on TSMCs 5nm process.

One-pagers:

Racetrack memory (RTM)

Device memory is usually stored as electric charge across a capacitor or as a state of a set of transistors, however magnetism provides a new way of complementing or replacing SSDs or DRAM with a low latency/jitter, energy efficient, and non-wearing solution: RTM, which stores information between magnetic domains in nanowires shows promise.

Originally developed by researchers at IBM and now in the experimental stage with various other institutions, RTM aims to combine the high performance and reliability of solid-state memory with the cost-effectiveness and density of traditional magnetic storage. RTM, unlike conventional memory which stores information in transistors and capacitors, uses “racetracks” or nano wires that hold multiple magnetic regions separated by thin domain walls or gaps. These domain walls represent the information to be stored, and information is moved from storage to read/write heads by shifting the magnetic domains up and down the wire using electrical currents.

The shifting action, as well as how quick and efficient it is to read and write information to the wires, leads to the potential for RTM to become part of future memory devices. Additionally, unlike flash storage which requires distinguishing charge levels in a capacitor that wears out over time, racetrack nanowires do not degrade with use. The use case and capabilities of RTM-based memory depends on its structure and the design of the device, just as with flash and DRAM based technologies. Whether the wires are arranged in two- or three-dimensional arrays, how many read/write heads are used to get and store information in parallel, or how error correction is optimised for shift-based memory movement all influence where and how RTM can be used and what advantages/disadvantages it presents.

AI workloads - now the fastest growing use case for high-performance hardware - could potentially see significant improvements in throughput/latency and flops/watt with processors using RTM. With compilers that can breakdown high-dimensional matrix operations (most of what AI models do internally) into instructions suited to reading and writing from arrays of racetracks, hardware that utilises RTM to compliment or replace DRAM-based memory could theoretically lead to large improvements in performance. Currently, IBM appears to be the industry leader in RTM research, and many universities experimenting independently on their own prototypes.

Baseboard management controllers (BMCs)

A baseboard (or motherboard) hosts all the components of a server and connects them via copper wires or sockets and cables. The microcontroller which manages much of the state and functionality of the board is known as the baseboard management controller (BMC) and is one of the most important yet often overlooked parts of a server.

BMCs are dedicated chips on the baseboard that perform a variety of functions crucial to managing a server. Primarily, they enable out-of-band (OOB) management through their own dedicated network interfaces, allowing administrators to remotely power-cycle servers, update BIOS/UEFI firmware, or diagnose failures even when the host operating systems are unresponsive. Modern BMCs are usually built on ARM or RISC-V architectures - even in x86 servers - but operate independently from the CPUs and draw power through their own separate rails. For temperature and power management they interface with thermal sensors, voltage regulators, and fans/other cooling systems, and so can modify power supply/frequency to components to enforce programmable temperature limits. To interact with a BMC directly, most manufacturers provide APIs to expose key functions. The industry standard that has emerged over the past decade is Redfish, which provides ways to interact with the BMC through POST/GET calls, Python, and Ansible to name a few.

As servers tend towards hundreds of cores per CPU, multiple GPUs, and additional NICs/switches all on the same board, the demand on the BMCs increases super-linearly. Individual components become denser and more performant with every iteration, but the number of sensors and the frequency of incoming data also increases, resulting in an even greater load on the management controller and connections. This trend towards increased networking and processing load has resulted in BMCs having a higher core count and using high-bandwidth connectivity such as 400G NICs and PCIe 5.0 to support the data collection and processing required for managing increasingly complex baseboards.

Some BMCs implement advanced predictive maintenance via small AI models, to help prevent damage and wear on components, and incorporate specialised units such as NPUs to offload these small AI workloads. It’s also likely that BMCs will eventually integrate more and higher bandwidth memory such as HBM stacks to cache sensor data, enabling better management functionality on chip rather than sending data to remote management servers. This shift mirrors industry’s push toward “lights-out” datacentres, where fewer and fewer hardware faults require human intervention.

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)

This month’s updates:

• DeepSeek’s hardware inventory possibly much greater than reported

• Samsung and Micron still competing for HBM, fighting for Nvidia’s GPUs

• Nvidia’s GB200: Mixed sentiment from industry on persistent overheating reports

• Qualcomm and Arm to compete again: The datacentre Arm-based CPU market

• The incredible engineering of the world's best gaming GPU – the Nvidia RTX 5090

Vendor spotlight:

• GigaIO

One-pagers:

• Overclocking

• Intel AVX/AMX

This month's updates:

DeepSeek possibly under-reports AI hardware infrastructure

The AI model that wiped an estimated $1.5 Trillion in value off global stocks was not an automated trading or banking service gone wrong, but just a free to use - and very capable - chatbot. DeepSeek’s R1 was reportedly trained on just 2048 H800 GPUs, but some sources claim that significantly more compute was used.

Though the paper claims that only 2048 Nvidia H800 GPUs and 5.6M USD were used during model training, speculation and rumours from industry have brought the legitimacy of this statement into doubt. DeepSeek (owned by Chinese hedge fund “High-flyer”) open-sourced their free-to-use “R1” model and shocked the market by quickly reaching second place on various public leaderboards and benchmarks, just short of OpenAI’s o1. What’s more, the model is significantly smaller and cheaper to run than its American/European counterparts and is freely available for anyone to build upon and run locally. But many sources claim it’s likely that DeepSeek used a lot more compute than they report.

Huida (Nvidia’s trading name in the Chinese market) designed the H800 AI accelerator in response to the 2022 US chips act, with it being essentially a reduced bandwidth H100 tailored to avoid export restrictions and meet the high demand for AI training compute from restricted countries. Both accelerators are almost equivalent except for the H800 supporting only 8 x 50GB/s NVLink channels instead of the 18 available on H100s. ScaleAI CEO Alexandr Wang states that industry rumours hint at DeepSeek owning over 50,000 H100 GPUs, likely acquired through Singapore to circumvent trade restrictions, and SemiAnalysis stating that its more likely about 10,000 each of the H800 and H100s, with 30,000 H20s

Based on their “Fire-flyer” paper, it’s evident that DeepSeek staff have extensive experience in optimising and maintaining infrastructure containing thousands of Nvidia GPUs with InfiniBand fabrics. The paper outlines how they designed and benchmarked a cluster of ~10,000 A100 GPUs, significantly deviating from Nvidia reference architecture for large cost reductions with relatively small performance drops. Most notably, their use of Connecting GPUs and IB NICs to CPUs via PCIe ports directly with no switching and developing their own collective communications library “HFReduce” which outperforms NCLL on their P2P NVLink setup, are both unique and very resourceful initiatives.

Samsung and Micron still in the race for HBM

Last quarter, SK-Hynix and Nvidia both confirmed plans for 6^th gen. HBM4 with the memory manufacturer agreeing to meet the 1H2025 deadline for sampling their devices on Nvidia’s “Rubin” architectures. Now, Samsung and Micron both formally announce that they too are still competing for the future of this market.

In mid-4Q24, Nvidia requested SK-Hynix to accelerate their development of HBM4 devices, preparing samples for testing and validation on Nvidia’s “Rubin” prototypes due in 2Q25. This resulted in them bringing forward their roadmap by 6 months, setting the date for mass production to the end of 3Q25, a timeline that originally had not been shared by their competitors.

Now, to regain their shrinking share of the HBM market, Samsung announced that they too will be releasing HBM4 samples for the end of 2Q25, likely to compete for the same trials on Nvidia hardware. It’s unclear whether Samsung will meet this deadline due to recent heat/power management issues with their (4^th and 5^th gen.) implementations, with Nvidia repeatedly failing Samsung’s attempts at passing validation. Their specific HBM4 designs also reportedly include using world-leading 1c 10nm DRAM dies, with logic dies being incorporated into the device for multiple, smarter buffers and providing some compute-in-memory capabilities. On top of this, Samsung also have declared intent to develop custom HBM4 devices for chip manufacturers such as Microsoft and Meta who are likely to be in the market for memory with narrower bus widths and higher overall density.

In addition, Micron, the smallest of the 3 major memory manufacturers has announced that they are conducting final tests for their implementation of high-density 16hi HBM3e, with mass production due before 2026. Despite being behind SK-Hynix in 16hi, Micron might well have a better chance than Samsung of increasing their HBM market share over the next year due to them famously winning the Nvidia H200 contract for 8hi HBM3e and hence being able to make significant investments in expanding their production capacity over the coming years. Unlike the other two, Micron keeps its HBM4 mass production plans to 2026/27.

Mixed market sentiment on Nvidia’s GB200 rack-scale SKUs

Nvidia’s B200 GPUs might possibly be at the centre of another overheating issue, this time stemming from the extreme density of their rack-scale SKUs, the NVL36/72s. Taiwanese suppliers such as Foxconn, Winstron, and QCT amongst the largest, claim that these rumours are incorrect and that orders are being fulfilled as expected.

From as far back as mid-3Q24, rumours have been circulating of Nvidia’s “Blackwell” rack SKUs experiencing overheating issues, causing issues for major customers such as Microsoft and Amazon. Recently, these reports have resurfaced as reductions in order volumes and an increase in demand for older “Hopper” H1/200 GPUs have been confirmed, with OpenAI reportedly asking Microsoft to provide more capacity on H100s to make up for the delays.

With B200s having a TDP of up to 1000W and Grace CPUs designed for 500W, a single 1RU GB200 tray with 2 Grace and 4 Blackwell chips is estimated to have a maximum power draw of ~6300W under sustained high utilisation, requiring advanced direct liquid cooling solutions that very few datacentres in the world have the infrastructure for. In total, an NVL72 rack holds 18 such compute trays as well as 9x1RU NVswitch trays for the GPU-GPU data fabric, all connected with active copper cabling. Whilst a variety of OEMs have now released their own implementations of the liquid cooled GB200 NVL72, it’s likely that testing outside of curated lab environments has been rare.

Taiwanese manufacturers and integrators such as Foxconn, Winstron, and QCT all deny these rumours, reporting that the current shipments are on schedule and are not affected by overheating issues. Instead, analysts cite cost vs benefit calculations compared to the tried and tested H1/200 HGX servers that are widely available to many buyers, as well as the timeline for Nvidia’s B300 (previously named the B200 Ultra) and “Rubin” series GPUs being close enough to affect ordering times for hyperscalers.

Qualcomm and Arm to compete over datacentre Arm CPUs

Another established mobile chipmaker now moving into the datacentre CPU space, Qualcomm has been hiring for server design roles recently. At the same time Arm may be preparing for an acquisition of the high-performing Arm CPU designer, Ampere.

With Nvidia’s Grace chips having shown success in the Arm server CPU market - albeit likely due to them being packaged in the GB200 rack-scale SKUs - and Apple showing signs of entering the same space with names such as Broadcom and Foxconn, it was always expected that other semiconductor design teams would follow suit. Now, both Qualcomm and Arm appear to be taking significant actions towards this, with aggressive hiring and discussions of acquisitions.

Qualcomm, having previously forayed into this space in 2017 with their Centriq 2400 series, couldn’t gain significant traction in the market due to the dominance of x86-based competitors and the difficulty of porting many applications to Arm architectures. Now, preferences and capabilities in industry have shifted towards energy efficiency, evidenced by a movement towards names such as Nvidia’s grace, Ampere’s Altra, and AWS’s Graviton to name a few. To confirm their intent, Qualcomm have hired Intel’s server CPU lead and have been openly looking for other CPU and server design roles.

At the same time, Arm have reportedly entered discussions along with SoftBank on acquiring Ampere, the market leader now in high performance, high core-count ARM server CPU. Until now, Arm has kept to developing and licensing semiconductor IP such as their individual cores for mobile and server applications (the Cortex and Neoverse series) which have seen success in processors made by Nvidia, AWS and Google to name a few. Ampere, on the other hand develop very high-core count processors such as their 96-192 core “AmpereOne” M and MX series and their upcoming “Aurora” 512-core SKU scheduled for 2025. Some of the concerns that regulators might have include possible conflicts between existing licensing agreements, Oracle’s 29% share in Ampere, and Ampere’s own plans for an IPO which are still underway.

The incredible engineering of Nvidia’s RTX 5090

Nvidia showcases their expertise in the gaming GPU space with their latest series of RTX graphics card built on their new “Blackwell” architecture. For the 5090 - the most performant GPU in the lineup - early samples have already been taken apart and reverse engineered, revealing just how unique its engineering is.

At their CES keynote earlier this month, Nvidia officially revealed the entire RTX 50XX series, though samples had likely been sent to various testing and review groups as far back as December. Among the many reviewers, the Der8auer group - who received a special “Founders edition” (FE) version of the 5090 – commented on how each of the 58 cooling fins are uniquely shaped, resulting in just two fans able to manage a device with a TDP of 575W in a FHFL PCIe form factor.

Its counterpart in the current generation, the RTX 4090 as released in 2022, was notably difficult to acquire due to cryptocurrency miners and low-tier AI datacentres using it as a cheaper and more efficient option than datacentre grade devices which are priced for the enterprise and require warranties and licensing among other things. The RTX 5090 reference design (RD) which will be sold as the standard version is similarly expected to be extremely difficult to buy due to incredible demand.

As for its specs (1 2 3 4):

• Using TSMCs 4nm process node

• 92.2B transistors in total within 744mm²

• 21,760 CUDA cores

• Clocking up to 2.4GHz (2.54 for the FE)

• 32GB of GDDR7 (512-bit bus width)

• 1.79TB/s memory bandwidth

• Air-cooled TDP of 575W

• PCIe 5.0 x16 in a FHFL form factor

• 680 5^th gen. Tensor cores

• Starting at ~$2000 (estimated)

Vendor spotlight:

GigaIO

A relatively small American OEM, GigaIO decided to forgo integrating network fabrics tightly with compute and memory and instead decided to extend PCIe to outside the chassis. With their incredibly underrated “FabreX” tech stack, they’re able to connect up to 64 nodes and present them as one “giga node” to software.

Optimised for edge inferencing, the Gryf suitcase-sized supercomputer packs up to 2.5KW in a airplane form factor. Gryf comes with six field-replaceable sleds that can be customised between four categories - compute, AI accelerator, networking, and storage units. The sleds are all internally connected by a FabreX backend network as well as allowing for up to 100GbE scale out, and multiple Gryf units can be connected using FabreX as a scale-up fabric. Currently, the compute sleds support 64 core AMD Milan CPUs, and the AI accelerators that fit the size and power envelope are limited to Nvidia’s L40S, though it appears other FHFL options are being qualified currently. Storage sleds have a capacity of up to 246TB, leading to well over a PB of managed storage in a Gryf optimised to act as a GPFS server.

The FabreX networking stack essentially takes PCIe out from within the confines of a chassis and allows for composing a large amount of compute resources over many physical servers into one giant virtual machine. Technology like this does already exist but suffers from a variety of limitations and performance issues due to the challenges involved in virtualising away many different communication protocols and networking stacks. With FabreX, everything is within the PCIe domain, and so using GigaIO’s FabreX NICs and switches, everything can appear as if it is on one motherboard. This allows for aggregating large amounts of compute resources such as up to 64 GPUs into one virtual server, or for dynamically changing the setup of a machine to vary the amount of memory or networking available. This has led to a variety of bold claims that GigaIO makes on the performance of their solutions.

One-pagers:

Overclocking

When power draw and cooling are no longer concerns, overclocking can offer a way to get more performance from the same hardware quickly, though the types of applications that can benefit from pushing CPUs and GPUs beyond their recommended bounds are limited.

All digital processors execute tasks by breaking them into “cycles”, synchronized to an internal clock - a high-frequency signal measured in mega/gigahertz (M/GHz). Each cycle allows the processor to perform operations like fetching data, executing instructions, or writing results. Modern CPUs use techniques such as pipelining to overlap these tasks across multiple cycles, but all operations are still tied to the clock’s rhythm. For example, a CPU might complete 1–4 instructions per cycle depending on its architecture. The clock rate - essentially the speed of this internal metronome - directly influences how quickly these operations occur. Higher clock speeds reduce latency for tasks like gaming or single-threaded applications, where rapid execution of individual instructions matters most.

However, increasing the clock rate (overclocking) comes with trade-offs. While it boosts performance, it also raises power consumption and heat output, with each 10% increase in clock speed typically raises power draw by 20-30%, straining cooling systems and risking thermal throttling.

Modern CPUs use dynamic frequency scaling to balance efficiency and performance, but overclocking bypasses these safeguards, prioritizing raw speed. This involves manually adjusting the CPU’s clock rate and voltage via BIOS settings or software tools, often used by gamers and enthusiasts to maximize frame rates or benchmark scores. Some AI workloads, like small-scale inference, may also benefit from faster serial computations, though parallel tasks (e.g., large-scale model training) depend more on core count and memory bandwidth.

CPUs typically offer a wide variation in allowable clock rates due to the inherent flexibility of their design and having to orchestrate the various components of a server such as memory and I/O, whereas GPUs and more application-specific processors tend to have a narrower acceptable range.

Intel’s AVX/AMX – Advanced Vector/Matrix eXtensions

Application performance on any processors depends not only on the inherent capability of the hardware, but also the software that runs on top of it. Intel’s AVX/AMX (advanced vector/matrix extensions) are instruction sets add-ons that optimise the use of Intel CPU/NPUs for modern data-heavy workloads.

Modern CPUs rely on instruction sets - collections of commands that define how hardware processes data - to execute tasks. AVX, introduced in 2011, expanded Intel’s x86 capabilities by adding 256-bit vector registers (YMM), enabling CPUs to perform operations on multiple data points, from 4 x FP64 to 32 x Int8 simultaneously. AVX accelerated tasks like image processing, physics simulations, and machine learning inference, where identical operations on large datasets benefit from parallel execution. Later, AVX-512 (2016) doubled register width to 512 bits (ZMM), further boosting throughput for scientific computing and AI training, though at higher power costs. However, with AI workloads demanding even higher arithmetic intensity, a newer instruction set with additional data structures was needed.

AMX, launched in 2023 with Intel’s Sapphire Rapids CPUs, targets matrix operations by introducing 2D tiles, which are dedicated on-chip memory blocks for matrix operations. Unlike AVX’s general-purpose registers, AMX tiles are optimized for matrix multiplications, a core operation in neural networks. Using tiles allow developers to further optimise processing very large regions of memory by exploiting locality and parallelism further. This design mirrors GPU-style efficiency for AI tasks while retaining CPU flexibility for mixed workloads. For example, a CPU with AMX can handle both database queries and real-time inference to a degree, making it ideal for edge AI or hybrid cloud environments.

AVX and AMX reflect Intel’s strategy to bridge the gap between CPUs and specialized accelerators. AVX’s vector parallelism suits diverse workloads, from video encoding to physics simulations, while AMX’s matrix focus competes directly with GPUs in AI efficiency. Though AVX-512’s power demands limit its adoption, AMX’s lower energy footprint positions it as a scalable AI solution suitable for high-bandwidth CPUs such as Sapphire rapids onwards.

(Want to see next month’s edition early? Follow this GitHub repo for the latest updates and early looks at the newsletter as it’s written - Also check out my LinkedIn!)