Planet Igalia

Andy Wingo: nominal types in webassembly

2026-03-10T08:19:34+00:00

Before the managed data types extension to WebAssembly was incorporated in the standard, there was a huge debate about type equality. The end result is that if you have two types in a Wasm module that look the same, like this:

(type $t (struct i32))
(type $u (struct i32))

Then they are for all intents and purposes equivalent. When a Wasm implementation loads up a module, it has to partition the module’s types into equivalence classes. When the Wasm program references a given type by name, as in (struct.get $t 0) which would get the first field of type $t, it maps $t to the equivalence class containing $t and $u. See the spec, for more details.

This is a form of structural type equality. Sometimes this is what you want. But not always! Sometimes you want nominal types, in which no type declaration is equivalent to any other. WebAssembly doesn’t have that, but it has something close: recursive type groups. In fact, the type declarations above are equivalent to these:

(rec (type $t (struct i32)))
(rec (type $u (struct i32)))

Which is to say, each type is in a group containing just itself. One thing that this allows is self-recursion, as in:

(type $succ (struct (ref null $succ)))

Here the struct’s field is itself a reference to a $succ struct, or null (because it’s ref null and not just ref).

To allow for mutual recursion between types, you put them in the same rec group, instead of each having its own:

(rec
 (type $t (struct i32))
 (type $u (struct i32)))

Between $t and $u we don’t have mutual recursion though, so why bother? Well rec groups have another role, which is that they are the unit of structural type equivalence. In this case, types $t and $u are not in the same equivalence class, because they are part of the same rec group. Again, see the spec.

Within a Wasm module, rec gives you an approximation of nominal typing. But what about between modules? Let’s imagine that $t carries important capabilities, and you don’t want another module to be able to forge those capabilities. In this case, rec is not enough: the other module could define an equivalent rec group, construct a $t, and pass it to our module; because of isorecursive type equality, this would work just fine. What to do?

cursèd nominal typing

I said before that Wasm doesn’t have nominal types. That was true in the past, but no more! The nominal typing proposal was incorporated in the standard last July. Its vocabulary is a bit odd, though. You have to define your data types with the tag keyword:

(tag $v (param $secret i32))

Syntactically, these data types are a bit odd: you have to declare fields using param instead of field and you don’t have to wrap the fields in struct.

They also omit some features relative to isorecursive structs, namely subtyping and mutability. However, sometimes subtyping is not necessary, and one can always assignment-convert mutable fields, wrapping them in mutable structs as needed.

To construct a nominally-typed value, the mechanics are somewhat involved; instead of (struct.new $t (i32.const 42)), you use throw:

(block $b (result (ref exn))
 (try_table
  (catch_all_ref $b)
  (throw $v (i32.const 42)))
 (unreachable))

Of course, as this is a new proposal, we don’t yet have precise type information on the Wasm side; the new instance instead is returned as the top type for nominally-typed values, exn.

To check if a value is a $v, you need to write a bit of code:

(func $is-v? (param $x (ref exn)) (result i32)
  (block $yep (result (ref exn))
   (block $nope
    (try_table
     (catch_ref $v $yep)
     (catch_all $nope)
     (throw_ref (local.get $x))))
   (return (i32.const 0)))
  (return (i32.const 1)))

Finally, field access is a bit odd; unlike structs which have struct.get, nominal types receive all their values via a catch handler.

(func $v-fields (param $x (ref exn)) (result i32)
  (try_table
   (catch $v 0)
   (throw_ref (local.get $x)))
  (unreachable))

Here, the 0 in the (catch $v 0) refers to the function call itself: all fields of $v get returned from the function call. In this case there’s only one, othewise a get-fields function would return multiple values. Happily, this accessor preserves type safety: if $x is not actually $v, an exception will be thrown.

Now, sometimes you want to be quite strict about your nominal type identities; in that case, just define your tag in a module and don’t export it. But if you want to enable composition in a principled way, not just subject to the randomness of whether another module happens to implement a type structurally the same as your own, the nominal typing proposal also gives a preview of type imports. The facility is direct: you simply export your tag from your module, and allow other modules to import it. Everything will work as expected!

fin

Friends, as I am sure is abundantly clear, this is a troll post :) It’s not wrong, though! All of the facilities for nominally-typed structs without subtyping or field mutability are present in the exception-handling proposal.

The context for this work was that I was updating Hoot to use the newer version of Wasm exception handling, instead of the pre-standardization version. It was a nice change, but as it introduces the exnref type, it does open the door to some funny shenanigans, and I find it hilarious that the committee has been hemming and hawwing about type imports for 7 years and then goes and ships it in this backward kind of way.

Next up, exception support in Wastrel, as soon as I can figure out where to allocate type tags for this new nominal typing facility. Onwards and upwards!

Yeunjoo Choi: Smarter Chromium GN in Vim with gn-language-server

2026-03-10T03:06:00+00:00

GN Language Server for Chromium development was announced on chromium-dev. It’s very easy to install in VSCode, NeoVim or Emacs. But how can we configure it with classic Vim + YCM?

Setup

First, install the language server with Cargo.

cargo install --locked gn-language-server

Then, add this to your vimrc.

let g:ycm_language_server = [
      \ {
      \   'name': 'gn',
      \   'cmdline': [ 'gn-language-server' ],
      \   'filetypes': [ 'gn' ],
      \ }
  \ ]

That easy, right?

What’s Working

Hover Documentation

Go To Imports

Go To Dependencies

Current Limitations

The following features are not working yet. They may need more configuration or further work:

Code Folding

Classic Vim and YCM don’t support LSP-based folding, and I’m not a big fan of that feature anyway. But you can configure another plugin that supports LSP-based folding, or simply rely on indent-based folding.

Go To Definition

When I try to go to the definition of template, I get an error KeyError: 'uri'. I’m not sure whether this is caused by my local configuration, but it needs further investigation.

Igalia WebKit Team: WebKit Igalia Periodical #59

2026-03-09T20:02:33+00:00

Update on what happened in WebKit in the week from March 2 to March 9.

As part of this week's handful of news, WebKitGTK and WPE WebKit now have support for Gamepad's "VibationActuator" property, the video decoding limit is now configurable at runtime in addition to build time, and an interesting fix that makes WebKit render fonts like other browsers by making it blend text incorrectly (!).

Cross-Port 🐱

Using libmanette's rumble support, enabled Gamepad VibrationActuator for WebKitGTK and WPE WebKit.

With these changes, playEffect() can be used to play dual-rumble vibration effects.

Multimedia 🎥

GStreamer-based multimedia support for WebKit, including (but not limited to) playback, capture, WebAudio, WebCodecs, and WebRTC.

VIDEO_DECODING_LIMIT is now configurable at runtime, in addition to build time. That will allow vendors that share a single binary build on different platforms to fine-tune their needs without a rebuild.

Graphics 🖼️

Landed a change that tweaks the text rendering done with Skia. With this change, the text looks more natural now - just like in other browsers. However, this is done by blending text incorrectly as a compromise.

Releases 📦️

One more set of release candidates for the upcoming stable branch, WebKitGTK 2.51.93 and WPE WebKit 2.51.93, have been published. For those interested in previewing the upcoming 2.52.x series this release is expected to be quite stable. Reporting issues in Bugzilla are, as usual, more than welcome.

That’s all for this week!

Tiago Vignatti: Accessibility and PDF documents

2026-03-04T13:00:00+00:00

Accessibility
#

When we think of accessibility, we tend to picture it as something designed for a small minority. The reality is much broader: 16% of the world’s population — 1.3 billion people — live with a significant disability¹. In Brazil alone, where I live, that means around 14.4 million people report some form of disability². And those numbers capture only permanent disabilities.

Igalia WebKit Team: WebKit Igalia Periodical #58

2026-03-02T20:11:00+00:00

Update on what happened in WebKit in the week from February 23 to March 2.

This installment of the periodical brings news about support for Qualcomm qtivdec2 and qtivenc2 on GStreamer, GPU texture atlas creation and replay substitution, enhancement of the scroll gesture in WPE, and two new releases: WebKitGTK 2.51.92 and WPE WebKit 2.51.92.

Cross-Port 🐱

Multimedia 🎥

GStreamer-based multimedia support for WebKit, including (but not limited to) playback, capture, WebAudio, WebCodecs, and WebRTC.

Work on adding support for the Qualcomm GStreamer qtivdec2 and qtivenc2 elements is on-going

Graphics 🖼️

Implemented GPU texture atlas creation and replay substitution in the Skia painting engine on GTK/WPE. After recording, raster images are packed into GPU atlases via BitmapTexture, with two upload paths: an optimized DMA-buf path that memory-maps GPU buffers and dispatches uploading to a dedicated worker thread, and a synchronous GL fallback using BitmapTexture::updateContents(). Atlas uploads are synchronized across workers using a countdown-latch fence. During replay, SkiaReplayCanvas intercepts raster image draws and substitutes them with atlas texture draws, mapping source coordinates into atlas space.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

The recent WPE WebKit 2.51.92 release is the first one to have its WPEPlatform documentation online, but it was not included in the tarball. This issue has been corrected and tarballs for future releases will also include this documentation.

Scrolling using touch input with WPEPlatform would result in scrolling faster when more than one touch point was in effect. The gesture detector has been fixed to make scrolling have always a consistent speed.

Releases 📦️

The third —and likely the last— release candidates for the upcoming stable branch, WebKitGTK 2.51.92 and WPE WebKit 2.51.92, have been published. For those interested in previewing the upcoming 2.52.x series this release is expected to be quite stable; but there might be still some rough edges. Reporting issues in Bugzilla are, as usual, more than welcome.

That’s all for this week!

Ziran Sun: A Day in “State of the Browser 2026” Conference

2026-03-02T18:11:17+00:00

The “State of the Browser 2026” Conference was held on Saturday, the 28th of February in The Barbican Centre, London. It is a yearly conference organised by London Web Standards. This is year is the 14th Edition.

From Igalia, this year we had Luke Warlow and myself attended in person, Javier Fernández attended online. My colleague Stephanie Stimac introduced this event to Igalia a couple of years ago. Now Igalia has become one of the sponsors for this great event. Luke had participated this event previously so it’s very helpful to understand more about this event from his note.

The event is a one-day, single-track conference that is community focused. While queuing for the registrations, a couple of attendees commented that talks for this event had been very good in the past few years. I’d say, this year was not an exception. I thoroughly enjoyed the talks, and the whole experiences.

Talks throughout the day covered a wide variety of topics including CSS features, accessibility, JS footprint, playing with gaming APIs and the art of connecting to people etc.. As someone who loves food, maybe I can describe it as a feast with content, taste, depth, variety…and a bit fun factor?

The open talk was Anchor positioning by Bramus Van Damme. The walk-through on the feature with examples were pretty cool, especially the case of a popover… with a little triangle (You’ll know what I mean if you look up the talk). Igalia worked on popover for Firefox in 2024, sponsored by Google. It’s really great to see that anchor positioning is in Firefox – popover has now found its place.

It was nice to hear that Igalians’ names were mentioned in the Temporal talk by Jason Williams from Bloomberg. A big shout out to Philip Chimento, Ujjwal Sharma who have participated substantially in the discussions about standardizing Temporal over the years and my fellow Igalians who have been writing spec PRs and tests for the feature. Check on Tim Chevalier’s blog on “Implementing the Temporal proposal in JavaScriptCore” if you’d like to find out more.

The atmosphere of the event was friendly, inclusive and energetic. I was very happy bumping into some ex-colleagues and making new friends.

One final note – This event brings a range of attendees, many are web developers. There are representatives from companies and browser vendors etc.. For some web developers, “Igalia” is a new name. I had a question like “Oh, is it the company with rainbow colours in the sponsors?”. Yes, Igalia is a private, worker-owned, employee-run cooperative model consultancy focused on open source software[1]. And Igalia has been a part of the Interop Project since its inception in 2021. Here is Igalia’s “rainbowy” logo :-).

Frédéric Wang: Stage d'implémentation des normes Web (session 2026)

2026-02-24T23:00:00+00:00

Les candidatures pour les « stages de programmation informatique » d’Igalia sont officiellement ouvertes jusqu’à début avril. Ils offrent aux étudiant·e·s l’occasion de participer au développement de logiciels libres tout en étant rémunéré·e·s 7 000 € brut pour 450 heures, réparties de juin à décembre 2026.

Comme chaque année, j’encadrerai un·e étudiant·e sur l’« Implémentation des normes Web » (Web Standards en anglais). L’objectif étant de modifier les navigateurs (Chromium, Firefox ou Safari…) afin d’améliorer le support de technologies Web (HTML, CSS, DOM…). Il faudra notamment étudier les spécifications correspondantes et écrire des tests de conformité. Notez bien que ce n’est pas un stage de développement Web mais de développement C++.

Un des objectifs de ce programme étant de lutter contre les discriminations professionnelles, tout le monde (y compris celles et ceux qui se sentent sous-représenté·e·s dans le secteur informatique) sont invité·e·s à candidater. Depuis 2016, mon équipe « Web Platform » a ainsi encadré 13 étudiant·e·s de différents pays dans le monde (Espagne, Inde, Italie, Australie, Cameroun, Chine, Vietnam, Angleterre et États-Unis) dont 7 femmes. L’année dernière, nous avions sélectionné Charlotte McCleary, une Américaine non-voyante qui a travaillé sur l’accessibilité dans Firefox au cours de son stage et a depuis rejoint Fizz Studio. J’aimerais encourager les étudiant·e·s Sourd·e·s à postuler et donne dans la vidéo ci-dessous une brève présentation du programme en LSF (en espérant que ce soit compréhensible et que vous serez indulgents avec mon piètre niveau en langue des signes 😅):

Si vous êtes intéréssé·e·s, remplissez ce formulaire en cochant la case Web Standards et en précisant éventuellement que vous avez trouvé cette offre via mon site Web. Enfin, si vous connaissez des étudiant·e·s qui pourraient participer, n’hésitez pas à partager l’annonce !

Igalia WebKit Team: WebKit Igalia Periodical #57

2026-02-23T19:52:49+00:00

Update on what happened in WebKit in the week from February 9 to February 23.

In this week we have a nice fix for video streams timestamps, a fix for a PDF rendering regression, support for rendering video buffers provided by Qualcomm video decoders, and a fix for a font selection issue. Also notable we had a new WPE Android release, and the libsoup 3.6.6 release.

Cross-Port 🐱

Added a new webkit_feature_list_find() convenience function to the public API, which searches for a WebKitFeature given its identifier.

Multimedia 🎥

GStreamer-based multimedia support for WebKit, including (but not limited to) playback, capture, WebAudio, WebCodecs, and WebRTC.

Opportunistically fix decoding timestamps to prevent deletion of preexisting samples when PTS doesn't conflict, fixing potential glitches when inserting videos (eg: ad insertion).

Graphics 🖼️

Fixed a PDF rendering regression caused by the canvas 2D operation recording feature, where switching between the recording canvas and the GPU surface canvas failed to preserve the full save/restore nesting, clip stack, and transparency layer state. Replaced the fragile state-copying approach with a state replay mechanism in GraphicsContextSkia that tracks the full sequence of save restore, clip, and transparency layer operations, then reconstructs the exact nesting on the target canvas when flushing a recording.

Added support for rendering video buffers provided by Qualcomm hardware-accelerated decoders, with aid from the EXT_YUV_target OpenGL extension.

Fixed the font selection issue that the system fallback font cache mixed up different font styles.

Releases 📦️

WPE Android 0.3.2 has been released, and prebuilt packages are available at the Maven Central repository. This is a stable maintenance release which updates WPE WebKit to 2.50.5, which is the most recent stable release.

libsoup 3.6.6 has been released with numerous bug and security fixes.

That’s all for this week!

Mauricio Faria de Oliveira: page_owner Part 2: optimizing output

2026-02-23T00:00:00+00:00

This blog post is part of a series about the page_owner debug feature in the Linux memory management subsystem, related to the talk Improving page_owner for profiling and monitoring memory usage per allocation stack trace presented at Linux Plumbers Conference 2025.

Part 1 is a quick introduction to page_owner and its debugfs files.
Part 2 describes challenges with processing page_owner files over time and a solution with new debugfs files in Linux v6.19.

Problem: stack traces over time

As described in Part 1, page_owner’s debugfs files contain stack traces for the most part:

/sys/kernel/debug/page_owner has one stack trace per allocated page, and
/sys/kernel/debug/page_owner_stacks/show_stacks lists the stack traces that allocated pages.

Reading and processing a significant amount of stack traces incurs a non-trivial computational cost in CPU and memory (copying to, and processing in, userspace) and storage usage, as the total size for such long strings might become large. This shouldn’t be an issue if done only once, but it does pose a concern if done repeatedly.

Take the processing of stack traces one step further and that concern materializes into a technical problem:

How to store information (say, number of pages) per-stack trace and over time?

For that, the stack trace must become a key to be assigned values from multiple reads over time. However, keys are usually numbers or somewhat short identifiers, not such long strings as stack traces (although doable, that is computationally more expensive in CPU and memory usage).

Workaround: stack trace hashing

One possible solution to this problem is hashing the stack traces and using the resulting hash values as keys.

However, this is inefficient with page_owner since there is significant duplication of stack traces on both debugfs files:

In the page_owner file, even on a single read, some stack traces may have tens/hundreds/thousands of duplicates; and they compound on multiple reads over time.
In the show_stacks file, there are no duplicates on a single read, but duplicates frequently happen on multiple reads over time.

With a high ratio of duplication, the dominant component in computational cost is the hashing step, which is significantly more expensive than the remaining step that simply use the resulting keys for storing values.

Additionally, the hashing step is usually repeated with the same data set (stack traces present in previous reads), which means that most of the calculations are discarded and done again on every read – wasting time and computational resources.

For illustration purposes, compare the execution time of script page_owner-to-show_stacks.py, which parses the page_owner file hashing the stack traces (with the extremely fast XXH3_64) and accumulating the number of pages per stack trace, reporting it at the end – basically mimicking show_stacks – with just reading the equivalent file.

The single read with hashing is 38.55 times slower:

# time ./page_owner-to-show_stacks.py </sys/kernel/debug/page_owner >/dev/null

real 0m1.542s
user 0m1.486s
sys 0m0.057s

# time cat /sys/kernel/debug/page_owner_stacks/show_stacks >/dev/null

real 0m0.040s
user 0m0.000s
sys 0m0.040s

So, considering the single-read results with the page_owner file, it’s not compelling to use it for multiple reads. However, multiple reads of the show_stacks file instead should perform better, though, as it contains unique stack traces and likely a lower ratio of duplication on multiple reads than in a single read of the former file.

Check the execution time of script show_stacks-over-time.py, which parses copies of show_stacks (collected over time), similarly hashing the stack traces and storing the number of pages per stack trace over time (that is, per copy).

For 100 copies, the execution time is almost 1 second:

# time ./show_stacks-over-time.py show_stacks.{1..100} >/dev/null

real 0m0.944s
user 0m0.900s
sys 0m0.044s

That is a great improvement (comparing to processing a single read of the page_owner file), but this is just a particular case on a lightly stressed, small VM with 1 GiB RAM. There is still the computational cost of hashing, which might increase processing time in cases with more stack traces (that is, a greater number of different code paths for memory allocation were exercised in the kernel).

Solution: stack trace handle numbers

The hashing of stack traces is only required in order to obtain a unique identifier for each stack trace, so that it can be used as a key. However, if such an identifier were already available, the hashing step (and associated computational cost) could be avoided altogether.

Fortunately, that is now the case with Linux 6.19! The stack trace storage used by page_owner ( stackdepot) provides a handle number to uniquely refer to stack traces – which meets the requirement.

Linux 6.19 contains two new debugfs files with handle numbers for optimized output:

/sys/kernel/debug/page_owner_stacks/show_handles: this lists nr_base_pages: per handle: (instead of per stack trace as in show_stacks)
/sys/kernel/debug/page_owner_stacks/show_stacks_handles: this lists handle: per stack trace (for resolving handle numbers to stack traces)

For the example in the previous post, show_stacks contains:

# cat /sys/kernel/debug/page_owner_stacks/show_stacks
...
 get_page_from_freelist+0x1416/0x1600
 __alloc_frozen_pages_noprof+0x18c/0x1000
 alloc_pages_mpol+0x43/0x100
 folio_alloc_noprof+0x56/0xa0
 page_cache_ra_unbounded+0xd9/0x230
 filemap_fault+0x305/0x1000
 __do_fault+0x2c/0xb0
 __handle_mm_fault+0x6f4/0xeb0
 handle_mm_fault+0xd9/0x210
 do_user_addr_fault+0x205/0x600
 exc_page_fault+0x61/0x130
 asm_exc_page_fault+0x26/0x30
nr_base_pages: 9643

...

While, for the same snippet, show_handles contains:

...
handle: 27000838
nr_base_pages: 9643

...

And the handle number can be resolved to a stack trace with show_stacks_handles:

...
 get_page_from_freelist+0x1416/0x1600
 __alloc_frozen_pages_noprof+0x18c/0x1000
 alloc_pages_mpol+0x43/0x100
 folio_alloc_noprof+0x56/0xa0
 page_cache_ra_unbounded+0xd9/0x230
 filemap_fault+0x305/0x1000
 __do_fault+0x2c/0xb0
 __handle_mm_fault+0x6f4/0xeb0
 handle_mm_fault+0xd9/0x210
 do_user_addr_fault+0x205/0x600
 exc_page_fault+0x61/0x130
 asm_exc_page_fault+0x26/0x30
handle: 27000838

...

Comparison: `show_stacks` vs. `show_handles`

From the previous post, for show_stacks:

# time cat /sys/kernel/debug/page_owner_stacks/show_stacks \
 | wc --bytes | numfmt --to=iec
402K

real 0m0.042s
user 0m0.004s
sys 0m0.046s

Now, for show_handles:

# time cat /sys/kernel/debug/page_owner_stacks/show_handles \
 | wc --bytes | numfmt --to=iec
31K

real 0m0.015s
user 0m0.004s
sys 0m0.019s

That is only 7.7% of the size and 35.7% of the time! Nice improvements.

Finally, compare the execution time of script show_handles-over-time.py with the previous one; it uses handle numbers as keys for stack traces instead of hashing them.

For 100 copies, the execution time is approximately 1/3 of a second, roughly 3 times faster.

# time ./show_handles-over-time.py show_stacks_handles show_handles.ln.{1..100} >/dev/nul

real 0m0.348s
user 0m0.319s
sys 0m0.030s

Conclusion

The original debugfs files provided by page_owner consist mainly of stack traces, which isn’t an efficient format for reading and processing repeatedly.

In order to store the number of pages used per stack trace over time, the stack traces must be converted to keys for storing values over time, for which hashing can be used. However, even efficient hashing algorithms incur a significant overhead.

In order to address this issue, Linux 6.19 provides new debugfs files for page_owner with handle numbers, which are unique identifiers for stack traces and can be used as keys, instead of hashing.

This optimizes the reading and processing of page_owner information, as it reduces the amount of data copied from kernel to userspace and allows storing the number of pages per stack trace over time without the overhead of hashing.

Scripts

Script 1: `page_owner-to-show_stacks.py`

#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Script to parse /sys/kernel/debug/page_owner, hashing the stack trace
# of each page and accumulating the number of pages per stack trace.
# At the end, print all stack traces and their number of pages in a format
# like /sys/kernel/debug/page_owner_stacks/show_stacks.
#
# Usage: page_owner-to-show_stacks.py </sys/kernel/debug/page_owner
#
# Author: Mauricio Faria de Oliveira <[email protected]>

import re
import sys
import xxhash

re_page = re.compile('^Page allocated via order ([0-9]+)')
re_stack = re.compile('^ ')
re_empty = re.compile('^$')

pages = {} # key -> number of pages
stacks = {} # key -> stack trace

for line in sys.stdin:

 # middle lines: try stack trace first as it occurs more often
 if re_stack.match(line):
 stack = stack + line
 continue
 
 # first line
 match = re_page.match(line)
 if match:
 order = int(match.group(1));
 stack = ''
 continue

 # last line
 if re_empty.match(line):
 key = xxhash.xxh3_64_hexdigest(stack)
 nr_pages = 2 ** order

 if key in pages:
 pages[key] += nr_pages
 else:
 pages[key] = nr_pages
 stacks[key] = stack

 continue

for key in stacks.keys():
 print(" " + stacks[key].strip())
 print("nr_base_pages: " + str(pages[key]))
 print()

Script #2: `show_stacks-over-time.py`

#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Script to parse /sys/kernel/debug/page_owner_stacks/show_stacks in multiple
# reads, hashing each stack trace and recording the number of base pages per
# stack trace in each read.
# At the end, print all stack traces and their number of pages in each read.
#
# Usage: show_stacks-over-time.py <read1> <read2> <read3> ... <read N>
#
# Author: Mauricio Faria de Oliveira <[email protected]>

import re
import sys
import xxhash

re_pages = re.compile('^nr_base_pages: ([0-9]+)')
re_stack = re.compile('^ ')
re_empty = re.compile('^$')

stacks = {}	# key -> stack trace (all reads)
pages = {}	# key -> array of number of pages (per read)
read = 0	# number of the current read

if len(sys.argv) < 2:
	exit(1)

files = sys.argv[1:]
nr_files = len(files)

for file in files:
	with open(file, 'r') as fd:
		stack = ''
		for line in fd:
	
			# first lines
			if re_stack.match(line):
				stack = stack + line
				continue
				
			# next to last line
			match = re_pages.match(line)
			if match:
				nr_pages = int(match.group(1));
				continue
		
			# last line
			if re_empty.match(line):
				key = xxhash.xxh3_64_hexdigest(stack)
		
				if key not in stacks:
					stacks[key] = stack;

				if key not in pages:
					pages[key] = {}

				pages[key][read] = nr_pages
		
				stack = ''
				continue

		read += 1

for key in stacks.keys():
	print(" " + stacks[key].strip())

	pages_per_read = []
	for read in range(nr_files):
		nr_pages = 0
		if read in pages[key]:
			nr_pages = pages[key][read]
		pages_per_read.append(str(nr_pages))
	
	print(' '.join(pages_per_read))
	print()

Script #3: `show_handles-over-time.py`

#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Script to parse /sys/kernel/debug/page_owner_stacks/show_handles in multiple
# reads, collecting handle numbers and recording the number of base pages per
# handle number in each read.
# At the end, print all stack traces and their number of pages in each read,
# resolving handle numbers with /sys/kernel/debug/page_owner_stacks/show_stacks_handles.
#
# Usage: show_handles-over-time.py <show_stacks_handles> <read1> <read2> <read3> ... <read N>
#
# Author: Mauricio Faria de Oliveira <[email protected]>

import re
import sys
import xxhash

re_pages = re.compile('^nr_base_pages: ([0-9]+)')
re_stack = re.compile('^ ')
re_empty = re.compile('^$')
re_handle = re.compile('^handle: ([0-9]+)')

stacks = {}	# handle number -> stack trace (all reads)
pages = {}	# handle number -> array of number of pages (per read)
read = 0	# number of the current read

if len(sys.argv) < 3:
	exit(1)

resolver = sys.argv[1]
files = sys.argv[2:]
nr_files = len(files)

for file in files:
	with open(file, 'r') as fd:
		for line in fd:
	
			# first line
			match = re_handle.match(line)
			if match:
				handle = int(match.group(1))
				continue
				
			# next to last line
			match = re_pages.match(line)
			if match:
				nr_pages = int(match.group(1));
				continue
		
			# last line
			if re_empty.match(line):
				key = handle
		
				if key not in pages:
					pages[key] = {}

				pages[key][read] = nr_pages
		
				continue

		read += 1

with open(resolver, 'r') as fd:
 stack = ''

 for line in fd:

 # first line
 if re_stack.match(line):
 stack = stack + line
 continue

 # next to last line
 match = re_handle.match(line)
 if match:
 handle = int(match.group(1))
 continue
 
 # last line
 if re_empty.match(line):
 stacks[handle] = stack
 stack = ''
 continue

for key in pages.keys():
	print(" " + stacks[key].strip())

	pages_per_read = []
	for read in range(nr_files):
		nr_pages = 0
		if read in pages[key]:
			nr_pages = pages[key][read]
		pages_per_read.append(str(nr_pages))
	
	print(' '.join(pages_per_read))
	print()

Mauricio Faria de Oliveira: page_owner Part 1: a quick introduction

2026-02-20T00:00:00+00:00

What is `page_owner`?

In the Linux kernel, page_owner is a debug feature that tracks the memory allocation (and release) of pages in the system – so as to tell the ‘owner of a page’ ;-).

For each memory allocation, page_owner stores its order, GFP flags, stack trace, timestamp, command, process ID (PID) and thread-group ID (TGID), and more. It also stores some information when pages are freed (stack trace, timestamp, PID and TGID).

With page_owner, one can find out “What allocated this page?” and “How many pages are allocated by this particular stack trace, PID, or comm”, for example.

This is struct page_owner in Linux v6.19. It stores additional information per-page, as an extension of struct page with CONFIG_PAGE_EXTENSION.

struct page_owner {
 unsigned short order;
 short last_migrate_reason;
 gfp_t gfp_mask;
 depot_stack_handle_t handle;
 depot_stack_handle_t free_handle;
 u64 ts_nsec;
 u64 free_ts_nsec;
 char comm[TASK_COMM_LEN];
 pid_t pid;
 pid_t tgid;
 pid_t free_pid;
 pid_t free_tgid;
};

Usage

In order to use page_owner, build the kernel with CONFIG_PAGE_OWNER=y (see mm/Kconfig.debug) and boot the kernel with page_owner=on.

The debugfs file /sys/kernel/debug/page_owner provides the information in struct page_owner for every page, listed per PFN (page frame number).

This example shows the entry for a page (line continuation added for clarity) – it tells “What allocated this page?”:

# cat /sys/kernel/debug/page_owner
...
Page allocated via order 0, \
 mask 0xd2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), \
 pid 5640, tgid 5640 (stress-ng-brk), ts 414987114269 ns
PFN 0x114 type Unmovable Block 0 type Unmovable Flags 0x200(workingset|node=0|zone=0)
 get_page_from_freelist+0x1416/0x1600
 __alloc_frozen_pages_noprof+0x18c/0x1000
 alloc_pages_mpol+0x43/0x100
 new_slab+0x349/0x460
 ___slab_alloc+0x811/0xd90
 __kmem_cache_alloc_bulk+0xb8/0x1f0
 __prefill_sheaf_pfmemalloc+0x42/0x90
 kmem_cache_prefill_sheaf+0xa9/0x240
 mas_preallocate+0x32f/0x420
 __split_vma+0xdc/0x300
 vms_gather_munmap_vmas+0xa4/0x240
 do_vmi_align_munmap+0xe9/0x180
 do_vmi_munmap+0xcb/0x160
 __vm_munmap+0xa7/0x150
 __x64_sys_munmap+0x16/0x20
 do_syscall_64+0xa4/0x310

...

One can use tools/mm/page_owner_sort to process the information in the file, or come up with custom commands, scripts, or programs.

For example: calculate the total size of pages allocated by stress-ng-brk with any order, in MiB:

# COMM=stress-ng-brk
# cat /sys/kernel/debug/page_owner \
 | awk -F '[ ,]' \
 '/^Page allocated via order .* \('${COMM}'\)/ { PAGES+=2^$5 } 
 END { print PAGES*4096/2**20 " MiB" }'
0.0429688 MiB

More information about page_owner is available in Documentation/mm/page_owner.rst.

Problem: output size

In the page_owner file, note the significant amount of text that is produced per-page: 745 bytes, in the example above.

Considering a system with 1 GiB of RAM and 4 kB pages, fully allocated, with similarly sized entries per page, the output size might reach approximately 186 MiB! (745 [bytes/page] * (2**30 [bytes of RAM] / 4096 [bytes/page]) / 2**20 [bytes/MiB])

For validation, a test VM with 1 GiB of RAM after just a warm-up level of stress (stress-ng --sequential --timeout 1) produced 125 MiB, which was not quick to read even in idle state:

# time cat /sys/kernel/debug/page_owner \
 | wc --bytes | numfmt --to=iec
125M

real 0m3.009s
user 0m0.512s
sys 0m3.542s

While this might not be a serious issue for reading and processing the file only once, it can likely impact a sequence of operations.

Alternative: optimized output

Fortunately, another debugfs file, /sys/kernel/debug/page_owner_stacks/show_stacks, provides an optimized output for obtaining the memory usage per stack trace. Even though it doesn’t address all needs as the generic output, it resembles the default operation of page_owner_sort (without PFN lines) and provides an often interesting information for kernel development or analysis.

This example shows the entry for a stack trace – it tells “How many pages are allocated by this particular stack trace?”

# cat /sys/kernel/debug/page_owner_stacks/show_stacks
...
 get_page_from_freelist+0x1416/0x1600
 __alloc_frozen_pages_noprof+0x18c/0x1000
 alloc_pages_mpol+0x43/0x100
 folio_alloc_noprof+0x56/0xa0
 page_cache_ra_unbounded+0xd9/0x230
 filemap_fault+0x305/0x1000
 __do_fault+0x2c/0xb0
 __handle_mm_fault+0x6f4/0xeb0
 handle_mm_fault+0xd9/0x210
 do_user_addr_fault+0x205/0x600
 exc_page_fault+0x61/0x130
 asm_exc_page_fault+0x26/0x30
nr_base_pages: 9643

...

The nr_base_pages field tells the number of base pages (i.e., not huge pages) allocated by a stack trace. So, this particular stack trace for readahead (page_cache_ra_unbounded()) has allocated approximately 37 MiB (9643 [pages] * 4096 [bytes/page] / 2**20 [ bytes/MiB]).

Note this file is more efficient for this particular purpose: just 402 KiB in less than 0.05 seconds. (That is 0.3% of the size and 1.7% of the time):

# time cat /sys/kernel/debug/page_owner_stacks/show_stacks \
 | wc --bytes | numfmt --to=iec
402K

real 0m0.042s
user 0m0.004s
sys 0m0.046s

Conclusion

The page_owner debug feature (enabled with CONFIG_PAGE_OWNER=y and page_owner=on) provides information about the memory allocation of pages in the system in debugfs files /sys/kernel/debug/page_owner with a generic format (dense description per-page) and /sys/kernel/debug/page_owner_stacks/show_stacks with an optimized format (number of base pages per stack trace).

Alex Bradbury: Minipost: Additional figures for per-query energy consumption of LLMs

2026-02-17T12:00:00+00:00

Last month I wrote up a fairly long piece on per-query energy consumption of LLMs using the data from InferenceMAX (note: InferenceMAX has since been renamed to InferenceX). Much of the write-up was dedicated to exploring what you can actually conclude from these figures and how that interacts with some of the implementation decisions in the benchmark, but I feel the results still give a useful yardstick. Beyond concerns about overly-specialised serving engine configurations and whether the workload is representative of real-world model serving in a paid API host, the other obvious limitation is that InferenceMAX is only testing GPT-OSS 120b and DeepSeek R1 0528 when there is a world of other models out there. I dutifully added "run my own tests using other models" to the todo list and here we are. By "here we are" I of course mean I made no progress towards that goal but Zach Mueller at Lambda started publishing model cards with the needed data - thanks Zach!

The setup for Lambda is simple - each model card lists the observed token generation throughput and total throughput (along with other stats) for an input sequence length / output sequence length (ISL/OSL) of 8192/1024, as benchmarked using vllm bench serve. The command used to serve the LLM (using sglang or vllm depending on the model) is also given. As a starting point this is no worse than the InferenceMAX data, and potentially somewhat better due to figures being taken from a configuration that's not overly specialised to a particular query length.

The figures each Lambda model card gives us that are relevant for calculating the energy per query are: the hardware used, token generation throughput and total token throughput (input+output tokens). Other statistics such as the time to first token, inter-token latency, and parallel requests tested help confirm whether this is a configuration someone would realistically use. Using an equivalent methodology to before, we get the Watt hours per query by:

Determining the total Watts for the GPU cluster. We take the figures used by SemiAnalysis (2.17kW for a single B200) and multiply by the number of GPUs.
Calculate the joules per token by dividing this total Watts figure by the total token throughput. This gives a weighted average of the joules per token for the measured workload, reflecting the ratio of isl:osl.
Multiply this weighted average of joules per token by the tokens per query (isl+osl) to get the joules per query. Then divide by 3600 to get Wh.

Collecting the data from the individual model cards we can generate the following (as before, using minutes of PlayStation 5 gameplay as a point of comparison):

data = {
    "Qwen/Qwen3.5-397B-A17B": {
        "num_b200": 8,
        "total_throughput": 11092,
    },
    "MiniMaxAI/MiniMax-M2.5": {
        "num_b200": 2,
        "total_throughput": 8062,
    },
    "zai-org/GLM-5-FP8": {
        "num_b200": 8,
        "total_throughput": 6300,
    },
    "zai-org/GLM-4.7-Flash": {
        "num_b200": 1,
        "total_throughput": 8125,
    },
    "arcee-ai/Trinity-Large-Preview": {
        "num_b200": 8,
        "total_throughput": 15611,
    },
}

# 8192 + 1024
TOKENS_PER_QUERY = 9216

# Taken from <https://inferencex.semianalysis.com/>
B200_KW = 2.17

# Reference power draw for PS5 playing a game. Taken from
# <https://www.playstation.com/en-gb/legal/ecodesign/> ("Active Power
# Consumption"). Ranges from ~217W to ~197W depending on model.
PS5_KW = 0.2


def wh_per_query(num_b200, total_throughput, tokens_per_query):
    total_cluster_kw = num_b200 * B200_KW
    total_cluster_watts = total_cluster_kw * 1000
    # joules_per_token is a weighted average for the measured mix of input
    # and output tokens.
    joules_per_token = total_cluster_watts / total_throughput
    joules_per_query = joules_per_token * tokens_per_query
    # Convert joules to watt-hours
    return joules_per_query / 3600.0

def ps5_minutes(wh):
    ps5_watts = PS5_KW * 1000
    return (wh / ps5_watts) * 60.0

MODEL_WIDTH = 31
WH_WIDTH = 8
PS5_WIDTH = 8

header = f"{'Model':<{MODEL_WIDTH}} | {'Wh/q':<{WH_WIDTH}} | {'PS5 min':<{PS5_WIDTH}}"
separator = f"{'-' * MODEL_WIDTH} | {'-' * WH_WIDTH} | {'-' * PS5_WIDTH}"

print(header)
print(separator)

for model, vals in data.items():
    wh = wh_per_query(vals["num_b200"], vals["total_throughput"], TOKENS_PER_QUERY)
    ps5_min = ps5_minutes(wh)

    wh_str = f"{wh:.2f}" if wh < 10 else f"{wh:.1f}"
    print(f"{model.strip():<{MODEL_WIDTH}} | {wh_str:<{WH_WIDTH}} | {ps5_min:.2f}")

This gives the following figures (reordered to show Wh per query in ascending order, and added a column for interactivity (1/TPOT)):

Model	Intvty (tok/s)	Wh/q	PS5 min.
zai-org/GLM-4.7-Flash (bf16)	34.0	0.68	0.21
MiniMaxAI/MiniMax-M2.5 (fp8)	30.3	1.38	0.41
arcee-ai/Trinity-Large-Preview (bf16)	58.8	2.85	0.85
Qwen/Qwen3.5-397B-A17B (bf16)	41.7	4.01	1.20
zai-org/GLM-5-FP8 (fp8)	23.3	7.05	2.12

As a point of comparison, the most efficient 8 GPU deployment of fp8 DeepSeek R1 0528 from my figures in the previous article was 3.32 Wh per query.

And that's all I really have for today. Some interesting datapoints with hopefully more to come as Lambda puts up more model cards in this format. There's a range of interesting potential further experiments to do, but for now, I just wanted to share this initial look.

Article changelog

2026-02-17: Initial publication date.

Alex Bradbury: shandbox

2026-02-11T12:00:00+00:00

shandbox is a simple Linux sandboxing script that serves my needs well. Perhaps it works for you too? No dependencies between a shell and util-linux (unshare and nsenter).

In short, it aims to provide fairly good isolation for personal files (i.e. your $HOME) while being very convenient for day to day use. It's designed to be run as an unprivileged user - as long as you can make new namespaces you should be good to go. By default /home/youruser/sandbox shows up as /home/sandbox within the sandbox, and other than standard paths like /usr, /etc, /tmp, and so on it's left for you to either copy things into the sandbox or expose them via a mount. There's a single shared sandbox (i.e. processes within the sandbox can see and interact with each other, and the exposed sandbox filesystem is shared as well), which trades off some ease of use for the security you might get with a larger number of more targeted sandboxes. On the other hand, you only gain security from a sandbox if you actually use it and this is a setup that offers very low friction for me. The network is not namespaced (although this is something you could change with a simple edit).

Usability is both subjective and highly dependent on your actual use case, so the tradeoffs may or may not align with what is interesting for you! Bubblewrap is an example of a mature alternative unprivileged sandboxing tool that offers a lot of configurability as well as options with greater degrees of sandboxing. Beyond that, look to Firecracker based solutions or gvisor. shandbox obviously aims to provide a reasonable sandbox as much as Linux namespaces alone are able to offer, but if you're looking for a security property stronger than "makes it harder for something to edit or access unwanted files" it's down to you to both carefully review its implementation and consider alternatives.

Usage example

$ shandbox run uvx pycowsay
Installed 1 package in 5ms

  ------------
< Hello, world >
  ------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
$ shandbox status
running (pid 1589364)

log:
  2026-02-11 13:02:51 stopped
  2026-02-11 13:05:06 started (pid 1589289)
$ shandbox add-mount ~/repos/llvm-project /home/sandbox/llvm-project
mounted /home/asb/repos/llvm-project -> /home/sandbox/llvm-project
$ shandbox run touch /home/sandbox/llvm-project/write-attempt
touch: cannot touch '/home/sandbox/llvm-project/write-attempt': Read-only file system
$ shandbox remove-mount /home/sandbox/llvm-project
unmounted /home/sandbox/llvm-project
$ shandbox add-mount --read-write ~/repos/llvm-project /home/sandbox/llvm-project
mounted /home/asb/repos/llvm-project -> /home/sandbox/llvm-project
$ shandbox run touch /home/sandbox/llvm-project/write-attempt

shandbox enter will open a shell within the sandbox for easy interactive usage. As a convenience, if the current working directory is in $HOME/sandbox (e.g. $HOME/sandbox/foo) then the working directory within the sandbox for shandbox run or shandbox enter will be set to the appropriate path within the sandbox (/home/sandbox/foo in this case). i.e., the case where this mapping is trivial. Environment variables are not passed through.

Functionality overview

shandbox start: Start the sandbox, creating the necessary namespaces and mount layout. Fails if the sandbox is already running.
shandbox stop: Stop the sandbox by killing the process holding the namespaces. Fails if the sandbox is not running.
shandbox restart: Stop the sandbox and start it again.
shandbox status: Print whether the sandbox is running and if it is, the pid. Also print the last 20 lines of the log.
shandbox enter: Open bash within the sandbox, starting the sandbox first if it's not already running.
shandbox run <command> [args...]: Run a command inside the sandbox. The current working directory is translated to an in-sandbox path if it falls within the sandbox home directory. Starts the sandbox first if it isn't already running.
shandbox add-mount [--read-write] <host-path> <sandbox-path>: Bind-mount a host path into the running sandbox. Mounts are read-only by default; pass --read-write to allow writes. The sandbox must already be running. Both directories and individual files are supported.
shandbox remove-mount <sandbox-path>: Remove a previously added bind mount from the running sandbox.

Implementation approach

The core sandboxing functionality is provided by the Linux namespaces functionality exposed by unshare and nsenter. The script's implementation should be quite readable but I'll try to summarise some key points here.

The goal is that:

Within the sandbox, you appear as an unprivileged user, with uid and gid equal to your usual Linux user.
It should be possible to expose additional files or directories to the sandbox once it's running.
Applications running within the sandbox have no way (modulo bugs or vulnerabilities in the kernel or accessible applications) of reaching files on the host filesystem that aren't explicitly exposed.
- To underline: This is a goal, it is not a guarantee.
It's possible to launch multiple processes within the sandbox which can all see each other, and have the same shared sandboxed filesystem.
This is all doable as an unprivileged user.

To implement that:

Two sets of namespaces are used to provide this isolation: the outer 'shandbox_root' has the user mapped to root within the namespace and retains access to standard / (allowing us to mount additional paths into after the sandbox has started). The inner 'shandbox_user' represents a new user namepsace mapping our uid/gid to an unprivileged user, but other namespaces are shared with 'shandbox_root'. Sandboxed processes are launched within the namespaces of 'shandbox_user'.
The process IDs of the initial process within 'sandbox_root' and 'sandbox_user' are saved and recalled so the script can use nsenter to enter the namespace.
To help make it easier to tell when you're in the sandbox, a dummy /etc/passwd is bind-mounted naming the current user as sandbox.
When shandbox start is executed, the necessary directories are bind mounted in a directory that will be used as root (/) for the user sandbox in .local/share/shandbox/root. This happens within the sandbox_root namespace, which then uses unshare again to create a new user namespace with an unprivileged user, executing within a chroot.
'sandbox_root' retains access to the host filesystem, which is necessary to allow mounting additional paths after the fact. Without this requirement, we could likely rewrite shandbox start to use pivot_root.

Making it your own

The script should be straight-forward enough to customise to your needs if they're not too dissimilar to what is offered out of the box. Some variables at the top provide things you may be more likely to want to change, such as the home directory location, and a list of files or directories in $HOME to always bind-mount into the sandbox home:

SANDBOX_HOME_DIR="$HOME/sandbox"
HOME_FILES_TO_MAP=".bashrc .vimrc"
HOME_DIRS_TO_MAP=".vim bin"
SB_HOME="/home/sandbox"
SB_PATH="$SB_HOME/bin:/usr/local/bin:/usr/bin"

Article changelog

2026-02-11: Initial publication date.

José Dapena: Container Timing: measuring web components performance

2026-02-10T00:00:00+00:00

Over the last year, as part of the collaboration between Igalia and Bloomberg to improve web performance observability, I worked on a new web performance API: Container Timing. This standard aims to make component-level performance measurement as easy as page-level metrics like LCP and FCP.

My focus has been writing the native implementation in Chromium, which is now available behind a feature flag.

In this post, I will explain why this API is needed, how it works, and how you can experiment with it today. In a follow-up post, I will dive deep into the implementation details within the Blink rendering engine.

The problem: measuring component performance #

We currently use Largest Contentful Paint (LCP) and First Contentful Paint (FCP) to measure web page loading performance. Both metrics are page-scoped, meaning they evaluate the user perceived load speed for full page.

The Element Timing API shifts the focus to individual DOM elements. By targetting specific elements, like hero images or a headers, we can measure their specific rendering performance independent of the rest of the page.

However, modern web development is component-based. Developers build complex widgets (as grids, charts, feeds or panels) that are made of many elements. It is not trivial to understand the performance of those components:

LCP may not be useful as another large image painting could delay it.
Measuring a web component with Element Timing may require instrumenting all the significant elements one by one.

The solution: Container Timing #

This is where Container Timing comes in! With the new specification, a web developer can mark subtrees of the DOM as “containers”. Then, it provides performance entries aggregating the painting time of that subtree.

This way, we can answer: “when did a specific component finish painting its content?”.

Some examples:

Breaking down the contributors to the initial page load: with Container Timing we can focus on the components that are more relevant to the user experience.
Single page application navigation: when a soft navigation shows a new component on the screen, we can obtain painting information for it.
Lazy-loaded components: Tracking when a widget that loads below the fold is fully visible.
Third-party content: Monitoring the performance of ads or embedded widgets.

You just need to add, to the top element of the subtree, the new attribute containertiming. When you add it to an HTML element, the browser will track all the painting updates of that element and its descendants.

What happens under the hood? The browser will start monitoring the rendering pipeline for paints that contribute to representing the subtree. When a new frame is painted, if that paints new areas for that subtree, it reports a performance entry showing the increase in painted area. It is similar to LCP, but for a specific subtree!

How to use Container Timing? #

Using the API is straightforward. First, mark the containers you want to track in HTML:

<div id="my-widget" containertiming="widget-load">
  <img src="graph.png" />
  <p>Loading data...</p>
</div>

Then, use a PerformanceObserver to listen for container entries:

const observer = new PerformanceObserver((list) => {
  list.getEntries().forEach((entry) => {
    console.log(`Container '${entry.identifier}' painted.`);
    console.log(`Time: ${entry.startTime}`);
    console.log(`Size: ${entry.size}`); // The area painted
  });
});

observer.observe({ type: "container", buffered: true });

When the web contents load, new Performance entries will be emitted with the container updates.

Which entry will be interesting? The API lets you choose what best fits your needs! Some ideas:

The most important entry could be the last one: the one that increased the painted area for the last time. Something similar to LCP.
Or maybe the last one that contributed a significant size increase?
Or the last one before a user interaction?

A native implementation for Chromium #

In the initial steps of the specification, Jason Williams wrote a polyfill that worked on top of Element Timing. This was very useful to understand and polish the kind of information the specification could provide. However, this had its own performance impact.

Deprecation Notice:

The polyfill is now deprecated and no longer maintained, as the native API cannot be fully replicated using Element Timing. Please use the native implementation for accurate results.

So I started a native implementation in Chromium. The main idea was working on top of the already existing implementation for Element Timing, and add the remaining bits.

In my next blog post I will go through the implementation details. But, for this post, it is relevant to state that the goals of this native implementation were:

Minimizing the overhead. It should be almost zero when elements are not interesting to Container Timing, and very fast and light when paints were relevant.
It should reuse as much as possible of the already existing logic for Element Timing.

The native implementation has landed and is available in Chromium144+, but still behind the ContainerTiming feature flag.

You can experiment with this feature locally by passing the following flag to Chromium at startup:

chrome --enable-blink-features=ContainerTiming

Or you can just enable the “Experimental Web Platform features” in chrome://flags.

Upcoming trials #

So now, it is time to collect feedback from the actual web developers.

We have already presented the specification in several conferences (as BlinkOn 20 or Performance.now() 2024). And discussions are ongoing in the Web Performance Working Group.

We just announced the Dev Trial in the blink-dev mailing list! The feature is now officially ready for testing.

What’s next? We are also preparing an Origin Trial, that will allow developers to test the specification in production for a subset of their users.

If you want to provide feedback, we are collecting it in the explainer ticket tracker.

Wrapping up #

With Container Timing, you will be able to measure paintings at the web component level, filling a significant gap in the web performance monitoring landscape.

If you struggled with finding out the ready time of your widgets, just try it! It is available, under the feature flags ContainerTiming, in Chromium Stable today.

And stay tuned! In a follow up post, I will go through the native implementation details in Chromium.

Thanks! #

This has been done as part of the collaboration between Bloomberg and Igalia. Thanks!

References #

Igalia WebKit Team: WebKit Igalia Periodical #56

2026-02-09T23:21:30+00:00

Update on what happened in WebKit in the week from February 2 to February 9.

The main event this week was FOSDEM (pun intended), which included presentations related to WebKit; but also we got a batch of stable and development releases, asynchronous scrolling work, OpenGL logging, cleanups, and improving the inspector for the WPE work.

Cross-Port 🐱

Graphics 🖼️

While asynchronous scrolling for mouse wheel events was already supported, scrollbar layers were still being painted on the main thread. This has been changed to paint scrollbars on the scrolling thread instead, which avoids scrollbars to “lag” behind scrolled content.

Fixed flickering caused by the combination of damage tracking and asynchronous scrolling for mouse wheel events.

It is now possible to enable debug logging for OpenGL contexts using the new GLContext log channel, which takes advantage of the message events produced by the widespread KHR_debug extension.

Figuring out the exact location inside WebKit that triggered an OpenGL issue may still be challenging with this aid, and therefore a backtrace will be appended in case of errors to help pinpoint the source, when the log channel is enabled at the “debug” level with GLContext=debug.

Configuring the build with USE_SKIA=OFF to make WebKit use the Cairo graphics library is no longer supported. Using Skia has been the default since late 2024, and after two full years the 2.54.0 release (due in September 2026) will be the first one where the choice is no longer possible.

WebKitGTK 🖥️

The “on demand” hardware acceleration policy has been rarely used lately, and thus support for it has been removed. Note that this affects only the GTK port when built with GTK 3—the option never existed when using GTK 4.

Existing GTK 3 applications that use WEBKIT_HARDWARE_ACCELERATION_POLICY_ON_DEMAND will continue to work and do not need rebuilding: they will be promoted to use the “always enabled” policy starting with WebKitGTK 2.54.0 (due in September 2026).

The Web Inspector has received support for saving data to local files, allowing things such as saving page resources or exporting the network session to a HAR archive.

Note that using the Web Inspector locally is supported when using the WPEPlatform API, and the keyboard shortcut Ctrl+Shift+I may be used to bring it up.

Releases 📦️

WebKitGTK 2.50.5 and WPE WebKit 2.50.5 have been released. These are stable maintenance releases that improves stability, correct bugs, and fixes small rendering issues.

The second release candidates for the upcoming stable branch, WebKitGTK 2.51.91 and WPE WebKit 2.51.91, have been published as well. Those using those to preview the upcoming 2.52.x series are encouraged to provide bug reports in Bugzilla for any issue they may experience.

Community & Events 🤝

We have published a blog post on our work implementing the Temporal proposal in JavaScriptCore, WebKit's JavaScript engine.

This year's edition of FOSDEM took place in Brussels between January 31st and February 1st, and featured a number of sessions related to WebKitGTK and WPE:

The Web Platform on Linux devices with WebKit: where are we now?, by Mario Sánchez, is a good introduction-level talk about the GTK and WPE WebKit ports.
WebRTC support in WebKitGTK and WPEWebKit with GStreamer: Current status and plans by Philippe Normand. Exactly what it says on the tin.
Interop and MathML Core by Eri Pazos, about the ongoing effort to improve how different Web engines handle MathML—including WebKit!

The videos for the talks are already available, too.

That’s all for this week!

Andy Wingo: six thoughts on generating c

2026-02-09T13:47:44+00:00

So I work in compilers, which means that I write programs that translate programs to programs. Sometimes you will want to target a language at a higher level than just, like, assembler, and oftentimes C is that language. Generating C is less fraught than writing C by hand, as the generator can often avoid the undefined-behavior pitfalls that one has to be so careful about when writing C by hand. Still, I have found some patterns that help me get good results.

Today’s note is a quick summary of things that work for me. I won’t be so vain as to call them “best practices”, but they are my practices, and you can have them too if you like.

static inline functions enable data abstraction

When I learned C, in the early days of GStreamer (oh bless its heart it still has the same web page!), we used lots of preprocessor macros. Mostly we got the message over time that many macro uses should have been inline functions; macros are for token-pasting and generating names, not for data access or other implementation.

But what I did not appreciate until much later was that always-inline functions remove any possible performance penalty for data abstractions. For example, in Wastrel, I can describe a bounded range of WebAssembly memory via a memory struct, and an access to that memory in another struct:

struct memory { uintptr_t base; uint64_t size; };
struct access { uint32_t addr; uint32_t len; };

And then if I want a writable pointer to that memory, I can do so:

#define static_inline \
  static inline __attribute__((always_inline))

static_inline void* write_ptr(struct memory m, struct access a) {
  BOUNDS_CHECK(m, a);
  char *base = __builtin_assume_aligned((char *) m.base_addr, 4096);
  return (void *) (base + a.addr);
}

(Wastrel usually omits any code for BOUNDS_CHECK, and just relies on memory being mapped into a PROT_NONE region of an appropriate size. We use a macro there because if the bounds check fails and kills the process, it’s nice to be able to use __FILE__ and __LINE__.)

Regardless of whether explicit bounds checks are enabled, the static_inline attribute ensures that the abstraction cost is entirely burned away; and in the case where bounds checks are elided, we don’t need the size of the memory or the len of the access, so they won’t be allocated at all.

If write_ptr wasn’t static_inline, I would be a little worried that somewhere one of these struct values would get passed through memory. This is mostly a concern with functions that return structs by value; whereas in e.g. AArch64, returning a struct memory would use the same registers that a call to void (*)(struct memory) would use for the argument, the SYS-V x64 ABI only allocates two general-purpose registers to be used for return values. I would mostly prefer to not think about this flavor of bottleneck, and that is what static inline functions do for me.

avoid implicit integer conversions

C has an odd set of default integer conversions, for example promoting uint8_t to signed int, and also has weird boundary conditions for signed integers. When generating C, we should probably sidestep these rules and instead be explicit: define static inline u8_to_u32, s16_to_s32, etc conversion functions, and turn on -Wconversion.

Using static inline cast functions also allows the generated code to assert that operands are of a particular type. Ideally, you end up in a situation where all casts are in your helper functions, and no cast is in generated code.

wrap raw pointers and integers with intent

Whippet is a garbage collector written in C. A garbage collector cuts across all data abstractions: objects are sometimes viewed as absolute addresses, or ranges in a paged space, or offsets from the beginning of an aligned region, and so on. If you represent all of these concepts with size_t or uintptr_t or whatever, you’re going to have a bad time. So Whippet has struct gc_ref, struct gc_edge, and the like: single-member structs whose purpose it is to avoid confusion by partitioning sets of applicable operations. A gc_edge_address call will never apply to a struct gc_ref, and so on for other types and operations.

This is a great pattern for hand-written code, but it’s particularly powerful for compilers: you will often end up compiling a term of a known type or kind and you would like to avoid mistakes in the residualized C.

For example, when compiling WebAssembly, consider struct.set‘s operational semantics: the textual rendering states, “Assert: Due to validation, val is some ref.struct structaddr.” Wouldn’t it be nice if this assertion could translate to C? Well in this case it can: with single-inheritance subtyping (as WebAssembly has), you can make a forest of pointer subtypes:

typedef struct anyref { uintptr_t value; } anyref;
typedef struct eqref { anyref p; } eqref;
typedef struct i31ref { eqref p; } i31ref;
typedef struct arrayref { eqref p; } arrayref;
typedef struct structref { eqref p; } structref;

So for a (type $type_0 (struct (mut f64))), I might generate:

typedef struct type_0ref { structref p; } type_0ref;

Then if I generate a field setter for $type_0, I make it take a type_0ref:

static inline void
type_0_set_field_0(type_0ref obj, double val) {
  ...
}

In this way the types carry through from source to target language. There is a similar type forest for the actual object representations:

typedef struct wasm_any { uintptr_t type_tag; } wasm_any;
typedef struct wasm_struct { wasm_any p; } wasm_struct;
typedef struct type_0 { wasm_struct p; double field_0; } type_0;
...

And we generate little cast routines to go back and forth between type_0ref and type_0* as needed. There is no overhead because all routines are static inline, and we get pointer subtyping for free: if a struct.set $type_0 0 instruction is passed a subtype of $type_0, the compiler can generate an upcast that type-checks.

fear not `memcpy`

In WebAssembly, accesses to linear memory are not necessarily aligned, so we can’t just cast an address to (say) int32_t* and dereference. Instead we memcpy(&i32, addr, sizeof(int32_t)), and trust the compiler to just emit an unaligned load if it can (and it can). No need for more words here!

for ABI and tail calls, perform manual register allocation

So, GCC finally has __attribute__((musttail)): praise be. However, when compiling WebAssembly, it could be that you end up compiling a function with, like 30 arguments, or 30 return values; I don’t trust a C compiler to reliably shuffle between different stack argument needs at tail calls to or from such a function. It could even refuse to compile a file if it can’t meet its musttail obligations; not a good characteristic for a target language.

Really you would like it if all function parameters were allocated to registers. You can ensure this is the case if, say, you only pass the first n values in registers, and then pass the rest in global variables. You don’t need to pass them on a stack, because you can make the callee load them back to locals as part of the prologue.

What’s fun about this is that it also neatly enables multiple return values when compiling to C: simply go through the set of function types used in your program, allocate enough global variables of the right types to store all return values, and make a function epilogue store any “excess” return values—those beyond the first return value, if any—in global variables, and have callers reload those values right after calls.

what’s not to like

Generating C is a local optimum: you get the industrial-strength instruction selection and register allocation of GCC or Clang, you don’t have to implement many peephole-style optimizations, and you get to link to to possibly-inlinable C runtime routines. It’s hard to improve over this design point in a marginal way.

There are drawbacks, of course. As a Schemer, my largest source of annoyance is that I don’t have control of the stack: I don’t know how much stack a given function will need, nor can I extend the stack of my program in any reasonable way. I can’t iterate the stack to precisely enumerate embedded pointers (but perhaps that’s fine). I certainly can’t slice a stack to capture a delimited continuation.

The other major irritation is about side tables: one would like to be able to implement so-called zero-cost exceptions, but without support from the compiler and toolchain, it’s impossible.

And finally, source-level debugging is gnarly. You would like to be able to embed DWARF information corresponding to the code you residualize; I don’t know how to do that when generating C.

(Why not Rust, you ask? Of course you are asking that. For what it is worth, I have found that lifetimes are a frontend issue; if I had a source language with explicit lifetimes, I would consider producing Rust, as I could machine-check that the output has the same guarantees as the input. Likewise if I were using a Rust standard library. But if you are compiling from a language without fancy lifetimes, I don’t know what you would get from Rust: fewer implicit conversions, yes, but less mature tail call support, longer compile times... it’s a wash, I think.)

Oh well. Nothing is perfect, and it’s best to go into things with your eyes wide open. If you got down to here, I hope these notes help you in your generations. For me, once my generated C type-checked, it worked: very little debugging has been necessary. Hacking is not always like this, but I’ll take it when it comes. Until next time, happy hacking!

Andy Wingo: ahead-of-time wasm gc in wastrel

2026-02-06T15:48:17+00:00

Hello friends! Today, a quick note: the Wastrel ahead-of-time WebAssembly compiler now supports managed memory via garbage collection!

hello, world

The quickest demo I have is that you should check out and build wastrel itself:

git clone https://codeberg.org/andywingo/wastrel
cd wastrel
guix shell
# alternately: sudo apt install guile-3.0 guile-3.0-dev \
#    pkg-config gcc automake autoconf make
autoreconf -vif && ./configure
make -j

Then run a quick check with hello, world:

$ ./pre-inst-env wastrel examples/simple-string.wat
Hello, world!

Now give a check to gcbench, a classic GC micro-benchmark:

$ WASTREL_PRINT_STATS=1 ./pre-inst-env wastrel examples/gcbench.wat
Garbage Collector Test
 Creating long-lived binary tree of depth 16
 Creating a long-lived array of 500000 doubles
Creating 33824 trees of depth 4
	Top-down construction: 10.189 msec
	Bottom-up construction: 8.629 msec
Creating 8256 trees of depth 6
	Top-down construction: 8.075 msec
	Bottom-up construction: 8.754 msec
Creating 2052 trees of depth 8
	Top-down construction: 7.980 msec
	Bottom-up construction: 8.030 msec
Creating 512 trees of depth 10
	Top-down construction: 7.719 msec
	Bottom-up construction: 9.631 msec
Creating 128 trees of depth 12
	Top-down construction: 11.084 msec
	Bottom-up construction: 9.315 msec
Creating 32 trees of depth 14
	Top-down construction: 9.023 msec
	Bottom-up construction: 20.670 msec
Creating 8 trees of depth 16
	Top-down construction: 9.212 msec
	Bottom-up construction: 9.002 msec
Completed 32 major collections (0 minor).
138.673 ms total time (12.603 stopped); 209.372 ms CPU time (83.327 stopped).
0.368 ms median pause time, 0.512 p95, 0.800 max.
Heap size is 26.739 MB (max 26.739 MB); peak live data 5.548 MB.

We set WASTREL_PRINT_STATS=1 to get those last 4 lines. So, this is a microbenchmark: it runs for only 138 ms, and the heap is tiny (26.7 MB). It does collect 30 times, which is something.

is it good?

I know what you are thinking: OK, it’s a microbenchmark, but can it tell us anything about how Wastrel compares to V8? Well, probably so:

$ guix shell node time -- \
   time node js-runtime/run.js -- \
     js-runtime/wtf8.wasm examples/gcbench.wasm
Garbage Collector Test
[... some output elided ...]
total_heap_size: 48082944
[...]
0.23user 0.03system 0:00.20elapsed 128%CPU (0avgtext+0avgdata 87844maxresident)k
0inputs+0outputs (0major+13325minor)pagefaults 0swaps

Which is to say, V8 takes more CPU time (230ms vs 209ms) and more wall-clock time (200ms vs 138ms). Also it uses twice as much managed memory (48 MB vs 26.7 MB), and more than that for the total process (88 MB vs 34 MB, not shown).

improving on v8, really?

Let’s try with quads, which at least has a larger active heap size. This time we’ll compile a binary and then run it:

$ ./pre-inst-env wastrel compile -o quads examples/quads.wat
$ WASTREL_PRINT_STATS=1 guix shell time -- time ./quads 
Making quad tree of depth 10 (1398101 nodes).
	construction: 23.274 msec
Allocating garbage tree of depth 9 (349525 nodes), 60 times, validating live tree each time.
	allocation loop: 826.310 msec
	quads test: 860.018 msec
Completed 26 major collections (0 minor).
848.825 ms total time (85.533 stopped); 1349.199 ms CPU time (585.936 stopped).
3.456 ms median pause time, 3.840 p95, 5.888 max.
Heap size is 133.333 MB (max 133.333 MB); peak live data 82.416 MB.
1.35user 0.01system 0:00.86elapsed 157%CPU (0avgtext+0avgdata 141496maxresident)k
0inputs+0outputs (0major+231minor)pagefaults 0swaps

Compare to V8 via node:

$ guix shell node time -- time node js-runtime/run.js -- js-runtime/wtf8.wasm examples/quads.wasm
Making quad tree of depth 10 (1398101 nodes).
	construction: 64.524 msec
Allocating garbage tree of depth 9 (349525 nodes), 60 times, validating live tree each time.
	allocation loop: 2288.092 msec
	quads test: 2394.361 msec
total_heap_size: 156798976
[...]
3.74user 0.24system 0:02.46elapsed 161%CPU (0avgtext+0avgdata 382992maxresident)k
0inputs+0outputs (0major+87866minor)pagefaults 0swaps

Which is to say, wastrel is almost three times as fast, while using almost three times less memory: 2460ms (v8) vs 849ms (wastrel), and 383MB vs 141 MB.

zowee!

So, yes, the V8 times include the time to compile the wasm module on the fly. No idea what is going on with tiering, either, but I understand that tiering up is a thing these days; this is node v22.14, released about a year ago, for what that’s worth. Also, there is a V8-specific module to do some impedance-matching with regards to strings; in Wastrel they are WTF-8 byte arrays, whereas in Node they are JS strings. But it’s not a string benchmark, so I doubt that’s a significant factor.

I think the performance edge comes in having the program ahead-of-time: you can statically allocate type checks, statically allocate object shapes, and the compiler can see through it all. But I don’t really know yet, as I just got everything working this week.

Wastrel with GC is demo-quality, thus far. If you’re interested in the back-story and the making-of, see my intro to Wastrel article from October, or the FOSDEM talk from last week:

Slides here, if that’s your thing.

More to share on this next week, but for now I just wanted to get the word out. Happy hacking and have a nice weekend!

Igalia Compilers Team: Igalia’s Compilers Team - A 2025 Retrospective

2026-02-06T00:00:00+00:00

Hey, hey, it’s the beginning of a new year and before we sprint too far into 2026, let’s take a quick breather, zoom out, and celebrate what Igalia’s awesome compilers team got up to in 2025. Over the past year we’ve been deeply involved in shaping and shipping key Web and JavaScript standards, which includes not just participating in committees but also chairing and actively moving the proposals forward. We worked on major JavaScript runtimes and foundational ahead-of-time compilers including LLVM and Mesa, as well as JIT CPU emulation, and smaller language VMs.

Some big highlights of this year included our work on FEX and Mesa that helped Valve with their upcomimg gaming devices - the Steam Frame and the Steam Machine (we talk more about this in a dedicated blog post), our continued involvement in supporting RISC-V in contemporary compilers, and our key role in multiple WebAssembly implementations.

Standards #

In 2025, our standards work focused on parts of JavaScript developers touch every day like time, numbers, modules and more. Across TC39, WHATWG, WinterTC and internationalization ecosystems, we helped move proposals forward while turning specifications into running, interoperable code. So yep, let’s talk about our most significant standards contributions from the year!

Temporal #

It’s been an exciting year for the Temporal proposal, which adds a modern date-and-time API to JavaScript. For starters, MDN published their API documentation for it, which created a huge surge of interest.

On the shipping front: Firefox shipped their implementation of the proposal and it’s now available in Firefox 139. Chrome moved their implementation to beta in late 2025, and released it in early 2026. Meanwhile, we’ve been steadily working on getting Temporal into Safari, with support for correct duration math and the PlainMonthDay and PlainYearMonth types added during 2025/early 2026. You can read more about this in our recent post on implementing Temporal.

Alongside that, we’ve been working on the Intl Era and Month Code proposal, which has expanded in scope beyond era codes and month codes to cover other calendar-specific things that a JS engine with Intl must implement. This allows developers to make use of a number of commonly-used non-Gregorian calendars, including but not limited to the calendar used in Thailand, the Japanese Imperial calendar, and Islamic calendars.

Decimal #

A lot of our recent work around the Decimal proposal has now migrated to a newer similarly number-focused effort called Amount (formerly known as "Measure" and officially renamed in 2025). The proposal reached Stage 1 at the November 2024 TC39 plenary. We also launched a polyfill. Since then, we have iterated on the Amount API and data model a number of times in plenary. So while it started 2025 at stage 1 and remains at stage 1 heading into 2026, the design is noticeably sharper, thanks to a lot of TC39 discussions. We’re lined up to keep it pushing forward next year.

And because numerics work benefits a ton from regular iteration, in late 2024, we also kicked off a biweekly community call ("JS Numerics") for those in TC39 interested in proposals related to numbers, such as Decimal, Amount, intl-keep-trailing-zeros, etc. We still host it, and it’s turned out to be a genuinely productive place to hash things out without waiting for plenary.

Source Maps #

We implemented draft range mappings implementations on a number of systems: WebKit, Justin Ridgewell’s source map decoder, a source map validator, and more.

We also facilitated source map TG4 meetings and assisted with advancing proposals such as the scopes proposal. Throughout the year, we continued serving as editors for the ECMA-426 specification, landing a steady stream of improvements and clarifications.

Modules #

We pushed JavaScript’s module system forward on multiple fronts, especially around reducing the impact of modules on application startup:

we advanced the import defer proposal, which allows modules to be be synchronously lazily evaluated, to Stage 3 in TC39. We are working on its implementations in V8 and WebKit, and we implemented it in Babel, webpack (together with other community members) and TypeScript.
we presented export defer and pushed it to Stage 2 in TC39: it allows more granular lazy evaluation, as well as built-in browser support for tree-shaking of re-exports.

We are among the most active members of the "Modules Harmony" group, an unofficial group within TC39 that aims at improving the capabilities of ESM to improve native adoption, while making sure that all modules proposals are well-coordinated with each other.

AsyncContext #

And over in the AsyncContext proposal world, we spent 2025 focusing on how the proposal should integrate with various web APIs. The way AsyncContext interacts with the web platform is unusually pervasive, and more challenging to figure out than the core TC39 proposal itself.

In a first for a TC39 proposal, it is not also going through the WHATWG stages process, where it has reached Stage 1. This gives us a clearer path to iterate with direct feedback from browser engines.

Unicode standards #

We have been working on Unicode MessageFormat, which is a Unicode standard for localizable dynamic message strings, designed to make it simple to create natural sounding localized messages.

In 2025, we helped the ICU4C implementation of Unicode MessageFormat align with ongoing specification changes. We also carried out experimental work on the custom function interface to support more extensible formatting formatting capabilities, which is currently under review.

WinterTC #

In December 2024, WinterTC was formed to replace WinterCG as an official ECMA Techincal committee to achieve some level of API interoperability across server-side JavaScript runtimes, especially for APIs that are common with the web.

We started chairing (together folks from Deno), and became involved in admin tasks. Over the course of the year, we:

Identified a core set of Web APIs that should be shared across runtimes and standardized it as the Minimum Common Web API specification, which was officially published at the ECMA General Assembly in December.
Started identifying a subset of the WPT test suite that covers the Minimum Common Web API, and made some headway towards clarifying which parts of the Fetch specification server-side runtimes should support, and which they shouldn’t.

Additionally, if you’re curious, we gave two talks about WinterTC: one at the Web Engines Hackfest together with Deno folks, the other chair of WinterTC; and one at JSConf.JP.

Node.js #

In Node.js, our work in 2025 spanned interoperability, proxy integration, and adding support for HTTP/HTTPS proxy and shipping integration of System CA certificates across platforms.

On the module side, we delivered interoperability features and bug fixes for require(esm) and helped stabilize it (read more about it in our colleague Joyee’s blog), shipped synchronous and universal loader hooks (now promoted to release candidate), integrated TypeScript into the compile cache, and improved the portability of the cache. Check out Joyee’s talk at JSConf JP if you are interested in learning more about these new module loader features.

We also strengthened System CA certificate integration along with JavaScript APIs for reading and configuring trusted CAs globally, adding built-in HTTP/HTTPS proxy support, and expanding documentation for using Node.js in enterprise environments.

Additionally, we started migration to the new V8 CppHeap model in Node.js and improved its V8 Platform integration.

V8 #

On the V8 side of things, we worked on HeapProfiler::QueryHolders, a companion API to the QueryObjects API.

We worked on extending the HeapStatistics API to include a new field that tracks the total of bytes allocated in an Isolate since its creation. This counter excludes allocations that happen due to GC operations and it’s intended to be used to create memory regression tests. Here’s the CL highlighting these changes.

We also started working on implementation of the import defer proposal on V8. This proposal extends the syntax of ESM imports to allow a mode where the evaluation of an imported module is deferred until its first access. From our work in Node.js, we upstreamed a few improvements and bug fixes in V8’s embedder API and startup snapshot implementation. We also contributed to Node.js’s V8 upgrade and upstreamed patches to address issues discovered in the upgrade.

As part of our collaboration with Cloudflare we added v8::IsolateGroup: a new unit that owns an independent pointer-compression cage. We then also enabled multiple cages per process (“multi-cage”), so thousands of isolates aren’t forced into one < 4 GiB region. Finally, we extended this to multiple sandboxes: one sandbox per isolate group instead of a single process-wide sandbox. In the end this work helped Cloudflare to enable the sandbox in Cloudflare workers.

Babel #

Our team also helps co-maintianing Babel. The build tools area is very active nowdays, and we strongly believe that alongside the innovation happening in the ecosystem companies need to invest on ensuring that the older and widely used tools keep being actively maintained and improving over time.

LLVM #

In LLVM, we helped extend auto-vectorization to take full advantage of the RISC-V vector extension’s many innovative features.

After four years of development by contributors from multiple organizations including Igalia, we finally enabled EVL tail folding for RISC-V as an LLVM default.

This work took advantage of the new VPlan infrastructure, extending it and developing it iteratively in-tree when needed to give us the ability to model a relatively complex vectorization scheme.

We also added full scalable segmented access support and taught the loop vectorizer to make smarter cost model decisions.

Building on top of this, we achieved improvements in RISC-V vectorization. In parallel, we also worked on LLVM scheduling models for the SpacemiT-x60 RISC-V processor, scoring a whopping 16% performance improvement.

Regarding WebAssembly in LLVM we landed a number of commits that improve size and performance of generated code, and added support for a few ISD nodes that enable vectorization for otherwise sequential codegen.

Mesa/IR3 #

We continued work on improving IR3, the Mesa compiler backend for Qualcomm Adreno GPUs. We implemented support for alias instructions novel to the a7xx generation of GPUs, significantly improving register pressure for texture instructions. We also refactored the post-RA scheduler to be able to reuse the legalization logic, significantly improving its accuracy when calculating instruction delays and, consequently, reducing latency.

We also added debug tooling to easily identify the shader that causes problems, among many other optimizations, implementations of new instructions, and bug fixes.

Guile and Whippet #

This year we also made some interesting progress on Whippet, a no-dependencies embeddable garbage collector. We were able to integrate Whippet into the Guile Scheme implementation, replacing Guile’s use of the venerable Boehm-Demers-Weiser library. We hope to merge the integration branch upstream over the next months. We also wrote up a paper describing the innards of some of Whippet’s algorithms.

We think Whippet is interesting whereever a programming language needs a garbage collector: it’s customizable and easy to manage, as it is designed to be "vendored" directly into a user’s source code repository. We are now in the phase of building out examples to allow for proper performance evaluation; after a bespoke Scheme implementation and Guile itself, we also wrote a fresh ahead-of-time compiler for WebAssembly, which in the near future will gain support for the garbage collection WebAssembly extensions, thanks to Whippet. For more info on our progress, check out Andy Wingo’s blog series.

FEX #

For the FEX x86 JIT emulator for ARM64, we worked on X87 Floating-Point Emulation, implemented x87 invalid operation bit handling in F80 mode, fixed IEEE 754 unordered comparison detection, and added f80 stack xchg optimization for fast path.

Besides further fixes for instruction implementations, we also worked on memory and stability improvements, protecting the last page of CodeBuffer, and implementing gradual memory growth. Finally, we also did some infrastructure work by upgrading the codebase to clang-format-19 and adding UBSAN support.

This year’s FEX work focused on x87 floating-point correctness and 32-bit compatibility—both critical for Valve’s Steam Frame, the ARM-powered VR headset they announced in November that uses FEX to run x86 games.

The x87 improvements matter because many games and middleware still use legacy floating-point code. Subtle deviations from Intel’s behavior—wrong exception flags, incorrect comparison semantics—cause crashes or weird behavior. Fixing invalid operation exceptions, IEEE 754 comparisons, and optimizing the x87 stack pass eliminated entire classes of compatibility bugs.

The 32-bit fixes are just as important. A huge chunk of Steam’s catalog is still 32-bit, and even 64-bit games often ship 32-bit launchers. Getting fcntl and addressing modes right means these games just work without users needing to do anything.

In total, this work gave Valve confidence that the Steam Frame could ship with solid library coverage, letting them announce the device on schedule.

Alright, that’s a wrap on our 2025 retrospective! We hope you had as much fun reading it as we had writing it, and building all the things we talked about along the way. We’ll see you next year with another roundup; until then, you can keep up with our latest work on the team blog.

Manuel Rego: FOSDEM 2026

2026-02-05T00:00:00+00:00

Last weekend I was in Brussels attending FOSDEM. A big event with lots of people and lots of things happening in parallel, where it’s impossible to be everywhere.

Browser and web platform 🌐 #

My main participation was in the Browser and web platform devroom, a new devroom (it seems it already happened back in the days, but not in the recent years) where I had the chance to speak about Servo with a talk titled The Servo project and its impact on the web platform ecosystem. My colleagues from Igalia Eri Pazos and Mario Sánchez Prada were also speaking in that devroom about Interop and MathML Core and The Web Platform on Linux devices with WebKit: where are we now? respectively. The room was fully packed a big part of the day, not unexpectedly many people are interested in the web platform.

MathML #

Eri was the first igalian talking in the room, they summarized the work Igalia has been doing for many years in MathML, on the standards side proposing the MathML Core spec, and on the implementation side bringing it back to Chromium and improving it in Gecko and WebKit.

In the talk, Eri went into deep detail about the last additions we have been adding around MathML: math-depth, math-shift, RTL mirroring, font-family: math, etc. This work is part of an agreement with the Sovereign Tech Fund, big thanks for your support.

MathML is more ready than ever for production, someone from arXiv.org in the audience mentioned that they are shipping it on millions of webpages today. Waiting for the day when Wikipedia switches to it by default, it will be a huge milestone.

WebKit on Linux #

Mario talked about the Linux ports of WebKit: WebKitGTK and WPE, both maintained by Igalia.

He explained what they are, the differences between them, reviewed their history and highlighted the big progress on the recent years, with multiple improvements in several areas: WebPlatform API, WebKit Container SDK, switch from Cairo to Skia graphics library, etc.

If you are curious about the status of things regarding them, you shouldn’t miss his talk.

Servo #

My talk started with an introduction to the Servo project and the current status of things. I showed a few demos about how Servo works and some of the things it can do already. After that introduction, I explained how Servo has been contributing to the wider web platform ecosystem.

Like for the rest of talks, slides and video are already available if you want to know all the details. Kudos to the organization for being so quick.

Picture of my talk with the slides about conclusions at the back

As an anecdote, the night before the talk a new project based on Servo was published, a browser developed fully with web technologies using Servo underneath. I couldn’t resist the urge to build it, play with it and add it to the presentation. It looks really cool what Servo can do these days.

Screencast of the new Beaver browser based on Servo

In addition, in the same devroom there was another Servo talk, this time by Taym, one of the Servo maintainers, Implementing Streams Spec in Servo web engine, where he explained all the work behind adding Streams support to Servo.

I am also very happy to met many Servo contributors at the event: Delan Azabani (@delan), Eri Pazos (@eerii), Jonathan Schwender (@jschwe), Martin Robinson (@mrobinson), Taym Haddadi (@Taym95), Tim van der Lippe (@TimvdLippe). We had the chance to have some informal conversations about the project, discussing some technical topics and ideas about things we can do in Servo.

The feedback about Servo has been extremely positive, people are really happy with the evolution of the project and excited about the future.

Apart from that, we also had the opportunity to talk to the nice folks from NLnet and the Sovereign Tech Agency who both have ongoings collaborations around Servo. The work these organizations do is really important for the open software development, and more should learn from them and join forces to try to fix the funding issues in FLOSS (more about this later, when talking about Marga’s keynote).

Igalia #

Igalia presence in the open source community is very big, and not unexpectedly we have more talks at FOSDEM this year. This is the full list:

Also our work in different projects was mentioned in several talks and conversations, we’re really happy regarding all the good feedback we got about Igalia contributions.

Keynote #

Marga Manterola was doing one of the keynotes talking about funding open source software: Free as in Burned Out: Who Really Pays for Open Source? . First time Igalia was doing a keynote at FOSDEM, in the year we celebrate our 25th anniversary, and in a historical auditorium where May 68 started in Brussels. 🎉

Picture of Marga Manterola's keynote at FOSDEM 2026

The talk was great, Marga explained many of the issues with open source software sustainability and some potential ideas about how to improve the situation. This is a recurring topic in many conversations these days, we should find a way to get this fixed somehow.

There I learnt about the Open Source Pledge, an interesting initiative to get companies donating 2,000 USD per developer to open source software maintainers. 💰

Wrap up #

All in all, it was a nice but very busy weekend in Brussels, weather was ok (a bit cold but not rainy) and waffles were delicious as usual. 🧇😋

Next big event on my calendar is the Web Engines Hackfest in June, more than 50 people have already registered and a bunch of Servo folks will be there too. If you’re interested in the web platform and willing to discuss about different topics, we would be very happy to host you there.

Brian Kardell: What if we just

2026-02-04T05:00:00+00:00

What if we just

What if a better answer to a question I've been struggling with for more than a decade is just... Way simpler? Sharing a potentially half-baked idea for discussion.

Back in 2013 I wrote Dropping the F-Bomb on Web Standards. The core argument was simple: the web works best when developers can invent “slang,” and standards bodies behave more like dictionary editors — watching what people actually say, then paving the cow paths that clearly matter.

It fed into the Extensible Web Manifesto (which followed) and over the years I've continued to push for study of what people are really doing. I have helped add features to the HTTPArchive crawl and built tools to analyze this data.

But it's hard. It's biased. It's incomplete. Even the best crawl misses huge swaths of the web — anything behind logins, paywalls, dashboards, internal tools, or private deployments. And all of them have limits. It requires a ton of follow-up analysis and raises almost as many questions as it answers.

So lately I've been wondering (a bit like Kramer):

What if we just... voluntarily shared this information?

We don't need a formal standard or anyone's permission, we could just... share it, and build tools to share it easily in a well known format at a well known URL.

It could give us insight into the use of custom elements behind logins and paywalls and so on too, and tell us where they come from (a git repo, for example)...

Lots of things that are common happened through community effort and adoption. Normally you get something from it - robots.txt helped your site from being aggressively scraped in problematic ways, ads.txt helped say something about monetization, feed.rss helped syndicate, and so on. What do you get out of sharing this kind of info?

Individually, I'm not sure. But, collectively the benefit is clear: We'd finally have a real, ecosystem‑wide index of custom elements and how they're used. and hopefully a way to shape useful standards on them easily.

As to what that would look like, I'm not sure.

The community defined Custom Element Manifest already has a bit of uptake and tooling - we could just publish that to a well known URL. It might be too much, or too little.. A simpler manifest of just element names and URLs of packages/repositories that supply them would even be nice.

Is it too much? Too little? Maybe.

Is it worth trying? I think so.

What do you think? Is this worth trying somehow?

Igalia WebKit Team: WebKit Igalia Periodical #55

2026-02-02T20:11:18+00:00

Update on what happened in WebKit in the week from January 26 to February 2.

A calm week for sure! The highlight this week is the fix for scrolling not starting when the main thread is blocked.

Cross-Port 🐱

Graphics 🖼️

Fixed the problem of wheel event async scrolling doesn't start while the main thread is blocked. This should make WebKit feel more responsive even on heavier websites.

That’s all for this week!

Brian Kardell: Maintaining the Bridges

2026-02-02T05:00:00+00:00

Maintaining the Bridges

Thoughts and analogies about infrastructure...

I live in Pittsburgh, Pennsylvania — “The Steel City,” once the beating heart of American steelmaking. In 1902, U.S. Steel’s first full year of operation, it produced 67% of all steel in the United States. By 1943, the company employed more than 340,000 people. We burned so much coal that Pittsburgh earned the nickname “Hell with the lid off.” Streetlights sometimes ran at noon because the sky was that dark.

A photo of Pittsburgh dark with smoke at midday. (You can search more about this it if that interests you, here's one nice piece with a few pictures).

The city’s geography didn’t make things any easier. Pittsburgh is carved by mountains, valleys, and the three rivers — the Allegheny and Monongahela merging to form the Ohio. That topography, combined with the industrial boom, meant we built a lot of bridges. It helps that when your city is literally manufacturing the materials, you get a hometown discount.

A view down river of Pittsburgh's 3 sisters" bridges and several others.

One of them, the Hot Metal Bridge — just a mile or two from my house — once carried ladle cars full of molten iron between the blast furnaces and mills of J&L Steel. During World War II, 15% of America’s steelmaking capacity crossed that bridge, up to 180 tons per hour.

These bridges were originally built by private companies with a clear profit motive: to move coal, ore, steel, or workers. Others were toll bridges, run by private companies the way you’d run a turnpike or ferry.

But more bridges meant more industry, which meant more people, which meant more bridges. You can see where this goes.

Even by the late 1800s we were beginning to publicly fund them. By the 1920s–1930s Allegheny County’s bridge program bought out many of the private bridges and replaced many of them. By the time the New Deal and Interstate era arrived, the private‑toll era was basically over - and since then over 90% of Pittsburgh's public bridges were funded by federal programs (we still have some private industry use bridges).

So what does any of this have to do with software?

Aside from giving me an excuse to talk about my city (which I enjoy), Pittsburgh’s bridges are a useful metaphor for the infrastructure we rely on in tech, in two important ways:

Becoming a public good Private investment built early bridges, just like private companies built much of what we've got now in terms of browser engines, search index, foundational libraries and so on - but eventually they stopped becoming optional. I think we're only now starting to really understand that we need a lot of this to be a public good in the same kind of way. These are the roads and bridges of the modern world.
Building something new is exciting. Maintaining it, not so much. A lot of my city's physical infrastructure is aging and some of it has been neglected. It's somehow way easier to get people to build new things than to take care of the old stuff. The public notices a new bridge! The ribbon-cutting gets a photo op and celebration. The maintenance budgets and crews struggle to even get funding.

In fact, even when things are fairly well funded, it doesn't mean they're kept up to date. While researching to write this piece I realized that a lot of the Wikipedia data about Pittsburgh (and many topics!) is actually really out of date. It's cool to write the article with these cool facts, but it's not so cool to do the work to keep it up... Or, maybe thats just not what you want to do anymore. Or maybe you were incarcerated, or you died, or you went to Mars - idk.

The point is that writing the thing in the first place is only half the battle. If most of your entry on a city was written two decades ago, a lot of what it details about the economics, population, jobs, and so on are probably not very accurate!

It's no different with software. It's cool and fun to build a new thing or add a new feature to an existing thing, but keeping them maintained is annoying. New mechanisms arrive that you might need to adapt to. Underlying code bit rots. All of it needs release teams and Q&A and reviews and fixes and updates at global scales, even if no new features were added. But very few people actually want to do that, and almost nobody wants to pay for it.

More Public Funding

I'd really love for societies around the world to come to the realization that a lot of the online things we've built are, like roads and bridges, now necessary - and figure out how we can publicly fund enough of them that important things without an obvious and direct profit motive can get done. MathML and SVG are two easy examples of this, but there are plenty more. Maybe XSLT is another example. Perhaps if we had good funding for those things, their ongoing survival wouldn't be questioned.

I feel like there is a lot of room here for improvement from the status quo. It doesn't even have to start with governments. Any ways that we expand the pool of funding avilable and diversifying, it helps.

Igalia Compilers Team: Implementing the Temporal proposal in JavaScriptCore

2026-02-02T00:00:00+00:00

Image source

For the past year, I've been working on implementing the Temporal proposal for date and time handling in JavaScript, in JavaScriptCore (JSC). JavaScriptCore is the JavaScript engine that's part of the WebKit browser engine. When I started, Temporal was partially implemented, with support for the Duration, PlainDate, PlainDateTime, and Instant types. However, many test262 tests related to Temporal didn't pass, and there was no support for PlainMonthDay, PlainYearMonth, or ZonedDateTime objects. Further, there was no support for the relativeTo parameter, and only the "iso8601" calendar was supported.

Duration precision (landed) #

Conceptually, a duration is a 10-tuple of time components, or a record with the fields "years", "months", "weeks", "days", "hours", "seconds", "milliseconds", "microseconds", and "nanoseconds".

One way durations are used is to represent the difference between two dates. For example, to find the length of time from a given date until the end of 2027, I could write the following JS code:

> const duration = (new Temporal.PlainDate(2026, 1, 26)).until(new Temporal.PlainDate(2027, 12, 31), { largestUnit: "years" })
> duration
Temporal.Duration <P1Y11M5D> {
  years: 1,
  months: 11,
  days: 5
}

The until method in this case returns a duration comprising one year, eleven months, and five days. Because durations can represent differences between dates, they can also be negative:

> const duration = (new Temporal.PlainDate(2027, 12, 31)).until(new Temporal.PlainDate(2026, 1, 26), { largestUnit: "years" })
> duration
Temporal.Duration <P1Y11M5D> {
  years: -1,
  months: -11,
  days: -5
}

When converted to nanoseconds, the total of days, hours, minutes, seconds, milliseconds, microseconds, and nanoseconds for a duration may be a number whose absolute value is as large as 10⁹ × 2⁵³. This number is too large to represent either as a 32-bit integer or as a 64-bit double-precision value. (If you're wondering about the significance of the number 2⁵³, see the MDN documentation on JavaScript's MAX_SAFE_INTEGER.)

To understand why we need to be able to work with such large numbers, consider totaling the number of nanoseconds in a duration. Following on the previous example’s definition of the variable duration:

> duration.total({unit: "nanoseconds", relativeTo: new Temporal.PlainDate(2025, 12, 15)})
60912000000000000

There are 60912000000000000 nanoseconds, or about 6.1e16, in a period of one year, eleven months, and five days. Since we want to allow this computation to be done with any valid start and end date, and valid years in Temporal range from -271821 to 275760, the result can get quite large. (By default, Temporal follows the ISO 8601 standard for calendars, which entails using a proleptic Gregorian calendar. Also note that this example uses a PlainDate, which has no time zone, so computations are not affected by daylight savings time; when computing with the Temporal ZonedDateTime type, the specification ensures that time zone math is done properly.)

To make it easier for implementations to fulfill these requirements, the specification represents durations internally as Internal Duration Records and converts between JavaScript-level duration objects and Internal Duration Records (which I'll call "internal durations") as needed. An internal duration pairs the date component of the duration (the years, months, weeks, and days fields) with a "time duration", which is a single integer that falls within an accepted range, and can be as large as 2⁵³ × 10⁹ - 1.

Implementations don't have to use this representation, as long as the results are observably the same as what the specification dictates. However, the pre-existing implementation didn't suffice, so I re-implemented durations in a way that closely follows the approach in the specification.

This work has been landed in JSC.

New date types #

Temporal's date types include PlainDate, PlainDateTime, Instant, ZonedDateTime, PlainMonthDay, and PlainYearMonth. The latter two represent partial dates: either a pair of a month and a day within that month, or a pair of a year and month within that year. Partial dates are a better solution for representing dates where not all of the fields are known (or not all of the fields matter) than full dates with default values for the missing bits.

Temporal's ZonedDateTime type represents a date along with a time zone, which can either be a numeric offset from UTC, or a named time zone.

I implemented PlainMonthDay and PlainYearMonth with all their operations. ZonedDateTime is fully implemented and the first pull request in a series of PRs for it has been submitted.

The relativeTo parameter #

What if you want to convert a number of years to a number of days? Temporal can do that, but there's a catch. Converting years to days depends on what year it is, when using the ISO 8601 calendar (similar to the Gregorian calendar), because the calendar has leap years. Some calendars have leap months as well, so converting years to months would depend on what year it is as well. Likewise, converting months to days doesn't have a consistent answer, because months vary in length.

For that reason, the following code will throw an exception, because there's not enough information to compute the result:

> const duration = Temporal.Duration.from({ years: 1 })
> duration.total({ unit: "days" })
Uncaught RangeError: a starting point is required for years total

The above definition of duration can still be made to work if we pass in a starting point, which we can do using the relativeTo parameter:

> duration.total({ unit: "days", relativeTo: "2025-01-01" })
365
> duration.total({ unit: "days", relativeTo: "2024-01-01" })
366

The string passed in for the relativeTo parameter is automatically converted to either a PlainDate or a ZonedDateTime, depending on which format it conforms to.

I implemented support for the relativeTo parameter on all the operations that have it; once the implementations for all the date types land, I'll be submitting this work as a series of pull requests.

Calendars #

Representing dates with non-ISO8601 calendars is still very much a work in progress. The ICU library can already do the basic date computations, but much glue code is necessary to internally represent dates with non-ISO8601 calendars and call the correct ICU functions to do the computations. This work is still underway. The Temporal specification does not require support for non-ISO8601 calendars, but a separate proposal, Intl Era Month Code, proposes a set of calendars to be supported by conformant implementations.

Testing Temporal #

The JavaScript test suite is called test262 and every new proposal in JavaScript must be accompanied by test262 tests. Not all JS implementations are required to support internationalization, so Temporal tests that involve non-ISO calendars or named time zones (other than the UTC time zone) are organized in a separate intl402 subdirectory in test262.

The test262 suite includes 6,764 tests for Temporal, with 1,791 of these tests added in 2025. Igalia invested hundreds of hours on increasing test coverage over the past year.

Status of work #

All of this work is behind a flag in JSC in Technology Preview, so to try it out, you'll have to pass the --useTemporal=1 flag.

All of the implementation work discussed above (except for non-ISO calendars) is complete, but I've been following an incremental approach to submitting the code for review by the JSC code owners. I've already landed about 40 pull requests over the course of 2025, and expect to be submitting at least 25 more to complete the work on PlainYearMonth, ZonedDateTime, and relativeTo.

Based on all the code that I've implemented, 100% of the non-intl402 test262 tests for Temporal pass, while the current HEAD version of JSC passes less than half the tests.

My colleagues at Igalia and I look forward to a future JavaScript standard that fully integrates Temporal, enabling JavaScript programs to handle dates more robustly and efficiently. Consistent implementation of the proposal across browsers is a key step towards this future. Step by step, we're getting closer to this goal.

We thank Bloomberg for sponsoring this work.

Qiuyi Zhang (Joyee): Tickering with Node.js Core on ARM64 Windows

2026-01-31T10:25:49+00:00

A while back, I wrote about Building Node.js on Windows using the new ClangCL support, which was done on an actual x64 Windows machine

Manuel Rego

2026-01-30T00:00:00+00:00

On my way to FOSDEM where tomorrow I’ll be talking about Servo in the Browser and web platform devroom. See you there!

Alice Boxhall: Reference Target: having your encapsulation and eating it too

2026-01-30T00:00:00+00:00

Three years ago, I wrote a blog post about How Shadow DOM and accessibility are in conflict.

I explained how the encapsulation provided by shadow roots is a double-edged sword, particularly when it comes to accessibility. Being able to programmatically express relationships from one element to another is critical for creating user experiences which don’t rely on visual cues - but elements inside a shadow root aren’t available to be referenced from elements in the light DOM. This encapsulation, however, is what allows component authors to create accessible components which can be safely reused in any context, without necessarily requiring any particular dependencies or extra build steps.

In the year or so following, even more heroic attempts were made to square this circle, and finally one seems likely to stick: Reference Target. In this post I’ll explain how this feature works, why I like it, and what the situation is right now with the spec and implementation (thanks in part to Igalia’s NLNet funding).

A quick introduction #

referenceTarget is a new property on shadow root objects which lets you nominate an element in the shadow root’s subtree which should be the target of any attribute-based reference to the shadow host.

As an example, imagine that you have a <custom-input> component, which has an <input> tucked away in its shadow root. This is a pattern which is ubiqutous in custom element libraries, as it allows the custom element to use composition to enhance the behaviour of a built-in element.

<label for="track">Track name:</label>
<custom-input id="track">
  #shadowRoot
  | <input id="inner-input">
</custom-input>

We can set the shadow root’s referenceTarget to allow the <label> to correctly label the inner <input>:

// in the constructor for the custom-input:

const shadowRoot = this.attachShadow({mode: 'open'});
shadowRoot.referenceTarget = 'inner-input';
shadowRoot.innerHTML = '<input id="inner-input">`;

This lets the label refer to the <custom-input> just like it would refer to an <input>; the <custom-input> transparently proxies the reference through to the encapsulated <input>.

In this example, we’ve set the referenceTarget property directly on the ShadowRoot object, but it can also be set declaratively when using the <template> element to create the shadow root:

<label for="track">Track name:</label>
<custom-input id="track">
  <template shadowRootMode="open"
            shadowRootReferenceTarget="inner-input">
    <input id="inner-input">
  </template>
</custom-input>

This works equally well for any attribute which refers to other elements like this - even if you set it via a reflected property like commandForElement:

<button id="settings-trigger">Site settings</button>
<custom-dialog id="settings-dialog">
  #shadowRoot referenceTarget="inner-dialog"
  | <dialog id="inner-dialog">
  |   <button id="close" aria-label="close"
  |           commandFor="inner-dialog"
  |           command="request-close"></button>
  |   <slot></slot>
  | </dialog>
  <fieldset>
    <legend>Colour scheme:</legend>
    <label for="dark">
      <input type="radio" id="dark" name="appearance" value="dark" checked>
      Dark
    </label>
    <!-- TODO: more colour schemes -->
  </fieldset>
</custom-dialog>

// Someone probably has a good reason why they'd do it this way, right?

const settingsButton = document.getElementById('settings-trigger');
settingsButton.command = 'show-modal';
settingsButton.commandForElement = document.getElementById('settings-dialog');

This lets the <custom-dialog> behave exactly like a <dialog> for the purposes of the command and commandForElement properties.

Why I like it #

In my earlier blog post I explained that I was concerned that the Cross-root ARIA delegation and reflection proposals introduced a bottleneck problem. This problem arose because it was only possible to refer to one element per attribute, rather than allowing arbitrary cross-shadow root references.

This proposal absolutely doesn’t solve that problem, but it reframes the overall problem such that I don’t think it matters any more.

The key difference between reference target and the earlier proposals is that reference target is a catch-all for references to the shadow host, rather than requiring each attribute to be forwarded separately. This solves a specific problem, which I alluded to above: how can custom element authors encapsulate the behaviour of a given built-in HTML element while also allowing other elements to refer to the custom element as if it was the built-in element?

I believe this more narrow problem definition accounts for a significant proportion - not all, but many - cases where references need to be able to cross into shadow roots. And it makes the API make much more sense to me - if you’re using the for attribute to refer to a <custom-input>, you’re not meant to need to know that you’re actually referring to an enclosed <input>, you just want the <custom-input> to be labelled. This API makes the enclosed <input> an implementation detail. And since a shadow root can only have one host, it makes sense that it can only have one reference target.

Adjacent, as-yet unsolved problems #

Arbitrary cross-shadow root references #

As mentioned above, one adjacent problem is the problem of element references which do need to refer to specific elements within a shadow root, rather than a stand-in for the shadow host.

The explainer gives two examples of this: aria-activedescendant on a combobox element which needs to refer to an option inside of a shadow root, and ARIA attributes like aria-labelledby, aria-describedby and aria-errormessage which may need a computed name for the component which excludes some parts.

I think we need to be careful about generalising this problem, though. As I describe later in the explainer, I think we might be able to get better solutions by solving more specific problems - as we have with reference target.

If you have another example of where you need to refer to specific elements within a shadow root, you can leave a comment on this issue collecting use cases.

Attribute forwarding #

While reference target allows other elements to refer to the encapsulated element, custom element authors may also want to allow developers using their component to use standard HTML and ARIA attributes on the host element and have those apply to the encapsulated element.

For example, you might like to support popoverTarget on your <custom-button> element:

<custom-button popoverTarget="languages">Language</custom-button>
<custom-menu id="languages" popover>
  <custom-menuitem>Nederlands</custom-menuitem>
  <custom-menuitem>Fryslân</custom-menuitem>
  <custom-menuitem>Vlaams</custom-menuitem>
</custom-menu>

There is an issue for the attribute forwarding idea; leave a comment there if this is an idea you’d like to see pursued.

Form association #

Custom elements can be specified as form-associated, but there’s no way to associate an encapsulated form-associated built-in element (such as <input>) with an enclosing <form>.

For example, the <custom-input> above could be nested in a <form> element, but the enclosed <input> wouldn’t be associated with the <form> - instead, you’d have to use setFormValue() on the custom element and copy the value of the <input>.

Spec and implementation status #

In brief: the spec changes seem to be in good shape, Chromium has the most feature-complete implementation and there are significantly less-baked implementations in WebKit and Firefox.

Spec changes #

There are open pull requests on the HTML and DOM specs. Since these PRs are still being reviewed, the concepts and terminology below might change, but this is what we have right now. These changes have already had a few rounds of reviews, thanks to Anne van Kesteren, Olli Pettay and Keith Cirkel.

The DOM change:

adds the concept of a reference target
adds the referenceTarget property to the ShadowRoot object.

The HTML change is where the actual effect of the reference target is defined.

Element reference attribute type #

One key change in the HTML spec is the addition of an attribute type for “element reference” attributes. This formalises in HTML what has previously been referred to as an ID reference or IDREF. This term isn’t currently used in HTML, and since the addition of reflected IDL Element attributes, IDs aren’t strictly necessary, either.

Before this change, whenever an attribute in the HTML spec was required to match another element based on its ID, this was written out explicitly where the attribute was defined. For example, the definition of the <label> element’s for attribute currently reads:

The for attribute may be specified to indicate a form control with which the caption is to be associated. If the attribute is specified, the attribute’s value must be the ID of a labelable element in the same tree as the label element. If the attribute is specified and there is an element in the tree whose ID is equal to the value of the for attribute, and the first such element in tree order is a labelable element, then that element is the label element’s labeled control.

Since reference target affects how this type of reference works, and is intended to apply for every attribute which refers to another element, it was simpler to have one central definition.

Reference target resolution #

For a reference target to actually do something, we need to define what effect it has. This is defined, quite straightforwardly, in the steps to resolve the reference target:

If element is not a shadow host, or element’s shadow root’s reference target is null, then return element.

Let referenceTargetValue be the value of element’s shadow root’s reference target.

Let candidate be the first element in element’s shadow root whose ID matches referenceTargetValue.

If no such element exists, return null.

Return the result of resolving the reference target on candidate.

These steps are recursive: if a shadow root’s reference target has its own shadow root, and that shadow root has a reference target, we keep descending into the nested shadow root.

One slightly subtle design choice here is that if a shadow root has a reference target which doesn’t refer to any element - for example, an empty string, or a value which doesn’t match the ID of any element in its subtree - the resolved reference target is null, not the shadow host.

For example, if you tried to use popoverTarget to refer to a shadow host which had a popover attribute, but had an invalid reference target on its shadow root, the popoverTarget attribute won’t actually target anything:

<button id="more-actions" popoverTarget="actions-popover" aria-label="more actions">…</button>

<!-- Even though this has a popover attribute, the button won't toggle it! -->
<custom-popover id="actions-popover" popover>
  <template shadowRootMode="open"
            shadowRootReferenceTarget="0xDEADBEEF">
    <div id="help-im-trapped-in-a-shadow-root" popover>
      <slot></slot>
    </div>
  </template>
</custom-dialog>

Resolved and unresolved attr target elements #

Like many spec concepts, this one is a real mouthful.

This lets us be very clear about whether reference target resolution has happened when we’re talking about what element an attribute refers to.

If we’re reflecting an attribute to its IDL counterpart, we now use the unresolved attr target element. For example, if we had the DOM defined in the previous example, and we wanted to get the popoverTargetElement for the "settings-trigger" button:

const moreActions = document.getElementById("more-actions");

// This will log the <custom-popover> element (!)
console.log(moreActions.popoverTargetElement);

(In spec terms: the <custom-popover> element is the unresolved popoverTarget target element for the <button>.)

This might also be a bit surprising; we spent quite a bit of time going back and forth on this, since we thought developers might want to know that the popoverTarget isn’t actually targeting anything. However, using the unresolved target lets us have a very close parallel between setting and getting the popoverTargetElement, as well as preserving the shadow root’s encapsulation.

The resolved attr target element, meanwhile, is what will be used when actually doing something with the attribute - such as triggering a popover, or computing a label’s labeled control, or determining an element’s accessible description.

In the above example, the resolved popoverTarget target element for the button is null. And, going back to the examples we’ve seen earlier:

the resolved commandFor target element for the Settings button is the inner <dialog> - clicking the button will open the <dialog>.
the resolved for target element for the <label> is the inner <input> - clicking the label will focus the input, and the input’s computed accessible name will be “Track name”.

“Referring to” #

For convenience, we define the concept of an attribute “referring to” an element:

A single-element reference attribute attr on an element X refers to an element Y as its target if Y is the resolved attr target element for X.

So, for example, the commandFor attribute on the Settings button refers to the inner <dialog> element.

Sets of element references #

All of the above used single element references as examples, but there are attributes which can refer to more than one element. For example, almost all of the ARIA attributes which refer to other elements refer to multiple elements in an ordered list - one such is aria-errormessage, which can refer to one or more elements which should be exposed as specifically as an error message for an element which is marked as invalid.

We define a set of element references attribute type, as well as a couple of subtypes which impose constraints such as ordering or uniqueness, as well as what it means for one of these attributes to refer to another element, and how to get the resolved and unresolved attr target elements for these attributes.

While these are slightly more complex than the single element versions, they follow the same basic logic. The only marginally significant difference is that since they produce lists of elements, if a shadow root’s reference target is invalid, no element is added to the list for that unresolved attr target, instead of returning null.

Using these concepts in the rest of the spec #

Now that we’ve defined these spec concepts, we have to update each place in the spec where we previously used the “whose ID is equal to the value of the blahblah attribute” wording.

Returning to our good friend popoverTarget, we can see a relatively straightforward example.

The definition of the popoverTarget attribute now reads:

If specified, the popovertarget attribute value must be a valid single-element reference attribute referring to an element with the popover attribute.

And now, get the popover target element determines popoverElement like this:

Let popoverElement be node’s resolved popovertarget target element.

<label> association is a bit more complex, since we wanted descendants of the <label> to be correctly labelled when using reference target:

To determine a label element label’s labeled control, run these steps:

If label’s for attribute is specified, then:

If the resolved for target element is not null, and the resolved for target element is a labelable element, return that element.

Otherwise, return null.

For each descendant descendant of label in tree order:

Let candidate be the result of resolving the reference target on descendant.

If candidate is a labelable element, return candidate.

Return null.

There is also a PR open on the ARIA spec to introduce this terminology there.

Implementation status #

Chromium has the most complete implementation, though it may not quite be up to date with the latest spec changes. Any developers wanting to try it out should get the latest build of Chrome or Edge and flip on the Experimental Web Platform Features flag. If you do try it out, I’d love to hear any feedback you might have!

WebKit and Firefox (tracking bug) each have a prototype implementation, available behind respective feature flags (ShadowRootReferenceTargetEnabled for WebKit and dom.shadowdom.referenceTarget.enabled for Firefox), which should pass at least most of the existing WPT tests - however, the WPT tests are insufficient to test all of the functionality, and the functionality which couldn’t be tested via WPTs hasn’t been implemented yet in these engines. The Chromium implementation included adding many Chromium-specific tests for the behaviour which can’t be tested via WPTs, as well as implementing that behaviour.

Currently, WPT tests can only test the computed accessible name and computed accessible role for an element, as well as testing DOM methods and user actions like clicking. However, reference target impacts the accessibility tree in many ways - not only via ARIA attributes, but via attributes like popoverTarget being exposed in the accessibility tree as an accessible relation.

And, importantly, changes to the accessibility tree can require certain notifications to be fired to assistive technology APIs - and reference target introduces several new ways to change the accessibility tree. Adding, changing, or removing a shadow root’s referenceTarget may cause changes in the resolved target elements for attributes, causing accessibility tree changes and potentially requiring notifications. Likewise, inserting an element with an ID which matches a shadow root’s referenceTarget could also cause a shadow host’s resolved reference target to change, also potentially causing the accessibility tree to change.

There are two complementary projects currently underway which will allow us to write much richer tests for accessibility tree functionality in browsers:

Once we can write WPT tests which actually test the full spectrum of expected behaviour for reference target, we’ll be able to actually make it an official interop focus area.

The prototype implementation work in WebKit and Firefox, as well as the spec work done by Igalia, was generously funded by a grant from NLNet Foundation, while the implementation work in Chromium and much of the remainder of the spec work was done by Microsoft engineers on the Edge team.

Qiuyi Zhang (Joyee): Improving Single Executable Application Building for Node.js

2026-01-26T22:11:31+00:00

Recently, I landed a change that moves the Single Executable Application (SEA) build process directly into Node.js core - a hobby project I’d been tinkering with for some time

Igalia WebKit Team: WebKit Igalia Periodical #54

2026-01-26T21:00:55+00:00

Update on what happened in WebKit in the week from January 19 to January 26.

The main event this week has been the creation of the branch for the upcoming stable series, accompanied by the first release candidate before 2.52.0. But there's more: the WPE port gains hyphenation support and the ability to notify of graphics buffer changes; both ports get graphics fixes and a couple of new Web features, and WPE-Android also gets a new stable release.

Cross-Port 🐱

Implemented support for the :open pseudo-class on dialog and details elements. This is currently behind the OpenPseudoClass feature flag.

Implemented the source property for ToggleEvent. This can be used to run code dependent on the triggering element in response to a popover or dialog toggle.

Graphics 🖼️

Fixed the rendering glitches with wheel event asynchronous scrolling, which occurred when the page was scrolled to areas not covered by tiles while the main thread was blocked.

Support for hyphenation has been added to WPE. This requires libhyphen and can be disabled at build-time with the USE_LIBHYPHEN=OFF CMake option.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

WPEPlatform gained support to notify changes in the configuration of graphics buffers allocated to render the contents of a web view, either by handling the WPEView::buffers-changed signal or by overriding the WPEViewClass.buffers_changed virtual function. This feature is mainly useful for platform implementations which may need to perform additional setup in advance, before updated web view contents are provided in the buffers configured by WebKit.

Releases 📦️

WPE-Android 0.3.0 has been released, and prebuilt packages are available at the Maven Central repository. The main change in this this version is the update to WPE WebKit 2.50.4, which is the most recent stable release.

A new branch has been created for the upcoming 2.52.x stable release series of the GTK and WPE WebKit ports. The first release candidates from this branch, WebKitGTK 2.51.90 and WPE WebKit 2.51.90 are now available. Testing and issue reports in Bugzilla are welcome to help with stabilization before the first stable release, which is planned for mid-March.

That’s all for this week!

Enrique Ocaña: Igalia Multimedia contributions in 2025

2026-01-26T09:34:37+00:00

Now that 2025 is over, it’s time to look back and feel proud of the path we’ve walked. Last year has been really exciting in terms of contributions to GStreamer and WebKit for the Igalia Multimedia team.

With more than 459 contributions along the year, we’ve been one of the top contributors to the GStreamer project, in areas like Vulkan Video, GstValidate, VA, GStreamer Editing Services, WebRTC or H.266 support.

Igalia’s contributions to the GStreamer project

In Vulkan Video we’ve worked on the VP9 video decoder, and cooperated with other contributors to push the AV1 decoder as well. There’s now an H.264 base class for video encoding that is designed to support general hardware-accelerated processing.

GStreaming Editing Services, the framework to build video editing applications, has gained time remapping support, which now allows to include fast/slow motion effects in the videos. Video transformations (scaling, cropping, rounded corners, etc) are now hardware-accelerated thanks to the addition of new Skia-based GStreamer elements and integration with OpenGL. Buffer pool tuning and pipeline improvements have helped to optimize memory usage and performance, enabling the edition of 4K video at 60 frames per second. Much of this work to improve and ensure quality in GStreamer Editing Services has also brought improvements in the GstValidate testing framework, which will be useful for other parts of GStreamer.

Regarding H.266 (VVC), full playback support (with decoders such as vvdec and avdec_h266, demuxers and muxers for Matroska, MP4 and TS, and parsers for the vvc1 and vvi1 formats) is now available in GStreamer 1.26 thanks to Igalia’s work. This allows user applications such as the WebKitGTK web browser to leverage the hardware accelerated decoding provided by VAAPI to play H.266 video using GStreamer.

Igalia has also been one of the top contributors to GStreamer Rust, with 43 contributions. Most of the commits there have been related to Vulkan Video.

Igalia’s contributions to the GStreamer Rust project

In addition to GStreamer, the team also has a strong presence in WebKit, where we leverage our GStreamer knowledge to implement many features of the web engine related to multimedia. From the 1739 contributions to the WebKit project done last year by Igalia, the Multimedia team has made 323 of them. Nearly one third of those have been related to generic multimedia playback, and the rest have been on areas such as WebRTC, MediaStream, MSE, WebAudio, a new Quirks system to provide adaptations for specific hardware multimedia platforms at runtime, WebCodecs or MediaRecorder.

Igalia Multimedia Team’s contributions to different areas of the WebKit project

We’re happy about what we’ve achieved along the year and look forward to maintaining this success and bringing even more exciting features and contributions in 2026.

Luke Lau: Closing the gap, part 2: Probability and profitability

2026-01-25T16:00:00+00:00

Welcome back to the second post in this series looking at how we can improve the performance of RISC-V code from LLVM.

Previously in part 1 we looked at how we can use LNT to analyze performance gaps, then identified and fixed a missed fmsub.d opportunity during instruction selection, giving a modest 1.77% speedup on a SPEC CPU 2017 benchmark.

In this post we’ll be improving another SPEC benchmark by 7% by teaching the loop vectorizer to make smarter cost modelling decisions. It involves a relatively non-trivial analysis, but thanks to LLVM’s modular infrastructure we can do it in just a handful of lines of code. Let’s get started.

Analysis

Just like last time, all fruitful performance work begins by analysing some workloads. In the last post we had already run some comparisons of SPEC CPU 2017 benchmarks on LNT, so we can return to those results and pick another benchmark to focus on. Here’s one that’s 12% slower than GCC:

531.deepsjeng_r is a chess engine that tied first in the World Computer Chess Championships back in 2009. It consists of a lot bitwise arithmetic and complex loops, since the state of the game is encoded in 64 element arrays: one element for each square on the board. Unlike 508.namd_r from last time, there’s no floating point arithmetic.

Drilling into the profile and its list of functions, right off the bat we can see that one function is much slower on LLVM. On GCC qsearch(state_t*, int, int, int, int) makes up 9.1% of the overall cycles, but on LLVM it’s 16.1%. And if we click in on the function and view the cumulative total of cycles spent in user mode, Clang takes 74.6 billion cycles to do what takes GCC only 37.7 billion cycles.

Left shows Clang taking 74.6 billion cycles, right shows GCC taking 37.7 billion.

So there’s probably something we can improve upon here, but it’s not immediately obvious from staring at the disassembly. qsearch is a pretty big function with a couple hundred instructions, so switching to the CFG view gives us a better overview.

On LLVM’s side we see the offending loop that’s consuming so many cycles: It’s long, vectorized, and completely if-predicated: there’s no control flow inside the loop itself. This is typical of a loop that’s been auto-vectorized by the loop vectorized. If you look at the load and store instructions you can see that they are masked with the v0.t operand, stemming from the original control flow that was flattened.

But on the GCC side there’s no equivalent vectorized loop. The loop is in there somewhere, but all the loops are still in their original scalar form with the control flow intact. And if we look at the edges coming from the loop headers, we can see that most of the time it visits one or two basic blocks and then branches back up to the header. Most of the blocks in the loop are completely cold.

Unfortunately the sources for deepsjeng aren’t open source so we can’t share them in this post, but the very rough structure of the loop is something like this:

for (i = 0; i < N; i++) {
    if (foo[i] == a) {
        if (bar[i] == b) {
            if (baz[i] == c) {
                qux[i] = 123;
                // lots of work here...
            }
        }
    }
}

For any given iteration, it’s statistically unlikely that we enter the first if statement. It’s even more unlikely that the second if’s condition is also true. And even more so for the third nested if where we eventually have lots of work to compute.

In a scalar loop this doesn’t matter because if an if statement’s condition is false, then we don’t execute the code inside it. We just branch back to the start of the loop. But with a vectorized loop, we execute every single instruction regardless of the condition.

This is the core of the performance gap that we’re seeing versus GCC: Given that the majority of the work in this loop is so deeply nested in the control flow, it would have been better to have not vectorized it given that we need to if-convert it.

Cost modelling

One of the hardest problems when making an optimizing compiler is to know when an optimization is profitable. Some optimizations are a double edged sword that can harm performance just as much as they can improve it (if not more), and loop vectorization falls squarely into this category. So rather than blindly applying optimizations at any given opportunity, LLVM has detailed cost models for each target to try and estimate how expensive or cheap a certain sequence of instructions is, which it can then use to evaluate whether or not a transform will be a net positive.

It’s hard to overstate the amount of effort in LLVM spent fine tuning these cost models, applying various heuristics and approximations to make sure different optimizations don’t shoot themselves in the foot. In fact there are some optimizations like loop distribute that are in-tree but disabled by default due to the difficulty in getting the cost model right.

So naturally, we would expect that the loop vectorizer already has a sophisticated solution for the problem we’re seeing in our analysis: Given any predicated block that’s if-converted during vectorization, we would expect the scalar cost for that block to be made slightly cheaper because the scalar block may not always be executed. And the less likely it is to be executed, the cheaper it should be — the most deeply nested if block should be discounted more than the outermost if block.

So how does the loop vectorizer handle this?

/// A helper function that returns how much we should divide the cost of a
/// predicated block by. Typically this is the reciprocal of the block
/// probability, i.e. if we return X we are assuming the predicated block will
/// execute once for every X iterations of the loop header so the block should
/// only contribute 1/X of its cost to the total cost calculation, but when
/// optimizing for code size it will just be 1 as code size costs don't depend
/// on execution probabilities.
///
/// TODO: We should use actual block probability here, if available. Currently,
///       we always assume predicated blocks have a 50% chance of executing.
inline unsigned
getPredBlockCostDivisor(TargetTransformInfo::TargetCostKind CostKind) {
  return CostKind == TTI::TCK_CodeSize ? 1 : 2;
}

We’ve come across a load bearing TODO here. Either the block is executed or its not, so it’s a fifty/fifty chance.

On its own this hardcoded probability doesn’t seem like an unreasonable guess. But whilst 50% may be an accurate estimate as to whether or not a branch will be taken, it’s an inaccurate estimate as to whether or not a block will be executed. Assuming that a branch has a 1/2 chance of being taken, the most deeply nested block in our example ends up having a 1/2 * 1/2 * 1/2 = 1/8 chance of being executed.

for (i = 0; i < N; i++) {
    if (foo[i] == a) {
        // 1/2 chance of being executed
        if (bar[i] == b) {
            // 1/4 chance of being executed
            if (baz[i] == c) {
                // 1/8 chance of being executed
                // ...
            }
        }
    }
}

The fix to get the loop vectorizer to not unprofitably vectorize this loop will be to teach getPredBlockCostDivisor to take into account control flow between blocks.

It’s worth mentioning the fact that a hardcoded constant managed to work well enough up until this point is the sign of an good trade off. 1% of the effort for 90% of the benefit. A patch can go off the rails very easily by trying to implement too much in one go, so deferring the more complex cost modelling here till later was an astute choice. Incremental development is key to making progress upstream.

VPlan cost modeling

To get a better picture of how the loop vectorizer is calculating the cost for each possible loop, lets start with a simplified LLVM IR reproducer:

; for (int i = 0; i < 1024; i++)
;   if (c0)
;     if (c1)
;       p1[p0[i]] = 0; // extra work to increase the cost in the predicated block

define void @nested(ptr noalias %p0, ptr noalias %p1, i1 %c0, i1 %c1) {
entry:
  br label %loop

loop:
  %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
  br i1 %c0, label %then.0, label %latch

then.0:
  br i1 %c1, label %then.1, label %latch

then.1:
  %gep0 = getelementptr i32, ptr %p0, i32 %iv
  %x = load i32, ptr %gep0
  %gep1 = getelementptr i32, ptr %p1, i32 %x
  store i32 0, ptr %gep1
  br label %latch

latch:
  %iv.next = add i32 %iv, 1
  %done = icmp eq i32 %iv.next, 1024
  br i1 %done, label %exit, label %loop

exit:
  ret void
}

We can run opt -p loop-vectorize -debug on this example to see how the loop vectorizer decides if it’s profitable to vectorize the loop or not:

$ opt -p loop-vectorize -mtriple riscv64 -mattr=+v nested.ll -disable-output -debug
...
LV: Found an estimated cost of 0 for VF 1 For instruction:   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c0, label %then.0, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c1, label %then.1, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep0 = getelementptr i32, ptr %p0, i32 %iv
LV: Found an estimated cost of 1 for VF 1 For instruction:   %x = load i32, ptr %gep0, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep1 = getelementptr i32, ptr %p1, i32 %x
LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 0, ptr %gep1, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   br label %latch
LV: Found an estimated cost of 1 for VF 1 For instruction:   %iv.next = add i32 %iv, 1
LV: Found an estimated cost of 1 for VF 1 For instruction:   %done = icmp eq i32 %iv.next, 1024
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %done, label %exit, label %loop
LV: Scalar loop costs: 3.
...
Cost of 1 for VF vscale x 4: induction instruction   %iv.next = add i32 %iv, 1
Cost of 0 for VF vscale x 4: induction instruction   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
Cost of 1 for VF vscale x 4: exit condition instruction   %done = icmp eq i32 %iv.next, 1024
Cost of 0 for VF vscale x 4: EMIT vp<%4> = CANONICAL-INDUCTION ir<0>, vp<%index.next>
Cost of 0 for VF vscale x 4: EXPLICIT-VECTOR-LENGTH-BASED-IV-PHI vp<%5> = phi ir<0>, vp<%index.evl.next>
Cost of 0 for VF vscale x 4: EMIT-SCALAR vp<%avl> = phi [ ir<1024>, vector.ph ], [ vp<%avl.next>, vector.body ]
Cost of 1 for VF vscale x 4: EMIT-SCALAR vp<%6> = EXPLICIT-VECTOR-LENGTH vp<%avl>
Cost of 0 for VF vscale x 4: vp<%7> = SCALAR-STEPS vp<%5>, ir<1>, vp<%6>
Cost of 0 for VF vscale x 4: CLONE ir<%gep0> = getelementptr ir<%p0>, vp<%7>
Cost of 0 for VF vscale x 4: vp<%8> = vector-pointer ir<%gep0>
Cost of 2 for VF vscale x 4: WIDEN ir<%x> = vp.load vp<%8>, vp<%6>, vp<%3>
Cost of 0 for VF vscale x 4: WIDEN-GEP Inv[Var] ir<%gep1> = getelementptr ir<%p1>, ir<%x>
Cost of 12 for VF vscale x 4: WIDEN vp.store ir<%gep1>, ir<0>, vp<%6>, vp<%3>
Cost of 0 for VF vscale x 4: EMIT vp<%index.evl.next> = add nuw vp<%6>, vp<%5>
Cost of 0 for VF vscale x 4: EMIT vp<%avl.next> = sub nuw vp<%avl>, vp<%6>
Cost of 0 for VF vscale x 4: EMIT vp<%index.next> = add nuw vp<%4>, vp<%0>
Cost of 0 for VF vscale x 4: EMIT branch-on-count vp<%index.next>, vp<%1>
Cost of 0 for VF vscale x 4: vector loop backedge
Cost of 0 for VF vscale x 4: EMIT-SCALAR vp<%bc.resume.val> = phi [ ir<0>, ir-bb<entry> ]
Cost of 0 for VF vscale x 4: IR   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ] (extra operand: vp<%bc.resume.val> from scalar.ph)
Cost of 0 for VF vscale x 4: EMIT vp<%3> = logical-and ir<%c0>, ir<%c1>
Cost for VF vscale x 4: 17 (Estimated cost per lane: 2.1)
...
LV: Selecting VF: vscale x 4.
LV: Minimum required TC for runtime checks to be profitable:0
LV: Interleaving is not beneficial.
LV: Found a vectorizable loop (vscale x 4) in nested.ll
LV: Vectorizing: innermost loop.
LEV: Unable to vectorize epilogue because no epilogue is allowed.
LV: Loop does not require scalar epilogue
LV: Loop does not require scalar epilogue
Executing best plan with VF=vscale x 4, UF=1

First we see it work out the cost of the original scalar loop, or as the vectorizer sees it, the loop with a vectorization factor (VF) of 1. It goes through each instruction calling into TargetTransformInfo, and arrives at a total scalar cost of 3. You might have noticed though, if you went through and manually summed up the individual instruction costs you would have gotten a total cost of 4. However the load and store instructions belong to the predicated then.1 block, so they have their cost divided by 2 from getPredBlockCostDivisor.

For the vectorized loop, the loop vectorizer uses VPlan to cost the one plan for a range of different VFs¹. VPlan is an IR specific to the loop vectorizer to help represent various vectorization strategies, which is why you see all the EMIT and WIDEN “recipes” in the output. It calculates a total cost for the loop and divides it by the estimated number of lanes — we’re working with scalable vectors on RISC-V so the target needs to make an estimate of what vscale is — and arrives at 2.1 per lane. There’s no predication discount applied here because it’s a vectorized loop. 2.1 is cheaper than 3, so it ultimately picks the vectorized loop.

BlockFrequencyInfo

Computing an accurate probability that a given block will be executed is a non-trivial task, but thankfully LLVM already has an analysis we can use for this called BlockFrequencyInfo.

BlockFrequencyInfo computes how often a block can be expected to execute relative to other blocks in a function. It in turn uses another analysis called BranchProbabilityInfo to work out how likely a branch to a specific block is going to be taken. And because BranchProbabilityInfo uses profiling information when available, it can give you much more accurate block frequencies when compiling with PGO. Otherwise it will fall back to guessing the probability of a branch being taken, which is just 50/50 a lot of the time, but sometimes influenced by interesting heuristics too: like the probability of icmp eq i32 %x, 0 is 0.375 instead of 0.5, and floats have a near zero chance of being NaN.

Plugging BlockFrequencyInfo into the loop vectorizer is straightforward, all we need to do is tell the pass manager that we want to access BlockFrequencyInfo from LoopVectorizePass:

PreservedAnalyses LoopVectorizePass::run(Function &F,
                                         FunctionAnalysisManager &AM) {
   ...
   BFI = &AM.getResult<BlockFrequencyAnalysis>(F);
   ...
}

(BlockFrequencyAnalysis is the pass that computes the analysis result BlockFrequencyInfo, if you’re wondering why the names are different)

Then we can use it to lookup the relative frequencies of whatever block and work out the probability of it being executed in the loop:

uint64_t LoopVectorizationCostModel::getPredBlockCostDivisor(
    TargetTransformInfo::TargetCostKind CostKind, const BasicBlock *BB) {
  if (CostKind == TTI::TCK_CodeSize)
    return 1;

  uint64_t HeaderFreq =
      BFI->getBlockFreq(TheLoop->getHeader()).getFrequency();
  uint64_t BBFreq = BFI->getBlockFreq(BB).getFrequency();
  return HeaderFreq / BBFreq;
}

The frequencies returned from BlockFrequencyInfo are relative to the the entry block of a function. So if a block has a frequency of 50 and the entry block has a frequency of 100, then you can expect that block to execute 50 times for every 100 times the entry block is executed.

You can use this to work out probabilities of a block being taken in a function, so in this example that block has a 50/100 = 50% chance of being executed every time the function is executed. However this only works in the case that the CFG has no loops: otherwise a block may be executed more times than the entry block and we’d end up with probabilities greater than 100%.

If we want to calculate the probability of a block being executed inside a loop though, that’s fine since the loop vectorizer currently only vectorizes inner-most loops², i.e. loops that contain no other loops.

We can consider the frequencies of each block in the loop relative to the frequency of the header block. To give a brief loop terminology recap, the header is the first block inside the loop body which dominates all other blocks in the loop, and is the destination of all backedges. So the header is guaranteed to have a frequency greater than or equal to any other block in the loop — this invariant is important as we’ll see later.

Then to calculate the probability of a block in a loop being executed, we divide the block frequency by the header frequency. To work out how much we should divide the cost of the scalar block by, we return the inverse of that.

Trying out this change on our sample loop, first we’ll see the debug output from BlockFrequencyInfo as it’s computed:

$ opt -p loop-vectorize -mtriple riscv64 -mattr=+v nested.ll -disable-output -debug
...
block-frequency-info: nested
 - entry: float = 1.0, int = 562949953421312
 - loop: float = 32.0, int = 18014398509481984
 - then.0: float = 16.0, int = 9007199254740992
 - then.1: float = 8.0, int = 4503599627370496
 - latch: float = 32.0, int = 18014398509481984
 - exit: float = 1.0, int = 562949953421312

loop is the header block and then.1 is the nested if block, and with BlockFrequencyInfo’s frequency we get a probability of 8/32 = 0.25. So we would expect then.1’s scalar cost to be divided by 4:

...
LV: Found an estimated cost of 0 for VF 1 For instruction:   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c0, label %then.0, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c1, label %then.1, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep0 = getelementptr i32, ptr %p0, i32 %iv
LV: Found an estimated cost of 1 for VF 1 For instruction:   %x = load i32, ptr %gep0, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep1 = getelementptr i32, ptr %p1, i32 %x
LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 0, ptr %gep1, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   br label %latch
LV: Found an estimated cost of 1 for VF 1 For instruction:   %iv.next = add i32 %iv, 1
LV: Found an estimated cost of 1 for VF 1 For instruction:   %done = icmp eq i32 %iv.next, 1024
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %done, label %exit, label %loop
LV: Scalar loop costs: 2.
...
Cost for VF vscale x 4: 17 (Estimated cost per lane: 2.1)
...
LV: Selecting VF: 1.
LV: Vectorization is possible but not beneficial.

then.1s scalar cost is now 2/4 = 0, so the total cost of the scalar loop is now 2 and the loop vectorizer no longer decides to vectorize. If we try this out on 538.deepsjeng_r, we can see that it no longer vectorizes that loop in qsearch either. Success!

Running it again on LNT showed a ~7% speedup in execution time. Not just as fast as GCC yet, but a welcome improvement for only a handful of lines of code.

Upstreaming

Now that we know the fix we want to land, we can start to think about how we want to upstream this into LLVM.

If we run llvm-lit --update-tests llvm/test/Transforms/LoopVectorize, we actually get quite a few unexpected test changes. One of the side effects of using BlockFrequencyInfo is that tail folded loops no longer discount the scalar loop if it wasn’t predicated to begin with. A tail folded loop is a loop where the scalar epilogue is folded into the vector loop itself by predicating the vector operations:

// non-tail folded loop:
// process as many VF sized vectors that fit in n
for (int i = 0; i < n - (n % VF); i += VF)
  x[i..i+VF] = y[i..i+VF];
// process the remaining n % VF scalar elements
for (int i = n - (n % VF); i < n; i++)
  x[i] = y[i];

// tail folded loop:
for (int i = 0; i < n; i += VF)
  x[i..i+VF] = y[i..i+VF] mask=[i<n, i+1<n, ..., i+VF-1<n];

However because this block is technically predicated due to the mask on the vector instructions, the loop vectorizer applied getPredBlockCostDivisor to the scalar loop cost even if the original scalar loop had no control flow in its body. BlockFrequencyInfo here can detect that if the block had no control flow, its probability of being executed is 1 and so the scalar loop cost isn’t made cheaper than it needs to be. I split off and landed this change separately, since it makes the test changes easier to review.

Now that the remaining changes in llvm/test/Transforms/LoopVectorize looked more contained, I was almost ready to open a pull request. I just wanted to quickly kick the tyres on llvm-test-suite with a few other targets, since this wasn’t a RISC-V specific change. The plan was to quickly collect some stats on how many loops were vectorized, check for any anomalies when compared to beforehand, and then be on our way:

$ cd llvm-test-suite
$ ninja -C build
...
[222/7278] Building C object External/...nchspec/CPU/500.perlbench_r/src/pp.c.o
FAILED: External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o 
/root/llvm-test-suite/build.x86_64-ReleaseLTO-a/tools/timeit --summary External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o.time /root/llvm-project/build/bin/clang -DDOUBLE_SLASHES_SPECIAL=0 -DNDEBUG -DPERL_CORE -DSPEC -DSPEC_AUTO_BYTEORDER=0x12345678 -DSPEC_AUTO_SUPPRESS_OPENMP -DSPEC_CPU -DSPEC_LINUX -DSPEC_LINUX_X64 -DSPEC_LP64 -DSPEC_SUPPRESS_OPENMP -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/dist/IO -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/cpan/Time-HiRes -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/cpan/HTML-Parser -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/ext/re -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/specrand -march=x86-64-v3 -save-temps=obj     -O3 -fomit-frame-pointer -flto -DNDEBUG   -w -Werror=date-time -save-stats=obj -save-stats=obj -fno-strict-aliasing -MD -MT External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -MF External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o.d -o External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -c /root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /root/llvm-project/build/bin/clang-19 -cc1 -triple x86_64-unknown-linux-gnu -O3 -emit-llvm-bc -flto=full -flto-unit -save-temps=obj -disable-free -clear-ast-before-backend -main-file-name pp.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=none -relaxed-aliasing -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64-v3 -debugger-tuning=gdb -fdebug-compilation-dir=/root/llvm-test-suite/build.x86_64-ReleaseLTO-a -fcoverage-compilation-dir=/root/llvm-test-suite/build.x86_64-ReleaseLTO-a -resource-dir /root/llvm-project/build/lib/clang/23 -Werror=date-time -w -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -vectorize-loops -vectorize-slp -stats-file=External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.stats -faddrsig -fdwarf2-cfi-asm -o External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -x ir External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.bc
1.      Optimizer
2.      Running pass "function<eager-inv>(float2int,lower-constant-intrinsics,chr,loop(loop-rotate<header-duplication;prepare-for-lto>,loop-deletion),loop-distribute,inject-tli-mappings,loop-vectorize<no-interleave-forced-only;no-vectorize-forced-only;>,infer-alignment,loop-load-elim,instcombine<max-iterations=1;no-verify-fixpoint>,simplifycfg<bonus-inst-threshold=1;forward-switch-cond;switch-range-to-icmp;switch-to-arithmetic;switch-to-lookup;no-keep-loops;hoist-common-insts;no-hoist-loads-stores-with-cond-faulting;sink-common-insts;speculate-blocks;simplify-cond-branch;no-speculate-unpredictables>,slp-vectorizer,vector-combine,instcombine<max-iterations=1;no-verify-fixpoint>,loop-unroll<O3>,transform-warning,sroa<preserve-cfg>,infer-alignment,instcombine<max-iterations=1;no-verify-fixpoint>,loop-mssa(licm<allowspeculation>),alignment-from-assumptions,loop-sink,instsimplify,div-rem-pairs,tailcallelim,simplifycfg<bonus-inst-threshold=1;no-forward-switch-cond;switch-range-to-icmp;switch-to-arithmetic;no-switch-to-lookup;keep-loops;no-hoist-common-insts;hoist-loads-stores-with-cond-faulting;no-sink-common-insts;speculate-blocks;simplify-cond-branch;speculate-unpredictables>)" on module "External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.bc"
3.      Running pass "loop-vectorize<no-interleave-forced-only;no-vectorize-forced-only;>" on function "Perl_pp_coreargs"
 #0 0x0000556ff93ab158 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/root/llvm-project/build/bin/clang-19+0x2d5c158)
 #1 0x0000556ff93a8835 llvm::sys::RunSignalHandlers() (/root/llvm-project/build/bin/clang-19+0x2d59835)
 #2 0x0000556ff93abf01 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007f305ce49df0 (/lib/x86_64-linux-gnu/libc.so.6+0x3fdf0)
 #4 0x0000556ffaa0dbfb llvm::LoopVectorizationCostModel::expectedCost(llvm::ElementCount) (/root/llvm-project/build/bin/clang-19+0x43bebfb)
 #5 0x0000556ffaa22a0d llvm::LoopVectorizationPlanner::computeBestVF() (/root/llvm-project/build/bin/clang-19+0x43d3a0d)
 #6 0x0000556ffaa36f3b llvm::LoopVectorizePass::processLoop(llvm::Loop*) (/root/llvm-project/build/bin/clang-19+0x43e7f3b)
 #7 0x0000556ffaa413eb llvm::LoopVectorizePass::runImpl(llvm::Function&) (/root/llvm-project/build/bin/clang-19+0x43f23eb)
 ...
...

A crash when building for X86. No assertion message, but a backtrace that points to the loop vectorizer cost model. Unfortunately this did not turn out to be simple to debug and instead turned into a whole other ordeal, so I’ll leave the details of that rabbit hole to the next post. But in the meantime, here are some hints if you want to guess what went wrong:

The crash stems from a SIGFPE signal
It only occurs when building on X86. Building on AArch64 is unaffected, even when cross-compiling to X86
It only occurs with LTO

Hopefully this also gives a bit of insight into the type of upstream work that we carry out at Igalia. If you have an LLVM or RISC-V project that we could help with, [email protected]">feel free to reach out.

The scalar loop is also modeled in VPlan, but currently costed with the legacy cost model and not the VPlan itself. This is another load bearing TODO. ↩
Whilst not enabled default, there is experimental support for outer loop vectorization in the VPlan native path. ↩

Igalia Compilers Team: Legacy RegExp features in JavaScript

2026-01-20T00:00:00+00:00

In June 2025, I joined the Igalia Coding Experience program. My role was to implement the TC39 proposal Legacy RegExp Features in SpiderMonkey, the JavaScript engine in Mozilla Firefox. This wasn't my first proposal implementation. I'd already implemented the Error.isError and Iterator.range TC39 proposals in SpiderMonkey, but implementing the Legacy RegExp Features proposal involved delving deeper into the Mozilla codebase, and new challenges for me.

To begin with, I created an implementation plan with a timeline of how I was going to approach the proposal. Additionally, I added links to the codebase where I thought I was going to make changes as per the specification, which helped me have a clear starting point and path for integrating the feature. It also meant I could get feedback from SpiderMonkey developers before actually beginning the implementation.

The Legacy RegExp features proposal disables legacy static properties and RegExp.prototype.compile for instances of proper subclasses of RegExp as well as for cross-realm regexps.

The following operations are modified in SpiderMonkey:

RegExp.prototype.compile(pattern, flags) #

This method reinitializes an existing RegExp object with a new pattern and/or flags. It modifies the RegExp object in place rather than creating a new one.

Modification: The proposal modifies RegExp.prototype.compile to throw errors for objects that are not direct instances of the RegExp as well as for cross-realm mismatches. The compile() method initializes a RegExp object similar to the way a RegExp literal is created, bypassing any preprocessing of the pattern that might be done by a RegExp subclass's constructor, and potentially breaking a subclass's custom "exec" method. Thus, compile is disallowed for subclasses. It is now forbidden for a RegExp compile method to be applied to a RegExp object belonging to a different realm, as this would typically result in static properties of the incorrect realm being updated.

Example of newly restricted behaviour:

(base) $ ./mach run
 0:00.29 /Users/default/firefox/obj-aarch64-apple-darwin25.2.0/dist/bin/js
js> let g = newGlobal();
js> let re = g.RegExp("x");
js> RegExp.prototype.compile.call(re);
typein:3:26 TypeError: RegExp operation not permitted on object from different realm
Stack:
  @typein:3:26
js>

To explain each line of the JavaScript code in detail:

let g = newGlobal() creates a new JavaScript global object in SpiderMonkey, similar to opening a new window in a browser. Each global object has its own realm. A realm is a JavaScript execution context that contains its own set of global objects and built-in functions. Every object in SpiderMonkey has a realm pointer which identifies which realm it belongs to.
let re = g.RegExp(“x”) creates a new RegExp object from g's realm, with a distinct instance of the RegExp constructor. Although the object behaves like one created from RegExp("x"), the two are not wholly compatible with one another.
RegExp.prototype.compile.call(re) invokes the compile() method with the regexp initialized above for a realm returned from newGlobal(). Per step 5 of the modified RegExp.prototype.compile() algorithm in the proposal, this results in a TypeError exception being thrown.

Initially, I added my changes in regexp_compile_impl(), but when testing with ./mach try auto, the feature failed test262 cross-realm tests when run with the ion eager and --more-compartments flag. Debug output showed that when invoking the RegExp.prototype.compile(re) both the receiver or (this`) of the RegExp.prototype.compile() method, and the RegExp object were in the same realm while they weren’t. In other words, the cross-realm check was passing, when it should have been failing, according to the test expectations.

By the time execution reached regexp_compile(), the CallNonGenericMethod<IsRegExpObject, regexp_compile_impl> wrapper had already processed the "receiver" or "this" of the compile method. According to the CallNonGenericMethod documentation, if args.thisv() is not of the correct type, it will attempt to unwrap this and if successful, call the implementation function on the unwrapped this. For a bit of context on this, SpiderMonkey has a concept of Wrapper objects, which decorate an object in a sort of proxy membrane to provide security boundary enforcement. For instance, ensuring that a method can be invoked or a field can be written to from the presently entered compartment. Unwrapping an object means removing that proxy membrane, to access the actual object, similar to how you’d unwrap a gift. This can be done using js::CheckedUnwrapStatic().

With --more-compartments, CallNonGenericMethod in regexp_compile() was automatically unwrapping cross-compartment proxies through CallMethodIfWrapped before calling regexp_compile_impl().

This unwrapping process also switched the JSContext to the target object's realm. This meant that by the time my realm checks executed in regexp_compile_impl(), both cx->realm() and the RegExp object's realm pointed to the same realm (the object's home realm), making them appear equal even in genuine cross-realm call scenarios where the original call came from a different realm.

So I moved the same-realm testing and [[LegacyFeaturesEnabled]] bit testing to regexp_compile(), just before CallNonGenericMethod is called and added js::CheckedUnwrapStatic() to unwrap any proxy wrappers before checking the realm. This ensures we’re checking the realm of the actual RegExp object and not the compartment wrappers around it.

Subclass Instances #

As mentioned above, the RegExp method RegExp.prototype.compile() re-initializes a RegExp using a newly created matcher for the specified pattern and flags. The proposal adds some restrictions to this which prevent oddities such as subclasses not functioning as expected (for instance, by not preprocessing the pattern and adding context used by their exec() implementation). More importantly, when applied to a cross-realm object, this would result in execution modifying the static RegExp members for the incorrect realm.

The proposal modifies the behavior so that legacy static properties are only updated when direct instances of the built-in RegExp constructor are used, not subclass instances or cross-realm objects, using similar logic to RegExp.prototype.compile():

If SameValue(thisRealm, rRealm) is true, then

i. If the value of R’s [[LegacyFeaturesEnabled]] internal slot is true, then

a. Perform UpdateLegacyRegExpStaticProperties(%RegExp%, S, lastIndex, e, capturedValues).

ii. Else,

a. Perform InvalidateLegacyRegExpStaticProperties(%RegExp%).

The properties are specced and implemented as accessors with a getter and no setter, except for RegExp.input (and its alias RegExp.$_), which remains writable. Inside each of the accessors, if the receiver this and the %RegExp% realm intrinsic (the standard RegExp constructor) are not the same, we throw a TypeError.

(base) $ ./mach run
 0:00.28 /Users/default/firefox/obj-aarch64-apple-darwin25.2.0/dist/bin/js
js> /a(b)c/.exec("abc"); 
["abc", "b"]
js> RegExp.$1 
"b"
js> new RegExp("a(b)").exec("ab"); 
["ab", "b"]
js> RegExp.$1                       
"b"
js> new (class extends RegExp {})("a(b)").exec("ab"); 
["ab", "b"]
js> RegExp.$1                                          
typein:6:1 TypeError: RegExp static property 'static_paren1_getter' is invalid
Stack:
  @typein:6:1
js>

/a(b)c/.exec("abc"); RegExp.$1  // should return "b"
new RegExp("a(b)").exec("ab"); RegExp.$1  // "b"
new (class extends RegExp {})("a(b)").exec("ab"); RegExp.$1 //  throws

Normalisation of RegExp Static Properties #

RegExp static properties are now defined as configurable and non-enumerable. This is so that the associated features may be easily removed by using the JavaScript delete operator. This is important for consistency with modern ECMA262 and for allows for applications to further reduce the number of side-affect producing globals, including VM native methods.

In SpiderMonkey, the legacy static properties are defined in RegExp.cpp. To implement the proposal, I enclosed the properties with a NIGHTLY_BUILD directive, removing the JS_PROP_PERMANEN and JS_PROP_ENUMERATE flags to make them configurable and non-enumerable for the Nightly environment, where they can be tested by the community. Outside of Nightly, we continue supporting the old implementation for beta/release environments.

Then, I updated the test262 AnnexB RegExp tests to support the change and to limit the tests to Nightly.

Understanding the Implementation: Challenges and Solutions #

1. Creative Bit Packing #

Once the legacy RegExp statics were normalised, the next step was adding a LegacyFeaturesEnabled internal slot. This slot keeps a reference to its constructor and is checked whenever legacy features are accessed. If the RegExp is a subclass instance or is is associated with a different realm, the slot indicates that legacy features should throw an error.

Initially, I added the slot to the RegExpObject :

static const unsigned LEGACY_FEATURES_ENABLED_SLOT = 3;

This presented a couple of problems for me:

The number of reserved slots must match the allocation kind defined in FOR_EACH_OBJECT_ALLOCKIND(D). The number of reserved slots increased to 5, which meant that I had to choose between OBJECT6 or OBJECT8. During implementation, I somehow missed OBJECT6 and went with OBJECT8.
I knew that I’d get some pushback in code review, as my changes increased the size of the RegExp Object by 32 bytes (four 8-byte slots). I could see that there was a way for Boolean flags to share a slot but I didn't know how to implement my changes without breaking the JIT.

I decided to leave the implementation as is and wait for SpiderMonkey engineers / reviewers to give me feedback and their preference on how to add the Boolean.

During code review, my reviewer Iain pointed out that since we’re only storing a single bit of information (whether legacy features are enabled or not), and the existing FLAGS_SLOT only uses 8 bits, I could store the legacy features in the unused higher bits.

The slot implementation includes a getter, bool legacyFeaturesEnabled(), that reads the bit from the FLAGS_SLOT; and a setter, setLegacyFeaturesEnabled(bool), that writes the bit to the FLAGS_SLOT.

The new approach involved defining some constants based on the size of RegExp Flags so that the code keeps working if RegExpFlags gets bigger in future:

static const size_t RegExpFlagsMask = JS::RegExpFlag::AllFlags;
static const size_t LegacyFeaturesEnabledBit = Bit(8);

static_assert((RegExpFlagsMask & LegacyFeaturesEnabledBit) == 0,
              "LegacyFeaturesEnabledBit must not overlap");

RegExpFlagsMask has a bit set to 1 if that bit is part of the RegExpFlags, and 0 otherwise. The lowest 8 bits are currently set to other RegExp flags, which leaves us with the highest bits to pack our slot in.

We perform two operations: raw & RegExpFlagsMask, which gets only the traditional RegExp flags; and raw & ~RegExpFlagsMask, which gets everything apart from the RegExp flags.Those are bits 0-7. We use bit 8 to store LegacyFeaturesEnabled. When we read the flags, we mask off any bits that are not part of the RegExpFlags.

return JS::RegExpFlags(raw & RegExpFlagsMask);

When we write to the flags, we combine the new value of the RegExpFlags bits (flags.value()) with the old value of the other bits in (raw & RegExpFlagsMask).

uint32_t newValue = flags.value() | (raw & ~RegExpFlagsMask);
setFixedSlot(FLAGS_SLOT, Int32Value(newValue));

When we read the LegacyFeaturesEnabledBit, we check if it’s set. When we write it, we take the existing raw value and either set or clear the LegacyFeaturesEnabledBit.

2. Lazy Evaluation #

The proposal specifies RegExp properties as internal slots of the RegExp Object, and the abstract operations UpdateLegacyRegExpStaticProperties (C, S, startIndex, endIndex, capturedValues) and InvalidateLegacyRegExpStaticProperties(C) were initially confusing. The confusion came from a specification detail: we need to eagerly update the properties at a specific point in time, as opposed to SpiderMonkey’s lazily evaluated implementation.

It was the first time I had come across lazy evaluation and thought, naively, that it would be possible to change the implementation to eagerly update static properties after a successful match. This didn't work for a few reasons.

First, lazy evaluation is heavily embedded in the JIT, so the idea of just changing that was… ambitious. Second, lazy evaluation is a way to defer regexp evaluation until RegExp properties are accessed. Third, there’s no observable difference to the end user whether the RegExp properties were lazily or eagerly evaluated. Lastly, internal slots are a way for ECMA262 to describe the internal state of the object.

So, UpdateLegacyRegExpStaticProperties (C, S, startIndex, endIndex, capturedValues) wasn’t needed, as it codifies already existing behaviour in SpiderMonkey. For InvalidateLegacyRegExpStaticProperties(C), my mentor suggested implementing it as a boolean flag in RegExpStatics.

When a subclass or cross-realm regexp executes, this flag is set to true, preventing legacy static properties from being accessed. The flag is cleared after normal RegExp executions, allowing legacy features to work for standard RegExp instances.

Because InvalidateLegacyRegExpStaticProperties(C) marks the values of the static properties as unavailable by setting the internal slots to empty, in step 4 of the accessors GetLegacyRegExpStaticProperty(C, thisValue, internalSlotName), we throw a TypeError if the static properties are invalidated.

Then, we add the equivalent code in the JIT path and so that when a regexp is executed, we lazily store enough information to be able to rerun the regexp later if the RegExpStatics are accessed.

3. Gating the implementation behind a preference #

The first step to implementing a TC39 proposal in SpiderMonkey is adding a preference for it. This allows the feature to be enabled or disabled at runtime, which is important in gating the feature until it has been tested enough for release.

With this proposal, it was awkward, because this was not a new syntax or library method, but behavioral modifications to the existing RegExp static properties and the compile() method.

At first, I enclosed my changes in an #ifdef NIGHTLY_BUILD directive so that they are only available in the nightly environment. But given the potential for web compatibility risks, we needed to put the changes behind a preference. That way, we can flip the feature back in case we break something.

This created an awkward situation: the static RegExp properties themselves (like RegExp.$1, RegExp.input) are defined in regexp_static_props, which is baked into the static RegExp JSClass and embedded in the binary at compile time. I ended up wrapping these property definitions in an #ifdef NIGHTLY_BUILD, meaning they only exist in Nightly builds.

But the behavior of these properties — that is, whether accessing them should throw errors for subclasses and cross-realm regexps — is gated behind a runtime preference. This is even more awkward, because it will change behaviour in Nightly even without the preference enabled.

Thus, the preference only controls whether the new throwing behavior is active. As Iain noted, there wasn't a particularly clean way to avoid this. We'd need two parallel RegExp classes and then have to switch between them at runtime based on the pref, which seemed like overkill.

The compromise was to ship the properties in Nightly, use the preference to control the new behavior, and rely on extra-careful testing.

4. Wild Goose Chase #

Around August, when I had the initial implementation working without memory optimization or centralized legacy and realm checks, I was updating legacy regexp statics in RegExpBuiltinExec() only when matches succeeded.

RegExpBuiltinExec() has two execution paths: a forTest path for RegExp.prototype.test (where we can skip allocating a result object) and a normal path for full execution. I had legacy feature validation in both paths, but only for successful matches.

My mentor suggested we needed to update the legacy regexp statics not just on success, but also on failure. That made sense from a spec perspective, so I spent the next week and a half trying to figure out how to implement this. I was looking into the execution paths, trying to understand where and how to trigger updates on failed matches.

After about a week, we realized that they had misread the proposal! Oops. Turns out, SpiderMonkey doesn't update legacy regexp properties on failure at all: it just returns the last successful result. I'd been chasing a solution to a problem that didn't actually exist in the implementation.

Next Steps and Final Thoughts #

The "Legacy RegExp features in JavaScript" proposal is, at the time of this writing, in stage 3 of the TC39 process, meaning the proposal is stable and no further changes can be made to it. There are potential backward compatibility risks and any attempt to use a disabled feature will throw a Type Error. More on that can be found in the Breaking Hazards portion of the proposal.

Before implementing this proposal I had briefly interacted with C++ on a production level codebase when working on the Error.isError proposal, but working on legacy RegExp properties was a deeper dive into C++ and browser internals, which was difficult but also very much appreciated!

Working on this proposal exposed gaps in my knowledge but also gave me confidence in navigating large C++ codebases. I’m particularly grateful to my mentor, and Daniel Minor and Iain Ireland (from the SpiderMonkey team) for pointing me in the right direction and brainstorming solutions with me.

Igalia WebKit Team: WebKit Igalia Periodical #53

2026-01-19T19:25:32+00:00

Update on what happened in WebKit in the week from December 26 to January 19.

We're back! The first periodical of 2026 brings you performance optimizations, improvements to the memory footprint calculation, new APIs, the removal of the legacy Qt5 WPE backend, and as always, progress on JSC's Temporal implementation.

Cross-Port 🐱

The memory footprint calculation mechanism has been unified across GTK, JSC, and WPE ports. Therefore, the expensive /proc/self/smaps is not used anymore and the WPE uses /proc/self/statm with extra cache now to prevent frequent file reading.

Added a new webkit_context_menu_get_position() function to the API that allows obtaining the pointer coordinates, relative to the web view origin, at the moment when a context menu was triggered.

Additionally, behaviour of context menus has been made more consistent between the GTK and WPE ports, and handling of GAction objects attached to menu items has been rewritten and improved with the goal of better supporting context menus in the WPE port.

JavaScriptCore 🐟

The built-in JavaScript/ECMAScript engine for WebKit, also known as JSC or SquirrelFish.

In JavaScriptCore's implementation of Temporal, fixed a bug in Temporal.PlainTime.from that read options in the wrong order, which caused a test262 test to fail.

In JavaScriptCore's implementation of Temporal, fixed several bugs in PlainYearMonth methods and enabled all PlainYearMonth tests that don't depend on the Intl object. This completes the implementation of Temporal PlainYearMonth objects in JSC.

Graphics 🖼️

In WebKit's Skia graphics backend, fixed GrDirectContext management for GPU resources. Operations on GPU-backed resources must use the context that created them, not the current thread's context. The fix stores GrDirectContext at creation time for NativeImage and uses surface->recordingContext()->asDirectContext() for SkSurface, correcting multiple call sites that previously used the shared display's context incorrectly.

Damage propagation has been added to the recently-added, non-composited mode in WPE.

In WebKit's Skia graphics backend for GTK/WPE, added canvas 2D operation recording for GPU-accelerated rendering. Instead of executing drawing commands immediately, operations are recorded into an SkPicture and replayed in batch when the canvas contents are needed, reducing GPU state change overhead for workloads with many small drawing operations, improving the MotionMark Canvas Lines performance on embedded devices with low-end tiled GPUs.

Due to Qt5 not receiving maintenance since mid-2025, the WPE Qt5 binding that used the legacy libwpe API has been removed from the tree. The Qt6 binding remains part of the source tree, which is a better alternative that allows using supported Qt versions, and is built atop the new WPEPlatform API, making it a future-proof option. The WPE Qt API may be enabled when configuring the build with CMake, using the ENABLE_WPE_QT_API option.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

The WPEScreenSyncObserver class has been improved to support multiple callbacks. Instead of a single callback set with wpe_screen_sync_observer_set_callback(), clients of the API can now use wpe_screen_sync_observer_add_callback() and wpe_screen_sync_observer_remove_callback(). The observer will be paused automatically when there are no callbacks attached to it.

That’s all for this week!

Manuel Rego: Servo 2025 Stats

2026-01-14T00:00:00+00:00

This is a brief blog post to highlight the growth of the Servo community in recent years, particularly since Igalia took over the project maintenance in 2023.

Note that this doesn’t talk about the technical achievements, though there have been tons of them in the last years. A picture is worth a thousand words so just take a look at this slide from my latest Servo talk which shows how google.com was rendered with Servo at the beginning of 2023 vs September 2025.

Slide showing screenshots of Servo rendering google.com in January 2023 vs September 2025

PRs numbers #

So like we did last year, let’s take a look at the PRs merged on the main Servo repository on GitHub since 2018.

	2018	2019	2020	2021	2022	2023	2024	2025
PRs	1,188	986	669	118	65	776	1,771	3,183
Contributors	27.33	27.17	14.75	4.92	2.83	11.33	26.33	42.42
Contributors ≥ 10	2.58	1.67	1.17	0.08	0.00	1.58	4.67	8.50

PRs: total number of PRs merged.
Contributors: average number of contributors per month.
Contributors ≥ 10: average number of contributors that have merged more than 10 PRs per month.

As a clarification, these numbers don’t include PRs from bots (dependabot and Servo WPT Sync).

Checking this we can see we are close to double the numbers from last year! The numbers in 2025 are way bigger than in the previous years (even checking the numbers from 2018-2019), showing a healthy community working on Servo.

The next chart is a different view of the same data but split per month, with the number of PRs landed every month, the number of contributors and the number of contributors with more than 10 patches. It shows the evolution over the years and the high activity last year.

Number of contributors #

Now let’s focus on the last 3 years, since the project reactivation, and the numbers of contributors to the Servo project.

	2023	2024	2025
Contributors	54	129	146
≥ 100 PRs	1 (2%)	3 (2%)	8 (5%)
≥ 10 PRs	8 (15%)	29 (22%)	43 (29%)
Only 1 PR	31 (57%)	53 (41%)	55 (38%)

The number of contributors to Servo has tripled since 2023, reaching 146 different contributors in 2025.

If we analyze the rest of the data in this table, we can see that the percentage of contributors that do a single PR to Servo in a year has been reduced, meaning that Servo contributors are now usually doing more than one PR to the project.

If we check the number of contributors that have done more than 10 PRs in a year, we see the percentage almost doubling from 15% to 29% in the last 3 years.

And for the top contributors doing more than 100 PRs in a year, we have gone from 1 in 2023 and 3 in 2024 to 8 last year, which represent the 5% of the Servo contributors, showing a good team of very active contributors to the project.

WPT pass-rate #

Let’s take a look at WPT evolution in 2025.

2025	January 1st	December 31st	Diff
Score %	48.2%	61.6%	+13.4pp
Subtests (passed/total)	1396647/1998146	1866247/1998146	+469,600
Subtests %	69.9%	93.4%	+23.5pp

Evolution of WPT pass rates for Servo in 2025

You can check more information about WPT pass-rates at Servo’s website (where you can also find an explanation of the Score number).

Note that these numbers differ from wpt.fyi because we’re still not running all the WPT tests in Servo, so the total numbers here are smaller.

It’s not easy to extract conclusions from this data, but it shows the Servo project keeps progressing and supporting more web platform features as time passes.

Sometimes these numbers grow artificially as new tests are added to WPT for features that Servo already supports (for example, the biggest jump last year was in October getting 188,281 new subtests passing without any change in Servo, just because new tests were added to WPT).

GitHub stars #

Evolution of GitHub stars for Servo from star-history.com

We are about to reach 35,000 stars on GitHub. It’s good to see the project has not stopped growing since the beginning, and the curve has become steeper in recent years.

Other #

If we check to the official project roles, we have now:

5 administrators
17 TSC members
25 maintainers
18 contributors

We have also started doing Servo releases, we have done 3 so far.

Also the TSC has setup sponsorship tiers for donations. We got 4 bronze sponsors in 2025 and we hope to increase the number of sponsorships in 2026.

Regarding donations, we have defined a funding process to request usage of that money. We are currently using it to sponsor Josh Matthews’ contributions, and pay for self-hosted runners to speed up CI times.

Servo has been present in several events last year, we ended up giving 10 talks all around the globe.

Wrap-up #

The idea here was to do a quick recap of the Servo stats in 2025. Taking a look at these numbers every now and then is useful, and gives you a different perspective about the status of the project, that one can easily ignore during the day-to-day tasks.

In general things have grown a lot in 2025, who knows what would happen in 2026, but we hope we can at least keep similar numbers or maybe even keep growing them further. That would be really great news for the Servo project.

Igalia is really proud of what the whole Servo community has achieved together in the recent years, and we hope for a bright future for the project going forward.

As an aside note, by the end of the month I’ll be at FOSDEM talking about Servo, other Servo folks like Delan Azabani and Martin Robinson will also be there. If you are around, don’t hesitate to say hi and ask anything about the project.

Miyoung Shin: Our Journey to support Extensions for embedders

2026-01-13T02:00:28+00:00

A History of Extensions for Embedders — and Where We’re Heading

Chromium’s Extensions platform has long been a foundational part of the desktop browsing experience. Major Chromium-based browsers—such as Chrome and Microsoft Edge—ship with full support for the Chrome Extensions ecosystem, and user expectations around extension availability and compatibility continue to grow.

In contrast, some Chromium embedders— for instance, products built directly on the //content API without the full //chrome stack—do not naturally have access to Extensions. Similarly, the traditional Chrome for Android app does not support Extensions. While some embedders have attempted to enable limited Extensions functionality by pulling in selected pieces of the //chrome layer, this approach is heavyweight, difficult to maintain, and fundamentally incapable of delivering full feature parity.

At Igalia we have been willing to help on the long term-goal of making Extensions usable on lightweight, //content-based products, without requiring embedders to depend on //chrome. This post outlines the background of that effort, the phases of work so far, the architectural challenges involved, and where the project is headed.

Note: ChromeOS supporting extensions (ChromeOS has announced plans to incorporate more of the Android build stack) is not the same thing as Chrome-Android App supporting extensions. The two codepaths and platform constraints differ significantly. While the traditional Chrome app on Android phones and tablets still does not officially support extensions, recent beta builds of desktop-class Chrome on Android have begun to close this gap by enabling native extension installation and execution.

Tracking bug: https://issues.chromium.org/issues/356905053

Extensions Architecture — Layered View

The following diagram illustrates the architectural evolution of Extensions support for Chromium embedders.

Traditional Chromium Browser Stack

At the top of the stack, Chromium-based browsers such as Chrome and Edge rely on the full //chrome layer. Historically, the Extensions platform has lived deeply inside this layer, tightly coupled with Chrome-specific concepts such as Profile, browser windows, UI surfaces, and Chrome services.

+-----------------------+
|      //chrome         |
|  (UI, Browser, etc.)  |
+-----------------------+
|     //extensions      |
+-----------------------+
|      //content        |
+-----------------------+

This architecture works well for full browsers, but it is problematic for embedders. Products built directly on //content cannot reuse Extensions without pulling in a large portion of //chrome, leading to high integration and maintenance costs.

Phase 1 — Extensions on Android (Downstream Work)

In 2023, a downstream project at Igalia required extension support on a Chromium-based Android application. The scope was limited—we only needed to support a small number of specific extensions—so we implemented:

basic installation logic,
manifest handling,
extension launch/execution flows, and
a minimal subset of Extensions APIs that those extensions depended on.

This work demonstrated that Extensions can function in an Android environment. However, it also highlighted a major problem: modifying the Android //chrome codepath is expensive. Rebasing costs are high, upstream alignment is difficult, and the resulting solution is tightly coupled to Chrome-specific abstractions. The approach was viable only because the downstream requirements were narrow and controlled.

I shared this experience at BlinkOn Lightning Talk: “Extensions on Android”.

Phase 2 — Extensions for Embedders
( //content + //extensions + //components/extensions )

Following Phase 1, we began asking a broader question:

Can we provide a reusable, upstream-friendly Extensions implementation that works for embedders without pulling in the //chrome layer?

Motivation

Many embedders aim to remain as lightweight as possible. Requiring //chrome introduces unnecessary complexity, long build times, and ongoing maintenance costs. Our hypothesis was that large portions of the Extensions stack could be decoupled from Chrome and reused directly by content-based products.

One early idea was to componentize the Extensions code by migrating substantial parts of //chrome/*/extensions into //components/extensions.

+-------------------------+
| //components/extensions |
+-------------------------+
|      //extensions       |
+-------------------------+
|       //content         |
+-------------------------+

Proof-of-concept : Wolvic

We tested this idea through Wolvic , a VR browser used in several commercial
solutions. Wolvic has two implementations:

a Gecko-based version, and
a Chromium-based version built directly on the //content API.

Originally, Extensions were already supported in Wolvic-Gecko, but not in Wolvic-Chromium. To close that gap, we migrated core pieces of the Extensions machinery into //components/extensions and enabled extension loading and execution in a content-only environment.

By early 2025, this work successfully demonstrated that Extensions could run without the //chrome layer.

Demo video::
https://youtube.com/shorts/JmQnpC-lxR8?si=Xf0uB6q__j4pmlSj

Design document:
https://docs.google.com/document/d/1I5p4B0XpypR7inPqq1ZnGMP4k-IGeOpKGvCFS0EDWHk/edit?usp=sharing

However, this work lived entirely in the Wolvic repository, which is a fork of Chromium. While open source, this meant that other embedders could not easily benefit without additional rebasing and integration work.

This raised an important question:

Why not do this work directly in the Chromium upstream so that all embedders can benefit?

Phase 3 — Extensions for Embedders
(//content + //extensions)

Following discussions with the Extensions owner ([email protected]), we refined the approach further.

Rather than migrating functionality into //components, the preferred long-term direction is to move Extensions logic directly into the //extensions layer wherever possible.

+-----------------------+
|      Embedder UI      | (minimal interfaces)
+-----------------------+
|      //extensions     |
+-----------------------+
|       //content       |
+-----------------------+

This approach offers several advantages:

clearer layering and ownership,
fewer architectural violations,
reduced duplication between Chrome and embedders,
a cleaner API surface for integration.

We aligned on this direction and began upstream work accordingly.

Tracking bug: https://issues.chromium.org/issues/358567092

Our goals for Content Shell + //extensions are:

Embedders should only implement a small set of interfaces, primarily for UI surfaces (install prompts, permission dialogs) and optional behaviors.
Full Web Extensions APIs support
w3c standard : https://w3c.github.io/webextensions/specification/
Chrome Web Store compatibility
Embedders should be able to install and run extensions directly from the Chrome Web Store.

Short-term Goal: Installation Support

Our immediate milestone is to make installation work entirely using //content + //extensions.

Current progress:

.zip installation support already lives in //extensions
Migrating Unpacked directory installation from //chrome to //extensions
(including replacing Profile with BrowserContext abstractions)
Moving .crx installation code from //chrome → //extensions

As part of this effort, we are introducing clean, well-defined interfaces for install prompts and permission confirmations:
Chrome will continue to provide its full-featured UI
Embedders can implement minimal, custom UI as needed

What Comes Next:

Once installation is fully supported, we will move on to:

Chrome Web Store integration flows
Core WebExtensions APIs required by commonly used extensions

Main Engineering Challenge — Detaching from the Chrome Layer

The hardest part of this migration is not moving files—it is breaking long-standing dependencies on the //chrome layer.

The Extensions codebase is large and historically coupled to Chrome-only concepts such as:

Profile
Browser
Chrome-specific WebContents delegates
Chrome UI surfaces
Chrome services (sync, signin, prefs)

Each migration requires careful refactoring, layering reviews, and close collaboration with component owners. While the process is slow, it has already resulted in meaningful architectural improvements.

What’s Next?

In the next post, We’ll demonstrate:

A functioning version of Extensions running on top of
//content + //extensions only — capable of installing and running extensions app.

from Igalia side, we continue working on ways to make easier integrating Chromium on other platforms, etc. This will mark the first end-to-end, //chrome-free execution path for extensions in content-based browsers.

Stay tuned!

Alex Bradbury: Per-query energy consumption of LLMs

2026-01-07T12:00:00+00:00

How much energy is consumed when querying an LLM? We're largely in the dark when it comes to proprietary models, but for open weight models that anyone can host on readily available, albeit eye-wateringly expensive, hardware this is something that can be measured and reported, right? In fact, given other people are doing the hard work of setting up and running benchmarks across all kinds of different hardware and software configurations for common open weight models, can we just re-use that to get a reasonable figure in terms of Watt-hours (Wh) per query?

For the kind of model you can run locally on a consumer GPU then of course there's some value in seeing how low the per-query energy usage might be on a large scale commercial setup. But my main interest is in larger and more capable models, the kind that you wouldn't realistically run locally and end up using in a pay-per-token manner either directly with your host of choice or through an intermediary like OpenRouter. In these cases where models are efficiently served with a minimum of 4-8 GPUs or even multi-node clusters it's not easy to get a feel for the resources you're using. I'm pretty happy that simple back of the envelope maths shows that whether providers are properly amortising the cost of their GPUs or not, it's implausible that they're selling per-token API access for open models at below the cost of electricity. That gives a kind of upper bound on energy usage, and looking at the pennies I spend on such services it's clearly a drop in the ocean compared to my overall energy footprint. But it's not a very tight bound, which means it's hard to assess the impact of increasing my usage.

We can look at things like Google's published figures on energy usage for Gemini but this doesn't help much. They don't disclose the length of the median prompt and its response, or details of the model used to serve that median query meaning it's not helpful for either estimating how it might apply to other models or how it might apply to your own usage (which may be far away from this mysterious median query). Mistral released data on the per query environmental impact (assuming for a 400 token query), but the size of the Mistral Large 2 model is not disclosed and they don't calculate a Wh per query figure. CO2 and water per query are very helpful to evaluate a particular deployment, but the actual energy used is a better starting point that can be applied to other providers assuming different levels of carbon intensity. If one of the API providers were to share statistics based on a real world deployment of one of the open models with a much higher degree of transparency (i.e. sharing stats on the number of queries served during the period, statistics on their length, and measured system power draw) that would be a useful source of data. But today we're looking at what we can conclude from the InferenceMAX benchmark suite published results.

I'd started looking at options for getting good figures thinking I might have to invest in the hassle and expense of renting a multi-GPU cloud instance to run my own benchmarks, then felt InferenceMAX may make that unnecessary. After writing this up along with all my provisos I'm perhaps tempted again to try to generate figures myself. Anyway, read on for a more detailed look at that benchmark suite. You can scroll past all the provisos and jump ahead to the figures giving the Wh/query figures implied by the benchmark results across different GPUs, different average input/output sequence lengths, and for gpt-oss 120B and DeepSeek-R1-0528. But I hope you'll feel a bit guilty about it.

If you see any errors, please let me know.

High-level notes on InferenceMAX

InferenceMAX benchmark suite has the stated goal to "provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation." They differentiate themselves from other benchmarking efforts noting "Existing performance benchmarks quickly become obsolete because they are static, and participants often game the benchmarks with unrealistic, highly specific configurations."

The question I'm trying to answer is "what is the most 'useful AI' I can expect for a modern GPU cluster in a realistic deployment and how much energy does it consume". Any benchmark is going to show peak throughput higher than you'd expect to achieve in real workload and there's naturally a desire to keep it pinned on a specific model for as long as it isn't totally irrelevant in order to enable comparisons as hardware and software evolves with a common point of reference. But although I might make slightly different choices about what gets benchmarked and how, the InferenceMAX setup at first look seems broadly aligned with what I want to achieve.

They benchmark DeepSeek-R1-0528 (both at the native fp8 quantisation and at fp4) which is a 671B parameter model with 37B active weights released ~7 months ago and seems a fair representative of a large MoE open weight model. gpt-oss-120b is also benchmarked, providing a point of comparison for a much smaller and efficient to run model. Different input sequence length and output sequence length (ISL and OSL - the number of input and output tokens) are tested: 1k/1k, 1k/8k, 8k/1k, which provides coverage of different query types. Plus tests against a wide range of GPUs (including the 72-GPU GB200 NVL72 cluster) and sweeps different settings.

At the time of writing you might reasonably consider to be 'InferenceMAX' is split into around three pieces:

The frontend website you can see at inferencemax.semianalysis.com (not currently open source but planned to be)
The script for executing queries against the LLM serving infrastructure and collecting stats (currently in a seperate repo but planned to be incorporated into the main InferenceMAX repository),
The wrapper/runner scripts and GitHub actions workflows that live in the main InferenceMAX repository.
- This is actively contributed to by at least Nvidia and AMD engineers.

GitHub Actions is used to orchestrate the runs, ultimately producing a zip file containing JSON with the statistics of each configuration (e.g. here). The benchmark_serving.py script is invoked via the run_benchmark_serving wrapper in benchmark_lib.sh which hardcodes some options and passes through some others from the workflow YAML. The results logged by benchmark_serving.py are processed in InferenceMAX's process_result.py helper which will produce JSON in the desired output format. Together, these scripts provide statistics like throughput (input and output token), end to end latency, interactivity (output tokens per second) etc.

Further studying the benchmark setup

So, let's look at the benchmarking logic in more detail to look for any surprises or things that might affect the accuracy of the Wh-per-query figure I want to generate. I'll note that InferenceMAX is an ongoing project that is actively being developed. These observations are based on a recent repo checkout, but of course things may have changed since then if you're reading this post some time after it was first published.

Looking through I made the following observations. Some represent potential issues (see the next subheading for a list of the upstream issues I filed), while others are just notes based on aspects of the benchmark I wanted to better understand.

One of the required arguments to the benchmark serving script is --random-range-ratio. This is set by default to 0.8 in benchmark-tmpl.yml and in benchmark-multinode-tmpl.yml and is not overridden elsewhere.
- This argument is ultimately used in sample_random_requests. It uses np.random.randint to sample input/output lengths between the range_ratio * {input,output}_len and {input,output}_len.
- Taken together, this logic means for for a workload advertised as having 8k input or output tokens (8192), the benchmark will actually run with an average ~7373 (0.9*num_tokens, due to the length being a random number between 0.8*num_tokens and num_tokens) tokens.
- Because the throughput figures are calculated using the actual input and output token lengths, the figure does represent what was observed, it's just the workload doesn't quite match the description. The reported end to end latency for instance will be misleadingly lower than you would get for a workload that actually did have the expected input / output sequence lengths.
The various request functions in backend_request.func.py will set output.success = False if they don't get a HTTP 200 status code back for a request. There is no logic to retry a refused request and metrics will be calculated skipping any failed requests. This means an overloaded server will perform better on this benchmark for metrics like E2E latency and TTFT if it refuses requests rather than accept them and serve them slowly. As the number of failed requests isn't included in the results json it's not easy to tell if this is a factor for any benchmarks.
Many of the various scripts in the benchmarks/ subdirectory set a max-model-len parameter or the similar --max_seq_len parameter for trt-llm (e.g. the b200 config which if I'm not mistaken will ultimately be set from the max_model_len defined in generate_sweep_configs.py. This parameter is documented in vllm and in TensortRT-LLM and controls the maximum supported length of a request, including both the prompt and any generated output. Setting it 20 or 200 tokens above the sum of the benchmarked ISL+OSL to minimise memory use does not seem like a realistic real-world deployment, which seems the wrong choice given the InferenceMAX complaint that in other suites "participants often game the benchmarks with unrealistic, highly specific configurations". Benchmarks naturally show a 'best case', but if you're generating figures like $ per M tokens it's a figure that makes little sense if it reflects a configuration you wouldn't feasibly use/sell.
Throughput is calculated in benchmark_serving.py based on the total number of tokens divided by the duration of the benchmark. This is then normalised on a per-GPU basis in process_result.py. No problems here, I just wanted to clarify the source of the figure.
In terms of the source of the input tokens themselves, we can see that --dataset-name random is always passed to benchmark_serving.py. This leads to sample_random_requests being called, which will pick random token ids and create a list of tokens of the desired length (mapping these randomly picked IDs to tokens).
- The --ignore-eos flag is passed to the benchmark_serving.py script which will in turn set this option in the JSON when making the LLM request. backend_request_func.py sets this and also sets max_tokens to the desired output_len which should ensure that the response has that exact desired number of output tokens. ignore_eos means that the LLM server will keep generating tokens even after seeing the end of sequence token.
- It's interesting that some of the benchmark configurations enable multi-token prediction, and presumably find it beneficial even given the totally random token inputs. Is it possible that such configurations benefit from undesirable looped outputs (due to a combination of random inputs and continuing to sample tokens past the EOS marker) that potentially are very predictable and give an extra boost?
The --num-prompts parameter controls the total number of requests that are issued. The benchmark script is written so it will wait for all of these to complete (either successfully or unsuccessfully). This is typically set to the concurrency times 10, but some benchmark setups set it higher (presumably as the default figure finishes too quickly for good results).
In terms of how requests are submitted with a certain level of concurrency:
- See above for a discussion of the total number of requests
- --request-rate inf is always passed, so there's no limit on submitting requests up to the concurency limit.
- It precomputes a list of requests to submit and then uses a semaphore to limit concurrency but otherwise continuously submits requests up to the concurrency limit, and then waits until they call complete.
There are no tests that the configuration is serving the model with the expected quality currently, but there's an issue tracking at least adding a simple quality benchmark. Although none of the explored settings should impact the quality of output, it's always possible they trigger a bug and in this case it's not interesting to benchmark.
It would be helpful for reproducibility if more complete system information for the benchmark runners was released. This is being worked on.
You should of course consider whether the tested input and output sequence lengths correspond to a workload you are interested in (thank you to Aaron Zhao for reminding me to mention this. This benchmarking approach also doesn't consider caching. Both factors could be highly relevant if trying to estimate energy cost for a long context chat or 'agentic' flow. But I'm happy enough with the tested workloads as a starting point, and my main focus here is trying to get a degree of comfort with the reported numbers for the ISL/OSL combinations they've chosen to test.

Filed issues

I ended up filing the following issues upstream:

FIXED Token throughput per MW is described as reflecting the generated tokens but is actually processed+generated tokens
- The companion article introducing InferenceMAX has previously defined throughput as the rate at which the GPU generates tokens yet the figure displayed in the UI was the total number of output and input tokens per second. The definition in the article has now been fixed, and changes to the UI make it more obvious based on context that throughput refers to input+output tokens (as y-axis metric options now exist to show "input token throughput per GPU" and "output token throughput per GPU").
- This talking head video from Nvidia seems to make the same error, talking about the number of tokens 'generated' per second per GPU when looking at the relevant results these sem to be the total throughput (i.e. output plus the much faster to process input tokens).
Presented input/output token throughput per GPU for disaggregated setups not usefully comparable to standard multi-gpu setups
- In disaggregated setups you have some number of GPUs dedicated to prefill (processing input tokens) and some number dedicated to decode (generating output tokens). In this case, the reported input/output throughput figures refer to the input or output throughput per prefill GPU or per decode GPU. It doesn't make sense (IMHO) to plot this figure against the input/output throughput figures for a non-disaggregated setup. To make it comparable, the input/output throughput per GPU should be calculated by averaging across the whole cluster rather than just the GPUs dedicated to prefill or decode respectively.
Standard deviatation of interactivity (std_intvty) in result json is incorrectly calculated
- Not a big issue as the figure isn't used anywhere. Interactivity (tokens/second) metrics are calculated from the recorded time per output token. 1000/$tpot_metric is correct for the mean, median, and p99 figures but mathematically incorrect for the standard deviation. e.g. a small standard deviation for time per output token will result in a huge standard deviation being computed for interactivity.
FIXED Reference kW figures no longer shown in frontend for each GPU
- At some point updates to the frontend logic meant that the per-GPU kW figures used in calculating the token throughput per utility MW were no longer displayed. This has now been fixed.
How will full workflow run output be retained beyond 90 days
- The benchmark frontend helpfully links to the GitHub Actions run that generated the displayed results and has a datepicker to view previous results. Clicking through to GitHub means you can download the original .zip of the JSON format benchmark results which is something I take advantage of in the analysis later in this article. According to GitHub docs, the maximum retention period for Actions artifacts and logs is 90 days for a public repo. It would be good to have a mechanism so that this information is backed up rather than lost.
Contents of CONFIG_DIR path as used in launch_gb200-nv.sh is undisclosed
- Most benchmark configuration lives in the main repository, but unfortunately one of the Nvidia DeepSeek R1 configurations relies on a config dir that's not publicly available meaning it can't be audited or reproduced. This is a case where tightening up benchmark rules and review process can hopefully avoid it happening in the future.
Reconsider allowing setting max_model_len / max_seq_len to isl+osl+tiny_margin
- As explained above, a number of benchmarks set max_model_len (or for Nvidia's TensorRT, --max_seq_len) to some figure that is just above ISL+OSL. Although some degree of tuning is expected, to me this goes against the idea that "We want server configs to reflect real world deployments as much as possible" and the stated goal "to provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation". It's hard to imagine a realistic deployment that would configure their serving engine in a way such that it errors if input+output tokens passes ~2k tokens for instance. Looking at the DeepSeek R1 0528 providers on OpenRouter, the vast majority offer greater than 128k context.
- By my understanding, with PagedAttention the KV cache is dynamically allocated anyway so this setting would largely impact other data structures. Plus vllm at least contains a startup check that there is sufficient VRAM to serve at least one request at the maximum configured context. I would really like to see what impact this setting has on benchmarks.
- The repository maintainers renamed my issue to a title that doesn't reflect my report. I'm hopeful they will review my recent comment and title it back.
Some reported metrics will be inflated if a serving engine sheds load
- This covers the observation made above that failed requests are simply skipped. As the number of failed requests isn't tracked, it's not easy to see if a particular configuration may appear better (better E2E latency, lower time to first token) as a reset of shedding load rather than queueing.
- The repository maintainers renamed this issue to "[feature suggestion for vllm/vllm benchmark_serving]" and closed it. I'm hopeful they will read my response and reconsider on the grounds that:
  - The benchmark_serving script isn't doing anything "wrong" necessarily. It is simply making an implementation choice with potential impact on results that the InferenceMAX harness isn't tracking.
  - The script is planned to be added to the repo soon anyway.
Benchmarked ISL and OSL averages 0.9*target_length meaning results are over-optimistic.
- This is the problem mentioned above where the introduced variance in input/output sequence length has an average lower than the headline rate. As noted, this means specifically the end to end latency figure is misleading, but also impacts tokens/second and throughput to the extent that the cost of serving a query doesn't scale with O(n).
- This will be fixed by PR 339 which upstreams the benchmark_serving.py script and in that modified branch changes sample_random_requests to sample a range with multiplier between 1 - RANGE_RATIO and 1 + RANGE_RATIO.

In the best case, you'd hope to look at the benchmark results, accept they're probably represent a higher degree of efficiency than you'd likely get on a real workload, that an API provider might achieve 50% of that and double the effective cost per query to give a very rough upper estimate on per-query cost But that only really works if the reported benchmark results roughly match the achievable throughput in a setup configured for commercial serving. Given the tuning to specific isl/osl values, I'm not at all confident thats the case and I don't know how wide the gap is.

Generating results

Firstly I wrote a quick script to check some assumptions about the data and look for anything that seems anomalous. Specifically:

Check that total throughput per GPU matches what you'd expect based on the input token and output token throughput per GPU, even in the disaggregated case. i.e. the total thoughput per GPU averaged over the whole cluster should equal the sum of the input and output throughput per GPU provided those figures are averaged over the whole cluster.
The ratio of input token throughput to output token throughput should be almost equal to the to the ratio of input to output tokens in the benchmark's workload. If not, there is something surprising that needs investigating.

Based on the information available in the generated result JSON and the reported all-in power per GPU (based on SemiAnalysis' model), we can calculate the Watt hours per query. First calculate the joules per token (watts per GPU divided by the total throughput per GPU). This gives a weighted average of the joules per token for the measured workload (i.e. reflecting the ratio of isl:osl). Multiplying joules per token by the tokens per query (isl+osl) gives the joules per query, and we can just divide by 3600 to get Wh.

There is some imprecision because we're constructing the figure for e.g. 8192/1024 ISL based on measurements with an average 0.9*8192 input and 0.9*1024 output length. The whole calculation would be much simpler if the benchmark harness recorded the number of queries executed and in what time, meaning we can directly calculate the Wh/query from the Wh for the system over the benchmark duration divided by the number of queries served (and remembering that in the current setup each query is on average 90% of the advertised sequence length).

This logic is wrapped up in a simple script.

There's been a recent change to remove the 'full sweep' workflows in favour of only triggering a subset of runs when there is a relevant change. But I grabbed my results from before this happened, from a December 15th 2025 run. However when finalising this article I spotted Nvidia managed to land some new NVL72 DeepSeek R1 0528 configurations just before Christmas, so I've merged in those results as well, using a run from December 19th. All data and scripts are collected together in this Gist.

Results

As well as giving the calculated Wh per query, the script also gives a comparison point of minutes of PS5 gameplay (according to Sony, "Active Power Consumption" ranges from ~217W to ~197W depending on model - we'll just use 200W). The idea here is to provide some kind of reference point for what a given Wh figure means in real-world times, rather than focusing solely on the relative differences between different deployments. Comparisons to "minutes of internet streaming" seem popular at the moment, presumably as it's because an activity basically everyone does. I'm steering away from that because I'd be comparing one value that's hard to estimate accurately and has many provisos to another figure that's hard to estimate accurately and has many provisos, which just injects more error and uncertainty into this effort to better measure/understand/contextualise energy used for LLM inference.

I'm now going to cherry-pick some results for discussion. Firstly for DeepSeek R1 0528 with 8k/1k ISL/OSL, we see that the reported configurations that give a usable level of interactivity at fp8 report between 0.96-3.74 Wh/query (equivalent to 0.29-1.12 minutes of PS5 gaming). The top row which is substantially more efficient is the newer GB200 NVL72 configuration added at the end of last year. It's not totally easy to trace the configuration changes given they're accompanied by a reworking of the associated scripts, but as far as I can see the configuration ultimately used is this file from the dynamo repository. Looking at the JSON the big gain comes from significantly higher prefill throughput (with output throughput per GPU remaining roughly the same). This indicates the older results (the second row) were bottlenecked waiting for waiting for prefill to complete.

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp8 DS R1 0528 8k/1k	39.5	36.5	gb200 dynamo-sglang (72 GPUs disagg, conc: 2048, pfill_dp_attn, dec_dp_attn)	0.96	0.29
fp8 DS R1 0528 8k/1k	31.3	55.2	gb200 dynamo-sglang (72 GPUs disagg, conc: 1024, pfill_dp_attn, dec_dp_attn)	3.13	0.94
fp8 DS R1 0528 8k/1k	20.9	48.8	h200 trt (8 GPUs, conc: 64, dp_attn)	3.32	1.00
fp8 DS R1 0528 8k/1k	19.5	49.6	h200 sglang (8 GPUs, conc: 64)	3.39	1.02
fp8 DS R1 0528 8k/1k	23.9	39.9	b200-trt trt (8 GPUs, conc: 64)	3.39	1.02
fp8 DS R1 0528 8k/1k	22.3	44.5	b200 sglang (8 GPUs, conc: 64)	3.74	1.12

Now taking a look at the results for an fp4 quantisation of the same workload, the result is significantly cheaper to serve with similer or better interactivity and the NVL72 setup Nvidia submitted does have a significant advantage over the 4/8 GPU clusters. This time we see 0.63-1.67 Wh/query (equivalent to 0.19-0.50 minutes of PS5 power draw while gaming). Serving at a lower quantisation impacts the quality of results of course, but the improved efficiency, including on smaler 4 GPU setups helps demonstrate why models like Kimi K2 thinking are distributed as "native int4", with benchmark results reported at this quantisation and quantisation aware training used to maintain quality of result.

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 DS R1 0528 8k/1k	41.6	24.6	gb200 dynamo-trt (40 GPUs disagg, conc: 1075, pfill_dp_attn, dec_dp_attn)	0.63	0.19
fp4 DS R1 0528 8k/1k	22.8	43.2	b200-trt trt (4 GPUs, conc: 128, dp_attn)	0.93	0.28
fp4 DS R1 0528 8k/1k	18.7	59.3	b200 sglang (4 GPUs, conc: 128)	1.25	0.38
fp4 DS R1 0528 8k/1k	30.3	39.4	b200 sglang (4 GPUs, conc: 64)	1.67	0.50

Looking now at the 1k/8k workload (i.e. generating significant output) and the cost is 15.0-16.3 Wh/query (equivalent to 4.49-4.89 minutes of PS5 power draw while gaming). As expected this is significantly higher than the 8k/1k workload as prefill (processing input tokens) is much cheaper per token than decode (generating output tokens)

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp8 DS R1 0528 1k/8k	42.5	176.3	b200 sglang (8 GPUs, conc: 64)	15.0	4.49
fp8 DS R1 0528 1k/8k	31.9	232.2	h200 sglang (8 GPUs, conc: 64)	15.9	4.76
fp8 DS R1 0528 1k/8k	31.2	237.9	h200 trt (8 GPUs, conc: 64)	16.3	4.88
fp8 DS R1 0528 1k/8k	39.1	189.5	b200-trt trt (8 GPUs, conc: 64)	16.3	4.89

Again, fp4 has a significant improvement in efficiency:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 DS R1 0528 1k/8k	29.7	251.5	b200-trt trt (4 GPUs, conc: 256, dp_attn)	2.73	0.82
fp4 DS R1 0528 1k/8k	37.7	197.5	b200-trt trt (8 GPUs, conc: 256, dp_attn)	4.31	1.29
fp4 DS R1 0528 1k/8k	34.2	221.2	b200 sglang (4 GPUs, conc: 128)	4.75	1.43
fp4 DS R1 0528 1k/8k	33.1	223.1	b200-trt trt (4 GPUs, conc: 128)	4.79	1.44

As you'd expect for a much smaller model at native fp4 quantisation, GPT-OSS-120B is much cheaper to serve. e.g. for 8k/1k:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 GPT-OSS 120B 8k/1k	45.8	20.8	b200-trt trt (1 GPUs, conc: 128)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	93.1	10.5	b200-trt trt (2 GPUs, conc: 128, dp_attn)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	44.3	21.4	b200 vllm (1 GPUs, conc: 128)	0.11	0.03
fp4 GPT-OSS 120B 8k/1k	145.7	6.7	b200-trt trt (2 GPUs, conc: 64, dp_attn)	0.14	0.04
fp4 GPT-OSS 120B 8k/1k	103.8	9.2	b200 vllm (2 GPUs, conc: 64)	0.20	0.06

Or for 1k/8k:

Workload	Intvty (tok/s)	E2EL (s)	Details	Wh/Q	PS5 min
fp4 GPT-OSS 120B 1k/8k	80.5	91.6	b200-trt trt (1 GPUs, conc: 128)	0.49	0.15
fp4 GPT-OSS 120B 1k/8k	72.3	102.0	b200 vllm (1 GPUs, conc: 128)	0.55	0.16
fp4 GPT-OSS 120B 1k/8k	144.9	51.1	b200-trt trt (2 GPUs, conc: 128, dp_attn)	0.55	0.17
fp4 GPT-OSS 120B 1k/8k	129.4	57.0	b200-trt trt (2 GPUs, conc: 128)	0.61	0.18

Conclusion

Well, this took rather a lot more work than I thought it would and I'm not yet fully satisfied with the result. Partly we have to accept a degree of fuzziness about marginal energy usage of an individual query - it's going to depend on the overall workload of the system so there's going to be some approximation when you try to cost a single query.

I'm glad that InferenceMAX exists and am especially glad that it's open and publicly developed, which is what has allowed me to dive into its implementation to the extent I have and flag concerns/issues. I feel it's not yet fully living up to its aim of providing results that reflect real world application, but I hope that will improve with further maturation and better rules for benchmark participants. Of course, it may still make most sense to collect benchmark figures myself and even if doing so, being able to refer to the benchmarked configurations and get an indication of what hardware can achieve what performance is helpful in doing so. Renting a 72-GPU cluster is expensive and as far as I can see not typically available for a short time, so any benchmarking run by myself would be limited to 4-8 GPU configurations. If the gap in efficiency is huge for such setups vs the NVL72 then these smaller setups are maybe less interesting.

If I found the time to run benchmarks myself, what would I be testing? I'd move to DeepSeek V3.2. One of the big features of this release was the movement to a new attention mechanism which scales much closer to linearly with sequence length. With e.g. Kimi Linear and Qwen3-Next, other labs are moving in a similar direction experimentally at least. I'd try to set up 8 GPU configuration with sglang/vllm configured in a way that it would be capable of serving a commercial workload with varied input/output sequence lengths and test this is the case (Chutes provide their deployed configs which may be another reference point). I'd want to see how much the effective Wh per million input/output tokens varies depending on the different isl/osl workloads. These should be relatively similar given the linear attention mechanism, and if so it's a lot easier to estimate the rough energy cost of a series of your own queries of varied length. I would stick with the random input tokens for the time being.

So where does that leave us? All of this and we've got figures for two particular models, with one benchmark harness, a limited set of input/output sequence lengths, and a range of potential issues that might impact the conclusion. I think this is a useful yardstick / datapoint, though I'd like to get towards something that's even more useful and that I have more faith in.

Article changelog

2026-02-17:
- Changed GitHub links to point to SemiAnalysisAI/InferenceX rather than InferenceMAX/InferenceMAX, as they were broken by the upstream rename.
2026-01-09:
- Fix broken link.
- Add note that more complete system info would be helpful for reproducibility.
- Add note about variety of input/output sequence lengths tested.
2026-01-07: Initial publication date.

Brian Kardell: The Secret Life of Custom Elements

2026-01-06T05:00:00+00:00

The Secret Life of Custom Elements

Twenty years ago last month, Google published an analysis of "slightly over a billion documents," a snapshot of the web that helped shape the early direction of HTML5. It followed a lineage of smaller, more personal studies — individuals poking at the web to answer some narrow question, often with datasets that would easily fit on a thumb drive today. For about half those two decades, I’ve been arguing that we need more study of the web, not less. The platform evolves faster than our understanding of it, and the only way to know what the web actually is — not what we imagine it to be — is to look.

Every month the HTTP Archive quietly captures a snapshot of the web as it actually exists—not the idealized web that we hope for, but the messy, improvised, duct‑taped reality of millions of sites in the wild. I’ve been collecting and studying these elements for the last six years.

This new dataset is the largest I’ve ever worked with: Billions of pages, hundreds of thousands of distinct non-standard element names, and a long tail that stretches into places no standards body has ever seriously examined. And unlike the Google study, which looked for patterns in class names, this dataset captures the long tail of non‑standard elements — the names people invent for actual elements when the platform doesn’t give them what they need.

What emerges is a portrait of the web as it is lived: messy, inventive, repetitive, global, and full of reinvention. It’s also a mirror held up to the platform itself.

But, it's also much more complex to study than I could have imagined a decade ago, and I really wish that the W3C (and member orgs which include academia) had taken up the charge to begin to figure out how to really study the web and use that information to inform standards work.

What's difficult about it...

One problem is that the dataset itself has some fairly extreme bias. The crawl doesn't hit anything that isn't on the public internet - that means it excludes intranets which are massive. In fact, most of my career was spent working on intranets. The crawl captures only home pages, plus the target of whatever it interprets as the largest link on that page. It also can't get to anything that requires login - which means that for a site like twitter or bluesky or mastodon, you're going to get something very unrepresentative of any of those. So, one challenge I'd love to see us trying to tackle is how to get even better data representation. It's hard to "pave cowpaths" if they're in a country we can't even see into.

Initially I had this idea that we could watch for the adoption of tags - imagining that we'd get some that would become very popular, just like we did with JavaScript libraries and frameworks. However, it turns out that this is not the signal it might first appear to be. An element appearing in tens of thousands or even hundreds of thousands of pages is often simply because they are part of a larger successful system. If Wix or Shopify create some custom elements that work behind the WYSIWYG tooling, and lots of people use it to create their pages - then suddenly that element gets very very popular - even if it isn't actually particularly good. In fact, we can see shifts in the data where the teams themselves changed their minds and another version supplants the first very quickly because it's simply internal.

Then, I thought that perhaps what we can do with the dataset instead, is to squint at it and look a little more abstractly at what people are naming their elements and see if people are re-solving similar problems. Do we find, for example, multiple non-standard element names that appear to be about tabs? Yes! Clearly that is indicative that we need a native element, right? Maybe. It's a bit more nuanced than that. Here are the most commonly re-created/repeated non-standard element themes:

Navigation
Headers and footers
Carousels and sliders
Modals
Search bars
Product cards
Login forms
Cookie banners
Accordions
Tabs
Toasts
Breadcrumbs

While we don't have several of these in standard HTML, we do have native <header>, <footer>, <nav>, <dialog>, and <search> elements, and even accordions via the name attribute of <details>. And yet, the wild still contains hundreds or thousands of custom elements with names like <app-header>, <site-footer>, <main-nav>, <modal-dialog>, <search-box>, and <accordion-panel>.

Native primitives may exist, but not at the same level of abstraction as these. <header> and <footer> in HTML are structural, not behavioral. <dialog> is behavioral, but not styled. <search> exists, but doesn’t solve autocomplete, filtering, or results.

So developers build those - and, if you stop and think about it, not all non-standard elements are equally as undesirable. Many of them will be simple decorations or thin wrappers that do use their native counterparts. Where there is definitely some interesting thing to study is where there is clear generic need where the platform doesn't provide anything close. Above, tabs, for example.

Observations..

Here are many observations from the data, in no real particular order of importance.

Forms and Inputs: Tweaked, Wrapped, and Re‑Wrapped

Forms and inputs are a great example of the constant re-invention I just described. Sometimes it's because the native element is insufficient, but that's not necessarily the case. In some cases they're just slight wrappers. Among them are lots and lots of "pickers" and "selecters" that show up...

<custom-select>
<date-picker>
<variant-picker>
<quantity-selector>

There is already a lot of ongoing work to make native form elements (including selects) require less code and just be more stylable and flexible, and the data at least suggests that such efforts will be very welcome.

Hidden Machinery

A surprising number of elements aren’t UI components at all. They’re runtime markers:

<ng-container>
<router-outlet>
<astro-island>
<ion-router-outlet>
<next-route-announcer>

These exist because frameworks need declarative boundaries for hydration, routing, rendering or template expansion. I suppose it is debatable wither these are an indicator of “missing HTML features”, or just how much.

Carousels (and sliders... and toasts)

I don't love carousels, but it's hard to deny that they are popular. There are dozens of distinct and identifiable carousel/slider elements in the dataset and they appear a lot. I really dislike a few bits of Google's attempt to make CSS-only carousels possible, but it's pretty clear why they chose to tackle that problem. I guess it is worth stressing again the bias in the dataset here - if there is a page I most expect to see a carousel, it is exactly the primary one the archive crawls. So, while it is the most popular in the dataset, I don't know that it is the most popular all-around. You can see why Google winds up with their proposals though, toasts are on that top list too.

Structural semantics?

There are a few broad categories where the main point seems to be "semantics". That is, very often many of these don't actually do anything, beyond provide some hooks, mainly for styling. They aren't actually even custom elements sometimes (or maybe even often) - just non-standard elements.

e-commerce

Dozens of these surround e-commerce. There are tens of thousands of sites that use elements with names (and variants).

Product & merchandising

<product-card>
<product-title>
<product-price>
<product-rating>
<product-variant>
<product-gallery>
<product-description>
<product-badge>

Pricing & money

<price-money>
<sale-price>
<compare-at-price>
<discount-amount>
<currency-display>

Inventory & availability

<stock-status>
<pickup-availability>
<delivery-estimate>
<inventory-level>

Cart & checkout

<cart-items>
<cart-count>
<checkout-button>
<order-summary>

Very interestingly they are often used alongside actual machine readable semantics via jsonLD in the same markup.

While the vast majority of these elements appear because of common tooling, the fact that there are dozens of variants of similar names appearing on smaller numbers of sites indicates there is something widely interesting here. It's hard to say what it is other than that it would be nice to have a common structural semantic that would work for both purposes.

I guess the biggest surprise here is that if it's true, why hasn't such a thing arisen already? It is entirely within the community's power to develop such a thing. Perhaps the answer is that there is just so much variance it isn't easily plausible. Maybe templating would somehow allow us to achieve a common pattern which achieved this based on the shared jsonLD semantics.

Publishing & Editorial Semantics

CMSes and news sites often invent tags for editorial structure, and many of these are sticking around.

Content structure

<article-header>
<article-summary>
<article-author>
<article-date>
<article-tags>
<article-tag>
<article-category>
<byline>
<dateline>
<pullquote>
<footnote>

Taxonomy

<tag-list>
<category-label>
<topic-header>

These reflect the needs of journalism and long‑form content.

Social & Community Semantics

These show up in comment systems, forums, and social platforms.

User‑generated content

<comment>
<comment-list>
<comment-item>
<comment-author>
<comment-content>
<comment-date>
<comment-form>

Identity

<user-avatar>
<user-name>
<profile-card>

These encode relationships and interactions, not UI patterns.

Events

<event-date>
<event-location>
<event-schedule>
<event-details>

Again, these are domain objects, not widgets - and they have well established schema.org or microformats as well.

Invoicing

<invoice>
<invoice-line>
<invoice-total>
<invoice-summary>

Before the web came along, there were already national and international standards around electronically trading informtation like invoices - and when XML was sold, invoices were a common example. Here we are again.

"Namespaced" Elements

Several elements like `o:p`, `rdf:rdf`, `dc:format`, `cc:work`, `fb:like`, `g:plusone` appear in the top 100. These basically were thinking of an XHTML future (namespacing) that never really arrived. However, HTML has always allowed it - so that's just the tag name. In many ways, it's just as good. Interestingly, these may be some of the better examples of what I'd like to see happen - they are widely understood.

Conversely, while hugely successful, the share buttons are more an indication of a desire than something we could actually standardize in precisely that way. They also point to a desire _in time_. Google Plus doesn't even exist anymore, `fb:like` is from a time when Facebook was at the top of the most interesting places to be. Maybe one of the things we've learned is that this is way handier to do at the browser/OS levels? I suppose the Web Share API was a part of thinking how we'd deal with this.

The fact that they both still appear so much is also kind of an indication of age of the page and slow replacement of underlying tools.

Typos, Encoding Errors, and the Weird Stuff

One of the most delightful parts of the dataset is the long tail of what are almost certainly just typos:

<prodcut-card>
<navgation>
<contianer>

The fact that these can appear on tens of thousands of sites because they are part of common tooling helps re-enforce that not every non-standard element is a signal. :)

In conclusion...

I wish that I could say "Ah ha - the data says very clearly that these are the specific things we should definitely 'just write down' now" in the way that I imagined a decade ago, but I don't think we're there yet. I guess if I had to give three things I'd like to see happen from here they'd be:

We need lots more effort in thinking about how to study these things. I would love to see real investment in this space. This year, at last, the W3C is hiring someone to study the web. I'm not yet sure what that looks like but I look forward to trying to discuss more with them.
We need a real community effort - an Underwriters Labs for custom elements, with participation and funding from orgs with money. We don't necessarily need "the one true tabs" as much as we need a place to find what I expect will be a very few sets of tabs as custom elements which we can trust like we trust native elements. Given a little bit of time, I have faith that this will naturally sort itself into a few 'winners'.
That community effort might also include things which won't ever have native implmentations, but which lay down some kind of light semantic meaning or compound styling structure that we all begin to agree on - like product cards or breadcrumbs.

A lot of this is pretty adjacent/close to the ideas behind OpenUI and it's possible some of this could just happen there. However, due mainly to limits and participation, OpenUI has really not really produced custom elements or worked to somehow list or grade and promote them (though we did study them quite a bit in the tabs research). The effort led by Brad Frost to think about a "global design system" in particular might be closer to some of these ideas.

Andy Wingo: pre-tenuring in v8

2026-01-05T15:38:38+00:00

Hey hey happy new year, friends! Today I was going over some V8 code that touched pre-tenuring: allocating objects directly in the old space instead of the nursery. I knew the theory here but I had never looked into the mechanism. Today’s post is a quick overview of how it’s done.

allocation sites

In a JavaScript program, there are a number of source code locations that allocate. Statistically speaking, any given allocation is likely to be short-lived, so generational garbage collection partitions freshly-allocated objects into their own space. In that way, when the system runs out of memory, it can preferentially reclaim memory from the nursery space instead of groveling over the whole heap.

But you know what they say: there are lies, damn lies, and statistics. Some programs are outliers, allocating objects in such a way that they don’t die young, or at least not young enough. In those cases, allocating into the nursery is just overhead, because minor collection won’t reclaim much memory (because too many objects survive), and because of useless copying as the object is scavenged within the nursery or promoted into the old generation. It would have been better to eagerly tenure such allocations into the old generation in the first place. (The more I think about it, the funnier pre-tenuring is as a term; what if some PhD programs could pre-allocate their graduates into named chairs? Is going straight to industry the equivalent of dying young? Does collaborating on a paper with a full professor imply a write barrier? But I digress.)

Among the set of allocation sites in a program, a subset should pre-tenure their objects. How can we know which ones? There is a literature of static techniques, but this is JavaScript, so the answer in general is dynamic: we should observe how many objects survive collection, organized by allocation site, then optimize to assume that the future will be like the past, falling back to a general path if the assumptions fail to hold.

my runtime doth object

The high-level overview of how V8 implements pre-tenuring is based on per-program-point AllocationSite objects, and per-allocation AllocationMemento objects that point back to their corresponding AllocationSite. Initially, V8 doesn’t know what program points would profit from pre-tenuring, and instead allocates everything in the nursery. Here’s a quick picture:

A linear allocation buffer containing objects allocated with allocation mementos

Here we show that there are two allocation sites, Site1 and Site2. V8 is currently allocating into a linear allocation buffer (LAB) in the nursery, and has allocated three objects. After each of these objects is an AllocationMemento; in this example, M1 and M3 are AllocationMemento objects that point to Site1 and M2 points to Site2. When V8 allocates an object, it increments the “created” counter on the corresponding AllocationSite (if available; it’s possible an allocation comes from C++ or something where we don’t have an AllocationSite).

When the free space in the LAB is too small for an allocation, V8 gets another LAB, or collects if there are no more LABs in the nursery. When V8 does a minor collection, as the scavenger visits objects, it will look to see if the object is followed by an AllocationMemento. If so, it dereferences the memento to find the AllocationSite, then increments its “found” counter, and adds the AllocationSite to a set. Once an AllocationSite has had 100 allocations, it is enqueued for a pre-tenuring decision; sites with 85% survival get marked for pre-tenuring.

If an allocation site is marked as needing pre-tenuring, the code in which it is embedded it will get de-optimized, and then next time it is optimized, the code generator arranges to allocate into the old generation instead of the default nursery.

Finally, if a major collection collects more than 90% of the old generation, V8 resets all pre-tenured allocation sites, under the assumption that pre-tenuring was actually premature.

tenure for me but not for thee

What kinds of allocation sites are eligible for pre-tenuring? Sometimes it depends on object kind; wasm memories, for example, are almost always long-lived, so they are always pre-tenured. Sometimes it depends on who is doing the allocation; allocations from the bootstrapper, literals allocated by the parser, and many allocations from C++ go straight to the old generation. And sometimes the compiler has enough information to determine that pre-tenuring might be a good idea, as when it generates a store of a fresh object to a field in an known-old object.

But otherwise I thought that the whole AllocationSite mechanism would apply generally, to any object creation. It turns out, nope: it seems to only apply to object literals, array literals, and new Array. Weird, right? I guess it makes sense in that these are the ways to create objects that also creates the field values at creation-time, allowing the whole block to be allocated to the same space. If instead you make a pre-tenured object and then initialize it via a sequence of stores, this would likely create old-to-new edges, preventing the new objects from dying young while incurring the penalty of copying and write barriers. Still, I think there is probably some juice to squeeze here for pre-tenuring of class-style allocations, at least in the optimizing compiler or in short inline caches.

I suspect this state of affairs is somewhat historical, as the AllocationSite mechanism seems to have originated with typed array storage strategies and V8’s “boilerplate” object literal allocators; both of these predate per-AllocationSite pre-tenuring decisions.

fin

Well that’s adaptive pre-tenuring in V8! I thought the “just stick a memento after the object” approach is pleasantly simple, and if you are only bumping creation counters from baseline compilation tiers, it likely amortizes out to a win. But does the restricted application to literals point to a fundamental constraint, or is it just accident? If you have any insight, let me know :) Until then, happy hacking!

Qiuyi Zhang (Joyee): require(esm) in Node.js: from experiment to stability

2026-01-05T10:36:13+00:00

More than a year ago, I set out to revive require(esm) in Node.js and

Qiuyi Zhang (Joyee): require(esm) in Node.js: implementer's tales

2026-01-05T10:36:11+00:00

In earlier posts, I wrote about reviving require(esm) and its iteration process

Jasmine Tang: Rewriting analysis based warnings fall through

2026-01-05T00:00:00+00:00

Uhhh hey :)

Qiuyi Zhang (Joyee): require(esm) in Node.js

2025-12-30T18:58:11+00:00

Recently I landed experimental support for require()-ing synchronous ES modules in Node.js, a feature that has been long overdue

Igalia WebKit Team: WebKit Igalia Periodical #52

2025-12-25T18:26:43+00:00

Update on what happened in WebKit in the week from December 16 to December 25.

Right during the holiday season 🎄, the last WIP installment of the year comes packed with new releases, a couple of functions added to the public API, cleanups, better timer handling, and improvements to MathML and WebXR support.

Cross-Port 🐱

Landed support for font-size: math. Now math-depth can automatically control the font size inside of <math> blocks, making scripts and nested content smaller to improve readability and presentation.

Two new functions have been added to the public API:

webkit_context_menu_item_get_gaction_target() to obtain the GVariant associated with a context menu item created from a GAction.
webkit_context_menu_item_get_title() may be used to obtain the title of a context menu item.

Improved timers, by making some of them use the timerfd API. This reduces timer “lateness”—the amount of time elapsed between the configured trigger time, and the effective one—, which in turn improves the perceived smoothness of animations thanks to steadier frame delivery timings. Systems where the timerfd_create and timerfd_settime functions are not available will continue working as before.

On the WebXR front, support was added for XR_TRACKABLE_TYPE_DEPTH_ANDROID through the XR_ANDROID_trackables extension, which allows reporting depth information for elements that take part in hit testing.

Graphics 🖼️

Landed a change that implements non-composited page rendering in the WPE port. This new mode is disabled by default, and may be activated by disabling the AcceleratedCompositing runtime preference. In such case, the frames are rendered using a simplified code path that does not involve the internal WebKit compositor. Therefore it may offer a better performance in some specific cases on constrained embedded devices.

Since version 2.10.2, the FreeType library can be built with direct support for loading fonts in the WOFF2 format. Until now, the WPE and GTK WebKit ports used libwoff2 in an intermediate step to convert those fonts on-the-fly before handing them to FreeType for rendering. The CMake build system will now detect when FreeType supports WOFF2 directly and skip the conversion step. This way, in systems which provide a suitable version of FreeType, libwoff2 will no longer be needed.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

The legacy libwpe-based API can now be disabled at build time, by toggling the ENABLE_WPE_LEGACY_API CMake option. This allows removal of uneeded code when an application is exclusively using the new WPEPlatform API.

WPE Android ↗ 🤖

Adaptation of WPE WebKit targeting the Android operating system.

AHardwareBuffer is now supported as backing for accelerated graphics surfaces that can be shared across processes. This is the last piece of the puzzle to use WPEPlatform on Android without involving expensive operations to copy rendered frames back-and-forth between GPU and system memory.

Releases 📦️

WebKitGTK 2.50.4 and WPE WebKit 2.50.4 have been released. These stable releases include a number of important patches for security issues, and we urge users and distributors to update to this release if they have not yet done it. An accompanying security advisory, WSA-2025-0010, has been published (GTK, WPE).

Development releases of WebKitGTK 2.51.4 and WPE WebKit 2.51.4 are available as well, and may be used to preview upcoming features. As usual, bug reports are welcome in Bugzilla.

Community & Events 🤝

Paweł Lampe has published a blog post that discusses various pre-rendering techniques useful in the context of using WPE on embedded devices.

That’s all for this week!

Eric Meyer: Targeting by Reference in the Shadow DOM

2025-12-19T15:04:58+00:00

I’ve long made it clear that I don’t particularly care for the whole Shadow DOM thing. I believe I understand the problems it tries to solve, and I fully acknowledge that those are problems worth solving. There are just a bunch of things about it that don’t feel right to me, like how it can break accessibility in a number of ways.

One of those things is how it breaks stuff like the commandFor attribute on <button>s, or the popoverTarget attribute, or a variety of ARIA attributes such as aria-labelledby. This happens because a Shadow DOMmed component creates a whole separate node tree, which creates a barrier (for a lot of things, to be clear; this is just one class of them).

At least, that’s been the case. There’s now a proposal to fix that, and prototype implementations in both Chrome and Safari! In Chrome, it’s covered by the Experimental Web Platform features flag in chrome://flags. In Safari, you open the Develop > Feature Flags… dialog, search for “referenceTarget”, and enable both flags.

(Disclosure: My employer, Igalia, with support from NLnet, did the WebKit implementation, and also a Gecko implementation that’s being reviewed as I write this.)

If you’re familiar with Shadow DOMming, you know that there are attributes for the <template> element like shadowRootClonable that set how the Shadow DOM for that particular component can be used. The proposal at hand is for a shadowRootReferenceTarget attribute, which is a string used to identify an element within the Shadowed DOM tree that should be the actual target of any references. This is backed by a ShadowRoot.referenceTarget API feature.

Take this simple setup as a quick example.

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent">
	<template>
		<input id="setting" type="checkbox" aria-checked="indeterminate">
		<span id="box"></span>	
	</template> </sp-checkbox>

Assume there’s some JavaScript to make that stuff inside the Shadow DOM work as intended. (No, nothing this simple should really be a web component, but let’s assume that someone has created a whole multi-faceted component system for handling rich user interactions or whatever, and someone else has to use it for job-related reasons, and this is one small use of that system.)

The problem is, the <label> element’s for is pointing at consent, which is the ID of the component. The actual thing that should be targeted is the <input> element with the ID of setting . We can’t just change the markup to <label for="setting"> because that <input> is trapped in the Shadow tree, where none in the Light beyond may call for it. So it just plain old doesn’t work.

Under the Reference Target proposal, one way to fix this would look something like this in HTML:

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent">
	<template shadowRootReferenceTarget="setting">
		<input id="setting" type="checkbox" aria-checked="indeterminate">
		<span id="box"></span>	
	</template> </sp-checkbox>

With this markup in place, if someone clicks/taps/otherwise activates the label, it points to the ID consent . That Shadowed component takes that reference and redirects it to an effective target  — the reference target identified in its shadowRootReferenceTarget attribute.

You could also set up the reference with JavaScript instead of an HTML template:

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent"></sp-checkbox>

class SpecialCheckbox extends HTMLElement {
	checked = "mixed";
	constructor() {
		super();
		this.shadowRoot_ = this.attachShadow({ 
			referenceTarget: "setting"
		});
		
		// lines of code to Make It Go
	
	}
}

Either way, the effective target is the <input> with the ID of setting .

This can be used in any situation where one element targets another, not just with for . The form and list attributes on inputs would benefit from this. So, too, would the relatively new popoverTarget and commandFor button attributes. And all of the ARIA targeting attributes, like aria-controls and aria-errormessage and aria-owns as well.

If reference targets are something you think would be useful in your own work, please give it a try in Chrome or Safari or both, to see if your use cases are being met. If not, you can leave feedback on issue 1120 to share any problems you run into. If we’re going to have a Shadow DOM, the least we can do is make it as accessible and useful as possible.

Have something to say to all that? You can add a comment to the post, or [email protected]?subject=In%20reply%20to%20%22Targeting%20by%20Reference%20in%20the%20Shadow%20DOM%22">email Eric directly.

Pawel Lampe: WPE performance considerations: pre-rendering

2025-12-19T00:00:00+00:00

This article is a continuation of the series on WPE performance considerations. While the previous article touched upon fairly low-level aspects of the DOM tree overhead, this one focuses on more high-level problems related to managing the application’s workload over time. Similarly to before, the considerations and conclusions made in this blog post are strongly related to web applications in the context of embedded devices, and hence the techniques presented should be used with extra care (and benchmarking) if one would like to apply those on desktop-class devices.

The workload #

Typical web applications on embedded devices have their workloads distributed over time in various ways. In practice, however, the workload distributions can usually be fitted into one of the following categories:

Idle applications with occasional updates - the applications that present static content and are updated at very low intervals. As an example, one can think of some static dashboard that presents static content and switches the page every, say, 60 seconds - such as e.g. a static departures/arrivals dashboard on the airport.
Idle applications with frequent updates - the applications that present static content yet are updated frequently (or are presenting some dynamic content, such as animations occasionally). In that case, one can imagine a similar airport departures/arrivals dashboard, yet with the animated page scrolling happening quite frequently.
Active applications with occasional updates - the applications that present some dynamic content (animations, multimedia, etc.), yet with major updates happening very rarely. An example one can think of in this case is an application playing video along with presenting some metadata about it, and switching between other videos every few minutes.
Active applications with frequent updates - the applications that present some dynamic content and change the surroundings quite often. In this case, one can think of a stock market dashboard continuously animating the charts and updating the presented real-time statistics very frequently.

Such workloads can be well demonstrated on charts plotting the browser’s CPU usage over time:

As long as the peak workload (due to updates) is small, no negative effects are perceived by the end user. However, when the peak workload is significant, some negative effects may start getting noticeable.

In case of applications from groups (1) and (2) mentioned above, a significant peak workload may not be a problem at all. As long as there are no continuous visual changes and no interaction is allowed during updates, the end-user is unable to notice that the browser was not responsive or missed some frames for some period of time. In such cases, the application designer does not need to worry much about the workload.

In other cases, especially the ones involving applications from groups (3) and (4) mentioned above, the significant peak workload may lead to visual stuttering, as any processing making the browser busy for longer than 16.6 milliseconds will lead to lost frames. In such cases, the workload has to be managed in a way that the peaks are reduced either by optimizing them or distributing them over time.

First step: optimization #

The first step to addressing the peak workload is usually optimization. Modern web platform gives a full variety of tools to optimize all the stages of web application processing done by the browser. The usual process of optimization is a 2-step cycle starting with measuring the bottlenecks and followed by fixing them. In the process, the usual improvements involve:

using CSS containment,
using shadow DOM,
promoting certain parts of the DOM to layers and manipulating them with transforms,
parallelizing the work with workers/worklets,
using the visibility CSS property to separate painting from layout,
optimizing the application itself (JavaScript code, the structure of the DOM, the architecture of the application),
etc.

Second step: pre-rendering #

Unfortunately, in practice, it’s not uncommon that even very well optimized applications still have too much of a peak workload for the constrained embedded devices they’re used on. In such cases, the last resort solution is pre-rendering. As long as it’s possible from the application business-logic perspective, having at least some web page content pre-rendered is very helpful in situations when workload has to be managed, as pre-rendering allows the web application designer to choose the precise moment when the content should actually be rendered and how it should be done. With that, it’s possible to establish a proper trade-off between reduction in peak workload and the amount of extra memory used for storing the pre-rendered contents.

Pre-rendering techniques #

Nowadays, the web platform provides at lest a few widely-adapted APIs that provide means for the application to perform various kinds of pre-rendering. Also, due to the ways the browsers are implemented, some APIs can be purposely misused to provide pre-rendering techniques not necessarily supported by the specification. However, in the pursuit of good trade-offs, all the possibilities should be taken into account.

Before jumping into particular pre-rendering techniques, it’s necessary to emphasize that the pre-rendering term used in this article refers to the actual rendering being done earlier than it’s visually presented. In that sense, the resource is rasterized to some intermediate form when desired and then just composited by the browser engine’s compositor later.

Pre-rendering offline #

The most basic (and limited at the same time) pre-rendering technique is one that involves rendering offline i.e. before the browser even starts. In that case, the first limitation is that the content to be rendered must be known beforehand. If that’s the case, the rendering can be done in any way, and the result may be captured as e.g. raster or vector image (depending on the desired trade-off). However, the other problem is that such a rendering is usually out of the given web application scope and thus requires extra effort. Moreover, depending on the situation, the amount of extra memory used, the longer web application startup (due to loading the pre-rendered resources), and the processing power required to composite a given resource, it may not always be trivial to obtain the desired gains.

Pre-rendering using canvas #

The first group of actual pre-rendering techniques happening during web application runtime is related to Canvas and OffscreenCavas. Those APIs are really useful as they offer great flexibility in terms of usage and are usually very performant. However, in this case, the natural downside is the lack of support for rendering the DOM inside the canvas. Moreover, canvas has a very limited support for painting text — unlike the DOM, where CSS has a significant amount of features related to it. Interestingly, there’s an ongoing proposal called HTML-in-Canvas that could resolve those limitations to some degree. In fact, Blink has a functioning prototype of it already. However, it may take a while before the spec is mature and widely adopted by other browser engines.

When it comes to actual usage of canvas APIs for pre-rendering, the possibilities are numerous, and there are even more of them when combined with processing using workers. The most popular ones are as follows:

rendering to an invisible canvas and showing it later,
rendering to a canvas detached from the DOM and attaching it later,
rendering to an invisible/detached canvas and producing an image out of it to be shown later,
rendering to an offscreen canvas and producing an image out of it to be shown later.

When combined with workers, some of the above techniques may be used in the worker threads with the rendered artifacts transferred to the main for presentation purposes. In that case, one must be careful with the transfer itself, as some objects may get serialized, which is very costly. To avoid that, it’s recommended to use transferable objects and always perform a proper benchmarking to make sure the transfer is not involving serialization in the particular case.

While the use of canvas APIs is usually very straightforward, one must be aware of two extra caveats.

First of all, in the case of many techniques mentioned above, there is no guarantee that the browser will perform actual rasterization at the given point in time. To ensure the rasterization is triggered, it’s usually necessary to enforce it using e.g. a dummy readback (getImageData()).

Finally, one should be aware that the usage of canvas comes with some overhead. Therefore, creating many canvases or creating them often, may lead to performance problems that could outweigh the gains from the pre-rendering itself.

Pre-rendering using eventually-invisible layers #

The second group of pre-rendering techniques happening during web application runtime is limited to the DOM rendering and comes out of a combination of purposeful spec misuse and tricking the browser engine into making it rasterizing on demand. As one can imagine, this group of techniques is very much browser-engine-specific. Therefore, it should always be backed by proper benchmarking of all the use cases on the target browsers and target hardware.

In principle, all the techniques of this kind consist of 3 parts:

Enforcing the content to be pre-rendered being placed on a separate layer backed by an actual buffer internally in the browser,
Tricking the browser’s compositor into thinking that the layer needs to be rasterized right away,
Ensuring the layer won’t be composited eventually.

When all the elements are combined together, the browser engine will allocate an internal buffer (e.g. texture) to back the given DOM fragment, it will process that fragment (style recalc, layout), and rasterize it right away. It will do so as it will not have enough information to allow delaying the rasterization of the layer (as e.g. in case of display: none). Then, when the compositing time comes, the layer will turn out to be invisible in practice due to e.g. being occluded, clipped, etc. This way, the rasterization will happen right away, but the results will remain invisible until a later time when the layer is made visible.

In practice, the following approaches can be used to trigger the above behavior:

for (1), the CSS properties such as will-change: transform, z-index, position: fixed, overflow: hidden etc. can be used depending on the browser engine,
for (2) and (3), the CSS properties such as opacity: 0, overflow: hidden, contain: strict etc. can be utilized, again, depending on the browser engine.

The scrolling trick

While the above CSS properties allow for various combinations, in case of WPE WebKit in the context of embedded devices (tested on NXP i.MX8M Plus), the combination that has proven to yield the best performance benefits turns out to be a simple approach involving overflow: hidden and scrolling. The example of such an approach is explained below.

Suppose the goal of the application is to update a big table with numbers once every N frames — like in the following demo: random-numbers-bursting-in-table.html?cs=20&rs=20&if=59

With the number of idle frames (if) set to 59, the idea is that the application does nothing significant for the 59 frames, and then every 60th frame it updates all the numbers in the table.

As one can imagine, on constrained embedded devices, such an approach leads to a very heavy workload during every 60th frame and hence to lost frames and unstable application’s FPS.

As long as the numbers are available earlier than every 60th frame, the above application is a perfect example where pre-rendering could be used to reduce the peak workload.

To simulate that, the 3 variants of the approach involving the scrolling trick were prepared for comparison with the above:

In the above demos, the idea is that each cell with a number becomes a scrollable container with 2 numbers actually — one above the other. In that case, because overflow: hidden is set, only one of the numbers is visible while the other is hidden — depending on the current scrolling:

With such a setup, it’s possible to update the invisible numbers during idle frames without the user noticing. Due to how WPE WebKit accelerates the scrolling, changing the invisible numbers, in practice, triggers the layout and rendering right away. Moreover, the actual rasterization to the buffer backing the scrollable container happens immediately (depending on the tiling settings), and hence the high cost of layout and text rasterization can be distributed. When the time comes, and all the numbers need to be updated, the scrollable containers can be just scrolled, which in that case turns out to be ~2 times faster than updating all the numbers in place.

To better understand the above effect, it’s recommended to compare the mark views from sysprof traces of the random-numbers-bursting-in-table.html?cs=10&rs=10&if=11 and random-numbers-bursting-in-table-prerendered-1.html?cs=10&rs=10&if=11 demos:

While the first sysprof trace shows very little processing during 11 idle frames and a big chunk of processing (21 ms) every 12th frame, the second sysprof trace shows how the distribution of load looks. In that case, the amount of work during 11 idle frames is much bigger (yet manageable), but at the same time, the formerly big chunk of processing every 12th frame is reduced almost 2 times (to 11 ms). Therefore, the overall frame rate in the application is much better.

Results

Despite the above improvement speaking for itself, it’s worth summarizing the improvement with the benchmarking results of the above demos obtained from the NXP i.MX8M Plus and presenting the application’s average frames per second (FPS):

Clearly, the positive impact of pre-rendering can be substantial depending on the conditions. In practice, when the rendered DOM fragment is more complex, the trick such as above can yield even better results. However, due to how tiling works, the effect can be minimized if the content to be pre-rendered spans multiple tiles. In that case, the browser may defer rasterization until the tiles are actually needed. Therefore, the above needs to be used with care and always with proper benchmarking.

Conclusions #

As demonstrated in the above sections, when it comes to pre-rendering the contents to distribute the web application workload over time, the web platform gives both the official APIs to do it, as well as unofficial means through purposeful misuse of APIs and exploitation of browser engine implementations. While this article hasn’t covered all the possibilities available, the above should serve as a good initial read with some easy-to-try solutions that may yield surprisingly good results. However, as some of the ideas mentioned above are very much browser-engine-specific, they should be used with extra care and with the limitations (lack of portability) in mind.

As the web platform constantly evolves, the pool of pre-rendering techniques and tricks should keep evolving as well. Also, as more and more web applications are used on embedded devices, more pressure should be put on the specification, which should yield more APIs targeting the low-end devices in the future. With that in mind, it’s recommended for the readers to stay up-to-date with the latest specification and perhaps even to get involved if some interesting use cases would be worth introducing new APIs.

Delan Azabani: Web engine CI on a shoestring budget

2025-12-18T10:00:00+00:00

Servo is a greenfield web browser engine that supports many platforms. Automated testing for the project requires building Servo for all of those platforms, plus several additional configurations, and running nearly two million tests including the entire Web Platform Tests. How do we do all of that in under half an hour, without a hyperscaler budget for compute and an entire team to keep it all running smoothly, and securely enough to run untrusted code from contributors?

We’ve answered these questions by building a CI runner orchestration system for GitHub Actions that we can run on our own servers, using ephemeral virtual machines for security and reproducibility. We also discuss how we implemented graceful fallback from self-hosted runners to GitHub-hosted runners, the lessons we learned in automating image rebuilds, and how we could port the system to other CI platforms like Forgejo Actions.

This is a transcript post for a talk I gave internally at Igalia.

Let's talk about how Servo can have fast CI for not a lot of money.

Servo is a greenfield web browser engine. And being a web browser engine, it has some pretty demanding requirements for its CI setup.

We have to build and run Servo for many platforms, including three desktop platforms and two mobile platforms.

We have to run many, many tests, the main bulk of which is the entire Web Platform Tests suite, which is almost 2 million subtests. We also have several smaller test suites as well, like the WebGPU tests and the DevTools tests and so on.

We have to build Servo in many different configurations for special needs. So we might build Servo with the oldest version of Rust that we still support, just to make sure that still works. We might build Servo as a library in the same way that it would be consumed by embedders. We have to lint the codebase. We have to build with optimizations for nightly and monthly releases. We have to build with other optimizations for benchmarking work, and so on.

And as you'll see throughout this talk, we do this on GitHub and GitHub Actions, but GitHub Actions is a very frustrating CI service, and it has many baffling limitations. And as time goes on, I think Servo being on GitHub and GitHub Actions will be more for the network effects we had early on than for any particular merits of these platforms.

On GitHub Actions, GitHub provides their own first-party runners.

And these runners are very useful for small workloads, as well as the logic that coordinates workloads. So this would include things like taking a tryjob request for "linux", and turning that into a run that just builds Servo for Linux. Or you might get a try job request for "full", and we'll turn that into a run that builds Servo for all of the platforms and runs all of the tests.

But for a project of our scale, these runners really fall apart once you get into any workload beyond that.

They have runners that are free of charge. They are very tiny, resource constrained, and it seems GitHub tries to cram as many of these runners onto each server as they possibly can.

They have runners that you can pay for, but they're very very expensive. And I believe this is because you not only pay a premium for like hyperscaler cloud compute rates, but you also pay a premium on top of that for the convenience of having these runners where you can essentially just flip a switch and get faster builds. So they really make you pay for this.

Not only that, but using GitHub hosted runners you can't really customize the image that runs on the runners besides being able to pull in containers, which is also kind of slow and not useful all the time. If there are tools and dependencies that you need that aren't in those images, you need to install them every single time you run a job or a workflow, which is such a huge waste of time, energy, and money, no matter whose money it is.

There are also some caching features on GitHub Actions, but they're honestly kind of a joke. The caching performs really poorly, and there's not a lot you can cache with them. So in general, they're not really suitable for doing things like incremental builds.

So we have all these slow builds, and they're getting slower, and we want to make them faster. So we considered several alternatives.

The first that comes to mind are these third-party runner providers. These are things like Namespace, Warpbuild, Buildjet. There's so many of them. The caveat with these is that they're almost always... almost as expensive per hour as GitHub's first-party runners. And I think this is because they try to focus on providing features like better caching, that allow you to accumulate less hours on their runners. And in doing so, they don't really have any incentive to also compete on the hourly rate.

There is one exception: there's a system called RunsOn. It's sort of an off-the-shelf thing that you can grab this software and operate it yourself, but you do have to use AWS. So it's very tightly coupled to AWS. And both of these alternatives, they often lack support for certain platforms on their runners. Many of them are missing Windows or missing macOS or missing both. And RunsOn is missing macOS support, and probably won't get macOS support for the foreseeable future.

We considered offloading some of the work that these free GitHub hosted runners do onto our own servers with, let's call them like, "proxy jobs". The idea would be that you'd still use free GitHub hosted runners, but you'd do the work remotely on another server. The trouble with this is that then you're still using these free GitHub hosted runners, which take up your quota of your maximum concurrent free runners.

And it's also tricky to do this without losing access to the broader ecosystem of prebuilt Actions. So these Actions are like steps that you can pull in that will let you do useful things like managing artifacts and installing dependencies and so on. But it's one thing to say, okay, my workload is this shell script, and I'm going to run it remotely now. That's easy enough to do, but it's a lot harder to say, okay, well, I've got this workflow that has a bunch of scripts and a bunch of external Actions made by other people, and I'm going to run all of this remotely. I'm going to make sure that all of these Actions are also compatible with being run remotely. That's a lot trickier. That said, you should probably avoid relying too heavily on this ecosystem anyway. It's a great way to get locked into the platform, and working with YAML honestly really sucks.

So we could set up an entire CI service. There are some CI services like Jenkins, Bamboo... honestly, most CI services nowadays have support for built-in container orchestration. So they can spin up containers for each runner, for each job, autonomously. And while they can do containers, none of them have really solved the problem of virtual machine orchestration out of the box. And this is a problem for us, because we want to use virtual machines for that security and peace of mind, which I'll explain in more detail a little bit later. And seeing as we lack the dedicated personnel to operate an entire CI service and have that be on the critical path — we didn't want to have someone on call for outages — this was probably not the best option for us at the moment.

What we decided to do was set up some self-hosted runners.

These solve most of our problems, primarily because we can throw as much hardware and resources at these runners as we want. We can define the contention ratios.

And better yet, we can also customize the images that the runners use. By being able to customize the runner images, we can move a lot of steps and a lot of work that used to be done on every single workflow run, and do it only once, or once per build. We only rebuild the runner images, say, once a week. So that significantly saves... it cuts down on the amount of work that we have to do.

This is not just installing tools and dependencies, but it's also enabling incremental builds quite powerfully, which we'll see a little bit later on.

How much faster is this system with the self-hosted runners? Well, it turns out, quite a lot faster.

We've got this typical workflow that you use when you're testing your commits, when you're making a pull request. And if you kick off a tryjob like this, several of the jobs that we've now offloaded onto self-hosted runners are now taking 70, 80, even 90% less time than they did on GitHub hosted runners, which is excellent.

And even for the web platform tests, we found more modest time savings with the capacity that we've allocated to them. But as a result, even though the savings are a bit more modest, what's worth highlighting here is that we were able to go from twenty runners — we had to parallelise this test suite across twenty runners before — and now we only need three.

Some things that make our system unique and that we're pretty proud of — and these things apply to all versions of our system, which we've been developing over the last 12 to 18 months.

The first is that we build on top of the native CI service of the Git forge. So in this case, it's GitHub and GitHub Actions. It could be Forgejo and Forgejo Actions in the future, and we're working on that already.

We also want to give users more or less a transparent user experience, the idea being that users should not notice any changes in their day-to-day work, besides the fact that their builds have gotten faster. And I think for the most part, we have achieved that.

Our system supports all of the platforms that GitHub Actions supports for their runners, including Linux, Windows, and macOS, and we could even support some of the other platforms that Forgejo Actions supports in the future, including BSD.

We have the ability to set up a job so that it can try to use self-hosted runners if they're available, but fall back to GitHub hosted runners if there's none available for whatever reason, like maybe they're all busy for the foreseeable future, or the servers are down or something like that. We have the ability to fall back to GitHub hosted runners, and this is something that was quite complicated to build, actually. And we have a whole section of this talk explaining how that works, because this is something that's not actually possible with the feature set that GitHub provides in their CI system.

It's secure enough for a large public project like Servo, where we don't necessarily know all of our contributors all that well personally. And this is in large part because we use virtual machines, instead of just containers, for each run and for each job. My understanding is that it is possible to build a system like this securely with containers, but in practice it's a lot harder to get that right as a security boundary than if you had the benefit of a hypervisor.

And of course it is all completely self-hosted, which makes it about as cheap as it gets, because your operating expenses are really just the costs of bare metal compute.

Now, how cheap is that? Well, in our deployment in Servo, we spend about 300 EUR per month on servers that do general-purpose runners, and these handle most of our workload.

If we were to compare that to if we were running on GitHub-hosted runners, there'd be almost an order of magnitude increase in costs, somewhere like 1400 EUR to over 2000 EUR per month if we were doing the same work on GitHub-hosted runners.

There are also significant increases if we went with third-party runner providers as well, although not quite as much.

But in truth, this is actually kind of an unfair comparison, because it assumes that we would need the same amount of hours between if we were running on self-hosted runners or if we were running on GitHub hosted runners. And something that you'll see throughout this talk is that this is very much not the case. We spend so many fewer hours running our jobs, because we have to do so much less work on them. So in reality, the gap between our expenses with these two approaches would actually be a lot wider.

There are three ways to use our CI runner system.

The first one is the way that GitHub designed self-hosted runners to be used. The way they intend you to use self-hosted runners is that when you define a job, you use this runs-on setting to declare what kind of runners your job should run on. And you can use labels for GitHub hosted runners, or you can define arbitrary labels for your own runners and select those instead. But you have to choose one or the other in advance. And if you do that, you can't have any kind of fallback, which was a bit of a drawback for us, especially early on. One benefit of this, though, is that you get the natural ability to queue up jobs. Because if, at the time of queuing up a new job, if there's no self-hosted runners available yet, the job will stay in a queued state until a self-hosted runner becomes available. And it's nice that you essentially get that for free. But you have no ability to have this kind of graceful fallback.

So we built some features to allow you to do graceful fallback. And how this works is that each of the servers that operates these runners has a web API as well. And you can hit that web API to check if there are runners available and reserve them. Reserving them is something you have to do if you're doing graceful fallback. But I'll explain that in more detail a bit later on. And in doing so, because you have a job that can check if there are runners available, you can now parameterize the runs-on setting and decide "I want to use a self-hosted runner, or a GitHub hosted runner". It's unclear if this is going to be possible on Forgejo Actions yet, so we may have to add that feature, but it's certainly possible on GitHub Actions.

Now, the downside of this is that you do lose the ability, that natural ability, to queue up jobs, and I'll explain why that is a bit later. But in short, we have a queue API that mitigates this problem, because you can hit the queue API, and it can have a full view of the runner capacity, and either tell you to wait, or forward your request once capacity becomes available.

Some things that you can do with our system, that you can't do with GitHub hosted runners. One of them is check out the repo significantly faster.

Something about GitHub Actions and how it works is that, if you run a job, you run a workflow in a repo, you don't actually get a checkout of the repo. You don't get a clone of the repo out of the box, unless you explicitly add a step that does a checkout. And this is fine for the most part, it works well enough for most users and most repos.

But the Servo repo has over 130,000 files across 6,000 directories, and that's just the tracked files and directories. And as a result, even if we use things like shallow clones, there's just kind of no getting around the fact that cloning this repo and checking it out is just slow. It's unavoidably slow.

And it's especially slow on Windows and macOS, where the filesystem performance is honestly often pretty poor compared to Linux. So we want to make our checkouts faster.

Well, what we can do is, we can actually move the process of cloning and checking out the repo from the build process, and move that into the image build process. So we only do it once, when we're building the runner images.

Then what we can do is go into the jobs that run on self-hosted runners, and switch out the stock checkout action with our own action. And our own action will just use the existing clone of the repo, it will fetch the commit that we actually need to build on, then switch to it, and check it out.

And as a result, we can check out the Servo repo pretty reliably in a couple of seconds, instead of having to check out the entire repo from scratch.

Something that flows on from that, though, is that if we are baking a copy of the Servo repo into our runner images, well, what if we just bake a copy of the built artifacts as well? Like, why don't we just build Servo, and bake that into the image?

And this will allow us to do incremental builds, because Servo is a Rust project, and we use Cargo, and Cargo supports incremental builds. As a result, by doing this, when you run a job on our CI system, most of the time we only have to rebuild a handful of crates that have changed, and not have to rebuild all of Servo from scratch.

Now, this is not perfect. Sometimes we'll have some compilation units that get falsely rebuilt, but this works well enough, for the most part, that it's actually a significant time savings.

I also probably wouldn't trust this for building artifacts that you actually want to release in a finished product, just because of the kinds of bugs that we've seen in Cargo's incremental build support.

But for everything else, just your typical builds where you do like commit checks and such, this is very very helpful.

Servo has this system deployed, and it's had it deployed for at least the last year or so.

Nowadays we have three servers which are modestly large, and we use these for the vast majority of our workload. We also have two smaller servers that we use for specific benchmarking tasks.

The stack on these servers, if you could call it that, is largely things that I personally was very familiar with, because I built this. So we've got NixOS for config management, the hypervisor is libvirt and Linux KVM, and the storage is backed by ZFS. The process of actually building the images, and managing the lifecycle of the virtual machines though, is done with some custom orchestration tooling that we've written.

This tooling consists of two services. The monitor service, which runs on every server that operates self-hosted runners, and the queue service, which is single and global.

So the monitor service does the vast majority of the work. It rebuilds virtual machine images, these templates. It clones the templates into virtual machines for each runner, for each job. It registers the runners with the CI service using its API, so that it can receive jobs. And it can also label the runners to tie them to specific jobs when asked to reserve them. This is optional, but it is a required step if we're doing graceful fallback.

Now, with graceful fallback, you do lose the ability to naturally queue up jobs. So what we've put on top of that is a single queue service that sits in front of the cluster, and it essentially acts as a reverse proxy. It's quite thin and simple, and there's a single one of them, not one per server, for the same reason as the general principle of, like, when you go to a supermarket, it's more efficient to have a single large queue, a single long queue of customers that gets dispatched to a bunch of shopkeepers. That's more efficient than having a sort of per-shopkeeper queue, especially when it comes to, like, moving jobs dynamically in response to the availability.

So we've got a whole section here about how graceful fallback works. I might cut this from shorter versions of the talk, but let's jump into it.

On GitHub Actions, every job has this runs-on setting, that defines what kind of runner it needs to get assigned to. It has to define this in advance, before the job runs.

And annoyingly, once you choose, "I want to run on a GitHub hosted runner" or "I want to run on a self-hosted runner", once you decide that, there's no turning back. Now, if you've decided that my job needs to run on a self-hosted runner, it can only run on a self-hosted runner. And now the outcome of your job, and the outcome of your workflow, now depends on that job actually getting a self-hosted runner with the matching criteria and succeeding. There's no way for that to become optional anymore.

And you might say, okay, well, maybe I can work around this by spinning up my self-hosted runners, but just giving them the same labels that GitHub assigns to their runners, maybe it'll do something smart? Like maybe... it will run my self-hosted runners if they're available and fall back. But no, the system has no sense of priority and you can't even define any sense of priority of like, I want to try this kind of runner and fall back to another kind of runner. This is simply not possible with the GitHub Action system.

So it may seem like it's impossible to do graceful fallback, but we found a way.

[self-hosted label] | no => [GitHub-hosted label] 2. $ echo " /> [self-hosted label] | no => [GitHub-hosted label] 2. $ echo " /> [self-hosted label] | no => [GitHub-hosted label] 2. $ echo " /> [self-hosted label] | no => [GitHub-hosted label] 2. $ echo "label=${label}" | tee -a $GITHUB_OUTPUT - Use the step output in runs-on runs-on: ${{ needs.decision.outputs.label }} - But two decisions must not be made concurrently">

What we can do is add a decision job, for each workload job that may need to run on self-hosted runners. And we prepend this job, and its job is to choose a runner label.

So how it essentially works is: it checks if a runner is available, and based on that, either chooses a self-hosted label or a GitHub hosted label. And it chooses it and sets it in an output. And this output can get pulled in... in our workload job, the job that actually does our work, because now you can parameterize the runs-on setting, so that it takes the value from this previous decision job.

Unfortunately, it seems like this may not be possible in Forgejo Actions just yet, so we might have to develop support for that, but it's certainly possible in GitHub Actions today, and it has been for quite a while.

The one caveat with this approach is that you need to be careful to only do this decision process one at a time. You should not do two instances of this process concurrently and interleave them.

- runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)" /> - runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)" /> - runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)" /> - runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)">

The reason for this is something you'll see if you think about a lot of the answers that you get on Stack Overflow and the GitHub forums. If you look up, "how do I solve this problem? How do I define a job where I can fall back from self-hosted runners to GitHub hosted runners?"

And most of the answers there, they'll have a problem where they check if there are any runners that are available, and then, they will make the decision, committing to either a self-hosted runner or a GitHub hosted runner. The trouble is that in between, if another decision job comes in and tries to make the same kind of decision, they can end up "earmarking" the same runner for two jobs. But each runner is only meant to run one job, and it can only run one job, so one of the jobs will get left without a runner.

So we can start to fix this by actually reserving the runners when we're doing graceful fallback. And how we've done it so far, is that we've used the GitHub Actions API to label the runner when we want to reserve it, and we label it with a unique ID. Then the workload job can put that unique ID, that label, in its runs-on setting. Instead of a general runner label, it can tie itself to this specific, uniquely identified runner label.

And we did it this way initially, because it allowed us to do it inside the decision job, at first. I think in the future, we will have to move away from this, because on Forgejo Actions, the way runner labels work is quite different. They're not something that you can sort of update after the fact. In fact, they're actually kind of defined by the runner process. So this approach for reserving runners won't work on Forgejo Actions. We'll probably have to do that internally on the runner servers. But in the meantime, we use labeling. Yeah, so at first we did this inside the decision job.

There are some problems with this. One of them is that now the decision job has to have a GitHub token that has permissions to manage runners and update their labels. And this is something that we'd like to avoid if possible, because we'd like our jobs to have least privilege that they need to do their work.

And something I didn't mention is that reserving runners in this way doesn't actually solve the problem on its own, because you've now transformed the problem to being, okay, we're going to check if the runner is not yet reserved. We're going to check if there's an unreserved runner, and then, we're going to label the runner. But in between, there's a moment where another process doing the same thing could make the same decision. And as a result, if we just did this, we could end up with a situation where one runner gets assigned two unique labels, but it can only fulfill one of them. So we have that same problem again.

So you might say, okay, well, it looks like GitHub Actions has this neat job concurrency feature. I mean, they say you can use this to define a job in a way where only one of them will run at a time, and you can't run them concurrently, so let's try using that to solve this problem.

What you'll find, though, is that if you try to solve the problem with job concurrency, as soon as there's even the slightest bit of contention, you'll just have your runs starting to fail spuriously, and you'll have to keep rerunning your jobs, and it'll be so much more annoying.

And the reason for this is that, if you look more closely at the docs, job concurrency essentially has a queue limited to one job. So at a maximum, you can have one job, one instance of the job that's running, one instance of the job that's queued, and then if another instance of the job comes in while you have one running and one queued, then those extra jobs just get cancelled, and the build fails. So unfortunately, job concurrency does not solve this problem.

So to solve these problems, what we did is we moved that decision and runner reservation process out of the decision jobs, and into the servers that operate the runners themselves. And we do this with an API that runs on the servers.

One smaller problem you might notice though, is that there's still a possibility that you could reserve a runner, but then after you've reserved the runner, it might fail to actually run. And this has become a lot less likely in our system, in our experience nowadays, now that we've ironed out most of the bugs, but it can still happen from time to time, usually due to infrastructure or network connectivity failures.

And we wanted to solve this by setting a time limit on how long a job can be queued for, because if it can't actually get a runner in practice, it will get stuck in that queued state indefinitely. But unfortunately, while we can set a time limit on how long a job can run for once it's been assigned, we can't actually set a time limit on how long a job can be queued for.

So we have to rely on the default built-in limit for all jobs, where I think there's like a limit of a day or two? So like, 24 or 48 hours or so, for how long a job can be queued? And this is just a really long time. So as a result, whenever this happens, you essentially have to cancel the job manually, or if you don't have permission to do that, you have to go ask someone who has permission, to go and cancel your job for you, which is really annoying.

So we solved that using timeout jobs. A timeout job is a sibling to your workload job, and it acts like a watchdog, and it ensures that it actually got a runner when you expect to get a self-hosted runner.

And how that works is, we wait a short amount of time, just like a minute or two, which should be long enough for the runner to actually start running the job, and then we query the API of the CI service to check if the workload job actually started running, or if it's still in that queued state. If it's still queued after two minutes, we cancel the run.

Unfortunately, we can't just cancel the job run. We do have to cancel the whole workflow run, which is quite annoying. But, you know, it's GitHub, nothing is surprising anymore.

Thankfully, we only have to run this when we actually get a self-hosted runner, so we can make it conditional. But over time, in Servo's deployment, we have actually stopped using these, to free up some of those GitHub-hosted runner resources.

One challenge that comes in making these timeout jobs is identifying the workload jobs uniquely, so we can look up whether it's still queued or whether it's started running. There are unique IDs for each job run. These are just like an incrementing number, and you'd think we'd be able to use this number to look up the workload job uniquely and robustly.

Unfortunately, you can't know the run ID of the job [correction 2025-12-24] ~~until it starts, and it may not ever start... or at least you may not know it~~ until the workflow runs, and there can be many instances of the job in the workflow because of workflow calls. Workflow calls are a feature that essentially allows you to inline the contents of a workflow in another as many times as you like. And as a result, you can have multiple copies, multiple instances of a job that run independently within one workflow run. So we definitely need a way to uniquely look up our instance of the workload job.

The trouble is that the only job relationship you can do in GitHub Actions is a needs relationship, and that's inappropriate for our situation here, because we can say that the workload job needs the decision job, we can say the timeout job needs the decision job — and in fact we do both of these, we "need" to do both of these — but we can't say that the timeout job needs the workload job, because of how needs works.

How needs works is that if job A needs job B, then job B has to actually get assigned a runner, and run, and complete its run — it has to finish — before job A can even start. And in this situation, we're making a timeout job to catch situations where the workload job never ends up running, so if we expressed a needs relationship between them, then the timeout job would never run, in these cases at least.

And even if we could express a needs relationship between jobs, like maybe we could walk the job tree, and go from the timeout job, through the decision job via the needs relationship, and then walk back down to the workload job using the same kind of needs relationship... unfortunately, none of these needs relationships are actually exposed in the API for a running workflow. So like, they're used for scheduling, but when you actually go and query the API, you can't tell what job needs what other job. They're just three jobs, and they're unrelated to one another.

in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))" /> in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))" /> in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))" /> in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))">

So how we had to end up solving this is, we had to tie these jobs together by generating a unique ID, a UUID, and putting it in the friendly name, like the display name of the job, like this.

And to query the CI service to find out if that job is still queued, we need to query it for the whole workflow run, and just look at all of the jobs, and then find the job whose name contains that unique ID.

Then we can check the status, and see if it's still queued. This is really, really silly. Yes, we really do have to string match the name, which is bananas! But this is GitHub Actions, so this is what we have to do.

One thing I didn't mention is that being able to reserve runners needs to be kind of a privileged operation, because we don't just want an arbitrary client on the internet to be able to erroneously or maliciously reserve runners. Even if they may not be able to do a whole lot with those runners, they can still deny service.

So to use the monitor API to do graceful fallback and to request and reserve a runner, normally we would require knowledge of some kind of shared secret, like an API token, and that's what we've done for most of the life of this system.

The trouble with this is that there are many kinds of workflow runs that don't have access to the secrets defined in the repo. A big one is pull_request runs. Most of the time, pull_request runs don't have access to secrets defined in the repo. And there is another kind of run called a pull_request_target run, and those do have access to secrets, but they also have some pretty gnarly security implications that mean that in general, you wanna avoid using these for pull requests anyway.

So if you're stuck with pull_request runs for your pull requests, does that mean that you can't use self-hosted runners? How do we allow pull_request runs to request and reserve runners in a way that it can prove that its request is genuine and authorized?

What we do is we use artifacts. We upload a small artifact that encodes the details of the request and publish the artifact against the run. So in the artifact, we'd say, "I want two Ubuntu runners" or "one Windows runner" or something like that.

And then we would hit the monitor API, we hit a different endpoint that just says, go to this repo, go to this run ID, and then check the artifacts! You'll see my request there! And this does not require an API token, it's not a privileged operation. What's privileged is publishing the artifact, and that's unforgeable; the only entity who can publish the artifact is the workflow itself.

All we have to do then, all we have to be careful to do, is to delete the artifact after we reserve the runner, so that it can't be replayed by a malicious client. And in the event that deleting the artifact fails, we can also set the minimum auto-delete period of, I think at the moment it's 24 hours, just in case that fails.

Graceful fallback normally means that you lose the ability to naturally queue jobs for self-hosted runners. And this happens because when you hit the API, requesting, you know, are there any available runners, please reserve one for me. If there aren't any available at the time of the request, we will immediately fall back to running on a GitHub hosted runner.

But the thing is, is that our self-hosted runners are generally so much faster! A GitHub hosted run might take five times as long as the equivalent self-hosted run, and if that was the case, it would actually be beneficial to wait up to 80% of that time, and we'd still probably save time.

So to increase the utilization of our self-hosted runners, what we can do is run a small queue service that sits in front of the monitors, and it essentially acts as a reverse proxy. It will take the same kind of requests for reserving runners as before, but it will have a global view of the availability and the runner capacity at any given moment.

And based on that, it will either respond to the client saying, go away and please try again later, or it will forward the request onto one of the monitors based on the available capacity.

We also have some lessons here about how to automate building virtual machine images, essentially. And these are lessons that we've learned over the last year or two.

GitHub Actions uses Packer for their first-party runner images, so they use Packer to build those images. And our monitor service also automates building the images, but we don't use Packer.

Initially, we had a handful of scripts that we just kicked off manually whenever we needed to update our images, but we've now fully automated the process. And we've done this using some modules in our monitor service, so there's some high-level Rust code that drives these image rebuilds, and it even uses reflink copies to take advantage of copy-on-write time savings and space savings with ZFS.

Now, one of the complications we ran into when building images, over the past year or so, is that we used to pull a sort of clever trick where, we would do as much of the process of configuring the virtual machine image on the host, actually, as possible, rather than doing configuration inside the guest, and having to spin up the guest so that we can configure it. And we were able to do this by essentially mounting the root file system of the guest image, on the host, and then injecting files, like injecting tools and scripts and other things that are needed by the image and needed to configure the image.

But we stopped being able to do this eventually because, well, essentially because of macOS. macOS has no file system that's usable for building Servo that can also be mounted on Linux with write support. Because like, just think about them, right? We've got HFS+, which Linux can write to but only if there's no journaling and you can't install macOS on HFS+ without journaling. There's APFS, which Linux has no support for. There's exFAT, which has no support for symlinks, so a lot of tools like uv will break. There's NTFS, which we thought would be our savior, but when we tried to use it, we ran into all sorts of weird build failures, which we believe are due to some kind of filesystem syncing race condition or something like that, so NTFS was unusable as well.

In the end, what we had to do is, if a guest image needed any tools or dependencies, it had to download them from a web server on the host.

We eventually settled on a consistent approach to automating the process of installing the OSes and configuring them for each runner template.

And the approach that we used was to install the OS using whatever method is easiest, and to bootstrap the config using any native config management system, if there is one included with the operating system.

But once we've kicked off that process, we then use the native config management as little as possible. And we do this because a lot of the config management tools that are built into these operating systems are quite quirky and annoying, and they are built for needs that we don't have, the primary need being that they can manage the configuration of a system over time, keeping it up to date with any changes. The thing about these runner images, though, is that each runner image only needs to get built once, it only needs to get configured once, and then after after that it'll be cloned for each runner, and then it will only run once, and this kind of sort of "one shot" use case is something that's kind of overkill with config management systems.

We can just do it with the usual scripting and automation facilities of the operating system. How does that look like in practice?

Well for Linux, we install the OS using pre-built Ubuntu cloud images, that we just download from the mirrors.

We bootstrap the config using cloud-init, which is such a painful... it's so painful to use cloud-init. We use it because it's included with the operating system, so that means it's the fastest possible way to get started.

We use it as little as possible: we just configure the logs to go to a TTY, we configure the network, so we can connect to the network, and then once we're on the network, we just curl and run a bash script which does the rest.

The same goes for Windows.

We install the operating system using an automated answers file called autounattend.xml. There's a nice little generator here which you can use if you don't want to have to set up a whole Windows server to set up unattended installations. You generate that XML file, and you can also use that XML file to bootstrap the config.

Again we use it as little as possible, because writing automations as XML elements is kind of a pain. So we essentially just set up a scheduled task to run the next stage of the config, we install the network driver, we import some registry settings, and we reboot. That's it. The rest of it is done with a PowerShell script.

The same goes for macOS.

Now, unfortunately, installing the OS and bootstrapping the config does have to be done by hand. And this is because if you want to automate a macOS installation, your two options, more or less, are enterprise device management solutions, which cost a lot of money and mean that you have to have a macOS server around to control and orchestrate the servers. But if you don't want to use one of those enterprise solutions, what most open systems that are faced with this problem end up doing is to throw OpenCV at the problem. I've seen several projects use OpenCV to OCR the setup wizard, which is... it's certainly a bold strategy. It's not really for me.

What I decided to do instead is just install the OS by hand, and pipe curl into sh to kick off the config management process. And this is something that we only really have to do once, because we do it once, and then we take a snapshot of it, and then we never have to do it again, at least until the next version of macOS comes out.

So this bootstrap script just does a handful of minimal things: it enables automatic login, it sets up a LaunchAgent to ensure that we can run our own code on each boot, and then it does a handful of other things which it honestly doesn't really have to do in this script. We could probably do these things in the zsh script which we then curl and run. And that's where the remainder of the work is done.

So looking forward, some things that we'd like to do with this system.

The first is to decouple it from Servo. So we built this CI system quite organically over the past 12 to 18 months, and we built it around Servo's needs specifically, but we think this system could be more broadly useful for other projects. We'll just have to abstract away some of the Servo specific bits, so that it can be more easily used on other projects, and that's something we're looking into now.

Something else that we'll have to do sooner or later is add support for macOS runners on Apple Silicon, that is ARM64, and the reason we have to do this is that macOS 26, which is the most recent version of macOS that came out in September, that's just a couple months ago, that is the last version of macOS that will support x86 CPUs. And at the moment, our macOS runners run on x86 CPUs, on the host and in the guest.

This is a little bit complicated because at the moment, our macOS runners actually run on Linux hosts, using a Linux-KVM-based sort of Hackintosh-y kind of solution. And there is no counterpart for this for arm64 hosts and guests, and I'm not sure there ever will be one. So we're going to have to port the system so that it can run on macOS hosts, so we can use actual Mac hardware for this, which is easy enough to do, and that's in progress.

But we're also going to have to port the system so it can run with other hypervisors. And this is because, although libvirt supports macOS hosts, the support for the macOS Hypervisor framework and Virtualization framework is not mature enough to actually run macOS guests in libvirt. And I'm not sure how long that will take to develop, so in the meantime, we've been looking at porting the system so that when you're on a Mac, you can run with UTM instead, and that's been working fairly well so far.

We're also looking at porting the system so that it can run with Forgejo Actions and not just GitHub Actions. So Forgejo Actions is an open alternative to GitHub Actions that tries to be loosely compatible with GitHub Actions, and in our experience, from experimentation so far, we found that it mostly is loosely compatible. We think we'll only have to make some fairly minor changes to our system to make it work on both CI systems.

That said, this CI system could potentially be more broadly applicable to other CI services as well, because virtual machine orchestration is something that we haven't really seen any CI services have a great built-in solution for. So if this interests you and you want to use it on your project on some other CI service, then we'd appreciate knowing about that, because that could be something we would explore next.

The remaining ideas are things that I think we could look into to make our runners more efficient.

The big one is autoscaling. So at the moment when you set up a server to operate some self-hosted runners, you essentially have to statically pre-configure how many runners of each kind of runner you want to be kept operating. And this has worked well enough for us for the most part, but it does mean that there's some kind of wasted resources sometimes, when the moment-to-moment needs of the jobs that are queued up aren't well fitted to the composition of your runner configuration. So if we had the ability to dynamically respond to demand, or some kind of autoscaling, I think we could improve our runner utilization rates a little bit, and sort of get more out of the same amount of runner capacity, the same amount of server capacity.

There's a couple ideas here, also, about reducing boot times for the runners, which can be quite helpful if you have a big backlog of jobs queued up for these servers, and this is because time spent booting up each runner, each virtual machine, is time that cannot be spent doing real work.

So two ways we can think of to reduce these boot times are, to have hot spares ready to go, the idea being that, if you can spin up more runners than you actually intend to run concurrently, and just have them sitting around, then you can kind of amortize the boot times, and sort of get the boot process process out of the way. And the way you do this is by spinning up a whole bunch of runners, say maybe you spin up like twenty runners, even though you only intend to run four of them concurrently.

And what you do is you give these runners a token amount of RAM to start with. You give them like one gig of RAM instead of 16 or 32 gigs of RAM. And then when a job comes in, and you actually want to assign the runner out so that it can do the work, then you dynamically increase the RAM from one gig, or that token amount, to the actual amount, like 16 gigs or 32 gigs. And this should be fairly easy to do in practice. This is actually supported in libvirt using a feature known as memory ballooning. But there are some minor caveats, like you do lose the ability to do certain optimizations, like you can't do huge pages backing on the memory anymore. But for the most part, this should be fairly technically simple to implement.

Something that could be more interesting in the longer term is microVMs, things like Firecracker, which as I understand it, these microVMs can sort of take the concept of paravirtualization to its logical extreme. And what it means is that on kernels that support being run as microVMs, you can boot them in like one or two seconds, instead of 20 or 30 or 60 seconds. And this could save a great deal of time, at least for jobs that run on Linux and BSD. I don't know if I said Linux and macOS, but I meant Linux and BSD.

So yea, we now have a system that we use to speed up our builds in Servo's CI, and it works fairly well for us. And we think that it's potentially useful for other projects as well.

So if you're interested to find out more, or you're interested to find out how you can use the system in your own projects, go to our GitHub repo at servo/ci-runners, or you can go here for a link to the slides. Thanks! article > section:not(#specificity) { position: relative; display: flex; flex-flow: column nowrap; padding: 1em 0; } article > section:not(#specificity) > * { flex: 0 0 auto; } article > section:not(#specificity) > ._slide { max-height: 50vh; flex: 0 0 auto; position: sticky; top: 0; text-align: center; } article > section:not(#specificity) > ._slide > img { max-width: 100%; max-height: 50vh; } article > section:not(#specificity) > ._spacer { flex: 1 0 1em; }

Andy Wingo: in which our protagonist dreams of laurels

2025-12-17T22:42:40+00:00

I had a dream the other evening, in which I was at a large event full of hackers—funny, that this is the extent of my dreams at the moment; as a parent of three young kids, I don’t get out much—and, there, I was to receive an award and give a speech. (I know, I am a ridiculous man, even when sleeping.) The award was something about free software; it had the trappings of victory, but the vibe among attendees was numbness and bitter loss. Palantir had a booth; they use free software, and isn’t that just great?

My talk was to be about Guile, I think: something technical, something interesting, but, I suspected, something inadequate: in its place and time it would be a delight to go deep on mechanism but the moment seemed to call for something else.

These days are funny. We won, objectively, in the sense of the goals we set in the beginning; most software is available to its users under a free license: Firefox, Chromium, Android, Linux, all the programming languages, you know the list. So why aren’t we happy?

When I reflect back on what inspired me about free software 25 years ago, it was much more political than technical. The idea that we should be able to modify our own means of production and share those modifications was a part of a political project of mutual care: we should be empowered to affect the systems that surround us, to the extent that they affect us.

To give you an idea of the milieu, picture me in 1999. I left my home to study abroad on another continent. When I would go to internet cafés I would do my email and read slashdot and freshmeat as one did back then, but also I would often read Z magazine, Noam Chomsky and Michael Albert and Michael Parenti and Arundhati Roy and Zapatistas and all. I remember reading El País the day after “we” shut down the World Trade Organization meeting in Seattle, seeing front-page pictures of pink-haired kids being beat up by the cops and wishing I were there with them. For me, free software fit with all of this: the notion that a better world was possible, and we could build it together.

I won’t lie and say that the ideals were everything. I think much of my motivation to program is selfish: I like to learn, to find out, to do. But back then I felt the social component more strongly. Among my cohort, though, I think we now do free software because we did free software; the motive sedimented into mechanism. These are the spoils of victory: free is the default. But defaults lack a sense of urgency, of the political.

Nowadays the commons that we built is the feedlot of large language models, and increasingly also its waste pond. The software we make is free, but the system in which it is made is not; Linux Magazine 1, Z magazine 0.

All of this makes me think that free software as a cause has run its course. We were the vanguard, and we won. Our dreams of 25 years ago are today’s table stakes. Specifically for my copyleft comrades, it seems that the role of copyright as a societal lever has much less purchase; taken to its conclusion, we might find ourselves siding with Disney and OpenAI against Google.

If I had to choose an idea from the 90s to keep, I would take “another world is possible” over the four freedoms. For me, software freedom is a strategy within a broader humanist project of liberation. It was clever, in that it could motivate people from a variety of backgrounds in a way that was on the whole positive for the humanist project. It inspired me as a meaningful way in which I could work towards a world of people caring for each other. In that spirit, I would like to invite my comrades to reflect on their own hierarchy of principles; too often I see people arguing the fine points of “is this software free” according to a specific definition without appreciating the ends to which the software freedom definition is a means.

Anyway, it turns out that I did win something, the Award for the Advancement of Free Software, for my work on Guile over the years. My work on Guile has waxed and waned, and in these last few years of parenthood it has been rather the latter, but I am proud of some of the technical hacks; and it has been with a heart-warming, wondrous delight that I have been a spectator to the rise of Guix, a complete operating system built on Guile. Apart from its quite compelling technical contributions, I just love that Guix is a community of people working together to build a shared project. I am going to the Guix days in a month or so and in past years it has been such a pleasure to see so many people there, working to make possible another world.

In my dream, instead of talking about Guile, I gave a rousing and compelling impromptu invective against Palantir and their ilk. I thought it quite articulate; I was asleep. In these waking hours, some days later, I don’t know what I did say, but I think I know what I would like to have said: that if we take the means of free software to be the ends, then we will find ourselves arguing our enemies are our friends. Saying that it’s OK if some software we build on is made by people who facilitate ICE raids. People who build spy software for controlling domestic populations. People who work for empire.

What I would like to say is that free software is a strategy. As a community of people that share some kind of liberatory principles of which free software has been a part, let use free software as best we can, among many other strategies. If it fits, great. If you find yourself on the same side of an argument as Palantir, it’s time to back up and try something else.

Pablo Saavedra: Verifying ARM vs THUMB2 Instruction Sets in ELF Binaries

2025-12-16T08:17:53+00:00

When working with embedded Linux systems, in ARM-based boards, sometimes you need to determine whether a binary was compiled with ARM or THUMB2 instruction sets. This is a quick reference guide for checking this without relying on heavy tools like readelf or objdump. The Core Concept ARM uses a clever trick to distinguish between ARM […]

Igalia WebKit Team: WebKit Igalia Periodical #51

2025-12-15T19:58:42+00:00

Update on what happened in WebKit in the week from December 8 to December 15.

In this end-of-year special have a new GMallocString helper that makes management of malloc-based strings more efficient, development releases, and a handful of advancements on JSC's implementation of Temporal, in particular the PlainYearMonth class.

Cross-Port 🐱

Added GMallocString class to WTF to adopt UTF8 C strings and make them WebKit first class citizens efficiently (no copies). Applied in GStreamer code together with other improvements by using CStringView. Fixed other two bugs about string management.

JavaScriptCore 🐟

The built-in JavaScript/ECMAScript engine for WebKit, also known as JSC or SquirrelFish.

In JavaScriptCore's implementation of Temporal, added the with method for PlainYearMonth objects, as well as the equals, compare, and valueOf methods, and also the from method. Also implemented the toPlainDate method for PlainYearMonth objects.

Releases 📦️

Development releases of WebKitGTK 2.51.3 and WPE WebKit 2.51.3 are now available. These include a number of API additions and new features, and are intended to allow interested parties to test those in advance, prior to the next stable release series. As usual, bug reports are welcome in Bugzilla.

That’s all for this week!

José Dapena: Maintaining Chromium downstream: how can upstream help?

2025-12-11T00:00:00+00:00

As I write often, maintaining a downstream of Chromium is not easy. A lot of effort falls on the shoulders of the teams embedding Chromium, or creating products on top of the upstream Chromium project.

We covered this in the previous chapters of my series of blog posts about maintaining Chromium downstreams. Now, this post is going to be a bit different.

I start with a question:

What can upstream Chromium do to help downstreams?

This very same question was discussed in the Web Engines Hackfest breakout session that originated most of these posts. In this blog post, I will share some of the most interesting answers that came up in that session.

Better componentization #

One of the ideas was to move code around more aggressively to make it easier to reuse. Specifically, refactoring to move more and more code from //chrome to //components.

Chromium has gone a long way in that direction. Each of these changes allows downstreams to directly use only the components they need, instead of working on top of //chrome. But there is still room for improvement.

Some parts of //chrome are still not refactored and could be very useful, especially for downstreams shipping a browser. Some examples:

Tabs implementation.
Profiles.
Synchronization.

Improve extensibility #

In the same direction, supporting easier ways to provide alternative implementations, and add custom software components, was considered important.

Some examples:

Making it easier to support Chrome extensions without using //chrome, would allow implementing new browsers without bundling the Chromium UI.
Going further in the direction of what has been done with Ozone: the Chromium platform abstraction layer that helps to implement the support for a variety of OS (including Linux and X11). Similar steps could be taken at other levels to improve OS integration (system hardware encryption, accelerated video codecs, system IPC, and so on).

Downstream advocacy #

A very interesting proposal was to create the role of downstream advocates in the Chrome community.

They would act as an entry point for downstream projects wanting to interact with the Chrome community and be an official communication channel for downstreams to report their needs.

This would also increase awareness of the different ways Chromium is used by downstreams.

Today there are two channels that are somewhat similar: the Chromium Embedders mailing list and the #embedders Slack channel.

A two-way problem #

So far, three different problems raised by downstreams have been covered, and they seem like fair requests to the Chromium community.

But there is also work to do on the downstreams side.

Can downstreams contribute more of their work to upstream? Not only in code, but also in all the maintenance activities.

There is also code written for very specific downstream needs that could land upstream, as long as it does not become a burden to the common project. That means ownership and enough work bandwidth need to be in place.

Where are we now? #

There is a major change in the Chromium community: the creation of the Supporters of Chromium Based Browsers. What does it mean for embedders? Could it be a good way to channel requirements from the different downstream projects?

Two years after the Web Engines Hackfest session, we can see some improvements. But the general question is still valid:

What can upstream Chromium do to help downstreams?

The last post #

The next post in this series will be the last one. It will cover some typical problems downstream projects are facing today.

References #

Luke Lau: Closing the LLVM RISC-V gap to GCC, part 1

2025-12-09T16:00:00+00:00

At the time of writing, GCC beats Clang on several SPEC CPU 2017 benchmarks on RISC-V¹:

LLVM developers upstream have been working hard on the performance of generated code, in every part of the pipeline from the frontend all the way through to the backend. So when we first saw these results we were naturally a bit surprised. But as it turns out, the GCC developers have been hard at work too.

Sometimes a bit of healthy competition isn’t a bad thing, so this blog post is the first in a series looking at the work going on upstream to improve performance and catch up to GCC.

Please note that this series focuses on RISC-V. Other targets may have more competitive performance but we haven’t measured them yet. We’ll specifically be focusing on the high-performance application processor use case for RISC-V, e.g. compiling for a profile like RVA23. Unfortunately since we don’t have access to RVA23 compatible hardware just yet we’ll be benchmarking on a SpacemiT-X60 powered Banana Pi BPI-F3 with -march=rva22u64_v. We don’t want to use -mcpu=spacemit-x60 since we want to emulate a portable configuration that an OS distribution might compile packages with. And we want to include the vector extension, as we’ll see in later blog posts that optimization like auto-vectorization can have a major impact on performance.

Where to start?

It goes without saying that a vague task like “make LLVM faster” is easier said than done. The first thing is to find something to make fast, and while you could read through the couple dozen million lines of code in LLVM until inspiration strikes, it’s generally easier to start the other way around by analyzing the code it generates.

Sometimes you’ll get lucky by just stumbling across something that could be made faster when hacking or poring through generated assembly. But there’s an endless amount of optimizations to be implemented and not all of them are equally impactful. If we really want to make large strides in performance we need to take a step back and triage what’s actually worth spending time on.

LNT, LLVM’s nightly testing infrastructure, is a great tool for this task. It’s both a web server that allows you to analyze benchmark results, and a command line tool to help run the benchmarks and submit the results to said web server.

As the name might imply, it’s normally used for detecting performance regressions by running benchmarks daily with the latest revision of Clang, flagging any tests that may have become slower or faster since the last revision.

But it also allows you to compare benchmark results across arbitrary configurations. You can run experiments to see what effects a flag has, or see the difference in performance on two pieces of hardware.

Moreover, you can pass in different compilers. In our case, we can do two “runs” with Clang and GCC. Here’s how we would kick these off:

for CC in clang riscv64-linux-gnu-gcc
do
  lnt runtest test-suite bpi-f3-rva22u64_v-ReleaseLTO \
    --sandbox /var/lib/lnt/ \
    --test-suite=path/to/llvm-test-suite \
    -DTEST_SUITE_SPEC2017_ROOT=path/to/cpu2017 \
    --cc=$CC \
    --cflags="-O3 -flto -march=rva22u64_v" \
    --cxxflags="-O3 -flto -march=rva22u64_v" \
    --benchmarking-only \
    --build-threads=16 \
    # cross-compile and run on another machine over ssh
    --toolchain=rva22u64_v.cmake \	
    --remote-host=bpi-f3 \
    # fight noise, run each benchmark 3 times on the same core
    --exec-multisample=3 \
    --run-under="taskset -c 5" \
    # submit the results to a web server for easy viewing
    --submit=https://mylntserver.com/submitRun
done

This command does a lot of heavy lifting. First off it invokes CMake to configure a new build of llvm-test-suite and SPEC CPU 2017 with -O3 -flto -march=rva22u64_v. But because compiling the benchmarks on the Banana Pi BPI-F3 would be painfully slow, we’ve specified a CMake toolchain file to cross-compile to riscv64-linux-gnu from an x86-64 build machine. Here’s what the toolchain file looks like:

# rva22u64_v.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
set(ARCH riscv64)

If you’ve got your cross toolchain sysroots set up in the right place in /usr/riscv64-linux-gnu/, things should “just work” and CMake will magically build RISC-V binaries. On Debian distros you can install the *-cross packages for this:

$ apt install libc6-dev-riscv64-cross libgcc-14-dev-riscv64-cross
libstdc++-12-dev-riscv64-cross

(You could also use mmdebstrap, or see Alex Bradbury’s guide to this)

After the benchmarks are built it rsyncs over the binaries to the remote machine, and then sshes into it to begin running the benchmarks. It will expect the sandbox path where the binaries are built on the remote to also exist on the host, so something like /var/lib/lnt should work across both. The BPI-F3 can also produce some noisy results, so the --exec-multisample=3 and -run-under="taskset -c 5" tell it to run the benchmarks multiple times and pin them to the same core.

Finally it generates a report.json file and submits it to the web server of choice. Navigate to the web interface and you’ll be shown two “machines”, LNT’s parlance for a specific combination of hardware, compiler and flags. You should see something like: bpif3-rva22u64_v-ReleaseLTO__clang_DEV__riscv64 and bpif3-rva22u64_v-ReleaseLTO__gcc_DEV__riscv64. Clicking into one of these machines will allow you to compare it against the other.

Profiling

Once on the LNT web interface you’ll be presented with a list of benchmarks with a lot of red percentages beside them. We now know what is slower, but next we need to know why they’re slower. We need to profile these benchmarks to see where all the cycles are spent and to figure out what Clang is doing differently from GCC.

LNT makes this easy, all you need to do is add --use-perf=profile to the lnt runtest invocation and it will perform an additional run of each benchmark wrapped in perf record. Profiling impacts run time so it runs it separately to avoid interfering with the final results. If you want to override the default events that are sampled you can specify them with --perf-events=cycles:u,instructions:u,....

LNT will take care of copying back the collected profiles to the host machine and encoding them in the report, and in the web interface you’ll notice a “Profile” button beside the benchmark. Click on that and you’ll be brought to a side by side comparison of the profiles from the two machines:

From here you can dive in and see where the benchmark spends most of its time. Select a function from the dropdown and choose one with a particularly high percentage: This is how much it makes up overall of whatever counter is active in the top right, like cycles or instructions. Then do the same for the other run and you’ll be presented with the disassemblies side-by-side below. Most importantly, information about the counters is displayed inline with each instruction, much like the output of perf annotate.

You might find the per-instruction counter cycle information to be a bit too fine-grained, so personally I like to use the “Control-Flow Graph” view mode in the top left. This groups the instructions into blocks and lets you see which blocks are the hottest. It also shows the edges between branches and their destinations which makes identifying loops a lot easier.

A real example

Lets take a look at how we can use LNT’s web interface to identify something that GCC does but Clang doesn’t (but should). Going back to the list of SPEC benchmark results we can see 508.namd_r is about 17% slower, so hopefully we should find something to optimize in there.

Jumping into the profile we can see there’s a bunch of functions that all contribute a similar amount to the runtime. We’ll just pick the hottest one at 14.3%, ComputeNonbondedUtil::calc_pair_energy_fullelect(nonbonded*). It’s a pretty big function, but in GCC’s profile 71% of the dynamic instruction count comes from this single, albiet large block.

Looking at Clang’s profile on the opposite side we see a similar block that accounts for 85% of the function’s instruction count. This slightly higher proportion is a small hint that the block that Clang’s producing is sub-optimal. If we take the hint and stare at it for long enough, one thing starts to stand out is that Clang generates a handful of fneg.d instructions which GCC doesn’t:

	fneg.d  fa0, fa0
	fneg.d  ft0, ft0
	fneg.d  ft2, ft2
	fmul.d  fa3, ft5, fa3
	fmul.d  fa0, fa3, fa0
	fmul.d  ft0, fa3, ft0
	fmul.d  fa3, fa3, ft2
	fmadd.d fa2, fa4, fa2, fa0
	fmadd.d ft6, fa4, ft6, ft0
	fmadd.d fa4, fa4, ft1, fa3

fneg.d rd, rs1 negates a double and fmul.d multiplies two doubles. fmadd.d rd, rs1, rs2, rs3 computes (rs1*rs2)+rs3, so here we’re doing some calculation like (a*b)+(c*-d).

These fneg.ds and fmadd.ds are missing on GCC. Instead it emits fmsub.d, which is entirely absent from the Clang code:

	fmul.d fa1,fa4,fa1
	fmul.d ft10,fa4,fa5
	fmsub.d ft10,ft7,fa0,ft10
	fmsub.d fa5,ft7,fa5,fa1
	fmul.d fa1,fa4,fa1
	fmsub.d fa1,ft7,fa0,fa1

fmsub.d rd, rs1, rs2, rs3 computes (rs1*rs2)-rs3, so GCC is instead doing something like (a*b)-(c*d) and in doing so avoids the need for the fneg.d. This sounds like a missed optimization in LLVM, so lets take a look at fixing it.

Writing the (right) fix

The LLVM RISC-V scalar backend is pretty mature at this stage so it’s surprising that we aren’t able to match fmsub.d. But if you take a look in RISCVInstrInfoD.td, you’ll see that the pattern already exists:

// fmsub: rs1 * rs2 - rs3
def : Pat<(any_fma FPR64:$rs1, FPR64:$rs2, (fneg FPR64:$rs3)),
          (FMSUB_D FPR64:$rs1, FPR64:$rs2, FPR64:$rs3, FRM_DYN)>;

We’ll need to figure out why this pattern isn’t getting selected, so lets start by extracting the build commands so we can look under the hood and dump the LLVM IR:

$ cmake -B build -C cmake/caches/ReleaseLTO.cmake --toolchain=...
$ ninja -C build 508.namd_r -t clean
$ ninja -C build 508.namd_r -v
...
[44/45] : && llvm-project/build.release/bin/clang++ --target=riscv64-linux-gnu -march=rva22u64_v -O3 -fomit-frame-pointer -flto -DNDEBUG -fuse-ld=lld ... -o External/SPEC/CFP2017rate/508.namd_r/508.namd_r

This is an LTO build so the code generation step is actually happening during link time. To dump the IR we can copy and paste the link command from the verbose output and append -Wl,--save-temps to it, which in turn tells the Clang driver to pass --save-temps to the linker².

$ llvm-project/build.release/bin/clang++ -Wl,--save-temps --target=riscv64-linux-gnu -march=rva22u64_v -O3 -fomit-frame-pointer -flto -DNDEBUG -fuse-ld=lld ... -o External/SPEC/CFP2017rate/508.namd_r/508.namd_r 
$ ls External/SPEC/CFP2017rate/508.namd_r/508.namd_r*
External/SPEC/CFP2017rate/508.namd_r/508.namd_r
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.0.preopt.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.2.internalize.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.4.opt.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.5.precodegen.bc

The bitcode is dumped at various stages, and 508.namd_r.0.5.precodegen.bc is the particular stage we’re looking for. This is after all the middle-end optimisations have run and is as close as we’ll get before the backend begins. It contains the bitcode for the entire program though, so lets find the symbol for the C++ function and extract just that corresponding LLVM IR function:

$ llvm-objdump -t 508.namd_r | grep calc_pair_energy_fullelect
...
000000000004562e l     F .text	0000000000001c92 _ZN20ComputeNonbondedUtil26calc_pair_energy_fullelectEP9nonbonded
$ llvm-extract -f 508.namd_r.0.5.precodegen.bc --func _ZN20ComputeNonbondedUtil26calc_pair_energy_fullelectEP9nonbonded \
  | llvm-dis > calc_pair_energy_fullelect.precodegen.ll

Now quickly grep the disassembled LLVM IR to see if we can find the source of the fnegs:

  %316 = fneg double %315
  %neg = fmul double %mul922, %316
  %317 = tail call double @llvm.fmuladd.f64(double %mul919, double %314, double %neg)

This looks promising. We have a @llvm.fmuladd that’s being fed by a fmul of a fneg, which is similar to the (a*b)+(c*-d) pattern in the resulting assembly. But looking back to our TableGen pattern for fmsub.d, we want (any_fma $rs1, $rs2, (fneg $rs3)), i.e. a llvm.fmuladd fed by a fneg of a fmul.

One thing about floating point arithmetic is that whilst it’s generally not associative, we can hoist out the fneg from the fmul since all negation does is flip the sign bit. So we can try to teach InstCombine to hoist the fneg outwards like (fmul x, (fneg y)) -> (fneg (fmul x, y)). But if we go to try that out we’ll see that InstCombine already does the exact opposite:

Instruction *InstCombinerImpl::visitFNeg(UnaryOperator &I) {
  Value *Op = I.getOperand(0);
  // ...
  Value *OneUse;
  if (!match(Op, m_OneUse(m_Value(OneUse))))
    return nullptr;

  if (Instruction *R = hoistFNegAboveFMulFDiv(OneUse, I))
    return replaceInstUsesWith(I, R);
  // ...
}

Instruction *InstCombinerImpl::hoistFNegAboveFMulFDiv(Value *FNegOp,
                                                      Instruction &FMFSource) {
  Value *X, *Y;
  if (match(FNegOp, m_FMul(m_Value(X), m_Value(Y)))) {
    // Push into RHS which is more likely to simplify (const or another fneg).
    // FIXME: It would be better to invert the transform.
    return cast<Instruction>(Builder.CreateFMulFMF(
        X, Builder.CreateFNegFMF(Y, &FMFSource), &FMFSource));
  }

InstCombine usually has good reasons for canonicalizing certain IR patterns, so we need to seriously reconsider if we want to change the canonical form. InstCombine affects all targets and it could be the case that some other backends have patterns that match fmul (fneg x, y), in which case we don’t want disturb them. However for RISC-V we know what our patterns for instruction selection are and what form we want our incoming IR to be in. So a much better place to handle this in is in RISCVISelLowering.cpp, which lets us massage it into shape at the SelectionDAG level, in a way that’s localized to just our target. “Un-canonicalizing” the IR is a common task that backends end up performing, and this is what the resulting patch ended up looking like:

--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -20248,6 +20248,17 @@ SDValue RISCVTargetLowering::PerformDAGCombine(SDNode *N,
       return V;
     break;
   case ISD::FMUL: {
+    using namespace SDPatternMatch;
+    SDLoc DL(N);
+    EVT VT = N->getValueType(0);
+    SDValue X, Y;
+    // InstCombine canonicalizes fneg (fmul x, y) -> fmul x, (fneg y), see
+    // hoistFNegAboveFMulFDiv.
+    // Undo this and sink the fneg so we match more fmsub/fnmadd patterns.
+    if (sd_match(N, m_FMul(m_Value(X), m_OneUse(m_FNeg(m_Value(Y))))))
+      return DAG.getNode(ISD::FNEG, DL, VT,
+                         DAG.getNode(ISD::FMUL, DL, VT, X, Y));
+

And if we rebuild our benchmark after applying it, we can see we the fmsub.ds getting matched, saving a couple of instructions:

@@ -983,18 +983,15 @@
        fld     ft2, 48(a5)
        fld     ft3, 64(a5)
        fld     ft4, 72(a5)
-       fneg.d  fa0, fa0
-       fneg.d  ft0, ft0
-       fneg.d  ft2, ft2
        fmul.d  fa3, ft5, fa3
        fmul.d  fa0, fa3, fa0
        fmul.d  ft0, fa3, ft0
        fmul.d  fa3, fa3, ft2
        fld     ft2, 0(s1)
        fmul.d  fa4, ft5, fa4
-       fmadd.d fa2, fa4, fa2, fa0
-       fmadd.d ft6, fa4, ft6, ft0
-       fmadd.d fa4, fa4, ft1, fa3
+       fmsub.d fa2, fa4, fa2, fa0
+       fmsub.d ft6, fa4, ft6, ft0
+       fmsub.d fa4, fa4, ft1, fa3

All in all this ended up giving a 1.77% improvement in instruction count for the 508.namd_r benchmark. It’s still not nearly as fast as GCC, but we’re a little bit closer than before we started.

What’s next?

Hopefully this has given you an overview of how to identify opportunities for optimization in LLVM, and what a typical fix might look like. The analysis is really the most important part, but if you don’t feel like setting up an LNT instance yourself locally Igalia runs one at cc-perf.igalia.com³. We run llvm-test-suite and SPEC CPU 2017 nightly built with Clang and GCC on a small set of RISC-V hardware⁴, but hopefully to be expanded in future. Feel free to use it to investigate some of the differences between Clang and GCC yourself, and maybe you’ll find some inspiration for optimizations.

In the next post in this series I’ll talk about a performance improvement that recently landed related to cost modelling.

Compiled with -march=rva22u64_v -O3 -flto, running the train dataset on a 16GB Banana Pi BPI-F3 (SpacemiT X60), with GCC and Clang from ToT on 2025-11-25. ↩
LLD in this case, configurable through CMake with -DCMAKE_LINKER_TYPE=LLD. ↩
The LLVM foundation is also in the process of rebooting its canonical public server, which should hopefully be up and running in the coming months. ↩
Currently it consists of a few Banana Pi BPI-F3s and some HiFive Premier P550s, the latter of which were generously donated by RISC-V International. ↩

Igalia WebKit Team: WebKit Igalia Periodical #50

2025-12-08T20:26:32+00:00

Update on what happened in WebKit in the week from December 1 to December 8.

In this edition of the periodical we have further advancements on the Temporal implementation, support for Vivante super-tiled format, and an adaptation of the DMA-BUF formats code to the Android port.

Cross-Port 🐱

JavaScriptCore 🐟

The built-in JavaScript/ECMAScript engine for WebKit, also known as JSC or SquirrelFish.

Implemented the toString, toJSON, and toLocaleString methods for PlainYearMonth objects in JavaScriptCore's implementation of Temporal.

Graphics 🖼️

BitmapTexture and TextureMapper were prepared to handle textures where the logical size (e.g. 100×100) differs from the allocated size (e.g. 128×128) due to alignment requirements. This allowed to add support for using memory-mapped GPU buffers in the Vivante super-tiled format available on i.MX platforms. Set WEBKIT_SKIA_USE_VIVANTE_SUPER_TILED_TILE_TEXTURES=1 to activate at runtime.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

The WPEBufferDMABufFormats class has been renamed to WPEBufferFormats, as it can be used in situations where mechanisms other than DMA-BUF may be used for buffer sharing—on Android targets AHardwareBuffer is used instead, for example. The naming change involved also WPEBufferFormatsBuilder (renamed from WPEBufferDMABufFormatsBuilder), and methods and signals in other classes that use these types. Other than the renames, there is no change in functionality.

That’s all for this week!

Enrique Ocaña: Meow: Process log text files as if you could make cat speak

2025-12-05T11:16:07+00:00

Some years ago I had mentioned some command line tools I used to analyze and find useful information on GStreamer logs. I’ve been using them consistently along all these years, but some weeks ago I thought about unifying them in a single tool that could provide more flexibility in the mid term, and also as an excuse to unrust my Rust knowledge a bit. That’s how I wrote Meow, a tool to make cat speak (that is, to provide meaningful information).

The idea is that you can cat a file through meow and apply the filters, like this:

cat /tmp/log.txt | meow appsinknewsample n:V0 n:video ht: \ ft:-0:00:21.466607596 's:#([A-za-z][A-Za-z]*/)*#'

which means “select those lines that contain appsinknewsample (with case insensitive matching), but don’t contain V0 nor video (that is, by exclusion, only that contain audio, probably because we’ve analyzed both and realized that we should focus on audio for our specific problem), highlight the different thread ids, only show those lines with timestamp lower than 21.46 sec, and change strings like Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp to become just AppendPipeline.cpp“, to get an output as shown in this terminal screenshot:

Cool, isn’t it? After all, I’m convinced that the answer to any GStreamer bug is always hidden in the logs (or will be, as soon as I add “just a couple of log lines more, bro” ).

Currently, meow supports this set of manipulation commands:

Word filter and highlighting by regular expression (fc:REGEX, or just REGEX): Every expression will highlight its matched words in a different color.
Filtering without highlighting (fn:REGEX): Same as fc:, but without highlighting the matched string. This is useful for those times when you want to match lines that have two expressions (E1, E2) but the highlighting would pollute the line too much. In those case you can use a regex such as E1.*E2 and then highlight the subexpressions manually later with an h: rule.
Negative filter (n:REGEX): Selects only the lines that don’t match the regex filter. No highlighting.
Highlight with no filter (h:REGEX): Doesn’t discard any line, just highlights the specified regex.
Substitution (s:/REGEX/REPLACE): Replaces one pattern for another. Any other delimiter character can be used instead of /, it that’s more convenient to the user (for instance, using # when dealing with expressions to manipulate paths).
Time filter (ft:TIME-TIME): Assuming the lines start with a GStreamer log timestamp, this filter selects only the lines between the target start and end time. Any of the time arguments (or both) can be omitted, but the - delimiter must be present. Specifying multiple time filters will generate matches that fit on any of the time ranges, but overlapping ranges can trigger undefined behaviour.
Highlight threads (ht:): Assuming a GStreamer log, where the thread id appears as the third word in the line, highlights each thread in a different color.

The REGEX pattern is a regular expression. All the matches are case insensitive. When used for substitutions, capture groups can be defined as (?CAPTURE_NAMEREGEX).

The REPLACEment string is the text that the REGEX will be replaced by when doing substitutions. Text captured by a named capture group can be referred to by ${CAPTURE_NAME}.

The TIME pattern can be any sequence of numbers, : or . . Typically, it will be a GStreamer timestamp (eg: 0:01:10.881123150), but it can actually be any other numerical sequence. Times are compared lexicographically, so it’s important that all of them have the same string length.

The filtering algorithm has a custom set of priorities for operations, so that they get executed in an intuitive order. For instance, a sequence of filter matching expressions (fc:, fn:) will have the same priority (that is, any of them will let a text line pass if it matches, not forbidding any of the lines already allowed by sibling expressions), while a negative filter will only be applied on the results left by the sequence of filters before it. Substitutions will be applied at their specific position (not before or after), and will therefore modify the line in a way that can alter the matching of subsequent filters. In general, the user doesn’t have to worry about any of this, because the rules are designed to generate the result that you would expect.

Now some practical examples:

Example 1: Select lines with the word “one”, or the word “orange”, or a number, highlighting each pattern in a different color except the number, which will have no color:

$ cat file.txt | meow one fc:orange 'fn:[0-9][0-9]*' 000 one small orange 005 one big orange

Example 2: Assuming a pictures filename listing, select filenames not ending in “jpg” nor in “jpeg”, and rename the filename to “.bak”, preserving the extension at the end:

$ cat list.txt | meow 'n:jpe?g' \
's:#^(?<f>[^.]*)(?<e>[.].*)$#${f}.bak${e}' train.bak.png sunset.bak.gif

Example 3: Only print the log lines with times between 0:00:24.787450146 and 0:00:24.790741865 or those at 0:00:30.492576587 or after, and highlight every thread in a different color:

$ cat log.txt | meow ft:0:00:24.787450146-0:00:24.790741865 \ ft:0:00:30.492576587- ht: 0:00:24.787450146 739 0x1ee2320 DEBUG … 0:00:24.790382735 739 0x1f01598 INFO … 0:00:24.790741865 739 0x1ee2320 DEBUG … 0:00:30.492576587 739 0x1f01598 DEBUG … 0:00:31.938743646 739 0x1f01598 ERROR …

This is only the begining. I have great ideas for this new tool (as time allows), such as support for parenthesis (so the expressions can be grouped), or call stack indentation on logs generated by tracers, in a similar way to what Alicia’s gst-log-indent-tracers tool does. I might also predefine some common expressions to use in regular expressions, such as the ones to match paths (so that the user doesn’t have to think about them and reinvent the wheel every time). Anyway, these are only ideas. Only time and hyperfocus slots will tell…

By now, you can find the source code on my github. Meow!

Brian Kardell: Standards Queues

2025-12-04T05:00:00+00:00

Standards Queues

The hardest part of web standards isn’t even the technology — it’s the queues. And that’s the real problem I keep coming back to.

Pools, Queues, and Bottlenecks

As programmers, we’re familiar with these kinds of problems: if things enter faster than they leave, they back up. We often need to prioritize among the backlog. The standards process is like several of those queues stacked together.

Ideas enter the system far faster than they leave — and they can come from anywhere. But to progress, you need implementers. They are finite, already busy, and often advancing their own priorities. On top of that, every proposal competes for wide review in privacy, security, architecture, accessibility, and internationalization. Each of those specialties is even more finite, even more busy, and even more backed up.

So an idea lands in hundreds — even thousands — of inboxes, waiting for attention. We might not even notice it as it whips past among all the others. Even if we do, it might just get starred in email or left open in a tab for “later.” Sometimes that idea is a book, or an explainer, or suddenly has 20 replies. Instead of needing five minutes to read and consider, it becomes intimidating.

At some point it just sits. It might wait weeks, months, or even years before someone comments. Why? Because everyone has jobs with other tasks. The queues are full.

And the longer it sits, the more things change around it. The more it unloads from memory. The more intimidating it becomes to return to. It has to get through a whole lot of asynchronous back-and-forth between implementers, spec writers, and test writers before reaching baseline usability.

Along the way, if coordination isn’t strong (and historically it hasn’t been), once work is invested it’s hard to throw away. It’s hard to propose breaking changes or add stop energy.

Real Impacts

This is why something like :focus-visible can take seven years. Realistically, it required only a few days of effective discussion. The tests and development weren’t that hard.

The hard part was agreeing on what it should do and which tests it should pass. Most of that difficulty came from the fact that you couldn’t get everyone to sit down and focus concurrently. Implementations — and thus real focus — were years apart.

Checking Fitness

Getting “unstuck” isn’t just about moving something forward. One major appeal of the standards process is wide review, but for this to work we need ways for things to fail early enough to shift efforts.

Sometimes failure happens after months or even years. That’s frustrating and demoralizing. It’s like standing in a long DMV line, inching forward, only to discover at the end that you’re in the wrong building.

All of this is made worse by the fact that queues keep getting fuller.

Things that help

Interop

The Interop project illustrates the very end of the process, and in many ways the simplest.

Without intervention, each implementer historically built their own priority queue from all possible shortcomings of their browser engine. There’s a huge pool of things to choose from. I’ve written before about how WPT and the dashboard aren’t the best way to view this, but there are currently almost 23k subtests that fail in every browser (or almost 11k that fail and aren’t marked tentative).

Interop coordinates efforts to choose an achievable set of things from this gigantic pool that meet strict criteria: all that’s left is implementation work. It’s been hugely successful because it ensures delivery. It also helps us deliver early when people are excited about common priorities. In those cases, the impact is huge — we can go from mostly words to three interoperable implementations in one year. Amazing.

Still, every year a huge number of things remain in the pool that we can’t afford to take up. The pool keeps growing.

Joint Meetings

The WHATWG has started holding regular joint meetings with groups like OpenUI and CSSWG. This is valuable because it allows the right people to agree on an agenda and discuss directly, rather than leaving issues to sit unnoticed or requiring endless pings for attention.

W3C's TPAC is an annual event with five days of meetings (both W3C and WHATWG), many of them joint with wide-review specialists. These are dedicated times to get a lot of people in the same rooms for extended periods. The availability for hallway conversations also matters a lot: You can corner people there in ways that are much harder when everyone is remote. More progress happens at TPAC than in half the rest of the year combined.

Timely coordination — and investment to make it plausible — is still the single biggest problem we face in standards. I'd love to see us find ways to improve that.

Igalia WebKit Team: WebKit Igalia Periodical #49

2025-12-02T14:15:36+00:00

Update on what happened in WebKit in the week from November 24 to December 1.

The main highlights for this week are the completion of `PlainMonthDay` in Temporal, moving networking access for GstWebRTC to the WebProcess, and Xbox Cloud Gaming now working in the GTK and WPE ports.

Cross-Port 🐱

Multimedia 🎥

GStreamer-based multimedia support for WebKit, including (but not limited to) playback, capture, WebAudio, WebCodecs, and WebRTC.

Xbox Cloud Gaming is now usable in WebKitGTK and WPE with the GstWebRTC backend, we had to fix non-spec compliant ICE candidates handling and add a WebRTC quirk forcing max-bundle in PeerConnections to make it work. Happy cloud gaming!

Support for remote inbound RTP statistics was improved in 303671@main, we now properly report framesPerSecond and totalDecodeTime metrics, those fields are used in the Xbox Cloud Gaming service to show live stats about the connection and video decoder performance in an overlay.

The GstWebRTC backend now relies on librice for its ICE. The Sans-IO architecture of librice allows us to keep the WebProcess sandboxed and to route WebRTC-related UDP and (eventually) TCP packets using the NetworkProcess. This work landed in 303623@main. The GNOME SDK should also soon ship librice.

Support for seeking in looping videos was fixed in 303539@main.

JavaScriptCore 🐟

The built-in JavaScript/ECMAScript engine for WebKit, also known as JSC or SquirrelFish.

Implemented the valueOf and toPlainDate for PlainMonthDay objects. This completes the implementation of Temporal PlainMonthDay objects in JSC!

WebKitGTK 🖥️

The GTK port has gained support for interpreting touch input as pointer events. This matches the behaviour of other browsers by following the corresponding specifications.

Fixed an issue that prevented WPE from processing further input events after receiving a secondary mouse button press.

Fixed an issue that caused right mouse button clicks to prevent processing of further pointer events.

WPE Platform API 🧩

New, modern platform API that supersedes usage of libwpe and WPE backends.

We landed a patch to add a new signal in WPEDisplay to notify when the connection to the native display has been lost.

Infrastructure 🏗️

Modernized the CMake modules used to find libtasn1, libsecret, libxkbcommon, libhyphen, and Enchant libraries.

Note that this work removed the support for building against Enchant 1.x, and only version 2 will be supported. The first stable release to require Enchant 2.x will be 2.52.0 due in March 2026. Major Linux and BSD distributions have included Enchant 2 packages for years, and therefore this change is not expected to cause any trouble. The Enchant library is used by the GTK port for spell checking.

Community & Events 🤝

We have published an article detailing our work making MathML interoperable across browser engines! It has live demonstrations and feature tables with our progress on WebKit support.

We have published new blogs post highlighting the most important changes in both WPE WebKit and WebKitGTK 2.50. Enjoy!

That’s all for this week!

Alex Bradbury: QEMU-based instruction execution counting

2025-12-02T12:00:00+00:00

Although analysing performance by way of instruction counting has obvious limitations, it can be helpful (especially when combined with appropriate analysis scripts) to get rapid feedback on the impact of code generation changes or to explore hypotheses about why code from one compiler might be performing differently from another - for instance, by looking at instruction mix in the most executed translation blocks. In this post we'll look at how to capture the necessary data to perform such an analysis using a QEMU plugin. Future posts will give details of the analysis scripts I've used, and walk through an example or two of putting them to use.

Modifying QEMU

Over the past few years, QEMU's plugin API has developed a fair bit. QEMU includes several plugins, and hotblocks provides almost what we want but doesn't allow configurability of the number of blocks it will print information on. I submitted a small patch series (and [email protected]/">submitted it a second time addressing this and other minor issues found along the way. The series has now been [email protected]/">accepted by the maintainer.

To build QEMU with this patch:

git clone https://github.com/qemu/qemu && cd qemu
git checkout v10.1.2
cat - <<'EOF' > hotblocks.patch
index 98404b6885..8ecf033997 100644
--- a/contrib/plugins/hotblocks.c
+++ b/contrib/plugins/hotblocks.c
@@ -73,28 +73,29 @@ static void exec_count_free(gpointer key, gpointer value, gpointer user_data)
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
     g_autoptr(GString) report = g_string_new("collected ");
-    GList *counts, *it;
+    GList *counts, *sorted_counts, *it;
     int i;

     g_string_append_printf(report, "%d entries in the hash table\n",
                            g_hash_table_size(hotblocks));
     counts = g_hash_table_get_values(hotblocks);
-    it = g_list_sort_with_data(counts, cmp_exec_count, NULL);
+    sorted_counts = g_list_sort_with_data(counts, cmp_exec_count, NULL);

-    if (it) {
+    if (sorted_counts) {
         g_string_append_printf(report, "pc, tcount, icount, ecount\n");

-        for (i = 0; i < limit && it->next; i++, it = it->next) {
+        for (i = 0, it = sorted_counts; (limit == 0 || i < limit) && it;
+             i++, it = it->next) {
             ExecCount *rec = (ExecCount *) it->data;
             g_string_append_printf(
-                report, "0x%016"PRIx64", %d, %ld, %"PRId64"\n",
+                report, "0x%016"PRIx64", %d, %ld, %"PRIu64"\n",
                 rec->start_addr, rec->trans_count,
                 rec->insns,
                 qemu_plugin_u64_sum(
                     qemu_plugin_scoreboard_u64(rec->exec_count)));
         }

-        g_list_free(it);
+        g_list_free(sorted_counts);
     }

     qemu_plugin_outs(report->str);
@@ -170,6 +171,13 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
                 fprintf(stderr, "boolean argument parsing failed: %s\n", opt);
                 return -1;
             }
+        } else if (g_strcmp0(tokens[0], "limit") == 0) {
+            char *endptr = NULL;
+            limit = g_ascii_strtoull(tokens[1], &endptr, 10);
+            if (endptr == tokens[1] || *endptr != '\0') {
+                fprintf(stderr, "unsigned integer parsing failed: %s\n", opt);
+                return -1;
+            }
         } else {
             fprintf(stderr, "option parsing failed: %s\n", opt);
             return -1;
diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
index 4a7d1f4178..e8793b0f9c 100644
--- a/docs/about/emulation.rst
+++ b/docs/about/emulation.rst
@@ -463,6 +463,18 @@ Example::
   0x000000004002b0, 1, 4, 66087
   ...

+Behaviour can be tweaked with the following arguments:
+
+.. list-table:: Hot Blocks plugin arguments
+  :widths: 20 80
+  :header-rows: 1
+
+  * - Option
+    - Description
+  * - inline=true|false
+    - Use faster inline addition of a single counter.
+  * - limit=N
+    - The number of blocks to be printed. (Default: N = 20, use 0 for no limit).

 Hot Pages
 .........
EOF
patch -p1 < hotblocks.patch
./configure --prefix=$(pwd)/inst --target-list="riscv32-linux-user riscv64-linux-user"
make -j$(nproc)
cd ..

Using this plugin to capture statistics from running a binary under qemu-user

Assuming you have an appropriate sysroot, you can run a binary and have the execution information emitted to stderr by doing something like:

QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain \
  my_rv64_binary

This produces output like:

collected 2229 entries in the hash table
pc, tcount, icount, ecount
0x00007fffee7012ba, 1, 1, 3737
0x00007fffee7012be, 1, 3, 3737
0x00007ffff741e738, 1, 23, 1074
0x00007fffee71bb38, 1, 5, 884
0x00007ffff741bb2e, 1, 11, 662
...

This listing indicates the address of the translation block, the number of times it's been translated, the number of instructions it contains, and the number of times it was executed. Note that a translation block is not the same as a basic block in the compiler. A translation block can span multiple basic blocks in the case of fallthrough, and this can also mean an instruction may show up in multiple translation blocks.

At least for my use cases, I need something a bit more involved than this. In order to add collection of these statistics to an existing benchmark harness I need a wrapper script that transparently collects these statistics to a file. It's also helpful to capture the runtime address of executable mappings for loaded libraries, allowing translation blocks to be attributed easily to either the binary itself or libc, libm etc. We have gdb connect to QEMU's gdbserver in order to dump those mappings. Do ensure you're using a recent version of QEMU (the version suggested in the patch application instructions is definitely good) for this as I wasted quite some time running into a bug with file descriptor numbers that caused odd breakage.

This qemu-forwarder.sh script will capture the plugin's output in a .qemu_out file and the mappings in a .map file, both of which can be later consumed by a detailed analysis script.

#!/bin/sh
QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
QEMU="$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain"

SUFFIX=""
if [ -e "$1.qemu_out" ]; then
  NUM=1
  while [ -e "$1.qemu_out.$NUM" ]; do
    NUM=$((NUM + 1))
  done
  SUFFIX=".$NUM"
fi

GDB_SOCK=$(mktemp -u)
setarch $(uname -m) -R $QEMU -g $GDB_SOCK -D $1.qemu_out$SUFFIX "$@" &
QEMU_PID=$!

RETRY_COUNT=0
while ! [ -e "$GDB_SOCK" ]; do
  RETRY_COUNT=$((RETRY_COUNT + 1))
  if [ $RETRY_COUNT -eq 10 ]; then
    echo "Timed out waiting for gdb socket to be created"
    exit 1
  fi
  sleep 0.1
  if ! kill -0 $QEMU_PID 2>/dev/null; then
    echo "QEMU process died before gdb socket was created"
    wait $QEMU_PID
    exit $?
  fi
done

gdb -batch \
  -ex "set pagination off" \
  -ex "target remote $GDB_SOCK" \
  -ex "break main" \
  -ex "continue" \
  -ex "set logging file $1.map$SUFFIX" \
  -ex "set logging enabled on" \
  -ex "info proc mappings" \
  -ex "detach" > /dev/null 2>&1
wait $QEMU_PID

The above will work under LLVM's lit, though you will need to use a recent enough version that doesn't strip HOME from the environment (or else edit the script accordingly). It also produces output in sequentially numbered files, again motivated by the desire to run under this script from lit as used by llvm-test-suite's SPEC configuration which can involve multiple invocations of the same binary for a given benchmark (e.g. 500.perlbench_r).

Analysing the output

A follow-up post will introduce the scripting I've built around this.

Recording and analysing results from running SPEC

Assuming you have qemu-forwarder.sh, in your llvm-test-suite directory:

CONF=clang-head-test
CLANG_BIN_DIR=$HOME/llvm-project/build/release/bin
CFLAGS="-march=rv64gc_zba_zbb_zbs"
cat - <<EOF > $CONF.cmake
set(CMAKE_SYSTEM_NAME Linux)

set(CMAKE_SYSROOT $HOME/rvsysroot)

set(CMAKE_C_COMPILER $CLANG_BIN_DIR/clang)
set(CMAKE_CXX_COMPILER $CLANG_BIN_DIR/clang++)

set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_C_FLAGS_INIT "$CFLAGS")
set(CMAKE_CXX_FLAGS_INIT "$CFLAGS")

set(CMAKE_LINKER_TYPE LLD)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
EOF
cmake -G Ninja \
  -B build.$CONF \
  --toolchain=$CONF.cmake \
  -DTEST_SUITE_SPEC2017_ROOT=~/cpu2017 \
  -DTEST_SUITE_SUBDIRS=External/SPEC \
  -DTEST_SUITE_COLLECT_CODE_SIZE=OFF \
  -DTEST_SUITE_COLLECT_COMPILE_TIME=OFF \
  -DTEST_SUITE_USER_MODE_EMULATION=ON \
  -DTEST_SUITE_RUN_UNDER=$(pwd)/qemu-forwarder.sh
cmake --build build.$CONF
$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand' build.$CONF

The 526.blender_r test takes twice as long as the others, so you may wish to skip it by instead executing something like:

$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand|blender' build.$CONF

If you want to re-run tests you must delete the previous .qemu_out and .map files, which can be done with:

[ -n "build.$CONF" ] && find "build.$CONF" -type f -name "*.qemu_out*" -exec sh -c '
    for q_file do
        base_path="${q_file%.qemu_out*}"
        rm -f "$q_file" "${base_path}.map"*
    done
' sh {} +

In order to compare two SPEC builds, you can use something like the following hacky script. Using the captured translation block execution data to generate a plain executed instruction count is overkill as the example tests/tcg/plugin/insn.c can easily dump for this for you directly. But by collecting the data upfront, you can easily dive right into a more detailed analysis when you see a surprising difference in executed instruction counts without rerunning the binary.

#!/usr/bin/env python3

from pathlib import Path
from collections import defaultdict
import sys

def collect_totals(root_dir):
    totals = defaultdict(int)
    root_path = Path(root_dir)/"External"

    for file_path in root_path.rglob("*.qemu_out*"):
        benchmark_name = file_path.parts[4]

        try:
            with file_path.open("r") as f:
                file_total = 0
                for line in f:
                    parts = line.strip().split(',')
                    # Only sum lines that match the expected format.
                    if len(parts) == 4 and parts[2].strip().isdigit():
                        # icount * ecount.
                        file_total += int(parts[2]) * int(parts[3])
                totals[benchmark_name] += file_total
        except Exception as e:
            print(f"Error reading {file_path}: {e}")

    return totals

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: spec-compare-helper <dir_a> <dir_b>")
        sys.exit(1)

    dir_a, dir_b = sys.argv[1], sys.argv[2]
    totals_a = collect_totals(dir_a)
    totals_b = collect_totals(dir_b)

    benchmarks = sorted(set(totals_a.keys()) | set(totals_b.keys()))

    print(f"{'Benchmark':<20} {'DirA':>15} {'DirB':>15} {'Diff (%)':>10}")
    print("=" * 60)

    for benchmark in benchmarks:
        val_a = totals_a.get(benchmark, 0)
        val_b = totals_b.get(benchmark, 0)
        diff_pct = ((val_b - val_a) / val_a * 100) if val_a else float("inf")

        print(f"{benchmark:<20} {val_a:>15} {val_b:>15} {diff_pct:>9.2f}%")

Which produces output looking something like this:

Benchmark                       DirA            DirB   Diff (%)
============================================================
500.perlbench_r         180245097594    182078714777      1.02%
502.gcc_r               220874510659    219647717585     -0.56%
505.mcf_r               131589945456    134271153130      2.04%
508.namd_r              220648061019    216682202888     -1.80%
510.parest_r            291341820355    291844973715      0.17%
511.povray_r             31911866906     31103201809     -2.53%
519.lbm_r                94166321698     86910581403     -7.71%
520.omnetpp_r           138002605692    137676301622     -0.24%
523.xalancbmk_r         283566182007    284735075518      0.41%
525.x264_r              380165035845    379862173371     -0.08%
526.blender_r           660528270138    659361380750     -0.18%
531.deepsjeng_r         355058534962    349621355155     -1.53%
538.imagick_r           238573643488    238560676372     -0.01%
541.leela_r             421886351310    405423320484     -3.90%
544.nab_r               415595728542    391443973852     -5.81%
557.xz_r                132548718317    130229753780     -1.75%

It's worth highlighting that as we're running this under user-mode emulation, the dynamic instruction count naturally never counts any instructions on the kernel side that you would see if profiling a real system.

Article changelog

2025-12-15: Note that the qemu patches have now been accepted in the maintainer's tree.
2025-12-02: Initial publication date.

Manuel Rego

2025-12-02T00:00:00+00:00

You can now easily customize find-in-page with the new ::search-text pseudo-element, that is shipping in Chromium 144.0.7547. 🚀

Find more details on the blog post by Stephen Chenney. Thanks to Bloomberg for sponsoring this work.

Alex Bradbury: Minipost: Olmo 3 training cost

2025-12-01T12:00:00+00:00

Recently I jotted down some notes on LLM inference vs training costs for DeepSeek and I wanted to add on an additional datapoint for training cost based on the recently released Olmo3 models from the Allen Institute for AI ("Ai2"). The model family has 7B and 32B parameter models, with 'Think' variants available for 7B and 32B but so far only a 7B 'Instruct' non-reasoning version (but watch this space). What's particularly interesting about the Olmo models to me is that beyond providing open weights, the training scripts and datasets are openly available as well.

Going by the reported benchmarks at least it's competitive with less open models at a similar size, and importantly they've increased the supported context length from the rather limiting 4k tokens supported by the Olmo 2 series to a much more usable 64k tokens. Given the relatively small size these models are less capable than relatively chunky models like DeepSeek R1/V3.x or Kimi K2, but I've been impressed by the capability of 32B dense models for basic queries, and from my non-scientific testing both the 32B and 7B Olmo3 variants seem to do a reasonable job of summarising things like discussion threads. You can experiment yourself at playground.allenai.org.

Energy required for training Olmo 3

One of the neat things about this level of openness is that it should act as a floor in terms of performance for future models of this size class assuming they're appropriately funded and don't take too many risks chasing novelty. Rerunning the training process with an updated dataset and some minor tweaks is something you could imagine doing on some regular cadence, ideally as a shared endeavour. Imagining this effort in the future, how much energy is required? The initial version of the detailed Olmo 3 technical report unfortunately has little to say on this. We can get a back of the envelope figure in terms of GPU hours for pre-training based on the reported 7700 tokens per second per GPU for the 7B base model and 1900 tokens per second for the 32B base model and the ~6T token dataset. But even better than that, we can just ask the Ai2 folks (sometimes the internet really does work wonderfully!). After asking on their public Discord I was rapidly furnished with this helpful answer:

For some detailed numbers, we measured power consumption throughout training, along with total GPU hours. We used ~234k H100 hours to pretrain the 7B, and ~1.05m H100 hours to pretrain the 32B. 1900 TPS is generally what our trainer is capable of, but with restarts, evaluations, checkpointing, and occasional network issues, the 32B took 1.05m hours. We measured an average power consumption of ~621W while pretraining the 7B and ~649W while pretraining the 32B, and this means that our GPUs consumed ~146MWh for the 7B and ~681MWh for the 32B. We'll include more detailed GPU hour information in a future version of the paper, including for post-training!
Ai2 Olmo 3 team on their Discord.

So that's 0.681 GWh in GPU power draw for pretraining the 32B model and 0.146 GWh in GPU power draw for pretraining the 7B model. As noted in the quote, this is inclusive of restarts, checkpointing etc. But perhaps won't include previous early stage experimentation. I look forward to an updated technical report with full details, but pretraining should cover the bulk of the compute requirements (as a reference point, today's DeepSeek V3.2 paper found it notable that the post-training compute budget exceeded 10% of the pretraining cost).

The 0.681 GWh figure doesn't account for full system power and cooling cost. I'd love to be corrected, but I believe a 1.5x-2x multiplier would be an assumption towards the upper end. But for the sake of this yardstick comparison let's look at a few comparisons based on the reported number:

0.681 GWh of electricity would cost about £180k at UK residential rates (capped at 26.35p per kWh currently). Substantially less in the USA.
A larger leisure centre with a pool consumes ~2.5 GWh of energy per year. I don't know if the idea of a "leisure centre" translates outside of the UK, but basically it's a swimming pool plus gym, squash/tennis courts etc.
- The linked page claims ~2 GWh of energy in gas and 0.5 GWh in electricity. For the gas, to compare like with like you'd need to consider the source of energy for the electricity used for Olmo training.
0.681 GWh is ~0.11% of LHC's annual 600 GWh energy consumption or ~0.05% of CERN's annual consumption.
We can estimate a Boeing 787-9 flying from London Heathrow to SFO consumes jet fuel containing ~0.58 GWh of energy.
- Calculated with 8638km distance, 5.62kg fuel/km (taking the most economic 787-9 long haul figure from this table on Wikipedia and 11.95kWh/kg specific energy of jet fuel).
- This is a yardstick rather than a direct comparison. A direct comparison to the GWh of electricity used for the GPU compute of the LLM would depend on the source of the electricity. If it was e.g. gas rather than solar/hydro/wind then you'd want to compare the number of GWh consumed to create that electricity which would of course be higher.
- As a further point of reference, FlightAware indicates 5 separate direct LHR to SFO flights scheduled per day.

More efficient LLM training

We can hope for new breakthroughs, more efficient hardware, better datasets and so on. But here is some work I noticed in the area. Fair warning: this isn't my field, and we have to recognise applying a research result to a production training run is sure to have challenges even if the research suggests the trade-offs are worthwhile. So consider this vague gesticulating about seemingly interesting work that is going on and find someone who knows what they're talking about to confirm the degree to which it is interesting/viable.

Mixture of Experts (MoE) models are substantially cheaper to train which is one reason the industry has moved in that direction. The next Ai2 Olmo model is expected to be MoE. The Qwen blog has a graph comparing the relative training cost in GPU hours of the dense Qwen3-32B vs Qwen3-30B-A3b vs Qwen3-Next-80B-A3B, where the latter makes further architectural changes, reporting a 10.7x reduction. ~2.5x of that is going to come from the reduced corpus size (15T tokens down from 36T), but that still leaves plenty of improvement from other factors.
Maybe it will be shown viable to train in lower precision such as MXFP8 or even NVFP4, which would allow much more throughput for a similar energy budget. Nvidia have worked to demonstrate this can be effective for both formats (see also this work from MIT).
Also from Nvidia, Nemotron Elastic showed a model architecture that allows deriving smaller models without doing a separate pre-training runs.

Finally, the cheapest way to train an LLM from scratch is...to find a way to avoid the need to. For models like Olmo 3 that release the base model and checkpoints, people can apply their own post-training or perform additional pre-training.

Bonus comparison point: Apertus

Apertus is a Swiss project to produce an open LLM, with 70B and 8B models released so far. Their full tech report notes the following "Once a production environment has been set up, we estimate that the model can be realistically trained in approximately 90 days on 4096 GPUs, accounting for overheads. If we assume 560 W power usage per Grace-Hopper module in this period, below the set power limit of 660 W, we can estimate 5 GWh power usage for the compute of the pretraining run."

Article changelog

2025-12-04: Add link to "Four Over Six" NVFP4 training paper.
2025-12-02: Added clarifying note about energy via gas in the leisure centre comparison.
2025-12-01: Initial publication date.

Alex Bradbury: Minipost: Benchmarking the Hetzner AX102 vs CCX53

2025-11-30T12:00:00+00:00

I recently had reason to do a quick comparison of the performance of the Hetzner AX102 dedicated server and the high-end 'dedicated' CCX53 VPS on Hetzner Cloud and thought I may as well write up the results for posterity. I'm incapable of starting a post without some kind of disclaimer so here comes the one for this post: naturally the two products have major differences in terms of flexibility (spin-up/down at will, vs pay a small setup fee and endure a wait time depending on hardware availability). So depending on your use case, your requirements with respect to that flexibility may override any cost differential.

Specs

All costs are exclusive of VAT, assuming the lowest cost data center location, and inclusive of IPv4 address.

AX102:

16 core Ryzen 9 7950X3D (32 threads)
128GB DDR5 RAM
2 x 1.92TB NVMe
104 EUR/month, 39 EUR one-off setup fee.

CCX53

Unknown AMD CPU exposing 32vCPU (physical cores? threads?)
128GB RAM
600GB NVMe
192.49 EUR/month maximum charge. 0.3085 EUR per hour (if you keep the same VPS active over the month it won't exceed the monthly price cap, so you effectively get a small discount on the per-hour cost).

Benchmark

Building Clang+LLVM+LLD, everyone's favourite workload! Both systems are running an up to date Arch Linux (more details on setting this up on the CCX53 in the appendix below) with clang 21.1.6. The dedicated machine has the advantage of RAID 0 across the two SSDs, but also has encrypted rootfs configured. I didn't bother to set that up for the CCX53 VPS.

sudo pacman -Syu --needed clang lld cmake ninja wget
LLVM_VER=21.1.6
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VER}/llvm-project-${LLVM_VER}.src.tar.xz
tar -xvf llvm-project-${LLVM_VER}.src.tar.xz
cd llvm-project-${LLVM_VER}.src

cmake -G Ninja \
  -DLLVM_ENABLE_PROJECTS='clang;lld' \
  -DLLVM_TARGETS_TO_BUILD="all" \
  -DLLVM_CCACHE_BUILD=OFF \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DLLVM_ENABLE_LLD=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -S llvm \
  -B build
time cmake --build build

printf "### Version info ###\n"
clang --version | head -n 1

On both machines, ninja shows 5575 build steps.

Results:

AX102
- 10m27s (627s)
CCX53
- 14m11s (851s, about 1.36x the AX102)

Running the clang and LLVM tests with ./build/bin/llvm-lit -s --order=lexical llvm/test clang/test (which shows 9402 tests) gives:

AX102
- 3m39s (219s)
CCX53
- 4m28s (268s, about 1.24x the AX102)

I ran these multiple times, and in the case of the CCX53 across two different VMs in different regions and saw only a few percentage points variance.

Focusing on the results for build clang/llvm/lld, let's figure out the cost for 1000 from-scratch builds. Not so much as it's a representative workload, but because it gives an easy to compare metric that captures both the difference in price and in performance. So calculating time_per_build_in_hours * 1000 * cost_per_hour:

AX102
- (626.6 / 3600) * 1000 * (104/720) = 25.14 EUR
- Or if you include the setup fee and assume it's amortised over 12 months:
  - (626.6/3600) * 1000 * ((104 + (39/12))/720) = 25.93 EUR
CCX53
- (850.6 / 3600) * 1000 * (192.49/720) = 63.17 EUR
- Or using the 0.3085 EUR/hr price which you would pay if you didn't run for the whole month:
  - (850.6 / 3600) * 1000 * 0.3085 = 72.89 EUR

Appendix: CCX53 Arch Linux setup

This could be scripted, but I just created the VPS via their web UI. Then after it was provisioned, used that web UI to have it boot into a rescue system. Then do an Arch bootstrap that roughly mirrors the one I use on a dedicated build machine except that we don't bother with encrypting the rootfs. The CCX* server types at least use UEFI so we can keep using efistub for boot.

First get a bootstrap environment and enter it:

wget http://mirror.hetzner.de/archlinux/iso/latest/archlinux-bootstrap-x86_64.tar.zst
tar -xvf archlinux-bootstrap-x86_64.tar.zst --numeric-owner
sed -i '1s;^;Server=https://mirror.hetzner.de/archlinux/$repo/os/$arch\n\n;' root.x86_64/etc/pacman.d/mirrorlist
mount --bind root.x86_64/ root.x86_64/ # See <https://bugs.archlinux.org/task/46169>
printf "About to enter bootstrap chroot\n===============================\n"
./root.x86_64/bin/arch-chroot root.x86_64/

Now set info that will be used throughout the process:

export NEW_HOST_NAME=archvps
export PUBLIC_SSH_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfpPQ1j+XLsapAhONAQmvu6TZGT5y8jeziM4Vio1NrA asb@plurp"
export NEW_USER=asb

And now proceed to set up the disks, create filesystems, perform an initial bootstrap and chroot into the new rootfs:

pacman-key --init
pacman-key --populate archlinux
pacman -Sy --noconfirm xfsprogs dosfstools

sfdisk /dev/sda <<EOF
label: gpt

start=1MiB, size=255MiB, type=uefi
start=256MiB, type=linux
EOF

mkfs.fat -F32 /dev/sda1
mkfs.xfs /dev/sda2

mount /dev/sda2 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot
pacstrap /mnt base linux linux-firmware efibootmgr \
  xfsprogs dosfstools \
   python3 \
  openssh sudo net-tools git man-db man-pages vim
genfstab -U /mnt >> /mnt/etc/fstab

printf "About to enter newrootfs chroot\n===============================\n"
arch-chroot /mnt

Do final configuration from within the chroot:

sed /etc/locale.gen -i -e "s/^\#en_GB.UTF-8 UTF-8.*/en_GB.UTF-8 UTF-8/"
locale-gen
# Ignore "System has not been booted with systemd" and "Failed to connect to bus" error for next command.
systemd-firstboot --locale=en_GB.UTF-8 --timezone=UTC --hostname="$NEW_HOST_NAME"
ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules # disable persistent network names

# No longer need to disable large fallback image as Arch stopped generating it
# by default

printf "efibootmgr before changes:\n==========================\n"
efibootmgr -u
# Set up efistub
efibootmgr \
  --disk /dev/sda \
  --part 1 \
  --create \
  --label 'Arch Linux' \
  --loader /vmlinuz-linux \
  --unicode "root=/dev/sda2 rw initrd=\initramfs-linux.img" \
  --verbose
printf "efibootmgr after changes:\n=========================\n"
efibootmgr -u

mkswap --size=8G --file /swapfile
cat - <<EOF > /etc/systemd/system/swapfile.swap
[Unit]
Description=Swap file

[Swap]
What=/swapfile

[Install]
WantedBy=multi-user.target
EOF
systemctl enable swapfile.swap

cat - <<EOF > /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes
Address=$(ip -6 addr show dev eth0 scope global | grep "scope global" | cut -d' ' -f6)
Gateway=$(ip route show | head -n 1 | cut -d' ' -f 3)
Gateway=fe80::1
EOF
systemctl enable systemd-networkd.service systemd-resolved.service systemd-timesyncd.service
printf "PasswordAuthentication no\n" > /etc/ssh/sshd_config.d/20-no-password-auth.conf
systemctl enable sshd.service
useradd -m -g users -G wheel -s /bin/bash "$NEW_USER"
usermod --pass='!' root # disable root login
chmod +w /etc/sudoers
printf "%%wheel ALL=(ALL) ALL\n" >> /etc/sudoers
chmod -w /etc/sudoers
mkdir "/home/$NEW_USER/.ssh"
printf "%s\n" "$PUBLIC_SSH_KEY" > "/home/$NEW_USER/.ssh/authorized_keys"
chmod 700 "/home/$NEW_USER/.ssh"
chmod 600 "/home/$NEW_USER/.ssh/authorized_keys"
chown -R "$NEW_USER:users" "/home/$NEW_USER/.ssh"

Now set password:

passwd "$NEW_USER"

Then ctrl-d twice and set a symlink for resolv.conf:

ln -sf ../run/systemd/resolve/stub-resolv.conf root.x86_64/mnt/etc/resolv.conf

Finally, reboot.

Remember to ssh-keygen -R $THE_IP_ADDRESS so you don't get ssh host verification errors.

Article changelog

2025-11-30: Initial publication date.

Alex Bradbury: Minipost: LLM inference vs training costs for DeepSeek

2025-11-29T12:00:00+00:00

Tl;dr: Based on published data from DeepSeek, we can estimate it takes something like ~70 days of inference traffic (served by DeepSeek themselves, ignoring any other providers) to match the GPU hours used for the final training run for V3 and R1.

Simon Willison recently reshared some figures on inference costs for LLMs. I couldn't agree more with the comment further down that thread "The big AI labs continue to be infuriatingly opaque about the actual figures for their total electricity and water consumption".

A number of responses wonder about the cost of training. If you accept the reported figures for serving a query, what impact does it have if you amortise the energy spent training the model over the served queries? Mistral did this for their lifecycle analysis but they grouped together "training and inference" and kept confidential the ratio of energy for training vs inference by reporting a figure that combined the training cost with 18 months of usage. The thread reminded me of another datapoint available for DeepSeek that seemed worth writing up. I think this gives some helpful intuition for the amortised cost of training for a widely used model of that size, but to state the obvious any attempt to apply that intuition to other models is totally reliant on how widely used it is.

DeepSeek have published figures both on training and on inference for DeepSeek's website and API users. I will attempt to consistently refer to the figure for training as "final run training cost" to reflect the fact the number of GPU hours used in experimentation and failed attempts isn't reported. For final run training for DeepSeek-R1:

2.788M H800 GPU hours for V3 which serves as the R1 base (see Table 1 in the V3 technical report).
0.147M H800 GPU hours for building R1 on top of V3 (see Supplementary Table 4 in the supplementary information for the R1 Nature article).
Total: 2.935M H800 GPU hours

Now for inference, back in February DeepSeek wrote up details of their inference system giving details of cost of serving, profit margin, and load over a 24h period. So yes, we're extrapolating from this datapoint and assuming it's representative. Given the worldwide inference of DeepSeek R1/V3 is surely much larger (being openly licensed there are many vendors who are serving it), I'm not overly worried about this aspect. Their reported average inference serving infrastructure occupancy is 226.75 nodes (each node containing 8 H800 GPUs), meaning 43536 H800 GPU hours per day. At that rate, it will take ~67.5 days of traffic for the same number of H800 GPU hours to be used for inference as for the final training run.

All this to say, for a widely used model of DeepSeek R1 scale when looking at the cost of inference, accounting for the amortised final run training cost is more likely to be a multiplier of 2x or less rather than something much larger. In terms of energy, this does assume that the power draw of the H800 GPUs while running inference is similar to the draw during training. And to underline again, the reported training cost surely doesn't include experimentation, aborted runs etc.

Article changelog

2025-11-29: Initial publication date.

Eri Pazos: interop and mathml core

2025-11-27T00:00:00+00:00

math { font-size: 2em; } .math-example { display: flex; justify-content: center; align-items: center; & > div { display: inline-grid; grid-template-columns: 1fr 2px 1fr; grid-template-rows: fit-content(0); grid-gap: var(--spacing); & > img { height: 100%; width: auto; border: none; padding: 0.25rem; } @media (width 768px) { grid-template-columns: 1fr; text-align: center; justify-items: center; & > hr { width: 100%; } } } }

Interoperability makes the web better for everyone, allowing users to have a great experience regardless of their choice of browser. We have been working on MathML Core making across browser engines as part of an agreement with the Sovereign Tech Fund. There are some exciting developments and new features!

Interoperability makes the web better for everyone, allowing users to have a great experience regardless of their choice of browser. We have many standards that shape how the internet should work, drafted from consensus between different engine makers and third parties. While having specs on how everything should function is great, we still need to align the different browser implementations. This can be tricky as all of them have their peculiarities, and not all browsers agree on what is a priority for them. The goal of the Interop program is to select a few important features that all engines will prioritize, so users and editors can finally benefit from them.

A few months ago I joined Igalia's web platform team (and I'm really happy about it!). Thanks to an agreement with the Sovereign Tech Fund, this year we will be working on MathML and other important Interop areas.

This post contains MathML examples. Each formula is represented twice. Your browser renders the left one from the HTML code, while on the right there is a pre-printed SVG as a reference of how it should look. Keep in mind that most of these features are either experimental or have just landed, so you may need the latest version of a browser to view them correctly.

A bit of history

MathML was first published in 1998, and it grew to be a gigantic project that sought to define how mathematical notation should be rendered. However, due to its complexity, the implementations of the browser engines were wildly different and incomplete. This meant that editors could not rely on it, since users would see very different content depending on what they were browsing with.

<math>
  <msubsup>
    <mo>∫</mo>
    <mn>0</mn>
    <mn>1</mn>
  </msubsup>
  <mrow>
    <msup>
      <mi>x</mi>
      <mn>2</mn>
    </msup>
    <mo>+</mo>
    <mn>1</mn>
  </mrow>
</math>

∫ 0 1 x 2 + 1

This is why MathML Core was born. It is a small subset of MathML 3 that is feasible to implement in browsers. It is based on the parts of the specification that are used in practice, adding important implementation details and testing.

To illustrate why this is important, Chromium had support for some parts of MathML when it was forked from WebKit. However, it proved to be very difficult to maintain and complete, so it was removed in 2013. My colleague Frédéric Wang led the effort to create a new implementation based on MathML Core, which was shipped in 2023, a huge milestone for the standard.

We are in a very exciting moment in the MathML history, since all three major browser engines have overlapping support. However, there is still work to be done to align the different implementations so they follow the MathML Core specification. The goal is that one could write formulas on a website and have it look the same everywhere (like Wikipedia, which is now transitioning to native MathML instead of prerendered SVGs).

So, what have we been working on?

RTL mirroring

Some scripts are written from right to left, including Arabic. Browsers should be able to correctly render text and math in either direction, making use of the Unicode BiDi specification and the rtlm font feature. However, the existing implementations either didn't support mirroring or had hacky behaviour that didn't work correctly for all cases. Read this explainer that Frédéric made for a great visualization of the differences.

<link rel="stylesheet" href="https://fred-wang.github.io/MathFonts/XITS/mathfonts.css"/>

<math>
  <mrow>
    <mo>{</mo>
    <mfrac>
      <mn>5</mn>
      <mn>6</mn>
    </mfrac>
    <mo>)</mo>
  </mrow>
  <msqrt>
    <mfrac>
      <mn>3</mn>
      <mn>4</mn>
    </mfrac>
  </msqrt>
  <msub displaystyle="true">
    <mo>∲</mo>
    <mi>C</mi>
  </msub>
</math>

<math dir="rtl">
  <mrow>
    <mo>{</mo>
    <mfrac>
      <mn>٥</mn>
      <mn>٦</mn>
    </mfrac>
    <mo>)</mo>
  </mrow>
  <msqrt>
    <mfrac>
      <mn>٣</mn>
      <mn>٤</mn>
    </mfrac>
  </msqrt>
  <msub displaystyle="true">
    <mo>∲</mo>
    <mi>ج</mi>
  </msub>
</math>

{ 5 6 ) 3 4 ∲ C { ٥ ٦ ) ٣ ٤ ∲ ج

There are two cases when it comes to mirroring. If there is a corresponding mirrored character (e.g. opening parenthesis to closing parenthesis), it is called character-level mirroring or Unicode BiDi, and the browser just needs to swap one character for the other. Sadly, this doesn't apply to every operator.

Take the contour clockwise integral. If we just mirror the symbol by applying a reflection symmetry about a vertical line, the arrow is suddenly pointing in the other direction, making it counterclockwise. This changes the meaning of the formula!

To avoid this, the rtlm font feature can use glyph-level mirroring to provide a different set of correctly mirrored glyphs. Glyphs plural since a math symbol can have different size variants to accommodate multiple contents. Not only that, when the variants are not enough, there are glyphs for assembling arbitrarily long operators.

<link rel="stylesheet" href="https://fred-wang.github.io/MathFonts/XITS/mathfonts.css"/>

<math>
  <msqrt>
    <mspace height="0.8em" width="0.8em" style="background: tomato"></mspace>
  </msqrt>
  <msqrt>
     <mspace height="1.5em" width="0.8em" style="background: gold"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="2.5em" width="0.8em" style="background: mediumseagreen"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="4.5em" width="0.8em" style="background: cornflowerblue"></mspace>
  </msqrt>
</math>

<math dir="rtl">
  <msqrt>
    <mspace height="0.8em" width="0.8em" style="background: tomato"></mspace>
  </msqrt>
  <msqrt>
     <mspace height="1.5em" width="0.8em" style="background: gold"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="2.5em" width="0.8em" style="background: mediumseagreen"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="4.5em" width="0.8em" style="background: cornflowerblue"></mspace>
  </msqrt>
</math>

No browser engine supported glyph-level mirroring for MathML operators, so we had to implement it in all of them. Thankfully harfbuzz, the underlying font rendering library used by Chromium and Firefox, already supported it. WebKit is a work in progress, since there is more complexity because of different ports using different backends. As for character-level mirroring, Chromium and WebKit did it right, but Firefox applied reflection symmetry instead of replacing the correct pair. The changes in Firefox and Chromium are now stable and ready to be used!

Feature	Firefox	WebKit	Chromium
Character level mirroring (BiDi)	✅✨	✅	✅
Glyph level mirroring (rtlm)	✅✨	🚧	✅✨

`math-shift` and `math-depth`

Details are important, especially when rendering complex and layered formulas. One may think that a few pixels do not make that much of a difference. However, when you have multiple levels of nesting, offsets, and multiple elements, a slight change can make everything look ugly at best, wrong at worst.

Enter math-shift: compact. Look at this example from the MathML Core spec:

<math display="block">
  <msqrt>
    <msup>
      <mi>x</mi>
      <mn style="color: mediumseagreen">2</mn>
    </msup>
  </msqrt>
  <mo>≠</mo>
  <msup>
    <mi>x</mi>
    <mn style="color: cornflowerblue">2</mn>
  </msup>
</math>

x 2 ≠ x 2

At first glance, you may not see anything too different. But looking closely, the green "2" on the left is a bit lower than then blue one on the right. It is trying to fit under the square root bar. This is what LaTeX calls cramped mode.

Chromium already supported the definition given by MathML Core, so that left Firefox and WebKit, both of which used hardcoded rules for specific cases in C++ objects. MathML Core takes another approach, and incentivizes using CSS styling rules instead.

Another interesting property is math-depth. It is used to make nested elements, such as those inside fractions, scripts or radicals a bit smaller. That way, if you have an exponent of an exponent of an exponent (of an exponent...), each one is displayed a bit tinier than the last.

<math display="block">
  <msup>
    <mi>A</mi>
    <msup>
      <mi style="color: cornflowerblue">A</mi>
      <msup>
        <mi style="color: mediumseagreen">A</mi>
        <mi style="color: tomato">A</mi>
      </msup>
    </msup>
  </msup>
  <mo>+</mo>
  <mroot>
    <mi>A</mi>
    <mi style="color: mediumseagreen">A</mi>
  </mroot>
  <mo>+</mo>
  <mfrac>
    <mrow>
      <mi>A</mi>
      <mo>+</mo>
      <mfrac>
        <mi style="color: cornflowerblue">A</mi>
        <mi style="color: cornflowerblue">A</mi>
      </mfrac>
    </mrow>
    <mi>A</mi>
  </mfrac>
</math>

A A A A + A A + A + A A A

In this case, Firefox and Chromium already had compliant implementations, so only WebKit needed to catch up. Support for math-depth and the scriptlevel attribute (which allows to modify this depth) has now landed, while a patch for font-size: math (which sets the size of the element based on its depth) is on the way.

Feature	Firefox	WebKit	Chromium
`math-shift: compact`	✅✨	✅✨	✅
`math-depth`	✅	✅✨	✅
`font-size: math`	✅	🚧	✅
`scriptlevel`	✅	✅✨	✅

Other work

Rendering unknown elements as mrow

MathML 3 defined 195 elements. MathML Core focuses on about 30, leaving the rest to styling or polyfills. This means deprecating some features that were previously implemented in some browsers, like mfenced, semantics, and maction, as it would be too difficult to make them interoperable right now. To prevent breaking existing content too much, they are rendered like an mrow.

`font-family: math`

Selecting a good math font is essential for rendering. Stretchy operators, math symbols, and italics are not available with every font, so without one they are presented very poorly. font-family: math is a CSS property that specifies that the content should use a suitable font for mathematics. Previously browsers had a hardcoded list of CSS fallbacks, but now this has been standardized and implemented.

Android doesn't come with a math font installed, so it mixes symbols from different fonts, producing a rather unappealing result:

`mathvariant` and `text-transform: math-auto`

Single letter identifiers inside a <mi> tag are treated as variables, and so they should be rendered with fancy italics. This is still supported by MathML Core. However, MathML 3 allows a plethora of transformations using mathvariant, from bold to gothic text. The new spec says that while italic transformation should still happen by default, other text should use the specific Unicode codepoint directly, as it just adds too much complexity for the browser implementation.

text-transform: math-auto is a CSS property applied by default to <mi> elements that enables the italic transformation for them. Setting the new mathvariant attribute to normal will make the text-transform of the element be none, removing the italic styling.

`DisplayOperatorMinHeight` and Cambria Math

Microsoft made a mistake in Cambria Math, one of the math fonts used in Windows. They switched the DisplayOperatorMinHeight and DelimitedSubFormulaMinHeight, so operators weren't being displayed correctly. Some browsers had a workaround for this, but a more general fix was implemented in harfbuzz, so we removed the workarounds in favour of relying on the upstream library instead.

Animation for `math-*` properties

When implementing math-shift in Firefox, we noticed that the spec said the new properties are not supposed to be animatable. In the new CSS spec, most properties are defined as animatable (fun!). After some discussion with the MathML Working Group, we decided to change the spec, and we are adding this feature to the browser engines.

@keyframes math-anim { 0% { color: royalblue; math-depth: 1; } 20% { color: mediumseagreen; } 40% { color: gold; } 60% { color: tomato; math-depth: 3; } 80% { color: mediumpurple; } 100% { color: royalblue; math-depth: 1; } } #anim-target { animation: math-anim 5s infinite; } #anim-container { height: 4.5rem; & > math { font-size: 4rem; } }

x 2

Feature	Firefox	WebKit	Chromium
Render unknown elements as `mrow`	✅✨	✅✨	✅
`font-family: math`	✅✨	✅✨	✅
`text-transform: math-auto`	✅	✅✨	✅
New `mathvariant` behaviour	✅	🚧	✅
`DisplayOperatorMinHeight` fix	✅✨	✅✨	✅✨
Animation for `math-*` properties	✅✨	🚧	🚧

What's next?

Many of these improvements have already shipped, but our work continues on making mathematics more interoperable in browsers. This includes some exciting new features ahead:

Updates to the operator dictionary: MathML Core revamped the existing list of operators and their default layouts. Additionally, there is a new compact form that removes redundancies.
More improvements to operator stretching and spacing: There are still some inconsistencies between browsers and some long standing bugs that we would love to tackle.
Handling positioned elements and forbidding floats in MathML: Like flex or grid, MathML doesn't create floating children for elements with a math display type. However, they can still have out of flow positioned children. At the moment this isn't consistent across browsers and it is something we want to improve.

Working on MathML is very rewarding, specially because of the people that have helped along the way. I'd like to specially thank my colleague @fredw, reviewers from Mozilla, Apple and Google, and the W3C Math Working Group. Also @delan for reviewing the first draft of this post.

We are very grateful to the Sovereign Tech Fund for supporting this work!

Igalia WebKit Team: WebKit Igalia Periodical #48

2025-11-24T20:12:28+00:00

Update on what happened in WebKit in the week from November 17 to November 24.

In this week's rendition, the WebView snapshot API was enabled on the WPE port, further progress on the Temporal and Trusted Types implementations, and the release of WebKitGTK and WPE WebKit 2.50.2.

Cross-Port 🐱

A WebKitImage-based implementation of WebView snapshot landed this week, enabling this feature on WPE when it was previously only available in GTK. This means you can now use webkit_web_view_get_snapshot (and webkit_web_view_get_snapshot_finish) to get a WebKitImage-representation of your screenshot.

WebKitImage implements the GLoadableIcon interface (as well as GIcon's), so you can get a PNG-encoded image using g_loadable_icon_load.

Remove incorrect early return in Trusted Types DOM attribute handling to align with spec changes.

JavaScriptCore 🐟

The built-in JavaScript/ECMAScript engine for WebKit, also known as JSC or SquirrelFish.

In JavaScriptCore's implementation of Temporal, implemented the with method for PlainMonthDay objects.

In JavaScriptCore's implementation of Temporal, implemented the from and equals methods for PlainMonthDay objects.

Releases 📦️

WebKitGTK 2.50.2 and WPE WebKit 2.50.2 have been released.

These stable releases include a number of patches for security issues, and as such a new security advisory, WSA-2025-0008, has been issued (GTK, WPE).

It is recommend to apply an additional patch that fixes building with the JavaScriptCore “CLoop” interpreter is enabled, which is typicall for architectures where JIT compilation is unsupported. Releases after 2.50.2 will include it and manual patching will no longer be needed.

That’s all for this week!

Planet Igalia

Andy Wingo: nominal types in webassembly

cursèd nominal typing

fin

Yeunjoo Choi: Smarter Chromium GN in Vim with gn-language-server

Setup

What’s Working

Hover Documentation

Go To Imports

Go To Dependencies

Current Limitations

Code Folding

Go To Definition

Igalia WebKit Team: WebKit Igalia Periodical #59

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Tiago Vignatti: Accessibility and PDF documents

Accessibility #

Igalia WebKit Team: WebKit Igalia Periodical #58

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Ziran Sun: A Day in “State of the Browser 2026” Conference

Frédéric Wang: Stage d'implémentation des normes Web (session 2026)

Igalia WebKit Team: WebKit Igalia Periodical #57

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Mauricio Faria de Oliveira: page_owner Part 2: optimizing output

Mauricio Faria de Oliveira: page_owner Part 1: a quick introduction

Alex Bradbury: Minipost: Additional figures for per-query energy consumption of LLMs

Alex Bradbury: shandbox

Usage example

Functionality overview

Implementation approach

Making it your own

José Dapena: Container Timing: measuring web components performance

The problem: measuring component performance #

The solution: Container Timing #

How to use Container Timing? #

A native implementation for Chromium #

Upcoming trials #

Wrapping up #

Thanks! #

References #

Igalia WebKit Team: WebKit Igalia Periodical #56

Cross-Port 🐱

Graphics 🖼️

WebKitGTK 🖥️

WPE WebKit 📟

Releases 📦️

Community & Events 🤝

Andy Wingo: six thoughts on generating c

static inline functions enable data abstraction

avoid implicit integer conversions

wrap raw pointers and integers with intent

fear not memcpy

for ABI and tail calls, perform manual register allocation

what’s not to like

Andy Wingo: ahead-of-time wasm gc in wastrel

hello, world

is it good?

improving on v8, really?

zowee!

Igalia Compilers Team: Igalia’s Compilers Team - A 2025 Retrospective

Standards #

Temporal #

Decimal #

Source Maps #

Modules #

AsyncContext #

Unicode standards #

WinterTC #

Node.js #

Accessibility
#

fear not `memcpy`