In the first slides from the only public presentation (2015) we claimed that the idea behind Arcan was to find the crunchy middle between a display server, a game engine and a multimedia processor.

The control plane to this “desktop engine” was designed for a scripting API targeting entry-level developers. Lua remain as the weapon of choice for that role as a better follow up to the ‘BASIC’ of the home computer era, but also as an intermediate for other tools to compile to.

To make a long story short: That, my friends, is a browser; just average in a few areas, yet create something interesting out of orchestrated mediocrity — not unlike musical theatre.

In order to get anywhere on such a broad topic we will need to limit the scope a bit or we will be here all day. The web itself is as much, or more, social structures growing and evolving as a bacterium on a technological substrate than it is any singular piece of technology.

Even if some cry out in doom and gloom about the dominance of Chrome ‘the browser’ (albeit also Chrome as Chrome OS and Chrome as Electron), do remember that it is still subject to other pieces like the server, the protocols, the app and the structural glue that is ‘the link‘. If you want to offset one part, you ought to consider how well it plays with the others as well.

This article will only focus on the technical side to the user facing access to this substrate. That is the browser software itself. An example of the makings of another ‘web’ to go with it, sponsored by NLnet (https://nlnet.nl/project/Arcan-A12/), is covered in ‘Arcan-a12: Weaving a Different Web‘ but it is architecturally loosely coupled.

To make things a little easier than what is normal around here, I will split it into a simplified ‘short form’ and as two heavier text sections.

The first section, Browser Breakdown, covers the problem space and some cherry picked history. The second, Arcan as Browser Design, goes into how we account for it.

The browser broken down is the end result of “the document” trying to envelop the computing world. The browser as proposed is “the desktop” trying to reach out.

The short form can be found here:

To demonstrate that this is existing technology, the presentation is also provided as an Arcan appl, viewable through “arcan-net arcan.divergent-desktop.org explain” (it is running on a potato used for active development).

That one happens to have been published a while ago. According to the logs, about two people have managed to find it. It seems there are lessons from the past about other meanings of “Search” that might be useful to rediscover now when more and more people and places alike are all llmpty on the inside and the noise floor is reaching the ceiling.

Some suggested extended reading on the topics that will be covered here and in the next section:

• The History of PDF (historical, light read)
• Information Management: A Proposal (1990) (historical, intermediate)
• NeWS: A Networked and Extensible Windowing System (video, pdf) (historical, advanced)
• Inside look at a modern web browser (current, multiple parts, light read)
• SOK: On the Analysis of Web Browser Security (current, advanced)
• A File Structure for the Complex, The Changing and the Intermediate (historical, advanced)
• Five Walled Gardens: Why Browsers are Essential to the Internet and How Operating Systems are Holding Them Back (current, political)

Browser Breakdown

The big question to unpack is if the browser is supposed to ‘present and navigate interlinked documents?’ in the spirit of the original much-too-simple hypertext, or be ‘an operating system agnostic substrate for networked applications’.

Lets start with a hypothetical speedrun through history.

You are back in the cosy times of the early 1990ies when computing was slow, storage was small and floppy disks got corrupted simply by staring at them for too long. You are struck with the brilliant idea of developing a software product that is supposed to just show very simple documents with formatted text in various typefaces, sizes and styles. Throw in the odd image or two. Maybe some media clip, but that can come later.

These documents should naturally be hosted and shared across networks of computers as that is faster and more convenient than copying things to floppy disks.

It turns out great with a lot of satisfied users that soon wants more. At the same time, windowing systems become more common. These force your simple document viewer to support moving things around to react to the window resizing. Users naturally want to handle different outputs, such as printers.

The documents these users have created started to form packs and people wanted ways for them to cross reference each-other and navigate these references with the click of a button. You oblige by adding ‘links’ to other documents and the option to open them in a new window.

Networks at this time are slow and servers are quite busy, so the next logical step would be some caching to make sure you only retrieve a linked document or its embedded images when something has actually changed.

A ‘popular’ kind of document turned out to be ‘forms’ for people to fill out, print and post because those are like cocaine to governments and businesses alike. With that you add some way for the user to type into parts of these forms to fill them out before printing. This will save many from wasting time deciphering messy handwriting. Both throughput and accuracy improves greatly.

Still people manage to fill them out wrong — to err is human and all that. It would save everyone time if you just add some easy way to have presets to chose from; a list box here and a radio box there. Soon enough every UI ‘widget’ imaginable has managed to join in, but you still think of it as a ‘document’ in the end.

Why even go through the pains of printing and posting? We can just sneak in some magic to POST the filled form as an e-mail or send it to the same place the document was hosted. Even better, we can embed some code to make sure that some of the inputs the user provides can be checked for errors locally and tell the user about any errors.

If something is made programmable, it follows that some crazy hacker kids will figure out creative ways to take advantage of this to do all sorts of unintended things. It did not take long before the documents started spawning little cats chasing the mouse pointer around.

Documents reshaping themselves as they were being viewed became the new normal. That is a big change in expectations for all stakeholders.

You should be able to see where this is going by now — It doesn’t take that many feature requests for a static document to become a dynamic application, similar to how few wrong turns it take to make a regular language grammar into a recursively enumerable one.

To keep up with new discoveries in everything from image compression to network communication lots of new code got written in order to support new media formats and more advanced protocols.

The easiest way to make software do new things is to add more code, and the path of least resistance is typically the one people follow. This is where software architecture matters and stops being just boxes and arrows on a whiteboard somewhere; it sets the constraints for how new code combines with the old.

Just adding more code to solve for new features rather than refactoring to be more clever about things comes at a cost. More code means more potential bugs. Mix in economic incentives and some of those bugs turn into opportunities to exploit which calls for mitigation strategies.

One of the more potent ones are sometimes called sandboxing (a terrible term but that is another can of worms best left unopened) and comes with heavy requirements on security boundaries. Re-architecting to do that successfully is not for the faint of heart or light of wallet.

In the browser case, such a project comes with another interesting side effect: you are now compositing and routing inputs and outputs between a number of interactive sources and sinks across several privilege boundaries.

That is how you end up with something like this:

Congratulations, you’ve made a display server with the weakest windowing system in town. Instead of leveraging the IPC system for better ways of collaborating with local software they stuck to crude left-overs from the outer windowing system like clipboards, drag and drop or, sigh, “portals”.

A historically interesting tangent is that Microsoft was on this trajectory way back when and got close to a stronger solution through its COM/OLE/ActiveX path before falling into the antitrust trials pit (not that the idea went away).

Now every few years there is some take on the “Java Applet” form of assimilation by forced cross compilation to an embedded executable format referenced from the document (which turns the document into a de-facto format for linking and loading yet with few of the insights embedded in proper executable formats) to target a virtual machine. Shockwave Flash being a notable one, and “WebAssembly” a more recent attempt.

What if, instead of trying to regress towards “simple” in the browser domain (just to see the same story repeat itself) or in the data model (and still be stuck with the same browser story) – we assume the networked application endgame from the start and design for that?

This leads us to …

Arcan as a Browser Design

To summarise the previous section – the browser started as linked documents breaking down into a networked application with ‘the document’ organically turning into ‘an executable format’.

In a spider graph form they went from this:

To this:

These are, of course, rough estimates. Improvements along each feature axis is exponentially more difficult and expensive.

The expected and required feature-space span the entire gradient:

It is worthwhile to note that it can transition from any point on the gradient to another. You get no guarantees about where you start, or if it will stay that way, unless you disable expected features with unpredictable results.

We can say the processing is iterative and the linking (as in program execution) is dynamic and mutable (events triggering code pulling in remote code hooking events).

In contrast, Arcan started at roughly this back in the early 2000s when hope was high and life worth living:

It is now here:

The processing looks like this:

We can say the processing is planar, static and recursive. With planar we mean that if there is a document stage, there is a discrete and decoupled transition from that into the app form. You know, a build system: compile, link/package, execute.

This means that the “as document” part of the runtime are separate tools not part of the browser itself. It only concerns itself with being an execution environment for the networked application. This keeps the browser part from having to walk an endless trail of outdated formats. Some are already conditioned to this approach with that of “static site generators”, it is just the output format that needs some change to move away from document to document translation.

With static it means that the compiled app is not capable of loading code outside of its own package, except for a set of preset helper scripts. This is the default with a user controlled opt-in rather than an opt-out as in “install extension to disable javascript”.

With recursive it means that navigating a link starts another instance of the browser that would either embed itself as any other data source in the parent, or take its place.

The emphasis on the display server aspect throughout the years here is no coincidence: it’s the weak-point of the dominant browser model and one that has very little documentation. It is also the reason why the apps developed and demonstrated here has mainly been desktop oriented — to politely point out that “you missed a spot”.

Chrome splits its app execution into a number of processes that are roughly “Network”, “Browser”, “UI”, “Storage”, “GPU”, “Device”, “Render” and “Plugin”. This is a logical breakdown of the internal division of responsibility. For privilege separation, each process role has a configured policy on which system resources it can access.

Arcan splits up its app execution into a number of processes by information transfer boundaries:

• Decode – computer to human representation (images, videos, music, voice synthesis, …) into GPU buffers, pixel transfers and audio sample streams.
• Encode – human to computer representation (recording, streaming, language models, object-character recognition, …).
• Network – name to address resolving, discovery and retrieving remote resources.
• Terminal – outer operating system space controls.

These come as a set of external clients that are picked for each instance – meaning that the supported formats, protocols and so on can be swapped out without any changes to the engine itself and per application basis. They are dependency and license “sponges”.

Their security model comes from a combination of least-privilege and capabilities. “Decode” is security wise the most sensitive one as that is where parsing of untrusted inputs go. “Encode” is privacy wise the most sensitive one as that is where the real ‘you’ distil into digital form.

After the connection is setup, it needs no file system access at all. The system calls required match an OpenBSD pledge of “stdio sendfd recvfd” which is a hard surface to break through. Even then we have microarchitectural attacks to consider in the most hardened setups. This is where the ability to live-migrate any shmif client to a networked form comes in — you can compartment to cheap one-off devices like single board computers and the architecture still holds. Memory is never safe, wear a helmet.

State store and list of permitted integrations with local software are in an SQLite database that is also picked for each instance. That local software can, of course, also be a different browser.

This becomes a hierarchy that looks like this:

Not pictured here are the ‘hook-scripts’ facility which corresponds to ‘browser extensions’ in the web sense and the ‘developer tools’ sense.

The key component is the inter-process communication system (IPC). The one in Chrome, Mojo, is highly generic: an interface description language (IDL), a binding code generator and there you go. The qualities of the IDL limits what can be communicated and the binding generator affects the developer effort it takes to use. Part of the interface description links in ancillary specifications. An important one is the sandbox policy to apply.

We have seen it before: MIDL, CORBA, Fuchsia IDL, D-Bus, Wayland, Binder (AIDL) etc. ad nauseam. One may wonder how many generic ways we may need in order to eventually be specific.

The pitch for those are always ‘extendable / flexible’. I read that as ‘afraid to commit to anything’. It is a tempting model if you are unsure what you are supposed to be building (or organisation decoupling where you want formal barriers for communication between teams) and kick the bucket down the road. It is less tempting if there is ample history and plenty of examples to learn from.

The generic approach come at a cost. Code generated creeps into, or even dictates, the design of both sides and it ends up much bulkier and more verbose. It is rare for ancillary tooling support for debugging, report generation, testing, profiling etc. to be included.

In a previous life as a security auditor, a very successful path to victory was to locate any IDLs in use; patch to integrate fuzzing; deploy and wait for things to blow up. They typically do.

The IPC in Arcan, SHMIF, is highly specific. It is a lock-stepped data model with a C API. The message types and fields came from the needs of a full desktop OS, which has changed very little over the years. There is no reason to be clever, flexible or extendable. Just look at what others have done before and generalise from that. Validate against existing examples and use cases. Easy peasy.

Where we did put effort into being clever was two parts. Crash resilience and Alternate Representations.

Both are heavyweight topics in their own right, a dissertation or three, but for this context the motivation for resilience is easy enough – it allows us to redirect user workloads to relaunched instances (recovery), to others or other devices across the network (redirection) for collaboration or stronger security compartments, or to a future version of ourselves (upgrade).

For alternate representations, well we don’t all look at the world with the same eyes or desires now do we? The one that should be obvious is accessibility where the screen reader tradition is to snapshot composition state back into a document form and then navigate through voice synthesis and keyboard controls.

A less obvious one is for debugging – be able to visually inspect and interact with the dynamics of the software rather than what it presents itself as. The other side of the same coin comes from malware reversing as that kind of work leverages the same kind of mechanisms. Process parasites (or as some call it, ‘Fileless malware‘) have been a thing for a very long time now, yet the generalised tools to shake them out have been sorely lacking for just as long.

In closing, there is a lot of technical detail that had to be omitted for this piece to not be completely indigestible but at the high level, there is not much more to say about the Arcan project as such. There are things left to do, but the end is in sight. Midnight draws near.

For those on site that want to have a chat, our DECT extension is 6681. Incidentally the same number as the control port for when Arcan was used to provide overlay, texting, memes, scheduling etc. for a pirate university TV station back in the early 2000s.

Before going into the project updates, there is tragic news to relay that put a massive damper on spirits and overall productivity for the last few months: Our most beloved of project members, Elijah “moon-child” Stone died on the 9th of September at the young age of 22. Elijah had been with the project since his early teens and was consistently kind, caring, courteous and clever well beyond his years. The 0.8 topic branch on one of his favourite subjects, performance engineering, will be dedicated to his memory. Our thoughts are with his family and partner.

Community Updates

It has been quite a while since we switched from GitHub to Fossil for development. Since we don’t expect people to tool around an uncommon tool we also mirror to git hosted on Codeberg.

A final friendly warning to packagers is to use those repositories. The ones on GitHub will no longer receive any mirror updates and any changes over there are likely to be of a more incendiary nature.

As covered in more detail, (part 1, part 2) we have had a longer in-person hackathon. The main outputs from those is that Alexander has a port of Gamescope to save us from keeping up to date with all the special quirks that running Steam over Xwayland takes. The clip below shows it running Baldur’s Gate 3.

Magnus keeps chipping away at his platform plugin for Qt5/Qt6 that largely works as intended for the likes of Qbittorrent and Binary Ninja but struggles still with hybrid 2D/3D complex window managed applications like FreeCad.

He also created patches to KeepassXC and a script (added upstream) to Durden to integrate them. Both are recommended reading.

Valts is still busy with his portable viewer for the A12 protocol that should soon be usable with some of the neat bits we cover further below.

In the bin of experimental applications, we have Atro with “Lasso” that is a hybrid ‘interactive canvas’ form of window manager.

On top of providing a number of bug fixes across the board, Bohdan created Xkbd2Lua for statically translating X Keyboard Layouts to our own format, removing the need to lug libxkbcommon around for those that want even fewer traces of X11 around in their lives.

Ariel has been toiling away at one approach for a bootable complete Arcan+Durden+Cat9 setup from as static a build as possible matching Arcan as OS design (there will be more). As part of that there is a nix oneliner that should hopefully work on a few setups:

nix run --impure 'git+https://codeberg.org/ingenieroariel/arcan?ref=nix-flake-build&dir=nix'

Status Update: Arcan

As normal, check the changelog for the fine grained changes.

A lot has happened on the network side, thanks in large part to the continued support from NLnet and a longer write up for that is coming up in a little while.

To start with, there is now support for ML-KEM as Post-Quantum cryptography to protect against ‘collect now, decrypt later’ (should the fantasy computers ever materialise). This is implemented as part of the forward secrecy ratcheting rekeying process. The same has been extended to set a signature verification key for file transfers and a way to limit a high number of search requests during high load.

Connection resumption and Casting

Clients that act as sources now has connection resumption support. This means that if the network connection is lost, the source application is kept alive and re-paired when you connect back.

In the following clip I first host arcan running pipeworld on my networked machine and connect. The window pops up and I create a few cells to show that there is data/state remotely. I close the window and reconnect. The window reappears just as I left it.

There is also a –cast option added. This lets the first user that connects become the “driver” over the hosted application. Any subsequent connections gets a read-only copy of the stream.

Unified and Referential Links

The other major network changes have to do with the directory server part. To recap, any endpoint can have one out of three roles: Source, Sink or Directory. Normally a source hosts some kind of application and provides, either inbound or outbound, access to a Sink. The Directory works as a self-hosted rendezvous for discovery, but also state, file-store and coordination between multiple clients using the same appl.

For instance, if I have a directory server hosting the Durden desktop appl and multiple clients download and run it from the directory, they can use the messaging domain that the directory provides to synch clipboard state or share input devices like one would do with Synergy/Barrier in the past.

The admin API for configuring the directory server has received two new functions, reference_directory for referential links and link_directory for unified links.

A referential link lets users with access to directory server ‘myserver’ access a referenced directory ‘myfriend’ and so on, letting them browse a larger network. Using the command-line tool, the following example:

arcan-net --path myfriend myserver@ myappl

Would have the client connect to myserver, then download and run myappl from the referenced directory myfriend. The path can/be/arbitrary/long. Connection primitives are negotiated by each step in the chain, which takes us into the very interesting space of transitive trust-discovery models.

In the example above, myserver gets a DIROPEN request for myfriend from the client. Myserver knows that this is a referential link, and asks myfriend for connection primitives on behalf of the client – forwarding the public key previously used by the client to authenticate to myserver. Myfriend returns with either direct connection information (ip, port etc.) or a request that myserver tunnels the traffic if it is not directly reachable.

The unified link is invisible to the user and is a more privileged connection. It lets multiple directories form a shared namespace such that they can access/host/mirror the same resources as if they were one logical server.

Say that you have a server in your home and another hosted on a VPS somewhere. With a unified link between the two, your devices can access the one when you are at home, and hand over to the other when you go outside.

Dynamically Hosted Directed Sources

For every directory hosted appl it is possible to slot in a controller. A controller is a set of server side scripts that regulate messaging and resource access for more advanced networked applications. The scripting API has received some new functions that are worth looking into.

The first is launch_target. This is best explained through an example. Say my directory server configuration database has a launch target defined, like:

arcan_db add_target "chromium" /usr/bin/Xarcan -redirect -exec chromium

I then slot in a controller for the ‘durden’ appl by doing this:

# mkdir durden; echo '
function durden_join(cl)
  launch_target("chromium", {}, cl)
end ' > durden/durden.lua

# arcan-net --sign-tag mytag --push-ctrl durden myserver@

This would package the controller for durden, sign it with whatever is assigned to mytag in the local keystore, and upload it to the directory server marked as myserver in the local keystore.

The server verifies permissions and that the signing key match previous signatures, and assigns a runner VM. Now the next time any client would run the durden appl:

# arcan-net myserver@ durden

It would spawn an instance of chromium that connects as a source only visible to the specific client, with a polite suggestion to sink it immediately. The client does so automatically and the window pops up.

There are more controls to add here for state management, sandboxing details, letting the controller script inject events and so on. The point of the feature is that the beefier server now has a mechanism for fine grained application hosting.

Speaking of, if the client end doesn’t have the full Arcan stack but something simpler like the Smash viewer, it is now possible (opt-in) to let the directory server host the arcan side of the equation turning the client into a very thin one:

# arcan-net --host-appl myserver@ durden

External Resource Resolver

Other changes to the controller development side is that the event handlers used to to list/download/upload files can now be hooked up to an external resolver.

The way it normally works is that you add an event handler, like this:

function durden_load(cl, name)
    return name
end

This does nothing fancy, it just forwards whatever the client requested to the server-side file-store. The function in this form is just to add any client specific block/name translation. Another option would be to return the result of open_nonblock to dynamically generate the data to be transferred.

If I modify the server’s config.lua to this:

function ready()
    launch_resolver("durden", "/usr/bin/myresolver")
end

Any resource requests forwarded from the runner scripts to the ‘durden’ appl will be routed to an external process. This can be any shmif client that implements a handler for the BCHUNK_IN / BCHUNK_OUT events. A simple test client can look like this: https://codeberg.org/letoram/arcan/src/branch/master/tests/core/a12resolve/a12resolve.c

Had this been the assigned resolver, storage requests would be rejected outright and load requests would get the ‘test.mp4’ file contents regardless of what was asked for. The point of this feature is to provide caching and translation to other file providers. Ones currently being looked into covers regular URLs, Magnet-to-torrent and IPFS.

Custom Debugger Integration

The remainder of changes are mostly on the developer side of things. Recall that we once wrote a debugger frontend to Cat9. This implemented the Debug Adapter Protocol, but was designed in such a way that we could support other protocols. DAP is too heavy for our own needs, so a lighter one was thrown together to match how we use the Lua VM in both the engine and the directory server. This lets us debug locally as well as with a corresponding directory controller in lockstep mapped as remote threads.

The following clip shows the current state of that in Cat9:

It can also be attached on-demand on a scripting error. There is some heavy lifting left on the frontend side before this is completely seamless, but in a not too distant future we will be at a point where you can pause-edit-update-continue a fleet of devices running a single Arcan application at once.

With that we have all the building blocks in place for more interesting networked Arcan applications. The first target for that will be a community chat application to move us away from Discord.

48 of the bottles of Mate did arrive, so to fill the hole we speed-ran through them and none us will have another bottle again, or until the next time — whichever comes first.

For show and tell it was finally Björn’s time to shine like a very distant and faint star. First squishing a number of long standing bugs, such as primary screen sometimes not getting the right identity/serial/make/model at startup breaking monitor settings persistence; unreliable transition between 16-bit, 24-bit, 30-bit and fp16 composition modes; cat9 subwindows failing to materialise when hosted over a network and directory server hosted sinks randomly shutting down (spoiler: always check your signal masks when execve()ing).

Then, apparently, he had snuck in a little edit mode into cat9, promptly overdoing it by adding vim style input modes, folds and language support. The following clip captures the current state.

The tangent was justified by some long rant about feature archetypes and a pilgrimage in search for the answer to the existential question `if a text editor is predestined to eventually grow a shell, and a shell is predestined to become a text editor, will they end up in the same place?`

With the aforementioned fixes we saw first signs of arcan-net -l 6680 -- /usr/bin/gamescope supertuxkart running, with a little caveat left to fix:

The details are fairly interesting. When trying to pass accelerated buffers from a shmif source, it is always possible for the other end to reject acceleration. The buffer passing routine then tries to gracefully convert it into pixels and try a slow path. This does not currently work when the shmif context structure is not actually managing the GPU, as the screenshot illustrates. The network tool (arcan-net) currently always rejects accelerated buffers and tuxcart-on-psychedelics it is.

To celebrate its 30 year anniversary we watched the aptly named hacking movie ‘Hackers’. Magnus confessed to not having seen it even once. Oh to be young again, and also a robot.

Speaking of that, the QPA 3D support commits materialised, then proclaimed to be working flawlessly, only to promptly break as soon as others tried it out. Somewhat disheartened by the fact and followed by “I fixed the race condition now but sometimes it goes black for some reason” on repeat, he asked for help and help was given.

The following clip is an artist’s rendition of Björn providing sagely advise about swap-chains, fencing and buffer lifecycles during resize.

Valts, meanwhile, started his days with the work from the previous night unavailable to him. Swallowing a large dose of what the kids would call copium, “It is not Nix. There is no way it is Nix. It was Nix.” could be heard.

This was promptly followed by adding a UI for configuring smash that we will opt to not show here to spare any would-be-UX designers from feeling overwhelmed.

All in all the participants confessed to having mild to moderate fun, with faint hopes of meeting up again at the next Chaos Communication Congress – should we manage to score some tickets.

The theme this time around is “Project Wayhem” with hem meaning home in Swedish, and it taking place in my renovation project/eternal money sink fort out in the middle of nowhere. Three brave travellers fought the perils and outrageous ticket prices of public transport in Sweden and came out victorious.

After an unhealthy amount of greets, rants and assorted beverages, we eventually managed to get to work.

First up in the show and tell was Alexander with his SHMIF backend for Gamescope that got massaged into working shape. The repository can be found here: https://codeberg.org/sashabjorkman/GameScope

The following clip shows it in action, playing Baldur’s Gate 3.

He also commissioned an artist for an event-appropriate mascot:

Scandinavian hedgehog being the only non-rubbish wild animal currently living in my backyard.

While we have paths for running x11 and wayland clients already, such as Xarcan and our native wayland support, nobody wants to work on the wayland implementation and the Xorg DDX works well enough for clients with reasonable use of desktop resources. Steam is not such a client and definitely not reasonable.

Since the goal-posts for anything “”Wayland”” these days are mounted on roombas powered by jet engines, the best effort / reward solution is Gamescope: write a backend for that and let them exhaust their resources chasing down the complementary IPC system of the day or the next ‘protocol’ extension to join the animal farm.

We celebrated this victory with a few runs of the true eternal game of the year (tuxkart) on the only machine out of 10 or so with functioning bluetooth support for the controllers. Instead of the 2.4 meter projection screen in a calibrated viewing environment, we thus huddled around the tiniest laptop screen around.

Next up was Magnus with his Qt QPA (found here: https://codeberg.org/vimpostor/qtarcan). He fought bravely, and eventually won, getting 3D accelerated windows working. This is one of the few remaining items before the fabled feature completion.

After a number of hours staring into the void and the void staring back, “fucking hell I can finally see my application, this is huge” could be heard from several kilometres away.

While everyone is waiting for the commits to materialise, the following screenshot is proof that he is not a pathological liar:

Valts (thanks NLnet!) being true to his nature, was much less dramatic and kept toiling away at the alternate, portable and lightweight implementation of our network protocol, that can be found here: https://smash.tase.lv — inching closer to supporting multiple windows. He was also trying his best to say good things about Nix and that was promptly ignored.

Björn (thanks NLnet!) while not busy making good use of his many academic degrees by cooking; writing this blog post; crafting cocktails and preparing snacks for each movie night, spent his time trying to debug why every n’th modeset between SDR and HDR caused the amdgpu kernel module to crash. He got absolutely nowhere and remain a disappointment to most everyone.

To salvage some remaining shreds of dignity, the 8+ year old backend for SDL2 got resurrected.

The purpose of that is threefold: One is to pretend to actually do something useful. The other is making sure there is an reference to compare to when eventually repeating the process for SDL3. The third reason is simply that we need more varied clients to test network transparency and local/remote migration with.

With several days to go, there might be more things to report on in the near future. With any luck, the 60 bottles of Club Mate ordered weeks ago might actually arrive before everyone heads back.

First, a few links to previous parts:

• Chasing the dream of a terminal-free CLI (2016, rant)
• Dawn of a new comand line interface (2017, architecture)
• Day of a new command line interface: shell (2022, design)
• Lash#Cat9 (2022, proof and development)

On top of those, and the many related demos, we have specialised tangents on accessibility and networking. The only remaining thing we have yet to cover is the programmer’s interface to all of this.

Before slipping into that dark night, I would like to summarise the mental model I use for ‘terminals’:

Terminal emulators emulate a machine that has never existed, a fantasy computer. They are constructed just like other emulators for the weakest of devices; a big old state table and switch (*pc++){case OP_1: implement OP_1(); case OP_2 ...} with a quirk being that the real horrors are hidden elsewhere. Many indulge in expanding this fantasy computer, adding all kinds of crazy expansion boards for faster video modes and co-processors. This is nothing new in itself — we have done that to practically every computer ever.

The tragedy is that it is taken seriously as a default.

Another day at the office is just running a terminal emulator (xterm) inside a terminal emulator (console) so you can use a terminal emulator-translator (ssh) to emulate more terminals (tmux). It is emulation all the way down instead of as a last resort for compatibility.

The other quirk is the flora of instruction sets to emulate and the subset selection heuristics. These are downplayed as ‘just text’ but everything that touches American Standard Code for Information Interchange are raw instructions for this machine. Programs for it are everywhere and most of them transform or expand the code, without being treated with the same scrutiny as one would for other code generators such as compilers.

A related take worth highlighting is this quote:

It feels a little unsettling that what we now use, effectively, as a data encoding format (ASCII) continues to hold – and it’s up to us to handle when writing software, too – things that correspond not to textual data, but TTY key press and serial link events, and that they follow you everywhere, even in places where the are no teletypes, no serial links, and no keys. Like most people who poked at their innards I used to think that terminal emulators are nasty, but for the longest time I thought that’s just because we took the “emulator” part a bit too seriously and didn’t bother to question it because we’re intellectually lazy. Now I realize it’s legitimately hard to question it, because a lot of assumptions are actually baked into the damn data model – terminal emulators aren’t leaky abstractions, they’re quite literally the abstraction being leaked :-D.

[email protected]

Computer people have tried to tame this beast before, using abstraction libraries such as TurboVision and Curses (hence the title of this post). One might be tricked into thinking that keeping these interfaces and substituting the implementation with something befitting the real computing environment would be enough. That is where the ‘leaky abstraction’ part meets ‘just text’ pops up like a little demon and plants itself firmly on your shoulder. Since it is so easy to sneak in arbitrary instructions in-band, people have done a lot of that, all over the place — going so far as to publishing cutesy little articles encouraging the malpractice.

Which building blocks do we have to plug the leaks with?

The Arcan project is suspiciously large in scope. Much of it is not needed for this particular problem. There are a few parts to single out, and a simplified build-mode (-DBUILD_PRESET=client) to assist with that. This will produce:

• libarcan-tui
• libarcan-shmif
• libarcan-shmif-server
• libarcan-a12
• afsrv_terminal
• arcan-net

[libarcan-tui] is the meat in this dish. It fits the ‘ncurses’ bin (.. and ‘libreadline’ and other widgets). There are also Lua bindings with some extra conveniences to smooth over other related POSIX details to help escape the clutches of /dev/tty.

Its reference implementation uses [libarcan-shmif] to talk to the desktop, using some variant of [libarcan-shmif-server] for the other half of that equation. One could substitute in [libx11/win32/…] for those, but some features make that quite difficult. For an technical walkthrough how those work, see the recent writeup of A deeper dive into the SHMIF IPC system.

To fill the portability gap across mobile and desktop, we have a (young) separate project, smash, managed by Valts Liepiņš. For the Zig capable, this is a good place to support.

[libarcan-a12] provides the reference implementation for [A12], the wire transport for this across devices. [arcan-net] providing a standalone binary wrapper for common use cases.

[afsrv_terminal] is the last piece in the equation. It is a standalone binary which, at first glance, is a terminal emulator(!) that outputs into [libarcan-tui]. That is needed for compatibility and proof of feature parity. When passed an environment (ARCAN_ARG=cli=lua) it ignores the terminal emulation part; enables the Lua bindings with some added functions useful specifically for shell work; and runs a bootstrap script that loads the real shell. An example of such is the Lash#Cat9 link from before.

Enough babbling. Time for some code.

We start with a low level C ‘hello world’, then repeat it using the Lua bindings:

#include <arcan_tui.h>

static void redraw(struct tui_context *C)
{
  arcan_tui_erase_screen(C, false);
  struct tui_screenattr text =
      arcan_tui_defattr(C, TUI_COL_ALERT);
  arcan_tui_writestr(C, "hello world", text);
}

static void resized(struct tui_context *C,
                    size_t px_w, size_t px_h,
                    size_t cols, size_t rows, void *T)
{
  redraw(C);
}

int main(int argc, char* argv[])
{
  struct tui_cbcfg cbcfg = {
   .resized = resized
  };

  arcan_tui_conn *con =
    arcan_tui_open_display("myname", "hello");

  struct tui_context *tui =
    arcan_tui_setup(con, NULL, &cbcfg, sizeof cbcfg);

  if (!tui)
    return EXIT_FAILURE;

/* draw something immediately */
  redraw(tui);
  arcan_tui_refresh(&tui);

/* main processing loop */
  int inf[] = {STDIN_FILENO};
  while(1){

/* block (-1) until there are events happening on a set of contexts or inbound data on a set of descriptors (inf) */
    struct tui_process_res result =
      arcan_tui_process(&tui, 1, inf, 1, -1);

/* synch any changes imposed by the event handlers */
    arcan_tui_refresh(&tui);
  }

/* shutdown, with no error message */
  arcan_tui_destroy(tui, NULL);

  return EXIT_SUCCESS;
}

There are quite a few nuances to cover here already.

For starters, opening a display matches the pattern used in EGL and elsewhere in using an opaque pointer to allow the implementation to support different outer display systems. It takes an immutable ‘name’ of the application and a mutable ‘identity’ (e.g. the current open document, path or other resource).

With the display, we can acquire a context (window) and set a table of event handlers. The table also lets the implementation know which features you support/need and which ones you do not.

Some of the possible handlers include:

• Input: Text codepoint, Abstract announced label, Native keys with modifiers
• Input: Mouse buttons and motion
• Window Management State: Visibility, size and colourscheme changes
• Acquiring new window contexts
• Language and location preferences
• Clipboard paste
• State transfer requests
• File transfer requests
• Seeking requests (horizontal / vertical scrolling)

Note: Passing the size of the vtable doubles as an additive version identifier. This makes it possible for features to be appended to the table without breaking backwards compatibility.

A tui context represents a single window. You can request additional ones from a set of types, and have ones pushed to you as a user request for alternate representations (mainly accessibility and debugging).

Calls to arcan_tui_refresh will atomically forward changes to the contents of the window. If nothing has changed it will return immediately.

When drawing we have a lot of shaping attributes to apply, from the regular bold/italic/underline/strikethrough to shaping hints, double-width characters, ligature substitutions, border drawing and foreground/background colours.

Note: Colours are a special case. The terminal tradition is ugly - there used to be a small set of reference colours, the emulator remapping them as desired (RED is now YELLOW) and several different extensions to specify explicit red/green/blue values or a wider palette. Here there is a semantic palette (ALERT, LABEL, ...) as well as the legacy one. The values they map to are tracked per window and can be dynamically updated by the outer display system (this triggers a 'recolor' event). They are resolved to red/green/blue values when the window is packed on refresh.

The process function takes a set of tui contexts, a set of file descriptors, their sizes and a timeout. The set of contexts is for handling multiple windows and the set of file descriptors and timeout for the common pattern of being input triggered.

Note: Since the UI processing is all out of band, STDIN and STDOUT are left intact and you can mask all signal handling. This removes the need for isatty() and you get both pipeline processing through an intact STDIN/STDOUT and an interactive UI. The choice of colouring or not colouring output is up to the final presentation.

In Lua, it would look like this:

tui = require 'arcantui'
local function redraw(wnd)
 local attr = tui.attr({fg = tui.ALERT})
 wnd:write_to(0, 0, "Hello World")
end

local handlers = 
{
  resized = redraw
}

wnd = tui.open("myname", "hello", {handlers = handlers})
redraw(wnd)

repeat
  wnd:refesh()
until not wnd:process()

There are a lot of bells and whistles in order to cover everything people have ever done with terminals, far too many to go through here. You have helpers for asynchronous transfers between file descriptors with progress updates; arbitrarily many cursors; verifying if a certain unicode codepoint has a representation in the current set of fonts; spawning new processes with embed-able windows; window manager integration controls; per row attributes; notifications and much more. Then there are widgets for showing buffers, navigating lists of items and so on.

More Windows

Since there is no restriction on a single screen we can be more adamant about avoiding premature composition – the desktop and its window manager should naturally be responsible for such decisions.

To request a new window, you do this:

static bool on_window(struct tui_context *T,
                      arcan_tui_connection *C,
                      uint32_t id,
                      uint8_t type,
                      void *T)
{
/* we use a custom id to distinguish between multiple
 * request and windows pushed to us */
    if (id != 0xcafe)
        return false;

    struct tui_cbcfg cbcfg = {
/* fill in as needed */
    };

    struct tui_context* new = 
          arcan_tui_setup(T, C, &cbcfg, sizeof(cbcfg));

/* window allocation can always fail */
    if (!new)
       return false;

/* do something with new */
  return true;
}

/* set in initial handler table before context creation */
cbcfg.subwindow = on_window;

...

struct tui_subwnd_req req = {
 .hint = TUIWND_SPLIT_LEFT,
 .rows = 80,
 .cols = 25
};

arcan_tui_request_subwnd_ext(tui, TUI_WND_TUI,
                             0xcafe, req, sizeof(req));

As you can see it is very similar to how the original context was created. [new] should be added to the processing loop, or spawned into a thread with its own separate one.

Note: The context setup now uses a parent as a reference. This inherits all dynamic properties regarding locale, colours and text so there is a starting point for the implementation to work with.

Among the nuances is the HINT. This is a way to communicate to the outer display system how its window is intended to be used. Among the possible values are for setting it to a discrete tab, but also for embedding into other windows or be ‘swallowed’ (take the parents place until closed).

In this clip from Cat9 you can see that in two forms:

In the first form we take the contents of a previous job, request a new window and move the job output into it.

The second one is something more advanced. The requested type is actually HANDOVER, and we use the helper:

pid_t arcan_tui_handover(struct tui_context*,
                         arcan_tui_conn*,
                         const char *path,
                         char *const argv[],
                         char *const env[], int flags);

This helper creates a new process that inherits window ownership. The context can still be referenced as a proxy and used with the arcan_tui_wndhint function to reposition or reanchor.

This is useful for embedding as done in the clip, where we also get feedback on scaling and dimensions. This is where some of [shmif] bleeds through; if the new process is graphical and uses that library, such as the video player and PDF viewer in the clip, we can embed and even interact with graphical applications even though the TUI process itself has no concept of bitmap graphics.

In Lua, it would look like this:

wnd:new_window("handover",
    function(wnd, new)
        if not new then
           return
        end
        local in, out, err, pid = 
           wnd:phandover("/usr/bin/afsrv_terminal", "")
    end
)

Reading a line

Lets take a look at the common case: a command-line, i.e. our ‘libreadline’ replacement. This is a rather complicated affair since it involves text editing; suggestions; prompt; auto-completion; help information; keybinding controls; masking characters for password entry; searchable command history and so on.

Some of that you can see in this clip, also from Cat9.

The prompt is live updated (shown by the ticking clock), feedback on syntax/validation errors are highlighted, toggle-able extended help as to what a completion does and error messages explaining the validation failure.

#include <arcan_tui_readline.h>

...
struct tui_context *T;
/* context has been setup like before */
...

struct tui_readline_opts options = 
{
  .anchor_row = 1
  .allow_exit = true
};

arcan_tui_readline_setup(T, options, sizeof options);
int status;
char* out;

while (1){
    struct tui_process_res result =
      arcan_tui_process(&T, 1, NULL, 0, -1);

    status = arcan_tui_readline_finished(T, &out);
    if (status)
        break;
}

if (status == READLINE_STATUS_DONE){
/* do something with *out */
   free(out);
}

This swaps out relevant parts of whatever previous handler table you have for ones provided by the readline widget, and reverts it once finished. There are a lot of options to pick from when setting up the widget, including callback handlers for requesting history, validating current input, password masking, filtering undesired characters and so on.

For Lua, we can go with a more involved example:

local readline = 
  wnd:readline(
               function(_, message)
                   print("you entered", message)
               end
              )

readline:set_prompt("type something: ")
readline:set_history({"things", "from", "the", "past"})
readline:suggest({"some", "thing", "to", "add"})

while (wnd:process()) do
    wnd:refresh()
end

Networking

All this would be only marginally useful if it did not also work across the network. Remote administration and similar tasks are, of course, passport holding citizens in this world and even more work has gone into the networking protocol itself.

Say that you have written a new application using libarcan-tui and you want to share it with others or access it from another computer. From the developer angle, there is nothing to do, the existing tooling provides everything.

We will omit authentication key management here, since that is an big topic in its own right. That’s done with the ‘soft-auth’ argument for the pull apprach, and a12:// rather than a12s:// for the push one.

Speaking of push and pull, running something like:

arcan-net --soft-auth -l 6680 -- /path/to/my_thing

Would set the device up to serve an instance of my_thing to every inbound connection. This is what we mean with ‘pull’, the display end connects, gets served an application and pulls the output to its display. This is what you would be most familiar with through ssh.

Push flips the direction around and is what you would expect from X11 style remoting with something like DISPLAY=some.host:port xterm.

Here ‘my.other.host’ would be setup with arcan-net --soft-auth -l 6680. To push my_thing: ARCAN_CONNPATH="a12://my.other.device" /path/to/my_thing.

Both have its uses. Normally I have a number of devices fuzzing, doing traffic monitoring, or monkey testing with a core dump crash handler installed. All of these attach their corresponding TUI interaction tool and on-demand push to whichever device I am currently working on directly, or to a fallback multiplexing server that keeps them around until I check in.

What is interesting in the network case is how it differs from terminal times over the wire. This is nuanced with many factors in play as congestion control isn’t exactly XON/XOFF or the RS-232 RTS/CTS of yore. Is the priority interactive latency? memory consumption locally? memory consumption remotely? low network bandwidth?

To our disadvantage there is potentially more data per packed cell, but ZSTD compresses that very well. At the same time it is only the visible changes that are transferred, not some theatre piece with ASCII-character LOGO turtle on acid playing the lead. Any back-pressure propagates from the network tool to the source so pending updates merge there.

In a simple line-mode screen, ssh + your shell doing a ‘find /’ would start sending a colossal amount of data on the wire. Here the strategy is partly up to the shell.

In the following clip from Cat9, you can see that it detects a data heavy job and switches to a slow updating statistics view until it terminates or I instruct it to show what is going on. Importantly I can keep on working and I am not shoving 127MiB of data over the wire.

For an alt-screen tool, like vim and tmux, they instead get to pay the price for cursor movements to draw at non-continuous locations; style changes; border drawing characters and so on. Then suffer all the crazy things ncurses does (like padding or delaying writes to align with the baudrate of the connection). Then suffer the crazy things emulators have to do (no incoming data last n miliseconds, then it’s probably safe to draw). Then suffer the crazy things they themselves have to do to understand if a key is a key or a control character.

As per the old roadmap, 0.7 is the last window of opportunity for any trailing features. For 0.8 we finally get to pull out the performance tuning tricks, 0.9 for hardening the attack surfaces, to be followed by me disappearing in a puff of smoke.

I am also happy to say that we have received two new NLnet grants, one administrated by me, the other by Cipharius. Both of these build on the networking layer. For my part we’ll extend the directory server end to have a more flexible programmable remote side, for supporting linking multiple ones together and making the entire stack ‘turnkey’ easy to deploy and use.

In the other end there will be a simpler access layer for letting a device join your ‘one desktop, many devices’ network and extend it with its storage and sensors (cameras, displays, IMUs, and so on), or act as a portable viewer, Smash, for accessing it from within the confines of the corporate walled gardens.

I have often and, rightly so, been accused of not being a very public person. As an exemption to that rule I did partake in an interview by the venerable David Chisnall which can be read over at lobste.rs here: https://lobste.rs/s/w3zkxx/lobsters_interview_with_bjorn_stahl — if you are at all curious about my background and motivation.

Let’s briefly look at the building blocks in place:

IPC system – SHMIF.

Way out of terminal insanity – From The Dawn of a new Command Line Interface into The Day of a new Command-Line Interface: Shell.

Covering special needs – accessibility, debugging and security.

Networking – A12: Visions of the Fully Networked Desktop.

Legacy Strategy – SHMIF and A12 were both verified to ensure that anything that could be done (arcan vs Xorg part 1, part 2) through previous popular tools and protocols can still be done and persist onwards (missing – bringing X along).

All these latch into the scheme layed out in ‘Arcan as OS design‘.

Now we can braid these strands together into one rope, whip it into shape and see what should dangle from it.

This release doesn’t have many demonstrable items for the core engine itself, mostly fixes all across the board. Instead we will recap some changes and ongoing work for Lash#Cat9 and Xarcan. Our reference desktop environment, Durden, will get a separate release post when some of the tools has received more polish work.

As a teaser though, the following clip shows how the network tool can map in the key-local private file store and route it to our decode plumber:

In a similar vein, the following clip shows the same approach to picking a file from the directory store, setting it as the drag source and then dropping it into Cat9 and letting it expand it into a job view.

This act as the development baseline for more things, particularly letting controller appls running server-side define other namespaces, scan / index and annotate, distribute across a network of directories and search.

Other highlights for that release will be a new streaming/sharing composition tool; trainable mouse gestures; new initial setup configuration helper; desktop icons; on-screen keyboard; input typer helper with dictionary/IME completion; stereoscopic support for Xreal Air style glasses and more bespoke displays such as Looking Glass (lenticular) and Tilt5 (projected, retroreflective AR).

Lash#Cat9

Technically, Lash is a Lua scripting environment that is wrapper around our curses replacement, libarcan-tui, adding some convenience features to make it easier to write a command-line shell. Cat9 is the current reference shell for this.

The major features added to Cat9 over the last ~2 years since its inception has all received individual write ups:

In A spreadsheet and a debugger walk into a shell we cover a Debug Adapter Protocol Implementation as well as an interactive spreadsheet that can be populated by regular shell commands. The following two clips come from that article:

In Cat9 Microdosing: stash and list we cover a ‘ls’ replacement that effectively turns it into a file manager, and a scratchpad for accumulating files into a workset of files to monitor for changes, or forward to compression / transfer tools. The following two clips from that article:

In Cat9 Microdosing: each and contain we add the option to absorb the output of previous or current jobs into a collection that can then be referenced by other commands as a single unit, as well as defining asynchronous processing options over a range of data from other jobs (best used with the stash mentioned above). The following two clips come from that article:

The next planned such write-up contains a social timeline for things like mastodon and email timeline by frontending ‘madonctl’ and ‘himalaya’ as well as more developer tools like scm based integration. The following clip is a sneak peak from that:

Xarcan

In the last release post we hinted at how Xarcan can be used to keep your own window manager, letting Arcan act as a display driver — as well as a security, control and configuration plane that selectively merge in arcan native clients with options for how, or if, X clients gets to see inputs and clipboard action.

The following clip was used for that:

It’s almost as if what Adam Jackson said long ago rings true: (https://lists.freedesktop.org/archives/wayland-devel/2017-September/034960.html)

One thing I’ve never really been thrilled with about the Xwayland design is that the wayland compositor wants also to be the X window manager, and that all the related state about window position and whatnot is just internal to the compositor. xwin and xquartz don’t have this problem, you can run twm as your window manager and still move windows, but in xwayland that doesn’t work because wayland refuses to allow you to position your own window for (what I consider) essentially religious reasons [5]. And as a result the compositor gets a lot more complicated and you _still_ need to change the X server to get things to work. What I’d have preferred is a wl_x11_interop protocol that Xwayland could use to send this kind of advisory state, which mutter could optionally only expose to Xwayland if it really wanted to be less functional than X.

‘Essentially religious’ reasons indeed. What really happens is that Xarcan synchronises X11 ConfigureWindow and Stacking information with the corresponding VIEWPORT events.

This lets the Arcan side of the WM equation take the output video and decompose it into degenerate regions of linked coordinate spaces. It then uses these regions to determine if it should forward input or not.

For Arcan clients, it creates empty placeholder windows in X11 space, and swaps out its contents for the real one during scanout. It lets the X11 side know there is an object there, but prevents it for having access to its contents – unless desired.

That last part can be solved with controlled sharing. In the following clip you can see how I drag the contents of an Arcan window into it, letting legacy ‘screen sharing’ or ‘screenshotting’ (as flawed as those concepts are, but that is a long rant) applications see and capture it:

There is quite a lot more going on and planned:

An interesting coming change to arcan-shmif will have it re-use the ‘platform’ layer in Arcan that is responsible for managing input devices and controlling displays when there is no other arcan-shmif server to connect to.

This will effectively turn the Xarcan DDX into a standalone Xserver that works as expected, but at the same time keep up to date with features and changes to the KMS/GBM part of the graphics stack.

This lets us drop a whole lot of cruft from Xorg: XNest/Xephyr is no longer needed, neither is XAce or XFree86. We can eliminate quite a few slowdowns and pitfalls in startup.

Since it’s trivial for us to compartment and apply policy based on client origin, it will also be possible to partially share in the other direction, such as letting Xarcan act exclusively as an input-driver in order to bring xf86-input-wacom and XIM along.

Before then we have a few minor immediate cleanups left, mainly fixing regressions in GLAMOR caused by changes to DRI3 and PRESENT and some work-arounds for XDND to be able to work between multiple, networked, X servers. It’s not hard, just tedious.

Now it’s time to head off to 38c3. See you next year for the final reveal.

The write-up assumes a basic computer science background, while the sections prefixed with ‘comment’ are more advanced.

History

SHMIF, or “SHared Memory InterFace” has a long history, dating back to around 2007. It was first added to cover the need to least-privilege separate all parsing of untrusted data in the main engine, simply because the ffmpeg libraries couldn’t stop corrupting memory – generally a bad thing and we had more than enough of that from GPU drivers doing their part.

With parsers sandboxed, it evolved to also work as a linker interposed- or injected shellcode- way of manipulating 3rd party audio/video processing and event loop without getting caught doing so. Rumour has it that it was once used to automate a lot of the tedium in games such as World of Warcraft, and was not caught doing so.

It was written to be portable across many operating systems. The initial version ran on both Windows, OSX, BSDs and Linux. There were also non-public versions that ran on Android and iOS. These days the focus remains on BSDs and Linux, with the networking version of the same model, “A12”, intended to retain compatibility with the others.

Its design is based on lessons learned from emulating arcade games of yore, as they represent the most varied and complex display systems to date. The data model evolved from increasingly complex experiments, up to- and beyond- the point of painstakingly going through every single dispatch function in X11 to guarantee that we did not miss anything. The safety and recovery aspects come from many lessons learned breaking and fixing control systems for power grids. The debugging and performance choices came from working on a last-resort debugging tiger team on (mainly) Android.

Layout

There is a shared memory region, and a set of OS specific primitives to account for inadequacies in how various kernels expose controls over memory allocation and use. Combined we refer to these as a segment. The first one established is referred to as the ‘primary’ and it is the only one that is guaranteed on a successful connection. Additional ones are negotiable, and the default is to reject any new allocation. This decision is ultimately left to the window management policy.

Comment: In this form, there is no mechanism for a client to influence allocations in the server end. Further compromises are possible (as we come to later) in order to gain more features, but for a hardened setup, this is one out of several ways we reduce the options for exploiting any vulnerabilities or staging a denial of service attack.

The shared memory is split into a fixed static region and a dynamic one.

The following figure shows the rough contents of these regions:

The order of the fields in the static region is organic, it has been extended over time. To avoid breaking compatibility, changes have been appended as more metadata was needed. The region marked ‘aux’ is 0- sized by default; it is only used for negotiating advanced features e.g. HDR metadata and VR device support.

Some of the more relevant and non-obvious members of the static regions are:

• DMS – Dead Man’s Switch. If it is ever modified the segment is considered dead. After this point no modifications to the page will be processed by the other side. (See ‘Safety Measures’ section).
• Verification Cookie. This is a checksum comprised of calculating the offsets and values of other members in the region. Both sides periodically calculate and compare this value to detect version mismatches or corruption.
• Inbound/Outbound event buffers. – These are fixed slot ring buffers of 128b events. They can be thought of as asynchronous ‘system’ calls (See ‘Event Processing’ section).
• Segment Token. A unique identifier for this specific segment. This can be used by the client end to reference other events if the identifier has been shared by some other mechanism. The ‘VIEWPORT’ event, for instance, instructs window management for repositioning or embedding segments owned by other clients or threads.

The entire memory region is treated as an unsafe contested area; one side populates it with changes it wants to see done and through some synchronisation trigger, and the other side verifies and applies or rejects them.

Comment: For debugging and inspection, this means a single snapshot of the mapped memory range is sufficient to inspect the state of the connection and trivial to write analysis, fuzzing and reporting tools for.

The raw layout is not necessarily exposed to the consumer of the corresponding library. Instead a context structure (struct arcan_shmif_cont) contains the developer-relevant pointers to the corresponding subregions.

Comment: While the implementations for this interface live in userspace, the design intent was to be able to have the server end live completely in a kernel, and have this act as the sole system call interface.

Each segment has a type that is transferred once from the client to the server during the REGISTER event (or when requesting a new one through a SEGREQ event). This is mainly a window management and server hint to control event response, but also determines if video and audio buffers are for (default) client to server or (screen recording and similar features) server to client.

First Connection (Client)

The client end comes as a library, libarcan-shmif. The rough skeleton that we will unpack here looks like this.

#include <arcan_shmif.h>
int main(int argc, char **argv)
{
  struct arg_arr args;
  struct arcan_shmif_cont C =
    arcan_shmif_open(SEGID_APPLICATION,
                     SHMIF_ACQUIRE_FATALFAIL,
                     &args);

  struct arcan_shmif_initial *config;
  arcan_shmif_initial(&C, &config);

/* send audio/video */
/* event processing */

  arcan_shmif_drop(&C);
}

The SEGID_ part is a hint to the server as to the intended use of this connection and how it could manage its resource allocation and scheduling. There is a handful of types available, but APPLICATION is a safe generic one. A video player would be wise to use MEDIA (extremely sensitive to synchronisation but not input), while a game would use, well, GAME (high resource utilisation, input latency and most-recent presentation more important than “perfect” frames).

The FATALFAIL part simply marks that there is no point to continue if a connection can’t be negotiated. It saves some error checking and unifies 'fprintf(stderr, "Couldn't connect")' like shenanigans.

The arg_arr ‘args’ is a form of passing command line arguments to the client without breaking traditional getopt/argv. It can be used to check for key=value pairs through something like ‘if (arg_lookup(&args, "myopt", 0, &val)){ ... }‘ .

A good question here would be, how does the client know to find the server? The actual mechanism is OS dependent, but for the POSIX case there are two main options that the library is looking for: the ARCAN_CONNPATH and ARCAN_SOCKIN_FD environment variables. The value for CONNPATH is the name of a connection point and is defined by the server side.

Comment: The connection point name is semantic. This stands in contrast to how Xorg does with its DISPLAY=:[number] where number normally came from the virtual terminal the user was starting Xorg from. The server end can spawn multiple connection points with different names and apply different policies based on the name.

ARCAN_SOCKIN_FD is used to reference a file descriptor inherited into the process running arcan_shmif_open. This is used when the server itself spawns the new process. It is also used in the special case of ARCAN_CONNPATH being set to “a12://” or “a12s://”. This form actually starts arcan-net to initialise a network connection to a remote host, which creates a single-use shmif server for the client to bind to. This is one of the ways we translate from local IPC to network protocol.

The 'arcan_shmif_initial' part gives the information needed to create correct content for the first frame. This includes user preferred text format (size, hinting), output density (DPI aware drawing over ‘scaling’ idiocy), colour scheme (contrast for accessibility, colour blindness or light/dark) and even locale (to migrate away from LC_… /getlocale) and location (latitude/longitude/elevation).

Comment: For accelerated graphics it also contains a reference to the GPU device to use for rendering, this lets the server compartment or load-balance between multiple accelerators.

Now for the ‘send audio/video’ part.

shmif_pixel px = SHMIF_RGBA(0x00, 0xff, 0x00, 0xff);
for (size_t y = 0; y < C.h; y++)
  for (size_t x = 0; x < C.w; x++)
    C.vidp[Y * C.pitch + x] = px;

arcan_shmif_signal(C, SHMIF_SIGVID);

This fills the dynamic video buffer part of the segment with a full opaque green pixel in linear RGB in whatever packing format the system considers native (embedded in the SHMIF_RGBA macro).

Comment: While on most systems (these days) that would be 32-bit RGBA, it is treated as compile time native as CPU endianness would be. Low-end embedded might want RGB565, special devices like eInk might want RGB800 and so on. 

There are a lot of options available here, but most have to deal with special synchronisation or buffering needs. These are covered in the ‘Special Case’ sections on Synchronisation and on Accelerated Graphics.

For this example we omitted the aural representation, but if you have a synthesizer core or even tone-generator the same pattern apply; switch vidp for audp and SHMIF_SIGVID for SHMIF_SIGAUD (it is a bitmask, use both if you have both).

Comment: The common distinction between audio and video is something we strongly oppose. It causes needless complexity and suffering trying to have one IPC system for audio, then another for video and then trying to repair and synchronise  the two after the fact. It is one of those historical mistakes that should have ended yesterday, but the state of audio on most systems is almost as bad as video.

At this stage we are already done (13 lines of code, zero need for error handling) but for something more polite, we will flesh out the ‘event processing’ part.

struct arcan_event ev;
while (arcan_shmif_wait(&C, &ev)){
 if (ev.category == EVENT_IO){
/* mouse, keyboard, eyetracker, ... handling goes here */
 }

 switch(ev.tgt.kind){
 case TARGET_COMMAND_EXIT:
/* any custom cleanup goes here */
 break;
 case TARGET_COMMAND_RESET:
/* assume that we are back where we started */
 default:
 break;
 }
}

This will block until an event is received, though more options are covered in the section on ‘Synchronisation’. No action is ever expected of us, we just get polite suggestions ‘it would be nice if you do something about this’. The category part will only be EVENT_IO or EVENT_TARGET and the next section will dip into why.

Comment: The _RESET event in particular is interesting and will be covered in the 'Recovery' Special Case. It can be initiated by the outer desktop for whatever reason, and just suggest 'go back to whatever your starting state was, I have forgotten everything' but is also used if the server has crashed and the implementation recovered from it, or is shutting down and have already handed responsibilities over to another.

The event data model cover 32 different server to client possibilities, and 22 client to server. Together they cover everything needed for a full desktop and more, but it is descriptive, not normative. React to the ones relevant to you, ignore the others.

First Connection (Server)

There are two implementations for the server end; one inside the arcan codebase tailored to work better with its more advanced resource management, crash resiliency and scripting runtime. The other comes as a library, libarcan-shmif-server, and is mainly used by the arcan-net networking tool which translates this into the A12 network protocol.

Let’s walk through a short example which accepts a single client connection, and in the next section do the same thing for the client application end. Normal C foreplay is omitted for brevity.

#include <arcan_shmif.h>
#include <arcan_shmif_server.h>

struct shmifsrv_client *cl = 
  shmifsrv_allocate_connpoint("demo", NULL, S_IRWXU, fd);

shmifsrv_monotonic_rebase();

This creates a connection point for a client to bind to. There are two alternatives, shmifsrv_spawn_client and shmifsrv_inherit_connection. Spawn takes care of creating a process with the primitives inside. Inherit takes some preexisting primitive and builds from there. Both return the same shmifsrv_client structure.

Comment: For a controlled embedded or custom OS setting, the spawn client approach is the safest bet. The inherit connection approach is for when there is a delegate responsible for spawning processes and reduces the number of system calls needed to a bare minimum.

The shmifsrv_monotonic_rebase() call sets up the internal timekeeping necessary to provide a coarse (25Hz) grained CLK (clock) signal.

Now we need some processing that interleaves with rendering/input processing loops, which is a larger topic and out of scope here.

int status;
while ((status = shmifsrv_poll(cl) <= CLIENT_NOT_READY)
{
/* event and buffer processing goes here */
}

if (status == CLIENT_DEAD)
{
  shmifsrv_free(cl, SHMIFSRV_FREE_FULL);
  exit(EXIT_SUCCESS);
}

It is possible to extract an OS specific identifier for I/O multiplexing so that _poll is only invoked when there is some signalled inbound data via shmifsrv_client_handle().

Comment: The flag passed to _free determines the client side visibility. It is possible to just free the server side resources and not signal the dead man's switch. This can be used to transparently pass the client to another shmif server process or instance.

Before we get to the event and buffer processing part, there is also some timekeeping that should be managed outside of a higher frequency render loop.

int left;
int ticks = shmifsrv_monotonic_tick(&left);
while (ticks--){
  shmifsrv_tick(cl);
}

This provides both integrity and liveness checks and manages client requested timers. The (left) returns the number of milliseconds until the next tick. This is used as feedback for a more advanced scheduler, if you have one (and you should).

Now to the event processing:

struct arcan_event buf[64];
size_t n_events, index = 0;

if ((n_events = shmifsrv_dequeue_events(cl, buf, 64)){
  while (index != n_events){
    struct arcan_event* ev = &buf[index++];
    if (shmifsrv_process_event(cl, ev))
      continue;

/* event handlers go here */
  }
}

This will dequeue, at most, 64 events into the buffer. Each event is forwarded back into the library in order to allow a subset of internally managed ones. They are just routed through the developer to allow complete visibility. You can use arcan_shmif_eventstr() to get a human readable representation of its contents.

Comment: The reason for having a limit is that a clever and malicious client could set things up in a way that would race to stall the server or exhaust its file descriptor space as part of a denial of service, either to affect the user directly or as part of trying to make an exploitation chain more robust.

Now for the ‘event processing’ part.

if (ev->category != EVENT_EXTERNAL){
  fprintf(stderr, "unexpected event category\n");
  continue;
}

switch (ev->ext.kind){
case EVENT_EXTERNAL_REGISTER:
/* only allow this once on the client */
  arcan_shmifsrv_enqueue(cl, &(struct arcan_event){
   .category = TARGET_COMMAND,
   .tgt = {
     .kind = TARGET_COMMAND_ACTIVATE
   }
  }
break;
default:
break;
}

The event data model has a lot of display server specific nuances to it, neither is necessary except for the one above. This unlocks the client from the ‘preroll’ state where it accumulates information received into the “arcan_shmif_initial” structure as covered in the client section. Any information necessary for a client to produce a correct first frame goes before the ‘ACTIVATE’ one. The most likely ones you want is DISPLAYHINT, OUTPUTHINT and FONTHINT to instruct the client about the size it will be scaled to, the density, colourspace and subchannel layout it will be presented through, as well as the preferred size of the most important of primitives, text.

Comment: There are a number of event categories, but only one reserved for clients (EVENT_EXTERNAL). The other categories are for display server internals. The reason they are exposed over SHMIF is for the 'server' end to be split across many processes and still interleave with event processing in the server. This allows us to have external sensors, input drivers etc. all as discrete threads or processes without changing anything else in the architecture. It also allows a transformation from using it as a kernel-userspace boundary to a microkernel form.

The last part is to deal with ‘buffer processing’ part of the previous code.

/* separate function */
bool audio_cb(shmif_asample *buf,
              size_t n_samples,
              unsigned channels, unsigned rate,
              void *tag)
{
  /* forward buf[n_samples] to audio device or mixer
   * configured to handle [channels] at [rate] */
  return true;
}

if (status & CLIENT_VBUFFER_READY){
  struct shmifsrv_vbuffer vbuf = shmifsrv_video(cl);
  /* forward vbuf content to GPU */
  shmifsrv_video_step(cl);
}

if (status & CLIENT_ABUFFER_READY){
  shmifsrv_audio(cl, audio_cb, NULL);
}

The contents of vbuf is nuanced. There is a raw buffer or opaque GPU system handle + metadata (timing, dirty regions, …), or TPACK (see section on ‘Text Only Windows’) and a set of flags corresponding to the ‘presentation-hints’ on how buffer contents should be interpreted concerning coordinate system, alpha blending and so on.

Comment: Most of the graphics processing properties are things any competent scanout engine has hardware acceleration for, excluding TPACK (and even ancient graphics adaptors used to have those as 'text mode' display resolution). There are support functions to unpack this into a compact list of text lines and their colouring and formatting in "arcan_tui.h" as arcan_tui_tunpack().

Comment: For accelerated GPU handles it is possible to refuse it by sending a BUFFERFAIL event. This will force the client implementation to convert accelerated GPU-local content into the shared pixel format on their end. This is further covered in 'Special Case: Accelerated Graphics'. It doubles as a security measure, preventing the client from submitting command buffers to the GPU that will never finish and livelock composition that way (or leverage any of the many vulnerabilities GPU side). On a hardened system this would be used in tandem with IO-MMU isolation.

In total this lands us with less than 100 lines of code with very granular controls, a fraction of what other systems need to just boilerplate graphics alone. If you only want a 101 level – take on how SHMIF works, we are done; there is a lot more to it if the topic fascinates you, but it gets more difficult from here.

Synchronisation

While one might be tempted to think that ‘display servers’ are about well, providing access to the display, its real job is actually desktop IPC with soft realtime constraints. The bread and butter for such systems is synchronisation. If you fail to realise this you are in for a world of hurt and dealing with it after the fact will brew a storm of complexity.

Comment: It is also the hardest problem in the space - figuring out who among many stakeholders knows what; when do they know it; when is something said relevant or dated. All of those are difficult as is, but it gets much worse when you also need to factor in resonance effects, malicious influence and that some stake holders reason asynchronously about some things, and synchronously about others. As icing on an already fattening cake you need to weigh in the domain specific (audio / video) nuances. Troubleshooting boils down to profiling, and problems manifest as 'jitter' and 'judder' and how those fit with human cognition. Virtual Reality is a good testing ground here even if you are otherwise uninterested in the space.

Comment: beginner mistakes here are fairly easy to spot; if someone responds to synchronisation problems by increasing buffer sizes (landing in a version of the network engineering famous 'buffer bloat' problem) or arbitrary sleep calls (even though some might be necessary without adequate kernel level primitives) they are only shifting the problem around.

Recommended study here is ‘queuing theory’ and ‘signal processing’ for a deeper understanding.

To revisit the previous code examples on the client end:

arcan_shmif_signal(&C, SHMIF_SIGVID);

This is synchronous and blocking. The thread will not continue execution until the server end has said it is ok (the shmifsrv_video_step code). The server can defer this in order to prioritise other clients or to stagger releases to mitigate the ‘thundering herd’ problem.

For normal applications, this is often sufficient and comparable to ‘VSYNC’. When you have tighter latency requirements and/or it is costly to produce a frame, you need something more. The historically ‘easier’ solution has been to just add another buffer:

arcan_shmif_resize_ext(&C, C.w, C.h,
                       (struct shmif_resize_ext){
                         .vbuf_cnt = 2
                       });

_resize and _resize_ext calls are both also synchronous and blocking. This is because the server end needs the controls to guarantee that enough memory is available and permitted. It will recalculate all the buffer offsets (vidp, audp, …) and verification cookie in the context and possibly move the base address around to satisfy virtual or physical memory constraints.

Comment: Some accelerated display scanout controllers have hard requirements on physically continuous memory at fixed linear addresses and those are a limited and scarce resource and why such resize request might fail, especially in tight embedded development settings. Same applies when dealing with virtual GPUs in virtual machines and so on. The other option to still satisfy a request is to buffer in the server end, causing an extra copy with increased power consumption and less memory bandwidth available for other uses.

The request above would make the first arcan_shmif_signal call return immediately, and only block if another signal call happens before the server is able to consume the buffer from the first. Otherwise the context pointer (C.vidp) will be changed to point to the new free buffer slot. This also has the added cost of adding another display refresh period of latency.

Comment: It is possible to increase the buffer count even further, but this changes the semantics to indicate that only the most recently submitted buffer should be considered and others can be discarded. This counters the latency problem of the double buffering problem at the expense of memory consumption. This has historically been called 'triple buffering' but, to much confusion, has also been used for the 'double buffering' behaviour with just deeper buffer queues and is thus meaningless.

Not every part of a buffer might have actually changed. A common optimisation is to annotate which region that should be considered, and for regular UI applications (blinking cursor, single widget updates, …) this substantially cuts down on memory transfers. To cover this you can mark such regions with calls to arcan_shmif_dirty() prior to signalling.

Comment: While some might be tempted to annotate every pixel, there are strong diminishing returns as soon as you go above just one region due to constraints on memory transfers. Internally the shmif client library implementation will just merge multiple calls to _dirty into the extents of all changes. For the triple buffering behaviour mentioned in the previous comment, dirty regions won't have any effect at all as changes only present in the one buffer would not guarantee to transfer over in the next and the cost of trying to merge them on the composition end would cancel out the savings in the first place.

There are more synchronisation nuances to cover, but to avoid making this section even more exhausting, we will stick to the two most relevant. The first of these look like this:

arcan_shmif_signal(&C, SHMIF_SIGVID | SHMIF_SIGBLK_NONE);

This returns immediately and you can chose to check if it is safe to draw into the video buffer yourself (through arcan_shmif_signalstatus), or to simply continue writing into the buffer and risk ‘tearing’ in favour of lower latency. This amounts to what is commonly called ‘Disable VSYNC’ in games.

Comment: For those that have written games in days of yore, you might also recall 'chasing the beam', unless dementia has taken over by now. Since game rendering can have predictable write patterns and display scanout can have predictable read patterns it is possible to align your writes such that you raster lines up to just before the one the display is currently reading. This is neither true for modern rendering nor is it true for modern displays, 'the framebuffer' is a lie. Still, for emulation of old systems, it is possible, but impractical, to repeatedly access the 'vpts' field of the static region to figure out how many milliseconds are left until the next VBLANK and stage your rendering accordingly.

The last option is to to keep the SHMIF_SIGBLK_NONE behaviour, but adding the flag SHMIF_RHINT_VSIGNAL_EV to C.hints prior to a _resize call. This will provide you with a TARGET_COMMAND_STEPFRAME event and you can latch your rendering to that one alone and let your event loop block entirely.

Comment: Enabling STEPFRAME event delivery by sending a CLOCKREQ request provides a secondary path for advanced latency management as it enables feedback in presentation timing. Sampling the 'vpts' field of the static region would provide information about upcoming deadline and STEPFRAME contains metadata about presentation timing as to when the contents was actually presented to scale up/down effects and quality. Current game development is full of these kinds of tricks.

Event Processing

Event loop construction is an interesting and nuanced topic. We start by returning to the naive one introduced in the client section:

struct arcan_event ev;
while (arcan_shmif_wait(&C, &ev)){
/* interpretation goes here */
}

Responding to each event that arrives here should be as fast as possible. This is easy most of the time. Whenever a response includes rendering however, response times can vary by a lot. Some events are more prone to this, with mouse motion and resize requests being common offenders.

What happens then is that the number of events in the incoming queue starts to grow. If the rate of dispatched events is lower than that of the incoming one, we get buffer back-pressure.

This applies to both the server and the client side. Each side has different constraints and call for different mitigation. The server end is more vulnerable here as it has multiple clients to process, and higher costs for processing events as most prompt some form of managerial decision.

Comment: Events from client A can directly or indirectly cause a larger number of responses from clients B and C (amplification), which in turn can cascade into further responses from A. This can snowball fast into 'event storms' and back-pressure building up in others as 'resonance effects'.

One small change to the loop improves on the client end of the equation:

bool process_event(struct arcan_event *ev)
{
/* interpretation now goes here */
}

struct arcan_event ev;

while (arcan_shmif_wait(&C, &ev)){
  bool dirty = process_event(&ev);
  size_t cap = PP_QUEUE_SIZE;

  while (arcan_shmif_poll(&C, &ev) > 0 && cap--){
    dirty |= process_event(&ev);
  }

  if (dirty){
    render();
  }
}

This will flush out as much of the inbound queue (or up to a cap corresponding to the size of the ring buffer) as possible, and only render after all have been applied. This prevents earlier events in the queue from being overdrawn by responses to later ones in the queue.

Comment: Since the event queue is a ring-buffer visible to both sides, it is possible for either party to atomically inspect the head and tail values to determine the current state of the other end, as well as incoming workload. This is a powerful advantage over other possible carriers, e.g. sockets.

The full data model would take another lengthy post to flesh out, so we will only look at one event which highlights library internals. That event is ‘TARGET_COMMAND_DISPLAYHINT’. This event is used to indicate the size that the server end would prefer the window to have. The client is free to respond to this by resizing to the corresponding dimensions. If it doesn’t, the server can still scale and post process – it has the final say on the matter.

As mentioned earlier, resize is synchronous and blocking due to its systemic cost so it makes sense to keep them at a minimum. Some of that responsibility falls on the window manager to ensure that a drag resize using a 2kHz mouse doesn’t also result in 2000 DISPLAYHINTs. Even if that would happen, the implementation has another trick up its sleve.

There is a small number of events which are considered costly and can be coalesced. When _wait or _poll encounters such an event, it sweeps the entire pending queue looking for similar ones, merging them together into the one eventually returned, providing only the most recent state.

Comment: There is a tendency for IPC systems to be designed as generally as possible, even if their actual narrow use is known. This defers the decision to commit to any one domain specific data model, making optimisations such as this one impossible -- you can't coalesce or discard what you don't know or understand, at least not responsibly with predictable outcome. This does not make complexity go away, to the contrary, now every consumer has increased responsibility to manage queuing elsewhere. The problem doesn't magically disappear just because you have an XML based code generator or some-such nonsense.

Most of this article has been about a single segment, though in ‘rich’ applications you would have more: child windows, popups, custom mouse cursors and so on. We already mentioned it is possible to request more, even though only one is ever guaranteed. This is not more difficult than setting up the primary one. You submit a SEGREQ event with a custom identifier and type. Eventually you either get a NEWSEG event or REQFAIL event back with the same identifier. For the NEWSEG you forward the event structure to arcan_shmif_acquire and you get a new arcan_shmif_cont structure back.

What does this have to do with queue management? Well, each new segment has their own separate queues and each segment can be processed and rendered on separate threads independent of each other. There is a monotonic per-client global counter and timestamp as part of each event to account for ordering requirements between ‘windows’, but in practice those are exceedingly rare.

A final part about the events themselves. SHMIF is an IPC system, it is not a protocol, it doesn’t cross device boundaries. We have a separate and decoupled networking protocol specifically for that. As an IPC system we can and should take advantage of device and operating system specific nuances.

Two such details is that each event has a fixed size, 128 bytes (64 did not cover all cases) which amounts to 2 cache lines for the vast majority of architectures out there. They are in linear continuous buffers at a native aligned base with access patterns that prefetch very well. The packing of different fields is tied to the system ABI which is designed to be optimal for whatever you are running on.

Safety Measures

We are almost done with the overall walkthrough, then we can finish off with some special cases and exotic features. Before then there is one caveat to cover.

In previous sections we have brushed upon a few tactics that protect against misuse; the validation cookie to detect corruption, version and code generation mismatches as well as the dead man’s switch. There is still one glaring issue from shared memory event management and audio/video signalling solution: what happens if the other end livelocks or crashes while we are locked waiting for a response?

In a socket based setup the common ‘solution’ for a crash in the other end would cause it to detach and you can detect that. For a live lock it is to resort to a kind of ping-pong protocol and somehow disambiguate between that and a natural stall for some other part of the system (very often, GPU driver).

By default (there is a flag to disable this) each segment gets a guard thread. This guard thread periodically (default: every second) checks the aliveness of a monitoring process identifier that the server filled out, as well as if the dead man’s switch has been released. If that happens, it immediately unlocks all internal semaphores causing any locked call into the shmif library to release and any further calls to error out so the natural event loop takes hold. This setup is also used to not only detect- but recover from- crashes (see ‘Special Case: Recovery and Migration’).

This might not be enough for troubleshooting or even communicating to a user that something is wrong. For this we have the ‘last_words’ part of the memory region. This is an area the client can fill out with a human presentable error message that the server end can forward to relevant stakeholders.

The Arcan engine itself splits out into two parts. One potentially privileged parent supervision process that is used to negotiate device access, and the main engine. This supervision process also acts as a watchdog. Every time the engine enters and exits a dangerous area, e.g. the graphics platform or the scripting VM, it registers a timestamp with the parent. If this exceeds some threshold, the parent first signals the engine to try and gracefully recover (the scripting VM is able to do that) and if the problem persists, shuts down the engine. This will trigger the guard threads inside clients and they, in turn, enter a reconnect, migrate or shutdown state.

Special Case: Handover Allocation

As we covered previously, requesting a new segment to use as a window takes a type that indicates its role and purpose. One such type that sticks out is ‘SEGID_HANDOVER’. This means that the real type will be provided later and that the segment will be handed over to a new client.

To better illustrate, let’s take a code example:

arcan_shmif_enqueue(&C, &(struct arcan_event){
  .category = EVENT_EXTERNAL,
  .ext.kind = ARCAN_EVENT(SEGREQ),
  .ext.segreq.kind = SEGID_HANDOVER
});

...

uint32_t new_token;

/* in event handler, 'ev' being the inbound event */
case TARGET_COMMAND_NEWSEGMENT:
  arcan_shmif_handover_exec(&C, ev,
                            "/path/to/something",
                            argvv, envv,
                            0 /* detach-options */);  
  new_token = ev.ext.ioevs[4].uiv;
break;

This would launch “/path/to/something” so that when it calls arcan_shmif_open it will actually use the segment we received in order to connect. We can then use new_token in other events to manage some of it, e.g. reposition its windows, inject input and more. All of this retains the chain of trust: the server end knows who invited the new client in and can treat it accordingly.

This can be used to embed other clients into your own window, to build an external window manager and so on. In our ‘command lines without terminal emulation’ shell, Lash#Cat9, we use that to manage graphical clients while still being ‘text only’ ourselves. For other examples, see the article on Another Low Level Arcan Client: A Trayicon Handler.

Special Case: Recovery and Migration

A key feature of SHMIF is that it can redirect and reconnect clients manually. Through this we can even transition back and forth between local and networked operations. The section on ‘Safety Measures’ covered how it works in SHMIF internals. There is also an article on ‘Crash Resilient Wayland Compositing’ (2017) that demonstrates this.

When a client connects, the library enqueues a ‘REGISTER’ event that contains a generated UUID. This can be leveraged by the window manager to persist location on the desktop and so on. At any stage it can also send a ‘DEVICEHINT’ event back.

This event is used to provide an opaque handle for GPU access in operating systems which requires that (which can further be used to load balance between multiple GPUs), but it can also mention a ‘fallback connection point’. Should the server end die (or pretend that it has died), the library will try to connect to that connection point instead.

If it is successful, it will inject the ‘TARGET_COMMAND_RESET’ event that we covered earlier. We will use the following clip from “A12: Visions of the Fully Networked Desktop“. as a starting point.

In it, you see Lash#Cat9 CLI shell inside the ‘Durden’ Window manager having a video clip as an embedded handover allocation. This has previously used the local discovery feature of the network protocol to detect that the tablet in front of the screen (a Surface Go) is available as a sink and has added it as an icon in the statusbar — unfortunately occluded by the monitor bezel in the clip.

When I drag the window and drop it on that icon, Arcan sends a DEVICEHINT with the connection primitive needed for the network embedded into the event. It then pulls the dead man’s switch, forcing the shmif library to go into recover. Since it remembers the connection from the DEVICEHINT, it reconnects and rebuilds itself there.

This feature is not only leveraged for network migration as shown, but also for compartmentalisation between multiple instances; for crash recovery; for driver upgrades and for upgrading the display server itself. All using the same code paths.

Special Case: Accelerated Graphics

Many ‘modern’ clients unfortunately have a hard dependency to a GPU, and unfortunately the mechanisms for binding accelerated graphics between display server and client are anything but portable.

Comment: Khronos (of OpenGL and Vulkan fame) tried to define a solution of their own (OpenWF) that failed miserably. What happened instead is even worse; the compromise once made for embedded systems, 'EGL' got monkey patched with a few extensions that practically undoes near all of its original design and purpose, and it is suddenly what most are stuck with.

There is a lot of bad blood and vitriol on the subject that we will omit here and just focus on the SHMIF provided interface. Recall the normal way of starting a SHMIF client:

#include <arcan_shmif.h>
int main(int argc, char **argv)
{
  struct arg_arr args;
  struct arcan_shmif_cont C =
    arcan_shmif_open(SEGID_APPLICATION,
                     SHMIF_ACQUIRE_FATALFAIL,
                     &args);
}

This does still apply. A client is always expected to start a normal connection first, and then try to bootstrap that to accelerated, which can fail. The reasoning for that is iff permissions or GPU driver problems stop us from providing an accelerated connection, the regular one can still be used to communicate that to the user rather than have them dig through trace outputs for the answer.

To extend a context to being accelerated you can do something like this:

struct arcan_shmifext_setup cfg = {
  .api = API_OPENGL,
  .major = 4,
  .minor = 2
/* other options go here */
};

int status = arcan_shmifext_setup(&C, &cfg);
if (status != SHMIFEXT_OK){
/* configuration couldn't be filled */
}

There are a number of options to provide in the config that requires some background with OpenGL etc. to make any sense so we skip those. If you know, you know and if you don’t, enjoy the bliss. If the setup is OK, the passed ‘cfg’ is modified to return the negotiated values, which might be slightly different than what you requested.

Afterwards, you can then continue with arcan_shmifext_lookup() to extract the function pointers to the parts of the requested API that you need to use, bound to the driver end of the created context.

When writing platform backends to existing applications, they often provide their own way of doing all this and we do need a way to work with that. If there already is a context living in your process and you want to manually export and forward a resource, it is possible through:

size_t n_planes = 4;
struct shmifext_buffer_plane planes[n_planes];
n_planes = arcan_shmifext_export_image(&C, (uintptr_t) MyDisplay, (uintptr_t) my_texture_id, n_planes, planes);

if (n_planes){
  arcan_shmifext_signal_planes(&C,
                               SHMIF_SIGVID,
                               n_planes, planes);
}

There are a several more support functions with similar patterns for context management, importing already exported images and so on, but this should be enough to get an idea of the process.

Special Case: Debugging and Accessibility

We have already shown how the client end can request new segments through SEGREQ events and how those are provided through NEWSEGMENT events coming back. Another nuance to this is that the server end can push a NEWSEGMENT without having to wait for a SEGREQ in advance.

This can be used to probe for support for things, such as custom client mouse cursors, or to signal a paste or drag and drop action (clipboard is just yet another segment being pushed), as the server end will know if the client mapped the segment or not.

There is nothing stopping us from actually mapping and populating the segment from within libarcan-shmif, and there are two cases where that is actually done. One is for SEGID_DEBUG and another for SEGID_ACCESSIBILITY.

There are two longer articles related to how this works in more depth, one on ‘Leveraging the Display Server to Improve Debugging’ and another on ‘Accessible Arcan: Out of Sight’.

If one of these are received, libarcan-shmif will (unless the feature is compiled out) internally spawn a new thread. In the debugging case it will provide a text interface for attaching a debugger, for exploring open files, inspecting environment and memory from within the process itself. In the accessibility case it will latch on to frame delivery (SHMIF_SIGVID) in order to overlay text descriptions that gets clocked to the video frames being delivered and the dirty regions being updated.

Special Case: Text-only Windows

There are more reasons as to why a set of fonts and desired font size is provided during the preroll state and why there is a ‘text rows and columns’ field in the static region.

Among the hints that can be set for the video region, there is SHMIF_RHINT_TPACK. This changes the interpretation of the contents of the video buffer region to use a packing format (TPACK) which is basically a few bytes of headers and then a number of rows where each row has a header covering how it should be processed (shaping, right-to-left) along with a number of cells with the formatting, colour and possible font local glyph indices (for ligature substitutions).

The format is complete enough to draw anything a terminal emulator would be able to throw at it, but also do things the terminal emulator can’t, such as annotation layers or borders that fall outside of the ‘grid’, saving precious space.

This approach lets the server end handle the complex task of rendering text. It also means that the costly glyph caches, GPU related acceleration primitives like distance fields and so on can all be shared between windows. It means that the server can apply the heuristic for ‘chasing the beam’ style minimal latency or tailor updates to the idiosyncrasy of eInk displays when appropriate, and that the actual colours used will fit with the overall visual theme of the desktop while letting the client focus on providing ‘just text’.

While it is possible to build these yourself, there is a higher level abstraction support library, ‘libarcan-tui’ for that purpose. The details of that, however, is a story for another time.

Recently the stars aligned and I had some time over to work on the accessibility story, particularly on the ‘no vision’ parts. Other forms (e.g. limited mobility, low vision) will be covered in future articles but most things are already in place, including eye tracking and bespoke devices like stream decks.

Here is a short recording of the first run of a clean installation, setting things up.

This is enabled by default during first setup. The first question presented will be to disable it, but there is no hidden trapdoor combination or setting to enable it for the first time.

The following recording shows using the builtin menu system to start a terminal emulator and open a PDF and the mouse cursor to navigate, with OCR results as I go. These will be elaborated on further in this article.

There is a previous higher-level article which covered a rougher outline of what is intended, but this one is more specific about work that can be used today — albeit with some rough corners still.

One detail to take away from that article is how the architecture splits user data processing into specific one-purpose replaceable programs. These can be isolated and restricted to a much stronger degree than a more generic application.

These are referred to as frameservers. They have a role (archetype) and the ones of interest here are decode and encode. Decode translates from a computer native representative to a human presentable one, like image loading into a pixel soup or annotated text into sound via synthesised speech. Encode goes from potentially lossy computer to human translation such as pixel soup to text via OCR, image description or transcribing audio.

Another detail is that the “screen reader” here is not the traditionally detached component that tries to stitch narration together through multiple sidebands. Instead, it is an “always present first class mechanism” that the window manager should leverage. There is no disconnect between how we provide visual information from aural and they naturally blend with the extensive network transparency as part of our ‘many devices, one desktop’ design target.

Key Features

Here is a short list of the things currently in place:

• Multiple simultaneous positional voice profiles for different types of information
• On-Demand Client requested Accessibility windows
• Force-injected client accessibility fallback with default OCR latched to frame updates
• Buffered Text Input with lookup oracle
• All desktop interaction controls as a file-system
• Command-line shell aware
• Keyboard controlled OCR
• Content aware mouse- audio feedback
• Special controls for text-only windows tracking changes

There is a lot more planned or in progress:

• Premade bootable live / VM image and generator
• Bindings to additional client support (AccessKit)
• Compatibility preset profiles to help transition from NVDA/Orca
• Extend accessibility fallback with description oracles (LLM:offline, …)
• Extend speech process format with waveform/tone/samples for formatting
• Extended lookup oracle
• Stronger language controls/toggles
• Scope based ‘few-line’ text editor extension to shell
• Haptic support
• Braille Terminal output (lacks hardware access currently)
• Indexer for extracting or generating alt-text descriptions

Screen Reading Basics

Let’s unpack some of what happens during setup.

The TTS tool in Durden uses one or many speech profiles that can be found in the durden/devmaps/tts folder. Each profile describe one voice, how its speech synthesis should work, which kinds of information it should convey and any input controls.

They allow for multiple voices to carry different information at different positions to form a soundscape, so you can use ‘clean’ voices for important notifications and ‘fast robotic’ for dense information where actions like flush pending speech buffer doesn’t accidentally cancel out something important.

A compact form starts something like this:

model = "English (Great Britain)",
gain = 1.0, gap = 10, pitch = 60, rate = 180, range = 60, channel = "l", name = "basic_eng", punct = 1, cappitch = 5000

These are just the basic voice synthesis parameters one would expect. Then it gets better.

actions = {
 select = {"title", "title_text"},
 menu = "menu",
 clipboard = "clip",
 clipboard_paste = "clip-paste",
 notification = "notify"
}

bindings = {
 m1_r = "/global/tools/tts/voices/basic_eng/flush",
 m1_t = "/global/tools/tts/voices/basic_eng/slow_replay"
}

These tell what this voice gets to do. The action keys mark which subsystems it should jack into, and some custom prefix announcement as value. The profile above would present all menu navigation, system notifications, clipboard access and window title on selection.

The bindings are keyboard binding overlays that take priority when the voice is active. Just like any other binding, they map to paths in the virtual filesystem that Durden is structured around. The two examples shown cancels all pending speech for the specific voice, or turns down the speech rate to low and repeats the last message then returns it back to the voice default. There are, of course, many others to chose from including things like key echo and so on.

Holding down any of the meta or accessibility bound buttons for a few seconds without activating a specific keybinding will play the current ones back for you to ease learning or refresh your memory.

By adding a position = {-10, 0, -10} attribute to the profile the system switches to 3D positional audio and, in this example, positions the voice to your back left. This feature also introduced the /target/audio/position=x,y,z which lets you move any audio from the selected window to a specific position around you, along with /target/audio/move=x,y,z,dt which would slide the voice around over time.

With an event trigger, e.g. /target/triggers/select/add=/target/audio/position=0,0,0 and /target/triggers/deselect/add=/target/audio/position=10,0,-10 the soundscape around you also match window management state itself.

The cursor profile has a few more things to it:

cursor = {
  alt_text = "over ",
  xy_beep = {65.41, 523.25},
  xy_beep_tone = "sine",
  xy_tuitone = "square",
  xy_tuitone_empty = "triangle"
  ...
}

The alt_text option would read any tagged UI elements with their text description. XY beeps specifies the frequency range to map the mouse cursor coordinates based on their screen position so that the pitch and gain changes as you slide it across the screen.

The waveform used to generate the wave will also change with the type of contents the cursor is over, so that text windows such as terminal emulators will get a distinct tone and distinguishes between empty and populated cells.

The following clip shows navigating over UI elements, an browser window and a terminal window. You can also hear how the ‘select text to copy to clipboard’ doubles as a more reliable means of hearing text contents in uncooperative windows.

There are also more experimental parts to the cursor, such as using a GPU preprocessing stage to attenuate edge features and then convert a custom region beneath the cursor into sounds. While it takes some training to decipher, this is another form of seeing with sound and applies to any graphical content, including webcam feeds. After some hours I (barely) managed to play some graphical adventure games with it.

Kicking it up a notch

Time to bring out the spice weasel and get back to the OCR part I briefly mentioned earlier.

The astute reader of this blog will recall the post on Leveraging the “Display Server” to Improve Debugging. A main point of that is that the IPC system is designed such that the window manager can push a typed data container (window) and the client can map and populate it. If a client doesn’t, there is an implementation that comes along with the IPC library. That is not only true for the debug type but for the accessibility one as well.

This means that with the push of a button we can probe the accessibility support for a window, and if none exists, substitute our own. This is partly intended to provide support for AccessKit which will complete the solution with cooperative navigation of the data model of an application.

The current fallback spawns an encode session which latches frame delivery to the OCR engine. The client doesn’t get to continue rendering without the OCR pass completed so the contents is forced to stay in synch.

This also where our terminal replacement comes in, particularly the TUI library used to provide something infinitely better than ncurses. The current implementation turns the accessibility implementation into a window attachment with a larger font (for the ‘low vision’ case), the shell populates it with the most important content and the text to speech tool kicks in.

Such text-only windows also get added controls, /global/tools/tts/voices/basic_eng/text_window/(speak_new, at_cursor, cursor_row, synch_cursor, step_row=n) for one dedicated reading cursor, that you move around separately.

Our custom CLI shell, Cat9, probes accessibility at startup. If found, it will adjust its presentation, layout and input control to match so that there is instant feedback. There is still the normal view to explore with keyboard controls, but added aural cues. The following clip demonstrates this both visually and aurally:

All this is, of course, network transparent.

A final related nugget covers both accessibility and security. The path /target/input/text lets you prepare a text locally that will be sent as simulated discrete characters. These are spaced based on a typing model, meaning that for the many poor clients that stream over a network, someone with signal processing 101 doing side channel analysis for reconstructing plaintext from encrypted channel metadata will be none the wiser.

This is useful for other things as well. For an input prompt one can set an oracle which provides suggestion completions. This is provided by the decode frameserver through hunspell, though other ones are just around the corner for filling the role of Input Method Engines, Password Manager integration, more complex grammar suggestions and offline LLM.

The following clip shows how I first type something into this tool, then read it back from the content of the window itself.

The current caveat is that it still does not work with X11 and Wayland clients due to their embarrassingly poor input models. Some workarounds are on their way, but there are a lot of problems to work around, especially for non-latin languages.

Previous episodes include: The Day of a new Command-Line Interface: Shell (high level) Whipping up a new Shell – Lash#Cat9 (technical demo), Cat9 Microdosing: Stash and List (feature highlight), Cat9 Microdosing: Each and Contain (feature highlight)

For this round we have added an interactive spreadsheet representation and a Debug Adapter Protocol implementation. These come as two discrete sets of builtins (groups of commands), ‘dev’ and ‘spreadsheet’ with a surprise interaction or two.

The intent for the dev builtin is to eventually collect more and more useful tools for managing all aspects of software development, from source control management to building, testing and fuzzing.

Starting with the spreadsheet. By typing:

builtin spreadsheet
new

You would get something like the following screenshot:

The following clip shows how I spawn new spreadsheet, populated by a CSV source and using the mouse cursor to interact with the layout.

This complements the form intended for media processing that was presented a few years ago as “Pipeworld” and re-uses the same language and parsing logic.

Cells can be populated with static contents, expressions like =max(A1:A4) or even shell commands through the ! prefix, e.g. !date +%Y:%M:%S. The following clip shows some basic expression use, as well as forced reprocessing of expressions and shell commands.

More useful is to populate the sheet with outputs from some command and processed by Lua patterns. The following screenshot shows the result of running insert #0 4 separate "%s+:" !cat /proc/cpuinfo

Exporting can be done using the existing copy builtin with control over output format, subrange and so on: copy #4(csv, compact, a1:b5) to export the resolved values of a1,b1,a2,b2 … as CSV.

There is still more to be done for this to be a complete replacement for the likes of sc-im, and to add things like plotting via gnuplot (if I can ever get their plot language to behave) and graphviz, as well more experimental things like importing Makefiles – — but at least for my, admittedly humble, spreadsheet uses it is good enough for daily driving.

Onwards to the Debugger

To start things you would do something like the following:

builtin dev
debug launch ./test

The ‘dev’ builtin set is used as it will eventually accumulate all developer task related commands like building, deployment, source control management and so on.

The following clip shows the default behaviour for that in action. In it you can see how multiple discrete jobs are created for managing threads, breakpoints and so on, detachable and with mouse cursor handling in place.

In the clip I also showed the process of stepping through threads, spawning source view, setting and toggling breakpoints, inspecting registers and modifying target variables.

I have spent thousands of hours staring at the GDB CLI prompt, and hated nearly every second of it. Not because of the relaxing task of debugging code or exploring software state itself, but for the absolutely atrocious terminal thrashing interface even with mitigation layers such as pwndbg. In fairness, a debugger TUI is in the deepest end of the complexity pool to get going.

There is a lot that goes into handling the protocol, and quite a few telltale signs of its designers, so we have just passed the point of basic bring-up. Importantly it is all composable with the data manipulation, filtering and transfer tools we already have elsewhere.

As an example of that we have ‘contain’ from the previous article, for instance, to bunch all the subwindows together into one contained job, useful when running multiple debug sessions side by side to step through multi-process issues.

We do have some other conveniences in place. Stepping controls are defined by the granularity of the job it represents, so stepping in the disassembly view would step instructions, while stepping in the source view would go by line or statement and so on.

Now to close the loop and mix in the spreadsheet part. In the following clip you see me picking a few registers and thread source location that gets added to a watch set. Whenever thread execution is stopped, these will be resampled and updated.

I then click the ‘spreadsheet’ option which will create a spreadsheet and populate it with the contents of the watchset as I go.

With all this in place we can almost start stitching the many other related projects together, from the data visualization from Senseye (closing in on 10 years…) with the window management from Pipeworld to the harnessing and setup from Leveraging the Display Server to Improve Debugging and build the panopticon of debugging from the plan presented in “Retooling and Securing Systemic Debugging” (2012, doi:10.1007/978-3-642-34210-3_10). But that is for another time.

This time we get contain for merging jobs together into datasets and each for batched command execution.

First a quick look at the latest UI convenience: view #job detach . This simply allows us to take any job and pop it out into its own window. The following clip shows how it works with ‘list’, initiated with mouse drag on the job bar. List was chosen as it uses both keyboard and mouse navigation, as well as spawns new windows of its own.

Onwards to contain. In the following clip I create a new job container by typing contain new. I then spawn a few noisy jobs and tell contain to adopt them through contain add.

By default contain will show an overview, coloured by their current run status. I can step through the job outputs either by clicking on their respective index in the job bar or type in contain show 1 (or any other valid index).

The container can also be set to automatically capture new jobs. In the following clip I spawn such a container and then run some commands. Those get added into the container automatically.

Contain meshes with commands like repeat, applying the action for all contained jobs at once. It gets spicier when I chose to merge the output of multiple contained jobs, either by right-clicking their entry in the job bar, or manually by running contain #0 show 1 2. These are then treated as a single dataset for any other commands, e.g. copy #0(1-100) that operate on the data for a job.

Contain even applies to interactive jobs. In the following clip I contain a ‘list’ in a detached window and show that mouse navigation is still working.

Moving on to each. Each is related to ‘for’ in Bash and similar shells, locally known as the syntax that I never recall when I need to and rarely get to do precisely what I want. Since we accumulate previous command outputs in discrete and typed contexts, we can avoid the “for I in file1 file2 file 3 do xyz $I done;’ form and reference the data to operate through our job and slicing syntax.

Starting simple, running this:

each #0(1,3,5-7) !! cat $arg

Anything before !! will be treated as part of the each command, and anything after will be reprocessed and parsed with $arg substituted for the sliced data, with some special sauce as $dir which will check if it is referring to a file and substitute its path, or use the path of the referenced job.

While it might at the quickest of glances look similar to the ‘for’ setup, the actual processing is anything but. Recall that everything we do here is asynchronous. If I would swap out ‘cat $arg‘ for ‘v! cat $arg‘ each invocation would spawn a new vertically split window, attach a legacy terminal emulator to it, and run the cat command.

Each also supports processing arguments:

each (sequential) #0(1,3,5-7) !!open $arg

Would functionally make it into a playlist. In this clip you can see how the media in the stash opens, and each time I close the window it launches the next in line.

Since we are not fighting for a single stdin/stdout pipeline, we have more interesting options:

each (merge) #0 !!cat $arg

This joins forces with the contain command by spawning a new container and attach the new jobs automatically.

The contained set of jobs also interact well with other commands, like trigger or repeat. In the following clip I repeat the previous form of running each on a stash of files. I then run the merge / cat command listed above and you can see how the commands keep progressing in parallel. Running repeat on the container would repeat the commands that had finished executing, merging output with the previous run, while letting the ongoing ones continue until completion.

The container here would also respect commands like trigger. Having a stash of makefiles, running them through a contained each like this:

each (merge) #stash !! make -f $arg -C $dir
trigger #0 ok alert "completed successfully"

Would treat each in the stash as a makefile, dispatch make, merge it into a container, associate a trigger with all merged jobs completing successfully and trigger a desktop notification.

That is enough for this time, next time around we will (likely) see what we can do to assist developer tooling such as the venerable gdb.