XkbSucks

The xkb extension is the worst piece of essential Linux software that wasn’t written by Lennart Poettering. It’s complex and the documentation is terse and woefully insufficient. It fails to explain what xkb is, why it exists, and how to use it. There are essential file formats which are literally undocumented. Searching for better documentation turns up a handful of incomplete explanations which may solve a narrow need, but don’t explain the broader context of how this thing works.

Faced with this, many users -- who just want to remap a key or two -- bounce off the whole mess and fall back to xmodmap, which is actually documented, and works on a simple and easy-to-understand model which isn’t infuriating to actually use.

This project is the fruit of sweat and dismay, and contains both an explanation of what the hell is going on with xkb, and a starting point to actually customize keymap behavior.

Goals

What I want out of my input devices is this:

• A Hyper modifier, for Emacs.
• A Compose key, for entering accented / non-Latin characters.
• Automatic configuration when a keyboard is connected. I shouldn’t have to manually run a command to set the map. Specifically, I have a Bluetooth keyboard which disconnects after a few minutes of idle time and reconnects when you press a key. The frequent reconnects mean that the keymap is constantly reset.
• Separate keymaps per keyboard. I primarily use an Ergodox, but also use the built-in keyboard on my laptop. They need somewhat different keymaps, so any system that applies the same map to both boards comes with tradeoffs.

Approach

To meet all these needs, the X server itself has to be configured with InputClass configurations, one per type of keyboard, which uses some customized xkb files to specify how it should work.

Instead of xkb, why not...

Every time I’ve tried to explain what I’m trying to do, someone tries to be helpful in the following way:

Xkb?  Why don’t you just do <something which works for them, but only solves 50% of my problems, which they don’t know because they don’t have those problems> instead?

So let’s talk about alternatives, and why they don’t work.

xmodmap

At first, xmodmap seems compelling. It’s massively simpler, has reasonable documentation, and has been around for decades, so it’s well understood. Unfortunately, xmodmap manipulates a single, global keymap, used for all connected keyboards. It can’t have different maps per keyboard, and doesn’t persist configuration across keyboard un/plug events.

KMonad

KMonad is another compelling-at-first option; it allows remapping input events at a low-level, fires when devices are plugged in (thanks to udev), and doesn’t depend on X, so it works across the whole system.

But, KMonad’s strengths are also its downfall for my needs. Unfortunately, when the USB-IF defined the Human Input Device class, they didn’t include a scancode for Hyper — so it’s impossible to have a Hyper key that works in hardware, because the protocol can’t represent it. They hyper modifier exists in X, so you have to change the keymap in X to get one.

setxkbmap or xkbcomp

Both these are non-starters:

Both options are almost as hard to get working as configuring the X server, and still don’t meet every need: The keymaps are global, and reset if a keyboard is disconnected and reconnected. While xkbcomp claims to support setting the map for one device only, I’ve never been able to make it work.

What is xkb?

Xkb is an X server extension for configuring keyboards, which replaces the older xmodmap way of doing things.

Why does it exist? Why wasn’t xmodmap good enough?

Understanding this requires a history lesson.

In the 1980s, computers were stacks of big ol’ boxes. Whole cubic feet of computer, CRT monitor, a full-sized keyboard and mouse would litter your desk. Even then, they often connected to even bigger stacks of boxes over the network, because those were the the actual fast computers. The bulk means the hardware didn’t move much, you set the boxes on your desk, maybe stashed some underneath, and didn’t do anything unless you got a new one or had to move your desk or something. You had one keyboard, one mouse, and (usually) no way to plug in more than one of each. Nothing supported hot-plugging, and everyone said that unplugging while the machine was on might ruin the workstation. But even if you did, the hardware interfaces were so simple that they couldn’t tell if the keyboard and mouse were plugged in or not.

This is the kind of system X was designed to run on, and the early versions of X did a fine job of it. You might customize your keymap a little, or change it if you got a new keyboard, or decided to remap things, but that was it. X shipped different default keymaps for the keyboards it supported, you picked one, and that was that.

Stuff got faster through the 80s and 90s, but the overall picture was largely the same. Maybe your heap of local boxes was fast enough to run stuff on its own, so you didn’t need the giant machine in the dedicated room down the hall anymore. But the local stuff, what you actually touched? That was about the same. There were some minor changes, like PCs moving from 101-key to 103-key boards, some new hardware, things that kept the X maintainers cranking out keymaps for them.

Then, around 2005, laptops started to get good. Before that, they were a major compromise: tiny, dim, hard-to-read screens, not much storage, a handful of hours of battery life, and low performance. But somewhere in this mid-aughts timeframe, they got good. Really good. Good enough that instead of being a secondary computer you used when you weren’t at your desk, maybe it was your only computer.

This changed the game significantly. Instead of a single hard-wired keyboard, you might use the built-in laptop keyboard when traveling, but connect a full-size keyboard at the office. You might use a remote that shows up as a keyboard when presenting slides. You might connect to a different keyboard when on-site somewhere, or carry a smaller than full-size, but larger than the laptop-size external keyboard. The smaller body of the laptop means that the keyboard can be laid out very differently than the keyboards. A single, fixed keymap no longer gets the job done, because you no longer have a single, fixed hardware configuration.

Also, the proliferation of laptops and companies tried a ton of different layouts to maximize the usability of a small keyboard. Which meant the number of keyboard maps that had to ship with X exploded. And since they were largely similar — the QWERTY (or whatever your local standard layout is) keys are going to be the same, it’s the extra stuff that varies — the maintenance burden of those keymaps increased.

Xkb is intended to solve these problems.

Note: I’m glossing over some timing. Xkb launched with X11R6.1 in 1996, so it’s not that laptops drove the creation of xkb. But the general shift towards ephemeral hardware configurations and hotplugging was accelerated massively around that era, which made it harder to ignore xkb like you could do for the previous nine years.

How does xkb work?

This isn’t remotely a full explanation, since that’s above my pay grade and not my job. But this is pretty close to the explanation I wish somebody had written for me.

Simpler tools provide a full keymap, that is, an association of the logical action which occurs when every physical key is pressed. If you have a 105-key keyboard, its keymap needs 105 entries, or some of the keys won’t do anything. Xkb breaks this model down into smaller pieces.

Instead of one giant keymap, xkb computes a full keymap by assembling several smaller keymaps. There might be one map for a QWERTY layout, common to all staggered QWERTY boards, which get combined with maps that only have keyboard-model-specific features. The result is reduced burden for the X maintainers and more flexibilty for the end user.

With that framework in place, let’s look at the different pieces of the xkb puzzle. We’re going to go bottom-up, since it’s impossible to understand how the high-level pieces work without knowing the lower-level parts they operate on.

XkbDir

The X server configuration allows you to tell it where all the xkb files live, using the XkbDir option. All xkb files are stored here, broken out into subdirectories for individual components.

On most Linux machines, XkbDir should be /usr/share/X11/xkb. The location may vary on other machines, but the internal structure will be the same.

You can only configure one XkbDir, there’s no support for search paths or anything like that. This will be important later.

Keycodes

Keycodes files abstract over hardware implementation differences. A PC USB keyboard might return keycode 42 when you press the A key, and an Amiga might return keycode 71. The keycodes get selected based on the hardware & OS platform, and map those to a common set of symbolic names. This allows the higher-level keymapping to happen independent of the platform. Modern Linuxen use the evdev keycodes.

Unless you’re working with an extremely weird setup, you don’t need to mess with these. Where "extremely weird" means a platform Xorg has never run on, or an entirely new keyboard hardware interface.

Geometry

These files declare a graphical layout for specific keyboards. While I like the idea of having a properly-shaped Ergodox in the, like, one UI place that ever uses these, the format is undocumented, as near as I can tell. I assume there’s a tool used to graphically lay these out, and a way to preview your work, but I’ve been unable to find what those tools actually are.

They aren’t critical for my needs, so they get shoved in the Ignore Pile.

Types

Types contain shift-level configurations, that is, which X events are emitted when a key is pressed along with a modifier. These aren’t relevant to my usecase, so I’m ignoring them as well. More information is in this doc.

Compat

Compat contains workarounds for extremely old X clients, which don’t understand xkb. I don’t care about them, and you shouldn’t either.

Symbols

This is where things get interesting. Symbols files are keymaps which associate the symbolic names of keys — the stuff defined in the keycodes files — with the X protocol key events. Each file contains a logical grouping of multiple, related maps. The specific map to use is selected with the notation file(variant), where variant is a keymap declared within file. For example, /usr/share/X11/xkb/symbols/ctrl contains:

// Eliminate CapsLock, making it another Ctrl.
partial modifier_keys
xkb_symbols "nocaps" {
    replace key <CAPS> { [ Control_L, Control_L ] };
    modifier_map  Control { <CAPS>, <LCTL> };
};

The file contains different mappings for the Control key (generally), and this map puts left Control on the capslock key. Recommended!

The file and map names can be anything, but generally follow a usecase(option) style. See this doc for an explanation of the format.

Rules and models

Rules define how all the other pieces get assembled into a complete keymap. Each complete keymap in the set of all configured keymaps is called a "model," and this corresponds to what the user selects when configuring their keyboard.

There appears to be zero end-user documentation for the format of this file. It seems to be roughly:

// Lines starting with two slashes are comments.
//
// Lines which look like:
//     ! foo = bar
// Declare that the following lines are associating bar with foo.  I
// don’t know the list of valid identifiers.
//
// Lines which look like:
//     baz = quux
// Associate the bar named quux with the foo named baz.

// Example:
// Declare $pcmodels to be four different PC keyboard layouts.
! $pcmodels = pc101 pc102 pc104 pc105

// Define the geometry for a keyboard model
! model = geometry

// The geometry for all the models contained in $pcmodels is in the %m
// (the self-name, for example pc101) variant of the geometry/pc file.
$pcmodels = pc(%m)

While every other component of xkb is extremely flexible, and allows building the map from smaller pieces, the same can’t be said of the rules files. You can pick only one set of rules, and it has to have enough information to handle every possible configuration you could want. Which is deeply unfortunate.

Options

Options are defined in the rules, and are a way to specify common personal preference tweaks that should be added into an otherwise standard keymap. Effectively, they allow overlaying partial keymaps on top of a base.

Going back to remapping caps to control, the standard /usr/share/X11/xkb/rules has:

// Define an option which changes keymap symbols:
! option	=	symbols

// The xkb option named "ctrl:nocaps" adds the nocaps variant taken from the
// symbols/ctrl file.
  ctrl:nocaps		=	+ctrl(nocaps)

rules.lst and rules.xml

Another mystery. Both these files have the same information, in two different formats: XML and another undocumented one, either the same as rules or very similar. They present a user-understandable description of the models defined in the rules file. As with rules, they have to contain everything, and you can’t break things down any further. I assume the XML is canonical, and the .lst is generated from those. The tooling to do this is likely trapped in the Xorg build system, and is not readily available to users.

Fortunately, they’re also not important for what I want to do, so they get thrown on the Ignore Pile as well.

Putting the pieces together

First off, I want a Hyper modifier on every keyboard. On normie keyboards, I like to use the key in between the right control and alt keys. On a modern ThinkPad, that’s Print Screen. So in symbols/hyper, a partial map is defined for that:

// Print Screen is mapped to Hyper
partial modifier_keys
xkb_symbols "print_screen" {
    replace key <PRSC> { [ Hyper_R ] };
    include "hyper(base)"
};

This includes another keymap, hyper(base). That’s defined in the same file:

partial modifier_keys
xkb_symbols "base" {
    modifier_map Control { <LCTL> };
    modifier_map Mod1 { <LALT>, <RALT> };
    modifier_map Mod3 { Hyper_L, Hyper_R };
    modifier_map Mod4 { <LWIN>, <RWIN> };
};

This is needed because the stock maps assign Mod3 to a different modifier, and even though the map explicitly says replace key, it actually adds it, so it emits, like, Hyper+Alt or something silly like that. I know, right?

On my Tex Shinobi, the key between right alt and control is Menu, so define a map for that:

// Menu is mapped to Hyper
xkb_symbols "menu" {
    replace key <MENU> { [ Hyper_R ] };
    include "hyper(base)"
};

New Rules

With the symbols defined, rules have to be added to make them options:

! option           = symbols
hyper:menu         = +hyper(menu)
hyper:print_screen = +hyper(print_screen)

Since you can only have a single rules defined per keyboard, the approach I took was to concatenate this to the end of the standard rules/base file and output it to rules/sucks. There really should be a better way of doing this.

Where do I put all this stuff?

Ah, ha ha, time for another little xkb joke to get played on us here. Remember XkbDir? All xkb files have to be in there. So the options are all terrible:

• Put the files in the system directory. This requires root, and might confict with the stock things — if you want a new mapping for control, it can’t go in symbols/ctrl, it has to be symbols/myctrl or something else that doesn’t already exist. Or you can modify the stock files (yeuch).
• Make a new directory, like /usr/local/share/X11/xkb. But now you have to reconfigure the X server to point there, and copy all the standard files there, too.
• Throw your hands up and use xmodmap.

I chose the first option, and made a Debian package for the files, so I’m not spewing unmanaged junk all over the filesystem.

Xorg configuration

With all that nonsense out of the way, it’s time to actually use it. User Xorg config snippets belong in /etc/X11/xorg.conf.d, and package-supplied ones should be in /usr/share/X11/xorg.conf.d. This package uses the latter, but if you’re not using it, put them in the former.

Here’s an example for the Shinobi:

Section "InputClass"
    Identifier "Tex Shinobi (USB)"
    MatchUSBID "04d9:0407"                       # Only apply to this device.
    matchiskeyboard "on"
    Option "XkbRules" "xkbsucks"                 # Use custom rules.
    Option "XkbOptions" "ctrl:nocaps,hyper:menu" # Add the desired options
EndSection


Section "InputClass"
    Identifier "Tex Shinobi (Bluetooth)"
    MatchProduct "TEX-BLE-KB-1"                  # Bluetooth has no USB ID.
    MatchIsKeyboard "on"
    Option "XkbRules" "xkbsucks"
    option "XkbOptions" "ctrl:nocaps,hyper:menu"
EndSection

Hopefully, with all the splaining I did earlier, this makes some kind of sense now.

One thing that hasn’t come up until now: These being in the InputClass section of the config means they apply to any keyboard they match, as soon as they’re connected (or if they’re connected when X starts). If you plugged in two identical keyboards, they’d get configured just the same. The alternative is InputDevice, which configures a specific keyboard, usually identified by its evdev device node. This allows you to have one Shinobi use one keymap, and a second Shinobi use a different one. In practice, you don’t usually want that, and it’s difficult to reliably tell which keyboard is which. InputClass is what’s needed.

For the built-in ThinkPad keyboard, things are more irritating. They’re still connected with a 1980s-style i8042 interface, so there’s not a USB ID to match on, nor is there a meaningful vendor or product name. It shows up to libinput as "AT Translated Set 2 keyboard." So, while it’s safe to drop the Shinobi configuration on any machine — it has no effect unless a Shinobi is connected — you can only install the ThinkPad configutarion on machines with that specific hardware. Say, using Ansible to set up classes of machines and having a role that applies to them. Annoying! But here’s the file:

Section "InputClass"
    Identifier "ThinkPad Built-In Keyboard"
    MatchIsKeyboard "on"
    MatchProduct "AT Translated Set 2 keyboard"

    Option "XkbRules" "xkbsucks"
    Option "XkbOptions" "ctrl:nocaps,hyper:print_screen"
EndSection

With both those copied to a suitable xorg.conf.d, you can restart X and live the life of smoothly functioning keyboard layouts.

Oh, yeah...

So, what else is in here besides the README?

• A handful of partial maps I use.
• A partial rules file which declares options for those.
• Some xorg conf snippets which apply them to classes of keyboards.
• A Makefile to build the rules and install everything.
• Debian packaging which bundles it all up nicely.
• This README, which is an order of magnitude larger than literally anything else in here.

My sincere hope is that this explanation and practical working example of how to wrangle this godforsaken system saves others from suffering the same pain I did to produce it.