Fork me on GitHub!
Alex Baines

Layout independent keys with Linux & X11

With all the talk of new display managers like Mir and Wayland rising, this is perhaps an outdated topic to discuss, but with overall documentation about X11 and Xkb keyboard handling being pretty sparse, I thought I’d note my findings down anyway.

Getting a layout independent ID for a key - the same concept as a scancode - is especially useful for games where you want to bind keys for specific finger positions like WASD without worrying if the player is using an AZERTY keyboard, or an alternate layout like dvorak or colemak.

Doing this on Linux and X11 turns out to actually be pretty easy, although not well documented.

To give some context, X11 has two indentifiers related to keys: KeySyms and KeyCodes. KeySyms are 32-bit ints which represent any symbol that might appear on a key. Each key can have multiple KeySyms, for example the Q key on a QWERTY keyboard has (at least) two: q and Q (with alt-gr on a UK keyboard I get @ too).

KeyCodes are 8-bit numbers ranging from 8 to 255 which each represent a physical key. That same Q key only has one KeyCode no matter if you hold shift or not. The mapping of KeyCodes to physical keys is implementation defined though - depending on the keyboard driver in use, the Q key could have two differing KeyCodes between two computers.

However, in modern Linux distributions the keyboard driver is almost invariably going to be evdev - so these KeyCodes can actually be used just fine as a layout independent key identifier!

Wait so you’re saying we’ve solved this already? Yup, so if that’s all you wanted to know you can just look up the KeyCode in xev and stop reading. But let’s suppose that evdev was not ubiquitous, could we still get a layout independent ID for each key? If you use a small bit of Xkb arcane magic, then the answer, as I understand it, is yes.

It turns out XKb knew this was what we wanted all along, and as part of its RMLVO system has names for every key on a standard US keyboard that are independent of what is actually printed on the keys. If you take a look at the files in /usr/share/X11/xkb/keycode/ you’ll see keys labelled with names such as “AD01”. This corresponds to the key 4 rows up (the D) and 1 across - on QWERTY it’d be that Q key again.

XKB Names
XKB's generic key names.

That’s all well and good, but how can we map this to a KeyCode or KeySym at runtime? Handily, XKB stores an array of these “AD01” style names with the array’s index corresponding to the X11 KeyCode. So it’s just a matter of iterating through this array, finding the strange name of the key you’re interested in, and noting down what the index was. An example:

#if 0
    gcc -std=c99 $0 -o xkb-key-id -lX11 && ./xkb-key-id && exit
#endif
#include <X11/XKBlib.h>
#include <stdio.h>
#include <string.h>
#define countof(x) (sizeof(x)/sizeof(*x))

// Define which keys we want to look up
enum { MY_KEY_UP, MY_KEY_DOWN, MY_KEY_LEFT, MY_KEY_RIGHT };

struct KeyInfo {
    const char* pos_name;
    int keycode;
} keys[] = {
    [MY_KEY_UP]    = { "AD02" }, // 'w' on qwerty
    [MY_KEY_DOWN]  = { "AC02" }, // 's' on qwerty
    [MY_KEY_LEFT]  = { "AC01" }, // 'a' on qwerty
    [MY_KEY_RIGHT] = { "AC03" }, // 'd' on qwerty
};

int main(void){

    // Open a handle to the X11 display server + initialize XKB
    int ev, err, maj = XkbMajorVersion, min = XkbMinorVersion, res;
    Display* dpy = XkbOpenDisplay(NULL, &ev, &err, &maj, &min, &res);
    if(!dpy) return 1;

    // Find the keycodes for the keys we're interested in.
    XkbDescPtr xkb = XkbGetKeyboard(dpy, XkbAllComponentsMask, XkbUseCoreKbd);
    for(int i = 0; i < xkb->max_key_code; ++i){
        for(int j = 0; j < countof(keys); ++j){
            if(strncmp(keys[j].pos_name, xkb->names->keys[i].name,
                    XkbKeyNameLength) == 0){
                keys[j].keycode = i;
            }
        }
    }

    // Get the state + group for XkbLookupKeySym as (sort of) described by
    // XKBProto.pdf section 2.2.2. (With probably broken wrapping?)
    // I recommend just using the xkey.state field when you get a KeyPress
    // or KeyRelease, but this example doesn't do any event handling.
    XkbStateRec xkb_state = {};
    XkbGetState(dpy, XkbUseCoreKbd, &xkb_state);
    int key_state = xkb_state.group << 13;

    // Print out the keycodes / keysyms we got
    for(int i = 0; i < countof(keys); ++i){
        KeySym sym = NoSymbol;
        XkbLookupKeySym(dpy, keys[i].keycode, key_state, NULL, &sym);
        printf("Key %d: \"%s\", keycode: %d, keysym: %s\n",
               i, keys[i].pos_name, keys[i].keycode, XKeysymToString(sym));
    }

    return 0;
}

The KeyCodes printed out should match the codes listed in the KeyCode files, and also the KeyCodes from xev. I haven’t thouroughly tested it, but I expect it should work regardless of the keyboard driver in use.

To make use of these KeyCodes you’d compare them against the xkey.key_code field of KeyPress and KeyRelease events from X11 - if they match you know which key was pressed/released.

If you want to get KeySyms from this, then you should be a bit careful - XKeycodeToKeysym will only give you the KeySym from “group 1” of the keys. This means that if a user has multiple keyboard layouts configured which they switch between, then only the KeySym from the first layout will be returned, even if they’re currently using the second.

That’s probably why the function was deprecated - you should use XLookupKeysym instead which uses the struct you get from KeyPress / KeyRelease events (i.e. &event.xkey), and takes the currently active group into account. Likewise for XkbKeycodeToKeysym which takes the state field of the same event that has the group encoded in its upper bits (You could calculate it manually if you’re careful).

If you want text input as well then use Xutf8LookupString which gives you the correct KeySym as well as correct unicode text, accounting for dead keys, compose keys and other complexities. I wrote a small example some time ago for that here.

Hopefully that sheds some light on Linux + X11’s keyboard situation. TL;DR: KeyCodes are good enough for layout independence, but there’s other fun stuff lurking in the dusty X11 headers.

Thanks for reading, If you have any comments, corrections or questions, drop me an email: alex @ this domain.

Dares, Bots, and Ports, oh my!

Ludum Dare

Last week I took part in Ludum Dare 35, and created a game in 48 hours using C and SDL2. I called it “VampShift: The Bloodening”, and following from the competition’s theme of “shape-shift”, you play as a vampire that has to progress through rooms via shape-shifting into a bat.

VampShift: The Bloodening

You can play the web version here, or see its Ludum Dare entry page here, which includes links to the downloadable version for Linux & Windows.

I also live-streamed the development of the game on Twitch. The VoDs are available at https://twitch.tv/insofaras/profile.

You might be curious about the “insofaras” user name there. I originally picked that (rather pretentious :/) name on twitch intending it to just be a random anonymous account, but lately I’ve been doing some projects under that name that I think are worth de-anonymizing it for.

The first of which is:

Insobot

Insobot is an IRC bot written in C99 with a modular structure; it can load and reload modules in the form of shared libraries (.so) at runtime without disconnecting from the server. Various modules exist for it including, at the time of writing, modules for storing quotes, expanding URLs, showing twitch uptime / new followers, and giving the schedule for Handmade Hero.

The reason that last module exists is because Insobot basically spawned out of my involvement with the community around Handmade Hero (known as Handmade Dev or Handmade Network), and my interest in creating a project that uses a more limited subset of C++ than I was used to.

The community seems to appreciate insobot, and have used my insobot code as the basis for their hmd_bot, which is designed to replace their previous hmh_bot that was written in python. Perhaps the main reason insobot is “appreciated” though is due to one of its modules that I haven’t mentioned yet - the Markov chain module.

It’s a pretty standard Markov chain implementation that gets words — and connections between words — from the IRC chat. When mentioned, or on a random chance, it will form a sentence of its own and send that back for all to see.

For something that doesn’t really amount to much more than a lot of rand(), it has had some amusing results, several of which have been shared on twitter:

Insobot Quote
Showing some understanding?
Insobot Quote
Strangely appropriate
Insobot Quote
Not a fan of new programming languages
Insobot Quote
Insulting my people

The source code of Insobot is avialable at github.com/insofaras/insobot. I’ll probably be moving it to my main github account at some point.

4coder

The second major project that I took on under the name Insofaras is helping to port the 4coder code editor to Linux.

4coder

4coder is a project by Allen Webster to create an improved editor for C and C++ programmers. He’s already implemented the standard text editor side of things, and is planning to add more advanced features that take advantage of the editor itself having an understanding of the programming languages in use.

One of its defining features is the customisation layer, implemented through loading a user-created DLL / shared-library. In contrast to the scripting languages often found in other editors, this approach allows extension of the editor with native compiled code, appropriate for its target audience of C programmers.

Mr Handmade Hero himself — Casey Muratori — has already found the current alpha build good enough to replace Emacs; he recently switched to using 4coder on his Twitch streams. That is, in the words of Allen Webster, awesome.

My involvement with 4coder has been mainly implementing the stubbed-out functions that were present in the Linux platform layer. This boils down to looking at how the windows side is doing things, and translating that into a POSIX and Linux approach.

I’ve enjoyed doing this Linux porting work, if you have a project that you would like a Linux version of, contact me: alex @ this site.

More Updates >