Alex Baines - Updates

Layout independent keys with Linux & X11

Sat, 03 Sep 2016 00:00:00 +0000

With all the talk of new display managers like Mir and Wayland rising, this is perhaps an outdated topic to discuss, but with overall documentation about X11 and Xkb keyboard handling being pretty sparse, I thought I’d note my findings down anyway.

Getting a layout independent ID for a key - the same concept as a scancode - is especially useful for games where you want to bind keys for specific finger positions like WASD without worrying if the player is using an AZERTY keyboard, or an alternate layout like dvorak or colemak.

Doing this on Linux and X11 turns out to actually be pretty easy, although not well documented.

To give some context, X11 has two indentifiers related to keys: KeySyms and KeyCodes. KeySyms are 32-bit ints which represent any symbol that might appear on a key. Each key can have multiple KeySyms, for example the Q key on a QWERTY keyboard has (at least) two: q and Q (with alt-gr on a UK keyboard I get @ too).

KeyCodes are 8-bit numbers ranging from 8 to 255 which each represent a physical key. That same Q key only has one KeyCode no matter if you hold shift or not. The mapping of KeyCodes to physical keys is implementation defined though - depending on the keyboard driver in use, the Q key could have two differing KeyCodes between two computers.

However, in modern Linux distributions the keyboard driver is almost invariably going to be evdev - so these KeyCodes can actually be used just fine as a layout independent key identifier!

Wait so you’re saying we’ve solved this already? Yup, so if that’s all you wanted to know you can just look up the KeyCode in xev and stop reading. But let’s suppose that evdev was not ubiquitous, could we still get a layout independent ID for each key? If you use a small bit of Xkb arcane magic, then the answer, as I understand it, is yes.

It turns out XKb knew this was what we wanted all along, and as part of its RMLVO system has names for every key on a standard US keyboard that are independent of what is actually printed on the keys. If you take a look at the files in /usr/share/X11/xkb/keycode/ you’ll see keys labelled with names such as “AD01”. This corresponds to the key 4 rows up (the D) and 1 across - on QWERTY it’d be that Q key again.

XKB's generic key names.

That’s all well and good, but how can we map this to a KeyCode or KeySym at runtime? Handily, XKB stores an array of these “AD01” style names with the array’s index corresponding to the X11 KeyCode. So it’s just a matter of iterating through this array, finding the strange name of the key you’re interested in, and noting down what the index was. An example:

#if 0
    gcc -std=c99 $0 -o xkb-key-id -lX11 && ./xkb-key-id && exit
#endif
#include <X11/XKBlib.h>
#include <stdio.h>
#include <string.h>
#define countof(x) (sizeof(x)/sizeof(*x))

// Define which keys we want to look up
enum { MY_KEY_UP, MY_KEY_DOWN, MY_KEY_LEFT, MY_KEY_RIGHT };

struct KeyInfo {
    const char* pos_name;
    int keycode;
} keys[] = {
    [MY_KEY_UP]    = { "AD02" }, // 'w' on qwerty
    [MY_KEY_DOWN]  = { "AC02" }, // 's' on qwerty
    [MY_KEY_LEFT]  = { "AC01" }, // 'a' on qwerty
    [MY_KEY_RIGHT] = { "AC03" }, // 'd' on qwerty
};

int main(void){

    // Open a handle to the X11 display server + initialize XKB
    int ev, err, maj = XkbMajorVersion, min = XkbMinorVersion, res;
    Display* dpy = XkbOpenDisplay(NULL, &ev, &err, &maj, &min, &res);
    if(!dpy) return 1;

    // Find the keycodes for the keys we're interested in.
    XkbDescPtr xkb = XkbGetKeyboard(dpy, XkbAllComponentsMask, XkbUseCoreKbd);
    for(int i = 0; i < xkb->max_key_code; ++i){
        for(int j = 0; j < countof(keys); ++j){
            if(strncmp(keys[j].pos_name, xkb->names->keys[i].name,
                    XkbKeyNameLength) == 0){
                keys[j].keycode = i;
            }
        }
    }

    // Get the state + group for XkbLookupKeySym as (sort of) described by
    // XKBProto.pdf section 2.2.2. (With probably broken wrapping?)
    // I recommend just using the xkey.state field when you get a KeyPress
    // or KeyRelease, but this example doesn't do any event handling.
    XkbStateRec xkb_state = {};
    XkbGetState(dpy, XkbUseCoreKbd, &xkb_state);
    int key_state = xkb_state.group << 13;

    // Print out the keycodes / keysyms we got
    for(int i = 0; i < countof(keys); ++i){
        KeySym sym = NoSymbol;
        XkbLookupKeySym(dpy, keys[i].keycode, key_state, NULL, &sym);
        printf("Key %d: \"%s\", keycode: %d, keysym: %s\n",
               i, keys[i].pos_name, keys[i].keycode, XKeysymToString(sym));
    }

    return 0;
}

The KeyCodes printed out should match the codes listed in the KeyCode files, and also the KeyCodes from xev. I haven’t thouroughly tested it, but I expect it should work regardless of the keyboard driver in use.

To make use of these KeyCodes you’d compare them against the xkey.key_code field of KeyPress and KeyRelease events from X11 - if they match you know which key was pressed/released.

If you want to get KeySyms from this, then you should be a bit careful - XKeycodeToKeysym will only give you the KeySym from “group 1” of the keys. This means that if a user has multiple keyboard layouts configured which they switch between, then only the KeySym from the first layout will be returned, even if they’re currently using the second.

That’s probably why the function was deprecated - you should use XLookupKeysym instead which uses the struct you get from KeyPress / KeyRelease events (i.e. &event.xkey), and takes the currently active group into account. Likewise for XkbKeycodeToKeysym which takes the state field of the same event that has the group encoded in its upper bits (You could calculate it manually if you’re careful).

If you want text input as well then use Xutf8LookupString which gives you the correct KeySym as well as correct unicode text, accounting for dead keys, compose keys and other complexities. I wrote a small example some time ago for that here.

Hopefully that sheds some light on Linux + X11’s keyboard situation. TL;DR: KeyCodes are good enough for layout independence, but there’s other fun stuff lurking in the dusty X11 headers.

Thanks for reading, If you have any comments, corrections or questions, drop me an email: alex @ this domain.

Dares, Bots, and Ports, oh my!

Fri, 22 Apr 2016 00:00:00 +0000

Ludum Dare

Last week I took part in Ludum Dare 35, and created a game in 48 hours using C and SDL2. I called it “VampShift: The Bloodening”, and following from the competition’s theme of “shape-shift”, you play as a vampire that has to progress through rooms via shape-shifting into a bat.

You can play the web version here, or see its Ludum Dare entry page here, which includes links to the downloadable version for Linux & Windows.

I also live-streamed the development of the game on Twitch. The VoDs are available at https://twitch.tv/insofaras/profile.

You might be curious about the “insofaras” user name there. I originally picked that (rather pretentious :/) name on twitch intending it to just be a random anonymous account, but lately I’ve been doing some projects under that name that I think are worth de-anonymizing it for.

The first of which is:

Insobot

Insobot is an IRC bot written in C99 with a modular structure; it can load and reload modules in the form of shared libraries (.so) at runtime without disconnecting from the server. Various modules exist for it including, at the time of writing, modules for storing quotes, expanding URLs, showing twitch uptime / new followers, and giving the schedule for Handmade Hero.

The reason that last module exists is because Insobot basically spawned out of my involvement with the community around Handmade Hero (known as Handmade Dev or Handmade Network), and my interest in creating a project that uses a more limited subset of C++ than I was used to.

The community seems to appreciate insobot, and have used my insobot code as the basis for their hmd_bot, which is designed to replace their previous hmh_bot that was written in python. Perhaps the main reason insobot is “appreciated” though is due to one of its modules that I haven’t mentioned yet - the Markov chain module.

It’s a pretty standard Markov chain implementation that gets words — and connections between words — from the IRC chat. When mentioned, or on a random chance, it will form a sentence of its own and send that back for all to see.

For something that doesn’t really amount to much more than a lot of rand(), it has had some amusing results, several of which have been shared on twitter:

Showing some understanding?

Strangely appropriate

Not a fan of new programming languages

Insulting my people

The source code of Insobot is avialable at github.com/insofaras/insobot. I’ll probably be moving it to my main github account at some point.

4coder

The second major project that I took on under the name Insofaras is helping to port the 4coder code editor to Linux.

4coder is a project by Allen Webster to create an improved editor for C and C++ programmers. He’s already implemented the standard text editor side of things, and is planning to add more advanced features that take advantage of the editor itself having an understanding of the programming languages in use.

One of its defining features is the customisation layer, implemented through loading a user-created DLL / shared-library. In contrast to the scripting languages often found in other editors, this approach allows extension of the editor with native compiled code, appropriate for its target audience of C programmers.

Mr Handmade Hero himself — Casey Muratori — has already found the current alpha build good enough to replace Emacs; he recently switched to using 4coder on his Twitch streams. That is, in the words of Allen Webster, awesome.

My involvement with 4coder has been mainly implementing the stubbed-out functions that were present in the Linux platform layer. This boils down to looking at how the windows side is doing things, and translating that into a POSIX and Linux approach.

I’ve enjoyed doing this Linux porting work, if you have a project that you would like a Linux version of, contact me: alex @ this site.

Ramblings about C++ and Linux's perf.

Fri, 28 Aug 2015 00:00:00 +0000

I’ve been watching some Handmade Hero as of late, a series in which Casey Muratori creates a game from about as scratch as you can get - not even using the standard C runtime library.

I can’t say I fully agree with his reluctance to use certain C++ features like templates, but the series has shown some benefits to writing your own base code instead of relying on the standard library.

Taking that to heart, I have started a small standard library of my own, with some additions and replacements for some parts of C++’s STL. I don’t plan to stop using std:: things in my code, but writing my own versions of certain bits is, if nothing else, educational.

I'm not reinventing it, honest!

After I found std::experimental::string_view from the draft of the next C++ version I started using it anywhere I didn’t need the mutability of std::string, but being experimental it isn’t yet fully supported across compilers. Its interoperability with std::string isn’t that great either, which prompted me to start my alternate standard library - altlib

with two classes to replace them both.

These two classes, alt::StrMut and alt::StrRef, aren’t really all that interesting. They perform a little bit better than the std equivalents but its nothing to write home about. The more interesting thing is a Linux tool I discovered while debugging their performance, suitably named perf.

A lot of people say the debugging tools on Linux suck compared to other OS’s. While I agree that gdb’s interface is pretty user-unfriendly, I’ve always considered valgrind a great counterexample to that argument. Maybe I’m just tool-ignorant but I think perf is yet more proof that Linux has its fair share of awesome debugging tools.

The problem that I faced was determining why

alt::StrMut s;
for(int i = 0; i < 100000; ++i){
	s.append("some random string literal");
}

was considerably slower than doing the same thing for std::string. So, in my search for a tool to help me I found perf, compiled a small test case, and ran perf record on it to collect some performance data.

The perf tool dumps out a perf.data file, which can then be analyzed with perf report. This was when my mind was slightly blown, as perf gives an amazing annotated view of the assembly. As well as having neat arrows to trace the flow of jmp instructions, it lists the hot parts of the code that are bottlenecking the program’s performance. If you compile with debugging symbols, it intermixes the assembly with the source code too.

I was quickly able to see the following code was the main bottleneck, in a simple strlen equivalent function:

But that seemed very odd to me, I had created the StrMut class so that its append method took a const StrRef&, which the string literal could be implicitly converted to with some constructors like this:

template<size_t N>
StrRef(const (&str)[N]) : length(N-1), /* [...] */

StrRef(const char* str) : length(StrLen(str)), /* [...] */

I had assumed that a string literal would bind to the first one, which gets the literal’s length at compile time with no need for a StrLen call, but it was the second constructor that was actually getting called.

After some searching, I found a stackoverflow answer that seems to suggest that a string literal binds more easily to the latter style due to the former being a template call. I’m not sure if it should really work that way, but apparently it does. I changed the second constructor to take a plain char* without the const to stop literals from binding to it, there’s probably a better fix but that’ll do for the time being.

So I suppose I should thank the Handmade Hero series, for inspiring me to write code which happened to have a bug that - in turn - introduced me to perf. It will be a cool new tool in my Linux C++ debugging adventures!

New Game: Lawn-Mowing Robot

Sat, 28 Mar 2015 00:00:00 +0000

I felt unusually inspired recently after seeing lots of awesome puzzle games being made with Puzzlescript. Specifically, Train Braining and Mirror Isles by Draknek which you should totally check out if you haven’t already.

I’m no puzzle-game master like he is, but I had a go at making my own. I also don’t claim to be good at naming things - that’s why it’s called “Lawn-Mowing Robot”…

The game plays a bit weirdly, but it’s only 8 levels and runs in a browser, so I’d love for you to give it a try from the link below.

It’s my first proper puzzle game so if you have any comments or feedback drop me an email. Thanks!

Website Updated

Fri, 27 Mar 2015 00:00:00 +0000

I’ve spent some time updating the website to have a better front page and a new updates section, the most recent of which appear on it.

The new section will just be for informative but infrequent updates about the site (like this one!) or stuff I’m doing and wanted to share.

I also switched the site to HTTPS using a cloudflare free account. It seems to be working well so far - I’m surprised they offer such a service for free with seemingly no strings attached.

Game engine ported to JS with Emscripten

Fri, 13 Mar 2015 00:00:00 +0000

I’ve been messing with Emscripten and managed to port my OpenGL/SDL2 game engine to Javascript & WebGL with it.

The Engine itself is still pretty crap and you probably shouldn’t use it, but you could theoretically make web-games with it now, so that’s something.