superjamie.github.io

Caddy container with DNS provider

Sat, 02 Nov 2024 00:00:00 +0000

I’ve been using Caddy as a reverse proxy for a while. It’s very well documented and easy to configure.

Gone are the days of figuring out HTTPS and paying for a certificate, many webservers and proxies offer an automated TLS certificate via Let’s Encrypt automated challenges. Getting a valid TLS certificate is just one line in a config file.

The default way to do this is with a “HTTP Challenge” where the webserver must listen on port 80, and the provider makes sure the webserver asking for the certificate is the webserver at the domain.

I want to move to Wildcard domains, which requires a “DNS Challenge”, where the webserver contacts the DNS provider via an API.

Caddy implements this as additional modules, however modules cannot be dynamically loaded, you need to either download a Caddy binary from the build service, or build Caddy yourself.

This wasn’t very appealing to me, I just want a container that works.

Luckily, I found the Caddy Gateway Containerfile which shows an example of an automated build and copy into the regular Caddy container.

I modified this a little for my needs:

ARG CADDY_VERSION=2

FROM docker.io/library/caddy:${CADDY_VERSION}-builder AS builder

RUN XCADDY_SETCAP=0 XCADDY_SUDO=0 xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM docker.io/library/caddy:${CADDY_VERSION}

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Now I can podman build . and I get a new image at the end of it:

[2/2] STEP 2/2: COPY --from=builder /usr/bin/caddy /usr/bin/caddy
[2/2] COMMIT
--> 4d7a6ac83a0
4d7a6ac83a0783425c0b569684190c8233a484374ddb1fa24239903e01e1bf48

I can give this a tag:

$ podman image tag 4d7a6ac83a07 caddy-cloudflare:latest

Done! I now have the regular Caddy image with the built binary in it.

Any time the base container updates, I can just rebuild it and continue like normal.

Goodbye, Neovim

Sun, 09 Apr 2023 00:00:00 +0000

I’ve been using Neovim for a while now and it’s been fun. I didn’t see any real killer feature for it, but I wrote previously about other reasons to switch to Neovim.

However, I kept all my config in Vimscript and didn’t migrate to Lua for a couple of reasons.

First, I need to use Vim on systems where Neovim isn’t easily available, and I didn’t feel like manually updating the AppImage.

Second, I was being careful in case something made Neovim unviable for me.

That day has come with Neovim’s removal of the cscope feature in Neovim 0.9.

LSP is great for small codebases where you can easily do a build or generate the compilation database, however it’s not very enjoyable for large codebases like the Linux kernel or maybe one of the modern BSDs, where a build to generate compile_commands.json is a large undertaking, and especially if you switch around different historical revisions of the codebase a lot, which is my exact usecase.

For that sort of code browsing, it’s much better to use GNU Global tags and the gtags-cscope.vim plugin.

Nothing else in Neovim really made a huge difference to me. The vim-gtk3 Ubuntu/Debian package supports set clipboard=unnamedplus for yank persistence, and at least Vim 8 supports :terminal buffers too.

Maybe I can finally pull my finger out and get the t_SI/t_SR/t_EI escape sequences working under tmux so Vim has the nice guicursor too.

Edit: Well, that was surprisingly easy, just set the escape sequences (press actual Ctrl+k then Esc, don’t use literal ESC). Reference.

" guicursor in terminal
set t_SI=ESC[6\ q
set t_SR=ESC[4\ q
set t_EI=ESC[2\ q

Anyway, goodbye Neovim, it was fun while it lasted.

HP t530 Thin Client with Linux

Sat, 11 Feb 2023 00:00:00 +0000

I love collecting gadgets. I recently noticed you can get these thin clients pretty cheap off eBay. These were going for AU$60 (US$40) so I picked one up.

The full specifications are on HP’s website but in summary:

CPU: AMD GX-215JJ System-on-Chip (x86_64), dual-core 1.5 GHz (boost to 2.0 GHz)
RAM: DDR4 1866 SODIMM, one slot up to 16G
GPU: Radeon R2E, roughly equal to HD4550 or GeForce 9400GT
Display ports: 2x DisplayPort, can do dual 4K
Video decode: x264 and x265 in hardware at 1080p
WiFi: Intel 3168NGW (802.11ac), iwlwifi driver, PCI ID 8086:24fb
LAN: RealTek Gigabit, r8169 driver, PCI ID 10ec:8168
Storage: M.2 SATA 6.0 Gbps, 2280/2260/2242 up to 512G
USB: 2.0 and 3.0, 1x USB-A and USB-C on front, several on rear

It’s also fanless so is completely silent.

That’s really good for the sort of things I like to play with.

Mine came with some modern Windows and Secure Boot enabled, but that was easy to disable in the BIOS.

Booting Ubuntu 22.04 (kernel v5.15) worked without a problem. It found all the hardware, including the internal speaker, sound output via video, Ethernet, and WiFi. Using Linux on few-year-old hardware is delightfully boring these days, everything just works.

Plugging into a TV with HDMI worked fine with a DisplayPort-to-HDMI cable off eBay. Mine also includes the optional VGA out connector.

I had a spare SSD lying around, and bought 16G RAM off eBay (ironically, for more than the cost of the entire thin client).

Performance in emulators is good enough for me:

PSX (DuckStation) - 100% speed at 3x scaling (720p), struggled with 5x (1080p). Tested with Wipeout 3 (US).
PSP (PPSSPP) - 100% speed at 1x scaling, struggled with any more. Tested with Outrun 2006 Coast 2 Coast.

Sadly PS2 (PCSX2) isn’t an option. Gran Turismo 3 ran at ~40% speed even with graphics settings turned down.

This is a better option than a Raspberry Pi 4 if you’re looking for a movie/TV box for Kodi, or something to run emulators on like RetroArch or Lakka or the PC install of RetroPie, or just use with a wireless keyboard and mouse from the couch. It’s also still cheaper than even a 5th Gen Celeron Intel NUC goes for these days.

It also supports virtualization CPU extensions, so there’s no reason you couldn’t run VMs or even a small fleet of container hosts on it, maybe useful for Kubernetes or OpenShift learning.

Overall, I’m very happy with this.

Here’s a YouTube video showing the unit and the easy access to the internal components:

Repeated Keypresses - an application usage anti-pattern

Sat, 28 Jan 2023 00:00:00 +0000

Repeated keypresses are an application usage anti-pattern. We should try to recognise when we’re using repeated keypresses, and replace them with more meaningful alternatlves.

This whole idea comes from disabling your arrow keys in the Vim editor, because repeatedly pressing Up,Up,Up,Up is an inefficient way to use the editor. For that particular movement you could have used 4k to move up three lines.

There are many others places that repeated keypresses can be removed, here are a few ideas for:

Vim
Web browsers
Desktop Alt+Tab and application launching

Vim

If you get serious about learning Vim, one of the first things you should do is disable the arrow keys to train yourself away from them.

Unfortunately, that often leads to just repeating the same anti-pattern with the hjkl movement keys, like pressing kkkk to move up 4 lines.

Drew Neil (Vimcasts videos, Practical Vim book) has a post about this: Habit breaking, habit making (2013) where he introduces the idea of disabling the arrow and hjkl keys, and shows the old vim-hardmode plugin which is deprecated now.

The modern version of hardmode is vim-hardtime. I recommend using this and enabling it by default in your ~/.vimrc. Here are my settings:

let g:hardtime_default_on = 1
let g:hardtime_maxcount = 3

This lets me use one of the motion keys up to three times like jjj (to address overshoots and nearby quick movements) but no more.

To move around text, you can use the fF (forward/backward to character) or tT (to character) motions, or the /? keys (search forward/backward).

You can combine these with editing commands, for example, given the cursor at █:

The █quick brown fox jumped over the lazy dog.

You could delete to “lazy” with dtl (delete, to, “l”). This gives us:

The █lazy dog.

What about more difficult text with similar characters?

She █sells sea shells by the sea shore.

This is where vim-sneak becomes useful. It uses the same ideas as the fFtT mappings, but matches two characters.

We could delete up to “shore” with dzsh; (delete, zneak, “sh”, next match). The “next match” is used because “shells” also starts with “sh”. This gives us:

She █shore.

The zZ mappings are used in motions because sS is more commonly used by vim-surround.

Moving around text normally, you use the sS motions.

To just move to “shore” we would have used ssh; (sneak to, sh, next match). This gives us:

She sells sea shells by the sea █shore.

vim-sneak also adds little coloured annotations so you can see all the “next matches” available on the page, and move to them with one letter. See the vim-sneak page for an image of those.

To move around lines easier, you could use vim-numbertoggle, which sets relative numbers and absolute numbers in a sensible way. Again see the link for a great image.

Web Browser

We all use web browsers with a lot of tabs these days, you probably find yourself using Ctrl+PgUp/PgDn to move through tabs. But that’s a repeated keypress. Is there a better way?

If you have 9 or less tabs open, you can move to them with Alt+number. For example, moving to the first tab is Alt+1, moving to the second tab is Alt+2 and so on. But we hit a limitation with 9 tabs, because Alt+8 is the 8th tab, and Alt+9 is the rightmost tab.

You can cycle through tabs with Ctrl+Tab and cycle backwards with Ctrl+Shift+Tab, but again this is a repeated keypress.

Firefox (the superior browser) allows tab search in its awesomebar. Hit Ctrl+L to focus the address bar and start typing an open tab match. You’ll get an option “Switch to Tab”, so press down and Enter and you’re there!

Firefox also allows you to map Ctrl+Tab to “switch to last tab”, for easily moving back and forth between two tabs. In Preferences, this is Ctrl+Tab cycles through tabs in recently used order.

Chrome/Chromium doesn’t have this as far as I can see. I don’t use the Chromium family of browsers at all, so I don’t know of an option there. Maybe there’s an extension? Maybe you should use Firefox.

Both browsers support this Ctrl+L to focus the address bar where you can:

Search with your default search engine: type a word and press Enter
Go to a .com domain: type the domain like “google” and press Ctrl+Enter to go to google.com

Both browsers support searching in the current page with Ctrl+f.

Firefox also supports finding clickable text with the ‘ (single quote, next to semicolon in US layout) then Enter to follow the link. This is great for browsing text-heavy pages like Wikipedia. Try it now with ‘wiki in Firefox (the search is not case sensitive).

Generally you should learn your browser keyboard shortcuts:

Alt+Tab

Here’s a spicy one. The pattern of switching through application with repeated Alt*Tab is one of the worst desktop usage patterns.

Tiling window managers like i3 and Sway try to avoid this by having applications on different workspaces (virtual desktops) which you can switch to with Win+number, or by having applications in a split which you switch to with the Vim direction keys Win+hjkl. This is pretty good.

The rofi application launcher does smart partial matching of commands or applications to launch, narrowing down options as you type.

So you can rofi -show drun and start typing fir then just press Enter to launch Firefox.

i3 and Sway bind rofi -show drun to Win+d for this purpose.

I propose to change this to rofi -combi-modi window,drun,run -show combi which adds a window switcher. You can use windowcd to limit it to the current workspace. I bind this to Win+Space for my desktop, but you could bind it to Win+d if you use i3/Sway, or take the plunge and bind it to Alt+Tab.

Now you can hit your key, type the partial name of the window you want, and switch to it with Enter. Much quicker and better than cycling through Alt+Tab until you get there.

This has the added advantage that if the appication isn’t already open, it’s launched instead!

This is also a much better way to launch applications than clicking a “Start” button and browsing through a menu. I prefer this to the “activities” idea that desktops like GNOME/Pantheon/Unity try to implement.

Summary

Find repeated keypresses in your current workflow and eliminate them.

This will give you a more efficient and more intentioned application usage experience.

Get the computer to do what you actually want quickly, instead of doomscrolling through all available options.

What replacements have you found which are useful?

Other reasons to switch to Neovim

Mon, 09 Jan 2023 00:00:00 +0000

Neovim has become very popular as a replacement for plain Vim. Most advice about moving to Neovim is centred around the LSP and Lua features, but I don’t care about those. This post explores a few other reasons you might want to switch to Neovim.

Better Cursor

In Vim, regardless of what mode or context you’re in, the cursor is always the same. Usually the terminal’s default block cursor.

In Neovim, the cursor changes to a vertical bar in Insert mode, and changes to an underbar when you’re in a character context like replace.

This seems a very minor difference, but it made a big difference to me, notably helping determine where text would be inserted when I entered insert or append.

While you can apply a similar thing with plain Vim using ANSI escapes, I couldn’t get those working in tmux, despite many cryptic settings copy-pasted from StackOverflow.

I didn’t realise how much I loved this feature until I had it. Now I’d be very unwilling to give it up.

(according to codekoalas this is actually the guicursor feature of GVim)

Yank Persistence

For those who use Vim in tmux, it’s always been a pest to have one Vim instance open and wish to yank/paste across to another Vim instance in another pane/window/session.

This “just works” in Neovim.

As an added bonus, you can even yank something, quit the editor altogether, open a new editor, and paste - and that works too!

While it’s possible to work around this with the system clipboard in Vim, Neovim’s native persistence does this better.

Built-in Terminal

In Neovim you can run :terminal and a buffer is opened with a terminal emulator. You can even open a command directly in a split like :vsplit term://top:

You can enter insert or append mode to run commands, then when you have output ready you can Ctrl+\ Ctrl+n to return to Normal mode.

If that’s too much to remember, the help suggests a more normal binding:

if has('nvim')
    tnoremap <Esc> <C-\><C-n>
endif

Now the terminal buffer is treated like any other text buffer in Normal mode.

I regularly switch between editor and terminal in my work, copy and pasting between the two, but I’d really rather use Vim full-time. I’m looking forward to using this more.

Proper VSCode integration

I like to dabble in Visual Studio Code, though it usually ends up annoying me. Anyway.

It has a Vim emulation plugin which is sort of like Vim but not quite, and you can’t use actual Vim plugins in it.

However, the neovim plugin isn’t an emulation, it’s actual real Neovim.

This means all your plugins and config work just like a real editor.

This is possible because Neovim completely decouples the frontend and backend of the editor, so the editor can be embedded anywhere.

You don’t have to switch entirely

There’s a lot of info out there about rewriting your config in Neovim’s native Lua, but I still have to SSH into systems where Neovim isn’t available and I can’t (or can’t be bothered) getting the latest Neovim working.

But that’s fine.

Front-and-center on the documentation page is :help nvim-from-vim which guides you through using your existing ~/.vimrc and everything else right in Neovim.

So I can keep the same VimScript config for places where I don’t have Neovim, and I can use all the above handy features in places where I do have Neovim.

Best of both worlds.

Which I guess is what this post is all about - using Vim but getting something even better. That’s what Neovim is all about!

Distro Surfing 2022 Part 2

Sat, 07 Jan 2023 00:00:00 +0000

After my last post about trying out various Linux distros, my friend Christian said he was using Sway which is the Wayland version of i3. It’s been a while since I used i3 so I wanted to give it a try again and went hunting for a LiveCD of it.

I ended up finding several more spins of desktop environments, so gave them a go:

Sway

Sway 1.7 on Ubuntu Sway 22.10

Pretty nice. I use a set of keybindings similar to i3 already, so this wasn’t a huge gap for me. The built-in help popup showed me how to drive it and some experimentation got me comfortable. It was a little difficult to set the bar to the bottom of the screen but I figured it out. I had some weird font scaling issues with foot terminal unless I set dpi-aware=no. Some minor polish issues like no graphical NetworkManager installed by default, but that’s more the distro than the DE. This confirms my thought that I’d probably be very comfortable using i3 now, though I imagine it would take some learning and config time to get it perfect, which MATE and Cinnamon do almost by default. Good desktop that makes me want to explore it more.

Rating: 4/5

Unity

Unity 7.6 in Ubuntu Unity 22.04

Another Mac copy but they actually get the top bar right, with File/Edit/etc controls up there. The rest of the desktop didn’t thrill me, I didn’t like the launcher or “lens” concept. It hijacks too many keyboard shortcuts that I want to use. Lots of animation and blue and eye candy cruft. I’d be unhappy using this, just as I would be with Gnome.

Rating: 0/5

Deepin

Deepin 5.10? on Ubuntu DDE 22.04

Interesting looking desktop with lots of big buttons and transparency. Is this made for beginners? Seems to be based on KDE. The only reason I know this is because the very first thing I tried (switching to dark mode) resulted in a constant flood of ”KWin Exited Unexpectedly” messages and made the system completely unusable and I had to force power off. Trying again gave me many things which just didn’t work, keyboard shortcuts which can’t be changed, window transparent corners which don’t work properly so the window has a square shadow in the corner. Evidently needs a lot more work.

Rating: 0/5

Kylin

UKUI 3.1 on Ubuntu Kylin 22.04

Chinese desktop environment, even defaults to Chinese language by default. Wikipedia says this started out as a MATE fork, but has lost a lot of the customisation and options that MATE has. What’s left is a strange desktop with few configuration options that reminds me more of using Windows XP or 7 than Linux. Seems to use applications from different graphical toolkits (GTK, Qt, etc) to get whatever job done that it needs. The English translation is not the best, and even with English selected, some text is not translated. I wouldn’t use it.

Rating: 0/5

Bonus: Android

I don’t really expect to use Android as a desktop, but I was interested to see how these ran as a way to play mobile games on the computer. I tried:

Bliss v11.13 (Nov 2020)

This didn’t go so well, it seems sometimes laptop touchpad clicks registered and sometimes they didn’t. It couldn’t decide what it wanted to use for a Home app, and I got a lot of force closes. Playing Monument Valley was very laggy and registered some clicks half a minute after making them. I think this was Android 8 or 9.

PrimeOS 2.13 (July 2022)

This worked a lot better! Games played smoothly, even 3D games like Riptide GP Renegade. It even comes with a keymapper to help play games, though it seems to be full of ads. I could also install all the usual things like Firefox and Termux so was quite at home. This is based on Android 11. It even has what it calls a “bridge” which allows execution of Arm games on x86 somehow. If I want to try Android games on the computer I’ll give this a go again.

Small JSON libraries in C

Thu, 05 Jan 2023 00:00:00 +0000

I wanted a serialisation and parsing library to create and load savegames in plain C. Considering Cataclysm: Dark Days Ahead uses JSON, I figured that was good enough for me. Now to find a library.

A large list of libraries is available at the Native JSON Benchmark repository. This also lists scores of some standards conformance tests, and offers some comparative performance numbers of speed and memory usage.

I’m aware of Jansson and json-c, however these are large shared libraries made up of many individual files. I am looking for something with one or two files that I can compile right into my project.

My first try was Parson but the API was very confusing to me, and I couldn’t understand how to build arrays of integer values. There’s even an answer on StackOverflow which didn’t enlighten me. No go there.

Next I looked at cJSON which is nice and well documented. I particularly like the single data type used for all objects, the memory ownership model, and the fact you can still change an object after you’ve added it to something. This leads to a good coding pattern like:

allocate root object
allocate branch object
add branch object to root
add data to branch object

Then if any step fails, you can jump to a failure case which frees the root object, and frees any branches and data you’ve added.

However, I wasn’t a huge fan of the mixed-case coding style and the 64% standards conformance score didn’t inspire confidence, so I kept looking.

I then tried json-builder and its accompanying json-parser, which ended up being exactly what I’m after.

It has a very similar structure to cJSON, with a single data type json_value and same memory model.

Despite having almost no documentation, the API is very concise and the examples given are simple enough to understand it.

It also uses all snake_case and underscores, so appears to me visually.

An example of the above coding pattern in json-builder is:

#define ARRAY_SIZE 4
char *json_output(void)
{
    /* data to serialise */
    int my_array[ARRAY_SIZE] = { 1, 2, 3, 4 };

    /* create root object which will contain all other data */
    json_value *root_obj = json_object_new(0);
    if (!root_obj)
        return NULL;

    /* create an array, and if that works, instantly attach to the root */
    json_value *array_branch = json_array_new(0);
    if (!array_branch)
        goto out;
    json_object_push(root_obj, "array", array_branch);

    /* walk the array, appending values */
    for (size_t i = 0; i < ARRAY_SIZE ; i++) {
        json_value *array_int = json_integer_new(my_array[i]);
        if (!array_int)
            goto out;
        json_array_push(array_branch, array_int);
    }

    /* serialise */
    char *buffer = calloc(1, json_measure(root_obj));
    if (!buffer)
        goto out;
    json_serialize(buffer, root_obj);

    /* clean up and exit in the success case */
    json_builder_free(root_obj);
    return buffer;

out:
    /* any time we land here, the root object owns all the data, so
     * we can clean up with this one call to free the root object */
    json_builder_free(root_obj);
    return NULL;
}

This gives us a nice JSON object containing a named array:

{ "array": [ 1, 2, 3, 4 ] }

If you’re unfamiliar with this exit style with goto, it’s quite common in the Linux kernel and other similar system software. I don’t know if there’s a name for this but I call them “goto trees”. More at:

Anyway, I’m very happy with this, I will use json-builder and json-parser.

Distro Surfing 2022

Sat, 24 Dec 2022 00:00:00 +0000

Update: I also have a Part 2

I have a preferred desktop setup with MATE, but I like to go distro surfing every so often to look at what other desktop environments have to offer, know what’s out there, and maybe get some new ideas.

Here’s what I looked at recently, in no particular order:

Cinnamon

Cinnamon 5.6.5 in Linux Mint 21.1

Really good! I could configure everything I wanted. All the keyboard shortcuts I’m used to (except Super+Space). Could turn off lots of animations and thumbnails and desktop and other cruft. This is honestly probably a better desktop environment than MATE, even if it does use a little bit more RAM. I’d happily use Mint and Cinnamon right now. I would need to figure out how to write a one pixel window border.

Rating: 5 stars!

XFCE

XFCE v4.16 in Linux Mint 21.1

Unusable. Inconsistent theming with dark settings, some windows appear with dark borders, some appear with light borders. Strange to have two places to configure keyboard shortcuts (Keyboard settings, Window Manager settings). Some key presses and mouse clicks visually register but don’t actually do anything. Trying again works. Most importantly, after removing some default shortcut, the panel took over Super and broke my whole workflow. A broken disappointment.

Rating: 0 stars

GNOME

GNOME 42.2 in Ubuntu 22.04
GNOME 42.5 in Pop OS 22.04

I dislike the mouse-dependent “activities” flyover paradigm this is going for, it’s like it’s trying to copy the Mac desktop without actually understanding the ideas behind the environment. No way to disable desktop icons. No way to center windows. Keyboard settings are very tedious, requiring an extra click to set every setting. Some button text overrides its borders. Big ugly top window borders that waste so much space. Side dock taking up unnecessary space, though it can be covered by windows. Top panel taking up more space for minimal functionality, and I dislike top panels as they’re an ergonomic anti-pattern. The irony of having “privacy” options which default to the bad settings. I’d be very unhappy if I had to use this.

Rating: 0 stars

Pantheon

elementary 6.1

Does the whole “activities” thing like Mac too. Copies Mac in some unintuitive ways (eg: similarly useless top bar like GNOME) but doesn’t box you into its own ideas so much. Not a fan of the default terminal, which has transparency and no way to change the colour theme. “Code” editor has Vim emulation but no way to tell what mode you’re in. “Web” browser based on Epiphany is limited but supports Firefox Sync. Overall not my preference, would take a lot of configuration and extra apps, still probably stuck with big top window borders and top panel. Better than GNOME and maybe the path of least resistance to get Mac users onto Linux.

Rating: 2 stars

LXQt

LXQt 0.17.0 in Lubuntu 22.04

Like going back in time 15 years. Janky unmatched theming, basic inferior apps, Openbox window manager. Needs dark widget theme, better text editor, better terminal. If I used this I’d be configuring Openbox manually with the text file like I did in 2009 and be using apps from other DEs. At least it is minimal and gets out of the way. Viable with a lot of effort but not optimal.

Rating: 1 star

Budgie

Ubuntu Budgie 22.10

I had read this was a whole new DE, but it’s obviously just GNOME with very minor changes. No thanks.

Rating: 0 stars

KDE

KDE Plasma 5.20 in Kubuntu 22.04

Very surprising! I’d always heard how this was a very polished and modern environment, but it actually felt very “old Linux” with lots of settings everywhere. So many settings that it’s a little overwhelming at times. There are some animations and other heavy cruft which appear to be all configurable. Kate editor looks intersting, many options including Vim mode, Git, and even LSP support! Quite powerful how you can configure toolbars for each application. Nice to see all the apps don’t start with K anymore like they used to in KDE3.

I can see why people love KDE, they’ve invested a lot of time learning it (so feel that same self-taught pride like driving a car) and setting it up just how they want. The developers obviously respect the users and try to make it all things to all people, and they appear to have largely succeeded. Not my primary choice but I think I could grow to enjoy it.

I like the idea of binding Win+. to an emoji/unicode selector, I will steal that.

Rating: 3 stars

Bonus: Ventoy

https://www.ventoy.net/

Not a desktop environment, but a shoutout to this great USB drive booloader. It lets you copy ISOs to a USB drive and boot from them with a menu. Much easier than dd‘ing every image individually.

A small story about premature optimisation

Thu, 15 Dec 2022 00:00:00 +0000

There’s that saying in software “Premature optimisation is the root of all evil”, popularised by Donald Knuth and which some say can be attributed to Tony Hoare. The full quote from Knuth’s paper Structured Programming with go to Statements reads:

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

For a little roguelike tile engine I’m writing, I wanted to do a character lookup from Unicode to Codepage 437. The canonical reference is seen in cp437.h and the same lookup table is in libtcod too.

Going the other way is easy, from Codepage 437 to Unicode, simply use the character code to look up the array.

For example, the second character is the smiley face, so we pick TABLE[1] and get 0x263A which sure enough is Unicode ☺. What a nice O(1) algorithm.

But what if we want to do the reverse lookup?

The most naive way is just scan the table from start to end, but that becomes an O(n) algorithm, and especially bad if I’m looking up the later characters like line drawing or the second-to-last “black square” ■.

A co-worker and I did some bike-shedding about this and thought of several ways to improve it, but when I went to implement it I just did the dumb worst algorithm to get it done.

I measured the performance of drawing random Codepage 437 characters to the screen and got about 315 frames per second.

I then performed the same test drawing a random unicode character and causing the lookup to happen for every character and got… about 315 frames per second.

So, at least for now, this part of my program is not the bottleneck and doesn’t need further optimisation.

(the utility of measuring frames per second in a turn-based game engine is an exercise left to the reader)

Light mode is actually quite scary

Tue, 13 Dec 2022 00:00:00 +0000

The dark-mode vs light-mode discussion popped up on Hacker News again so I wanted to summarise all I cound find about it.

In chronological order:

An overview of standards and guidelines for visual display terminals (Sept 1984)

Suggests that contrast ratio over 20:1 is associated with user eye fatigute, and dark-mode displays are associated with “higher comfort” and “higher user acceptance”. Also discusses many other points like green-screen vs orange-screen, keyboard height, etc.

Smaller pupil size and better proofreading performance with positive than with negative polarity displays (Sept 2013)

“Positive polarity” (light mode) results in better proofreading performance.

Digital eye strain: prevalence, measurement and amelioration (April 2018)

Digital eye strain (DES) is most likely associated with not blinking for long periods, too-small fonts, or undiagnosed eye defects (astigmatism). Blue light might be a factor.

Reading and Myopia: Contrast Polarity Matters (July 2018)

Prolonged exposure to black-on-white (light mode) thins the choroid which is associated with myopia, while white-on-black (dark mode) thickens the choroid which inhibits myopia.

Blue light may not be as disruptive to our sleep patterns as originally thought

Blue light might not be so bad, bright yellow light produced poor sleep patterns #inmice. Suggests dim cool colours in evening, bright warm colours in the day.

Dark Mode vs. Light Mode: Which Is Better? - NN Group (Feb 2020)

Light mode is better for small cognitive tasks, but the effect decreases with age. People with cloudy vision defects might perform better with light mode. Cites the “Reading and Myopia” paper above.

Which colour scheme is better? - Brent from Stitcher (Sept 2020)

Coding with a dark theme adds measurable milliseconds of cognitive delay. Etienne Grandjean’s “Ergonomic Aspects of Visual Display Terminals” (1980) [1, 2, 3, 4] determined that it’s easier to proofread dark text on light background (light mode). I wish there was a copy of this paper online.

https://kevquirk.com/is-dark-mode-such-a-good-idea/ (2020)

Cites the NN Group page about light mode performance increase. Cites a 2019 Vice article which cites the “Digital eye strain” study. Dark mode doesn’t save battery unless it’s true #000 black on OLED screen. Users should be given flexibility to choose. Somehow comes to the conclusion that dark mode damages eyes?

https://medium.com/codesphere-cloud/should-you-really-be-coding-in-dark-mode-1c34c7cf5f99 (June 2021)

Cites the “Smaller pupil size” paper. Cites the “Blue light” study and says that screen use altogether before bed isn’t great. True-black-on-OLED power usage again. Users should be given flexibility to choose.

I actually started this post intending to write about accepting Brent from Stitcher’s challenge and try light mode for a week with the following:

Terminal: https://github.com/solarized/xresources/blob/master/Xresources.light
Vim: https://github.com/altercation/vim-colors-solarized with background=light
Firefox: https://addons.mozilla.org/en-US/firefox/addon/zen-fox/
GTK: https://github.com/Ferdi265/numix-solarized-gtk-theme

However, having sorted through the above, two major points stand out to me:

Risk of myopia increases with light mode

Use of dark mode also builds resilience against myopia.

This is the most important to me.

My eyesight is fairly perfect and I’d like to keep it that way.

Cognitive advantage of light mode is unsure

Whilst there is a repeatable cognitive advantage of light mode:

It decreases with age after 20s.

I’m older than that now, so any cognitive advantage seems irrelevant to me.

All research seems to be on instantaneous “glancing” cognition, like reading a sign, or proofreading of small passages.

I don’t see research specifically to exposure for 8+ hours a day and to the cognition of knowledge work like programming and problem-solving.

Even if it makes me work a little slower, saving my eyes is more important so that I can continue to be productive into the future.

Summary

With that, I’m switching back to dark mode.

Swap CSS stylesheet with JavaScript

Tue, 06 Dec 2022 00:00:00 +0000

I wanted a button to toggle between a light and dark stylesheet for my dynamic Markdown renderer.

This proved to be a lot more difficult than I expected.

The code to achieve this is:

<html>
 <head>
  <link id="sheet_light" rel="stylesheet" type="text/css" href="light.css" />
  <link id="sheet_dark"  rel="stylesheet" type="text/css" href="dark.css" disabled/>

<script type="text/javascript">
 function swap() {
  if (document.getElementById("sheet_dark").disabled == true) {
   document.getElementById("sheet_dark").removeAttribute('disabled');
   document.getElementById("sheet_light").disabled = true;
  } else {
   document.getElementById("sheet_light").removeAttribute('disabled');
   document.getElementById("sheet_dark").disabled = true;
  }
 }
</script>
</head>
<body>
 <button id="sheet_button" onclick="swap();">&#x1F4A1;</button>
</body>
</html>

A few notes on the implementation:

There are other solutions around which suggest to modify the href attribute of the link, however this causes a momentary “flicker” the first time the style is swapped, as the browser loads the new stylesheet. This flicker occurred every time I swapped the sheet in Chromium, but not in Firefox.

Using the disabled property of the link object avoids this, because the browser loads both stylesheets on page load. Switching between the sheets by toggling the disabled property switches between already-loaded stylesheets and avoids the flicker.

Just toggling disabled to true/false did work for me, but I found several answers on StackOverflow which say it’s better to remove the propertly altogether with .removeAttribute().

Unrelated to this implementation but very annoying to figure out, document.getElementById("whatever").href and document.getElementById("whatever").getAttribute("href") are not the same thing. The former is the full URL including the http:// onwards, but the attribute is just the file’s basename. I only discovered this by using the JavaScript console in Firefox.

Hopefully this pops up in someone’s search results and saves them the hours it took me to figure this out.

Hacker News Solarized Dark Nice theme for Stylus

Sun, 20 Nov 2022 00:00:00 +0000

How to do SDL 2D Fullscreen and Scaling

Sun, 06 Nov 2022 00:00:00 +0000

While tinkering with roguelike development, I ended up writing my own 2D tile engine in SDL. It takes a Dwarf Fortress Tileset of Codepage 437 symbols, and allows you to create a window on a modern graphical OS which behaves like a DOS text window.

If you’ve done any roguelike dev before this will sound familiar to you, and you probably know that ncurses/PDCurses do this in the terminal, and at least one of the better grahical alternatives:

A DOS-sized window of 80x25 characters with the EGA 8x14 or VGA 8x16 font ends up quite small on a modern high-res screen, so you’ll want to scale this up.

Scaling up pixel graphics like this always looks bad at arbitrary scaling sizes, so you’ll want to scale by integer factors (eg: 2x or 3x, but not 2.5x or 1.333x).

When you set your SDL window fullscreen with SDL_SetWindowFullscreen(SDL_WINDOW_FULLSCREEN_DESKTOP) you’ll want to restrict scaling to the largest sensible size for the display, and center the display so it draws in the middle of the screen.

I had done all this manually, multiplying my SDL_RenderFillRect by a scaling factor, working out my content size and how large the client window was so I could center the content, etc. I actually hadn’t done that last part about maximum scale factor for a fullscreen viewport and wasn’t exactly sure of the best way to do it.

Turns out I’d been doing things the hard way. HexDecimal (author of libtcod) recently informed me of two incredibly useful SDL functions:

Work out your content width/height and provide it to SDL_RenderSetLogicalSize, enable integer scaling with SDL_RenderSetIntegerScale. Now when you go fullscreen, the content is automatically scaled to the largest possible integer scaling factor for the content/screen, and is automatically centered in the fullscreen viewport.

I also discovered:

SDL_RenderSetScale

When the window is not fullscreen, this will do scaling for you, though you still have to work out the correct window size and resize with SDL_SetWindowSize.

Be sure to disallow SDL_RenderSetScale when the window is fullscreen, otherwise SDL gets confused.

This allowed me to rip out several lines of my own code and maths, leverage the library, and end up with a superior result.

Hopefully it helps you too!

Many thanks to HexDecimal and the positive helpful sharing nature of the entire roguelike development community.

C Static Analysis Tools - Part 2

Fri, 04 Nov 2022 00:00:00 +0000

Earlier in the year I wrote about static analysis tools for C and wanted to cover a couple more.

Let’s continue with our simple array-out-of-bounds test from the last post:

int main(void)
{
	int my_array[3] = { 0 };

	for(int i = 0; i < 6; i++)
		my_array[i] = i;

	return 0;
}

GCC Static Analysis

Red Hat’s David Malcolm has been busy adding a static analyser to GCC with the -fanalyzer switch, covered in Red Hat Developer blog posts:

Unfortunately I couldn’t get it to find an error on the above code, neither with GCC-11 in Ubuntu 22.04, nor with Compiler Explorer’s current “static analysis” or “trunk” versions of GCC (which appear to be daily builds).

I emailed David and logged by bug with GCC for this.

Turns out Compiler Explorer’s gcc (static analysis) branch was almost 3 years out of date, logged an Issue to fix that too.

PVS-Studio

I came across this commercial static analyzer from a series of blog posts about writing good code:

Usually commercial products like this are inaccessible to us mere mortals, and I expect a paid license of this runs into at least 4 figures per seat. They don’t even offer single licenses, only commercial volume.

However, they provide a free license to students, open source projects, and even non-commercial personal projects. Very nice!

It’s also available via Compiler Explorer as a plugin. In the compiler window just go Add Tool and PVS-Studio.

This is awesome, and a good way to get your product in front of people who’ll later use it in industry, where companies are more willing pay for it.

From the above blog posts you can see that PVS-Studio catches lots of things that other static analyzers don’t, and it picks up our incorrect access too:

The documentation for all analyzer warnings is available here: https://pvs-studio.com/en/docs/warnings/.

<source>:6:1: warning: V557 Array overrun is possible. The value of 'i' index could reach 5.

Not as friendly as cppcheck was in the last post, but does the job.

How to ncursesw (as of 2022)

Sat, 06 Aug 2022 00:00:00 +0000

Roguelike authors might be interested in using Unicode in curses, particularly for drawing Codepage 437 symbols in modern OSes, or just to get “beautiful” characters like Brogue.

This post is more-or-less an updated version of this howto page from 2014:

http://dillingers.com/blog/2014/08/10/ncursesw-and-unicode/

This page explains the new API well with some function cheatsheet too:

https://reversed.top/2019-02-05/more-than-256-curses-color-pairs/

Journeyman on Discord has written a small header to make CP437 symbols easier in C:

https://github.com/Journeyman-dev/cp437.h

Prerequisites

This assumes you have the following:

A terminal which supports Unicode - mate-terminal and guake do for me
A font with Unicode ligatures - PxPlus IBM VGA 8x16 does for me
Know how to type Unicode symbols on your system. On Linux I do:
- MATE desktop: Ctrl+Shift and hold them, u and release it, type hex code, release all keys
- vim Insert Mode: Ctrl+v and release it, u and release it, type hex code
The ncurses and ncursesw library package, development package, documentation package. On Ubuntu 22.04 this is at least: libncurses6 libncursesw6 libncurses-dev ncurses-doc

man ncurses

There are two common configurations of the library:

ncursesw
  the so-called "wide"  library, which handles multibyte characters (see the
  section on ALTERNATE CONFIGURATIONS). The "wide" library includes all of the
  calls from the "normal" library.  It adds about one third more calls using
  data types which store multibyte characters:

  cchar_t
    corresponds to chtype. However it is a structure, because more data is
    stored than can fit into an integer. The characters are large enough to
    require a full integer value - and there may be more than one character per
    cell. The video attributes and color are stored in separate fields of the
    structure.

    Each cell (row and column) in a WINDOW is stored as a cchar_t.

    The setcchar(3X) and getcchar(3X) functions store and retrieve the data
    from a cchar_t structure.

  wchar_t
    stores a “wide” character. Like chtype, this may be an integer.

  wint_t
    stores a wchar_t or WEOF - not the same, though both may have the same size.

  The "wide" library provides new functions which are analogous to functions in
  the "normal" library. There is a naming convention which relates many of the
  normal/wide variants: a "_w" is inserted into the name.  For example, waddch
  becomes wadd_wch.

ALTERNATE CONFIGURATIONS

  --enable-widec
    The configure script renames the library and (if the --disable-overwrite
    option is used) puts the header files in a different subdirectory. All of
    the library names have a "w" appended to them, i.e., instead of

      -lncurses

    you link with

      -lncursesw

    You must also enable the wide-character features in the header file when
    compiling for the wide-character library to use the extended
    (wide-character) functions. The symbol which enables these features has
    changed since XSI Curses, Issue 4:

    * Originally, the wide-character feature required the symbol
      _XOPEN_SOURCE_EXTENDED but that was only valid for XPG4 (1996).

    * Later, that was deemed conflicting with _XOPEN_SOURCE defined to 500.

    * As of mid-2018, none of the features in this implementation require a
      _XOPEN_SOURCE feature greater than 600. However, X/Open Curses, Issue 7
      (2009) recommends defining it to 700.

    * Alternatively, you can enable the feature by defining NCURSES_WIDECHAR
      with the caveat that some other header file than curses.h may require a
      specific value for _XOPEN_SOURCE (or a system-specific symbol).

    The curses.h file which is installed for the wide-character library is
    designed to be compatible with the normal library's header. Only the size
    of the WINDOW structure differs, and very few applications require more
    than a pointer to WINDOWs.

    If the headers are installed allowing overwrite, the wide-character
    library's headers should be installed last, to allow applications to be
    built using either library from the same set of headers.

Headers

Before any library includes in all your files, even if not using ncurses in those files, define:

#define _XOPEN_SOURCE_EXTENDED

If you are including <stdarg.h> to get variadics, you must include that before including ncurses.

Aside: you should be including system libraries (like stdarg) before external libraries (like ncursesw) anyway, see https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes:

Include headers in the following order: Related header, C system headers, C++ standard library headers, other libraries’ headers, your project’s headers.

Finally, to include ncurses with wide support, do not use the regular #include <curses.h>, instead you need to:

#include <ncursesw/curses.h>

Compilation

When linking, ensure your linker is picking up the wide version of the library with:

-lncursesw

I presume if you are using other curses helpers like panels, they still go first, eg:

-lpanel -lncursesw

but I haven’t tested this yet.

Locale - System

Check your locale in the terminal with:

$ locale
LANG=en_AU.UTF-8
LANGUAGE=en_AU:en
LC_CTYPE="en_AU.UTF-8"
...

On Ubuntu, you can change what locales to generate interactively with:

sudo dpkg-reconfigure locales

Or uncomment locales of interest in here:

$ grep -E "^[^#]" /etc/locale.gen
en_AU.UTF-8 UTF-8
en_US.UTF-8 UTF-8

Then either of these commands probably work:

dpkg-reconfigure --frontend noninteractive locales
sudo locale-gen

I suggest to always generate en_US.UTF-8 even if it’s not your primary locale, because most systems probably have the US locale (which have not intentionally excluded English altogether).

Locale - Program Code

The default locale C does not support wide characters by default. Set your program’s locale before starting curses with:

#include <locale.h>
#include <ncursesw/curses.h>

int main(void)
{
    setlocale(LC_ALL, "en_US.UTF-8");
    initscr();

If your curses program prints escape codes like ^@ instead of Unicode, you haven’t got locales working right.

Using Wide Characters

To get the wide character type wchar_t:

#include <stddef.h>

A wchar_t literal is a “long” character or string qualified by L, so:

wchar_t my_wide_character = L'a';
wchar_t my_wide_string[] = L"Hello";

wchar_t my_hammer_and_sickle_unicode = L'☭'
wchar_t my_hammer_and_sickle_codepoint = L'\x262d' // ☭ 

Long string literals are null-terminated with the long null character: L'\0'

ncursesw also supports its own “complex character type” cchar_t which includes a character and attributes like bold, colorpair, etc. You can pack its contents and attributes with setcchar(), which takes arguments of:

Pointer to cchar_t to pack
Pointer to wide character string, terminated with wide null L\0
ncurses wide attributes starting WA_*
short color pair
options not used here (it’s for when you need more than SHRT_MAX colorpairs)

cchar_t my_cchar = { 0 };
setcchar(&my_cchar, L"☭", WA_NORMAL, colorpair(C_RED, C_BLK), NULL);
mvadd_wch(1, 1, &my_cchar);
refresh();
wint_t keypress = { 0 };
int ret = get_wch(&keypress);

Functions - ncurses

Change to using all wide character and new API functions, don’t use any of the old functions anymore.

Set attributes with wattr_set(). Use new API version of the attribute macros, like changing A_BOLD to WA_BOLD.

Get input characters with get_wch(). Note the return value is a status - OK for wide character, KEY_CODE_YES for function key, ERR for error. The actual keypressed is placed into the *wch parameter you provide to the function. This is different to the old getch() which returns the key pressed.

I don’t see the use of packing entire strings full of complex characters (cchar_t), so for strings, use wchar_t and long strings and place with mvaddwstr() or mvwaddwstr().

For single characters, you can pack a cchar_t with setcchar() and place it with mvadd_wch() or mvwadd_wch().

However you can also skip using cchar_t altogether and write single long characters (eg: L"@") with the long string functions.

A complete list of wide curses functions is in the source like:

https://github.com/mirror/ncurses/blob/master/include/curses.wide

Functions - Wide Strings

To use wide string functions you must #include <wchar.h>. Some references:

A few replacement functions are:

Purpose	Old	New
string length	`strlen()`	`wcslen()`
string n length	`strnlen()`	`wcsnlen()`
string width	N/A	`wcswidth()`
string compare	`strcmp()`	`wcscmp()`
string n compare	`strncmp()`	`wcsncmp()`
string copy	`strcpy()`	`wcscpy()`
string n copy	`strncpy()`	`wcpncpy()`
memory set	`memset()`	`wmemset()`
string to long int	`strtol()`	`wcstol()`
string to double	`strtod()`	`wcstod()`

Convert old char strings to wchar_t with mbstowcs() “multibyte string to wide-character string” from <stdlib.h>, or swprintf with %hs reference
Note file I/O functions like fgetws(), fputws(), fwprintf(), swprintf(), etc

A Complete Example

#define _XOPEN_SOURCE_EXTENDED

/* gcc -o ncwtest -std=c99 -Wall -Wextra -Wpedantic main.c -lncursesw */

#define SCREEN_X  80
#define SCREEN_Y  24

#include <inttypes.h>
#include <locale.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>

#include <ncursesw/curses.h>

/**
 * Get the value of our custom curses color pair.
 *
 * @param fg  Foreground color using one of the COLOR_ macros
 * @param bg  Background color using one of the COLOR_ macros
 * @return    Curses color pair to use with COLOR_PAIR()
 */
static short colorpair(short fg, short bg)
{
        // the original 16 colorpairs should not be modified
        return 16 + fg + (bg * 8);
}

void curses_on(void)
{
        initscr();

        if (has_colors() && can_change_color()) {
                start_color();
                for (short bg = 0; bg < 8; bg++) {
                        for (short fg = 0; fg < 8; fg++) {
                                short pair = colorpair(fg, bg);
                                init_pair(pair, fg, bg);
                        }
                }
        }

        cbreak();
        noecho();
        keypad(stdscr, TRUE);
        curs_set(0);
}

void curses_off(void)
{
        erase();
        refresh();
        endwin();
}

int main(void)
{
        setlocale(LC_ALL, "en_US.UTF-8");
        curses_on();

        //wchar_t my_sym[] = { L'\x262d', L'\0' }; // ☭
        cchar_t my_cchar = { 0 };
        attr_t my_attrs = WA_NORMAL;
        setcchar(&my_cchar, L"☭", my_attrs, colorpair(COLOR_RED, COLOR_BLACK), NULL);

        mvadd_wch(1, 1, &my_cchar);

        for (int y = 1; y < (SCREEN_Y - 1); y++) {
                for (int x = 1; x < (SCREEN_X - 2); x++) {
                        mvadd_wch(y, x, &my_cchar);
                }
        }

        refresh();

        wint_t keypress = { 0 };
        int ret = get_wch(&keypress);

        curses_off();
        return 0;
}

History

v1.0 - 2022-08-06 - Initial commit
v1.1 - 2022-09-26 - Added wide char stuff, made some header bits clearer
v1.2 - 2022-09-27 - Clarified text more, simplified standalone example
v1.3 - 2022-10-04 - Wide character function table, added some more common functions

C Static Analysis Tools

Tue, 24 May 2022 00:00:00 +0000

Static analysis of computer programs can be simply defined as automated inspection of the source code to find programming errors. These might be very obvious errors that human code inspection or software testing would probably have caught, but also less visible coding errors which are hard for humans to find and might even compile and run okay.

For example, this code has an obvious array out-of-bound error. This array has three members starting at zero, so we cannot access member three or four or five:

int main(void)
{
	int my_array[3] = { 0 };

	for(int i = 0; i < 6; i++)
		my_array[i] = i;

	return 0;
}

It will even compile without complaint:

$ gcc array-out-of-bounds.c -o array-out-of-bounds
$ echo $?
0

Though running it (correctly) produces an abort:

$ ./array-out-of-bounds 
*** stack smashing detected ***: terminated
Aborted (core dumped)

The top Stack Overflow answer about static analysis tools lists PC-Lint and Splint as the top answers.

PC-Lint is proprietary and expensive (~US$400 per license, over a decade ago) so inaccessible to me.

Splint is open source, but development seems inactive with the last commit over 3 years ago and I couldn’t get it running anyway.

What other options do we have?

GCC Compiler Warnings

One way we could have detected this was better GCC warnings, which performs a sort of static analysis:

$ gcc -Wall array-out-of-bounds.c -o array-out-of-bounds
array-out-of-bounds.c: In function ‘main’:
array-out-of-bounds.c:3:13: warning: variable ‘my_array’ set but not used [-Wunused-but-set-variable]
    3 |         int my_array[3] = { 0 };
      |             ^~~~~~~~

However, that’s just telling me I haven’t used the variable I defined, not about the actual programming error.

My preference for GCC compiler flags is:

-std=c99 -Wall -Werror -Wpedantic

(considering the Linux Kernel has updated to C11, maybe I should too!)

You can also add -Werror if you wish any warnings to be considered as errors and stop compilation.

cppcheck

http://cppcheck.net/

cppcheck is a static analysis tool for both C and C++ which is actively developed and has packages and plugins for many environments.

On Ubuntu, install with sudo apt install cppcheck

Using it is very easy, just run it with your source file as an argument:

$ cppcheck array-out-of-bounds.c
Checking array-out-of-bounds.c ...
array-out-of-bounds.c:6:11: error: Array 'my_array[3]' accessed at index 5, which is out of bounds. [arrayIndexOutOfBounds]
  my_array[i] = i;
          ^
array-out-of-bounds.c:5:19: note: Assuming that condition 'i<6' is not redundant
 for(int i = 0; i < 6; i++)
                  ^
array-out-of-bounds.c:6:11: note: Array index out of bounds
  my_array[i] = i;
          ^

That’s found our problem and explained it nicely.

There are other arguments, I like to enable all checks, but suppress missingIncludeSystem so that it doesn’t check system library includes.

I add this to my Makefile template like so:

.PHONY: check

check: $(SRCS)
	cppcheck --enable=all --std=c99 --suppress=missingIncludeSystem $^

Now I can run make check and get this output.

Clang Static Analysis

https://clang-analyzer.llvm.org/

The Clang compiler frontend also has a static analysis tool.

On Ubuntu, install with sudo apt install clang-tools

This works by running scan-build in front of your usual compilation command.

We can run it like:

$ scan-build gcc array-out-of-bounds.c -o array-out-of-bounds
scan-build: Using '/usr/lib/llvm-14/bin/clang' for static analysis
scan-build: Analysis run complete.
scan-build: Removing directory '/tmp/scan-build-2022-05-24-103834-26933-1' because it contains no reports.
scan-build: No bugs found.

Oh dear, no bugs found. That’s unfortunate. I logged an Issue with the LLVM project.

This has found other errors for me in the past on larger projects, so don’t take this as any suggestion that the Clang Analyzer sucks or doesn’t work. As the webpage says, it’s still a work in progress.

One particularly nice feature is how scan-build works. It uses the shell environment variables CC and CXX to override the compiler with its own tool. This supports gcc, g++, and even make.

So an existing project with a Makefile has nothing to do to add Clang static analysis, simply run the usual command:

$ scan-build make

Summary

Static analysis tools are another tool in the toolbox that developers should use to ensure their code is free of errors.

As we have found, not every tool is perfect and some things will pick up warnings or errors that others don’t.

So it’s a good idea to add as many checks to your code as you reasonably can, both compiler warnings and static analyzers.

Part 2

C Static Analysis Tools - Part 2

Raspberry Pi OS (Raspbian) Bash PS1 Prompt Explained

Mon, 24 Jan 2022 00:00:00 +0000

The bash shell has variable PS1 which controls how the prompt is displayed.

You can find many examples of how to customise the prompt: https://duckduckgo.com/?q=Bash+PS1

For a long time I have kept this PS1 around from Raspbian, but today I wondered what everything actually did:

PS1="\[\033]0;\u@\h: \w\007\]\[\e[1;32m\]\h\[\e[1;32m\]:\[\e[1;34m\]\w \[\e[0m\]\\$ "

You can just paste this into bash to see what it looks like. It will show something like this, with your computer’s name as hostname and your cursor blinking at _:

hostname:~ $ _

If you change to a directory, then that directory name is added to the prompt, eg:

hostname:~ $ cd Downloads
hostname:~/Downloads $ _

I started looking into this, and the first part of it baffled me:

PS1="\[\033]0;\u@\h: \w\007\]

So I skipped that for now and focused on the rest, I already knew what these did:

\[\e[1;32m\]\h\[\e[1;32m\]:\[\e[1;34m\]\w \[\e[0m\]\\$ 

The bash colours are explained better by others (with actual colour) at pages such as:

https://www.shellhacks.com/bash-colors/
https://www.lihaoyi.com/post/BuildyourownCommandLinewithANSIescapecodes.html
or just search around for “bash colors” or “ansi colors”

They are explained as:

\[  start non-printing characters
  \e[1;32m  bold green
          \]  end non-printing characters
            \h  hostname
              \[  start non-printing characters
                \e[1;32m  set text color to bold/bright green
                        \]  end non-printing characters
                          :  literal colon character ":"
                           \[  start non-printing characters
                             \e[1;34m  set text color to bold/bright blue
                                     \]  end non-printing characters
                                       \w  current working directory, with home abbreviated as ~
                                         _  a literal space character " "
                                          \[  start non-printing characters
                                            \e[0m  reset text to no color
                                                 \]  end non-printing characters
                                                   \\$

And the last line creates the prompt. If effective UID is 0 (eg: you’ve done sudo -s to open a root shell) then it becomes #, otherwise if you’re a non-root user then it is $.

Now, back to the first part:

PS1="\[\033]0;\u@\h: \w\007\]

We can already spot some of these, such as \] and \] to delimit non-printing characters, \w for the current working directory with ~ for home, and I happen to know \u@\h shows username@hostname. But what are the rest?

The answer is hidden over here on the Ask Ubuntu StackExchange site.

The \033] is an Octal character named ESC, and the ] after it means an “Operating System Command” or OSC.

We can see in the xterm Control Sequence manual that 0; means “Set Text Parameters” and “Change Icon Name and Window Title”. The username@hostname: cwd now makes sense, that’s the graphical terminal window’s title text!

The \007 is an ASCII BEL character, which is used to end the OSC.

So looking at it again:

PS1="\[\033]0;\u@\h: \w\007\]

We can explain this as:

     \[  begin a sequence of non-printing characters
       \033 octal 033 "Esc"
           ]  OSC
            0;  change icon and window name to:
              \u@\h  username@hostname
                   :  a literal colon and space ": "
                     \w  working directory, with home abbreviated as ~
                       \007  octal 007 terminal bell, end the OSC
                           \]  end non-printing characters

The answer I linked on Ask Ubuntu isn’t the most upvoted answer, but it is the correct answer.

Looking at the author of the answer, it shouldn’t come as a surprise that it’s the correct answer.

It’s written by Thomas Dickey, a long-time open source software developer, and maintainer of many low-level pieces of software such as ncurses and xterm itself!

How to install Ubuntu with LUKS Encryption on LVM

Thu, 25 Nov 2021 00:00:00 +0000

Delay Loop in 6502 Assembly

Tue, 14 Jan 2020 00:00:00 +0000

The Android app Learn 6502 Assembly provides a reasonably good 6502 CPU emulator and debugger, integrated with an Android text editor and a small game-console-like interface (memory-mapped graphics, d-pad, A and B).

The emulator seems to be the one from Easy 6502.

It provides several example programs, but at least the “game-like” programs run way too fast to be pratical on my modern phone.

So let’s make a delay. We’ll do this by counting from 0 to 65535 (a 16-bit variable).

Within the game loop, you can call subroutine JSR delay

delay:
  ; save state. we'll be using the A register, so save it off to RAM
  ; and restore it before leaving. use another address if needed
  STA $40
  ; zero the A register ready for our use
  LDA #$00
  ; store that zero off to RAM too. registers only hold 8-bits so
  ; we'll store the high byte of our number off to RAM
  STA $41  ; high byte
delayloop:
  ADC #01
  ; usually adding 1 into A won't set the Zero bit. when the reigster
  ; overflows and the value of A becomes #$00, the Zero flag will be
  ; set. BNE jumps when the Z bit is not set
  BNE delayloop
  ; so we only land here after we've counted past #$FF
  ; clear carry flag so we don't add it next ADC
  CLC
  INC $41
  ; same strategy is used here, we jump unless the INC overflowed
  ; back to zero
  BNE delayloop
  ; clear carry flag so we don't add it next ADC
  CLC
  ; exit
  ; restore state of the A register
  LDA $40
  RTS
~~~

Or without comments:

~~~
delay:
  STA $40  ; save state
  LDA #$00
  STA $41  ; high byte
delayloop:
  ADC #01
  BNE delayloop
  CLC
  INC $41
  BNE delayloop
  CLC
  ; exit
  LDA $40  ; restore state
  RTS

Thanks Revenant for the catch where I missed CLC.

Reversing for Dummies

Thu, 28 Apr 2016 00:00:00 +0000

Notes for a talk given at Brisbane SecTalks.

Jamie Bainbridge - http://jbainbri.github.io/

Why?

To understand how a program works.

Break copy protection
Commercial/military espionage
Security analysis
Understand a segfault/panic/bug and fix/patch it
Add features
Re-implementation in another language or create a clone

What?

OS-specific instructions to link and execute a program
Several binary formats
- MS-DOS: COM and MZ
- Win16: NE
- Win32/Win64: PE
- System V ABI: ELF - we’ll be talking about this one
ELF binaries are made up of sections (each section links to one or more segments)
Common section headers:
- .text for executable code
- .data for initialized data
- .rodata for read-only initialized data
- .bss for uninitialized data, will be blank
- .plt for the linker to link in library functions
Within a code section we’ll usually have symbol names

How?

Non-assembly Tools

strings finds ASCII text within the file
file gives a basic fie type and some info
readelf and nm to read more stuff

Decompilers

These disassemble a binary then try to make source code back out of it. Original variable names are not preserved, nor are inlined functions, modern optimising compilers might make this a very confusing mess. May work ok for very simple programs, probably impractical with anything serious.

Disassemblers

objdump, dasm2.pl, ndisasm, radare2, IDA Pro, Hopper. These present binary in assembly code, sometimes with niceties like showing offsets and strings. radare has a scriptable API, IDA has a GUI!

Debuggers

gdb, WinDbg, OllyDbg, IDEs like Visual Studio or Eclipse. Allow you to load an executable, watch/step live execution, stop execution with breakpoints, view/edit registers, stack frames, other memory contents.

Assembly

A processor understands machine language or machine code, these are binary instructions which are loaded into the CPU and executed each clock cycle.

A compiler takes source code, assembles it into machine language object files, and a linker creates a binary out of your various assembled files and the libraries you used.

Assembly language is a human-readable representation of binary machine code instructions, assembly translates directly into machine code.

A CPU is made up of registers which can store a word-width of data (8-bit, 16-bit, 32-bit, 64-bit, etc), and has some way to access system memory.

The actual instructions are concerned with:

MOV - moving data around registers
SUB, ADD - doing mathematical operations on register contents
CMP - comparing values
JMP, CALL - moving to other parts of the program
or doing so conditionally:
- JE - jump if the last comparison was equal
- JNE - jump if the last comparison was not equal)
NOP - doing nothing

Each instruction mnemonic translates directly into hex.

Reference for x86 assembly instructions: http://ref.x86asm.net/
This is a 32-bit binary so we’ll want: http://ref.x86asm.net/coder32.html

Syntax

There are two common syntaxes for asm, AT&T syntax and Intel syntax.

AT&T syntax places the destination last:

MOV  $50, A    ; move hex 50 into register A

Intel syntax places the destination first:

MOV  A, $50    ; move hex 50 into register A

Tools

file

Uses “magic” to classify a file. Magic is just a list of known signatures, like how a PNG image always starts with 89 50 4e 47 ( PNG).

$ file ./reverseme1 
./reverseme1: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID
[sha1]=8835077d876c5544ac9766af68c2e98d3c137917, not stripped

strings

Looks for ASCII strings 4 or more characters long. Helps to remove the non-printable characters from a binary.

$ strings ./reverseme1 | grep flag
False_flag{RobotBully}!
DEBUG off. No flag for yews!

Just run it

$ chmod +x ./reverseme1 
$ ./reverseme1
DEBUG off. No flag for yews!

dasm2.pl

A perl script reverser. The author claims this is only useful if your system doesn’t have objdump but it finds strings from .data sections and resolves jumps to symbol names, which not many others do!

$ dasm2.pl -p -output=dasm.txt ./reverseme1
$ less dasm.txt

objdump

$ objdump -d -j .text --file-offsets --prefix-addresses --show-raw-insn ./reverseme1  | awk '/<main/' > objd.txt
$ less objd.txt

This debugger has a disassembler too:

disassemble /r

CTF Reverse Me 1

Look for where the “DEBUG off” string is printed, understand why.

The debug message is printed after a jump:

dasm2

  08048551  sub     $c,%esp                         ;  referenced from jump(s) at 080484df; 

  08048554  push    $8048618                        ;  reference to data : "DEBUG off. No flag for yews!"

  08048559  call    08048310                        ;  (this is a call to puts, to print the above string)

objdump

08048551 <main+0xf6> (File Offset: 0x551) 83 ec 0c             	sub    $0xc,%esp
08048554 <main+0xf9> (File Offset: 0x554) 68 18 86 04 08       	push   $0x8048618
08048559 <main+0xfe> (File Offset: 0x559) e8 b2 fd ff ff       	call   08048310 <puts@plt> (File Offset: 0x310)

The jump relies on a condition:

dasm2

  080484d4  movl    $0,-14(%ebp)                    ;  reference to data : "putchar"

  080484db  cmpl    $0,-14(%ebp)                    ;  reference to data : "putchar"

  080484df  je      08048551                        ;  
  080484e1  movl    $0,-c(%ebp)                     ;  reference to data : "putchar"

  080484e8  jmp     0804852c                        ;  

objdump

080484d4 <main+0x79> (File Offset: 0x4d4) c7 45 ec 00 00 00 00 	movl   $0x0,-0x14(%ebp)
080484db <main+0x80> (File Offset: 0x4db) 83 7d ec 00          	cmpl   $0x0,-0x14(%ebp)
080484df <main+0x84> (File Offset: 0x4df) 74 70                	je     08048551 <main+0xf6> (File Offset: 0x551)
                      ^^^^^^^^^^^^^^^^^^
080484e1 <main+0x86> (File Offset: 0x4e1) c7 45 f4 00 00 00 00 	movl   $0x0,-0xc(%ebp)
080484e8 <main+0x8d> (File Offset: 0x4e8) eb 42                	jmp    0804852c <main+0xd1> (File Offset: 0x52c)

Let’s just turn off that jump:

$ hexdump -C ./reverseme1 | egrep -A1 "0*04d. "
000004d0  00 86 04 08 c7 45 ec 00  00 00 00 83 7d ec 00 74  |.....E......}..t|
000004e0  70 c7 45 f4 00 00 00 00  eb 42 8d 55 d4 8b 45 f4  |p.E......B.U..E.|

$ hexdump -C ./reverseme1done | egrep -A1 "0*04d. "
000004d0  00 86 04 08 c7 45 ec 00  00 00 00 83 7d ec 00 90  |.....E......}...|
000004e0  90 c7 45 f4 00 00 00 00  eb 42 8d 55 d4 8b 45 f4  |..E......B.U..E.|

And run again:

$ ./reverseme1done
flag{}

References

dasm2.pl

https://www.lxtreme.nl/source/dasm2/

ELF binaries

x86 Assembly and Disassembly

Resources

http://microcorruption.com/ - Web-based reversing CTF with great debugger and fictional corporate espionage story, uses the actual Texas Instruments MSP430 assembly language.
http://www.crackmes.de/ - The reversers’ playground. A community of people who post and solve each others’ binaries for fun. Ranges from very easy to insanely difficult.
http://tcrf.net/ - People who reverse old video games to understand the file formats, look for unpublished content, create editing tools, etc
http://www.romhacking.net/ - Index of video game hacks to change graphics, gameplay, translate, etc
https://github.com/djyt/cannonball - This guy spent 4 years reversing the Outrun arcade ROM, re-implementing it in C++ and SDL, then started extending it with widescreen, 60fps, track editors, etc.

Raspberry Pi Vpn Router

Sat, 24 Oct 2015 00:00:00 +0000

title: Raspberry Pi VPN Router external-url: https://gist.github.com/superjamie/ac55b6d2c080582a3e64 —-

Running a Python script only if it's not already running

Fri, 31 Oct 2014 00:00:00 +0000

I have a simple Linux graphical desktop environment, using the Openbox window manager and tint2 panel. I use gsimplecal to display a small GTK Calendar widget when I click the panel clock. When I click the clock again, the calendar disappears.

gsimplecal is written in C++ but I thought it might be fun to rewrite it in Python.

The hardest part was detecting the running process, so a script would find another instance of itself then kill that instance and exit, but run if there was no existing instance. I came up with this fairly universal solution:

#!/usr/bin/python

from os import getpid
from sys import argv, exit
import psutil  ## pip install psutil

myname = argv[0]
mypid = getpid()
for process in psutil.process_iter():
if process.pid != mypid:
    for path in process.cmdline():
    if myname in path:
        print "process found"
        process.terminate()
        exit()

## your program here...

The 8-Byte Two-Step Redux

Sun, 21 Sep 2014 00:00:00 +0000

Over on Hacker News, Z posted a fantastic article called The 8-Byte Two-Step where the concept of aligning to byte boundaries was explored.

I thought this was a fantastic post, it goes right down to binary and logical operators then back up again.

After understanding the alignment, Z then writes three other ways of doing the same thing, disassembles them and benchmarks them.

Discussing this on HN, I said:

I have banned myself from using printfs to figure out things like this. Instead I would use a debugger and breakpoints to view the live variables in their different data formats.

In GDB, that’s p/t for binary and p/d for decimal.

To which Z replied:

I would like to try the method you describe… I’ll be curious to use your method and see how it compares.

So I figured I’d show how. Knowledge of the original post The 8-Byte Two-Step is required before progressing further.

I had to include <stdint.h> for the uint_64t, and I removed the printf()s. This is the code I am using:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <stdint.h>

#define MQ_ALIGNSIZE 8

char *as_binary(uint32_t n) {
  int bit_i = 31;
  char *bits;
  uint64_t on_bit;

  bits = malloc(sizeof(char) * 33);

  for (int i = 0; i < 32; i++, bit_i--) {
    on_bit = 1u << bit_i;
    bits[i] = (on_bit & n) ? '1' : '0';
  }
  bits[32] = '\0';

  return bits;
}

int main(int argc, char **argv) {
  uint32_t size = 0;
  uint32_t temp, a, b;
  size = atoi(argv[1]);

  a = (size + MQ_ALIGNSIZE - 1);
  b = ~(MQ_ALIGNSIZE - 1);
  temp = a & b;

  return 0;
}

Compile with debugging symbols (add -g) so we can use a debugger:

$ cc -g -std=c99 aligner.c -o aligner

Start the debugger with the program loaded:

$ gdb ./aligner

Add a commandline parameter to the program:

(gdb) set args 1023

Set a breakpoint at line 32 (just before the return):

(gdb) b aligner.c:32

Run and let GDB pause at the breakpoint. Here’s it’s about to run the return but has stopped before doing so. Now the application is frozen at this point in time:

(gdb) r
Starting program: /home/superjamie/code/8-byte-two-step/aligner 1023

Breakpoint 1, main (argc=2, argv=0x7fffffffdf08) at aligner.c:33
33        return 0;

Looking at the size variable:

(gdb) p/t size
$1 = 1111111111
(gdb) p/d size
$2 = 1023

And the rest of the variables:

(gdb) p/t a
$3 = 10000000110
(gdb) p/d a
$4 = 1030

(gdb) p/t b
$5 = 11111111111111111111111111111000
(gdb) p/d b
$6 = 4294967288

(gdb) p/t temp
$7 = 10000000000
(gdb) p/d temp
$8 = 1024

One obvious advantage of the printf() method is that it looks a lot nicer, remember Z’s output:

$ ./align_debug 1023
size: 00000000000000000000001111111111 (1023)
   a: 00000000000000000000010000000110 (1030)
   b: 11111111111111111111111111111000 (4294967288)
temp: 00000000000000000000010000000000 (1024)

I find GDB to also be a fantastic disassembler and lets you see the contents of registers as you run a program.

Try the following:

$ gdb ./aligner
(gdb) layout asm
(gdb) layout regs

This turns on GDB’s disassembly layout and register monitor. You’ll see the entire binary disassembled and the contents of the registers above. They’ll be blank because we’re not running anything yet.

Add our argument as before, and set a breakpoint:

(gdb) set args 1023
(gdb) b main

This sets a breakpoint as we enter main(). Assembly wise, we’ll have just setup main()’s stack frame (push %rbp and mov %rsp,%rbp) be ready to start executing.

We can type stepi to proceed to the next assembly instruction, then press the Enter key to repeat that stepi command.

The disassembler will step down as we execute, and the register monitor will highlight registers which change value.