NHollmann's RSS Feed

Run dynamic code in C on Unix

Sat, 03 Apr 2021 10:00:00 GMT

Run dynamic code in C on Unix

This is just a short extension of my previous post regarding dynamic code execution on Windows. While you can find examples of this technique for unix on other pages, I liked to try it out myself and post it here for documentation purposes. Please read my previous post for more information, espacially regarding the limitations.

How it works (on Unix)

The technique is really the same as on windows, only that we need to call other system functions to allocate and manage our memory. As always, the full sourcecode is on GitHub.

#include <stdio.h>
#include <stdbool.h>
#include <string.h>
#include <sys/types.h>
#include <sys/mman.h>

/**
* Helper function to execute dynamic code.
* 
* @param code a pointer to the code to execute
* @param length the length of the code
* @return true if the code run successfully, otherwise false
*/
bool run_code(const void* code, size_t length)
{
	// Allocate new memory for code execution.
	// We don't set the execute flag because we won't want to
	// open doors to malicious software.
	void* dynMemory = mmap(0, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	if (dynMemory == MAP_FAILED)
	{
		printf("Alloc Error\n");
		return false;
	}

	// Copy code to new memory.
	memcpy(dynMemory, code, length);

	// Change protection.
	// We remove the write flag and set the execute flag on that page.
	if (mprotect(dynMemory, length, PROT_EXEC))
	{
		printf("Protect Error\n");
		return false;
	}

	// Call the new function in memory.
	int result = ((int (*)(void))dynMemory)();
	printf("Result: %d\n", result);

	// At the end we free the memory.
	if (munmap(dynMemory, length))
	{
		printf("Free Error\n");
		return false;
	}

	return true;
}

The explanation for this code is the same as in the Windows version, only with some different functions:

Windows	Unix
VirtualAlloc	mmap
VirtualProtect	mprotect
VirtualFree	munmap

Make sure you don’t run code from an untrusty source. It will run with the same permissions as the calling process/thread.

Links

Full sourcecode on GitHub: GitHub Gist

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/run-dynamic-code-unix/.

Run dynamic code in C on Windows

Thu, 01 Apr 2021 22:00:00 GMT

Off-Topic: With this blogpost I may waste my chance for a good aprils fool but neither the less, I just publish it today. On the other hand this technique is, while real, so niche, that it maybe could be seen as a joke to somebody…

Run dynamic code in C on Windows

Lately I’m experimenting with running machinecode from within an C program. I found some examples of this technique for Linux systems using mmap and mprotect, but nothing for Windows. So in this short post I wan’t to write about how this works on windows. Maybe there are already some nice guides about this topic, but I didn’t found them. Writing about this myself helps me to document my work and maybe it helps somebody else.

I really have a problem with finding a name for this technique. If you know a common name for it, please write me on Twitter. Until then I stay with “running dynamic code” :)

What I want to achieve

I want to be able to supply some raw opcodes to a function and let them be executed at runtime. This can be useful for example in a JIT compiler, where the sourcode is compiled just before the program (or even function) is run (just-in-time). In such a compiler the required opcodes are normally generated on the fly and not saved in a (executable) file but stored in memory.

Another usage example would be a compressed or encrypted executable. There are some niche usecases for this, like some maleware that wan’t to disguise their presence, or systems with really low persistent memory but enough RAM to decompress the executable at runtime. A compressed or encrypted executable would need an uncompressed loader that first uncompresses/decrypts the main application and then launches it.

At the moment I’m just interested in this technique by itself without any real usecase. But maybe in the future I might look into JIT compiling and executable compression.

Limitations

First a disclaimer: This technique is more of an expert/niche thing and should not be used without a second thought. You may even introduce a (many) security flaws or hard to debug code. In most cases there are better ways to solve an problem than this. For exmple, if you wan’t an application wich allows for user-modification like plugins or mods, you could use a dynmic library (.dll/.so/.dylib) which is loaded at runtime. Or even an external but production ready scripting language like Lua.

Another downside of this technique is that it is heavily unportable. At least, you need to generate the right opcodes for the processor architecture you’re using. For example, the opcodes for an x86-64 processor are completly different then for an ARM processor. On some architiectures you need to apply extra steps like clearing a special cache for this to work. Some architectures even don’t support this technique directly, like AVR. AVR uses different memory for code and data, and you would need to reflash your code memory on the microcontroller at runtime. This is indeed possible, but it would havily reduce the lifetime of such a microcontroller, as their flash memory isn’t really optimized for this usecase and has normaly a guranteed lifetime of about 10,000 write-erase cycles 1.

But even if you know on what processor it runs, you may need to account for differences in the operating system, espacially if you want to directly call standard library functions.

If you want to use this technique in production I would recommend to look for a library which helps with running code in a more cross-platform manner.

How it works (on Windows)

Lets get our hands dirty! In this post I will only look at the function which runs the opcodes but you can find the full sourcecode on GitHub.

#include <stdio.h>
#include <stdbool.h>
#include <Windows.h>

/**
* Helper function to execute dynamic code.
* 
* @param code a pointer to the code to execute
* @param length the length of the code
* @return true if the code run successfully, otherwise false
*/
bool run_code(const void* code, size_t length)
{
	// Allocate new memory for code execution.
	// We don't set the execute flag because we won't want to
	// open doors to malicious software.
	// dynMemory must be declared as volatile, as some Release
	// optimizations cause problems on VirtualFree. Maybe some
	// sort of reordering.
	volatile LPVOID dynMemory = VirtualAlloc(NULL, length, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
	if (dynMemory == NULL)
	{
		printf("Alloc Error: %ld\n", GetLastError());
		return false;
	}

	// Copy code to new memory.
	memcpy(dynMemory, code, length);

	// Change protection.
	// We remove the write flag and set the execute flag on that page.
	DWORD oldProtection;
	if (!VirtualProtect(dynMemory, length, PAGE_EXECUTE, &oldProtection))
	{
		printf("Protect Error: %ld\n", GetLastError());
		return false;
	}

	// Call the new function in memory.
	int result = ((int (*)(void))dynMemory)();
	printf("Result: %d\n", result);

	// At the end we free the memory.
	if (!VirtualFree(dynMemory, 0, MEM_RELEASE))
	{
		printf("Free Error: %ld\n", GetLastError());
		return false;
	}

	return true;
}

This function takes a pointer to some raw bytes and the length of it. It then uses the WinAPI function VirtualAlloc to allocate some memory pages for our code to live in. VirtualAlloc allocates enough whole pages so our length-long code will fit. These new pages are declared as readable and writable but not executable. This is because a memory page should never be executable AND writable at the same time. This is called W^X (write XOR execute) and you can read more about it here. The result of VirtualAlloc is a pointer to our newly allocated memory and it is saved in an volatile LPVOID pointer. The volatile keyword is needed because it disables some optimizations that caused the program to crash when compiled with msvc in release mode. In debug mode everything workes as expected even without volatile. To be honest, at the moment I don’t know why it breaks in release, maybe some sort of instruction reordering? But it is just another example why you should be careful with this technique. If somebody knows why this breaks in release, please write me, I really would like to know.

In the next step we copy our code from the supplied pointer to the newly created, page-aligned memory. After we copied our code, we don’t need to modifiy the memory anymore. Because of this we use VirtualProtect to modifiy the memory permissions to executable but not writable. The old permissions will be saved in oldProtection. We aren’t interested in these, but without suppling a valid pointer as the fourth parameter to VirtualProtect, the function will fail according to the documentation.

After this we can jump into our code. We do this by casting the pointer to the code to a function pointer and then calling it directly. The syntax for this is at least cryptic so let me go through this in more detail:

((int (*)(void))dynMemory)()

Here we cast our pointer to a function pointer which doesn’t expect any parameters and returns an integer. The (void) denotes that this function doesn’t take parameters. The (*) with the paranthesis is required for a function pointer cast. The int at the beginning denotes as expected the return type. And lastly the paranthesis at the end is there because we want to call this function directly.

On 64bit Windows the integer return value of a function is expected to be in the EAX register, so you opcode should put a meaningful value in this register. This is the easiest way to communicate with the calling code from within the dynamic code. If you’re not interested in a return value, you can change the type of the function in the cast to void. On the other side, if you want to supply the dynmic code with values, you can define parameters instead of void. The example code on GitHub performs an addition and subtraction and then returns the value in eax. The result is printed directly after the call in the printf.

Make sure you don’t run code from an untrusty source. It will run with the same permissions as the calling process/thread.

Last but not least we need to free our code memory. This is done with a call to VirtualFree.

Links

Full sourcecode on GitHub: GitHub Gist

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/run-dynamic-code-windows/.

Writing Images: Windows Bitmap

Sun, 14 Mar 2021 08:00:00 GMT

Writing Images: Windows Bitmap

Last time we looked at the very simple Portable Anymap format to easily write images. But there is a catch with portable anymaps: they are a little but uncommon. While macOS and most Linux desktops support them even with thumbnails, there is no default support on Windows. Also, most webbrowsers don’t support them either. They are also a little bit unknown in non-development environments. This time we’ll look at a much more common file format: Windows Bitmap.

Windows bitmaps with their iconic file extension .bmp are undestood by almost every software that processes images. As you will see they are just a little bit more complicated than Portable Anymaps and thus easy to write and read. On the other side, BMPs have a lot of restrictions which limit their use. One limit is, that BMP cannot save any transparency information. Another problem is the size of BMPs because they normaly save uncompressed image data.

The Bitmap File Format

We won’t look at every detail of the bitmap file format but focus on the important features and properties so that we can write our own uncompressed bitmaps easily and fast. If you want are more indepth look at this file format I recommend the Wikipedia article.

The bitmap file format is a completely binary file format. This isn’t bad at all, in fact, while it is a little more inconvienient for humans, it makes the data size smaller and its format is exactly descibed. You will see that there is no black magic required to write such binary bitmap files. There is just one thing you need to be aware: the byte order. I’ve already written an article about that topic. Windows bitmaps always save their multibyte data as little endian.

A Bitmap starts with two headers, one that describes the file itself (called the BITMAPFILEHEADER in the documentation) and one that describes the saved image (the BITMAPINFOHEADER). You always need to write both, but technicaly it doesn’t make a difference if there were one big header or two smaller ones.

In the next two subsections I present you the members of these headers. It may look a bit overwhelming at first, but stay with me. You will see that many of them are always constant or are easily filled. If you don’t understand everything at first you can also jump directly to the code, it will help you understand everything.

One last word about the data type informations in these tables. These types are from the Microsoft documentation and are commonly used in Windows code. They are in fact just different names for common datatypes. To write bitmaps, you only need to understand what they represent. Here is a short table:

Type	Description
WORD	2 byte / 16 bit unsigned integer
DWORD	4 byte / 32 bit unsigned integer, DWORD means DOUBLE WORD
LONG	4 byte / 32 bit signed integer

BITMAPFILEHEADER

This header is the first thing found in a bitmap file.

Name	Data type	Size	Short description
bfType	WORD	2 bytes	This is always `"BM"`
bfSize	DWORD	4 bytes	The filesize in bytes
bfReserved1	WORD	2 bytes	Reserved and unused, must be 0, but some software uses it regardless
bfReserved2	WORD	2 bytes	Reserved and unused, must be 0, but some software uses it regardless
bfOffBits	DWORD	4 bytes	Offset to the image data from the beginning of the file

These names are directly from the Microsoft bitmap documentation. It is linked at the Format specifications and Links section.

BITMAPINFOHEADER

Directly after the BITMAPFILEHEADER follows this header.

Name	Data type	Size	Description
biSize	DWORD	4 bytes	size of this header, here we can use the constant value 40
biWidth	LONG	4 bytes	width of the image
biHeight	LONG	4 bytes	height of the image, but this value can be negative, I come to this later
biPlanes	WORD	2 bytes	not used anymore, must be set to 1
biBitCount	WORD	2 bytes	bits per pixel, can be 1, 4, 8, 16, 24 or 32
biCompression	DWORD	4 bytes	which compression is used, can be 0, 1, 2 or 3
biSizeImage	DWORD	4 bytes	the size of the image data in byte, can be 0 if no compression is used
biXPelsPerMeter	LONG	4 bytes	horizontal resolution in pixels per meter, often just set to 0
biYPelsPerMeter	LONG	4 bytes	vertical resolution in pixels per meter, often just set to 0
biClrUsed	DWORD	4 bytes	if no color table is used, this can be set to 0
biClrImportant	DWORD	4 bytes	if no color table is used, this can be set to 0

In this header are some things that need to be addressed in greater detail. Let’s start with the most wierd thing: The height can be negative. If the height is positive, the image data is expected to be written from bottom to top. I don’t know why this was decided, but it seems that this is the most common form bitmaps are saved. Some of the compression options are also only available, if this direction is choosen. If the height is negative the image is expected to be written in the IMHO more natural top to bottom direction.

Now we look at the biBitCount value. It describes how many bits are expected for a single pixel. The values 1, 4 and 8 are for indexed colors. This means we have a color table where we define all colors that can appear in an image. We then only save an index of an color in this table as the actual image data. We won’t look into the benefits and problems with this approch today, as we only look at 24 bits per pixel. This is the most simple way to write images because every channel (red, green and blue) get a full byte per pixel (3 * 8 bit).

Data

And now the best thing: as we save bitmaps uncompressed we can easily write out our pixel data. A pixel with only 0 bytes represents black and only 255 bytes represents white. But there are two catches: bitmaps save BGR (blue-green-red) data, not RGB (red-green-blue). Today you probably save your images in memory in RGB so you need to inverse the byte order before putting out a pixel.

The second catch is the already described order of rows, from bottom to top. If your image data is already ordered this way, you can easily dump them out to a file. But if you save them from top to bottom, you need to reverse them while writing or flip them before. Flipping beforehand may have some performance benefits by utilising the processor cache lines better, at least if implemented correctly. But in this article we just flip the order while writing it, even if it may not be the best performing solution. My focus in this post is more on simplicity than performance.

Implementation

As I did with the portable anymap I show you now a simple implementation of a windows bitmap writer in C.

#include <stdio.h>
#include <stdint.h>

#define WIDTH 300
#define HEIGHT 200

// This changes the padding behaviour of the
// compiler. I come to this later.
#pragma pack(push, 1)

typedef struct {
    uint16_t bfType;
    uint32_t bfSize;
    uint16_t bfReserved1;
    uint16_t bfReserved2;
    uint32_t bfOffBits;
} BitmapFileHeader;

typedef struct {
    uint32_t biSize;
    int32_t biWidth;
    int32_t biHeight;
    uint16_t biPlanes;
    uint16_t biBitCount;
    uint32_t biCompression;
    uint32_t biSizeImage;
    int32_t biXPelsPerMeter;
    int32_t biYPelsPerMeter;
    uint32_t biClrUsed;
    uint32_t biClrImportant;
} BitmapInfoHeader;

// This restores the padding behaviour.
#pragma pack(pop)

int main()
{
    FILE* outfile = fopen("out.bmp", "wb");

    // Both headers have a combined size of 54 bytes.
    // We need WIDTH * HEIGHT pixels and each of them needs
    // three bytes (one for each channel).
    uint32_t filesize = 54 + WIDTH * HEIGHT * 3;

    // First we fill the file header.
    BitmapFileHeader fileHeader;
    fileHeader.bfType = 'B' | 'M' << 8; // reversed because of Little Endian 
    fileHeader.bfSize = filesize;
    fileHeader.bfReserved1 = 0;
    fileHeader.bfReserved2 = 0;
    fileHeader.bfOffBits = 54; // both headers have a size of 54 byte

    // Next we fill the info header.
    BitmapInfoHeader infoHeader;
    infoHeader.biSize = 40; // this header is 40 bytes long
    infoHeader.biWidth = WIDTH;
    infoHeader.biHeight = HEIGHT;
    infoHeader.biPlanes = 1; // must be 1
    infoHeader.biBitCount = 24; // 24 bit per pixel = 1 byte per channel
    infoHeader.biCompression = 0; // no compression
    infoHeader.biSizeImage = 0; // 0 because we don't use compression
    infoHeader.biXPelsPerMeter = 0;
    infoHeader.biYPelsPerMeter = 0;
    infoHeader.biClrUsed = 0;
    infoHeader.biClrImportant = 0;

    // Then we write them both to the file.
    // This only works because of the padding changes
    // and ignores endianess. Normaly you should NOT
    // do this, but this simplifices this example.
    fwrite(&fileHeader, sizeof(BitmapFileHeader), 1, outfile);
    fwrite(&infoHeader, sizeof(BitmapInfoHeader), 1, outfile);

    // We write the image from bottom to top.
    // In this example the image data is created 
    // procedurally, so we really don't need to do it
    // this way. But if you would read the image data
    // from an array, you could use the following loops:
    for (int y = HEIGHT - 1; y >= 0; y--)
    {
        for (int x = 0; x < WIDTH; x++)
        {
            // Image data creation:
            int r = (x / (float)WIDTH) * 255;
            int g = (y / (float)WIDTH) * 255;
            int b = (y / (float)HEIGHT) * 255;

            // We write a single pixel in BGR order
            unsigned char colors[3] = {b, g, r};
            fwrite(colors, 3, 1, outfile);
        }
    }

    fclose(outfile);
    return 0;
}

This example outputs the following image: I actually converted the original BMP to PNG for a smaller size and because my site is not configured serve BMP files.

You might have noticed I used types like uint16_t from stdint.h instead of WORD, DWORD, LONG. This makes this code more portable as the other types are Windows specific. If you already include windows.h, you can easily use those types instead. Actually, you might use BITMAPFILEHEADER and BITMAPINFOHEADER directly.

Because this is only an example how to write a bitmap file, I used code to create an image procedurally. And in this case the orientation of the image doen’t really matter. Normaly it would, so for demonstration purposes my Y-loop writes the data from bottom to top, as you would with an actual image, maybe read from an array oder rendered via raytracing on the fly. I hope this clarifies the writing procedure.

Last but not least I need to talk about padding, packed structs and why this example is not very portable. Maybe you’ve already noticed the #pragma pack(...) directives in this example. The reason they exists is, that C structs may have something called a padding. Simply put, a padding is introduced in a struct to align the members of a struct to adresses that are dividable by some factor. For example a 2 byte uint16t is aligned to an even address while a 4 byte uint32t is aligned to an address divisible by 4. If a 2 byte member in a struct is followed by a 4 byte member, a 2 byte padding is introduced, so that the second member is correctly aligned. Excactly this happens with the BITMAPFILEHEADER in the example code between bfType and bfSize. This causes the struct to have a size of 16 instead of 14 bytes. This will corrupt our header so we can’t read our image. Padding is normaly useful, because it allows optimised memory access. But if we wan’t to write a struct directly to a file, it causes havok.

The solution to this is using a packed struct. A packed struct does not contain any padding, so we can write it out without problems. Here we come to the first problem: There is no standarised way to declare a packed struct. I choosed the #pragma pack(...) directive, because it seems to have been adopted by many compilers. The #pragma pack(push, 1) directive saves the current packing status on a stack and also sets it to 1, which means that we disable padding completly. The #pragma pack(pop) restores the old status.

There are a lot of problems with this solution:

I’ve already written that it is not standarised. If your compiler doesn’t support it, your code wouldn’t work.
Accessing struct membery that are not aligned may be slow. This isn’t that important if you just use this struct to save one image file, but if you use the same struct for some calculations it may hurt your performance.
It is also heavily coupled to the x86 processor architecture.

The x86 architecture is really common in personal computers, but it is by far not the only architecture out there. Padding and alignment are specifics of the processor architecture. Some architectures even require some sort of alignment, or you can’t access the membery. Also we ignored the endianness completly, because x86 uses little endian, as the BMP file format do. But on big endian systems all multibyte members need to be swapped.

Using the solution presented in the example really helped to keep the code short and understandable. It is also okay for small experiments or demos. But for anything just a little bit more important you shouldn’t do it that way. Most of the time it is better to output each member of the struct seperatly (preferably with some buffering). This way you have a lot of repetitve code but on the other hand you don’t need to deal with padding or alignment and also you can deal with endianess on each struct member as needed.

Format specifications and Links

Wikipedia
Microsoft Docs: BITMAPFILEHEADER
Microsoft Docs: BITMAPINFOHEADER
Sourcecode on GitHub: GitHub Gist

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/writing-bmp-files/.

Writing Images: Portable Anymap

Thu, 29 Aug 2019 14:00:00 GMT

Writing Images: Portable Anymap

Most of the time, image formats are somewhat complex and require some sort of compression. But there are some exceptions and Portable Anymaps are such an exception. Portable anymaps are not a single file format but a group of formats for saving Bitmaps, Grayscale and RGB images.

There are 7 different formats:

Format	Magic Number	File Ending
ASCII Bitmap	P1	.pbm
ASCII Graymap	P2	.pgm
ASCII Pixmap	P3	.ppm
Binary Bitmap	P4	.pbm
Binary Graymap	P5	.pgm
Binary Pixmap	P6	.ppm

The seventh format is a generalization of the binary types and we just ignore it in this article.

ASCII

The ASCII format is really simple. Ineed, it is so simple that you can export images in this format from every language that can output text to a file or console. Yes, you could even use Brainfuck to export portable anymaps.

All portable anymaps begin with a ASCII header in the following format:

MAGIC_NUMBER
WIDTH HEIGHT
MAX_VALUE

MAGIC_NUMBER is one of the 7 format identifiers we already looked at. WIDTH and HEIGHT are simply the width and height of the image in pixels. We look at MAX_VALUE a little bit later.

The fields in the header are seperated by a whitespace so you could even write every field in its own line or write the whole header in one line.

The actual image data begins right after the header. They are encoded as human readable ASCII numbers seperated by whitespace. Bitmaps can just use the 0 and 1 character, graymaps and pixmaps can use numbers from 0 to MAX_VALUE. Graymaps use one number per pixel while pixmaps use 3 numbers per pixel, one for each channel. The channel order is red, green, blue.

While not required, it is common to spereate lines in the image with a newline and also use a padding for smaller numbers so they align in columns. This makes it easier to read by humans. Look at the examples on this page to better undestand what I mean.

Comments

Portable Anymaps can also include comments. Comments start with a # character and end with a carriage return or newline.

Depth

Now it is time to look at the MAX_VALUE header field. It describes what the highest possible value in an image may be. It must be greater than 0 and less than 65536. In a graymap, a pixel with a value of MAX_VALUE would be white. In a pixmap, if all three channels of a pixel are MAX_VALUE, it also would be white. The most common value for MAX_VALUE is 255, because this is the maximal value one byte can hold.

Because portable bitmaps can only have 0 and 1 as values, there is no MAX_VALUE field for them.

Binary

While ASCII is human readable and easy to output, it does take up a lot of space. Even for a single value we need 2 bytes (1 whitespace + 1 data byte) but it can easy be more. Saving the same file in the binary mode we can save a lot space but we sacrifice readability.

To output a portable anymap in binary you just output the header in ASCII but directly after the last whitespace of the header we output the image data. Make sure that you don’t output any whitespaces after the last one in the header. They get interpreted as image values.

There is a little catch when using the binary mode: If MAX_VALUE is less than 256 the values are saved in one byte per pixel in graymaps and one byte per RGB channel in pixmaps. If MAX_VALUE is equal or greater than 256 values are saved with two bytes using Big Endian.

Implementation ASCII

There is nothing really special about the implementation of a portable anymap writer. I’ve decided to show you a simple example using C and writing a RGB image to a file. You can also find a C++ sample in the full sourcecode. The code should be easy to read even for a beginner.

#include <stdio.h>

#define WIDTH 300
#define HEIGHT 200

int main()
{
    FILE *outfile = fopen("out.ppm", "wb");

    fprintf(outfile, "P3");

    fprintf(outfile, "\n%d %d\n255\n", WIDTH, HEIGHT);

    for (int y = 0; y < HEIGHT; y++)
    {
        for (int x = 0; x < WIDTH; x++)
        {
            int r = (x / (float)WIDTH) * 255;
            int g = (x / (float)WIDTH) * 255;
            int b = (y / (float)HEIGHT) * 255;

            fprintf(outfile, "%3d %3d %3d ", r, g, b);
        }
        // Output newlines so the result is a little bit more readable.
        fprintf(outfile, "\n");
    }

    fclose(outfile);
    return 0;
}

Implementation Binary

This implementation isn’t very different from the ASCII version. Notice that the header is still saved as ASCII characters.

#include <stdio.h>

#define WIDTH 300
#define HEIGHT 200

int main()
{
    FILE *outfile = fopen("out.ppm", "wb");

    // Now it is P6 instead of P3
    fprintf(outfile, "P6");

    fprintf(outfile, "\n%d %d\n255\n", WIDTH, HEIGHT);

    for (int y = 0; y < HEIGHT; y++)
    {
        for (int x = 0; x < WIDTH; x++)
        {
            int r = (x / (float)WIDTH) * 255;
            int g = (x / (float)WIDTH) * 255;
            int b = (y / (float)HEIGHT) * 255;

            unsigned char colors[3] = {r, g, b};
            fwrite(colors, 3, 1, outfile);
        }
    }

    fclose(outfile);
    return 0;
}

Examples

Here are some examples using ASCII anymaps. They are all written by hand.

Bitmap

Graymap

Pixmap

P3
3 3
255
0   0   0   100 100 100  255 255 255
255 0   0   0   255   0    0   0 255
255 255 0   0   255 255  255   0 255

Full sourcecode

You can find the full sourcecode for C and C++ on GitHub.

Format specifications

Portable Bitmap: PBM
Portable Graymap: PGM
Portable Pixmap: PPM

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/writing-portable-anymap/.

Byte order: Big Endian vs. Little Endian

Thu, 15 Aug 2019 20:00:00 GMT

This article will lay the foundation for further articels on the topic of writing binary files.

So, what exactly do I mean with byte ordering? What is this Endianness?

Byte ordering

When writing down numbers, for example 153, we compose the number of multiple digits. In this case 1, 5 and 3. But just knowing these digits isn’t enough. Also their position in the number, their ordering, is important. While the 3 alone has a higher value than the 1, in our example the 1 means one hundret, because it appers on the first of three digits. In other words, it is the most significant digit, while the 3 is the least significant digit.

The memory of a computer is composed of bytes, which can hold a number between 0 and 255. If you want to save bigger numbers, you can just put more bytes together. Two bytes (or 16 bit) can hold a value between 0 and 65535. But here is the interesting thing: In which order do you save those 2 bytes? Should the most significant byte come first in memory, like we write down our numbers?

There is no right answer to this questions as both approches were taken on different processors. The ones who save the most significant byte first are called big endian and the ones who save the least significant byte first are called little endian. In the past there were even some mixed endian systems with a totally crazy byte layout but luckily they didn’t take over so we ignore them.

Today the most personal computers are using little endian because it’s the endianness of the widespread Intel x86-architecture. You should keep this in mind if you ever analyze raw byte data. The bytes might be in a different order than you first expect.

If you want to read more on this topic, look at the Wikipedia article, it features a lot more details.

Why is this important for binary file formats?

The ordering of the bytes in memory isn’t really important. You calculations will work and you don’t need to think about it. Problems arise when you try to send data from one computer to another with a different byte ordering. This happens a lot when dealing with internet traffic or file formats. They both are designed for sharing between different computers.

So we need to deal with different byte orderings. How? There are two solutions:

Allow both orderings and detect while reading which one was used.
This way you can always write with your own ordering but if you try
to read a file you might need to correct the ordering.
For detection many formats use some mark like the BOM of Unicode files.
Just specify for the format which byte ordering should be used.
Sounds simple and it is! Indeed, most network protocols took this approach
and decided that everything sent should be in big endian. Also many file formats
we look at decided to use just little endian. In this scenario we may need to convert
the data on writing and reading, if we use the “wrong” ordering.

To recap, we only need to correct data if:

we have multibyte data. If everything just uses a single byte we don’t need a correction.
we detected that our systems order doesn’t match the needed order.

How do we detect the ordering?

I will use C++ for this articles code.

So first, lets write functions which allow us to detect the byte order:

union _endian_test {
    uint16_t word;
    uint8_t byte[2];
};

int isLittleEndian() {
    _endian_test test;
    test.word = 0xAA00;

    return test.byte[0] == 0x00;
}

int isBigEndian() {
    _endian_test test;
    test.word = 0xAA00;

    return test.byte[0] == 0xAA;
}

Whooaa, what is happing in this snippet? The most important bit is the union at the beginning. A union is a little bit like a struct, with some important difference: Every member uses exactly the same memory. A union is a little bit special and they aren’t as useful as structs or classes. But for this task they shine!

Let’s concentrate on the isLittleEndian(): We first create a new endiantest and set the 16 bit unsigned int word to 0xAA00. This value was picked randomly, every other value would have worked, as long as the first byte is different from the second one. Because one byte can be representated by exactly two hexadecimal digits it is easy to work with them when the ordering of bytes is more important than the content itself.

And now the magic: if test.byte[0] == 0x00 is true, we know we are on a little endian system. Why? Because we use a union, test.byte[0] uses the exact same memory as the first byte of test.word. test.byte[1] would have used the same memory as the second byte of test.word.

If this first byte is 0x00 and we put 0xAA00 in word, this means that the first byte corresponds to the least significant byte.

In isBigEndian() we use the same logic but check if test.byte[0] == 0xAA which would mean that the first byte corresponds to the most significant byte.

And how do we correct the order?

To correct the order, we just need to swap the bytes. The important thing here is that you don’t do anything like adding, subtracting or so when the ordering of the data isn’t the same as the one of your system.

Let’s look at a simple byte swap for 16 bit integers:

uint16_t swap16(uint16_t in)
{
    return (in >> 8) | (in << 8);
}

This is short. We just shift all the bits by 8 positions in the input one time to the left and one time to the right. Then combine the two with a binary OR. Shift operations on unsigned integers always shift zeros in. This doesn’t work for signed ints, so we always need to cast them to unsigned before we swap them.

This is what’s happening step by step:

0xFF03 <=> 1111 1111 0000 0011
1111 1111 0000 0011 >> 8 => 0000 0000 1111 1111
1111 1111 0000 0011 << 8 => 0000 0011 0000 0000
0000 0000 1111 1111 OR 0000 0011 0000 0000 => 0000 0011 1111 1111
0000 0011 1111 1111 <=> 0x03FF

I’ve implemented this swapping also for 32 bit and 64 bit but I just include them without explanation. The idea is really the same, it’s just more bytes. In case of 64 bit I’ve reused the swap32 bit to avoid using tons of zeros in the source. This way I save some error prone repetition. Try to understand what is happening yourself.

uint32_t swap32(uint32_t in)
{
    return  ((in & 0xFF000000) >> 24) | ((in & 0x00FF0000) >>  8) | 
            ((in & 0x0000FF00) <<  8) | ((in & 0x000000FF) << 24);
}

uint64_t swap64(uint64_t in)
{
    uint64_t a = swap32(0xFFFFFFFF & in);
    uint64_t b = swap32(((0xFFFFFFFFL << 32) & in) >> 32);

    return (a << 32) | b;
}

So now we can detect the endianness of the system and we can swap bytes. Great! This is really everything we need. The most of the time, at least when writing data, we know what the endianness of the data should be but we don’t know our own endianness. So we should write helper for that job that swap the bytes only if we have a different endianness then we want.

uint16_t toLittleEndian16(uint16_t in) 
{
    return isLittleEndian() ? in : swap16(in);
}

uint16_t toBigEndian16(uint16_t in)
{
    return isBigEndian() ? in : swap16(in);
}

Nothing really special. I’ve excluded the four functions for 32 bit and 64 bit but you can find them on GitHub because I’ve bundled the code of this article as a SHL and uploaded it on GitHub: nh_byteorder.hpp.

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/byte-order/.

Binary file formats introduction

Wed, 14 Aug 2019 23:34:00 GMT

This article is just an introduction to a complete series of articles about binary file formats.

In this series we will learn about the structure of many different binary file formats and how to read and write them. First we will focus on writing various binary image file formats, because we can use this knowledge in many projects on this blog.

But why should you be interested in such formats? Today the most data is either in XML or JSON and if I really need to write a binary file there is definitely a library to do so, right?

Yes, exactly. And you should use them! They are well tested and have many more features than anything we will build in this series. But using something doesn’t mean that you really understand it. That is, if you want to understand it at all. I was fascinated about the inner workings of ANYTHING since I was a young boy. And there is no better way in understanding something than building it yourself. Your implementation doesn’t even need to be perfect or well optimized. As long as your implementation works it’s good enough.

If you’re not as that interest in the inner workings of some of the most common file formats we use every day, then this series will just bore you and you shouldn’t read it.

But if you’re at least a little bit like me I will take you to a journy of understanding some of the genius techniques used in everyday file formats. We will first look at the simpler formats like Portable Anymaps and Bitmaps and then gradually go over to the more complicated formats. While we first stick to images, I plan to look at other formats like .tar or .zip Archives later.

The article about the byte order which will come out soon is the foundation of many other articles in this series.

So stay tuned!

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/binary-file-formats/.

Single Header Libraries

Wed, 14 Aug 2019 19:20:00 GMT

This article is targeted towards beginners. It should clarify what a single header library (SHL) is and how you could create our own. When I use or create single header libraries in later articles I can just link to this article for clarification.

This article is about a concept in C and C++, the most other languages don’t even have header files so you can’t create a single header library.

To understand what a single header library (SHL) is, first we look at the single parts of this name one by one. We start with the last one, because it has the most meaning.

Library

Almost every langauge have some sort of libraries. In general one could say, a software library is a collection of functions and datatypes, that can’t normaly run directly. Instead it provides it’s features to other software (programs and other libraries). This allows you to use the same code in different projects easily. A user of a library only needs to know the public interface, not the internal workings. Examples for libraries are the C++ STL or libSDL.

If you ever used C or C++, you should know what a header file is. But for the sake of completeness:

In C/C++ a function must be deklared befor its name can be used in any way. This not only holds for functions in the same source file (conventionally .c or .cpp) but also applies to functions from different source files or external libraries. To declare them, it is enough to write down the function header with the parameters and return type:

// Declaration
int add(int, int);

int main()
{
    // Usage
    printf("%d", add(1, 2));
    return 0;
}

// Definition / Implementation
int add(int a, int b) {
    return a + b;
}

As you may have spotted, it is even enough to just write down the types of the parameters, their names are optional in the declaration. Still you normaly write them down too, so the function documents itself.

Declaring the function header for all functions you want to use, especially if they come from libraries, would be really cumbersome. For this reason C and C++ use so called header files (conventionally .h or .hpp). They include normaly no code to run (although that would be very possible), instead they include the function headers of alle functions that are part of the public interface of some software module. These header files can then be included in own source files by using #include.

Single

The “single” just means in this context that our complete library is contained in a single file so it can be easily embedded in different projects.

And now everything together!

Going by these parts of the name a SHL is nothing more than a software library which consist of a single header file. That’s it.

Why are single header libraries useful?

Libraries are part of the daily life of a developer. But often it can be cumbersome to embed them in new projects (at least in C and C++). First, you need to compile it for all your target platforms by yourself. The build processes may differ from library to library and often the setup and compilation takes a lot time and don’t even work on the first try. Some libraries even need other dependencies to work properly. In general, many C and C++ libraries have a somewhat difficult setup which may need to be done more than once for each platform.

SHLs have a solution for that: They consist of a single file that can be copied easily in any project. There is no big setup, they just use the build process of your own project to compile themselves. They can also be included in the version control system so they are available for everyone trying to build your project.

This a big plus for open source projects which will be used by many people around the world.

In which situation are they not useful?

But even SHLs have their limits. One big problem is that you need to recompile the whole library for every computer in every project. This takes time. Classic static or dynamic libraries can be compiled once for each target platform and then the build products (.lib, .so, .dll, .dylib, etc.) can be copied between computers.

In my opinion, the scope of a library is the strongest criteria to decide if a SHL or a classic library should be the way to go. Where the limit is, however, must be determined individually for each project.

How do you include/use a SHL?

The exact usage naturally differs from SHL to SHL but there are some similar patterns.

You decide on a source file in your project which is going to include the implementation of the whole SHL and there you define a preprocessor constant before including the SHL. If other source files in your project want to use the library they just need to include it without defining the constant.

Let’s look at an example: The SHL ”stb_image” allows to read many image formats in a C/C++ program.

First we need to implement the library in a C/C++ source file:

#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"

In this file the stb_img.h header will expand all the implementations of the reading functions. Every other source file, which also want to read images can just include this header again but without setting STB_IMAGE_IMPLEMENTATION:

#include "stb_image.h"

Often you will observe that the file which includes the implementation will take a lot longer in the compilation step. This is logical because even if we don’t see it, the SHL injects a lot of functionality in this file. For this reason you should use a build system like make which onle recompiles source files if their contents changed since the last compilation. Additionally you should choose a source file for the implementaiton that doesn’t change that often, maybe even a special file just for this SHL.

How do you create your own SHL?

First, we need some code which can be used usefully in different projects. For this article, we expect that we already have something like that.

After that you must decide if your library should work with C and C++ or just for one of these languages. If you use C++ only features like classes your library definitely won’t work with C code. A library written in C can be used in C++, but it often feels not nativ. The usage of such a library just don’t feel “C++ly”. With the help of the preprocessor it is possible to write a SHL wich works behves for C and C++ like anyone would expect but this may increases the effort a lot.

Beside that, writing a SHL is really easy thanks to the preprocessor. Let’s look at a example with a ~~useless~~ fictional math library:

#ifndef MINI_MATH_H
#define MINI_MATH_H

/* Public Interface */

int add(int a, int b);
int sub(int a, int b);

#ifdef MINI_MATH_IMPLEMENTATION_H

/* Implementation */

int add(int a, int b)
{
    return a + b;
}

int sub(int a, int b)
{
    return a - b;
}

#endif /* MINI_MATH_IMPLEMENTATION_H */

#endif /* MINI_MATH_H */

Thats all to create your very own SHL. It’s important that only on .c/.cpp file defines MINI_MATH_IMPLEMENTATION_H, otherwise you get linker conflicts when building your project.

If you want to use utility functions in the implementation that should not be part of the public interface place them in the MINI_MATH_IMPLEMENTATION_H part. THis also holds for external headers which are not needed for the interface.

There are some different approaches in the development of a SHL. For example you could use a program that bundles many files into a single header file. This allows you to split your library code in meaningful submodules without loosing the comfort of a SHL after the bundling.

This is the end of this first small article.

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/single-header-libraries/.

First Blogpost

Tue, 13 Aug 2019 14:38:00 GMT

echo "Hello World!"

This is my first entry in this blog. Here I will answer some general questions about this blog.

Why do I want to write a blog?

I really love to read articles on Medium, Dev.to or blogs by other developers. Often I get the desire to write about some themes myself. Other than that, I like to teach people new skills. Because of that I decided to start my own blog.

I will experiment with publishing articles from this site on Medium and Dev.to too. I don’t know if I keep it that way.

What is this blog about?

Everything I like to write about. I don’t want to restrict myself on this term. In gerneral, the content will be of a technical nature, most of the time about software dvelopment. Some themes will be more adequate for beginners, others for intermediate to advanced users. The most articles are expecting that you already can progam. Some of the articles will be more like a tutorial, that you can work through. Otherwise they are about things I want to learn myself so they only describe how I approch something.

Who am I anyway?

To answer this questions I have the about me site. Currently it isn’t really filled with content but I will change that. Under projects you can find a selection of projects I’ve worked on in the past.

How can you reach me?

Best via Twitter, because I look there regulary. DMs are open for everyone, but you just can tag me on a twwet, I will read that too. Also, for private requests you can reach me with the email in the legal notices. It may take some time until I answere there.

I really would appreciate feedback on the articles on this blog. At the moment I don’t have a comment feature because I don’t want to embed external providers like Disqus. Maybe I write an own system for that, but for now there aren’t any comments possible.

Which system do I use for this blog?

This site was created with Gatsbyjs. It is a static site generator which takes React applications written in JavaScript and builds HTML, CSS and JS Bundles from that. This allws me to use a simple WebServer without any serverside logic to operate this blog. I write the articles as Markdown files. Who knows, maybe I write an article about Gatsby ;)

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/first-blogpost/.

Project: Grenzhus Geo

Mon, 12 Aug 2019 21:53:00 GMT

Introduction

Grenzhus Geo is an Android App I developed back in middle school together with my classmate Adrian. It was a voluntary project to improve our history grades. Although it’s really some time ago (the app was last updated on November 23, 2014) I thought it could be interesting to write about it and recap our results.

So, whats the point of the app?

The App

Our school was located not far away from the former inner german border which split the DDR from the BRD. We contacted a border museum called ”Grenzhus” in Schlagsdorf. With their help we developed a concept for a geo-caching like app where the user needs to find barcodes which we placed on some historical points along the former border. Each point has its own descriptive text which included some historical facts about that point.

This way a user can walk around the area and search for barcodes while also learning something about history.

Technical challanges

I think one of the biggest challanges was reading out the GPS sensor and converting its coordinates to the correct pixel spots on screen. We decided to embed a own, simplified map and not Google Maps or Open Street Map. The app only makes sense if you are near Schlagsdorf so there is no need for a fancy, scrollable map.

To convert the coordinates we just used a linear interpolation between some points we determined empirically. This is definitly the wrong way to do it but the total area of we needed to support wasn’t that big it did work really good.

Screenshots

The map view	The detail view

Conclusion

It was a fun project and one of my first contacts with Android App development. I’ve learned about our history and also about geological coordinates.

A big problem was the missing iPhone support. While it wasn’t possible to spend 79€ at that time for a yearly apple developer membership, the missing support excluded about half of our classmates.

Also, in retrospective the design of the app was really ugly.

A big shout out to our history teacher and Dr. Wagner who gave us this oppertunity and supported us during the development.

Links

The App is not maintained anymore, neither digital nor were the barcodes replaced. But you can find it on the Google playStore.

Here are some german news articles regarding our app:

This article was posted to my blog at nicolashollmann.de. You can read it online at https://nicolashollmann.de/blog/grenzhus-geo/.