Tix3Dev’s Blog

apoptOS-Series-1: Introduction

2022-06-06T00:00:00+00:00

Foreword

Whether or not you already are familiar with apoptOS, this blog post will talk about the project itself and what you can expect from this series. Enjoy the ride!

What is apoptOS?

apoptOS an open source project currently only developed by myself, Yves Vollmeier aka Tix3Dev. It’s a modern x86_64 UNIX-like microkernel-based operating system written in C and Assembly. The main purpose of it in general is to help me understand computers better whilst having fun, as I like building stuff :D

Especially I want to understand the following topics better:

Microkernels
UNIX-like systems
Scheduling
Messaging (IPC)
Syscalls
Userland
ELF
Filesystems

What is this series about?

In this series of blog posts, I will document milestones regarding the development of apoptOS and my learning process. This will include the topics mentioned above.

NOTE: All the blog posts are treated as snapshots in time, meaning I won’t update e.g. a list of features currently available.

Milestone - Memory management

apoptOS-Series-2: Milestone - Memory management

2022-06-06T00:00:00+00:00

Foreword

In this blog post I will talk about the concepts for memory management in apoptOS and how to use them.

Physical memory management

Generally speaking, this part is in charge of managing actual memory in RAM, i.e. the addresses used here directly correspond to memory in RAM. In order to manage that kind of memory, physical memory, we need to know how the bootloader, Limine, set up the memory map. To get this information, we can use the stivale2_struct_tag_memmap struct provided by the stivale2 boot protocol.

Bitmap based page frame allocator

Now that we know where the usable entries / memory locations are, we can store those informations in a data structure. apoptOS makes use of a bitmap (hence the name bitmap based allocator), where each bit in the bitmap corresponds to a page / block of memory (hence the name page frame allocator) through a mapping system. The value of a bit says if it’s corresponding page is free or used. Note that in apoptOS pages are 4 KiB in size.

Before we can allocate or free anything, we need to initialize it with this function:

void pmm_init(struct stivale2_struct *stivale2_struct);

It basically does everything we’ve talked about so far, i.e. setting up the memory map and the bitmap.

To allocate contiguous pages, we can either use this:

void *pmm_alloc(size_t page_count);

or preferably this:

void *pmm_allocz(size_t page_count);

Which does the same thing as pmm_alloc except it also memsets the memory returned to zero (-> “clean memory” -> should be preferred).

Allocating consists of finding a free bit in the bitmap, setting it to used and returning the corresponding memory.

To free contiguous pages, the following is used:

void pmm_free(void *pointer, size_t page_count);

Freeing consists of setting the bit corresponding to the pointer passed as an argument to free again, so that the memory can be used again.

Slab allocator

The bitmap based page frame allocator is great for page-sized allocations, but when it comes to fine-grained allocations, there would be too much memory wastage. That’s why I’ve implemented a slab allocator, inspired by the design proposed by Jeff Bonwick. The basic concept lays in object caching, i.e. storing preallocated objects in a customly defined cache. When allocating, no real memory has to be allocated, only the address of the preallocated object has to be returned. Same goes for freeing: No real memory is freed, only the object is marked as free again.

A cache consists of a linked list of so called slabs. If one slab happens to be fully allocated, a new slab will automatically be created if specified by the user. That’s why in theory, when allocating with the slab allocator, one can’t run out of memory.

A slab itself consists of a freelist or in other words a linked list of control buffers (bufctl).

There are some restrictions though, as the maximum size for a cached object is 512 bytes (so called small slabs). If you want to know more about this issue, please consult the paper by Jeff Bonwick. malloc solves this problem as we’ll see later.

Now let’s learn how to use the slab allocator:

First, we are going to use slab_cache_create to create a cache for a specific task, in this example inodes for a filesystem.

slab_cache_t *slab_cache_create(const char *name, size_t slab_size, slab_flags_t flags);

// create a cache where each slab is 64 bytes in size, the cache has a description for 
// debugging: "filesystem inodes" and flags to panic if any problem occurs

slab_cache_t *filesystem_inode_cache = slab_cache_create("filesystem inodes", 64, SLAB_PANIC);

Now to allocate an object of size 64 from the newly created cache, we will use the follwing function:

void *slab_cache_alloc(slab_cache_t *cache, slab_flags_t flags);

It might look something like this:

// create a cache where each slab is 64 bytes in size, the cache has a description for 
// debugging: "filesystem inodes" and flags to panic if any problem occurs

slab_cache_t *filesystem_inode_cache = slab_cache_create("filesystem inodes", 64, SLAB_PANIC);

// allocate an object from the cache, set flags to panic if any problem occurs
// and automatically grow the cache if it runs out of slabs

void *inode_ptr_1 = slab_cache_alloc(filesystem_inode_cache, SLAB_PANIC | SLAB_AUTO_GROW);

NOTE: We can also manually grow or reap the cache, by using the corresponding functions:

void slab_cache_grow(slab_cache_t *cache, size_t count, slab_flags_t flags);

void slab_cache_reap(slab_cache_t *cache, slab_flags_t flags);

To free our freshly allocated object, we will use the following function:

void slab_cache_free(slab_cache_t *cache, void *pointer, slab_flags_t flags);

Our final code might look something like this - I’ve also added a pretty neat debugging tool, to dump the cache:

void slab_cache_dump(slab_cache_t *cache, slab_flags_t flags);

// create a cache where each slab is 64 bytes in size, the cache has a description for 
// debugging: "filesystem inodes" and flags to panic if any problem occurs

slab_cache_t *filesystem_inode_cache = slab_cache_create("filesystem inodes", 64, SLAB_PANIC);

// allocate an object from the cache, set flags to panic if any problem occurs
// and automatically grow the cache if it runs out of slabs

void *inode_ptr_1 = slab_cache_alloc(filesystem_inode_cache, SLAB_PANIC | SLAB_AUTO_GROW);

// let's check how our cache looks like at the moment

slab_cache_dump(filesystem_inode_cache, SLAB_PANIC);

// do something nice with inode_ptr_1 :D

// we don't need our object anymore, so let's free it

slab_cache_free(filesystem_inode_cache, inode_ptr_1, SLAB_PANIC);

And that’s how easy it is to use the slab allocator in apoptOS!

Virtual memory management

The basic idea of virtual memory is to have a bigger address space, ranging (in our case) from 0x0 to 0xFFFFFFFFFFFFFFFF. To get from virtual memory to physical memory, we use a mapping system. On x86_64 we are provided with paging. In apoptOS, to be specific, we make use of 4-level paging. I won’t go into the details of how paging works, but instead how to use the interface.

To initialize paging and map all memory regions, we’ll use:

void vmm_init(struct stivale2_struct *stivale2_struct);

The resulting virtual memory layout looks like this:

First 4 GiB identical:
0x0 - 0x100000000 mapped to 0x0 - 0x100000000

First 4 GiB for higher half data:
0xFFFF800000000000 - 0xFFFF800100000000 mapped to 0x0 - 0x100000000

First 4 GiB for heap:
0xFFFF900000000000 - 0xFFFF900100000000 mapped to 0x0 - 0x100000000 

First 2 GiB for higher half code:
0xFFFFFFFF80000000 - 0x0001000000000000 mapped to 0x0 - 0x80000000 

All memory map entries:
At higher half data start to all entries in memory map 

To just map a page, we can use:

void vmm_map_page(uint64_t *page_table, uint64_t phys_page, uint64_t virt_page, uint64_t flags);

And to unmap a page:

void vmm_unmap_page(uint64_t *page_table, uint64_t virt_page);

We can also map a whole range using:

void vmm_map_range(uint64_t *page_table, uint64_t start, uint64_t end, uint64_t offset, uint64_t flags);

And to unmap a whole range:

void vmm_unmap_range(uint64_t *page_table, uint64_t start, uint64_t end);

Whenever we need to map something outside of src/kernel/memory/virtual/vmm.c, we’ll need to know the address of the root page table. To get it, we have to use:

uint64_t *vmm_get_root_page_table(void);

Heap memory

Sometimes we don’t want to allocate a whole page with pmm_alloc or create a slab cache and allocate from it via slab_cache_alloc. That might be because we don’t want to waste one whole page for a small allocation, so we’d have to use the slab allocator, but then the sizes might change, so we’d have to create a lot of caches. To fix this issue, apoptOS provides a generic interface, where we don’t have to initialize anything before an allocation. The size we want to allocate can also be anything arbitrary.

TL;DR apoptOS provides a heap interface through malloc and free.

In src/kernel/kernel.c - kinit_all the following is already called, so this function will never have to be used again:

void malloc_heap_init(void);

Allocations and frees from now on are as easy as:

void *malloc(size_t size);

and

void free(void *pointer);

Addresses returned by malloc are always from the heap memory region at 0xFFFF900000000000 (see memory layout in “Virtual memory management”).

Detailed implementation

Last but not least, I’d like to show how I’ve implemented the heap in more detail, as I’ve come up with a quite unique concept. I haven’t done any benchmarks on it yet, but at the moment this allocator will suffice time and efficiency wise. The slab allocator might also be changed in the future to also use large slabs, so this design won’t be needed anymore. Simplicity-wise I can only recommend it!

malloc is a mix between the bitmap based page frame allocator and the slab allocator. Former of which is used for allocations above 512 bytes, latter of which is used for allocations below 512 bytes.
malloc_heap_init will create 8 caches, with sizes being power of two, starting with 4 and ending with 512. apoptOS uses 4 as the smallest possible size, as there will always be metadata, which is 2 bytes already.
malloc will check if the requested size is below or above 512 (to use the appropriate allocator).

-> size <= 512:
- Update size by adding the size of the metadata struct
- Convert updated size to index that’ll be used to access the array of slab caches (e.g., size=5 -> index=1 (corresponding to slab size 8))
NOTE: This conversion is done by:
```
 size_t get_slab_cache_index(size_t size);
```
Which is a very efficient algorithm (three conditions only), compromising looks of the code. See repo for details.
- Allocate memory with slab allocator (argument=index)
- Put metadata struct at beginning of allocated memory - Metadata contains slab caches index
-> size > 512
- Update size by adding the size of the metadata struct and page aligning it
- Convert udpated size to page count
- Allocate memory with bitmap based page frame allocator (argument=page count)
- Put metadata struct at beginning of allocated memory - Metadata contains page count
free will check if the pointer ends with 0xFFF bits by bitwise ANDing

-> not true
- Get slab caches index from metadata at pointer passed as argument
- Free memory with slab allocator (argument=index)
-> true
- Get page count from metadata at pointer passed as argument
- Free memory with bitmap based page frame allocator (argument=page count)

And that’s the whole deal with memory. Of course, everything is very basic, but for now it wouldn’t make sense to implement more, instead let’s start adding more features to apoptOS!

References

https://github.com/Tix3Dev/apoptOS/blob/9a4fde1fe18abb4fb91d89a29ae21bde7c0cbea5/src/kernel/memory/physical/pmm.c
https://github.com/Tix3Dev/apoptOS/blob/9a4fde1fe18abb4fb91d89a29ae21bde7c0cbea5/src/kernel/memory/virtual/vmm.c
https://github.com/Tix3Dev/apoptOS/blob/9a4fde1fe18abb4fb91d89a29ae21bde7c0cbea5/src/kernel/memory/dynamic/slab.c
https://github.com/Tix3Dev/apoptOS/blob/9a4fde1fe18abb4fb91d89a29ae21bde7c0cbea5/src/kernel/libk/malloc/malloc.c
https://github.com/limine-bootloader/limine
https://github.com/stivale/stivale
https://people.eecs.berkeley.edu/~kubitron/courses/cs194-24-S14/hand-outs/bonwick_slab.pdf

Using OSDev-related documentation

2022-02-17T00:00:00+00:00

Foreword

It can be a quite daunting task for newcomers to the OSDev world to read documentation like the Intel System Programming Guide or UEFI specifications. Even once the desired part is found in the sheer endless documentation, it’s still hard to make use of those technical information’s.

As I am slowly getting better at extracting such information’s, I wanted to share a few tips and tricks using an example.

Example: Implementing RSDP structure for ACPI

In case you don’t know yet, in order to parse ACPI tables like MADT etc., it’s necessary to find the “Root System Description Pointer” (short: RSDP). After that, we need to extract information from it.

So what I did, as hopefully every programmer would do, was to search for this structure (in my case in the UEFI ACPI specification). And fortunately, also found it:

First step done!

Next step is to read all information’s contained in this table. It’s really worth reading it carefully and trying to understand everything (although 100% isn’t necessary, as it’s already very helpful just having heard the terms for the future). But as always, if you don’t understand a key part, you can just search for it on the internet. E.g., I had to search for OEM and thanks to that, I now know that it stands for “Original equipment manufacturer” and a common example are computer manufacturers that integrate OEM parts such as processors or software into their own product.

That’s cool and all, but we still need to “convert” this table into code, which is our last step.

I started off by writing some pseudocode, which eventually will turn into C code:

typedef struct
{
    // ACPI version 1.0
    var signature;
    var checksum;
    var oemid;
    var revision;
    var rsdt_address;

    // ACPI version 2.0+
    var length;
    var xsdt_address;
    var extended_checksum;
    var reserved;

} rsdp_structure_t;

(The fact that this is a struct just comes from the ACPI context.)

As you can see, for now I only defined a struct with non-existing data types. Our main goal now is to change them into proper ones.

Let’s start changing the easy ones, whose types are described in the “Description” part:

typedef struct
{
    // ACPI version 1.0
    var signature;
    var checksum;
    var oemid;
    var revision;
    uint32_t rsdt_address;  // <-

    // ACPI version 2.0+
    var length;
    uint64_t xsdt_address;  // <-
    var extended_checksum;
    var reserved;

} rsdp_structure_t;

As we can see and will see, it’s very important to know how big a data type is. This should help fresh things up:

// Some important C data types for the x86_64 architecture

char var1;	// 1 byte
short var2;	// 2 bytes
int var3;	// 4 bytes
long var4;	// 8 bytes

// Generally what would be preferred in OSDev

uint8_t var5;	// 8 bits   = 1 byte
uint16_t var6;	// 16 bits  = 2 bytes
uint32_t var7;	// 32 bits  = 4 bytes
uint64_t var8;	// 64 bits  = 8 bytes

Next up we can tell by the naming that the following ones are a string and we also get the byte length that they take up, which is just the [n], as one char is one byte:

typedef struct
{
    // ACPI version 1.0
    char signature[8];	// <-
    var checksum;
    char oemid[6];	// <-
    var revision;
    uint32_t rsdt_address;

    // ACPI version 2.0+
    var length;
    uint64_t xsdt_address;
    var extended_checksum;
    var reserved;

} rsdp_structure_t;

If we have a look at the byte length column, we can see that the following ones have the same length (namely 1 byte = 8 bits):

typedef struct
{
    // ACPI version 1.0
    char signature[8];
    uint8_t checksum;	// <-
    char oemid[6];
    uint8_t revision;	// <-
    uint32_t rsdt_address;

    // ACPI version 2.0+
    var length;
    uint64_t xsdt_address;
    uint8_t extended_checksum;	// <-
    var reserved;

} rsdp_structure_t;

The table in the documentation says, that the field “Length”, has a byte length of 4 bytes = 32 bits

typedef struct
{
    // ACPI version 1.0
    char signature[8];
    uint8_t checksum;
    char oemid[6];
    uint8_t revision;
    uint32_t rsdt_address;

    // ACPI version 2.0+
    uint32_t length;	// <-
    uint64_t xsdt_address;
    uint8_t extended_checksum;
    var reserved;

} rsdp_structure_t;

Finally, we’ve got the field “Reserved”, which has a byte length of 3. Oh no, what now? We don’t have a data type with that size… Actually, that’s an easy fix, let’s just use 3 uint8_t -> 3 * 1 byte = 3 bytes.

typedef struct
{
    // ACPI version 1.0
    char signature[8];
    uint8_t checksum;
    char oemid[6];
    uint8_t revision;
    uint32_t rsdt_address;

    // ACPI version 2.0+
    uint32_t length;
    uint64_t xsdt_address;
    uint8_t extended_checksum;
    uint8_t reserved[3];

} rsdp_structure_t;

And that’s it, our final struct, ready to use for ACPI table parsing!

References

https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf

Markdown Syntax Preview

2021-10-21T00:00:00+00:00

Philosophy:

Markdown is intended to be as easy-to-read and easy-to-write as is feasible.

Readability, however, is emphasized above all else. A Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions. While Markdown’s syntax has been influenced by several existing text-to-HTML filters — including Setext, atx, Textile, reStructuredText, Grutatext, and EtText — the single biggest source of inspiration for Markdown’s syntax is the format of plain text email.

To this end, Markdown’s syntax is comprised entirely of punctuation characters, which punctuation characters have been carefully chosen so as to look like what they mean. E.g., asterisks around a word actually look like emphasis. Markdown lists look like, well, lists. Even blockquotes look like quoted passages of text, assuming you’ve ever used email. Inline HTML

Markdown’s syntax is intended for one purpose: to be used as a format for writing for the web.

Markdown is not a replacement for HTML, or even close to it. Its syntax is very small, corresponding only to a very small subset of HTML tags. The idea is not to create a syntax that makes it easier to insert HTML tags. In my opinion, HTML tags are already easy to insert. The idea for Markdown is to make it easy to read, write, and edit prose. HTML is a publishing format; Markdown is a writing format. Thus, Markdown’s formatting syntax only addresses issues that can be conveyed in plain text.

For any markup that is not covered by Markdown’s syntax, you simply use HTML itself. There’s no need to preface it or delimit it to indicate that you’re switching from Markdown to HTML; you just use the tags.

The only restrictions are that block-level HTML elements — e.g. <div>, <table>, <pre>, <p>, etc. — must be separated from surrounding content by blank lines, and the start and end tags of the block should not be indented with tabs or spaces. Markdown is smart enough not to add extra (unwanted) <p> tags around HTML block-level tags. Source

Basic Markdown Formatting

Headings

# This is an <h1> tag
## This is an <h2> tag
### This is an <h3> tag
#### This is an <h4> tag
##### This is an <h5> tag
###### This is an <h6> tag

Emphasis

*This text will be italic*
_This will also be italic_

**This text will be bold**
__This will also be bold__

_You **can** combine them_

Result:

This text will be italic

This will also be italic

This text will be bold

This will also be bold

You can combine them

Lists

Inordered:

* Milk
* Bread
    * Wholegrain
* Butter

Result:

Milk
Bread
- Wholegrain
Butter

Ordered:

Tidy the kitchen
Prepare ingredients
Cook delicious things

Result:

Tidy the kitchen
Prepare ingredients
Cook delicious things

Images

![Alt Text](url)

Result:

Links

[link](http://example.com)

Result:

link

Blockquotes

As Kanye West said:

> We're living the future so
> the present is our past.

Result:

As Kanye West said:

We’re living the future so the present is our past.

Horizontal Rules

---

Result:

Code Snippets

Indenting by 4 spaces will turn an entire paragraph into a code-block.

Result:

.my-link {
    text-decoration: underline;
}

Reference Lists & Titles

**The quick brown [fox][1], jumped over the lazy [dog][2].**

[1]: https://en.wikipedia.org/wiki/Fox "Wikipedia: Fox"
[2]: https://en.wikipedia.org/wiki/Dog "Wikipedia: Dog"

Result:

The quick brown fox, jumped over the lazy dog.

Escaping

\*literally\*

Result:

*literally*

Embedding HTML

<button class="button-save large">Big Fat Button</button>

Result:

Advanced Markdown

Note: Some syntax which is not standard to native Markdown. They’re extensions of the language.

Strike-throughs

~~deleted words~~

Result:

~~deleted words~~

Automatic Links

https://ghost.org

Result:

https://ghost.org

Markdown Footnotes

Work in Ghost:

The quick brown fox[^1] jumped over the lazy dog[^2].

[^1]: Foxes are red
[^2]: Dogs are usually not red

Result:

The quick brown fox¹ jumped over the lazy dog².

GitHub Flavored Markdown

Syntax Highlighting

```javascript
function fancyAlert(arg) {
  if(arg) {
    $.facebox({div:'#foo'})
  }
}
```

Result:

function fancyAlert(arg) {
  if (arg) {
    $.facebox({ div: "#foo" });
  }
}

Task Lists

- [x] @mentions, #refs, [links](), **formatting**, and <del>tags</del> supported
- [x] list syntax required (any unordered or ordered list supported)
- [x] this is a complete item
- [ ] this is an incomplete item

Result:

@mentions, #refs, links, formatting, and ~~tags~~ supported
list syntax required (any unordered or ordered list supported)
this is a complete item
this is an incomplete item

Tables

You can create tables by assembling a list of words and dividing them with hyphens - (for the first row), and then separating each column with a pipe |:

First Header | Second Header | Third Header |
------------ | ------------- ----------------
Content from cell 1 | Content from cell 2 | Content from cell 3 |
Content in the first column | Content in the second column | Content in the third column |
Content in the forth row A | Content in the forth row B | Content in the forth row C |

First Header	Second Header	Third Header
Content from cell 1	Content from cell 2	Content from cell 3
Content in the first column	Content in the second column	Content in the third column
Content in the forth row A	Content in the forth row B	Content in the forth row C

References

http://blog.ghost.org/markdown/
https://guides.github.com/features/mastering-markdown/

Foxes are red ↩
Dogs are usually not red ↩