MooFoo blogs

Data compression

2025-12-01T00:00:00Z

I have been curious about how data compression works for a very long time - back in the day there was an archiver race, with WinRar and KGB archiver being insanely good at it. Bear in mind, those were the days where you had to use 1.44MB 3.5" magnetic "floppy" disks to transfer data - that was the era before DVD and even CD disks. And what WinRar could offer was no joke - you could compress 14MB Doom 2 and split it into 5 tomes , write each of them to a separate disk and then unarchive them on another computer - archiver would prompt you to insert the next disk with the next tome when it is done decompressing the current one.

But I digress. I was curious how WinRar can do that and how the size of the data is actually reduced.

And recently I got back to this topic. What I found was actually quite interesting.

Back in the day I had an idea to transform the data like so: for each character, replace it with a short code followed by the number of times it is repeated in a row:

AAAAABBB (8 characters)

  |
  v

A5B3 (4 characters - 50% compression!)

This is exactly what run-length encoding is doing. The problem with this solution was that even this very text does not contain a whole lot of repeated characters in a row, so it is quite inefficient:

Hello world (11 characters)

     |
     v

H1e1l2o1 1w1o1r1l1d1 (20 characters - negative 100% compression)

The next idea I had was to represent each character's original binary code with a shorter one:

Hello world

H (72 = 01001000 => 000000001)
e (101 = 01100101 => 00000010)
l (108 = 01101100 => 00000011) <------\
l (108 = 01101100 => 00000011) <-----/|
o (111 = 01101111 => 00000100) <-\    |
  (32 = 00100000 => 00000101)    |    |
w (119 = 01110111 => 00000110)   |    |
o (111 = 01101111 => 00000100) <-/    |
r (114 = 01110010 => 00000111)        |
l (108 = 01101100 => 00000011) <------/
d (100 = 01100100 => 00001000)

Original:

Hello world
01001000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 01100100

New:

00000001 00000010 00000011 00000011 00000100 00000101 00000110 00000100 00000111 00000011 00001000

On itself this does not present any compression, but what if characters were packed into half-bytes? The longest short code is only 4 (meaningful) bits, so we could get rid of most of zeros:

Old:

0000'0001 0000'0010 0000'0011 0000'0011 0000'0100 0000'0101 0000'0110 0000'0100 0000'0111 0000'0011 0000'1000

(11 bytes)

New:

0001'0010 0011'0011 0100'0101 0110'0100 0111'0011 1000'0000

(6 bytes)

This is a reduction of about 50%! But this method does not preserve the mapping of characters to short codes. If we were to add this mapping, the message size would be actually much larger:

01001000 00000001 01100101 00000010 01101100 00000011 01101111 00000100 00100000 00000101 01110111 00000110 01110010 00000111 01100100 00001000
H        code     e        code     l        code     o        code     <space>  code     w        code     r        code     d        code

This is already 16 bytes. Then add the 6 bytes for the encoded message and you get 22 bytes - a -100% (negative 100%) compression again.

If the initial text was much, much longer, this could actually be an improvement, but here's the challenge: once the number of unique letters in the initial text is larger than 15 (the maximum number of variants that could be represented by 4 bits), the compression rate drops below zero - simply because we can not represent every character in less than a full byte anymore.

Literal decades of break and I learned about Huffman encoding . The idea is somewhat similar - represent each letter of the input text with a short(er) code. The algorithm looks like this: get all the unique letters of the input text, order them by the number of their occurrences in the text, in decreasing order:

l => 3
o => 2
d => 1
r => 1
w => 1
" " => 1
e => 1
H => 1

Within this list, order the pairs with the same number of occurrences by the character code, in ascending order:

l => 3
o => 2
" " => 1
H => 1
d => 1
e => 1
r => 1
w => 1

Imagine each of these entries as a node:

Then, take a pair of nodes with the smallest number of occurrences each and merge them into a new node with the number of occurrences as a sum of two original nodes:

Repeat this process until there is just one node left:

And again:

There are four candidates for the smallest amongst the (2) nodes: (o, 2) and three internal nodes; in this case the (o, 2) node will become the left branch (because the ordering of the characters - none of the internal nodes has the character assigned, but the (o, 2) node has it; its code is lexicographically larger than the empty character, assuming empty character has code of 0 ):

Note how the node (l, 3) has moved to the middle of the list, between (4) and (2) nodes - this is needed to preserve the ordering by the number of occurrences.

There are two more nodes (2) , join them:

Now the smallest two nodes are (l, 3) and (4) :

And lastly there are (7) and (4) nodes to merge:

Then, following the rule "left branch -> '0', right branch -> '1'" , mark each branches of this tree:

Lastly, by following the branches from the root node of this tree, gather the branch labels into the binary code for each of the leaf nodes:

(l, 3, 0b01)  (o, 2, 0b11)  (" ", 1, 0b100)  (H, 1, 0b101)  (d, 1, 0b0010)  (e, 1, 0b0011)  (r, 1, 0b0000)  (w, 1, 0b0001)

This is called Huffman encoding or Huffman tree. Using these codes, the initial text Hello world is encoded as following:

H   e    l  l  o      w    o  r    l  d
101 0011 01 01 11 100 0001 11 0000 01 0010

joining into bytes

10100110 10111100 00011100 00010010

This could be programmed in Ruby like so:

s = 'Hello world'
a = s.chars.uniq.map {|c| {char: c, count: s.chars.count(c)}}.sort {|a, b| [b[:count], a[:char]] <=> [a[:count], b[:char]]}
pq = a.map { |c| {char: c[:char], count: c[:count], left: nil, right: nil} }

while pq.size > 1 do
  left, right = pq.pop(2)

  if left[:count] < right[:count]
    left, right = right, left
  end

  pq << {char: nil, count: left[:count] + right[:count], left: left, right: right}

  pq.sort! {|a,b| [b[:count], a[:char] || ''] <=> [a[:count], b[:char] || '']}
end

q = [{node: pq[0], code: ''}]

codes = {}

while q.size > 0 do
  e = q.shift

  if e[:node][:left].nil? && e[:node][:right].nil?
    codes[e[:node][:char]] = e[:code]
    next
  end

  q.push({node: e[:node][:left], code: e[:code] + '0'}) if !e[:node][:left].nil?

  q.push({node: e[:node][:right], code: e[:code] + '1'}) if !e[:node][:right].nil?
end

or in C++:

#include <string>
#include <map>
#include <queue>
#include <iostream>

struct Node {
    char c;
    int count;
    Node* left;
    Node* right;
};

struct CodeNode {
    Node* node;
    std::string code;
};

int main() {
    const std::string s = "Hello world";

    std::map<char, int> freq;

    for (auto i : s) {
        if (freq.find(i) == freq.end()) {
            freq[i] = 1;
        } else {
            freq[i]++;
        }
    }

    auto cmpNode = [](Node* a, Node* b) {
        if (a->count != b->count) {
            return a->count > b->count;
        }

        if ((a->c == static_cast<char>(255)) != (b->c == static_cast<char>(255))) {
            return b->c == static_cast<char>(255);
        }

        return a->c < b->c;
    };

    std::priority_queue<Node*, std::vector<Node*>, decltype(cmpNode)> pq(cmpNode);

    for (auto i : freq) {
        pq.push(new Node{.c = i.first, .count = i.second, .left = nullptr, .right = nullptr});
    }

    while (pq.size() > 1) {
        auto* left = pq.top();
        pq.pop();

        auto* right = pq.top();
        pq.pop();

        if (left->count < right->count) {
            std::swap(left, right);
        }

        pq.push(new Node{ .c = static_cast<char>(255), .count = left->count + right->count, .left = left, .right = right });
    }

    std::queue<CodeNode*> q;

    q.push(new CodeNode{ .node = pq.top(), .code = std::string() });

    std::map<char, std::string> codes;

    while (!q.empty()) {
        auto* e = q.front();
        q.pop();

        if (!e->node->left && !e->node->right) {
            codes[e->node->c] = e->code;
            continue;
        }

        if (e->node->left) {
            q.push(new CodeNode{ .node = e->node->left, .code = e->code + "0" });
        }

        if (e->node->right) {
            q.push(new CodeNode{ .node = e->node->right, .code = e->code + "1" });
        }
    }

    for (auto i : codes) {
        std::cout << "'" << i.first << "'" << ": " << i.second << std::endl;
    }

    return 0;
}

If the Huffman table is known, the source could be decoded by traversing from the root: if the next bit in the input sequence is 0 - traverse the left child of the current node, if the next bit is 1 - traverse the right child of the current node; if the leaf node is reached - emit a corresponding character:

In Ruby:

encoded = s.chars.map {|c| codes[c]}.join
tree = pq[0]

def decode(tree, encoded)
  root = tree
  res = ''

  encoded.chars.each do |c|
    root = if c == '0' then root[:left] else root[:right] end

    if not root[:char].nil?
      res += root[:char]
      root = tree
    end
  end

  res
end

Or in C++:


#include <string>
#include <map>
#include <queue>
#include <iostream>

struct Node {
    char c;
    int count;
    Node* left;
    Node* right;
};

struct CodeNode {
    Node* node;
    std::string code;
};

Node* buildHuffmanTree(std::string_view s) {
  std::map<char, int> freq;

  for (auto i : s) {
    if (freq.find(i) == freq.end()) {
      freq[i] = 1;
    } else {
      freq[i]++;
    }
  }

  auto cmpNode = [](Node *a, Node *b) {
    if (a->count != b->count) {
      return a->count > b->count;
    }

    if ((a->c == static_cast<char>(255)) != (b->c == static_cast<char>(255))) {
      return b->c == static_cast<char>(255);
    }

    return a->c < b->c;
  };

  std::priority_queue<Node *, std::vector<Node *>, decltype(cmpNode)> pq(
      cmpNode);

  for (auto i : freq) {
    pq.push(new Node{
        .c = i.first, .count = i.second, .left = nullptr, .right = nullptr});
  }

  while (pq.size() > 1) {
    auto *left = pq.top();
    pq.pop();

    auto *right = pq.top();
    pq.pop();

    if (left->count < right->count) {
      std::swap(left, right);
    }

    pq.push(new Node{.c = static_cast<char>(255),
                     .count = left->count + right->count,
                     .left = left,
                     .right = right});
  }

  return pq.top();
}

std::string encode(Node* tree, std::string_view s) {
  std::queue<CodeNode *> q;

  q.push(new CodeNode{.node = tree, .code = std::string()});

  std::map<char, std::string> codes;

  while (!q.empty()) {
    auto *e = q.front();
    q.pop();

    if (!e->node->left && !e->node->right) {
      codes[e->node->c] = e->code;
      continue;
    }

    if (e->node->left) {
      q.push(new CodeNode{.node = e->node->left, .code = e->code + "0"});
    }

    if (e->node->right) {
      q.push(new CodeNode{.node = e->node->right, .code = e->code + "1"});
    }
  }

  std::string res = "";

  for (auto ch : s) {
    res += codes[ch];
  }

  return res;
}

std::string decode(Node* tree, std::string_view s) {
  std::string res = "";
  auto root = tree;

  for (auto ch : s) {
    if (ch == '0') {
      root = root->left;
    } else {
      root = root->right;
    }

    if (!root->left && !root->right) {
      res += root->c;
      root = tree;
    }
  }

  return res;
}

int main() {
    const std::string s = "Hello world";

    auto root = buildHuffmanTree(s);
    
    auto encoded = encode(root, s);
    auto decoded = decode(root, encoded);

    std::cout << "Input: " << s << std::endl;
    std::cout << "Encoded: " << encoded << std::endl;
    std::cout << "Decoded: " << decoded << std::endl;

    return 0;
}

To construct a Huffman tree from encodings table (given the encodings are sorted in the descending order of their frequency), a very simple algorithm is used - traverse the tree by the bits of the character encoding and if there is no node matched, add to the last seen node - a left child if a bit is 0 or a right child if a bit is 1 .

As an example from above, consider the following encodings table (sorted already):

l => 0b01; o => 0b11; " " => 0b100; H => 0b101; d => 0b0010; e => 0b0011; r => 0b0000; w => 0b0001

Iterate over the encodings one by one, starting with l => 0b01 :

encoding: (l, 0b01) tree is empty (has only root node), the current node is root:

first bit: 0 , add a new left child to the current node and make this new child node the current one:

next bit: 1 , add a new right child to the previously added node and make it the new current node:

since this is the last bit, add the value to the current node:

Next encoding, o => 0b11 , reset the current node to the root:

first bit: 1 , add a new right child node to the current node and make it the new current:

next bit: 1 , add a new right child node and make it current:

since this is the last bit, add the value to the current node:

Next encoding, " " => 0b100 , reset the current node (*) to the root:

first bit: 1 , no changes to the tree, since the right child of the current node exists:

next bit: 0 , add a new left child to the current node and make it the new current (*):

since this is the last bit, add a value to the current node:

Next encoding is H => 0b101 , reset the current node (*) to root:

first bit is: 1 , no changes to the tree since the node exists, just make it current (*):

next bit is: 0 , no changes to the tree since the node exists, just make it current (*):

next bit is: 1 , add a right child node and make it current (*):

since this was the last bit, add a value to the current node:

And without going step-by-step, for all the remaining encodings: encoding: d => 0b0010 :

encoding: e => 0b0011 :

encoding: r => 0b0000 :

encoding: w => 0b0001 :

If the initial message ( Hello world ) is encoded with this algorithm (to four bytes: 0xA6 0xBC 0x1C 0x12 ), in order to properly decode it, the tree or the code table has to be given alongside with the encoded message:

message: 'Hello world' (11 bytes)
encoded: 0xA6 0xBC 0x1C 0x12 (4 bytes)
table:   {"l"=>"01", "o"=>"11", " "=>"100", "H"=>"101", "r"=>"0000", "w"=>"0001", "d"=>"0010", "e"=>"0011"} (8 keys + 8 values = 16 bytes)
compression = 11 bytes -> 20 bytes = negative 100%

This is not entirely a compression algorithm, though. Algorithms such as ZIP (DEFLATE) and JPEG (although slightly different in details, the general approach still applicable) go further to compress the table itself. To understand how they work, one needs to understand the tricks behind them.

Canonical Huffman codes

There is one issue with just general Huffman encoding: there are multiple ways to build a different tree for the same input. Refer to the early stage of encoding from above:

(l, 3)  (2)   (o, 2)      (2)               (2)
      /     \           /     \           /     \
 (" ", 1) (H, 1)    (d, 1)  (e, 1)     (r, 1) (w, 1)

Here's the non-deterministic part: which pair of nodes to merge next? Depending on the choice, the code for each character would be different (although the code length would remain the same).

That's where canonical Huffman codes come into play: this simple algorithm unifies the logic, transforming any valid Huffman tree for a given input into the same tree (so the codes remain the same).

To obtain canonical codes from any valid Huffman tree for a given input, the codes are transformed following two simple rules:

sort the codes by code length (ascending) and then by character (also ascending)
starting with current_code = 0 and current_length = 0 , iterate over each code length (assume iterator is length_i ):
if the length is greater than current_length , left shift current_code by the difference length_i - current_length
increment current_code , filling the unused left-most bits with zeroes
update current_length to length_i

In the example above, the codes are:

l:   01
o:   11
" ": 100
H:   101
r:   0000
w:   0001
d:   0010
e:   0011

The lengths of those codes are:

l:   2 (Huffman code: '01', length of '01' = 2)
o:   2 (Huffman code: '11', length of '11' = 2)
" ": 3 (Huffman code: '100', length of '100' = 3)
H:   3 (Huffman code: '101', length of '101' = 3)
r:   4 (Huffman code: '0000', length of '0000' = 4)
w:   4 (Huffman code: '0001', length of '0001' = 4)
d:   4 (Huffman code: '0010', length of '0010' = 4)
e:   4 (Huffman code: '0011', length of '0011' = 4)

Making these codes canonical requires sorting by Huffman code's length first and then by character (I have left old codes in braces for reference):

l:   2 (old code: 01, its length = 2)
o:   2 (old code: 11, its length = 2)
" ": 3 (old code: 100, its length = 3)
H:   3 (old code: 101, its length = 3)
d:   4 (old code: 0010, its length = 4)
e:   4 (old code: 0011, its length = 4)
r:   4 (old code: 0000, its length = 4)
w:   4 (old code: 0001, its length = 4)

Then, re-assign codes, starting with 0 and padding to the length of old code:

l:   00 (old code length = 2, next code = 00 + 1 = 01)
o:   01 (old code length = 2, next code = 01 + 1 = 10)
" ": 100 (old code length = 3; but previous entry ('o' = 01) has length 2, so shift left by 1 bit ('10' << 1 == '100'); next code = 100 + 1 = 101)
H:   101 (old code length = 3; next code = 101 + 1 = 110)
d:   1100 (old code length = 4; but previous entry ('H' = 101) has length 3, so shift left by 1 bit ('110' << 1 == '1100'); next code = 1100 + 1 = 1101)
e:   1101 (old code length = 4; next code = 1101 + 1 = 1110)
r:   1110 (old code length = 4; next code = 1110 + 1 = 1111)
w:   1111 (old code length = 4)

Interestingly, this code did not use the previous codes themselves. This implies the codes could be calculated automatically if only the lengths of the codes are given.

For the example above, the encoding table is:

l:   00
o:   01
" ": 100
H:   101
d:   1100
e:   1101
r:   1110
w:   1111

Encoded message with this table looks like this:

message:

Hello world

encoded:

101 1101 00 00 01 100 1111 01 1110 00 1100

joined:

10111010000011001111011110001100

In Ruby:

def build_canonical_table(codes)
    codes = codes.sort_by {|k, v| [v.length, k]}
    canonical_codes = {}
    curr_code = 0
    prev_length = 0

    codes.each do |char, old_code|
        curr_code <<= (old_code.length - prev_length)
        canonical_codes[char] = curr_code.to_s(2).rjust(old_code.length, '0')
        curr_code += 1
        prev_length = old_code.length
    end

    canonical_codes
end

In C++:

std::vector<std::pair<char, std::string>> sorted_codes(codes.begin(), codes.end());

std::sort(sorted_codes.begin(), sorted_codes.end(),
    [](const auto& a, const auto& b) {
        if (a.second.length() != b.second.length()) {
            return a.second.length() < b.second.length();
        }

        return a.first < b.first;
    });

std::map<char, std::string> canonical_codes;

int curr_code = 0;
int prev_length = 0;

for (auto [ch, code] : sorted_codes) {
    curr_code <<= (code.length() - prev_length);
    canonical_codes[ch] = std::bitset<32>(curr_code).to_string().substr(32 - code.length());
    curr_code++;
    prev_length = code.length();
}

ZIP, DEFLATE

DEFLATE algorithm compresses the list of codes lengths using a constant table, defined by the standard (or algorithm description, if you will).

The constant table is defined in the archiver code itself, so it remains the same for all archives and does not need to be a part of archive itself.

The archive, on the other hand, stores a list of 256 values, each representing the length of the code of the corresponding character.

For example, archive may store a list like this:

0, 0, 2, 2, 4, 0, 0, 0, 0, ... (total 256 elements, all the rest are zeros)

But this is highly inefficient, so this list is also comressed using Huffman encoding. The difference is that this list is first compressed using run-length encoding (from the very top of this blog) and the result is compressed using the pre-defined Huffman table.

Step-by-step this looks like this: first, the original content is compressed. This produces a Huffman table. Assume this table of canonical Huffman codes for input Hello world :

l  : "00"
o  : "01"
" ": "100"
H  : "101"
d  : "1100"
e  : "1101"
r  : "1110"
w  : "1111"

As described above, for canonical Huffman codes, storing only codes' lengths is enough to reconstruct the codes :

"00"   : 2
"01"   : 2
"100"  : 3
"101"  : 3
"1100" : 4
"1101" : 4
"1110" : 4
"1111" : 4

So the table of codes' lengths looks like this:

1: 0 (codes of length 1: none)
2: 2 (codes of length 2: two codes)
3: 2 (codes of length 3: two codes)
4: 4 (codes of length 4: four codes)

This table looks very much like an indexed array (1-based, though - indexes start at 1 instead of 0 like in C), so it could be represented as an array.

Given this data, at any point Huffman codes could be re-created:

code = 0

// lengths: [0, 2, 2, 4]

length_i = 1, codes_length_i = 0;
  no codes of length 1 => skip

length_i = 2, codes_length_i = 2;
  code (0) << 1 => code = 00;
  create <codes_length_i (2)> codes:
    emit code (00), code += 1 (code = 01);
    emit code (01), code += 1 (code = 10);

length_i = 3, codes_length_i = 2;
  code (10) << 1 => code = 100;
  create <codes_length_i (2)> codes:
    emit code (100), code += 1 (code = 101)
    emit code (101), code += 1 (code = 110);

length_i = 4, codes_length_i = 4;
  code (110) << 1 => code = 1100;
  create <codes_length_i (4)> codes:
    emit code (1100), code += 1 (code = 1101);
    emit code (1101), code += 1 (code = 1110);
    emit code (1110), code += 1 (code = 1111);
    emit code (1111), code += 1 (code = 10000);

So for the list of codes lengths [0, 2, 2, 4] the re-created codes will be

In Ruby:

def recreate_canonical_huffman_codes(codes_of_length)
  code = 0
  res = []
  
  codes_of_length.each_with_index do |n, len|
    code <<= 1

    n.times do
      res << code.to_s(2).rjust(len+1, '0')
      code += 1
    end
  end

  res
end

For the second phase, the original symbols have to be correlated with these codes somehow. DEFLATE does not store the symbols or their corresponding encodings at any point - instead it uses smart algorithm to encode the encodings (so meta!) for all possible symbols.

Referring back to the compressed content's table of codes:

l  : "00" (code length: 2)
o  : "01" (code length: 2)
" ": "100" (code length: 3)
H  : "101" (code length: 3)
d  : "1100" (code length: 4)
e  : "1101" (code length: 4)
r  : "1110" (code length: 4)
w  : "1111" (code length: 4)

In a similar manner to codes lengths' table, a table of all possible 256 byte values (ASCII table, effectively) is created and the corresponding byte's code length is written to this table:

0x00: 0
0x01: 0
0x02: 0
...
0x20 (" "): 3
...
0x48 (H)  : 3
...
0x64 (d)  : 4
0x65 (e)  : 4
...
0x6c (l)  : 2
...
0x6f (o)  : 2
...
0x72 (r)  : 4
...
0x77 (w)  : 4
...

Instead of hexadecimal values, here's decimal representation:

32  (" ") : 3
72  (H)   : 3
100 (d)   : 4
101 (e)   : 4
108 (l)   : 2
111 (o)   : 2
114 (r)   : 4
119 (w)   : 4

This list could be written as follows:

index: [0, 1, 2, 3, 4, ..., 31, 32, 33, 34, 35, ..., 71, 72, 73, ..., 99, 100, 101, 102, ..., 108, ..., 111, 112, 113, 114, ..., 119, ..., 255 ]
value: [0, 0, 0, 0, 0, ..., 0,  3,  0,  0,  0,  ...,  0,  3,  0, ...,  0,   4,   4,   0, ...,   2, ...,   2,   0,   0,   4, ...,   4, ..., 0 ]

This list contains a lot of duplications and a lot of zeroes. It is then compressed using the run-length algorithm (from the very top of this blog), which now makes sense:

(0, 32), (3, 1), (0, 39), (3, 1), (0, 27), (4, 1), (4, 1), (0, 6), (2, 1), (0, 2), (2, 1), (0, 2), (4, 1), (0, 4), (4, 1), (0, 136)

But DEFLATE does it smarter - it uses pre-defined table of symbols, again:

for zero value, use zero itself
for values 1..15, use the value itself as a code
to repeat previous value 3..6 times, use code 16 and 3 extra bits to represent how many repetitions ( 0 - repeat previous value 3 times, 1 - repeat previous value 4 times, 10 - repeat previous value 5 times, 11 - repeat previous value 6 times)
to repeat value 0 3..10 times, use code 17 and extra bits
to repeat value 0 11..138 times, use code 18 and extra bits

And then DEFLATE compresses the table of codes' lenghts with Huffman encoding. Since there is a very limited number of characters to encode ( 1..15 , 16 , 17 , 18 - 19 values total), the maximum code length, even if all of these values were used, would be about 4 bits. The values 16 , 17 and 18 will always be followed by a pre-defined number of bits (extra bits describing a parameter), so when decoding, if the value decoded is one of those three ( 16 , 17 or 18 ), the following pre-defined number of bits are interpreted differently.

Look at this process step-by-step. First, the code lenghts array is replaced with the codes from above using run-length algorithm:

Repeated zeros are represented by following this simple table:

single zero: 0
two zeros: 0, 0 (two zero bytes)
3..10 zeros: 17 followed by
- 0 for 3 zeros
- 1 for 4 zeros
- 2 for 5 zeros
- etc.
- 7 for 10 zeros
11..138 zeros: 18 followed by
- 0 for 11 zeros
- 1 for 12 zeros
- etc.
- 127 for 138 zeros

These extra bits are not encoded using Huffman codes. The rest of the codes (raw values 0..15 , 16 , 17 , 18 ) are encoded according to their frequencies.

(0, 32)  -> 18 (repeat '0' 32 times), followed by extra bits representing 32 repetition: 32 - 11 = 21 (0b00010101)
(3, 1)   -> 3
(0, 39)  -> 18 (repeat '0' 39 times), followed by extra bits representing 39 repetitions: 39 - 11 = 28 (0b00011100)
(3, 1)   -> 3
(0, 27)  -> 18 (repeat '0' 27 times), followed by extra bits representing 27 repetitions: 27 - 11 = 16 (0b00010000)
(4, 1)   -> 4
(4, 1)   -> 4
(0, 6)   -> 17 (repeat '0' 6 times), followed by extra bits representing 6 repetitions: 6 - 3 = 3 (0b00000011)
(2, 1)   -> 2
(0, 2)   -> [0, 0]
(2, 1)   -> 2
(0, 2)   -> [0, 0]
(4, 1)   -> 4
(0, 4)   -> 17 (repeat '0' 4 times), followed by extra bits representing 4 repetitions: 4 - 3 = 1 (0b00000001)
(4, 1)   -> 4
(0, 136) -> 18 (repeat '0' 136 times), followed by extra bits representing 136 repetitions: 136 - 11 = 125 (0b01111101)

After this, the codes lengths list becomes:

18, 21 (0b00010101), 3, 18, 28 (0b00011100), 3, 18, 16 (0b00010000), 4, 4, 17, 3 (0b00000011), 2, 0, 0, 2, 0, 0, 4, 17, 1 (0b00000001), 4, 18, 125 (0b01111101)

In Ruby:

def running_codes_lengths(codes)
    codes_lengths = (0..255).map {|i| (codes[i.chr] || '').length}

    result = []
    i = 0

    while i < codes_lengths.size
        len = codes_lengths[i]
        run_length = 1

        while i + run_length < codes_lengths.size && codes_lengths[i + run_length] == len
            run_length += 1
        end

        i += run_length

        if len == 0
            while run_length > 0
                if run_length >= 11
                    # code 18, repeat '0' 11..138 times
                    diff = [138, [11, run_length].max].min
                    result << [18, diff - 11]
                    run_length -= diff
                elsif run_length >= 3
                    # code 17, repeat '0' 3..10 times
                    diff = [10, [3, run_length].max].min
                    result << [17, diff - 3]
                    run_length -= diff
                else
                    result += [0] * run_length
                    run_length = 0
                end
            end
        elsif len != 0 && run_length >= 3
            result << len
            run_length -= 1

            while run_length > 0
                # code 16, repeat previous value 3..6 times
                diff = [6, [3, run_length].max].min
                result << [16, diff - 3]
                run_length -= diff
            end
        else
            run_length.times { result << len }
            run_length = 0
        end
    end

    result
end

In C++:

struct CodeLengthNode {
    size_t length;
    int extra_bits;
};

std::vector<CodeLengthNode*> code_lengths;

for (auto i = 0; i < 256; i++) {
    auto length = canonical_codes[i].length();
    auto run_length = 1;

    while (i + run_length < 256 && canonical_codes[i + run_length].length() == length) {
        run_length++;
    }

    i += run_length - 1;

    if (length == 0) {
        while (run_length > 0) {
            if (run_length >= 11) {
                // code 18, repeat '0' 11..138 times
                auto diff = std::min(138, std::max(11, run_length));

                code_lengths.push_back(new CodeLengthNode{ .length = 18, .extra_bits = diff - 11 });

                run_length -= diff;
            }
            else if (run_length >= 3) {
                // code 17, repeat '0' 3..10 times
                auto diff = std::min(10, std::max(3, run_length));

                code_lengths.push_back(new CodeLengthNode{ .length = 17, .extra_bits = diff - 3 });

                run_length -= diff;
            }
            else {
                for (auto t = 0; t < run_length; t++) {
                    code_lengths.push_back(new CodeLengthNode{ .length = 0, .extra_bits = 0 });
                }

                run_length = 0;
            }
        }
    }
    else if (length != 0 && run_length >= 3) {
        code_lengths.push_back(new CodeLengthNode{ .length = length, .extra_bits = 0 });
        run_length--;

        while (run_length > 0) {
            auto diff = std::min(6, std::max(3, run_length));
            code_lengths.push_back(new CodeLengthNode{ .length = 16, .extra_bits = diff - 3 });
            run_length -= diff;
        }
    }
    else {
        for (auto t = 0; t < run_length; t++) {
            code_lengths.push_back(new CodeLengthNode{ .length = length, .extra_bits = 0 });
        }

        run_length = 0;
    }
}

To compress this list, ignore the extra bits and count values 0..18 only:

18, 21, 3, 18, 28, 3, 18, 16, 4, 4, 17, 3, 2, 0, 0, 2, 0, 0, 4, 17, 1, 4, 18, 125

without extra bit values:

18, 3, 18, 3, 18, 16, 4, 4, 17, 3, 2, 0, 0, 2, 0, 0, 4, 17, 1, 4, 18

counts:

18 : 4
3  : 3
16 : 1
4  : 4
17 : 2
2  : 2
0  : 4
1  : 1

Next, order these mappings by count, followed by the number itself (key of this dictionary / hashmap):

Then, create Huffman tree for these counts:

step 1:

step 2:

step 3:

step 4:

Following the rule "left branch => 0, right branch => 1", traverse the tree and assign codes to each leaf node:

This yields:

(0, 4, 00) (4, 4, 01) (18, 4, 100) (3, 3, 101) (2, 2, 1100) (17, 2, 1101) (1, 1, 1110) (16, 1, 1111)

0 (4 occurrences)  : 00
4 (4 occurrences)  : 01
18 (4 occurrences) : 100
3 (3 occurrences)  : 101
2 (2 occurrences)  : 1100
17 (2 occurrences) : 1101
1 (1 occurrences)  : 1110
16 (1 occurrences) : 1111

Then, assign the canonical Huffman codes:

step 1: sort codes by lengths and code itself:

0  : 00    (code length: 2)
4  : 01    (code length: 2)
18 : 100   (code length: 3)
3  : 101   (code length: 3)
2  : 1100  (code length: 4)
17 : 1101  (code length: 4)
1  : 1110  (code length: 4)
16 : 1111  (code length: 4)

becomes

0  : 00    (code length: 2)
4  : 01    (code length: 2)
3  : 101   (code length: 3)
18 : 100   (code length: 3)
1  : 1110  (code length: 4)
2  : 1100  (code length: 4)
16 : 1111  (code length: 4)
17 : 1101  (code length: 4)

step 2: assign incrementing and shifting code:

code = 0, previous_length = 0

0  : 00    (code length: 2), code = 00, next code = 01
4  : 01    (code length: 2), code = 01, next code = 10
3  : 101   (code length: 3), length > previus_length => shift code, code = 100, next code = 101
18 : 100   (code length: 3), code = 101, next code = 110
1  : 1110  (code length: 4), length > previous_length => shift code, code = 1100, next code = 1101
2  : 1100  (code length: 4), code = 1101, next code = 1110
16 : 1111  (code length: 4), code = 1110, next code = 1111
17 : 1101  (code length: 4), code = 1111

Going back to what the initial codes' lengths list was (the thing being encoded):

18, 21 (0b00010101), 3, 18, 28 (0b00011100), 3, 18, 16 (0b00010000), 4, 4, 17, 3 (0b00000011), 2, 0, 0, 2, 0, 0, 4, 17, 1 (0b00000001), 4, 18, 125 (0b01111101)

Now encoded with this new Huffman encoding:

100, 00010101, 101, 100, 00011100, 101, 100, 1111, 00010000, 01, 01, 1101, 101, 00000011, 1100, 00, 00, 1100, 00, 00, 01, 1101, 1110, 00000001, 01, 100, 01111101

joined:

1000001010110110000011100101100111100010000010111011010000001111000000110000000111011110000000010110001111101

And the tree for this encoding could be just a list of 19 elements (one for each code used in this code lengths encoding) of codes lengths (for the encoded code lengths list):

0  : 00    (code length: 2)
4  : 01    (code length: 2)
3  : 101   (code length: 3)
18 : 100   (code length: 3)
1  : 1110  (code length: 4)
2  : 1100  (code length: 4)
16 : 1111  (code length: 4)
17 : 1101  (code length: 4)

simplified:

0  : 2
1  : 4
2  : 4
3  : 3
4  : 2
16 : 4
17 : 4
18 : 3

becomes

index: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 ]
value: [ 2, 4, 4, 3, 2, 0, 0, 0, 0, 0,  0,  0,  0,  0,  0,  0,  4,  4,  3 ]

or just

2, 4, 4, 3, 2, 0, 0, 0, 0, 0,  0,  0,  0,  0,  0,  0,  4,  4,  3

Note one interesting observation: since there is a very limited number of symbols used to encode code lengths, the maximum code for them is 7 , which is 3 bits long. This means the new encodings for code lengths table could be packed into 3 bit chunks, further reducing the size of this tree:

codes lengths (for codes lengths, so meta!):

2, 4, 4, 3, 2, 0, 0, 0, 0, 0,  0,  0,  0,  0,  0,  0,  4,  4,  3

encoded into binary:

010, 100, 100, 011, 010, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 100, 100, 011

joined:

010100100011010000000000000000000000000000000000100100011

Now, build the header for the archive, consisting of code lengths encoded with DEFLATE:

code lengths tree:

010100100011010000000000000000000000000000000000100100011

code lengths for each character, encoded:

1000001010110110000011100101100111100010000010111011010000001111000000110000000111011110000000010110001111101

in bytes:

01010010 01110100 00000000 00000000 00000000 00000000 10010001 11000001 01001011 00000111 00101100 11110001 00000101 11011010 00000111 10000001 10000000 11101111 00000000 10110001 111101

in hex:

0x52 0x74 0x00 0x00 0x00 0x00 0x91 0xC1 0x4B 0x07 0x2C 0xF1 0x05 0xDA 0x07 0x81 0x80 0xEF 0x00 0xB1 0x3D

And build the content of the archive, which is just the encoded message (from a while ago):

encoded message:

10111010000011001111011110001100

in bytes:

10111010 00001100 11110111 10001100

in hex:

0xBA 0x0C 0xF7 0x8C

Joining header and the body:

0x52 0x74 0x00 0x00 0x00 0x00 0x91 0xC1 0x4B 0x07 0x2C 0xF1 0x05 0xDA 0x07 0x81 0x80 0xEF 0x00 0xB1 0x3D 0xBA 0x0C 0xF7 0x8C

Funny enough, the length of this archive is much longer than the original message itself - 25 bytes vs original 11 - more than negative 100% increase in size. But the important part is that this is due to the fact that the message is short and most symbols only occur once. On a longer text the compression is actually good: a typical AI response compression:

Source length: 3994
Encoded length: 2632
> encoded length: 2573
> header length: 59
>> header1 length: 8
>> header2 length: 51
Compression rate: 51%

Decoding works the same three-step way, just in the reverse:

decode canonical Huffman tree for codes' lengths
decode codes' lengths and build the canonical Huffman tree again
decode the message using that tree

The only tricky part there is decoding the codes' lengths, since it involves re-building the symbols + extra bits for repeated values.

Decoding codes' lengths:

def decode_code_lengths_lengths(header)
    bits = header.bytes.map {|ch| ch.to_s(2).rjust(8, '0')}.join

    code_lengths = []

    (0..18).each do |e|
        start_bit = e * 3
        three_bits = bits[start_bit, 3]
        length = three_bits.to_i(2)
        code_lengths[e] = length
    end

    code_lengths
end

Then the canonical Huffman codes are reconstructed for decoding next stage of codes' lengths:

def recreate_huffman_codes(code_lengths)
    symbols_with_lengths = code_lengths.each_with_index.filter {|len, _| len > 0}.map {|len, s| [s, len] }.sort_by {|s, len| [len, s]}

    res = Array.new(code_lengths.size)
    prev_length = 0
    code = 0

    symbols_with_lengths.each do |s, len|
        code <<= (len - prev_length)
        res[s] = code.to_s(2).rjust(len, '0')
        code += 1
        prev_length = len
    end

    res
end

This is then used to decode running lengths (compressed lengths):

def decode_compressed_code_lengths(bits, codes_lengths_tree)
    tree_inv = codes_lengths_tree.each_with_index.to_h

    compressed_code_lengths = []
    bit_pos = 0
    buf = ''
    decoded_count = 0

    while bit_pos < bits.length && decoded_count < 256
        buf += bits[bit_pos]
        bit_pos += 1

        if tree_inv.key?(buf)
            symbol = tree_inv[buf]
            buf = ''

            case symbol
            when 16
                extra_bits = bits[bit_pos, 2].to_i(2)
                bit_pos += 2
                compressed_code_lengths << [16, extra_bits]
                decoded_count += 3 + extra_bits # code 16 repeats 3-6 repetitions
            when 17
                extra_bits = bits[bit_pos, 3].to_i(2)
                bit_pos += 3
                compressed_code_lengths << [17, extra_bits]
                decoded_count += 3 + extra_bits # code 17 repeats 3-10 repetitions
            when 18
                extra_bits = bits[bit_pos, 7].to_i(2)
                bit_pos += 7
                compressed_code_lengths << [18, extra_bits]
                decoded_count += 11 + extra_bits # code 18 repeats 11-138 repetitions
            else
                compressed_code_lengths << symbol
                decoded_count += 1 # any other code adds 1 instance of itself
            end
        end
    end

    compressed_code_lengths
end

Which is then used to reconstruct all codes' lengths for all symbols:

def decode_running_lengths(compressed_code_lengths)
    res = []

    compressed_code_lengths.each do |e|
        if e.is_a?(Array)
            code, extras = e

            case code
            when 16
                repeats = 3 + extras
                repeats.times { res << res.last }

            when 17
                repeats = 3 + extras
                repeats.times { res << 0 }

            when 18
                repeats = 11 + extras
                repeats.times { res << 0 }
            end
        else
            res << e
        end
    end

    res
end

These last two methods are there just to showcase the same process as encoding, but in reverse. Practically speaking, it does not make sense to build the intermediate running lengths encoding (with codes 16 , 17 and 18 ) and the actual list could be built in-place:

def decode_codes_lengths(header, codes_lengths_tree)
    bits = header.bytes.map {|b| b.to_s(2).rjust(8, '0')}.join

    tree_inv = codes_lengths_tree.each_with_index.to_h

    bit_pos = 0
    buf = ''

    res = []

    while bit_pos < bits.length && res.size < 256
        buf += bits[bit_pos]
        bit_pos += 1

        if tree_inv.key?(buf)
            symbol = tree_inv[buf]
            buf = ''

            case symbol
            when 16
                extra_bits = bits[bit_pos, 2].to_i(2)
                bit_pos += 2
                repeats = 3 + extra_bits
                repeats.times { res << res.last }
            when 17
                extra_bits = bits[bit_pos, 3].to_i(2)
                bit_pos += 3
                repeats = 3 + extra_bits
                repeats.times { res << 0 }
            when 18
                extra_bits = bits[bit_pos, 7].to_i(2)
                bit_pos += 7
                repeats = 11 + extra_bits
                repeats.times { res << 0 }
            else
                res << symbol
            end
        end
    end

    res
end

This list is then used to create the ultimate canonical Huffman codes for the message itself (the "body" of the archive), by calling recreate_huffman_codes :

def decode(header1, header2, body)
    code_lengths_lengths = decode_code_lengths_lengths(header1)
    tree = recreate_huffman_codes(code_lengths_lengths)
    codes_lengths = decode_codes_lengths(header2, tree)

    codes = recreate_huffman_codes(codes_lengths)
    decode_body(body, codes)
end

Decoding the body with this tree is simple:

def decode_with_tree(bits, tree)
    tree_inv = tree.each_with_index.to_h

    res = []
    buf = ''

    bits.each_char do |bit|
        buf += bit

        if tree_inv.key? buf
            res << tree_inv[buf]
            buf = ''
        end
    end

    res
end

def decode_body(body, codes_lengths)
    bits = body.bytes.map {|b| b.to_s(2).rjust(8, '0')}.join
    bytes = decode_with_tree(bits, codes_lengths)
    bytes.pack('C*').force_encoding('UTF-8')
end

Note how these methods use string.bytes instead of string.chars - this is to make the algorithm work with UTF-8 characters and not only ASCII.

For binary data it can be hit or miss. For example, a PNG is already a compressed binary data, so DEFLATE does not prove too useful:

Raw data (PNG file): 2356 bytes
Encoded: 2397 bytes
Compression ratio: -1%

For a larger uncompressed data (so potentially more of the same characters being used more frequently), like a TTF font:

Raw data (TTF font): 7316 bytes
Encoded: 5161 bytes
Compression ratio: 29%

JPEG, DHT

Define Huffman Table uses a much simpler approach - it stores the list of codes lengths using an index-as-value approach - a list of 16 values effectively translated as "element at index 'i' tells how many codes of length 'i' are there in the table" . This list is followed by the list of raw characters used in the message.

For example, an array like this

[0, 0, 2, 2, 4, 0, 0, 0, 0, 0,  0,  0,  0,  0,  0,  0]

is expanded like so:

index: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
value: [0, 0, 2, 2, 4, 0, 0, 0, 0, 0,  0,  0,  0,  0,  0,  0]

codes of length 0: 0
codes of length 1: 0
codes of length 2: 2
codes of length 3: 2
codes of length 4: 4
codes of length 5: 0
codes of length 6: 0
codes of length 7: 0
codes of length 8: 0
codes of length 9: 0
codes of length 10: 0
codes of length 11: 0
codes of length 12: 0
codes of length 13: 0
codes of length 14: 0
codes of length 15: 0

For the Hello world message, the archive header, comprising of codes' lengths and the alphabet, would look as following:

canonical Huffman codes:

l:   00   (length: 2)
o:   01   (length: 2)
" ": 100  (length: 3)
H:   101  (length: 3)
d:   1100 (length: 4)
e:   1101 (length: 4)
r:   1110 (length: 4)
w:   1111 (length: 4)

number of codes per length:

[ 0, 0, 2, 2, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]

alphabet (in order same order as canonical Huffman codes table):

[ l, o, " ", H, d, e, r, w ]

Followed by the encoded message, the entire archive would look like this:

codes per length:

[ 0, 0, 2, 2, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]

in bytes:

0x00 0x00 0x02 0x02 0x04 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

alphabet:

[ l, o, " ", H, d, e, r, w ]

in bytes:

0x6C 0x6F 0x20 0x48 0x64 0x65 0x72 0x77

encoded message:

10111010000011001111011110001100

in bytes:

0xBA 0x0C 0xF7 0x8C

combined:

0x00 0x00 0x02 0x02 0x04 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x6F 0x20 0x48 0x64 0x65 0x72 0x77 0xBA 0x0C 0xF7 0x8C

The total length is 28 , almost triple the original length!

Implementing this in Ruby:

# using the three helpers from the previous implementations - build_tree, build_table, build_canonical_table
def build_canonical_codes(message)
  build_canonical_table(build_table(build_tree(message.bytes)))
end

def encode(message)
  codes = build_canonical_codes(message)
  grouped_codes = codes.values.group_by { |code| code.length }

  codes_count_by_length = grouped_codes.transform_values { |cs| cs.size }

  header1 = (1..15).map { |len| codes_count_by_length[len] || 0 }
  header2 = codes.keys
  body = message.bytes.map { |byte| codes[byte] }.join.chars.each_slice(8).map { |slice| slice.join.ljust(8, '0').to_i(2) }

  {
    header1: header1,
    header2: header2,
    body: body
  }
end

Then, the final message would be comprised of joining those parts together:

r = encode('Hello world')
[ :header1, :header2, :body ].map { |k| r[k].pack('C*').force_encoding('UTF-8') }.join

As I said before, for a simple message such as Hello world the archive is 27 bytes long, which is longer than the original data itself. But for something longer, like the AI chatbot response from before it is actually a bit better:

Original message: 3994
Encoded message: 2598
> header1: 15
> header2: 90
> body: 2573
Compression ratio: 49%

Decoding this is much simpler than DEFLATE: all there is to do is to reconstruct canonical Huffman codes from the number of codes by length (16-element-array)

# using `recreate_huffman_codes` helper from before
# decode_with_table is slightly different from decode_with_tree
def decode_with_table(bits, table)
    res = []
    buf = ''

    bits.each_char do |bit|
        buf += bit

        if table.key? buf
            res << table[buf]
            buf = ''
        end
    end

    res
end

def decode(message)
  bytes = message.bytes

  counts_encoded = bytes[0, 15]
  counts = counts_encoded.each_with_index.map { |n, len| [ len + 1 ] * n }.flatten
  codes = recreate_huffman_codes count

  symbols = bytes[15, codes.size]
  table = codes.zip(symbols).to_h
  
  body_encoded = bytes[15 + codes.size ..]
  body_bits = body_encoded.map { |b| b.to_s(2).rjust(8, '0') }.join
  body_decoded = decode_with_table(body_bits, table)

  body_decoded.pack('C*').force_encoding('UTF-8')
end

By the way, all the code in this blog is available in a separate repo .

This algorithm works much worse with binary data, however:

Raw data (PNG file): 2356 bytes
Encoded: 2605 bytes
Compression ratio: -2%

On a larger uncompressed binary data (same TTF font as before):

Raw data (TTF font): 7316 bytes
Encoded: 5336 bytes
Compression ratio: 27%

Clean before running a Spring Boot application in IntelliJ Idea

2025-07-12T00:00:00Z

My teammate has faced an issue when running our SpringBoot application in IntelliJ Idea: there were certain Gradle artifacts cached between switching the branches, which prevented the application from running. The issue is easily solved by running clean Gradle task. But my teammate was running it using IntelliJ's UI. And by default, it does not do a clean build before running - just the normal build, if the files have changed.

The solution was a bit of UI trickery.

In the run menu, edit the run configuration (from the meatball menu):

From the run configuration window, click the "Modify options" link:

In the "Before launch" section of a very long drop-down menu, click the "Add before launch task":

From the "Add task" pop-up menu select "Run Gradle task":

In the Gradle tak window fill out the "Gradle project" and "Tasks" fields (they both have auto-complete):

Bear in mind: this advice might become obsolete as IntelliJ team changes their new UI.

Modal text editors

2025-03-27T00:00:00Z

I was using vim on-and-off since my pal showed me Linux on my first ever computer, back in circa 2005. I then learned emacs back in 2015 and used it for remote development (over SSH). That was the first time I used a bunch of plugins to improve my developer experience. Then, circa 2022, I heard about an interesting take on vim ideology - Kakoune . I liked how it allowed user to see what he is about to operate on before actually performing an action. I did not end up using it often, but I liked the idea. Then at the end of 2024, I heard about Helix - another take on Vim ideology (or more like Kakoune at that point). At some random morning, scrolling the suggested videos on YouTube, I stumbled upon a semi-interesting (at the time) video about Vim productivity tips . Creator showed an interesting plugin, flash, which allowed him to quickly jump between any two points on the screen - much faster and easier than what I usually would do in Vim (relative line numbers and motions). I really liked the idea, to a point where I was on the edge of giving it a try. Since we have a few people in our team using NeoVim or Vim plugin for VS Code or IntelliJ Idea, I thought about using each of them for a while for work and comparing the experience.

IntelliJ Idea, baseline

For a number of years, the way I develop in IntelliJ is quite keyboard-centric - I disable tabs and use Shift-Shift menu and its tabs to navigate the project and perform actions.

The quick action menu has 6 tabs:

Classes
Files
Symbols
Actions
Text
All (combined of the above)

For the most part I only use Classes, Files and Actions tabs via a keyboard shortcut. From the Actions tab I rarely use more than "rename", "extract variable/method" or "copy path/reference".

The only other two features of the IDE I actually use are the file tree (so that I can see the structure of the project - it comes handy for Java packages organization, for example) which I configure to automatically focus the file currently opened in the editor and the terminal, which I use for pretty much everything else (including Git interactions and running / building project).

Actually, IntelliJ Idea gives you the statistics of features you use in the Help -> My Productivity menu.

In my case it looks like this (ordered by the usage frequency, descending - from most used to least):

Recent files
Go to declaration
Code completion
Syntax-aware selection ( Cmd + w )
Find in files
Context actions (suggestions like "remove unused", "replace with X", "import Y", etc.)
Find (in current file)
Toggle comment
Go to file
Go to class
Rename
Find and replace (in current file)
Generate code (constructors, getters and setters)
Go to implementation
Introduce variable
Change case (camel case)
Implement methods

Some of these (implement methods, introduce variable, generate code) might be hard to achieve in either of editors, so they are not requirements, but rather nice to haves.

So this would be my measure of success in other editors (in terms of comfort) - on top of movement in a document, I would like to compare the editors in terms of moving in the project (files, symbols / classes and references) and quick actions.

NeoVim

I thought about trying NeoVim since it was the hype at the time. I was dumbfolded by the fact it did not even come with the default config file to work with (in a stumbling contrast to Helix, which has :config-open and :config-reload commands built in). Yet I kept going, spending days perfecting my config. I switched my terminal emulator from Warp to Ghostty (so that I can see images right inside my terminal, improving file manager experience), installed a good dozen of plugins and messed with color schemes.

My full neovim config is actually public .

After almost a week, my NeoVim plugin setup looked like this:

Lazy - plugin manager
Telescope - for fzf and quick actions
Neo-tree - the file tree

These three plugins alone allowed me to already be quite a bit productive compared to vanilla Vim experience. But it was not a complete setup:

telescope-file-browser (?) - as an alternative for Neo-tree
telescope-ui-select - for LSP integration (code actions, specifically)
mason - for LSP management
mason-lspconfig - a middleware between mason and lspconfig
nvim-lspconfig - for configuring LSP in NeoVim
nvim-treesitter - for treesitter integration
toggleterm - terminal integration, toggleable; obsolete when using zellij
vim-sleuth - automatically detects tabwidth settings for the buffer
Comment.nvim - toggle comment for lines and blocks

A rather large set of plugins is required for autocomplete to work:

nvim-cmp - autocomplete engine
cmp-nvim-lsp - autocomplete integration for LSP
cmp-path - autocomplete for file paths
LuaSnip - snippets for Lua
cmp_luasnip - autocomplete integration for luasnip
friendly-snippets - a set of snippets
lspkind.nvim - icons for autocomplete options

Those plugins make up for almost a complete setup.

But I wanted to further improve the Vim experience somewhat so motions become easier (like in the video from the top of the post):

nvim-treesitter-textobjects (?) - for moving between text blocks (paragraphs, function / class scopes, parameters, etc.)
nvim-surround - for operations on surrounding characters (braces, brackets, quotes, ticks, etc.)
flash - the fast movements
vim-visual-multi - true multi-cursor editing (unlike visual block mode, which would require macros for the best experience)

And there are a few quality-of-life improvements:

undotree - local file history
noice - moves command line and search line to the middle of the screen, in a popup panel
indent-blankline - to show indentation lines
cmp-cmdline - for autocomplete suggestions in the Noice command popup
nvim-treesitter-context - shows the current context as fixed lines at the top of the screen (like in VSCode)
outline with outline-treesitter-provider - shows symbols defined in the current buffer
quicker - allows for better quickref buffer experience
telescope-live-grep-args - allows to search not by text (or regex) alone, but also by filename / filetype (and more)

The config I have at the moment allows the following:

file tree
- <leader>n toggles NeoTree
- ~~when NeoVim opens, NeoTree opens too~~ later I decided against this, since sometimes I want to just edit a single file without a care for the directory containing it
- NeoTree follows the currently open file
- NeoTree shows hidden files (including dot files and directories, like .github , which are hidden by default)
- when NeoTree is the last open buffer, NeoVim will quit
movements
- s performs a Flash search (with literal references to different points in text)
- S-s performs a treesitter Flash search
terminal
- trying to utilize zellij instead of built-in NeoVim terminal, following the "one tool for its purpose" principle
- <leader>t toggles a terminal
- C-\ C-n switches terminal to Normal mode, but I also remapped it to <Esc><Esc>
- i switches terminal to Insert mode
navigation
- C-left / C-right goes through the jumplist (similarly to back/forward in IntelliJ / VSCode)
editing
- ys<motion><character> wraps selected text in a specified character
- cs<old char><new char> changes the wrapped character around text under cursor from <old char> to <new char>
- ds<char> deletes wrapped character around text under cursor
- gcc comments / uncomments the line
- gc comments the selected block of text
- <Esc> in Normal mode sends nohlsearch to remove search highlights
- <leader>u shows a list of changes (undo tree), aka local file history
- C-n / C-<down> / C-<up> creates a new cursor for the same word occurrences / below current cursor / above current cursor
LSP
- S-k shows the hint for the symbol under cursor
- <leader>ca (built-in gra since NeoVim 0.11) shows c ode a ctions for the symbol under cursor in a Telescope panel
- treesitter-textobjects provides custom scopes (like w for word or W for WORD ):
  - ac / ic - outer class code / inner class code
  - af / if - outer function / inner function
  - as - local scope
  - ai / ii - outer function call / inner function call ( i for invocation )
  - ap / ip - outer / inner parameter
  - ar / ir - outer / inner return statement
- <leader>o shows list of symbols defined in the current buffer (outline)
- gr (built-in grr since NeoVim 0.11) shows a list of LSP references; <C-q> sends the list of locations to quick buffer
- gd shows a list of LSP definitions
- gI (built-in gri since NeoVim 0.11) shows a list of LSP implementations
- grn (built-in since NeoVim 0.11) renames a symbol under cursor
- C-s (built-in since NeoVim 0.11) shows method signature reference
- ]d / [d (built-in) goes to previous / next diagnostic message location (e.g. warnings, errors, suggestions in the code)
- C-w,d shows the diagnostic message at cursor
Telescope
- <leader><leader> opens a quick action menu (Telescope builtin)
- <leader>f opens search in files
- <leader>gf opens search only in files tracked by Git
- <leader>b opens a list of NeoVim buffers, ignoring current buffer and sorting by last used timestamp
- <leader>/ performs a live-grep (with arguments) on files
- C-q (in the search results; needs Zellij in the locked mode - C-g - to prevent conflict) moves files to quickref buffer; then
  - ]c / [c for next / previous quickref entry
  - <leader>q toggles quickref buffer
  - <leader>l toggles locations buffer

Aside from ridiculous amount of time spent configuring Neovim, it ticks most of my boxes:

Recent files ✅ <leader>b
Go to declaration ✅ gd
Code completion ✅ ( nvim-cmp ) C-space
Syntax-aware selection ✅ ( nvim-treesitter with customized incremental_selection ): initialize treesitter selection mode with gsn and then increment / decrement node selection with gss and gsm
Find in files ✅ <leader>/ ( telescope + telescope-live-grep-args , rg -powered)
Context actions ✅ gra
Find in current file ✅ /
Toggle comment ✅ gcc / gc<object> (e.g. gcaw - comment word)
Go to file ✅ <leader>f
Go to class ✅ <leader>S ( telescope + LSP)
Rename ✅ grn
Find and replace in current file :%s/<search>/<replace>
Generate code ❌
Go to implementation ✅ gri
Introduce variable ❌
Change case ✅ ( text-case.nvim or through external program)
Implement methods ❌

Helix

Actually, my first try was Helix, but I immediately stumbled upon the first blocker - the task I was working on at that time involved some Mustache templates. And Helix did not support it even on the most basic level (syntax highlighting). Moreover, Helix did not have plugins at the time, so there was little I could do for that particular task. The reason I wanted at least syntax highlighting had to do with the issue I was working on, which had misplaced conditionals in the template file, resulting in an incorrect rendering. I liked, however, how Helix came with a lot of handy utilities out of the box - the file picker, treesitter integration (so I could jump between the rest of Java / TypeScript codebase with ease).

I tried configuring a file tree via Yazi , which has an integration example on their website (using zellij terminal multiplexer panels), but it just refused to work for me on my Mac. That was when I timeboxed this and decided to switch my focus to NeoVim for the moment being.

After a while of just living my life, a new version of Helix has dropped, 25.07. And it introduced a few quality o flife changes. For example, the file explorer, which is now built in. It is not as powerful as Yazi, but it does the job. So I decided to switch to Helix for a while.

Upon getting back to Helix some time after, there was a new release, 25.07.1 which introduced a few quality of life improvements, including the file browser, sort of addressing the file tree issue I had before. Invoked with <leader>e (compared to <lead>f which just lists all the files in one long list), this one allows you to see directories. It is sort of an immediate-mode file manager as in it only shows the contents of a selected directory, but it is arguably more useful solution to the file tree problem than a list of all the files (recursively listing sub-directories). Additionally, tree-sitter was replaced so the syntax highlighting should be better - this would most likely be helpful in content like my blogs, often featuring multiple languages on top of the main Markdown.

Something that I liked about Helix was how it handles selections. For the most part, native multi-cursor is a great start - you just have to remember the difference between splitting the selection and selecting inside the selection - S (split) vs s (select) - both would create multiple cursors, just in a different way.

Different selection objects, including the ones provided by the tree-sitter - with the match mode you can select different objects ( ma and mi are your friends). But you can also surround the selection ( ms<char> ) and change the surrounding ( mr<from><to> ) and delete the surrounding ( md<char> ), which I find pretty handy sometimes.

Some things I find bearable are the use of external tools for some actions, instead of plugins. Since Helix lacks plugin system in any shape or form (as of late October 2025), you can just pipe the selection to the external program. One such case is changing the case of selection - I use this often , especially in conjunction with multiple cursors - to quickly edit, say test data or a bunch of constants or to refactor multiple class fields. In this case I found a comment on Github suggesting the use of ccase - you first need to install this Rust utility, ccase, and then you'll be able to send your selection(-s) by using :pipe ccase -t screamingsnake , for instance.

Something I did not realize until writing this blog is that there are some really useful default keybindings for insert mode, like Ctrl+w to delete the previous word, Alt+w to delete the next word, Ctrl+k to delete to the end of the line and Ctrl+x to trigger autocomplete.

One annoying thing about Helix is the buffer picker (available via <lead>b ) - it shows the list of open buffers (files). It does not appear to be sorting the elements of the list and the first element is always the current buffer . This is not really helpful if you want to quickly go the previously edited file. To make things worse, it does not seem to have any configuration around it, so the behaviour is pretty much set in stone.

I also am still missing one feature which made me try NeoVim in the first place, which is flash.nvim for quick jumps on the screen. Helix team did add something similar to HopWord command from hop.nvim , called goto_word , available via gw key shortcut. But unlike hop.nvim , Helix implementation is barely configurable (you can only configure the alphabet used for making labels) and comes with just one mode - HopWord . Additionally, it enforces exactly two-character abbreviations (jump labels). I thought this is rather limiting and since Helix is an open-source project, I decided to implement a behaviour similar to leap.nvim , since it is much less obstructive (it does not cover your entire screen in labels, making it super hard to figure out where you want to jump) and it allows to narrow down the places to jump by having prefix search. So I raised a PR and a suggestion discussion on Helix Github repo. To which the team just said "we don't want it" and dismissed the proposal:

We made intentional choices when implementing the goto_word behavior. We were quite aware if the nvim plugins. We are not going to replace or add alternative commands

One comment pointed to the plugin system when it is available:

I'm not inclined to add more jumping commands as core Helix commands like gw. There are a lot of different spins on jumping functions in Neovim plugins and I believe that future work should be done in plugins (once available) rather than as core faetures.

For context, plugin system has been discussed for over three years (as the moment of this writing, since September 2022) and apparently has been worked on for over two years (since October 2023). So I have no hope of seeing it released in the near future.

But instead of giving up, I decided to give a try to this work-in-progress plugin system and after a week a flash.hx plugin was born.

Here is my condensed cheatsheet of key shortcuts I use in Helix:

Pickers
- <lead>b buffer list
- <lead>e file explorer; far superior to file list
- <lead>/ search
- <lead>d show diagnostics (hints, errors, warnings, etc)
Multiple cursors
- , collapse all cursors into one
- s select in selection; create a new cursor at each selection; takes a regexp as a pattern
- S split selection; create a new cursor at each occurrence of pattern; takes regexp as a pattern
Selection
- v enters visual (selection) mode; this is how you can select multiple words - vwwww
- m enters marking mode; this is how you can select paragraphs, text and LSP objects, add, change and remove wrapping characters (like brackets and quotes); few examples:
  - mi( - select inside parentheses
  - ma" - select everything inside double-quotes and the double-quotes themselves
  - ms[ - surround selection with square brackets
  - mr[( - replace the surrounding square brackets with regular braces
  - mif - select the body of a function (method)
- = / < / > - format / unindent / indent selection
- :pipe or | - send selection to an external program and replace selection with the output of said program
  - as of version 25.7 you can also use interpolation with %{} , for instance, %{cursor_line} or %{buffer_name} , so you can pass in the filename to the external tool
- % - select an entire buffer (file)
- <lead><c> / <lead><C> - comment/uncomment the selection
Go to
- gl / gh - go to end of line / start of line (contrary to $ and ^ in Vim)
- gg / ge - go to first line / last line of the file
- Ctrl+i / Ctrl+o - go back and forth in your jumplist (places where the cursor has been placed; jumplist itself is available with <lead>j )
- gd / gD - go to definition / declaration
- gr - go to references
Copying (yanking)
- "+y - copy to the OS clip buffer (translates to "change the buffer to the + - OS clipboard and then yank the selection")

After a few weeks of working in Helix I was pleasantly surprised by how much out-of-the-box stuff it comes shipped with. And I did not have to spend a whole lot of time configuring it (aside from that week I spent developing flash.hx ).

It checks most of my boxes too:

Recent files ✅ <leader>b
Go to declaration ✅ gd
Code completion ✅ C-x
Syntax-aware selection ✅ m<selection><see tooltip>
Find in files ✅ <leader>/ (simplified)
Context actions ✅ <leader>a (depends on LSP)
Find in current file ✅ /
Toggle comment ✅ <leader>c
Go to file ✅ <leader>f
Go to class ✅ <leader>S (capital S for workspace symbols, lowercase s for current file symbols) (partial)
Rename ✅ <leader>r
Find and replace in current file ⚠️ (unconventinal): select scope ( % for current buffer), then use s to select occurrences or S to split selections into multiple cursors, then use actions - i or c to change the selections, d to delete selections
Generate code ❌
Go to implementation ✅ gi (depends on LSP)
Introduce variable ❌
Change case ✅ (through external tool): select text, then | (pipe it), then specify external program ( ccase -t snake , for instance)
Implement methods ❌

Kakoune

TBD

Final thoughts

I was surprised by how nice these seemingly limiting editors have become over the years (last time I used them was over-configured Emacs back in circa 2015)!

Although rough around the edges and coming with massive pros and massive cons, both editors are being actively developed to become even nicer.

Neovim:

👍 comes with some quite nice things out of the box
👎 the out-of-the-box niceities need LSP to be plugged in, which comes through a plugin manager
👍 insane amount of plugins for all sorts of things
👎 a proportionally insane amount of time required to find and configure the plugins to make for a comfortable experience
👍 overall, the experience that you can create is on its own league
👍 has a nice file tree implementation (yes, I like to see the structure of the directory and its parents)
👍 has very nice semantic and LSP selection expand
👍 has an OG Flash implementation
👍 has nice surround behaviour
👍 file picker powered by ripgrep so you can filter all you want
👎 both selection expand and surround are a bit tricky to trigger ( gsn followed by gss ; ysaw" and cs[( and ds" )

Helix:

👍 comes with a ton of nice things out of the box
👍 none of the niceities require extensive configuration (if any at all)
👎 very slow development cycle, so new features do not come out often (like twice a year, I believe)
👎 no plugin system, making the new features even less frequent or even possible
👎 very strongly opinionated developers, so if you are not comfortable with a feature - tough luck!
👎 does not have file tree
👎 file picker is very basic - only offers filtering by partial path match
👎 semantic selection expansion is not exactly expansion , but more like "select this exact scope"
👍 surround and semantic selection triggers are consistent e.g. make much more sense ( ms( , mr([ , md[ ; maf , maa )

My current stance is that Neovim is a much stronger contender with a much broader set of (much better implemented) features, but the amount of time you have to spend to get to that state is enormous. And whilst much nicely organised out of the box, Helix is very much undercooked (in my opinion).

Hence, for powerful and actual workloads - use Neovim, it is worth spending all that time configuring it for that purpose. For a casual relaxing but very restricted editing - Helix is an-okay choice.

Strongly-typed front-end: experiment 2, simple application, in Gleam / Lustre

2024-12-20T00:00:00Z

import gleam/float
import gleam/option.{type Option, None, Some}
import gleam/string
import lustre
import lustre/attribute.{value}
import lustre/element.{text}
import lustre/element/html.{button, div, input, p, select}
import lustre/event.{on_click, on_input}

type Shape {
  Circle
  Square
}

type Msg {
  ShapeChanged(s: Option(Shape))
  ValueChanged(x: Float)
  CalculateArea
}

type State {
  State(shape: Option(Shape), value: Float, area: Option(Float))
}

const pi = 3.14

fn calculate_area(shape: Shape, x: Float) -> Float {
  case shape {
    Circle -> pi *. x *. x
    Square -> x *. x
  }
}

fn init(_flags) -> State {
  State(shape: None, value: 0.0, area: None)
}

fn update(model: State, msg: Msg) -> State {
  case msg {
    ShapeChanged(s) -> State(..model, shape: s)
    ValueChanged(v) -> State(..model, value: v)
    CalculateArea ->
      State(
        ..model,
        area: option.map(model.shape, fn(s) { calculate_area(s, model.value) }),
      )
  }
}

fn handle_value_change(s: String) -> Msg {
  case string.is_empty(s) {
    True -> ValueChanged(0.0)
    False ->
      case float.parse(s) {
        Ok(x) -> ValueChanged(x)
        _ -> ValueChanged(0.0)
      }
  }
}

fn handle_shape_change(s: String) -> Msg {
  case s {
    "circle" -> ShapeChanged(Some(Circle))
    "square" -> ShapeChanged(Some(Square))
    _ -> ShapeChanged(None)
  }
}

fn view(model: State) {
  div([], [
    select([on_input(handle_shape_change)], [
      html.option([value("")], "Select shape"),
      html.option([value("circle")], "Circle"),
      html.option([value("square")], "Square"),
    ]),
    input([value(float.to_string(model.value)), on_input(handle_value_change)]),
    button([on_click(CalculateArea)], [text("Calculate area")]),
    p([], [
      text(
        "Area: " <> option.unwrap(option.map(model.area, float.to_string), ""),
      ),
    ]),
  ])
}

pub fn main() {
  let app = lustre.simple(init, update, view)
  let assert Ok(_) = lustre.start(app, "#app", Nil)

  Nil
}

Resulting bundle is quite big sitting at a whopping 65.9kb

Iterating a vector in C++

2024-10-31T00:00:00Z

Such a simple topic - iterating over a vector, is it even worth discussing?

Interestingly enough, there is a difference in how exactly you iterate - be it using iterators, for(:) sugar or plain old for(i=0; i<vec.size(); ++i) .

Let us see what output does a compiler produce in each of these cases.

sample1 (simple for loop):

std::vector<int> data{ 5, 3, 2, 1, 4 };

for (auto i = 0; i < data.size(); ++i) {
  moo(data[i]);
}

        mov     DWORD PTR [rbp-20], 0
        jmp     .L3
.L4:
        mov     eax, DWORD PTR [rbp-20]
        movsx   rdx, eax
        lea     rax, [rbp-64]
        mov     rsi, rdx
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::operator[](unsigned long)
        mov     eax, DWORD PTR [rax]
        mov     edi, eax
        call    moo(int)
        add     DWORD PTR [rbp-20], 1
.L3:
        mov     eax, DWORD PTR [rbp-20]
        movsx   rbx, eax
        lea     rax, [rbp-64]
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::size() const
        cmp     rbx, rax
        setb    al
        test    al, al
        jne     .L4

but

sample2 (reversed for loop):

for (auto i = data.size() - 1; i >= 0; --i) {
  moo(data[i]);
}

        lea     rax, [rbp-64]
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::size() const
        sub     rax, 1
        mov     QWORD PTR [rbp-24], rax
.L3:
        mov     rdx, QWORD PTR [rbp-24]
        lea     rax, [rbp-64]
        mov     rsi, rdx
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::operator[](unsigned long)
        mov     eax, DWORD PTR [rax]
        mov     edi, eax
        call    moo(int)
        sub     QWORD PTR [rbp-24], 1
        jmp     .L3

also

sample3 (foreach):

for (auto i : data) {
  moo(i)
}

        lea     rax, [rbp-80]
        mov     QWORD PTR [rbp-24], rax
        mov     rax, QWORD PTR [rbp-24]
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::begin()
        mov     QWORD PTR [rbp-88], rax
        mov     rax, QWORD PTR [rbp-24]
        mov     rdi, rax
        call    std::vector<int, std::allocator<int> >::end()
        mov     QWORD PTR [rbp-96], rax
        jmp     .L3
.L4:
        lea     rax, [rbp-88]
        mov     rdi, rax
        call    __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >::operator*() const
        mov     eax, DWORD PTR [rax]
        mov     DWORD PTR [rbp-28], eax
        mov     eax, DWORD PTR [rbp-28]
        mov     edi, eax
        call    moo(int)
        lea     rax, [rbp-88]
        mov     rdi, rax
        call    __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >::operator++()
.L3:
        lea     rdx, [rbp-96]
        lea     rax, [rbp-88]
        mov     rsi, rdx
        mov     rdi, rax
        call    bool __gnu_cxx::operator!=<int*, std::vector<int, std::allocator<int> > >(__gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > > const&, __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > > const&)
        test    al, al
        jne     .L4

but with -O1

sample1 (simple for loop):

        movabs  rax, 12884901893
        movabs  rdx, 4294967298
        mov     QWORD PTR [r12], rax
        mov     QWORD PTR [r12+8], rdx
        mov     DWORD PTR [r12+16], 4
        mov     QWORD PTR [rsp+8], rbp
        mov     rbx, r12
        jmp     .L10
.L20:
        add     rbx, 4
        cmp     rbp, rbx
        je      .L19
.L10:
        mov     edi, DWORD PTR [rbx]
        call    moo(int)
        jmp     .L20

sample2 (reversed for loop):

        movabs  rax, 12884901893
        movabs  rdx, 4294967298
        mov     QWORD PTR [rbp+0], rax
        mov     QWORD PTR [rbp+8], rdx
        mov     DWORD PTR [rbp+16], 4
        lea     rbx, [rbp+16]
        jmp     .L4
.L8:
        sub     rbx, 4
.L4:
        mov     edi, DWORD PTR [rbx]
        call    moo(int)
        jmp     .L8

sample3 (foreach):

        mov     r12, rax
        lea     rbp, [rax+20]
        mov     QWORD PTR [rsp+16], rbp
        movabs  rax, 12884901893
        movabs  rdx, 4294967298
        mov     QWORD PTR [r12], rax
        mov     QWORD PTR [r12+8], rdx
        mov     DWORD PTR [r12+16], 4
        mov     QWORD PTR [rsp+8], rbp
        mov     rbx, r12
        jmp     .L10
.L20:
        add     rbx, 4
        cmp     rbx, rbp
        je      .L19
.L10:
        mov     edi, DWORD PTR [rbx]
        call    moo(int)
        jmp     .L20

Interesting how iterating forwards adds an extra boundary check and a jump ( cmd rbx, rbp and je .L19 in samples #1 and #3), whereas iterating backwards does not.

But this find actually must come with one pretty big caveat: the cache lines. The number of assembly instructions is a pretty poor measure of performance - after all, CPU instructions are ridiculously fast, compared to any form of IO - specifically memory access.

The code is also available on Github: [ https://github.com/shybovycha/iterate-over-vector-in-cpp/ ]

The benchmarking results seem to prove the assumption: bacwards iteration seems to be faster:

Test	Min	Max	50%	90%	95%	delta
sample1 ( `++i` )	1325447	3762733	1339309	1416805	1429724	+7%
sample2 ( `--i` )	1290571	2596630	1308953	1334955	1342008	0%
sample3 ( `for(:)` )	1438847	3034474	1460698	1506001	1508657	+11%

Out of curiosity, I also benchmarked using the postfix operator instead of prefix: i++ (postfix) instead of ++i (prefix), which shows prefix operator is slightly faster:

Test	Min	Max	50%	90%	95%	delta
sample1 prefix ( `++i` )	1325447	3762733	1339309	1416805	1429724	+11%
sample1 postfix ( `i++` )	1292195	4155596	1308721	1370542	1382136	+7%
sample2 prefix ( `--i` )	1290571	2596630	1308953	1334955	1342008	+5%
sample2 postfix ( `i--` )	1256929	2598426	1267088	1277779	1282787	0%

Interestingly enough, there is some difference:

backward iteration with postfix decrement for (auto i=vec.size()-1; i>0; i--) is the fastest
backward iteration with prefix decrement --i is

5% slower than i--

forward iteration with postfix increment i++ is

3% slower than --i
7% slower than i--

forward iteration with prefix increment ++i is the slowest:

11% slower than i--
7% slower than --i
4% slower than i++

The above tests were ran 10000 times on a heap-allocated ( std::vector ) 10000 int elements.

Standard library iterators

Out of sheer curiosity, I decided to test how using an iterator would affect the results ( sample4 ):

for (auto it = a.begin(); it != a.end(); ++it) {
  moo(*it);
}

With both prefix and suffix (postfix) variations:

for (auto it = a.begin(); it != a.end(); it++) {
  moo(*it);
}

And the reverse iterator ( sample5 ):

for (auto it = a.end(); it != a.begin(); it--) {
  moo(*it);
}

Test	Min	Max	50%	90%	95%	delta
sample1 prefix ( `++i` )	1325447	3762733	1339309	1416805	1429724	+11%
sample1 postfix ( `i++` )	1292195	4155596	1308721	1370542	1382136	+7%
sample2 prefix ( `--i` )	1290571	2596630	1308953	1334955	1342008	+4%
sample2 postfix ( `i--` )	1256929	2598426	1267088	1277779	1282787	0%
sample3 ( `for(:)` )	1438847	3034474	1460698	1506001	1508657	+17%
sample4 prefix ( `++it` )	1470583	2834386	1495310	1561023	1564850	+22%
sample4 postfix ( `it++` )	1461634	2695552	1535905	1547865	1556815	+21%
sample5 prefix ( `--it` )	1462156	2802714	1470567	1478957	1512621	+18%
sample5 postfix ( `it--` )	1447920	2831859	1471334	1479670	1490088	+16%

The results are a bit surprising, looking at the assembly code generated by G++:

.L25:
        cmp     rbx, r13
        je      .L10
        mov     rbp, r13
        jmp     .L11
.L27:
        add     rbp, 4
        cmp     rbx, rbp
        je      .L26
.L11:
        mov     edi, DWORD PTR [rbp+0]
        call    moo(int)
        jmp     .L27
.L26:
        test    r13, r13
        je      .L18
        mov     rsi, r12
        sub     rsi, r13
.L15:
        mov     rdi, r13
        call    operator delete(void*, unsigned long)
        ; ... cleanup & exit ...
        ret

Clang 17.0.1 is a slightly different story:

.LBB0_1:
        cmp     r14, rax
        jne     .LBB0_2
        sub     r14, r15
        movabs  rax, 9223372036854775804
        cmp     r14, rax
        mov     qword ptr [rsp], r15
        je      .LBB0_11
        mov     rbp, r14
        sar     rbp, 2
        cmp     rbp, 1
        mov     rax, rbp
        adc     rax, 0
        lea     rdx, [rax + rbp]
        mov     rcx, r12
        cmp     rdx, r12
        jbe     .LBB0_14
        add     rax, rbp
        jae     .LBB0_16
.LBB0_17:
        test    r12, r12
        je      .LBB0_18
.LBB0_19:
        lea     rdi, [4*r12]
        call    operator new(unsigned long)@PLT
        mov     r15, rax
        jmp     .LBB0_21
.LBB0_14:
        mov     rcx, rdx
        add     rax, rbp
        jb      .LBB0_17
.LBB0_16:
        mov     r12, rcx
        test    r12, r12
        jne     .LBB0_19
.LBB0_18:
        xor     r15d, r15d
.LBB0_21:
        mov     dword ptr [r15 + 4*rbp], r13d
        test    r14, r14
        mov     rbx, qword ptr [rsp]
        jle     .LBB0_23
        mov     rdi, r15
        mov     rsi, rbx
        mov     rdx, r14
        call    memmove@PLT
.LBB0_23:
        test    rbx, rbx
        je      .LBB0_25
        mov     rdi, rbx
        call    operator delete(void*)@PLT
.LBB0_25:
        lea     rbp, [r15 + 4*rbp]
        lea     rax, [r15 + 4*r12]
        movabs  r12, 2305843009213693951
        jmp     .LBB0_26
.LBB0_3:
        cmp     r15, r14
        je      .LBB0_7
        lea     r14, [r15 - 4]
.LBB0_5:
        mov     edi, dword ptr [r14 + 4]
        call    moo(int)@PLT
        add     r14, 4
        cmp     r14, rbp
        jne     .LBB0_5
.LBB0_7:
        test    r15, r15
        je      .LBB0_9
        mov     rdi, r15
        call    operator delete(void*)@PLT
.LBB0_9:
        xor     eax, eax
        add     rsp, 8
        pop     rbx
        pop     r12
        pop     r13
        pop     r14
        pop     r15
        pop     rbp
        ret
.LBB0_11:
        lea     rdi, [rip + .L.str.1]
        call    std::__throw_length_error(char const*)@PLT
        jmp     .LBB0_30
        jmp     .LBB0_30
        mov     qword ptr [rsp], r15
.LBB0_30:
        mov     r14, rax
        cmp     qword ptr [rsp], 0
        je      .LBB0_32
        mov     rdi, qword ptr [rsp]
        call    operator delete(void*)@PLT
.LBB0_32:
        mov     rdi, r14
        call    _Unwind_Resume@PLT

Caching impact

One other assumption is that iterating over a vector backwards affects the memory caching in a pretty poor manner. This is a rather complex scenario to test, since there are two potential scenarios on how this could happen:

swap memory usage - happens when the dataset is so big it does not fit in the available RAM and OS has to switch to using disk instead of RAM; this is the worst-case scenario
data locality / CPU cache usage - CPUs usually try to predict memory access patterns and pre-load more data than actually required by a given operation; when the data block is accessed at random points (indices), CPU fails to pre-load the correct slices of memory

Since this blog is about iterating over a vector, it would be seem to be easier to generate absurdely large chunks of data and try to iterate over them. But in reality it would only happen with absurdely large files - my machine, for instance, sports 32GB of RAM, so it would be quite a task to generate multiple 32GB files.

Simulating random memory access will break the purpose of iterating over the elements one by one - in these cases we either use iterators to access a linked list (each element is in random part of RAM) or access elements by index. And there is no way to iterate over them by incrementing an index. Alternatively, we have to dereference indexes from another block of memory. Either way it has nothing to do comparing for loops.

Simulating accessing multiple cache lines could actually be easier - we just need to replace integers with structures of variable size and use std::list (linked list) instead of std::vector (single block of memory):

struct MyStruct {
  bool f0; // 1 byte
  float f1[100]; // sizeof(float) * 100 = 4 * 100 = 400 bytes
  double f2[53]; // sizeof(double) * 53 = 8 * 53 = 424 bytes
  char f3[64]; // sizeof(char) * 64 = 1 * 64 = 64 bytes
}; // total size = 889 bytes, likely to span across multiple cache lines

std::list<MyStruct> a;

and generate a large enough number of these objects:

#include <random>

std::random_device rd;
std::mt19937_64 gen(rd());

std::uniform_int_distribution<int> bool_dist(0, 1);
std::uniform_real_distribution<float> float_dist(0.0f, 1000.0f);
std::uniform_int_distribution<int> int_dist(std::numeric_limits<int>::min(), std::numeric_limits<int>::max());

for (auto i = 0; i < 10000; ++i) {
  MyStruct e;

  e.f0 = static_cast<bool>(bool_dist(gen));

  for (auto t = 0; t < 100; ++t)
    e.f1[t] = float_dist(gen);

  for (auto t = 0; t < 53; ++t)
    e.f2[t] = static_cast<double>(float_dist(gen));

  for (auto t = 0; t < 64; ++t)
    e.f3[t] = static_cast<char>(int_dist(gen) % 255);

  a.push_back(e);
}

Just to prevent memory leaks, cleanup the test data at the end:

for (auto& item : a) {
  delete item.f3;
}

a.clear();

These benchmarks take significantly longer time to run and produce big numbers:

Test	Min	Max	50%	90%	95%	delta
sample1 ( `++i` )	7882057	15385569	7948012	8004355	8031673	0%
sample2 ( `--i` )	7915870	16297070	8971293	9109493	9151567	+13%
sample3 ( `for(:)` )	8094894	16496526	8159004	8229138	8259364	+3%

Compare this to a structure which actually fits in one cache line. Cache line size is apparently 64 B - that is sixty-four bytes - on most CPUs ( AMD Zen , Intel Ice Lake ). My Apple M1 laptop has twice as much, 128 B , as shown by running sysctl -a | grep 'hw.cachelinesize' .

If the structure was reorganized to fit in that limit, like so:

struct MyStruct {
  bool f0; // 1 byte
  float* f1; // sizeof(float*) = 8 bytes
  double* f2; // sizeof(double*) = 8 bytes
  char* f3; // sizeof(char*) = 8 bytes
}; // total size = 25 bytes, likely to fir in a single cache line

The timings are considerably lower for the backwards iteration:

Test	Min	Max	50%	90%	95%	delta
sample1 ( `++i` )	7575860	20005185	7640651	7700057	7725933	0%
sample2 ( `--i` )	7466816	18068148	7657415	8109531	8208603	+6%
sample3 ( `for(:)` )	7832724	14264212	7882930	7932295	7960718	+3%

This highlights how predictive cache loading and data element size that fits into a single CPU cache line actually impacts RAM reads - by a significant margin.

Conclusion

In simplest case (vector of numbers), the worst case scenario is within 11% difference, with backwards iteration being the fastest, followed by forward iteration and foreach being the slowest of the bunch. Prefix decrement in the backwards iteration being the fastest. Bear in mind: these numbers are in nanoseconds , meaning the worst case scenario is 10000 (ten thousand) elements of variable size being iterated in 1.5 ms and the fastest being 1.3 ms - that is milliseconds . For comparison, rendering a frame at 120 FPS rate would take about 8.3 ms . And if you were to process some requests (in a client-server system), you would be able to handle 699 requests per second while iterating over this list. So realistically, the way you iterate a vector of integers, any of these techniques would only really matter if you are working on a really high performance system.

That being said, this is all true while the vector elements fit in one CPU cache line or cache block, that is L1 and L2 caches.

So if the data element does not fit in a block of 64 B (or 128 B on some CPUs), the performance impact of backwards iteration is going to be more significant. In this case we are talking about 8 ms vs 9.1 ms . In terms of FPS, the difference would be 125 FPS vs 109 FPS . Or about 15 requests per second more.

Gantt chart. Part 4

2024-06-27T00:00:00Z

Seems like every two years or so I hop on my Gantt chart implementation and rework it completely.

Last few attempts ( rev. 1 , rev. 2 , rev. 3 ) were alright, but I was never quite satisfied with the implementation - be it SVG, which has a toll on a browser and has quite limited customization functionality or Canvas API, with same limited customization but being fast.

With the recent introduction of grid layouts in CSS, now supported in all browsers , now seems like a perfect time to revisit the old implementations once again:

CodeSandbox / live demo

This revision now has a proper horizontal scrolling on the panel with bars - meaning the labels on the left panel stay in place whilst the left panel is scrollable. Moreover, the chart is now relies on pure HTML and CSS (being rendered with React though), making it is possible to use rich markup inside the bars and labels.

Implementation steps

The data for the tests is going to look like this:

export const data = [
  {
    id: 1,
    name: "epic 1"
  },
  {
    id: 2,
    name: "epic 2"
  },
  {
    id: 3,
    name: "epic 3"
  },
  {
    id: 4,
    name: "story 1",
    parent: 1
  },
  {
    id: 5,
    name: "story 2",
    parent: 1
  },
  {
    id: 6,
    name: "story 3",
    parent: 1
  },
  {
    id: 7,
    name: "story 4",
    parent: 2
  },
  {
    id: 8,
    name: "story 5",
    parent: 2
  },
  {
    id: 9,
    name: "lorem ipsum dolor atata",
    parent: 5
  },
  {
    id: 10,
    name: "task 2",
    parent: 5
  }
];

The main component, <Gantt> , initially was implementated as follows:

import React, { useMemo } from "react";

import style from "./gantt.module.css";

const LeftPaneRow = ({ id, name }) => {
  return <div className={style.row}>{name}</div>;
};

const LeftPane = ({ items }) => {
  return (
    <div className={style.left_pane}>
      <div className={style.left_pane_header}>/</div>

      <div className={style.left_pane_rows}>
        {items.map((item) => (
          <LeftPaneRow key={item.id} {...item} />
        ))}
      </div>
    </div>
  );
};

const RightPaneRow = ({ id, name }) => {
  return (
    <div className={style.row}>
      <div className={style.entry} style={{ left: 0 }}>
        {id}
      </div>
    </div>
  );
};

const RightPane = ({ items }) => {
  return (
    <div className={style.right_pane}>
      <div className={style.right_pane_header}>...scale...</div>
      <div className={style.right_pane_rows}>
        {items.map((item) => (
          <RightPaneRow key={item.id} {...item} />
        ))}
      </div>
    </div>
  );
};

export const flattenTree = (items) => {
  const queue = [];

  items.filter(({ parent }) => !parent).forEach((item) => queue.push(item));

  const result = [];
  const visited = new Set();

  while (queue.length > 0) {
    const item = queue.shift();

    if (visited.has(item.id)) {
      continue;
    }

    result.push(item);
    visited.add(item.id);

    items
      .filter((child) => child.parent === item.id)
      .forEach((child) => queue.unshift(child));
  }

  return result;
};

export const Gantt = ({ items }) => {
  const itemList = useMemo(() => flattenTree(items), [items]);

  return (
    <div className={style.gantt}>
      <LeftPane items={itemList} />
      <RightPane items={itemList} />
    </div>
  );
};

The core of the proper representation of this diagram is the CSS:

.gantt {
  display: grid;
  grid-template: 1fr / auto 1fr;
  grid-template-areas: "left right";
  width: 100%;
}

.gantt .left_pane {
  display: grid;
  grid-area: left;
  border-right: 1px solid #bbb;
  grid-template: auto 1fr / 1fr;
  grid-template-areas: "corner" "rows";
}

.gantt .left_pane .left_pane_rows {
  display: grid;
  grid-area: rows;
}

.gantt .left_pane .left_pane_header {
  display: grid;
  grid-area: corner;
}

.gantt .right_pane {
  display: grid;
  grid-template: auto 1fr / 1fr;
  grid-template-areas: "scale" "rows";
  grid-area: right;
  overflow: auto;
}

.gantt .right_pane .right_pane_rows {
  width: 10000px; /*temp*/
  display: grid;
  grid-area: rows;
}

.gantt .right_pane .right_pane_header {
  display: flex;
  grid-area: scale;
}

.gantt .row {
  height: 40px;
  align-items: center;
  display: flex;
}

.gantt .right_pane .row {
  position: relative;
}

.gantt .right_pane .row .entry {
  position: absolute;
  background: #eeeeee;
  padding: 0.1rem 0.5rem;
  border-radius: 0.4rem;
}

Good, we now have two panels with items aligned in rows and the right panel being scrollable if it gets really long. Next thing, position: absolute is absolutely disgusting - we use grid layout already! Instead, split each row into the same number of columns using grid and position the elements in there:

const RightPaneRow = ({ id, name, columns, start, end }) => {
  const gridTemplate = `auto / repeat(${columns}, 1fr)`;
  const gridArea = `1 / ${start} / 1 / ${end}`;

  return (
    <div
      className={style.row}
      style={{
        gridTemplate,
      }}
    >
      <div
        className={style.entry}
        style={{
          gridArea,
        }}
      >
        {id}
      </div>
    </div>
  );
};

and clean up the CSS a bit (like removing the position: absolute and reducing the width from 10000px down to 1000px ):

.gantt .right_pane .right_pane_rows {
  width: 1000px; /*temp*/
  display: grid;
  grid-area: rows;
}

.gantt .row {
  height: 40px;
  align-items: center;
  display: grid;
}

.gantt .right_pane .row {
  position: relative;
}

.gantt .right_pane .row .entry {
  background: #eeeeee;
  padding: 0.1rem 0.5rem;
  border-radius: 0.4rem;
}

Now, let's position the elements in each row using the column index:

const RightPanelRowEntry = ({ id, start, end, children }) => {
  const gridArea = `1 / ${start} / 1 / ${end}`;

  return (
    <div
      className={style.entry}
      style={{
        gridArea,
      }}
    >
      {children}
    </div>
  );
};

const RightPaneRow = ({ id, name, columns, start, end }) => {
  const gridTemplate = `auto / repeat(${columns}, 1fr)`;
  const gridArea = `1 / ${start} / 1 / ${end}`;

  return (
    <div
      className={style.row}
      style={{
        gridTemplate,
      }}
    >
      <div
        className={style.entry}
        style={{
          gridArea,
        }}
      >
        {id}
      </div>
    </div>
  );
};

const RightPaneHeaderRow = ({ columns, children }) => {
  const gridTemplate = `auto / repeat(${columns}, 1fr)`;

  return (
    <div
      className={style.right_pane_header_row}
      style={{
        gridTemplate,
      }}
    >
      {children}
    </div>
  );
};

const RightPaneHeader = ({ children }) => {
  return <div className={style.right_pane_header}>{children}</div>;
};

const RightPane = ({ items, columns }) => {
  const columnHeaders = [...Array(columns)].map((_, idx) => (
    <RightPaneHeader>{idx + 1}</RightPaneHeader>
  ));

  const rows = items.map((item) => (
    <RightPaneRow key={item.id} columns={columns}>
      <RightPanelRowEntry {...item}>{item.id}</RightPanelRowEntry>
    </RightPaneRow>
  ));

  return (
    <div className={style.right_pane}>
      <RightPaneHeaderRow columns={columns}>{columnHeaders}</RightPaneHeaderRow>
      <div className={style.right_pane_rows}>{rows}</div>
    </div>
  );
};

And add corresponding new CSS styles:

.gantt .right_pane .right_pane_header_row {
  display: grid;
  grid-area: scale;
}

.gantt .right_pane .right_pane_header_row .right_pane_header {
  display: grid;
  align-items: center;
  text-align: center;
}

This requires start and end defined for each entry:

export const data = [
  {
    id: 1,
    name: "epic 1",
    start: 1,
    end: 12,
  },
  {
    id: 2,
    name: "epic 2",
    start: 2,
    end: 4,
  },
  {
    id: 3,
    name: "epic 3",
    start: 9,
    end: 11,
  },
  {
    id: 4,
    name: "story 1",
    parent: 1,
    start: 6,
    end: 7,
  },
  // ...
};

And, to make it not repeat a dozen of inline CSS styles, we can utilize CSS variables:

const RightPaneRow = ({ id, columns, children }) => {
  return (
    <div className={style.row}>
      {children}
    </div>
  );
};

const RightPanelRowEntry = ({ id, start, end, children }) => {
  return (
    <div
      className={style.entry}
      style={{
        "--col-start": start,
        "--col-end": end,
      }}
    >
      {children}
    </div>
  );
};

const RightPane = ({ items, columns }) => {
  const columnHeaders = [...Array(columns)].map((_, idx) => (
    <RightPaneHeader>{idx + 1}</RightPaneHeader>
  ));

  const rows = items.map((item) => (
    <RightPaneRow key={item.id} columns={columns}>
      <RightPanelRowEntry {...item}>{item.id}</RightPanelRowEntry>
    </RightPaneRow>
  ));

  return (
    <div className={style.right_pane} style={{ "--columns": columns }}>
      <RightPaneHeaderRow>{columnHeaders}</RightPaneHeaderRow>
      <div className={style.right_pane_rows}>{rows}</div>
    </div>
  );
};

We can also re-use the same row for header:

const RightPaneHeaderRow = ({ children }) => {
  return <div className={style.right_pane_header_row}>{children}</div>;
};

And corresponding CSS:

.gantt .right_pane .right_pane_header_row {
  display: grid;
  grid-area: scale;

  grid-template: auto / repeat(var(--columns, 1), 1fr);
}

.gantt .right_pane .row {
  position: relative;

  grid-template: auto / repeat(var(--columns, 1), 1fr);
}

.gantt .right_pane .row .entry {
  background: #eeeeee;
  padding: 0.1rem 0.5rem;
  border-radius: 0.5rem;
  align-items: center;
  text-align: center;

  grid-area: 1 / var(--col-start, 1) / 1 / var(--col-end, 1);
}

I like to also change the fonts, since the default sans-serif just looks terrible:

@import url("https://fonts.googleapis.com/css2?family=Assistant:[email protected]&display=swap");

:root {
  font-family: "Assistant", sans-serif;
  font-optical-sizing: auto;
  font-weight: 300;
  font-style: normal;
  font-variation-settings: "wdth" 100;
}

And maybe add some grid lines for the rows:

.gantt .row:first-child {
  border-top: 1px solid var(--border-color, #eee);
}

.gantt .row {
  padding: 0 0.75rem;
  border-bottom: 1px solid var(--border-color, #eee);
}

Now let's add some padding to separate parent and child items of a chart:

const LeftPaneRow = ({ level, id, name }) => {
  const nestingPadding = `${level}rem`;

  return (
    <div className={style.row} style={{ "--label-padding": nestingPadding }}>
      {name}
    </div>
  );
};

.gantt .left_pane .row {
  padding-left: var(--label-padding, 0);
}

and fill out the level property when flattening the item tree:

export const flattenTree = (items) => {
  const queue = [];

  items
    .filter(({ parent }) => !parent)
    .forEach((item) => queue.push({ level: 0, item }));

  const result = [];
  const visited = new Set();

  while (queue.length > 0) {
    const { level, item } = queue.shift();

    if (visited.has(item.id)) {
      continue;
    }

    result.push({ ...item, level });
    visited.add(item.id);

    items
      .filter((child) => child.parent === item.id)
      .forEach((child) => queue.unshift({ item: child, level: level + 1 }));
  }

  return result;
};

And automate the number of columns calculation:

export const Gantt = ({ items }) => {
  const itemList = flattenTree(items);

  const startsAndEnds = items.flatMap(({ start, end }) => [start, end]);
  const columns = Math.max(...startsAndEnds) - Math.min(...startsAndEnds);

  return (
    <div className={style.gantt}>
      <LeftPane items={itemList} />
      <RightPane items={itemList} columns={columns} />
    </div>
  );
};

In order to make chart panel scrollable, one can set a width CSS property for the .right_pane_rows and .right_pane_header_row :

.gantt .right_pane .right_pane_rows {
  width: 2000px;
}

.gantt .right_pane .right_pane_header_row {
  width: 2000px;
}

The last bit for a this prototype would be to have a scale for the columns.

Assume a chart item has an abstract start and end fields - these could be dates or some domain-specific numbers (like a week in a quarter or a sprint, etc.). Those will then need to be mapped onto column index. Then the chart width (in columns) would be the difference between the smallest start value and the biggest end value:

export const Gantt = ({ items, scale }) => {
  const itemList = flattenTree(items).map((item) => ({
    ...item,
    ...scale(item), // assuming `scale` function returns an object { start: number; end: number }
  }));

  const minStartItem = minBy(itemList, (item) => item.start);
  const maxEndItem = maxBy(itemList, (item) => item.end);

  const columns = maxEndItem.end - minStartItem.start;

  return (
    <div className={style.gantt}>
      <LeftPane items={itemList} />
      <RightPane items={itemList} columns={columns} />
    </div>
  );
};

The minBy and maxBy helper functions could be either taken from lodash or manually defined like this:

const minBy = (items, selector) => {
  if (items.length === 0) {
    return undefined;
  }

  let minIndex = 0;

  items.forEach((item, index) => {
    if (selector(item) < selector(items[minIndex])) {
      minIndex = index;
    }
  });

  return items[minIndex];
}

For better navigation around this code we can add some types:

interface GanttChartItem {
  id: string;
  name: string;
}

interface GanttChartProps {
  items: GanttChartItem[];
  scale: (item: GanttChartItem) => { start: number; end: number };
}

function minBy<T>(items: T[], selector: (item: T) => number): T | undefined {
  // ...
}

export const Gantt = ({ items, scale }: GanttChartProps) => {
  // ...
};

export default function App() {
  const scale = ({ start, end }) => {
    return { start: start * 2, end: end * 2 };
  };

  return <Gantt items={data} scale={scale} />;
}

We can extend this even further by adding an API to provide labels for columns:

interface GanttChartProps {
  // ...
  scaleLabel: (column: number) => React.Element;
}

export const Gantt = ({ items, scale, scaleLabel }: GanttChartProps) => {
  // ...

  return (
    <div className={style.gantt}>
      <LeftPane items={itemList} />
      <RightPane items={itemList} columns={columns} scaleLabel={scaleLabel} />
    </div>
  );
};


const RightPane = ({ items, columns, scaleLabel }) => {
  const columnHeaders = [...Array(columns)].map((_, idx) => (
    <RightPaneHeader>{scaleLabel(idx)}</RightPaneHeader>
  ));

  // ...
};

export default function App() {
  const scale = ({ start, end }) => ({ start, end });
  };

  const scaleLabel = (col) => `${col}`;

  return <Gantt items={data} scale={scale} scaleLabel={scaleLabel} />;
}

This new API can then be utilized to show month names, for instance:

export default function App() {
  const scale = ({ start, end }) => {
    return { start, end };
  };

  const months = [
    "Jan",
    "Feb",
    "Mar",
    "Apr",
    "May",
    "Jun",
    "Jul",
    "Aug",
    "Sep",
    "Oct",
    "Nov",
    "Dec",
  ];

  const scaleLabel = (col) => months[col % 12];

  return <Gantt items={data} scale={scale} scaleLabel={scaleLabel} />;
}

Moreover, it is now possible to inline HTML and CSS in the name of each chart item:

export const LeftPaneRow = ({ level, name }) => {
  const nestingPadding = `${level}rem`;

  return (
    <div className={style.row} style={{ "--label-padding": nestingPadding }}>
      <span dangerouslySetInnerHTML={{__html: name}}></span>
    </div>
  );
};

And then in data.json (note that FontAwesome requires its CSS on a page in order to work):

[
  {
    id: 7,
    name: '<i style="font-family: \'FontAwesome\';" class="fa fa-car"></i>&nbsp;story with FontAwesome',
    parent: 2,
    start: 4,
    end: 6,
  },
  {
    id: 9,
    name: 'inline <em><b style="color: #5ebebe">CSS</b> color</em> <u style="border: 1px dashed #bebefe; padding: 2px; border-radius: 2px">works</u>',
    parent: 5,
    start: 5,
    end: 6,
  },
]

The API can be further improved by providing the render function for the bars' labels:

export const RightPane = ({ items, columns, scaleLabel, barLabel }) => {
  const rows = items.map((item) => (
    <RightPaneRow key={item.id} columns={columns}>
      <RightPaneRowEntry {...item}>{barLabel ? barLabel(item) : <>{item.id}</>}</RightPaneRowEntry>
    </RightPaneRow>
  ));

  // ...
};

export interface GanttChartProps {
  items: GanttChartItem[];
  scale: (item: GanttChartItem) => { start: number; end: number };
  scaleLabel: (column: number) => React.Element;
  barLabel: (item: GanttChartItem) => React.Element;
}

export const Chart = ({
  items,
  barLabel,
  scale,
  scaleLabel,
}: GanttChartProps) => {
  // ...
  return (
    <div className={style.gantt}>
      <LeftPane items={itemList} />
      <RightPane
        items={itemList}
        columns={columns}
        scaleLabel={scaleLabel}
        barLabel={barLabel}
      />
    </div>
  );
};

and then in App component:

import pluralize from "pluralize";

export default function App() {
  const barLabel = ({ start, end }) => (
    <>
      {end - start} {pluralize("month", end - start)}
    </>
  );

  // ...

  return (
    <Chart
      items={data}
      scale={scale}
      scaleLabel={scaleLabel}
      barLabel={barLabel}
    />
  );
};

So why Redux is bad?

2024-06-26T00:00:00Z

Redux is generally considered a bad choice for state management on front-end. But check out an average application' React application (or a few fragments of it):

import { Provider, useAtom } from 'jotai';
import { useQuery } from 'react-query';

const useInfo = () => {
  const { data, error, isLoading } = useQuery({
    queryKey: [ 'info' ],
    queryFn: () => fetch('/info').then(r => r.json()),
    staleTime: Infinity,
  });

  return {
    info: data,
    isLoading,
    error,
  };
};

const HomeWithoutProvider = () => {
  const { raiseToast } = useToast();

  const [ initialRender, setInitialRender ] = useState(false);

  const [ pageType, setPageType ] = useAtom(pageTypeAtom);

  const { info } = useInfo();

  useEffect(() => {
    if (info.isNewVersionAvailable) {
      raiseToast({
        // ...
      });
    }
  }, [info]);

  return (
    <div>...</div>
  );
};

const Home = () => (
  <Provider>
    <HomeWithoutProvider />
  </Provider>
);

const Routes = () => (
  <BrowserRouter>
    <Suspense fallback={<Loader />}>
      <Route element={<Home />} path="/" />
    </Suspense>
  </BrowserRouter>
);

const root = React.createRoot(document.getElementById('root'));

root.render(
  <StrictMode>
    <StyleThemeProvider>
      <ToastProvider>
        <GlobalErrorHandler>
          <ReactQueryProvider>
            <Routes />
          </ReactQueryProvider>
        </GlobalErrorHandler>
      </ToastProvider>
    </StyleThemeProvider>
  </StrictMode>
);

Usually most of the components ( Home , Routes ) and hooks ( useInfo ) are in separate files, but for the sake of simplicity I combined them all into one code block.

What I find suboptimal with this code is that it has at least three obvious different state management systems:

jotai for shared atoms (pieces of global state)
React.useState for internal component state
various React.Context s ( StyleThemeProvider , ToastProvider , GlobalErrorHandler , etc.)

On top of those, there are less obvious state management systems:

react-router uses internal router state, which could be treated as global application state
react-query uses its internal cache for each query
react-hook-form uses the form state of a component' ancestor (which could be declared on any level above the current component)

In the pursuit of encapsulation and reducing the boilerplate, front-end developers came up with all of these solutions aiming to solve the problem of managing application state.

So what exactly is this problem? And what are the issues all of the aforementioned solutions try to address?

As I see it, there are two competing camps:

containing the logic in small reusable chunks (hooks, components)
sharing chunks of state between different parts of the application

There are some side-tracks like dealing with asynchronous actions (like fetching the data from server), changing the state of external components (like showing a toast message), reducing the unnecessary re-renders.

Back in the day, Redux seemingly addressed these areas to a degree. Redux implements Flux architecture, which was compared to MVC (Model-View-Controller) architecture back in the day:

It became especially popular after Angular.JS' MVVM (Model-View-ViewModel) architecture implementation was considered slow with its dirty checks and constant re-rendering.

It could be said that Redux is being shipped in recent versions of React itself - with the use of useReducer() hook. One would rarely use Redux on its own, often sticking to a somewhat opinionated stack of reselect (for derived state), react-redux (to connect() components to the store) and redux-thunk or redux-sagas (for asynchronous dispatch() calls).

The aforementioned component could be implemented with the "conventional" (old) Redux approach like so:

import { createStore, combineReducers, applyMiddleware } from 'redux';
import { connect } from 'react-redux';
import { createSelector } from 'reselect';
import { thunk } from 'redux-thunk';

const infoReducer = (state = { isNewVersionAvailable: false }, action) => {
  switch (action.type) {
    case 'INFO_LOADED':
      return { ...state, ...action.payload };

    default:
      return state;
  }
};

const pageDataReducer = (state = { pageType: undefined }, action) => {
  switch (action.type) {
    case 'SET_PAGE_TYPE':
      return { ...state, pageType: action.pageType };

    default:
      return state;
  }
};

const toastReducer = (state = { isOpen: false, content: undefined }, action) => {
  switch (action.type) {
    case 'SHOW_TOAST':
      return { ...state, isOpen: true, content: action.payload };

    default:
      return state;
  }
};

const rootReducer = combineReducers({
  info: infoReducer,
  pageData: pageDataReducer,
  toast: toastReducer,
});

const store = createStore(rootReducer, applyMiddleware(thunk));

// Meet thunks.
// A thunk in this context is a function that can be dispatched to perform async
// activity and can dispatch actions and read state.
// This is an action creator that returns a thunk:
const loadInfoAction = () =>
  (dispatch) =>
    fetch('/info')
      .then(r => r.json)
      .then(payload => dispatch({ type: 'INFO_LOADED', payload }));

const raiseToastAction = (content) =>
  (dispatch) =>
    dispatch({ type: 'SHOW_TOAST', payload: content });

const setPageTypeAction = (pageType) =>
  (dispatch) =>
    dispatch({ type: 'SET_PAGE_TYPE', pageType });

const Home = ({ info, pageType, loadInfo, raiseToast, setPageType }) => {
  useEffect(() => {
    loadInfo();
  }, []);

  useEffect(() => {
    if (info.isNewVersionAvailable) {
      raiseToast({
        // ...
      });
    }
  }, [info]);

  return (
    <div>...</div>
  );
};

const mapStateToProps = ({ info, pageData: { pageType } }) => ({ info, pageType });

const mapDispatchToProps = (dispatch) => ({
  raiseToast: (content) => dispatch(raiseToastAction(content)),

  setPageType: (pageType) => dispatch(setPageTypeAction(content)),

  loadInfo: () => dispatch(loadInfoAction()),
});

const HomeContainer = connect(mapStateToProps, mapDispatchToProps)(Home);

const Routes = () => (
  <BrowserRouter>
    <Suspense fallback={<Loader />}>
      <Route element={<HomeContainer />} path="/" />
    </Suspense>
  </BrowserRouter>
);

const root = React.createRoot(document.getElementById('root'));

root.render(
  <Routes />
);

In my opinion, Redux is not suitable for complex projects for a few reasons:

it combines all states (both local and global) in one big messy furball; managing it is quite a hurdle
- as the project complexity grows, one can not just change a piece of state or selectors without affecting the entirety of the project (and teams)
- combining reducers into one supermassive function makes any state update unreasonably long and complex process (remember: each reducer returns a new state instance; now imagine having even a hundred of reducers, each of which returns a new state)
it is easy for a component to be re-rendered on any change to the state; a lot of effort goes into making sure selectors are well memoized and not re-calculated
asynchronous actions are a big unsolved mystery (are you going to use thunks, flux, sagas or something else?)

On a flip side, the idea itself could actually bring a lot of positives if cooked properly:

there is only one possible flow of data: via dispatch() call, through the reducers and back to the components connected to the store via component props
- this is supposed to make following the data (e.g. debugging the application) easy
components are pretty much stateless at this point, encapsulated and not having side effects leaking everywhere
logic is nicely separated from the representation and is encapsulated in the reducers (and maybe, to a small extent, in selectors)

Elm utilizes the language features and its own runtime combined with Redux-like architecture to improve some aspects of the more traditional pure JS way of things, where there are only opinionated libraries and no one way of doing things.

Consider Elm architecture and how it compares to Redux:

all the states are still combined into one big cauldron of chaos
by default, any component is just a function returning an array; the entire application will be rerendered on each state change, which is still suboptimal, since literally all components are connected to the store
asynchronous actions are handled separately by the runtime in a similar way to synchronous actions; each action returns a new state and a command (triggering the asynchronous processing)
- since commands are handled by the runtime and there's a handful of commands, all of them will (eventually) circle back to dispatching messages just like components do, following the same one-way data flow
reducers are a lot faster, since they are essentially a big switch..case statement (which is cheap)

The above component could be re-implemented in Elm as follows:

import Browser
import Html exposing (..)
import Html.Attributes exposing (style)
import Html.Events exposing (..)
import Http
import Json.Decode exposing (Decoder, map, field, bool)

main =
  Browser.element
    { init = init
    , update = update
    , subscriptions = subscriptions
    , view = view
    }

type alias Model =
  { info : Maybe (Result Http.Error Info)
  , toast : Maybe Toast
  , pageType : PageType
  }

type alias Info =
  { quote : String
  , source : String
  , author : String
  , year : Int
  }

type alias Toast =
  { content : String }

type PageType = Page1 | Page2

init : () -> (Model, Cmd Msg)
init _ =
  ({ info = Nothing, toast = Nothing, pageType = Page1 }, loadInfo)

type Msg
  = GotInfo (Result Http.Error Info)
  | ShowToast Toast
  | SetPageType PageType

update : Msg -> Model -> (Model, Cmd Msg)
update msg model =
  case msg of
    GotInfo result ->
      case result of
        Ok info ->
          ({ model | info = Just (Ok info) }, Cmd.none)

        Err e ->
          ({ model | info = Just (Err e) }, Cmd.none)

    ShowToast t ->
      ({ model | toast = Just t }, Cmd.none)

    SetPageType p ->
      ({ model | pageType = p }, Cmd.none)

subscriptions : Model -> Sub Msg
subscriptions model =
  Sub.none

view : Model -> Html Msg
view model =
  div []
    [ h2 [] [ text "Demo App" ]
    , viewInfo model.info
    , viewToast model.toast
    ]

viewInfo : Maybe (Result Http.Error Info) -> Html Msg
viewInfo mbInfoResult =
  case mbInfoResult of
    Nothing ->
      text "Loading..."

    Just infoResult ->
      case infoResult of
        Err _ ->
          div []
            [ text "Could not load info" ]

        Ok info ->
          div []
            [ text "App loaded" ]

viewToast : Maybe Toast -> Html Msg
viewToast mbToast =
  case mbToast of
    Nothing ->
      div [] []

    Just toast ->
      div [] [ text toast.content ]

loadInfo : Cmd Msg
loadInfo =
  Http.get
    { url = "/info"
    , expect = Http.expectJson GotInfo infoDecoder
    }

infoDecoder : Decoder Info
infoDecoder =
  map Info
    (field "isNewVersionAvailable" bool)

The good bits are:

forcing to handle all possible actions (messages) and results (HTTP success and error scenarios)
expressive language features (union types, strong typing, records, switch-case expressions) ensure robust code (as in this code does not leave room for mistakes like null/undefined/unhandled exceptions/unhandled code path/wrong value type)
no leeway for various ways to get things done (as in there is only one way to handle HTTP requests, only one way to handle asynchronous message dispatches, only one way to parse HTTP responses)

But if Redux spreads things apart compared to modern React, Elm feels like it spreads things further apart by handling effect results separately (like sending HTTP request, parsing HTTP response and processing the result by dispatching another message).

One other example would be PureScript (or rather Halogen ). Purescript itself elevates the complexity to the skies and beyond, by making you run around with monads like a headless chicken. Consider "simple" example of sending a HTTP request:

module Main where

import Prelude

import Affjax.Web as AX
import Affjax.ResponseFormat as AXRF
import Data.Either (hush)
import Data.Maybe (Maybe(..))
import Effect (Effect)
import Effect.Aff.Class (class MonadAff)
import Halogen as H
import Halogen.Aff (awaitBody, runHalogenAff)
import Halogen.HTML as HH
import Halogen.HTML.Events as HE
import Halogen.HTML.Properties as HP
import Halogen.VDom.Driver (runUI)
import Web.Event.Event (Event)
import Web.Event.Event as Event

main :: Effect Unit
main = runHalogenAff do
  body <- awaitBody
  runUI component unit body

type State =
  { loading :: Boolean
  , username :: String
  , result :: Maybe String
  }

data Action
  = SetUsername String
  | MakeRequest Event

component :: forall query input output m. MonadAff m => H.Component query input output m
component =
  H.mkComponent
    { initialState
    , render
    , eval: H.mkEval $ H.defaultEval { handleAction = handleAction }
    }

initialState :: forall input. input -> State
initialState _ = { loading: false, username: "", result: Nothing }

render :: forall m. State -> H.ComponentHTML Action () m
render st =
  HH.form
    [ HE.onSubmit \ev -> MakeRequest ev ]
    [ HH.h1_ [ HH.text "Look up GitHub user" ]
    , HH.label_
        [ HH.div_ [ HH.text "Enter username:" ]
        , HH.input
            [ HP.value st.username
            , HE.onValueInput \str -> SetUsername str
            ]
        ]
    , HH.button
        [ HP.disabled st.loading
        , HP.type_ HP.ButtonSubmit
        ]
        [ HH.text "Fetch info" ]
    , HH.p_
        [ HH.text $ if st.loading then "Working..." else "" ]
    , HH.div_
        case st.result of
          Nothing -> []
          Just res ->
            [ HH.h2_
                [ HH.text "Response:" ]
            , HH.pre_
                [ HH.code_ [ HH.text res ] ]
            ]
    ]

handleAction :: forall output m. MonadAff m => Action -> H.HalogenM State Action () output m Unit
handleAction = case _ of
  SetUsername username -> do
    H.modify_ _ { username = username, result = Nothing }

  MakeRequest event -> do
    H.liftEffect $ Event.preventDefault event
    username <- H.gets _.username
    H.modify_ _ { loading = true }
    response <- H.liftAff $ AX.get AXRF.string ("https://api.github.com/users/" <> username)
    H.modify_ _ { loading = false, result = map _.body (hush response) }

Now add the halogen-store package to the mix to make use of Redux-like state management:

module Main where

import Prelude

import Affjax.Web as AX
import Affjax.ResponseFormat as AXRF
import Data.Either (hush)
import Data.Maybe (Maybe(..))
import Effect (Effect)
import Effect.Aff.Class (class MonadAff)
import Halogen as H
import Halogen.Aff as HA
import Halogen.HTML as HH
import Halogen.HTML.Events as HE
import Halogen.HTML.Properties as HP
import Halogen.VDom.Driver (runUI)
import Web.Event.Event (Event)
import Web.Event.Event as Event
import Halogen.Store.Monad (class MonadStore, updateStore, runStoreT)
import Halogen.Store.Connect (Connected, connect)
import Halogen.Store.Select (selectAll)
import Effect.Aff (launchAff_)

data StoreAction
  = StoreSetUsername String
  | StoreMakeRequest
  | StoreReceiveResponse (Maybe String)

reduce :: State -> StoreAction -> State
reduce store = case _ of
  StoreSetUsername username ->
    store { username = username, result = Nothing }
  StoreMakeRequest ->
    store { loading = true }
  StoreReceiveResponse response ->
    store { loading = false, result = response }

initialStore :: State
initialStore = { username: "", loading: false, result: Nothing }

main :: Effect Unit
main = launchAff_ do
  body <- HA.awaitBody
  root <- runStoreT initialStore reduce component
  void $ runUI root unit body

type State =
  { loading :: Boolean
  , username :: String
  , result :: Maybe String
  }

data Action
  = SetUsername String
  | MakeRequest Event
  | ReceiveState (Connected State Unit)

deriveState :: Connected State Unit -> State
deriveState { context: { username, loading, result }, input: _ } =
  { username: username
  , loading: loading
  , result: result
  }

component :: forall query output m. MonadAff m => MonadStore StoreAction State m => H.Component query Unit output m
component =
  connect selectAll $ H.mkComponent
    { initialState: deriveState
    , render
    , eval: H.mkEval $ H.defaultEval
      { handleAction = handleAction
      , receive = Just <<< ReceiveState
      }
    }

render :: forall m. State -> H.ComponentHTML Action () m
render st =
  HH.form
    [ HE.onSubmit \ev -> MakeRequest ev ]
    [ HH.h1_ [ HH.text "Look up GitHub user" ]
    , HH.label_
        [ HH.div_ [ HH.text "Enter username:" ]
        , HH.input
            [ HP.value st.username
            , HE.onValueInput \str -> SetUsername str
            ]
        ]
    , HH.button
        [ HP.disabled st.loading
        , HP.type_ HP.ButtonSubmit
        ]
        [ HH.text "Fetch info" ]
    , HH.p_
        [ HH.text $ if st.loading then "Working..." else "" ]
    , HH.div_
        case st.result of
          Nothing -> []
          Just res ->
            [ HH.h2_
                [ HH.text "Response:" ]
            , HH.pre_
                [ HH.code_ [ HH.text res ] ]
            ]
    ]

handleAction :: forall output m. MonadAff m => MonadStore StoreAction State m => Action -> H.HalogenM State Action () output m Unit
handleAction = case _ of
  SetUsername username -> do
    updateStore $ StoreSetUsername username

  MakeRequest event -> do
    H.liftEffect $ Event.preventDefault event
    username <- H.gets _.username
    updateStore $ StoreMakeRequest
    response <- H.liftAff $ AX.get AXRF.string ("https://api.github.com/users/" <> username)
    updateStore $ StoreReceiveResponse (map _.body (hush response))

  ReceiveState input ->
    H.put $ deriveState input

The really nice things about this approach are:

components could be self-sufficient, as opposed to Elm:
- they can have both internal state and communicate with the external application via Aff
- they can be extracted into separate modules, making them actually reusable components
state selectors and connecting to a store are seamlessly implemented based on existing Halogen tools (subscriptions)
it feels like you do not have to worry about state growing big, since each component explicitly declares which parts of that messy furball it needs (derives)

The bad news is that everything relies on monads and transformers - lifting, mapping, flat-mapping are just the very tip of the iceberg. Once you hit some mysterious error - it is quite tricky to understand what is going on. Unlike Elm, which has really nicely structured, formatted and presented both error message, its location and ways to fix it.

Just looking at the type definitions is nauseaing at best:

handleAction :: forall output m. MonadAff m => MonadStore StoreAction State m => Action -> H.HalogenM State Action () output m Unit

And then there is this bit, lifting everything to the same monad and then mapping and flat-mapping it to get the response body:

MakeRequest event -> do
  H.liftEffect $ Event.preventDefault event
  username <- H.gets _.username
  updateStore $ StoreMakeRequest
  response <- H.liftAff $ AX.get AXRF.string ("https://api.github.com/users/" <> username)
  updateStore $ StoreReceiveResponse (map _.body (hush response))

And do not forget that this monad is merely describing the computation, you will have to run it at some point:

main = runHalogenAff do
  body <- HA.awaitBody
  root <- runStoreT initialStore reduce component
  let ui = runUI root unit body
  ui

The example on halogen-store suggests using launchAff_ , but then you will have to cast the return value type to match the monad of the main function ( Effect Unit ) or lift runStoreT to the Effect Unit monad - whichever you find suitable:

main = launchAff_ do
  body <- HA.awaitBody
  root <- runStoreT initialStore reduce component
  let ui = runUI root unit body
  void ui

But having to worry about all these intricacies actually strengthens the point that PureScript is not for the faint-harted - Elm prevails here.

The other drawback is that mixing logic in both component and the reducers is weird - it is not clear from the Flux architecture where the side-effects should live - like network calls, asynchronous actions, actions triggering other actions. Redux is known for suffering from all of these areas.

Elm solves this nicely with commands.

Halogen kind of takes a step backwards from Elm - it does have subscriptions, but it does not prevent you from issuing side effects from the handleActions . And halogen-store does not have a recipe for complex chained actions.

Ultimately, I don't think Redux is bad - the idea to have a full visibility into all possible application interactions is scary in a complex project and it is hard to come up with a clean way to work around it, but single-way data flow is actually nice.

Interesting how developers went from "we don't want Angular.js dirty checks - it is not clear where the data is flowing" to "we don't want a single point of contention for all application interactions" .

In my eyes, the four technologies (modern-day React, Redux, Elm and Purescript) all come with their own pros and massive cons and there is no good or one-size-fits-all solution among them. And none of them ultimately solves the problem of managing application state and interactions in a non-bloated way. Maybe Angular or React 19 have an answer?

Bun is still undercooked

2024-05-17T00:00:00Z

my Skunkworks project was trying out Bun . it was not a successful project, but there are some learnings:

TL;DR: I think Bun is still undercooked and despite being super cool and competitive on paper, it is a bit too early to use it in big projects. But hey, it works for my blog!

What is Bun

Bun is a combined alternative for NodeJS, package manager ( npm / yarn / pnpm ), bundler ( vite / webpack / esbuild ) and test runner ( vite / jest ).

Bun is ridiculously fast

Command	Yarn	Bun
`yarn install`	`49 sec`	`7.5 sec`
`vite build`	`14 sec`	`0.9 sec`
`vite test`	`forever`	`4.5 sec`

This most likely has to do with what happens in those tools - Bun went with parsing files as ASTs, applying transformations and running them in memory (to the best of my knowledge, digging through the Bun code)

Some things work out of the box

Dependency management works like a charm. No questions asked. Bun is just 7x faster. Comparing the node_modules directories:

Only in node_modules_yarn:
    .yarn-state.yml
    @aashutoshrathi
    @isaacs
    @npmcli
    @pkgjs
    @tootallnate
    abbrev
    agentkeepalive
    aggregate-error
    aproba
    are-we-there-yet
    asynciterator.prototype
    cacache
    chownr
    clean-stack
    color-support
    console-control-strings
    deep-equal
    delegates
    depd
    eastasianwidth
    encoding
    env-paths
    err-code
    es-get-iterator
    exponential-backoff
    foreground-child
    fs-minipass
    gauge
    graceful-fs
    has
    has-unicode
    humanize-ms
    ip
    is-lambda
    jackspeak
    jsonc-parser
    make-fetch-happen
    minipass-collect
    minipass-fetch
    minipass-flush
    minipass-pipeline
    minipass-sized
    minizlib
    mkdirp
    negotiator
    node-gyp
    nopt
    npmlog
    object-is
    p-map
    promise-retry
    retry
    set-blocking
    smart-buffer
    socks
    socks-proxy-agent
    ssri
    stop-iteration-iterator
    string-width-cjs
    strip-ansi-cjs
    tar
    unique-filename
    unique-slug
    wide-align
    wrap-ansi-cjs

Only in node_modules_bun:
    confbox
    es-object-atoms
    word-wrap

Curious to see if those packages missing in bun's node_modules are actually used anywhere.

Plugins

In Relational Migrator we use few plugins with vite, namely svgr , vanilla-extract and sentry . Bun only supports limited esbuild plugins and does not have the aforementioned plugins. Some of them work with minimal changes, some of them do not work entirely.

`svgr` plugin

svgr worked with minimal alterations:

import svgrEsbuildPlugin from 'esbuild-plugin-svgr';

Bun.build({
    plugins: [
        svgrEsbuildPlugin() as unknown as BunPlugin,
    ]
})

But required to change the imports from

import { ReactComponent as DatabaseAccessImage } from './assets/database-access-image.svg';

import DatabaseAccessImage from './assets/database-access-image.svg';

`vanilla-extract` plugin

This one loads vite server to compile the CSS and does not work no matter what I tried, throwing the following errors all over the place:

error: Styles were unable to be assigned to a file. This is generally caused by one of the following:

- You may have created styles outside of a '.css.ts' context
- You may have incorrect configuration. See https://vanilla-extract.style/documentation/getting-started
      at getFileScope (.../frontend/node_modules/@vanilla-extract/css/fileScope/dist/vanilla-extract-css-fileScope.cjs.dev.js:35:11)
      at generateIdentifier (.../frontend/node_modules/@vanilla-extract/css/dist/vanilla-extract-css.cjs.dev.js:175:7)
      at style (.../frontend/node_modules/@vanilla-extract/css/dist/vanilla-extract-css.cjs.dev.js:374:19)
      at .../frontend/src/shared/leafygreen-ui/badge/badge.css.ts:4:28

Followed by

error: Module._load is not a function. (In 'Module._load(file, parentModule)', 'Module._load' is undefined)
    at .../frontend/src/components/mapping-banner.css.ts:1:0

`sentry` plugin

This one was trivial and did not complain (I did not check if it actually works):

Bun.build({
    plugins: [
        sentryEsbuildPlugin({
                disable: !process.env.SENTRY_AUTH_TOKEN,
                org: 'mongodb-org',
                project: 'relational-migrator-frontend',
                telemetry: false,
                sourcemaps: {
                        filesToDeleteAfterUpload: '**/*.map',
                },
        }) as unknown as BunPlugin,
    ]
})

Bundling

Bun is great to run bundling, testing or manage packages from CLI when things are relatively simple. When you need plugins (for instance), the interactions become tricky. For specifying and configuring bundle-time plugins one needs to use Bun's JS/TS API and make a custom build script:

await Bun.build({ ... });

By default, Bun does not log anything, which is actually quite inconvenient - not even build failures are logged. One has to get the result of Bun.build() and manually process them, which is a bit of a bummer:

const result = await Bun.build(...);

if (!result.success) {
  console.error('Build failed');

  for (const message of result.logs) {
    console.error(message);
  }
} else {
  console.info('Build succeeded');
}

Configuration

Another inconvenient interaction - some actions require entire scripts (like build configuration, serving files, etc.). Then there is a config file, bunfig.toml where users can specify some configurations for Bun.

Running tests

This one had the most issues on my end.

`react-testing-library`

Bun declares support for react-testing-library , which worked as expected.

Browser APIs

Had to use happy-dom and configure it in the bunfig.toml to enable some of the UI testing features (such as access to the window object). Yet, happy-dom still lacks support for Canvas API , for instance.

`test.each`

It is one example of Bun's partial compatibility with Jest - with Jest one can use nice-ish string interpolation to generate test name:

test.each`
    currentStep | lastCompletedStep | progressType
    ${0}        | ${0}              | ${'active'}
    ${1}        | ${0}              | ${'inactive'}
    ${1}        | ${2}              | ${'checked'}
  `(
    'returns $progressType for $currentStep and $lastCompletedStep',
    ({ currentStep, lastCompletedStep, progressType }) => { })
);

With bun:test it is slightly different - you can't use arguments out of order, nor do you have access to their names. Neither can you use this nice syntactic sugar for defining test cases in a table manner.

const cases = [
    // currentStep | lastCompletedStep | progressType
    [ 0, 0, 'active' ],
    [ 1, 0, 'inactive' ],
    [ 1, 2, 'checked' ],
  ];

  test.each(cases)(
    'For %p and %p returns %p',
    (currentStep, lastCompletedStep, progressType) => {
      expect(getProgressType(currentStep, lastCompletedStep)).toEqual(
        progressType
      );
    }
  );

Oh, and there is no describe.each() functionality at all, which makes defining suites of tests more tedious.

Mocks

Mocks work as expected, out of the box. There are mocks for system clock, which is nice. Had to replace vi.fn() with mock() and a corresponding import { mock } from 'bun:test'; .

`ObjectContaining` matchers

When using nested matchers in the ObjectContaining , some of them are missing in Jest compatibility (like expect.toBeNumber ):

expect(nodes).toEqual([
        expect.objectContaining({
          id: 'node-1',
          position: { x: expect.toBeNumber(), y: expect.toBeNumber() },
        }),
]);

Had to use expect.any(Number) instead:

expect(nodes).toEqual([
        expect.objectContaining({
          id: 'node-1',
          position: { x: expect.any(Number), y: expect.any(Number) },
        }),
]);

Using `Fragment` import alongside `<>`

If a component contains both import { Fragment } from 'react' and uses a shorthand <> , Bun will yell at test time (but not at build time, interestingly enough):

SyntaxError: Cannot declare an imported binding name twice: 'Fragment'.

If you specify a different jsxFragmentFactory in tsconfig.json and set "jsx": "react" (and not "react-jsx" or anything), you will get further.

After meddling with Bun source code itself, I figured something (like it parses files' AST and modifies them to add missing imports, like Fragment but it ends up with duplicates), but even after applying some crude hacks to prevent it from adding those duplicate statements, I could not get to fix the issues. Left a comment on Bun's Github issue , but from my experience developers do not pay enough attention to those.

Ended up manually changing sources for the libraries in question in node_modules folder directly (just for the test), which did actually help. Might be worth changing it in the libraries directly, but that won't work with everything.

Ace editor

It still is kinda impossible to use UMD/AMD modules in conjunction with TS in Bun tests - the nature of UMD is that once the file is imported, it uses IIF to define stuff, but Bun does not tolerate this (I presume it only parses the AST of the imported file but does not actually execute it in the right order).

Hence Ace editor, which uses UMDs, can not really be used as intended.

Bun's meat and potatoes

I did a bit of digging in Bun's source code and it seems... immature - commented out code, ignored tests, thousand-line-functions and files ( js_parser.zig has 23.3k LOC ). And this is on top of using Zig, which is still at version 0.12 (as of writing of this post, 10 May 2024 ) and has quite limited standard library (no remove and find methods in lists, no hash sets, etc.).

Bottom line

My experience shows that Bun might fine to be used in new and low-risk projects, but it is not ready for a drop-in replacement in existing or more or less complex projects.

Strongly-typed front-end: experiment 2, simple application, in PureScript / Halogen

2024-05-17T00:00:00Z

A more "conventional" way to implement the front-end application in PureScript would be using a framework called Halogen .

Starting off with a "hello world" example:

module Main where

import Prelude

import Effect (Effect)
import Halogen as H
import Halogen.Aff as HA
import Halogen.HTML as HH
import Halogen.HTML.Events as HE
import Halogen.VDom.Driver (runUI)

main :: Effect Unit
main = HA.runHalogenAff do
  body <- HA.awaitBody
  runUI component unit body

data Action = Increment | Decrement

component =
  H.mkComponent
    { initialState
    , render
    , eval: H.mkEval $ H.defaultEval { handleAction = handleAction }
    }
  where
  initialState _ = 0

  render state =
    HH.div_
      [ HH.button [ HE.onClick \_ -> Decrement ] [ HH.text "-" ]
      , HH.div_ [ HH.text $ show state ]
      , HH.button [ HE.onClick \_ -> Increment ] [ HH.text "+" ]
      ]

  handleAction = case _ of
    Increment -> H.modify_ \state -> state + 1
    Decrement -> H.modify_ \state -> state - 1

Adding the utility code akin to the other technologies:

data Shape = Circle | Square

calculateArea :: Maybe Shape -> Float -> Float
calculateArea Nothing _ = 0
calculateArea (Just Circle) value = pi * value * value
calculateArea (Just Square) value = value * value

getShape :: String -> Maybe Shape
getShape "circle" = Just Circle
getShape "square" = Just Square
getShape _ = Nothing

This resurfaces few differences from Haskell, Elm and others:

there is no pi constant in the Prelude , so need to import one of the available definitions , I went with Data.Number
Float is not a type; there is Number , however
0 is not a Number , it is Int , confusing the audience

These are all minor differences, however. But this code is not a conventional PureScript either - it is working against the good practices of functional programming and thus defeats the purpose of these experiments. Examples of this are the heavy reliance on String instead of using the available type system.

Let us change that a bit:

import Data.String.Read (class Read)

data Shape = Circle | Square

calculateArea :: Shape -> Number -> Number
calculateArea Circle value = pi * value * value
calculateArea Square value = value * value

instance Read Shape where
  read = case _ of
    "square" -> Just Square
    "circle" -> Just Circle
    _ -> Nothing

instance Show Shape where
  show = case _ of
    Square -> "square"
    Circle -> "circle"

Now, to the UI:

import Halogen.HTML.Properties as HP

render state =
  HH.div_
    [
      HH.select [] [
        HH.option [ HP.value "" ] [ HH.text "Select shape" ],
        HH.option [ HP.value (show Circle) ] [ HH.text (show Circle) ],
        HH.option [ HP.value (show Square) ] [ HH.text (show Square) ]
      ],
      HH.input [],
      HH.div_ [ HH.text "<area>" ]
    ]

In the application state we need to store the selected shape and the value, so we can utilize records for that:

initialState _ = { shape: Nothing, value: Nothing }

Then we need to modify the possible actions. Let's stick to the same approach of utilizing the type system:

data Action = ChangeValue (Maybe Number) | ChangeShape (Maybe Shape)

The thing glueing the two together is the handleAction function:

handleAction = case _ of
  ChangeValue value ->
    H.modify_ \state -> state { value = value }
  ChangeShape shape ->
    H.modify_ \state -> state { shape = shape }

Here, unlike Haskell (to my best knowledge), the placeholder variable is being used for pattern matching against the only function argument. So instead of a little verbose

handleAction action = case action of
  -- ...

you can use this placeholder variable and just provide the branches for each of its possible values:

handleAction = case _ of
  -- ...

Modifying the state is done using the Halogen.Hooks.HookM.modify_ function, which allows us to only use the previous state value and provide a new state value, without the need to mess with monads. In turn, we modify the state record using the record syntax:

state { shape = newShapeValue }

Now the only bit left is tying the UI with the actions:

import Halogen.HTML.Events as HE
import Data.String.Read (read)
import Data.Number as N
import Data.Tuple (Tuple(..))

render state =
  HH.div_
    [
      HH.select [ HE.onValueChange onShapeChanged ] [
        HH.option [ HP.value "" ] [ HH.text "Select shape" ],
        HH.option [ HP.value (show Circle) ] [ HH.text (show Circle) ],
        HH.option [ HP.value (show Square) ] [ HH.text (show Square) ]
      ],
      HH.input [ HE.onValueChange onValueChanged ],
      HH.div_ [ HH.text "<area>" ]
    ]

onShapeChanged v = ChangeShape (read v)

onValueChanged v = ChangeValue (N.fromString v)

showArea state =
  case res of
    Nothing ->
      HH.text "Choose shape and provide its parameter"

    Just (Tuple shape area) ->
      HH.text $ "Area of " <> (show shape) <> " is " <> (show area)

  where
    res = do
      shape <- state.shape
      value <- state.value
      let area = calculateArea shape value
      pure (Tuple shape area)

Here is where most fun and benefit from using PureScript comes into play.

First of all, the HE.onValueChange event handler (the onShapeChanged and onValueChanged functions) - it will be called with the new value for the input instead of an entire event object. This allows us to skip unpacking the raw value from that object.

Then, the action dispatchers take the value from the input and try to parse it, returning a Maybe a :

onShapeChanged :: String -> Maybe Shape
onShapeChanged v = ChangeShape (read v)

onValueChanged :: String -> Maybe Number
onValueChanged v = ChangeValue (N.fromString v)

It is actually a quite important part, since the shape might not be selected (making the <select> value an empty string) and the value might be either a blank string or not a valid number string. PureScript does not allow us to not handle these cases, so whenever we parse the user input, we get a Maybe a value and we have to handle both scenarios when the value is valid and when it is not.

The function showArea is where this neatness comes together - we handle both values as one, using the Data.Tuple type to pair them together:

res = do
  shape <- state.shape -- unpacks `Shape` from `Maybe Shape`
  value <- state.value -- unpacks `Number` from `Maybe Number`
  let area = calculateArea shape value -- always returns a Number, since both `shape` and `value` are always provided
  pure (Tuple shape area) -- returns a tuple of shape and area, packed in a `Maybe`

The above code will shortcircuit whenever at any point it is trying to unpack a value from a Nothing and the whole do block will return Nothing .

Putting it all together:

module Main where

import Prelude

import Data.Maybe (Maybe(..))
import Data.Number as N
import Data.String.Read (class Read, read)
import Data.Tuple (Tuple(..))
import Effect (Effect)
import Halogen as H
import Halogen.Aff as HA
import Halogen.HTML as HH
import Halogen.HTML.Events as HE
import Halogen.HTML.Properties as HP
import Halogen.VDom.Driver (runUI)

data Shape = Circle | Square

calculateArea :: Shape -> Number -> Number
calculateArea Circle value = N.pi * value * value
calculateArea Square value = value * value

instance Read Shape where
  read = case _ of
    "square" -> Just Square
    "circle" -> Just Circle
    _ -> Nothing

instance Show Shape where
  show = case _ of
    Square -> "square"
    Circle -> "circle"

data Action = ChangeValue (Maybe Number) | ChangeShape (Maybe Shape)

component =
  H.mkComponent
    { initialState
    , render
    , eval: H.mkEval $ H.defaultEval { handleAction = handleAction }
    }
  where
  initialState _ = { shape: Nothing, value: Nothing }

  render state =
    HH.div_
      [
        HH.select [ HE.onValueChange onShapeChanged ] [
          HH.option [ HP.value "" ] [ HH.text "Select shape" ],
          HH.option [ HP.value (show Circle) ] [ HH.text (show Circle) ],
          HH.option [ HP.value (show Square) ] [ HH.text (show Square) ]
        ],
        HH.input [ HE.onValueChange onValueChanged ],
        HH.div_ [ showArea state ]
      ]

  onShapeChanged v = ChangeShape (read v)

  onValueChanged v = ChangeValue (N.fromString v)

  showArea state =
    case res of
      Nothing ->
        HH.text "Select shape and provide its value"

      Just (Tuple shape area) ->
        HH.text $ "Area of " <> (show shape) <> " is " <> (show area)
    where
      res = do
        shape <- state.shape
        value <- state.value
        let area = calculateArea shape value
        pure (Tuple shape area)

  handleAction = case _ of
    ChangeValue value ->
      H.modify_ \state -> state { value = value }
    ChangeShape shape ->
      H.modify_ \state -> state { shape = shape }

main :: Effect Unit
main = HA.runHalogenAff do
  body <- HA.awaitBody
  runUI component unit body

More than TypeScript

2024-05-12T00:00:00Z

Back in 2011 frontend was a very different thing - JavaScript had no class , Object.entries / Object.keys , promises were a proof of concept idea (unless you used 3rd party library bluebird ) and Node was v0.10.

Then came CoffeeScript , which added nice helper features to JavaScript - list comprehensions, classes, string interpolation and if statements (meaning you could use them for variable assignment):

# if statements
text = if happy and knowsIt
  chaChaCha()
else if sexy
  knowsIt()
else if tooSexy
  removeShirt()
else
  showIt()

# list comprehensions
courses = [ 'greens', 'caviar', 'truffles', 'roast', 'cake' ]
menu = (i, dish) -> "Menu Item #{i}: #{dish}"
menu i + 1, dish for dish, i in courses

# ranges and list comprehensions
countdown = (num for num in [10..1])

# iterating over object entries
yearsOld = max: 10, ida: 9, tim: 11

ages = for child, age of yearsOld
  "#{child} is #{age}"

Whilst it was still compiled to an inferior ES5 JavaScript, it helped to organise the code and make it substantially cleaner. The one drawback is that it did not provide any type safety (only added recently, via @flow and you would still have to duplicate your classes if you wanted to use it - once in coffeescript and once in @flow annotations) . Anyhow, CoffeeScript was a good tool for the task, if you ask me.

Then came Dart , Flow and TypeScript, which were also compiled to ES5 JavaScript, but instead of adding new syntax features, they aimed to solve a different problem - by introducing types they sought to reduce the number of runtime errors with type checks at compile time.

This sounded so good that many developers and companies immediately got on board. Alas, the new tech still suffered from the same issue as JavaScript itself and most common cause of runtime errors - the null and undefined still was a thing, causing the exact same runtime errors.

The one true benefit offered by TypeScript over the others was that it allowed to seamlessly use existing JavaScript code. And, provided you have the type signatures for that JavaScript code, it could even perform type checking on it too, effectively reducing the requirements for using TypeScript in the existing codebase. No wonder it was an easy buy-in for many projects.

Fast-forward to 2024 ( twelve years since its first release) and TypeScript dominates the frontend world. Over the years it seems to have been focused on improving the type system in terms of what can you do with types - union types, partial types, etc. It still suffers from the original issues though and there still are not too many new syntactic features to pair with its powerful type system (like pattern matching).

With the new EcmaScript standards, classes and promises becoming the first-class citizens in all browsers (even Internet Explorer / Edge, when it was still around), the APIs and syntax became more mature ( Object.entries , async / await to reduce callback hell , for .. of , const / let , string interpolation and many others). There are still no list comprehensions or conditional expressions / pattern matching though.

TypeScript still does help in what I think is a small subset of highly-specific scenarios like navigating the code (jump to definition / declaration / find usages) in the IDE and refactoring the code. But IDEs matured as well, code navigation is not as much of a feature unique to TypeScript anymore as it used to be (in the era of Sublime Text). TypeScript can prevent some really naive errors at compile time like using a number instead of an object, but I don't think developers run into them these days - because, again, IDEs are really powerful these days, even VSCode and they help eliminate such mistakes to a large degree.

Let's consider a real-world scenario (from my work project): server returns a list of LocationType objects. Each one of the objects can be either a Collection , a Document (in a specific collection) or a Table , each with its own subset of fields, specific to the type. We need to handle each case differently (display them on the UI differently).

The OpenAPI spec:

tableLocation:
  type: object
  required:
    - table
  properties:
    table:
      type: string

collectionLocation:
  type: object
  required:
    - collection
  properties:
    collection:
      type: string

documentLocation:
  type: object
  required:
    - collection
    - document
  properties:
    collection:
      type: string
    document:
      type: string

location:
  oneOf:
    - $ref: "#/components/tableLocation"
    - $ref: "#/components/collectionLocation"
    - $ref: "#/components/documentLocation"

And the TypeScript code generated by OpenAPI for the above spec looks like this:

interface CollectionLocation {
    collection: string;
}

interface DocumentLocation {
    collection: string;
    document: string;
}

interface TableLocation {
    table: string;
}

function instanceOfCollectionLocation(value: object): boolean {
    let isInstance = true;
    isInstance = isInstance && "collection" in value;

    return isInstance;
}

function instanceOfDocumentLocation(value: object): boolean {
    let isInstance = true;
    isInstance = isInstance && "collection" in value;
    isInstance = isInstance && "document" in value;

    return isInstance;
}

function instanceOfTableLocation(value: object): boolean {
    let isInstance = true;
    isInstance = isInstance && "table" in value;

    return isInstance;
}

function CollectionLocationFromJSONTyped(json: any, ignoreDiscriminator: boolean): CollectionLocation {
    if ((json === undefined) || (json === null)) {
        return json;
    }
    return {
        'collection': json['collection'],
    };
}

function DocumentLocationFromJSONTyped(json: any, ignoreDiscriminator: boolean): DocumentLocation {
    if ((json === undefined) || (json === null)) {
        return json;
    }
    return {
        'collection': json['collection'],
        'document': json['document'],
    };
}

function TableLocationFromJSONTyped(json: any, ignoreDiscriminator: boolean): TableLocation {
    if ((json === undefined) || (json === null)) {
        return json;
    }
    return {
        'table': json['table'],
    };
}

type JobLocation = CollectionLocation | DocumentLocation | TableLocation;

function JobLocationFromJSONTyped(json: any, ignoreDiscriminator: boolean): JobLocation {
    if ((json === undefined) || (json === null)) {
        return json;
    }
    return { ...CollectionLocationFromJSONTyped(json, true), ...DocumentLocationFromJSONTyped(json, true), ...TableLocationFromJSONTyped(json, true) };
}

This might be a bit too harsh on TypeScript as a language, considering this is not the best implementation (in my opinion), but this highlights one of the issues with TypeScript: this very same code caused a number of runtime errors caught by users - not exceptions, not compilation errors, but wrong UI behaviour - on the UI all locations were treated as a table location.

The reason why this was happening is the code itself - it does not really handle the choice type JobUpdateLocation correctly and instead of a choice type it returns a union type, to put it roughly - instead of oneOf it returns essentially allOf object.

Now, even if we were to rewrite it by hand (which would defeat the purpose of using OpenAPI and could be harder to keep in sync between client and server code), we would end up with something like this:

type CollectionLocation = {
    collection: string;
}

type DocumentLocation = {
    collection: string;
    document: string;
}

type TableLocation = {
    table: string;
}

function instanceOfCollectionLocation(value: object): boolean {
    return ("collection" in value) && !("document" in value);
}

function instanceOfDocumentLocation(value: object): boolean {
    return ("collection" in value) && ("document" in value);
}

function instanceOfTableLocation(value: object): boolean {
    return ("table" in value);
}

function CollectionLocationFromJSONTyped(json: any): CollectionLocation | undefined {
    if ((json === undefined) || (json === null)) {
        return undefined;
    }
    return {
        'collection': json['collection'],
    };
}

function DocumentLocationFromJSONTyped(json: any): DocumentLocation | undefined {
    if ((json === undefined) || (json === null)) {
        return undefined;
    }
    return {
        'collection': json['collection'],
        'document': json['document'],
    };
}

function TableLocationFromJSONTyped(json: any): TableLocation | undefined {
    if ((json === undefined) || (json === null)) {
        return undefined;
    }
    return {
        'table': json['table'],
    };
}

type JobLocation = CollectionLocation | DocumentLocation | TableLocation;

function JobLocationFromJSONTyped(json: any): JobLocation | undefined {
    if ((json === undefined) || (json === null)) {
        return undefined;
    }
    if (instanceOfCollectionLocation(json) && !instanceOfDocumentLocation(json) && !instanceOfTableLocation(json)) {
      return CollectionLocationFromJSONTyped(json);
    }
    if (instanceOfDocumentLocation(json) && !instanceOfCollectionLocation(json) && !instanceOfTableLocation(json)) {
      return DocumentLocationFromJSONTyped(json);
    }
    if (instanceOfTableLocation(json) && !instanceOfCollectionLocation(json) && !instanceOfDocumentLocation(json)) {
      return TableLocationFromJSONTyped(json);
    }
    return undefined;
}

This is quite a verbose code, with few bad practices in place (the use of any and object , need to always be conscious of undefined ), but this is literally what we ended up doing (the helpers, not redefining the types).

The types in TypeScript (or rather classes and interfaces) are also a point of a few confusing tricks you have to keep in mind - if we were to use class instead of type , as follows

class CollectionLocation {
    constructor(public collection: string) {}
}

class DocumentLocation {
    constructor(public collection: string, document: string) {}
}

class TableLocation {
    constructor(public table: string) {}
}

we would eventually run into the similar bug at runtime, which is a perfectly valid behaviour from the perspective of TypeScript, because of type compatibility , meaning DocumentLocation and CollectionLocation could be used interchangeably, since they have a subset of compatible fields:

const a: CollectionLocation = new DocumentLocation('col', 'doc'); // ok
const b: DocumentLocation = new CollectionLocation('col'); // also ok
const c: TableLocation = a; // not ok

This would not work if CollectionLocation , DocumentLocation and TableLocation were types instead:

const a: CollectionLocation = { collection: 'col', document: 'doc' }; // not ok
const b: DocumentLocation = { collection: 'col' }; // not ok
const c: TableLocation = b; // not ok

And the code to parse location from JSON is actually quite ugly. It would benefit so much from switch expressions or pattern matching !

And we still have to remember to handle those potentially undefined values whenever we use the helper functions. And here's an example from literally few days ago:

interface Job {
    projectId: string;
}

const jobsInProgress: Job[];

const message = useProjectStatus(
    jobsInProgress[0]?.projectId ?? ''
);

const useProjectStatus = (projectId: string) {
    const { data } = useQuery({ queryFn: () => fetch(`/project/${projectId}`) });

    return data;
};

The above code started throwing an error, since server responded with 500 Server Error . Reason was quite simple - the projectId , which we recently started to validate on the server (expecting it to be a valid ID), was a blank string. Interesting thing: no one has questioned the very line causing the "default value" for projectId to become an empty string:

jobsInProgress[0]?.projectId ?? ''

And issues like these are unbelievably common in front-end world, while being quite tricky to detect and resolve. To fix this particular issue we added a client-side validation (to an extent) to run the query only when the projectId value is provided:

useQuery({
  enabled: !!projectId,
  queryFn: () => fetch(`/project/${projectId}`)
})

The problem remains: it is really easy to miss this rather small fallback to an empty string.

There is a solution in Scala world that somewhat addresses this issue - refined types , which allows to have something along the lines of:

import eu.timepit.refined.*
import eu.timepit.refined.api.Refined
import eu.timepit.refined.string.*

type NonBlankString = MatchesRegex[".+"]

def useProjectStatus(projectId: String Refined NonBlankString) = ???

This would require wrapping all the values passed to fetchData in refineV[NonBlankString]() call and handling the case when the validation fails:

def generateId(): String = List("some", "").last

def fetchData(id: String Refined NonBlankString) = println(s"fetching '$id'...")

def main(args: Array[String]) = {
  for {
    id <- refineV[NonBlankString](generateId())
  } yield fetchData(id)
}

But in the land of TypeScript, there is only so much you can do - TypeScript only works at compile time.

The above examples might sound like very far-fetched edge case scenarios, but keep in mind: this is the code generated automatically by one of the most popular tools from a trivial schema. This is not as far-fetched as it might seem.

Can we do better in TypeScript? Something like refined in Scala? What if we had a powerful type system and syntax to support it? And, if possible, get rid of the null and undefined along the way?

The problem of null and undefined can be mitigated to an extent by using some concepts of functional programming, similarly to how I showcased some time ago . It would be quite hard to achieve, though, given the problem remains imbued in the language itself. Moreover, targeting the issues described above, it would take an entire standard library to really reduce the possibility of the issue:

const jobInProgressMaybe: Maybe<Job> = new List<Job>(jobsInProgress).first();
const projectIdMaybe: Maybe<string> = jobInProgress.map(j => j.projectId);

useQuery({
  enabled: projectIdMaybe.isSome(),
  queryFn: () =>
    projectIdMaybe
      .flatMap(projectId => fetch(`/project/${projectId}`))
      .orElse(Promise.reject('no projectId'))
})

We could try to address the empty string issue with the extensive type system:

type NonEmptyString<T extends string> = '' extends T ? never : T;
type MyString<T extends string> = T;

function fetchData<T extends string>(id: NonEmptyString<T>) {
  console.log(`Fetching ${id}`);
}

But it would only work if all the values are known at compile time, which is easily broken with the simplest test:

function generateId(): NonEmptyString<string> {
  return ['moo', ''][1] as NonEmptyString<string>;
}

fetchData(''); // not ok
fetchData(generateId()); // ok, but guaranteed undefined behaviour at runtime

And it would take even more effort to make all types uniquely identifiable (to solve the choice type problem).

The way most developers would approach solving similar issues in a real-world project would be (at best) adding some linters, checkers and relying on automated tests and high-quality code reviews. In my experience, this is a rather flimsy excuse rather than a real solution and it does not work most of the time - especially in edge case scenarios.

This is where I'd suggest to use another language altogether, which, similarly to CoffeeScript and TypeScript back in the day, solved some problems at compile time. And suggest I will.

There is a big warning before I proceed though: another technology is a rather big decision, not to be taken lightly. Some might see it as an impossible switch, only few small or indie developers on rather unimportant projects would make. But remember it was the same story with ES6, TypeScript, bundlers and pretty much any significant upgrade in the past.

Balance bike is a good tool to get you going - it gets you from walking to moving fast. But if you want to get faster and further, you have to drop it at some stage in favour of a more advanced bike.

Similarly to how TypeScript and ESNext got you from plain callback-hell-infested JavaScript code to a better place - you can refactor code faster, it saves you from a few errors at compile time, the code is much cleaner and conscise now. But if you really want to get even further, you will have to make a leap of faith, make an investment into the future.

Here is my big controversial suggestion: a pure functional language, one with strong type system, which does not have a concept of null and undefined in the first place, with a nice sweet syntax.

Before landing on a specific choice, check out Elm (dead, but a good starting point) and PureScript, in that order. Let me explain.

Elm is like a very simplified Haskell - it is a pure functional language with a subset of Haskell syntax. It has a nice compiler with really good error messages. It enforces a structure for your application (redux-like). It gives you a gentle introduction to the functional programming concepts and it targets browsers (web applications). With its architecture, you can look at the message (action from redux) type and see exactly what are all possible operations in the application (which makes reading code and getting to know new codebases much easier).

On a bad note, it is not being developed since 2019, it comes with an entire runtime (saves you from runtime errors, but blows up the bundle size) and it is a all-or-nothing commitment for the project - it is an all-in-one platform and if you want to gradually update your application from React - sorry, you will have to rewrite entire parts of you application entirely in Elm. The good point turned bad, having all possible actions defined in one message type make complex applications really complex (with one massive type definition, an issue very familiar to developers who had to deal with Redux).

here could have been an Elm code sample

The next step on this journey would be PureScript. It is an actively developed language, it has a minimal footprint after compiled to JS (much smaller than Elm), it has a very rich ecosystem and, best of all, it has a very simple interop with JS and it can compile just one module. Top it up with Halogen framework and you effectively got yourself Elm on steroids. The downside is that it is slightly more complex platform (language and framework) compared to Elm, so the learning curve is a bit steeper.

The above example of CoffeeScript code could be written in plain PureScript like this:

foreign import happy :: Boolean
foreign import knowsIt :: Boolean
foreign import sexy :: Boolean
foreign import tooSexy :: Boolean
foreign import chaChaCha :: String
foreign import knowsItStr :: String
foreign import removeShirt :: String
foreign import showIt :: String

import Data.Array ((..), mapWithIndex)
import Data.Map as M
import Data.Tuple (Tuple(..))

-- if statements with multiple branches become pattern matching
text
  | happy && knowsIt = chaChaCha
  | sexy = knowsItStr
  | tooSexy = removeShirt
  | otherwise = showIt

-- list comprehensions become function application
courses = [ "greens", "caviar", "truffles", "roast", "cake" ]

-- string interpolation is possible via an external packages
-- https://pursuit.purescript.org/packages/purescript-interpolate
import Data.Interpolate as I
menu' i dish = I.i "Menu Item " i ": " dish

-- https://pursuit.purescript.org/packages/purescript-template-strings
import Data.TemplateString.Unsafe ((<~>))
menu'1 i dish = "Menu Item ${i}: ${dish}" <~> { i: i, dish: dish }

import Data.TemplateString ((<->))
import Data.Tuple.Nested ((/\))
menu'2 i dish = "Menu Item ${i}: ${dish}" <-> [ "i" /\ i, "dish" /\ dish ]

-- pure PureScript string interpolation
menu i dish = "Menu Item " <> (show i) <> ": " <> dish

x i dish = menu (i + 1) dish

-- can not just call a function and ignore its result
x' = mapWithIndex x courses

-- ranges become Array monad
-- countdown :: Array Int
countdown = do
  num <- 10 .. 1
  pure num

-- JavaScript objects exist in a separate package
import Foreign.Object as FO
yearsOld' = FO.fromHomogeneous { max: 10, ida: 9, tim: 11 }

-- object as Map
yearsOld = M.fromFoldable [Tuple "max" 10, Tuple "ida" 9, Tuple "tim" 11]

y child age = (show child) <> " is " <> (show age)
ages = map y yearsOld
ages = map y yearsOld'

The real deal with this approach is how to migrate from an existing (most likely) React/TypeScript/(webpack | vite) ecosystem to PureScript?

Expanding on Scott Wlaschin's talk, you can (and probably should) separate the pure application logic from IO, potentially utilising the foreign imported functions to interact with the existing JS code (libraries). This way you keep your application logic error-free, and all the errors that can happen are shifted towards the presentation layer (MVC/MVP, remember this concept?).

This would be the best strategy for the most projects, migrating one bit at a time and making the application less and less error prone whilst not wreaking the havok by rewriting everything from scratch (very few businesses will buy into that).

The bigger issue is that most modern frontend apps I have seen are so mangled in mixing the business logic and the presentation layer, it would be challenging (to say the least) to unmangle it back to a reasonable code. Check how we handle UI action, triggering a HTTP request and updating both the UI (to display the request progress/status) and the application state (for other parts of the UI) at the same time.

here could have been a real-world application interaction handling code sample

Calling PureScript code from JavaScript (based on FFI example in PureScript book ):

module Test where

import Prelude

gcd :: Int -> Int -> Int
gcd 0 m = m
gcd n 0 = n
gcd n m
  | n > m     = gcd (n - m) m
  | otherwise = gcd (m - n) n

data ZeroOrOne a = Zero | One a

inc :: ZeroOrOne Int -> ZeroOrOne Int
inc Zero = Zero
inc (One n) = One (n + 1)

_zero = Zero
_one = One 1
_two = One 2

and then in JS:

import Test from 'Test.js';

Test.gcd(15)(20);

const _zero = new Test.Zero();
const _one = new Test.One(1);
const _two = new Test.One(2);

console.log(Test.inc(_zero));
console.log(Test.inc(_one));
console.log(Test.inc(_two));

In the other direction (calling JS code from PureScript):

export const setItem = key => value => () =>
  window.localStorage.setItem(key, value);

export const getItem = key => () =>
  window.localStorage.getItem(key);

and then in PureScript:

foreign import setItem :: String -> String -> Effect Unit

foreign import getItem :: String -> Effect Json

import Data.Argonaut (class DecodeJson, class EncodeJson)
import Data.Argonaut.Decode.Generic (genericDecodeJson)
import Data.Argonaut.Encode.Generic (genericEncodeJson)
import Data.Generic.Rep (class Generic)

-- define PhoneType

derive instance Generic PhoneType _

instance EncodeJson PhoneType where encodeJson = genericEncodeJson
instance DecodeJson PhoneType where decodeJson = genericDecodeJson

processItem :: Json -> Either String Person
processItem item = do
  jsonString <- decodeJson item
  j          <- jsonParser jsonString
  decodeJson j

main = do
  item <- getItem "person"
  initialPerson <- case processItem item of
    Left  err -> do
      log $ "Error: " <> err <> ". Loading examplePerson"
      pure examplePerson
    Right p   -> pure p

To align this with the original problem about JobLocation , here's how this would look like in PureScript:

module Main where

import Prelude
import Data.Argonaut (jsonParser)
import Data.Argonaut.Decode (decodeJson, (.:), printJsonDecodeError)
import Data.Argonaut.Decode.Class (class DecodeJson)  
import Data.Argonaut.Decode.Error (JsonDecodeError(..))
import Data.Argonaut.Encode (encodeJson, (:=), (~>))
import Data.Argonaut.Encode.Class (class EncodeJson)
import Data.Bifunctor (bimap)
import Data.Either (Either(..))
import Data.Generic.Rep (class Generic)
import Data.Show.Generic (genericShow)
import Effect (Effect)
import Effect.Console (log)

data JobLocation
  = CollectionLocation { collection :: String }
  | DocumentLocation { collection :: String, document :: String }
  | TableLocation { table :: String }

derive instance Eq JobLocation
derive instance Generic JobLocation _
instance showJobLocation :: Show JobLocation where
  show a = genericShow a

instance DecodeJson JobLocation where
  decodeJson json = do
    obj <- decodeJson json
    
    -- Check which fields are present
    maybeCollection :: Maybe String <- obj .:? "collection"
    maybeDocument :: Maybe String <- obj .:? "document" 
    maybeTable :: Maybe String <- obj .:? "table"
    
    let hasCollection = isJust maybeCollection
    let hasDocument = isJust maybeDocument
    let hasTable = isJust maybeTable
    
    case hasCollection, hasDocument, hasTable of
      true, true, false -> do
        collection <- obj .: "collection"
        document <- obj .: "document"
        pure $ DocumentLocation { collection, document }
        
      true, false, false -> do
        collection <- obj .: "collection"
        pure $ CollectionLocation { collection }
        
      false, false, true -> do
        table <- obj .: "table"
        pure $ TableLocation { table }
        
      _, _, _ -> Left $ AtKey "structure" $ UnexpectedValue json

parseJobLocation :: String -> Either String JobLocation
parseJobLocation jsonStr = jsonParser jsonStr >>= (decodeJson >>> bimap printJsonDecodeError identity)

-- Example usage
examples :: Effect Unit
examples = do
  let collectionJson = """{"collection": "users"}"""
  case parseJobLocation collectionJson of
    Left err -> log $ "Collection parse error: " <> err
    Right location -> do
      log $ "Parsed CollectionLocation: " <> show location
  
  let documentJson = """{"collection": "users", "document": "user123"}"""
  case parseJobLocation documentJson of
    Left err -> log $ "Document parse error: " <> err
    Right location -> do
      log $ "Parsed DocumentLocation: " <> show location
  
  let tableJson = """{"table": "analytics"}"""
  case parseJobLocation tableJson of
    Left err -> log $ "Table parse error: " <> err
    Right location -> do
      log $ "Parsed TableLocation: " <> show location
  
  let invalidJson = """{"data": "something"}"""
  case parseJobLocation invalidJson of
    Left err -> log $ "Expected error: " <> err
    Right location -> log $ "Unexpected success: " <> show location

main :: Effect Unit
main = do
  examples

While this code is type safe and will handle all of the edge cases just perfectly, I find it pretty hard to read. Parts that I would be unable to understand in a month of not working with this code, like this one:

parseJobLocation jsonStr = jsonParser jsonStr >>= (decodeJson >>> bimap printJsonDecodeError identity)

It could be rewritten in an imperative way:

parseJobLocation jsonStr = do
  json <- jsonParser jsonStr
  bimap printJsonDecodeError identity (decodeJson json)

But I doubt this makes it any more readable - the bimap printJsonDecodeError identity (decodeJson json) part, specifically. This is a common issue with Haskell and PureScript - some of the functions are weirdly named and can only be understood either with experience or by reading the docs (which, in turn, are also quite cryptic). Take the bimap function for example. Its signature looks like this:

bimap :: forall a b c d. (a -> b) -> (c -> d) -> f a c -> f b d

This does not really help non-seasoned Haskeller.

The docs only say this:

The bimap function maps a pair of functions over the two type arguments of the bifunctor.

I had to use it since there are two function calls in parseJobLocation : jsonParser :: String -> Either String Json and decodeJson :: Json -> Either JsonDecodeError a . This means if you just pipe the output of the first to the second ( jsonParser jsonStr >>= decodeJson ), the return type should match - because of the nature of the >>= operator: >>= :: (b -> c) -> m a b -> m a c , meaning for Either a b and a function b -> c the return type would have to be Either a c , in our case (Json -> a) -> (Either String Json) should return Either String a , but decodeJson returns Either JsonDecodeError a instead. Effectively, >>= operates on the second argument of a functor (in case of Either a b , operator >>= changes b ), whereas we want to change it first and then change the first argument as well (making the chain Either String Json -> Either JsonDecodeError a -> Either String a ). But this has to be done in one go - or, rather, in one argument of >>= . We can create such function by combining decodeJson which returns Either JsonDecodeError a and another function, which converts that middle argument, JsonDecodeError into String , making it Either String a . We achieve this by using the >>> operator which is an alias for composeFlipped and is defined as composeFlipped :: a b c -> a c d -> a b d . In fact, instead of using bimap _ identity _ it is better to use lmap _ _ which would only apply function to the left side of Either :

parseJobLocation jsonStr = do
  json <- jsonParser jsonStr
  lmap printJsonDecodeError (decodeJson json)

Explanations like those make it a huge barrier to entry for newcomers.

Incorporating the above solution in an existing TypeScript project might look a tad cumbersome. For Vite, there is a PureScript plugin:

import { defineConfig } from 'vite';
import purescript from 'vite-plugin-purescript';

export default defineConfig({
  plugins: [
    // ...
    purescript(),
  ],
});

The tricky part is that we have to re-define types in TypeScript:

type CollectionLocation = { collection: string };
type DocumentLocation = { collection: string; document: string };
type TableLocation = { table: string };

type JobLocation = CollectionLocation | DocumentLocation | TableLocation;

Moreover, since the function returns an Either , we would need that one too:

interface Left<E> {
  readonly _tag: 'Left';
  readonly value0: E;
}

interface Right<A> {
  readonly _tag: 'Right';
  readonly value0: A;
}

type Either<E, A> = Left<E> | Right<A>;

And then importing the function and calling it from TypeScript:

import { parseJobLocation } from '../output/JobLocation';

const loc = parseJobLocation('{ "collection": "col", "document": "doc" }');

if (loc._tag === 'Right') {
    // ...
}

Quite the bother, right? That's why it is most beneficial if the entire critical section can be written in PureScript entirely. In my case, the JobLocation is used for displaying a corresponding UI element (in React), so there's little benefit at a cost of lots of boilerplate and not the best experience defining those parsers.

I personally prefer Scala 3 implementation (not the integration with TypeScript part though, just the JSON parser implementation):

import io.circe.*
import io.circe.parser.*
import cats.implicits.*
import cats.effect.{IO, IOApp}
import io.circe.generic.semiauto.*

enum JobLocation:
  case CollectionLocation(collection: String)
  case DocumentLocation(collection: String, document: String)
  case TableLocation(table: String)

object JobLocation:
  given Decoder[DocumentLocation] = deriveDecoder
  given Decoder[CollectionLocation] = deriveDecoder
  given Decoder[TableLocation] = deriveDecoder

object JobLocationApp extends IOApp.Simple:

  import JobLocation.*

  def parseJobLocation(jsonString: String): Either[Error, JobLocation] =
    parse(jsonString).flatMap { json =>
      json.as[TableLocation].widen[JobLocation] orElse
      json.as[DocumentLocation].widen[JobLocation] orElse
      json.as[CollectionLocation].widen[JobLocation]
    }

  def run: IO[Unit] =
    val testCases = List(
      """{"collection": "users"}""",
      """{"collection": "orders", "document": "order-123"}""",
      """{"table": "analytics"}""",
      """{"invalid": "data"}"""
    )

    testCases.traverse_ { jsonStr =>
      parseJobLocation(jsonStr) match
        case Right(location) =>
          IO.println(s"Parsed: $jsonStr -> $location")

        case Left(error) =>
          IO.println(s"Failed to parse: $jsonStr -> ${error.getMessage}")
    }

This is a really straightforward implementation in my opinion. In this example, the parsing is literally boiled down to "try parsing TableLocation and if it returns Left , try parsing DocumentLocation instead, and if that returns Left , try parsing CollectionLocation , otherwise return what you got".

Alternatively, circe , the JSON parsing library for scala-cats used in the example above, provides an even neater way:

object JobLocation:
  given Decoder[DocumentLocation] = deriveDecoder
  given Decoder[CollectionLocation] = deriveDecoder
  given Decoder[TableLocation] = deriveDecoder

  given Decoder[JobLocation] =
    summon[Decoder[TableLocation]].widen[JobLocation] or
    summon[Decoder[DocumentLocation]].widen[JobLocation] or
    summon[Decoder[CollectionLocation]].widen[JobLocation]

def parseJobLocation(jsonString: String): Either[Error, JobLocation] =
    parse(jsonString).flatMap(_.as[JobLocation])

In this code, the parser for the parent type, JobLocation is defined as a combined parser of whichever case class manages to get parsed first:

summon[Decoder[TableLocation]].widen[JobLocation] or
summon[Decoder[DocumentLocation]].widen[JobLocation] or
summon[Decoder[CollectionLocation]].widen[JobLocation]

Just to reiterate, I do understand that converting the application (and developers) to this new weird technology is an almost impossible task, especially in a large long-lived project. One way to reason about it and justify the transition is the resilience requirements of a project (the need for actually error-prone code) and the amount of time and effort spent to date on finding and fixing those nasty bugs and undefined behaviours in an application.

IO impact

2024-05-10T00:00:00Z

At MongoDB I work on a Relational Migrator project - a tool which helps people migrate their relational database to MongoDB. And recently we grew interested in the performance of our tool. Due to the nature of the migrations, they are usually extremely long (potentially even never ending, for some scenarios). It is a rather valuable information to know where we can speed things up.

Hence we ran a profiler on a relatively big database of 1M rows. And this was what we saw:

The handleBatch method is where the meat and potatoes of our migration logic reside. It lasts for approx. 6.5 sec . We could have debated on which parts of this flame graph we could optimize (and we actually did), but we first decided to take a quick look at the same graph from the higher level - not the CPU time (when CPU is actually doing the active work) but the total time:

The entire application run took 4,937 sec ( 1hr 22min 17sec ). Of which, the migration itself took only 130 sec :

The biggest chunk of it was writing to MongoDB database at 120 sec :

The actual migration logic is really just 3.5 sec :

So out of 130 sec of the actual migration run, the actual logic took 3.5 sec or mere 2.69% . The rest was just IO (input/output). Which we also saw on the thread timeline:

Most time all the threads spent sleeping.

This is not new information, just a reminder that the slowest part of pretty much any application is input-output.

Simulating network outages in Docker

2024-02-08T00:00:00Z

Recently, for my work project I needed to simulate a system of ours going offline for a period of time (similar to having a network outage on a customer side).

I figured there are two ways to do it:

using Docker' networks
using host OS' firewall

With Docker network it is as easy as

docker network disconnect <network-name> <container-name>

This way you disconnect a container from a network (even if it is a bridge network, exposing the container to the host OS).

You can find a list of networks with docker network ls and list of containers with docker ps .

To roll it back (simulate recovery from an outage), simply

$ docker network connect <network-name> <container-name>

Disconnecting a container from a network is okay, but sometimes you might want to have a fine-grain control over the outage, like forbid a specific IP or port being accessed by your container.

With the firewall it is totally possible, but the steps differ for Linux and OSX.

In Linux you use iptables and control a rule group specific to Docker only:

$ iptables -I DOCKER -p tcp --dport 27017 -j DROP

The above command will block all TCP traffic on port 27017 for all Docker containers. To revert this, run

$ iptables -I DOCKER -p tcp —dport 22 ACCEPT

To control a specific IP address use the -s parameter:

$ iptables -I DOCKER -p tcp -s 192.168.0.10 --dport 27017 -j DROP
$ iptables -I DOCKER -p tcp -s 192.168.0.10 --dport 27017 -j ACCEPT

On the other hand, OSX uses a tool with BSD roots, pf . You can use the /etc/pf.conf file to mess with firewall rules.

Blocking traffic is achieved by adding a rule like below to the /etc/pf.conf file:

block drop out quick proto tcp from any to any port 27017

followed by reloading the rule list with

$ pfctl -f /etc/pf.conf

In case pf is disabled, one is enabled running

$ pfctl -e

Re-enabling is as easy as removing (or commenting out) the rule line in the /etc/pf.conf file and reloading it with pfctl -f /etc/pf.conf .

How unique are your bundles?

2024-01-04T00:00:00Z

In the modern front-end development we are all used to package managers, transpilation and bundling. These concepts are the biproduct of the tools which we hoped would simplify developer' burden and speed up the development.

However, how confident are you these tools are actually doing good job?

Developers seem comfortable off-loading the processing power to users' machine (browser, predominantly). We are not surprised by seeing slow websites anymore. A simple blog is downloading 55MB of JavaScript? Seems fine nowadays.

I currently work on a fairly small tool ( MongoDB Relational Migrator ), which also utilizes TypeScript, React and, of course, bundling. We use Vite for that. Our bundles are split into chunks (but I have accounted for that, too).

I went ahead and wrote a rather simple script which parses the bundles (using TypeScript compiler API, because why not), extracting the function definitions (both arrow functions and plain old function ) and counts how many times they occur in the file. For this last bit, to make sure I am not counting a => true and x => true as different occurrences, I am minimizing the function definition with uglifyjs and counting the SHA256 hashes of the minimized functions (just to have a reasonable key in my hashmap instead of entire function code).

These are my findings.

Out of 54 chunks, 47 are not css-in-js chunks. Out of 47 remaining, 7 have any significant duplication (over 5%). But when they do, they do it hard: duplication varies between 18% and a whopping 42% of sheer file size. Absolute numbers are also astonishing: 33% to 59% functions are duplicates.

Found 15192 functions, 8963 are unique (59%)
Duplicates length: 1518418 bytes out of 3537579 bytes are duplicate code (42.92%)

Found 1202 functions, 494 are unique (41.1%)
Duplicates length: 130649 bytes out of 340227 bytes are duplicate code (38.4%)

Found 513 functions, 231 are unique (45.03%)
Duplicates length: 50160 bytes out of 136057 bytes are duplicate code (36.87%)

Found 598 functions, 267 are unique (44.65%)
Duplicates length: 57607 bytes out of 164737 bytes are duplicate code (34.97%)

Found 17 functions, 10 are unique (58.82%)
Duplicates length: 1932 bytes out of 6532 bytes are duplicate code (29.58%)

Found 154 functions, 98 are unique (63.64%)
Duplicates length: 11140 bytes out of 45135 bytes are duplicate code (24.68%)

Found 968 functions, 651 are unique (67.25%)
Duplicates length: 52616 bytes out of 281406 bytes are duplicate code (18.7%)

I thought my code might be wrong, so I looked into the bundle code itself. Here's a short excerpt:

Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},yR.apply(this,arguments)}var Zce;function bR(){return bR=Object.assign?Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},bR.apply(this,arguments)}var Wce;function wR(){return wR=Object.assign?Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},wR.apply(this,arguments)}var Uce;function $R(){return $R=Object.assign?Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},$R.apply(this,arguments)}var Gce;function OR(){return OR=Object.assign?Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},OR.apply(this,arguments)}var Kce;function xR(){return xR=Object.assign?Object.assign.bind():function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e},xR.apply(this,arguments)}

See how the following fragment of code repeats multiple times:

function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e}

In fact, this exact same fragment of code repeats 137 times in the same piece of bundle chunk (same file):

By the way, this is a production build of our front-end, built using Vite, with minification enabled.

The raw length of this function code is 146 characters. So in a single place, in a single file, you have 136 * 146 = 19_992 bytes of waste. Meaning, browser has to load these 20KB of code, parse it and create 136 duplicating functions.

Looking at the overall size of 3.5MB of code in this chunk and its insane 41% duplicated code (in sheer bytes, not occurrences, so 1.5MB wasted), imagine how much faster this single file could have been loaded in a browser.

I was keen on seeing what functions get duplicated most often and ran my script on an entire build output directory. Here are top 80-ish offenders:

Function code (minimized)	Copies
`function(r){for(var t=1;t<arguments.length;t++){var n,o=arguments[t];for(n in o)Object.prototype.hasOwnProperty.call(o,n)&&(r[n]=o[n])}return r}`	2205
`function n(){return Object.assign&&Object.assign.bind(),n.apply(this,arguments)}`	1197
`function n(){return Object.assign,n.apply(this,arguments)}`	1008
`function(){}`	753
`function(i){this.a=i}`	250
`function(n,e){if(null==n)return{};for(var r,t={},f=Object.keys(n),u=0;u<f.length;u++)r=f[u],0<=e.indexOf(r)\|\|(t[r]=n[r]);return t}`	191
`function(e,r){if(null==e)return{};var t,n=function(e,r){if(null==e)return{};for(var t,n={},l=Object.keys(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|(n[t]=e[t]);return n}(e,r);if(Object.getOwnPropertySymbols)for(var l=Object.getOwnPropertySymbols(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|Object.prototype.propertyIsEnumerable.call(e,t)&&(n[t]=e[t]);return n}`	187
`function(e,r){return r=r\|\|e.slice(0),Object.freeze(Object.defineProperties(e,{raw:{value:Object.freeze(r)}}))}`	159
`function(n){return this===n}`	119
`function(r,t){if("object"!=typeof r\|\|null===r)return r;var e=r[Symbol.toPrimitive];if(void 0===e)return("string"===t?String:Number)(r);e=e.call(r,t\|\|"default");if("object"!=typeof e)return e;throw new TypeError("@@toPrimitive must return a primitive value.")}`	113
`function(r,e,t){return i=function(r,e){if("object"!=typeof r\|\|null===r)return r;var t=r[Symbol.toPrimitive];if(void 0===t)return String(r);t=t.call(r,e);if("object"!=typeof t)return t;throw new TypeError("@@toPrimitive must return a primitive value.")}(e,"string"),(e="symbol"==typeof i?i:String(i))in r?Object.defineProperty(r,e,{value:t,enumerable:!0,configurable:!0,writable:!0}):r[e]=t,r;var i}`	111
`function(t){t=function(t,r){if("object"!=typeof t\|\|null===t)return t;var i=t[Symbol.toPrimitive];if(void 0===i)return String(t);i=i.call(t,r);if("object"!=typeof i)return i;throw new TypeError("@@toPrimitive must return a primitive value.")}(t,"string");return"symbol"==typeof t?t:String(t)}`	111
`function(e,n,r){return n in e?Object.defineProperty(e,n,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[n]=r,e}`	104
`function(c,i){He.call(this,c,i)}`	94
`function(n,r){(null==r\|\|r>n.length)&&(r=n.length);for(var e=0,l=new Array(r);e<r;e++)l[e]=n[e];return l}`	93
`function(){return!0}`	92
`function(){return!1}`	78
`function(){return new gt(this)}`	77
`function(r){if(Array.isArray(r))return r}`	77
function(){throw new TypeError(`Invalid attempt to destructure non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.`)}	77
`function(t){return t&&"object"==typeof t&&"default"in t?t:{default:t}}`	76
`function(i,t){this.a=i,this.b=t}`	58
`function(){return this.a}`	49
`function(t,e){var r,n=Object.keys(t);return Object.getOwnPropertySymbols&&(r=Object.getOwnPropertySymbols(t),e&&(r=r.filter(function(e){return Object.getOwnPropertyDescriptor(t,e).enumerable})),n.push.apply(n,r)),n}`	49
`function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(c,e),Object.defineProperty(y,e,t.get?t:{enumerable:!0,get:function(){return c[e]}}))}`	49
`function(){return c[b]}`	49
`function(a,e,i){var r,t=i["aria-label"],n=i["aria-labelledby"],c=i.title;switch(a){case"img":return t\|\|n\|\|c?(l(r={},"aria-labelledby",n),l(r,"aria-label",t),l(r,"title",c),r):{"aria-label":"".concat(e.replace(/([a-z])([A-Z])/g,"$1 $2")," Icon")};case"presentation":return{"aria-hidden":!0,alt:""}}}`	49
`function(i){Di(this,i)}`	48
`function(r){return Object.getOwnPropertyDescriptor(e,r).enumerable}`	48
`function(){throw M(new De)}`	46
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var n,u,e=[],a=!0,o=!1;try{for(t=t.call(l);!(a=(n=t.next()).done)&&(e.push(n.value),!r\|\|e.length!==r);a=!0);}catch(l){o=!0,u=l}finally{try{a\|\|null==t.return\|\|t.return()}finally{if(o)throw u}}return e}}`	44
`function(n){throw M(new De)}`	39
`function(r){var n;return r&&"object"==typeof r&&"default"in r?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	39
`function n(r){return n(r)}`	38
`function(n){return typeof n}`	38
`function(o){return o&&"function"==typeof Symbol&&o.constructor===Symbol&&o!==Symbol.prototype?"symbol":typeof o}`	38
`function(r){var n;return r&&r.__esModule?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	37
`function(){return this.b}`	33
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var e,n,u,a,f=[],i=!0,o=!1;try{if(u=(t=t.call(l)).next,0===r){if(Object(t)!==t)return;i=!1}else for(;!(i=(e=u.call(t)).done)&&(f.push(e.value),f.length!==r);i=!0);}catch(l){o=!0,n=l}finally{try{if(!i&&null!=t.return&&(a=t.return(),Object(a)!==a))return}finally{if(o)throw n}}return f}}`	33
`function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}`	32
`function(n){return Ei(n)}`	30
`function(n){return L(fn,X,2,n,6,1)}`	30
`function(n){}`	29
`function(r){if(typeof Symbol<"u"&&null!=r[Symbol.iterator]\|\|null!=r["@@iterator"])return Array.from(r)}`	28
function(){throw new TypeError(`Invalid attempt to spread non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.`)}	28
`()=>{}`	27
`function(){return null}`	27
`function(n,o){e.exports=o(m,on(),dn)}`	27
`function(i){Fi(this,i)}`	23
`function(n,o){e.exports=o(dn,on(),m)}`	22
`function(){throw M(new Fe(Re((xe(),Ds))))}`	21
`function(){return this}`	20
`function(i,t){this.b=i,this.a=t}`	19
`function(n,e){throw M(new De)}`	18
`function(){return 0}`	17
`function(r){Object.defineProperty(e,r,Object.getOwnPropertyDescriptor(t,r))}`	16
`()=>{var l;return null!=(l=e.options.debugAll)?l:e.options.debugHeaders}`	16
`function(n){return n}`	15
`function(c){Kr.call(this,c)}`	15
`function(){return this.c}`	15
`function(){return this.d}`	15
`function(n,r){return n}`	14
`d=>d.id`	14
`function(){var r=function(t,o){return(r=Object.setPrototypeOf\|\|({__proto__:[]}instanceof Array?function(t,o){t.__proto__=o}:function(t,o){for(var n in o)Object.prototype.hasOwnProperty.call(o,n)&&(t[n]=o[n])}))(t,o)};return function(t,o){if("function"!=typeof o&&null!==o)throw new TypeError("Class extends value "+String(o)+" is not a constructor or null");function n(){this.constructor=t}r(t,o),t.prototype=null===o?Object.create(o):(n.prototype=o.prototype,new n)}}`	14
`function(_,o){_.__proto__=o}`	14
`function(o,r){for(var t in r)Object.prototype.hasOwnProperty.call(r,t)&&(o[t]=r[t])}`	14
`function(n){return!1}`	13
`function(t){var l=-1,n=null==t?0:t.length;for(this.clear();++l<n;){var r=t[l];this.set(r[0],r[1])}}`	12
`function(a,e,l){var i,r=l["aria-label"],t=l["aria-labelledby"],n=l.title;switch(a){case"img":return r\|\|t\|\|n?(f(i={},"aria-labelledby",t),f(i,"aria-label",r),f(i,"title",n),i):{"aria-label":"".concat(e.replace(/([a-z])([A-Z])/g,"$1 $2")," Icon")};case"presentation":return{"aria-hidden":!0,alt:""}}}`	12
`a=>a`	12
`a=>a()`	12
`function(n,a){n.a=a}`	11
`function(){ia.call(this)}`	11
`function(i,t,h){this.a=i,this.b=t,this.c=h}`	11
`function(n,r){return r}`	11
`function(){return this.a.gc()}`	11
`function(e){return e&&e.__esModule?e:{default:e}}`	11
`()=>n(!1)`	11
`function(i){this.b=i}`	10
`function(c){this.c=c}`	10
`function(){return this.f}`	10
`function(n){return n\|\|"div"}`	10
`function(n,r,i){var l;return i=null!=(l=i)?l:"div",n\|\|("string"==typeof(null==r?void 0:r.href)?"a":i)}`	10

Let's dive deeper, shall we?

Imagine for a second that we could define the duplicated functions once and then just reuse the short name instead (sounds reasonable, does it not?).

But not all of those functions could be de-duplicated in such way (at least not so easily). Some of these functions use the outer closure functions and variables (something defined outside of the function itself), so we can skip these. For instance, function(i){Di(this,i)} and function(){throw M(new De)} can be ignored.

Then, there are function using this . These might be tricky ( this is not what you think is a famous JavaScript mantra).

Lastly, some (if not most) of the functions could be either replaced with arrow functions or standard library function. But that is uncertain - one must understand what a function does first.

With those points in mind, let's look at the offenders once again:

Function code (minimized)	Copies	Notes
`function(r){for(var t=1;t<arguments.length;t++){var n,o=arguments[t];for(n in o)Object.prototype.hasOwnProperty.call(o,n)&&(r[n]=o[n])}return r}`	2205	spread operator?
`function n(){return Object.assign&&Object.assign.bind(),n.apply(this,arguments)}`	1197	`Object.assign` methods?
`function n(){return Object.assign,n.apply(this,arguments)}`	1008	`Object.assign` properties?
`function(){}`	753	no-op
`function(n,e){if(null==n)return{};for(var r,t={},f=Object.keys(n),u=0;u<f.length;u++)r=f[u],0<=e.indexOf(r)\|\|(t[r]=n[r]);return t}`	191	?
`function(e,r){if(null==e)return{};var t,n=function(e,r){if(null==e)return{};for(var t,n={},l=Object.keys(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|(n[t]=e[t]);return n}(e,r);if(Object.getOwnPropertySymbols)for(var l=Object.getOwnPropertySymbols(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|Object.prototype.propertyIsEnumerable.call(e,t)&&(n[t]=e[t]);return n}`	187	?
`function(e,r){return r=r\|\|e.slice(0),Object.freeze(Object.defineProperties(e,{raw:{value:Object.freeze(r)}}))}`	159	?
`function(r,t){if("object"!=typeof r\|\|null===r)return r;var e=r[Symbol.toPrimitive];if(void 0===e)return("string"===t?String:Number)(r);e=e.call(r,t\|\|"default");if("object"!=typeof e)return e;throw new TypeError("@@toPrimitive must return a primitive value.")}`	113	?
`function(r,e,t){return i=function(r,e){if("object"!=typeof r\|\|null===r)return r;var t=r[Symbol.toPrimitive];if(void 0===t)return String(r);t=t.call(r,e);if("object"!=typeof t)return t;throw new TypeError("@@toPrimitive must return a primitive value.")}(e,"string"),(e="symbol"==typeof i?i:String(i))in r?Object.defineProperty(r,e,{value:t,enumerable:!0,configurable:!0,writable:!0}):r[e]=t,r;var i}`	111	?
`function(t){t=function(t,r){if("object"!=typeof t\|\|null===t)return t;var i=t[Symbol.toPrimitive];if(void 0===i)return String(t);i=i.call(t,r);if("object"!=typeof i)return i;throw new TypeError("@@toPrimitive must return a primitive value.")}(t,"string");return"symbol"==typeof t?t:String(t)}`	111	?
`function(e,n,r){return n in e?Object.defineProperty(e,n,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[n]=r,e}`	104	?
`function(n,r){(null==r\|\|r>n.length)&&(r=n.length);for(var e=0,l=new Array(r);e<r;e++)l[e]=n[e];return l}`	93	array spread?
`function(){return!0}`	92	always-true
`function(){return!1}`	78	always-false
`function(r){if(Array.isArray(r))return r}`	77	self explanatory
`function(){throw new TypeError('Invalid attempt to destructure non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.')}`	77	`isSymbol` ?
`function(t){return t&&"object"==typeof t&&"default"in t?t:{default:t}}`	76	?
`function(t,e){var r,n=Object.keys(t);return Object.getOwnPropertySymbols&&(r=Object.getOwnPropertySymbols(t),e&&(r=r.filter(function(e){return Object.getOwnPropertyDescriptor(t,e).enumerable})),n.push.apply(n,r)),n}`	49	?
`function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(c,e),Object.defineProperty(y,e,t.get?t:{enumerable:!0,get:function(){return c[e]}}))}`	49	?
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var n,u,e=[],a=!0,o=!1;try{for(t=t.call(l);!(a=(n=t.next()).done)&&(e.push(n.value),!r\|\|e.length!==r);a=!0);}catch(l){o=!0,u=l}finally{try{a\|\|null==t.return\|\|t.return()}finally{if(o)throw u}}return e}}`	44	?
`function(r){var n;return r&&"object"==typeof r&&"default"in r?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	39	?
`function n(r){return n(r)}`	38	`Function.apply` ?
`function(n){return typeof n}`	38	`typeof`
`function(o){return o&&"function"==typeof Symbol&&o.constructor===Symbol&&o!==Symbol.prototype?"symbol":typeof o}`	38	?
`function(r){var n;return r&&r.__esModule?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	37	`import` ?
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var e,n,u,a,f=[],i=!0,o=!1;try{if(u=(t=t.call(l)).next,0===r){if(Object(t)!==t)return;i=!1}else for(;!(i=(e=u.call(t)).done)&&(f.push(e.value),f.length!==r);i=!0);}catch(l){o=!0,n=l}finally{try{if(!i&&null!=t.return&&(a=t.return(),Object(a)!==a))return}finally{if(o)throw n}}return f}}`	33	?
`function(n){}`	29	no-op
`function(r){if(typeof Symbol<"u"&&null!=r[Symbol.iterator]\|\|null!=r["@@iterator"])return Array.from(r)}`	28	?
`function(){throw new TypeError('Invalid attempt to spread non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.')}`	28	?
`()=>{}`	27	no-op
`function(){return null}`	27	always-null
`function(){return this}`	20	always-this
`function(){return 0}`	17	always-zero
`function(n){return n}`	15	identity
`function(n,r){return n}`	14	always-first-argument
`d=>d.id`	14	`.id`
`function(){var r=function(t,o){return(r=Object.setPrototypeOf\|\|({__proto__:[]}instanceof Array?function(t,o){t.__proto__=o}:function(t,o){for(var n in o)Object.prototype.hasOwnProperty.call(o,n)&&(t[n]=o[n])}))(t,o)};return function(t,o){if("function"!=typeof o&&null!==o)throw new TypeError("Class extends value "+String(o)+" is not a constructor or null");function n(){this.constructor=t}r(t,o),t.prototype=null===o?Object.create(o):(n.prototype=o.prototype,new n)}}`	14	?
`function(_,o){_.__proto__=o}`	14	`Object.is_a` ?
`function(o,r){for(var t in r)Object.prototype.hasOwnProperty.call(r,t)&&(o[t]=r[t])}`	14	?
`function(n){return!1}`	13	always-false
`a=>a`	12	identity
`a=>a()`	12	call first argument
`function(n,a){n.a=a}`	11	enum/const definition?
`function(n,r){return r}`	11	always-second-argument
`function(e){return e&&e.__esModule?e:{default:e}}`	11	import default
`function(c){this.c=c}`	10	enum/const definition?

As a matter of fact, someone on the internet did a very similar research few years ago. So I hoped to see the improvement in the build tools over the years.

As I mentioned above, our front-end is bundled with Vite. Let's see if using esbuild or bun (since both are fairly new and stand out in terms of architecture and performance) do a better job.

With few small adjustments to make things fair (e.g. build the same thing in the same way), like disabling the plugins for Vite, setting up svgr loader, here are some build time stats:

yarn install :

➤ YN0000: Done with warnings in 17s 798ms
yarn  12.11s user 19.69s system 175% cpu 18.122 total

bun install :

warn: esbuild's postinstall script took 748.9ms

 1028 packages installed [1.82s]
  Removed: 2
bun install  0.22s user 0.65s system 47% cpu 1.849 total

Bundler	Build time
bun	0.43s
esbuild	2.57s
vite	85.04s
webpack	138.64s

And the analysis of the built bundles:

vite :

Found 968 functions, 651 are unique (67.25%)
Found 598 functions, 267 are unique (44.65%)
Found 154 functions, 98 are unique (63.64%)
Found 17 functions, 10 are unique (58.82%)
Found 15192 functions, 8963 are unique (59%)
Found 1202 functions, 494 are unique (41.1%)
Found 513 functions, 231 are unique (45.03%)
= Total 18644 functions, 10714 are unique (57.4%)

Duplicates length: 52616 bytes out of 281406 bytes are duplicate code (18.7%)
Duplicates length: 57607 bytes out of 164737 bytes are duplicate code (34.97%)
Duplicates length: 11140 bytes out of 45135 bytes are duplicate code (24.68%)
Duplicates length: 1932 bytes out of 6532 bytes are duplicate code (29.58%)
Duplicates length: 1518418 bytes out of 3537579 bytes are duplicate code (42.92%)
Duplicates length: 130649 bytes out of 340227 bytes are duplicate code (38.4%)
Duplicates length: 50160 bytes out of 136057 bytes are duplicate code (36.87%)
= Total 1822522 out of 4511673 bytes are duplicate code (40.3%)

esbuild :

Found 46654 functions, 28952 are unique (62.06%)
Duplicates length: 6905599 bytes out of 9645594 bytes are duplicate code (71.59%)

bun :

Found 31113 functions, 25755 are unique (82.78%)
Duplicates length: 446020 bytes out of 5696964 bytes are duplicate code (7.83%)

webpack :

Found 2898 functions, 1434 are unique (49.48%)
Duplicates length: 320940 bytes out of 4645589 bytes are duplicate code (6.91%)

And a deeper analysis of the duplicated functions:

esbuild :

Function	Copies
`function(r){for(var t=1;t<arguments.length;t++){var n,o=arguments[t];for(n in o)Object.prototype.hasOwnProperty.call(o,n)&&(r[n]=o[n])}return r}`	2216
`function n(){return Object.assign&&Object.assign.bind(),n.apply(this,arguments)}`	1204
`function n(){return Object.assign,n.apply(this,arguments)}`	1010
`function(){}`	844
`function(t){return t&&"object"==typeof t&&"default"in t?t:{default:t}}`	260
`function(i){this.a=i}`	250
`function(n,e){if(null==n)return{};for(var r,t={},f=Object.keys(n),u=0;u<f.length;u++)r=f[u],0<=e.indexOf(r)\|\|(t[r]=n[r]);return t}`	203
`function(e,r){if(null==e)return{};var t,n=function(e,r){if(null==e)return{};for(var t,n={},l=Object.keys(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|(n[t]=e[t]);return n}(e,r);if(Object.getOwnPropertySymbols)for(var l=Object.getOwnPropertySymbols(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|Object.prototype.propertyIsEnumerable.call(e,t)&&(n[t]=e[t]);return n}`	194
`function(e,r){return r=r\|\|e.slice(0),Object.freeze(Object.defineProperties(e,{raw:{value:Object.freeze(r)}}))}`	160
`function(r,t){if("object"!=typeof r\|\|null===r)return r;var e=r[Symbol.toPrimitive];if(void 0===e)return("string"===t?String:Number)(r);e=e.call(r,t\|\|"default");if("object"!=typeof e)return e;throw new TypeError("@@toPrimitive must return a primitive value.")}`	137
`function(r,e,t){return i=function(r,e){if("object"!=typeof r\|\|null===r)return r;var t=r[Symbol.toPrimitive];if(void 0===t)return String(r);t=t.call(r,e);if("object"!=typeof t)return t;throw new TypeError("@@toPrimitive must return a primitive value.")}(e,"string"),(e="symbol"==typeof i?i:String(i))in r?Object.defineProperty(r,e,{value:t,enumerable:!0,configurable:!0,writable:!0}):r[e]=t,r;var i}`	134
`function(t){t=function(t,r){if("object"!=typeof t\|\|null===t)return t;var i=t[Symbol.toPrimitive];if(void 0===i)return String(t);i=i.call(t,r);if("object"!=typeof i)return i;throw new TypeError("@@toPrimitive must return a primitive value.")}(t,"string");return"symbol"==typeof t?t:String(t)}`	134
`()=>{}`	129
`function(n){return this===n}`	119
`function(e,n,r){return n in e?Object.defineProperty(e,n,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[n]=r,e}`	115
`function(n,r){(null==r\|\|r>n.length)&&(r=n.length);for(var e=0,l=new Array(r);e<r;e++)l[e]=n[e];return l}`	106
`function(c,i){Bs.call(this,c,i)}`	94
`function(){return!0}`	93
`function(r){if(Array.isArray(r))return r}`	83
`function(){throw new TypeError(` Invalid attempt to destructure non-iterable instance.\nIn order to be iterable, non-array objects must have a Symbol.iterator method. `)}`	83
`function(){return!1}`	79
`function(){return new cu(this)}`	77
`function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(b,e),Object.defineProperty(E,e,t.get?t:{enumerable:!0,get:function(){return b[e]}}))}`	76
`function(){return b[P]}`	76
`function(a,e,l){var i,r=l["aria-label"],t=l["aria-labelledby"],n=l.title;switch(a){case"img":return r\|\|t\|\|n?(s(i={},"aria-labelledby",t),s(i,"aria-label",r),s(i,"title",n),i):{"aria-label":"".concat(e.replace(/([a-z])([A-Z])/g,"$1 $2")," Icon")};case"presentation":return{"aria-hidden":!0,alt:""}}}`	76
`function(r){var n;return r&&r.__esModule?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	67
`function(n){return typeof n}`	64
`function(o){return o&&"function"==typeof Symbol&&o.constructor===Symbol&&o!==Symbol.prototype?"symbol":typeof o}`	64
`function n(r){return n(r)}`	63
`function(t,e){var r,n=Object.keys(t);return Object.getOwnPropertySymbols&&(r=Object.getOwnPropertySymbols(t),e&&(r=r.filter(function(e){return Object.getOwnPropertyDescriptor(t,e).enumerable})),n.push.apply(n,r)),n}`	61
`function(i,t){this.a=i,this.b=t}`	58
`function(r){var n;return r&&"object"==typeof r&&"default"in r?r:(n=Object.create(null),r&&Object.keys(r).forEach(function(e){var t;"default"!==e&&(t=Object.getOwnPropertyDescriptor(r,e),Object.defineProperty(n,e,t.get?t:{enumerable:!0,get:function(){return r[e]}}))}),n.default=r,Object.freeze(n))}`	50
`function(r){if(typeof Symbol<"u"&&null!=r[Symbol.iterator]\|\|null!=r["@@iterator"])return Array.from(r)}`	49
function(){throw new TypeError(`Invalid attempt to spread non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.`)}	49
`function(){return this.a}`	49
`function(i){p0(this,i)}`	48
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var n,u,e=[],a=!0,o=!1;try{for(t=t.call(l);!(a=(n=t.next()).done)&&(e.push(n.value),!r\|\|e.length!==r);a=!0);}catch(l){o=!0,u=l}finally{try{a\|\|null==t.return\|\|t.return()}finally{if(o)throw u}}return e}}`	46
`function(){throw St(new Ss)}`	46
`function(n){throw St(new Ss)}`	39
`()=>{"use strict";Vu(),Du()}`	38
`function(l,r){var t=null==l?null:typeof Symbol<"u"&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var e,n,u,a,f=[],i=!0,o=!1;try{if(u=(t=t.call(l)).next,0===r){if(Object(t)!==t)return;i=!1}else for(;!(i=(e=u.call(t)).done)&&(f.push(e.value),f.length!==r);i=!0);}catch(l){o=!0,n=l}finally{try{if(!i&&null!=t.return&&(a=t.return(),Object(a)!==a))return}finally{if(o)throw n}}return f}}`	37
`function(){return this.b}`	33
`function(){X(x)}`	32
`function(n){}`	31
`a=>a()`	30
`function(n){return V1(n)}`	30
`function(n){return dn(oi,mr,2,n,6,1)}`	30
`function(){return null}`	29
`function(){return this}`	24
`()=>{"use strict";Du()}`	23
`function(i){g0(this,i)}`	23
`function(t,r){var e;if(t)return"string"==typeof t?k(t,r):"Map"===(e="Object"===(e=Object.prototype.toString.call(t).slice(8,-1))&&t.constructor?t.constructor.name:e)\|\|"Set"===e?Array.from(t):"Arguments"===e\|\|/^(?:Ui\|I)nt(?:8\|16\|32)(?:Clamped)?Array$/.test(e)?k(t,r):void 0}`	22
`function(t,r){var e;if(t)return"string"==typeof t?P(t,r):"Map"===(e="Object"===(e=Object.prototype.toString.call(t).slice(8,-1))&&t.constructor?t.constructor.name:e)\|\|"Set"===e?Array.from(t):"Arguments"===e\|\|/^(?:Ui\|I)nt(?:8\|16\|32)(?:Clamped)?Array$/.test(e)?P(t,r):void 0}`	22
`function(n){return null!=n&&n instanceof Array}`	21
`function(r){if(Array.isArray(r))return P(r)}`	21
`function(e){return e&&e.__esModule?e:{default:e}}`	21
`function(t,r){var e;if(t)return"string"==typeof t?Q(t,r):"Map"===(e="Object"===(e=Object.prototype.toString.call(t).slice(8,-1))&&t.constructor?t.constructor.name:e)\|\|"Set"===e?Array.from(t):"Arguments"===e\|\|/^(?:Ui\|I)nt(?:8\|16\|32)(?:Clamped)?Array$/.test(e)?Q(t,r):void 0}`	21
`function(){throw St(new Os(il((qs(),_g))))}`	21
`function(n){return n}`	20
`function(n){return null!=n&&n.nodeType===Node.ELEMENT_NODE}`	20
`function(e){throw Error("Received unhandled value: ".concat(e))}`	20
`function(r,n){return Array.isArray(r)?r.concat(n):"string"==typeof r?r:void 0}`	19

bun :

Function	Copies
`function(){}`	739
`function(i){this.a=i}`	250
`function(r){for(var t=1;t<arguments.length;t++){var n,o=arguments[t];for(n in o)Object.prototype.hasOwnProperty.call(o,n)&&(r[n]=o[n])}return r}`	197
`()=>{}`	141
`function(n){return this===n}`	119
`function(c,f){f7.call(this,c,f)}`	94
`function(){return!0}`	91
`function(){return new p9(this)}`	77
`function(){return!1}`	76
`function(i,t){this.a=i,this.b=t}`	58
`function(n,e){if(null==n)return{};for(var r,t={},f=Object.keys(n),u=0;u<f.length;u++)r=f[u],0<=e.indexOf(r)\|\|(t[r]=n[r]);return t}`	53
`function(r,t){if("object"!=typeof r\|\|null===r)return r;var e=r[Symbol.toPrimitive];if(void 0===e)return("string"===t?String:Number)(r);e=e.call(r,t\|\|"default");if("object"!=typeof e)return e;throw new TypeError("@@toPrimitive must return a primitive value.")}`	51
`function(){return this.a}`	49
`function(e,r){if(null==e)return{};var t,n=function(e,r){if(null==e)return{};for(var t,n={},l=Object.keys(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|(n[t]=e[t]);return n}(e,r);if(Object.getOwnPropertySymbols)for(var l=Object.getOwnPropertySymbols(e),o=0;o<l.length;o++)t=l[o],0<=r.indexOf(t)\|\|Object.prototype.propertyIsEnumerable.call(e,t)&&(n[t]=e[t]);return n}`	48
`function(r,e,t){return i=function(r,e){if("object"!=typeof r\|\|null===r)return r;var t=r[Symbol.toPrimitive];if(void 0===t)return String(r);t=t.call(r,e);if("object"!=typeof t)return t;throw new TypeError("@@toPrimitive must return a primitive value.")}(e,"string"),(e="symbol"==typeof i?i:String(i))in r?Object.defineProperty(r,e,{value:t,enumerable:!0,configurable:!0,writable:!0}):r[e]=t,r;var i}`	48
`function(t){t=function(t,r){if("object"!=typeof t\|\|null===t)return t;var i=t[Symbol.toPrimitive];if(void 0===i)return String(t);i=i.call(t,r);if("object"!=typeof i)return i;throw new TypeError("@@toPrimitive must return a primitive value.")}(t,"string");return"symbol"==typeof t?t:String(t)}`	48
`function(i){T6(this,i)}`	48
`()=>{R3(),G3()}`	46
`function(){throw x0(new w7)}`	46
`function(e,r){return r=r\|\|e.slice(0),Object.freeze(Object.defineProperties(e,{raw:{value:Object.freeze(r)}}))}`	45
`function(n,r){(null==r\|\|r>n.length)&&(r=n.length);for(var e=0,l=new Array(r);e<r;e++)l[e]=n[e];return l}`	41
`function(n){throw x0(new w7)}`	39
`function(r){if(Array.isArray(r))return r}`	36
`function(){throw new TypeError("Invalid attempt to destructure non-iterable instance.\\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.")}`	36
`function(){return this.b}`	33
`function(n){return n2(n)}`	30
`function(n){return J1($5,p1,2,n,6,1)}`	30
`function(t,e){var r,n=Object.keys(t);return Object.getOwnPropertySymbols&&(r=Object.getOwnPropertySymbols(t),e&&(r=r.filter(function(e){return Object.getOwnPropertyDescriptor(t,e).enumerable})),n.push.apply(n,r)),n}`	29
`function(l,r){var t=null==l?null:"undefined"!=typeof Symbol&&l[Symbol.iterator]\|\|l["@@iterator"];if(null!=t){var e,n,u,f,i=[],a=!0,o=!1;try{if(u=(t=t.call(l)).next,0===r){if(Object(t)!==t)return;a=!1}else for(;!(a=(e=u.call(t)).done)&&(i.push(e.value),i.length!==r);a=!0);}catch(l){o=!0,n=l}finally{try{if(!a&&null!=t.return&&(f=t.return(),Object(f)!==f))return}finally{if(o)throw n}}return i}}`	29
`function(n){}`	29
`function(){return null}`	28
`function(e){return Object.getOwnPropertyDescriptor(Z,e).enumerable}`	25
`function(e){Object.defineProperty(Z,e,Object.getOwnPropertyDescriptor(W,e))}`	25
`function(n){return n}`	23
`function(i){C6(this,i)}`	23
`()=>{G3()}`	22
`function(){throw x0(new C7(n7((y7(),KG))))}`	21
`function(){return this}`	19
`function(i,t){this.b=i,this.a=t}`	19
`function(n){return!1}`	18
`function(n,w){throw x0(new w7)}`	18
`function(){return 0}`	17
`function(n){return typeof n}`	17
`function(o){return o&&"function"==typeof Symbol&&o.constructor===Symbol&&o!==Symbol.prototype?"symbol":typeof o}`	17
`()=>{R3()}`	16
`()=>{var e;return null!=(e=Z.options.debugAll)?e:Z.options.debugHeaders}`	16
`function(n,r){return n}`	15
`function(c){hX.call(this,c)}`	15
`function(){return this.c}`	15
`function(){return this.d}`	15
`function(e,n,r){return n in e?Object.defineProperty(e,n,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[n]=r,e}`	14
`function(r){if("undefined"!=typeof Symbol&&null!=r[Symbol.iterator]\|\|null!=r["@@iterator"])return Array.from(r)}`	14
`function(){throw new TypeError("Invalid attempt to spread non-iterable instance.\\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.")}`	14
`d=>d.id`	14
`()=>W(!1)`	13
`a=>a`	12
`function(t){var l=-1,n=null==t?0:t.length;for(this.clear();++l<n;){var r=t[l];this.set(r[0],r[1])}}`	12
`function(n,c){}`	12
`function(n,r){return r}`	11
`function(n,a){n.a=a}`	11
`function(){_V.call(this)}`	11
`function(i,t,h){this.a=i,this.b=t,this.c=h}`	11
`function(){return this.a.gc()}`	11
`function(i){this.b=i}`	10
`function(c){this.c=c}`	10
`function(){return this.f}`	10

Interestingly enough, all three tools handled the job bad in different aspects:

vite was the slowest and produced second biggest bundle
esbuild was the fastest and produced the largest bundle
bun was slower than esbuild by a split of hair, but produced smallest bundle with least duplicates

Bonus points to bun for installing node modules in a link of an eye.

In terms of duplicates, however, all three failed miserably (in my opinion), with the best result being the bundle produced by bun with 18% duplicates and the rest having almost half the bundle wasted.

For the most part, bundlers seem to be doing a pretty bad job at tree shaking and keep a lot of those utility functions' duplicates. One can estimate how much of a wasted space these use, by multiplying the function code length by the number of duplicates minus one (for one definition).

Let's imagine some of the above functions could be de-duplicated. What are the benefit of that? For the most part, the front-end can load faster for users - simply because there is less bytes to transfer. On top of that, there are less functions to be created in memory. So technically , the front-end can act faster. Although, on the modern machines the difference between having one function and few thousand of the same function is negligible.

Here is a shortened list of top abusers from different bundlers for our tool:

Function	vite		esbuild		bun
Function	#	Bytes	#	Bytes	#	Bytes
`function(){}`	753	9036	844	10128	739	8868
`function(){return!0}`	92	1840	93	1860	91	1820
`function(){return!1}`	78	1560	79	1580	76	1520
`function(){return null}`	27	621	29	667	28	644
`function(){return this}`	20	460	24	552	19	437
`function(){return 0}`	17	340	n/a	0	n/a	0
`function(n){return!1}`	13	273	n/a	0	18	378
`function(n){}`	29	377	31	403	29	377
`function(n){return n}`	15	315	20	420	23	483
`function(n){return typeof n}`	n/a	0	64	1792	17	476
`function(n){return this===n}`	n/a	0	119	3332	119	3332
`function(n,r){return n}`	14	322	n/a	0	15	345
`function(n,r){return r}`	11	253	n/a	0	11	253
`function(n,c){}`	n/a	0	n/a	0	12	180
`()=>{}`	27	162	129	774	141	846
`a=>a`	12	48	n/a	0	12	48
`a=>a()`	12	72	n/a	0	n/a	0
`function(r){ for(var t=1;t<arguments.length;t++){ var n,o=arguments[t]; for(n in o) Object.prototype.hasOwnProperty.call(o,n)&&(r[n]=o[n]) } return r }`	2205	317520	2216	319104	197	28368
`function n(){ return Object.assign&&Object.assign.bind(),n.apply(this,arguments) }`	1197	95760	1204	96320	n/a	0
`function n(){return Object.assign,n.apply(this,arguments)}`	1008	58464	1010	58580	n/a	0
`function(n,r){(null==r\|\|r>n.length)&&(r=n.length); for(var e=0,l=new Array(r);e<r;e++)l[e]=n[e]; return l}`	93	9672	106	11024	n/a	0
`function(r){if(Array.isArray(r))return r}`	77	3157	83	3403	36	1476
`function(){ throw new TypeError('Invalid attempt to destructure non-iterable instance.\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.')}`	77	13244	83	14276	36	6192
`function(t){ return t&&"object"==typeof t&&"default"in t?t:{default:t} }`	76	5624	260	19240	n/a	0
`function(_,o){_.__proto__=o}`	14	392	n/a	0	n/a	0
`function(o,r){ for(var t in r) Object.prototype.hasOwnProperty.call(r,t)&&(o[t]=r[t]) }`	14	1176	n/a	0	n/a	0
`function(e){return e&&e.__esModule?e:{default:e}}`	11	539	21	1029	n/a	0
`function(e,n,r){ return n in e?Object.defineProperty(e,n,{ value:r,enumerable:!0,configurable:!0,writable:!0 }):e[n]=r,e }`	n/a	0	115	13570	n/a	0
`function(r,n){ return Array.isArray(r)?r.concat(n):"string"==typeof r?r:void 0 }`	n/a	0	19	1520	n/a	0
`function(n){return null!=n&&n instanceof Array}`	n/a	0	21	987	n/a	0
`function(n,e){ if(null==n) return{}; for(var r,t={},f=Object.keys(n),u=0;u<f.length;u++) r=f[u],0<=e.indexOf(r)\|\|(t[r]=n[r]); return t }`	n/a	0	n/a	0	53	6890
`function(r,t){ if("object"!=typeof r\|\|null===r)return r; var e=r[Symbol.toPrimitive]; if(void 0===e)return("string"===t?String:Number)(r); e=e.call(r,t\|\|"default"); if("object"!=typeof e)return e; throw new TypeError("@@toPrimitive must return a primitive value.") }`	n/a	0	137	36853	51	13719
`function(e,r){ return r=r\|\|e.slice(0), Object.freeze(Object.defineProperties(e,{raw:{value:Object.freeze(r)}})) }`	n/a	0	160	17600	n/a	0
`function(o){ return o&&"function"==typeof Symbol&& o.constructor===Symbol&&o!==Symbol.prototype?"symbol":typeof o }`	n/a	0	64	7424	n/a	0
`function(r){ if("undefined"!=typeof Symbol&& null!=r[Symbol.iterator]\|\|null!=r["@@iterator"]) return Array.from(r) }`	n/a	0	n/a	0	14	1624

Interestingly enough, aside from a lot of () => {} and (a, b) => a and () => true (as I call them, utility) functions, there are a lot of ES6 / TypeScript helpers such as class definition and spread operator variants, presumingly made to be compatible with ES5-only browsers. Maybe if we had targeted only platforms supporting the latest ES features we would get better results?

Well, not quite much:

bundle sizes:

Bundler	Bundle size
bun	5.4M
esbuild	9.2M
esbuild (tuned)	8.0M
vite	7.1M
vite (tuned)	3.8M
webpack	4.4M

vite :

Found 15983 functions, 9581 are unique (59.94%)
Duplicates length: 1495985 bytes out of 4019326 bytes are duplicate code (37.22%)

esbuild :

Found 41736 functions, 29224 are unique (70.02%)
Duplicates length: 3406606 bytes out of 8347230 bytes are duplicate code (40.81%)

webpack is (should be) already using the target config option from tsconfig.json (which is set to ESNext in our case) and bun does not really have a whole lot of customization in this regard.

One unlikely possibility when these duplicates can be faster than having just one function is that running the nearly-defined code (in terms of a single block of code) might be slightly faster than making code jumps. This is super far-fetched idea from low-level programming, when CPU does not have to jump thousands or millions of (assembly) instructions back and forth but literally a few instead. This won't justify using verbose ES5-compatible code on ESnext browser, however. How about we run a very synthetic benchmark to check just this one theory?

function f(){return!1}

console.time('1');
for(var i=0;i<100000;i++){var a=[];for(var t=0;t<20000;t++)a.push(Math.random()*1000000);var x=a.filter(f).length}
console.timeEnd('1');

console.time('2');
for(var i=0;i<100000;i++){var a=[];for(var t=0;t<20000;t++)a.push(Math.random()*1000000);var f=function(){return!1},x=a.filter(f).length}
console.timeEnd('2');

The results are actually quite stable:

1: 17029ms
1: 16998ms
1: 16903ms

2: 21877ms
2: 21811ms
2: 21821ms

Having just one function instance is approx. 23% faster. But what happens at consequitive runs?

1: 9194ms
1: 9159ms
1: 14044ms
1: 13882ms
1: 13975ms
1: 9205ms
1: 14026ms

2: 21821ms
2: 13843ms
2: 13866ms
2: 13854ms
2: 13961ms

2: 21718ms
2: 13952ms
2: 13925ms
2: 13923ms

Seems like CPU does indeed do a little bit of instruction caching and branch prediction (first run is visibly slower than the subsequent runs). But the observation still holds: having one function definition instead of many copies (even "near" copies) has a much bigger impact.

With that being said, there is one interesting thing to try here: what if we actually replace some of those bulky duplicates with one-time declarations, within the same bundle?

Without performing an in-depth code analysis and optimization, I came up with the following naive implementation:

analyze the code and pick a few functions (biggest abusers) to fix them up
extract all function names, function parameter names (including potential destructuring objects in function params) and all variable and constant names
for each function to be de-duplicated, create a unique name (using the variable and function parameter and function names to avoid any clashes)
from the end of the file to the beginning, remove all occurrences of the function declaration and replace all function references with the unique name
add the function declarations to the beginning of the file

This approach is rather naive, since it does not account for a number of edge cases. For instance, if there are two functions to be replaced:

function(e){for(var r=1;r<arguments.length;r++){var t,l=arguments[r];for(t in l)Object.prototype.hasOwnProperty.call(l,t)&&(e[t]=l[t])}return e}

function p(){return(p=Object.assign||function(e){for(var r=1;r<arguments.length;r++){var t,l=arguments[r];for(t in l)Object.prototype.hasOwnProperty.call(l,t)&&(e[t]=l[t])}return e}).apply(this,arguments)}

e.g. one includes the other, the algorithm could have potentially evicted cases like this.

Before proceeding further, it is a good idea to test if the cleaned up bundle can safely replace the original one. Hence I just stuffed it in the static assets folder of our project and ran it with the modified bundle.

This way I figured few issues with the naive approach:

two function definitions are causing stack overflow:
- $z=function n(){return Object.assign,n.apply(this,arguments)}
- $q=function n(){return n=Object.assign&&Object.assign.bind(),n.apply(this,arguments)}
some of the empty functions are actually used as constructors (ES5-compatible OOP model) which is only discovered by finding the expressions like $FnName.prototype.something = somethingElse;
some functions are named and then referenced later in the code
some functions are not used at all:

For the shorthand functions I first tried manually fixing them up - had to replace them with $z=function(){return $z=Object.assign.bind(),$z.apply(this,arguments)} alikes. This worked, so I created an AST transformer to handle these one-line return-only functions:

const simplifyFunction = (code, fname) => {
    const tmpFilename = '_tmp';

    fs.writeFileSync(tmpFilename, code, 'utf-8');

    const root = ts.createSourceFile(
        tmpFilename,
        code,
        ts.ScriptTarget.ESNext,
        /* setParentNodes */ true
    );

    let rootFnName = undefined;

    const parse = (node) => {
        if (ts.isFunctionDeclaration(node) && ts.isIdentifier(node.name) && node.name.escapedText !== '') {
            rootFnName = node.name.escapedText;
            return;
        }

        ts.forEachChild(node, child => {
            if (child) {
                parse(child);
            }
        });
    };

    parse(root);

    if (!rootFnName) {
        fs.rmSync(tmpFilename);
        return code;
    }

    const transformer = (ctx) => (sourceFile) => {
        const visit = (node) => {
            if (ts.isIdentifier(node) && node.escapedText === rootFnName) {
                return ts.factory.createIdentifier(fname);
            }

            if (
                ts.isFunctionDeclaration(node) &&
                ts.isBlock(node.body) &&
                node.body.statements.length === 1 &&
                ts.isReturnStatement(node.body.statements[0]) &&
                ts.isBinaryExpression(node.body.statements[0].expression)
            ) {
                const next = ts.factory.createFunctionDeclaration(
                    [],
                    undefined,
                    undefined,
                    [],
                    [],
                    undefined,

                    ts.factory.createBlock([
                        ts.factory.createReturnStatement(
                            ts.factory.createComma(
                                ts.factory.createAssignment(
                                    ts.factory.createIdentifier(fname),
                                    node.body.statements[0].expression.left
                                ),

                                node.body.statements[0].expression.right
                            )
                        )
                    ])
                );

                return ts.visitEachChild(next, visit, ctx);
            }

            return ts.visitEachChild(node, visit, ctx);
        };

        return ts.visitNode(sourceFile, visit);
    };

    const s = ts.createSourceFile(tmpFilename, code, ts.ScriptTarget.ESNext);
    const { transformed } = ts.transform(s, [ transformer ]);

    const newCode = ts.createPrinter({ omitTrailingSemicolon: true })
        .printFile(transformed.find(({ fileName }) => fileName === tmpFilename));

    fs.rmSync(tmpFilename);

    return newCode;
};

The transformer is essentially a two-pass processor: it first parses the code and identifies the first function declaration. If none was found - it just returns the original code. If there was a so-called "root function" defined, it then replaces all identifier with that "root function" name with the alias provided as fname . It also replaces the return statements in form of a return something && something() with return alias = something, alias() .

This approach is different from simply using uglifyjs to just try and minimize the code - it is way more complex (compared to just one function call). It results in few extra whitespaces being added. But using uglifyjs messes things up again and TypeScript compiler does not have an option for minimizing the output. But few extra whitespaces are totally acceptable given the much bigger savings from this transformation.

With the constructors I simply excluded them from being de-duplicated. This resulted in 155 fewer substitutions (in Vite mode), which is negligible on the overall scale of the problem.

As for the named functions which are in the global scope and are referenced later down the line, I had to create a list of "backwards-compatible aliases", mapping those old function names onto the new unique names.

I ended up with a three-pass parser-transformer utility (not really "simple script" anymore). The passes being:

figuring out the duplicate function declarations and replacing them from the end to the start of the code (to minimize the chance of writing over the just-changed code)
removing the duplicate global-scope named functions and replacing them with shorthand aliases to the de-duplicated declarations
replace the usages of all known aliases and shorthand-named de-duplicated functions with their corresponding new (generated) names

Other than those, replacing the original bundle with the optimized one worked like a charm!

The results? With the threshold of 20 duplicates or more:

Bundler	Before optimization
Bundler	Bundle size	Total functions	Unique functions	Unique functions, %	Duplicate code, %
bun	6.2M	8903	7443	83.6%	0.78%
esbuild	8.7M	13057	10250	78.5%	3.9%
vite	3.9M	3502	2365	67.53%	6.39%
webpack	4.4M	2898	1434	49.48%	6.91%
After optimization
bun	6.2M (same)	7865 (-1038)	7355 (-88)	93.52% (+9.92%)	0.51% (-0.27%)
esbuild	8.5M (-0.2M)	3265 (-9792)	2990 (-7260)	91.58% (+13.08%)	0.62% (-3.28%)
vite	3.6M (-0.3M)	2483 (-1019)	2277 (-88)	91.7% (+24.17%)	1.68% (-4.71%)
webpack	4.1M (-0.3M)	1484 (-1414)	1375 (-59)	92.65% (+43.17%)	0.43% (-6.48%)

In conclusion, the bundlers do a pretty average job at optimizing the bundles, even in production mode with some extra tuning. And if some brave soul is willing to invest even more time and effort than I did into developing a sophisticated solution (potentially improving the existing tools, like uglifyjs or bundlers themselves), the numbers can be improved even further. It would be really interesting to see what would the results be running this optimizer on a bigger bundle.

In my humble opinion, Bun does produce the cleanest bundle. It might be not the smallest one, but it has least unnecessary stuff. On top of that, it is the fastest tool in JS world I have ever used. By the way, this very blog is built using Bun and React SSR - on Github Actions it takes around a minute to build and publish:

TypeScript classes are not what you think

2023-06-05T00:00:00Z

There was a talk at CPPCon 2017 by Matt Goldbolt with an inspirational title, What has my compiler done for me lately? . It actually resonates with me quite a bit, especially in the world where we try to push so hard for statically typed compiled languages on front-end.

Recently I recalled one of (many) aspects why I think TypeScript is useless (as a way to introduce strong type system to JavaScript world) in many cases.

To be realistic and not just throw bare accusations around, this was inspired by the work on contracts for a new microservice, specifically - the request & response types to be used in both client and the back-end of the microservice. The types were similar to the types used on a database layer and one of the developers just returned objects of that DB layer model type on a controller (endpoint) level, which confused me.

Consider the code below:

class A {
    constructor(public moo: string, public foo: number) {}
}

class B {
    constructor(public moo: string, public foo: number, public zoo?: string[]) {}
}

const b: B = new B('ololo', -3.14, ['1']);
const a: A = b;
const c: B = a;

This works because of a (rather questionable) design decision by TypeScript team called Type compatibility , where any two classes or interfaces that have overlapping public fields are deemed compatible and can be mutually interplaceable.

However, in most languages with reasonable type system, you would expect two different classes to be just that - two different classes - in the case of a code above, an object of class B can never be assigned to a variable of type A and vice versa.

There is a way to achieve this in TypeScript, however (a bit cumbersome, though): by hiding the properties and only exposing them through non-getter/non-setter methods, when needed:

class A {
    constructor(private moo: string, private foo: number) {}

    getMoo() {
        return this.moo;
    }

    getFoo() {
        return this.foo;
    }
}

class B {
    constructor(private moo: string, private foo: number, private zoo?: ReadonlyArray<string>) {}

    getMoo() {
        return this.moo;
    }

    getFoo() {
        return this.foo;
    }

    getZoo() {
        return this.zoo;
    }
}

const b: B = new B('ololo', -3.14, ['1']);
const a: A = b; // Type 'B' is not assignable to type 'A'. Types have separate declarations of a private property 'moo'.(2322)
const c: B = a; // Property 'getZoo' is missing in type 'A' but required in type 'B'.(2741)

Without knowing this "feature" beforehand, one might end up with an inconsistent code or errors down the line (when somebody decides to modify the DB layer model and gets errors on an API layer). And TypeScript does not really help here.

How not to use monads

2023-01-31T00:00:00Z

Recently I have watched a video titled "Optionals and errors in Haskell & Rust - monads by example" . In the video, the author makes a simple application simulating the retrieval of a HTML document via URL and checking if the document contains the title. This is a half-simulated experience, aimed at the beginners to demonstrate how the error handling is done in Rust and Haskell.

However, author in his code has made quite a few bad decisions. And that's what I want to make a focus of this blog: how one should not use monads and handle errors in Haskell.

Author's original source looks like this:

{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE MultiWayIf #-}
{-# LANGUAGE OverloadedRecordDot #-}
{-# LANGUAGE OverloadedRecordUpdate #-}

import Control.Monad (forM_)
import Data.List (isInfixOf)
import Data.Maybe (fromMaybe)
import Text.Printf (printf)

data Error = Error (String) deriving (Show)

data Doc = Doc {head :: Maybe Head}

data Head = Head {title :: Maybe String}

data Summary = Summary
  { title :: Maybe String,
    ok :: Bool
  }
  deriving (Show)

readDoc :: String -> Either Error Doc
readDoc url =
  if
      | isInfixOf "fail" url -> Left $ Error $ printf "Bad read of %s" url
      | otherwise ->
          Right $
            if
                | isInfixOf "head-missing" url -> Doc {head = Nothing}
                | isInfixOf "title-missing" url ->
                    Doc {head = Just Head {title = Nothing}}
                | isInfixOf "title-empty" url ->
                    Doc {head = Just Head {title = Just ""}}
                | otherwise ->
                    Doc
                      { head =
                          Just Head {title = Just $ printf "Title of %s" url}
                      }

buildSummary :: Doc -> Summary
buildSummary doc =
  Summary {title = doc.head >>= (.title), ok = True}

readAndBuildSummary :: String -> Summary
readAndBuildSummary url = case readDoc url of
  Left err -> Summary {title = Nothing, ok = False}
  Right doc -> buildSummary doc

isTitleNonEmpty :: Doc -> Maybe Bool
isTitleNonEmpty doc = do
  head <- doc.head
  title <- head.title
  return $ not $ null title

isTitleNonEmpty' :: Doc -> Maybe Bool
isTitleNonEmpty' doc = not <$> null <$> (doc.head >>= (.title))

isTitleNonEmpty'' :: Doc -> Maybe Bool
isTitleNonEmpty'' doc = not . null <$> (doc.head >>= (.title))

readWhetherTitleNonEmpty :: String -> Either Error (Maybe Bool)
readWhetherTitleNonEmpty url = do
  doc <- readDoc url
  return $ isTitleNonEmpty doc

readWhetherTitleNonEmpty' :: String -> Either Error (Maybe Bool)
readWhetherTitleNonEmpty' url = isTitleNonEmpty <$> readDoc url

main :: IO ()
main = do
  let urls = ["good", "title-empty", "title-missing", "head-missing", "fail"]
  forM_ urls $ \url -> do
    putStrLn $ printf "Checking \"https://%s/\":" url
    -- Summary.
    let summary = readAndBuildSummary url
    putStrLn $ printf "  Summary: %s" $ show summary
    putStrLn $ printf "  Title: %s" $ fromMaybe "" summary.title
    -- Has title.
    let hasTitle = readWhetherTitleNonEmpty url
    -- let hasTitleSure = either (const False) id $ fromMaybe False <$> hasTitle
    let hasTitleSure = fromMaybe False $ either (const Nothing) id hasTitle
    putStrLn $
      printf "  Has title: %s vs %s" (show hasTitle) (show hasTitleSure)

He then proceeded to wrapping this code in IO monad and ended up with the following:

{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE MultiWayIf #-}
{-# LANGUAGE OverloadedRecordDot #-}
{-# LANGUAGE OverloadedRecordUpdate #-}

import Control.Monad (forM_)
import Data.Functor ((<&>))
import Data.List (isInfixOf)
import Data.Maybe (fromMaybe)
import Text.Printf (printf)

data Error = Error (String) deriving (Show)

data Doc = Doc {head :: Maybe Head}

data Head = Head {title :: Maybe String}

data Summary = Summary
  { title :: Maybe String,
    ok :: Bool
  }
  deriving (Show)

-- readDoc

readDoc :: String -> IO (Either Error Doc)
readDoc url =
  return
    if
        | isInfixOf "fail" url -> Left $ Error $ printf "Bad read of %s" url
        | otherwise ->
            Right $
              if
                  | isInfixOf "head-missing" url -> Doc {head = Nothing}
                  | isInfixOf "title-missing" url ->
                      Doc {head = Just Head {title = Nothing}}
                  | isInfixOf "title-empty" url ->
                      Doc {head = Just Head {title = Just ""}}
                  | otherwise ->
                      Doc
                        { head =
                            Just Head {title = Just $ printf "Title of %s" url}
                        }

-- buildSummary

buildSummary :: Doc -> Summary
buildSummary doc =
  Summary {title = doc.head >>= (.title), ok = True}

-- readAndBuildSummary

readAndBuildSummary :: String -> IO Summary
readAndBuildSummary url = do
  docOrError <- readDoc url
  return $ case docOrError of
    Left err -> Summary {title = Nothing, ok = False}
    Right doc -> buildSummary doc

readAndBuildSummary' :: String -> IO Summary
readAndBuildSummary' url =
  readDoc url >>= \docOrError -> return $ case docOrError of
    Left err -> Summary {title = Nothing, ok = True}
    Right doc -> buildSummary doc

readAndBuildSummary'' :: String -> IO Summary
readAndBuildSummary'' url =
  readDoc url <&> \case
    Left err -> Summary {title = Nothing, ok = True}
    Right doc -> buildSummary doc

-- isTitleNonEmpty

isTitleNonEmpty :: Doc -> Maybe Bool
isTitleNonEmpty doc = do
  head <- doc.head
  title <- head.title
  return $ not $ null title

isTitleNonEmpty' :: Doc -> Maybe Bool
isTitleNonEmpty' doc = not <$> null <$> (doc.head >>= (.title))

isTitleNonEmpty'' :: Doc -> Maybe Bool
isTitleNonEmpty'' doc = not . null <$> (doc.head >>= (.title))

-- readWhetherTitleNonEmpty

readWhetherTitleNonEmpty :: String -> IO (Either Error (Maybe Bool))
readWhetherTitleNonEmpty url = do
  docOrError <- readDoc url
  return $ do
    doc <- docOrError
    return $ isTitleNonEmpty doc

readWhetherTitleNonEmpty' :: String -> IO (Either Error (Maybe Bool))
readWhetherTitleNonEmpty' url =
  readDoc url >>= \docOrError ->
    return $ docOrError >>= \doc -> return $ isTitleNonEmpty doc

readWhetherTitleNonEmpty'' :: String -> IO (Either Error (Maybe Bool))
readWhetherTitleNonEmpty'' url = (isTitleNonEmpty <$>) <$> readDoc url

readWhetherTitleNonEmpty''' :: String -> IO (Either Error (Maybe Bool))
readWhetherTitleNonEmpty''' url = readDoc url <&> (<&> isTitleNonEmpty)

-- main

main :: IO ()
main = do
  let urls = ["good", "title-empty", "title-missing", "head-missing", "fail"]
  forM_ urls $ \url -> do
    putStrLn $ printf "Checking \"https://%s/\":" url
    -- Summary.
    summary <- readAndBuildSummary url
    putStrLn $ printf "  Summary: %s" $ show summary
    putStrLn $ printf "  Title: %s" $ fromMaybe "" summary.title
    -- Has title.
    hasTitle <- readWhetherTitleNonEmpty url
    -- let hasTitleSure = either (const False) id $ fromMaybe False <$> hasTitle
    let hasTitleSure = fromMaybe False $ either (const Nothing) id hasTitle
    putStrLn $
      printf "  Has title: %s vs %s" (show hasTitle) (show hasTitleSure)

I see a few issues with this code:

the abuse of IO monad
the excessive (an I mean unnecessary) use of Maybe monad
too many unnecessary readDoc calls (bear in mind: it is supposed to be a network call + document parse, so quite a heavy function)

Improvements

My suggestions on how to make their code better:

only stick to IO where the function is actually interacting with the external world; in this case this would be either main or readDoc (when it actually makes a network call)
eliminate the abuse of Maybe Bool - in the functions where the Maybe Bool is used ( isTitleNonEmpty and its variations) just Bool would suffice
only call readDoc once and use the result down the pipeline, no need to call it multiple times ( readAndBuildSummary , readWhetherTitleNonEmpty )
reduce the nested if statements to guard expressions
replace the case eitherValue expressions with the either function
remove the dead / unused / unnecessary code (like those duplicated functions - readWhetherTitleNonEmpty' , readWhetherTitleNonEmpty'' and readWhetherTitleNonEmpty''' )

Use `IO` monad only for the functions which interact with the outside world

IO monad tells compiler and the people who read the code the function operates outside of the program internals. Like actually performing input-output, working with network, operating system, etc.

You can think of any such function as an unsafe part of an application.

In the example above, most of the functions do not comply with that - they are simply a side-effect of poor monad operations and could be easily changed as follows:

isTitleNonEmpty :: Doc -> Maybe Bool
isTitleNonEmpty doc = do
  head <- doc.head
  title <- head.title
  return $ not $ null title

buildSummary :: Doc -> Summary
buildSummary doc =
  Summary {title = doc.head >>= (.title), ok = True}

-- there is absolutely no need for this function
readWhetherTitleNonEmpty :: String -> IO (Either Error (Maybe Bool))
readWhetherTitleNonEmpty url = do
  docOrError <- readDoc url
  return $ do
    doc <- docOrError
    return $ isTitleNonEmpty doc

-- this is also redundant
readAndBuildSummary :: String -> IO Summary
readAndBuildSummary url = do
  docOrError <- readDoc url
  return $ case docOrError of
    Left err -> Summary {title = Nothing, ok = False}
    Right doc -> buildSummary doc

-- this is the (reduced) existing code
main__old :: IO ()
main__old = do
  let url = "good"

  putStrLn $ printf "Checking \"https://%s/\":" url

  summary <- readAndBuildSummary url

  putStrLn $ printf "  Summary: %s" $ show summary
  putStrLn $ printf "  Title: %s" $ fromMaybe "" summary.title

  hasTitle <- readWhetherTitleNonEmpty url

-- note how the document is only passed down to the functions which actually operate on the document
-- the only two functions using `IO` monad here are `main` and `readDoc`, as both perform input-output operations
main :: IO ()
main = do
  let url = "good"

  putStrLn $ printf "Checking \"https://%s/\":" url

  docOrError <- readDoc url

  let summary = case docOrError of
    Left err -> Summary {title = Nothing, ok = False}
    Right doc -> buildSummary doc

  putStrLn $ printf "  Summary: %s" $ show summary
  putStrLn $ printf "  Title: %s" $ fromMaybe "" summary.title

  let hasTitle = isTitleNonEmpty doc

Reduce calls to `IO` functions

Pure functions must return the exact same value for the same arguments, regardless of how many times they are called.

IO (and a lot of other monads, like Reader , Writer , etc.) work around this rule by changing the outside world. So even though a function technically returns the same value every time, the side-effect of an IO monad can potentially change something elsewhere, outside of the function itself.

This makes the programs non-pure and might cause a lot of negative impact. These side effects are what is thought to be the root of all evils in (functional) programming. They make it hard to figure out what a program actually does. They make the functions and programs hard to test and prove to be correct.

From the performance perspective, if a function actually does change the outside world, it can't be memoized and it won't be optimized by the compiler to only use its result, computed once.

Think calling a function which tries to create a user in a system or drop a database table. This would happen on every call. If you call such function twice, it will most likely fail (due to database failures, hopefully).

But what if the function creates data instead and there are no checks for duplicates in the external system? Like appending a log line. If such function was to be called multiple times, it can quickly bloat the storage. This is a potential security issue as well.

So instead of calling such functions (recap: functions returning IO monad), why not call them once and store the result? Then the entire program can operate on the stored result value.

Of course, there are plethora of cases where such behaviour (calling such functions multiple times) is actually desired. But in the sample code from above, this is not the case.

Check the transformed code from the previous point how this solution is utilized:

main :: IO ()
main = do
  let url = "good"

  putStrLn $ printf "Checking \"https://%s/\":" url

  -- store the result of fetching a document by its URL
  docOrError <- readDoc url

  -- work on the stored value instead of sending an extra HTTP request
  let summary = case docOrError of
    Left err -> Summary {title = Nothing, ok = False}
    Right doc -> buildSummary doc

  putStrLn $ printf "  Summary: %s" $ show summary
  putStrLn $ printf "  Title: %s" $ fromMaybe "" summary.title

  -- work on the stored value again, preventing one more HTTP request
  let hasTitle = case docOrError of
    Left err -> False
    Right doc -> isTitleNonEmpty doc

Eliminate the abuse of `Maybe Bool`

There's simply no need to go overboard with Maybe in methods which simply check whether something is present or not.

Maybe Bool introduces a tri-state:

Just False
Just True
Nothing

In cases where this is desired, is much better to represent each of the state with a semantic value:

type HeaderPresence = NoHeader | EmptyHeader | HasHeader

This way the code is much easier to follow and debug.

For the sample in question even this approach is redundant - the function simply checks whether the value (title) is present or not, so a simple Bool would suffice:

isTitleNonEmpty__old :: Doc -> Maybe Bool
isTitleNonEmpty__old doc = do
  head <- doc.head
  title <- head.title
  return $ not $ null title

isTitleNonEmpty :: Doc -> Bool
isTitleNonEmpty doc =
  not $ null title
  where
    title = doc.head doc >>= head.title

main__old :: IO ()
main__old = do
  let url = "good"

  docOrError <- readDoc url

  hasTitle <- case docOrError of
    Left err -> None
    Right doc -> isTitleNonEmpty__old doc

  let hasTitleSure = fromMaybe False $ either (const Nothing) id hasTitle

  putStrLn $
    printf "  Has title: %s vs %s" (show hasTitle) (show hasTitleSure)

main :: IO ()
main = do
  let url = "good"

  putStrLn $ printf "Checking \"https://%s/\":" url

  docOrError <- readDoc url

  -- a simple True | False value, no need to mess with Maybe
  let hasTitle = case docOrError of
    Left err -> False
    Right doc -> isTitleNonEmpty doc

  -- note: no need to wrap & unwrap Maybe and Either
  putStrLn $ printf "  Has title: %s" (show hasTitle)

Reduce the nested `if` statements - use guard expressions

The original readDoc function uses a rather unnecessarily bulky if statement:

readDoc :: String -> IO (Either Error Doc)
readDoc url =
  return
    if
        | isInfixOf "fail" url -> Left $ Error $ printf "Bad read of %s" url
        | otherwise ->
            Right $
              if
                  | isInfixOf "head-missing" url -> Doc {head = Nothing}
                  | isInfixOf "title-missing" url ->
                      Doc {head = Just Head {title = Nothing}}
                  | isInfixOf "title-empty" url ->
                      Doc {head = Just Head {title = Just ""}}
                  | otherwise ->
                      Doc
                        { head =
                            Just Head {title = Just $ printf "Title of %s" url}
                        }

Haskell provides a built-in language feature for just this case, which really does make code cleaner a lot:

readDoc :: String -> IO (Either Error Doc)

readDoc url
  | isInfixOf "fail" url = return $ Left $ Error $ printf "Bad read of %s" url
  | isInfixOf "head-missing" url = return $ Right Doc { head = Nothing }
  | isInfixOf "title-missing" url = return $ Right Doc { head = Just Head { title = Nothing } }
  | isInfixOf "title-empty" url = return $ Right Doc { head = Just Head { title = Just "" } }
  | otherwise = return $ Right Doc { head = Just Head { title = Just $ printf "Title of %s" url } }

Use existing functions on monads instead of `case` statements

In my experience, for the most part, you want to simply apply a function to a Maybe or an Either .

And, again, for the most part, you either ignore one of the states of the monad (be it Nothing or Left ), or apply a different function to it.

There are handy helper functions for these situations which do make the code cleaner - fromMaybe and either .

Consider few examples:

hasTitle__old <- case docOrError of
  Left err -> None
  Right doc -> isTitleNonEmpty__old doc

hasTitle <- either (const None) (isTitleNonEmpty__old)

let summary__old = case docOrError of
  Left err -> Summary {title = Nothing, ok = False}
  Right doc -> buildSummary doc

-- needs a nice helper function
invalid_summary = Summary { title = Nothing, ok = False }

let summary = either invalid_summary buildSummary

let hasTitle__old = case docOrError of
  Left err -> False
  Right doc -> isTitleNonEmpty doc

let hasTitle = either (const False) isTitleNonEmpty

End result

Here's what the code looks like when applying all of the above suggestions and tidying up the code:

{-# LANGUAGE DuplicateRecordFields #-}

import Control.Monad (forM_)
import Data.List (isInfixOf)
import Data.Maybe (fromMaybe)
import Text.Printf (printf)

newtype Error = Error String deriving (Show)

newtype Doc = Doc {_head :: Maybe Head}

newtype Head = Head {title :: Maybe String}

data Summary = Summary
  { title :: Maybe String,
    ok :: Bool
  }
  deriving (Show)

-- helpers for Repl.It is meh with language extensions

docHead :: Doc -> Maybe Head
docHead = _head

headTitle :: Head -> Maybe String
headTitle = title

summaryTitle :: Summary -> Maybe String
summaryTitle = title

-- implementations

readDoc :: String -> IO (Either Error Doc)
readDoc url
  | "fail" `isInfixOf` url = return $ Left $ Error $ printf "Bad read of %s" url
  | "head-missing" `isInfixOf` url = return $ Right Doc {_head = Nothing}
  | "title-missing" `isInfixOf` url = return $ Right Doc {_head = Just Head {title = Nothing}}
  | "title-empty" `isInfixOf` url = return $ Right Doc {_head = Just Head {title = Just ""}}
  | otherwise = return $ Right Doc {_head = Just Head {title = Just $ printf "Title of %s" url}}

buildSummary :: Doc -> Summary
buildSummary doc =
  Summary {title = docHead doc >>= headTitle, ok = True}

invalidSummary :: Summary
invalidSummary = Summary {title = Nothing, ok = False}

readAndBuildSummary :: Either Error Doc -> Summary
readAndBuildSummary = either invalidSummary buildSummary

readAndBuildTitle :: Summary -> String
readAndBuildTitle summary = fromMaybe "" (summaryTitle summary)

isTitleNonEmpty :: Doc -> Bool
isTitleNonEmpty doc =
  not $ null title
  where
    title = docHead doc >>= headTitle

readWhetherTitleNonEmpty :: Either Error Doc -> Bool
readWhetherTitleNonEmpty = either (const False) isTitleNonEmpty

-- main

main :: IO ()
main = do
  let urls = ["good", "title-empty", "title-missing", "head-missing", "fail"]

  forM_ urls $ \url -> do
    putStrLn $ printf "Checking \"https://%s/\":" url

    docOrError <- readDoc url

    -- Summary
    let summary = readAndBuildSummary docOrError
    let title = readAndBuildTitle summary

    putStrLn $ printf "  Summary: %s" $ show summary
    putStrLn $ printf "  Title: %s" $ title

    -- Has title
    let hasTitle = readWhetherTitleNonEmpty docOrError

    putStrLn $ printf "  Has title: %s" (show hasTitle)

Floyd-Warshall algorithm, revised

2023-01-12T00:00:00Z

In this blog I would like to brush upon Floyd-Warshall algorithm implementation I have described previously .

See, generally, when explaining Floyd-Warshall algorithm, the graph is given as an adjacency matrix - an V x V matrix, where V is the number of vertices (nodes) in a graph and each matrix value representing the distance between each vertex. For instance, the value G[1][3] = 12 represents the edge from vertex 1 to vertex 3 with the value of 12 .

Aside from adjacency matrix, there are two other common ways to represent a graph:

incidence matrix of size V x E for V vertices and E edges, where each row represents an edge value ( G[i][j] > 0 for edge i -> j , G[i][j] < 0 for edge j -> i , G[i][j] = 0 for no edge between vertices i and j )
adjacency list, where each vertex is represented as an object with a list of its connections

Consider the graph from the previous blog:

It can be represented in three different ways:

adjacency list:

v0: [ [2 -> 1] ]
v1: [ [2 -> 6], [0 -> 7] ]
v2: [ [3 -> 5] ]
v3: [ [1 -> 2] ]

adjacency matrix:

   |  v0  v1  v2  v3
---+------------------
v0 |   0   0   1   0
v1 |   7   0   6   0
v2 |   0   0   0   5
v3 |   0   2   0   0

incidence matrix:

   |  e0  e1  e2  e3  e4
---+---------------------
v0 |   0   0   0   7   0
v1 |   0   0   2   0   0
v2 |   1   0   0   0   6
v3 |   0   5   0   0   0

However, arguably the most common and most memory-efficient way to represent graph in a real-world application would be the adjacency list.

Hence I thought a bit of a revised implementation is needed, since the previous one (aka "classic" implementation) relies on matrices. And those matrices contain quite a bit of unused data. Take the graph above as an example: it has 4 vertices and 5 edges. An adjacency matrix would use 16 numbers, incidence matrix would use 20 numbers and adjacency list would use 5 tuples of 2 numbers ( 10 numbers, so to speak).

"Classic" Floyd-Warshall algorithm implementation constructs an additional matrix for the shortest paths' distances (at most it can reuse the existing one representing the graph itself) and an additional V x V matrix for path reconstruction.

With a graph represented as an adjacency list (or just a list of node objects), the algorithm can use something like a dictionary (or a hashmap) to reduce the memory consumption. Subjectively, it also becomes easier to read and understand the algorithm.

First, we'd need few definitions:

class Node
{
    public string Name;
    public Dictionary<Node, int> Connections;

    public Node(string name)
    {
        this.Name = name;
        this.Connections = new Dictionary<Node, int>();
    }

    public override string ToString()
    {
        return this.Name;
    }
}

class Graph
{
    public List<Node> Nodes;

    public Graph()
    {
        this.Nodes = new List<Node>();
    }
}

class Path
{
    public Node From;
    public Node To;
    public List<Node> Nodes;
    public int Distance;

    public Path(Node from, Node to)
    {
        this.Distance = 0;
        this.From = from;
        this.To = to;
        this.Nodes = new List<Node>();
    }
}

The Path class is a bit excessive, as it really only needs a list of Node objects, but I decided to make it verbose on purpose - to make it easier to debug.

The algorithm then becomes like this:

Dictionary<(Node from, Node to), Node> FindAllPaths(Graph g)
{
    var distance = new Dictionary<(Node from, Node to), int>();
    var pathThrough = new Dictionary<(Node from, Node to), Node>();

    foreach (var nodeFrom in g.Nodes)
    {
        foreach (var nodeTo in g.Nodes)
        {
            if (nodeFrom == nodeTo)
            {
                distance.Add((nodeFrom, nodeTo), 0);
            }
            else
            {
                distance.Add((nodeFrom, nodeTo), int.MaxValue);
            }
        }

        foreach (var nodeTo in nodeFrom.Connections.Keys)
        {
            distance[(nodeFrom, nodeTo)] = nodeFrom.Connections[nodeTo];
            pathThrough.Add((nodeFrom, nodeTo), nodeTo);
        }

        pathThrough.Add((nodeFrom, nodeFrom), nodeFrom);
    }

    foreach (var nodeThrough in g.Nodes)
    {
        foreach (var nodeFrom in g.Nodes)
        {
            foreach (var nodeTo in g.Nodes)
            {
                if (nodeFrom == nodeTo || distance[(nodeThrough, nodeTo)] == int.MaxValue || distance[(nodeFrom, nodeThrough)] == int.MaxValue)
                {
                    continue;
                }

                if (distance[(nodeFrom, nodeTo)] > distance[(nodeFrom, nodeThrough)] + distance[(nodeThrough, nodeTo)])
                {
                    distance[(nodeFrom, nodeTo)] = distance[(nodeFrom, nodeThrough)] + distance[(nodeThrough, nodeTo)];
                    pathThrough[(nodeFrom, nodeTo)] = pathThrough[(nodeFrom, nodeThrough)];
                }
            }
        }
    }

    return pathThrough;
}

One can easily see the separate parts of the algorithm - initializing the intermediate storage (for the paths and distances) and actually finding the shortest paths. It is pretty much the same algorithm as before with the only difference being the use of dictionary instead of a matrix.

Path reconstruction bit is also a little bit different (not too much though):

Path ReconstructPath(Node from, Node to, Dictionary<(Node from, Node to), Node> allPaths)
{
    if (!allPaths.ContainsKey((from, to)))
    {
        return null;
    }

    var path = new Path(from, to);

    var tempNode = from;

    path.Nodes.Add(tempNode);

    while (tempNode != to)
    {
        var nextNode = allPaths[(tempNode, to)];

        path.Distance += tempNode.Connections[nextNode];
        path.Nodes.Add(nextNode);

        tempNode = nextNode;
    }

    return path;
}

Now, there is one major improvement we can make to this algorithm. See, when we initialize the dictionaries for paths and distances, we follow the same logic as with the classic implementation, putting in the values for edges which do not exist in a graph:

foreach (var nodeFrom in g.Nodes)
{
    // this bit will add entries for inexistent edges
    foreach (var nodeTo in g.Nodes)
    {
        if (nodeFrom == nodeTo)
        {
            distance.Add((nodeFrom, nodeTo), 0);
        }
        else
        {
            distance.Add((nodeFrom, nodeTo), int.MaxValue);
        }
    }

    // ...
}

But as a matter of fact, we do not really have to store these values - instead, we could simply have a check whether the needed entry exists in the dictionary or not:

Dictionary<(Node from, Node to), Node> FindAllPaths(Graph g)
{
    var distance = new Dictionary<(Node from, Node to), int>();
    var pathThrough = new Dictionary<(Node from, Node to), Node>();

    foreach (var nodeFrom in g.Nodes)
    {
        // -- no more unnecessary entries --

        foreach (var nodeTo in nodeFrom.Connections.Keys)
        {
            distance.Add((nodeFrom, nodeTo), nodeFrom.Connections[nodeTo]);
            pathThrough.Add((nodeFrom, nodeTo), nodeTo);
        }

        pathThrough.Add((nodeFrom, nodeFrom), nodeFrom);
    }

    foreach (var nodeThrough in g.Nodes)
    {
        foreach (var nodeFrom in g.Nodes)
        {
            foreach (var nodeTo in g.Nodes)
            {
                if (nodeFrom == nodeTo)
                {
                    continue;
                }

                // ++ but one extra check here, to prevent unnecessary iterations ++
                if (!distance.ContainsKey((nodeFrom, nodeThrough)) || !distance.ContainsKey((nodeThrough, nodeTo)))
                {
                    continue;
                }

                // ++ and one extra check here, to annotate the existence of a new, indirect shortest path between nodeFrom and nodeTo ++
                if (!distance.ContainsKey((nodeFrom, nodeTo)) || distance[(nodeFrom, nodeTo)] > distance[(nodeFrom, nodeThrough)] + distance[(nodeThrough, nodeTo)])
                {
                    distance[(nodeFrom, nodeTo)] = distance[(nodeFrom, nodeThrough)] + distance[(nodeThrough, nodeTo)];
                    pathThrough[(nodeFrom, nodeTo)] = pathThrough[(nodeFrom, nodeThrough)];
                }
            }
        }
    }

    return pathThrough;
}

For that matter, we don't really need to have a separate dictionary for path reconstruction - since both distance and pathThrough dictionaries serve the same purpose (have the exact same key, but different value), we can smash those two together and use even less memory:

Dictionary<(Node from, Node to), (Node through, int distance)> FindAllPaths(Graph g)
{
    var pathThrough = new Dictionary<(Node from, Node to), (Node through, int distance)>();

    foreach (var nodeFrom in g.Nodes)
    {
        foreach (var nodeTo in nodeFrom.Connections.Keys)
        {
            pathThrough.Add((nodeFrom, nodeTo), (nodeTo, nodeFrom.Connections[nodeTo]));
        }

        pathThrough.Add((nodeFrom, nodeFrom), (nodeFrom, 0));
    }

    foreach (var nodeThrough in g.Nodes)
    {
        foreach (var nodeFrom in g.Nodes)
        {
            foreach (var nodeTo in g.Nodes)
            {
                if (nodeFrom == nodeTo)
                {
                    continue;
                }

                if (!pathThrough.ContainsKey((nodeFrom, nodeThrough)) || !pathThrough.ContainsKey((nodeThrough, nodeTo)))
                {
                    continue;
                }

                if (!pathThrough.ContainsKey((nodeFrom, nodeTo)) || pathThrough[(nodeFrom, nodeTo)].distance > pathThrough[(nodeFrom, nodeThrough)].distance + pathThrough[(nodeThrough, nodeTo)].distance)
                {
                    pathThrough[(nodeFrom, nodeTo)] = (nodeThrough, pathThrough[(nodeFrom, nodeThrough)].distance + pathThrough[(nodeThrough, nodeTo)].distance);
                }
            }
        }
    }

    return pathThrough;
}

Essentially, nothing special - just using the tuples for both the intermediate node and the distance for the shortest paths.

And with a tiny bit of a change, path reconstruction:

Path ReconstructPath(Node from, Node to, Dictionary<(Node from, Node to), (Node through, int distance)> allPaths)
{
    if (!allPaths.ContainsKey((from, to)))
    {
        return null;
    }

    var path = new Path(from, to);

    var tempNode = from;

    path.Nodes.Add(tempNode);

    while (tempNode != to)
    {
        var (nextNode, distance) = allPaths[(tempNode, to)];

        path.Distance += distance;
        path.Nodes.Add(nextNode);

        tempNode = nextNode;
    }

    return path;
}

Parser monad

2022-12-06T00:00:00Z

Few years ago I did a course project for the "Functional programming" course at Jagiellonian University. The project was to implement a program for solving systems of equations... in Haskell.

For me back then, the entire concept of monads was quite foreign and scary, so I implemented a very crude state machine to parse the equations as strings from STDIN:

parseRow :: String -> Integer -> [Integer] -> [String] -> ([Integer], [String])
parseRow [] state coefficients var_names
	| state == 7 = (reverse coefficients, reverse var_names)
	| otherwise = error ("Invalid equation (state: " ++ show state ++ "; coefficients: " ++ show coefficients ++ "; var_names: " ++ show var_names ++ ")")

parseRow (c:cs) state coefficients var_names
	| (state == 1) && (c == '-') = parseRow cs 2 ((-1) : coefficients) var_names
	| state == 1 && isNum c = parseRow cs 3 (c_int : coefficients) var_names
	| state == 1 && isAlpha c = parseRow cs 4 (1 : coefficients) ([c] : var_names)
	| state == 2 && isNum c = parseRow cs 3 (repl_k : drop 1 coefficients) var_names
	| state == 2 && isAlpha c = parseRow cs 4 coefficients ([c] : var_names)
	| (state == 3) && (c == '=') = parseRow cs 5 coefficients var_names
	| state == 3 && isAlpha c = parseRow cs 4 coefficients ([c] : var_names)
	| state == 3 && isNum c = parseRow cs 3 (new_k : drop 1 coefficients) var_names
	| (state == 4) && (c == '-') = parseRow cs 2 ((-1) : coefficients) var_names
	| (state == 4) && (c == '+') = parseRow cs 2 (1 : coefficients) var_names
	| state == 4 && isAlphaNum c = parseRow cs 4 coefficients (new_v : drop 1 var_names)
	| (state == 4) && (c == '=') = parseRow cs 5 coefficients var_names
	| (state == 5) && (c == '-') = parseRow cs 6 ((-1) : coefficients) var_names
	| state == 5 && isNum c = parseRow cs 7 (c_int : coefficients) var_names
	| state == 6 && isNum c = parseRow cs 7 (repl_k : drop 1 coefficients) var_names
	| state == 7 && isNum c = parseRow cs 7 (new_k : drop 1 coefficients) var_names
	| otherwise = parseRow cs state coefficients var_names
	where
		k = abs $ head coefficients
		k_sign = sign (head coefficients)
		c_int = atoi c
		new_k = k_sign * ((k * 10) + c_int)
		repl_k = k_sign * c_int
		v = if not (null var_names) then head var_names else []
		new_v = v ++ [c]

parseLine :: String -> ([Integer], [String])
parseLine s = parseRow s 1 [] []

However, recently I (hopefully) got a grip of monads, so I decided to modify my old project in Haskell to make it prettier.

One of the improvements I made was the parsing part. Since one of the requirements for the course project was to not use any third-party libraries, I decided to stick to this and implement a parser monad.

This proved to be an interesting task and made the entire parsing part quite neat:

equationFactor :: Parser Fraction
equationFactor = do
    _sign <- factorSign

    zeroOrMore (sat isSpace)

    factor <- fmap (fromMaybe (1%1)) (zeroOrOne rationalFactor)

    return (_sign * factor)

equationMember :: Parser (Fraction, String)
equationMember = do
    factor <- equationFactor

    zeroOrMore (sat isSpace)
    zeroOrOne (sat (== '*'))
    zeroOrMore (sat isSpace)

    nameFirst <- oneOrMore (sat isAlpha)
    nameRest <- zeroOrMore (sat isAlphaNum)

    zeroOrMore (sat isSpace)

    return (factor, nameFirst ++ nameRest)

-- An equation consists of a list of pairs (factor, variable name) and a free member
equation :: Parser ([(Fraction, String)], Fraction)
equation = do
    members <- oneOrMore equationMember

    zeroOrMore (sat isSpace)
    sat (== '=')
    zeroOrMore (sat isSpace)

    freeMember <- rationalFactor

    return (members, freeMember)

This is all possible thanks to the Parser monad. Under the cut - the implementation details.

This material is heavily based on the Dave Sands' lectures about parsing in Haskell.

The main idea is that a parser is an annotated function from an input string to maybe something (parsing result) and a string (the unparsed part):

newtype Parser a = P (String -> Maybe (a, String))

Here, P is just the type constructor.

So the function wrapped by the Parser monad takes a String as an input, parses as much as possible and returns what it managed to parse together with the rest of the string, which was not parser. The two possible results hence are Just (a, String) representing the parsed data ( a part of Maybe (a, String) ) and the unparsed remainder of a string ( String part of Maybe (a, String) ). The other possible result is Nothing , meaning the input string can not be parsed with a given parser function.

Based on this explanation, one can build combinations of the parsers using the chaining of the Maybe monad and the boolean logic ( AND and OR operations, predominantly).

Helper functions

There are few trivial parsers which to help on this endeavour:

success a - always succeeds, returning a Just (a, str) (some default value and an entire input string) for any input string
failure - always fails, returning Nothing for any input string
item - parses any one arbitrary character, returning this character wrapped in Just (chr, str) (first character and the rest of the string) if the input string is non-empty or Nothing if the input string is empty
parser1 <|> parser2 - tries the first parser, parser1 and if it succeeds - returns the result, but if it fails - returns whatever running the second parser, parser2 returns
parser1 >>= (\x -> parser2) - chains two parsers together, runs parser1 and passes its output string to the second parser, parser2 ; but since a parser result is a tuple, not just a string that can be passed to the second parser, this function takes a function from the output of the first parser to the second parser
parserFn2 <*> parser1 - chains two parsers together, runs parser1 (yes, in the reverse order) and applies parserFn2 to the result; where parserFn2 is a function wrapped in a parser ( Parser (a -> a) )

Since a parser is just a function String -> Maybe (a, String) , in order to actually parse something, one needs to run it. Running a parser is as easy as executing a function parse , passing it a parser and an input string:

>>> parse (success 1) "abc"
Just (1,"abc")

>>> parse (failure) "abc"
Nothing

>>> parse item "abc"
Just ('a',"bc")

>>> parse (item <|> success '!') "abc"
Just ('a',"bc")

>>> parse (item <|> success '!') ""
Just ('!',"")

>>> parse (item >>= (\_ -> item)) "abc"
Just ('b',"c")

>>> parse (item >>= (\_ -> item)) "abc"
Just ('b',"c")

>>> parse (success (\x -> if x == '-' then (-1) else 1) <*> item) "-123"
Just (-1,"123")

>>> parse (success (\x -> if x == '-' then (-1) else 1) <*> item) "123"
Just (1,"23")

The aforementioned helper functions are defined as following:

success :: a -> Parser a
success a = P (\str -> Just (a, str))

failure :: Parser a
failure = P (\_ -> Nothing)

item :: Parser Char
item = P $ \str -> case str of
  "" -> Nothing
  (ch:chs) -> Just (ch, chs)

instance Alternative Parser where
  empty = failure

  p1 <|> p2 = P $ \str -> case parse p1 str of
    Nothing -> parse p2 str
    result -> result

instance Monad Parser where
  p1 >>= p2 = P $ \str -> case parse p1 str of
    Just (a, str') -> parse (p2 a) str'
    Nothing -> Nothing

  return = pure

Basic parsers

These basic building blocks are then extended with a bit more complex building blocks, which are a little bit more handy for building complex parsers:

sat (ch -> Bool) - returns a parser which only succeeds if the first character of a (non-empty) input string satisfies the function ( ch -> Bool )
zeroOrMore p - returns a parser which succeeds if the parser p (passed as a param) succeeds on the input string (potentially multiple times) or fails - regardless, the resulting parser will succeed
zeroOrOne p - returns a parser which succeeds if the parser p (passed as a param) succeeds exactly once on an input string or if it fails
oneOrOne p - returns a parser which succeeds if the parser p (passed as a param) succeeds once or more on an input string

With these building blocks, one can build more complex parsers:

Parsing a digit

>>> digit = sat isDigit
>>> parse digit "123abc"
Just ('1',"23abc")

Parsing a number (as a string)

>>> numberStr = oneOrMore (sat isDigit)
>>> parse numberStr "123abc"
Just ("123","abc")

Since Parser is an instance of Monad , you can use fmap or <$> to combine it with other functions:

Parsing a number (as a non-negative number)

>>> naturalNumber :: Parser Integer; naturalNumber = read <$> oneOrMore (sat isDigit)
>>> parse naturalNumber "123abc"
Just (123,"abc")

Parsiung a potentially negative number

>>> sign = fmap (maybe 1 (\_ -> -1)) (zeroOrOne (sat (== '-')))
>>> intNumber = ((*) <$> sign) <*> naturalNumber
>>> parse intNumber "-123abc"
Just (-123,"abc")

The code above requires a bit of an explanation, I guess. The first part,

sign = fmap (maybe 1 (\_ -> -1)) (zeroOrOne (sat (== '-')))

parses a potential - sign at the beginning of a string and returns either 1 or -1 , aka the sign multiplier. The tricky part is fmap (maybe 1 (\_ -> -1)) ... - the zeroOrOne parser will return Maybe a -- that is, Maybe (Maybe a, String) . So if the wrapped part (in this case - sat (== '-') ) is present in the string, it will return Just a (in this case - Just '-' ). Hence we need to cast this Maybe Char to something reasonable - a number would do. We call the maybe helper with two params - the first one is what would be returned if it is applied to Nothing and the second one is a function which will be called on the value wrapped in Just the thing is applied to:

>>> maybe 1 (\_ -> -1) (Just '-')
-1
>>> maybe 1 (\_ -> -1) Nothing
-1
>>> maybe False (\ch -> ch == '-') (Just '-')
True
>>> maybe True (\ch -> ch == '-') Nothing
True

The fmap bit then applies this function (returned by maybe 1 (\_ -> -1) ) to the value wrapped by the next argument:

>>> fmap (maybe 1 (\_ -> -1)) (Just (Just '-'))
Just (-1)
>>> fmap (maybe 1 (\_ -> -1)) (Just Nothing)
Just 1

This very same code could be rewritten as follows:

(maybe 1 (\_ -> -1)) <$> (Just Nothing)

Finally, the second part:

((*) <$> sign) <*> number

The left part of it is just like the previously rewritten function from fmap to <$> , so this entire line can be written down as follows:

(fmap (*) sign) <*> number

What it does is applies the (*) function (multiplication) to the value wrapped in the second argument. In this case it is sign , which is a Parser Integer . This is the tricky bit: the type of this expression is not Parser Integer . It is Parser (a -> a) , a parser of a function. This function, wrapped in a Parser type, can be applied to whatever other parser returns and hence chained together. The <*> operator, as mentioned before, chains the two parsers. So the entire expression applies the multiplication operation to the value returned by the sign parser and the value returned by the number parser.

TL;DR: the whole thing analyzes the first character of a string (without consuming it) and returns either 1 or -1 ; it then multiplies this value by the number returned by the number parser.

>>> parse intNumber "-123"
Just (-123,"")
>>> parse intNumber "123"
Just 123

Using a <|> operator, one can parse integer (both negative and non-negative) numbers in this weird manner:

negativeNumber = (sat (== '-')) >> (* (-1)) <$> read <$> oneOrMore (sat isDigit)
positiveNumber = read <$> oneOrMore (sat isDigit)
number2 = negativeNumber <|> positiveNumber

>>> parse number2 "-42"
Just (42,"")
>>> parse number2 "123"
Just (123,"")

Combining parsers

To give a few practical examples of combining the parsers into something meaningful, consider the parser of an integer equation:

-3x1 + 4*x2 = 5

There are few helpful smaller parsers mentioned above, like intNumber .

But we would need a bit more than that: an equation is a series of members ( -3x1 , 4*x2 ), where each member is a combination of a factor ( -3 , 4 ) and a variable ( x1 , x2 ). There might be either an addition or subtration between members. Lastly, equation must end with an equal sign followed by a free-standing number.

Let's see if we can come up with a plan for this.

We can start at the bottom and define parsers for each bit of an equation or start from the top and define the parser for an entire equation and go down to every member and factor. Let's start with the small bits, the members:

equationMember :: Parser (Integer, String)
equationMember = do
    -- first, the numeric factor
    factor <- equationFactor -- TODO

    -- then, optionally, a multiplication sign, potentially surrounded by any number of whitespace
    zeroOrMore (sat isSpace)
    zeroOrOne (sat (== '*'))
    zeroOrMore (sat isSpace)

    -- followed by the variable name - only letters first, then either letters or numbers
    nameFirst <- oneOrMore (sat isAlpha)
    nameRest <- zeroOrMore (sat isAlphaNum)

    -- potentially surrounded by any number of whitespace
    zeroOrMore (sat isSpace)

    -- lastly, return the tuple - numeric factor and the name of a variable
    return (factor, nameFirst ++ nameRest)

This should be relatively straightforward to follow. We do not have the equationFactor defined which is used in this parser, so here it is:

equationFactor :: Parser Integer
equationFactor = do
    -- optionally, a sign (+ or -)
    _sign <- factorSign -- TODO

    -- surrounded by any number of whitespace
    zeroOrMore (sat isSpace)

    -- followed by a number
    factor <- fmap (fromMaybe 1) (zeroOrOne intNumber)

    -- combine the numeric factor and a member sign (+ or -) for a final factor number
    return (_sign * factor)

The factorSign is yet to be implemented. The reason why I extract it into a separate parser is because one-liner would look ugly:

factorSign :: Parser Integer
factorSign = positiveSign <|> negativeSign <|> (success 1)

positiveSign = fmap (\_ -> 1) (sat (== '+'))

negativeSign = fmap (\_ -> -1) (sat (== '-'))

This works by either matching the + sign and converting it to 1 or matching the - sign and converting it to -1 or succeeding with 1 (by default, when there is no sign).

Lastly, an entire equation parser would be a rather simple combination of the above parsers:

equation :: Parser ([(Integer, String)], Integer)
equation = do
    -- first, match all the members as a list of tuples (factor, variable name)
    members <- oneOrMore equationMember

    -- potentially surrounded by any number of whitespace, the '=' sign
    zeroOrMore (sat isSpace)
    sat (== '=')
    zeroOrMore (sat isSpace)

    -- the last, free-standing member
    freeMember <- intNumber

    -- finally, return the tuple of members and a free-standing member
    return (members, freeMember)

You can try it on the equation from the example above:

>> parse equation "-3x1 + 4*x2 = 5"
Just (([(-3,"x1"),(4,"x2")],5),"")

The finished project contains a little bit more - it implements rational calculus (rational factors instead of just pure integers). The entire code is also hosted on GitHub .

ConfigScript

2022-11-07T00:00:00Z

Once upon a time I thought OGRE was overly too complicated - all those unnecessary script files, custom formats, a ton of setup hassle. That was until I tried figuring out an entire modern 3D rendering stack from scratch. That's when I realized configuring a rendering process (rendering pipeline) can be tricky. And that's when I realized configuring application with config files can be helpful. OGRE suddenly became very appealing to me and I really began to appreciate all the work the devs have put into it.

One good aspect of OGRE was the "unnecessary" script and configuration files. But the syntax of those files looked much cleaner than that of JSON:

// This is a comment
object_keyword Example/ObjectName
{
    attribute_name "some value"

    object_keyword2 "Nested Object"
    {
        other_attribute 1 2 3
        // and so on..
    }
}

I thought if I could harvest anything from OGRE into my OpenGL application, that would be the configuration based on this format (rather than Lua or whatever scripts).

Hence I crafted this simple grammar in ANTLR4 to parse these files:

grammar ConfigScript;

config : (object | comment)* EOF ;

object
    : Identifier '{' property* '}'
    | Identifier STRING '{' (property)* '}'
    ;

property : Identifier propertyValue ;

propertyValue
    : vector
    | INT
    | FLOAT
    | BOOL
    | STRING
    | objectValue
    ;

objectValue
    : '{' property* '}'
    | STRING '{' (property)* '}'
    ;

vector
    : INT+
    | FLOAT+
    ;

comment : LINE_COMMENT | BLOCK_COMMENT ;

STRING : DOUBLE_QUOTED_STRING | SINGLE_QUOTED_STRING ;

BOOL : 'true' | 'false' ;

DOUBLE_QUOTED_STRING : '"' DoubleQuoteStringChar* '"' ;
SINGLE_QUOTED_STRING : '\'' SingleQuoteStringChar* '\'' ;

Identifier : ALPHA (ALPHA | NUM)* ;

fragment SingleQuoteStringChar : ~['\r\n] ;
    // : ~['\\\r\n]
    // | SimpleEscapeSequence ;

fragment DoubleQuoteStringChar : ~["\r\n] ;
    // : ~["\\\r\n]
    // | SimpleEscapeSequence ;

// fragment SimpleEscapeSequence : '\\' ['"?abfnrtv\\] ;

INT : '0'
    | '-'? [1-9] [0-9]*
    ;

FLOAT : ('+' | '-')? NUM+ '.' NUM+ ;

WHITESPACE : [ \r\n\t]+ -> skip ;
ALPHA : [a-zA-Z_] ;
NUM : [0-9] ;

LINE_COMMENT : '//' ~[\r\n]* -> skip ;
BLOCK_COMMENT : '/*' .*? '*/' -> skip ;

The only difference is that the object name can only be a quoted string:

// This is a comment
object_keyword "Example/ObjectName" // <--- this can not be just Example/ObjectName
{
    attribute_name "some value"

    object_keyword2 "Nested Object"
    {
        other_attribute 1 2 3
        // and so on..
        /* block comment */
    }
}

The only thing left with this parser thing is to compile it and use in a project.

ANTLR version incompatibility

It's bragging time! For C++ users, life was never easy. Yet, some good samaritan created the CMake thing. It is ugly as heck, it is barebones and it does not really simplify the life by a whole lot, yet it allows devs to somehow manage build process a bit better than Makefiles. But it is one more tool to learn, keep up-to-date and hate. Then, Microsoft came up with a package manager for C++, vcpkg. And they decided to integrate it with CMake, which became quite popular back then. One more tool to learn, keep up-to-date and hate. But not as much as CMake - this one actually helps quite a bit. Like when you have to add a library or two to your project, it becomes much easier than using barebones CMake. And that's where we come to a point of keeping the thing up-to-date. The vcpkg repository is technically community-driven. Meaning people from all around the world are responsible for keeping repository in good shape, but only Microsort-approved (employed?) people can approve merges to the repo. And often this only happens when somebody has to use the port (dependency from vcpkg repo) and sees an issue and has enough time and passion to go and fix it up.

That's a long brag, but the thing is: if you go to ANTLR4 website and just download the JAR and use it to generate the parser & lexer sources for your C++ project, you won't get it to work with the runtime from vcpkg (unless somebody stands up and updates the port). The thing is: vcpkg port provides runtime for ANTLR4 4.10.1 , while the generator on ANTLR4 website has version 4.11.1 . This is an issue, since between these two versions, there were breaking changes in the ANTLR4 runtime source, so the code generated by the newer version can not be used with the newer version of runtime. Simply because classes were moved around or even created, like ::antlr4::internal::OnceFlag , which never existed in 4.10.1 but was added in 4.11.1 together with an entire header file and namespace.

Luckily, some good guy by a gesture of sheer good will created a PR to fix the issue (by updating the version of ANTLR in vcpkg repository), which should be merged in by the time this blog will be published.

You have to match the ANTLR4 source generator version with the version of the runtime provided by the vcpkg port.

Actual implementation

That thing aside, there are few tricks when using the port in the code.

For the vcpkg port, the vcpkg.json file should contain the antlr4 dependency:

{
  "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg/master/scripts/vcpkg.schema.json",
  "name": "antlr-configscript-cpp",
  "version-string": "0.1.0",
  "dependencies": [
      "antlr4"
  ]
}

The CMakeFile.txt however should look for antlr4-runtime package, link the antlr4_shared or antlr4_static library and explicitly add the include directories from the internal variable, ${ANTLR4_INCLUDE_DIR} :

cmake_minimum_required(VERSION 3.20 FATAL_ERROR)

project(antlr_configscript_cpp VERSION 0.1.0 LANGUAGES CXX)

add_executable(antlr_configscript_cpp
    "main.cpp"
    "gen_parser/ConfigScriptLexer.cpp"
    "gen_parser/ConfigScriptLexer.h"
    "gen_parser/ConfigScriptParser.cpp"
    "gen_parser/ConfigScriptParser.h"
    "gen_parser/ConfigScriptBaseListener.cpp"
    "gen_parser/ConfigScriptBaseListener.h"
)

set_property(TARGET antlr_configscript_cpp PROPERTY CXX_STANDARD 20)

// linking ANTLR4
find_package(antlr4-runtime CONFIG REQUIRED)
target_link_libraries(antlr_configscript_cpp PRIVATE antlr4_shared)
target_include_directories(antlr_configscript_cpp PRIVATE ${ANTLR4_INCLUDE_DIR})

With those issues sorted out, the code to parse the source for the grammar is relatively straightforward:

#include <iostream>

#include <antlr4-runtime.h>

#include "gen_parser/ConfigScriptLexer.h"
#include "gen_parser/ConfigScriptParser.h"
#include "gen_parser/ConfigScriptBaseListener.h"

#pragma execution_character_set("utf-8")

int main()
{
    const auto configSource = R"(
// This is a comment
object_keyword "Example/ObjectName" // <--- this can not be just Example/ObjectName
{
    attribute_name "some value"

    object_keyword2 "Nested Object"
    {
        other_attribute 1 2 3
        // and so on..
        /* block comment */
    }
}
)";

    antlr4::ANTLRInputStream input(configSource);

    ConfigScriptLexer lexer(&input);

    antlr4::CommonTokenStream tokens(&lexer);

    ConfigScriptParser parser(&tokens);

    antlr4::tree::ParseTree* tree = parser.config();

    auto s = tree->toStringTree(&parser);

    std::cout << "Parse Tree: " << s << std::endl;

    return 0;
}

This program would yield an abstract syntax tree looking like this:

(config (object object_keyword "Example/ObjectName" { (property attribute_name (propertyValue "some value")) (property object_keyword2 (propertyValue (objectValue "Nested Object" { (property other_attribute (propertyValue (vector 1 2 3))) }))) }) <EOF>)

Now, what do we do with it? Well, first we need to implement our own version of ConfigScriptBaseListener :

class MyConfigListener : public ConfigScriptBaseListener {
public:
    virtual void enterConfig(ConfigScriptParser::ConfigContext* ctx) override { }
    virtual void exitConfig(ConfigScriptParser::ConfigContext* ctx) override { }

    virtual void enterObject(ConfigScriptParser::ObjectContext* ctx) override { }
    virtual void exitObject(ConfigScriptParser::ObjectContext* ctx) override { }

    virtual void enterProperty(ConfigScriptParser::PropertyContext* ctx) override { }
    virtual void exitProperty(ConfigScriptParser::PropertyContext* ctx) override { }

    virtual void enterPropertyValue(ConfigScriptParser::PropertyValueContext* ctx) override { }
    virtual void exitPropertyValue(ConfigScriptParser::PropertyValueContext* ctx) override { }

    virtual void enterObjectValue(ConfigScriptParser::ObjectValueContext* ctx) override { }
    virtual void exitObjectValue(ConfigScriptParser::ObjectValueContext* ctx) override { }

    virtual void enterVector(ConfigScriptParser::VectorContext* ctx) override { }
    virtual void exitVector(ConfigScriptParser::VectorContext* ctx) override { }

    virtual void enterComment(ConfigScriptParser::CommentContext* ctx) override { }
    virtual void exitComment(ConfigScriptParser::CommentContext* ctx) override { }

    virtual void enterEveryRule(antlr4::ParserRuleContext* ctx) override { }
    virtual void exitEveryRule(antlr4::ParserRuleContext* ctx) override { }
    virtual void visitTerminal(antlr4::tree::TerminalNode* node) override { }
    virtual void visitErrorNode(antlr4::tree::ErrorNode* node) override { }
};

Then we need to pass an instance of this new class to the TreeWalker so that we can process the tree node-by-node:

auto listener = std::make_unique<MyConfigListener>();

auto walker = std::make_unique<antlr4::tree::ParseTreeWalker>();

walker->walk(listener.get(), tree);

But for that to properly work, we'd need to handle every context separately - meaning whenever we enter a nested tree node (like IntVector ), we would want to have the pointer to a this temporary vector to be able to add elements to. And when we exit this node, we would need a pointer to whatever the parent of that vector was, to be able to add this vector to that parent.

This might get out of hand quite quickly.

Alternatively, and arguably more convenient way to handle this is using attributes and actions in the grammar itself:

grammar ConfigScript;

@header {
    #include <variant>
    #include <any>
}

config : objects=object* EOF ;

object
    returns [
        std::string name
    ]
    : Identifier objectValue { $name = $Identifier->getText(); }
    ;

property
    returns [
        std::string name,
        std::any value
    ]
    : Identifier propertyValue { $name = $Identifier->getText(); antlrcpp::downCast<ObjectValueContext*>(_localctx->parent)->propertyMap[$name] = $value; }
    ;

propertyValue
    : intVector { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $intVector.elements; }
    | floatVector { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $floatVector.elements; }
    | INT { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = std::stoi($INT.text); }
    | FLOAT { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = std::stof($FLOAT.text); }
    | BOOL { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = static_cast<bool>($BOOL.text == "true"); }
    | STRING { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $STRING.text; }
    | objectValue { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $objectValue.propertyMap; }
    ;

objectValue
    returns [
        std::string classifier,
        std::map<std::string, std::any> propertyMap
    ]
    : '{' property* '}'
    | STRING '{' (property)* '}' { $classifier = $STRING.text; }
    ;

intVector
    returns [ std::vector<int> elements ]
    : INT+ { auto v = $ctx->INT(); std::for_each(v.begin(), v.end(), [&](auto* node) { _localctx->elements.push_back(std::stoi(node->getText())); }); }
    ;

floatVector
    returns [ std::vector<float> elements ]
    : FLOAT+ { auto v = $ctx->FLOAT(); std::for_each(v.begin(), v.end(), [&](auto* node) { _localctx->elements.push_back(std::stof(node->getText())); }); }
    ;

This way we couple the parser with the specific language (C++ in this case), but this allows us to write bare minimum code afterwards:

auto objects = antlrcpp::downCast<ConfigScriptParser::ConfigContext*>(tree)->object();

However, there quite a few tricks involved.

First of all, the huge benefit is that this way we are free to specify code to be run on entering each rule and we have an access to all of the rule context:

propertyValue
    : intVector { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $intVector.elements; }
    | floatVector { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $floatVector.elements; }
    | INT { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = std::stoi($INT.text); }
    | FLOAT { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = std::stof($FLOAT.text); }
    | BOOL { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = static_cast<bool>($BOOL.text == "true"); }
    | STRING { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $STRING.text; }
    | objectValue { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $objectValue.propertyMap; }
    ;

See how the nested rules are referenced by their names ( intVector is accessed with $intVector , INT is accessed with $INT , rule text is accessed via $RULE.text ). There is also the current rule' context available through _localctx .

All of this logic is injected into the code generated by ANTLR, so you can access it as-is and see what is available and what not.

But the issue is: whatever the code is that you write in the rules, it won't be compiled when generating the parser code - it would be inserted into the generated code as-is. You will have to make sure it compiles as a separate step.

Some of the rules allow you to define what other data will can be accessed (like the integer value of an INT rule or elements of the intVector rule):

intVector
    returns [ std::vector<int> elements ]
    : INT+ { auto v = $ctx->INT(); std::for_each(v.begin(), v.end(), [&](auto* node) { _localctx->elements.push_back(std::stoi(node->getText())); }); }
    ;

This also acts as the initialization of the additional context fields.

Second, all of these antlrcpp::downCast<Context*>(_localctx->parent) are required to access the parent context (so no $parent or something).

Accessing attribute: $vector.value even though it is not valid C++ code, since $vector will be resolved to a context pointer ( VectorContext* ), hence we have to use all those antlrcpp::downCast .

Accessing repeated rules as a vector: $INT.begin() and $INT.end() would resolve in something else (see the above, accessing attribute). Hence a trick is to access it via context: $ctx->INT().begin() .

Accessing parent attribute:

propertyValue
    : vector { $property::value.emplace($vector.elements); }

The $property::value won't work and will throw missing code generation template NonLocalAttrRefHeader . I am unsure how to fix this correctly (this should be valid, according to documentation), so I hacked my way through:

propertyValue
    : vector { antlrcpp::downCast<PropertyContext*>(_localctx->parent)->value = $vector.elements; }

And a simple program that prints out the parsed config:

#include <iostream>
#include <map>
#include <variant>

#include <antlr4-runtime.h>

#include "gen_parser/ConfigScriptLexer.h"
#include "gen_parser/ConfigScriptParser.h"
#include "gen_parser/ConfigScriptBaseListener.h"

#pragma execution_character_set("utf-8")

void printAny(std::any value)
{
    if (value.type() == typeid(std::string))
    {
        std::cout << "(str){ " << std::any_cast<std::string>(value) << " };";
    }
    else if (value.type() == typeid(int))
    {
        std::cout << "(int){ " << std::any_cast<int>(value) << " };";
    }
    else if (value.type() == typeid(float))
    {
        std::cout << "(float){ " << std::any_cast<float>(value) << " };";
    }
    else if (value.type() == typeid(std::vector<int>))
    {
        auto vec = std::any_cast<std::vector<int>>(value);

        std::cout << "(int[]){ ";

        for (auto val : vec)
        {
            std::cout << val << ", ";
        }

        std::cout << " };";
    }
    else if (value.type() == typeid(std::vector<float>))
    {
        auto vec = std::any_cast<std::vector<float>>(value);

        std::cout << "(float[]){ ";

        for (auto val : vec)
        {
            std::cout << val << ", ";
        }

        std::cout << " };";
    }
    else if (value.type() == typeid(std::map<std::string, std::any>))
    {
        auto vec = std::any_cast<std::map<std::string, std::any>>(value);

        std::cout << "(obj){ ";

        for (auto& val : vec)
        {
            std::cout << val.first << " = ";

            printAny(val.second);
        }

        std::cout << " };";
    }
}

int main()
{
    const auto configSource = R"(
// This is a comment
object_keyword "Example/ObjectName" // <--- this can not be just Example/ObjectName
{
    attribute_name "some value"

    object_keyword2 "Nested Object"
    {
        other_attribute 1 2 3
        // and so on..
        /* block comment */
    }
}
)";

    antlr4::ANTLRInputStream input(configSource);

    ConfigScriptLexer lexer(&input);

    antlr4::CommonTokenStream tokens(&lexer);

    ConfigScriptParser parser(&tokens);

    antlr4::tree::ParseTree* tree = parser.config();

    auto s = tree->toStringTree(&parser);

    std::cout << "Parse Tree: " << s << std::endl;

    auto objects = antlrcpp::downCast<ConfigScriptParser::ConfigContext*>(tree)->object();

    std::cout << "Objects found: " << objects.size() << std::endl;

    for (auto o : objects)
    {
        std::cout << "[" << o->name << "] { ";

        auto objectValue = o->objectValue();
        auto propertyMap = objectValue->propertyMap;

        for (auto& p : propertyMap)
        {
            std::cout << p.first << " = ";

            printAny(p.second);

            std::cout << std::endl;
        }

        std::cout << " } " << std::endl;
    }

    return 0;
}

Hope this gives you a brief introduction into ANTLR4 and some of the tips on implementing your own parsers.

OpenGL: advanced samples

2022-10-26T00:00:00Z

Damn this blog was long coming - I picked my interest in shaders over a decade ago, started to actually learn something in early 2021 and started writing this blog at the end of 2021. Now the time has finally come to share this with the world!

For a long time I was keen in learning low-level computer graphics APIs and algorithms, but only recently actually made any progress towards that goal. I have spent few weeks if not months learning about shaders, graphics pipeline and rendering techniques, so this write-up was long forecoming. There might be some mistakes, since I am not an expert in CG and OpenGL and there is a big chunk missing, namely rendering animated 3D models, compute shaders and tesselation, but I hope to fix those in the future.

One might ask: why OpenGL? Why not Vulkan (or DirectX 12)? Why even bother with these emulaiton techniques - why not jump straight to raytracing? Well, it is the end of October 2022 at the moment of publishing this blog, the most recent GPU on the market is GeForce RTX 4090, which is yet to arrive and its RRP (recommended retailer price) is A$2959 (as advertised by nVidia):

And most of those cards are gone in a blink of an eye and used for mining whatever-coins. My PC runs GTX1080, which is still more than enough for most of the games I play (being Genshin Impact at most).

I think it is still not a widely spread consumer market, and whilst raytracing, DLSS and all those fine things sound super cool and nice, it is still a niche market. That is not to say I won't be looking into that in the near future. I just think OpenGL still has place in this world, at least as a starting point to build a decent understanding of the modern graphics pipeline, some basic rendering techniques and justify switching to more recent APIs. OpenGL as well as DirectX 11 are still a very feasible fallback for those sytems which do not run top-notch hardware or are struggling running those massive open-world games in anything higher than 1080p@60fps.

In this blog I will try to condense the knowledge I have acquired. I will skip most basic parts such as initializing context, creating window and so on, since most of the tutorials and articles on the Internet focus on those topics. Unlike most tutorials and articles out there, this blog will be all about rendering techniques .

This article was heavily inspired by few blogs on russian website ( Habr ), namely "super-modern OpenGL" ( part 1 and part 2 ) - they are quite messy and lack a lot of material on really interesting topics (again, rendering techniques). This blog partially builds on top of those two and uses a lot more other materials (references provided).

This blog is enormous, so blow is the table of contents for the topics covered. Feel free to jump straight to the topic that picks your interest, screenshots are provided:

Setting up
Basics I promised not to focus on
Optimization techniques
Rendering techniques
Common rendering techniques

The source code is available on GitHub .

Setting up

Just to get this out of the way: I have used few available libraries to simplify my work. You can totally use anything else in your projects with greater success, these were just my decisions which seemed right for me at that time.

SFML to manage the window creation, input and loading images
globjects to simplify some of the OpenGL entity management (framebuffers, textures, shader programs, vertex attribute objects and buffers)
glm for maths (algebra and matrices, predominantly)
assimp to load 3D models

The project uses CMake to handle the build process and vcpkg to manage the dependencies. I have also tried xmake but because it lacks IDE support (I have used Visual Studio) , it did not quite fit me.

In these projects I have tried to utilize the new C++ features (C++17 onwards) as much as possible, but I might be missing few great improvements it gives (mostly around containers and memory ownership) .

I also have implemented a sample of using imgui for user interface way down the line as I thought it was not all that helpful for my projects, but apparently, I could have saved myself a lot of debugging time if I added few windows and buttons here and there. Hence I encourage you to look into that too.

Basics I promised not to focus on

Again, to get this out of the way and not go into too much detail on these topics, few words on how I approached OpenGL from my very much outdated background (think OpenGL ver. 1) .

Getting started with SFML

Creating a window and handling user input pretty much remained the same:

#include <SFML/Window.hpp>

int main()
{
#ifdef SYSTEM_DARWIN
    auto videoMode = sf::VideoMode(2048, 1536);
#else
    auto videoMode = sf::VideoMode(1024, 768);
#endif

    sf::Window window(videoMode, "Hello SFML Window!");

    while (window.isOpen())
    {
        sf::Event event {};

        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
            {
                window.close();
            }
        }
    }

    return 0;
}

Getting started with globjects

The approach to rendering has changed drastically in past decades: from passing each vertex and its attributes to OpenGL it has grown to better suit modern graphics cards' architectures and now looks like passing chunks of data to be stored in the graphics memory and calling graphics driver to run pre-compiled programs on the GPU.

In order to run the program on GPU, it first needs to be compiled from the source. And just like any ordinary program for CPU, the program for GPU (called "shader program" or just "shader") has multiple stages when it runs - you can think about these stages as stages in a big data processing pipeline - you get some input and tell your executor (GPU in this case) how to process and transform the data. This pipeline, however, has few pre-defined stages with pre-defined (to some extent) input and output formats. The stages we will be using in all our samples are:

vertex shader stage - takes a bunch of vertices with bunch of attributes as flat integer or float numbers or packs of 2, 3 or 4 of those numbers, aka vectors; it transforms those vertices (if needed, generally transforms vertices as if they were seen by the "camera") and returns each vertex with attributes assigned to each vertex (the attributes could be arbitrary - numbers or vectors)
geometry shader stage - takes each vertex and produces the geometry it will be part of (think lines, triangles or polygons); returns new vertices with vertex attributes assigned to each vertex (numbers or vectors)
fragment shader stage - takes each geometry (line or triangle) formed by multiple vertices with attributes assigned to each vertex and projects them onto screen (or texture, aka "render target"), returning a pixel data suitable for each render target (one target - one type of pixel data accepted)

There are more of those stages, but those are the most commonly used ones. Each of them is customizable by a separate shader. That is, OpenGL will throw an error if you try to return fragment shader data from vertex shader.

The way you render anything on the screen is: you take a bunch of vertices in 3D space, assign them some data (like the color or normal or texture coordinates), pack them into a buffer object (that is, just a buffer full of data). You then have to tell OpenGL what the data format is. This is done using vertex attribute object - this is essentially a descriptor, telling OpenGL how it should read and process the bytes stored in the buffer.

Here's a comprehensive example of a simple shader initialization and rendering:

#include <fstream>
#include <iostream>

#include <glbinding/gl/gl.h>

#include <globjects/Buffer.h>
#include <globjects/Error.h>
#include <globjects/Program.h>
#include <globjects/Shader.h>
#include <globjects/VertexArray.h>
#include <globjects/VertexAttributeBinding.h>
#include <globjects/base/File.h>
#include <globjects/base/StaticStringSource.h>
#include <globjects/globjects.h>

#include <glm/vec2.hpp>

#include <SFML/OpenGL.hpp>
#include <SFML/Window.hpp>

#ifdef WIN32
using namespace gl;
#endif

int main()
{
    sf::ContextSettings settings;
    settings.depthBits = 24;
    settings.stencilBits = 8;
    settings.antialiasingLevel = 4;
    settings.majorVersion = 3;
    settings.minorVersion = 2;
    settings.attributeFlags = sf::ContextSettings::Attribute::Core;

#ifdef SYSTEM_DARWIN
    auto videoMode = sf::VideoMode(2048, 1536);
#else
    auto videoMode = sf::VideoMode(1024, 768);
#endif

    sf::Window window(videoMode, "Hello OpenGL!", sf::Style::Default, settings);

    globjects::init([](const char* name) {
        return sf::Context::getFunction(name);
    });

    globjects::DebugMessage::enable(); // enable automatic messages if KHR_debug is available

    globjects::DebugMessage::setCallback([](const globjects::DebugMessage& message) {
        std::cout << "[DEBUG] " << message.message() << std::endl;
    });

    std::cout << "[INFO] Initializing..." << std::endl;

    auto cornerBuffer = globjects::Buffer::create();

    std::cout << "[INFO] Creating shaders..." << std::endl;

    std::cout << "[INFO] Compiling vertex shader...";

    auto vertexShaderSource = globjects::Shader::sourceFromFile("media/vertex.glsl");
    auto vertexShaderTemplate = globjects::Shader::applyGlobalReplacements(vertexShaderSource.get());
    auto vertexShader = globjects::Shader::create(static_cast<gl::GLenum>(GL_VERTEX_SHADER), vertexShaderTemplate.get());

    std::cout << "done" << std::endl;

    std::cout << "[INFO] Compiling fragment shader...";

    auto fragmentShaderSource = globjects::Shader::sourceFromFile("media/fragment.glsl");
    auto fragmentShaderTemplate = globjects::Shader::applyGlobalReplacements(fragmentShaderSource.get());
    auto fragmentShader = globjects::Shader::create(static_cast<gl::GLenum>(GL_FRAGMENT_SHADER), fragmentShaderTemplate.get());

    std::cout << "done" << std::endl;

    std::cout << "[INFO] Linking shader programs...";

    auto renderingProgram = globjects::Program::create();
    renderingProgram->attach(vertexShader.get(), fragmentShader.get());

    std::cout << "done" << std::endl;

    std::cout << "[INFO] Creating VAO...";

    auto vao = globjects::VertexArray::create();

    cornerBuffer->setData(
        std::array<glm::vec2, 4> {
            { glm::vec2(0, 0), glm::vec2(1, 0), glm::vec2(0, 1), glm::vec2(1, 1) }
        },
        static_cast<gl::GLenum>(GL_STATIC_DRAW));

    vao->binding(0)->setAttribute(0);
    vao->binding(0)->setBuffer(cornerBuffer.get(), 0, sizeof(glm::vec2));
    vao->binding(0)->setFormat(2, static_cast<gl::GLenum>(GL_FLOAT));
    vao->enable(0);

    std::cout << "done" << std::endl;

    std::cout << "[INFO] Done initializing" << std::endl;

    while (window.isOpen())
    {
        sf::Event event {};

        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
            {
                window.close();
                break;
            }
        }

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        ::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));

        renderingProgram->use();

        vao->drawArrays(static_cast<gl::GLenum>(GL_TRIANGLE_STRIP), 0, 4);

        renderingProgram->release();

        window.display();
    }

    return 0;
}

vertex shader:

#version 410

layout (location = 0) in vec2 corner;

out vec4 color;

out gl_PerVertex
{
    vec4 gl_Position;
};

void main()
{
    gl_Position = vec4(corner * 2.0 - 1.0, 0.0, 1.0);
    color = vec4(corner, 0.0, 1.0);
}

fragment shader:

#version 410

layout (location = 0) out vec4 fragColor;

in vec4 color;

void main()
{
    fragColor = color;
}

Camera and getting started with glm

A lot of operations are described as mathematical operations on bunch of vectors with the use of matrices.

Most of the transformations could be done on either CPU (think C++ code) or GPU (think shader code). But keep in mind: if you do a certain operation in vertex shader - it will be executed for each vertex you render with that shader. If you do it in fragment shader - it will be executed for each pixel rendered. Those numbers might be really big and vastly different (for a simple scene with just a plane - 4 vertices vs 1024 * 768 = 786432 pixels, for a big scene - say 2K objects with 3M vertices = 2*10^3 * 3*10^6 = 6*10^9 vertices vs same 786432 pixels).

GLSL comes packed with quite a lot of mathematical operations already, whilst for C++ we can use glm to handle all those nitty-gritty matrix multiplications and inverse matrix calculations.

In order to communicate some data from the CPU to the shader running on GPU, you can use uniform variables.

Here's a simple example of camera implementation with glm:

#include <glm/ext/matrix_clip_space.hpp>
#include <glm/ext/matrix_transform.hpp>
#include <glm/gtx/rotate_vector.hpp>
#include <glm/mat4x4.hpp>
#include <glm/vec2.hpp>
#include <glm/vec3.hpp>

// ...

int main()
{
    // ...

    const float fov = 45.0f;

    const float cameraMoveSpeed = 1.0f;
    const float cameraRotateSpeed = 10.0f;

    glm::vec3 cameraPos = glm::vec3(0.0f, 0.0f, 3.0f);
    glm::vec3 cameraUp = glm::vec3(0.0f, 1.0f, 0.0f);
    glm::vec3 cameraRight = glm::vec3(1.0f, 0.0f, 0.0f);
    glm::vec3 cameraForward = glm::normalize(glm::cross(cameraUp, cameraRight));

    sf::Clock clock;

    // ...

    while (window.isOpen())
    {
        // measure time since last frame, in seconds
        float deltaTime = static_cast<float>(clock.restart().asSeconds());

        glm::vec2 currentMousePos = glm::vec2(sf::Mouse::getPosition(window).x, sf::Mouse::getPosition(window).y);
        glm::vec2 mouseDelta = currentMousePos - glm::vec2((window.getSize().x / 2), (window.getSize().y / 2));
        sf::Mouse::setPosition(sf::Vector2<int>(window.getSize().x / 2, window.getSize().y / 2), window);

        float horizontalAngle = (mouseDelta.x / static_cast<float>(window.getSize().x)) * -1 * deltaTime * cameraRotateSpeed * fov;
        float verticalAngle = (mouseDelta.y / static_cast<float>(window.getSize().y)) * -1 * deltaTime * cameraRotateSpeed * fov;

        cameraForward = glm::rotate(cameraForward, horizontalAngle, cameraUp);
        cameraForward = glm::rotate(cameraForward, verticalAngle, cameraRight);

        cameraRight = glm::normalize(glm::rotate(cameraRight, horizontalAngle, cameraUp));

        if (sf::Keyboard::isKeyPressed(sf::Keyboard::W))
        {
            cameraPos += cameraForward * cameraMoveSpeed * deltaTime;
        }

        if (sf::Keyboard::isKeyPressed(sf::Keyboard::S))
        {
            cameraPos -= cameraForward * cameraMoveSpeed * deltaTime;
        }

        if (sf::Keyboard::isKeyPressed(sf::Keyboard::A))
        {
            cameraPos -= glm::normalize(glm::cross(cameraForward, cameraUp)) * cameraMoveSpeed * deltaTime;
        }

        if (sf::Keyboard::isKeyPressed(sf::Keyboard::D))
        {
            cameraPos += glm::normalize(glm::cross(cameraForward, cameraUp)) * cameraMoveSpeed * deltaTime;
        }

        glm::mat4 projection = glm::perspective(
            glm::radians(fov),
            (float) window.getSize().x / (float) window.getSize().y,
            0.1f,
            100.0f);

        glm::mat4 view = glm::lookAt(
            cameraPos,
            cameraPos + cameraForward,
            cameraUp);

        renderProgram->setUniform("model", model);
        renderProgram->setUniform("view", view);
        renderProgram->setUniform("projection", projection);

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        ::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));

        renderProgram->use();

        vao->drawArrays(static_cast<gl::GLenum>(GL_TRIANGLE_STRIP), 0, 4);

        renderProgram->release();

        window.display();
    }
}

vertex shader:

#version 410

layout (location = 0) in vec3 vertexPos;

out vec4 color;

out gl_PerVertex
{
    vec4 gl_Position;
};

uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;

void main()
{
    gl_Position = projection * view * model * vec4(vertexPos, 1.0);
    color = vec4(vertexPos, 1.0);
}

fragment shader:

#version 410

layout (location = 0) out vec4 fragColor;

in vec4 color;

void main()
{
    fragColor = color;
}

Optimization techniques

Passing data to the shader

Bear in mind: if you want to pass just few bits of data - it is fine to use few uniforms, but if you want to pass a lot of data (think few matrices and lists of data) - you're much better off using buffers. That is, uniform buffer objects (UBO) or even better - shader shared buffer objects (SSBO) - the latter can store much larger amounts of data (according to OpenGL standard guarantees, UBO can store up to 16 kB whilst SSBO - up to 128 MB , but most drivers will let you store up to graphic card's memory limit) and you can have arrays of data in SSBOs. Coupled with user-defined data structures (think struct but in GLSL), you can pass, for example, list of light sources' descriptors to your shader with SSBO.

Here's how you can do it:

struct alignas(16) PointLightDescriptor
{
    glm::vec3 position;
    float strength;
    glm::vec4 color;
};

int main()
{
    // ...

    std::vector<PointLightDescriptor> pointLights{
        { glm::vec3(-1.75f, 3.85f, -0.75f), 0.5f, glm::vec4(1.0f, 1.0f, 1.0f, 1.0f) }
    };

    auto pointLightDataBuffer = std::make_unique<globjects::Buffer>();

    pointLightDataBuffer->setData(pointLights, static_cast<gl::GLenum>(GL_DYNAMIC_COPY));

    // ...

    while (window->isOpen())
    {
        pointLightDataBuffer->bindBase(GL_SHADER_STORAGE_BUFFER, 5);

        // bind a shader

        pointLightDataBuffer->unbind(GL_SHADER_STORAGE_BUFFER, 5);
    }
}

fragment shader:

struct PointLight
{
    vec3 position;
    float strength;
    vec4 color;
};

layout (std430, binding = 5) buffer PointLightData
{
    PointLight pointLight[];
} pointLightData;

void main()
{
    for (int i = 0; i < pointLightData.pointLight.length(); ++i)
    {
        PointLight light = pointLightData.pointLight[i];

        vec3 lightDirection = normalize(light.position - fragmentPosition);
    }
}

Deferred rendering

Rendering directly to the output (window), also known as "direct rendering", is not very optimal with most techniques (especially post-processing and special effects). For the most part I will try to use technique called "deferred rendering", where you first render your entire scene into multiple textures with the same size as the output window and then you apply various techniques to those textures - this way you do not have to render entire scene multiple times.

Since you can have multiple render targets, you can render just one fragment (pixel) attribute to one render target and then use it in local post-processing calculations. Trick is: you render scene once. In the geometry shader, you render each vertex' data to a different render target. Then you will not need to process each vertex, but each pixel instead. If you want to calculate shadows or reflections or bloom - you will only have to do it for each pixel, not for each vertex, which might save you heaps of GPU resources.

Batch geometry rendering

You can pack the vertex data for your scene however you want. For example, you can have one buffer with just the vertices' positions and another buffer with just vertices' normals (and so on for each of the vertex attributes). But if the geometry you are trying to render is not changing with time, you can store all the data in one buffer - you just have to tell OpenGL how the bytes are packed in the buffer. For example, you can have position, followed by normal, followed by texture coordinates. You just tell OpenGL to have three attributes - one of the size of 3 floats, the next one of the size of three floats and the last one of the size of two floats.

Moreover, you can store multiple objects' vertices in one buffer! The reason for that would be to minimize the number of buffer and shader switching OpenGL has to make in order to render a certain amount of geometry.

You can combine rendering multiple different geometries with instanced rendering and use glMultiDrawElementsIndirect with a list of "drawing commands" to render a whole lot of geometry from buffers stored in GPU memory.

Top it up with the fact that you can store the draw commands in a buffer and you can generate this buffer using compute shaders (not covered yet) and you can have a rendering engine which works almost entirely on the GPU, saving you a lot of synchronization and communication between CPU and GPU.

To understand how the glMultiDrawElementsIndirect works, one must understand the concept of a draw command and how it works in DirectX 12 and Vulkan. The idea is that you prepare all the geometry and the supporting data, store it in video memory and then only tell GPU to render all that mess in a specific order. This way you don't need to send huge chunks of data from CPU to GPU every frame - this spares computer quite some work.

The draw command describes which chunks of data that are already stored in video memory to use for rendering and how to render them.

A typical draw command is represented as the following C++ structure:

struct StaticGeometryDrawCommand
{
    unsigned int elementCount; // number of elements (triangles) to be rendered for this object
    unsigned int instanceCount; // number of object instances
    unsigned int firstIndex; // offset into GL_ELEMENT_ARRAY_BUFFER
    unsigned int baseVertex; // offset of the first object' vertex in the uber-static-object-buffer
    unsigned int baseInstance; // offset of the first instance' per-instance-vertex-attributes; attribute index is calculated as: (gl_InstanceID / glVertexAttribDivisor()) + baseInstance
};

The issue with this approach is: how to pass the per-instance data? Since all the data used by glDrawMulti* is per-vertex. SSBOs to the rescue!

struct alignas(16) StaticObjectData
{
    glm::vec2 albedoTextureSize;
    glm::vec2 normalTextureSize;
    glm::vec2 emissionTextureSize;
    unsigned int instanceDataOffset;
};

struct alignas(16) StaticObjectInstanceData
{
    glm::mat4 transformation;
};

// create two arrays: per-object data and per-instance data

std::vector<StaticObjectData> m_objectData;
std::vector<StaticObjectInstanceData> m_objectInstanceData;

// duck
m_objectData.push_back({
    // TODO: automate this
    .albedoTextureSize = glm::vec2(0, 0),
    .normalTextureSize = glm::vec2(0, 0),
    .emissionTextureSize = glm::vec2(0, 0),
    .instanceDataOffset = 0
});

m_objectInstanceData.push_back({ .transformation = glm::mat4() });

// lantern
m_objectData.push_back({ .albedoTextureSize = glm::vec2(1024, 1024),
                            .normalTextureSize = glm::vec2(1024, 1024),
                            .emissionTextureSize = glm::vec2(0, 0),
                            .instanceDataOffset = 1 });

m_objectInstanceData.push_back({ .transformation = glm::translate(glm::vec3(0.0f, 0.5f, 0.0f)) });

// scroll
m_objectData.push_back({ .albedoTextureSize = glm::vec2(1024, 1024),
                            .normalTextureSize = glm::vec2(0, 0),
                            .emissionTextureSize = glm::vec2(1024, 1024),
                            .instanceDataOffset = 2 });

m_objectInstanceData.push_back({ .transformation = glm::translate(glm::vec3(0.0f, 0.5f, -0.5f)) });

// pen
m_objectData.push_back({ .albedoTextureSize = glm::vec2(512, 512),
                            .normalTextureSize = glm::vec2(512, 512),
                            .emissionTextureSize = glm::vec2(0, 0),
                            .instanceDataOffset = 3 });

m_objectInstanceData.push_back({ .transformation = glm::translate(glm::vec3(0.5f, 0.5f, 0.5f)) });

// generate object data buffer
auto objectDataBuffer = std::make_unique<globjects::Buffer>();

objectDataBuffer->setData(m_objectData, static_cast<gl::GLenum>(GL_DYNAMIC_COPY));

// generate object **instance** data buffer
auto objectInstanceDataBuffer = std::make_unique<globjects::Buffer>();

objectInstanceDataBuffer->setData(m_objectInstanceData, static_cast<gl::GLenum>(GL_DYNAMIC_COPY));

Note how object data contains the texture sizes - this is done so that all the textures can be stored in one texture array (aka 3D texture). This allows for easy texture indexing (using object index instead of having some arbitrary number of texture variables), making this approach extensible. But texture arrays have a limitation: you can have as many textures as you would like in one array, but all textures must have same width and height. However, that does not mean all the pixels matter - you can store smaller textures using their original size and just ignore the rest of the texture data. But in order to do that, you would need to sample the texture with a scale factor. See the shader code below for details.

Then, for even more performance boost, you can put all objects' data into one big data buffer - since we will use drawing commands to render, we can not worry about different buffers. That is, each buffer's element will contain all the necessary vertex data - position, normal and UV coordinates. However, we will still need a separate buffer for indices:

std::vector<unsigned int> m_indices;

std::vector<StaticGeometryDrawCommand> m_drawCommands;

struct NormalizedVertex
{
    glm::vec3 position;
    glm::vec3 normal;
    glm::vec2 uv;
};

std::vector<NormalizedVertex> normalizedVertexData;
unsigned int baseVertex = 0;

for (auto& scene : scenes)
{
    for (auto& mesh : scene->meshes)
    {
        const auto numVertices = mesh.vertexPositions.size();

        for (size_t i = 0; i < numVertices; ++i)
        {
            auto position = mesh.vertexPositions[i];
            auto normal = mesh.normals[i];
            auto uv = mesh.uvs[i];

            normalizedVertexData.push_back({ .position = position, .normal = normal, .uv = uv });
        }

        m_indices.insert(m_indices.end(), mesh.indices.begin(), mesh.indices.end());

        StaticGeometryDrawCommand drawCommand {
            .elementCount = static_cast<unsigned int>(mesh.indices.size()),
            .instanceCount = 1, // TODO: generate commands dynamically whenever the data is changed
            .firstIndex = 0,
            .baseVertex = baseVertex,
            .baseInstance = 0
        };

        m_drawCommands.push_back(drawCommand);

        baseVertex += numVertices;
    }
}

auto vao = std::make_unique<globjects::VertexArray>();

// generate draw command buffer
auto drawCommandBuffer = std::make_unique<globjects::Buffer>();
drawCommandBuffer->setData(m_drawCommands, static_cast<gl::GLenum>(GL_DYNAMIC_DRAW)); // draw commands can technically be changed

// generate vertex data buffer
auto geometryDataBuffer = std::make_unique<globjects::Buffer>();
geometryDataBuffer->setData(normalizedVertexData, static_cast<gl::GLenum>(GL_STATIC_DRAW));

vao->binding(0)->setAttribute(0);
vao->binding(0)->setBuffer(geometryDataBuffer.get(), offsetof(NormalizedVertex, position), sizeof(NormalizedVertex)); // number of elements in buffer, stride, size of buffer element
vao->binding(0)->setFormat(3, static_cast<gl::GLenum>(GL_FLOAT)); // number of data elements per buffer element (vertex), type of data
vao->enable(0);

vao->binding(1)->setAttribute(1);
vao->binding(1)->setBuffer(geometryDataBuffer.get(), offsetof(NormalizedVertex, normal), sizeof(NormalizedVertex)); // number of elements in buffer, stride, size of buffer element
vao->binding(1)->setFormat(3, static_cast<gl::GLenum>(GL_FLOAT)); // number of data elements per buffer element (vertex), type of data
vao->enable(1);

vao->binding(2)->setAttribute(2);
vao->binding(2)->setBuffer(geometryDataBuffer.get(), offsetof(NormalizedVertex, uv), sizeof(NormalizedVertex)); // number of elements in buffer, stride, size of buffer element
vao->binding(2)->setFormat(2, static_cast<gl::GLenum>(GL_FLOAT)); // number of data elements per buffer element (vertex), type of data
vao->enable(2);

// generate element buffer
auto elementBuffer = std::make_unique<globjects::Buffer>();
elementBuffer->setData(m_indices, static_cast<gl::GLenum>(GL_STATIC_DRAW));

vao->bindElementBuffer(elementBuffer.get());

Rendering is then as simple as binding those buffers and calling glMultiDrawElementsIndirect :

drawCommandBuffer->bind(static_cast<gl::GLenum>(GL_DRAW_INDIRECT_BUFFER));
objectDataBuffer->bindBase(static_cast<gl::GLenum>(GL_SHADER_STORAGE_BUFFER), 4);
objectInstanceDataBuffer->bindBase(static_cast<gl::GLenum>(GL_SHADER_STORAGE_BUFFER), 5);

vao->bind();

vao->multiDrawElementsIndirect(
    static_cast<gl::GLenum>(GL_TRIANGLES),
    static_cast<gl::GLenum>(GL_UNSIGNED_INT),
    0,
    m_drawCommands.size(),
    0);

simpleProgram->release();

drawCommandBuffer->unbind(static_cast<gl::GLenum>(GL_DRAW_INDIRECT_BUFFER));
objectDataBuffer->unbind(static_cast<gl::GLenum>(GL_SHADER_STORAGE_BUFFER), 4);
objectInstanceDataBuffer->unbind(static_cast<gl::GLenum>(GL_SHADER_STORAGE_BUFFER), 5);

vao->unbind();

Vertex shader:

#version 460

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    flat uint objectID;
    flat uint instanceID;
} vsOut;

struct ObjectData
{
    vec2 albedoTextureSize;
    vec2 normalTextureSize;
    vec2 emissionTextureSize;
    uint instanceDataOffset; // use this field to get instance data: StaticObjectInstanceData.transformation[StaticObjectData.objectData[gl_DrawID].instanceDataOffset + gl_InstanceID]
};

layout (std430, binding = 4) buffer StaticObjectData
{
    ObjectData[] objectData;
};

layout (std430, binding = 5) buffer StaticObjectInstanceData
{
    mat4[] transformations;
};

uniform mat4 projection;
uniform mat4 view;

void main()
{
    vsOut.fragmentPosition = vertexPosition;
    vsOut.normal = vertexNormal;
    vsOut.textureCoord = vec2(vertexTextureCoord.x, vertexTextureCoord.y);
    vsOut.objectID = gl_DrawID;
    vsOut.instanceID = gl_InstanceID;

    uint objectInstanceIndex = objectData[gl_DrawID].instanceDataOffset + gl_InstanceID;

    mat4 model = transformations[objectInstanceIndex];

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

Fragment shader:

#version 460

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    flat uint objectID;
    flat uint instanceID;
} fsIn;

struct ObjectData
{
    vec2 albedoTextureSize;
    vec2 normalTextureSize;
    vec2 emissionTextureSize;
    uint instanceDataOffset;
};

layout (std430, binding = 4) buffer StaticObjectData
{
    ObjectData[] objectData;
};

uniform sampler2DArray albedoTextures;
uniform sampler2DArray normalTextures;
uniform sampler2DArray emissionTextures;

void main()
{
    vec2 maxTextureSize = textureSize(albedoTextures, 0).xy;
    vec2 uvFactor = objectData[fsIn.objectID].albedoTextureSize / maxTextureSize;
    vec2 uv = vec2(fsIn.textureCoord.x, fsIn.textureCoord.y) * uvFactor;

    vec4 albedo = texture(albedoTextures, vec3(uv.x, uv.y, fsIn.objectID));

    fragmentColor = albedo;
}

[blog] Opengl 4.4 scene rendering techniques

Texture handles

This optimization allows you to spare those glActivateTexture and referring textures by some magical numbers. You create a texture once and freeze its params by using the handles (so you can not change texture params once you have started using its handle), but what you gain is one handle, stored as a 64-bit number (roughly: texture address in GPU memory) which you can then pass to multiple shaders without bothering with those texture IDs.

// create texture - nothing changes here
auto shadowMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

shadowMapTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB10),
    glm::vec2(shadowMapSize, shadowMapSize),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr);

// copy texture to GPU memory; this could be done once (not on each frame render) to save some time
shadowMapTexture->textureHandle().makeResident();

// at this stage, texture parameters (dimensions, filtering, wrapping, etc.) can not be changed anymore - you would have to create a new texture

// remove texture from GPU memory
shadowMapTexture->textureHandle().makeNonResident();

Rendering to different textures in geometry shader

A neat trick to render scene multiple times to different textures (for instance, rendering into cubemap) is to use geometry shader - you simply bind multi-layer texture (cubemap or a texture array) and use the gl_Layer output variable in the geometry shader to write to a specific texture layer:

C++ program:

struct alignas(16) PointLightData
{
    glm::vec3 lightPosition;
    float farPlane;
    std::array<glm::mat4, 6> projectionViewMatrices;
};

// prepare the shader program with vertex, geometry and fragment shaders
auto pointShadowMappingVertexSource = globjects::Shader::sourceFromFile("media/shadow-mapping-point.vert");
auto pointShadowMappingVertexShaderTemplate = globjects::Shader::applyGlobalReplacements(pointShadowMappingVertexSource.get());
auto pointShadowMappingVertexShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_VERTEX_SHADER), pointShadowMappingVertexShaderTemplate.get());

if (!pointShadowMappingVertexShader->compile())
{
    std::cerr << "[ERROR] Can not compile point shadow mapping vertex shader" << std::endl;
    return 1;
}

auto pointShadowMappingGeometrySource = globjects::Shader::sourceFromFile("media/shadow-mapping-point.geom");
auto pointShadowMappingGeometryShaderTemplate = globjects::Shader::applyGlobalReplacements(pointShadowMappingGeometrySource.get());
auto pointShadowMappingGeometryShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_GEOMETRY_SHADER), pointShadowMappingGeometryShaderTemplate.get());

if (!pointShadowMappingGeometryShader->compile())
{
    std::cerr << "[ERROR] Can not compile point shadow mapping fragment shader" << std::endl;
    return 1;
}

auto pointShadowMappingFragmentSource = globjects::Shader::sourceFromFile("media/shadow-mapping-point.frag");
auto pointShadowMappingFragmentShaderTemplate = globjects::Shader::applyGlobalReplacements(pointShadowMappingFragmentSource.get());
auto pointShadowMappingFragmentShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_FRAGMENT_SHADER), pointShadowMappingFragmentShaderTemplate.get());

if (!pointShadowMappingFragmentShader->compile())
{
    std::cerr << "[ERROR] Can not compile point shadow mapping fragment shader" << std::endl;
    return 1;
}

auto pointShadowMappingProgram = std::make_unique<globjects::Program>();
pointShadowMappingProgram->attach(pointShadowMappingVertexShader.get(), pointShadowMappingGeometryShader.get(), pointShadowMappingFragmentShader.get());

// prepare the cubemap texture

auto pointShadowMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_R), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

pointShadowMapTexture->bind();

const auto shadowMapSize = 2048;

for (auto i = 0; i < 6; ++i)
{
    ::glTexImage2D(
        static_cast<::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_X + i),
        0,
        GL_DEPTH_COMPONENT,
        shadowMapSize,
        shadowMapSize,
        0,
        GL_DEPTH_COMPONENT,
        GL_FLOAT,
        nullptr);
}

pointShadowMapTexture->unbind();

// prepare the framebuffer

auto pointShadowMappingFramebuffer = std::make_unique<globjects::Framebuffer>();
pointShadowMappingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), pointShadowMapTexture.get());

// prepare a list of six view projection matrices

glm::mat4 cameraProjection = glm::perspective(glm::radians(fov), (float)window.getSize().x / (float)window.getSize().y, 0.1f, 100.0f);

glm::mat4 cameraView = glm::lookAt(
    cameraPos,
    cameraPos + cameraForward,
    cameraUp);

const float nearPlane = 0.1f;
const float farPlane = 10.0f;

glm::mat4 pointLightProjection = glm::perspective(glm::radians(90.0f), static_cast<float>(shadowMapSize / shadowMapSize), nearPlane, farPlane);

std::array<glm::mat4, 6> pointLightProjectionViewMatrices{
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(-1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 1.0f, 0.0f), glm::vec3(0.0f, 0.0f, 1.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, -1.0f, 0.0f), glm::vec3(0.0f, 0.0f, -1.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 0.0f, 1.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 0.0f, -1.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
};

PointLightData pointLightData{ pointLightPosition, farPlane, pointLightProjectionViewMatrices };

// store the view projection matrices in SSBO

auto pointLightDataBuffer = std::make_unique<globjects::Buffer>();

pointLightDataBuffer->setData(pointLightData, static_cast<gl::GLenum>(GL_DYNAMIC_COPY));

// ...

// bind framebuffer and shared stored buffer object with parameters

pointLightDataBuffer->bindBase(GL_SHADER_STORAGE_BUFFER, 5);

pointShadowMappingFramebuffer->bind();

pointShadowMappingProgram->use();

// render entire scene

pointShadowMappingProgram->release();

pointShadowMappingFramebuffer->unbind();

geometry shader:

void main()
{
    for (int face = 0; face < 6; ++face)
    {
        gl_Layer = face;

        for (int vertex = 0; vertex < 3; ++vertex)
        {
            fragmentPosition = gl_in[vertex].gl_Position;
            gl_Position = pointLight.projectionViewMatrices[face] * fragmentPosition;
            EmitVertex();
        }

        EndPrimitive();
    }
}

Rendering techniques

Shadow mapping

The most straightforward idea of implementing shadow mapping is: you render a scene depths (each pixel represents a distance from camera to the object) from the perspective of a light source to a separate render target. Then you render your scene normally from the perspective of a camera and for each pixel you compare its distance to the light source (take position of a pixel and subtract position of a light source from it) - if scene pixel is further from the light than the same pixel' depth in the light space - there's some other thing blocking the light, so this pixel is in the shadow.

For this algorithm, you would need a shadow map texture and a framebuffer it is attached to:

auto shadowMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

shadowMapTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB10),
    glm::vec2(2048, 2048),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr);

auto framebuffer = std::make_unique<globjects::Framebuffer>();
framebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), shadowMapTexture.get());
framebuffer->setDrawBuffers({ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), static_cast<gl::GLenum>(GL_NONE) });

auto renderBuffer = std::make_unique<globjects::Renderbuffer>();
renderBuffer->storage(static_cast<gl::GLenum>(GL_DEPTH24_STENCIL8), 2048, 2048);
framebuffer->attachRenderBuffer(static_cast<gl::GLenum>(GL_DEPTH_STENCIL_ATTACHMENT), renderBuffer.get());

Then you render the entire scene to that framebuffer from the position of the light using the orthographic projection and setting the viewport to the shadow map' size:

glm::vec3 lightPosition = glm::vec3(0.0f, 3.0f, 4.0f); // cameraPos;

const float nearPlane = 0.1f;
const float farPlane = 10.0f;
glm::mat4 lightProjection = glm::ortho(-10.0f, 10.0f, -10.0f, 10.0f, nearPlane, farPlane);

glm::mat4 lightView = glm::lookAt(lightPosition, glm::vec3(0.0f, 0.0f, 0.0f), glm::vec3(0.0f, 1.0f, 0.0f));

glm::mat4 lightSpaceMatrix = lightProjection * lightView;

::glViewport(0, 0, 2048, 2048);

framebuffer->bind();

::glClearColor(static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(1.0f));
::glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// same as
// framebuffer->clearBuffer(static_cast<gl::GLenum>(GL_COLOR_BUFFER_BIT | GL_DEPTH), 0, glm::vec4(1.0f));

glEnable(GL_DEPTH_TEST);
glEnable(GL_CULL_FACE);

// cull front faces to prevent peter panning the generated shadow map
glCullFace(GL_FRONT);

shadowMappingProgram->use();

shadowMappingProgram->setUniform("lightSpaceMatrix", lightSpaceMatrix);
shadowMappingProgram->setUniform("model", chickenModel->getTransformation());

// render scene

framebuffer->unbind();

shadowMappingProgram->release();

glEnable(GL_CULL_FACE);
glCullFace(GL_BACK);

Finally, you restore the viewport to the window (screen) size and render the scene, taking data from shadow map into consideration:

::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));
::glClearColor(static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(1.0f));
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

shadowRenderingProgram->use();

shadowRenderingProgram->setUniform("lightPosition", lightPosition);
shadowRenderingProgram->setUniform("lightColor", glm::vec3(1.0, 1.0, 1.0));
// ambientColorUniform->set(glm::vec3(1.0f, 1.0f, 1.0f));
// materialSpecularUniform->set(12.0f);
shadowRenderingProgram->setUniform("cameraPosition", cameraPos);

shadowRenderingProgram->setUniform("projection", cameraProjection);
shadowRenderingProgram->setUniform("view", cameraView);
shadowRenderingProgram->setUniform("lightSpaceMatrix", lightSpaceMatrix);

// render the scene again

shadowMapTexture->unbindActive(0);

shadowRenderingProgram->release();

Shadow mapping vertex shader:

#version 410

layout (location = 0) in vec3 vertexPosition;

out gl_PerVertex
{
    vec4 gl_Position;
};

uniform mat4 lightSpaceMatrix;
uniform mat4 model;

void main()
{
    gl_Position = lightSpaceMatrix * model * vec4(vertexPosition, 1.0);
}

Shadow mapping fragment shader:

#version 410

void main()
{
    gl_FragDepth = gl_FragCoord.z;
}

Final rendering pass vertex shader:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec4 fragmentPositionInLightSpace;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;
uniform mat4 lightSpaceMatrix;

void main()
{
    vsOut.fragmentPosition = vec3(model * vec4(vertexPosition, 1.0));
    vsOut.normal = vertexNormal;
    vsOut.textureCoord = vertexTextureCoord;
    vsOut.fragmentPositionInLightSpace = lightSpaceMatrix * model * vec4(vertexPosition, 1.0);

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

Final rendering fragment shader:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec4 fragmentPositionInLightSpace;
} fsIn;

uniform sampler2D shadowMap;
uniform sampler2D diffuseTexture;

uniform vec3 lightPosition;
uniform vec3 lightColor;
uniform vec3 cameraPosition;

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    vec2 shadowMapCoord = (fsIn.fragmentPositionInLightSpace.xy / fsIn.fragmentPositionInLightSpace.w) * 0.5 + 0.5;
    float occluderDepth = texture(shadowMap, shadowMapCoord.xy).r;
    float thisDepth = fsIn.fragmentPositionInLightSpace.z / fsIn.fragmentPositionInLightSpace.w;

    if (thisDepth > 1.0)
    {
        return 0.0;
    }

    return 1.0;
}

void main()
{
    vec3 color = texture(diffuseTexture, fsIn.textureCoord).rgb;
    vec3 normal = normalize(fsIn.normal);

    // ambient
    vec3 ambient = 0.3 * color;

    // diffuse
    vec3 lightDirection = normalize(lightPosition - fsIn.fragmentPosition);
    float diff = max(dot(lightDirection, normal), 0.0);
    vec3 diffuse = diff * lightColor;

    // specular
    vec3 viewDirection = normalize(cameraPosition - fsIn.fragmentPosition);
    vec3 halfwayDirection = normalize(lightDirection + viewDirection);
    float spec = pow(max(dot(normal, halfwayDirection), 0.0), 64.0);
    vec3 specular = spec * lightColor;

    // calculate shadow
    float shadow = shadowCalculation(normal, lightDirection);

    vec3 lighting = ((shadow * (diffuse + specular)) + ambient) * color;

    fragmentColor = vec4(lighting, 1.0);
}

This technique, however, introduces a number of artifacts.

The simplest and easily solved issue is the rough edge of the shadow where there should be none, caused by the bounds of the light projection matrix. One can simply add a check in the shader to see if the pixel is out of the shadow map bounds and light those pixels by default.

Shadow aliasing - when you are trying to render a whole lot of objects on a small texture, shadow edges have stair-like shape instead of smooth lines. The bigger the shadow map, the more cube-like shadows will become. These artifacts are addressed by a number of different techniques, like cascaded shadow maps (aka parallel-slice aka parallel-split shadow mapping).

Sometimes you might see lines of shadows where there should be none. This is caused by the lack of precision of the shadow map - the depth values simply do not have enough data to represent that half-of-a-pixel depth. This could be somewhat mitigated by adding a small factor to the depth sampled from a shadow map. But such a solution presents a yet new artifact - peter-panning, when the objects rendered "fly" over a surface beneath them. Sometimes light might "bleed" through the objects, lighting the surfaces where they should be in the shadow. This artifact is also known as "light bleeding".

There are other algorithms addressing the above artifacts - PCF and variance shadow mapping.

Biasing

This technique fixes the aliasing issue by "offsetting" the surface the shadow is cast upon by some small number (realistically you do not offset anything - you just adjust the calculations).

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    vec2 shadowMapCoord = (fsIn.fragmentPositionInLightSpace.xy / fsIn.fragmentPositionInLightSpace.w) * 0.5 + 0.5;
    float occluderDepth = texture(shadowMap, shadowMapCoord.xy).r;
    float thisDepth = fsIn.fragmentPositionInLightSpace.z / fsIn.fragmentPositionInLightSpace.w;

    if (thisDepth > 1.0)
    {
        return 0.0;
    }

    // this biasing fixes aliasing artifact, but introduces peter-panning
    float bias = max(0.05 * (1.0 - dot(normal, lightDirection)), 0.005);

    return thisDepth - bias < occluderDepth ? 1.0 : 0.0;
}

This fixes the aliasing, indeed, but introduces the peter-panning effect, making object appear flying or hovering over the surface upon close inspection.

Percentage-close filtering

Percentage-close filtering, or PCF, is a technique of smoothing out those hard-edge shadows. Instead of calculating whether the pixel (say, on a surface) is in shadow or not you calculate the median of all the neighbours of an are around that pixel (say, 5x5 pixels).

PCF is implemented in the final shadow rendering fragment shader:

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    vec2 shadowMapCoord = (fsIn.fragmentPositionInLightSpace.xy / fsIn.fragmentPositionInLightSpace.w) * 0.5 + 0.5;
    float occluderDepth = texture(shadowMap, shadowMapCoord.xy).r;
    float thisDepth = fsIn.fragmentPositionInLightSpace.z / fsIn.fragmentPositionInLightSpace.w;

    if (thisDepth > 1.0)
    {
        return 0.0;
    }

    // this biasing fixes aliasing artifact, but introduces peter-panning
    float bias = max(0.05 * (1.0 - dot(normal, lightDirection)), 0.005);

    // PCF
    float shadow = 0.0;
    vec2 texelSize = 1.0 / textureSize(shadowMap, 0);

    for (int x = -1; x <= 1; ++x)
    {
        for (int y = -1; y <= 1; ++y)
        {
            float pcfDepth = texture(shadowMap, shadowMapCoord.xy + vec2(x, y) * texelSize).r;

            shadow += thisDepth - bias < pcfDepth  ? 1.0 : 0.0;
        }
    }

    shadow /= 9.0;

    return shadow;
}

Variance shadow mapping

Variance shadow mapping is yet another technique of smoothing hard shadows. Unlike PCF, it has constant computation time. The idea is that you calculate the probability of a pixel being in shadow. If the probability is high enough - you treat it as if it was in the shadow. For the calculations, the two numbers (called "moments") are calculated during the shadow mapping, essentially being a function of the pixel depth (in light space) and a function of the same depth, squared. During the final scene rendering, you substitute the values in Chebyshev's inequality to get the probability.

The implementation is as simple as having the following shadow mapping fragment shader:

float linearizeDepth(float depth)
{
    float nearPlane = 1.0;
    float farPlane = 100.0;

    return (2.0 * nearPlane) / (farPlane + nearPlane - depth * (farPlane - nearPlane));
}

float linstep(float _min, float _max, float v)
{
    return clamp((v - _min) / (_max - _min), 0, 1);
}

float reduceLightBleeding(float p_max, float Amount)
{
    // Remove the [0, Amount] tail and linearly rescale (Amount, 1].
    return linstep(Amount, 1, p_max);
}

// compute an upper bound on the probability that the currently shaded surface (at depth t) is occluded
float ChebyshevUpperBound(vec2 moments, float t)
{
    // One-tailed inequality valid if t > Moments.x
    if (t <= moments.x)
    {
        return 1.0;
    }

    // Compute variance
    float variance = moments.y - (moments.x * moments.x);
    variance = max(variance, 0.001);

    // Compute probabilistic upper bound
    float d = t - moments.x;
    return variance / (variance + d * d);
}

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    vec3 shadowMapCoord = (fsIn.fragmentPositionInLightSpace.xyz / fsIn.fragmentPositionInLightSpace.w) * 0.5 + 0.5;
    vec3 shadowMapSample = texture(shadowMap, shadowMapCoord.xy).rgb;
    vec2 moments = shadowMapSample.xy;
    float fragmentDepth = shadowMapCoord.z; // alternatively, use linearizeDepth(shadowMapCoord.z);

    // Compute the Chebyshev upper bound.
    float p = step(fragmentDepth, moments.x);
    float mu = moments.x;
    float sigma2 = max(moments.y - moments.x * moments.x, 0.0002);

    float d = fragmentDepth - mu;
    float p_max = sigma2 / (sigma2 + (d * d));
    p_max = clamp(max(p, p_max), 0, 1);

    return reduceLightBleeding(p_max, 0.5);
}

Cascaded shadow mapping

This is rather quality- and performance-optimization technique. It could also be used for not just shadow mapping, but for various objects rendering (like rendering grass, small terrain details, terrain itself, etc.). The idea is that there is no need to render high-detailed shadows further from the camera. Assume you have a large landscape. If you render detailed shadows for an entire landscape to rather limited texture - you will end up having each object have big pixelated shadow. Whilst you want objects closer to the camera look sharp and nice. And the trees in the distance - well, there is no need for them to have that nice shadows.

Thus, you "divide" your camera space into few sections (aka "cascades") and render shadows for each of those cascade separately, using the same size of the shadow map texture. The trick is that the cascade closest to the camera is the smallest, so the shadows have very high level of detail. The section of a camera space a bit further away is larger, but since the shadow map has the same size - the quality of shadows is lower. The furthest section of the camera space has the largest size, so the quality of the shadows in the same size of the shadow map will be the worst. But since the objects in that section are also furthest away from the camera - they don't have to have the most detailed shadows. In fact, you can go even further and only split half of camera space into three (or more, if you will) cascades, leaving the other half, the furthest one from the camera, completely out of the shadow mapping process. This way the objects far away won't even have any shadows at all.

For the most part, the logic is implemented in the main application, where you have to set up the projection matrices for the shadow mapping and run the shadow mapping rendering stage multiple times (for each of the projection matrix).

Alternatively, this could be used in conjunction with other two optimization techniques - SSBO and geometry shader - you send the projection matrices to the shader via SSBO and then instead of issuing rendering commands multiple times from the application side (CPU), everything is done on the GPU, without the need to switch back and forth between shader context and application context (roughly put - without the need to formally finish the rendering, let the CPU / application know about this and make it issue a new bunch of rendering commands, which essentially will be the same - just the projection matrix will be different).

Setting up shadow mapping shaders:

auto shadowMappingVertexSource = globjects::Shader::sourceFromFile("media/shadow-mapping.vert");
auto shadowMappingVertexShaderTemplate = globjects::Shader::applyGlobalReplacements(shadowMappingVertexSource.get());
auto shadowMappingVertexShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_VERTEX_SHADER), shadowMappingVertexShaderTemplate.get());

if (!shadowMappingVertexShader->compile())
{
    std::cerr << "[ERROR] Can not compile vertex shader" << std::endl;
    return 1;
}

auto shadowMappingGeometrySource = globjects::Shader::sourceFromFile("media/shadow-mapping.geom");
auto shadowMappingGeometryShaderTemplate = globjects::Shader::applyGlobalReplacements(shadowMappingGeometrySource.get());
auto shadowMappingGeometryShader = std::make_unique<globjects::Shader>(gl::GL_GEOMETRY_SHADER, shadowMappingGeometryShaderTemplate.get());

if (!shadowMappingGeometryShader->compile())
{
    std::cerr << "[ERROR] Can not compile geometry shader" << std::endl;
    return 1;
}

auto shadowMappingFragmentSource = globjects::Shader::sourceFromFile("media/shadow-mapping.frag");
auto shadowMappingFragmentShaderTemplate = globjects::Shader::applyGlobalReplacements(shadowMappingFragmentSource.get());
auto shadowMappingFragmentShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_FRAGMENT_SHADER), shadowMappingFragmentShaderTemplate.get());

if (!shadowMappingFragmentShader->compile())
{
    std::cerr << "[ERROR] Can not compile fragment shader" << std::endl;
    return 1;
}

auto shadowMappingProgram = std::make_unique<globjects::Program>();

shadowMappingProgram->attach(shadowMappingVertexShader.get(), shadowMappingGeometryShader.get(), shadowMappingFragmentShader.get());

auto shadowMappingModelTransformationUniform = shadowMappingProgram->getUniform<glm::mat4>("modelTransformation");
auto shadowMappingLightViewProjectionMatrices = shadowMappingProgram->getUniform<std::vector<glm::mat4>>("lightViewProjectionMatrix");
auto lightViewProjectionMatricesUniform = shadowMappingProgram->getUniform<std::vector<glm::mat4>>("lightViewProjectionMatrix");

And a few constants:

std::vector<glm::mat4> lightViewProjectionMatrices;
std::vector<float> splitDepths;
const std::vector<float> splits{ { 0.0f, 0.05f, 0.2f, 0.5f, 1.0f } };

// these vertices define view frustum in screen space coordinates
constexpr std::array<glm::vec3, 8> _cameraFrustumSliceCornerVertices{
    {
        { -1.0f, -1.0f, -1.0f }, { 1.0f, -1.0f, -1.0f }, { 1.0f, 1.0f, -1.0f }, { -1.0f, 1.0f, -1.0f },
        { -1.0f, -1.0f, 1.0f }, { 1.0f, -1.0f, 1.0f }, { 1.0f, 1.0f, 1.0f }, { -1.0f, 1.0f, 1.0f },
    }
};

Then, the shadow mapping stage:

{
    lightViewProjectionMatrices.clear();
    splitDepths.clear();

    glm::mat4 proj = glm::inverse(cameraProjection * cameraView);

    std::array<glm::vec3, 8> _entireFrustum{};

    std::transform(
        _cameraFrustumSliceCornerVertices.begin(),
        _cameraFrustumSliceCornerVertices.end(),
        _entireFrustum.begin(),
        [proj](glm::vec3 p) {
            glm::vec4 v = proj * glm::vec4(p, 1.0f);
            return glm::vec3(v) / v.w;
        }
    );

    std::array<glm::vec3, 4> _frustumEdgeDirections {};

    for (auto i = 0; i < 4; ++i)
    {
        _frustumEdgeDirections[i] = glm::normalize(_entireFrustum[4 + i] - _entireFrustum[i]);
    }

    const float _depth = farPlane - nearPlane;

    for (auto splitIdx = 1; splitIdx < splits.size(); ++splitIdx)
    {
        // frustum slice vertices in world space
        std::array<glm::vec3, 8> _frustumSliceVertices;

        for (auto t = 0; t < 4; ++t)
        {
            _frustumSliceVertices[t] = _entireFrustum[t] + _frustumEdgeDirections[t] * _depth * splits[splitIdx - 1];
            _frustumSliceVertices[4 + t] = _entireFrustum[t] + _frustumEdgeDirections[t] * _depth * splits[splitIdx];
        }

        // TODO: also check if camera is looking towards the light

        glm::vec3 _frustumSliceCenter(0.0f);

        for (auto p : _frustumSliceVertices)
        {
            _frustumSliceCenter += p;
        }

        _frustumSliceCenter /= 8.0f;

        glm::vec3 _frustumRadiusVector(0.0f);

        for (auto p : _frustumSliceVertices)
        {
            auto v = p - _frustumSliceCenter;

            if (glm::length(_frustumRadiusVector) < glm::length(v))
            {
                _frustumRadiusVector = v;
            }
        }

        // calculate new light projection
        glm::vec3 _forward = glm::normalize(_lightDirection);
        glm::vec3 _right = glm::cross(_forward, glm::vec3(0.0f, 1.0f, 0.0f));
        glm::vec3 _up(0.0f, 1.0f, 0.0f);
        glm::mat4 _lightView = glm::lookAt(_frustumSliceCenter - glm::normalize(_lightDirection) * glm::length(_frustumRadiusVector), _frustumSliceCenter, _up);

        const float _frustumRadius = glm::length(_frustumRadiusVector);

        glm::mat4 _lightProjectionViewMatrix = glm::ortho(
            _frustumSliceCenter.x - _frustumRadius,
            _frustumSliceCenter.x + _frustumRadius,
            _frustumSliceCenter.y - _frustumRadius,
            _frustumSliceCenter.y + _frustumRadius,
            0.0f,
            _frustumSliceCenter.z + 2.0f * _frustumRadius
        ) * _lightView;

        lightViewProjectionMatrices.push_back(_lightProjectionViewMatrix);

        splitDepths.push_back(_depth * splits[splitIdx] * 0.7f);
    }

    shadowMappingLightViewProjectionMatrices->set(lightViewProjectionMatrices);
    shadowRenderingLightViewProjectionsUniform->set(lightViewProjectionMatrices);
    shadowRenderingSplitsUniform->set(splitDepths);
}

Then geometry shader would be:

#version 410

layout(triangles) in;

/*
 this shader will be emitting data to 4 shadow map layers as per 4 frustum splits;
 each triangle fed into this geometry shader will be projected into each shadow map layer;
 hence this shader will emit 4 layers * 3 vertices (per each input primitive / geometry) = 12 vertices;
 note how the input vertices are in the world space, so we need to project them to the clip space in here
*/
layout(triangle_strip, max_vertices = 12) out;

out gl_PerVertex
{
  vec4 gl_Position;
};

// out int gl_Layer;

uniform mat4 lightViewProjectionMatrix[4]; // as per 4 frustum splits

void main()
{
    for (int split = 0; split < 4; ++split)
    {
        // input primitive index
        for (int i = 0; i < gl_in.length(); ++i)
        {
            // in a 3D texture, which layer do we project our input primitive to
            gl_Layer = split;

            // project an input primitive using the corresponding projection matrix from the uniform
            gl_Position = lightViewProjectionMatrix[split] * gl_in[i].gl_Position;

            EmitVertex();
        }

        EndPrimitive();
    }
}

The rendering stage would use the projection matrices too. So setting up the rendering shader:

auto shadowRenderingFragmentShaderSource = globjects::Shader::sourceFromFile("media/shadow-rendering.frag");
auto shadowRenderingFragmentShaderTemplate = globjects::Shader::applyGlobalReplacements(shadowRenderingFragmentShaderSource.get());
auto shadowRenderingFragmentShader = std::make_unique<globjects::Shader>(static_cast<gl::GLenum>(GL_FRAGMENT_SHADER), shadowRenderingFragmentShaderTemplate.get());

if (!shadowRenderingFragmentShader->compile())
{
    std::cerr << "[ERROR] Can not compile chicken fragment shader" << std::endl;
    return 1;
}

auto shadowRenderingProgram = std::make_unique<globjects::Program>();
shadowRenderingProgram->attach(shadowRenderingVertexShader.get(), shadowRenderingFragmentShader.get());

auto shadowRenderingModelTransformationUniform = shadowRenderingProgram->getUniform<glm::mat4>("model");
auto shadowRenderingViewTransformationUniform = shadowRenderingProgram->getUniform<glm::mat4>("view");
auto shadowRenderingProjectionTransformationUniform = shadowRenderingProgram->getUniform<glm::mat4>("projection");

auto shadowRenderingLightPositionUniform = shadowRenderingProgram->getUniform<glm::vec3>("lightPosition");
auto shadowRenderingLightColorUniform = shadowRenderingProgram->getUniform<glm::vec3>("lightColor");
auto shadowRenderingCameraPositionUniform = shadowRenderingProgram->getUniform<glm::vec3>("cameraPosition");

auto shadowRenderingLightViewProjectionsUniform = shadowRenderingProgram->getUniform<std::vector<glm::mat4>>("lightViewProjections");
auto shadowRenderingSplitsUniform = shadowRenderingProgram->getUniform<std::vector<float>>("splits"); // distance from camera to zFar, e.g. (far - cameraPosition.z) * splitFraction

And finally, the fragment shader for the final rendering stage:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec4 viewPosition;
    vec4 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

uniform mat4 lightViewProjections[4];
uniform float splits[4];

uniform sampler2DArray shadowMaps;
uniform sampler2D diffuseTexture;

uniform vec3 lightPosition;
uniform vec3 lightColor;
uniform vec3 cameraPosition;

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    float cameraViewDepth = fsIn.viewPosition.z;

    for (int i = 0; i < 4; ++i)
    {
        if (cameraViewDepth < splits[i])
        {
            vec4 fragmentPositionInLightSpace = lightViewProjections[i] * fsIn.fragmentPosition;
            vec3 shadowPos1 = fragmentPositionInLightSpace.xyz / fragmentPositionInLightSpace.w;
            vec3 shadowMapCoord = shadowPos1 * 0.5 + 0.5;
            float thisDepth = shadowMapCoord.z;

            if (thisDepth > 1.0)
            {
                continue;
            }

            float occluderDepth = texture(shadowMaps, vec3(shadowMapCoord.xy, i)).r;

            float bias = max(0.05 * (1.0 - dot(normal, lightDirection)), 0.005);

            return thisDepth - bias > occluderDepth ? 0.25 : 1.0;
        }
    }

    return 0.0;
}

void main()
{
    vec3 color = texture(diffuseTexture, fsIn.textureCoord).rgb;
    vec3 normal = normalize(fsIn.normal);

    // ambient
    vec3 ambient = 0.3 * color;

    // diffuse
    vec3 lightDirection = normalize(lightPosition - vec3(fsIn.fragmentPosition) / fsIn.fragmentPosition.w);
    float diff = max(dot(lightDirection, normal), 0.0);
    vec3 diffuse = diff * lightColor;

    // specular
    vec3 viewDirection = normalize(cameraPosition - vec3(fsIn.fragmentPosition) / fsIn.fragmentPosition.w);
    vec3 halfwayDirection = normalize(lightDirection + viewDirection);
    float spec = pow(max(dot(normal, halfwayDirection), 0.0), 64.0);
    vec3 specular = spec * lightColor;

    // calculate shadow
    float shadow = shadowCalculation(normal, lightDirection);

    vec3 lighting = ((shadow * (diffuse + specular)) + ambient) * color;

    fragmentColor = vec4(lighting, 1.0);
}

Note how now the shadow calculation function makes use of the projection matrices, so instead of rendering the scene multiple times again, we simply pick the shadow data from a corresponding shadow map buffer.

Anti-aliasing

You might have noticed that rendering scene from the framebuffer results in most lines looking like stairs, with very distinct hard edges around every shape. This is most obvious during the deferred rendering.

In order to prevent (or rather somewhat mitigate) this issue, the technique called "anti-aliasing" is used. It softens the pixels around those hard edges so they gradually change color instead of hard jump.

Multi-sampled anti-aliasing (MSAA)

I have never understood the concept of multi-resolution pyramid (even when working on my second M.Sc. thesis) until I got myself into all this OpenGL stuff. The way you can soften the rapid change of a color in neighbour pixels is to upscale the image image few times and then downscale it back. This way you will loose a lot of detail, getting an average color value for each scaling level. OpenGL actually comes bundled with the multi-sampled rendering feature - when you create a framebuffer you can specify the number of samples for texture.

But you can implement one by yourself, explicitly, using one rather simple shader program and a deferred rendering technique.

First rendering pass would be rendering a scene to a frame buffer. The last (or second, in this overly-simplified example) rendering pass would run on top of that frame buffer and compute the new one using the simple shader program:

vertex shader for the MSAA last rendering pass:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec2 textureCoord;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

void main()
{
    vsOut.fragmentPosition = vertexPosition;
    vsOut.textureCoord = vec2(1.0 - vertexTextureCoord.x, vertexTextureCoord.y);

    gl_Position = vec4(vertexPosition, 1.0);
}

fragment shader for the last rendering pass:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec2 textureCoord;
} fsIn;

uniform sampler2D diffuseTexture;

vec4 msaa( sampler2D tex, vec2 uv ) {
    vec2 resolution = textureSize(tex, 0);

    vec4 singleSample = texture(tex, uv);

    float a = (3.0 / 8.0) * (1.0 / resolution.x);
    float b = (1.0 / 8.0) * (1.0 / resolution.y);

    vec4 acc = vec4(0.0);

    acc += texture(tex, (uv + vec2(-a, b)));
    acc += texture(tex, (uv + vec2(a, -b)));
    acc += texture(tex, (uv + vec2(-b, -a)));
    acc += texture(tex, (uv + vec2(b, a)));
    acc /= 4.0;

    // vec4 color = pow(acc, vec4(1.0 / 2.2));

    return acc;
}

void main()
{
    vec4 color = msaa(diffuseTexture, fsIn.textureCoord);

    fragmentColor = color;
}

Fast-approximation anti-aliasing (FXAA)

Multi-sampled rendering is fine, but usually it comes at a cost of higher memory consumption - for each multi-sampled texture you have to allocate memory. The more multi-sampled textures you have - the more memory your application consumes.

Fast approximation works at constant time and memory and thus is one of the most popular anti-aliasing algorithms used. It is also quite easy to implement, which makes it all more appealing.

The algorithm actually detects the edges on the image and then blends the colors in the direction perpendicular to the edge.

That sounds easy on paper, but how does it actually detect the edges? The algorithm calculates the luminance of the pixels surrounding a given central pixel by calculating a dot product of each pixel' color and a constant luminance vector. It then compares the luminance in two directions vertically and two directions horizontally to find the edge' direction.

The only substantial difference in the code for FXAA is in the fragment shader for the last rendering pass:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec2 textureCoord;
} fsIn;

uniform sampler2D diffuseTexture;

#define FXAA_SPAN_MAX 16.0
#define FXAA_REDUCE_MUL   (1.0 / FXAA_SPAN_MAX)
#define FXAA_REDUCE_MIN   (1.0 / 128.0)
#define FXAA_SUBPIX_SHIFT (1.0 / 8.0)

vec4 fxaa( sampler2D tex, vec2 uv2 )
{
    vec2 res = textureSize(tex, 0);
    vec2 rcpFrame = 1. / res;

    vec4 uv = vec4( uv2, uv2 - (rcpFrame * (0.5 + FXAA_SUBPIX_SHIFT)));

    vec3 rgbNW = texture(tex, uv.zw).xyz;
    vec3 rgbNE = texture(tex, uv.zw + vec2(1, 0) * rcpFrame.xy).xyz;
    vec3 rgbSW = texture(tex, uv.zw + vec2(0, 1) * rcpFrame.xy).xyz;
    vec3 rgbSE = texture(tex, uv.zw + vec2(1, 1) * rcpFrame.xy).xyz;
    vec4 texColor = texture(tex, uv.xy);
    vec3 rgbM  = texColor.xyz;

    vec3 luma = vec3(0.299, 0.587, 0.114);
    float lumaNW = dot(rgbNW, luma);
    float lumaNE = dot(rgbNE, luma);
    float lumaSW = dot(rgbSW, luma);
    float lumaSE = dot(rgbSE, luma);
    float lumaM  = dot(rgbM,  luma);

    float lumaMin = min(lumaM, min(min(lumaNW, lumaNE), min(lumaSW, lumaSE)));
    float lumaMax = max(lumaM, max(max(lumaNW, lumaNE), max(lumaSW, lumaSE)));

    vec2 dir;
    dir.x = -((lumaNW + lumaNE) - (lumaSW + lumaSE));
    dir.y =  ((lumaNW + lumaSW) - (lumaNE + lumaSE));

    float dirReduce = max(
        (lumaNW + lumaNE + lumaSW + lumaSE) * (0.25 * FXAA_REDUCE_MUL),
        FXAA_REDUCE_MIN
    );

    float rcpDirMin = 1.0/(min(abs(dir.x), abs(dir.y)) + dirReduce);

    dir = min(
        vec2( FXAA_SPAN_MAX,  FXAA_SPAN_MAX),
        max(
            vec2(-FXAA_SPAN_MAX, -FXAA_SPAN_MAX),
            dir * rcpDirMin
        )
    ) * rcpFrame.xy;

    vec3 rgbA = (1.0/2.0) * (
        texture(tex, uv.xy + dir * (1.0 / 3.0 - 0.5)).xyz +
        texture(tex, uv.xy + dir * (2.0 / 3.0 - 0.5)).xyz);

    vec3 rgbB = rgbA * (1.0 / 2.0) + (1.0 / 4.0) * (
        texture(tex, uv.xy + dir * (0.0 / 3.0 - 0.5)).xyz +
        texture(tex, uv.xy + dir * (3.0 / 3.0 - 0.5)).xyz);

    float lumaB = dot(rgbB, luma);

    if ((lumaB < lumaMin) || (lumaB > lumaMax))
        return vec4(rgbA, texColor.a);
    else
        return vec4(rgbB, texColor.a);
}

void main()
{
    vec4 color = fxaa(diffuseTexture, fsIn.textureCoord);

    fragmentColor = color;
}

Screen-space ambient occlusion (SSAO)

As you might know, rendering big and complex scenes is expensive in both time, memory and computational resources. It is not always desirable or even possible to render all effects like reflections and shadows for each and every polygon of the scene.

To address this issue, techniques like ambient occlusion (specifically, screen-space ambient occlusion, in that they only work in screen space, combined with deferred rendering, as a post-processing step) exist. This technique in particular allows simulating the shadow cast by the objects by darkening the pixels which are occluded by other objects from the perspective of light.

Sounds very much like shadow mapping, doesn't it? But the big difference here is that in shadow mapping you need to render an entire scene from the perspective of each light source. Even if you trim the scene to only the objects visible by the light (which is a quite nice optimization, but for the most part it is also expensive), you will still have to render every polygon.

In contrast, ambient occlusion is calculated for a fixed number of samples around a given pixel. On top of that, the pixels we analyze are only in screen space. On top of that, we do not need to render all the geometry around a given pixel - we re-use the position information rendered as one of the attributes in first pass of a deferred rendering. To reduce the cost of this algorithm even further, you do not need to render anything from the light perspective - you just calculate the sample rays of light for each light, knowing its position and direction.

You will need to store the position of each fragment in camera space (yes, camera space, not light space). Then, for each pixel, you iterate each source of light to calculate the direction of light and samples for a given pixel' position in camera space. You assume you have a sphere of a given constant radius around the pixel' position and you generate a certain constant number of random samples within that sphere. Since the samples are also in the camera space, you can use these positions as-is. Given the information about positions of neighbour pixels in camera space, you then compare those positions to the samples within the sphere - if the sample's position is further from the camera than the neighbour pixel's position - this sample is occluded by some polygon. You then calculate the ratio of occluded samples to the non-occluded ones and based on that give the original pixel its shadow value.

[learnopengl] SSAO

Setting up the initial render pass framebuffer is a bit tricky - you will need multiple attachments for each of the data component:

auto deferredFragmentPositionTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentPositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentPositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentPositionTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredFragmentNormalTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentNormalTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentNormalTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentNormalTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredFragmentAlbedoTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentAlbedoTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentAlbedoTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentAlbedoTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGBA8),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_UNSIGNED_BYTE),
    nullptr
);

auto deferredFragmentDepthTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentDepthTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_DEPTH_COMPONENT),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_DEPTH_COMPONENT),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredRenderingFramebuffer = std::make_unique<globjects::Framebuffer>();
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), deferredFragmentPositionTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1), deferredFragmentNormalTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT2), deferredFragmentAlbedoTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), deferredFragmentDepthTexture.get());

// tell framebuffer it actually needs to render to **BOTH** textures, but does not have to output anywhere (last NONE argument, iirc)
deferredRenderingFramebuffer->setDrawBuffers({
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0),
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1),
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT2),
    static_cast<gl::GLenum>(GL_NONE)
});

deferredRenderingFramebuffer->printStatus(true);

You will also need two buffers for blur:

auto temporaryTexture1 = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

temporaryTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
temporaryTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

temporaryTexture1->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGBA8),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_UNSIGNED_BYTE),
    nullptr
);

auto temporaryFramebuffer = std::make_unique<globjects::Framebuffer>();

temporaryFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), temporaryTexture1.get());
temporaryFramebuffer->setDrawBuffers({ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), static_cast<gl::GLenum>(GL_NONE) });

temporaryFramebuffer->printStatus(true);

auto temporaryTexture2 = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

temporaryTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
temporaryTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

temporaryTexture2->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGBA8),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_UNSIGNED_BYTE),
    nullptr
);

auto temporaryFramebuffer2 = std::make_unique<globjects::Framebuffer>();

temporaryFramebuffer2->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), temporaryTexture2.get());
temporaryFramebuffer2->setDrawBuffers({ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), static_cast<gl::GLenum>(GL_NONE) });

temporaryFramebuffer2->printStatus(true);

And the random data (vector offsets) for the SSAO algorithm, which we will generate in the application and store in a small texture:

std::uniform_real_distribution<float> randomFloats(0.0, 1.0); // random floats between [0.0, 1.0]
std::default_random_engine generator;

const auto ssaoKernelSamples = 64;

std::vector<glm::vec3> ssaoKernel;

for (unsigned int i = 0; i < ssaoKernelSamples; ++i)
{
    glm::vec3 sample(
        randomFloats(generator) * 2.0f - 1.0f,
        randomFloats(generator) * 2.0f - 1.0f,
        randomFloats(generator)
    );

    sample = glm::normalize(sample);
    sample *= randomFloats(generator);

    float scale = static_cast<float>(i) / static_cast<float>(ssaoKernelSamples);

    // lerp(a, b, f) = a + f * (b - a);
    // scale = lerp(0.1f, 1.0f, scale * scale);
    scale = 0.1f + (scale * scale * (1.0f - 0.1f));

    sample *= scale;

    ssaoKernel.push_back(sample);
}

std::vector<glm::vec3> ssaoNoise;

for (unsigned int i = 0; i < 16; i++)
{
    glm::vec3 noise(
        randomFloats(generator) * 2.0f - 1.0f,
        randomFloats(generator) * 2.0f - 1.0f,
        0.0f
    );

    ssaoNoise.push_back(noise);
}

auto ssaoNoiseTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

ssaoNoiseTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_NEAREST));
ssaoNoiseTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_NEAREST));
ssaoNoiseTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<GLint>(GL_REPEAT));
ssaoNoiseTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<GLint>(GL_REPEAT));

ssaoNoiseTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(4.0f, 4.0f),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    &ssaoNoise[0]
);

auto ssaoKernelTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_1D));

ssaoKernelTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_NEAREST));
ssaoKernelTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_NEAREST));

ssaoKernelTexture->image1D(
    0,
    static_cast<gl::GLenum>(GL_RGBA16F),
    static_cast<float>(ssaoKernelSamples),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    &ssaoKernel[0]
);

With the rendering itself, there are no tricks - you have three passes.

On the first pass you bind the deferred rendering buffer (with a few attachments, that is) and render scene as normal. The textures attached to the deferred rendering buffer will be populated with the data.

This data is then used on the second rendering pass to calculate the dark edges (amboent occlusion) on the frame rendered (aka in the screen space) and blur them.

On the last rendering pass you combine the framebuffers to obtain the final image:

// first render pass - prepare for deferred rendering by rendering to the entire scene to a deferred rendering framebuffer's attachments
{
    deferredRenderingFramebuffer->bind();

    deferredRenderingPrePassProgram->use();

    // render scene

    deferredRenderingPrePassProgram->release();

    deferredRenderingFramebuffer->unbind();
}

// second render pass - calculate & blur the ambient occlusion
{
    temporaryFramebuffer->bind();

    ::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));
    ::glClearColor(static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(1.0f));
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    ssaoProgram->use();

    deferredFragmentPositionTexture->bindActive(2);
    deferredFragmentNormalTexture->bindActive(3);

    ssaoNoiseTexture->bindActive(5);
    ssaoKernelTexture->bindActive(6);

    ssaoProgram->setUniform("positionTexture", 2);
    ssaoProgram->setUniform("normalTexture", 3);
    ssaoProgram->setUniform("albedoTexture", 4);

    ssaoProgram->setUniform("ssaoNoiseTexture", 5);
    ssaoProgram->setUniform("ssaoKernelTexture", 6);

    ssaoProgram->setUniform("cameraPosition", cameraPos);
    ssaoProgram->setUniform("projection", cameraProjection);
    ssaoProgram->setUniform("view", cameraView);

    quadModel->bind();
    quadModel->draw();
    quadModel->unbind();

    deferredFragmentPositionTexture->unbindActive(2);
    deferredFragmentNormalTexture->unbindActive(3);

    ssaoNoiseTexture->unbindActive(5);
    ssaoKernelTexture->unbindActive(6);

    ssaoProgram->release();

    temporaryFramebuffer->unbind();

    // blur
    temporaryFramebuffer->bind(static_cast<gl::GLenum>(GL_READ_FRAMEBUFFER));
    temporaryFramebuffer2->bind(static_cast<gl::GLenum>(GL_DRAW_FRAMEBUFFER));

    temporaryFramebuffer->blit(
        static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0),
        std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
        temporaryFramebuffer2.get(),
        std::vector<gl::GLenum>{ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0) },
        std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
        static_cast<gl::ClearBufferMask>(GL_COLOR_BUFFER_BIT),
        static_cast<gl::GLenum>(GL_NEAREST));

    // same as
    // glReadBuffer(GL_COLOR_ATTACHMENT0);
    // glDrawBuffer(GL_COLOR_ATTACHMENT0);
    // glBlitFramebuffer(0, 0, window.getSize().x, window.getSize().y, 0, 0, window.getSize().x, window.getSize().y, static_cast<gl::ClearBufferMask>(GL_COLOR_BUFFER_BIT), static_cast<gl::GLenum>(GL_NEAREST));

    temporaryFramebuffer->unbind();
    temporaryFramebuffer2->unbind();

    blurProgram->use();

    const auto blurPasses = 10;

    // for the initial blur pass, use the texture from the bloomFramebuffer as an input

    // we do not need anything extra here, since the bloomBlurFramebuffer2 (which we read from) will already contain the data from the bloomBrightnessTexture

    blurProgram->setUniform("blurInput", 0);

    for (auto i = 0; i < blurPasses; ++i)
    {
        // bind one framebuffer to write blur results to and bind the texture from another framebuffer to read input data from (for this blur stage)
        if (i % 2 == 0)
        {
            // bind the new target framebuffer to write blur results to
            temporaryFramebuffer->bind();
            // bind the texture from the previous blur pass to read input data for this stage from
            temporaryTexture2->bindActive(0);
            // tell shader that we want to use horizontal blur
            blurProgram->setUniform("isHorizontalBlur", true);
        }
        else
        {
            // bind the new target framebuffer to write blur results to
            temporaryFramebuffer2->bind();
            // bind the texture from the previous blur pass to read input data for this stage from
            if (i > 0)
                temporaryTexture1->bindActive(0);
            // tell shader that we want to use vertical blur
            blurProgram->setUniform("isHorizontalBlur", false);
        }

        // render quad with the texture from the active texture
        quadModel->bind();
        quadModel->draw();
        quadModel->unbind();

        if (i % 2 == 0)
        {
            // unbind the active framebuffer
            temporaryFramebuffer->unbind();
            // unbind the active texture
            temporaryTexture2->unbindActive(0);
        }
        else
        {
            temporaryFramebuffer2->unbind();
            temporaryTexture1->unbindActive(0);
        }
    }

    blurProgram->release();
}

// third render pass - merge textures from the deferred rendering pre-pass into a final frame
{
    ::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));
    ::glClearColor(static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(1.0f));
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    deferredRenderingFinalPassProgram->use();

    deferredFragmentPositionTexture->bindActive(2);
    deferredFragmentNormalTexture->bindActive(3);
    deferredFragmentAlbedoTexture->bindActive(4);

    temporaryTexture1->bindActive(5);

    pointLightDataBuffer->bindBase(GL_SHADER_STORAGE_BUFFER, 5);

    deferredRenderingFinalPassProgram->setUniform("positionTexture", 2);
    deferredRenderingFinalPassProgram->setUniform("normalTexture", 3);
    deferredRenderingFinalPassProgram->setUniform("albedoTexture", 4);

    deferredRenderingFinalPassProgram->setUniform("ssaoTexture", 5);

    deferredRenderingFinalPassProgram->setUniform("cameraPosition", cameraPos);
    deferredRenderingFinalPassProgram->setUniform("projection", cameraProjection);
    deferredRenderingFinalPassProgram->setUniform("view", cameraView);

    quadModel->bind();
    quadModel->draw();
    quadModel->unbind();

    pointLightDataBuffer->unbind(GL_SHADER_STORAGE_BUFFER, 5);

    deferredFragmentPositionTexture->unbindActive(2);
    deferredFragmentNormalTexture->unbindActive(3);
    deferredFragmentAlbedoTexture->unbindActive(4);

    temporaryTexture1->unbindActive(5);

    deferredRenderingFinalPassProgram->release();
}

The fragment shader for the first render pass:

#version 410

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

layout (location = 0) out vec3 fsPosition;
layout (location = 1) out vec3 fsNormal;
layout (location = 2) out vec4 fsAlbedo;

uniform sampler2D diffuseTexture;
uniform sampler2D normalMapTexture;

void main()
{
    fsPosition = fsIn.fragmentPosition;

    fsNormal = texture(normalMapTexture, fsIn.textureCoord).rgb;

    if (length(fsNormal) == 0.0)
    {
        fsNormal = fsIn.normal;
    }

    fsAlbedo = texture(diffuseTexture, fsIn.textureCoord);
}

Note how it has three output vectors , each of which will be bound to the corresponding framebuffer attachment.

The fragment shader for SSAO (second render pass) :

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec2 textureCoord;
} fsIn;

uniform sampler2D positionTexture;
uniform sampler2D normalTexture;

uniform sampler2D ssaoNoiseTexture;
uniform sampler1D ssaoKernelTexture;

uniform mat4 projection;

void main()
{
    vec3 fragmentPosition = texture(positionTexture, fsIn.textureCoord).xyz;
    vec3 normal = texture(normalTexture, fsIn.textureCoord).rgb;

    float radius = 0.5;
    float bias = 0.025;

    vec2 screenSize = textureSize(positionTexture, 0);
    vec2 noiseSize = textureSize(ssaoNoiseTexture, 0);
    float kernelSize = textureSize(ssaoKernelTexture, 0);

    vec2 noiseScale = screenSize / noiseSize;

    vec3 randomVec = normalize(texture(ssaoNoiseTexture, fsIn.textureCoord * noiseScale).xyz);

    vec3 tangent = normalize(randomVec - normal * dot(randomVec, normal));
    vec3 bitangent = cross(normal, tangent);
    mat3 TBN = mat3(tangent, bitangent, normal);

    float occlusion = 0.0;

    for (int i = 0; i < kernelSize; ++i)
    {
        vec3 samplePosition = TBN * texture(ssaoKernelTexture, i).xyz;

        if (dot(samplePosition, normal) < 0.0)
            samplePosition *= -1.0;

        samplePosition = fragmentPosition + samplePosition * radius;

        vec4 offsetUV = projection * vec4(samplePosition, 1.0);
        offsetUV.xyz /= offsetUV.w;
        offsetUV.xy = offsetUV.xy * 0.5 + 0.5;

        vec4 offsetPosition = texture(positionTexture, offsetUV.xy);

        float rangeCheck = smoothstep(0.0, 1.0, radius / abs(fragmentPosition.z - offsetPosition.z));

        occlusion += (samplePosition.z >= offsetPosition.z + bias ? 1.0 : 0.0) * rangeCheck;
    }

    occlusion = (occlusion / kernelSize);

    fragmentColor = vec4(vec3(occlusion), 1.0);
}

The blur fragment shader:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

uniform sampler2D blurInput;

uniform bool isHorizontalBlur;

float weight[5] = float[] (0.2270270270, 0.1945945946, 0.1216216216, 0.0540540541, 0.0162162162);

void main()
{
    vec2 textureOffset = 1.0 / textureSize(blurInput, 0);
    vec3 result = texture(blurInput, fsIn.textureCoord).rgb * weight[0];

    if (isHorizontalBlur) {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
        }
    } else {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(0.0, textureOffset.y * i)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(0.0, textureOffset.y * i)).rgb * weight[i];
        }
    }

    fragmentColor = vec4(result, 1.0);
}

The final render pass fragment shader:

#version 430

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

layout (location = 0) out vec4 fragmentColor;

struct PointLight
{
    vec3 position;
    float strength;
    vec4 color;
    // float farPlane;
    // mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer PointLightData
{
    PointLight pointLight[];
} pointLightData;

uniform sampler2D positionTexture;
uniform sampler2D normalTexture;
uniform sampler2D albedoTexture;
uniform sampler2D ssaoTexture;

uniform vec3 cameraPosition;

uniform mat4 view;
uniform mat4 projection;

float attenuation_constant = 1.0;
float attenuation_linear = 0.09;
float attenuation_quadratic = 0.032;

void main()
{
    vec3 fragmentPosition = texture(positionTexture, fsIn.textureCoord).xyz;
    vec3 normal = texture(normalTexture, fsIn.textureCoord).rgb;
    vec4 albedoColor = texture(albedoTexture, fsIn.textureCoord);

    vec3 viewDirection = normalize(cameraPosition - fragmentPosition);

    float ambientOcclusion = texture(ssaoTexture, fsIn.textureCoord).r;

    vec4 lighting = albedoColor * 0.3 * ambientOcclusion;

    for (int i = 0; i < pointLightData.pointLight.length(); ++i)
    {
        PointLight light = pointLightData.pointLight[i];

        vec3 lightDirection = normalize(light.position - fragmentPosition);

        float lightDistance = length(light.position - fragmentPosition);
        float attenuation = 1.0 / (attenuation_constant + (attenuation_linear * lightDistance) + (attenuation_quadratic * lightDistance * lightDistance));

        vec4 diffuse = max(dot(normal, lightDirection), 0.0) * albedoColor * light.color;

        lighting += diffuse * attenuation;
    }

    fragmentColor = lighting;
}

Since all of the rendering is happening in passes on the frame buffers, the vertex shaders are very simple - just passing the parameters through to the next stage.

Horizon-based ambient occlusion (HBAO)

Screen-space ambient occlusion algorithm (aka SSAO) is fine, but it can use even further improvement. Instead of calculating random samples inside imaginary sphere, you can be a little smarter about those samples and only generate them in a hemisphere, oriented towards the camera. Then, instead of checking the camera-space positions of samples and pixels, you can use the "horizon" of a current pixel and calculate the "horizion" for each of the samples. You can then use the angle difference between those two "horizons" to see if pixels are potentially occluded by some other polygon.

This technique gives better results for corners and edges than conventional SSAO technique.

[nVidia] Image-Space Horizon-Based Ambient Occlusion

The difference between SSAO and HBAO implementations lays in literally one fragment shader, calculating the ambient occlusion values (the second pass, from the SSAO paragraph) :

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec2 textureCoord;
} fsIn;

uniform sampler2D positionTexture;
uniform sampler2D normalTexture;

uniform sampler2D hbaoNoiseTexture;
uniform sampler1D hbaoKernelTexture;

uniform mat4 projection;

const float PI = 3.14;
const float NUM_SAMPLE_DIRECTIONS = 10.0;
const float NUM_SAMPLE_STEPS = 10.0;
const float INTENSITY = 2.0;

const float radius = 30;
const float bias = 0.5;

void main()
{
    vec3 fragmentPosition = texture(positionTexture, fsIn.textureCoord).xyz;
    vec3 normal = texture(normalTexture, fsIn.textureCoord).rgb;

    vec2 screenSize = textureSize(positionTexture, 0);
    vec2 noiseSize = textureSize(hbaoNoiseTexture, 0);

    vec2 noiseScale = screenSize / noiseSize;

    const float stepPixels = radius / (NUM_SAMPLE_STEPS + 1);
    const float alpha = 2 * PI / float(NUM_SAMPLE_DIRECTIONS);

    float occlusion = 0.0;

    for (float i = 0; i < NUM_SAMPLE_DIRECTIONS; ++i)
    {
        float angle = alpha * i;

        vec4 random = texture(hbaoNoiseTexture, fsIn.textureCoord * noiseScale);

        // vec2 dir = vec2(cos(angle), sin(angle));
        // vec2 cosSin = random.xy;
        // vec2 direction = vec2(dir.x * cosSin.x - dir.y * cosSin.y, dir.x * cosSin.y + dir.y * cosSin.x);
        // vec2 direction = rotateDirection(vec2(cos(angle), sin(angle)), random.xy);

        vec2 direction = vec2(cos(angle) * random.x - sin(angle) * random.y, cos(angle) * random.y + sin(angle) * random.x);

        float rayPixels = random.z * (stepPixels + 1.0);

        for (float t = 0; t < NUM_SAMPLE_STEPS; ++t)
        {
            vec2 sampleUV = round(rayPixels * direction) * (1.0 / screenSize) + fsIn.textureCoord;
            vec3 samplePosition = texture(positionTexture, sampleUV).xyz;

            rayPixels += stepPixels;

            vec3 sampleDirection = samplePosition - fragmentPosition;
            float v1 = dot(sampleDirection, sampleDirection);
            float v2 = dot(normal, sampleDirection) * 1.0 / sqrt(v1);
            occlusion += clamp(v2 - bias, 0.0, 1.0) * clamp(v1 * (-1.0 / (radius * radius)) + 1.0, 0.0, 1.0);
        }
    }

    occlusion *= INTENSITY / (NUM_SAMPLE_DIRECTIONS * NUM_SAMPLE_STEPS);
    occlusion = clamp(occlusion, 0.0, 1.0);

    fragmentColor = vec4(vec3(occlusion), 1.0);
}

Volumetric lighting

Have you ever seen those beautiful rays of light passing through the clouds and becoming actually visible rays ? There are lots of these in games too. They are also known as "god rays". They are actually tiny particles (presumingly dust) hanging in the air and reflecting the light.

The technique which allows to achieve such an effect (and tons of others) is called "raymarching".

The idea is that you store the pixels' depth and position information in both light space and camera space and then you cast an imaginary ray from the camera towards each pixel rendered. You then split this ray into a constant number of parts and project the resulting points into light space. You then compare the position of the point on the ray in light space to the depth value stored in the light space, and if the depth value stored is greater than the position on the ray - the point on the ray is visible from the perspective of a light, so you make the pixel on the screen a bit brighter. The more parts of the ray are visible by the light - the brighter the pixel will be.

The implementation for this technique is actually pretty straightforward: this is a two-pass deferred rendering application.

First, you set up a framebuffer with multiple attachments, for each of the scene parameters:

auto deferredFragmentPositionTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentPositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentPositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentPositionTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredFragmentNormalTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentNormalTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentNormalTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentNormalTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredFragmentAlbedoTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentAlbedoTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentAlbedoTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentAlbedoTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGBA8),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_UNSIGNED_BYTE),
    nullptr
);

auto deferredFragmentLightSpacePositionTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentLightSpacePositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentLightSpacePositionTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentLightSpacePositionTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB32F),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredFragmentDepthTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

deferredFragmentDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
deferredFragmentDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

deferredFragmentDepthTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_DEPTH_COMPONENT),
    glm::vec2(static_cast<float>(window.getSize().x), static_cast<float>(window.getSize().y)),
    0,
    static_cast<gl::GLenum>(GL_DEPTH_COMPONENT),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr
);

auto deferredRenderingFramebuffer = std::make_unique<globjects::Framebuffer>();
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), deferredFragmentPositionTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1), deferredFragmentNormalTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT2), deferredFragmentAlbedoTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT3), deferredFragmentLightSpacePositionTexture.get());
deferredRenderingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), deferredFragmentDepthTexture.get());

// tell framebuffer it actually needs to render to **BOTH** textures, but does not have to output anywhere (last NONE argument, iirc)
deferredRenderingFramebuffer->setDrawBuffers({
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0),
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1),
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT2),
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT3),
    static_cast<gl::GLenum>(GL_NONE)
});

deferredRenderingFramebuffer->printStatus(true);

Then, you render scene to that framebuffer:

deferredRenderingFramebuffer->bind();

deferredRenderingPrePassProgram->use();

// render scene

deferredRenderingPrePassProgram->release();

deferredRenderingFramebuffer->unbind();

The vertex shader for this pass is:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;
// layout (location = 3) in vec3 vertexTangent;
// layout (location = 4) in vec3 vertexBitangent;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec3 lightSpaceCoord;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;
uniform mat4 lightSpaceMatrix;

void main()
{
    vec4 position = model * vec4(vertexPosition, 1.0);
    vsOut.fragmentPosition = position.xyz / position.w;
    vsOut.normal = transpose(inverse(mat3(view * model))) * vertexNormal;
    vsOut.textureCoord = vertexTextureCoord;

    vec4 lightSpaceCoord = lightSpaceMatrix * model * vec4(vertexPosition, 1.0);
    vsOut.lightSpaceCoord = lightSpaceCoord.xyz / lightSpaceCoord.w;

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

And the fragment shader is:

#version 410

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec3 lightSpaceCoord;
} fsIn;

layout (location = 0) out vec3 fsPosition;
layout (location = 1) out vec3 fsNormal;
layout (location = 2) out vec4 fsAlbedo;
layout (location = 3) out vec3 fsLightSpaceCoord;

uniform sampler2D diffuseTexture;
uniform sampler2D normalMapTexture;

void main()
{
    fsPosition = fsIn.fragmentPosition * 0.5 + 0.5;

    fsNormal = texture(normalMapTexture, fsIn.textureCoord).rgb;

    if (length(fsNormal) == 0.0)
    {
        fsNormal = fsIn.normal;
    }

    fsNormal = normalize(fsNormal.xyz) * 0.5 + 0.5;

    fsAlbedo = texture(diffuseTexture, fsIn.textureCoord);

    fsLightSpaceCoord = fsIn.lightSpaceCoord.xyz * 0.5 + 0.5;
}

In the final render pass you just combine the textures of the previous render pass(-es) into the final frame:

// final render pass - merge textures from the deferred rendering pre-pass into a final frame
{
    ::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));
    ::glClearColor(static_cast<gl::GLfloat>(1.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(0.0f), static_cast<gl::GLfloat>(1.0f));
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    deferredRenderingFinalPassProgram->use();

    deferredFragmentPositionTexture->textureHandle().makeResident();
    deferredFragmentNormalTexture->textureHandle().makeResident();
    deferredFragmentAlbedoTexture->textureHandle().makeResident();

    deferredFragmentLightSpacePositionTexture->textureHandle().makeResident();
    shadowMapTexture->textureHandle().makeResident();

    pointLightDataBuffer->bindBase(GL_SHADER_STORAGE_BUFFER, 5);

    deferredRenderingFinalPassProgram->setUniform("positionTexture", deferredFragmentPositionTexture->textureHandle().handle());
    deferredRenderingFinalPassProgram->setUniform("normalTexture", deferredFragmentNormalTexture->textureHandle().handle());
    deferredRenderingFinalPassProgram->setUniform("albedoTexture", deferredFragmentAlbedoTexture->textureHandle().handle());

    deferredRenderingFinalPassProgram->setUniform("lightSpaceCoord", deferredFragmentLightSpacePositionTexture->textureHandle().handle());
    deferredRenderingFinalPassProgram->setUniform("shadowMap", shadowMapTexture->textureHandle().handle());

    deferredRenderingFinalPassProgram->setUniform("cameraPosition", cameraPos);
    deferredRenderingFinalPassProgram->setUniform("projection", cameraProjection);
    deferredRenderingFinalPassProgram->setUniform("view", cameraView);

    deferredRenderingFinalPassProgram->setUniform("lightSpaceMatrix", lightSpaceMatrix);
    deferredRenderingFinalPassProgram->setUniform("sunDirection", -lightPosition);
    deferredRenderingFinalPassProgram->setUniform("sunColor", glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

    quadModel->bind();
    quadModel->draw();
    quadModel->unbind();

    pointLightDataBuffer->unbind(GL_SHADER_STORAGE_BUFFER, 5);

    deferredFragmentPositionTexture->textureHandle().makeNonResident();
    deferredFragmentNormalTexture->textureHandle().makeNonResident();
    deferredFragmentAlbedoTexture->textureHandle().makeNonResident();

    deferredFragmentLightSpacePositionTexture->textureHandle().makeNonResident();
    shadowMapTexture->textureHandle().makeNonResident();

    deferredRenderingFinalPassProgram->release();
}

Using the fragment shader, which increases the brightness of the pixels based on the information from the light projection buffer. This is done using counting (or the "accumulating") the number of intersections of the ray from the viewing position towards the light source (by checking the light projection buffer - the position of each pixel in the light space) :

#version 460

#extension GL_ARB_bindless_texture : require

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

layout (location = 0) out vec4 fragmentColor;

struct PointLight
{
    vec3 position;
    float strength;
    vec4 color;
    // float farPlane;
    // mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer PointLightData
{
    PointLight pointLight[];
} pointLightData;

layout(bindless_sampler) uniform sampler2D positionTexture;
layout(bindless_sampler) uniform sampler2D normalTexture;
layout(bindless_sampler) uniform sampler2D albedoTexture;

layout(bindless_sampler) uniform sampler2D lightSpaceCoord;
layout(bindless_sampler) uniform sampler2D shadowMap;

uniform vec3 cameraPosition;
uniform vec3 sunDirection;
uniform vec4 sunColor;

uniform mat4 view;
uniform mat4 projection;
uniform mat4 lightSpaceMatrix;

float attenuation_constant = 1.0;
float attenuation_linear = 0.09;
float attenuation_quadratic = 0.032;

const float NB_STEPS = 30;
const float G_SCATTERING = 0.858;

#define M_PI 3.1415926535897932384626433832795

void main()
{
    vec3 fragmentPosition = texture(positionTexture, fsIn.textureCoord).xyz * 2 - 1;
    vec3 normal = texture(normalTexture, fsIn.textureCoord).rgb * 2 - 1;
    vec4 albedoColor = texture(albedoTexture, fsIn.textureCoord);

    vec3 viewDirection = normalize(cameraPosition - fragmentPosition);

    vec3 startPosition = cameraPosition;
    vec3 endRayPosition = fragmentPosition;

    vec3 rayVector = endRayPosition.xyz- startPosition;

    float rayLength = length(rayVector);
    vec3 rayDirection = normalize(rayVector);

    float stepLength = rayLength / NB_STEPS;

    vec3 raymarchingStep = rayDirection * stepLength;

    vec3 currentPosition = startPosition;

    vec4 accumFog = vec4(0.0);

    for (int i = 0; i < NB_STEPS; i++)
    {
        // basically perform shadow mapping
        vec4 worldInLightSpace = lightSpaceMatrix * vec4(currentPosition, 1.0f);
        worldInLightSpace /= worldInLightSpace.w;

        vec2 lightSpaceTextureCoord1 = (worldInLightSpace.xy * 0.5) + 0.5; // [-1..1] -> [0..1]
        float shadowMapValue1 = texture(shadowMap, lightSpaceTextureCoord1.xy).r;

        if (shadowMapValue1 > worldInLightSpace.z)
        {
            // Mie scaterring approximated with Henyey-Greenstein phase function
            float lightDotView = dot(normalize(rayDirection), normalize(-sunDirection));

            float scattering = 1.0 - G_SCATTERING * G_SCATTERING;
            scattering /= (4.0f * M_PI * pow(1.0f + G_SCATTERING * G_SCATTERING - (2.0f * G_SCATTERING) * lightDotView, 1.5f));

            accumFog += scattering * sunColor;
        }

        currentPosition += raymarchingStep;
    }

    accumFog /= NB_STEPS;

    // fade rays away
    // accumFog *= currentPosition / NB_STEPS);

    // lighting *= accumFog;

    vec4 fragmentPositionInLightSpace1 = lightSpaceMatrix * vec4(fragmentPosition, 1.0);
    fragmentPositionInLightSpace1 /= fragmentPositionInLightSpace1.w;

    vec2 shadowMapCoord1 = fragmentPositionInLightSpace1.xy * 0.5 + 0.5;

    // vec3 fragmentPositionInLightSpace2 = texture(lightSpaceCoord, fsIn.textureCoord).xyz;

    // vec2 shadowMapCoord2 = fragmentPositionInLightSpace2.xy;

    float thisDepth = fragmentPositionInLightSpace1.z;
    float occluderDepth = texture(shadowMap, shadowMapCoord1).r;

    float shadowFactor = 0.0;

    if (thisDepth > 0.0 && occluderDepth < 1.0 && thisDepth < occluderDepth)
    {
        shadowFactor = 1.0;
    }

    fragmentColor = ((albedoColor * shadowFactor * 0.3) + (accumFog * 0.7));
}

Common rendering techniques

There are few simpler effects and techniques, which are more widely known (in that more tutorials on the Internet discuss these to some degree), but I feel like describing them as well.

Bloom

Bloom is a relatively simple effect, but it looks really nice.

The idea is that you render the scene to a framebuffer, storing the light emitting value per pixel. You then blur the resulting image (using Gaussian blur, for instance) and blend the original scene image with the blurred one.

The only complex part here is Gaussian blur.

const auto antiAliasingSamples = 4;

auto bloomColorTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D_MULTISAMPLE));

bloomColorTexture->image2DMultisample(
    antiAliasingSamples,
    static_cast<gl::GLenum>(GL_RGB16F),
    glm::vec2(window.getSize().x, window.getSize().y),
    static_cast<gl::GLboolean>(true));

auto bloomBrightnessTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D_MULTISAMPLE));

bloomBrightnessTexture->image2DMultisample(
    antiAliasingSamples,
    static_cast<gl::GLenum>(GL_RGB16F),
    glm::vec2(window.getSize().x, window.getSize().y),
    static_cast<gl::GLboolean>(true));

auto renderBuffer = std::make_unique<globjects::Renderbuffer>();

renderBuffer->storageMultisample(antiAliasingSamples, static_cast<gl::GLenum>(GL_DEPTH24_STENCIL8), window.getSize().x, window.getSize().y);

auto bloomFramebuffer = std::make_unique<globjects::Framebuffer>();

bloomFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), bloomColorTexture.get());
bloomFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1), bloomBrightnessTexture.get());

bloomFramebuffer->attachRenderBuffer(static_cast<gl::GLenum>(GL_DEPTH_STENCIL_ATTACHMENT), renderBuffer.get());

// tell framebuffer it actually needs to render to **BOTH** textures, but does not have to output anywhere (last NONE argument, iirc)
bloomFramebuffer->setDrawBuffers({ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT1), static_cast<gl::GLenum>(GL_NONE) });

auto temporaryOutputTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

temporaryOutputTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
temporaryOutputTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

temporaryOutputTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));
temporaryOutputTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));

temporaryOutputTexture->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB16F),
    glm::vec2(window.getSize().x, window.getSize().y),
    0,
    static_cast<gl::GLenum>(GL_RGB),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr);

auto temporaryOutputFramebuffer = std::make_unique<globjects::Framebuffer>();

temporaryOutputFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), temporaryOutputTexture.get());

temporaryOutputFramebuffer->printStatus(true);

auto bloomBlurTexture1 = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

bloomBlurTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
bloomBlurTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

bloomBlurTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));
bloomBlurTexture1->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));

bloomBlurTexture1->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB16F),
    glm::vec2(window.getSize().x, window.getSize().y),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr);

auto bloomBlurFramebuffer1 = std::make_unique<globjects::Framebuffer>();

bloomBlurFramebuffer1->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), bloomBlurTexture1.get());

bloomBlurFramebuffer1->printStatus(true);

auto bloomBlurTexture2 = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));

bloomBlurTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
bloomBlurTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

bloomBlurTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));
bloomBlurTexture2->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_EDGE));

bloomBlurTexture2->image2D(
    0,
    static_cast<gl::GLenum>(GL_RGB16F),
    glm::vec2(window.getSize().x, window.getSize().y),
    0,
    static_cast<gl::GLenum>(GL_RGBA),
    static_cast<gl::GLenum>(GL_FLOAT),
    nullptr);

auto bloomBlurFramebuffer2 = std::make_unique<globjects::Framebuffer>();

bloomBlurFramebuffer2->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), bloomBlurTexture2.get());

With all that preparatory work, the rendering is a multi-step (multi-pass) process - first pass is to render scene to the bloomFramebuffer , filling the bloomColorTexture and bloomBrightnessTexture :

bloomFramebuffer->bind();

// render scene

bloomFramebuffer->unbind();

The shader for the first rendering pass is only different in fragment phase:

#version 410

layout (location = 0) out vec4 fragmentColor;
layout (location = 1) out vec4 brightColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec4 fragmentPositionInLightSpace;
} fsIn;

uniform sampler2D shadowMap;
uniform sampler2D diffuseTexture;
uniform sampler2D specularMapTexture;
uniform sampler2D emissionMapTexture;

uniform vec3 lightPosition;
uniform vec3 lightColor;
uniform vec3 cameraPosition;

uniform vec3 emissionColor;

float attenuation_constant = 1.0;
float attenuation_linear = 0.09;
float attenuation_quadratic = 0.032;

float shadowCalculation(vec3 normal, vec3 lightDirection)
{
    vec3 shadowMapCoord = (fsIn.fragmentPositionInLightSpace.xyz / fsIn.fragmentPositionInLightSpace.w) * 0.5 + 0.5;
    float occluderDepth = texture(shadowMap, shadowMapCoord.xy).r;
    float thisDepth = shadowMapCoord.z;

    if (thisDepth > 1.0)
    {
        return 0.0;
    }

    float bias = max(0.05 * (1.0 - dot(normal, lightDirection)), 0.005);

    // PCF
    float shadow = 0.0;
    vec2 texelSize = 1.0 / textureSize(shadowMap, 0);

    for (int x = -1; x <= 1; ++x)
    {
        for (int y = -1; y <= 1; ++y)
        {
            float pcfDepth = texture(shadowMap, shadowMapCoord.xy + vec2(x, y) * texelSize).r;

            shadow += thisDepth - bias < pcfDepth  ? 1.0 : 0.0;
        }
    }

    shadow /= 9.0;

    return shadow;
}

void main()
{
    vec3 color = texture(diffuseTexture, fsIn.textureCoord).rgb;
    vec3 normal = normalize(fsIn.normal);

    // ambient
    vec3 ambient = 0.3 * color;

    // diffuse
    vec3 lightDirection = normalize(lightPosition - fsIn.fragmentPosition);
    float diff = max(dot(lightDirection, normal), 0.0);
    vec3 diffuse = diff * lightColor;

    // specular
    vec3 viewDirection = normalize(cameraPosition - fsIn.fragmentPosition);
    vec3 halfwayDirection = normalize(lightDirection + viewDirection);
    float spec = pow(max(dot(normal, halfwayDirection), 0.0), 64.0);
    vec3 specular = spec * lightColor;

    // attenuation
    float distance = length(lightPosition - fsIn.fragmentPosition);
    float attenuation = 1.0 / (attenuation_constant + attenuation_linear * distance + attenuation_quadratic * (distance * distance));

    // calculate shadow; this represents a global directional light, like Sun
    float shadow = shadowCalculation(normal, lightDirection);

    // these are the multipliers from different light maps (read from corresponding textures)
    float specularCoefficient = texture(specularMapTexture, fsIn.textureCoord).r;
    float emissionCoefficient = texture(emissionMapTexture, fsIn.textureCoord).r;

    vec3 lighting = (1 - emissionCoefficient) * ((shadow * ((diffuse * attenuation) + (specular * specularCoefficient * attenuation))) + (ambient * attenuation)) * color + (emissionColor * emissionCoefficient * 4.0);

    float brightness = dot(lighting, vec3(0.2126, 0.7152, 0.0722));

    if (brightness > 1.0) {
        brightColor = vec4(lighting, 1.0);
    } else {
        brightColor = vec4(0.0, 0.0, 0.0, 1.0);
    }

    fragmentColor = vec4(lighting, 1.0);
}

The second scene rendering blurs the framebuffer's data by rendering the texture to a quad (that is, a rectangle with same dimensions as the screen; this makes a deferred rendering) and making each pixel an average of the neighbour pixels from the previous step:

// copy data from the multisampled framebuffer to non-multisampled textures
// since the bloomFramebuffer has two color attachments, we need to blit twice
// but doing so removes the need in some additional conditions when running the blur

bloomFramebuffer->bind(static_cast<gl::GLenum>(GL_READ_FRAMEBUFFER));
temporaryOutputFramebuffer->bind(static_cast<gl::GLenum>(GL_DRAW_FRAMEBUFFER));

bloomFramebuffer->blit(
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0),
    std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
    temporaryOutputFramebuffer.get(),
    std::vector<gl::GLenum>{ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0) },
    std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
    static_cast<gl::ClearBufferMask>(GL_COLOR_BUFFER_BIT),
    static_cast<gl::GLenum>(GL_NEAREST));

// same as
// glReadBuffer(GL_COLOR_ATTACHMENT0);
// glDrawBuffer(GL_COLOR_ATTACHMENT0);
// glBlitFramebuffer(0, 0, window.getSize().x, window.getSize().y, 0, 0, window.getSize().x, window.getSize().y, static_cast<gl::ClearBufferMask>(GL_COLOR_BUFFER_BIT), static_cast<gl::GLenum>(GL_NEAREST));

temporaryOutputFramebuffer->unbind();
bloomFramebuffer->unbind();

bloomFramebuffer->bind(static_cast<gl::GLenum>(GL_READ_FRAMEBUFFER));
bloomBlurFramebuffer2->bind(static_cast<gl::GLenum>(GL_DRAW_FRAMEBUFFER));

bloomFramebuffer->blit(
    static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0),
    std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
    bloomBlurFramebuffer2.get(),
    std::vector<gl::GLenum>{ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0) },
    std::array<gl::GLint, 4>{ 0, 0, static_cast<int>(window.getSize().x), static_cast<int>(window.getSize().y) },
    static_cast<gl::ClearBufferMask>(GL_COLOR_BUFFER_BIT),
    static_cast<gl::GLenum>(GL_NEAREST));

bloomFramebuffer->unbind();
bloomBlurFramebuffer2->unbind();

// third pass - blur the data stored in the bloom framebuffer with two-pass Gauss blur

bloomBlurProgram->use();

const auto blurPasses = 10;

// for the initial blur pass, use the texture from the bloomFramebuffer as an input

// we do not need anything extra here, since the bloomBlurFramebuffer2 (which we read from) will already contain the data from the bloomBrightnessTexture

bloomBlurProgram->setUniform("blurInput", 0);

for (auto i = 0; i < blurPasses; ++i)
{
    // bind one framebuffer to write blur results to and bind the texture from another framebuffer to read input data from (for this blur stage)
    if (i % 2 == 0)
    {
        // bind the new target framebuffer to write blur results to
        bloomBlurFramebuffer1->bind();
        // bind the texture from the previous blur pass to read input data for this stage from
        bloomBlurTexture2->bindActive(0);
        // tell shader that we want to use horizontal blur
        bloomBlurProgram->setUniform("isHorizontalBlur", true);
    }
    else
    {
        // bind the new target framebuffer to write blur results to
        bloomBlurFramebuffer2->bind();
        // bind the texture from the previous blur pass to read input data for this stage from
        if (i > 0)
            bloomBlurTexture1->bindActive(0);
        // tell shader that we want to use vertical blur
        bloomBlurProgram->setUniform("isHorizontalBlur", false);
    }

    // render quad with the texture from the active texture
    quadModel->bind();
    quadModel->draw();
    quadModel->unbind();

    if (i % 2 == 0)
    {
        // unbind the active framebuffer
        bloomBlurFramebuffer1->unbind();
        // unbind the active texture
        bloomBlurTexture2->unbindActive(0);
    }
    else
    {
        bloomBlurFramebuffer2->unbind();
        bloomBlurTexture1->unbindActive(0);
    }
}

bloomBlurProgram->release();

The fragment shader for blur render pass:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

uniform sampler2D blurInput;

uniform bool isHorizontalBlur;

float weight[5] = float[] (0.2270270270, 0.1945945946, 0.1216216216, 0.0540540541, 0.0162162162);

void main()
{
    vec2 textureOffset = 1.0 / textureSize(blurInput, 0);
    vec3 result = texture(blurInput, fsIn.textureCoord).rgb * weight[0];

    if (isHorizontalBlur) {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
        }
    } else {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(0.0, textureOffset.y * i)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(0.0, textureOffset.y * i)).rgb * weight[i];
        }
    }

    fragmentColor = vec4(result, 1.0);
}

As with most cases described in this blog, the vertex shader for both stages is just the same old projection, nothing fancy:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;

void main()
{
    vsOut.fragmentPosition = vec3(model * vec4(vertexPosition, 1.0));
    vsOut.normal = vertexNormal;
    vsOut.textureCoord = vertexTextureCoord;

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

Except there is no need to project anything when rendering to the quad, so the difference is only in the last line for the blur shader:

gl_Position = vec4(vertexPosition, 1.0);

Now, the last rendering pass is quite simple, but that's where the magic becomes reality: we mix the original picture, rendered to the temporaryOutputTexture (first render pass) with the blurred bright pixels from bloomBlurTexture2 (the second render pass) to get the glow effect:

bloomOutputProgram->use();

// here we use our temporary non-multisampled texture as one of the inputs for merge bloom effect stages, since OUR shader would only work with one sample layer
temporaryOutputTexture->bindActive(0);
bloomBlurTexture2->bindActive(1);

bloomOutputProgram->setUniform("colorOutput", 0);
bloomOutputProgram->setUniform("blurOutput", 1);

quadModel->bind();
quadModel->draw();
quadModel->unbind();

temporaryOutputTexture->unbindActive(0);
bloomBlurTexture2->unbindActive(1);

bloomOutputProgram->release();

Fragment shader:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
} fsIn;

uniform sampler2D blurInput;

uniform bool isHorizontalBlur;

float weight[5] = float[] (0.2270270270, 0.1945945946, 0.1216216216, 0.0540540541, 0.0162162162);

void main()
{
    vec2 textureOffset = 1.0 / textureSize(blurInput, 0);
    vec3 result = texture(blurInput, fsIn.textureCoord).rgb * weight[0];

    if (isHorizontalBlur) {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(textureOffset.x * i, 0.0)).rgb * weight[i];
        }
    } else {
        for (int i = 1; i < 5; ++i) {
            result += texture(blurInput, fsIn.textureCoord + vec2(0.0, textureOffset.y * i)).rgb * weight[i];
            result += texture(blurInput, fsIn.textureCoord - vec2(0.0, textureOffset.y * i)).rgb * weight[i];
        }
    }

    fragmentColor = vec4(result, 1.0);
}

Vertex shader is same deferred rendering vertex shader - without any projection going on.

Particle systems

Particle simulation is not really a technique in itself - it is a very big topic (how you design it from the code perspective, how do you simulate particles optimally - concurrent programming, SIMD, calculations on the GPU, etc.). But the way you render bunch of particles could be qualified as a rendering technique. It is mostly about instanced rendering.

The standard implementations include calculating every particle's params (position, rotation, texture animation frame etc.) on CPU and passing that data to the GPU to be rendered. More interesting approach is to do all the calculations on the GPU itself, only passing a handful of parameters from CPU, like the number of particles and their origin - essentially, the particle emitter parameters and then using the computing shader to do the rest.

Skybox

Sky can really add atmosphere to your game. Rendering it might be as simple as rendering an inverted cube with texture on the insides of its faces and moving it along with the camera (or rather rendering it in camera space, not world space).

The skybox can use normal textures or it could use cubemaps - literally just a cube with textures on each of its six sides. Cubemaps allow you to calculate the texture coordinates by only having a vector from the center of the cube in the direction of view.

[learnopengl] Cubemaps

In order to implement skybox, one first needs a cubemap texture:

sf::Image m_top, m_bottom, m_left, m_right, m_front, m_back;

// load textures with SFML

std::map<gl::GLenum, sf::Image> skyboxTextures{
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_X), m_right },
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_NEGATIVE_X), m_left },
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_Y), m_top },
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_NEGATIVE_Y), m_bottom },
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_Z), m_back },
    { static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP_NEGATIVE_Z), m_front },
};

auto skyboxTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP));

skyboxTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
skyboxTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));

skyboxTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<GLint>(GL_CLAMP_TO_EDGE));
skyboxTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<GLint>(GL_CLAMP_TO_EDGE));
skyboxTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_R), static_cast<GLint>(GL_CLAMP_TO_EDGE));

skyboxTexture->bind();

for (auto& kv : skyboxTextures)
{
    const auto target = kv.first;
    auto& image = kv.second;

    if (target == gl::GL_TEXTURE_CUBE_MAP_POSITIVE_Y || target == gl::GL_TEXTURE_CUBE_MAP_NEGATIVE_Y)
    {
        image.flipVertically();
    }
    else
    {
        image.flipHorizontally();
    }

    ::glTexImage2D(
        static_cast<::GLenum>(target),
        0,
        static_cast<::GLenum>(GL_RGBA8),
        static_cast<::GLsizei>(image.getSize().x),
        static_cast<::GLsizei>(image.getSize().y),
        0,
        static_cast<::GLenum>(GL_RGBA),
        static_cast<::GLenum>(GL_UNSIGNED_BYTE),
        reinterpret_cast<const ::GLvoid*>(image.getPixelsPtr()));
}

skyboxTexture->unbind();

In my sample I manually create a box mesh with inverted normals:

std::vector<glm::vec3> vertices{
    { -1.0f, 1.0f, 1.0f },
    { -1.0f, -1.0f, 1.0f },
    { 1.0f, -1.0f, 1.0f },
    { 1.0f, 1.0f, 1.0f },
    { -1.0f, 1.0f, 1.0f },
    { -1.0f, -1.0f, 1.0f },
    { -1.0f, -1.0f, -1.0f },
    { -1.0f, 1.0f, -1.0f },
    { 1.0f, 1.0f, 1.0f },
    { 1.0f, -1.0f, 1.0f },
    { 1.0f, -1.0f, -1.0f },
    { 1.0f, 1.0f, -1.0f },
    { -1.0f, 1.0f, -1.0f },
    { -1.0f, -1.0f, -1.0f },
    { 1.0f, -1.0f, -1.0f },
    { 1.0f, 1.0f, -1.0f },
    { -1.0f, 1.0f, -1.0f },
    { -1.0f, 1.0f, 1.0f },
    { 1.0f, 1.0f, 1.0f },
    { 1.0f, 1.0f, -1.0f },
    { -1.0f, -1.0f, -1.0f },
    { -1.0f, -1.0f, 1.0f },
    { 1.0f, -1.0f, 1.0f },
    { 1.0f, -1.0f, -1.0f },
};

std::vector<glm::vec3> normals{
    { 0.0f, 1.0f, 0.0f },
    { 0.0f, 1.0f, 0.0f },
    { 0.0f, 1.0f, 0.0f },

    { 0.0f, -1.0f, 0.0f },
    { 0.0f, -1.0f, 0.0f },
    { 0.0f, -1.0f, 0.0f },

    { -1.0f, 0.0f, 0.0f },
    { -1.0f, 0.0f, 0.0f },
    { -1.0f, 0.0f, 0.0f },

    { 0.0f, 0.0f, -1.0f },
    { 0.0f, 0.0f, -1.0f },
    { 0.0f, 0.0f, -1.0f },

    { 1.0f, 0.0f, 0.0f },
    { 1.0f, 0.0f, 0.0f },
    { 1.0f, 0.0f, 0.0f },

    { 0.0f, 0.0f, 1.0f },
    { 0.0f, 0.0f, 1.0f },
    { 0.0f, 0.0f, 1.0f },

    { 0.0f, 1.0f, 0.0f },
    { 0.0f, 1.0f, 0.0f },
    { 0.0f, 1.0f, 0.0f },

    { 0.0f, -1.0f, 0.0f },
    { 0.0f, -1.0f, 0.0f },
    { 0.0f, -1.0f, 0.0f },

    { -1.0f, 0.0f, 0.0f },
    { -1.0f, 0.0f, 0.0f },
    { -1.0f, 0.0f, 0.0f },

    { 0.0f, 0.0f, -1.0f },
    { 0.0f, 0.0f, -1.0f },
    { 0.0f, 0.0f, -1.0f },

    { 1.0f, 0.0f, 0.0f },
    { 1.0f, 0.0f, 0.0f },
    { 1.0f, 0.0f, 0.0f },

    { 0.0f, 0.0f, 1.0f },
    { 0.0f, 0.0f, 1.0f },
    { 0.0f, 0.0f, 1.0f },
};

std::for_each(vertices.begin(), vertices.end(), [this](glm::vec3 p) { return p * m_size; });

std::vector<unsigned int> indices{
    2, 1, 0,
    2, 0, 3,
    4, 5, 6,
    7, 4, 6,
    10, 9, 8,
    10, 8, 11,
    12, 13, 14,
    15, 12, 14,
    18, 17, 16,
    18, 16, 19,
    20, 21, 22,
    23, 20, 22,
};

std::vector<glm::vec2> uvs{
    { 0.333333f, 0.500000f },
    { 0.333333f, 0.000000f },
    { 0.000000f, 0.000000f },
    { 0.000000f, 0.500000f },
    { 0.000000f, 1.000000f },
    { 0.000000f, 0.500000f },
    { 0.333333f, 0.500000f },
    { 0.333333f, 1.000000f },
    { 1.000000f, 1.000000f },
    { 1.000000f, 0.500000f },
    { 0.666666f, 0.500000f },
    { 0.666666f, 1.000000f },
    { 0.333333f, 1.000000f },
    { 0.333333f, 0.500000f },
    { 0.666666f, 0.500000f },
    { 0.666666f, 1.000000f },
    { 0.340000f, 0.500000f },
    { 0.666666f, 0.500000f },
    { 0.666666f, 0.000000f },
    { 0.340000f, 0.000000f },
    { 0.666666f, 0.500000f },
    { 0.666666f, 0.000000f },
    { 1.000000f, 0.000000f },
    { 1.000000f, 0.500000f },
};

// use AbstractMesh to render this

The only difference in rendering skybox is the use of samplerCube for texture in the fragment shader:

#version 410

in VS_OUT
{
    vec3 textureCoords;
} fsIn;

out vec4 fragmentColor;

uniform samplerCube cubeMap;

void main() {
    fragmentColor = texture(cubeMap, fsIn.textureCoords);
}

In vertex shader, however, the textureCoords variable is set using the camera view matrix passed as a uniform variable:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexUV;

out VS_OUT
{
    vec3 textureCoords;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;

void main()
{
    vec4 pos = projection * view * vec4(vertexPosition, 1.0);
    gl_Position = pos; //.xyww;
    vsOut.textureCoords = vertexPosition;
}

And passing the matrix to the uniform variable in the application (thanks globjects for a beautiful API) :

skyboxRenderingProjectionTransformationUniform->set(cameraProjection);
skyboxRenderingViewTransformationUniform->set(glm::mat4(glm::mat3(cameraView)));

Point lighting

When people are talking about point light sources, they usually mean light attenuation (light intensity fading with the distance from the light source) and object shading calculation for each light source around a given polygon (you calculate the Blinn-Phong shading considering each light source and light attenuation).

A bit more interesting here is the shadow casting technique. Instead of rendering shadow map as a simple 2D texture, you will need to utilize the cubemap, rendering an entire scene for each light source six times. But bear in mind: this is a very calculation-intensive technique, so refrain from doing so on the large landscapes. Alternatively, you should limit the light sources to the ones that are visible (or whose shadows are visible) in the camera space.

The implementation is quite interesting - it is an inversion of a cubemap rendering technique - instead of sampling the cubemap texture we now render to the cubemap texture. Initializing this texture:

auto pointShadowMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_R), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

pointShadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

pointShadowMapTexture->bind();

const auto shadowMapSize = 2048;

for (auto i = 0; i < 6; ++i)
{
    ::glTexImage2D(
        static_cast<::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_X + i),
        0,
        GL_DEPTH_COMPONENT,
        shadowMapSize,
        shadowMapSize,
        0,
        GL_DEPTH_COMPONENT,
        GL_FLOAT,
        nullptr);
}

pointShadowMapTexture->unbind();

This texture will be attached to a framebuffer:

auto pointShadowMappingFramebuffer = std::make_unique<globjects::Framebuffer>();
pointShadowMappingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), pointShadowMapTexture.get());

pointShadowMappingFramebuffer->printStatus(true);

Since the cubemap is effectively six different textures, we will need to render to all of them, using different light projection matrices. We can somewhat optimize this by using geometry shader. Passing the matrices could be optimized using the SSBO:

struct alignas(16) PointLightData
{
    glm::vec3 lightPosition;
    float farPlane;
    std::array<glm::mat4, 6> projectionViewMatrices;
};

auto pointLightDataBuffer = std::make_unique<globjects::Buffer>();

// calculate the 6 light projection matrices

glm::mat4 pointLightProjection = glm::perspective(glm::radians(90.0f), static_cast<float>(shadowMapSize / shadowMapSize), nearPlane, farPlane);

std::array<glm::mat4, 6> pointLightProjectionViewMatrices{
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(-1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 1.0f, 0.0f), glm::vec3(0.0f, 0.0f, 1.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, -1.0f, 0.0f), glm::vec3(0.0f, 0.0f, -1.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 0.0f, 1.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
    pointLightProjection * glm::lookAt(pointLightPosition, pointLightPosition + glm::vec3(0.0f, 0.0f, -1.0f), glm::vec3(0.0f, -1.0f, 0.0f)),
};

PointLightData pointLightData{ pointLightPosition, farPlane, pointLightProjectionViewMatrices };

pointLightDataBuffer->setData(pointLightData, static_cast<gl::GLenum>(GL_DYNAMIC_COPY));

Then, on each frame, bind the SSBO and render scene to a framebuffer with cubemap texture attached to it:

// bind the buffer on each frame

pointShadowMappingFramebuffer->bind();

pointLightDataBuffer->bindBase(GL_SHADER_STORAGE_BUFFER, 5);

pointShadowMappingProgram->use();

// render frame

pointLightDataBuffer->unbind(GL_SHADER_STORAGE_BUFFER, 5);

pointShadowMappingFramebuffer->unbind();

pointShadowMappingProgram->release();

For shadow mapping pass, the shaders are as following:

geometry shader:

#version 430

layout (triangles) in;

// we emit 6 triangles for one input triangle - to be written to 6 textures of the cubemap
layout (triangle_strip, max_vertices = 18) out;

struct PointLight
{
   vec3 lightPosition;
   float farPlane;
   mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer pointLightData
{
   PointLight pointLight;
};

out vec4 fragmentPosition;

void main()
{
    for (int face = 0; face < 6; ++face)
    {
        gl_Layer = face;

        for (int vertex = 0; vertex < 3; ++vertex)
        {
            fragmentPosition = gl_in[vertex].gl_Position;
            gl_Position = pointLight.projectionViewMatrices[face] * fragmentPosition;
            EmitVertex();
        }

        EndPrimitive();
    }
}

fragment shader:

#version 430

in vec4 fragmentPosition;

struct PointLight
{
    vec3 lightPosition;
    float farPlane;
    mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer pointLightData
{
    PointLight pointLight;
};

void main()
{
    float distance = length(fragmentPosition.xyz - pointLight.lightPosition) / pointLight.farPlane;

    gl_FragDepth = distance;
}

And for the final render pass (sampling the shadow cubemap) :

vertex shader:

#version 430

in vec4 fragmentPosition;

struct PointLight
{
    vec3 lightPosition;
    float farPlane;
    mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer pointLightData
{
    PointLight pointLight;
};

void main()
{
    float distance = length(fragmentPosition.xyz - pointLight.lightPosition) / pointLight.farPlane;

    gl_FragDepth = distance;
}

fragment shader:

#version 430

layout (location = 0) out vec4 fragmentColor;

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
} fsIn;

uniform samplerCube shadowMap;
uniform sampler2D diffuseTexture;
uniform sampler2D specularMapTexture;
uniform sampler2D emissionMapTexture;

struct PointLight
{
    vec3 lightPosition;
    float farPlane;
    mat4 projectionViewMatrices[6];
};

layout (std430, binding = 5) buffer pointLightData
{
    PointLight pointLight;
};

uniform vec3 lightColor;
uniform vec3 cameraPosition;

// TODO: make these params uniforms
// uniform vec3 ambientColor;
// uniform vec3 diffuseColor;
// uniform float materialSpecular;
uniform vec3 emissionColor;

float attenuation_constant = 1.0;
float attenuation_linear = 0.09;
float attenuation_quadratic = 0.032;

const int pcfSamples = 20;
vec3 pcfSampleOffsetDirections[pcfSamples] = vec3[]
(
   vec3(1, 1, 1), vec3(1, -1, 1), vec3(-1, -1, 1), vec3(-1, 1, 1),
   vec3(1, 1, -1), vec3(1, -1, -1), vec3(-1, -1, -1), vec3(-1, 1, -1),
   vec3(1, 1, 0), vec3(1, -1, 0), vec3(-1, -1, 0), vec3(-1, 1, 0),
   vec3(1, 0, 1), vec3(-1, 0, 1), vec3(1, 0, -1), vec3(-1, 0, -1),
   vec3(0, 1, 1), vec3(0, -1, 1), vec3(0, -1, -1), vec3(0, 1, -1)
);

float shadowCalculation(vec3 normal)
{
    vec3 fragmentToLight = fsIn.fragmentPosition - pointLight.lightPosition;
    float occluderDepth = texture(shadowMap, fragmentToLight).r * pointLight.farPlane;
    float thisDepth = length(fragmentToLight);

    float bias = 0.15;

    // PCF
    /*float shadow = 0.0;
    float viewDistance = length(fsIn.fragmentPosition - cameraPosition);
    float diskRadius = (1.0 + (viewDistance / farPlane)) / 50.0;

    for (int i = 0; i < pcfSamples; ++i)
    {
        float closestDepth = texture(shadowMap, fragmentToLight + pcfSampleOffsetDirections[i] * diskRadius).r * farPlane;

        if (thisDepth - bias < closestDepth)
            shadow += 1.0;
    }

    shadow /= float(pcfSamples);*/

    float shadow = (thisDepth - bias) < occluderDepth ? 1.0 : 0.0;

    return shadow;
}

void main()
{
    vec3 color = texture(diffuseTexture, fsIn.textureCoords).rgb;
    vec3 normal = normalize(fsIn.normal);

    // ambient
    vec3 ambient = 0.3 * color;

    // diffuse
    vec3 lightDirection = normalize(pointLight.lightPosition - fsIn.fragmentPosition);
    float diff = max(dot(lightDirection, normal), 0.0);
    vec3 diffuse = diff * lightColor;

    // specular
    vec3 viewDirection = normalize(cameraPosition - fsIn.fragmentPosition);
    vec3 halfwayDirection = normalize(lightDirection + viewDirection);
    float spec = pow(max(dot(normal, halfwayDirection), 0.0), 64.0);
    vec3 specular = spec * lightColor;

    // attenuation
    float lightRadius = 10.0;
    float distance = length(pointLight.lightPosition - fsIn.fragmentPosition) / lightRadius;
    float attenuation = 1.0 / (attenuation_constant + attenuation_linear * distance + attenuation_quadratic * (distance * distance));

    // calculate shadow; this represents a global directional light, like Sun
    float shadow = shadowCalculation(normal);

    // these are the multipliers from different light maps (read from corresponding textures)
    float specularCoefficient = texture(specularMapTexture, fsIn.textureCoords).r;
    float emissionCoefficient = texture(emissionMapTexture, fsIn.textureCoords).r;

    // vec3 lighting = ((shadow * ((diffuse * attenuation) + (specular * specularCoefficient * attenuation))) + (ambient * attenuation)) * color + (emissionColor * emissionCoefficient);
    vec3 lighting = ((shadow * ((diffuse) + (specular * specularCoefficient))) + (ambient)) * color + (emissionColor * emissionCoefficient);

    fragmentColor = vec4(lighting, 1.0);
}

Note how the shadow map is samplerCube and sampling it is using the vec3 instead of vec2 as with planar (2D) textures. Also note how lighting has something called "attenuation" now - this is the fading-out factor - the further the pixel is from the light source - the more bleak the lighting effect would be.

Reflections

Reflections on the water surface are relatively simple to implement - for each pixel you calculate the direction to the camera upside-down (simply create a new view matrix with "up" vector pointing down) and then use the GLSL function reflect() to calculate the reflected vector. You then render take the color at the reflected position and add it (multiplied by some transparency factor) to the original pixel' color.

The concept of cubemaps is not only useful for skyboxes, but could also be used to calculate reflections on the objects. For each reflective object you render the scene from its position to the cubemap texture (so you will have to render the scene six times, which is huge overhead in itself) and when rendering the object, you reflect the camera view vector (vector from the pixel in world space to camera position) and sample the cubemap with that vector. You then add the sampled color to the source pixel color to receive a nice reflective surface effect.

This technique is very similar to both point light shadow mapping and skybox rendering combined.

We will need the cubemap texture to render to:

auto reflectionMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP));

reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_R), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

reflectionMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));

reflectionMapTexture->bind();

const auto reflectionMapSize = 2048;

for (auto i = 0; i < 6; ++i)
{
    ::glTexImage2D(
        static_cast<::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_X + i),
        0,
        GL_RGBA8,
        reflectionMapSize,
        reflectionMapSize,
        0,
        GL_RGBA,
        GL_UNSIGNED_INT,
        nullptr);
}

reflectionMapTexture->unbind();

And on top of that, for things to not look mangled, we will need a depth texture. Which also has to be a cubemap:

auto reflectionMapDepthTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_CUBE_MAP));

reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));

reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_R), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));

reflectionMapDepthTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(0.0f, 0.0f, 0.0f, 1.0f));

reflectionMapDepthTexture->bind();

for (auto i = 0; i < 6; ++i)
{
    ::glTexImage2D(
        static_cast<::GLenum>(GL_TEXTURE_CUBE_MAP_POSITIVE_X + i),
        0,
        GL_DEPTH_COMPONENT,
        reflectionMapSize,
        reflectionMapSize,
        0,
        GL_DEPTH_COMPONENT,
        GL_FLOAT,
        nullptr);
}

reflectionMapDepthTexture->unbind();

They both are then attached to a framebuffer:

auto reflectionMappingFramebuffer = std::make_unique<globjects::Framebuffer>();
reflectionMappingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), reflectionMapTexture.get());
reflectionMappingFramebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), reflectionMapDepthTexture.get());

// tell framebuffer it actually needs to render to **BOTH** textures, but does not have to output anywhere (last NONE argument, iirc)
reflectionMappingFramebuffer->setDrawBuffers({ static_cast<gl::GLenum>(GL_COLOR_ATTACHMENT0), static_cast<gl::GLenum>(GL_NONE) });

reflectionMappingFramebuffer->printStatus(true);

Then, we have to render scene... well, 6 times - once for each side of the reflection cubemap:

// this is a cubemap, hence aspect ratio **must** be 1:1
glm::mat4 reflectionProjection = glm::perspective(glm::radians(90.0f), 1.0f, nearPlane, farPlane);

// TODO: technically, this should be calculated as AABB / 2
const auto reflectionOffset = glm::vec3(0.0f, 0.05f, 0.0f);

reflectionMappingProgram->setUniform("projectionViewMatrices[0]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)));
reflectionMappingProgram->setUniform("projectionViewMatrices[1]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(-1.0f, 0.0f, 0.0f), glm::vec3(0.0f, -1.0f, 0.0f)));
reflectionMappingProgram->setUniform("projectionViewMatrices[2]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(0.0f, 1.0f, 0.0f), glm::vec3(0.0f, 0.0f, 1.0f)));
reflectionMappingProgram->setUniform("projectionViewMatrices[3]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(0.0f, -1.0f, 0.0f), glm::vec3(0.0f, 0.0f, -1.0f)));
reflectionMappingProgram->setUniform("projectionViewMatrices[4]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(0.0f, 0.0f, 1.0f), glm::vec3(0.0f, -1.0f, 0.0f)));
reflectionMappingProgram->setUniform("projectionViewMatrices[5]", reflectionProjection * glm::lookAt(reflectiveModelPosition + reflectionOffset, reflectiveModelPosition + glm::vec3(0.0f, 0.0f, -1.0f), glm::vec3(0.0f, -1.0f, 0.0f)));

reflectionMappingFramebuffer->bind();

reflectionMappingProgram->use();

// render scene EXCEPT the mirror object

reflectionMappingFramebuffer->unbind();

reflectionMappingProgram->release();

For this we use geometry shader:

#version 430

layout (triangles) in;

// we emit 6 triangles for one input triangle - to be written to 6 textures of the cubemap
layout (triangle_strip, max_vertices = 18) out;

in VS_OUT
{
    vec4 vertexPosition;
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
} gsIn[];

out GS_OUT
{
    vec4 vertexPosition;
    vec4 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
} gsOut;

uniform mat4 projectionViewMatrices[6];

// out vec4 fragmentPosition;

void main()
{
    for (int face = 0; face < 6; ++face)
    {
        gl_Layer = face;

        for (int vertex = 0; vertex < 3; ++vertex)
        {
            gsOut.vertexPosition = gsIn[vertex].vertexPosition;
            gsOut.fragmentPosition = projectionViewMatrices[face] * gsIn[vertex].vertexPosition;
            gsOut.normal = gsIn[vertex].normal;
            gsOut.textureCoords = gsIn[vertex].textureCoords;

            gl_Position = projectionViewMatrices[face] * gsIn[vertex].vertexPosition;

            EmitVertex();
        }

        EndPrimitive();
    }
}

Vertex shader for reflection mapping:

#version 430

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoords;

out VS_OUT
{
    vec4 vertexPosition;
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 modelTransformation;

void main()
{
    vsOut.vertexPosition = modelTransformation * vec4(vertexPosition, 1.0);
    vsOut.fragmentPosition = vec3(modelTransformation * vec4(vertexPosition, 1.0));
    vsOut.normal = vertexNormal;
    vsOut.textureCoords = vertexTextureCoords;

    gl_Position = vsOut.vertexPosition;
}

And fragment shader:

#version 430

in GS_OUT
{
    vec4 vertexPosition;
    vec4 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
} fsIn;

uniform sampler2D diffuseTexture;

void main()
{
    vec4 color = texture(diffuseTexture, fsIn.textureCoords);

    // TODO: add lighting component here
    gl_FragColor = color;
}

Note how the output is a 4-component vector instead of a regular 3-component one.

When rendering the reflective object on the final render pass, we sample the reflection cubemap texture and use the reflect function in vertex shader to calculate the sampling vector from the camera view vector:

#version 430

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
    vec3 reflectedDirection;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;

uniform vec3 cameraPosition;

void main()
{
    vsOut.fragmentPosition = vec3(model * vec4(vertexPosition, 1.0));
    vsOut.normal = vertexNormal;
    vsOut.textureCoords = vertexTextureCoord;
    vsOut.reflectedDirection = reflect(normalize(vsOut.fragmentPosition - cameraPosition), vertexNormal);

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

In the fragment shader we then combine the color sampled from reflection texture and the object' own (albedo) color:

#version 430

layout (location = 0) out vec4 fragmentColor;

in VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoords;
    vec3 reflectedDirection;
} fsIn;

uniform samplerCube reflectionMap;
uniform sampler2D diffuseTexture;

// uniform vec3 lightColor;
uniform vec3 cameraPosition;

uniform mat4 model;

void main()
{
    vec4 reflectionColor = texture(reflectionMap, fsIn.reflectedDirection);
    vec4 albedoColor = texture(diffuseTexture, fsIn.textureCoords);

    fragmentColor = mix(albedoColor, reflectionColor, 0.6);
}

MooFoo blogs

Data compression

Canonical Huffman codes

ZIP, DEFLATE

JPEG, DHT

Clean before running a Spring Boot application in IntelliJ Idea

Modal text editors

IntelliJ Idea, baseline

NeoVim

Helix

Kakoune

Final thoughts

Strongly-typed front-end: experiment 2, simple application, in Gleam / Lustre

Contents

Iterating a vector in C++

Standard library iterators

Caching impact

Conclusion

Gantt chart. Part 4

Contents

Implementation steps

So why Redux is bad?

Bun is still undercooked

What is Bun

Bun is ridiculously fast

Some things work out of the box

Plugins

svgr plugin

vanilla-extract plugin

sentry plugin

Bundling

Configuration

Running tests

react-testing-library

Browser APIs

test.each

Mocks

ObjectContaining matchers

Using Fragment import alongside <>

Ace editor

Bun's meat and potatoes

Bottom line

Strongly-typed front-end: experiment 2, simple application, in PureScript / Halogen

Contents

More than TypeScript

IO impact

Simulating network outages in Docker

How unique are your bundles?

TypeScript classes are not what you think

How not to use monads

Improvements

Use IO monad only for the functions which interact with the outside world

Reduce calls to IO functions

Eliminate the abuse of Maybe Bool

Reduce the nested if statements - use guard expressions

Use existing functions on monads instead of case statements

End result

Floyd-Warshall algorithm, revised

Parser monad

Helper functions

Basic parsers

Parsing a digit

Parsing a number (as a string)

Parsing a number (as a non-negative number)

Parsiung a potentially negative number

Combining parsers

ConfigScript

ANTLR version incompatibility

Actual implementation

OpenGL: advanced samples

Setting up

Basics I promised not to focus on

Getting started with SFML

Getting started with globjects

Camera and getting started with glm

Optimization techniques

Passing data to the shader

Deferred rendering

Batch geometry rendering

Texture handles

`svgr` plugin

`vanilla-extract` plugin

`sentry` plugin

`react-testing-library`

`test.each`

`ObjectContaining` matchers

Using `Fragment` import alongside `<>`

Use `IO` monad only for the functions which interact with the outside world

Reduce calls to `IO` functions

Eliminate the abuse of `Maybe Bool`

Reduce the nested `if` statements - use guard expressions

Use existing functions on monads instead of `case` statements