I write C++ code almost every day, be it for work or for fun (yes, we exist). While I have great tools for benchmarking runtime performance in my toolbox, I never really tried "benchmarking compile times".

With build times continuously creeping up regardless of the build caches I throw at the problem, I wanted to understand what really takes so long to compile.

I don't intend to showcase specific problems, but rather show how one can analyze their own codebase to find the pain points.

How do we do this?

The idea is to use the -ftime-trace compiler flag available in Clang to generate a JSON trace of the compilation process and then visualize it. Thanks to Aras Pranckevičius and Anton Afanasyev, this is part of Clang since version 9.

You can use this with any build system that supports passing custom flags to the compiler. If you are using Bazel and building within the sandbox, have a look at this Bazel issue as you might need to jump through some hoops to get the trace file out of the sandbox. Enabling the --sandbox_debug option is what I did.

Example program

I will use a simple example program that uses std::variant and std::visit with multiple types to demonstrate what the generated trace looks like.

Don’t look too much into what the code is doing, it’s just an example of a variant with a couple types, not doing anything fancy or important.

#include <cstddef>
#include <string>
#include <variant>
#include <vector>

struct Sizer
{
    std::size_t operator()(int) const { return sizeof(int); }
    std::size_t operator()(double) const { return sizeof(double); }
    std::size_t operator()(const std::string &v) const { return v.size(); }
    std::size_t operator()(bool) const { return sizeof(bool); }
    std::size_t operator()(const std::vector<int> &v) const { return v.size() * sizeof(int); }
};

int main()
{
    using Value = std::variant<int, double, std::string, bool, std::vector<int>>;
    std::vector<Value> values{42, 3.14, std::string("hello"), true, std::vector{1, 2, 3}};

    std::size_t total{0};
    for (const auto &v : values)
    {
        total += std::visit(Sizer{}, v);
    }
    return static_cast<int>(total);
}

Compiler Explorer

Generating the trace

First, make sure you are using Clang as your compiler. Then, add the -ftime-trace flag to your compiler flags. By default, this will generate a json file named <name>.json just next to the object file.

Visualizing the trace

Once you have the trace file, you can use either the chrome://tracing tool in a Chromium-based browser or the Perfetto UI to visualize it.

Flame graphs are great. Clicking one of the sections gives us information about what operation it is related to.

Unsurprisingly, given how simple our program is, a big portion of the time is spent parsing the included files.

In a larger program, if you are seeing such expensive headers, it might be a good idea to check and see if the header in question can be split up or thinned down a bit into smaller headers that can be included as needed.

Tools like Include What You Use are also great to identify unnecessary includes across your codebase. Of course, in our example case, there is not much to improve on this side.

Going further in the timeline, we can also clearly see the time spent instantiating our variant and the vector of variants.

What now?

Go ahead and try this out on your own codebase! If you are looking for more details, check out the Clang developer documentation on Performance Investigation and the Perfetto documentation.

There is two main approaches when it comes to delaying something in embedded systems. In most basic terms, busy waiting is the act of giving a long and unnecessary work to the CPU, so you can block it from processing the next operation. This is wasteful both in terms of CPU resources and power. In contrast, interrupt based waiting is the act of starting a timer which will wait on its own without blocking the CPU until the time comes. When the timer runs out; it will go ahead and interrupt the CPU and ask it to do whatever task that is assigned to that timer.

This project is an example of interrupt based waiting, but I will explain both options with examples. Below you can see a chart which shows the difference of busy waiting and interrupt based waiting.

https://users.ece.utexas.edu/%7Evalvano/Volume1/E-Book/C12_Interrupts.htm

1. Busy Waiting - the blocking selfish way

The selfish and the easy way of delaying something is just blocking the CPU with some unnecessary work, usually by creating loops that accomplish nothing but look like they do to the compiler. If compiler realises that your new shiny loop is not doing anything useful it will just throw it out in the optimization phase. You can always disable or change the optimization level, but I will just assume that you are using the default level of optimization.

An example to this can be; assigning or reading a value from a volatile variable over and over. The below code will block the CPU for however long it takes for the CPU to process that operation for 100 times.

// Define a volatile variable
#define SOMEVAR (*(volatile uint8_t *) (0x80))

for (int i = 0; i < 100; ++i) { 
    // Set a value to the volatile variable. 
    // If this is not volatile, the compiler will run this only once and 
    // optimize out the rest of it to save time and resources since it 
    // basically does nothing. But because it  is volatile it has no way
    // to know if this value will be the same since declaring as volatile 
    // means that this value can change any time by
    // a factor that is not known to the compiler.
    SOMEVAR = 0x00;
}

If manually implemented this option requires the developer to perform some calculations in order to understand how long this delay will take, or you can always do trial and error to guesstimate the time it takes and try to get it to a value close to what you are looking for but that's going to take some time going back and forth and will require an oscilloscope to get it at least close to the value you looking for (no, you can't use your phone's timer).

The delay(ms) function that all of us know from the Arduino lib is an example of busy waiting.

2. Interrupt Based Waiting - non blocking way

Another and arguably the better way of waiting for something is using timers with interrupts to free the CPU from the waiting task. This works as in that when the timer is triggered it will go ahead and interrupt the CPU from whatever it is doing and will ask it to do what it wants immediately. For more details about how interrupts work, visit the website of University of Texas at Austin.

An example code snippet for an interrupt based timer using the 16-bit Timer 1 of ATmega328P can be found below.

ISR (TIMER1_OVF_vect) {
    PORTB ^= (1 << PORTB5); // Toggle the 5th data register of PORTB
    TCNT1 = 64755; // 50 ms for 16MHz clock
}

int main() {
    // Set 5th data direction register of PORTB. A set value means output
    DDRB = (1 << DDB5);
    // 50 ms for 16MHz clock
    TCNT1 = 64755;
    // Set normal counter mode
    TCCR1A = 0x00;
    // Set 1024 pre-scaler
    TCCR1B = (1<<CS10) | (1<<CS12);
    // Set overflow interrupt enable bit
    TIMSK1 = (1 << TOIE1); 
    sei(); // Enable interrupts globally
    while(1)
    {
        // Do anything, this timer is non-blocking.
        // It will interrupt the CPU only when needed
    }
    // Add return so compilers don't cry about it being missing.
    // Under normal circumstances this will never be hit
    return 0;
}

This creates a non-blocking interrupt based timer that will toggle the 5th data register of PORTB every 50ms. But how do we calculate how long we need to wait? Unless we need to change the wait time in the runtime we should pre-compute the value for TCNT1 which is in basic terms what determines the wait time.

Calculate the wait time

So how do we calculate the value we put on TCNT1 according to the wait time we want.

1. Divide Frequency to pre-scaler

Since we are using the Timer 1 of the ATmega328P and this timer is running on 16MHz we divide this value by our selected pre-scaler of 1024.

Freq/Pre-scaler => 16MHz/1024 = 15.625KHz

2. Calculate the tick time

Then we get the 1 over of this value to find our tick time. Which is 64 micro seconds.

1/15.625KHz = 64μs

3. Get delay time

Our clock is 16-bit therefore we need to subtract t/64μs from 2^16 = 65536 which will in turn give us the value we need to put in TCNT1. Where t is the time we want to wait in milliseconds.

TCNT1 = 2^16 - (t/64μs)

For example, if we want to have wait time of 100ms we would calculate it as;

TCNT1 = 2^16 - (100/0.064) => TCNT1 = 65536 - 1562 => TCNT1 = 63974

Example output

Below you can see an example output displayed on a simulated oscilloscope when the timer is set to have a delay of 1 second.

Source code

The full source code to compliment this post is available in my Github profile. There you can find an example interrupt based timer that toggles a pin every 50ms by changing the data registers. Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

TP-Link TD864W is a modem/router combo that is widely used in Turkey mostly because it was given for free to every DSL subscriber by a major ISP in Turkey for years. Its software was modified for use by that particular ISP by TP-Link and is very limited on functionality especially when it comes to setting manual DNS and VPN settings. Today I am going to try to find a way into the shell of this modem/router combo firstly by correctly identifying the serial ports on the PCB, reverse engineering the pinouts with a basic multimeter, then connecting a USB UART so I can read the output and issue commands to the device from my computer.

Figure 1. An external photo

Examining the board

Unfortunately, this device has no FCC filing so we have to take a look ourselves. When we take the very annoying plastic cover off and take a look at internal photos of the device we can see a very convenient 4-pin port which looks very much like a UART port (Outlined in red below in Figure 2). We can also see some chips which might come in handy later such as RTL8676 outlined in yellow and FL128SAIF00 outlined in blue in Figure 2.

Figure 2. An internal photo of the device

Figuring out the pinout

Unfortunately, there is no standard for the UART pinouts therefore we will need to figure it out ourselves. There are different ways to identify the pinout of the port such as using a logic analyzer or a multimeter. Since I don't have a logic analyzer we will take the slower route and check every pin against R(ground), R(Vcc), and also watch their voltage range while the device is booting in order to identify their uses. But before we can do this we need a Ground and Vcc reference.

That's where the aforementioned RTL8676 and FL128SAIF00 chips come in handy. Since there is no way I'm gonna manage to freehand probe the pins on the RTL8676 (Outlined in yellow in Figure 2.) we are going to look for the datasheet of FL128SAIF00 (Outlined in blue in Figure 2.). If we take a look at the datasheet of FL128SAIF00, we can see that the pin-2 is Vcc and pin-10 is GND. We are going to use these pins as our Vcc and GND reference respectively while probing for our UART pins.

Figure 3. Close-up of the pins

Figure 4. Measurement results for the pins

Now that the measurements are done we can see that pin-4 is Vcc since it has zero resistance against Vcc and has a stable 3V3 output, we can also see that pin-3 is Ground since it has no resistance against GND and is always on zero Volts. This leaves us the pin-1 and pin-2 in which we have to decide which is TX and RX. The easiest way to do this is just to connect our multimeter in voltage measurement mode and check each pin while the device is booting. As long as the UART is not disabled, there will be a lot of output on the TX pin which means it will fluctuate in value. In this case, the pin-2 is TX since its voltage fluctuates between 1V7 to 3V3 during boot.

The best way to confirm our findings is to connect a USB UART interface and create a serial connection between our computer and the device.

Figure 5. USB UART interface connected to the pins

Connecting to the device

After a mediocre solder work, we can connect our USB UART interface to the pins that we have identified earlier. Using a serial monitor such as PuTTY we can connect to our device. One thing that we still don't know about the connection is the baud rate. The best way to find the baud rate of a device is to just analyze it with an oscilloscope. Well, I don't have an oscilloscope lying around in my home so we will once again take the slower route and just try some common baud rates starting from 115200 because from experience this is what most of the routers I have seen run on.

Figure 6. Settings for PuTTY serial connection

Figure 7. Boot sequence output

Thankfully we found the right baud rate on our first try, if you happen to be not as lucky as me you can check this question on the electronics Stack Exchange to get a better idea around the topic. And look what the 4th line says. Let's reboot and spam ESC until we get to the boot console.

Figure 8. Entering the boot console

Well, that was easy. Let's see what we can do here in Figure 9.

Figure 9. Boot console commands

Let's try the bootline first it is probably going to ask us to input a string, lets see if it has a limit on how many chars can be put on maybe we can do some easy buffer overflow here.

Figure 10. Bootline and info commands

Well, it has a set limit on the input so we are not going to get any easy buffer overflows here, we can later come back to this with edge cases if we need to. We also run the info command which displays some basic but very useful information about the device including our newly set bootline.

Figure 11. Displaying memory with d command

The d command takes a memory address and length parameter then displays the memory region for that address in cleartext. We can easily use this to dump the memory of the device for further research. You can find a basic Python script I wrote which takes the PuTTY log output and clears anything but the memory data and gives you a clean memory dump so you can use it for further research or reverse-engineering on my Github within one week from the publication date.

The rest of the commands in the boot console seems self-explanatory. They even gave us a TFTP command which we can use to flash a new firmware to the device from another host.