ops.tips

A Raspberry PI Concourse Worker

Ciro S. Costa — Sun, 19 May 2019 00:00:00 +0000

Hey,

I’ve recently bought a Raspberry Pi 3B+, and that seemed like a great target for a Concourse worker run at.

It turns out that just the process of building and adapting the Concourse binary itself was already an interesting thing to do, so here I share the lessons learned and how that process looked like.

To provide ARM-compiled binaries for node_extra_exporter (a Rust-based Prometheus exporter for exposing some metrics that the traditional node_exporter doesn’t expose), it felt like having an ARM-based machine in my Concourse cluster would be a great idea - have an extra job for building the ARM binary and there you go!

If you’d like to know what does it take to get a Concourse worker ready to take workloads on ARM, make sure you stick to the end.

Concourse workers

Regardless of your knowledge of Concourse, the tl;dr is that being a “continuous thing-doer”, it needs some form of compute nodes to do the things you want - these nodes are the workers.

To perform the job of “doing things”, in any platform, these workers are made of three components:

a “containerizer”, that manages containers
a “volumizer”, that manages volumes
“beacon”, the piece that registers the worker against the cluster and manages its lifecycle

In the case of Linux, of those three components, two¹ (the “volumizer” and “beacon”) are already part of the concourse binary. This means that we not only need to build concourse, but we also need to build the “containerizer”.

¹ there’s an expection, but that won’t be covered here.

Compiling the Concourse binary

Being Concourse a project entirely written in Go (actually, there’s also the UI, which is in Elm), all that we need at this stage is the Go toolchain.

As almost all of the Go code necessary for building Concourse is now under a single repository (github.com/concourse/concourse), and Go modules is the way that dependencies are handled there, this is what should be the most straightforward part.

Given that installing Go is quite straightforward, I decided to do that right in my Raspberry PI (why not?).

This means that just the following two steps are required:

# clone the main concourse repository
#
git clone https://github.com/concourse/concourse .


# build the main Concourse binary and put the result of
# the compilation into `$GOPATH/bin/`.
#
go install -v ./cmd/concourse

The problem though, is that the Raspberry PI is not as powerful as one would imagine: despite the 4 core SoC, it takes no less than seven minutes once all of the dependencies are already in place. Yeah, SEVEN MINUTES already having fetched all of the dependencies. Without the dependencies: 20 minutes.

For a full picture of the dashboard, click here to check out the interactive snapshot).

As a side note, it was quite interesting for me to see how there are clearly very different phases that the compiler toolchain goes trough when performing the build (not including the dependency fetching that is not in this panel).

Cross compiling Go code

Knowing that if I’d need to recompile this multiple times, it’d be crazy slow and, thus, super time consuming, I decided to go with cross compiling - this way we could just use all of the speed we have available and, in the end, just ship the binaries.

Another benefit is that it’d be easy for anyone to build it too! No need for an actual Raspberry PI (or any ARMv7) to build Concourse.

The good news is that Go makes that whole job easy for us, making the whole cross compilation dependant on just few flags:

GOOS the name of the target operating system
GOARCH the name of the target architecture
GOARM the version of ARM that want to target

For Go code that is free of any calls to C code (via CGO), this “Just Works™” by the magic of the default Go toolchain - the go compilation infrastructure has all the necessary bits to perform the right translation to the right architectures and OSes that it supports.

If you’re curious about the intermediate representation that the Go compiler generates and how that becomes machine-dependant code, checkout the following example views of Go’s SSA:

ps.: to compile reproduce: GOSSAFUNC=main go build -gcflags "-S" main.go

However, being free of calls to C code is not a property of all projects - in the case of Concourse itself, dex, one of our dependencies, depends on go-sqlite3, which has bindings to C, which means that we now depend on CGO.

├ github.com/concourse/concourse/skymarshal/storage
  ├ github.com/concourse/dex/storage
  ├ github.com/concourse/dex/storage/sql
    ├ database/sql
    ├ database/sql/driver
    ...
    └ github.com/mattn/go-sqlite3
      ├ database/sql
      ├ database/sql/driver
      ..
      ├ unsafe
      └ C                       << CGO!!

If you’re curious about an example in go-sqlite3 where CGO gets used, check out sqlite3.go.

To see what I mean by “CGO doesn’t ‘just works’”, let’s try doing cross compilation with just those flags:

CGO_ENABLED=1 \
        GOARCH=arm \
        GOARM=7 \
        GOOS=linux \
                go build ./cmd/concourse/
# runtime/cgo
gcc: error: unrecognized command line option '-marm'; 
            did you mean '-mabm'?

And the reason why it wouldn’t work out of the box makes sense: when it comes to CGO, you’re not only in the realm of the Go toolchain, but you’re also relying on the infrastructure to build the C code as well, and there, things are slightly different. Let’s expand on it.

Compiling CGO code

As an example of how cross compilation with CGO looks like, let’s assume that we have a super extra very efficient library in C that is optimized to print strings to stdout.

First, starting with the declaration of the function we want to consume from Go (in the printer.h file):

#include <stdio.h>
void super_optimized_print(char* str);

Then, in the definition (printer.c):

#include "printer.h"
void super_optimized_print(char* str)
{
	printf("%s\n", str);
	return;
}

And now, finally, our Go code that uses CGO (main.go):

package main

// #include "printer.h"
// #include <stdlib.h>
import "C"
import "unsafe"

func main() {
	str := C.CString("hello world")
	defer C.free(unsafe.Pointer(str))

	C.super_optimized_print(str)
}

We can then try to build all of that code to our own OS and ARCH, and nothing really needs to be changed: you give the package to the go compiler, and let it do its job:

go build -v .

As, in this case, our C code is definitely portable, and we’re targetting our own machine architecture, all works. Let’s now try to build to a different platform though.

Cross compiling CGO code

However, if, again, we try the cross compilation, it’ll fail with the very same error we saw before:

CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 go build -v .
gcc: error: unrecognized command line 
     option '-marm'; did you mean '-mabm'?

The good news is that without even looking at the Go code, we can see how that separation of “who compiles what” happens by tracing all of the execves that happen:

PCOMM     PID      PPID     ARGS
--------------------------------------------------------
go        29352    26142    go build .
cgo       29361    29352    cgo -V=full
compile   29362    29352    compile -V=full
compile   29365    29352    compile -V=full
compile   29367    29352    compile -V=full
compile   29376    29352    compile -V=full
asm       29381    29352    asm -V=full
asm       29382    29352    asm -V=full
cgo       29394    29352    cgo -objdir /tmp/go-build123931463/b003/...
gcc       29399    29394    gcc -E -dM -marm -I /tmp/go-build1239314...

For the output above, execsnoop from iovisor/bcc was used.

And that seems perfectly reasonable - to build C code, one needs to have a C compiler, which is something totally out of the realm of where the Go team should be focusing on, reason why it calls out to a C compiler.

The failure that we see there though comes from the fact that to have GCC performing the cross compilation, we need to install separate packages before, and properly adjust the C compiler to point to the right compiler.

Luckly, letting Go know which compiler to use when performing the build comes down to setting the CC environment variable (see golang/go#gccBaseCmd()).

CGO_ENABLED=1 \
        GOOS=linux \
        GOARCH=arm \
        GOARM=7 \
        CC=arm-linux-gnueabihf-gcc \
        go build -v
# workkss!!

PCOMM     PID    PPID   ARGS
go        30825  26142  go build -v .
cgo       30834  30825  cgo -V=full
compile   30835  30825  compile -V=full
compile   30840  30825  compile -V=full
compile   30841  30825  compile -V=full
compile   30842  30825  compile -V=full
asm       30857  30825  asm -V=full
asm       30862  30825  asm -V=full
asm       30863  30825  asm -V=full
asm       30872  30825  asm -V=full
cgo       30877  30825  cgo -objdir /tmp/go-build130487924/b003/ ...
arm-linux-30882  30877  /usr/bin/arm-linux-gnueabihf-gcc -E -dM ...
cc1       30883  30882  /usr/lib/gcc-cross/arm-linux-gnueabihf/7/cc1 -E ...
...

Now, with the Concourse binary built, we can move to its dependencies.

Building Guardian, the containerizer

As Concourse steps and resource checks run in Containers¹, and there’s a piece of the Concourse worker that is responsible for that, but Concourse as a team is not necessarily in the business of creating container runtimes, Concourse uses a separate component for doing so: Guardian(gdn), an implementation of the Garden interface for container management.

With the process of creating Linux containers being all standardized now (see opencontainers/runtime-spec), Guardian takes the approach of leveraging what’s already there, wrapping runc, the de facto implementation of the Runtime Spec, allowing consumers of the Garden interface to have containers created by Runc, without leaking the implementation details to the Garden interface.

The detail here though, is that while most of gdn is pure Go, there are many bits from runc (gdn’s dependency) that are C based, and Guardian itself depends on other binaries (one being a C program).

Another detail in the process os building gdn is that, by default, gdn is not suitable for multiple architectures due to the way the way that it interacts with runc when it comes to asking runc to block specific syscalls.

// Seccomp represents syscall restrictions
//
// By default, only the native architecture of the kernel is allowed to be used
// for syscalls. Additional architectures can be added by specifying them in
// Architectures.
//
type Seccomp struct {
	DefaultAction Action     `json:"default_action"`
	Architectures []string   `json:"architectures"`
	Syscalls      []*Syscall `json:"syscalls"`
}

That means that, in gdn itself, we needed to have in the architectures slice, ARM included:

 var seccomp = &specs.LinuxSeccomp{
         DefaultAction: specs.ActErrno,
         Architectures: []specs.Arch{
                 specs.ArchX86_64,
                 specs.ArchX86,
                 specs.ArchX32,
+                specs.ArchARM,
         },
         Syscalls: []specs.LinuxSyscall{

As we know how to build all of those in a cross-platform way, we just need to follow the same recipe: set the right compiler, and there you go.

¹: in platforms that support containers.

Building Baggageclaim, the volumizer

As mentioned before, the “volumizer” is already part of the worker, thus, by building concourse (the binary), we already have baggageclaim built.

The only detail for baggageclaim is that if the backing filesystem for it is set to be btrfs, then the machine that runs the worker needs to have the btrfs CLI on it (built to the right platform).

Running the Concourse worker

At this point, we can already have our Concourse worker in a state that it could run - it has all of its dependencies, even though it has no base resource types, meaning that it wouldn’t be able to have any steps or even checks running.

The reason for that is that the root filesystem that ends up being fetched by Concourse to run a container needs to come from somewhere - the resource type that is configured to retrieve those bits. As there are none to do so, nothing fetch the base, thus, nothing can run.

To break out of that, we have to create a resource type to ship with our cross platform build that is able to retrieve root filesystems that are built for the specific platform that we target.

As we’re targetting non Linux AMD64, that meant that we’d need to go through the cross compilation dance again, now for the registry-image resource.

The problem though, is that not only compiling would be sufficient - when a registry client asks for a container image that lives in a registry, it has to specify what’s the platform that such image is created for.

By default, the underlying library that registry-image-resource uses assuming the linux amd64 tuple, thus, I created a pull request (PR) to address that: https://github.com/concourse/registry-image-resource/pull/36.

Generating the modified registry-image-resource

As a base resource type is defined by having a rootfs and resource_metadata.json (see https://concourse-ci.org/tasks.html#task-image-resource), the easiest way of getting to a rootfs would be to have a container image generated from a Dockerfile and then extracting the final rootfs, and placing that into a tarball.

Having that rootfs.tgz that contains the root filesystem for the modified registry-image-resource, that would mean that I could then distribute this resource in the tarball that contains all of the necessary bits for Concourse, effectively bootstrapping the whole thing!

While that sounds great, we have to remember that the registry-image-resource makes requests to external systems.

The problem here is that to have the rootfs properly created, we need to execute some commands within that container that creates the rootfs to get ca-certificates so that we can make requests to HTTPS endpoints.

# the final representation of the registry-image 
# Concourse resource type.
#
FROM rootfs-${arch} AS registry-image-resource

COPY --from=registry-image-resource-build \
        /assets/ \
        /opt/resource/

RUN apt update -y && \
        apt install -y ca-certificates && \
        rm -rf /var/lib/apt/lists/*

As the base image of that container must be an ARM-based image, it means that any binaries that we try executing there, will be executing instructions that only ARM machines can run. Damn!

To overcome that, we have at least two options:

create a bootstrapping container image once in the target architecture, or
emulate the target architecture.

While 1 sounds like something that could work, 2 is now quite simple to achieve, if you’re using a MacOS or Windows 10 machine.

Since not too long ago, Docker for Desktop has been shipping their internal VM with the right hooks to be able to emulate other architectures in a very transparent way for the developer (see the recent announcement: Building Multi-Arch Image for Arm and X86 with Docker Desktop).

Automating the process of building an ARM-based Concourse distribution

As manually building all of those binaries is not fun at all, I made the whole process buildable by creating a multi-stage Dockerfile (see cirocosta/concourse-arm#Dockerfile).

Given that to build such Dockerfile we’d need to have a task that is able to do so, the Dockerfile builds a version of the builder-task, which wraps genuinetools/img, which is able to build container images from Dockerfiles (using buildkit).

Yeah, that’s a lot of names in a single article!

Summarizing

In the end, we build a bunch of stuff!

For instance, consider the building of the binaries:

And, for the registry-image-resource rootfs:

The good news is that it’s all declared in that very same Dockerfile I mentioned, making the build mostly reproducible.

Some surprises

While trying to figure out what was going wrong with Guardian, I wanted so much to use dlv to troubleshoot what was going on, but, unfortunatelly, it doesn’t support any 32bit systems at the moment.
I didn’t know that in some Linux distros you’d have to run modprobe config to have the /proc/config.gz file accessible to check the configuration used to build that Kernel - interesting to know!

Closing Thoughts

In the end, it turns out that it’s not super complicated to achieve building a Go project that depends on some dependencies (having some C code involved too) to other architectures - set the right variables here and there, make use of some emulation if needed, and there you go.

It’s quite cool what you can do using cross compilation, and how a combination of Go and container images with a well defined set of build steps can make the whole process of building for multi platforms work great even when you don’t have access to those platforms.

I’m very curious about the whole movement of supporting other architectures other than amd64, so it’s nice to start having a foot in this space.

Even though we made use of cross compilation, there were steps where we were still needing to run some details in such architecture, which makes me think that it might be worth investing in having Concourse workers running smoothly on these other platforms too.

Please let me know what you think! I’m @cirowrc on Twitter.

See you!

How Linux creates sockets and counts them

Ciro S. Costa — Tue, 16 Oct 2018 00:00:00 +0000

Hey,

If you’ve been working with web servers for a little while, you certainly have already hit the classic “address already in use” (EADDRINUSE).

Here in this article, we go through not only how to see whether such condition as conditioned to happen (by looking at the list of open sockets), but also verify in the actual Kernel code paths where that check happens.

In case you’ve been wondering about how the socket(2) syscall works where are these sockets stored, make sure you stick to the end!

This is the sixth article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

What are these sockets about?

Sockets are the constructs that allow processes on different machines to communicate through an underlying network, being also possibly used as a way of communicating with other processes in the same host (through Unix sockets).

The analogy that really stuck with me is the one presented in the book Computer Networking: A top-down approach.

At a very high-level, we can think of the server machine as this “house” with a set of doors.

With each door corresponding to a socket, the client can arrive at the door of the house and “knock” at it.

Right after knocking (sending the SYN packet), the house then automatically responds back with a response (SYN+ACK), which is then acknowledged by the house (yep, smart house with a “smart door”).

Meanwhile, while the process just sits there within the house, the clients get organized by the “smart house”, which creates two lines: one for those that the house is still greeting, and another one for those that it finished greeting.

Whenever new clients land in the second line, the process can then let it come in.

Once this connection gets accepted (the client is told to come in), the server is then able to communicate with it, transmitting and receiving data at wish.

One detail to note is that the client doesn’t really “get in” - the server creates a “private door” in the house (a client socket) and then communicates with the client from there.

If you’d like to follow the step by step of implementing a TCP server in C, make sure you check this article! Implementing a TCP server.

Where to look for the list of sockets in my system?

Having the mental model of how the TCP connection establishment looks like, we can now “get into the house” and explore how the machine is creating these “doors” (the sockets), how many doors our house has and in which state they are (are they closed? are they opened?).

For doing so, let’s consider the example of a server that just creates a socket (the door!) and does nothing with it.

// socket.c - creates a socket and then sleeps.
#include <stdio.h>
#include <sys/socket.h>


/**
 * Creates a TCP IPv4 socket and then just
 * waits.
 */
int
main(int argc, char** argv)
{
	// The `socket(2)` syscall creates an endpoint for communication
	// and returns a file descriptor that refers to that endpoint.
	//
	// It takes three arguments (the last being just to provide greater
	// specificity):
	// -    domain (communication domain)
	//      AF_INET              IPv4 Internet protocols
	//
	// -    type (communication semantics)
	//      SOCK_STREAM          Provides sequenced, reliable,
	//                           two-way, connection-based byte
	//                           streams.
	int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
	if (err == -1) {
		perror("socket");
		return err;
	}


        // Just wait ...
        sleep(3600);

	return 0;
}

Under the hood, such simple syscall ends up triggering a whole bunch of internal methods (more on that in the next session) that at some point allows us to seek for information about active sockets under three different files: /proc/<pid>/net/tcp, /proc/<pid>/fd, and /proc/<pid>/net/sockstat.

While the fd directory presents us a list of files opened by the process, /proc/<pid>/net/tcp file gives us information regarding currently active TCP connections (in their various states) under the process network namespace. sockstat, on the other hand, acts more like a summary.

Starting with the fd directory, we can see that after the socket(2) call we can see the socket file descriptor in the list of file descriptors:

# Run socket.out (gcc -Wall -o socket.out socket.c)
# and leave it running in the background
./socket.out &
[2] 21113
 
# Check out that are the open files that the process has.
ls -lah /proc/21113/fd
dr-x------ 2 ubuntu ubuntu  0 Oct 16 12:27 .
dr-xr-xr-x 9 ubuntu ubuntu  0 Oct 16 12:27 ..
lrwx------ 1 ubuntu ubuntu 64 Oct 16 12:27 0 -> /dev/pts/0
lrwx------ 1 ubuntu ubuntu 64 Oct 16 12:27 1 -> /dev/pts/0
lrwx------ 1 ubuntu ubuntu 64 Oct 16 12:27 2 -> /dev/pts/0
lrwx------ 1 ubuntu ubuntu 64 Oct 16 12:27 3 -> 'socket:[301666]'

Given that from a simple call to socket(2) we don’t have a TCP connection, there’s no relevant information to be gathered from /proc/<pid>/net/tcp.

From the summary (sockstat), we can guess that we’re increasing the number of allocated TCP sockets:

# Check the summary regarding socket.
cat /proc/21424/net/sockstat
sockets: used 296
TCP: inuse 3 orphan 0 tw 4 alloc 106 mem 1
UDP: inuse 1 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

To make sure that we’re really increasing the alloc number, we can modify the source code above and allocate 100 sockets instead:

+ for (int i = 0; i < 100; i++) {
      int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
      if (err == -1) {
          perror("socket");
          return err;
      }
+ }

Now, checking that again, we can see the alloc at a higher number:

cat /proc/21456/net/sockstat

                   bigger than before!
                                |
sockets: used 296          .----------.
TCP: inuse 3 orphan 0 tw 4 | alloc 207| mem 1
UDP: inuse 1 mem 0         *----------*
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

Now, the question is - how does the socket gets created under the hood?

What happens under the hood when the socket syscall gets called?

socket(2) is pretty much a factory that produces the underlying structures for handling operations on such socket.

Making use of iovisor/bcc, we can trace the deepest invocation that happens in the sys_socket call stack and from there understand each step.

|  socket()
|--------------- (kernel boundary)
|  sys_socket    
|       (socket, type, protocol)
|  sock_create   
|       (family, type, protocol, res)
|  __sock_create 
|       (net, family, type, protocol, res, kern)
|  sock_alloc    
|       ()
˘

Starting from sys_socket itself, this syscall wrapper is the first thing to be touched at kernelspace, being responsible for performing various checks and preparing some flags to pass down to subsequent invocations.

Once preliminary validations have been performed, it allocates in its stack a pointer to a struct socket, the struct that will end up holding the non-protocol specific information about the socket:

/**
 * Defined `socket` as a syscall with the
 * following arguments:
 * - int family;        - the communication domain
 * - int type; and      - the communication semantics
 * - int protocol.      - a specific protocol within a
 *                        certain domain and semantics.
 *                       
 */
SYSCALL_DEFINE3(socket, 
        int, family, 
        int, type, 
        int, protocol)
{
        // A pointer that is meant to be pointed to
        // a `struct sock` that contains the whole
        // socket definition after it gets properly
        // allocated by the socket family.
	struct socket *sock;
	int retval, flags;


	// ... Checks some stuff and prepare some flags ...
        // Create the underlying socket structures.
	retval = sock_create(family, type, protocol, &sock);
	if (retval < 0)
		return retval;


        // Allocate the file descriptor for the process so
        // that it can consume the underlying socket from
        // userspace.
	return sock_map_fd(sock, flags & (O_CLOEXEC | O_NONBLOCK));
}


/**
 * High level wrapper of the socket structures.
 */
struct socket {
	socket_state            state;
	short                   type;
	unsigned long           flags;
	struct sock*            sk;
	const struct proto_ops* ops;
	struct file*            file;
        // ...
};

Given that at the moment that we create the socket we can choose between different types and protocol families (like, UDP, UNIX, and TCP), the struct socket contains an interface (struct proto_ops*) that defines the basic constructs that sockets implement (regardless of their families/type), which gets initiated on the next method called: sock_create.

/**
 * Initializes `struct socket`, allocating the
 * necessary memory for it, as well as filling
 * the necessary information associated with
 * the socket.
 * 
 * It:
 * - Performs some argument checking;
 * - Runs a security check hook for `socket_create`
 * - Initializes the actual allocation of the `struct socket`
 *   (letting the `family` do it according to its own rules)
 */
int __sock_create(struct net *net, 
        int family, int type, int protocol, 
        struct socket **res, int kern)
{
	int err;
	struct socket *sock;
	const struct net_proto_family *pf;

        // Checks if the protocol range.
	if (family < 0 || family >= NPROTO)
		return -EAFNOSUPPORT;
	if (type < 0 || type >= SOCK_MAX)
		return -EINVAL;


	// Triggers custom security hooks for socket_create.
	err = security_socket_create(family, type, protocol, kern);
	if (err)
		return err;


	 // Allocates a `struct socket` object and ties it to
         // a file under the `sockfs` filesystem.
        sock = sock_alloc();
	if (!sock) {
		net_warn_ratelimited("socket: no more sockets\n");
		return -ENFILE;	/* Not exactly a match, but its the
				   closest posix thing */
	}

	sock->type = type;

        // Tries to retrieve the protocol family methods
        // for performing the family-specific socket creation.
        pf = rcu_dereference(net_families[family]);
	err = -EAFNOSUPPORT;
	if (!pf)
		goto out_release;


        // Executes the protocol family specific 
        // socket creation method.
        //
        // For instance, if our family is AF_INET (ipv4)
        // and we're creating a TCP socket (SOCK_STREAM),
        // a specific method for handling such type of socket
        // is called.
        //
        // If we were specifying a local socket (UNIX),
        // then another method would be called (given that
        // such method would implement the `proto_ops` interface
        // and have been loaded).
	err = pf->create(net, sock, protocol, kern);
	if (err < 0)
		goto out_module_put;
        // ...
}

Continuing with our deep dive, we can now look closely at how the actual struct socket gets allocated by sock_alloc().

What that method does is allocate two things: a new inode, and a socket object.

These two are bound together via the sockfs filesystem, which is then responsible for not only keeping track of socket information in the system, but also providing the translation layer between regular filesystem calls (like write(2)) and the network stack (regardless of the underlying communication domain).

By tracing sock_alloc_inode, the method responsible for allocating the inode in sockfs, we’re able to see how that gets set up:

trace -K sock_alloc_inode
22384   22384   socket-create.out      sock_alloc_inode
        sock_alloc_inode+0x1 [kernel]
        new_inode_pseudo+0x11 [kernel]
        sock_alloc+0x1c [kernel]
        __sock_create+0x80 [kernel]
        sys_socket+0x55 [kernel]
        do_syscall_64+0x73 [kernel]
        entry_SYSCALL_64_after_hwframe+0x3d [kernel]


/**
 *	sock_alloc	-	allocate a socket
 *
 *	Allocate a new inode and socket object. The two are bound together
 *	and initialized. The socket is then returned. If we are out of inodes
 *	NULL is returned.
 */
struct socket *sock_alloc(void)
{
	struct inode *inode;
	struct socket *sock;

        // Given that the filesystem is in-memory,
        // perform the allocation using the kernel
        // memory.
	inode = new_inode_pseudo(sock_mnt->mnt_sb);
	if (!inode)
		return NULL;


        // Retrieves the `socket` struct from
        // the `inode` that lives in `sockfs`
	sock = SOCKET_I(inode);


        // Sets some filesystem aspects so that
	inode->i_ino = get_next_ino();
	inode->i_mode = S_IFSOCK | S_IRWXUGO;
	inode->i_uid = current_fsuid();
	inode->i_gid = current_fsgid();
	inode->i_op = &sockfs_inode_ops;


        // Update the per-cpu counter (which can then be
        // used by `sockstat` to and other systems
        // to quickly know the socket count).
	this_cpu_add(sockets_in_use, 1);
	return sock;
}


static struct inode *sock_alloc_inode(
        struct super_block *sb)
{
	struct socket_alloc *ei;
	struct socket_wq *wq;

        // Create an entry in the kernel cache 
        // taking the necessary memory for it.
	ei = kmem_cache_alloc(sock_inode_cachep, GFP_KERNEL);
	if (!ei)
		return NULL;

	wq = kmalloc(sizeof(*wq), GFP_KERNEL);
	if (!wq) {
		kmem_cache_free(sock_inode_cachep, ei);
		return NULL;
	}

        
        // Performs the most basic initialization
        // possible
	ei->socket.state = SS_UNCONNECTED;
	ei->socket.flags = 0;
	ei->socket.ops = NULL;
	ei->socket.sk = NULL;
	ei->socket.file = NULL;

        // Returns the underlying vfs inode.
	return &ei->vfs_inode;
}

Sockets and resource limits

Given that a filesystem inode can be referred from the userspace from a file descriptor, after we have the underlying Kernel structs all set up, sys_socket is then responsible for generating a file descriptor for the user (going through the resource limits validations as presented in Process resource limits under the hood.

If you’ve wondered why it is the case that you might receive a “too many open files” error for socket(2), that’s the reason - it goes through the same resource limits checks:

static int
sock_map_fd(struct socket* sock, int flags)
{
	struct file* newfile;

        // Do you recall this one? This is the method
        // the kernel ends up performing a check against
        // resource limits and making sure that we don't
        // get past the limits!
	int          fd = get_unused_fd_flags(flags);
	if (unlikely(fd < 0)) {
		sock_release(sock);
		return fd;
	}

	newfile = sock_alloc_file(sock, flags, NULL);
	if (likely(!IS_ERR(newfile))) {
		fd_install(fd, newfile);
		return fd;
	}

	put_unused_fd(fd);
	return PTR_ERR(newfile);
}

Counting the number of sockets in the system

If you’ve paid attention to the sock_alloc call, there was a part of it that took care of increasing the number of sockets that are “in-use”.

struct socket *sock_alloc(void)
{
	struct inode *inode;
	struct socket *sock;

        // ....

        // Update the per-cpu counter (which can then be
        // used by `sockstat` to and other systems
        // to quickly know the socket count).
	this_cpu_add(sockets_in_use, 1);
	return sock;
}

Being this_cpu_add a macro, we can look at its definition and understand a bit more about it:

/*
 * this_cpu operations (C) 2008-2013 Christoph Lameter <cl@linux.com>
 *
 * Optimized manipulation for memory allocated through the per cpu
 * allocator or for addresses of per cpu variables.
 *
 * These operation guarantee exclusivity of access for other operations
 * on the *same* processor. The assumption is that per cpu data is only
 * accessed by a single processor instance (the current one).
 * 
 * [...]
 */

Now, given that we’re always adding to sockets_in_use, we can at least guess that if we go through the method that is registered for /proc/net/sockstat is going to use that value, which it really does (with also performing an addition over the values registered for each CPU):

/*
 *	Report socket allocation statistics [mea@utu.fi]
 */
static int sockstat_seq_show(struct seq_file *seq, void *v)
{
	struct net *net = seq->private;
	unsigned int frag_mem;
	int orphans, sockets;

        // Retrieve the counters related to TCP sockets.
	orphans = percpu_counter_sum_positive(&tcp_orphan_count);
	sockets = proto_sockets_allocated_sum_positive(&tcp_prot);

        // Show the stats!
        // As we saw in the beginning of the article,
        // `alloc` show all of those that were allocated
        // and might not be in an "in-use" state yet.
	socket_seq_show(seq);
	seq_printf(seq, "TCP: inuse %d orphan %d tw %d alloc %d mem %ld\n",
		   sock_prot_inuse_get(net, &tcp_prot), orphans,
		   atomic_read(&net->ipv4.tcp_death_row.tw_count), sockets,
		   proto_memory_allocated(&tcp_prot));
	// ...
	seq_printf(seq,  "FRAG: inuse %u memory %u\n", !!frag_mem, frag_mem);
	return 0;
}

What about namespaces?

As you might’ve noticed, there’s no logic in the code related to namespaces when it comes to counting how many sockets where allocated.

That’s something that at first really surprised me given that I thought the networking stack was something that was the most namespaced, but it turns out that there are still some points that are not.

interesting - `/proc/<pid>/net/tcp` is namespaced, but `/proc/<pid>/net/sockstat` is not (it still isn't, the patch wasn't accept) pic.twitter.com/BcaVCAOczY
— Ciro S. Costa (@cirowrc) October 16, 2018

If you’d like to see that by yourself, make sure you follow the article Using network namespaces and a virtual switch to isolate servers.

The gist of it is that you can create a bunch of sockets, see sockstat, then create a network namespace, get into it, and then see that although you can’t see the TCP sockets from the whole system over there (namespaces in action!), you can see the total number of allocated sockets in the system (not namespaced).

# Create a bunch of sockets using our
# example in C
./sockets.out


# Check that we have a bunch of sockets
cat /proc/net/sockstat
sockets: used 296
TCP: inuse 5 orphan 0 tw 2 alloc 108 mem 3
UDP: inuse 1 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0


# Create a network namespace
ip netns add namespace1


# Get into it
ip netns exec namespace1 /bin/bash


# Check how `/proc/net/sockstat` shows the same
# number of allocated sockets.
TCP: inuse 0 orphan 0 tw 0 alloc 108 mem 3
UDP: inuse 0 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

Closing thoughts

It’s interesting to see how after exploring the inner workings of the Kernel by just being curious about /proc is leading to answers to why some specific behaviors that I’ve seen in daily operations work in such way.

Given that this is just the first article that is about /proc/net and I’ve already learned a lot, I can’t wait to start digging deeper into the rest of it!

If you’d like to follow along with me, make sure you subscribe to the mailing list.

In case you have any questions or thoughts you’d like to share, please let me know!

I’m cirowrc on Twitter, and I’d love to chat with you!

Have a good one, Ciro

Resources

Using /proc to get a process' current stack trace

Ciro S. Costa — Sun, 14 Oct 2018 00:00:00 +0000

Hey,

The file covered in this article, /proc/<pid>/stack, is the one that motivated me to learn more about /proc and get The Month of Proc.

It’s such a useful thing when you’re unaware of what is the state of a given process. Meanwhile, I’ve noticed that it’s not very well known by people getting started with Linux.

Here in this post, you’ll get to know more about how procfs can gather a process' stack trace, as well as get an idea of its usefulness.

This is the fifth article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

A process that blocks

To kick things off, let’s start with the tailoring of a process that blocks - a TCP server.

In its most simplistic form, we can have a single-threaded TCP server that just receives traffic in a given thread and then processes its results.

Naturally, there are two places where we can see the server blocking: at the accept(2) phase, of the read(2) phase - the first blocks until a client finishes the TCP handshake; the second, until data is available for read.

Here’s how a simplistic implementation in C would look like considering just the first blocking part (accepting):

/**
 * server_listen takes care of setting up the
 * passive socket that receives incoming TCP
 * connections.
 *
 * Returns the file descriptor corresponding to
 * the passive socket, of -1 in the case of
 * error.
 *
 * See https://ops.tips/blog/a-tcp-server-in-c/#creating-a-socket
 * for a reference implementation.
 */
int
server_listen();

/**
 * server_accept_and_close accepts connections that
 * finish their TCP handshake through the provided
 * @listen_fd argument.
 *
 * Once the connection is `accept`ed, it gets closed
 * immediately.
 */
int
server_accept_and_close(int listen_fd)
{
	int                conn_fd;
	int                err;
	socklen_t          client_len;
	struct sockaddr_in client_addr;

	client_len = sizeof(client_addr);

	printf("Accepting connections ...\n");

	// Accept connections from the completed
	// connection queue (pops the latest conn
	// that succesfully finished the 3-way
	// handshake).
	//
	// Given that the queue might be empty, it
	// waits (i.e., blocks) until there's a connection
	// there.
	err = (conn_fd = accept(
	         listen_fd, (struct sockaddr*)&client_addr, &client_len));
	if (err == -1) {
		perror("accept");
		return err;
	}

	printf("Client connected! Going to close now.\n");

	// Mark the connection as closed so that we
	// can get back to accepting new connections.
	err = close(conn_fd);
	if (err == -1) {
		perror("close");
		return err;
	}

	return 0;
}

/**
 * main execution - creates a passive socket
 * bound to a specific port that keeps receiving
 * incoming connections, accepting them and then
 * marking them as closed immediately.
 */
int
main(int argc, char** argv)
{
	int err;
	int listen_fd = server_listen();

	if (listen_fd == -1)
		return 1;

	for (;;) {
		err = server_accept_and_close(listen_fd);
		if (err == -1)
			return 1;
	}

	return 0;
}

If you’re not into how a TCP server can be implemented in C, make sure you check A TCP Server in C for a complete description of it.

Run the server, and then see it blocking!

Viewing the process kernel stack trace

Once the server is blocked, we can jump into /proc and check out what’s going on in the Kernel and figure out that it’s blocked on the accept syscall:

# Gather the in-kernel stack trace of
# the `accept.out` process (the simplistic
# TCP server that we're running).
cat /proc/$(pidof accept.out)/stack
[<0>] inet_csk_accept+0x246/0x380
[<0>] inet_accept+0x45/0x170
[<0>] SYSC_accept4+0xff/0x210
[<0>] SyS_accept+0x10/0x20
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

Although it might look like a weird stack trace at first, the structure is very straightforward to reason about.

Each line represents a function that was called (from looking at the stack calls), having the first part, that [<0>] thing, being the kernel address of the function, while the second, that do_syscall_64+... being the symbol name with the corresponding offset.

If [<0>] looks weird (like, not an actual address at all), it’s because it’s intended to be like that.

When fs/proc/base.c#proc_pid_stack (the method that gets called by the /proc implementation of the virtual filesystem) iterates over the stack frames, we can see that it hardcodes [<0>] as the address to be displayed:

/**
 * Iterates through the kernel stack frames of
 * the current task, displaying the kernel 
 * addresses of each function, as well as
 * their symbol name and offset.
 */
static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,
			  struct pid *pid, struct task_struct *task)
{
	struct stack_trace trace;
	unsigned long *entries;
	int err;
	int i;


        // Allocate some memory so that we can have
        // in our execution the whole stack of the
        // process (up to a given depth).
        //
	// The first argument to the kmalloc
	// is the size of the block to be allocated.
        //
	// The second argument is allocation flag.
        //
	// GFP_KERNEL means that allocation is performed on
	// behalf of a process running in the kernel space.
	//
	// This means that the calling function is executing
	// a system call on behalf of a process.
	//
	// Using GFP_KERNEL means that kmalloc can put the
	// current process to sleep waiting for a page when
	// called in low-memory situations.
        // 
	// See https://elixir.bootlin.com/linux/v4.15/source/include/linux/gfp.h#L219
	entries = kmalloc(MAX_STACK_TRACE_DEPTH * sizeof(*entries), GFP_KERNEL);
	if (!entries)
		return -ENOMEM;


        // With the space properly allocated, we can now
        // prepare the `stack_trace` struct and pass
        // it down to the function that will get that
        // for us.
	trace.nr_entries	= 0;
	trace.max_entries	= MAX_STACK_TRACE_DEPTH;
	trace.entries		= entries;
	trace.skip		= 0;


        // Make sure that we have mutual exclusion in
        // place.
	err = lock_trace(task);
	if (!err) {
                // Capture the stack trace!
		// https://elixir.bootlin.com/linux/v4.15/source/arch/x86/kernel/stacktrace.c#L69
		save_stack_trace_tsk(task, &trace);

                // Iterate over each frame captured.
                //
                // *************************
                // HERE is where the `[<0>]` is hardcoded.
                // *************************
		for (i = 0; i < trace.nr_entries; i++) {
			seq_printf(m, "[<0>] %pB\n", (void *)entries[i]);
		}
		unlock_trace(task);
	}
	kfree(entries);

	return err;
}

Looking at the file where the function is defined, we can git blame that seq_printf and see who’s there to blame for putting that hardcoded [<0>].

Guess what - Torvalds did the change!

Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Mon Nov 27 16:45:56 2017 -0800

    proc: don't report kernel addresses in /proc/<pid>/stack

    This just changes the file to report them as zero, although maybe even
    that could be removed.  I checked, and at least procps doesn't actually
    seem to parse the 'stack' file at all.

    And since the file doesn't necessarily even exist (it requires
    CONFIG_STACKTRACE), possibly other tools don't really use it either.

    That said, in case somebody parses it with tools, just having that zero
    there should keep such tools happy.

    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/fs/proc/base.c b/fs/proc/base.c
index 31934cb9dfc8..28fa85276eec 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -443,8 +443,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,
                save_stack_trace_tsk(task, &trace);

                for (i = 0; i < trace.nr_entries; i++) {
-                       seq_printf(m, "[<%pK>] %pB\n",
-                                  (void *)entries[i], (void *)entries[i]);
+                       seq_printf(m, "[<0>] %pB\n", (void *)entries[i]);
                }
                unlock_trace(task);
        }

Not being a Kernel expert (at all!), I tried to understand what that %pB and %pK are all about - I’ve never used such kind of formatting with printf after all.

Looking at the docs for printk format specifiers, we can see what that very specialized formatting is all about:

The B specifier results in the symbol name with offsets and should be used when printing stack backtraces.

[The K specifier is used …] For printing kernel pointers which should be hidden from unprivileged users.

Meaning that yeah, previously you could retrieve the kernel addresses, but not anymore, for the reasons presented by Linus.

When the stack does not help much

While it’s very clear why knowing the in-kernel stack trace in the example above was useful, it’s not all that much when it comes to servers that make use of async io (like most of the modern web servers do).

Here’s how a code that is very similar to the TCP acceptor we wrote above in C looks like in Go:

package main

import (
	"net"
)

func main () {
        // Create the necessary underlying data
        // structures for listening on port 1337
        // on all interfaces.
	listener, err := net.Listen("tcp", ":1337")
	if err != nil {
		panic(err)
	}

        // Release all the resources when leaving
	defer listener.Close()

	for {
                // Accept a connection from the
                // backlog of connections that
                // finalized the 3-way handshake
		conn, err := listener.Accept()
		if err != nil {
			panic(err)
		}

                // Close them right away
		conn.Close()
	}
}

Although in the code above we spawn no goroutines other than the main one, under the hood, the Go runtime ends up setting a single event pool file that allows us to monitor multiple file descriptors and not block on them individually.

By the way, Julia Evans has a great blog post on async IO - make sure you check it out! It’s called Async IO on Linux: select, poll, and epoll.

We can notice that by looking at which syscall the Kernel is blocked at when our process runs:

# Display the stack of every thread that
# pertains to the `go_accept` command that
# is running.
find /proc/$(pidof go_accept)/task -name "stack" | \
        xargs -I{} /bin/sh -c 'echo {} ; cat {}'

/proc/17019/task/17019/stack
[<0>] ep_poll+0x29c/0x3a0
[<0>] SyS_epoll_pwait+0x19e/0x220
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

/proc/17019/task/17020/stack
[<0>] futex_wait_queue_me+0xc4/0x120
[<0>] futex_wait+0x10a/0x250
[<0>] do_futex+0x325/0x500
[<0>] SyS_futex+0x13b/0x180
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

/proc/17019/task/17021/stack
[<0>] futex_wait_queue_me+0xc4/0x120
...

/proc/17019/task/17022/stack
[<0>] futex_wait_queue_me+0xc4/0x120
...

/proc/17019/task/17023/stack
[<0>] futex_wait_queue_me+0xc4/0x120
...

/proc/17019/task/17024/stack
[<0>] futex_wait_queue_me+0xc4/0x120
...

Notice that differently from the C code, here I’m looking at the stack of each task under the task group identified by the PID of the go_accept command.

Given that Go will run more than one thread when it starts (so that we can schedule goroutines to run across the poll of actual threads), we can take a look at the stack across all of the threads and see their stack (in the end, each thread is a task, so we can take the stack of each of them).

If we use dlv, we can see why it’s the case that we have those 5 threads just waiting with a futex_wait, while there’s another one blocked on ep_poll (the actual block on async IO):

# Attach to the currently running Go
# process with delve so that we can
# check from the userspace perspective
# what is going on with the Go runtime
dlv attach $(pidof go_accept)


# Check the thread pool
(dlv) threads
* Thread 17019 at .../sys_linux_amd64.s:671 runtime.epollwait
  Thread 17020 at .../sys_linux_amd64.s:532 runtime.futex
  Thread 17021 at .../sys_linux_amd64.s:532 runtime.futex
  Thread 17022 at .../sys_linux_amd64.s:532 runtime.futex
  Thread 17023 at .../sys_linux_amd64.s:532 runtime.futex
  Thread 17024 at .../sys_linux_amd64.s:532 runtime.futex


# Check the pool of goroutines
(dlv) goroutines
[4 goroutines]
  Goroutine 1 - ...netpoll.go:173 internal/poll.runtime_pollWait (0x427146)
  Goroutine 2 - ...proc.go:303 runtime.gopark (0x42c74b)
  Goroutine 3 - ...proc.go:303 runtime.gopark (0x42c74b)
  Goroutine 4 - ...proc.go:303 runtime.gopark (0x42c74b)


# Switch to goroutine 1
(dlv) goroutine 


# See the userspace stack that got us
# to being parked at `epoll wait`
(dlv) stack
 0  0x000000000042c74b in runtime.gopark
    at /usr/local/go/src/runtime/proc.go:303
 1  0x0000000000427a99 in runtime.netpollblock
    at /usr/local/go/src/runtime/netpoll.go:366
 2  0x0000000000427146 in internal/poll.runtime_pollWait
    at /usr/local/go/src/runtime/netpoll.go:173
 3  0x000000000048e81a in internal/poll.(*pollDesc).wait
    at /usr/local/go/src/internal/poll/fd_poll_runtime.go:85
 4  0x000000000048e92d in internal/poll.(*pollDesc).waitRead
    at /usr/local/go/src/internal/poll/fd_poll_runtime.go:90
 5  0x000000000048fc20 in internal/poll.(*FD).Accept
    at /usr/local/go/src/internal/poll/fd_unix.go:384
 6  0x00000000004b6572 in net.(*netFD).accept
    at /usr/local/go/src/net/fd_unix.go:238
 7  0x00000000004c972e in net.(*TCPListener).accept
    at /usr/local/go/src/net/tcpsock_posix.go:139
 8  0x00000000004c86c7 in net.(*TCPListener).Accept
    at /usr/local/go/src/net/tcpsock.go:260
 9  0x00000000004d55f4 in main.main
    at /tmp/tcp/main.go:16
10  0x000000000042c367 in runtime.main
    at /usr/local/go/src/runtime/proc.go:201
11  0x0000000000456391 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

Having now both the userspace and kernelspace stacks, we can properly identify what’s going on with the Go application.

Closing thoughts

The conclusion? IMO, /proc/<pid>/stack (or the equivalent /proc/<pid>/task/<task_id>/stack) is great, but it only takes us so far.

In the end, we need a mix of userspace and kernel space tools that can help us debug the state in which a system is present.

Luckily, from time to time new tools like dlv show up, pprof improves, and even more powerful tools to inspect the Kernel emerge.

I hope this article was useful for you! Please let me know if you have any questions.

I’m @cirowrc on Twitter, and I’d love to chat!

Have a good one!

Process resource limits under the hood

Ciro S. Costa — Sat, 13 Oct 2018 00:00:00 +0000

Hey,

When dealing with web servers, it’s not uncommon to face the problem of “too many files open”.

Usually, people solve that by modifying a number using ulimit -n, and then, sometimes that doesn’t work as such setting is per-process, and not system-wide.

In this article, I go through how that ulimit setting works under the hood, and how you can make use of /proc to inspect other processes limits without the need for additional tools.

This is the fourth article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

What are resource limits

Resource limits can be thought as ceilings that can be set for specific resources from the Kernel that a given process might end up consuming.

These limits are made up of two values:

soft: limits the number of resources that the process may consume; and
hard: a ceiling on the value to which the soft limit may be adjusted by an unprivileged process.

Throughout this article, I’ll be covering only one resource - the number of open files -, but there are many other resources that can have their consumption limited via resource limits.

So, to set a quick example, let’s consider the following program that opens n files and leaves them open:

package main

import (
	"flag"
	"fmt"
	"io/ioutil"
	"os"
)

/**
 * Take as user input the number of files
 * that we want to create and keep open.
 */
var files = flag.Uint("files", 10, "number of files to open")

/**
 * open-files - opens files and leaves them open.
 *
 * Usage: ./open-files -files=<number_of_files>
 *
 */
func main() {
	flag.Parse()

	if *files == 0 {
		fmt.Fprintf(os.Stderr, "number of files must be > 0")
		os.Exit(1)
	}

	/**
	 * Show the PID of the current process so
	 * that we can mess with it in another
	 * terminal.
	 */
	fmt.Println("pid: ", os.Getpid())

	/**
	 * Starts the process of creating N files
	 * and leaving them open.
	 */
	for i := 0; i < int(*files); i++ {
		fmt.Println(i, "files open")

		tmpfile, err := ioutil.TempFile("", "example")
		if err != nil {
			fmt.Fprintf(os.Stderr, "failed to create and " +
                                "open tmp file: %v\n", err)
			return
		}

		defer os.Remove(tmpfile.Name())
	}
}

Running them on a shell with a high limit of open files, we can see it properly working:

./open-files
pid:  726
0 files open
1 files open
2 files open
3 files open
4 files open
5 files open
6 files open
7 files open
8 files open
9 files open

Now, if we make that limit very small - 10 -, we can see it failing:

# Set the maximum number of open files to 10.
ulimit -n 10

# Try to open 10 files.
#
# Naturally, it should fail, given that 
# the execution of `open-files` will also open
# some shared libraries before opening the other
# 10 files.
./open-files
pid:  732
0 files open
1 files open
2 files open
3 files open
4 files open
5 files open
6 files open
failed to create and open tmp file: 
  open /tmp/example484763765: too many open files

How does the kernel limit the number of files open by a process?

In the case of open files, we can see in the Kernel code how that’s strictly enforced by following the methods invoked in the code path for an open(2) syscall:

          ^   __alloc_fd
          |   get_unused_fd_flags
(kernel)  |   do_sys_open
          |   sys_openat
          |   do_syscall_64
          |   entry_SYSCALL_64_after_hwframe
----------+--------------------------
(user)    |   open(2)

Having everything started at do_syscall and going towards do_sys_open, nothing particularly interesting happens when it comes to resource limits.

Things get interesting when it comes to get_unused_fd_flags, which then calls __alloc_fd with the resource limit boundaries:

/**
 * Sets up the ranges for __alloc_fd
 * so that when trying to allocate a
 * file descriptor, it performs some
 * bounds checking.
 */
int get_unused_fd_flags(unsigned flags)
{
	return __alloc_fd(
                current->files, 0, 
                rlimit(RLIMIT_NOFILE),
                flags);               
}

/*
 * Tries to allocate a file descriptor
 * for the process, marking it as busy.
 */
int __alloc_fd(struct files_struct *files,
	       unsigned start, unsigned end, unsigned flags)
{
	unsigned int fd;
	int error;
	struct fdtable *fdt;

        // ...

        // tries to gather the next available
        // file descriptor
	fd = start;
	if (fd < files->next_fd)
		fd = files->next_fd;
	if (fd < fdt->max_fds)
		fd = find_next_fd(fdt, fd);

        // Verifies if the file descriptor that
        // we have available falls within the
        // boundaries set by the open files
        // resource limit.
	error = -EMFILE;
	if (fd >= end)
		goto out;

	// ...
}

Knowing where all these checks take place, we can tailor a specific bpftrace program that targets __alloc_file and then lets us know what is the result from it.

BEGIN
{
        printf("Looking for EMFILE when opening\n");
}


// Place a return probe in the `__alloc_fd`
// so that we can verify whether an error is
// returned from it.
kretprobe:__alloc_fd /comm == "open-files"/
{
        printf("returned: %d\n", retval);
}

Leaving the tracing running in a terminal while we run open-files in another with a low limit:

# Run the trace program and then
# see the values returned.
# 
# What we expect: file descriptors being
# properly created up until `9`, then `-EMFILE`
# (24) at the 10th try.
#
# #define EMFILE 24 /* Too many open files */

sudo bpftrace ./limit-check.d
Attaching 2 probes...
Looking for EMFILE when opening
returned: 3
returned: 4
returned: 5
returned: 6
returned: 7
returned: 8
returned: 9
returned: -24

Under the hood of the ulimit “command”

Now that we know how the Kernel is able to check for the limits whenever our process tries to create a program, it’s time to discover what that ulimit -n command is all about.

The first thing one might notice is that ulimit is not really a binary that the shell is calling - it’s usually a builtin command that your shell provides.

For instance, looking at ash, the shell that busybox provides, we can see it in ash’s code:

/**
 * Implementation of all of the
 * ulimit builtin functionality.
 *
 * No `execve` calling a `/usr/bin/ulimit`
 * or somethign like that.
 *
 * Pure code implemented from ground up.
 */
int FAST_FUNC
shell_builtin_ulimit(char **argv)
{
        struct rlimit limit;

         // ...
        for (l = limits_tbl; 
                l != &limits_tbl[ARRAY_SIZE(limits_tbl)]; 
                l++) {

                // ...
                if (!opts)
                        opts = OPT_hard + OPT_soft;
                if (opts & OPT_hard)
                        limit.rlim_max = val;
                if (opts & OPT_soft)
                        limit.rlim_cur = val;
                if (setrlimit(l->cmd, &limit) < 0) {
                        bb_perror_msg("error setting limit");
                        return EXIT_FAILURE;
                }
        }
}

ps.: the same is valid for bash - see bash/builtins/ulimit.def.

Being a builtin or not, it doesn’t matter - at some point, it touches the Kernel with a syscall.

For doing so, three syscalls can be used (see man 2 prlimit):

setrlimit and getrlimit: respectively, sets and gets resource limits corresponding to the currently running process (and its future children); and
prlimit, which allows settings and gettings resource limits corresponding to an arbitrary process.

As we can see from the description, in the end, prlimit is what really matters.

If we look under the hood of setrlimit and getrlimit, we even discover that they’re just wrapping parts of the prlimit functionality (see kernel/sys.c):

SYSCALL_DEFINE2(getrlimit, 
        unsigned int, resource, 
        struct rlimit __user *, rlim)
{
	struct rlimit value;
	int ret;

	ret = do_prlimit(current, resource, NULL, &value);
	if (!ret)
		ret = copy_to_user(rlim, &value, sizeof(*rlim)) ? -EFAULT : 0;

	return ret;
}


SYSCALL_DEFINE2(setrlimit, 
        unsigned int, resource, 
        struct rlimit __user *, rlim)
{
	struct rlimit new_rlim;

	if (copy_from_user(&new_rlim, rlim, sizeof(*rlim)))
		return -EFAULT;
	return do_prlimit(current, resource, &new_rlim, NULL);
}

So, how does prlimit work?

A sample ulimit implementation

We can see its functionality in place by creating a quick program in C that is able to change the resource limits regarding the number of open files associated with a given process id:

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <unistd.h>

// A hardcoded number that sets the maximum number
// of open files that the supplied PID can handle.
#define MAX_OPEN_FILES 10

// Hardcode the resource that we'll be changing.
//
// RLIMIT_NOFILE - number of files that can be
//                 opened by this process.
#define RESOURCE RLIMIT_NOFILE

/**
 * limit-open-files - limits the number of open files that
 *                    a given PID can hold to 10.
 *
 * Usage: ./limit-open-files.out <PID>
 *
 */
int
main(int argc, char** argv)
{
	int        err = 0;

        // parse the arguments supplied via
        // flags (code omitted for brevity).
	struct cli cli = { 0 };
	cli_parse(argc, argv, &cli);

        // Initialize the data structures that
        // carry information about the process
        // resource limits.
        //
        // `old` is supposed to have its values
        // filled by the kernel at the `get` 
        // phase of `prlimit`.
	struct rlimit old = { 0 };

        // `new` is supposed to contain the values
        // that we want to change.
	struct rlimit new = {
		.rlim_cur = cli.soft,
		.rlim_max = cli.hard,
	};

	// Perform a "get-and-set" operation.
	//
	// By passing non-NULL `new` and `old` values,
	// the values supplied at `new` will be used as
	// the new values for such resource.
	//
	// The `old` will get filled with the previous
	// values.

        // int prlimit(
        //
        //   pid_t pid,      --> process ID to affect
        // 
        //   int resource,   --> resource to tweak / gather info
        // 
        //   const struct rlimit *new_limit, --> if set; new limit 
        //                                       to place.
        // 
        //   struct rlimit *old_limit   --> if non-nil, gathers info 
        //                                  about the previous soft 
        //                                  and hard limits.
        // );
        // 
	err = prlimit(cli.pid, RLIMIT_NOFILE, &new, &old);
	if (err == -1) {
		perror("prlimit - get and set:");
		return 1;
	}

	printf("before: soft=%lld; hard=%lld\n",
	       (long long)old.rlim_cur,
	       (long long)old.rlim_max);

	// Perform a `get` operation to retrieve the
	// current values set for the resource.
	err = prlimit(cli.pid, RLIMIT_NOFILE, NULL, &old);
	if (err == -1) {
		perror("prlimit - get:");
		return 1;
	}

	printf("now:    soft=%lld; hard=%lld\n",
	       (long long)old.rlim_cur,
	       (long long)old.rlim_max);

	return 0;
}

Run it against a particular process and then see it changing its resources limits:

# Modify the limit of open files that process
# 29871 can hold to soft=12 and hard=12.
./limit-open-files.out -p 29871 -s 12 -h 12
before: soft=1024; hard=1048576
now: soft=12; hard=12


# Trying to modify it again, we can see that 
# the process was previously set to `12` 
# (given that we had just changed to 12).
./limit-open-files.out -p 29871 -s 12 -h 12
before: soft=12; hard=12
now: soft=12; hard=12


# Try to increase the hard limit from 12 to 13.
./limit-open-files.out -p 29871 -s 12 -h 13
prlimit - get and set:: Operation not permitted

Notice how in the end we were not able to complete our desired operation of increasing hard limit.

That’s because some of the operations that prlimit performs (like increasing the hard limit) require elevated privileges, which we didn’t have when running it as an unprivileged user.

Making use of a bpftrace tool called capable we’re able to see how prlimit checks for CAP_SYS_RESOURCE when trying to elevate the hard limit:

sudo capable
UID    PID    COMM             CAP  NAME                 AUDIT
1001   29882  limit-open-file  24   CAP_SYS_RESOURCE     1

Now that we’re aware of the prlimit syscall, can we see the Kernel code that sets and gets these limits?

Setting and getting resource limits at the Kernel level

Tracing down the methods invoked by prlimit, we can see that all ends up in the method do_prlimit at kernel/sys.c:

Here’s a break down of it, full of comments of my own (and mutual exclusion - through locks - stripped out):

/**
 * `do_prlimit` constitutes the underlying functionality
 * of `prlimit` (which is also partly used by `getrlimit`
 * and `setrlimit`).
 */
int do_prlimit(struct task_struct *tsk, unsigned int resource,
		struct rlimit *new_rlim, struct rlimit *old_rlim)
{
	struct rlimit *rlim;
	int retval = 0;

	// Checks if the resource is a valid one (given that
        // we can provide whatever we want from the syscall
        // interface).
	if (resource >= RLIM_NLIMITS)
		return -EINVAL;

        // If we specified a non-NULL `new_rlim`, it means
        // that we're planning to set some limits on a given
        // resource.
	if (new_rlim) {
		// Check if the values even make sense.
		if (new_rlim->rlim_cur > new_rlim->rlim_max)
			return -EINVAL;

		// Check if it'd be getting above the system-wide limit
		// already set (sysctl)
		if (resource == RLIMIT_NOFILE &&
                                new_rlim->rlim_max > sysctl_nr_open)
			return -EPERM;
	}

	// Grab the current resource limits for
	// the desired resource.
	//
	// - Maybe we'll be able to see that /proc/pid/limits
	//   looks at the very same thing (tsk->signal->rlim)?
	rlim = tsk->signal->rlim + resource;

	// If we're going to set new values,
	// that is, if we're going to change limits, then ...
	if (new_rlim) {

		// If we're increasing the hard limit, make
		// sure that the user has the proper capabilities.
		if (new_rlim->rlim_max > rlim->rlim_max &&
				!capable(CAP_SYS_RESOURCE))
			retval = -EPERM;

		// Go ahead and perform the update, but first,
                // apply a security check.
		if (!retval)
			retval = security_task_setrlimit(tsk, resource, new_rlim);

		// ...
	}

	// If everything went right so far,
	// update `old_rlim` with the values of
	// what has been captured as the current
	// limits as of before updating.
	if (!retval) {
		if (old_rlim)
			*old_rlim = *rlim;
		if (new_rlim)
			*rlim = *new_rlim;
	}
}

Cool! So now we know that the limits (per-task) are set at task->signal->rlim + <rlimit_offset>.

You might’ve noticed that we didn’t explicitly look at task->signal->rlim in the section “How does the kernel limits the number of files open by a process”.

To remember you, this is what we saw:

/**
 * Sets up the ranges for __alloc_fd
 * so that when trying to allocate a
 * file descriptor, it performs some
 * bounds checking.
 */
int get_unused_fd_flags(unsigned flags)
{
	return __alloc_fd(
                current->files, 0, 
                rlimit(RLIMIT_NOFILE),
                flags);               
}

Looking deeper into it, we can start inspecting what is that rlimit(RLIMIT_NOFILE) thing, which leads to the fact that under the hood, it looks at the current task and inspects ->signal->rlim in the next function that gets called (task_rlimit):

/**
 * Looks at the current task's resource
 * limit.
 *
 * ps.: notice how it doesn't look at the **hard**
 *      limit, but the **soft** limit.
 */
static inline unsigned long task_rlimit(const struct task_struct *tsk,
		unsigned int limit)
{
	return READ_ONCE(tsk->signal->rlim[limit].rlim_cur);
}

So, now that we know how limits can be updated, as well as read, we can start imagining how /proc/<pid>/limits works under the hood.

Gathering process limits via /proc

Not only we’re able to retrieve information about what are the limits set for a given process via the prlimit(2) syscall, we can also do that by inspecting /proc, more specifically, /proc/<pid>/limits (where <pid> corresponds to the process ID of the process we want to know the limits).

For instance, we can check the limits of the current process either via ulimit -n or /proc/self/limits (/proc/self is a link to /proc/<pid> where <pid> is the current process' pid).

# Check the limit of open files
# using `ulimit` (`prlimit` under the hood).
ulimit -n
1024


# Check the limit of open files
# using the `proc` filesystem under `/proc`.
cat /proc/self/limits  | grep 'open files'
Limit           Soft Limit  Hard Limit   Units
Max open files  1024        1048576      files

Knowing from the past articles (see What is /proc?) how procfs is able to register methods to respond to Virtual Filesystem (VFS) calls, we can start digging into how the method registered for handling calls to /proc/<pid>/limits work.

/**
 * Display limits for a process 
 */
static int proc_pid_limits(struct seq_file *m, struct pid_namespace *ns,
			   struct pid *pid, struct task_struct *task)
{
	unsigned int i;
	unsigned long flags;

	struct rlimit rlim[RLIM_NLIMITS];

        // Make sure we have exclusive access
        // to the task struct while reading
        // it.
	if (!lock_task_sighand(task, &flags))
		return 0;

        // Copy to our current execution all of
        // the limits that are set for the task
        // provided as argument.
	memcpy(rlim, task->signal->rlim, 
                sizeof(struct rlimit) * RLIM_NLIMITS);

        // Release the exclusive access
	unlock_task_sighand(task, &flags);

        // Print the header
        seq_printf(m, "%-25s %-20s %-20s %-10s\n",
		  "Limit", "Soft Limit", "Hard Limit", "Units");

        // Iterate over each limit and then display it.
	for (i = 0; i < RLIM_NLIMITS; i++) {
		if (rlim[i].rlim_cur == RLIM_INFINITY)
			seq_printf(m, "%-25s %-20s ",
				   lnames[i].name, "unlimited");
		else
			seq_printf(m, "%-25s %-20lu ",
				   lnames[i].name, rlim[i].rlim_cur);
                // ...
	}

	return 0;
}

That’s it!

Why expose limits through procfs if prlimit already exists

I don’t know!

While the kernel doesn’t provide convenient ways of accessing some of its internal resources, for limits it’s definitely not the case - prlimit can supply all of its functionality.

Do you know why? Please let me know!

Closing thoughts

It was interesting to see how a resource limit is actually applied by the kernel, and in the end, discover that /proc/<pid>/limits is very redundant when looking at the functionality that a syscall (prlimit) already provides.

If you have any further questions, or just want to connect, let me know! I’m cirowrc on Twitter.

Have a good one!

Resources

Here are some interesting books to learn more about some of the concepts highlighted here:

Why top and free inside containers don't show the correct container memory

Ciro S. Costa — Fri, 12 Oct 2018 00:00:00 +0000

Hey,

Something that is very common to get wrong when starting with Linux containers is to think that free and other tools like top should report the memory limits.

Here you’ll not only go through why that happens and how to get it right, but also take a look at where is the Kernel looking for information when you ask it for memory statistics.

Also, if you’re curious about how the code for keeping track of per-cgroup page counter looks, stick to the end!

This is the third article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

Running top within a container

To get a testbed for the rest of the article, consider the case of running a single container with a memory limit of 10MB in a system that has 2GB of RAM available:

# Check the amount of memory available
# outside the container (i.e., in the host)
free -h
      total   used   free   available
Mem:   1.9G   312M   385M        1.5G

# Define the total number of bytes that
# will dictate the memory limit of the
# container.
MEM_MAX="$((1024 * 1024 * 10))"


# Run a container using the ubuntu image
# as its base image, with the memory limit
# set to 10MB, and a tty as well as interactive
# support.
docker run \
        --interactive \
        --tty \
        --memory $MEM_MAX \
        ubuntu

With the container running, we can now check what are the results from executing top over there:

top -bn1

Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
         .----------------.
         |                |
KiB Mem :| 2040940 total, | 117612 free,   651204 used,  1272124 buff/cache
KiB Swap:|       0 total, |      0 free,        0 used.  1196972 avail Mem
         *--+-------------*
  PID USER  |   PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root  |   20   0   18508   3432   3016 S   0.0  0.2   0:00.02 bash
   12 root  |   20   0   36484   3104   2748 R   0.0  0.2   0:00.00 top
            |
            *---> Not really what we 
                  expect, that is 2GB!!

As we outlined before, not what one would typically expect (it reports the total available memory as seen in the host - not showing the 10MB limit at all).

What about free? Same thing:

free -h
        total    used     free   available
Mem:     1.9G    612M     131M        1.2G
Swap:      0B      0B       0B

How the top and free tools gather memory statistics

If we go inspect what are the syscalls being used by both top and free, we can see that they’re making use of plain open(2) and read(2) calls:

# Check what are the syscalls being
# used by `free`
strace -f free
...
openat(AT_FDCWD, "/proc/meminfo", O_RDONLY) = 3

                        .-------.
                        |       v
read(3, "MemTotal:      | 2040940 kB\nMemF"..., 8191) = 1307
...                     |       
                     That is 2GB!



# Check what are the syscalls being used
# by `top`
strace -f top -p 19282  -bn1
...
openat(AT_FDCWD, "/proc/meminfo", O_RDONLY) = 5
lseek(5, 0, SEEK_SET)                   = 0
read(5, "MemTotal:        2040940 kB\nMemF"..., 8191) = 1307
...                             ^
                                |
             2GB again  --------*

Looking at those return values (what it’s read), we can spot that the “problem” is coming from /proc/meminfo, which free and top are just blindly trusting.

Before we go check what the Kernel is doing when reporting those values, let’s quickly remember how a container gets memory limits set.

Setting container limits

The way that Docker (ok, runc) ends up setting the container limits is via the use of cgroups.

As very well documented in the man page (see man 7 cgroups:

Control cgroups, usually referred to as cgroups, are a Linux kernel feature which allows processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored.

To see that in action, consider the following program that allocates memory in chunks of 1MB:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MEGABYTE (1 << 20)
#define ALLOCATIONS 20


/**
 * alloc - a "leaky" program that just allocated
 *         a predefined amount of memory and then
 *         exits.
 */
int
main(int argc, char** argv)
{
	printf("allocating: %dMB\n", ALLOCATIONS);


	void* p;
	int   i = ALLOCATIONS;


	while (i-- > 0) {
		// Allocate 1MB (not initializing it
		// though).
		p = malloc(MEGABYTE);
		if (p == NULL) {
			perror("malloc");
			return 1;
		}

		// Explicitly initialize the area that
		// has been allocated.
		memset(p, 65, MEGABYTE);

		printf("remaining\t%d\n", i);
	}
}

We can see that without any limits, we can keep allocating past 20MB without problems.

# Keep allocating memory until the 20MB
# mark gets reached.
./alloc.out
allocating: 20MB
remaining	19
remaining	18
...
remaining	1
remaining	0

That changes after we put our process under a cgroup with memory limits set:

# Create our custom cgroup
mkdir /sys/fs/cgroup/memory/custom-group

# Configure the maximum amount of memory
# that all of the processes in such cgroup
# will be able to allocate
echo "$((1024 * 1024 * 10))" > \
        /sys/fs/cgroup/memory/custom-group/memory.limit_in_bytes

# Put the current process tree under such
# cgroup
echo $$ > \
        /sys/fs/cgroup/memory/custom-group/tasks

# Try to allocate the 20MB
./alloc.out
allocating: 20MB
remaining	19
remaining	18
remaining	17
remaining	16
remaining	15
remaining	14
remaining	13
remaining	12
Killed

Looking at the results from dmesg, we can see what happened:

                        our thing getting killed!
                                  .------------.
[181346.109904] alloc.out invoked | oom-killer:|
                                  *------------*
[181346.109906] alloc.out cpuset=/ mems_allowed=0
[181346.109911] CPU: 0 PID: 22074 Comm: alloc.out Not tainted 4.15.0-36-generic #39-Ubuntu
[181346.109911] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[181346.109912] Call Trace:
[181346.109918]  dump_stack+0x63/0x8b
[181346.109920]  dump_header+0x71/0x285
[181346.109923]  oom_kill_process+0x220/0x440
[181346.109924]  out_of_memory+0x2d1/0x4f0
[181346.109926]  mem_cgroup_out_of_memory+0x4b/0x80
[181346.109928]  mem_cgroup_oom_synchronize+0x2e8/0x320
[181346.109930]  ? mem_cgroup_css_online+0x40/0x40
[181346.109931]  pagefault_out_of_memory+0x36/0x7b
[181346.109934]  mm_fault_error+0x90/0x180
[181346.109935]  __do_page_fault+0x4a5/0x4d0
[181346.109937]  do_page_fault+0x2e/0xe0
[181346.109940]  ? page_fault+0x2f/0x50
[181346.109941]  page_fault+0x45/0x50

                        Killed!
...               ____________________________
                 /                            \
[181346.109950] Task in /custom-group killed as 
                a result of limit of /custom-group
[181346.109954] memory: usage 10240kB, limit 10240kB, failcnt 56
[181346.109954] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[181346.109955] kmem: usage 940kB, limit 9007199254740988kB, failcnt 0
[181346.109955] Memory cgroup stats for /custom-group: cache:0KB rss:9300KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:9248KB inactive_file:0KB active_file:0KB unevictable:0KB
[181346.109965] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[181346.110005] [21530]     0 21530     5837     1381    90112        0             0 bash
[181346.110011] [22074]     0 22074     3440     2594    69632        0             0 alloc.out
[181346.110012] Memory cgroup out of memory: Kill process 22074 (alloc.out) score 989 or sacrifice child
[181346.318942] Killed process 22074 (alloc.out) total-vm:13760kB, anon-rss:8988kB, file-rss:1388kB, shmem-rss:0kB
[181346.322003] oom_reaper: reaped process 22074 (alloc.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So we can see pretty well that limits are being enforced.

Again, why is /proc telling us that we have 2GB of memory?

Memory limits set by cgroups are not namespaced

The reason why is that the memory retrieved by /proc/meminfo is not namespaced.

Differently from other things like listing pids from /proc, when the file_operations that procfs implements reach the point of gathering memory information, it doesn’t acquire a namespaced view of it.

For instance, let’s compare the way that listing the differences in showing contents under /proc/ (listing the directory entries) and /proc/meminfo.

In the case of listing /proc (see How is /proc able to list process IDs?), we can see procfs taking the namespace reference and using it:

int proc_pid_readdir(struct file *file, struct dir_context *ctx)
{
        // Takes the namespace as seen by the file
        // provided.
	struct pid_namespace *ns = file_inode(file)->i_sb->s_fs_info;

        // ...
        
        // Iterates through the next available tasks
        // (processes) as seen by the namespace that
        // we are within.
	for (iter = next_tgid(ns, iter);
	     iter.task;
	     iter.tgid += 1, iter = next_tgid(ns, iter)) {
                // ...
        }

        // ...
}

Meanwhile, in the case of reading /proc/meminfo, that doesn’t happen at all (well, as expected, it’s not about namespaces! It’s about cgroups):

static int meminfo_proc_show(struct seq_file *m, void *v)
{
	struct sysinfo i;
	// ...

        // Populate the sysinfo struct with memory-related
        // stuff
	si_meminfo(&i);

        // Add swap information
	si_swapinfo(&i);
        
        // ... start displaying

	show_val_kb(m, "MemTotal:       ", i.totalram);
	show_val_kb(m, "MemFree:        ", i.freeram);

        // ...
}

As expected, no single reference to namespaces (or cgroups).

Also, si_meminfo, the method that fills the sysinfo interface takes some global values and bring it to /proc/meminfo, has no idea about cgroups either:

/**
 * The struct that holds part of the memory information
 * that ends up being displayed in the end.
 */
struct sysinfo {
	__kernel_long_t uptime;		/* Seconds since boot */
	__kernel_ulong_t loads[3];	/* 1, 5, and 15 minute load averages */
	__kernel_ulong_t totalram;	/* Total usable main memory size */
	__kernel_ulong_t freeram;	/* Available memory size */
	__kernel_ulong_t sharedram;	/* Amount of shared memory */
	__kernel_ulong_t bufferram;	/* Memory used by buffers */
	__kernel_ulong_t totalswap;	/* Total swap space size */
	__kernel_ulong_t freeswap;	/* swap space still available */
	__u16 procs;		   	/* Number of current processes */
	__u16 pad;		   	/* Explicit padding for m68k */
	__kernel_ulong_t totalhigh;	/* Total high memory size */
	__kernel_ulong_t freehigh;	/* Available high memory size */
	__u32 mem_unit;			/* Memory unit size in bytes */
	char _f[20-2*sizeof(__kernel_ulong_t)-sizeof(__u32)];	/* Padding: libc5 uses this.. */
};

/**
 * Fills the `sysinfo` struct passed as a pointer
 * with values collected from the system (globally
 * set).
 */
void si_meminfo(struct sysinfo *val)
{
	val->totalram = totalram_pages;
	val->sharedram = global_node_page_state(NR_SHMEM);
	val->freeram = global_zone_page_state(NR_FREE_PAGES);
	val->bufferram = nr_blockdev_pages();
	val->totalhigh = totalhigh_pages;
	val->freehigh = nr_free_highpages();
	val->mem_unit = PAGE_SIZE;
}

Interesting fact: totalram_pages (reported from MemTotal) can change - see this StackOverflow question: Why does MemTotal in /proc/meminfo change?.

Who’s controlling the allocation of memory?

If you’re now wondering where we end up reaching that limit that we set in the cgroup, we need to look at the path that a memory allocation takes.


   alloc.out	(our process)
      |
      |
      *--> task_struct (process descriptor)
            |
            |
   	    *--> mm_struct (memory descriptor)
   	           |
   	           |
   m_cgroup <------*
   |
   +------> page_counter memory
   |          |
   |          *--> { atomic_long_t count, unsigned long limit }
   |
   |
   *------> page_counter swap

Within the Kernel, each process created (in our case, alloc.out) is referenced internally via a process descriptor task_struct:

struct task_struct {
	struct thread_info		thread_info;

        // ... 
 
	unsigned int			cpu;
	struct mm_struct		*mm;

Such process descriptor references a memory descriptor mm defined as mm_struct:

struct mm_struct {
	struct vm_area_struct *mmap;		/* list of VMAs */
	unsigned long mmap_base;		/* base of mmap area */
	unsigned long task_size;		/* size of task vm space */

	// ...

#ifdef CONFIG_MEMCG
	struct mem_cgroup *mem_cgroup;
#endif

}

Such memory descriptor references a mem_cgroup, a data structure that keeps track of the cgroup semantics for memory limiting and accounting:

struct mem_cgroup {
	struct cgroup_subsys_state css;

	/* Private memcg ID. Used to ID objects that outlive the cgroup */
	struct mem_cgroup_id id;

	/* Accounted resources */
	struct page_counter memory;
	struct page_counter swap;
        
        // ...
}

Such cgroup data structure then references some page counters (memory and swap, for instance) defined via the page_counter struct, which are responsible for keeping track of usage and providing the limiting functionality when someone tries to acquire a page:

struct page_counter {
	atomic_long_t count;
	unsigned long limit;

        // The parent CGROUP (remember, cgroups are
        // hierarchical!)
	struct page_counter *parent;
        
        // ...
};

Whenever a process needs some pages assigned to it, page_counter_try_charge goes through the cgroup memory hierarchy, trying to charge a given number of pages, which in case of success (new value would be smaller than the limit), it updates the counts, otherwise, it triggers OOM behavior.

Using bcc to trace page_counter_try_charge, we can see how the act of page_faulting leads to mem_cgroup_try_charge calling page_counter_try_charge:

25641   25641   alloc.out       page_counter_try_charge
        page_counter_try_charge+0x1 [kernel]
        mem_cgroup_try_charge+0x93 [kernel]
        handle_pte_fault+0x3e3 [kernel]
        __handle_mm_fault+0x478 [kernel]
        handle_mm_fault+0xb1 [kernel]
        __do_page_fault+0x250 [kernel]
        do_page_fault+0x2e [kernel]
        page_fault+0x45 [kernel]

Tracing a cgroup running out of memory

If we’re even more curious and decide to trace the page_counter_try_charge arguments, we can see the tries failing in the case when we’re within a container and try to grab more memory than we’re allowed to.

Using bpftrace, we’re able to tailor a small program that inspects the page_counter used in page_counter_try_charge and see how the limit changes over time (until the point that we reach the exhaustion - receiving an OOM then).

#include <linux/page_counter.h>


BEGIN
{
        printf("Tracing page_counter_try_charge... Hit Ctrl-C to end.\n");
        printf("%-8s %-6s %-16s %-10s %-10s %-10s\n",
                "TIME", "PID", "COMM", "REQUESTED", "CURRENT", "LIMIT");

	@epoch = nsecs;
}


kprobe:page_counter_try_charge
{
        $pcounter = (page_counter*)arg0;

        $limit = $pcounter->limit;
        $current = $pcounter->count.counter;
        $requested = arg1;

        printf("%-8d %-6d %-16s %-10ld %-10ld %-10ld\n",
                (nsecs - @epoch) / 1000000,
                pid,
                comm,
                $requested,
                $current,
                $limit
        );
}

Running the tracer with a shell session put into the cgroup that limits our memory, we can see it running out of pages:

sudo bpftrace ./try-charge-counter.d
Attaching 2 probes...
Tracing page_counter_try_charge... Hit Ctrl-C to end.
TIME     PID     REQUESTED  CURRENT    LIMIT
...
3301     25980   32         1288       2560
3302     25980   32         1320       2560
...
3307     25980   1          2553       2560
3307     25980   32         2554       2560
                        .--------------------.
3307     25980   1      |   2554       2560  |
3308     25980   32     |   2555       2560  |
3308     25980   1      |   2555       2560  |
3308     25980   32     |   2556       2560  |
3308     25980   1      |   2556       2560  |
3308     25980   32     |   2557       2560  |
3308     25980   1      |   2557       2560  |
3308     25980   32     |   2558       2560  |
                        *----------.---------*
                                   |
                                still possible
                                to increase the
                                number of pages 
                                   ...

3308     25980   1          2558       2560
3308     25980   32         2559       2560
3308     25980   1          2559       2560
3308     25980   32         2560       2560 * LIMIT REACHED
3308     25980   1          2560       2560 *
3308     25980   1          2560       2560 *
                             |          |
                             *-----.----*
                                   |
   Whoopsy, can't allocate  <------*
   anymore!

Closing thoughts

Although I’ve understood that meminfo wasn’t namespaced, it wasn’t clear for my why.

Going through the exercise of tailoring a quick program to inspect the arguments passed to page_counter_try_charge was very interesting (and easier than I thought!).

Shout out to bpftrace once again for allowing us to go deep into the Kernel with ease!

If you have any further questions, or just want to connect, let me know! I’m cirowrc on Twitter.

Have a good one!

Resources

How is /proc able to list process IDs?

Ciro S. Costa — Thu, 11 Oct 2018 00:00:00 +0000

Hey,

Continuing with the Month of /proc, today’s blog post is about how reading /proc works (yep, the directory) .

Not only is this article’s content about /proc, but also about reading directories in general (expect syscalls and Kernel inspection).

If you’ve been curious about how listing directory entries works under the hood, this is for you!

This is the second article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

Listing directory entries in Linux

Whenever you issue ls in a Linux system, three things happen:

/bin/ls (or the equivalent of it) is executed;
the given directory is opened, and
a syscall is issued to read the directory entries of such directory.

We can discover all these three things by making use of strace, a utility that allows us to trace the syscalls called by a given process:

# Create a directory somewhere
mkdir /tmp/ciro

# Run `strace` with the option of 
# tracing child processes as they are
# created.
strace -f ls /tmp/ciro
execve("/bin/ls", ["ls", "/tmp/ciro/"], 0x7ffd091bfe30 /* 20 vars */) = 0
...
        # Open the file, passing some flags to it.
        openat(AT_FDCWD,     
                "/tmp/ciro/",
                O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

        # Read the directory entries.
        getdents(3,             
                /* 2 entries */, 32768)     = 48

While execve is interesting in it self, we’re not going to focus on it in this article - we can know just that it’s what executes the given program.

Opening a directory

The next syscall is openat.

This one is pretty much the same as open(2), which we covered in the last article (see What is /proc - section: Translating a file read to an internal kernel method), except that it has a special flag set: O_DIRECTORY.

According to the man open:

O_DIRECTORY: If pathname is not a directory, cause the open to fail.

That is pretty much an optimization so that we can avoid an extra stat(2) just for checking if the file is a directory before doing any following directory operations.

If you’re curious about how that works, check out the following trace:

do_last
path_openat
do_sys_open
sys_openat
do_syscall_64

Where do_last, which finishes the file opening, performs the following check:

//                     .--> flag set if 
error = -ENOTDIR; //   |    `O_DIRECTORY` is passed
//                     |    to open(2)
//                     |
if ((nd->flags & LOOKUP_DIRECTORY) && 
        !d_can_lookup(nd->path.dentry)) {

        goto out;       // ends up returning ENOTDIR
}

If you’d like to know more about open(2), make sure you also the last article: what is /proc.

There I cover how open works under the hood when the Virtual Filesystem interacts with procfs.

Another great resource is the book Understanding the Linux Kernel, 3rd Ed. It goes down to what the kernel does when opening a file too!

Reading directory entries from userspace

Now, once the directory has been opened, that is, in the Kernel we have a file description and in userspace we have the file descriptor, we can make use of the getdents syscall.

The system call getdents() reads several linux_dirent structures from the directory referred to by the open file descriptor fd into the buffer pointed to by dirp.

Something interesting about this one is that it’s not wrapped by glibc, meaning that we need to call it via the syscall(2) method ourselves.

I don’t really know why this syscall in specific is not wrapped! Do you? Please let me know! Reach at cirowrc on Twitter!

Not being a glibc-wrapped syscall, we need to call it directly with syscall, providing the arguments that it expects and memory areas for the Kernel to fill (so we can retrieve the response).

Here’s how we can do it (check the comments!):

#include <fcntl.h>
#include <linux/types.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

// Total size that the buffer that we'll allocate
// in the stack can have.
#define BUF_SIZE 1024

// The directory we'll end up reading.
const char* proc_directory_path = "/proc";

/**
 * Provide the structure that fits our current kernel structure
 * for directory entries (see linux#include/linux/dirent.h).
 */
struct linux_dirent64 {
	long           d_ino;    /* 64-bit inode number */
	off_t          d_off;    /* 64-bit offset to next structure */
	unsigned short d_reclen; /* Size of this dirent */
	unsigned char  d_type;   /* File type */
	char           d_name[]; /* Filename (null-terminated) */
};

/**
 * Reads the directory entries from `/proc`.
 *
 * It does so using by using the non-wrapped syscall
 * getdents64 until it returns 0 entries.
 */
int
main(int argc, char** argv)
{
	// `buf` that holds a piece of allocated memory to
	// be given to the Kernel to retrieve data.
	//
	// Given that we're initializing it with `BUF_SIZE`,
	// this is allocating `BUFSIZE * 1` bytes in the stack.
	char buf[BUF_SIZE];

	// file descriptor to hold the open file (directory)
	int fd;

	// number of directory entries that were read and put
	// in the buffer that we allocated in the stack.
	int directory_entries_read;

	// error code to exit.
	int err = 0;


        // Open the directory so that we're able to let the
        // kernel deal with the underlying file description.
	fd = open(proc_directory_path, O_RDONLY | O_DIRECTORY);
	if (fd == -1) {
		perror("open");
		return 1;
	}

	for (;;) {
		// Call the `getdents64` syscall passing the buffer
		// to the kernel so that it can fill with directory
		// entries.
		directory_entries_read =
		  syscall(SYS_getdents64, fd, buf, BUF_SIZE);
		if (directory_entries_read == -1) {
			perror("SYS_getdents64");
			err = 1;
			break;
		}

		if (directory_entries_read == 0) {
			err = 0;
			break;
		}

		struct linux_dirent64* entry;

		// Given that the Kernel filled our array of memory with
		// `N` entries (directory_entries_read), iterate over those
		// structs using the right offset.
		for (int off = 0; off < directory_entries_read;) {
			entry = (struct linux_dirent64*)(buf + off);

			printf("entry: %s\n", entry->d_name);
			off += entry->d_reclen;
		}
	}

	// Given that we're done with reading
	// from the file, close it to free the
	// underlying structures allocated for
	// it (and make further reads fail).
	close(fd);
	return err;
}

IF you’re not very familiar with some of the Linux concepts presented above, a great book that covers that is The Linux Programming Interface.

I’ve based some of the explanations from it!

Having that, we’re able to read a directory with pure C.

Given that the interface is the same for any filesystem, we can swap /proc by /something.txt and it should work the same way - if /something.txt is a directory and we have the right permissions, done!

Under the hood of getdents

With the knowledge of what happens at the userspace (getdents), now it’s time to look at what happens under the hood - once getdents crosses to kernelspace.

Remembering that the filesystems need to implement the corresponding methods from file_operations interface, we can guess that:

there is a method in such interface for listing directory entries, and
such method gets called by sys_getdents at some point.

The first point can be confirmed by looking at the interface itself:

/**
 * file_operations describe an interface that
 * filesystems must implement in order to handle
 * calls from syscalls that interact with filesystems.
 */
struct file_operations {
	loff_t (*llseek) (struct file *, loff_t, int);
	ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
	ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
	// ...

        // The method for iterating over the directory
        // entries under a given directory.
	int (*iterate) (struct file *, struct dir_context *);

        // A method that is similar to `iterate` but allows
        // multiple calls to be performed simultaneously.
	int (*iterate_shared) (struct file *, struct dir_context *);
	// ...
} __randomize_layout;

Now, regarding the second point (checking where in the sys_getdents path one of those two methods gets called), we can look at the syscall implementation at fs/readdir.c.

It first starts by verifying if the task can perform such call at sys_getdents, then it goes to iterate_dir, which is all about offloading the directory entries iteration to the implementor of the file_operations interface.

Looking at the at the actual Linux implementation, we can see how iterate_dir makes use of the underlying filesystem implementation of iterate_shared (f->f_op->iterate_shared):

int iterate_dir(struct file *file, struct dir_context *ctx)
{
	struct inode *inode = file_inode(file);
	bool shared = false;
	int res = -ENOTDIR;

        // making sure that either the iterate_shared or
        // iterate methods of the file_operations interface
        // are implemented
	if (file->f_op->iterate_shared)
		shared = true;
	else if (!file->f_op->iterate)
		goto out;

	// ...

        // is the directory really there?
        // if so, hand the file to the underlying
        // filesystem implementation and let it 
        // iterate over the directory entries.
	res = -ENOENT;
	if (!IS_DEADDIR(inode)) {
		ctx->pos = file->f_pos;

                // In either case, let the filesystem
                // implementation do it.
		if (shared)
			res = file->f_op->iterate_shared(file, ctx);
		else
			res = file->f_op->iterate(file, ctx);

                // update the reading offset
                // ("file pointer")
		file->f_pos = ctx->pos;

                // notify that we accessed the file
		fsnotify_access(file);
		file_accessed(file);
	}
	
        // ...
}

How procfs handles getdents calls

Now, to know where is the implementation of either iterate_shared or iterate from proc, we can go over to the procfs source code (at fs/proc) and search for the method signature (iterate or iterate_shared):

ubuntu@bionic:~/linux/fs/proc$ ag iterate_shared
fd.c
269:	.iterate_shared	= proc_readfd,
353:	.iterate_shared	= proc_readfdinfo,

generic.c
306:	.iterate_shared	= proc_readdir,

root.c
190:	.iterate_shared	= proc_root_readdir,

proc_net.c
184:	.iterate_shared	= proc_tgid_net_readdir,

base.c
2215:	.iterate_shared	= proc_map_files_readdir,
2587:	.iterate_shared	= proc_attr_dir_readdir,
3009:	.iterate_shared	= proc_tgid_base_readdir,
3402:	.iterate_shared	= proc_tid_base_readdir,
3612:	.iterate_shared	= proc_task_readdir,

namespaces.c
143:	.iterate_shared	= proc_ns_dir_readdir,

proc_sysctl.c
847:	.iterate_shared	= proc_sys_readdir,

To filter that list out, we can make use of funccount from iovisor/bcc to check which of those methods get called whenever we issue a call to getdents on /proc:

root@bionic:~# funccount 'proc_*readdir'
FUNC                                    COUNT
proc_readdir                                1
proc_root_readdir                           2
proc_pid_readdir                            2

Having narrowed our scope, now we can learn about those three functions.

From the naming, we can guess that proc_root_readdir is responsible for being the first to respond to a request to list all directory entries from /proc.

Such affirmation can be confirmed by looking at the proc_dir_entry set:

/*
 * This is the root "inode" in the /proc tree..
 */
struct proc_dir_entry proc_root = {
	.low_ino	= PROC_ROOT_INO, 
	.namelen	= 5, 
	.mode		= S_IFDIR | S_IRUGO | S_IXUGO, 
	.nlink		= 2, 
	.count		= ATOMIC_INIT(1),
	.proc_iops	= &proc_root_inode_operations, 

        // Sets the implementation of the `file_operations`
        // interface to use.
	.proc_fops	= &proc_root_operations,
	.parent		= &proc_root,
	.subdir		= RB_ROOT_CACHED,
	.name		= "/proc",
};


/*
 * The root /proc directory is special, as it has the
 * <pid> directories. Thus we don't use the generic
 * directory handling functions for that..
 */
static const struct file_operations proc_root_operations = {
	.read		 = generic_read_dir,

        // Sets the method for iterating over directory entries.
	.iterate_shared	 = proc_root_readdir,
	.llseek		= generic_file_llseek,
};

/**
 * Starts the process of listing the entries from
 * `/proc`.
 */
static int proc_root_readdir(struct file *file, struct dir_context *ctx)
{
        // Check if we're still in the context of
        // listing non-PID entries.
	if (ctx->pos < FIRST_PROCESS_ENTRY) {
		int error = proc_readdir(file, ctx);
		if (unlikely(error <= 0))
			return error;
		ctx->pos = FIRST_PROCESS_ENTRY;
	}

        // Having already read the non-pid directory
        // entries (like `/proc/meminfo`), now go 
        // list the PIDs.
	return proc_pid_readdir(file, ctx);
}

The first part (listing non-pid directories) doesn’t reveal a lot for us - it goes through a list of directories that have registered via their corresponding directory entries structs.

The second though (listing processes directories), is the thing.

Before we get there, we need to review the differences between how the Kernel looks at PIDs and threads compared to the userspace.

Linux and its pids

While in the userspace we’re accustomed to the term pid (process identifier), it’s not the same thing for tgid (thread group id).

Whenever we create a process in userspace, a PID is received.

Now, considering that this process creates a thread, we can see from userspace that this thread inherits such PID.

USERSPACE:

         (pid=123)
        my_root_proc      .--> my_root_proc (pid=123)
           |              |
           *----> fork ---+
                          |
                          *->  my_root_proc (pid=123)

When it comes to the kernel space though, a pid refers to a single execution, so that those things now differ:

KERNEL:

         (pid=123)
        my_root_proc      .--> my_root_proc (pid=123)
           |              |
           *----> fork ---+
                          |
                          *->  my_root_proc (pid=124) 
                                (new pid!)

What unites them is the notion of a tgid (thread group id). This is a property that gets inherited so that we can keep track of who initiated the whole three:

KERNEL:

         (pid=123,tgid=123)
        my_root_proc      .--> my_root_proc (pid=123,tgid=123)
           |              |
           *----> fork ---+
                          |
                          *->  my_root_proc (pid=124,tgid=123)
                                (new pid!)

With that in mind, let’s proceed.

How procfs lists process IDs

The whole implementation of process listing can be found at proc_pid_readdir:

/**
 * For the /proc/ directory itself, 
 * after non-process stuff has been done.
 */
int proc_pid_readdir(struct file *file, struct dir_context *ctx)
{
	// ...

        // Given that calls to `/proc`
        // are namespaced (check out `ls /proc`
        // from within a docker container),
        // we start by grabbing the PID namespace
        // of the current task executing the 
        // getdents call.
	struct pid_namespace *ns = file_inode(file)->i_sb->s_fs_info;

	// ... do some checks ...

        // Iterate over all thread group ids 
        // (`tgid`s), capturing the task struct
        // associated with them.
	for (iter = next_tgid(ns, iter);
	     iter.task;
	     iter.tgid += 1, iter = next_tgid(ns, iter)) {
		char name[PROC_NUMBUF];
		int len;

		cond_resched();
		if (!has_pid_permissions(ns, iter.task, HIDEPID_INVISIBLE))
			continue;

                // convert the tgid to a string that we can
                // return in the dir entries
		len = snprintf(name, sizeof(name), "%d", iter.tgid);
		ctx->pos = iter.tgid + TGID_OFFSET;
		if (!proc_fill_cache(file, ctx, name, len,
				     proc_pid_instantiate, iter.task, NULL)) {
			put_task_struct(iter.task);
			return 0;
		}
	}
	ctx->pos = PID_MAX_LIMIT + TGID_OFFSET;
	return 0;
}

If you’re curious about how the kernel is able to fill that struct task that next_tgid gets, then make sure you stick with the Month of proc!

In the next articles we go further into what information we can grab from such tasks.

Closing thoughts

Having never wrapped a syscall using C before, it was a great exercise to learn how that’s done.

I had never really given attention to how a process can list contents from a given directory - it was pretty clear for me that it involved opening a file and then issuing a certain syscall, but really learning what happens under the hood was amazing.

I’d like also to point out how helpful bcc and bpftrace are for learning about how Linux works internally. Kudos for everyone involved!

If you have any further questions or would like to drop a comment, let me know! I’m cirowrc on Twitter.

Have a good one!

Resources

What is /proc?

Ciro S. Costa — Wed, 10 Oct 2018 10:00:00 +0000

Hey,

Although many people that are accustomed to Linux are aware of the existence of /proc and what some files over there can do, many lack the understanding of what goes behind the scenes to power such filesystem (myself included before writing this article).

If you’ve been wondering about how /proc works under the hood, stay tuned!

This is the first article in a series of 30 articles around procfs: A Month of /proc.

If you’d like to keep up to date with it, make sure you join the mailing list!

What is procfs

Procfs is a special virtual filesystem that can be mounted in your directory tree, allowing processes in userspace to read kernel information conveniently - using regular file I/O operations (like read(2) and write(2)).

           process 123: how many files do proc321
                  |          has open?
                  | 
(userspace)       *---> ls /proc/321/fd
                         \----+------/
                              |    ^
------------------------------|----|------------
                              |    |
                      .---<---*    |
                      |            |
                     kernel        *-----------<------. 
(kernelspace)         |                               |
                      *--> list number of open file   |
                           descriptors for proc `321` |
                           in the root namespace      |
                                 |                    |
                                 *------------>-------'
                                     there you go!

The “virtual” comes from the fact that there’s not really a block device (like a solid-state drive - SSD) that serves the files that we can access under the place where you mount procfs (usually /proc).

Instead, there’s just some code implementing the filesystem interface that gets called whenever you issue reads and writes against those particular locations.

For instance, when a user asks for the limits that apply to a given process, the following path gets followed under the hood:

        cat /proc/13323/limits

                
(userspace)     fd = open("/proc/13323/limits")
                n = read(fd, buf, bufsize)
                     |
---------------------|--------------
                    vfs (common interface for interacting with
                     |   any filesystem)
                     |
                     *-> who's responsible for this `/proc`
                         mount?
                         procfs! let it handle the call.
                          |
                          |
(kernelspace)             *-> hey procfs, take this `read` call
                              for `/proc/13323/limits` please!
                                 |
                   sure! <-------*
                   I'll write the response
                   to the file.
                     |
                     *---> linux/fs/proc/base.c#proc_pid_limits
                           for limit := range limits {
                                fprintf(file, limit)
                           }

Using a tracer like bcc’s trace.py, we can see the kernel stack getting the proc_pid_limit command getting called:

PID     TID     COMM            FUNC
21450   21450   cat             proc_pid_limits
        proc_pid_limits+0x1 
        seq_read+0xe5 
        __vfs_read+0x1b 
        vfs_read+0x8e 
        sys_read+0x55 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

Contrasting Procfs with a regular filesystem

A nice way of viewing the difference between the two is looking at how does the kernel path compare.

Let’s say we have a file /myfile.txt that lives on a disk that makes use of EXT4 as a filesystem.

If we were to read this file (making that that it’s not cached), this is how it’d look like:

        cat /myfile.txt

                
(userspace)     fd = open("/myfile.txt")
                n = read(fd, buf, bufsize)
                     |
---------------------|--------------
                    vfs (common interface for interacting with
                     |   any filesystem)
                     |
                     *-> who's responsible for this `/` mount?
                         ext4! let it handle the call.
                          |
                          |
(kernelspace)             *-> hey ext4, take this `read` call
                              for `/myfile.txt` please!
                                 |
                   sure! <-------*
                   Oh, I know that this file exists in the disk!
                   Let me request the underlying block device driver
                   for it.
                     |
                     *---> hey whoever is in charge of /dev/sda1, 
                        please hand me the contents of my file!
                                |
          Oh, this is not ------*
          in my cache; let me ask the disk for what
          is in the blocks where this file exists.

We can see that in the case of the regular file, the read(2) call ends up getting down to the block device driver that issues the read against a real device.

Using the same tracer that we used before, we can check that, differently from when reading from /proc, at this time, the path is much longer (goes deep down to the actual blk_* methods that handle block devices):

# Drop the caches so that the call ends up in
# a very low-level call to the block device 
# driver.
echo "3" > /proc/sys/vm/drop_caches

# Perform a read
cat ./myfile.txt

# See the trace results:
PID     TID     COMM            FUNC
28653   28653   cat             blk_start_request
        blk_start_request+0x1 
        scsi_request_fn+0xf5 
        __blk_run_queue+0x43 
        queue_unplugged+0x2a 
        blk_flush_plug_list+0x20a 
        blk_finish_plug+0x2c 
        __do_page_cache_readahead+0x1da 
        ondemand_readahead+0x11a 
        page_cache_sync_readahead+0x2e 
        generic_file_read_iter+0x7fb 
        ext4_file_read_iter+0x56 
        new_sync_read+0xe4 
        __vfs_read+0x29 
        vfs_read+0x8e 
        sys_read+0x55 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

Although they look very different after vfs_read, everything feels the same for those consuming the vfs interface.

If you’d like to refresh some concepts around Linux in general (including filesystems), I recommend reading The Linux Programming Interface (chapter 12 covers /proc a bit, and chapter 14 is about filesystems!)

Reading from and writing to procfs

Not only being able to give you some introspection into what is the current state of a given process or the system as a while, /proc is also able to let you modify some of the behaviors of the system.

For instance, in the example above, we dropped the caches by performing a write(2) operation against /proc/sys/vm/drop_caches from the userspace.

To summarize:

when it comes to read(2) operations, it can be seen as an interface to introspect kernel data structures associated with either the whole system or a particular process; and
when it comes to write(2), it can be used to change some kernel parameters at runtime.

Translating a file read to an internal kernel method

Coming back to the interaction between user space and kernel space, you might’ve noticed in the previous sections that there was a common thing sitting between the EXT4 filesystem and procfs: the virtual filesystem (vfs).

By having this layer that sits between any syscall related to a filesystem, vfs is able to present a consistent API while letting different implementations to provide their functionality behind the scenes. No need for user programs to know about what’s the filesystem under the hood.

The way that the Kernel does this translation is pretty nifty.

Here’s an overview of how a read(2) from userspace gets down to a filesystem-specific implementation of a read:

        fd = open("file", flags);
        read(fd, buf, bufsize)
           |
 ----------+-------------------------- (userspace boundary)
           |
  .-------*
  |
  *-> ksys_read(fd, buf, bufsize)
         |                        .--> performs a file lookup,
         |                        |    gathering file information
         |                        |    from a given file descriptor (per-process)
         |                        |    to get a file description (system-wide)
         |                        |
         *-> struct fd f = fdget_pos(fd) 
             vfs_read(f.file, buf, bufsize) -> performs the actual
                       |                     read utilizing the info
                       |                   from the file gathered before.
             .---------*
             |                   
             *-> f.file contains a pointer to a `file_operations` struct,
                which can be thought as an interface that specifies file
                operations like `read`, `write`, etc
                |
                *--> depending on the mount, a specific implementation of
                     such interface is referenced there.
                     |
                     *--> f.file->f_op->read(...)
                                   ^     ^
                                   |     |
                                   |     *-- implementation
                                   |      
                                interface

Whenever a file is opened, the userspace program receives a reference to the open file - the “file descriptor”.

For instance, in the following program, a file descriptor is retrieved after openning the file , being printed to stdout and then closed:

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

// Compile with `gcc -Wall ./main.c
//
// ps.: assumes `/tmp/file.txt` has been created before.
int main (int argc, char** argv) {
	int fd = open("/tmp/file.txt", 0);
	if (fd == -1) {
		perror("open");
		return 1;
	}

        // result: fd=3
        printf("fd=%d\n", fd)

        close(fd);

	return 0;
}

This file descriptor points to an open file description, a system-wide entry that is the thing in the kernel that contains the implementation of the file operations interface depending on the file system that such file resides (see struct file in include/linux/fs.h), as well as keeping track of other details.

/**
 * The file description that gets created when an
 * `open(2)` is called from userspace.
 */
struct file {
	// f_count keep track of the number of references
        // being hold for this file.
        atomic_long_t		f_count;

        // f_pos records the current file offset
        // (a.k.a. file pointer)
        loff_t	f_pos; 


        // f_op contains a pointer to an implementation
        // of the `file_operations` interface - a file
        // operation table.
	const struct file_operations *f_op;

	// ... many more
}


/**
 * Interface for vfs to interfact with.
 * 
 * This is meant to be implemented by the filesystems
 * so that VFS can transparentely interact with them.
 */
struct file_operations {
	struct module *owner;
	loff_t (*llseek) (struct file *, loff_t, int);
	ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
	ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
	ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
	ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
	// ...
} __randomize_layout;

To see that a file description (struct file) gets created when an open (or openat) is issued, we can trace get_empty_filp, the function that gets called to allocate a new object.

32648   32648   a.out           get_empty_filp
        get_empty_filp+0x1      # finds an unused file structure 
                                # and returns a pointer to it
        do_filp_open+0x9b 
        do_sys_open+0x1bb 
        sys_openat+0x14 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

After retrieving a file object, it then starts initializing the struct, filling the fields as they need.

Given that at this moment the kernel has already loaded the inode related to this file into memory, and knowing that the inode holds the pointer to the implementation of the file_operations interface for the underlying filesystem, vfs is then able to set the file operations accordingly, such that whenever a further file operations come, it just follows the pointers: f.file->f_op->read(...)

Now, if that file lived under an xfs filesystem, pretty much the same would happen, except that the inode would be loaded for xfs, which would have xfs file operations, which would then be called when reading through f.file->f_op->read (read would now be an xfs read).

In the case of procfs, f.file->f_op->read is a procfs read.

If you’d like to know more about this area, make sure you get a copy of Understanding the Linux Kernel, 3rd Ed. You can get more insights into VFS from the chapter 12 (The Virtual Filesystem).

Tip: To look around the Linux source code, check the Elixir Cross Referencer. It allows you to search references and find definitions around the code base across different Linux releases. Check out how linux#fs/open.c looks like.

Getting `/proc` in your tree

Although in most modern Linux distributions procfs is probably already mounted under /proc, it’s possible that it is not.

In such case, mounting it requires only the necessary privileges and executing mount with the right type (proc):

# Mount the `proc` device under `/proc`
# with the filesystem type specified as
# `proc`.
#
# note.: you could mount procfs pretty much
#        anywhere you want and specify any
#        device name.
#
#
#         .------------> fs type
#         |    
#         |     .------> device (dummy identifier
#         |    |         in this case - anything)
#         |    |
#         |    |     .-> location
#         |    |     |
#         |    |     |
mount -t proc proc /proc

Once the mount point is there, we can now access it:

# Search for the `meminfo` file in the
# procfs mountpoint
ls /proc | grep meminfo
meminfo

If you’d like to know more about mounting things in a directory tree, make sure you check out The Linux Programming Interface.

This is a great book to have - I’m always consulting it from time to time.

Procfs after VFS

Once we got our /proc mountpoint set up, we can start looking at what happens once we start interacting with it.

After understanding the functionality of vfs (and how it can trigger the specific read of a given filesystem by following the file->f_op->read pointers), it’s a matter of looking at how the file operations implementation of procfs looks like.

Differently from a regular filesystem (like ext4), procfs needs to set different handlers for different files, given that each file ends up in the execution of a different method in the kernel.

Taking the example of reading from /proc/<pid>/limits (from the beginning of the article) and a different file, like /proc/<pid>/wchan, we can see how they differ:

@@ -1,4 +1,4 @@
-        proc_pid_limits+0x1 [kernel]
+        proc_pid_wchan+0x1 [kernel]
        seq_read+0xe5 [kernel]
        __vfs_read+0x1b [kernel]
        vfs_read+0x8e [kernel]
        sys_read+0x55 [kernel]
        do_syscall_64+0x73 [kernel]
        entry_SYSCALL_64_after_hwframe+0x3d [kernel]

Now, what happens inside proc_pid_limits, or what /proc/<pid>/limits is all about … that’s something for another article!

Closing thoughts

It’s very interesting how flexible VFS ends up being.

The way that it presents a consistent interface for applications, letting different filesystem implementions deal with adapting themselves to such interface is pretty interesting.

Coming from a Golang background, I found pretty neat the way that the concept of an interface is applied in this Kernel code (which is all in C, as you might’ve noticed).

In the following articles I’ll go on with exploring some files under /proc, getting deep down into what are those methods doing, so, stay tuned!

If you have any questions or would like to drop some feedback for me, feel free to reach me on Twitter! I’m @cirowrc over there.

Have a good one!

Resources

Aside from regular man pages, two books were referenced in the article (and used during the research):

The Linux Programming Interface: Ch. 12 covering /proc, and Ch. 14 on Filesystems; and
Understanding the Linux Kernel, 3rd Ed. Ch 12 on VFS (The Virtual Filesystem).

A month of /proc

Ciro S. Costa — Wed, 10 Oct 2018 00:00:00 +0000

Hey!

I’ve been wanting to better understand what are some of the capabilities of procfs (/proc), so I thought of a little challenge for me:

for each day, for 30 days, I’ll be posting a new article about a single file under /proc that should be useful for someone either introspecting a system (or wanting to mutate some kernel param).

Given that so many tools touch the /proc mountpoint to gather information, and that it’s such a great place to learn more about where you can look for the reasons why a given process or the entire system is malfunctioning, I bet this it going to be great for many of the readers.

If you’d like to go along with me in such journey, join the mailing list!

Differently from the many other articles around the web covering /proc, the focus of this series is more on the details and internals of each call.

For instance, the plan is not only to cover questions like:

What is /proc/<pid>/fd?

But to include answers to questions like:

When I call ls /proc/<pid>/fd, what happens under the hood?

I still don’t have a full list of which files I’ll be touching (and in what order), so, please, let me know if you’d like me to cover a specific one!

This article will be updated whenever a new post comes out, so feel free to keep track of this one.

Have a good one!

The articles so far

What is /proc? - as the first article in the series, this one focuses on the fundamentals around procfs, the underlying implementation that responds to calls to /proc, as well as vfs.
How is /proc able to list process IDs? - given that directories are files, the first file I wanted to look for was the root directory of /proc. In this article, I go through how listing directories work in Linux and at which point procfs takes care of searching for PIDs and returning them as directory entries for the user.
Why top and free inside containers don’t show the correct container memory? - this one covers some of the underlying aspects of how containers limits resources (tip: cgroups!) and why is the retrieval of memory information uncontained. If you’re into k8s and Docker, check this one out!
Process limits under the hood - go through the internals of implementing a ulimit utility in C, inspecting how the Kernel limits the number of open files for a given process and then finally seeing how /proc/pid/limits works. Expect some tracing and a bunch of Kernel methods exposed!
Using /proc to get a process' current stack trace - go through the process of creating a simple TCP server that blocks, and then using /proc to identify the stack trace of the method that led the kernel to block at a given syscall. In this post we also look at how that C stack compares to one generated by the Go runtime.
How Linux creates sockets and counts them - a deep dive into how the Linux kernel creates a Socket under the hood, and how it keeps track of the number of sockets that it has allocated.

Would you like to keep up with the articles?

Click here to signup for the newsletter!

A bpftrace Ansible role

Ciro S. Costa — Mon, 08 Oct 2018 00:00:00 +0000

Hey,

Today I spotted something very interesting in my Twitter (I’m @cirowrc there, by the way):

To learn bpftrace, there is a reference guide and one-liners tutorial: https://t.co/EztuxTK9iJ https://t.co/foCH2TUCpv
— Brendan Gregg (@brendangregg) October 8, 2018

That’s right, dtrace functionality for Linux via eBPF “superpowers”.

To be very honest, I know almost nothing about dtrace itself, but I’ve heard about it many times - it brings to the operator a framework that allows he/she to observe what’s going on throughout the stack: be it in the user space, or the kernel space.

I’ll be taking a deeper look at it soon, but meanwhile, bpftrace is out there, and you can check it out right now!

In this quick post, you can check a sample Ansible that’s able to install the dependencies of bpftrace as well as build and install it.

The bpftrace ansible role

The role is pretty much a follow up of the INSTALL guide from bpftrace, targetting at an Ubuntu Bionic (18.04) installation.

---
- name: 'add llvm apt key'
  apt_key:
    url: 'https://apt.llvm.org/llvm-snapshot.gpg.key'
    state: 'present'

- name: 'add llvm apt repo'
  apt_repository:
    repo: 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-6.0 main'
    state: 'present'
    update_cache: 'yes'

- name: 'install bpftrace dependencies'
  apt:
    update_cache: 'yes'
    cache_valid_time: 3600
    name: '{{ item }}'
    state: 'present'
  with_items:
    - 'bison'
    - 'clang-6.0'
    - 'clang-6.0-doc'
    - 'clang-tools-6.0'
    - 'cmake'
    - 'flex'
    - 'g++'
    - 'git'
    - 'libclang-6.0-dev'
    - 'libclang-common-6.0-dev'
    - 'libclang1-6.0'
    - 'libelf-dev'
    - 'libfl-dev'
    - 'libllvm6.0'
    - 'llvm-6.0'
    - 'llvm-6.0-dev'
    - 'llvm-6.0-runtime'
    - 'zlib1g-dev'


- name: 'clone bpftrace repository'
  git:
    repo: 'https://github.com/iovisor/bpftrace'
    dest: '/usr/share/bpftrace'
    force: 'yes'

- name: 'create build directory'
  file:
    path: '/usr/share/bpftrace/build'
    state: 'directory'
    mode: 0755

- name: 'prepare makefile'
  command: 'cmake -DCMAKE_BUILD_TYPE=DEBUG ..'
  args:
    chdir: '/usr/share/bpftrace/build'

- name: 'build'
  command: 'make -j3'
  args:
    chdir: '/usr/share/bpftrace/build'

- name: 'install'
  command: 'make -j3'
  args:
    chdir: '/usr/share/bpftrace/build'

Run it against your machine (with privileges) and you should be good to go!

You can find related roles (like one for installing bcc) here: github.com/cirocosta/mylinux.

Please let me know if you need any help! I’m cirowrc on Twitter.

Have a good one!

Article recommendation using Hugo

Ciro S. Costa — Sat, 06 Oct 2018 00:00:00 +0000

Hey,

This month I’ve been interested in increasing the overall time that a user spends navigating here in the blog.

Being article recommendation something that the blog was missing, I went for it.

Check out how you can do it for your static site too!

The idea

Given that my content usually is tagged, I thought that one easy way of adding article recommendation would be to simply take the union between the articles tagged with the same tag as the current one and then randomly list them.

For instance, consider the case of this article: A UDP server and client in Go.

Being the article tagged as Linux, Networking, and Go, we can infer that there are three pools of content that might be similar to what’s been there in this article (having some overlap between them).

Moving forward, we can think that once we addressed the possible articles to recommend, the next step is to give preference to some of them: rank higher those with more overlap in more categories.

Let’s implement that then.

Implementing a Recommended Articles list in Hugo

As a first step, we retrive all the pages that are not the currently page that we are seeing:

// Assign the current scope (the current page) to
// a variable named `$currentArticle` so that within
// other scopes we are able to still reference the 
// current page.
{{ $currentArticle := . }}

// From the list of all pages from the site, only keep
// those whose name is different from the name of the
// current article and whose kind is `page`.
//
// ps.: here we could check something else, like permalink
// or another unique identifier.
// 
// ps.: here you'd probably pick a specific section. To do
// so, perform another `where`.
{{ $articles := where 
        (where $.Site.Pages ".Kind" "eq" "page") 
        ".Title" "!=" $currentArticle.Title }}

note.: the where above must be inlined. Here I broke it in multiple lines just for achieving better readability.

Next, we now create two lists:

one that references all articles with at least two tags that are in the set of tags in the current article; and
a list that references all articles with a single tag in the set of tags of the current article.

We can call these two veryRelevantArticles and relevantArticles (accordingly):

// Instantiate each of them with an empty slice.
{{ $veryRelevantArticles := slice }}
{{ $relevantArticles := slice }}

With the variables set, we can now start iterating over our list of all article pages and checking how many intersections they have with the set of tags from our current page:

// Iterate over each of the articles from the list 
// of article pages
{{ range $idx, $article := $articles }}
        // Compute the number of tag intersactions.
        {{ $numberOfIntersections := len (
                intersect $article.Params.tags $currentArticle.Params.Tags
        ) }}

        // For those pages with a big number of 
        // intersections (>= 2), put in the first
        // slice.
        {{ if (ge $numberOfIntersections 2) }}
                {{ $veryRelevantArticles = 
                        $veryRelevantArticles | append $article }}
        // For the rest (single intersaction), put in the 
        // second slice.
        {{ else if (eq $numberOfIntersections 1) }}
                {{ $relevantArticles = 
                        $relevantArticles | append $article }}
        {{ end }}

        // note.: I'm ignoring those with 0 intersections.
{{ end }}

Once we’ve gotten all interesting articles, now we can create an ordered list starting from those with the biggest number of recommendations to those with the lowest, i.e., we can create a list that corresponds to the concatenation of the two variables we created above:

// Create an empty slice to hold the final list
{{ $recommendedArticles := slice }}

// For each very recommended article, append to the
// list.
{{ range $veryRelevantArticles }} 
        {{ $recommendedArticles = $recommendedArticles | append . }} 
{{ end }}

// For each recommended article, append to the
// list.
// 
// This will lead to something like 
// [very, very, rec, rec, rec....]
{{ range $relevantArticles }} 
        {{ $recommendedArticles = $recommendedArticles | append . }} 
{{ end }}

With the list created, now it’s time to show it.

Displaying the recommendation list

With a list containing those with the biggest number of intersections first and then the ones that have the less number of intersections, we can move forward with displaying that.

For this Blog, I took the approach of showing a shuffle of the very first five that are picked from such list.

<ul>

// For every article in the set of the first 5
// recommended articles shuffled, show their
// anchor.
{{ range (shuffle (first 5 $recommendedArticles)) }}
<li>
  <a href="{{ .Permalink }}">
    {{ .Title }}
  </a>
</li>
{{ end }}

</ul>

This guarantees that if I have some articles with big intersections with the content of the current article, they get displayed (even though unordered).

If you have a lot of content, and a lot of tags, you might want to create more categories (not only veryRelevant and relevant.

In such case, something more elaborate could be done.

Closing thoughts

It’s interesting to see how much we can accomplish without “an actual language”.

Although Hugo gives us some simple primitives, we can build upon that and achieve some pretty satisfactory results.

What about you? Have you ever achieved something similar doing something different? Please let me know!

Also, if you have any questions, feel free to drop me a message at @cirowrc on Twitter.

Have a good one!

A UDP server and client in Go

Ciro S. Costa — Sun, 30 Sep 2018 00:00:00 +0000

Hey,

While it’s prevalent to see implementations of TCP servers in Golang, it’s not very common to see the same when it comes to UDP.

Besides the many differences between UDP and TCP, using Go it feels like these are pretty much alike, except for little details that arise from each protocol specifics.

If you feel like some Golang UDP knowledge would be valuable, make sure you stick to the end.

As an extra, this article also covers the underlying differences between TCP and UDP when it comes to the syscalls that Golang uses under the hood, as well as some analysis of what the Kernel does when those syscalls get called.

Stay tuned!

Overview

As a goal for the blog post, the final implementation should look like an “echo channel”, where whatever a client writes to the server, the server echoes back.

        .---> HELLO!! -->-.
        |                 |
client -*                 *--> server --.
   ^                                    |
   |                                    |
   *---<----- HELLO!! ---------<--------*

Being UDP a protocol that doesn’t guarantee reliable delivery, it might be the case that the server receives the message, and it might be the case that the client receives the echo back from the server.

The flow might complete successfully (or not).

        .---> HELLO!! -->-.
        |                 |
client -*                 *--> server --.
                                        |
                                        |
            whoops, lost! -----<--------*

Not being connection-oriented, the client won’t really “establish a connection”, like in TCP; whenever a message arrives at the server, it won’t “write a response back to the connection”, it will only direct a message to the address that wrote to it.

With that in mind, the flow should look like this:

TIME    DESCRIPTION

t0      client and server exist

          client                         server
          10.0.0.1                       10.0.0.2


t1      client sends a message to the server

          client      ------------>      server
          10.0.0.1         msg           10.0.0.2
                       (from:10.0.0.1)
                       (to:  10.0.0.2)


t2      server receives the message, then it takes
        the address of the sender and then prepares
        another message with the same contents and
        then writes it back to the client

          client      <------------      server
          10.0.0.1         msg2          10.0.0.2
                       (from:10.0.0.1)
                       (to:  10.0.0.2)


t3      client receives the message

          client                         server
          10.0.0.1                       10.0.0.2
        
           thxx!! :D :D


ps.: ports omitted for brevity

That said, let’s see how that story rolls out in Go.

Also,

If you’d like a real deep dive, make sure you consider these books:

The first takes the approach of going from the very high level part of the networking stack (application layer), and then goes down to the very bottom, explaining the details of the protocols in there as it goes through them - if you need an excellent refresher on networking concepts without digging into the implementation details, check this one out!

The other two are more about Linux and Unix in general - very worthwhile if you’re more focused on the implementation.

Have a good reading!

Sending UDP packets using Go

Kicking off with the whole implementation at once (full of comments), we can start depicting it, understanding piece by piece, until the point that we can understand each and every interaction that happens behind the scenes.

// client wraps the whole functionality of a UDP client that sends
// a message and waits for a response coming back from the server
// that it initially targetted.
func client(ctx context.Context, address string, reader io.Reader) (err error) {
	// Resolve the UDP address so that we can make use of DialUDP
	// with an actual IP and port instead of a name (in case a
	// hostname is specified).
	raddr, err := net.ResolveUDPAddr("udp", address)
	if err != nil {
		return
	}

	// Although we're not in a connection-oriented transport,
	// the act of `dialing` is analogous to the act of performing
	// a `connect(2)` syscall for a socket of type SOCK_DGRAM:
	// - it forces the underlying socket to only read and write
	//   to and from a specific remote address.
	conn, err := net.DialUDP("udp", nil, raddr)
	if err != nil {
		return
	}

	// Closes the underlying file descriptor associated with the,
	// socket so that it no longer refers to any file.
	defer conn.Close()

	doneChan := make(chan error, 1)

	go func() {
		// It is possible that this action blocks, although this
		// should only occur in very resource-intensive situations:
		// - when you've filled up the socket buffer and the OS
		//   can't dequeue the queue fast enough.
		n, err := io.Copy(conn, reader)
		if err != nil {
			doneChan <- err
			return
		}

		fmt.Printf("packet-written: bytes=%d\n", n)

		buffer := make([]byte, maxBufferSize)

		// Set a deadline for the ReadOperation so that we don't
		// wait forever for a server that might not respond on
		// a resonable amount of time.
		deadline := time.Now().Add(*timeout)
		err = conn.SetReadDeadline(deadline)
		if err != nil {
			doneChan <- err
			return
		}

		nRead, addr, err := conn.ReadFrom(buffer)
		if err != nil {
			doneChan <- err
			return
		}

		fmt.Printf("packet-received: bytes=%d from=%s\n",
			nRead, addr.String())

		doneChan <- nil
	}()

	select {
	case <-ctx.Done():
		fmt.Println("cancelled")
		err = ctx.Err()
	case err = <-doneChan:
	}

	return
}

Having the client code, we can now depict it, exploring each of its nuances.

Address resolution

Before we even start creating a socket and carrying about sending the information to the server, the first thing that happens is a name resolution that translates a given name (like, google.com) into a set of IP addresses (like, 8.8.8.8).

The way we do that in our code is with the call to net.ResolveUDPAddr, which in a Unix environment, goes all the way down to performing the DNS resolution via the following stack:

(in a given goroutine ...)
    >>  0  0x00000000004e5dc5 in net.(*Resolver).goLookupIPCNAMEOrder
           at /usr/local/go/src/net/dnsclient_unix.go:553
    >>  1  0x00000000004fbe69 in net.(*Resolver).lookupIP
           at /usr/local/go/src/net/lookup_unix.go:101
        2  0x0000000000514948 in net.(*Resolver).lookupIP-fm
           at /usr/local/go/src/net/lookup.go:207
        3  0x000000000050faca in net.glob..func1
           at /usr/local/go/src/net/hook.go:19
    >>  4  0x000000000051156c in net.(*Resolver).LookupIPAddr.func1
           at /usr/local/go/src/net/lookup.go:221
        5  0x00000000004d4f7c in internal/singleflight.(*Group).doCall
           at /usr/local/go/src/internal/singleflight/singleflight.go:95
        6  0x000000000045d9c1 in runtime.goexit
           at /usr/local/go/src/runtime/asm_amd64.s:1333

(in another goroutine...)
0  0x0000000000431a74 in runtime.gopark
   at /usr/local/go/src/runtime/proc.go:303
1  0x00000000004416dd in runtime.selectgo
   at /usr/local/go/src/runtime/select.go:313
2  0x00000000004fa3f6 in net.(*Resolver).LookupIPAddr           <<
   at /usr/local/go/src/net/lookup.go:227
3  0x00000000004f6ae9 in net.(*Resolver).internetAddrList       <<
   at /usr/local/go/src/net/ipsock.go:279
4  0x000000000050807d in net.ResolveUDPAddr                     <<
   at /usr/local/go/src/net/udpsock.go:82
5  0x000000000051e63b in main.main
   at ./resolve.go:14
6  0x0000000000431695 in runtime.main
   at /usr/local/go/src/runtime/proc.go:201
7  0x000000000045d9c1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

If I’m not mistaken, the overall process looks like this:

it checks if we’re giving an already IP address or a hostname; if a hostname, then
looks up the host using the local resolver according to the lookup order specified by the system; then,
eventually performs an actual DNS request asking for records for such domain; then,
if all of that succeeds, a list of IP addresses is retrieved and then sorted out according to an RFC; which gives us the highest priority IP from the list.

Follow the stack trace above and you should be able to see by yourself the source code where the “magic” happens (it’s an interesting thing to do!).

With an IP address chosen, we can proceed.

note.: this process is not different for TCP.

The book Computer Networking: A top-down approach has a great section about DNS.

I’d really recommend you going through it to know more about.

I also wrote a blog post about writing something that is able to resolve A records from scratch using Go: Writing DNS messages from scratch using Go.

TCP Dialing vs UDP Dialing

Instead of using a regular Dial commonly used with TCP, for our UDP client, a different method was used: DialUDP.

The reason for that is that we can enforce the type of address passed, as well as receive a specialized connection: the “concrete type” UDPConn instead of the generic Conn interface.

Although both Dial and DialUDP might sound like the same (even when it comes to the syscalls used while talking to the kernel), they end up being pretty different concerning the network stack implementation.

For instance, we can check that both methods use connect(2) under the hood:

// TCP - performs an actual `connect` under the hood,
// trying to establish an actual connection with the
// other side.
net.Dial("tcp", "1.1.1.1:53")

// strace -f -e trace=network ./main
//  [pid  4891] socket(
        AF_INET, 
 -----> SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 
        IPPROTO_IP) = 3
//  [pid  4891] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
...

// UDP - calls `connect` just like TCP, but given that
// the arguments are different (it's not SOCK_STREAM),
// the semantics differ - it constrains the socket 
// regarding to whom it might communicate with.
net.Dial("udp", "1.1.1.1:53")

// strace -f -e trace=network ./main
// [pid  5517] socket(
        AF_INET, 
 -----> SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 
        IPPROTO_IP) = 3
// [pid  5517] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16) = 0
...

While they’re pretty much the same, from the documentation we can see how they are semantically different depending on the way we configure the socket created via the socket(2) call that happens before connect(2):

If the socket sockfd is of type SOCK_DGRAM, then addr is the address to which datagrams are sent by default, and the only address from which datagrams are received.

If the socket is of type SOCK_STREAM or SOCK_SEQ‐PACKET, this call attempts to make a connection to the socket that is bound to the address specified by addr.

Should we be able to verify that with the TCP transport the Dial method would perform the act of really connecting to the other side? Sure!

./tools/funccount -p $(pidof main) 'tcp_*'
Tracing 316 functions for "tcp_*"... Hit Ctrl-C to end.
^C
FUNC                                    COUNT
tcp_small_queue_check.isra.28               1
tcp_current_mss                             1
tcp_schedule_loss_probe                     1
tcp_mss_to_mtu                              1
tcp_write_queue_purge                       1
tcp_write_xmit                              1
tcp_select_initial_window                   1
tcp_fastopen_defer_connect                  1
tcp_mtup_init                               1
tcp_v4_connect                              1
tcp_v4_init_sock                            1
tcp_rearm_rto.part.61                       1
tcp_close                                   1
tcp_connect                                 1
tcp_send_fin                                1
tcp_rearm_rto                               1
tcp_tso_segs                                1
tcp_event_new_data_sent                     1
tcp_check_oom                               1
tcp_clear_retrans                           1
tcp_init_xmit_timers                        1
tcp_init_sock                               1
tcp_initialize_rcv_mss                      1
tcp_assign_congestion_control               1
tcp_sync_mss                                1
tcp_init_tso_segs                           1
tcp_stream_memory_free                      1
tcp_setsockopt                              1
tcp_chrono_stop                             2
tcp_rbtree_insert                           2
tcp_set_state                               2
tcp_established_options                     2
tcp_transmit_skb                            2
tcp_v4_send_check                           2
tcp_rate_skb_sent                           2
tcp_options_write                           2
tcp_poll                                    2
tcp_release_cb                              4
tcp_v4_md5_lookup                           4
tcp_md5_do_lookup                           4

In the case of UDP though, in theory, it merely takes care of marking the socket for reads and writes to the specified address.

Going through the same process that we did for TCP (going further from looking at the syscall interface), we can trace the underlying kernel methods used by both DialUDP and Dial to see how they differ:

./tools/funccount -p $(pidof main) 'udp_*'
Tracing 57 functions for "udp_*"... Hit Ctrl-C to end.
^C
FUNC                                    COUNT
udp_v4_rehash                               1
udp_poll                                    1
udp_v4_get_port                             1
udp_lib_close                               1
udp_lib_lport_inuse                         1
udp_init_sock                               1
udp_lib_unhash                              1
udp_lib_rehash                              1
udp_lib_get_port                            1
udp_destroy_sock                            1

Much… Much less.

If we go even further, try to explore what happens at each of these calls, we can notice how connect(2) in the case of TCP ends up really transmitting data (to establish perform the handshake, for instance):

PID     TID     COMM            FUNC
6747    6749    main            tcp_transmit_skb
        tcp_transmit_skb+0x1
        tcp_v4_connect+0x3f5
        __inet_stream_connect+0x238
        inet_stream_connect+0x3b
        SYSC_connect+0x9e
        sys_connect+0xe
        do_syscall_64+0x73
        entry_SYSCALL_64_after_hwframe+0x3d

While in the case of UDP, nothing is transmitted, just some set up is performed:

PID     TID     COMM            FUNC
6815    6817    main            ip4_datagram_connect
        ip4_datagram_connect+0x1 [kernel]
        SYSC_connect+0x9e [kernel]
        sys_connect+0xe [kernel]
        do_syscall_64+0x73 [kernel]
        entry_SYSCALL_64_after_hwframe+0x3d [kernel]

If you’re not convinced yet that these two are really different (in the sense that the TCP one sends packets to establish the connection, while UDP doesn’t), we can set up some triggers in the network stack to tell us whenever packets flow:

# By creating a rule that will only match
# packets destined at `1.1.1.1` and that
# match a specific protocol, we're able
# to see what happens at the time that
# `connect(2)` happens with a given protocol
# or another.
iptables \
        --table filter \
        --insert OUTPUT \
        --jump LOG \
        --protocol udp \
        --destination 1.1.1.1 \
        --log-prefix="[UDP] "

iptables \
        --table filter \
        --insert OUTPUT \
        --jump LOG \
        --protocol tcp \
        --destination 1.1.1.1 \
        --log-prefix="[TCP] "

Now, run Dial against a TCP target and DialUDP target and compare the differences.

You should only see [TCP] logs:

[46260.105662] [TCP] IN= OUT=enp0s3 DST=1.1.1.1 SYN URGP=0
[46260.120454] [TCP] IN= OUT=enp0s3 DST=1.1.1.1 ACK URGP=0
[46260.120718] [TCP] IN= OUT=enp0s3 DST=1.1.1.1 ACK FIN URGP=0
[46260.150452] [TCP] IN= OUT=enp0s3 DST=1.1.1.1 ACK URGP=0

If you’re not familiar with the inner workings of dmesg, check out my other blog post - dmesg under the hood.

By the way, The Linux Programming Interface is a great book to know more about sockets and other related topics!

Writing to a UDP “connection”

With our UDP socket properly created and configured for a specific address, we’re now on time to go through the “write” path - when we actually take some data and write to the UDPConn object received from net.DialUDP.

A sample program that just sends a little bit of data to a given UDP server would be as follow:

// Perform the address resolution and also
// specialize the socket to only be able
// to read and write to and from such
// resolved address.
conn, err := net.Dial("udp", *addr)
if err != nil {
        panic(err)
}
defer conn.Close()

// Call the `Write()` method of the implementor
// of the `io.Writer` interface.
n, err = fmt.Fprintf(conn, "something")

Given that conn returned by Dial implements the io.Writer interface, we can make use of something like fmt.Fprintf (that takes an io.Writer as its first argument) as let it call Write() with the message we pass to it.

If interfaces and other Golang concepts are not clear for you yet, make sure you check Kernighan’s book: The Go Programming Language.

yeah, from the guy who wrote The C Programming Language with Dennis Ritchie.

Under the hood, UDPConn implements the Write() method from the io.Writer interface by being a composition of conn, a struct that implements the most basic methods regarding writing to and reading from a given file descriptor:

type conn struct {
	fd *netFD
}

// Write implements the Conn Write method.
func (c *conn) Write(b []byte) (int, error) {
	if !c.ok() {
		return 0, syscall.EINVAL
	}
	n, err := c.fd.Write(b)
	if err != nil {
		err = &OpError{Op: "write", Net: c.fd.net, Source: c.fd.laddr, Addr: c.fd.raddr, Err: err}
	}
	return n, err
}


// UDPConn is the implementation of the Conn 
// and PacketConn interfaces for UDP network 
// connections.
type UDPConn struct {
	conn
}

Now, knowing that in the end fmt.Fprintf(conn, "something") ends up in a write(2) to a file descriptor (the UDP socket), we can investigate even further and see how does the kernel path look for such write(2) call:

PID     TID     COMM            FUNC
14502   14502   write.out       ip_send_skb
        ip_send_skb+0x1 
        udp_sendmsg+0x3b5 
        inet_sendmsg+0x2e 
        sock_sendmsg+0x3e 
        sock_write_iter+0x8c 
        new_sync_write+0xe7 
        __vfs_write+0x29 
        vfs_write+0xb1 
        sys_write+0x55 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

At that point, the packet should be on its way to the other side of the communication channel.

Receiving from a UDP “connection” in a client

The act of receiving from UDPConn can be seen as pretty much the same as the “write path”, except that at this time, a buffer is supplied (so that it can get filled with the contents that arrive), and we don’t really know how long we have to wait for the content to arrive.

For instance, we could have the following code path for reading from a known address:

buf := make([]byte, *bufSize)
_, err = conn.Read(buf)

This would turn into a read(2) syscall under the hood, which would then go through vfs and turn into a read from a socket:

22313   22313   read            __skb_recv_udp
        __skb_recv_udp+0x1 
        inet_recvmsg+0x51 
        sock_recvmsg+0x43 
        sock_read_iter+0x90 
        new_sync_read+0xe4 
        __vfs_read+0x29 
        vfs_read+0x8e 
        sys_read+0x55 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

Something important to remember is that when it comes to reading from the socket, that’s going to be a blocking operation.

Given that a message might never return from such socket, we can get stuck waiting forever.

To avoid such situation, we can set a reading deadline that would kill the whole thing in case we wait for too long:

buf := make([]byte, *bufSize)

// Sets the read deadline for now + 15seconds.
// If you plan to read from the same connection again,
// make sure you expand the deadline before reading
// it.
conn.SetReadDeadline(time.Now().Add(15 * time.Second))
_, err = conn.Read(buf)

Now, In case the other end takes too long to answer:

read udp 10.0.2.15:41745->1.1.1.1:53: i/o timeout

Receiving from a UDP “connection” in a server

While that’s great for the client (we know whom we’re reading from), it’s not for a server.

The reason why is that at the server side, we don’t know who we’re reading from (the address is unknown).

Differently from the case of a TCP server where we have accept(2) which returns to the server implementor the connection that the server can write to, in the case of UDP, there’s no such thing as a “connection to write to”. There’s only a “whom to write to”, that can be retrieved by inspecting the packet that arrived.

WITH READ

  "Hmmm, let me write something to
   my buddy at 1.1.1.1:53"

   client --.
            |
            | client: n, err := udpConn.Write(buf)
            | server: n, err := udpConn.Read(buf)
            |
            *---> server
                  "Oh, somebody wrote me something!
                   I'd like to write back to him/her,
                   but, what's his/her address?
                   
                   I don't have a connection... I need
                   an address to write to! I can't to
                   a thing now!"



WITH READFROM

   client --.
            |
            | client: n, err := udpConn.Write(buf)
            | server: n, address, err := udpConn.Read(buf)
            |
            *---> server
                  "Oh, looking at the packet, I can
                   see that my friend Jane wrote to me,
                   I can see that from `address`!
                   
                   Let me answer her back!"

For that reason, on the server, we need the specialized connection: UDPConn.

Such specialized connection is able of giving us ReadFrom, a method that instead of just reading from a file descriptor and adding the contents to a buffer, it also inspects the headers of the packet and gives us information about who sent the package.

Its usage looks like this:

buffer := make([]byte, 1024)

// Given a buffer that is meant to hold the
// contents from the messages arriving at the
// socket that `udpConn` wraps, it blocks until
// messages arrive. 
//
// For each message arriving, `ReadFrom` unwraps
// the message, getting information about the
// sender from the protocol headers and then
// fills the buffer with the data.
n, addr, err := udpConn.ReadFrom(buffer)
if err != nil {
        panic(err)
}

An interesting way of trying to understand how things work under the hood is looking at the plan9 implementation (net/udpsock_plan9.go).

Here’s how it looks (with comments of my own):

func (c *UDPConn) readFrom(b []byte) (n int, addr *UDPAddr, err error) {
        // creates a buffer a little bit bigger than
        // the one we provided (to account for the header of
        // the UDP headers)
	buf := make([]byte, udpHeaderSize+len(b))

        // reads from the underlying file descriptor (this might
        // block).
	m, err := c.fd.Read(buf)
	if err != nil {
		return 0, nil, err
	}
	if m < udpHeaderSize {
		return 0, nil, errors.New("short read reading UDP header")
	}

        // strips out the parts that were not readen
	buf = buf[:m]

        // interprets the UDP header
	h, buf := unmarshalUDPHeader(buf)
        
        // copies the data back to our supplied buffer
        // so that we only receive the data, not the header.
	n = copy(b, buf)
	return n, &UDPAddr{IP: h.raddr, Port: int(h.rport)}, nil
}

Naturally, under Linux, that’s not the path that readFrom takes. It uses recvfrom which does the whole “UDP header interpretation” under the hood, but the idea is the same (except that with plan9 it’s all done in userspace).

To verify the fact that under Linux we’re using recvfrom, we trace UDPConn.ReadFrom down (you can use delve for that):

0  0x00000000004805b8 in syscall.recvfrom
   at /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1641
1  0x000000000047e84f in syscall.Recvfrom
   at /usr/local/go/src/syscall/syscall_unix.go:262
2  0x0000000000494281 in internal/poll.(*FD).ReadFrom
   at /usr/local/go/src/internal/poll/fd_unix.go:215
3  0x00000000004f5f4e in net.(*netFD).readFrom
   at /usr/local/go/src/net/fd_unix.go:208
4  0x0000000000516ab1 in net.(*UDPConn).readFrom
   at /usr/local/go/src/net/udpsock_posix.go:47
5  0x00000000005150a4 in net.(*UDPConn).ReadFrom
   at /usr/local/go/src/net/udpsock.go:121
6  0x0000000000526bbf in main.server.func1
   at ./main.go:65
7  0x000000000045e1d1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

At the kernel level, we can also check what are the methods involved:

24167   24167   go-sample-udp   __skb_recv_udp
        __skb_recv_udp+0x1 
        inet_recvmsg+0x51 
        sock_recvmsg+0x43 
        SYSC_recvfrom+0xe4 
        sys_recvfrom+0xe 
        do_syscall_64+0x73 
        entry_SYSCALL_64_after_hwframe+0x3d

A UDP Server in Go

Now, going to the server-side implementation, here’s how the code would look like (heavily commented):

// maxBufferSize specifies the size of the buffers that
// are used to temporarily hold data from the UDP packets
// that we receive.
const maxBufferSize = 1024

// server wraps all the UDP echo server functionality.
// ps.: the server is capable of answering to a single
// client at a time.
func server(ctx context.Context, address string) (err error) {
	// ListenPacket provides us a wrapper around ListenUDP so that
	// we don't need to call `net.ResolveUDPAddr` and then subsequentially
	// perform a `ListenUDP` with the UDP address.
	//
	// The returned value (PacketConn) is pretty much the same as the one
	// from ListenUDP (UDPConn) - the only difference is that `Packet*`
	// methods and interfaces are more broad, also covering `ip`.
	pc, err := net.ListenPacket("udp", address)
	if err != nil {
		return
	}

	// `Close`ing the packet "connection" means cleaning the data structures
	// allocated for holding information about the listening socket.
	defer pc.Close()

	doneChan := make(chan error, 1)
	buffer := make([]byte, maxBufferSize)

	// Given that waiting for packets to arrive is blocking by nature and we want
	// to be able of canceling such action if desired, we do that in a separate
	// go routine.
	go func() {
		for {
			// By reading from the connection into the buffer, we block until there's
			// new content in the socket that we're listening for new packets.
			//
			// Whenever new packets arrive, `buffer` gets filled and we can continue
			// the execution.
			//
			// note.: `buffer` is not being reset between runs.
			//	  It's expected that only `n` reads are read from it whenever
			//	  inspecting its contents.
			n, addr, err := pc.ReadFrom(buffer)
			if err != nil {
				doneChan <- err
				return
			}

			fmt.Printf("packet-received: bytes=%d from=%s\n",
				n, addr.String())

			// Setting a deadline for the `write` operation allows us to not block
			// for longer than a specific timeout.
			//
			// In the case of a write operation, that'd mean waiting for the send
			// queue to be freed enough so that we are able to proceed.
			deadline := time.Now().Add(*timeout)
			err = pc.SetWriteDeadline(deadline)
			if err != nil {
				doneChan <- err
				return
			}

			// Write the packet's contents back to the client.
			n, err = pc.WriteTo(buffer[:n], addr)
			if err != nil {
				doneChan <- err
				return
			}

			fmt.Printf("packet-written: bytes=%d to=%s\n", n, addr.String())
		}
	}()

	select {
	case <-ctx.Done():
		fmt.Println("cancelled")
		err = ctx.Err()
	case err = <-doneChan:
	}

	return
}

As you might have noticed, it’s not all that different from the client! The reason why is that not having an actual connection involved (like in TCP), both client and servers end up going through the same path: preparing a socket to read and write from and to, then checking the content from the packets and doing the same thing over and over again.

Closing thoughts

It was great to go through this exploration, checking what’s going on behind the scenes in the Go source code (very well written, by the way), as well as in the Kernel.

I think I finally got a great workflow when it comes to debugging with Delve and verifying the Kernel functions with bcc, maybe I’ll write about that soon - let me know if that’d be interesting!

If you have any questions, please let me know! I’m @cirowrc on Twitter, and I’d love to receive your feedback.

Have a good one!

Retrieving the full path of a process on MacOS (and exploring procfs)

Ciro S. Costa — Tue, 25 Sep 2018 00:00:00 +0000

Hey,

Another day I was trying to make sure that a given process that I was running was using a specific binary that I had built, but I couldn’t figure out: ps would only show me the non-absolute path.

# How could I know what is the absolute path of the
# `hugo` binary, assuming that I could have multiple
# `hugo` binaries in `$PATH`?
ps
  PID TTY           TIME CMD
 4153 ttys000    5:14.98 hugo serve     <<<
 9035 ttys001    0:00.04 /Applications/iTerm.app/Content...
 9037 ttys001    0:00.10 -bash
 9086 ttys001    0:02.27 /usr/local/Cellar/macvim/8.1-15...
 9236 ttys002    0:00.04 /Applications/iTerm.app/Content...
 9238 ttys002    0:00.10 -bash

If I were using Linux though, I thought, that’d be easy: head to /proc, search for the pid of the process and then check what exe links to; done.

# (on a Linux machine ...)
#
# See `hugo` will still not show up with the absolute path
# like on MacOS.
ps aux | grep hugo
ubuntu    2275  0.0  0.0 101852   748 pts/0    Sl+  00:26   0:00 hugo serve

# Given that the proc filesystem can provide us with some
# more information about the process, check out the `exe`
# link (which should provide a link to the actual executable).
stat /proc/2275/exe
  File: /proc/2275/exe -> /usr/local/bin/hugo
  Size: 0         	Blocks: 0          IO Block: 1024   symbolic link
Device: 4h/4d	Inode: 140106      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1001/  ubuntu)   Gid: ( 1001/  ubuntu)
Access: 2018-09-24 00:26:37.167004005 +0000
Modify: 2018-09-24 00:26:27.391004005 +0000
Change: 2018-09-24 00:26:27.391004005 +0000
 Birth: -

In this post, I go through how we can gather such information on a MacOS, and what the procfs in Linux is all about.

tl;dr: /proc on Linux is dope; on MacoS: compile a little code that uses proc_pidpath from libproc, or install pidpath.

The /proc filesystem in Linux

In “Linux land”, there’s this thing called “procfs”.

It’s a virtual filesystem - in the sense that there are no real regular files in your disk that map to the filesystem representation - that allows a user (in userspace) to perform some introspection about its current running process and others as well.

From the kernel docs:

The proc file system acts as an interface to internal data structures in the kernel.

It can be used to obtain information about the system and to change certain kernel parameters at runtime (sysctl).

The way the interaction with it is set up is pretty nifty:

each process receives a given path under /proc (like, /proc/<pid>), and then
as subdirectories of this path, various other files and subdirectories are present to allow deeper introspection about the specific pid.

# Display the files and directories present at the
# very root of `/proc`.
#
# Here we can find the list of PIDs that we can access,
# as well as some more system-wide information and 
# settings that we can tweak.
ls -lah /proc
total 4.0K
dr-xr-xr-x 124 root     root       0 Sep 24 23:56 .
drwxr-xr-x  24 root     root    4.0K Sep 24 23:57 ..
dr-xr-xr-x   9 root     root       0 Sep 24 23:56 1
dr-xr-xr-x   9 root     root       0 Sep 25 00:54 2016
dr-xr-xr-x   9 root     root       0 Sep 24 23:57 417
...
-r--r--r--   1 root     root       0 Sep 25 01:25 sched_debug
-r--r--r--   1 root     root       0 Sep 25 01:25 schedstat
dr-xr-xr-x   4 root     root       0 Sep 25 01:25 scsi
lrwxrwxrwx   1 root     root       0 Sep 24 23:56 self -> 2574
...


# Getting into a specific pid path, we're able to
# gather more information about the specifics of
# a given process.
ls -lah /proc/472
total 0
dr-xr-xr-x   9 root root 0 Sep 24 23:57 .
dr-xr-xr-x 123 root root 0 Sep 24 23:56 ..
...
lrwxrwxrwx   1 root root 0 Sep 25 01:30 cwd -> /
-r--------   1 root root 0 Sep 25 01:30 environ
lrwxrwxrwx   1 root root 0 Sep 24 23:57 exe -> /lib/systemd/systemd-udevd
dr-x------   2 root root 0 Sep 24 23:57 fd
lrwxrwxrwx   1 root root 0 Sep 25 01:30 root -> /
-rw-r--r--   1 root root 0 Sep 25 01:30 sched
...

Not only that, procfs is very helpful when you’re not sure if a process is blocked on something you didn’t expect (like a write(2) to an nfs mount point that is malformed due to a bad set of servers not responding), or something simple as your process sleeping when you didn’t want to:

# Sleep for 33 days on the background
sleep 33d &
[1] 2786


# Check what's the state of the process
cat /proc/2786/stat
2786 (sleep) S ...
 |     |     |
 |     |     `-> state (interruptible sleep)
 |     `-> command being run (sleep command)
 `-> process id (the pid we used before)


# Check what's the stack trace (from the kernel
# perspective) that led the process to this 
# sleep state
cat /proc/2786/stack
[<0>] hrtimer_nanosleep+0xd8/0x1d0
[<0>] SyS_nanosleep+0x72/0xa0
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

procfs under the hood

What’s interesting about being virtual is that the implementation of procfs is able to generate the representation of the filesystem on the fly - whenever you issue an I/O call like read(2), Linux answers back with what you asked for, be it the list of file descriptors opened by a given process, or the list of environment variables that were set at process startup time.

For instance, if tracing the execution of cat /proc/<pid>/meminfo down, we can find the path that the read(2) syscall takes:

# stack trace of `cat /proc/<pid>/meminfo`
        meminfo_proc_show
        proc_reg_read
        __vfs_read
        vfs_read
        sys_read
        do_syscall_64
        entry_SYSCALL_64_after_hwframe


# stack trace of `cat /file.txt` 
# on an ext4 mount point
        ext4_file_read_iter
        __vfs_read
        vfs_read
        sys_read
        do_syscall_64
        entry_SYSCALL_64_after_hwframe

Very different from a regular read (as shown in the second stack trace), there’s no real file on disk being accessed - just meminfo_proc_show returning the contents related to what the user asked for: virtual memory stuff.

By the way, if you’re interested in knowing more about related subjects, a great reference for this type of knowledge is The Linux Programming Interface: A Linux and UNIX System Programming Handbook.

Now to MacOS.

The libproc library in MacOS

Differently from Linux, it feels like we can’t know all that much about how things work on MacOS.

After searching a bit on how to accomplish how to gather information about a process, libproc showed up.

As mentioned in libproc.h:

/*
 * This header file contains private interfaces 
 * to obtain process information.
 *
 * These interfaces are subject to change in future releases.
 */

One thing to note:

the interfaces are private - no guaranteed compatibility with future releases.

This has been elucidated by an Apple staff member on post at the Apple’s developer forum regarding gathering process information:

[…] Apple has not put a lot of effort into providing APIs for getting this sort of information.

What APIs that do exist were either inherited from OS X’s predecessor OSs or were added primarily to meet our internal requirements rather than the needs for third-party developers.

Thus, you will find a lot of places where these APIs are: incomplete; incorrect; poorly documented and aren’t as binary compatible as they should be.

Anyway, we can still make use of it - more specifically, we can make use of proc_pidpath, a method that takes a pid (the pid of the process that we want to know more about), a buffer where the path should be written to, and the buffer size.

int proc_pidpath(
  int pid,              // pid of the process to know more about
  void * buffer,        // buffer to fill with the abs path
  uint32_t  buffersize  // size of the buffer
);

That said, we can go ahead and create our Go binary that can handle both Linux and MacOS by specifying two different compilation targets.

A Golang binary that suits Linux and MacOS

Given that libproc will not be a thing under Linux, we can start by creating a pidpath_linux.go file that is meant to be compiled only on Linux, and another file, pidpath_darwin.go, aimed at MacOS machines.

The Linux one is rather simple: it follows the /proc/<pid>/exe symlink, and that’s it:

// +build linux
package main

import (
	"os"
	"strconv"
)

func GetExePathFromPid(pid int) (path string, err error) {
	path, err = os.Readlink("/proc/" + strconv.Itoa(pid) + "/exe")
	return
}

The MacOS version though, needs a little bit more.

Given that we’d access libproc via C, we can leverage CGO.

// +build darwin
package main

// #include <libproc.h>
// #include <stdlib.h>
// #include <errno.h>
import "C"

import (
	"fmt"
	"unsafe"
)

// bufSize references the constant that the implementation
// of proc_pidpath uses under the hood to make sure that
// no overflows happen.
//
// See https://opensource.apple.com/source/xnu/xnu-2782.40.9/libsyscall/wrappers/libproc/libproc.c
const bufSize = C.PROC_PIDPATHINFO_MAXSIZE

func GetExePathFromPid(pid int) (path string, err error) {
        // Allocate in the C heap a string (char* terminated
        // with `/0`) of size `bufSize` and then make sure
        // that we free that memory that gets allocated
        // in C (see the `defer` below).
	buf := C.CString(string(make([]byte, bufSize)))
	defer C.free(unsafe.Pointer(buf))

        // Call the C function `proc_pidpath` from the included
        // header file (libproc.h).
	ret, err := C.proc_pidpath(C.int(pid), unsafe.Pointer(buf), bufSize)
	if ret <= 0 {
		err = fmt.Errorf("failed to retrieve pid path: %v", err)
		return
	}

        // Convert the C string back to a Go string.
	path = C.GoString(buf)
	return
}

That done, we can now consume GetExePathFromPid in our application.

To see that in place, check out cirocosta/pidpath.

Closing thoughts

It was interesting to me to check out how different things are in MacOS land.

Although I use a Macbook Pro as a personal computer (and a Mac at work), I’ve not really paid attention to these little details.

Also, /proc is just so valuable! Definitely worth knowing more about other functionality over there. Make sure you check out The Linux Programming Interface: A Linux and UNIX System Programming Handbook.

If you have any questions, or suggestions to improve this blog post, please let me know! I’m cirowrc, and I’d love to chat.

Have a good one!

Dmesg under the hood

Ciro S. Costa — Sat, 22 Sep 2018 00:00:00 +0000

Hey,

This week I wanted to discover a bit more about how dmesg works under the hood. In the past, I wanted to have alerting based on error messages popping at dmesg, so, maybe by trying to create something that uses the same thing that dmesg uses under the hood, I could better understand it.

Also, I knew that in some systems, it was possible to gather the same information from kern.log and that dmesg was all about reading “the kernel ring buffer”, but, what did that mean?

More specifically, where was dmesg reading from?

In this blog post, I go through some exploration to better understand the whole picture.

You can jump to the tl;dr here.

Producing logs from kernel space

Whenever something in the Linux kernel (which could be a module) wants to do the equivalent of a printf, a similar utility (printk) can be used.

That allows the program to give information back to the developer (or any other user) in userspace to know more about what’s going on with the module - be a warning message, stack traces or some pure debugging information.

To test out producing logs from the Kernel, I came up with some ideas:

iptables has a LOG target that essentially produces some logs from iptables itself whenever the target is, well, targetted. Being iptables something that runs in kernel space (not the cli, I mean - the actual thing), we could see logs coming out of the kernel;
writing a minimal kernel module should not be something very hard - many people did that before. I could write a printk("hello-world") and that’s it!
check if there’s something simpler that we could use to produce the logs (maybe the Kernel exposes a standard interface for doing that?)

Being this an exploration, why not try all of them?

Producing logs from iptables

Although this blog post is not particularly about iptables, we can use it as an example where interacting with something that is going on under the hood deep in the kernel is interesting for a user that sits on userspace.

From iptables man page:

Iptables is used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Several different tables may be defined.

These mentioned rules can be either very simple things (drop a packet if it comes from a given source), or some very complicated chain of rules that depend on a bunch of conditionals.

Acknowledging that it’s very useful to be able to visualize the packets that are hitting a specific chain of rules, the LOG target came to existence.

LOG: Turn on kernel logging of matching packets.

When this option is set for a rule, the Linux kernel will print some information on all matching packets (like most IP/IPv6 header fields) via the kernel log.

Here’s an example set up that allows us to see that working:

# List the current state of the `filter` table.
# As we can see below, nothing is set up:
# - any packet arriving w/ our machine as the
#   destination is accepted;
# - any packet arriving that should be forwarded
#   is dropped;
# - any packet that is sent from our machine is
#   accepted (can go through).
iptables --table filter --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


# Add a rule to the OUTPUT chain to log every packet
# that is destined towards 8.8.8.8 (google's public
# dns) 
iptables \
        --table filter \
        --insert OUTPUT \
        --jump LOG \
        --destination 8.8.8.8 \
        --log-prefix="[google-dns-out]"

# Check that the rule has been placed in the OUTPUT 
# chain of the `filter` table
iptables --table filter --list OUTPUT --numeric
Chain OUTPUT (policy ACCEPT)
target   prot opt source       destination
LOG      all  --  0.0.0.0/0    8.8.8.8        LOG flags 0 level 4 prefix "[google-dns-out]"


# Perform a DNS query targetting 8.8.8.8
dig example.com @8.8.8.8
... example.com.		18272	IN	A	93.184.216.34

# Verify the request packet in our logs (dmesg)
dmesg | grep google-dns-out
[66458.608136] [google-dns-out]IN= OUT=enp0s3 SRC=10.0.2.15 DST=8.8.8.8 LEN=80 TOS=0x00 PREC=0x00 TTL=64 ID=30602 PROTO=UDP SPT=39573 DPT=53 LEN=60

Producing logs from a Linux loadable kernel module

Not being a driver developer myself (honestly, having never written a kernel module before), it was interesting to look for resources that would teach me how to accomplish this task.

Based on a guide from The Linux Kernel Labs, I came up with the following code for a very simple kernel module:

/**
 * `module.h` contains the most basic functionality needed for
 * us to create a loadable kernel module, including the `MODULE_*`
 * macros, `module_*` functions and including a bunch of other
 * relevant headers that provide useful functionality for us
 * (for instance, `printk`, which comes from `linux/printk.h`,
 * a header included by `linux/module.h`).
 */
#include <linux/module.h>

/**
 * Following, we make use of several macros to properly provide
 * information about the kernel module that we're creating.
 *
 * The information supplied here are visible through tools like
 * `modinfo`.
 *
 * Note.: the license you choose here **DOES AFFECT** other things -
 * by using a proprietary license your kernel will be "tainted".
 */
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Ciro S. Costa");
MODULE_DESCRIPTION("A hello-world printer");
MODULE_VERSION("0.1");

/** hello_init - initializes the module
 *
 * The `hello_init` method defines the procedures that performs the set up
 * of our module.
 */
static int
hello_init(void)
{
         // By making use of `printk` here (in the initialization),
         // we can look at `dmesg` and verify that what we log here
         // appears there at the moment that we load the module with
         // `insmod`.
	printk(KERN_INFO "HELLO-WORLD: Hello world!\n");
	return 0;
}

static void
hello_exit(void)
{
        // similar to `init`, but for the removal time.
	printk(KERN_INFO "HELLO-WORLD: Bye bye world!\n");
}

// registers the `hello_init` method as the method to run at module
// insertion time.
module_init(hello_init);

// similar, but for `removal`
module_exit(hello_exit);

ps.: the source code is available at github.com/cirocosta/hello-world-lkm

Load the module, and there you go:

dmesg | grep HELLO-WORLD
[62076.224353] HELLO-WORLD: Hello world!

Producing logs from kmsg

https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg

A standard interface that we can use to insert messages into the printk buffer is exactly the same that we can use to read the messages from it: /dev/kmsg.

From the kernel documentation:

Injecting messages: Every write() to the opened device node places a log entry in the kernel’s printk buffer.

That is, whatever we echo "haha" as a single write to /dev/kmsg goes into the buffer.

Let’s try it then:

# Generate the message from my current unprivileged
# user and then pipe to the privileged `tee` which is
# able to write into `/dev/kmsg` (yep, /dev/kmsg can be
# see and read, but to write you need more privileges)
echo "haha" | sudo tee "/dev/kmsg"


# Check that we can see our message is indeed there:
dmesg | grep haha
[67030.010334] haha

Seeing kernel logs from user space

Having already spoiled the blog post by mentioning kmsg in the section before, there’s not a bunch to talk now.

The kernel provides to us /dev/kmsg in userspace, a special device that allows us to read from the ring buffer (with multitenancy in mind), and that’s what dmesg uses.

# Trace the syscalls `openat` and `read` to verify
# that it reads from `/dev/kmsg`
strace -e openat,read -f dmesg > /dev/null
openat(AT_FDCWD, "/dev/kmsg", O_RDONLY|O_NONBLOCK) = 3
read(3, "5,0,0,-;Linux version 4.15.0-34-"..., 8191) = 192
read(3, "6,1,0,-;Command line: BOOT_IMAGE"..., 8191) = 142
read(3, "6,2,0,-;KERNEL supported cpus:\n", 8191) = 31
read(3, "6,3,0,-;  Intel GenuineIntel\n", 8191) = 29

Going even deeper, whenever a read(2) syscall is emitted targetting such file, the kernel triggers kernel/printk/devkmsg_read internally, taking a message from the circular queue, formatting it and then sending back to the user.

# We can use `iovisor/bcc#examples/trace/stacknoop.py` to
# gather the stack trace from the execution of `devkmsg_read`
# to verify that the `read` gets right there when `vfs_read`
# is called.
./stacksnoop.py -v devkmsg_read
TIME(s)            COMM         PID    CPU FUNCTION
1.875072956        tail         1565   0   devkmsg_read
	devkmsg_read
	vfs_read
	sys_read
	do_syscall_64
	entry_SYSCALL_64_after_hwframe

A similar path is followed when writing to the circular queue as well: whenever a write(2) is issued, at some point devkmsg_write is called.

devkmsg translates such call to the equivalent of a printk, which then takes the path of reaching log_store, the method that ends up finally taking a free space from the queue and adding the log message:

# In one terminal
echo "haha" > /dev/kmsg

# In another terminal
./stacksnoop.py -v log_store
TIME(s)            COMM         PID    CPU FUNCTION
1.786375046        bash         1450   2   log_store
	log_store
	printk_emit
	devkmsg_write
	new_sync_write
	__vfs_write
	vfs_write
	sys_write
	do_syscall_64
	entry_SYSCALL_64_after_hwframe

By knowing that the kernel provides this interface, we can implement a simple program that constantly performs read(2)s against such device and then parses the contents (messages) that arrive.

Interpreting the kmsg messages

Knowing that every read(2) syscall performed against /dev/kmsg returns us a single message, we’re left to:

implementing a reader that continuously looks at /dev/kmsg and then extracts the raw messages from there; as well as
implementing the parsing of those messages.

The later is the most interesting, imo.

Each message comes packed in the following format: a list of comma-separated info fields and a message (these, separated by a semicolon).

//
//                  INFO		              MSG
//     .------------------------------------------. .------.
//    |                                            |        |
//    |	int	int      int      char, <ignore>   | string |
//    prefix,   seq, timestamp_us,flag[,..........];<message>
//

There are four standard fields in the info section, leaving room for other fields in the future.

From the priority field, we can extract two pieces of information: priority and facility.

// DecodePrefix extracts both priority and facility from a given
// syslog(2) encoded prefix.
//
//	   facility    priority
//      .-----------.  .-----.
//      |           |  |     |
//	7  6  5  4  3  2  1  0    bits
//

With the rest of the fields, we’re able to know when the message was produced and what ID does such message carry in the sequence of messages that have been put in the buffer.

If you’re interested in seeing this parsing in the form of actual code, I wrote a Golang implementation that you can check here: github.com/cirocosta/dmesg_exporter/kmsg/kmsg.go.

Closing thoughts

It was great to have the opportunity of going deep into the stack and verifying how components from the kernel can communicate with userspace applications.

That led me to create dmesg_exporter, a tool for exporting “dmesg” logs metrics so that people can eventually alert messages marked as “critical” arriving from a specific facility.

During the writing of this blog post, I made use of the following book: The Linux Programming Interface. Please make sure you check it out! It’s definitely one of the best books on Linux out there.

If you’ve had any questions, or have ideas for improvements, please let me know! I’m cirowrc on Twitter, and I’d love to hear from you!

Have a good one!

Analyzing Tcpdump capture in real-time with Wireshark

Ciro S. Costa — Sun, 09 Sep 2018 00:00:00 +0000

Hey,

I’ve been doing some work in a Linux VM recently, and having to use secure copy (scp) every time just to copy a tcpdump capture to my MacOS machine to inspect it with Wireshark is not very fun.


        VM                              MACOS

tcpdump --> capture
                                scp linux --> /tmp/capture
                                wireshark /tmp/capture

Given that Wireshark can inspect packets flowing in real time, what if Wireshark perform the equivalent of a tail -f operation on a given packet file?

It turns out that this is possible and pretty simple.

tl;dr: wireshark -k -i <(tail -f -c +0 $PCAP_FILE)

What’s behind `tail`

While tail is pretty much just what it claims to be (a program that outputs the last part of a given fail), it has a pretty neat functionality of being able to “follow” a file (keep track of its status), and, whenever new things come (content gets appended to it), print to stdout these additions.

file_fd = open(/tmp/file.txt)
notification_fd = inotify(file_fd)

while(1):
        select(notification_fd)
        print(read(fd))

tail initializes inotify to keep track of changes happening to a file (let’s say, file.txt;
then it waits for a change on the underlying file;
when the change occurs, tail then reads all that has not been read so far and then keeps track of the position where it was; then
when a new change occurs, does the same thing: reads everything from the old position until the end (read(2) keeps track of the offset under the hood - seeking is only needed at the first time, assuming that arguments like -c and -n are specified).

If you’re curious to see this in practice, you can either check the source code or set up a simple test scenario where you can strace the execution of tail -f.

Opening a capture file with Wireshark

Having installed Wireshark on my MacOS using the regular procedure (downloading the package and then performing the regular installation), wireshark binary should already be in your $PATH.

Looking at the help (wireshark --help), we can see an interesting option to use (-r):

wireshark --help
Wireshark 2.6.2 (v2.6.2-0-g1b3cedbc)
Interactively dump and analyze network traffic.
...

Input file:
  -r <infile>   set the filename to read 
                from (no pipes or stdin!)

Reading tcpdump packet capture in realtime with Wireshark

While being able to read a packet capture in Wireshark is already interesting by itself, it doesn’t solve the problem stated at the beginning of the blog post.

Ideally, it’d be interesting if Wireshark was able to handle the result of a tail -f that could come to it either via stdin (we could pipe tail -f to wireshark), or via a pipe (we could provide the pipe to a flag).

while new_packets:
        tcpdump --->
                /tmp/capture.pcap
                        ------> 
                                wireshark

Having no success with the first option (supplying stuff via stdin), I went with the second (supplying a pipe to -r, the option that makes Wireshark read from a file):

# Execute the `wireshark` binary with the result 
# of the bash process substitution as the argument 
# of `-r`.
#
# The process substitution creates a temporary
# pipe that takes the output of the `tail -f`
# process and makes it readable by `wireshark`.
wireshark \
        -r <(tail -f -c +0 /tmp/capture.pcap)

Wireshark not happy with file being a pipe

No success, but, a pretty clear direction: use -i.

Replace -r with -i and see it “not failing” (nothing starts):

wireshark \
        -i <(tail -f -c +0 /tmp/capture.pcap)

That’s because we’re missing an extra flag (-k): the one that tells Wireshark to start capturing the packets right away:

wireshark \
        -k \
        -i <(tail -f -c +0 /tmp/capture.pcap)

Closing thoughts

It’s great how we can combine something like bash process substitution with a program like Wireshark and achieve a much better result compared to what I had before.

As I’ve decided to go with a MacbookPro for personal use while I still do a bunch of Linux development, having these tricks under the belt helps a lot.

Do you have some that you use all the time as well?

Please let me know!

By the way, if you want to get in touch, I’m very open - reach me at @cirowrc on Twitter!

Cheers

Passing the results of a command as a file to another script

Ciro S. Costa — Sun, 09 Sep 2018 00:00:00 +0000

Hey,

Being accustomed to tying together multiple bash programs with the pipe operator, sometimes I’ve seen myself not being able to easily do so when a command expected the input to come from a file instead of stdin.

Given that `dirname` doesn't expect input from `stdin`, piping to it doesn't work as expected

Using process substitution we can make sure that pretty much every command can perform the equivalent of taking contents from standard input.

tl;dr: dirname <(echo "/var/lib/my/file.txt")

Rationale

Although it’s prevalent for some programs like jq (the command-line JSON processor) to receive things from its standard input, process them, and then give the response back on the standard output, not every program is made like that.

JQ receiving JSON from stdin and procuding a filtered output at stdout

Some programs, like dirname, expect that the input comes as a form of positional argument (others usually expect a flag).

While that might not look like a big deal, for some use cases where you want to streamline operations between multiple processes, it might feel like you can’t easily do that when a file is expected.

The programmer that created the command line interface for the program needs to be aware that someone might want to use its software in such manner and then code it for that.

Example: consuming from `stdin` using Go

For instance, if we’re doing a byte counter in Go, that’s how we’d do it:

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
)

// maxBufferSize defines the maximum size of
// the buffer that we're creating for temporarily
// holding the contents of what's coming from
// stdin.
const maxBufferSize = 4096

func main() {
	var totalRead uint64

	buffer := make([]byte, maxBufferSize)
	reader := bufio.NewReader(os.Stdin)

        // Keep reading in a buffered fashion
        // from `stdin` until an EOF is returned
        // in the form of an error.
        //
        // When that happens, output to `stdout`
        // the total count and exit.
	for {
		n, err := reader.Read(buffer)
		if err != nil {
			if err == io.EOF {
				break
			}

			fmt.Fprintln(os.Stderr, err.Error())
			os.Exit(1)
		}

		totalRead += uint64(n)
	}

        // Write the `stdout` the total amount of
        // bytes that were read.
	fmt.Println(totalRead)
}

Compile the code with a standard go build and check that it indeed works as expected:

# Create a 1MB file named `./1M`
dd if=/dev/zero of=./1M bs=1024 count=1024


# Confirm that we sent 1024*1024 bytes to the file
ls -lah | grep 1M
-rw-r--r--  1 cirocosta  wheel   1.0M  4 Sep 21:13 1M


# Check how many bytes are there in this file by
# running our go code by sending the whole contents
# of the file to the standard input of our command
cat ./1M | ./byte-counter
1048576

Example: consuming from a file using Go

If we were about to change this code to not take from stdin, but instead, read from a file, the changes wouldn’t be significant: take the filename either from a flag or a positional argument and then read it.

diff --git a/tmp/before b/tmp/after
index fe7f39c..e4102af 100644
--- a/tmp/before
+++ b/tmp/after
@@ -16,8 +16,24 @@ const maxBufferSize = 4096
 func main() {
        var totalRead uint64

+       if len(os.Args) < 2 {
+               fmt.Fprintln(os.Stderr, "error: not enough arguments")
+               fmt.Fprintln(os.Stderr, "Usage: byte-counter <filename>")
+               os.Exit(1)
+       }
+
+       fileName := os.Args[1]
+
+       // Open the file that the user provided us
+       // with a path. 
+       // Given that `os.Open` returns os.File which 
+       // implements the `io.Reader` interface, we don't 
+       // need to change the rest of the code.
+       reader, err := os.Open(fileName)
+       if err != nil {
+               fmt.Fprintln(os.Stderr, err)
+               os.Exit(1)
+       }
+
+       defer file.Close()
+
        buffer := make([]byte, maxBufferSize)
@@ -41,8 +57,5 @@ func main() {

Even better, because it doesn’t matter to us what the reader is (the interface only cares about the methods that the implementor implements), we can modify the code very little.

Example: consuming from both a file and stdin using Go

We can go even deeper and simplify our reading logic and make it handle both cases - when it should read from a file and when it should read from the standard input:

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"os"
)

// maxBufferSize defines the maximum size of
// the buffer that we're creating for temporarily
// holding the contents of what's coming from
// stdin.
const maxBufferSize = 4096

func main() {
	var (
		err    error
		reader io.Reader
		writer = ioutil.Discard
	)

	// Make sure that the user always supplies
	// a positional argument to be explcit that
	// it either wants to consume the input of
	// stdin or a file.
	if len(os.Args) < 2 {
		fmt.Fprintln(os.Stderr, "error: not enough arguments")
		fmt.Fprintln(os.Stderr, "Usage: byte-counter (filename|-)")
		os.Exit(1)
	}

	if os.Args[1] == "-" {
		reader = os.Stdin
	} else {
		reader, err = os.Open(os.Args[1])
		if err != nil {
			fmt.Fprintln(os.Stderr, err)
			os.Exit(1)
		}

		defer reader.(*os.File).Close()
	}

	// Use the `io.Copy` method to read the contents
	// of the file into a temporary buffer and then
	// write those to the discard facility (/dev/null).
	//
	// Naturally, there's no need for the `write` part,
	// but this simplifies our code.
	n, err := io.Copy(writer, reader)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	// Write the `stdout` the total amount of
	// bytes that were read.
	fmt.Println(n)
}

Given that for io.Copy all that matters is something that implements the io.Reader interface (and well, both stdin and a file can be referred by a file descriptor in Linux anyway - pretty close semantics), we can use both os.Stdin and the os.File interchangeably with io.Copy. Neat!

Using process substitution to pass a temporary file consuming the output of a command

While it’s pretty well known that bash can interpolate the results of the execution of a given command (like, echo $(date) becomes echo Sun 9 Sep 2018 20:53:41 EDT), it is also capable of interpolating the execution of a command with a named pipe (see the article Introduction to Named Pipes for some great info about what pipes are).

Here is where process substitution comes in place.

Whenever bash spots the <(cmd) operator, it runs the given cmd and then makes its standard output be connected to a temporary named pipe that it creates.

dirname <(echo "/var/lib/my/file.txt")
                |
                *-----> 1. creates a named pipe under /dev/fd/<number>
                *-----> 2. substitutes the execution by the file path

                        dirname /dev/fd/<number>

                        *---> executes the `dirname` command

The interpolation then substitutes such syntax by the path to this temporary file and now whatever program you’re running is capable of reading from this temporary file!

ps: the resulting temporary file is not a regular file.

If the command you’re supplying this named pipe to checks if the file is a regular file, the entire operation might fail (as the file types are different).

Closing thoughts

Having a little bit of extra knowledge about some capabilities of bash is definitely handy.

I really hope I enjoyed this quick bash tip tied with a bit of Go as well.

Please let me know if you have any questions or additions! I’m cirowrc on Twitter and would love your feedback!

Cheers,

Ciro

Measuring HTTP response times with cURL

Ciro S. Costa — Tue, 17 Jul 2018 00:00:00 +0000

Hey,

Oftentimes I see myself needing to check response times while making a bunch of requests quickly. Most of the tools out there either do not continuously make new requests (via an entirely new TCP connection) or don’t expose connection times.

Snapshot of the `httpstat` tool

httpstat is a great tool that does the job really well. It’s a single binary that you can put under your $PATH and have nice stats, but it’s yet another tool, and you might just be inside a container (or maybe a sealed environment?) and want to check response times quickly.

curl comes very handy when needing to have an easy way of displaying response times. Tied with a little of shell scripting, you can even create a comma-separated values output that allows you to see how it changes over time.

Discovering the right curl flags

Looking at man curl (curl’s man page), we can see that there’s an interesting flag named --write-out that we can leverage to gather further insights about the connection:

-w, --write-out <format>
        Make curl display information on stdout 
        after a completed transfer. 
        
        The format is a string that may contain plain 
        text mixed with any  number  of  variables.

        The variables present in the output format will 
        be substituted by the value or text that curl 
        thinks fit, as described below. 
        
        All variables are specified  as %{variable_name}.

What are these variables? Let’s see some of them:

http_code           The  numerical  response  code that was found 
                    in the last retrieved HTTP(S) or FTP(s) transfer.

time_appconnect     The time, in seconds, it took from the start until 
                    the SSL/SSH/etc connect/handshake to the remote host 
                    was completed.

time_connect        The time, in seconds, it took from the start until 
                    the TCP connect to the remote host (or proxy) was 
                    completed.

time_namelookup     The time, in seconds, it took from the start until the name 
                    resolving was completed.

time_pretransfer    The time, in seconds, it took from the start until the 
                    file transfer was just about to begin.

time_starttransfer  The time, in seconds, it took from the start until the first 
                    byte was just about to be transferred.

time_total          The total time, in seconds, that the full operation lasted.

Knowing those variables, it becomes a matter of creating a format (a template) to pass to the --write-out flag specifying which variables we want to see.

# Check out how much time it's going to take for the
# name resolution to happen.
curl --write-out '%{time_namelookup}' https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
0.068856

You can see that it took 0.06 seconds, but it also contains some information there that doesn’t really matter for me - if all we want is the timing information, why get all that information from the body?

We can get rid of that by telling curl to send the body to the null device:

#  -o, --output <file> Write output to <file> instead of stdout
curl \
        --output /dev/null \
        --write-out '%{time_namelookup}' \
        https://google.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100   220    0     0    462      0 --:--:-- --:--:-- --:--:--   462
0.069249

Well, we still have some stuff left - there are these extra header lines that tells us curl’s progress.

Fortunately, we can get rid of that too:

# -s, --silent
# Silent  or  quiet  mode.  
# Don't  show progress meter or error messages.  
# Makes Curl mute. 
# It will still output the data you ask for.
curl \
        --output /dev/null \
        --silent \
        --write-out '%{time_namelookup}' \
        https://google.com
0.004344

Cool, now that we have the information we want, let’s tailor a script retrieves all the information needed.

A snippet for printing a CSV of response times using CURL

Knowing how the command looks like, now it’s a matter of creating a better defined templated and let it run for n times:

#!/bin/bash

# Set the `errexit` option to make sure that
# if one command fails, all the script execution
# will also fail (see `man bash` for more 
# information on the options that you can set).
set -o errexit

# This is the main routine of our bash program.
# It makes sure that we've supplied the necessary
# arguments, then it prints a CSV header and then
# keeps making requests and printing their responses.
#
# Note.: because we're calling `curl` each time in
#        the loop, a new `curl` process is created for 
#        each request. 
#       
#        This means that a new connection will be made 
#        each time.
#       
#        Such property might be useful when you're testing
#        if a given load-balancer in the middle might be
#        messing up with some requests.
main () {
  local url=$1

  if [[ -z "$url" ]]; then
    echo "ERROR:
  An URL must be provided.

  Usage: check-res <url>

Aborting.
    "
    exit 1
  fi

  print_header
  for i in `seq 1 10000`; do
    make_request $url
  done
}

# This method does nothing more that just print a CSV
# header to STDOUT so we can consume that later when
# looking at the results.
print_header () {
  echo "code,time_total,time_connect,time_appconnect,time_starttransfer"
}

# Make request performs the actual request using `curl`. 
# It specifies those parameters that we've defined before,
# taking a given `url` as its parameter.
make_request () {
  local url=$1

  curl \
    --write-out "%{http_code},%{time_total},%{time_connect},%{time_appconnect},%{time_starttransfer}\n" \
    --silent \
    --output /dev/null \
    "$url"
}

main "$@"

Trying it out:

./check-res https://google.com
code,time_total,time_connect,time_appconnect,time_starttransfer
301,0.469397,0.125566,0.292027,0.469221
301,0.380109,0.049464,0.204842,0.379735
301,0.389428,0.048424,0.208052,0.389043
301,0.731204,0.047314,0.547336,0.730819
301,0.379709,0.048266,0.205082,0.379577
301,0.384067,0.048614,0.211338,0.383765

Closing thoughts

While there are tools out there that are able to do this, it’s very useful to have some options to quickly test out when you’re in the middle of an on-call.

I hope this article was useful for you! If you have any questions or just want to chat a bit, I’m @cirowrc on Twitter.

I’m planning to post some other articles regarding useful shell scripts that I use, so make sure you also subscribe to the mailing list to get updates on that topic.

Have a good one!

Ciro

How to minify and bundle assets using Hugo

Ciro S. Costa — Mon, 09 Jul 2018 00:00:00 +0000

Hey,

Hugo v0.43 just got released today (see the GitHub Releases page for Download links and notes about all that’s included in v0.43) with great news!

From my point of view, the most significant feature in this release (as noted in the gohugo.io website) is the introduction of the asset pipeline (Hugo Pipes), a way of allowing the user to specify transformations to be applied to assets (css, javascript, svg and more) right from the template.

The tl;dr is illustrated in the release docs:

{{ $styles := resources.Get "scss/main.scss" |
        toCSS |
        postCSS |
        minify |
        fingerprint }}

<link   rel="stylesheet" 
        href="{{ $styles.Permalink }}" 
        integrity="{{ $styles.Data.Integrity }}" 
        media="screen">

As I was in the middle of refactoring this blog’s template to better support Accelerated Mobile Pages Project (AMP), this release came just in time.

You can check why this is useful for AMP below, but first, some intro.

What’s minification all about?

Although we usually code for humans (hopefully), placing a great deal of space between elements, adding comments and making either function names or variables verbose, the browser doesn’t really care about all of that.

For instance, consider the following Javascript file (491 bytes):

// maybeRemoveTrailingSlash - removes the trailing slash (`/`) 
// of a given string if it has one. Otherwise, nothing is done.
function maybeRemoveTrailingSlash(str) {
   // Perform an early termination if it doesn't match
   // the type of string that we're looking for.
   if (!str.endsWith("/")) {
      return str;
  }

  // Remove the last character.
  return str.substring(0, str.length - 1);
}

console.log(maybeRemoveTrailingSlash("a"));
console.log(maybeRemoveTrailingSlash("a/"));

When we consider the fact that every byte that you add in a given file needs to shipped via an unreliable network (that might be very expensive for the user in the case of Mobile phones), that’s us requiring our customers to not only wait longer, but pay more to get our content.

By minifying our assets we can remove all that’s needed for a human to understand the code (thus, effectively reducing its size), while still maintaining its semantics.

The code above could then become something like this:

// Minified - 
a=(n)=>n.endsWith("/")?n.substring(0,n.length-1):n
b=console.log
b(a("a"))
b(a("a/"))

That’s a reduction from 491 bytes to 85 bytes!

What about bundling?

Bundling allows us to take a series of files (for instance, base.css which specifies the base styling of our website, and syntax.css, which specifies coloring for of pre blocks) and concatenate them all such that the end result is a single file.

Although nowadays that’s a practice that is not very useful anymore in many cases (see Musings on HTTP/2 and Bundling), there’s still AMP forcing us to adhere to such practices - all custom styles must be defined in the head section of the page within an <style amp-custom> element (see AMP - Supported CSS).

Hugo Pipes comes very handy for this.

The asset bundling and minification pipeline

Knowing that we have those two primitives to build our asset pipeline upon, we can go ahead and actually put it into code.

Having my _default/single.html (the template that defines the page of a single article) defined like this:

<!doctype html>
<html lang="en">
  <head prefix="og: http://ogp.me/ns# article: http://ogp.me/ns/article#">
    {{ partial "content/single/base_head.html" . }}
    {{ partial "content/single/ld.html" . }}
  </head>

  <body class="article">
    {{ partial "content/single/main.html" . }}

    {{ partial "footer.html" . }}
    {{ partial "style.html" . }}  <!-- THIS IS WHERE I'LL USE HUGO PIPES -->
    {{ partial "tracking.html" . }}
  </body>
</html>

I can then move on and define that partial (partials/style.html):

{{ $cssOpts := (dict "targetPath" "styles/syntax.css") }}
{{ $base := resources.Get "css/index.css" }}
{{ $syntax := resources.Get "css/syntax.css" }}

{{ $style := slice $base $syntax | resources.Concat "bundle.css" | minify | fingerprint }}

<link rel="stylesheet"
      href="{{ $style.Permalink }}"
      integrity="{{ $style.Data.Integrity }}">

ps.: the snippet above is horizontally scrollable.

Once an article gets rendered, that <link> element becomes what you might expect:

<link rel="stylesheet"
      href="https://ops.tips/bundle.min.e838f1fc0247d7eff52385847b5a86ec8747ed58aeeb35d32f8c22926256d7ec.css"
      integrity="sha256-6Djx/AJH1&#43;/1I4WEe1qG7IdH7Viu6zXTL4wikmJW1&#43;w=">

ps.: you can check this out by prepending view-source: to this article page and go to the end of the page if you’re using Chrome.

Making the minified bundle AMP ready

If you’re into AMP though, you know that what we did above doesn’t provide a valid AMP page.

That’s because you can’t make use of link elements unless it’s for specifying a custom font resource. We have to put the CSS code in the html file in the head section inside a style element. Period.

To do that, I make use of Hugo’s Custom Output Formats.

Just like I have a _default/single.html template, I create an extra _default/single.amp.html, in which I can then specify a different partial: style.amp.html.

<!doctype html>
<html amp lang="en">
  <head prefix="og: http://ogp.me/ns# article: http://ogp.me/ns/article#">
    {{ partial "content/single/base_head.html" . }}
    {{ partial "content/single/ld.html" . }}
    {{ partial "amp-boilerplate.html" . }}
    {{ partial "style.amp.html" }}  <!-- HERE'S THE STYLE PARTIAL -->
  </head>
  <body class="article">
    {{ partial "content/single/main.html" . }}

    {{ partial "footer.amp.html" . }}
    {{ partial "tracking.amp.html" . }}
  </body>
</html>

Properly placed in the head as AMP mandates, we then implement the partial:

{{ $cssOpts := (dict "targetPath" "styles/syntax.css") }}
{{ $base := resources.Get "css/index.css" }}
{{ $syntax := resources.Get "css/syntax.css" }}

{{ $style := slice $base $syntax | resources.Concat "bundle.css" | minify }}

<style amp-custom>
{{ $style.Content | safeCSS }}
</style>

Said and done, now we have inlined bundled and minified CSS for our AMP pages:

Closing thoughts

Being a Hugo user for not a long time (since I started this blog in November 2017), Hugo has always been a piece of cake to deal with, allowing me to have my blog up and running without knowing a lot about it while still providing me a great deal of value as I learn more and more.

I’m delighted that this new release as it removes complexity - while you needed to do those optimizations and further templating yourself before, now it’s even easier (my Makefile just sent a Thanks").

If you have any questions or found something weird here, please let me know! I’m cirowrc on Twitter.

In case you want to receive updates about content like this, make sure you subscribe to the mailing list.

Have a good one!

Ciro

Blog Archive

Ciro S. Costa — Fri, 06 Jul 2018 00:00:00 +0000

Gists

Ciro S. Costa — Fri, 06 Jul 2018 00:00:00 +0000

Differently from the blog posts, these are smaller posts which are usually more direct; not a lot of plain-text explanation - more code.

Shell: replacing a variable with the contents of a file

Ciro S. Costa — Thu, 05 Jul 2018 00:00:00 +0000

Hey,

I’ve been trying to improve the Hugo template that I created for this blog, and one of the things I wanted to do was inject the contents of an index.css file into a index.amp.html layout file - Accelerated Mobile Pages (AMP) requires us to have our custom styles inlined in the HTML (see Custom Amp Styles).

Here I go through a way of achieving that - using awk - and a way of not achieving that - using sed (naively).

If you ever need to template a file with contents of another, this gist is for you.

ps.: yes, with Hugo you could use partials, but let’s say you want to just have some sort of poor man’s templating engine.

If you want to skip the introduction, jump to the awk section.

Getting started with the naive way - sed without escapeing

Knowing that sed is all about performing substitions, it was the first tool I tried.

Unfortunatelly, it didn’t work as I expected.

# Create the file that contains the contents that we want
# to insert into another file at a specific location.
echo 'xxxxxxx
yyyyyyy
zzzzzzz' > ./content.txt

# Create the template that we want to have the variable
# substitution.
echo '11111111
22222222
__REPLACE_ME__
33333333' > ./template.txt


# Perform the substituion of `__REPLACE_ME__` by a single
# string just to test the `sed` syntax.
sed 's/__REPLACE_ME__/SOMETHING_NEW/g' ./template.txt
11111111
22222222
SOMETHING_NEW
33333333


# Well, if it worked for that simple case, why not just 
# replace `SOMETHING_NEW' by all the contents of the
# file? We can get that via `cat` anyway:
CONTENT=$(cat ./content.txt)
sed "s/__REPLACE_ME__/$CONTENT/g" ./template.txt

sed: 1: "s/__REPLACE_ME__/xxxxxxx
 ...": unescaped newline inside substitute pattern

As we can see, that doesn’t work.

The main reason is that the command was not properly formatted - there are newlines not being escaped when we perform the substitution.

That happens because bash is performing a simple text substitution when we specify $CONTENT.

To see that working, make use of the xtrace option invoking bash directly:

# By invoking bash with the `-x` flag, we enable the 
# `xtrace` option (which is usually added to scripts
# when debugging by specifying `set -o xtrace` or
# `set -x`).
#
# The options has the effect of expanding every command
# and variables that are supplied to it.
#
# This way, we're able to see what bash used as the full
# command after substituting the $CONTENT variable.
/bin/bash -x -c \
        "sed \"s/__REPLACE_ME__/$CONTENT/g\" ./template.txt"

+ sed 's/__REPLACE_ME__/xxxxxxx
yyyyyyy
zzzzzzz/g' ./template.txt
sed: 1: "s/__REPLACE_ME__/xxxxxx ...": 
unescaped newline inside substitute pattern

Knowing that, we can either:

escape the newlines before supplying $CONTENT to sed; or
make use of something that has greater flexibility (like awk).

With 1, it means that we’d need to get into regular expressions and some fancy sed syntax (see this stackoverflow answer).

With 2, we can create a little program that we can be very verbose about what’s doing and make the whole process pretty explicity. I’ll go with this one.

Moving on - using AWK to template the file

AWK (just like sed) is a pretty ubiquotous tool in Unix systems, meaning that you can find it everywhere.

It allows you to create data-driven programs where given a set of rules, actions are performed whenever a match happens.

By taking data from files (either stdin or regular files supplied via arguments), awk contiguously checks whether your matchers match and then it so, proceeds with the action specified.

“The awk utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs.”

“The basic function of awk is to search files for lines (or other units of text) that contain certain patterns.”

— The GNU Awk User’s Guide

Given that, we can start modelling a solution for our problem in terms of matches:

we definitely want to match __REPLACE_ME__ at a given point; and
we certainly want to match lines those lines that are not meant to be replaced and just printed to stdout.

We can also model it in terms of actions:

we want to replace the __REPLACE_ME__ tag; and
we want to just print to the standard output what doesn’t match the __REPLACE_ME__ tag.

Now, it’s a matter of adhering to the awk specifics and implementing it.

note.: because awk gives us a fully featured programming language, here I include parameter validation and the ability for the user to specify the pattern that (s)he wants to replace by using an environment variable (PATTERN).

#!/usr/bin/awk -f

# templater - takes a file and replaces a variable
# in a given template with the contents of another file.

# err() - Prints a supplied `text` to standard error.
# @text: Text to be printed to stderr.
function err (text) {
        print text > "/dev/stderr"
}


# The BEGIN matcher is a special type of matcher that
# gets executed whenever the AWK program is starting
# and no records have been matched yet.
BEGIN {
        if (ARGC != 3) {
                err("Error: not enough arguments.")
                err("")

                err("Usage: ./templater <content_file> <template_file>")
                err("Aborting.")
                exit 1
        }

        if (length(ENVIRON["PATTERN"]) == 0) {
                err("Error: no pattern specified.")
                err("")

                err("Specify a pattern via the `PATTERN` environment variable.")
                err("For example: ")
                err("  PATTERN=__CONTENT__ templater contents.txt template.txt")
                err("Aborting.")
                exit 1
        }
}

# By using the `NR=FNR` pattern we're able to specify
# an action that we want to perform only on the first
# file that we supply via the command line.
#
# FNR is a counter that keeps track of the current line
# in the current file that is being processed.
#
# NR is a counter that keeps track of the total number
# of lines that have been processed so far.
#
# By trying to match `NR==FNR` we can perform an action
# in the very first file. To visualize that, we can set
# up an experiment:
#
#       $ cat file1
#       a
#       b
#       c
#
#       $ cat file2
#       d
#       e
#
#       $ awk '{print FILENAME, NR, FNR, $0}' file1 file2
#       file1 1 1 a
#       file1 2 2 b
#       file1 3 3 c
#       file2 4 1 d -> not equal -> starts the second one
#       file2 5 2 e -> not equal
#
# In the action we can then store all the lines from
# the first file in memory so that we can use it later
# when we find the string to replace.
#
# By specifying the `next` statement, no further matching
# is performed for this record (line).
#
# ps.: we could also check `FILENAME`, like:
#       FILENAME==ARGV[1]
NR==FNR {
        content_lines[n++]=$0;
        next;
}

# Once we find the string to replace, we iterate over
# all the lines that we stored (from the first file)
# and then once we're done, we force AWK to immediately
# stop processing the current record so that it doesn't
# print `__CONTENT__` and don't proceed with performing
# further matches for this record (line).
#
# ps.: if you didn't want to take a variable here, for
# instance, have a fixed pattern to replace, you could
# simply use `/PATTERN/ { ... }`.
$0 ~ ENVIRON["PATTERN"] {
        for (i = 0; i < n; i++) {
                print content_lines[i];
        }
        next
}

# Given that 1 always evaluates to `true`, this is a match
# that will always occur.
#
# As we can either omit an action or a match (not both!),
# we can use a catch-all match (1) and let awk use the
# default action (print current line).
#
# This has the effect of printing all lines that didn't
# match the other matches that we specified above.
1

As we’re already specifying the interpreter directive (#!/usr/bin/awk -f), we can execute the program by making it executable and then feeding some data to it:

# Make it executable
chmod +x ./templater.awk

# Execute it passing the necessary arguments
PATTERN=__REPLACE_ME__ \
        ./templater.awk \
        content.txt \
        template.txt
11111111
22222222
xxxxxxx
yyyyyyy
zzzzzzz
33333333


# Using `-` as the `template.txt` argument
# and feeding its `stdin` also works:
export PATTERN=__REPLACE_ME__
echo "this is
very cool
__REPLACE_ME__
right?" | ./templater.awk \
        content.txt \
        -
this is
very cool
xxxxxxx
yyyyyyy
zzzzzzz
right?

Closing thoughts

Although I’ve copied and pasted AWK snippets now and then, I’ve never really understood the intrisics of them.

Now, going through its documentation, it feels like I have so much more power when it comes to processing text in the terminal.

I hope you feel that way too! If you’re willing to know more, make sure you check out the gawk manual. There you have multiple examples and it’s also very detailed.

If you have any questions or just want to make contact, feel free to reach me at @cirowrc on Twitter.

Have a good one!

finis

Developing eBPF code with autocompletion support

Ciro S. Costa — Sat, 30 Jun 2018 00:00:00 +0000

Hey,

Some days ago I wrote about how you can configure YouCompleteMe to navigate the Linux source code.

One great benefit of going through such setup is that it showed me how I could use the very same concepts to develop eBPF code better by leveraging autocompletion and jumps to definitions and declarations.

While it’s common for people to just embed their eBPF code directly into Python scripts, I find such approach very hard to debug (not being very accustomed to Kernel stuff).

Given that not all eBPF code targets the same subsystems in the Kernel, some peculiarities arise when trying to provide proper autocompletion.

ps.: I’m still learning these things!

Autocompleting eBPF compiled via Clang without BCC

If you’re performing the builds yourself (calling clang with the bpf backend specified via -target bpf), nothing more than a basic .ycm_extra_conf.py like showed in the article mentioned before is needed.

Usually, all you need is referencing /usr/include and you’re done.

def FlagsForFile( filename, **kwargs ):
        return { 'flags': [
            '-I/usr/include',
            '-I/usr/local/include',
            '-std=gnu99', # this allows us to leverage `asm`
            '-xc'
        ]}

ps.: if you need to reference internal kernel structures, make sure you include the proper paths from your local kernel source. For instance, if I have the kernel cloned at /home/ubuntu/linux, then I’d probably add something like -I/home/ubuntu/linux/include or whatever else you need from it.

As a more specific example, in case you’re targetting ebpf programs for tc, it might be useful that you leverage the helpers from iproute2. It ships with a set of helpers that might be useful for you when using eBPF with tc, so it might be worth cloning iproute2 and manually installing their bpf helpers file:

# Clone the IPROUTE2 repository and get into the directory
git clone git://git.kernel.org/pub/scm/network/iproute2/iproute2.git 
cd ./iproute2

# Copy the `bpf_api.h` helpers file that lives under `./include`
# to your `/usr/include` directory.
#
# ps.: this could be anywhere - including your current source tree.
install \
        -m 0644 \
        ./include/bpf_api.h \
        /usr/include/iproute2

Now, create a bpf hello world that gets loaded into the ingress path and you’ll see that the definitions properly show up:

#include <iproute2/bpf_api.h>
#include <linux/bpf.h>

__section("ingress") 
int cls_ingress(struct __sk_buff* skb)
{
        (void)skb;
	printt("Hello World\n");

	return TC_ACT_UNSPEC;
}

With the autocompletion working, we can quickly visualize what are the fields that the __sk_buff structure gives us:

More importantly, we can get our Vim cursor on top of it and jump to its declaration (YcmCompleter GoToDeclaration).

Given that there are helpers defined in iproute2/bpf_api.h, and that we include it (and YouCompleteMe knows where to find this file given that we specified the include list at .ycm_extr_conf.py), we can do the same for the bpf helper functions (as shown in the image at the beginning of the article).

Autocompleting eBPF compiled using BCC

If you’re using bcc to get your eBPF in, then the story changes a little bit - there’s a subset of code that gets injected behind the scenes (not very good for us, trying to provide the whole source code to the autocompletion engine).

Such subset is what gives you some handy definitions like BPF_HASH.

Not only BCC provides helpers, but it actually acts as part of the Clang compilation process, transforming the eBPF source code that we provide.

For instance, we can see how bcc adds some helpers:

// Helpers are inlined in the following file (C). 
// Load the definitions and pass the partially compiled 
// module to the B frontend to continue with.
auto helpers_h = ExportedFiles::headers()
        .find("/virtual/include/bcc/helpers.h");

if (helpers_h == ExportedFiles::headers().end()) {
        fprintf(stderr, "Internal error: missing bcc/helpers.h");
        return -1;
}

if (int rc = load_includes(helpers_h->second))
        return rc;

BLoader b_loader(flags_);

Source code from bpf_module.cc#L970-l980

I’m still not completely sure about how it interacts with the LLVM toolchain, but it seems like it’s something like this:

ps.: There’s a great presentation by Alexei Starovoitov illustrating the process: Slides - BPF in LLVM and kernel.

If we want to leverage those definitions (from the injected helpers.h) in our autocompletion though, we need to reference that file manually. The catch here is that if you look at it, there’s an extra line at both the beginning and end that would invalidate our use - it makes the whole code a comment.

R"********(
/*
 * Copyright (c) 2015 PLUMgrid, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.

 [...]

        } while (0);
#endif
)********"

Naturally, that’s not acceptable for us as including it would fail.

To remove the first and last line, we can then make use of head and tail:

# Create the location at which we plan to store the `helpers.h`
# file.
mkdir -p /usr/local/include/bcc


# Clone the bcc project
git clone https://github.com/iovisor/bcc


# `tail -n +2` removes the first line by making
# it send to `stdout` only the last `n` lines starting
# at the second line.
#
# `head -n -1` does the opposite: it the first `n-1`
# lines (that is, all of the lines except the last one)
# and sends them to stdout.
#
# Given that we're doing those two operations in a pipeline
# manner, in the end, we have removed the first and the
# last lines.
#
# Piping the result to `/usr/local/include/bcc/helpers.h` makes 
# the modified content available at such location.
cat ./bcc/src/cc/export/helpers.h | \
        tail -n +2  | \
        head -n -1 | \
        sudo tee -a > /usr/local/include/bcc/helpers.h

Now, let’s create an example to be used with bcc to see the autocompletion working.

First, create a Python wrapper that is going to take our source and then make use of the BCC toolchain.

The BCC toolchain uses its frontend to translate our code to a Clang-compatible code which gets turned into BPF bytecode via the LLVM BPF backend.

from bcc import BPF

# BPF creates a new BPF module with the given
# source code that we can specify either via
# `src_file` or `text`.
#
# `trace_print` reads the kernel debug trace pipe
# and then prints what's there to stdout.
#
# This way, we're able to easily visualize our
# debug statements issued via `bpf_trace_printk`.
BPF(src_file='./trace_vfs_open.c').trace_print()

With the wrapper set, we can the code our trace_vfs_open.c file:

/**
 * If `BPF_HASH` is already defined, it means that we're
 * in the BCC compilation path, so we don't need to import
 * `bcc/helpers.h` as its been already added to this file
 * already.
 */
#ifndef BPF_HASH
#include <bcc/helpers.h>
#endif

/**
 * kprobe__vfs_open() - instrument the vfs_open call with a custom handler
 * @ctx: registers and the bpf context (unused).
 *
 * Instruments the internall call `vfs_open` (`fs/open.c`) which is responsible
 * for allocating a file structure, initializing it and then submitting a
 * subsequent `open` call to the filesystem.
 *
 * By using the special prefix `kprobe__`, `bcc` will automatically
 * attach the kprobe for us to the kernel method `sys_statfs`.
 */
int
kprobe__vfs_open(void* __attribute__((unused)) ctx)
{
	/**
	 * `bpf_trace_printk` is a method that gets defined by the bcc
	 * toolchain - see [1].
	 *
	 * It defines a `printk`-like facility for debugging (that should
	 * really just be used for quick debugging).
	 *
	 * [1]:
	 * https://github.com/iovisor/bcc/blob/d17d5a8fd4f3b8a9638c8326a77b56ba56dc5eec/src/cc/frontends/clang/b_frontend_action.cc#L840-L852
	 */
	bpf_trace_printk("vfs_open called\n");

	return 0;
}

Although the example is pretty contrived, we can already see where the autocompletion support shines - we can discover which operations we can perform with the ctx argument:

If we switch this trace from vps_open to something more elaborate, like tcp_v4_connect, then we can see how such functionality shines. Consider the tcpv4connect.py example from the BCC repository:

int kprobe__tcp_v4_connect(
        struct pt_regs *ctx, 
        struct sock *sk)
{
	u32 pid = bpf_get_current_pid_tgid();
	// stash the sock ptr for lookup on return
	currsock.update(&pid, &sk);
	return 0;
};

If we wanted to know more about sock, using the autocomplete feature we can quickly do it:

Closing thoughts

Not being a Kernel developer myself, I found that by leveraging some additional tooling made me much more productive when learning about eBPF.

It seems to me that if more people who are not very educated about Linux start getting into eBPF, then the more approachable this area tends to get.

So, I hope that this helped you! Please let me know if you find something weird (or completely wrong). I’m cirowrc on Twitter.

Have a good one!

finis

Blocking ingress traffic to Docker swarm worker machines

Ciro S. Costa — Wed, 27 Jun 2018 00:00:00 +0000

Hey,

When Docker Swarm mode got announced, one of the big features included was the routing mesh.

Image from docs.docker.com

Although the feature indeed works as expected, there’s the possibility that you might not want to have all of your nodes accepting connections and performing the job of a load-balancer.

In this blog post, I go through what are the fundamental blocks that the routing mesh uses under the hood so we can block such kind of traffic on specific machines.

The ingress load-balancing setup lifecycle

Given a cluster of machines, whenever a service publishes a port, any machine can be targetted at such port, and an internal load-balancer (IPVS) will send the traffic to a host containing an instance of such service.

For instance, assume that we have the following service definition:

version: '3.2'
services:
  nginx:
    image: 'nginx:alpine'
    ports:
      - '8000:80'
    deploy:
      replicas: 1

And that we have three machines: worker-1, worker-2 and manager.

Once the service is pushed to a Docker Swarm manager, it gets the service instantiated in the form of tasks - only one in this case (replicas == 1)-, which turns into a container in a machine chosen by the manager.

Having a port published (target=80,published=8000), Swarm proceeds with the configuration of the routing mesh, which happens at two moments:

at the time that the service gets created, all participating nodes in the cluster set up their IPVS configuration to have a virtual server listening on port 8000.
at the moment the task lands in a node and a container is created, then all machines update their IPVS configuration again, but now to include a real server that corresponds to the machine in which the container landed, forming the mesh.

Each machine now has the published port (8000) set to listen for incoming connections so that you can target any of them and have the connection adequately established to a container.

If it happens that the nginx container dies and the scheduler (Docker Swarm) creates a task for placing the container somewhere else, or it happens that we have a scale out situation (increasing the number of replicas to two, for instance), all machines become aware of that and update their IPVS configurations.

If the service is removed (or a published port gets unpublished), then again, all participants get to know the change in the overall state and then reflect such changes in their local IPVS configuration.

Under the hood - how Ingress gets traffic directed to containers

If at this point you’ve tried to see how IPVS gets set up, you might’ve noticed that nothing shows up by running ipvsadm --list.

That’s because such configuration is isolated in a namespace (ingress_sbox), and to have a full understanding of how the packet flow looks like, we need to take a look at the iptables rules set in the default namespace to make those packets get into the ingress namespace which is then responsible for doing the whole load-balancing.

To do so, let’s assume a more straightforward scenario:

there are two machines (machine1 and machine2), both members of the swarm cluster (it doesn’t matter which one is the manager);
there’s a nginx service deployed (publishing port 8000 for the target port 80), having the one (and only one) replica landed in machine1; and
we’re reaching the nginx service from machine2.

When the packet gets received by the machine, its device issues an interrupt. A driver that has registered itself as a handler for that takes the packet data from the device (via direct memory access - DMA) and then fills the proper kernel representation of it in its driver (skb). After this, the packet starts flowing through a series of steps in the kernel before it’s sent to a local application or forwarded to a different interface.

One of these steps is passing the packet through the kernel firewall, which can be configured from userspace using iptables (and many others), allowing you to set rules that can do all sorts of packet mangling, filtering, forwarding and more.

These rules that we specify are placed in either preconfigured chains - hooks in the packet flow that are activated at certain points - or custom ones that we can reference from those that are preconfigured.

These chains live under tables, which aggregate a set of chains by their purpose.

For now, let’s focus on the nat table - the table where rules are set to implement network address translation. From the man page:

nat: This table is consulted when a packet 
     that creates a new connection is encountered.

It  consists  of  four  built-ins: 
- PREROUTING (for altering packets as soon as they come in), 
- INPUT (for altering packets destined  for  local  sockets),  
- OUTPUT  (for altering  locally-generated  packets before routing), and
- POSTROUTING (for altering packets as they are about to go out).

Being even more specific, let’s focus on the PREROUTING chain and those chains that are referenced there:

tip: As a way of quickly grasping over all the additions to iptables that Docker made, I like to run iptables-save so we can see all that’s set across all tables.

# List all the chains from the NAT table (by default,
# iptables lists the chains from the FILTER table).
iptables --list --table nat

# As the packet has not been generated in our machine,
# packets get to this chain first before a routing 
# decision is taken (i.e., before it's either set that
# the packet is indeed for this machine - INPUT - or that 
# it's meant to be sent to another machine - FORWARD).
Chain PREROUTING (policy ACCEPT)
target          prot opt source     destination         
# Using the `addrtype` iptables extension, it looks at
# the packets and then performs matching based on its
# destination address type.
#
# When this rule was set, it was configured to match the
# address type of the destination address to LOCAL - a
# local address.
#
# ps.: LOCAL does not mean only 127.0.0.0/8 CIDR used for
# loopback addresses, but any address that represents the
# local machine (as per the routing table).
DOCKER-INGRESS  all  --  anywhere   anywhere       ADDRTYPE match dst-type LOCAL
DOCKER          all  --  anywhere   anywhere       ADDRTYPE match dst-type LOCAL

Chain DOCKER-INGRESS (2 references)
target          prot opt source     destination   
# Matches packets that make use of the TCP protocol and 
# that are targetted towards port 8000, applying `DNAT`ing
# to them, effectively substituting the destination address
# to 172.18.0.2.
DNAT            tcp  --  anywhere   anywhere       tcp dpt:8000 to:172.18.0.2:8000
RETURN          all  --  anywhere   anywhere  


Chain DOCKER (2 references)
target          prot opt source     destination

Now, taking a quick look at ip addr (lists all the devices in the current network namespace together with their addresses), we can notice that such ip (172.18.0.2) must belong to the device connected to the docker_gwbridge bridge:

# Verify that the `docker_gwbridge` interface that belongs
# to the bridge device is indeed the gateway for the 172.18.0.1/16
# network.
ip addr show docker_gwbridge
13: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:af:92:92:f6 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:afff:fe92:92f6/64 scope link 
       valid_lft forever preferred_lft forever

# Check which interfaces have been connected to this bridge.
bridge link show docker_gwbridge
25: veth79f1181 state UP @if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master docker_gwbridge state forwarding priority 32 cost 2 
29: veth8aec496 state UP @if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master docker_gwbridge state forwarding priority 32 cost 2 

# We can see that there are two veth pairs connected to it.
#
# Naturally, being veth pairs, from this network namespace we can see
# one side of the pairs, while the other side have their interfaces in
# different network namespaces.
#
# We can corroborate the hypothesis that there exists at least two different
# namespaces by checking that under the hood the `docker_gwbridge` docker network
# has two hidden containers:
docker network inspect \
        --format '{{ json .Containers }}' \
        docker_gwbridge | jq
{
  "ddf1af4f39efaeb556964351800f282aca75d509537d166ff78d5d12b8911cef": {
    "Name": "gateway_ddf1af4f39ef",
    "EndpointID": "5903ff858bc2d5...",
    "MacAddress": "02:42:ac:12:00:03",
    "IPv4Address": "172.18.0.3/16",
    "IPv6Address": ""
  },
  "ingress-sbox": {
    "Name": "gateway_ingress-sbox",
    "EndpointID": "650cc71d444f...",
    "MacAddress": "02:42:ac:12:00:02",
    "IPv4Address": "172.18.0.2/16",     # <<< the ip we saw!
    "IPv6Address": ""
  }
}

# Get inside this container and see what's going on there when
# it comes to IPVS.
nsenter \
        --net=/var/run/docker/netns/ingress_sbox \
        ipvsadm --list --numeric
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  257 rr
  -> 10.255.0.6:0                 Masq    1      0          0       

# Translating that output to pure English, it means:
# - for those connections marked with the firewall mark 257, load-balance
#   them using NAT (masquerading) between the list of servers below (given
#   that we have a single replica, there's only one address there).
#
# To understand where that IP address comes from, get into the nginx
# container and check its interfaces:
 docker exec $NGINX_CONTAINER_ID ip addr show eth0
26: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:ff:00:04 brd ff:ff:ff:ff:ff:ff
    inet 10.255.0.4/16 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever

# Now if you're wondering where this network comes from, checkout 
# `docker network ls` and inspect the `ingress` network:
docker network inspect ingress --format '{{ json .IPAM }}' | jq
{
  "Driver": "default",
  "Options": null,
  "Config": [
    {
      "Subnet": "10.255.0.0/16",
      "Gateway": "10.255.0.1"
    }
  ]
}

The last thing that’s missing understanding from the ingress sandbox is: how does fwmark (firewall mark) that IPVS uses gets set? The answer is, one more time, iptables.

However, the difference now is that this gets set in the mangle table:

# Get inside the `ingress_sbox` namespace and then issue
# the iptables command to list the chains and rules that
# are set in the `mangle` table.
nsenter \
        --net=/var/run/docker/netns/ingress_sbox \
        iptables --list --table mangle
Chain PREROUTING (policy ACCEPT)
target     prot src  dst 
# For every packet that is destined to port 8000 (the
# nginx' published port), set the firewall mark to 0x101).
MARK       tcp  any  any  dpt:8000 MARK set 0x101

Chain OUTPUT (policy ACCEPT)
# For every packet that is destined to a virtual IP that swarm
# assigns to a specific service (10.255.0.3), also mark the packet
# with the corresponding virtual service mark.
#
# Given that this destination (10.255.0.3) lives in the ingress
# network, this is only accessible from within a container.
target     prot opt source               destination         
MARK       all  --  anywhere             10.255.0.3           MARK set 0x101

# Given that in iptables that value is set in its hexadecimal
# form, we can look at what 0x101 means in the decimal form
# (shown in the `ipvsadm` listing).
printf "%d\n" 0x101
257

By looking at the man page of iptables-extensions we can see what that target does:

“MARK - This target is used to set the Netfilter mark value associated with the packet.”

From https://linux.die.net/man/8/iptables

And then, looking at ipvsadm man page we can also check how it can make use of such mark:

“Using firewall-mark virtual services provides a convenient method of grouping together different IP addresses, ports, and protocols into a single virtual service. This is use‐ ful for both simplifying configuration if a large number of virtual services are required and grouping persistence across what would otherwise be multiple virtual ser‐ vices.”

From https://linux.die.net/man/8/ipvsadm

In summary, what goes on is:

A packet is destined for our machine (192.168.10.4); then
if it’s destination port matches a published port, we take the original destination address (our machine address) and replace it by a different address (172.18.0.2). As this is performed before a routing decision has been made, it has the effect of making the packet be routed to the gateway_ingress-sbox container that contains the IPVS load-balancer rules.
once the packet lands in ingress_sbox, it passes through its own iptables rules. This time, it goes through the mangle table, which (in the case of being targetted at port 8000), has the effect of adding a marker to the packet (fwmark).
landing in its destination (ipvs), the packet’s fwmark indicates the service that the packet needs to be load-balancer to. IPVS then picks one of the target real servers (using a given scheduling algorithm - roundrobin in this case) and then forwards it to the destination.
As the destination servers are in the ingress networks, the overlay setup takes care of making the packets reach the destination address accordingly.

Changing IPTables to block Ingress traffic

Having some idea of what goes on under the scenes, we can now jump into the process of putting barriers in the middle of these steps such that traffic can’t go through.

Looking at the last two images, we can imagine that we can either put a barrier between eth0 and ingress_sbox, or we can but a barrier between ingress_sbox and ipvs

If we perform the first block (cut traffic between eth0 and ingress_sbox), then we’re going to be blocking connections that are made from one machine to the physical interface of another, but we’d still allow traffic originating in the host itself.

Taking the second approach looks like a more protective block.

Let’s create some services and some networks and apply the block to see that in action.

# Create a network named `mynet` that uses the
# overlay network driver.
docker network create \
  --driver overlay \
  mynet


# Create two services in the same network:
#
# 1. nginx1 that specifies a public port mapping
#    8123 --> 80
#
# 2. nginx2 that specifies a public port mapping
#    8124 --> 80
docker service create \
  --detach \
  --no-resolve-image \
  --network mynet \
  --publish 8123:80 \
  --name nginx1 \
  nginx:alpine

docker service create \
  --detach \
  --no-resolve-image \
  --network mynet \
  --publish 8124:80 \
  --name nginx2 \
  nginx:alpine


# Create the `CURL_ARGS` variable that is going to
# hold the non-positional arguments that we'll be
# using throughout the example. 
#
# --connect-timeout: specifies the maximum amount of
#                    time that `connect(2)` should
#                    take. This is needed because we're
#                    simply dropping packets.
#
# --silent: removes the verbose curl information
#
# --output: allows us to specify a file that should
#           be used as the stdout for the body. 
#           Using /dev/null we simply discard it.
#
# --write-out: allows us to specify a template of text
#              containing information regarding the
#              connection / request. By specifying
#              %{http_code} we can display the HTTP status
#              code of the request (000) on connection
#              failures.
CURL_ARGS="--connect-timeout 0.5 \
--silent \
--write-out '%{http_code}' \
--output /dev/null"

# Verify that both of the services can be 
# reached from machine1 by targetting its
# own loopback interface.
curl $CURL_ARGS \
  localhost:8123
200
curl $CURL_ARGS \
  localhost:8124
200


# Get into machine2 and verify
# that it can indeed connect to the
# services by targetting the machine's
# interface (eth0).
ssh ubuntu@machine2
machine2 $ curl $CURL_ARGS \
  machine1:8123
200
machine2 $ curl $CURL_ARGS \
  machine2:8124
200


# Still in machine2, verify that the
# service is accessible in its own IPVS 
# load-balancer by trying to reach the 
# service making a request to its own loopback
# interface.
machine2 $ curl $CURL_ARGS \
  localhost:8123
200
machine2 $ curl $CURL_ARGS \
  localhost:8124
200


# Get back to machine1.
exit


# Get into the `ingress_sbox` network namespace.
#
# To do this we can either make use of `nsenter`
# like we did before or make use of `ip netns` which 
# assumes that the network namespace files live under
# /var/run/netns.
#
# As docker creates its network namespace files under
# /var/run/docker/netns, we can get over such default
# by creating a soft link.
ln -s /var/run/docker/netns /var/run/netns


# Given that Docker doesn't make use of the `filter`
# table and doesn't change it at all during service
# changes, we can include specific rules in specific chains
# there to drop desired connections.
#
# By including a rule in the INPUT table, we're able to
# filter packets that are destined to a local address that
# represents the current machine.
ip netns exec ingress_sbox \
  iptables -A INPUT -p tcp -m tcp --dport 8123 -j DROP


# Verify that the `nginx` (8123) can't be 
# accessed by touching `machine1` ports
curl $CURL_ARGS \
  localhost:8123
000

curl $CURL_ARGS \
  machine1:8123
000

# Verify that when targetting machine2, the service is 
# available:
curl $CURL_ARGS \
  machine2:8123
200

# Get into machine2 and verify that it can't access
# the service by targetting machine1
ssh ubuntu@machine2
machine2 $ curl $CURL_ARGS \
  machine1:8123
000

# But that it can access it via itself
machine2 $ curl $CURL_ARGS \
  localhost:8123
200


# Go back to machine1 (manager) and create a new 
# service and verify that docker didn't mess with 
# the `ingress_sbox` rules:
docker service create \
  --detach \
  --no-resolve-image \
  --network mynet \
  --publish 8125:80 \
  --name nginx3 \
  nginx:alpine


# Verify that it still fails (which is what we expect)
curl $CURL_ARGS \
  localhost:8123
000

Closing thoughts

It was very interesting to go through all of this as a way of learning how to set up a simple block for a given set of ports that wouldn’t mess with the default ingress load-balancing and other Docker-specific chains.

Please let me know if I got something wrong - I’m new to networking and going through this definitely helped me improve and I’d appreciate a lot to be corrected if I presented a misleading or completely wrong information here.

You can reach me at cirowrc on Twitter!

Have a good one!

finis

How to set up a Private Docker Registry using AWS S3

Ciro S. Costa — Sat, 23 Jun 2018 00:00:00 +0000

Hey,

Some time ago I needed to check whether AWS EC2 instance profiles would work fine with Docker Registry, and guess what? It does!

What interests me the most in such setup is how it facilitates a scenario where you can have a highly available internal registry, having a load-balancer in front of a set of registries and granting privileges to only those who really need it.

Given that the pushes to the registry (and pulls from it) usually involve large amounts of data being transferred, with a well-done load-balancing we can scale the throughput with ease.

Here’s a setup that makes use of terraform and the official Docker Registry image to achieve the first scenario.

The terraform setup

By taking a top-down approach, the infrastructure resources that Terraform is meant to create is composed of only two top-level resources:

the instance that will hold the docker registry; and
the S3 bucket that is meant to store the registry data.

The bucket can be created using the policy specified in the official registry documentation:

# Create a AWS S3 bucket that is encrypted by default
# at the server side using the name provided by the
# `bucket` variable.
#
# Given that we're not specifying an ACL, by default
# the `private` canned ACL is used, which means that
# only the owner gets FULL_CONTROL access (and no
# one else).
resource "aws_s3_bucket" "encrypted" {
  bucket        = "${var.bucket}"
  force_destroy = true

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

# Set up the bucket policy to allow only a 
# specific set of operations on both the root
# of the bucket as well as its subdirectories.
#
# Here we also explicitly set who's able to have
# such capabilities - those that make use of the
# role that we defined in `permissions.tf`. 
resource "aws_s3_bucket_policy" "main" {
  bucket = "${var.bucket}"

  policy = <<POLICY
{
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Effect": "Allow",
            "Principal": {
                "AWS": "${aws_iam_role.main.arn}"
            },
            "Resource": "arn:aws:s3:::${var.bucket}/*",
            "Sid": "AddCannedAcl"
        },
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Principal": {
                "AWS": "${aws_iam_role.main.arn}"
            },
            "Resource": "arn:aws:s3:::${var.bucket}"
        }
    ],
    "Version": "2012-10-17"
}
POLICY
}

Naturally, there’s one piece there in the policies that we didn’t create yet at this point - the role to which we grant access to.

This role is the one that we’re going to ship in the form of an instance profile to the EC2 instances such that they can have access to the S3 buckets.

# The bucket-root policy defines the API actions that are
# allowed to take place on the bucket root directory.
#
# Given that by default nothing is allowed, here we list
# the actions that are meant to be allowed.
#
# This list of actions that are required by the Docker Registry
# can be found in the official Docker Registry documentation.
data "aws_iam_policy_document" "bucket-root" {
  statement {
    effect = "Allow"

    actions = [
      "s3:ListBucket",
      "s3:GetBucketLocation",
      "s3:ListBucketMultipartUploads",
    ]

    resources = [
      "arn:aws:s3:::${var.bucket}",
    ]
  }
}

# The bucket-subdirs policy defines the API actions that
# are allowed to take place on the bucket subdirectories.
# 
# Given that by default nothing is allowed, here we list
# the actions that are meant to be allowed.

# Just like the policy for the root directory of the bucket,
# this list of necessary actions to be allowed can be found in 
# the official Docker Registry documentation.
data "aws_iam_policy_document" "bucket-subdirs" {
  statement {
    effect = "Allow"

    actions = [
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject",
      "s3:ListMultipartUploadParts",
      "s3:AbortMultipartUpload",
    ]

    resources = [
      "arn:aws:s3:::${var.bucket}/*",
    ]
  }
}

# Define a base policy document that dictates which principals
# the roles that we attach to this policy are meant to affect.
#
# Given that we want to grant the permissions to an EC2 instance,
# the pricipal is not an account, but a service: `ec2`.
#
# Here we allow the instance to use the AWS Security Token Service
# (STS) AssumeRole action as that's the action that's going to
# give the instance the temporary security credentials needed
# to sign the API requests made by that instance.
data "aws_iam_policy_document" "default" {
  statement {
    effect = "Allow"

    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type = "Service"

      identifiers = [
        "ec2.amazonaws.com",
      ]
    }
  }
}

# Creates an IAM role with the trust policy that enables the process
# of fetching temporary credentials from STS.
#
# By tying this trust policy with access policies, the resulting 
# temporary credentials generated by STS have only the permissions 
# allowed by the access policies.
resource "aws_iam_role" "main" {
  name               = "default"
  assume_role_policy = "${data.aws_iam_policy_document.default.json}"
}

# Attaches an access policy to that role.
resource "aws_iam_role_policy" "bucket-root" {
  name   = "bucket-root-s3"
  role   = "${aws_iam_role.main.name}"
  policy = "${data.aws_iam_policy_document.bucket-root.json}"
}

# Attaches an access policy to that role.
resource "aws_iam_role_policy" "bucket-subdirs" {
  name   = "bucket-subdirs-s3"
  role   = "${aws_iam_role.main.name}"
  policy = "${data.aws_iam_policy_document.bucket-subdirs.json}"
}

# Creates the instance profile that acts as a container for that
# role that we created that has the trust policy that is able
# to gather temporary credentials using STS and specifies the
# access policies to the bucket.
#
# This instance profile can then be provided to the aws_instance
# resource to have it at launch time.
resource "aws_iam_instance_profile" "main" {
  name = "instance-profile"
  role = "${aws_iam_role.main.name}"
}

With the permissions created, as well as the bucket, what’s left now is creating the instance with its corresponding networking and AMI:

provider "aws" {
  region  = "${var.region}"
  profile = "${var.profile}"
}

# We could create an extra VPC, properly set a subnet and
# have the whole thing configured (internet gateway, an updated
# routing table etc).
#
# However, given that this is only a sample, we can make use of
# the default VPC (assuming you didn't delete your default VPC
# in your region, you can too).
data "aws_vpc" "main" {
  default = true
}

# Provide the public key that we want in our instance so we can
# SSH into it using the other side (private) of it.
resource "aws_key_pair" "main" {
  key_name_prefix = "sample-key"
  public_key      = "${file("./keys/key.rsa.pub")}"
}

# Create a security group that allows anyone to access our
# instance's port 5000 (where the main registry functionality
# lives).
#
# Naturally, you'd not do this if you're deploying a private
# registry - something you could do is allow the internal cidr
# and not 0.0.0.0/0.
resource "aws_security_group" "allow-registry-ingress" {
  name = "allow-registry-ingress"

  description = "Allows ingress SSH traffic and egress to any address."
  vpc_id      = "${data.aws_vpc.main.id}"

  ingress {
    from_port   = 5000
    to_port     = 5000
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name = "allow_registry-ingress"
  }
}

# Allow SSHing into the instance
resource "aws_security_group" "allow-ssh-and-egress" {
  name = "allow-ssh-and-egress"

  description = "Allows ingress SSH traffic and egress to any address."
  vpc_id      = "${data.aws_vpc.main.id}"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name = "allow_ssh-and-egress"
  }
}

# Template the registry configuration with the desired
# bucket and region information.
#
# Note.: no credentials are needed given that the golang
# aws sdk is going to retrieve them from the instance
# role that we set to the machine.
data "template_file" "registry-config" {
  template = "${file("./registry.yml.tpl")}"

  vars {
    bucket = "${var.bucket}"
    region = "${var.region}"
  }
}

# Template the instance initialization script with information
# regarding the region and bucket that the user configured.
data "template_cloudinit_config" "init" {
  gzip          = true
  base64_encode = true

  part {
    content_type = "text/cloud-config"

    content = <<EOF
#cloud-config
write_files:
  - content: ${base64encode("${data.template_file.registry-config.rendered}")}
    encoding: b64
    owner: root:root
    path: /etc/registry.yml
    permissions: '0755'
EOF
  }

  part {
    content_type = "text/x-shellscript"
    content      = "${file("./instance-init.sh")}"
  }
}

# Create an instance in the default VPC with a specified
# SSH key so we can properly SSH into it to verify whether
# everything is worked as intended.
resource "aws_instance" "main" {
  instance_type        = "t2.micro"
  ami                  = "${data.aws_ami.ubuntu.id}"
  key_name             = "${aws_key_pair.main.id}"
  iam_instance_profile = "${aws_iam_instance_profile.main.name}"
  user_data            = "${data.template_cloudinit_config.init.rendered}"

  vpc_security_group_ids = [
    "${aws_security_group.allow-ssh-and-egress.id}",
    "${aws_security_group.allow-registry-ingress.id}",
  ]
}

output "public-ip" {
  description = "Public IP of the instance created"
  value       = "${aws_instance.main.public_ip}"
}

Now, aside from the Terraform setup that has already been done at this point, the next step is configuring the registry itself.

As we’re using cloud-init’s write_files to send our templated registry configuration, this is how such configuration looks like:

# -*- mode: yaml -*-
# vi: set ft=yaml :
# 
# Template of a registry configuration.
# 
# It expects the folling variables:
# - region: AWS region where the S3 bucket is deployed to; and
# - bucket: the name of the bucket to push registry data to.
#
# Without modifying the default command line arguments of the
# registry image, put this configuration under 
# `/etc/docker/registry/config.yml`.
version: 0.1
log:
  level: "debug"
  formatter: "json"
  fields:
    service: "registry"
storage:
  cache:
    blobdescriptor: "inmemory"
  s3:
    region: "${region}"
    bucket: "${bucket}"
http:
  addr: ":5000"
  secret: "a-secret"
  debug:
    addr: ":5001"
    # In the `master` branch there's already native support for
    # Prometheus metrics.
    # 
    # If you want to make use of this feature: 
    # 1. clone `github.com/docker/distribution`;
    # 2. build the image (`docker build -t registry .`);
    # 3. reference the recently built image in the initialization.
    prometheus:
      enabled: true
      path: "/metrics"
  headers:
    X-Content-Type-Options: [ "nosniff" ]
health:
  storagedriver:
    enabled: true
    interval: "10s"
    threshold: 3

Update: Note that there I specified a Prometheus endpoint - that’s because in a future release the Docker registry will be able to export metrics to prometheus; if you’re curious about it, checkout how to retrieve metrics using prometheus in which I tie these metrics with the newest Grafana Prometheus heatmap support that just landed in Grafana v5.1.

And that’s it! Create an instance initialization script and there you go! You have a registry backed by S3.

Such script doesn’t need to be complicated - something like the following would do it:

# Run docker registry with the configuration provided
# at `/etc/registry.yml` as a readonly volume mount
# to the registry container.
#
# Here I make use of the host networking mode (no extra
# network namespace is created), but you're free to make 
# use of port forwarding as you wish.
docker run \
        --detach \
        --volume /etc/registry.yml:/etc/docker/registry/config.yml:ro` \
        --network host \
        registry

Note that all the setup is independent of a particular container scheduler.

In the case of a scheduler like Kubernetes or Swarm you could provide the proper labels to the instance and make sure that the registry container gets scheduled to such instance type.

The whole code that’s been described here can be found in the cirocosta/sample-docker-registry-aws repository.

If you have any questions or found something odd, feel free to get in touch! I’m cirowrc on Twitter.

Have a good one!

finis

Retrieving Docker Registry metrics using Prometheus

Ciro S. Costa — Sat, 23 Jun 2018 00:00:00 +0000

Hey,

Today I was looking at the internal struct that ends up being filled as the result of parsing the Docker Registry configuration, and doing that I found that in the master branch of the repository there’s already support for metrics scraping by Prometheus (see configuration.go), something that used to be only available in OpenShift (see openshift/origin issue).

It surprised me that this addition came not super recently:

commit e3c37a46e2529305ad6f5648abd6ab68c777820a
Author: tifayuki <tifayuki@gmail.com>
Date:   Thu Nov 16 16:43:38 2017 -0800

    Add Prometheus Metrics
    
    at the first iteration, only the following metrics are collected:
    
      - HTTP metrics of each API endpoint
      - cache counter for request/hit/miss
      - histogram of storage actions, including:
        GetContent, PutContent, Stat, List, Move, and Delete
    
    Signed-off-by: tifayuki <tifayuki@gmail.com>

What about giving it a try?

So, first, start by building an image right from the master branch:

# Clone the Docker registry repository
git clone https://github.com/docker/distribution

# Build the registry image using the provided Dockerfile
# that lives right in the root of the project.
#
# Here I tag it as `local` just to make sure that we
# don't confuse it with `registry:latest`. 
docker build --tag registry:local .

With the image built using the latest code, tailor a configuration that enables the exporter:

version: 0.1
log:
  level: "debug"
  formatter: "json"
  fields:
    service: "registry"
storage:
  cache:
    blobdescriptor: "inmemory"
  filesystem:
    rootdirectory: "/var/lib/registry"
http:
  addr: ":5000"
  debug:
    addr: ":5001"
    prometheus:
      enabled: true
      path: "/metrics"
  headers:
    X-Content-Type-Options: [ "nosniff" ]

Run the registry with this configuration and note how :5001/metrics will provide you with the metrics you expect.

# Make a request to the metrics endpoint and filter the
# output so we can see what the metrics descriptions are.
#
# ps.: each of those metrics end up expanding to multiple
# dimensions via labels.
curl localhost:5001/metrics --silent | ag registry_ | ag HELP

HELP registry_http_in_flight_requests The in-flight HTTP requests
HELP registry_http_request_duration_seconds The HTTP request latencies in seconds.
HELP registry_http_request_size_bytes The HTTP request sizes in bytes.
HELP registry_http_requests_total Total number of HTTP requests made.
HELP registry_http_response_size_bytes The HTTP response sizes in bytes.
HELP registry_storage_action_seconds The number of seconds that the storage action takes
HELP registry_storage_cache_total The number of cache request received

Wanting to see how useful these metrics can be, I set up an environment in AWS where there’s an EC2 instance with a registry instance that’s backed by an S3 bucket as the storage tier (you can check more about how to achieve it here: How to set up a private docker registry using AWS S3).

With the registry up, it was now a matter of having Prometheus and Grafana running locally so I could start making some queries:

version: '3.3'
services:

  # Create a "pod-like" container that will serve as both the
  # network entrypoint for both of the containers as well as
  # provide a common ground for them to communicate over localhost
  # (given that they'll share the same network namespace).
  pod:
    container_name: 'pod'
    ports:
      - '9090:9090'
      - '3000:3000'
    image: 'alpine'
    tty: true

  grafana:
    container_name: 'grafana'
    depends_on: [ 'pod' ]
    network_mode: 'service:pod'
    image: 'grafana/grafana:5.2.0-beta3'
    restart: 'always'

  prometheus:
    container_name: 'prometheus'
    depends_on: [ 'pod' ]
    network_mode: 'service:pod'
    image: 'prom/prometheus'
    restart: 'always'
    volumes: 
      - './prometheus.yml:/etc/prometheus/prometheus.yml'

Given that prometheus is making use of a local configuration under ./prometheus.yml, this one looked like the following:

global:
  scrape_interval: '15s'
  evaluation_interval: '15s'
  scrape_timeout: '10s'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: [ 'localhost:9090' ]

  - job_name: 'registry'
    # Here I just specified the public IP address of the
    # EC2 instance that I had holding the registry at that
    # time. 
    #
    # Naturally, you wouldn't keep it open like this.
     static_configs:
      - targets: [ '52.67.104.105:5001' ]

Now, to reproduce the panel showed in the beginning, head over to our grafana dashboard and create a heatmap panel with the following query:

rate(registry_http_request_duration_seconds_bucket{handler="blob_upload"}[10m])

That’s it!

If you have any questions or found something odd, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis

Navigating the Linux Kernel source tree with YouCompleteMe

Ciro S. Costa — Thu, 21 Jun 2018 00:00:00 +0000

Hey,

These days I’ve been doing some research that involves looking at the Linux kernel to figure things out.

Since I’ve been using YouCompleteMe for a reasonable amount of time, and that I love how simple it is to configure it, why not give a try to using it for inspecting the Linux kernel?

Example of YouCompleteMe searching available functions.

ps.: for a browser experience, bootlin is AMAZING. It lets you navigate through the code providing links to definitions and making you go back in time to previous versions of the source - it’s great! Here I focus on navigating the Kernel source using VIM and a terminal though.

Check out how to tailor a YouCompleteMe configuration that suits the Linux kernel source code.

Getting YouCompleteMe ready

The first thing to do is getting YCM ready.

As I’m constantly doing some experiments in AWS and from time to time running a VM here and there, I automated the process using Ansible and Packer (with Ansible I describe how to get to the desired state, with Packer I build a base image) - check out cirocosta/mylinux.

Below is how the YouCompleteMe part of it looks like:

- name: 'install ycm'
  command: './install.py --go-completer --clang-completer'
  environment: 
    PATH: "{{ ansible_env.PATH }}:/usr/local/go/bin"
  args:
    chdir: '{{ user_home }}/.vim/bundle/YouCompleteMe'

ps.: You might notice that I also specify --go-completer there. That’s not needed for C code (like Linux), so you can get rid of that.

pps.: Check out my Vim Ansible role to see the full installation of my Vim setup.

As I use pathogen to provide Vim plugins to my environment, there’s just the need to clone the repository to .vim/bundle (with submodules) and then running the installation script.

Setting up the YouCompleteMe configuration

Given that the Kernel does not make use of CMake or Ninja, we can’t make use of the compilation database that CMake would create (a database that tells which commands were used to compile a given file).

Update (30 Jun, 2018): it turns out that you can get a full fidelity compilation database even without cmake - the bear utility takes the approach of intercepting the build calls, gather the relevant info and generate a complete compilation database. Having such database ready, you can either make full use of it and ignore an extra .ycm_extra_conf.py or gather flags you find useful from there and tailor your special config. Thanks lanzaio!

However, we can still have such mapping by providing a .ycm_extra_conf.py file that does the job of delivering the compilation flags for any given file. The whole point of this python file is to make the project owner implement the YCM interface for detecting flags for files:

Given a source file, give me its compilation flags.

As it’s common in C project to have a “catch all” target in a Makefile (like %.o: %.c) that provides the same flags to every file, we can have a pretty simplified .ycm_extra_conf.py file:

def FlagsForFile(filename, **kwargs):
  return { 'flags': ['-I/usr/include', '-std=gnu89', '-xc'] }

For Linux, that’s not very different - gather the flags used, supply them to FlagsForFile and we’re done.

The necessary flags can be gathered by exposing two variables that get set in the Makefile: LINUXINCLUDE and USERINCLUDE. That’s because, for just autocompletion and source code navigation, includes and definitions are sufficient.

To dump those variables, create an extra target in ./Makefile:

diff --git a/Makefile b/Makefile
index 019a5a02..88e1ad10 100644
--- a/Makefile
+++ b/Makefile
@@ -1792,6 +1792,10 @@ endif    # skip-makefile
 PHONY += FORCE
 FORCE:
 
+show-includes:
+       @$(foreach include, $(LINUXINCLUDE), echo $(include);)
+       @$(foreach include, $(USERINCLUDE), echo $(include);)
+
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a v

Then prepare the .config file that would be necessary for the tiniest possible kernel:

make tinyconfig

With the configuration set, now, executing make specifying the show-includes target dumps the variables we need:

# Piping it to `sort -u` to remove the duplicates
# and keep the list sorted.
make show-includes | sort -u
-I./arch/x86/include
-I./arch/x86/include/generated
-I./arch/x86/include/generated/uapi
-I./arch/x86/include/uapi
-I./include
-I./include/generated/uapi
-I./include/uapi
-include
./include/linux/kconfig.h

As the next step is to supply those variables to YouCompleteMe, we need first to realize that the daemon (ycmd) will not deal well with these relative paths.

In our Python script we can deal with that though:

import os

CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))

include_dirs = [
    './arch/x86/include',
    './arch/x86/include/generated',
    './arch/x86/include/generated/uapi',
    './arch/x86/include/uapi',
    './include',
    './include/generated/uapi',
    './include/uapi',
]


include_files = [
    './include/linux/kconfig.h',
]

flags = [
    '-D__KERNEL__',
    '-std=gnu89',
    '-xc',
    '-nostdinc',
]

def FlagsForFile( filename, **kwargs ):
    """
    Given a source file, retrieves the flags necessary for compiling it.
    """
    for dir in include_dirs:
        flags.append('-I' + os.path.join(CURRENT_DIR, dir))

    for file in include_files:
        flags.append('-include' + os.path.join(CURRENT_DIR, file))

    return { 'flags': flags }

Now, open a file (like, ./net/socket.c) and check the autocompletion working! You can also check how “jump to” declarations and definitions also work.

Example of YouCompleteMe autocompleting a kernel struct.

If you have any questions or found any error, please let me know! I’m cirowrc on Twitter.

Have a good one!

A minimal TCP Client in C

Ciro S. Costa — Wed, 20 Jun 2018 00:00:00 +0000

Hey,

This week I needed to check whether in a given situation a given error would occur when connecting to a TCP server, so why not go back to the man pages and review the steps?

Here’s what I come up with! I hope it’s going to be useful for you who’s reading.

/**
 * Start by specifying our dependencies.
 *
 * As we're using only standard dependencies you'd find in a
 * Linux machine, these can be found under `/usr/include`.
 *
 * - `stdio.h` gives us the standard io (input and output) methods
 *   like `fprintf` that allows us to format a given string and
 *   write it to a specific file descriptor (usually stderr or
 *   stdout).
 *
 * - `unistd.h` defines constants, types and methods that provides
 *   access to the POSIX API. Here we make use of the `close(2)`
 *   syscall definition that can be found there.
 *
 * - `arpa/inet` provides definitions for internet operations, most
 *   notably, it gives us ways of converting addresses from string
 *   formats (presentation) to numeric ones (and vice-versa).
 *
 *   Given that `arpa/inet` includes `netinet/in.h`, it also provides
 *   us with the definitions of `sockaddr_in`, the structure used
 *   to define internet addresses.
 *
 * - `sys/socket.h` provides us the `socket(2)` syscall and its
 *   necessary definitions.
 */
#include <arpa/inet.h>
#include <stdio.h>
#include <sys/socket.h>
#include <unistd.h>

/**
 * ADDRESS defines the IP of the server that we should
 * connect to.
 *
 * In this exampe, it must be a standard IPV4 address with
 * four octects representing the IP in a string format -
 * no names are allowed given that we don't perform name
 * resolution.
 *
 * Using GCC, this value can be specified by using the
 * -D arguments with the address itself having escaped
 *  quotes (for instance: -DADDRESS=\"127.0.0.1\"). Without
 *  the escaping, `127.0.0.1` would be supplied instead of
 *  `"127.0.0.1"`, causing errors.
 */
#ifndef ADDRESS
#define ADDRESS "127.0.0.1"
#endif

/**
 * PORT represents the server port we want to connect to.
 *
 * Naturally, it must live in the range of [1,65k).
 */
#ifndef PORT
#define PORT 8000
#endif

int
main()
{
	int                ret = 0;
	int                conn_fd;
	struct sockaddr_in server_addr = { 0 };

	// Specifiy the communication domain in which the address lives:
	// AF_INET      - IPV4;
	// AF_INET6     - IPV6;
	// AF_UNIX      - Local communication (used w/ unix sockets).
	server_addr.sin_family = AF_INET;

	// Assign the port that we supplied in host byte order in the
	// form of network byte order (`htons` performs the conversion).
	//
	// On amd64 (x86-64), the byte ordering of the host is little endian
	// (o.e., the least significant byte comes first).
	//
	// However, when it comes to networking, the byte ordering is big
	// endian - the most significant byte comes first.
	//
	// This means that a value (like 8000) defined in our machine needs
	// to be properly translated to the network byte order before being
	// sent.
	server_addr.sin_port = htons(PORT);

	// Fill the destination address with a 4-byte (32bit) unsigned integer
	// in network byte order after converting the address from the string
	// representation (e.g., "127.0.0.1").
	//
	// Given that we might see a failure in the conversion (e.g., if we
	// supply an invalid server_addr pointer or an invalid string), check
	// for these errors.
	//
	// Note that differently from the majority of syscalls and glibc
	// methods, success is defined with `1`, and not `0`. The information of
	// what represents success can often be found in man pages - e.g., `man
	// inet_pton`.
	ret = inet_pton(AF_INET, ADDRESS, &server_addr.sin_addr);
	if (ret != 1) {
		if (ret == -1) {
			perror("inet_pton");
		}
		fprintf(stderr,
		        "failed to convert address %s "
		        "to binary net address\n",
		        ADDRESS);
		return -1;
	}

	fprintf(stdout, "CONNECTING: address=%s port=%d\n", ADDRESS, PORT);

	// Create an endpoint for communication in our machine (our side) that
	// is meant to communicate over the AF_INET (IPv4) domain, using
	// SOCK_STREAM semantics (sequenced, reliable, two-way, connection-base
	// byte streams - TCP, for instance).
	//
	// Note that at this point, no communication has been made to an
	// external server yet - this operation is entirely local.
	conn_fd = socket(AF_INET, SOCK_STREAM, 0);
	if (conn_fd == -1) {
		perror("socket");
		return -1;
	}

	// Connect our local endpoint (represented by the socket file
	// descriptor) to the address specified by `server_addr`.
	//
	// On a TCP connection, `connect` passes to the kernel the
	// responsability of perming the TCP handshake, blocking the call while
	// the kernel goes forward (when nonblocking flag - SOCK_NONBLOCK - is
	// not set in the socket).
	//
	// Once the handshake has been succesful, on the server side the
	// connection is then put into a queue so that the server can
	// `accept(2)` on a passive socket to make use of such established
	// connection.
	ret =
	  connect(conn_fd, (struct sockaddr*)&server_addr, sizeof(server_addr));
	if (ret == -1) {
		perror("connect");
		return -1;
	}

	// After the connection has been properly established, we could go
	// forward with `read(2)` and `write(2)` calls.
	//
	// As in this example we don't want to read or write from it, we can
	// proceed with a `shutdown(2)`, which takes care of terminating our
	// side of the channel.
	//
	// By specifying `SHUT_RDWR`, not only furtes receptions are
	// dissallowed, but transmissions to.
	ret = shutdown(conn_fd, SHUT_RDWR);
	if (ret == -1) {
		perror("shutdown");
		return -1;
	}

	// Once the connection got properly terminated, now we can proceed with
	// actually closing the file descriptor
	ret = close(conn_fd);
	if (ret == -1) {
		perror("close");
		return -1;
	}

	return 0;
}

If you spot anything weird, just let me know! I’m cirowrc on Twitter.

Have a good one!

Using network namespaces and a virtual switch to isolate servers

Ciro S. Costa — Tue, 12 Jun 2018 00:00:00 +0000

Hey,

I’ve been working on an tc + ebpf-based load-balancer that I’ll soon talk about here in this blog, and one of the things I wanted to do was test such load-balancing feature.

In my line of thought, I wondered: if I’m going to redirect the traffic by changing the destination address of the packets, that means that I’ll need somehow to have a different internet set up in the machine where I could put processes listening on those addresses.

Having worked with Docker and implemented a tiny container runtime myself, it seemed clear that going with network namespaces, virtual interfaces and bridging would get the job done.

How these two relate to each other become clearer after looking at what jobs they do:

network namespaces, according to man 7 network_namespaces:

Network namespaces provide isolation of the system resources associated with networking: network devices, IPv4 and IPv6 protocol stacks, IP routing tables, firewall rules, the /proc/net directory, the /sys/class/net directory, various files under /proc/sys/net, port numbers (sockets), and so on.

virtual interfaces provide us with virtualized representations of physical network interfaces; and
the bridge gives us the virtual equivalent of a switch.

That said, we can combine these three virtual components and create a virtual network inside the host without the need of VMs. Very lightweight (and with a single dependency - iproute2).

Setting up

Assuming we want to simulate load-balancing across two different servers, that would mean that we’d set up the machine to have:

two network namespace (in this case, we can think of each network namespace as a different computer);
two veth pairs (we can think of each pair as two ethernet cards with a cable between them); and
a bridge device that provides the routing to these two namespaces (we can think of this device as a switch).

First, let’s start with the namespaces:

# Making use of the `ip` command from the `iproute2` 
# package we're able to create the namespaces.
#
# By convention, network namespace handles created by
# iproute2 live under `/var/run/netns` (although they
# could live somewhere, like `docker` does with its
# namespaces - /var/run/docker/netns`).
ip netns add namespace1
ip netns add namespace2

# Check that iproute2 indeed creates the files
# under `/var/run/netns`.
tree /var/run/netns/
/var/run/netns/
├── namespace1
└── namespace2

Once the namespaces have been set up, we can confirm the isolation by taking advantage of ip netns exec and executing some commands there:

# List all the interfaces with their corresponding
# configurations.
#
# We can verify that inside the namespace, only the
# loopback interface has been set.
ip netns exec namespace1 \
        ip address show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

At this point, we can start creating the veth pairs and associating one of their sides to their respective namespaces.

# Create the two pairs.
ip link add veth1 type veth peer name br-veth1
ip link add veth2 type veth peer name br-veth2

# Associate the non `br-` side
# with the corresponding namespace
ip link set veth1 netns namespace1
ip link set veth2 netns namespace2

These pairs act as tunnels between the namespaces (for instance, namespace1 and the default namespace).

Now that the namespaces have an additional interface, check out that they’re actually there:

# Differently from before, now we see the
# extra interface (veth1) that we just added.
ip netns exec namespace1 \
        ip address show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
30: veth1@if29: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:92:25:3c:79:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0

As we can see, the veth interface has no IPV4 address associated with it.

We can do so by making use of ip addr add from within the corresponding network namespaces:

# Assign the address 192.168.1.11 with netmask 255.255.255.0
# (see the `/24` mask there) to `veth1`.
ip netns exec namespace1 \
        ip addr add 192.168.1.11/24 dev veth1


# Verify that the ip address has been set.
ip netns exec namespace1 \
        ip address show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
30: veth1@if29: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:92:25:3c:79:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.2/24 scope global veth1
       valid_lft forever preferred_lft forever


# Repeat the process, assigning the address 192.168.1.12 with 
# netmask 255.255.255.0 to `veth2`.
ip netns exec namespace2 \
        ip addr add 192.168.1.12/24 dev veth2

Although we have both IPs and interfaces set, we can’t establish communication with them.

That’s because there’s no interface in the default namespace that can send the traffic to those namespaces - we didn’t either configure addresses to the other side of the veth pairs or configured a bridge device.

With the creation of the bridge device, we’re then able to provide the necessary routing, properly forming the network:

# Create the bridge device naming it `br1`
# and set it up:
ip link add name br1 type bridge
ip link set br1 up

# Check that the device has been created.
ip link | grep br1
33: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

With the bridge created, now it’s time to connect the bridge-side of the veth pair to the bridge device:

# Set the bridge veths from the default
# namespace up.
ip link set br-veth1 up
ip link set br-veth2 up

# Set the veths from the namespaces up too.
ip netns exec namespace1 \
        ip link set veth1 up
ip netns exec namespace2 \
        ip link set veth2 up

# Add the br-veth* interfaces to the bridge
# by setting the bridge device as their master.
ip link set br-veth1 master br1
ip link set br-veth2 master br1

# Check that the bridge is the master of the two
# interfaces that we set (i.e., that the two interfaces
# have been added to it).
bridge link show br1
29: br-veth1@if30: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br1 state disabled priority 32 cost 2
31: br-veth2@if32: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br1 state disabled priority 32 cost 2

Now, it’s a matter of giving this bridge device an address so that we can target such IP in our machine’s routing table making it a target for connections to those interfaces that we added to it:

# Set the address of the `br1` interface (bridge device)
# to 192.168.1.10/24 and also set the broadcast address
# to 192.168.1.255 (the `+` symbol sets  the host bits to
# 255).
ip addr add 192.168.1.10/24 brd + dev br1

Checking our routing table (from the default namespace), we can see that any requests with a destination to our namespaces (192.168.1.0/24) go through our bridge device:

default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.10 # <<<<<<<<<

We can verify that we indeed have connectivity:

# Check the connectivity from the default namespace (host)
ping 192.168.1.12
PING 192.168.1.12 (192.168.1.12) 56(84) bytes of data.
64 bytes from 192.168.1.12: icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from 192.168.1.12: icmp_seq=2 ttl=64 time=0.061 ms

# We can also reach the interface of the other namespace
# given that we have a route to it.
ip netns exec namespace1 \
        ip route
192.168.1.0/24 dev veth1 proto kernel scope link src 192.168.1.11

# Let's reach the other then iface then.
ip netns exec namespace1 \
        ping 192.168.1.12

Given that the routing table from namespace1 doesn’t have a default gateway, it can’t reach any other machine from outside the 192.168.1.0/24 range.

# Try to reach Google's DNS servers (8.8.8.8).
#
# Given that there's no route for something that doesn't 
# match the `192.168.1.0/24` range, 8.8.8.8 should be unreachable.
ip netns exec namespace1 \
        ping 8.8.8.8
connect: Network is unreachable

To fix that, the first step is giving the namespaces a default gateway route:

# Execute the command to add the default gateway in all
# the network namespaces under `/var/run/netns`.
#
# The command is going to add a default gateway which should
# be used for all connections that doesn't match the other
# specific routes. 
#
# 192.168.1.10 corresponds to the address assigned to the
# bridge device - reachable from both namespaces, as well as
# the host machine.
ip -all netns exec \
        ip route add default via 192.168.1.10

# Check how the routing table looks inside the namespace
ip netns exec namespace1 \
        ip route
default via 192.168.1.10 dev veth1
192.168.1.0/24 dev veth1 proto kernel scope link src 192.168.1.11

Show we be able to reach the internet now? Not yet.

# Try to reach Google's DNS servers (8.8.8.8).
ip netns exec namespace1 \
        ping 8.8.8.8
        PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

Although the network is now reachable, there’s no way that we can have responses back - packets from external networks can’t be sent directly to our 192.168.1.0/24 network.

To get around that, we can make use of NAT (network address translation) by placing an iptables rule in the POSTROUTING chain of the nat table:

# -t specifies the table to which the commands
# should be directed to. By default it's `filter`.
#
# -A specifies that we're appending a rule to the
# chain the we tell the name after it;
#
# -s specifies a source address (with a mask in 
# this case).
#
# -j specifies the target to jump to (what action to
# take).
iptables \
        -t nat \
        -A POSTROUTING \
        -s 192.168.1.0/24 \
        -j MASQUERADE

Although NAT is set to work for the packets originating from 192.168.1.0/24, there’s still (yet another) one more configuration to do: enable packet forwarding (maybe this is already active in your case):

# Enable ipv4 ip forwarding
sysctl -w net.ipv4.ip_forward=1

# Send some packets to 8.8.8.8
ip netns exec namespace1 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=43 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=61 time=35 ms

Now (finally), we’re good! We have connectivity all the way:

the host can direct traffic to an application inside a namespace;
an application inside a namespace can direct traffic to an application in the host;
an application inside a namespace can direct traffic to another application in another namespace; and
an application inside a namespace can access the internet.

Closing thoughts

Did we need bridge at all? That depends.

If the intention was to have the communication going through the host to a namespace (or vice-versa) directly, just setting the pairs would be enough.

Just in case you spot any mistake, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis

Adding privileged containers to Docker Swarm mode

Ciro S. Costa — Tue, 05 Jun 2018 00:00:00 +0000

Hey,

There’s been some time since an issue has been open under SwarmKit to address the lack of support to having privileged containers (see https://github.com/docker/swarmkit/issues/1030).

Although pull requests have been made (see https://github.com/docker/swarmkit/pull/1129, for instance), it seems like there’s no intention by the Docker team to have the feature added before entitlements get into moby (see https://github.com/moby/moby/issues/32801).

Knowing those facts, and given that I’ve forked docker before (I blogged before on how to compile and run your own docker fork), why not give a try forking it and add support to privileged containers?

Here’s how the journey looked like.

Let’s get started then!

1. The idea

The whole point of the modifications is as outlined in the swarmkit pull request:

“When create some container, we need privileged=True. But swarmkit does not support this.”

https://github.com/docker/swarmkit/issues/1030#issue-161106957

That is, the --privileged flag that exists in a regular docker run does not exist in docker service create (and the subsequent API calls used by the CLI).

Having that enables us to have any interesting functionality that requires more privileges than the default ones chosen by the Docker contributors. For instance, with --privileged you can:

run docker in docker (dind);
run an NFS server;
create special devices; and
more.

As shown in the picture above, there are many places where our privileged annotation has to go on.

Let’s put them in place.

2. Modifying the CLI

Not wanting to interact with the new feature using the remote api directly (using curl, for instance), I decided to start messing with the docker/cli repository.

Here is where we’re able to expose the --privileged flag to users of the docker client. For this part, check out the privileged branch from my fork: https://github.com/cirocosta/docker-cli/tree/privileged

For any Go developer who has already developed an application with flag parsing and all that stuff, the first thing to search for is the place where the definition of the flags to be parsed live.

Given that the docker client has multiple subcommands, the organization in the repository follows the same pattern:

tree ./cli/command -L 1
./cli/command
|-- bundlefile
|-- checkpoint
|-- image
|-- inspect
...
|-- network
|-- node
...
|-- service     # <<<<
|-- stack
...
`-- volume

Getting there under service, there’s an opts.go file with all the options that the subcommands of service needs. There you go:

diff --git a/cli/command/service/opts.go b/cli/command/service/opts.go
index 6d427451..b61b6c0f 100644
--- a/cli/command/service/opts.go
+++ b/cli/command/service/opts.go
@@ -483,6 +483,7 @@ type serviceOptions struct {
        stopSignal      string
+       privileged      bool
        mounts          opts.MountOpt
        dns             opts.ListOpts
        dnsSearch       opts.ListOpts
@@ -622,6 +623,7 @@ func (options *serviceOptions) ToService(ctx context.Context, apiClient client.N
                                Groups:     options.groups.GetAll(),
                                StopSignal: options.stopSignal,
                                TTY:        options.tty,
+                               Privileged: options.privileged,
                                ReadOnly:   options.readOnly,
                                Mounts:     options.mounts.Value(),
                                DNSConfig: &swarm.DNSConfig{
@@ -795,6 +797,8 @@ func addServiceFlags(flags *pflag.FlagSet, opts *serviceOptions, defaultFlagValu
+       flags.BoolVar(&opts.privileged, flagPrivileged, false, "Give extended privileges to the service")
+       flags.SetAnnotation(flagPrivileged, "version", []string{"1.35"})
        flags.BoolVarP(&opts.tty, flagTTY, "t", false, "Allocate a pseudo-TTY")

@@ -878,6 +882,7 @@ const (
+       flagPrivileged              = "privileged"
        flagUpdateDelay             = "update-delay"

If we were able to compile cli, then voilà, we’d have --privileged setting a Privileged boolean to true in the service spec.

Why wouldn’t it compile yet? Because the service spec doesn’t have such field! Time to modify SwarmKit.

3. Modifying SwarmKit

Looking at api/specs.proto, we can see the definition of the messages that go through the SwarmKit API.

There we can find one type of message that carries the type of information that fits our purpose: ContainerSpec.

“Container specifies runtime parameters for a container.”

see swarmkit#api/specs.proto line 164

That could only mean one thing - we had to proceed to add a “privileged” to this struct.

Being a property of a protobuf message, that also meant that I’d need to regenerate the Golang definitions that are made up from the protobuf specification. That is, before proceeding, I’d need to grab the protobuf compiler tooling ready.

The installation goes like this:

grab a protoc (protobuf compiler) release; and
install the binary and the includes.

So, go to the protobuf releases page and grab the x86-64 (amd64) version.

That’s a zip that comes with the following:

Archive:  ./protoc-3.5.1-linux-x86_64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-12-22 19:21   include/
        0  2017-12-22 19:21   include/google/
        0  2017-12-22 19:21   include/google/protobuf/
     3781  2017-12-22 19:21   include/google/protobuf/struct.proto
     6283  2017-12-22 19:21   include/google/protobuf/type.proto
    36277  2017-12-22 19:21   include/google/protobuf/descriptor.proto
     7734  2017-12-22 19:21   include/google/protobuf/api.proto
     2422  2017-12-22 19:21   include/google/protobuf/empty.proto
        0  2017-12-22 19:21   include/google/protobuf/compiler/
     8200  2017-12-22 19:21   include/google/protobuf/compiler/plugin.proto
     5483  2017-12-22 19:21   include/google/protobuf/any.proto
     8196  2017-12-22 19:21   include/google/protobuf/field_mask.proto
     3745  2017-12-22 19:21   include/google/protobuf/wrappers.proto
     5975  2017-12-22 19:21   include/google/protobuf/timestamp.proto
     4890  2017-12-22 19:21   include/google/protobuf/duration.proto
     2352  2017-12-22 19:21   include/google/protobuf/source_context.proto
        0  2017-12-22 19:21   bin/
  4433736  2017-12-21 19:11   bin/protoc
      715  2017-12-22 19:21   readme.txt
---------                     -------
  4529789                     19 files

The /bin/protoc is the binary that we need to have in our $PATH, while those protobuf definitions under include must be placed in our default include path (/usr/local/include).

In summary, here’s what you can do:

# Grab the release zip
curl \
        -SL \
        -o protoc.zip \
        https://github.com/google/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip

# Unzip it
unzip ./protoc.zip

# Move the correpsonding files
sudo mv ./bin/protoc /usr/local/bin
sudo mv ./include/google /usr/local/include/google

With that, we have protobuf set for generating swarmkit protobuf files.

note.: we don’t need to install github.com/golang/protobuf/protoc-gen-go given that swarmkit uses protoc-gen-gogoswarm.

With that done, we’re not able to apply the changes to our local clone of swarmkit:

--- a/agent/exec/dockerapi/container.go
+++ b/agent/exec/dockerapi/container.go
@@ -209,6 +209,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
                PortBindings: c.portBindings(),
                Init:         c.init(),
                Isolation:    c.isolation(),
+               Privileged:   c.spec().Privileged,
        }
 
        // The format of extra hosts on swarmkit is specified in:


diff --git a/cmd/swarmctl/service/flagparser/container.go b/cmd/swarmctl/service/flagparser/container.go
index 1507a168..37f11dd1 100644
--- a/cmd/swarmctl/service/flagparser/container.go
+++ b/cmd/swarmctl/service/flagparser/container.go
@@ -15,6 +15,15 @@ func parseContainer(flags *pflag.FlagSet, spec *api.ServiceSpec) error {
                spec.Task.GetContainer().Image = image
        }
 
+       if flags.Changed("privileged") {
+               privileged, err := flags.GetBool("privileged")
+               if err != nil {
+                       return err
+               }
+
+               spec.Task.GetContainer().Privileged = privileged
+       }
+
        if flags.Changed("hostname") {
                hostname, err := flags.GetString("hostname")
                if err != nil {

diff --git a/api/specs.proto b/api/specs.proto
index 14448d04..a5945ec3 100644
--- a/api/specs.proto
+++ b/api/specs.proto
@@ -318,6 +318,9 @@ message ContainerSpec {
        // PidsLimit prevents from OS resource damage by applications inside the container 
        // using fork bomb attack.
        int64 pidsLimit = 25;
+
+       // Privileged gives extended privileges to the container
+       bool privileged = 26;
 }

protobuf definition updated, now generate the golang protobuf files using swarmkit’s makefile target:

# Install extra dependencies that might be needed.
make setup

# Generate the protobuf golang files.
make generate

# Build the swarkit binaries
make binaries

With everything getting properly compiled, that means that we should now proceed to the next step - incorporating the setting of privileged flag to the ContainerSpec message whenever dockerd receives a Privileged flag in its service spec.

4. Modifying the Docker Daemon

Having a local fork, incorporating the modified swarmkit into the fork doesn’t take much more than setting up the vendoring tool and changing a file here and there.

As docker makes use of vndr to vendor its dependencies, we can modify the vndr configuration pointing swarmkit at our forked repository instead of the regular docker/swarmkit one and have it update the vendor directory with it:

# Get the current commit 
swarmkit $ FORKED_SWARMKIT_COMMIT=$(git rev-parse HEAD)
5598a39a89bd7482864c6279ae5544d9e8aa5299

# Go to our forked moby/moby repository
swarmkit $ cd ../docker

# Set a variable that defines the repository for `vndr`
# to use to gather our forked source code.
FORKED_SWARMKIT_REPO=https://github.com/cirocosta/swarmkit

# Prepare the variables to be used in the SED replacement
SED_FROM="\(github.com/docker/swarmkit\) \([a-zA-Z0-9]*\)"
SED_TO="\1 $FORKED_SWARMKIT_COMMIT $FORKED_SWARMKIT_REPO"

# Perform the replacement in `vendor.conf`.
# ps.: you could just grab an editor of your choice
# (like vim) and do the changes there.
#
# Here I'm using `sed` just to make the changes
# reproducible by copying and pasting.
docker $ sed \
  --in-place \
  "s#$SED_FROM#$SED_TO#g" \
  ./vendor.conf

# Update the dependencies
vndr

With the dependencies updated (i.e., with our swarmkit code in!), build the binaries (they will now include our swarmkit code):

# Run the `binary` target specified in ./Makefile
make binary

This will make use of a local instance of Docker that you must have running and then generate the daemon binaries at ./bundle:

./bundles/
├── binary-daemon
│   ├── docker-containerd
│   ├── docker-containerd-ctr
│   ├── docker-containerd-ctr.md5
│   ├── docker-containerd-ctr.sha256
│   ├── docker-containerd.md5
│   ├── docker-containerd.sha256
│   ├── docker-containerd-shim
│   ├── docker-containerd-shim.md5
│   ├── docker-containerd-shim.sha256
│   ├── dockerd -> dockerd-dev
│   ├── dockerd-dev
│   ├── dockerd-dev.md5
│   ├── dockerd-dev.sha256
│   ├── docker-init
│   ├── docker-init.md5
│   ├── docker-init.sha256
│   ├── docker-proxy
│   ├── docker-proxy.md5
│   ├── docker-proxy.sha256
│   ├── docker-runc
│   ├── docker-runc.md5
│   └── docker-runc.sha256
└── latest -> .

Naturally, without any changes to docker/docker itself, no flags would be passed to the recently modified SwarmKit ContainerSpec struct.

diff --git a/api/types/swarm/container.go b/api/types/swarm/container.go
index 0041653c9..c16c55fd8 100644
--- a/api/types/swarm/container.go
+++ b/api/types/swarm/container.go
@@ -57,6 +57,7 @@ type ContainerSpec struct {
        Privileges      *Privileges             `json:",omitempty"`
        StopSignal      string                  `json:",omitempty"`
        TTY             bool                    `json:",omitempty"`
+       Privileged      bool                    `json:",omitempty"`
        OpenStdin       bool                    `json:",omitempty"`
        ReadOnly        bool                    `json:",omitempty"`
        Mounts          []mount.Mount           `json:",omitempty"`
diff --git a/daemon/cluster/convert/container.go b/daemon/cluster/convert/container.go
index 0a34fc73e..9eb1feb4e 100644
--- a/daemon/cluster/convert/container.go
+++ b/daemon/cluster/convert/container.go
@@ -17,6 +17,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
        if c == nil {
                return nil
        }
+
        containerSpec := &types.ContainerSpec{
                Image:      c.Image,
                Labels:     c.Labels,
@@ -32,6 +33,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
                OpenStdin:  c.OpenStdin,
                ReadOnly:   c.ReadOnly,
                Hosts:      c.Hosts,
+    Privileged: c.Privileged,
                Secrets:    secretReferencesFromGRPC(c.Secrets),
                Configs:    configReferencesFromGRPC(c.Configs),
                Isolation:  IsolationFromGRPC(c.Isolation),
@@ -228,6 +230,7 @@ func containerToGRPC(c *types.ContainerSpec) (*swarmapi.ContainerSpec, error) {
                Groups:     c.Groups,
                StopSignal: c.StopSignal,
                TTY:        c.TTY,
+    Privileged: c.Privileged,
                OpenStdin:  c.OpenStdin,
                ReadOnly:   c.ReadOnly,
                Hosts:      c.Hosts,
diff --git a/daemon/cluster/executor/container/container.go b/daemon/cluster/executor/container/container.go
index 69d673bd3..829c5dcc3 100644
--- a/daemon/cluster/executor/container/container.go
+++ b/daemon/cluster/executor/container/container.go
@@ -351,6 +351,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
        hc := &enginecontainer.HostConfig{
                Resources:      c.resources(),
                GroupAdd:       c.spec().Groups,
+               Privileged:     c.spec().Privileged,
                PortBindings:   c.portBindings(),
                Mounts:         c.mounts(),
                ReadonlyRootfs: c.spec().ReadOnly,

Compile it again, and now everything should be fine!

Closing thoughts

To finish the whole thing, update the vendoring of the docker/cli fork to incorporate those changes (yes, docker/cli depends on docker/docker, which depends on docker/swarmkit) and now everything should compile.

Modify your docker.service to use the built binaries, call docker service create using the recently built docker binary and you’re good to go!

If you found any mistakes or felt a little lost, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis

Compiling and running your own forked Docker release

Ciro S. Costa — Sun, 03 Jun 2018 00:00:00 +0000

Hey,

Some days ago I needed to tweak Docker a little bit to check if changing some parameters of SwarmKit would make it work more reliably.

Given that the value I needed changing was hardcoded, no /etc/docker/daemon.json configuration would do the job. That meant that a fork would be needed.

It turns out that building and running a forked version of Docker is not complicated once you get to know what are the pieces involved.

What does the docker-ce package installs on my system?

As I’ve always been installing docker using apt, the first step was to get it running right from the binaries.

Not knowing what to expect from the building process, I started looking at what are the binaries that the docker-ce package installs on the system.

You can check which binaries are these by looking at dpkg --listfiles <pkg_name> (it lists the files installed to your system from pkg_name):

# List the files that the docker-ce package brought
# us and then filter out those that are not in
# the `/usr/bin` directory.
sudo dpkg --listfiles docker-ce | \
  grep /usr/bin

/usr/bin
/usr/bin/docker
/usr/bin/docker-containerd
/usr/bin/docker-containerd-ctr
/usr/bin/docker-containerd-shim
/usr/bin/docker-init
/usr/bin/docker-proxy
/usr/bin/docker-runc
/usr/bin/dockerd

Aside from /usr/bin/docker, which is the Docker CLI, the others are components that are run behind the scenes by the Docker daemon (dockerd).

dockerd is the daemon that keeps running as a server, taking commands from your docker client (usually from a unix socket) and then acting upon them, creating containers leveraging the other docker-* binaries.

Running the docker daemon behind the scenes

To have dockerd running at all times, docker-ce (the package) sets two systemd configuration files:

# List the files installed by `docker-ce` and the
# filter out those without a systemd in the path.
sudo dpkg --listfiles docker-ce | \
  grep systemd

/lib/systemd
/lib/systemd/system
/lib/systemd/system/docker.service      # <<<< 
/lib/systemd/system/docker.socket       # <<<<

These are systemd units, files that describe resources that should be taken care by systemd - the default init process used by most distributions (e.g., Ubuntu and Debian).

The later (docker.socket) controls the socket that dockerd uses to listen for requests made by a client.

Given that generally every .socket file must have a corresponding .service unit, the docker.socket is tied to the docker.service unit.

[Unit]
# By telling the unit that it's part of `docker.service`
# we're tying a relationship between them, making a restart
# on `docker.service` propagate to this unit (the same for
# a stop but not for a start).
PartOf=docker.service


# Information about the socket that systemd should supervise.
[Socket]
# The address to listen on for a stream (`SOCK_STREAM`).
# Given that it starts with a `slash`, a filesystem socket (`AF_UNIX`) 
# is created.
ListenStream=/var/run/docker.sock

# File permissions that should be used in the socket file.
SocketMode=0660
SocketUser=root
SocketGroup=docker


# This section carries information regarding installation of the unit.
# Its sole purpose is to provide `systemd` information regarding `enable`
# and `disable` commands issued by `systemctl` during installation or
# uninstallation of units.
#
# ps.: without this section, `systemctl enable` doesn't work.
[Install]
# Creates a symbolic link in the `.wants` directory of the `sockets` target,
# having the effect of making it a dependency of another unit.
#
# This is telling systemd to pull in the unit when starting sockets.target.
WantedBy=sockets.target

ps.: a cool effect of having the .socket unit is that you can keep the socket unit active (thus, having /var/run/docker.sock created) and the docker unit inactive (dockerd not running) at machine startup time such that only when a connection is made to the socket, docker starts.

The former (docker.service) is a service unit that not only executes the dockerd daemon, but it also keeps track of its liveness and makes sure that this is only initialized after some dependencies are met.

It looks like this (with comments added by me):

[Unit]
# Here we specify some of the dependencies that `dockerd` has.
# Although `After` only configures service startup order,
# We place a strict dependency on `docker.socket`.
#
# - `network-online` is a target that waits until the network is
#   considered up. After this target is met, it's supposed that
#   connections to network resources can be established.
#
# - `firewalld.service` lets us wait for any firewalld configs to
#   go on before docker starts. Under a regular ubuntu installation
#   this target generally doesn't exist.
#
# - `docker.socket` makes us start after the socket activation.
#
After=network-online.target docker.socket firewalld.service

# `Wants=` establishes a weak dependency on something.
# This means that in contrast with `Requires=`, if something goes 
# wrong with it, it can still proceed.
Wants=network-online.target

# Enforces a hard dependency on `docker.socket` unit - if it fails,
# then this service will also fail.
Requires=docker.socket


[Service]
# Makes use of what's specified in ExecStart as the main process
# of the service and then only proceeds to activate other dependencies
# (i.e., other units that specify a dependency on this service) after
# the process (dockerd) sends a notification to systemd via `sd_notify`.
#
# See moby/cmd/dockerd/daemon_linux.go to check the `notifySystem` call.
Type=notify
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0

# As systemd can act as a Linux containers daemon, this clearly conflicts
# with the interests of the docker daemon over the containers controlled
# by docker.
#
# Specifying `delegate=yes` we make systemd grant all the resource control
# to the main process started by `ExecStart`.
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s


[Install]
WantedBy=multi-user.target

Having those two files and the binaries, we can get Docker running from a set of binaries.

Let’s now generate them then.

Building right from the docker-ce repository

Although I’m not from Docker Inc, it seems like all the docker packaging for the community edition happens with what’s there in the docker/docker-ce repository.

Let’s make use of such repository then:

# Clone the docker-ce repository to somewhere
git clone https://github.com/docker/docker-ce
cd ./docker-ce

# Checkout to the tag you want.
# ps.: you can gather the list of tags by 
# running git tag.
git checkout v18.03.1-ce

# Having `make` installed, perform the build of 
# the static binaries.
#
# This will invoke a series of build steps that
# make use of the source code that lives under
# the `components` directory.
#
# ps.: you're required to have a running `docker`
# daemon to proceed (`dockerd` is built using docker).
#
# If you're curious, on a `t2.medium` machine without
# any previous runs of this command (i.e, zero cache),
# this takes the following `time`:
#       real	9m44.885s
#       user	0m9.320s
#       sys	0m0.634s
make static DOCKER_BUILD_PKGS=static-linux


# Check that results from the build
tree ./components/packaging/static/build/linux

./components/packaging/static/build/linux
├── docker
│   ├── docker
│   ├── docker-containerd
│   ├── docker-containerd-ctr
│   ├── docker-containerd-shim
│   ├── dockerd
│   ├── docker-init
│   ├── docker-proxy
│   └── docker-runc
└── docker-18.03.1-ce.tgz

That’s it! Send that .tgz to the machines you want, unpack it to the desired destination (/usr/bin if you don’t want to change the systemd scripts) and you’re done.

If all you wanted was a way to build docker from the ground and have the binaries, you can stop here.

Modifying docker (the daemon) from the main repository

Because docker-ce is just a repository that gathers three others (docker/docker-ce-packaging, moby/moby and docker/cli), it’s better to keep our modifications under a fork of the real thing (moby/moby) and then (optionally) use a fork of docker-ce to centralize our modifications.

When cloning moby/moby you’ll notice that the process of producing binaries is fairly straightforward: having docker already running, issue make binary and you’ll have a bundles directory filled with the daemon binaries.

# `docker/docker` has been renamed to `moby/moby`, 
# but given that I want to keep the golang import
# paths working, cloning from `docker/docker` makes
# things easier.
git clone https://github.com/docker/docker

# Set up the git remotes to have my fork's source
# code in.
git remote set-url origin https://github.com/cirocosta/docker
git remote add upstream https://github.com/docker/docker

# Grab the code
git fetch --all

# Check out to a branch with the modifications
git checkout privileged

# Build the binaries
make binary

Being dockerd made up of several components (like swarmkit and libnetwork), make sure that when you update those, you also update moby/moby’s references (there’s a vendor.conf file there that vndr - the vendoring tool - uses to fetch the dependencies). Thankfully, it’s very easy to make use of a fork in vendor.conf - you specify the import path, the git reference of your modified content and then the fork.

For instance, having modified docker/swarmkit, I updated vendor.conf:

IMPORT PATH                GIT REF       MY FORK
github.com/docker/swarmkit 268f203dda... https://github.com/cirocosta/swarmkit

ps.: the same is true for other repositories, like docker/cli.

If all you need is a modified daemon (and not a client), then now you’re done - distribute the binaries and you’re good.

Closing thoughts

It looks like the Docker contributors have spent a big time making sure that building Docker is not hard. What bothered me a bit is that it’s quite huge when it comes to dependencies - updating a dependency (using vndr) will end up in fetching a bunch of repositories.

Aside from the dependency complexity (which is understandable), I felt that the process was pretty straightforward - now I get why there are so many contributors.

If you had any question or spotted a mistake, please let me know - I’m cirowrc on Twitter!

Have a good one!

finis

Shell: how to add a prefix to the output of multiple commands

Ciro S. Costa — Sun, 13 May 2018 00:00:00 +0000

Hey,

This week I wanted to make the logs of a Docker build easier to read, and remembering how some tools prefix their outputs with a name, I thought that doing so would be a good way to go.

In such scenario, I had the following structure:

.
├── Dockerfile          # The Dockerfile that instruct the 
│                       # image building process
│
└── scripts             # directory full of scripts to be run
    ├── first.sh        # Ideally, each script would have its name
    ├── second.sh       # used as the prefix in the logs when they
    └── third.sh        # get executed.

For this example, let’s assume that each of these scripts output a dummy text.

cat ./scripts/{first,second,third}.sh
#!/bin/bash
echo "hello, first here"
#!/bin/bash
echo "hello, second here"
#!/bin/bash
echo "hello, third here"

Knowing how to run these three scripts would give a hint to how to proceed.

Here I had some options:

use find and `xargs together;
use find with its -exec argument;
using a for together with variable expansion.

Although all of them are capable of achieving the goal, I’d always first give a try to find piping to xargs - there are some caveats to running for loops over files that might break your script in ways you don’t expect (given that there’s a variable expansion going on, files named after command-line arguments could break your execution later).

# Search for all the files under the `./scripts`
# directory that end with `.sh`.
#
# Pipe the result of the search to `xargs`, which
# then executes the command supplied in the positional
# arguments replacing the replacement string supplied
# to `-I` (that is, when it finds `lol.sh`, it executes
# `bash lol.sh`.
find ./scripts -name "*.sh" \
        | xargs -I {} bash {}
hello, first here
hello, third here
hello, second here

However, given that we’ll need to make use of piping to have our outputs prefixed, that’d mean making xargs call an extra shell for that.

# If we want to pipe the result of `bash` to another
# command, then we need to use an extra shell to process
# the piping operator and set up the piping itself.
find ./scripts -name "*.sh" \
        | xargs -I {} /bin/sh -c 'bash {} | wc -c'
      18
      18
      19

So, in this case, using a for loop seems to be the way to go (knowing that none of our files there are going to break us):

for script in *.sh; do
        bash $script | wc -c
done
      18
      19
      18

Now, to solve the problem we introduced before (prefixing the output of the executions), we can leverage the fact that we can take the output produced by those executions and then mutate them accordingly such that the final result has a prefix.

Naturally, sed is a great candidate for mutating streams of strings.

# Create a stream of multiline text
printf 'haha\nhaha\nhaha\n'
haha
haha
haha

# Create a stream of multiline text and pipe
# it to `sed`.
#
# Using the `s/regular expression/replacement/`
# we can provide a regular expression that matches
# the very initial part of the string and then 
# replace that by a prefix.
printf 'haha\nhaha\nhaha\n' | sed 's/^/prefix: /'
prefix: haha
prefix: haha
prefix: haha

Putting all together, we can make use of script name in the place of prefix in the sed rule and achieve our goal:

# Iterate over all the script files under ./scripts.
# 
# Given that the expansion will result in names that
# include the `./scripts` prefix and the `.sh` suffix,
# we can make use of bash's prefix and suffix replacement
# syntax to get rid of that and end up with the pure name.
#
# Once clean names are got, then it's time to use in `sed`.
for script in ./scripts/*.sh; do
  name=${script#./scripts/}
  name=${name%.sh}

  bash $script | sed "s/^/[${name}] /"
done

[first] hello, first here
[second] hello, second here
[third] hello, third here

As an extra tip, if you happen to have your output being forwarded to a log collector and you see many lines (that should be separate) comming together, it might be the case that sed is performing some buffering. You can get rid of such behavior by executing sed with stdbuf.

That’s it!

Please let me know if you have any questions or if you spot anything wrong. I’m cirowrc on Twitter.

Have a good one!

finis

Using HAProxy maps with Access control lists (acl)

Ciro S. Costa — Thu, 10 May 2018 00:00:00 +0000

Hey,

If you’ve ever wondered whether you can tie Access Control Lists (ACLs) with maps in HAProxy, the answer is: yes.

Let’s tailor a scenario here:

based on a map, decide whether a request to a given domain should be answered by the current frontend or not - in the negative case, forward the request to a different frontend.

One application that can be thought from this is the case where some domains are meant to be served by both a plain HTTP frontend and a HTTPS frontend too, but some other domains must be redirected to HTTPS when a request is made to HTTP.

For instance:

http://test.example.com and https://test.example.com OK;
https://prod.example.com is OK but no http://prod.example.com such that when a request is made to http://prod.example.com, this should be redirected to the HTTPS frontend.

That said, this is how our configuration would look like:

Let’s write it then.

# Load the `./script.lua` script that defines some services
# that are meant to respond with `SERVICE_N OK` when a request
# is made to them.
#
# Having these services defined allows us to not need a real
# server behind the scenes.
global
	lua-load	./script.lua


defaults
	mode		http
	timeout		client 10000
	timeout		server 10000
	timeout		connect 1000


# The frontend `fe1` is the one that clients are meant to
# make requests to in order to access the backends as mapped
# by the `./domain-map` file.
#
# The contents of the `domain-map` file are:
#
#       foo.com backend_1
#       bar.com backend_2
#
# This way, requests that are not for `foo.com` or `bar.com`
# are sent to the other frontend (fe2) based on an ACL, otherwise,
# the respective backend is used.
frontend		fe1
	bind		127.0.0.1:8000
	acl		has_domain hdr(Host),map_str(./domain-map) -m found
	redirect	prefix http://127.0.0.1:8001 if ! has_domain
	use_backend	%[req.hdr(host),lower,map_str(./domain-map)]


# A sample frontend that always responds with a specific
# service (demonstration purposes).
frontend		fe2
	bind		127.0.0.1:8001
	http-request	use-service lua.service3


backend			backend_1
	http-request	use-service lua.service1


backend			backend_2
	http-request	use-service lua.service2

Being service1, service2 and service3:

core.register_service("service1", "http", function (applet)
	local response = "SERVICE 1 OK"

	applet:set_status(200)
	applet:start_response()
	applet:send(response)
end)

core.register_service("service2", "http", function (applet)
	local response = "SERVICE 2 OK"

	applet:set_status(200)
	applet:start_response()
	applet:send(response)
end)

core.register_service("service3", "http", function (applet)
	local response = "SERVICE 3 OK"

	applet:set_status(200)
	applet:start_response()
	applet:send(response)
end)

We can see it in practice:

# Make a request with `foo.com` as the destination host.
# Given that `foo.com` is a mapped backend, the request
# will get directed to a backend controlled by the frontend
# itself (backend_1)
curl \
        --location \
        --header "Host: foo.com" \
        --verbose \
        localhost:8000/
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: foo.com
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Transfer-encoding: chunked
< 
SERVICE 1 OK

# Just like in the case above, `bar.com` is also a
# domain controlled by this frontend.
curl \
        --location \
        --header "Host: bar.com" \
        --verbose \
        localhost:8000/
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: bar.com
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Transfer-encoding: chunked
< 
* Connection #0 to host localhost left intact
SERVICE 2 OK


# Differently from the two cases above, `test.com`
# is not included in the frontend1 map.
#
# This means that the ACL we placed there will evaluate
# `has_domain` to false and then will redirect us to
# the second frontend  (which always responds with
# SERVICE 3). 
#
# Check out how we get redirected to the other frontend.
curl \
        --location \
        --header "Host: test.com" \
        --verbose \
        localhost:8000/
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: test.com
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Content-length: 0
< Location: http://127.0.0.1:8001/      # << REDIRECTION
<                                       # sends us to another frontend

* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://127.0.0.1:8001/'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8001 (#1)
> GET / HTTP/1.1
> Host: 127.0.0.1:8001
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Transfer-encoding: chunked
< 
* Connection #1 to host 127.0.0.1 left intact
SERVICE 3 OK

That’s it!

If you have any questions or spotted something off, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis

Making HAProxy respond 200 OK to health checks

Ciro S. Costa — Sat, 05 May 2018 00:00:00 +0000

Hey,

This is a quick post for those who might need to perform HTTP health checks against a running HAProxy instance.

From my perspective, three ways of doing it:

using the monitor-uri directive (you should use this one);
using a custom HTTP file with errorfile directive;
using a lua script.

While the first and seconds should work in any HAProxy installation, the third requires lua, which your HAProxy binary might have support or not.

ps.: In the past, I wrote about how to install haproxy with Lua on MacOS.

The gist of it can be summarized in a single haproxy.conf file that contains three frontends - each representing a way of doing it.

#	This configuration demonstrates how several frontends
#	can be created in different ways to indicate
#	via HTTP that the HAProxy instance is running.
#
#	The usefulness of these arise in scenarios like
#	AWS NLB where no TCP health check can be specified
#	and you can't modify the health check headers.
global
	lua-load	./script.lua


defaults
	mode		http
	timeout		client 10000
	timeout		server 10000
	timeout		connect 1000


#	The first way of doing it is via lua scripting:
#	a script (which registers a service) is loaded
#	and then when a request is made to a frontend
#	with the directive `use-service`, the script
#	does the job of responding with 200OK.
frontend		status-lua
	bind		127.0.0.1:8000
	http-request	use-service lua.status_service


#	The second way is using `monitor-uri` (the
#	proper way of doing since you can make the
#	monitor fail in other parts of the configuration
#	and there's no lua involved).
frontend		status-monitor
	bind		127.0.0.1:8001
	monitor-uri	/


#
#	The third way is using a "hack": setting errorfile
#	to respond with a 200OK when a 503 happens, we're
#	able to indicate that HAProxy is active.
frontend		status-errorfile
	bind		127.0.0.1:8002
	http-request	set-log-level silent
	errorfile	503 ./200.http

With that, we now need to tailor a script.lua file that defines the status_service that is used by the first frontend:

-- `core` is a static class provided by haproxy containing all
-- the haproxy methods we can use.


-- `register_init` registers a function to be executed after
-- configuration parsing.
core.register_init(function ()
	core.log(core.info, "script loaded: case-200-ok")
end)

-- `register_service` registers a lua function to be executed
-- as a service. Once it's been properly registered, the service
-- can be referenced in the haproxy configuration as `lua.<svc_name>`.
--
-- Depending on the mode (tcp or http), the applet is either an
-- AppletHTTP or AppletTCP class instance.
core.register_service("status_service", "http", function (applet)
	local response = "OK"

	applet:set_status(200)
	applet:start_response()
	applet:send(response)
end)

And then, for the last frontend, tailor a 200.http file:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/plain
User-Agent: *
Allow: /

ps.: given that we’re defining the whole HTTP message, it must end with carriege return and line feed (\r\n).

That’s it!

Load haproxy.conf and you’re good to go.

If you have any doubts or spotted any mistakes, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis

Blocking EC2 Metadata service from Docker containers in AWS

Ciro S. Costa — Sun, 29 Apr 2018 00:00:00 +0000

Hey,

Those who already have deployed Docker to an EC2 instance might have noticed (or not) that, within the containers, you’re able to perform requests to the EC2 metadata service, discovering information about the host from within these containers.

While in some cases that’s a desired thing (i.e., not needing to explicitly pass credentials to containers and make use of instance profiles to authenticate against AWS services), sometimes it ends up leaking information that the containers (which maybe you don’t control) shouldn’t have access to.

As a way of showing both the usefulness of the metadata service and a setup where containers can’t access the service, one could tailor an instance like this:

A regular EC2 instance containing two components:

cirocosta/awsmon service: a daemon that gathers host metrics that AWS doesn’t automatically retrieve for you (like memory and load) and forwards it to AWS CloudWatch - without any hardcoded credentials, it makes use of the aws metadata service to retrieve the credentials necessary for putting metrics in cloudwatch;
docker running containers that I want not to be able to access the EC2 metadata service.

It turns out that there’s not much documentation out there on how to accomplish such kind of setup.

In this blog post, I go through the steps of making regular processes be able to connect to a destination while at the same time blocking containers from doing so (while still not removing all their external connectivity).

What’s EC2 metadata service about

The instance metadata service is a service that you can access by making requests to a well-defined address from within an EC2 instance: http://169.254.169.254.

# Check what are the types of metadata values that
# we can gather.
curl 169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
iam/                    # << this is interesting
instance-action
instance-id
...


# Retrieve the public IPV4 address of the current
# instance.
curl 169.254.169.254/latest/meta-data/public-ipv4
18.231.176.21


# Retrieve the instance profile associated with the
# instance that we're running.
curl http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2018-04-29T13:29:01Z",
  "InstanceProfileArn" : "arn:aws:iam::111111111:instance-profile/put-metrics-profile",
  "InstanceProfileId" : "AAAABBBBBBBCCCCCDDD"
}


# Retrieve the credentials generated for the role (default) 
# assigned to the instance profile associated with the instance.
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/default
{
  "Code" : "Success",
  "LastUpdated" : "2018-04-29T13:28:16Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "AAAEEEEEIIIIOOO",
  "SecretAccessKey" : "aaaaaaaa2222222233333330000aaa",
  "Token" : "a-very-long-token-that-looks-to-be-base64-encoded=",
  "Expiration" : "2018-04-29T19:50:14Z"
}

Not only information regarding the machine can be retrieved as you can see - security credentials are also first-class citizens in the metadata service.

When you create an instance with an instance profile attached, this instance can retrieve the role credentials by making requests to this service.

Given that the service (like almost any other in AWS) is throttled, making too many requests to the service will make it unavailable for the machine.

If you’re using the instance metadata service to retrieve AWS security credentials, avoid querying for credentials during every transaction or concurrently from a high number of threads or processes, as this may lead to throttling. Instead, we recommend that you cache the credentials until they start approaching their expiry time.

A user container could create a denial of service just by making too many requests to it, making the service unavailable to services in the machine that need to use it.

Blocking any connectivity to the EC2 metadata service

The most straightforward way to block any requests to the EC2 metadata service is by modifying the routing table of the instance.

# Show the kernel's IP routing table
route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.0.1      0.0.0.0         UG    100    0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker_gwbridge
172.31.0.0      0.0.0.0         255.255.240.0   U     0      0        0 eth0
172.31.0.1      0.0.0.0         255.255.255.255 UH    100    0        0 eth0


# Place a rejection for the fixed IP of the
# metadata service
route add -host 169.254.169.254 reject

# Check the updated IP table
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.0.1      0.0.0.0         UG    100    0        0 eth0
169.254.169.254 -               255.255.255.255 !H    0      -        0 -
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker_gwbridge
172.31.0.0      0.0.0.0         255.255.240.0   U     0      0        0 eth0
172.31.0.1      0.0.0.0         255.255.255.255 UH    100    0        0 eth0

Given that we’re rejecting traffic at the routing decision moment, no traffic (either from a container of from the host) is able to get to that address.

# Try to make a request to the EC2 metadata service
# right from the host.
curl http://169.254.169.254/ -v
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Immediate connect fail for 169.254.169.254: No route to host
* Closing connection 0
curl: (7) Couldnt connect to server

In a container, we can’t connect either:

# Run a container and then try the same
# command.
#
# Given that at some point the process inside
# the container will end up issuing a `connect`
# that goes through the routing table, it'll
# fail.
docker run -it alpine /bin/sh
apk add --update curl
curl http://169.254.169.254/ -v
*   Trying 169.254.169.254...
* TCP_NODELAY set
* connect to 169.254.169.254 port 80 failed: Host is unreachable
* Failed to connect to 169.254.169.254 port 80: Host is unreachable
* Closing connection 0
curl: (7) Failed to connect to 169.254.169.254 port 80: Host is unreachable

If all you need is make sure that happens after you extracted initial metadata that is static (e.g., the instance’s region, the instance’s id etc), this is enough and no concerns regarding Docker changing iptables would be raised.

In the case of needing to preserve host connectivity to the service and disallowing docker containers to access it, then a different approach is needed.

ps.: you can remove the “reject route” by issuing route del 169.254.169.254 reject.

Using internal networks to block external requests

When making use of overlay networks, one easy way of prohibiting requests to external services is creating internal-only networks.

These are networks without connectivity to the docker_gwbridge that allows containers to make external requests.

For instance:

# Create two networks: one that is internal-only and another
# that is a regular network with external connectivity.
docker network create \
        --driver overlay \
        --attachable \
        --internal mynet-internal
docker network create \
        --driver overlay \
        --attachable mynet

# Run a container in the network with external connectivity
docker run \
        --network mynet \
        --tty \
        --detach \
        --name with-conn \
        alpine /bin/sh

docker exec with-conn ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=51 time=2.603 ms
64 bytes from 8.8.8.8: seq=1 ttl=51 time=2.629 ms
^C


# Run a container in the network without external connectivity
docker run \
        --network mynet-internal \
        --tty \
        --detach \
        --name without-conn \
        alpine /bin/sh

docker exec without-conn ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network unreachable

Naturally, this covers a very particular use case - total isolation from outside the network -, not very suitable for most people.

If you’re curious regarding the difference between these two networks, check out how the interfaces and routing are set up inside the namespaces of these containers we just created.

In the one with a network that enables external connectivity, there’s an extra interface that ends up connected to the docker_gwbridge interface in the host via an extra interface set up within the container’s network namespace:

Any requests that are not directed towards the internal network are then routed to docker_gwbridge in the host.

Given that, in the host, docker configures the necessary iptables rules to NAT outgoing packets that originate from the docker_gwbridge interface, packets can flow to the outside world without problems.

In the other container though, there’s only the overlay network interface and no extra route to a default gateway like in the first.

Putting everything together and assuming that there’s an extra container without connectivity, the overview is like the following:

Using classid marking

Given that every container runs in a cgroup, and that it’s possible to make Docker use a parent cgroup that encapsulates all the containers, we could eventually use net_cls.classid to put a little mark in every packet and then make use of that mark in iptables to block certain traffic.

Although it’s indeed possible to have net_cls.classid setup, we can’t make use of such mark from outside the network namespace of the container (as the classid is restricted by the namespace).

Such limitation means that to go with such strategy we’d need to put specific iptables rules in every container namespace created.

Not good.

A solution: using the DOCKER-USER chain

Recent Docker daemons include an extra chain that is never changed by the daemon - the DOCKER-USER chain -, meaning that we can modify it without having to worry about it being flushed or messed up by Docker.

As the table sits right in the forward path that packets from containers to external networks, we can place our filtering rules right there and then block any container access to the EC2 metadata server.

iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target            prot opt source        destination         
DOCKER-USER       all  --  anywhere      anywhere     # <<<<<<<<<
DOCKER-ISOLATION  all  --  anywhere      anywhere            
ACCEPT            all  --  anywhere      anywhere     ctstate RELATED,ESTABLISHED
DOCKER            all  --  anywhere      anywhere            
ACCEPT            all  --  anywhere      anywhere            
ACCEPT            all  --  anywhere      anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)                      # <<<<<<<<<
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

One easy way to making sure that this is the right chain to place rules is by adding a rule that will simply lead to the LOG target, giving us palpable results regarding matching when running some scenarios.

# Insert a rule to the DOCKER-USER chain to
# jump to the LOG target such that we can
# visualize the logs in the kernel logs
# whenever a container reaches 1.1.1.1.
#
# Because `DOCKER-USER` comes referenced
# in FORWARD, it'll only be matched in the
# case of the host forwarding the connections
# from the containers, and not those being
# initiated directly from the host (not 
# namespaced).
iptables \
        --insert DOCKER-USER \
        --jump LOG \
        --destination 1.1.1.1 \
        --log-prefix="[container]"

# Insert rule to the OUTPUT chain to
# jump to the LOG target such that we can
# visualize the packets sent to 1.1.1.1 
# from the host.
iptables \
        --insert OUTPUT \
        --jump LOG \
        --destination 1.1.1.1 \
        --log-prefix="[host]"

# Check how our two rules have been placed in
# the two chains that matter.
iptables \
        --list \
        --numeric
(...)
Chain OUTPUT (policy ACCEPT)
target     prot opt source         destination
LOG        all  --  0.0.0.0/0      1.1.1.1       LOG flags 0 level 4 prefix "[host]"

(...)
Chain DOCKER-USER (1 references)
target     prot opt source         destination
LOG        all  --  0.0.0.0/0      1.1.1.1       LOG flags 0 level 4 prefix "[container]"
RETURN     all  --  0.0.0.0/0      0.0.0.0/0


# Clear the kernel ring buffer so we
# don't check old logs
dmesg --clear

# Send some packets from the host
ping 1.1.1.1 -c 5

# Check dmesg for kernel logs.
#
# ps.: you could initiate a second terminal and
# follow the logs with the `--follow` opt.
dmesg
[ 3632.384997] [host]IN= OUT=eth0 SRC=172.31.5.64 DST=1.1.1.1 LEN=84 ... SEQ=1 
[ 3633.386725] [host]IN= OUT=eth0 SRC=172.31.5.64 DST=1.1.1.1 LEN=84 ... SEQ=2 
[ 3634.388343] [host]IN= OUT=eth0 SRC=172.31.5.64 DST=1.1.1.1 LEN=84 ... SEQ=3 
[ 3635.389922] [host]IN= OUT=eth0 SRC=172.31.5.64 DST=1.1.1.1 LEN=84 ... SEQ=4 
[ 3636.391557] [host]IN= OUT=eth0 SRC=172.31.5.64 DST=1.1.1.1 LEN=84 ... SEQ=5 

# Send some packets from a container 
docker run \
        --rm \
        alpine \
        ping 1.1.1.1 -c 5

# From the host, check `dmesg` and see that
# there are no other new `[host]` entries, but
# only new `[container]` entries as we wanted.
#
# Note that there's an included `IN` interface and
# and an `OUT` as we expected given that we're in
# the middle of a forwarding path.
dmesg
[ 3724.759112] [container]IN=docker0 OUT=eth0 (...) 0 SRC=172.17.0.3 DST=1.1.1.1 LEN=84 ... SEQ=0 
[ 3725.759295] [container]IN=docker0 OUT=eth0 (...) 0 SRC=172.17.0.3 DST=1.1.1.1 LEN=84 ... SEQ=1 
[ 3726.759484] [container]IN=docker0 OUT=eth0 (...) 0 SRC=172.17.0.3 DST=1.1.1.1 LEN=84 ... SEQ=2 
[ 3727.759673] [container]IN=docker0 OUT=eth0 (...) 0 SRC=172.17.0.3 DST=1.1.1.1 LEN=84 ... SEQ=3 
[ 3728.759849] [container]IN=docker0 OUT=eth0 (...) 0 SRC=172.17.0.3 DST=1.1.1.1 LEN=84 ... SEQ=4

Knowing which chain to use to place a filter rule, now it’s matter of dropping the connections to the EC2 metadata service.

# Delete (optionally) the log rule.
#
# ps.: The `1` argument is the number of the
# rule in the chain.
iptables \
        --delete DOCKER-USER 1

# Add the rule to reject traffic to the ec2 metadata
# service.
iptables \
        --insert DOCKER-USER \
        --destination 169.254.169.254 \
        --jump REJECT

# Try to access the metadata service from the host.
curl 169.254.169.254/latest/meta-data/instance-id
i-0648792307a6e6610

# Try to access the metadata service from the container.
docker run -it alpine /bin/sh
apk add --update curl 
curl -v 169.254.169.254/
*   Trying 169.254.169.254...
* TCP_NODELAY set
* connect to 169.254.169.254 port 80 failed: Connection refused
* Failed to connect to 169.254.169.254 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 169.254.169.254 port 80: Connection refused

That’s it!

Make sure that you have the iptables configuration properly set up at boot (given that the changes are not permanent) and you’re done.

If you have any questions or saw anything wrong in the blog post, please let me know! I’m cirowrc on Twitter.

Have a good one!

Ciro

Running Docker with a forked RunC

Ciro S. Costa — Wed, 28 Feb 2018 00:00:00 +0000

Hey,

While Docker takes some time to expose more options to service creation (for instance, limiting the maximum number of PIDs of a service), it’s important that they are still enforced (at least with a default number).

Many docker options that are not exposed to docker swarm mode yet - see add more options to service create / `service update.

It turns out though that there’s a way of adding such functionality without forking SwarmKit and Docker just to add that if you’re fine with setting a single default - adding a modified runtime to containerd (which is transparently managed by docker for you) and then making it the default runtime.

Can you run Docker with a forked runc?

That’s possible because the Docker daemon is not the process that gets executed (in the sense of performing an execve) when a container is meant to be ran - it delegates the action to containerd which then controls a list of runtimes (runc by default) which is then responsible for creating a new process (calling the defined runtime as specified in the configuration parameters) with some isolation and only then execveing the entrypoint of that container.

Regarding interactions between docker, containerd, and runc, we can understand that without even looking at the source code - plain strace and pstree can do the job.

Without any container running, perform a ps aux | grep docker.

ps aux | grep docker
root      1002  /usr/bin/dockerd -H fd://
root      1038  docker-containerd --config /var/run/docker/containerd/containerd.toml

This shows us that we have two daemons running - the docker daemon and the docker-containerd daemon.

Given that dockerd interacts heavily with containerd all the time and the last is never exposed to the internet, it makes sense to bet that its interface is unix-socket based.

The communication between Docker and ContainerD

We can check that by looking at the docker-conatainerd configuration (/var/run/docker/containerd/containerd.toml):

root = "/var/lib/docker/containerd/daemon"
state = "/var/run/docker/containerd/daemon"
# ...

[grpc]
  address = "/var/run/docker/containerd/docker-containerd.sock"
  uid = 0
  gid = 0

Another options is to just grab docker-containerd pid and inspect its file descriptors:

# Retrieve the PID of `containerd`
CONTAINERD_PID="$(ps -C docker-containerd -o pid= | tr -d ' ')"

# Check what are the file descriptors associated with
# that process by looking at proc fs
sudo ls -lah /proc/$CONTAINERD_PID/fd
...
 6 -> socket:[17674]
 7 -> socket:[17675]
 8 -> socket:[17676]
 9 -> socket:[18597]

# That doesn't help much, aside from seeing that there are
# some sockets created. 
# 
# We can gather more information looking at the inspection
# performed by `lsof` though.
#
# Given that the inode of the sockets are known, we can filter
# the result to focus only on a single socket if we want
sudo lsof -p $CONTAINERD_PID | grep 17674
CMD        FD   TYPE  NODE  NAME
docker-co  6u   unix  17674 /var/run/docker/containerd/docker-containerd.sock type=STREAM

We can see how docker delegates all the work of setting up the container to containerd by looking at the write(2)s performed by docker to the containerd’s unix socket right before creating the container.

# Retrieve the PID of `dockerd`
DOCKERD_PID="$(ps -C dockerd -o pid= | tr -d ' ')"
BUFSIZE=1024

# Start strace following all forks and allowing it to
# print big lines (as big as 1024 bytes).
#
# By grepping `mycontainer` we can follow our trace of 
# `HOSTNAME=mycontainer` which should land at some point
# at containerd and in the end go through runc.
sudo strace -f \
        -e write \
        -s $BUFSIZE \
        -p $DOCKERD_PID 2>&1 | \
        grep containerd

[pid  1062] write(11, 
        "...io.containerd.runtime.v1.linux\22P\n!containerd.linux.runc.RuncOptions\22+"
        "\n\vdocker-runc\22\34/var/run/docker/runtime-runc*\340\235\1\n6types.containerd"
        ".io/opencontainers/runtime-spec/1/Spec\22\244\235\1{\"ociVersion\":\"1.0.1\",\"
>>>>>>  \"HOSTNAME=mycontainer\",\"NGINX_VERSION=1.13.9\"],\"cwd\":\"/\",\"capabilities\""
        ":{\"bounding\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CA, 16393 <unfinished ...>

So we can see that docker is writing all that config to the file descriptor 11.

Looking at /proc/$DOCKERD_PID/fd doesn’t reveal much though - it doesn’t tell us what’s the end of that socket (just like with a plain TCP client-server communication, when you chat over unix sockets, clients also create a socket on their end):

sudo ls -lah /proc/$DOCKERD_PID/fd | grep '11 ->'
11 -> socket:[17757]

To determine what’s the end of that socket (the server-side unix socket that is in PASSIVE state), we can use ss:

# Check out what is the INODEs involved in the established
# connecition between a file descriptor 11 in the system
# (we could have more processes with a connection of fd=11 as
# that's per-process, but that's ok) and another peer.
#
# With both INODEs in hand we can use `lsof` again to inspect
# what are them
sudo ss -a --unix -p | grep fd=11
Netid  State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port                
u_str  ESTAB      0      0       * 17757              * 17758           users:(("dockerd",pid=1002,fd=11))

# Check what those sockets are all about
sudo lsof -U | grep '17757\|17758'
CMD                PID   FD   TYPE ADDR                INODE  NAME
dockerd            1002  11u  unix 0xffff90edf6db6800  17757 type=STREAM
docker-containerd  1038  10u  unix 0xffff90edf6db6000  17758 /var/run/docker/containerd/docker-containerd.sock type=STREAM

The instantiation of containers by ContainerD

In that {CONTAINER_CONFIG} there’s one line that stadands out for what we’re looking for here (changing the default runtime):

containerd.linux.runc.RuncOptions
        \22+\n\vdocker-runc
                \22\34/var/run/docker/runtime-runc*\340\235\1\n6
                types.containerd.io/opencontainers/runtime-spec/1/Spec\22\244\235\1
        {\"ociVersion\":\"1.0.1\"

We can’t see very well from the result of strace, but we can inspect that looking at the configuration used by containerd by using ctr, the command line utility that helps us interact with the containerd daemon:

# Check out what the containerd directory structure looks like
tree /var/lib/docker/containerd/daemon
.
├── io.containerd.content.v1.content
│   └── ingest
├── io.containerd.metadata.v1.bolt
│   └── meta.db
├── io.containerd.runtime.v1.linux
│   └── moby                            # <<< THE NAMESPACE
│       └── ac550b5a0083269e9866...     # <<< THE CONTAINER
├── io.containerd.snapshotter.v1.btrfs
└── io.containerd.snapshotter.v1.overlayfs
    └── snapshots


# Gather the containers that have been spawn in the
# moby namespace.
sudo docker-containerd-ctr \
        --namespace moby \
        --address /var/run/docker/containerd/docker-containerd.sock \
        containers ls
CONTAINER        ..   IMAGE    RUNTIME                           
ac550b5a0083269e9..   -        io.containerd.runtime.v1.linux 

# Gather the information related to the runtime of the container.
#
# This should reveal which binary it's used to create the actual
# containers.
sudo docker-containerd-ctr \
        --namespace moby  \
        --address /var/run/docker/containerd/docker-containerd.sock \
        containers info \
        ac550b5a0083269e9866e0d868e34e2fd35e1c5c6de31df00f481734d94a3ff7 | \
        jq '.Runtime'
{
  "Name": "io.containerd.runtime.v1.linux",
  "Options": {
    "type_url": "containerd.linux.runc.RuncOptions",
    "value": "Cgtkb2NrZXItcnVuYxIcL3Zhci9ydW4vZG9ja2VyL3J1bnRpbWUtcnVuYw=="
  }
}

# Decode the base64 value such that we can understand
# what's in there
echo "Cgtkb2NrZXItcnVuYxIcL3Zhci9ydW4vZG9ja2VyL3J1bnRpbWUtcnVuYw==" | base64 -d
docker-runc/var/run/docker/runtime-runc

We can now verify that containerd indeed makes use of runtime-runc when it initializes a container by, again, using strace, but this time, on containerd:

# Trace the execution of the running docker-containerd but
# filter the syscall tracing to only catch the `execve` calls
# such that we can see which process images are being used for
# the new processes.
sudo strace -f \
        -e execve \
        -p $CONTAINERD_PID


# A containerd-shim is created to execute `docker-runc` and keep
# track of its execution, allowing it to run the container entrypoint 
# process and not have to stay around after its execution.
#
# It's also responsible for keeping the IO and performing some extra
# cleanup roles if necessary.
#
# Note.: although this is spawn by containerd, it can be reparented
# by the machine pid1.
[pid  3749] execve("/usr/bin/docker-containerd-shim", [
        "docker-containerd-shim", 
        "-namespace", "moby", 
        "-workdir", "/var/lib/docker/containerd/daemo"..., 
        "-address", "/var/run/docker/containerd/docke"..., 
        "-containerd-binary", "/usr/bin/docker-containerd", 
        "-runtime-root", "/var/run/docker/runtime-runc"], [/* 7 vars */]) = 0


# docker-runc starts the process of creating the OCI container bundle
[pid  3755] execve("/usr/bin/docker-runc", [
        "docker-runc", 
        "--root", "/var/run/docker/runtime-runc/mob"..., 
        "--log", "/run/docker/containerd/daemon/io"..., 
        "--log-format", "json", 
        "create", 
        "--bundle", "/var/run/docker/containerd/daemo"..., 
        "--pid-file", "/run/docker/containerd/daemon/io"..., 
        "a919d61879fac203b1f2f78ddee3903c"...], [/* 7 vars */]) = 0


# docker-runc then starts the actual container
[pid  3807] execve("/usr/bin/docker-runc", [
        "docker-runc", 
        "--root", "/var/run/docker/runtime-runc/mob"..., 
        "--log", "/run/docker/containerd/daemon/io"..., 
        "--log-format", "json", 
        "start", 
        "a919d61879fac203b1f2f78ddee3903c"...], [/* 7 vars */]) = 0


# the container entrypoint gets executed
[pid  3771] execve("/usr/sbin/nginx", [
        "nginx", 
        "-g", "daemon off;"], [/* 4 vars */] <unfinished ...>



# We can now inspect the process tree of the container's entrypoint
# to see which processes are left (and check what is the first process
# when the namespaces got changed).
NGINX_PID=3771
sudo pstree \
        --show-pids \
        --ascii \
        --long \
        --ns-changes \
        --show-parents $NGINX_PID

systemd(1)
        ---dockerd(1002)
                ---docker-containerd(1038)
                        ---docker-containerd-shim(3749)
                                ---nginx(3771,ipc,mnt,net,pid,uts)

Regardless of what we use in that runc component, if it’s an OCI compliant runtime, then a container can be run.

Forking runc

Being a regular Golang project, make sure you have what’s needed to build a Go project (set your $GOPATH and stuff accordingly).

# Retrieve the `opencontainers/runc` and have it
# under `GOPATH/src`.
#
# This way we can very quickly perform the modifications
# to the source code, compile and see if they indeed 
# work.
go get -v github.com/opencontainers/runc

# Get into the source code destination
cd $GOPATH/src/github.com/opencontainers/runc

# Here you should see `runc` already built.
./runc --help
NAME:
   runc - Open Container Initiative runtime

runc is a command line client for running applications 
packaged according to the Open Container Initiative (OCI) 
format and is a compliant implementation of the Open Container 
Initiative specification.

Although with that we have runc already built, this version doesn’t have seccomp support (which is activated with an extra build tag).

If you don’t have it yet, gather the dependency (on Ubuntu it’s libseccomp-dev) and then build it with seccomp as a build tag:

# Verify that there's no libseccomp being open when
# we execute ./runc (that is, it's not compiled with
# support to seccomp) - nothing shows as it's not 
# there.
sudo strace -e openat ./runc --help 2>&1 | grep seccomp

# Remove the binary without seccomp
rm ./runc

# Update the package information from all the
# configured sources
sudo apt update -y

# Install the development package of libseccomp
sudo apt install -y \
        libseccomp-dev

# Build `runc` again, but now specifying that we
# want seccomp as well as apparmor.
#
# Note.: the default `BUILDTAGS` in the Makefile
# uses only `seccomp`. Docker requires both though.
make BUILDTAGS='seccomp apparmor'

# Verify that libseccomp is being loaded:
sudo strace -e openat ./runc --help 2>&1 | grep seccomp
openat(AT_FDCWD, 
        "/lib/x86_64-linux-gnu/libseccomp.so.2", 
        O_RDONLY|O_CLOEXEC) = 3

Now that we have a fresh build of runc, we can tell Docker to use our own version instead of docker-runc (the default runtime).

Adding a new runtime to the Docker daemon configuration

Head over to the daemon configuration file (/etc/docker/daemon.json) and add a new field:

{
    "runtimes": {
        "our-runtime": {
            "path": "/usr/local/bin/our-runc"
        }
    }
}

Link the runc generated in $GOPATH/src/github.com/opencontainers/runc to /usr/local/bin/our-runc and then tell dockerd to reload (send a SIGHUP to the dockerd process):

# Link the custom runc to `/usr/local/bin/our-runc`
# such that we can update the binary with a simple
# `make` and not have to copy to `/usr/local/bin`
# all the time.
sudo ln -sf $(realpath ./runc) /usr/local/bin/our-runc

# Tell dockerd to reload
sudo kill -s SIGHUP $(pgrep dockerd)

# Check that the daemon actually got the signal to
# reload and it's really doing it.
# 
# By default, dockerd will output the full configuration
# that it loaded and will be in use now.
#
# ps.: not every configuration will be reloaded with a
#      soft-reload via SIGHUP. Check the docs.
sudo journalctl -u docker.service -f

dockerd[1002]: time="2018-0..." level=info msg="Got signal to reload configuration, reloading from: /etc/docker/daemon.json"
dockerd[1002]: time="2018-0..." level=info msg="Reloaded configuration: {\"mtu\":1500, ...
        \"runtimes\":
                {\"our-runtime\":{\"path\":\"/usr/local/bin/our-runc\"},
                \"runc\":{\"path\":\"docker-runc\"}},
        \"default-runtime\":\"runc\" ...

Parsing that configuration that dockerd showed us (in JSON), we can highlight some things:

---
# If we don't specify a runtime, the runtime named `runc` 
# from the runtimes object is used.
#
# Naturally, this can be configured
default-runtime: 'runc'
runtimes:
  our-runtime:          # our runtime got loaded
    path: '/usr/local/bin/our-runc'
  runc:                 # default `runc` runtime that docker has
    path: 'docker-runc' # $PATH resolution can be performed

# Check in `docker info` if the configuration got
# properly loaded
docker info | \
        grep -i runtime

Runtimes: our-runtime runc
Default Runtime: runc

Cool, with that set, let’s modify our-runc and run a container.

Modifying runc to place a default PID limit

Given that Docker swarm mode doensn’t allow us to place a limit on the number of PIDs that a container can hold, we can go directly to runc and modify that:

diff --git a/libcontainer/cgroups/fs/pids.go b/libcontainer/cgroups/fs/pids.go
index f1e37205..418a5152 100644
--- a/libcontainer/cgroups/fs/pids.go
+++ b/libcontainer/cgroups/fs/pids.go
@@ -23,6 +23,11 @@ func (s *PidsGroup) Apply(d *cgroupData) error {
        if err != nil && !cgroups.IsNotFound(err) {
                return err
        }
+
+       if d.config.PidsLimit == 0 {
+               d.config.PidsLimit = 500
+       }
+
        return nil
 }

Now, compile it with make BUILDTAGS='seccomp apparmor' and run a container.

# Run the container specifying that we want `our-runtime` (`our-runc`)
# to be used as the container runtime.
docker run \
        --rm \
        --name c1 \
        --runtime our-runtime 
        nginx:alpine

# Gather the full ID of the container such that we can
# look at the cgroup filesystem and check if the pids
# limit has really been set.
CONTAINER_ID=$(docker inspect c1 | jq -r '.[] | .Id')

# Look at the `pids.max` key to verify if the 500
# is really set - the default value when we don't
# specify (per the changes we did).
cat /sys/fs/cgroup/pids/docker/$CONTAINER_ID/pids.max
500

That’s it! We’re running a Docker container with a modified runc.

Now, if you’re willing to always run containers with that runc, set the default-runtime property in /etc/docker/daemon.json and you’re good to go.

Closing thoughts

It’s great that Docker is modular enough to allow us to perform some quick modifications like this.

Being this building block for PaaSes, I guess this is more of a requirement than a feature.

Please let me know if you have any questions or want to point out a mistake I made.

I’m @cirowrc on Twitter,

Have a good one!

finis

Implementing a TCP server in C

Ciro S. Costa — Thu, 22 Feb 2018 00:00:00 +0000

Hey,

I’ve wanted to cover some Linux networking basics, and I felt that going through the exercise of setting up a straightforward TCP server would do good, and writing a guide during that process would certainly force me to go over details that I’d usually skip.

If you’re curious (or just wanted a tutorial) on how to set up a TCP server using Linux and C, this is for you.

The overview

At the very high level, there are two actors: a client and a server.

The server has a know IP address that represents a machine to connect to at some point (this address could be retrieved via DNS) and a known port (that represents the service that we want to talk to in that machine).

At this moment, nothing can happen - TCP has its communication based on connections and without some handshaking and connection establishment, no data can be sent.

So, that’s what happens at the first moment.

Once the connection gets established, it’s then up to the application to decide whether there’s indeed a server component and a client component. They could be peers and synchronize data back and forth, for instance.

For an application protocol like HTTP/1.1 though, there’s a well-defined client-server model, where a client issues requests with specific methods, and a server that processes these requests and gives back results.

Being at the application-level, HTTP simply assumes that there’s a way for the server to send data back to the client and vice-versa. If there’s a channel in which messages can be sent, all good for HTTP.

In case you’re not familiar with these terms or the way the networking layers fit together, make sure you get a copy of Computer Networking: A top-down approach.

It goes “layer-by-layer”, allowing the reader to build the understanding of networking concepts from a high-level to as low level as it gets.

The socket

The interface that allows the interaction between the application layer (e.g., HTTP/1.1) and the transport layer (e.g., TCP) is the socket.

An analogy that I remember from college is the one of a series of houses and its doors:

“Considering that houses are applications in machines, then sockets are their doors: to get a message from one to another, it needs to cross a door of the first house and a door of the second.”

In C, we can access such socket interface via a file descriptor that is returned by the socket(2) syscall.

As this interface is required for any communication to happen (it’s the abstraction presented to us by our TCP/IP implementation under the hood), in my example I started by creating a struct that keeps track of it:

/**
 * Encapsulates the properties of the server.
 */
typedef struct server {
	// file descriptor of the socket in passive
	// mode to wait for connections.
	int listen_fd;
} server_t;

This way we can pass the server_t struct around the functions that depend on that socket.

Our main routine can then be declared:

/**
 * Creates a socket for the server and makes it passive such that
 * we can wait for connections on it later.
 */
int server_listen(server_t* server);


/**
 * Accepts new connections and then prints `Hello World` to
 * them.
 */
int server_accept(server_t* server);


/**
 * Main server routine.
 *
 *      -       instantiates a new server structure that holds the
 *              properties of our server;
 *      -       creates a socket and makes it passive with
 *              `server_listen`;
 *      -       accepts new connections on the server socket.
 *
 */
int
main()
{
	int err = 0;
	server_t server = { 0 };

	err = server_listen(&server);
	if (err) {
		printf("Failed to listen on address 0.0.0.0:%d\n", 
                        PORT);
		return err;
	}

	for (;;) {
		err = server_accept(&server);
		if (err) {
			printf("Failed accepting connection\n");
			return err;
		}
	}

	return 0;
}

Let’s implement each of those methods.

Creating a Socket

To have the socket created, the first thing we do is call the socket(2) syscall specifying the type of communication protocol to be used (TCP, in this case) and the domain in which we’re using it (IPv4).

note.: the domain is relevant because we could be using, e.g., unix sockets to communicate - not internet / network specific.

Looking at socket(2) man page:

SOCKET(2)      Linux Programmer's Manual          SOCKET(2)

NAME
       socket - create an endpoint for communication

SYNOPSIS
       int socket(int domain, int type, int protocol);

DESCRIPTION
       socket()  creates  an  endpoint for communication and 
       returns a file descriptor that refers to that endpoint.

       The file descriptor returned by a successful call will be 
       the lowest-numbered file descriptor  not  currently open 
       for the process.

We can see that if we succeed, we end up with a file descriptor that we can reference later. This file descriptor that we receive is what we can store under server->listen_fd:

	// The `socket(2)` syscall creates an endpoint for communication
	// and returns a file descriptor that refers to that endpoint.
	//
	// It takes three arguments (the last being just to provide greater
	// specificity):
	// -    domain (communication domain)
	//      AF_INET              IPv4 Internet protocols
	//
	// -    type (communication semantics)
	//      SOCK_STREAM          Provides sequenced, reliable,
	//                           two-way, connection-based byte
	//                           streams.
	err = (server->listen_fd = socket(AF_INET, SOCK_STREAM, 0));
	if (err == -1) {
		perror("socket");
		printf("Failed to create socket endpoint\n");
		return err;
	}

To verify that we really end up with a file descriptor right after the socket(2) call, check out lsof (which lists open files):

# Capture the PID of the server process that only
# calls `socket(2)` and the `pause(2)` to wait for
# a signal indefinitely.
SERVER_PROC=$(pgrep server.out)


# List the open files of this process but filter
# out those that do not contain `sock` in the 
# line.
lsof | ag $SERVER_PROC | ag sock
COMMAND    PID    TYPE    NODE  NAME
server.out 8824   sock   34368  protocol: TCP


# Check out what file descriptors are assigned
# to our process by inspecting the `proc` virtual
# filesystem.
ls -lah /proc/$SERVER_PROC/fd
 .
 ..
 0 -> /dev/pts/0
 1 -> /dev/pts/0
 2 -> /dev/pts/0
 3 -> socket:[34368]

While at university I read this great book by W. Richard Stevens. As it’s in its title - it’s all about the sockets networking API in Unix systems.

At this point we have a socket that is both not connected and can’t accept connections, but what if we try to write to this socket?

(extra) Writing to a socket in CLOSED state

Naturally, things break:

        err = (server->listen_fd = socket(AF_INET, SOCK_STREAM, 0));
        // ...

        err = write(server->listen_fd, "hey", 3);
        if (err == -1) {
                perror("write");
                printf("failed to write\n");
                return err;
        }

Compile it and run:

$ ./server.out
$

Nothing shows! That’s because the default behavior of a write to a closed socket is to fail with SIGPIPE, which has the default action of killing thr process (although you can catch the signal and do whatever you want):

strace ./server.out
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
write(3, "hey", 3)                      = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=11008, si_uid=1001} ---
+++ killed by SIGPIPE +++

Looking at the man pages:

man 7 signal
       SIGPIPE  13   Broken pipe: write to pipe with no
                                  readers.


man 2 write
       EPIPE    fd  is  connected  to  a  pipe  or  socket whose 
                reading end is closed.  When this happens the 
                writing process will also receive a SIGPIPE signal.  

                (Thus, the write return value is seen only if 
                the program catches, blocks or ignores this signal.)

Very neat.

Binding the socket to an address

Although we have a socket, this socket isn’t bound to an address yet. This is where another syscall comes: bind(2).

bind(2) takes a socket and a well defined structure that lets you tell it about the address you want to bind the socket to.

For an IPv4 application like we’re creating here, we make use of sockaddr_in which allows you to specify a 32-bit address (the IPv4 address) and a 16-bit number (the port).

	struct sockaddr_in server_addr = { 0 };

	// `sockaddr_in` provides ways of representing a full address
	// composed of an IP address and a port.
	//
	// SIN_FAMILY   address family   AF_INET refers to the address
	//                               family related to internet
	//                               addresses
	//
	// S_ADDR       address (ip) in network byte order (big endian)
	// SIN_PORT     port in network byte order (big endian)
	server_addr.sin_family = AF_INET;

        // INADDR_ANY is a special constant that signalizes "ANY IFACE",
        // i.e., 0.0.0.0.
	server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
	server_addr.sin_port = htons(PORT);

With the sockaddr_in structure set, it’s time to call bind:


	// bind() assigns the address specified to the socket referred 
        // to by the file descriptor (`listen_fd`).
        //
        // Here we cast `sockaddr_in` to `sockaddr` and specify the
        // length such that `bind` can pick the values from the
        // right offsets when interpreting the structure pointed to.
	err = bind(server->listen_fd,
	           (struct sockaddr*)&server_addr,
	           sizeof(server_addr));
	if (err == -1) {
		perror("bind");
		printf("Failed to bind socket to address\n");
		return err;
	}

Once bind is called, now our socket is attached to a specific address, but we still can’t see it on ss or any other program like that, and that’s because our socket is still in the CLOSED state (no connection state at all).

Making the socket listen for connections

As, by default, the socket is created for active connections (acting as a client), we must make it passive using listen(2) if we want to accept connections:

LISTEN(2)      Linux Programmer's Manual    LISTEN(2)

NAME
       listen - listen for connections on a socket

SYNOPSIS
       int listen(int sockfd, int backlog);

DESCRIPTION
       listen()  marks  the socket referred to by sockfd as 
       a passive socket, that is, as a socket that will be 
       used to accept incoming connection requests using 
       accept(2).

With listen(2) we then mark the socket with SO_ACCEPTCON to listen for connections and specify a backlog size - the maximum amount of pending connections that can be enqueued for that socket.

	// listen() marks the socket referred to by sockfd as a passive socket,
	// that is, as a socket that will be used to accept incoming connection
	// requests using accept(2).
	err = listen(server->listen_fd, BACKLOG);
	if (err == -1) {
		perror("listen");
		printf("Failed to put socket in passive mode\n");
		return err;
	}

Calling this syscall also has the effect of moving our socket to a different state.

Instead of CLOSED, now the socket is put into LISTEN state (represents waiting for a connection request from any remote TCP and port - RFC 793 - TCP):

# Check that `lsof` identifies our socket listening on
# port 8080
lsof -i:8080
COMMAND    TYPE DEVICE NAME
server.out IPv4 38593  TCP *:http-alt (LISTEN)

# Check that `ss` properly recognizes our socket as `listen`
ss \
        --listening \
        --numeric \
        --tcp  | ag 8080
LISTEN     0      128          *:8080                     *:*

We can also look at /proc/net/tcp to see exactly the same as what the commands above are seeing (i.e., going under the hood):

# Get the inode of our socket.
#
# Given that `socket(2)` creates a file descriptor
# that is the next fd in the sequence (0, 1, and 2
# are already taken), we know that `3` references
# our socket.
 SERVER_SOCKET=stat \
        --dereference \
        --printf %i \
        /proc/$SERVER_PROC/fd/3

# Check the state of our socket inode in the
# /proc/net/tcp table:
cat /proc/net/tcp | ag $SERVER_SOCKET
  sl  local_address rem_address   st  
   1: 00000000:1F90 00000000:0000 0A   ...

st defines the state of the socket, so we can know what 0A (10 in decimal) means by looking at /usr/src/linux-headers-<LINUX_VER>/include/net/tcp_states.h:

enum {
	TCP_ESTABLISHED = 1,
	TCP_SYN_SENT,                   // 02
	TCP_SYN_RECV,                   // 03
	TCP_FIN_WAIT1,                  // ...
	TCP_FIN_WAIT2,                  
	TCP_TIME_WAIT,
	TCP_CLOSE,
	TCP_CLOSE_WAIT,                 // ...
	TCP_LAST_ACK,                   // 09
	TCP_LISTEN,                     // 0A  <<<<<<<<<
	TCP_CLOSING,
	TCP_NEW_SYN_RECV,

	TCP_MAX_STATES
};

As we wanted, it’s in the TCP_LISTEN state.

To know more about /proc and other Linux interfaces, make sure you check the book The Linux Programming Interface.

(extra) Messing up with the backlog

As at this point we’re able to listen for income connections, but we don’t accept them, what would happen if people started trying to connect to our server?

One easy way of simulating the backlog exhaustion is by putting our server up and then start creating connections to it.

With a call to pause(2) right after listen(2) we’ll start enqueuing connections without closing them, so if we create a bunch of telnet clients that connect to our server, we should see them failing after N connections (where N equals the size of the backlog).

# In one terminal, start the server.
# ps.: I modified the code to have the
# backlog set to 4.
./server.out

# In another terminal, start a handful of 
# connections using telnet.

for i in $(seq 1 10); \
        do sleep 1; \
        telnet localhost 8080 & \
done

[1] 11696                       # <<<<< worked
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

telnet localhost 8080
[2] 11698                       # <<<<< worked
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

telnet localhost 8080
[3] 11700                       # <<<<< worked
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

telnet localhost 8080
[4] 11702                       # <<<<< worked
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

telnet localhost 8080
[5] 11704                       # <<<<< worked
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.


# From now on, the backlog got to its maximum, 
# so it won't accept more connections.

telnet localhost 8080
[6] 11707
Trying 127.0.0.1...
[7] 11709
Trying 127.0.0.1...
[8] 11711
Trying 127.0.0.1...
[9] 11713
Trying 127.0.0.1...
[10] 11715
Trying 127.0.0.1...

What that means is that we can directly affect that rate at which connections get processed by our server without even accepting them in the first place.

Accepting connections

Once a connection lands on the server that has a passive socket for that destination address, it’s up to the application to decide to do something with does connections or not.

As the connections are being established, they float around two queues that are limited by the backlog value:

an incomplete connection queue: keeps track of connections that just arrived and didn’t finish the three-way handshake;
a completed connection queue: keeps track of connections that successfully finished the three-way handshake and can be utilized by the server.

What accept(2) ends up doing is looking at the completed connections first in first out (FIFO) queue and popping from it, giving to the application a file descriptor that represents that connection (such that it can send data or whatever). If the queue is empty, it just blocks.

ACCEPT(2)    Linux Programmer's Manual ACCEPT(2)

NAME
       accept, accept4 - accept a connection on a socket

SYNOPSIS
       int accept(int sockfd, 
                  struct sockaddr *addr, 
                  socklen_t *addrlen);

DESCRIPTION
       The  accept()  system  call  is  used  with  connection-based 
       socket types (SOCK_STREAM, SOCK_SEQPACKET).  
       
       It extracts the first connection request on the queue of pending 
       connections for the listening  socket,  sockfd, creates  a  new 
       connected socket, and returns a new file descriptor referring to 
       that socket.  
       
       The newly created socket is not in the listening state.  
       The original socket sockfd is unaffected by this call.

       The argument sockfd is a socket that has been created with socket(2), 
       bound to a local address with  bind(2), and is listening for connections 
       after a listen(2).

With that said, the code:

int
server_accept(server_t* server)
{
	int err = 0;
	int conn_fd;
	socklen_t client_len;
	struct sockaddr_in client_addr;

	client_len = sizeof(client_addr);

	err =
	  (conn_fd = accept(
	     server->listen_fd, (struct sockaddr*)&client_addr, &client_len));
	if (err == -1) {
		perror("accept");
		printf("failed accepting connection\n");
		return err;
	}

	printf("Client connected!\n");

	err = close(conn_fd);
	if (err == -1) {
		perror("close");
		printf("failed to close connection\n");
		return err;
	}

	return err;
}

And that’s it!

Now our server can adequately receive connections and act upon them if we want.

Closing thoughts

I found that going through this rather simple example was great - I took the time to review some concepts and make the whole flow even more concrete in my mind.

If you’re looking for the implementation, make sure you check github.com/cirocosta/hstatic.

I plan to make this a simple HTTP server that serves static files, but that’ll come next.

Just in case I got something wrong or you just want to chat about something related, get in touch with me on Twitter! I’m @cirowrc there.

Have a good one!

finis

Resources

Throughout the article, some books were mentioned.

Here’s the list of them:

Using C to inspect Linux syscalls

Ciro S. Costa — Wed, 21 Feb 2018 00:00:00 +0000

Hey,

I came through a great article this week (Write yourself a strace in 70 lines of code) and decided to give a try, going through every step with a bunch of care such that I’d understand each piece.

While writing the code, I came up with the following schematics of how everything works together:

Let’s dive into it.

Executing a new process with tracing enabled

By controlling the execution of a new process (being its parent) we can set this new process to have tracing enabled.

This is possible by how the creation of new processes work in Linux: fork from the parent (in which case there’s going to exist two very similar processes - a clone is made), then perform some actions and switch the process image to a new one.

This means that in our main execution we end up with something in the following lines:

/**
 * Main execution - parses CLI arguments forming the runtime_t struct
 * and then starts the tracer by forking the execution and then
 * starting the main routine of the child (tracee) as well as the
 * tracing capabilities on the parent.
 */
int
main(int argc, char** argv)
{
	setbuf(stdout, NULL);

	if (argc < 2) {
		fprintf(stderr, "usage: %s prog args\n", argv[0]);
		return 1;
	}

        // Initialize the `runtime_t`, a custom structure
        // that takes care of parsing the CLI arguments.
        //
        // Check the repository to know more.
	runtime_t runtime = { 0 };
	runtime_init(&runtime, argc, argv);
	runtime_show(&runtime);

	// Create a new process such that we can spawn a new
	// execution (via `execvp`) via a process that has already
	// declared that it wants to be traced (PTRACE_TRACEME).
	//
	// In the other process (parent), we implement the tracing
	// capabilities that will inspect the child's registers and
	// interrupt its executions.
	pid_t child = fork();
	switch (child) {
		case 0:
			do_child(&runtime);
			break;
		case -1:
			perror("fork");
			fprintf(stderr, "failed to fork process");
			exit(1);
			break;
		default:
			do_trace(child);
			break;
	}

	runtime_destroy(&runtime);

	return 0;
}

Having do_child execute the child functionality and do_trace the tracer (parent), we can see how the child enables tracing and then signals itself a SIGSTOP to stop its execution:

/**
 * Main execution of the child process after
 * forking from the parent.
 *
 * Its `argc` and `argv` are a subset of the parent's
 * `argv` and `argv`, thus containing only the subprocess
 * arguments (e.g.: tracer myprog arg1 --> myprog arg1).
 */
int
do_child(runtime_t* runtime)
{
	int err = 0;

	// Explicitly tell that we want to get traced
	err = ptrace(PTRACE_TRACEME);
	if (err == -1) {
		return err;
	}

	// As a child that gets traced ends up with all the
	// signals send to it being sent to the parent, signalling
	// itself will signal the parent.
	//
	// To inform the parent then, we send a signal to
	// outselves which will also stop us.
	err = kill(getpid(), SIGSTOP);
	if (err == -1) {
		return err;
	}

	// ...
}

Meanwhile, the parent waits for the child to change its process state (to go from RUNNING to STOPPED or vice-versa) using waitpid.

Once the child process gets to the paused state (i.e., once the child raises SIGSTOP to itself), the parent can then resume execution of its main routine to set some ptrace options before it tells the child to wake up and continue its main routine.


/**
 * Main execution of the tracer.
 */
int
do_trace(pid_t child)
{
	int err = 0;
	int status;
	int syscall;
	int retval;

	// Wait until the child starts its main execution
	err = waitpid(child, &status, 0);
	if (err == -1) {
		return err;
	}

	// Set some options in regards to the child process.
	//
	// Here we also set ptrace option: PTRACE_O_TRACESYSGOOD.
	//
	// This option forces the kernel to set bit 7 in the signal
	// number such that we can know if that was a syscall trap
	// that we caught.
	err = ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD);
	if (err == -1) {
		return err;
	}

        // ...
}

At this moment the child is set to be traced and the parent can allow the child to resume its execution.

Catching a child’s syscalls

Once the parent finished setting the options needed to properly analyze the child’s behavior, it needs to tell it (the child) to start running again, and then wait on the child’s PID so that it can be notified of the syscall execution once it happens (in the child).

It does so by making use of ptrace(2) with the flag PTRACE_SYSCALL set and then using waitpid to wait for the syscall execution in the child (whenever the child executes a syscall it’ll get stopped - changing the process state - which gets noticed by the parent).

Regarding code, we can wrap the logic in a function called wait_for_sycall which does what’s been described and also checks some information carried with the signal that tells the tracer what happened to the child:

/**
 * Keeps the child running until either entry to
 * or exit from a system call.
 */
int
wait_for_syscall(pid_t child)
{
	int status;
	int err = 0;

	while (1) {
		// Calling ptrace with PTRACE_SYSCALL makes
		// the tracee (child) continue its execution
		// and stop whenever there's a syscall being
		// executed (SIGTRAP is captured).
		err = ptrace(PTRACE_SYSCALL, child, 0, 0);
		if (err == -1) {
			return err;
		}

		// Wait until the next signal arrives
		// When the running tracee enters ptrace-stop, it
		// notifies its tracer using waitpid(2)
		// (or one of the other "wait" system calls).
		waitpid(child, &status, 0);

		// Ptrace-stopped tracees are reported as returns
		// with pid greater than 0 and WIFSTOPPED(status) true.
		//
		// -    WIFSTOPPED(status) returns true if the child
		//      process was stopped by delivery of a signal.
		//
		// -    WSTOPSIG(status) returns the number of the signal
		//      which caused the child to stop - should only be
		//      employed if WIFSTOPPED returned true.
		//
		//      by `and`ing with `0x80` we make sure that it was
		//      a stop due to the execution of a syscall (given
		//      that we set the PTRACE_O_TRACESYSGOOD option)
		if (WIFSTOPPED(status) && WSTOPSIG(status) & 0x80) {
			return 0;
		}

		// Check whether the child exited normally.
		if (WIFEXITED(status)) {
			return 1;
		}
	}

	return 0;
}

Then, to catch every syscall made in the child, just use it in a loop:

	while (1) {
		err = wait_for_syscall(child);
		if (err != 0) {
			break;
		}

		syscall =
		  ptrace(PTRACE_PEEKUSER, child, sizeof(long) * ORIG_RAX);
		fprintf(stderr, "syscall(%d) = ", syscall);

		err = wait_for_syscall(child);
		if (err != 0) {
			break;
		}

		retval = ptrace(PTRACE_PEEKUSER, child, sizeof(long) * RAX);
		fprintf(stderr, "%d\n", retval);
	}

ps.: we use wait_for_syscall twice as we’re being notified regarding the syscall enter (start of the syscall execution) and the syscall exit (termination of the syscall).

That’s it!

Looking up syscalls by number

To test if this is indeed working, I created a simple program that just writes to stdout:

#include <unistd.h>

int
main()
{
	return write(1, "hello world", 11);
}

Nothing fancy, just a write(2) syscall execution.

Let’s run it then:

# Compile the code we just created
# with `-static` to produce a static binary.
gcc main.c -O2 -static -o main.out

# Compile the test case (the program that writes
# to stdout
gcc case.c -O2 -static -o case.out

# Run the tracer with `case.out` as the tracee
./main.out ./case.out

syscall(59) = 0
syscall(12) = 31203328
syscall(12) = 31207872
syscall(158) = 0
syscall(63) = 0
syscall(89) = 47
syscall(12) = 31343040
syscall(12) = 31346688
syscall(21) = -2
syscall(1) = hello world11
syscall(231) =

As we wanted, it prints syscall followed by the number and then after the syscall exits, the exit number from it.

// Pick the syscall number by inspecting
// the values at the tracee's registers
// that were set at the moment that the
// user program was about to perform the 
// syscall.
syscall = ptrace(PTRACE_PEEKUSER, child, sizeof(long) * ORIG_RAX);
fprintf(stderr, "syscall(%d) = ", syscall);

wait_for_syscall(child);

// Pick the syscall number by inspecting
// the values at the tracee's registers
// once it returned from the syscall
retval = ptrace(PTRACE_PEEKUSER, child, sizeof(long) * RAX);
fprintf(stderr, "%d\n", retval);

Those numbers that we see represent the actual syscalls that were performed.

We can perform a reverse lookup to determine what each number is by looking at /usr/include/x86_64-linux-gnu/asm/unistd_64.h:

#define __NR_execve 59
#define __NR_brk 12
#define __NR_arch_prctl 158
#define __NR_uname 63
#define __NR_readlink 89
#define __NR_access 21
#define __NR_write 1
#define __NR_exit_group 231

meaning that our program started by executing execve (to get ./case.out program running), then it did some glibc stuff and then ended with write, printing our message to stdout.

Closing thoughts

It seems very useful for me to know that we can fully control the execution of the syscalls that our program performs, eventually injecting faults and tampering results. This seems very useful in the context of chaos engineering.

One downside of the way we did here is that we’re stoping for every single syscall, which might considerably slow down an application. To counterfeit that, I plan to explore the use of SECCOMP together with ptrace which would allow us to get in the middle of specific syscalls based on a filter that we provide.

I’m very grateful for Nelson who put Write yourself an strace in 70 lines of code up. I had a bunch of joy following that article and learning from it. Thanks!

Please let me know if you have any questions or spot something odd. I’d appreciate!

Have a good one,

finis

How to set up AWS EFS across multiple availability zones using Terraform

Ciro S. Costa — Mon, 19 Feb 2018 00:00:00 +0000

Hey,

this is a quick follow up for the article I wrote about the locks quota of AWS EFS (The first limit you’ll hit on AWS EFS: Locks) and an article about settings up AWS networking with Terraform (A practical look at basic AWS Networking with Terraform).

Here I take the concept of creating multiple subnets in a VPC explored in the second article and then tie with the AWS EFS provisioning tips from the first one.

The architecture

As mentioned in the first article, in AWS EFS you have a file system that is managed by Amazon (per-region) that you can then mount in subnets in a VPC to make use of the filesystem.

Having the networking set up, it’s a matter of iterating over the subnets and creating the mount points.

Let’s check out how that looks like in Terraform terms.

Creating a multi-az AWS EFS set up with Terraform

First things first, get your VPC ready with at least one subnet for each availability zone you want to cover. In my case, I made use of a module I’ve written before (see how I set up networking with Terraform):

# Explicitly maps subnet names to CIDR and
# availability zone that subnets should live.
#
# This allows us to call the `networking` module
# once and have all the subnet creation taken
# care of.
#
# By giving a name to each subnet we're able to
# get the generated subnet ID after the provisioning
# is done by looking up for the ID in a map that
# comes as the output of the module.
variable "az-subnet-mapping" {
  type = "list"

  default = [
    {
      name = "us-east-1a"
      az   = "us-east-1a"
      cidr = "10.0.0.0/24"
    },
    {
      name = "us-east-1b"
      az   = "us-east-1b"
      cidr = "10.0.1.0/24"
    },
  ]
}

# Make use of the `networking` module as defined in 
# the `./networking` directory of the `cirocosta/sample-aws-networking`
# repository.
module "networking" {
  source = "github.com/cirocosta/sample-aws-networking//networking"
  cidr   = "10.0.0.0/16"

  "az-subnet-mapping" = "${var.az-subnet-mapping}"
}

That gives us two availability zones covered by two subnets (one for each).

With the subnets ready, I make use of a custom efs module (the one we’ll create below):

module "efs" {
  source = "./efs"

  name          = "shared-fs"
  subnets-count = "${length(var.az-subnet-mapping)}"
  subnets       = "${values(module.networking.az-subnet-id-mapping)}"
  vpc-id        = "${module.networking.vpc-id}"
}

Having the expectations set, let’s write the module.

Creating the AWS EFS Terraform module

The way we used the EFS module above made explicit what’s the interface of it:

it takes a name that describes what our elastic file system is about - this can be used for tagging the resource and naming the EFS;
it takes a list of subnets (and a count because of some Terraform problems with using count together with a dynamic variable) - these are the subnets that the AWS EFS will mount to;
a vpc id that allows us to retrieve information about the VPC that the subnets belong to and then pick the CIDR to allow traffic from and to.

With that deliniated, I start shaping the module file structure:

.
├── inputs.tf   # declares the variables that serve as input
├── main.tf     # defines the main resources and data sources
└── outputs.tf  # defines the outputs that can be used by other
                # invocations

0 directories, 3 files

The inputs.tf is fairly simple, so I’ll skip (you can look it here: cirocosta/aws-efs-sample).

The main.tf looks like this:

# Gathers information about the VPC that was provided
# such that we can know what CIDR block to allow requests
# from and to the FS.
data "aws_vpc" "main" {
  id = "${var.vpc-id}"
}

# Creates a new empty file system in EFS.
#
# Although we're not specifying a VPC_ID here, we can't have
# a EFS assigned to subnets in multiple VPCs.
#
# If we wanted to mount in a differente VPC we'd need to first
# remove all the mount points in subnets of one VPC and only 
# then create the new mountpoints in the other VPC.
resource "aws_efs_file_system" "main" {
  tags {
    Name = "${var.name}"
  }
}

# Creates a mount target of EFS in a specified subnet
# such that our instances can connect to it.
#
# Here we iterate over `subnets-count` which indicates
# the length of the `var.subnets` list.
#
# This way we're able to create a mount target for each
# of the subnets, making it available to instances in all
# of the desired subnets.
resource "aws_efs_mount_target" "main" {
  count = "${var.subnets-count}"

  file_system_id = "${aws_efs_file_system.main.id}"
  subnet_id      = "${element(var.subnets, count.index)}"

  security_groups = [
    "${aws_security_group.efs.id}",
  ]
}

# Allow both ingress and egress for port 2049 (NFS)
# such that our instances are able to get to the mount
# target in the AZ.
#
# Additionaly, we set the `cidr_blocks` that are allowed
# such that we restrict the traffic to machines that are
# within the VPC (and not outside).
resource "aws_security_group" "efs" {
  name        = "efs-mnt"
  description = "Allows NFS traffic from instances within the VPC."
  vpc_id      = "${var.vpc-id}"

  ingress {
    from_port = 2049
    to_port   = 2049
    protocol  = "tcp"

    cidr_blocks = [
      "${data.aws_vpc.main.cidr_block}",
    ]
  }

  egress {
    from_port = 2049
    to_port   = 2049
    protocol  = "tcp"

    cidr_blocks = [
      "${data.aws_vpc.main.cidr_block}",
    ]
  }

  tags {
    Name = "allow_nfs-ec2"
  }
}

Once the AWS EFS is properly created, we then care about one thing: the address available for us to perform the NFSv4.1 mounting.

One gotcha is that the VPC must have DNS hostname resolution set.

output "mount-target-dns" {
  description = "Address of the mount target provisioned."
  value       = "${aws_efs_mount_target.main.0.dns_name}"
}

You might notice that I’m taking the address of the first mount target without worrying about the subnet and availability zone that the address comes from. That’s because it really doesn’t matter - in each subnet the DNS resolution will be performed correctly, returning the address of the mountpoint in that availability zone.

[
    {
        "dns_name": "fs-4b698303.efs.us-east-1.amazonaws.com",
        "file_system_id": "fs-4b698303",
        "id": "fsmt-7011eb38",
        "ip_address": "10.0.0.238",
        "network_interface_id": "eni-b0388b7c",
        "security_groups.#": "1",
        "security_groups.4275814226": "sg-2a644a5d",
        "subnet_id": "subnet-8b4d19a4"
    },
    {
        "dns_name": "fs-4b698303.efs.us-east-1.amazonaws.com",
        "file_system_id": "fs-4b698303",
        "id": "fsmt-7111eb39",
        "ip_address": "10.0.1.160",
        "network_interface_id": "eni-78127885",
        "security_groups.#": "1",
        "security_groups.4275814226": "sg-2a644a5d",
        "subnet_id": "subnet-9b2935d0"
    }
]

See? Different IP addresses, subnets and network interfaces, but same address.

Making use of AWS EFS in an EC2 instance

Let’s start by declaring an instance (it could be an autoscaling group with a launch configuration that carries the address of the AWS EFS mount point in that VPC):

# Create an EC2 instance that will interact with an EFS
# file system that is mounted in our specific availability
# zone.
resource "aws_instance" "inst1" {
  ami               = "${data.aws_ami.ubuntu.id}"
  instance_type     = "t2.micro"
  key_name          = "${aws_key_pair.main.key_name}"
  availability_zone = "us-east-1a"
  subnet_id         = "${module.networking.az-subnet-id-mapping["us-east-1a"]}"

  vpc_security_group_ids = [
    "${aws_security_group.allow-ssh-and-egress.id}",
  ]

  tags {
    Name = "inst1"
  }
}

Once that instance gets up, SSH into it:

# Lower the permissions of the key 
chmod 400 ./keys/key.rsa

# Get into the instance by making use of
# the private key that corresponds to the 
# private side of the key-pair that we generated
# (the instance has the public key in its
# ~/.ssh/authorized_keys).
ssh -i ./keys/key.rsa ubuntu@inst1

Once you’re there, let’s mount EFS:

# The mount location refers to the destination
# in the file system where EFS will be mounted
# to.
MOUNT_LOCATION="/mnt/efs"

# Mount target is the target of the mount - that
# address that we received from the EFS invocation.
# 
# Using the EFS module we just created, you could 
# get this address by interpolation:
#
#       ${module.efs.mount-target-dns}
#
MOUNT_TARGET="fs-4b698303.efs.us-east-1.amazonaws.com"

# Retrieve the necessary packages for `mount` to work
# properly with NFSv4.1
sudo apt update -y
sudo apt install -y nfs-common

# Create the directory that will hold the mount
sudo mkdir -p $MOUNT_LOCATION

# Mount the EFS mount point as a NFSv4.1 fs
sudo mount \
    -t nfs4 \
    -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 \
    $MOUNT_TARGET:/ $MOUNT_LOCATION

# Check whether it has been successfuly mounted or not:
df -h | grep efs
fs-8d6d87c5.efs.us-east-1.amazonaws.com:/  8.0E     0  8.0E   0% /mnt/efs

mount | grep nfs
fs-8d6d87c5.efs.us-east-1.amazonaws.com:/ on 
        /mnt/efs type nfs4 
        (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,
                hard,proto=tcp,timeo=600,retrans=2,sec=sys,
                clientaddr=10.0.0.249,local_lock=none,
                addr=10.0.0.140)

Closing thoughts

Mounting an AWS EFS file system into multiple availability zones is not complicated. It has some details here and there but the overall experience is very straightforward.

If you don’t forget to have DNS support for your VPC and also have a way of getting the DNS address of the EFS mount, you’re good to go.

In case you have any questions, please let me know!

I’m @cirowrc.

Have a good one!

finis

A practical look at basic AWS Networking with Terraform

Ciro S. Costa — Sun, 18 Feb 2018 00:00:00 +0000

Hey,

I noticed that recently a bunch of people has been getting to the website through queries about HAProxy in the context of AWS - mainly how to receive traffic through a set of instances and forward them to another set of machines.

As I aim at making this blog as much practical as I can, I feel that if I taught the basics around AWS networking first, then I could have the ground set for further discussions about either HAProxy, NGINX or other options out there.

The client point of view

In the point of view of the client, be it AWS or anywhere else, the lifecycle is the same:

it makes requests to an endpoint (under a given domain that resolves to an assigned IP);
somehow someone processes its request;
it receives a response back.

That endpoint that takes the request from the client and then calls the application magic behind the scenes could either be a Content Delivery Network (CDN) point of presence (PoP) that would forward the request to another region or be a machine that we own.

For simplicity, let’s assume it’s one of our machines.

The components

A very common scenario where many AWS networking concepts can be exercised is the following:

three machines are meant not to be touched directly by other applications on the internet;
each of those machines (srv1, srv2 and srv3) have application services (maybe just the same application but spread in three machines) which are meant to be accessed publicly via a well-defined URL;
lb (our load-balancer machine) has a public interface that is meant to receive traffic from the internet and another interface that can reach the machines in the internal network - this is the machine that should be touched when traffic comes to api.cirocosta.com.

To have these machines communicating with each other, they’re put into a common virtual network that we can define - in AWS terms, this big network is the AWS VPC (Virtual Private Cloud).

These virtual networks (you can have more than your if you wish) are totally separate from each other (so you can have many networks using the same IP address range, for instance) and can be created in a minute.

Another property of them is that they naturally span across multiple availability zones in a given region.

When it comes to provisioning EC2 instances, only having a VPC is not enough. From that big range of IPs that we allocated for our network, it’s needed that the user configures subnets too.

By separating the VPC in subnets we’re able to partition that network across the availability zones and configure further properties, including whether it should:

be totally private (i.e, have no default internet gateway assigned to it);
assign public IPs (even though ephemeral) to the instances when they get into the subnet;
assign an IPv6 address to the instance.

I usually think of subnets as a further specification of details about the network that I want to put instances to.

In the end, we usually have something like this:

With the ground set, let’s terraform it!

Creating an AWS VPC using Terraform

The first resource I start creating is aws_vpc:

# The VPC that spans across multiple availability zones.
#
# Given the CIDR 10.0.0.0/16, we can have IPs from 10.0.0.1
# up to 10.0.255.254. Essentially we can host 65k IPs in
# that range. 
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

The VPC by itself gives us nothing more than that cidr block that we specified. Creating this resource (terraform apply) we can see it in describe-vpcs:

aws ec2 describe-vpcs \
        --profile=$PROFILE

{
    "Vpcs": [
        {
            "CidrBlock": "10.0.0.0/16",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-896491e1",
                    "CidrBlock": "10.0.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "DhcpOptionsId": "dopt-a20007c7",
            "InstanceTenancy": "default",
            "IsDefault": false,
            "State": "available",
            "VpcId": "vpc-76d00d11"
        }
    ]
}

Something I didn’t mention before is that there’s an extra resource that gets attached to the VPC when you create it: a routing table.

aws ec2 describe-route-tables \
        --profile=$PROFILE

{
    "RouteTables": [
        {
            "Associations": [
                {
                    "Main": true,
                    "RouteTableAssociationId": "rtbassoc-0f885c69",
                    "RouteTableId": "rtb-852df2e2"
                }
            ],
            "PropagatingVgws": [],
            "RouteTableId": "rtb-852df2e2",
            "Routes": [
                {
                    "DestinationCidrBlock": "10.0.0.0/16",
                    "GatewayId": "local",
                    "Origin": "CreateRouteTable",
                    "State": "active"
                }
            ],
            "Tags": [],
            "VpcId": "vpc-76d00d11"
        }
    ]
}

Each VPC created carries with it an implicit router, coming with a main route table that we can modify. By default, it only comes with a route that determines where traffic should go when packets are sent to internal instances.

This might look like not much, but it’s enough to allow us to create subnets, add instances to them and have these instances communicating with each other. Nevertheless, if we want to download software from the internet to update our instances, we must be able to get traffic sent out to the internet as well.

To do so, an internet gateway is required.

The internet gateway is a piece of infrastructure that can be seen as a router that will take packets from instances inside the network and forward them to the outside (as well as receive back packets from the established connection).

The instances that are created within subnets within this VPC use such component as a target in its internal route table and then when it establishes a connection to an outside resource, the gateway takes care of setting up that connection.

Fortunately, we don’t need to take care of this component - you only attach it to your VPC and then AWS takes care of scaling, making it redundant and highly available to all the instances within the VPC.

Let’s now create this internet gateway and add the route to our main route using Terraform then.

Creating an Internet Gateway using Terraform

To grant internet access to the VPC we make use of the aws_internet_gateway resource and aws_route:

# Internet gateway to give our VPC access to the outside world
resource "aws_internet_gateway" "default" {
  vpc_id = "${aws_vpc.default.id}"
}

# Grant the VPC internet access by creating a very generic
# destination CIDR ("catch all" - the least specific possible) 
# such that we route traffic to outside as a last resource for 
# any route that the table doesn't know about.
resource "aws_route" "internet_access" {
  route_table_id         = "${aws_vpc.main.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.main.id}"
}

Once the creation finishes, check out the route table:

aws ec2 describe-route-tables \
        --profile=$PROFILE

 {
     "RouteTables": [
         {
             "Associations": [
                 {
                     "Main": true,
                     "RouteTableAssociationId": "rtbassoc-0f885c69",
                     "RouteTableId": "rtb-852df2e2"
                 }
             ],
             "PropagatingVgws": [],
             "RouteTableId": "rtb-852df2e2",
             "Routes": [
                 {
                     "DestinationCidrBlock": "10.0.0.0/16",
                     "GatewayId": "local",
                     "Origin": "CreateRouteTable",
                     "State": "active"
-                }
+                },
+                {
+                    "DestinationCidrBlock": "0.0.0.0/0",
+                    "GatewayId": "igw-caa366ae",
+                    "Origin": "CreateRoute",
+                    "State": "active"
+                }
             ],
             "Tags": [],
             "VpcId": "vpc-76d00d11"
         }
     ]
 }

It’s there!

Creating AWS VPC Subnets

Differently from a VPC, each subnet is assigned to a specific availability zone, but each availability zone can have more than one subnet (belonging to the same VPC) though.

This means that if we want to cover two availability zones with instances, we must create two subnets at least.

With Terraform, I see that the most elegant way of doing so is by using the aws_subnet resource paired with a list of maps:

# Creates N subnets according to the subnet mapping described in
# the `az-subnet-mapping` variable.
#
# The variable is a list of maps in the following form:
#
#   [  { name: "crazydog", az: "name-of-the-az", cidr: "cidr-range" } , ... ]
#
# For instance:
#
#   [ { name =  "sub1", az = "us-east-1a", cidr = "192.168.0.0/24"  } ]
#
resource "aws_subnet" "main" {
  count = "${length(var.az-subnet-mapping)}"

  cidr_block              = "${lookup(var.az-subnet-mapping[count.index], "cidr")}"
  vpc_id                  = "${aws_vpc.main.id}"
  map_public_ip_on_launch = true
  availability_zone       = "${lookup(var.az-subnet-mapping[count.index], "az")}"

  tags = {
    Name = "${lookup(var.az-subnet-mapping[count.index], "name")}"
  }
}

This way we’re able to be very explicit about which subnets we want in each region and what CIDR they take (we can even vary the size of these subnets without relying on some weird CIDR math).

To make use of it, just declare the variable:


variable "az-subnet-mapping" {
  type        = "list"
  description = "Lists the subnets to be created in their respective AZ."

  default = [
    {
      name = "subnet1"
      az   = "sa-east-1a"
      cidr = "10.0.0.0/24"
    },
    {
      name = "subnet1"
      az   = "sa-east-1c"
      cidr = "10.0.1.0/24"
    },
  ]
}

and then, after applying it, see what subnets to we have around:

aws ec2 describe-subnets \
        --profile=beld

{
    "Subnets": [
        {
            "AssignIpv6AddressOnCreation": false,
            "AvailabilityZone": "sa-east-1a",
            "AvailableIpAddressCount": 251,
            "CidrBlock": "10.0.0.0/24",
            "DefaultForAz": false,
            "Ipv6CidrBlockAssociationSet": [],
            "MapPublicIpOnLaunch": true,
            "State": "available",
            "SubnetId": "subnet-936734f4",
            "VpcId": "vpc-76d00d11"
        },
        {
            "AssignIpv6AddressOnCreation": false,
            "AvailabilityZone": "sa-east-1c",
            "AvailableIpAddressCount": 251,
            "CidrBlock": "10.0.1.0/24",
            "DefaultForAz": false,
            "Ipv6CidrBlockAssociationSet": [],
            "MapPublicIpOnLaunch": true,
            "State": "available",
            "SubnetId": "subnet-b20122ea",
            "VpcId": "vpc-76d00d11"
        }
    ]
}

There we go! With the subnets set, we can start creating our instances. Nevertheless, we can take some time to refactor our code a bit.

Refactoring - making networking a Terraform module

So far all the code has been thrown out to a main.tf.

That’s perfect if you’re dealing with just a pair or two of resources. However, when the infrastructure starts growing, it becomes harder and harder to maintain.

Here we face one of those situations. We’re passing our subnet definitions via a list of maps, but after the subnets get created, it becomes hard for us to retrieve the subnet IDs without passing this complexity to the resource that will need to pick that ID.

One way of making that whole process easier is to abstract the whole networking set up into a Terraform module.

To do so, I started by picking all those resources related to networking and put them in a file in a different directory called networking. The file structure looks like this:

.
├── keys                # contains pub and private keys (for this example)
│   ├── key.rsa 
│   └── key.rsa.pub
├── keys.tf             # manages `key` resource
├── main.tf             # top-level tf file - configures providers and vars
├── networking          # networking moduel - sets the whole networking
│   │ 
│   ├── inputs.tf       # declares the variables taken as input
│   ├── main.tf         # main resource creation (vpc, igw, routes, subnets ...)
│   └── outputs.tf      # results from the execution of the module
│     
└── terraform.tfstate

The rationale behind setting this kind of structure is that we can think of the Terraform modules as functions that take some inputs, creates some resources and then produces some outputs.

Accessing generated subnet IDs via maps using Terraform

Once the subnets have been generated, we’ll have their IDs which we can then use to attach instances to (or set as the based subnet to have an autoscaling group placing instances to).

The problem is that without doing some interpolation trick, we can’t access those IDs by name.

Once aws_subnet is executed with the count set, we end up with a plain list of aws_subnet resources.

Given that at this point we already separated our networking configuration from the rest, at output.tf we can make use of zipmap to have access to those subnet IDs using plain key-value lookup:

# Creates a mapping between subnet name and generated subnet ID.
#
# Given an `aws_subnet` resource created with `count`, we can access
# properties from the list of resources created as a list by using
# the wildcard syntax:
# 
#     <resource>.*.<property>
#
# By making use of `zipmap` we can take two lists and create a map
# that uses the values from the first list as keys and the values
# from the seconds list as values.
#
# Example output:
#
#     az-subnet-id-mapping = {
#       subnet1 = subnet-3b92c15c
#       subnet2 = subnet-a90023f1
#     }
#
output "az-subnet-id-mapping" {
  value = "${zipmap(aws_subnet.main.*.tags.Name, aws_subnet.main.*.id)}"
}

Having easy access to the subnet IDs, it’s time to create the instances.

Creating EC2 Instances in AWS Subnets

As we need an AMI to create the instances, we start by picking the latest one released by Canonical.

# Pick the latest ubuntu artful (17.10) ami released by the
# Canonical team.
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-artful-17.10-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  # 099720109477 is the id of the Canonical account
  # that releases the official AMIs.
  owners = ["099720109477"]
}

Usually, you’d pick an AMI that you prepared yourself already containing the software needed to run your application (maybe it’s just docker and some agent to send metrics / whatever), but for testing, picking a simple base Ubuntu image suffices.

With the AMI in hands, let’s wrap the whole thing with all that’s left: making use of the networking module that we specified, set the desired subnets and specify some instances:

variable "profile" {
  description = "Profile with permissions to provision the AWS resources."
}

variable "region" {
  description = "Region to provision the resources into."
  default     = "sa-east-1"
}

provider "aws" {
  region  = "${var.region}"
  profile = "${var.profile}"
}

# By specifying the submodule here we'll have
# as an end result 2 subnets created in those
# two availability zones.
#
# Given that we defined the module's output, 
# wherever we interpolate with that output,
# terraform will establish the dependency and
# make sure that the subnets have been properly
# created.
module "networking" {
  source = "./networking"
  cidr   = "10.0.0.0/16"

  "az-subnet-mapping" = [
    {
      name = "subnet1"
      az   = "sa-east-1a"
      cidr = "10.0.0.0/24"
    },
    {
      name = "subnet2"
      az   = "sa-east-1c"
      cidr = "10.0.1.0/24"
    },
  ]
}

# Create the first instance in `subnet1`.
#
# Here we get the subnet ID to put this instance
# by performing a simple key-value lookup over
# the map that we created in the output of the
# networking module.
resource "aws_instance" "inst1" {
  instance_type = "t2.micro"
  ami           = "${data.aws_ami.ubuntu.id}"
  key_name      = "${aws_key_pair.main.id}"
  subnet_id     = "${module.networking.az-subnet-id-mapping["subnet1"]}"
}

resource "aws_instance" "inst2" {
  instance_type = "t2.micro"
  ami           = "${data.aws_ami.ubuntu.id}"
  key_name      = "${aws_key_pair.main.id}"
  subnet_id     = "${module.networking.az-subnet-id-mapping["subnet2"]}"
}

Shoot terraform apply, and you’re good to go!

aws ec2 describe-instances \
        --profile=$PROFILE

{
    "Reservations": [
        {
            "Instances": [
                {
                    "InstanceId": "i-0394b9700680faf8b",
                    "InstanceType": "t2.micro",
                    "Placement": {
                        "AvailabilityZone": "sa-east-1a"
                    },
                    "PublicIpAddress": "18.231.174.76",
                    "PrivateIpAddress": "10.0.0.191",
                    "VpcId": "vpc-4ff52828"
                }
            ]
        },
        {
            "Instances": [
                {
                    "InstanceId": "i-04714fbbd5df26497",
                    "InstanceType": "t2.micro",
                    "Placement": {
                        "AvailabilityZone": "sa-east-1c"
                    },
                    "PrivateIpAddress": "10.0.1.140",
                    "PublicIpAddress": "18.231.163.99",
                    "VpcId": "vpc-4ff52828"
                }
            ]
        }
    ]
}

Naturally, we could now try to access the instances via SSH.

Unfortunatelly, that would not work.

Accessing an AWS EC2 Instance - Setting the security groups

Whenever traffic comes to our instances, those packets pass by EC2’s virtual firewall that filters traffic before it gets to our instances.

While that’s a good analogy that makes sense in the most traditional way of thinking of a centralized firewall, security groups are a bit different.

They’re tied directly to the instances and can also be used to filter internal traffic - you can use it even to decide if two instances within the same network can communicate or not. Even more specifically, these security groups get tied to the virtual network interfaces of the instances.

A more accurate representation would be one that puts that “AWS Firewall” more tied to the instance:

The way AWS EC2 security group works is:

a security group is created (at this moment, everything is denied by default);
rules are specified and attached to a security group (these rules specify which kind of traffic is allowed - either inbound or outbound);
an instance is raised with a given security group specified.

When we created the instances in the Terraform file above, we didn’t specify a security group. In that case, AWS takes the lead and assigns a default security group to our instances.

Let’s check:

aws ec2 describe-instances \
        --profile=$PROFILE | \
        jq '.[] | .[] | .Instances | .[] | .SecurityGroups | .[]' | \
        jq -s '.'

[
  {
    "GroupName": "default",
    "GroupId": "sg-948f3ef2"
  },
  {
    "GroupName": "default",
    "GroupId": "sg-948f3ef2"
  }
]

So, as we see, both EC2 instances are making use of the default security group.

We can also check what’s defined for this group:

aws ec2 describe-security-groups \
        --group-ids sg-948f3ef2 \
        --profile=$PROFILE

{
    "SecurityGroups": [
        {
            "Description": "default VPC security group",
            "GroupId": "sg-948f3ef2",
            "GroupName": "default",
            "IpPermissions": [
                {
                    "IpProtocol": "-1",
                    "IpRanges": [],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "UserIdGroupPairs": [
                        {
                            "GroupId": "sg-948f3ef2",
                            "UserId": "461528445026"
                        }
                    ]
                }
            ],
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1",
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "UserIdGroupPairs": []
                }
            ],
            "OwnerId": "461528445026",
            "VpcId": "vpc-bd4e93da"
        }
    ]
}

This way, we can adapt our Terraform file to fit our needs - have egress all available and the SSH port open.

# Create a security group that will allow us to both
# SSH into the instance as well as access prometheus
# publicly (note.: you'd not do this in prod - otherwise
# you'd have prometheus publicly exposed).
resource "aws_security_group" "allow-ssh-and-egress" {
  name = "main"

  description = "Allows SSH traffic into instances as well as all eggress."
  vpc_id      = "${module.networking.vpc-id}"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name = "allow_ssh-all"
  }
}

and then, with that, reference this security group:

resource "aws_instance" "inst1" {
  instance_type = "t2.micro"
  ami           = "${data.aws_ami.ubuntu.id}"
  key_name      = "${aws_key_pair.main.id}"
  subnet_id     = "${module.networking.az-subnet-id-mapping["subnet1"]}"

  vpc_security_group_ids = [
    "${aws_security_group.allow-ssh-and-egress.id}",
  ]
}

Closing thoughts

With our EC2 instances running, being able to communicate with each other and granting us access to do whatever we want, we’re able to move forward with more interesting things.

To get traffic from the Internet and provide a real service we could either assign an Elastic IP to an instance (or one for each) and then configure our DNS settings or put a network load-balancer in front of them.

I wanted to share this configuration as this is one of the pieces that when I got started with AWS felt quite hard to understand - as there’s a bunch of terminology around it and few gotchas.

All the code cited here can be found in the cirocosta/sample-aws-networking repository.

If you have any questions, make sure to ask me at @cirowrc on Twitter. Maybe I wrote something completely wrong here too! I’d really appreciate if you let me know.

Have a good one!

finis

How to collect Docker Daemon Metrics

Ciro S. Costa — Sun, 18 Feb 2018 00:00:00 +0000

Hey,

While people generally know (and agree) that cAdvisor is the guy in the room to keep track of container metrics, there’s a sort of hidden feature of the docker daemon that people don’t take into account: the daemon by itself can be monitored too - see Collect Docker Metrics with Prometheus.

You indeed can determine whether the daemon is running by checking the systemd metrics via cAdvisor (if you’re running the Docker daemon as a systemd service), you can’t know much more than that.

Using the native Docker daemon metrics, you can get much more.

In this post we go through:

How to turn Docker daemon metrics on

To turn the feature on, change your daemon.json (usually at /etc/docker/daemon.json) file to allow experimental features and tell it an address and path to expose the metrics.

For instance:

{
        "expermental": true,
        "metrics-addr": "0.0.0.0:9100"
}

With that done, check out whether it’s indeed exporting the metrics:

curl localhost:9100/metrics

        # HELP builder_builds_failed_total Number of failed image builds
        # TYPE builder_builds_failed_total counter
        builder_builds_failed_total{reason="build_canceled"} 0
        builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
        ...

With the endpoint working, you’re now able to have Prometheus scraping this endpoint.

Gathering Docker daemon metrics on MacOS

If you’re a Docker-for-mac user, then there’s a little difference.

First, you’ll not have the /etc/docker/daemon.json file - this is all managed by the Docker mac app, which doesn’t prevent you from tweaking the daemon configurations though.

Head over to the preferences pannel:

Once there, head to the Daemon tab and switch to the advanced mode.

With that configuration set, you’re now able to properly reach the docker-for-mac daemon metrics: curl localhost:9100 from your mac terminal.

Tailoring a Docker metrics Grafana Dashboard

Once our metric collection is set up, we can now start graphing data from our Prometheus instance.

To demonstrate which kind of metrics we can retrieve, I set up a sample repository (cirocosta/sample-collect-docker-metrics) with some preconfigured dashboards that you can use right away.

There, I configured Prometheus to look for the Docker daemon at host.docker.internal, a DNS name that in a Docker for Mac installation resolves to the IP of the Xhyve VM that Docker is running on (see Networking features in Docker for Mac).

global:
  scrape_interval: '5s'
  evaluation_interval: '5s'

scrape_configs:

  - job_name: 'docker'
    static_configs:
      - targets:
        - 'host.docker.internal:9100'
        labels:
          instance: 'instance-1'

With Prometheus now scraping my Docker for Mac daemon, we can jump to the Grafana Web UI and start creating some dashboards and graphs.

Docker Daemon CPU usage

The one I started with was CPU usage to answer the question “How much CPU usage is the Docker daemon process taking”.

Given that all Prometheus Golang clients report their process CPU usage, we’re able to graph that (just like any other process level metric that are also revealed by Prometheus clients).

From the Prometheus documentation, these are:

Metric	Description
`process_cpu_seconds_total`	Total user and system CPU time spent in seconds
`process_open_fds`	Number of open file descriptors
`process_max_fds`	Maximum number of open file descriptors
`process_virtual_memory_bytes`	Virtual memory size in bytes
`process_resident_memory_bytes`	Resident memory size in bytes
`process_heap_bytes`	Process heap size in bytes
`process_start_time_seconds`	Start time of the process since unix epoch in seconds

Knowing that, we can then graph the CPU usage of the dockerd process:

What the query does is:

it grabs the total CPU seconds spent by the Daemon process, then
calculates the per-second instantaneous rate from the five-minutes vector of CPU seconds.

Docker container start times heatmap

A handy type of metric that we can graph from the daemon metrics is heatmaps. With them, we can spot unusual spikes in actions that the Docker daemon performs.

From the Grafana documentation:

The Heatmap panel allows you to view histograms over time

Since Grafana v5.1, we’ve been able to make use of Heatmaps when gathering metrics from Prometheus instances (see this commit).

All that the query, in this case, is doing is:

gathering all the Prometheus histogram buckets' values for the container action seconds metric labeled with the “start” action; then
calculating the rate in a one-minute vector.

The second step is necessary to make the data non-commulative, such that we can see at which point the most significant start time happened.

Number of running Docker containers

Another interesting type of metric that we can gather is gauge-like metrics that tell us about how many containers are being supervised by the daemon and in what states they are.

This can indicate how loaded the machine is. Differently from cAdvisor, here we can determine how many containers are indeed running, are still in create state, or even in a pause state.

Closing thoughts

By having the Docker daemon exporting its own metrics, we’re able to enhance even more the observability of the system when used in conjunction with cAdvisor.

While cAdvisor gives us the in-depth look of how the containers consume the machine resources, the Daemon metrics tells us how the daemon itself is consuming the machine resources and how well it’s performing what it should do - manage containers.

There are certainly more interesting metrics that you can gather, but I hope these ones will give you an idea of what’s possible.

If you enjoyed the article, please share it! I’d appreciate a lot!

You can also get related content by signing up to the newsletter below.

I’m also available for questions and feedback - hit me at @cirowrc any time.

Have a good one!

Ciro

Nginx HTTP2 Server Push

Ciro S. Costa — Tue, 13 Feb 2018 00:00:00 +0000

Hey,

some days ago HTTP2 server push has been added to Nginx (at least the open source version).

That’s great news since this was one of the most interesting features from HTTP2 that Nginx was lacking.

Given that I didn’t use server push myself so far, I decided to learn a bit more about it and give a try to Nginx' implementation.

A minimal HTTP2 Server push example in Go

To establish a base around knowing whether the HTTP2 push functionality is indeed working or not, we can implement quick example in Go.

Following the example of Yoshiki, the server serves two endpoints:

/               -->     index.html   ( <img src="/image.svg" />
/image.svg      -->     image.svg itself

That way we can see whether Chrome would retrieve the image pushed by the / handler when we request /.

I started tailoring main.go:

// handleImage is the handler for serving `/image.svg`.
//
// It does nothing more than taking the byte array that
// defines our SVG image and sending it downstream.
func handleImage(w http.ResponseWriter, r *http.Request) {
	var err error

	w.Header().Set("Content-Type", "image/svg+xml")
	_, err = w.Write(assets.Image)
	must(err)
}

// main provides the main execution of our server.
//
// It makes sure that we're providing the required
// flags: cert and key.
//
// These two flags are extremely important because
// browsers will only communicate via HTTP2 if we
// serve the content via HTTPS, meaning that we must
// be able to properly terminate TLS connections, thus,
// need a private key and a certificate.
func main() {
	flag.Parse()

	if *key == "" {
		fmt.Println("flag: key must be specified")
		os.Exit(1)
	}

	if *cert == "" {
		fmt.Println("flag: cert must be specified")
		os.Exit(1)
	}

	http.HandleFunc("/", handleIndex)
	http.HandleFunc("/image.svg", handleImage)

	must(http.ListenAndServeTLS(":"+strconv.Itoa(*port), *cert, *key, nil))
}

Nothing fancy there, the big deal comes next: the handler for index.html.

This handler is the one that knows about what are the assets the index.html have (in our example, image.svg) such that it can tell the browser to start fetching them ahead of time.

With Go1.8+ we do that obtaining an http.Pusher by casting the writer supplied to our handler and then using the Push method:

// handleIndex is the handler for serving `/`.
//
// It first checks if it's possible to push contents via
// the connection. If so, then it pushes `/image.svg` such
// that at the same moment  that the browser is fetching
// `index.html` it can also start retrieving `image.svg`
// (even before it knows about the existence in the html).
func handleIndex(w http.ResponseWriter, r *http.Request) {
	var err error

	pusher, ok := w.(http.Pusher)
	if ok {
		must(pusher.Push("/image.svg", nil))
	}

	w.Header().Add("Content-Type", "text/html")
	_, err = w.Write(assets.Index)
	must(err)
}

Compile it and let it run.

Verifying HTTP2 Server Push with Google Chrome

Running the application on port 443 (requires some privileges), head to Chrome at https://localhost, open the Networks tab in the developer tools and see the push indication:

ps.: the Not Secure indication is not a big deal - the browser doesn’t trust the certificate we provided. If you wan’t to see it green you can either get a real certificate (LetsEncrypt is very handy for this if you have a domain and a webserver on such domain that you can use to retrieve a cert from them) or, in Mac, use Keychain to trust your certificate.

To get the details around how that push got initiated, we can use the chrome://net-internals page to debug the HTTP2 connection that we got established:

Note that we’re not pushing contents downstream right from the index.html handler, but instead in the push functionality we just signalize that the browser should perform another request.

ps.: From the Go’s side we can’t consume those push frames yet: see https://github.com/golang/go/issues/18594 and PR https://go-review.googlesource.com/#/c/net/+/85577/.

Inspecting the HTTP2 streams using Wireshark

Aside from using chrome://net-internals, we can use Wireshark.

Chrome, Firefox and even Go are able to produce NSS-formatted key log files that gives the necessary secrets to Wireshark so it can decrypt the streams and interpret them.

Using either Chrome of Firefox you can get the file written to our filesystem by initializing them with the environment variable ``SSLKEYLOGFILE` that indicates where this file should live.

For instance, using MacOS:

SSLKEYLOGFILE=~/tlskey.log \
        /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome &

Once the browser is running, open Wireshark, go to preferences, then protocols and then finally reach the SSL configurations. There you set the (Pre)-Master-Secret log filename to the name you put on SSLKEYLOGFILE environment variable:

Now head to https://localhost and see the HTTP2-filtered packets flowing on the loopback interface:

Done!

ps.: make sure you’re running the application on port 443. For some reason http2 filter won’t work on a different port. I guess that’s because the http2 filter is probably hardcoded to 443, but who knows?

Installing NGINX from source

I’m not very sure if there’s already a released version of NGINX with the latest commit from some days ago, so I decided to go with a build right from source.

Here I’m using Ubuntu Artful (17.10) on VirtualBox.

For that I set up a quick Vagrantfile:

Vagrant.configure(2) do |config|
  config.vm.hostname = "artful"
  config.vm.box = "ubuntu/artful64"
  config.vm.box_check_update = false
  config.vm.network "forwarded_port", guest: 443, host: 443

  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 3
  end

  config.vm.synced_folder ".", "/vagrant", disabled: true
end

With the virtual machine up, I get there and start the actual thing process.

# Clone the git mirror (the official version control
# system used by nginx is mercurial)
git clone https://github.com/nginx/nginx


# Get into the cloned repository
cd ./nginx


# Run auto configuration to check for dependencies and
# prepare installation files.
#
# Make sure you run this command from the `nginx` directory
# instead of `./auto`. 
./auto/configure


# Here it should probably tell you that you that it 
# didn't find some dependencies (like `pcre` and others).
#
# Let's install them
sudo apt install -y \
        libpcre3 libpcre3-dev \
        zlib1g-dev \
        libssl-dev


# Configure our build with some modules.
# - http_v2_module      provides support for HTTP/2
# - http_ssl_module     provides the necessary support
#                       for dealing with TLS
./auto/configure \
        --with-http_v2_module \
        --with-http_ssl_module


# After running the command we should have a Makefile
# in the `nginx` directory (current working directory)
ls | grep Makefile
Makefile

Now that we have the Makefile ready, it’s a matter of triggering the build:

# Run the build process with a concurrency of 4
make -j4


# Check that `nginx` has been produced
./objs/nginx -v
nginx version: nginx/1.13.9

All set! Time to explore this new NGINX version.

Configuring NGINX with HTTP2

To get started with the most minimal configuration possible, I set up a static website configuration with HTTP2 support with server push configured for our root path:

# Do not daemonize - this makes it easier to test
# new configurations as any stupid error would be
# more easily caught when developing.
daemon off;

events {
        worker_connections              1024;
}

http {

        # Explicitly telling what mime types we're
        # supporting just for the sake of explicitiness
        types {
                image/svg+xml           svg svgz;
                text/html               html;
        }

        server {
                # Listen on port 8443 with http2 support on.
                listen                  8443 http2;


                # Enable TLS such that we can have proper HTTP2
                # support using browsers
                ssl on;
                ssl_certificate         certs/cert_example.com.pem;
                ssl_certificate_key     certs/key_example.com.pem;


                # For the root location (`index.html`) we perform
                # a server push of `/image.svg` when serving the
                # content to the end user.
                location / {
                        root            www;
                        http2_push      "/image.svg";
                }


                # When pushing the asset (`image.svg`) there's no need
                # to push additional resurces.
                location /image.svg {
                        root            www;
                }
        }
}

That works exactly as we wanted: if you head over / (that serves index.html) you get index.html and also the PUSH_PROMISE to retrieve image.svg:

Unsurprisingly, this is almost identical to the Go example we set up.

HTTP2 Server push using Nginx as a Proxy

As it’s very common to use NGINX as a reverse proxy, this new feature also supports such mode.

Although NGINX does not support HTTP2 backends, it can act as an HTTP2 frontend that translates requests back to origin servers as HTTP1.1, letting TLS termination and HTTP2 capabilities to itself.

The tricky part of server push is knowing what to push - something that a proxy doesn’t know.

To get over that, NGINX adhered to the spec of using the Link header as means of informing it what to push.

Let’s go back to Go to implement and HTTP1.1 server that informs NGINX when to push then.


// In the case of HTTP1.1 we make use of the `Link` header
// to indicate that the client (in our case, NGINX) should
// retrieve a certain URL.
//
// See more at https://www.w3.org/TR/preload/#server-push-http-2.
func handleIndex(w http.ResponseWriter, r *http.Request) {
	var err error

	if *http2 {
		pusher, ok := w.(http.Pusher)
		if ok {
			must(pusher.Push("/image.svg", nil))
		}
	} else {
                // This ends up taking the effect of a server push
                // when interacting directly with NGINX.
                //
                // Note that I'm returning `/proxy/image.svg` instead
                // of `/image.svg`. 
                //
                // This has the effect of NGINX taking this as a normal 
                // standard request and processing it through its location
                // pipeline (which ends up in the image Go handler we defined
                // here).
		w.Header().Add("Link", 
			"</proxy/image.svg>; rel=preload; as=image")
	}

	w.Header().Add("Content-Type", "text/html")
	_, err = w.Write(assets.Index)
	must(err)
}

In NGINX, not muched changed as well:

http {
 
         # Explicitly telling what mime types we're
@@ -16,6 +18,11 @@ http {
                 text/html               html;
         }
 
+        # Add an upstream server to proxy requests to
+        upstream sample-http1 {
+                server localhost:8080;
+        }
+
         server {
                 # Listen on port 8443 with http2 support on.
                 listen                  8443 http2;
@@ -27,6 +34,10 @@ http {
                 ssl_certificate         certs/cert_example.com.pem;
                 ssl_certificate_key     certs/key_example.com.pem;
 
+                # Enable support for using `Link` headers to indicate
+                # origin server push
+                http2_push_preload on;
+
 
                 # For the root location (`index.html`) we perform
                 # a server push of `/image.svg` when serving the
@@ -42,6 +53,15 @@ http {
                 location /image.svg {
                         root            www;
                 }
+
+                # Act as a reverse proxy for requests going to /proxy/*.
+                # Because we don't want to rewrite our endpoints in the
+                # Go app, rewrite the path such that `/proxy/lol` ends up
+                # as `/lol`.
+                location /proxy/ {
+                        rewrite         /proxy/(.*) /$1 break;
+                        proxy_pass      http://sample-http1;
+                }
         }
 }

I enabled http2_push_preload for the whole server and then made all request to /proxy/ be proxied to our upstream sample-http1 server.

The result? The following:

At first that didn’t make much sense to me: two image.svg? What?

However, looking at the paths, it was clear: there was a request being made by the browser (/image.svg) and another coming from the server push (/proxy/image.svg). There was that extra one because I didn’t change the HTML, thus, Chrome was requesting it (as it should!).

If you’re curious about what goes on at the server host, this is a tcpdump of what goes on:

# Collect some packets within the host.
# -i: interface to snoop (loopback)
# -A: present body in ASCII format
# pos' args: only packets destined to 8080
sudo tcpdump \
        -i lo \
        -A \
        dst port 8080


# When requesting `/proxy/` nginx picks our
# proxy location and then recreates the request
# as an HTTP1.0 request to `sample-http1` upstream.
GET / HTTP/1.0
Host: sample-http1
Connection: close


# In consequence of the server-push the browser 
# performs the request (reusing the HTTP2 conenction)
# which ends up rewritten by NGINX as an HTTP1.0 request
# to our origin.
GET /image.svg HTTP/1.0
Host: sample-http1
Connection: close

Closing thoughts

It’s pretty cool to see that NGINX is integrating server push. While it doesn’t support HTTP2 backends, having server push at the proxy level is a very neat way of letting HTTP1 clients take advantage of this functionality.

I hope HAProxy comes next with an implementation that also makes use of the Link header (as that seems to be the standard).

If you have any questions or notice a mistake, please let me know!

All of the code cited here is aggregated in this repository: cirocosta/sample-nginx-http2. Feel free to open issues / PR if you find something to be improved.

I’m cirowrc on Twitter, by the way.

Have a good one!

finis

An Example of Go RPC Client and Server

Ciro S. Costa — Sat, 10 Feb 2018 00:00:00 +0000

Hey,

I’ve been exploring the new Go support for AWS Lambda that has been announced by AWS early this year (see Announcing Go Support for AWS Lambda), and there’s a little detail of it that I found very interesting: at its core, it uses Go’s net/rpc package as a mean of communication between the AWS infrastructure and your code.

Having never used net/rpc myself, I decided to explore it.

Architecting an RPC hello world

As net/rpc gives us ways of making a piece of code communicate with another piece of code over the wire, a natural example is a “Hello World” in a client-server architecture.

To achieve that, I created this repository: cirocosta/sample-rpc-go.

The file structure looks like the following:

.
├── Makefile            # instructions for building and running the
|                       # example
|
├── client              # RPC client
│   └── client.go       # provides the interface which executes the
|                       # command remotely.
|
├── core                # Shared functionality.
│   └── core.go         # - implements the actual handler that gets
|                       #   executed in the server;
|                       # - provides the go representation of the 
|                       #   messages exchanged between client and server.
|
├── main.go             # CLI
|                       #
└── server              # RPC server - provides the interface which
    |                   # allows a client to communicate with it
    └── server.go       # over the wire.

That allow us to keep the project really simple as main executes either the server or client code based on a flag (-server) and all the rest of the functionality is implemented independently.

Create a client and a server, and you’ll be able to communicate.

Let’s do that, then!

Creating a Go RPC Handler

The main piece of an RPC server is the methods that get executed when the client makes a call to them, the p part of the name - procedures.

For those accustomed to HTTP, in RPC land the equivalent of an endpoint (like, POST /say-hello) is a service/procedure name (HelloSayer.Say). Go constructs a map of these service endpoints by taking an exported struct that has a set of exported methods and making them available.

Naturally, these methods must obey to a certain interface so that net/rpc can properly provide (de)serialization of the request and response objects and perform the right method execution:

it must take two arguments and return an error object;
the first argument is the request and second is the response;
the second (response) must be a pointer - in case of failure (an error in the service), error is returned, and the response is sent as nil.

Looking at the Go source code (src/net/rpc/server.go), we can understand how it’s done behind the scenes.

// Register publishes in the server the set of methods of the
// receiver value that satisfies the following conditions:
//	- exported method of exported type
//	- two arguments, both of exported type
//	- the second argument is a pointer
//	- one return value, of type error
// It returns an error if the receiver is not an exported type or has
// no suitable methods. It also logs the error using package log.
// The client accesses each method using a string of the form "Type.Method",
// where Type is the receiver's concrete type.
func (server *Server) Register(rcvr interface{}) error {
	return server.register(rcvr, "", false)
}

// ciro: `register` does the hard work of taking the struct and then analyzing
// its methods to register them.
//
// In the end, `net/rpc` constructs a map that maps endpoint names 
// (`Struct.Method`) to the actual implementations, as well as  
// performing the (de)serialization accordingly  such that the methods 
// are properly called.
func (server *Server) register(rcvr interface{}, name string, useName bool) error {
	s := new(service)
	s.typ = reflect.TypeOf(rcvr)
	s.rcvr = reflect.ValueOf(rcvr)
	s.name = reflect.Indirect(s.rcvr).Type().Name()

	// Install the methods
        // ciro: this inspects the methods and verify (for each) whether they 
        // adhere to the "interface" that net/rpc expects - first arg is 
        // represents the request and is not a pointer ; second is the response 
        // and must be  pointer, and then it also has an error return.
	s.method = suitableMethods(s.typ, true)

        // store the service with the given service name
	if _, dup := server.serviceMap.LoadOrStore(s.name, s); dup {
		return errors.New(
                        "rpc: service already defined: " + 
                        sname)
	}
	return nil
}

So, for a Hello World I end up with this:

type Response struct {
	Message string
}

type Request struct {
	Name string
}

type Handler struct {}

func (h *Handler) Execute(req Request, res *Response) (err error) {
	if req.Name == "" {
		err = errors.New("A name must be specified")
		return
	}

	res.Message = "Hello " + req.Name
	return
}

It adheres to the interface that net/rpc expects and is simple enough for our example.

Creating a Go RPC Server and Client

Once the hander has been defined, creating the server is easy if you’ve set up an HTTP or TCP server before. The only difference is that before actually listening for incoming connections, you have to call rpc.Register and pass an instance of the struct that carries the exported methods you define (i.e., your service handlers).

// Publish our Handler methods
rpc.Register(&core.Handler{})

// Create a TCP listener that will listen on `Port`
listener, _ = net.Listen("tcp", ":"+strconv.Itoa(Port))

// Close the listener whenever we stop
defer listener.Close()

// Wait for incoming connections
rpc.Accept(listener)

on the client side, it’s almost the same as establishing a TCP connection:

var (
        addr     = "127.0.0.1:" + strconv.Itoa(Port)
        request  = &core.Request{Name: Request}
        response = new(core.Response)
)


// Establish the connection to the adddress of the
// RPC server
client, _ = rpc.Dial("tcp", addr)
defer c.client.Close()


// Perform a procedure call (core.HandlerName == Handler.Execute)
// with the Request as specified and a pointer to a response
// to have our response back.
_ = c.client.Call(core.HandlerName, request, response)
fmt.Println(response.Message)

Let’s see how it works in practice.

Go RPC in Practice

Given the simplicity of having a client and server, I got curious about how the underlying communication works.

By default, the Go RPC mechanism uses encoding/gob to (de)serialize the messages that come and go from net/rpc. This means that we’d not have an easy time deciphering what’s going on by inspecting the wire. Nonetheless, net/rpc is pluggable enough that other encoding mechanisms can be used - for instance, the official jsonrpc which is much simpler to inspect.

First, a connection is established (plain TCP), and the request is sent:

{
    "id": 0,
    "method": "Handler.Execute",
    "params": [
        {
            "Name": "ciro"
        }
    ]
}

As no error happens, the response cames back with error: null and the result (our Response message):

{
    "error": null,
    "id": 0,
    "result": {
        "Message": "Hello ciro"
    }
}

That’s it!

Closing thoughts

Having never used net/rpc before, I felt that there’s a lot of value having a working RPC system so fast.

At the same time, it’s very hard to find great resources about this package online.

I started searching why and it turns out that net/rpc is in a “freeze” state - no more development is meant to go on in this package - and you can clearly see that: no method of this package takes context, although it makes a lot of sense for some of them (e.g., the client’s .Call method could easily have request cancellation with a context just like you have with net/http).

If you’re curious about whether people use net/rpc out there, the answer is: yes. Most notably, minio, which saw a decrease in performance when evaluating gRPC. Pretty interesting.

What about you? Have you ever used it? Did you, like me, went straight to gRPC without checking net/rpc? I’d love to know!

Reach me at Twitter @cirowrc at any time.

Have a good one!

finis

The first limit you'll hit on AWS EFS: Locks

Ciro S. Costa — Sat, 10 Feb 2018 00:00:00 +0000

Hey,

a month ago I was looking at some issues with a container running a MySQL instance against an NFS-mounted directory on the host (should you do this? maybe not).

The issues seemed pretty weird as I’m not a MySQL guy and there were all sorts of errors popping up related to disk quotas.

Sure, we did have a bunch of space:

df -h
fs-<...>.amazonaws.com:/  8.0E  1.6G  8.0E   1% /mnt/nfs

The problem, as the logs revealed, was that InnoDB wasn’t able to grab a lock that it wanted.

That seemed even weirder to me as I thought we’d never hit a limit in this front.

Making the case

My idea to get through this was to try to replicate what the user was facing - he couldn’t take his MySQL instance up.

Let’s bring something up and look at some MySQL details:

# Create a container named `inst1` which has
# two environment variables set which alters
# the behavior of the mysql instance startup.
#
# The code that runs at startup time can be found
# at the repository that holds this image:
# https://hub.docker.com/mysql
docker run \
        --detach \
        --name inst1 \
        --env MYSQL_DATABASE=testdb \
        --env MYSQL_ALLOW_EMPTY_PASSWORD=true \
        mysql

These two environment variables allow us to have a running MySQL instance with an empty password and a database testdb created.

If you’re curious about it, this comes from mysql’s image entrypoint:

# If the string `MYSQL_DATABASE` has been
# specified, then add the database creation
# string to the list of mysql commands to 
# run after the process has started.
# (line 159 of docker-entrypoint.sh)
if [ "$MYSQL_DATABASE" ]; then
        echo "CREATE DATABASE IF NOT EXISTS \`$MYSQL_DATABASE\` ;" | \
                "${mysql[@]}"
        mysql+=( "$MYSQL_DATABASE" )
fi

With this, we can start looking at the structure of the filesystem that MySQL creates.

# get into the container
docker exec \
        --interactive \
        --tty \
        inst1 \
        /bin/bash


# From within the container, get into the directory 
# where mysql puts our database files and everything 
# that we should persist
cd /var/lib/mysql


# Find all the directories that are out there
find . -type d

        .
        ./sys
        ./mysql
        ./testdb
        ./performance_schema


# Check that indeed, these are just databases
echo "SHOW DATABASES;" | mysql 

        Database
        information_schema
        mysql                   <<<<<<<
        performance_schema      <<<<<<<
        sys
        testdb                  <<<<<<<

Cool, but what about the locks?

We can count how many are there by either making use of lslocks or looking at /proc/locks. The benefit of using lslocks here is that it’ll resolve the paths for us so we can know which files of the MySQL instance that are holding locks:

lslocks
COMMAND PID  TYPE SIZE MODE  M START END PATH
mysqld    1 POSIX  48M WRITE 0     0   0 ./ib_logfile0
mysqld    1 POSIX  48M WRITE 0     0   0 ./ib_logfile1
mysqld    1 POSIX  76M WRITE 0     0   0 ./ibdata1
mysqld    1 POSIX  12M WRITE 0     0   0 ./ibtmp1
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/plugin.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/innodb_table_stats.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/innodb_index_stats.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/gtid_executed.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/server_cost.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/engine_cost.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/time_zone_leap_second.ibd
mysqld    1 POSIX  96K WRITE 0     0   0 ./mysql/servers.ibd
mysqld    1 POSIX 320K WRITE 0     0   0 ./mysql/time_zone_name.ibd
mysqld    1 POSIX 144K WRITE 0     0   0 ./mysql/time_zone.ibd
mysqld    1 POSIX  12M WRITE 0     0   0 ./mysql/time_zone_transition.ibd
mysqld    1 POSIX 464K WRITE 0     0   0 ./mysql/time_zone_transition_type.ibd

Well, no testdb there.

So we know that it’s not locking at the database level.

Maybe it does lock but differently as there is a way of performing a mutually exclusive flush per-database - FLUSh tables WITH read lock.

We can confirm that it locks at the table-level:

# Create the query that creates a table called 
# `example` with just two dummy columns.
table_creation_query="
CREATE TABLE example ( 
        id INT, 
        data VARCHAR(100) 
);"


# Execute the query against our database
mysql --database testdb <<< "$table_creation_query"


# Verify that a lock has been created
lslocks | grep testdb
mysqld    1 POSIX  96K WRITE 0     0   0 ./testdb/example.ibd

Cool, we now know that MySQL will hold locks per-table.

How does MySQL behave on lock starvation?

For someone with close to zero knowledge about MySQL, I thought about forcing the system to fail MySQL when it tried to get a lock by setting limits on the number of locks that the process could take.

The idea is to make use of the setrlimit syscall via the ulimit utility:

GETRLIMIT(2)    Linux Programmer's Manual         GETRLIMIT(2)

NAME
       getrlimit, setrlimit, prlimit - get/set resource limits

SYNOPSIS
       int setrlimit(int resource, const struct rlimit *rlim);

...

       The  getrlimit()  and  setrlimit()  system  calls  get  
       and set resource limits respectively.  Each resource has 
       an associated soft and hard limit ...


           struct rlimit {
               /* Soft limit */
               rlim_t rlim_cur;  

               /* Hard limit (ceiling for rlim_cur) */
               rlim_t rlim_max;  
           };

It essentially means that we can put some limits on some resources by specifying some numbers (hard and soft limit). The resource that we want to limit in this case is the number of locks a process can take:

       RLIMIT_LOCKS
              A limit on the combined number of flock(2) locks 
              and fcntl(2) leases that this process may establish.

That sounds pretty close but we don’t know if mysql is indeed making use of these calls.

We know that lslocks does see a new lock when we create a new table but … does it use one of those syscalls?

We can check that using strace:

# Get the PID of the MySQL daemon instance as seen by
# the host.
docker top inst1
PID          ..   CMD
861379       ..   mysqld
861589       ..   /bin/bash
864033       ..   /bin/bash


# Cool, PID of `mysqld` is 861379.
# From the host we can specify that pid to strace
# and tell it to attach to all the processes raised 
# by it. 
#
# This way we can look at every syscall
# made by them while it runs.
#
# Here I'm making use of the following arguments:
# -f:   Trace  child  processes (and "lightweight processes")
# -e:   Filter the calls so that we only get notified when
#       there's a syscall that matches the term
# -p:   Pid to attach to (and its children)

sudo strace -f -e fcntl -p 861379 
strace: Process 861379 attached with 28 threads


# Now that we got attached, in a separate terminal, create a new
# table by sending the table creation command to the
# container's mysql binary (client) that will get the command
# sent to the mysql daemon.
echo "CREATE TABLE example2 ( id INT, data VARCHAR(100) );" | 
        docker exec inst1 \
                mysql --database testdb


# Now look at our strace results!
# We spotted some `fcntl` syscalls with
# the lock-related parameters
[pid 862376] fcntl(44, F_SETLK, {       <<<<<
        l_type=F_WRLCK,                 <<<<<
        l_whence=SEEK_SET, 
        l_start=0, 
        l_len=0}) = 0

[pid 862376] fcntl(44, F_SETLK, {       <<<<<<
        l_type=F_WRLCK,                 <<<<<<
        l_whence=SEEK_SET, 
        l_start=0, 
        l_len=0}) = 0

By the declaration of fcntl we can check what those arguments are about:

fcntl - manipulate file descriptor

int fcntl(int fd, int cmd, ... /* arg */ );

FD:             file descriptor to act upon
CMD:            action to perform
ARG:            well, arguments

fd=44           (acting uppon filedescriptor 44)
cmd=F_SETLK     (acquiring a write - exclusive - lock)

But, how do we know what is that 44? What that file descriptor points to?

To know that we can look at the table of file descriptors open by a given process.

From within the container (we could do this from the outside too) we can check that:

# Get inside the container
docker exec -it inst1 /bin/bash

# By looking at the proc virtual filesystem we
# can check what files are open.
#
# Because we're running a container that runs MYSQL
# as the PID 1, we look at the properties for the 
# process who's PID is 1.
#
readlink /proc/1/fd/44 
/var/lib/mysql/testdb/example2.ibd

Exactly! Our table example2 has a lock associated with it (as we wanted) which gets set using fcntl, a syscall that can be blocked by setrlimit calls.

So now we know that we can make use of setrlmit with RLIMIT_LOCKS and expect that the kernel will limit the process' resources.

With Docker, we make use of it by specifying an extra argument to the run command: --ulimit.

# Run the same command as before that spawns
# a mysql container but this time with the limit
# of 25 locks. 
docker run \
        --detach \
        --name inst1 \
        --ulimit locks=25 \
        --env MYSQL_DATABASE=testdb \
        --env MYSQL_ALLOW_EMPTY_PASSWORD=true \
        mysql

To see it in action, let’s try to create 30 tables.

#!/bin/bash

# exit on any failed execution
set -o errexit

# main routine that calls the
# table creation method 30 times
main () {
        for i in $(seq 1 30); do
                create_table mytable$i
        done
}

# execute a command within the mysql
# container to create a table.
# args:
#       1) name of the table
create_table () {
        local name=$1

        echo "CREATE TABLE $name ( 
        id INT, 
        data VARCHAR(100) 
);" | docker exec -i inst1 \
        mysql --database=testdb
}

main

Guess what? It doesn’t work! The execution goes past 30 tables without a problem. We’re not limiting the locks as we wanted.

The reason why it’s not working is that since Linux 2.4 setrlimit simply stopped limiting locks, meaning that we were relying on a feature that is not even there anymore.

How can we get get through that?

Messing up our syscalls

There’s a Linux syscall (ptrace) that can help us.

Quoting the man pages, it provides means by which a trace program may observe and control the execution of another process, even changing the tracee’s memory and registers.

So, given that we know exactly what syscall brings us the trouble (puts locks), we can get in the way of the syscall invocations for a given process, change it and then mess with it such that the tracee (mysqld in our case) thinks that it doesn’t have enough disk quotas (free locks).

Given that strace (the tool) makes use of ptrace and that it allows us to temper syscalls, we can make use of it to emulate a disk quota issue (after n fctnl calls trying to lock, fail with ENOLCK (“Too many segment locks open, lock table is full, or a remote locking protocol failed”).

# Run `strace` targetting the process at pid $PID
# as well as it's children.
#
# Inject an FCNTL error in case the syscall has been
# executes more than 25 times.
strace \
        -p $PID \
        -f \
        -e fault=fcntl:error=ENOLCK:when=25+

Running the script above (the one that creates 30 tables) runs, we can see the InnoDB logs:

mysqld: ready for connections.
Version: '5.7.21'  socket: '/var/run/mysqld/mysqld.sock'  
                   port: 3306  MySQL Community Server (GPL)
[ERROR] InnoDB: Unable to lock ./testdb/mytable22.ibd error: 37
[ERROR] InnoDB: Operating system error number 37 in a file operation.

                                        \/ \/ \/ \/
[ERROR] InnoDB: Error number 37 means 'No locks available'
                                        /\ /\ /\ /\

[ERROR] InnoDB: Cannot create file './testdb/mytable22.ibd'

Cool, we failed the table creation process!

Now it’s time to do it on AWS.

Terraforming a test scenario in AWS

The scenario is simple: single machine and an EFS mount in the same AZ (Availability Zone).

To provision that I made use of Terraform such that I can make it very declarative and reproducible.

I structured it like the following:

.
├── main.tf             # data retrieval and output
├── efs.tf              # provisiong the filesystem and a mount target
│                       # in the specific AZ that our instance is in
├── instance.tf         # creates an instance in the region and AZ we want
├── keys.tf             # provisions the key in the region
├── keys                # dumb keys for this one-off experiment
│   ├── key.rsa
│   └── key.rsa.pub
└── terraform.tfstate   # state

1 directory, 7 files

If you’re curious about the whole Terraform files, check them out at cirocosta/efs-locks-sample, but I wanted to highlight efs.tf.

Whenever you create an EFS filesystem you created it by region (note that not all regions support it - I think it’s something like only 5) and then within that region, you create mount points that the EFS attach to within subnets you already provisioned.

For instance, say you’re in us-east-1 and have subnets on us-east-1a and us-east-1b. You create two mount points, and then your instances in those two availability zones can connect to it.

In Terraform terms that would mean that you’d have a list of subnets and availability zones that you want your EFS mounts in and then you’de iterate over them (using count) to create the mount points.

Here I’m sticking with a single AZ to make things easier.

# Creates a new empty file system in EFS.
resource "aws_efs_file_system" "main" {
  tags {
    Name = "locks-test-fs"
  }
}

# Creates a mount target of EFS in a specified subnet
# such that our instances can connect to it.
resource "aws_efs_mount_target" "main" {
  file_system_id  = "${aws_efs_file_system.main.id}"
  subnet_id       = "${var.subnet}"
  security_groups = ["${aws_security_group.efs.id}"]
}

# Allows both ingress and egress for port 2049 (nfs)
# such that our instances can get to the mount
# target in the AZ.
resource "aws_security_group" "efs" {
  name        = "efs-mnt"
  description = "Allows NFS traffic from instances within the VPC."
  vpc_id      = "${data.aws_vpc.default.id}"

  ingress {
    from_port = 2049
    to_port   = 2049
    protocol  = "tcp"

    cidr_blocks = [
      "${data.aws_vpc.default.cidr_block}",
    ]
  }

  egress {
    from_port = 2049
    to_port   = 2049
    protocol  = "tcp"

    cidr_blocks = [
      "${data.aws_vpc.default.cidr_block}",
    ]
  }

  tags {
    Name = "allow_nfs-ec2"
  }
}

The output of that is then an address that we use to mount an NFS partition - which we then use to mount in a directory that we wish the have the file system.

#!/bin/bash

# Performs the mounting of an EFS mount target into
# predefined MOUNT_LOCATION in the filesystem.


set -o errexit

readonly MOUNT_LOCATION="${MOUNT_LOCATION:-/mnt/efs}"
readonly MOUNT_TARGET="${MOUNT_TARGET}"


# Main routine - checks for the MOUNT_TARGET variable
# that does not have a default (can be empty) making 
# sure that it gets specified.
main () {
        if [[ -z "$MOUNT_TARGET" ]]; then
                echo 'ERROR: 
        MOUNT_TARGET environment variable must be specified.
                '
                exit 1
        fi

        echo "INFO:
        Mounting shared filesystem.

        MOUNT_LOCATION  $MOUNT_LOCATION
        MOUNT_TARGET    $MOUNT_TARGET
        "

        mount_nfs
}


# Performs the actual mounting of the EFS target
# in the AZ we specified into a directory we defined
# via MOUNT_LOCATION.
mount_nfs() {
        sudo mkdir -p $MOUNT_LOCATION

        sudo mount \
                -t nfs4 \
                -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 \
                $MOUNT_TARGET:/ $MOUNT_LOCATION
}

main

Having the script above executed, we’re ready to go (make sure you have nfs-common installed, by the way).

Exhausting the EFS mount target locks

Per the AWS documentation we can check how the quotas look like:

Each unique mount on the instance can acquire up to a total of 8,192 locks across a maximum of 256 unique file/process pairs. For example, a single process can acquire one or more locks on 256 separate files, or 8 processes can each acquire one or more locks on 32 files.

See Limits for Client EC2 Instances

That means that just to make sure that we’ll indeed exhaust it, we can create a single MySQL instance and then make it create some three hundred tables.

Adopting that script I showed above, this means:

local number_of_tables="$1"

if [[ -z "$number_of_tables" ]]; then
        echo "ERROR:
        An argument (number of tables) must be supplied.
        "
        exit 1
fi

for i in $(seq 1 $number_of_tables); do
        create_table mytable$i
done

then, calling it with 300:

./create-many-tables.sh 300

we can see that after a little while, it breaks:

ERROR 1030 (HY000) at line 1: Got error 168 from storage engine

Looking at the number of locks that the system is holding:

# Count the number of locks being held from the efs mount
lslocks | grep 'efs' | wc -l
256

We can clearly see the quota in place.

Naturally, InnoDB complained and failed:

# docker logs ...
[Note] mysqld: ready for connections.
[ERROR] InnoDB: Unable to lock ./testdb/mytable241.ibd error: 122
[ERROR] InnoDB: Operating system error number 122 in a file operation.
[ERROR] InnoDB: Error number 122 means 'Disk quota exceeded'
[ERROR] InnoDB: Cannot create file './testdb/mytable241.ibd'

That’s it, 256 locks in a machine and you’re done.

Is this a problem with NFS in general?

I thought this limit that AWS imposes could be a hard limit imposed by the protocol itself (NFSv4.1), but it turns out that this isn’t the case.

If you create yourself an NFS server, mount it and try running a stress test that grabs as many locks as possible, you can go up to the millions (even though that takes some load on the server).

To make sure that this is indeed the case, I set up a quick c program that stresses only this piece (checkout at cirocosta/efs-locks-sample/stress).

/**
 * Struct that keeps track of the runtime arguments
 * that we supply via the CLI.
 */ 
typedef struct runtime_t {
	int number_of_locks;
	char* base_directory;
} runtime_t;

/**
 * Prints to STODUT the runtime configuration passed.
 */
void
runtime_show(runtime_t* runtime)
{
	// some printf to show the runtime
}

/**
 * Creates a file under a directory (base) with a number (i)
 * and then grabs a write lock to it.
 */
void
create_file(const char* base, const int i)
{
	FILE* fp;
	char* filename;
	struct flock lock = { 0 };
	int err = 0;

        // make the lock exclusive (write lock)
	lock.l_type = F_WRLCK;

	// .. do some work to have the filename properly set
        // try to acquire the lock
	err = fcntl(fileno(fp), F_SETLKW, &lock);
	if (err == -1) {
		perror("fcntl");
		fprintf(stderr, "Couldn't acquire lock on %s", filename);
		exit(1);
	}

        // just write something stupid to the file
	fprintf(fp, "dummy");
	free(filename);
}

/**
 * Creates `N` files with write a write lock grabbed for each
 * under a base directory `base`.
 */
void
runtime_create_files(runtime_t* runtime)
{
	// ...
}

/**
 * Main execution.
 *
 * Parses the cli arguments (number of locks
 * and base directory) and then starts the creation of the
 * files and locks.
 *
 * Once lock creation is done, pauses its execution until
 * a signal arrives.
 */
int
main(int argc, char** argv)
{
	// ...

	runtime_t runtime = {
		.number_of_locks = atoi(argv[1]), 
                .base_directory = argv[2],
	};

	runtime_show(&runtime);
	runtime_create_files(&runtime);
        
        // ... wait for signals

	return 0;
}

As expected, we hit the limit pretty fast in AWS while with a regular NFS server we can go to the sky.

Closing thoughts

What can you do if you need MySQL on EFS? Well, this is the funny part: not much!

That’s a limit that you can’t tell AWS to raise.

I’m not very sure you’d want to run MySQL on EFS (or any other NFS mount) as it’s clearly not endorsed by MySQL, but I guess that people using other software might hit problems in scenarios that make sense using EFS (NFS in general).

I guess the best is to deal with proper MySQL replication on normal EBS disks, SSD on some other machines or go straight with the new NVMe-based instances.

Have you faced problems like this? Did you spot any error?

Please let me know!

I’m cirowrc on Twitter and would appreciate knowing what your thoughts are!

Have a good one,

finis

Initializing Grafana with preconfigured dashboards

Ciro S. Costa — Fri, 09 Feb 2018 00:00:00 +0000

Hey,

this week the team behind Grafana released a beta release of Grafana v5 (which is almost there - check the milestone) and I got very happy to know that now it’s easier to get an instance from ground up without having to use the UI to configure the dashboards.

The niceness of that feature is that now you can redeploy a Grafana container that has a bunch of dashboards without needing to go through the UI to configure them.

While previously you’d had to manually import dashboards and configure the data sources using the UI, now it’s all declarative.

Here I’ll go through an example where I set up Prometheus as Grafana’s source such and then perform the dashboard initialization without using the UI.

If you’re only concerned with the Grafana initialization and want to skip Prometheus, go to the Configuring Grafana section.

Setting the stage

I created a little repository that contains some source code that can be used to check out the feature: cirocosta/sample-grafana.

It’s structured like this:

.
├── Makefile
├── docker-compose.yml          # docker-compose file that
│                               # aggregates `prometheus`,
│                               # `node_exporter` and `grafana`.
│ 
├── grafana                     # Grafana config
│   ├── Dockerfile              # Dockerfile that adds config to the image
│   ├── config.ini              # Base configuration
│   ├── dashboards              # Pre-made dashboards
│   │   └── mydashboard.json    # Sample dashboard
│   └── provisioning            # Configuration for automatic provisioning at
│       │                       # grafana startup.
│       ├── dashboards          
│       │   └── all.yml         # Configuration about grafana dashboard provisioning
│       └── datasources
│           └── all.yml         # Configuration about grafana data sources provisioning
│ 
└── prometheus                  # Prometheus config
    ├── Dockerfile
    └── config.yml

The docker-compose.yml file that aggregates the three containers works like this:

version: '3.3'
services:

  # Prometheus uses the image resulting form the build
  # of `./prometheus` which simply packs some configuration
  # into the form of an image.
  #
  # It could instead, use volume-mounts.
  prometheus:
    build: './prometheus'
    container_name: 'prometheus'
    ports:
      - '9090:9090'

  # The grafana container uses the image resulting from the build
  # of `./grafana` which simply packs some configuration into the
  # form of an image.
  #
  # This could instead be a volume-mounted container.
  grafana:
    build: './grafana'
    container_name: 'grafana'
    ports:
      - '3000:3000'

  # You'd typically not run `node_exporter` in a container like
  # this.
  # If so, make sure you're using `network_mode=host` to give the
  # host network namespace to the containers as well as `pid=host`. 
  # 
  # Eventually it would also make sense to extend it with other 
  # privileges and mounts. 
  #
  # I've never run it in a container so I can't endorse doing so.
  node_exporter:
    image: 'quay.io/prometheus/node-exporter'
    container_name: 'node_exporter'

With that we end up with the following scenario:

Note that there’s an extra component: database.

Grafana records dashboard modifications, alerts, organizations, users, and more (see grafana/pkg/services/sqlstore). All of this needs some persistence.

In the past, Grafana used to use Elasticsearch for that (it was more like a Kibana extension back then). Nowadays, when nothing is configured, a sqlite3 database is created by the Grafana binary, but you’re also free to use either mysql or postgresql as external databases.

Configuring Prometheus

To have a well configured datasource, Prometheus needs some configuration (checkout the configuration at sample-grafana/prometheus/config.yml):

global:
  scrape_interval: '5s'
  evaluation_interval: '5s'

scrape_configs:

  # `node` takes care of scraping node_exporter instances
  # that gives us all sorts of information about the host
  # in which the exporter runs.
  - job_name: 'node'
    static_configs:

      # Given that we're only testing a single instance that
      # we know the address very well, statically set it here
      # with a custom instance label so that it looks better
      # in the ui (and metrics).
      - targets:
        - 'node_exporter:9100'
        labels:
          instance: 'instance-1'

This allows us to query Prometheus and get the metrics from node_exporter that get the scrape targets registered with the node job:

curl http://localhost:9090/api/v1/query\?query\=node_memory_MemAvailable

{
    "data": {
        "result": [
            {
                "metric": {
                    "__name__": "node_memory_MemAvailable",
                    "instance": "instance-1",
                    "job": "node"
                },
                "value": [
                    1518184109.51,
                    "7857156096"
                ]
            }
        ],
        "resultType": "vector"
    },
    "status": "success"
}

If you’re new to Prometheus, I highly recommend Prometheus: Up & Running from Brian Brazil.

Now we’re set to go configure Grafana.

Configuring Grafana

The new release of Grafana introduced and extra configuration path: provisioning.

########## Paths ###########
[paths]
# folder that contains provisioning config files 
# that grafana will apply on startup and while running.
provisioning = conf/provisioning

This provisioning directory can have other two directories inside it:

datasources: holds a list of .yaml files that configure data sources (e.g, Prometheus instance …).
dashboards: holds a list of .yaml files with configuration files that tell Grafana where to look for dashboards to initialize on startup;

Starting with datasources, this is where I tell Grafana where it should look for Prometheus:

# provisioning/datasources/all.yml
# https://github.com/cirocosta/sample-grafana/blob/master/grafana/provisioning/datasources/all.yml

datasources:
-  access: 'proxy'                       # make grafana perform the requests
   editable: true                        # whether it should be editable
   is_default: true                      # whether this should be the default DS
   name: 'prom1'                         # name of the datasource
   org_id: 1                             # id of the organization to tie this datasource to
   type: 'prometheus'                    # type of the data source
   url: 'http://prometheus:9090'         # url of the prom instance
   version: 1                            # well, versioning

With that set, whenever Grafana starts up it picks the datasource (see pkg/services/provisioning/datasources if you want to know more by checking the source code) and applies it to its database.

ps.: This datasources file can also contain a delete_datasources object which would tell Grafana about datasources to remove from its database (assuming you’re not starting from a fresh database).

As just having the data source is not enough, we’d then configure the provisioning/dashboards directory. There you can add a file describing how Grafana can pick dashboards whenever it starts up.

# provisioning/dashboards/all.yml
# https://github.com/cirocosta/sample-grafana/blob/master/grafana/provisioning/dashboards/all.yml

- name: 'default'       # name of this dashboard configuration (not dashboard itself)
  org_id: 1             # id of the org to hold the dashboard
  folder: ''            # name of the folder to put the dashboard (http://docs.grafana.org/v5.0/reference/dashboard_folders/)
  type: 'file'          # type of dashboard description (json files)
  options:
    folder: '/var/lib/grafana/dashboards'       # where dashboards are

With that configured, make sure your dashboards are in the place they should be (/var/lib/grafana/dashboards in this example) and you’re ready to go.

Example

Clone the cirocosta/sample-grafana repository.

This repository contains the file structure described above.

With make run you can take the infrastructure up (node_exporter, grafana and prometheus) if you have docker and docker-compose installed.

# clone the repo and get there
git clone https://github.com/cirocosta/sample-grafana
cd ./sample-grafana

# run the infra
make run

# open the grafana web page
open http://localhost:3000

Now you should see a login screen (which you can enter with admin as user and admin as password).

After logged in, voilà, the datasource has been already configured and you already have a dashboard (mydashboard) set:

Heading towards the dashboard, we can see that our scrape target is really being scraped and that Grafana is properly retrieving the information it needs from Prometheus (as the data source has been properly configured):

Updating dashboards

As the files that represent the dashboards are all JSON files, the easiest way to get them with the modifications you performed is targetting the Grafana API with your credentials and saving the JSON files to the directory.

If you’re managing just one dashboard or another you can do this by hand though - go to settings and retrieve the JSON for the dashboard:

If you’re managing some more (which I guess it’s the case), there’s a better way of doing it.

Scripting dashboard retrieval

That manual process I described is not all that hard to be scripted.

By inspecting the network requests issued by the browser, we can notice that Grafana exposes some interesting endpoints via the /api path:

/api/search             allows us to retrieve different types of objects
                        related to our account / organization, including
                        one that really matters to us: dash-db (dashboards
                        db I guess)

/api/dashboards/db      retrieves the configuration of a dash-db object

(the routes are registered under grafana/grafana/pkg/api/api.go)

So the idea of the script is:

retrieve a list of dashboard names (get it from dash-db objects in the list that /api/search gives us);
for each dashboard, get its configuration;
save the configuration to a file in a specific location.

The script looks like this (see update-dashboards.sh):

#!/bin/bash

# Updates local dashboard configurations by retrieving
# the new version from a Grafana instance.
#
# The script assumes that basic authentication is configured
# (change the login credentials with `LOGIN`).
#
# DASHBOARD_DIRECTORY represents the path to the directory
# where the JSON files corresponding to the dashboards exist.
# The default location is relative to the execution of the
# script.
#
# URL specifies the URL of the Grafana instance.
#

set -o errexit

readonly URL=${URL:-"http://localhost:3000"}
readonly LOGIN=${LOGIN:-"admin:admin"}
readonly DASHBOARDS_DIRECTORY=${DASHBOARDS_DIRECTORY:-"./grafana/dashboards"}


main() {
  local dashboards=$(list_dashboards)
  local dashboard_json

  show_config

  for dashboard in $dashboards; do
    dashboard_json=$(get_dashboard "$dashboard")

    if [[ -z "$dashboard_json" ]]; then
      echo "ERROR:
  Couldn't retrieve dashboard $dashboard.
      "
      exit 1
    fi

    echo "$dashboard_json" >$DASHBOARDS_DIRECTORY/$dashboard.json
  done
}


# Shows the global environment variables that have been configured
# for this run.
show_config() {
  echo "INFO:
  Starting dashboard extraction.
  
  URL:                  $URL
  LOGIN:                $LOGIN
  DASHBOARDS_DIRECTORY: $DASHBOARDS_DIRECTORY
  "
}


# Retrieves a dashboard ($1) from the database of dashboards.
#
# As we're getting it right from the database, it'll contain an `id`.
#
# Given that the ID is potentially different when we import it
# later, to make this dashboard importable we make the `id`
# field NULL.
get_dashboard() {
  local dashboard=$1

  if [[ -z "$dashboard" ]]; then
    echo "ERROR:
  A dashboard must be specified.
  "
    exit 1
  fi

  curl \
    --silent \
    --user "$LOGIN" \
    $URL/api/dashboards/db/$dashboard |
    jq '.dashboard | .id = null'
}


# lists all the dashboards available.
#
# `/api/search` lists all the dashboards and folders
# that exist in our organization.
#
# Here we filter the response (that also contain folders)
# to gather only the name of the dashboards.
list_dashboards() {
  curl \
    --silent \
    --user "$LOGIN" \
    $URL/api/search |
    jq -r '.[] | select(.type == "dash-db") | .uri' |
    cut -d '/' -f2
}

main "$@"

Execute it with the right variables, and you’re done!

Closing thoughts

It’s very nice to see that Grafana is going.

As the de facto tool for graphing I’m very happy that Grafana has been improving so much over time. This new version is even more responsive and has this little feature that really helps.

Congratulations to everybody involved!

If you have any questions or noticed that I made a mistake somewhere, please let me know!

I’m cirowrc on Twitter.

Have a good one!

finis

Simulating AWS tags in local Prometheus

Ciro S. Costa — Thu, 08 Feb 2018 00:00:00 +0000

Hey,

some time ago I had the challenge of setting a monitoring system for some machines in AWS.

I end up taking Prometheus as the system to retrieve the metrics from the sources and Grafana as the way of graphing them. Nothing fancy, pretty standard.

After that, I had a problem though. How could I test this setup locally?

Everything was very much tied to how service discovery worked within an AWS environment. The machines would get fetched from EC2’s DescribeInstances with all the tags that were set up for them. With the relabelling feature, I was able to make use of those tags to improve the metrics and have a better sense of data. How could I replicate that locally?

Check out how I end up solving that.

ps.: this is one way - here are undoubtedly many others.

Tailoring the AWS Prometheus configuration

As each environment has its specificities regarding how to gather information, about where are the machines and what information can be retrieved from they, I created separate configuration files that achieve the same end goal: expose machine metrics, docker daemon metrics as well as container metrics.

prometheus
    └── config
        ├── aws.yml             # aws-specific configuration
        └── vagrant.yml         # vagrant-specific configuration

As each machine has (mostly) the same exporters, they’re all alike with just a few changes.

ps: some machines have different exporters, e.g., load-balancer machines have haproxy_exporter.

Here’s a Prometheus configuration aiming at getting information about AWS machines that enhance the metrics with custom AWS tags:

# AWS configuration

global:
  scrape_interval: '10s'
  evaluation_interval: '10s'

scrape_configs:

  # Collect metrics about the node.
  # 
  # These are plain `node_exporter` metrics that
  # are scrapped to reveal all sorts of machine 
  # metrics like cpu, inode utilization for each fs, 
  # memory, iface statistics ...
  # 
  # In this example I'm not specifying the AWS access token,
  # secret key and regions because the AWS golang client will
  # get it from the environment using the default namings, 
  # leaving us only to need to specify the port to scrape.
  - job_name: 'node'
    ec2_sd_configs:
      - port: 9100
    relabel_configs:

      # Because we could have multiple environments running
      # in the same region or some ephemeral machines that
      # are not meant to be scrapped, filter out those that
      # do not contain a specific tag.
      - source_labels:
        - '__meta_ec2_tag_tips_ops_environment'
        regex: 'dev1'
        action: 'keep'

      # Gather the instance id and use it as the name of the
      # instance.
      #
      # This allows us to not use the weird hostname that aws provides
      # but instead, the instance ID that AWS we can gather from the
      # service discovery.
      - source_labels:
        - '__meta_ec2_instance_id'
        target_label: 'instance'

      # To enhance the intel and understand better what's going on,
      # here I also retrieve an extra tag that the machines are 
      # supposed to have.
      # 
      # In this example I set `tips_ops_node_type` to the machine
      # tag (e.g., m4.4xlarge) but this could be anything.
      # 
      # Check `local-aws-prom/infrastructure/main.tf` to see the
      # tag being set.
      - source_labels:
        - '__meta_ec2_tag_tips_ops_node_type'
        target_label:  'tips_ops_node_type'

You can see that there I retrieve a custom tag that I set in the machine and then enhance the metrics with it. That tag, however, comes from ec2_sd (the EC2 service discovery).

Once Prometheus starts with such configuration, we can see our relabelling taking place. Having two instances, one with the proper tags and another without, the first gets accepted and the second, dropped.

More than that, the first gets some extra labels extracted from the EC2 tags.

Preparing the local configuration

To test that locally we can use static_config labeling for each target.

For instance, this is how the Vagrant Prometheus configuration would look like:

# VAGRANT configuration
   
global:
  scrape_interval: '5s'
  evaluation_interval: '5s'

scrape_configs:

  # Collect metrics about the node.
  # 
  # These are plain `node_exporter` metrics that
  # are scrapped to reveal all sorts of machine 
  # metrics like cpu, inode utilization for each fs, 
  # memory, iface statistics ...
  - job_name: 'node'

    relabel_configs:

      # Gather the instance id and use it as the name of the
      # instance.
      #
      # This allows us to not use `${ADDR}:${PORT}` in the instance
      # names but instead, the instance ID that AWS gives to us.
      #
      # In this case, a fake `i-<instance>` identifier.
      - source_labels:
        - '__meta_ec2_instance_id'
        target_label: 'instance'

      # Collect the type of the node and add it as a tag to our
      # metrics. 
      #
      # This allows us to make use of node types in our
      # queries and dashboards (e.g., if you want to graph the
      # average CPU usage of a specific machine type like
      # m4.4xlarge)
      - source_labels:
        - '__meta_ec2_tag_tips_ops_node_type'
        target_label:  'opstips_node_type'


    # Machine configuration acting as if we were in aws.
    #
    # This has the effect of emulating the AWS labels that
    # the aws service discovery would give us.
    #
    # Note that here I'm setting more information
    # than I actually use - not all of these are
    # being used in the relabelling phase so those
    # not used will be dropped and not persisted.
    #
    # Because I'm simulating a two-node scenario
    # I'm listing only 2 static addresses here with
    # the desired labels. If you needed to test against more
    # VMs, this would need to be updated. Probably a better
    # way would be setting up DNS service discovery with
    # a custom DNS server locally.
    static_configs:
      - targets:
        - '${MACHINE_IP}:9110'
        labels:
          __meta_ec2_tag_tips_ops_environment: 'dev1'
          __meta_ec2_tag_tips_ops_node_type: 'opstips-fleet-masters'
          __meta_ec2_instance_type: 't2.medium'
          __meta_ec2_instance_id: 'i-inst1'

      - targets:
        - '${MACHINE_IP}:9120'
        labels:
          __meta_ec2_tag_tips_ops_environment: 'dev1'
          __meta_ec2_tag_tips_ops_node_type: 'opstips-fleet-slaves'
          __meta_ec2_instance_type: 't2.medium'
          __meta_ec2_instance_id: 'i-inst2'

To have two machines running with node_exporter I setup a little Vagrantfile which would provision them as wanted:

# Provisioning script that downloads the node_exporter
# binary from the GitHub releases page and then initiates
# it with a simple `nohup`.
#
# ps.:  for something more robust you'd typically prepare
#       a systemd service and have it enabled to keep it running
#       even between restarts.
$script = <<SCRIPT
URL=https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz

curl \
    -o /tmp/node_exporter.tgz \
    -SL \
    $URL
    
tar \
    xzf /tmp/node_exporter.tgz \
    -C /tmp/ \
    --strip-components=1

nohup /tmp/node_exporter &
SCRIPT


# Prepare the machine configurations.
# Here I'm statically configuring what are the ports
# that will be exposed such that in our prometheus configuration
# we can target them directly without needing DNS or any other
# discovery mechanisms.
Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/artful64"
  config.vm.box_check_update = false
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.provision "shell", inline: $script

  config.vm.define "machine1" do |machine1|
    machine1.vm.network "forwarded_port", guest: 9100, host: 9110
  end

  config.vm.define "machine2" do |machine2|
    machine2.vm.network "forwarded_port", guest: 9100, host: 9120
  end

  config.vm.provider "virtualbox" do |v|
    v.memory = 512
    v.cpus = 1
  end
end

After a vagrant up, we should have both machines running with node_exporter in the respective ports:

curl localhost:9110/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
...

curl localhost:9120/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
...

Now get prometheus up (in my case I tailored a Dockerfile that takes care of adding the right configuration depending on the environment - aws or vagrant), and we should be able to see the targets:

And, that’s it!

With the targets being properly scrapped and with Prometheus performing the right label transformation, we’re all set.

A Grafana dashboard that makes use of those labels would be able to run properly against both AWS and local instances.

As it can be seen in the above comparison, the labeling works great.

Closing thoughts

Even though it’s fairly simple to get some instances running in AWS to prepare dashboards and test Prometheus alert rules, I find useful to be able to run it locally. With a set up like this, you can test relabelling easily without the need of any AWS instances.

Naturally, this technique can be applied to other cloud providers.

Have you been monitoring instances in AWS with Prometheus? Did you spot a mistake in the article? Please let me know!

I’m cirowrc on Twitter and would be very happy to answer any questions or just hear something about the article.

Thanks!

Have a good one,

finis

What happens when a pipeline is created in Concourse CI?

Ciro S. Costa — Mon, 05 Feb 2018 00:00:00 +0000

Hey,

this blog post is the continuation of the previous post (How to build and run Concourse CI locally). In this one, I try to understand better how one of its main components works internally to keep track of pipelines and have builds triggered.

Let’s dig into what goes on when a pipeline is submitted to Concourse.

The API

As a consumer of the “concourse service”, the API server is the first point that we touch. The API server in this case is atc:

From the words of the atc repository:

atc is the brain of Concourse. It’s responsible for scheduling builds across the cluster of workers, providing the API for the system, as well as serving the web interface.

In the last post, when we submitted a pipeline to Concourse, atc was the component that the cli (fly) was sending it to.

To have an example, let’s take that very same pipeline:

jobs:
- name: 'hello'
  plan:
  - task: 'say-hello'
    config:
      platform: 'linux'
      image_resource:
        type: 'docker-image'
        source: {repository: 'alpine'}
      run:
        path: 'echo'
        args: ['hello']

and then submit it using fly:

fly \
        set-pipeline \
        --pipeline=hello-world \
        --config=./hello.yml \
        --target=local

Fly is interesting in its own; in summary it does the follow (see commands/set_pipeline and commands/internal/setpipelinehelpers):

creates an internal representation of the new configuration submitted (parses the yaml it got from --config);
loads a possible existing configuration for that pipeline (asks atc for it based on the pipeline name);
performs a diff between the existing pipeline and the new one;
shows a message highlighting these diffs and then asks for confirmation to proceed;
applies the configuration by making use of the concourse client go-concourse ;
the configuration save call made from go-concourse touches the following endpoint in atc: /api/v1/teams/:team_name/pipelines/:pipeline_name/config".

Now that the configuration landed in atc, api/configserver/save.go#SaveConfig gets touched.

This piece does two things:

performs a second validation (fly already validated it, but as this api might be public it takes the defensive approach);
saves the configuration.

Fair enough, but, what “saves the configuration” means?

Saving the pipeline configuration

Saving such configuration means making the pipeline persistent.

The process of saving the pipeline is based on two possibilities:

we’re creating a plain new pipeline;
there’s already a pipeline with such name.

In the first case, it sets a row in the pipelines table and creates an extra table to keep track of build events for that pipeline (pipeline_build_events_<pipeline_id>d) - pretty neat. Once that’s done, it starts populating the other tables (jobs, resources, resource_types, …).

In the case that there’s already a pipeline established for that name, an update is performed, all previous jobs, resources and resource_types are made inactive, and task caches are cleared. Once that’s done, now it populates the jobs, resources, … tables with active entries (such that in the end, it has only the new ones marked as active).

What’s in the DB?

After we ran set-pipeline, we should have some stuff filled in our database.

# log in as `postgres` (we know we don't need password
# because of the `atc` initialization at `./dev/atc`)
psql \
        --username=postgres \
        --no-password


# list the databases that we have
\l
                                 List of databases
   Name    |
-----------+
 atc       |    <<<<
 postgres  |
 template0 |
           |
 template1 |
           |
(4 rows)


# connect to the ATC database
\c atc
You are now connected to database "atc" as user "postgres".


# list all the tables that are there
\dt                                               
                                          
|         Name             |
+--------------------------+
| base_resource_types      |
| build_events             |
...
| jobs_serial_groups       |
| next_build_inputs        |
| pipeline_build_events_2  | <<< the generated build_events table
| pipelines                | <<< our pipeline should be here
...
| worker_task_caches       |
| workers                  |
(32 rows)


# Retrieve all the pipelines saved
SELECT * FROM pipelines;
 version | id |    name     | paused | ordering |        last_scheduled         | team_id | public | groups 
---------+----+-------------+--------+----------+-------------------------------+---------+--------+--------
       3 |  2 | hello-world | f      |        2 | 2018-02-05 18:58:00.205094-05 |       1 | f      | null
(1 row)

While that’s interesting per se, looking at the table definitions, we can understand a bit more about how data is structured internally by looking at the references.

Referenced by:

    # `builds` keeps track of the build executions that
    # are initiated. It has fields that mark the status
    # of the build, what's the engine that ran the
    # tasks, if it was manually triggered or not,
    # which pipeline the build belongs to ...
    TABLE "builds" 
        CONSTRAINT "builds_pipeline_id_fkey" 
        FOREIGN KEY (pipeline_id) 
        REFERENCES pipelines(id) 
        ON DELETE CASCADE

    TABLE "jobs" 
        CONSTRAINT "jobs_pipeline_id_fkey" 
        FOREIGN KEY (pipeline_id) 
        REFERENCES pipelines(id) 
        ON DELETE CASCADE

    TABLE "resource_types" 
        CONSTRAINT "resource_types_pipeline_id_fkey" 
        FOREIGN KEY (pipeline_id) 
        REFERENCES pipelines(id) 
        ON DELETE CASCADE

    TABLE "resources" 
        CONSTRAINT "resources_pipeline_id_fkey" 
        FOREIGN KEY (pipeline_id) 
        REFERENCES pipelines(id) 
        ON DELETE CASCADE

And then to understand what are the most critical queries (or at least guess), look at the indices:

Indexes:
    "pipelines_pkey" 
        PRIMARY KEY, btree (id)

    "pipelines_name_team_id" 
        UNIQUE CONSTRAINT, 
        btree (name, team_id)

    "pipelines_team_id" 
        btree (team_id)

What’s next?

The next step is getting to know more about how does a work register itself against tsa, another component that makes up the architecture.

I’d like to dig deep into how that process looks like and what are the components involved.

If you’re curious about it too, check out the next posts!

Let me know if you have any questions or comments. I probably got something wrong here, and I’d really like to know about it - I’m cirowrc on Twitter, feel free to reach out.

Have a good one!

finis

How to build and run Concourse CI locally

Ciro S. Costa — Sun, 04 Feb 2018 00:00:00 +0000

Hey,

close to a year ago I got in touch with concourse.ci when evaluating a tool to perform builds whenever changes happened in a git repository and even though the team I worked with end up creating a custom tailored system (we needed much less than what concourse - or something like Jenkins - provided) I think Concourse does a lot right.

This post is not meant to teach you the concepts behind Concourse or provide use cases, but let you know how you can run a development version of it locally if you’re willing to contribute to the project.

Setting up the development environment

Concourse (and the CloudFoundry project as a whole it seems) performs vendoring using git submodules.

There’s always a repository somewhere that aggregates dependencies and gives a consistent view of how things group together. In the case of Concourse, it’s github.com/concourse/concourse.

To the extent of this post there’s no need to run all the parts of Concourse (as we’re only exploring atc and PostgreSQL), but before breaking things apart, I think it’s good to get an example running.

Pick that repo and put it somewhere in your FS (doesn’t need to be under your current $GOPATH) and then initialize the submodules (if you have any doubts, check the CONTRIBUTING.md file at this repository):

# clone the repository that aggregates the whole
# project and its dependencies
git clone https://github.com/concourse/concourse
cd concourse

# initialize the submodules
git submodule update --init --recursive

# if not using direnv (https://github.com/direnv/direnv),
# execute the commands described under `.envrc` file in
# the current shell session (that `.envrc` is essentially
# setting the GOPATH go the `./src` directory under 
# concourse/concourse.
source .envrc

With that set, I update the ./dev/Procfile file which describes how to run the dockerized version of the postgres database and make atc point to the DB instead of a local instance:

--- a/dev/Procfile
+++ b/dev/Procfile
@@ -1,4 +1,4 @@
-atc: ./atc
+atc: ./atc-dockerdb
 tsa: ./tsa
 worker: ./worker
-db: ./db
+db: ./dockerdb

With that set, start:

./dev/start

Using default tag: latest
latest: Pulling from concourse/con...
Digest: sha256:ac10eadf32798da0567...
Status: Image is up to date for co...
10:21:25    atc | Starting atc on ...
10:21:25    tsa | Starting tsa on ...
10:21:25     db | Starting db on p...
10:21:25 worker | Starting worker ...
10:21:26     db | 2939be72c6e24ddb...
10:21:26    tsa | {"timestamp":"15...
10:21:26     db | Terminating db
10:21:26 worker | {"timestamp":"15...
10:21:28    atc | creating databas...
10:21:31    atc | creating databas...

Using FLY locally

To interact with the Concourse API (provided by atc) we use the fly command line interface (CLI). It’s the piece that makes use of go-concourse to communicate with ATC in a standard fashion.

Installing it is simple after all submodules have been initialized. Head to concourse/concourse/src/github.com/concourse/fly and build it:

# Make sure you have  your `GOPATH` properly set by sourcing 
# the `.envrc` file
source .envrc

# get to the directory where the source of the FLY command
# line is. 
# This should be there after you've installed the submodules.
cd ./src/github.com/concourse/fly

# By using `go install` we'll have the final result of
# the build in `$GOPATH/bin` which should be in your
# $PATH after sourcing `.envrc`. 
go install -v

# check if everything is working
fly --version
0.0.0-dev

With the CLI can now create a target that we’ll use to perform the commands against:

# Authenticates against a concourse-url (`127.0.0.1:8080`) and 
# saves the auth details under an alias (`local`) in `~/.flyrc`.
#
# As we're not specifying a `team` (`--team-name`) it'll use the default
# one (`main`) - a team can be thought as a namespace under which pipelines
# and builds fall into.
#
# When testing concourse locally (and not dealing with auth) we can stay
# with `main` (which is also an administrator team) and not worry about
# auth at all (`atc` is being initialized with the `--no-really-i-dont-want-any-auth`
# flag).
fly \
        login \
        --concourse-url http://127.0.0.1:8080 \
        --target local 

# Create a pipeline
echo '
jobs:
- name: hello
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: alpine}
      run:
        path: echo
        args: ["hello"]
' > /tmp/hello.yml

# Submit the pipeline
 fly \
        set-pipeline \
        --pipeline hello-world \
        --config ./hello.yml \
        --target local 

# List the pipelines to make sure it's there
fly \
        pipelines \
        --target local
name         paused  public
hello-world  yes     no

Ok, everything is set up!

Running the pipeline from concourse web application

At the first time you shoot localhost:8080 on your browser, no pipeline will show:

That’s because our sample pipeline is not public and we’re not authenticated.

Head to login (http://localhost:8080/login) and just click right in main and login. Because atc has been put up with --no-really-i-dont-want-any-auth, no questions will be asked.

The next step is enabling the pipeline (unpausing it).

Once that’s done, select the pipeline and start a new build:

It should now be started:

I’m still now very into what’s going on at this moment but my guess is:

atc received a build request
given that there’s a worker that registered itself against tsa, atc schedules the build to that worker (as it matches the constraints - platform == linux)
a container is started at that worker somehow, which then reports back to atc the progress of the build
that’s all persisted and also reproduced to the ui something.

ps.: I might be totally wrong here - check out the next posts to know if that’s the case!

Some seconds later, we see that it succeeded:

What’s next?

The next step is getting into what’s going on when we create a new pipeline.

Concourse is made of a database component - when we create a new pipeline, how does that interaction look like?

That’s something to be covered next.

Please let me know what you think about Concourse - have you already used it?

I’m cirowrc on Twitter, let me know if you have any questions!

Have a good one,

finis

Creating a simple extension to block websites

Ciro S. Costa — Fri, 26 Jan 2018 00:00:00 +0000

Hey,

since some months ago I’ve been struggling a bit with the amount of information that I consume in a single day.

It’s just outrageous that sometimes I’d put myself in a zombie mode and just keep following news (even if something like HackerNews which, in theory, should be a good source of interesting information) for so long and not really produce much. I decided to try blocking all these sources for a while and see how it goes.

To do so, I looked for some Chrome extensions that would do the job.

There are several of them out there, as you might expect, but why not give a try implementing it? I’d learn something new (and procrastinate with a “reason”).

td;dr: check out this example: https://developer.chrome.com/extensions/webRequest#examples.

Capturing HTTP requests

Having worked with Firefox extensions before, I knew this couldn’t be all that hard.

From the several Javascript APIs for extensions that are out there, there’s one that is very suitable for our needs: webRequest (yeah, I’m linking to MDN - nowadays the Firefox extension ecosystem uses the same standardized APIs as Chrome does).

From the description of it:

Use the chrome.webRequest API to observe and analyze traffic and to intercept, block, or modify requests in-flight.

Exactly what we need.

Given an HTTP request initiated by the browser we can use this API to mess with it at any point in the request pipeline:

image from MDN: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest

Because we’re aiming at blocking a specific set of requests we can pick the very first one: onBeforeRequest.

onBeforeRequest (optionally synchronous) fires when a request is about to occur. This event is sent before any TCP connection is made and can be used to cancel or redirect requests. (https://developer.chrome.com/extensions/webRequest)

In practice, that means adding the following to a javascript file:

// Limit the requests for which events are
// triggered.
//
// This allos us to have our code being executed
// only when the following URLs are matched.
// 
// ps.: if we were going to dynamically set the
//      URLs to be matched (used a configuration
//      page, for example) we'd then specify the 
//      wildcard <all_urls> and then do the filtering
//      ourselves.
const filter = {
  urls: [
    '*://news.ycombinator.com/*',
  ],
}

// Extra flags for the `onBeforeRequest` event.
//
// Here we're specifying that we want our callback
// function to be executed synchronously such that
// the request remains blocked until the callback 
// function returns (having our filtering taking 
// effect).
const webRequestFlags = [
  'blocking',
];

// Register our function that takes action when a request
// is initiated and matches the provided filter that we
// specified in the options.
//
// Because we outsourced the URL filtering to chrome itself
// all we need to do here is always cancel the request (as
// it matches the filter of unwanted webpages).
window.chrome.webRequest.onBeforeRequest.addListener(
  page => {
    console.log('page blocked - ' + page.url);

    return {
      cancel: true,
    };
  },
  filter,
  webRequestFlags,
);

The functionality can be illustrated as follows:

That’s it!

Closing thoughts

It’s pretty evident that this isn’t suitable for the case where you want to change the list of URLs to the blacklist regularly. There’s a reason why: I don’t keep changing the source of my distractions (lol).

Another point to be made is that it’s effortless to get rid of it: just remove the extension. The rationale for not putting much thought into making it hard to remove is that when impulsively visiting a website like Twitter I’m not totally freaking out of control of myself - I just got into the habit and by breaking the cycle I can stop it (this comes from the great book The Power of Habit).

If you have any questions, just drop by Twitter and let me know - I’m @cirowrc there. You can also subscribe to the mailing list below to receive some up to date content from time to time.

Have a nice one!

finis

Replacing HAProxy ACLs by HAProxy Maps

Ciro S. Costa — Thu, 18 Jan 2018 00:00:00 +0000

Hey,

This is a quick tip on how you can switch from using ACLs (access control lists) to use maps for selecting backends based on a request parameter.

The usefulness of using maps instead of acls is that sometimes it might be easier to update the mappings rather than the ACLs (as well as allowing us to perform mutations on the maps without needing to spawn a new process due to the possibility of using the runtime api via unix sockets).

Overview

As an example, consider the job of selecting a backend based on a destination host as advertised by the Host header.

That backend might be configured with a great number of servers that are meant to deal with those requests arriving for that host.

For instance, a client performs a request against our load-balancer (HAProxy) which lives under a known address (test.com):

GET /hello HTTP/1.1
Host: test.com
User-Agent: Chrome

Once that requests lands on HAProxy, that HTTP request is then parsed and we’re able to make use of the HTTP headers to determine which set of servers (i.e., which backend) should take care of that request (one possibility is not matching a backend and then sending the traffic to a no-op “nomatch” backend that just replies back with 404).

The usual way of selecting a backend is using ACLs (access control lists) - for a more in-depth documentation of ACLs, see ACL basics on the official HAProxy documentation.

Selecting a backend using ACLs

To get started, let’s create a minimal configuration that can handle our example:

# Apply the `mode http` to every frontend and backend that
# doesn't specify a mode.
# 
# This is useful for us just to don't repeat ourselves throughout
# the config.
defaults
                mode http



# This main http frontend will be responsible for handling every
# request that comes to our load-balancer on port 80 (the default
# http port).
frontend        http
                bind                        *:80 

# Simple ACL to send traffic to our custom tailored web server.
#
# It looks at the `Host` header of the request, and if it matches
# `test.com`, then checking that `acl` will evaluate to `true`. 
                acl                         host_matches hdr_dom(host) test.com
# Make use of the backend `desired_backend` if the acl `host_matches`
# evaluates to `true`.
                use_backend                 desired_backend if host_matches
# By default use the `not_found` no-op backend.
                default_backend             not_found



# A static backend that will just serve some files from a local server
backend         desired_backend
                server                      myserver localhost:8000



# A dumb "not-found" auto-responder using HAProxy's errorfile
# directive.
#
# As the ACL that send traffic to this backend is the default
# (and least prioritized), when we reach this backend, it means
# that we didn't find a desired backend, thus we should serve
# a 404 instead of the generic 503 that haproxy gives back to
# clients when something goes wrong.
backend         not_found
                errorfile                   503 /tmp/sample/errorfiles/404.http

With that configuration, you can take HAProxy up and start seeing the requests with Host set to test.com succeeding and those with a different Host (for instance, Host: localhost) failing (404).

Looking at what’s going on we can see that that’s a job that is very well suited for a map lookup.

Selecting a backend using Maps

Using maps, we can remove acl completely and just rely on use_backend with the map lookup syntax.

defaults
                mode http

frontend        http
                bind                        *:80 

# Use the backend found in the map whose key matches the
# `Host` header.
#
# If nothing is found, use the default `not_found` backend
# (which will 404).
                use_backend	%[req.hdr(host),lower,map_str(/tmp/sample/maps/map1,not_found)]


backend         desired_backend
                server                      myserver localhost:8000


backend         not_found
                errorfile                   503 /tmp/sample/errorfiles/404.http

As you can see, we can simplify the configuration by making use of the map, but we need to know in advance what are the available backends and also have a map file.

The map file is far from complicated though - it’s a simple line-by-line key-value file where each key is separated from its value using a space:

test.com desired_backend
another-host.com another_backend
another-blabla.com haha_backend
...

Whenever you spin up HAProxy (or tell it to reload), it looks at this file and then puts that mapping into memory such that it can very quickly perform the lookups.

One way to make sure that your map has been properly loaded by issuing Unix Socket Commands against the HAProxy runtime API:

 # Make haproxy create a stats and runtime api socket
 # at `/tmp/haproxy.sock` with admin permissions
+ global
+                 stats socket /tmp/haproxy.sock mode 666 level admin

# Issuing the `show map` command to the running HAProxy
# instance.
#
# Here I pipe to a privileged instance of `socat` such
# that we can properly access the `/tmp/haproxy.sock`
# unix socket.
#
# If you didn't need extra privileges to raise HAProxy,
# then you don't need `sudo`.
#
# In the output you can see an internal address and
# the key-value pair.
echo "show map /tmp/sample/maps/map1" | \
        sudo socat stdio /tmp/haproxy.sock          
0x7fbb72d013d0 test.com desired_backend

One of the benefits of using maps is that we can use the Unix sockets API to modify the map without needing to restart the HAProxy process (there’s no easy way of changing the ACLs with the unix socket api):

# Send the `add map` command through the unix socket so that 
# internally, HAProxy will update the map.
#
# Note that if HAProxy restarts, it won't know about this
# newly added domain - it'll look for the mappings specified
# at the file and that'll be its source of truth.
#
# To keep consistency, make sure you also modify the map file.
echo "add map /tmp/sample/maps/map1 another-host.com another_backend" | \
        sudo socat stdio /tmp/haproxy.sock

# Check the newly added domain
echo "show map /tmp/sample/maps/map1" | \
        sudo socat stdio /tmp/haproxy.sock
0x7fbb72d013d0 test.com desired_backend
0x7fbb72d02a30 another-host.com another_backend

Closing thoughts

HAProxy maps might offer a great way of reducing (or simplifying) the usage of ACLs in your configuration.

Here I showed one use of it - one that’s actually very common where the name of the backends are well known and updating the map that matches a domain with a backend is needed - but there are certainly many more uses that one can think of.

Please let me know if you need any help or spotted an error - you can find me on Twitter (@cirowrc).

Have a good one!

finis

How to publish a blog using AWS

Ciro S. Costa — Mon, 15 Jan 2018 00:00:00 +0000

Hey,

I just got an idea about a tool, and given that I think it’s a type of thing that pairs nicely with a blog to drive people to it, I decided to create one.

Given that I already have everything set up for this one (ops.tips, I mean), I thought about sharing my setup with you so that you can also adopt it - if you think it’s cool.

ps.: this is not meant to be advertising for AWS. I’m just talking about my use-case here.

update: added a CDN invalidation step to the Travis-CI deployment script - thanks a lot Corey Quinn!

How it looks

The whole process of delivering the content of my blog is based on four steps:

create the content (markdown)
build and publish the content (send generated HTML to somewhere)
serve the build result (accept connections and serve HTML)
invalidating old objects in the edge caches

Naturally, the first case doesn’t matter for this blog post.

Building and publishing the content

All these blog posts are plain markdown files with some custom templating rules.

To get markdown processed and HTML generated I make use of Hugo.

You define a theme for your content and then based on the theme and what you write in some markdown files it generates a specific HTML.

The code that does so (automatically in travis-ci after I push to a specific branch) is fairly simple:

# Build the markdown files and output
# (verbosely) information about the 
# process.
hugo -v

# Go to the directory where the static
# files now live
cd ./public 

# Send the new files to the configured S3 
# (and delete the old ones that are not in
# the new desired state) with a special
# max-age header configuration so that both
# browsers and CDNs cache it for some time.
aws s3 sync \
        --delete \
        --cache-control max-age=10800 \
        ./ s3://$BUCKET

# Create a new invalidation that will simply
# invalidate any objects that we have in our
# website bucket such that we serve fresh content
# to the users that are requests files from the
# edge locations
#
# ps.: users that already downloaded our 
#      resources will still have then until
#      the browser cache gets invalidated
aws cloudfront \
        create-invalidation \
        --distribution-id $DISTRIBUTION_ID \
        --paths '/*'

Once the code has been built and sent to S3, it comes the time to post the content to S3.

Serving the website

As the final htmls lands on S3, it’s a matter of serving it.

S3 allows you to serve your content directly from it (see Hosting a Static website on S3). The problem of doing so is that we can’t configure it very much. It’s not flexible at all.

Another drawback from serving directly from it is that if we have users from a different continent trying to get content from our website, they’ll have to make requests that land in our continent, making their response times not very good.

A solution to that (which AWS already provides and that integrates very smoothly with S3) is to use a CDN (in this case, CloudFront).

To make clear why it makes sense to use CloudFront (or any other CDN, to be honest), we can think of how browsers interact with our servers.

The browser-server interaction

When the user requests our website (say, myblog.com), it has first to get at least one IP that tells it the location to establish a TCP connection to (so that it can make an HTTP request to request a file).

This process is called name resolution.

Once the browser has the IP of the server to contact, it starts a TCP connection which, in our case, will require negotiating some parameters and exchanging a certificate which the browser decides whether it’s trusty or not.

In the case that the browser trusts the certificate received, it can then start sending application-layer requests (in our case, HTTP2 frames).

Having a service like CloudFront makes a huge difference here as this process involves several roundtrips (there’s a great article from CloudFlare about this - Introducing Zero Round Trip Time Resumption (0-RTT)). The less it takes to perform a roundtrip, the better.

Because a CDN has all these points of presence (PoP) all over the world (and it has our certificate in all of them - at least at some point), the TLS termination happens much closer to the client.

In the case that the PoP also has the file you need, it can serve it directly from there without having to have your file coming from a distant location (S3).

Now you might ask me, does it matter? In a typical scenario where you just launch a blog post, do you have all those cache hits?

From the CloudFront data I can tell that yeah, it definitely matters:

Even though my blog is pretty slim and has not many large images (as I optimize them before publishing), it’s still a decent amount of traffic that is not being directed to a far origin.

The actual thing - terraform module

To put all of this into practice, I make use of Terraform to create the necessary resources in AWS.

The module is made of three files: outputs.tf, inputs.tf and main.tf.

inputs.tf defines the variables that are required by the module:

variable "domain" {
  type        = "string"
  description = "name of the domain for the distribution and to alias route53 to"
}

variable "bucket" {
  type        = "string"
  description = "name of the bucket that will hold the files (access logs goes to {bucket}-log"
}

variable "storage-secret" {
  type        = "string"
  description = "a secret user-agent that is sent to all requests from CF to S3"
}

variable "acm-certificate-arn" {
  type        = "string"
  description = "the aws resource number of the certificate that has been manually provisioned"
}

Those variables are used in the main.tf which manages the resources:

# Creates the bucket that will hold logs about
# access to S3 objects.
# The number of access shouldn't be very big given
# that we're caching things at the edge.
#
# In theory, it should have, per day, a maximum of
# number_of_pops*number_of_objects log lines coming
# from S3.
#
# As we also log all the requests from CloudFlare, we
# should have `n` entries for CDN access (as the requests
# are recorded).
#
# To not keep all of these log records that
# we will almost never look at we transition
# these log files to infrequent access storage
# to reduce costs after 30 days.
resource "aws_s3_bucket" "website_log_bucket" {
  bucket        = "${var.bucket}-logs"
  acl           = "log-delivery-write"
  force_destroy = true

  lifecycle_rule {
    enabled = true

    transition {
      days          = "30"
      storage_class = "STANDARD_IA"
    }
  }
}

# The main bucket that holds the actual HTML files
# that are generated.
#
# It defines what's an error document (404.html) and
# what's the `index.html` of it (if we wanted to access
# the website from S3 directly).
#
# It has versioning enabled such that when we modify
# or delete files they are not really deleted and then
# we can go back if we want.
#
# After 15 days we expire the non-current versions as
# at that point it's probably all good there.
#
# Here we also set the bucket policy so that we only
# allow the gets from those having a secret user-agent
# and is a request from aws.
resource "aws_s3_bucket" "website_bucket" {
  bucket = "${var.bucket}"

  website {
    index_document = "index.html"
    error_document = "404.html"
  }

  versioning {
    enabled = true
  }

  lifecycle_rule {
    enabled = true

    noncurrent_version_expiration {
      days = "15"
    }
  }

  logging {
    target_bucket = "${aws_s3_bucket.website_log_bucket.id}"
    target_prefix = "website-bucket-logs/"
  }

  policy = <<POLICY
{
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:UserAgent": "${var.storage-secret}"
                }
            },
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Resource": "arn:aws:s3:::${var.bucket}/*",
            "Sid": "PublicReadAccess"
        }
    ],
    "Version": "2012-10-17"
}
POLICY
}

# A user that we can use to perform the deployment.
#
# The idea here is that if we want to put the deployment
# process (i.e., uploading of new contents to S3) in a
# CI system we don't want to give it many privileges -
# just the bare minimum which is `put` and `list` for that
# bucket.
resource "aws_iam_user" "main" {
  name = "${var.bucket}-deployer"
}

# Create ACCESS and SECRET keys that we can feed our tools
resource "aws_iam_access_key" "main" {
  user = "${aws_iam_user.main.name}"
}

# Assign a policy to the user that restricts its actions.
resource "aws_iam_user_policy" "main" {
  name = "${aws_iam_user.main.name}-deploy-policy"
  user = "${aws_iam_user.main.name}"

  policy = <<POLICY
{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::${var.bucket}"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::${var.bucket}/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}
POLICY
}

# Create the CloudFront distribution that:
# - has IPV6 enabled
# - is spread across all the PoPs (PriceClass_All)
# - has a default root object for our `/` requests
# - sends requests to the origin with our secret header
# - is tied to the S3 origin
# - has logging configured
# - has caching enabled
# - enforces https
# - sets a minimum TLS version
resource "aws_cloudfront_distribution" "website_cdn" {
  enabled             = true
  is_ipv6_enabled     = true
  price_class         = "PriceClass_All"
  http_version        = "http2"
  default_root_object = "index.html"

  "origin" {
    origin_id   = "origin-bucket-${aws_s3_bucket.website_bucket.id}"
    domain_name = "${aws_s3_bucket.website_bucket.website_endpoint}"

    custom_origin_config {
      origin_protocol_policy = "http-only"
      http_port              = "80"
      https_port             = "443"
      origin_ssl_protocols   = ["TLSv1"]
    }

    custom_header {
      name  = "User-Agent"
      value = "${var.storage-secret}"
    }
  }

  logging_config {
    include_cookies = false
    bucket          = "${aws_s3_bucket.website_log_bucket.bucket_domain_name}"
    prefix          = "cdn-logs/"
  }

  custom_error_response {
    error_code            = "404"
    error_caching_min_ttl = "360"
    response_code         = "404"
    response_page_path    = "/404.html"
  }

  "default_cache_behavior" {
    min_ttl                = "0"
    default_ttl            = "3600"
    max_ttl                = "3600"
    target_origin_id       = "origin-bucket-${aws_s3_bucket.website_bucket.id}"
    viewer_protocol_policy = "redirect-to-https"
    compress               = true

    allowed_methods = [
      "GET",
      "HEAD",
      "DELETE",
      "OPTIONS",
      "PATCH",
      "POST",
      "PUT",
    ]

    cached_methods = [
      "GET",
      "HEAD",
    ]

    "forwarded_values" {
      query_string = "false"

      cookies {
        forward = "none"
      }
    }
  }

  "restrictions" {
    "geo_restriction" {
      restriction_type = "none"
    }
  }

  "viewer_certificate" {
    acm_certificate_arn      = "${var.acm-certificate-arn}"
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1"
  }

  aliases = [
    "${var.domain}",
  ]
}

# Retrieve information about the manually configured
# public hosted zone.
data "aws_route53_zone" "primary" {
  name         = "${var.domain}."
  private_zone = false
}

# Creates an ALIAS record that ties our domain to
# the CloudFlare distribution.
#
# We could also have a resource that CNAMEs `www` such
# that we'd have `www` set for our website as well.
resource "aws_route53_record" "cdn-alias" {
  zone_id = "${data.aws_route53_zone.primary.zone_id}"
  name    = "${var.domain}"
  type    = "A"

  alias {
    name                   = "${aws_cloudfront_distribution.website_cdn.domain_name}"
    zone_id                = "${aws_cloudfront_distribution.website_cdn.hosted_zone_id}"
    evaluate_target_health = false
  }
}

Aside from running a main.tf that imports the module and creates the resource I just have to do two things manually:

tell GoDaddy to use Route53 as the nameservers for this domain (such that we can make use of ALIAS records to have the domain tied to CloudFront);
accept the terms of service for the certificate generation.

And that’s it! Push to git and everything is live.

Closing thoughts

The primary objection to doing things this way is that there are already great players out there to serve static files, and I get the point. For me, however, being able to guarantee a great performance, control each step, get all the metrics I need and still have the same great user experience, makes publishing with AWS great - set it up once (takes 30m) and then never worry again.

I think now there’s only one thing that I’d improve:

automatic cache invalidation when publishing: would allow me to make sure clients are getting the last published content at the time I post (supposing I want to quickly edit something or make sure that the index.html that lists all pages is up to date on all locations).

Update: this is actually pretty simple and doens’t require any Lambda stuff. Corey Quinn from LastWeekInAWS gave me an excelent idea: use the AWS CLI to perform the invalidations. This way we can tie it to the travis-ci deployment script and we’ll have the invalidations on every push. Pretty neat!

To achieve that I could have an AWS lambda function that is triggered when I finish pushing to S3. It hasn’t been an issue so far as I’ve only needed to expire the cache twice (in those cases I just went to the console and clicked few buttons, no big deal).

If you liked to post and would want to see more content related to AWS and coding in general, make sure to subscribe to the mailing list below. You can also reach me at Twitter at any time.

Have a good one!

finis

Configuring HAProxy with HTTP2 support

Ciro S. Costa — Mon, 08 Jan 2018 00:00:00 +0000

Hey,

Recently, HAProxy 1.8 got announced, and it came with some pretty good news:

HTTP/2 is automatically detected and processed in HTTP frontends negotiating the “h2” protocol name based on the ALPN or NPN TLS extensions.

At the moment the HTTP/2 frames are converted to HTTP/1.1 requests before processing, so they will always appear as 1.1 in the logs (and in server logs).

No HTTP/2 is supported for now on the backend, though this is scheduled for the next steps.

HTTP/2 support is still considered EXPERIMENTAL, so just like for multi-threading, in case of problem you may end up having to disable it for the time it takes to solve the issue.

If you’re already making use of TLS with HAProxy, HTTP/2 support should be a piece of cake.

The example

The whole blog post is centered around the idea of putting HAProxy as a reverse proxy for a service that sits behind the test.com domain.

As mentioned in the announcement, all the traffic that flows into HAProxy as HTTP/2 gets translated to HTTP/1.1 requests before processing, meaning that you can put your HTTP/1.1 server as a backend service without any problems.

Another consequence of the HTTP/2 to HTTP/1.1 translation is that all the http mode directives that you have in the haproxy configuration are still valid and you don’t need to change them.

Building HAProxy from source

If you’re a MacOS user and don’t want to wait until the HAProxy Brew formula gets updated, or just don’t want to wait for the official HAProxy docker image either, it’s pretty straightforward to build it from source.

For either macOS or Linux, the first steps are all the same. The only difference comes at the moment of running make (different flags are needed):

# Set the version that we'll download
HAPROXY_BRANCH="1.8"
HAPROXY_VERSION="1.8.4"

# Retrieve the release of HAProxy
curl -SOL http://www.haproxy.org/download/$HAPROXY_BRANCH/src/haproxy-$HAPROXY_VERSION.tar.gz

# Untar it
tar xzf ./haproxy-$HAPROXY_VERSION.tar.gz

# Get into the directory
cd ./haproxy-$HAPROXY_VERSION

With the directory ready, now it’s time to run make with the appropriate flags.


# Install (or upgrade) openssl:
brew upgrade openssl
==> Upgrading openssl 

# Once the install is finished, brew will tell you where to
# look for the openssl headers (include) and the compiled
# library that you can use with software that requires it
# (in our case, HAPROXY)
... 
For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include
For pkg-config to find this software you may need to set:
    PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig


# Start the actual compilation.
#
# These options can be looked up at the Makefile file.
#
# We're essentially telling it to use some default options for
# OSX as well as specifing that we want a specific regex library
# (pcre), enable use of OpenSSL and zlib support to provide us
# support for deflate (gives us gzip compression).
#
# SSL_* options tells the compiler where to look the symbol
# declarations and definition.
make -j6 \
USE_ZLIB=1 \
USE_PCRE=1 \
USE_OPENSSL=1 \
SSL_LIB=/usr/local/opt/openssl/lib \
SSL_INC=/usr/local/opt/openssl/include \
TARGET=osx


# Verify whether the binary has been compiled with  all the
# properties that we need.
./haproxy -vvvv
HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
Build options :
  TARGET  = osx                                         <<<<
  CPU     = generic
...
Built with OpenSSL version : OpenSSL 1.0.2n  7 Dec 2017
Running on OpenSSL version : OpenSSL 1.0.2n  7 Dec 2017 <<<<<<
OpenSSL library supports TLS extensions : yes           <<<<<<
OpenSSL library supports SNI : yes                      <<<<<<
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 <<<<<
Built with transparent proxy support using:
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.41 2017-07-05               <<<<<<
Running on PCRE version : 8.41 2017-07-05               <<<<<<
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11                        <<<<<<
...
Available polling systems :
     kqueue : pref=300,  test result OK                 <<<<<
...

If you’re on Linux, the process is mostly the same. Add some dependencies, switch some parameters when building with make and you’re good to go.

# Update the package information from all the configured
# sources.
sudo apt update -y

# Install the dependencies
sudo apt install -y \
        openssl-dev \
        zlib-dev \
        libpcre-dev

# Perform the actual compilation
make -j6 \
TARGET=linux2628 \
USE_OPENSSL=1 \
USE_ZLIB=1 \
USE_PCRE=1

ps.: if you also need Lua and is curious to know how to integrate HAProxy with Letsencrypt, make sure you check how to get TLS certificates with Letsencrypt and HAProxy. There I build both HAProxy and lua from scratch on an Ubuntu 17.04 box.

Generating certificates

As browsers only implement HTTP2 over TLS, we must have some certificates in place if we want to test HAProxy HTTP2 support using a web browser.

The process of generating self-signed certificates (for tests) is far from difficult though - all we need to do is call openssl with some arguments:

openssl req \
        -x509 \
        -sha256 \
        -newkey rsa:4096 \
        -keyout "test.com.key" \
        -out "test.com.pem" \
        -days 730 \
        -nodes \
        -subj "/C=BR/ST=SaoPaulo/L=SaoPaulo/O=TestOrg/OU=TestUnit/CN=test.com"
Generating a 4096 bit RSA private key
...............................................................................................................................................................................++
...........................++
writing new private key to 'test.com.key'
-----

With the certificate written to the test.com.pem file (and the private key to test.com.key), we inspect the certificate and make sure that what we want is set up:

openssl x509 -in ./test.com.pem -text -noout 

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 17902607379759041757 (0xf872d68f4fd30cdd)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=BR, ST=SaoPaulo, L=SaoPaulo, O=TestOrg, OU=TestUnit, CN=test.com
        Validity
            Not Before: Mar  1 13:47:12 2018 GMT
            Not After : Feb 29 13:47:12 2020 GMT
        Subject: C=BR, ST=SaoPaulo, L=SaoPaulo, O=TestOrg, OU=TestUnit, CN=test.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

ps.: I’ve created a little script that helps provisioning certificates that bundle multiple domains (via SAN - Subject Alternative Name): cirocosta/certgen. It’s a one-file bash script that automates the process

Having the certificate and its associated private key created, we must create the final file that HAProxy uses to terminate the TLS connections. THis file is a combination of both the private key and the certificate. From the configuration manual:

[crt…] designates a PEM file containing both the required certificates and any associated private keys.

This file can be built by concatenating multiple PEM files into one (e.g. cat cert.pem key.pem > combined.pem).

In our case, just concatenate those two:

# Concatenate the certificate and the private key
# such that HAProxy can deal with a single file
# that has both of them.
cat ./test.com.pem test.com.key > haproxy_test.com

With the certificates set, let’s jump to HAProxy.

Setting up HAProxy with HTTPS support

First, we’ll set up HAProxy to route requests from HTTPS to an HTTP backend.

global
# Setting the maximum size of the Diffie-hellman parameters
# used in the TLS negotiation such that HAProxy won't warn us 
# about the low default value of 1024.
                tune.ssl.default-dh-param   2048


defaults
# As we're sticking with using HTTP/1.1 all the way
# we can set the mode in the `defaults` section and
# have it applied everywhere.
#
# If you set only in the frontend, then haproxy will
# complain about letting backend be tcp.
                mode http


frontend        https
# Binding to port 443 forces us to have higher privileges
# when launching HAProxy, but it allows us not to have to
# set the `:<port>` in the URL when connecting to it using
# a browser (as `https` will imply port 443).
#
# The certificate set here is the one that contains both the
# private key and the actual `.pem` that we generated using
# openssl.
#
# ps.: if you have *a bunch* of certificates to serve, then 
# you should switch to `crt-list` as HAProxy has a limit on
# the size of each line in this config file.
                bind                        *:443  ssl crt /tmp/sample/certs/haproxy_test.com

# Simple ACL to send traffic to our custom tailored web server.
#
# This way we can exercise a "not found" when not having `HOST`
# properly set.
#
# This also demonstrates how HAProxy is indeed decyphering the
# traffic (otherwise it wouldn't catch the Host header).
                acl                         host_matches hdr_dom(host) test.com
                use_backend                 desired_backend if host_matches
                default_backend             not_found


# Our server that serves `/tmp/sample/www` using python's 
# SimpleHTTPServer module.
backend         desired_backend
                server                      myserver localhost:8000



# A dumb "not-found" auto-responder using HAProxy's errorfile
# directive.
#
# As the ACL that send traffic to this backend is the default
# (and least prioritized), when we reach this backend, it means
# that we didn't find a desired backend, thus we should serve
# a 404 instead of the generic 503 that haproxy gives back to
# clients when something goes wrong.
backend         not_found
                errorfile                   503 /tmp/sample/errorfiles/404.http

Run HAProxy and head over to https://localhost.

At this moment you should see HAProxy serving the HTTP response defined in the errorfile (404.http).

Now, change your /etc/hosts (or local DNS) to contain an entry for test.com and head to https://test.com: now you should see the page being served.

With TLS termination going on, debugging becomes a bit hard without providing extra help for Wireshark.

Using Wireshark to debug the traffic

To have Wireshark being able to interpret the traffic, we must provide it the keys to decrypt the whole traffic.

SSLKEYLOGFILE=~/tlskey.log \
        /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome &

# go to your website ...
# check what Chrome put in the file

cat tlskey.log 
CLIENT_RANDOM d3245ce2f3eabc7895a99c66de03d30be1f8798fe5401dd10dcdf0b2db566e33 2adf73c515bb4fc0171b19f991a6aaecd3899c18fb83f162f926f6239f8e76c431a01d379ede29a4ae31bd37dcb48964
CLIENT_RANDOM 6b8148cc629e9a605f58e13c7a57feaecfb54db2365991c7316cc7705a9fff8b 8c2d1a94f7e8d4b65e115230e725cdda022fda8eefd4f235dde977c3ce7c0b52f95ddd9958193e18ea2b60afb6274a2a
CLIENT_RANDOM 4c6a4da0b4bc5985180c8eb7bc4264d1794a24e4c19c8b1e8bbfe544f737f67d 367b1594d6903a0a93f1b145bf321bb8740df860abdabfd3ef44dce118bed8c92a49a03a1d740f6cd22b4934d1174466
CLIENT_RANDOM d8316ea981c4bf35636c185b186af42dc257ce9b88752f9f9848150298433b9b 1742b4d08c170bf86a70c44130d782db173db32446d5350295395e5ed2c787b5f79772c7f48910b12d0223d85dd5dec0
CLIENT_RANDOM 93ffc7c33a0a38476266961d7eb2c3b7e62ca41db99129f50648779871170feb c8dc729c7f0a3ef195fe856d2fd8f081d0d9519c52ca87eaa9e33c8c77bb8619f691cb99c9ba9ba8ef4968ce97e74d32
CLIENT_RANDOM 27b31452a53d48abfa322f931e0f7a65e5ad846cd682005a9b52a579c03703c1 13121dc8cf9c012f4d5f27a93a2281d3769a3566926ebed7c4fd7ab7d11ae16f210e9268befad340b3d2803f44a5b570
CLIENT_RANDOM 322c12247007fc09a4ddff6ab8c8256b8cb40bc6c82769cde936d28e812c8ced 4455a438772c75fb685418a46801ad63b61459801d217669c7b9ed371ed2959e9da0180dd749496e83f8a78996781b98
CLIENT_RANDOM 3cde72add3576eb96f4cdf7820e2555188a500ea816f835112feb5d195c91c10 34d07cd567fb9287dad18df4c31e3b38f602d8e2ed668942bad45858c76243f4d3f9f0012ca56f1fd5f4fb2ddaafd125
CLIENT_RANDOM 5d6baf61132868f69136415114c9cf2b1134bb509669899cb3ed23610f7d4b5d 1f584a5d6407bc5b2eecef36480af8d4adfb8481cb0e74b8e91f05b3501d4101db34eb47a76edb248b3f73cc1bad005c

With this, we’re already able to decrypt TLS traffic.

Open Wireshark’s preferences (cmd + , ), head to Protocols > SSL and then change (pre)-Master-Secret log filename to the filename you specific in SSLKEYLOGFILE.

Run HAProxy and navigate to the website - you should be able to see the traffic in wireshark:

Enabling HTTP2 in HAProxy

Once HTTPS has been set up, enabling HTTP/2 in HAProxy is a matter of including the alpn h2 directive to the bind line such that whenever the browser tells HAProxy that it can take HTTP/2 traffic, HAProxy does the job of serving it.

 # ... defaults configuration ...

 # HTTP2 frontend server from https portj
 frontend        https
-     bind  *:443  ssl crt <certificate>                 
+     bind  *:443  ssl crt <certificate> alpn h2,http/1.1

Now, reload HAProxy with the new configuration and the traffic should be served via HTTP/2.

To make sure that that’s the case, get to https://test.com and open the HTTP/2 tab of chrome://net-internals:

There we should be able to see the HTTP/2 session originated by Chrome to HAProxy which proxies the requests to our HTTP/1.1 server.

Inspecting the HTTP2 traffic

To have a better view of what goes on, we can make use of Wireshark.

Unlike with plain HTTPS though, to inspect these HTTP2 streams we must first make sure that our operating system trusts the server certificate.

Using MacOS and Chrome, you can head to the Security tab in the developer tools and then open “view certificate” to open the certificate popup.

With that popup open, drag the certificate to your desktop and open it (it’ll ask for your password and launch Keychain Access).

Once keychain is open, perform a right click on the test.com certificate and then click on get info. This should open a new popup.

Now it’s a matter of making MacOS trust the cert:

With the certificate trusted, now we can go back to Wireshark and see the HTTP/2 streams (notice that Chrome might still tell you that the certificate is not secure as it’s a self-signed certificate, but that’s ok):

Extra - Installing CURL with HTTP2 support

curl by default doesn’t come with HTTP/2 support out of the box, and if you’re using MacOS and didn’t specify an extra flag, that’s for sure. As mentioned on the official website:

libcurl uses this 3rd party library for the low-level protocol handling parts. The reason for this is that HTTP/2 is much more complex at that layer than HTTP/1.1 (which we implement on our own) and that nghttp2 is an already existing and well functional library.

So, to have HTTP/2 enabled with Curl, you need to build it with nghttp2 (see HTTP/2 with curl).

Using brew, it’s very simple to have that though.

The brew formula for curl supports an extra flag that installs it for you:

# some other options ...
  option "with-gssapi", "Build with GSSAPI/Kerberos authentication support."
  option "with-libmetalink", "Build with libmetalink support."

#        \/ \/ what we want!
  option "with-nghttp2", "Build with HTTP/2 support (requires OpenSSL)"

This means that you’d go to Terminal and install it like this:

# install cURL with nghttp2 support
brew install curl --with-nghttp2

# link the formula to replace the system cURL
brew link curl --force

# reload the shell and then issue a request against
# our `test.com` server specifying the `-k` to allow
# any certificates:
curl -k https://test.com -v

* Rebuilt URL to: https://test.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to test.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2             <<<<< negotiating HTTP/2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/openssl/cert.pem
  CApath: /usr/local/etc/openssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
...
* Using HTTP2, server supports multi-use                <<<<< HTTP/2
* Connection state changed (HTTP/2 confirmed)           
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff354804400)
> GET / HTTP/2
> Host: test.com
> User-Agent: curl/7.56.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< server: SimpleHTTP/0.6 Python/2.7.10
< date: Thu, 01 Mar 2018 18:06:30 GMT
< content-type: text/html
< content-length: 423
< last-modified: Thu, 01 Mar 2018 15:31:02 GMT
< 
<!DOCTYPE html>
<html>
  ...

Closing thoughts

It’s cool to see how easy is to have a basic HTTP/2 setup with HAProxy. Although it’s not the new kid on the block, it’s catching up regarding features with other more recent load-balancers.

It still lacks server push and full HTTP/2 support (it can’t connect via HTTP/2 with a backend), but it can already deliver great results.

Please let me know if you spot anything off; I’m @cirowrc.

Have a good one!

finis

Sending files via gRPC

Ciro S. Costa — Tue, 02 Jan 2018 00:00:00 +0000

Hey,

some time ago I got curious about whether gRPC would be something suitable for sending files over the wire.

One of its goodness is the native support of streams, so, why wouldn’t it be?

A service and a message

To get the idea going, I took the approach of defining a minimum viable service, one that takes some chunks and then, once received, counts how many bytes of the actual content has been received.

These chunks were defined like the following:

message Chunk {
        bytes Content = 1;
}

As the service takes this as a stream, we can define it like so (see cirocosta/gupload/messaging/service.proto):

service GuploadService {
        rpc Upload(stream Chunk) returns (UploadStatus) {}
}

enum UploadStatusCode {
        Unknown = 0;
        Ok = 1;
        Failed = 2;
}

message UploadStatus {
        string Message = 1;
        UploadStatusCode Code = 2;
}

To better visualize what’s going on, we have the following:

In essence, this means that we get a file handle and once the connection is started, we start splitting its contents in chunks and sending these fragments as Chunk messages over the gRPC connection that got established. Once messages arrive at the server, we unpack those messages (which contains the raw bytes in the Content field. After all the transmission has been finalized, an UploadStatus message is sent to the client, and the channel is closed.

You can check the code in cirocosta/gupload.

The gist of the client is, with some parts removed for brevity, the following:

func (c *ClientGRPC) UploadFile(ctx context.Context, f string) (stats Stats, err error) {

        // Get a file handle for the file we 
        // want to upload
        file, err = os.Open(f)

        // Open a stream-based connection with the 
        // gRPC server
	stream, err := c.client.Upload(ctx)
	
        // Start timing the execution
	stats.StartedAt = time.Now()

        // Allocate a buffer with `chunkSize` as the capacity
        // and length (making a 0 array of the size of `chunkSize`)
	buf = make([]byte, c.chunkSize)
	for writing {
                // put as many bytes as `chunkSize` into the
                // buf array.
		n, err = file.Read(buf)

                // ... if `eof` --> `writing=false`...

		stream.Send(&messaging.Chunk{
                        // because we might've read less than
                        // `chunkSize` we want to only send up to
                        // `n` (amount of bytes read).
                        // note: slicing (`:n`) won't copy the 
                        // underlying data, so this as fast as taking
                        // a "pointer" to the underlying storage.
			Content: buf[:n],
		})
	}

        // keep track of the end time so that we can take the elapsed
        // time later
	stats.FinishedAt = time.Now()

        // close
	status, err = stream.CloseAndRecv()
}

For the server, almost the same - receive the stream connection and then gather each message:

// Upload implements the Upload method of the GuploadService
// interface which is responsible for receiving a stream of
// chunks that form a complete file.
func (s *ServerGRPC) Upload(stream messaging.GuploadService_UploadServer) (err error) {
        // while there are messages coming
	for {
		_, err = stream.Recv()
		if err != nil {
			if err == io.EOF {
				goto END
			}

			err = errors.Wrapf(err,
				"failed unexpectadely while reading chunks from stream")
			return
		}
	}

END:
        // once the transmission finished, send the
        // confirmation if nothign went wrong
	err = stream.SendAndClose(&messaging.UploadStatus{
		Message: "Upload received with success",
		Code:    messaging.UploadStatusCode_Ok,
	})
	// ...

	return
}

With that set we can start measuring the transfer time.

Scenarios

To keep the transmission running for little while I picked a payload of 143M (.git of github.com/moby/moby as an uncompressed .tar).

The server and client shared the same host (macOS 16 GB 1600 MHz DDR3, 2.2 GHz Intel Core i7 Retina, 15-inch, Mid 2015) and communicated via loopback.

Apparently, this is not the best way of testing this kind of stuff, but I think the overall idea is still valid anyway.

I wanted to understand how changing the size of the chunks would affect the overall transmission time. I hypothesized that there was going to exist a significant number between a big chunk and a small chunk that would be optimal for the transmission.

With that said, I created the following test script:

#!/bin/bash

main () {

  # for every chunk size in 
  # "16, 32, 64, 128 ... 2097152 (2MB)"
  for k in $(seq 4 21); do
    
    # run the upload 50 times
    for i in $(seq 1 50); do
      upload_file_via_grpc_tcp_compressed "$k"
    done

  done
}

upload_file_via_grpc_tcp () {
  local number=$1

  printf "$number,"

  gupload upload \
    --address localhost:1313 \
    --chunk-size $(./shift 1 $number) \
    --file ./files.tar 
}

main

shift is just a little program that performs the bit shifting of two arguments:

int main(int argc, char** argv) {
	// ... validates argc and argv
	int number = atoi(argv[1]);
	int shifts = atoi(argv[2]);

	printf("%d\n", (number << shifts));
	return 0;
}

which means that our test case would produce a CSV: <number_of_bitshifts>,<time>, for instance:

4,28418808431
4,31305514409
4,30451840141
...
5,16610844861
5,15894009998
...
6,8175781898
6,8074899000
...

Having those two columns, we can then put it in Excel, create a pivot table and then gather insights about the data.

Results

I let that script run for a while, here are the results:

From the results, it looks like if we have a tiny chunk size (16, 32, 64 and 128), we can’t deliver too much over the network. That’s clear via the CPU usage: I was getting close to 200% on each process (client and server), clearly seeing the transfer being bottlenecked by the CPU, which is not the desired.

As we move to more significant values it becomes evident that the time to transfer the data falls considerably, but we start seeing a big variance in the times - sometimes it was performing pretty well, sometimes it was pretty bad.

It seems like at 1024bytes (1KB) we have a decent throughput.

I was expecting something like 32KB: (1 << 15) to be the most optimal but it looks like it’s one of the worst scenarios regarding variance.

Maybe that’s something that’s only observed when running both client and server on the same machine … I’m not sure.

As in this scenario I wasn’t making use of TLS I thought that perhaps the underlying gRPC would not be using HTTP2 (I need to confirm this!); thus I could probably get some different results running with TLS enabled.

That wasn’t the case though:

The variance is shifted a bit but, still, very close results.

Comparing with plain HTTP2

I wanted to have a baseline to compare these results, so I made client and server a simple interface that other types of clients and servers could implement.

With that set I created a HTTP2 client and server that you can check out at the repository - https://github.com/cirocosta/gupload under ./core/client_h2.go and ./core/server_h2.go.

I think it’s evident that for this workload a raw transfer of bytes over http2 should be faster than the gRPC one as there’s no encoding and decoding of messages - just raw transmission of data.

Closing thoughts

I didn’t want to draw many conclusions from this but just make clear in my head that it’s critical to understand what’s your workload and how the underlying transport works to get the best results possible. It’s common to get caught in fallacies of distributed computing and just start assuming things. This week I tried to pretend less and test more.

If you liked this article, there’s another blog post I wrote about buffering where I explore where buffering takes most effect, and it’s very interesting to see how different network conditions affect transmission.

If you’ve got interested in this article, you probably should also check out The Site Reliability Workbook - Practical Ways to Implement SRE.

It’s about practical examples of Google’s experience, and the internal gRPC equivalent (Stubby) as well as GRPC itself is part of it.

Again, it was all “loopback-based” so take it with some salt.

If you want to get the results, here’s the Excel file: grpc-uploads.xslx.

Let me know if I got anything wrong; I’d love to understand more about it and see what you think.

Have a good one!

finis

Should you buffer your writes?

Ciro S. Costa — Sun, 31 Dec 2017 00:00:00 +0000

Hey,

some weeks ago I wrote a Docker logging plugin that acts as an oklog forwarder: oklog-docker-plugin.

oklog makes use of a fairly straightforward format for ingesting logs: send it a line with some predefined fields and ta-da, it’ll be properly ingested.

While coding the plugin, I remember considering whether I should buffer these stuff or not.

My rationale was that I should probably look at the tradeoff of between immediate availability of the logs and some kind of network optimization by reducing the number of writes that I’d perform, thus reduce the number of packets inflight in a given time.

That got me to wonder how much of perceived differences we could find via buffering writes and how does that change as we switch from a given size to another.

What I expected: in the beginning, the more we cache, the better; we’ll achieve some peak, and then after that, we’ll start getting worse than not buffering or buffering at a given limit given that the whole write will be broken in parts. Does this hold true for real scenarios? How does it perform?

Testing it

To test that, I managed to have the following:

set a given reasonable amount of data to transfer
prepare a test executable that would transfer this with the different buffer sizes by passing a different parameter
run each test execution multiple times; average the results for each.

The easiest way I could find to have buffering set was to make use of setvbuf(3):

SETBUF(3)                 Linux Programmer's Manual                SETBUF(3)

NAME
       setbuf, setbuffer, setlinebuf, setvbuf - stream buffering operations

       ...

       int setvbuf(FILE *stream, char *buf, int mode, size_t size);

       The setvbuf() function may be used on any open stream to  change  its
       buffer.

The coolness of this method is that we can either provide a buffer of our own (and then, say, inspect it at any time we want) or let glibc take care of that. We can skill all the buffering if desired by setting a flag or make it perform buffering based on newlines. Something to take notice is that setvbuf is applied to a FILE, which is not our primary target when making use of sockets.

This means that to benefit from those methods we have to associate a socket with a FILE first and just after that set the buffering mode for methods that make use of that stream.

Establishing connections and associating sockets with FILE streams

To associate a socket with a file stream we have to, right after getting a socket, make use of fdopen(3):

The fopen() function opens the file whose name is the string pointed to by path and associates a stream with it.

Given that when we establish a TCP connection, we have both ways (read and write), we can associate the socket with to separate file streams.

To make that easy to deal with I started by creating a struct (see when-to-buf/conn.h) that would hold these two streams and then deal with establishing the connections.

typedef struct conn {
	FILE* rx;
	FILE* tx;
	int fd;
} t_conn;

The struct gets initialized differently depending on whether we’re dealing with a client (active) or server (passive) connection.

For instance, on the server side we accept(2) on the listen_fd and then prepare the t_conn struct:

int
init_server_conn(t_conn* connection, int listen_fd)
{
	int conn_fd;
	struct sockaddr_in client_addr;
	socklen_t client_len = sizeof(client_addr);

        // `accept` extracts the first conn request from the queue of 
        // pending connections for the listening socket `listen_fd`
        // and then created a new connected socket.
	conn_fd =
	  accept(listen_fd, (struct sockaddr*)&client_addr, &client_len);
	if (conn_fd == -1) {
		perror("failed to accept connection - accept");
		return 1;
	}

	connection->fd = conn_fd;

        // with `fdopen` we create a FILE stream for the `read`
        // side of the socket
	connection->rx = fdopen(conn_fd, "r");
	if (connection->rx == NULL) {
		perror("failed to associate socket with rx stream - fdopen");
		return 1;
	}

        // to create the second FILE stream we first duplicate the
        // socket file descriptor and then open it with write only
        // capabilities
	connection->tx = fdopen(dup(conn_fd), "w");
	if (connection->tx == NULL) {
		perror("failed to associate socket with tx stream - fdopen");
		return 1;
	}

	return 0;
}

Once we have t_conn set up, writing to it became as simple as writing to a file. Any method that takes a FILE (like fprintf or fwrite…) could now be an entry point to writing to that connection.

Writing to the connection

To perform the buffered writes, setvbuf end up looking like the following (see when-to-buf/client.c#work_on_connection):

int
work_on_connection(t_conn* connection, int bufsize)
{
        int to_write = bufsize;
	char* write_buf;
	// ... 

        // if we want to buffer our writes, then prepare a 
        // buffer that will hold the data prior to pushing it
        // to the real underlying connection via `write(2)`s
	if (bufsize > 0) {
		write_buf = malloc(bufsize * sizeof(char));
		if (write_buf == NULL) {
			perror("failed to allocate write_buf");
			return 1;
		}

                // make `glibc` make use of our buffer and always buff
                // up to `bufsize`.
		err = setvbuf(connection->tx, write_buf, _IOFBF, bufsize);
		if (err != 0) {
			perror("failed to set block buffer - setvbuf");
			return 1;
		}
	        // ...
        }

	for (;;) {
                // ...
                // send data to `fwrite` such that it can perform
                // the buffering as performed above and then
                // end up writing to the connection at some point
		n = fwrite(SOURCE_BUFFER + total_written,
		           sizeof(char),
		           to_write,
		           connection->tx);
                // ...
	}
        // ...
}

With that set, it was now time to make sure that our underlying write(2) calls were being buffered as we intended.

To test that, the easiest way was to set SOURCE_BUFFER to something small, like 1024 bytes

- #define SRC_BUFSIZE (1 << 25)
+ #define SRC_BUFSIZE (1 << 10)
  diff client.c

and then verify for different bufsizes how many write(2) calls are made.

First we start with a big buffer (bufsize=1024):

# Run `strace` watching the execution of the 
# `./client` executable taking the arguments 
# `127.0.0.1` (address) and `1024` (bufsize).
#
#  ps.: -w : summarise time different between executions
#       -c : count the executions

# In this case, we expect to have a *very*  low number
# of `write(2)`s as we should be able to fill the whole
# thing in a single `write(2)` call, with no buffering
# at all.

strace -c -f -w ./client 127.0.0.1 1024

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 33.22    0.000288          72         4           close
 18.92    0.000164          82         2           write        <<<<<<<<<<
 13.03    0.000113         113         1           execve
  ...
  0.35    0.000003           3         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.000867                    36         3 total
                                      /\ 
                                      ||
                                  low amount (Y)

After that, with a very small buffer (bufsize=1):

# Now, with a `bufsize` that's as low as 1 byte, we should
# expect to have the number of `write(2)`s to be pretty big
# (close to 1024):
strace -c -f -w ./client 127.0.0.1 1


% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 94.58    0.011862          12      1025           write        <<<<<<<<<
  2.72    0.000341          85         4           close
  ...
  0.02    0.000003           3         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.012542                  1059         3 total
                                    /\ 
                                    ||
                            a bunch of syscalls

cool, so we have what we need setup:

control bufsize via CLI arguments
have buffered writes with different buffer sizes

now it’s time to make it run!

Running some scenarios

The first scenario I wanted to test was the one that I could predict the outcome pretty well.

communication between client and server via loopback

As the loopback interface isn’t more than a virtual NIC and doesn’t even makes use of Ethernet, we should be able to transfer these packets as fast as possible regardless of the size (ps.: we’d not have everything being sent in a single packet as we still make use of IP).

That said, the improvements should be seen as we get bigger and bigger buffer sizes.

It really is - overall, increasing the size of the buffers help.

Is this too far from reality? Well, in some cases. For those deploying Kubeneretes services, it’s very common to have sidecars, i.e, collocation of containers. In this case, you’re running another container in the same shared namespaces, and you’re entirely able to communicate via loopback (e.g, for relaying logs!). It’s debatable if you should be using the whole network stack in such case (as opposed to unix sockets), but it’s a case.

Next, I wanted to check how it performs when there are some failures and delays in the communication. In theory, this is where buffers should shine after all.

Introducing problems in the network

To introduce both consistent and eventual delays, as well as the ability to drop packages at a given rate, I made use of tc(1). Change the queueing discipline of the interface, and you’ll have a messed up scenario*.

Keeping the total transfer size constant (1GB), I tried five scenarios. First I started with package loss:

Random package loss applied to 0.1% of all packages

Random package loss applied to 1% of all packages

Random package loss applied to 5% of all packages

Nothing too interesting here. The shape of the data looks like the same as in the perfect loopback scenario. It might vary a little bit more but nothing impressive.

The game changes when it comes to delays though.

Consistent delay of 5ms to all packages

Consistent 10ms to all packages

Bursting delay - added delay of 100ms (+/- 10ms) with the next random element depending on 25% on the last one

If you’re curious about how to perform the same kind of traffic shaping in your own loopback, these were the rules used:

tc qdisc add dev lo root netem loss 0.1%
tc qdisc change dev lo root netem loss 1%
tc qdisc change dev lo root netem loss 5%
tc qdisc change dev lo root netem delay 100ms 10ms 25%
tc qdisc change dev lo root netem delay 5ms

Closing thoughts

I found pretty surprising how delays introduced a much more diverse result compared to package loss.

In the future, I might explore this a bit more, but I think this is enough and worth sharing.

If you spot any mistakes or think that I might have gotten everything wrong, please let me know! I’d appreciate a lot perfecting this and understanding better the results.

You can get the Excel worksheet that I created here: when-to-buf.xslx.

I’m cirowrc on Twitter, feel free to drop me a message there!

Have a good one,

finis

Writing DNS messages from scratch using Go

Ciro S. Costa — Thu, 28 Dec 2017 00:00:00 +0000

Hey,

Yesterday I read a pretty interesting article from James Routley with the title Let’s hand write DNS messages. It goes all the way down to preparing a UDP query by hand and then interpreting it by reading the bytes received back.

That was cool mainly because it makes clear that DNS messages are not all that complicated.

It takes some encoding/decoding to create a message and understand its result but, still, not hard.

I took the opportunity to go through the article and the DNS RFC and then implement myself something similar. The result is cirocosta/rawdns.

The post is not intended to explain why DNS exists and what it’s meant to do.

If you’d like to know more about DNS as a whole, make sure you check - Computer Networking: A top-down approach.

Now, to the article.

Why recreate the wheel?

It’s pretty evident that the work I did is unnecessary. There are great packages for creating DNS servers/clients (like the excellent miekg/dns which I even used in a simple DNS recursive server I wrote (cirocosta/sdns some time ago). If all you need is resolving the IP of a host, just use plain “net”/LookupAddr, it does the job much better than my tool.

The good side of doing though is that I had the opportunity to:

review some networking concepts
check out how to do UDP with Go (I had only dealt with UDP in C before)
review some bitwise operations
have a more in-depth view of how the DNS protocol works

Should you do the same? Well, maybe.

It’s a common practice to do something similar in a networking class, but in my case, we didn’t do it (had some other assignments).

The types

I started the project by looking at which types I should deal with.

Fortunately, there are just a few.

DNS Message: the piece that goes in both ways (request and response), always contains the header and then other pieces (like question and RR) depending on the message being a request or response;
DNS Message - Header: this gives general information about what’s coming in the message, like, how many queries are there … how many answers … what is the ID that identifies this “session” and some other pieces of information. It’s what tells you how to parse what’s coming next and give context to the semantics.
DNS Message - Question: contains the actual query that you want to perform against a nameserver. It can be of many types and classes; here I just carried about one: A records against recursive servers.
DNS Message - RR: representation of a resource record (in this case, we can treat it as a plain answer). Regardless of the type, this always comes packed with the same format, but it leaves a field (RDATA) to be parsed after the fact depending on the type.

This translated to the following files in cirocosta/rawdns:

rawdns
└── lib
    ├── header.go               // header
    ├── header_test.go
    ├── message.go              // full DNS msg
    ├── question.go             // question
    ├── question_test.go
    ├── rr.go                   // rr
    └── rr_test.go

As the whole message (message.go) is pretty much a composition of Header, Questions and Answers, I started by making sure that each one of them could be both parsed from a sequence of bytes as well as they could create a sequence of bytes from their definition.

The tricky part is that all the types we have in Go are byte aligned, while some parts of the header section, for instance, take only 3 bits, or 4 bits … some others, just 1bit. This means that we have to fit that kind of information in an 8bit type.

A little bit of digression - getting and set bits in a byte-aligned language

The whole idea of fitting pieces of information in a single byte is very natural to think about as we can easily refer to “this bit of this byte” but some languages (like Go) gives no native interface to that.

Looking at the description of the DNS message header, we have the following description:

        FIRST BYTE                SECOND BYTE
   ----------------------- || ---------------------

     0  1  2  3  4  5  6  7  0  1  2  3  4  5  6  7
   +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
   |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
   +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

The structure can be thought as two bytes which have values placed at different offsets with some maximum values that they can take (given the number of bits that they can hold):

   first_byte:
   {
      QR:      offset=0 , size=1bit
      Opcode:  offset=1 , size=4bit 
      AA:      offset=5 , size=1bit
      TC:      offset=6 , size=1bit
      RD:      offset=7 , size=1bit
   }

   second_byte:
   {
      RA:      offset=0 , size=1bit
      Z:       offset=1 , size=3bit 
      RCODE:   offset=4 , size=4bit
   }

Back in the days in which I used to do some C I remember there was a pretty good way of dealing with this: bit fields. In Go, however, there are no language features that can help us directly (see golang-nuts - Bit fields/binary parsing).

In C this allows us to make use of a struct to represent a byte and then use field access operators to gather these bit values without having to deal with bitwise operations.

What I mean is, suppose that you have a multiplayer game where four players are connected and in your loop, you have to evaluate what’s the next position of each player based on their directions.

We could establish that these directions come in the form of an enum which takes 2bits to represent any direction:

typedef enum direction {
        NORTH = 0,
        EAST,
        SOUTH,
        WEST
} e_direction;

Being a 4-player game, we can store this information in a single byte:


     0  1  2  3  4  5  6  7 
   +--+--+--+--+--+--+--+--+
   |DIR1 |DIR2 |DIR3 |DIR4 |
   +--+--+--+--+--+--+--+--+
     /\
   +-------------------------------+
   | direction of the first player |
   +-------------------------------+

such that in C that would translate two:

typedef struct player_directions {
        unsigned char p1_dir : 2;
        unsigned char p2_dir : 2;
        unsigned char p3_dir : 2;
        unsigned char p4_dir : 2;
} t_player_directions;

so in our loop, we could check the directions by accessing the fields of this 1byte struct:

/* definitions above ... */

int main () {
  t_players_directions directions = {
    .p1_dir = NORTH,
    .p2_dir = NORTH,
    .p3_dir = SOUTH,
    .p4_dir = EAST,
  };

  printf("sizeof(directions)=%ld\n", sizeof(t_players_directions));
  printf("p1_dir=%x\n", directions.p1_dir);
  printf("p2_dir=%x\n", directions.p2_dir);
  printf("p3_dir=%x\n", directions.p3_dir);
  printf("p4_dir=%x\n", directions.p4_dir);

  return 0;
}

/**
 *   $  gcc -o main ./main.c
 *   $  ./main
 *
 *        sizeof(directions)=1
 *        p1_dir=0
 *        p2_dir=0
 *        p3_dir=2
 *        p4_dir=1
 *
 **/

What about Go, then? Bit shifting.

When setting values: set the bits in the right position and then merge it to the value you want them in.

   given that we want to pack it in
   1 byte:

     0  1  2  3  4  5  6  7   ==>  dirs uint8 = 0
   +--+--+--+--+--+--+--+--+
   |DIR1 |DIR2 |DIR3 |DIR4 |
   +--+--+--+--+--+--+--+--+

   assume that we want the following 
   values packed:
   dir1 = 0     ===> 0 0
   dir2 = 0     ===> 0 0
   dir3 = 2     ===> 1 0
   dir4 = 1     ===> 0 1

   then it means that we need to get
   dir1 shifted all the way to the 
   first position, then dir2 to the
   second, dir3 to the third and dir4
   to the forth.

   dirs = 0             - start with 0 0 0 0 0 0 0 0
                        
   dirs |= 0 << 6       - move `00` six times and "merge"

   0 0 0 0 0 0 0 0 
   d1  ----------

   dirs |= 0 << 4       - move `00` four times and "merge"

   0 0 0 0 0 0 0 0 
   d1  d2  -------

   dirs |= 2 << 2       - move `10` two times and "merge"

   0 0 0 0 1 0 0 0 
   d1  d2  d3  ---

   dirs |= 1 << 0       - move `01` 0 times and "merge"

   0 0 0 0 1 0 0 1 
   d1  d2  d3  d4

That in Go code turned out to the following (for the case of the header):

//				0  1  2  3  4  5  6  7
//      0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |                      ID                       |
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |                    QDCOUNT                    |

func (h Header) Marshal() (res []byte, err error) {
	var (
		buf       = new(bytes.Buffer)
		h1_0 byte = 0
		h1_1 byte = 0
	)

        // using `binary.Write` we can take the 16bit
        // int that the ID is made of and have it split
        // in two bytes following big endian order (most
        // significant bit first, less significant second)
	binary.Write(buf, binary.BigEndian, h.ID)

	// first 8bit part of the second row
	// QR :		0
	// Opcode:	1 2 3 4
	// AA:		5
	// TC:		6
	// RD:		7
	h1_0 = h.QR << (7 - 0)
	h1_0 |= byte(h.Opcode) << (7 - (1 + 3))
	h1_0 |= h.AA << (7 - 5)
	h1_0 |= h.TC << (7 - 6)
	h1_0 |= h.RD << (7 - 7)

	// second 8bit part of the second row
	// RA:		0
	// Z:		1 2 3
	// RCODE:	4 5 6 7
	h1_1 = h.RA << (7 - 0)
	h1_1 |= h.Z << (7 - 1)
	h1_1 |= byte(h.RCODE) << (7 - (4 + 3))

	buf.WriteByte(h1_0)
	buf.WriteByte(h1_1)

	binary.Write(buf, binary.BigEndian, h.QDCOUNT)

        // ..
}

When retrieving though, the idea is the same. The only difference this time is that you shift to the right and apply a mask to the remaining bits:

  0 0 0 0 1 0 0 1   current value    
  d1  d2  d3  d4

  how to know what's set for `d3`?

  shift the whole thing to the right by
  two places:

  0 0 0 0 0 0 1 0
  --- d1  d2  d3 

  and then mask the remaining (in this case
                               it's not necessary
                               but I assume we don't
                               know that in advance)
  0 0 0 0 0 0 1 0
& 0 0 0 0 0 0 1 1 
  ---------------
  0 0 0 0 0 0 1 0

  this way we know that at d3 we have `2` as the value.

  i.e.: d3 = (dirs >> 2) & ((1 << 2) -1)

If you’re wondering why (1 << 2) - 1, that’s because we want n places to the right filled with 1s and all the places to the left with 0s such that when we AND them, only the n to the right remain.

0 0 0 0 0 0 0 1     1   = (1 << 1 ) - 1
0 0 0 0 0 0 1 1     3   = (1 << 2 ) - 1
0 0 0 0 0 1 1 1     7   = (1 << 3 ) - 1
0 0 0 0 1 1 1 1     15  = (1 << 4 ) - 1
0 0 0 1 1 1 1 1     31  = (1 << 5 ) - 1
0 0 1 1 1 1 1 1     63  = (1 << 6 ) - 1
0 1 1 1 1 1 1 1     127 = (1 << 7 ) - 1
1 1 1 1 1 1 1 1     255 = (1 << 8 ) - 1

If you’re curious to see how that looks like, check out the method lib/header.go#UnmarshalHeader from cirocosta/rawdns:

//      0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |                      ID                       |
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
//    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
//    |                    QDCOUNT                    |
//    .......

func UnmarshalHeader(msg []byte, h *Header) (n int, err error) {
	var (
		h1_0 byte = 0
		h1_1 byte = 0
	)

	// ID is formed by the first two bytes such that
	// it results in an uint16.
	// As this comes from the network in UDP packets we 
        // can assume that it comes in BigEndian (network byte 
        // order), thus, consider the first byte of each,
	// the most significant.
	h.ID = uint16(msg[1]) | uint16(msg[0])<<8

	// take the first byte of the second row (QR, Opcode, AA, TC and RD)
	h1_0 = msg[2]

	// Each value is got from right bitshifting
	// followed by masking such that we end up
	// with only that number.
	// Here we're starting from right to left.
	h.RD = h1_0 & masks[0]
	h.TC = (h1_0 >> 1) & masks[0]
	h.AA = (h1_0 >> 2) & masks[0]
	h.Opcode = Opcode((h1_0 >> 3) & masks[3])
	h.QR = (h1_0 >> 7) & masks[0]

        // ...
}

Once you can recreate messages from a sequence of bytes and write bytes from a structure, you’re ready to start communicating with a real server.

Asking the questions

This is the easiest part (ps.: assuming we’re not entirely implementing all the details of the protocol, like truncating messages when above 512bytes and making use of some security features).

The DNS RFC states:

The DNS assumes that messages will be transmitted as datagrams or in a byte stream carried by a virtual circuit. While virtual circuits can be used for any DNS activity, datagrams are preferred for queries due to their lower overhead and better performance. …

The Internet supports name server access using TCP [RFC-793] on server port 53 (decimal) as well as datagram access using UDP [RFC-768] on UDP port 53 (decimal).

That’s essentially meaning that we should make use of UDP when sending queries and that these queries should be performed against port 53 (by default).

That in Go is translated to a UDP dial:

c.conn, err = net.Dial("udp", cfg.Address)
if err != nil {
        err = errors.Wrapf(err,
                "failed to create connection to address %s",
                cfg.Address)
        return
}

which allows us to write() a sequence of bytes to it and then wait for responses.

The fun thing is that UDP is connectionless. That means that we’re not setting up a full-fledged connection like with TCP, performing ACKS and NACKS all the way to make sure things go smoothly in the channel.

Here I didn’t make use of timeouts and other mechanisms to control the possible scenario where no message comes back, but in practice, that’s totally needed.

// LookupAddr queries a DNS server specified in
// the client initialization for A records concerning
// the address `addr`.
func (c *Client) LookupAddr(addr string) (ips []string, err error) {
        // ...
        // Construct the query struct which
        // represents the whole message.
        // Here we're specifying that we
        // have a single question, that
        // recursion is resired, that the
        // message is a query and then in
        // the question we specify that we
        // want A records, the type is "internet"
        // and the address is 'addr'.
	queryMsg := &Message{
		Header: Header{
			ID:      id,
			QR:      0,
			Opcode:  OpcodeQuery,
			QDCOUNT: 1,
			RD:      1,
		},
		Questions: []*Question{
			{
				QNAME:  addr,
				QTYPE:  QTypeA,
				QCLASS: QClassIN,
			},
		},
	}

        // Because each piece of the message can be
        // marshalled as a sequence of bytes (`Header`
        // implements a `Marshal()` method, as well as
        // `Question` ..), the marshalling of `Message`
        // becomes simply a sequence of `Marshall` calls
        // which sum up to a sequence of bytes
	payload, _ = queryMsg.Marshal()

        // Write the bytes that represent our query to
        // the "connection"
	_, err = c.conn.Write(payload)

        // Prepare a buf to receive the response.
        // We're assuming here the best scenario where
        // the response is always less than 512 as, if it
        // was, then we'd need to tie together multiple
        // messages.
	buf := make([]byte, 1024)
	_, err = c.conn.Read(buf)

	responseMsg := &Message{}

        // construct the Message struct from the sequence
        // of bytes.
	err = UnmarshalMessage(buf, responseMsg)

        // return the IPs (as we asked for IPv4 address,
        // each one will have 4bytes).

        // 3.4.1. A RDATA format
        //     +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
        //     |                    ADDRESS                    |
        //     +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
        //
        // where: 
        // ADDRESS         A 32 bit Internet address.
        //
        // Hosts that have multiple Internet addresses will have multiple A
        // records.
        ips = make([]string, len(responseMsg.Answers)
	for idx, answer := range responseMsg.Answers {
                ips[idx] = answer.RDATA[idx*4:idx*4+4]
	}

	return
}

With that, we’re set!

There are some other details like compression that are implemented in rawdns that you can check out, but the gist of it is described above.

Closing thoughts

I found the experiment to be very cool.

At first I was expecting a tedious task, but in the end, it was fun to look at how well documented the RFC is and how “future-proof” it was intended to be (even though not everything got really implemented).

I plan to learn more about DNSSEC soon, so this was a good start.

If you want to know more or just get in touch, reach me at @cirowrc on Twitter.

Have a good one!

finis

Before you go, please make sure you check out Computer Networking: A top-down approach; there you can learn more about DNS and other related networking concepts that will help you understand more the whole stack.

Can we manage multiple GitHub repositories in a better way?

Ciro S. Costa — Mon, 25 Dec 2017 00:00:00 +0000

Hey,

I’ve been trying to put myself in the skin of project managers this week and started thinking about some questions related to managing GitHub-based projects. Just trying to grasp what the experience would look like if I needed to gather answers to some very fundamental questions about how my team performs.

The conclusion that came out from this is that project managers might be having a hard time with GitHub.

I see that there are all these tools like ZenHub and CodeTree, but I still don’t see them as great solutions for reporting team activity.

They help when it comes to tracking issues and organizing them, but I have the impression that they can’t go much farther from that (don’t get me wrong, I think it is okay! they do a great job at what they propose).

To get a sense of what’s missing, I started thinking about a project.

What is the project?

Having a very active GitHub user, I know pretty well that there’s a bunch of repositories that, to be honest, I don’t even care, they’re not meant to be tracked, and their activities don’t matter at all.

It turns out that this is also the same for companies - some repositories that are just “lab” experiments, others are quick demos, and some forks were used to contribute to other repositories. No big deal.

Except that it might get in the way when looking for all the GitHub activity across all the repositories.

Can I track just some repositories of all of those that my organization has in GitHub?

Let’s define our project as a set of repositories from all the repositories that our GitHub organization has (it could also aggregate repositories from other organizations).

Think of our project as the intersection of an open source (OSS) only GitHub organization (which has all the repositories public) and a private organization with only private repositories. As I said, not all of them matter and the same is valid for the public ones.

What is the status of our project?

As someone who’s shipping some binaries and that has continuous integration (CI) systems tied to my repositories, it’d be handy to get a sense of how things are going.

Say that github.com/myorg has to release github.com/myorg/docker-image as a Docker image. How can we, in a glance, know whether the current branch is in good shape to be released? I.e., that all the tests pass and that the image is built just fine?

We’ve been doing this forever adding to our README.md a badge that tells people what’s the status of all these sorts of things. I discover a repository out there and then boom, it’s all there.

When you’re at project management though, that’s not how it works (I suppose).

What was the last release of the project repositories?

That’s easy, you might say. Jump to the repository and go to the releases page.

For sure, you can do that. What about when you have 13 repositories and need to make sure things are on track?

What’s the status? Are we green in all master branches? Are the tests falling all the time?

Could I obtain this information from Travis-CI, or maybe Circle-CI? Well, again, sure! However, it turns out that sometimes there’s also Jenkins. Occasionally, other systems advertise their status to commits and pull requests.

Why should I log into each of them just to get a view of the status of the repositories that they’re taking care?

Let’s say GitHub suddenly adds status and release tags to their repositories visualization.

Is that better? Maybe. Does it solve the problem? Well, sometimes, but not ours. Having more than a single organization already brakes the flow of “knowing how things are across multiple organizations.”

Damn, what did I do the whole week?

Monday morning and you go to the standup meeting. Now it’s your time to tell the team what you’ve been working on.

But, wait.

What did I do?

Maybe I’m kinda stupid to not remember things, but I guess there are also some other people out there who just forgive what they’ve been working on last week like me. Perhaps my head goes blank on a topic when I finish working on it, but, who knows? Maybe people are alike.

The point here is that even though you have all your work very documented on GitHub (you created issues, your pull requests were very well written and descriptive, you reviewed pull requests …) there’s no easy way to gather that in a concise manner.

Thinking about the team as a whole we can then multiply that by N. Maybe the other person is also missing something that (s)he did and it would be pretty important for me to know (or just interesting, anyway).

The contribution viz

Looking at what GitHub has we can think about its “contribution map”.

It’s interesting for sure but cmon, that does nothing more than should how “hard” (focus on the air quotes) you’ve been working during some days throughout the year.

Honestly? It doesn’t bring much to the table.

What were my pull requests about?

Next, we have pull request activity. This one is pretty cool, but there’s a little detail that bugs me:

it doesn’t allow me to filter the dates.

What if it’s also essential for me to make sure that my team is having all their pull requests peer-reviewed? What if it matters to know that they’re only getting merged after test passes?

What was I committing?

This topic is a bit controversial in the sense that it might be valid to say that in theory, your commits don’t matter when looking at the high level as every meaningful work would get in the form of pull requests which would be peer-reviewed and tested.

For that reason I think we can come clean out of this.

GitHub could let us filter the dates and maybe the status of the CI in each commit … but perhaps it’s not very useful.

Pull request reviews

From time to time I’ve been migrating from just writing code to reviewing a bunch of code.

Even though I’m somewhat new to this, I think this is a place where proper tooling helps.

Right now it’s pretty good to see what you’ve done.

But what about what I have to do?

In some more busier weeks it might happen that I just can’t review the code right away as the notification comes, but I don’t want to have people pinging me (I wouldn’t like to do that to others) and unfortunately GitHub doesn’t make easy for you to see what’s pending very easily when it comes to reviews.

Good good, what’d be cool then?

Sincerely? I don’t know!

I’ve been recentely experimenting with dgraph and it appears to me that indexing GitHub information in a graph database could lead to great answers.

Not that it’s something that you wouldn’t be able to do right now with their REST interface, but it feels like by reducing the difficulty of exploring the relationships between repositories, labels and pull requests, I could maybe ask more interesting questions.

For instance, consider the following schema:

Id:             int @index(int)         .
Type:           string @index(exact)    .
Name:           string @index(exact)    .
Login:          string @index(exact)    .
ClosedAt:       dateTime @index(hour)   .
Owner:          uid @reverse            .
Repository:     uid @reverse            .

By indexing the repositories, issues, labels and users that are interacting with a repository (say, cirocosta/cr), we could quickly see which labels are the most used querying dgraph with something like:

{
  some_query(func: eq(Name, "cr")) {
    uid
    Name
    Issues: ~Repository {
      Title
      Labels {
        Name
      }
    }
  }
}

which gives us back:

Interesting? Well, maybe. I have the impression that the most interesting is the possibility - have better ways of asking those questions and not being bottlenecked at the retrieval.

My goal is to soon be able to perform some sort of survival analysis on this data.

Survival analysis is a branch of statistics for analyzing the expected duration of time until one or more events happen, such as a death in biological organisms and failure in mechanical systems.

The survival function is a function that gives the probability that a patient, device, or another object of interest will survive beyond any given specified time.[1]

In theory, we should try to minimize this number for bugs assuming that we keep the good practice of always submitting issues and marking them correctly. We should also see that those issues labeled with higher priorities should get lower values (indicating that they “survive” less, in the sense that they get closed soon).

With this kind of calculation we could then estimate the expected amount of time that an issue marked with a given label takes to get into the “death state” (closed).

Closing thoughts

I see that there are many tools that can soon emerge to make querying GitHub data better. The information is there, being produced daily and with great quality. It looks like it’s a matter of exploring it better.

Do share these impressions? Are you using a tool to auxiliate your team gather more insights into how it’s operating?

What would make you improve by knowing data from your development team?

I’d love to hear from you! I’m cirowrc on Twitter, feel free to drop a message there!

Have a good one,

finnis

Creating a hello-world API using Swagger and Go

Ciro S. Costa — Tue, 05 Dec 2017 00:00:00 +0000

Hey,

I’ve been recentely developing, once again, a service that needs to expose an HTTP api which has to be publicly exposed and well documented. Guess what?! That’s not new! Write a new service that talks via HTTP and you’ll have to document the interface and write, again, all the same boilerplate for dealing with logging requests, handling preflight requests made by browsers, checking parameters and so on and so on.

The greatness of using swagger is that we can simply skip all of that part and write a swagger.yml (pretty declarative) which contains the definition of all of our endpoints.

Given the definition, it generates stub code that we implement in our servers in the language you want (you’d probably do this only one time) and then have the client code generated for as many languages as there are swagger generator.

Aside from that, documentation is all done for you - write a decent swagger.yml and you can have automatic documentation generated.

In this tutorial I go through what the process of creating a simple server looks like using go.

ps.: if you want to check the final result, see github.com/cirocosta/hello-swagger.

Getting go-swagger

First step is getting the swagger command line interface.

If you already have go, use go get:

go get -u -v github.com/go-swagger/go-swagger/cmd/swagger

swagger --help

Usage:
  swagger [OPTIONS] <command>

Swagger tries to support you as best as possible when building APIs.

It aims to represent the contract of your API with a language agnostic description of your application
in json or yaml.

...

The tools gives us a handful of commands aimed at generating and validating configurations as well as producing code according to an API description.

Initializing the Golang project

To initialize the project, first create a repository in your $GOPATH. In my case:

cd go/src/github.com/cirocosta
mkdir hello-swagger
cd ./hello-swagger

The next step depends on how you manager your project dependencies. I’ve been using Masterminds/glide for a long time so that’s what I’ll be using (feel free to even not vendor your dependencies if you don’t think that’s a necessary thing).

glide init

$ tree
.
├── glide.lock
└── glide.yaml

The first dependency I’ll retrieve is go-arg, which gives us a simple way of retrieving flags from a CLI application.

glide get github.com/alexflint/go-arg

[INFO]	Preparing to install 1 package.
[INFO]	Attempting to get package github.com/alexflint/go-arg
...
[INFO]	--> Exporting github.com/alexflint/go-arg
[INFO]	Replacing existing vendor dependencies
..

With it we can create main.go, the entrypoint of our application:

// main declares the CLI that spins up the server of
// our API.
// It takes some arguments, validates if they're valid
// and match the expected type and then intiialize the
// server.
package main

import (
	"fmt"
	"github.com/alexflint/go-arg"
)


// cliArgs defines the configuration that the CLI
// expects. By using a struct we can very easily
// aggregate them into an object and check what are
// the expected types.
// If we need to mock this later it's just a matter
// of reusing the struct.
type cliArgs struct {
	Port int `arg:"-p,help:port to listen to"`
}

var (
	// args is a reference to an instantiation of
	// the configuration that the CLI expects but
	// with some values set.
	// By setting some values in advance we provide
	// default values that the user might provide
	// or not.
	args = &cliArgs{
		Port: 8080,
	}
)

// main parses the arguments from the CLI as specified
// by our configuration described in `cliArgs` and then
// populates the `args` reference we defined in the `vars`
// section above.
func main () {
	arg.MustParse(args)
	fmt.Printf("port=%d\n", args.Port)
}

To install it by running a single command we can tailor a Makefile:

install:
	go install -v

.PHONY: install

and then check if all went well:

make
hello-swagger --help

Usage: hello-swagger [--port PORT]

Options:
  --port PORT, -p PORT   port to listen to [default: 8080]
  --help, -h             display this help and exit

hello-swagger --port 1337
port=1337

With the CLI configured, let’s move to the actual API and code generation.

Define the API and generate the code

.
├── Makefile            # makefile that summarizes the
│                       # build and code generation procedures
│       
├── glide.lock          # a lock-file with all the dependencies
│                       # specified with their respective versions
│       
├── glide.yaml          # a high-level definition of our directly
│                       # imported dependencies
│                       
├── main.go             # our application entrypoint
│       
├── swagger             # swagger definition and generated code
│   │       
│   └── swagger.yml
└── vendor              # dependencies
    ├── github.com
    └── ...

swagger: "2.0"
info:
  title: 'Hello'
  version: '0.0.1'
paths:
  /hostname:
    get:
      operationId: 'getHostname'
      produces:
      - 'text/plain'
      responses:
        200:
          description: 'returns the hostname of the machine'
          schema:
            description: 'the hostname of the machine'
            type: 'string'

Update the Makefile to properly produce the code that our server will consume and nothing more

# The `validate` target checks for errors and inconsistencies in 
# our specification of an API. This target can check if we're 
# referencing inexistent definitions and gives us hints to where
# to fix problems with our API in a static manner.
validate:
	swagger validate ./swagger/swagger.yml


# The `gen` target depends on the `validate` target as
# it will only succesfully generate the code if the specification
# is valid.
# 
# Here we're specifying some flags:
# --target              the base directory for generating the files;
# --spec                path to the swagger specification;
# --exclude-main        generates only the library code and not a 
#                       sample CLI application;
# --name                the name of the application.
gen: validate
	swagger generate server \
		--target=./swagger \
		--spec=./swagger/swagger.yml \
		--exclude-main \
		--name=hello


# just added `gen` and `validate`
.PHONY: install gen validate

Now generate the swagger code

make gen
swagger generate server \
		--target=./swagger \
		--spec=./swagger/swagger.yml \
		--exclude-main \
		--name=hello
2017/10/24 14:57:06 building a plan for generation
2017/10/24 14:57:06 planning definitions
2017/10/24 14:57:06 planning operations
2017/10/24 14:57:06 grouping operations into packages
...
  * github.com/jessevdk/go-flags

You can get these now with: go get -u -f swagger/...
...

```
 Check that it has been generated
```

$ tree -L 3
.
├── Makefile
│   ...
├── main.go
└── swagger
    ├── restapi                 # code generated by
    │   │                       # go-swagger.
    │   ├── configure_hello.go
    │   ├── doc.go
    │   ├── embedded_spec.go
    │   ├── operations
    │   └── server.go
    └── swagger.yml             # our definition

Add dependencies to main and let glide update your vendor

// ...

import (
	"github.com/alexflint/go-arg"

        _ "github.com/cirocosta/hello-swagger/swagger/models"
        _ "github.com/cirocosta/hello-swagger/swagger/restapi"
        _ "github.com/cirocosta/hello-swagger/swagger/restapi/operations"
        _ "github.com/go-openapi/loads"
        _ "github.com/go-openapi/runtime/middleware"
        _ "github.com/go-openapi/swag"
)

// ...

$ glide up -v
[INFO]	Downloading dependencies. Please wait...
[INFO]	--> Fetching updates for github.com/alexflint/go-arg
...
[INFO]	--> Fetching updates for github.com/go-openapi/validate
[INFO]	--> Fetching updates for golang.org/x/net
..
[INFO]	Project relies on 24 dependencies.
..

$ tree -L 2
.
├── Makefile
├── glide.lock
├── glide.yaml
├── main.go
├── swagger
│   ├── restapi
│   └── swagger.yml
└── vendor
    ├── github.com      # some new here as well
    ├── golang.org      # new
    └── gopkg.in        # new

Implement the functionality

Now that we have all the dependencies set up we need to create the server and implement the handler functionality.

package main

import (
	"log"
	"os"

	"github.com/alexflint/go-arg"
	"github.com/cirocosta/hello-swagger/swagger/models"
	"github.com/cirocosta/hello-swagger/swagger/restapi"
	"github.com/cirocosta/hello-swagger/swagger/restapi/operations"
	"github.com/go-openapi/swag"
	"github.com/go-openapi/loads"
	"github.com/go-openapi/runtime/middleware"
)

type cliArgs struct {
	Port int `arg:"-p,help:port to listen to"`
}

var (
	args = &cliArgs{
		Port: 8080,
	}
)

func main() {
	arg.MustParse(args)

	swaggerSpec, err := loads.Analyzed(restapi.SwaggerJSON, "")
	if err != nil {
		log.Fatalln(err)
	}

	api := operations.NewHelloAPI(swaggerSpec)
	server := restapi.NewServer(api)
	defer server.Shutdown()

	server.Port = args.Port


        // Implement the handler functionality.
        // As all we need to do is give an implementation to the interface
        // we can just override the `api` method giving it a method with a valid
        // signature (we didn't need to have this implementation here, it could
        // even come from a different package).
	api.GetHostnameHandler = operations.GetHostnameHandlerFunc(
		func(params operations.GetHostnameParams) middleware.Responder {
			response, err := os.Hostname()
			if err != nil {
				return operations.NewGetHostnameDefault(500).WithPayload(&models.Error{
					Code: 500,
					Message: swag.String("failed to retrieve hostname"),
				})
			}

			return operations.NewGetHostnameOK().WithPayload(response)
		})

        // Start listening using having the handlers and port 
        // already set up.
	if err := server.Serve(); err != nil {
		log.Fatalln(err)
	}
}

To verify whether it’s all fine:

# in one terminal session
$ hello-swagger          
2017/10/24 15:32:47 Serving hello at http://[::]:8080

# in another terminal session
$ hostname
cirocosta.local

$ curl localhost:8080/dadsadas
{"code":404,"message":"path /dadsadas was not found"}

$ curl localhost:8080/hostname
cirocosta.local

Done

A minimal Docker Ansible role

Ciro S. Costa — Mon, 04 Dec 2017 00:00:00 +0000

Hey,

there are plenty of Docker Ansible roles out there but it turns out that most of them are overly complex while at the same time it’s very simple to install it.

Actually, most of the times not even Ansible is needed: if you have a single machine and just needs to have it there, Docker already got you covered on how to install Docker:

head to get.docker.com
pick a script that installs it from a specific release channel and done (if you’re curious - or worried - about the development of that script, head to github.com/docker/docker-install).
configure user permissions

If you’re into Ansible though, maybe because you use it to make a bunch of machines reach a given state or makes use of it to bake machine images, here’s my suggestion of minimal Docker role.

The role

The role consists of three directories following Ansible’s patterns (see Ansible - Roles).

If you’re not familiar with it, it looks like this:

docker
├── defaults            # Variables used by the docker role.
│   └── main.yml        # Here I place some default variables
│                       # like the `apt` repository and a default
│                       # docker version to be installed.
│
├── files               # static files that we plan to use in the
│   └── daemon.json     # role (i.e, send to the host) but not
│                       # modify (it's not a template)
│ 
└── tasks               # the actual list of actions to take.
    └── main.yml

The main part of this role is the tasks/main.yml file which defines the actions to take:

---
# As an initial step we make sure that we have our
# dependencies ready. Here we're installing just
# two:
# - apt-transport-https makes us be able to use
#   TLS in the transport of packages coming
#   from APT repositories
# - ca-certificates gives us a bundle of common
#   certificate authorities' certificates
- name: 'install docker dependencies'
  apt:
    name: '{{ item }}'
    state: 'present'
  with_items:
    - 'apt-transport-https'
    - 'ca-certificates'


# Because apt makes use of public key crypto to fetch
# packages we must tell it what's the public key of the
# source that is signing the packages we want to retrieve,
# that is, we need to add the repository's key.
- name: 'add docker repo apt key'
  apt_key:
    url: 'https://download.docker.com/linux/ubuntu/gpg'
    id: '9DC858229FC7DD38854AE2D88D81803C0EBFCD88'
    state: 'present'
  register: 'add_repository_key'
  ignore_errors: true


# Add the official docker apt repository so that `apt`
# can list packages from it and then fetch them from
# there.
# With `update_cache` we force an `apt update` which
# would essentially be the equivalent of updating the
# list of packages from a list of source repositories.
- name: 'add Docker repository'
  apt_repository:
    repo: '{{ docker_apt_repository }}'
    state: 'present'
    update_cache: 'yes'


# With the list of packages updated we can install
# a specific version of the `docker-ce` package. This
# way we can declaratively tell the role which version
# of docker we want: a stable (17.09, for instance) or an 
# edge (17.11-rc3)?
- name: 'install docker'
  apt:
    name: 'docker-ce={{ docker_version }}'
    state: 'present'


# Once Docker has finished the installation (which involves
# setting a systemd service) we have the option to either
# enable that service or not. By enabling it, systemd hooks
# the docker unit into specific places such that whenever the
# machine boots we have this service started.
- name: 'enable docker systemd service'
  service:
    name: 'docker'
    state: 'started'
    enabled: 'yes'


# As we can configure the docker daemon via the configuration
# file `/etc/docker/daemon.json` here we take the opportunity
# of placing one of our own at the relevant destination.
- name: 'prepare default daemon configuration'
  copy:
    src: 'daemon.json'
    dest: '/etc/docker/daemon.json'


# If you use something like `docker swarm mode` it's
# very common to have dangling containers around.
# By setting a cron job to clean thing ups every N
# hours we make sure that dangling containers don't 
# stay around for too long.
- name: 'set periodic docker system prune'
  cron:
    name: 'docker-prune'
    minute: '0'
    hour: '*/2'
    job: 'docker container prune -f'


# Without adding the unprivileged 
# user to the docker group we can't 
# make use of the socket that is activated
# by systemd. Here we take a list of users that
# we want to make part of the `docker` group and
# do it.
- name: 'add users to docker group'
  user:
    name: '{{ item }}'
    groups: 'docker'
    append: 'yes'
  with_items: '{{ docker_group_members }}'
  when: 'docker_group_members is defined'

This file makes use of what’s defined under defaults/main.yml, which looks like this:

---
# The specific version that we aim at installing.
# If we wanted to get whatever is the latest not 
# marked as a release candidate we could instead
# not specify a version but simply `docker-ce`.
docker_version: '17.09.0~ce-0~ubuntu'

# The release channel to look for packages.
# If your curious about what are the channels and which
# versions do they have, head to
# https://download.docker.com/linux/ubuntu/dists/zesty/ (or
# any other distro you want).
docker_apt_release_channel: 'stable'

# The URL of the apt repository.
# Here we're picking the values from the knowledge
# that ansible already took from the system. This way
# we can make fewer changes in this code when changing
# from one distro to another.
# Note that there's a compatibility matrix but in general
# new versions of Ubuntu are well covered.
docker_apt_repository: 'deb https://download.docker.com/linux/{{ ansible_distribution|lower }} {{ ansible_distribution_release }} {{ docker_apt_release_channel }}'

# List of users that we want to add to the
# `docker` group. As mentioned, by adding them to
# the `docker` group they can access the Unix socket
# that Docker places at `/var/run/docker.sock` (by default)
# which the docker CLI uses to communicate with
# the docker daemon.
docker_group_members: 
  - 'ubuntu'

While the Docker daemon configuration looks like:

{
    "experimental": true,
    "icc": false,
    "max-concurrent-downloads": 30,
    "max-concurrent-uploads": 30,
    "metrics-addr": "0.0.0.0:9102",
    "storage-driver": "overlay2",
    "userland-proxy": false
}

If you’re not very sharp about Docker configuration, this is what the last file means:

enabling experimental mode allows us to build images with --squash;
icc means that container won’t be able to communicate with each others over the bridge network;
max-concurrent-(downloads|uploads) increases the number of downloads/uploads (layers) at a given time;
metrics-addr makes the daemon expose Prometheus metrics at all interfaces on port 9102;
storage-driver sets the storage driver to overlay2 (see docs.docker.com - Select a storage driver);
userland-proxy deactivates the use of the userland proxy that docker places for each port mapping - this way we rely solely on iptables for the port mapping. I’d like to go deeper into this one but I’d need to research more first (please send me some articles!).

With the role ready we can now prepare a playbook that makes use of it and test it against a real (in this case, virtual) machine.

Testing it with Vagrant

The easiest way I can think of testing this is making use of Vagrant. I wrote a bit about it when I was setting a VM in an article regarding augmenting swap space in Linux but the gist of it is that it allows us to create a little piece of code that describes how a VM (or a container) should be prepared.

Here, just like in the article I mentioned, I’ll be creating an Ubuntu 17.04 (zesty) machine with 512MB of RAM and 1 CPU. The difference now is that when the machine gets up we’ll be turning ansible against it and executing our role.

To do so, let’s prepare the project directory structure a bit:

.
├── ansible             # ansible related configuration
│   │                   # and execution files
│   │   
│   ├── ansible.cfg     # general configuration
│   ├── playbooks       # playbooks that execute the roles
│   │   │               # and tasks we want
│   │   └── provision-vagrant.yml
│   └── roles           # the docker role described before
│       └── docker
│           ├── defaults
│           │   └── main.yml
│           ├── files
│           │   └── daemon.json
│           └── tasks
│               └── main.yml
└── vagrant             # vagrant configuration
    └── Vagrantfile

With that structure ready, let’s populate the files. The roles/docker/* ones should already be done given the previous sections, now the Vagrantfile should look like the following:

Vagrant.configure(2) do |config|
        # - set the image to be used to be ubuntu/zesty64
        # - set the hostname of the machine
        # - do not check for base image updates
        # - do not sync the default vagrant directory
        config.vm.box = "ubuntu/zesty64"
        config.vm.hostname = "test-machine"
        config.vm.box_check_update = false
        config.vm.synced_folder ".", "/vagrant", disabled: true


        # - configure some parameters from the virtualbox provider
        config.vm.provider "virtualbox" do |v|
                v.memory = 512
                v.cpus = 1
        end


        # hook ansible into the process of provisioning
        # the machine.
        # -     using the ansible configuration file
        #       located at `../ansible/ansible.cfg`, 
        #       execute the playbook at `playbooks/provision-vagrant.yml`
        config.vm.provision "ansible" do |ansible|
                ansible.playbook = "../ansible/playbooks/provision-vagrant.yml"
                ansible.config_file = "../ansible/ansible.cfg"
        end
end

Now moving to the ansible configuration, let’s tell it where it should look for the roles. In ansible/ansible.cfg we can put the following:

[defaults]
roles_path    = ./roles

Now moving to the playbook itself, configure it to target the host that Vagrant will create and make use of the Docker role we just created:

---
# This is a little role that I always execute at first
# when I'm provision machines from the ground up. Because
# some roles might require knowledge about the machine
# (gathered via `gather_facts`) and obtaining such knowledge
# require python, with this role we can bootstrap the 
# python installation with plain ansible SSH communication.
#       -       by not gathering facts and execution 
#               almost manually the `apt install` script
#               we can install it.
# Check the role at https://github.com/cirocosta/example-docker-ansible
- hosts: 'default'
  any_errors_fatal: true
  become: true
  gather_facts: false
  roles:
    - 'bootstrap'


# * Target whatever is the default group of hosts;
# * If any errors happen in the middle, stop the
# whole execution;
# * Make sure to `become` a privileged user;
# * Gather facts about the machine (requires
# python);
# * Execute the `docker` role.
- hosts: 'default'
  become: true
  any_errors_fatal: true
  roles:
    - 'docker'

Having the playbook set up, we can now go back to the root of the repository, enter in the vagrant directory and execute have the ansible execution at the launch of the Vagrant machine.

cd ./vagrant
vagrant up

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/zesty64'...
...
==> default: Machine booted and ready!
...
==> default: Running provisioner: ansible...
    default: Running ansible-playbook...

PLAY [default] ...........................
TASK [bootstrap : bootstrap python] ......
changed: [default]

...

TASK [docker : install docker dependencies] ************************************
ok: [default] => (item=[u'apt-transport-https', u'ca-certificates'])

TASK [docker : add docker repo apt key] ****************************************
ok: [default]
...

That’s it!

Can we simplify it even more?

For sure!

As I mentioned, get.docker.com already install Docker for you. It won’t allow you to specify the version you want but it does the job. As it’s a command that you’d manually copy and paste into the terminal, we can mimic that with Ansible’s command module:

---
# Fetches the docker installation script
# from `get.docker.com` and then pipes the
# script to `sh` so that it gets executed.
- name: 'install docker'
  command: 'bash -c "curl -fsSL https://get.docker.com/ | sh"'

Closing thoughts

From having this Ansible role to having a process of baking AWS AMIs or images for GCE is just a matter of adding Packer to the mix or even manually just targetting a VM and snapshotting it. I tend to keep using a distribution for a reasonable time so I don’t see many reasons to try to make my Ansible roles very complicated with support to a bunch of distros so that’s why I wrote this article.

You can find this example in github.com/cirocosta/example-docker-ansible.

Assuming you have Ansbile and Vagrant installed it should be a matter of running make run from the root of the repository.

Please let me know if I left something uncovered or if you want to share how you’re doing this kind of stuff! I’d really like to know.

Thanks,

Ciro

Incremental backups using GNU Tar and S3

Ciro S. Costa — Sat, 02 Dec 2017 00:00:00 +0000

Hey,

a friend of mine recently told me about a way of performing incremental backups making use of the ubiquitous tar tool.

I was really impressed that such a useful thing could be hidden there without people talking about it (or is it just me who didn’t know? anyway …)

Here are my two cents on how you can tailor a script that will backup a directory incrementally sending the snapshots to S3 and then recovering later.

In the end, I also include restic as an alternative.

Goal

The ultimate goal is pretty simple to describe:

perform backups that we can go back to if we need and that we can store in a remote location;

With an extra feature that is very nice to have: restore the backups up to a specific date.

The idea of restoring to a point-in-time snapshot is that if we mess up and backup our mess we want to go back to a previous backup that wasn’t messed up.

Consider the following example: starting at t0 we perform our initial snapshots that corresponds to an addition of all the present files and directories. Next, in t1, we perform a second snapshot which accounts for only the addition of a new file. In a third time, t2, we perform another snapshot which differs from the last one by only adding a new file. Now assume that at t3 we perform a new snapshot but we shouldn’t have snapshotted as we added a wrong file (e.g, the database was already broken and our snapshotting routine took the snapshot).

If we can go back in time to any snapshot that we want (instead of relying on a single backup), then we can recover even from a bad snapshot taken at the wrong moment. In our example, that would mean not recovering to t3, but instead, recover to t2 where we took a good snapshot.

With that in hands, we can make sure that even if we wrongly backup a directory where things got removed we can still go to a snapshot taken previously and get to that state.

Install the dependencies - `gnu-tar` (macOS)

If you’re using MacOS you can retrieve gnu-tar using brew as it already has a formula (see gnu-tar formula):

brew install gnu-tar
gtar --help

Usage: gtar [OPTION...] [FILE]...
GNU 'tar' saves many files together into a single tape or disk archive, and can
restore individual files from the archive.

Examples:
  tar -cf archive.tar foo bar  # Create archive.tar from files foo and bar.

If you don’t want to have to prefix with g we can pass an additional argument, --with-default-names as we can see in the formula (line 16 from gnu-tar.rb):

  option "with-default-names", "Do not prepend 'g' to the binary"

  # ...

  test do
    tar = build.with?("default-names") ? 
        bin/"tar" : 
        bin/"gtar"
    # ...
  end

so that we install it with the following command:

brew install gnu-tar --with-default-names

I’ll stick with gtar (installation with --with-default-names). You can know whether you’re using the BSD or GNU version by looking at help:

tar --help

       ++==== BSD
       ||
       \/
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
  -c Create  -r Add/Replace  -t List  -u Update  -x Extract
...

Cool, let’s jump to the backups.

Performing incremental backups

The manuals of tar contain a very interesting section Using tar to perform incremental dumps that states:

The option `--listed-incremental' instructs tar to operate 
on an incremental archive with additional metadata stored in 
a standalone file, called a snapshot file. The purpose of this 
file is to help determine which files have been changed, added 
or deleted since the last backup, so that the next incremental 
backup will contain only modified files. The name of the snapshot 
file is given as an argument to the option:

`--listed-incremental=file'

That essentially means that we can create some various tar files and keep track of this sequence of files by mutating an index file that we provide to the --listed-incremental flag.

So, how to perform incremental backups using Tar?

create a list that keeps track of all the files that compose the full state of the system up until a given point in time
create the initial tar with the full current state
keep creating files with only the difference (increments) and updating the file that lists these snapshots that we take.

As an example, assume we have the following file structure:

.
├── rootfs
│   └── 1.txt
└── snapshots

2 directories, 1 file

We want to back up /rootfs and, which right now contains 1.txt, and send the snapshots to the snapshots directory.

Starting with the first and second point (creating a list that keeps track of all the files that compose the full state and then creating the first snapshots that contain all the state):

gtar \
        --create \
        --no-check-device \
        --file=./snapshots/1.tar \
        --listed-incremental=./snapshots/index \
        ./rootfs

which leds us to the following:

tree
.
├── rootfs
│   └── 1.txt
└── snapshots
    ├── 1.tar
    └── index

2 directories, 3 files


cat ./snapshots/index 
GNU tar-1.29-2
1511892293436890000151189201680544478167772208590981353./rootfsY1.txt%

Now mutate the state of rootfs directory one more time by adding 2.txt and 3.txt and then perform a second snapshot:

# mutate the rootfs directory
echo "2.txt" > ./rootfs/2.txt
echo "3.txt" > ./rootfs/3.txt
tree
.
├── rootfs
│   ├── 1.txt
│   ├── 2.txt
│   └── 3.txt
└── snapshots
    ├── 1.tar
    └── index

# perform a second snapshot that should
# contain only the addition of the two files
 gtar \
        --create \
        --no-check-device \
        --file=./snapshots/2.tar \
        --listed-incremental=./snapshots/index \
        ./rootfs

# verify that the new snapshot has been created
# but we still have a single `index` file that
# lists the contents
tree
.
├── rootfs
│   ├── 1.txt
│   ├── 2.txt
│   └── 3.txt
└── snapshots
    ├── 1.tar
    ├── 2.tar
    └── index

# check the updated index file
cat ./snapshots/index 
GNU tar-1.29-2
151189253560533700001511892488471449116167772208590981353./rootfsN1.txtY2.txtY3.txt

# check the contents of the first snapshot
# (it should only contain 1.txt)
tar -tvf ./snapshots/1.tar 
drwxr-xr-x  0 cirocosta wheel       8 Nov 28 16:00 ./rootfs/
-rw-r--r--  0 cirocosta wheel       2 Nov 28 16:00 ./rootfs/1.txt

# check the contents of the first snapshot
# (it should only contain 2.txt and 3.txt)
tar -tvf ./snapshots/2.tar
drwxr-xr-x  0 cirocosta wheel      22 Nov 28 16:08 ./rootfs/
-rw-r--r--  0 cirocosta wheel       6 Nov 28 16:08 ./rootfs/2.txt
-rw-r--r--  0 cirocosta wheel       6 Nov 28 16:08 ./rootfs/3.txt

Cool, having that all we need to do is restore to check if it’s really working.

# create a separate directory to put the 
# restoration files. This is important for
# the demo because otherwise `gtar` would
# remove the files from `rootfs` that were
# not at the snapshot at that given moment.
mkdir ./restore
cd restore
gtar \
        --extract \
        --listed-incremental=../snapshots/index \
        --file=../snapshots/1.tar 

# check if only `1.txt` is there in the 
# rootfs from the first moment
tree
.
└── rootfs
    └── 1.txt

# remove the contents so that we start again
# with a clean rootfs
rm -rf ./rootfs

# to restore all the files until the last snapshot
# we must perform the extraction of all the snapshots
# from the first to the last, one after another
gtar \       
        --extract \
        --listed-incremental=../snapshots/index \
        --file=../snapshots/1.tar 
gtar \
        --extract \
        --listed-incremental=../snapshots/index \
        --file=../snapshots/2.tar 

# now check that all the files of the last snapshot
# are there
tree
.
└── rootfs
    ├── 1.txt
    ├── 2.txt
    └── 3.txt

1 directory, 3 files

That’s it!

Integrating with S3

Someone who has worked with S3 knows that now syncing this to S3 is not a big deal - make use of the AWS CLI and then it’ll push the new / changed files to a bucket that you want.

Besides the simplicity of uploading content to S3, I’d like to highlight a simple way of testing it locally. The easiest way to do it, in my opinion, is using minio which provides us an implementation of the S3 API that we can use locally:

Minio is an open source object storage server with Amazon S3 compatible API. Build cloud-native applications portable across all major public and private clouds.

If you have Docker installed, having minio running is one command away:

# Run a container in the background using the image 
# `minio/minio` publishing the target port 9000 as 9000 
# on the host and initialize it with the command `server`
# with the argument `/data` (location where minio
# will save the contents - this is a directory you'd
# like persisted somehow, here, we don't care).
# With MINIO_ACCESS_KEY and MINIO_SECRET_KEY we set some
# testing credentials that we can use with the AWS CLI tool.
docker run \
        --publish 9000:9000 \
        --name minio \
        --detach \
        --env 'MINIO_ACCESS_KEY=accesskey' \
        --env 'MINIO_SECRET_KEY=secretkey' \
        minio/minio \
        server /data

Now to make use of it we can install the AWS CLI and configure it:

# install `pip` to fetch python packages (in
# this case, `awscli`)
sudo easy_install pip

# fetch the `awscli` package.
pip install awscli --upgrade

Configure the credentials:

aws configure
AWS Access Key ID [None]: accesskey
AWS Secret Access Key [None]: secretkey
Default region name [us-east-1]: 
Default output format [None]:

# enable AWS signature V4 for the minio server
aws configure set default.s3.signature_version s3v4

# make sure we don't have previous environment
# variables set
unset AWS_DEFAULT_REGION    
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY

With everything configured, we can create a bucket and then sync our snapshots:

aws \
        --endpoint-url=http://localhost:9000 \
        s3 mb \
        s3://backups
make_bucket: backups

# get into the snapshots directory
cd ./snapshots

# Sync the current directory (snapshots) with
# the S3 bucket.
# Only new files will be sent on each new run of
# the command. If there are no changes, nothing
# is sent. 
# Because it's our first run, everything is sent.
aws \
        --endpoint-url=http://localhost:9000 \
        s3 sync \
        ./ s3://backups
upload: ./2.tar to s3://backups/2.tar                             
upload: ./1.tar to s3://backups/1.tar                           
upload: ./index to s3://backups/index

Now, to retrieve all the files from that bucket:

# copy from a remote location (s3 bucket `backups`)
# to the current location `./` recursively.
aws \
        --endpoint-url=http://localhost:9000 \
        s3 cp \
        s3://backups ./ \
        --recursive
download: s3://backups/index to ./index                           
download: s3://backups/1.tar to ./1.tar                         
download: s3://backups/2.tar to ./2.tar

Alternative - restic

In the search of a better UX for performing these incremental backups, I stumbled upon restic, a backup program in Go that takes care of sending the backups to a remote location and presents a delightful view of the snapshots that have been taken. Besides that, it also takes care of encrypting the contents and performing integrity checks, pretty cool.

To install it on macOS you can go to the restic/releases GitHub page and take the latest release (in my case, version 0.8.0) and then fetch the .bz2 file, decompress it and put in your $PATH:

# Fetch the .bz2 file with some options:
# --show-error  will fail with error messages
#               if something goes wrong
# --remote-name instructs `curl` to save the
#               file with the name it receives
#               from the remote
# --location    follows redirects if there are
#               any
curl \
        --show-error \
        --remote-name \
        --location \
        https://github.com/restic/restic/releases/download/v0.8.0/restic_0.8.0_darwin_amd64.bz2

# Decompress the file
bzip2 \
        --decompress \
        ./restic_0.8.0_darwin_amd64.bz2

# Move it to somewhere in your $PATH
sudo mv ./restic_0.8.0_darwin_amd64 /usr/local/bin/restic

# Modify the permission bits (make it executable)
sudo chmod +x /usr/local/bin/restic

# Check if it worked
restic --help

Having restic set up we can perform a local snapshot as we performed before with gtar:

# Create a snapshots directory that will
# hold the repository structure that
# restic uses to keep track of the changes
# and store the snapshotted content.
mkdir snapshots

# Create a rootfs that will contain the data
# that we plan to backup.
mkdir rootfs
echo "1" > rootfs/1.txt

        tree
        .
        ├── rootfs
        │   └── 1.txt
        └── snapshots


# initialize the restic repository at `./snapshots`
restic init --repo ./snapshots 

        enter password for new backend: 
        enter password again: 
        created restic backend 100efe3e68 at ./snapshots


# check how the filesystem looks like after 
# the initialization
tree
        .
        ├── rootfs
        │   └── 1.txt
        └── snapshots
            ├── config
            ├── data
            │   ├── 00
            │   ├── 01
            ...
            │   └── ff
            ├── index
            ├── keys
            │   └── 04b12ce19886a5d25...
            ├── locks
            └── snapshots

# Perform our first snapshot of `rootfs`
# saving it in our repository that we
# initialized before.
restic --repo ./snapshots backup ./rootfs 

        enter password for repository: 
        password is correct
        scan [./rootfs]
        scanned 1 directories, 1 files in 0:00
        [0:00] 100.00%  0B/s  2B / 2B  2 / 2 items  0 errors  ETA 0:00 
        duration: 0:00, 0.00MiB/s
        snapshot 9171fe56 saved


# With the snapshot creation succeeded, let's
# check how the snapshots dir changed.
tree
        .
        ├── rootfs
        │   └── 1.txt
        └── snapshots
            ├── config
            ├── data
            │   ├── 00
            ...
            │   ├── 06          # some new stuff
            │   │   └── 0623cae88fefc31f07c09a11e8743b9eb94877883d458a6bcc2ec8a870bccf37
            ...
            ├── index           # index dir has some content
            │   └── 89cbbfae048c49f7abbaa5ac929971550537eb97195039f0a70220890b08d259
            ...
            ...
            └── snapshots       # snapshot!
                └── 9171fe56882a6ac7d4cc8836ec096dc761698dd2d47bfe8b0f9a6b82c061b7dc


# Having the snapshot created we can
# now create a `restored` directory where
# we'll put the results of the restoration
mkdir restored

# Perform the restoration by pointing restic
# at the snapshots directory and a target
# directory that will receive the restored files.
restic \
        --repo ./snapshots \
        restore \
        --target ./restored \
        latest

        enter password for repository: 
        password is correct
        restoring <Snapshot 9171fe56 of [./rootfs] (...) to ./restored


# Go to the `restored` directory ad check if it worked
cd restored 
tree
        .
        └── rootfs
            └── 1.txt

# We can also check what are the files in
# a given snapshot as well as list all the
# snapshots we've taken so far.

# Let's first list the snapshots
restic --repo ./snapshots snapshots

        enter password for repository:
        password is correct
        ID        Date                 Host             Tags        Directory
        ----------------------------------------------------------------------
        9171fe56  2017-12-01 09:17:51  cirocosta.local              ./rootfs
        ----------------------------------------------------------------------
        1 snapshots

# With the ID of the snapshot we can then
# inspect the contents of the snapshot
restic \
        --repo ./snapshots \
        ls 9171fe56

        enter password for repository:
        password is correct
        snapshot 9171fe56 of (...):
        /rootfs
        /rootfs/1.txt

# Now, what if we create a new file, snapshot
# it and then check the contents?

# Create a new file:
echo "2" > ./rootfs/2.txt

# Take the new snapshot
restic --repo ./snapshots backup ./rootfs
        enter password for repository: 
        password is correct
        using parent snapshot 9171fe56
        scan [./rootfs]
        scanned 1 directories, 2 files in 0:00
        [0:00] 100.00%  0B/s  4B / 4B  3 / 3 items  0 errors  ETA 0:00 
        duration: 0:00, 0.00MiB/s
        snapshot 3b5e8701 saved


# Look at the snapshots that we created
restic --repo ./snapshots snapshots      
        ID        Date                 Host             Tags        Directory
        ----------------------------------------------------------------------
        9171fe56  2017-12-01 09:17:51  cirocosta.local              /tmp/cc/rootfs
        3b5e8701  2017-12-01 10:38:12  cirocosta.local              /tmp/cc/rootfs
        ----------------------------------------------------------------------
        2 snapshots


# Check the state of the latest snapshot
# (we `latest` is an alias for the last snapshot)
restic --repo ./snapshots ls latest      
        /rootfs
        /rootfs/1.txt
        /rootfs/2.txt


# Interesting. Looking at the latest snapshot
# reveals all the files, not only the addition
# of `2` - it shows us the complete representation
# of the final state.

# If we look at the first snapshot though,
# then it shows only the final state up to
# that point in time.
 restic --repo ./snapshots ls 9171fe56
        /rootfs
        /rootfs/1.txt

It works! We have our files there.

It’s important to note that right now restic does not have support for compression but it performs deduplication across the backups and the backups are all incremental.

As it already has S3 support we can test it against minio:

# run the minio container just like before
docker run \
        --publish 9000:9000 \
        --name minio \
        --detach \
        --env 'MINIO_ACCESS_KEY=accesskey' \
        --env 'MINIO_SECRET_KEY=secretkey' \
        minio/minio \
        server /data


# Initialize the S3 (minio) repository passing
# the AWS credentials as environment variables
# and the repository pointing to our local minio
# instance
export AWS_ACCESS_KEY_ID=accesskey
export AWS_SECRET_ACCESS_KEY=secretkey
restic \
        --repo s3:http://localhost:9000/restic-test \
        init

        enter password for new backend: 
        enter password again: 
        created restic backend 0871674e7e at s3:http://localhost:9000/restic-test


# now, perform the snapshot:
restic \
        --repo s3:http://localhost:9000/restic-test \
        backup ./rootfs 

        enter password for repository: 
        password is correct
        scan [/tmp/cc/rootfs]
        scanned 1 directories, 2 files in 0:00
        [0:00] 100.00%  0B/s  4B / 4B  3 / 3 items  0 errors  ETA 0:00 
        duration: 0:00, 0.00MiB/s
        snapshot e870fc54 saved


# check that restic created the contents there 
# in S3 (minio):
aws \
        --endpoint-url=http://localhost:9000 \
        s3 ls --recursive \
        s3://restic-test

        2017-12-01 10:48:41        155 config
        2017-12-01 10:49:01        178 data/25/25fe4bc29169f29...
        2017-12-01 10:49:01       1312 data/5b/5b525003c0450a4...
        2017-12-01 10:49:01        661 index/e691cc0bd3ebc2e9c...
        2017-12-01 10:48:41        460 keys/50a8c58f10d1c4ae1d...
        2017-12-01 10:49:01        247 snapshots/e870fc54bb482...

Closing thoughts

I was very happy knowing that there’s an easy way of creating these incremental snapshots using standard Linux utilities. For sure there are better ways (or more modern at least) but this one described in the article seems to be pretty much enough for various cases.

Restic reveals to be pretty good for my use case (performing some snapshots and having them sent directly to S3) but it has some limitations like not having compression. As I never used it very extensively, that’s the biggest one for me (ps.: if you want to see a comparison of Restic against some other tools, make sure you check https://github.com/gilbertchen/benchmarking).

Some resources used for this article are:

GNU Tar Manual - Performing incremental dumps - great section in the man pages of tar on how to perform incremental backups;
Source of the GNU-TAR homebrew formula;
minio - the S3-compatible storage used in this article so simulate S3;
github.com/restic/restic - the backup tool to use in comparison with the TAR method.

What about you? Have you been performing incremental backups? What are you using to do it? Please let me know, I’m cirowrc on Twitter and would really like to know more about!

Have a good one,

Ciro.

finis

Updating the Docker version in Travis-CI

Ciro S. Costa — Tue, 28 Nov 2017 00:00:00 +0000

Hey,

for some good time, Travis has been allowing anyone to run Docker in their infrastructure - see Using Docker in Builds - be it on a paid or free plan.

Something that is not outlined there is that it’s very common to not have the latest version of Docker running in your builds. For some people, that’s a problem.

Updating the version

To update the version we must, before using Docker in the build, set a script that will fetch Docker from the official apt repository and perform the upgrade.

This script can either be inlined in the .travis.yaml definition or be called via an extra file.

Personally, I prefer an extra file.

sudo: 'required'

language: 'generic'

services:
  - 'docker'

before_install:
  - './.travis/main.sh'

script:
  - 'docker version'

notifications:
  email: false

And the script looks like this:

#!/bin/bash

# Performs any provisioning needed for a clean build.
#
# This script is meant to be used either directly
# as a `before_install` step such that the next step
# in the Travis build have the environment properly 
# setup.
#
# The script is also handy for debugging - SSH into
# the machine and then call `./.travis/main.sh` to
# have all dependencies set.

set -o errexit

main() {
  setup_dependencies

  echo "INFO:
  Done! Finished setting up Travis-CI machine.
  "
}

# Takes care of updating any dependencies that the
# machine needs.
setup_dependencies() {
  echo "INFO:
  Setting up dependencies.
  "

  sudo apt update -y
  sudo apt install --only-upgrade docker-ce -y

  docker info
}

main

Let it run and Docker should be updated.

Looking at build system information we can see which version Travis used as the Docker version by default:

docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false

Then, verify that our provisioning step got executed:

$ ./.travis/main.sh
INFO:
  Setting up dependencies.
  
Ign:1 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty InRelease
Hit:2 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty-updates InRelease
....

And see that we end up with the desired version:

$ docker version
Client:
 Version:	18.02.0-ce
 API version:	1.36
 Go version:	go1.9.3
 Git commit:	fc4de44
 Built:	Wed Feb  7 21:16:47 2018
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm
Server:
 Engine:
  Version:	18.02.0-ce
  API version:	1.36 (minimum version 1.12)
  Go version:	go1.9.3
  Git commit:	fc4de44
  Built:	Wed Feb  7 21:15:21 2018
  OS/Arch:	linux/amd64
  Experimental:	false

That’s it!

Conclusion

In Travis-CI it’s almost guaranteed that you’ll not have, by default, the latest version of Docker. Nevertheless, it’s very easy to update it: just apt update && apt upgrade <docker> and you’re good to go.

Just in case you need some guidance, here’s an example repository I set up to make this even clearer: cirocosta/sample-travis-docker-update.

If you’re curious about the execution, check it out: https://travis-ci.org/cirocosta/sample-travis-docker-update.

Having any questions, make sure you ping me on Twitter! I’m @cirowrc there.

Have a good one!

finis

Forcing (from inside) the redirection of all outputs of a bash script to a file

Ciro S. Costa — Mon, 27 Nov 2017 00:00:00 +0000

Hey,

this is a quick tip that might save you some time if you need to have all the output of a script going out to a file and you don’t have control over the execution of it.

Here I go through two possible ways to tackle the problem:

using exec to redirect the stdout and stderr to a specific file;
using a subprocess and controlling its stdout and stderr.

Both achieve the same, but are slightly different.

When do you need this?

The typical case for needing to ensure (from within the script) that the logs end up in a file is when you can’t control a shell execution of your script (if you had, then you could simply forward both standard out and standard error to a file without modifying the script).

Note.: if you’re using AWS EC2 AutoScaling groups with a custom script for initializing your instance, you don’t necessarily need to use this to get your logs sent to a file - you can already grab them at /var/log/cloud-init-output.log even if you don’t use cloud-init -, unless you want to change the default file.

Using exec to redirect stdout and stderr

The whole thing is rather simple, although it relies on some bash details that most people don’t know:

#!/bin/bash

set -o errexit

readonly LOG_FILE="/var/log/script.log"

# Create the destination log file that we can
# inspect later if something goes wrong with the
# initialization.
sudo touch $LOG_FILE

# Make sure that the file is accessible by the user
# that we want to give permissions to read later
# (maybe this script is executed by some other user)
sudo chown ubuntu $LOG_FILE

# Open standard out at `$LOG_FILE` for write.
# This has the effect 
exec 1>$LOG_FILE

# Redirect standard error to standard out such that 
# standard error ends up going to wherever standard
# out goes (the file).
exec 2>&1

The whole deal is about making use of exec, which is document in man bash:

      exec [-cl] [-a name] [command [arguments]]
              If command is specified, it replaces the shell.  
              No  new  process  is  created.
              The  arguments  become the arguments to command.

So, nothing fancy, just like a execvp syscall would do - replace the image of the current process and initialize the program we pass.

The catch is in the end of the description of the command though:

              ...
              If command is not specified, any redirections  
              take  effect in the current shell, and the return 
              status is 0.  If there is a redirection error, the 
              return status is 1.

It means that by making use of exec without a program, we can affect the redirection of the current process (the one executing our commands).

To understand that better, I started pausing the script at several break points to see what’s going on under the hood from the perspective of /proc/$PROC/fd.

At the first moment (I), nothing fancy - the standard redirections to the pseudo terminal are set:

ls -lah /proc/$(pgrep script.sh)/fd
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
255 -> /tmp/script.sh

At the second momement though (II), the file has already been open and the standard out got redirected:

ls -lah /proc/$(pgrep script.sh)/fd
0 -> /dev/pts/1                 # stdin, untouched
1 -> /var/log/script.log        # redirected!
2 -> /dev/pts/1                 # not redirected to the log file
                                # yet
255 -> /tmp/script.sh

At the third moment (III - final of the script), as expected, we have everybody redirecting to the log file:

0 -> /dev/pts/1                 # stdin, untouched
1 -> /var/log/script.log
2 -> /var/log/script.log
255 -> /tmp/script.sh           # other file descriptors - untouched

If you’re extra curious, take a look at the use of syscalls during the script execution using strace:

# Open the log file with 666 permissions, create
# if it doesn't exist (`O_CREAT`), start writing
# in the file from the very beginning if it already
# exists (O_TRUNC) and open it as write only 
# (`O_WRONLY`).
#
# With the succesfull `openat`, it tells us that
# we can access it via the file descriptor `3`.
openat(AT_FDCWD, 
        "/var/log/script.log", 
        O_WRONLY|O_CREAT|O_TRUNC, 
        0666) = 3
...

# dup2 duplicates a file descriptor (first 
# argument) creating a copy of it and giving
# it the number specified in the second 
# argument (1, in this case). 
#
# This comes from `1>$LOG_FILE` where `$LOG_FILE`
# is being pointed to as the file descriptor `3`.
dup2(3, 1)                              = 1

# Close the first file descriptor associated
# with out log files as we don't use it directly
# in the rest of the application - it was only
# openned to be duplicated later.
close(3)                                = 0

# Duplicate the stdout to also be used as the
# stderr (`2>&1`).
dup2(1, 2)                              = 2

Redirecting subprocess stdout and stderr

Using bash, any command (or series of them) inside a parenthesis group (( and )) gets executed in a subshell (a copy of the original shell process but in a forked one).

From the bash man pages:

       A  compound  command is one of the following.  
       In most cases a list in a command's description
       may be separated from the rest of the command by 
       one or more newlines, and may be followed  by
       a newline in place of a semicolon.

 >>>>> (list) list  is  executed in a subshell environment.

Let’s take an example:

#!/bin/bash

set -o errexit

say_hello() {
  echo "Hello! pid=$BASHPID"
}

# Calls `say hello` from the main
# process
say_hello

# Creates a subshell (in a sub process)
# and then calls `say_hello`.
#
# By sleeping we can look at `pstree` and
# figure out how the process tree looks
# like (the outer process will wait this
# inner process)
(
  say_hello
  sleep 3600
)

The execution leads to:

# Run the script and leave it running in the
# background
./script.sh &
Hello! pid=20127
Hello! pid=20128

# Check out the process tree of the recently
# executed script (`./script.sh`)
pstree -p $!
script.sh(20127)
        ───script.sh(20128)
                ───sleep(20129)

Being a compound command, IO redirection can be properly done:

#!/bin/bash

set -o errexit

readonly LOG_FILE="/var/log/script.log"

say_hello() {
  echo "Hello! pid=$BASHPID"
}

# Calls `say hello` from the main
# process.
#
# Logs the hello message to the process
# stdout as no redirection has been
# performed at this point.
say_hello

# Creates a subshell (in a sub process)
# and then calls `say_hello`.
#
# The subshell that gets created has both
# stdout and stderr redirected to the LOG
# file `/var/log/script.log`.
(
  say_hello
  sleep 3600
) &>$LOG_FILE

Execute the script and then inspect /proc/<pid>/fd and you should see the redirection taking place:

# Look at the file descriptors of the 
# parent process
ls -lah /proc/20370/fd
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
255 -> /tmp/script.sh

# Look at the file descriptors of the
# child process (subshell) that got
# the redirection:
ls -lah /proc/20371/fd
0 -> /dev/pts/1
1 -> /var/log/script.log
2 -> /var/log/script.log

Closing thoughts

Maybe you’ll never need to do this kind of thing, but now you know how it works haha.

If you have any questions or just want to chat about something, reach me at @cirowrc on Twiter.

Have a good one!

finis

Inspecting Docker images without pulling them

Ciro S. Costa — Sun, 26 Nov 2017 00:00:00 +0000

Hey,

depending on what you’re trying to build it might happen that part of it involves inspecting a Docker image from a registry but you can’t afford to pull it.

It turns out that there’s an API that allows you to perform exactly that - be it DockerHub or a private registry.

The Docker Registry HTTP API is the protocol to facilitate distribution of images to the docker engine. It interacts with instances of the docker registry, which is a service to manage information about docker images and enable their distribution.

Testing it locally

The first step to test it locally is raising a registry from the library/registry image.

# Run a container named `registry` in the background.
# Map the internal port 5000 to the host on port 5000.
docker run \
        --detach \
        --name registry \
        --publish '5000:5000' \
        registry

Check that it’s definitely working:

# Fetch an image from an external registry.
# Because we're not explicitly specifying a
# registry address it's going to use dockerhub's 
# address. 
# For instance, we could use registry-1.docker.io/library/registry
# instead of just `registry` (library repository is also
# another default value).
docker pull busybox

# Tag the image we just fetched to now be named
# `localhost:5000/test`.
docker tag busybox localhost:5000/test

# Push the image. 
# Because we have explicitly specified a registry
# in the name of the image, it'll be pushed to the
# registry at the given address (localhost:5000)
docker push localhost:5000/test

The push refers to repository [localhost:5000/busybox]
0271b8eebde3: Pushed 
latest: digest: sha256:91ef6c1c52b166be02645b8efee30d1ee65362024f7da41c404681561734c465 size: 527

Having the local registry working, we can move to the script that inspects images right from the registry metadata. The following script contains all that it takes to retrieve it and relies only on two dependencies: bash and jq.

#!/bin/bash

set -o errexit

# Address of the registry that we'll be 
# performing the inspections against.
# This is necessary as the arguments we
# supply to the API calls don't include 
# such address (the address is used in the
# url itself).
readonly REGISTRY_ADDRESS="${REGISTRY_ADDRESS:-localhost:5000}"


# Entry point of the script.
# If makes sure that the user supplied the right
# amount of arguments (image_name and image_tag)
# and then performs the main workflow:
#       1.      retrieve the image digest
#       2.      retrieve the configuration for
#               that digest.
main() {
  check_args "$@"

  local image=$1
  local tag=$2
  local digest=$(get_digest $image $tag)

  get_image_configuration $image $digest
}


# Makes sure that we provided (from the cli) 
# enough arguments.
check_args() {
  if (($# != 2)); then
    echo "Error:
    Two arguments must be provided - $# provided.

    Usage:
      ./get-image-config.sh <image> <tag>

Aborting."
    exit 1
  fi
}


# Retrieves the digest of a specific image tag,
# that is, the address of the uppermost of a specific 
# tag of an image (see more at 
# https://docs.docker.com/registry/spec/api/#content-digests).
# 
# You can know more about the endpoint used at
# https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest
get_digest() {
  local image=$1
  local tag=$2

  echo "Retrieving image digest.
    IMAGE:  $image
    TAG:    $tag
  " >&2

  curl \
    --silent \
    --header "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    "http://$REGISTRY_ADDRESS/v2/$image/manifests/$tag" |
    jq -r '.config.digest'
}


# Retrieves the image configuration from a given
# digest.
# See more about the endpoint at:
# https://docs.docker.com/registry/spec/api/#pulling-a-layer
get_image_configuration() {
  local image=$1
  local digest=$2

  echo "Retrieving Image Configuration.
    IMAGE:  $image
    DIGEST: $digest
  " >&2

  curl \
    --silent \
    --location \
    "http://$REGISTRY_ADDRESS/v2/$image/blobs/$digest" |
    jq -r '.container_config'
}


# Run the entry point with the CLI arguments
# as a list of words as supplied.
main "$@"

ps.: It’s important to note that the API calls need to specify the type of content that it accepts (application/vnd.docker.distribution.manifest.v2+json).

Let’s now check if it’s working for real:

chmod +x ./get-image-config.sh
./get-image-config.sh test latest

{
  "Hostname": "3fbce8bb8947",
  "Domainname": "",
...
  "Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  ],
  "Cmd": [
    "/bin/sh",
    "-c",
    "#(nop) ",
...
  "OnBuild": null,
  "Labels": {}
}

Cool, it indeed works!

While that does the job for registries that require no authentication, it’s not suitable for DockerHub. When checking a public image there, we need to perform an extra step before getting the digest of an image - retrieve a token. With such token in hand, we can then inspect either public or private images.

Going back to scripting, let’s create a different one to deal with this case - call it get-public-image-config.sh (this is for brevity sake, using some other programming language you could place some conditionals and detect each case).

The additional code can be placed in a method called get_token which only takes image as an argument:

# Retrieves a token that grants access to the
# registry.docker.io (dockerhub registry) access
# to pull a specific image.
#
# note.:        the token that we retrieve is valid only
#               for that image.
# note.:        we get the token from `auth.docker.io` and
#               not `registry.docker.io`.
# usage.:       get_token "library/nginx"
#
get_token() {
  local image=$1

  echo "Retrieving Docker Hub token.
    IMAGE: $image
  " >&2

  curl \
    --silent \
    "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" \
    | jq -r '.token'
}

With the token in hands it’s just a matter of making use of it on the other calls.

If we were targetting private images we’d modify get_token a little bit: on the call to get the token from auth.docker.io we’d need to do it with a DockerHub username and password pair (without authentication we can only have access to public images). To do so, specify in that call an authorization header (--user flag in curl):

# Retrieves a token that grants access to
# a private image named `image` on registry.docker.io.
# note.:        the user identified by `DOCKER_USERNAME`
#               and `DOCKER_PASSWORD` must have access
#               to the image.
get_token() {
  local image=$1

  echo "Retrieving Docker Hub token.
    IMAGE: $image
  " >&2

  curl \
    --silent \ 
    --u "$DOCKER_USERNAME:$DOCKER_PASSWORD" \
    "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" \
    | jq -r '.token'
}

With the token in hards we could now retrieve the digest of a given image and tag (pay attention to the extra Authorization: Bearer $token header that we added):

# Retrieve the digest, now specifying in the header
# the token we just received.
# note.:        $token corresponds to the token
#               that we received from the `get_token`
#               call.
# note.:        $image must be the full image name without
#               the registry part, e.g.: `nginx` should
#               be named `library/nginx`.
get_digest() {
  local image=$1
  local tag=$2
  local token=$3

  echo "Retrieving image digest.
    IMAGE:  $image
    TAG:    $tag
    TOKEN:  $token
  " >&2

  curl \
    --silent \
    --header "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    --header "Authorization: Bearer $token" \
    "https://registry-1.docker.io/v2/$image/manifests/$tag" \
    | jq -r '.config.digest'
}

With that we can have a full script that retrieves public images from Dockerhub (check how in main we first retrieve a token, then we pass that token to the following methods):

#!/bin/bash

set -o errexit

main() {
  check_args "$@"

  local image=$1
  local tag=$2
  local token=$(get_token $image)
  local digest=$(get_digest $image $tag $token)

  get_image_configuration $image $token $digest
}

get_image_configuration() {
  local image=$1
  local token=$2
  local digest=$3

  echo "Retrieving Image Configuration.
    IMAGE:  $image
    TOKEN:  $token
    DIGEST: $digest
  " >&2

  curl \
    --silent \
    --location \
    --header "Authorization: Bearer $token" \
    "https://registry-1.docker.io/v2/$image/blobs/$digest" \
    | jq -r '.container_config'
}

get_token() {
  local image=$1

  echo "Retrieving Docker Hub token.
    IMAGE: $image
  " >&2

  curl \
    --silent \
    "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" \
    | jq -r '.token'
}

# Retrieve the digest, now specifying in the header
# that we have a token (so we can pe...
get_digest() {
  local image=$1
  local tag=$2
  local token=$3

  echo "Retrieving image digest.
    IMAGE:  $image
    TAG:    $tag
    TOKEN:  $token
  " >&2

  curl \
    --silent \
    --header "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    --header "Authorization: Bearer $token" \
    "https://registry-1.docker.io/v2/$image/manifests/$tag" \
    | jq -r '.config.digest'
}

check_args() {
  if (($# != 2)); then
    echo "Error:
    Two arguments must be provided - $# provided.
  
    Usage:
      ./get-image-config.sh <image> <tag>
      
Aborting."
    exit 1
  fi
}

main "$@"

Note.: again, you must add the full name of the image (official images use the library repository so nginx should be referred as library/nginx).

To make sure that it works, run it against an image like nginx:

# because `nginx` comes from the set of
# official images we must prepend `library/`.
./get-public-image-config.sh library/nginx latest

Retrieving Docker Hub token.
    IMAGE: library/nginx
  
Retrieving image digest.
    IMAGE:  library/nginx
    TAG:    latest
    TOKEN:  eyJhbGciVFER...ZsNw
  
Retrieving Image Configuration.
    IMAGE:  library/nginx
    TOKEN:  eyJhb...ZsNw
    DIGEST: sha256:9e7424e5dbaeb9b28fea44d8c75b41ac6104989b49b2464b7cbbed16ceeccfc3
  
{
  "Hostname": "22e255475f18",
  "Domainname": "",
  ...
  "Labels": {
    "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
  },
  "StopSignal": "SIGTERM"
}

Extra - old images

If you try to retrieve an image that is not very new (say, that has some 2 years) you’ll notice that the script I posted above might not work.

The reason for that is that images that have been pushed to the docker registry a long time ago won’t use the second version of the V2 manifest. However, they still present the image configuration even though in a regular string.

Bellow is a script that deals with that case:

#!/bin/bash

set -o errexit

main() {
  check_args "$@"

  local image=$1
  local tag=$2
  local token=$(get_token $image)
  local old_config=$(get_old_config $image $tag $token)

  get_image_configuration "$old_config"

}

get_image_configuration () {
  local old_config=$1

  echo "$old_config" | jq -r '.history[0].v1Compatibility' | jq '.container_config'
}

get_token() {
  local image=$1

  echo "Retrieving Docker Hub token.
    IMAGE: $image
  " >&2

  curl \
    --silent \
    "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" \
    | jq -r '.token'
}

get_old_config() {
  local image=$1
  local tag=$2
  local token=$3

  echo "Retrieving image digest.
    IMAGE:  $image
    TAG:    $tag
    TOKEN:  $token
  " >&2

  curl \
    --silent \
    --header "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    --header "Authorization: Bearer $token" \
    "https://registry-1.docker.io/v2/$image/manifests/$tag" \
    | jq -r '.'
}

check_args() {
  if (($# != 2)); then
    echo "Error:
    Two arguments must be provided - $# provided.
  
    Usage:
      ./get-image-config.sh <image> <tag>
      
Aborting."
    exit 1
  fi
}

main "$@"

If you’re looking for the difference, look at main. Essentially we abandon the idea of retrieving a digest and simply pick the “old config”. From the old config, we look at the first blob in the list which represents the uppermost layer - the layer that contains all the info altogether. From there we parse that plain-text JSON and then get the config.

Update

From the feedbacks the article received, two alternatives (that you can use right now) appeared constantly:

github.com/GoogleCloudPlatform/container-diff - Diff your Docker containers - it looks pretty interesting but even though you can specify remote:// to an image when analyzing it, it looks like it always pulls the entire image. Maybe I got something wrong?
github.com/projectatomic/skopeo - “Work with remote images registries - retrieving information, images, signing content” - well, does exactly what it says! For the scope of retrieving image / repositories configuration before pulling them, totally worth it.

By the way, if you wan’t to try Skopeo, building it from source is very easy on MacOS:

# install gpgme
brew install gpgme

# clone the repository to the golang-specific directory
# (make sure you have Go installed and GOPATH set 
# before).
git clone \
        https://github.com/projectatomic/skopeo \
        $GOPATH/src/github.com/projectatomic/skopeo

# get into the repository you just clones
cd $GOPATH/src/github.com/projectatomic/skopeo 

# run the binary-local target 
# (this is a regular `go build` with some
# tags and flags set.
make binary-local

# now you should have `skopeo` in the repository
# directory
./skopeo --help

NAME:
   skopeo - Various operations with container images and container image registries

USAGE:
   skopeo [global options] command [command options] [arguments...]
   
VERSION:
   0.1.28-dev commit: 78b29a5c2f05b4026876728e7651bad31193216c

Now inspecting an image or a repository from Dockerhub is one command away:

./skopeo \
        --override-os=linux \
        inspect docker://docker.io/fedora
{
    "Name": "docker.io/library/fedora",
    ...
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:a8ee583972c2295bb76704d4defe5116d5e4dd7ba3767aaa2cc8fcf71088ee06"
    ]
}

Note that here I’m specifying the --override-os flag to the command. The reason is that otherwise it’ll try to inspect the image or repository filtering by digests marked with OS=darwin. If you’re using Linux you’d not need to use that flag.

Closing thoughts

Interacting with DockerHub or a private registry is not all that hard, it’s just not very documented. Having these scripts it becomes pretty easy to get it working on any language you want - just add some checks, parse the image names and you should be good to go.

Here are the resources mentioned in the article:

Please let me know if I got anything wrong and/or if there’s an easier way to do it.

Have a good one!

finis

Getting TLS certificates with Letsencrypt and HAProxy

Ciro S. Costa — Sat, 25 Nov 2017 00:00:00 +0000

Hey,

with the upcoming release of HAProxy 1.8 (see the blog post at haproxy.com) it’ll be possible to keep your stack behind the goodness of http2 without changing your code at all.

That’s pretty cool as you can have very perceptive differences under real-life scenarios. One thing to notice is that browsers only establish these connections if you’re HTTPS ready, and that means having TLS certificates in your load-balancer (or regular server).

Here are my 2 cents on how you can have a fully functioning HAProxy set up with certificate generation via Letsencrypt.

Dependencies

There are a handful of dependencies that need to be in place: certbot, lua, openssl-dev,zlib-dev, libpcre-dev and haproxy itself. If you already have all of those set, just skip to the next session: HAProxy Configuration section.

Here I’m making use of Ubuntu zesty (17.04) so there might be some differences between what I document here and your OS specifics.

Certbot

Start by adding certbot’s apt repository:

# after issuing the command press `enter` to accept
sudo add-apt-repository ppa:certbot/certbot

 This is the PPA for packages prepared by Debian Let's Encrypt Team and backported for Ubuntu(s).
 More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keybox '/tmp/tmp028qrfr4/pubring.gpg' created
gpg: /tmp/tmp028qrfr4/trustdb.gpg: trustdb created
gpg: key 8C47BE8E75BCA694: public key "Launchpad PPA for certbot" imported
gpg: Total number processed: 1
gpg:               imported: 1
OK

then update:

sudo apt update -y

Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty InRelease
Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-updates InRelease [89.2 kB]  
Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-backports InRelease [89.2 kB]
Get:4 http://ppa.launchpad.net/certbot/certbot/ubuntu zesty InRelease [21.3 kB]         
Hit:5 http://security.ubuntu.com/ubuntu zesty-security InRelease        
...

and now install the desired package, python-certbot:

sudo apt install -y python-certbot

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  certbot dialog python-acme python-asn1crypto python-certifi python-chardet
  python-configargparse python-configobj python-cryptography python-dialog
...


certbot help

-------------------------------------------------------------

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. The most common SUBCOMMANDS and flags are:

...

Once we have that we know that certbot is ready to request certificates from Letsencrypt for us.

HAProxy

Before installing HAProxy itself we need to get its dependencies right. HAProxy by itself can’t serve content from a directory like a static hosting web server would so to do that we must make use of lua which allows us to very easily extend HAProxy functionality. Then, the first dependency we’ll get is lua.

# set the version of Lua that we want to install.
# is a variable that we can reference later.
LUA_VERSION=5.3.3

# install a development library that lua depends on
sudo apt install -y \
        libreadline-dev

# fetch lua's source code for the version we want
curl \
        -SOL https://www.lua.org/ftp/lua-$LUA_VERSION.tar.gz

# create a directory to hold that source code
sudo mkdir \
        -p /usr/src/lua 

# extract the source code to the directory we want
sudo tar \
        -xzf lua-$LUA_VERSION.tar.gz \
        -C /usr/src/lua --strip-components=1 

# build Lua using a concurrency equivalent to the number
# of processors we have (and targetting Linux)
sudo make \
        -C /usr/src/lua \
        -j "$(getconf _NPROCESSORS_ONLN)" \
        linux 

# install it so that it's widely accessible
sudo make \
        -C /usr/src/lua \
        install

To proceed with the HAProxy installation we need to know where the lua headers and compiled library are:

# search for the headers
find /usr/local/include -name "*lua*"
/usr/local/include/lua.h
/usr/local/include/lualib.h
/usr/local/include/lua.hpp
/usr/local/include/luaconf.h

# search for the lib
find /usr/local/lib -name "*lua*"
/usr/local/lib/liblua.a
/usr/local/lib/lua

The next dependencies can all be fetched from apt though. We need three:

OpenSSL: for full TLS support (e.g, SNI extension);
PCRE: to not rely on libc’s regex support but the more common Perl regex
ZLIB: for providing deflate and gzip compression algorithms

sudo apt install -y \
        libpcre3-dev \
        libssl-dev \
        zlib1g-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
...

Now that our dependencies are all ready to be used we can proceed with HAProxy itself.

For obtaining the source code we can head to the downloads page (haproxy.org/download/1.8/src/) and fetch the latest version (I’m getting the last release candidate of 1.8 because this version gives us HTTP2 support, which is something I plan to write about soon).

# fetch the .tar.gz of the source code of haproxy 
curl -SOL http://www.haproxy.org/download/1.8/src/haproxy-1.8-rc3.tar.gz

# decompress it to the current directory
tar xzf ./haproxy-1.8-rc3.tar.gz

Once everything is in place it’s just a matter of compiling it with some flags that signalize the build process what is our environment and where it can find the Lua dependency:

# get into the source code directory
cd ./haproxy-1.8-rc3/

# build the code with Lua, gzip compression, PCRE regex
# targetting a recent kernel.
make \
        TARGET=linux2628 \
        USE_OPENSSL=1 \
        USE_ZLIB=1 \
        USE_PCRE=1 \
        USE_LUA=1 \
        LUA_LIB_NAME=lua \
        LUA_LIB=/usr/local/lib/ \
        LUA_INC=/usr/local/include

# make it available from `$PATH` so that
# we can call `haproxy` from anywhere and
# also have `man` pages set
sudo make install

If you wonder why TARGET=linux2628, head to the Makefile at haproxy’s source and check this:

ifeq ($(TARGET),linux2628)
  # This is for standard Linux >= 2.6.28 with 
  # netfilter, epoll, tproxy and splice.
  USE_NETFILTER         = implicit
  USE_POLL              = implicit
  USE_EPOLL             = implicit
  USE_TPROXY            = implicit
  USE_LIBCRYPT          = implicit
  USE_LINUX_SPLICE      = implicit
  USE_LINUX_TPROXY      = implicit
  USE_ACCEPT4           = implicit
  USE_FUTEX             = implicit
  USE_CPU_AFFINITY      = implicit
  ASSUME_SPLICE_WORKS   = implicit
  USE_DL                = implicit
  USE_THREAD            = implicit
else

In summary, it means that it’ll use some recent kernel capabilities to improve our performance.

After the compilation finishes you should have something like this:

./haproxy -vvvv
HA-Proxy version 1.8-rc3-34650d5 2017/11/11
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

From the output, we can make sure that what we wanted has been properly compiled. Now we can move to the HAProxy configuration.

HAProxy Configuration

With all the dependencies installed we can proceed with the actual HAProxy configuration.

Because we need to have HAProxy performing something that it can’t do by default as mentioned - serve static files from a directory - we need to add a Lua plugin that does the job. That’s needed to serve the challenge that letsencrypt gives us when checking if we can serve content from a given domain (webroot).

Thanks to Jan (github.com/janeczku), we don’t have to write our own:

# get out of the haproxy directory
cd ../

# fetch the lua script code that will serve the challenges
# placed by certbot at a specific directory (webroot)
git clone https://github.com/janeczku/haproxy-acme-validation-plugin

# go to the repository directory and check what are the 
# files that we have there
cd ./haproxy-acme-validation-plugin
tree
.
├── LICENSE
├── README.md
├── acme-http01-webroot.lua
├── cert-renewal-haproxy.sh
└── haproxy.cfg.example

In the acme-http01-webroot.lua make sure you set the non_chroot_webroot variable to a location where we’ll set certbot to put challenges on:

--
-- Configuration
--
-- When HAProxy is *not* configured with the 'chroot' option you must set an absolute path here and pass 
-- that as 'webroot-path' to the letsencrypt client

acme.conf = {
	["non_chroot_webroot"] = "/tmp/webroot"
}

With the Lua plugin in place all we need to do next is specify that path to the plugin in our haproxy.cfg:

# notice that we're specifying in 
# `lua-load` the location of the Lua
# script. Make sure you reference it
# according to your configuration.
global
                maxconn                   8192
                log                       127.0.0.1  local0
                tune.maxrewrite           16384
                tune.bufsize              32768
                tune.ssl.default-dh-param 2048
                max-spread-checks         200
                spread-checks             5
		lua-load                  /home/ubuntu/haproxy-acme-validation-plugin/acme-http01-webroot.lua


# Configure some default values that
# frontends and backends can inherit
# Here I'm setting some dummy timeouts.
# In another blog post we can go through
# some real values there.
defaults
                log       global
                retries   3
                option    redispatch
                option    dontlog-normal
                mode      http

                timeout   http-request 10m
                timeout   client 10m
                timeout   connect 10m
                timeout   server 10m
                timeout   http-keep-alive 10m
                timeout   tunnel 10m
                timeout   client-fin 10m
                timeout   server-fin 10m

# The HTTP frontend serving only as a way of redirecting traffic
# to port :443 where it can serve the real content via HTTPS
# and also accept the requests from letsencrypt.
# The big thing here is `use-service` which essentially means that
# when the `url_acme_http01` ACL is `true` it'll execute the lua 
# script that we registered.
# The name `lua.acme-http01` comes from the lua script itself:
#
#       core.register_service("acme-http01", "http", acme.http01)
#
frontend        http
		bind		*:80
                acl		url_acme_http01 path_beg /.well-known/acme-challenge/
                http-request	use-service lua.acme-http01 if METH_GET url_acme_http01
                redirect	scheme https if METH_GET !url_acme_http01

# Serve the HTTPS traffic.
# For each request that comes it takes the server name indicated 
# in SNI (the TLS extension that gives to an encrypted connection the 
# equivalent of a Host header) it looks on the list of certificates
# that it loaded from the `crt-list` file and then uses that certificate.
# As we're offloading the TLS resolution from the backend to the 
# frontend, `backend_test` will receive unencrypted content as
# if there was a plain TCP connection coming (no need to deal with
# certificates there - only HAProxy has to care about it).
frontend        https
		bind		*:443
                default_backend backend_test


# Dummy backend connecting haproxy to a server on the same machine.
# The server at `localhost:8080` is a simple HTTP server expecting
# regular non-encrypted connections.
backend         backend_test
                server test-server localhost:8080

To make sure we got our configuration right we can make use of the -c flag of HAProxy to perform some sort of “dry run” and only check the config. Assuming we have the configuration at ~/haproxy.cfg:

haproxy --help 
...
Usage : haproxy -f <cfgfile|cfgdir>] [opt]
        ...
        -c check mode : only check config files and exit
        ...

haproxy -c -f ~/haproxy.cfg
Configuration file is valid

As the configuration is valid we can start HAProxy with a privileged user (as it’s going to bind on ports 80 and 443):

sudo haproxy -f ./haproxy.cfg
[info] 328/123916 (374961) : [acme] http-01 plugin v0.1.1

Now that we got HAProxy working we can check if the lua plugin is really receiving the requests when we hit port 80 on the expected path:

# in a separate terminal, make a request to the
# haproxy instance on the path which should respond
# to letsencrypt requests
curl localhost:80/.well-known/acme-challenge/aa
resource not found

# on the HAProxy terminal, check the logs
[warning] 328/124202 (374961) : [acme] http-01 token not found: aa (client-ip: 127.0.0.1)

As it’s all working as expected we can request a certificate to letsencrypt.

# Create the directory where challenges will
# be placed. This is the directory that you have
# to be set earlier in the lua script in the
# `acme.conf` field.
mkdir /tmp/webroot

# make certbot initiate the certificate generation
# process using the webroot method, placing challenges
# at the directory /tmp/webroot and creating the certificate
# for the domain `cirocosta.com`
sudo certbot certonly --webroot -w /tmp/webroot -d cirocosta.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
...
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/cirocosta.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/cirocosta.com/privkey.pem
   Your cert will expire on 2018-02-23. To obtain a new or tweaked
...

With the certificates in hand now we have to create a new file that HAProxy can use to terminate the TLS connections. This new file is a concatenation of the private key and the certificate we received from letsencrypt:

# concatenate those files (as I used the root user to
# provision them - via certbot - they were created as
# root, so I must be `root` to see them).
sudo cat \
        /etc/letsencrypt/live/cirocosta.com/privkey.pem \
        /etc/letsencrypt/live/cirocosta.com/fullchain.pem > \
                ~/cirocosta.com

Then update the haproxy.cfg file to make use of the certificate when binding to port 443:

 frontend        https
#                bind		*:443   (before)
                 bind		*:443   ssl crt /home/ubuntu/cirocosta.com
                 default_backend backend_test

Then perform a soft-reload HAProxy after checking if the configuration is fine (it checks the cert):

# check the configuration
haproxy -c -f ~/haproxy.cfg 
Configuration file is valid

# soft-reload the current instance by creating a new
# one passing the pid of the old instance via the `-sf`
# flag
sudo haproxy \
        -f ./haproxy.cfg \
        -sf $(pidof haproxy)
[info] 328/131241 (376196) : [acme] http-01 plugin v0.1.1

# in the terminal where the old HAProxy was running
# you should see the following (perfectly fine) messages:
[WARNING] 328/131241 (374961) : Stopping frontend http in 0 ms.
[WARNING] 328/131241 (374961) : Stopping frontend https in 0 ms.
[WARNING] 328/131241 (374961) : Stopping backend backend_test in 0 ms.
[WARNING] 328/131241 (374961) : Proxy http stopped (FE: 3 conns, BE: 0 conns).
[WARNING] 328/131241 (374961) : Proxy https stopped (FE: 0 conns, BE: 0 conns).
[WARNING] 328/131241 (374961) : Proxy backend_test stopped (FE: 0 conns, BE: 0 conns).

And that’s it!

Even without a properly configured server, we should already be able to see that the HTTPS setup is fine:

Closing thoughts

Even though the guide is quite extensive if you consider the process of building HAProxy and Lua from scratch, there’s not much going on:

Make HAProxy serve the challenges at port 80 on a specific path that letsencrypt expects
Concatenate certificates and private keys
Update the HAProxy configuration
Reload HAProxy

Once that’s automated you’re ready to serve your customers with HTTPS / TLS without dropping connections and having to pay for certificates.

Even if the part of setting up the dependencies seem like too much you can tailor a Dockerfile that does all of that and you’re good to go.

If you enjoyed the guide and are willing to learn more about related content, make sure you subscribe to the newsletter. In case you find mistakes I made or think there’s something to improve, just let me know, I’m cirowrc on Twitter.

Have a good one!

finis

Update (13 Feb, 2018)

If you’re interested in provisioning TLS certificates, you’re probably also interested in HTTP/2.

I just finished writing a blog post on how to make use of HTTP/2 Server Push with NGINX for those out there making use of NGINX.

Have a good one!

Pushing Docker images right from Travis-CI

Ciro S. Costa — Fri, 24 Nov 2017 00:00:00 +0000

Hey,

oftentimes I have some code that ends up in a Docker image which I want to have pushed to Dockerhub. Docker itself is capable of automatically building images from git repositories but it has the restriction that I can’t specify a certain Docker daemon version to run the build. This used to be important (not sure if it still is) as in the stable version of Docker we already have been using multi-stage builds but the Dockerhub builder didn’t allow us to use it.

As I already make use of Travis to run my tests I think it’s a no-brainer to tie the image building to Travis as well - if tests pass and we’re on the master branch, build the image and push to docker hub.

Updating the docker daemon

As I mentioned, I like to force Travis to use a newer Docker than they use by default. To speed some things I also like to configure /etc/docker.json so that we can increase the concurrency of downloads and uploads (it really matters depending on how many images you have and whether you build them in parallel).

To achieve that I use the following structure:

.
├── .travis
│   └── main.sh
├── .travis.yml
├── README.md
├── Dockerfile
└── Makefile

.travis is a directory that holds scripts to update the Travis machine. This way if I need to use the debug feature of Travis I don’t have to follow the entire .travis.yml recipe and instead just run ./.travis/main.sh from there to have the dependencies set.

The file looks like the following:

#!/bin/bash

# Set an option to exit immediately if any error appears
set -o errexit

# Main function that describes the behavior of the 
# script. 
# By making it a function we can place our methods
# below and have the main execution described in a
# concise way via function invocations.
main() {
  setup_dependencies
  update_docker_configuration

  echo "SUCCESS:
  Done! Finished setting up Travis machine.
  "
}

# Prepare the dependencies that the machine need.
# Here I'm just updating the apt references and then
# installing both python and python-pip. This allows
# us to make use of `pip` to fetch the latest `docker-compose`
# later.
# We also upgrade `docker-ce` so that we can get the
# latest docker version which allows us to perform
# image squashing as well as multi-stage builds.
setup_dependencies() {
  echo "INFO:
  Setting up dependencies.
  "

  sudo apt update -y
  sudo apt install realpath python python-pip -y
  sudo apt install --only-upgrade docker-ce -y

  sudo pip install docker-compose || true

  docker info
  docker-compose --version
}

# Tweak the daemon configuration so that we
# can make use of experimental features (like image
# squashing) as well as have a bigger amount of
# concurrent downloads and uploads.
update_docker_configuration() {
  echo "INFO:
  Updating docker configuration
  "

  echo '{
  "experimental": true,
  "storage-driver": "overlay2",
  "max-concurrent-downloads": 50,
  "max-concurrent-uploads": 50
}' | sudo tee /etc/docker/daemon.json
  sudo service docker restart
}

main

With that set, we can move on to the Travis configuration file (.travis.yml):

# make use of vm's 
sudo: 'required'

# have the docker service set up (we'll
# update it later)
services:
  - 'docker'

# prepare the machine before any code
# installation scripts
before_install:
  - './.travis/main.sh'

# first execute the test suite.
# after the test execution is done and didn't
# fail, build the images (if this step fails
# the whole Travis build is considered a failure).
script:
  - 'make test'
  - 'make image'

# only execute the following instructions in
# the case of a success (failing at this point
# won't mark the build as a failure).
# To have `DOCKER_USERNAME` and `DOCKER_PASSWORD`
# filled you need to either use `travis`' cli 
# and then `travis set ..` or go to the travis
# page of your repository and then change the 
# environment in the settings pannel.
after_success:
  - if [[ "$TRAVIS_BRANCH" == "master" ]]; then
      docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD ;
      make push-image ;
    fi

# don't notify me when things fail
notifications:
  email: false

As you might have noticed the whole functionality is behind a Makefile (from the make commands). That’s on purpose to facilitate possible debugging sessions (you’d SSH into the Travis machine and just ./.travis/main.sh, make test to run the whole thing).

The Makefile can be as simple as something like the following:

IMAGE := cirocosta/dind

test:
	true

image:
	docker build -t $(IMAGE) .

push-image:
	docker push $(IMAGE)


.PHONY: image push-image test

If you want an example to check out, see cirocosta/dind.

Closing thoughts

There’s no need to complicate a lot the process of automatically pushing images after a successful test run using travis-ci. I really enjoy the idea of only pushing an image when things are fine as it gives a good guarantee that the code that is being packed as the image will run.

To be even more sure about the wellness of the image you could add an extra suit of integration tests that creates docker containers using the image and then asserts that it works as expected.

If you’d like to have a container-based environment that runs your unit tests and on the side (concurrently) run a build in a VM that prepares your docker images and run integration tests, you can totally do that with Travis-ci - it’s a matter of adapting a previous article I wrote: Concurrently running container-based and VM tests in Travis-CI.

If you’d like to learn more about integrating Travis-ci into your projects or automation in general, make sure you subscribe to the mailing list.

Here are some resources related to the article:

Have a good one!

finis

Changing the name of an index in Elasticsearch

Ciro S. Costa — Tue, 21 Nov 2017 00:00:00 +0000

Hey,

today I happened to write a script to solve a specific problem that it looks like a good deal of people face: rename a Elasticsearch index.

Naturally, there are documented solutions but I didn’t find quickly a script that would get me where I wanted - all the data from an index named a now being queryable in an index named b with all the properties set.

Note.: the following code is aimed at Elasticsearch 2.4.6.

Here it comes then.

Reindexing step by step

There are four steps to get towards our goal:

 Create an Elasticsearch index and populate it with some data;

 Get the configurations of the original index;

 Create the new index with the desired configuration;

```
 Run `_reindex` action;
```
```
 Drop the old index.
```

0. Create an Elasticsearch index and populate it with some data

To create an index using the default parameters (e.g, number of shards and replicas) we can issue a POST against the Elasticsearch HTTP endpoint specifying the desired index (in this case, acme-production:

curl \
        -XPOST \
        http://localhost:9200/acme-production
{
    "acknowledged": true
}

Which, naturally, has no data indexed:

curl localhost:9200/_cat/indices\?v

health status index           pri rep docs.count docs.deleted store.size pri.store.size 
green  open   acme-production   1   0          0            0       130b           130b

Now we populate it with some data:

curl \
        -XPOST \
        -d '{"title":"Hello world"}' \
        http://localhost:9200/acme-production/test/hello

{
    "_id": "hello",
    "_index": "acme-production",
    "_shards": {
        "failed": 0,
        "successful": 1,
        "total": 1
    },
    "_type": "test",
    "_version": 1,
    "created": true
}

Which we can verify by looking again at the /_cat/indices endpoint:

curl localhost:9200/_cat/indices\?v

health status index           pri rep docs.count docs.deleted store.size pri.store.size 
green  open   acme-production   1   0          1            0      2.9kb          2.9kb

1. Get the configurations of the original index

Because the renaming is nothing more than “create, copy and delete” we need to create a new index with the properties from the old one. To properly achieve that we must then copy the old configuration:

index_config=$(curl \
        localhost:9200/acme-production/_settings,_mappings | \
        jq '.[]')

echo "$index_config"

{
    "mappings": {
        "test": {
            "properties": {
                "title": {
                    "type": "string"
                }
            }
        }
    },
    "settings": {
        "index": {
            "creation_date": "1511285883996",
            "number_of_replicas": "0",
            "number_of_shards": "1",
            "uuid": "h3z8Hq_5SKuu7MwEJZYcVQ",
            "version": {
                "created": "2040699"
            }
        }
    }
}

ps.: here I’m making use of jq, the lightweight command-line JSON processor, in order to get the mappings and settings objects from within the bigger object returned by the call to /<index>/_settings,_mappings. This way we’re able to assign that to a variable and then utilize directly later.

2. Create the new index with the desired configuration

Using the old configuration (stored in the index_config variable) we’re able to create the the index based on it:

curl \
        -XPUT \
        -d "$index_config" \
        http://localhost:9200/acme-staging

{
    "acknowledged": true
}

ps.: even though there’s an uuid in the $index_config object there, it doesn’t matter - it’ll get replaced by a new uuid in the new index.

3. Run `_reindex` action

Having both indices properly configured we’re ready to have the data from the old index in the new one:

source=acme-production
dest=acme-staging
payload="{
  \"source\": {
    \"index\": \"$source\"
  },
  \"dest\": {
    \"index\": \"$dest\",
    \"version_type\": \"internal\"
  }
}"

curl \
        -XPOST \
        -d "$payload" \
        http://localhost:9200/_reindex

{
    "batches": 1,
    "created": 1,
    "failures": [],
    "noops": 0,
    "requests_per_second": "unlimited",
    "retries": 0,
    "throttled_millis": 0,
    "throttled_until_millis": 0,
    "timed_out": false,
    "took": 48,
    "total": 1,
    "updated": 0,
    "version_conflicts": 0
}

By this time you should already have your new index populated. Now it’s a matter of deleting the old index:

4. Drop the old index

If you have no intention of making use of the old index, now it’s time to drop it:

curl \
        -XDELETE \
        http://localhost:9200/acme-production

{
    "acknowledged": true
}

Update - Using aliases and avoiding complete index copies

I received some pretty interesting feedback on Reddit that I’d like to share here.

It turns out that sometimes we can avoid reindexing by making use of aliases (see Elasticsearch Indices Aliases).

The idea is that when we need to reference what’s covered by an index by using another name we can create some kind of “pointer” to the real index and perform all the normal operations against this pointer (alias). The API that allows doing that allows us to essentially CRUD (create, remove, update and delete) aliases, making it totally possible for us to perform what we want: achieve a “renaming” of an index, even if only virtually.

Let’s do it then.

First, create an acme-production index just like before and add some data:

curl \
        -XPOST \
        http://localhost:9200/acme-production
curl \
        -XPOST \
        -d '{"title":"Hello world"}' \
        http://localhost:9200/acme-production/test/hello

health status index           pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   acme-production   5   1          1            0      3.4kb          3.4kb

then create an alias named acme-staging:

payload='{
    "actions": [
        {
            "add": {
                "alias": "acme-staging",
                "index": "acme-production"
            }
        }
    ]
}'

curl \
        -XPOST \
        -d "$payload" \
        http://localhost:9200/_aliases

if we check the indices we’ll see that we don’t have any new index though:

curl localhost:9200/_cat/indices\?v
health status index           pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   acme-production   5   1          1            0      3.4kb          3.4kb

But that we do have aliases:

curl localhost:9200/_cat/aliases\?v
alias        index           filter routing.index routing.search 
acme-staging acme-production -      -             -

which allows us to perform queries against acme-staging and retrieve data from acme-production:

# request against the index
curl http://localhost:9200/acme-production/test/hello
{"_index":"acme-production","_type":"test","_id":"hello","_version":1,"found":true,"_source":{"title":"Hello world"}}

# now against the alias
curl http://localhost:9200/acme-staging/test/hello
{"_index":"acme-production","_type":"test","_id":"hello","_version":1,"found":true,"_source":{"title":"Hello world"}}

Now, what if we want to disallow requests to the old index? As if we had really renamed it and not duplicated? Then we need to close the old index using the open/close index api:

curl -XPOST http://localhost:9200/acme-production/_close

then we can try to get from acme-production:

 curl http://localhost:9200/acme-production/test/hello   

{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index":"acme-production"}],"type":"index_closed_exception","reason":"closed","index":"acme-production"},"status":403}

Cool, what we wanted, huh? Now, if we try to get from acme-staging:

curl http://localhost:9200/acme-staging/test/hello
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index":"acme-production"}],"type":"index_closed_exception","reason":"closed","index":"acme-production"},"status":403}

we can’t retrieve either.

It sounds logical to me that we can’t as the alias is just a pointer to the other index (which was closed).

So, to sum up, if you want to have new indices to point to an existing one (as if you were renaming), aliases will save you and you’ll need to perform 0 copying of data.

If you need to have something like “rename” and disallow access to the old index, then alias won’t help you (will have to use the reindex + delete strategy.

I never used aliases before and it’s pretty good to know that they exist! It can definitely be very useful some times.

Closing thoughts and resources

As someone who never really dug deep into how Elasticsearch works, I found very easy the whole concept of reindexing. The official documentation is pretty good and with it, I was able to quickly solve the problem. Kudos Elasticsearch team!

I plan to do some more posts regarding Elasticsearch in the future. If you want to come along and learn together, make sure you subscribe to the mailing list.

Thanks,

finis

Executing multiple commands in SSH session against multiple machines

Ciro S. Costa — Mon, 20 Nov 2017 00:00:00 +0000

Hey,

it’s not uncommon for me to have to execute a quick command against a set of machines. Naturally, the easiest way of performing that against a single machine is using ssh:

readonly private_key='./key.rsa'
readonly command='echo test'
readonly ip='10.0.0.2'
readonly user='ubuntu'
readonly port='22'

echo "$command" | ssh \
        -i ${private_key} \
        -p ${port} \
        ${user}:${ip}

Having the command properly wrapped in terms of variables now it’s just a matter making it a method, looping through a list of machines and executing the command.

# Executes a given command via SSH
#       $1:             IP of the machine;
#       $2:             port of the machine to 
#                       connect to.
#       $USER:          user to use when executing
#                       the commands.
#       $SCRIPT:        global variable with the
#                       full multiline script to
#                       execute.
#       $PRIVATE_KEY:   global variable with the
#                       full path to a private
#                       key that is authorized by
#                       the machines.
execute_script_in_machine () {
        local ip=$1
        local port=$2

        echo "$SCRIPT" | ssh \
                -o "StrictHostKeyChecking=no" \
                -i "$PRIVATE_KEY" \
                -p $port \
                $USER@$ip
}

The list of machines might be an array that we just iterate through:

readonly TARGETS=(
        "127.0.0.1:2200"
        "127.0.0.1:2201"
        "127.0.0.1:2202"
)

As we provide the port argument separated from the ip argument we must split the strings during the iteration. Thankfully to bash we are covered by some internal operators. For instance:

#!/bin/bash

readonly TARGETS=(
        "firstIP:firstPORT"
        "secondIP:secondPORT"
)

# when iterating over an array we must 
# get each element from the array accessor
# `VARIABLE[@]` and not only `VARIABLE`.
main () {
        local ip
        local port

        for target in "${TARGETS[@]}"; do
                ip=${target%:*}
                port=${target#*:}

                printf "ip=$ip\tport=$port\n"
        done
}

main

leads to the following:

bash test.sh 
ip=firstIP      port=firstPORT
ip=secondIP     port=secondPORT

which essentially means that now we need to plumb these two pieces (the parsing of the target list and the command execution) and have the full working script:

#!/bin/bash

readonly USER="ubuntu"
readonly PRIVATE_KEY="./key.rsa"
readonly SCRIPT='echo -----STARTING;
  echo "WhoAmI? $(whoami)";
  echo -----DONE;
'
readonly TARGETS=(
  "127.0.0.1:2020"
  "127.0.0.1:2021"
  "127.0.0.1:2022"
)

main () {
        local ip
        local port

        for target in "${TARGETS[@]}"; do
                ip=${target%:*}
                port=${target#*:}
        done
}

execute_script_in_machine () {
        local ip=$1
        local port=$2

        echo "$SCRIPT" | ssh \
                -o StrictHostKeyChecking=no \
                -i "$PRIVATE_KEY" \
                -p $port \
                $USER@$ip
}

main

That’s it!

I included an extra section after some feedback. It describes an alternative called mpssh that does the job of executing commands over SSH across many machines. Make sure you check it out!

Closing thoughts

This is just a quick script that might be handy if you want to inspect the state of something across many machines. I really don’t advise to make use of this as a way of provisioning machines as there are much better solutions out there (like ansible) which take a more declarative approach, are less distribution-specific and provides the great benefit of idempotence.

Having some bash skills are nice and can really save some time if need to get something done quickly using standard Unix tools. If you’re interested in knowing more about some bash/shell stuff, make sure you subscribe to the mailing list.

I don’t have many resources to share regarding this article but I always end up reading something here and there from the advanced bash-scripting guide on TLDP. I’d not stress too much on a guide as at least in my case most of what I know comes from solving stuff when needed. After a while, you get the gotchas and become good enough to get to the goal.

Have a good one!

finis

Update - `mpssh` as an alternative

Hey, a reader pointed out that there are some other very lightweight solutions that do the job very well. The alternative I liked the most is mpssh.

If you’re a MacOS user and uses brew: brew install mpssh. Under Linux, clone the repository ndenev/mpssh and then run make from the root (assuming you have at least gcc and make).

SYNOPSIS
        mpssh \
                [-besvV] \
                [-o directory] \
                [-u username] \
                [-f hosts] \
                [-p procs] \
                <command>

   The mpssh utility executes multiple parallel ssh binary 
   instances in order to connect to a list of hosts (specified 
   in the hosts file) and execute the given <command> on each 
   of them.

So, first thing to do is create a hosts file. There you can specify in each line the ip that you want to connect to. For instance:

echo "10.0.0.1
10.0.0.2
10.0.0.3"  > /tmp/hosts

Next step is configuring ~/.ssh/config: mpssh seems to now have an easy way of specifying a private key to use so you have to specify an IdentityFile in the ~/.ssh/config file. For instance:

Host *
    IdentityFile /tmp/key.rsa

With that we can already execute a command across those machines:

mpssh --user ubuntu --nokeychk --file /tmp/hosts 'echo "test"'

MPSSH - Mass Parallel Ssh Ver.1.3.3
(c)2005-2013 Nikolay Denev <ndenev@gmail.com>

  [*] read (3) hosts from the list
  [*] executing "echo "test"" as user "ubuntu"
  [*] strict host key check disabled
  [*] spawning 3 parallel ssh sessions

10.0.0.1 -> test
10.0.0.2 -> test
10.0.0.3 -> test

  Done. 3 hosts processed.

Aside from just executing the commands in all of them, it can have a ratelimit in terms of parallelization (e.g, given a list of 100 machines, execute in parallel 3 at a time). It’s also possible to output the result of the execution of each machine in a separate file. Pretty good!

Concurrently running container-based and VM tests in Travis-CI

Ciro S. Costa — Sat, 18 Nov 2017 00:00:00 +0000

Hey,

oftentimes I see myself having to run two types of tests in travis-ci: unitary and integration tests.

For the first category, that usually means running some plain Golang code that interacts with no other services (e.g, it uses a mocked database or it mocks HTTP requests). In the other scenario, spawning Docker containers which end up creating a database that is used in the test - requiring docker to be present, thus, VM-based builds.

Using the matrix property we can tailor our build matrix and make use of both types of build infrastructures: container-based (pretty fast, super good for unit tests) and VM-based (pretty good for raising services and Docker containers).

Snippet

Below is a very commented snippet (.travis.yml) detailing how to run more than one type of build infrastructure. Note that here I’m making use of language: go but that works just fine for anything else.

language: 'go'

go:
  - '1.9.1'

# `matrix` allows us to include or exclude environments
# specifying properties for each of them.
#
# As `sudo` is a property like any other, we can change
# it in `matrix` such that we can have one build environment
# running in a container - useful for very quick unit tests - and 
# another build running in a Travis VM - where we can make
# use of `docker` and create many containers, very suitable
# for integration testing.
matrix:
  include:
    # mark via environment-variables that we're in a unit-test
    - env: 'TEST_SUITE=unit'
      sudo: false

    # sudo-enabled specific configuration
    - env: 'TEST_SUITE=integration'
      sudo: true
      services:
        - 'docker'
      # most of the times I set a `setup.sh` script that updates
      # the image dependencies so that if I want to debug the
      # build later it's very easy to run all the setup by just
      # running a single command.
      before_install: 
        - './.travis/setup.sh'
      before_script:
        - 'make image'

# These properties that are not in `include` will be run for both
# environments.
install:
  - 'make'

# Having `test-<TEST_SUITE>` different in each environment we can
# then run different types of tests in different environments.
script:
  - make test-${TEST_SUITE}

# Evict sending notifications when things fail.
# I'm definitely not a fan of being alerted by
# this when developing.
notifications:
  email: false

That’s it! Let’s create a test project and check how that works.

 create a new GitHub repository (in this case I'm creating [cirocosta/travis-multiinfra](https://github.com/cirocosta/travis-multiinfra));

 sync your account in `travis-ci.org` or `travis-ci.com` (works for both offerings);

```
 enable builds for that repository
```

 populate the repo with some code and a `yaml` file like the above

 push to git and check the build at Travis-ci.

In cirocosta/travis-multiinfra’s .travis.yml I have a simplified .travis.yml:

language: 'go'

go:
  - '1.9.1'

matrix:
  include:
    - env: 'TEST_SUITE=unit'
      sudo: false

    - env: 'TEST_SUITE=integration'
      sudo: true
      services:
        - 'docker'

install:
  - 'true'

script:
  - 'true'

notifications:
  email: false

When your build starts you should see two instantiations: one for the container-based build and another for the VM-based.

Note: the amount of parallelization depends on your plan. If you’re on the free plan on travis.org or you’re already running other builds you won’t be able to execute multiple builds in parallel. You’ll still have the multiple infrastructures though.

To check configuration was used for each job, head to one of them and then click on view config.

Here’s how the unitary (container) looks like:

{
    "dist": "trusty",
    "env": "TEST_SUITE=unit",
    "go": "1.9.1",
    "group": "stable",
    "install": [
        "true"
    ],
    "language": "go",
    "os": "linux",
    "script": [
        "true"
    ],
    "sudo": false
}

and the integration (VM):

{
    "dist": "trusty",
    "env": "TEST_SUITE=integration",
    "go": "1.9.1",
    "group": "stable",
    "install": [
        "true"
    ],
    "language": "go",
    "os": "linux",
    "script": [
        "true"
    ],
    "services": [
        "docker"
    ],
    "sudo": true
}

Note that the vm-based contains the service as we want but the first (container-based) doesn’t.

Closing thoughts

Being a long time Travis-ci user I really enjoy the flexibility that it brings as well as the peace of mind that having a CI infrastructure setup brings. I got very excited when Travis announced container-based build and migrated some projects right away (it was mostly a matter of putting sudo: false and replacing the apt get ... calls). Their container-based builds are pretty fast and can do a bunch for sure but not everything is doable there. For these other situations, the VM-based runtime is needed and being able to run both of them without changing a lot the configuration (or worse, imagine if I had to create a separate repository?) is pretty good.

If you want to know more, make sure you look at their official docs on build customization: docs.travis-ci: customizing the build.

I’ll be following up soon on how I publish this blog. As a heads up, I use Travis to build the static files and sync with an S3 bucket that serves the content to CloudFront. If you want to know more, make sure you subscribe to the mailing list 👌

Please let me know if I got something wrong.

Thanks!

finis

Why my Ubuntu container doesn't execute profile scripts?

Ciro S. Costa — Mon, 13 Nov 2017 00:00:00 +0000

tl;dr: you’re missing the --login flag in your bash execution.

Hey,

yesterday I was trying to test some ansible roles I had written and to do so I was using a containerized version of Ubuntu that I tailored for doing the job - cirocosta/ubuntu.

The image does its job pretty well: it can run with systemd as the PID 1, it has SSH and I can mess around with systemd services, units .. etc etc as if I had a working “VM” (given the set of boundaries of how far we can simulate a VM). Anyway … I faced an issue: my ansible roles were setting some scripts at /etc/profile.d/ but they were not being executed! How dare they?

It turns out that something pretty simple was going on - the default shell (bash in that case) wasn’t being executed as “login shell”.

Looking at bash(1)’s man page:

--noprofile
	Do not read either the system-wide startup file 
	/etc/profile or any of the personal initialization 
	files ~/.bash_profile, ~/.bash_login, or ~/.profile. 
	By default, bash reads these files when it is invoked 
	as a login shell (see INVOCATION below).

Invocation

	A login shell is one whose first character of argument 
	zero is a -, or one started with the --login option.

i.e, when we run bash <blabla> by default, it won’t run as a login shell and it turns out that profile files are only read when running it as login. Does it? We can easily check that with strace:

# create a script to be sourced by bash on startup
echo "export FOO=bar" > /etc/profile.d/my-script.sh

# create a simplistic exec file
echo "echo hey!" > ./exec.sh

# execute it without '--login' 
strace -e open bash exec.sh 
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
open("exec.sh", O_RDONLY)               = 3
hey!
+++ exited with 0 +++

# execute it with '--login' 
strace -e open bash --login exec.sh 
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
..
open("/dev/tty", O_RDWR|O_NONBLOCK)     = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
open("/etc/profile", O_RDONLY)          = 3
open("/etc/profile.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
open("/etc/profile.d/my-script.sh", O_RDONLY) = 3
open("exec.sh", O_RDONLY)               = 3
hey!
+++ exited with 0 +++

Closing thoughts

Do you need a full ubuntu container? Should you care about /etc/profile.d in a container? Probably not. Honestly, aside from testing ansible roles, I never needed.

It’s pretty cool how we can super easily verify a behavior making use of a combination of man to check the docs and strace to check the syscalls involved. Very handy.

If you have any questions or just noticed that I wrote something that’s totally wrong or that could be made better, reach out! I’m @cirowrc on Twitter. You can also subscribe to the newsletter to keep up with some stuff that I find very useful.

Cheers!

A Dockerfile tailored for Golang applications

Ciro S. Costa — Sun, 12 Nov 2017 00:00:00 +0000

Hey,

The first time I got involved with Docker was three and a half years ago while I was still an undergraduate student.

By that time, the only thing that used to matter to me in regards to Docker was the fact that I was able to run “a minimalistic VM” (which is almost an absurd way of referring to Docker or Linux containers in general).

Remember the time when the blogs were all about “how a container compares to a VM”?

Some years later (having dealt with Docker in production a lot), the most interesting piece for me (and others it seems) became the container image - this immutable thing that carries all of your dependencies - create a little recipe specifying the steps to reach the final desired state and ship that packed filesystem. A “Tarball++".

In regards to Golang, do we need much effort to create a working image? Totally not!

Here in this blog post, we go through:

The file structure of a common Golang project

For almost every Go project, I tend to always structure my Golang application in the same way:

.
├── .editorconfig
├── Dockerfile          # By placing the Dockerfile on the root
│                       # we can make it very simple - when executing
│                       # the `docker build` command from the root
│                       # the context of the build will carry all the
│                       # relevant Go files and dependencies (assuming 
│                       # you have a `vendor` directory at least).
├── Makefile
├── vendor
├── lib                 # A sample Golang package (you'd probably not call
│   ├── foo.go          # it `lib`, but something more meaningful).
│   ├── foo_test.go
│   ├── something.go
│   └── something_test.go
├── anotherpackage      # Another go package.
│   ├── bar.go
│   └── bar_test.go
├── main.go             # The `main` package.
└── VERSION

This way, I have to make minimal changes to this Dockerfile and the Makefile that I use (check the Makefile at Minimal Golang Makefile).

By keeping it simple like that we can build almost any kind of Docker images for all sorts of Golang projects without having to learn weird Makefile syntax or something complex for such a simple task.

An initial approach to a Golang Docker Image

If we take a multi-step approach to reach a very minimal Go Dockerfile, we can start with a path where we pack all the Golang source into it, build the code and then we’re done with it.

# Retrieve the `golang:alpine` image to provide us the 
# necessary Golang tooling for building Go binaries.
#
# Here I retrieve the `alpine`-based just for the 
# convenience of using a tiny image.
FROM golang:alpine

# Add the `main` file that is really the only Golang 
# file under the root directory that matters for the 
# build 
ADD ./main.go /go/src/github.com/cirocosta/l7/main.go

# Add all the files from the packages that I own
ADD ./lib /go/src/github.com/cirocosta/l7/lib

# Add vendor dependencies (committed or not)
# I typically commit the vendor dependencies as it
# makes the final build more reproducible and less
# dependant on dependency managers.
ADD ./vendor /go/src/github.com/cirocosta/l7/vendor

# 0.    Set some shell flags like `-e` to abort the 
#       execution in case of any failure (useful if we 
#       have many ';' commands) and also `-x` to print to 
#       stderr each command already expanded.
# 1.    Get into the directory with the golang source code
# 2.    Perform the go build with some flags to make our
#       build produce a static binary (CGO_ENABLED=0 and 
#       the `netgo` tag).
# 3.    copy the final binary to a suitable location that
#       is easy to reference in the next stage
RUN set -ex && \
  cd /go/src/github.com/cirocosta/l7 && \       
  CGO_ENABLED=0 go build \
        -tags netgo \
        -v -a \
        -ldflags '-extldflags "-static"' && \
  mv ./l7 /usr/bin/l7

# Set the binary as the entrypoint of the container
ENTRYPOINT [ "l7" ]

While this is an approach that indeed works, it has two drawbacks:

it makes you have to update the Dockerfile whenever a new package is created, or the filestructure changes, and
it ends up in an image that contains the entire Go compiler, Make and other tools that are not needed for the consumer of this image.

We can make it better.

Improving the Golang Dockerfile using multi-stage builds

One improvement that we can perform is making use of multi-stage builds.

This tackles the second drawback outlines in the last section.

The whole idea is that you can separate your build process into stages such that each stage marks an entirely new base image which can COPY files from previous stages.

FROM golang:alpine AS builder

ADD ./main.go /go/src/github.com/cirocosta/l7/main.go
ADD ./lib /go/src/github.com/cirocosta/l7/lib
ADD ./vendor /go/src/github.com/cirocosta/l7/vendor

RUN set -ex && \
  cd /go/src/github.com/cirocosta/l7 && \       
  CGO_ENABLED=0 go build \
        -tags netgo \
        -v -a \
        -ldflags '-extldflags "-static"' && \
  mv ./l7 /usr/bin/l7

# Create the second stage with the most basic that we need - a 
# busybox which contains some tiny utilities like `ls`, `cp`, 
# etc. When we do this we'll end up dropping any previous 
# stages (defined as `FROM <some_image> as <some_name>`) 
# allowing us to start with a fat build image and end up with 
# a very small runtime image. Another common option is using 
# `alpine` so that the end image also has a package manager.
FROM busybox

# Retrieve the binary from the previous stage
COPY --from=builder /usr/bin/l7 /usr/local/bin/l7

# Set the binary as the entrypoint of the container
ENTRYPOINT [ "l7" ]

Just by adding this extra stage you can see the differences in practice.

At least the size of the Go toolchain will be reduced (that’s at least 30MB).

ps.: it’s not everytime that 30MB makes a difference. I’m sure that early optimization is a problem, but this is such a straightforward optimization that everyone can take it without much thought.

The minimal Golang Dockerfile

The final optimization missing is reducing those lines that add specific directories to the builder stage.

While that’s something useful to be done when you have RUN steps in between (it can let you cache directories and avoid re-running build steps all the time), in this case, it’s not very useful if you only have a handful of dependencies and no big files in the repository.

To have an even simple Dockerfile, we can then replace those adds by a wildcard ADD ./ that simply puts everything we have into the image build context:

FROM golang:alpine AS builder

# Add all the source code (except what's ignored
# under `.dockerignore`) to the build context.
ADD ./ /go/src/github.com/cirocosta/l7/

RUN set -ex && \
  cd /go/src/github.com/cirocosta/l7 && \       
  CGO_ENABLED=0 go build \
        -tags netgo \
        -v -a \
        -ldflags '-extldflags "-static"' && \
  mv ./l7 /usr/bin/l7

FROM busybox

# Retrieve the binary from the previous stage
COPY --from=builder /usr/bin/l7 /usr/local/bin/l7

# Set the binary as the entrypoint of the container
ENTRYPOINT [ "l7" ]

Some remarks

You might want to end up with a FROM alpine instead of FROM busybox and then run apk add --update ca-certificates if your binary needs to perform requests to HTTPS endpoints - just using busybox will lead to an image that doesn’t contain root CA certificates which would make HTTPS requests fail.

If you’re interested in learning how to deal with Docker, Golang or just wanting to learn more about software engineering in general, make sure you subscribe to the mailing list!

In case of questions or if you spot something odd, please get in touch with me at cirowrc on Twitter.

Have a good one!

finis

LVM on loopback devices

Ciro S. Costa — Sat, 11 Nov 2017 00:00:00 +0000

Hey,

I remember back when I was introduced to Arch Linux this thing called LVM but I never really used it and knew what were its capabilities.

I simply couldn’t understand why someone would want to resize a partition on the fly (guess what, I’ve done that so many times in AWS nowadays). Why would someone want to snapshot? Why LVM?

Now many years after my first Arch Linux installation I understand what are its benefits but I’ve never really set up on a Linux machine myself though. Here in this article, I go through the process of how one can play around with LVM without needing to have an extra disk or format anything.

ps.: If you’re looking for high-level concepts of LVM, this is not really the place you’re looking for. This is just hands-on on how to get it running with a loopback device (/dev/loop*).

ps.: in all of the code below the “hashtags” before the command (#) indicates that we’re executing it with a privileged user.

ps.: this is an area I’m not expert at all! Please be kind of pointing to me the places I got stuff wrong and I’ll update right away!

Getting real

First, let’s verify that we have some loopback devices

# ls /dev | grep loop
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
loop-control

Now create a file to hold the data that will be written to the first loopback device. This is important because we’re wanting to simulate a real disk so we have to reserve some space for it.

# dd if=/dev/zero of=./lvm0.img bs=50 count=1M
1048576+0 records in
1048576+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 1.06171 s, 49.4 MB/s

The next step is linking the loopback devices with the file we just created. This is important because the file itself is just another regular file in the OS. LVM can’t get set up on a regular file though - it demands a special block device (like those you have under /dev).

# losetup /dev/loop0 ./lvm0.img

Verify that the loopback device has been recognized as an actual disk

# fdisk -l
Disk /dev/loop0: 50 MiB, 52428800 bytes, 102400 sectors <<<<<<
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x14f750bf

# lsblk
lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0    7:0    0  50M  0 loop          <<<<<<< (not mounted yet)
sda      8:0    0  10G  0 disk 
└─sda1   8:1    0  10G  0 part /
sdb      8:16   0  10M  0 disk

Check all the partitions that we have in our disks - we should see 0 partitions in /dev/loop0. For doing so we can use GNU parted (parted command in ubuntu zesty).

# man parted

NAME
       GNU Parted - a partition manipulation program

SYNOPSIS
       parted [options] [device [command [options...]...]]

...
      -l, --list
              lists partition layout on all block devices
...

Now, let’s execute it:

# parted -l

Model: VBOX HARDDISK (scsi)
Disk /dev/sda: 10.7GB                           <<<< NOT THE LOOPBACK
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  10.7GB  10.7GB  primary  ext4         boot

So now what we need to do is format the device and set it to use LVM. For doing so we can make use of fdisk. As we don’t want to go through the process using the interactive section we can make use of sfdisk.

From man sfdisk you can see that it’s a “script-oriented tool for partitioning any block device”. Here we only have to submit the same inputs we’d use with fdisk but in the stdin of the executable:

# echo ",,8e,," | sfdisk /dev/loop0
Checking that no-one is using this disk right now ... OK

Disk /dev/loop0: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Created a new DOS disklabel with disk identifier 0x9f5773d1.
/dev/loop0p1: Created a new partition 1 of type 'Linux LVM' and of size 49 MiB.
/dev/loop0p2: Done.

New situation:

Device       Boot Start    End Sectors Size Id Type
/dev/loop0p1       2048 102399  100352  49M 8e Linux LVM

The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
Syncing disks.

What does ,,8e,, means? It’s the same as “create a partition starting at the default start, with the default size, being LVM and using the default value for the bootable property”.

We can understand that better by looking at man sfdisk:

# man sfdisk.8
...
       Unnamed-fields format

                     start size type bootable

LIKE WHAT WE DID
           >>>>>         ,     , 8e ,       ,

              where each line fills one partition descriptor.

You can check that 8e is the type of LVM by using fdisk manually, typing m to look for help, and then l to list known partition types:

#fdisk

press m
press l

Command (m for help): l

 ...
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility   
                                        ++++++++++++++
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM|      df  BootIt         
                                        /\++++++++++++
                                        || 
                                        the 8e we used!
 ...

Cool, we have our /dev/loop0 device partitioned with LVM. Using the same commands we used before we can make sure that the statement is true:

# fdisk -l

fdisk -l
Disk /dev/loop0: 50 MiB, 52428800 bytes, 102400 sectors
...

Device       Boot Start    End Sectors Size Id Type
/dev/loop0p1       2048 102399  100352  49M 8e Linux LVM

But if you go back to the output of the sfdisk command you can see there was an error:

The kernel still uses the old table. 
The new table will be used at the next reboot or 
after you run partprobe(8) or kpartx(8).

We can confirm that indeed there’s something wrong if we make use of a command that relies on the mentioned table.

We can be faced with such problem by using our first LVM command: lvmdiskscan

# lvmdiskscan 
  /dev/loop0 [      50.00 MiB] 
  /dev/sda1  [      10.00 GiB] 
  /dev/sdb   [      10.00 MiB] 
  1 disk
  2 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

What? We just created an LVM partition on a disk (on the loopback device) and it says that there’s no LVM devices? That’s the mentioned problem in place.

To fix it we can make use of partx to force an update in the kernel table:

# partx --update /dev/loop0

and then see the new output:

# lvmdiskscan

  /dev/loop0p1 [      49.00 MiB] 
  /dev/loop0   [      50.00 MiB] 
  /dev/sda1    [      10.00 GiB] 
  /dev/sdb     [      10.00 MiB] 
  1 disk
  3 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

But we still don’t have a physical volume that LVM can use.

# lvmdiskscan -l
  WARNING: only considering LVM devices
  0 LVM physical volume whole disks
  0 LVM physical volumes

To create that we now make use of pvcreate:

PVCREATE(8)                     System Manager's Manual                    PVCREATE(8)

NAME
       pvcreate — initialize a disk or partition for use by LVM

...

Examples
       Initialize  partition  #4 on the third SCSI disk and the entire fifth SCSI disk
       for later use by LVM:

       pvcreate /dev/sdc4 /dev/sde

So that’s cool, we can just follow the example provided and do ourselves:

# lvmdiskscan -l
  WARNING: only considering LVM devices
  0 LVM physical volume whole disks
  0 LVM physical volumes

# pvcreate /dev/loop0
WARNING: dos signature detected on /dev/loop0 at offset 510. Wipe it? [y/n]: y
  Wiping dos signature on /dev/loop0.
  Physical volume "/dev/loop0" successfully created.

# lvmdiskscan  -l
  WARNING: only considering LVM devices
  /dev/loop0   [      50.00 MiB] LVM physical volume
  0 LVM physical volume whole disks
  1 LVM physical volume

So now we have our first physical volume. Next step is creating a volume group:

VGCREATE(8)                     System Manager's Manual                    VGCREATE(8)

NAME
       vgcreate — create a volume group

SYNOPSIS
       vgcreate [opts...] VolumeGroupName PhysicalDevicePath

DESCRIPTION
       vgcreate creates a new volume group called VolumeGroupName using the block spe‐
       cial device PhysicalDevicePath.


# vgcreate myvg /dev/loop0
  Volume group "myvg" successfully created

With vgdisplay we can then check that everything went fine:

# vgdisplay 
  --- Volume group ---
  VG Name               myvg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  ...
  VG Size               48.00 MiB
  ...

Now that we have a volume group we can create a logical volume using a desired amount of size from the volume group:

# lvcreate --size 10M --name lv1 myvg
 Rounding up size to full physical extent 12.00 MiB
  Logical volume "lv1" created.

Having the logical volume created we can inspect the before and after of vgdisplay:

--- initial	2017-11-11 18:54:11.000000000 -0200
+++ after	2017-11-11 18:54:23.000000000 -0200
@@ -3,11 +3,11 @@
   System ID             
   Format                lvm2
   Metadata Areas        1
-  Metadata Sequence No  1
+  Metadata Sequence No  2
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
-  Cur LV                0
+  Cur LV                1
   Open LV               0
   Max PV                0
   Cur PV                1
@@ -15,6 +15,6 @@
   VG Size               48.00 MiB
   PE Size               4.00 MiB
   Total PE              12
-  Alloc PE / Size       0 / 0   
-  Free  PE / Size       12 / 48.00 MiB
+  Alloc PE / Size       3 / 12.00 MiB
+  Free  PE / Size       9 / 36.00 MiB
   VG UUID               TmGsRi-3SZg-kU9Q-FTOI-72rG-XaYB-xkxXYB

Clearly, we can see that we took some space from the pool of all available space of the volume group (as it was allocated to the logical volume).

To finish, just see that our logical volume is not listed under lsblk:

# lsblk
NAME       MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0        7:0    0  50M  0 loop 
├─myvg-lv1 253:0    0  12M  0 lvm  
└─loop0p1  259:0    0  49M  0 loop 
sda          8:0    0  10G  0 disk 
└─sda1       8:1    0  10G  0 part /
sdb          8:16   0  10M  0 disk

Having that we’re set to use the partition for anything we want - mounting XFS, EXT4 .. any filesystem you want there and have all the benefits that LVM gives us (which is not the focus of this article).

Closing thoughts

Compared to just set up up a block device with a different filesystem, setting up LVM takes a big number of steps. Besides this, it’s pretty cool to see that because the interface (a special block device) is so simple we can totally skip the need for a real device and work in the same manner. Once meet the needs of the interface we’re done. That’s pretty cool IMHO.

What I plan to do next is script that process and set up multiple loopback devices so that I can experiment with the concept of having multiple volumegroups together with docker volumes.

Let’s see if that works out well 🙌.

If you’re struggling with a concept related to this or do something related to your work, your side project .. make sure you subscribe to the newsletter - you’ll be interested in other content similar to this.

Have a good one!

Resources

I used two resources (beside man pages) that really helped me along the way. I want to thank these two for the great resources they provided:

Anothony’s blog: LVM Loopbackv HOW-TO - this is what I used the most to write the article. It goes through almost the same as I do here but I think I explained more some pieces that he just passed by.
Digital Ocean: How to use LVM To Manage storage devices on Ubuntu 16.04: another pretty great article that covers the process of using LVM on Ubuntu but not really focused on loopback devices.

finis

Minimal Golang Makefile

Ciro S. Costa — Sat, 11 Nov 2017 00:00:00 +0000

Hey,

Most of the time I see myself creating a Golang project: be it a one-file single-purpose thing or a more complex project.

To facilitate that process, I try always to keep the structure the same such that there’s no need to put much thought into the process of bootstrapping a new project - for me, the less friction to get started, the better.

The file structure usually looks like this:

.
├── .editorconfig       # Allows you to define and maintain 
│                       # a consistent editor configuration
│                       # for your project.
│ 
├── Dockerfile          # Define the steps to build a Docker
│                       # image that contains just the software
│                       # I'm building and nothing more.
│                        
│                       # The benefits that I see here is that 
│                       # it makes explicit for anyone what are
│                       # the dependencies. It also provides a
│                       # reproducible way for people to build
│                       # your software without worrying about
│                       # external dependencies, like, the 
│                       # compiler toolchain or some shared 
│                       # libraries.
│                       
├── Makefile    
├── lib                 # A package that we might use in `main`
│   ├── foo.go
│   ├── foo_test.go
│   ├── something.go
│   └── something_test.go
├── main.go             # `main` package.
└── VERSION

Some things to pay attention:

 [`editorconfig`](https://editorconfig.org/) is a real deal

I use vim with set shiftwidth=2; set tabstop=2 set by default, meaning that I always have two spaces set for identation.

Go, on the other hand, is all about tabs.

The standard formatting tool doesn’t even allow you to use spaces - see cmd/gofmt: remove -tabs and -tabwidth flags.

 People tend to expect a [Dockerfile](https://docs.docker.com/engine/reference/builder/) nowadays

And I get it!

Because with a standard Docker builder we’re able to do the equivalent of vendoring the whole filesystem, it allows the developer to set the versions for all the toolchain around building the code, and that’s very convenient for someone who’s wanting to consume the software from your repository.

Now, let’s get to the point - the Golang Makefile.

The (annotated) minimal makefile for Go

In this case, I extracted the Makefile from a test project I started some time ago (a “load-balancer” that makes use of fasthttp: cirocosta/l7).

If you’ve never used make before (or if you don’t remember the syntax), there’s this awesome Gist from Isaacs.

ps.: the snippet below is scrollable - if you’re seeing the text cut, perform a horizontal scroll.

# I usually keep a `VERSION` file in the root so that anyone
# can clearly check what's the VERSION of `master` or any
# branch at any time by checking the `VERSION` in that git
# revision.
#
# Another benefit is that we can pass this file to our Docker 
# build context and have the version set in the binary that ends
# up inside the Docker image too.
VERSION         :=      $(shell cat ./VERSION)
IMAGE_NAME      :=      cirocosta/l7


# As a call to `make` without any arguments leads to the execution
# of the first target found I really prefer to make sure that this
# first one is a non-destructive one that does the most simple 
# desired installation. 
#
# It's very common to people set it as `all` but it could be anything 
# like `a`.
all: install


# Install just performs a normal `go install` which builds the source
# files from the package at `./` (I like to keep a `main.go` in the root
# that imports other subpackages). 
#
# As I always commit `vendor` to `git`, a `go install` will typically 
# always work - except if there's an OS limitation in the build flags 
# (e.g, a linux-only project).
install:
	go install -v


# Keeping `./main.go` with just a `cli` and `./lib/*.go` with actual 
# logic, `tests` usually reside under `./lib` (or some other subdirectories).
#
# By using the `./...` notation, all the non-vendor packages are going
# to be tested if they have test files.
test:
	go test ./... -v


# Just like `test`, formatting what matters. As `main.go` is in the root,
# `go fmt` the root package. Then just `cd` to what matters to you (`vendor`
# doesn't matter).
#
# By using the `./...` notation, all the non-vendor packages are going
# to be formatted (including test files).
fmt:
        go fmt ./... -v


# This target is only useful if you plan to also create a Docker image at
# the end. 
#
# I really like publishing a Docker image together with the GitHub release
# because Docker makes it very simple to someone run your binary without
# having to worry about the retrieval of the binary and execution of it
# - docker already provides the necessary boundaries.
image:
	docker build -t cirocosta/l7 .


# This is pretty much an optional thing that I tend always to include.
#
# Goreleaser is a tool that allows anyone to integrate a binary releasing
# process to their pipelines. 
#
# Here in this target With just a simple `make release` you can have a 
# `tag` created in GitHub with multiple builds if you wish. 
#
# See more at `gorelease` github repo.
release:
	git tag -a $(VERSION) -m "Release" || true
	git push origin $(VERSION)
	goreleaser --rm-dist

.PHONY: install test fmt release

Save that content in the Makefile file in root directory of the project, create a VERSION file with something like 0.0.1 (semver) and you’re ready to go.

Naturally, vendor directories and all of that is already covered.

Projects with many packages and vendor dependencies

In bigger projects, you might have a bunch of packages and they’re not as flat as I described here.

Regardless of how big it is, given that in our targets we’ve been specifying to the tools that the paths should be recursively traversed (./...), we don’t need to keep updating the Makefile.

Although this one I supplied is minimal, it stays minimal even if your project grows.

It might happen though, that you want to include an extra tool that requires exact paths to directories where Golang files live. Maybe your static analyser doesn’t have the equivalent of ./....

In that case, we can still keep our Makefile minimal.

One way is adapting the Makefile to handle both nesting and multiple packages “intelligently” by finding them ahead of time and then suppliying these to your targets.

First things first, we can combine find and xargs utilities to gather the directories:

GO_SRC_DIRS := $(shell \
	find . -name "*.go" -not -path "./vendor/*" | \
	xargs -I {} dirname {}  | \
	uniq)

GO_TEST_DIRS := $(shell \
	find . -name "*_test.go" -not -path "./vendor/*" | \
	xargs -I {} dirname {}  | \
	uniq)

# Shows the variables we just set.
# By prepending `@` we prevent Make
# from printing the command before the
# stdout of the execution.
show:
	@echo "SRC  = $(GO_SRC_DIRS)"
	@echo "TEST = $(GO_TEST_DIRS)"

Now, testing it:

make show

SRC  = ./lib ./lib/sub ./lib .
TEST = ./lib

With those lists as variables, we can make use of them as dependencies for our new fmt and test targets:

test: $(GO_TEST_DIRS)
	@for dir in $^; do \
		pushd ./$$dir > /dev/null ; \
		go test -v ; \
		popd > /dev/null ; \
	done;

fmt: $(GO_SRC_DIRS)
	@for dir in $^; do \
		pushd ./$$dir > /dev/null ; \
		go fmt ; \
		popd > /dev/null ; \
	done;

As $^ targets all the dependencies, on both targets we’re able to use the $ˆ selector to iterate over the directories of each.

Note that I’m not using cd to get into the directories. That’s because not knowing how deep in the file structure we’ll go, just stacking the directory changes with pushd is easier as to get back to the original place we just need to popd.

Closing thoughts

I think it’s very handy to keep a standard way of performing basic operations across multiple repositories. In my experience this reduces the friction of moving from one project to another. Having a common flow of how to build, create an image and publish a project using a Makefile has helped me in such area.

Do you think the same? What are your thoughts?

Reach me on Twitter at any time @cirowrc and subscribe to the list if you’re a Golang developer or simply likes software stuff!

finis

How to install HAProxy with Lua support on MacOS

Ciro S. Costa — Fri, 27 Oct 2017 00:00:00 +0000

Hey,

Since 2016, a great Lua script has been used by many people deploying HAProxy instances that need to allow LetsEncrypt certificates generation: haproxy-acme-validation-plugin.

I’ve even written about how to respond to HTTP requests right from HAProxy: Making HAProxy respond 200 OK to health checks.

Many other scripts that extend HAProxy’s functionality have been made, but that’s not the point of this post.

If you ever need (or want) to test this script in a MacOS machine (or any other Lua script that enhances HAProxy), you’d need a special build of it: one that comes with Lua support.

Homebrew users have an easy time with this - since this commit, an extra option has been added: --lua, making the how installation a breeze:

# Install the `HAPROXY` brew formula with
# the extra `lua` option.
brew install haproxy \
        --with-lua

However, what if you want to install the very latest HAProxy version from source?

Installing HAProxy from source with Lua support

To install HAProxy from source on a Mac, we need to follow some steps:

install HAProxy dependencies (you can discover these using brew info haproxy if you have brew);
gather the source code from the official website;
“untar” it; and
compile the code using a set of flags that will allow us to build with the proper Lua support.

Aiming at Lua 5.3, it’s required that you first install it (you can use Homebrew for this):

brew install lua@5.3

To properly satisfy the first step though, it’s also important that you have the other dependencies as well:

# Discover what are the dependencies
# that have been set for HAProxy
brew info haproxy
==> Dependencies
Required: openssl ✔, pcre ✔
Optional: lua ✘

# Install the dependencies
brew install openssl
brew install pcre

Having that done, proceed with the download of the version you want (at this time, 1.9-dev0 is the very latest release):

# Set the version that we want to get the source
VERSION=1.9-dev0

# Grab the haproxy source from their website.
#
# Note that differently from the stable releases,
# the development version sits under `devel`.
#
# Make sure you properly modify the URL when you
# use the stable versions.
wget http://www.haproxy.org/download/1.9/src/devel/haproxy-$VERSION.tar.gz
tar xzf haproxy-$VERSION.tar.gz
cd haproxy-$VERSION

Being in the source directory of the HAProxy version, make sure that there’s no --export-dynamic property set in the LUA_LD_FLAGS line of the Makefile.

Such flag is not available in the MacOS linker, so, with it, your build will fail.

diff --git a/Makefile b/Makefile
index 26b55db..d02c858 100644
--- a/Makefile
+++ b/Makefile
@@ -629,7 +629,7 @@ check_lua_inc = $(shell if [ -d $(2)$(1) ]; then echo $(2)$(1); fi;)
 
 BUILD_OPTIONS   += $(call ignore_implicit,USE_LUA)
 OPTIONS_CFLAGS  += -DUSE_LUA $(if $(LUA_INC),-I$(LUA_INC))
-LUA_LD_FLAGS := -Wl,--export-dynamic $(if $(LUA_LIB),-L$(LUA_LIB))
+LUA_LD_FLAGS := -Wl $(if $(LUA_LIB),-L$(LUA_LIB))
 ifeq ($(LUA_LIB_NAME),)
 # Try to automatically detect the Lua library
 LUA_LIB_NAME := $(firstword $(foreach lib,lua5.3 lua53 lua,$(call check_lua_lib,$(lib),$(LUA_LD_FLAGS))))

With the Makefile fixed, proceed with the compilation:

# build the source code.
# for doing this you must at least have a recent C compiler

make -j6 \
TARGET=osx \
USE_KQUEUE=1 \
USE_POLL=1 \
USE_PCRE=1 \
USE_THREAD=1 \
USE_OPENSSL=1 \
USE_ZLIB=1 \
USE_LUA=1 \
LUA_LIB_NAME=lua \
LUA_LIB=/usr/local/lib/ \
LUA_INC=/usr/local/include \
SSL_LIB=/usr/local/opt/openssl/lib \
SSL_INC=/usr/local/opt/openssl/include \
ADDLIB=-lcrypto

# Check if everything was correctly installed.
./haproxy -vvvv
HA-Proxy version 1.9-dev0-b306650 2017/11/26
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>


..
  OPTIONS = USE_ZLIB=1 USE_POLL=1 USE_KQUEUE=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
...
Built with Lua version : Lua 5.3.4
...

# link it to `/usr/local/bin/haproxy` (which is in my $PATH) so I
# can access it directly from `cli`
ln -s $(realpath haproxy) /usr/local/bin/haproxy

That’s it!

Now you should have the binary ready to be used.

If you want to know how to get TLS certificates with LetsEncrypt and HAProxy, make sure you check a blog post I wrote about the topic: Getting TLS certificates with Letsencrypt and HAProxy.

In case you have any questions, please let me know! I’m @cirowrc on Twitter and would love your feedback.

Also, make sure you subscribe to the mailing list if you like the topic!

Have a good one!

finis

HAProxy Docker Container Logs

Ciro S. Costa — Fri, 20 Oct 2017 00:00:00 +0000

If you simply take the official HAProxy docker image, you’ll quickly see that your logs will not show.

That’s because by default, HAProxy won’t log to stdout - you need to have a facility (rsyslog) that will take those logs and ship it to somewhere.

A Dockerfile with RSYSLOG for HAProxy logging

Although adding rsyslog is straightforward with the alpine-based image, we can go further with linking the generated haproxy.log from rsyslog to /dev/stdout such that whenever rsyslog writes HAProxy logs, it goes directly to stdout (such that any log aggregation driver from Docker can pick).

Given that we need to have some configuration files and a custom entrypoint, I created the following file structure:

.
├── Dockerfile          # image description
├── entrypoint.sh       # executable script to launch both
|                       # haproxy and rsyslog.
└── etc                 # configuration files
    ├── haproxy.cfg     # haproxy config
    └── rsyslog.conf    # rsyslog config

1 directory, 5 files

The Dockerfile doesn’t do much:

starts from the official HAProxy image;
adds rsyslog;
creates the file that rsyslog will log contents to;
links the haproxy log with the stdout device.

It looks like this:

FROM haproxy:1.8-alpine

RUN set -x	&& \
	apk upgrade --update					&&  \
	apk add bash ca-certificates rsyslog	                &&  \

	mkdir -p /etc/rsyslog.d/				&&  \

	touch /var/log/haproxy.log				&&  \

        # here's the catch: by creating a soft-link that 
        # links /var/log/haproxy.log to /dev/stdout whatever 
        # rsyslogd writes to the file will endup being
        # propagated to the standard output of the container
	ln -sf /dev/stdout /var/log/haproxy.log

# Include our configurations (`./etc` contains the files that we'd
# need to have in the `/etc` of the container).
ADD ./etc/ /etc/

# Include our custom entrypoint that will the the job of lifting
# rsyslog alongside haproxy.
ADD ./entrypoint.sh /usr/local/bin/entrypoint

# Set our custom entrypoint as the image's default entrypoint
ENTRYPOINT [ "entrypoint" ]

# Make haproxy use the default configuration file
CMD [ "-f", "/etc/haproxy.cfg" ]

Now looking at the configuration files, we have two: the rsyslog configuration and the haproxy configuration.

The rsyslog is tailored with a facility (local0) to be used solely by haproxy.

# Loads the imudp into rsyslog address space
# and activates it.
# IMUDP provides the ability to receive syslog
# messages via UDP.
$ModLoad imudp

# Address to listen for syslog messages to be 
# received.
$UDPServerAddress 0.0.0.0

# Port to listen for the messages
$UDPServerRun 514

# Take the messages of any priority sent to the
# local0 facility (which we reference in the haproxy
# configuration) and send to the haproxy.log 
# file.
local0.* -/var/log/haproxy.log

# Discard the rest
& ~

The HAProxy configuration then, with local0 as the target for the logs:

# Setting `log` here with the address of 127.0.0.1 will have the effect
# of haproxy sending the udp log messages to its own rsyslog instance
# (which sits at `127.0.0.1`) at the `local0` facility including all
# logs that have a priority greater or equal than debug
global
                maxconn                   2046
                log                       127.0.0.1       local0  debug


# By default we want to use the same logging parameters as defined
# in the global section.
defaults
                log                       global


# Simple frontend that will take some HTTP requests from port :80
# and then always pick the `backend_default` default backend.
#
# Naturally, this configuration you'd replace by whatever makes more
# sense to your application.
frontend        http
                bind                      :80
                default_backend           backend_default

# A non-existent backend that would never return - again, this would
# be replaced by something that makes sense to your application, maybe
# something that gets generated whenever a new container goes up or
# it could be picked via DNS (as HAProxy now supports SRV records).
backend         backend_default
                server                    local-server 127.0.0.1:8080

We can then raise those two components (HAProxy and RSYSLOG) by making the entrypoint do so:

#!/bin/bash

set -o errexit
set -o nounset

readonly RSYSLOG_PID="/var/run/rsyslogd.pid"

main() {
  start_rsyslogd
  start_lb "$@"
}

# make sure we have rsyslogd's pid file not
# created before
start_rsyslogd() {
  rm -f $RSYSLOG_PID
  rsyslogd -n
}

# Starts the load-balancer (haproxy) with 
# whatever arguments we pass to it ("$@")
start_lb() {
  exec haproxy "$@"
}

main "$@"

Create the image and you’re good to go!

Closing thoughts

Redirecting all the logs from rsyslog to the standard out device makes haproxy logs play nice with docker default logging. It also has the upside of allowing us to not be concerned about log rotation from within the load-balancer container - either the log aggregation platform that takes the HAProxy logs would take care of it or even the standard json-file driver would do the log rotation.

This configuration is fairly simplistic - it doesn’t account the eventual failure of rsyslog that might occur. In that case, I’d say the best would be to kill the container and let it be recreated (although I never saw rsyslog diying like that).

Please let me know if you have any questions or if you have any suggestions.

I’m @cirowrc and would appreciate!

Have a good one!

finis

Augmenting Linux Swap Space

Ciro S. Costa — Sun, 24 Sep 2017 00:00:00 +0000

Hey,

the first time I had a machine provisioned using a set of scripts I noticed that I had a hardcoded value for the amount of swap space that it could handle.

Damn! All my machines are using the same amount of swap! Can I fix that?

It turns out that increasing the swap space is pretty straightforward.

tl;dr: just create an extra file and activate it - Linux will add it to the count of available swap space.

If you’ve ever needed to extend the amount of swap space that you have in your Linux machine, be it Ubuntu, CentOS or any other distribution, this blog post is for you.

Simulating

To simulate the action, we can follow some steps:

Prepare a virtual machine where we can mess with swap space.
Set up a single swap area and activate it
Expand the total amount of swap by creating an extra file, setting it as a swap area and enabling it.

Once the simulation is done, we can verify that we can end up with a system that has more swap space than before.

1. Preparing a VM

The easiest way of creating a VM that I know is making use of the excellent Vagrant.

It allows you to prepare a ruby file that tells it how to create a set of machines given a spec that you define.

What I like the most of it is that even though it’s ruby (and not something like yaml) it’s very declarative (I know nothing about ruby and can go with it pretty well).

Having vagrant properly installed we need to create a Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# run this little script that changes
# the password of the ubuntu user after
# the machine gets booted up
$script = <<SCRIPT
echo "ubuntu:admin" | chpasswd
SCRIPT

Vagrant.configure(2) do |config|
        # set the image to be used to be ubuntu/zesty64
        config.vm.box = "ubuntu/zesty64"
        # set the hostname of the machine
        config.vm.hostname = "test-machine"
        # do not check for base image updates
        config.vm.box_check_update = false
        # do not sync the default vagrant directory
        config.vm.synced_folder ".", "/vagrant", disabled: true
        # provision the machine with a custom script
        config.vm.provision "shell", inline: $script
        # configure some parameters from the virtualbox provider
        config.vm.provider "virtualbox" do |v|
                v.memory = 512
                v.cpus = 1
        end
end

Running vagrant up gives us a virtual machine.

vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/zesty64'...
==> default: Matching MAC address for NAT networking...
...
    default: 22 (guest) => 2200 (host) (adapter 1)
...
==> default: Setting hostname...
==> default: Running provisioner: shell...
    default: Running: inline script

Now we can SSH into the machine at any time:

vagrant ssh default

# or

ssh ubuntu@localhost -p 2200

# password is `admin` as set by the provisioning
# script that we defined in the Vagrantfile

Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-26-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 ...

ubuntu@test-machine:~$

Now that we have a VM running we can give it some swap space. All the following commands are meant to be executed inside the VM.

2. Activating some wrong swap space

The first thing to do is allocate some space on a disk that Linux can refer to when swapping. Here we can choose two approaches: fallocate or dd.

fallocate is used to manipulate the allocated disk space for a file, either to deallocate or preallocate it. For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks.

This is much faster than creating a file by filling it with zeroes as one would do with dd. You can check it with the following snippet:

swap_file=/mnt/1g.swap
time dd if=/dev/zero of=$swap_file bs=1M count=1024

1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.51977 s, 707 MB/s

real	0m1.534s
user	0m0.008s
sys	0m0.756s

Now comparing to the fallocate command:

rm -rf $swap_file
time fallocate --length 1g $swap_file;

real	0m0.022s
user	0m0.000s
sys	0m0.000s

ps.: this amount of time might not matter much if we’re allocating just a small amount like 1g and it’s not at instance-startup time.

However, having something like 64G could take some unnecessary time from the instance startup.

After space on disk has been reserved for swap we need now to make it so that Linux understands that file as a linux swap area designed for such purpose - mkswap is the command to do the job.

mkswap $swap_file

mkswap: /mnt/1g.swap: insecure permissions 0644, 0600 suggested.
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=0ef91bd0-3691-4dde-848c-44699289d733

Once the operation succeeds, we need to activate the swap area. swapon is the command that does so. You just need to specify the file and Linux will keep a reference to it to swap in when needed.

free -h
              total        used        free      shared  buff/cache   available
Mem:           486M         62M        361M        1.8M         62M        406M
Swap:            0B          0B          0B
                /\
                || NO SWAP

swapon $swap_file

free -h
              total        used        free      shared  buff/cache   available
Mem:           486M         62M        360M        1.8M         62M        406M
Swap:          1.0G          0B        1.0G
                /\
                || SWAP ACTIVATED

Ok, we got some swap space! Now, can we make it bigger?

3. Extending it

Sure, making it bigger is just a matter of adding an extra file where we can submit swap memory to it and making sure that Linux keeps track of it.

Just execute the same commands as before but targetting a new file:

swap_file=/mnt/2g.swap
dd if=/dev/zero of=$swap_file bs=1M count=2048
mkswap $swap_file
swapon $swap_file

After all, we should now have 3G of swap space:

free -h
              total        used        free      shared  buff/cache   available
Mem:           486M         62M        7.1M        1.8M        416M        406M
Swap:          3.0G          0B        3.0G

That’s it!

Gotchas

Q: Can you allocate as much space as you want?

No, there’s an amount in the number of pages that Linux will allow to be addressed by swap area headers.

Q: Can you create as many swap areas as you want?

No, there’s a limit on that amount as well and each allocated swap area can be found in /proc/swaps.

Closing thoughts

This was one of those simple little things that you learn by doing and really has no secrets.

I already knew how to increase swap because I had previously hardcoded things before and wanted to change later but this one I also faced in the middle of a call when a machine was literally close to getting OOM triggered because all the RAM had been taken and swap space was almost done.

To have just a little bit more of time to investigate the leak I just had to increase the swap space and then let it crash after the investigation. These steps did the job.

By the way, if you want to know more about Linux in general, some tips regarding on-call stuff, operations, and software engineering you should definitely subscribe to the newsletter - I occasionally send some links and content that I’d really like to receive.

If you have any questions or just want to chat a little bit, let me know! I’m @cirowrc on Twitter.

Have a good one!

finis

ops.tips

A Raspberry PI Concourse Worker

Concourse workers

Compiling the Concourse binary

Cross compiling Go code

Compiling CGO code

Cross compiling CGO code

Building Guardian, the containerizer

Building Baggageclaim, the volumizer

Running the Concourse worker

Generating the modified registry-image-resource

Automating the process of building an ARM-based Concourse distribution

Summarizing

Some surprises

Closing Thoughts

How Linux creates sockets and counts them

What are these sockets about?

Where to look for the list of sockets in my system?

What happens under the hood when the socket syscall gets called?

Sockets and resource limits

Counting the number of sockets in the system

What about namespaces?

Closing thoughts

Resources

Using /proc to get a process' current stack trace

A process that blocks

Viewing the process kernel stack trace

When the stack does not help much

Closing thoughts

Process resource limits under the hood

What are resource limits

How does the kernel limit the number of files open by a process?

Under the hood of the ulimit “command”

A sample ulimit implementation

Setting and getting resource limits at the Kernel level

Gathering process limits via /proc

Why expose limits through procfs if prlimit already exists

Closing thoughts

Resources

Why top and free inside containers don't show the correct container memory

Running top within a container

How the top and free tools gather memory statistics

Setting container limits

Memory limits set by cgroups are not namespaced

Who’s controlling the allocation of memory?

Tracing a cgroup running out of memory

Closing thoughts

Resources

How is /proc able to list process IDs?

Listing directory entries in Linux

Opening a directory

Reading directory entries from userspace

Under the hood of getdents

How procfs handles getdents calls

Linux and its pids

How procfs lists process IDs

Closing thoughts

Resources

What is /proc?

What is procfs

Contrasting Procfs with a regular filesystem

Reading from and writing to procfs

Translating a file read to an internal kernel method

Getting /proc in your tree

Procfs after VFS

Closing thoughts

Resources

A month of /proc

The articles so far

A bpftrace Ansible role

The bpftrace ansible role

Article recommendation using Hugo

The idea

Implementing a Recommended Articles list in Hugo

Displaying the recommendation list

Closing thoughts

A UDP server and client in Go

Overview

Sending UDP packets using Go

Address resolution

Getting `/proc` in your tree

What’s behind `tail`

Example: consuming from `stdin` using Go