Adrian Todorov

GPU Sharing Done Right: Secrets, Security, and Scaling

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sun, 15 Mar 2026 13:21:59 +0200

Presentation delivered at the Southern California Linux Expo (SCALE), 23rd Edition in March 2026.

Multi-tenant Kubernetes with GPU sharing is a compelling model for AI infrastructure, but it requires careful design to balance performance with security. This session shows how to build a secure and scalable environment where multiple teams can run GPU workloads without compromising isolation or access control. We’ll cover open source options like KAI Scheduler and vCluster and demonstrate how to integrate external tools for secret management and dynamic access policies, all within an architecture that lets teams feel like they have their own cluster—while behind the scenes, resources are efficiently pooled and shared.

Links

Slide Deck

Dynamic Identities and Secrets for your applications on Nomad

adrian.todorov@atodorov.ninja (Adrian Todorov) — Thu, 05 Mar 2026 15:16:05 +0200

Description

A presentation delivered at the December 2024 Paris HashiCorp User Group, covering Nomad Workload Identity and how it can be used to securely introduce identities (for auth to third parties) and secrets to applications running in Nomad. Using GCP Workload Identity Federation and offline trust between GCP IAM and Nomad’s Workload Identity, all based on OIDC/JWT.

Links

GitHub repository with demo code | Slide Deck

Orchestrating Edge AI workloads on a Jetson Orin Nano with Nomad

adrian.todorov@atodorov.ninja (Adrian Todorov) — Fri, 27 Jun 2025 16:49:06 +0100

Introduction

Recently I’ve been tinkering around with my Home Assistant setup, more specifically adding voice control. I wanted to enhance it with a local LLM (large language model) to be able to do more advanced natural language processing (I’m in a multilingual household, and Home Assistant’s voice control is pretty static - you have to predefine phrases and actions, you can’t just improvise). I could have used a cloud service, but not only is that not fun, but it also has privacy implications. So I wanted to run it all locally, and preferably on a small, power-efficient device that can handle the workload.

Enter the Jetson Orin Nano from NVIDIA.

Jetson Orin Nano

The Jetson Orin Nano is a powerful, compact AI development board from NVIDIA. It has a 6 core ARM CPU, a Tensor GPU with 1024 CUDA/32 tensor cores, and 8GB of 102 GB/s (relatively slow, but usable) RAM. It’s a great platform for running small AI workloads locally, and is very power efficient. Recently NVIDIA released a software update (confusingly called Jetson Orin Nano Super) that added a significant performance improvement (mostly in terms of TOPS and memory bandwidth).

It runs “Jetpack”, a NVIDIA customised (with all the NVIDIA drivers and libraries) version of Ubuntu, and there is a pretty good community, ecosystem and documentation around it. In particular, Jetson Containers is a very good collection of ready-to-use containers (with correct versions of CUDA and all necessary configs, etc.) for popular AI / ML / robotics-related software.

As the underlying OS is based on Ubuntu, the software catalogue is vast, including HashiCorp Nomad, to plug into a larger homelab setup and orchestrate everything from a common control plane.

In my case, I’d like to run Ollama, a local LLM hosting server, to integrate it with Home Assistant, and maybe some other AI workloads. They need to run on the Jetson, but consumers don’t (such as Open WebUI, a frontend for Ollama, or Home Assistant). They can run elsewhere in my lab to avoid wasting precious memory that could be used as VRAM for the Large Language Models. For instance, OpenWebUI takes a good 800+ MB of RAM, and the Jetson has only 8GB, so it would be a waste to run it there.

Nomad

Nomad is a powerful, flexible, and easy to use orchestrator that can run on a wide range of platforms, including edge devices like the Jetson Orin Nano. It supports a variety of workloads, including containers and raw executables, and crucially for Edge scenarios, supports various distributed clusters topologies and tolerates unreliable networks.

Nomad supports linux/arm64, so everything should work out of the box. Furthermore, Nomad has an NVIDIA Device Driver so there should be direct support to fingerprint (discover and make targetable via attributes) the GPU and attach it to jobs.

This doesn’t sound too complex, right? While building it, I was pleasantly surprised with how simple the whole process was. Let’s dive in.

Jetson setup

NVIDIA have a detailed getting started tutorial on how to set up the Jetson Orin Nano, so I won’t go into too much detail about it here. The only tricky part was that the one I got was running a very old firmware version, so I had to jump through multiple steps to update it to a recent one that allows running the latest Jetpack 6.x (the version that delivers the drastically improved performance that led to NVIDIA calling it “Jetson Orin Nano Super”). Other than that, it was basically flashing a micro SD card, inserting it into the Jetson, and booting it up. Rinse, repeat this step several times until you get to the desired version.

An NVME drive is strongly recommended to store the heavy stuff (container images, models, etc.) - microSD cards are more expensive per GB at the higher end, aren’t made for such heavy random write I/O, and probably won’t survive that long. The Jetson Orin Nano has two available M.2 slots (a third one is used for the WiFi card), a PCIe 3.0 x4 in the 2280 format, and a PCIe 3.0 x2 in the 2230 format.

To allow for remote headless control, enable SSH from the Ubuntu GUI setup, and configure SSH keys.

Performance tuning

To get the most out of the Jetson, you need to disable the Desktop GUI (which just wastes resources), and set the power mode to “Max Performance”, which came with Jetpack 6.x. By default there’s a cap of 15W power consumption, which can be increased to 25W or “unlimited” - the setting I went for. This can be done using the nvpmodel command, which is part of the Jetpack installation. The command to set the power mode to unlimited is:

# set the power mode to unlimited
sudo /usr/sbin/nvpmodel -m 0

# disable desktop GUI to free up resources
sudo systemctl set-default multi-user.target

# reboot for the changes to take effect and everything to be cleaned
sudo reboot

Installing Nomad

If you’re new to Nomad, consider taking a look at the Introduction to Nomad.

Nomad has releases for arm64 (the CPU architecture of the Jetson family) available via an APT repository which works for Jetpack.

Connect to the Jetson via SSH, and run the following commands to install Nomad:


sudo apt-get update && sudo apt-get install wget gpg coreutils
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main"  | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install nomad

# Install CNI plugins which are used to configure networking for Nomad allocations
export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)
export CNI_PLUGIN_VERSION=v1.7.1
curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-linux-${ARCH_CNI}-${CNI_PLUGIN_VERSION}".tgz && \

We also need a configuration file for the Nomad agent to know which addresses to listen on, which folder to use for the Nomad and allocation data (on the NVME to avoid overtaxing the SD card), etc. In my case I’m connecting to an existing Nomad cluster, but it can also be a standalone single machine cluster with only the Jetson. The configuration file is in HCL format, and should be placed in /etc/nomad.d/.

hcl /etc/nomad/client.hcl


name = "jetson-nano-01"
region = "global"
datacenter = "home"

# the address the Nomad client will be available on, important to adjust especially on public networks where 
# it shouldn't be 0.0.0.0
bind_addr = "0.0.0.0"

# the folder where Nomad stores its data, which we want on the NVME drive
data_dir = "/ssd/nomad"

log_level = "INFO"
enable_syslog = true

client {
    enabled = true

    # the address of the Nomad servers/control plane; 
    # if there are none and this will be a single node cluster, add a server block 
    # with enabled = true and bootstrap_expect = 1 in a separate configuration file
    # in the same folder (e.g. /etc/nomad/server.hcl), and use 172.0.0.1 for the server address
    servers = ["nomad-servers.home"]
    # Node Pool, used to group nodes with similar capabilities (in this case, Jetsons)
    node_pool = "jetsons"
}

Setting up the NVIDIA Jetson GPU/APU for use with Nomad

Nomad has a concept called device drivers, which enable Nomad to fingerprint (discover) devices, collect information about them, and make them available for scheduling and targeting. They are with an open spec, anyone can write their own, and there are community ones for USB and Xlinix FPGAs, as well as an official NVIDIA driver. It discovers NVIDIA GPUs, collects their info (type, wattage, memory use, etc.) and allows you to deploy workloads specifically targeting them, like so:

hcl Example job using an NVIDIA GPU


job "gpu-test" {
  datacenters = ["dc1"]
  type = "batch"

  group "smi" {
    task "smi" {
      driver = "docker"

      config {
        image = "nvidia/cuda:12.8.1-base-ubuntu22.04"
        command = "nvidia-smi"
      }

      resources {
        device "nvidia/gpu" {
          count = 1

          # Add an affinity for a particular model
          affinity {
            attribute = "${device.model}"
            value     = "Tesla H100"
            weight    = 50
          }
        }
      }
    }
  }
}

I thought I’d need to use it, so compiled my own (currently there are no arm64 releases), tried running it… and got a bunch of errors from NVML, the underlying NVIDIA library for GPU management and monitoring. After some research, I discovered that contrary to my expectations, Jetsons aren’t supported by NVML, at all. It goes as far as detecting there’s an NVIDIA device of type “Orin”, and that’s it, none of the other methods such as getting the name, wattage, memory, etc. work.

Therefore the NVIDIA device driver, which relies entirely on NVML, is out. But is it actually needed?

Turns out, no, not really. CUDA workloads detect the integrated GPU on their own and directly use it. The only thing needed for containers is the NVIDIA container toolkit (which installs a special nvidia runtime), which comes preinstalled with Jetpack. The last part of the puzzle is a way to target the correct client, the one with the GPU, but this is easily solved by the fact that I put the Jetson in its own node pool, and use that to target it.

Note: Nomad can run applications with raw_exec where binaries/scripts are executed directly on the host OS, with no isolation. This could be marginally faster than with containers, but comes at the expense of increased difficulty when managing dependencies such as CUDA and various other libraries’ versions. There is already some complexity due to the dependency on the host NVIDIA driver and kernel, which is enough of a potential minefield, so I decided to stick to Docker containers for now.

Speaking of Docker containers, this brings me back to the jetson-containers repository by Dustin Franklin. It contains ready to use (with all necessary libraries and dependencies) containers for popular robotics, IoT, AI, ML, etc. frameworks/tools/platforms, as well as wrapper tools to start them with the appropriate settings. That’s a great starting point for running workloads on the Jetson, and what I’ll use to make my life easier. The containers are available on Docker Hub, and can be pulled and run like any other container.

Deploying Ollama and Open WebUI on Nomad

With all this set up, we can now deploy Ollama and Open WebUI on Nomad. The process is pretty straightforward, and is similar to deploying any other container on Nomad. The only difference is that Docker needs to use the nvidia runtime - NVIDIA’s Jetson guides recommend setting it as the default for the docker daemon (in /etc/docker/daemon.json), but it can also be specified in Nomad with runtime in the job spec.

hcl Ollama job spec


job "ollama" {
  # targeting the "jetsons" node pool which contains the Jetson Orin Nano
  datacenters = ["*"]
  node_pool = "jetsons"
  namespace = "llm" # namespace for all LLM-related jobs, needs to be created first
  type      = "service"

  group "ollama" {
    # persistent volume to store Ollama's models, data, cache, etc.
    volume "ollama" {
      type            = "host"
      source          = "ollama"
      access_mode     = "single-node-single-writer"
      attachment_mode = "file-system"
    }

    # service and network to make Ollama's API discoverable to other jobs, like Open WebUI
    # and Home Assistant
    service {
      name     = "ollama"
      port     = "ollama"
      provider = "nomad"
    }
    network {
      port "ollama" {
        static = 11434
      }
    }
    # Ollama Docker container (from jetson-containers), using the NVIDIA runtime
    task "ollama" {
      driver = "docker"
      config {
        image = "dustynv/ollama:r36.4.0"
        command = "ollama"
        args    = ["serve"]
        runtime = "nvidia"
        shm_size = 8192
        ports = ["ollama"]
      }
      # env vars to configure Ollama
      env {
        OLLAMA_HOST = "0.0.0.0"
        OLLAMA_MODELS = "/ollama/models"
        OLLAMA_LOGS = "/ollama/logs"
      }
      volume_mount {
        volume      = "ollama"
        destination = "/ollama"
      }
      resources {
        cpu    = 9000
        memory = 7168 # leave a GB for the OS, Nomad and other processes
      }
    }
  }
}

Next, a Nomad job spec for Open WebUI:

hcl OpenWebUI job spec


job "openwebui" {
  datacenters = ["*"]
  namespace = "llm"
  node_pool = "default" # default node pool, so that it doesn't run on the Jetson
  type       = "service"

  group "openwebui" {
    volume "openwebui" {
      type            = "host"
      source          = "openwebui"
      access_mode     = "single-node-single-writer"
      attachment_mode = "file-system"
    }

   service {
      name = "open-webui"
      port = "http"
      provider = "nomad"
    }

    network {
        port "http" {
            static = 8080
            to = 8080
        }
    }

  task "open-webui" {
    driver = "docker"
    config {
        image   = "ghcr.io/open-webui/open-webui:0.6"
        ports   = ["http"]
    }
    resources {
        cpu    = 600
        memory = 1024
    }

  # dynamic template to inject the Ollama server address into the OpenWebUI container,
  # with automatic restart on changes
  template {
    data = <<EOH
    # any extra environment variables needed by OpenWebUI can be added here
    # such as ENABLE_WEB_SEARCH=true and WEB_SEARCH_ENGINE=duckduckgo
OLLAMA_BASE_URL=http://{{range nomadService "ollama"}}{{ .Address}}:{{ .Port }}{{end}}
EOH

    destination = "secrets/file.env"
    env         = true
    change_mode = "restart"
    }
   volume_mount {
        volume      = "openwebui"
        destination = "/app/backend/data"
      }
  
  }
 }

For both jobs, I used Nomad volumes to create a persistent storage for the data (for models, cache, configuration). Volumes can be managed by CSI plugins, but I used the newly released builtin dynamic host volumes, which are much simpler to set up and use. They are created on the host, and mounted into the container. Example volume definition that will create a volume called ollama in the the Nomad data_dir (/ssd/nomad/host_volumes) folder on the host:

name            = "ollama"
namespace       = "llm"
type            = "host"
plugin_id       = "mkdir"
node_pool       = "jetsons"

capacity_min = "10G"
capacity_max = "100G"

capability {
  access_mode     = "single-node-single-writer"
  attachment_mode = "file-system"
}

Deploying all of those is a matter of running nomad volume run and nomad job run with the job spec files. The jobs will be scheduled on the Jetson Orin Nano, the Ollama server will be available on port 11434, and OpenWebUI on port 8080.

nomad volume create jobs/llm/ollama.volume.hcl
nomad volume create jobs/llm/openwebui.volume.hcl

nomad job run jobs/llm/ollama.hcl
nomad job run jobs/llm/openwebui.hcl

After a few minutes, the Ollama server should be up and running, and you can access it via the OpenWebUI at http://jetson|other-machine-IP:8080. Clients such as Home Assistant can connect to Ollama at http://jetson-IP:11434.

Conclusion

The NVIDIA Jetson Orin Nano is a decently powerful (for limited home use) AI development board, and with Nomad it’s very easy to orchestrate AI workloads on it and integrate it into a larger homelab setup.

Dynamic Identities and Secrets for your applications on Nomad

adrian.todorov@atodorov.ninja (Adrian Todorov) — Thu, 05 Dec 2024 15:16:05 +0200

Description

Links

GitHub repository with demo code | Slide Deck

Running a multi-gig Home Network in 2024

adrian.todorov@atodorov.ninja (Adrian Todorov) — Wed, 03 Jul 2024 19:01:40 +0200

Introduction

Why a multi-gig home network? What is a multi-gig network anyway? The second question is easy, it’s a network that runs at a speed higher than 1Gbps (usually one of or a combination of 2.5/5/10Gbps), which until a few years ago was almost exclusively reserved for the enterprise space due to hardware costs. But it isn’t anymore, there are quite a few relatively affordable options nowadays. The first question is a bit more complex, but the short answer is: because I can. The long answer is: because I enjoy tinkering with networking (to some extent), and because I want to have a fast network at home. My Internet Service Provider (ISP), Free (yes, really) already gives me lots of bandwith - 5Gbps down and 700Mbps up, so why not have a network that can handle as much as possible from that? I have a few bandwith intensive workloads at home - streaming from my NAS, video game streaming from the cloud / my gaming PC to my TV, remote storage for my Nomad cluster from my NAS, game downloads, backups, etc.

I embarked on this journey in 2022, but it took me some time to finish implementing everything and find the time to write about it, so I’m documenting it from here in 2024 (it didn’t actually take me 2 years to set up everything, I just did things sequentially with massive gaps in between). I’ll try to remember as much as possible, but I’ll also try to keep it up to date with the latest information I have.

Hardware

Router

One of the most important parts, the centerpriece of the network. I wanted to use a router with some very specific advanced features, which my ISP provided box doesn’t have (even if it has a lot of features for an ISP-provided one, such as WireGuard and OpenVPN, ad-blocking DNS, etc.), hence the need for a separate, proper one. Unfortunately there’s no way to entirely remove my ISP’s router (because it has an integrated 10G-EPON module to be able to actually use the fibre link to my home), but I can put it in bridge mode and use my own router behind it for all the logic and network management.

Requirements

Multiple (at least 3) multi-gig Ethernet ports (ideally 10Gbps) - due to the physical constraints in my network cupboard, and the need for the ISP’s router in bridge mode, there simply isn’t the space for a switch there too, hence the need for a router with multiple multi-gig ports (at least one for the “WAN” link to the ISP router, and two others for the two rooms with lots of equipment). As for why Ethernet, the place I live in is wired for Ethernet in every room, following a standard that should give me up to 10Gbps, so I want to take advantage of that, even if I know that it’s not perfect and consumes lots of power/heat compare to SFP+.
VLAN support (basic stuff)
VPN support, both as a server but also, and more importantly, as a client with something like ipsets to direct only certain traffic through VPNs (to avoid a few specific geoblocks)
(relatively) affordable
(relatively) low power consumption and (relatively) low noise and (relatively) small size - it’s going to live in a fully closed network cupboard, so it can’t be too loud or too hot, and there isn’t a ton of space either
Wi-Fi optional to not needed, because I have a separate Unifi WiFi access point that I’m happy with, and in any case the network cupboard being weirdly located would result in poor coverage
software support - I don’t want a router from a vendor that will stop updating it within a year or two, and then being stuck with outdated/insecure device at the core of my network. I was looking for something that has a good community around it, and/or a vendor that has a good track record of updating their devices for a long time

Options

It took quite some digging to find a router that fits all these requirements - most that did from the hardware side (multiple multi-gig ports) are “gamer routers” from the likes of Asus, Netgear and TP-Link (such as the Asus GT-AXE16000 or Netgear Nighthawk RS700S). Not only do they have Wi-Fi onboard which I don’t need, Netgear and TP-Link have… let’s say far from steallar reputation in terms of their software’s security and quickness/length of updates. Asus is a bit better because they use ASUSWRT (based on the Tomato network firmware), which is open source, and there’s a third party/community Asuswrt-Merlin project which builds on top of that, but it’s still not ideal. Their router that seemed to most closely match my requirements, the Asus GT-AXE16000, wasn’t supported by Asuswrt-Merlin and it was unclear when that support would land (and although I love contributing to open source, networking firmwares are way above my capabilities).

On the other side were the “prosumer” routers, such as those from Ubuquiti and Mikrotik. My previous network used to run on an Edgerouter-X, which was a great little router, so I was familiar with Ubiquiti’s software and hardware. However their product strategy (deprecating the Edge* line and focusing on the Unifi line, which is UI-managed and has less flexibility and features), as well as the overall quality of their software, and importantly in this case, the size of their routers (the Unifi Dream Machines are huge) made me look elsewhere. Which is a shame, because I plan on using their Access Points for WiFi, and it would have been great to be able to manage the whole network from a single central point. Mikrotik was a bit of a wildcard, I had never used their products, but they seemed to have a good reputation in the networking community, and their routers were relatively affordable. However, none had more than one multi-gig (and it was only 2.5Gbps) Ethernet port (like most vendors, and for very good reasons, they prioritise SFP+ ports for 1Gbps+).

By chance, I stumbled upon an article by ServeTheHome about an inexpensive 2.5Gbps mini PC that can serve as a router from Topton/Qotom (apparently appears under different brand names around the internet), based on a decent Intel CPU and with 4x Intel 2.5Gbps Ethernet ports. After some digging around I found a 6x 2.5Gbps port version, still using Intel NICs (Intel i226-V), which is important for compatibility with various OSes (the seller proposed shipping it with pfSense or OPNsense, but I guessed any *nix OS would be fine based on the specs and hardware involved). A big added benefit is the fanless design. The description of the device was exactly what I was looking for, and the price was very reasonable (~300 euros with shipping), so I decided to give it a try. I ordered it from AliExpress, and it arrived in a few weeks, much faster than planned.

Hardware specs:

Intel Celeron N5105 (4 cores, 4 threads, 2.0GHz base, 2.9GHz turbo) - decent low-powered CPU, more than enough for a home router, even with multiple 2.5Gbps ports, VPNs, etc.
8GB DDR4 RAM - more than enough, in normal use it uses less than 1.5GB
128GB SSD - more than enough, the whole OS + config + everything takes around 5GB
6x Intel i225-V 2.5Gbps Ethernet ports - currently I only use 4 of them, but it provides some flexibility for the future (like connecting a backup 4G modem, or after I upgrade my ISP’s router to one that has more than one 2.5Gbps port, make an aggregate of two ports and load balance between the two for 2x2.5Gbps Internet)

The exact model I got doesn’t seem to be in stock anymore, but there are all sorts of similar ones with newer processors and various combinations of ports available on Amazon, Ebay, AliExpress.

Operating System

Since I went with custom hardware, I had to also pick an appropraite OS. I initially tried OPNSense, but the configuration by ClickOps wasn’t my thing, especially for my scenario where I have 3 ports that need to be bridged and configured the same; it was a lot of clicking around. I hesitated rolling a regular Linux distro such as Debian, but in the end decided to go for a more specialized one, VyOS. It is based on Vyatta, same as EdgeOS from Ubiquiti which I already knew from my previous router, the venerable ER-X. It’s entirely CLI-based - even better, a declarative CLI, like real networking software, and has a good community and documentation. All the configuration is in a single file (effectively) with some Git-like commit/compare/rollback/etc. features, including an extremely powerful commit-confirm which will commit configuration changes, and reboot if you don’t explicitly confirm them; this is brilliant for network equipment where a misconfiguration can easily cut off your own admin access. The biggest downside is that Long Term Support Releases are for paying customers/contributors only; for everyone else, you either have to compile your own, or use the rolling release, which is what I went with. I’m not too worried about it, because VyOS makes it very easy to backup the configuration (it’s just a text file automatically backed up to my NAS on each configuration commit), so if anything goes wrong I can just reflash the device and restore the configuration.

VyOS’ documentation is pretty good on all the basics, all options and features, and importantly for people getting started, there are configuration blueprints which show you how to achieve complete things (like a site to site VPN). There’s also a good community of people writing about it, like this blog series by Level Zero Networking, and many EdgeOS-related blog posts, docs, forum posts, etc. also apply or are easy to translate.

Switches

I needed 3 switches - one for the living room (for the TV, console, media box, etc.), and two chained ones for the office - one covering my homelab, (3 Nomad mini PCs + another mini PC), and the other for the rest of the office (NAS, work and personal laptops, Raspberry Pis running Home Assistant, assorted projects, another 3 Nomad nodes, etc.) in two different corners of the room.

Requirements

A few multi-gig Ethernet ports (at least 2.5Gbps, but also at least one 10Gbps port for my NAS so that both my Nomad cluster and other uses can saturage their links) for the uplink and a few devices that can take advantage of it, but mostly 1Gbps ports; PoE isn’t needed because I have exactly one device that can use it, an Access Point, and it’s powered by a PoE injector
“managed” (mostly VLAN support)
(relatively) affordable
(relatively) small and (relatively) low noise - 1 of them would be in the living room, and slightly visible, so a full rackmount switch would be out of place

Options

For switches, there were much more options available on the market, with quite a lot of flexibility in terms of port combinations. After going over:

Mikrotik - CRS312-4C+8XG-RM with 4x combo 10G Ethernet/SFP+ ports, 8 1/2.5/5/10G Ethernet ports
Netgear:
- GS110EMX with 2x 1/2.5/5/10G Ethernet ports, 8x 1G Ethernet ports
- MS510TX with 1x SFP+, 1x1/2.5/5/10G, 2x 1/2.5/5G, 2x1/2.5G, 4x1G Ethernet ports
- MS510TXM with 2x SFP+, 4x1x1/2.5/5/10G, 4x1/2.5G
Zyxel XGS1250-12 with 1xSFP+, 3x1/2.5/5/10G, 8x 1G Ethernet ports.

I settled on the Zyxel, it was by far the cheapest option (~190 euros vs ~270 for the Netgear GS110EMX), and it provides a very good combination of ports for my needs.

I use the SFP+ port on one of them for 10Gbps to my NAS, a 10Gbps Ethernet cable to chain two of the switches (thus getting 10Gbps between my Nomad homelab and the NAS serving its persistent volumes), and 2.5Gbps uplinks to the router. The web UI they come with is a bit weird and it took me reading the docs twice to understand how to properly configure the VLANs, especially trunks (having a port that allows multiple vlans), but it does the trick. Also, there’s an OpenWRT firmware for those switches with a lot of extra functionality if someone needs more than the stock firmware provides, or if they want to convert the non-managed switches (which are cheaper) to a managed version.

I use VLANs extensively to separate the IoT devices from the rest of the network, and to have a separate LAB network for my Nomad homelab Nomad.

Wireless Access Point

I’m currently using an older Unifi Access Point (AC-PRO) that only does WiFi 5 (formerly known as ac), doesn’t support WiFi6 nor WiFi 6E, and only has an 1Gbps uplink (technically two of them, but the second one can only serve as a backup) so it can’t fully benefit from my fast wired network. I plan on upgrading to a new model that can make use of a multi-gig uplink, and to prepare for the future WiFi 6, 6E and 7 devices on my network. WiFi frequencies are quite congested where I live (and I don’t help with 3 2.4GHz networks, 2 Zigbee networks and a ton of Bluetooth devices over the same 2.4Ghz band), so moving to WiFi 6/6E/7 should have a noticeable impact. Hopefully.

Overall I’m happy with Ubiquiti’s Unifi line for Access Points - they work pretty well and the UI is perfectly fine for configuring them (unlike for a router), so I’ll probably get a Unifi U7 Pro or a U7 Pro Max that has a 2.5Gbps uplink, which is perfect for my setup.

In terms of networks, I have 3 separate SSIDs (WiFi networks, mapped to VLANs) - one for Internet of Things devices to keep them separate and only give them internet access, one for guests, and one for everything else.

Next Steps

Overal, I’m very happy with the current setup. It works very well, is super reliable, allows me to do everything I want to do (including, importantly, conditional VPN for certain traffic), and is relatively easy to manage (no ongoing maintenance other than a VyOS update very few weeks).

Hardware

Outside of the upgrade of the WiFi Access Point, one day I might upgrade the router to one with a better uplink (or figure out a way to add one to the current router, e.g. with a USB3 adaptor) to make use of more of my ISP’s bandwith, but that’s not a priority right now.

Observability

An important part of any network is observability - knowing what happens in it, bandwith actually used (to see if there are any bottlenecks that merit an upgrade), what devices are connected, how much they use (e.g. my washing machine downloading gigabytes from the Internet would not be normal), etc. I’ll use Grafana Alloy with Grafana Cloud-hosted Prometheus, monitoring the various devices with the old and battle tested SNMP.

Configuring Home Assistant OS with VLANs

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sun, 09 Jun 2024 19:10:40 +0200

Introduction

(if you know why you’re here, jump to the next section)

VLANs (virtual local area networks) are virtual networks that allow you to segment a single physical network into multiple logical ones. It’s a great way to isolate devices from each other and keep your network a bit more secure, especially with IoT/smart devices (as the famous saying goes, the s in IoT stands for security). In my case, all my “smart” devices are on a separate VLAN because I don’t trust them enough, and Home Assistant, which manages them all, needs to be accessible from both the IoT network and my main network. I use Home Assistant OS, which is a pre-built image for Raspberry Pi and generally pretty great, but hides a lot of things from the user, so the usual Linux configuration options are not available (you can’t just SSH in and use nmcli or /etc/network/interfaces).

Commands

You need to install the SSH & Web Terminal add-on to get a terminal to be able to run commands on your Home Assistant OS, configure it with a username and password/authorized key (important even if you want to use it directly in the UI, otherwise it refuses to start), and start it. Afterwards, you can “Open Web UI” / SSH in, and run the following commands using the ha cli:

# gets the network configuration, including the name of the network interface (something like end0)
ha network info

and then, for a static network / if you want your Home Assistant to have a static IP:

 ha network vlan $NETWORK_INTERFACE $VLAN_ID --ipv4-method static --ipv6-method disabled
  --ipv4-address 172.16.1.5/24 --ipv4-gateway 172.16.1.1 --ipv4-nameserver 172.16.1.1

or for DHCP:

 ha network vlan $NETWORK_INTERFACE $VLAN_ID --ipv4-method auto --ipv6-method disabled

Logging on Nomad and log aggregation with Loki

adrian.todorov@atodorov.ninja (Adrian Todorov) — Fri, 09 Jul 2021 18:38:43 +0100

Introduction

When running a task orchestrator like Nomad or Kubernetes, there’s usually a bunch of different instances ( containers, micro-VMs, jails, etc. ) running, more or less ephemerally, across a fleet of servers. By default all logs would be local to the nodes actually running the stuff we want to run, making it burdensome to debug, correlate events, alert, etc., especially if the node crashes, hence why it’s a well established practice to collect all logs they emit to a central location, where all of those actions happen.

One of the most popular log management stacks is the so-called ELK ( ElasticSearch for log indexing, Logstash for parsing/transforming them, and Kibana for visualisation, with either Filebeat or Fluentd / Fluent bit for log collection), which has a few drawbacks - most notably heavy resource consumption and licence changes, the latter leading to numerous forks, which will probably result in some chaos/incompatibilities in the future.

A recent-ish contender in that space is Grafana Labs’ Loki. It’s a lightweight Prometheus-inspired tool, which can be run as a bunch of microservices for better and easier scale-out, or in monolithic all-in-one mode. In contrast to ElasticSearch, it only indexes labels ( which are user defined ), the logs themselves ( chunks ) are stored as-is, separately. That makes it more flexible and much cheaper with regards to storage and compute.

There’s a variety of storage options - Cassandra (index and chunk), GCS/S3 (chunk), BigTable (chunk), DynamoDB(chunk) and the monolithic-only local storage ( which uses BoltDB internally for indexing). The last one is great for getting started/testing/non-huge projects, and the lack of redundancy ( since it writes to the local filesystem) can be offset by the storage provider ( e.g. a CSI plugin, GlusterFS, DRBD, or good old NFS). Visualisation from Loki is done with Grafana ( not really surprising since they’re both made by the same company), via the LogQL PromQL-inspired query language.

Having been using Loki for a few months in production, I’m really like it and hence this article on logs and using it on and with Nomad.

There are two main ways to get logs from Nomad tasks into Loki ( one of which only works for Docker tasks), and I’ll discuss the pros and cons of each one later on.

How logs on Nomad work

By default Nomad writes task logs to the alloc/logs folder shared by all tasks in an allocation, which is stored locally on the client node running it. That makes accessing logs slightly burdensome and look like the following:

# get all current allocations for the job
$ nomad job status random-logger-example 
ID            = random-logger-example
Name          = random-logger-example
Submit Date   = 2021-04-27T23:28:29+02:00
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
random      0       0         1        0       0         0

Latest Deployment
ID          = beffde05
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
random      1        1       0        0          2021-04-27T23:38:29+02:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
529b074e  a394f493  random      0        run      running  8s ago   3s ago

# get the logs for an allocation
$ nomad alloc logs 529b074e

As soon as there’s more than one allocation, you have to get the logs for each one by one; if you want to see the logs of an old allocation ( assuming it hasn’t been garbage collected already ), you have to run nomad job status with the -all-allocs flag to include non-current versions.

Specific log options for the task driver can be configured on multiple levels - per task, per client, or in certain cases, most notably Docker, per task driver.

On the task level one can send all logs to a central syslog server (easily adaptable to fluentd, Splunk, etc. ) like so:

task "syslog" {
      driver = "docker"
      config {
        logging {
          type = "syslog"
          config = {
            syslog-address         = "tcp://my-syslog-server:10514"
            tag                    = "${NOMAD_TASK_NAME}"
          }
        }
      }
}

This is perfectly fine, except that you have to do it on each and every task, which can quickly become burdensome, especially if your logging configuration changes one day, and you need to be on Docker Engine 20.10+, or use a logging driver that supports writing the logs locally as well as to the remote endpoint, otherwise docker logs and nomad alloc logs are unusable because the logs get sent directly. To facilitate that, there is use the new Docker driver-wide logging configuration, but then you’re limited only to Docker logging drivers, and, the bigger issue in my opinion, you’re lacking context (unless the stuff you run detects and specifically logs that context).

The, in my humble opinion, better option, is to have something that runs automatically on all client nodes, via a system job, and collects all logs, like what everyone does with their Kubernetes clusters via Promtail, Fluend/Fluentbit, Datadog agent, Splunk Connect, etc. The biggest challenges with that are:

context - it’s good to know everything about where the logs are from - which task, namespace, client node, etc.
discovery - how to find all new allocations to ship the logs of?

Recently, there have been some great news concerning those two challenges in regards to Docker-based tasks - this PR on Nomad that adds extra Docker labels with job, task, task group, node, namespace names, this PR that enables default logging options for the Docker driver on the Nomad side.

I’ll describe how to collect and ship logs from Nomad, and then briefly go over how to run Loki, a lightweight log aggregation tool by Grafana Labs, on Nomad.

Logs collection

Nomad makes each allocation’s logs available through the API, so ideally our logging agent should be able to connect to it directly. The only mainstream one that does that at the time of writing is:

Filebeat

Filebeat, Elastic’s log collection agent, has an auto-discovery module for Nomad, experimentally via this PR which hasn’t been released in a stable version yet. If one wants to use ElasticSearch, or any of the compatible alternatives or companions ( Logstash, Kafka), that’s great. In theory one can configure Filebeat to output to a file, and then parse that file with another logging agent ( like Vector or Promtail) and ship the logs somewhere else, but IMHO that seems too much hassle for little benefit.

Promtail

Grafana provide a log collecting companion to Loki in the form of Promtail, which is lightweight and easy to configure, but much more limited compared to filebeat, fluent* or Vector in terms of compatible sources. The biggest downside is that for Docker logs it can’t use Docker labels for extra context, and having only the container name and ID is often insufficient.

Vector

Vector is a very fast and lightweight observability agent that can collect logs and metrics, transform and ship them to a variety of backends by Timber, a logging SaaS. They were recently acquired by Datadog, who also have an agent and logging backend, so they obviously saw some value in them. And, very importantly for this story, Vector’s Docker logs source can use Docker labels to enrich the logs’ context, and there’s a Loki sink ( backend ). So, let’s see about deploying Vector as a system job to collect logs from Docker.

First, we need to allow Vector to access the Docker daemon and configure Nomad to add extra metadata; the easiest and cleanest way to do that is to pass the Docker socket as a host volume, which need to be declared at the host (Nomad client) level, and then mounted in the Vector job. We can do that read-only, and we can use ACLs to limit who can mount it to avoid allowing every job to mount it and do whatever.

hcl Nomad client configuration for the Docker sock host volume and extra Docker labels


plugin "docker" {
  config {  
    # extra Docker labels to be set by Nomad on each Docker container with the appropriate value
    extra_labels = ["job_name", "task_group_name", "task_name", "namespace", "node_name"]
  }
}

client {
  host_volume "docker-sock-ro" {
    path = "/var/run/docker.sock"
    read_only = true
  }
}

hcl An ACL policy allowing the host volume to be mounted as read-only


host_volume "docker-sock-ro" {
  policy = "read" # read = read-only, write = read/write, deny to deny
}

Second, actually mount the newly available host volume inside the Vector task:

hcl Declaring and mounting the Docker sock volume



group "vector" {
    ...
    volume "docker-sock" {
      type = "host"
      source = "docker-sock"
      read_only = true
    }
    task "vector" {
      ...
      volume_mount {
        volume = "docker-sock"
        destination = "/var/run/docker.sock"
        read_only = true
      }
    }
}

Third, tell Vector to collect Docker’s logs and send them to Loki, enriching them with the metadata extracted from the Docker labels:

toml Vector configuration file


          data_dir = "alloc/data/vector/"
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki" 
            inputs = ["logs"] 
            endpoint = "http://loki.example"
            encoding.codec = "json" 
            healthcheck.enabled = true 
            # since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
            # we need to escape them twice, once for TOML, once for Vector
            labels.job = "{{ label.com\\.hashicorp\\.nomad\\.job_name }}"
            labels.task = "{{ label.com\\.hashicorp\\.nomad\\.task_name }}"
            labels.group = "{{ label.com\\.hashicorp\\.nomad\\.task_group_name }}"
            labels.namespace = "{{ label.com\\.hashicorp\\.nomad\\.namespace }}"
            labels.node = "{{ label.com\\.hashicorp\\.nomad\\.node_name }}"
            # remove fields that have been converted to labels to avoid having them twice
            remove_label_fields = true

And that’s it.

Full example Vector job file with all those components and dynamic sourcing of the Loki address from Consul:

hcl Vector job file


job "vector" {
  datacenters = ["dc1"]
  # system job, runs on all nodes
  type = "system" 
  update {
    min_healthy_time = "10s"
    healthy_deadline = "5m"
    progress_deadline = "10m"
    auto_revert = true
  }
  group "vector" {
    count = 1
    restart {
      attempts = 3
      interval = "10m"
      delay = "30s"
      mode = "fail"
    }
    network {
      port "api" {
        to = 8686
      }
    }
    # docker socket volume
    volume "docker-sock" {
      type = "host"
      source = "docker-sock"
      read_only = true
    }
    ephemeral_disk {
      size    = 500
      sticky  = true
    }
    task "vector" {
      driver = "docker"
      config {
        image = "timberio/vector:0.14.X-alpine"
        ports = ["api"]
      }
      # docker socket volume mount
      volume_mount {
        volume = "docker-sock"
        destination = "/var/run/docker.sock"
        read_only = true
      }
      # Vector won't start unless the sinks(backends) configured are healthy
      env {
        VECTOR_CONFIG = "local/vector.toml"
        VECTOR_REQUIRE_HEALTHY = "true"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available 
      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }
      # template with Vector's configuration
      template {
        destination = "local/vector.toml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }}
        left_delimiter = "[["
        right_delimiter = "]]"
        data=<<EOH
          data_dir = "alloc/data/vector/"
          [api]
            enabled = true
            address = "0.0.0.0:8686"
            playground = true
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki" 
            inputs = ["logs"] 
            endpoint = "http://[[ range service "loki" ]][[ .Address ]]:[[ .Port ]][[ end ]]" 
            encoding.codec = "json" 
            healthcheck.enabled = true 
            # since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
            # we need to escape them twice, once for TOML, once for Vector
            labels.job = "{{ label.com\\.hashicorp\\.nomad\\.job_name }}"
            labels.task = "{{ label.com\\.hashicorp\\.nomad\\.task_name }}"
            labels.group = "{{ label.com\\.hashicorp\\.nomad\\.task_group_name }}"
            labels.namespace = "{{ label.com\\.hashicorp\\.nomad\\.namespace }}"
            labels.node = "{{ label.com\\.hashicorp\\.nomad\\.node_name }}"
            # remove fields that have been converted to labels to avoid having the field twice
            remove_label_fields = true
        EOH
      }
      service {
        check {
          port     = "api"
          type     = "http"
          path     = "/health"
          interval = "30s"
          timeout  = "5s"
        }
      }
      kill_timeout = "30s"
    }
  }
}

Overall, that’s pretty a pretty good solution for the problem, with a lightweight and fast logging agent, some context, and everything is configured from within Nomad; however, that only works for Docker tasks and requires some (light) configuration on each client node.

nomad_follower (+Promtail)

There’s a project called nomad_follower, which uses Nomad’s API to tail allocation logs. It requires no outside (of the Nomad system job running it) configuration, and compiles all the logs with metadata and context in a file which can then be scraped by a logging agent and sent to whatever logging backend you have. It’s a bit more obscure compared to Vector or Filebeat, and debugging it might require diving in the code ( speaking from experience, it didn’t accept my logfmt formatted logs because it was looking at a timestamp starting in the first 4 characters of each line (used to deal with multi-line logs), but mine started after 5.. so i had to fork and patch it). Nonetheless, coupled with a logging agent it’s a fairly decent solution that works for all task types.

Configuration is done via env variables ( like the file where to write all logs, the Nomad/Consul service tag to filter on, etc.), including Nomad API access/credentials. The logs are stored in JSON, with JSON-formatted logs inside the data field, non-JSON (like logfmt) logs are in the message field, and there’s a bunch of metadata alongside them:

alloc_id
job_name
job_meta
node_name
service_name
service_tags
task_meta
task_name

hcl Example nomad_follower job file


  group "log-shipping" {
    count = 1
    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }
    ephemeral_disk {
      size = 300
    }
    task "nomad-follower" {
      driver = "docker"
      config {
        image = "sofixa/nomad_follower:latest"
      }
      env {
        VERBOSE    = 4
        LOG_TAG    = "logging"
        LOG_FILE   = "${NOMAD_ALLOC_DIR}/nomad-logs.log"
        # this is the IP of the docker0 interface
        # and Nomad has been explicitly told to listen on it so that Docker tasks can communicate with the API
        NOMAD_ADDR = "http://172.17.0.1:4646" 
        # Nomad ACL token, could be sourced via template from Vault
        NOMAD_TOKEN = "xxxx" 
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available 
      resources {
        cpu    = 100
        memory = 512
      }
    }
  }

That will result in all allocations running on the same node as an allocation from this task with the logging tag to get their logs collected and stored in ${NOMAD_ALLOC_DIR}/nomad-logs.log. Now all we need is to get something to read, parse and send those logs; considering I’d like to use Loki for storage, Promtail seems like the best option, but of course any of the alternatives could do the job just as well.

Promtail’s configuration is split into scrape (collection), pipeline that parses/transforms/extracts labels, and generic ( Loki adress, ports for healthcheck, etc.). It has a few ways to scrape logs, the one we need in this case is the static, which tails a file; and we also need to parse the various fields ( via a json pipeline stage) and mark some as labels ( so they’re indexed and we can search by them).

To scrape and parse the logs pre-collected by nomad_follower, we need a configuration similar to this one:

yaml Example Promtail YAML configuration file


server:
  # port for the healthcheck
  http_listen_port: 3000
  grpc_listen_port: 0
positions:
  filename: ${NOMAD_ALLOC_DIR}/positions.yaml
client:
  url: http://loki.example/loki/api/v1/push
scrape_configs:
- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: nomad
      __path__: "${NOMAD_ALLOC_DIR}/nomad-logs.log"
  pipeline_stages:
    # extract the fields from the JSON logs
    - json:
        expressions:
          alloc_id: alloc_id
          job_name: job_name
          job_meta: job_meta
          node_name: node_name
          service_name: service_name
          service_tags: service_tags
          task_meta: task_meta
          task_name: task_name
          message: message
          data: data
    # the following fields are used as labels and are indexed:
    - labels:
        job_name:
        task_name:
        service_name:
        node_name:
        service_tags:
    # an example regex to extract a field called time from within message( which is for non-JSON formatted logs,
    # so the assumption is that they're in the logfmt format,
    # and a field time= is present with a timestamp in, which is the actual timestamp of the log)
    - regex:
        expression: ".*time=\\\"(?P<timestamp>\\S*)\\\"[ ]"
        source: "message"
    - timestamp:
        source: timestamp
        format: RFC3339

A full example system job file, with nomad_follower and Promtail, with a template to dynamically source Loki’s address from within Consul:

hcl Example log_follower job file


job "log-shipping" {
  datacenters = ["dc1"]
  type = "system"
  namespace = "logs"
  update {
    max_parallel      = 1
    min_healthy_time  = "10s"
    healthy_deadline  = "3m"
    progress_deadline = "10m"
    auto_revert       = false
  }
  group "log-shipping" {
    count = 1
    network {
      port "promtail-healthcheck" {
        to = 3000
      }
    }
    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }
    ephemeral_disk {
      size = 300
    }
    task "nomad-forwarder" {
      driver = "docker"
      env {
        VERBOSE    = 4
        LOG_TAG    = "logging"
        LOG_FILE   = "${NOMAD_ALLOC_DIR}/nomad-logs.log"
        # this is the IP of the docker0 interface
        # and Nomad has been explicitly told to listen on it so that Docker tasks can communicate with the API
        NOMAD_ADDR = "http://172.17.0.1:4646"
        # Nomad ACL token, could be sourced via template from Vault
        NOMAD_TOKEN = "xxxx"
      }
      config {
        image = "sofixa/nomad_follower:latest"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available
      resources {
        cpu    = 100
        memory = 512
      }
    }
    task "promtail" {
      driver = "docker"
      config {
        image = "grafana/promtail:2.2.1"
        args = [
          "-config.file",
          "local/config.yaml",
          "-print-config-stderr",
        ]
        ports = ["promtail-healthcheck"]
      }
      template {
        data = <<EOH
server:
  http_listen_port: 3000
  grpc_listen_port: 0

positions:
  filename: ${NOMAD_ALLOC_DIR}/positions.yaml

client:
  url: http://loki.example/loki/api/v1/push
scrape_configs:
- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: nomad
      __path__: "${NOMAD_ALLOC_DIR}/nomad-logs.log"
  pipeline_stages:
    # extract the fields from the JSON logs
    - json:
        expressions:
          alloc_id: alloc_id
          job_name: job_name
          job_meta: job_meta
          node_name: node_name
          service_name: service_name
          service_tags: service_tags
          task_meta: task_meta
          task_name: task_name
          message: message
          data: data
    # the following fields are used as labels and are indexed:
    - labels:
        job_name:
        task_name:
        service_name:
        node_name:
        service_tags:
    # use a regex to extract a field called time from within message( which is for non-JSON formatted logs,
    # so the assumption is that they're in the logfmt format,
    # and a field time= is present with a timestamp in, which is the actual timestamp of the log)
    - regex:
        expression: ".*time=\\\"(?P<timestamp>\\S*)\\\"[ ]"
        source: "message"
    - timestamp:
        source: timestamp
        format: RFC3339
EOH
        destination = "local/config.yaml"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available
      resources {
        cpu    = 500
        memory = 512
      }
      service {
        name = "promtail"
        port = "promtail-healthcheck"
        check {
          type     = "http"
          path     = "/ready"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Running Loki

Note: I personally consider it somewhat of an anti-pattern to run your log aggregation system on the same cluster as the one it’s aggregating logs from. If the cluster explodes, you can’t really access the logs to debug what happened and why. For smaller use cases, or if critical failure isn’t probable, it’s perfectly fine and will probably never cause any issues; past a certain point though, I’d recommend splitting it ( and other similarly scoped tools like monitoring) into a separate management cluster, or using the hosted version, which includes a decent free tier (50GB and 14 days retention when it comes to logs).

Like I mentioned last time, even though one can include YAML configuration from Consul in Nomad jobs via the template stanza, I prefer to have as much as possible in the Nomad file directly for better versioning and rollbackability ( YAML stored in Consul evolves independently of the Nomad job’s lifecycle, so rollbacking the latter won’t do anything about the former ).

To run a basic, monolithic Loki service with local storage and unlimited retention, a host volume for said storage, Traefik as a reverse proxy (with TLS and a basic auth middleware attached, since Loki doesn’t do auth itself) and Loki’s YAML configuration file embedded, you need something along these lines:

job "loki" {
  datacenters = ["dc1"]
  type        = "service"
  update {
    max_parallel      = 1
    health_check      = "checks"
    min_healthy_time  = "10s"
    healthy_deadline  = "3m"
    progress_deadline = "5m"
  }
  group "loki" {
    count = 1
    restart {
      attempts = 3
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }
    network {
      port "loki" {
        to = 3100
      }
    }
    volume "loki" {
      type      = "host"
      read_only = false
      source    = "loki"
    }
    task "loki" {
      driver = "docker"
      config {
        image = "grafana/loki:2.2.1"
        args = [
          "-config.file",
          "local/loki/local-config.yaml",
        ]
        ports = ["loki"]
      }
      volume_mount {
        volume      = "loki"
        destination = "/loki"
        read_only   = false
      }
      template {
        data = <<EOH
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  # Any chunk not receiving new logs in this time will be flushed
  chunk_idle_period: 1h       
  # All chunks will be flushed when they hit this age, default is 1h
  max_chunk_age: 1h           
  # Loki will attempt to build chunks up to 1.5MB, flushing if chunk_idle_period or max_chunk_age is reached first
  chunk_target_size: 1048576  
  # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  chunk_retain_period: 30s    
  max_transfer_retries: 0     # Chunk transfers disabled
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
EOH
        destination = "local/loki/local-config.yaml"
      }
      resources {
        cpu    = 512
        memory = 256
      }
      service {
        name = "loki"
        port = "loki"
        check {
          name     = "Loki healthcheck"
          port     = "loki"
          type     = "http"
          path     = "/ready"
          interval = "20s"
          timeout  = "5s"
          check_restart {
            limit           = 3
            grace           = "60s"
            ignore_warnings = false
          }
        }
        tags = [
          "traefik.enable=true",
          "traefik.http.routers.loki.tls=true",
          # the middleware has to be declared somewhere else, we only attach it here
          "traefik.http.routers.loki.middlewares=loki-basicauth@file", 
        ]
      }
    }
  }
}

Conclusion

So, which logging agent to use with Nomad ? As always, it depends. If you have ElasticSearch or any of the compatible alternatives, Filebeat is probably the best option due to the native support. If you only run Docker-based tasks, Vector collecting Docker Daemon’s logs (with Docker labels for context) is a pretty decent option. If neither, or you want more context ( like custom metadata) than what is available with Vector, nomad_follower is for you.

Discuss

Hacker News

Twitter

Using Traefik on Nomad

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sat, 27 Mar 2021 23:20:43 +0100

Introduction

Traefik is a great load balancer, which uses dynamic configuration from a variety of providers, notably in this case Consul Catalog, which Nomad jobs can register into, providing a fast and easy way of having automatic virtual hosts and load balancing (ingress) for all of our Nomad jobs. There’s already a decent basic tutorial on Hashicorp Learn about doing just that, so I’ll focus on more advanced patterns and use cases that I’ve had to deal with.

A full Nomad job file can be found here.

Base

Traefik’ s configuration comes in two parts - static, like on which ports to listen, tracing, logs, etc. and dynamic, which comes from a third-party source(like Consul’ s service catalogue) and is used to dynamically create routers (aka virtual hosts, which can be HTTP, TCP or UDP) and services(aka backends).

The static configuration can be provided via CLI flags on start-up, or auto-reloaded YAML / TOML configuration files (which in the case of Traefik running on Nomad can come from Consul or Vault via the template stanza), and contains the basics - entrypoints(on which ports to listen), access logs, metrics, etc. and middlewares (Attached to the routers, pieces of middleware are a means of tweaking the requests before they are sent to your service).

Personally I prefer to have as much as possible in the Nomad file for better version history and rollbackability (if the configuration of a middleware comes from a YAML file stored in Consul, its updates are independent of Nomad, and you if you rollback your Nomad deployment to version X, it won’ t rollback the value in Consul as well), and specifically in the case of Traefik, I think that CLI flags are a bit leaner and sufficiently readable for the basics.

A small example, minimal Traefik configuration with the dashboard and API enabled, healthcheck, Prometheus-compatible metrics, access logs :

yaml YAML version


# to make use of the file, we need to tell Traefik to watch the folder where it is
args = ["--providers.file.directory=local/traefik/"]
template {
  data        = <<EOH
  ping: {}
  accessLog: {}
  api:
    dashboard: true
  metrics:
    entryPoint: metrics
EOH
  destination = "local/traefik/base-config.yaml"

bash CLI flags version


  args = [
    "--ping=true",
    "--accesslog=true",
    "--api=true",
    "--metrics=true",
    "--metrics.prometheus=true",
    "--metrics.prometheus.entryPoint=metrics",
  ]

I’ll provide most examples from here on in the CLI format, but converting them to TOML or YAML shouldn’t be very complicated in most cases with the help of the Traefik docs, which show everything in the different versions.

The bare minimum to get Traefik running under Nomad looks like this:

task "traefik" {
  driver = "docker"
  config {
    image = "traefik:2.4"
    args = [
      "--entryPoints.http.address=:80",
      "--accesslog=true",
      "--api=true",
      "--metrics=true",
      "--metrics.prometheus=true",
      "--metrics.prometheus.entryPoint=http",
      "--ping=true",
      "--ping.entryPoint=http",
    ]
    ports = ["http"]
  }

Static configuration from Consul via templates

Traefik has a file configuration provider (with automatic and hot reloads, hence the noop change_mode), and with the following argument --providersfile.directory=local/traefik/ we can use Nomad templates to generate extra configuration coming from Consul:

hcl Consul template


template {
  data        = "{{ key \"traefik/auth\" }} "
  destination = "local/traefik/auth.yaml"
  change_mode = "noop"
}

As I already mentioned though, I prefer to have most of Traefik’ s configuration in the Nomad job file directly for rollbackability.

TLS configuration from Vault via templates

Using the same argument --providers.file.directory=local/traefik/ and the following templates, we can have TLS certificates stored in Vault with restarts (due to the fact that Traefik doesn’t support hot reload of certificates).

hcl manual TLS from Vault


template {
  data = <<EOH
    tls:
      certificates:
        - certFile: "local/secret/example.com.crt"
          keyFile: "local/secret/example.com.key"
   EOH
  destination = "local/traefik/cert.yaml"
  change_mode = "noop"
}
template {
  data        = "{{ with secret \"secret/data/example.com\"  }}{{.Data.data.key}}{{end}}"
  destination = "local/secret/example.com.key"
  change_mode = "restart"
  splay       = "1m"
}
template {
  data        = "{{ with secret \"secret/data/example.com\"  }}{{.Data.data.crt}}{{end}}"
  destination = "local/secret/example.com.crt"
  change_mode = "restart"
  splay       = "1m"
}

Consul Catalog

To be able to create dynamic routers based on the services we have in Consul, Traefik needs some configuration (full doc here) on itself, and then tags on those services telling it the specifics:

bash flags version


  "--providers.consulcatalog=true",
  "--providers.consulcatalog.exposedByDefault=false",
  "--providers.consulcatalog.endpoint.address=http://172.17.0.1:8500",
  "--providers.consulcatalog.prefix=traefik",
  "--providers.consulcatalog.defaultrule=Host(`{{ .Name }}.example.com`)",

toml TOML version


[providers.consulCatalog]
  prefix           = "traefik"
  exposedByDefault = false
  defaultRule      = "Host(`{{ .Name }}.{{ index .Labels \"customLabel\"}}`)"
  constraints      = "Tag(`a.tag.name`)"
  [providers.consulCatalog.endpoint]
    address = "http://172.17.0.1:8500"

Basically we enable the Consul Catalog provider and point it to a Consul agent, determine the “prefix” (tags on Consul services starting with prefix will be looked at for extra configuration, so for prefix = traefik, tags like traefik.http.routers.api.service = api@internal will be used), define whether or not services are exposed by default (IMHO they shouldn’t be), the default routing rule (Host {{.Name }}.example.com will use service-name.example.com by default, over-ridable per-service via tags).

Optionally we can also add constraints, which allows to do complex matching on tags to determine if a router should be created or not, mostly useful if exposedByDefault is true.

Note on the Consul address - since Traefik runs inside Docker, pointing it to the default localhost:8500 won’t work unless we run it in network_mode = host (sharing the host’s networking, more on that below); in my case, I’m pointing it to the host’s docker0 IP, 172.17.0.1, which is accessible from all Docker containers, and is the same across all my hosts, and making Consul listen on the docker0 on top of usual localhost via the following configuration on the consul agent.

hcl Consul agent configuration to make its API and DNS interfaces available from within docker containers


  addresses {
      http = "127.0.0.1 {{ GetInterfaceIP \"docker0\" }}"
  }

There’s also bunch of more advanced options around Consul auth, API read consistency requirements and refreshInterval.

Here’s how to make use of the provider in the Nomad job file via the service stanza:

hcl Example Traefik configuration via Consul tags


service {
  name = "my-service"
  port = "https"
  tags = [
    # HTTPS-only service which will be called when the URL is example.com or example.org/something
    # with my-middleware attached
    "traefik.enable=true",
    "traefik.http.routers.my-service.rule=Host(`example.com`) || (Host(`example.org`) && Path(`/something`))",
    "traefik.http.routers.my-service.tls=true",
    "traefik.http.routers.my-service.middlewares=my-middleware",
  ]
}

If exposedByDefault is true and you have all the configuration you need by default (e.g.http->https redirect and other middlewares you need on the entrypoint level), you don’t even need per-service configuration unless you want to make something specific.

Networking

There are two main ways you can configure Nomad in regards to Traefik networking - host networking or with static ports. Host networking, which is the same as Docker’s --network="host", adds the task to the host’s network namespace and shares its network stack, and is generally not recommended.

Static ports are host-level ports that are declared statically (compared to regular Nomad ports, which are ephemeral and random on the host level, and rather impractical for incoming http / https traffic) and forwarded to the task, like so:

hcl single network ports example


group "traefik" {
  network {
    port "http" {
      static = 80
    }
    port "https" {
      static = 443
    }
  }
  task "traefik" {
    driver = "docker"
    config {
      image = "traefik:2.4"
      args = [
        "--entryPoints.http.address=:80",
        "--entryPoints.https.address=:443",
      ]
      ports = ["http", "https"]
      ...

This declares the ports 80 and 443, and attaches them to the Traefik task, which uses them as entrypoints named http and https.

If you have multiple networks, you can make use of Nomad’s host_network configuration, where you declare the available networks with aliases on the Nomad client and target them with the host_network option on ports. ( In reality that’s what Nomad does behind the scenes, using the first network it finds for a hidden default host_network, and attaching ports to it). Do note however, that if you declare host_networks, ports which don’t specify the host_network will use the default value of default, and if no such network exists, allocations will fail (silently for system jobs).

hcl multiple host_networks example


group "traefik" {
  network {
    port "http-priv" {
      static       = 80
      host_network = "private"
    }
    port "https-priv" {
      static       = 443
      host_network = "private"
    }
    port "http-pub" {
      static       = 80
      host_network = "public"
    }
    port "https-pub" {
      static       = 443
      host_network = "public"
    }
  }
  task "traefik" {
    driver = "docker"
    config {
      image = "traefik:2.4"
      args = [
        "--entryPoints.http.address=:80",
        "--entryPoints.https.address=:443",
      ]
      ports = ["http-priv", "https-priv", "http-pub", "https-pub"]
      ...

This will make available the http and https Traefik endpoints on both networks, and can of course be adapted to have specific endpoints only on specific networks.

If for some reason the above doesn’t work for you (like if you use VRRP with a virtual IP floating between multiple machines, host_network might not work due Nomad fingerprinting the networks on start-up, and since the floating IP won’t be available on all machines simultaneously, Nomad won’t bind to it), you can resort to host-mode networking (again, it ’ s generally discouraged) where the task shares the host’s networking, like so:

hcl host networking


group "traefik" {
  network {
    port "http" {
      static = 80
    }
    port "https" {
      static = 443
    }
  }
  task "traefik" {
    driver = "docker"
    config {
      image = "traefik:2.4"
      args = [
        "--entryPoints.http.address=:80",
        "--entryPoints.https.address=:443",
      ]
      ports        = ["http", "https"]
      network_mode = "host"
      ...

With this, Traefik will be available on ports 80 and 443 on all network interfaces on the host.

Security

Traefik’s API, dashboard and metrics endpoints should be protected for security reasons, which can be done by putting them on a separate entrypoint which is firewalled off and/or adding middlewares with auth. The API and dashboard are represented by a special service, api@internal, and Prometheus-compatible /metrics can be put on a dedicated service as well (via themanualRouting = true option, which will create a prometheus@internal service ).

Here’s a basic example of that, making use of tags on the Consul service of Traefik to dynamically configure the routes and middlewares:

hcl securing traefik’s API, dashboard and metrics


task "traefik" {
  driver = "docker"
  config {
    image = "traefik:2.4"
    args = [
      # defining 3 entrypoints, on ports 80, 443 and 8080 ( for admin stuff)
      # and putting the healthcheck and metrics (with manual routing) on the admin endpoint
      "--entryPoints.http.address=:80",
      "--entryPoints.https.address=:443",
      "--entryPoints.admin.address=:8080",
      "--entrypoints.http.http.redirections.entryPoint.to=https",
      "--accesslog=true",
      "--api=true",
      "--metrics=true",
      "--metrics.prometheus=true",
      "--metrics.prometheus.entryPoint=admin",
      "--metrics.prometheus.manualrouting=true",
      "--ping=true",
      "--ping.entryPoint=admin",
      "--providers.consulcatalog=true",
      "--providers.consulcatalog.endpoint.address=http://172.17.0.1:8500",
      "--providers.consulcatalog.prefix=traefik",
    ]
    ports = ["http", "https", "admin"]
  }
  service {
    name = "traefik"
    port = "https"
    tags = [
      # using Consul service tags prefixed by traefik (as defined in `--providers.consulcatalog.prefix`) 
      # to configure api&dashboard routers
      # with a headers check ( spoofable ) and a basic auth middleware attached inline
      "traefik.enable=true",
      "traefik.http.routers.api.rule=Host(`traefik.example.com`) && HeadersRegexp(`X-Real-Ip`, `^(10.1.1.1)$`)",
      "traefik.http.routers.api.service=api@internal",
      "traefik.http.routers.api.middlewares=basic-auth",
      "traefik.http.middlewares.basic-auth.basicauth.users=admin:xxx",
    ]
    # healthcheck using the appropriate port
    check {
      name     = "alive"
      type     = "http"
      port     = "admin"
      path     = "/ping"
      interval = "5s"
      timeout  = "2s"
    }
  }
}

Minimising downtime during updates

Traditionally, an ingress/load balancer/reverse proxy such as Traefik would run on all client nodes and route appropriately. In such a scenario, you need some way to distribute the incoming traffic between the client nodes (such as round-robin DNS, L4 load balancing on the router/provider, BGP/Anycast, etc.), and to know their health and stop sending requests their way if they’re unavailable. One of the potential scenarios to deal with gracefully are node draining or configuration updates.

To achieve that on Nomad with minimal downtime, we use system jobs, staggered updates that don’t restart all instances simultaneously with the uppdate stanza, and a combination of the kill_timeout task parameter and Traefik’s lifeCycle.requestAcceptGraceTimeout and lifeCycle.graceTimeOut to allow for requests to finish gracefully:

job "traefik" {
  region      = "global"
  datacenters = ["dc1"]
  type        = "system"
  # only one Traefik instance will be restarted at a time, with 1 minute delay between each such action
  # and automatic rollback to the previous version if the new one doesn't pass the health check
  update {
    max_parallel = 1
    stagger      = "1m"
    auto_revert  = true
  }
  # Nomad will wait for 30s after sending the kill signal to the task before forcefully shutting it down
  # by default it's 10s ( not enough to properly drain connections )
  # and the maximum is limited by the max_kill_timeout setting on the Nomad client ( default 30s)
  kill_timeout = "30s"
  group "traefik" {
    task "traefik" {
      driver = "docker"
      config {
        image = "traefik:2.4"
        # Traefik will continue serving new requests for 15s while failing its healthcheck,
        # giving time to downstream load balancers to take it out of their pool
        # and will *then* wait for 10s for existing requests to finish before shutting down,
        # before Nomad forcefully kills it 30s after initiating
        args = [
          "--entryPoints.http.address=:80",
          "--entryPoints.http.transport.lifeCycle.requestAcceptGraceTimeout=15s",
          "--entryPoints.http.transport.lifeCycle.graceTimeOut=10s",
          "--entryPoints.https.address=:443",
          "--entryPoints.https.transport.lifeCycle.requestAcceptGraceTimeout=15s",
          "--entryPoints.https.transport.lifeCycle.graceTimeOut=10s",
          ...
        ]
        ...
      }
    }

Sidecars

A sidecar is a task that runs alongside the main task doing something auxiliary like collecting logs, metrics, traces. I run two sidecars with my Traefik instances, one for logs and another for traces, due to the heavy load and specific configuration required for it.

Promtail for logs

Promtail is a logging agent from Grafana Labs that can get logs from a variety of sources, and then send them to Loki, a highly-scalable yet lightweight log aggregation system inspired by Prometheus. It’s drastically simpler and lighter than ElasticSearch (sometimes a default choice for centralised log management) to setup, and does a very good job, so that’s what I use (there’s a blog post in the works on that too).

Here’s an example task block for a Promtail sidecar that can collect Traefik’s logs, parse them with a regex, and add some labels (index fields for searching), with the associated lifecycle policy(sidecar, poststart), healthcheck, resource limitations:

task "promtail" {
  driver = "docker"
  config {
    image = "grafana/promtail:2.2.0"
    args = [
      "-config.file",
      "local/config.yaml",
      "-print-config-stderr",
    ]
    # the only port required is for the healthcheck
    ports = ["promtail_healthcheck"]
  }
  # the template with Promtail's YAML configuration file, configuring the files to scrape,
  # the Loki server to send the logs to (based on a registered Consul service, but it could be a fixed URL),
  # the regex to parse the Common Log Format (https://en.wikipedia.org/wiki/Common_Log_Format) used for access logs,
  # and the labels ( HTTP method and status code)
  template {
    data = <<EOH
      server:
        http_listen_port: 3000
        grpc_listen_port: 0
      positions:
        filename: /alloc/positions.yaml
      client:
        url: {{ range service "loki" }}{{ .Address }}:{{ .Port }}{{ end }}/loki/api/v1/push
      scrape_configs:
      - job_name: local
        static_configs:
        - targets:
            - localhost
          labels:
            job: traefik
            __path__: "/alloc/logs/traefik.std*.0"
        pipeline_stages:
          - regex:
              expression: '^(?P<remote_addr>[\w\.]+) - (?P<remote_user>[^ ]*) \[(?P<time_local>.*)\] "(?P<method>[^ ]*) (?P<request>[^ ]*) (?P<protocol>[^ ]*)" (P<status>[\d]+) (?        P<body_bytes_sent>[\d]+) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"?'
          - labels:
              method:
              status:
    EOH
    destination = "local/config.yaml"
    }
    resources {
      cpu    = 50
      memory = 256
    }
    # poststart, and sidecar=true, so Promtail will start *after* Traefik ( since it has nothing to do before Traefik isup and running),
    # and run for as long as it does
    lifecycle {
      hook    = "poststart"
      sidecar = true
    }
    # a service for a health check to determine the state of Promtail
    service {
      check {
        type     = "http"
        port     = "promtail_healthcheck"
        path     = "/ready"
        interval = "10s"
        timeout  = "2s"
      }
    }
}

OpenTelemetry collector for traces

OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data(metrics, logs, and traces) for analysis in order to understand your software’s performance and behaviour.

It’s the future standard for telemetry adopted by just about everyone in the branch (Datadog, AWS, Splunk, Google,Elastic, Honeycomb, Lighstep come to mind), which would allow easily switching backends while keeping the same instrumentation and collector. At the time of writing, only the tracing part is stable, logs and metrics are a work in progress in various stages (Prometheus / OpenMetrics-compatible metrics are in alpha and should be ready by the end of November 2021, while the logs spec is at the design stage), so personally I’d only use it for that part while waiting for the rest to be ready.

The OpenTelemetry(OTEL) collector is an agent you can run alongside tasks (as a sidecar) or on each Nomad client as a system job to collect traces (and one day metrics and logs).

Specifically in regards to Traefik, we can use the latter’s Jaeger tracing compatibility combined with the Jaeger thrift compact receiver in OTEL. Due to the potentially heavy traffic, I run a collector per Traefik, as a sidecar task, just in case.

Here’s an example task block with the OpenTelemetry collector sidecar which sends traces to a Jaeger server (it could be anything else supported by OTEL), with extra tags set via the JAEGER_TAGS env variable:

task "opentelemetry-agent" {
  driver = "docker"
  config {
    image = "otel/opentelemetry-collector-contrib:0.22.0"
    args = [
      "--config=local/otel/config.yaml",
    ]
    # pass the healthcheck and Jaeger Thrift compact ( UDP ) ports 
    ports = ["otel_health", "jaeger_thrift_compact"]
    # extra JAEGER_TAGS
    env {
      JAEGER_TAGS = "mytag=value,mytag2=test"
    }
  }
  # it's a good idea to limit the amount of resources available to the collector
  resources {
    cpu    = 128 # Mhz
    memory = 256 # MB
  }
  # prestart, and sidecar=true, so the OTEL collector will start *before* Traefik, and run for as long as it does
  lifecycle {
    hook    = "prestart"
    sidecar = true
  }
  # a service for a health check to determine the state of the OTEL collector
  service {
    check {
      name     = "health"
      type     = "http"
      port     = "otel_health"
      path     = "/"
      interval = "5s"
      timeout  = "2s"
    }
  }
  # the template with OTEL's YAML configuration file, defining the Jaeger Thrift compact (UDP) receiver,
  # Jaeger Thrift ( HTTP ) exporter to a centrally running Jaeger server registered with Consul
  # and some boilerplate ( healthcheck, batching, retries)
  template {
    data = <<EOH
         receivers:
           jaeger:
             protocols:
               thrift_compact:
         exporters:
           jaeger_thrift:
             url:  {{ range service "jaeger-api-thrift" }}{{ .Address }}:{{ .Port }}{{ end }}/api/traces
             timeout: 2s
           logging:
             loglevel: debug
         processors:
           batch:
           queued_retry:
         extensions:
           health_check:
         service:
           extensions: [health_check]
           pipelines:
             traces:
              receivers: [jaeger]
              processors: [batch]
              exporters: [jaeger_thrift]
    EOH
    destination = "local/otel/config.yaml"
  }
}

And to make use of it, Traefik needs the following extra arguments:

"--tracing.jaeger=true",
"--tracing.jaeger.localAgentHostPort=${NOMAD_ADDR_jaeger_thrift_compact}",

To send traces in the Jaeger thrift compact protocol to the IP and port(NOMAD_ADDR) of the jaeger_thrift_compact port.

Discuss

Hacker News

Twitter

Please support Web Monetization if you want less ads on the web

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sun, 07 Mar 2021 14:50:37 +0100

Note: Since it came up last time, I’m not in any way affiliated with any of the entities discussed in this article

What is Web Monetization

Web Monetization is a browser API that allows content creators to monetize content, and content consumers to pay for it, anonymously, without ads or any friction, like paywalls and subscriptions.

The basic concept is that a content creator signals they’re accepting payments, and the content consumer, via a browser extension, natively in the browser or by account linking, streams micropayments via the Interledger, which supports a wide variety of providers and currencies ( real and crypto). The point is, as a consumer you pay for what you consume directly ( instead of indirectly via tracking and ads ) and proportionally.

It’s based on an open specification which is still a work in progress, but fully functional.

Why you should participate in it

A lot of people don’t like ads - be it because they find them wasteful, too obnoxious, too insistent, too intrusive, too dumb, etc. Even more people dislike tracking ( the art of following user actions across the internet to serve them more “relevant” ads based on their interests ), because that centralises a lot of power and knowledge in a few big companies ( Google, Facebook, etc. ), and allows precise targeting, which has been used for some nasty things. If I may quote the EFF directly:

Over the years, the machinery of targeted advertising has frequently been used for exploitation, discrimination, and harm. The ability to target people based on ethnicity, religion, gender, age, or ability allows discriminatory ads for jobs, housing, and credit. Targeting based on credit history - or characteristics systematically associated with it - enables predatory ads for high-interest loans. Targeting based on demographics, location, and political affiliation helps purveyors of politically motivated disinformation and voter suppression.

A big part of the aforementioned people, and plenty of others, use ad blockers ( either as browser plugins like AdBlock Plus or Ublock (Origin), or DNS-level like PiHole and AdGuard Home ), which block ads, trackers and various analytics software, which aren’t necessarily the same as trackers - their main purpose is to analyse and aggregate users of the website, by country, language, browser, device type (mobile, PC), how they came upon the website, how much time they spent on what page, etc. compared to trackers’ main purpose, which is to analyse per-user (anonymised, but still) interests and behaviour and recommend related ads.

There are proposals by Google to supposedly improve upon the experience of tracking ( in part due to major browsers like Firefox blocking some of the technology enabling such tracking ), but there are major issues with them.

Nonetheless, for a lot of web content creators, ads remain the main way to make money of the content they produce, which could be just to compensate costs, as a reward for the time spent, or even potentially to allow transferring to full time content producing. And for a lot of content consumers, blocking ads removes the only way to “pay”, baring some direct donations scheme.

Ideally, there should be an easy way to directly pay for the content you consume. In some cases you can subscribe and pay a monthly fee, but that could never cover it all - nobody will pay for every small blog, recipe/repair tutorial/gardening tips website/YouTube channel. How much value do they bring to you ? How much would you pay for them? How would you know they’re worth it without using them first, and why would they allow you to use them for free, when most users would be one-shot coming from a search engine? And then there’s non-profit open/crowdsourced institutions like Wikipedia and the Internet Archive ( which is Web Monetized) whose principles make them incompatible with ads and paywalls/subscriptions and force them to rely on donations.

Enter Web Monetization - with almost no friction, content consumers pay-per-use ( time spent ), automatically, and more or less anonymously. If content creators desire, there can be exclusive features only for “paying” ( participating in Web Monetization ) users, or no ads for them.

Funnily enough, the concept is nothing new - the French Minitel ( more info on Wikipedia and a fascinating read by the IEEE) system, in a sense a predecessor of the Internet, which among other things, contained specifically paying per time spent on pages.

How it works

At the time of writing, there is only one Web Monetization provider - Coil.com, but the API is open so in theory anyone can join in and compete with them.

For content consumers, they provide browser extensions ( which are FOSS under the Apache 2.0 licence and available on GitHub) and subscriptions ($5 USD/month), which are spent at the rate of $0.36 USD an hour, providing for roughly 13 hours of content consumption. After $4.50 is spent, the rate is lowered to last until the end of the month. Said subscription also includes some perks, such as ad-free experiences on Imgur with Emerald and Cinnamon ( a subscription-based video content platform).

For content creators, you need an account with a virtual wallet providers that supports ILP ( the Interledger protocol used to make the magic happen, like Uphold and Gatehub ) and an HTML meta tag with the address of your wallet ( like so <meta name="monetization" content="$ilp.uphold.com/8Q6r7F3XjXEi"> ) in the code of your website. The latter can be added manually or via any of the existing plugins (there are ones for Hugo, Gatsby, Svelte, Wordpress). For some cases where that’s impossible, Coil allow you to connect with YouTube and Twitch directly.

Anonymity and privacy are provided by the way things work - that wallets are randomly generated IDs, Coil transfer the money, the wallet provider knows who the recipient is, the content creators knows how much they’ve received, but nobody knows everything. Here’s a post by Coil’s co-founder and CTO Ben Sharafian discussing wallet-side and sender-site (based on PrivacyPass) privacy techniques they and the wallet providers use to ensure everyone’s privacy is respected.

Limitations

Of course, that could never replace premium subscription services like the Financial Times, or even direct ones like Substack due to the limited amounts involved, and it isn’t trying to - the point is to help monetise smaller sized websites ( and Youtube/Twitch channels ) or open/crowdsourced/non-profit institutions by providing a decent alternatives to obnoxious ads (which millions of people block anyway ) and paywalls. For instance the Internet Archive and Hacker Noon are web monetized; and so is this and some other small-scale personal blogs. It’s more similar to Buy me a coffee, Patreon and GitHub Sponsors, only more automated, ubiquitous and with less effort.

However, it’s relatively recent and not very popular yet - according to the WebMotized Twitter bot there’s only 1482 web monetized websites so far. Some more participation is needed, on both sides ( content creators and consumers) to make it a viable alternative to ads and some paywalls. I already did my part

Conclusion & Summary

Are you a content consumer or provider, or both? Don’t like ads and privacy invading tracking? Participate in Web Monetization!

If I’ve convinced you, feel free to sign up for Coil’s membership to contribute and/or set up web monetization on your website. If not, check out the Discuss section below, I’d really like to hear why not and what did I miss.

Discuss

Hacker News: https://news.ycombinator.com/item?id=26375857

Twitter: https://twitter.com/sofixa/status/1368561842835099649

Why you should take a look at Nomad before jumping on Kubernetes

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sat, 27 Feb 2021 20:05:43 +0100

Pre-introduction

Recently I stumbled upon and then stumbled upon again on David Anderson’s interesting post about “new Kubernetes”, based on a discussion he had with Vallery Lancey about what they would do differently if they were rewriting Kubernetes from scratch. Interestingly, a decent part of the proposals for a “new Kubernetes” are design choices made by Hashicorp for Nomad, which is a pretty underrated orchestrator, and drastically simpler ( one of the main goals of said “new Kubernetes”).

Some people are aware that Docker Swarm kinda exists but is abandonware/on life support, and isn’t really recommended anymore, but it still comes up in discussions due to how easy it is to use. For most, that leaves Kubernetes as the only “serious” option, but it is a very complex piece of software, with a lot of moving parts, which isn’t actually required or need in most cases.

This inspired me to write a series on Nomad, what it is, why it’s great, where it’s lacking and how to use it.

Introduction - what is Nomad and why it’s great

Hashicorp’s Nomad is a simple to run and maintain, yet very flexible task scheduler/orchestrator. It relies on plugins for execution, autoscaling and other features, and can run pretty much anything via its task drivers - Docker, contairnerd, LXC, rkt, podman, Java, fork/exec, QEMU, firecracker, FreeBSD jails.

It comes in the form of a single binary, run in two modes (server, in groups of 3 or 5, which make scheduling decisions and host the APIs and configuration, and an unlimited number of workers which actually run whatever it is you want to run), and can be automatically clustered via Consul. The configuration ( both for jobs and of Nomad itself) is in HCL (I’ll get into more detail about how great that is a bit later) or JSON (mainly for when the jobs are submitted by machines/scripts/tooling and not humans). Multiple clusters can be connected via multi-region federation for sharing ACLs and for API forwarding ( you can submit a job or request logs to any server for any region and it will be forwarded to the appropriate server). Deployments can be complex out of the box ( rolling, canary, blue/green), and everything is version controlled and rollbackable.

Like most HashiCorp tools, it’s “open core”, meaning that the majority of features are available in an open source version, and some more advanced/enterprise-y ones ( in Nomad’s case, multi-region/cluster deployments - deploying something simultaneously to multiple separate clusters, policy as code with Sentinel and similar ) require upgrading to Nomad Enterprise.

Primitives

job is a declarative file which contains groups of tasks, each task being a container/binary/anything run by an exec driver
system jobs (run on all client nodes, equivalent to Kubernetes DaemonSets, for monitoring/logging agents/load balancers)
periodic jobs (equivalent to cronjobs)
service, which registers as a Consul service and is thus discoverable ( via API or DNS)
deployment, each version of a job, they’re tracked and can be rollbacked to
allocation, each instance of a task ( group ) on a node
namespace, a logical unit to organise jobs in and ACLs around

Jobs

Example of a very basic job that runs a Docker container (jaegertracing/all-in-one:1.21), with limits of 1000Mhz of CPU and 1024MB of RAM, and registers the service with Consul:

job "jaeger" {
        type = "service"
        group "api" {
            task "jaeger" {
                driver = "docker"
                config { 
                  image = "jaegertracing/all-in-one:1.21"
                }
                resources {
                  cpu = 1000
                  memory = 1024
                }
                service {
                  name = "jaeger-query"
                }
            }
        }            
}

Note that this is a very basic job, there are no healthchecks, no persistent storage, no extra configuration, no update strategy, no autoscaling, no exposed ports.

Deployment history and rollback

Nomad tracks each job’s full definitions and deployment history, and allows you to easily rollback and compare them, via the UI, CLI or API, e.g.:

# List the versions of the job named "opentelemetry-collector"
$ nomad job history opentelemetry-collector
Version     = 1
Stable      = false
Submit Date = 2021-01-08T21:30:30+01:00

Version     = 0
Stable      = true
Submit Date = 2021-01-08T21:29:48+01:00

# Check the difference between versions
$ nomad job history -p opentelemetry-collector
Version     = 1
Stable      = false
Submit Date = 2021-01-08T21:30:30+01:00
Diff        =
+/- Job: "opentelemetry-collector"
+/- Task Group: "opentelemetry-collector"
  +/- Task: "opentelemetry-collector"
    +/- Config {
          args[0]:  "--config=local/otel/config.yaml"
      +/- image:    "otel/opentelemetry-collector-contrib:0.15.0" => "otel/opentelemetry-collector-contrib:0.16.0"
          ports[0]: "health"
          ports[1]: "jaeger_thrift_compact"
        }

Version     = 0
Stable      = true
Submit Date = 2021-01-08T21:29:48+01:00

# Revert job "opentelemetry-collector" to version 0
$ nomad job revert opentelemetry-collector 0

State tracking and job planning

Nomad keeps the desired state and its history, and with nomad job plan, similar to terraform plan, allows us to preview what will change upon applying a new job file. There’s also a feature to verify nothing has changed between the plan and run (equivalent to terraform apply with a plan file) with the -check-index flag:

$ nomad job plan otel.nomad
+/- Job: "otel"
+/- Task Group: "opentelemetry" (1 create/destroy update)
  +/- Task: "opentelemetry-collector" (forces create/destroy update)
    +/- Config {
          args[0]:  "--config=local/otel/config.yaml"
      +/- image:    "otel/opentelemetry-collector-contrib:0.15.0" => "otel/opentelemetry-collector-contrib:0.20.0"
          ports[0]: "health"
          ports[1]: "jaeger_thrift_compact"
        }
Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 413
To submit the job with version verification run:

nomad job run -check-index 413 otel.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

Overall, it’s a very useful feature, especially when collaborating, locally or via CI/CD.

Checking the status and logs

To check the status of a job, there are a few commands under nomad job and nomad alloc

$ nomad job status otel
ID            = otel
Name          = otel
Submit Date   = 2021-02-27T20:41:29+01:00
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
otel      0       0         1        0       0         0

Latest Deployment
ID          = ea533b6f
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
otel      1        1       1        0          2021-02-27T20:51:45+01:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
89031cfd  d3cbeb7e  otel      0        run      running  20s ago  4s ago

# logs are at the allocation level ( similar to Kubernetes, where they're at the container level), so we get them with the alloc id
$ nomad alloc logs 89031cfd
[...]

lifecycle and sidecars

Nomad allows defining the lifecycle of tasks in task groups, and their status, with the lifecycle stanza. We can have prestart ( for initialisation ), poststart ( companion, for proxying (aka ambassador and adapter pattern in Kubernetes )) or poststop for clean up, and via the sidecar bool we define whether or not it should run as long as the main task(s), e.g.:

  task "init" {
    lifecycle {
      hook = "prestart"
      sidecar = false
    }
    driver = "docker"
    config {
      image = "alpine/httpie"
      command = "http"
      args = [
        "POST",
        "https://some-internal-service-for-provisioning-stuff.local/v1/new",
        "job_id='${NOMAD_JOB_ID}!'"
      ]
    }
  }

  task "fluentd" {
    lifecycle {
      hook = "poststart" # should start after the main task
      sidecar = true # should run as long as the main task does, and be restarted if it fails
    }
    driver = "docker"
    config {
      image = "fluentd/fluentd"
    }
    ...
  }

  task "main-app" {
    ...
  }

  task "cleanup" {
    lifecycle {
      hook = "poststop"
    }
    driver = "docker"
    config {
      image = "alpine"
      command = "rm -rf"
      args = [
        "/var/lib/volume-with-super-secret-data"
      
    }
  }

ACL/RBAC

ACL ( access-control list ), or RBAC ( role-based access control ) as it’s known in Kubernetes, allow defining who can do what, so that not everyone with network access can have full administrator privileges and run/stop whatever. Nomad’s ACL system is pretty similar to Consul and Vault’s, and uses JSON ( mostly for non-humans ) or HCL to define policies with rules, which describe what action is allowed on what object.

# a basic policy which allows the predefined read policy with read-only access to list and read:
# job, volume and scaling details, and extra capabilities for job creation and log access within the default namespace
namespace "default" {
  policy = "read"
  capabilities = ["submit-job","dispatch-job","read-logs"]
}

Assignment of policies is done only via the CLI, unlike Kubernetes where that happens in YAML, as does policy management:

# create/update the policy within Nomad
nomad acl policy apply -description "Application Developer policy" my-policy my-policy.hcl
nomad acl token create -name="Test token" -policy=my-policy -type=client
Accessor ID  = 4e3c1ac7-52d0-6c68-94a2-5e75f17e657e
Secret ID    = 0be3c623-cc90-3645-c29d-5f0629084f68
Name         = Test token
Type         = client
Global       = false
Policies     = [my-policy]
Create Time  = 2021-02-10 18:41:53.851133 +0000 UTC
Create Index = 15
Modify Index = 15

Just for comparison, the (in my opinion weirdly verbose to write due to YAML ) syntax for the equivalent in Kubernetes:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: test
rules:
- apiGroups: [""] 
  resources: ["pods", "services"] # a Nomad job contains both the pod equivalents ( task groups ) and services
  verbs: ["get", "watch", "list", "logs", "create", "update", "patch"]

And with this token, either as an env variable ( NOMAD_TOKEN) or flag (-token) for the CLI or HTTP header ( X-Nomad-Token) for the API, we can do things.

ACL policies and tokens are optionally shared with federated clusters for simplified management. We can also have ephemeral tokens via Vault’s Nomad Secret Backend, which generates single/short-use tokens with specific policies, but there’s no implicit or explicit job role equivalent to Kubernetes’ Service Accounts, one has to pass through Vault for that ( assign a Vault policy to the job).

Simplicity

Nomad is easy to install, maintain, update and scale, even with “advanced” features such as linking multiple clusters across datacenters/regions.

Running locally

Running Nomad locally for development/testing is just a matter of downloading the binary and running nomad agent -dev ( significantly easier than microk8s or minikube or kind), and the same goes for Consul and Vault ( which you might need to replicate the production Nomad environment locally).

Upgrading

Upgrading Nomad servers is just a matter of replacing the binary and restarting the service. There are detailed upgrade guides which list the main changes and potential breaking ones/things to take care of, but it’s relatively rare ( and will be even less so since it’s already on 1.0+). Clients need to be drained first before upgrading ( for which there’s also a detailed guide ), and the behaviour of jobs during that operation can be tweaked via the migrate stanza.

Monitoring

Nomad collects extensive metrics on itself and everything it runs within it, which can be send to compatible agents ( statsD, Datadog ) or scraped by a Prometheus/OpenMetrics-compatible scraper, and they even have a guide on setting up Prometheus to monitor and alert.

Cluster/multi-cluster joining

Forming/joining a cluster can be done manually via nomad server join, via the server_join configuration block( which can use static IPs/DNS, or dynamic cloud auto-join ( based on cloud provider tags or similar)), or via Consul. Federation is done by joining a server in another cluster via its WAN IP/DNS and port:

nomad server join 172.31.26.138:4648

Integrations

Nomad has an extensive API (which includes cool recent additions like the event stream, allowing the creation integrations and tools that act based on what happens in your Nomad cluster), and it integrates well with a bunch of other tools, some of them from Hashicorp, nicely complementing Nomad to rival the features of the more feature-rich Kubernetes and its ecosystem.

Things like Service Discovery and K/V storage (Services and ConfigMaps respectively) and secret storage (Secrets) and even features part of the larger Kubernetes ecosystem like service mesh are delegated to other, existing, well-used and battle tested parts of the HashiStack (Consul and Vault), which makes sense - it follows the Unix philosophy “do one thing and do it well”, and makes things easier for Hashicorp and its users - Consul and Vault are already heavily used, stable and very popular in their respective niches, and the decoupling and separation of concerns allows us to use them outside of Nomad (e.g. you can use Vault, hosted or not on Nomad, for secret storage/dynamic secrets/auth for apps running in Nomad or anywhere else; maybe you even already have a running Vault cluster, and you can just connect your Nomad cluster to it and you have the same storage for all your secrets, regardles of where they are used from).

The big downside is that if you just want to run a Nomad cluster, you kinda need two other tools to install, maintain, stay up to date on, etc. but seeing that all three are similar ( single binary, Raft consensus algorithm, great documentation including very detailed upgrade guides ), it’s not that complex to achieve.

There are also integrations, via Consul, with third-party tools such as Traefik and Fabio for automatic Reverse proxy/Load Balancing.

Integrated Templating

From Consul or Vault

One way that you can make use of Vault and Consul in Nomad job files is via the template stanza (configuration block) in nomad jobs, which allows the creation of files or environment variables based on templates, and is based Hashicorp’s venerable consul-template.

# creating a YAML configuration file from Consul K/V and services, Nomad metadata and Vault secrets:
template {
  data = <<EOH
  ---
    bind_port:   {{ env "NOMAD_PORT_db" }}  # the port "db"
    scratch_dir: {{ env "NOMAD_TASK_DIR" }} # the task folder
    node_id:     {{ env "node.unique.id" }} # the unique ID of the Nomad node
    service_key: {{ key "service/my-key" }} # populated by service/my-key from Consul K/V store
    loki_addr:   {{ range service "loki" }}{{ .Address }}:{{ .Port }}{{ end }} # populated by the IP and port of the service named "loki" in Consul's Service catalogue
    some_secret: {{ with secret \"secret/data/my-secret\"  }}{{.Data.data.value}} {{end}}" # populated by the secret/my-secret secret in Vault
  EOH
  destination = "local/file.yml"
}
# doing the same but instead of writing in a YAML file, export the values as env variables:
template {
 ...
  destination = "secrets/file.env" # secrets is a special, per-task folder that you can use to store secrets and isn't browseable via API/CLI/UI. More on that below
  env         = true
}

HCLv2

HCLv2 greatly improves upon HCLv1 with more dynamism and better type management. One can use variables, for loops, include files, etc. Some examples with common use cases:

Using an “env” variable to determine if we’re in local dev or production, and set variables accordingly; and using a list variable with the default arguments in all cases, default ones for local/production, additional arguments, and merging them.

Note: local variables are local to the file, and variable variables are more akin to function parameters ( they can be passed via file, env variable, with defaults, etc.)

variable "env" {
  default = "local"
}
variable "args" {
    type = list(string)
    default = []
}
locals {
  datacenter = var.env == "local" ? "dc1" : "eu-west-1"
  count = var.env == "local" ? 1 : 5
  default_args = ["--something"]
  local_args = ["--verbose"]
  prod_args = ["--production-mode 1", "--secure"]
  args = var.env == "local" ? concat(local.default_args, local.local_args, var.args) : concat(local.default_args, local.prod_args, var.args)
}

job "test" {
  ...
  datacenters = [local.datacenter] # Nomad expects a list here, hence the []
  group "test"{
    count = local.count
    task "test" {
      driver = "docker"
      config {
        image = "registry.test.com:0.1"
        args = [for a in var.args: a] # Nomad expects a list here, hence the []
      }
  }
}

For comparison, either of those are impossible in Kubernetes-land without third-party tooling - Helm, jsonnet, tanka, ytt, etc. for basic templating/logic, which come with a lot of overhead on top of YAML, and things like Vault Agent for sidecar secret injection or the Secrets Store CSI driver, which allows reading secrets as filesystem objects via a CSI driver. Of course, Kubernetes ConfigMaps and Secrets exist for the latter issue, but there are several limitations compared to the Nomad way of doing things:

Kubernetes Secrets aren’t really secret, they’re base64 encoded strings stored in etcd ( so you’re relying on it being encrypted for “security”)
Kubernetes ConfigMaps and Secrets are fully static, you can’t throw in any dynamism ( an if, a for loop, the IP/port of a service, etc.)
both are disconnected from deployments - if you update a ConfigMap used by a Pod, the latter needs to be restarted for the changes to be recongized ( compared to Nomad’s change_mode which allows you to control how and if the task is restarted/reloaded upon configuration change)
you can’t mix and match, like having a single file with secrets, a service address, K/V config, etc. ( unless you do somethig manually with an initContainer and bash)
you can’t have dynamic secrets ( like AWS or database credentials with a limited time to live) without external tooling taking care of renews and restarts

Deployment primitives

Nomad supports natively a few deployment methods - rolling updates ( each instance is updated X at a time ), canary ( a small part of the running instances are updated, they’re monitored for anomalies/bugs/errors/etc., and if everything is fine over some period of time, the rest are updated as well) and blue/green (two equally-sized environments are running, blue and green one, but only one is serving traffic; the other one gets the updated version in full, tests are run, and once all is good, traffic is switched to it ), and can do automated rollbacks.

All of that, including timeouts, counts, etc. is configured via the update stanza :

# generic configuration
count = 3 # 3 allocations 
update {
  max_parallel = 1 # number of instances to upgrade in parallel
  health_check = "checks" # what determines the state of the allocation - it could be health *checks*, *state* (if all tasks are running) or *manual* for human/monitoring/etc. via the API
}

# canary
count = 3 # 3 allocations 
update {
  max_parallel = 1
  canary = 1 # 1 canary allocation
}

# blue/green
count = 3 # 3 allocations 
update {
  max_parallel = 3 
  canary = 3 # canary=max_parallel=count => Green env
}

Canaries or blue/green deployments can be promoted via the CLI, web UI or API ( e.g. automatically based on metrics/logs/traces data). Hashicorp have a detailed-ish guide with examples on their HashiCorp Learn platform.

Networking

Networking on Nomad is, at its base, very simple and basic, at least when it comes to containers. By default each task gets an IP (as provided by Docker/etc.) in bridge mode, in a shared namespace with the other tasks in its group ( to allow sidecar proxies ). Ports can be exposed by using host networking, dynamic or static port forwarding and accessed directly or via sidecar proxies, or via a CNI plugin if advanced features are needed.

The more or less standard/recommended behaviour is inspired by Google’s Borg - any scheduled task group gets a random port from the host if it requires it, and service discovery is responsible for linking and discovering services across the dynamic ports. In Nomad’s case, that’s done via Consul ( either DNS, API or in Nomad with templating). It’s also possible to have the containers be IPv6-only, and advertise that to Consul.

More advanced options include scheduling ports only on specific networks and only exposing tasks via Consul Connect/Service Mesh, which can be used with Kubernetes, Nomad, regular virtual or physical machines.

Storage

By default, allocations are ephemeral and with no persistence. Each one gets a shared alloc folder, to which all tasks within the allocation can read and write, with 3 standard folders inside - data for ephemeral_disk-declared data ( more on that later), logs which contains all tasks’ logs, and tmp, which used as scratch space by task drivers. Besides that, each tasks gets its own folder ( on the same level as alloc ), with local ( to be used at will, for e.g. configuration files), secrets (for secrets, and is therefore unavailable for browsing via the UI, API or nomad alloc fs command, unlike the others) and tmp subfolders. Tasks can’t access other tasks’ folders, so cross-task things ( e.g. if you’re using a sidecar log shipper for custom logs) should use the alloc folder.

There’s an ephemeral_disk stanza, which allows for somewhat persistent storage:

job "example" {
  group "example" {
    ephemeral_disk {
      migrate = true
      size    = 500 # in MB
      sticky  = true
    }
  }
}

This will create a 500MB “disk” ( it isn’t an enforced limit, it’s more of a capacity planning thing and it’s used for scheduling decisions) that should be migrated to another node if the one it’s currently on gets drained ( migrate = true), and Nomad will try to place updated allocations on the same node and move the ephemeral_disk ( corresponding to the alloc/data on the allocation level and local on the task level folders ) to it. This is all best-effort, with zero guarantees and of course, if the node fails, the data will be lost.

Actual persistence is achieved in two main ways:

Host volumes ( folders on the hosts mounted inside the allocations ), redundant via some external to the Nomad cluster tooling ( NFS, GlusterFS, Ceph, Portworx, etc.)
CSI volumes ( more on that below )

Plugins

A lot of the key components of Nomad are delegated to plugins ( some included and distributed with Nomad, some external, and all with an open spec, so anyone can contribute custom plugins), such as autoscaling, task drivers, advanced networking (CNI) and storage (CSI) to enable greater flexibility.

Task drivers

As mentioned before, Nomad can schedule and run all sorts of different things ( Docker, contairnerd, LXC, rkt, podman, Java, fork/exec, QEMU, firecracker, FreeBSD jails, Windows IIS). A few of them are maintained by the community, and there’s an official guide on writing custom ones, which is infinitely more flexible than Kubernetes, which is fixed on Docker/contairnerd ( even if projects such as kubevirt exist).

CSI and CNI

In a bid to make Nomad more compatible with cloud-native software, it implements the CSI and CNI specs for storage and networking respectively, meaning that you can use plugins following those specs ( popularised by Kubernetes and hosted by the CNCF) with Nomad. In theory that means that Nomad can tap in to a vast ecosystem of existing tooling and plugins and one can ( more or less ) easily move from Kubernetes to Nomad and keep the same third-party helpers/tooling/infrastructure - external storage system ( AWS EBS/EFS or NetApp or Ceph, etc.), network overlay ( Weave, Flannel, etc.), but in practice there are some limitations, most notably the fact that not the whole CSI spec is implemented. Nonetheless, that’s a great direction and the HashiCorp team is working on improving.

Autoscaling

Nomad supports autoscaling of various types, managed by external autoscalers, and Hashicorp provide the Nomad Autoscaler project, which supports horizontal application ( add more instances of task group) and cluster (add more Nomad workers) scaling in the open source version, and dynamic application sizing ( add more resources to a task based on its actual real life usage) in Nomad Enterprise, but anyone can implement a custom autoscaler via Nomad’s API.

Nomad Autoscaler supports plugins for the following components:

APM ( what to query for application metrics to use for decisions ) - Prometheus, Datadog or a limited integrated Nomad one
strategy ( based on the current and desired state, decide what to do ) - target value ( what the value of a metric should be) and the Enterprise-only dynamic application sizing average strategy, max strategy, percentile strategy
target ( what to scale ) - Nomad task group, dynamic application sizing for horizontal application scaling, and AWS Autoscaling Group, Azure Virtual Machine Scale Set for cluster autoscaling ( a GCP Managed Instance group plugin is in the works ).

And as per usual, anyone can write custom plugins for specific use cases.

Advantages

Easy, lightweight and low maintenance

Nomad is a single binary, with relatively few moving components ( optionally, but more often than not ) required - most notably Consul, Vault and potentially some plugins. Upgrading, adding extra nodes etc, is easy. Making the jump from test to production is also straightforward and consists of more or less the following:

have 3 or 5 servers with Nomad and Consul
enable ACLs
enable cross-node communication encryption
add monitoring (via e.g. Prometheus or any Prometheus-compatible tool)
enable Autopilot, which automates some operations-related tasks

Roughly speaking, you can be up and running from zero to pre-production within a day or two, and maintenance is minimal outside of updates, which rarely introduce breaking changes.

Scale

On top of all that, Nomad can also scale well. Hashicorp did a few experiments with that, first with 1 million containers back in 2016, and in December 2020, for the 1.0 release, they ran 2 million containers across 6,100 AWS Spot instances, with just 3 Nomad servers (AWS i3.16xlarges, but still) orchestrating all of that.

HCL

No YAML

As I mentioned, Nomad eschews the wildly popular YAML and uses Hashicorp’s own HCL ( Hashicorp Configuration Language ), which was created because none of JSON or YAML were good enough for configuration purposes, and that’s still as true as it was a few years ago. JSON is overly verbose and not really fit for configuration ( e.g. no comments - I know that JSON5 exists, but isn’t widely supported, and the official implementations are few, and mostly archived ), and YAML is just terrible beyond 10-20 lines or 3-4 levels. It’s so bad, there’s an entire site dedicated on the subject - https://noyaml.com/, which i won’t quote in it’s entirety, but let’s just say that using spaces for logic is a bad idea and leads to weird, undebuggable errors when copy/pasting and templating via external tools ( such as Helm or Jinja - anyone seen Ansible’s “error at line X, but it probably isn’t there” message? ).

Furthermore, both are limited in that templating/logic is impossible/hacky/dependent on third-party tools ( like Helm, jsonnet/tanka, kustomize or ytt), while HCL can do ( in some cases since not that long ago, but nonetheless ) basic logic such as if/else, for loops, dynamic blocks, variable substitution, data types ( maps, lists, objects ), strict typing thereof, file including and plenty of other goodies. You can have basic logic in your configuration if you need it, you can comment it, you can copy/paste at will without fear of breaking anything, you can split your configuration into multiple files ( e.g. sharing a sidecar logging/tracing/etc. agent). It’s actually pretty great! There are even (sadly third-party) tools to fmt ( in Go, and some other Hashicorp tools, like terraform, there’s format fmt subcommand to format your code according to a basic set of rules, making code more easily readable and always looking the same ).

Self-contained jobs

A very nice thing about nomad job files is that they contain everything related to the job - all the groups and tasks, sidecars, resultant services, configuration, etc. HCL remains legible even at length ( especially within an IDE or text editor with enough features to be able to highlight corresponding curly brackets {} ), but of course if one wants, there’s a file function to import file contents, so some parts can be split into different files if required/preferred,

CSI, CNI

Hashicorp opting for widely used standards where appropriate and practical is a big advantage, allowing Nomad users to profit from developments done in the larger ecosystem. There are caveats, however, and some improvements are needed in Nomad ( e.g. a more complete implementation of the CSI spec ); nonetheless, it’s a great direction and the bases are already there.

Extensibility

The fact that a lot of the main functionalities ( task drivers, device drivers, autoscaling, storage and networking ) are offloaded to plugins is absolutely great. It allows you to move at your own pace when something gets deprecated ( e.g. if you were using rkt and hadn’t had the time to move to something else yet, the fact that you can still run the rkt task driver, even if it’s deprecated and no longer a part of Nomad, is probably reassuring) or add your custom use cases ( e.g. the Firecracker and IIS task drivers were added by community users that needed them). And at the same time, the basic ones are baked in and distributed with the Nomad binary, so you don’t need to install anything to do the “standard” things ( Docker, raw fork/exec, etc.). If you want to make Nomad autoscale on your Proxmox or Nutanix AHV cluster, you can develop it yourself and aren’t stuck waiting for Hashicorp to do it.

Disadvantages

Ecosystem

The Kubernetes ecosystem is massive. There are entire companies, tools and whole niches being built around it ( ArgoCD, Rook, Istio, etc. etc. etc.). In some cases tools exist only because Kubernetes is itself so complex - Helm, Kustomize, there are a bunch of web UIs and IDEs ( Octant, Kubevious, Lens, etc.), specialised tooling to get an overview into the state and security of your Kubernetes cluster ( Sonobuoy, kube-hunter, kube-bench, armosec, pixie). Furthermore, there are literally hundreds of operators that allow abstracting the running of complex software within Kubernetes.

There isn’t a lot of software that even comes close, and Nomad doesn’t. There’s a decent amount of tools it works with, and CNI and CSI support helps it tap into the wider Kubernetes/cloud native ecosystem, but it still isn’t as popular. In practice this means that for running some things, you’re on your own - e.g. if you want to use PostgreSQL, Cassandra, Prometheus, CockroachDB or anything of the like, you can’t just use existing operator with a few CRDs, you have to write the full Nomad configuration for it to work.

Managed service

At the time of writing, there is no Nomad managed service akin to Google Kubernetes Engine, Amazon Elastic Kubernetes Service, Digital Ocean Kubernetes, etc.; if you want to use Nomad, you have to run it yourself ( maintenance is minimal, but still, in some cases offloading the maintenance to a cloud provider is worth the cost even for easy-to-maintain software). Hashicorp have their Hashicorp Cloud Platform, which uses Nomad underneath for their managed Vault and Consul offerings; and Armon Dadgar, co-founder of Hashicorp, said at Hashiconf US 2020 that they are planning on offering a HCP Nomad. Until that arrives ( or AWS decide to make a Nomad managed service), you’re on your own.

Enterprise

Hashicorp are a for-profit company with open core software distribution model. That presents some risks, most notably that some day they might decide to take Nomad ( or Terraform or Consul for that matter) in a direction you don’t like, or, to the extreme, shift features from the open source Nomad version to the Enterprise one - like when InfluxData removed clustering from InfluxDB OSS and moved it to InfluxDB Enterprise. I personally consider that highly unlikely ( because doing so would destroy most of the goodwill they have within the community, make conversion to Enterpise versions harder, etc. etc.) and they genuinely seem like nice people, but it’s still a possibility. In any case, there’s an inherent conflict of interest - Hashicorp wouldn’t allow features competing with their Enterprise offering to be added to Nomad OSS, because that would lower it’s value proposition.

Policy management

One of the main things I’d consider “missing” in Nomad OSS is policy management ( as in Open Policy Agent, Gatekeeper and company - roughly equivalent to Kubernetes admission controllers) - it’s done only via Sentinel which is IMHO a fine policy as code DSL, but it’s only available in Nomad Enterprise. One can work around this with conftest’s HCLv2 parser in CI/CD, coupled with ACLs forcing users to only submit jobs via “GitOps”; but it’s certainly not as powerful (e.g. you can’t easily use runtime data from Nomad).

vs “New Kubernetes”

As I started with my inspiration, “new kubernetes”, it’s fitting to end comparing Nomad to it.

"[…] my guide star is Go. If Kubernetes is C++, what would the Go of orchestration systems look like? Aggressively simple, opinionated, grown slowly and warily, and you can learn it in under a week and get on with what you were actually trying to accomplish"

~ David Anderson

Nomad mostly fits that bill - it’s simple and opinionated. In terms of the features proposed, it does by default:

mutability
version control
better-controlled deployments
simplified networking with IPv6 or with task groups getting randomly allocated ports by default
virtual machines (and FreeBSD Jails, and LXC, and etc.)
alternative to CSI storage although it’s nothing special, just allowing you to mount host folders (same as Kubernetes)

And it allows, with some work, under some conditions and with some limitations:

highly distributed clusters - more specifically, Nomad’s cluster federation is decently featured and stable ( but deploying jobs across multiple regions simultaneously is only available in Nomad Enterprise), and can probably be done over WAN via Consul Mesh/Terminating/Ingress gateways ( also useful for inter-service traffic and failover). Nevertheless, Nomad isn’t designed for clients to stay disconnected from the servers for a long time, and Nomad will reschedule allocations on “lost” nodes after a certain ( tunable ) time has elapsed, so for “edge” scenarios a cluster per location is more appropriate - as CloudFlare do

However, Nomad doesn’t have the concept of control loops, which drastically simplifies it and allocation debugging, but also limits how far you can integrate with it from within to control its behaviour.

Conclusion

Overall, Nomad is a pretty great, simple, opinionated and flexible orchestrator. It has some advantages in features over Kubernetes ( stable cluster federation, task drivers, version control, integrated deployment logic and templating, no YAML), but lacks some other features (CRD), polish ( full CSI spec ) and ecosystem depth.

Some scenarios where Nomad would be more appropriate than Kubernetes:

on-prem deployments ( even with RKE/VMware Tanzu/kubeadm etc., running Nomad on bare metal/virtual machines is drastically easier )
multi-DC/region presence ( due to cluster federation )
not only container needs (QEMU, Java, raw exec, FreeBSD jails, IIS, etc.)
small(-er) sized teams
limited amount of things to deploy ( e.g. using Kubernetes for Django + Redis + PostgreSQL is certainly overkill; Nomad’s overhead is acceptable for such a stack)
following the KISS ( keep it simple, stupid ) principle

Some scenarios where it might not be more appropriate:

it might be useful to tap into the wider Kubernetes ecosystem, e.g. operators - if you want to run PostgreSQL, Redis, Cassandra, ElasticSearch, Kafka with limited human resources, it might be easier to do so via Kubernetes Operators ( whether or not such operational complexity, even abstracted, is worth it with a limited team, is an entirely different discussion)
if you’re, by choice or not, limited to a single public cloud provider, and can outsource the management of Kubernetes (or even a higher level product, like AWS’ Fargate or GCP’s Google Cloud Run/AppEngine), like with GKE and especially Autopilot, there’s little left to you; ( even then, there’s still some complexity that might be unneccessary - e.g. dealing with AWS EKS to run Wordpress might be overkill)

Coming up next

I intend to write a few more articles on Nomad, mostly to tell my recent experiences with specific things around it, like log management ( how to run Loki on top of Nomad, ship logs to it), tracing ( Jaeger, integrate with Traefik/Consul, etc.).

Discuss

Hacker News

Twitter

Comparing Kubernetes managed services across Digital Ocean, Scaleway, OVHCloud and Linode

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sun, 14 Jun 2020 18:26:15 +0200

This article was updated:

15/06 to add an SLA & Support section (thanks T from the DevOps’ish Telegram group for the great idea); add the just released option to deploy Traefik v2 with Kapsule; explicitly mention that instance pricing is hourly

19/06 to update Kapsule’s latest available version (v1.18.4, less than 22 hours after public release!)

21/07 to mention Scaleway’s price hike and newer Kapsule versions

Introduction

Everybody knows about the “big three” cloud providers - Amazon Web Services, Google Cloud Platform and Microsoft Azure. However, there are a few other, less known competitors, which provide fewer services, but have some advantages (most notably in terms of cost and complexity) and are perfectly viable choices, depending on size and requirements. In recent years, with the advent of Kubernetes, the features gap is being bridged because with a managed k8s service and some related ones (load balancers, block and object storage, managed databases) one can go a long way before needing the full might of AWS’ 175 (at time of writing) services. And thanks to the ubiquity of Kubernetes, migrating to a new, more feature complete cloud provider’s isn’t that complex ( a lot less than if you have custom orchestration and cloud-specific deployment logic), if you do someday outgrow your current choice.

Especially when starting (be it in a hobby capacity, a side project, or a start-up), relatively simple configuration and pricing can be a huge advantage. Everybody could do with not needing schematics like this one by Corey Quinn:

In this article I’d like to compare how some of those smaller, more developer-oriented providers fare in terms of managed Kubernetes and associated services. I’ll also throw in a brief comparison to AWS or GCP from time to time to put things in perspective.

Note: When I say “managed Kubernetes service”, I mean a service where the master nodes (Kubernetes control plane) are hosted and managed by the provider. It’s not about Container as a service services like AWS Fargate, Google Cloud Run or Scaleway Serverless where Kubernetes might be running behind the scenes, but you only deal with it via an abstraction layer (like Knative) at the container level.

Criteria

Kubernetes features
complexity
tooling
ecosystem
cost

Competitors

I’ll compare Digital Ocean, Scaleway, OVHCloud and Linode, some of the most popular traditional VPS providers, which have also branched out into the managed Kubernetes space, keeping with their developer-oriented, low cost, low complexity model. Why only them? They were the only ones I could find within the first two pages of Google search that weren’t enterprise-only (IONOS) or too similar in model, complexity and pricing to AWS/GCP/Azure (IBM, Oracle). Civo was also pointed out to me on the DevOps’ish Telegram group, but they’re only in private beta.

Brief history and overview

Digital Ocean

Founded in 2011 as a VPS provider (“droplets” in DO parlance), their product range has been extended to add managed databases (PostgreSQL, MySQL and Redis), managed Kubernetes (main focus of this article), load balancers and object storage (Spaces). They have mostly a good reputation and are easy to use. Currently they have 8 locations around the world (New York and San Francisco in the USA, Amsterdam in the Netherlands, Singapore, London in the UK, Frankfurt in Germany, Toronto in Canada and Bangalore in India).

Scaleway

Formerly a sub-brand of the Online.net (founded in 1999) part of the French Iliad group which offered custom-made ARM servers, since last year it’s the main brand for their public cloud offerings, which includes bare metal servers, traditional VPSes (x86), object storage, archival storage, managed Kubernetes (Kapsule, which I’ve explored before on my blog, serverless offerings (Functions and Containers as a service), IoT/message queue, managed databases (PostgreSQL and soon MySQL), load balancers and soon AI inference. Currently they have 2 locations (Paris, France and Amsterdam, Netherlands), with a third one in Poland coming sometime in 2020, and others planned for “Asia and Latin America” “by end-2020”.

OVHCloud

Founded in 1999 and formerly known as OVH, they started as a managed hosting and VPS provider, and have expanded in telecoms, managed OpenStack, VMware vSphere and Kubernetes, email hosting, managed databases (MySQL, MariaDB, PostgreSQL, Redis), load balancers, Anti-DDoS, Data and Analytics platform. It’s by far the biggest of the bunch, in terms of services, clients and locations (Roubaix, Strasbourg, Gravelines, Paris in France, Franfkurt in Germany, Warsaw in Poland, London in the UK, Beauharnois in Canada, Vint Hill in the USA, Singapore, Sydney in Australia).

Linode

Created in 2003 as a VPS provider (“Linodes” in Linode parlance), Linode have grown to also provide object storage, load balancers and Kubernetes.

Note: not all of the locations listed for the different cloud providers have their respective managed Kubernetes service

Getting started

Creating an account

Creating an account is basically the same thing across all four providers - sign up, confirm your email, set up a payment method, done.

Digital Ocean and OVHCloud require you create a “project” to organise your resources. In DO, that’s just an organisational unit to group certain types of resources (but not Kubernetes clusters, for instance), and makes no difference, functionalities or billing-wise. In OVHCloud, a “public cloud project” is the level on which you interact with billing, quotas and IAM (Identity and Access Management, users and their rights).

In Digital Ocean you can create teams and add users with different access levels (which boil down to owner with full access, billing access, and access to DO resources); on OVHCloud the same can be done on the “public cloud project” level (with similar roles for billing and developers); Linode allows adding users with detailed ACLs (create a new linode, power on/off, create images, delete, etc.). Scaleway used to be the only ones without anything similar, but they recently added Organizations in beta, which allow sharing access at different levels (similar to DO) with different team members.

Navigating the web UI

All of the web interfaces are pretty similar - clean, with a lot of information provided at a glance and many actions readily available, with the exception of OVHCloud (I’ll get into the disaster it is in a bit).

Digital Ocean

Simple, elegant, everything is there at a glance. You can see billing usage, create new resources, check out documentation, switch projects and accounts/teams from the dashboard, and there’s a menu with all services on the left. Each resource creation presents an estimate of the monthly bill for it, which is pretty nice to visualise. You can search by name or IP. On each droplet’s page there is integrated monitoring with basic CPU, memory, load, disk, disk I/O, bandwidth graphs and basic alerting rules; backup and restore options (including a virtual console or boot from a recovery ISO). Overall, great UX.

Scaleway

Pretty similar to the Digital Ocean - simple, clean, everything is easily accessible. There’s a “Create” button up top, a detailed resources and billing overview, links to documentation, and a menu on the side with all services. You can attach a virtual console to the instances (requires a user with password auth enabled though), use cloud-init for the initial bootstrap, “protect” them to prevent accidental halt/deletion, and every creation previews an estimated monthly and hourly price. Each resource page (compute, object storage, etc.) has direct links to the associated documentation, which is handy.

OVHCloud

Completely different thing. Their “console” (OVHCloud Control Panel or Manager), is actually a mix of a bunch of different consoles for different branches of products, and is painfully slow (each console takes ~10s to load when first accessing it) and chaotic. It absolutely shows they’re a bigger company with various product ranges that evolved weirdly over the years, yet they could probably organise them better. AWS have everything from basic compute through satellite communications to quantum computers and the AWS Console is much clearer. Things aren’t helped by the confusing naming schemes and product lines - Web, Server, Public cloud, Telecom, Sunrise are product lines/sub-consoles, with unclear, sometimes overlapping product names like Cloud Web and Hosting under Web; Private Cloud under Cloud and Public Cloud under Public cloud. Some services are referenced by different names throughout documentation, blog posts and consoles. It’s a complete mess and it requires some clicking around (which is slow) or reading documentation just to understand how to get to the parts that interest you, which only brings up more questions:

Cloud, Cloud Web, Private Cloud and Public Cloud are listed as separate products/product lines. If OVH Load balancer is under Cloud and Managed Kubernetes is under Public Cloud, does that mean I can’t use load balancers with their k8s service?

Thankfully they provide a Product catalogue that at least lists all services with descriptions and categories in a somewhat clear fashion:

It would appear that Private cloud/Hosted Private Cloud/SDDC is a VMware-centred offering, and Public Cloud is OpenStack-based (which is why you get a Horizon interface on top of the OVHCloud one), their Kubernetes service being based on the latter.

Linode

Back a to clean and nice interface. You get a quick glance of existing resources, an easy way to create new ones via “Create button”, a powerful search, a sidebar menu with all services and settings, and cost estimations on each creation. Similar to DO, each instance’s interface also has some basic graphs.

CLI & API

Digital Ocean

doctl, Golang-based (static binaries for many platforms, distribution via homebrew and snapd) CLI, somewhat resembles kubectl in organisation (doctl subject action, e.g. doctl kubernetes cluster list), and can do pretty much everything Digital Ocean related - listing, creating, updating various resources and related actions (like getting a k8s cluster’s kubeconfig, and even, for snapd installs, connecting/integrating it with kubectl and ssh), inlcuding billing related(getting invoices).

Their API covers pretty much everything and has libraries in multiple languages. There are great docs and guides.

Scaleway

scw, Golang-based (static binaries for many plaforms, distribution via homebrew and Chocolatey), is inspired by the docker CLI, only manages compute instances and is currently under heavy redevelopment (v1.2 -> v2, in beta, and also following the kubectl mantras) to handle the new Scaleway services (including Kapsule).

Their API covers all services and is well documented.

OVHCloud

OVH have no actively maintained CLI, ovh-cli being archived since ~2018.

There’s an API, and associated Python and PHP SDK’s which cover some scenarios, but, weirdly, the Kuberentes methods are deprecated. Apparently they were moved under the /cloud/ section. Docs are much more basic (OpenAPI-style descriptions of the parameters, little extra explications, context or examples ) than DO or Scaleway.

Linode

linode-cli is a Python-based (distibution and installation via PyPi) flexible CLI that allows creating/updating/deleting/listing pretty much any Linode resource (Linodes, domains, managed kubernetes, etc.).

Their API is fully featured, and they have libraries and tools for various languages and tasks.

Conclusion

All 4 have complete APIs, but Digital Ocean, Scaleway and Linode also have great tooling (libraries, SDKs, CLIs)

Terraform providers

All 4 have terraform providers with varying levels of maturity. Digital Ocean, Scaleway and Linode cover all of their services (including Kubernetes), while OVHCloud’s one doesn’t (somewhat understandable considering their complex web of services and products).

Kubernetes

Digital Ocean Kubernetes Service

Digital Ocean Kubernetes Service is currently available in 10 datacenters across 8 locations covering most of the world (North America, Europe and Asia). It supports recent-ish versions of Kubernetes (v1.17, the latest being v1.18), and new clusters can be created via the web interface, API, CLI and terraform. Node pools can be composed of Standard, CPU-Optimised, Memory-optimised or General Purpose droplets, and you can mix and match them as you want to fit different workloads. Cluster-autoscaler is available for dynamically sized node pools, but only when creating via terraform or doctl; via the web UI a node pool can only be modified, but not created, as autoscaling. Clusters come with cilium as the CNI plugin, a Kubernetes dashboard (with automatic SSO when coming from the DO web interface), automatic patches in maintenance windows (only patch versions, e.g. v1.17.0 to v1.17.1, but not upgrades like v1.17 to v1.18, those have to be manually launched), basic cluster and node-level metrics with some nice graphs, thanks to the their do-agent which runs on all nodes, which can even be used for basic alerting (only simple rules, metric X is above or below fixed value Y for duration Z on droplet or droplets by tag).

DO also provide a basic managed Container Registry, which is however in early availability and pricing is specific and fixed (you need to subscribe do a Digital Ocean Spaces(their object storage) plan at $5/month for 250GB of storage and 1TB outbound data transfer), but access is easy (you need a DO API token, which can be read-only or read/write) from within your CLI or Kubernetes cluster.

Digital Ocean’s cluster-autoscaler, Cloud Controller Manager (which allows the creation of Load balancers from within the cluster) and CSI plugin are all open source and available on GitHub to consult and contribute to. Documentation and how-to tutorials are top notch, and explain important bits like creating and configuring load balancers, adding block storage, etc.

Operations (cluster creation, upgrades) are pretty fast and usually complete within a few minutes.

All in all, you get a pretty complete and feature rich solution - the only thing I’d say is “missing” is log management, something one might take for granted coming from AWS/GCP/Azure.

Scaleway Kapsule

Scaleway Kapsule is currently only available in the Paris region, but stays very close to Kubernetes releases (v1.18.6, latest release, was available less than a day after being on GitHub), which is really rare, I haven’t found another provider so up to date (for reference, the closest, OVHCloud, are on v1.18.1, GCP are in preview on v1.17.6, and AWS are still on v1.16.8). Clusters can be created via the web UI, API and terraform, all with the same options, which are pretty rich:

CNI (Container Network Interface) can be any one of cilium (default), calico, weave or flannel
ingress can be any of none (default), nginx or traefik (versions 1.7 or 2.2), which can also be modified in-place after creation
autoscaling and autohealing of nodes
nodes come in a variety of sizes from 3 different product lines, DEV (2-4 vCPUs and 2-12GB RAM), GP1 (General Purpose, 4-48 vCPUs and 16-256GB RAM) and RENDER (which include dedicated NVIDIA Tesla P100 16GB GPUs alongside 10 cores (half of an Intel Xeon Gold 6148))

There’s a default node pool and you can add extra ones later on, each with its own autoscaling configuration. Besides that, you can opt in Kapsule deploying a Kubernetes dashboard for you (in a dedicated namespace, following best practices). Every cluster also comes with a wildcard DNS you can use to test things, but there’s no monitoring/metrics/alerting included (besides metrics-server running on clusters by default), only docs about about how you can add them yourself. They do, however, have a basic managed container registry available, where storage is cheap (€0.025/GB/month) and networking is free within the same region (€0.03/GB/month outside), but access to a private instance of it requires a Scaleway API Token, each of which has full read/write access to the whole of your Scaleway account, so it’s not great, security-wise.

Scaleway’s Cloud Controller Manager and CSI plugin are also open source and available on GitHub, and their documentation and tutorials are also pretty great and detailed, and include important things like creating load balanacers, using Loki for logs management and plenty of others. Cluster upgrades are available via the API (and terraform).

Operations (creation, upgrades, scale ups and downs) are pretty fast.

Similar to Digital Ocean, you get a pretty complete and feature rich managed Kubernetes, with the main things “missing” being integrated monitoring/metrics/alerting and log management.

OVHCloud Managed Kubernetes Service

OVHCloud’s Kubernetes service is only available in two datacenters, in Canada and France (more to come in 2020), and, even if they keep up with Kubernetes versions faster than DO (v1.18.1 is latest available, and they promise they’ll have new versions available in the quarter after release), it works a bit differently, and not necessarily in a good way. Organisationally, Kubernetes clusters and nodes are under their Public Cloud services, and billing is under the respective Public Cloud project. You have to first create a cluster (control plane hosted by OVHCloud), and only after it’s created you can add individual nodes, each of which is rather slow (at least compared to DO and Scaleway) - ~5 minutes for the cluster creation (and apparently it’s a feature because they even notify you by email when your cluster has been created) and 5-10 minutes for each node. Node pools and autoscaling are on their official roadmap and are available as experimental features in their API. On that roadmap are also bare metal nodes, which can be great for performance. By default clusters are on the “maximum security” security policy, which means they get all patch updates automatically, and you can do upgrades manually (via the web interface or the API).

Nodes come in different flavors (sizes) and flavor families (CPU, RAM, IOPS (which come with dedicated NVMe drives)), but not all are available depending on location (e.g. at the time of writing, in Gravelins, France, anything with more 8vCPUs from the CPU optimised or 30GB RAM from the memory optimised wasn’t available, and there were no instances at all, of any type, from the IOPS-optimised family). Canal(calico for policy and flannel for networking) is used as the CNI plugin.

OVHCloud’s Managed Container Registry is far more advanced than Digital Ocean or Scaleway’s, since it’s based on CNCF’s Harbor project, and it includes detailed RBAC (Role-Based Access Control), vulnerability scans of images, and Helm chart hosting alongside OCI-compatible images, but pricing is a bit specific and inflexible - there are different tiers and you pay for a combination of storage (200GB, 600GB or 2TB) and number of concurrent connections (15, 45, 90), while networking is free. There’s no included by default monitoring/metrics/alerting or logging, but they have a (somewhat old, but probably still usable) doc on how to send metrics from your clusters to their OVH Observability/Metrics platform (based on Warp10), and another one about sending logs to their Logs Data Platform (based on Graylog). Their documentation is pretty decent, even if there are some losses in translation with French idioms/phrases slipping in the English versions.

Operations feel slow - on top of the aforementioned 5-10 minutes per adding a node, a load balancer takes around the same time to create, and even the Kubernetes API sometimes takes a bit more to respond than usual. OVHCloud run Kubernetes on top of OpenStack and probably use the official cloud-provider-openstack and the associated cloud controller manager and Cinder CSI plugin.

Overall, it’s hit and miss. There are some great things about OVHCloud’s Kubernetes service, mostly in related services (Harbor is a great registry, there are fully featured metrics and logs platforms), but the Kubernetes itself is direly missing in even basic features like node pools, and the pricing on those related services is on static plans (instead of pay per use) and in general everything about it feels slow and clunky. At least their roadmap is public and they intend to improve some of the major downfalls, and hopefully work on the speed and UX.

Linode Kubernetes Engine

Linode’s LKE is available in 10 locations across North America, Europe and Asia Pacific (and they even provide a handy speed test page to help you pick the closest region), and, similar to Digital Ocean, have a relatively recent version of Kubernetes (v1.17.x, the latest being v1.18.x) available. Clusters can be created via terraform, web interface, CLI or API, come with calico as the CNI and their node pools can be composed of nodes from the Standard (1-32 vCPUs, 2-192GB RAM), Dedicated (2-48 dedicated virtual cores, 4-96GB RAM) or High Memory (1-16 CPU, 24-300GB RAM) ranges; sadly their GPU instances are not available. There are basic graphs per instance (CPU, IPv4 and IPv6 network traffic, and Disk I/O) coming from their Longview agent, which is deployed on all Kubernetes nodes, which can then also be turned into basic alerting (enabled by default with sane thresholds), as well as decent audit logs.

There’s no cluster (node) autoscaling, no cluster upgrades (v1.16 to v1.17), and no node patches/upgrades, only automatic control plane patches by Linode. And even though they don’t provide a managed Docker registry, they have a how-to guide on how to set up one with their Object Storage and the Docker registry Helm chart, and, of course, if you’re going to run one yourself, you can always install Harbor for the extra features.

Their documentation is pretty detailed and of good quality, and so are their open source tools (CLI, SDK, API, terraform provider, cloud controller manager, CSI plugin).

Operations are pretty fast (and you can even quantify it thanks to their logs, which show it took ~60s for an instance to become available).

Overall, LKE is decent but lacking some very important features (cluster upgrades and autoscaling).

Features comparison

Legend

feature/service present and in GA
feature/service present, in GA, and much more advanced than competitors'
feature/service present in beta
feature/service not present
feature/service on roadmap
feature available, but it costs extra or is in a separate service billed separately

Feature	Digital Ocean	Scaleway	OVHCloud	Linode	Google
Up to date Kubernetes	v1.17	v1.18	v1.18	v1.17	v1.17 in preview
Free control plane					only for the first zonal cluster
Kubernetes dashboard		optional			replaced by a GKE dashboard
Ingress Controller		optional, Nginx/ Traefik v1 or v2			optional, GKE Ingress
Bare metal Instances
GPU Instances
Cluster Autoscaler
Cluster Patches (v1.17.x)
Automatic Cluster Patches
Cluster Upgrades (v1.xx)
Metrics
Logs
API
Terraform provider
IAM/ Team management
CNI plugin	cilium	cilium, calico, weave or flannel	canal	calico	GKE custom or calico
VPC (Private network)
Unlimited network transfer			except for SYD and SGP*
IPv6 dualstack
Multi-zone/ datacenter clusters

* “Outbound public traffic is also free and unlimited, with the exception of the Singapore (SGP) and Sydney (SYD) regions, where 1,024 GB/month per project, per datacentre is available. Each additional GB will be charged for” Source

Additional Services	Digital Ocean	Scaleway	OVHCloud	Google
Block storage
Object Storage	Spaces	Object Storage	Object storage (based on Swift)	Google Cloud Storage
Archival Storage		C14	Cloud Archive	Cloud Storage for data archiving
Load Balancer
Managed Container Registry	basic	basic	Harbor	GCR
Managed Databases	PgSQL, MySQL, Redis	PgSQL, MySQL*	PgSQL, MySQL, MariaDB, Redis	a lot
Secrets Management				Secret Manager
Managed Message broker		IoT Hub (MQTT)	ioStream	Pub/Sub
Serverless (Functions)		Knative		Cloud Functions
Serverless (Containers)		Knative		Cloud Run
AI		AI Inference	Data & Analytics	Cloud AI
Data Warehouse			Data Analytics Platform (based on Hadoop)	BigQuery
Data Processing			Data Processing (based on Spark)	Dataprep/ Dataproc/ Dataflow

* Only PgSQL is currently publicly available as a managed database by Scaleway, MySQL is in private beta

Pricing

All four providers have relatively simple pricing when it comes to their managed Kubernetes services - the control plane is free, you only pay for compute, memory, storage and networking (load balancers, extra public IPs) you consume, which are billed with hourly granularity (practical for autoscaling). There’s however a caveat with some of them - Digital Ocean and Linode have an account-level monthly public network transfer cap based on amount and size of instances you run (starting at 1TB/droplet for DO and 2TB/linode for Linode, per full month), and anything on top is billed ($0.01/GB). Scaleway and OVHCloud have free network transfer, with the latter having an exception for instances located in their Singapore and Sydney datacenters, which are capped at 1024 GB/project/month (but they don’t have the Kubernetes service anyway).

Compute

Back in April Scaleway posted this comparison in prices between them and competitors (i checked the numbers and they aren’t lying):

(Linode are missing, but their pricing is identical to Digital Ocean’s - and I mean that literally, the only difference in Linode Standard Plans vs Standard Droplets is the network transfer each instance adds to your cap - compute, memory, even local storage space are the same and with the same price)

Note Pricing is in USD regardless of location for Digital Ocean and Linode, in EUR for Scaleway, and OVHCloud propose different currencies depending on your location (EUR in Eurozone countries, PLN in Poland, CAD in Canada, etc.) The course at the time of writing is 1 USD = 0.89EUR | 1 EUR = 1.13 USD , but that doesn’t change the scale too drastically because the difference between Scaleway and OVHCloud, Digital Ocean/Linode, and then AWS/GCP/Azure is too big. In further comparisons, I’ll use prices in their original currency, as billed, and in parenthesis convert the values to euros/US dollars using today’s course.

And it’s a great comparison, which shows they’re ~~slightly cheaper~~ pretty much the same with their price hike starting 01/08/2020 when it comes to General Purpose instances compared to OVHCloud, and very cheaper compared to DO/Linode, and even more so to AWS/GCP/Azure. However they provide little flexibility in terms of CPU/RAM combinations, so if one requires a different ratio, they might not be the best option.

Storage

Block Storage

Provider	Storage class	Price per GB/month
Digital Ocean	SSD, 800-5000 IOPS, 20-200 MB/sec depending on block size	$0.10 (€0.089)
Scaleway	SSD, 5000 IOPS	€0.08 ($0.09)
Scaleway	SSD+, 10000 IOPS (coming mid-2020)	€0.12 ($0.14)
OVHCloud	Classic, 250 IOPS guaranteed	€0.04 / 0.06 CA$ ($0.045 )
OVHCloud	High speed, up to 3000 IOPS	€0.08 / 0.138 CA$ ($0.09)
Linode	up to 5000 IOPS, 150MB/s	$0.10 (€0.089)

Scaleway seem to have the best price/performance rate, Linode and Digital Ocean are again very similar, and OVHCloud are in a category of their own with very cheap but very slow storage, and a “high speed” class which is slower than the competition but more acceptable.

Object storage

Provider	Storage included	Additional storage cost GB/month	Outbound Transfer included	Additional Outbound Transfer GB/month	Basic price per month
Digital Ocean	250GB	$0.02 (€0.018)	1TB	$0.01 (€0.0089)	$5 (€4.45)
Scaleway	75GB	€0.01 ($0.011)	75GB	€0.01 ($0.011)	0
OVHCloud	0	€0.01 / 0.015 CA$ ($0.011)	0	€0.01 / 0.015 CA$ ($0.011)	0
Linode	250GB	$0.02 (€0.018)	1TB	$0.01 (€0.0089)	$5 (€4.45)

Once again, Scaleway are cheaper than everyone else (to be fair, they have the same prices as OVHCloud, but they have a decent free tier of 75GB of storage and 75GB transfer/month to start), and Digital Ocean and Linode have identical pricing.

Networking

Load balancers

Provider	Load balancer class	Price per month
Digital Ocean	Load balancer	$10 (€8.90)
Scaleway	LB-GP-S, 200Mbps	€8.99 ($10.11)
Scaleway	LB-GP-M, 500Mbps	€19.99 ($22.47)
Scaleway	LB-GP-L, 1Gbps, Multicloud	€49.99 ($56.20)
OVHCloud	Network Load Balancer	€10 / 16 CA$ ($11.24)
Linode	Nodebalancer	$10 (€8.90)

Digital Ocean and Linode, with identical pricing once more, come out slightly cheaper than Scaleway, which are the only ones with different tiers and guaranteed throughput and OVHCloud.

Network transfer

As I already mentioned, OVHCloud and Scaleway have unlimited (in volume) network transfer, while Linode and Digital Ocean apply an account-wide cap based on number and type of instances (starting from 1TB/droplet for DO, 2TB/ Linode for Linode), and anything extra costs $0.01/GB.

SLA & Support

Provider	Support tier	SLA	Response time	Price per month
Digital Ocean	Basic	99.99% droplets and block storage	No guarantee	0
Scaleway	Basic	99.9% network, 99.99% instances	No guarantee	0
Scaleway	Developer	99.9% network, 99.99% instances	<12 hours	€2.99
Scaleway	Business	99.9% network, 99.99% instances	<2 hours	“Starting at €49”
Scaleway	Enterprise	99.9% network, 99.99% instances	<30 minutes	“Starting at €499”
OVHcloud	Standard	99.999% instances	No guarantee, ~8 hours (working hours)	0
OVHcloud	Premium	99.999% instances	No guarantee, ~2 hours (working hours)	€50 / 72 CA$, 12 months minimum
OVHcloud	Business	99.999% instances	30mn - 8 hours depending on severity (24/7)	“starting at” €250 / 360 CA$
OVHcloud	Enterprise	99.999% instances	15 minutes for P1 (24/7)	“starting at” €5000 / “contact us”
Linode	Basic	99.99% for all services	No guarantee	0
AWS	Basic	99.99% for EC2	No guarantee	0
GCP	Basic	99.5% for a single GCE instance	No guarantee	0

You can calculate how much is 99.9% with uptime.is

OVHCloud and Scaleway are the only ones to offer business suport, and the former have the best SLA (99.999%, better than AWS EC2 at 99.99 or GCP at 99.5, which, to be fair, make it easier to run multi-zonal/regional, so an individual instance’s availability is of lesser importance), compared to everyone else’s 99.99%.

Conclusion

Overall, it’s a pretty close match between Scaleway and Digital Ocean, with the latter beating based on availability (8 locations vs 1), but the former edging on features alone, and then some more based on pricing. OVHCloud simply aren’t ready yet (no node pools and no autoscaling are deal breakers) and with a poor UX and tooling, while Linode is too basic for most non-test workloads (recreating clusters to upgrade Kubernetes versions is too cumbersome) while costing almost exactly the same as DO.

Of course, neither of them are on par with AWS or GCP in terms of geographical locations or services, and probably never could be. Nevertheless, they’re more than sufficient for lots of workloads, hitting a sweet spot of simple, cost-effective and feature-rich (or at least having the basic building blocks for those features - thanks to Operators running pretty much anything yourself is getting easier), while still using widely available standards (Kubernetes) that allow mobility across cloud providers, if the needed arises.

Getting started with Scaleway's managed Kubernetes service - Kapsule

adrian.todorov@atodorov.ninja (Adrian Todorov) — Thu, 21 Nov 2019 18:26:15 +0200

Introduction

Scaleway is a French cloud provider that mostly specialises in (custom designed) bare metal ARM servers, standard VPSes, and has recently started adding some additional services like x86 bare metal servers, Load Balancers, a new and improved object storage, managed databases, container registry, managed firewalls, and, hotly anticipated, a managed Kubernetes Service, Kapsule. There’s plenty of competition in the managed Kubernetes space, but Scaleway have a few potential advantages, most notably:

the possibility to have bare metal and ARM-based node pools (not yet available in the public beta, but ARM is kind of Scaleway’s speciality, so it’d be surprising if they didn’t offer it)
decent integrated ecosystem - Load Balancers, Container Registry, block and object storage, and plenty of others soon
pricing, which is on par with Digital Ocean’s pricing (even if the smallest possible node type is pretty big at 4vCPU/16GB RAM and at 39 euros/month), the control plane is free, and associated services aren’t expensive - load balancers are at 9 euros/month, container registry is at 0.025 euros/GB/month for storage, with networking free within the same region and at 0.03 euros/GB/month inter-region
cluster upgrades, which aren’t offered by everyone (and for a long time neither Digital Ocean nor OVH proposed it, and Kapsule is younger than either of them)
cluster autoscaling (based on the cluster autoscaler Kubernetes project), which, again, isn’t offered by everyone
EU-based (datacenters in Paris and Amsterdam)

So, let’s give a Kapsule cluster a spin and see if it lives up to my expectations.

Kapsule Overview

Currently Kapsule is in beta, so some features aren’t ready yet (like cluster autoscaling or upgrades from the web UI), and for now only the Paris region is supported.

Kapsule clusters can use calico, weave, flannel or cilium for the overlay network, and upon creation Scaleway can optionally deploy traefik (1.x, for now) or nginx ingress controllers, and the Kubernetes dashboard (which of course you can do later on your own, via Helm or otherwise, but it’s a nice touch). You get an automatic wildcard DNS with your cluster’s id.nodes.k8s.fr-par.scw.cloud, which can come in handy for testing.

Deploying a cluster

For now, cluster deployment is possible via the API, the web console or terraform (which sadly wasn’t available when i started writing this post).

Let’s deploy a small cluster via the API. First you’ll need your organization id and a token secret key, both available from the credentials page of the Scaleway console, and export the latter for easier reuse to SCALEWAY_TOKEN . Then, create a JSON file along these lines (updating the organization_id with yours), which is the bare minimum to create a 1.14.8 (on purpose, to test cluster upgrades to a more recent version, 1.15.x, later on) Kubernetes cluster with 2 GP-1S nodes in the default node pool, with calico and treafik ingress:

Creating the following resources will cost you money, as per Scaleway's pricing page.

{
  "organization_id": "xxx",
  "name": "k8s-test",
  "version": "1.14.8",
  "cni": "calico",
  "dashboard": true,
  "ingress": "traefik",
  "default_pool_commercial_type": "gp1_s",
  "default_pool_autoscaling": false,
  "default_pool_size": 2
}

Then POST it to the Scaleway API in the right region (fr-par only for now):

curl -XPOST  -H "X-Auth-Token: ${SCALEWAY_TOKEN}" -d @data.json https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters

And we get a response straight away with the cluster

                          

ID and the DNS wildcard: style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;display:grid;">{ "autoscaler_config": { "balance_similar_node_groups": false, "estimator": "binpacking", "expander": "random", "expendable_pods_priority_cutoff": -10, "ignore_daemonsets_utilization": false, "scale_down_delay_after_add": "10m", "scale_down_disable": false }, "cluster_ip": "", "cluster_port": 0, "cluster_url": "https://f50f0126-b994-47a9-9949-b68a3ed1335b.api.k8s.fr-par.scw.cloud:6443", "cni": "calico", "created_at": "2019-09-15T15:32:01.278854200Z", "current_core_count": 0, "current_mem_count": 0, "current_node_count": 0, "description": "", "dns_wildcard": "*.f50f0126-b994-47a9-9949-b68a3ed1335b.nodes.k8s.fr-par.scw.cloud", color:#3c3d38">    "id": "f50f0126-b994-47a9-9949-b68a3ed1335b", "name": "k8s-bal", "organization_id": "xxx", "region": "fr-par", color:#3c3d38">    "status": "creating", "sub_status": "no_details", "tags": [], "updated_at": "2019-09-15T15:32:01.278854200Z", "version": "1.14.8" }

Note down the cluster ID, and store it in a varialbe (like CLUSTER_ID) for future use, so that you can check the status and get the kubeconfig file:

curl -XGET  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}"
curl -XGET  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}/kubeconfig?dl=1" > /tmp/mycluster.kubeconfig

Once the cluster is ready, we can check connectivity with kubectl:

export KUBECONFIG=/tmp/mycluster.kubeconfig
kubectl cluster-info
Kubernetes master is running at https://f50f0126-b994-47a9-9949-b68a3ed1335b.api.k8s.fr-par.scw.cloud:6443
CoreDNS is running at https://f50f0126-b994-47a9-9949-b68a3ed1335b.api.k8s.fr-par.scw.cloud:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Testing the traefik ingress controller

Let’s create a simple Ingress for the Treafik dashboard deployed for us by Scaleway to test it and the Traefik Ingress Controller.

First, generate a file which will contain the basic auth credentials with htpasswd (apache2-utils package on Debian/Ubuntu), with the -B option which forces bcrypt instead of the horribly outdated and insecure MD5 used by default, and create a Kubernetes secret in the kube-system namespace based on it :

htpasswd -c -B ./auth admin
New password:
Re-type new password:
Adding password for user admin

kubectl create secret generic traefik-dash-basic-auth --from-file auth --namespace=kube-system

Then, create an Ingress object for the Traefik dashboard, swapping XXX in the host field with your cluster id:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    traefik.ingress.kubernetes.io/auth-type: basic
    traefik.ingress.kubernetes.io/auth-secret: traefik-dash-basic-auth
spec:
  rules:
  - host: dashboard.XXX.nodes.k8s.fr-par.scw.cloud
    http:
      paths:
      - path:
        backend:
          serviceName: ingress-traefik
          servicePort: admin

and apply it:

kubectl apply -n=kube-system -f traefik-ingress.yml

After that, going to dashboard.XXX.nodes.k8s.fr-par.scw.cloud should get you to the traefik dashboard, protected by the username and password you created with htpasswd.

So our cluster is up and running, and so is the Traefik ingress.

Kapsule with Scaleway’s Container Registry

Scaleway provide a managed container registry service, called simply “Container Registry”. It’s rather barebones - no security scanning or anything, just a basic Docker Registry that is separated in namespaces which have regionally unique names, can’t be recreated and can either be publicly accessible or private. If they are private, accessing them requires Scaleway API tokens, which aren’t scoped and thus give full read/write access to the Scaleway API, which isn’t great. You only pay for storage (0.025/GB/month) and traffic outside of the same region (a Kapsule cluster in fr-par pulling from a Container Registry in fr-par will cost nothing).

To use it, you only need to create a namespace(more details on that below), docker login, docker build / docker tag your image, and docker push it, like this:

docker login rg.fr-par.scw.cloud -u anyuser -p ${SCALEWAY_TOKEN}
docker build . -t rg.fr-par.scw.cloud/sofixa/golang-example-web-server-k8s:latest
docker push rg.fr-par.scw.cloud/sofixa/golang-example-web-server-k8s

To be able to use the container image created from inside Kapsule, if your registry is private, you need to configure the registry authentication on your cluster (link to the official docs), by first creating the secret in your namespace with the Scaleway token:

kubectl -n=<your-namespace> create secret docker-registry regcred --docker-server=rg.fr-par.scw.cloud --docker-username=anyuser --docker-password=${SCALEWAY_TOKEN} --docker-email=<your-email>

And then using imagePullSecrets in your template spec on Deployments/Daemon Sets/Stateful Sets/Pods:

spec:
  selector:
      matchLabels:
        name: golang-test # Label selector that determines which Pods belong to the DaemonSet
  template:
    metadata:
      labels:
        name: golang-test # Pod template's label selector
    spec:
      containers:
      - name: golang-test
        image: rg.fr-par.scw.cloud/xxx/golang-example-web-server-k8s:1.0
      imagePullSecrets: 
        - name: regcred

Kapsule with Scaleway’s Load balancer

Scaleway’s Load Balancer service is an active-passive Load balancer that supports multiple frontends (listeners) and multiple backends (targets), which can be of different types, with pre-defined healthchecks for backends like PostgreSQL, MySQL, Redis, LDAP or plain old TCP or HTTP. It can do SSL offloading (via automatically provisioned Let’s Encrypt certificates) or SSL passthrough (by configuring a frontend and backend with TCP/443).

To create a Scaleway Load Balancer from inside Kapsule, you need to create a Service of Type LoadBalancer, like this:

apiVersion: v1
kind: Service
metadata:
  name: golang-test
spec:
  selector:
    name: golang-test
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer

Which creates a LB with one frontend on port 80, one backend with all Kapsule nodes containing your service (targeting it on port 80), with round-robin load balancing (the default). The Scaleway Cloud Controller doesn’t seem to be fully documented yet, so here is a list of the available annotations (kudos to ben from Scaleway’s Community Slack for sharing them):

service.beta.kubernetes.io/scw-loadbalancer-forward-port-algorithm #annotation to choose the load balancing algorithm
service.beta.kubernetes.io/scw-loadbalancer-sticky-sessions #annotation to enable cookie-based session persistence
service.beta.kubernetes.io/scw-loadbalancer-sticky-sessions-cookie-name #annotation for the cookie name for sticky sessions
service.beta.kubernetes.io/scw-loadbalancer-health-check-type #health check used
service.beta.kubernetes.io/scw-loadbalancer-health-check-delay #time between two consecutive health checks
service.beta.kubernetes.io/scw-loadbalancer-health-check-timeout #additional check timeout, after the connection has been already established
service.beta.kubernetes.io/scw-loadbalancer-health-check-max-retries #number of consecutive unsuccessful health checks, after wich the server will be considered dead
service.beta.kubernetes.io/scw-loadbalancer-health-check-http-uri #URI that is used by the "http" health check
service.beta.kubernetes.io/scw-loadbalancer-health-check-http-method #method used by the "http" health check
service.beta.kubernetes.io/scw-loadbalancer-health-check-http-code #HTTP code that the "http" health check will be matching against
service.beta.kubernetes.io/scw-loadbalancer-health-check-mysql-user #MySQL user used to check the MySQL connection when using the "mysql" health check
service.beta.kubernetes.io/scw-loadbalancer-health-check-pgsql-user #PgSQL user used to check the PgSQL connection when using the "pgsql" health check
service.beta.kubernetes.io/scw-loadbalancer-send-proxy-v2 #annotation that enables PROXY protocol version 2 (must be supported by backend servers)
service.beta.kubernetes.io/scw-loadbalancer-timeout-server # maximum server connection inactivity time
service.beta.kubernetes.io/scw-loadbalancer-timeout-connect #maximum initical server connection establishment time
service.beta.kubernetes.io/scw-loadbalancer-timeout-tunnel #maximum tunnel inactivity time
service.beta.kubernetes.io/scw-loadbalancer-on-marked-down-action# annotation that modifes what occurs when a backend server is marked down

Details about them and their possible values are available on the Load Balancer API docs.

Cluster upgrades

One of the most interesting features of Kapsule is the possibility to do rolling cluster upgrades, let’s test it!

We’ll create a random service behind a Load Balancer to test if our service stays up during the upgrade, as it should. To see what’s going on, the service will based on a DaemonSet (a pod running on each node) with a simple Golang http server which prints the node hostname. Along the way we’ll test Scaleway’s Load Balancer and Container Registry services.

Prerequisites

First, create your container registry with a similar JSON file, editing the appropriate lines:

{
  "name": "xxx",
  "description": "My awesome container registry",
  "organization_id": "xxx",
  "is_public": false
}

and POST it to Scaleway’s API endpoint for Container Registry namespaces:

https://api.scaleway.com/registry/v1/regions/{region}/namespaces

curl -XPOST  -H "X-Auth-Token: ${SCALEWAY_TOKEN}" -d @data.json https://api.scaleway.com/registry/v1/regions/fr-par/namespaces

Second, clone the example code in my GitHub repository , build its Docker container, and push it to your registry:

git clone git@github.com:sofixa/golang-example-web-server-k8s.git
cd golang-example-web-server-k8s/
docker build . -t rg.fr-par.scw.cloud/xxx/golang-example-web-server-k8s:1.0
docker push rg.fr-par.scw.cloud/xxx/golang-example-web-server-k8s

Once the image is successfully uploaded to the registry, let’s create a DaemonSet from it and a Load Balancer in front with the manifest files (based on the examples from the Load Balancer and the Container Registry sections) in the same repository ( the DaemonSet manifest will try to use a regcred secret to authentify to your registry, so if your registry is public, you should remove the imagePullSecrets part from it):

kubectl create namespace test-golang
namespace/test-golang created

kubectl apply -n=test-golang -f daemon-set.yml
daemonset.apps/golang-test created

kubectl apply -n=test-golang -f load-balancer.yml
service/golang-test created

To get the Load Balancer’s external IP, get the corresponding service and copy the EXTERNAL-IP:

kubectl get svc -n=test-golang
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
golang-test   LoadBalancer   10.38.179.245   51.159.25.121   80:31120/TCP   5d23h

Once the load balancer is ready, curl-ing it’s public IP should get you a 200 response code with a similar response body:

curl -v http://51.159.25.121   
* Rebuilt URL to: http://51.159.25.121/
*   Trying 51.159.25.121...
* TCP_NODELAY set
* Connected to 51.159.25.121 (51.159.25.121) port 80 (#0)
> GET / HTTP/1.1
> Host: 51.159.25.121
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 11 Nov 2019 20:11:57 GMT
< Content-Length: 65
< Content-Type: text/plain; charset=utf-8
<
Hello world from scw-k8s-test-default-16b716e81c7a4!
* Connection #0 to host 51.159.25.121 left intact

Notice the response, which contains the node name, composed of the following:

cluster name k8s-test
node pool name default
node id 16b716e81c7a4

The test

Now, let’s set up vegeta, a HTTP load testing tool, to test if our cluster continues to respond during a cluster upgrade. Follow the installation instructions, and do a quick test on your load balancer’s IP:

echo "GET http://51.159.25.121" | vegeta attack -rate=1/s
wResult	Attack
              SeqCode	TimestampLatencBytesOutBytesInError
                                                           Body
Timeb���6P<bAAHello world from scw-k8s-test-default-3cb12e55883e4!
d���6*<H.�AAHello world from scw-k8s-test-default-3cb12e55883e4!

Everything seems to be working (ignore the binary output, vegeta output isn’t meant to be parsed directly by humans), you can quit/Ctrl+C the process. You can now launch a continuous “attack” in vegeta parlance, storing the results in a json file:

echo "GET http://51.159.25.121" | vegeta attack -rate=1/s | vegeta encode > results.json

cat-ing the file should result in a similar output, which contains all sorts of useful information for a normal load test (the main purpose of vegeta), but for our test case, only the response code and body (which is base64 encoded) matter:

cat results.json | jq
  "attack": "",
  "seq": 0,
  "code": 200,
  "timestamp": "2019-11-11T21:24:15.375938851+01:00",
  "latency": 21093314,
  "bytes_out": 0,
  "bytes_in": 65,
  "error": "",
  "body": "SGVsbG8gd29ybGQgZnJvbSBzY3ctazhzLWVsb3F1ZW50LWxlaG1hbm4tZGVmYXVsdC0zY2IxMmU1NTg4M2U0IQo="
}

A base64decode / base64 -d on that body will get the same result as a bare curl:

head -n 1 results.json | jq .body | base64 -i -d
Hello world from scw-k8s-test-default-3cb12e55883e4!

Once the vegeta load test is running, time to launch an upgrade to the Kapsule cluster.

The upgrade

Always read the full Kubernetes release notes when doing an upgrade. API's get depreceated, breaking changes happen, stuff is no longer compatible, you should always check.

Kubernetes cluster upgrades are done in two main stages:

the control plane and all its components
the worker node pools, a node at a time

During the control plane upgrade, it remains accessible (worst case scenario you might get an EOF or two). For a worker node to be updated, it needs to be drained from all pods. Scaleway do this for you automatically, but for that to happen there must be enough resources on the other nodes for all the pods required to be scheduled.

Usually you can only upgrade a minor version at a time (e.g. 1.14.x to 1.15.x, but not 1.16.x directly), and this is the case with Scaleway. In any case, doing the upgrade over API is cooler, so that’s what we’ll do. All that’s required is a small JSON file with the version wanted (remember, only patches or a single minor version upwards are supported) and a boolean wether we want to upgrade the nodes as well (duh).

{
  "version": "1.15.5",
  "upgrade_pools": true
}

All that’s left to do is to POST that to the /upgrade endpoint:

curl -XPOST -d @data.json  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}/upgrade

Which would get a similar result:

{
  "region": "fr-par",
  "id": "ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b",
  "organization_id": "37a7df83-e2f2-43aa-a181-170a52aec2ac",
  "created_at": "2019-11-11T19:46:54.261230Z",
  "updated_at": "2019-11-11T20:49:16.331469260Z",
  "name": "k8s-test",
  "description": "",
  "cluster_ip": "",
  "cluster_port": 0,
  "current_node_count": 2,
  "status": "updating",
  "sub_status": "deploy_controlplane",
  "version": "1.15.5",
  "cni": "calico",
  "tags": [],
  "current_core_count": 8,
  "current_mem_count": 34359738368,
  "cluster_url": "https://ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b.api.k8s.fr-par.scw.cloud:6443",
  "dns_wildcard": "*.ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b.nodes.k8s.fr-par.scw.cloud",
  "autoscaler_config": {
    "scale_down_disable": false,
    "scale_down_delay_after_add": "10m",
    "estimator": "binpacking",
    "expander": "random",
    "ignore_daemonsets_utilization": false,
    "balance_similar_node_groups": false,
    "expendable_pods_priority_cutoff": -10
  }
}

To check the status of the upgrade, you can use GET the status of the cluster and the node pools via the Scaleway API on the cluster, pools and nodes endpoints or kubectl get nodes:

curl -XGET  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}"
{
  "region": "fr-par",
  "id": "ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b",
  "organization_id": "37a7df83-e2f2-43aa-a181-170a52aec2ac",
  "created_at": "2019-11-11T19:46:54.261230Z",
  "updated_at": "2019-11-11T21:04:47.430331Z",
  "name": "k8s-test",
  "description": "",
  "cluster_ip": "",
  "cluster_port": 0,
  "current_node_count": 2,
  "status": "ready",
  "sub_status": "deploy_controlplane",
  "version": "1.15.5",
  "cni": "calico",
  "tags": [],
  "current_core_count": 8,
  "current_mem_count": 34359738368,
  "cluster_url": "https://ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b.api.k8s.fr-par.scw.cloud:6443",
  "dns_wildcard": "*.ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b.nodes.k8s.fr-par.scw.cloud",
  "autoscaler_config": {
    "scale_down_disable": false,
    "scale_down_delay_after_add": "10m",
    "estimator": "binpacking",
    "expander": "random",
    "ignore_daemonsets_utilization": false,
    "balance_similar_node_groups": false,
    "expendable_pods_priority_cutoff": -10
  }
}

curl -XGET  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}/pools"
{
  "total_count": 1,
  "pools": [
    {
      "region": "fr-par",
      "id": "feb2c164-8805-4130-b5d3-57889dc35652",
      "cluster_id": "ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b",
      "created_at": "2019-11-11T19:46:54.266926Z",
      "updated_at": "2019-11-11T21:02:09.443695Z",
      "name": "default",
      "current_node_count": 2,
      "status": "updating",
      "version": "1.16.2",
      "commercial_type": "gp1_xs",
      "autoscaling": false,
      "size": 2,
      "min_size": 2,
      "max_size": 2,
      "current_core_count": 8,
      "current_mem_count": 34359738368,
      "container_runtime": "docker",
      "autohealing": false
    }
  ]
}

curl -XGET  -H "X-Auth-Token: ${SCALEWAY_TOKEN}"  "https://api.scaleway.com/k8s/v1beta2/regions/fr-par/clusters/${CLUSTER_ID}/nodes"
{
  "total_count": 2,
  "nodes": [
    {
      "region": "fr-par",
      "id": "16b716e8-1c7a-4486-9861-067717cd44ea",
      "cluster_id": "ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b",
      "created_at": "2019-11-11T19:47:52.240391Z",
      "updated_at": "2019-11-11T21:06:40.605505Z",
      "pool_id": "feb2c164-8805-4130-b5d3-57889dc35652",
      "status": "ready",
      "npd_status": {
        "DiskPressure": "False",
        "KernelDeadlock": "False",
        "MemoryPressure": "False",
        "NetworkUnavailable": "False",
        "PIDPressure": "False",
        "Ready": "True"
      },
      "name": "scw-k8s-test-default-16b716e81c7a4",
      "public_ip_v4": "51.158.69.231",
      "public_ip_v6": null
    },
    {
      "region": "fr-par",
      "id": "3cb12e55-883e-404e-9617-9716a1ab22aa",
      "cluster_id": "ae336b4a-7875-4150-8b1c-ed1e4c9f3b2b",
      "created_at": "2019-11-11T19:47:54.410118Z",
      "updated_at": "2019-11-11T21:06:40.732254Z",
      "pool_id": "feb2c164-8805-4130-b5d3-57889dc35652",
      "status": "notready",
      "npd_status": {
        "DiskPressure": "Unknown",
        "KernelDeadlock": "False",
        "MemoryPressure": "Unknown",
        "NetworkUnavailable": "False",
        "PIDPressure": "Unknown",
        "Ready": "Unknown"
      },
      "name": "scw-k8s-test-default-3cb12e55883e4",
      "public_ip_v4": "51.15.217.30",
      "public_ip_v6": null
    }
  ]
}

kubectl get nodes         
NAME                                             STATUS                        ROLES    AGE   VERSION
scw-k8s-test-default-16b716e81c7a4   Ready                         <none>   77m   v1.15.5
scw-k8s-test-default-3cb12e55883e4   NotReady,SchedulingDisabled   <none>   76m   v1.14.8

Upgrades are pretty quick (<5 minutes) if your cluster is empty, as was mine, but as usual, your mileage might vary. However, they aren’t fully non-disruptive - checking the negative results of the vegeta run, we see there were a few timeouts:

less results.json | jq '. | select (.code != 200)'
{
  "attack": "",
  "seq": 41,
  "code": 0,
  "timestamp": "2019-11-11T22:02:41.187717035+01:00",
  "latency": 30000794729,
  "bytes_out": 0,
  "bytes_in": 0,
  "error": "Get http://51.159.25.121: net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
  "body": null
}
{
  "attack": "",
  "seq": 42,
  "code": 0,
  "timestamp": "2019-11-11T22:02:42.187457957+01:00",
  "latency": 30000935369,
  "bytes_out": 0,
  "bytes_in": 0,
  "error": "Get http://51.159.25.121: net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
  "body": null
}
{
  "attack": "",
  "seq": 43,
  "code": 0,
  "timestamp": "2019-11-11T22:02:43.187827619+01:00",
  "latency": 30000555385,
  "bytes_out": 0,
  "bytes_in": 0,
  "error": "Get http://51.159.25.121: net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
  "body": null
}

It looks like the Load balancer’s health check isn’t frequent enough to detect that the backend is down during upgrades. The Load Balancer API docs show that it is possible to tweak the delay between checks (not available via the web UI though). As mentioned before, there are a few (not yet publicly documented) annotations we can use on our service object for that (the available choices and detailed explanations are available on the Load Balancer API page):

service.beta.kubernetes.io/scw-loadbalancer-health-check-delay # time between two consecutive health checks, in milliseconds
service.beta.kubernetes.io/scw-loadbalancer-health-check-timeout # additional check timeout, after the connection has been already established
service.beta.kubernetes.io/scw-loadbalancer-health-check-max-retries # number of consecutive unsuccessful health checks, after wich the server will be considered dead

This is what our Service object could look like (example values, the precise ones will vary depending on your environment):

apiVersion: v1
kind: Service
metadata:
  name: golang-test
  annotations:
    service.beta.kubernetes.io/scw-loadbalancer-health-check-delay: 10000 # a check every 10s
    service.beta.kubernetes.io/scw-loadbalancer-health-check-timeout:  3000 # a 3s timeout, which can be dangerously low
    service.beta.kubernetes.io/scw-loadbalancer-health-check-max-retries: 2 # 2 retries
spec:
  selector:
    name: golang-test
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer

Summary

Pros:

quick to provision and good provisioning tooling (API, terraform)
good amount of features/integrations (cluster upgrades, autohealing, optional installation of dashboard, ingress for Kapsule, automatic cetrs via Let’s Encrypt and various healthchecks for Load Balancer)
Container Registry is good enough and inexpensive (0.025 eur/GB/month for storage and 0.03 eur/GB/month traffic outside of the same region)
Kubernetes versions come out pretty quickly (~1 week for 1.16)
ecosystem is good - Scaleway have managed Databases, Object and Block storage, Load Balancers (which will soon be multi-cloud) and a few potentially very interesting services in beta/preview like Serverless Functions, VPCs, IoT Hub, AI Inference, Domains (which includes promissing features like “Dynamic routing capabilities: balance traffic depending on resource health and more to come” ), striking, for me, a fine balance between more basic platforms (Linode, Vultr, Digital Ocean to an extent) and full featured clouds (AWS, GCP, Azure) while still retaining pricing closer to the former

Cons:

only relatively expensive instance types are available (starting at 40 eur/month), for now, making Kapsule less affordable compared to the main competitors - hopefully that will change soon, and we might even get bare metal nodes one day
Container Registry auth requires a full Scaleway token, which gives full read/write API access - read-only tokens would be a cool addition
documentation is somewhat lacking in some areas
there’s no way to monitor/get statistics from services like Load Balancer or Container Registry; for Kapsule Heapster and metrics-server are preinstalled, so there’s access to Kubernetes-level statistics, but nothing else
there’s no way to create and manage Scaleway ressources (other than Load Balancer and block storage) from inside Kapsule, like with GCP’s Cloud Connector (Note: you can probably use AppsCode Kubeform for that, since it’s just a Kubernetes CRD wrapper around terraform providers, and Scaleway’s terraform provider is pretty decent)
cluster upgrades can result in small (a few seconds) downtimes with the default configuration, some fine tuning is required

So, in conclusion, Kapsule and the Scaleway stack are very interesting and evolving quickly, but come with some rough edges (well, Kapsule is still in beta, so completely normal). Once those are polished, Scaleway Kapsule and the rest of their ecosystem will make for a very compelling development / small-to-mid scale cloud environment, with just the right amount of managed services and features for a bargain price, and EU data sovereignty, which can be a requirement/plus in some cases.

Using HashiCorp Sentinel to validate Terraform configuration/plan

adrian.todorov@atodorov.ninja (Adrian Todorov) — Thu, 23 May 2019 15:22:15 +0200

Let me preface this by saying that i really like HashiCorp, their products and their open core business model. Their Enterprise stack is great with awesome features (Vault selective intercluster replication, Terraform team collaboration tooling, Sentinel Policy as Code, etc.), but if itss for personal use or a small business case that doesn’t require all Terraform Enterpirse Features and associated budget, using the freely available Sentinel Simulator in a somewhat hacky way could be a viable option (certainly beats having to write a custom tool to achieve the same thing).

Use case

I have some terraform configuration files, and mostly, plan files that i’d like to validate before applying. Things like the terraform version is appropriate, resources to add aren’t too big/expensive, there’s always a tag with some key, etc. I could write a custom tool to achieve that, or i could take advantage of Sentinel, HashiCorp’s Policy as Code framework/tool, which is sadly closed source and available only in Enterprise versions of HashiCorp’s stack. However, HashiCorp have Sentinel Simulator freely available to download, which “enables you to develop and test Sentinel policies locally on your own machine or in a CI environment”, which sounds great for what i need.

Getting started with Sentinel Simulator

Start by downloading Sentinel Simulator’s latest version for your OS and architecture, setting the appropriate permissions (chmod +x sentinel on *nix) and executing the binary, which should result in a similar output:

./sentinel
Usage: sentinel [--version] [--help] <command> [<args>]

Available commands are:
    apply      Execute a policy and output the result
    doc        Show documentation for an import from a doc file
    fmt        Format Sentinel policy to a canonical format
    test       Test policies
    version    Prints the Sentinel Simulator version

Once that’s out of the way, you can start writing a policy - which is just a file with rules that assert and parse conditions and return true/false (pass/fail).

An example policy is the following:

# file: basic.sentinel
hour = 4
main = rule { hour >= 0 and hour < 12 }

With it, we declare a static variable hour which is 4, and in the main rule (each policy needs to have a main rule) we check if the hour variable is between 0 and 12. Since it is, we’ll get a true/pass with exit code 0 when doing a sentinel apply:

sentinel apply basic.sentinel
Pass

Rules can be very complex (Sentinel’s rule getting started guide), and most interestingly, can use dynamic data sources with imports, which are, roughly speaking, plugins that communicate with an external data source like time, terraform plan output, etc.

An example use of the time import and more complex rules (which do the same thing, checking if the hour is between 0 and 12), with a print:

# file: time.sentinel
import "time"

hour = time.now.hour
minute = time.now.minute
print("current time is", hour, minute)

past_midnight = rule { hour >= 0 } 
before_noon   = rule { hour < 12 }

main = rule {
    past_midnight and 
    before_noon
}

Here we’re importing the “time” plugin, and using the time.now.hour attribute (which gives us the current hour) to compare against 0 and 12, allowing us to do time-based rules (e.g. enforce “read-only Friday”).

In my case, since it’s 15:25, it will fail:

date
Thu May 23 15:25:29 UTC 2019
./sentinel apply time.sentinel
Fail

Execution trace. The information below will show the values of all
the rules evaluated and their intermediate boolean expressions. Note that
some boolean expressions may be missing if short-circuit logic was taken.

Print messages:

current time is 15 25

FALSE - time.sentinel:10:1 - Rule "main"
  TRUE - time.sentinel:11:5 - past_midnight
    TRUE - time.sentinel:7:24 - hour >= 0
  FALSE - time.sentinel:12:5 - before_noon
    FALSE - time.sentinel:8:24 - hour < 12

FALSE - time.sentinel:8:1 - Rule "before_noon"

TRUE - time.sentinel:7:1 - Rule "past_midnight"

You can do much more than simple comparisons (contains, is, not contains, is not, is not undefined, etc. - full spec of the Sentinel language here)

Advanced reading

The good stuff

Sentinel, when used as a part of Terraform Enterpirse, has some terraform-related plugins like tfplan, which allow to do validations based on planned values. Sentinel Simulator, however, doesn’t have those closed source plugins, so the only way to use it in a similar fashion is via its JSON parser plugin, since terraform configuration and plan can be previewed as JSON (Note: terraform show (plan|state) -json exists only in terraform 0.12, for prior versions you’d need to use a tool like tfjson).

If you are already on terraform 0.12, skip to the relevant section.

Using tfjson

First you need to install a (functional) version of tfjson, like tapirs/tfjson via go get:

go get github.com/tapirs/tfjson
tfjson
usage: tfjson terraform.tfplan

As an example, I manage my Digital Ocean Kubernetes cluster via terraform, I’ll increase the node_count to 3, and do a terraform plan -out terraform.plan :

terraform plan -out terraform.plan                                                                                                                    
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

digitalocean_kubernetes_cluster.my_digital_ocean_cluster: Refreshing state... (ID: f8ee8505-af76-437e-864f-3751371374d8)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ digitalocean_kubernetes_cluster.my_digital_ocean_cluster
      node_pool.0.node_count: "2" => "3"


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: terraform.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "terraform.plan"

Running tfjson on the terraform.plan file results in a JSON with the resources that will get added/modified with the appropriate values:

tfjson terraform.plan
{
    "destroy": false,
    "digitalocean_kubernetes_cluster.my_digital_ocean_cluster": {
        "destroy": false,
        "destroy_tainted": false,
        "node_pool.0.node_count": "3"
    }
}

Sentinel can read JSON, either inline via the json plugin or via the global values configuration, which is available as a configuration file (sentinel apply -config config.json) or an a command-line flag (sentinel apply -global key=value).

Assuming tfjson’s output, an example policy that checks destroy is true (meaning that applying the plan will result in destruction of resources - normally you’d want to check the opposite but this is just a test) :

# file: tf.policy

print("input.destroy is",input.destroy)
main = rule {input.destroy is true}

Now we just need to pass the output of tfjson as the global variable input to Sentinel:

sentinel apply -global input="`tfjson terraform.plan`" tf.policy
Fail

Execution trace. The information below will show the values of all
the rules evaluated and their intermediate boolean expressions. Note that
some boolean expressions may be missing if short-circuit logic was taken.

Print messages:

input.destroy is false

FALSE - tf.policy:2:1 - Rule "main"

As we can see, the JSON output of tfjson was successfully read, parsed and evaluated.

Sadly, this would only work for the simplest of checks (destroy is false) due to the way tfjson’s output is structured and Sentinel parses JSON (you can’t iterate on root-level keys). To be able to do more complex checks, a hacky workaround is required, to encapsulate tfjson’s output before passing it to Sentinel:

sentinel apply -global input="{\"data\": `tfjson terraform.plan`}" tf.policy

With the slighly modified policy:

# file: tf.policy

print("input.data.destroy is", input.data.destroy)

main = rule {input.data.destroy is true}

Now a more complex policy with multiple advanced checks:

# file: tf.policy

print("input.data.destroy is", input.data.destroy)

no_destroys = rule {
	input.data.destroy is false
}

dont_touch_my_kube = func(data) {
	for data as i, j {
		// skipping the destroy value since there is a dedicated rule for it
		if i == "destroy" {
			continue
		} else {
			print(i)
			if i contains "digitalocean_kubernetes_cluster" {
				print("Refusing modification of digitalocean_kubernetes_cluster resources")
				return (false)
			}
		}
	}
	return(true)
}

main = rule { no_destroys and dont_touch_my_kube(input.data) }

Result:

sentinel apply -global input="{\"data\": `tfjson terraform.plan`}" tf.policy 
Fail

Execution trace. The information below will show the values of all
the rules evaluated and their intermediate boolean expressions. Note that
some boolean expressions may be missing if short-circuit logic was taken.

Print messages:

input.data.destroy is false
digitalocean_kubernetes_cluster.my_digital_ocean_cluster
Refusing modification of digitalocean_kubernetes_cluster resources

FALSE - tf.policy:23:1 - Rule "main"
  TRUE - tf.policy:23:15 - no_destroys
    TRUE - tf.policy:4:2 - input.data.destroy is false
  FALSE - tf.policy:23:31 - dont_touch_my_kube(input.data)

TRUE - tf.policy:3:1 - Rule "no_destroys"

Since there are no destroys, the first rule is successful, but the second (which is a function to be able to properly parse the JSON object) fails since there are modifications to be done on a digitalocean_kubernetes_cluster resource, which isn’t allowed.

Terraform show -json (starting from terraform 0.12)

Since terraform version 0.12 (which includes some very cool new features, if you haven’t checked it out here’s the announcement blog post), there’s a -json option to terraform show which works on terraform plans, so you can preview them in JSON format, perfect for checks via Sentinel or other tools. Let’s see what that looks like with Sentinel:

terraform -v
Terraform v0.12.0
+ provider.digitalocean v1.3.0
terraform plan -out terraform.plan
terraform show -json terraform.plan 
...

This should result in a pretty sizeable JSON, including:

terraform version
providers’ configuration
all current resources
planned changes
output changes
yet unknown values
variables

Overall, it’s pretty cool and you can get wild with policies. Off the top of my head:

providers’ configuration doesn’t have static values, only variables / Vault data source
terraform version is good
there are no variables with X or Y (like passwords or generic admin username)
no resources are used at the root level (only modules)
and the plain old (value X isn’t bigger than Y), like AWS/GCP/Azure instance types, Lambda memory settings, security groups SSH open from 0.0.0.0/, etc.

A few examples for inspiration:

# file tf.policy
allowed_node_sizes = [
	"s-1vcpu-2gb",
]

allowed_regions = [
	"ams3",
]

allowed_tf_versions = [
	"0.12.0",
]

node_pool_max_size = 3

tf_version = rule {
	input.terraform_version in allowed_tf_versions
}

no_destroys = func() {
	for input.resource_changes as change {
		if change.change.actions contains "destroy" {
			print("No destroys allowed")
			return (false)
		}
	}
	return (true)
}

max_nodepool_size = func() {
	for input.resource_changes as change {
		if change.address contains "digitalocean_kubernetes_cluster" {
			for change.change.after.node_pool as pool {
				if pool.node_count > node_pool_max_size {
					error("Node pool", pool.name, "shouldn't exceed max node size", node_pool_max_size, "it's currently", pool.node_count)
				}
			}
		}
	}
	return (true)
}

region_check = func() {
	for input.configuration.root_module.resources as resource {
		if resource.expressions.region.constant_value not in allowed_regions {
			return (false)
		}
	}
	return (true)
}

node_size = func() {
	for input.resource_changes as change {
		if change.address contains "digitalocean_kubernetes_cluster" {
			for change.change.after.node_pool as pool {
				if pool.size not in allowed_node_sizes {
					return (false)
				}
			}
		}
	}
	return (true)
}

main = rule {
	tf_version and no_destroys() and region_check() and max_nodepool_size() and node_size()
}

This example policy checks the following:

node types
number of nodes
no destroys
terraform version hasn’t been updated
no resources are created outside of the allowed region (AMS3)

To run it, you need to apply sentinel with terraform show’s output:

sentinel apply -global input="`terraform show -json terraform.plan`" tf.policy

It took me roughly 15 minutes to write and test basic functionality, and i have a decent chunk of stuff covered (for my limited use case).

TL;DR

Sentinel is an awesome Policy As Code tool from HashiCorp which can be run locally via Sentinel Simulator and be used to validate any sort of JSON, like the output from a terraform plan.

Coming up next

While searching for a Policy/Validation as code tool, i asked around the DevOps’ish Telegram group and someone proposed conftest, which seems more flexible, advanced and complicated (with a Go-like syntax compared to Sentinel’s HCL-like DSL), not to mention free, open source and usable directly without workarounds. I’d like to give it a try and compare the two tools.

Creating a static blog with Hugo on Firebase Hosting delivered with Drone CI

adrian.todorov@atodorov.ninja (Adrian Todorov) — Tue, 23 Oct 2018 20:22:21 +0200

Preface

Hello there. After years of hesitation, i finally got the courage to start my blog as an excuse to play with some cool tech i don’t have the chance to do at work. Of course, there is no fun in using something easy like WordPress, so i chose Hugo, a static site generator. This way i can have everything in Git, and have a CI/CD pipeline that deploys it somewhere.

The stack

I’ve been eyeing Hugo for more than a year, and i prefer its syntax and flexibility to Jekyll, so that’s that. For Git GitHub should do the trick; for CI/CD, i’ve always liked Drone on paper - configuration is in simple .yml file per repo, everything is a plugin (which in itself is just a Docker container), and it seems like a good opportunity to give it an actual go.

Originally i thought about using AWS S3 for hosting the static files, and CloudFront for CDN and SSL, but i already work with AWS and it’s something i’ve already done, so there was no challenge. Google Cloud Platform it is then, with its nice “Always Free” tier.

However, upon some reading on the Google Cloud Storage (GCS) docs and Henry Lawson’s blog post on the subject i realised that blindly applying what i know about AWS won’t work. Like him, i imagined it the following way:

Static content gets uploaded to GCS
Google Cloud CDN serves it with a custom SSL cert

However, there are a few problems with that - GCS doesn’t need CDN because it does caching by default, but it doesn’t support SSL with custom domains. Not a problem, i’ll just use Cloud CDN for that! Well, apparently it requires a Load Balancer, which costs money and seems way too overkill.

So, Firebase Hosting enters the stage. It’s oriented towards static site hosting, supports SSL with custom domains and also has an always free tier. Furthermore, the Firebase suite has some pretty interesting products which can come to use for a static blog/ongoing experiment, most notably Cloud Firesotre, a managed NoSQL database, which, coupled with Cloud Functions i can use to add some dynamic parts.

Roadmap

I’m assuming you already have Hugo installed and running - if that’s not the case, you can check out Hugo’s Quick Start guide.

I’ll start by setting up Firebase and DroneCI, and then creating a pipeline to generate the static files with Hugo and deploy them to Firebase Hosting. Once that’s up and running, i’ll look into adding the following:

tests before deploying the pipeline - a spellchecker, for instance
a dynamic comment section and a newsletter with the help of Cloud Functions and other parts of the Firebase/Google Cloud Platform suite
search with lunr.js, fuse.js, algolia or something else

So, let’s get started.

Getting started with Firebase Hosting

First you need to create your Firebase project. To do that, you have to connect to the Firebase Console with a Google account, and click on the big “Add project” button. There you have to create a project with a cool name (in my case it’s just “Blog”, you can be more original than that), or use an existing one from Google Cloud Platform. Once its created / selected, we arrive at the pretty slick looking Firebase console. There are plenty of features, but for now i’ll just stick to the CLI (easier to document, gitify, redo one day if necessary).

To start using the Firebase CLI, you need to have Node.js and npm installed, afterwards it’s as easy a single command to install and another one to login (which opens a page where you sign in with your Google account and authorise the CLI to access your Google Cloud Platform):

npm install -g firebase-tools
firebase login

Then you have to create a folder which will contain our project (or if you already have a folder with Hugo, just cd inside), and initialise the Firebase CLI in it:

mkdir blog && cd blog
firebase init

The CLI then launches an interactive setup which asks questions, like which services do we want to setup (for now just Hosting ), what Firebase project are we working on, security rules, the folder we want to publish to Firebase Hosting (public by default and a sensible choice considering it’s also Hugo’s default):

You're about to initialize a Firebase project in this directory:

  /home/ato/blog

? Which Firebase CLI features do you want to setup for this folder?
Press Space to select features, then Enter to confirm your choices.
 ◯ Database: Deploy Firebase Realtime Database Rules
 ◯ Firestore: Deploy rules and create indexes for Firestore
 ◯ Functions: Configure and deploy Cloud Functions
❯◉ Hosting: Configure and deploy Firebase Hosting sites
 ◯ Storage: Deploy Cloud Storage security rules

=== Project Setup

First, let's associate this project directory with a Firebase project.
You can create multiple project aliases by running firebase use --add,
but for now we'll just set up a default project.

? Select a default Firebase project for this directory: (Use arrow keys)
  [don't setup a default project]
❯ Blog (blog-314450)
  [create a new project]

=== Hosting Setup

Your public directory is the folder (relative to your project directory) that
will contain Hosting assets to be uploaded with firebase deploy. If you
have a build process for your assets, use your build's output directory.

? What do you want to use as your public directory? (public)
? Configure as a single-page app (rewrite all urls to /index.html)? (y/N)
✔  Wrote public/404.html
✔  Wrote public/index.html

i  Writing configuration info to firebase.json...
i  Writing project information to .firebaserc...

✔  Firebase initialization complete!

So now you have the Firebase CLI installed and configured.

Since it installed a default index.html in the public folder (public in my case), you can test if everything is running fine locally with the following command:

firebase serve --only hosting
=== Serving from '/home/ato/blog'...

i  hosting[blog-314450]: Serving hosting files from: public
✔  hosting[blog-314450]: Local server: http://localhost:5000

Opening http://localhost:5000 should give us the default Firebase page.

Now, let’s deploy it to Firebase Hosting:

firebase deploy                                       
=== Deploying to 'blog-314450'...
i  hosting[blog-314450]: beginning deploy...
i  hosting[blog-314450]: found 2 files in public
✔  hosting[blog-314450]: file upload complete
i  hosting[blog-314450]: finalizing version...
✔  hosting[blog-314450]: version finalized
i  hosting[blog-314450]: releasing new version...
✔  hosting[blog-314450]: release complete

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/blog-314450/overview
Hosting URL: https://blog-314450.firebaseapp.com

If you go check the URL at the end, you should see the same default Firebase index.html.

So far, Firebase Hosting is up and running. Time to deploy our Hugo generated website to it!

Generating static files with Hugo and deploying them on Firebase Hosting

Generating static HTML with Hugo is as simple as running a single command, which will put the files in the public folder (by default, modifiable with the publishDir parameter in your site’s config):

hugo
Building sites …
                   | EN  
+------------------+----+
  Pages            | 37  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     | 14  
  Processed images |  0  
  Aliases          | 13  
  Sitemaps         |  1  
  Cleaned          |  0

And that’s it, you should now have the HTML ready in the public folder, and all that’s left is to run firebase deploy and deploy to your website on Firebase Hosting:

➜  blog git:(master) ✗ firebase deploy                                                                                                                                                                

=== Deploying to 'blog-314450'...

i  deploying hosting

i  hosting[blog-314450]: beginning deploy...
i  hosting[blog-314450]: found 67 files in public
✔  hosting[blog-314450]: file upload complete
i  hosting[blog-314450]: finalizing version...
✔  hosting[blog-314450]: version finalized
i  hosting[blog-314450]: releasing new version...
✔  hosting[blog-314450]: release complete

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/blog-314450/overview
Hosting URL: https://blog-314450.firebaseapp.com

Refreshing the webpage should now show your webite up and running, fully static and for free.

Now, that was simple enough, but you don’t want to actually do this manually every time, so let’s set up a CI/CD pipeline that will rebuild the Hugo site and deploy it to Firebase every time there’s a new commit (it goes without saying, Hugo uses text files for everything, and if it’s text and it matters, it should be stored in Git).

Getting started with Drone

Drone is a cloud-native CI/CD system, heavily based around Docker. Pipelines are defined with YAML files, contain multiple stages and each stage is a plugin (which is a Docker container that uses environment variables to read it’s configuration and it does stuff). There’s a plethora of official plugins, and it’s pretty easy to create a new one in Golang or even Bash - basically anything that could run inside Docker. There’s already an existing plugin for Hugo, which will handle the generation of static files.

Note: at the time of writing, the latest version of Drone is 0.8

“Installation” is pretty easy - you just have to run two Docker containers available on Docker Hub. It’s even easier with Docker Compose, which orchestrates that with a relatively simple YAML file (for getting started with it, you can check Digital Ocean’s great installation guide ).

After you have Docker Compose up and running, create the following docker-compose.yml file:

# Docker Compose configuration version
version: '2'

# With the services block we define each "service" (Docker container and associated configuration) we want to run
services:
  drone-server: # drone-server, which is the backend
    image: drone/drone:0.8 # the Docker image we're using, which is drone/drone (group/project:version) from the public Docker Hub
    ports: # the list of ports to expose - with an optional host-port:container port mapping
      - 80:8000
      - 9000
    volumes: # Docker Volumes, the way to manage persistent data, in this case Drone's internal SQLite database
      - drone-server-data:/var/lib/drone/
    restart: always # If the container dies / crashes, dockerd will restart it
    environment: # Drone-specific environment variables we use to configure our container
      - DRONE_OPEN=true # is our Drone instance publicly available, true or false
      - DRONE_HOST=${DRONE_HOST} # the URL on which Drone will be accessible, in a <scheme>://<hostname> format
      - DRONE_SECRET=${DRONE_SECRET} # a random secret string shared between drone and drone-agent to secure communication
      # activate GitHub authentication for Drone - http://docs.drone.io/install-for-github/
      - DRONE_GITHUB=true
      - DRONE_GITHUB_CLIENT=${DRONE_GITHUB_CLIENT}
      - DRONE_GITHUB_SECRET=${DRONE_GITHUB_SECRET}

  drone-agent: # drone-agent, which is the frontend
    image: drone/agent:0.8 #
    command: agent # a command to execute upon starting the container, in this case launching Drone Agent
    restart: always
    depends_on: # explicit dependency to make sure the Agent will start only after Drone Server is alive and well
      - drone-server
    volumes: # we're passing the Docker socket so that Drone Agent can manage the host's dockerd and spawn containers on it
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=drone-server:9000 # drone-server will resolve to the drone-server container's IP automagically
      - DRONE_SECRET=${DRONE_SECRET} # same secret we passed to drone-server
volumes:  # we define the drone-server-data volume
  drone-server-data:

Highlighted are the lines you probably should change, as well as those that require your input - ${SOMETHING} needs to be replaced with the appropriate value (GitHub credentials, Drone URL, etc.). The most important ones are DRONE_SECRET and the version control configuration (GitHub, GitLab, Gitea, Gogs and BitBucket Cloud are supported ).

Once you’ve modified the file to your liking, run the following command to get everything up:

docker-compose up

If everything is fine, you should see something like this (basically docker-compose creating the Docker containers and what they need to run(virtual network, volume) and then Drone’s logs, which are pretty verbose ):

Creating network "dronecompose_default" with the default driver
Creating dronecompose_drone-server_1
Creating dronecompose_drone-agent_1
Attaching to dronecompose_drone-server_1, dronecompose_drone-agent_1
drone-server_1  | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
drone-server_1  |  - using env:	export GIN_MODE=release
drone-server_1  |  - using code:	gin.SetMode(gin.ReleaseMode)
drone-server_1  |
drone-server_1  | [GIN-debug] GET    /logout                   --> github.com/drone/drone/server.GetLogout (12 handlers)
drone-server_1  | [GIN-debug] GET    /login                    --> github.com/drone/drone/server.HandleLogin (12 handlers)
drone-server_1  | [GIN-debug] GET    /api/user                 --> github.com/drone/drone/server.GetSelf (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/user/feed            --> github.com/drone/drone/server.GetFeed (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/user/repos           --> github.com/drone/drone/server.GetRepos (13 handlers)
drone-server_1  | [GIN-debug] POST   /api/user/token           --> github.com/drone/drone/server.PostToken (13 handlers)
drone-server_1  | [GIN-debug] DELETE /api/user/token           --> github.com/drone/drone/server.DeleteToken (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/users                --> github.com/drone/drone/server.GetUsers (13 handlers)
drone-server_1  | [GIN-debug] POST   /api/users                --> github.com/drone/drone/server.PostUser (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/users/:login         --> github.com/drone/drone/server.GetUser (13 handlers)
drone-server_1  | [GIN-debug] PATCH  /api/users/:login         --> github.com/drone/drone/server.PatchUser (13 handlers)
drone-server_1  | [GIN-debug] DELETE /api/users/:login         --> github.com/drone/drone/server.DeleteUser (13 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name   --> github.com/drone/drone/server.PostRepo (16 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name   --> github.com/drone/drone/server.GetRepo (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/builds --> github.com/drone/drone/server.GetBuilds (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/builds/:number --> github.com/drone/drone/server.GetBuild (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/logs/:number/:pid --> github.com/drone/drone/server.GetProcLogs (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/logs/:number/:pid/:proc --> github.com/drone/drone/server.GetBuildLogs (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/files/:number --> github.com/drone/drone/server.FileList (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/files/:number/:proc/\*file --> github.com/drone/drone/server.FileGet (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/secrets --> github.com/drone/drone/server.GetSecretList (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/secrets --> github.com/drone/drone/server.PostSecret (16 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/secrets/:secret --> github.com/drone/drone/server.GetSecret (16 handlers)
drone-server_1  | [GIN-debug] PATCH  /api/repos/:owner/:name/secrets/:secret --> github.com/drone/drone/server.PatchSecret (16 handlers)
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name/secrets/:secret --> github.com/drone/drone/server.DeleteSecret (16 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/registry --> github.com/drone/drone/server.GetRegistryList (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/registry --> github.com/drone/drone/server.PostRegistry (16 handlers)
drone-server_1  | [GIN-debug] GET    /api/repos/:owner/:name/registry/:registry --> github.com/drone/drone/server.GetRegistry (16 handlers)
drone-server_1  | [GIN-debug] PATCH  /api/repos/:owner/:name/registry/:registry --> github.com/drone/drone/server.PatchRegistry (16 handlers)
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name/registry/:registry --> github.com/drone/drone/server.DeleteRegistry (16 handlers)
drone-server_1  | [GIN-debug] PATCH  /api/repos/:owner/:name   --> github.com/drone/drone/server.PatchRepo (16 handlers)
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name   --> github.com/drone/drone/server.DeleteRepo (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/chown --> github.com/drone/drone/server.ChownRepo (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/repair --> github.com/drone/drone/server.RepairRepo (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/move --> github.com/drone/drone/server.MoveRepo (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/builds/:number --> github.com/drone/drone/server.PostBuild (16 handlers)
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name/builds/:number --> github.com/drone/drone/server.ZombieKill (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/builds/:number/approve --> github.com/drone/drone/server.PostApproval (16 handlers)
drone-server_1  | [GIN-debug] POST   /api/repos/:owner/:name/builds/:number/decline --> github.com/drone/drone/server.PostDecline (16 handlers)
drone-agent_1   | {"time":"2018-09-26T20:52:39Z","level":"debug","message":"request next execution"}
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name/builds/:number/:job --> github.com/drone/drone/server.DeleteBuild (16 handlers)
drone-server_1  | [GIN-debug] DELETE /api/repos/:owner/:name/logs/:number --> github.com/drone/drone/server.DeleteBuildLogs (16 handlers)
drone-server_1  | [GIN-debug] GET    /api/badges/:owner/:name/status.svg --> github.com/drone/drone/server.GetBadge (12 handlers)
drone-server_1  | [GIN-debug] GET    /api/badges/:owner/:name/cc.xml --> github.com/drone/drone/server.GetCC (12 handlers)
drone-server_1  | [GIN-debug] POST   /hook                     --> github.com/drone/drone/server.PostHook (12 handlers)
drone-server_1  | [GIN-debug] POST   /api/hook                 --> github.com/drone/drone/server.PostHook (12 handlers)
drone-server_1  | [GIN-debug] GET    /stream/events            --> github.com/drone/drone/server.EventStreamSSE (12 handlers)
drone-server_1  | [GIN-debug] GET    /stream/logs/:owner/:name/:build/:number --> github.com/drone/drone/server.LogStreamSSE (15 handlers)
drone-server_1  | [GIN-debug] GET    /api/info/queue           --> github.com/drone/drone/server.GetQueueInfo (13 handlers)
drone-server_1  | [GIN-debug] GET    /authorize                --> github.com/drone/drone/server.HandleAuth (12 handlers)
drone-server_1  | [GIN-debug] POST   /authorize                --> github.com/drone/drone/server.HandleAuth (12 handlers)
drone-server_1  | [GIN-debug] POST   /authorize/token          --> github.com/drone/drone/server.GetLoginToken (12 handlers)
drone-server_1  | [GIN-debug] GET    /api/builds               --> github.com/drone/drone/server.GetBuildQueue (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/         --> github.com/drone/drone/server/debug.IndexHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/heap     --> github.com/drone/drone/server/debug.HeapHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/goroutine --> github.com/drone/drone/server/debug.GoroutineHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/block    --> github.com/drone/drone/server/debug.BlockHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/threadcreate --> github.com/drone/drone/server/debug.ThreadCreateHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/cmdline  --> github.com/drone/drone/server/debug.CmdlineHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/profile  --> github.com/drone/drone/server/debug.ProfileHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/symbol   --> github.com/drone/drone/server/debug.SymbolHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] POST   /api/debug/pprof/symbol   --> github.com/drone/drone/server/debug.SymbolHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /api/debug/pprof/trace    --> github.com/drone/drone/server/debug.TraceHandler.func1 (13 handlers)
drone-server_1  | [GIN-debug] GET    /metrics                  --> github.com/drone/drone/server/metrics.PromHandler.func1 (12 handlers)
drone-server_1  | [GIN-debug] GET    /version                  --> github.com/drone/drone/server.Version (12 handlers)
drone-server_1  | [GIN-debug] GET    /healthz                  --> github.com/drone/drone/server.Health (12 handlers)

The first part indicates which container is the log coming from (most of them during startup are from drone-server; during builds they’re mostly on the agent side), what system is it coming from (GIN is an HTTP framework for Golang) and various details. All seems fine, so let’s do our first pipeline!

I'd highly recommend you setup an SSL cert ([Drone docs about that](http://docs.drone.io/configure-ssl/), or even [Let's Encrypt](http://docs.drone.io/configure-lets-encrypt/) which is also supported by Drone)

Creating your first pipeline with Drone

Connect to Drone’s web UI, which is available at DRONE_HOST. Once there, it will automatically log you in / demand that you log in with your version control sytem (Drone doesn’t handle authentification, it delegates that part to the VCS). After that, you can enable DroneCI on the projects you want it, add a .drone.yml file, commit it to master and off we go!

A .drone.yml file is just a bunch of YAML which defines our pipeline and its stages (which can be based on regular Docker containers with the commands parameter or special Drone plugins (which are special Docker containers)), with conditions (only run stage X on master/tag; if the pipeline succeeds/fails notify via Slack, etc. etc.)

pipeline:
  hugo:
    image: plugins/drone-hugo:latest
    validate: true

*plugins* points to the [DockerHub organisation](https://hub.docker.com/r/plugins/) for drone-plugins

This defines a pipeline with a single stage, called “hugo”, which uses the plugins/drone-hugo:latest image from DockerHub (if you specify the full URL it can be a custom Docker Registry), with the validate:true option (it will check the configuration files for errors). You can read the full documentation of the Drone Hugo Plugin, but that’s the base, with optional parameters like theme, buildDrafts, etc.

So now all that is left is to add the .drone.yml file to your repo and push it to GitHub/equivalent.

Commit and push, and now the pipeline should run successfully:

+ hugo check
Contains some verification checks

+ hugo

                   | EN  
+------------------+----+
  Pages            | 22  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     | 14  
  Processed images |  0  
  Aliases          |  7  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 312 ms

Now that you have the static files generated, the next step is having them deployed to Firebase Hosting.

Adding a deploy to Firebase step in our pipeline

First you need to deal with authentification. Firebase supports token authentification (instead of with your regular Google Account), and for that you need to run the following command:

firebase login:ci

Which will open a browser window, and guide you through connecting to your Google Account and giving the appropriate rights to the token, and in the end print out the token.

As you know, every time you commit credentials to Git, little kittens die. Luckily Drone can easily manage secrets with the Drone CLI (which you need tp install and login into first). To add a new secret to the Drone store, use the drone secret add command:

drone secret add \
  -repository sofixa/blog \
  -image sofixa/drone-firebase \
  -name token \
  -value "$FIREBASE_TOKEN"

This would add a secret available to the sofixa/blog repository, only on stages running with the sofixa/drone-firebase image, with the name token and value of $FIREBASE_TOKEN(which you should replace with the token you got from firebase login:ci).

To add a second stage which deploys to Firebase using the token secret you’ve just added, modify the .drone.yaml :

pipeline:
  hugo:
    image: plugins/hugo:latest
    validate: true
  firebase:
    image: sofixa/drone-firebase
    project_id: blog-314450
    message: Autodeploy of commit $$COMMIT
    targets: hosting
    # this will extrapolate the token secret as a parameter with its value
    secrets:
      - source: token
        target: plugin_token

Now commit and push to your Git Repository, and Drone should pick up an extra step (firebase) and deploy your Hugo generated static files to Firebase hosting with a similar output:

Now using project blog-314450

=== Deploying to 'blog-314450'...

i  deploying hosting
i  hosting[blog-314450]: beginning deploy...
i  hosting[blog-314450]: found 47 files in public
i  hosting: adding files to version [0/47] (0%)
✔  hosting[blog-314450]: file upload complete
i  hosting[blog-314450]: finalizing version...
✔  hosting[blog-314450]: version finalized
i  hosting[blog-314450]: releasing new version...
✔  hosting[blog-314450]: release complete

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/blog-314450/overview
Hosting URL: https://blog-314450.firebaseapp.com

And that’s it! You know have a static website with Hugo on Firebase Hosting, delivered automatically every time you commit in your repository.

Possible improvements:

spellcheck stage
scheduled runs to pick up planned articles with a future publish date
…

Hello, world!

adrian.todorov@atodorov.ninja (Adrian Todorov) — Sat, 21 Apr 2018 02:50:28 +0200

adrian.todorov@atodorov.ninja (Adrian Todorov) — Mon, 01 Jan 0001 00:00:00 +0000

Hi, my name is Adrian Todorov and this is my personal blog.

Currently I’m a Staff Solutions Architect at HashiCorp, giving advice on Nomad, Vault, Consul, and everything they interact with.

I started my career in tech as a web developer, and afterwards I moved to Site Reliability Engineering, focusing more on various types of infrastructure, their automation and monitoring. I spent a few years running vSphere, Kubernetes and Nomad in production, as well as covering the role of a virtualisation and storage admin, and an AWS architect; developing internal tooling, and automating all the things I had to do more than once with extensive use of Terraform (including homemade Terraform providers), Ansible, Python and Golang.

Proud alumni of the first class of the original Ecole 42, which taught me a lot about programming, debugging, and the importance of community.

At home I like exploring technology I can’t use at work, so I have some home automation (with Home Assistant), and my personal infrastructure is managed with Terraform, Ansible and GitHub on top of a local lab running Nomad, combined with CloudFlare, GCP and Scaleway (managed Kubernetes). Everything is monitored with Grafana Cloud (Prometheus, Loki, Grafana).

Tech I’m interested in and have running somewhere:

The HashiStack, obviously (Nomad, Consul, Vault, Terraform)
Orchestrators in general, mostly Nomad and Kubernetes
Observability (Prometheus, Grafana, Loki, Jaeger, InfluxDB once upon a time, OpenTelemetry, DataDog, NewRelic, etc.)
Automating all the things
Python, Golang, Bash
Home automation with Home Assistant, with tons of Zigbee devices, some ESP32s, and a few Raspberry Pis scattered around
Networking, specifically with VyOS and Ubiquiti gear, with some cheap multi-gig Zyxel switches

adrian.todorov@atodorov.ninja (Adrian Todorov) — Mon, 01 Jan 0001 00:00:00 +0000

Tracee

Debugging Scratch containers