Jon Williams

Cilanto Corn Black Bean Salad

2024-04-09T00:00:00+00:00

I know the photo doesn’t have the tortilla strips; deal with it.

I promise I haven’t been hacked and turned into a recipe SEO blog: here’s a salad recipe that I designed in 2023. I’m not much of a chef, but this is a big hit at the beach or summer parties. I usually just wing the proportions and it turns out fine – it’s not rocket science. Don’t stress!

Ingredients

One or two big bundles of cilantro. This is your base green; you want it to approximate the volume of beans & corn (together). Don’t be fooled by parsley!
Some (4-6) corn tortillas. I like white corn tortillas, but yellow can work too I suppose.
4 limes. You may want more if you’re planning on keeping the salad in the fridge for a couple days.
2 avocados or more if you can afford it. I’ve left these out before; it was fine I guess.
A wheel of cotija cheese. You could get it ground, I suppose. Usuing a whole wheel might be overkill, I end up using half to two thirds depending on the volume of cilantro, beans and corn.
Dry black beans. Canned beans work but it’s not the same without the cumin.
Greek yogurt. I use Fage, the firmer texture is really superior.
A big ass red onion.
4 to 8 jalapeños. Depends on how spicy you like it.
Olive oil or some other kind of oil for frying the tortillas.
A cup of sugar.
2 or 3 ears of corn. I’ve used frozen corn and it threw the whole taste off; beware.

Spices

Cumin.
Sea salt.
Chili powder; I like the dark chili powder.

Optional Ingredients

Radishes, 6 to 12. These look nice in photos.

Tortilla strips

Cut the tortillas into strips. They should be about half the length of your pinky & a little wider.
Put the tortillas in a pan with enough oil to cover them. I always feel like I’m wasting oil but you want to fry them until they’re super crispy.
Fry them on medium high heat for a couple minutes. Some brown & black areas is ideal. You may need to do two batches.
Place the tortilla strips in a container or a paper bag. Toss with 1 tablespoon chili powder and one tablespoon sea salt.

Beans

Search the internet for how to cook black beans. I usually just presoak them overnight and then throw them on for 90 minutes or so.
Throw a generous amount of cumin in. Maybe you don’t like cumin; do something else.

Corn

Shuck the corn.
Start boiling some water. When it boils, throw the sugar in.
Boil for about 5 minutes.
Use a knife to cut the kernels off the cob.

Assembly

Rough cut the cilantro, throw into bowl.
Add the beans and corn.
Cut up the jalapeños and add them. Don’t touch your eyes.
Add the juice of 4 limes. This will help the salad keep in the fridge for a couple days.
Cut up the red onion and add it.
Toss a bit.
Add the cotija cheese.
(Optional) cut up the radishes and throw them in.
Toss more.

Serving

Cut the avocados length wise into thin slivers.
Cut remaining limes into wedges.
Let your guests put tortilla strips, avocado, lime juice, yogurt and optional drizzle of olive oil on themselves.

Quick tools.go invocation

2023-07-03T00:00:00+00:00

Many places (including the wiki) recommend managing your CLI tool dependencies with a tools.go file. I’ve found this helpful as well. My contribution to this is a bash/zsh function (gt) that allows a shorthand invocation of a vendored tool.

function gt {
  arg=$1;
  shift 2> /dev/null
  [[ $? == 1 ]] && echo "Usage: gt tool" && return 1
  root=$(dirname "$(go env GOMOD)")
  cd "$root" || return 1
  cmd=$(go list -f '{{ join .Imports "\n" }}' -tags tools tools.go | grep -E "\/$arg\$" )
  if [[ $cmd == "" ]]; then
    echo cmd \""$arg"\" not in tools.go
  else
    go run "$cmd" "$@"
  fi;
  cd - 1> /dev/null || return 1
}

So now instead of invoking go run github.com/bufbuild/buf/cmd/buf lint, you can run gt buf lint. This will only work within a Go project where a tools.go file exists.

If you’re using zsh, here’s tab completion:

function _gt {
  if ((CURRENT == 2)); then
    root=$(dirname "$(go env GOMOD)")
    cd "$root" || return 1
    compadd $(go list -f '{{ join .Imports "\n" }}' -tags tools tools.go | sed -e "s#.*/##" )
    cd - 1> /dev/null || return 1
  elif ((CURRENT > 2)); then
    shift words
    ((CURRENT--))
    _normal -p mycmd
  fi
}
compdef _gt gt

If you haven’t seen tools.go before, the TL;DR is:

Invoke cli utilities – specifically Go commands (package main) in other modules – via go run. For example go run github.com/bufbuild/buf/cmd/buf lint.
Reference the cli utilities in a stub tools.go file in the root of your projects.

//go:build tools
// +build tools

package foobar

import (
	_ "github.com/bufbuild/buf/cmd/buf"
	_ "github.com/golangci/golangci-lint/cmd/golangci-lint"
	_ "github.com/mfridman/tparse"
	_ "github.com/twitchtv/twirp/protoc-gen-twirp"
	_ "golang.org/x/vuln/cmd/govulncheck"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

Your command-line utilities will be versioned like everything else in your project & will benefit from the ecosystem around that (Dependabot, govulncheck, etc.).

Launch OpenBSD vmd Guests on Demand from SSH

2023-03-20T00:00:00+00:00

I was annoyed that Electron apps Other people are annoyed that Signal doesn’t exist and wrote helpful writeups. don’t run on my OpenBSD laptop so I decided to run them inside a virtual machine. Unfortunately, this laptop is underpowered by 2023 standards & persistent Linux virtual machines would be competing with all the other memory hogs. gopls

I’d been kicking around the idea of using an ssh ProxyCommand to launch transient EC2 instances connected to long-lived EBS volumes so I figured – why not implement this for vmd hosts? I could forward Linux X11 apps to my desktop & be able to use Signal. But not Visual Studio Code; there is no way that is usable with 4GB of memory. I tried!

How does this work?

OpenSSH has the concept of session multiplexing] over a single connection. By wrapping the master connection with a ProxyCommand, I tie the master connection to the lifetime of the virtual machine. When it starts, the VM starts; when it exits the VM exits.

vmctl-ssh-wrapper.sh, when invoked by OpenSSH, parses the output of vmctl to determine if the vm is already running. If it isn’t, we try to bring it up, and schedule it for eventual shutdown – we will not shutdown vms that were manually started.

We determine the guest’s IP address by parsing the output of ifconfig tap and looking for interfaces with a “description” field matching the name of our requested virtual machine. By convention, the guest’s DHCP-assigned ip is immediately above the address assigned to the host’s tap interface.

Once we have the IP, we poll until the guest’s ssh server comes up. When it does, the socket is connected to our ssh client. When the script exits, we (may) invoke vmctl once again to schedule a shutdown.

VM Setup

vmd is the OpenBSD virtualization daemon. There’s plenty of tutorials on how to install Linux over the fake serial port, including the one I linked earlier.

A fresh wrinkle is that newer Linux kernels will not boot. “MMIO is unfinished in vmd(8).” So stick to an OS release known to be working, and don’t blindly jump to the next major release.

/etc/vm.conf

vm "ubuntu" {
    memory 2G 
#    boot device cdrom
    cdrom "/home/jon/vm/mini.iso"
    disk "/home/jon/vm/ubuntu.img"
    interfaces 1
    local interface tap
    owner jon
    disable

}

SSH Client Configuration

I added a block to my ssh configuration so that requests to ubuntu.vmctl.host are be serviced by the virtual machine defined as ubuntu. The ControlPersist block allows a 10 minutes idle period (no active X or ssh clients) before shutting down.

~/.ssh/config:

Host *.vmctl.host
  ProxyCommand ~/.skel/bin/vmctl-ssh-wrapper.sh %h %p
  ControlMaster auto
  ControlPersist 10m
  ForwardX11 yes

Just Let Me Install It, Already!

Source here. This is my first shell script in a while, so gentle feedback is welcome.

This works pretty well for Signal! Who knows, maybe Slack or VSCode might even be possible on a nicer laptop.

Isolating problematic Cgo code

2022-03-09T00:00:00+00:00

Introduction

KCTV_bot watches an HLS video stream and posts screengrabs to Twitter. Because the video source (North Korean state television) is not regularly available, some image processing must be performed to recognize when the channel is live.

Although the code is written in Go, the native options for decoding segments of video to get at individual frames are underwhelming. Fortunately, there exist Cgo wrapperscharlestamz/goav for the popular audio/video library ffmpeg.

The process of decoding an MPEG video segment to iterate over each individual video frame is a bit involved. To get an idea of the complexity, begin reading HandleSegment at the call to AvformatAllocContext(). Although the program was stable over short time intervals, it frequently crashed while running unsupervised. Furthermore, I would often return to my computer to see that the memory usage had ballooned to many gigabytes. Typical memory usage is around a gigabyte, with roughly half of that being accounted for by memory allocated inside Go — as reported by ReadMemStats(). After a few attempts at tracing memory leaks in Cgo proved mostly fruitless, I decided to try separating the program into two processes.

What is HLS (HTTP Live Streaming)?

Briefly, HLS is a popular format for streaming content. A piece of content, be it live or on-demand, is represented as a series of short (~10 second) media files contained in a playlist. A live playlist will be repeatedly fetched by a player to discover new media segments.

Architecture

Parent process, directly invoked from the command line to handle the majority of the tasks:
- Downloading the playlist to check for new segments. Every Target Duration, download the playlist and look for new segments.
- Fetching segments
- Image pattern recognition Is this an image of color bars or a test pattern? Is it a black screen? Is the image moving?
- Maintenance of a state machine Has the image been mostly moving for the last 30 seconds? If so, begin storing images to post.
- Posting tweets.
Child process, invoked by the parent process.
- An rpc service that, when provided with a raw segment (a bunch of MPEG bytes), returns a slice of frames ([]image.Image) to the caller.
- The child process is completely stateless — there is no dependency on previous rpc calls; a freshly restarted instance of the child process is ready to serve requests.

Implementation

Some of the code below has been edited for clarity.

Starting the child process

Spawning the child process is handled by spawnChild within internal/worker/parent.go.

First, the parent process listens on a Unix domain socket and retrieve an os.File struct corresponding to this socket. This struct contains the file descriptor of the socket. It may be surprising to see an empty UnixAddr passed into ListenUnix instead of a path to a file. This is a Linuxism that allows us to use a Unix socket on a read-only file system.

ul, err := net.ListenUnix("unix",
	&net.UnixAddr{})
if err != nil {
    return err
}
f, err := ul.File()
if err != nil {
    return err
}

A special flag is appended to a slice of arguments to the child process. The go1.18+ fuzzing system is very similar to our approach. We prepare the child process for execution and add our socket to ExtraFiles slice on the exec.Cmd struct:

args := append([]string{}, os.Args[1:]...)
args = append(args, "-worker")
cmd := exec.CommandContext(ctx, os.Args[0], args...)
cmd.ExtraFiles = append(cmd.ExtraFiles, f)

Passing the listening socket’s FD via ExtraFiles allows the child process to Accept() connections. While it is possible for the child process to call ListenUnix directly & avoid passing ExtraFiles, the parent’s DialUnix call may occur before the child process has begun listening.

The parent process next dials two connections and creates an net/rpc client. One handles rpc request/responses, and the second passes newly opened file descriptors to the child process. See Why two client connections? below.

connRPC, err := net.DialUnix("unix", nil,
	ul.Addr().(*net.UnixAddr))
if err != nil {
    return err
}
p.conn = conn

connFD, err := net.DialUnix("unix", nil,
	ul.Addr().(*net.UnixAddr))
if err != nil {
    return err
}

client := rpc.NewClient(connRPC)

Child startup

The child process next translates the file descriptor passed via ExtraFiles back into a Listener. The first 3 file descriptors are reversed for standard i/o , so ExtraFiles[0] corresponds to an FD of 3.

Simplified from internal/worker/child.go:

f := os.NewFile(3, "unix")
if f == nil {
    return fmt.Errorf("nil for fd %d", fd)
}
listener, err := net.FileListener(f)
if err != nil {
    return fmt.Errorf("net.FileListener: %w", err)
}

The Accept loop for the child process is contained in runWorker (internal/worker/child.go).

server := rpc.NewServer()

// pointer to a struct holding the goav code
server.Register(segApi)
conn, err := listener.Accept()
if err != nil {
    return errors.Wrap(err, "listener.Accept")
}
server.ServeConn(conn)

The child has now attached an RPC server to the Unix socket connection from the parent. segApi’s methods may now be invoked from the parent process. Specifically HandleSegment

Making calls to the child process

Again, the net/rpc documentation may be helpful. The parent process is able to make calls to child as such:

return client.Call("GoAV.HandleSegment", request, resp)

request and response are pointers to:

type Request struct {
	FD uintptr
}
type Response struct {
	RawImages []image.Image
}

Why are you passing file descriptors between processes?

Request contains an integer file descriptor for each segment. I explored passing the segment to goav in a number of ways.

A path to a temporary file.
- This had the disadvantage of disk i/o, and could leave files around when the program crashed.
- goav would not wait for the rest of the data to download once the end of a partially downloaded file was reached.
A path to a temporary FIFO.
- This allows forward progress on a partially downloaded file.
- Same problems as temporary files and a bit complex.
Passing http Body() - an instance of the io.ReadCloser interface.
- It wasn’t obvious to me how to call goav on data already in memory.
- This does not work across process boundaries.
The serialized byte slice of the entire segment (~10 mb for our stream).
- goav cannot begin decoding the segment until it has been fully read from the http response.
- It wasn’t obvious to me how to call goav on data already in memory.
- When moving to process separation, this added additional copies
  1. Copy from Body to a []byte
  2. Serialize each Request using encoding/gob.
  3. Unserialize the Request using encoding/gob.

Instead of any of these approaches, I ended up passing segments’ file descriptors to the child process in along with the RPC call. Previous code examples were simplified to hide this complexity. The implementation of sending and receiving file descriptors can be found in pkg/unixmsg/send_fd.go.

func SendFd(conn *net.UnixConn, fd uintptr) error {
	rights := syscall.UnixRights(int(fd))
	dummy := []byte("x")
	n, oobn, err := conn.WriteMsgUnix(dummy, rights, nil)
	if err != nil {
		return fmt.Errorf("err %v", err)
	}
	if n != len(dummy) {
		return fmt.Errorf("short write %v", conn)
	}
	if oobn != len(rights) {
		return fmt.Errorf("short oob write %v", conn)
	}
	return nil
}

func RecvFd(conn *net.UnixConn) (uintptr, error) {
	buf := make([]byte, 32)
	oob := make([]byte, 32)
	_, oobn, _, _, err := conn.ReadMsgUnix(buf, oob)
	if err != nil {
		return 0, err
	}
	scms, err := syscall.ParseSocketControlMessage(oob[:oobn])
	if err != nil {
		return 0, err
	}
	if len(scms) != 1 {
		return 0, fmt.Errorf("count not 1: %v", len(scms))
	}
	scm := scms[0]
	fds, err := syscall.ParseUnixRights(&scm)
	if err != nil {
		return 0, err
	}
	if len(fds) != 1 {
		return 0, fmt.Errorf("fd count not 1: %v", len(fds))
	}
	return uintptr(fds[0]), nil
}

Under the hood, this is calling the I_SENDFD ioctl on one of the Unix socket connections the parent process stood up earlier. Because file descriptors are scoped to a process, the receiving process must read a structure out of the connection to determine the integer value of the file descriptor it has been passed. The values passed from the parent will not be the same as the values received in the child, despite corresponding to the same resource.

The FD passed to the child process corresponds to a PipeReader returned from io.Pipe(). The HTTP body is streamed to the PipeWriter as it downloads; enabling the goav calls to begin decoding without waiting for the entire response to be read into memory.

Simplified version of handling a segment body from internal/stream/seg_consumer.go:

resp, err := s.httpGet(ctx, url)
if err != nil {
    return errors.Wrap(err, "httpGet")
}
defer resp.Body.Close()

r, w, err := os.Pipe()
if err != nil {
    return errors.Wrap(err, "os.Pipe")
}
defer r.Close()
defer w.Close()

go func() {
    if _, err := io.Copy(w, resp.Body); err != nil {
        log.WithError(err).Warn("io.Copy")
    }
    w.Close()
}()

request := &segment.Request{FD: r.Fd()}
return s.ProcessSegment(ctx, request)

In the child process, the ffmpeg calls are passed a filename that corresponds to an open file descriptor returned by RecvFd.

file := fmt.Sprintf("/proc/self/fd/%d", fd) // This is a Linuxism
pFormatContext := avformat.AvformatAllocContext()
avformat.AvformatOpenInput(&pFormatContext, file, nil, nil)

Why two client connections?

Socket control messages are considered out-of-band (OOB) data and are read into a separate slice by ReadMsgUnix. Attempting to read available OOB data will always discard at least 1 byte of in-band data. This dropped byte would cause problem for the the rpc server we attached to the parent to child connection, so we ended up using two connections. It may be possible to make an abstraction on top of UnixConn that allows multiplexing both messages on a single connection. But for a project this frivolous, this is good enough for now. See proposal: net: add ability to read OOB data without discarding a byte for more detail.

Perhaps removing net/rpc entirely and returning individual frames immediately after decoding would be a cleaner solution?

Conclusion

This code has some warts resulting from its origins as an ANSI HLS player. Nevertheless, I’m pleased that this is now able to run fairly stable without constant care & feeding. I plan on moving this into my local Kubernetes cluster & expect plenty of new problems from the limited resources.

Source: WIZARDISHUNGRY/hls-await
Twitter: KCTV_bot

Introducing @KCTV_bot

2022-03-08T00:00:00+00:00

I recently finished adding Twitter support to my Go project for monitoring intermitant HLS streams (hls-await). The result is @KCTV_bot - a Twitter bot that will live tweet screencaps from DPRK television whenever it goes on the air.

There’s more documentation to come, but for now, enjoy a few screencaps:

It's currently 3:02PM in Pyongyang & KCTV is on the air! pic.twitter.com/PKDERMD8RN
— 🇰🇵 KCTV bot 📺 (@KCTV_bot) March 7, 2022

pic.twitter.com/CvKIF0VfVq
— 🇰🇵 KCTV bot 📺 (@KCTV_bot) March 5, 2022

Speed Up GoMock with Conditional Generation

2019-12-22T00:00:00+00:00

Recently I’ve been frustrated by the slow reflection performance of Go’s mockgen when running go generate ./... on a large project. I’ve found it useful to use the Bourne shell built-in test command to conditionally invoke mockgen iif:

the destination is older than the source file
or the destination does not exist

go generate does not implement any kind of parallelism, so the slow performance of mockgen, while in source mode, has become a bit of a drag; thus –

package ordering

import (
	"context"
)

//go:generate sh -c "test client_mock_test.go -nt $GOFILE && exit 0; mockgen -package $GOPACKAGE -destination client_mock_test.go github.com/whatever/project/ordering OrderClient"

type OrderClient interface {
	Create(ctx context.Context, o *OrderRequest) (*OrderResponse, error)
	Status(ctx context.Context, orderRefID string) (*OrderResponse, error)
	Cancel(ctx context.Context, orderRefID string) (*OrderResponse, error)
}

On my fairly large project, this reduces many generate runs from the order of 45 seconds to 2 or 3 seconds.

(The above code sample probably works in source mode, but has been contrived for simplicity.)

Deploying Anycast DNS Using OpenBSD and BGP

2018-09-23T00:00:00+00:00

My home network is connected to NYCMesh, a community-owned open network. Recently, the failure of an SD card inside a Raspberry Pi at an adjacent large hub has left my area of the network without a caching recursive resolver to serve DNS for both the .mesh TLD and the wider internet. I stood up my own instance of the 10.10.10.10 anycast DNS resolver to service DNS in my neighborhood of the network.

Overview

Inside the mesh, DNS is serviced by the anycast IP address 10.10.10.10 by announcing a BGP route for this IP address. Nodes near to me will use my instance for DNS resolution because the routing topology will prefer my instance over a distant instance.

The major components of this build will be:

OpenBSD - Operating system for my network gateway and my anycast resolver instance.
vmd - Hypervisor for hosting a virtualized copy of OpenBSD, for running the resolver instance.
unbound - Caching recursive DNS resolver, for serving DNS requests to clients.
nsd - Authoritative DNS server, for supplying .mesh to unbound.
relayd - Application layer gateway that keeps a route from my BGP instance to the DNS instance up, if health checks are passing.
OpenBGP - Border Gateway Protocol daemon, included in OpenBSD.

The gateway machine has already been configured as a router to allow forwarding of packets, and functions as a router, LAN DNS forwarder, and web server.

Setup Virtual Machine

The virtual machine base system is installed mostly using the autoinstall facility, you may prefer a manual installation.

/etc/vm.conf:

vm "nycmesh-dns" {
    enable
    owner jon:wheel
    memory 512M
    # First disk from 'vmctl create "/home/vm/nycmesh-dns.img" -s 1G'
    disk "/home/vm/nycmesh-dns.img"
    #boot "/bsd.rd" # For install
    interface {
        switch "vmnet"
        locked lladdr 00:00:0A:46:91:C2
    }
}

/etc/dhcpd.conf:

authoritative;
option domain-name "bongo.zone";
use-host-decl-names on;
filename "auto_install";

# vmd service zone
subnet 10.70.145.192 netmask 255.255.255.224 {
  range 10.70.145.196 10.70.145.222;
  option routers 10.70.145.193;
  option domain-name-servers 10.70.145.1, 10.10.10.10, 10.70.131.129;

  host nycmesh-dns {
    fixed-address nycmesh-dns.bongo.zone, 10.10.10.10;
    hardware ethernet 00:00:0A:46:91:C2;
  }

}

/var/www/htdocs/default/nycmesh-dns-install.conf:

# autoinstall response file for unattended installation
# https://man.openbsd.org/autoinstall
#Password for root account = plaintext / encrypt(1) / "*************" to disable
Password for root account = *************
Change the default console to com0 = yes
Which speed should com0 use = 19200
Public ssh key for root account = ssh-rsa AAAA…XYZZY [email protected]
Start sshd(8) by default = yes
Do you expect to run the X Window System = no
Setup a user = no
Allow root ssh login = prohibit-password
What timezone are you in = America/New_York
Which disk is the root disk = sd0
URL to autopartitioning template for disklabel = https://kibble.bongo.zone/disklabel.min
Location of sets = http
HTTP proxy URL = none
HTTP Server = cdn.openbsd.org
Server directory = /pub/OpenBSD/6.3/amd64
Set name(s) = -comp* -game* -x* -man*

We may now access this virtual machine only via ssh.

Zonefile pull on the VM

NYCMesh generally uses kresd/knot as their DNS server and keeps the zone files and configuration in a git repo. Because OpenBSD has a fairly old version of knot, I decided to use the base system DNS servers to serve the zone files. (I should probably move this to a Linux VM running kresd/knot to be in line with the rest of the mesh.)

First I checked out a copy of the git repo using anonymous HTTP so I wouldn’t need github credentials on the VM.

pkg_add git python-2.7.14p1 bash
git clone https://github.com/nycmeshnet/nycmesh-dns.git

I setup a script to auto-pull the zonefile updates, based on the same script for Linux/Unbound.

/root/nycmesh-dns/deploy-nsd.sh:

#!/usr/local/bin/bash
export PATH=$PATH:/usr/local/bin

# OpenBSD + Unbound + NSD

cd /root/nycmesh-dns
git pull

NEWCOMMIT=`git rev-parse HEAD`
OLDCOMMIT=`cat commit`

if [ "$NEWCOMMIT" == "$OLDCOMMIT" ]
then
  exit 0
fi

python makereverse.py
cp -f *.zone /var/nsd/zones/master
rcctl restart nsd unbound
git rev-parse HEAD > commit

I later added a cron entry.

*/10    *       *       *       *       cd /root/nycmesh-dns && /root/nycmesh-dns/deploy-nsd.sh 2>&1 > /dev/null

Setup NSD and Unbound on the VM

First tweak the networking confiruration (/etc/hostname.vio0):

dhcp
inet alias 10.10.10.10/32

nsd will serve zone files from git.

/var/nsd/etc/nsd.conf:

server:
        hide-version: yes
        verbosity: 1
        database: "" # disable database

## bind to a specific address/port
ip-address: 127.0.0.1@53

remote-control:
        control-enable: yes

zone:
        name: "mesh"
        zonefile: "master/mesh.zone"
zone:
        name: "10.in-addr.arpa"
        zonefile: "master/10.in-addr.arpa.zone"
zone:
        name: "59.167.199.in-addr.arpa"
        zonefile: "master/59.167.199.in-addr.arpa.zone"

unbound will serve as a recursive resolver.

/var/unbound/etc/unbound.conf:

server:
        private-domain: "mesh"
        domain-insecure: "mesh"
        do-not-query-localhost: no
        #interface: 127.0.0.1
        interface: 10.10.10.10
        interface: 10.70.145.194
        interface: 127.0.0.1@5353       # listen on alternative port
        interface: ::1
        do-ip6: no

        prefetch: yes

        # override the default "any" address to send queries; if multiple
        # addresses are available, they are used randomly to counter spoofing
        outgoing-interface: 10.70.145.194

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: 10.0.0.0/8 allow
        access-control: 199.167.59.0/24 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow

        hide-identity: yes
        hide-version: yes

remote-control:
        control-enable: yes
        control-use-cert: no
        control-interface: /var/run/unbound.sock

forward-zone:
        name: "mesh."
        forward-addr: 127.0.0.1
forward-zone:
        name: "10.in-addr.arpa."
        forward-addr: 127.0.0.1
forward-zone:
        name: "59.167.199.in-addr.arpa."
        forward-addr: 127.0.0.1

Start both servers and try to see if you can resolve ns.mesh.

nycmesh-dns# rcctl restart nsd unbound
nsd(ok)
nsd(ok)
unbound(ok)
unbound(ok)
nycmesh-dns# host ns.mesh 10.10.10.10
Using domain server:
Name: 10.10.10.10
Address: 10.10.10.10#53
Aliases:

ns.mesh has address 10.10.10.11

relayd health check

relayd adds a route to 10.0.10.10/32 if the healthcheck passes. If the DNS server stops responding, the route is removed from the kernel and BGP retracts it from peers.

/usr/local/bin/mesh-dns-health-check.sh:

#!/bin/sh
! host -W 1 ns.mesh. $1

/etc/relayd.conf:

log updates

timeout 2000
interval 3
table <dns-servers> { nycmesh-dns.bongo.zone ip ttl 1 retry 0 }
router "anycast-dns" {
  route 10.10.10.10/32
  #forward to <dns-servers> check icmp
  forward to <dns-servers> check script "/usr/local/bin/mesh-dns-health-check.sh"
  rtlabel export
}

Start relayd and verify the route gets added.

kibble# rcctl restart relayd
kibble# traceroute 10.10.10.10
traceroute to 10.10.10.10 (10.10.10.10), 64 hops max, 40 byte packets
 1  10.10.10.10 (10.10.10.10)  1.162 ms  0.327 ms  0.485 ms

BGP announcement of `10.10.10.10`

Setting up BGP is a whole task in and of itself, but I have included a partial BGP configuration for reference.

/etc/bgpd.conf:

# global configuration
AS 65009
router-id 10.70.130.139
network 10.70.145.0/24
network 199.167.59.73/32
network inet static # This is the line that causes our dynamically inserted routes to get picked up
#network inet connected
# restricted socket for bgplg(8)
socket "/var/www/run/bgpd.rsock" restricted

# neighbors and peers
group "nycmesh" {
        neighbor 10.70.130.138 {
                remote-as 64996
                descr   "Node 1340"
                announce self
        }
}

# do not send or use routes from EBGP neighbors without
# further explicit configuration
#deny from ebgp
#deny to ebgp

allow from group nycmesh

dockertest timeouts

2018-08-10T00:00:00+00:00

Added Docker container timeouts to dockertest.

Rhombic Shift Register 0.6.0 Released

2018-04-18T00:00:00+00:00

Photo

The Pulsum Quadratum Rhombic Shift Register (RSR) is a four-voice, demuxing shift register for the VCVRack modular synthesis environment. The RSR consists of four looping “analog” shift registers whose inputs and outputs are switched via control voltage. By skillfully applying modulation, automatic variations upon source patterns may be generated. Using the Rhombic Shift Register with a sequenced melody allows you to design complex arabesque patterns and arpeggiations. On the other hand, you need not need not use a sequencer or keyboard to play the RSR. Good results have been obtained with quantized stepped and random voltages, feedback patches and other unconventional sources. Commonly termed “West coast synthesis”

The RSR is now available for $15 from VCVRack.

Announcing the Rhombic Shift Register

2018-03-23T00:00:00+00:00

I’ve been working on software modular synth design and finally have a fully featured module ready to share with early adopters.

Photo

The Pulsum Quadratum Rhombic Shift Register (RSR) is a four-voice, demuxing shift register for the VCVRack modular synthesis environment. The RSR consists of four looping “analog” shift registers whose inputs and outputs are switched via control voltage. By skillfully applying modulation, automatic variations upon source patterns may be generated. Using the Rhombic Shift Register with a sequenced melody allows you to design complex arabesque patterns and arpeggiations. On the other hand, you need not need not use a sequencer or keyboard to play the RSR. Good results have been obtained with quantized stepped and random voltages, feedback patches and other unconventional sources.

~~A time-limited beta version is available at the Pulsum Quadratum website.~~

NB: This has been superceded by a commercial version.

Lenovo Ideapad 120S

2018-03-18T00:00:00+00:00

I picked up a Lenovo Ideapad 120S for ~$150USD, a relative steal even for such a puny machine. I’m compiling my notes on hardware support under Linux and OpenBSD as I go.

General notes

The trackpad is pretty great for a cheap laptop, but not as nice as a MacBook’s. Windows 10 performance is terrible; taking multiple hours to install Windows updates. Battery life is good; right now I’m seeing ~7 hours reported remaining at 90% capacity under Debian.

Keyboard

The keyboard is of a chiclet design, similar to current generation MacBooks. Key caps are slightly rounded on the bottom, giving the keys a shield shape. The position of the Fn and Ctrl keys is my major complaint with the keyboard; they should be swapped.

Esc

Tab

CapsLk

Shift

Ctrl

❖

Alt

Storage

The device comes with Windows 10 Home 64-bit installed on it. The hard drive appears to be a 64 GB MMC reader. The drive is partitioned with:

an EFI partition, ~256 MB FAT
a Microsoft Reserved Partition, 16 MB
an OEM recovery partition, 10 GB NTFS
the Windows 10 Home partition, ~50 GB NTFS

Although it would be possible to resize the NTFS partition to 35 GB or so and install another operating system alongside Windows, I chose to nuke the entire partition map so that I wouldn’t be space constrained.

BIOS/EFI

The system has a small button on the right hand side you may depress with a paper clip to enter the BIOS/EFI configuration screen. Disable “Secure Boot” before attempting to install another operating system. You may also adjust the boot order here. Insert your USB drive before entering the configuration menu and allow it to boot first.

There are also options related to enabling virtualization features (I turned these on).

Debian Testing (buster)

The install image I picked was debian-testing-amd64-netinst.iso ; I burned it to a flash drive by running (on a Mac)

sudo dd if=debian-testing-amd64-netinst.iso of=/dev/diskXXX bs=1m

The trackpad did not work during install, so I used the text-based installer. It was totally functional upon reboot.

You will be prompted for missing non-free firmware for the Atheros wireless card. I followed the instructions and loaded the firmware via a second USB drive. The installer prompted me again for missing firmware, this time with a much shorter list (including ath10k). You may safely proceed. If your installer prompts you to manually select your network card (instead of proceeding to WiFi configuration), you messed up; restart the installation.

OpenBSD

So far I’ve tried 6.2 and a snapshot of pre-release 6.3. I will note where 6.3 differs. This section is subject to revision

Attempting to boot with the system in EFI mode results in a blank screen after the BOOTX64 prompt. Booting using BIOS gets you into the installer in 6.2. 6.3 successfully boots the installer using EFI.

The internal storage is an MMC device, which although supposedly supported & and visible in the dmesg, does not show up as a sd device. We are unable to proceed further unless we install to a USB drive. The Atheros wireless isn’t supported and shows up as See dmesg:

vendor "Atheros", unknown product 0x0042 (class network subclass miscellaneous, rev 0x31) at pci2 dev 0 function 0 not configured

I’ve noticed problems with mass storage devices not showing up, which seems to relate to the USB hub

uhub0: device problem, disabling port 5

After installing 6.3 snapshot to a second USB drive, selecting the USB drive as the boot target and booting, we see the regular 6.3 multiprocessor kernel spin up as expected. After a bit, we get a blank screen like we did booting the 6.2 install with BIOS and are unable to proceed further.

Forcing Clang to statically link against an installed library

2018-02-02T00:00:00+00:00

When building redistributable binary plugins, we cannot rely on the end user having installed library dependencies. In my case, I need my code to be statically linked against libusb and librtlsdr.

In the past, the venerable GCC allowed us to specify static linking by specifying a switch such as -l:rtlsdr and dynamic linking with -lrtlsdr. The modern LLVM Clang compiler is now the C/C++ compiler of choice on many platforms including in Apple’s Xcode. Its linker lacks an option to force static linking when resolving a library passed in. When both a static and dynamic version of a library exist, we must explicitly pass the path to the static library if we wish to link statically with Clang.

Static linking with full paths

c++ -o plugin.dylib object.cpp.o … /usr/local/Cellar/libusb/1.0.21/lib/libusb-1.0.a /usr/local/Cellar/librtlsdr/0.5.3/lib/librtlsdr.a

Dynamic linking (no full paths needed)

c++ -o plugin.dylib object.cpp.o … -lusb-1.0 -lrtlsdr -lusb-1.0

Getting your build system (e.g. make) to determine the exact path to a static version of a library can be trying. Fortunately many modern packages include pkg-config which will let you do things such as the following in your Makefile

PKGCONFIG= pkg-config
PACKAGES= libusb-1.0 librtlsdr

# FLAGS will be passed to both the C and C++ compiler

FLAGS += $(shell $(PKGCONFIG) --cflags $(PACKAGES))

The compiler will get the flags -I/usr/local/Cellar/librtlsdr/0.5.3/include/ -I/usr/local/Cellar/libusb/1.0.21/include/libusb-1.0 to set the include path. pkg-config also can be used to set linker flags with --libs; however, in my experience the --static option does not correctly emit static linking options. My trick for getting make to emit the correct static linking options is:

LDFLAGS +=$(shell $(PKGCONFIG) --variable=libdir libusb-1.0)/libusb-1.0.a
LDFLAGS +=$(shell $(PKGCONFIG) --variable=libdir librtlsdr)/librtlsdr.a

I’ve had varying success with passing static libraries on the command line, particularly running into problems on Linux, but it works on Windows (with some conditional code for library naming) and Mac.

If a library lacks pkg-config, you could parse the output of ld -v to enumerate search paths for find.

Validating Static Linking

Windows	Linux	Mac
`objdump file.dll`	`ldd file.so`	`otool -L file.dylib`

RTL SDR FM module for VCVRack

2018-01-22T00:00:00+00:00

⊕ Download:
Binaries
Source

As previously mentioned, I’ve been developing modules for the VCVRack software modular environment. Today I’m releasing the compiled version of RTL_SDR, a voltage-controlled FM radio tuner for RTL-SDR dongles. You should be able to find one on Amazon for cheap.

Messing around with VCVRack

2017-12-15T00:00:00+00:00

Update: Beta version available. ~~The source code for this is missing right now because I'm working on commercial VCVRack development.~~

I’ve been excited by all the activity around VCVRack, an open source software modular synthesizer for desktop computers. I decided to try my hand at writing a simple shift register module for it. The guts of a module is a C++ function that gets called once per output sample. Implementation was dead-simple; perhaps I’ll expand on this module.

The panel SVG labels do not match function! From top to bottom, the components are:

Trigger in + trigger led
CV in
4x Shift out + CV led (would be nice to have this indicate scale degree with an RGB value)

~~Source Code~~

Ableton Drum Rack for MFB-522

2017-11-29T00:00:00+00:00

Core 808 to MFB-522.adg (6.0K) I've been spending time working on music stuff, and decided to share my Ableton Drum Rack for the MFB-522 drum machine.

The MFB-522 Drumcomputer is a discontinued drum machine from MFB that sounds very similar to the Roland TR-808. While you can sequence it internally, I prefer to use Ableton Live or an Arturia Beatstep Pro to control it. To program drum sequences when I only have my laptop, I like to use a sample-based Ableton drum rack such as Kit-Core 808. You should be able to drop this Ableton rack onto your Core Drum sequence to sequence your MFB-522.

The MFB-522’s MIDI implementation uses a layout similar to the General MIDI percussion mapping but deviating on number of notes. Furthermore, the MIDI implementation chart inside the MFB-522 manual (PDF) has several errors in which notes/note numbers correspond to a particular drum sound; keep this in mind when referring to it. Ableton’s core drum racks seem to follow General MIDI Percussion, with their main kick drum located at C1. I’ve omitted the 808 maracas (doesn’t have an equivalent in the MFB-522) and congas (triggered by the toms when a two-position switch is flipped). The short clap and short bass drum have been mapped an octave below the regular clap and bass drum.

Two-Factor Authentication for OpenBSD

2016-04-22T00:00:00+00:00

This post is adapted from my OpenBSD guide in the totp-util wiki.

I recently set up a semi-public OpenBSD box, and thought I could stand to lock down password logins, especially for the root user. A popular system for two-factor authentication is TOTP:

In a typical two-factor authentication application, user authentication proceeds as follows: a user enters username and password into a website or other server, generates a one-time password for the server using TOTP running locally on a smartphone or other device, and types that password into the server as well. The server then also runs TOTP to verify the entered one-time password. For this to work, the clocks of the user’s device and the server need to be roughly synchronized (the server will typically accept one-time passwords generated from timestamps that differ by ±1 time interval from the client’s timestamp). A single secret key, to be used for all subsequent authentication sessions, must have been shared between the server and the user’s device over a secure channel ahead of time. If some more steps are carried out, the user can also authenticate the server using TOTP.

I wrote totp-util to simplify the process of setting up Google Authenticator on UNIX systems.

Install utilities

npm install -g https://github.com/WIZARDISHUNGRY/totp-util
pkg_add login_oath

User setup

run totp-util to setup ~/.totp-key
Scan the code in Google Authenticator

Setup authentication and SSH

We’re assuming everyone on the server is using SSH key authentication. Change /etc/login.conf to force TOTP+password when not using public key authentication.

# Default allowed authentication styles
auth-defaults:auth=-totp-and-pwd,skey:

Edit /etc/ssh/sshd_config to force SSH logins by root to use both an ssh key and a totp/password.

Match User root
AuthenticationMethods publickey,password

Restart sshd and update the login capabilities database.

/etc/rc.d/sshd restart
cap_mkdb /etc/login.conf

Now regular users should be able to authenticate with just SSH (or a password plus TOTP token) but root will need password, ssh-key and a TOTP token.

Logging in

$ ssh root@machine   
Authenticated with partial success.
user@machine's password: 123456/password

dump1090 on OpenBSD

2016-03-31T00:00:00+00:00

Fixed dump1090 compilation on OpenBSD.

dump1090 on OpenBSD

2016-03-31T00:00:00+00:00

Fixed dump1090 compilation on OpenBSD.

Block Tweet Sponsors

2014-09-29T00:00:00+00:00

April, 2016: this is broken, likely by a Twitter API change.

@twitter Stop putting things in my feed from people I don't follow. Hurts the signal-to-noise ratio and decreases utility of the platform.

— Nullsleep (@Nullsleep) September 17, 2014

I find the promoted tweets on Twitter super annoying, so I threw together a PHP (sorry!) script to block tweet sponsors. If anyone has an idea on how to get the mobile timeline so I never see ads for mobile games with in-purchasing, hit me up.

Fix Joli ORM for Titanium SDK

2014-09-12T00:00:00+00:00

Fixed Joli.js for newer versions of Titanium SDK.

Jon Williams

Cilanto Corn Black Bean Salad

Ingredients

Spices

Optional Ingredients

Tortilla strips

Beans

Corn

Assembly

Serving

Quick tools.go invocation

Launch OpenBSD vmd Guests on Demand from SSH

How does this work?

VM Setup

SSH Client Configuration

Just Let Me Install It, Already!

Isolating problematic Cgo code

Introduction

What is HLS (HTTP Live Streaming)?

Architecture

Implementation

Starting the child process

Child startup

Making calls to the child process

Why are you passing file descriptors between processes?

Why two client connections?

Conclusion

Introducing @KCTV_bot

Speed Up GoMock with Conditional Generation

Deploying Anycast DNS Using OpenBSD and BGP

Overview

Setup Virtual Machine

Zonefile pull on the VM

Setup NSD and Unbound on the VM

relayd health check

BGP announcement of 10.10.10.10

Further reading

dockertest timeouts

Rhombic Shift Register 0.6.0 Released

Announcing the Rhombic Shift Register

Lenovo Ideapad 120S

General notes

Keyboard

Storage

BIOS/EFI

Debian Testing (buster)

OpenBSD

Forcing Clang to statically link against an installed library

Static linking with full paths

Dynamic linking (no full paths needed)

Validating Static Linking

RTL SDR FM module for VCVRack

Messing around with VCVRack

Ableton Drum Rack for MFB-522

Two-Factor Authentication for OpenBSD

Install utilities

User setup

Setup authentication and SSH

Logging in

dump1090 on OpenBSD

dump1090 on OpenBSD

Block Tweet Sponsors

Fix Joli ORM for Titanium SDK

BGP announcement of `10.10.10.10`