cat /dev/null > /proc/mind

Integrating netbox with Authentik

Sat, 25 Jun 2022 00:00:00 +0000

Netbox has support for SSO integration out of the box, however some extra work is required to make it work with authentik correctly.

Setting up Authentik

In authentik, create an OAuth2/OpenID Provider (under Resources/Providers) with these settings:
- Name: Netbox
- Signing Key: Select any available key
Take note of the client ID & secret for later usage
Create an application with these settings:
- Name: Netbox
- Slug: netbox-slug
- Provider: Netbox

Setting up Netbox

Building the image

This step is only required for docker. Netbox comes with the SSO python package (social-auth-core) pre-installed, however not all the optional depedencies are installed due to relying on libraries that may not be present¹.

Luckily the image is made to be easily extendable:

FROM netboxcommunity/netbox:v3.2.5

RUN /opt/netbox/venv/bin/python -m pip install --upgrade 'social-auth-core[openidconnect]'

Configuration

For the python configuration file, we’ll combine the netbox documentation for connecting to Okta² with the generic OpenID connection backend³ from social-core

REMOTE_AUTH_BACKEND = 'social_core.backends.open_id_connect.OpenIdConnectAuth'
SOCIAL_AUTH_OIDC_OIDC_ENDPOINT = 'https://authentik.company/application/o/netbox-slug/'
SOCIAL_AUTH_OIDC_KEY = '<ID from step 2>'
SOCIAL_AUTH_OIDC_SECRET = '<secret from step 2>'
SOCIAL_AUTH_PROTECTED_USER_FIELDS = ['groups']

If groups is not set to be protected, you’ll receive a an error from Django about not being able to set a many-to-many field.

Caveats

Currently this setup does not handle groups or superuser status. If that functionality is required, an authentik LDAP outpost can be used instead.

'Ghost' changes in kubernetes' server-side apply

Sun, 20 Feb 2022 00:00:00 +0000

Starting in version 1.22, server-side apply has become officially “stable” in kubernetes, and in most is truly an upgrade from client-side applies. It respects fields that other server-side tools mark as controlled, and allows for applying larger and more complex objects.

However, there are a couple “gotchas” to be aware of. Most are either big loud errors, or called out explicitly in the documentation, but one thing I can across while working with server-side apply diffs (N.B.: client-side diffs are also sent to the server in new versions of kubernetes, it’s a little confusing). Every time I went to apply a change, certain objects, particularly deployments, would have a timestamp in metadata.managedFields which would always change even if nothing else did:

diff -u -N /tmp/LIVE-902559977/rbac.authorization.k8s.io.v1.ClusterRole..awx-operator /tmp/MERGED-2370271049/rbac.authorization.k8s.io.v1.ClusterRole..awx-operator
--- /tmp/LIVE-902559977/rbac.authorization.k8s.io.v1.ClusterRole..awx-operator  2022-02-20 17:25:02.204297070 -0500
+++ /tmp/MERGED-2370271049/rbac.authorization.k8s.io.v1.ClusterRole..awx-operator       2022-02-20 17:25:02.204297070 -0500
@@ -27,7 +27,7 @@
       f:rules: {}
     manager: tanka
     operation: Apply
-    time: "2022-02-20T21:55:47Z"
+    time: "2022-02-20T22:25:02Z"
   - apiVersion: rbac.authorization.k8s.io/v1
     fieldsType: FieldsV1
     fieldsV1:

The tool I’m using is tanka, but the same thing shows up when manually diffing. Even switching back to client-side diffing didn’t reveal anything unexpected. After spending way too long staring at the generated YML against the live YAML, I found that most of these instances were due to kubernetes converting values, with some example as follows:

'1024Mi' -> '1Gi', in requests/limits.memory
'2000m' -> '2', in requests/limits.cpu

There were also some fields that also were removed from the live objects when set to null (e.g. spec.selector in services) which also caused the same timestamp change. In all these cases, the server-side apply was smart enough to determine that nothing actually changed, but since the fields were technically different between the new and live object, the timestamp was still updated.

Tracking Geo IP information with Vector, Loki and Grafana

Mon, 29 Nov 2021 00:00:00 +0000

Normally my logging stack of choice is ELK, but recently I’ve been digging more into Loki, designed to be the logging equivalent of Prometheus. It’s been very interesting dealing with a low-cardinality log system after getting so used to ELK, but one feature I’ve definitely been missing is the ability to easily add GeoIP from arbitrary logs. There’s an open issue for adding it, but in the mean time one comment suggested using vector.

In my case, the log lines I was interested looked like this:

1637792122 I conn_pool_manager->255.152.255.207: Handshake dropped: connection backoff.
1637792123 I conn_pool_manager->71.255.178.255: Handshake dropped: certificate rejected.
1637792126 I conn_pool_manager->94.255.255.178: Adding incoming connection: fd:2074.

The first step is to get them into loki with the information we need. Vector’s remap language is extremely powerful and allows us to create this configuration:

# Watch the log files
[sources.in]
type = "file"
include = [ "/var/log/connections/*.log" ]
ignore_older_secs = 600
read_from = "beginning"


# Parse log message, mainly to extract the original timestamp and IP address
[transforms.remap_conn_pool]
inputs = [ "in"]
type = "remap"
source = '''
  . |= parse_regex(.message, r'^(?P<timestamp>\d+)\W(?P<level>\w)\Wconn_pool_manager->(?P<ip_address>[\d\.]+):(?P<trailer>.*)') ??
       {"err": "could not parse"}
  .timestamp = parse_timestamp(.timestamp, "%s") ?? now()
'''

# Use a locally downloaded MaxMind DB to generate GeoIP info
[transforms.geo_ip]
type = "geoip"
inputs = [ "remap_conn_pool" ]
database = "/etc/vector/GeoLite2-City.mmdb"
source = "ip_address"
target = "geoip"

# For debugging purposes
[sinks.out]
inputs = ["geo_ip"]
type = "console"
encoding.codec = "json"

# Send output to loki
[sinks.loki]
inputs = ["geo_ip"]
type = "loki"
endpoint = "https://loki.smuth.me/"
encoding.codec = "json"
labels = {"app"= "vector"}

Now that the information is in Loki, we can use the Geomap panel to display the locations inside grafana. In this particular case, this query was all that was needed:

count_over_time(
  {app="vector"}
  | json geoip_longitude="geoip.longitude",geoip_latitude="geoip.latitude"
  | geoip_latitude != ""
  [$__range])

This gets all JSON lines from logs created by the vector config with GeoIP information, filters for non-empty values, and counts each entry for the range provided. However, in order to make it more palatable for grafana, we need to apply some transforms to the data in grafana, namely labels-to-fields, merge, and then converting the lat and long from strings to numbers. Here is an example panel JSON, to get started with.

What are those zero byte files on my glusterfs bricks?

Sun, 02 Aug 2020 00:00:00 +0000

While troubleshooting some issues on a distributed GlusterFS cluster, I came across the presence of a bunch of odd zero-byte files with empty permissions on the bricks.

132504 1 ---------T 2 1000 1000 0 Nov 28 02:09 /data/gluster/tank/brick-xxxxxxxx/brick/example/file.txt

I had known they were there previously, but since the bricks were composed of ZFS zpools, there was very little danger of running out of inodes or anything, so I had let them be. This time however, they seemed to be related to the issue at hand, and after some digging, I found they were being used to maintain hard link information.

The mailing list has strict instructions to not meddle with them, but in my particular case they ended up being the root of the problem. At some point in the past, it seems the cluster was in a partially available state, and some files had gotten into a state where they existing on multiple bricks (which never happens under normal cirumstances). During a later cleanup effort, they had been replaced with hardlinks to identical files. This caused the very unusual symptom of gluster reporting client-side the existence of multiple files with the same exacts paths, one of which was the actual file, the others being the zero-byte files at hand.

Interestingly, most programs dealt with this reasonably well. Since the files still had different inodes, anything that scanned the effected parent directories discarded the bad files as unreadable and continued on. It was only when programs attempted to open the files directly by the path that things went haywire. Generally they’d get a handle to the good path, but under certain circumanstances they’d get the bad one. It was possible to coax the client cache into recognizing the good path again, but ultimately I ended up deleting the bad files directly from the bricks in order to resolve the problem.

Finding the log for deleted files in git

Sat, 08 Feb 2020 00:00:00 +0000

Recently I came across a blog post explaining how to get the history of a deleted file: https://dzone.com/articles/git-getting-history-deleted, which pointed to yet another blog with the simple solution https://feeding.cloud.geek.nz/posts/querying-deleted-content-in-git/:

git log -- deleted_file.txt

However, neither explained exactly why this works, and I got interested. -- is normally the convention that means “pass everything after this into the program as a literal string. For example, if for some crazy reason you had a file named -h (which is perfectly legal), rm -h would just bring up the man page, while rm -- -h would work as expected (as would rm ./-h). However, this is just a convention and git might be free to ignore it.

As the man page actually points out, git does indeed treat these differently:

[--] <path>...
    Show only commits that are enough to explain how the files that match the specified paths came to be. See
    History Simplification below for details and other simplification modes.

    Paths may need to be prefixed with -- to separate them from options or the revision range, when confusion
    arises.

The short story, according to the History Simplification section, appears to be that the default behavior is able to accurately track changes across moves/renames, but they also needed a way to say “track this exact path only”.

Enabling NFS mounts in Proxmox 5.2 LXC containers

Thu, 11 Oct 2018 00:00:00 +0000

Edit: This post has been made obsolete in recent releases, which allow enabling these flags via both the web UI and the pct CLI tool.

Normally, Proxmox doesn’t allow mounting NFS mounts directly in containers due to security concerns. In the past it was possible to modify apparmor profiles directly in order to allow it, as seen here and here. However, as of this commit, that method is no longer an option, due to the apparmor profiles now being generated dynamically when the containers are started (you can view these profiles at /var/lib/lxc/${CID}/apparmor/lxc-${CID}_<-var-lib-lxc>). Instead, modifying apparmor directly is no longer necessary, as you can now add the undocumented feature setting to the pct.conf files for individual containers. For instance:

features: mount=nfs

allows NFS mounting for the specified container, although a restart will be necessary for the setting to take effect. You can verify this by running grep nfs /var/lib/lxc/${CID}/apparmor/lxc-${CID}_\<-var-lib-lxc\>, which return a line like mount fstype=nfs. Other filesystems can also be allowed by separating the different types with semicolons.

This change was released in version 2.0-28 of the pve-container package, so it’s easy tell if you are affected:

$ dpkg -s pve-container | grep '^Version:'
Version: 2.0-28

Since this change only applies on container start-up, it’s possible to upgrade the package first without impacting any running containers.

SSH public key shenanigans

Wed, 19 Oct 2016 00:00:00 +0000

A fun little fact I discovered about SSH: when you specify a private key to use, it checks ${key}.pub for hints about how to parse the private key, without warning. Under normal operations this is never a problem, but you need to replace a private key in-place, and don’t update the .pub file, authentication will fail:

$ ls -la
ssh.key ssh.key.pub
$ ssh user@host echo ping
user@host's password: ^C
$ mv ssh.key.pub ssh.key.pub.bak
$ ssh user@host echo ping
Last login: Tue Oct 19
ping

This can be partially seen in the output of the ssh client’s -vvv option. On the left is with the public key preset, on the right is without:

debug1: identity file ./ssh.key type 2                   | debug1: identity file ./ssh.key type -1
...
debug2: dh_gen_key: priv key bits set: 131/256           | debug2: dh_gen_key: priv key bits set: 125/256
debug2: bits set: 529/1024                               | debug2: bits set: 532/1024
...
debug2: bits set: 512/1024                               | debug2: bits set: 482/1024
...
debug2: key: ./ssh.key (0x7fdea492cb30)                  | debug2: key: ./ssh.key ((nil))
...
debug3: send_pubkey_test                                 | debug1: read PEM private key done: type DSA
                                                         > debug3: sign_and_send_pubkey
debug2: we sent a publickey packet, wait for reply         debug2: we sent a publickey packet, wait for reply
debug3: Wrote 528 bytes for a total of 1653              | debug3: Wrote 592 bytes for a total of 1717
debug1: Authentications that can continue: publickey,pas | debug1: Authentication succeeded (publickey).
debug2: we did not send a packet, disable method         | debug2: fd 5 setting O_NONBLOCK

Interestingly, I wasn’t able to find any official documentation that mentions this, and only figured it out after resorting to strace.

Enabling jumbo frames on Proxmox 4

Sun, 02 Oct 2016 15:58:01 +0000

While turning on jumbo frames on a basic interface is straightforward (ip link set eth0 mtu 9000), Proxmox’s use of bridges to connect VMs makes things much more interesting. To start with, all interfaces connected to the bridge must have their MTUs upgraded first, otherwise it will give you an unhelpful error. Note that interfaces connected to the bridge include those of running containers/VMs.¹

# ip a | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
4: veth102i0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
# ip link set dev vmbr0 mtu 9000
RTNETLINK answers: Invalid argument
# ip link set dev eth0 mtu 9000
# ip link set dev veth102i0 mtu 9000
# ip link set dev vmbr0 mtu 9000

Note that setting the MTU on vmbr0 is technically unnecessary, since bridges inherit the smallest MTU of the slaved devices².

However, this only enables it temporarily. The jumbo frames will be lost at the next reboot. To fix this, ideally we could simply define the MTU via the /etc/network/interfaces file. However, due to a series of bugs, I was not able to get this working, even with the post-up option.³⁴ My (hacky) solution was to write a simple script that goes through and raises the MTU for each interface assigned to a bridge, and run it every 5 minutes via cron.

#!/bin/bash
intf="$1"
mtu="$2"

if [[ -z "$intf" || -z "$mtu" ]]; then
  echo "Usage: $0 interface mtu"
  exit 3
fi
        
for lower in /sys/devices/virtual/net/"${intf}"/lower_*; do
  child_intf="$(basename "$lower" | sed 's/lower_//')"
  if [[ "$(cat "$lower"/mtu)" -lt "$mtu" ]]; then
    ip link set "$child_intf" mtu "$mtu"
  fi
done

Saving this to /usr/local/bin/br_mtu and running it as /usr/local/bin/br_mtu vmbr0 9000 takes care of the host, and we can test that out with ping:

$ ping -c 5 -M do -s 8000 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 8000(8028) bytes of data.
8008 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.42 ms
8008 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1.42 ms
8008 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=1.43 ms
8008 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=1.51 ms
8008 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=1.45 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4057ms
rtt min/avg/max/mdev = 1.425/1.449/1.512/0.058 ms

(Note that the network stack all the way to the target host will need to have jumbo frames enabling for this to work)

However, the containers/VMs will still need to be told to use jumbo frames on their internal interfaces. As long as it’s enabled on the host, you should be able to simply follow your OS’s instructions for doing so.

A word of warning: On one of my hosts, after a reboot while testing some of these changes, containers with “Start at boot” set to yes were unreachable, and I had to delete and re-add the interfaces for them to work correctly again. This is probably due to all the changes I was making, and not a direct result of changing the MTU.

Sources

Upgrading from PHP 5.5 to 5.6 on FreeBSD

Sun, 31 Jul 2016 00:00:00 +0000

Recently PHP 5.5 got EOL’d, but PHP 5.6 will be supported for another two years. On Debian, this is a just a matter of upgrading the php5 package, but FreeBSD splits it out into two packages: php55 and php56, not to mention that extensions are also split out this way. The fact that I’ve installed php via ports also complicates things.

Doing the deed

This assumes portmaster is installed.

Listing the installed php55 packages:

$ pkg info | grep php55
php55-5.5.38                   PHP Scripting Language
php55-ctype-5.5.38             The ctype shared extension for php
etc...

Get the original ports path they were installed from (pkg query is a fantastic command to have in your toolbelt), and convert any 5.5 refereces to 5.6:

$ pkg query "%o" $(pkg info | grep php55 | cut -f 1 -d ' ' | tr '\n' ' ') \
  | sed 's/55/56/g'
lang/php56
extproc/php56-ctype
etc...

For the sake of brevity, I’m going to place the list of packages into a temporary file

pkg query "%o" $(pkg info | grep php55 | cut -f 1 -d ' ' | tr '\n' ' ') \
  | sed 's/55/56/g' > /tmp/php56-packages.txt

I’m pretty sure that all the packages I care about have 5.6 equivalents, but just to be sure, let’s check that those directories exist in the ports directory:

while read d; do [[ -d /usr/local/"$d" ]] && echo "$d does not exist"; done < /tmp/php56-packages.txt

Copy over the ports' options directory. (skip this step if you think any packages may need to be compiled differently):

mkdir -p /var/db/ports/lang_php56/ /var/db/ports/lang_php56-extensions/
cp /var/db/ports/lang_php55/options /var/db/ports/lang_php56/
cp /var/db/ports/lang_php55-extensions/options /var/db/ports/lang_php56-extensions/

Now begins the actual changes! Build 5.6 over 5.5 with portmaster:

sudo portmaster -n -o /usr/ports/lang/php56 lang/php55

Do a quick dry run with portmaster (if there are any new options that can be set, portmaster will open up a prompt):

portmaster -n $(cat /tmp/php56-packages.txt | tr '\n' ' ')

Install the other packages sequentially, in the same manner as the main php56 package:

cat /tmp/php56-packages.txt | grep -v 'lang/php56$' | \
  while read p; do echo portmaster -D --no-confirm -o "/usr/ports/$p" "$(echo "$p" | sed 's/56/55/g')"; done

Restart any necessary daemons, and you’re done! This was a learning process for me, I’m sure this could be compressed into a nice script. Of course, if you need to upgrade many machines, or downtime is an issue, building packages from the ports system (possible with portmaster’s -g flag), and then installing them wholesale would cut down on the amount of time spent compiling everything.

Making Terraform work with PowerDNS 4

Sun, 01 May 2016 00:00:00 +0000

Edit: This post has been made obsolete by a pull request I opened in the terraform repository: https://github.com/hashicorp/terraform/pull/7819

I’ve really enjoyed using PowerDNS as my DNS server at home. Most people only think of BIND and dnsmasq when it comes to DNS, while ignoring this stable, scalable, secure database-backed offering that powers some really large deployments. But enough proselytizing! I’m in the middle of trying to migrate my infrastructure to be controlled via Terraform (mostly). I figure this will help me consolidate and track most of my VPSs and the like.

The problem

PowerDNS has a nice HTTP/JSON API that can be used, but it changed URL locations in version 4, from / to /api/v1/, which makes any Terraform changes return powerdns_record.dns: Failed to create PowerDNS Record: Error creating record set: exmaple.com:::A, reason: "Not Found", which isn’t exactly helpful (for the record, that’s the error message returned by the API whenever a bad URL is requested).

The solution

Unfortunately there’s no easy fix, short of breaking compatibility for one thing or another. However, what I decided to do was to use nginx as a reverse proxy for the API. My configuration looks like this:

server {
    listen 8000;
    server_name _;

    location /api/v1/ {
        proxy_pass http://localhost:8081/;
    }

    location / {
        proxy_pass http://localhost:8081/api/v1/;
    }
}

This lets us use both the new and old locations simultaneously. I’m sure this can be done in Apache as well, but wasn’t up for installing it just to test something so simple.

Picking a blogging platform.

Sun, 24 Apr 2016 00:00:00 +0000

I’ve been bouncing around between picking a blog platform, and realized I should really settle down and stick with one. This is mostly a page to track my decision-making, and isn’t meant to be an objective comparison of different platforms. The one thing I'’ll be limiting myself to is static site generators. They’re light, easy to write to, and I can keep my blog in source control.

The markup language

I’d really rather avoid having to learn a wholle new markup, so that effectively limits me to either Markdown or ReST. While I really like the expressiveness of ReST, I don’t think I’ll be needing the advanced feautures enough to merit having to deal with the syntax.

The tool’s language

Meh, I really don’t care. As long it takes less than 30 seconds to build the site, that’s good enough for me.

Templating

It would be nice if it had a minimal black on white theme available, but I’m not really averse to creating my own.

Hosting

I’ll be most likely hosting this on github pages. Integration with that would be a bonus, but I’m not going to specifically check for that.

Candidates

A list of platforms that might meet my needs:

Nikola

My previous platform. I thinks it’s a great product, but it just didn’t strike the right chord.

Jekyll

By far and away the most popular static site generator. However, the YY/MM/DD/ folder structure it enforces irks me much more than it should.

Hugo

The strongest contender so far. Simple and does what I need.

My decision

I decided to go with Hugo. It’s written in Go, the new hotness, doesn’t force a directory structure on me, and has a really nice selection of themes I can use (I’m using angels-ladder currently.

Using shush as a crontab wrapper

Sat, 11 Apr 2015 00:00:00 +0000

Cron is a great tool for linux servers, but it's a very limited in it's capabilities (since it follows the Unix philosophy), so when I started to run up against those limits, I began doing all sorts of bash trickery to accomplish what I needed to happen, but that swiftly started giving me even more problem. At work, I use the Jenkins CI tool as a cron replacement (great tool, allows for distributed runs, queuing tasks, emails on failure, etc), but it seemed rather heavy weight for a homelab. Thankfully, there are a lot of cron wrappers/replacements out there, but I settled on shush, a neat little wrapper script for cron.

There are a couple reasons I like it:

The ability to manually run scripts under shush before adding them to cron
Automatic crontab management
Simple C binary, very few dependencies
Text based config file
Locking and timeout handling

Basically how it works is that there's a designated directory that holds all the shush configuration files. These files can be named anything, although I usually have them be extensionless for simplicity. You can then run any of these files with shush -c <directory> <config_file_name>. If you use the default directory of $HOME/.shush/, you can just run shush <config_file_name>. However, in order to have the config file run regularly, you need to set the schedule setting, and run shush -c <directory> -u to update the crontab file.

The man page has some more documentation, but I'll at least break down the example config file to make it more understandable.

Set the command shush -c /etc/shush -u to run every day at 9 PM:

command=shush -c /etc/shush -u
schedule=0 9 * * *

Use a lockfile. If a lockfile already exists, send an email to root and root-logs, then abort the job:

lock=notify=root root-logs,abort

Send out an email notification if the script is still running after 5 minutes, but keep the script alive:

timeout=5m,notify=root root-logs

Print stderr output first, use the text format, and set the email subject:

stderr=first
format=text
Subject=Crontab Daily Update

Always send an email to root-logs:

[logs]
to=root-logs

If one of the various failure conditions applies, send an email to root:

[readers]
if=$exit != 0 || $outlines != 1 || $errsize > 0 || U
to=root
format=rich

Shush is a very powerful tool, and I'm very happy with the ways that I've been able to implement it in my homelab. However, it can be a bit confusing for a beginner, and the regexp matching features leave much to be desired. With the intention of fixing things up, I've create a repository at https://github.com/smuth4/shush, which I can hopefully use to clean up things, and get the project active again. Feel free to compile, try out and maybe even contribute to this neat little tool!

Flashing LSI SAS 9211-8i with EFI

Sun, 08 Feb 2015 00:00:00 +0000

I recently went on an upgrade crusade to my homelab, and as part of that, upgraded FreeNAS to 9.3. When I did, there was a non-urgent alert about a driver mismatch for my LSI HBA (FreeNAS expected 16, LSI had 12). Thus, I decided to upgrade the firmware.

Directions

This assumes that your server can boot directly into an EFI shell. It might require a Shell.efi for some motherboards, but I can't tell you much more than that, as it boots straight to EFI on mine. I'm going to be flashing mine to IT mode (so it passes the disks through directly to ZFS), but the steps to upgrading the firmware in IR mode is very similar.

To do this, you're going to need a USB drive (formatted with something like Rufus) and 3 files:

sas2flash.efi - The executable that will be doing the flashing
mptsas2.rom - The BIOS ROM
2118it.bin/2118ir.bin - The firmware binary

All three of these files can be found on LSI's website. The two downloads you need are Installer_PXX_for_UEFI and 9211-8i_Package_PXX_IR_IT_Firmware_BIOS_for_MSDOS_Windows, where XX is the version you want. If you need an older version like I did, you can find them through the download search page.

Now open up Installer_PXX_for_UEFI.zip, find sas2flash_efi_ebc_rel/sas2flash.efi, and put that on the USB drive. Similarly, open up 9211-8i_Package_PXX_IR_IT_Firmware_BIOS_for_MSDOS_Windows.zip, extract Firmware/HBA_9211_8i_IT/2118it.bin and sasbios_rel/mptsas2.rom, and put those on the USB drive as well.

Alright, now that we have our files, we can boot into the EFI shell.

Verify you're in the USB root directory with ls. If not, try map to list the devices, mount <device> and then <device>: to mount and enter the USB.

Note

For this next step, apparenlty it's possible to upgrade multiple cards at once. I prefer the slow but safe way, since there's a (slim) chance of bricking a card, and flashed each card one at a time, manually pulling them in and out.

Verify we can see the hardware, and that it's a version we want to update:

sas2flash.efi -listall

First we erase the old firmware:

sas2flash.efi -o -e 6

Then we install the new one:

sas2flash.efi -o -f 2118it.bin -b mptsas2.rom

And Bob's your uncle! Enjoy your new and exciting passthrough disk I/O.

Useful links:

http://digitalcardboard.com/blog/2014/07/09/flashing-it-firmware-to-the-lsi-sas-9211-8i-hba-2014-efi-recipe/

http://brycv.com/blog/2012/flashing-it-firmware-to-lsi-sas9211-8i/

http://linustechtips.com/main/topic/104425-flashing-an-lsi-9211-8i-raid-card-to-it-mode-for-zfssoftware-raid-tutorial/

Using Heritrix to archive sites to a directory structure

Tue, 12 Aug 2014 00:00:00 +0000

So I one day I found myself in the market for a good web archiver. Specifically, there were some interesting open directories I wanted to mirror. My ideal solution would be a web front end around wget, but a little bit of research and testing showed that such an architecture would be too simplistic for the level of detail I wanted. There were a couple spider frameworks I tried out, like scrapy, but I wasn't enthusiastic about the prospect of trying to roll my own solution, when I knew sites like the Internet Archive had the exact kind of thing I had in mind, and they use the Heritrix engine archive their material. The Heritrix Wikipedia page mentions that it can output in the same directory format as wget (perfect!), but there's no citation for that, and the Heretrix documentation is unorganized, to say the least.

Setting it up

Software

I used Heritrix 3.2.0, the most recent stable version, for this project.

Steps

This is not going to be a full tutorial on how to use Heritrix. Be sure to read the documentation, and to read the default job configuration file before starting a large job.

Install Heritrix as per the instructions and get it started. Navigate to the web interface and create a new job with the standard configuration.

Next we want to edit the disposition chain, which starts at line 335 in my default configuration. The first bean defined should be the warcWriter, which, obviously, writes out scraped content to WARC files. WARC files are perfect for preserving websites exactly how they were accessed, but are a little too clumsy to be convenient.

After the WARC bean, add the follow code:

<bean id="mirrorWriter" class="org.archive.modules.writer.MirrorWriterProcessor">
</bean>

Then, in the dispositionProcessors bean, remove the line <ref bean="warcWriter"/> from the the processors list, and add <ref bean="mirrorWriter"/>.

That's pretty much all that's needed to get started. There are more parameters that can be tweaked, which can be found in the 3.2.0 Javadoc, but the defaults are not anything surprising.

Check_mk and FreeNAS, Pt. 3

Thu, 10 Jul 2014 00:00:00 +0000

A continuation of Check_mk and FreeNAS, Pt. 2

Now that we have all our smart data nicely set up, let's see if we can't get some stats on I/O speed. I'm pretty sure FreeNAS is supposed to have a I/O section in its "Reports" section, but for whatever reason, it's not in my install, and I'd like to have the data in Nagios in any case.

Just like with the SMART data, we're going to write a small script that the check_mk agent can use. Unlike the SMART script, getting IO stats is incredibly easy.

Yep, that's all it is. We're only really interesting in the drives being used in ZFS, but you could open it up to all drives if you wanted to.

Next up is to let check_mk be able to recognize the agent's output. The check script that I use can be found here. It should be placed in the "checks/" directory of your check_mk install.

I also created a quick template for pnp4nagios, found here, which should be placed in the "templates/" directory.

After all this, we've finally got a solid set up for tracking disks in FreeNAS. For each disk, there will be three associated services: SMART data, temperature (taken from the SMART data), and iostat data.

Check_mk and FreeNAS, Pt. 2

Fri, 25 Apr 2014 00:00:00 +0000

A continuation of Check_mk and FreeNAS, Pt. 1.

So I've got my check_mk on set up on my NAS, and it's monitoring stuff beautifully. However, it's not monitoring something very near and dear to my heart for this server: S.M.A.R.T. data. FreeNAS comes with smartctl, and there's already S.M.A.R.T. data plugins for the linux agents, so I figured this wouldn't be a big deal. And I was right! All I had to do was add the following script to my plugins/ folder for check_mk to find, and the server picked it up automatically.

Note

The linux script has a lot fancier checking for edge cases. I figured FreeNAS would be homogeneous enough that it wouldn't be worth converting all those edge cases, so this is a very simple script, shared on a "works for me" basis.

Check_mk and FreeNAS

Sun, 23 Feb 2014 00:00:00 +0000

Note

Software involved:

FreeNAS 9.2.0
OMD 1.10 (check_mk 1.2.2p3)

FreeNAS is great, and the web interface makes it easy and simple to understand my NAS's overall structure. However, my favored method of monitoring in my homelab is OMD with Check_mk, while FreeNAS prefers a self-contained collectd solution. We're in luck however, in that FreeNAS is heavily based on FreeBSD, which check_mk happens to have a plugin for, so it shouldn't be too hard to set things up the way I like them. There are two possible ways to do this:

Enable inetd and point it to the check_mk_agent
Call check_mk_agent over ssh through as shown in the datasource programs documentation

I decided to go the second way, as I prefer to avoid making manual changes to FreeNAS if I can avoid it.

If you do decided to go the inetd route, this thread may come in useful.

Agent Setup

The first thing we need to do is set up a user with a home directory where we can store the check_mk_agent program. If you already have a non-root user set up for yourself (which is good practice), that will work perfectly fine (they may need root access to collect certain data points). If you want to be more secure, you can set up a check_mk-only user, and limit it to just the agent command, which I will explain below.

Once the user is set up with a writeable home directory, it's as simple as copying check_mk_agent.freebsd into the home directory. Run it once or twice to make sure it's collecting data correctly.

Check_mk setup

From here it's basically following the instructions on the datasource programs documentation link. Here's a quick overview of the steps involved:

Add datasource programs configuration entry to main.mk. It will looks something like this:

datasource_programs = [
  ( "ssh -l omd_user <IP> check_mk_agent", ['ssh'], ALL_HOSTS ),
]

Set up password-less key authentication ssh access for omd_user to FreeNAS
(Optional) Limit omd_user to onyl check_mk_agent command by placing command="check_mk_agent"
Add ssh tag to FreeNAS host, through WATO or configuration files
Enjoy the pretty graphs and trends!

Tweaks

So we've got everything set up, but not everything is perfect.

Network

The first thing that I noticed was missing was a network interface counter. check_mk_agent was outputting a 'netctr' section, which seemed to have all the necessary information, but it wasn't being recognized in check-mk inventory, as it's been superseeded by lnx_if. It's possible to re-enable netctr, but not on a per-host basis.

BASH Documentation

Mon, 13 Jan 2014 00:00:00 +0000

One of the things that always bothers me is lack of proper documentation. Now, I’m lazy just like everyone else, but if I’m going to document something, I prefer to do it properly and keep it up to date. I’ve inhierited a nice suite of bash scripts, which aren’t really complicated, but they all have the same copy & pasted header that’s dated from 2003. Not exactly helpful.

So while I have a wiki that explains how some of the processes work on a higher level, it would be nice to have clean documentation in my bash scripts. Ideally, it would be embeddable, exportable and human readable. Basically, I shouldn’t have to maintain two files, I should be able to paste it somewhere else if need be, and I should be able to maintain it without any external tools whatsoever, if I wanted to.

Here are a list of options I found while browsing around:

The old-fashioned embedded comments
bashdoc (awk + ReST structure via python’s docutils)
embedded Perl POD (via a heredoc hack)
ROBODoc

Of these choices, POD seems a bit bloated to be inside a script, and ROBODoc looks way overblown for my simple needs, so I’ve decided to go with bashdoc. I’m already working with ReST, via this blog, and it fits pretty much all the criteria. Plus, it has few dependencies (awk, bash and python’s docutils) and doesn’t require a package for itself, so I wouldn’t feel bad about setting this up on production servers (although I should really set it up as a git hook in the script repo or something). However, documentation for bashdoc is quite limited (irony at it’s finest). The best way to figure out what is going on is to read lib/basic.awk, and the docutils source code, which isn’t exactly everyone’s cup of tea. That said, it shouldn’t be too difficult to build a small template I can copy and paste everywhere, which will hoepfully be more useful than the current header.

Fun with Basic PHP Optimization

Thu, 09 Jan 2014 00:00:00 +0000

A while ago I came across a full-featured PHP application for controlling a daemon. It worked well with a small data set, but quickly became laggy with a dataset numbering in the thousands. Admittedly, it really wasn’t built for that kind of load, so I removed it and controlled the daemon manually, which wasn’t a big deal.

Then a while later, I came across a post by someone who managed to mitigate the problem by shifting a particularly expensive operation to an external python program. Obviously, this was not exactly the most elegant solution, so I decided to take a look at the problematic section of code.

It looked something like this:

<?php
for ($i = 0; $i<count($req->val); $i+=$cnt) {
  $output[$req->val[$i]] = array_slice($req->val, $i+1, $cnt-1);
}
?>

Looks pretty basic, right? It cycles through an array ($req) and splits it into a new dictionary ($output) based on a fixed length ($cnt). However, if we turn this into a generic big O structure, with the values borrowed from this serverfault post, the problem quickly becomes apparent.

<?php
for ($i = 0; $i<O(n); $i+=$cnt)
  $output[$req->val[$i]] = O(n)
?>

Taking into account the for loop, this would appear to mean that the operation is O(2n2), in contrast to the very similar array_chunk O(n) function. So how do we optimize this? The most important thing to do is make it so php can complete this in one loop over the array. Everything else will be a nice improvement, but when scaling, the big O is king.

Here’s the new code:

<?php
foreach($req->val as $index=>$value)
{
  if($index % $cnt == 0)
  {
    $current_index = $value;
    $output[$current_index] = array();
  }
  else
    $output[$current_index][] = $value;
}
?>

We’ve dropped the for/count() loop in favor of foreach, and eliminated slicing in favor of appending to newly created elements. In a real world test, this cut down the response time of the module from 12s to 4s on average. A pretty big improvement for a pretty small change!

Welcome to my blog

Thu, 09 Jan 2014 00:00:00 +0000

Hopefully I’ll be able to build this into a little repository of my tips, tricks and hacks as I navigate the strange and wonderful world of information technology. Maybe I’ll even improve my writing, or help someone out with an obscure problem. The possibilities are endless! (For specific definitions of endless.)