Geoff's Braindump

Setting Up Kubernetes On A Budget With Ceph Volumes

Thu, 24 May 2018 00:00:00 +0000

I have been really excited to use Kubernetes (aka k8s) in a semi-production way on my home server; however, there is only one of said server and not the 3+ that seem to be typical of a minimal kubernetes deployment. At the same time, I know there’s minikube and similar, but those give a strong sense of development-time, experimental usage. I want a small scale, but real kubernetes experience.

For me, an additional requirement to make it a real deployment is that it should support stateful services. One of the things I like about kubernetes is that persistent volumes are a first-class construct supported in various real-world ways. Since I’m working with a tight budget and a single server, I narrowed my search to open source, software-defined storage solutions. Ceph workeds out well for me since it was easy to setup, scaled down without compromises, and is full of featurea.

In this article I’ll show how to use a little libvirt+KVM+QEMU wrapper script to create three VMs, deploy kubernetes using kube-adm and overlay a ceph cluster using ceph-deploy. The setup might appear tedious, but the benefits and ease of kubernetes usage afterwards are well worth it

My environment

If you’re following along, it might help to know what I’m using to see how that aligns with what you’re using. My one and only home server is an Intel NUC with

i5 dual-core, hyperthreaded processor
16 GB RAM
240 GB SSD with about 100 GB dedicated to VM images
Ubuntu Xenial 16.04.4
Kernel 4.4.0-124

It has plenty of RAM for three VMs, but I’m going to be knowingly overcommitting the vCPU count of 6 (2 x 3 VMs) since there’s only 4 logical processors on my system. I’m not going to be running stressful workloads, so hopefully that works out.

Setup virtual machines

Baremetal or VMs?

My original desire was to avoid virtual machines (VMs) entirely and run both kubernetes and ceph directly “on metal”. I found that was technically possible with options such as “untainting” the k8s master node; however, striving for zero downtime really requires distinct nodes in order to achieve seamless cluster scheduling.

As such, one of the wheels I re-invented was to create a small tool to help me create VMs with minimal requirements. All that it requires is installation of the packages:

libvirt-bin
qemu-kvm

Networking setup for virtualization

Since I’ll eventually be port forwarding certain traffic from my ISP’s cable modem, I need the VMs to be directly routable on my LAN. For that I needed to configure the Linux kernel bridging by following this.

In my case, I set my /etc/network/interfaces with the following since I am using my LAN’s DHCP server to assign a fixed IP address. So the dhcp option eliminates the need specify address, gateway, and DNS.

auto br0
iface br0 inet dhcp
        bridge_ports eno1
        bridge_maxwait 0
        bridge_fd 0

To make sure iptable filtering and such of the host doesn’t interfere with the guest VMs doing their own, we ease packet forwarding by creating /etc/sysctl.d/60-bridge-virt.conf with

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

To apply those settings immediately, run:

sudo /lib/systemd/systemd-sysctl --prefix=/net/bridge

To ensure the settings are applied on reboot after networking is started, then create the file /etc/udev/rules.d/99-bridge.rules with the content:

ACTION=="add", SUBSYSTEM=="net", KERNEL!="lo", RUN+="/lib/systemd/systemd-sysctl --prefix=/net/bridge"

Create VMs using libvirt+KVM+QEMU

Download my helper script from my libvirt-tools repo and set it to be executable:

wget https://raw.githubusercontent.com/itzg/libvirt-tools/master/create-vm.sh
chmod +x create-vm.sh

I am going to create three VMs. My network has static IP address space starting at 192.168.0.150, so I’m configuring my VMs starting from that address.

The volume on my machine that contains /var/lib/libvirt/images on my system only has 112G so I am going to allocate an extra disk device of 20G for the three VMs. In total they will use 3 x (8G root + 20G extra) = 84G. The volumes are thin provisioned, but it’s still good to plan for maximum usage since I have limited host volume space (SSD == good for speed, but bad for capacity).

View full usage and default values of the helper script by running

./create-vms.sh --help

Make sure you have an SSH key setup since the helper script will tell cloud-init to install your current user’s ssh key for access as the user ubuntu. To confirm, list your keys using:

ssh-keygen -l

Create the first VM:

sudo ./create-vm.sh --ip-address 192.168.0.150 --extra-volume 20G nuc-vm1

You should now be able to ssh to the VM as the user ubuntu

ssh [email protected]

however, if that doesn’t seem to be working you can attach to the VM’s console using:

virsh console nuc-vm1

Use Control-] to detach from the console.

Repeat the same invocation for the other two VMs changing the IP address and name:

sudo ./create-vm.sh --ip-address 192.168.0.151 --extra-volume 20G nuc-vm2
sudo ./create-vm.sh --ip-address 192.168.0.152 --extra-volume 20G nuc-vm3

Simplify ssh access

Adding /etc/hosts entries for the VMs and their IPs will help ease the remainder of the setup tasks. In my case I added these:

192.168.0.150 nuc-vm1
192.168.0.151 nuc-vm2
192.168.0.152 nuc-vm3

To avoid having to specify the default ubuntu user for ssh’ing to each VM, add this, replacing the host names with yours, to ~/.ssh/config:

Host nuc-vm1 nuc-vm2 nuc-vm3
   User ubuntu

Confirm you can ssh into each, which also gives you a chance to confirm and accept the host fingerprint for later steps:

ssh nuc-vm1

While you’re in there you could also upgrade packages to get the VMs up to date before installing more stuff:

sudo apt update
sudo apt upgrade -y
# ...and sudo reboot if the kernel was included in the upgraded packages

Create kubernetes cluster

Pre-flight checks

ssh to the first VM and install kubeadm and its prerequisites.

The VMs were configured (via defaults) to generate unique MAC addresses for their bridged network device, but here’s an example of confirming:

$ ssh nuc-vm1 ip link | grep "link/ether"
    link/ether 52:54:00:91:76:e5 brd ff:ff:ff:ff:ff:ff
$ ssh nuc-vm2 ip link | grep "link/ether"
    link/ether 52:54:00:93:4c:83 brd ff:ff:ff:ff:ff:ff
$ ssh nuc-vm3 ip link | grep "link/ether"
    link/ether 52:54:00:b7:88:ad brd ff:ff:ff:ff:ff:ff

Likewise, the VMs were created with the default behavior of generating a UUID per domain/node. Here is confirmation of that:

$ ssh nuc-vm1 sudo cat /sys/class/dmi/id/product_uuid
781526BF-31E7-4339-8424-6B886A432968
$ ssh nuc-vm2 sudo cat /sys/class/dmi/id/product_uuid
AF93665F-6137-4125-9480-08B99B040BE8
$ ssh nuc-vm3 sudo cat /sys/class/dmi/id/product_uuid
C76F05E7-0289-4479-91CA-3D47A48096F4

Install Docker and Kubernetes packages

Install the recommended version of Docker by ssh’ing into the first node and starting an interactive sudo session with

sudo -i

Use the installation snippet provided in the Installing Docker section.

Still in the interactive sudo session, install kubeadm, kubelet, and kubectl.

Repeat the same for the other nodes installing Docker and steps after that.

Create the cluster

Before running kubeadm init skip to the pod network section to see what parameters should be passed. I’m going to use kube-router, so I’ll pass --pod-network-cidr=10.244.0.0/16. You can find more information about using kube-router with kubeadm here.

The full kubeadm command to run is:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Use the commands it provides at the end to enable kubectl access from the regular ubuntu user on the VM:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Be sure to note the kubeadm join command it provided since it will be used when joining the other two VMs to the cluster.

Now you can install the networking add-on, kube-router in this case:

kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

After about 30 seconds you should see the kube-router pods running using:

kubectl get pods --all-namespaces -w

Join the other two VMs to the kubernetes cluster

ssh to the next VM and become root with sudo -i. Then use the kubectl join command output from the init call earlier, such as

kubeadm join 192.168.0.150:6443 --token b2pp7j.... --discovery-token-ca-cert-hash sha256:...

If you ssh back to the first node, you should see the new node become ready after about 30 seconds:

$ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
nuc-vm1   Ready     master    22h       v1.10.3
nuc-vm2   Ready     <none>    23s       v1.10.3

Repeat the kubeadm join on the third VM.

Create a distinct user config

On the master node’s VM I ran the following and saved its output to a file on my desktop:

sudo kubeadm alpha phase kubeconfig user --client-name itzg

By default that user can’t do much of anything, but you can quickly fix that by creating a cluster role binding to the builtin cluster-admin role. In my case I ran:

kubectl create clusterrolebinding itzg-cluster-admin --clusterrole=cluster-admin --user=itzg

I used a naming convention of <user>-<role>, but you can use whatever naming convention you want for the first argument – it just needs to distinctly name the binding of user(s) to role(s).

Create ceph cluster

Now we’ll switch gears and ignore the fact that the three VMs are participating in a kubernetes cluster. We’ll overlay a ceph cluster onto them. My goal is to create a storage pool dedicated to kubernetes that can be used to provision rbd volumes.

In the same spirit as kubeadm ceph provides an extremely useful tool called ceph-deploy that will take care of setting up our VMs as a ceph cluster. It also comes with great instructions, that start here.

At the time of this writing, the latest stable release is luminous, so that’s what I’ll be using in the next few steps.

Most of the steps I’ll run with ceph-deploy will be invoked from the main/baremetal host, but really it can be done from any node that can ssh to the three VMs.

Setup ceph-deploy

First add the release key

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

Then, add the repository:

echo deb https://download.ceph.com/debian-luminous/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

Finally, update the repo index and install ceph-deploy

sudo apt update
sudo apt install ceph-deploy

The tool is going to place some important config files in the current directory, so it’s a good idea to create a new directory and work within there:

mkdir ceph-cluster
cd ceph-cluster

Before proceeding, make sure you did the ~/.ssh/config setup above to configure ubuntu as the default ssh user for accessing the three VMs. Since that user already has password-less sudo access, it’s ready to use for ceph deployment.

Initialize cluster config

Initialize the ceph cluster configuration to point at what will be the initial monitor node. Think of the ceph monitor kind of like the kubernetes master/api node.

ceph-deploy new nuc-vm1

Install packages on ceph nodes

Now, install the ceph packages on all three VMs. I had to pass the --release argument to avoid it defaulting to the prior “jewel” release.

ceph-deploy install --release luminous nuc-vm1 nuc-vm2 nuc-vm3

Create a ceph monitor

After a few minutes of installing packages, the initial monitor can be setup:

ceph-deploy mon create-initial

To enable automatic use of ceph on the three nodes, distribute the cluster config using

ceph-deploy admin nuc-vm1 nuc-vm2 nuc-vm3

Create a ceph manager

As of the luminous release, a manager node is required. I’m just going to run that also on the first VM:

ceph-deploy mgr create nuc-vm1

Setup each node as a ceph OSD

Way back in the beginning of all this, you might remember that I specified an extra volume size of 20GB for each VM. That extra volume is /dev/vdc on each VM and will be used for the OSD on each:

ceph-deploy osd create --data /dev/vdc nuc-vm1
ceph-deploy osd create --data /dev/vdc nuc-vm2
ceph-deploy osd create --data /dev/vdc nuc-vm3

You can confirm the ceph cluster is healthy and contains the three OSDs by running:

ssh nuc-vm1 sudo ceph -s

To enable running ceph commands from the host, copy the admin keyring and config file into the local /etc/ceph:

sudo cp ceph.client.admin.keyring ceph.conf /etc/ceph/

Joining ceph and kubernetes

Create a storage pool

Create the pool with 128 placement groups, as recommended here:

sudo ceph osd pool create kube 128

In order to avoid the libceph error “missing required protocol features” when kubelet mounts the rbd volume apply this adjustment:

sudo ceph osd crush tunables legacy

Create and store access keys

Export the ceph admin key and import it as a kubernetes secret:

sudo ceph auth get client.admin|grep "key = " |awk '{print  $3'} |xargs echo -n > /tmp/secret
kubectl create secret generic ceph-admin-secret \
   --type="kubernetes.io/rbd" \
   --from-file=/tmp/secret

Create a new ceph client for accessing the kube pool specifically and import that as a kubernetes secret:

sudo ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube'|grep "key = " |awk '{print  $3'} |xargs echo -n > /tmp/secret
kubectl create secret generic ceph-secret \
   --type="kubernetes.io/rbd" \
   --from-file=/tmp/secret

I might be doing something wrong, but I found I had to also save the client.kube keyring on each of the kubelet nodes:

ssh nuc-vm1 sudo ceph auth get client.kube -o /etc/ceph/ceph.client.kube.keyring
ssh nuc-vm2 sudo ceph auth get client.kube -o /etc/ceph/ceph.client.kube.keyring
ssh nuc-vm3 sudo ceph auth get client.kube -o /etc/ceph/ceph.client.kube.keyring

Setup RBD provisioner

The out-of-tree RBD provisioner is pre-configured to manage dynamic allocation of RBD persistent volume claims, so download and extract the necessary files:

wget https://github.com/kubernetes-incubator/external-storage/archive/master.zip
unzip master.zip "external-storage-master/ceph/rbd/*"

Go into the deploy directory and apply the provisioner:

cd external-storage-master/ceph/rbd/deploy
kubectl apply -f rbac

Define a storage class

Create a storage class definition file, such as storage-class.yaml containing:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: rbd
provisioner: ceph.com/rbd
parameters:
  monitors: 192.168.0.150:6789
  pool: kube
  adminId: admin
  adminSecretName: ceph-admin-secret
  userId: kube
  userSecretName: ceph-secret
  imageFormat: "2"

Apply the storage class:

kubectl apply -f storage-class.yaml

Try it out

Test the storage class by applying a persistent volume claim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

Verify that the claim was provisioned and is bound by using

kubectl describe pvc claim1

Let’s test otf the persistent volume backed by ceph rbd by running a little busybox container that just sleeps so that we can exec into it:

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod1
spec:
  containers:
  - name: ceph-busybox
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - name: data
      mountPath: /data
      readOnly: false
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: claim1

Now exec into the pod using

kubectl exec -it ceph-pod1 sh

You can cd /data and touch/edit files in that directory, delete the pod, and re-create a new one to confirm the content from the persistent volume claim sticks around.

Written with StackEdit.

Elegant array to set conversion with Java 8 streams

Sat, 04 Mar 2017 00:00:00 +0000

Before Java 8, it was always a little awkward to take a given array of objects and populate a Set (or other collection) out of them. The various collections have constructors that accept other collections, so you could use Arrays.toList(...). However…let’s use an elegant, Java 8 way instead:

public Set<String> convert(String... members) {
    return Stream.of(members)
            .collect(Collectors.toSet());
} 

Dealing with inconsistent JSON with Jackson deserializer tweaking

Fri, 03 Mar 2017 00:00:00 +0000

Dealing with inconsistent JSON with Jackson deserializer tweaking

Let’s say we have a model class we want to read from JSON:

@Data
public class NotWeird {
    String name;
    Details details;

    @Data
    public static class Details {
        int count;
    }
}

but we need to deal with varying structure of details like this

{
  "name": "weird1",
  "details": 5
}

and this

{
  "name": "weird2",
  "details": {
    "count": 6
  }
}

In our fabricated scenario, maybe there was an older version of our JSON structure where details used to be a single field. Or perhaps we’re provided a human-friendly shortcut where only setting count of a details doesn’t required the whole object.

To recreate this scenario and create a solution, let’s create a Spring Boot batch-like application that will autowire a Jackson ObjectMapper for us.

We’ll use these dependencies in our pom.xml:

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-configuration-processor</artifactId>
    <optional>true</optional>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-web</artifactId>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
	<groupId>org.projectlombok</groupId>
	<artifactId>lombok</artifactId>
	<optional>true</optional>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-test</artifactId>
	<scope>test</scope>
</dependency>

Even though we’re not creating a web app, we need the org.springframework.http.converter.json.Jackson2ObjectMapperBuilder provided by spring-web. We’ll disable the web application context by using SpringApplicationBuilder:

@SpringBootApplication
public class InconsistentJsonApplication {

	public static void main(String[] args) {
		new SpringApplicationBuilder(InconsistentJsonApplication.class)
				.web(false)
				.run(args);
	}
}

Before getting to an application runner, let’s establish a properties component:

@ConfigurationProperties("app") @Component @Data
public class AppProperties {
    boolean failOnEmptyFileList = true;
    boolean exitWhenFinished = true;
}

We’ll wire that and the ObjectMapper auto configured by Boot into an ApplicationRunner:

@Slf4j
@Service
public class Loader implements ApplicationRunner {

    private ObjectMapper objectMapper;
    private AppProperties properties;

    @Autowired
    public Loader(ObjectMapper objectMapper, AppProperties properties) {
        this.objectMapper = objectMapper;
        this.properties = properties;
    }

In our runner we’ll “load” the JSON content by reading them into POJOs, logging, and exiting:

@Override
public void run(ApplicationArguments args) throws Exception {
    if (properties.isFailOnEmptyFileList()) {
        Assert.notEmpty(args.getNonOptionArgs(), 
	        "Pass at least one JSON filename on the command line");
    }

    List<NotWeird> objects = args.getNonOptionArgs().stream()
            .map(this::load)
            .collect(Collectors.toList());

    log.info("Loaded: {}", objects);

    if (properties.isExitWhenFinished()) {
        System.exit(0);
    }
}

private NotWeird load(String filename) {
    try {
        final NotWeird obj = objectMapper.readValue(
	        new File(filename), NotWeird.class);

        return obj;
    } catch (IOException e) {
        log.warn("Unable to process file {}", filename, e);
        return null;
    }
}

If you run the code at this point, the readValue will fail on the first JSON snippet as follows (with newlines added for clarity):

com.fasterxml.jackson.databind.JsonMappingException: 
    Can not construct instance of me.itzg.json.model.NotWeird$Details: 
      no int/Int-argument constructor/factory method to deserialize from Number value (5)
      at [Source: weird1.json; line: 3, column: 14] 
      (through reference chain: me.itzg.json.model.NotWeird["details"])

There are probably several ways to solve this, but let’s use a solution where we can intercept the deserialization of details and handle it in a polymorphic/adapting way.

By declaring a com.fasterxml.jackson.databind.Module component, Boot will take care of autowiring that into any ObjectMappers :

@Component
public class CustomModule extends Module {
    @Override
    public String getModuleName() {
        return "custom";
    }

    @Override
    public Version version() {
        return new Version(1, 0, 0, null, null, null);
    }

The meat of our solution involves adding a com.fasterxml.jackson.databind.deser.BeanDeserializerModifier:

@Override
public void setupModule(SetupContext context) {
    context.addBeanDeserializerModifier(new BeanDeserializerModifier() {
        @Override
        public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config,
                                                      BeanDescription beanDesc,
                                                      JsonDeserializer<?> deserializer) {

            return deserializer;
        }
    });

}

Already with those code our application will run, but will still complain about the first JSON snippet since we didn’t actually modify the deserializer yet. Let’s do that by using replaceProperty of BeanDeserializer.

That method takes two SettableBeanProperty objects, the original and the one to put in its place. Where do we get those? It turns out that we can a) ask the existing deserializer for one and b) derive a new instance from that one.

We’ll be good and only tweak the class we want especially since we know that a BeanDeserializer is indeed used in that case:

public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config,
                                              BeanDescription beanDesc,
                                              JsonDeserializer<?> deserializer) {

    if (NotWeird.class.isAssignableFrom(beanDesc.getBeanClass())) {
		// ...
    }

Now we can ask it for the property definition it is using by default for our “details” field:

if (NotWeird.class.isAssignableFrom(beanDesc.getBeanClass())) {
    final BeanDeserializer beanDeserializer = (BeanDeserializer) deserializer;

    final SettableBeanProperty property = beanDeserializer.findProperty("details");

	// ...
}

With property we can derive a customized instance with our deserializer:

final SettableBeanProperty property = beanDeserializer.findProperty("details");

final SettableBeanProperty ourProp = 
	property.withValueDeserializer(new DetailsDeserializer());
beanDeserializer.replaceProperty(property, ourProp);

All that’s left is to implement that DetailsDeserializer:

public class DetailsDeserializer extends JsonDeserializer<NotWeird.Details> {
    @Override
    public NotWeird.Details deserialize(JsonParser p, DeserializationContext ctxt) 
            throws IOException, JsonProcessingException {
        // ...return one
    }
}

As a starting point, we’ll read it as an object (which will throw the same exception):

public NotWeird.Details deserialize(JsonParser p, DeserializationContext ctxt)
        throws IOException, JsonProcessingException {
    final NotWeird.Details details = 
	    p.getCodec().readValue(p, NotWeird.Details.class);

    return details;
}

Let’s hope for the best and wrap the default strategy in a try-catch, use the raw parse tree, and try to process the entry as a single numeric value:

NotWeird.Details details;
try {
    details = p.getCodec().readValue(p, NotWeird.Details.class);
} catch (JsonMappingException e) {
    final TreeNode treeNode = p.getCodec().readTree(p);
    final JsonToken token = treeNode.asToken();

    details = new NotWeird.Details();

    if (token.isNumeric()) {
		// ...
    } else {
        ctxt.handleUnexpectedToken(NotWeird.Details.class, token, p, 
	        "Unsupported content for details");
    }

The given DeserializationContext has several “handle*” methods, which are meant for exactly these cases.

Let’s optimistically parse it as an integer and set count of the details with that value:

if (token.isNumeric()) {
    final String str = treeNode.toString();
    try {
        details.setCount(
                Integer.parseInt(str)
        );
    } catch (NumberFormatException e1) {
        ctxt.handleWeirdStringValue(NotWeird.Details.class, str,
                "Could not parse into an integer");
    }
}

Finally, with that surgically precise adjustment, the application processes both JSON snippets and produces the log (newlines added for clarity) we wanted:

Loaded: [
  NotWeird(name=weird1, details=NotWeird.Details(count=5)), 
  NotWeird(name=weird2, details=NotWeird.Details(count=6))]

Run a Minecraft server container with attached host directory

Sun, 21 Aug 2016 00:00:00 +0000

Attaching host directories to a Docker container instance and getting the read/write permissions correct can actually be a little tricky. By default attached directories are accessed as the same user running the Docker daemon, which is usually root.

My Minecraft server image has a security feature where the java process runs as a non-root user (UID=1000 by default); however, that adds another level of complexity to attached host directory access if your current user isn’t set at user ID 1000.

To counteract that possible mismatch environment variable options are provided to define the user ID and group ID of the process that will run java and access the /data attach point:

-e UID=uid
-e GID=gid

Hands-on Example

First, setup a host directory owned and modifiable by your current user:

sudo mkdir /minecraft
sudo chown -R $(id -un) /minecraft

At this point the directory is empty, but we can startup a new container and let it populate that directory with defaults:

docker run -d \
  --name mc-attached-data \
  -v /minecraft:/data \
  -e UID=$(id -u) -e GID=$(id -g) \
  -e EULA=TRUE \
  -p 25565:25565 \
  itzg/minecraft-server

From the host-side you can browse the active Minecraft data files:

ls /minecraft

You can even stop and remove the container and the populated host directory remains intact. At this point you can adjust the server.properties, replace the world data with one you downloaded, etc.

docker stop mc-attached-data
docker rm mc-attached-data

vi /minecraft/server.properties

At any time after that you can start up a fresh new container, attach the same directory to /data, and this server instance will use that previously configured data directory. In the following example I gave the container a distinct name, “mc-reuse”.

docker run -d \
  --name mc-reuse \
  -v /minecraft:/data \
  -e UID=$(id -u) -e GID=$(id -g) \
  -e EULA=TRUE \
  -p 25565:25565 \
  itzg/minecraft-server

Lessons learned switching to Java Alpine Docker base image

Sat, 09 Jul 2016 00:00:00 +0000

Somewhere around the Java 8u92 versions, the official Docker images for Java started providing Alpine variants of the base images. I believe the benefits of that option are smaller footprint and smaller security attack surface.

Both are fantastic benefits, but there’s some gotchas if you’re used to a Debian (or similar) base image.

Doesn’t include curl or wget

For the reasons above, the Alpine variant doesn’t come with curl or wget. Avoid the temptation to use apk to install those packages and instead take this as a reminder to better leverage Docker.

The ADD Dockerfile instruction already supports source files retrieved via URL. So rather than using RUN to issue a curl/wget, just add a file from a URL natively, such as

ADD https://somehost/somefile.zip /tmp/somefile.tgz

useradd is adduser

If your Dockerfile or scripts are using the command useradd (from Debian land), then use adduser instead. BUT, check out the next couple of sections.

Default system user has no shell

Creating a system user with

adduser -S sysuser

will result in a user than can’t be used with an su -c invocation since no shell is configured by default. Instead explicitly set a shell, such as

adduser -S -s /bin/sh sysuser

Use su to drop privileges

Since sudo isn’t installed by default and we want to minimize adding more packages (to lower security exposure), then use su instead. To run a command as a system user (with an explicit shell configured as above), use something like

su -c "command" sysuser

Don’t use an extra dash in su

Originally I was putting a dash before the username when invoking su thinking that it was a definitive way to tell it where the command ending, like

su -c "command" - sysuser

It turns out giving the dash disables propagation of the environment variables, which is bad since Docker containers use environment variables to pass inputs.

Manually allocate orphaned ES shards to a node

Wed, 06 Jul 2016 20:28:48 +0000

…or: why are no shards being allocated to a node?

Do a

GET _cat/shards

and you’ll see something like

logstash-2015.01.08-raw 4 p STARTED     10238    1.1mb 172.17.0.2 es-01
logstash-2015.01.08-raw 4 r UNASSIGNED                                  
test                    4 p UNASSIGNED                                  
test                    4 r UNASSIGNED                                  
test                    0 p UNASSIGNED                                  
test                    0 r UNASSIGNED                                  
test                    3 p UNASSIGNED                                  
test                    3 r UNASSIGNED                                  
test                    1 p UNASSIGNED                                  
test                    1 r UNASSIGNED                                  
test                    2 p UNASSIGNED                                  
test                    2 r UNASSIGNED                                  

but that’s annoying because we added a second node…but nothing is allocated to it:

Doing

GET _cat/allocation

I’m getting

13.6gb 1gb 14.7gb 92 58fc33034c60 172.17.0.2  es-01      
13.6gb 1gb 14.7gb 92 ff50dd0b4d0e 172.17.0.26 es-00      
                                             UNASSIGNED

What’s up with es-00? Let’s try manually allocating a shard to it:

POST _cluster/reroute
{
  "commands": [
    {
      "allocate": {
        "index": "test",
        "shard": 0,
        "node": "es-00",
        "allow_primary": true
      }
    }
  ]
}

which responds with (newlines inserted for clarity)

{
  "error": "RemoteTransportException[[es-01][inet[/172.17.0.2:9300]][cluster:admin/reroute]];
    nested: ElasticsearchIllegalArgumentException[
      [allocate] allocation of [test][0] on node [es-00][lE3e-Po_QQiXMk_n10jO9Q][ff50dd0b4d0e][inet[/192.168.0.100:9300]] is not allowed,
		YES(shard is not allocated to same node or host)]
		[YES(shard is primary)][YES(below primary recovery limit of [4])]
		[YES(no allocation awareness enabled)][YES(total shard limit disabled: [-1] <= 0)]
		[NO(less than required [15.0%] free disk on node, free: [7.08796015987245%])]
		[YES(no snapshots are currently running)]
	]; ",
  "status": 400
}

Looking for the NO items, we find the cause:

NO(less than required [15.0%] free disk on node, free: [7.08796015987245%])

Written with StackEdit.

Using CircleCI to perform Maven Releases

Wed, 06 Jul 2016 00:00:00 +0000

Your Project

The following operations are performed after you have at least initialized your project for git source control with git init.

Add build-support module

A helper script is needed to execute the Maven release (and deploy) sequence, so first add the build-support sub-module to your project:

git submodule add https://github.com/moorkop/build-support.git

settings.xml

Add a settings.xml to your project’s base directory to convey the server credentials needed to publish to Bintray. The following content can be used as is for that file:

<?xml version='1.0' encoding='UTF-8'?>
<settings xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd'
          xmlns='http://maven.apache.org/SETTINGS/1.0.0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
    <servers>

        <server>
            <id>bintray</id>
            <username>${env.BINTRAY_USER}</username>
            <password>${env.BINTRAY_API_KEY}</password>
        </server>

    </servers>
</settings>

pom.xml

Some specifics will need to be declared within your pom.xml.

SCM section

Add the following section to the top level of your pom.xml but replace the following placeholders with your GitHub specifics:

[[user]]
[[repo]]

<scm>
  <connection>scm:git:https://github.com/[[user]]/[[repo]].git</connection>
  <url>https://github.com/[[user]]/[[repo]]</url>
  <tag>HEAD</tag>
</scm>

Plugins section

Declare the latest deploy plugin to avoid a bug in older versions that would leave out the git commit step. Add the following text as is in the build > plugins section:

<plugin>
  <artifactId>maven-release-plugin</artifactId>
  <version>2.5.3</version>
</plugin>

Distribution management section

To complement the Bintray configuration in settings.xml, add the following text as is to the top level of your pom.xml:

<distributionManagement>
  <repository>
    <id>bintray</id>
    <name>bintray</name>
    <url>https://api.bintray.com/maven/${env.BINTRAY_REPO_OWNER}/${env.BINTRAY_REPO}/${project.artifactId}/;publish=1
    </url>
  </repository>
</distributionManagement>

circle.yml

We’ll hook the release process into the deployment stage of your CircleCI build configuration. Use the following snippet as an example for yours. You can specify your mainline branch instead of master, if needed.

deployment:
  releases:
    branch: master
    commands:
      - build-support/handle-mvn-release.sh

Since a git sub-module is being used you’ll also need to add this:

checkout:
  post:
    - git submodule sync
    - git submodule update --init

Bintray

Next, you will need to prepare a Bintray repository and project.

API Key

Instead of your password, you will use your API key to deploy to Bintray. Obtain your API key from your profile, in the edit section.

Create project entry

Create a project entry with the same name as your project’s artifactId and note the name of the containing repository since you’ll need that for the BINTRAY_REPO variable, below.

CircleCI project

You’re almost done now. The last piece is configuring the project environment variables and push access for GitHub.

Setup GitHub user key access

To enable git push access from within your builds, go to your project’s settings and the section:

Permissions > Checkout SSH keys

Click the action to “Create and add user key”, as shown here:

Setup project environment variables

Also in your project’s settings, add these environment variables:

BINTRAY_USER
BINTRAY_API_KEY
BINTRAY_REPO_OWNER
BINTRAY_REPO

Double-check Ubuntu Version

At the time of writing this, the git push works only with the choice of “Ubuntu 12.04” in the project’s Build Environment settings:

Perform a release

When you’re ready to perform a release, invoke the CircleCI parameterized build API to trigger a build.

In the POST body, replace these placeholders:

[[tag]] : this will be the Git tag of the release and the default release name. GitHub recommends you prefix the tag with a “v” and use semantic versioning.
[[version]] : this will be the Maven project’s version, which is typically the same as the tag but without the “v” prefix.
[[next]] : this is the next snapshot version you’ll want to use for continued development after the release. I recommend using a two-part shortening of the semantic version with the minor version bumped up to the next.
[[email]] : your email address as configured in your GitHub account

In the POST URL, replace these parameters (or use the param editor in a tool like Postman):

:username : GitHub username
:project : GitHub project name
:branch : the branch to build, which needs to match the branch configured in your circle.yml, above.
:token : an API token allocated from the API Tokens section in your account settings

curl -X POST -H "Content-Type: application/json" -d '{
    "build_parameters": {
        "MVN_RELEASE_TAG": "[[tag]]",
        "MVN_RELEASE_VER": "[[version]]",
        "MVN_RELEASE_DEV_VER": "[[next]]-SNAPSHOT",
        "MVN_RELEASE_USER_EMAIL": "[[email]]",
        "MVN_RELEASE_USER_NAME": "Via CircleCI"
    }
}' "https://circleci.com/api/v1/project/:username/:project/tree/:branch?circle-token=:token

Enjoy

Your API-triggered-build includes “Build Parameters” that you can inspect later to see what exact release was performed: