linuxctl.com

Developing a Nomad Job Ansible Module

Fri, 21 Apr 2023 00:00:00 +0000

Installing the module

The module source code can be found at github.com/gbolo/hashicorp-ansible. In the near future I may add it to my ansible galaxy account for easier installation along with adding proper docs. Until then you will need to install it manually by simply copying a couple folders from the git repo linked above.

# retrieve the plugin directory from the git repo
❯ git clone https://github.com/gbolo/hashicorp-ansible /tmp/hashicorp-ansible
❯ mv /tmp/hashicorp-ansible/plugins .

# confirm the structure looks something like this
❯ tree plugins
plugins
├── modules
│   ├── consul_acl_bootstrap.py
│   ├── consul_acl_get_token.py
│   ├── consul_acl_policy.py
│   ├── consul_acl_token.py
│   ├── consul_connect_intention.py
│   ├── consul_get_service_detail.py
│   ├── nomad_acl_bootstrap.py
│   ├── nomad_acl_policy.py
│   ├── nomad_acl_token.py
│   ├── nomad_csi_volume.py
│   ├── nomad_job_parse.py
│   ├── nomad_job.py
│   ├── nomad_namespace.py
│   └── nomad_scheduler.py
└── module_utils
    ├── consul.py
    ├── debug.py
    ├── nomad.py
    └── utils.py

# ensure that your ansible.cfg file has these set
❯ cat ansible.cfg
...
library      = ./plugins/modules
module_utils = ./plugins/module_utils

Using the module

Now that it’s installed, let’s use this module to create a simple nomad job task in your playbook:

- name: ensure nomad job
  run_once: true
  nomad_job:
    url: "{{ nomad_server_url }}"
    management_token: "{{ nomad_acl_management_token }}"
    state: present
    namespace: default
    hcl_spec: |
      job "podinfo" {
        type = "service"
        meta {
          owner = "gbolo"
        }
        namespace = "default"
        group "app" {
          network {
            port "http" {
              to = 9898
            }
          }
          service {
            name = "podinfo"
            port = "http"
          }
          task "app" {
            driver = "docker"
            config {
              image = "stefanprodan/podinfo"
              ports = ["http"]
              args = [
                "./podinfo",
                "--port=9898",
                "--level=debug",
              ]
            }
          }
        }
      }

`--check` mode

specifying --check in an ansible playbook run allows you to determine whether or not a change will occur, without that change actually being executed. It’s like a “Dry Run”. In order to accomplish this we must ensure that our ansible module can properly detect changes. This is perfect opportunity to use the nomad API job plan endpoint, since this endpoint invokes a dry-run of the scheduler for the job.

In the actual module_utils/nomad.py module we have the following code snippets:

# API endpoint for job plan
URL_JOB_PLAN = "{url}/v1/job/{id}/plan?namespace={namespace}"
...

# helper class for use in module
class NomadAPI(object):
...
  # method returns the result of job plan
  def plan_job(self, id, body):
      return self.api_request(
          url=URL_JOB_PLAN.format(url=self.url, id=id, namespace=quote_plus(self.namespace)),
          method="POST",
          body=body,
          json_response=True,
      )

In the actual nomad_job module we have the following code snippets:

# use nomad plan to help decide if we should submit a job
if module.params.get("state") == "present":
    plan = nomad.plan_job(
        job_id,
        json.dumps(
            dict(
                Job=parsed_job,
                Diff=True,
            )
        ),
    )

...

# determine if the job spec changed by inspecting the diff type.
if plan["Diff"].get("Type") != "None":
    result["changed"] = True

`--diff` mode

specifying --diff in an ansible playbook run is easily my favorite feature. This flag should output the actual change in a human readable diff format. Not all ansible modules support such a flag, since the author of the module must explicitly write this functionality. The Nomad API endpoint for job plan returns a diff report that is not very human readable. For example, check out how the response looks like when we make two simple changes; adding a new meta field and changing the docker image tag of a running nomad job:

POST /v1/job/podinfo/plan

{
...
    "Diff": {
        "Fields": [
            {
                "Annotations": null,
                "Name": "Meta[project]",
                "New": "test",
                "Old": "",
                "Type": "Added"
            }
        ],
        "ID": "podinfo",
        "Objects": null,
        "TaskGroups": [
            {
                "Fields": null,
                "Name": "app",
                "Objects": null,
                "Tasks": [
                    {
                        "Annotations": [
                            "forces create/destroy update"
                        ],
                        "Fields": null,
                        "Name": "app",
                        "Objects": [
                            {
                                "Fields": [
                                    {
                                        "Annotations": null,
                                        "Name": "args[0]",
                                        "New": "./podinfo",
                                        "Old": "./podinfo",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "args[1]",
                                        "New": "--port=9898",
                                        "Old": "--port=9898",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "args[2]",
                                        "New": "--level=debug",
                                        "Old": "--level=debug",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "image",
                                        "New": "stefanprodan/podinfo:6.3.4",
                                        "Old": "stefanprodan/podinfo",
                                        "Type": "Edited"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "ports[0]",
                                        "New": "http",
                                        "Old": "http",
                                        "Type": "None"
                                    }
                                ],
                                "Name": "Config",
                                "Objects": null,
                                "Type": "Edited"
                            }
                        ],
                        "Type": "Edited"
                    }
                ],
                "Type": "Edited",
                "Updates": {
                    "create/destroy update": 1
                }
            }
        ],
        "Type": "Edited"
    },
...
}

This output is quite large for such a simple change and is likely intended to be parsed before returning it to human eyes. The nomad binary does this quite nicely (its even colored in the shell!):

❯ nomad job plan artifacts/simple.hcl
+/- Job: "podinfo"
+   Meta[project]: "test"
+/- Task Group: "app" (1 create/destroy update)
  +/- Task: "app" (forces create/destroy update)
    +/- Config {
          args[0]:  "./podinfo"
          args[1]:  "--port=9898"
          args[2]:  "--level=debug"
      +/- image:    "stefanprodan/podinfo" => "stefanprodan/podinfo:6.3.4"
          ports[0]: "http"
        }

To accomplish a similar result in my ansible module I found this excellent python library github.com/strigo/nomad-diff. I made this an optional dependency in the ansible module so that it does not fail if the system does not have it installed:

# import nomad_diff if it is available on the system
_nomad_diff_available = False
import importlib.util

nomad_diff_spec = importlib.util.find_spec("nomad_diff")
if nomad_diff_spec is not None:
    import nomad_diff
    _nomad_diff_available = True

...

# do a nice diff if the system has nomad_diff available
if _nomad_diff_available and plan.get("Diff") is not None:
    try:
        result["diff"] = dict(prepared=nomad_diff.format(plan["Diff"], colors=True, verbose=False))
    except:
        # if we can't get a diff, it's not a big deal...
        pass

Now if we do an ansible run and this task gets called while --diff flag is specified we get this nice feedback (it is also colored when ran in the shell):

❯ ansible-playbook --diff -i inventory/gbolo.digitalocean.yml playbook.yml
...

TASK [ensure nomad job] *****************************************************************************
+/- Job: "podinfo"
+   Meta[project]: "test"
+/- Task Group: "app" (1 create/destroy update)
  +/- Task: "app" (forces create/destroy update)
    +/- Config {
          args[0]:  "./podinfo"
          args[1]:  "--port=9898"
          args[2]:  "--level=debug"
      +/- image:    "stefanprodan/podinfo" => "stefanprodan/podinfo:6.3.4"
          ports[0]: "http"
        }
changed: [gbolo-alpha-mrz01]

Preserving the Original HCL

Nomad 1.6.0 has introduced support for storing the original job HCL spec in the (following PR)[https://github.com/hashicorp/nomad/pull/16763]. I originally created this module before this came out, but adding support for this was really simple as I just had to modify the API call:

result["submit_response"] = nomad.create_or_update_job(
    job_id,
    json.dumps(
        dict(
            Job=parsed_job,
            # Submission structure is new in Nomad 1.6.0
            # it preserves the original HCL spec.
            Submission=dict(
                Format="hcl2",
                Source=module.params.get("hcl_spec"),
            ),
        )
    ),
)

Now when we use the ansible module, the original HCL spec will be preserved in the UI just like the nomad binary does.

Why not use the community module?

Fair question, why didn’t I choose to use the community.general.nomad_job module?

I did try it out, and it works fine enough, however (at the time of this writing) it fell short in a few areas in my opinion:

It requires the pip package python-nomad. Which I feel is a bit overkill for the simple Nomad API.
It lacks a --diff mode, which I feel is essential when doing dry-runs.
It does not preserve the original HCL job spec, which is a new feature in Nomad (introduced in 1.6.0). In order for this module to even support that, the underlying library python-nomad would need to first get that support. See why I think it’s not always a great idea to use a library when dealing with a simple REST API.

The above reasons were along with the fact that I was creating other ansible modules that interact with other areas of the Nomad API (nomad_acl_token, nomad_acl_policy) that did not have any community based modules was more than enough motivation for me to create this nomad_job module.

x509 Certificate Manual Signature Verification

Fri, 03 Feb 2017 00:00:00 +0000

What Does “Signing a Certificate” Mean?

A Certificate Authority (CA) utilizes asymmetric cryptography to form a key pair. This key pair is usually referred to as the public key and the private key. Messages encrypted with one key, can only be decrypted with the other key. The public key is advertised and known to all, however the private key is kept secret and should only been known by the CA. When a Certificate Authority (CA) signs a certificate, what it actually does is hash the certificate then encrypt that hash with it’s private key. This hex code is then embedded into the certificate along with information on how it was derived called the Signature Algorithm. Therefore, in order for one to verify that a certificate was signed by a specific CA, we would only need to possess the following:

the public key of the CA (issuer)

the Signature and Algorithm used to generate signature

The Verification Process

Obtaining the two listed items above is not a difficult task. Let’s examine how we would do this manually

Obtaining the Issuer’s Public Key

The issuer of a x.509 certificate should have it’s own x.509 certificate (that’s also signed if it’s an Intermediate CA, or slef signed if Root CA) to prove it’s authenticity. Once obtaining this certificate, we can extract the public key with the following openssl command:

openssl x509 -in /tmp/rsa-4096-x509.pem -noout -pubkey > /tmp/issuer-pub.pem

Extracting the Signature

Now let’s take a look at the signed certificate. The signature (along with algorithm) can be viewed from the signed certificate using openssl:

openssl x509 -in /tmp/ec-secp384r1-x509-signed.pem -noout -text
...
Signature Algorithm: sha384WithRSAEncryption
     ad:cd:dd:71:cf:8a:79:2c:2d:86:5b:a2:08:7e:00:e0:28:30:
     17:f3:b0:75:a6:12:46:fc:f8:75:9c:4a:57:2f:89:f9:8a:c6:
     03:e4:58:fa:c4:53:13:a7:67:18:60:17:fc:0d:e4:1a:ba:86:
     fd:14:a7:a5:cb:98:a3:d3:ea:40:a8:71:eb:cd:07:b7:e3:16:
     33:8d:ad:45:9f:ca:c7:ca:77:0a:ec:10:42:2a:85:8d:79:de:
     f0:a8:af:38:bc:70:2d:c1:5c:d7:b7:bb:a3:e8:5c:44:d3:89:
     a6:f0:00:8a:b7:1f:df:fd:5b:2a:9e:d3:9c:5f:ec:ad:7a:b8:
     5b:3b:6c:f5:8a:9c:4f:e5:84:d1:c9:ba:af:d5:9c:75:e8:43:
     17:2f:3c:be:e2:49:ef:6a:e1:28:73:f6:57:aa:4e:ec:a3:f9:
     7d:6f:c5:ce:83:20:fc:51:e0:3b:9f:7b:51:06:d3:29:28:08:
     19:99:2b:ca:fb:9e:7c:e9:5e:38:89:ec:71:f8:a4:aa:be:e0:
     1f:65:ed:24:d7:c0:13:aa:0c:f4:b6:f1:24:62:85:38:ac:05:
     f0:d0:de:69:91:dd:98:9d:9a:46:10:30:ac:6a:6a:20:d5:3c:
     c6:10:b4:2b:aa:d9:38:58:07:f2:41:12:23:45:1c:a8:ca:41:
     11:f2:fd:dd:6f:1a:83:ba:4e:ff:ef:b2:0a:7c:32:9e:0d:6b:
     01:17:76:69:59:9c:25:90:54:3d:08:f3:e9:3d:48:14:11:1d:
     09:59:6e:a9:22:1b:f5:c2:80:ee:4d:e7:f5:df:0a:b2:27:f3:
     05:68:3e:23:99:64:bb:d9:dc:25:95:1a:5f:b5:be:7d:0e:b9:
     47:e4:c6:34:0c:22:0a:ad:d0:c5:f9:be:cb:e3:c9:ca:71:d3:
     ea:f6:62:33:2f:08:c4:1e:d6:1e:aa:4b:5c:dd:bc:66:43:18:
     d9:56:68:93:cd:3a:ef:51:b3:67:8a:e9:bd:07:04:95:23:10:
     6c:42:2e:0d:28:a8:b4:c1:81:ba:1b:7e:9d:5c:48:16:6c:04:
     de:53:e6:7d:aa:cb:52:dd:8f:55:8d:09:1f:cf:32:70:ec:81:
     ea:00:07:52:8d:95:2b:b8:b6:ac:b6:36:0a:fd:02:f3:0e:42:
     20:1f:ce:1b:ba:c7:f4:aa:01:53:fb:af:9c:b5:e2:d5:61:7e:
     3c:2b:16:de:22:2b:43:13:f8:5a:6e:ab:c2:0a:85:96:6f:7b:
     b3:ef:49:0f:5f:bf:01:e9:52:f8:de:44:12:23:f5:e6:78:de:
     f6:e0:40:8e:97:89:45:92:f7:73:ee:41:71:7a:7c:de:79:10:
     f2:db:c3:86:85:3c:ac:fe

In the above example, we can tell by the algorithm name sha384WithRSAEncryption that SHA-384 is the cryptographic hash function used and that it was encrypted via RSA.

We can take this hex and dump it to disk as a binary like this:

# extract hex of signature
SIGNATURE_HEX=$(openssl x509 -in /tmp/ec-secp384r1-x509-signed.pem -text -noout -certopt ca_default -certopt no_validity -certopt no_serial -certopt no_subject -certopt no_extensions -certopt no_signame | grep -v 'Signature Algorithm' | tr -d '[:space:]:')

# create signature dump
echo ${SIGNATURE_HEX} | xxd -r -p > /tmp/cert-sig.bin

Decrypting the Signature (RSA)

Now that we have both the encrypted dump of the signature as well as the public key of the issuer. We can decrypt the signature like so:

openssl rsautl -verify -inkey /tmp/issuer-pub.pem -in /tmp/cert-sig.bin -pubin > /tmp/cert-sig-decrypted.bin

We can now finally view the hash with openssl

openssl asn1parse -inform der -in /tmp/cert-sig-decrypted.bin
0:d=0  hl=2 l=  65 cons: SEQUENCE          
2:d=1  hl=2 l=  13 cons: SEQUENCE          
4:d=2  hl=2 l=   9 prim: OBJECT            :sha384
15:d=2  hl=2 l=   0 prim: NULL              
17:d=1  hl=2 l=  48 prim: OCTET STRING      [HEX DUMP]:0C8EDBABEA1EAF2DB57A2A7CC29E2AAA9103167659572DFE10C0D7FF62BA74EC5481D6F049B956775E2BA3E11C5A046A

The digest is in the last line: 0C8EDBABEA1EAF2DB57A2A7CC29E2AAA9103167659572DFE10C0D7FF62BA74EC5481D6F049B956775E2BA3E11C5A046A

Hashing the Original Signature

Now that we got a hash of the orginal certificate, we need to verify if we can obtain the same hash by using the same hashing function (in this case SHA-384). In order to do that, we need to extract just the body of the signed certificate. Which, in our case, is everything but the signature. The start of the body is always the first digit of the second line of the following command:

openssl asn1parse -i -in /tmp/ec-secp384r1-x509-signed.pem
    0:d=0  hl=4 l= 856 cons: SEQUENCE          
    4:d=1  hl=4 l= 320 cons:  SEQUENCE     
...

We can extract this data and store it to disk like so:

openssl asn1parse -in /tmp/ec-secp384r1-x509-signed.pem -strparse 4 -out /tmp/cert-body.bin -noout

Finally, we can run this through the same hashing function to determine the digest

openssl dgst -sha384 /tmp/cert-body.bin
SHA384(/tmp/cert-body.bin)= 0c8edbabea1eaf2db57a2a7cc29e2aaa9103167659572dfe10c0d7ff62ba74ec5481d6f049b956775e2ba3e11c5a046a

As you can see, both hashes match, so we can now confirm that:

/tmp/rsa-4096-x509.pem did sign /tmp/ec-secp384r1-x509-signed.pem

The Finished Script

Now that we went through that manual process, I have put together a script which undergoes a similar process to determine the valididty of a signature. Now in the real world, your browser will be tasked with validating a chain of certificates not just the certificate that signed your domain’s cert. This script only checks if CERT A signed CERT B.

This script should not be relied upon in any shape, way or form. Simply educational.

Here is an example of how to use this script

# supply the signed cert (-s) and issuer's cert (-i) in pem encoding
sh x509-cert-sig-verify.sh -i /tmp/rsa-4096-x509.pem -s /tmp/ec-secp384r1-x509-signed.pem

Issuer Certificate: /tmp/rsa-4096-x509.pem
Signed Certificate: /tmp/ec-secp384r1-x509-signed.pem
Verified OK

# when no arguments supplied
sh x509-cert-sig-verify.sh
  Missing REQUIRED option -i <ISSUER_CERTIFICATE>

  usage: x509-cert-sig-verify.sh [OPTIONS...]

  -i issuer-cert  required  path or url of issuer x509 cert
  -s signed-cert  required  path or url of signed x509 cert
  -v              optional  verbose logging

  *WARNING*
  This script was made for educational purposes.
  VERIFICATION MAY BE WRONG, USE AT YOUR OWN RISK!

If you made it this far down the post, you are awarded the source of the script!

Syncthing - Why You Should be Using it

Sun, 23 Apr 2017 00:00:00 +0000

A Brief Log of the Journey

Before I get into why I feel Syncthing is the best option for a self-hosted syncing solution, let me give an account of my history with syncing software.

Start of the Journey - DropBox

Looking through my email inbox, I found the welcome email I received when I first joined Dropbox. It was delivered in 2011, and I remember how much I loved it back then. Finally, I had a convenient way to sync files across all my computers. Life was easier back then, ignorance was bliss. I was happy for a while, but when I started running my own server at home full time, I began thinking:

I have my Linux box at home with all this free storage available doing almost nothing. why don’t I start using it for my syncing needs?

This immediately made me think of the potential benefits of the self-hosted options:

All-you-can-eat storage for free!
I can put some sensitive/personal files in my shares, and not worry about it being in the cloud.
Gain experience/knowledge with this type of use case.

Of course, I didn’t think much about the issues I may run into.

My two years with OwnCloud

My first foray into te self-hosted options was OwnCloud. I believe at the time it was pretty much the only option I knew of. Since I did a bunch of web development back then with php, OwnCloud was very attractive to me. Installation couldn’t be simpler, just extract the tar into your webroot, set up a mysql database with provided schema and you were ready. Just NAT the port to my OwnCloud VM (which was running Apache at the time with mod_php) and it was now publicly available. At the time there was no let’s encrypt, but browsers also had less of a problem with self-signed TLS certificates. I remember being very proud of it, and would take any opportunity to talk about how cool my set up was. However, my time with OwnCloud was not without it’s issues:

OwnCloud was slow

I have nothing against PHP, in fact I have written most of my web applications in PHP and I know it very well. But, for what ever reason, I found OwnCloud to be slow at times.

Sync conflicts

This would happen often enough, that I had to create a script to periodically run across my shares to scan for them. I will cut OwnCloud some slack, after all they were kind of pioneers in this area. It served my needs for two years but it was time to try something else.

My two years with SeaFile

I divorced OwnCloud and married SeaFile in just an afternoon. Although the installation of SeaFile was significantly more involved, I had already become a talented Linux System Administrator by then. Fronted with nginx, and secured with legitimately trusted TLS certificates obtained for free thanks to Let’s Encrypt, I didn’t think it could get much better than this. Speed was much improved and the whole experience just felt more polished. Things were great, but as my knowledge in the field grew so did my concerns. Although I had control of the in-transit security with TLS terminated on my nginx VM, authentication was still handled by username and password. Also the Android app wasn’t the greatest and would occasionally failed to sync all my photos properly. The last straw was when SeaFile split into two due to some bizarre conflict between two companies.

Syncthing - The Light at the end of the Tunnel

I like to think of Syncthing as my final destination after years searching for the perfect solution. It has a ton of benefits so lets go over them.

Protected with TLS Mutual Authentication

No simple username + password here! When you initialize Syncthing for the first time, it will create an Elliptic Curve Key Pair (although older versions used RSA). It then uses a hash function (sha256) on it’s self-signed x509 certificate to determine it’s unique Device ID as described here. This Device ID must now be shared with other devices you would like to communicate with, which means you are explicitly telling other devices to only trust the holder of the private key which created the public key mentioned in the x509 certificate. This is a far more secure authentication method than any of the other products I used in the past. Sure it is a little less convenient than the others (although the QR code helps), but I am willing to sacrifice some convenience for much improved security. Also, only perfect forward secrecy ciphers are supported, which means if your keys get compromised some how, your previous transfers (if recorded) will still be safe. More details on the protocol spec can be found here.

Distributed Peer to Peer Model

Unlike the other products, there is no central server in Syncthing, it’s completely distributed. Every installation has the same capabilities. Any node in the network can die and it wont disrupt syncing as long as your shares are not shared with only that one dead endpoint. This makes maintenance much easier.

Written in Go

The previous two self-hosted products were made with PHP and Python respectively. Usually, I wouldn’t even mentioned choice of programming language, but I feel strongly about choice of Go for this project. Go (AKA Golang) is one of the most performant programming languages available today, and also enjoys having one of the best modern http and crypto libraries. When you think Google, you think massive http traffic and top of the line encryption. The quality of those two libraries is a reflection of Google’s strength and I think is a perfect marriage for a product like Syncthing.

Fully Open Source

While this is not unique to this product, I still feel that it’s an important quality to mention. It’s also created and maintained by a really talented and devoted guy, Jakob Borg (calmh), who has done an incredible job with this thing. My hats off to him and everyone else who has contributed to this project.

Focus Only on Syncing Files

Unlike SeaFile or OwnCloud, Syncthing’s purpose is only to sync files! There is no code to provide additional features like calendars, document editing, exc. This is great for me, because I never needed any of the other stuff, I was just interested in syncing files! I feel confident in knowing that every commit to this project only has one purpose: to improve the file syncing capabilities of the product.

Cross Platform

Last but not least, cross platform! While the other products were also cross platform (for server side), it’s great to know that the exact same code base is being used on all platforms since there is no separate code base for clients and servers. As a result, my Android app for Syncthing works perfectly. I guess Golang has a bit to do with this as well, but it also speaks volumes of the great design of Syncthing.

TL;DR

Bottom line, if your looking for the most secure self-hosted file synchronization software out there, look no further! Syncthing has you covered, go check it out!

Ansible - Interacting with external REST API

Fri, 20 Jan 2017 00:00:00 +0000

Ansible modules

So Ansible already has modules that can interact directly with rabbitmq. I only need ansible to create a user and vhost that the sensu servers and clients can use. The ones I would normally use if my rabbitmq hosts were not dockerized are:

rabbitmq_user - Adds or removes users to RabbitMQ
rabbitmq_vhost - Manage the state of a virtual host in RabbitMQ

RabbitMQ REST API

Unfortunatly, it appears that all these modules cannot interact with remote rabbitmq daemons. Since we are running rabbitmq in a docker container, we will not have ssh daemon in the docker container for ansible to connect to. However, installing the admin plugin for rabbitmq exposes a REST API, which we can use to create vhosts and users. Fortunatly, there already is an offical docker image which has the admin plugin already installed: 3.6-management-alpine

Now we can simply bring up this rabbitmq docker container and explore the API:

docker run -d --name rabbitmq-mgmt1 -p 127.0.0.1:15672:15672 rabbitmq:3.6-management-alpine

We can use curl to take a look at the vhosts and users:

$ curl -s -u guest:guest http://127.0.0.1:15672/api/vhosts | python -m json.tool
[
    {
        "name": "/",
        "tracing": false
    }
]

$ curl -s -u guest:guest http://127.0.0.1:15672/api/users | python -m json.tool
[
    {
        "name": "guest",
        "password_hash": "X8uG3fwwu3+R3VCEvv0/XC5WI2YcEqYai3+xtb8CT9OaJ0Fl",
        "hashing_algorithm": "rabbit_password_hashing_sha256",
        "tags": "administrator"
    }
]

Creating Ansible Playbook for REST API Integration

According to the api-docs, these endpoints support PUT requests to insert data. Instead of doing curls, let’s build a customizable playbook for Ansible to execute. Lets start by defining the variables we need:

---
- hosts: localhost
  connection: local
  gather_facts: no

  vars:

    rabbitmq_auth_user: "guest"
    rabbitmq_auth_password: "guest"
    rabbimq_rest_api_url: "http://127.0.0.1:15672/api"
    rabbitq_vhost_sensu: "sensu"
    rabbitq_user_sensu: "sensu"

    rabbitq_user_sensu_request_body:
      password: "test"
      tags: ""

    rabbitmq_permissions_sensu_request_body:
      configure: ".*"
      write: ".*"
      read: ".*"

As listed above, we need an rest endpoint to talk to along with the credentials. Also we need the name of the vhost and user we want to create, along with the request body for the user and permissions. Great! Now let’s make our tasks.

We want to first list if our vhost is already present. If it is not, the REST endpoint will return a 404 on a GET request, if the vhost is not present. We need to store this response, so we can use it later to decide if we should do a second request to PUT this vhost on the system. Also, we want the task to fail on any response codes other than 200 (present) or 404 (not found).

  tasks:

    - name: check if sensu vhost is present
      uri:
        url: "{{ rabbimq_rest_api_url }}/vhosts/{{ rabbitq_vhost_sensu }}"
        method: GET
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 200,404
        timeout: 10
      register: request_vhost

    - name: debug
      debug:
        var: request_vhost

    - name: create vhost
      uri:
        url: "{{ rabbimq_rest_api_url }}/vhosts/{{ rabbitq_vhost_sensu }}"
        method: PUT
        HEADER_Content-Type: "application/json"
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 204
        timeout: 10
      when:
        - request_vhost.status == 404

Running this playbook produces the correct result; It will add the vhost if not present, but will skip it if present:

$ ansible-playbook rabbitmq-api-playbook.yml

PLAY [localhost] ***************************************************************

TASK [check if sensu vhost is present] *****************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "request_vhost": {
        "changed": false,
        "content_length": "55",
        "content_type": "application/json",
        "date": "Fri, 10 Jan 2017 21:21:18 GMT",
        "json": {
            "error": "Object Not Found",
            "reason": "\"Not Found\"\n"
        },
        "msg": "HTTP Error 404: Object Not Found",
        "redirected": false,
        "server": "MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)",
        "status": 404,
        "url": "http://127.0.0.1:15672/api/vhosts/sensu",
        "vary": "Accept-Encoding, origin"
    }
}

TASK [create vhost] ************************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   

# Running a second time, skips the step called "create vhost"
$ ansible-playbook rabbitmq-api-playbook.yml

PLAY [localhost] ***************************************************************

TASK [check if sensu vhost is present] *****************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "request_vhost": {
        "cache_control": "no-cache",
        "changed": false,
        "content_length": "32",
        "content_type": "application/json",
        "date": "Fri, 10 Jan 2017 21:23:39 GMT",
        "json": {
            "name": "sensu",
            "tracing": false
        },
        "msg": "OK (32 bytes)",
        "redirected": false,
        "server": "MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)",
        "status": 200,
        "url": "http://127.0.0.1:15672/api/vhosts/sensu",
        "vary": "Accept-Encoding, origin"
    }
}

TASK [create vhost] ************************************************************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Fortunately, the same logic is needed for creating a user is similar but requires us to send a body in json format for PUT requests. Personally, I love yaml and prefer to represent my variables in yaml format, then convert the dictionary to json format with a filter called to_json. The tasks looks like this:

    - name: create user
      uri:
        url: "{{ rabbimq_rest_api_url }}/users/{{ rabbitq_user_sensu }}"
        method: PUT
        HEADER_Content-Type: "application/json"
        body: "{{ rabbitq_user_sensu_request_body | to_json }}"
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 204
        timeout: 10
      when:
        - request_user.status == 404

The Finished Playbook

OK great! Notice how we have to set the HEADER_Content-Type parameter, and how we convert the rabbitq_user_sensu_request_body variable to json format. Now we can put everything together and create the full playbook. Also the great thing about this playbook is that its idempotent. We can run it many times without it changing anything unless the change is needed.

---

- hosts: localhost
  connection: local
  gather_facts: no

  vars:

    rabbitmq_auth_user: "guest"
    rabbitmq_auth_password: "guest"
    rabbimq_rest_api_url: "http://127.0.0.1:15672/api"
    rabbitq_vhost_sensu: "sensu"
    rabbitq_user_sensu: "sensu"

    rabbitq_user_sensu_request_body:
      password: "test"
      tags: ""

    rabbitmq_permissions_sensu_request_body:
      configure: ".*"
      write: ".*"
      read: ".*"

  tasks:

    - name: check if sensu vhost is present
      uri:
        url: "{{ rabbimq_rest_api_url }}/vhosts/{{ rabbitq_vhost_sensu }}"
        method: GET
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 200,404
        timeout: 10
      register: request_vhost

    - name: debug
      debug:
        var: request_vhost

    - name: create vhost
      uri:
        url: "{{ rabbimq_rest_api_url }}/vhosts/{{ rabbitq_vhost_sensu }}"
        method: PUT
        HEADER_Content-Type: "application/json"
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 204
        timeout: 10
      when:
        - request_vhost.status == 404

    - name: check if sensu user is present
      uri:
        url: "{{ rabbimq_rest_api_url }}/users/{{ rabbitq_user_sensu }}"
        method: GET
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 200,404
        timeout: 10
      register: request_user

    - name: debug
      debug:
        var: request_user

    - name: create user
      uri:
        url: "{{ rabbimq_rest_api_url }}/users/{{ rabbitq_user_sensu }}"
        method: PUT
        HEADER_Content-Type: "application/json"
        body: "{{ rabbitq_user_sensu_request_body | to_json }}"
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 204
        timeout: 10
      when:
        - request_user.status == 404

    - name: check if permissions are present
      uri:
        url: "{{ rabbimq_rest_api_url }}/permissions/{{ rabbitq_vhost_sensu }}/{{ rabbitq_user_sensu }}"
        method: GET
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 200,404
        timeout: 10
      register: request_permission

    - name: debug
      debug:
        var: request_permission

    - name: create permissions
      uri:
        url: "{{ rabbimq_rest_api_url }}/permissions/{{ rabbitq_vhost_sensu }}/{{ rabbitq_user_sensu }}"
        method: PUT
        HEADER_Content-Type: "application/json"
        body: "{{ rabbitmq_permissions_sensu_request_body | to_json }}"
        user: "{{ rabbitmq_auth_user }}"
        password: "{{ rabbitmq_auth_password }}"
        force_basic_auth: yes
        status_code: 204
        timeout: 10
      when:
        - request_permission.status == 404

Ideally, you would create a role for this. Perhaps I will show that in another future post.

Hardening Nginx

Sun, 17 Jul 2016 00:00:00 +0000

Default Configuration

I like to keep the default configuration file very sparse and just do includes. I always start by removing the default server block in /etc/nginx/nginx.conf. I then follow that up by removing all files from /etc/nginx/conf.d

Disable Server Token

Really simple, just add this to the server block. All it does is refrain nginx from displaying it’s version number. Throw this file in /etc/nginx/conf.d/00_disable-tokens.conf

# disable information leak
Server_tokens        off;

Disable Unknown Virtual Hosts

I sometimes create a wildcard entry in DNS for certain domains. Doing this may invite http requests for sub-domains which do not exist on your nginx server. Also, attackers sometimes scan the web, and do not even provide a host header when hitting your nginx server. The point here is, we need to create a default server which will catch all these bad host headers and return 444 (nothing)! Throw this file in /etc/nginx/conf.d/01_catch-bad-vhost.conf

# Catch vhosts which we do not host!
server {

  listen        80  default_server;
  server_name  	_; # some invalid name that won't match anything

  # return nothing!
  return        444;
}

TLS Hardening

This next section will deal with TLS hardening

Disable non-SNI requests

Similar to the above section where we disabled unknown virtual hosts, we want to do the same for SNI. You see, we can’t see the user’s host header yet because the tls connection handshake hasn’t happened yet. Since we may have many different tls certificates to offer depending on the domain you would like to visit, they invented SNI to have the client tell the web server which domain they would like to visit as part of the TLS Client Hello message. This would allow the web server to look up the appropriate certificate to respond with. While not all clients can support this (like Internet Explorer on Windows XP), I choose to enforce it anyways. So when a request comes in on my https port that fails to have an SNI or provides an SNI that I do not know about, lets give back a weak self signed certificate and immediately hang up (return a 444!). Throw this file in /etc/nginx/conf.d/01_catch-bad-sni.conf

# Catch SNIs which we do not host!
server {

  listen        443 ssl;
  server_name  	_; # some invalid name that won't match anything

  ssl_certificate     /etc/nginx/ssl/weak/weak.crt;
  ssl_certificate_key /etc/nginx/ssl/weak/weak.key;

  # return nothing!
  return        444;
}

The above configuration needs weak TLS Certificates to be provided. You can create and sign your own > by reading this: openssl reference

TLS Allowed Protocols and Cipher Suites

Choosing cipher suites will hopefully be a thing from the past with the up-comming TLS 1.3 spec. Until then I would highly recommend mozilla’s cipher suite selections. They break it down into compatibility categories like modern, intermediate, old. Ofcourse I would highly recommend choosing from the modern section (this is a hardening post after all). Also I would only enable TLS 1.1 and 1.2. Throw this file in /etc/nginx/conf.d/00_tls-cipher-suites.conf

# allowed protocols and ciphers
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!MD5';

Diffie-Hellman Parameter for EDH Cipher Suites

You will notice that I have chosen some cipher suites that contain EDH. The E stands for Ephemeral. This basically means that these suites provide forward secrecy that can prevent your past sessions (if they were recorded by man-in-middle) to be decrypted should the server private key get compromised. Creating Diffie-Hellman Parameters help better maintain integrity of this key exchange

# generate parameters - may take a while
openssl dhparam -out dhparam4096.pem 4096

Once those parameters have been created, Throw this file in /etc/nginx/00_tls-dhparams.conf

# Diffie-Hellman parameter for DHE ciphersuites
ssl_dhparam /etc/nginx/tls/dhparam4096.pem;

Enable HTTP Strict Transport Security (HSTS)

If everything on your domain is encrpyted, you should probably set this http header. It basically tells the browser that it should expect everything in this domain be accessed via an encrypted channel (https). This would help in situations where an attacker would like to perform ssl stripping on your connections. While this is not a silver bullet by any means, it just adds another layer (a small layer) of additional security. Throw this file in /etc/nginx/00_tls-hsts.conf or individually to certain domains.

# HSTS (HTTP Strict Transport Security) avoids ssl stripping
add_header Strict-Transport-Security max-age=15768000; # six months
# use this only if all subdomains support HTTPS!
# add_header Strict-Transport-Security "max-age=15768000; includeSubDomains";

Migrating old Ubuntu XEN VMs to VMware

Mon, 12 Oct 2015 00:00:00 +0000

Most recently, I had to migrate a pair of VMs under the control of a third-party entity who were running some sort of xen environment.

I was not given any specific details about the environment, just two image files which I was told were the root drives of their existing VMs. After personally being asked by two of my superiors to get this done ASAP due to some kind of “emergency situation”, I reluctantly agreed. Near the end of the work day, a colleague and I began to work on this. This is what we found:

These images were about 1GB each in size and in .img format. They were not full disks but rather just partitions (from previous experience with xen, I am aware that you can specify partitions and not just full disks). After mounting them, we were surprised to find out that one of them was a root drive of a ancient Ubuntu 6.06 server! Time to roll up our sleeves and get to work.

Migrating a XEN Ubuntu 6.06.2 LTS VM over to VMware Manually

There’s no way that the vmware converter can handle this job since it doesn’t like the xen kernel, filesystem or bootloader. OK that’s fine, we can do this manually. Let’s do this one first and prepare the destination VM; we created the VM with oldest emulated hardware available:

Guest OS: Ubuntu 32-Bit
Disk Controller: BusLogic Parallel
Network Adapter: E1000

Step 0

Great, now lets boot our VM to one of our favourite Linux Sys-Admin live ISOs… Finnix! After enabling ssh for comfortable terminal experience we got straight to work.

Step 1

Let’s partition and format our new 1GB drive to be similar to existing image:

parted -s /dev/sda mklabel msdos
parted -a optimal /dev/sda mkpart primary 0% 100%
mkfs.reiserfs /dev/sda1

Step 2

Now lets mount both the xen image file provided by the 3rd-part (which we placed on an NFS share) and our new partition and copy over contents:

mkdir /mnt/{local,nfs,loop}
/etc/init.d/rpcbind start
mount nas:/zpool1/misc /mnt/nfs/
mount -o loop /mnt/nfs/vm_xen_ubu_root.img /mnt/loop/
mount /dev/sda1 /mnt/local/
cp -rp /mnt/loop/* /mnt/local/

Step 3

OK, lets prepare to chroot in there now

mount -t proc proc /mnt/local/proc/
mount -t sysfs sys /mnt/local/sys/
mount -o bind /dev /mnt/local/dev/
chroot /mnt/local/

Step 4

OK we are in! Lets get dns working and try apt:


vi /etc/resolv.conf
apt-get update

…
Failed to fetch http://security.ubuntu.com/ubuntu/dists/dapper-security/main/binary-i386/Packages.gz 404 Not Found [IP: 91.189.91.13 80]
…
Failed to fetch http://archive.ubuntu.com/ubuntu/dists/dapper/main/binary-i386/Packages.gz 404 Not Found [IP: 91.189.91.15 80]
…

Oh no! Looks like this release is too old, we need to find modify our apt sources to find alternative URLs

cat /etc/*release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=6.06
DISTRIB_CODENAME=dapper
DISTRIB_DESCRIPTION=”Ubuntu 6.06.2 LTS”

OK lets replace all references to http://archives.ubuntu.com with http://old-releases.ubuntu.com and try again:

vi /etc/apt/sources.list
apt-get update

Get:1 http://old-releases.ubuntu.com dapper Release.gpg [189B]
Get:2 http://old-releases.ubuntu.com dapper-updates Release.gpg [198B]
Get:3 http://old-releases.ubuntu.com dapper-backports Release.gpg [198B]
Get:4 http://old-releases.ubuntu.com dapper-security Release.gpg [198B]
Hit http://old-releases.ubuntu.com dapper Release
Hit http://old-releases.ubuntu.com dapper-updates Release
…
Reading package lists… Done

Step 5

Success! Taking a look at the /boot/ reveals that this VM was para-virtualized and had no kernel or bootloader. Lets get that installed.

apt-get install lilo
apt-cache search linux
apt-get install linux-image-2.6.15-28-server

# Lets activate the partition
lilo -A /dev/sda 1

Step 6

Let’s do some last minute cleanup before the grand reboot. Check /etc/inittab file to make sure tty1 is set properly then check /etc/fstab for proper mounts (modify or comment out some lines) and reboot. Oh, you may also want disable some services from running on boot if you are missing partitions (like we were)… you maybe want to also make a backup of the shadow file and insert your own root password so that you can login after it boots. Now we can finally reboot.

Works! Ok lets move on to the second VM which seems to be a much more modern version of Linux.

Migrating a XEN Ubuntu 12.04.4 LTS VM over to VMware Manually

Now i know there are better ways to migrate this more modern version, but we will use the same approach but with more a recent filesystem, bootloader and kernel. We can also bump up the disk and network driver:

Guest OS: Ubuntu 64-Bit
Disk Controller: VMware Paravirtual
Network Adapter: VMXNET 3

Step 0

Lets boot our VM to Finnix again, this time selecting the 64-bit version and again enabling ssh.

Step 1-4

Let’s zoom by these steps since they are similar to the ones above with some differences in filesystem:

parted -s /dev/sda mklabel msdos
parted -a optimal /dev/sda mkpart primary 0% 100%
mkfs.ext4 /dev/sda1
mkdir /mnt/{local,nfs,loop}
/etc/init.d/rpcbind start
mount nas:/zpool1/misc /mnt/nfs/
mount -o loop /mnt/nfs/vm_xen_ubu12_root.img /mnt/loop/
mount /dev/sda1 /mnt/local/
cp -rp /mnt/loop/* /mnt/local/
mount -t proc proc /mnt/local/proc/
mount -t sysfs sys /mnt/local/sys/
mount -o bind /dev /mnt/local/dev/
chroot /mnt/local/
vi /etc/resolv.conf
apt-get update

…
Reading package lists… Done

Step 5

Excellent! apt sources are still good. We also have many more kernels to choose from. Lets try using one of the virtualization optimized kernels

apt-cache search linux | grep virtual
apt-get install linux-image-extra-virtual

Installing the kernel also triggered an install of grub which took care of the bootloader for us. Now all that’s left is to re-enable the tty1 as it has been replaced with xen console hvc0. this setting is no longer in the /etc/inittab (since upstart) but can be found here:

cat /etc/init/hvc0.conf
# hvc0 – getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345] and (
not-container or
container CONTAINER=lxc or
container CONTAINER=lxc-libvirt)
stop on runlevel [!2345]
respawn
exec /sbin/getty -8 38400 hvc0

mv /etc/init/hvc0.conf /etc/init/tty1.conf
sed -i -e ‘s/hvc0/tty1/g’ /etc/init/tty1.conf

Step 6

Like last time, remember to do some last minute cleanup before the reboot. This time we won’t have to cross our fingers too tight. Reboot successful!

Grafana & InfluxDB - Getting Started

Tue, 15 Sep 2015 00:00:00 +0000

InfluxDB is a great time-series database for storing server performance metrics like CPU load, memory utilization, network usage, etc. Unlike traditional relational databases like mysql, InfluxDB can store this data in a much smaller amount of disk space while returning the data at much faster rate.

This means we can greatly increase the resolution of our graphs by collecting more data to make our statistics more granular. We are going to poll our systems using collectd every 10 seconds, send this data over the network to our InfluxDB server which has built-in capabilities to read this collectd formatted data. Then we will use another growing-in-popularity opensource tool called Grafana (forked from Kibana) to display this data from InfluxDB in a beautiful dashboard. Here is a quick guide to get you started:

# install and configure collectd
root@watcher ~ $ yum install epel-release
root@watcher ~ $ yum install collectd

# add this file
root@watcher ~ $ vim /etc/collectd.d/influxdb.conf
LoadPlugin network
<Plugin network>
	Server "127.0.0.1" "25826"
</Plugin>

# enable and start collectd
root@watcher ~ $ systemctl start collectd
root@watcher ~ $ systemctl enable collectd

# follow these instructions from: https://influxdb.com
root@watcher ~ $ wget https://s3.amazonaws.com/influxdb/influxdb-0.9.3-1.x86_64.rpm
root@watcher ~ $ yum localinstall influxdb-0.9.3-1.x86_64.rpm

# clean up a bit
root@watcher ~ $ mkdir /etc/influxdb/
root@watcher ~ $ mv /etc/opt/influxdb/influxdb.conf /etc/influxdb/influxdb.conf.orig
root@watcher ~ $ rm -rf /etc/opt/influxdb
root@watcher ~ $ /opt/influxdb/influxd config > /etc/influxdb/influxdb.conf
root@watcher ~ $ chown -R influxdb:influxdb /etc/influxdb/

# modify, change config
root@watcher ~ $ vim /etc/init.d/influxdb
CONFIG=/etc/influxdb/influxdb.conf

# modify, enable collectd
vim /etc/influxdb/influxdb.conf
[collectd]
  enabled = true
  bind-address = ":25826"
  database = "collectd"
  retention-policy = ""
  batch-size = 5000
  batch-timeout = "10s"
  typesdb = "/usr/share/collectd/types.db"

# enable and start influxdb
root@watcher ~ $ chkconfig influxdb on
root@watcher ~ $ service influxdb start

# create database for collectd
root@watcher ~ $ curl -G 'http://localhost:8086/query' --data-urlencode "q=CREATE DATABASE collectd"
root@watcher ~ $ curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "q=SHOW DATABASES"

# install, enable, start grafana
root@watcher ~ $ yum install https://grafanarel.s3.amazonaws.com/builds/grafana-2.1.3-1.x86_64.rpm

root@watcher ~ $ systemctl start grafana-server
root@watcher ~ $ systemctl enable grafana-server

open your browser to this server’s port 3000 (http://x.x.x.x:3000) to connect to grafana, then add influxdb to data sources by clicking Data Sources > Add new. See below (u:root/p:root)

Now to make a graph, click Dashboards > Home > New; then Add Panel > Graph; then click on graph title > edit and follow screenshot below.

After playing around with some of the other settings (like stack) and clicking save dashboard (top), you should get a graph similar to:

Building Tiny Secure Docker Containers

Sun, 12 Mar 2017 00:00:00 +0000

A security chief once told me something along these lines:

Security is layered like an onion

Funny enough, this reminds me of the layers of a docker image. Lowering your attack vector is essential to security. Generally speaking, the more libraries and binaries you have installed on your system, the greater the odds that your system will be susceptible to vulnerabilities discovered tomorrow (all things being equal). Obviously, there is no silver bullet for security, but any hardening guide I ever read always emphasised that:

Minimization of your installation is the most powerful security tool

This, ofcourse, does not mean that larger docker images are always less secure. However, I would say that it is a decent indicator.

Image Size compared to CVEs

Let’s compare image sizes of the most popular base OS images on dockerhub as of today (2016-03-12):

Image	Size	C/M CVE
CentOS 7	70MB	21
CentOS 6	69MB	27
Debian 9	42MB	3
Debian 8	51MB	6
Debian 7	36MB	6
Ubuntu 16.10	41MB	6
Ubuntu 16.04	50MB	8
Ubuntu 14.04	66MB	13
OpenSuse 42.2	48MB	20
OpenSuse 42.1	38MB	20
Alpine 3.5	2MB	1
Alpine 3.4	2MB	0
busybox 1.26-musl	723KB	0

It’s no surprise that most offcial images on the docker hub now include an alpine version! Not only is the base image anymore between 20 to 35 times smaller but also contains far less known vulnerabilities of type critical or major.

How Small can a Container Get?

Using Alpine Linux as your base image can produce pretty small images. However, applications written in languages that can be statically compiled down to a single binary can often make the smallest of container images. This is because you don’t actually need any external libraries for statically compiled applications and can therefore usually skip choosing a base image from above and instead create a docker container from SCRATCH. Let’s take a modern language like Golang for example (which is getting extremely popular!). Let’s create a tiny container for a test web application I wrote in Go to learn the language. Side note: If you interested in learning go, I highly recommend this udemy online course which is taught by gopher Todd McLeod.

Building a container image from SCRATCH

You might be surprised to know that a docker container can function with just a binary in it. Remember that a docker container is essentially just a process. Sometimes, just a set of processes. The kernel is provided by the Host Operating System. It also doesn’t need all the stuff your Host OS contains such as ssh, systemd, syslog, exc. Let’s demonstrate this:

## Pull the application from the repo
git pull https://github.com/gbolo/go-tinyapi.git

## Statically compile this go application (requires go, duh!)
cd go-tinyapi; CGO_ENABLED=0 GOOS=linux go build -o tinyapi

## Verify that this binary has no dependancies on external libraries
ldd tinyapi
	not a dynamic executable

## Build the docker Image
docker build -t tinyapi .

## Run the Docker Container:
docker run --rm -p 8080:8080 tinyapi

## (OPTIONAL) -- Too Lazy
## If you don't want to compile this on your own, I have a pre-made image for you:
####  docker run --rm -p 8080:8080 gbolo/tinyapi:v0.1

Taking a look at the Dockerfile:

Once the docker container is ready, you can open another console and hit it’s various endpoints:

curl -v http://127.0.0.1:8080/
curl -v http://127.0.0.1:8080/alwaysok
curl -v http://127.0.0.1:8080/healthcheck
curl -v http://127.0.0.1:8080/panic

The final endpoint /panic will cause the application to exit when hit killing and erasing your container (your welcome!). Going back to the first console, you can view the logs the application generated to stdout. As you can see, this is an example of a fully functional (yet useless) docker container which has absolutely nothing on the filesystem which it does not need. Pretty neat!

TL;DR

Docker containers should be treated more like a process rather than a Virtual Machine. Build your container around your application, instead of just throwing it in at the last step. Include only what is absolutely needed in your container in order for your application to function properly under all circumstances. This minimalist approach will almost always produce a more secure container. As a bonus, your smaller container will also be more distributable. Now go out there and start making some damned small containers!

OpenSSL - Reference

Mon, 05 Dec 2016 00:00:00 +0000

OpenSSL is quite an extensive project. The docs for the cli (openssl commands) gives you an overview on just how many things you can do with openssl.

All openssl commands listed below were taken from manual of version 1.1.0 and tested with the same version.

Generating Key Material

Commands related to the generation of private key material for asymmetric encryption. Remember to always protect your private keys!

Elliptic Curve (EC)

The only Elliptic Curve algorithms that OpenSSL currently supports are Elliptic Curve Diffie Hellman (ECDH) for key agreement and Elliptic Curve Digital Signature Algorithm (ECDSA) for signing/verifying. Reasons to use EC over RSA include:

Shorter keys needed for equivalent protection compared to RSA.

Lower CPU/Memory requirements.

Better Performance.

References: ecparam | ec

# list available EC curves
openssl ecparam -list_curves

# create EC parameters (using a curve from above list) and a private key but discard parameters
openssl ecparam -name secp384r1 -genkey -noout -out /tmp/ec-secp384r1-key.pem

# create EC parameters (using a curve from above list) and a private key
openssl ecparam -name secp384r1 -genkey -out /tmp/ec-secp384r1-key.pem

# view EC parameters
openssl ecparam -in /tmp/secp384r1-key.pem -noout -text

# encrypt the private key with a symmetric key algorithm
# options are: -aes128|-aes192|-aes256|-camellia128|-camellia192|-camellia256|-des|-des3|-idea
openssl ec -in /tmp/ec-secp384r1-key.pem -aes256 -out /tmp/ec-secp384r1-aes256-key.pem

# remove encryption of a private key
openssl ec -in /tmp/ec-secp384r1-aes256-key.pem -out /tmp/ec-secp384r1-key.pem

# extract public key from private key
openssl ec -in /tmp/ec-secp384r1-key.pem -pubout -out /tmp/ec-secp384r1-pub.pem

Rivest-Shamir-Adleman (RSA)

RSA is the most common type of Public/Private Key. For that reason, it is the most compatible.

References: genrsa | rsa

# generate rsa private key of length 4096 bits
openssl genrsa -out /tmp/rsa-4096-key.pem 4096

# encrypt the private key with a symmetric key algorithm
# options are: -aes128|-aes192|-aes256|-camellia128|-camellia192|-camellia256|-des|-des3|-idea
openssl rsa -in /tmp/rsa-4096-key.pem -aes256 -out /tmp/rsa-4096-aes256-key.pem 4096

# remove encryption of a private key
openssl rsa -in /tmp/rsa-4096-aes256-key.pem -out /tmp/rsa-4096-key.pem

# extract public key from private key
openssl rsa -in /tmp/rsa-4096-key.pem -pubout -out /tmp/rsa-4096-pub.pem

x509 Certificates

An X.509 certificate is a digital certificate used in PKI for authentication. x.509 is the format (standard) used for TLS. The certificate includes the public key along with additional data to help provide authentication.

Creating and Viewing x509 Certificates

Commands related to the generation and viewing of x509 certificates.

References: x509

# create a x509 certificate signed by a private key generated above
openssl req -new -x509 -key /tmp/ec-secp384r1-key.pem -out /tmp/ec-secp384r1-x509.pem -days 365

# view this certificate in human readable format
openssl x509 -in /tmp/ec-secp384r1-x509.pem -text -noout

# view the public key of a x509 certificate in pem format
openssl x509 -in /tmp/ec-secp384r1-x509.pem -noout -pubkey

# convert the encoding of a certificate (text/bin)
# the default is pem, so we need to specify in/outform to der
openssl x509 -in /tmp/ec-secp384r1-x509.pem -outform der -out /tmp/ec-secp384r1-x509.der
openssl x509 -in /tmp/ec-secp384r1-x509.der -inform der -out /tmp/ec-secp384r1-x509.pem

Certificate Signing Request (CSR)

Commands related to the generation, viewing, and verification of Certificate Signing Requests.

References: req | verify

# create a CSR from a private key with sha384 digest
openssl req -new -sha384 -key /tmp/ec-secp384r1-key.pem -nodes -out /tmp/ec-secp384r1-csr.pem

# view a CSR in human readable format
openssl req -in /tmp/ec-secp384r1-csr.pem -noout -text

# sign the CSR with a CA certificate
openssl x509 -req -sha384 -days 360 -in /tmp/ec-secp384r1-csr.pem -CA /tmp/ca-rsa-4096-x509.pem -CAkey /tmp/ca-rsa-4096-key.pem -CAcreateserial -out /tmp/ec-secp384r1-x509-signed.pem

Verification of x509 Certificates

There are two things I usually want to verify about a signed x.509 Certificate:

Did this private key create this certificate?
Did this other x509 certificate sign this certificate?

We can verify that an x509 certificate was generated by a specific private key by comparing the public keys. We can do this for both EC and RSA keys.

References: dgst

# compare the hashes of the public keys
openssl x509 -in /tmp/ec-secp384r1-x509.pem -noout -pubkey | openssl dgst -sha1
openssl ec -in /tmp/ec-secp384r1-key.pem -pubout | openssl dgst -sha1

For signature verification, we also need the certificate of the issuer.

References: verify

# verify the signature
openssl verify -verbose -CAfile /tmp/ca-rsa-4096-x509.pem /tmp/ec-secp384r1-x509-signed.pem

UPDATE - I have written a blog post that includes a script for verifying signatures. Can be found here: x.509 Certificate Signature Verification

Symmetric Encryption

Openssl can be used to encrypt/decrypt data using either keys derived from passphrase or pre-made keys.

References: enc

# list available symmetric ciphers
openssl enc -ciphers

# encrypt a file with aes-256-cbc
openssl enc -aes-256-cbc -salt -in /tmp/rsa-4096-key.pem -out /tmp/rsa-4096-key.pem.aes

# decrypt the file created above
openssl enc -d -aes-256-cbc -salt -in /tmp/rsa-4096-key.pem.aes -out /tmp/rsa-4096-key.pem

Performance Testing

OpenSSL is also a very convenient way to test the encryption capabilities of your server. This is really helpful when deciding what ciphers to use, and the performance impact of them.

References: speed

# basic aes tests software only (no hardware cpu instruction set aes-in)
openssl speed aes-128-cbc aes-192-cbc aes-256-cbc

# enable hardware cpu instruction aes-in
openssl speed -evp aes-128-cbc aes-192-cbc aes-256-cbc

# increase threads
openssl speed -elapsed -evp aes-128-cbc -multi 4

# list available suites (-help triggers error and displays suites)
openssl speed -help

Let's Encrypt Muli-Domain Across Unique IPs

Sat, 28 May 2016 00:00:00 +0000

Let’s Encrypt is awesome! This service allows you to automate the retrieval of as many valid TLS certificates as you wish, as long as you can “prove” that you own the domain. One of the first proofs that they offered was the http-01 challenge. This proof works by essentially sending your domain a random HTTP GET request string which your lets-encrypt client must receive and send back.

Problems with the `http-01` Challenge

By default, the client will listen on port 80 to receive this request, which may be a problem if your webserver is already listening on port 80 to serve your domain’s web traffic. Another potential issue is that you may have subdomains on different hosts and would like to avoid having to install the lets-encrypt agent on each one. Summary of problem:

port 80 may be in use already by a real webserver
forcing certbot to listen on port 80 during renewal will cause a small outage
you need to run certbot on all servers which resolve to the domain list

The Solution

Fortunately, there are many ways to avoid these issues, my favourite being the use of docker containers and nginx's reverse proxy capabilities. Let’s draw out what a solution would look like (at a high level):

The solution involves setting up a central server with a docker container for the lets-encrypt client (certbot) and having nginx reverse-proxy only HTTP requests specific to these challenges back to your certbot docker container.

The Solution Guide

This guide assumes that you already have nginx serving traffic for your domains and that you have some experience with docker.

STEP 1

Install docker and create a directory for certbot config and data

# Install and start docker (on centos/rhel)
yum install -y docker && systemctl start docker

# Create some directories for certs and configs
mkdir -p /opt/certbot/{config,data}

STEP 2

Create Config files for your domains

cat /opt/certbot/config/linuxctl.com.conf

# Example config file
rsa-key-size = 4096
text = True
non-interactive = True
# If an existing cert covers some subset of the requested names, always expand and replace it with the additional names.
expand = True

email = gbolo@linuxctl.com
domains = linuxctl.com, www.linuxctl.com, blog.linuxctl.com, static.linuxctl.com, repo.linuxctl.com, site1.lab.linuxctl.com, site2.lab.linuxctl.com

# https://certbot.eff.org/docs/using.html#standalone
authenticator = standalone
standalone-supported-challenges = http-01

# tells Certbot to continue with cert generation if only some of the specified domain authorazations can be obtained
allow-subset-of-names = true

STEP 3

Set up nginx to reverse proxy requests for challenges:

# Use this Location section on central server to proxy these requests to local certbot:

  location ~ /.well-known/acme-challenge/ {
      allow all;
      proxy_pass       http://127.0.0.1:8080;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
  }

# Use this Location section on remote servers to proxy requests back to your central nginx server:

  location ~ /.well-known/acme-challenge/ {
      allow all;
      proxy_pass       http://CENTRAL_SERVER;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
  }

# !NOTICE!
# MAKE SURE THAT YOUR CENTRAL NGINX SERVER IS CONFIGURED TO HANDLE REQUESTS FOR THESE REMOTE DOMAINS!

STEP 4

Launch a docker container for certbot, bound to 127.0.0.1:8080:

 docker run -d --name certbot --restart=always \
   -p 127.0.0.1:8080:80 \
   -v /opt/certbot/config:/config:ro \
   -v /opt/certbot/data:/etc/letsencrypt \
   gbolo/alp-letsencrypt

STEP 5

Verify that you got your certs

# Check container logs:
docker logs certbot

# Check data directory
tree -L 3 /opt/certbot/data
/opt/certbot/data
├── accounts
│   └── acme-v01.api.letsencrypt.org
│       └── directory
├── archive
│   └── linuxctl.com
│       ├── cert1.pem
│       ├── chain1.pem
│       ├── fullchain1.pem
│       └── privkey1.pem
├── csr
│   └── 0000_csr-letsencrypt.pem
├── keys
│   └── 0000_key-letsencrypt.pem
├── live
│   └── linuxctl.com
│       ├── cert.pem -> ../../archive/linuxctl.com/cert1.pem
│       ├── chain.pem -> ../../archive/linuxctl.com/chain1.pem
│       ├── fullchain.pem -> ../../archive/linuxctl.com/fullchain1.pem
│       └── privkey.pem -> ../../archive/linuxctl.com/privkey1.pem
└── renewal
    └── linuxctl.com.conf

Now that you have your certs in a central location, you can distribute them periodically to any servers that needs them via scp or rsync through a cronjob.

To be fair, the letsencrypt team has provided additional challenges and methods of receiving certs. The method describe above was what I used since near the beginning of the letsencrypt service going live. Other methods are described here: https://certbot.eff.org/docs/using.html#getting-certificates

Docker Storage Driver - Devicemapper

Mon, 18 Apr 2016 00:00:00 +0000

Docker relies on storage engines to layer images. The default storage driver depends on who packaged docker for your OS. Fortunately, it’s not too difficult to change; However you may lose your images and containers so it’s best to decide on a driver when you begin.

Storage Driver Options

Currently (docker 1.11) has the following engines available:

Technology	Driver Name
OverlayFS	overlay
AUFS	aufs
Btrfs	btrfs
Device Mapper	devicemapper
VFS*	vfs
ZFS	zfs

To see your current storage driver try:

docker info | grep -i "storage driver"
Storage Driver: devicemapper

Setting up Device Mapper Driver

Choosing the right storage engine requires some reading. Personally, I went with devicemapper direct-lvm since I have experience working with lvm and it seems to be well supported by docker. Here is how I got it configured on a fresh docker host running centOS 7:

# stop docker
root@c7-1 ~ $ systemctl stop docker

# create your vg,lv
root@c7-1 ~ $ lvcreate --wipesignatures y -n thinpool vg-docker -l 95%VG
Logical volume "thinpool" created.

root@c7-1 ~ $ lvcreate --wipesignatures y -n thinpoolmeta vg-docker -l 1%VG
Logical volume "thinpoolmeta" created.

root@c7-1 ~ $ lvconvert -y --zero n -c 512K --thinpool vg-docker/thinpool --poolmetadata vg-docker/thinpoolmeta
WARNING: Converting logical volume vg-docker/thinpool and vg-docker/thinpoolmeta to pool's data and metadata volumes.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted vg-docker/thinpool to thin pool.

# create this lvm profile
root@c7-1 ~ $ cat /etc/lvm/profile/docker-thinpool.profile
activation {
     thin_pool_autoextend_threshold=80
     thin_pool_autoextend_percent=20
}

# apply the above profile, and verify that your lv is being monitored
root@c7-1 ~ $ lvchange --metadataprofile docker-thinpool vg-docker/thinpool
Logical volume "thinpool" changed.

root@c7-1 ~ $ lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
thinpool vg-docker twi-a-t--- 28.50g 0.00 0.02 monitored

# configure docker deamon
root@c7-1 ~ $ cat /etc/docker/daemon.json
{
         "storage-driver": "devicemapper",
         "storage-opts": [
                 "dm.thinpooldev=/dev/mapper/vg--docker-thinpool",
                 "dm.use_deferred_removal=true"
         ]
}

# This will delete all your images and containers!
root@c7-1 ~ $ rm -rf /var/lib/docker/*

# start docker
root@c7-1 ~ $ systemctl start docker

# verify
...
Server Version: 1.11.0
Storage Driver: devicemapper
Pool Name: vg--docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 320.3 MB
Data Space Total: 30.6 GB
Data Space Available: 30.28 GB
Metadata Space Used: 180.2 kB
Metadata Space Total: 318.8 MB
Metadata Space Available: 318.6 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-12-01)
...

Git - preserve history when moving files

Mon, 21 Mar 2016 00:00:00 +0000

While working on my vSummary github repo, I came to a point where I had decided to change the folder structure of it. At this point I had about 60-ish commits to it. So I began by using the git mv command to move and rename directories then finished off with the usual commit and push commands. However, when I took a look at some of the individual files in the repo I noticed that the commit history for them were gone. While github still had all my commits, the history for individual files/directories seemed to have been erased with the restructuring commit. I did not expect that, but according to github, this is the expected behaviour: Why does github not track renames

Fortunately, I was able to fix this by rolling back my latest commit and using this gist script to preserve the history of my restructuring: https://gist.github.com/emiller/6769886.js

#!/bin/bash
#
# git-mv-with-history -- move/rename file or folder, with history.
#
# Moving a file in git doesn't track history, so the purpose of this
# utility is best explained from the kernel wiki:
#
#   Git has a rename command git mv, but that is just for convenience.
#   The effect is indistinguishable from removing the file and adding another
#   with different name and the same content.
#
# https://git.wiki.kernel.org/index.php/GitFaq#Why_does_Git_not_.22track.22_renames.3F
#
# While the above sucks, git has the ability to let you rewrite history
# of anything via `filter-branch`. This utility just wraps that functionality,
# but also allows you to easily specify more than one rename/move at a
# time (since the `filter-branch` can be slow on big repos).
#
# Usage:
#
#   git-rewrite-history [-d/--dry-run] [-v/--verbose] <srcname>=<destname> <...> <...>
#
# After the repsitory is re-written, eyeball it, commit and push up.
#
# Given this example repository structure:
#
#   src/makefile
#   src/test.cpp
#   src/test.h
#   src/help.txt
#   README.txt
#
# The command:
#
#   git-rewrite-history README.txt=README.md  \     <-- rename to markdpown
#                       src/help.txt=docs/    \     <-- move help.txt into docs
#                       src/makefile=src/Makefile   <-- capitalize makefile
#
#  Would restructure and retain history, resulting in the new structure:
#
#    docs/help.txt
#    src/Makefile
#    src/test.cpp
#    src/test.h
#    README.md
#
# @author emiller
# @date   2013-09-29
#

function usage() {
  echo "usage: `basename $0` [-d/--dry-run] [-v/--verbose] <srcname>=<destname> <...> <...>"
  [ -z "$1" ] || echo $1

  exit 1
}

[ ! -d .git ] && usage "error: must be ran from within the root of the repository"

dryrun=0
filter=""
verbose=""
repo=$(basename `git rev-parse --show-toplevel`)

while [[ $1 =~ ^\- ]]; do
  case $1 in
    -d|--dry-run)
      dryrun=1
      ;;

    -v|--verbose)
      verbose="-v"
      ;;

    *)
      usage "invalid argument: $1"
  esac

  shift
done

for arg in $@; do
  val=`echo $arg | grep -q '=' && echo 1 || echo 0`
  src=`echo $arg | sed 's/\(.*\)=\(.*\)/\1/'`
  dst=`echo $arg | sed 's/\(.*\)=\(.*\)/\2/'`
  dir=`echo $dst | grep -q '/$' && echo $dst || dirname $dst`

  [ "$val" -ne 1  ] && usage
  [ ! -e "$src"   ] && usage "error: $src does not exist"

  filter="$filter                            \n\
    if [ -e \"$src\" ]; then                 \n\
      echo                                   \n\
      if [ ! -e \"$dir\" ]; then             \n\
        mkdir -p ${verbose} \"$dir\" && echo \n\
      fi                                     \n\
      mv $verbose \"$src\" \"$dst\"          \n\
    fi                                       \n\
  "
done

[ -z "$filter" ] && usage

if [[ $dryrun -eq 1 || ! -z $verbose ]]; then
  echo
  echo "tree-filter to execute against $repo:"
  echo -e "$filter"
fi

[ $dryrun -eq 0 ] && git filter-branch -f --tree-filter "`echo -e $filter`"

Once you are finished with your history rewrites, you can force a push like: git push origin master --force

Docker Networking Options - Bridge

Wed, 16 Mar 2016 00:00:00 +0000

By Default, when docker is installed, it will automatically create a bridged network which represents the docker0 interface which is present in all Docker installations. We can view our docker networks using the docker network ls command:

[root@c7-1 ~]# docker network ls
NETWORK ID          NAME                DRIVER
0249796ab0b7        none                null                
47b6faccef28        host                host                
1ecba84224cb        bridge              bridge

the Docker daemon connects containers to this network by default. This network also has a subnet associated with it. We can view more details about this network by issuing the docker network inspect command:

[root@c7-1 ~]# docker network inspect 1ecba84224cb
[
    {
        "Name": "bridge",
        "Id": "1ecba84224cbcc476cc1c1b08f625ac33f9424647664538e545a5c3b1298b08d",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.42.1"
                }
            ]
        },
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        }
    }
]

[root@c7-1 ~]# ifconfig docker0
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.42.1  netmask 255.255.0.0  broadcast 0.0.0.0

We can also notice that the gateway for this network is our docker0 interface on the host. We can also create additional networks on our host, such as a bridge network or an overlay network. The problem with using the built in default bridge is that If you want to communicate with container names you must connect the containers via the legacy docker run --link. Also we can create pockets of isolation for our containers using multiple of these bridged networks. Let’s take a look at creating a new bridged network with our own subnet, which allows us to use some new features that are not present with the default bridge:

[root@c7-1 ~]# docker network create --subnet=192.168.77.0/24 --gateway=192.168.77.1 www
749a3fdd6eb9cfae661e4257fde682f4978936acb5696f865bbfe2f5f7868ca0

[root@c7-1 ~]# docker network ls
NETWORK ID          NAME                DRIVER
0249796ab0b7        none                null                
47b6faccef28        host                host                
1ecba84224cb        bridge              bridge              
749a3fdd6eb9        www                 bridge              

[root@c7-1 ~]# docker network inspect 749a3fdd6eb9
[
    {
        "Name": "www",
        "Id": "749a3fdd6eb9cfae661e4257fde682f4978936acb5696f865bbfe2f5f7868ca0",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.77.0/24",
                    "Gateway": "192.168.77.1"
                }
            ]
        },
        "Containers": {},
        "Options": {}
    }
]

[root@c7-1 ~]# ifconfig | grep -iB1 192.168.77.1
br-749a3fdd6eb9: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.77.1  netmask 255.255.255.0  broadcast 0.0.0.0

To use this new network, we can add the --net=www option when we run our container. We can also use Network-scoped alias for name resolution in our containers with the option --net-alias name. Let’s say I wanted to make 3 containers which talk to each other to deliver my web application such as:

nginx –> php-fpm –> mysql

I can now put these 3 containers in this bridged network and create a separate network for my minecraft container. We also have the flexibility to put the nginx container into multiple bridged networks, since it may need to proxy requests for containers providing different services which are isolated from each other.

[root@c7-1 ~]# docker run -itd --name=nginx --net=www --ip=192.168.77.2 --net-alias=nginx busybox
17ab90c3994c635afcd96e128ed2124c21923c202594e5185b6bf0ba4299b80c
[root@c7-1 ~]# docker run -itd --name=php-fpm --net=www --net-alias=php-fpm busybox
de5b86ca5009287e70c66cfb5cca3979f667c39ea2ee1c639f1334a709045a14
[root@c7-1 ~]# docker run -itd --name=mysql --net=www --net-alias=mysql busybox
e4679102aa2afd7a9803374209ab615b3a7c63596c12df2660463edb15a2465f

[root@c7-1 ~]# docker network inspect www
[
    {
        "Name": "www",
        "Id": "749a3fdd6eb9cfae661e4257fde682f4978936acb5696f865bbfe2f5f7868ca0",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.77.0/24",
                    "Gateway": "192.168.77.1"
                }
            ]
        },
        "Containers": {
            "17ab90c3994c635afcd96e128ed2124c21923c202594e5185b6bf0ba4299b80c": {
                "Name": "nginx",
                "EndpointID": "c6f23376071ec35497ddc76a8eb68d0044fa19f542f42ed4644e857762b25496",
                "MacAddress": "02:42:c0:a8:4d:02",
                "IPv4Address": "192.168.77.2/24",
                "IPv6Address": ""
            },
            "de5b86ca5009287e70c66cfb5cca3979f667c39ea2ee1c639f1334a709045a14": {
                "Name": "php-fpm",
                "EndpointID": "c5a6df5c236f5bd070e2d25edcf52a6634a3d4d8e59705deecd19980ecf165e8",
                "MacAddress": "02:42:c0:a8:4d:03",
                "IPv4Address": "192.168.77.3/24",
                "IPv6Address": ""
            },
            "e4679102aa2afd7a9803374209ab615b3a7c63596c12df2660463edb15a2465f": {
                "Name": "mysql",
                "EndpointID": "1dbf62a4d49a7ce356fea6356a0c283002f227927180f1215798425269b63594",
                "MacAddress": "02:42:c0:a8:4d:04",
                "IPv4Address": "192.168.77.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

Each container on this network can reach each other by the name defined in the option --net-alias=name. Now lets connect nginx container to another bridge network and spin up another container which is part of it:

[root@c7-1 ~]# docker network connect minecraft nginx

[root@c7-1 ~]# docker run -itd --name=minecraft --net=minecraft --net-alias=minecraft busybox
0e48fad03075fd3a39636b66de13f6201a3b4767808b2b62d537eaa567bc2b1a

[root@c7-1 ~]# docker network inspect minecraft
[
    {
        "Name": "minecraft",
        "Id": "72740d4610049bb48782947474fac6d4e25ecfede427ba07c3f1e5077cd48b50",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.78.0/24",
                    "Gateway": "192.168.78.1"
                }
            ]
        },
        "Containers": {
            "0e48fad03075fd3a39636b66de13f6201a3b4767808b2b62d537eaa567bc2b1a": {
                "Name": "minecraft",
                "EndpointID": "e28dfdd943c275a0f95f5a908f80505a5afebf2fabd787610fa67ed5cdbd13c4",
                "MacAddress": "02:42:c0:a8:4e:03",
                "IPv4Address": "192.168.78.3/24",
                "IPv6Address": ""
            },
            "17ab90c3994c635afcd96e128ed2124c21923c202594e5185b6bf0ba4299b80c": {
                "Name": "nginx",
                "EndpointID": "3bb0f6b4962d418b6ffe0844a280b75ccd38a984ea581f14b9effe02ea55e461",
                "MacAddress": "02:42:c0:a8:4e:02",
                "IPv4Address": "192.168.78.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

Testing Connectivity

Now that we have everything setup, lets experiment with connectivity between containers:

[root@c7-1 ~]# docker attach nginx

/ # ping -c1 mysql
PING mysql (192.168.77.4): 56 data bytes
64 bytes from 192.168.77.4: seq=0 ttl=64 time=0.307 ms

--- mysql ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.307/0.307/0.307 ms
/ # ping -c1 php-fpm
PING php-fpm (192.168.77.3): 56 data bytes
64 bytes from 192.168.77.3: seq=0 ttl=64 time=0.134 ms

--- php-fpm ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.134/0.134/0.134 ms
/ # ping -c1 minecraft
PING minecraft (192.168.78.3): 56 data bytes
64 bytes from 192.168.78.3: seq=0 ttl=64 time=0.150 ms

--- minecraft ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.150/0.150/0.150 ms

[root@c7-1 ~]# docker attach minecraft

/ # ping mysql
ping: bad address 'mysql'
/ # ping -c1 192.168.77.4
PING 192.168.77.4 (192.168.77.4): 56 data bytes

--- 192.168.77.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

As expected, our nginx container can talk to the other 4 because it’s in both networks. The other containers can only resolve and ping other containers in their own network. Mission accomplished!

Docker Networking - Change docker0 Subnet

Tue, 02 Feb 2016 00:00:00 +0000

When you install docker, by default it will create a bridged interface docker0 with a 172.17.0.0/16 subnet for container networking. It will also create a MASQUERADE rule on your POSTROUTING iptables chain for container NAT. If this subnet is being used elsewhere on your network, then you should change this default subnet to avoid losing connectivity to these other networks.

Lets view this chain, stop docker and remove this subnet from your docker0 interface:

[root@arch1 ~]# iptables -nL POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0

[root@arch1 ~]# systemctl stop docker
[root@arch1 ~]# ip link set dev docker0 down
[root@arch1 ~]# ip addr del 172.17.0.1/16 dev docker0

Now we need to configure a new subnet for docker0 to use. All we need to do is reconfigure docker0 then start the docker daemon back up and docker will pick up the change and reconfigure the iptables chain:

[root@arch1 ~]# ip addr add 172.17.77.1/24 dev docker0
[root@arch1 ~]# ip link set dev docker0 up
[root@arch1 ~]# systemctl start docker

[root@arch1 ~]# iptables -nL POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.77.0/24       0.0.0.0/0           

[root@arch1 ~]# docker network ls | grep bridge            
f96903f1d9e1        bridge              bridge              

[root@arch1 ~]# docker network inspect f96903f1d9e1
[
    {
        "Name": "bridge",
        "Id": "f96903f1d9e1e278878402b56abae411b0c85f17ab6d3b61c7e56833d623ae3d",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.77.0/24",
                    "Gateway": "172.17.77.1"
                }
            ]
        },
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        }
    }
]

MAKING THE CHANGE PERSISTENT

To make this change PERMANENT, we need to change the way we start up the docker deamon by modifying either the init script or systemd unit file and pass the --bip option:

cat /usr/lib/systemd/system/docker.service | grep -i ExecStart
ExecStart=/usr/bin/docker daemon --bip=172.17.77.1/24 -H fd://

UPDATE - 2016-04-12

Docker now supports the ability to read configuration files from a json file, which by default, is located at /etc/docker/daemon.json. So instead of modifying the flags of the initialization of the docker daemon, we can simply create this file with the following content:

{
  "bip": "172.17.77.1/24"
}

Bootstrapping Hyperledger Fabric 1.0

Wed, 09 Aug 2017 00:00:00 +0000

I am proud to say that I have been working with the Hyperledger Fabric project nearly since it’s inception almost two years ago. While frustrating at times in the beginning (as is the case with most open-source, popular, and pre-alpha software), I have grown to enjoy working with the Fabric. It has improved my skills in a variety of areas such as golang, docker, encryption, pkcs11, continuous integration, and many more. Since the Fabric recently went 1.0, this blog post will focus on how to bootstrap the fabric without the aid of cryptogen tool.

Introduction

The Hyperledger Fabric is an opensource project which provides you a platform/framework to implement/build a blockchain (distributed ledger) application. It is modular in design, and is capable of running chaincode (smart contracts) which essentially is your customized logic on how to write/read from the ledger. Ofcourse this project is tough to explain in just a sentence, so you can read more about it here.

Fabric Components

In order for this blog post to make any sense, you must first understand how the Hyperledger Fabric works. I highly recommend reading the official documentation to accomplish this. Here are some links to components/concepts of interest:

Why not use pre-generated crypto material?

The Fabric repo provides a tool which you can compile to produce crypto material (called cryptogen) required for MSP. While this tool is great for development and testing purposes, I feel that is unacceptable to use for a production grade deployment. In my opinion, private key material should be completely private and secured in pkcs11 based HSMs. Also, the Fabric CA server should be aware of any certificate that it signs. In the current design, the Fabric Peer and Orderer cannot create it’s own MSP directory. It is assumed that the administrator would pre-create this and provide it to them. In older versions of the Fabric (when the orderer didn’t even exist), the Peer was able to enroll itself against the Fabric CA (called membersrvc back then) provided that you supplied the correct secret in it’s config file. In order to accomplish something similar today, we will need to include the fabric-ca-client binary on our fabric peer host.

Bootstrapping the Peer

As mentioned previously, we need the fabric-ca-client binary to enroll with the Fabric CA server in order for us to begin the construction of our MSP directory. A successful enrollment with this binary will produce the following:

The ECDSA private key stored in the keystore directory (if SW based BCSSP is used)
The Enrollment x509 Certificate signed by the CA will be stored in the signcerts directory
The Fabric CA self-signed x509 root Certificate will be stored in the cacerts directory

While this is a great start, we still need two more directories to complete our MSP:

tlscacerts directory is needed to set up our chain of trust for TLS connections
admincerts directory must contain x509 certificates corresponding to an administrator

Obtaining the Administrator Certificate

Before we Enroll the peer we will need to enroll an Administrator user, so that we can obtain it’s x509 Certificate to pass along to the peer. When doing this for development/testing purposes, on our local laptop, we can use local docker volumes to share data between our containers. In this case, we would spin up a container that completes an enrollment for the administrator user and stores it on a volume. Then when the peer starts up, we can have some custom entrypoint logic to copy this certificate to its own local MSP directory after it enrolls itself.

In a more secure production grade deployment, we would enroll the administrative user on a seperate machine, attached to an HSM to secure it’s private key. We can then store it’s x509 enrollment certificate in a git repository and deploy it with a configuration management host such as ansible.

Custom Entrypoint Logic

Since we wish to have the peer enroll it’s self we need to create a custom entrypoint script. This entrypoint script should take in additional values from environment variables for enrollment purposes. This will include things such as the url to the Fabric-CA server and valid credentials.

It is important to note that, for a production grade deployment, these credentials should be registered by the administrator with a max_enrollments set to 1. This will ensure that once these credentials are used once, they can never be re-used again.

In my public test repository, I use the following scripts to accomplish this:

Testing out the Bootstrapping

The bootstrapping i covered above is actually more involved then I explained. For example, I also bootstrap the orderer and use provisional genesis block creation. However, you are free to see and test the full setup in my public git repository. Also note that the images used in this repository are not the official images and are instead created and maintained by myself and are available here:

You will need git, docker, and docker-compose installed in order to try it out for yourself. Here we go:

# Clone the repository
$ git clone https://github.com/gbolo/dockerfiles.git

# Change directory to where the compose file is
$ cd dockerfiles/hyperledger-fabric/softhsm/compose/

# Run the wrapper script which will fetch the required images and compose the environment
$ ./wrapper-test1.sh

Details left out

There is a lot going on with the demonstration provided above that I did not touch on. If I touched on everything, then this blog post would be many times longer. If there’s some questions about fabric or what’s going on in this demo output please feel free to leave a comment below, or find me (gbolo) on the official Hyperledger rocket chat (you will need Linux Foundation credentials to join this chat). For those of you who would like to suggest or make improvments to this boot-strapping demo, please feel free to submit a Pull Request to gbolo/dockerfiles

Virtualbox Control Script

Wed, 02 Jul 2014 00:00:00 +0000

Having been a heavy VirtualBox user while under Linux, I once came across this useful interactive script that I could run in my drop-down terminal that would easily allow me to manage my VMs. So useful was it, that it was aliased on all workstations i used, and carried around in my dropbox account. Here is a screenshot of it in action:

vbctl.sh

#!/usr/bin/env bash
##################################################################################################
# vbctl.sh
# VirtualBox Control
# This script scans and shows information on all virtual machines installed on the current host
# allows to perform basic tasks (start, pause, stop, etc)
# Designed particularly for headless machines
#
# Version 1.0 - May 2, 2011
# Version 1.1 - July 2011   - Enable VRDE activation, improve presentation.
# Version 1.2 - August 2011 - Fix machine listing when having sharedfolders.
# Version 1.3 - August 2011 - Using UUID instead of name for all functions.
#                             Correctly handle machine names with spaces
#                             Code cleaning and ANSI colors on menus.
#                             Animation while reading machines status.
#                             New option to change VRDE port.
#                             Removal of temporary files on exit.
#
# Version 1.4 - September 2011 - Added more color to main menu (guru meditation)
#                                Added option to mount and unmount cd/dvd images
#                                Grouped actions in commands menu
#
# Version 1.5 - September 2011 - Minor fix to list iso and img files for floppy
#                                controllers.
#
# Version 1.6 - October 2012   - Incorporating changes by Lance Norskog from Redwood City, CA
#                                 - Replacing 'seq' with a function to make it run under MacOS X
#                                 - Changed interpreter to /usr/bin/env to make the script more portable
#                                Minor cosmetic changes in VRDE and Load/Eject functions of the VM menu
#                                
# Version 1.7 - January 2013   - Minor fix in seq function to prevent printing an extra delimiter char
#
# Version 1.8 - November 2013  - Incorporating new features made by Sofia Perez
#                                 - Clone VM
#                                 - Destroy VM
#                                
# Version 1.9 - November 2013  - Colored progress bar when loading VM data on TTY that support color
#                                Support for dumb terminals without color (rotating wheel and vm state report)
#
#
# Developed on Oracle Virtual Box 4.0.4 on linux
# Tested on Linux and MacOS X
#
#
# List of contributors:
#
# Carlos Acosta Alamo
#    Original versions 1.0 .. 1.2
#    http://oracleexamples.wordpress.com
#    Oracle DBA
#
# Joao Figueiredo
#    Version 1.3 .. 1.5, 1.7, 1.9
#    http://www.enide.net
#
# Lance Norskog
#    Version 1.6
#    http://
#
# Sofia Perez
#    Version 1.8
#    http://
#
#
# Licensed under Creative Commons Attribution 3.0 Unported (CC BY 3.0)
# http://creativecommons.org/licenses/by/3.0/
# http://creativecommons.org/licenses/by/3.0/legalcode
#
###################################################################################################

## Current version
VERSION=1.9

## Virtualbox suggested VRDE ports
MIN_VRDE_PORT=5000
MAX_VRDE_PORT=5100

#
## Enable color only on these terminal types
#
## You can add more if you know they support color. Just 'echo $TERM' in your shell and add
## the terminal name to this list as another '-o "$TERM" = "terminalName"'
#
if [ "$TERM" = "ansi" -o "$TERM" = "xterm" -o "$TERM" = "linux" -o "$TERM" = "screen" ]; then
  ## ANSI foreground colors
  NORMAL="\033[0m"

  DARKRED="\033[31m"
  RED="\033[31;1m"

  DARKGREEN="\033[32m"
  GREEN="\033[32;1m"

  DARKYELLOW="\033[33m"
  YELLOW="\033[33;1m"

  DARKBLUE="\033[34m"
  BLUE="\033[34;1m"

  DARKPURPLE="\033[35m"
  PURPLE="\033[35;1m"

  DARKCYAN="\033[36m"
  CYAN="\033[36;1m"

  WHITE="\033[37;1m"
  UNDERLINEWHITE="\033[37;4m"
  WHITEONRED="\033[37;41;1m"

  BAR_BACK_COLOR="\033[41m"
  BAR_FORE_COLOR="\033[47m"

  USE_COLOR=1
else
  ## No colors
  NORMAL=""

  DARKRED=""
  RED=""

  DARKGREEN=""
  GREEN=""

  DARKYELLOW=""
  YELLOW=""

  DARKBLUE=""
  BLUE=""

  DARKPURPLE=""
  PURPLE=""

  DARKCYAN=""
  CYAN=""

  WHITE=""
  UNDERLINEWHITE=""
  WHITEONRED=""

  BAR_BACK_COLOR=""
  BAR_FORE_COLOR=""

  USE_COLOR=0
fi

## wheel animation
WHEEL_INDEX=1
WHEEL_CHARS[0]="-"
WHEEL_CHARS[1]="\\"
WHEEL_CHARS[2]="|"
WHEEL_CHARS[3]="/"
function animate_wheel()
{
    # remove old char
    echo -e "\b\b\c"

    # print char
    echo -e "${WHEEL_CHARS[$WHEEL_INDEX]} \c"

    # increment index
    WHEEL_INDEX=`expr $WHEEL_INDEX + 1`
    if [ $WHEEL_INDEX -ge 4 ]; then WHEEL_INDEX=0; fi
}

# macos has no 'seq', had to replace it.
# $1 number of items (1... up to $1)
# $2 separating character
# SEQ is the output variable
SEQ=
function seqN() {
    SEQ=
    C=$1
    GAP="$2"
    N=1
    while [ $C -gt 0 ]
    do
        C=`expr $C - 1`

        if [ $C -gt 0 ]; then
            SEQ="$SEQ$N$GAP"
        else
            SEQ="$SEQ$N"
        fi

        N=`expr $N + 1`
    done
}

#$1 nbr of spaces
function print_spaces()
{
    #SAVEIFS=$IFS
    IFS="-"

    if [ $1 -le 0 ]; then
        return;
    fi
    seqN $1 " "
    SPACES=$(echo $SEQ | sed 's/[0-9]//g')
    echo -e -n ${SPACES}

    #IFS=$SAVEIFS
    unset IFS
}

# $1 extra string for header
function print_header()
{
    FREE_SPACE=55 # 80 - 23 for "VirtualBox Console 1.x" - 2 for []
    REQUIRED_SPCS=`expr ${FREE_SPACE} - ${#1} + 1`

    clear
    echo -e "${WHITE}VirtualBox Console ${VERSION}${NORMAL} \c"
    print_spaces ${REQUIRED_SPCS}
    echo -e "[$1]"
    echo "--------------------------------------------------------------------------------"
    echo ""
}


# progress bar functions
BAR_SIZE=40 # default

# print bar background on current line
# $1 text message to place before the progress bar
# $2 number of steps
function initialize_bar() {

    local i=0
    BAR_SIZE=`expr 80 - ${#1} - 1`

    # bar max steps and remaining chars due to the integer arithmetic
    bar_max=$2
    bar_cur=0
    bar_chars_left=`expr $BAR_SIZE % $bar_max`


    # output message without changing the line and leave a white space
    echo -n -e "$1 "

    seqN $BAR_SIZE " "

    echo -n -e ${BAR_BACK_COLOR}
    for i in $SEQ; do
        echo -n " "
    done
    echo -n -e ${NORMAL}

    # move cursor back to start
    for i in $SEQ; do
        echo -n -e "\b"
    done
}

function increment_bar() {

    local i=0

    let bar_cur=$bar_cur+1;
    let nbr_chars=$BAR_SIZE/$bar_max;

    # add one of the remaining chars (due to the integer arithmetics) every % chars_left of current pos
    if [ $bar_chars_left -gt 0 ]; then
        let nbr_chars=$nbr_chars+1;
        bar_chars_left=`expr $bar_chars_left - 1`
    fi

    local ifs_safe=$IFS
    IFS=" "

    seqN $nbr_chars " "
    for i in $SEQ; do
        echo -n -e "${BAR_FORE_COLOR} "
    done
    echo -n -e "${NORMAL}"

    IFS=$ifs_safe
}


function refresh_vmdetails ()
{
    print_header "scanning for virtual machines"
    #echo -e "Please wait - \c"

    VMDETAILS=.vbox_vmdetails
    TEMPFILE=.vbox_temp
    TEMPFILE2=.vbox_temp2

    local nbr_vms=`VBoxManage list vms|wc -l`
    if [ $USE_COLOR -gt 0 ]; then initialize_bar "Found $nbr_vms VMs, fetching details" "$nbr_vms"
    else echo -e "Found $nbr_vms VMs, fetching details - \c"
    fi

    # JP: IFS is the field separator in BASH, default is space, so I'm redefining it to \n
    # JP: so it's possible to iterate VM names with spaces in the for loop
    IFS=$(echo -en "\n\b")

    VM_GETINFO="Name:|Guest OS:|Memory size:|Number of CPUs:|State:|VRDE:|Hardware UUID:"
    index=1
    rm -f $VMDETAILS >/dev/null 2>&1

    # JP: redefined the awk field separator to \" to get the VM names with spaces
    for i in `VBoxManage list vms|awk -F\" '{print $2}'` ; do
        if [ $USE_COLOR -gt 0 ]; then increment_bar
        else animate_wheel
        fi

        VBoxManage showvminfo "$i"|egrep "$VM_GETINFO">$TEMPFILE2
        head -7 $TEMPFILE2 >$TEMPFILE
        while read vmdetail; do
            echo -n "myvm$index) "$vmdetail >>$VMDETAILS
            echo "" >>$VMDETAILS
        done<$TEMPFILE
        index=$(($index+1));
    done

    # JP: restore IFS state otherwise all echos print a lot of spaces
    unset IFS
}

function list_vm_details ()
{
    print_header "virtual machine control options"
    grep $1 $VMDETAILS|sed -e "s/"$1"/   /"|grep -v VRDE
    grep $1 $VMDETAILS|sed -e "s/"$1"/   /"|grep VRDE|sed -e 's/\(.*\) (\([A-Za-z0-9\. ]*\), \([A-Za-z0-9\ ]*\),.*/\1 (\2, \3)/'
}

function select_vm ()
{
response_valid=1
while [ $response_valid -gt 0 ];do
    print_header "select id from the list or exit with CTRL-C"
    for VMID in `grep "Name:" $VMDETAILS|awk '{print $1}'`
    do
        VMID=$VMID" "
        VMNAME=`grep $VMID $VMDETAILS|sed -e 's/myvm//'|grep "Name:"`
        VRDE=`grep $VMID $VMDETAILS|sed -e "s/${VMID}//g"|grep "VRDE:"|sed -e 's/, Multi.*//' -e 's/(//'`
        VMSTATE=`grep $VMID $VMDETAILS|sed -e "s/${VMID}//g"|grep "State:"|sed 's/(since.*//'`"  "

        if [ $USE_COLOR -eq 0 ]; then
          SHORTVMSTATE="- [`echo $VMSTATE | awk -F\: '{print $2}'|awk '{gsub(/^[ \t]+|[ \t]+$/,""); print;}'`] "
        fi


        case $VMSTATE in
            *"powered off"*) echo -e $DARKRED $VMNAME ${NORMAL}${SHORTVMSTATE}- $VRDE ;;
            *"aborted"*) echo -e $RED $VMNAME ${NORMAL}${SHORTVMSTATE}- $VRDE ;;
            *"running"*) echo -e $DARKGREEN $VMNAME ${NORMAL}${SHORTVMSTATE}- $VRDE ;;
            *"paused"*) echo -e $YELLOW $VMNAME ${NORMAL}${SHORTVMSTATE}- $VRDE ;;
            *"guru meditation"*) echo -e $WHITEONRED $VMNAME - ${VRDE} ${NORMAL}${SHORTVMSTATE} ;;
            *) echo -e $NORMAL $VMNAME - $VMSTATE ;;
        esac
    done # for

    No_VMs=`VBoxManage list vms|wc -l`

    echo ""
    echo -n "Select a VM Id: ";read SELECTED_VM
    if echo $SELECTED_VM | grep "^[0-9]*$">/dev/null
        then
          if [ $SELECTED_VM -gt 0 -a $SELECTED_VM -le $No_VMs ];then
              response_valid=0
         fi
    fi

done #while

SELECTED_VM=myvm$SELECTED_VM")"
SELECTED_VM_NAME=`grep $SELECTED_VM $VMDETAILS|grep "Name:"|awk -F\: '{print $2}'|awk '{gsub(/^[ \t]+|[ \t]+$/,""); print;}'`
SELECTED_VM_UUID=`grep $SELECTED_VM $VMDETAILS|grep "Hardware UUID:"|awk -F\: '{print $2}'`
#SELECTED_VM_STATUS=`grep $SELECTED_VM $VMDETAILS|grep "State:"|awk '{print $3}'`

list_vm_details $SELECTED_VM

echo "--------------------------------------------------------------------------------"
}


function STARTUP_VM ()
{
if  grep $SELECTED_VM $VMDETAILS|grep "VRDE:"|grep "enabled" >/dev/null
then
    echo "Starting up"
else
    read -s -n 1 -p "Warning, VRDE is not enabled, enable it? (Y/[N])" SETVRDE
    if [ $SETVRDE == "Y" ] || [ $SETVRDE == "y" ]
    then
        response_valid=1
        while [ $response_valid = 1 ];do
            echo
            read -p "Enter port number (between $MIN_VRDE_PORT and $MAX_VRDE_PORT): " PORTNUMBER
            if [ $PORTNUMBER -ge $MIN_VRDE_PORT -a $PORTNUMBER -le $MAX_VRDE_PORT ] >/dev/null 2>&1;then
                VBoxManage modifyvm $SELECTED_VM_UUID --vrde on
                VBoxManage modifyvm $SELECTED_VM_UUID --vrdeport $PORTNUMBER
                response_valid=0
            else
                echo "Invalid port number, please choose between $MIN_VRDE_PORT and $MAX_VRDE_PORT"
            fi
        done
    fi
fi
VBoxManage startvm  $SELECTED_VM_UUID --type headless
}

function change_vrde_port ()
{
response_valid=1
while [ $response_valid = 1 ];do
    echo ""
    read -p "Enter port number (between $MIN_VRDE_PORT and $MAX_VRDE_PORT, 0 to return): " PORTNUMBER
    if [ $PORTNUMBER -eq 0 ]; then
      echo "Nothing changed"
      sleep 1
      return;
    elif [ $PORTNUMBER -ge $MIN_VRDE_PORT -a $PORTNUMBER -le $MAX_VRDE_PORT ] >/dev/null 2>&1;then
        VBoxManage modifyvm $SELECTED_VM_UUID --vrde on
        VBoxManage modifyvm $SELECTED_VM_UUID --vrdeport $PORTNUMBER
        response_valid=0
    else
        echo "Invalid port number, please try again"
    fi
done
}

control_vm()
{
response_valid=1
while [ $response_valid -gt 0 ];do
    echo "Available Options"
    echo ""
    echo -e "  1) ${GREEN}STARTUP${NORMAL} (Powered off or Saved)      5) ${YELLOW}Reset${NORMAL}"
    echo -e "  2) ${DARKYELLOW}Pause${NORMAL}                               6) ${YELLOW}Save state${NORMAL}"
    echo -e "  3) ${DARKGREEN}Resume${NORMAL} (from paused)                7) ${YELLOW}Power Off${NORMAL}"
    echo -e "  4) ${DARKRED}ACPI power button${NORMAL}"
    echo -e ""
    echo -e "  8) ${PURPLE}Change VRDE port${NORMAL}                    10) ${DARKGREEN}Clone${NORMAL}"
    echo -e "  9) ${BLUE}Load/Eject removable media${NORMAL}          11) ${DARKRED}Destroy${NORMAL}"
    echo -e "\n  0) Select another machine"
    echo -n -e "\nOption: " ;read VM_ACTION

    if echo $VM_ACTION | grep "^[0-9]*$">/dev/null
     then
       if [ $VM_ACTION -ge 0 -a $VM_ACTION -le 11 ];then
        response_valid=0
       fi
    fi
done  #while

case  $VM_ACTION in
        1 ) STARTUP_VM
            echo -n "Press enter to continue..."; read dummy
            ;;
        2 ) VBoxManage controlvm $SELECTED_VM_UUID pause
            echo -n "Press enter to continue..."; read dummy
        ;;
        3 ) VBoxManage controlvm $SELECTED_VM_UUID resume
            echo -n "Press enter to continue..."; read dummy
        ;;
        4 ) VBoxManage controlvm $SELECTED_VM_UUID acpipowerbutton
            echo -n "Press enter to continue..."; read dummy
        ;;
        5 ) VBoxManage controlvm $SELECTED_VM_UUID reset
            echo -n "Press enter to continue..."; read dummy
        ;;
        6 ) VBoxManage controlvm $SELECTED_VM_UUID savestate
            echo -n "Press enter to continue..."; read dummy
        ;;
        7 ) VBoxManage controlvm $SELECTED_VM_UUID poweroff
            echo -n "Press enter to continue..."; read dummy
        ;;
        8 ) change_vrde_port
        ;;
        9 ) clear
            list_vm_details $SELECTED_VM
            echo "-------------------------------------------------------------------------------"
            ask_user_for_controller

            if [ ${opt} -ne 0 ]; then
              echo ""
              ask_user_for_mount_umount_action "$CTRL_NAME"
              echo ""
              case $opt in
                1) echo ""
                   mount_drive "$CTRL_NAME" "$CTRL_PORT" "$CTRL_DEVICE" "$SELECTED_VM_UUID"
                   echo -n "Press enter to continue..."; read dummy
                ;;
                2) echo ""
                   umount_drive "$CTRL_NAME" "$CTRL_PORT" "$CTRL_DEVICE" "$SELECTED_VM_UUID"
                   echo -n "Press enter to continue..."; read dummy
                ;;
                *) echo "Nothing changed"
                   sleep 1
                ;;
              esac
            else
              echo "Nothing changed"
              sleep 1
            fi

        ;;
        10 )  SELECTED_VM_STATUS=`grep $SELECTED_VM $VMDETAILS|grep "State:"|awk '{print $3}'`
          if [ $SELECTED_VM_STATUS != "powered" ] ; then
            echo -e "You must turn off the virtual machine."        
          else
            read -p "Enter name of new virtual machine (empty to cancel): " VM_NEW
            if [ "$VM_NEW" == "" ]; then
              echo "Nothing changed."
            else
              echo "Start cloning..."
              vboxmanage clonevm $SELECTED_VM_UUID --mode all --options keepnatmacs --name $VM_NEW --register
            fi
          fi

          echo -n "Press enter to continue..."; read dummy
        ;;
        11 )  SELECTED_VM_STATUS=`grep $SELECTED_VM $VMDETAILS|grep "State:"|awk '{print $3}'`
          if [ $SELECTED_VM_STATUS != "powered" ] ; then
            echo -e "You must turn off the virtual machine."        
          else
            read -s -n 1 -p "Are you sure to destroy the virtual machine \"$SELECTED_VM_NAME\"? (Y/[N]) " VM_DESTROY
            if [ "$VM_DESTROY" == "Y" ] || [ "$VM_DESTROY" == "y" ]; then
              echo -e "\nRemoving virtual machine \"$SELECTED_VM_NAME\"..."
              vboxmanage unregistervm $SELECTED_VM_UUID --delete
            else
              echo -e "\nNothing changed."
            fi
          fi

          echo -n "Press enter to continue..."; read dummy
        ;;

        0 ) ;;

esac
}




###############################################################################
###############################################################################
# self contained load/eject floppy/cd/dvd image functions
###############################################################################

# $1 lower limit, $2 upper limit, $3 prompt
# $opt is the output
function read_delimited()
{
  response_valid=1;
  while [ $response_valid -gt 0 ]; do

    read -p "$3" opt;
    [ "$opt" == "" ] && opt=-1;
    if [ $opt -ge $1 ] && [ $opt -le $2 ]; then
      response_valid=0;
    else
      echo "Invalid number, type a number between $1 and $2"
      echo ""
    fi

  done # while
}

# inputs: $SELECTED_VM_UUID, $SELECTED_VM_NAME
# outputs: $CTRL_NAME, $CTRL_PORT, $CTRL_DEVICE
function ask_user_for_controller()
{
  TMP3=.cdchangetmp.txt

  # get all vm info
  VBoxManage showvminfo ${SELECTED_VM_UUID} > ${TMP3}

  # change InternalFieldSeparator to \n
  IFS=$'\n'

  # get interfaces names
  total_ifaces=1
  for IFACE in `cat ${TMP3} | grep "Storage Controller Name" | awk -F: '{print $2}'` ; do
    controller[$total_ifaces]=${IFACE}
    total_ifaces=`expr $total_ifaces + 1`
  done

  # restore to (white spaces and similar)
  unset IFS;

  # decrement one in the counter so it represents the last index of the array controller
  [ $total_ifaces -gt 1 ] && total_ifaces=`expr $total_ifaces - 1`;

  # trim interface names
  seqN $total_ifaces " "
  for i in $SEQ; do
      controller[$i]=`echo ${controller[$i]}`;
  done


  # now get all controller ports for the interface and show them
  echo "Available interfaces on ${SELECTED_VM_NAME}"
  total_ports=1
  seqN $total_ifaces " "
  for i in $SEQ; do

    echo "${controller[$i]}"
    this_controller_nbr_ports=0;

    IFS=$'\n'
    for PORT in `cat ${TMP3} | grep -v "Storage Controller Name" | grep ${controller[$i]} | sed -e 's/(UUID:.*)//g'` ; do
      ports[$total_ports]=${PORT};
      echo "   $total_ports) ${ports[$total_ports]}"
      total_ports=`expr $total_ports + 1`;
      this_controller_nbr_ports=`expr $this_controller_nbr_ports + 1`;
    done
    unset IFS;

    if [ $this_controller_nbr_ports -eq 0 ]; then
      echo "   -) no devices found";
    fi

    echo "";

  done

  echo "   0) Do nothing and go back";
  echo "";

  # restore to (white spaces and similar)
  unset IFS;

  # delete the temporary file
  rm -f ${TMP3}

  # decrement one in the counter so it represents the last index of the array ports
  [ $total_ports -gt 1 ] && total_ports=`expr $total_ports - 1`;

  # ask for controller
  read_delimited 0 $total_ports "Option: "
  #echo "You selected $opt, ${ports[$opt]}"

  # get controller name and trim it
  CTRL_NAME=`echo ${ports[$opt]} | sed -e 's/\([A-Za-z0-9 ]*\)(\([0-9]*\), \([0-9]*\)): \(.*\)/\1/'`;
  CTRL_NAME=`echo $CTRL_NAME`;

  # get port and device
  CTRL_PORT=`echo ${ports[$opt]} | sed -e 's/\([A-Za-z0-9 ]*\)(\([0-9]*\), \([0-9]*\)): \(.*\)/\2/'`;
  CTRL_DEVICE=`echo ${ports[$opt]} | sed -e 's/\([A-Za-z0-9 ]*\)(\([0-9]*\), \([0-9]*\)): \(.*\)/\3/'`;

  #echo "Interface [${CTRL_NAME}]"
  #echo "Port      [${CTRL_PORT}]"
  #echo "Device    [${CTRL_DEVICE}]"
}

# inputs: $1 controller name
# outputs: $opt
function ask_user_for_mount_umount_action()
{
  echo "Action on $1"
  echo "  1) Load Floppy/CD/DVD image"
  echo "  2) Eject image"
  echo "  0) Do nothing and go back"

  read_delimited 0 2 "Option: "
}

# inputs: $1 controller name, $2 port, $3 device, $4 VM_UUID
# outputs: none
function umount_drive()
{
  # Example:
  # VBoxManage storageattach 1a03448b-baeb-4b3c-a811-e3b26d4be8e4 --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium emptydrive

  echo "Ejecting from $1"
  VBoxManage storageattach $4 --storagectl "$1" --port $2 --device $3 --type dvddrive --medium emptydrive
}

# inputs: $1 controller name, $2 port, $3 device, $4 VM_UUID
# outputs: none
function mount_drive()
{
  # ask for file name
  while true; do
    echo "Available disk images"
    list_of_isos=`ls -1 *.iso *.img *.ima 2> /dev/null`
    IFS=$'\n'
    for file in $list_of_isos; do
      echo "  $file";
    done
    unset IFS;
    echo " "

    read -p "Filename (no name to cancel): " filename;

    if [ "$filename" == "" ] ; then
      echo ""
      echo "Nothing changed";
      return;
    fi

    if [ -f "$filename" ] ; then
      break;
    else
      echo "$filename not found, check name and try again";
    fi

  done

  # mount
  echo "Loading $filename on $1"
  VBoxManage storageattach $4 --storagectl "$1" --port $2 --device $3 --type dvddrive --medium "$filename"
}

###############################################################################
# end of load/eject cd/dvd images
###############################################################################
###############################################################################



function cleanup_before_exit()
{
    echo -e "\nSee you soon...";
    rm -f ${VMDETAILS} ${TEMPFILE} ${TEMPFILE2}
}

####
#### MAIN
####

#set -x

if [ "`which vboxmanage`" == "" ] || [ "`which VBoxManage`" == "" ]; then
  echo -e "Error: `basename $0` requires VirtualBox utilities to be installed and available in the \$PATH\n       Make sure vboxmanage is available and try again.\n"
  exit 1
fi

trap "cleanup_before_exit;exit" SIGINT SIGTERM
clear
while [ 1 = 1 ];do
    refresh_vmdetails
    select_vm
    #read -s -n 1 -p "Press a key to continue.."
    control_vm
done

Netappp 7-mode - CLI Reference

Tue, 14 Jan 2014 00:00:00 +0000

The great thing about Netapps are that they were designed with CLI in mind first. The web GUI came later. Because of that, the CLI is a joy to work with. This post is for reference purposes

The Basics

setup (Re-Run initial setup)
halt (Reboots controller into bootrom)
reboot (Reboots the connected controller)
sysconfig -a (Dumps the system configuration)
storage show disk (shows physical information about disks)
passwd (Changes the password for the current user)
sasadmin shelf (shows a graphical layout of your shelves with occupied disk slots)
options trusted.hosts x.x.x.x or x.x.x.x/nn (hosts that are allowed telnet, http, https and ssh admin access. x.x.x.x = ip address, /nn is network bits)
options trusted.hosts * (Allows all hosts to the above command)

Check NFS clients and Traffic

## Enable nfs.per_client_stats.enable
options nfs.per_client_stats.enable on

## List NFS Clients connected/working
nfsstat -l

## Reset list
nfsstat -z

Modifying NFS Exports

## Read exports:
rdfile /etc/exports

## Write new export file:
wrfile /etc/exports
** press ctrl+c when done **

## Apply exports
exportfs -a

Networking

## rc file (startup config):
ifgrp create lacp lacptrunk1 -b ip e3a e3b  // create lacp interface using e3a and e3b
vlan create lacptrunk1 300 301 302 310 315  // trunk these vlans on this lacp interface
ifconfig lacptrunk1-315 10.131.15.12 netmask 255.255.255.0 partner lacptrunk1-315 // assign tagged IP to interface

## realtime live changes
vlan add lacptrunk1 315  // add new vlan 315 to lacp interface
ifconfig lacptrunk1-315 10.131.15.12 netmask 255.255.255.0 partner lacptrunk1-315 // assign tagged IP to interface

Diagnostics

Press DEL at boot up during memory test followed by boot_diags and select all
priv set diags (Enter diagnostics CLI mode from the Ontap CLI)
priv set (Return to normal CLI mode from diagnostics mode)

Software

software list (Lists software in the /etc/software directory)
software delete (Deletes software in the /etc/software directory)
software update 8.1RC2_e_image.zip -r (Install software. The -r prevents it rebooting afterwards)

Aggregates

aggr create aggregate_name (Creates an Aggregate)
aggr destroy aggregate_name (deletes an Aggregate)
aggr offline aggregate_name (takes an Aggregate offline)
aggr online aggregate_name (brings an Aggregate online)
aggr status (shows status of all aggregates)
aggr status aggregate_name (show status of a specific Aggregate)
aggr show_space aggregate_name (shows specific aggregate space information)

Volumes

vol create volume_name (Creates a volume)
vol status (gives the status of all volumes)

Snapshots

snap create volume_name snapshot_name (create a snapshot)
snap list volume_name (List snapshots for a volume)
snap delete volume_name snapshot_name (delete a snapshot on a volume)
snap delete -a volume_name (Deletes all snapshots for a volume)
snap restore -s snapshot_name volume_name (Restores a snapshot on the specified volume name)
options cifs.show_snapshot on (Sets snapshot directory to be browse-able via CIFS)
options nfs.hide_snapshot off (Sets snapshot directory to be visible via NFS)

SnapMirror

options snapmirror.enable on (turns on SnapMirror. Replace on with off to toggle)
vol restrict volume_name (Performed on the Destination. Makes the destination volume read only which must be done for volume based replication)
snapmirror initialize -S srcfiler:source_volume dstfiler:destination_volume (Performed on the destination. This is for full volume mirror. For example snapmirror initialize -S filer1:vol1 filer2:vol2)
snapmirror status (Shows the status of snapmirror and replicated volumes or qtree’s)
snapmirror status -l (Shows much more detail that the command above, i.e. snapshot name, bytes transferred, progress, etc)
snapmirror quiesce volume_name (Performed on Destination. Pauses the SnapMirror Replication. If you are removing the snapmirror relationship this is the first step.)
snapmirror break volume_name (Performed on Destination. Breaks or disengages the SnapMirror Replication. If you are removing the snapmirror relationship this is the second step followed by deleting the snapshot)
snapmirror resync volume_name (Performed on Destination. When data is out of date, for example working off DR site and wanting to resync back to primary, only performed  when SnapMirror relationship is broken)
snapmirror update -S srcfiler:volume_name dstfiler:volume_name (Performed on Destination. Forces a new snapshot on the source and performs a replication, only if an initial replication baseline has been already done)
snapmirror release volume_name dstfiler:volume_name (Performed on Destination. Removes a snapmirror destination)

Cluster (HA Partner)

cf enable (enable cluster)
cf disable (disable cluster)
cf takeover (take over resources from other controller)
cf giveback (give back controller resources after a take over)

Disks

disk show (Show disk information)
disk show -n (Show unowned disks)

## Show spares
vol status -r

LUNS

lun setup (runs the cli lun setup wizard)
lun create -s 10g -t windows_2008 -o noreserve /vol/vol1/lun1 (creates a lun of 10GB with type Windows 2008, sets no reservation and places it in the following volume or qtree)
lun offline lun_path (takes a lun offline)
lun online lun_path (brings a lun online)
lun show -v (Verbose listing of luns)

Fiber FCP

fcp config 0b down (Brings down fcp adapter 0b)
fcadmin config -t target 0a (Changes adapter from initiator to target)
fcadmin config (lists adapter state)
fcp start (Start the FCP service)
fcp stop (Stop the FCP service)
fcp show adapters (Displays adapter type, status, FC Nodename, FC Portname and slot number)
fcp nodename (Displays fiber channel nodename)
fcp show initiators (Show fiber channel initiators)
fcp topology show 1a (Show topology for adapter 1a)
fcp portname show -v (shows list of all avilable used/unused WWPNs)
fcp wwpn-alias set alias_name (Set a fiber channel alias name for the controller)
fcp wwpn-alias remove -a alias_name (Remove a fiber channel alias name for the controller)
igroup show (Displays initiator groups with WWN’s)

CIFS

cifs setup (cifs setup wizard)
cifs restart (restarts cifs)
cifs shares (displays cifs shares)
cifs status (show status of cifs)
cifs domain info (Lists information about the filers connected Windows Domain)
cifs testdc ip_address (Test a specific Windows Domain Controller for connectivity)
cifs prefdc (Displays configured preferred Windows Domain Controllers)
cifs prefdc add domain address_list (Adds a preferred dc for a specific domain i.e. cifs prefdc add netapplab.local 10.10.10.1)
cifs prefdc delete domain (Delete a preferred Windows Domain Controller)
vscan on (Turns virus scanning on)
vscan off (Turns virus scanning off)
vscan reset (Resets virus scanning)

SIS / Deduplication

sis status (Shows SIS status)
sis config (Shows SIS config)
sis on /vol/vol1 (Turns on deduplication on vol1)
sis start -s /vol/vol1 (Runs deduplication manually on vol1)
sis status -l /vol/vol1 (Displays deduplication status on vol1)
df -s vol1 (View space savings with deduplication)
sis stop /vol/vol1 (Stops deduplication on vol1)
sis off /vol/vol1 (Disables deduplication on vol1)
aggr show_space aggr0 -g

DNS

dns flush (Flushes the DNS cache)
/etc/resolv.conf (edit this file to change your dns servers)

BigchainDB on CentOS 7

Sat, 19 Mar 2016 00:00:00 +0000

It’s been the year of the no-sql distributed databases for me. Technically, it started a few years back when I discovered the ELK stack, but this year I found myself evaluating many more of these distributed databases for an exciting project at my workplace. A few months back I started playing with Apache Cassandra and was really impressed with it’s capabilities. However, I since switched my focus to evaluate a new kid on the block (or should I say block chain?) called bigchaindb which may be more suited to the requirements of our project.

Since we are mostly a RedHat shop, I decided to try and get this running on the latest CentOS 7. This application has a few requirements:

RethinkDB
Python 3.4+
OpenSSL ==with ECC==

RethinkDB is fairly simple to install and Python 3.4 is available on the epel repo. However, openssl on CentOS 7 only ships with these three ECC curves compiled in:

root@c7-dev3 ~ $ openssl ecparam -list_curves
  secp384r1 : NIST/SECG curve over a 384 bit prime field
  secp521r1 : NIST/SECG curve over a 521 bit prime field
  prime256v1: X9.62/SECG curve over a 256 bit prime field

In order to get more ECC curves, we are going to have to compile openssl from source. To preserve the current default openssl installation on this OS, I will be compiling it, then placing it to a non-standard location. I consider this a safer approach then just replacing what we already have installed. So let’s get started:

# Download source and compile
root@c7-dev3 ~ $ yum -y install gcc
root@c7-dev3 ~ $ mkdir -p /opt/openssl-1.0.1s/build; cd /opt/openssl-1.0.1s
root@c7-dev3 /opt/openssl-1.0.1s $ wget https://www.openssl.org/source/openssl-1.0.1s.tar.gz
root@c7-dev3 /opt/openssl-1.0.1s $ tar xzvf openssl-1.0.1s.tar.gz
root@c7-dev3 /opt/openssl-1.0.1s $ mv openssl-1.0.1s src; cd src
root@c7-dev3 /opt/openssl-1.0.1s/src $ export CFLAGS="-fPIC"
root@c7-dev3 /opt/openssl-1.0.1s/src $ ./config --prefix=/opt/openssl-1.0.1s/build shared enable-ec enable-ecdh enable-ecdsa
root@c7-dev3 /opt/openssl-1.0.1s/src $ make all
root@c7-dev3 /opt/openssl-1.0.1s/src $ make install

# Check if we got more curves
root@c7-dev3 /opt/openssl-1.0.1s/src $ openssl ecparam -list_curves | wc -l
3
root@c7-dev3 /opt/openssl-1.0.1s/src $ /opt/openssl-1.0.1s/build/bin/openssl ecparam -list_curves | wc -l
73

Alright, now that we got the openssl curves we needed, we can continue installing the rest:

# Install RethinkDB
root@c7-dev3 ~ $ wget https://download.rethinkdb.com/centos/6/`uname -m`/rethinkdb.repo -O /etc/yum.repos.d/rethinkdb.repo
root@c7-dev3 ~ $ yum -y install rethinkdb

# Install python 3.4 and some other deps
root@c7-dev3 ~ $ yum -y install epel-release
root@c7-dev3 ~ $ yum -y install libffi-devel gcc-c++ redhat-rpm-config python34-devel

# Install pip
root@c7-dev3 ~ $ curl https://bootstrap.pypa.io/get-pip.py | python3.4

# Install bigchain and specify the openssl headers and libraries
root@c7-dev3 ~ $ CFLAGS='-I /opt/openssl-1.0.1s/src/include' LDFLAGS='-L /opt/openssl-1.0.1s/build/lib -Wl,-rpath=/opt/openssl-1.0.1s/build/lib' pip --no-cache-dir install bigchaindb

# start rethinkdb & bigchaindb
root@c7-dev3 ~ $ rethinkdb --daemon && bigchaindb start

MemSQL - Cheatsheet

Wed, 16 Dec 2015 00:00:00 +0000

MemSQL is a cool distributed In-Memory Database which offers high performance, sharded horizontal scale-out design, High Availability (with Enterprise edition), and the familiar SQL syntax. It also ships with a great looking GUI that displays most of information you need to know about your cluster.

However, some things are better dealt with in the cli, this page is for my reference:

Listing status

# List Nodes
memsql-ops memsql-list                          
 ID       Agent Id  Process State  Cluster State  Role        Host         Port
 0669D57  A116b09   RUNNING        CONNECTED      MASTER      10.1.10.51  3306
 B98DA2F  Abefe6b   RUNNING        CONNECTED      AGGREGATOR  10.1.10.52  3306
 32BA03D  A116b09   RUNNING        CONNECTED      LEAF        10.1.10.51  3307
 9472F86  Abefe6b   RUNNING        CONNECTED      LEAF        10.1.10.52  3307
 B1776D6  Abefe6b   RUNNING        CONNECTED      LEAF        10.1.10.52  3308
 B97142B  A116b09   RUNNING        CONNECTED      LEAF        10.1.10.51  3308


# Show leaves (connect via mysql)
mysql> SHOW LEAVES;
+-------------+------+--------------------+-------------+-----------+--------+--------------------+------------------------------+
| Host        | Port | Availability_Group | Pair_Host   | Pair_Port | State  | Opened_Connections | Average_Roundtrip_Latency_ms |
+-------------+------+--------------------+-------------+-----------+--------+--------------------+------------------------------+
| 10.1.10.51 | 3307 |                  1 | 10.1.10.52 |      3308 | online |                 18 |                        0.262 |
| 10.1.10.52 | 3307 |                  1 | 10.1.10.51 |      3308 | online |                 18 |                        0.561 |
| 10.1.10.52 | 3308 |                  2 | 10.1.10.51 |      3307 | online |                 17 |                        0.465 |
| 10.1.10.51 | 3308 |                  2 | 10.1.10.52 |      3307 | online |                 17 |                        0.189 |
+-------------+------+--------------------+-------------+-----------+--------+--------------------+------------------------------+
4 rows in set (0.00 sec)

# Show partitions (select db)
mysql> SHOW PARTITIONS;                     
+---------+-------------+------+--------+--------+
| Ordinal | Host        | Port | Role   | Locked |
+---------+-------------+------+--------+--------+
|       0 | 10.1.10.52 | 3308 | Master |      0 |
|       0 | 10.1.10.51 | 3307 | Slave  |      0 |
|       1 | 10.1.10.51 | 3308 | Master |      0 |
|       1 | 10.1.10.52 | 3307 | Slave  |      0 |
|       2 | 10.1.10.52 | 3308 | Master |      0 |
|       2 | 10.1.10.51 | 3307 | Slave  |      0 |
|       3 | 10.1.10.51 | 3308 | Master |      0 |
|       3 | 10.1.10.52 | 3307 | Slave  |      0 |
|       4 | 10.1.10.52 | 3308 | Master |      0 |
|       4 | 10.1.10.51 | 3307 | Slave  |      0 |
|       5 | 10.1.10.51 | 3308 | Master |      0 |
|       5 | 10.1.10.52 | 3307 | Slave  |      0 |
|       6 | 10.1.10.52 | 3308 | Master |      0 |
|       6 | 10.1.10.51 | 3307 | Slave  |      0 |
... TRUNCATED

Upgrading memsql

# Download the latest ops and bin from: http://download.memsql.com/releases/latest/memsqlbin_amd64.tar.gz http://www.memsql.com/download/thanks/

# Upgrade memsql
root@memsql-1 ~ $ memsql-ops memsql-upgrade --file-path /home/skroot/memsqlbin_amd64_5.0.8.tar.gz
Adding a MemSQL file
Unpacking archive to determine its version
Copying file into MemSQL Ops data directory
Registering file with MemSQL Ops
Successfully added a MemSQL file with version 5.0.8
Are you sure you want to upgrade your cluster to MemSQL version 5.0.8? This will temporarily shut down your cluster. [y/N] y
Do you want to back up the data directories in your cluster before the upgrade? If you choose no, we will use less disk space during the upgrade, but the risk of data loss will be greater. [Y/n] y
Starting MemSQL Upgrade
Currently on state OfflineClusterUpgrade.OfflineClusterUpgradeInit
Currently on state OfflineClusterUpgrade.SnapshottingDatabases
0/6 MemSQL nodes have snapshotted their databases
1/6 MemSQL nodes have snapshotted their databases
2/6 MemSQL nodes have snapshotted their databases
4/6 MemSQL nodes have snapshotted their databases
6/6 MemSQL nodes have snapshotted their databases
Currently on state OfflineClusterUpgrade.StoppingCluster
0/6 MemSQL nodes stopped
1/6 MemSQL nodes stopped
2/6 MemSQL nodes stopped
6/6 MemSQL nodes stopped
Currently on state OfflineClusterUpgrade.UpgradingCluster
0/6 MemSQL nodes have upgraded
1/6 MemSQL nodes have upgraded
3/6 MemSQL nodes have upgraded
4/6 MemSQL nodes have upgraded
5/6 MemSQL nodes have upgraded
6/6 MemSQL nodes have upgraded
Currently on state OfflineClusterUpgrade.StartingCluster
0/6 MemSQL nodes started
1/6 MemSQL nodes started
3/6 MemSQL nodes started
5/6 MemSQL nodes started
6/6 MemSQL nodes started
Currently on state OfflineClusterUpgrade.DeletingOldInstalls
0/6 MemSQL nodes have cleaned up their old installs
3/6 MemSQL nodes have cleaned up their old installs
6/6 MemSQL nodes have cleaned up their old installs
Currently on state OfflineClusterUpgrade.OfflineClusterUpgradeSuccess
Successfully upgraded cluster to MemSQL version 5.0.8

# Upgrade memsql ops
root@memsql-1 ~ $ memsql-ops agent-upgrade --file-path /home/skroot/memsql-ops-5.0.3.tar.gz              
Adding a MemSQL Ops file
Unpacking archive to determine its version
Copying file into MemSQL Ops data directory
Registering file with MemSQL Ops
Successfully added a MemSQL Ops file with version 5.0.3
Starting upgrade
Upgrading followers
Upgrading primary agent
Successfully upgraded all agents

Mon, 01 Jan 0001 00:00:00 +0000

Contrib

Mon, 01 Jan 0001 00:00:00 +0000

Open-source is very important to me. I am extremely grateful for all the open-source projects I have encountered through-out the years that have helped me develop solutions in both my professional and personal life. Without the gift of open-source I would not have been able to accomplish much in my career! Being an active part of the open-source community continues to motivate me and it is with great pleasure and pride that I contribute back to the community that has given me so many cool things!

If you would like to see open-source projects that I started follow this link.

Open-source Community Contributions

The list below is filtered to include any significant pull requests that I have contributed to various open-source projects as well as any significant issues that I have opened that led to a PR.

Project	Year	Description	Link(s)
hashicorp/consul (Orchestration)	2023	bug fix: AccessorID in request body should be optional when updating an ACL token	PR
hashicorp/nomad (Orchestration)	2023	bug: API returning 404 when attempting to create/update ACL policy with invalid rules	Issue
hashicorp/nomad (Orchestration)	2023	bug: ACL Token ExpirationTTL is broken or not documented properly	Issue
ansible-collections/community.general (Automation)	2023	improvements to the keycloak_realm_key module	PR
dev-sec/ansible-collection-hardening (Automation)	2023	bug fix in os hardening role	PR
gofiber/fiber (HTTP Framework)	2022	support getting client IP more accurately and reliably	PR
mkeeler/consul-mock-api (Mock/Testing)	2022	return a slice of api.ACLPolicyListEntry for /v1/acl/policies	PR
lestrrat-go/jwx (crypto)	2022	support for additional signers aws KMS or pkcs11	Issue
lestrrat-go/jwx (crypto)	2021	support for jwks json marshalling	Issue
ory/hydra (OIDC)	2021	add support for RFC 8414 pkce info to the OpenID Connect discovery endpoint	PR
sensu/sensu-go (Monitoring)	2019	strengthen TLS security posture	PR
sensu/sensu-go (Monitoring)	2019	separate TLS config for dashboard	PR
sensu/sensu-go (Monitoring)	2019	set sensuctl http client timeout	PR
gliderlabs/logspout (Logging)	2018	adding TLS client auth and more control of CA trust store	PR
kubernetes/kubespray (Kubernetes)	2017	allow admission control plug-ins to be configured	PR
hyperledger/fabric (Blockchain)	2017	(HSM) Fix issue with pubkey pkcs11 CKA_PRIVATE	Commit
hyperledger/fabric (Blockchain)	2017	allow compiling functions implemented in assembly	Commit
hyperledger/fabric (Blockchain)	2017	configurable pulling of chain code base images	Commit
upfluence/sensu-client-go (Monitoring)	2017	bug fix: add default subscription or sensu server will crash	PR

Projects

Mon, 01 Jan 0001 00:00:00 +0000

A list of various open-source projects I created over the years, and the story behind them. If you would like to see contributions I made to other open-source projects follow this link.

gbolo/aws-power-toggle

A web UI and API for quickly starting and stopping AWS environments. I wrote this tool back in 2018 to help members of other teams easily manage the state of the environment(s) that I would build out for them in aws via our IaC (terraform + ansible). Whether it was a sales team doing a pitch on various demo environments or a dev team working on various dev environments, this tool could accommodate it. The main goal was to reduce the aws bill ofcourse 🤑. While not as cheap as running terraform destroy, shutting down the compute of an environment could save a ton of money. It also is much quicker to start/stop an environment then running through the infrastructure code to redeploy. I had a great time building this out, and was a great learning experience for using the aws api.

Source Code	Year	Language(s)	Demo Link
github.com/gbolo/aws-power-toggle	2018	`go`, `vue.js`	https://awspt-demo.linuxctl.com

gbolo/vSummary

vSummary is an open source tool for collecting and displaying a summary of your vSphere Environment(s). Back in 2012, I was the sole member of the virtualization team. This meant that I was on the hook for the virtualization design, deployment, and support 😉. Over the next few years, I had grown the company from zero VMs up to about 4k. This was spread across 4 datacenters, 3 countries, 2 continents. At the time (vSphere 5) VMware did not have any tools for a single pane of glass view. So I originally wrote this tool back in 2015 in php! It actually had a few components back then so it wasn’t entirely php (the original poller was written in PowerShell). A year later in 2016 I open-sourced it on github. Then a couple years later in 2018 I redesigned and rewrote the whole thing as a perfect excuse to learn go 😎. Best decision ever, as go quickly became my favorite language and put me in an incredible position to be able to contribute to many other popular open-source projects that are also written in go.

Source Code	Year	Language(s)	Demo Link
github.com/gbolo/vsummary	2016	`go`, `javascript`	https://vsummary.linuxctl.com

gbolo/muggle-k8s

A deployment for Kubernetes without the use of any “magic” 🧙‍♂️ (this means no kubeadm usage BTW). This repository is mainly for educational purposes (although I have occasionally used it for the purpose of testing a candidate’s knowledge of kubernetes during interviews). The goal being to fully understand how to deploy each component of the kubernetes infrastructure without the use of any magical tools like kubeadm. From a security perspective, I also want to explore the use of my own PKI as well as to harden the system and tightly control network and ACLs.

Source Code	Year	Language(s)
github.com/gbolo/muggle-k8s	2020	`ansible`

gbolo/protego

Protego is a self-hosted REST API service, intended to be used in conjunction with nginx’s auth_request module, with the goal of providing transparent IP based ACLs to your existing HTTP service(s). protego_authorize_flow.png

Source Code Year Language(s)

github.com/gbolo/protego 2020 go

Source Code	Year	Language(s)
github.com/gbolo/protego	2020	`go`

gbolo/muggle-oidc

I wanted to learn the ins-and-outs of OIDC, so I thought it would be a great idea to write my own OIDC client application and integrate it with various providers. I skimmed through a good portion of RFC 6749 and RFC 7636 as a reference when implemeting much of the functionality. I was really interested in the private_key_jwt client authentication type since it offers much strong authentication than client_secret so I also spent a bunch of time on that too. It was really nice to see my client work well after I was done with it! I should also disclaimer that this client is for educational purposes only and you should probably not attempt to use it for anything else 😅.

Source Code	Year	Language(s)
github.com/gbolo/muggle-oidc	2021	`go`

Dockerfiles

I was a fairly early adopter of Docker. I kept a ton of Dockerfiles and docker-compose in this repo over the years as I worked on various things. I don’t really maintain much of the stuff I no longer use anymore. It seemed to get some stars over the years, so I guess it was useful to some.

Source Code	Year	Purpose
github.com/gbolo/dockerfiles	2016	`docker`

Ansible Roles

I wrote a ton of ansible roles over the years. Most of them are very specific to my employer’s needs and are stored privately. However, there are a few generic ones I decided to share (although I don’t really maintain them anymore)

Source Code	Year	Purpose
github.com/gbolo/ansible-role-docker	2016	sets up docker just the way I like it ;)
github.com/gbolo/ansible-role-nginx	2017	sets up a hardened nginx on RHEL
github.com/gbolo/ansible-role-desktop	2017	helps configure and maintain my desktops/laptops
github.com/gbolo/ansible-nfs	2016	sets up NFS servers and clients on RHEL
github.com/gbolo/ansible-role-mongodb	2017	sets up mongodb on RHEL
github.com/gbolo/ansible-ipfs	2016	sets up multi-tenant ipfs
github.com/gbolo/ansible-dev	2016	bunch of roles to support container orchestration and fabric

Search

Mon, 01 Jan 0001 00:00:00 +0000

linuxctl.com

Developing a Nomad Job Ansible Module

Installing the module

Using the module

--check mode

--diff mode

Preserving the Original HCL

Why not use the community module?

x509 Certificate Manual Signature Verification

What Does “Signing a Certificate” Mean?

The Verification Process

Obtaining the Issuer’s Public Key

Extracting the Signature

Decrypting the Signature (RSA)

Hashing the Original Signature

The Finished Script

Syncthing - Why You Should be Using it

A Brief Log of the Journey

Start of the Journey - DropBox

My two years with OwnCloud

OwnCloud was slow

Sync conflicts

My two years with SeaFile

Syncthing - The Light at the end of the Tunnel

Protected with TLS Mutual Authentication

Distributed Peer to Peer Model

Written in Go

Fully Open Source

Focus Only on Syncing Files

Cross Platform

TL;DR

Ansible - Interacting with external REST API

Ansible modules

RabbitMQ REST API

Creating Ansible Playbook for REST API Integration

The Finished Playbook

Hardening Nginx

Default Configuration

Disable Server Token

Disable Unknown Virtual Hosts

TLS Hardening

Disable non-SNI requests

TLS Allowed Protocols and Cipher Suites

Diffie-Hellman Parameter for EDH Cipher Suites

Enable HTTP Strict Transport Security (HSTS)

Migrating old Ubuntu XEN VMs to VMware

Migrating a XEN Ubuntu 6.06.2 LTS VM over to VMware Manually

Step 0

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

Migrating a XEN Ubuntu 12.04.4 LTS VM over to VMware Manually

Step 0

Step 1-4

Step 5

Step 6

Grafana & InfluxDB - Getting Started

Building Tiny Secure Docker Containers

Are Size and Security Related?

Image Size compared to CVEs

How Small can a Container Get?

Building a container image from SCRATCH

TL;DR

OpenSSL - Reference

Generating Key Material

Elliptic Curve (EC)

Rivest-Shamir-Adleman (RSA)

x509 Certificates

Creating and Viewing x509 Certificates

Certificate Signing Request (CSR)

Verification of x509 Certificates

Symmetric Encryption

Performance Testing

Let's Encrypt Muli-Domain Across Unique IPs

Problems with the http-01 Challenge

The Solution

The Solution Guide

`--check` mode

`--diff` mode

Problems with the `http-01` Challenge