Developing a Nomad Job Ansible Module

Fri, 21 Apr 2023 00:00:00 +0000

Installing the module

The module source code can be found at github.com/gbolo/hashicorp-ansible. In the near future I may add it to my ansible galaxy account for easier installation along with adding proper docs. Until then you will need to install it manually by simply copying a couple folders from the git repo linked above.

# retrieve the plugin directory from the git repo
❯ git clone https://github.com/gbolo/hashicorp-ansible /tmp/hashicorp-ansible
❯ mv /tmp/hashicorp-ansible/plugins .

# confirm the structure looks something like this
❯ tree plugins
plugins
├── modules
│   ├── consul_acl_bootstrap.py
│   ├── consul_acl_get_token.py
│   ├── consul_acl_policy.py
│   ├── consul_acl_token.py
│   ├── consul_connect_intention.py
│   ├── consul_get_service_detail.py
│   ├── nomad_acl_bootstrap.py
│   ├── nomad_acl_policy.py
│   ├── nomad_acl_token.py
│   ├── nomad_csi_volume.py
│   ├── nomad_job_parse.py
│   ├── nomad_job.py
│   ├── nomad_namespace.py
│   └── nomad_scheduler.py
└── module_utils
    ├── consul.py
    ├── debug.py
    ├── nomad.py
    └── utils.py

# ensure that your ansible.cfg file has these set
❯ cat ansible.cfg
...
library      = ./plugins/modules
module_utils = ./plugins/module_utils

Using the module

Now that it’s installed, let’s use this module to create a simple nomad job task in your playbook:

- name: ensure nomad job
  run_once: true
  nomad_job:
    url: "{{ nomad_server_url }}"
    management_token: "{{ nomad_acl_management_token }}"
    state: present
    namespace: default
    hcl_spec: |
      job "podinfo" {
        type = "service"
        meta {
          owner = "gbolo"
        }
        namespace = "default"
        group "app" {
          network {
            port "http" {
              to = 9898
            }
          }
          service {
            name = "podinfo"
            port = "http"
          }
          task "app" {
            driver = "docker"
            config {
              image = "stefanprodan/podinfo"
              ports = ["http"]
              args = [
                "./podinfo",
                "--port=9898",
                "--level=debug",
              ]
            }
          }
        }
      }

`--check` mode

specifying --check in an ansible playbook run allows you to determine whether or not a change will occur, without that change actually being executed. It’s like a “Dry Run”. In order to accomplish this we must ensure that our ansible module can properly detect changes. This is perfect opportunity to use the nomad API job plan endpoint, since this endpoint invokes a dry-run of the scheduler for the job.

In the actual module_utils/nomad.py module we have the following code snippets:

# API endpoint for job plan
URL_JOB_PLAN = "{url}/v1/job/{id}/plan?namespace={namespace}"
...

# helper class for use in module
class NomadAPI(object):
...
  # method returns the result of job plan
  def plan_job(self, id, body):
      return self.api_request(
          url=URL_JOB_PLAN.format(url=self.url, id=id, namespace=quote_plus(self.namespace)),
          method="POST",
          body=body,
          json_response=True,
      )

In the actual nomad_job module we have the following code snippets:

# use nomad plan to help decide if we should submit a job
if module.params.get("state") == "present":
    plan = nomad.plan_job(
        job_id,
        json.dumps(
            dict(
                Job=parsed_job,
                Diff=True,
            )
        ),
    )

...

# determine if the job spec changed by inspecting the diff type.
if plan["Diff"].get("Type") != "None":
    result["changed"] = True

`--diff` mode

specifying --diff in an ansible playbook run is easily my favorite feature. This flag should output the actual change in a human readable diff format. Not all ansible modules support such a flag, since the author of the module must explicitly write this functionality. The Nomad API endpoint for job plan returns a diff report that is not very human readable. For example, check out how the response looks like when we make two simple changes; adding a new meta field and changing the docker image tag of a running nomad job:

POST /v1/job/podinfo/plan

{
...
    "Diff": {
        "Fields": [
            {
                "Annotations": null,
                "Name": "Meta[project]",
                "New": "test",
                "Old": "",
                "Type": "Added"
            }
        ],
        "ID": "podinfo",
        "Objects": null,
        "TaskGroups": [
            {
                "Fields": null,
                "Name": "app",
                "Objects": null,
                "Tasks": [
                    {
                        "Annotations": [
                            "forces create/destroy update"
                        ],
                        "Fields": null,
                        "Name": "app",
                        "Objects": [
                            {
                                "Fields": [
                                    {
                                        "Annotations": null,
                                        "Name": "args[0]",
                                        "New": "./podinfo",
                                        "Old": "./podinfo",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "args[1]",
                                        "New": "--port=9898",
                                        "Old": "--port=9898",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "args[2]",
                                        "New": "--level=debug",
                                        "Old": "--level=debug",
                                        "Type": "None"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "image",
                                        "New": "stefanprodan/podinfo:6.3.4",
                                        "Old": "stefanprodan/podinfo",
                                        "Type": "Edited"
                                    },
                                    {
                                        "Annotations": null,
                                        "Name": "ports[0]",
                                        "New": "http",
                                        "Old": "http",
                                        "Type": "None"
                                    }
                                ],
                                "Name": "Config",
                                "Objects": null,
                                "Type": "Edited"
                            }
                        ],
                        "Type": "Edited"
                    }
                ],
                "Type": "Edited",
                "Updates": {
                    "create/destroy update": 1
                }
            }
        ],
        "Type": "Edited"
    },
...
}

This output is quite large for such a simple change and is likely intended to be parsed before returning it to human eyes. The nomad binary does this quite nicely (its even colored in the shell!):

❯ nomad job plan artifacts/simple.hcl
+/- Job: "podinfo"
+   Meta[project]: "test"
+/- Task Group: "app" (1 create/destroy update)
  +/- Task: "app" (forces create/destroy update)
    +/- Config {
          args[0]:  "./podinfo"
          args[1]:  "--port=9898"
          args[2]:  "--level=debug"
      +/- image:    "stefanprodan/podinfo" => "stefanprodan/podinfo:6.3.4"
          ports[0]: "http"
        }

To accomplish a similar result in my ansible module I found this excellent python library github.com/strigo/nomad-diff. I made this an optional dependency in the ansible module so that it does not fail if the system does not have it installed:

# import nomad_diff if it is available on the system
_nomad_diff_available = False
import importlib.util

nomad_diff_spec = importlib.util.find_spec("nomad_diff")
if nomad_diff_spec is not None:
    import nomad_diff
    _nomad_diff_available = True

...

# do a nice diff if the system has nomad_diff available
if _nomad_diff_available and plan.get("Diff") is not None:
    try:
        result["diff"] = dict(prepared=nomad_diff.format(plan["Diff"], colors=True, verbose=False))
    except:
        # if we can't get a diff, it's not a big deal...
        pass

Now if we do an ansible run and this task gets called while --diff flag is specified we get this nice feedback (it is also colored when ran in the shell):

❯ ansible-playbook --diff -i inventory/gbolo.digitalocean.yml playbook.yml
...

TASK [ensure nomad job] *****************************************************************************
+/- Job: "podinfo"
+   Meta[project]: "test"
+/- Task Group: "app" (1 create/destroy update)
  +/- Task: "app" (forces create/destroy update)
    +/- Config {
          args[0]:  "./podinfo"
          args[1]:  "--port=9898"
          args[2]:  "--level=debug"
      +/- image:    "stefanprodan/podinfo" => "stefanprodan/podinfo:6.3.4"
          ports[0]: "http"
        }
changed: [gbolo-alpha-mrz01]

Preserving the Original HCL

Nomad 1.6.0 has introduced support for storing the original job HCL spec in the (following PR)[https://github.com/hashicorp/nomad/pull/16763]. I originally created this module before this came out, but adding support for this was really simple as I just had to modify the API call:

result["submit_response"] = nomad.create_or_update_job(
    job_id,
    json.dumps(
        dict(
            Job=parsed_job,
            # Submission structure is new in Nomad 1.6.0
            # it preserves the original HCL spec.
            Submission=dict(
                Format="hcl2",
                Source=module.params.get("hcl_spec"),
            ),
        )
    ),
)

Now when we use the ansible module, the original HCL spec will be preserved in the UI just like the nomad binary does.

Why not use the community module?

Fair question, why didn’t I choose to use the community.general.nomad_job module?

I did try it out, and it works fine enough, however (at the time of this writing) it fell short in a few areas in my opinion:

It requires the pip package python-nomad. Which I feel is a bit overkill for the simple Nomad API.
It lacks a --diff mode, which I feel is essential when doing dry-runs.
It does not preserve the original HCL job spec, which is a new feature in Nomad (introduced in 1.6.0). In order for this module to even support that, the underlying library python-nomad would need to first get that support. See why I think it’s not always a great idea to use a library when dealing with a simple REST API.

The above reasons were along with the fact that I was creating other ansible modules that interact with other areas of the Nomad API (nomad_acl_token, nomad_acl_policy) that did not have any community based modules was more than enough motivation for me to create this nomad_job module.

python on linuxctl.com