In a couple of templates I worked on I came across the need to create a variable list of numbered strings that is a product of a different list.

Dealing with this kind of thing is easy when you’re dealing with resources (that support count). It’s less straight-forward when you need to create a local variable, say, to use for a resource’s value, or an output, that don’t.

Let’s have a look at a dirty hack that can help.

Task 1

I needed to generate an output that is a list of URLs with an incrementing port number, e.g.,:

http://server.com:9000
http://server.com:9001
...etc

Trick was, I needed a different number of these based on the length() of a list variable. After poking around the Interwebz, I’ve managed to pull together a solution that worked for me, like so:

variable "var1" {
  default = [
    "something",
    "something else",
    "something yet else"
  ]
}

variable "fqdn" {
  default = "server.com"
}

resource "null_resource" "port_list" {
  count = "${length(var.var1)}"

  triggers {
    port = "${9990 + count.index}"
  }
}

output "My_Cool_Servers" {
  value = "${formatlist("http://${var.fqdn}:%s",
    null_resource.port_list.*.triggers.port)}"
}

How this works:

• The null_resource of course supports count, which we can use in setting the triggers.port attribute. We can then read it as null_resource.port_list.[x].triggers.port, or access as a list just as we do above – null_resource.port_list.*.triggers.port.
• We do the later, and use the result as the input to the formatlist function, that we can stick directly into an output’s value.

Task 2

This one involved adding something to a template_file. You of course have already discovered that resource (or datasource) parameters don’t support count. Thus the way to insert something akin to the above into a template_file is to prepare a local variable with the complete value, and then do the substitution.

Say my template file is a BIND DNS zone, and I need to insert some NS records; except I don’t know ahead of time how many of those I’ll have!

Let’s see what we can do:

variable "nameservers" {
  type = "list"
  # default for illustration purposes:
  default = ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
}

variable "dns_domain" {
  default = "blah.com"
}

variable "dns_subdomain" {
  default = "test"
}

variable "zone_serial" {
  default = "2018050101"
}

# Let's set up our null_resource with the bits we need
resource "null_resource" "ns" {
  # This creates a number of strings we need for NS records, based on the
  # list of nameserver IPs
  count = "${length(var.nameservers)}"

  triggers {
    # This creates "nsN" string, N based on the count.index
    ns_host = "ns${format("%d", count.index + 1)}"

    # This creates "nsN.test.blah.com" string, N based on the count.index
    ns_fqdn = "ns${format("%d", count.index + 1)}.${var.dns_subdomain}.${var.dns_domain}"
  }
}

# Now, we need to prepare the list of NS and A records for our nameservers
#
locals {
  # Create a list of "  IN NS nsN.test.blah.com." strings
  in_ns_list = "${formatlist("    IN NS %s.",
    null_resource.ns.*.triggers.ns_fqdn)}"

  # Pull the list into a single string, joined by the new line character
  in_ns_string = "${join("\n", local.in_ns_list)}"

  # Create a list of "nsN  IN A x.x.x.x"
  ns_in_a_list = "${formatlist("%s  IN A %s",
    null_resource.ns.*.triggers.ns_host,
    var.nameservers)}"

  # Pull the list into a string joined by new lines
  ns_in_a_string = "${join("\n", local.ns_in_a_list)}"
}

# Finally, we're ready to render our template
#
data "template_file" "glb_zone" {
  template = <<EOF
; $TTL used for all RRs without explicit TTL value

$TTL    30

$ORIGIN $${dns_subdomain}.$${dns_domain}.

@  1D  IN  SOA ns1.$${dns_subdomain}.$${dns_domain}. hostmaster.$${dns_subdomain}.$${dns_domain}. (
                $${zone_serial} ; serial
                3H ; refresh
                15 ; retry
                1w ; expire
                3h ; nxdomain ttl
                )

$${in_ns}

$${ns_in_a}

EOF

  vars {
    dns_domain    = "${var.dns_domain}"
    dns_subdomain = "${var.dns_subdomain}"
    zone_serial   = "${var.zone_serial}"
    in_ns         = "${local.in_ns_string}"
    ns_in_a       = "${local.ns_in_a_string}"
  }
}

output "my_zone" {
  value = "${data.template_file.glb_zone.rendered}"
}

What we do here is:

• Create a number of copies of null_resource, one per entry in var.nameservers, and format them appropriately. The ns_fqdn one isn’t strictly necessary, but it is useful to make template more DRY (I used it in a couple more places in addition to what’s shown above).
• We then formatlist() both with the subdomain/domain/IP(s), so we have a list of IN NS and IN A entries we need.
• Then we convert each list into a single string (which is what var parameters of our template_file expect), where the original list items are joined by a newline \n character.
• Last, we use the resulting stings “blah-blah\nblah-blah” as values to substitute template vars.

Closing comments

I don’t pretend to be the ultimate authority on Terraform – what you see served my needs, and I’ll just leave it at that.

If you know of a better way to do this – please kindly leave a comment, and thank you!

Pulse Virtual Traffic Manager (vTM) v18.1 introduced new Service Discovery mechanism to help with situations where vTM sits in front of an application made up from dynamic components.

Many organisations are now either use or experimenting with Kubernetes (K8s). Let’s see if we can get this Service Discovery mechanism to help us expose an application running on K8s to the Internet:

• In a way that works both in public clouds and on-prem;
• Supports session persistence and per-pod health checks;
• Can scale really well; and
• Is highly available*.

*The above assumes that you operate a production-grade HA cluster of vTMs somewhere where they can reach your K8s nodes running your application pods.

Our test bed set-up

First, we’ll need a K8s cluster to deploy our sample app to, that we can then discover. For the purpose of this blog, I’ll be using Quick Start for Kubernetes by Heptio on AWS. I simply deploy a new copy of the Quick Start into a new VPC, wait until it’s all up, and then retrieve my kubeconfig by using command shown in the CloudFormation Output:

Once we have it, quickly test that we can talk to the cluster, and let’s move on (we’ll assume that you’ve saved the kubeconfig file into your current working directory):

export KUBECONFIG=$(pwd)/kubeconfig

kubectl get nodes

# [ .. output skipped ..]

Next, we’ll need to procure a Developer Edition of vTM 18.1 to play with. If you have a host with Docker installed on it, the easiest way to get around it is to use the unofficial copy I’ve put up on Docker Hub – dk114/pulse-vtm:18.1. On your Docker host, you could run something on the lines of:

docker run --name vTM --rm -e ZEUS_DEVMODE=yes -e ZEUS_EULA=accept -e ZEUS_PASS=abc123 -p 9090:9090 -p 9070:9070 -p 80:80 -p 443:443 --privileged -t -d dk114/pulse-vtm:18.1

This should pull and deploy a vTM 18.1 in Developer Mode for you, and make it accessible through its UI interface (on port 9090) and REST API (port 9070). It will also expose ports 80 and 443, which at the beginning won’t be listening.

Alternatively, you can deploy an HA cluster of vTMs into AWS using my CloudFormation Template, making sure to set vTM version parameter to 181 when deploying. You will need an existing VPC with at least one public subnet that you will need to pass to the template when deploying. You can use the same VPC that Heptio’s Quick Start creates. To allow vTMs to talk to pods running on K8s nodes, add the id of the Security Group called k8s-cluster-security-group created by Heptio’s Quick Start to the vTM template’s parameter EnvSGs, e.g., sg-b47d5fcd.

For the rest of this blog, we’ll assume you’re using vTM in a Docker container.

Intro to vTM’s Service Discovery

On your Docker host, you should be able to access your vTM container through your host’s IP and port 9090; in my case I’m running it on my laptop, and can access it through https://127.0.0.1:9090. The page will throw a warning that it’s using a self-signed cert, which is fine in this case. Proceed to get to the login page, and log in with admin / abc123 (password we’ve set through ZEUS_PASS variable in our docker run command above).

Once logged in, hit Catalogs -> Service Discovery:

This will bring you to the page where we can add our own Service Discovery plugin.

The documentation for Service Discovery is included in the vTM User Guide, page number 79 (pdf page 99).

In a nutshell, this feature allows you to upload any arbitrary program to your vTM as a Custom User Plugin. You can then create a pool, and choose your plugin as the means to populate that pool with IPs and ports of this pool’s nodes.

vTM will then run your plugin program every X seconds (the number you can specify; 10 by default), and read / parse the output that plugin returns to find IPs, port numbers, and state for the nodes in your pool. It will then compare what the plugin just returned with the current state of the pool – IPs, ports, node state, etc., and figure out if it needs to make any changes. If not – nothing further is done; if yes – vTM will update pool’s config accordingly.

This may all sound harder than it is, so let’s get our hands dirty and create a very simple (if useless) plugin to see how it works.

Create a file called test-plugin.sh with the following contents:

#!/bin/bash

cat <<EOF
{
    "version":1,
    "nodes":[
        { "ip":"192.0.2.0", "port":80 },
        { "ip":"192.0.2.1", "port":81, "draining":true }
    ],
    "code":200
}
EOF

If this script is run, it will output a JSON file that’s been copied from the vTM User Guide, page number 83 (pdf page 103).

Let’s see what happens if we add it to our vTM. On the Service Discovery page click Custom User Plugins, then Upload Plugin -> Browse -> select your test-plugin.shfile, and click Upload.

Once uploaded, you’ll be able to Test plugin, by running it once with optional Arguments. Since our plugin doesn’t need any, simply click Test to see what vTM thinks this plugin is returning:

In our case, as expected, it is seeing two nodes, one of which is draining.

Ok, let’s create a pool that uses this plugin:

Hit Create Pool, and on the next page you’ll have a chance to adjust settings of the Service Discovery, including arguments, scheduling interval (how often it is run), and run timeout.

Ok, let’s see what we now have in our pool! Click Services -> Pools -> Test to see what’s going on:

As you can see, you now have nodes in your pool, that came from our (admittedly useless) plugin.

Let’s reflect and take this in a bit.

What you essentially have is an ability to run any code directly on vTM to do any sort of thing to discover what the nodes for your pool should be. It could be as simple as running a curl to fetch a file that your application’s provisioning system populates with IPs / Ports of your app components, to something that talks to AWS API and K8s API to dynamically find components of your application that is distributed across on-premises and public cloud.

Why not event-driven?

Next question that pops into your mind is probably “why couldn’t vTM listen for external events, instead of polling? This is a valid question, but when you consider the complexity involved in making this work, universally, vs. the benefit you get, in most cases simple polling would suffice just fine.

Additionally, the current implementation allows your plugins to tell vTM to adjust its polling interval (up or down) by using the optional interval_override parameter.

Ok, so when can we get to K8s?

How about “now”?

First, clone or download the following repo: https://github.com/dkalintsev/vTM-K8s-ServiceDiscovery, and change your working directory to the one you’ve downloaded the repo into.

Disclaimer Note: I wrote this plugin as a “proof of concept”, and it should not be used in production “as-is”. Please treat this code as “unsupported” and “experimental”. The idea is to provide a working prototype that can be used as an inspiration for your own implementation.

You should have something like this; we’ll get to what’s what in here in a moment:

.
|____deploy-template
| |____files
| | |____K8s-get-nodeport-ips.sh
| | |____my-kubeconf
| |____main.tf
| |____terraform.tfvars
| |____variables.tf
|____Manifests
| |____K8s-get-nodeport-ips.yaml
| |____my-nginx.yaml
|____README.md
|____test-plugin.sh

The game plan for this exercise is as follows:

• Our Service Discovery plugin will be a script that calls kubectl to find the IPs and Port number for the Pods we’d like to send traffic to
• The script then formats this information in the shape vTM expects it and prints it out
• vTM reads the output, and updates a pool, as/if necessary.

Let’s break it down.

The plugin itself

In the repo you’ve cloned above, there’s a file deploy-template/files/K8s-get-nodeport-ips.sh, which will do the work for us. It may look a bit complex, but once you strip down all data massaging, prerequisites, and error handling logic – it’s very simple:

• Take a name of a K8s Service that you pass it as a parameter;
• Confirm that the type of this Service is NodePort;
• Find the ephemeral port K8s allocated to that NodePort – this is the port we’re looking for;
• Find the pods backing our Service by reading Service’s Endpoint list;
• Find the IPs of the K8s hosts where each of the pods runs – these are the IPs we’re after;
• Format the IPs + port into a JSON string vTM expects, and print it out.

What does this plugin need?

Since our plugin is a simple Shell script, the work of talking to K8s API and processing JSON falls on to “helpers”. In our case we use kubectl and jq, both of which do not ship with vTM.

vTM has a concept of Extra Files in its Catalogs. All files uploaded into that section of the Catalogs end up in the directory $ZEUSHOME/zxtm/conf/extra/, which is automatically replicated between vTMs in a cluster.

Our plugin uses this to its advantage. When it is run, it will check if kubectl and jq are already available, and, if not, it will, if told to through a -g parameter, attempt downloading them from their respective origins and place under the extras directory mentioned above.

What about authentication when talking to K8s?

The easiest but the least secure way to let the plugin talk to your K8s cluster is to supply it with a copy of kubeconfig file we’ve downloaded at the start of this blog.

Let’s however pretend that we care about security, and do it a bit better. Instead of using the K8s admin user, let’s create a new one with a restricted set of permissions that only allow R/O access to what it needs.

In the repo you’ve downloaded, have a look at the file Manifests/K8s-get-nodeport-ips.yaml. When created, it makes a ServiceAccount called vtm-user and similarly named ClusterRole with restricted permissions, and then binds them together. As the result, user vtm-user has the restricted permissions that we’re after.

Next up, we need to create a kubeconfig file that uses our vtm-user instead of the default admin.

In the repo, there’s a file deploy-template/files/my-kubeconf that you can use as a guide:

1) Open your real kubeconfig file and copy the top part of it:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [Cert contents skipped]
    server: https://k8s-k8sst-apiloadb-<blah>-<blah>.ap-southeast-2.elb.amazonaws.com
  name: kubernetes

2) Edit the deploy-template/files/my-kubeconf and replace the respective part with the one you’ve copied above, leaving everything starting with the line contexts: and below intact.

3) Create the ServiceAccount, ClusterRole, and their binding for our vtm-user:

kubectl create -f Manifests/K8s-get-nodeport-ips.yaml

4) Retrieve the token for the vtm-user:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep vtm-user | awk '{print $1}')

You should see something similar to:

Name:         vtm-user-token-6rzkh
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=vtm-user
              kubernetes.io/service-account.uid=d11872bd-4f32-11e8-84a3-060a49790776

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciO[ ...blah-blah... ]4So2k8g <<== WE'RE AFTER THIS
ca.crt:     1025 bytes
namespace:  11 bytes

5) Copy the long string after the token: and replace the simlar-looking long string inside your deploy-template/files/my-kubeconf.

6) Test if you got it all right:

kubectl --kubeconfig=deploy-template/files/my-kubeconf get nodes

# You should see a list of nodes in you cluster

kubectl --kubeconfig=deploy-template/files/my-kubeconf get secrets

# You should get an error message, since vtm-user isn't authorised to do that

Ok, we’re all good; so let’s get ourselves something to discover:

kubectl create -f Manifests/my-nginx.yaml

This should spin up one instance of an Nginx pod that serves a document containing its own internal hostname on port 80, that is then exposed via a Service called my-nginx-service on a NodePort.

Run kubectl get pods to make sure our Nginx pod is running, and let’s get our vTM to see it!

Adding plugin to vTM

Following the same process as we did for test-plugin.sh above, add deploy-template/files/K8s-get-nodeport-ips.sh as a Custom User Plugin.

Next, we’ll need to add our my-kubeconf file to Catalogs -> Extra Files -> Miscellaneous:

At this point, we should have all moving bits in place, so we can test it out! Fill the following into the Arguments for our Custom User Plugin K8s-get-nodeport-ips.sh, and hit “Test”:

-s my-nginx-service -p http -c my-kubeconf -g

Refer to the comments inside the plugin code for details, but what we’re telling it is:

• Look for the K8s Service called my-nginx-service
• Get a NodePort value that corresponds to that Service’s port called “http”
• Use the file from Extras directory called my-kubeconf to communicate with the K8s cluster
• get the prerequisite helpers (kubectl and jq), if they’re not already in Extras

After clicking “Test” for the very first time, it will take a good few moments, depending on your Internet speed, as it will be downloading a copy of kubectl and jq (and curl, if you’re running a Docker version of vTM). Then, if all went well, we’ll see something like this:

It worked!

It’s now time to create a pool and see what happens then.

Hit “Create Pool”, and then go back to the list of pools to see if the nodes got populated:

Looks good! Now, what happens if we scale up our Nginx deployment to two pods? Let’s find out:

kubectl scale deployment my-nginx --replicas=2

Wait 10 seconds (default poll timer), refresh – and here we go, two nodes:

Summary

What we’ve just done is added a new, completely bespoke Service Discovery plugin to our vTM 18.1, and got it to discover our “application” running on K8s, exposed through a NodePort Service.

If our vTM was able to reach the IP addresses of K8s nodes, we could have started directing traffic to them; however in our sandbox where vTM was running on my laptop while K8s cluster was sitting in AWS, this would not work.

In a more realistic scenario, you would have a vTM cluster configured for High Availability with Traffic IPs sitting next to your K8s cluster(s). This set-up would solve the following problems:

1) Scalability: you would be able to direct as much traffic to your application as your vTM cluster can handle, which is typically a lot. vTMs in a cluster can operate as “all-active”, so capacity a cluster can handle scales directly with the number of nodes in the cluster.

2) High availability: vTM cluster running outside of K8s can use its built-in high availability features built around Traffic IPs, without any limitations.

3) The need to dance around with K8s Cluster IPs is eliminated; especially in AWS where VPC doesn’t support more specific routes. Traffic to your K8s Service is sent directly to the IPs of the actual nodes where your pods are running. Yes, since we’re using NodePort, this traffic will then be handled by kube-proxy, but since we have our Service configured with externalTrafficPolicy: Local*, it will only forward traffic to the pods residing on the node. Traffic from vTMs will never be sent to a node that’s just gone down, because we’ll discover the fact that the pods on the failed node went down, and the IP of that node will be gone from the vTM’s pool.

*Note: at the time of writing (8 May 2018) Heptio K8s appears to have a bug that prevents use of Services with externalTrafficPolicy: Local – traffic gets silently dropped. See this issue for any updates. This means that it won’t be possible to guarantee session persistence and per-pod health checks from vTM. One possible work-around would be to use hostPort on your application pods; however this will require changes to the Service Discovery plugin code.

You probably noticed that my-nginx.yaml includes node anti-affinity configuration in the Deployment’s spec. This is needed to ensure that each K8s node where vTM sends traffic has only one replica of your application’s pod, so that session persistence and per-pod health checks can work.

You should take this into consideration and decide whether these benefits (persistence + health checks) outweigh the “cost” associated with this: you won’t be able to run more than one replica of your app pod on the same K8s node. One possible work-around is to have more nodes than the maximum number of replicas your app requires. If your K8s nodes are VMs, this should be relatively easy to do.

Um, what’s with those Terraform-looking files in the repo?

The repo also contains a Terraform template that can deploy a pool with Service Discovery using freshly-released Terraform Provider for vTM. Please feel free to play around with it. See the my other blog post on intro to Terraform with vTM to get yourself started.

To give it a quick go, you can do the following:

• Update the deploy-template/terraform.tfvars file and make sure that vtm_rest_ip is set to the IP of your vTM, and vtm_password is set accordingly.
• Change your working directory to deploy-template, and run the following command:

docker run --rm -v $PWD:/root/k8s-sd -w /root/k8s-sd dk114/try-vtmtf:1.1 /bin/ash -c "terraform init && terraform apply -auto-approve"

This will run the dk114/try-vtmtf:1.1 Docker container that is an Alpine linux with a copy of terraform + Terraform Provider for vTM, that will apply the Terraform template from your current directory. After it successfully applies, your vTM should have a new uniquely-named copy of our plugin, similarly uniquely-named copy of the kubeconfig file, and a new pool configured to use this plugin.

See you next time!

Continuing from Part 1, Part 2, and Part 3, in this final instalment we’ll finish our configuration by adding things like SSL offload and L7 routing.

vTM has an embedded language called TrafficScript, that can be used to run business logic over Requests (what vTM sees from the client) and/or Responses (what vTM sees from the pool node). It has full access to everything in Requests and Responses (if vTM is configured to perform SSL Offload), which includes headers, cookies, request strings, body, etc.. Based on the logic you put into your TrafficScript (TS), vTM can do all kinds of things – check out my older post on integrating 3rd party services with your Single Page App web site, for example. For this post, we’ll use it to do L7 routing.

Terraform supports including external code in-line, which is perfect for small chunks of such code, but I like separating concerns. Create a directory files under our working directory try-vtmtf, and inside it create a file called vs1_request_rule.tpl with the following contents:

if(http.getPath() == "/api") {
    pool.use("${pool_name}");
}

This is a template for TS code with one variable – pool_name. Running terraform apply will inject a name of our API pool to produce the actual TrafficScript code. Our VS1 will then run it on all incoming requests, and if HTTP Path contains the string /api, vTM will send that request to the nodes in the pool with the name of our API pool that we’ve passed to it.

Let’s add this into our main.tf:

# Our Request Rule needs API pool name, so we handle this through a template
data "template_file" "vs1_request_rule" {
  template = "${file("${path.module}/files/vs1_request_rule.tpl")}"

  vars {
    pool_name = "${vtm_pool.api_pool.name}"
  }
}

resource "vtm_rule" "vs1_l7_routes" {
  name    = "${local.uniq_id}_VS1-L7-Routes"
  content = "${data.template_file.vs1_request_rule.rendered}"
}

This will create a template data source called vs1_request_rule of type template_file with one variable – pool_name that we set to the name of our API pool. When accessed, this data source will take the contents of the template file set in template parameter, inject the variable(s), and return the resulting file.

We can see how this is used in the vtm_rule resource, where content parameter consumes what this data source produces.

Since template_file requires yet another provider, we’ll need to run terraform init once more before proceeding, so that Terraform can download it. Once done, you can run terraform plan and see the rendered contents of the template in the proposed creation of the vtm_rule.

Note: In cases where your rules are small (such as above), you may prefer to include them directly into the template using heredoc syntax. It is certainly more compact, and comes to personal preference as to whether you want to mix Terraform templating code (HCL) with other code (TrafficScript) or not.

Here’s an example for how you could do the same as above (separate TrafficScript file and data "template_file" "vs1_request_rule") using this combined approach, where the TS code is included between the EOF markers. In this case we inject the pool name by referring directly to the appropriate Terraform resource – ${vtm_pool.api_pool.name}.

resource "vtm_rule" "vs1_l7_routes" {
  name = "${local.uniq_id}_VS1-L7-Routes"

  # Content is TrafficScript, with an embedded Terraform variable
  #
  content = <<EOF
if(http.getPath() == "/api") {
    pool.use("${vtm_pool.api_pool.name}");
}
EOF
}

To make this rule active we’ll need to add it to our VS as a parameter; but let’s take care of SSL offload prerequisites first.

Our template requires a certificate, so we are going to create a self-signed one. In our try-vtmtf container, run the following:

openssl req -newkey rsa:2048 -nodes \
    -keyout domain.key -out domain.csr \
    -subj "/C=US/ST=New York/L=Brooklyn/O=Example Brooklyn Company/CN=*.corp.com"

openssl x509 -signkey domain.key \
    -in domain.csr -req \
    -days 365 -out domain.crt

This should have produced three files – domain.crt, domain.csr, and domain.key. Paste their contents (starting / ending with -----BEGIN.. / -----END..) into the three new variables in our terraform.tfvars file between the respective EOF markers:

ssl_cert_pub = <<EOF
-----BEGIN CERTIFICATE-----
MIIDUjCCAjoCCQDGcM4C0OfWCDANBgkqhkiG9w0BAQsFADBrMQswCQYDVQQGEwJV
[ ... skipped ... ]
64+OgfhZ2bb3eiFsm2Azf55AWqbb5pr3tDwrBidg5R176Cl6l00=
-----END CERTIFICATE-----
EOF

ssl_cert_pri = <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCq0CEK/ua37JDM
[ ... skipped ... ]
htQ3q8eLTYwApsXNHdGwQyME
-----END PRIVATE KEY-----
EOF

ssl_cert_req = <<EOF
-----BEGIN CERTIFICATE REQUEST-----
MIICsDCCAZgCAQAwazELMAkGA1UEBhMCVVMxETAPBgNVBAgMCE5ldyBZb3JrMREw
[ ... skipped ... ]
K4nc/E5mq4JkmBEdiDg/yxUnUvI=
-----END CERTIFICATE REQUEST-----
EOF

Hopefully now it’s double-clear why we don’t want to check terraform.tfvars into code repository!

Let’s add the corresponding variables into our variables.tf:

variable "ssl_cert_pri" {
  description = "Private SSL Cert for the Virtual Server"
}

variable "ssl_cert_pub" {
  description = "Public SSL Cert for the Virtual Server"
}

variable "ssl_cert_req" {
  description = "Signing Request for the SSL Cert for the Virtual Server"
}

And the code for the SSL Server Key resource to the main.tf:

# SSL Server Certificates for the Virtual Server's SSL Offload.
#
resource "vtm_ssl_server_key" "ssl_cert" {
  name = "${local.uniq_id}-server-corp.com"
  note = "SSL Server Cert for corp.com"

  private = "${var.ssl_cert_pri}"
  public  = "${var.ssl_cert_pub}"
  request = "${var.ssl_cert_req}"
}

Finally, let’s update our Virtual Server so that it:

• Uses our TrafficScript Rule;
• Has SSL Offload turned on; and
• Listens on the port 443 instead of 80.

Replace the resource "vtm_virtual_server" "vs1" { .. } that you have in your main.tf with the following:

# The Virtual Server
#
resource "vtm_virtual_server" "vs1" {
  name          = "${local.uniq_id}_VS1"
  enabled       = "true"
  listen_on_any = "${local.should_listen_on_any}"

  # We need to use compact() to get rid of values "" which will be there
  # in case we didn't have any TIP Groups
  listen_on_traffic_ips = ["${compact(local.tigs_list)}"]

  # Default pool = "Main"
  pool                    = "${vtm_pool.main_pool.name}"
  port                    = "443"
  protocol                = "http"
  ssl_decrypt             = "true"
  ssl_server_cert_default = "${vtm_ssl_server_key.ssl_cert.name}"
  request_rules           = ["${vtm_rule.vs1_l7_routes.name}"]
}

The notable changes are:

• port: 80 -> 443
• Added ssl_decrypt, ssl_server_cert_default, and request_rules

Now, run terraform apply, and if all went well – you should have your configuration in its final state, all wrapped up and working!

If not – do not despair My copy of the template is available in my GitHub repo, where you can download it and compare with yours.

And once you’re done playing – run terraform destroy, to see the configuration you’ve applied to your vTM disappear as a whole, leaving the rest of the configuration (if any) intact.

Happy terraforming!

Carrying on from Part 1 and Part 2, in this instalment we’ll continue adding to our configuration.

If you’re back with us after a break – feel free to go over the part 1 and 2 again, and if you’re following along – make sure your set-up is all good, and the very last exercise from Part 2 completes correctly.

In the last exercise we’ve added one pool that can be populated with any number of nodes by supplying IP:Port for those nodes through a variable. Since the second pool isn’t any different, simply add the following to your main.tf, variables.tf, and terraform.tfvars, and re-run terraform apply to see if it works:

# main.tf
#
data "vtm_pool_nodes_table_table" "api_pool_nodes" {
  count    = "${length(var.api_nodes)}"
  node     = "${var.api_nodes[count.index]}"
  priority = "1"
  state    = "active"
  weight   = "1"
}

resource "vtm_pool" "api_pool" {
  name             = "${local.uniq_id}_API-Pool"
  monitors         = ["Ping"]
  nodes_table_json = "[${join(",", data.vtm_pool_nodes_table_table.api_pool_nodes.*.json)}]"
}

# variables.tf
#
variable "api_nodes" {
  description = "List of nodes for the 'API' pool"
  default     = []
}

# terraform.tfvars
api_nodes = ["192.168.1.100:80", "192.168.2.100:80"]

If all went well, your vTM will now have two pools with two nodes in each.

So, all these resources..

You probably been wondering about where you can find all the possible choices you have for vTM resources in your templates. Those familiar with vTM’s REST API have likely noticed that the resource names look suspiciously like vTM REST resources with vtm_ slapped in front, with resource parameters very much a 1:1 transplant.

And you will be right – the resources (and data sources) that vTM Provider exposes closely match those in the REST API. There are a few minor changes where internal Terraform variables clashed with parameter names in vTMs REST schema and have been slightly changed to aviod it. In addition, however, vTM Provider exposes a number of special data sources: *_list and *_table. You’ll meet them in this blog post very soon.

The complete set of resources, data sources, and useful examples can be found in the vTM Terraform Provider documentation (REST API v4.0 / vTM 17.2 and later) or (REST API v5.2 / vTM 18.1 and later).

variable "vtm_tig_eips" {
  description = "List of AWS Elastic IPs to be used for Traffic IP Group"
  default     = []
}

# This returns a list populated with "name" values of all traffic managers
# in the target cluster. We need this to create the Traffic IP Group.
#
data "vtm_traffic_manager_list" "cluster_machines" {
  # No parameters needed
}

locals {
  tig_name = "${local.uniq_id}_TrafficIPGroup"
}

# Traffic IP Group for our Virtual Server.
# By default, we create the "singlehosted" type.
#
resource "vtm_traffic_ip_group" "tip_group" {
  count       = "${signum(length(var.vtm_tig_eips))}"
  name        = "${local.tig_name}"
  mode        = "ec2vpcelastic"
  ipaddresses = "${var.vtm_tig_eips}"
  machines    = ["${data.vtm_traffic_manager_list.cluster_machines.object_list}"]
}

Second, we use count as a condition to create the resource. The length(var.vtm_tig_eips) will return a number of IPs supplied in the variable, or 0. We’re good for when it returns a 0, which will cause this resource to not be created; but we don’t want more than one Traffic IP Group in case our variable had multiple IPs in it! This is where Terraform’s built-in function signum() comes handy – it will keep 0 a 0, and convert any positive number into 1, which is what we want.

Since we have not included any IPs for our vtm_tig_ips, running terraform plan should say that no changes are required.

Last but not least, we defined a name for our TIP Group as a Local Variable. The need for this will become clear in a few moments.

Let’s test if our logic works. The configuration may not be valid, but at least we’ll know it was applied. Add the following to our terraform.tfvars, and run terraform apply:

vtm_tig_eips = ["2.2.2.2", "5.5.5.5"]

Your vTM’s log should display a Warning “VPC Elastic IP addresses can only be configured on EC2-VPC instances”, and Services -> Traffic IP Groups should have a broken TIP Group with our two IP addresses “2.2.2.2” and “5.5.5.5”.

Let’s undo this for now. Edit the terraform.tfvars, put a # in front of the vtm_tig_eips line, and then re-run terraform apply. The TIP Group should disappear from your vTM.

Ok, let’s add our Virtual Server (VS) before winding up for today. To start with, let’s add a simple HTTP VS. We’ll add SSL Offload and L7 routing to it in our next part.

Edit your main.tf and add the following block:

locals {
  # If var.vtm_tig_eips is empty, we should not use TIP Group
  should_listen_on_any = "${length(var.vtm_tig_eips) == 0 ? true : false}"

  # This is to provide the list of TIP Group Names to listen_on_traffic_ips
  # paremeter of the Virtual Server. If We don't have any Traffic IPs, this
  # list should be empty; if we do - it should have a name of our TIP Group.
  tigs_list = ["${length(var.vtm_tig_eips) == 0 ? "" : local.tig_name}"]
}

# The Virtual Server
#
resource "vtm_virtual_server" "vs1" {
  name          = "${local.uniq_id}_VS1"
  enabled       = "true"
  listen_on_any = "${local.should_listen_on_any}"

  # We need to use compact() to get rid of values "" which will be there
  # in case we didn't have any TIP Groups
  listen_on_traffic_ips = ["${compact(local.tigs_list)}"]

  # Default pool = "Main"
  pool     = "${vtm_pool.main_pool.name}"
  port     = "80"
  protocol = "http"
}

Since we’re working around two cases here – we either do or don’t have Traffic IP Group – we need to make sure our Virtual Server has correct values for the related parameters: listen_on_any and listen_on_traffic_ips. The code above first defines a couple Local Variables that are set based on condition, and then uses them as values for the respective parameters of the VS1.

listen_on_traffic_ips deserves a special mention. For it to not take effect when we don’t use TIPs, the value it receives must be an empty list – []. However the best we can do with the tigs_list variable is to set it to a list that has one empty element, [""], which isn’t good enough. Luckily for us, compact() can get rid of the empty element, and we’re off to races!

Now, why did we define a local variable to hold a name for our TIP Group? This is to handle the case where the vtm_tig_ips is empty, and the resource "vtm_traffic_ip_group" "tip_group" is not created. Usually we would refer to the .name parameter of our resource to fetch what we need; but what do we do if the resource creation is conditional?

Unfortunately at the moment Terraform will always evaluate both sides of a condition. This means we can’t just create a logic that says “if there are Traffic IPs, create TIP Group and get the name from it; otherwise don’t create a group and set the name to []”. If we did, Terraform would always try to evaluate both, and fail in the case where we don’t have any TIPs specified.

So the work-around for that in our specific case is to create a name for our TIP Group as a local variable that is not dependent on whether actual TIP Group resource exists or not.

Let’s run terraform apply once more to marvel at our creation, and wrap up with that for now.

To sum it up, in this part we talked about:

• Handling conditional logic in our templates,
• How to use that logic to affect creation of resources (through count), and
• How to use it to manipulate parameters for resources that depend on such conditions.

See you in the final Part 4!

Continuing from Part 1, today we’ll make our template do something useful. If you’re following along – make sure your set-up is all good, and the very last exercise of Part 1 completes correctly.

Ready? Let’s carry on!

To remind you what we’re templating, here’s our sample app’s diagram again:

As far as vTM goes, we have four elements we’ll need to configure:

• 2 x Pools (Main and API)
• Traffic IP Group
• Virtual Server

Note that our playpen setup is missing a few bits and pieces, to say the least. We have a single vTM, no ability to raise Traffic IPs, and no actual backend servers. However, the thing is that the template we’ll create will work just fine if we point it at a different vTM (cluster) in a real environment, and give it the right IPs as the input parameters!

But first things first

Remember how in the last exercise we had to type the input values of the variables by hand? This will get old very quickly, so let’s take care of this. When you run Terraform, it will look in the current directory for the file named terraform.tfvars, that can provide values for the variables defined within the template. Following the reasoning in Part 1, I personally like to keep values in this file that are specific to the copy of the template I’m deploying. Let’s make ours. Create a new file called terraform.tfvars in our try-vtmtf directory, and add the following to it:

vtm_rest_ip = "172.16.0.10"

vtm_password = "abc123"

Note that this file often contains sensitive information (as you can see above) such as cloud and login credentials, so it’s an extremely good idea to add terraform.tfvars into your ~/.gitignore_global. The topic of handling sensitive info with terraform is a lot bigger than this blog allows, so keep in mind this file, along with the state files that terraform will create once you run the apply command.

Wait a sec..

Most resources in vTM have a name field that is used by other resources to refer to them; e.g., a Virtual Server refers to the pool by its name. Since our templates could be in theory applied to the same infrastructure (in this case – vTM) multiple times, we need to make sure that we:

• Can find all bits that a particular instance of a template has created; and
• Our resource names (assigned in the template!) don’t clash.

Fortunately, terraform supports generation of random strings that are stable within a template instance, but is created anew once instance is destroyed or an entirely new instance of the template is created. Let’s add one.

# Random string to make sure each deployment of this template
# includes unique string in all resources' names. This should allow
# deployment of more than one copy of this template to the same vTM
# cluster as long as the unique things like IP addresses used for
# Traffic IP Groups are taken care of elsewhere.
#
resource "random_string" "instance_id" {
  length  = 4
  special = false
  upper   = false
}

locals {
  # Create a local var with the value of the random instance_id
  uniq_id = "${random_string.instance_id.result}"
}

The code above creates a local variable uniq_id, that is a String that is a random mix of 4 lower-case letters and numbers. We’ll use it when we get to name our resources.

Let’s add our first pool

We have two pools, each of which serves different parts of our faux web site. In our diagram we have two nodes in each pool. Let’s start with the Main pool. Edit out main.tf, and add the following to it:

resource "vtm_pool" "main_pool" {
  name     = "${local.uniq_id}_Main-Pool"
  monitors = ["Ping"]

  nodes_table {
    node = "10.1.0.10:80"
  }

  nodes_table {
    node = "10.2.0.10:80"
  }
}

Hmm, hold on. Looks like we’ll need to create a separate nodes_table section for each node we want in our pool. But how can we template the configurations where we don’t know the number of pool nodes ahead of time, e.g., when a list of these nodes is passed in as a parameter? It looks like Terraform doesn’t handle this very well at this point; so what do we do?

Fortunately, vTM Provider offers an ability to describe resource parameters that take input as tables, such as nodes_table above, through special data sources. It can then take output of these data sources as the parameter value. Don’t worry, it will come clearer in a moment. Data sources support use of count (which we’ll talk about below soon), so we’re in luck! Let’s see what this looks like.

Replace the resource "vtm_pool" "main_pool" { .. } that you’ve just added (per above) with the following:

data "vtm_pool_nodes_table_table" "main_pool_nodes" {
  # Repeat as many times as we have nodes in our node list variable
  count = "${length(var.main_nodes)}"

  # Get the node from the var.main_nodes list
  node = "${var.main_nodes[count.index]}"
}

resource "vtm_pool" "main_pool" {
  name     = "${local.uniq_id}_Main-Pool"
  monitors = ["Ping"]

  # The data.vtm_pool_nodes_table_table.main_pool_nodes.*.json returns a list
  # of string, each string is a JSON for one node. We need to wrap this into
  # a JSON list, so we add "[]" at the ends, and join the node strings with ","
  # in the middle for a resulting "[{..},{..}]" string that nodes_table_json
  # is expecting
  #
  nodes_table_json = "[${join(",", data.vtm_pool_nodes_table_table.main_pool_nodes.*.json)}]"
}

Ok, there’s a fair bit going on above. First of all, we have a new variable, main_nodes, which we’ll need to add to our variables.tf in a moment.

Second, vtm_pool_nodes_table_table is a Data Source, which is a type of special read-only resource. This particular one is designed to create a JSON string populated with values for specified parameters. In our case, it’s a pool node parameter node.

You can also see a parameter count, that is set to the length of variable main_nodes. Since this variable is of a type list, length will equal the number of entries in it. In our case, the list will be populated with IP:Port strings for our pool nodes, e.g., "10.1.0.10:80" and "10.2.0.10:80".

The way count works is that when it’s present in a resource or data source, terraform will create count many copies of this resource or data source. In our case there will be two, each of which will return a JSON representation of the corresponding node, with value for the node derived from a count-th element within the main_nodes list.

Then we get to our pool. The first thing you’ll notice is that value for the name parameter is made up by adding our uniq_id string to the literal _Main-Pool, so as discussed above we avoid potential clash, and can identify bits of vTM config that belong together.

The second bit is that nodes_table_json parameter is formed by combining the outputs of count outputs of the vtm_pool_nodes_table_table into a string that conforms to syntax of a JSON table. It works like this:

• Each vtm_pool_nodes_table_table output is a {}
• join() connects them all into one string joined by ,, essentially {},{}
• Finally, they are combined with literal [ and ] at the start and the end, to a resulting [{},{}] JSON string we want.

The cool thing about it is that it will work irrespective of how many items are in the main_nodes list – zero or a hundred, so all we need to do is to generate a main_nodes list that matches our environment, and terraform will do the rest.

Whew. Ok, let’s add our new variable main_nodes to our variables.tf and terraform.tfvars, and give it a spin!

variables.tf:

variable "main_nodes" {
  description = "List of nodes for the 'Main' pool"

  # For example, ["1.1.1.1:80", "2.2.2.2:80"]
  default = []
}

terraform.tfvars:

main_nodes = ["10.1.0.10:80", "10.2.0.10:80"]

Note that variable "main_nodes" is supplied with a default value of []. This will cause Terraform to not prompt you for the value of this variable if you have not specified it, for example, through terraform.tfvars as above.

I did this because a configuration where a pool has no nodes it technically valid and useful. An example would be a situation where my configuration may evolve and change when I repeatedly apply the same template, but with different input parameters. I may start with the empty list of nodes in one pool but add them later; while the rest of the configuration will be ready in place.

Now, let’s see what this can do! In our try-tfvtm Docker container (or your own setup), from inside our try-tfvtm directory, run: terraform init, followed by terraform plan. You need to run the init again because we’ve added a new provider to our template – random, and terraform needs to download a copy of that Provider plugin from the Internet.

If all went well, you should see terraform think for a little while, then spit out a bit of output ending with something like:

**Plan: 2 to add, 0 to change, 0 to destroy.**

Take your time to have a look at the output. This is the representation of what terraform will do, once you run terraform apply. You’ll notice that terraform shows many more parameters for the resource we’ve defined (vtm_pool.main_pool). The ones that we haven’t specified are the defaults. This output is often useful in determining how your resource will be configured in the parts that you didn’t specify; bau also for figuring out what parameters your resource has for you to play with.

Before wrapping up this part, let’s run the terraform apply, and see what happens.

Terraform will again display the plan output and ask you to type “yes” to confirm that you let it make the proposed changes. If you don’t want to be prompted, add -auto-approve to the end of the command, so it reads in full as terraform apply -auto-approve

If all went well, you should see:

**Apply complete! Resources: 2 added, 0 changed, 0 destroyed.**

You can wrap up with that, or go to https://172.16.0.10:9090 (change the 172.16.0.10 to your vTM’s IP), and visit Services -> Pools, where you should see a new pool named [4 random symbols]_Main-Pool with 2 nodes in it – 10.1.0.10 and 10.2.0.10, both with port 80.

To sum it up, in this part we learned:

• How to save values for Terraform variable parameters in terraform.tfvars file,
• How to name vTM resources to avoid conflicts between multiple instances of a template applied to the same vTM,
• What are the Data Sources and Resources, and
• How to create a vTM Pool with a variable number of nodes in it.

See you in Part 3!

The freshly-released Pulse Virtual Traffic Manager (vTM) v18.1 comes out with a Terraform Provider for vTM. The provider ships with 100% coverage for all vTM’s REST API resources, and includes support for the API version 4.0 that goes back to vTM 17.2, and the API version 5.2 that includes all the newest features that shipped with vTM 18.1.

In this 4-part post we’ll do a quick introduction of Terraform provider for vTM, and show how it can help you support the needs of your applications.

Hopefully Terraform does not need much introduction. It is one of the most prominent Infrastructure as Code tools, that can provision a wide variety of IaaS, PaaS, SaaS, and on-prem services through the use of Providers. The role of a Provider is to take the details needed to create / read / update / delete resources specific to a particular Service (e.g., AWS), and make them available through the Terraform’s language (HCL).

The beauty of this approach is in that you can describe a collection of resources supported by any combination of the Providers using the same language (HCL). And you can mix and cross-reference these resources in one template, too.

It’s worth noting that Pulse were clearly not the first ones to realise that this approach would fit very well with treating vTM as an infrastructure that provides load balancing “services” to applications sitting behind it. At this point I’m personally aware of two independent implementations of Terraform provider for vTM, created by customers themselves: One and Two.

So without further ado, let’s dive into it.

An example topology

For this blog I wanted a very simple topology that’s fairly typical: a single domain (corp.com) with a single property (www), and two backends – main (/) and API (/api).

A cluster of 2 x vTMs sitting in front of the infrastructure that serves this provides a few basic functions:

• Terminates/offloads SSL
• Performs L7 routing based on the request Path
• Provides High Availability for the IPs that match the property’s FQDN by using Traffic IPs.

If you’d like to follow at home

I’ve created a Docker image (based on Apline Linux) with terraform itself + both versions of terraform provider for vTM installed in /usr/local/bin. You can run a copy of it so that you can test things out as we go:

mkdir try-vtmtf && cd try-vtmtf

docker run -it --rm -v $PWD:/root/try-vtmtf -w /root/try-vtmtf dk114/try-vtmtf:1.1 /bin/ash

terraform --version
# You should see "Terraform v0.11.7"

The docker run command included above will start a Docker container, and mount your current directory (try-vtmtf) under the path /root/try-vtmtf. Any changes you make to files that reside in this directory will persist on the your main host running container. Also any changes you make to files in this directory from outside the container will be immediately visible to programs running inside (such as terraform). This means you can edit the files we’ll be working on using an editor on your host, and terraform inside the container will “see” your changes as soon as you save them to disk.

If you prefer to create your own setup, download the following:

• Terraform itself
• Terraform provider for vTM – linux64 or
• Unofficial build of Terraform provider for vTM – darwin64

The MacOS/darwin provider will come down named as terraform-provider-vtm_v4.0.0-darwin. You’ll need to rename it and remove the -darwin part, so it’s called just terraform-provider-vtm_v4.0.0, then make it executable (it’s a static binary), and place somewhere in your $PATH.

Note that the offcial instructions for installing plugins differs from this approach. I found that both work. Placing plugin somewhere in $PATH however has benefit of making it available to users system-wide without asking them to make their own copy.

You can continue working in another Terminal window on our template in the directory you’ve created (try-vtmtf).

If you already have a copy of vTM you can play with – note down its management IP address, make sure REST API access is enabled (typically on port 9070), and that your machine can reach it. If not – you can get one like so:

docker run --name vTM --rm -e ZEUS_DEVMODE=yes -e ZEUS_EULA=accept -e ZEUS_PASS=abc123 -p 9090:9090 -p 9070:9070 -p 80:80 -p 443:443 --privileged -t -d dk114/pulse-vtm:17.4

This should download a Docker image with a vTM 17.4, and run it in the Developer Mode (all features on, throughput-limited to 1Mbps). It sets password for the admin user to abc123; change as you see fit. You should then be able to reach it from your terraform container on the IP of your machine (172.16.0.10 in my case):

curl -u admin:abc123 -k https://172.16.0.10:9070/

# I get the response:
# {"children":[{"name":"api","href":"/api/"}]}

If you got to this point successfully – sweet!

Let’s begin

Terraform works on templates; but with a little twist – when you run terraform <operation>, it will look inside your current directory for all files named *.tf, combine them behind the scenes, and treat them as if it was a single file. So each template will need its own separate directory; in our case the one we’ve created and changed into – try-tfvtm.

Most templates have one or more variables that can be set with default values, or given those values at the run time. Terraform then takes those values, combines them with the template code, and calculates the final desired state, which is what it will work to achieve when you run terraform apply.

Also templates are often shared between people, and to help quickly figure out what are the variables used in a template, it’s often useful to split variable definitions into a separate file. Let’s do that, and create two files: main.tf and variables.tf:

# Inside our try-vtmtf directory:
touch main.tf variables.tf

Edit the main.tf, and add the following block to it:

provider "vtm" {
  base_url        = "https://${var.vtm_rest_ip}:${var.vtm_rest_port}/api"
  username        = "${var.vtm_username}"
  password        = "${var.vtm_password}"
  verify_ssl_cert = false
  version         = "~> 4.0.0"
}

This block of code will tell terraform that we have a vTM (or maybe a vTM cluster) that can be reached on https://${var.vtm_rest_ip}:${var.vtm_rest_port}, authenticated to using ${var.vtm_username} and ${var.vtm_password}; that it’s using a self-signed cert (that we should none the less trust for now), and that we’ll talk to this vTM using REST API v4.0. At the time of writing this includes vTM versions 17.2x, 17.3, 17.4, and 18.1.

All these ${} bits are variables, and we’ll need to define them. Let’s do that. Save your main.tf for now, edit variables.tf, and add the following to it:

variable "vtm_rest_ip" {
  description = "IP or FQDN of the vTM REST API endpoint, e.g. '192.168.0.1'"
}

variable "vtm_rest_port" {
  description = "TCP port of the vTM REST API endpoint"
  default     = "9070"
}

variable "vtm_username" {
  description = "Username to use for connecting to the vTM"
  default     = "admin"
}

variable "vtm_password" {
  description = "Password of the $vtm_username account on the vTM"
}

As you can see, two of our variables – vtm_rest_port and vtm_username have defaults. When thinking about when to set a default, I personally think “is this value going to be applicable in all/most cases where I’ll be using this template?” If yes, then I set the default. If not – we’ll be passing value for it during run time.

Ok, it’s time to give it a go, even though our template doesn’t do anything useful yet! Save both our files, and from inside your try-vtmtf container check that you’re in /root/try-vtmtf directory and can see your main.tf and variables.tf files in it. If we’re good, do this:

ls -l
# total 8
# -rw-r--r-- 1 root root 234 Apr 26 03:51 main.tf
# -rw-r--r-- 1 root root 423 Apr 26 03:51 variables.tf

terraform init

# You should see messages saying:
#    "Initializing provider plugins...
#    Terraform has been successfully initialized!"

terraform plan

# Terraform should prompt you for the values of
# var.vtm_password and
# var.vtm_rest_ip

Give it the values, and if all is good and your try-vtmtf container can reach your vTM – you should see a message informing you in happy green letters: “No changes. Infrastructure is up-to-date.”

To sum it up, in this part we learned:

• What is a Terraform template,
• How to declare a connection to a vTM (cluster), and
• What the variables look like in Terraform and what they’re useful for.

At this point, pour yourself some beverage to celebrate getting your set-up into the shape for the Part 2, where we’ll actually create some useful configuration.

Quick announcement – I’ve published a new version of the CloudFormation template that deploys a HA cluster of Pulse vTM instances into an existing VPC.

Repo: https://github.com/dkalintsev/Pulse-vTM-CloudFormation

Highlights:

• Suitable for using as a module in bigger stacks, e.g., CloudFormation or Terraform that take care of the VPC and other infrastructure. Has been tested with the later.
• Updated with the latest vTM AMIs for 17.2r2, 17.3, 17.4, and 18.1.
• Supports additional Security Groups (e.g., when your target instances use SG with Source based on SG name).
• Can take UserData from Pulse Service Director’s Cloud Registrations for self-registration.

Please see README for full details, especially the Disclaimer at the beginning.

I recently went through a somewhat painful exercise figuring out how to implement this, and that’s what I’m here to share with you.

As they say – “failure is mother of all learning”, or something on those lines (actually I just made it up), but anyway.

What I failed to achieve is make it work with optional SGs Parameter type List<AWS::EC2::SecurityGroup::Id>, that displays a nice drop-down of all your existing SGs when deploying through UI. Instead, I have to use CommaDelimitedList, which isn’t nearly as nice, but hey – at least it works.

The sticker was in the validation logic (see Gist below): when user doesn’t make any selection for Parameter with type List<AWS::EC2::SecurityGroup::Id>, CloudFormation declares this Parameter as “unknown”, so it can’t be used for things like Fn::Equals and such. That’s what I learned (and proudly twitted about with the #awswishlist tag, asking for a way to validate if such Parameter is empty)

So, enough talking, off to show the goods.

Notes:

1. Yes, the logic spider nest in SecurityGroup attribute of InstanceLaunchConfig looks, um, unattractive; but it works.
2. You may probably want to make sure the value you supply for AdditionalSGs doesn’t have any spaces in it.
3. I declare PrimarySG as a Parameter here – it’s just for brevity so I don’t have to include an actual AWS::EC2::SecurityGroup Resource that in turn requires a VPC, and so on.
4. What’s with all Splits and Joins? Well, SecurityGroups is a List, which means that we first must make a String from all our SGs, and then Split the result. Since AdditionalSGs is a CommaDelimitedList (and not a String), it needs to be run through Fn::Join (line 46), as well. Yeah, ugly.

I am making a buch of assumptions about what you, dear reader, already know – so shout out in comments if something is still muddy.

With that, on to it.

There’s no point arguing that Single Page Apps (SPAs) are in vogue. One of the approaches to SPAs is to move business logic into the client, and leverage a range of API-based services to provide needed server-side functionality. Here’s a very good argument presented on this topic: Joe Emison – 10X Product Development.

A side effect of following this pattern is your SPA code ends up on a server that only knows how to serve your static client content (html / js / css / images). This may present a problem when you try to integrate your SPA with another stand-alone application, for example, blog hosted by WordPress.

Let’s have a look in a bit more detail, followed by what can be done about it.

A bit of a background

I’m not an SEO guru, but my understanding is that your appearance in search results is better if your whole web property appears behind the same domain. For example, if your main web page is corp.com, it’s better for your web image if the URL for your company’s blog is something like corp.com/blog, as opposed to blog.corp.com, which may count as a separate web property.

When your web page is hosted on a “traditional” web server, serving your SPA contents for corp.com and your self-hosted WordPress blog from corp.com/blog is not a problem. If traffic levels for your site are moderate and high availability is not a major concern, it’s a perfectly acceptable solution.

If, however, you’d like to cater for unpredictable traffic levels of, well, any magnitude, and/or don’t want to deal with undifferentiated heavy lifting that is IT infrastructure administration of your web servers, you may be using specialised services to both host your SPA and your blog. Which brings us right back to our SEO problem.

Let’s say you used an AWS S3 Static Web Site Hosting for your corp.com SPA. You set up DNS for corp.com to point to S3, and the life is good. That is, until you want to add a /blog, but hosted by WordPress.com. The problem here is that the IP address(es) for corp.com, which happens to be the common part of both corp.com and corp.com/blog can point to either S3, where your SPA lives, or to WordPress, where your blog is.

If you try to send a request to S3 for something that isn’t there, S3 will tell you that those are not the droids you’re looking for:

This is where in-line HTTP request and response processing comes to rescue. Recognising this and several other related needs, AWS has just announced service called Lambda@Edge. This service allows you to run Node.js code at CloudFront locations that can modify your web site visitors’ HTTP requests and CloudFront / origin servers’ responses sent back to them. For example, to solve our problem above, code may look for the /blog in the request path, and redirect that request to a different origin (WordPress blog).

The service is currently in preview, and its functionality is in early days; however it will likely mature rapidly. In meanwhile, if you have a similar need to what’s described above, here is how you can do it with a Brocade vADC.

Setup

For our demo solution, I’ve grabbed a copy of AngularJS Tutorial, slightly modified it to point at my blog for the /blog integration, uploaded it into an S3 bucket, and turned on S3 Web Site Hosting on it.

I also created a domain test.11-14.net to represent our hypothetical corp.com, and got a Let’s Encrypt SSL certificate for it to demonstrate serving S3 web site content via SSL.

The last bit was to deploy a Brocade Virtual Traffic Manager Developer Edition, and create a virtual server with some TrafficScript to process HTTP requests and responses.

When all is in place, we can hit our web site and see it in action. (Um, since I’m running the vADC in my own AWS account, I’ve stopped the vADC used to record this demo to save money.)

Opening the main page shows us the SPA, sitting behind our pretty URL:

..while when we hit the blog..

..we’re presented with our blog, appearing integrated with the main site, and even all article links looking pretty – see the bottom of the screenshot when the mouse is pointing at the header of the post on screen:

So, how does it all work?

vADC Configuration

First, let’s set up our backend. Our page has two sources: an S3 bucket with the SPA, and the blog hosted by WordPress.com. To match this, we create two Pools:

The first one, unimaginatively called blog, is the WordPress site. When I put telecomoccasionally.wordpress.com:443 into the Node detail, vADC resolves the FQDN, sees it matching two IPs, and puts those IPs as the pool nodes. I chose “Full HTTPS” as the health monitor, since WordPress serves my blog over HTTPS. For HTTPS connections to this pool, vADC will terminate the client SSL session, and open a different one to the backend pool.

The second one, test-SPA-s3, is an HTTP node that points to the S3’s IP address. It’s HTTP, since S3 doesn’t provide HTTPS for web sites hosted on it. For client-side HTTPS connections leading to this pool, vADC will perform SSL offload.

Next, we create the Virtual Servers – one for HTTP (port 80), and one for HTTPS (port 443):

Virtual Servers are the ones listening to client connections, and serve the content from Pools accordingly.

I’ve configured both virtual servers to point at the pool test-SPA-s3, but as you’ll see below, we’ll be overriding this in our script. You can also see that there’s a green “tick” against “Rules” for both virtual servers, and this is where the TrafficScript is applied.

vADC provides ability to run three Rule scripts for each Virtual Server:

• Request: script is run over what client sends to vADC. The result of the script processing is then sent to the backend pool.
• Response: script is run over what the backend pool member responds with to vADC. The result of the script processing is then returned to the client.
• Transaction Complete: relevant for the transactions, such as “when the connection is closed or when a complete HTTP, SIP, or RTSP response has been sent to the client”

For our example, we’ll be using Request and Response rules, which will do the following:

• Virtual Server will expect to see connections for the domain test.11-14.net
• Request rule attached to the Virtual Server will check if the path in the request has string /blog in it
• If it does not, the request will be routed to the pool test-SPA-s3, pointing at the S3 bucket hosting the SPA code
• If it does, the request will be routed to the pool blog, pointing at our blog site on wordpress.com
• In both cases, vADC will rewrite the Host: header accordingly, since S3 will expect it to match the S3 bucket name, and WordPress will expect it to match the blog page (telecomoccasionally.wordpress.com).

Additionally, the Response rule will look through the body of the responses from S3 and WordPress, and substitute the original S3 or WordPress URL strings with the appropriate “pretty URL” – test.11-14.net or test.11-14.net/blog accordingly.

This will ensure that for example any links on the blog appear to be “pretty”, giving an illusion of the blog being completely integrated with the main site.

Here’s what the rules look like; I think you should have no trouble figuring out what the code does:

Granted, you potentially may need to manipulate other things like other headers and cookies; but TrafficScript is there to help – you have access to all parts of requests and responses.

And there you have it – nice clean integration!

Bonus material

While the ability to make changes to the contents of requests and responses is cool, it’s not all you can do with TrafficScript. Here’s one cool use case we’ve seen – in this implementation, web site behind vADC provides a “Red Carpet” experience to the site’s visitors based on a result of an in-line API call to find out visitor’s loyalty program status. Once visitor’s eligibility is established, they are directed to a separate backend pool, and given a more preferential QoS treatment:

# Extract the Customer ID from the request:
$CustID = http.getFormParam("CID");

# Look them up to see what program level they are currently at:
$CIDLookup = http.request.get("http://cidlookup.corp.com/cidLookup.php?CID=".$CustID);

# If they have platinum status, let's roll out the Red Carpet:
if(string.containsI($CIDLookup, "platinum")) {

   # We have a dedicated pool of servers for Platinum Members:
   pool.select("pool_Platinum_Custs");

   # And apply a pair of special Bandwidth Classes so we don’t
   # slow them down when the site is under load:
   request.setBandwidthClass("BW_Platinum_CID_REQ");
   response.setBandwidthClass("BW_Platinum_CID_RES");
}

Looks good, but..

..I don’t have experience with vADC, and how to deploy and configure it!

If you’re in AWS – CloudFormation to rescue! I’ve been working on a few generations of CloudFormation templates to help smooth out deployment of vADC clusters into AWS. A good place to start is the variant that deploys two vADC nodes into a cluster across two AZs, and pushes vADC configuration to it using vADC Puppet Module. Template should cover most of your needs when it comes to vADC deployment and configuration. You can pull it apart and integrate bits into your own solution, no problem. All code is Creative Commons and is completely free for any use as a whole or in part.

You can find the template, along with a reasonable README that explains what’s going on there by clicking here.

P.S. The template above deploys a pair of vADCs as static EC2 instances. If you’d like to manage your vADC cluster by AWS Autoscaling, feel free to drop by here.

Happy ADCing!

In this short post I’ll cover a situation we’ve encountered at one of our customers where things “should” have worked but didn’t, and what was the reason for that.

I’ve been contacted to help solve a mysterious issue that popped up during a customer PoC, where a 6740 HW VTEP connected to an NSX environment was behaving in unexpected ways.

Everything looked great on the control plane – VTEP and MAC tables, both on Controllers and hosts, and on the underlay data plane – everything that should be able to ping each other, could. Traffic capture on ESXi host showed that a VM on a logical switch bridged to a VLAN through HW VTEP were sending traffic correctly encapsulated, and receiving traffic sent from a physical server.

But, two things weren’t happening: no traffic from VM arrived at the physical server, and BFD sessions between the HW VTEP and replication nodes were refusing to come up.

The first reaction was to check whether ESXi hosts’ VTEPs and the VTEP on 6740 were in different IP subnets. Yep, they were.

Here are the relevant bits of a mock 6740’s config:

rbridge-id 10
!
interface Port-channel 20
description L2 to ESXi hosts VTEP segment
switchport
switchport mode trunk
switchport trunk allowed vlan all
!
interface Ve 110
description Default gw for ESXi VTEPs
ip address 192.168.0.1/24
!
interface Ve 112
description HW VTEP
ip address 192.168.1.10/24
vrrp-extended-group 112
virtual-mac 02e0.5200.00xx
virtual-ip 192.168.1.1
short-path-forwarding
!
overlay-gateway NSXv
type hardware-vtep
ip interface Ve 112 vrrp-extended-group 112
attach rbridge-id add 10
attach vlan 3000
activate
!
nsx-controller nsxv
ip address 172.16.0.100 port 6640
activate

ESXi hosts were connected to the 6740 on L2 through the Port-Channel 20, with their VTEPs sitting in the VLAN 110. Default gateway (pay attention now, this is important!) for ESXi VXLAN IP stack was set to 192.168.0.1, which is as you see above is on the interface Ve 110.

We could ping perfectly fine between either Ve 110 or Ve 112 and ESXi VTEPs, so it wasn’t a problem with connectivity. VXLAN packets leaving ESXi hosts had the correct outer DMAC = MAC of the Ve 110, since it’s the default gateway that is used to reach HW VTEP’s IP (192.168.1.10 on Ve 112).

On ESXi hosts acting as replication nodes, we could see incoming and outgoing VXLAN packets with BFD PDUs inside to and from HW VTEP. Yet, no worky.

Next thing we tried was to switch over from VRRP-E VTEP to Loopback, by adding interface Loopback 1 and switching from Ve 112 to Loopback 1 in the overlay-gateway configuration.

rbridge-id 10
!
interface Port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan all
!
interface Loopback 1
ip address 10.10.10.10/32
!
interface Ve 110
ip address 192.168.0.1/24
!
interface Ve 112
ip address 192.168.1.10/24
vrrp-extended-group 112
virtual-mac 02e0.5200.00xx
virtual-ip 192.168.1.1
short-path-forwarding
!
overlay-gateway NSXv
type hardware-vtep
ip interface Loopback 1
attach rbridge-id add 10
attach vlan 3000
activate
!
nsx-controller nsxv
ip address 172.16.0.100 port 6640
activate

Well, still no worky. Puzzled, we’ve brought in some engineering help (hi Ram!). After some further collective head-scratching, we’ve noticed that the "In" packet counters on BFD sessions on the 6740 weren’t increasing, meaning that BFD packets from ESXi hosts weren’t reaching the BFD function. (Remember – BFD packets between HW VTEP to ESXi travel inside VXLAN tunnel with VNI=0).

After seeing this, we decided to try and delete the VRRP-E configuration from Ve 112 (the only VRRP-E interface in this particular configuration). Lo and behold – we were cooking with gas!

With the problem out of the way, it was the time to dig in to understand what caused the problem, and how to prevent it in future.

The first clue was (surprise!) in the documentation:

If the VXLAN packet entering a VDX VTEP-enabled device on a Layer 3 interface (such as a routing next hop) is different from the VRRP-E based-VE interface configured for the VTEP, but at an ingress interface where the VTEP VRRP-E VLAN is also configured, and the final destination is an NSX-configured VXLAN tunnel (as identified by the VXLAN tunnel parameters of source IP and destination IP), then the VXLAN traffic is routed to the VTEP interface in the VDX and is a candidate for VXLAN to VLAN bridging, but only if the destination mac of the ingressing VXLAN traffic is the same as that of the virtual mac of the VTEP VRRP-E session.

(A bit hard to parse for me, but once you know what you’re looking for, it kind of makes sense).

So let’s see how this applies to our situation.

Remember, since 6740’s VTEP is in a different IP subnet from ESXi hosts’ VTEPs, said ESXi hosts will use their default gateway as the next hop, which in our case happens to be on the same 6740 as the HW VTEP. So ESXi hosts will set outer DMAC in their VXLAN packets to the MAC of the Ve 110, which is not the same as the VRRP-E virtual MAC of Ve 112. And, in line with the quote above, 6740 will drop these VXLAN packets.

If we look carefully, we note that the Port-Channel 20 has switchport trunk allowed vlan all on it, which of course includes VLAN 112, that matches Ve 112 that has VRRP-E enabled. I know it’s a different VLAN ID! Looks like it doesn’t matter, though.

In further lab testing we found that if we kept the original configuration (with the Ve 112 used as the VTEP), but deleted VLAN 112 from Port-Channel 20, we were back to happy days.

Another way would be to follow recommendation in the document linked above, which is:

…creating VE interfaces for each of the ingressing transport VLANs on all the RBridges in the VCS, and then configuring them with the same VRRP-E VRID and virtual-mac address as the VRRP-E VRID and virtual-mac address that was configured for the VTEP.

Translating this for our situation: our “ingressing transport VLAN” is VLAN 110, so all we need to do is change Ve 110 configuration so it looks something like:

!
interface Ve 110
ip address 192.168.0.110/24
vrrp-extended-group 112
virtual-mac 02e0.5200.00xx
virtual-ip 192.168.0.1

The changes are:

• Switch the main interface IP to a different one in the same IP range;
• Add vrrp-extended-group 112 with the corresponding vitual-mac, and virtual-ip set to the Ve 110’s original IP address.

Make sure that you use the same group number (112) as the one used on the Ve used for VTEP.

This solution probably makes a lot of sense since you’d typically have a pair of 6740s for your HW VTEP, and configure default gw redundancy for ESXi VTEPs with VRRP-E on Ve 110. Just make sure to use the same VRRP-E group ID.

Q&A

Q: What happens if I have multiple VRRP-E groups, each with different group ID?
A: one of the virtual MACs will be chosen randomly and programmed into ports. You don’t get to choose, AFAIK.

Q: So what do I do?
A: You should be able to safely use the same group ID for all relevant VRRP-E groups. It will use the same MAC on the associated VLANs, which should be fine since VLANs are separate MAC domains.

Q: Does it matter if I use Loopback or Ve for my HW VTEP?
A: No.

Q: Is there an alternative solution?
A: You can add an external L3 hop between your ESXi VTEPs and the VTEP on a 6740. For an example, please see the “Infrastructure” diagram my previous post on the topic. The L3 hop in question is the one labelled “Router (Data)”.

Q: What about removing VLAN IDs of Ve interfaces with VRRP-E from the interface where VXLAN packets coming in?
A: Yep, can do that, too. As long as you remove all VLANs with Ve interfaces that have VRRP-E. Likely scenario is where you have only a single 6740 node, like in our pictured case.