I'm pretty excited to be able to share with you the first beta release of RoadCaptain!

RoadCaptain is an app that makes riding on Zwift even more fun and can really push your limits in Watopia.

How? Simple: you are no longer limited to the fixed routes in Watopia, with RoadCaptain you can build your own routes and explore Watopia even more.

Always wanted to do 3 laps on the Volcano as a warm up followed by blasting through the Jungle Loop? Now you can!

Read all about it here!

When I started down this path I've had quite a few people ask me "WHY?? DEAR $DIETY WHY DO YOU WANT THIS?"

Fair enough, given the whole ecosystem around containers and especially Docker and Kubernetes is largely focused on Linux and *nix's we are a .Net shop and most of our developers work on Windows machines. Switching people over purely for the sake of running a kubernetes (k8s) environment is not something I'd encourage.

One might say: "But you can create a k8s cluster in the cloud right?", well, yes that is certainly possible and most cloud providers give you free credits to do that. The drawback is that it forces you to always have an Internet connection available (repeat after me: "The network is always reliable, there is no latency"). I've found that having resources available locally works well for me, but this is definitely something you'll need to decide for yourself.

Having said that, this is about setting up a k8s cluster on Windows 10.

Here be dragons

I want to be honest with you, figuring this out has given me a few more grey hairs and I fear I've worn down some of my more choiciest swear words. This was probably more due to my own stubornness than documentation not being available. Still, it's always a challenge to piece together the right information and discarding whatever knowledge you have from working on different platforms (I used to code on a Mac).

Common approach

What you'll find is when searching for k8s on Windows is that people will recommend minikube and that is usually a good option. Unfortunately I've had a lot of trouble getting to get this setup to work reliably on my machine. The main reason for this seems to be in the networking area and IPv4/IPv6 configurations. At home it works like a charm (IPv4 only network), however at the office we have a IPv4/IPv6 combined network and things break down rapidly. I've tried disabling IPv6 on the network adapters (both host and the k8s VM) but to no avail.

"Use docker"

Luckily we have the Internet. After searching how to resolve the IPv4/IPv6 problems I came across this comment that made me go: "well.... BLEEP..."

So I set out to see if I could get the Docker way to work. (Narrator: it worked)

Starting out

There are a few things you'll need before setting off on this quest of getting a k8s cluster working on your machine:

• Windows 10 Enterprise, Pro or Education edition with Hyper-V enabled (the link documents how to do that)
• Docker Desktop (I tested with Version 2.0.0.2 (30215), stable channel and build 0b030e1) previous versions may work but YMMV. According to the docs it should work from version 18.06 Stable.
• kubectl (I recommend to install this using Chocolatey)

Configuring docker

After installing Docker (and rebooting! It's Windows after all) you will need to enable k8s. Open the Docker settings application and select the Kubernetes option, you should see this:

Make sure the Enable Kubernetes and Show system containers (advanced) options are selected.
Docker will prompt you that it needs to install k8s support, click Ok when it does.

After the installation is complete, the Docker configuration is complete.

Verifying Kubernetes works

As Docker already manages the k8s cluster we need to check if it is properly running and that you can access it.

From a PowerShell or Command window use kubectl to check connectivity:

$> kubectl cluster-info

That should provide you with output similar to the following:

Kubernetes master is running at https://localhost:6445
KubeDNS is running at https://localhost:6445/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

If you're a command line hero, you'll be fine from this point on. However I recommend also deploying the dashboard so you can have a bit quicker look at your k8s environment.

Setting up the k8s dashboard

Open a PowerShell console and run the following commands (from the official docs):

• kubectl create secret generic kubernetes-dashboard-certs --from-file=$HOME/certs -n kube-system
• kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
• kubectl proxy

At this point you can access the dashboard on http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login but you'll be greeted with this:

Select the Kubeconfig option and select the config file from your %USERPROFILE%\.kube directory. After clicking Sign In you should see the dashboard. If not please check the docs.

NOTE: The dashboard deployment is now using v1.10.1. You may want to check here if that's the latest version before deploying.

Man the helm

At this point you'll have a working k8s cluster running on your Windows machine, without minikube. The next step is deploying a helm chart to your cluster.

I've provided a sample helm chart that you get here, clone the repository and cd into the directory. Next, run helm install . and wait for the command to complete and it should end with something like:

NOTES:
1. Get the application URL by running these commands: http://k8s.demo.net/

Now when you try to access that URL you'll be sadly disappointed because it will report a connection refused error because there is nothing listening at that address.

Knock-knock

This is weird, the helm chart contains an ingress specification so it should be accessible right? When we check the dashboard it even shows listed under the ingresses. What gives?

Well by default there is no ingress controller active so there is nothing that will accept HTTP(s) traffic and route it to our service. Fortunately we can solve this problem by adding the nginx ingress controller.

Open a PowerShell console and run the following command:

$> helm install stable/nginx-ingress --name ingress-nginx --namespace ingress-nginx --wait

Once this command completes you should be able to connect to the service. Let's see:

$> curl.exe -v "http://k8s.demo.net/"
* Could not resolve host: k8s.demo.net
* Closing connection 0
curl: (6) Could not resolve host: k8s.demo.net

Ok, well great. It still doesn't work. Actually this was to be expected, your local machine is not using a DNS server that knows how to map the domain name k8s.demo.net to 127.0.0.1 where the ingress controller is listening at.

The easiest way to fix this is by adding an entry to the HOSTS file. Open up a PowerShell window with administrative privileges (you'll need this to edit the file) and run:

$> notepad c:\windows\system32\drivers\etc\hosts

and add the following entry:

127.0.0.1    k8s.demo.net

Save the file and run curl again:

$> curl.exe -v "http://k8s.demo.net/"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to k8s.demo.net (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: k8s.demo.net
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.15.6
< Date: Mon, 28 Jan 2019 13:59:07 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 6
< Connection: keep-alive
< X-App-Name: http-echo
< X-App-Version: 0.2.3
<
hello
* Connection #0 to host k8s.demo.net left intact

Success!

The finish line

At this point you have a fully working kubernetes cluster running on your machine and you have deployed a service that you can call from your machine using a proper domain name.

There are a few things to keep in mind. To run the dashboard you'll have to run kubectl proxy, it's possible to expose the dashboard using NodePort but I haven't tested that yet.

The HOSTS file "hack" isn't very user friendly. If you're working with the same services all the time this shouldn't be too big of a deal but there should be a way to do this easier with a local DNS server for example.

For now, happy k8s'ing. If you have questions feel free to reach out via Twitter @Codenizer.

Acknowledgements

This wouldn't have been possible without a lot of people writing Github issues, official documentation and blog posts. I've added a list below that hopefully captures most of the resources I've used for this journey.

• Scott Hanselman's blog: https://www.hanselman.com/blog/HowToSetUpKubernetesOnWindows10WithDockerForWindowsAndRunASPNETCore.aspx
• Rancher docs that pointed me to nginx ingress: https://rancher.com/blog/2018/2018-05-18-how-to-run-rancher-2-0-on-your-desktop/
• Docker documentation: https://github.com/docker/for-win
• Nginx ingress docs: https://kubernetes.github.io/ingress-nginx/examples/docker-registry/
• k8s dashboard wiki: https://github.com/kubernetes/dashboard/wiki
• Github issue for HOSTS file: https://github.com/docker/for-win/issues/1901
• Setting up ingress on minikube: https://medium.com/@Oskarr3/setting-up-ingress-on-minikube-6ae825e98f82
• Helm: https://helm.sh/
• Hashicorp echo server: https://hub.docker.com/r/hashicorp/http-echo

Many people in recent years have jumped on the (micro) services bandwagon for many reasons. But, as with radioactively-spider-bitten-students, with great services come great responsibility... eh, well, observability.

As more and more services are introduced into your landscape, it becomes more difficult to keep track if services are running. More importantly, how do you quickly see which services are affected if one dies?

Healthy, what does it mean?

First things first: How do we know if a service is healthy?

Most services depend on other components to function properly, for example a database, a cache or another service. If any of these are not available or are not responding quickly enough, we can say that the service is unhealthy.

Still each service knowing abouth the health of its own components doesn't do us a lot of good just yet. We're in the "A tree falls in the forest and there is nobody there does it make a sound?" territory right now.

A first approach here is to have the checks log an error when it fails its criteria, something along the lines of Health test failed. We can now leverage our logging infrastructure (Splunk, ELK, LogzIO etc) to trigger alerts for us.

And then suddenly your Slack #operations channel contains these little messages:

[ERROR] ReallyImportantService is unhealthy

Ok, well now.

So obviously we know the service failed but we still need to resort to our logs and figure out what part of the service is failing. Database? Messagebus? ¯\_(ツ)_/¯

We need to go deeper

What we need is a way to ask the service itself what's wrong instead of digging through our alerts and logs. Given that we're dealing with API's that are accessible over HTTP anyway, maybe we should leverage that. We could create an endpoint on our API that provides us with the results of the health checks.

But why reinvent the wheel? Fortunately for us the folks at Beamly have already done a lot of the thinking for us. They came up with the Simple Service Status Endpoints or SE4 that defines four endpoints to quickly get information about the health of a service.

For our goal of seeing what is going on we'll be using these two:

• /service/healthcheck/gtg which tells us if the service is OK (/gtg: good-to-go)
• /service/healthcheck which gives a list of dependencies and if they are OK

I won't go in too much detail (the linked doc does that better) but we now have a common way of accessing health information for all our services. You can imagine that this helps a lot when you have a lot of services, you don't need to write each integration separately.

Mapping expedition

Unfortunately we're still not a lot closer to where we want to be: a view of our landscape with health indicators.

Even though right now we can get detailed information about a service, we need a way to determine which service is using what. If you read the SE4 document, you'll see that each test has a test_name and we can use that to identify the dependency.

But how do we match identifiers to services?

You will probably have already adopted a naming convention for your services. Perhaps like this https://environment-domain-api.mycompany.com/ to create https://production-user-api.mycompany.com/. Your application name would be user-api and this is a good candidate for an identifier to use in the test_name for services that depend on the User API.

The problem is, we still need to find all our services. But before we can find our services, what environments do we have?

When you run your services in The Cloud^TM, you'll get the benefit of infrastructure management APIs. These allow you to inspect what you have deployed and where you can find these services (and many other things like configuration, firewall settings, you name it).

For example if you run on Azure, you could query App Service instances, on AWS you could look for Application Load Balancers or you can use Consul or Zookeeper on your own infrastructure.

My current discovery method is slightly less sophisticated: a config file.... but it does the job.

The key point here is that you use the data from service discovery to determine which services make up the landscape for a particular environment (and therefore what an environment means to you).

Pulling everything together we can now find out which services exist, what the relations are between them and what additional dependencies exist.

Visualizing

We have our discovery, naming and endpoints, but still we have no visual representation of the landscape. The next step is figuring out how to show this.

Fortunately there are a lot of libraries available to do this for us, vis.js is one I've found to work well. We can transform our list of services and dependencies into their graph equivalents: nodes and edges. The healthy state of the services and dependencies is then used to color the nodes and edges red or green.

A sample graph then looks like this:

Of course this is a very simple example but I hope it gives you an idea about how it works in scenarios with many more services.

Wrapping up

We've seen how we moved from having no information on our service health to a visual overview that helps to quickly spot the problem and the scope of it. This can be a big benefit when alerts go off and you quickly need to diagnose a problem. It also allows you to easily communicate a problem scope to stakeholders.

At $company we're using Voltmeter which is an implementation of what I've talked about in this post. Hopefully you'll find it useful and it allows you to get started using this yourself to monitor service health.

This week I tried to set up a Specflow acceptance test project in one of the applications I'm working on. I already knew that Specflow support for netcore is work in progress. So thinking myself very clever, I created a project targeting net462 with the assumption that it would JustWork^TM

Of course, it didn't. The code behind files were not generated from the add-in on save 😕

Okay, so I knew you could let Specflow generate the code behind on build, or actually BeforeBuild. Good! Let's try that!

<Import Project="packages\specflow\2.2.1\tools\TechTalk.Specflow.targets" />

that should just.... shit:

The imported project "C:\git\project\packages\specflow\2.2.1\tools\TechTalk.Specflow.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

Problem #1:
packages\ is not in the solution directory anymore because reasons (hint: VS2017 + new NuGet) 😶. Great, so where do we reference the Specflow targets from? Luckily there is a MSBuild property $(NuGetPackageRoot) that points to that path, sooooooo:

<Import Project="$(NuGetPackageRoot)\specflow\2.2.1\tools\TechTalk.Specflow.targets" />

whoop, project loads again! Let's build!

Hmpf. No tests in the test explorer, NCrunch shows nothing... bugger.

Problem #2:
Code behind files still not being generated. Why didn't it work?
A lot of digging turned up a StackOverflow post that explained why the BeforeBuild wasn't called. TL;DR: in VS2017 the build system uses sensible defaults to reduce the noise in .csproj files. That has the unfortunate side effect that BeforeBuild isn't called, which Specflow depends on.

Cool, applied the import magicking done, let's build:

TechTalk.Specflow.targets(47,5): error : Object reference not set to an instance of an object.

$^$%^#@$@#$^$^ not again!

This one took some digging in the Specflow sources. This little bit of source code spelunking led me to MsBuildProjectReader.cs and the aptly named method IsNewProjectSystem(). The method checks if the root node of the .csproj file contains an Sdk property, which new style projects do!
Oh but I just removed that to fix the BeforeBuild problem.... shit.

I wonder if.... ok yeah, adding Sdk="" still loads the project, let's see if it builds....

TechTalk.Specflow.targets(47,5): error : Object reference not set to an instance of an object.

(╯°□°）╯︵ ┻━┻

More source code spelunking later, it turns out that we need a specflow.json in the project because for VS2017 style projects Specflow is also changing to JSON instead of app.config (see here)

Adding a specflow.json with just {} as it's contents....

TechTalk.Specflow.targets(47,5): error : Object reference not set to an instance of an object.

Double (╯°□°）╯︵ ┻━┻

Ok, contents matter. Digging into JsonConfigurationLoader I created the following specflow.json:

{
	"SpecFlow": {
		"UnitTestProvider": {
			"Name": "xUnit"
		}
	}
}

Hit build.....

BOOM! A wild MyFeature.feature.cs appears!

NCrunch suddenly turns green, TestExplorer shows a test, life is good!

Recap

Pulling this all together, you'll need a .csproj that looks like this:

<Project Sdk="">
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />

  <PropertyGroup>
    <TargetFramework>net462</TargetFramework>
  </PropertyGroup>
  
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
  
  <Import Project="$(NuGetPackageRoot)\specflow\2.2.1\tools\TechTalk.Specflow.targets" />
</Project>

and a specflow.json like this:

{
	"SpecFlow": {
		"UnitTestProvider": {
			"Name": "xUnit"
		}
	}
}

You will get the code behinds to be generated on-build. It's a far cry from the generate-on-save behaviour of the add-in, but at least it gets us going.

When you use the official Splunk Docker container the default configuration is that the UI shows time in the UTC time zone. This can be quite confusing if you are not actually in that particular time zone (or deal with daylight saving time....)

Fortunately Splunk allows you to select the time zone you want in the UI (it's in the top right corner, click "Account settings"). But if you frequently recycle the container this gets a bit tedious and it's something you'll easily forget.

As it turns out, you can set the default for all users via a configuration file that acts as a template for all users. The only thing that is needed is to create a user-prefs.conf file in the directory /opt/splunk/etc/system/local. (Found this solution here). The file looks like this:

[general]
eai_app_only = False
eai_results_per_page = 25
tz = Canada/Alberta

In this example I've configured the default UI time zone to be Alberta in Canada (UTC-06:00). You can simply set the tz = to the time zone you need.

Putting this together in a docker-compose.yml it will look like this:

version: "3.2"

volumes:
  opt_splunk_etc:
  opt_splunk_var:

services: 
  splunk:
    hostname: splunkenterprise
    image: splunk/splunk:latest
    environment:
      SPLUNK_START_ARGS: --accept-license
      SPLUNK_ENABLE_LISTEN: 9997
      SPLUNK_ADD: tcp 1514
    volumes:
      - type: volume
        source: opt_splunk_etc
        target: /opt/splunk/etc
      - type: volume
        source: opt_splunk_var
        target: /opt/splunk/var
      - type: bind
        source: ./user-prefs.conf
        target: /opt/splunk/etc/system/local/user-prefs.conf
        read-only: true
    ports:
      - "8000:8000"
      - "9997:9997"
      - "8088:8088"
      - "1514:1514"

The interesting bit here is the bind volume that maps the user-prefs.conf file into the right location inside the container.

Easy as.

Today I've had the interesting challenge to help Ester trying to figure out why a Specflow test wasn't behaving as it should. Of course the test ran fine in the CI pipeline but locally, no joy.

Some background

The application that this test is for generates a file from data in a database, which it then uploads to a FTP server. Easy as.

The test we were looking at basically does the following:

1. Clear the files from the FTP server
2. Set up test data in the database
3. Run the application
4. Verify that the file has been uploaded to the FTP server

Because this test uses an FTP server, Ester already set up a local FileZilla server and configured the test and application to point to that server. So far, so good.

However the test failed at step 1...

Time to investigate!

The start of our journey

Okay, so what failed exactly? The method that clears the files from the FTP server is nothing special, it just lists the files in the directory (which we get from settings) and then deletes them one by one.

The error we saw was that the file we attempted to delete didn't exist. Which is odd as we just got that file from a list (NLST) operation. Looking at the path supplied to the delete command it looked like this:
ftp://localhost/workdir/workdir/somefile.dat apparently the directory workdir was included twice. How did that happen? We certainly didn't put it in the format string twice.

Apparently FileZilla includes the directory name in the result of a NLST operation. So if you do NLST workdir you get workdir/file1, workdir/file2 etc instead of file1, file2. According to djb NLST should return an abbreviated list whereas LIST returns the full path for every file.

So our first problem is a FTP server with weird behavior... nice!

Enter Docker!

Okay, so FileZilla doesn't work for us, let's pick another FTP server. Luckily Nathan has already poked around and created a Docker container ready to use with pure-ftpd. After copying the necessary docker-compose.yml and DOCKERFILE we were good to go:

C:\src\application> docker-compose up
Traceback (most recent call last):
...
...
  File "site-packages\docker\transport\npipesocket.py", line 49, in connect
pywintypes.error: (2, 'WaitNamedPipe', 'The system cannot find the file specified.')
docker-compose returned -1

Not very encouraging.... what the bleep is WaitNamedPipe anyway?

Some Googling only revealed posts with this issue from at least a year ago. Probably updating Docker couldn't hurt.

Oh how wrong I was...

After downloading the latest Docker version and running the installer, it told me: "Hahaha! I only work on Windows 10!" Great. Okay so I should've read the site more carefully and downloaded the Docker Toolbox instead because that supports Windows 7.

One reboot later...

Error: Unable to connect to system D-Bus (2/3): D-Bus not installed

Shit.

Then I saw another message that the Virtual Box service could not be found. Interesting... I opened up Virtual Box to see if I could start the Docker VM, that didn't work. Also it told me there was an update available. So on a hunch I updated Virtual Box, thinking that re-installing probably would correct the situation.

Another reboot later...

DEVBOX ~
$ 

Yay! It works, Docker is up!

DEVBOX ~ 
$ docker-compose up
...
...
Attaching to pureftpd-test...

Awww yis!
By now, everything was working and I was able to connect to the FTP server successfully via IP address 192.168.155.165 (<-- remember this).

Fixing them tests

Okay so now we had a working environment, correctly configured test suite and application to point to the FTP server. Time to run the test again and see if it works.

Hoo-rah!! Test cleanup worked, setup worked, running the application worked. Only the assertion failed... bummer!

Let's see if the file got created on the FTP server... nope. Weird. I thought we had everything working, right? Double checking the configuration it certainly looks that way. Time to debug, launch the application from Visual Studio directly and see if it is behaving weird.

Stepping through the file upload I noticed something weird:

PASV
227 Entering Passive Mode (127,0,0,1,216,3004)

127.0.0.1? But wait a minute, we're connecting to 192.168.155.165 aren't we?

No wonder the upload fails. The FTP server is telling the application to connect to localhost but nothing is listening there. What's going on?

It turns out PASV instructs the FTP server to tell the client the port and the IP address to connect to. But because the FTP server thinks it's running on localhost, it will tell that to the client causing a mismatch. Luckily there is RFC-2428 that introduces the EPSV verb which only returns the port number to connect to.

So what causes this address confusion in the first place? Docker maps ports to localhost right? Yes, it does. However because of how Docker works on Windows 7 this doesn't quite work all the way.
Normally a Docker container runs directly on the host OS, which makes mapping the ports and IPs straightforward. 127.0.0.1 in the container is mapped to 127.0.0.1 on the host.
However because Docker on Windows 7 runs inside a VirtualBox Linux VM the host is not Windows but the Linux VM! Also VirtualBox doesn't map on 127.0.0.1 but real IP addresses the mismatch occurs.
This image shows the difference between Docker on Windows 7 vs Windows 10:

Luckily the PASV vs EPSV issue is easily fixable, we just changed:
ftpClient.DataConnectionType = FtpDataConnectionType.PASV;
to:
ftpClient.DataConnectionType = FtpDataConnectionType.EPSV;

FTP upload: solved!

Fixing them tests (2)

Unfortunately, the test still didn't pass. The file we expected still didn't exist on the FTP server... or does it?

Looking at the code for the assertion it was using a FileHelper class. Wait a minute, FileHelper? We're using a FTP server right?

So it turns out that the test was set up to expect the FTP location to be mapped to a local folder which then gets inspected. Major assumption there! But "no worries" (as our Aussie colleagues would say), we can just map the appropriate volume in the Docker container!

Yes, well, no.

Because Docker on Windows 7 is something of a hacktastic solution, mapping volumes actually doesn't quite work. We've specified the folder in the docker-compose.yml but it kept turning up as not mapped in the Docker container. Apparently we are out of luck here until we switch to Windows 10 (not something I'll do to satisfy a test though).
Update: Although VirtualBox isn't used anymore, the IP address jumble hasn't gone away. (Thanks Pat)

Changing to a different tack: why are we looking at the local file system anyway? Aren't we interested in whether the file exists on the FTP server?

The easy solution was just to use the FtpHelper instead of the FileHelper and hey presto, test passes!

Lessons learned

I think the most important thing to learn from investigating this test failure is to first try and understand what the test attempts to do and, for me personally, first to draw a picture of what is going on. Something like this:

would have helped a lot in figuring out which part(s) of the test were failing and identify the assertion that was checking the local directory instead of the FTP server.

Another important point is that you cannot depend on the behavior of 3rd party systems. Something that became painfully clear with FileZilla vs pure-ftpd for example. I think we need to have a look at the behavior of our 3rd party and their FTP server to ensure we can reliably replicate it in our test environments.

Also where Docker was involved, there is some (arcane) knowledge you need to have when attempting to troubleshoot issues. I feel that the "just use Docker, you're problems will be solved" attitude doesn't always hold up. Docker is a piece of software you'll need to grok in order to use it effectively. I'll definitely be looking at organizing some training/workshop around this in the next few weeks.

Conclusion

Test that shit. Really, do it.

All right, a bit of sleep and a lot of coffee, NDC day 2 is about to kick off so I've got a little bit of time to continue with the recap of day 1.

When feature flags go bad

This talk by Edith Harbaugh (@Edith_H) followed nicely on the one from Sam Newman and continued his story on feature toggles.

Feature toggles are a way to enable (or disable!) features in a deployed system (or application, or service) in a way that is non-disruptive to the usage of the application. This immediately brings us to the first key point about feature toggles: if you can only toggle at deployment time, it's configuration! Feature toggles are meant to be used on a running system without incurring downtime.

A nice example Edith shared with us is one from PagerDuty, they have certain feature flags around features that improve performance. But they disable them before they go to lunch. Why? Well everybody wants to have lunch in peace and quiet right? This shows us that adopting feature toggles allows you to have control over your live systems.

Although we might think of using feature toggles only as a on-or-off scenario, you can use this mechanism to give some users early access to certain features. Or even make sure some people don't get access (like your competitors!). A feature toggle itself is just a toggle, but the strength of this pattern is that it allows you to use a lot of information about a user to flip a toggle at run-time. Imagine only enabling a feature for people who visit NDC for the first time, are taller than 6 ft. and haven't shaved in a week.... Well you get the point.

What's really important when implementing feature toggles is that they should always be short lived. There is nothing more dangerous than a feature toggle that remains and is suddenly flipped. Ask the people at Knight Capital... oh wait, you can't because they went bankrupt¹. When you introduce a feature toggle, you introduce technical debt (the old feature). Be sure to remove them as quickly as possible.

AWS Serverless with .Net core

One of the things I like so much about conferences is that you get to meet the people building the frameworks and tools you are using every day. Norm Johanson is part of the team that builds the AWS tooling for the .Net platform.

The focus of this talk was demonstrating how you can use C# and dotnet core to build functions that run on AWS Lambda. Lambda is the serverless computing platform built by Amazon and for a long time it's only been possible to run NodeJS workloads in there. However since November 2016 it is possible to use dotnet core, so now we can leverage C# to build functions and use the tools we love to do it.

The AWS tooling is supported in VS 2017 (via the extension gallery), VS 2015 (as downloadable package from Amazon) and the dotnet core CLI. The beauty is that the Visual Studio extensions use the dotnet core CLI to do all the work. This also makes it super easy to integrate this in CI pipelines, no Visual Studio install needed or custom PowerShelling involved. The extensions also give you project templates (in both VS and dotnet core CLI) that will help you bootstrap a Lambda function project.

In his demo Norm showed that you can easily deploy the functions to AWS from within Visual Studio. However he was quick to point out that this is nice for getting to know Lambda but you should actually leverage tools like CloudFormation to manage your deployments. AWS does provide a number of ready to use templates (or stacks as they call it) which makes setting that up a lot easier.

When building functions with dotnet core it is important to realise that only netcoreapp1.0 is currently supported because Amazon only wants long-term support platforms. The expectation is that when dotnet core 2.0 is released it will become available relatively soon.

To take this even further, it is also possible to run full ASP.Net Core apps in AWS Lambda! Using API gateway to expose the app you can run a webapp (or WebAPI) completely serverless. Be aware though that all static content is now also being processed by your lambdas so you might want to route those calls directly to S3 (pro-tip!)

Check out the video Q&A with Norm about Lambda here

Recap

So, after day 1 we can already say that the session agenda at NDC is full of good stuff! We've gone from cowboys, to authorization, toggled our way through features and merge conflicts to running applications in serverless environments. And this was just day 1!
What's really great is that everybody here is so relaxed, I've met some cool people and have been able to talk to a bunch of the speakers. My cunning plan² is also starting to work out and I've recorded a bunch of video Q&A's with the speakers. (Which you can find here)

I don't think I've ever seen a conference open this way. Everybody gathered on the exhibition floor around a stage in the center with a massive screen hanging above it with a countdown timer. When the timer ran out we could hear the intro of the Dire Straits song "Money for nothing", what followed was a very NDC special version of the song! The lyrics were changed to introduce the speakers, really cool!

Once the song was done and the intro video of the NDC organizers was finished it was time for...

The Keynote

And then there was a cowboy on the stage... A what?? Yes, a cowboy. Well he sure looked like one but it was time for Dylan Beattie to kick off NDC Oslo.

What better way to start than asking if there are any questions? Not that he forgot the rest of his presentation and skipped to the end but rather to set the stage (ha ha, pun intended) for what he was going to talk about.

Because if we don't ask questions, we never start figuring anything out and we grind to a halt. Dylan talked a lot about the what drives us as human beings (and us as developers even more) to go forth and find out "hey, can I eat that?"

From there he took us through the process of trying to answer a question. For example, "What is the capital of Norway?" Depending on the context you're asking it in, it can either be Oslo, a ridiculously large number or the letter 'N'.

In the process of answering questions us humans have, along the way, invented some pretty amazing stuff. From fire, books and calculus to boats, rockets and GPS. Toss in the World Wide Web and if you look really close, it was all done in support of answering questions. (To which we all know the ultimate answer is 42)

Dylan had one final piece of advice for us: "Use flatscreens". In the style of Baz Lurhmann's "Wear sunscreen" he told us to go forth, ask why a lot and enjoy NDC Oslo!

Implementing authentication and authorization

In this session by Dominick Baier & Brock Allen the focus was more on authorization than authentication. It turns out that authentication is a relatively simple problem in this space, sure getting it right is a challenge but there are only so many variations.

The trick however is how to do authorization.

From their experiences (and to be honest my own as well), authorization is a very application specific problem, and although there are some reusable mechanisms there are always enough things that make that approach not work.

What happens a lot is that tokens generated by identity providers get abused to store permissions and claims. This leads to very large tokens and put the responsibility of working out what claim means what with the identity provider instead of the application (and it's business domain). For example Delete may mean something completely different in service A than service B, but now you can't differentiate other than to push more permissions in the token.

Instead we should be looking at mapping identity to permissions in the context of the service and it's business domain. That means that this mapping should happen not in the identity provider but in an authorization provider. That provider can be a completely separate service (even centralised for many services) or even an in-memory one that lives inside of the context of the service itself.

Key is that mapping identity to permissions is a domain specific problem and should be solved there.

Serverless: Reality or BS

Serverless systems are one of the main themes here at NDC, and this talk tries to shed some light on the different architectural styles and platforms to build serverless systems. Lynn Langit is an independent cloud architect who has experience building these systems on the well known platforms of Amazon, Google, and Microsoft.

Serverless systems started when AWS Lambda was introduced, or rather when S3 became available. The simplest form of serverless is a static site running from an S3 bucket! Let that sink in for a minute. Because AWS was the first provider to jump in this space they currently have the platform that is furthest ahead compared to Google or Azure. However both of them are catching up rapidly and in case of Azure with some nice features like Logic Functions.

Of course there are some caveats to building serverless platforms. The success largely depends on the type of workload you have. An example is the Australian census. Built as a government IT project, it was way too expensive, delivered too late and when it went live it crashed because millions of people tried to use it at the same time. Two students rebuilt it as part of a hackathon project using AWS lambda and simulated the load. It cost three days and $500 to build and run!

So if you have peak load or run processes on a schedule, serverless computing can help you a long way. Instead of paying for servers sitting idle, you pay for actual usage and don't worry about managing servers (with the additional cost that is involved there)

Also you should be worrying about vendor lock-in. These vendors are good at giving you freebies to get you on board, but once there and you are using many of their services it can become really hard to switch providers. Plan accordingly for this when designing your serverless architecture. Have an exit strategy and consider whether using micro or nano instances would also be a possibility.

Feature branches and toggles

This talk by Sam Newman is about whether you should use feature branches or practice trunk-based development with feature toggles. Hint: it's the latter.

Some good examples of the pain of merging lead us to the research done by the people behind "The state of DevOps" reports. They concluded that practicing trunk-based development leads to faster feature delivery, less problems with integrating changes from many developers and generally better software.

A good contrast was made about using pull requests in open source projects. These projects deal with untrusted committers that submit changes and with the pull request mechanism you as maintainer get much better tools to manage that. However most teams consist of developers who are trusted committers, they're your team mates! Therefore it makes more sense to not use PRs but instead make sure that everybody commits small changes frequently and thereby reduce the pain of integration.

Stay tuned for the rest of day 1, soon to follow!

After arriving in Oslo on Sunday and doing the touristy walkabout through the city, Monday saw the start of the Designing Microservices workshop by Sam Newman. Now personally I've been waiting for a good opportunity to jump on the microservices bandwagon for some time now and joining this workshop is definitely preparation to doing just that.

Of course with microservices being in the opinion of many, yours truly included, a bandwagon thing it doesn't hurt to actually try and understand what they actually are.

Now before I continue, these are some insights I've gained from the workshop but I'm not Sam and I highly recommend doing this workshop yourself if you have the opportunity. He's way better at explaining this (and arguably funnier)

With that out of the way, let's get go!

So then, should you build microservices?

Well the only answer to that is: It depends. (I have it on good authority that this is what a consultant would answer^[1]). But in truth, it really does. In the two days of the workshop we kept coming back to this question and I'll try and explain why it depends.

The micro of microservices is a bit of a misnomer because it implies a tiny tiny service. How tiny? Nobody really knows. But no fear! There is a better definition. Sam gave us the following:

Small Independently Deployable services that work together, modelled around a business domain

This reminded me a lot of Uncle Bob's definition of single responsibility, of which he says to consider it as single reason for change ^[2]. One of the key points is that these services can be deployed without requiring another service or application to be deployed in lock-step with it. In a way, a monolithic system is in fact independently deployable, it's one unit of deployment.

But again, the definition mentions small. But what is small? When is a service small enough? I think that answer could either be 42^[3] or that you need to define that for yourself. Size largely depends on what works for your team and organisation.

It is more important that the services you create are designed around business domains. This helps to isolate functionality for that domain inside a single service (which can use many others!). Also the responsibility for changing that service should be with one team and not many people making changes (and thus sharing responsibility, which doesn't work[4]).

When you start figuring out which parts of your application can be isolated and turned into microservices it is a good idea to read up on Domain Driven Design. Designing microservices is largely identifying bounded contexts and defining the models you expose to your consumers.

Defining the boundaries of your services will be the hardest part and what you need to keep in mind is the cost of change. Changing the shared model of your service when it's running in production is pretty high cost, moving a sticky note on a wall in your design phase is pretty darn cheap. Also keep in mind that when your system is still new it is perfectly fine to build as a modular monolith and split out into services later when the domains and contexts are more clear and less likely to change.

But wait! You are at a technical conference! What about the technology?

So far I've only talked about the design and businessy stuff but obviously there is more to it than that. Truth of the matter is, all the technology to build microservices already exists and has existed for a number of years already. It's more a matter of using it in the right way and applying practices rigorously.

Now you don't necessarily need to use REST to build microservices, XML-RPC or CORBA works just as well (and this is 1990's era technology!), however HTTP (and REST) give you advantages in providing a useful protocol for communication between services. Just be aware that for some circumstances it might make more sense to communicate over UDP because you need very high volume and losing some data is fine, keep your options open.

With the introduction of more services in your landscape you will need to focus more on how you deploy your services, manually deploying doesn't cut it anymore. Invest in a proper Continuous Integration / Continuous Delivery pipeline in which you can integrate automated testing. This helps reduce the time you need to spend to get services into production and provide more confidence that what is deployed actually works.

Once in deployed (to production), keeping track of how your landscape is performing becomes challenging. Like with deployment, manually checking status is madness. Manual alerting is even worse (getting that call from a user is a poor alerting mechanism!). You should collect logs from your various services in a centralised location like ELK or Splunk so you can diagnose failures better as looking for log files across many services becomes too painful.

Because in a microservices landscape an action performed by a user can cross multiple service boundaries it can be challenging to stitch together log entries when something goes wrong. To improve this it is a good idea to implement correlation id's and attach this to your log entries. It makes it that much easier to track calls across multiple services. Tools like Zipkin can leverage this to help track down which service is taking a long time for example.

One bit of advice Sam gave us, and I agree on this one, build this stuff in from the start! It is so much more difficult to do it afterwards (cost of change!). When you start building the first services and deploying them into production you will find out what works for your organisation and what you actualy need. Use that knowledge in the next services you build so you improve your overall platform while you are moving forward instead of trying to retrofit logging and monitoring onto existing services.

Another important thing to consider is using Infrastructure-as-Code to help you automate the deployment of your infrastructure. It also gives you more confidence that what is running in production is actually what you intended instead of a collection of special snowflakes that nobody can remember how they were built. With this approach you can also more easily destroy and rebuild environments which is really useful in for example testing scenarios or disaster recovery (to name a few, there are many more).

Of course there is a lot more to talk about on the technology side but I'll save that for the next blog post as this one got long already. More stuff from NDC to come, also I hope to do a lot more video Q-and-A sessions with some of the speakers so watch this space.

At $WORKPLACE we're currently busy porting over our XAML builds to the TFS 2015 build system and we wanted to be able to queue another build when the current build completes.
Fortunately for us the REST API makes that a trivial exercise and that particular bit was running fairly quickly.

However...

Because we have a build that will automatically get a version number we needed to pass that version number on to the next build in line. "Aha!", I hear you think, "You can use variables for that!" and yes, you are correct. TFS 2015 allows you, just as in XAML builds, to specify additional variables you can set at queue time.

Alas, the REST API documentation doesn't specify how to do that.

But after a bit of poking around in the JSON data that is returned when you retrieve information about a build I noticed the parameters property which seemed to contain the variables used at build time.

So when you want to pass variables when queueing a build through the REST API you can use something like this:

{
  "definition": {
    "id": 25
  },
  "sourceBranch": "refs/heads/master",
  "parameters": "{ \"version\": \"1.2.3.4\" }"
}

I'm willing to bet the following is going to feel very familiar to you:

Here you are at Awesome Company, happily working on the SuperProjectOfTheMonth and about every 15 minutes you run into this little thing that's costing you just a microsecond of brain activity to notice before you get on with whatever it was you're doing.

Now it'll only cost you a microsecond, I mean, I can't probably even blink that fast, but if that happens every 15 minutes I wouldn't get anything done, let alone remember what I was doing before.

Luckily for us, we work in IT and we can DoStuff^TM

At this point I'd like to make another bet:

You get all excited about the massive improvement you're going to make when suddenly a wild impediment appears and says "No we can't do this because reasons"

If you're Dutch (or have lived here for some time) you may have come across a tv-commercial that goes like this:

Two little kids are with their mother at a swimming pool and they seem to have lost their purple inflatable crocodile. When they go to the reception desk and see that it's propped against the wall behind the counter they ask if they can have it back.
The man behind the counter responds that they will need to fill in a form to which the mother replies: "But it's right there, that's the crocodile"
The man won't be budged and replies: "But you still will need to fill in the form" at which point the mother walks away with her children looking completely bewildered.

Now this is only a tv-commercial and may seem a bit "heh, funny" but it provides me with a nice example of what I see happening quite frequently.

Remember the bet I made earlier? I think that not only the fact that "reasons" pop up can be demotivating, there is a more fundamental problem here. Given previous experiences it may very well be that you won't even bother to change or do anything because people will say no anyway!

I feel that this way of thinking gradually gets you into a situation where you muddle on and become oblivious to the things you could change to improve your life: The purple crocodile has become real!

But fear not!

While it can feel like trying to swim up a waterfall you should make an effort to change. "But won't that be tedious?" I can hear you say. Yes it most likely will, but I'd like to offer some advice. When someone says No!, there is probably a reason behind it and it may be that the nay-sayer thinks that you are aware of these reasons. As a bonus they may even think you're stupid because you're too thick to understand them.
So my advice is: Find out what those reasons are.

Because when you do you can convince people why change is necessary. It can also help you clarity the need for change and you may learn something new as well. A new perspective never hurts!

And sometimes it just pays off to just do what you need and show people how you improved. And sometimes it's just easier to ask for forgiveness instead of permission.

I hope that you now no longer believe in purple crocodiles or if you do, just to give it a nice name and show how it, too, can change.

At the end of 2014 I had been toying with the idea to organize a conference for the developer group I work with. At the time I had two basic goals:

• Provide a platform for the devs to have a talk about the cool stuff they are working on
• Get all the devs together (again)

Now the second bullet might seem a bit odd to put there as us nerds have the unique ability to recognize others on sight, but we've had so many projects move through the building and people moving in and out of teams that it got a bit difficult to figure out who works where. Also knowing your fellow devs makes it a lot easier to solve problems by asking someone you know, to find a sparring partner or rubber duck.

As far as the first goal is concerned, I believe that in our dev group there are a lot of smart people that do interesting stuff on their projects, but also have fiddled with new libraries or tools that could help us build better software. At the regular dev meetings it's always a bit of a challenge to find people who want to talk about these topics, so how was I going to get them to speak at a conference? But more on that later.

The experience I usually have when I get a new idea is that I get really enthusiastic and full of energy you know, that conquer-the-world feeling. Having seen many ideas die a quick and messy death for various reasons I really wanted to avoid that this time around. So one of the first things I did was to pitch my idea to other devs and see if it was something people would like to see, a bit of market research if you will.

I think that from pitching my idea I learned the first lesson: get feedback early

Organizing something all by yourself can be very rewarding if you pull it off but it also means that you need to be careful that you don't suffer from tunnel vision and create something that you might find useful but others might not. One suggestion I got was to join the GoForIt days, our innovation challenge and to gather a group of people to help me hammer out the details of the conference.

Going for it

During the 24 hours of innovation (or: GoForIt day) I worked with a bunch of people (hi Jim, Michael & Henk!) to determine what the conference would look like, how it should be called, how many sessions etc. We had a lot of discussion about how large we should make it, just our dev group? Invite all devs? Do we invite only devs or should we try to get testers (QA engineers) and BA's to join us as well?

I think these questions led to lesson no 2: keep it real.

At first I wanted to aim big: get as many people as we can. 100+ attendees? Yes, please! However as this was the first time this conference is held we decided to manage our own expectations and scaled it down a bit and figured that if we could get about 60 people to show up that would be great.

But getting people to show up, that means having great sessions and that means getting speakers for those sessions. Which leads us to...

Volunteering Inviting speakers

Within any organization there are always the usual suspects who will show up to meet-ups and tech sessions and a number of those will jump to any chance to do a demo or presentation about something. So to make sure I got enough speakers to fill all the slots I had a talk to a couple of likely candidates and pitched the conference idea. Luckily for me the reactions I got were all positive so filling up the slots actually went pretty smoothly...

... until I got a cancellation.

Cue third lesson: always have a backup.

It's a sensible idea to have backups of your data but I think this also applies to basically anything else. Always have a plan B. Think about what could go wrong with plan A, it helps you identify problems that may occur later and to come up with alternatives for when that happens.
I'm not saying that you always (all right, yes I did say always) have to have a plan B but what helped me is that when the shit hits the fan I can think: all right, bad things happen but let's move on.

Luckily I was able to quickly find another speaker who was able to put together a presentation on very short notice.

Timing (and getting people to join)

One thing I haven't talked about so far is the actual set-up of the conference. During the GoForIt we had quite some discussion on how long the individual sessions should be and what the overall conference schedule would be like.

We decided to plan the conference on an afternoon instead of a full day as we thought it more likely for people to show up if it doesn't take up their whole day. We often face quite some pressure within projects that make it difficult for people to join these types of events.

Deciding on the length of a session is always a bit difficult, as a speaker it's easy to fill an hour or maybe even two on your particular topic and I've frequently found myself needing to trim a session to fit a time slot. For the conference we wanted to have enough time for both the presentation and a discussion, we wanted to make sure that the sessions were not just one-way sending of information. We thought that by making every time slot 1 hour that would work out nicely, 45 minutes of presentation and 15 minutes discussion.

Did that work? Yes, I think so. From my own session and the others that I've seen there was plenty time available for people to ask questions which is what we wanted. I did get feedback from a couple of the speakers that filling a 1 hour slot was too much and that it might prevent other people from hosting a session. It's definitely something we need to think on.

Having said that, one problem however was that we tried to cram too many sessions in one afternoon.

When we planned the conference we wanted to have enough sessions and I think we might have gone a bit overboard with that. The conference schedule ended up like this:

So I think the lesson here is: plan carefully!

It's easy with a project like this to focus too much on the content of the sessions because that is one of the things you obviously know and care about. But organizing an event is about so much more than just that. I remember speaking to someone from Microsoft a couple of years ago about the TechEd and TechDays events and one of the remarks was that they had to worry even about the temperature of the rooms! Wish I had remembered that particular conversation when I was doing the planning and not now when I'm writing this blog.

The thing we ran in to was that apparently Wednesday is a popular day to work at home or have a day off within our company. So it's likely we would have had more people join if we'd only have planned the conference on, say, a Thursday.

Also I think that most people just found 17:45 too late to stay around, starting earlier on the day would, in hindsight, have been better.

Communication, communication, communication

How do you announce an event like this? We figured that the newsletters that get e-mailed around every month would serve as a starting point, however we have seen that a lot of people just don't read them. To make sure we got enough exposure we decided to print posters and bring the event to everyones attention that way. Also we've got these nice tv-screens in our company restaurant on which we were able to put a promotional video.

The effect of all this? Not a whole lot.

It turned out that when I spoke to people before the conference a lot actually hadn't heard about it, despite our efforts at reaching as many people as we could. I think that we should have communicated the date and the general theme of the conference a lot sooner than the couple of weeks before we did.

I think that we also could have used the various team managers and lead developers more to promote the event and to encourage other people to join as well.

Conclusion

The question that kept running through my head during the conference and afterwards at the drinks was: can I call the conference a success?

I think I can safely say yes. We had great sessions, people actually turned up to watch them and the overall feel was great!

Would I do this again? Oh yes! It has been a great experience for me and I think I've learned a couple of valuable lessons in organizing the conference.

Thanks

All right, this isn't an award ceremony but I would like to thank a couple of people, Michael & Jim for the brainstorms and pulling this off, Henk for the reality checks, Wouter for being our fairy godfather and all the speakers for their enthusiasm.

Now that we've migrated to TFS 2013 we now have Team Rooms available for all the projects. TFS provides a number of default events such as work item state changes and builds that have completed. Because we use XL Deploy to handle our automated deployments I wanted to include those events to the team room as well.

XL Deploy has a nice REST API but unfortunately there is no way to register a hook when a deployment completes. Fortunately XL Deploy has a way to add rules that can execute extra steps at various points in a deployment. I've used this to add a step to the end of a deployment plan that uses a little Jython script to call a remote REST API whenever the deployment completes.

Adding a rule

XL Deploy stores it's rules under the $XLDEPLOYPATH/server/ext folder in the xl-rules.xml file and this is where we'll add the necessary configuration to run our script.
The rule definition looks like

 <rule name="DeploymentSucceeded" scope="post-plan">
    <steps>
        <jython>
          <order>99999</order>
          <description>Execute application deployed hook</description>
          <script-path>hooks/application-deployed.py</script-path>
          <jython-context>
            <application expression="true">context.deployedApplication.name</application>
            <environment expression="true">context.deployedApplication.environment.name</environment>
            <version expression="true">context.deployedApplication.version.name</version>
            <hooksDict expression="true">context.repository.read("Environments/Hooks")</hooksDict>
          </jython-context>
        </jython>
    </steps>
</rule>

I've set the rule scope to be post-plan to make sure that this step is only added after the validation and planning stages have completed. The rule adds a new jython step that instructs XL Deploy to call the script defined in script-path and is a relative path starting from $XLDEPLOYPATH/server/ext on the XL Deploy server.

To provide the script with some useful information you can define a number of parameters in the jython-context. These parameters will be made available as variables in the jython script. In this example I pass in the application, environment and version names. The hooksDict parameter is pointing to a dictionary under Environments/Hooks in the XL Deploy repository and it contains the URL of the REST API we want to call.

! Before you add this to your xl-rules.xml be sure to make a backup first!

Depending on the configuration of your XL Deploy instance you may need to restart the server before the rule is activated.

The Hooks dictionary

To be able to configure the URL that the script will be calling I've added a dictionary under the Environments folder in the XL Deploy repository. This dictionary contains the following settings:

The $application, $environment, $version and $username placeholders will be replaced by the Jython script later on. If you leave host empty the script won't run which makes it an easy way to disable the hook alltogether.

Creating the script

Now that we have the rule in place we need to act whenever the step is triggered. The script is actually fairly simple, it collects the variables, figures out the current username running the deployment and calls a REST service:

import httplib
from string import Template

if hooksDict != null and "host" in hooksDict.entries and "deployment-complete-url" in hooksDict.entries:
	username = context.task.username
	
	context.logOutput("Calling deployment-completed hook with: " + username + " deployed application " + application + " (" + version + ") to " + environment)
	
	s = Template(hooksDict.entries["deployment-complete-url"])
	url = s.substitute(application=application, version=version, environment=environment, username=username)

	conn = httplib.HTTPConnection(hooksDict.entries["host"])
	conn.request("GET", url)
	r1 = conn.getresponse()
	conn.close()

! Note that Jython is a Python derived language and is white-space sensitive.

In the script you can see that the deployment-complete-url is retrieved from the dictionary and the placeholders are replaced with the actual values. This allows a bit of flexibility when defining a REST service endpoint.

Creating the REST service

To demonstrate that this all works I put together a simple NodeJS app (this is pushing the term app) to receive the request that the Jython script makes. It will simply log the request URL to a file and return "OK":

var http = require("http");
var fs = require("fs");

var server = http.createServer(function(req, res) {
    fs.appendFile("/tmp/completed.log", req.url + "\n", function(err) {
        if(err) {
          res.writeHead(500, { "Content-Type": "text/plain" });
          res.end(err);
        }
        else {
            res.writeHead(200, { "Content-Type": "text/plain" });
            res.end("OK\r\n");
        }
    });

});

server.listen(8000, '127.0.0.1');

This service is started by running:

sauron@localhost> node restservice.js

Testing the deployment complete hook

To test the rule and the actual script I created the following in a local XL Deploy instance:

• Application: TestApp
• Package: TestPackage (without anything in it)
• Environment: Production

I also added the Hooks dictionary with the settings as described above.

On the deployment tab simply drag the package TestPackage in the left box and the environment Production in the right box and click Execute. Because there isn't actually anything in the package or the environment the deployment will start immediately and you should see sometthing like this:

And when the execution is complete you should see this:

If everything went OK then the file /tmp/completed.log (or wherever you've put it) should show:

/TestApp/Production/TestPackage/admin

Wrap up

While extending XL Deploy this way works and provides a nice and simple way to achieve what I wanted I'm not entirely convinced this is the best way to do this. It may be that a plugin is actually better suited and could be more powerfull but that is something I need to look at in the future.

In the meantime, this hook now allows me to post messages to the TFS team room whenever someone deploys an application with XL Deploy!

When looking to consume a WCF service in SQL Server Reporting Services (SSRS) I came across the XML datasource and a lot of articles showing me that while it is possible, it is a bit of a hassle with formatting the query and figuring out exactly which SOAP action to use and in what namespace.

Thats why I thought to have a go at creating a data source for SSRS that truly consumes a WCF service and would give me better metadata, like which fields are in the result set, which parameters the SOAP operation needs etc. So less of this:

<Query>
  <Method Name="MyMethodRequest" Namespace="http://tempuri.org/">
    <Parameters>
      <Parameter Name="Param1"><DefaultValue>ABC</DefaultValue></Param>
    </Parameters>
  </Method>
  <SoapAction>http://tempuri.org/MyService/MyMethod</SoapAction>
</Query>

but instead just:

MyMethod

And let the extension figure out the rest for you.

I dug around on MSDN and found a quite comprehensive article about writing a custom Data Provider Extension (DPE) that provides a walkthrough on how to create such an extension. It didn't take me all that long to create my own extension and have it generate WCF clients at runtime. Easy as pie you'd say.

Well.. not really. Deploying should be as simple as dropping the assembly in the correct location and editing the SSRS config file, but when I reloaded the web UI it didn't show my extension and the event log was riddled with errors. Yikes!

After a bit of digging it turns out that SSRS 2012 (the version I'm targeting) does not support the .Net 4.0 runtime but instead is based on .Net 2.0. Fortunately it did get an update so that .Net 3.5 is supported so we're good to go: WCF is supported in that version.

Having rebuilt the project against v3.5 I deployed the extension again but alas, the extension was still not visible. It took me a while to figure it out but I had switched the architecture from AnyCPU to x86 and that prevented the extension from loading. After I fixed that it worked like a charm!

I have put the sources for the extension up on GitHub so that other people looking to use WCF services in SSRS can use it as well. Have a look here for details on how to deploy and use the extension.