Dusted Codes

.NET Blazor

Sun, 19 Nov 2023 00:00:00 +0000

.NET Blazor has been touted as a revolutionary framework that allows .NET developers to build interactive web applications using C# instead of JavaScript. It's mainly aimed at ASP.NET Core developers who want to build SPA-like apps leveraging the .NET ecosystem and a wealth of existing libraries and tools available via NuGet. It's Microsoft's latest instalment in an attempt to gain traction amongst frontend developers. With the recent release of .NET 8 Microsoft announced even more improvements to Blazor, most notably introducing a new rendering mode called "Static Server-side Rendering (SSR)".

But what exactly is Blazor and how does it enable C# to work in the browser? More interestingly, how does it compare to traditional JavaScript based SPA frameworks with which it aims to compete with?

Blazor WASM

Blazor has been on a journey and in order to understand Blazor's current state one has to look at its evolution starting from the beginning. Launched in 2018, Blazor initially started as an experimental project, aiming to leverage WebAssembly to run C# directly in the browser, allowing developers to build SPAs using .NET. This idea was realized with Blazor WebAssembly, which allowed the .NET runtime to execute on the client.

Blazor WebAssembly, commonly abbreviated as Blazor WASM, offers the most SPA-like experience among all Blazor options. When a user first visits a Blazor WASM application, the browser downloads the .NET runtime along with the application's assemblies (lots of .dlls) and any other required content onto the user's browser. The downloaded runtime is a WebAssembly-based .NET runtime (essentially a .NET interpreter) which is executed inside the browser's WebAssembly engine. This runtime is responsible for executing the compiled C# code entirely in the browser.

Although Blazor WASM applications are primarily written in C#, they can still interoperate with JavaScript code. This allows the use of existing JavaScript libraries and access to browser APIs which are not directly exposed to WebAssembly.

While Blazor WASM has received plenty of initial praise and has been improved over time, it's also been met with key criticisms which often revolve around the following areas:

Initial load time:
The requirement to download the .NET runtime and application assemblies upon the first visit can result in a significant initial load time. This is even more evident in complex apps with large dependencies and especially over slow networks.
Performance:
Blazor WASM lags behind traditional JavaScript frameworks in terms of performance. The WebAssembly runtime is still generally slower than optimised JavaScript code for compute-intensive workloads.
Compatibility:
While WebAssembly is widely supported in modern browsers there may still be issues with older browsers or certain mobile devices which can limit the reach of a Blazor WASM application.
SEO challenges:
Beside the usual SEO challenges which all SPA frameworks come with, the additional longer load times and slower performance of Blazor WASM can negatively impact SEO rankings.
Complexities of interop with JavaScript:
While Blazor WASM allows for JavaScript interop, it can be cumbersome to use alongside complex JavaScript libraries or when there is a need for extensive interaction between C# and JavaScript functions. This complexity can lead to additional development overhead and potential performance bottlenecks. Unfortunately due several limitations the need for JavaScript interop is very common and therefore kind of undermines the whole premise of using Blazor in the first place.

Blazor Server

To counter act some of these critiques, Blazor Server was introduced a year after Blazor WebAssembly, enabling server-side C# code to handle UI updates over a SignalR connection. Unlike in Blazor WASM, the client-side UI is maintained by the server in a .NET Core application. After the initial request, a WebSocket connection is established between the client and the server using ASP.NET Core and SignalR.

When a user interacts with the UI, the event is sent over the SignalR connection to the server. The server processes the event and any UI updates are rendered on the server. The server then calculates the diff between the current and the new UI and sends it back to the client over the persistent SignalR connection. This process keeps the client and server UIs in sync. Since the UI logic runs on the server, the actual rendering logic as well as the .NET runtime doesn't need to be downloaded to the client, resulting in a much smaller download footprint, directly addressing one of the major criticisms of Blazor WASM.

However, while innovative in its approach, Blazor Server has several downsides of its own which need to be considered:

Latency:
Since every UI interaction is processed on the server and requires a round trip over the network, any latency can significantly affect the responsiveness of a Blazor Server app. This can be particularly problematic for users with poor network connections or those geographically distant from the server.
Scalability issues:
Each client connection with a Blazor Server app maintains an active SignalR connection (mostly via WebSockets) to the server. This can lead to scalability issues, as the server must manage and maintain state for potentially thousands of connections simultaneously.
Server resource usage:
Blazor Server apps are much more resource-intensive because the server maintains the state of the UI. This can lead to higher memory and CPU usage, especially as the number of connected clients increases.
Reliance on SignalR:
The entire operation of a Blazor Server app depends on the reliability of the SignalR connection. If the connection is disrupted, the app can't function. This reliance requires a robust infrastructure and potentially increases the complexity of deployment, especially in corporate environments with strict security requirements that may restrict WebSocket usage.
No offline support:
Unlike Blazor WebAssembly apps, Blazor Server requires a constant connection to the server. If the client's connection drops, the app stops working, and the current state can be lost. This makes Blazor Server unsuitable for environments where offline functionality is required.
ASP.NET Core Server requirement:
The reliance on SignalR also means that Blazor Server apps cannot be served from a Content Delivery Network (CDN) like other JavaScript SPA frameworks. Serverless deployments aren't possible and Blazor Server requires the deployment of a fully fledged ASP.NET Core server.

Blazor Static SSR

Despite Blazor's versatility, both the WASM and Server rendering modes suffer from serious drawbacks which make Blazor a difficult choice over traditional SPA frameworks, which by comparison don't share any of Blazor's problems and are architecturally simpler too.

Being aware of these challenges, Microsoft tackled some of the primary concerns of Blazor WASM and Server by rolling out Blazor Static SSR:

Blazor Static SSR, as shown in the diagram above, is a third rendering option which operates entirely independent of WASM or SignalR, instead leveraging an open HTTP connection to stream UI updates to the client. This approach, known as static site rendering, involves generating web pages server-side and transmitting the fully composed HTML to the client, where it then gets wired back into the DOM to function as a dynamic application.

During an initial page load, Blazor Static SSR behaves similarly to a traditional server-side application by delivering a complete HTML page to the user's browser. Additionally, it fetches a blazor.server.js script that establishes a long lived HTTP connection to an ASP.NET Core server. This connection is used to stream UI updates to the client. This architecture is more straightforward, much like a classic server-rendered website, yet it provides a dynamic, SPA-like experience by selectively updating portions of the DOM and therefore eliminating the need for full page reloads.

The benefits over Blazor WASM and Blazor Server are twofold:

Reduced load times:
There's no need for users to download the full .NET runtime and application files when visiting the website, and as they navigate through the site, complete page reloads are avoided.
Scalability:
No SignalR connection is required which greatly reduces the load on the server and removes many of the complexities around WebSocket connections.

Nonetheless, Blazor Static SSR is not an actual SPA framework in the traditional sense. It doesn't allow for rich interactivity beyond web forms and simple navigation. It also doesn't allow for real-time updates as there is no code running on the client after the initial page was loaded:

To combat this, Blazor starting with .NET 8 enables the mixing of different modes and introduces a fourth rendering option called Auto mode.

In order to add interactivity to a Blazor Static SSR website one has to go back to creating either Blazor WASM or Blazor Server components. The auto rendering option aims to counter the main issues of Blazor WASM's slow load times and Blazor Server's requirement for a SignalR connection by using both rendering modes at different times:

A Blazor component operating in Auto-mode starts off by establishing a SignalR connection to enable immediate interactivity and bypass extended load times. Concurrently, it discreetly fetches the .NET runtime and all necessary dependencies to function as a Blazor WASM application. For later visits, Blazor transitions from the Server to the WASM version, maintaining SPA responsiveness without further dependence on the SignalR connection.

It's a fascinating approach which undoubtedly doesn't lack creativity or ambition. Even so, Blazor Static SSR incorporated with interactive components poses some old and new challenges too:

No interactivity without WASM or SignalR:
The biggest drawback of Blazor Static SSR is that it still relies on Blazor WASM or SignalR to become an interactive framework, which means it inherits not just one, but all of the many unresolved downsides when running in Auto-mode.
Increased complexity:
Combining three different rendering modes adds a lot of complexity on the server and presents a steep learning curve for developers who must comprehend and manage those complexities effectively.
No serverless deployments:
Deployments from a CDN are still not possible due to the reliance on ASP.NET Core.
No offline support:
Blazor Static SSR minimises full page reloads but still requires an active connection to stream updates to the UI.
Caching challenges:
While static content is easily cacheable, dynamic content that changes frequently can be challenging to cache effectively, potentially missing out on valuable performance optimisations.

Having said that, Blazor Static SSR also comes with a few benefits when it's not mixed with WASM or Server together:

SEO Friendliness:
Since SSR applications pre-load all the content on the server and send it to the client as HTML, they are inherently SEO-friendly. This allows search engines to crawl and index the content more efficiently.
Fast initial load:
Blazor Static SSR can provide faster initial page loads compared to SPAs. This is because the HTML is ready to be rendered by the browser as soon as it's received, without waiting for client-side JavaScript to render the content.
Stability across browsers:
SSR applications often have more consistent behavior across different browsers since they don't rely on client-side rendering, which can sometimes be unpredictable due to browser-specific JavaScript quirks.

Blazor vs. traditional JavaScript SPAs

Overall Blazor is a remarkable achievement with buckets of originality and technical finesse, however with the exception of Blazor WASM, Blazor Server and Blazor Static SSR behave quite differently to traditional SPAs.

Neither Blazor Server or Blazor Static SSR load all the necessary HTML, JavaScript and CSS upfront. They have a hard dependency on an ASP.NET Core backend, can't be hosted serverless and require a constant connection to a server. The frontend is not separated from the backend and data is not fetched using APIs. Typical SPAs maintain state on the client side. The user's interactions with the application can change the state, and the UI updates accordingly without a server round-trip. Since SPAs don't require page reloads for content updates, they can offer a smoother and faster user experience that is similar to desktop applications. With conventional SPAs the same code can often be shared between web and mobile apps, another advantage over Blazor Server or Static SSR. The clean separation between the frontend and the backend makes the overall mental model simpler and allows to efficiently split the disciplines between different teams.

Blazor WASM vs. JavaScript SPAs

Blazor WASM stands out as the only rendering option which fully aligns with the ethos of a conventional SPA. Unfortunately the heavy nature of having to run the .NET Runtime over WebAssembly puts it at a significant disadvantage over comparable JavaScript frameworks.

Blazor Server vs. JavaScript SPAs

While Blazor Server is technically intriguing, offering a unique approach to web development, it paradoxically combines the limitations of both, a Single-Page Application and a server-intensive architecture, at the same time. To some extent Blazor Server represents a "worst of both worlds" scenario. Personally it's my least favourite option and I can't see any future in this design.

Blazor Static SSR vs. JavaScript SPAs

Blazor Static SSR deviates the most from the paradigm of a SPA. Apart from being placed under the Blazor brand it diverges significantly from the framework's initial architecture. Ironically this is where its strengths lie as well. Given that SPAs are inherently accompanied by their own set of challenges, the necessity for a SPA must be well-justified, or otherwise opting for a server-rendered application can be a more straightforward and preferable solution most of the times.

In my view, Blazor Static SSR is a compelling option that deserves to be its own framework, enabling .NET developers to enrich the functionality of everyday ASP.NET Core.

A word of caution

Would I opt for Blazor today? To be candid, probably not. While I maintain a hopeful stance on Blazor, I must remain truthful to myself. I've never been the person who blindly champions every Microsoft technology without critical thought. The truth is, currently Blazor is evolving into an unwieldy beast. In spite of its four rendering modes, intricate layers of complexity, and clever technical fixes, it still falls short when compared to established SPAs. This situation leads me to question the longevity of Microsoft's commitment and how long Blazor will be around. The parallels with Silverlight are hard to ignore, and without the .NET team delivering a technically sound framework, I find it hard to envision widespread adoption beyond a comparatively small group of dedicated C# enthusiasts who will accept any insanity over the thought of using JS.

An untapped opportunity?

As I reach the end of this blog post I want to finish on a positive note. I dare to say it, but could C# learn another thing from F#? Thanks to Fable, an F# to JavaScript transpiler, F# developers have been able to create rich interactive SPAs using F# for quite some time. Developed in 2016, Fable was originally built on top of Babel, an ECMAScript 2015+ to JavaScript compiler. Wouldn't something similar work for C#? As I see it this could pave the way for a very appealing C# framework that circumvents the complexities around WASM and SignalR.

Blazor not only in name but in glory too.

In fact, I'm quite surprised that we haven't seen such a development yet, but perhaps it's a matter of perspective. Maybe it has been a case of the wrong team looking at the wrong problem all along? After all the ASP.NET Core team excels in web development and not compiler design. Not every problem needs to be solved using SignalR or streaming APIs. Perhaps it's time to put a hold on more rendering modes and looking at Blazor through a different lens?

In my view, without doubt, this is the best path forward and I shall remain hopeful until then.

Creating a pretty console logger using Go's slog package

Wed, 23 Aug 2023 00:00:00 +0000

I had the privilege of attending GopherCon UK last week, and among the many captivating talks, one that stood out to me was "Structured Logging for the Standard Library" presented by Jonathan Amsterdam.

The presentation provided an insightful dive into Go's log/slog package. This talk couldn't have come at a better time given that I've just started on a new Go project, where I was eager to use Go's structured logging approach. The slog package in Go distinctly draws its inspiration from Uber's zap, making it a seamless transition for those who are well-acquainted with the latter. If you're already at ease with zap, you'll find yourself quickly at home with slog as well.

Currently, the slog library offers two built-in logging handlers: the TextHandler and the JSONHandler. The TextHandler formats logs as a series of key=value pairs, while the JSONHandler produces logs in JSON format. These handlers are greatly optimised for production scenarios, but can be somewhat verbose when troubleshooting applications during the development phase.

Recognizing this, I realized the necessity for a more visually friendly console logger tailored for local development purposes. Despite stumbling upon a code snippet within a blog post titled A Comprehensive Guide to Logging in Go with Slog, the implementation was both incomplete and flawed, rendering it unsuitable for my use case.

So my next question was: How difficult can it be to create one myself? Luckily, with a bit of clever hackery not that difficult at all!

Objectives

First let's address some of the shortcomings in the implementation outlined in the blog post referenced above. Unfortunately the custom handler fails to print preformatted attributes coming from the WithAttrs method. This means that attributes set on a parent logger are not propagated to child loggers at all. Additionally, the handler struggles to manage groups established on a parent logger too. Alongside these issues, there was no support for appending an error attribute, as well as addressing other edge cases which the original JSONHandler intuitively dealt with.

This didn't come as a huge surprise, given the difficulty of crafting a custom slog.Handler as highlighted by the Go team themselves. In fact, writing a custom slog.Handler is not to be taken lightly, and the Go team anticipates that only a select number of package authors will find themselves in the need of undertaking this task. To facilitate this, the Go team has thoughtfully provided a comprehensive guide to writing slog handlers to assist with this process.

Either way, I have no desire of writing a complete slog.Handler myself for something which is only ever going to be relevant during development time. With this in mind I set myself the following requirements:

Requirements:

Logs must be visually pleasing
Implementation must be complete
Only use packages from the standard library
Keep it super simple (as I'm lazy)

Non requirements:

Doesn't have to be very fast
Doesn't have to be very memory efficient

It's good to keep in mind that this "pretty" handler is tailored for development purposes and doesn't require blazing speed or intense memory efficiency. Since I won't be generating millions of logs on my local machine, this greatly simplifies the upcoming solution.

Final output:

The final logs should look something like this:

Here's another example by enabling debug logs and adding source information to them:

Evidently, these logs are designed for human readability through colouring and spacing. The default log attributes (time, level, message) are presented in a single line, while extra structured attributes are attached as a JSON object.

If you like this log style then keep on reading.

Creating a pretty console logger

For the purpose of this blog post I am calling this package prettylog but you can copy paste this logger into your own codebase and call it whatever you want.

Let's start with the function that will add color to the console output:

package prettylog

import (
	"fmt"
	"strconv"
)

const (
	reset = "\033[0m"

	black        = 30
	red          = 31
	green        = 32
	yellow       = 33
	blue         = 34
	magenta      = 35
	cyan         = 36
	lightGray    = 37
	darkGray     = 90
	lightRed     = 91
	lightGreen   = 92
	lightYellow  = 93
	lightBlue    = 94
	lightMagenta = 95
	lightCyan    = 96
	white        = 97
)

func colorize(colorCode int, v string) string {
	return fmt.Sprintf("\033[%sm%s%s", strconv.Itoa(colorCode), v, reset)
}

That's all that's required for straightforward coloured output, avoiding the need for an external dependency on the color package. I'm not even going to use all the colours listed above but I've included them regardless so you can adjust your logs to your own liking.

Moving forward, we'll create a struct called Handler (later used as prettylog.Handler):

type Handler struct {
	h slog.Handler
	b *bytes.Buffer
	m *sync.Mutex
}

The handler has three dependencies:

A "nested" slog.Handler which we wrap to effectively fulfil most of our handler's logic.
A *bytes.Buffer with the purpose to capture the output from the "nested" handler.
A mutex to guarantee thread safe access to our *bytes.Buffer.

All three dependencies will make more sense once we implement the Handle function.

The slog.Handler interface requires four methods to be implemented:

Enabled
WithAttrs
WithGroup
Handle

The Enabled method denotes whether a given handler handles a slog.Record of a particular slog.Level. The WithAttrs and WithGroup methods create child loggers with predefined Attrs.

For all three methods we can use the implementation of our nested handler:

func (h *Handler) Enabled(ctx context.Context, level slog.Level) bool {
	return h.h.Enabled(ctx, level)
}

func (h *Handler) WithAttrs(attrs []slog.Attr) slog.Handler {
	return &Handler{h: h.h.WithAttrs(attrs), b: h.b, m: h.m}
}

func (h *Handler) WithGroup(name string) slog.Handler {
	return &Handler{h: h.h.WithGroup(name), b: h.b, m: h.m}
}

The Handle method is where things get interesting.

Writing a log line is actually remarkably easy if one completely ignores groups and attributes to begin with:

const (
	timeFormat = "[15:04:05.000]"
)

func (h *Handler) Handle(ctx context.Context, r slog.Record) error {
	level := r.Level.String() + ":"

	switch r.Level {
	case slog.LevelDebug:
		level = colorize(darkGray, level)
	case slog.LevelInfo:
		level = colorize(cyan, level)
	case slog.LevelWarn:
		level = colorize(lightYellow, level)
	case slog.LevelError:
		level = colorize(lightRed, level)
	}

	fmt.Println(
		colorize(lightGray, r.Time.Format(timeFormat)),
		level,
		colorize(white, r.Message),
	)

	return nil
}

What about printing all the attributes that are added to the slog.Record or a parent logger? This is where the bytes buffer and the nested handler come into play.

The concept is simple:

We'll invoke the Handle function of the nested handler, but have it write to the *bytes.Buffer instead of the final io.Writer. We'll exclude the default log attributes such as time, level, and message from the nested handler to prevent repetition. Then, we'll append the remaining output as an indented JSON string to our log line. Since loggers need to function correctly when a single slog.Logger is shared among multiple goroutines, we also need to synchronize read and write access to the *bytes.Buffer using the mutex.

Let's encapsulate this behaviour in a function called computeAttrs:

func (h *Handler) computeAttrs(
	ctx context.Context,
	r slog.Record,
) (map[string]any, error) {
	h.m.Lock()
	defer func() {
		h.b.Reset()
		h.m.Unlock()
	}()
	if err := h.h.Handle(ctx, r); err != nil {
		return nil, fmt.Errorf("error when calling inner handler's Handle: %w", err)
	}

	var attrs map[string]any
	err := json.Unmarshal(h.b.Bytes(), &attrs)
	if err != nil {
		return nil, fmt.Errorf("error when unmarshaling inner handler's Handle result: %w", err)
	}
	return attrs, nil
}

The computeAttrs works as following:

It initially locks the mutex to ensure synchronized access across all goroutines utilizing the same logger or a child logger that shares the same *bytes.Buffer.
It defers the process of resetting the buffer (necessary to prevent outdated Attrs from previous Log calls) and releasing the mutex once the task is complete.
The Handle function of the inner slog.Handler is then invoked. This is where we compute a JSON object within the *bytes.Buffer, leveraging the capabilities of a slog.JSONHandler.
Lastly, the JSON buffer is transformed into a map[string]any after which the resulting object is returned to the caller.

Now, let's revisit our own Handle function and integrate the following code:

attrs, err := h.computeAttrs(ctx, r)
if err != nil {
    return err
}

bytes, err := json.MarshalIndent(attrs, "", "  ")
if err != nil {
    return fmt.Errorf("error when marshaling attrs: %w", err)
}

Through invoking computeAttrs, we can obtain the attrs map, which we subsequently convert into a neatly formatted (indented) JSON string using marshaling. Admittedly, this code isn't the most efficient approach (writing a JSON string into a buffer, deserializing it into an object, and then re-serializing it as a string), but it's worth mentioning that I couldn't identify a more effective method to obtain an indented JSON string from the slog.JSONHandler. Fortunately, as highlighted earlier, this handler isn't designed to achieve peak speed performance in any case.

Finally, we attach the formatted JSON string in a dark gray hue to our "pretty" log entry:

fmt.Println(
    colorize(lightGray, r.Time.Format(timeFormat)),
    level,
    colorize(white, r.Message),
    colorize(darkGray, string(bytes)),
)

The final Handle method looks as following:

func (h *Handler) Handle(ctx context.Context, r slog.Record) error {

	level := r.Level.String() + ":"

	switch r.Level {
	case slog.LevelDebug:
		level = colorize(darkGray, level)
	case slog.LevelInfo:
		level = colorize(cyan, level)
	case slog.LevelWarn:
		level = colorize(lightYellow, level)
	case slog.LevelError:
		level = colorize(lightRed, level)
	}

	attrs, err := h.computeAttrs(ctx, r)
	if err != nil {
		return err
	}

	bytes, err := json.MarshalIndent(attrs, "", "  ")
	if err != nil {
		return fmt.Errorf("error when marshaling attrs: %w", err)
	}

	fmt.Println(
		colorize(lightGray, r.Time.Format(timeFormat)),
		level,
		colorize(white, r.Message),
		colorize(darkGray, string(bytes)),
	)

	return nil
}

Only one last task remains. Currently, the nested slog.Handler writes the time, log level, and log message in addition to other custom attributes. However, since our handler is responsible for displaying these three default attributes, we need to configure the inner slog.Handler to bypass the slog.TimeKey, slog.LevelKey and slog.MessageKey attributes.

The most straightforward approach is to provide a function to the ReplaceAttr property of the slog.HandlerOptions. However, we wish to preserve the ability for an application to specify its individual ReplaceAttr function and slog.HandlerOptions. Therefore we must apply a final touch of trickery to "merge" a custom ReplaceAttr function with our own requirements:

func suppressDefaults(
	next func([]string, slog.Attr) slog.Attr,
) func([]string, slog.Attr) slog.Attr {
	return func(groups []string, a slog.Attr) slog.Attr {
		if a.Key == slog.TimeKey ||
			a.Key == slog.LevelKey ||
			a.Key == slog.MessageKey {
			return slog.Attr{}
		}
		if next == nil {
			return a
		}
		return next(groups, a)
	}
}

A helpful analogy for understanding the suppressDefaults function is to compare it to a middleware in an HTTP server. It takes in a next function that matches the same function signature as the ReplaceAttr property. It then performs filtering on slog.TimeKey, slog.LevelKey, and slog.MessageKey before continuing with next (if it's not nil).

With this in place, we're ready to create a constructor for our prettylog.Handler and assemble everything together:

func NewHandler(opts *slog.HandlerOptions) *Handler {
	if opts == nil {
		opts = &slog.HandlerOptions{}
	}
	b := &bytes.Buffer{}
	return &Handler{
		b: b,
		h: slog.NewJSONHandler(b, &slog.HandlerOptions{
			Level:       opts.Level,
			AddSource:   opts.AddSource,
			ReplaceAttr: suppressDefaults(opts.ReplaceAttr),
		}),
		m: &sync.Mutex{},
	}
}

The entire code can be found on GitHub.

Final result

Below are a few examples of how those pretty logs will look like.

Example of a logger with no *slog.HandlerOptions:

Creating a child logger with an additional group of attributes attached to it:

Making sure custom ReplaceAttr functions are supported:

Hopefully this blog post proved to be useful. It certainly provided me a valuable exercise for delving into the new log/slog package and gaining a better understanding of Go's latest structured logging capabilities.

How fast is ASP.NET Core?

Mon, 14 Nov 2022 00:00:00 +0000

In recent years the .NET Team has been heavily advertising ASP.NET Core as one of the fastest web frameworks on the market. The source of those claims has always been the TechEmpower Framework Benchmarks.

Take this slide from BUILD 2021, which Scott Hunter - Director of Program Management .NET, presented last year:

According to him .NET is more than 10 times faster than Node.js.

Scott also claims that .NET is faster than Java, Go and even C++, which is a huge boast if this is true!

Only recently Sébastien Ros, from the ASP.NET Core team, wrote this on Reddit:

In particular this sentence was super interesting to read:

Finally, even with the fastest Go web framework, .NET is still faster when using a high level stack (middleware, minimal APIs, …).

That is a bold claim and equally super impressive if true, so I was naturally curious to find out more about ASP.NET Core's performance and the TechEmpower Framework Benchmarks.

TechEmpower Benchmarks

TechEmpower is a software agency located in Los Angeles, California who run an independent framework benchmark on their servers. They publish all the results on their website as well as the framework code on GitHub.

The first thing that stood out to me was that the last official round (Round 21) was captured on 19th July 2022. The round before that (Round 20) ran in February 2021, which means there was a gap of more than a year between those two official rounds. I am not sure why they have only so few official rounds but I also discovered that they have a continuos benchmark run which can be viewed on their Results Dashboard. However, since the last official round was not that long ago and the difference between the results from Round 21 and the last completed run from the continuous benchmarks is not that big I decided to stick with Round 21 for my further analysis.

TechEmpower divides their tests into the following categories:

JSON serializers
Single query
Multiple queries
Cached queries
Fortunes
Data updates
Plaintext

The Fortunes benchmark is the gold standard of all benchmarks. It is the only one which tries to resemble a "real world scenario" which involves some reading from a database, sorting data by text, XSS prevention and it includes some server-side HTML template rendering too.

All the other test categories focus on an isolated aspect of a framework which makes it interesting for reading but useless when ranking web frameworks by general performance.

So let's take a closer look at the Fortunes benchmark from Round 21:

To my astonishment ASP.NET Core ranks 9th in place amongst the top 10 fastest frameworks! Two further flavours of the ASP.NET Core benchmark also rank 13th and 14th out of the 439 completed benchmark runs. That is very impressive indeed!

What are the different ASP.NET Core benchmarks?

Why does ASP.NET Core appear more than once in the benchmark results with varying performance metrics?

It turns out that there are in fact 15 different ASP.NET Core benchmarks which can be broadly subdivided into these four categories:

ASP.NET Core stripped naked
ASP.NET Core with middleware
ASP.NET Core MVC
ASP.NET Core on Mono

However, those are self-chosen names (by the .NET Team) and in order to get a real picture of what is being tested one has to look at the actual code itself. Luckily all the code is publicly available on GitHub.

I'm not interested in checking out 15 different implementations of various ASP.NET Core benchmarks so I decided to focus on the top performing ones by further narrowing down the 15 benchmarks into the best 7 out of the bunch:

I removed the Mono benchmarks and all the tests which used MySQL as the underlying database, because those tests performed significantly worse in comparison to the .NET Core with Postgres equivalents (which has the pg suffix in the labels).

Slowly the picture becomes clearer. The above screenshot also includes the framework "classification" which can be seen on the right hand side of the image. The top benchmark (which is the impressive one that ranks 9th overall) is classified as "Platform". The next three benchmarks are classified as "Micro" and the last three benchmarks are classified as "Full". There seems to be a very significant performance drop as one moves from the "Platform" tests down to the "Full" tests.

Similar to the naming of the framework benchmarks, the classification is not standardised or audited by TechEmpower employees either. Anyone can submit code with an arbitrary name and classification and get very little or no scrutiny at all by the repository maintainers. At least that was my impression (I once submitted an F# benchmark test).

Only the code itself can be used as a reliable source of truth to draw conclusions from those tests.

Luckily the code for all ASP.NET Core (on .NET Core) benchmarks can be found inside the /frameworks/CSharp/aspnetcore folder of the GitHub repository.

On 19th July 2022 (when Round 21 took place) the ASP.NET Core benchmark was divided into two projects:

Both of these web applications are very different so it is important to understand which one is used by which benchmark. This can be done by inspecting the config.toml file and the associated Dockerfile for the respective test case.

For example, the best ranking ASP.NET Core benchmark (aspcore-ado-pg) has the following configuration:

config.toml

[ado-pg]
urls.db = "/db"
urls.query = "/queries/"
urls.fortune = "/fortunes"
urls.cached_query = "/cached-worlds/"
approach = "Realistic"
classification = "Platform"
database = "Postgres"
database_os = "Linux"
os = "Linux"
orm = "Raw"
platform = ".NET"
webserver = "Kestrel"
versus = "aspcore-ado-pg"

aspcore-ado-pg.dockerfile

FROM mcr.microsoft.com/dotnet/sdk:6.0.100 AS build
WORKDIR /app
COPY PlatformBenchmarks .
RUN dotnet publish -c Release -o out /p:DatabaseProvider=Npgsql

FROM mcr.microsoft.com/dotnet/aspnet:6.0.0 AS runtime
ENV ASPNETCORE_URLS http://+:8080

# Full PGO
ENV DOTNET_TieredPGO 1
ENV DOTNET_TC_QuickJitForLoops 1
ENV DOTNET_ReadyToRun 0

WORKDIR /app
COPY --from=build /app/out ./
COPY PlatformBenchmarks/appsettings.postgresql.json ./appsettings.json

EXPOSE 8080

ENTRYPOINT ["dotnet", "PlatformBenchmarks.dll"]

The Dockerfile tells us that this test uses the /PlatformBenchmakrs code:

COPY PlatformBenchmarks .

From the config.toml file we can derive that the Fortune test invokes the /fortunes endpoint during the benchmark run.

Also the .NET Team specified this particular benchmark to be classified as a realistic approach in the config.toml file:

approach = "Realistic"

The "ASP.NET Core Platform" Benchmark

Cool, so what's inside this highly performant realistic ASP.NET Core application?

On first glance I didn't recognise a lot of what I'd normally consider a typical ASP.NET Core application (I've been developing professionally on ASP.NET and later ASP.NET Core since 2010).

The only thing that looked slightly familiar was the use of Kestrel (the .NET web server) inside Program.cs:

To my surprise this was also the only thing which I could recognise as an "ASP.NET Core" thing. The web application itself is not even initialised via one of the many ASP.NET Core idioms. Instead it creates a custom BenchmarkApplication as the listener on the configured endpoint.

An untrained eye might be thinking that builder.UseHttpApplication<T>() is a method that comes with Kestrel, but that is not the case either. The extension method as well as the HttpApplication class which is in use here are not things which you'd find in the actual ASP.NET Core framework. It is yet another custom class specifically written for this benchmark:

Not even the interface IHttpApplication comes from ASP.NET Core. This is also a custom made type which was specifically designed for the benchmark tests.

Looking further into the BenchmarkApplication.cs I was shocked by the sheer amount of finely tuned low level C# code that was tailor made for this (extremely simple) application.

Everything inside the /PlatformBenchmarks folder is custom code which you won't find anywhere in an official ASP.NET Core package.

A good example is the AsciiString class which is used to statically initialise huge chunks of the expected HTTP responses in advance:

Even though it is called AsciiString it's only a string in name:

In reality the AsciiString class is just a fancy (highly optimised) wrapper around a byte array which converts a string into bytes during initialisation. In the case of the Fortunes test the entire HTTP header (which the application is expected to return during a test run) is created upfront during application startup and then kept in memory for the entirety of the benchmark:

This is supposed to be a very simple application, something which a framework could probably squeeze into a single file of code, but the /PlatformBenchmarks project has many dozens of expertly crafted classes with all sorts of trickery applied to produce a desired outcome.

The extent to which the .NET Team went is extraordinary.

ASP.NET Core has many ways of implementing routing. They have Actions and Controllers, Endpoint Routing, Minimal APIs, or if someone wanted to operate on the lowest level of ASP.NET Core (= Platform), then they could work directly with the Request and Response objects from the HttpContext.

Neither of these options can be found /PlatformBenchmarks:

In fact, you won't find a HttpContext anywhere at all. It's almost like the .NET Team tried to avoid using ASP.NET Core at all cost, which is strange to say the least.

Sieving through the project reveals even more bizarre code which the .NET Team applied to "tweak" the benchmark score.

For instance take a look at the HTML templating implementation of the ASP.NET Core solution:

There is no HTML template at all. The whole point of the Fortunes benchmark is - amongst others - to test different web frameworks for how fast they can output templated HTML. In ASP.NET Core we have two templating engines, Razor Views and Razor Pages, of which none is being used here.

Instead there are more hardcoded statically initialised byte arrays:

Of course the question remains if these sort of tricks are allowed? The lines might be a bit blurry but I am certain that this implementation pushes the boundaries of what one might consider a real templating engine.

Web frameworks don't have to participate in every category of the TechEmpower Benchmark tests. In fact it is encouraged to only enter the categories which apply to a particular framework. For example, if a low level ASP.NET Core implementation (a real one which uses ASP.NET Core with HttpContext and so on) doesn't have template rendering included then it shouldn't enter the competition for Fortunes. If a higher level framework such as ASP.NET Core MVC has HTML template rendering available then it can enter the Fortunes benchmark. Entering the Fortunes competition with random C# code that doesn't resemble a real web framework at all makes very little sense and really just tarnishes the credibility of the entire TechEmpower Framework Benchmark test.

Perhaps I am being a little bit overly critical here, but this line of code really got me thinking:

Setting the Date HTTP header with a date time value is such a small task that you don't even need a framework to do this job. It should be no more than a single line of code:

response.WriteHeader("Date", DateTime.UtcNow.ToString())

However, the ASP.NET Core benchmark has a "slightly more optimised" solution to this task:

Setting a date time value has been so highly optimised that I can't even fit the entire code into a single screen. The creativity of finding ways to save computation cycles and therefore score higher in the benchmarks is truly astonishing. The DateHeader class is a static class (which means it only gets initialised once as a singleton and is then kept in memory) with a static DateTimeOffset value (of course already stored as a byte array). Additionally a System.Threading.Timer object is also statically initialised with a one second interval. This Timer will run on a separate thread and set a new date time value once every second:

private static readonly Timer s_timer = new Timer((s) => {
    SetDateValues(DateTimeOffset.UtcNow);
}, null, 1000, 1000);

You wonder how this is an optimisation? Well, the TechEmpower Benchmark will hit a web server many hundreds of thousand times per second to really test the limits of each framework. The DateHeader class will return the exact same timestamp for all those thousand requests and henceforth save itself from computing a new timestamp many thousand times. Then after one second the Timer (which runs on a separate thread) will sync a new timestamp exactly once and cache the timestamp for the next 300+ thousand requests. I'm impressed by the ingenuity. In all fairness the HTTP Date header doesn't accept timestamps more granular than a second and the TechEmpower guidelines mention this to be an accepted optimisation.

The only question I have is if this benchmark is testing ASP.NET Core why does it need to replicate something which ASP.NET Core already has out of the box?

Now I ask myself, are all the ASP.NET Core benchmarks "tweaked" like this?

What about other frameworks?

I needed to further investigate this!

ASP.NET Core Micro Benchmarks

After dissecting the "Platform" benchmark it was time to look at the "Micro" frameworks:

Looking at the respective Dockerfile it turns out that the "Micro" benchmarks use the code from the /Benchmarks folder, which looks like an actual ASP.NET Core application:

This benchmark immediately has a different vibe than the one before. I'm very pleased to see that it's actually using elements which come from ASP.NET Core itself. The Fortunes tests are initialised via conventional middleware like this:

The aspcore-mw-ado-pg benchmark is what most .NET developers would probably call a low level "Platform" ASP.NET Core implementation. There is no higher level routing, no content negotiation, no other cross-cutting middlewares, no EntityFramework and still no actual HTML template rendering either, but at least it's ASP.NET Core.

The middleware operates directly on the HttpContext to do basic routing:

This is okay and inline with the TechEmpower guidelines, because operating directly on the HttpContext is canonical for the framework (as opposed to the benchmark before):

In some cases, it is considered normal and sufficiently production-grade to use hand-crafted minimalist routing using control structures such as if/else branching. This is acceptable where it is considered canonical for the framework.

Although the middleware benchmark doesn't apply the AsciiString trickery any more, it still resorts to a "fake" templating engine:

Overall it is a much more realistic (albeit not perfect) benchmark!

ASP.NET Core Full Benchmarks

Finally it was time to check out the "MVC" benchmarks. It also derives its code from the /Benchmarks folder but instead of operating on the raw HttpContext it actually initialises the least required MVC middleware with the Razor View Engine:

The Controller Action is also kept very realistic and finally uses the actual ASP.NET Core templating engine:

[HttpGet("raw")]
public async Task<IActionResult> Raw()
{
    var db = HttpContext.RequestServices.GetRequiredService<RawDb>();
    return View("Fortunes", await db.LoadFortunesRows());
}

The Razor view matches what one would expect from this simple benchmark:

This is the most realistic ASP.NET Core application which actually meets the spirit of the Fortunes benchmark.

However, the results of this benchmark are very different to what Microsoft actively advertised to the .NET Community. The performance difference between a "fake" templating engine where a HTML response is being created in memory via a cached StringBuilder versus an actual templating engine which has to incur additional (expensive) I/O operations to read, parse and apply HTML templates from disk is enormous.

The latter only manages to serve 184k requests/sec and only ranks 109^th overall in the TechEmpower Framework Benchmarks for the Fortunes test. That is a staggering difference and something to be kept in mind when comparing ASP.NET Core to frameworks written in Java, Go or C++.

Other frameworks

Now that I've established a clearer picture of what the various ASP.NET Core benchmarks are, it was time to look at other frameworks too.

Java

The fastest Java benchmark which also uses Postgres as the underlying database is Jooby.

Their benchmark implementation is astonishingly simple. The entire fortune implementation is basically this block of code:

It uses a higher level router (get("/fortunes", ctx -> {}) as well as conventional database access methods and a real templating engine too:

This is pretty much the Java equivalent to the ASP.NET Core MVC (aka Full) benchmark.

The interesting part is that this completely unoptimised fully fledged Java MVC framework ranks overall 12^th in the Fortunes benchmark with an incredible 404k requests/sec. It is essentially more than twice as fast as the ASP.NET Core equivalent, still beats the "Micro" implementation of the ASP.NET Core benchmark (which skips all the expensive I/O operations by using a fake templating engine) and even manages to compete with the infamous /PlatformBenchmarks application which in all honesty due to its differences is not even worth a comparison.

No disrespect to ASP.NET Core (because 184k requests/sec is still an amazing result) but it doesn't come anywhere near this Java framework when it comes to performance. Credit where credit is due.

Go

What about Go?

Sébastien Ros (developer working on ASP.NET Core performance at Microsoft) specifically called out Go and claimed that ASP.NET Core is still faster than Go in a like-for-like comparison. I was personally very interested in this claim as I have migrated several .NET Core projects to Go and seen dramatic performance increases as a result of it.

At the time of writing this post the fasted Fortune benchmark is atreugo for Go.

Similar to Java, the actual Go implementation is kept extremely simple.

Routing is done via the framework provided idioms:

No shortcuts or trickery to be found here. The entire application for the Fortunes benchmark is basically less than 20 lines of code.

Templating is done the proper way too:

So where does this leave us overall? Well, just like with the fasted Java framework, the Go benchmark also compares to ASP.NET Core's "Full" implementation best. Anything other would simply not be fair. You cannot compare a benchmark which spits out in-memory crafted HTML (which is not even part of ASP.NET Core) versus one that actually uses a real templating engine that goes through expensive cycles of reading files from I/O, parsing them at runtime and having to execute their logic for every request (loops, variables, etc. in the template).

Nevertheless, the expensive Go implementation ranks 22^nd overall in the TechEmpower Fortunes Benchmark with an equally impressive 381k requests/sec. Not quite as fast as the Java one but still more than 2x faster than the equivalent test in ASP.NET Core.

C++

Hopefully this shouldn't be a big surprise, but currently C++ with the drogon framework leads the Fortunes benchmarks with a breathtaking 616k requests/sec which beats every other framework by a long stretch (except Rust where the gap is not that big)! What makes this achievement even more astonishing is that it manages to do this with a fully fledged MVC implementation. There is absolutely no shortcuts or trickery at play.

It even uses the CSP templating engine which looks like this:

I love .NET but there is no mental gymnastics that one could convincingly apply in which .NET comes on top of C++. Any benchmark that suggests otherwise knows it's not being honest with itself.

Rust, Node.js, Kotlin and PHP

Since the .NET Team started to campaign ASP.NET Core as a much faster web framework than many others I thought it would only be fair to further probe those claims.

Rust

Rust delivers 588k requests/sec and comes 2^nd in the overall Fortunes benchmark. It's the only other language platform which gives C++ a run for its money. The xitca-web framework accomplishes this unbelievable result with another proper MVC-like implementation and an actual templating engine.

Kotlin

Another great result is achieved by a Kotlin web framework with a very honest Fortunes implementation which uses the Rocker engine for its HTML templating. It pegs at 350k requests/sec and comes 29^th overall which is still 80 places ahead of the equivalent ASP.NET Core implementation.

Node.js

One claim which turned out to be (partially) true is that ASP.NET Core is faster than Node.js. Although only 3x and not 10x faster as it was claimed, ASP.NET Core still beats Polkadot convincingly, which is the highest ranking Node.js framework which had a comparable implementation to the "Micro" benchmark in ASP.NET Core. With only 125k requests/sec Node.js trails behind .NET.

PHP

Now this might actually take people by surprise, but if you haven't been paying attention then you might have missed all the work that has gone into PHP over the many years. Not least because Facebook invested a lot of effort into making PHP a better platform. It is now capable of serving an incredible 309k requests/sec with it's MVC-like implementation delivered by mixphp. That is still significantly faster than ASP.NET Core's MVC framework and certainly deserves a mention too!

Just(js)

If you are a JavaScript developer don't feel too bad about the Node.js benchmarks, because Just(js) will knock you off your socks with a spectacular 538k requests/sec. This is no joke, Just(js) comes 5^th in the Fortunes benchmark and is the only framework which competes in the realms of C++ and Rust. It is a remarkable achievement which is not something that happened by mistake. It is far ahead of every other ASP.NET Core benchmark and had to be mentioned as part of this post!

Is ASP.NET Core actually fast?

Yes, it certainly is!

Especially if you think back to what Classic ASP.NET was during the .NET Framework times then it becomes very clear that ASP.NET Core is world's apart from its darker past.

Make no mistake, ASP.NET Core is very fast and certainly doesn't need to shy away from a healthy competition. However, it is evidently not faster than Java, Go or C++. Perhaps it will get there one day but at the moment this is not the case. I am certain that we haven't seen the ceiling for ASP.NET Core just yet and I look forward to what the .NET Team will deliver next. ASP.NET Core is a great platform and even though it's not the fastest (yet), it is still a joy!

I wish Scott Hunter and the rest of the ASP.NET Core Team didn't feel the need to market ASP.NET Core based on soft lies and bad-faith claims to make ASP.NET Core stand out amongst its peers. I'm sure there is more to be proud of!

Sidenotes

One final interesting thing which came up during my research is that TechEmpower switched their cloud hosting environment from AWS to Azure around the time when Microsoft got interested in the tests. TechEmpower also receives its physical hardware for all their on-premise tests by Microsoft today.

Update after Twitter storm (15/11/2022)

Bad Microsoft?

No, I don't think the .NET Team had any malice in mind. I am confident the engineers were simply geeking out over performance improvements and then the marketing department probably got wind of it and started to conveniently cherry pick comparisons. It happens, but David Fowler from the ASP.NET Core team confirmed they will be more mindful about this going forward.

Fair comparisons?

The TechEmpower Framework Benchmarks run over 300 different frameworks. Web frameworks consist of many layers with a huge variety of functionality. It's going to be impossible to have a 100% fair comparison when these web frameworks don't even have feature parity. I think there is still some value in having basic ground rules on the big things (e.g. db, templating engine, etc.) and accepting that those benchmarks won't be perfect without any flaws. A Redditer perfectly pointed out all the other things that one has to take into account. Has the garbage collector been turned off for the benchmarks? Are they using a fully HTTP compliant router? Should frameworks use the same templating engine? It's complicated and hence why I personally think it was quite unfair by Microsoft to "smear" other frameworks as being slow based on cherry picked results from greatly inconsistent implementations. This was precisely the whole point of me writing this post.

ASP.NET Core Platform Benchmark

The .NET Team pointed out on Twitter that the "Platform" benchmark is the lowest level of the "Platform" where some parts were used by Kestrel and others not. I don't have an issue with that personally, but it seems the .NET Team cannot really articulate what "Platform" means. The Platform of what? It is not ASP.NET Core, so perhaps they mean the ".NET Platform" or maybe "Platform" is just a conveniently chosen name to label a random collection of low level APIs bundled together into a benchmark application. The point is it is not ASP.NET Core as far as I know and therefore labelling the test as "ASP.NET Core Platform" so "ASP.NET Core" shows up at the top of the benchmark table is slightly disingenuous.

Postgres Pipelining

The developer behind Just(js) pointed me towards an issue where "the ASP.NET team made a big collective (successful) effort to force the removal of postgres pipelining from the benchmarks for no particularly good reason". Honestly I don't know enough about it, but it made me look into other GitHub issues and there is certainly an interest at Microsoft to keep other frameworks in check.

There is also another issue from 2019 where someone pointed out that concatenating hardcoded strings is cheating and after Ben Adams tried to defend the ASP.NET Core benchmark it was finally ruled that it is indeed against the rules. The Rust framework which was part of that discussion made the necessary changes afterwards, but Ben and the .NET Team never adjusted theirs.

The type system is a programmer's best friend

Tue, 01 Nov 2022 00:00:00 +0000

I am tired of primitive obsession and the excessive use of primitive types to model a domain.

A string value is not a great type to convey a user's email address or their country of origin. These values deserve much richer and dedicated types. I want a data type called EmailAddress which cannot be null. I want a single point of entry to create a new object of that type. It should get validated and normalised before returning a new value. I want that data type to have helpful methods such as .Domain() or .NonAliasValue() which would return gmail.com and [email protected] respectively for an input of [email protected]. Such useful functionality should be embedded into those types. It provides safety, helps to prevent bugs and it immensely increases maintainability.

Well designed types with useful functionality guide a programmer to do the right thing.

For instance an EmailAddress could have two methods to check for equality:

Equals would return true if two (normalised) email addresses are identical.
EqualsInPrinciple would return true for inputs of [email protected] and [email protected] also.

These type specific methods would be extremely handy in a variety of scenarios. A user login should not fail if the user registered with [email protected] but then logs in with [email protected]. Equally it would be super convenient to match a user who contacted customer support from their non-aliased email address ([email protected]) to their registered account ([email protected]). Those are typical requirements which a simple string couldn't fulfil without a lot of additional domain logic scattered around a codebase.

Note: According to the official RFC the part of an email address before the @-sign could be case-sensitive, but all major email hosts treat them as case-insensitive and so it's not unreasonable for a domain type to take this knowledge into consideration.

Good types can prevent bugs

Ideally I want to go even further. An email address can be verified or unverified. It's common practice to validate an email address by sending a unique code to a person's inbox. These "business" interactions can be expressed through the type system as well. For example, let's have a second type called VerifiedEmailAddress. If you wish it can even inherit from an EmailAddress. I don't care, but ensure that there is only one place in the code which can yield a new instance of VerifiedEmailAddress, namely the service which is responsible for validating a user's address. All of a sudden the rest of the application could rely on this new type to prevent bugs.

Any function which is sending emails can lean on the safety of a VerifiedEmailAddress. Imagine what it would look like if an email address was expressed via a simple string. One would have to find/load the associated user account first, check for some obscure flag like HasVerifiedEmail or IsActive (which is the worst flag by the way because it tends to grow in meaning over time) and then hope that this flag was actually correctly set and not mistakenly initialised as true in some default constructor. There is too much room for error to go unchecked! Using a primitive string for an object which could get so easily expressed through its own type is simply lazy and unimaginative programming.

Rich types protect you from future mistakes

Another great example is money! I've lost count of how many applications express monetary values using the decimal type. Why? There are so many issues with that type that I find it incomprehensible. Where is the currency? Every domain that deals with people's money should have a dedicated type called Money. At the very least it should include the currency and some operator overloads (or other safety features) to prevent silly mistakes like multiplying $100 with £20. Besides, not every currency has only two digits after the decimal point. Some currencies such as the Bahraini or Kuwaiti dinar have three. If you deal with investments or bank loans in Chile then you better make sure that you render the Unidad de Fomento with 4 decimal points. These concerns are already important enough to warrant a dedicated Money type, but that's not even all.

Unless you build everything in house you will eventually have to deal with third party systems too. For instance, most payment gateways request and respond with money as integer values. Integer values don't suffer from the same rounding issues which are often associated with float or double types and therefore preferred over floating-point numbers. The only caveat is that values have to be transmitted in minor units (e.g. Cent, Pence, Diram, Grosz, Kopeck, etc.), which means that if your program deals with decimal values you'll have to constantly convert them back and forth when talking to an external API. As explained before not every currency uses two decimal points so it's not going to be a simple multiplication/division by 100 every time. Things can get very quickly difficult and matters could be significantly simplified if those business rules were encapsulated into a concise single type:

var x = Money.FromMinorUnit(100, "GBP"): £1
var y = Money.FromUnit(100.50, "GBP"): £1.50
Console.WriteLine(x.AsUnit()): 1.5
Console.WriteLine(x.AsMinorUnit()): 150

As if this was not already complicated enough countries have different currency formats to render money too. In the UK "Ten Thousand Pounds and Fifty Pence" would be represented as 10,000.50 but in Germany "Ten Thousand Euro and Fifty Cent" would be shown as 10.000,50. Just imagine the amount of money and currency related code that would be fragmented (and possibly duplicated with minor inconsistencies) across a codebase if those business rules were not put into a single Money type.

Additionally a dedicated Money type could include many more features which would make working with monetary values a breeze:

var gbp = Currency.Parse("GBP");
var loc = Locale.Parse("Europe/London");

var money = Money.FromMinorUnit(1000050, gbp);

money.Format(loc)        // ==> £10,000.50
money.FormatVerbose(loc) // ==> GBP 10,000.50
money.FormatShort(loc)   // ==> £10k

Sure modelling such a Money type would be a little bit of an effort to begin with, but once it has been implemented and tested to satisfaction then the rest of a codebase could rely on much greater safety and prevent the majority of bugs which would otherwise creep in over time. Even if small features such as the guarded initialisation of a Money object through either Money.FromUnit(decimal v, Currency c) or Money.FromMinorUnit(int v, Currency c) doesn't seem like much, it makes successive developers think every time whether the value which they received from a user input or external API is one or the other and therefore prevent bugs from the start.

Smart types can prevent unwanted side effects

The great thing about rich types is that you can shape them in whichever way you want. If I haven't sparked your own imagination yet then let me show you another great example of how a dedicated type can save your team from a huge operational overhead and even prevent security bugs.

Every codebase that I've ever worked with had something like a string secretKey or string password somewhere as a parameter of a function. Now what could possibly go wrong with these variables?

Imagine you have this (pseudo-)code:

try
{
    var userLogin = new UserLogin
    {
        Username = username
        Password = password
    }

    var success = _loginService.TryAuthenticate(userLogin);

    if (success)
        RedirectToHomeScreen(userLogin);

    ReturnUnauthorized();
}
catch (Exception ex)
{
    Logger.LogError(ex, "User login failed for {login}", userLogin);
}

The problem that arises here is that if an exception is thrown during the authentication process then this application would (accidentally) write the user's cleartext password into the logs. Now of course this code should never exist like this in the first place and you'd hope it would get caught during a code review before going to production but the reality is that this stuff happens over time. Most such bugs occur incrementally as time moves on.

Initially the UserLogin class could have had a different set of properties and this piece of code would have probably been fine during the initial code review. Years later someone might have modified the UserLogin class to include the cleartext password. Then this function would have not even shown up in the diff which was submitted for later review and violà you've just introduced a security bug. I am sure every developer with some years of experience would have run into a similar issues at some point during their career.

However this bug could have been easily prevented with the introduction of a dedicated type.

In C# (using this as my example language) the .ToString() method gets automatically called when an object gets written to a log (or anywhere else for that matter). Having this knowledge one could design a Password type like this:

public readonly record struct Password()
{
    // implementation goes here

    public override string ToString()
    {
        return "****";
    }

    public string Cleartext()
    {
        return _cleartext;
    }
}

It's only a minor change, but all of a sudden it would become impossible to accidentally output a cleartext password anywhere in the system. Isn't that great?

Of course you might still need the cleartext value during the actual authentication process but that is being made accessible via a very clearly named method Cleartext() so there is no ambiguity about the sensitivity of this operation and it automatically guides a developer to use this method with intention and care.

Dealing with a user's PII (e.g. National Insurance number, Tax number, etc.) would be the same principle. Model that information using dedicated types. Override default functions such as .ToString() to your benefit and expose sensitive data via accordingly named functions. You'll never leak PII into logs and other places that later might require a huge operation to scrub it out again.

These small tricks can go a long way!

Make it a habit

Every time you deal with data that has particular rules, behaviours or dangers associated with them think about how you could help yourself with the creation of an explicit type.

Continuing from my example of the Password type we can go even further once again!

Passwords get hashed before being stored in the database, right? Sure thing, but that hash is (of course) not just a simple string. At some point we will have to compare a previously stored hash with a newly computed hash during the login process. The problem is that not every developer is a security expert and therefore knows that comparing two hash strings could make your code vulnerable to timing attacks.

The recommended way of checking the equality of two password hashes is by doing it in a non-optimised way:

// Compares two byte arrays for equality. The method is specifically written so that the loop is not optimized.
[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)]
private static bool ByteArraysEqual(byte[] a, byte[] b)
{
    if (a == null && b == null)
    {
        return true;
    }
    if (a == null || b == null || a.Length != b.Length)
    {
        return false;
    }
    var areSame = true;
    for (var i = 0; i < a.Length; i++)
    {
        areSame &= (a[i] == b[i]);
    }
    return areSame;
}

Note: Code example taken from the original ASP.NET Core repository.

So it would only make sense to encode this particular functionality into a dedicated type:

public readonly record struct PasswordHash
{
    // Implementation goes here

    public override bool Equals(PasswordHash other)
    {
        return ByteArraysEqual(this.Bytes(), other.Bytes());
    }
}

If a PasswordHasher only returns values of type PasswordHash then even developers who don't know much about this topic will be forced to use a safe form of checking for equality.

Be thoughtful in how you model your domain!

Of course, it's almost needless to say that with everything in programming there is no clear right or wrong and there is always more nuance in people's personal use cases than what I could possibly convey in a single post, but my general suggestion is to think about how you could make the type system your best friend.

Many modern programming languages come with very rich type systems nowadays and I think on a broad spectrum we are probably heavily underutilising those as a way of improving our code.

Using Go generics to pass struct slices for interface slices

Sun, 04 Sep 2022 00:00:00 +0000

Have you ever tried to pass a struct slice into a function which accepts a slice of interfaces? In Go this won't work.

Let's have a quick look at an example. Let's assume we have an interface called Human and a function called GreetHumans which accepts a slice of humans and prints their names:

package main

import "fmt"

type Human interface {
	Name() string
}

func GreetHumans(humans []Human) {
	for _, h := range humans {
		fmt.Println("Hello " + h.Name())
	}
}

Then we have a separate struct which implements the Human interface:

type Hero struct {
	FirstName string
	LastName  string
}

func (h Hero) Name() string {
	return h.FirstName + " " + h.LastName
}

Nothing unusual so far.

Now one can create an object of type Hero and pass it into any function that requires an object of Human. That is expected.

However, the issue occurs when one deals with a slice of Hero and a function accepts a slice of Human.

This code won't compile:

func main() {
	heroes := []Hero{
		{FirstName: "Peter", LastName: "Parker"},
		{FirstName: "Bruce", LastName: "Wayne"},
	}

	GreetHumans(heroes) // <-- Compilation error here
}

Even though all heroes are humans the compiler won't accept this assignment.

You wonder why? Simply because Go doesn't want to hide expensive operations behind convenient syntax:

One has to iterate through a slice of Hero themselves and convert each object of Hero explicitly to a Human in order to cast the entire slice before passing it into the GreetHumans function. The cost of this conversion becomes immediately visible to the programmer.

What did Go programmers do up until recently?

Well there were mainly three options:

Create a conversion function for each individual type which implements the Human interface
Create a "generic" conversion function using interface{}
Create a "generic" conversion function using reflection

Option 1 is extremely tedious and option 2 and 3 provide very weak (or basically no) type safety (because checks would be only performed at runtime).

Since Go 1.18 one can use Generics to tackle the issue.

First we can modify the GreetHumans function to use Generics and therefore not require any casting at all:

func GreetHumans[T Human](humans []T) {
	for _, h := range humans {
		fmt.Println("Hello " + h.Name())
	}
}

This makes it possible to pass the heroes slice into the GreetHumans functions now.

However, sometimes it's not possible to make a function generic. Methods (functions on a type) require all type parameters to be on the type. Parameterized methods are not allowed.

The good news is that even in this case Generics can help to provide a much better conversion function then the previously listed options:

func CastToHumans[T Human](humans []T) []Human {
	result := []Human{}
	for _, h := range humans {
		result = append(result, h)
	}
	return result
}

Here the generic CastToHumans function provides type safety at the time of compilation. It still remains an expensive operation but at least it cannot be used in an improper way any longer.

I wasn't sure if this is going to work and I was positively surprised to find out that it does indeed.

It's another neat use case of Generics in Go!

Building a secure note sharing service in Go

Tue, 26 Jul 2022 00:00:00 +0000

Welcome to my first Go related article which I am releasing on my blog.

In this blog post I’ll be using Go to develop a super small web service which can be used by people or organisations to share secrets in a slightly more private way. We all have the occasional need to share a secret with a co-worker or another person. It may be an API key, a password or even some confidential data from a customer. When we share secrets via channels such as Slack, Teams or Email then we essentially send the secret to the servers of a complete stranger. We have no oversight over how the data is being handled, how long it will persist on third party servers and who the people are who have access to it. Sending secrets directly via Slack or Teams can also pose other unwanted side effects. For instance new employees who get added to an existing channel could discover previously shared confidential data via a channel's chat history. That could be a breach of security in itself if those employees didn't have the necessary clearance beforehand. Overall secrets and/or confidential data should never be shared directly via (untrusted) third party channels.

I thought writing a small data sharing app could be a good way of learning Go. The goal is to create a small web service which can be run as a single binary or from a Docker container inside a company's own infrastructure. Why rely on an (untrusted) third party service (noterip, safenote, onetimesecret, circumvent or privnote) if one could run their own?

The Foundation

This is going to be an MVP so we’ll be making some fast gains by keeping the service extremely simple and making use of Redis as the main persistence layer. Redis seems to be a good fit for an MVP as it can be easily hosted in a container and can be used as a distributed data store that can serve multiple instances of our app. We can also make use of the TTL (time to live) feature which gives us a quick and dirty implementation of short lived, self destructing links.

Our web service will be a simple Go executable which can also run in a container and which will implement basic functionality to persist and retrieve a secret.

The entire solution will be open source with an OSS friendly Apache 2.0 license so that people can fork it and make their own modifications to it.

I call this project self-destruct-notes:

For the purpose of this blog post I'll keep the service very rudimental and use as few third party dependencies as possible. I'm actually coding this project as I'm writing this blog post so one can follow the evolution of this app through this article or the associated commit history in Git.

Creating a new Go project

First let’s create a simple Go project to kick things off.

I'll start with a basic main.go file which yields a typical "hello world" message and then I run a couple go mod commands to initiliase the project:

main.go:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Terminal:

go mod init github.com/dustinmoris/self-destruct-notes
go mod tidy

Running go run . now will return hello world.

For the people who are new to Go (my main readership comes from .NET), the go mod commands are Go’s way of managing third party packages. They generate a go.mod and go.sum file when the service takes on any external dependencies. For now the go.mod file remains mostly empty:

go.mod:

module github.com/dustinmoris/self-destruct-notes

go 1.17

Creating a simple hello world web server

Next I’m going to change the main function from a hello world console application to a hello world web server. We’re going to read the PORT environment variable to establish the port which our HTTP server should be listening to. If that variable is not set then we’ll default to port 3000.

The “web server” itself will be a basic HTTP handler of the form func (http.ResponseWriter, *http.Request):

package main

import (
    "fmt"
    "net/http"
    "os"
)

func main() {
    port := os.Getenv("PORT")
    if len(port) == 0 {
        port = "3000"
    }
    addr := ":" + port

    fmt.Printf("Starting web server, listening on %s\n", addr)

    err := http.ListenAndServe(addr, http.HandlerFunc(webServer))
    if err != nil {
        panic(err)
    }
}

func webServer(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(200)
    w.Write([]byte("hello world"))
}

If I hit go run . now then our web server will start and I'll be able to curl localhost:3000 to get a hello world response:

Adding basic routing

Now that we have a super tiny web server running we can add basic routing for the three main endpoints which I'd like to support:

Get / (start page with a form to create a new note)
POST / (handles the form post to persist a new note)
GET /{noteID} (returns a previously saved note)

So far our web server is just a singe function and we had to use the http.HandlerFunc wrapper in order to implement the http.Handler interface on the function. Another way of defining the server would have been to create a struct type which implements the http.Handler interface by exposing a ServeHTTP function. Personally I prefer this option because I know that we will require some dependencies for Redis and other functionality later on and management of those dependencies will be easier that way.

Below I have refactored the webServer function into a Server struct and added some basic routing to it:

type Server struct{}

func (s *Server) ServeHTTP(
    w http.ResponseWriter,
    r *http.Request,
) {
    if r.Method == "GET" || r.Method == "HEAD" {
        noteID := strings.TrimPrefix(r.URL.Path, "/")
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(
            fmt.Sprintf(
                "You requested the note with the ID '%s'.",
                noteID)))
        return
    }

    if r.Method == "POST" && r.URL.Path == "/" {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("You posted to /."))
        return
    }

    w.WriteHeader(http.StatusNotFound)
    w.Write([]byte("Not Found"))
}

As you can see I didn’t use any third party packages to add more complex routing capabilities. Our server only needs basic functionality which can be easily satisfied by the standard library. Although this is slightly more verbose than in other languages it keeps the Go code very simple and easy to understand.

Additionally I have also changed the http.ListenAndServe function call to accept a new Server object:

http.ListenAndServe(addr, &Server{})

Next I'm splitting the web server code into smaller functions. I am creating a new handler for posting notes, another handler for getting notes and a helper function to return a 404 Not Found response.

After this refactoring the ServeHTTP function serves as the main routing handler and nothing else:

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.Method == "GET" || r.Method == "HEAD" {
        s.handleGET(w, r)
        return
    }
    if r.Method == "POST" && r.URL.Path == "/" {
        s.handlePOST(w, r)
        return
    }
    s.notFound(w, r)
}

func (s *Server) notFound(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusNotFound)
    w.Write([]byte("Not Found"))
}

func (s *Server) handlePOST(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("You posted to /."))
}

func (s *Server) handleGET(w http.ResponseWriter, r *http.Request) {
    noteID := strings.TrimPrefix(r.URL.Path, "/")
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(fmt.Sprintf("You requested the note with the ID '%s'.", noteID)))
}

That's a very quick and neat way to keep the Go web server code well structured and yet super simple at the same time.

Adding HTML views

The next step is to add some basic HTML to the project. We need a HTML template which we can return on the / index page when someone visits the service.

It's up to an individual to decide how they like to structure their code, but I generally place all web content which needs to get shipped alongside the Go executable into a /dist folder. With this in mind I've added two initial HTML templates to the project:

/dist/layout.html
/dist/index.html

The layout.html is the main layout page which can be re-used by other templates. Other programming languages might call this a "master page" (e.g. MVC). The index.html is the template which mainly includes the HTML form for creating a new note.

For simplicity I've kept them extremely small and I've added only a tiny bit of CSS to make the UI passable to the eye. If you're not familiar with Go's templating language then I'd recommend to have a quick look at the templates in my Git repository itself.

In order to respond with a HTML template on the index (/) route I added one helper function to our Server struct and covered the / route in the handleGET handler:

Helper function:

func (s *Server) renderTemplate(
	w http.ResponseWriter,
	r *http.Request,
	data interface{},
	name string,
	files ...string,
) {
	t := template.Must(template.ParseFiles(files...))
	err := t.ExecuteTemplate(w, name, data)
	if err != nil {
		panic(err)
	}
}

The renderTemplate function is just a small convenience method to parse all requested template files and initialise a *template.Template object and then write the result combined with the data model to the HTTP response body.

This method could be optimised by computing all known templates in advance but for brevity I'll keep it this way for now.

If you're new to Go and wonder what files ...string means in the function declaration then just know that this is a special form of specifying a string array where the function can accept a dynamic amount of string parameters at the end of a function call. It's similar to what params string[] does in C#.

Using the renderTemplate function from within the handleGET method looks like this now:

handleGET:

func (s *Server) handleGET(
	w http.ResponseWriter,
	r *http.Request,
) {
	path := r.URL.Path
	if path == "/" {
		s.renderTemplate(
            w, r, nil,
            "layout",
            "dist/layout.html",
            "dist/index.html")
		return
	}

	noteID := strings.TrimPrefix(path, "/")
	w.WriteHeader(http.StatusOK)
	w.Write([]byte(fmt.Sprintf("You requested the note with the ID '%s'.", noteID)))
}

In the handleGET function I check for the root path of the application and call the newly created renderTemplate function to complete the request. The index.html template doesn't require any model at this point and therefore I kept the data argument nil. The name argument is set to layout because this is the name I chose for the layout.html template at the top of the file.

Once everything is put together one should see the following UI when visiting http://localhost:3000:

Adding a Redis dependency

Now that the service can display a simple HTML page to create a note we need to implement the logic to actually handle the HTTP POST request and save the note on the backend. As mentioned before I'll use Redis for the purpose of this MVP.

First I'll add the required dependencies to the go.mod file and also reference them in the main.go file as part of the import declaration:

go.mod

module github.com/dustinmoris/self-destruct-notes

go 1.17

require (
	github.com/go-redis/cache/v8 v8.4.3
	github.com/go-redis/redis/v8 v8.11.4
)

main.go

import (
	"fmt"
	"net/http"
	"os"
	"strings"

	"github.com/go-redis/cache/v8"
	"github.com/go-redis/redis/v8"
)

In order to access the Redis cache from our Server struct we also need to add a reference in the struct declaration itself:

type Server struct {
	RedisCache *cache.Cache
}

Finally we can initialise a Redis cache object from within the application's main function and subsequently pass it into the server object before launching the service:

redisURL := os.Getenv("REDIS_URL")
if len(redisURL) == 0 {
    redisURL = "redis://:@localhost:6379/1"
}

redisOptions, err := redis.ParseURL(redisURL)
if err != nil {
    panic(err)
}
redisClient := redis.NewClient(redisOptions)
defer redisClient.Close()
redisCache := cache.New(&cache.Options{
    Redis: redisClient,
})
server := &Server{
    RedisCache: redisCache,
}

The code is mostly self explanatory but I'll do a quick run through anyway. Similar to the port we also read the Redis URL (connection string) from an environment variable which I chose to call REDIS_URL. If none was provided then we assume a default Redis instance to run behind the default port 6370. Using the redis.ParseURL function we can convert the "magic" string variable into a strongly typed object which encapsulates all the Redis options. Using the redisOptions object we can then initialise a Redis client. Go has a neat way of deferring the closure of the connection to the end of the function using the defer keyword:

defer redisClient.Close()

In the case of the main function that would be the case when the application shuts down. This is not quite the same but very similar to how one would write code using C#'s using statement.

Finally a cache object is being initialised using the redisClient and then passed into our Server struct.

Saving notes

Now that the Server struct has everything it needs we can implement the actual logic to persist a message. For that purpose we'll extend the handlePOST method, which is the handler which we call when someone sends a HTTP POST to the root (/) of the application.

For good measure let's run some basic validation first:

func (s *Server) handlePOST(
	w http.ResponseWriter,
	r *http.Request,
) {
	mediaType := r.Header.Get("Content-Type")
	if mediaType != "application/x-www-form-urlencoded" {
		s.badRequest(
			w, r,
			http.StatusUnsupportedMediaType,
			"Invalid media type posted.")
		return
	}

	err := r.ParseForm()
	if err != nil {
		s.badRequest(
			w, r,
			http.StatusBadRequest,
			"Invalid form data posted.")
		return
	}
	form := r.PostForm

    //...

If the HTTP request didn't include the application/x-www-form-urlencoded HTTP header then we return a 400 Bad Request response. The same is true if the form data cannot be successfully parsed by the server (because it didn't adhere to the url encoded format).

If everything was okay then we can access the posted form data via the r.PostForm property and assign it to a form variable.

Next I'll attempt to read the message and ttl (time to live) fields from the form and initialise matching variables to hold their values:

message := form.Get("message")
destruct := false
ttl := time.Hour * 24
if form.Get("ttl") == "untilRead" {
    destruct = true
    ttl = ttl * 365
}

A note will either persist for 24 hours or until read (but no longer than a maximum of 365 days). The destruct variable indicates whether a note should get destroyed immediately after it was opened or if it should remain until the end of the 24 hour period.

Using the initialised data we can create a note object of type Note:

type Note struct {
	Data     []byte
	Destruct bool
}

... then inside handlePOST prepare a note which will be subsequently stored in a db:

note := &Note{
    Data:     []byte(message),
    Destruct: destruct,
}

The last step before completing the request is to actually write the note to Redis:

key := uuid.NewString()
err = s.RedisCache.Set(
    &cache.Item{
        Ctx:            r.Context(),
        Key:            key,
        Value:          note,
        TTL:            ttl,
        SkipLocalCache: true,
    })
if err != nil {
    fmt.Println(err)
    s.serverError(w, r)
    return
}

The RedisCache object which our Server is holding makes the storing of the note super easy.

By the way, the s.badRequest(...) and s.serverError(...) functions are further two small convenience methods which I've added to the Server struct to make error responses slightly easier:

func (s *Server) badRequest(
	w http.ResponseWriter,
	r *http.Request,
	statusCode int,
	message string,
) {
	w.WriteHeader(statusCode)
	w.Write([]byte(message))
}

func (s *Server) serverError(
	w http.ResponseWriter,
	r *http.Request,
) {
	w.WriteHeader(http.StatusInternalServerError)
	w.Write([]byte("Ops something went wrong. Please check the server logs."))
}

Anyhow, coming back to handlePOST the last remaining job is to print a friendly message with a link to the newly created note at the end of a successful POST:

noteURL := fmt.Sprintf("%s/%s", s.BaseURL, key)
w.WriteHeader(http.StatusOK)
s.renderMessage(
    w, r,
    "Note was successfully created",
    template.HTML(
        fmt.Sprintf("<a href='%s'>%s</a>", noteURL, noteURL)))

There's a couple new things in this code. First there is the BaseURL variable on the Server struct. This is something that I've added when instantiating the server object in the main function. It makes the Server struct aware of the public URL which should get displayed to the user after a successful POST (after all we won't show them a localhost URL in production).

The other thing is the renderMessage method, which is another helper function which outputs a new HTML page called message.html with an anonymous model:

renderMessage

func (s *Server) renderMessage(
	w http.ResponseWriter,
	r *http.Request,
	title string,
	paragraphs ...interface{},
) {
	s.renderTemplate(
		w, r,
		struct {
			Title      string
			Paragraphs []interface{}
		}{
			Title:      title,
			Paragraphs: paragraphs,
		},
		"layout",
		"dist/layout.html",
		"dist/message.html",
	)
}

message.html

{{ define "header" }}
<style>
    h3 {
        margin: 2rem 0 1rem 0;
    }
</style>
{{ end }}

{{ define "content" }}

<h3>{{ .Title }}</h3>
{{ range $i, $p := .Paragraphs }}
<p>{{ $p }}</p>
{{ end }}

<p><a href="/">Back to home</a></p>

{{ end }}

Thanks to Docker we can easily spin up a new instance of Redis and try out the app so far:

docker run -p 6379:6379 redis:latest

If everything went according to plan then creating a note should return a response like this now:

Retrieving notes

At last we need to implement the logic to retrieve a previously saved note. This can be done inside the handleGET method. I would like to point out that handlePOST and handleGET are obviously not brilliant function names for a larger web application, but for this small app where our service has only ever 3 endpoints to deal with those names are appropriate enough. It keeps a nice balance between structure and simplicity for an app that won't need more than 250 lines of code when finished.

Reading a note from Redis is very easy using its key. In order to get the key we assume that everything that follows the forward slash in the URL will be part of the note ID:

noteID := strings.TrimPrefix(path, "/")

Then we can try to find the note inside Redis using its ID:

ctx := r.Context()
note := &Note{}
err := s.RedisCache.GetSkippingLocalCache(
    ctx,
    noteID,
    note)
if err != nil {
    s.badRequest(
        w, r,
        http.StatusNotFound,
        fmt.Sprintf("Note with ID %s does not exist.", noteID))
    return
}

If the user desired the note to be destroyed after it was opened then we should honour their request as well:

if note.Destruct {
    err := s.RedisCache.Delete(ctx, noteID)
    if err != nil {
        fmt.Println(err)
        s.serverError(w, r)
        return
    }
}

If everything was fine so far then we finish the request by outputting the contents of the note itself:

w.WriteHeader(http.StatusOK)
w.Write(note.Data)

Voila, this completes the handleGET method and the application itself:

func (s *Server) handleGET(
	w http.ResponseWriter,
	r *http.Request,
) {
	path := r.URL.Path
	if path == "/" {
		s.renderTemplate(
			w, r, nil,
			"layout",
			"dist/layout.html",
			"dist/index.html")
		return
	}

	noteID := strings.TrimPrefix(path, "/")

	ctx := r.Context()
	note := &Note{}
	err := s.RedisCache.GetSkippingLocalCache(
		ctx,
		noteID,
		note)
	if err != nil {
		s.badRequest(
			w, r,
			http.StatusNotFound,
			fmt.Sprintf("Note with ID %s does not exist.", noteID))
		return
	}

	if note.Destruct {
		err := s.RedisCache.Delete(ctx, noteID)
		if err != nil {
			fmt.Println(err)
			s.serverError(w, r)
			return
		}
	}

	w.WriteHeader(http.StatusOK)
	w.Write(note.Data)
}

Links

The full source code is available on GitHub and licensed under a permissive Apache-2.0 license.

I have also added a GitHub Action to build and publish a public Docker image on Docker Hub.

Next steps

This web service was just a little toy project to dip my toes into Go. I am not planning on making it much better beyond this point, although if someone would send me a PR I would probably accept it. If I did want to improve it I'd probably swap Redis for a different db and also encrypt the note before persisting it for additional security against database attacks.

How to run

The easiest way to start the public Docker image is by running this docker-compose:

docker-compose.yml

version: "3.9"

services:
  db:
    image: redis:latest
    ports:
      - "6379:6379"
  web:
    image: dustinmoris/self-destruct-notes:1.0.0
    environment:
      - PORT=3000
      - REDIS_URL=redis://:@db:6379/1
    ports:
      - "3000:3000"
    depends_on:
      - db

Terminal

docker compose up

Closing words

I love how quickly I was able to build a web service using Go. The standard library and the net/http package in particular are super powerful and include all the functionality which one would expect to build rich web applications with ease. I also admire how there was no unnecessary boilerplate to begin with. The web project feels very light-weight and close to the metal. There is no layers of bloatware, no hard requirement or pull towards a certain paradigm and no unnecessary additional packages for simple use cases like mine. At the moment the entire web application is essentially a single main.go file and less than a handful of HTML pages. However, it doesn't mean that the project is unstructured or "dirty" in any way. It's production ready code as far as I am concerned. It was very easy for me to expand on structure as the application grew, keeping things easy to understand and easy to maintain. Every single line of code has a purpose and adds functionality which makes sense.

If the application continued to grow then I could apply the same principles as I already did before and further break down the handleGET and handlePOST methods into smaller parts. I would continue to keep the main routing logic inside the ServeHTTP method of the Server struct, which is the main entry point of the (web) app.

I really appreciate the simplicity of Go and how it allowed me to add just enough complexity when I built out the app. For me Go enables "just-in-time-complexity" coding which feels very refreshing coming from .NET. I can start a project without any at all and only add more as and when I need.

Fund OSS through package managers

Tue, 28 Jun 2022 00:00:00 +0000

Open source software has become an integral part of our lives. Personally I don’t know a single app, website or other digital product which doesn’t make use of at least one open source project in their stack. We all know the value and importance of open source software and yet many open source maintainers struggle with its sustainability. A big part of that struggle is the inability to cover the cost for the time and effort put into developing and maintaining a successful OSS project over a long period of time. Of course, very few if any software developers start an OSS project with the intent to make an income from it, not least because it would be a very bad business idea, but if a project grows in users and (user) demands over a long period of time then it inevitably turns from an occasional I-do-what-I-want hobby into something more akin to a real world job.

Unsurprisingly nobody likes to put in hard work for free. Unfortunately most OSS developers never manage to make a dime from their projects.

There may be many reasons why OSS maintainers find it difficult to get compensated for their hard work, but fundamentally I believe that the issue lies within the fact that there's no easy way to pay for OSS. Of course we have a few fringe projects which try to fill the void, but none of them are integrated into the daily tools which developers rely on every day. It requires some good will and someone going out of their way in order to make a financial contribution towards OSS. Ideally it should be the other way around.

Donations don’t work

Personally I don't think that the model of donations work that well. Initiatives like Patreon, OpenCollective or GitHub Sponsors sound very noble and for very few people it might even generate enough income to compensate their work, but for the vast majority of OSS developers it just doesn’t help at all.

The problem with donations is that it promotes the wrong idea. It suggests that OSS is free by nature and only if a user feels charitable enough they might contribute a dollar or two to show some support. Fundamentally there is no set expectation to donate towards OSS and as a result most consumers won't pay anything at all.

It is important to understand that users are not to blame. They simply follow a social contract which has been understood by people since the beginning of time.

Digital street artists

Free open source projects are the digital equivalent of real world street artists.

We’ve all been to a town square with (free) street performances before. Only a small fraction of the audience will throw in a coin at the end of the show. Nobody thinks of you badly if you don't. After all nobody asked the street performers to put on a show. In most cases street artists will even begin their performance without an audience to start with. Only if the street artist performs well enough people will start to slowly congregate around them. It is not that different to how most open source projects operate today. The social contract is the same. It is the street artists who are grateful for the (voluntary) donations from their audience and not the other way around.

As a charity you need to beg

I often hear the argument that the entire software industry relies on OSS projects and therefore everyone should pay. I agree with the sentiment (to some extent) but then donations are the wrong model to achieve this goal.

Let’s look at another example from a different field. Five out of ten adults (in the developed world) will develop cancer at some point in their life. Yet less than one out of ten people will ever donate a single penny towards cancer research during their time. Nine out of ten people will be affected by cancer indirectly at the minimum. Despite cancer being a major issue in the developed world only very few people choose to voluntarily contribute towards the cause.

On the other hand people contribute towards a million other medical treatments and research projects by paying national insurance and income tax. The difference? It's done automatically via existing systems which we have in place.

What about Wikipedia? How many people use Wikipedia every day versus how many people choose to donate at least once a year? My point is that a user's willingness to pay voluntarily for a product or service is rarely related to its usefulness. Frequently people simply choose the path of least resistance. In the case of a free open source project that means no financial support at all.

Neither Cancer Research UK or Wikipedia would exist if they didn't spend a lot of time and energy every year to actively beg people for their support. This is not a model which we can reasonably expect from an average OSS developer.

Donations simply don't work.

Toxic expectations

Sometimes donations can be toxic too. As mentioned before, a donation is seen as a charitable act by the user towards the OSS maintainer. Open source developers often feel very grateful for their users' support and don't want to let them down. That feeling of gratitude and self imposed expectations can put maintainers under immense pressure too. It's hard to prioritise your own personal needs if someone who voluntarily pays for your coffee asks for your time. Even if one's supporters are the most understanding, kind and friendly people, it can be still extremely mentally taxing if one thinks that they're letting their supporters down. It takes a mentally strong individual to not fall into this trap. I wouldn't be surprised if it is often the mental strain rather than the physical work which drives OSS maintainers into the burnout stage.

Of course the situation isn't helped by the fact that some donors actually believe that their financial support entitles them to special treatment in return. The highest paying customer often demands the best table in the house. If someone tips their taxi driver well they probably expect help with their luggage also. I am not saying that this is right or wrong, but simply stating the fact that it's common enough for many people to believe that their donations buy them a favour too.

Whether it is intentional or not, the model of donations (or often camouflaged as "sponsorships") can put open source maintainers on the back foot and sometimes even be felt as more damaging than helpful under certain circumstances. Those concerns should not be taken lightly and I wholeheartedly believe that donations (or sponsorships) are a complete inappropriate way to sustain open source.

Convenience drives behaviour

It turns out that the vast majority of cinema goers don't mind to pay for films at all. There was a time when people were not so sure. In the late 90s and early 2000s ripping movies off the internet was so widely spread that one might have thought the opposite was true. However, the problem was never people's willingness to pay for films. Initially it just happened to be so much easier to download a movie illegally off the internet than buying a legitimate DVD. Today the tables have turned. Finding and ripping movies off the internet takes so much more effort than buying them on the Google or Apple store. Amazon Prime, Netflix, Disney+, Google Play or Apple TV have made film watching so incredibly convenient that for 99.9% of consumers any other alternative is just not worth their time.

We are a very busy society and convenience drives how we behave.

I believe that we should take this valuable lesson to the open source ecosystem an apply it in a similar way!

Package managers reimagined

Package managers are the central point of consuming third party OSS packages. It is the most straightforward and convenient way to publish, install or update an open source library for most programming languages. Package managers know everything about both sides of the transaction. They know who publishes a package and who consumes it. They know how many times a package gets installed in total or per party. They are already an established platform that developers could hardly live without today. They have fantastic and simple to use CLI tooling which could be easily extended to help with the sustainability of open source.

Firstly, package managers such as npm or NuGet should introduce the ability for users to sign-in with an account. It should be an optional feature so that anonymous access remains the default out of the box experience. However, this could enable package authors to decide if their package may be consumed by anonymous users or authenticated user only. In the simplest form nothing would change. A package author could simply specify that their OSS library can be consumed by anyone without any limitations at all, just like it is today. However, in addition they could start introducing pricing models based on a variety of options:

Disable anonymous access (only logged in users can install a package)
Allow first X amount of installs per user for free (e.g. for exploration)
If user installs package more than X-times they require to purchase a license

If for whatever reason a package is prohibited from being installed then the CLI should output the reason just like it does for a variety of non commerical reasons already today.

Examples:

> Please sign in to install package X

> You've exceeded Y free installs for package X. Choose one of the following license options to continue:

1. $49 for a lifetime license
2. Don't install package

Purchasing a license should be as straightforward as buying a subscription in the Apple Store. Those licensing fees could be yearly, monthly or a one off fee. I haven't thought of all the possibilities but over time package platforms should evolve to give their users increasing flexibility to pick the right model for themselves.

Flexible pricing

Other ideas would be to limit the installation of a package only in certain scenarios. For example one may consume a package anonymously and without any limitations as long as they consume it from their development machine, but they may need a license when installing it from a CI build.

Perhaps a package remains entirely free but users who want to have early access to bleeding edge features will require a license for the consumption of pre-release versions. Maybe a user that is a verified student will not be restricted at all.

Another option could be to offer the same package licensed under different conditions. A free download could be available for a GPL licensed version and differently licensed version for a fee. The possibilities are endless and each package author could pick an appropriate model between them and their user base.

Make license acquisition easy via CLI

The biggest challenge with introducing a commercial layer into the already existing complexity of package management is to not make it so cumbersome that it will become unusable. The key is that the vast majority (like 99%) of use cases should become satisfiable via the CLI. Nobody wants to have to open the browser, log into some platform and start entering card details or go through checkout screens in order to use a third party library. Package managers would have to capture a user's payment details once in advance and make the whole experience almost unnoticeable. There is already a good precedence for this model, although it might not be 100% transferable to package management it would be a good point to start with. I can already provision, update or extend expensive cloud infrastructure from the comfort of my CLI and at no point am I confronted with pesky payment or checkout forms. I believe a very similar experience could be introduced into package management in order to make OSS more sustainable!

Downstream dependencies

Downstream dependency management could pose another challenge which needs to be carefully thought through. Again, there are many ways how this could be solved, but in its simplest form one could say that the end user must hold a valid (free or commercial) license for all downstream dependencies. Personally this feels the most straightforward and least ambiguous model and also the easiest to start with from a package management platform's point of view.

Users can still circumvent and cheat

What if users cheat?

Yeah so what? There will always be a minority that will go out of their way to circumvent whatever restrictions have been imposed on them. However, they will have to jump through annoying hoops and take a path of constant resistance in order to achieve their goal. They will not be able to rely on the leading package managers for their language of choice. They will not be able to receive frictionless updates to their dependencies. They might have to run their own hosting or pay a fee elsewhere to create a back alley dependency channel. They will have to jump through additional hoops to stay anonymous or risk getting sued by the original package author or a big platform like npm itself.

If the legitimate path of acquiring a third party OSS package has been deeply engrained into developer's every day tools and made ridiculously easy then most programmers will simply not bother to go against the grain. It has worked for books, music and movies and there is little reason to believe that it won't work for the open source community either.

Final thoughts

Apart from package managers other platforms can play an active role in promoting the sustainability of OSS as well.

GitHub for example could allow repository owners to disable issue creation unless a user pays a minor fee. Users who submitted a successful pull request which later gets merged could get a credit towards their account which in turn can be used as payment as well.

Maybe GitHub can create a new tab for "development requests" to track user desired work. Those could be new features, creating sample projects or expanding in documentation. Repository owners can then set a funding goal for each development request and only commence with the work when the goal was successfully met.

I don't want to claim that I have thought of all the potential pros and cons for each those ideas but I would like to bring two major points home before I will wrap up with my post:

There is no shortage of creative ideas which could immensely improve the sustainability of OSS.
It is the platforms which developers use every day that must get involved. Paying for OSS is only awkward because it has not been normalised through the platforms which we use yet.

GitHub, GitLab, npm, NuGet, etc. the ball is in your court!

Can we trust Microsoft with Open Source?

Sat, 23 Oct 2021 00:00:00 +0000

Oh boy, what a week of .NET drama again. Not bored yet? Read on, but for this one you’ll need some stamina, because this one is different.

Before I start with my actual blog post let me give you a short disclaimer. This is not an issue of some outspoken or sceptical community members making a fuss out of nothing. I know it is easy to see it this way with incomplete information, but trust me on this one, you couldn’t be more wrong if you believed that's the case this time. This is a bigger issue, an issue internally playing out at Microsoft right now as we speak. Many people from Microsoft itself, who have been working really hard to build trust with the OSS community and market .NET as a real viable OSS platform are struggling right now. These are your .NET heroes who you probably dearly cherish and these folks are currently unable to publicly speak their minds. They struggle and they want YOU to speak for them. In fact, they want you to speak out right now and make your voice heard!

You don’t believe me? Don’t take my word for it, but how do you explain this to me?

The hot "Hot Reload" issue

Here is a short summary of what has happened…

For the majority of the last year the .NET Team was hugely focused on improving the “inner dev loop” in .NET. You might have heard many prominent .NET figures use exactly those words on countless public forums, live streams, conferences and public talks. It was one of the .NET Team’s highest priority items for .NET 6:

As such the team worked hard on making lots of great improvements to .NET, the SDK and the tooling around it. One of those big features was “Hot Reload” in the dotnet watch tool. I watched Scott Hunter give a talk and early demos of this feature many months ago when .NET 6 was still in baby shoes.

Hot reload wasn’t a fringe feature that might or might not have made it into the release of .NET 6. It was literally one of the flagship features that was in the making for a long time, and it was complete. After all, you don’t accumulate 1000s of lines of perfectly fine code by accident.

So what happened to it? Well, someone at Microsoft who wields great power has made the decision that features such as hot reload cannot be given away for free as part of the open source .NET SDK anymore. These features must be reserved to proprietary commercial products such as Visual Studio. In fact, there is a bigger internal strategy being formed at Microsoft to make Visual Studio the main IDE for .NET again, because some people at Microsoft are annoyed that Visual Studio Code and other third party tools have been undermining Visual Studio for way too long now. As a result the hot reload feature was ripped out of the SDK in a last minute effort, breaking all promises and conventions of Microsoft’s release candidate policies and announcing that hot reload will be a Visual Studio only feature going forward.

I know what you think. You might think I’m reading too much in-between the lines here, but trust me on this one that I am not wrong. This is exactly what is happening behind the scenes at Microsoft right now and the .NET team wants you to know it even though they can't say it. It’s not by accident that we’ve seen a fairly “coordinated” effort by Microsoft employees leaking a lot of internal infighting with the Verge and other media outlets yesterday.

The issue goes even deeper. Not only will hot reload not make it into .NET 6 or any future version of .NET, the entire dotnet watch tool will be discontinued in an effort to push Visual Studio as a viable product. This was not publicly announced yet and don’t count on anyone from Microsoft to publicly admit that, but if you pay close attention or if you happen to talk to a Microsoft developer privately over a coffee then you might come to that conclusion exactly.

Why the change of direction now? The truth is this hasn’t just happened now. In fact Microsoft has already started to make such subtle moves in the past. For example the Python extension for Visual Studio Code was never open sourced and released as a proprietary product to begin with. The once open source cross platform IDE MonoDevelop was hard forked by Microsoft and rebranded as "Visual Studio for Mac". All improvements and feature work since then have been proprietary and closed source. Yes, that is right, Visual Studio for Mac is a closed source version of the formerly open source MonoDevelop IDE after Microsoft acquired Xamarin. There was no need to turn an already long standing open source project into a proprietary commercial product. Especially since MonoDevelop was available on Linux and Mac and Visual Studio for Mac, as the name suggests, isn't. These subtle moves and many others which went mostly unnoticed have emboldened certain people at Microsoft to further pursue a less open strategy again.

I'm glad I am not the only one who noticed...

Rumours say more is to follow.

Visual Studio Code has been one of the most successful products which Microsoft has ever released. It has become staple to every software developer around the globe. It was a success to everyone except Visual Studio. Have you ever noticed that .NET - a Microsoft owned product - is the worst supported programming platform on Visual Studio Code - which also happens to be another Microsoft owned product? I’ve been complaining about this for years, because to me Visual Studio Code represents the future of .NET and a new gateway into growing the .NET platform beyond traditional die-hard Windows fans. However, Microsoft has been purposefully underfunding OmniSharp for years in an effort to push developers towards Visual Studio again.

Sure, you might think this is not a big deal and completely up to Microsoft to decide where they want to allocate their resources, and normally I would agree, but what if I tell you that internally at Microsoft employees are being actively punished by management if they contribute improvements or bug fixes to OmniSharp (the OSS .NET plugin for VS Code) which is seen as further undermining Visual Studio?

Ouch, that is a big accusation and hard to believe, so why don't you reach out to one of your favourite Microsoft employees who work on .NET or Roslyn or another OSS product under .NET and ask them for a comment? They most likely won’t admit it, because they are not allowed to, but more importantly they also won’t deny it.

As Damian Edwards said before, good that nobody let the cat out of the bag ;)

The big issue here is that Microsoft has an internal struggle going on at the moment. On one hand they want to be seen as a new version of Microsoft who loves Open Source, but on the other hand they want to actively block advances in OSS projects like the .NET SDK which could undermine their own commercial offerings. Microsoft^* doesn’t want features like hot reload to make it into the SDK. They won’t develop cool features like these any more and most frighteningly, they won’t accept pull requests or community contributions which could add these features back into the SDK - and that is a bleak outlook for the open source community in .NET.

*) Microsoft outside the .NET team

So here is my simple question…

Can we trust Microsoft with OSS?

I am not sure. It takes years to build trust and only a few moments to lose it all. Microsoft is a huge organisation and we as outsiders often get to see only a handful of selected people being tasked to spread a certain message via their huge followings in order to create a public image in favour of Microsoft, but what if those people leave?

Do you trust Microsoft with Open Source or do you actually trust people like Jon Galloway, Scott Hanselman, Scott Hunter, Guido van Rossum, David Fowler, Damian Edwards, Miguel de Icaza and a handful of other OSS champions who have been pushing the OSS message internally from the bottom up? What if these people leave .NET? Will Microsoft continue to play nicely with the community?

Would it worry you if Scott Hunter was moved away from .NET and someone new from a different division at Microsoft would have taken over? Remember your answer when you find out in which part of Microsoft Scott Hunter works today.

What can we do?

This is a call to action to all .NET community members out there. Anyone with a tiny bit of clout make your voice heard. Comment and upvote the issues on GitHub. Tweet your dissatisfaction and make sure to tag high profile folks at Microsoft so they see what you think! Send Microsoft a clear message that this undermines the trust which we’ve lent them over the last few years and that the betrayal of this trust can have long term consequences. This is not a threat by any means but really just the reality of the matter. Many developers in technical leadership roles and with hiring powers in their respective organisations have only stayed on .NET due to the big changes which came with .NET Core. People love .NET for what it is today, a more open, cross platform and community led platform. If Microsoft starts to change their hearts again then people will as well.

Please help to send the right message and make your voice heard!

!!! UPDATE 24/10/2021 !!!

Thanks to everyone who read this blog post and shared it on social media or spread the message in other ways. Yesterday evening (London, UK time) something amazing happened which can best be described in a single picture:

Honestly, this felt quite like some biblical moment. The entire community stepped up and made their voices heard. Everyone was respectful and worked together to get a clear message across:

We love the new open and cross platform .NET of today and we won't give up even an inch of it without a proper fight :)

Apparently this message made its way all the way to the top of Microsoft and the result speaks for itself:

Special thanks to Scott Hanselman and Scott Hunter who amongst many others worked tirelessly behind the scenes to make this happen.

Read Scott Hunter's official blog post for more information and don't be too hung up on the exact wording. It's corporate lawyer speak for admitting the mistake and thanking the community. We should take that as a huge win!

Congratulations to everyone!

.NET Basics

Mon, 24 May 2021 00:00:00 +0000

So you want to learn .NET but you are confused about the differences between .NET Framework and .NET Core or what the various versions of ASP.NET are or what the relationship is between C# and F#? If that is the case then you came to the right place. This guide will cover all the basics of .NET and shed some light on the various acronyms and buzz words behind it!

If you are new to .NET and you want to get a holistic overview of the entire platform and what parts really drive the framework and how they relate to each other then look no further because this blog post will cover them all!

Disclaimer: I am a (mostly) self taught developer and a non native English speaker. I have written this guide with the utmost care and to the best of my knowledge, but there is a good chance that I could have made a mistake or misrepresented some details in the guide below. If you find an issue then please be polite and let me know in the comments underneath or email me directly at hello [at] dusted.codes. I appreciate any type of feedback and will do my best to correct any mistakes as soon as I can. Thank you.

What is .NET?

.NET is the top level name of a Microsoft technology used for building software. It is a twenty year old platform which has seen a lot of change and innovation over the years and which spans across many different domains. .NET can be used to develop web applications, games, IoT, machine learning, Desktop and mobile applications and much more. The term ".NET" is often used in a very far reaching meaning and used synonymously for a wide range of smaller parts of the platform such as the original .NET Framework, the newer .NET Core, or languages such as C#, F# and VB.NET.

When people talk about ".NET" they could mean anything from ASP.NET to PowerShell. In order to understand .NET one has to look at all the different pieces that make up the platform and look at them individually to get the full picture.

Tip: The official website for .NET is dotnet.microsoft.com, but a much more memorable URL to all things .NET is dot.net, which will redirect a beginner to the most up to date resources around the Microsoft .NET platform.

C# vs. F# vs. VB.NET

First of all .NET is not a programming language. It's a development platform. In fact, you can use three different officially supported programming languages to develop with .NET:

C# (An object oriented language, very similar to Java)
F# (A functional first language, similar to OCaml)
VB.NET (An object oriented language which is the successor of VB6)

VB.NET and C# were the first officially supported languages for .NET. VB.NET is the successor of Visual Basic 6.0 and whilst extremely similar in syntax yet a different language and incompatible with the classic version of Visual Basic.

C# is a C-like object oriented language which first borrowed a lot of its early concepts from Java and was initially seen as an imitation of it. Today things stand very differently and C# is often being praised for its innovation and modern features which Java trails behind. The name C# is a word play from taking C++ and adding two more + signs to it, so that it forms the hash character, or as .NET developers like to call it, the "sharp" sign.

Fun fact: Initially C# was developed under the name "Cool" which stood for C-like Object Oriented Language but then was renamed to C# for trademark reasons.

A few years after .NET's initial release Microsoft Research and Don Syme developed a completely new language called F#. Unlike C# and VB.NET "F-Sharp" was designed as a functional first multi paradigm language which took a lot of inspiration from Ocaml, Erlang, Python, Haskell and Scala at the time. For many years F# has been the leading source of inspiration for new features in C# and was the first .NET language to introduce features such as Linq, Async and pretty much all of the new features starting from C# 7 and onwards. It was also the first language to go open source before all of .NET became public.

All three programming languages are primarily statically typed languages which means that the compiler will provide type safety checks during development and compilation. Those type safety checks can prevent many hard to catch runtime errors which could otherwise occur.

The opposite to a statically typed language is a dynamic one. Famous examples of dynamic languages are Python or JavaScript. For example, in C# one cannot accidentally assign a string value to a float variable but in JavaScript this would be totally fine.

Note: C# has the dynamic keyword which allows a developer to introduce dynamic behaviour into the language, but it remains an extremely rarely used feature which has been mostly reserved to excpeptional cases where interop with other systems is required.

IL Code and the CLI

A term which often gets mentioned alongside .NET is so called "IL Code". IL code stands for intermediate language code and is the code to which C#, F# and VB.NET get translated to when an application gets compiled.

In .NET it is perfectly possible (and not that uncommon) to have multiple projects of different languages mixed into a single solution. For example one can have several F# projects sitting alongside C# and talk to each other, forming a larger application.

This is possible because the CLI (Common Language Infrastructure, not to be mistaken with the "command line interface") is a language agnostic interface which allows code to be executed on different architectures without having the code to be rewritten for each specific platform.

At this point this might sound a little bit confusing but it will become much clearer when I will explain the .NET CLR in the next part.

Tip: One can use sharplab.io to translate .NET code into IL code and get a glimpse into the inner workings of the compiler.

SDK and Runtime (CLR)

Software programmers write applications with the help of very high level languages which have human readable constructs such as if, else, return, while, foreach, public, private and so on. Of course this is not how a binary machine works and therefore every application code which was written in a high level programming language must get translated into native machine code at some point in time. We can broadly characterise a programming language into "compiled" and "interpreted" languages and "managed" and "unmanaged" code.

Compiled vs. Interpreted

An interpreted language is a programming language where the code which gets written by a developer is the final code which gets shipped as the application. For example, in PHP a developer doesn't compile .php files into something else. PHP files are the final artefact which get shipped to a web server and only when an incoming request hits the server the PHP engine will read the .php files and interpret the code "just in time" to translate it into lower level machine code. If a developer made a mistake then they will not know until the code gets executed. This has the benefit that a developer can quickly modify raw .php files and get a quick feedback loop during development but on the other hand it means that many errors will not be found until the server runs the code. Typically an interpreted language also takes a slight performance hit because the runtime has to do the entire compilation at the time of execution.

In contrast a compiled language does all of the compilation work (or parts of it) ahead of time. For example Go requires code to be compiled directly into native machine code before an application can run. This means that a developer will be notified by the compiler about any potential errors well ahead of execution, but equally it means that changes to the code require an additional step during development before they can get tested.

.NET is also a compiled language, but unlike Go or Rust it doesn't compile directly into native machine code but into the intermediate language (IL Code). The IL code is the artefact which gets shipped as the final application and the .NET runtime will do the final compilation step from IL code to native machine code using the JIT (just in time compiler) at runtime. This is why .NET applications get deployed as .dll files and not raw .cs, .fs or .vb files. This is also how C# and F# and VB.NET can talk to each other because they communicate at the IL level and not before. The model which .NET follows is popular with many other programming languages and provides some notable benefits. It is usually much faster than interpreted code because much of the compilation is done well in advance, but then still slightly slower than low level languages such as Rust or C++ which can compile directly into native machine instructions.

The important take away is to understand why .NET has .dll files and that .NET requires a runtime to do the final compilation from IL code to machine code with the help of the JIT. This runtime is called the "CLR" (common language runtime) in .NET.

Fun fact: The source code compiler for C# and VB.NET is called Roslyn, whereas F# has its own compiler called the Fsharp.Compiler.Service and a console application called fsc (fsharp compiler) which can be used to invoke the FSharp.Compiler.Service from the command line.

Managed vs. Unmanaged

Managed code is simply code which requires the execution under the CLI (Common Language Infrastructure). Unmanaged code is code which runs outside the CLI. Without going too far into detail the point is that in theory one can invent any new framework (or language) which can run on .NET as long as it implements the CLI. If that is the case then the language can be translated into IL code and get executed under the CLR (Common Language Runtime). All these abstractions make it possible to not only have more than one language for .NET but also multiple frameworks such as .NET Framework, .NET Core or Mono (more on this later).

Fun fact: Microsoft has also developed a dynamic language runtime (DLR) which runs on the CLR and therefore can support dynamic languages on top of .NET. The most notable examples are IronPython and IronRuby, which are implementations of the Python and Ruby programming language for .NET.

SDK vs. Runtime

Now that the purpose of the CLR (.NET runtime) is a bit clearer one can also explain why .NET is available as two different installations:

SDK (Software Development Kit with the runtime)
Runtime (only)

The runtime is what an end user's machine or a server must have installed in order to run a .NET application. The SDK is the software development tool chain which allows programmers to develop .NET applications. In layman terms, the SDK is what produces a .dll file and the runtime is what can run it.

This is why the official .NET page offers both options to download:

Software developers must download the .NET SDK in order to build .NET applications, but a web server or end user machine should only install the .NET runtime which is the much smaller installation.

Another point which becomes very clear from the download page above is the loose relationship between the different .NET languages. As one can see C#, F# and VB.NET are at complete different stages in their life and therefore versioned differently. The languages can evolve independently and introduce new language features to their own liking as long as the associated compiler translates the source code into valid IL.

A careful observer might have also noticed the disparity between the .NET version, the .NET SDK and the .NET runtime. The official .NET version normally refers to the .NET runtime version, because that is essentially the final execution runtime which needs to be installed on a machine. The SDK can have a different version because the development tool chain can improve faster than the runtime itself and support new features and better development workflows whilst still targeting the same version of .NET.

Side note: Not every programming language requires an SDK and runtime. For example languages such as Rust or Go, which directly compile into native machine code, don't require a runtime. These languages only have a single download option available which normally represents the SDK for building software. Coming from one of these languges can make .NET feel unusual, but in essence .NET is not any different than for example Java, which also has the JDK (Java Development Kit) for building software and the JRE (Java Runtime Environment) for running it.

Summary of .NET components

So what is .NET?

Coming back to the original question of what is .NET? .NET is the combination of all of the different parts which have been described above. It's a platform which consists of languages, a CLI, the runtime (CLR) and an SDK for building software.

To make matters worse, .NET comes in three official versions:

.NET Framework
Mono
.NET Core

Even though .NET Framework, .NET Core and Mono are officially labelled as "frameworks", these flavours of .NET are simply different implementations of the CLI.

Fun fact:

In theory all three frameworks come with slightly different CLRs:

CLR (Original .NET Framework CLR)
Mono CLR (Mono's implementation of the CLR)
CoreCLR (The actual name of .NET Core's CLR)

The CLR is the platform specific part of .NET since it has to translate platform agnostic IL code into an architecture's specific machine code. This is why the .NET SDK and Runtime downloads come in so many different versions.

In order to support .NET on a completely new architecture (such as Apple Silicon for example) Microsoft only has to build a new architecture specific CLR (e.g. for macOS Arm64) and the rest will continue to work.

What is .NET Framework?

.NET Framework is the original .NET. It's the platform which was first released twenty years ago and which only worked on Windows and was regularly updated and shipped as part of Windows.

.NET Framework came in these major versions:

.NET Framework 1.0
.NET Framework 1.1
.NET Framework 2.0
.NET Framework 3.0
.NET Framework 3.5
.NET Framework 4.0
.NET Framework 4.5
.NET Framework 4.6
.NET Framework 4.7
.NET Framework 4.8

.NET Framework 4.8 was the last official version of .NET Framework and has been superseded by .NET 5 since then. There will be no new version of .NET Framework any more and all future development is conducted on .NET Core (renamed to .NET 5) and its future versions.

The original .NET Framework is what most people think of when they have negative connotations towards .NET. It was tightly coupled to Windows, the CLR could only run on Windows or Windows Server with IIS, it required Visual Studio to develop on it and it had absolutely no cross platform support. It was also relatively slow in execution and slow to evolve.

Overall it worked well on Windows but started to increasingly lack capabilities and meet modern software development demands.

What is Mono?

When Microsoft was still at war with Linux and had absolutely no appetite to support .NET on any other platform than Windows a crazy-genius person called Miguel de Icaza decided to develop the Mono Project - an open source cross platform alternative to .NET Framework.

Mono allowed Linux, macOS and other Unix developers to write and execute .NET applications on non Windows systems. The Mono project was led by Miguel as a collaborative open source project and attracted many supporters and even investors (through Xamarin) from all around the world. Despite its initial struggles and not having had any help from Microsoft, which meant it was always lagging slightly behind the latest version of .NET Framework, Mono still managed to become a very successful standalone framework which became very popular with the ALT.NET (alternative .NET) movement and matured incredibly well over the years.

Fun fact: Mono was only possible because it implemented the Common Language Infrastructure (CLI) which Microsoft released as an open standard (ECMA-335) in December 2000.

Only in 2016 when Microsoft changed its internal culture and started to adopt a more favourable relationship with the open source community they decided to acquire Xamarin (the company which owned Mono) and hired Miguel de Icaza as the lead.

Today Mono is part of the official .NET eco system and powers strategic products such as Microsoft's mobile development toolkit and Blazor, a .NET WASM (web assembly) runtime for running .NET in the browser.

What is .NET Core?

.NET Core is Microsoft's latest reinvention of the legacy .NET Framework with the promise of true cross platform support and improved performance. Initially .NET Core started as a complete re-write of the .NET Framework and was initially focused on the "core" parts of the framework which were needed for running web and console applications.

Since the release of .NET Core 1.0 Microsoft has steadily iterated over the product and eventually caught up with the original .NET Framework which led to the renaming of .NET Core to simply ".NET" again. The current version of .NET Core is called ".NET 5" and .NET 6 will be released later this year.

.NET Core has truly lived up to its promises and revived the entire .NET eco system from the first day. It runs on Linux, macOS and Windows, it is being openly developed on GitHub and runs under an OSS license, it's a more light-weight standalone product which is decoupled from Windows and incredibly fast in comparison to .NET Framework.

Fun fact:

.NET Core's CLR is called CoreCLR and has its own JIT which is called RyuJIT. Both projects are also open source and available on GitHub:

The term Ryu means dragon in Japanese and is a reference to a book about compilers famously known as the "Dragon Book".

Today .NET Core represents the foundation of all new innovation in .NET and is being released on a yearly schedule, with every second year producing an LTS (long term support) version (.NET 6 going to be the next one).

Without doubt, any new .NET developer should start with .NET 5 or higher when learning .NET.

What is .NET Standard?

.NET Standard is basically a short lived invention from the past, but because it still lingers around many corners of the internet it is worth quickly touching on as well.

Before .NET 5 became the unification of .NET Framework and .NET Core Microsoft created a specification called the ".NET Standard" which was meant to help developers to build Framework and Core compatible applications. .NET Standard was not a framework itself, but just a blueprint (specification) of available APIs.

It worked as following, the higher the version of .NET Framework was, the higher it would implement a version of .NET Standard. The same was true for .NET Core. This meant that a .NET developer could target a specific version of .NET Standard and then be confident that it would be compatible with certain versions of .NET Framework and .NET Core.

It was a worthwhile idea but unfortunately has always caused some confusion with .NET developers and finally got phased out with the unification of .NET 5.

Why is there no .NET Core 4?

.NET 5 became the first version of .NET to unify .NET Framework and .NET Core. As such it had to pick a version number which would reflect the natural progression of both frameworks and had to be higher than .NET Core 3.1 and .NET Framework 4.8. Simple as that.

Tip: You can visit versionsof.net to get an overview of all existing versions of .NET, including .NET Framework, .NET Core and Mono. It also highlights which versions are in long term support.

What is ASP.NET?

ASP.NET is the name of .NET's web platform. It is a collection of .NET libraries to build rich web applications and comes with .NET Framework. ASP.NET inherited its name from ASP (Active Server Pages), which was Microsoft's initial server side scripting language for dynamic web pages. ASP was an interpreted language just like PHP and pretty much a Redmond copy of it. ASP.NET on the other hand is an object oriented framework which compiles into IL code like everything else in .NET Framework. It was first released with .NET Framework 1.0 and is only available on Windows.

Much like the rest of .NET Framework, ASP.NET is mostly seen as a legacy platform, which is tightly coupled to Windows and requires hosting in IIS (Internet Information Services) - a proprietary Microsoft web server.

What is ASP.NET Core?

For a long time Microsoft was bleeding existing developers to new emerging technologies, such as Node.js, Docker or the Cloud. These tools were purposefully built to tackle the problems of modern web demand and for the most part incompatible with Microsoft's Windows centric ASP.NET. As a result Microsoft decided to develop a new version of ASP.NET with the release of .NET Core. The aim was to provide a more light weight, composable and cross platform compatible web platform which could compete with other technologies on a level playing field. ASP.NET Core was released with the first version of .NET Core and remained the main focus of its initial releases. It played a huge role in the success of .NET Core and the adoption of .NET by a whole new generation of developers, scoring consistently high in StackOverflow's yearly developer surveys.

ASP.NET Core is the future of web in .NET and often forms the baseline library for many other web frameworks too. One of those is ASP.NET Core MVC, an object oriented model-view-controller framework sitting on top of ASP.NET Core, which allows developers to build rich web applications in an object oriented class driven approach. Other notable ASP.NET Core web frameworks are Carter, Giraffe, Saturn, Freya or Falco, which have a slightly more light-weight and functional nature to building web applications in .NET.

In addition there are also .NET web frameworks that don't require ASP.NET Core at all. Famous examples of standalone web frameworks are WebSharper or Suave.

Last but not least ASP.NET Core can also be used completely on its own in a more bare metal approach. It is a very popular choice and something which the ASP.NET Core team is currently focused on and will probably evangelise more in the future.

Where do I start?

Unless someone has a very good reason not to, everyone should start with .NET Core (now .NET 5). As a developer one should download the latest .NET SDK and familiarise themselves with the dotnet command line tool.

The most important commands are dotnet build, which will restore dependencies and build an application, dotnet run which will build and launch an application and dotnet test which can be used to run some unit tests.

The easiest way to get started is by creating a simple console application:

dotnet new console

If someone wants to jump straight into web development then I'd recommend to begin with an empty ASP.NET Core application:

dotnet new web

This is a good starting point to slowly explore ASP.NET Core as a whole and learn about the architecture of the framework and how to compose bigger applications.

A great resource to learn ASP.NET Core and find out about different project types is practical-aspnetcore. Another fantastic resource is The Little ASP.NET Core Book, a short free e-book to help people learn about ASP.NET Core! For functional developers the SAFE Stack is a great place to get started too!

Who is dotnet-bot?

When Microsoft started to open source .NET they didn't want that all of the existing source will get attributed to a single person. Consequently Microsoft created a new GitHub user called dotnet-bot as a placeholder for the initial commit. Later it has evolved into .NET's official mascot:

You can design your own dotnet-bot mod at mod-dotnet-bot.net and also find existing swag in the official .NET Foundation repository.

Why is everything .NET?

At last one might ask why is everything called something something .NET? Isn't that causing a lot of confusion and part of the reason why guides like this have to be written in the first place? Well yes, but it's also important to remember that Microsoft is still a big corporation after all. That means that things have to be complicated enough to justify corporate prices and policies and Microsoft also doesn't want to disenfranchise all the corporate consultants who have made an entire career out of distilling everything .NET.

Just think of all the certifications alone!

Jokes aside, there is no real reason why everything is called something .NET. Someone at Microsoft probably really likes the name .NET and because they are boss everything will remain and continue to be .NET until they eventually retire :)

Useful links

Finally a list of some useful links:

.NET Homepage
.NET Blog
.NET Conversations
C# Documentation
F# Documentation
VB.NET Documentation
NuGet.org (npm for .NET)
.NET Foundation
Live .NET (.NET community stand-ups)
Versions of .NET
Themes of .NET (High level topics which the .NET team is working on)
Sites of .NET (Find all official .NET pages in one place)
Discover .NET (.NET community resources)
BuiltWithDot.Net (Collection of projects which have been built with .NET)
.NET Ketchup (Collection of weekly .NET news)

You don’t need Docker

Mon, 12 Apr 2021 00:00:00 +0000

Note: In this blog post I use the terms Docker and containers interchangeably. I know that Docker is only one of many container technologies and not always the best suited one (e.g. Kubernetes), but for the purpose of this blog post I don't differentiate between them.

„You don’t need Docker. I started my business on a small server under my desk. It took me more than 10 years before I reached the scale where I needed something like Docker. You’ll be fine for a very long time before you’ll have to worry about it!“

Who has never heard someone say something like this?

I hear it all the time. At least once a week I see someone tweet or blog about how Docker is not really needed and how they managed to get away without it. To be honest they are not wrong. Nobody really needs Docker, but then again “need” is a very strong word. The real question is “Do you want Docker”?

A different world

It is true, many successful websites and web applications started without Docker. Many also started without the “Cloud”. Some probably even started without NoSQL databases (there was a time when MySQL and Oracle were king), no Redis, no SendGrid or MailChimp, no Stripe or Braintree, no Angular or React, definitely no Vue, no serverless functions, no queues and distributed systems, no CSS frameworks, no CI/CD pipelines and probably not even virtual machines! Heck, some websites probably didn't even use jQuery or JavaScript to begin with!

However, would anyone really want to start a new internet business without these tools today? I don't think so, at least not if they aim for success!

It's easy to forget but when companies like MySpace or Facebook started the world was a very different place. We had no social media (that’s kind of obvious from this example), no iPads and iPhones, no smartwatches, no home speakers, we had no fibre optic cables leading to our homes and we didn’t have mobile broadband either. The aspiration of internet domination wasn’t a thing yet. Even when Facebook already reached huge success they were still only one of many other social networks on the web. We had many individual (often national) versions of something similar to Facebook for a very long time before the world wide web became a little bit less wide again. Internet usage was very different too. Psy didn't even break YouTube yet.

The world, our relationship with the internet and people's expectations were completely different than what they are today.

Instant scale was not a threat

Do you remember online bulletin boards (BBS)? Before Reddit we had many independent self-hosted internet forums. There was a time where there was no WordPress or Medium. The internet started off as a truly decentralised web with many independent websites, blogs, communities and even multiple search engines before Google took over. Internet users didn't all congregate in the same places then.

Websites had time to grow. The danger for an indy blog seeing traffic spikes from 3 users per week to tens of thousands of users in a single day was just not a credible threat. Today it doesn't matter how fringe or unknown a website is, any page could suddenly end up on Hacker News and learn what it means to get the infamous Hacker News hug of death. Maybe a few years ago it was possible to delay the thought of scale to a later stage, but today this is not possible if one wants to reap the benefits of sudden success.

A more patient crowd

The first time I used an instant messenger was at the time of ICQ. The first time I downloaded music was from Napster. Then I switched to KaZaA, then LimeWire, then eDonkey and later to Giganews which was a popular Usenet at that time. What they all had in common was the true nature of a decentralised web. They were all built on so called peer-to-peer networks. When my friends went offline then I couldn't send them a message any more. Messages just wouldn't arrive. There would be no connection and it would time out. When enough "peers" turned their computers off then my downloads would pause. It was totally normal to download content over a period of multiple days if not weeks. There was absolutely no expectation for things to happen in an instant moment.

Nowadays nobody would accept a download to take longer than a few seconds let alone a couple weeks. Patience has come down and expectations went up high. What was once a luxury experience is now the baseline bar. If a video doesn't stream in at least 1080p then it might as well never happen. If the quality is not right then people turn away. Startups, indy hackers, open source projects and even hobbyists cannot afford to offer a degraded service if they want to get traction in current times.

The internet was a toy

When I got my first computer the internet was nothing more than a toy. My relationships with my family and friends did not rely on the availability of WhatsApp. Jobs were not impacted if StackOverflow, GitHub or Slack were a bit slow. Today I can't even book a doctor's appointment without going online.

As we have become increasingly more dependent on the web, service providers have gained a higher responsibility in keeping their services alive. Today it's rather questionable if a business can still offer a meaningful SLA with a server under someone's desk.

Tolerance towards failure

Tolerance towards failure is another benefit which web applications had in the past. Today not so much anymore. Nobody expects Uber to be down. Binge watching is only possible because Netflix never goes offline. Music never stops playing when you're on Spotify. And if it did then people would lash out.

We've become so used to the high quality and availability of services that no glitches in the system go unnoticed anymore. No failure, no scalability issue and no data loss get past users without people grinding their teeth or writing angry tweets. Every incident has a lasting effect and can limit one's future potential of growth. For example, I have never hosted anything on GitLab myself but I sure know that they are infamous for being so awfully slow or losing production data without full recourse.

The network effect

Although everyone likes some quick gains, nobody likes to see their business outgrow their own ability of keeping up with demand. Scale plays a huge part in that realm. We are so interconnected that unless someone launches a product in a private invite-only group they won't be able to predict (or control) how fast they will grow. The internet has its own mind and nobody knows who will be famous tomorrow and what will go viral the day after. An innocent tweet, a short post on Reddit or a routine launch on Product Hunt can shift a new startup from 0 to 10,000 users in the span of 3 months. That level of virality is insane. Imagine having 10,000 willing customers on your doorstep and they can't sign in because someone told you that you won't have to think about scale for a good while. Don't become a victim of your own success.

Do you need want Docker?

Nobody really needs Docker (or containers per se) and I'm not going to claim that containers are the perfect silver bullet to all the issues listed above, but it remains an incredibly powerful tool which can address many of today's challenges in a very time and cost effective way. Sure there is an upfront investment to be made in learning container technology and putting it into practice, but by no means is it any harder or more time intensive than learning a new CSS framework or the latest flavour of JS. If anything, skills like Docker are much broader applicable and more transferable between programming languages, tech stacks and jobs.

Containers make builds predictable, they make deployments reliable and they make horizontal scaling a breeze. Containers are a great way of providing stable and backwards compatible APIs whilst keeping code complexity low. Containers can reduce infrastructure cost by running multiple applications on the same box. They can accelerate a team's productivity by running different feature branches at the same time and launch testing environments independent of hardware. Containers make blue green deployments mainstream and help to keep downtime low.

The question is, why would you not want Docker?

Using .env in .NET

Sun, 10 Jan 2021 00:00:00 +0000

.NET (Core) comes with a lot of bells and whistles. One of them is the sheer amount of managing application secrets and settings. Developers have a variety of options from which they can load application settings using the ConfigurationBuilder class. The official ASP.NET Core documentation lists all options as following:

Settings files, such as appsettings.json
Environment variables
Azure Key Vault
Azure App Configuration
Command-line arguments
Custom providers, installed or created
Directory files
In-memory .NET object

Additionally .NET developers can choose between strongly typed configuration classes, IOptions<T> wrappers, IOptionsSnapshot<T> wrappers or the IOptionsMonitor<T> interface to access their settings. Theoretically there are also the IOptionsChangeTokenSource<T>, IOptionsFactory<T>, IOptionsMonitorCache<T> interfaces and the OptionsManager<T> class, but most users will never need to use them.

Most modern cloud based applications don't even require half of those features. If anything the ASP.NET Core options pattern can feel a little bit bloated which can overcomplicate an application and make it more difficult to understand for people outside one's team. The much simpler and often entirely sufficient alternative are environment variables!

Tip: If you would like to learn more about the different configuration implementations in ASP.NET Core then check out Andrew Lock's blog where he wrote about several of the mentioned interfaces and explained how and when to use them.

Environment variables

In the cloud most settings are configured via environment variables. The ease of configuration, their wide spread support and the simplicity of environment variables makes them a very compelling option. Setting environment variables during development is a little bit more tricky though. It's not any harder than in the cloud, but it's significantly more inconvenient when someone wants to quickly add, remove or edit a variable. Additionally there is a risk of collision when working on multiple applications at the same time. Environment variables like LOG_LEVEL, SECRET_KEY or WEB_PORT are common enough to appear in more than one project. Having to constantly change those values when switching context can become tiresome. Luckily environment variables can be configured at different levels. They can be set on a machine level, user level or for a single process. The latter is the preferred solution during development. Dotenv (.env) files are a great way of making that easy!

The .NET way

Before explaining "Dotenv" files let's take a quick look at how configuration is typically done in .NET 5 (Core).

The framework strongly prescribes developers to create an appsettings.json file in the root of their project and configure their application settings in JSON:

{
    "Logging": {
        "Level": "Debug"
    },
    "Foo": "foo",
    "Bar": "bar",
    "Server": {
        "Port": 8080,
        "ForceHttps": true
    }
}

However not every cloud environment makes editing JSON files in a deployed application's directory easy and therefore most .NET developers still end up using environment variables in production. The ConfigurationBuilder makes it possible to specify more than one source and load configuration settings from various places:

var config =
    new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", true)
        .AddEnvironmentVariables()
        .Build();

The AddEnvironmentVariables instruction comes after AddJsonFile, which means that any environment variables which have been set would override a previously configured setting in appsettings.json. This is a common pattern and standard code seen in almost every .NET application.

Although one thing which is not obvious from the example above is the unidiomatic way of declaring environment variables in order to make this happen. The .NET configuration architecture has been primarily designed with JSON files in mind, which means that .NET developers have to configure nested settings with a double underscore (__) in environment variables:

LOGGING__LEVEL=Debug
FOO=foo
BAR=bar
SERVER__PORT=8080
SERVER__FORCEHTTPS=true

This is such an odd way of configuring environment variables that seeing names such as SERVER__FORCEHTTPS are almost a certain giveaway that the underlying architecture is in .NET.

The ALT.NET way (using .env)

I'd much rather keep my development environment as close to production as possible. Therefore I'd much rather use environment variables as the main configuration mechanism during development too.

What if instead of using a nested JSON document I could configure my application just like in production:

LOG_LEVEL=Debug
FOO=foo
BAR=bar
SERVER_PORT=8080
SERVER_FORCE_HTTPS=true

Well that's how developers would do it in many other programming languages where the use of .env files is more prevalent. A .env file is essentially just a flat file specifying environment variables like the ones above. When an engineer launches their application during development then the .env file gets parsed and all variables within it will get set on a process level before anything else tries to read them. As the values are set on a process level they will only persist during the currently executing process and vanish on shutdown.

This has several benefits over the appsettings.json file approach. First is the incredible simplicity. Secondly is predictability. There is no need to configure multiple configuration providers. An application only retrieves its settings from one source and nowhere else. There is also no complexity around what happens when certain settings are stored in one location (e.g. appsettings.json) and other settings in another (e.g. environment variables). Will they merge or replace each other? If an application relies on only environment variables then this is not something to worry about.

Another benefit is how engineers think of configuration. The appsettings.json approach invites developers to create overly complex configuration hierarchies. They are easy to read and change during development, but more cumbersome to manage in production.

For example take this snippet as an illustration:

{
    "Databases": {
        "SqlServer": {
            "ConnectionString": "foo-bar"
        },
        "Redis": {
            "Endpoint": "localhost:6379"
        }
    }
}

In JSON format this looks totally fine, but in reality it probably isn't. Apart from being a data persistence technology, Redis and SQL Server have very little in common. In fact they are probably used for complete different application functionalities. Thus it makes very little sense to group them under one universal Databases configuration node together.

Remember in production these will need to get configured as following:

DATABASES__SQLSERVER__CONNECTIONSTRING=foo-bar
DATABASES__REDIS__ENDPOINT=localhost:6379

This notion makes it much more obvious that the original configuration structure is unfit for everyday use in production.

If environment variables were the primary configuration strategy during development too, then developers would presumably name them more sensibly:

SQL_SERVER_CS=foo-bar
REDIS_ENDPOINT=localhost:6379

Fortunately using .env in .NET is a straightforward alternative to appsettings.json.

Loading .env files in C#

The code for loading and parsing a .env file is so simple that it hardly warrants the use of an external dependency via NuGet.

Personally I like to create a DotEnv.cs file in my C# project and copy the following code into it:

namespace YourApplication
{
    using System;
    using System.IO;

    public static class DotEnv
    {
        public static void Load(string filePath)
        {
            if (!File.Exists(filePath))
                return;

            foreach (var line in File.ReadAllLines(filePath))
            {
                var parts = line.Split(
                    '=',
                    StringSplitOptions.RemoveEmptyEntries);

                if (parts.Length != 2)
                    continue;

                Environment.SetEnvironmentVariable(parts[0], parts[1]);
            }
        }
    }
}

Then I add DotEnv.Load("..") at the beginning of the Main function inside my Program.cs file:

public static class Program
{
    public static async Task Main(string[] args)
    {
        var root = Directory.GetCurrentDirectory();
        var dotenv = Path.Combine(root, ".env");
        DotEnv.Load(dotenv);

        // Other code
    }
}

This makes sure that all environment variables get set before any class or function tries to access them.

Finally I specify environment variables as the only required configuration provider:

var config =
    new ConfigurationBuilder()
        .AddEnvironmentVariables()
        .Build();

Now I can add a .env file into the root of my application and configure environment variables like in production.

Of course the file doesn't have to be named .env and one can rename it to whichever name suits them best. Regardless which name one settles on, don't forget to add it to one's .gitignore file. Especially in open source projects you wouldn't want to commit development secrets into the public domain.

Loading .env files in F#

In F# the implementation is very similar to C#:

namespace YourApplication

module DotEnv =
    open System
    open System.IO

    let private parseLine(line : string) =
        Console.WriteLine (sprintf "Parsing: %s" line)
        match line.Split('=', StringSplitOptions.RemoveEmptyEntries) with
        | args when args.Length = 2 ->
            Environment.SetEnvironmentVariable(
                args.[0],
                args.[1])
        | _ -> ()

    let private load() =
        lazy (
            Console.WriteLine "Trying to load .env file..."
            let dir = Directory.GetCurrentDirectory()
            let filePath = Path.Combine(dir, ".env")
            filePath
            |> File.Exists
            |> function
                | false -> Console.WriteLine "No .env file found."
                | true  ->
                    filePath
                    |> File.ReadAllLines
                    |> Seq.iter parseLine
        )

    let init = load().Value

The only main difference is that the load() function has been made private and lazy loaded via the init variable, meaning that the code inside load will only get executed once, regardless of how often DotEnv.init gets called. This is to allow the loading of environment variables before Program.fs gets invoked.

In functional programming it is very common to make use of static variables and functions. For example I often load my application settings using a static module like this:

module Config =
    open System

    let private get key =
        DotEnv.init
        Environment.GetEnvironmentVariable key

    let secretKey = get "SECRET_KEY"
    let redisEndpoint = get "REDIS_ENDPOINT"

    // etc.

Those static values will get initialised as soon as the assembly loads into the domain, which is well in advance of any code being called in Program.fs. Therefore I have to place the DotEnv.init command inside the get helper function, making sure that settings from the .env file get initialised before the first Environment.GetEnvironmentVariable invocation. Given that DotNet.load() is lazy it will only execute once and not reload the .env file on subsequent calls.

Additionally I must also put the DotEnv.fs file as the first compilation item in the .fsproj file:

<ItemGroup>
    <Compile Include="DotEnv.fs" />
    <Compile Include="Other.fs" />
    <Compile Include="Stuff.fs" />
    <Compile Include="Program.fs" />
</ItemGroup>

All in all this completely replaces .NET's huge configuration pattern with an extremely simple solution. It's "cloud native" as Microsoft likes to call it and extremely easy to understand.

Just like in C# don't forget to add the .env file to your .gitignore rules!

Side notes

What if an environment variable changes?

A lot of complexity in .NET's configuration classes come from the "need" to react to changes. A web server is a long running process and if someone wants to change a value in appsettings.json then any functionality which relies on that setting also has to learn about the update. However most current application hosting solutions such as serverless functions or Kubernetes clusters automatically reload an application on configuration changes, so while it might be an interesting problem to think about, it's certainly more of a theoretical than practical issue. The simple .env solution works just fine.

Why not load environment variables via X?

Could I not just load environment variables via:

bash/PowerShell?
launchSettings.json?
this tool?
that tool?
etc.?

Yes, there are many ways to load environment variables into the process before launching an application. However, in this blog post I wanted to show a way which satisfies two requirements (which most of these tools don't):

It works for everyone, regardless of OS or IDE
It works during F5 debugging from an IDE as well

Ultimately it doesn't matter how you load environment variables, but if it can be done from within .NET so that it just works for everyone on every platform, and also works during debugging without any extra hacks then why not go with such a solution?

Existing OSS projects for .NET?

If you wondered if there are any existing .NET OSS projects to support .env files then you will be pleased to hear that there are some such as dotenv.net, dotnet-env and net-dotenv. I have not used any of them but they all seem to be actively maintained.

Equal pay for equal work

Sat, 02 Jan 2021 00:00:00 +0000

Remote work is on the rise, the economy is on a decline and businesses are re-structuring themselves around what many consider the new normal. Increasingly more organisations are opening themselves up to the possibility of working remotely beyond just the temporary measure in the fight against COVID-19. Many begin to see remote work as not only a necessary consequence, but rather as a new opportunity to diversify their workforce, increase employee productivity and cut some cost. The biggest cost saving obviously comes from office space, and not just in terms of raw square footage but also in location. When previously companies had to boast huge HQs in expensive areas, now they can spread themselves thinner across more affordable cities. Another big cost saving can come from employee salaries. A more distributed workforce and more work from home opportunities will inevitably mean a bigger talent pool competing for the same open positions, effectively giving employers a bigger choice.

However, when it comes to existing employees' salaries the direction taken by big tech companies couldn't be of starker contrast. Companies like Facebook or Twitter have made it clear that employees who choose to relocate to more affordable cities can expect big cuts to their existing salary whereas companies like Reddit announced that they won't reduce the salary of any of their 600 US workers regardless of where they live.

These recent announcements have revived a long standing debate around the topic of equal pay for equal work. Before COVID-19 GitLab has already sparked a lot of controversy around the concept of "cost of living" adjusted salaries. At GitLab two engineers who perform the exact same work could get vastly differently compensated based on where they live. It doesn't require a lot of imagination to understand that such policies can alienate a lot of good people and further contribute to the perception of an ever growing inequality gap. Personally I find cost of living adjusted salaries very problematic as they deflect from the real market forces which come into play. Those should be a decent minimum living wage and the forces of demand & supply. If someone is a professional in a niche market, a rare specialist in a scientific field or an exceptionally sought after engineer then cost of living should have no say in determining their pay. Those so called knowledge workers are often higher in demand than there is supply. A good indicator for the scarcity of one's profession is when their employer recruits talent from all across the world and advertises unique perks such as relocation bonuses, dental care or exceptional holiday packages. These benefits would not exist if there wasn't a fierce competition for a particular skill.

Regardless of the real economics at play, it still comes down to a good old negotiation where each person has to stand up for their own beliefs and reach an agreement on their pay. If you find yourself in a situation where your current salary might be at risk due to recent relocation then hopefully the following write-up can help you to negotiate equal pay for equal work. It is a curated list of relevant points to formulate a strong argument why one should receive the same or perhaps an even better pay for the same work carried out from a remote position.

Same duties, same pay

If you're reading this blog post then you're most likely not being paid for your time or place of work. You are being paid for your knowledge, your skills, your duties, the challenges which come with your work and most importantly the responsibilities of your role. Your professionalism and your personal commitment which you bring to your every day job will not diminish when you move into a new place. Whether you work remotely or not, you will still contemplate over that tough work problem in your spare time. You will still lose sleep over that big presentation the next day. You will still stay up late to meet an important deadline by the end of the month. You do it because you take pride in your work. You do it because of a sense of personal accountability to your team. It is those qualities which have earned you the trust of your manager over the years. It is those qualities which earned you a pay rise long ago. Changing postcodes doesn't take away your accomplishments from the past. Changing postcodes shouldn't take away the things which you have already earned before.

Same value, same pay

The value of your contributions hasn't been compromised by moving house. Customers still pay the same price for the products which you have helped to build. Sales figures didn't drop when you went remote. Why should your employer reduce your pay when they are not passing that cost saving onto their customers by cutting their own price? Based on the same principle why your employer won't reduce their prices when downsizing office, you also shouldn't accept a lower salary after moving place. The concept is the same and you shouldn't accept one principle for them, but another for yourself. As you continue to deliver the same value and quality as before, you equally deserve the same compensation in return.

Your savings, their savings

Let's be clear, when you transitioned from office to home it's not like you're the only one who benefited from a lower cost of living as a result. Whatever savings you might have made on your rent, your employer does now too. Every person in an office requires an extra desk. Every additional person requires a fraction more of bathroom space. More people mean bigger kitchens, bigger break out areas, more meeting rooms, larger hallways, bigger stairways and more facilities to comply with fire safety regulations. There is more pressure on lifts to avoid bottle necks during peak times. There is more building maintenance work to be done and more frequent cleaning intervals required. A bigger workforce automatically means more individual needs. Extra bicycle storage rooms, canteens, car parks, private offices, outdoor spaces, a bigger selection of refreshments and a larger variety of office perks are just a few to name. Unless your employer is prepared to share their own operational saving directly with you, why should you share your personal operational saving with them? Fundamentally their savings are theirs, and your savings are yours.

Their profit, your profit

Have you ever received a pay rise when your employer re-structured themselves to benefit from lower tax? Have you ever received a pay rise when your employer opened up a new factory in a less regulated place? Have you ever received a pay rise when your employer outsourced a call centre into a country without minimum wage? Probably not, because it was your employer and not you who took the risk. Guess what, when you are courageous enough to relocate to a different country, city or state, then any financial gains from your move are also only yours to claim. This shouldn't be a huge surprise as nothing comes without its own significant risk. Economic security, social safety nets, unemployment benefits, retirement support, access to public health services, cost of qualitative education or medical treatment are often some of the compromises which have to be taken into account. Simple things such as free playgrounds for children, access to public recreational grounds, well maintained parks, a good public transport system funded by tax or basic luxuries such as political stability or the ability to safely park a new car on a public road are many other cost calculations not to be dismissed. Of course this is not always the case, but those considerations are for you to make. Nobody else, particularly not your employer, who has a financial interest of dismissing or downplaying those issues should be making those calculations on your behalf. This is such a basic principle that it's almost offensive to suggest the opposite. Your employer should not decide what sort of lifestyle you deserve.

Higher cost, higher pay

Contrary to common belief, working from home is not cheap. First of all, working permanently from home requires an entire additional room. If your family lived in a three bedroom house before, now you need four. Thanks to COVID-19 most will agree that working from a sofa is neither practical, nor sustainable or realistic by any means. A proper office desk and a good ergonomic chair are a necessity at the least. Those requirements alone make a legitimate home office significantly more expensive than many would like to admit. Throw in a couple of monitors, a qualitative web cam, microphone, printer, shredder, peripheral devices, a whiteboard, noise cancelling headphones, a mesh router and a backup laptop (assuming you'll get a work laptop provided) then the true cost of a home office begins to reach eye watering levels.

Some employers will try to provide you those supplies, but any seasoned remote worker will tell you to decline such an offer. After all you're still equipping your own personal home. You shouldn't have to pick from a selection of office desks which don't match your walls. You shouldn't have to settle on a chair which doesn't appeal to your eye. You shouldn't have to accept a black bezelled screen when all your other equipment is in space grey. Most importantly though, you want to treat those things as your own. You don't want to change your monitors when you change your job. You don't want to have to go through your employer to make a warranty claim. You don't want to ask for permission to replace a worn out chair. Instead you want to get a pay rise which allows you to set up your home office in the most productive way. If your employer is smart enough then they will not want to manage all of that inventory (which they never get to see) either.

Speaking of inventory, the true cost of working from home does not stop there yet. When working remotely nothing disrupts productivity more than a low bandwidth internet connection. One cannot constantly have people cutting out in important meetings. Pair programming is not possible with a two second lag. Worst of it is when one cannot work at all, because their internet provider has yet another outage in a short period of time. Working professionally requires a professional internet connection, whether from an office or from home. A fibre optic connection from a reputable provider with a high SLA is often 3-4 times more expensive than what most households have today. It's a significant cost which remote workers can hardly avoid to pay.

Fibre optic is not always the holy grail though. About a year ago hundreds of London households were affected by a major outage because a residential construction site accidentally drilled into one of the main regional cables underground. Households, businesses and even entire hospitals were left without internet for several days. Now if this was to happen in a traditional office then employees would still go to work, stand by the coffee machine all day and still get paid. Meanwhile office management would go berserk and erratically try to workaround a problem which cannot be easily fixed. At some point the issue would get eventually resolved and everyone will carry on. However, if the same were to happen in a remote work environment then employees would be expected to put adequate counter measures into place. Unfortunately I was one of the households which was affected by this outage at the time. My options were to either sign up with a co-working space and work from there, or pay extra for unlimited mobile data as a backup plan. I opted for the latter as it made more sense and time has shown that exceptional cases tend to be more regular when one cannot afford for them to happen.

Finally remote workers - who essentially conduct business from home - also have to pay for a higher home insurance premium and for a significant increase in utility bills. All in all the recurring and one-off expenses are sky high and must be accounted in a remote worker's pay.

Not important then, not important now

Think back to the time when you got hired for your current role. Did anyone ask you about your cost of living then? During your interview did someone ask you about your monthly rent? I'd be surprised if this was the case. Instead you were probably asked to traverse a binary tree. You had to solve a whiteboard puzzle to justify your pay. Cost of living was never a concern. Many of your peers received the same pay before the housing market took a hike. Many of your peers might have not even paid any rent. What if someone inherited a home or bought extremely cheap a few years ago? Equal distribution was never your employer's aim. Adjusting salaries to cost of living cannot just be arbitrarily introduced when it suits your employer the most. You are not less worth when moving place.

Negotiate like a friend

Whatever one's situation is, however difficult a negotiation might be, treat people like they were your friends. Always be kind, remain friendly, negotiate in good faith and explain your thoughts and concerns with respect. The best way to achieve your goal is by getting people on your side. Make yourself a friend. Imagine if you were a manager and how far you would go yourself for someone who you respect. How many hierarchies would you fight in order to secure a valuable employee's pay? No supervisor or manager would want to lose a good employee on their team over a pay policy which is partially out of their control. People will jump through those hoops if you stand up for yourself in a positive and friendly way. Don't underestimate what an amicable negotiation can achieve. A hostile negotiator can only get what the other party has to give. A friendly negotiator however can get things which others thought were not even up for debate.

.NET for Beginners

Wed, 22 Jul 2020 00:00:00 +0000

Last night I came across this question on Reddit:

Hello, I am just starting to learn c# and i am about to start with a course on Udemy by Mosh Hamedani called "C# Basics for Beginners: Learn C# Fundamentals by Coding". I can see that he is using .NET Framework but i have read that .NET Core is newer and is the future? I am really not sure where to start and would appreciate if anyone could help me. I would like to learn C# to have a better understanding with OOP and to learn a programming language to help with my University course. Thank you

I am just starting to learn C# and i am confused about .NET framework & .NET Core, Reddit

First of all congratulations to the author for learning C# which is a great choice of programming language and for putting in the extra effort into better understanding the concepts of OOP (object oriented programming)! I genuinely hope that they'll have a great experience learning .NET and most importantly that they'll enjoy picking up some new skills and have fun along the way! I remember when I started to code (more than 20 years ago) and how much fun it was for me! It is also great to see how many people have replied with really helpful answers and provided some useful guidance to the original thread! It is a true testament to how supportive and welcoming the .NET community is! However, despite these positive observations I'm still a little bit gutted that such a question had to be even asked in the first place. I don't imply any fault on the author themselves - quite contrary - I actually really sympathise with them and can understand the sort of struggles which a new C# or F# developer may face. Questions like these really demonstrate how complex .NET has become over the years. It is a good time to take a step back and reflect!

High cognitive entry barrier

.NET has a high cognitive entry barrier. What I mean by this is that a new developer has to quickly familiarise themselves with rather boring and complex topics way too early in their educational journey. In particular questions around the differences between .NET, .NET Core, Mono, Xamarin, and the relation between C#, F#, VB.NET and what a target framework or a runtime is add very little to almost no benefit to one's initial learning experience. Most newcomers just want to read a tutorial, watch a video or attend a class and then be able to apply their newly learned skills in an easy and uncomplicated way. Ideally educational material such as purchased online courses or books from more than a few months ago should still be relevant today. Unfortunately neither of these are true for .NET. As a matter of fact it's almost given that any content which has been written more than a year ago is largely outdated today. Just think about the upcoming release of .NET 5 and how it will invalidate most of the lessons taught just a few months ago. Another prominent example is the very short lived .NET Standard. Once hugely evangelised and now de-facto dead. I personally think Microsoft, the .NET team and to some extent the wider .NET community has failed in making .NET a more beginner friendly language. I say this from a good place in my heart. Programming is an extremely powerful skill which can allow people to uplift themselves from all sorts of backgrounds and the harder we make it for beginners, the more exclusive we make the entry to our profession. I think the success of the next ten years will be defined by how accessible we make .NET to a whole new population of developers and I think there's some real work to be done in order to improve the current situation!

What about others?

Before you think that the current complexity is normal in programming and nothing endemic to .NET itself then I'm afraid to prove you wrong. Take a look at PHP. PHP has evolved a lot over the years and while it has introduced more complicated object oriented concepts and improved on its performance, it hasn't lost the beauty of its original simplicity at all. PHP is a great programming language to learn. Beginner content exist like sand on a beach. The online community, forums, Reddits and conferences are second to none. Most importantly, a student could pass on a several years old PHP book to another person and they would still get a lot of value from it. The language might have moved on, but many things which were taught on a beginner level are still 100% accurate today. This is a huge advantage which gets easily forgotten by more advanced developers like myself. Furthermore the distinction between language and the interpreter is not something which a new PHP developer has to understand. A new beginner doesn't have to wrap their head around such relatively unimportant topics to get started. They just know that they've installed PHP and it works.

Another great example (and there are many great examples) is Go. Go is not a new language at all. It's been around for more than 10 years and despite huge improvements to the language, the compiler and the standard library it has remained faithful to its original simple design. Similar to PHP a new developer doesn't have to think about complicated nuances between "target frameworks", weird gotchas if they write Go in one IDE or another and they certainly don't have to look up a complicated version matrix to understand the intricate relations between an SDK, the runtime and newly available language features. There is just one version which maps to a well defined list of features, bug fixes and improvements in Go. It's documentation is simple and easy to comprehend. It is a very beginner friendly language.

Improving .NET

Why is .NET so complicated? Well, the answer is perhaps not that easy itself. It's mostly history, legacy baggage, some bad decisions in the past and what I believe an overly eager desire (almost urgency) to change things for changes' sake. I will try to name a few selected issues from my personal observation, describe the problems which I have perceived and try to provide some constructive suggestions which I think could be a good improvement for the future.

Let me introduce you to the 6 Sins of .NET:

Language Spaghetti

If a person sets out to learn C# (like the author from the question above), what do they learn? Is it C# or .NET? The answer is both. C# doesn't exist without .NET and you cannot program .NET without C# (or F# or VB.NET for that matter). This is not a problem in itself, but certainly where some of the issues begin. A new beginner doesn't just learn C# but also has to learn the inner workings of .NET. Things get even more confusing when C# isn't the initial .NET language to learn. Supposedly the loose relationship between .NET languages shouldn't really matter because they compile into IL and become cross compatible. Except when they don't:

The BCL (Base Class Libraries) provide the foundation for all three languages, yet they are only written in C#. That's not really an issue unless entire features were written with only one language in mind and are extremely cumbersome to use from another. For example, F# still doesn't have native support for Task and Task<T>, converting between System.Collection.* classes and F# types is a painful undertaking and F# functions and .NET Action<T> objects don't map very well.

The most frustrating thing though is when changes to one language force additional complexity on another.

Interop issues between those three languages are a big burden on new developers, particularly when they start with something else but C#. However, as painful as this might be, interop problems are not the only complexity which a new beginner has to face. Try to explain to a new C# user in a meaningful way when they should use inheritance with standard classes, interfaces, abstract classes or interfaces with default method implementations. The differences between those options have been so watered down that one cannot explain the distinctions in a single coherent answer anymore.

Take this StackOverflow question for an example. Nothing which has been described in the accepted (and most upvoted) answer below isn't also true for interfaces with default method implementations today:

Another great example is the growing variety of C# data types. When is it appropriate to create a data class, an immutable data class, a mutable struct, an immutable struct, a tuple class, the new concept of named tuples or the upcoming record type?

Seasoned .NET developers are very opinionated about when to use each of these types and depending on who a new beginner will ask, or which StackOverflow thread they'll read, they will most likely get very different and often contradicting advice. Anyone who doesn't think that this is a massive problem must be in serious denial. Learning a programming language is hard enough on its own. Learning a programming language where two mentors (who you might look up to) give you contradicting advice is even harder.

The irony is that many of the newly added language and framework features are supposed to make .NET easier to learn:

(BTW, I have been a huge proponent of dropping the .csproj and .sln files for a very long time but previously Microsoft employees defended them as if someone offended their family, so it's nice to finally see some support for that idea too! :))

Don't get me wrong, I agree with Scott that *this particular feature* will make writing a first hello world app a lot easier than before, however, our friend Joseph Woodward makes a good point that nothing comes for free:

And he's not alone with this idea:

It's not just about learning how to write C#. A huge part of learning .NET is reading other people's code, which is also getting inherently more difficult as a result:

Whilst I do like and support the idea of adding more features to C#, I cannot ignore the fact that it also takes its toll.

Some people raised a good point that it might be time to consider making some old features obsolete:

Whatever one's personal opinion is, "feature bloat" is certainly becoming a growing concern in the C# community and Microsoft would be stupid not to listen or at least take some notes.

Given that F# is already a functional first multi paradigm language, and C# is definitely heading towards that direction too, maybe one day there's a future opportunity to consolidate both languages into one? (YES, I dared to suggest it!) Either that, or Microsoft should establish 100% feature parity so that interop is seamless between the languages and the only differentiating factor remains in their syntax - one geared towards a functional first experience and the other towards the object oriented equivalent.

Version Overflow

As mentioned above, all three .NET languages evolve independently from .NET. C# is heading towards version 9, F# is approaching version 5 and VB.NET is already on version 16. Meanwhile .NET Framework is on version 4.8 and .NET Core on version 3.1. Don't even get me started on Mono, Xamarin or ASP.NET.

Now I understand that all these things are very different and I'm comparing apples with oranges, but how is a new developer supposed to know all of that? All these components are independent and yet correlated enough to overwhelm a new developer with screens like this:

Even .NET developers with 5+ years of experience find the above information hard to digest. I know this for a fact because I ask this question in interviews a lot and it's rare to get a correct explanation. The problem is not that this information is too verbose, or wrong, or unnecessary to know, but rather the unfortunate case that it's just how big .NET has become. In fact it's even bigger but I believe that the .NET team has already tried their best to condense this screen into the simplest form possible. I understand the difficulty - if you've got a very mature and feature rich platform then there's a lot to explain - but nevertheless it's not a particularly sexy look.

In contrast to .NET this is the cognitive load which is thrown at a beginner in Go:

It's much simpler in every way. Admittedly it's not an entirely fair comparison because Go gets compiled directly into machine code and therefore there isn't a real distinction between SDK and runtime, but my point is still the same. There is certainly plenty of room for improvement and I don't think what we see there today is the best we can do.

Maybe there's some value in officially aligning language, ASP.NET Core and .NET (Core) versions together and ship one coherent release every time? Fortunately .NET 5 is the right step in that direction but in my opinion there's still more to do!

.NET Everywhere

Now this one will probably hit some nerves, but one of the *big* problems with .NET is that Microsoft is obsessed with the idea of .NET Everywhere. Every new development aims at unifying everything into one big platform, catering for every single possible use case imaginable:

(Thanks to the courtesy of Ben Adams I've updated the graphic to represent the full picture of .NET. Ben created this image for the purpose of his own blog which you can read on www.ageofascent.com/blog.)

In many ways it makes a lot of sense, but the angle taken is causing more harm than help. While it makes a lot of sense to walk on a stage and boast to potential customers that your product can solve all their existing and future problems, it's not always the best approach when you actually want to on-board new developers from all sorts of different industries.

Unifying the entire stack into a single .NET platform doesn't come without a price. For years things have been constantly moved around, new things have been created and others have been dropped. Only recently I had to re-factor my ASP.NET Core startup code yet again:

Every attempt at unifying things for unification's sake makes simple code more verbose. Without a doubt the previous code was a lot easier to understand. I applaud the concept of a generic host, but one has to wonder how often does a developer actually want to combine two servers into one? In my opinion there must be a real benefit in order to justify complexity such as having a builder inside another builder! How often does a developer want to *create* a web server and not *run* it as well? I'm sure these edge cases do exist, but why can't they be hidden from the other 99.9% of use cases which normally take place?

As nice and useful as the builder pattern may be, and whatever architectural benefits the lambda expression might give, to a C# newbie who just wants to create a hello world web application this is insane!

Remember, this is the equivalent in Go:

router := ... // Equivalent to the Startup class in .NET

if err := http.ListenAndServe(":5000", router); err != nil {
    // Handle err
}

Microsoft's "Swiss Army Knife" approach creates an unnecessary burden on new .NET developers in many different ways.

For example, here's the output of all the default .NET templates which get shipped as part of the .NET CLI (minus the Giraffe one):

They barely fit on a single screen. Again, it's great that Microsoft has Blazor as an answer to WASM, or that they have WPF as an option for Windows, but why are these things shipped together as one big ugly thing? Why can't there just be a template for a console app or class library and then some text which explains how to download more? This is a classic example where ".NET Everywhere" is getting into most users' way!

Speaking of fitting things into a single screen...

Whilst the above tweet was comical in nature, it's not far from the truth.

My honest constructive feedback is *less is more*!

It's rarely the case that a new .NET developer wants to be drowned in a sea of options before they can get started with some basic code. Most newcomers just want to set up the bare bones minimum to get up and running and write their first lines of code. They don't even care if it's a console app or an ASP.NET Core application.

I totally support the idea of a single .NET which can cater to all sorts of different needs, but it must be applied in a much less overwhelming way. Visual Studio's new project dialogue doesn't need to match Microsoft's marketing slides and the dotnet new command doesn't have to ship a template for every single type of app. It doesn't happen very often that a developer is first tasked to work on a line of business web application, then told to build a Windows forms store app and later asked to build a game. Absolutely nobody needs twenty different templates which span across ten different industries on their machine.

My advice would be to optimise the .NET experience for beginners and not worry about advanced developers at all. There's a reason why advanced users are called advanced. If a senior developer wants to switch from iOS to IoT development then they will know where to find the tools. Currently the .NET CLI ships FIFTEEN!!! different web application templates for a user to pick. How is a new C# developer supposed to decide on the right template if even experienced .NET developers scratch their head? Microsoft must understand that not bloating every tool or IDE with a million different options doesn't mean that users don't understand that these options do exist.

In my opinion the whole mentality around ".NET Everywhere" is entirely wrong. It should be a positive side effect and not the goal.

All Eyez on Me

Another problem which I have observed is Microsoft's fear of being ignored. Microsoft has a solution for almost every problem which a developer might face. It's an amazing achievement and something to be really proud of (and I really mean it), but at the same time Microsoft has to learn how to give developers some space.

Microsoft does not miss a single opportunity to advertise a whole range of products when someone just looks at one. Doing .NET development? Oh look, here is Visual Studio which can help you with that! Oh and by the way you can deploy directly to IIS! In case you wonder, IIS comes with Windows Server which also runs in Azure! Aha, speaking of Azure, did you know that you can also click this button which will automatically create a subscription and deploy to the cloud? In case you don't like right-click-publish we also have Azure DevOps, which is a fully featured CI/CD pipeline! Of course there's no pressure, but if you *do sign up now* then we'll give you free credits for a month! Anyway, it's just a "friendly reminder" so you know that in theory we offer the full package in case you need it! C'mon look at me, look at me, look at me now!

Again, I totally understand why Microsoft does what they do (and I'm sure there's a lot of good intention there), but it comes across the complete wrong and opposite way.

No wonder that the perception of .NET hasn't changed much in the outside world:

What Microsoft really tries to say is:

Hey folks, look we're different now, we are cross platform, we run everywhere and we want you to have a great experience. Here's a bunch of things which can help you on your journey.

Unfortunately what users *actually* understand is:

Hey, so .NET is a Microsoft product and it mostly only works with other Microsoft products, so here's a bunch of stuff which you will have to use!

On one hand Microsoft wants to create this new brand of an open source, cross platform development tool chain and yet on another they push Visual Studio and Azure whenever they talk .NET. This is sending mixed messages and confusing new developers, detracting from .NET's actual brilliance and doing a massive disservice to the entire development platform. The frustrating thing is that the more .NET is becoming open to the world, the more Microsoft is pushing for their own proprietary tools. Nowadays when you watch some of the most iconic .NET employees giving a talk on .NET then it's only 50% .NET and the rest advertising Windows, Edge and Bing. This is not what most .NET developers came to see and it doesn't happen in other programming communities such as Node.js, Rust or Go either. Besides that, if someone constantly advertises every new Microsoft flavour of the month then they also lose developer credibility over time.

The other thing is that questions and answers like these need to stop:

This question and particularly the answer are really bad, because they demonstrate that whilst C# and .NET are not necessarily tied to Visual Studio and Windows anymore, they still remain the most viable option to date. This sentiment is not good but unfortunately true. I know from my own experience that the Visual Studio Code plugin for C# is nowhere near as good as it should be. The same applies to F#. Why is that? It's not that Visual Studio Code is less capable than Visual Studio, but rather a decision by Microsoft to give it a lower priority and a lack of investment. I don't need to use JetBrains GoLand in order to be productive in Go, but I have to use Rider for .NET.

Microsoft needs to decouple .NET from everything else and make it a great standalone development environment if they want to compete with the rest.

C#, F# and .NET will always be perceived as a very Microsoft and Windows centric development environment when even the official .NET documentation page confirms a critic's worst fears:

Architecture Break Down

Go is 10 years old.

.NET Core is 4 years old - not even half the age of Go.

Go is currently on version 1.14 and .NET Core is already on its third major version. Both have been written from the ground up, but Microsoft has had arguably more experience developing .NET Core given that it was re-written from scratch with the knowledge of more than 14 years of supporting .NET Framework. How on earth did .NET Core end up with so many architectural changes that it is already on version 3.x?

Microsoft prides itself with developing .NET Core extremely fast and offering a comprehensive out of the box solution, but it is one of the most unstable and tiresome development environments which I have worked with in recent years. It is becoming increasingly exhausting trying to keep up with .NET Core's constant change. Microsoft would implement one thing one day and then completely replace it the other. Just think about the constant changes to the Startup.cs class, or the ever evolving HttpClient and its related helper types, the invention and demise of the JSON project file, .NET Standard or various JSON serialisers. The list goes on. Features such as CORS, routing and authorisation keep changing as more code gets rewritten and pushed down the pipeline and more types are being made obsolete from the Microsoft.AspNetCore.* namespace and replaced with new ones emerging in Microsoft.Extensions.*.

It's hard to keep up with .NET as an experienced developer let alone as a beginner. This constant change significantly reduces the lifespan and usefulness of books, videos, online tutorials, StackOverflow questions, Reddit threads and community blog posts. It's not only making .NET by an order of magnitude harder to learn but also financially less viable than others.

Good software and framework architecture should provide a stable foundation which is open for extension, but closed for change (sound familiar?). There was no need to implement v1 of ASP.NET Core in such a way that it now requires constant architectural change in order to support new innovation. Why has endpoint routing not been build on day one? Why does Microsoft not provide an adequate and feature complete replacement for Newtonsoft.Json before they released and encouraged the usage of System.Text.Json? Why are light weight and easy to understand routing handlers such as MapGet only an afterthought? Why is it that Microsoft never creates a GitHub issue for an existing successful .NET OSS project when they need something similar (maybe with a few changes or improvements) and rather invent their own competing in-house product which *always* causes pain in the community and indirectly forces users to re-write their codebase yet again?

There is no actual need to do any of this, only self imposed deadlines which force Microsoft to release ill written software, badly thought out framework features and an unnecessary burden on its current developer community. This constant change is extremely painful to say the least. It's the single worst feature of .NET and the main reason why I honestly couldn't recommend .NET to a programming novice in good faith.

There is really not much else to say other than *slow - down*. I wish the .NET and ASP.NET Core teams would take this criticism (which isn't new) more serious and realise how bad things have become. I know I keep banging about Go, but surely there is some valuable lesson to learn given how popular and successful Go has become in a relatively short amount of time? Maybe Go is too simple in comparison to .NET, but maybe the current pace of .NET is not the right approach either? It's important to remember that a less breaking .NET would pose such a smaller mental and financial burden on new developers from all across the world!

Name Overload

This blog post wouldn't be complete without mentioning Microsoft's complete failure of naming .NET properly in a user and beginner friendly way. I mean what is ".NET" anyway? First and foremost it's a TLD, which has nothing to do with Microsoft! Secondly there is no clear or uniform way of spelling .NET. Is it ".NET" or "dot net"? Maybe it was "DOTNET" or it could be "dot.net" like the newly registered domain dot.net? My friends still tease me by calling it "DOT NOT" whenever I mention it to them!

Finally when there was an opportunity to correct this long standing mistake by re-writing the entire platform and possibly giving it a new name then Microsoft decided to call it ".NET Core". If anyone thought it couldn't get any worse than Microsoft surely didn't disappoint! I cannot think of a more internet unfriendly name than ".NET Core". How do you even hashtag this? I've seen it all ranging from #dotnet to #dot-net, #dot-net-core, #dotnet-core, #netcore, #net-core, #dot-netcore and #dotnetcore.

I think everyone can agree that objectively speaking ".NET Core" was never a great name. Needless to say that ".NET Core" also completely messes with the internet history for ".NET Framework", which is exactly what everyone predicted before.

At least Microsoft is consistent with its naming. There's something comical in the fact that the only three supported languages in .NET are called CSharp, F# and VB.NET. Or was it C#, F Sharp and Visual Basic? Anyway, it was some combination of the three!

Final words

C#, F# and the whole of .NET is a great development platform to code, but it has also become overly complex which is holding new developers back. I've been working with it for many years and mostly enjoyed myself, however I won't lie and say that things haven't gotten a little bit out of hand lately. There is something to tell that after having .NET for 20 years the programming community still hasn't seen anything new or noteworthy since the creation of stackoverflow.com:

Meanwhile we've seen very prominent products being built with other languages spanning across multiple domains such as developer technologies (Docker, Kubernetes, Prometheus) to smaller static website generators (Hugo) or some of the most successful FinTech startups (Monzo) in the world.

.NET is a great technology for experienced developers who grew up with the platform as it matured, but I'm not sure if I'd still enjoy learning it today as much as I did in 2008. Whilst the complexity allows me to charge great fees to clients for writing software in .NET, I'd probably not recommend it to a friend who wants to learn how to code or I wouldn't use it for building my own startup from the grounds up.

The future success of .NET will be based on the developers which it can attract today.

The success of .NET in ten years will be based on the decisions made today.

I hope those decisions will be made wisely!

June 2021 Update:

If you are actually learning .NET and were looking for beginner content to get started then check out my latest blog post on .NET Basics which teaches all the foundational concepts around .NET and explains the inner workings of the platform.

GitHub Actions for .NET Core NuGet packages

Mon, 29 Jun 2020 00:00:00 +0000

Last weekend I migrated the Giraffe web framework from AppVeyor to GitHub Actions. It proved to be incredibly easy to do so despite me having some very specific requirements on how I wanted the final solution to work and that it should be flexible enough to apply to all my other projects too. Even though it was mostly a very straight forward job, there were a few things which I learned along the way which I thought would be worth sharing!

Here's a quick summary of what I did, why I did it and most importantly how you can apply the same GitHub workflow to your own .NET Core NuGet project as well!

Overview

CI/CD pipeline for .NET Core NuGet packages

First, let's look at the requirements which I set out for my final CI/CD pipeline to meet. Each of these points has a specific purpose which I think is applicable to most .NET Core NuGet projects and therefore explaining in more detail.

Branch and pull request trigger

CI builds are the first formal check as part of the software development feedback loop which don't come from a developer's machine itself. They are reproducible and reliable feedback and arguably cheap to run in the cloud. As such CI builds should run as frequently as possible so that new errors can be flagged up as soon as they occur.

On this premise I decided that each commit, regardless if it happened on a feature/*, hotfix/* or other branch, should trigger a CI build. Pull requests should trigger a CI build as well. It's a great way of validating the changes of a PR before deciding whether to merge. As a matter of fact, it's highly recommended to enforce this rule through GitHub itself.

In GitHub, under Settings and then Branches, one can set up branch protection rules for a repository:

Note that the available CI options get automatically updated whenever a CI pipeline is executed and therefore might not show up before the first workflow run has completed.

We can configure a GitHub Action to trigger builds for commits and pull requests on all branches by providing the push and pull_request option and leaving the branch definitions blank:

on:
  push:
  pull_request:

Test on Linux, macOS and Windows

.NET Core is cross platform compatible and so it's not a surprise that a NuGet library is expected work on Linux, macOS and Windows as well.

Running a CI job against multiple OS versions can be configured via a build matrix:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ ubuntu-latest, windows-latest, macos-latest ]

In this example I've named the "build" job build, which is an arbitrary value and can be changed to anything a user wants.

Create build artifacts

A build artifact is downloadable output which can be created and collected on each CI run. It can be anything from a single file to an entire folder full of binaries. In the case of a .NET Core NuGet library it is a very useful feature to create a super early version of a NuGet package as soon as a build has finished:

In combination with pull request triggers this is a super handy way of giving OSS contributors and OSS maintainers an easy way of downloading and testing a NuGet package as part of a PR.

It is also a nice way of letting users download and consume a "super early semi official" NuGet package which came from the project's official CI pipeline when someone is in absolute desperate need of applying a fix before an official release or pre-release has been created.

In GitHub a NuGet artifact can be easily created by first running the dotnet pack command as part of an earlier build step and subsequently using the upload-artifact@v2 action to upload the newly created *.nupkg as an artifact:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ ubuntu-latest, windows-latest, macos-latest ]
    steps:
      ...
      ...
      - name: Pack
        if: matrix.os == 'ubuntu-latest'
        run: dotnet pack -v normal -c Release --no-restore --include-symbols --include-source -p:PackageVersion=$GITHUB_RUN_ID src/$PROJECT_NAME/$PROJECT_NAME.*proj
      - name: Upload Artifact
        if: matrix.os == 'ubuntu-latest'
        uses: actions/upload-artifact@v2
        with:
          name: nupkg
          path: ./src/${{ env.PROJECT_NAME }}/bin/Release/*.nupkg

In the example above I'm using the pre-defined GITHUB_RUN_ID environment variable to specify the NuGet package version and a custom defined environment variable called PROJECT_NAME to specify which .NET Core project to pack and publish as an artifact. This has the benefit that the same GitHub workflow definition can be used across multiple projects with very minimal initial setup.

One might have also noticed that I used a wildcard definition for the project file extension .*proj. This has the additional benefit that the dotnet pack command will work for all types of .NET Core projects, which are .vbproj, .csproj and .fsproj.

Lastly I had to use the version 2 (@v2) of the upload-artifact action in order to use wildcard definitions in the artifact's path specification. If you run into a "missing file" error when trying to upload an artifact then make sure that you're using the latest version of this action. Before version 2 wildcards were not supported yet.

On another note, the if: matrix.os == 'ubuntu-latest' condition as part of the Pack and Upload Artifact steps has no special purpose except limiting the artifact upload to a single run from the previously defined build matrix. A single artifact upload is sufficient (the NuGet package doesn't change based on the environment where it has been packed) and therefore I simply chose ubuntu-latest because Ubuntu happens to be the fastest executing environment and therefore helps to keep the overall build time as low as possible. Windows workers seem to take generally longer to start than macOS or Ubuntu.

Push nightly releases to GitHub packages

You might have heard of the term "Nightly Build" before. A nightly build (or what I like to call a bleeding edge pre-release build) is a proper (formal) deployment of a build artifact to a place which makes general consumption almost as intuitive as an official release.

In the context of a NuGet package a "nightly release" is a NuGet library which normally gets pushed to a public NuGet feed which is just like the official NuGet Gallery, but not the gallery itself. This is a common pattern amongst .NET Core libraries because developers can configure more than one NuGet feed in their project via a NuGet.config file (see the NuGet.config reference for more information) and therefore consume a nightly build package the same way as an official release. Most commonly I've seen self hosted ProGet feeds or cloud hosted MyGet feeds to distribute "nightly builds" alongside the official NuGet gallery. However, GitHub's relatively new Packages feature makes an attractive alternative.

Setting up a nightly build pipeline to GitHub packages is fairly easy:

jobs:
  build:
    ...
    ...
  prerelease:
    needs: build
    if: github.ref == 'refs/heads/develop'
    runs-on: ubuntu-latest
    steps:
      - name: Download Artifact
        uses: actions/download-artifact@v1
        with:
          name: nupkg
      - name: Push to GitHub Feed
        run: |
          for f in ./nupkg/*.nupkg
          do
            curl -vX PUT -u "$GITHUB_USER:$GITHUB_TOKEN" -F package=@$f $GITHUB_FEED
          done

Unlike build artifacts nightly releases are not something which one would want to do on every build run. It makes sense to limit the creation of a pre-release/nightly deployment to a trigger which is at least one step closer to an official release than a casual git commit or a random pull request. If one uses Git flow or another similar branching strategy then the develop branch can be a natural gate keeper for a nightly release:

if: github.ref == 'refs/heads/develop'

Anything which gets pushed into the develop branch is per definition on the road map for the next official release and therefore a good trigger for a nightly build.

I've created a complete separate job called prerelease for this purpose alone. Just like the build job before, this name is completely random and can be changed to something entirely else. In addition the prerelease job should only execute after a successful build run:

needs: build

If I hadn't specified this then GitHub would try to run multiple jobs in parallel which is not desired in this case.

The following two steps are fairly self explanatory:

steps:
  - name: Download Artifact
    uses: actions/download-artifact@v1
    with:
      name: nupkg
  - name: Push to GitHub Feed
    run: |
      for f in ./nupkg/*.nupkg
      do
        curl -vX PUT -u "$GITHUB_USER:$GITHUB_TOKEN" -F package=@$f $GITHUB_FEED
      done

First I'm using the download-artifact@v1 action to obtain the artifact which has been uploaded under the name nupkg by the previous job. Then I use cURL to make a HTTP PUT request directly to GitHub's HTTP API in order to upload the downloaded *.nupkg package to the specified feed.

The name of the feed is determined through the GITHUB_FEED environment variable (more on this later). The GITHUB_TOKEN is a pre-defined environment variable which every GitHub Action has automatically created for. The GITHUB_USER variable is another global setting where I've set my GitHub username in one place.

GitHub packages issue with NuGet

Now one might wonder why I used a curl command to interact with GitHub's HTTP API if I could have used dotnet nuget push or nuget push instead? The short answer is because both of these CLI commands don't work with GitHub's packages feed today.

The dotnet nuget push command only works if the worker image is set to windows-latest, however, because the start-up time of a Windows worker is significantly longer than ubuntu-latest I rather trade a little bit of "cURL complexity" for an overall faster CI/CD pipeline. It is a personal choice and a trade off which I'm happy to make in this particular case (more on the benefit of speed later).

GitHub Packages Feed

If everything went to plan then the NuGet packages will get uploaded to the user's or organisation's own GitHub packages feed:

The packages are tagged with the GITHUB_RUN_ID (unless it was a GitHub release):

This is by design. It makes it very easy to associate a certain package version to a specific nightly run. It also makes it very obvious that a package version is a nightly build and not an official release and it's easy to know when a newer version is available since the GITHUB_RUN_ID is an incremental counter.

GitHub release trigger for official NuGet release

GitHub has a wonderful concept of releases, which is an extra layer on top of git tags and which provide a nice UI to create, manage and view a release:

Personally I like to use GitHub releases as a formal and conscious step to create, document and publish a NuGet package. For that reason I've added the release option as an additional CI trigger:

on:
  push:
  pull_request:
  release:
    types:
      - published

A GitHub release can have multiple trigger types such as a draft (e.g. created) or an edit (edited), a delete (deleted) and many more. A deployment should only get kicked off when an actual release has been published and therefore the published type has to be specified explicitly.

If a repository doesn't use GitHub releases then one can add normal git tags as a CI trigger instead. Git tag triggers would also invoke for a GitHub release (which uses git tags behind the scene), but they would also run whenever a developer pushes a git tag manually from their client.

In order to kick off a deployment for a GitHub release I created a job called deploy:

deploy:
  needs: build
  if: github.event_name == 'release'
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.301
    - name: Create Release NuGet package
      run: |
        ...
        ...
        dotnet pack -v normal -c Release --include-symbols --include-source -p:PackageVersion=$VERSION -o nupkg src/$PROJECT_NAME/$PROJECT_NAME.*proj
    - name: Push to GitHub Feed
      run: |
        for f in ./nupkg/*.nupkg
        do
          curl -vX PUT -u "$GITHUB_USER:$GITHUB_TOKEN" -F package=@$f $GITHUB_FEED
        done
    - name: Push to NuGet Feed
      run: dotnet nuget push ./nupkg/*.nupkg --source $NUGET_FEED --skip-duplicate --api-key $NUGET_KEY

Similar to the prerelease job the deploy job also requires the build job to finish first and additionally checks if the CI run was triggered by a GitHub release event:

deploy:
  needs: build
  if: github.event_name == 'release'

The actual deploy steps are very similar to everything else we've done so far. First it pulls the latest changes via the checkout@v2 action, then it installs the .NET Core SDK version 3.1.301 with the help of setup-dotnet@v1 and finally it runs a dotnet pack command with the official release version which is specified via the VERSION variable (more on this in the next section).

At last I'm using cURL to push to the GitHub packages feed again and the normal dotnet nuget push command to publish the final release to the official NuGet feed too.

Drive NuGet version from Git Tags

As mentioned above, the final NuGet version is determined through the VERSION variable. This variable doesn't exist and has to be created manually. Most .NET Core projects specify the package version in their *.csproj/*.fsproj file through the <Version>, <VersionSuffix> or <PackageVersion> property (if you don't know the differences check out Andrew Lock's blog post for further information). The downside of this is that the project's version has to be kept manually in sync with the GitHub release or git tag version and that it's mostly just meaningless metadata carried along in the project file which is not required until a new release is being published.

In my opinion a much better approach is to entirely remove all version properties from a .NET Core project file and derive the final package version from the submitted git tag during deployment. This is mostly better because there is never a risk of the NuGet package version being out of sync with the provided git tag version. As a developer you probably know that any manual sync is deemed to fail, so why put that strain on ourselves if we can do without it!

Luckily obtaining the git tag version from within a GitHub action is fairly easy. Assuming that a release is being tagged in the format of vX.X.X this bash script will extract the actual version from the GITHUB_REF variable:

- name: Create Release NuGet package
  run: |
    arrTag=(${GITHUB_REF//\// })
    VERSION="${arrTag[2]}"
    VERSION="${VERSION//v}"
    dotnet pack -v normal -c Release --include-symbols --include-source -p:PackageVersion=$VERSION -o nupkg src/$PROJECT_NAME/$PROJECT_NAME.*proj

For example, if I tagged a commit with v1.2.3, then the GITHUB_REF variable would contain refs/tags/v1.2.3.

The code arrTag=(${GITHUB_REF//\// }) converts all forward slash / characters into a whitespace and subsequently splits the GITHUB_REF variable by the whitespace character into an array called arrTag:

arrTag[0]: refs
arrTag[1]: tags
arrTag[2]: v1.2.3

The next couple lines grab the version tag from the third array element (second index) and later strip the leading v character from the value:

VERSION="${arrTag[2]}"
VERSION="${VERSION//v}"

If the git tag didn't include the v character (e.g. just 1.2.3) then the second line can be removed.

In the end the VERSION variable holds the correct release version of the imminent deployment and can be used to tag the final NuGet package as part of the dotnet nuget pack command:

dotnet pack -c Release --include-symbols --include-source -p:PackageVersion=$VERSION -o nupkg src/$PROJECT_NAME/$PROJECT_NAME.*proj

This is a very effective way of correctly versioning NuGet releases and keeping them automatically synced with GitHub releases (or manual git tags). It also enforces that a release can only happen when a proper git tag has been created.

Speed

Speed is paramount in a good CI/CD pipeline. The longer a single run takes, the more likely it is that multiple triggers will result in long queues of individual CI runs stacking up and therefore preventing developers from getting a fast developer feedback loop.

There's a few things which can be done to speed up a .NET Core NuGet pipeline.

Ubuntu over Windows

All jobs use the ubuntu-latest worker image except the first build job which uses a build matrix of three different OS versions to build and test against all major environments. Ubuntu workers start faster than others and therefore should be preferred over Windows images.

Avoid redundant dotnet restores

The build job has been optimised to not repeat the dotnet restore step unnecessarily by making use of the --no-restore and --no-build flags where possible:

- name: Checkout
  uses: actions/checkout@v2
- name: Setup .NET Core
  uses: actions/setup-dotnet@v1
  with:
    dotnet-version: 3.1.301
- name: Restore
  run: dotnet restore
- name: Build
  run: dotnet build -c Release --no-restore
- name: Test
  run: dotnet test -c Release --no-build

Avoid redundant NuGet caching

Setting the environment variable DOTNET_SKIP_FIRST_TIME_EXPERIENCE to true means that we can prevent the .NET CLI from wasting time on redundant package caching.

Turn off telemetry

Maybe not a huge gain, but surely turning off the .NET telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to true will shave off another few (milli)seconds.

Avoid pulling in extra dependencies

Not having to install extra utilities on a worker image means that the CI run doesn't have to waste extra time setting up additional tools. For example, instead of installing the standalone NuGet CLI one can use dotnet nuget which comes out of the box when .NET Core is set up as a dependency. Another example is to use curl when it already exists instead of pulling in another HTTP client for negligent benefits.

Bash over PowerShell

Running bash scripts is significantly faster than running PowerShell (pwsh), because PowerShell takes longer to load. Luckily all script blocks in GitHub actions are set to bash by default unless specified otherwise. Try to avoid using PowerShell scripts if not necessarily required (e.g. using a little bit more bash instead of fancier pwsh Cmdlets for establishing the NuGet release version as seen above).

Overall these micro improvements mean that an incoming pull request takes approximately two minutes to successfully build against the entire build matrix and produce a NuGet artifact as well:

Environment Variables

Now that the majority of the GitHub Action has been explained in detail we're just missing one final piece to complete the puzzle. Throughout this blog post I've been frequently referring to various environment variables which the script assumes to exist so that it is not specifically tied to a single project but rather applicable to many.

Some of those environment variables get created automatically by GitHub itself, but others have to be set up manually, which can be done either at the job level or globally at the top of the file:

env:
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  DOTNET_CLI_TELEMETRY_OPTOUT: true

  # Project name to pack and publish
  PROJECT_NAME: Giraffe

  # GitHub Packages Feed settings
  GITHUB_FEED: https://nuget.pkg.github.com/giraffe-fsharp/
  GITHUB_USER: dustinmoris
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  # Official NuGet Feed settings
  NUGET_FEED: https://api.nuget.org/v3/index.json
  NUGET_KEY: ${{ secrets.NUGET_KEY }}

The PROJECT_NAME variable is set to the .NET Core project name which is meant to get packed and published by this script. The script assumes that the solution follows the widely adopted folder structure of:

src/
+-- MyProject/
    +-- MyProject.csproj
tests/
+-- MyProject.Tests/
    +-- MyProject.Tests.csproj
MySolution.sln

In this example the PROJECT_NAME variable must be set to MyProject. Don't worry, if the solution contains more helper projects. Everything will get built and tested by the script. The build job executes a dotnet build and dotnet test from the root level of the repository, which means that it will pick up all projects from the repo's .sln file.

The GITHUB_FEED variable is a convenience pointer to the user's or organisation's GitHub feed. The GITHUB_USER variable requires a username of someone who has sufficient permissions to publish to the feed. As mentioned before, the GITHUB_TOKEN variable is auto-created by GitHub, which is why it's being assigned directly from ${{ secrets.GITHUB_TOKEN }}.

Finally the NUGET_FEED variable points towards the official NuGet gallery and the NUGET_KEY variable receives a private secret which must be set up manually either at the project or organisation level of the GitHub repository:

I configured the NUGET_KEY as an organisation wide secret, which means I don't have to set up any additional secrets for each repository any more.

If this rings your security bells then you are not entirely wrong. If you wonder if a malicious attacker could modify the GitHub workflow yaml file as part of a pull request and force bad code into the public domain then let me assure you that this won't be possible. GitHub makes it very clear that it doesn't pass secrets to workflows which were triggered by a pull request from a fork:

Secrets are not passed to workflows that are triggered by a pull request from a fork

The value for the NUGET_KEY secret has to be generated on www.nuget.org:

The End Result

The end result is a pretty elaborate CI/CD pipeline. All commits and pull requests will trigger a new run against a Linux, macOS and Windows environment and build and test the code across all these platforms. Additionally each build will produce a NuGet package as an artifact which can be downloaded and added to a local NuGet feed for test purposes or urgent matters. When features and bug fixes get eventually merged into the develop branch it will kick off a nightly build and publish an early pre-release version into the organisation's own GitHub packages feed. Finally when a release is being published an official NuGet package will be produced and not only pushed to the Github packages feed, but also to the official NuGet gallery. Nightly build packages are tagged with the workflow run ID and official packages with the associated git tag version. Everything is optimised for speed.

Four stages of a release

In total this set up supports the four stages of a release:

1. YOLO Release

Builds create a NuGet artifact, which is what I like to call the YOLO (you only live once) release. It's meant for super early testing or people who just don't give a damn :).

2. Nightly Builds

Merges into the develop branch will create an official nightly build which gets pushed into GitHub packages. It's still bleeding edge, but slightly more mature than the YOLO release.

3. Official pre-release packages

The next step in the release pipeline is an official pre-release package. It is basically the same as an official release package except that the git version tag follows the pre-release convention (e.g. v2.0.0-beta-23). Those packages are being released into the public NuGet gallery, but clearly marked as not a proper release candidate.

4. Official release packages

Finally the last release stage is a proper stable release. Same as before, it's triggered by a git tag, but this time with a stable release version (e.g. v2.0.0).

Workflow YAML

The final GitHub Action file where all the individual pieces are put together can be viewed on the official Giraffe repository.

Anyone is free to copy the build.yml file, apply custom changes to the environment variables at the top of the file and deploy it to their own .NET Core NuGet repository (it should just work)!

If the link above doesn't work or cannot tbe viewed then the entire build.yml file can also be seen in the script below:

build.yml

name: .NET Core
on:
  push:
  pull_request:
  release:
    types:
      - published
env:
  # Stop wasting time caching packages
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  # Disable sending usage data to Microsoft
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  # Project name to pack and publish
  PROJECT_NAME: Giraffe
  # GitHub Packages Feed settings
  GITHUB_FEED: https://nuget.pkg.github.com/giraffe-fsharp/
  GITHUB_USER: dustinmoris
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  # Official NuGet Feed settings
  NUGET_FEED: https://api.nuget.org/v3/index.json
  NUGET_KEY: ${{ secrets.NUGET_KEY }}
jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ ubuntu-latest, windows-latest, macos-latest ]
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.301
      - name: Restore
        run: dotnet restore
      - name: Build
        run: dotnet build -c Release --no-restore
      - name: Test
        run: dotnet test -c Release
      - name: Pack
        if: matrix.os == 'ubuntu-latest'
        run: dotnet pack -v normal -c Release --no-restore --include-symbols --include-source -p:PackageVersion=$GITHUB_RUN_ID src/$PROJECT_NAME/$PROJECT_NAME.*proj
      - name: Upload Artifact
        if: matrix.os == 'ubuntu-latest'
        uses: actions/upload-artifact@v2
        with:
          name: nupkg
          path: ./src/${{ env.PROJECT_NAME }}/bin/Release/*.nupkg
  prerelease:
    needs: build
    if: github.ref == 'refs/heads/develop'
    runs-on: ubuntu-latest
    steps:
      - name: Download Artifact
        uses: actions/download-artifact@v1
        with:
          name: nupkg
      - name: Push to GitHub Feed
        run: |
          for f in ./nupkg/*.nupkg
          do
            curl -vX PUT -u "$GITHUB_USER:$GITHUB_TOKEN" -F package=@$f $GITHUB_FEED
          done
  deploy:
    needs: build
    if: github.event_name == 'release'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.301
      - name: Create Release NuGet package
        run: |
          arrTag=(${GITHUB_REF//\// })
          VERSION="${arrTag[2]}"
          echo Version: $VERSION
          VERSION="${VERSION//v}"
          echo Clean Version: $VERSION
          dotnet pack -v normal -c Release --include-symbols --include-source -p:PackageVersion=$VERSION -o nupkg src/$PROJECT_NAME/$PROJECT_NAME.*proj
      - name: Push to GitHub Feed
        run: |
          for f in ./nupkg/*.nupkg
          do
            curl -vX PUT -u "$GITHUB_USER:$GITHUB_TOKEN" -F package=@$f $GITHUB_FEED
          done
      - name: Push to NuGet Feed
        run: dotnet nuget push ./nupkg/*.nupkg --source $NUGET_FEED --skip-duplicate --api-key $NUGET_KEY

Automate frequently used CLI commands

Tue, 02 Jun 2020 00:00:00 +0000

Welcome back to my blog! I haven’t written anything here for more than a year now, which is something I very much regret, but I mostly blame real life for it! Luckily due to the COVID-19 pandemic everything has slowed down a bit and I get to spend more time on my blog, side projects and some writing again. It’s always hard to get back to something after a long break, so I thought I'd start with a short blog post on a little productivity tip to break the current blogging hiatus!

Every developer has a range of commonly used CLI commands which they have to run frequently in order to execute some everyday tasks. These commands often have a bunch of additional flags and arguments attached to them which are really hard to remember if you only need to run them once every so often. One way of making it easier to run these commands is to configure short Aliases which are much easier to remember if you're using something like zsh or bash as your preferred shell (aliases also work in the KornShell and Bourne shell too). However, if you don't have a fancy Unix shell because you work on a restricted environment, an older version of Windows or you don't have the option of installing WSL (Windows Subsystem for Linux) on your Windows machine then there's another option which luckily works across most operating systems regardless which version or shell you're using!

Good old shell/batch scripts

Creating a shell (or batch) script with an easy to remember name and placing it in a directory which has been added to your OS's PATH environment variable is a very simple and easy way of achieving pretty much the same for the majority of automation tasks!

First create a new folder called something like useful-scripts in your home directory:

Windows

mkdir %userprofile%\useful-scripts

Unix

mkdir $HOME/useful-scripts

This will create a new directory called /useful-scripts under...

/users/<username>/useful-scripts on Linux or macOS
C:\Users\<username>\useful-scripts on Windows

Then you have to add this path to your PATH environment variable and voila, any shell or batch script which will be placed inside this folder will be available as if it was a command itself.

PATH on Windows

Note that when you're changing the PATH variable on Windows the set PATH=%PATH%;C:\Users\<username>\useful-scripts command will only set the path for the current terminal session and the change will not persist until the next one. In Windows 7 or later you can use the setx command to permanently set the PATH variable, however it's not very intuitive, because the command setx PATH %PATH%;C:\Users\<username>\useful-scripts will merge all values from your system wide PATH into your user specific PATH variable. Using setx /M will do the opposite, merge your user specific PATH values into the system wide PATH variable. Both are not recommended as it may cause unwanted side effects.

The GUI still remains the easiest way of permanently editing your PATH variable in Windows today.

Alternatively you can use the following PowerShell script too:

$currentPath = [Environment]::GetEnvironmentVariable("Path", "Machine")
$usefulScripts = "C:\Users\<username>\useful-scripts"
[Environment]::SetEnvironmentVariable("Path", $currentPath + ";" + $usefulScripts, "Machine")

PATH on Linux and macOS

Adding /users/<username>/useful-scripts to the PATH environment variable on Linux or macOS is fairly straight forward. You can either add it to the profile of your preferred shell or set it system wide by adding it to /etc/profile.d on Linux or /etc/paths.d on macOS.

Useful Scripts

Now let's explore some example scripts which might be useful to add to your newly created directory.

Switching kubectl context

A command which I frequently use is to authenticate the kubectl CLI with a specific Kubernetes cluster. I manage more than one Kubernetes cluster in Microsoft Azure as well as a few other ones in the Google Cloud Platform. Authenticating with an AKS cluster requires to run the following command:

az aks get-credentials --name <cluster-name> --resource-group <resource-group-name>

Authenticating with a cluster running inside the GKE requires a similar command:

gcloud container clusters get-credentials <cluster-name> --region <region-name>

If you manage multiple AKS clusters under different Azure subscriptions, or multiple GKE clusters under different Google Cloud projects then you'll also need to authenticate with the desired subscription/project before executing the get-credentials command:

gcloud config set project <gcp-project-name>

Having to type out all these commands every single time when wanting to connect to a specific cluster is a very tedious task and a good opportunity to shorten them into a more memorable abbreviation.

For example I'd like to only have to type gcp-cluster-1 if I want to authenticate with cluster-1 in GCP and only have to type aks-cluster-a when I'd like to authenticate with cluster-a in Azure.

Creating the following scripts inside useful-scripts will exactly enable this:

/users/<username>/useful-scripts/gcp-cluster-1

gcloud config set project <project-1>
gcloud container clusters get-credentials <cluster-1> --region <region-name>

/users/<username>/useful-scripts/gcp-cluster-2

gcloud config set project <project-2>
gcloud container clusters get-credentials <cluster-2> --region <region-name>

/users/<username>/useful-scripts/aks-cluster-a

az account set --subscription <subscription-name>
az aks get-credentials --name <cluster-1> --resource-group <resource-group-name>

Note that on Windows you'll have to add the .bat file extension for it to work and on macOS or Linux you'll have to give the execute permission to the newly created scripts:

chmod +x /users/<username>/useful-scripts/*

Cleaning up Docker containers and images

Another immensely useful script is to clean up old Docker containers and remove unused Docker images from your machine. If you frequently create and run Docker images locally then you will quickly accumulate many unused containers and unused or outdated images over time.

In older versions of Docker it was a bit of a mouthful to find and remove old Docker containers and images, but even with the latest Docker APIs and the availability of the prune command it's still nice to condense them into a single command:

Delete old containers (before Docker 1.13)

docker rm $(docker ps -a -q)

Delete untagged images (before Docker 1.13)

docker rmi $(docker images -a | grep "^<none>" | awk '{print $3}')

Delete old containers (Docker >= 1.13)

docker container -prune -f

Delete untagged images (Docker >= 1.13)

docker image prune -a -f

It can be more convenient to put both statements into a single docker-cleanup script:

docker container -prune -f
docker image prune -a -f

Output all custom scripts

Creating these sort of helper scripts is a nice way of simplifying and speeding up every day workflows. One script which I always like to include is an sos script which will output all other scripts in case I have ever forgetten any of the other scripts :)

sos.bat on Windows

dir %USERPROFILE%\useful-scripts

sos on Linux or macOS

ls $HOME/useful-scripts

This will enable me to run sos in order to get a list of all other available "commands".

Tips and tricks for ASP.NET Core applications

Thu, 28 Feb 2019 00:00:00 +0000

This is a small collection of some tips and tricks which I keep repeating myself in every ASP.NET Core application. There's nothing ground breaking in this list, but some general advice and minor tricks which I have picked up over the course of several real world applications.

Logging

Let's begin with logging. There are many logging frameworks available for .NET Core, but my absolute favourite is Serilog which offers a very nice structured logging interface for a vast number of available storage providers (sinks).

Tip 1: Configure logging before anything else

The logger should be the very first thing configured in an ASP.NET Core application. Everything else should be wrapped in a try-catch block:

public class Program
{
    public static int Main(string[] args) => StartWebServer(args);

    public static int StartWebServer(string[] args)
    {
        Log.Logger =
            new LoggerConfiguration()
                .MinimumLevel.Warning()
                .Enrich.WithProperty("Application", "MyApplicationName")
                .WriteTo.Console()
                .CreateLogger();

        try
        {
            WebHost.CreateDefaultBuilder(args)
                .UseSerilog()
                .UseKestrel(k => k.AddServerHeader = false)
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseStartup<Startup>()
                .Build()
                .Run();

            return 0;
        }
        catch (Exception ex)
        {
            Log.Fatal(ex, "Host terminated unexpectedly.");
            return -1;
        }
        finally
        {
            Log.CloseAndFlush();
        }
    }
}

Tip 2: Flush the logger before the application terminates

Make sure to put Log.CloseAndFlush(); into the finally block of your try-catch block so that no log data is getting lost when the application terminates before all logs have been written to the log stream.

Tip 3: Enrich your log entries

Configure your logger to automatically decorate every log entry with an Application property which contains a unique identifier for your application (typically a human readable name which identifies your app):

.Enrich.WithProperty("Application", "MyApplicationName")

This is extremely useful if you write logs from more than one application into a single log stream (e.g. a single Elasticsearch database). Personally I prefer to write logs from multiple (smaller) services of a coherent system into a single logging database and filter logs by properties.

Appending an additional Application property to all your application logs has the advantage that one can easily filter and view the overall health of a single application as well as getting a holistic view of the entire system.

Other really useful information which could be appended to your log entries is the application version and the environment name:

Log.Logger =
    new LoggerConfiguration()
        .MinimumLevel.Warning()
        .Enrich.WithProperty("Application", "MyApplicationName")
        .Enrich.WithProperty("ApplicationVersion", "<version number>")
        .Enrich.WithProperty("EnvironmentName", "Staging")
        .WriteTo.Console()
        .CreateLogger();

This will allow one to better visualise if issues had been resolved (or appeared) after a certain version has been deployed and it will also make it very easy to filter out any logs which might have accidentally been written from a different environment (e.g. a developer was debugging locally with the production connection string in their settings).

Startup Configuration

In ASP.NET Core there are two main places where features and functionality get configured. First there is the Configure method which can be used to plug middleware into the ASP.NET Core pipeline and secondly there is the ConfigureServices method to register dependencies.

For example adding Swagger to ASP.NET Core would look a bit like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    services.AddSwaggerGen(
        c =>
        {
            var name = "<my app name>"
            var version = "v1";

            c.SwaggerDoc(
                version,
                new Info { Version = version, Title = name });

            c.DescribeAllEnumsAsStrings();

            var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml";
            var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
            c.IncludeXmlComments(xmlPath);
        });
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app .UseSwagger()
        .UseSwaggerUI(
            c =>
            {
                var name = "<my app name>"
                var version = "v1";
                c.RoutePrefix = "";
                c.SwaggerEndpoint(
                    "/swagger/{version}/swagger.json", name);
            }
        )
        .UseMvc();
}

Middleware and dependencies are obviously two different things and therefore their configuration is split into two different methods, but from a developer's point of view it is very annoying that most features are configured across more than just one place.

Tip 4: Create 'Config' classes

One nice way to combat this is by creating a Config folder in the root of your ASP.NET Core application and create <FeatureName>Config classes for each feature/functionality which needs to be registered in Startup:

public static class SwaggerConfig
{
    private static string Name => "My Cool API";
    private static string Version => "v1";
    private static string Endpoint => $"/swagger/{Version}/swagger.json";
    private static string UIEndpoint => "";

    public static void SwaggerUIConfig(SwaggerUIOptions config)
    {
        config.RoutePrefix = UIEndpoint;
        config.SwaggerEndpoint(Endpoint, Name);
    }

    public static void SwaggerGenConfig(SwaggerGenOptions config)
    {
        config.SwaggerDoc(
            Version,
            new Info { Version = Version, Title = Name });

        config.DescribeAllEnumsAsStrings();

        var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml";
        var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
        config.IncludeXmlComments(xmlPath);
    }
}

By doing this one can move all related configuration of a feature into a single place and also nicely distinguish between the individual configuration steps (e.g. SwaggerUIConfig vs SwaggerGenConfig).

Afterwards one can tidy up the Startup class by invoking the respective class methods:

public void ConfigureServices(IServiceCollection services)
{
    services
        .AddMvc(MvcConfig.AddFilters)
        .AddJsonOptions(MvcConfig.JsonOptions);

    services.AddSwaggerGen(SwaggerConfig.SwaggerGenConfig);
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app .UseSwagger()
        .UseSwaggerUI(SwaggerConfig.SwaggerUIConfig)
        .UseMvc();
}

Tip 5: Extension methods for conditional configurations

Another common use case is to configure different features based on the current environment or other conditional cases:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
}

A neat trick which I like to apply here is to implement an extension method for conditional configurations:

public static class ApplicationBuilderExtensions
{
    public static IApplicationBuilder When(
        this IApplicationBuilder builder,
        bool predicate,
        Func<IApplicationBuilder> compose) => predicate ? compose() : builder;
}

The When extension method will invoke a compose function only if a given predicate is true.

Now with the When method someone can set up conditional middleware in a much nicer and fluent way:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app .When(env.IsDevelopment(), app.UseDeveloperExceptionPage)
        .When(!env.IsDevelopment(), app.UseHsts)
        .UseSwagger()
        .UseSwaggerUI(SwaggerConfig.SwaggerUIConfig)
        .UseMvc();
}

Exit scenarios

Tip 6: Don't forget to return a default 404 response

Don't forget to register a middleware which will return a 404 Not Found HTTP response if no other middleware was able to deal with an incoming request:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app .When(env.IsDevelopment(), app.UseDeveloperExceptionPage)
        .When(!env.IsDevelopment(), app.UseHsts)
        .UseSwagger()
        .UseSwaggerUI(SwaggerConfig.SwaggerUIConfig)
        .UseMvc()
        .Run(NotFoundHandler);
}

private readonly RequestDelegate NotFoundHandler =
    async ctx =>
    {
        ctx.Response.StatusCode = 404;
        await ctx.Response.WriteAsync("Page not found.");
    };

If you don't do this then a request which couldn't be matched by any middleware will be left unhandled (unless you have another web server sitting behind Kestrel).

Tip 7: Return non zero exit code on failure

Return a non zero exit code when the application terminates with an error. This will allow parent processes to pick up the fact that the application terminated unexpectedly and give them a chance to handle such a situation more gracefully (e.g. when your ASP.NET Core application is run from a Kubernetes cluster):

try
{
    // Start WebHost

    return 0;
}
catch (Exception ex)
{
    Log.Fatal(ex, "Host terminated unexpectedly.");
    return -1;
}
finally
{
    Log.CloseAndFlush();
}

Error Handling

Every ASP.NET Core application is likely going to have to deal with at least three types of errors:

Server errors
Client errors
Business logic errors

Server errors are unexpected exceptions which get thrown by an application. Normally these exceptions bubble up to a global error handler which will log the exception and return a 500 Internal Server Error response to the client.

Client errors are mistakes which a client can make when sending a request to the server. These normally include things like missing or wrong authentication data, badly formatted request bodies, calling endpoints which do not exist or perhaps sending data in an unsupported format. Most of these errors will get picked up by a built-in ASP.NET Core feature which will return a corresponding 4xx HTTP error back to the client.

Business logic errors are application specific errors which are not handled by ASP.NET Core by default because they are very unique to each individual application. For example an invoicing application might want to throw an exception when a customer tries to raise an invoice with an unsupported currency whereas an online gaming application might want to throw an error when a user ran out of credits.

These errors are often raised from lower level domain code and might want to return a specific 4xx or 5xx HTTP response back to the client.

Tip 8: Create a base exception type for domain errors

Create a base exception class for business or domain errors and additional exception classes which derive from the base class for all possible error cases:

public enum DomainErrorCode
{
    InsufficientCredits = 1000
}

public class DomainException : Exception
{
    public readonly DomainErrorCode ErrorCode;

    public DomainException(DomainErrorCode code, string message) : base(message)
    {
        ErrorCode = code;
    }
}

public class InsufficientCreditsException : DomainException
{
    public InsufficientCreditsException()
        : base(DomainErrorCode.InsufficientCredits,
                "User ran out of free credit. Please upgrade your plan to continue using our service.")
    { }
}

Include a unique DomainErrorCode for each custom exception type which later can be used to identify the specific error case from higher level code.

Afterwards one can use the newly created exception classes to throw more meaningful errors from inside the domain layer:

throw new InsufficientCreditsException();

This has now the benefit that the ASP.NET Core application can look for domain exceptions from a central point (e.g. custom error middleware) and handle them accordingly:

public class DomainErrorHandlerMiddleware
{
    private readonly RequestDelegate _next;

    public DomainErrorHandlerMiddleware(RequestDelegate next)
    {
        _next = next ?? throw new ArgumentNullException(nameof(next));
    }

    public async Task InvokeAsync(HttpContext ctx)
    {
        try
        {
            await _next(ctx);
        }
        catch(DomainException ex)
        {
            ctx.Response.StatusCode = 422;
            await ctx.Response.WriteAsync($"{ex.ErrorCode}: {ex.Message}");
        }
    }
}

Because every domain exception includes a unique DomainErrorCode the generic error handler can even implement a slightly different response based on the given domain error.

This architecture has a few benefits:

The domain layer can throw meaningful exceptions
The domain layer works nicely with the higher level web layer without tight coupling
Domain exceptions are clearly distinguishable from other errors
Domain exceptions are self documenting
The web layer can handle all domain errors in a unified way without having to replicate the same try-catch block across multiple controllers
The additional error code in the response can be parsed and understood by third party clients
The custom exception types can be easily documented through Swagger

Tip 9: Expose an endpoint which returns all error codes

When you followed tip 8 and implemented a custom exception type with a unique error code for each error case then it can be extremely handy to expose all possible error codes through a single API endpoint. This will allow third party clients to quickly retrieve a list of the latest possible error codes and their meaning:

[HttpGet("/error-codes")]
public ActionResult<IDictionary<int, string>> ErrorCodes()
{
    var values = Enum
        .GetValues(typeof(DomainErrorCode))
        .Cast<DomainErrorCode>();

    var result = new Dictionary<int, string>();

    foreach(var v in values)
        result.Add((int)v, v.ToString());

    return result;
}

Other Tips & Tricks

Tip 10: Expose a version endpoint

Another really useful thing to have in an API (or website) is a version endpoint. Often it can be extremely helpful to customer support staff, QA or other members of a team to quickly be able to establish what version of an application is being deployed to an environment.

This version is different than the customer facing API version which often only includes the major version number (e.g. https://my-api.com/v3/some/resource).

Exposing an endpoint which displays the current application version and the build date and time is a nice way of quickly making this information accessible to relevant people:

[HttpGet("/info")]
public ActionResult<string> Info()
{
    var assembly = typeof(Startup).Assembly;

    var creationDate = File.GetCreationTime(assembly.Location);
    var version = FileVersionInfo.GetVersionInfo(assembly.Location).ProductVersion;

    return Ok($"Version: {version}, Last Updated: {creationDate}");
}

Tip 11: Remove the 'Server' HTTP header

Whilst one is at configuring their ASP.NET Core application they might as well remove the Server HTTP header from every HTTP response by deactivating that setting in Kestrel:

.UseKestrel(k => k.AddServerHeader = false)

Tip 12: Working with Null Collections

My last tip on this list is not specific to ASP.NET Core but all of .NET Core development where a collection or IEnumerable type is being used.

How often do .NET developers write something like this:

var someCollection = GetSomeCollectionFromSomewhere();

if (someCollection != null && someCollection.Count > 0)
{
    foreach(var item in someCollection)
    {
        // Do stuff
    }
}

Adding a one line extension method can massively simplify the above code across an entire applicatoin:

public static class EnumerableExtensions
{
    public static IEnumerable<T> OrEmptyIfNull<T>(this IEnumerable<T> source) =>
        source ?? Enumerable.Empty<T>();
}

Now the above if statement can be reduced to a single loop like this:

var someCollection = GetSomeCollectionFromSomewhere();

foreach(var item in someCollection.OrEmptyIfNull())
{
    // Do stuff
}

Or converting the IEnumerbale to an IList and use the ForEach LINQ extension method to turn this into a one liner:

someCollection.OrEmptyIfNull().ToList().ForEach(i => i.DoSomething());

What tips and tricks do you have?

So this is it, this was my brief post on some tips and tricks which I like to apply in my personal ASP.NET Core development. I hope this was at least somewhat useful to someone?! Let me know what you think and please feel free to share your own tips and tricks which make your ASP.NET Core development life easier in the comments below!

Why you should learn F#

Mon, 17 Dec 2018 00:00:00 +0000

If you were thinking of learning a new programming language in 2019 then I would highly recommend to have a close look at F#. No matter if you are already a functional developer from a different community (Haskell, Clojure, Scala, etc.) or you are a complete newbie to functional programming (like I was 3 years ago) I think F# can equally impress you. F# is a functional first language. This means it is not a pure functional language but it is heavily geared towards the functional programming paradigm. However, because F# is also part of the .NET language family it is equally well equipped to write object oriented code too. Secondly F# is - contrary to common believe - an extremely well designed general purpose language. This means that F# is not only good for all sorts of "mathematical" stuff, but also for so much more. Without doubt F# is, like most other functional (algebraic) languages, greatly suited for this kind of work, but it is certainly not at the forefront of the creators of F# and neither a very common use case by most people who I know work with F#. So what is F# really good for? Well, the honest answer is almost anything! F# is an extremely pragmatic, expressive, statically typed programming language. Whether you want to build a distributed real time application, a service oriented web backend, a fancy looking single page app, mobile games, a line of business application or the next big social internet flop, F# will satisfy most if not all of your needs. As a matter of fact F# is probably a much better language for these types of applications than let's say Python, Java or C#. If you don't believe me then please continue reading and hopefully I will have convinced you by the end of this post!

Domain Driven Development

Before I started to write this article I asked myself why do I like F# so much? There are many reasons which came to my mind, but the one which really stood out to me was the fact that F# has some great capabilities of modelling a domain. After all, the majority of work which we do as software developers is to model real world processes into a digital abstraction of them. A language which makes this kind of work almost feel natural is immensely valuable and should not be missed.

Let's look at some code examples to demonstrate what I mean. For this task and for the rest of this blog post I'll be comparing F# with C# in order to show some of the benefits. I've chosen C# because many developers consider it as one of the best object oriented languages and mainly because C# is the language which I am the most proficient at myself.

Identifying bad design

A common use case in a modern application is to read a customer object from a database. In C# this would look something like this:

public Customer GetCustomerById(string customerId)
{
    // do stuff...
}

I have purposefully omitted the internals of this method, because from a caller's point of view the signature of a method is often all we know. Even though this operation is so simple (and very familiar) there are still a lot of unknowns around it:

Which values are accepted for the customerId? Can it have an empty string? Probably not, but will it instantly throw an ArgumentException or still try to fetch some user data?
Does the ID follow a specific format? What if the customerId has the correct format but is all upper case? Is it case sensitive or will the method normalise the string anyway?
What happens if a given customerId doesn't exist? Will it return null or throw an Exception? There's no way to find out without checking the internal implementation of this method (docs, decompilation, GitHub, etc.) or by testing against all sorts of input.
What happens if the database connection is down? Will it return the same result as if the customer didn't exist or will it throw a different type of exception?
How many different exception types will this code throw anyway?

The interface/signature of this method is not very clear in answering any of these questions. This is pretty poor given that the signature or interface of a method has the only purpose of defining a clear contract between the caller and the method itself. Of course there are many conventions which make C# developers feel safe, mostly by making broad assumptions about the underlying code, but at the end of the day these are only assumptions which can (and will eventually) lead to severe errors. If a library only slightly differs from an established convention then there is a high chance of introducing a bug which will catch them later.

If anything, conventions are rather weak workarounds for missing language features. Just like C# is perhaps seen as a better language than JavaScript, because of its statically typed feature, many functional programming languages are seen superior to C#, Java, and others, because of their domain modelling features.

There are ways of improving this code in C#, but none of those options are very straightforward (or often very cumbersome), which is why there is still plenty of code written like the one above.

F# makes correct code easy

F# on the other hand has a rich type system which allows developers to express the true state of a function. If a function might or might not return a Customer object then the function can return an object of type Option<'T>.

The Option<'T> type defines a return value which can either be something or nothing:

let getCustomerById customerId =
    match db.TryFindCustomerById customerId with
    | true, customer -> Some customer
    | false, _       -> None

It is important to understand that None is not another way of saying null, because null is truly nothing (there is nothing allocated in memory), whereas None is an actual object/case of type Option<'T>.

In this example the TryFindCustomerId method is a typical .NET member which has an out parameter defined like this:

bool TryFindCustomerById(string customerId, out Customer customer)

In F# you can use simple pattern matching to extract the out parameter on success:

match db.TryFindCustomerById customerId with
| true, customer -> Some customer
| false, _       -> None

The benefit of the Option<'T> type is not only that it is more expressive (and therefore more honest about the true state of the function), but also that it forces the calling code to implement the case of None, which means that a developer has to think of this edge case straight from the beginning:

let someOtherFunction customerId =
    match getCustomerById customerId with
    | Some customer -> // Do something when customer exist
    | None          -> // Do something when customer doesn't exist

Another extremely useful type which comes with F# is the Result<'T,'TError> type:

let validateCustomerId customerId =
    match customerId with
    | null -> Error "Customer ID cannot be null."
    | ""   -> Error "Customer ID cannot be empty."
    | id when id.Length <> 10 -> Error "Invalid Customer ID."
    | _ -> Ok (customerId.ToLower())

The validateCustomerId function will either return Ok with a normalised customerId or an Error object which contains a relevant error message. In this example 'T and 'TError are both of type string, but it doesn't have to be the same type and you can even wrap multiple types into a much richer return value such as Result<Option<Customer>, string list>.

The type system in F# allows for even more flexibility. One can easily create a new type which truly represents all possible outcomes of a function like getCustomerById:

type DataResult<'T> =
    | Success         of 'T option
    | ValidationError of string
    | DataError       of Exception

let getCustomerById customerId =
    try
        match validateCustomerId customerId with
        | Error msg -> ValidationError msg
        | Ok    id  ->
            match db.TryFindCustomerById id with
            | true, customer -> Some customer
            | false, _       -> None
            |> Success
    with ex -> DataError ex

The custom defined DataResult<'T> type declares three distinctive cases which the calling code might want to treat differently. By explicitly declaring a type which represents all these possibilities we can model the getCustomerById function in such a way that it removes all ambiguity about error- and edge case handling as well as preventing unexpected behaviour and forcing calling code to handle these cases.

F# makes invalid state impossible

So far we have always assumed that the customerId is a value of type string, but as we've seen this creates a lot of ambiguity around the allowed values for it and also forces a developer to write a lot of guard clauses to protect themselves from errors:

public Customer GetCustomerById(string customerId)
{
    if (customerId == null)
        throw new ArgumentNullException(nameof(customerId));

    if (customerId == "")
        throw new ArgumentException(
            "Customer ID cannot be empty.", nameof(customerId));

    if (customerId.Length != 10 || customerId.ToLower().StartsWith("c"))
        throw new ArgumentException(
            "Invalid customer ID", nameof(customerId));

    // do stuff...
}

The correct way of avoiding this anti-pattern is to model the concept of a CustomerId into its own type. In C# you can either create a class or struct to do so, but either way you'll end up writing a lot of boilerplate code to get the type to behave the way it should (eg. GetHashCode, Equality, ToString, etc.):

public class CustomerId
{
    public string Value { get; }

    public CustomerId(string customerId)
    {
        if (customerId == null)
            throw new ArgumentNullException(nameof(customerId));

        if (customerId == "")
            throw new ArgumentException(
                "Customer ID cannot be empty.",
                nameof(customerId));

        var value = customerId.ToLower();

        if (value.Length != 10 || value.StartsWith("c"))
            throw new ArgumentException(
                "Invalid customer ID",
                nameof(customerId));

        Value = value;
    }

    // Lots of overrides to make a
    // CustomerId behave the correct way
}

Needless to say that this is extremely annoying and the exact reason why it is so rarely seen in C#. Also a class is less favourable, because code which will accept a CustomerId will still have to deal with the possibility of null, which is not really a thing. A CustomerId should never be null, just like an int, a Guid or a DateTime can never be null. Once you've finished implementing a correct CustomerId type in C# you'll end up with 200 lines of code which itself open up a lot of room for further errors.

In F# we can define a new type as easily as this:

type CustomerId = private CustomerId of string

This version of CustomerId is basically a wrapper of string, but provides additional type safety, because one couldn't accidentally assign a string to a parameter of type CustomerId or vice versa.

The private access modifier prevents code from different modules or namespaces to create an object of type CustomerId. This is intentional, because now we can force the creation via a specific function like this:

module CustomerId =
    let create (customerId : string) =
        match customerId with
        | null -> Error "Customer ID cannot be null."
        | ""   -> Error "Customer ID cannot be empty."
        | id when id.Length <> 10 -> Error "Invalid Customer ID."
        | _ -> Ok (CustomerId (customerId.ToLower()))

The above implementation is extremely efficient and almost free of any noise. As a developer I didn't have to write a lot of boilerplate code and was able to focus on the actual domain which is what I really want:

The system has a type called CustomerId, which wraps a string.
The only way to create a CustomerId is via the CustomerId.create function, which does all of the relevant checks before emitting an object of CustomerId.
If a string violates the CustomerId requirements then a meaningful Error is returned and the calling code is forced to deal with this scenario.
The CustomerId object is immutable and non-nullable. Once successfully created all subsequent code can confidently rely on correct state.
The CustomerId type automatically inherits all other behaviour from a string, which means I didn't have to write a GetHashCode implementation, equality overrides, operator overloads and all of the other nonsense which I would have to do in C#.

This is a perfect example where F# can provide a lot of value with very few lines of code. Also because there is not much code to begin with there is very little room for making a mistake. The only real mistake I could have made is in the actual implementation of the CustomerId validation, which is more of a domain responsibility rather than a shortcoming of the language itself.

C# developers are not very used to model real world concepts like a CustomerId, an OrderId or an EmailAddress into their own types, because the language doesn't make it easy. These objects are often represented by very primitive types such as string or int and are being handled very loosely by the domain.

If you would like to learn more about Domain Driven Design in F# then I would highly recommend to watch Scott Wlaschin's Domain Modeling Made Functional presentation from NDC London. This is a fantastic talk with lots of food for thought and also the source of some of the ideas which I have introduced in this article:

Immutability and lack of Nulls

One of the greatest features of F# is that objects in F# are immutable by default and cannot be null. This makes it a lot easier to reason about code and also implement bug free applications. Not having to think twice if an object has changed state after passing it into a function or having to check for nulls has a huge impact on how easily someone can write reliable applications.

Saying goodbye to nulls

Tony Hoare, who invented (amongst many other great things) the null reference called it his billion dollar mistake. He even apologised for the creation of null during QCon London in 2009.

The problem with null is that it doesn't reflect any real state and yet has too many meanings at the same time. It's never clear if null means "unknown", "empty", "does not exist", "not initialised", "invalid", "some other error" or perhaps "end of line/file/stream/etc."? Today's scholars agree that the existence of null is certainly a mistake and hence why languages like C# try to slowly move away from it in their upcoming versions.

Fortunately F# never had nulls to begin with. The only way to force a null into F# is by interoperating with C# and not properly fencing it off.

Immutability > Mutability

Mutability is another topic where functional programming really shines. The problem is not mutability per se, but whether objects are mutable by default or not in a given language. It can only be one of the two and each programming language has to pick which one it wants it to be.

Immutability has the benefit of making code a lot easier to understand. It also prevents a lot of errors, because no class, method or function can change the state of an object after it had been created. This is particularly useful when objects get passed around between many different methods where the internal implementation is not always known (third party code, etc.).

On the other hand mutability doesn't have many benefits at all. It makes code arguably harder to follow, introduces a lot more ways for classes and methods to overstep their responsibility and lets poorly written libraries introduce unexpected behaviour. The small benefit of being able to directly mutate an object comes at a rather high cost.

Now the question is which one is easier, change an object in an immutable-by-default language, or introduce immutability in a mutable-by-default one?

I can quickly answer the first one by looking at F# and how it deals with the desire of changing an object's state. In F# mutations are performed by creating a new object with the modified values applied:

let c  = { Name = "Susan Doe"; Address = "1 Street, London, UK" }
let c' = { c with Address = "3 Avenue, Oxford, UK" }

This is a very elegant solution which produces almost the same outcome as if mutability was allowed (but without the cost).

Introducing immutability in C# is a little bit more awkward.

C# has no language construct which allows one to create an immutable object out of the box. First I have to create a new type, but I cannot use a class, because a class is a reference type which could be null. If null can be assigned to an object after it had been created then it is not immutable:

public class Customer
{
    public string Name { get; set; }
    public string Address { get; set; }
}

// Somewhere later in the program:

var c = new Customer();
c = null;

This leads me to using a struct:

public struct Customer
{
    public string Name { get; set; }
    public string Address { get; set; }
}

Now null is not an issue anymore, but the properties still are:

var c = new Customer();
c.Name = "Haha gotcha!";

Let's make the setters private then:

public struct Customer
{
    public string Name { get; private set; }
    public string Address { get; private set; }
}

Better, but not immutable yet. One could still do something like this:

public struct Customer
{
    public string Name { get; private set; }
    public string Address { get; private set; }

    public void ChangeName(string name)
    {
        Name = name;
    }
}

var c = new Customer();
c.ChangeName("Haha gotcha!");

The problem is not that ChangeName is public, but the fact that there is still a method which can alter the object's state after it was created.

Let's introduce two private backing fields for the properties and remove the setters altogether:

public struct Customer
{
    private string _name;
    private string _address;

    public string Name { get { return _name; } }
    public string Address { get { return _address; } }

    public Customer(string name, string address)
    {
        _name = name;
        _address = address;
    }
}

This looks perhaps better, but it's not (yet). A class member can still change the _name and _address fields from inside.

We can fix this by making the fields readonly:

public struct Customer
{
    private readonly string _name;
    private readonly string _address;

    public string Name { get { return _name; } }
    public string Address { get { return _address; } }

    public Customer(string name, string address)
    {
        _name = name;
        _address = address;
    }
}

Now this is immutable (at least for now), but a bit verbose. At this point we might as well collapse the properties into public readonly fields:

public struct Customer
{
    public readonly string Name;
    public readonly string Address;

    public Customer(string name, string address)
    {
        Name = name;
        Address = address;
    }
}

Alternatively with C# 6 (or later) we could also create readonly properties like this:

public struct Customer
{
    public string Name { get; }
    public string Address { get; }

    public Customer(string name, string address)
    {
        Name = name;
        Address = address;
    }
}

So far so good, but unless someone knows C# very well one could have easily gotten this wrong.

Unfortunately real world applications are never this simple though.

What if the Customer type would look more like this?

public class Address
{
    public string Street { get; set; }
}

public struct Customer
{
    public readonly string Name;
    public readonly Address Address;

    public Customer(string name, Address address)
    {
        Name = name;
        Address = address;
    }
}

var address = new Address { Street = "Springfield Road" };
var c = new Customer("Susan", address);
address.Street = "Gotcha";

At this point it should be evident that introducing immutability in C# is not as straightforward as someone might have thought.

This is another great example where the stark contrast between F# and C# really stands out. Writing correct code shouldn't be that hard and the language of choice can really make a difference.

SOLID made easy in F#

Object oriented programming is all about producing SOLID code. In order to understand and write decent C# one has to read at least five different books, study 20+ design patterns, follow composition over inheritance, practise TDD and BDD, apply the onion architecture, layer everything into tiers, MVP, MVVM and most importantly single responsibility all the things.

We all know the importance of these principles, because they are vital in keeping object oriented code in maintainable shape. Object oriented developers are so used to practise these patterns that it is unimaginable to them that someone could possibly produce SOLID code without injecting everything through a constructor. I've been there myself. The first time I saw functional code it looked plain wrong to me. I think most C# developers are put off by F# when they look at functional code for the very first time and don't see anything which looks familiar to them. There is no classes, no constructors and most importantly no IoC containers.

Functional code is often mistaken for procedural code to the inexperienced eye.

In functional programming everything is a function. The only design pattern which someone has to know is that a function is a first class citizen. Functions can be composed, instantiated, partially applied, passed around and executed.

There is this famous slide by Scott Wlaschin which nicely sums it up:

This slide deck is from Scott Wlaschin's Functional programming design patterns talk which can be viewed on Vimeo.

To hell with interfaces

In C# everything requires an interface. For example if a method requires to support multiple sort algorithms then the strategy pattern can help with this:

public interface ISortAlgorithm
{
    List<int> Sort(List<int> values);
}

public class QuickSort : ISortAlgorithm
{
    public List<int> Sort(List<int> values)
    {
        // Do QuickSort
        return values;
    }
}

public class MergeSort : ISortAlgorithm
{
    public List<int> Sort(List<int> values)
    {
        // Do MergeSort
        return values;
    }
}

public void DoSomething(ISortAlgorithm sortAlgorithm, List<int> values)
{
    var sorted = sortAlgorithm.Sort(values);
}

public void Main()
{
    var values = new List<int> { 9, 1, 5, 7 };
    DoSomething(new QuickSort(), values);
}

In F# the same can be done with simple functions:

let quickSort values =
    values // Do QuickSort

let mergeSort values =
    values // Do MergeSort

let doSomething sortAlgorithm values =
    let sorted = sortAlgorithm values
    ()

let main =
    let values = [| 9; 1; 5; 7 |]
    doSomething quickSort values

Functions in F#, which have the same signature, are interchangeable and don't require an explicit interface declaration. Both sort functions are therefore of type int list -> int list and can be alternatively passed into the doSomething function.

This is literally all it requires to implement the strategy pattern in in F#!

Let's look at a slightly more complex example to really demonstrate the strengths of F#.

Everything is a function

One of the most useful patterns in C# and one of my personal favourites is the decorator pattern. It allows adding additional functionality to an existing class without violating the open-closed principle of the SOLID guidelines. A password policy is a perfect example for this:

public interface IPasswordPolicy
{
    bool IsValid(string password);
}

public class BasePolicy : IPasswordPolicy
{
    public bool IsValid(string password)
    {
        return true;
    }
}

public class MinimumLengthPolicy : IPasswordPolicy
{
    private readonly int _minLength;
    private readonly IPasswordPolicy _nextPolicy;

    public MinimumLengthPolicy(int minLength, IPasswordPolicy nextPolicy)
    {
        _minLength = minLength;
        _nextPolicy = nextPolicy;
    }

    public bool IsValid(string password)
    {
        return
            password != null
            && password.Length >= _minLength
            && _nextPolicy.IsValid(password);
    }
}

public class MustHaveDigitsPolicy : IPasswordPolicy
{
    private readonly IPasswordPolicy _nextPolicy;

    public MustHaveDigitsPolicy(IPasswordPolicy nextPolicy)
    {
        _nextPolicy = nextPolicy;
    }

    public bool IsValid(string password)
    {
        if (password == null) return false;

        return password.ToCharArray().Any(c => char.IsDigit(c))
            && _nextPolicy.IsValid(password);
    }
}

public class MustHaveUppercasePolicy : IPasswordPolicy
{
    private readonly IPasswordPolicy _nextPolicy;

    public MustHaveUppercasePolicy(IPasswordPolicy nextPolicy)
    {
        _nextPolicy = nextPolicy;
    }

    public bool IsValid(string password)
    {
        if (password == null) return false;

        return password.ToCharArray().Any(c => char.IsUpper(c))
            && _nextPolicy.IsValid(password);
    }
}

public class Programm
{
    public void Main()
    {
        var passwordPolicy =
            new MustHaveDigitsPolicy(
                new MustHaveUppercasePolicy(
                    new MinimumLengthPolicy(
                        8, new BasePolicy())));

        var result = passwordPolicy.IsValid("Password1");
    }
}

During the instantiation of the passwordPolicy object one can decide which policies to use. A different password policy can be created without having to modify a single class. While this works really well in C#, it is also extremely verbose. There is a lot of code which had to be written for arguably little functionality at this point. I also had to use an additional interface and constructor injection to glue policies together. The passwordPolicy variable is of type IPasswordPolicy and can be injected anywhere a password policy is required. This is as good as it gets in C#.

The only thing which I could have possibly improved (by writing a lot more boilerplate code) would have been to add additional syntactic sugar to compose a policy like this:

var passwordPolicy =
    Policy.Create()
        .MustHaveMinimumLength(8)
        .MustHaveDigits()
        .MustHaveUppercase();

In F# the equivalent implementation is "just" functions again:

let mustHaveUppercase (password : string) =
    password.ToCharArray()
    |> Array.exists Char.IsUpper

let mustHaveDigits (password : string) =
    password.ToCharArray()
    |> Array.exists Char.IsDigit

let mustHaveMinimumLength length (password : string) =
    password.Length >= length

let isValidPassword (password : string) =
    mustHaveMinimumLength 8 password
    && mustHaveDigits password
    && mustHaveUppercase password

Just like in C# the passwordPolicy object implemented the IPasswordPolicy interface, the isValidPassword function implements the string -> bool signature which therefore can be interchanged with any other function which also implements string -> bool.

The F# solution is almost embarrassingly easy when compared to the overly complex one in C#. Yet I didn't have to compromise on any of the SOLID principles. Each function validates a single requirement (single responsibility) and can be tested in isolation. They can be swapped or mocked for any other function which also implements string -> bool and I can create multiple new policies without having to modify existing code (open closed principle):

let isValidPassword2 (password : string) =
    mustHaveMinimumLength 12 password
    && mustHaveUppercase password

Inversion of Control made functional

The only pattern which a functional developer has to understand is functions. To prove my point one last time I'll explore the Inversion of Control principle next.

First let's be clear what the Inversion of Control principle is, because many developers wrongly confuse it with the dependency injection pattern. The Inversion of Control principle states that a class shall never instantiate its own dependencies itself. Martin Fowler uses the term Hollywood Principle as in "Don't call us, we'll call you".

There are three distinctive design patterns which follow the IoC principle:

Dependency Injection
Factory
Service Locator

The Service Locator is considered an anti pattern so I won't go any further here.

The Factory pattern consists of two further sub-patterns:

The Dependency Injection pattern breaks down into three more sub-patterns:

Constructor Injection
Method Injection
Property Injection

Despite Constructor Injection being the most popular IoC pattern in object oriented programming, it is only one of many other patterns which follow the Dependency Inversion Principle. Each of these patterns is extremely useful and satisfies a specific use case which Constructor Injection couldn't do on its own.

This has nothing to do with F# directly, but I wanted to underline how the sheer number of different design patterns can sometimes be very confusing. It may take years for an OO software engineer to fully grasp the vast amount of concepts and understand how and when they play an important role.

Now that I got this out of the way let's take a look at how C# handles Dependency Injection via Constructor Injection:

public interface INotificationService
{
    void SendMessage(Customer customer, string message);
}

public class OrderService
{
    private readonly INotificationService _notificationService;

    public OrderService(INotificationService notificationService)
    {
        _notificationService =
            notificationService
            ?? throw new ArgumentNullException(
                nameof(notificationService));
    }

    public void CompleteOrder(Customer customer, ShoppingBasket basket)
    {
        // Do stuff

        _notificationService.SendMessage(customer, "Your order has been received.");
    }
}

Nothing should be surprising here. The OrderService has a dependency on an object of type INotificationService which is responsible for sending order updates to a customer.

There could be multiple implementations of the INotificationService, such as an SmsNotificationService or an EmailNotificationService:

public class EmailNotificationService : INotificationService
{
    private readonly EmailSettings _settings;

    public EmailNotificationService(EmailSettings settings)
    {
        _settings = settings;
    }

    public void SendMessage(Customer customer, string message)
    {
        // Do stuff
    }
}

Typically in C# these dependencies would get registered in an IoC container. I've skipped this part in order to keep the C# implementation small as it's already becoming large.

Now let's take a look at how dependency injection can be done in F#:

let sendEmailNotification emailSettings customer message =
    ()

let sendSmsNotification smsService apiKey customer message =
    ()

let completeOrder notify customer shoppingBasket =
    notify customer "Your order has been received."
    ()

That's it - Dependency Injection in functional programming is achieved by simply passing one function into another (basically what I've already been doing in the examples before)!

The only difference here is that the sendEmailNotification and sendSmsNotification functions do not share the same signature at the moment. Not only is emailSettings of a different type than smsService, but both functions also differ in the number of parameters they need. The sendEmailNotification function requires three parameters in total and the sendSmsNotification requires four. Furthermore the notify parameter of the completeOrder function doesn't know which concrete function will be injected and therefore doesn't care about anything except the Customer object and the string message. So how does it work?

The answer is partial application. In functional programming one can partially apply parameters of one function in order to generate a new one:

let sendEmailFromHotmailAccount =
    // Here I only apply the `emailSettings` parameter:
    sendEmailNotification hotmailSettings

let sendSmsWithTwillio =
    // Here I only apply the `smsService` and `apiKey` parameters:
    sendSmsNotification twilioService twilioApiKey

After partially applying both functions the newly created sendEmailFromHotmailAccount and sendSmsWithTwillio functions share the same signature again:

Customer -> string -> unit

Now both functions can be passed into the completeOrder function.

There is no need for an IoC container either. If one doesn't want to repeatedly pass all dependencies into the completeOrder function then partial application can be utilised once again:

let completeOrderAndNotify =
    emailSettings
    |> sendEmailNotification
    |> completeOrder

// Later in the program one would use:

completeOrderAndNotify customer shoppingBasket

If we compare this solution to the one from C# then there isn't much of a difference (except for simplicity). Classes mainly require their dependencies to be injected through their constructor and functions take other functions as a dependency. In C# all dependencies get registered only once at IoC container level. In F# all dependencies get "registered" only once through partial application. In both cases one can create mocks, stubs and fakes for their dependencies and unit test each class or function in isolation.

There is a few advantages with the functional approach though:

Dependencies can get "registered" (partially applied) closer to the functions where they belong.
Simpler by having a lot less code.
No additional (third party) IoC container required.
Dependency Injection is a pattern which has to be taught in OO programming whereas passing a function into another function is the most fundamental/normal thing one could do in functional programming.

Mark Seeman, author of Dependency Injection in .NET, did a fantastic talk on more advanced dependency patterns in F#. Watch his talk "From dependency injection to dependency rejection" on YouTube:

Simplicity

If there is one theme which has been consistent throughout this blog post then it must be the remarkable simplicity of F#. No matter if it is creating a new immutable type, expressing the true state of a function, modelling a domain or applying advanced programming patterns, F# always seems to have a slight edge over C#.

The abstinence of classes, complex design patterns, IoC containers, mutability, inheritance, overrides and interfaces has a few more benefits which come extremely handy at work.

First there is a lot less code to write. This makes applications smaller, faster to comprehend and much easier to maintain.

Secondly it allows for blazingly fast prototyping. In F# one can very quickly hack one function after another until a desired prototype has been reached. Furthermore, the additional work to transition from prototype to production is almost nothing. Since everything is a function and gets naturally compartmentalised into smaller functions the difference between a prototype- and a production-ready function is often very little.

Asynchronous programming

Speaking of simplicity, F# makes asynchronous programming strikingly easy:

let readFileAsync fileName =
    async {
        use stream = File.OpenRead(fileName)
        let! content = stream.AsyncRead(int stream.Length)
        return content
    }

There is a lot of great content available which explains the differences and benefits of F#'s asynchronous programming model so that I won't rehash everything again, but I would highly recommend to read Thomans Petricek's article on Async in C# and F#: Asynchronous gotchas in C#, followed by his blog series on Asynchronous C# and F#, including How do they differ? and How does it work?.

.NET Core

So far I've talked mostly about generic concepts of the functional programming paradigm, but there is a wealth of benefits which come specifically with F#. The obvious one is .NET Core. As we all know Microsoft is putting a lot of work into their new open source, cross platform, multi language runtime.

F# is part of .NET and therefore runs on all .NET runtimes, which include .NET Core, .NET Framework and Xamarin (Mono). This means that anyone can develop F# on either Windows, Linux or macOS. It also means that F# developers have access to a large eco system of extremely mature and high quality libraries. Because F# is a multi paradigm language (yes you can write object oriented code too if you want) it can reference and call into any third party package no matter if it was written in F#, C# or VB.NET.

Open Source

Long time before Microsoft embraced the OSS community they debuted with F# as their first language which was born out of Microsoft Research as an open source project from the get go. The open source community behind F# is very strong, with many contributions coming from outside Microsoft and driving the general direction of the language.

You can find all F# source code hosted on GitHub and start contributing by submitting an F# language suggestion first. When a suggestion gets approved then an RFC gets created with a corresponding discussion thread.

The F# language is under the direction of the F# Foundation with strong backing by Microsoft who is still the main driver of development.

Tooling

There is no match when it comes to tooling. Microsoft's .NET languages have always benefited from excellent tooling. Visual Studio was the uncontested leader for a long time, but in recent years the competition has racked up. JetBrains, the company who invented ReSharper, has released a new IntelliJ driven cross platform IDE called Rider. Meanwhile Microsoft developed a new open source editor called Code. Visual Studio Code has quickly emerged as the most popular development environment amongst programmers and boasts a huge marketplace of useful plugins. Thanks to Krzysztof Cieślak there is a superb extension called Ionide for F#.

Visual Studio, JetBrains Rider and Visual Studio Code with Ionide are three of the world's best programming IDEs which are cross platform compatible, run on all major operating systems and support F#.

F# conquering the web

As I mentioned at the very beginning F# is not just a language for algebraic stuff. Functional programming in general is a perfect fit for anything web related. A web application is basically a large function with a single parameter input (HTTP request) and a single parameter output (HTTP response).

F# on the Backend

F# has an abundance of diverse and feature rich web frameworks. My personal favourite is a library called Giraffe (disclaimer: I am the core maintainer of this project). Giraffe sits on top of ASP.NET Core, which means that it mostly piggybacks off the entire ASP.NET Core environment, its performance attributes and community contributions. In Giraffe a web application is composed through a combination of many smaller functions which get glued together via the Kleisli operator:

let webApp =
    choose [
        GET >=>
            choose [
                route "/ping" >=> text "pong"
                route "/"     >=> htmlFile "/pages/index.html"
            ]
        POST >=> route "/submit" >=> text "Successful" ]

Giraffe has also recently joined the TechEmpower Web Framework Benchmarks and ranks with a total of 1,649,957 req/sec as one of the fastest functional web frameworks available.

However, if Giraffe is not to your taste then there are many other great F# web libraries available:

Saturn (F# MVC framework built on top of Giraffe)
Suave (An entire web server written in F#)
Freya
WebSharper

ASP.NET Core and ASP.NET Core MVC are also perfectly compatible with F#.

F# on the Frontend

After F# set its mark on the server side of things it has also seen a lot of innovation on the frontend of the web.

Fable is an F# to JavaScript transpiler which is built on top of Babel, which itself is an extremely advanced JavaScript compiler. Babel, which is hugely popular and backed by large organisations such as Google, AirBnb, Adobe, Facebook, trivago and many more, is doing the heavy lifting of the compilation, whereas Fable is transpiling from F# to Babel's own abstract syntax tree. In simple terms you get the power of F# combined with the maturity and stability of Babel which allows you to write rich frontends in F#. Alfonso Garcia-Caro has done a magnificent job in merging the F# and JavaScript communities and recently released Fable 2 which comes with a two-fold speed boost as well as a 30%-40% reduced bundle size.

Fable and Babel are also open source and have a thriving community behind them.

On a complete different front Microsoft has worked on a new project called Blazor. Blazor is a single-page web application framework built on .NET that runs in the browser with WebAssembly. It supports all major .NET languages including F# and is currently in beta.

With the availability of Fable and Blazor there is a huge potential of what an F# developer can do on the web today.

F# Everywhere

F# is one of very few languages which can truly run anywhere! Thanks to .NET Core one can develop F# on any OS and run on any system. It can run natively on Windows, Linux and macOS or via a Docker container in a Kubernetes cluster. You can also run F# serverless functions in AWS Lambda or Azure Functions. Xamarin App Development brings F# to Android, iOS and Windows apps, and Fable and Blazor into the browser. Since .NET Core 2.1 one can even run F# on Alpine Linux and ARM! Machine learning, IoT and games are yet other areas where F# can be used today.

The list of supported platforms and architectures has been growing every year and I'm sure it will expand even further in the future!

Final Words

I've meant to write this blog post for a long time but never found the time to do it up until recently. My background is mainly C#, which is what I have been programming for more than ten years now and what I am still doing today. In the last three years I have taught myself F# and fallen madly in love with it. As a convert I get often asked what I like about F# and therefore I decided to put everything into writing. The list is obviously not complete and only a recollection of my own take on the main benefits of F#. If you think that I have missed something then please do not hesitate and let me know in the comments below. I see this blog post as an ever evolving resource where I hope I can point people to who have an interest in F#.

This blog post is also part of the F# Advent Calendar 2018 which has been kindly organised by Sergey Tihon again. Sergey does not only organise the yearly F# Advent Calendar, but also runs a weekly F# newsletter. Subscribe to his newsletter or follow him on Twitter and stay up to date with the latest developments on F#!

Useful Resources

Blog and Websites

Videos

Books

Conferences

Drawbacks of Stored Procedures

Thu, 29 Nov 2018 00:00:00 +0000

A few weeks ago I had a conversion with someone about the pros and cons of stored procedures. Personally I don't like them and try to avoid stored procedures as much as possible. I know there are some good reasons for using stored procedures (sometimes), but I'm also very well aware of the downsides which stored procedures bring with them.

This was not the first time that I had such a conversation and therefore I thought that I would quickly summarise all the reasons (and problems) which I had encountered with stored procedures in the past and put them into one concise blog post for future reference.

Testability

First and foremost business logic which is encapsulated in stored procedures becomes very difficult to test (if tested at all).

Some developers prefer to write a thin data access layer on top of stored procedures to workaround this issue, but even in this instance the extent of testing is mostly limited to a few integration tests only. Writing unit tests for any business logic inside a stored procedure is not possible, because there is no way to clearly separate the domain logic from the actual data. Mocking, faking or stubbing won't be possible either.

Debugging

Depending on the database technology debugging stored procedures will either not be possible at all or extremely clunky. Some relational databases, such as SQL Server, have some debugging capabilities and others none. There's nothing worse than having to use a database profiler to track down an application issue or to debug your database via print statements.

Versioning

Versioning is another crucial feature which stored procedures don't support out of the box. Putting stored procedure changes into re-runnable scripts and placing them into a version control system is certainly advisable, but it doesn't solve the problem that there is nothing inside a stored procedure which tells us which version a stored procedure is on and if there wasn't any other change being made after the latest script had been applied.

History

Similar to versioning, there's no history attached to stored procedures. Specifically if business logic spans across multiple stored procedures then it can be very difficult to establish the exact combination of different versions of different stored procedures at a given point in time.

Branching

Branching is a wonderful feature which enables the isolation of related software changes until a certain piece of work has been completed. This also allows development teams to work on multiple changes simultaneously without breaking each others' code.

As soon as a stored procedure requires to change then a development team will either face the maintenance of multiple database instances for their affected branches or have to coordinate the deployment of different stored procedures throughout the entire development life cycle.

Runtime Validation

Errors in stored procedures cannot be caught as part of a compilation or build step in a CI/CD pipeline. The same is true if a stored procedure went missing or another database error has crept into the application during the development process (e.g. missing permission to execute a stored procedure). In such a scenario a development team will often not know about the error until they execute the application. Catching fundamental mistakes like this can be very disruptive if it happens so late in the process.

Maintainability

Stored procedures introduce a cliff (or disconnect) between coherent functionality, because the domain logic gets split between the application- and the database layer. It's rarely clear where the line is drawn (e.g. which part of a query should go into the application layer and which part into the database layer?). Code which is divided between two disconnected systems makes it harder to read, comprehend and therefore reason about.

Fear of change

One of the biggest drawbacks of stored procedures is that it is extremely difficult to tell which parts of a system use them and which not. Especially if software is broken down into multiple applications then it's often not possible to find all references in one go (or at all if a developer doesn't have read access to all projects) and therefore it might be difficult to confidently establish how a certain change will affect the overall system. As a result stored procedures pose a huge risk of introducing breaking changes and development teams often shy away from making any changes at all. Sometimes this can lead to crippling new technological innovations.

Logging and Error handling

Robust software often relies on very sophisticated logging frameworks and/or error handling modules. An exception can get logged in several places, events can be raised based on different severity levels and custom notifications can be sent out to selective members of the team. However, business logic which is encapsulated inside a stored procedure cannot directly benefit from the same tools without having to either duplicate some of the code or introduce additional layers and workarounds.

Deployments

Not entirely impossible, but if a stored procedure has to change as part of a new application version then a zero downtime deployment will become a lot more difficult than without it. It's much easier to deploy and run two different versions of a web services than having to run two different versions of a set of stored procedures.

Conclusion

These are some of the issues which I had personally experienced when dealing with complex stored procedures in the past. There are obviously many good reasons for using stored procedures too, but overall I feel that the majority of these drawbacks are pretty big trade offs to swallow for very few benefits which can also be achieved without having to use a stored procedure.

If you think I've missed something or misstated one of the downsides of stored procedures then please let me know in the comments below.

ASP.NET Core Firewall

Mon, 22 Oct 2018 00:00:00 +0000

About a month ago I experienced an issue with one of my online services which is running in the Google Cloud and also protected by Cloudflare. I had noticed a spike in traffic which only showed up in my Google Cloud dashboard but not in Cloudflare. It was odd because all requests should normally route through Cloudflare's proxy servers but it seemed like someone was circumventing the DNS resolution and hitting my service directly via its exposed IP address. It was a significant issue, because the endpoint which was being hit was quite expensive and I had specifically configured Cloudflare to rate limit a caller to a maximum of 100 requests per second. Unfortunately someone must have spoofed my service's IP address and managed to bypass Cloudflare and the configured rate limit and was able to issue thousands of requests per second which put a huge strain on my rather cheap infrastructure.

After a quick Google search I discovered that bypassing Cloudflare is not that difficult and actually quite well documented on the internet. To my rescue I also discovered that Cloudflare publishes a list of all their IPv4 and IPv6 addresses which web administrators (is that even still a thing?) can use to set up IP address filtering on their web services to specifically prevent scenarios like this. I needed a quick solution and therefore went on another internet search for an ASP.NET Core middleware which would block all incoming requests which did not originate from a known Cloudflare address. The closest I could find was an article on a Client IP safelist, but it didn't allow me to "safelist" an entire IP address range like the ones which Cloudflare has made public (e.g. 103.21.244.0/22).

Knowing that I couldn't afford to run with this issue for much longer I decided to quickly hack my own IP address filtering middleware together. After a couple of hours of mad programming and copy pasting from Stackoverflow I had a quick and dirty solution deployed to production. It wasn't perfect, but it worked. My initial hack was able to validate an incoming IP address against all of Cloudflare's published CIDR notations and either grant or deny access to the requested resource. I was really happy how well it worked and after my pressing issue had been solved I wanted to deploy the same solution to all of my other ASP.NET Core services too.

A week later I published a slightly more polished version of the middleware as a NuGet package called Firewall. Today I deployed another version with major architectural improvements which made Firewall a much more flexible and useful library to a wider range of applications. In the rest of this blog post I would like to demonstrate some of the features which Firewall can do for an ASP.NET Core application.

How Firewall works

Firewall is an ASP.NET Core access control middleware. It primarily lets an application filter incoming requests based on their IP address and either grant or deny access. IP address filtering can be configured through a list of specific IP addresses and/or a list of CIDR notations:

using Firewall;

namespace BasicApp
{
    public class Startup
    {
        public void Configure(IApplicationBuilder app)
        {
            var allowedIPs =
                new List<IPAddress>
                    {
                        IPAddress.Parse("10.20.30.40"),
                        IPAddress.Parse("1.2.3.4"),
                        IPAddress.Parse("5.6.7.8")
                    };

            var allowedCIDRs =
                new List<CIDRNotation>
                    {
                        CIDRNotation.Parse("110.40.88.12/28"),
                        CIDRNotation.Parse("88.77.99.11/8")
                    };

            app.UseFirewall(
                FirewallRulesEngine
                    .DenyAllAccess()
                    .ExceptFromIPAddressRanges(allowedCIDRs)
                    .ExceptFromIPAddresses(allowedIPs));

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("Hello World!");
            });
        }
    }
}

The main feature can be enabled through the UseFirewall() extension method, which registers the FirewallMiddleware in the ASP.NET Core pipeline.

Rules for the Firewall are configured through the so called FirewallRulesEngine. The Firewall NuGet package comes with a set of default rules which are ready to use. For example the ExceptFromCloudflare() extension method will automatically configure the Firewall to retrieve the latest version of all of Cloudflare's IPv4 and IPv6 address ranges and subsequently validate incoming requests against them:

app.UseFirewall(
    FirewallRulesEngine
        .DenyAllAccess()
        .ExceptFromCloudflare());

A list of all currently available rules can be found on the project's documentation page.

Rules can be chained in the reverse order in which they will get evaluated against an incoming HTTP request:

var adminIPAddresses = new [] { IPAddress.Parse("1.2.3.4) };

app.UseFirewall(
    FirewallRulesEngine
        .DenyAllAccess()
        .ExceptFromCloudflare()
        .ExceptFromIPAddresses(adminIPAddresses)
        .ExceptFromLocalhost());

In the example above an incoming request will be first checked if it came from the same host, then if it came from the web administrator's home address and afterwards if it came from one of Cloudflare's IP addresses before the request will get denied. The request needs to satisfy only one of the rules in order to pass validation.

The reverse order of validation might seem a little bit weird at first, but it is simply explained by exposing the underlying architecture which is nothing more than a standard decorator composition pattern:

// Pseudo code:

var rules =
    new LocalhostRule(
        new IPAddressRule(
            new CloudflareRule(
                new DenyAllAccessRule())));

The FirewallRulesEngine is only syntactic sugar on top of the decorator pattern which allows a user to compose a set of rules without having to new up a bunch of classes and dependencies.

Custom Rules

Custom rules can either be configured via the ExceptWhen extension method or by creating a new class which implements the IFirewallRule interface:

var adminIPAddresses = IPAddress.Parse("1.2.3.4);

app.UseFirewall(
    FirewallRulesEngine
        .DenyAllAccess()
        .ExceptFromCloudflare()
        .ExceptWhen(ctx => ctx.Connection.RemoteIpAddress == adminIPAddress));

More complex rules can be created by implementing IFirewallRule:

public class IPCountryRule : IFirewallRule
{
    private readonly IFirewallRule _nextRule;
    private readonly IList<string> _allowedCountryCodes;

    public IPCountryRule(
        IFirewallRule nextRule,
        IList<string> allowedCountryCodes)
    {
        _nextRule = nextRule;
        _allowedCountryCodes = allowedCountryCodes;
    }

    public bool IsAllowed(HttpContext context)
    {
        const string headerKey = "CF-IPCountry";

        if (!context.Request.Headers.ContainsKey(headerKey))
            return _nextRule.IsAllowed(context);

        var countryCode = context.Request.Headers[headerKey].ToString();
        var isAllowed = _allowedCountryCodes.Contains(countryCode);

        return isAllowed || _nextRule.IsAllowed(context);
    }
}

There's a complete example of creating a custom rule available in the latest documentation.

X-Forwarded-For HTTP Header

Firewall has more features like a GeoIP2 powered CountryRule, detailed diagnostics for debugging and examples of how to load rule settings from external configuration providers, but one more ASP.NET Core feature which I wanted to specifically highlight here is the UseForwardedHeaders middleware.

If an application sits behind more than one proxy server (e.g. Cloudflare + a custom load balancer) then you'll need to enable the ForwardedHeader middleware in order to retrieve the correct client IP address in the HttpContext.Connection.RemoteIpAddress property:

public void Configure(IApplicationBuilder app)
{
    app.UseForwardedHeaders(
        new ForwardedHeadersOptions
        {
            ForwardedHeaders = ForwardedHeaders.XForwardedFor,
            ForwardLimit = 1
        }
    );

    app.UseCloudflareFirewall();

    app.Run(async (context) =>
    {
        await context.Response.WriteAsync("Hello World!");
    });
}

It is important to understand that this HTTP header is not guaranteed to be safe (as anything else which is client generated) and therefore it is not recommended to set the ForwardedLimit to a value greater than 1 unless the application is also set up with a list of trusted proxies (KnownProxies or KnownNetworks). If this is not done correctly then a malicious user could pretend to be a trusted source by setting the X-Forwarded-For header to a known (trusted) IP address.

If you think this short article was useful or if you've got your own ASP.NET Core website running behind Cloudflare then please go and check out the Firewall project and secure your application against unwanted traffic too.

Open Source Documentation

Mon, 08 Oct 2018 00:00:00 +0000

Since January 2017, which soon will be two years ago, I've been maintaining an open source project called Giraffe. Giraffe is a functional web framework for F# which allows .NET developers to build rich web applications on top of Microsoft's ASP.NET Core web framework in a functional first approach. Given that Giraffe is targeted at .NET web developers, who practise F# and who also like to use ASP.NET Core as their underlying web stack (as opposed to maybe WebSharper, Suave, Freya or others) I would consider Giraffe a fairly niche product.

However as niche as it may be, it still attracted a reasonable amount of developers who use it in a personal or professional capacity every day and as such documentation has become an integral part of maintaining the project from the get go.

As someone who has never maintained an open source project before I didn't really have much experience with this topic and therefore went with a very straightforward, simple and sort of lazy solution in the beginning. I put all initial documentation into the README.md file inside my Git repository.

Almost two years-, nearly 50 releases-, 47 total contributors-, more than 800 GitHub stars and more than 100 merged pull requests later the project's simple documentation approach hasn't changed much, and to be honest I have very little motivation to do something about it.

The main reason why I'm not particularly excited about migrating Giraffe's documentation to a different place is because I actually believe that the current way of how documentation is handled by Giraffe is a perfectly well working solution for its users, its contributors and myself.

In Giraffe the entire documentation - a full reference of what the web framework can or cannot do and how it integrates and operates within the ASP.NET Core pipeline - is placed in a single DOCUMENTATION.md file inside the project's Git repository, right next to the README.md file.

The documentation is not short either. There's quite a lot of content available and somehow this hasn't become a problem yet. The fact that all of the documentation is placed in a single large file has proven to be extremely advantageous if anything else.

Sometimes people ask me why I don't move the docs to a wiki page, like GitHub's wiki feature, or use GitHub pages or perhaps even a third party tool like Read the docs and my answer has always been the same: Because they all suck for documentation!

I find them particularly bad for documentation because they often fail to sufficiently address one of the two most important aspects of what makes good documentation in my opinion:

Help your users
Stay up to date (aka provide accurate information)

The first point is without doubt the most important aspect of all. If a project's documentation doesn't address 1.) sufficiently, then there's no point in even having documentation at all.

A single `DOCUMENTATION.md` file helps your users

Discovery

Before users can read (and hopefully benefit from) your documentation they need to be able to find it in the first place. Having all of your documentation in a single DOCUMENTATION.md file makes that discovery process a lot easier.

First the file is labelled in big capital letters stating "DOCUMENTATION". This is definitely a good start. Secondly it is right next to a file called README.md, which is a pretty well established (and understood) concept in the open source community. The chances that someone will find the DOCUMENTATION.md file which resides right next to the README.md file is considerably high I would say. An additional reference from the README.md file pointing to the DOCUMENTATION.md file often helps to eliminate any leftover chances of someone not finding my project's documentation.

Furthermore the discovery process is massively boosted by the fact that the DOCUMENTATION.md file is stored inside the project's Git repository. Given that most open source projects are hosted by one of the big Git hosting providers (GitHub, BitBucket, GitLab, etc.) there's a high probability that the DOCUMENTATION.md file will end up very high at a search engine's results page - and let's face it, this is probably how the majority of your users will search for documentation in the first place anyway. It will probably even outrank any custom homepage or wiki page which is rarely a coincidence. GitHub's, BitBucket's and other Git hosting provider's main business model is to provide a user friendly hosting platform for your open source projects as well as a user friendly platform for your own users to easily discover, browse and contribute to your project. These platforms have discoverability at their heart and are extremely well SEO optimised. If my project's DOCUMENTATION.md file can benefit from that optimisation then I'm all up for it!

Well understood structure

Once users find the link to the DOCUMENTATION.md file and click on it they will be presented with the actual content. At this point it is the maintainer's responsibility to make sure that the content is written and presented in such a way that it addresses the needs of its readers.

The ease of use and the initial experience of your users will often be determined by how familiar and comfortable they are with your documentation's structure. Wiki pages, GitHub pages, custom homepages, Read the docs pages and other third party tools have all their own take on how to structure a well laid out documentation. Is the menu on the top? Maybe on the left or right? Does it slide or collapse and where does it go when I open it on a mobile device with a much smaller screen size? Does it even have a menu? These questions seem extremely trivial, but they are often responsible for frustrations amongst users when they are not implemented in a good way.

I once had to read the documentation of a third party software which had so many menu items that the menu exceeded my screen's vertical length. Unfortunately at that time there was an issue with the website which didn't let me to scroll beyond the last visible item on the screen and I had to open my browser's developer tools to read the remaining items directly from the source code and open the links manually in a new tab. Needless to say that this was a terrible experience.

I'm sure that this issue has been fixed by now and I'm not saying that all third party tools have such a bad user experience, but regardless of what their actual UX is, each of them has a slightly different approach on how they structure their content. This is all good in the context of normal (commercial) websites, but it often forgets that documentation is much simpler than the usual website on the internet. Documentation doesn't require half of the things which a normal website can do today. Documentation is a read only exercise. Most importantly, documentation already has an ancient universally understood structure which every human is very likely to be familiar (and comfortable) with: The table of contents.

A table of contents is so simple and effective that it is still used across all various industries for any content which happens to be larger than a single page. Magazines, books, catalogues, contracts and manuals of all sorts of kind use a table of contents in order to structure their content in a user friendly way.

A table of contents let's one structure a large document into smaller pieces without having to divide the content into multiple pages. If it works for print, e-books or large PDFs, then I don't see why it wouldn't work for a project's DOCUMENTATION.md file which is hosted on the web:

Instead of having to break up an online documentation into multiple pages which need to be maintained individually by a person or team, a table of contents allows one to have a single large DOCUMENTATION.md file which can be maintained a lot easier without losing the convenience of a structured document.

There's also no ambiguity where a user will find the menu (= table of contents). It's always at the top (or the front) of a document, regardless if it's been opened on a large screen, a small mobile device or if it's printed on paper.

Search is king

Arguably the most important feature of good documentation (apart from the content itself) is the capability of quickly finding a certain piece of information. I would even go as far as to say that a website's search capability make or break a good documentation.

This again sounds like an extremely trivial problem and yet I feel like we're constantly getting it wrong to a point where a user can never really know whether they can trust a website's search box or not. Will the search give me meaningful suggestions when I type? Will it suggest relevant content even when I mistype something? How does the algorithm prioritise results? What key words do I need to search for? When I type "nodejs" will it also show me results which has "node.js" in the title? I'd hope so, but I can never be 100% sure.

For example there's a huge difference in the quality of results which I get from searching a programming question on Google or directly via StackOverflow.com. I want to get an answer from StackOverflow.com, but I still chose to search for it via Google because I know I'll get much better results.

The loss of trust in a website's search box has a deep implication on the user experience. If I'm browsing a website and I want to search for a specific topic which I am interested in and I don't get a perfect match to my first query then I'm rarely satisfied with my initial results. Often I will start altering my search query several times before accepting that the topic which I'm looking for might not be covered by the current state of documentation. Most likely I will even go to Google and try several search queries there before making any conclusions.

Another realisation which I have made is that nowadays it doesn't even matter how good the actual search of a website is. The mistrust in search boxes is so engrained in users' behaviour that even if a wiki page (or any other tool) has a perfectly functioning search box users will still go to a search engine and double check their results. It's kind of sad that users have been trained over time that the best way to search for any information is to leave the actual website where they want to get the information from and search for it on another estate.

The only search which I have found is more trusted than Google's search box is the browser's built-in text search (often quickly accessed through the CTRL+F or CMD+F shortcut). A single page DOCUMENTATION.md file can be easily and confidently searched by simply using a browser's built-in full text search - and regardless of what the results are - users know to trust it. From my experience this is the only time where a user actually trusts their initial search results and almost never re-validates via Google or any other search engine. In terms of user experience this is a huge benefit for keeping all documentation in a single large file. It makes finding information easy, predictable, trustworthy and extremely fast!

Other benefits for users

There's quite a few other benefits which users get from a single DOCUMENTATION.md file:

Search works offline. Having the documentation open in one tab allows a user to make effective use of it even when there is a loss of connectivity (e.g. sitting on a train, plane, airport, etc.).
Downloadable - The entire documentation can be downloaded with a single click and is usable as it is. This is particularly true for a Markdown file which is already human readable without any extra software. Given that most users probably have some sort of editor which can nicely format Markdown files makes it even better.
Print friendly - Some people like to have a print-out of the documentation. A single DOCUMENTATION.md file is extremely print friendly, since it doesn't need anything else in order to be user friendly for offline consumption.
Looks good on any screen size. A single large documentation Markdown file is extremely friendly towards all sort of screen sizes.

A single `DOCUMENTATION.md` file makes it easy to maintain

The most annoying thing about documentation is that it is extremely difficult to keep up to date if nobody really owns this responsibility. For an open source project this normally means that the lead maintainer has to constantly update the documentation after a new release has been published, which often proves to be a process which doesn't scale very well.

If all of the documentation is hosted in a single DOCUMENTATION.md file inside a project's Git repository then this responsibility can be easily shared with other contributors. There is a huge benefit in being able to search and replace function names and other code samples as part of the normal re-factoring process from an IDE. Without even having to actively think about documentation the normal search and replace feature in an IDE will automatically include any findings from the DOCUMENTATION.md too. It is also extremely useful to have the documentation being closely linked with the various branches of a project. This allows contributors and other maintainers to keep the documentation up to date as and when they make changes on a specific branch. Heck it can even be required to update the documentation as part of a pull request which gives the core maintainer a huge power over distributing this responsibility to other contributors as well.

Apart from never being out of sync another nice token is that there is never a delay in publishing the updated version of the documentation and the actual product release itself. As soon as a release is crafted and everything is merged back into master (from where the build will automatically deploy the latest version) the documentation has been merged into master as well and therefore become the latest updated iteration which matches the released code.

Overall experience

Personally I've not found a single downside of having a single large documentation file written up in Markdown and kept close to my code inside my project's Git repository yet. It has proven to be an extremely powerful pattern which allows me to easily keep Giraffe's documentation up to date, well maintained and extremely user friendly for everyone who's been using it so far.

I don't pretend that I've invented something new here or that I'm the first one doing this, but I simply wanted to state all the benefits which I have actively thought about when taking the decision to follow this pattern and I thought it would be worth sharing as I think other project's could certainly benefit of better documentation too.

Stop doing security (yourself)

Fri, 21 Sep 2018 00:00:00 +0000

Security has a very special place in software engineering. It is very different than other disciplines, because it is a problem space which cannot be approached with the same mindset as other problem spaces in our trade.

What makes security stand out so much is that it doesn't follow the same principles as the rest of software development. There is no different set of opinions. There is only one right opinion and the rest. There is not many ways how to achieve a secure something. There's only one very narrow, very particular and very unforgiving way of how to achieve a secure system (based on latest knowledge) and if you don't follow these steps very precisely then you'll end up with something which has more holes than a swiss cheese.

The other difference (between security and the rest of software engineering) is that you cannot delegate that competency to a single person. Your organisation might have one software architect, maybe one principal developer, maybe one Java or one database specialist, but if you have only one security expert then you have a problem. Security is such a complex topic, that even the most knowledgeable person on the planet will not know everything that is important.

Unfortunately, by nature, a system can never be assumed (or claimed) to be 100% secure, because there is no physics on this planet which could back this up. Therefore, the best and really only way which an organisation can use to its defence is to get their security model in front of as many eyes as possible. This is not an opinion, but a fact.

The most secure encryption is not the one which has been developed by the most knowledgeable person, but the one which has been reviewed, hacked and revised by the most people. There is a reason why all sophisticated cryptography algorithms are open to the public. Cryptographers deliberately want their work to be exposed to the largest possible audience and have it validated by them, because even they don't know how secure it is until it's been tested. This also explains why we have organisations specialising in penetration testing or why big organisations run public bug bounty programs.

With that in mind, my universal recommendation to any organisation which is not deeply involved in the InfoSec community is always to not do security by yourself. This can really not be stressed enough, but if you are not an industry leading expert in security, then don't even think about implementing your own password hasing algorithm, don't secure your API with a custom built authentication scheme and please don't build your own identity provider.

There is three reasons why:

You'll get it wrong
You don't have to do it, because others have already done it for you
You will get it wrong

If this little pep talk hasn't been convincing enough yet then there is another crucial reason why you shouldn't do security yourself: Security is out of your control and you'll probably live better by not having to deal with it.

Imagine your development team has made the choice to use Amazon's DynamoDb as their main data persistence layer. Two years down the line Amazon releases a new, improved and more state-of-the-art NoSQL database, but the development team doesn't find out about it until a few months later when one of the team members hears about it at a conference. After the conference the team sits together and decides to migrate to the new NoSQL database, but they won't do it for another six months, because they simply don't have the time and resources right now. Half of the team is on summer holiday, one person is sick, the development manager is on her honeymoon (so they lack formal approval anyway) and the person who originally came up with the current database architecture has left the company last year. It will take some time to come up with a good migration strategy and roll out in an efficient way. Luckily that is not an issue, because the current NoSQL database still works perfectly well and there's no immediate pressure to make a hasty change. The team is not bothered, nobody is stressing out and everyone enjoys their holidays before the team tackles the project later in the year.

Now imagine the same scenario but replace "database" with "identity provider" and replace "state-of-the-art" with "not full of security holes" and the whole story looks very different.

If someone drops a zero day vulnerability which affects your system then time is against you before you have even realised it. At this point you are already on the losing end and the problem which you're trying to tackle is not prevention, but damage control. The longer it will take you to fix, update and roll out a new version of your affected software the more likely you are going to be hit hard by this situation.

It will add very little consolation knowing that this whole disaster hasn't even been your fault. You might just have used a cryptographic implementation which was considered secure yesterday, but today you woke up to the news that some hackers have published a white paper on how to crack it in less than an hour. This was not a targeted attack against you, your business or your customers. This was simply a new discovery which has been dumped on the security community without much thought and now you and half of the world is affected by it. Before you think this type of stuff doesn't happen, let me quickly remind you of incidents like Heartbleed, Cloudbleed, Meltdown, Spectre, WannaCry, and so on.

In order to survive such a scenario you'll want to have certain things in place:

You'd hope to hear about the news before it's too late. This usually requires someone being extremely active in the InfoSec community, following multiple industry leading experts on Twitter, reading InfoSec blogs and being subscribed to InfoSec related RSS feeds and mailing lists.
You have a security response team which has the skillset, communication channels and authority within your organisation to deal with this matter as fast as possible
You have an extremely fast development life cycle. Your security response team can get the latest version of a project, make the necessary changes, get sufficient test coverage and deployment to your production systems turned around in as little time as possible
Your security response team is ideally available around the clock. You don't know when the news might come to light and given the huge time differences between different places it can be mission critical to have security employees working around the clock, or at least be prepared for an uncomfortable situation in the middle of the night. Your security team might wake up one or two hours away from the office and needs to have sufficient access to deal with such a situation away from their office desk.

Unless your business meets all of these requirements it might be flatout irresponsible to even think of writing your own custom security software if you don't have the means to deal with issues that come with it.

Now my main point is not to scare you of writing your own software, but to create some awareness that in the context of security you are probably better off by deferring these responsibilities to a third party which is a specialist in this field. Someone who lives and breathes security every day is much better equipped to deal with all the unknowns which we're confronted with every day.

Giraffe 1.1.0 - More routing handlers, better model binding and brand new model validation API

Fri, 16 Feb 2018 00:00:00 +0000

Last week I announced the release of Giraffe 1.0.0, which (apart from some initial confusion around the transition to TaskBuilder.fs) went mostly smoothly. However, if you have thought that I would be chilling out much since then, then you'll probably be disappointed to hear that today I've released another version of Giraffe with more exciting features and minor bug fixes.

The release of Giraffe 1.1.0 is mainly focused around improving Giraffe's routing API, making model binding more functional and adding a new model validation API.

Some of these features address some long requested functionality, so let's not waste any more time and get straight down to it.

Routes with trailing slashes

Often I've been asked how to make Giraffe treat a route with a trailing slash equal to the same route without a trailing slash:

https://example.org/foo/bar
https://example.org/foo/bar/

According to the technical specification a route with a trailing slash is not the same as a route without it. A web server might want to serve a different response for each route and therefore Giraffe (rightfully) treats them differently.

However, it is not uncommon that a web application chooses to not distinguish between two routes with and without a trailing slash and as such it wasn't a surprise when I received multiple bug reports for Giraffe not doing this by default.

Before version 1.1.0 one would have had to specify two individual routes in order to make it work:

let webApp =
    choose [
        route "/foo"  >=> text "Foo"
        route "/foo/" >=> text "Foo"
    ]

Giraffe version 1.1.0 offers a new routing handler called routex which is similar to route except that it allows a user to specify Regex in the route declaration.

This makes it possible to define routes with more complex rules such as allowing an optional trailing slash:

let webApp =
    choose [
        routex "/foo(/?)" >=> text "Foo"
    ]

The (/?) regex pattern denotes that there can be exactly zero or one slash after /foo.

With the help of routex and routeCix (the case insensitive version of routex) one can explicitly allow trailing slashes (or other non-standard behaviour) in a single route declaration.

Parameterised sub routes

Another request which I have seen on several occasions was a parameterised version of the subRoute http handler.

Up until Giraffe 1.0.0 there was only a routef and a subRoute http handler, but not a combination of both.

Imagine you have a localised application which requires a language parameter at the beginning of each route:

https://example.org/en-gb/foo
https://example.org/de-at/bar
etc.

In previous versions of Giraffe one could have used routef to parse the parameter and pass it into another HttpHandler function:

let fooHandler (lang : string) =
    sprintf "You have chosen the language %s." lang
    |> text

let webApp =
    choose [
        routef "/%s/foo" fooHandler
    ]

This was all good up until someone needed to make use of something like routeStartsWith or subRoute to introduce additional validation/authentication before invoking the localised routes:

let webApp =
    choose [
        // Doesn't require authentication
        routef "/%s/foo" fooHandler
        routef "/%s/bar" barHandler

        // Requires authentication
        requiresAuth >=> choose [
            routef "/%s/user/%s/foo" userFooHandler
            routef "/%s/user/%s/bar" userBarHandler
        ]
    ]

The problem with above code is that the routing pipeline will always check if a user is authenticated (and potentially return an error response) before even knowing if all subsequent routes require it.

The workaround was to move the authentication check into each of the individual handlers, namely the userFooHandler and the userBarHandler in this instance.

A more elegant way would have been to specify the authentication handler only one time before declaring all protected routes in a single group. Normally the subRoute http handler would make this possible, but not if routes have parameterised arguments at the beginning of their paths.

The new subRoutef http handler solves this issue now:


let webApp =
    choose [
        // Doesn't require authentication
        routef "/%s/foo" fooHandler
        routef "/%s/bar" barHandler

        // Requires authentication
        subRoutef "%s-%s/user" (
            fun (lang, dialect) ->
                // At this point it is already
                // established that the path
                // is a protected user route:
                requiresAuth
                >=> choose [
                    routef "/%s/foo" (userFooHandler lang dialect)
                    routef "/%s/bar" (userBarHandler lang dialect)
                ]
        )
    ]

The subRoutef http handler can pre-parse parts of a route and group a collection of cohesive routes in one go.

Improved model binding and model validation

The other big improvements in Giraffe 1.1.0 were all around model binding and model validation.

The best way to explain the new model binding and validation API is by looking at how Giraffe has done model binding in previous versions:

[<CLIMutable>]
type Adult =
    {
        FirstName  : string
        MiddleName : string option
        LastName   : string
        Age        : int
    }
    override this.ToString() =
        sprintf "%s %s"
            this.FirstName
            this.LastName

    member this.HasErrors() =
        if this.Age < 18 then Some "Person must be an adult (age >= 18)."
        else if this.Age > 150 then Some "Person must be a human being."
        else None

module WebApp =
    let personHandler : HttpHandler =
        fun (next : HttpFunc) (ctx : HttpContext) ->
            let adult = ctx.BindQueryString<Adult>()
            match adult.HasErrors() with
            | Some msg -> RequestErrors.BAD_REQUEST msg
            | None     -> text (adult.ToString())

    let webApp _ =
        choose [
            route "/person" >=> personHandler
            RequestErrors.NOT_FOUND "Not found"
        ]

In this example we have a typical F# record type called Adult. The Adult type has an override for its ToString() method to output something more meaningful than .NET's default and an additional member called HasErrors() which checks if the provided data is correct according to the application's business rules (e.g. an adult must have an age of 18 or over).

There's a few problems with this implementation though. First you must know that the BindQueryString<'T> extension method is a very loose model binding function, which means it will create an instance of type Adult even if some of the mandatory fields (non optional parameters) were not present in the query string (or badly formatted). While this "optimistic" model binding approach has its own advantages, it is not very idiomatic to functional programming and requires additional null checks in subsequent code.

Secondly the model validation has been baked into the personHandler which is not a big problem at first, but means that there's a lot of boilerplate code to be written if an application has more than just one model to work with.

Giraffe 1.1.0 introduces new http handler functions which make model binding more functional. The new tryBindQuery<'T> http handler is a stricter model binding function, which will only create an instance of type 'T if all mandatory fields have been provided by the request's query string. It will also make sure that the provided data is in the correct format (e.g. a numeric value has been provided for an int property of the model) before returning an object of type 'T:

[<CLIMutable>]
type Adult =
    {
        FirstName  : string
        MiddleName : string option
        LastName   : string
        Age        : int
    }
    override this.ToString() =
        sprintf "%s %s"
            this.FirstName
            this.LastName

    member this.HasErrors() =
        if this.Age < 18 then Some "Person must be an adult (age >= 18)."
        else if this.Age > 150 then Some "Person must be a human being."
        else None

module WebApp =
    let adultHandler (adult : Adult) : HttpHandler =
        fun (next : HttpFunc) (ctx : HttpContext) ->
            match adult.HasErrors() with
            | Some msg -> RequestErrors.BAD_REQUEST msg
            | None     -> text (adult.ToString())

    let parsingErrorHandler err = RequestErrors.BAD_REQUEST err

    let webApp _ =
        choose [
            route "/person" >=> tryBindQuery<Adult> parsingErrorHandler None adultHandler
            RequestErrors.NOT_FOUND "Not found"
        ]

The tryBindQuery<'T> requires three parameters. The first is an error handling function of type string -> HttpHandler which will get invoked when the model binding fails. The string parameter in that function will hold the specific model parsing error message. The second parameter is an optional CultureInfo object, which will get used to parse culture specific data such as DateTime values or floating point numbers. The last parameter is a function of type 'T -> HttpHandler, which will get invoked with the parsed model if model parsing was successful.

By using tryBindQuery<'T> there is no danger of encountering a NullReferenceException or the need of doing additional null check any more. By the time the model has been passed into the adultHandler it has been already validated against any data contract violations (e.g. all mandatory fields have been provided, etc.).

At this point the semantic validation of business rules is still embedded in the adultHandler itself. The IModelValidation<'T> interface can help to move this validation step closer to the model and make use of a more generic model validation function when composing the entire web application together:

[<CLIMutable>]
type Adult =
    {
        FirstName  : string
        MiddleName : string option
        LastName   : string
        Age        : int
    }
    override this.ToString() =
        sprintf "%s %s"
            this.FirstName
            this.LastName

    member this.HasErrors() =
        if this.Age < 18 then Some "Person must be an adult (age >= 18)."
        else if this.Age > 150 then Some "Person must be a human being."
        else None

    interface IModelValidation<Adult> with
        member this.Validate() =
            match this.HasErrors() with
            | Some msg -> Error (RequestErrors.BAD_REQUEST msg)
            | None     -> Ok this

module WebApp =
    let textHandler (x : obj) = text (x.ToString())
    let parsingErrorHandler err = RequestErrors.BAD_REQUEST err
    let tryBindQuery<'T> = tryBindQuery<'T> parsingErrorHandler None

    let webApp _ =
        choose [
            route "/person" >=> tryBindQuery<Adult> (validateModel textHandler)
        ]

By implementing the IModelValidation<'T> interface on the Adult record type we can now make use of the validateModel http handler when composing the /person route. This functional composition allows us to entirely get rid of the adultHandler and keep a clear separation of concerns.

First the tryBindQuery<Adult> handler will parse the request's query string and create an instance of type Adult. If the query string had badly formatted or missing data then the parsingErrorHandler will be executed, which allows a user to specify a custom error response for data contract violations. If the model could be successfully parsed, then the validateModel http handler will be invoked which will now validate the business rules of the model (by invoking the IModelValidation.Validate() method). The user can specify a different error response for business rule violations when implementing the IModelValidation<'T> interface. Lastly if the model validation succeeded then the textHandler will be executed which will simply use the object's ToString() method to return a HTTP 200 text response.

All functions are generic now so that adding more routes for other models is just a matter of implementing a new record types for each model and registering a single route in the web application's composition:

let webApp _ =
    choose [
        route "/adult" >=> tryBindQuery<Adult> (validateModel textHandler)
        route "/child" >=> tryBindQuery<Child> (validateModel textHandler)
        route "/dog"   >=> tryBindQuery<Dog>   (validateModel textHandler)
    ]

Overall the new model binding and model validation API aims at providing a more functional counter part to MVC's model validation, except that Giraffe prefers to use functions and interfaces instead of the System.ComponentModel.DataAnnotations attributes. The benefit is that data attributes are often ignored by the rest of the code while a simple validation function can be used from outside Giraffe as well. F# also has the benefit of having a better type system than C#, which means that things like the [<Required>] attribute have little use if there is already an Option<'T> type.

Currently this new improved way of model binding in Giraffe only works for query strings and HTTP form payloads via the tryBindQuery<'T> and tryBindFrom<'T> http handler functions. Model binding functions for JSON and XML remain with the "optimistic" parsing model due to the underlying model binding libraries (JSON.NET and XmlSerializer), but a future update with improvements for JSON and XML is planned as well.

In total you have the following new model binding http handlers at your disposal with Giraffe 1.1.0:

HttpHandler	Description
`bindJson<'T>`	Traditional model binding. This is a new http handler equivalent of `ctx.BindJsonAsync<'T>`.
`bindXml<'T>`	Traditional model binding. This is a new http handler equivalent of `ctx.BindAsync<'T>`.
`bindForm<'T>`	Traditional model binding. This is a new http handler equivalent of `ctx.BindFormAsync<'T>`.
`tryBindForm<'T>`	New improved model binding. This is a new http handler equivalent of a new `HttpContext` extension method called `ctx.TryBindFormAsync<'T>`.
`bindQuery<'T>`	Traditional model binding. This is a new http handler equivalent of `ctx.BindQueryString<'T>`.
`tryBindQuery<'T>`	New improved model binding. This is a new http handler equivalent of a new `HttpContext` extension method called `ctx.TryBindQueryString<'T>`.
`bindModel<'T>`	Traditional model binding. This is a new http handler equivalent of `ctx.BindModelAsync<'T>`.

The new model validation API works with any http handler which returns an object of type 'T and is not limited to tryBindQuery<'T> and tryBindFrom<'T> only.

Roadmap overview

To round up this blog post I thought I'll quickly give you a brief overview of what I am planning to tackle next.

The next release of Giraffe is anticipated to be version 1.2.0 (no date set yet) which will mainly focus around improved authentication and authorization handlers (policy based auth support), better CORS support and hopefully better Anti-CSRF support.

After that if nothing else urgent comes up I shall be free to go over two bigger PRs in the Giraffe repository which aim at providing a Swagger integration API and a higher level API of working with web sockets in ASP.NET Core.

Announcing Giraffe 1.0.0

Fri, 09 Feb 2018 00:00:00 +0000

I am pleased to announce the release of Giraffe 1.0.0, a functional ASP.NET Core web framework for F# developers. After more than a year of building, improving and testing the foundations of Giraffe it makes me extremely happy to hit this important milestone today. With the help of 32 independent contributors, more than a hundred closed GitHub issues and an astonishing 79 merged pull requests (and counting) it is fair to say that Giraffe has gone through many small and big changes which made it what I believe one of the best functional web frameworks available today.

The release of Giraffe 1.0.0 continues with this trend and also brings some new features and improvements along the way:

Streaming support

Giraffe 1.0.0 offers a new streaming API which can be used to stream (large) files and other content directly to a client.

A lot of work has been put into making this feature properly work like supporting conditional HTTP headers and range processing capabilities. On top of that I was even able to help iron out a few bugs in ASP.NET Core MVC's implementation as well (loving the fact that ASP.NET Core is all open source).

Conditional HTTP Headers

In addition to the new streaming API the validation of conditional HTTP headers has been exposed as a separate feature too. The ValidatePreconditions function is available as a HttpContext extension method which can be used to validate If-{...} HTTP headers from within any http handler in Giraffe. The function will self determine the context in which it is called (e.g. GET POST, PUT, etc.) and return a correct result denoting whether a request should be further processed or not.

Configuration of serializers

A much desired and important improvement was the ability to change the default implementation of data serializers and content negotiation. Giraffe 1.0.0 allows an application to configure the default JSON or XML serializer via ASP.NET Core's services container.

Detailed XML documentation

For the first time Giraffe has detailed XML documentation for all public facing functions available:

Even though this is not a feature itself, it aims at improving the general development experience by providing better IntelliSense and more detailed information when working with Giraffe.

Giraffe.Tasks deprecated

When Giraffe introduced the task {} CE for the first time it was a copy of the single file project TaskBuilder.fs written by Robert Peele. However, maintaining our own copy of the task CE is resource expensive and not exactly my personal field of expertise. Besides that, since the initial release Robert has made great improvements to TaskBuilder.fs whereas Giraffe's version has been lacking behind. When TaksBuilder.fs has been published to NuGet it felt like a good idea to deprecate Giraffe.Tasks and resort back to the original.

This allows me and other Giraffe contributors to focus more on the web part of Giraffe and let Robert do his excellent work on the async/task side of things. Otherwise nothing has changed and Giraffe will continue to build on top of Task and Task<'T>. If you use Giraffe.Tasks outside of a Giraffe web application then you can continue doing so by referencing TaskBuilder.fs instead.

Giraffe also continues to use exclusively the context insensitive version of the task CE (meaning all task objects are awaited with ConfigureAwait(false)). If you encouter type inference issues after the upgrade to Giraffe 1.0.0 then you might have to add an extra open statement to your F# file:

open FSharp.Control.Tasks.ContextInsensitive

This is normally not required though unless you have do! bindings in your code.

If you like the task {} computation expression then please go to the official GitHub repository and hit the star button to show some support!

TokenRouter as NuGet package

TokenRouter is a popular alternative to Giraffe's default routing API aimed at providing maximum performance. Given the complexity of TokenRouter and the fact that Giraffe already ships a default version of the routing API it made only sense to decouple the TokenRouter into its own repository.

This change will allow TokenRouter to become more independent and evolve at its own pace. TokenRouter can also benefit from having its own release cycle and be much bolder in introducing new features and breaking changes without affecting Giraffe.

If your project is using the TokenRouter API then you will need to add a new dependency to the Giraffe.TokenRouter NuGet package now. The rest remains unchanged.

Improved documentation

At last I have worked on improving the official Giraffe documentation by completely restructuring the document, providing a wealth of new information and focusing on popular topics by demand.

The documentation has also been broken out of the README, but remains as a Markdown file in the git repository for reasons which I hope to blog about in a separate blog post soon.

The complete list of changes and new features can be found in the official release notes.

Thank you for reading this blog and using Giraffe (and if you don't, then give it a try ;))!

Extending the Giraffe template with different view engine options

Thu, 21 Dec 2017 00:00:00 +0000

This is going to be a quick but hopefully very useful tutorial on how to create a more complex template for the dotnet new command line tool.

If you have ever build a Giraffe web application then you've probably started off by installing the giraffe-template into your .NET CLI and then created a new boilerplate application by running the dotnet new command:

dotnet new giraffe

(There is currently a bug with the .NET CLI which forces you to specify the -lang parameter.)

Previously this would have created a new Giraffe web application which would have had the Giraffe.Razor NuGet package included and a default project structure with MVC's famous Razor view engine.

Since today (after you've updated the giraffe-template to the latest version) you can choose between three different options:

giraffe - Giraffe's default view engine
razor - MVC Razor views
dotliquid - DotLiquid template engine

The dotnet new giraffe command optionally supports a new parameter called --ViewEngine or short -V now:

dotnet new giraffe -ViewEngine razor

If you are unsure which options are available you can always request help by running:

dotnet new giraffe --help

The output of the command line will display all available options and supported values as well:

Giraffe Web App (F#)

Author: Dustin Moris Gorski, David Sinclair and contributors

Options:
  -V|--ViewEngine
        giraffe    - Default GiraffeViewEngine
        razor      - MVC Razor views
        dotliquid  - DotLiquid template engine

        Default: giraffe

If you do not specify a view engine then the dotnet new giraffe command will automatically create a new Giraffe web application with the default GiraffeViewEngine engine.

Creating multiple project templates as part of one dotnet new template

There are many ways of how I could have programmed these options into the Giraffe template, but none of them are very obviously documented in one place. The documentation of the dotnet templating engine¹ is fairly scattered across multiple resources and hard to understand if you have never worked with it before. Part of today's blog post I thought I'll quickly sum up one of the options which I believed was the cleanest and most straight forward one.

Each view engine has a significant impact on the entire project structure, such as NuGet package dependencies, folder structure, code organisation and files which need to be included. I didn't want to hack around with #if - #else switches and introduce complex add-, modify- or delete rules and consequently decided that the easiest and least error prone way would be to create a complete independent template for each individual view engine first:

src
+-- giraffe-template.nuspec
|
+-- content
    |
    +-- .template.config
    |   +-- template.json
    |
    +-- DotLiquid.Template
    |   +-- Views
    |   +-- WebRoot
    |   +-- Program.fs
    |   +-- AppNamePlaceholder.fsproj
    |
    +-- Razor.Template
    |   +-- Views
    |   +-- WebRoot
    |   +-- Program.fs
    |   +-- AppNamePlaceholder.fsproj
    |
    +-- Giraffe.Template
        +-- WebRoot
        +-- Program.fs
        +-- AppNamePlaceholder.fsproj

I split the content of the Giraffe template into three distinctive sub templates:

DotLiquid.Template
Razor.Template
Giraffe.Template

As you can see from the diagram there's still only one .template.config\template.json file at the root of the content folder and only one giraffe-template.nuspec file.

The benefit of this structure is very simple. There is a clear separation of each template and each template is completely independent of the other templates which makes maintenance very straight forward. I can work on each template as if they were small projects with full Intellisense and IDE support and being able to build, run and test each application.

The next step was to create the --ViewEngine parameter inside the template.json file:

"symbols": {
    "ViewEngine": {
      "type": "parameter",
      "dataType": "choice",
      "defaultValue": "giraffe",
      "choices": [
        {
          "choice": "giraffe",
          "description": "Default GiraffeViewEngine"
        },
        {
          "choice": "razor",
          "description": "MVC Razor views"
        },
        {
          "choice": "dotliquid",
          "description": "DotLiquid template engine"
        }
      ]
    }
  }

All I had to do was to define a new symbol called ViewEngine of type parameter and data type choice. Then I specified all supported options via the choice array and set the giraffe option as the default value.

Now that the ViewEngine parameter has been created I was able to use it from elsewhere in the specification. The sources section of a template.json file denotes what source code should be installed during the dotnet new command. In Giraffe's case this was very easy. If the giraffe option has been selected, then the source code shall come from the Giraffe.Template folder and the destination/target folder should be the root folder of where the dotnet new command is being executed from. The same logic applies to all the other options as well:

"sources": [
    {
      "source": "./Giraffe.Template/",
      "target": "./",
      "condition": "(ViewEngine == \"giraffe\")"
    },
    {
      "source": "./Razor.Template/",
      "target": "./",
      "condition": "(ViewEngine == \"razor\")"
    },
    {
      "source": "./DotLiquid.Template/",
      "target": "./",
      "condition": "(ViewEngine == \"dotliquid\")"
    }
  ]

With this in place I was able to create a new giraffe-template NuGet package and deploy everything to the official NuGet server again.

This is literally how easy it is to support distinct project templates from a single dotnet new template.

Different templates with same groupIdentifier

Another very similar, but in my opinion less elegant way would have been to create three different template.json files and use the groupIdentifier setting in connection with the tags array to support three different templates as part of one. Unfortunately this option doesn't seem to be very well supported from the .NET CLI. Even though it works, the .NET CLI doesn't display any useful error message when a user makes a mistake or when someone types dotnet new giraffe --help into the terminal. It also doesn't allow a default value to be set which made it less attractive overall. I would only recommend to go with this option if you need to provide different templates based on the selected .NET language, in which case it works really well again.

If you have any further questions or you would like to know more about the details of the Giraffe template then you can visit the giraffe-template GitHub repository for further reference.

This blog post is part of the F# Advent Calendar in English 2017 blog series which has been kindly organised by Sergey Tihon. Hope you all enjoyed this short tutorial and wish you a very Merry Christmas!

1) Templating engine documentation

Various documentation for the dotnet new templating engine can be found across the following resources:

Custom templates for dotnet new
dotnet-template-samples (GitHub repo with a lot of useful examples)
How to create your own templates for dotnet new (Great blog post)
Available templates for dotnet new (Community built templates)
dotnet tempalting engine (Official dotnet templating engine GitHub repository)

Evolving my open source project from a one man repository to an OSS organisation

Wed, 06 Dec 2017 00:00:00 +0000

Over the last couple of months I have been pretty absent from my every day life, such as keeping up with Twitter, reading and responding to emails, writing blog posts, working on my business and maintaining the Giraffe web framework. It was not exactly what I had planned for, but my wife and I decided to take a small break and wonder through South America for a bit before coming back for Christmas again. It was a truly amazing experience, but as much as travelling was fun, it was not great for my open source project Giraffe which was just about to pick up momentum after the huge exposure via Scott Hanselman's and the Microsoft F# team's blog posts (huge thanks btw, feeling super stoked and grateful for it!).

Initially I did not plan to slow down on Giraffe, but it turns out that trying to get a decent internet connection and a few hours of quality work in between of adventure seeking, 16 hour bus journeys, one flight every 3 days on average, hostels, hotels, hurricanes and multi day treks in deserts and mountains is not that easy after all :) (who thought ey?).

As a result issues and pull requests started to pile up quicker than I was able to deal with them and things got a bit stale. Luckily there were a few OSS contributors who did a fantastic job in picking up issues, replying to questions, fixing bugs and sending PRs for new feature requests when I was simply not able to do so - which meant that luckily things were moving at least at some capacity while I was away (a very special thanks to Gerard who has helped with the entire project beyond imagination and has been a huge contributor to Giraffe altogether; I think it's fair to say that Giraffe wouldn't be the same without his endless efforts).

However, even though the community stepped up during my absence I was still the only person who was able to approve PRs and get urgent bug fixes merged before pushing a new release to NuGet, and that understandably caused frustrations not only for users, but also for the very people who were kind enough to help maintaining it.

As the owner of the project and someone who really believes in the future of Giraffe it became very apparent that this was not acceptable going forward and things had to change if I really wanted Giraffe to further grow and succeed in the wider .NET eco system. So when the community asked me to add additional maintainers to the project I did not even hesitate for a second and decided that it was time to evolve the project from a one man repository to a proper OSS organisation which would allow more people having a bigger impact and control of the future of Giraffe.

An OSS project is only as good as its community

I created giraffe-fsharp on GitHub and moved the Giraffe repository from my personal account to its new home. Furthermore I have initially added three more contributors to the organisation who now all have the required permissions to maintain Giraffe without my every day involvement. This doesn't mean that I don't want to work on Giraffe any longer or that I want to work less on it, but it means that I just wanted to remove myself as the single point of failure, which is very important for a number of reasons.

First there is the obvious point that if I'd disappear out of the blue for whatever reason it would have a huge impact on anyone who has trusted in the project and its longevity. If I'd be a large company where changing tech can be very expensive then I'd personally not be able to justify the usage of a project which could literally drop dead over night. It is a real concern which I understand and therefore try to address this with the transition to a proper OSS organisation. It's a first step to mitigate this very real risk and a commitment from my side to do whatever I can to keep Giraffe well and alive.

Secondly I can simply not imagine that I as a single person could possibly grow the project better or faster than a larger collective of highly talented developers who are motivated to help me. I would be stupid to refuse this offer and it's in my personal interest to help them to help me.

Thirdly and most importantly I don't want to lose the fun and joy of working on Giraffe. I strongly believe that .NET Core is an excellent platform and a future proof technology. I also believe that functional programming is yet to have its big moment in the .NET world and with F# being the only functional first language I think the potential for Giraffe is probably bigger than I can think of today. I think it's only a matter of time before its usage will outgrow my own physical capability of maintaining it and the last thing I want to happen is to burn out, have an OSS fatigue or completely lose motivation for it. The only way to avoid this from happening is by accepting the help of others and delegating more responsibilities to other people who share the same vision and have an interest in the project's success.

Therefore it makes me proud to have met these people and being able to expand Giraffe into a more structured organisation which I believe is the right way of addressing all these issues effectively.

More people need more structure

Now that there's more people using Giraffe and more people helping to maintain it I think the next important step is to further expand on a work flow which aims at providing at a minimum the same quality at which I would have maintained the project myself.

As such I have set up the following process to maximise quality, reduce risk and give orgaisations of all size the confidence of trusting into Giraffe:

All development work has to happen on individual branches (enforced via GitHub)
The develop branch must be at a releasable state at all times
Only a finished bug fix or feature enhancement can be pushed via a pull request into develop (enforced via GitHub)
Each PR against develop must be formally reviewed by at least one other core maintainer (enforced via GitHub)
If enough PRs has flown into develop and the team wants to schedule a new release then a pull request can be made against master
Only an owner (currently me) has permissions to approve a PR on master and hence triggering a new automated release to NuGet (enforced via GitHub)

I know this might seem like a lot of process for a fairly new project, but I think it is very important to establish a good working structure early on to keep the quality high no matter how big the project will grow in the future. This process guarantees that at least 3 separate pair of eyes (two core maintainers and one owner) will review every single line of code before it makes into an official release. At the same time this process should hopefully allow a frictionless collaboration between core maintainers up until the point of an official release without the necessity of my involvement. Things like a vacation, illness or other forms of temporary unavailability should not be an issue any longer.

Never stop growing

All of this is only a first step of what I hope will be a very long future for Giraffe. Nothing is set in stone and if we discover better or more efficient ways of working together then things might change in the future and I will most certainly blog about it in a follow up post again.

One other thing which is perhaps worth noting is that I also plan to join the .NET Foundation or the F# Foundation in the near future. I'll talk more about this in a follow up blog post as well!

Can I join giraffe-fsharp?

Short answer is yes, but you have to have contributed to the project as an outside collaborator first and shown an interest and familiarity with the project before getting an invite to become a member of the organisation. I certainly don't see an upper limit of motivated people who want to help and any form of contribution is more than welcome. If you like Giraffe and want to participate in it then feel free to go and check out the code, discuss your ideas via GitHub issues and send PRs for approved feature requests which will definitely see your code (after reviews) merged into the repository.

Please also be aware that any member of giraffe-fsharp must have two factor authentication enabled on their account. I am big on security and I don't see why an OSS project should be treated any less serious than a project which would be run by a private company.

I hope this was a helpful insight into where I see the project going, what has happend lately and which steps I am taking to get Giraffe out of infancy and make it a serious contender among all other web frameworks in the wild (and not just .NET ;))!

Thanks to everyone who has helped, blogged, used or simply talked about Giraffe! There's no better feeling than knowing that people use and like something you've built and I now it's time that Giraffe becomes not only mine but a community driven project which we can grow together.

P.S.: If you ever have a chance to pack your stuff and travel the world then I'd highly recommend you to do so! It's one of the most amazing things one can do in life and probably for many people a one in a lifetime experience. Go and explore different places, meet people, learn about new cultures and experience life from a different perspective. It's eye opening and very educational on so many levels! Follow me on Instagram for inspiration ;)

Giraffe goes Beta

Sat, 12 Aug 2017 00:00:00 +0000

After 6 months of working on Giraffe, lots of improvements, great community contributions and running several private as well as commercial web applications in production I am really happy to announce that Giraffe is finally going Beta. This might not sound like a big milestone to you, but given that I was very hesitant on pre-maturely labelling the project anything beyond Alpha and the many breaking changes which we had in the past it certainly feels like a big achievement to me! After plenty of testing, tweaking, re-architecting and some real world exposure I think we finally reached a point where I can confidently say that the external facing API will remain fairly stable as of now - and seems to work really well too ;).

What has changed

XmlViewEngine

Since the last blog post I have made several improvements to the previously known functional HTML engine, which has been renamed to a more generic XmlViewEngine now. The new changes allow the XmlViewEngine to be used beyond simple HTML views and can be used for things like generating dynamic XML such as SVG images and more. Personally I think the XmlViewEngine is the most feature rich and powerful view engine you can find in any .NET framework today and I will certainly dedicate a whole separate blog post on that topic alone in the near future soon.

Continuations instead of bindings

One particular big change which we recently had was the move from binding HttpHandler functions to chaining HttpFunc continuations instead. The HttpHandler function has changed its signature from a HttpContext -> Async<HttpContext option> to a HttpFunc -> HttpFunc, whereas a HttpFunc is defined as HttpContext -> Task<HttpContext option>.

The main difference is that a HttpHandler function doesn't return Some HttpContext any longer (unless you want to immediately short circuit the chain) and is responsible for invoking the next continuation function from within itself. If you think this sounds very similar to ASP.NET Core's middleware then you are not mistaken. It is the same concept which brings several benefits such as better control flow and improved performance.

Even though this posed a fundamental change in architecture, we didn't have to compromise on how easy it is to compose a larger web application in Giraffe:

let webApp =
    choose [
        GET >=>
            choose [
                route "/"       >=> renderHtml indexPage
                route "/signup" >=> renderHtml signUpPage
                route "/login"  >=> renderHtml loginPage
            ]
        POST >=>
            choose [
                route "/signup" >=> signUpHandler
                route "/login"  >=> loginHandler
            ]
        setStatusCode 404 >=> text "Not Found" ]

All credit goes to Gerard who worked on this big change entirely from concept to implementation on his own.

Tasks

Another architectural change was that Giraffe works natively with Task and Task<'T> objects now. Previously you would have had to convert from a C# Task<'T> to an F# Async<'T> workflow and then back again to a Task<'T> before returning the flow to ASP.NET Core, but not any longer.

If you paid close attention in the previous example then you might have noticed that a HttpFunc is defined as HttpContext -> Task<HttpContext option>. Apart from the additional convenience of not having to convert between tasks and asyncs any more, this change netted us a two figure % perf improvement overall.

From now on you can reference the Giraffe.Tasks module and make use of the new task {} workflow which will allow you to write asynchronous code just as easily as it was with F# async {} before:

open Giraffe.Tasks
open Giraffe.HttpHandlers

let personHandler =
    fun (next : HttpFunc) (ctx : HttpContext) ->
        task {
            let! person = ctx.BindModel<Person>()
            return! json person next ctx
        }

The original code for Giraffe's task implementation has been taken from Robert Peele's TaskBuilder.fs and minimally modified to better fit Giraffe's use case for a highly scalable ASP.NET Core web application.

Again, all credit goes to Gerard and his endless efforts in improving Giraffe's overall architecture and performance.

What to expect next?

Stability

First of all, as Giraffe has officially entered the Beta phase you can expect a much more stable API with no to minimal breaking changes going forward.

More performance

Next there are still many ways of improving the internals of Giraffe.Tasks which we think will yield even further perf improvements. Additionally Gerard plans to implement an alternative trie routing API which should promise another perf gain on web applications with large routing layers as well.

More sample applications and templates

Another area where I would like to focus on more in the future is in providing more sample applications and templates which will help people to get up and running with Giraffe in as little time as possible.

Also I would like to blog more about my own usage of Giraffe and show case a few production applications with a closer look at some stats, deployments and general tips & tricks.

I hope you like the latest changes and are still as excited about Giraffe as I am or even considering to build your next F# web application with the fastest functional .NET web framework you'll find anywhere today ;).

If you already use Giraffe for a commercial or hobby project please let me know in the comments below and I can feature you in the official GitHub repository if you like.

Thanks for reading and stay tuned until next time!

Functional ASP.NET Core part 2 - Hello world from Giraffe

Sun, 14 May 2017 00:00:00 +0000

This is a follow up blog post on the functional ASP.NET Core article from about two months ago. First of all I'd like to say that this has been the longest period I haven't published anything new to my blog since I started blogging in early 2015. The reason is because I have been pretty busy with a private project which I hope to write more about in the future, but more importantly I have been extremely busy organising my own wedding which took place at the end of last month :). Yes!, I've been extremely lucky to have found the love of my life and best friend and after being engaged for almost a year and a half we've finally tied the knot two weeks ago. Normally I don't blog about my private life here, but since this has been such a significant moment in my life I thought I should mention a few words here as well and let everyone know that the quiet time has been for a good reason and will not last for much longer now.

While this has primarily occupied the majority of my time I was also quite happy to see my functional ASP.NET Core project receive recognition from the community and some really great support from other developers who've been helping me in adding lots of new functionality since then. In this first blog post after my small break I thought I'd take the opportunity and showcase some of the work we've done since the initial release and explain some of the design decisions behind some features.

But first I shall say that the framework has been renamed to Giraffe.

ASP.NET Core Lambda is now Giraffe

ASP.NET Core Lambda was a good name in terms of being very descriptive of what it stood for, but at the same time there were plenty of other issues which led me and other people believe that a different name would be a better fit.

Initially I named the project ASP.NET Core Lambda, because, at its core it was a functional framework built on top of (and tightly integrated with) ASP.NET Core, so I put one and one together and went with that name.

However, it quickly became apparent that "ASP.NET Core Lambda" wasn't a great name for the following reasons:

ASP.NET Core Lambda is a bit of a tongue twister.
"ASP", ".NET", "Core" and "Lambda" are extremely overloaded words with more than one meaning. If the project turns out to be successful then any type of search or information lookup (e.g StackOverflow) would be an absolute nightmare with this name.
Specifically Lambda is associated with Amazon's serverless cloud offering which would add even more to the confusion.
Finally the name is not very tasteful. Let's be honest, the mix of capitalized and pascal cased words, the additional whitespace and the dot in the word makes the name look very busy and simply doesn't resemble an elegant or tasteful product.

As a result I decided to rename the project to something different and put the name up for a vote, which ultimately led to Giraffe. Looking back I think it was a great choice and I would like to thank everyone who helped me in picking the new name, as well as suggesting other great names which made the decision not easy at all.

I think Giraffe is a much better name now, because it is short, it is very clear and distinctive and there is no ambiguity around the spelling or pronunciation. There is also no other product called Giraffe in the .NET space and not really anything else which it could be mistaken with. The name Giraffe also hasn't been taken as a NuGet package which made things really easy. On top of that Giraffe gave lots of creative room for creating a beautiful logo for which I used 99designs.co.uk. I set up a design challenge there and the winner impressed with this clever design:

Now I can only hope that the product will live up to this beautiful logo and the new name, which brings me to the actual topic of this blog post.

Overview of new features

There has been quite a few changes and new features since my last blog post and there's a few of which I am very excited about:

Dotnet new template

One really cool thing you can do with the new .NET tooling is to create project templates which can be installed via NuGet packages.

Thanks to David Sinclair you can install a Giraffe template by running the following command:

dotnet new -i giraffe-template::*

This will install the giraffe-template NuGet package to your local templates folder.

Afterwards you can start using Giraffe as a new project type when running the dotnet new command:

dotnet new giraffe

This feature makes it significantly easier to get started with Giraffe now. The quickest way to get a working Giraffe application up and running is by executing these three commands:

dotnet new giraffe
dotnet restore
dotnet run

Everything should compile successfully and you should see a Hello-World Giraffe app running behind http://localhost:5000.

Nested routing

Another cool feature which has been added by Stuart Lang is nested routing.

The new subRoute handler allows users to create nested routes which can be very useful when logically grouping certain paths.

An example would be when an API changes it's authentication scheme and you'd want to group routes together which implement the same type of authentication. With the help of nested routing you can enable certain features like a new authentication scheme by only declaring it once per group:

let app =
    subRoute "/api"
        (choose [
            subRoute "/v1"
                (oldAuthentication >=> choose [
                    route "/foo" >=> text "Foo 1"
                    route "/bar" >=> text "Bar 1" ])
            subRoute "/v2"
                (newAuthentication >=> choose [
                    route "/foo" >=> text "Foo 2"
                    route "/bar" >=> text "Bar 2" ]) ])

In this example a request to http://localhost:5000/api/v1/foo will use oldAuthentication and a request to http://localhost:5000/api/v2/foo will end up using newAuthentication.

There is also a subRouteCi handler which is the case insensitive equivalent of subRoute.

Razor views

Next is the support of Razor views in Giraffe. Nicolás Herrera developed the first version of Razor views by utilising the RazorLight engine. Shortly after that I realised that by referencing the Microsoft.AspNetCore.Mvc NuGet package I can easily re-use the original Razor engine in order to offer a more complete and original Razor experience in Giraffe as well. While under the hood the engine changed from RazorLight to ASP.NET Core MVC the functionality remained more or less the same as implemented by Nicolás in the first place.

In order to enable Razor views in Giraffe you have to register it's dependencies first:

type Startup() =
    member __.ConfigureServices (svc : IServiceCollection,
                                 env : IHostingEnvironment) =
        Path.Combine(env.ContentRootPath, "views")
        |> svc.AddRazorEngine
        |> ignore

After that you can use the razorView handler to return a Razor page from Giraffe:

let model = { WelcomeText = "Hello World" }

let app =
    choose [
        route "/" >=> razorView "text/html" "Index" model
    ]

The above example assumes that there is a /views folder in the project which contains an Index.cshtml file.

One of the parameters passed into the razorView handler is the mime type which should be returned by the handler. In this example it is set to text/html, but if the Razor page would represent something different (like an SVG image template for example) then with the razorView handler you can also set a different Content-Type as well.

In most cases text/html is probably the desired Content-Type of your response and therefore there is a second handler called razorHtmlView which does exactly that:

let model = { WelcomeText = "Hello World" }

let app =
    choose [
        route  "/" >=> razorHtmlView "Index" model
    ]

A more involved example with a layout page and a partial view can be found in the SampleApp project in the Giraffe samples repository.

Using DotNet Watcher to reload the project on Razor page changes

If you come from an ASP.NET Core MVC background then you might be used to having Razor pages automatically re-compile on every page change during development, without having to manually restart an application. In Giraffe you can achieve the same experience by adding the DotNet.Watcher.Tools to your .fsproj and put a watch on all .cshtml files:

<ItemGroup>
    <DotNetCliToolReference Include="Microsoft.DotNet.Watcher.Tools" Version="1.0.0" />
</ItemGroup>

<ItemGroup>
    <Watch Include="**\*.cshtml" Exclude="bin\**\*" />
</ItemGroup>

By adding the watcher to your project file you can start making changes to any .cshtml file in your project and immediately see the changes take effect during a running Giraffe web application (without having to manually restart the app).

Dependency on Microsoft.AspNetCore.Mvc

One other thing which might sound a little bit strange is the dependency on the Microsoft.AspNetCore.Mvc NuGet package. It is essentially the full MVC library being referenced by Giraffe now and it has sparked a bit of confusion or disappointment amongst some users. Personally I think it really doesn't matter and I wanted to explain my thinking behind this design decision.

In order to get Razor views working in Giraffe there were three options available:

Implement Giraffe's own Razor engine
Use someone else's custom Razor engine
Use the original Razor engine

I certainly did not have an appetite for the first option, which is hopefully understandable, and therefore was left with the choice between the latter two.

At the time of writing there was only one .NET Core compatible custom Razor engine available, which is RazorLight. From what I know RazorLight is a very nice library and definitely highly recommended, but not necessarily the right choice for Giraffe.

When you ignore the name of the NuGet package for a second then there is really not much difference between referencing RazorLight or Microsoft.AspNetCore.Mvc in Giraffe. Both require a new NuGet dependency in the project and both are a library which exposes some functionality to render Razor views. The ASP.NET Core MVC package might be slightly bigger and offer more functionality than what Giraffe actually needs, but that doesn't really matter, because Giraffe ignores the rest and only uses what is needed for the Razor support. I think it is pretty normal that any given library often implements far more functionality than what a single project actually makes use of.

In the case of Giraffe I was faced with a trade-off between a dependency which uses slightly more KBs disk space, but in return offers a complete and original Razor experience vs. a slightly smaller library which offers a custom implementation of Razor pages.

As far as I see this issue there is absolutely no disadvantage in Giraffe using the MVC NuGet package in order to get the original Razor experience in comparison to using any other Razor library. I also believe that this option is more in line with Giraffe's goal to be tightly integrated with the original ASP.NET Core experience. Users benefit by getting the original, well documented and understood Razor features which makes portability of existing Razor views also significantly easier.

For me it's really about making smart choices and I truly believe that the strength of Giraffe is by standing on the shoulders of giants, which is ASP.NET Core MVC in this case.

Functional HTML view engine

Speaking of the Razor view engine, another really cool feature which has been added to Giraffe is a new programmatic way of creating views. Florian Verdonck helped me a lot with Giraffe over the last few weeks and one of his contributions was to port Suave's experimental Html engine to Giraffe.

I think the best way to describe the new Giraffe.HtmlEngine is by showing some code:

open Giraffe.HtmlEngine

let model = { Name = "John Doe" }

let layout (content: HtmlNode list) =
    html [] [
        head [] [
            title [] (encodedText "Giraffe")
        ]
        body [] content
    ]

let partial () =
    p [] (encodedText "Some partial text.")

let personView model =
    [
        div [] [
                h3 [] (sprintf "Hello, %s" model.Name |> encodedText)
            ]
        div [] [partial()]
    ] |> layout

let app =
    choose [
        route "/" >=> (personView model |> renderHtml)
    ]

This examples demonstrates well how easy it is to create complex views with features like layout pages, partial views and model binding. There's really nothing that can't be done with this programmatic way of defining view-model pages and if you think there's something missing then it is really easy to extend as well.

Kudos to the Suave guys for coming up with this brilliant view engine and thanks to Florian for suggesting this feature as well as liaising with the Suave guys and porting the code to Giraffe!

Content negotiation

The next feature which I'd like to show off is something which I did myself for a change. When exposing a web service endpoint you often want to respect the client's requested response type which is typically communicated via the HTTP Accept header.

For example a client might send the following information with a HTTP Accept header:

Accept: application/xml,application/json,text/html;q=0.8,text/plain;q=0.9,*/*;q=0.5

In this example the client says the following:

Please give me either an XML or JSON response, both is my most preferred choice
If you don't speak XML or JSON, then I'd like plain text as the next best option
If you don't speak plain text either, then please just send the response in HTML
If that doesn't suit you either, then just give me whatever you have

In Giraffe you can use the negotiate and negotiateWith handlers to return the client the best matching response based on the information passed through the request's Accept header.

The negotiate handler is very simple and speaks JSON, XML and plain text at the moment:

[<CLIMutable>]
type Person =
    {
        FirstName : string
        LastName  : string
    }
    override this.ToString() =
        sprintf "%s %s" this.FirstName this.LastNam

let app =
    choose [
        route  "/foo" >=> negotiate { FirstName = "Foo"; LastName = "Bar" }
    ]

By default the negotiate handler will check the request's Accept header and automatically serialize the model in either JSON, XML or plain text. If the client asks for plain/text then the negotiate handler will use the model's ToString() method otherwise it will use a JSON or XML serializer. Other mime types like text/html are not supported out of the box, because there is no default way to serialize an object into HTML.

However, if you want to support a wider range of accepted mime types then you can use the negotiateWith handler to set custom negotiation rules.

Let's assume you want to support two additional mime types, application/x-protobuf for Google's Protocol Buffers serialization and application/octet-stream for generic binary serialization.

First you would want to implement two new HttpHandler functions which can return a response of those exact types:

let serializeProtobuf x =
    // Implement protobuf serialization

let serializeBinary x =
    // Implement binary serialization

let protobuf (dataObj : obj) =
    setHttpHeader "Content-Type" "application/x-protobuf"
    >=> setBodyAsString (serializeProtobuf dataObj)

let binary (dataObj : obj) =
    setHttpHeader "Content-Type" "application/octet-stream"
    >=> setBodyAsString (serializeBinary dataObj)

Then you can use the two new HttpHandler functions to set up custom negotiation rules and use them with the negotiateWith handler:

let rules =
    dict [
        "*/*"                     , json
        "application/json"        , json
        "application/xml"         , xml
        "application/x-protobuf"  , protobuf
        "application/octet-stream", binary
    ]

let model = { FirstName = "Foo"; LastName = "Bar" }

let app =
    choose [
        route  "/foo" >=> negotiateWith rules model
    ]

You might find it more convenient to create a new negotiate handler altogether, which will make it much less verbose to use the custom rules in subsequent routes:

let negotiate2 = negotiateWith rules

let app =
    choose [
        route  "/foo" >=> negotiate2 { FirstName = "Foo"; LastName = "Bar" }
    ]

Even though there's still loads of room for improvement, I think this might be just enough for a large quantity of web applications already.

Model Binding

While the Accept HTTP header denotes what mime types a client understands (typically more than just one), the Content-Type HTTP header specifies which mime type a client/server has chosen to send a message with. This is very useful information when it comes to model binding.

Giraffe exposes five different model binding functions which can deserialize the content of a HTTP request into a strongly typed object. Four of them can bind a specific request type into a typed model and the fifth method picks the most appropriate model binding function based on the request's Content-Type header.

It's the easiest to demonstrate this with a quick example again. Let's assume we have the following record type in a web application:

[<CLIMutable>]
type Car =
    {
        Name   : string
        Make   : string
        Wheels : int
        Built  : DateTime
    }

Now I'd like to expose different endpoints which can be used to HTTP POST a car object to the web service:

open Giraffe.HttpHandlers
open Giraffe.HttpContextExtensions

let submitAsJson =
    fun (ctx : HttpContext) ->
        async {
            let! car = ctx.BindJson<Car>()
            // Do stuff
        }

let submitAsXml =
    fun (ctx : HttpContext) ->
        async {
            let! car = ctx.BindXml<Car>()
            // Do stuff
        }

let submitAsForm =
    fun (ctx : HttpContext) ->
        async {
            let! car = ctx.BindForm<Car>()
            // Do stuff
        }

let submitAsQueryString =
    fun (ctx : HttpContext) ->
        async {
            let! car = ctx.BindQueryString<Car>()
            // Do stuff
        }

let submitHowYouLike =
    fun (ctx : HttpContext) ->
        async {
            let! car = ctx.BindModel<Car>()
            // Do stuff
        }

let webApp =
    POST >=>
        choose [
            route "/json"  >=> submitAsJson
            route "/xml"   >=> submitAsXml
            route "/form"  >=> submitAsForm
            route "/query" >=> submitAsQueryString
            route "/any"   >=> submitHowYouLike ]

As you can see from the example, the model binding functions are extension methods on the HttpContext object and require to open the Giraffe.HttpContextExtensions module.

The ctx.BindJson<'T>() function will always try to retrieve an object by deserializing data from JSON. The ctx.BindXml<'T>() function behaves the same way but will try to deserialize from XML. The ctx.BindForm<'T>() function will bind a model from a request which has a Content-Type of application/x-www-form-urlencoded (typically a POST request from a HTML form element).

Sometimes you might want to bind a model from a query string, which could not only come from a HTTP POST but also from any other HTTP verb request. In this instance the ctx.BindQueryString<'T>() function can be used to bind the values from a query string to a strongly typed model.

At last you might want to allow a client to submit an object via any of the above mentioned options on the same endpoint. In this case your endpoint has to pick the correct model binding based on the Content-Type HTTP header and this can be achieved with the ctx.BindModel<'T>() function.

Since all model binding functions are extension methods of the HttpContext type they can be used from anywhere in a web application where you have access to the HttpContext object, which in Giraffe's case is every single HttpHandler function.

What's next?

There were quite a few breaking changes since the first release, but APIs are slowly maturing as I get more feedback and exposure of the framework. So far the library has been in an alpha stage and will probably remain for another few weeks before I get around to finish some more examples and test projects which will eventually lead to the beta phase.

Once the project is in beta I will try to focus my effort more on collecting a lot of additional feedback before I feel confident enough to declare the first RC and subsequently the official version 1.0.0.

Even though breaking changes are not always the end of the world I would like to avoid drastic fundamental changes (as seen recently) once the project has entered the first stable release. Therefore I have been fairly reluctant to prematurely label Giraffe beyond an alpha and will probably want to enjoy the freedom of breaking stuff for a tiny bit longer. At the end of the day it's about setting the right expectations and I don't help anyone by labeling v1.0.0 too early when I know there's still a fair bit of danger to potentially move stuff around.

However, having said that I do want to stress that the underlying system (ASP.NET Core and Kestrel) have been very stable for a while now and as long as you don't mind that a namespace or method might still change in the near future then Giraffe is absolutely fit for production. So please go ahead and give it a try if you like what you've seen in this blog post so far :).

This basically brings me to the end of this follow up article and I thought what better way to finish it off than by sharing some of our memories from our wonderful wedding (in case you ever wondered what a British-Indian/Austrian-Polish wedding looks like ;)).

It was a very long day, which started off with a civil ceremony in the morning...

Followed by a traditional Hindu ceremony shortly after lunch...

I had no idea how much fun Hindu ceremonies can be! There's lots of really fun and merry traditions which take place as part of us getting married. Then there's also a bit of banter between the two families. One of those little traditions is that the bride's family has to scyan the groom's shoes before the ceremony ends so that the groom can't leave the house and take his newly wedded wife away from her family - at least not without having to pay for getting his shoes back. Normally this results in a bit of shoe pulling between the bride's side and the groomsmen, but I think in our case it is fair to say that there was a bit of a cultural clash when someone from my family rugby tackled a guy who tried to sneak away with my shoes, lol...

Luckily nothing serious happened and after everyone had a great laugh we continued with the ceremony...

Until finally we were able to celebrate at the reception party in the evening...

Throughout the day our guests wrote us lovely (I think) messages on little papers and my family decided to throw all these messages into a wooden box with a nice bottle of Red, which we had to seal ourselves with nails and hammer. We are not allowed to open this box until in seven years time and then we can enjoy a nicely matured bottle of vino while reading all those wonderful memories from our big day. What a brilliant idea!

We had a fantastic day and before everyone stormed to the dance floor there was even a pretty impressive surprise firework display...

Getting married was a lot of fun and it all worked out so much better than we could have hoped for :)

Now we are ready for new adventures...

Functional ASP.NET Core

Tue, 07 Feb 2017 00:00:00 +0000

In December 2016 I participated in the F# Advent Calendar where I wrote a blog post on running Suave in ASP.NET Core. As part of that blog post I introduced the Suave.AspNetCore NuGet package which makes it possible to run a Suave web application inside ASP.NET Core via a new middleware.

So far this has been pretty good and as of last week the GitHub repository has been moved to the official SuaveIO GitHub organisation as well. A while ago someone even tweeted me that the performance has been pretty good too:

Even though this made me very happy there was still one thing that bugged me until today.

Why I created Suave.AspNetCore

My main motivation for running Suave inside ASP.NET Core was to benefit from the speed and power of Kestrel while still being able to build a web application in a functional approach. Suave.AspNetCore made this possible, but afterwards I realised that this was not my final goal yet.

Ultimately I would like to build a web application in ASP.NET Core with a functional framework which does not only benefit from Kestrel but also from the entire ASP.NET Core eco system, including other middleware such as static files, authentication, authorization, security, the flexibility of the config system, logging or simply being able to retrieve information from the current hosting environment and more.

There is a lot of great features in ASP.NET Core which have been carefully crafted by experts (e.g. security) and I wouldn't want to miss out on those based on the framework of my choice. Unfortunately with Suave and Suave.AspNetCore I am limited in what other middleware I can use in combination with a Suave ASP.NET Core web application.

Suave and ASP.NET Core - Buddies but not family

Suave doesn't naturally fit into ASP.NET Core the way MVC does.

The reason is because Suave is not only a web framework but more of a web platform similar to what ASP.NET Core is itself. It's probably never been intended to be integrated with ASP.NET Core in the first place.

Think of it like this, ASP.NET Core is a web platform which sets the ground work for building any web application and MVC is a framework on top of the platform, which enables building web applications with an object oriented Model-View-Controller design pattern. Ideally as an F# developer I would like to replace the object oriented MVC framework with a functional equivalent, but keep the rest of ASP.NET Core's offering at the same time. This is very difficult with Suave at the moment.

Suave has its own HTTP server, its own Socket server and its own HTTP abstractions. As a result Suave's Socket implementation is not compatible with ASP.NET Core's web socket server and Suave's HttpContext is vastly different from ASP.NET Core's HttpContext.

This is why the Suave.AspNetCore middleware has to translate one HttpContext into another and then back again. It works, but it is not ideal, because there's a lot of information that gets lost along the way (e.g. the User object in ASP.NET Core is of type IPrincipal and in Suave it is a Map<string, obj>). This is a limiting factor. For example there's no way to access the Authentication property, the Session object or the Features collection of the original ASP.NET Core HttpContext from inside a Suave application.

This means that even though I can run a Suave web application in ASP.NET Core, I still have to re-build a lot of the ground work that has been laid out for me by other ASP.NET Core middleware. In some cases this might be a minor problem, but in others I consider it a big issue. Especially when it comes to critical components such as security I would much rather want to rely on the implementation provided by Microsoft and other industry experts than (re-)inventing it myself.

None of this is Suave's fault though, because it has not been designed with ASP.NET Core in mind, which is more than fair, but for people like me who want to benefit from both platforms this is still an important issue to consider.

Why ASP.NET Core?

If Suave is already a well working standalone product for building web applications in a functional way, why would I even bother with ASP.NET Core? Well this is a good question and I can only answer it for myself.

For me the main reasons for using ASP.NET Core are the following:

Performance

Kestrel is extremely fast and the Microsoft team is working hard on making it even faster. Depending on what type of application you are trying to build this can be a big or a small selling point.

Personally I am working on a project which anticipates a significant load at most times and therefore performance is key. I also have a few smaller side projects which run in Docker containers in the cloud and the quicker these applications can handle web requests, the less I have to pay for additional computation power.

Security

As mentioned before I am very reluctant to using 3rd party security code which hasn't been vetted, audited and tested to the same depth as the one provided by Microsoft and the ASP.NET Core team. They have years of invaluable experience and I trust them with security more than other individuals who I barely know or even myself. Call me pessimistic, but security is such a complex topic that I don't even think that a single person should do this on their own. It's one of those things where I believe you have to have a big team of experts and resources behind you to stand a chance against the various vulnerabilities of today.

Laziness

I am a lazy developer. I don't enjoy building the 100th logging framework or coming up with yet another configuration API. I want to build new original ideas and not waste my time on stuff which has been done by thousands of other developers before me. ASP.NET Core is a platform which offers many things out of the box and essentially saves me a lot of valuable time.

Additionally it offers easy integration points for other 3rd party code and has a huge developer base behind it which gives me access to even more useful tools and libraries which can benefit my projects.

Community

The community around ASP.NET (Core) is probably the biggest of all .NET web frameworks. There is significantly more people reporting issues, working on bug fixes and building new features into the product than anyone else. The value of such a vibrant community shall not be underestimated. There's nothing better than having bugs fixed by other people before I even encounter them myself.

Another great side effect is that a lot of my questions might have already been answered on StackOverflow and that there is a ton of other useful blog posts explaining stuff that I don't have to work out myself. As an employer it might be also beneficial to pick a framework which has a bigger talent pool than others.

Impatience

I am very impatient and for me it is very important that certain issues get addressed fairly quickly. For instance in recent years many web servers were upgrading to support HTTP/2 which is an important improvement over HTTP/1.1. My chances of getting such critical updates are probably higher with a product which is used by thousands of developers than something else which is maybe a little bit more niche. This is not always true, but from my experience this is generally the case.

Fear of missing out

We don't know what the future holds for us. Tomorrow someone might release something incredibly awesome which might not be available on smaller platforms in its initial phase. I love working with bleeding edge technology and as such I do have a certain degree of FOMO when it comes to software innovations.

Support

Lastly ASP.NET Core, even though open source, remains an enterprise supported product and that has a lot of value as well. Over the years I had to replace many successful open source packages because the original maintainers got tired of supporting it and no one else stepped in to replace them, which meant they became stale and essentially completely out of date. There is no guarantee that Microsoft will support the ASP.NET (Core) platform for ever, but from the looks of it I don't expect it to go away any time soon either. In fact the current signs seem to suggest the exact opposite considering that ASP.NET itself has already more than a decade on its shoulders and Microsoft recently decided to invest in a complete new re-design to continue its success.

Not everyone might agree with my reasoning, but for me these are very compelling points to stick with ASP.NET Core as my preferred .NET web platform and try to use as much of its ready made features as possible.

Building a functional framework for ASP.NET Core

Now that I have explained why I would like to build a web application with ASP.NET Core I have the only problem that as an F# developer there's no ideal framework available yet.

This got me thinking what if I could build my own little micro framework in F# that borrows the functional design pattern of Suave, but embraces the power of ASP.NET Core?

Defining a functional HttpHandler

A functional ASP.NET Core web application could look as simple as a function which takes in a HttpContext and returns a HttpContext. In functional programming everything is a function after all. Inside that function it would have full access to the Request and Response object as well as all the other objects to successfully process an incoming web request and return a response.

I could call such a web function a HttpHandler:

type HttpHandler = HttpContext -> HttpContext

Not every incoming web request can or should be handled by a HttpHandler. For example if the request was made to a route which wasn't anticipated, like a static file which should be picked up by the static file middleware, then a HttpHandler should be able to skip this particular request.

In this case there should be an option to return nothing so that another HttpHandler can try to satisfy the incoming request or the calling middleware can defer the HttpContext to the next middleware:

type HttpHandler = HttpContext -> HttpContext option

By making the HttpContext optional the function can now either return Some HttpContext or None.

Additionally the HttpHandler shouldn't block on IO operations or other long running tasks and therefore return the HttpContext option wrapped in an asynchronous workflow:

type HttpHandler = HttpContext -> Async<HttpContext option>

This is slowly taking shape, but there's still something missing.

In ASP.NET Core MVC controller dependencies are automatically resolved during instantiation. This is very useful, because in ASP.NET Core dependencies are registered in a central place inside the ConfigureServices method of the Startup.cs class file.

Automatic dependency resolution is not really a thing in functional programming, because dependencies are normally functions and not objects. Functions can be passed around or partially applied which usually makes object oriented dependency management obsolete.

However, because most ASP.NET Core dependencies are registered as objects inside an IServiceCollection container, a HttpHandler can resolve these dependencies through an IServiceProvider object.

This is done by wrapping the original HttpContext and an IServiceProvider object inside a new type called HttpHandlerContext:

type HttpHandlerContext =
    {
        HttpContext : HttpContext
        Services    : IServiceProvider
    }

By changing the HttpHandler function definition we can take advantage of this new type:

type HttpHandler = HttpHandlerContext -> Async<HttpHandlerContext option>

With that one should be able to build pretty much any web application of desire. If you have worked with Suave in the past then this should look extremely familiar as well.

Combining smaller HttpHandlers to bigger applications

In principal there's nothing you cannot do with an HttpHandler, but it wouldn't be very practical to build a whole web application in one function. The beauty of functional programming is the composition of many smaller functions to one bigger application.

The simplest combinator would be a bind function which takes two HttpHandler functions and combines them to one:

let bind (handler : HttpHandler) (handler2 : HttpHandler) =
    fun (ctx : HttpHandlerContext) ->
        async {
            let! result = handler ctx
            match result with
            | None      -> return None
            | Some ctx2 -> return Some ctx2
        }

As you can see the bind function takes two different HttpHandler functions and a HttpHandlerContext. First it evaluates the first handler and checks its result. If the result was None then it will stop at this point and return None as the final result. If the result was Some HttpHandlerContext then it will take the resulting context and use it to evaluate the second HttpHandler. Whatever the second HttpHandler returns will be the final result in this case.

This pattern is often referred to as railway oriented programming. If you are interested to learn more about it then please check out Scott Wlaschin's slides and video on his website or this lengthy blog post on that topic.

One more thing that should be considered in the bind function is to check if a HttpResponse has been already written by the first HttpHandler before invoking the second HttpHandler. This is required to prevent a potential exception when a HttpHandler tries to make changes to the HttpResponse after another HttpHandler has already written to the response. In this case the bind function should not invoke the second HttpHandler:

let bind (handler : HttpHandler) (handler2 : HttpHandler) =
    fun (ctx : HttpHandlerContext) ->
        async {
            let! result = handler ctx
            match result with
            | None      -> return None
            | Some ctx2 ->
                match ctx2.HttpContext.Response.HasStarted with
                | true  -> return  Some ctx2
                | false -> return! handler2 ctx2
        }

To round this up we can bind the bind function to the >>= operator:

let (>>=) = bind

With the bind function we can combine unlimited HttpHandler functions to one.

The flow would look something like this:

Another very useful combinator which can be borrowed from Suave is the choose function. The choose function let's you define a list of multiple HttpHandler functions which will be iterated one by one until the first HttpHandler returns Some HttpHandlerContext:

let rec choose (handlers : HttpHandler list) =
    fun (ctx : HttpHandlerContext) ->
        async {
            match handlers with
            | []                -> return None
            | handler :: tail   ->
                let! result = handler ctx
                match result with
                | Some c    -> return Some c
                | None      -> return! choose tail ctx
        }

In order to better see the usefulness of this combinator it's best to look at an actual example.

Let's define a few simple HttpHandler functions first:

let httpVerb (verb : string) =
    fun (ctx : HttpHandlerContext) ->
        if ctx.HttpContext.Request.Method.Equals verb
        then Some ctx
        else None
        |> async.Return

let GET     = httpVerb "GET"    : HttpHandler
let POST    = httpVerb "POST"   : HttpHandler

let route (path : string) =
    fun (ctx : HttpHandlerContext) ->
        if ctx.HttpContext.Request.Path.ToString().Equals path
        then Some ctx
        else None
        |> async.Return

let setBody (bytes : byte array) =
    fun (ctx : HttpHandlerContext) ->
        async {
            ctx.HttpContext.Response.Headers.["Content-Length"] <- new StringValues(bytes.Length.ToString())
            ctx.HttpContext.Response.Body.WriteAsync(bytes, 0, bytes.Length)
            |> Async.AwaitTask
            |> ignore
            return Some ctx
        }

let setBodyAsString (str : string) =
    Encoding.UTF8.GetBytes str
    |> setBody

This already gives a good illustration of how easily one can create different HttpHandler functions to do various things. For instance the httpVerb handler checks if the incoming request matches a given HTTP verb. If yes it will proceed with the next HttpHandler, otherwise it will return None. The two functions GET and POST re-purpose httpVerb to specifically check for a GET or POST request.

The route function compares the request path with a given string and either proceeds or returns None again. Both, setBody and setBodyAsString write a given payload to the response of the HttpContext. This will trigger a response being made back to the client.

Each HttpHandler is kept very short and has a single responsibility. Through the bind and choose combinators we can combine many HttpHandler functions into one larger web application:

let webApp =
    choose [
        GET >>=
            choose [
                route "/"     >>= setBodyAsString "Index"
                route "/ping" >>= setBodyAsString "pong"
            ]
        POST >>=
            choose [
                route "/submit" >>= setBodyAsString "Submitted!"
                route "/upload" >>= setBodyAsString "Uploaded!"
            ]
    ]

Even though I've barely written any code this functional framework already proves to be quite powerful.

At last I need to create a new middleware which can run this functional web application:

type HttpHandlerMiddleware (next     : RequestDelegate,
                            handler  : HttpHandler,
                            services : IServiceProvider) =

    do if isNull next then raise (ArgumentNullException("next"))

    member __.Invoke (ctx : HttpContext) =
        async {
            let httpHandlerContext =
                {
                    HttpContext = ctx
                    Services    = services
                }
            let! result = handler httpHandlerContext
            if (result.IsNone) then
                return!
                    next.Invoke ctx
                    |> Async.AwaitTask
        } |> Async.StartAsTask

And finally hook it up in Startup.cs:

type Startup() =
    member __.Configure (app : IApplicationBuilder) =
        app.UseMiddleware<LambdaMiddleware>(webApp) |> ignore

This web framework shows what I love about F# so much. With very little code I was able quickly write a basic web application from the ground up. It looks very much like an ASP.NET Core clone of Suave, but with the difference that it fully embraces the ASP.NET Core architecture and its HttpContext.

Functional ASP.NET Core framework

By extending the above example with a few more useful HttpHandler functions to return JSON, XML, HTML or even templated (DotLiquid) views someone could create a very powerful ASP.NET Core functional web framework, which could be easily seen as an MVC replacement for F# developers.

This is exactly what I did and I named it ASP.NET Core Lambda, because I simply couldn't think of a more descriptive or "cooler" name. It is a functional ASP.NET Core micro framework and I've built it primarily for my own use. It is still very early days and in alpha testing, but I already use it for two of my private projects and it works like a charm.

How does it compare to other .NET web frameworks

Now you might ask yourself how does this compare to other .NET web frameworks and particularly to Suave (since I've borrowed a lot of ideas from Suave and from Scott Wlaschin's blog)?

I think this table explains it very well:

	Paradigm	Language	Hosting	Frameworks
MVC	Object oriented	C#	ASP.NET (Core) only	Full .NET, .NET Core
NancyFx	Object oriented	C#	Self-hosted or ASP.NET (Core)	Full .NET, .NET Core, Mono
Suave	Functional	F#	Primarily self-hosted	Full .NET, .NET Core, Mono
Lambda	Functional	F#	ASP.NET Core only	.NET Core

MVC and NancyFx are both heavily object-oriented frameworks mainly targeting a C# audience. MVC probably the most feature rich framework and NancyFx with its super-duper-happy-path are probably the most wide spread .NET web frameworks. NancyFx is also very popular with .NET developers who want to run a .NET web application self-hosted on Linux (this was a big selling point pre .NET Core times).

Suave was the first functional web framework built for F# developers. It is a completely independent standalone product which was designed to be cross platform compatible (via Mono) and self-hosted. A large part of the Suave library is compounded of its own HTTP server and Socket server implementation. Unlike NancyFx it is primarily meant to be self-hosted, which is why the separation between web server and web framework is not as clean cut as in NancyFx (e.g. the Suave NuGet library contains everything in one while NancyFx has separate packages for different hosting options).

ASP.NET Core Lambda is the smallest of all frameworks (and is meant to stay this way). It is also a functional web framework built for F# developers, but cannot exist outside of the ASP.NET Core platform. It has been tightly built around ASP.NET Core to leverage its features as much as possible. As a result it is currently the only (native) functional web framework which is a first class citizen in ASP.NET Core.

I think it has its own little niche market where it doesn't really compete with any of the other web frameworks. It is basically aimed at F# developers who want to use a ASP.NET Core in a functional way.

While all these web frameworks share some similarities they still have their own appliances and target a different set of developers.

Watch this space for more updates on ASP.NET Core Lambda in the future and feel free to try it out and let me know how it goes! So far I've been very happy with the results and it has become my goto framework for new web development projects.

Thank you Microsoft for being awesome

Tue, 24 Jan 2017 00:00:00 +0000

OK so I have to admit that I can have a pretty big mouth sometimes, which is certainly not a good quality to have. I don't shy away to be (overly) critical if things annoy me:

As you can see, about a year ago I was a little bit upset that a great new product received a not so great new name. I really hoped that ASP.NET Core would have been named something much "cooler" for my taste.

So my frustration continued...

(Let's be honest, it is probably a good thing that I don't have many Twitter followers.)

And even a year later I feel like I am still suffering sometimes:

But I don't care about the name anymore.

Because the product is actually really good.

Yes, the name is still an issue when you google for stuff, and yes, the tooling is still in preview, and yes, the support for F# is not even in RC yet, and yes, there's still lots of other little annoyances which would be nice to have fixed soon, but when I look past all that for a moment then I have to admit that the product is actually really freaking nice.

Being harsh is easy

Often it is very easy to be very quickly critical, but much harder to acknowledge someone's good work. It's already an effort to think about praising people who did a great job at work, let alone praising some third party who you are not even directly related to.

But I think as someone who depends a lot on these third parties, corporations like Microsoft, who own a lot of the tech which I love and use every day, and other contributors from the open source community, then it's important to give back to these people as well.

Particularly with companies like Microsoft many people feel entitled to be overly rude or critical if things don't work out immediately the way they want it to be (and not just Microsoft but really any company of similar size). It is so easy to forget good manners when one is thinking of a big company logo rather than actual human beings, because it is much easier to throw dirt at a logo than a person.

But behind these logos there is still human beings working hard on the products which we use every day and they are not any different from us. Women and men who got into software development and IT for the same good reasons as we did, who share the same passion as we do and who make the same mistakes as us, even when they have only the best in their intention.

Microsoft has made a lot of mistakes in the past (Vista, Windows 8, Silverlight, Windows Phone, Skype, you name it!) and we've not come short to let them know how much these products sucked, but in most recent years they've actually done an amazing turnaround and even taken the lead in some of the most exciting innovations of our field and I am certainly not the only one who has noticed it.

Awesome things by Microsoft

Here's a list of a some of the many awesome things that happened at Microsoft in recent years (in no particular order):

This is a lot of (radical) positive change in a comparatively short amount of time when you think about the size of the company!

For example, check out the Surface Studio introduction video which demonstrates 3D drawing on a Surface Studio:

You have to admit that this is awesome. I mean I am not even a designer and I get a watery mouth by looking at it.

To be honest the whole Microsoft Surface product series is awesome and I speak from personal experience. Last Black Friday I bought my partner a new Surface Pro 4 (I accidentally spilled a drink over her old laptop) and boy was I impressed with her new device. It's slick, it's fast and when I tried out the drawing with the pencil then my jaw dropped and I was left with pure jealousy (and I say that even though I have a Lenovo Yoga 900 which is a pretty neat device itself).

But that is not even why I am so pleased with Microsoft in recent days. As a .NET developer I am mostly impressed by the work the teams have put into .NET Core, ASP.NET Core, C#, F# and the tooling around it.

.NET Core, ASP.NET Core, F# and Open Source

Especially the ASP.NET Core team led by Scott Hanselman and Damian Edwards is an excellent example of how well Microsoft functions today. Don't get me wrong, I know that ASP.NET Core is still far away from being perfect and that there's still loads of stuff that needs to be done before everyone can use it, but I am still amazed by how far this team has come today.

They have taken a 10+ year old technology, completely revamped it and transitioned from an entirely closed and proprietary product team to one of the most approachable open source contributors that I have ever seen. Not only is their code on GitHub, but they are putting an immense amount of time and effort into keeping a close relationship to the .NET community. They are active on GitHub, Slack, Gitter, Twitter and heck they even live stream a weekly standup where they demo the latest stuff hot from the press, answer live questions to the community, give updates on the current development status and talk about the roadmap. They even spend time going through blog posts written by non-Microsofties where Jon Galloway presents a whole list of community written content at the beginning of each standup. This has become a nice ritual where the Microsoft team gives back to other open source contributors and help nourish a friendly and healthy community.

I find this is amazing work. I don't know many other teams in the world who put this much effort into their open source work.

This is not Microsoft trying to put some code on GitHub and then turning on the #OpenSource marketing machinery. No, this is Microsoft showing the rest of the world how open source is done right (and I am not trying to undermine the open source work by others here, just trying to highlight how well it's done at MSFT).

Also I'd like to point out that this team is doing the difficult balancing act of developing a completely new technology which is supposed to entertain an already existing (and very spoiled) community as well as trying to attract new programmers from the open source field. I guess this is much harder than just launching a new product into the market like Go, Rust or Node.js which didn't have an existing developer base to please.

FSharp

I have been a C# developer for almost the entirety of my professional career and only in the last year I really fell in love with functional programming and F#. For me F# is probably one of the most powerful languages on the market right now. It has the horse power of .NET under the hood with the beauty of a modern functional language on the surface and it has been open source, cross platform compatible and community driven from the very beginning of its time.

So with all of that I've got something to say which was overdue for a while now:

Thank you Microsoft for being awesome!

Seriously, a special thanks to all of the Microsoft teams who work on .NET Core, ASP.NET Core, Visual Studio, Visual Studio Code, C#, F# and the entire goodness around it!!! Your products are awesome and because of your great work I enjoy working in the .NET eco-system more than ever before!

This is not me being sarcastic, I am dead serious, so thank you guys!

I hope when you read this then you know that despite all the negativity that Microsoft has to suck up every now and then that there's also a lot of developers who think you are doing an amazing job and that without you we would be probably some poor suckers who would have to work with Node.js or some other soul destroying technology!

Join me in saying thank you

If you are a .NET developer who loves working with C# or F#, who thinks that Visual Studio (Code) is one of the best IDEs, who thinks that ASP.NET Core and .NET Core are awesome new development stacks then please join me in saying thank you to these guys so that they know that all of their hard work is paying off.

Oh and before someone labels me a Microsoft fanboy or thinks that there's a hidden agenda behind this blog post then let me tell you this:

Sometimes is just feels damn good to be nice.

...something which I would wish programmers would do much more to each other, particularly in such divisive times like today.

Error Handling in ASP.NET Core

Thu, 19 Jan 2017 00:00:00 +0000

Almost two years ago I wrote a blog post on demystifying ASP.NET MVC 5 error pages and error logging, which became one of my most popular posts on this blog. At that time of writing the issue was that there were an awful lot of choices on how to deal with unhandled exceptions in ASP.NET MVC 5 and no clear guidance or recommendation on how to do it the right way.

Fortunately with ASP.NET Core the choices have been drastically reduced and there is also a much better documentation on the topic itself. However, there's still a few interesting things to consider which I wanted to point out in this follow up blog post and clear up any remaining questions on ASP.NET Core error handling.

ASP.NET Core <> MVC

First I thought it's worth mentioning that the relationship between ASP.NET Core and ASP.NET Core MVC hasn't changed much since what it used to be in Classic ASP.NET.

ASP.NET Core remains the main underlying platform for building web applications in .NET Core while MVC is still an optional web framework which can be plugged into the ASP.NET Core pipeline. It's basically a NuGet library which sits on top of ASP.NET Core and offers a few additional features for the Model-View-Controller design pattern.

What that means in terms of error handling is that any exception handling capability offered by MVC will be limited to MVC. This will become much more apparent when we look at the ASP.NET Core architecture.

ASP.NET Core middleware

ASP.NET Core is completely modular and the request pipeline is mainly defined by the installed middleware in an application.

For better demonstration let's create a new boilerplate MVC application and check out the void Configure(..) method inside the Startup.cs class file:

public void Configure(
    IApplicationBuilder app,
    IHostingEnvironment env,
    ILoggerFactory loggerFactory)
{
    loggerFactory.AddConsole(Configuration.GetSection("Logging"));
    loggerFactory.AddDebug();

    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseDatabaseErrorPage();
        app.UseBrowserLink();
    }
    else
    {
        app.UseExceptionHandler("/Home/Error");
    }

    app.UseStaticFiles();

    app.UseIdentity();

    app.UseMvc(routes =>
    {
        routes.MapRoute(
            name: "default",
            template: "{controller=Home}/{action=Index}/{id?}");
    });
}

Because that is a lot of boilerplate code for a simple web application I'll trim it down to the main points of interest:

app.UseExceptionHandler("/Home/Error");
app.UseStaticFiles();
app.UseIdentity();
app.UseMvc(routes => ...);

What you can see here is fairly self explanatory, but there's a few key things to understand from this code. The app.Use...() (extension-) method calls are enabling several middleware by registering them with the IApplicationBuilder object. Each middleware will be made responsible for invoking the next middleware in the request pipeline, which is why the order of the app.Use...() method calls matter.

For example this is a rough skeleton of the StaticFileMiddleware:

public StaticFileMiddleware(RequestDelegate next, ...)
{
    // Some stuff before

    _next = next;

    // Some stuff after
}

public Task Invoke(HttpContext context)
{
    // A bunch of code to see if this middleware can
    // serve a static file that matches the HTTP request...

    // If not the code will eventually reach this line:
    return _next(context);
}

I cut out some noise to highlight the usage of the RequestDelegate variable.

As you can see each middleware must accept a RequestDelegate object in the constructor and each middleware must implement a method of type Task Invoke(HttpContext context).

The RequestDelegate is, as its name suggest, a delegate which represents the next middleware in the lifecycle. ASP.NET Core defers the responsibility of invoking it to the current middleware itself. For example if the StaticFileMiddleware is not able find a static file which matches the incoming HTTP request then it will invoke the next middleware by calling return _next(context); at the end. On the other hand if it was able to find the requested static file then it will return it to the client and never invoke the next or any subsequent middleware anymore.

This is why the order of the app.Use...() method calls matter. When you think about it the underlying pattern can be seen a little bit like an onion:

A HTTP request will travel from the top level middleware down to the last middleware, unless a middleware in between can satisfy the request and return a HTTP response earlier to the client. In contrast an unhandled exception would travel from the bottom up. Beginning at the middleware where it got thrown it would bubble up all the way to the top most middleware waiting for something to catch it.

In theory a middleware could also attempt to make changes to the response after it has invoked the next middleware, but this is normally not the case and I would advise against it, because it could result in an exception if the other middleware already wrote to the response.

Error handling should be the first middleware

With that in mind it is clear that in order to catch any unhandled exception an error handling middleware should be the first in the pipeline. Only then it can guarantee a final catch if nothing else caught the exception before.

Because MVC is typically registered towards the end of the middleware pipeline it is also clear that exception handling features (like the infamous ExceptionFilters) within MVC will not be able to catch every exception.

For more information on middleware please check out the official documentation.

Custom Exception Handlers

Now that middleware and exception handling hopefully makes sense I also wanted to quickly show how to create your own global exception handler in ASP.NET Core.

Even though there is already quite a few useful exception handlers in the Microsoft.AspNetCore.Diagnostics NuGet package available, it still might make sense to create your own one as well. For example one might want to have an exception handler which logs critical exceptions to Sentry by using Sentry's Raven Client for .NET or one might want to implement an integration with a bug tracking tool and log a new ticket for every NullReferenceException that gets thrown. Another option would be an integration with elmah.io.

There is many good reasons why someone might want to create additional exception handlers and it might even be useful to have multiple exception handlers registered at once. For example the first exception handler logs a ticket in a bug tracking system and re-throws the original exception. Then the next exception handler could log the error in ELMAH and re-trigger the original exception again. The final exception handler might catch the exception and return a friendly error page to the client. By having each exception handler focusing on a single responsibility they automatically become more re-usable across multiple projects and it would also enable to use different combinations on different environments (think dev/staging/production).

A good example of writing your own exception handling middleware is the default ExceptionHandlerMiddleware in ASP.NET Core.

A default exception handler boilerplate would look like this:

using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

namespace SomeApp
{
    public sealed class CustomExceptionHandlerMiddleware
    {
        private readonly RequestDelegate _next;
        private readonly ILogger _logger;

        public CustomExceptionHandlerMiddleware(
            RequestDelegate next,
            ILoggerFactory loggerFactory)
        {
            _next = next;
            _logger = loggerFactory.
                    CreateLogger<CustomExceptionHandlerMiddleware>();
        }

        public async Task Invoke(HttpContext context)
        {
            try
            {
                await _next(context);
            }
            catch (Exception ex)
            {
                try
                {
                    // Do custom stuff
                    // Could be just as simple as calling _logger.LogError

                    // if you don't want to rethrow the original exception
                    // then call return:
                    // return;
                }
                catch (Exception ex2)
                {
                    _logger.LogError(
                        0, ex2,
                        "An exception was thrown attempting " +
                        "to execute the error handler.");
                }

                // Otherwise this handler will
                // re -throw the original exception
                throw;
            }
        }
    }
}

In addition to the RequestDelegate the constructor also accepts an ILoggerFactory which can be used to instantiate a new ILogger object.

In the Task Invoke(HttpContext context) method the error handler basically does nothing other than immediately calling the next middleware. Only if an exception is thrown it will come into action by capturing it in the catch block. What you put into the catch block is up to you, but it would be good practice to wrap any non trivial code in a second try-catch block and default back to basic logging if everything else is falling apart.

I hope all of this made sense and that this blog post was useful again. Personally I find it extremely nice to see how well ASP.NET Core has evolved from its predecessor. If you have any more questions just drop me a comment below.

Running Suave in ASP.NET Core (and on top of Kestrel)

Thu, 15 Dec 2016 00:00:00 +0000

Ho ho ho, happy F# Advent my friends! This is my blog post for the F# Advent Calendar in English 2016. First a quick thanks to Yan Cui who has pointed out this calendar to me last year and a big thanks to Sergey Tihon who is organising this blogging event and was kind enough to reserve me a spot this year.

In this blog post I wanted to write about two technologies which I am particularly excited about: Suave and ASP.NET Core. Both are frameworks for building web applications, both are written in .NET and both are open source and yet they are very different. Suave is a lightweight web server written entirely in F# and belongs to the family of micro frameworks similar to NancyFx. ASP.NET Core is Microsoft's new cloud optimised web framework which has been built from the ground up on top of .NET Core and all of its goodness. Both are fairly new cutting edge technologies and both are extremely fun to work with.

What I like the most about Suave is that it's written in F# for F#. It is really well designed and embraces functional concepts like railway oriented programming in its core architecture. Lately I've been a big fan of functional programming and being able to build web applications in a functional way is not only very productive but also a heap of fun. ASP.NET Core is object oriented and closer related to C#, but nonetheless an extraodinary new web framework. After more than 14 years of developing (the old) ASP.NET stack Microsoft has completely revamped the platform and built something new which is extremely fast and flexible. I love Kestrel, I love how ASP.NET Core is completely modular and extendable (via middleware) and I love how it is cross platform compatible and supported by Microsoft (Mono you have served us well but I am glad to move on now). There's more than one good reason to go with either framework and that's why I really wanted to combine them together.

Ideally I would like to continue building web applications with Suave in F# and then plug them into the ASP.NET Core pipeline to run them on top of Kestrel and benefit from both worlds.

Suave inside ASP.NET Core in theory

In order to better understand Suave let's have a quick look at a simple web application:

open System
open Suave
open Suave.Successful
open Suave.Operators
open Suave.RequestErrors
open Suave.Filters

let simpleApp =
    choose [
        GET >=> choose [
            path "/" >=> OK "Hello world from Suave."
            path "/ping" >=> OK "pong"
        ]
        NOT_FOUND "404 - Not found."
    ]

[<EntryPoint>]
let main argv =
    startWebServer defaultConfig simpleApp
    0

Even in this simple example you can clearly see the core concept behind Suave. An application is always an assemble of one or many web parts. A WebPart is a function which takes a HttpContext and returns an option of HttpContext wrapped in an async workflow. Through combinators such as choose or >=> (and many others) one can compose a complex web application with routing, model binding, view engines and anything else that someone might want to do. At the end there is one top level function of type WebPart which takes in a HttpContext and returns a HttpContext. In this example this function is called simpleApp.

In theory the one thing required to plug a Suave web app into ASP.NET Core would be to take an incoming HTTP request from ASP.NET Core and convert it into an HttpContext in Suave, execute the top level web part, and then translate the resulting HttpContext back into an ASP.NET Core response:

The other thing which you get with Suave is a self hosted web server which is built into the framework and the traditional way of starting a Suave web application. The startWebServer function takes a SuaveConfig object and the top level WebPart as input parameters. The config object allows web server specific configuration such as HTTP bindings, request quotas, timeout limits and many more things to be set.

When putting a Suave app into ASP.NET Core then it would be running on a different web server under the hood (Kestrel by default) and it wouldn't necessarily make sense to use an existing SuaveConfig in this scenario. Considering that ASP.NET Core offers other natural ways of configuring server settings, I think it is fair to skip the SuaveConfig when merging Suave into ASP.NET Core and mainly focusing on a smooth WebPart integration.

Suave inside ASP.NET Core in practice

Taking theory into practise I thought I can make it happen, and when a programmer says that then it usually means to google for an existing solution first. I was lucky to discover Suave.Kestrel which is a super early alpha version of the above concept written by Krzysztof Cieślak. Krzysztof is the developer behind the Ionide project which makes F# development in Visual Studio Code even possible, and therefore a massive thanks to him and his great contributions as well!

Even though this project was a good start to begin with there was still loads of work left to do. First I started off by using the existing code and trying to extend it, but then I quickly realised that I was fighting more with the tools than writing any code, which lead me to the decision of creating an entirely new project written in C#. Why C#? Because the Visual Studio tooling for F# projects in .NET Core is non existent (at least at the moment). As much as I love F# if I cannot properly debug or reason with my code then I rather switch to C# and get the job done.

However as a seasoned C# developer that was not a big problem and in the end it wouldn't even matter if the library was written in C#, F# or VB.NET as long as it would allow an easy integration from an F# point of view. Moments like this make me really appreciate the flexibility of the .NET framework.

Introducing Suave.AspNetCore

After my initial start on the project it took me another 3 months (mainly because I had absolutely no time) to finally release the first version of Suave.AspNetCore and make it available. Since yesterday anyone can install Suave.AspNetCore as a NuGet package for a .NET Core application.

I decided to name the package Suave.AspNetCore because I thought it is a more representative name of what the NuGet package has to offer. While this library makes it perfectly possible to run Suave on top of Kestrel it is certainly not limited to it. Suave.AspNetCore gives a way of plugging a Suave WebPart into the ASP.NET Core pipeline and run it on any environment of someone's desire. In theory Suave can be run alongside NancyFx and ASP.NET Core MVC in the same ASP.NET Core application and let the middleware decide which framework is suited best to satisfy an incoming request.

Current release information

The current version should be able to deal with any incoming web request which can be handled by a Suave WebPart. One thing that is missing (but already in the works) is the support for Suave's web socket implementation.

I shall also note that Suave and F# itself don't have an official stable release for .NET Core yet and therefore the project as a whole should be taken with some caution.

Suave.AspNetCore in action

Ok enough of the talk and let's look at a demo.

First I'll start by using one of my existing F# ASP.NET Core project templates and upgrade it to .NET Core 1.1.

Then I add Suave, Suave.AspNetCore and Newtonsoft.Json to the dependencies:

"dependencies": {
    "Microsoft.FSharp.Core.netcore": "1.0.0-alpha-*",
    "Microsoft.AspNetCore.Diagnostics": "1.1.0",
    "Microsoft.AspNetCore.Server.Kestrel": "1.1.0",
    "Microsoft.Extensions.Logging.Console": "1.1.0",

    "Suave": "2.0.0-*",
    "Suave.AspNetCore": "0.1.0",
    "Newtonsoft.Json": "9.0.1"
  }

Next I move on to the Startup.fs file and create a Suave web application:

module App =
    let catchAll =
        fun (ctx : HttpContext) ->
            let json =
                JsonConvert.SerializeObject(
                    ctx.request,
                    Formatting.Indented)
            OK json
            >=> Writers.setMimeType "application/json"
            <| ctx

This app is very basic. You can see that it is a single web part which uses Json.NET to serialize the entire HttpContext and then later returns a successful response of the Json text with a mime type of application/json.

It is not hugely interesting but it is a nice function which will handle every incoming web request and output the resulting HttpContext in a more or less readable way. It's at least a good way of quickly verifying if the incoming web request has been correctly converted into a Suave HttpContext object.

Finally I go to the Startup class and hook up the Suave catchAll web app into the ASP.NET Core pipeline via a middleware:

type Startup() =
    member __.Configure (app : IApplicationBuilder)
                        (env : IHostingEnvironment)
                        (loggerFactory : ILoggerFactory) =
        app.UseSuave(App.catchAll) |> ignore

Save all, dotnet restore, dotnet build and dotnet run:

If everything is correct then going to http://localhost:5000/ should return a successful response like this:

You can check out the sample app in GitHub and try it yourself!

Differences between vanilla Suave and Suave.AspNetCore

After running a few tests of my own I noticed a few minor differences.

First I noticed that the original Suave web server converts all HTTP headers into lower case values. For example Content-Type: text/html would be stored as content-type: text/html in Suave's HTTP header collection. In contrast ASP.NET Core preserves the original casing. When using the Suave.AspNetCore middleware then it will match the original Suave behaviour, but can be easily overriden by setting the preserveHttpHeaderCasing parameter to true in the UseSuave method:

app.UseSuave(App.catchAll, true) |> ignore

Another difference which I found was that Suave always sets the host variable to 127.0.0.1 for local requests, even when I explicitely call the API with http://localhost:5000/. I wasn't able to find out why or where this is happening and if there is a good reason for it. In this case I didn't align with the original Suave behaviour and kept the values provided by ASP.NET Core.

Other than this I haven't found any big differences and hope that it is (mostly) bug free. The project is open source and I am open for ideas, help or suggestions of any kind.

Merry Christmas everyone!

Building and shipping a .NET Core application with Docker and TravisCI

Wed, 19 Oct 2016 00:00:00 +0000

With the .NET Core ecosystem slowly maturing since the first official release this year I started to increasingly spend more time playing and building software with it.

I am a big fan of managed CI systems like AppVeyor and TravisCI and one of the first things I wanted to work out was how easily I could build and ship a .NET Core application with one of these tools. This was a major consideration for me because I would have been less interested in building a .NET Core app if the deployment story wasn't great yet and I am not very keen in building my own CI server as I don't think this is the best use of a developer's time. Luckily I was very happy to find out that the deployment experience and integration with TravisCI is extremely easy and intuitive, which is what I will be trying to cover in this blog post today.

Up until now I was more or less tied down to AppVeyor as the only vendor which uses Windows Server VMs for its build nodes and therefore the only viable option of building full .NET framework applications. TravisCI and other popular CI platforms use Linux nodes for their build jobs and .NET support was limited to the Mono framework at most. However, with .NET Core being the first officially Microsoft supported cross platform framework my options have suddenly increased from one to many. TravisCI already offered a good integration with Mono and now that .NET Core is part of their default offering I was keen to give it a shot.

In this blog post I will be covering what I believe is a typical deployment scenario for a .NET Core application which will be shipped as a Docker image to either the official Docker Hub or a private registry.

1. Creating a .NET Core application

First I need to create a .NET Core application. For the purpose of this blog post I am just going to create a default hello world app and you can skip this step for the most part if you are already familiar with the framework. For everyone else I will quickly skim through the creation of a new .NET Core application.

Let's open a Windows command line prompt and navigate to C:\temp and create a new folder called NetCoreDemo:

cd C:\temp
mkdir NetCoreDemo
cd NetCoreDemo

Inside that folder I can run dotnet new --type console to create a new hello world console application:

For a full reference of the dotnet new command check out the official documentation.

If you don't have the .NET Core CLI available you need to install the .NET Core SDK for Windows (or your operating system of choice).

After the command has completed I can run dotnet restore to restore all dependencies followed by a dotnet run which will build and subsequently start the hello world application:

This is literally all I had to do to get a simple C# console app running and therefore will stop at this point and move on to the next part where I will set up a build and deployment pipeline in TravisCI.

If you want to learn more about building .NET Core applications then I would highly recommend to check out the official ASP.NET Core tutorials or read other great blog posts by developers who have covered this topic extensively.

2. Setting up TravisCI for building a .NET Core application

If you are not familiar with TravisCI yet (or a similar platform), then please follow the instructions to set up TravisCI with your source control repository and add a .travis.yml file to your project repository. This file will contain the entire build configuration for a project.

The first line in the .travis.yml file should be the language declaration. In our case this will be language: csharp which is the correct setting for any .NET language (including VB.NET and F#).

Next we need to set the correct environment type.

The standard TravisCI build environment runs on an Ubuntu 12.04 LTS Server Edition 64 bit distribution. This is no good for us because .NET Core only supports Ubuntu 14.04 or higher. Fortunately there is a new Ubuntu 14.04 (aka Trusty) beta environment available. In order to make use of this new beta environment we need to enable sudo and set the dist setting to trusty:

sudo: required
dist: trusty

Next I want to specify what version of Mono and .NET Core I want to have installed when running my builds. At the moment I am only interested in .NET Core so I am going to skip Mono and set the dotnet setting to the currently latest SDK:

language: csharp
sudo: required
dist: trusty
mono: none
dotnet: 1.0.0-preview2-003131

The next step is not required nor necessarily recommended, but more of my personal preference to disable the .NET Core Tools Telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1 during the install step of the TravisCI lifecycle:

install:
  - export DOTNET_CLI_TELEMETRY_OPTOUT=1

After that I have to set access permissions for two script files in the before_script step:

before_script:
  - chmod a+x ./build.sh
  - chmod a+x ./deploy.sh

The chmod command changes the access permissions of my build and deployment script to allow execution by any user on the system. TravisCI recommends to set chmod ugo+x which is effectdively the same as chmod a+x, where a is a shortcut for ugo.

Following before_script I am going to set the script step which is responsible for the actual build instructions:

script:
  - ./build.sh

At last I am giong to define a deploy step as well, which will automatically trigger only after the script setp has successfully completed:

deploy:
  - provider: script
    script: ./deploy.sh $TRAVIS_TAG $DOCKER_USERNAME $DOCKER_PASSWORD
    skip_cleanup: true
    on:
      tags: true

Here I am essentially calling a second script called deploy.sh and passing in three environment variables which I will explain in a moment. Additionally I defined the trigger to deploy for tags only. You can set up different deploy conditions, but in most cases you either want to deploy on each push to master or when a commit has been tagged. I chose the latter, because sometimes I want to publish an alpha or beta version of my application which is likely to be on a different branch than master and therefore the tag condition made more sense in my case.

The TRAVIS_TAG variable is a default environment variable which gets set by TravisCI for every build which has been triggered by a tag push and will contain the string value of the tag. DOCKER_USERNAME and DOCKER_PASSWORD are two custom environment variables which I have set through the UI to follow TravisCI's recommendation to keep sensitive data secret:

Another option would have been to encrypt environment variables in the .travis.yml file to keep those values secret. Both options are valid as far as I know and it is up to you which one you prefer.

Tip:

If you have to store access credentials to 3rd party platforms like a private registry or the official Docker Hub inside TravisCI then it is highly recommended to register a dedicated user for TravisCI and add that user as an additional collaborator to your Docker Hub repository, so that you can easily limit or revoke access when required:

After defining the script and deploy step I am basically done with the .travis.yml file.

Note that I purposefully didn't choose to place the individual build and deployment instructions directly into the script step, because I wanted to seperate out the actual build instructions from the TravisCI configuration.

This has a few advantages:

There is a clear distinction between environment setup and the actual build steps which are required to build and deploy the project. The .travis.yml file is the definition for the build environment and the build.sh and deploy.sh script files are the recipe to build and deploy an application.
The build and deploy scripts are completely independent from the CI platform and I could easily switch the CI provider at any given time.
The actual build and deployment scripts can be executed from anywhere. Both are a generic bash script which developers can run on their personal machines to build, test and deploy a project.

The last point is probably the most important in my view. Even though managed CI systems are super easy to integrate with, it can be a pain if you are tied down to a particular provider. Imagine you have a new developer joining your team and the first question they ask is how to build your project. It would be a pain to tell them to open up the .travis.yml file and follow all the instructions manually if you could just tell them to run build.sh and it will work.

If I put everything together then the final .travis.yml file will look something like this:

language: csharp
sudo: required
dist: trusty
mono: none
dotnet: 1.0.0-preview2-003131
install:
  - export DOTNET_CLI_TELEMETRY_OPTOUT=1
before_script:
  - chmod a+x ./build.sh
  - chmod a+x ./deploy.sh
script:
  - ./build.sh
deploy:
  - provider: script
    script: ./deploy.sh $TRAVIS_TAG $DOCKER_USERNAME $DOCKER_PASSWORD
    skip_cleanup: true
    on:
      tags: true

One last thing that I wanted to mention is that even though I said we are going to use Docker to deploy the project I didn't have to specify Docker as an extra service anywhere in the .travis.yml file. This is because unlike the standard TravisCI environment the Trusty beta environment comes with Docker pre-configured out of the box.

3. Building and deploying a .NET Core app from a bash script

Now that the build environment is set up in the .travis.yml file and we deferred the entire build and deployment logic to external bash scripts we have to actually create those scripts to complete the puzzle.

build.sh

The build.sh script is going to be very quick:

#!/bin/bash
set -ev
dotnet restore
dotnet test
dotnet build -c Release

The first line is not necessarily required, but it is good practice to include #!/bin/bash at top of the script so the shell knows which interpreter to run. The second line tells the shell to exit immediately if a command fails with a non zero exit code (set -e) and to print shell input lines as they are read (set -v).

The last three commands are using the normal dotnet CLI to restore, build and test the application.

deploy.sh

The deploy.sh script is going to be fairly easy as well. The first two lines are going to be the same as in build.sh and then I am assigning the three parameters that we are passing into the script to named variables:

#!/bin/bash
set -ev

TAG=$1
DOCKER_USERNAME=$2
DOCKER_PASSWORD=$3

Next I am going to use the dotnet CLI publish command to package the application and all of its dependencies into the publish folder:

dotnet publish -c Release

Now that everything is packaged up I can use the docker CLI to build an image with the supplied tag and the latest tag:

docker build -t repository/project:$TAG bin/Release/netcoreapp1.0/publish/.
docker tag repository/project:$TAG repository/project:latest

Make sure that repository/project matches your own repository and project name.

Lastly I have to authenticate with the official Docker registry and push both images to the hub:

docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
docker push repository/project:$TAG
docker push repository/project:latest

And with that I have finished the continuous deployment setup with Docker and TravisCI. The final deploy.sh looks like this:

#!/bin/bash
set -ev

TAG=$1
DOCKER_USERNAME=$2
DOCKER_PASSWORD=$3

# Create publish artifact
dotnet publish -c Release src

# Build the Docker images
docker build -t repository/project:$TAG src/bin/Release/netcoreapp1.0/publish/.
docker tag repository/project:$TAG repository/project:latest

# Login to Docker Hub and upload images
docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
docker push repository/project:$TAG
docker push repository/project:latest

Tip:

Some projects follow a naming convention where version tags begin with a lowercase v in git, for example v1.0.0, but want to remove the v from the Docker image tag. In that case you can use this additional snippet to create a variable called SEMVER which will be the same as TAG without the leading v:

# Remove a leading v from the major version number (e.g. if the tag was v1.0.0)
IFS='.' read -r -a tag_array <<< "$TAG"
MAJOR="${tag_array[0]//v}"
MINOR=${tag_array[1]}
BUILD=${tag_array[2]}
SEMVER="$MAJOR.$MINOR.$BUILD"

Place that snippet after the dotnet publish command in the deploy.sh and use $SEMVER instead of $TAG when building and publishing the Docker images.

If you want to see a full working example you can check out one of my open source projects where I use this setup to publish a Docker image of an F# .NET Core application.

Load testing a Docker application with JMeter and Amazon EC2

Tue, 27 Sep 2016 00:00:00 +0000

A couple of months ago I blogged about JMeter load testing from a continuous integration build and gave a few tips and tricks on how to get the most out of automated load tests. In this blog post I would like to go a bit more hands on and show how to manually load test a Docker application with JMeter and the help of Amazon Web Services.

I will be launching two Amazon EC2 instances to conduct a single load test. One instance will host a Docker application and the other the JMeter load test tool. The benefit of this setup is that Docker and JMeter have their own dedicated resources and I can load test the application in isolation. It also allows me to quickly tear down the Docker instance and vertically scale it up or down to measure the impact of it.

Launching a Docker VM

First I will create a new EC2 instance to host the Docker container. The easiest way of doing this is to go through the online wizard and select the Ubuntu 14.04 base image and paste the following bash script into the user data field to automatically pre-install the Docker service during the launch up:

#!/bin/bash

# Install Docker
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo bash -c 'echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" >> /etc/apt/sources.list.d/docker.list'
sudo apt-get update
sudo apt-get install linux-image-extra-$(uname -r) -y
sudo apt-get install apparmor
sudo apt-get install docker-engine -y
sudo service docker start

# Run [your] Docker container
sudo docker run -p 8080:8888 dustinmoris/docker-demo-nancy:0.2.0

At the end of the script I added a docker run command to auto start the container which runs my application under test. Replace this with your own container when launching the instance.

Simply click through the rest of the wizard and a few minutes later you should be having a running Ubuntu VM with Docker and your application container running inside it.

Make sure to map a port from the container to the host and open this port for inbound traffic. For example if I launched my container with the flag -p 8080:8888 then I need to add the port 8080 to the inbound rules of the security group which is associated with this VM.

Launching a JMeter VM

Next I am going to create a JMeter instance by going through the wizard for a second time. Just as before I am using Ubuntu 14.04 as the base image and the user data field to install everything I need during the launch-up:

#!/bin/bash

# Install Java 7
sudo apt-get install openjdk-7-jre-headless -y

# Install JMeter
wget -c http://ftp.ps.pl/pub/apache//jmeter/binaries/apache-jmeter-3.0.tgz -O jmeter.tgz
tar -xf jmeter.tgz

Don't forget to open the default SSH port 22 in the security group of the JMeter instance.

Only a short time later I have two successfully created VMs with Docker and JMeter being fully operational and ready to run some load tests.

Running JMeter tests

Running load tests from the JMeter instance is fairly straight forward now. I am going to remote connect to the JMeter instance, copy a JMeter test file on the machine and then launch the JMeter command line tool to run the load tests remotely. Afterwards I will download the JMeter results file and analyse the test data in my local JMeter GUI.

Download PuTTY SSH client tools

From here on I will describe the steps required to remote connect from a Windows desktop, which might be slightly different than what you'd have to do to connect from a Unix based system. However, most things are very similar and and it should not be too difficult to follow the steps from a Mac or Linux as well.

In order to SSH from Windows to a Linux VM you will have to download the PuTTY SSH client. Whilst you are on the download page you might also download the PSCP and PuTTYgen tools. One will be needed to securely transfer files between your Windows machine and the Linux VM and the other to convert the SSH key from the .pem to the .ppk file format.

Convert SSH key from .pem to .ppk

Before we can use PuTTY to connect to the Ubuntu VM we have to convert the SSH key which has been associated with the VM from the .pem to the .ppk file format:

Open puttygen.exe
Click on the "Load" button and locate the .pem SSH key file
Select the SSH-2 RSA option
Click on "Save private key" and save the key as a .ppk file

Once completed you can use the new key file with the PuTTY SSH client to remote connect to the EC2 instance.

Remote connect to the EC2 instance

Open putty.exe
Type the public IP of the EC2 instance into the host name field
Prepend ubuntu@ to the IP address in the host name field
(this is not necessarily required, but speeds up the login process later on)
On the left hand side in the tree view expand the "SSH" node and then select "Auth"
Browse for the .ppk private key file
Go back to "Session" in the tree view
Type in a memorable name into the "Saved Sessions" field and click "Save"
Finally click on the "Open" button and connect to the VM

At this point you should be presented with a terminal window and being connected to the JMeter EC2 instance.

Upload a JMeter test file to the VM

Now you can use the pscp.exe tool from a normal Windows command prompt to copy files between your local Windows machine and the Ubuntu EC2 instance in the cloud.

The first argument specifies the source location and the second argument the destination path. You can target remote paths by prepending the username and the saved session name to it.

For example I downloaded the pscp.exe into C:\temp\PuTTY and have an existing JMeter test plan saved under C:\temp\TestPlan.jmx which I would like to upload to the JMeter instance. I named the session in PuTTY demo-session and therefore can run the following command from the Windows command prompt:

C:\temp\PuTTY\pscp.exe C:\temp\TestPlan.jmx ubuntu@demo-session:TestPlan.jmx

Usually the upload is extremely fast. If you don't know how to create a JMeter test plan then you can follow the official documentation on building a basic JMeter web test plan.

Running JMeter from the command line

After uploading the .jmx file we can switch back to the PuTTY terminal and run the test plan from the JMeter command line.

If you followed all the steps from before then you can find JMeter under /apache-jmeter-3.0/bin/./jmeter on the EC2 instance. Use the -n flag to run it in non-GUI mode, the -t parameter to specify the location of the test plan and -l to set the path of the results file:

apache-jmeter-3.0/bin/./jmeter -n -t TestPlan.jmx -l results.jtl

Run this command, wait and watch the test being executed until it's completed.

Download the JMeter results file

Finally when the test has finished you can download the results file via the PSCP tool again:

C:\temp\PuTTY\pscp.exe ubuntu@demo-session:results.jtl C:\temp\

From here on everything should be familiar and you can retrospectively open the results.jtl from an available JMeter listener and analyse the data in the JMeter GUI.

With the help of a cloud provider like Amazon Web Services and Docker containers it is super easy to quickly spin up multiple instances and run many load tests at the same time without them interfering with each other. You can test different application versions or instance setups simultanuly and optimise for the best performance.

Creating a Slack bot with F# and Suave in less than 5 minutes

Mon, 22 Aug 2016 00:00:00 +0000

Slack has quickly gained a lot of popularity and became one of the leading team communication tools for developers and technology companies. One of the main compelling features was the great amount of integrations with other tools and services which are essential for development teams and project managers. Even though the list of apps seems to be huge, sometimes you need to write your own custom integration and Slack wants it to be as easy and simple as possible. How easy and how fast this can be done I will show you in this blog post.

A simple slash command

In most cases you will probably need to create one or more new slash commands which can be used by slack users to perform some actions.

For this tutorial let's assume I would like to create a new slash command to hash a given string with the SHA-512 algorithm. I would like my slack users to be able to type /sha512 <some string> into a channel and a slack bot to reply with the correct hash code.

The easiest way to achieve this is to create a new web service which will perform the hashing of the string and integrate it with the Slash Commands API.

Building an F# web service which integrates with Slash commands

Let's begin with the web service by creating a new F# console application and installing the Suave web framework NuGet package.

The Slash Commands API will make a HTTP POST request to a configurable endpoint and submit a bunch of data which will provide all relevant information to perform our action. First I want to model a SlackRequest type which will represent the incoming POST data from the Slash Commands API:

type SlackRequest =
    {
        Token       : string
        TeamId      : string
        TeamDomain  : string
        ChannelId   : string
        ChannelName : string
        UserId      : string
        UserName    : string
        Command     : string
        Text        : string
        ResponseUrl : string
    }

For this simple web service the only two relevant pieces of information are the token and the text which get submitted. The token represents a secret string value which can be used to validate the origin of the request and the text value represents the entire string which the user typed after the slash command. For example if I type /sha512 dusted codes then the text property will contain dusted codes in the POST data.

Inside this record type I'm also adding a little helper function to extract the POST data from a Suave.Http.HttpContext object:

static member FromHttpContext (ctx : HttpContext) =
    let get key =
        match ctx.request.formData key with
        | Choice1Of2 x  -> x
        | _             -> ""
    {
        Token       = get "token"
        TeamId      = get "team_id"
        TeamDomain  = get "team_domain"
        ChannelId   = get "channel_id"
        ChannelName = get "channel_name"
        UserId      = get "user_id"
        UserName    = get "user_name"
        Command     = get "command"
        Text        = get "text"
        ResponseUrl = get "response_url"
    }

Next I'll create a function to perform the actual SHA-512 hashing:

let sha512 (text : string) =
    use alg = SHA512.Create()
    text
    |> Encoding.UTF8.GetBytes
    |> alg.ComputeHash
    |> Convert.ToBase64String

Finally I will create a new Suave WebPart to handle an incoming web request and register it with a new route /sha512 which listens for POST requests:

let sha512Handler =
    fun (ctx : HttpContext) ->
        (SlackRequest.FromHttpContext ctx
        |> fun req ->
            req.Text
            |> sha512
            |> OK) ctx

let app = POST >=> path "/sha512" >=> sha512Handler

[<EntryPoint>]
let main argv =
    startWebServer defaultConfig app
    0

With that the entire web service - even though very primitive - is completed. The entire implementation is less than 60 lines of code:

open System
open System.Security.Cryptography
open System.Text
open Suave
open Suave.Filters
open Suave.Operators
open Suave.Successful

type SlackRequest =
    {
        Token       : string
        TeamId      : string
        TeamDomain  : string
        ChannelId   : string
        ChannelName : string
        UserId      : string
        UserName    : string
        Command     : string
        Text        : string
        ResponseUrl : string
    }
    static member FromHttpContext (ctx : HttpContext) =
        let get key =
            match ctx.request.formData key with
            | Choice1Of2 x  -> x
            | _             -> ""
        {
            Token       = get "token"
            TeamId      = get "team_id"
            TeamDomain  = get "team_domain"
            ChannelId   = get "channel_id"
            ChannelName = get "channel_name"
            UserId      = get "user_id"
            UserName    = get "user_name"
            Command     = get "command"
            Text        = get "text"
            ResponseUrl = get "response_url"
        }

let sha512 (text : string) =
    use alg = SHA512.Create()
    text
    |> Encoding.UTF8.GetBytes
    |> alg.ComputeHash
    |> Convert.ToBase64String

let sha512Handler =
    fun (ctx : HttpContext) ->
        (SlackRequest.FromHttpContext ctx
        |> fun req ->
            req.Text
            |> sha512
            |> OK) ctx

let app = POST >=> path "/sha512" >=> sha512Handler

[<EntryPoint>]
let main argv =
    startWebServer defaultConfig app
    0

Now I just need to build, ship and deploy the application.

Configuring Slash Commands

Once deployed I am ready to add a new Slash Commands integration.

Go into your team's Slack configuration page for custom integrations.
e.g.: https://{your-team-name}.slack.com/apps/manage/custom-integrations
Pick Slash Commands and then click on the "Add Configuration" button:

Choose a command and confirm by clicking on "Add Slash Command Integration":

Finally type in the URL to your public endpoint and make sure the method is set to POST:

Optionally you can set a name, an icon and additional meta data for the bot and then click on the "Save Integration" button.

Congrats, if you've got everything right then you should be able to go into your team's Slack channel and type /sha512 test to get a successful response from your newly created Slack integration now.

If you are interested in a more elaborate example with token validation and Docker integration then check out my glossary micro service.

BuildStats.info |> F#

Fri, 15 Jul 2016 00:00:00 +0000

After working on the project on and off for a few months I finally found enough time to finish the migration to F#, Suave and Docker of BuildStats.info.

The migration was essentially a complete rewrite in F# and a great exercise to learn F# and Suave.io as part of a small side project which runs in a production environment now. Working with F# and Suave was so much fun that I'm already planning to develop a couple more small projects in the very near future, but more on this at another time.

Apart from migrating to F# and Suave I had also dockerised the application and switched my hosting from an Azure Web App to Amazon EC2 Container Service, because it is considerably cheaper than Microsoft's ACS at the time of writing. I was also considering Docker Cloud and Google Container Service, but the fact that I can run a micro instance in Amazon for free for 12 months was the deciding factor which pushed me towards AWS and I am very happy so far.

Small side projects like this are always a great opportunity to try and learn new technologies beyond a simple hello world application and with that in mind I also decided to document the service endpoint with RAML, which is a very intuitive language for describing web service APIs. RAML was not entirely new to me, but it was the first time I used version 1.0 and some of its new features.

Last but not least I also switched my CI system from AppVeyor to TravisCI. I wasn't really planning to do this, but because I wanted to build the application with Mono on Linux I had to make that transition as well. Nevertheless I am still a big fan of AppVeyor and will continue using it as my primary CI sytem for all Windows based builds and Travis is just as great as well.

A lot of (good) things have been going on in my private life in the last 6 months and I didn't get as much time to blog and work on side projects as I wanted to do, but now that things have gotten a bit calmer again I hope to get more time to keep this blog more updated again and talk about all the stuff that I am doing every day.

Wish you all a great weekend and stay tuned :)

JMeter Load Testing from a continuous integration build

Fri, 17 Jun 2016 00:00:00 +0000

In the last two weeks I have been doing a series of load tests and the tool I've been using was Apache JMeter. JMeter is an open source, cross platform load testing tool written in Java. Unlike the Apache benchmarking tool (aka AB) JMeter has been specifically designed to load test static and dynamic web services with high accuracy. Running those tests manually in the beginning of a new project might be fun, but if you have a high traffic web service which needs periodic testing then integrating those tests into your CI system might be a good idea.

In this short blog post I will share a few lessons I have learned along the way...

Creating a JMeter test plan (.jmx file)

The first step would be to create a test plan which will represent the type of load you would like to test against. This should match the load you anticipate in your production system as much as possible. If you have multiple instances running behind a load balancer then you can test the load of one instance only. Also the maximum load your server can handle will depend on the size of the instance under test. If your production and test instances don't match up then you might have difficulties to draw meaningful conclusions after a test run.

Creating a JMeter test plan should be the only time you use the GUI to execute your tests. The GUI consumes many resources which will have an impact on your test run and could even limit you on the maximum load you'll be able to throw at your service. If you have never created a JMeter test plan then check out this great introduction tutorial which will walk you through the basic architecture of a JMeter test plan.

Note that listeners and assertions are very resource expensive elements and should be avoided in your test plan wherever possible. After a test run you can open the results file with any of the provided listeners and analyse the data retrospectively. For example if you don't log the HTTP response data in the test results file, which is generally recommended to minimise resource consumption, then you might need to keep a response assertion enabled to check if a HTTP request was successful or not. In contrast you can probably disable all duration assertions, because you will most likely log the latency of each request and therefore be able validate this metric afterwards.

Allocating enough memory in your JVM

If you run very large load tests then you might experience a Java OutOfMemory exception. This is not uncommon considering that the default memory allocation is only 512MB large. A good rule of thumb is to set the Java heap size to ~80% of your available memory. If you intend to set the heap size to a value greater than 2GB then you will have to install the 64-bit version of the JRE.

You can change the heap size allocation by opening the jmeter.bat file inside the /bin folder and edit the following line:

set HEAP=-Xms512m -Xmx512m

In this example I set the Java heap size to a maximum value of 4GB:

set HEAP=-Xms512m -Xmx4096m

Configuring JMeter properties

Another good way of optimising your load tests is to configure JMeter to only save the data which is required for later analysis. This again will reduce the overall resource consumption and allow you to run much larger tests.

Inside the /bin folder there is a jmeter.properties file, which holds all the default properties for JMeter. You should not modify this file directly, because otherwise you would lose your custom settings when upgrading to a newer version. Instead it's recommended to create a user.properties file in the same folder (or open the pre-existing empty one) and save all custom settings in this place. Configuration settings inside the user.properties file have higher precedence over jmeter.properties.

Search for the values beginning with jmeter.save.saveservice. inside the jmeter.properties file. These properties specify the results file configuration and let you customise what data will be stored during a test run. Go through each of those lines and decide whether you want to keep the default value or change it. In the latter case copy the original line and save it with your custom value inside the user.properties file.

For instance if you are mostly interested in the average throughput, various latencies and the error rate then you could tailor the JMeter properties to those values:

jmeter.save.saveservice.output_format=csv
jmeter.save.saveservice.assertion_results_failure_message=false
jmeter.save.saveservice.assertion_results=none
jmeter.save.saveservice.data_type=false
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_data=false
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.response_message=false
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.subresults=false
jmeter.save.saveservice.assertions=false
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.connect_time=false
jmeter.save.saveservice.samplerData=false
jmeter.save.saveservice.responseHeaders=false
jmeter.save.saveservice.requestHeaders=false
jmeter.save.saveservice.encoding=false
jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.url=false
jmeter.save.saveservice.filename=false
jmeter.save.saveservice.hostname=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.sample_count=true
jmeter.save.saveservice.idle_time=true
jmeter.save.saveservice.timestamp_format=ms
jmeter.save.saveservice.default_delimiter=,
jmeter.save.saveservice.print_field_names=true

One setting which is worth pointing out is the following:

jmeter.save.saveservice.output_format=csv

This lets you specify the format of the results file. By default it is set to csv and I would recommend you to keep it this way, unless you want to store the response data in your results. In that case you'll have to change it to xml.

However, storing results in CSV has a few advantages:

JMeter is quicker in saving data in CSV than in XML
It might be easier to analyse data with 3^rd party tools such as MS Excel
Even though it's possible with XML, it is much easier to merge multiple results files from a distributed test run in CSV format

Running JMeter from the command line

Runnning JMeter from the command line is not only the recommended way of running your tests, but also the best to execute them from an automated build step.

To run in non GUI mode you have to invoke the jmeter.bat file with the -n switch. Using the -t parameter lets you specify a test plan and the -l parameter the location of the results file:

jmeter -n -t MyTestPlan.jmx -l Results.jtl

When the test run completes JMeter will exit with code 0. You can check for that code when invoking the load test from an automated step. The results file will contain all information for subsequent test analysis. When JMeter encounters an error you can inspect the jmeter.log file for more information.

Combining multiple results files from a distributed test run

Sometimes the load you want to run is so big that you have to split the execution across two or more JMeter instances, because one instance is unable to simulate the load alone. In this case you can either use the JMeter client to control multiple remote instances or configure your CI system do it yourself. Either way it will work the exact same way. The JMeter test plan will have to be executed on each individual instance and will produce a separeate results file for you. When all tests finished you'll have to collect the results files and merge them into one single file for analysis.

If you're using the CSV format you can simply append the contents from one file to the end of another. Merging multiple CSV results files is usually as simple as a few copy paste bash commands. However, if you were using the XML format then you will have to append the data in the right place which will require a bit of additional scripting.

Analysing the results file (.jtl)

You can always open a JMeter results file with one of the existing listeners from the JMeter GUI. If your results were saved in CSV format then you can also open them in a spreadsheet or almost any other statistical software.

If you want to analyse and evaluate your results from an automated build then you will have to do some more scripting. It mostly depends what type of analysis you would like to perform, but in most cases you will be able to extract the relevant information with only a few simple commands.

For that purpose you could use PowerShell as the native language on Windows machines or if you are looking for something more portable then you could write a small C# or F# library, which can be either invoked directly from tools like CAKE, FAKE, the FSI or from a console application build on Mono. Just to give you a taster of how simple that can be you an check out this small F# application.

BlazeMeter for everyone

If everything from above sounds like too much work and you look for a more hassle free, extremely well functioning and feature rich solution then I can highly recommend you to sign up at BlazeMeter and let your tests run in the cloud by some real experts.

Custom error handling and logging in Suave

Tue, 31 May 2016 00:00:00 +0000

Some years ago when you wanted to develop a .NET web application it was almost given that it will run on IIS, but today we have a sheer amount of different web server technologies to our availability. If you are an F# developer or in the process of learning F# (like me) then you will most likely come across the Suave web framework.

Suave is a lightweight, non-blocking web server written in F#. It is fairly new to the .NET space, but works wonderfully well and is very idiomatic to functional paradigms. Even though it is still early days it has probably the coolest name and logo amongst its competitors already:

After working with Suave for more than a month now I can say that I find it very nice and intuitive. It is a well designed web framework which is easy to work with and allows rapid development if you know how it works. However, if you don't know how it works then you might struggle to get anything done at all. This is exactly what happened to me when I started adopting the framework in the beginning. The documentation is almost non existent and if you look for any advice on how to implement certain things then you are better off by browsing the GitHub repository directly. The lack of documentation is not a mistake, but rather a concious design decision which the Suave team explains as following:

In suave, we have opted to write a lot of documentation inside the code; so just hover the function in your IDE or use an assembly browser to bring out the XML docs.

Even though I found myself around I wish there would have been a little bit more documentation or more code examples available which would have put me on the right track straight away. With that in mind I will jump straight into some code examples myself and try to explain proper error handling and logging in Suave.

Hello World in Suave

Before we get started we need a simple web application to begin with:

open Suave
open Suave.Operators
open Suave.Successful
open Suave.Filters

let app =
    choose [
        GET >=> path "/" >=> OK "Hello World"
    ]

[<EntryPoint>]
let main argv =
    startWebServer defaultConfig app
    0

When I navigate to http://localhost:8083 I can see the "Hello World" message.

When I try to browse a path which doesn't exist then Suave will not serve the request by default. For example http://localhost:8083/foo will not return anything. If you want Suave to return a 404 (or anything else) for non existing paths then you can append a generic fallback case to the end of the webpart options:

let app =
    choose [
        GET >=> path "/" >=> OK "Hello World"
        NOT_FOUND "Resource not found."
    ]

The NOT_FOUND webpart and other pre-defined client error responses can be found in the Suave.RequestErrors namespace.

Now that I covered that, let's look at some proper error handling next.

Error Handling in Suave

In order to test error handling I need a route which throws an exception:

let errorAction =
    fun _ -> async { return failwith "Uncaught exception!!" }

let app =
    choose [
        GET >=> path "/" >=> OK "Hello World"
        GET >=> path "/error" >=> errorAction
        NOT_FOUND "Resource not found."
    ]

The default Suave error handler will return a 500 Internal Server error and a static message in plain text, unless you run your application locally in which case it will return the exception message in HTML.

If you want to change the default behaviour and provide a different implementation, then you can do this by setting a custom function of type ErrorHandler as part of the web server configuration.

The ErrorHandler type is defined as following:

type ErrorHandler = Exception -> String -> HttpContext -> WebPart

For example you can declare a new error handler like this:

let customErrorHandler ex msg ctx =
    // Change implementation as you wish
    INTERNAL_ERROR ("Custom error handler: " + msg) ctx

let customConfig =
    {
        defaultConfig with
            errorHandler = customErrorHandler
    }

[<EntryPoint>]
let main argv =
    startWebServer customConfig app
    0

Let's say you have a RESTful service and you want to return an error in Json instead of plain text. You could do this by amending the code as following:

let JSON_ERROR obj =
    JsonConvert.SerializeObject obj
    |> INTERNAL_ERROR
    >=> setMimeType "application/json; charset=utf-8"

let customErrorHandler ex msg ctx =
    JSON_ERROR ex ctx

The JSON_ERROR function is a custom helper function, which uses Newtonsoft.Json to serialize an object and pipe it through to the INTERNAL_ERROR function in combination with a mime type of "application/json; charset=utf-8".

You could go one step further and examine the Accept header of the incoming HTTP request and return the error response in a mime type which is supported by the client. The ctx parameter which is of type Suave.Http.HttpContext has all the relevant information to make that distinction:

type AcceptType =
    | Json
    | Xml
    | Other

let getAcceptTypeFromRequest ctx =
    match ctx.request.header "Accept" with
    | Choice1Of2 accept ->
        match accept with
        | "application/json" -> Json
        | "application/xml"  -> Xml
        | _                  -> Other
    | _                      -> Other

let customErrorHandler ex msg ctx =
    match getAcceptTypeFromRequest ctx with
    | Json  -> JSON_ERROR ex ctx
    | Xml   -> XML_ERROR ex ctx
    | Other -> INTERNAL_ERROR msg ctx

The XML_ERROR function doesn't exist by default and would need to be implemented similarly to the JSON_ERROR function from the previous example.

This code is not full proof, but you can get the idea of it. There is nothing you can't do with the error handler and you can tailor a custom implementation entirely to your own application requirements.

Let's move on to logging now.

Custom logging in Suave

Just like the error handler is set in the web server configuration the default logger can be overridden there as well, but with the small difference that the logger needs to be of type Suave.Logging.Logger.

The Logger interface has only one method for logging events:

abstract member Log : LogLevel -> (unit -> LogLine) -> unit

It's simple but enough to build custom loggers for most use cases:

let pressTheRedButton line =
    line |> ignore // Implement here

let logAsNormal level line =
    line |> ignore // Implement here

type CustomLogger() =
    interface Logger with
        member __.Log level line =
            match level with
            | LogLevel.Fatal -> pressTheRedButton line
            | _ -> logAsNormal level line

let customConfig =
    { defaultConfig with
        errorHandler = customErrorHandler
        logger = new CustomErrorLogger() }

The log level argument is an enum of type Suave.Logging.LogLevel and has the following options available:

Verbose
Debug
Info
Warn
Error
Fatal

It also provides a few helper methods to convert between its string and integer representation as well as a few overloads for comparison and an implementation of the IComparable and IEquatable interface. Check the full implementation for a complete overview.

Suave also provides a few default loggers which can be used out of the box. There is a ConsoleWindowLogger and an OutputWindowLogger which do exactly what they say.

Additionally there is another useful logger called CombiningLogger which lets you combine multiple loggers at once. This allows you to build smaller and more specialised loggers instead of one big bloated implementation:

let sendMessageToSlack line =
    line |> ignore // Implement here

let sendEmailToTeam line =
    line |> ignore // Implement here

type SlackLogger() =
    interface Logger with
        member __.Log level line =
            if level = LogLevel.Fatal then sendMessageToSlack line

type EmailNotifier() =
    interface Logger with
        member __.Log level line =
            if level >= LogLevel.Error then sendEmailToTeam line

let customConfig =
    { defaultConfig with
        errorHandler = customErrorHandler
        logger = CombiningLogger(
            [SlackLogger()
             EmailNotifier()
             OutputWindowLogger(LogLevel.Info)]) }

Again, there is no limit to what you can do with the logger implementation.

A popular logging framework in F# is Logary and there is even an adapter for Suave available. Funnily the official documentation for the Suave adapter is a bunch of Lorem Ipsum. Maybe it's the whole F# community which doesn't like documentation that much :) ? While this page is clearly still in progress you can find a simple example on the official Suave website in the meantime.

As you can see the Suave web framework can be very simple and powerful when you know how it works. I have to admit that the source code is well written, concise and often self explaining. Once you are familiar with the framework you will quickly find what you need, but until then you might be missing some general pointers to main topics which are common with every modern web application.

Watch US Netflix, Hulu and more from anywhere in the world

Sun, 01 May 2016 00:00:00 +0000

If you are reading this then I assume you are one of those unfortunate Netlix, Hulu and Co. users who does not live in the US and is upset that you are treated like a second class citizen by those companies? Well, you're damn right to be upset because you know you're paying the same amount of money as other users but getting a lot less for it which is just not fair. You are being discriminated because you don't live in the United States of America and the money in your pocket does not hold a picture of George Washington. It's a bloody pain and if you ask me it's an absolute disgrace that we still live in a time where we have to deal with those types of problems.

I don't even blame Netflix & Co. because these guys already get our money and they would be more than happy for us to watch whatever we want. It's some outdated laws and regulations made up by some poor souls in the content industry who force us into this miserable situation. Now the problem is that the internet is a global place and certainly does not know any borders. It has been the biggest motor of new economic growth for the last couple of decades and made the world a much smaller place than it used to be. If the internet has taught us one thing then it is that anyone can have anything, anywhere in the world with an instant effect and anyone who wants to convince us of the opposite is a dinosaur on the losing track. Patience and borders is a forgein concept which does not exist in the vocabulary of new generations, and that is for a good reason. They are impediments to innovation, growth and evolution.

Obviously the case is not as simple and clear cut as I make it sound, but it doesn't change the fact that there is a lot of stuff going awfully wrong at the moment and I feel like there is a lot of effort being made into the wrong direction. Instead of embracing the internet's full potential of global reach, companies are investing money and technology into setting up virtual borders and building detection software for people who violate those. While everyone else is making great steps forward, the media industry is desperately trying to stay resistent and not adapting to the new economy at all.

Playing cat and mouse with VPNs and Proxies

You probably know that one popular way of circumventing content restrictions is by streaming media via a VPN or proxy server. Every device which is connected to the internet has a so called IP Address. This address allows content providers to establish your geographic location and serve you a tailored view for your country. By using a proxy server you can pretend to be in a different location and trick a provider into serving you a much better offering than you would usually get. The concept is simple. Instead of streaming directly from Netflix & Co. you connect to an intermediate server, which is geographically located in the country of your desire and let that server stream the content for you on your behalf and forward it back to you. Sounds good in theory, except that Netflix and Co. have ways of detecting this spiel and will block you if not even cancel your account.

How does Netflix detect VPNs and Proxies?

The fact that they can do this is quite clever, because it is certainly not an easy thing to do. I don't know how they exactly do it, but there is some basic theory which might give us an idea how they detect whether you are using a VPN or not. First of all they are probably collecting a growing list of IP addresses which they know belong to popular VPN and proxy services. Those IP addresses get simply blacklisted and blocked. Another way of detecting proxy services would be by monitoring the amount of users connecting from the same IP address over a period of time. If you've got hundreds or thousands of users streaming from the same IP address then chances are high that this is a proxy server. Of course it could be a bunch of people streaming from a Starbucks, but even a Starbucks has to close its doors at some point in the day. A genuine user is probably going to work sometimes or at least has to sleep. If you detect streaming behaviour from an IP address which doesn't fit with normal human behaviour than it might be another indicator for dodgy activity. Now I am not saying that this is what Netflix does, and I am sure their detection system is much more sophisticated than this, but I want to share some ideas which demonstrate that detecting a VPN or proxy server is not always an impossible task.

So what can one do to trick those detection systems? Well, you'd have to stop sharing proxy servers for a beginning and make an IP look as normal as possible. Luckily setting up a private proxy server is really not that difficult and can be super cheap as well. As a matter of fact you can set up your own private proxy server entirely for free and I am going to show you how!

Amazon web services, Netflix's best friend (and ours)

Alright so you know Amazon right? It's the book shop which doesn't sell only books anymore. If you are a regular reader of my blog then you might know what I am going to say now, but if you are a non techie who came across this blog post through some other channel then you might be surprised to learn that Amazon was the pioneer of cloud providers. They are not only the largest, but also the most mature cloud operator in the world at the moment. Amazon is doing such a great job that Netflix runs its entire infrastructure in the cloud provided by Amazon.

The reason why I am telling you this is because Amazon's cloud has been so successsful that they are going mainstream by making cloud services easily available to anyone, people like you and me. If Netflix can deliver the latest season of House of Cards to the entire world with the help of Amazon web services, then surely we can host one tiny proxy server in the same cloud as well and circumvent their detection software, right? Yes, we can.

Setting up a proxy server with AWS EC2 in less than 10 minutes

Let's start off with some good news first. Amazon offers a 12 months free tier for new subscribers to their web services. This is an amazing offer and exactly what we are going to use to set up our private proxy. If you ask yourself what happens after those 12 months then wait until the end of this blog post.

First you need to sign up with Amazon web services. If you already have an account then you can skip this step, otherwise follow the instructions:

Go to the registration page and log in with your regular Amazon account
After the login select the personal account option and fill in your address details
On the next page fill in your payment information. This is required for the case when you exceed the free tier limitation, but don't worry about this, because it will not happen if you stick to my instructions
The next step is pretty cool. You will be displayed a 4 digit PIN code on the screen and receive an automated phone call from Amazon. You will be asked to enter the PIN on your phone and if you typed it in correctly then the call will end immediately and you will be directed to the next step in the registration process
From the list of support plans pick the basic (free) plan and click continue
Now you are done and you should see a message that your account will be activated within a few minutes and usually you'll get an email notification as well

Step 2: Create a private proxy with EC2

Once signed in you should see a list of available AWS Services. We are going to create a new EC2 instance which will become our private proxy server.

Click on the EC2 link from the menu:

In the top right corner make sure you have selected a US region, because after all we want our proxy to stream US content:

Next click on the Launch Instance button:

This opens up a 7-step wizard which will walk you through the configuration of a new EC2 instance. Don't worry, there is not much that needs to be done to get the proxy server up and running.

The first step lets you choose which image (AMI) to use for your new instance. An image is a snapshot of a pre-installed server. This allows you to create a new server which already has an operating system and other software installed so you don't have to do it manually each time.

For the purpose of the proxy server we don't need anything fancy and therefore can go with the Ubuntu Server, which is a free tier eligible Linux distribution:

On the next screen you can pick the size of the new instance. If you don't know what this means, think of it like the horse power of your new server.

Again, because we don't need anything fancy we can happily go with the t2.micro instance which is free tier eligible:

Don't click on the Review and Launch button yet! Confirm and continue by clicking on the Next: Configure Instance Details button.

On the third step there's a bunch of information available, but luckily the default values are exactly what we need and you don't have to change any of them, except one thing. Scroll down to the bottom and expand the Advanced Details section by clicking on the little arrow next to it:

You will be presented with a text field which can be used to specify additional commands which will run when launching the new EC2 instance. We will add a few commands which will automatically install and configure the Tinyproxy software. Tinyproxy is a free and open source proxy server for POSIX operating systems.

Copy the following code snippet into the text field:

#!/bin/bash
sudo -i
apt-get install tinyproxy
printf 'Allow xxx.xxx.xxx.xxx' >> /etc/tinyproxy.conf
/etc/init.d/tinyproxy stop
/etc/init.d/tinyproxy start

Replace the xxx.xxx.xxx.xxx with your own IP address.

This is an important step, because by default Tinyproxy does not allow any connections other than from the host itself. Therefore we need to configure who is allowed to connect to the proxy server and because we want to keep it private you will enable your own IP address only. Make sense?

The end result should look something like this, except with your own IP address:

Continue by clicking on the Next: Add Storage button.

There is literally nothing to do here or in the next step. Click Next: Tag Instance and immediately move on to Next: Configure Security Group.

This is the last important step during the configuration. Here we configure which ports will be open on the new instance. If this is the first EC2 instance you are going to create then you are likely not going to have any existing security groups set up yet.

By default the wizard will create a new security group for you and add one rule for port 22. This is the default port to SSH into your EC2 instance. Normally as a system administrator you would want to keep this port open, but for the simplicity of this setup we can overwrite it. We don't really need to SSH into the instance and if you really want to you can always edit the security group afterwards.

In the drop down select Custom TCP Rule and enter port 8888 into the Port Range field. Why port 8888? Because this is the default port which Tinyproxy listens to. Under Source pick the Custom IP option and enter your IP address in the field next to it and append "/32" to the end:

As you can see in the screen shot I also changed the security group name to something more meaningful. Feel free to pick your own name.

Click Review and Launch to get to the next screen and confirm there by clicking on Launch.

Almost done now! The final step is to create a private key pair. The private key pair is something you would need if you wanted to SSH into this instance, but as I said before, for the simple purpose of a proxy server you don't need to do this and therefore I will not go into any more detail. Just make sure you type in a meaningful name, something like "AWS Default Key Pair" or "AWS Proxy Server Key Pair" and hit the Download Key Pair button. Save this file somewhere in a secure place and keep it secret!

After that you should be able to click the Launch Instances button and let Amazon web services do the rest for you:

When you click on the instance id you get redirected back to the EC2 console where you can see your instance being initialized. It may take a few minutes until everthing is ready and once completed you should see the status to be "running" and all status checks to be OK:

The public IP address that you see with your instance is your new private proxy IP address! Take a note of it, because you will need it in the last step.

Step 3: Configure your device

The final step is to set up your device to connect via the proxy server. This is literally one single setting that you have to apply on your device. I have provided instructions for Windows, Android and iOS, but you can easily google for any other OS and find instructions online:

Changing proxy server settings in Windows

Follow these instructions even if you don't use Internet Explorer for browsing the internet. Proxy settings are a system wide setting and it doesn't matter through wich browser you open the dialog.

Open Internet Explorer by clicking the Start button and type Internet Explorer into the search box and click on the corresponding result from the list
Click the Tools button, and then click Internet Options
Click the Connections tab, and then click LAN settings
Select the Use a proxy server for your LAN check box
Enter the IP address of your proxy server and port 8888 into the text fields
Click OK and confirm everything

Change proxy server settings in Android

Go into the Settings
Click on the Wi-Fi menu item
Long tap on the network which you are currently connected to
Select the Modify network option
Expand the Advanced options
Selet the Manual option under Proxy
Type in the IP address of your proxy server into the Proxy hostname field and port 8888 into Proxy port
Save those settings

Change proxy server settings in iOS

Go into the Settings
Click on the Wi-Fi menu item
Click on the network which you are currently connected to
Select the Manual option under HTTP PROXY
Type in the IP address of your proxy server into the Server field
Type 8888 into the Port field

Test your connection

Now that everything is set up and running you should be able to stream US content from Netflix, Hulu and many more! A quick test to confirm that your proxy server is successfully running would be to google your own IP address and see that your IP appears different, from a US location now:

And what shall you do after your first 12 months of free tier eligibility? Well, I'd suggest you sign up a new account under a different email address. It takes only a couple of minutes which you have to invest every 12 months in order to run a free private proxy server. If that is too much effort then you might as well choose to let your server run and pay for its usage. The t2.micro instance costs only $0.013 per hour which would come down to $9.75 per month if you'd let it run continuously. However, in this case I'd suggest to switch it on and off as you need and reduce your cost to almost nothing.

Oh, and if you wonder about any of the IP addresses shown on the screenshots of this blog post then let me assure you that none of those servers exist anymore and you really have to go through the few minutes of setting up your own private proxy server ;)

Asynchronous F# workflows in NancyFx

Tue, 12 Apr 2016 00:00:00 +0000

I have been playing with F# quite a lot recently and to be honest I started to really like it. Besides the fact that F# is a very powerful language it is a lot of fun too! As a result I began to migrate one of my smaller web services from C# and ASP.NET MVC 5 to F# and NancyFx. Nancy as a .NET web framework works perfectly fine with any .NET language, but has certainly not been written with F# much in mind. While it works really well in C#, it can feel quite clunky with F# at some times. There are other web frameworks which are more idiomatic, but come with other tradeoffs instead.

However, when I started to migrate from C# to NancyFx and F# I had quite some difficulties to implement even the simplest things in the beginning. Asynchronous routes were one of those things. The fact that I am an F# beginner didn't help either. Unfortunately there is not a lot of documentation on NancyFx with F# available and therefore I thought I'd share some of my own examples in this blog post.

As I said before, some of the stuff is really simple and probably super easy for an experienced F# developer, but for me as a beginner it was not that obvious in the beginning.

A great start on NancyFx and F# is Michal Franc's blog post, where he shows how to register a normal route in NancyFx with F#:

type MyModule() as this =
    inherit NancyModule()
    do this.Get.["/"] <- fun ctx ->
        "Hello World" :> obj

This code is not much different from the equivalent C# implementation. MyModule inherits from the base NancyModule class and assigns a function to the root path of the HTTP GET verb. The only difference between C# and F# is that I had to explicitly upcast the string value to an object to match the method's expected signature.

Registering an asynchronous route is slightly different. The function expects an additional input parameter for the cancellation token and returns a Task or Task<T> object instead:

do this.Get.["/foo", true] <- fun ctx ct ->
    Task.FromResult("bar" :> obj)

This simple example works, but is not asynchronous, because Task.FromResult blocks on the current thread. What I really want to do is to call an asynchronous workflow from F# and execute it from an asynchronous route in NancyFx.

Asynchronous workflows in F# are different from Tasks in C# and therefore need to be converted from an Async<'T> object into a Task<T> in NancyFx.

Luckily the .NET Control.Async class offers plenty of predefined methods to make the translation very easy.

With Async.RunSynchronously one can convert an async workflow into a C# Task and execute it on the current thread synchronously:

do this.Get.["/foo"] <- fun ctx ct ->
  async {
      return "bar" :> obj
  }
  |> Async.RunSynchronously

This works, but it is still not asynchronous. If you want to run the async workflow in a non-blocking fashion in Nancy then you can pipe it to Async.StartAsTask which runs it asynchronously and returns a completed task:

do this.Get.["/foo", true] <- fun ctx ct ->
  async {
      return "bar" :> obj
  }
  |> Async.StartAsTask

In every case I had to upcast the string value to an object to match the expected return type. The Async class is full of useful functions which can run and convert asynchronous workflows from F# to C# and vice-versa.

If you are building a Nancy application in F# then you might also be interested in Fancy or Tiny Blue Robots' blog post, where both show some neat tricks on how to make Nancy feel a bit more functional.

With this I am going back to my F# work and wish everyone a happy F# day!

Filtering the AWS Service Health Dashboard

Thu, 31 Mar 2016 00:00:00 +0000

If you run anything on Amazon Web Services in production then you probably know the AWS Service Health Dashboard very well.

Sometimes when you experience a service disruption you might find yourself scrolling through the page and look for an update for one of your affected resources and then you probably wish that the icons were a little bit more distinguishable between healthy and previously unhealthy services:

Unless you have perfect eagle sight you probably struggle to quickly filter the page with your plain eye and could use some help to better visualise good from bad icons. At least this is how I feel and therefore I created this little JavaScript snippet to blend out healthy icons from the page, leaving only the bad ones visible:

[].forEach.call(document.querySelectorAll('[src="/images/status0.gif"]'), function(e) {e.style.display = "none";});

You can either copy this into your browser's console and run it directly from there or create a permanent bookmarklet for easy access in the future.

Automating CSS and JavaScript minification in ASP.NET MVC 5 with PowerShell

Sat, 26 Mar 2016 00:00:00 +0000

When I started building this blog I kept things very simple in the beginning. First there was nothing but my Hello World blog post and only later when I had more content I added more features over time. It didn't take me very long before I had to think about minifying static content such as CSS and JavaScript files to speed up page load times for my readers.

Minifying a CSS file is one of the most trivial tasks in web development and yet it is often more cumbersome than it has to be. It is too easy to forget updating a minified file when making some quick changes to the original file, or I'd update the minified file, but forget to swap the file paths in the HTML source code, leaving the live website pointing to the uncompressed version. These are typical mistakes which happen with manual tasks and every developer experiences at least once in their career. The best way to prevent these type of mistakes is to automate the entire process and reduce the human error.

The default project template in a Classic ASP.NET MVC web application offers runtime minification via the WebGrease NuGet library. It is not great but does the job for the lazy programmer. Runtime minification is not ideal because it puts additional load on the web server instead of doing the compression on the build server at an earlier stage. ASP.NET Core takes a different approach and promotes the use of post-build commands by utilizing Node modules to minify static assets. This is a much more elegant solution and works just as good.

The same an be achieved in a Classic ASP.NET application where it is perfectly feasible to use Node.js from a post-build event as well. Node is great when you care about cross platform compatibility, but it might be a slight overkill when the application only builds on a Windows machine and Node is not used anywhere else in the project. If an entire team works on Windows then you might as well use a technology which is already available to everyone. PowerShell would be one of those.

Tip: As an alternative to Node.js you can utilize CAKE or FAKE for cross platform compatible build events or entire build scripts. Both are open source and you get to use C# or F# through the entire project.

For my own website I am using PowerShell for the exact reason that I only work from a Windows machine and the website builds on a Windows server via AppVeyor.

In this blog post I will show how to use PowerShell to minify static assets from a post-build event in a Classic ASP.NET MVC 5 application and how to switch between compressed and uncompressed versions in Release and Debug mode.

Minifying CSS and JavaScript with PowerShell

Let's begin with the PowerShell script. In the first step I want to recursively find all CSS files within a given folder and exclude already minified files:

Get-ChildItem $SolutionDir -Recurse -Include *.css -Exclude *.min.css

$SolutionDir is a variable pointing to the root path of the solution. This variable will be assigned when calling the script from a post-build event. I will come back to this later again.

The next step is to iterate through all CSS files and minify them. This can be achieved by piping | the result from Get-ChildItem to a foreach loop % and call a function on each individual element:

Get-ChildItem $SolutionDir -Recurse -Include *.css -Exclude *.min.css | % {
    Compress-CssFile -CssFilePath $_
}

The $_ symbol represents each individual element in a foreach loop, which then gets assigned to the -CssFilePath parameter of the Compress-CssFile function.

Next I have to implement the Compress-CssFile function:

Function Compress-CssFile
{
    [CmdletBinding()]
    param
    (
        [string] $CssFilePath
    )

    # ToDo: Implement
}

This is the basic skeleton of the function. The param section declares all parameters which can be passed into the function and the [CmdletBinding()] attribute defines that global flags such as -Verbose or -Debug will be inherited from the calling context.

The actual implemetation can vary in many ways, but for this blog post I thought it would be a good exercise to use the public API of the CSSMinifier web service.

The API is very simple. All I have to do is to send a HTTP POST request to http://cssminifier.com/raw/ with the original CSS content in the body and subsequently receive the minified version from the body of the response.

The Compress-CssFile function only accepts the full file path of a CSS file and therefore I need to read all of its content first:

$cssFile = Get-Item -Path $CssFilePath
$content = [System.IO.File]::ReadAllText($cssFile.FullName)

Now with the content I can initialize a HTTP body object and invoke a HTTP POST request to the API:

$body = @{input = $content}
$response = Invoke-WebRequest -Uri "http://cssminifier.com/raw/" -Method Post -Body $body

Before processing any further I can validate if the request was successful:

if ($response.StatusCode -ne 200)
{
    throw "Pick your own error message"
}

If the request was successful I can grab the minified CSS content from the response and save it under the same location as the original file, but with the .min.css file extension instead:

$compressedContent = $response.Content
$newFilePath = $CssFilePath.Replace(".css", ".min.css")
Set-Content -Path $newFilePath -Value $compressedContent -Force

Note how I used the -Force flag on the Set-Content cmdlet to overwrite an existing file with the same name. This is required to update the minified file even if it already exists.

Finally I put all of the above PowerShell code into one file and add the $SolutionDir parameter at the top. I name the PowerShell file MinifyCss.ps1 and save it in the root folder of my ASP.NET solution:

[CmdletBinding()]
param
(
    [Parameter(Position = 0, Mandatory = $true)]
    [string] $SolutionDir
)

Function Compress-CssFile
{
    [CmdletBinding()]
    param
    (
        [string] $CssFilePath
    )

    $cssFile = Get-Item -Path $CssFilePath
    $content = [System.IO.File]::ReadAllText($cssFile.FullName)
    $body = @{input = $content}
    $response = Invoke-WebRequest -Uri "http://cssminifier.com/raw/" -Method Post -Body $body

    if ($response.StatusCode -ne 200)
    {
        throw "Pick your own error message"
    }

    $compressedContent = $response.Content
    $newFilePath = $CssFilePath.Replace(".css", ".min.css")

    Set-Content -Path $newFilePath -Value $compressedContent -Force
}

Get-ChildItem $SolutionDir -Recurse -Include *.css -Exclude *.min.css | % {
    Compress-CssFile -CssFilePath $_
}

This script is ready now. Implementing the same functionality for JavaScript files is trivial. Simply copy the MinifyCss.ps1 file and rename it to MinifyJavaScript.ps1. Change the implementation to point to the public JavaScript Minifier API and change the file extensions from .css to .js.

Calling PowerShell scripts from an ASP.NET post-build event

The next step is to call both PowerShell scripts from an ASP.NET post-build event.

This couldn't be any easier. Right click the project file of your ASP.NET project and select "Properties" from the menu or select the project file and hit Alt + Enter on your keyboard.

Go to the "Build Events" dialog and paste the following code into the post-build event command line:

if $(ConfigurationName) == Debug (
    echo "Skipping CSS minification in debug mode."
    echo "Skipping JavaScript minification in debug mode."
) else (
    %windir%\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NonInteractive -Command "$(SolutionDir)MinifyCss.ps1" $(SolutionDir)
    %windir%\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NonInteractive -Command "$(SolutionDir)MinifyJavaScript.ps1" $(SolutionDir)
)

This code block makes sure that we only execute the PowerShell scripts when the project doesn't build in Debug mode. This is desired because during development we might make frequent changes to the original CSS file and not want to minify the content until we are ready to build in Release mode.

The $(SolutionDir) placeholder is a reserved MSBuild macro which points to the root directory of the solution. It gets passed directly to the PowerShell script where it gets assigned to the equally named PowerShell variable. The rest happens in PowerShell.

Swap between original and minified files in ASP.NET MVC Razor views

The last piece in the puzzle is to swap between the original and the minified files in the ASP.NET MVC Razor views. In Debug mode we want to point to the original file, so that we can test CSS changes without any friction during development, but in all other cases we want to swap it for the minified version instead.

In order to distinguish between Debug and Release mode in an MVC razor view we need a little helper class:

public static class BuildProperties
{
    public static bool IsDebugMode()
    {
#if DEBUG
        return true;
#else
        return false;
#endif
    }
}

With this helper method we can easily switch between the .css and .min.css files in the HTML markup:

@if (BuildProperties.IsDebugMode())
{
    <link type="text/css" href="~/Assets/Css/site.css">
}
else
{
    <link type="text/css" href="~/Assets/Css/site.min.css">
}

If you use C# 6.0 in your razor views then you can write it even neater with this one liner where you don't have to repeat the file path twice:

<link type="text/css" href=@($"~/Assets/Css/site{(BuildProperties.IsDebugMode() ? "" : ".min")}.css")>

Voila, now you never have to worry about manually minifying static assets anymore. It just happens automatically during the Release build and the live website will reference the correct path to the minified file.

CircleCI build history charts and NuGet badges by Buildstats.info

Mon, 29 Feb 2016 00:00:00 +0000

Quick update on Buildstats.info. Two weeks ago I added CircleCI to the list of supported CI systems for the build history chart and last weekend I implemented a new badge for NuGet packages too.

CircleCI is the third continuous integration system which is supported by the build history chart now. AppVeyor and TravisCI are the other two. If you have a public open source project which is built by one of those systems then you might want to check out the official documentation for the build history chart. Its quite cool and lets you create SVG badges like the one I did for my blog:

On a complete separate note I also added a new SVG badge for NuGet packages.

I did not think of adding NuGet support in the beginning, but since Shields.io NuGet badges are broken for more than 2 weeks now I had to look for an alternative:

The issue has been reported, but it doesn't look like it will get fixed any time soon and so I went with my own solution.

For my personal projects I like to display the current version of my NuGet packages as well as the total number of downloads. With Shields.io I had to use two individual badges, but with Buildstats.info I can display both in one:

This is a first version which satisfied my own personal needs, but will likely expand over the next coming weeks providing more functionality for other users as well.

I have a few ideas, but if you are looking for something in particular then please feel free to file a feature request on the public GitHub repository.

Don't dispose externally created dependencies

Sat, 20 Feb 2016 00:00:00 +0000

Today I wanted to blog about a common mistake which I often see during code reviews or when browsing open source projects in relation to the IDisposable interface. Heck, even Microsoft got it wrong when they wrote on how to implement the UnitOfWork pattern in an ASP.NET MVC application.

What I mean is this, in a very simplified way:

public interface IRepository : IDisposable
{
    // Member definition
}

public class MyClass : IDisposable
{
    private readonly IRepository _repository;

    public MyClass(IRepository repository)
    {
        _repository = repository;
    }

    public void Dispose()
    {
        _repository.Dispose();
    }
}

The issue with the above implementation is this little detail:

public void Dispose()
{
    _repository.Dispose();
}

Disposing an externally created dependency is a bad idea and usually leads to severe bugs in an application. MyClass did not create the instance of IRepository, but decided to dispose it on behalf of the creator. However, MyClass has no information on who or what created the instance or for how long it is supposed to live. What if the object has been setup as a Singleton or an instance per HttpRequest? If that would be the case then any code which follows this and relies on an IRepository is broken now. MyClass has crossed its boundaries by interfering with the lifespan of an externally managed dependency.

It doesn't even matter what the current lifespan is. It could have been setup as a transient object, but the point is that a developer should be able to change the lifespan in the future without breaking the entire application. This does not only apply to the repository but to any dependency which is managed outside a class. The rule of thumb is that you cannot dispose an object which you did not create yourself.

The question is what shall we do with dependencies which implement IDisposable?

How to manage a dependency which implements IDisposable?

The intentions were definitely good. The repository implements IDisposable obviously for a good reason. IDisposable is usually used when a class has some expensive or unmanaged resources allocated which need to be released after their usage. Not disposing an object can lead to memory leaks. In fact it is so important that the C# language offers the using keyword to minimize risk of not cleaning up resources in certain edge cases such as the occurrence of an exception. Generally it should be a red flag if an instance of IDisposable has not been wrapped in a using block, which is another good indicator that the code from above has a fundamental problem. I am actually surprised that no one at Microsoft picked this up before publishing the article on the UnitOfWork pattern.

In most cases the creator of an object will be the IoC container. Therefore it should be the IoC container's responsibility to dispose an object after its usage, except that this is very difficult at this stage of an application. Either the object will live too long or too short which brings us back to the initial problem.

Typically if something implements IDisposable we want to dispose it as soon as possible. Often we even want to dispose it way sooner than the lifespan of the class where it has been used, which is yet another indicator that disposing it in the Dispose() method of MyClass is not a good place. This makes me doubt if injecting the repository through the constructor is a good idea at all.

Taking control of creating and disposing an IDisposable

Constructor injection is only one of many IoC patterns which we have to our availability. Another approach would be to use a factory which will issue a new repository on every request:

public interface IRepository : IDisposable
{
    // Member definition
}

public interface IRepositoryFactory
{
    IRepository Create();
}

public class MyClass
{
    private readonly IRepositoryFactory _repositoryFactory;

    public MyClass(IRepositoryFactory repositoryFactory)
    {
        _repositoryFactory = repositoryFactory;
    }

    public string GetSomething()
    {
        using (var repository = _repositoryFactory.Create())
        {
            return repository.GetSomething();
        }
    }
}

The constructor parameter has been changed from an IRepository to an IRepositoryFactory. The instantiation of the repository has been deferred to the actual time of usage and the Dispose() method has been made redundant. Additionally I was able to benefit from the using keyword and overall reduce the total amount of code. This solution is much more elegant, more robust and free of the initial error.

The big difference is that MyClass takes care of creating a repository now. Per definition a factory always creates a new instance, so there is no ambiguity about the object's lifespan. In terms of testability and extensibility nothing has really changed. The factory gets injected through the constructor which makes it easily exchangeable with another implementation or a mock in tests.

Dependency injection is not always the best option

This is a great example where dependency injection is not always the best suited IoC pattern. Dependency injection in all three forms (constructor, property and method injection) is only one of many possible ways of the dependency inversion principle. Factories and other creational design patterns are still useful and have their place in modern software architecture.

Subtle differences like the one from above can determine whether you have a severe bug in your application or not. Sometimes it is not even a question of stylistic preference.

Another good read on the proper use of the IDisposable interface is this amazing StackOverflow answer by Ian Boyd.

SHA-256 is not a secure password hashing algorithm

Mon, 08 Feb 2016 00:00:00 +0000

SHA-256 is not a secure password hashing algorithm. SHA-512 neither, regardless of how good it has been salted. Why not? Because both can be computed in the billions per minute with specialised hardware. If you are surprised to hear that you should continue reading...

What makes a good password hashing algorithm?

A password hash is the very last line of defence. Its only purpose is to prevent an attacker from gaining total control of a user when all other measures of security have been broken. This usually means to prevent the attacker from using the compromised data to access users' data on other websites, which could happen when a user re-uses a password. It is extremely important that a good hashing algorithm will resist all attempts of cracking it, at least for a significant period of time.

Since the attacker is in control of the raw user data there is nothing which can be done to prevent a crude brute force attack. However, this is not an easy undertaking and there are measures which can be put into place to prolong the attack and jeopardise its feasibility.

A good password hashing algorithm removes the slightest chance of a shortcut, leaving a brute force attack as the only attack surface and puts other barriers in place.

One-way functions

First of all this means that a password must always be stored with a cryptographic one-way function. If a password has been encrypted with an algorithm which allows decryption then there is no guarantee that an attacker has not already gained access to the secret key and immediately bypassed all gates of security.

Therefore encryption algorithms such as AES and RSA are not secure storage mechanisms for a password. The use of a one-way hash function is mandatory.

Pre-image and collision attacks

A password hash also needs to resist so called pre-image and collision attacks. In simple words it should not be possible to methodically find a value which can be computed to a given hash value. This crosses out hash functions such as MD5 and SHA-1 which have been proven to be vulnerable to such attacks.

Lookup tables and password salting

Another shortcut are lookup tables. A lookup table is a pre-computed table with hash values derived from commonly used passwords and dictionary entries. An attacker can easily match up a lookup table with the compromised hash values and look up the underlying plain text password. This is where the concept of salting comes into play.

A salt is a piece of text of certain length and complexity which is added to the original value before computing a hash. The idea is that the salt itself is random enough to generate a hash which will not exist in a pre-computed lookup table.

The salt is usually stored in plain text next to the hash value. This is required to allow a genuine login scenario with the original password. It doesn't matter if an attacker can see the salt, because it still invalidates a pre-computed lookup table.

Random salt per user

It is good practice to generate a random salt per user. If the same salt has been shared among all users then an attacker can quickly generate a new lookup table and is back at square one. However, if every user has an individual salt then it becomes significantly more difficult.

Additionally a random salt per user prevents the use of reverse lookup tables. A reverse lookup table is similar to a lookup table except that it matches up the password of multiple users at once. This is possible because many users pick the same (simple) password without knowing it.

Key-stretching algorithms

By using salts and eliminating the possibility of pre-computed lookup tables an attacker is forced to go down the route of a brute force attack. Even though it is extremely difficult it is not impossible. High end hardware with fast GPU can compute billions of hashes per minute.

How can one protect a password from being brute force attacked like this? The idea is to slow down the hashing function. This technique is called key stretching and is a specially crafted algorithm which is very hardware intensive. Such algorithms usually come with an iteration factor which needs to be carefully adjusted to the hardware used on a web server. This is the currently recommended way of storing passwords.

Popular key-stretching algorithms are:

The .NET framework has native built in support for PBKDF2 which comes in form of the Rfc2898DeriveBytes class. There is also an open source library for bcrypt in .NET.

Summary

As you can see good password hashing is more than just sticking a salt at the end of a password and shoving it into the SHA-256 hash function. In practical terms this is as bad as using MD5.

Correct password hashing is not too complicated either, but if it could be avoided all together it would be even better. There is always the option of defering the password handling to a 3rd party by using single sign-on options from trusted authorities such as Google, Facebook or Twitter.

If you have to do it yourself you should follow the guidelines from above and use a key derivation algorithm (=key stretching) in combination with a random salt per user and stick with the native implementation. Don't try to create your own algorithm as you will only get it wrong and end up with a hash function which can be easily parallelised on the CPU and potentially make it even worse.

Last but not least you should always encourage your users to chose strong and unique passwords and not limit them in doing so.

Understanding ASP.NET Core 1.0 (ASP.NET 5) and why it will replace Classic ASP.NET

Wed, 03 Feb 2016 00:00:00 +0000

ASP.NET has quite some years on its shoulders now. Fourteen years to be precise. I only started working with ASP.NET in 2008, but even that is already 8 years ago. Since then the framework went through a steady evolutionary change and finally led us to its most recent descendant - ASP.NET Core 1.0.

ASP.NET Core 1.0 is not a continuation of ASP.NET 4.6. It is a whole new framework, a side-by-side project which happily lives alongside everything else we know. It is an actual re-write of the current ASP.NET 4.6 framework, but much smaller and a lot more modular.

Only recently Scott Hanselman announced the final name will be ASP.NET Core 1.0 after we've got to know it under ASP.NET 5, ASP.NET vNext and Project K.

Some people claim that many things remain the same, but this is not entirely true. Those people mostly refer to MVC 6 which is a whole separate framework which can be plugged into ASP.NET, but doesn't have to. While MVC 6 remains very familiar, ASP.NET Core 1.0 is a big fundamental change to the ASP.NET landscape.

If you are a follower of the live community standups then you might have heard Damian Edwards saying how the team gets a lot of pressure (from above) to get an RTM out of the door. I am not surprised and can understand why ASP.NET Core 1.0 is strategically so important to Microsoft. It is probably a lot more vital to the future of .NET than we might think of it today.

ASP.NET Core 1.0 - What has changed?

A better question would be what has not changed. ASP.NET Core 1.0 is a complete re-write. There is no System.Web anymore and everything which came with it.

ASP.NET Core 1.0 is open source. It is also cross platform. Microsoft invests a lot of money and effort into making it truly cross platform portable. This means there is a new CoreCLR which is an alternative to Mono now. You can develop, build and run an ASP.NET Core 1.0 application either on Mono or the CoreCLR on a Mac, Linux or Windows machine. This also means that Windows technologies such as PowerShell are abandoned from ASP.NET Core 1.0. Instead Microsoft heavily integrates Node.js which can be utilized to run pre- and post build events with Grunt or Gulp.

It is also the reason why things like the .csproj file got replaced by the project.json file and why all the new framework libraries ship as NuGet packages. This was the only way to make development on a Mac a first class citizen.

But Microsoft went even further. Part of a great development experience is the editor of choice. Visual Studio was privileged to Windows users only. With Visual Studio Code Microsoft created a decent IDE for everyone now. Initially it was proprietary software but quickly became open source as well.

There are many more changes being made, but the common theme remains the same. Microsoft is dead serious about going open source and cross platform. Personally I think this is great. All of this is an amazing change and crucial to the long term success of ASP.NET.

ASP.NET Core 1.0 - Why did everything change?

One might wonder why this new direction towards the Mac and Linux community? Why does Microsoft invest so much money in attracting non-Windows developers? Visual Studio Code doesn't cost them anything, it is unlikely that they will use MS SQL server in their projects and there is a high chance that these web applications will end up somewhere on a Linux box in Amazon Web Services or the Google Cloud Platform. After all these are the technologies which non-Windows users are more familiar with.

My guess is that all of this doesn't matter for now. The truth is that an ASP.NET developer who cannot be monetized is still better than a non ASP.NET developer. This is particularly very true if you think that the .NET community is shrinking (relatively). This is just my own speculation, but I think Microsoft fears losing .NET developers, which means they are subsequently losing people who are more willing to pay for other Microsoft products such as MS SQL Server or Microsoft Azure.

If you are a .NET developer you might think this sounds crazy, but think of it from a different angle. Windows Desktop application development is slowly dying. There is no denial to that. Left is the mobile market and the web. Windows phones and tablets are still a drop on a hot stone in comparison to the market shares of iOS and Android. This leaves the web as a last resort. Now the web is an interesting one. After Silverlight's death ASP.NET is the only Microsoft product which competes with other web technologies such as Node, Ruby, Python, Java and more. This is a though battle for ASP.NET, because up until now you had to be a Windows user to be able to develop web applications with ASP.NET.

Lack of portability

In the last few years this problem has become even more prominent with many new languages gaining more popularity and putting ASP.NET into the shadows.

The biggest problem is that the .NET framework and ASP.NET are not cross platform compatible. As a web developer you are writing applications which can be understood by any browser, any OS and any device which is connected to the web. There are no limitations, but with ASP.NET you can only develop from a Windows machine. That doesn't make much sense when you think about it.

This limitation has an impact on the adoption of ASP.NET on several levels. Recruitment is a good example. There is a massive shortage of good software developers at the moment. Ask Ayende how hard it is to recruit a new talent. Imagine how much harder it is if you limit your talent pool to Windows users only? Not only do you waste more time and resources on the recruitment process itself, but also have to pay higher salaries for developers where the demand is higher than the supply.

It can be difficult for companies which are heavily committed to the .NET stack to change directions now, but what about startups? Many of today's biggest internet businesses were born out of small startups. They use free open source technologies such as PHP, Ruby, Python, Java or Node.js. This has a double negative effect for Microsoft. Not only did they lose the opportunity to sell ASP.NET, but they also send out the message that if you want to build a successful business you pick an open stack over proprietary software.

ASP.NET is probably one of the feature richest and fastest technologies you can find, but why would a startup care about this in the beginning? If they do well they can deal with this stuff later and if it doesn't go well then its good they didn't have to pay for a Microsoft license, right?

Chasing behind innovation

Another major implication of not being cross platform compatible is that current ASP.NET 4.6 developers are missing out on big innovations which are not immediately available on the Windows platform. Over the last years Microsoft was chasing after many innovations by providing its own version to the .NET community, but not always with success (Silverlight, AppFabric Cache, DocumentDb, Windows Phone, etc.). This is not a sustainable model.

As a result many ASP.NET developers live in silos today. We are at a point where Microsoft cannot keep up with the vast amount of technology anymore and ASP.NET developers miss out on big innovations such as containers and Docker and don't even realize it, because they know very little to nothing about it. This is a dangerous place to be.

Cross platform compatibility is more than just a fad. It is the key to innovation today and the only way to stay on top of the game!

So how does Classic ASP.NET fit into this new world? Not much to be honest. ASP.NET 4.6 has a really though time to keep up with this fast moving environment.

Except we have ASP.NET Core 1.0 now...

ASP.NET Core 1.0 - Reviving ASP.NET

This is where ASP.NET Core 1.0 comes into the limelight. It is built on the same core principles which helped other languages to popularity:

Free and open source
Cross platform compatible
Ease of access
Thin, fast, modular and extensible

On the plus side ASP.NET Core 1.0 can be developed with some of the greatest languages available right now, thinking of C# and F# in particular! This will stick out ASP.NET Core from other competitive frameworks.

What will happen to ASP.NET 4.6? I don't know, but I would argue that ASP.NET 4.6 is a dead horse in the long run. There is very little value in betting any more money on it. Microsoft wouldn't say this yet, but it is pretty obvious. ASP.NET Core 1.0 is the new successor and the only viable solution to address the aforementioned problems.

ASP.NET 4.6 will be soon remembered as Classic ASP.NET. It will not entirely disappear, just like Classic ASP has never fully disappeared, but new development will likely happen in ASP.NET Core going forward. I find it extremely exciting and the benefits of ASP.NET Core are too compelling to not switch over as soon as possible.

The only thing we need to hope for is that Microsoft will not become impatient now and mess up the release with an immature product which will cause more churn than attraction. Microsoft, please take the time to bake something to be proud of!

ASP.NET 5 like configuration in regular .NET applications

Tue, 12 Jan 2016 00:00:00 +0000

ASP.NET 5 is Microsoft's latest web framework and the new big thing on the .NET landscape. It comes with a whole lot of new features and other changes which makes it very distinctive from previous ASP.NET versions. It is basically a complete re-write of the framework, optimized for the cloud and cross platform compatible. If you haven't checked it out yet then you are definitely missing out!

Some of the newly introduced features are the new configuration options, which come as part of the Microsoft.Extensions.Configuration NuGet packages. They allow you a more flexible way of loading configuration values into an application and make it significantly easier to move an application across different environments without having to change configuration files or run through nasty configuration transformations as part of the process. Scott Hanselman has nicely summarized this topic in one of his recent blog posts.

It is needless to say that a lot of people are hugely excited about the new framework and many projects are being written in ASP.NET 5 right now, but there is also a significant amount of people who cannot easily migrate their existing projects to the new framework yet. This might be for various reasons but essentially means they are stuck on an older version of ASP.NET at least for a while.

However, it doesn't mean that those people cannot already benefit from some of the new ideas which have been publicly introduced in ASP.NET 5 such as the new configuration options.

Implementing ASP.NET 5 like configuration options

The idea is to provide multiple sources from where an app can load configuration values. Additionally we want to put those sources into an order of precedence, where one source overrules another.

While ASP.NET 5 implementes a Builder pattern, you can easily achieve the same thing with a good old Decorator.

First we need an interface which provides a method to retrieve a config value:

public interface IConfiguration
{
    string Get(string key);
}

Next I implement a simple class which loads a value from an app- or web.config file:

using System.Configuration;

public class AppConfigConfiguration : IConfiguration
{
    public string Get(string key)
    {
        return ConfigurationManager.AppSettings[key];
    }
}

This is trivial and probably the same as what you have in your current application, just that it has been wrapped in a class which is behind an interface.

Finally we can provide an additional implementation which loads a value from the environment variables:

using System;

public class EnvironmentVariablesConfiguration : IConfiguration
{
    private readonly IConfiguration _backupConfiguration;

    public EnvironmentVariablesConfiguration(
        IConfiguration backupConfiguration)
    {
        _backupConfiguration = backupConfiguration;
    }

    public string Get(string key)
    {
        var value = Environment.GetEnvironmentVariable(key);
        return value ?? _backupConfiguration.Get(key);
    }
}

This particular implementation wraps another IConfiguration implementation. If the setting does not exist in the environment variables then it defers to the next implementation. This could be anything and in this particular example will be the AppConfigConfiguration:

container.Register<IConfiguration>(
    new EnvironmentVariablesConfiguration(
        new AppConfigConfiguration()));

It is really as simple as that. The order of precedence is determined by the order of the classes being put together.

This pattern can be extended as far as you like. For example I could add two more classes and compose something like this:

container.Register<IConfiguration>(
    new EnvironmentVariablesConfiguration(
        new JsonFileConfiguration(
            new AzureTableStorageConfiguration(
                new AppConfigConfiguration()))));

Now I can access configuration values from other classes through an IConfiguration dependency:

var value = _configuration.Get("SomeKey");

With this trick you can easily implement a "cloud optimized" configuration in any version of ASP.NET and follow good practice patterns no matter where you code!

UPDATE:

I created a complete example of how this might look like in an ASP.NET 4.6 MVC 5 application and uploaded it to GitHub.

In this example I provide 3 configuration sources and two different ways of composing them together. Once via the Decorator pattern as shown in this blog post and another one using a fluent Builder pattern which is much more similar to the way it is done in ASP.NET Core.

Running NancyFx in a Docker container, a beginner's guide to build and run .NET applications in Docker

Thu, 07 Jan 2016 00:00:00 +0000

The quiet Christmas period is always a good time to explore new technologies and recent trends which have been on my list for a while. This Christmas I spent some time learning the latest ASP.NET framework, in particular how to run ASP.NET 5 applications on Linux via the CoreCLR and how to run a regular .NET 4.x web application via Mono in a Docker container. The latter is what I am going to talk about in this blog post today.

What is Docker?

I assume you have some basic knowledge of what Docker is, how it revolutionized the way we ship software into the cloud and what the benefits are of a container over a VM. If anything of this doesn't make sense, then I would highly recommend to make yourself familiar with the basic concept of containers and why it is desirable to run applications in a container first.

A few good resources to get you started are:

Setting up Docker on Windows

First I want to get Docker running locally so I can run and debug applications in a development environment. Luckily this has been made extremely easy for us. All I need is to download the Docker Toolbox for Windows and follow the instructions.

Docker Toolbox

After installation I will have three new applications:

VirtualBox
Kitematic
Docker Quickstart Terminal

If you have VirtualBox already installed then the installer will skip over this step. The important thing to know is that VirtualBox has an external API which can be used by other applications to manage VMs automatically. This is exactly what the Docker Machine does. It will create a new VM in VirtualBox with an image which has everything you need to run Docker there. Because it is all automated you never really have to worry about VirtualBox yourself.

Kitematic is a GUI client around the Docker Machine. At the moment it is very limited in functionality and therefore you will not need it either.

This leaves the Docker Terminal as the last application and the only thing which we will be using to run and manage Docker containers in a local environment.

Run your first Docker command from the Terminal

After a successful installation let's run a first Docker command to see if things generally work. When you open the terminal for the first time it will initialize the VM in VirtualBox. This may take a few seconds but eventually you should end up at a screen like this:

You don't have to open Kitematic or VirtualBox to get it running. As I said before, you can happily ignore those two applications, however, if you are curious you can look into VirtualBox and see the VM running as expected:

It's a Linux box loaded from the boot2docker.iso.

Back to the terminal I can now type docker version to get some basic version information about the Docker client and server application:

With that I am good to go with Docker now.

Maybe one thing which is worth mentioning at this point is the initial message in the Docker Terminal:

The IP address which is shown in the terminal is the endpoint from where you can reach your application later in this tutorial.

Creating a NancyFx web application for Docker

Now it is time to actually create a .NET web application which can run on Mono.

First I create a new project using the template for a regular console application, targeting .NET Framework 4.6.1.

The project is entirely empty except the Program.cs file:

class Program
{
    static void Main(string[] args)
    {
    }
}

Next I have to install 3 NuGet packages:

Install-Package Nancy
Install-Package Nancy.Hosting.Self
Install-Package Mono.Posix

The first package installs the NancyFx web framework. Nancy is a lightweight .NET framework for building HTTP based services. You can think of it like a counterpart of ASP.NET, except it has nothing to do with ASP.NET, IIS or the System.Web namespace.

You can still host Nancy applications on IIS, but you can equally host it somewhere else like a console application. This is exactly what we will do and why we install Nancy.Hosting.Self as the second package.

The third package installs the POSIX interface for Mono and .NET.

Having the Nancy packages installed I can now configure an endpoint and start a new Nancy.Hosting.Self.NancyHost:

using System;
using Nancy.Hosting.Self;

class Program
{
    static void Main(string[] args)
    {
        const string url = "http://localhost:8888";

        var uri = new Uri(url);
        var host = new NancyHost(uri);

        host.Start();
    }
}

This console application will exit immediately after launching and therefore I need to add something to keep it open such as a Console.ReadLine() command. Additionally I want to stop the host when I know the application is going to shut down:

host.Start();
Console.ReadLine();
host.Stop();

If I would want to run this on Windows then I would be done now, but on Linux I want to wait for Unix termination signals instead.

A way to detect if the application is running on Mono is with this little helper method:

private static bool IsRunningOnMono()
{
    return Type.GetType("Mono.Runtime") != null;
}

Another helper method exposes the Unix termination signals:

private static UnixSignal[] GetUnixTerminationSignals()
{
    return new[]
    {
        new UnixSignal(Signum.SIGINT),
        new UnixSignal(Signum.SIGTERM),
        new UnixSignal(Signum.SIGQUIT),
        new UnixSignal(Signum.SIGHUP)
    };
}

I add both methods to my Program class and change the Main method to support both, Windows and Unix termination:

host.Start();

if (IsRunningOnMono())
{
    var terminationSignals = GetUnixTerminationSignals();
    UnixSignal.WaitAny(terminationSignals);
}
else
{
    Console.ReadLine();
}

host.Stop();

This is what the final class looks like:

using System;
using Nancy.Hosting.Self;
using Mono.Unix;
using Mono.Unix.Native;

class Program
{
    static void Main(string[] args)
    {
        const string url = "http://localhost:8888";

        Console.WriteLine($"Starting Nancy on {url}...");

        var uri = new Uri(url);
        var host = new NancyHost(uri);
        host.Start();

        if (IsRunningOnMono())
        {
            var terminationSignals = GetUnixTerminationSignals();
            UnixSignal.WaitAny(terminationSignals);
        }
        else
        {
            Console.ReadLine();
        }

        host.Stop();
    }

    private static bool IsRunningOnMono()
    {
        return Type.GetType("Mono.Runtime") != null;
    }

    private static UnixSignal[] GetUnixTerminationSignals()
    {
        return new[]
        {
            new UnixSignal(Signum.SIGINT),
            new UnixSignal(Signum.SIGTERM),
            new UnixSignal(Signum.SIGQUIT),
            new UnixSignal(Signum.SIGHUP)
        };
    }
}

All I am missing now is at least one Nancy Module which serves HTTP requests. This is done by implementing a new module which derives from Nancy.NancyModule and registering at least one route. I setup a "Nancy: Hello World" message on the root / endpoint and an OS version string on the /os endpoint:

using System;
using Nancy;

public class IndexModule : NancyModule
{
    public IndexModule()
    {
        Get["/"] = _ => "Nancy: Hello World";
        Get["/os"] = _ => Environment.OSVersion.ToString();
    }
}

If I compile and run the application then I should be able to see the hello world message when visiting http://localhost:8888 and see the OS version at http://localhost:8888/os:

Running NancyFx in a Docker container

The application is very simple but certainly enough to deploy the first version in a Docker container.

Create a Dockerfile

First I need to build a Docker image which will contain the entire application and all of its dependencies. For this I have to create a recipe which defines what exactly goes into the image. The recipe is a Dockerfile, an ordinary human readable text file with instructions on how to compose an image. It is important to name the file exactly as shown, without a file extension and a capital "D".

It is good practice to add the Dockerfile into your project folder, because it may change when your project changes:

I also want to include the Dockerfile in the build output, therefore I have to change the "Build Action" setting to "Content" and "Copy to Output Directory" to "Copy always":

Visual Studio 2015 creates text files with UTF-8-BOM encoding by default. This adds an additional (invisible) BOM character at the very beginning of the text file and will cause an error when trying to build an image from the Dockerfile. The easiest way to change this is by opening the file in Notepad++ and changing the encoding to UTF-8 (without BOM):

You can also permanently change Visual Studio to save files without BOM.

Now that this is sorted I can open the file and start defining the build steps.

Every Dockerfile has to begin with the FROM instruction. This defines the base image to start with. Docker uses a layering system which is one of the reasons why Docker images are so light. You can find many official images to start with at the public Docker Hub.

Fortunately there is already an official Mono repository which we can use. The most recent image is 4.2.1.102 at the time of writing. As you can see the Mono image itself has the debian:wheezy image from the official Debian repository as its base. The Debian image has the empty scratch image as its base. When we use the Mono image we essentially build a new layer on top of an existing tree:

scratch
   \___ debian:wheezy
       \___ mono:4.2.1.102
           \___ {our repository}:{tag}

If you look at the official Mono repository you can see that the latest Mono image has multiple tags:

It depends on your use case which tag makes the most sense for your application. Currently they all have been built from the same Dockerfile, but only tag 4.2.1.102 is explicit enough to always guarantee the exact same build. Personally I would chose this one for a production application:

FROM mono:4.2.1.102

The next two instructions are very straight forward. I want to create a new folder called /app and copy all relevant files, which are required to execute the application, into this folder. Remember that the Dockerfile gets copied into the build output folder. This means that I basically have to copy everything from the same directory where the Dockerfile sits into the /app folder:

RUN mkdir /app
COPY . /app

My Nancy application has been configured to listen to port 8888. With the EXPOSE instruction I inform Docker that the container listens to this specific port:

EXPOSE 8888

Finally I have to run the application with Mono:

CMD ["mono", "/app/DockerDemoNancy.Host.exe", "-d"]

This is what the final Dockerfile looks like:

FROM mono:4.2.1.102
RUN mkdir /app
COPY . /app
EXPOSE 8888
CMD ["mono", "/app/DockerDemoNancy.Host.exe", "-d"]

There is a lot more you can do with a Dockerfile. Check out the Dockerfile reference for a complete list of available instructions.

Build a Docker image

Building a Docker image is extremely easy. Back in the Docker Terminal I navigate to the /bin/Release/ folder of my Nancy application:

cd /c/github/docker-demo-nancy/dockerdemonancy.host/bin/release

Next I run the docker build command and tag the image with the -t option:

docker build -t docker-demo-nancy:0.1.0 .

Don't forget the dot at the end. This is the path to the directory which contains the Dockerfile. Because I already navigated into the /bin/Release/ folder I just put a dot at the end.

The build process will go through each instruction and create a new layer after executing it. The first time you build an image you are likely not going to have the mono:4.2.1.102 image on disk and Docker will pull it from the public registry (Docker Hub):

As you can see the FROM instruction requires Docker to download 6 different images. This is because the mono:4.2.1.102 image and all of its ancestors (debian:wheezy) have 6 instructions in total, which result in 6 layered images.

A better way of visualizing this is by inspecting our own image.

Once the build is complete we can list all available images with the docker images command:

With docker history {image-id} I can see the entire history of the image, each layer it is made of and the command which is responsible for the layer:

This is quite clever! Anyway, I am getting carried away here, the point is we just created our first Docker image!

If you want to upload the image into a repository on Docker Hub or another private registry you can use docker tag to tag an existing image with a new tag and docker push to upload it to the registry.

Create and run a Docker container

Running a Docker container couldn't be easier. Use the docker run command to create and run a container in one go:

docker run -d -p 8888:8888 docker-demo-nancy:0.1.0

The -d option tells Docker to run the container in detached mode and the -p 8888:8888 option maps the container's port 8888 to the host's port 8888.

Afterwards you can run docker ps to list all currently running containers:

Great, now pasting {docker-ip}:8888 (the IP address from the beginning) into a browser should return the Nancy hello world message:

And going to {docker-ip}:8888/os should return "Unix 4.1.13.2":

This is pretty awesome. With almost no effort we managed to run a Nancy .NET application on Mono in a Docker container!

Tip: map the Docker IP address to a friendly DNS

You can map the Docker IP address to a friendly DNS by editing your Windows hosts file:

Open C:\Windows\System32\drivers\etc\hosts as an administrator
Add a new mapping to a memorable DNS, e.g: 192.168.99.100 docker.local
Save the file

Now you can type docker.local:8888 into your browser and get the same result:

Configure environment specific settings with Docker

The last thing I would like to show in this blog post is how to manage environment specific variables with a Docker container.

I think it is pretty obvious that you must never change a Docker image when you promote your Docker container from one environment to another. This means that the app.config which has been packed into the image must be the same for every environment. Even though this is not a new practice I still see a lot of people transforming config files between environments. This has to stop and Docker makes it easy to load environment variables when launching a container.

Let's make a small change to the Nancy IndexModule:

public IndexModule()
{
    var secret = Environment.GetEnvironmentVariable("Secret");

    Get["/"] = _ => "Nancy: Hello World";
    Get["/os"] = _ => Environment.OSVersion.ToString();
    Get["/secret"] = _ => secret ?? "not set";
}

It is a fairly straight forward change. I load an environment setting with the name "Secret" into a local variable and expose it later.

This environment setting could be anything, but typically it includes sensitive data like encryption keys, database connection strings or other environment specific settings such as error log paths.

Needles to say that exposing the secret to the public is only for the purpose of this demo to show that it works.

Now I need to compile the application and build a new Docker image again, following the same instructions as before. I tagged the new image with docker-demo-nancy:0.2.0.

Before I launch a new container I want to stop the current one to avoid a clash on port 8888, otherwise I would happily run them side by side.

After I ran docker stop {container-id} I launch a new container with:

docker run -d -p 8888:8888 -e Secret=S3cReT docker-demo-nancy:0.2.0

The docker run command takes in one or many -e options to specify environment settings. There are a few more options on specifiying environment settings, but the only one which you would ever want to use in a live environment is the --env-file option to load all environment variables from an external file.

This has many advantages:

You can easily ship environment settings to environments
You can easily provide many environment settings
Sensitive data will not show up in logs
The path to the file can be static which makes it easier to configure a scheduler to run containers in production

After launching the container with the secret setting I can run docker inspect {container-id} to load a whole bunch of information on the container. One piece of information is the environment variables which have been loaded for that container:

Going to docker.local:8888/secret will expose the secret environment variable now:

Recap

This brings me to the end of my first blog post on running .NET applications in Docker. I hope I could shed more light on some of the Docker basics and demonstrate how quick and easy you can build .NET applications for Docker.

For this demo I chose the NancyFx framework to build a web application, but I could have equally written a regular .NET application which can run on Mono or used ASP.NET 5 which does not only run on Mono but also on the new CoreCLR which is cross platform compatible.

Obviously there is a lot more that comes into running .NET apps in Docker which I haven't covered in this blog post. Some of these things are debugging applications in a Docker container, building Docker images from your CI and managing containers in production. Watch out for further blog posts where I will drill down into some of those topics!

The full source code of the demo application can be found on GitHub. I have also uploaded the Docker images to my public repository on Docker Hub.

Diagnosing CSS issues on mobile devices with Google Chrome bookmarklets

Thu, 31 Dec 2015 00:00:00 +0000

Yesterday, when I was browsing my blog on my mobile phone I discovered a small CSS issue on one of the pages. One of my recent blog posts had a horizontal scrollbar which shouldn't have been there. A page element caused an overflow, but it was not visible which element was responsible for it.

When I tried to diagnose the issue on my computer I struggled to replicate it to the same extent as it was present on my mobile phone. I was too lazy to go through the entire page source to search for the needle in the haystack and decided to quickly create a Google Chrome bookmarklet to help me with the investigation.

First I had to make the invisible visible.

With a little bit of Google's help and playing around in the Google Chrome Console (Ctrl + Shift + J) I put this little JavaScript snippet together:

[].forEach.call(
    document.getElementsByTagName("*"),
    function(e) {
        e.style.outline = "1px solid red";
});

When I execute this in the console it will outline every element on the page:

With this it should be easy to spot the overflowing element. Now I had to find a way to execute it inside the mobile version of Google Chrome.

For this purpose I created a new bookmarklet on my Desktop, which then got automatically synchronised to my phone.

A bookmarklet in Google Chrome allows you to execute JavaScript code on an already rendered page.

This is how you create it:

Ctrl + Shift + O in Google Chrome
Right click "Bookmarks bar" on the left
Click Add Page
Pick a random name, e.g.: Outline Elements
Type in the following snippet:

javascript: [].forEach.call(document.getElementsByTagName("*"), function(e) {e.style.outline = "1px solid red";});
Done!

When I click on the newly created bookmarklet I will get the same result as if I had executed the snippet in the console.

Only seconds later it appeared on my phone as well:

However, when I click on the bookmarklet from the bookmarks menu on my phone everything freezes and nothing happens. It turns out that I have to execute it from the address bar.

Just start typing the name of your bookmarklet and Google Chrome will auto-suggest the item for you:

Executing it from the address bar delivers the correct result:

This little trick quickly helped me to find the overflowing element on my phone without having to modify the original website. I use the same technique to remove advertising banners and other blocking content on several websites which normally don't display the entire content when you are not logged in (or a paying customer).

Design, test and document RESTful APIs using RAML in .NET

Wed, 23 Dec 2015 00:00:00 +0000

Building a RESTful API is easy. Building a RESTful API which is easy to consume is more difficult. There are three key elements which make a good API:

Intuitive design
Good documentation
Documentation which actually matches the implementation

Intuitive API design is obviously very important, but equally important and often neglected is good and complete documentation which makes it easier to build against your API.

Having said that we all know how difficult it is to keep documentation up to date. It is very loosely coupled to the actual implementation without any enforcement other than a human trial and error process.

In this article I would like to demonstrate how we can close this gap by building a RESTful API with a design- and test first approach using RAML.

I will showcase an entire end to end scenario by building a simple demo API and covering the following steps:

But first let me briefly introduce you to RAML:

RAML

RAML stands for RESTful API Modeling Language and this is exactly what it delivers.

If you have worked with Swagger or API Blueprint before then this should be familiar, except that RAML is designed to be human readable and remarkably easy to use.

At the time of writing there are two public specifications:

In this blog post I will be using RAML 0.8 and assume that you are familiar enough with the spec to follow my simple examples as part of the demo.

For further reading I would recommend to go through the official RAML tutorials explaining the basic concepts and more advanced features in your own time and at your own pace:

Now that you know what RAML is I will jump straight into the first part where I'll be using RAML to design an API:

1. Design an API using RAML

As I mentioned earlier, for the purpose of this demo I would like to build a very rudimental fake parcel delivery API with the following endpoint:

GET /status/{parcelId} will return the status of a parcel
PUT /status/{parcelId} will update the status of a parcel

RAML is a YAML based language and designed for human readability. The beauty of this is that you can write RAML in a basic editor without fancy syntax highlighting and it will be still easy to read and understand.

However, there is a good Atom plugin called API Workbench which I am using to kick start my API:

#%RAML 0.8
title: Parcel Delivery API
version: v1
baseUri: http://localhost/raml-demo-api/{version}
protocols: [ HTTP, HTTPS ]

Sidenote: I will not go through every single line of the sample code, but provide enough context so that it should be easy to follow.

At the top of the document I specified the RAML version, followed by the title of the API, the API version and the basic URI with a version placeholder. This allows me to introduce breaking changes in the future. The API shall also be called from both protocols, HTTP and HTTPS.

Next I define a single endpoint to set and get status information for a given parcel ID:

...
/status/{parcelId}:
  displayName: Parcel Status Information
  uriParameters:
    parcelId:
      displayName: Parcel ID
      type: string
      required: true
      minLength: 6
      maxLength: 6
      example: 123456

After this I define the contract for the GET operation, which shall return the current status of a parcel:

...
  get:
    description: Retrieves the current status for the specified parcel ID.
    responses:
      200:
        description: Current status.
        body:
          application/json:
            schema: |
              {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "title": "Delivery Status",
                "type": "object",
                "properties": {
                  "status": {
                    "description": "The current status of the delivery.",
                    "type": "string"
                  },
                  "updated": {
                    "description": "The date time the last status update.",
                    "type": "string"
                  }
                }
              }
            example: |
              {
                "status": "Parcel is out for delivery.",
                "updated": "2015-12-09T16:53:19.5168335+00:00"
              }

As you can see there is probably not much I have to explain. The GET operation returns a successful response with the 200 HTTP status code and a JSON payload containing the current status. Note how RAML allows me to provide an example alongside the schema. This will be particularly useful at a later point.

I am pleased with that and apply something similar for the PUT verb:

...
  put:
    description: Creates or updates the status for the specified parcel ID.
    body:
      application/json:
        schema: |
          {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "title": "Status Update",
            "type": "object",
            "properties": {
              "status": {
                "description": "The new status update message.",
                "type": "string"
              }
            }
          }
        example: |
          {
            "status": "Delivered and signed by customer."
          }
    responses:
      201:
        description: The status has been successfully updated.

The only difference is that the PUT operation expects a JSON object in the HTTP body and returns the 201 status code on success.

Now I'm almost done with designing my API. The only thing missing is to describe what happens if a consumer provides an invalid parcel ID. It might either not exist or be in the wrong format.

Because this applies to both operations, the GET and the PUT on my endpoint, I will define a trait with two additional error responses, which can be shared by multiple endpoints in RAML:

...
traits:
  - requiresValidParcelId:
      usage: |
        Apply this to any method which requires a valid Parcel ID in the request.
      responses:
        406:
          description: Parcel ID was in incorrect format.
          body:
            application/json:
              schema: ErrorMessage
              example: |
                {
                  "message": "Parcel ID has to be 6 characters long and may only contain digits."
                }
        404:
          description: Could not find the specified parcel ID.
          body:
            application/json:
              schema: ErrorMessage
              example: |
                {
                  "message": "Parcel ID not found."
                }

Every trait has a unique name. In this case I called it requiresValidParcelId.

Perhaps you noticed the schema: ErrorMessage in the declaration of the response body. Shared schemas is another feature of RAML. I created an ErrorMessage schema to describe the JSON payload of an error response:

...
schemas:
  - ErrorMessage: |
      {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Error Message",
        "type": "object",
        "properties": {
          "message": {
            "description": "The error message of the error.",
            "type": "string"
          }
        }
      }

Overall it is the same principle as to how I defined the success response in my previous example, except that this one is declared in a separate section which allows multiple re-use inside RAML.

At last I need to hook up my endpoint with the trait:

...
/status/{parcelId}:
  displayName: Parcel Status Information
  uriParameters:
    parcelId:
      displayName: Parcel ID
      type: string
      required: true
      minLength: 6
      maxLength: 6
      example: 123456
  is: [ requiresValidParcelId ]

The entire end result looks as follows:

#%RAML 0.8
title: Parcel Delivery API
version: v1
baseUri: https://raml-demo-api.azurewebsites.net/{version}
protocols: [ HTTP, HTTPS ]
schemas:
  - ErrorMessage: |
      {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Error Message",
        "type": "object",
        "properties": {
          "message": {
            "description": "The error message of the error.",
            "type": "string"
          }
        }
      }
traits:
  - requiresValidParcelId:
      usage: |
        Apply this to any method which requires a valid Parcel ID in the request.
      responses:
        406:
          description: Parcel ID was in an incorrect format.
          body:
            application/json:
              schema: ErrorMessage
              example: |
                {
                  "message": "Parcel ID has to be 6 characters long and may only contain digits."
                }
        404:
          description: Could not find the specified parcel ID.
          body:
            application/json:
              schema: ErrorMessage
              example: |
                {
                  "message": "Parcel ID not found."
                }
/status/{parcelId}:
  displayName: Parcel Status Information
  uriParameters:
    parcelId:
      displayName: Parcel ID
      type: string
      required: true
      minLength: 6
      maxLength: 6
      example: 123456
  is: [ requiresValidParcelId ]
  get:
    description: Retrieves the current status for the specified parcel ID.
    responses:
      200:
        description: Current status.
        body:
          application/json:
            schema: |
              {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "title": "Delivery Status",
                "type": "object",
                "properties": {
                  "status": {
                    "description": "The current status of the delivery.",
                    "type": "string"
                  },
                  "updated": {
                    "description": "The date time the last status update.",
                    "type": "string"
                  }
                }
              }
            example: |
              {
                "status": "Parcel is out for delivery.",
                "updated": "2015-12-09T16:53:19.5168335+00:00"
              }
  put:
    description: Creates or updates the status for the specified parcel ID.
    body:
      application/json:
        schema: |
          {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "title": "Status Update",
            "type": "object",
            "properties": {
              "status": {
                "description": "The new status update message.",
                "type": "string"
              }
            }
          }
        example: |
          {
            "status": "Delivered and signed by customer."
          }
    responses:
      201:
        description: The status has been successfully updated.

You can also explore the full RAML document in my RAML-Demo GitHub repository.

RAML has a lot more to offer than what I showed in this basic example, but hopefully this gives you a rough idea of how intuitive and powerful it can be!

Additionally there is a lot of useful tooling built around RAML. One of my favourites is the RAML Tools for .NET, which brings me to the second part of this blog post.

2. Generate a client from the RAML document

Now that I have a detailed specification of what my API should look like it is time to open up Visual Studio and get my hands dirty.

First I create an empty test project and include the RAML file (api.raml) in a solution folder to keep everything together:

For the next part I have to install the RAML Tools for .NET Visual Studio extension:

After a successful install I have an additional context menu when I right click the "References" item underneath my test project:

A click on that menu item pops up a pretty much self-explaining dialog:

I select the Upload option and navigate to the api.raml inside my solution folder. After confirmation I am presented with an Import RAML dialog:

The import process automatically detected my single endpoint and the only thing I had to change was the default client name to "ParcelDeliveryApiClient" in case I want to import another API at a later point.

Hitting the Import button finishes the remaining work and once completed I am seeing a new API reference in my project tree:

This was a very smooth and painless process and if successfully imported I should be able to create an instance of ParcelDeliveryApiClient in a new class file:

Amazing, let's explore the auto-generated client by writing some tests in the next step!

3. Write tests using the auto-generated client

With the ParcelDeliveryApiClient I can write integration tests against a real endpoint. At the moment I don't have a working API running anywhere so I define a provisional URI and a couple more parameters for my first test:

[TestFixture]
public class ParcelDeliveryApiTests
{
    [Test]
    public async Task IntegrationTest()
    {
        const string endpoint = "http://localhost/raml-demo-api/v1/status";
        const string parcelId = "123456";
        const string status = "Parcel is out for delivery.";
    }
}

Next I initialize the client and send a PUT request for the parcel ID 123456:

[Test]
public async Task IntegrationTest1()
{
    const string endpoint = "http://localhost/raml-demo-api/v1/status";
    const string parcelId = "123456";
    const string status = "Parcel is out for delivery.";

    var parcelDeliveryApiClient = new ParcelDeliveryApiClient(endpoint);

    var putResult =
        await parcelDeliveryApiClient.StatusParcelId.Put(
            new StatusParcelIdPutRequestContent
            {
                Status = status
            },
            parcelId);
}

The amazing thing is that the entire client code has been auto-generated during the import. Every member of the class such as the StatusParcelId or the Put(...) method, as well as the StatusParcelIdPutRequestContent DTO class have been auto-magically created for me.

Remember, all I have done so far was to describe my API using RAML and with a few additional clicks in Visual Studio I am now able to write full fletched integration tests against an API which doesn't even exist yet!

I find this pretty cool.

The expected result is HTTP status code 201:

[Test]
public async Task IntegrationTest()
{
    const string endpoint = "http://localhost/raml-demo-api/v1/status";
    const string parcelId = "123456";
    const string status = "Parcel is out for delivery.";

    var parcelDeliveryApiClient = new ParcelDeliveryApiClient(endpoint);

    var putResult =
        await parcelDeliveryApiClient.StatusParcelId.Put(
            new StatusParcelIdPutRequestContent
            {
                Status = status
            },
            parcelId);

    Assert.AreEqual(HttpStatusCode.Created, putResult.StatusCode);
}

For this integration test I'd like to add one more check:

[Test]
public async Task IntegrationTest()
{
    const string endpoint = "http://localhost/raml-demo-api/v1/status";
    const string parcelId = "123456";
    const string status = "Parcel is out for delivery.";

    var parcelDeliveryApiClient = new ParcelDeliveryApiClient(endpoint);

    var putResult =
        await parcelDeliveryApiClient.StatusParcelId.Put(
            new StatusParcelIdPutRequestContent
            {
                Status = status
            },
            parcelId);

    Assert.AreEqual(HttpStatusCode.Created, putResult.StatusCode);

    var getResult = await parcelDeliveryApiClient.StatusParcelId.Get(parcelId);

    Assert.AreEqual(HttpStatusCode.OK, getResult.StatusCode);
    Assert.AreEqual(status, getResult.Content.StatusParcelIdGetOKResponseContent.Status);
}

After the PUT I am firing a GET with the same parcel ID and expect another successful response with the updated status.

When I run this test it will fail on the first assertion, because at the moment there is nothing behind the endpoint yet, but I am going to fix this very soon.

This test obviously doesn't cover everything from the RAML document, but at this point it should be clear that with the auto-generated client I can test every single aspect of my API without having to write any client code myself.

Coupling the RAML file to the API

So how is this better than a normal integration test? The key benefit is that the client is a 1:1 replica of the RAML file. If the API changes I will have to update my tests, which requires me to update the client, which subsequently forces me to update the RAML first.

Besides that it took me only 10 seconds to generate a perfect abstraction of my API which can be used for more than just writing tests.

4. Implement the API to satisfy the tests

I have to admit this part has very little to do with RAML, but I thought it would be great to provide a full end to end example as part of this blog post.

For that reason I'll make it short and fast forward the implementation to the point where it will satisfy the test from above:

public StatusModule() : base("/v1/status")
{
    Get["/{parcelId}"] = ctx =>
    {
        if (!Statuses.ContainsKey(ctx.parcelId))
            return HttpStatusCode.NotFound;

        return new JsonResponse(
            Statuses[ctx.parcelId],
            new DefaultJsonSerializer());
    };

    Put["/{parcelId}"] = ctx =>
    {
        dynamic obj = JsonConvert.DeserializeObject(Request.Body.AsString());

        Statuses[ctx.parcelId] = new StatusInformation
        {
            Status = obj.status.Value,
            Updated = DateTime.UtcNow.ToString("o")
        };

        return HttpStatusCode.Created;
    };
}

This (quick and dirty) snippet doesn't implement the entire API, but enough to make the test go green.

After cheating myself through step 4 let's move on to the next and final part of this article and look at another important criteria of building a RESTful API.

5. Document and review an API using the Anypoint Platform

Before going any further let's quickly recap what I've done so far:

I designed an API using RAML
Used the RAML Tools for .NET to auto-generate a client
Wrote an integration test with the generated client
Implemented enough of the API to satisfy my test

It feels like I am almost done. So what about documentation? Well RAML is already human readable, it is accurate and tested against my actual API and all my tests are passing as well, so am I actually done?

No, not yet, there are two more things I have to solve:

At the moment the RAML document is only in my source control and likely not accessible to external consumers
Integration tests are great to give me confidence, but my stakeholders don't understand those green and red lights in Visual Studio (or CI system) and likely want to verify the API for themselves

This is where I find the Anypoint Platform extremely useful!

Anypoint Platform

Among many other features Anypoint allows me to document and publish my API with an interactive designer (much like the API Workbench but even richer) and create a live Portal at no cost.

The designer is exceptionally well done. It offers many features like syntax highlighting, intellisense, instant RAML validation and auto-suggestion of available nodes:

Another brilliant feature is the interactive preview when editing a RAML file. It visually displays every characteristic of your API in a beautiful interface, like those responses as an example:

It even goes as far as allowing me to interact with a mocked service while working on the RAML:

When I click the Try It button it displays me a form with all relevant parameters pre-populated with the values from the examples in my RAML:

From the UI I can quickly run requests against my API with as little friction as possible.

Finally you can publish your API into a Live Portal which is publicly accessible (can be private as well), where users and stakeholders can try the API in a live environment.

Try it yourself by executing some PUT and GET requests via the Live Portal of my demo API.

Any technical or non-technical person can review the API and validate if it works as expected and I as a developer cannot claim that a feature is done before it is available in the Live Portal.

This is how I envision API development in an agile environment.

How to make an IStatusCodeHandler portable in NancyFx

Wed, 25 Nov 2015 00:00:00 +0000

I am currently working on several micro services using the NancyFx framework with many projects sharing the same underlying architecture.

The IStatusCodeHandler interface is one of the core infrastructure components which is used by many Nancy projects to intercept and process a Nancy context before the final response gets returned to the client.

Having a few identical implemenations of IStatusCodeHandler I wanted to extract them into a re-usable NuGet package.

The problem is that Nancy automagically detects all implementations of IStatusCodeHandler and wires them up in the Nancy pipeline. In other words, if a library exposes an IStatusCodeHandler implementation then it gets automatically hooked into your Nancy application just by adding a reference to the assembly.

This is a nice feature, but makes it more difficult to include an IStatusCodeHandler in a shared library. I certainly don't want to modify an application's behaviour by simply adding a reference to the project.

Fortunately with some additional wiring you can easily expose an IStatusCodeHandler in a disabled state and allow your application to enable it only when required.

Here is an example of how to make an IStatusCodeHandler portable:

public class NotFoundStatusCodeHandler : IStatusCodeHandler
{
    private static bool _isEnabled = false;

    public static void Enable()
    {
        _isEnabled = true;
    }

    public bool HandlesStatusCode(
        HttpStatusCode statusCode,
        NancyContext context)
    {
        return _isEnabled && statusCode == HttpStatusCode.NotFound;
    }

    public void Handle(HttpStatusCode statusCode, NancyContext context)
    {
        // Do work
    }
}

I added a static Boolean field _isEnabled to the class definition. The field is initialised with false and only the Enable() method changes the value to be true.

Inside the IStatusCodeHandler.HandlesStatusCode method we check for the _isEnabled flag before evaluating the rest of the condition.

This will not stop Nancy from detecting and adding the IStatusCodeHandler into the pipeline, but will essentially suppress its behaviour entirely.

Enabling the handler is as easy as adding one line of code to your application startup event:

public class Bootstrapper : DefaultNancyBootstrapper
{
    protected override void ApplicationStartup(TinyIoCContainer container, IPipelines pipelines)
    {
        base.ApplicationStartup(container, pipelines);

        NotFoundStatusCodeHandler.Enable();
    }
}

The fact that the _isEnabled flag is static is not a problem, because it would either never change or only once during the application startup and remain that way for the rest of the application's lifetime.

It is a very simple but effective trick to make IStatusCodeHandler classes re-usable across multiple projects.

When to use Scrum? Waterfall vs. Scrum vs. Kanban vs. Scrumban

Sun, 08 Nov 2015 00:00:00 +0000

"Why", I love this word and it is probably my most favourite question to ask. It is also a curse at the same time, because if it is not followed by a satisfactory answer I can easily find myself in an unhappy situation.

This summer I decided to look for a new career opportunity and as such I went through many job interviews with various technology companies. Being a big fan of agile methodologies I was keen to learn exactly how their teams work and which processes they follow. Surprisingly, I often heard the same answer. A development manager or CTO would very proudly explain to me that all their teams work in Scrum. I however, was not very impressed with this answer.

All teams work in Scrum? Why?

Do all their projects really have the same requirements?

I doubt it. My guess is that all the teams work in Scrum because of the unfortunate circumstance that Scrum has become the unchallenged prime example of agile methodologies. The issue I have with this is that Scrum does not make you agile automatically. It can be agile when applied appropriately, but only if it's strengths match your requirements, otherwise it might be an impediment on its own.

Essence of Agile

Agile means to move quickly, be flexible and adaptable to rapid change.

If you didn't know anything about Scrum, would it sound very flexible to you if all teams have to follow the same rigid process?

Probably not and hence I ask myself how agile is such a company after all?

Sometimes I hear that an organisation invested a lot into implementing Scrum and therefore is unlikely to change their process any time soon. But hang on, someone invested a lot into becoming more agile and then they are averse to change? I find it a bit strange, because to me agile is all about change.

Fear of losing control

Self organising teams may seem daunting to a traditional manager. This is because managers are generally in charge of a team's process which puts them in control. Luckily Scrum is very heavy in process.

When waterfall became unpopular it was easy to find love for Scrum. It promises to be more agile, but also introduces a lot of new processes such as roles within a team, repetitive meetings, velocity based release planning, monitoring burndown charts and defining sprint goals.

I can see how a traditional manager felt very comfortable with the volume of processes and consequently implemented it across the entire organisation.

Self organising teams

Rolling out Scrum to all teams, making everyone follow the same iterations and maximising consistency in the process leaves very little to no room for self organising teams. At the end of the day there is nothing left to self organise and a team follows the same rigid patterns and reporting schedules as they did before, but now under the hat of Scrum.

Alternatively a much better, or in my opinion a more agile approach would be to say "I don't care how you work". Provide a team with enough coaching to understand the differences in agile methodologies and empower them to pick the right tool for themselves.

The manager can stop worrying about all the boring semantics of a process and focus on more relevant and exciting parts of the business - mainly the results. Additionally this will force a team to make success measurable, which is key in agile software development.

The power of trust

There is nothing more powerful than delegating responsibility to a team. It triggers a higher sense of ownership, pride in one's own work, intrinsic motivation and increase in productivity.

The team is responsible for it's own process and will take any action required to make it as smooth as possible. Situations where a team leaned back and waited for someone else to fix an issue belong in the past!

Scrum is not a silver bullet

Ultimately there is really no reason why every team should follow Scrum. The methodology of choice should meet a project's requirements rather than a manager's capability to control teams horizontally.

This also means that one team may change its workflow if the project changes or if the team switches to a different project.

So which project types are actually benefitting from Scrum?

When to use Scrum

Scrum is expensive

Scrum is a time consuming process. It takes a long time for a team to align their story point estimations and base velocity. Every day stand-up meetings, frequent backlog refinement and sprint planning meetings come with a price as well. A sprint is a fixed commitment and does not allow for any change or interruptions. Highly skilled people spend a lot of time doing secondary tasks which is not very cost effective either.

However, Scrum really pays off for projects which:

have stakeholders who frequently change their minds
require a quick feedback loop
use stakeholders' feedback to prioritise the next sprint
don't get many interruptions from everyday business
have a cross functional team

Scrum is great for projects with little baggage

Greenfield projects and projects in their early days are great candidates for Scrum. Stakeholders may have busy schedules and cannot afford to see a team every day. Regular 2-4 week sprint review meetings give them an opportunity to view the latest deliverables, clarify new requirements and define or change the scope of the following iteration.

The project is new, which means there is almost no maintenance work and the team can focus on a few shippable features every sprint. The team is also less worried about filling up the sprint with as much work as possible. Instead they commit to a couple of features which they are confident to present to the stakeholders at the end of a sprint.

The team gets a full iteration to produce value without any interruptions. Anything after that may be subject to change. This is where a sprint really lives up to its name.

The heavy Scrum process is vital to this type of project. Every single meeting and the restrictions of a committed sprint serve a real purpose and are advantageous to the success of a project.

However, if a project doesn't have any of the requirements listed above then Scrum can very quickly become uneconomical.

Waterfall is okay if nothing is expected to change

If your project doesn't anticipate any change then Scrum is certainly an overkill.

You might even question if an agile approach is the right tool at all?

There is no shame to consider a more traditional waterfall workflow for some projects.

If you need to migrate an old C++ library to a more modern language and the library is well documented, free of ambiguity and doesn't need any change then it might be easier to just get on with the work and not faff about with Scrum planning- and redundant sprint review meetings.

Maximum flexibility with Kanban

Kanban is great in many ways where Scrum has its limits. It is a lean system, which means one of it's main principles is to eliminate any waste. There are no iterations, no sprint planning meetings and therefore no story pointing, only one continuous flow.

Kanban is extremely flexible and suits a wide range of projects. In particular it works well for projects which:

have reached a certain level of maturity with many business as usual tasks, defects and smaller unrelated user stories
require maximum flexibility and frequent change of priorities
have multiple releases per week or per day
have many unscheduled releases
have less cross functional teams

Mature projects

When a project enters the phase where core features have been implemented and there's naturally less ambiguity about the general course of the project then the advantages of a Scrum driven process will gradually disappear.

This is often, but not necessarily the point when a project has had its first release and customers started using the system. New changes get pushed directly to consumers and enable a new channel for immediate feedback outside of Scrum.

Business as usual tasks

At this stage the project also starts picking up many business as usual tasks such as bugs, minor improvements or 3rd line support from developers.

While these tasks are considered as noise or impediments in Scrum, they are still important and first class citizens in Kanban.

Unscheduled releases

Some tasks or bug fixes are critical and cannot wait for the next release at the end of an interation. Unfortunately Scrum does not facilitate for these cases.

I have seen and practised many workarounds, but none of them are ideal:

Either a team plans all sprints below velocity to accommodate for the unknown
...or the PO cancels the current sprint to deal with the interruption
...or the PO removes user stories from the sprint backlog to make room for the unscheduled release mid sprint
...or the team simply deals with the event and shrugs their shoulders when they move half unfinished work into the next iteration

In particular when the last two circumstances become a habit then a sprint commitment loses all of its meaning and the team basically cries for a more flexible workflow like Kanban.

Less cross functional teams

Scrum is not very compatible with non cross functional teams. It either results in a lot of friction or in a waste of resources. Kanban doesn't entirely solve the problem, but is certainly a lot more friendly towards specialised roles and distributed efforts within a team.

Work tasks can be divided into several stages with different upper limits on current work in progress. This allows a team to tweak those limits to avoid blockages and optimise a smooth continuous flow.

Another benefit of Kanban is that bottlenecks are clearly visualised on the cumulative flow diagram and therefore quickly highlights inefficiencies or understaffed areas of the development cycle.

Continuous integration

Last but not least, Kanban is a perfect match for teams who aspire a continuous integration pipeline. Process flow and cycle time are key metrics in Kanban and teams iteratively improve on them to deliver innovation just-in-time to the market.

Caveats

Even though Kanban is applicable to many projects it not as widely used as Scrum. Due to it's fast pace it requires a good and robust test automation suite which makes it difficult for many teams to begin with.

Scrumban

While Scrum is a continuous sequence of time boxed iterations, Kanban has only one continuous flow. The repetition of process establishes routine and familarity. A well accustomed team could struggle to adopt Kanban and give up the routine.

This is where Scrumban comes into play, initially invented to help teams with the transition from Scrum to Kanban. In Scrumban a team primarily uses Scrum as their chosen way of work and mixes in concepts from Kanban to allow for specialised roles, work policies and a better flow.

One fundamental idea of Scrumban is to help teams and organizations develop their own set of Scrum-based processes and practices that work best for them.

Which process will you pick?

Back to my initial question, how agile can a company be if all teams have to follow Scrum? It may be possible that Scrum is the right fit for all their projects, but looking at the variety of different methodologies it is more than likely that this is not the case.

Scrum is a brilliant tool, but it is not an all purpose solution to becoming more agile. If Scrum is applied inapproriately it could decrease productivity, as well as frustrate everyone involved with it. Eventually people will end up blaming the process for the inadequacies of the implementor. Let's not stigmatise Scrum like we did with Waterfall.

Death of a QA in Scrum

Mon, 28 Sep 2015 00:00:00 +0000

The world is changing fast and the software industry even faster. Today I am a software developer with almost 9 years of commercial experience and recently I was told that this is about the time when a developer actually becomes good. Whether that is true or not I leave to someone else, but there is certainly a level of maturity which I have today and I didn't have a couple of years ago.

In those 9 years I was lucky enough to work in different teams, with different technologies and different approaches of agile methodologies.

However, one thing which has never changed was my role within the teams. Regardless of my actual job title I was a hands-on developer among other roles such as testers, business analysts, product owners, architects and UX experts.

The interesting part is that software developers and testers were always separated into two different roles. One guy (or girl!) was supposed to write code and another guy/girl was supposed to test it.

This model sounds great in theory, but in my own (subjective) experience it never really worked.

The separation of roles in Scrum

This is not another debate on manual vs. automated testing. When I say it never really worked then I mean in the context of working in an agile Scrum team with automated regression QA.

More precisely the distinction of developers and QA caused a lot of friction in our Scrum process and mostly we ended up with many issues like:

QA had very little work at the beginning of a sprint
At the end of a sprint we had a lot of QA tasks piling up
Many QA tasks didn't get done before the end of the sprint
Developers wrote features quicker than QA could test them
...and most importantly, it was impossible to have a developer write code for an entire iteration and on the last day have everything go through code review and QA without any issues

Basically through the separation of roles we ended up with a lot of difficulties, bottlenecks and inefficient use of resources.

A production line in Scrum

What was happening is that a user story got divided into several work tasks and each task was worked on by a different person in the team. It felt a lot like a production line:

While it was possible to do some parallel work on the development and QA task at the same time, it was not possible to close one before the other. We had inter team dependencies.

Unfortunately this production line is not a hybrid approach of Scrum with a pinch of Kanban, it is more of a mini waterfall within Scrum:

The team commits to deliver a fixed scope (sprint backlog) at a fixed date (end of sprint) with fixed resources (size of the team)
But the problem is that the scope is a set of unrelated work items instead of one or two shippable features
Because each work item is divided into different phases (development, review, QA, etc.) it is difficult to estimate it as a whole and the team ends up over-committing
Dependencies between the phases and bottlenecks only add up to it
And finally the separation of roles leads to a lot of last minute testing, not allowing any room for re-iteration and unexpected issues
Eventually the sprint goal will be missed
And early warning signs like a sprint burndown are not meaningful as the team never really knows if they are on track

When this happens the team usually doesn't think of completing one goal anymore. Individual people think of completing individual phases. The separation of roles is the same as the separation of concerns and results in sub teams within a team. While this is great as a coding practise it is poison to Scrum.

Scrum is one team, not a set of roles

The problem also becomes very prominent if the team starts using language like "we developers do this..., they QA do that..." and vice versa.

Scrum is built on the fundamental believe that one self-contained team works together towards a goal. There cannot be any sub teams, dependencies or bottle necks within a team. Each member of the team (except the PO and BA if you like) must be capable of working on each phase of a user story, otherwise your Scrum is deemed to fail.

What does this mean for developers and QA then? Well, there is simply not much space for a dedicated QA anymore!

Yes that's right, but before everyone starts to freak out now, I don't mean to get rid of all QA by sacking them. That would be mad and a true waste of many years of experience, valuable domain knowledge, skills and employee loyalty. We need to get rid of the QA as a separate role within a Scrum team!

We are all developers

I think it would be fair to say that we are all developers. At the end of the day writing good automation tests requires a lot of engineering skills and good coding practices as much as writing production code.

In order to make every team member responsible for writing production code as well as reviewing and writing tests we need to give everyone a title which reflects these responsibilities.

Make everyone a developer!

If you really need to distinguish between different levels of expertise then give your team members a title which relates to their experience rather than a role. You can turn your automation QA into a junior or mid-level developer. They already know how to write maintainable automation tests so they are not far away from writing maintainable production code. With a little bit of guidance from a senior member of the team it should be possible to train your current QA into a full member of the team.

Personally I have not made the experience of this change yet, but I witnessed a transition from manual testers to fully committed automation testers and the results were extremely good! The learning curve is very steep in the beginning, but to be honest the entire software industry is a big journey of change and it is not any different for a junior developer either.

As someone said to me before... it takes about 8-9 years before a developer actually becomes good...

Are you ready for the change?

Display build history charts for AppVeyor or TravisCI builds with an SVG widget

Sun, 30 Aug 2015 00:00:00 +0000

If you ever browsed a popular GitHub repository (like NUnit or Bootstrap) then you must have seen many of the available SVG badges which can be used to decorate a repository's README file.

While some repositories keep it very simple:

Others can be quite fancy:

These little widgets (or often called badges) are more of a gimmick rather than anything useful, but we love them because they give us an opportunity to visually highlight statistics or achievements which we are proud of.

Having a few of my own open source projects I also wanted to decorate my README files with one or more fancy widgets.

I quite like the little build charts which you get in VisualStudio's Team Explorer and I thought I could create something similar for my AppVeyor builds myself:

A couple days later I also added support for TravisCI builds, uploaded everything to GitHub and hosted the widget in Windows Azure.

You can see it in action in one of my public GitHub repositories.

Other repositories can use the widget as well by simply providing their own account and project name in the widget's URL. Additionally you can specify the number of builds to be shown and a switch to display or hide the build statistics.

For a complete up to date feature list and concrete code examples please visit the official project page.

Ideas, contributions, bug reports or any type of feedback is welcome!

Using C# 6 features in ASP.NET MVC 5 razor views

Fri, 21 Aug 2015 00:00:00 +0000

Recently I upgraded my IDE to Visual Studio 2015 and made instant use of many new C# 6 features like the nameof keyword or interpolated strings.

It worked (and compiled) perfectly fine until I started using C# 6 features in ASP.NET MVC 5 razor views:

Feature 'interpolated strings' is not available in C# 5. Please use language version 6 or greater.

The project compiles fine, but intellisense underlines my interpolated string in red and tells me that I can't use this feature in C# 5. Well I know that myself, but the real question is why does it think it is C# 5?

It is the compiler's fault

I knew I didn't have to change the .NET Framework version to .NET 4.6, because it is a language feature and not a .NET framework feature. The compiler is responsible for translating my C# 6 code into IL code which is supported by the framework.

However, saying that I don't get any errors at compilation time even though I made a lot of use of C# 6 features all over my project.

Maybe it is an intellisense bug in Visual Studio 2015? Not really, because when I start my project I get a yellow screen of death which matches the intellisense error:

ASP.NET Runtime compiler

The problem is at runtime when ASP.NET tries to compile the razor view. ASP.NET MVC 5 uses the CodeDOM Provider which doesn't support C# 6 language features.

Solutions

There are two solutions to fix the problem:

Upgrade your application to MVC 6 (which is still in beta at the time of writing)
Reference the Roslyn compiler in your project by using the compiler element in your web.config

The 2nd option is as easy as installing the CodeDOM Providers for .NET Compiler nuget package.

It replaces the CodeDOM provider with the new .NET compiler platform (aka Roslyn) compiler as a service API. After installing the nuget package in your MVC 5 project you will be able to use C# 6 features in razor views as well!

How to use RSA in .NET: RSACryptoServiceProvider vs. RSACng and good practise patterns

Thu, 13 Aug 2015 00:00:00 +0000

In my last blog post I wrote a little crash course on RSA and how it works without looking into any specific language implementations. Today I'd like to explore the native implementations of .NET and the new RSACng class which has been introduced with .NET Framework 4.6.

In .NET 4.6 you'll find three native RSA classes:

RSA Class

The RSA class in System.Security.Cryptography is an abstract class which cannot be instantiated itself. It is the base class for all other RSA implementations and exists since .NET 1.1 in the mscorlib assembly.

It derives from the abstract class AsymmetricAlgorithm, which itself implements IDisposable. This means every instance of an RSA implementation should be disposed after its usage to free up memory as soon as possible.

The base class defines many methods, but most likely you will be interested in one of these which come with the default RSA contract:

Encrypting data
Decrypting cipher data
Signing data
Signing a hash of data
Validating signed data
Validating a signed hash
A factory method to instantiate an implementation of RSA

Differences between .NET Frameworks

.NET 3.5 and earlier

In .NET 3.5 and earlier the RSA class was much smaller than it is today. It didn't have a contract for signing and validating data and only exposed two methods for encyrpting and decrypting a value:

Also notice how the methods accept a data object but no indication of which padding system should be used. Using an encyrption padding is critical to the security of your RSA implementation and therefore is somewhat lacking in the base contract.

After .NET 4.0

Starting with the .NET 4.0 framework the RSA class has been significantly extended. In addition to all the signing methods it received two new methods for encrypting and decrypting a message:

public virtual byte[] Encrypt(byte[] data, RSAEncryptionPadding padding)
public virtual byte[] Decrypt(byte[] data, RSAEncryptionPadding padding)

Interestingly they are not mentioned in the official MSDN documentation on the web, however when I decompile .NET 4.0's mscorlib I can see the two virtual methods:

This was a great addition for two reasons in particular:

It allows you to specify which padding should be used.
It matches the implementation of the RSACryptoServiceProvider. This is very nice, because the RSACryptoServiceProvider never bothered to implement the EncryptValue and DecryptValue methods and made it impossible to program against the native RSA contract in previous .NET versions.

Factory methods

The RSA class also implements two static factory methods to create an instance of RSA:

In all versions of .NET the default implementation is the RSACryptoServiceProvider:

using (var rsa = RSA.Create())
{
    Console.WriteLine(rsa.GetType().ToString());
    // Returns System.Security.Cryptography.RSACryptoServiceProvider
}

Overriding the default implementation

The factory methods are designed to work with the CryptoConfig setting from the machine.config file.

In order to override the default implementation you need to map a friendly algorithm name to a specific cryptography class.

Be aware that you have different machine.config files for each architecture and .NET framework:

32-bit
%windir%\Microsoft.NET\Framework\{.net version}\config\machine.config
64-bit
%windir%\Microsoft.NET\Framework64\{.net version}\config\machine.config

For example if you have a 64-bit application built with .NET 4.0 you need to modify the machine.config under this path:
%windir%\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config

For this demo I will map "SomeCustomFriendlyName" to the native RSACng class:

<mscorlib>
  <cryptographySettings>
    <cryptoNameMapping>
      <cryptoClasses>
        <cryptoClass
         MyRSAImplementation="System.Security.Cryptography.RSACng,
          System.Core, Version=4.0.0.0, Culture=neutral,
          PublicKeyToken=b77a5c561934e089" />
      </cryptoClasses>
      <nameEntry
        name="SomeCustomFriendlyName"
        class="MyRSAImplementation" />
    </cryptoNameMapping>
  </cryptographySettings>
</mscorlib>

Now when I run this...

using (var rsa = RSA.Create("SomeCustomFriendlyName"))
{
    Console.WriteLine(rsa.GetType().ToString());
}

...it will output "System.Security.Cryptography.RSACng".

If you don't specify a friendly algorithm name then the default Create() method will call RSA.Create("System.Security.Cryptography.RSA") under the covers.

Hence adding another nameEntry for "System.Security.Cryptography.RSA":

<mscorlib>
  <cryptographySettings>
    <cryptoNameMapping>
      <cryptoClasses>
        <cryptoClass
          MyRSAImplementation="System.Security.Cryptography.RSACng,
          System.Core, Version=4.0.0.0, Culture=neutral,
          PublicKeyToken=b77a5c561934e089" />
      </cryptoClasses>
      <nameEntry
        name="SomeCustomFriendlyName"
        class="MyRSAImplementation" />
      <nameEntry
        name="System.Security.Cryptography.RSA"
        class="MyRSAImplementation" />
    </cryptoNameMapping>
  </cryptographySettings>
</mscorlib>

...will now also output "System.Security.Cryptography.RSACng" when I run RSA.Create().GetType().ToString().

RSACryptoServiceProvider vs. RSACng

Where to find

RSACryptoServiceProvider exists in mscorlib since .NET 1.1 while RSACng is a very recent addition with .NET 4.6 and lives in System.Core.

CoreFX is open source and you can browse any implementation as well as the RSACng class on GitHub.

Implementation

Both classes are sealed and derive from the base RSA class and implement their members. However, both classes throw a NotSupportedException when calling EncryptValue and DecryptValue. You are forced to use the Encrypt and Decrypt methods which accept a padding.

Both classes are FIPS compliant!

Also worth mentioning is that both classes use the OS's underlying CSP (Crypto Service Provider) and CNG (Cryptography Next Generation) providers which are unmanaged code and should have considerably similar performance.

Difference

In short, the CNG implementation is...

...the long-term replacement for the CryptoAPI. CNG is designed to be extensible at many levels and cryptography agnostic in behavior.

Cryptography API: Next Generation, MSDN Microsoft

One difference which you will quickly notice is that the key in RSACng is managed by the CngKey class and can be injected into the constructor of RSACng, while RSACryptoServiceProvider is tied to its own key operations.

Support

CNG is supported beginning with Windows Server 2008 and Windows Vista. RSACng requires .NET framework 4.6 (or higher).

The Crypto API was first introduced in Windows NT 4.0 and enhanced in subsequent versions.

Good practise patterns

After all it doesn't really matter which implementation you choose if you abstract it away and program against a contract. This is good practise anyway and luckily .NET offers the abstract RSA class for this purpose out of the box.

Constructor injection vs. Factory method

Constructor injection is the de facto standard pattern for dependency injection in most cases. However, when you deal with an object which implements the IDisposable interface then constructor injection can be responsible for keeping the object longer alive than it needs to be.

Image I have this class:

public class MyClass
{
    private readonly RSA _rsa;

    public MyClass(RSA rsa)
    {
        _rsa = rsa;
    }

    public void sendSecureMessage(string message)
    {
        byte[] data;
        // Convert the message into data

        _rsa.Encrypt(data, RSAEncryptionPadding.Pkcs1);

        // More business logic
    }
}

Whatever happens after _rsa.Encrypt(...) doesn't require the RSA object any longer and it should get disposed immediately.

I could call _rsa.Dispose(); afterwards, but this would be a bad code smell, because:

MyClass is not the owner of the RSA instance. It didn't create it and therefore shouldn't dispose it.
The RSA instance could have been injected somewhere else as well, therefore disposing it would cause a bug.
There is a benefit of the using statement and the constructor injection pattern doesn't allow me to make use of it

This means we are better off by using a different dependency injection pattern. In this instance the factory pattern is more suitable:

public interface IRSAFactory
{
    RSA CreateRSA();
}

public class MyClass
{
    private readonly IRSAFactory _rsaFactory;

    public MyClass(IRSAFactory rsaFactory)
    {
        _rsaFactory = rsaFactory;
    }

    public void SendSecureMessage(string message)
    {
        byte[] data;
        // Convert the message into data

        using (var rsa = _rsaFactory.CreateRSA())
        {
            rsa.Encrypt(data, RSAEncryptionPadding.Pkcs1);
        }

        // More business logic
    }
}

I have created an IRSAFactory interface which defines a contract to create a new instance of RSA. Now I can inject this factory into MyClass and conveniently create a new RSA object on the fly and encapsulate it with the using statement to properly dispose it afterwards.

Why not using the native RSA.Create() factory method?

What is wrong with the native factory method? Why didn't I just do this:

public class MyClass
{
    public void SendSecureMessage(string message)
    {
        byte[] data;
        // Convert the message into data

        using (var rsa = RSA.Create())
        {
            rsa.Encrypt(data, RSAEncryptionPadding.Pkcs1);
        }

        // More business logic
    }
}

Theoretically this is absolutely fine, but it makes it more difficult to provide an alternative implementation in my unit test suite, because the native factory is designed to work with the machine.config and cannot be changed programmatically!

Wrapper for RSA

Alternatively I could have created a wrapper for the RSA class like this:

public abstract class RSAWrapper : RSA
{
    private static RSA _overridenDefaultRSA = null;

    public static void OverrideDefaultImplementation(RSA rsa)
    {
        _overridenDefaultRSA = rsa;
    }

    public static new RSA Create()
    {
        return _overridenDefaultRSA ?? RSA.Create();
    }
}

...and then use it like the original class:

public class MyClass
{
    public void SendSecureMessage(string message)
    {
        byte[] data;
        // Convert the message into data

        using (var rsa = RSAWrapper.Create())
        {
            rsa.Encrypt(data, RSAEncryptionPadding.Pkcs1);
        }

        // More business logic
    }
}

This would allow me to override the default implementation from the machine.config by calling into OverrideDefaultImplementation(RSA rsa) from my unit tests.

I personally prefer the first approach though, for the following reasons:

By injecting an IRSAFactory object into MyClass it is obvious from the outside which dependencies MyClass has
It allows me to provide different mocks and stubs for different unit tests while running them in parallel. This would be very tricky with the static factory method.
At some point I'd have to initialise the RSA key by using RSA.FromXmlString or RSA.ImportParameters. With the IRSAFactory (which is an abstract factory btw) I could provide additional methods to do this and test the interaction between them as well.

I hope this was useful and that I could shed more light on RSA in .NET.

There are a lot of resources on the internet showing how to use the RSACryptoServiceProvider and therefore I didn't want to re-iterate over the same topic again and focus more on some patterns beyond the default examples.

The beauty of asymmetric encryption - RSA crash course for developers

Sun, 28 Jun 2015 00:00:00 +0000

With the rapid growth of the internet and the vast business which is handled over the web it is not surprising that security has become an inevitable topic for any software developer these days.

Unfortunately security and in particular cryptography is a complex science on its own. It is very difficult to get it right and extremely easy to get it wrong.

As a developer I personally find the topic very interesting and challenging and therefore strive to understand the concepts of cryptography more than just the bare minimum.

With this in mind I would like to break down one of the most important cryptographic algorithms which is used in modern web development nowadays: Asymmetric encryption with RSA.

History

While modern cryptology has a long history dating back to AD 800, asymmetric encryption is only a very recent discovery starting in the mid-70s.

Previously every symmetric encryption algorithm had the fundamental problem of the secret key distribution. Two parties had to find a way to share a secret key via an insecure channel before they could successfully exchange private messages.

Only in 1977 (a year after Whitfield Diffie and Martin Hellman introduced the Diffie-Hellman key exchange algorithm) three scientists, Ron Rivest, Adi Shamir and Leonard Adleman, succeeded in inventing the first publicly known asymmetric encryption algorithm named RSA. Back then GCHQ, a UK intelligence agency has independently developed the first public key algorithm in 1973 and 1974, but this was kept secret until the mid 80s.

Today RSA is the de-facto standard asymmetric encryption algorithm and is used in many areas such as TLS/SSL, SSH, digital signatures and PGP.

Basics

Before cracking down the RSA algorithm I would like to scratch on some basics, which are essential to understand the nature of RSA.

Public-key encryption

Public-key encryption, as opposed to secret-key encryption, consists of a pair of keys - the public key which is used to encrypt a message and the private key, which is subsequently used to decrypt the cipher message.

Each private key has only one matching public key. A message encrypted with the public key can only be decrypted with the related private key.

Alice, Bob and Eve

Alice and Bob want to communicate privately and Eve wants to eavesdrop. Both, Alice and Bob have their individual public and private key pair.

Alice uses Bob's public key to encrypt a private message before sending it to Bob. Bob can use his private key to decrypt the message. Now Bob can use Alice's public key to reply to Alice without Eve being able to understand any of the transmitted data. Finally Alice decrypts Bob's message with her own private key.

The public key is available to everyone, while the private key is only known to the key holder. There is never the requirement to share a secret key via an insecure channel.

Integrity and Authenticity

Now theoretically Eve could equally use Alice's or Bob's public key and send them a private message. It wouldn't help her directly to read anyone's encrypted message, but she could trick either one of them to believe her message was from Bob or Alice and provoke them to make a mistake.

Eve could also intercept and tamper with the encrypted message, so that after decryption it will read something different.

Just by encrypting a message neither Bob or Alice can ever be sure that the message hasn't been modified (integrity) by Eve nor that the message wasn't sent from Eve in the first place (authenticity).

Luckily RSA offers a solution to this problem. Similar to encrypting a message, the algorithm can also be used for signing it. Bob can use his private key to compute a signature from his message. The signature would be unique to the message and no other message would compute the same signature.

Alice can then use Bob's public key to verify if the message and the attached signature match. Because she uses Bob's public key she can be sure that the message came from Bob. Only Bob can create a correct signature with his private key. Additionally Alice will easily know if the message has been modified or not. If Eve changed the message or the attached signature, then Alice will not be able to verify the signature with Bob's public key and therefore know that someone has tampered with the data.

By providing an additional signature Alice and Bob can trust each others messages.

Encrypting and signing

Encrypting or signing are not exclusive. For example Alice can first encrypt her message and then use the resulting cipher as the base for computing a valid signature. Afterwards Bob has to first verify the signature based on the cipher and when that goes well he can proceed to decrypt the message. Now both have established a trusted and secure way of communicating privately.

One-way functions

The concept of encrypting a message with one key and not being able to decrypt with the same key is based on one-way functions. As the name suggests the characteristic of a one-way function is that it is not reversible other than with a trial and error approach.

This can be achieved if there is an infinite amount of values which lead to the same result, if there is some information lost as part of the algorithm or if the time to decrypt takes immensely longer than to encrypt.

A simple example of a one-way function

Let's say the initial value is 264. The one-way function reads as following:

You start from the centre of a map. Now take your value and divide it by it's last digit. The result is a new value x. Now draw a line x centimetres north east and mark a new point on the map. Next take your original value and subtract it by x. The result is y. Draw another line, starting from the last point, y centimetres south west. The final point is the end result.

In this example we would divide 264 by 4 and retrieve 66 for x. Additionally we subtract 66 from 264 and retrieve y = 198. We draw both lines and determine the final point on the map, which represents the end result of the one-way function.

Now just from knowing the final point on the map and the definition of the function it is not possible to easily deduce the original value.

Modular arithmetic

Modular arithmetic is full of one-way functions. It is also known as clock arithmetic, because it can be illustrated by a finite amount of numbers arranged in a loop, like on a clock:

The dark circle represents the clock. The blue numbers represent the value 17. If you arrange all numbers from 1 to 17 clockwise in a loop, then the end value results in 5. In other words 17 mod 12 equals 5.

The short-cut and common way of calculating the modulus is by dividing the original value by x. The reminder equals the modulus.

The modulus operation is a great one-way function, because it is fairly simple and has an infinite amount of possible values giving the same result.

Prime numbers

Prime numbers are the last important building block of the RSA algorithm.

A prime number (or a prime) is a natural number greater than 1 that has no positive divisors other than 1 and itself.

Prime number, Wikipedia

Example of prime numbers: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, ...

What's important to know is that there is an infinite amount of prime numbers and there is no effective formula to calculate a prime number or to validate a given number to be prime or not.

Prime factorisation

Factorisation is the process of decomposing a positive integer into a product of smaller numbers.

For example the number 759 can be decomposed into 3 and 253.
In other words 759 = 3 * 253.

If one continues the process you will eventually end up with all numbers being prime numbers (= prime factorisation). In our case 253 can be further broken down into 11 and 23.

Finally you can say 759 = 3 * 11 * 23.

Primes become rarer as we progress through the integers and there is a race in finding the next highest prime number in history.

The RSA algorithm

As genius as the algorithm is, it is fairly short and can be demonstrated with a relatively small example.

Generating the public and private key

First Alice needs to generate her own public and private key pair. For this Alice picks two prime numbers p = 17 and q = 19.

She multiplies them together to retrieve the number n = 323.

Next she picks another prime e = 7.

Lastly she needs to calculate the value d. This is done by working out the following equation:
e * d mod ((p - 1) * (q - 1)) = 1
7 * d mod 288 = 1
d = 247

Private key

The private key is d = 247.

p and q are kept secret as well. d can only be calculated if p and q are known.

Public key

The public key is formed by n and e:
n = 323
e = 7

Note that n is the product of two prime numbers. It can only be decomposed into p and q. If p and q are large enough then it is de-facto impossible.

Encrypting a message

The public key is used to encrypt a message. Let's say Bob wants to encrypt the message "I love Alice". Internally all data is represented in binary format and binary numbers can be converted into decimals.

In C# you can quickly convert it using this snippet:

var message = "I love Alice";
var binary = Encoding.ASCII.GetBytes(message);
var decimals = binary.Select(b => Convert.ToInt32(b)).ToArray();

"I love Alice" is represented by this number sequence:
73, 32, 108, 111, 118, 101, 32, 65, 108, 105, 99, 101.

To keep this example short I will only encrypt the first letter of the message, which is represented by the number 73.

The formula to encrypt m = 73 is:
c = m^e mod n
c = 73⁷ mod 323
The cipher c = 112.

Decrypting the cipher

Alice receives the cipher 112 and can decrypt it using her private key d and the formula:
m = c^d mod n
m = 112²⁴⁷ mod 323
The first number of the original message is m = 73.

RSA is a brilliant one-way function which allows someone to reverse it only if the private key d is known.

Real world application

RSA is considerably slow due to the calculation with large numbers. In particular the decryption where d is used in the exponent is slow. There are ways to speed it up by remembering p and q, but it is still slow in comparison to symmetric encryption algorithms.

A common practise is to use RSA only for the encryption of a secret key, which then is used in a symmetric encryption algorithm. Typically the message to encrypt is a lot longer than the secret key itself, therefore this is a very effective method to benefit from the security of an asymmetric- and the speed of a symmetric encryption algorithm.

Key length

With the fast development of computer chips the recommended key length for RSA changes over time.

RSA claims that 1024-bit keys are likely to become crackable some time between 2006 and 2010 and that 2048-bit keys are sufficient until 2030. An RSA key length of 3072 bits should be used if security is required beyond 2030.

Asymmetric algorithm key lengths, Wikipedia

If you use RSA in your application, then you should periodically (every few years) recycle your keys and generate a new pair to meet current length recommendations and stay secure.

Future

The entire security of RSA is built on the fact that it is impractical to determine p and q by looking at n. If at some point in the future a mathematician finds a way to rapidly factor n, then RSA would become of no use.

Another interesting way of cracking most of today's crypto systems could be with the help of quantum computing, which as of now still remains a very theoretical topic.

Running free tier and paid tier web apps on the same Microsoft Azure subscription

Sun, 14 Jun 2015 00:00:00 +0000

Last week I noticed a charge of ~ £20 by MSFT AZURE on my bank statement and initially struggled to work out why I was charged this much.

I knew I'd have to pay something for this website, which is hosted on the shared tier in Microsoft Azure, but according to Microsoft Azure's pricing calculator it should have only come to £5.91 per month:

After a little investigation I quickly found the issue, it was due to a few on and off test web apps which were running on the shared tier as well.

This was clearly a mistake, because I was confident that I created all my test apps on the free tier, but as it turned out, after I upgraded my production website to the shared tier all my other newly created apps were running on the shared tier as well.

I simply didn't pay close attention during the creation process:

Evidentially every new web app gets automatically assigned to my existing app service plan, which I upgraded to the shared tier.

Luckily I learned my lesson after the first bill. However my initial attempt to switch my test apps back to the free tier was not as simple as I thought it would be. I cannot scale one app individually without affecting all other apps on the same plan:

The solution is to create a new app service plan and assign it to the free tier.

You can do this either when creating a new web app, by picking "Create new App Service plan" from the drop down:

Or when navigating to the new Portal, where you have the possibility to manage your app service plans:

This wasn't difficult at all, but certainly a mistake which can easily happen to anyone who is new to Microsoft Azure.

Another very useful thing to know is that if you choose the same data centre location for all your app service plans, then you can easily move a web app from one plan to another. This could be very handy when having different test and/or production stages (Dev/Staging/Production).

Effective defense against distributed brute force attacks

Fri, 01 May 2015 00:00:00 +0000

Protecting against brute force attacks can be a very tricky task.

Recently I was curious if there are any best practices to protect a website from distributed brute force attacks and I found a lot of interesting solutions:

Lock an account after X failed login attempts

The first method I found was very trivial. If a user reaches a certain limit of failed login attempts the website locks down the account and refuses any further access.

A genuine user can unlock his or her account by requesting a recovery link via email or by changing their password via the password reset function.

Problems with this pattern

Introduces a targeted DOS attack surface. An attacker could easily lock out an account by purposefully providing the wrong password several times to either to block an account from using the service entirely or to force the user into a recovery path, where the attacker might have found further vulnerabilities.
Doesn't protect against more sophisticated attacks (typically an attacker would pick the most common password an try it on all accounts, then pick the second most common password, etc.)
Introduces a potential enumeration attack. An attacker can purposely provide a wrong password and determine if a certain email address/username exists if the account gets locked or not.

Blocking IP Addresses with too many failed login attempts

This one is fairly simple and well known. If a certain IP address had too many failed login attempts, then further access from this IP address is denied.

Problems with this pattern

It doesn't help against a distributed brute force attack.
Opens the door for another DOS attack.
There is a good chance that users behind a shared network will lock themselves out if enough users type in a wrong password within a short period of time.

Whitelist-/Blacklisting IP Addresses

The idea is that a user can limit access to his/her account based on IP address rules. It can be as simple as allowing access to one single IP address, multiple addresses or more complex rules around IP address ranges or subnets.

Problems with this pattern

Impractical for most websites or web services.
This pattern requires a user to put effort into security configuration instead of being secure by default.
Can become a maintenance nightmare.

Increase artificially the login time after each failed attempt

This one I found very creative. Each failed login attempt causes the next failed login request to take longer by a factor X. A successful login will proceed in normal speed at any point of time. This allows a website to throttle a distributed brute force attack while providing good experience for a genuine user.

Problems with this pattern

If a genuine user makes a mistake shortly after an attack, they might end up with a long response time.
The website ends up unnecessarily wasting threads. This can result in a potential DOS attack again!?

Implement a challenge like a CAPTCHA

This appraoch is trying to stop automated bots from brute forcing an account by implementing a challenge, which supposedly can only be accomplished by a human. Captchas are a very popular solution, but there are many other creative approches to filter humans from machines which work on the same assumption.

Problems with this pattern

Bad user experience for the geuine user.
Computer learning and social engineering make it a tough challenge to come up with a good filter.

Additional verification step

Digital signatures, two factor authentication and many other patterns require an additional step of verification. They are highly effective against brute force attacks, but have their own down sides and might be impractical for many web services.

Combination of patterns?

Quickly you will find that one pattern on it's own might not do the trick. I tried to think of a good combination of patterns and potential pros and cons attached to them and my best idea was the following:

Monitor the average fail rate and CAPTCHAs

The website determines a natural rate of login failure over a certain period of time. Once this metric has been established it starts monitoring and counting failed login attempts going forward. When the number of failed login attempts significantly deviates from the natural rate then a CAPTCHA will be displayed on all subsequent login requests.

If the rate recovers then the CAPTCHA will be hidden from the login screen again. A very transparent website could even show a notification to the user explaining why the CAPTCHA is being displayed and remind the user to set a strong password if not done yet.

Pros

Effective against any type of brute force attack?
Good user experience.

Cons

Might be difficult to establish the initial variables.

Strict password policy

Another very viable approach is to simply not fight brute force attacks at all. Make sure your users have strong passwords and make brute force attempts rather harmless.

A good password policy is probably a good idea in any case. As always, security comes in layers.

If you know any other effective defense systems against distributed brute force attacks I'd be interested in hearing them!

Demystifying ASP.NET MVC 5 Error Pages and Error Logging

Mon, 06 Apr 2015 00:00:00 +0000

elmah.io loves this post and since we already use it as part of our official documentation for implementing custom error pages, we've decided to sponsor it. Visit elmah.io - Error Management for .NET web applications using ELMAH, powerful search, integrations with Slack and HipChat, Visual Studio integration, API and much more.

Custom error pages and global error logging are two elementary and yet very confusing topics in ASP.NET MVC 5.

There are numerous ways of implementing error pages in ASP.NET MVC 5 and when you search for advice you will find a dozen different StackOverflow threads, each suggesting a different implementation.

Overview

What is the goal?

Typically good error handling consists of:

Human friendly error pages
- Custom error page per error code (e.g.: 404, 403, 500, etc.)
- Preserving the HTTP error code in the response to avoid search engine indexing
Global error logging for unhandled exceptions

Error pages and logging in ASP.NET MVC 5

There are many ways of implementing error handling in ASP.NET MVC 5. Usually you will find solutions which involve at least one or a combination of these methods:

All these methods have a historical reason and a justifyable use case. There is no golden solution which works for every application. It is good to know the differences in order to better understand which one is applied best.

Before going through each method in more detail I would like to explain some basic fundamentals which will hopefully help in understanding the topic a lot easier.

ASP.NET MVC Fundamentals

The MVC framework is only a HttpHandler plugged into the ASP.NET pipeline. The easiest way to illustrate this is by opening the Global.asax.cs:

public class MvcApplication : System.Web.HttpApplication

Navigating to the implementation of HttpApplication will reveal the underlying IHttpHandler and IHttpAsyncHandler interfaces:

public class HttpApplication : IComponent, IDisposable, IHttpAsyncHandler, IHttpHandler

ASP.NET itself is a larger framework to process incoming requests. Even though it could handle incoming requests from different sources, it is almost exclusively used with IIS. It can be extended with HttpModules and HttpHandlers.

HttpModules are plugged into the pipeline to process a request at any point of the ASP.NET life cycle. A HttpHandler is responsible for producing a response/output for a request.

IIS (Microsoft's web server technology) will create an incoming request for ASP.NET, which subsequently will start processing the request and eventually initialize the HttpApplication (which is the default handler) and create a response:

The key thing to know is that ASP.NET can only handle requests which IIS forwards to it. This is determined by the registered HttpHandlers (e.g. by default a request to a .htm file is not handled by ASP.NET).

And finally, MVC is only one of potentially many registered handlers in the ASP.NET pipeline.

This is crucial to understand the impact of different error handling methods.

Breaking down the options

HandleErrorAttribute

The HandleErrorAttribute is an MVC FilterAttribute, which can be applied to a class or a method:

namespace System.Web.Mvc
{
    [AttributeUsage(
        AttributeTargets.Class | AttributeTargets.Method,
        Inherited = true,
        AllowMultiple = true)]
    public class HandleErrorAttribute : FilterAttribute, IExceptionFilter
    {
        // ...
    }
}

It's error handling scope is limited to action methods within the MVC framework. This means it won't be able to catch and process exceptions raised from outside the ASP.NET MVC handler (e.g. exceptions at an earlier stage in the life cycle or errors in other handlers). It will equally not catch an exception if the action method is not part of the call stack (e.g. routing errors).

Additionally the HandleErrorAttribute only handles 500 internal server errors. For instance this will not be caught by the attribute:

[HandleError]
public ActionResult Index()
{
    throw new HttpException(404, "Not found");
}

You can use the attribute to decorate a controller class or a particular action method. It supports custom error pages per exception type out of the box:

[HandleError(ExceptionType = typeof(SqlException), View = "DatabaseError")]]

In order to get the HandleErrorAttribute working you also need to turn customErrors mode on in your web.config:

<system.web>
    <customErrors mode="On" />
</system.web>

Use case

The HandleErrorAttribute is the most limited in scope. Many application errors will bypass this filter and therefore it is not ideal for global application error handling.

It is a great tool for action specific error handling like additional fault tolerance for a critical action method though.

Controller.OnException Method

The OnException method gets invoked if an action method from the controller throws an exception. Unlike the HandleErrorAttribute it will also catch 404 and other HTTP error codes and it doesn't require customErrors to be turned on.

It is implemented by overriding the OnException method in a controller:

protected override void OnException(ExceptionContext filterContext)
{
    filterContext.ExceptionHandled = true;

    // Redirect on error:
    filterContext.Result = RedirectToAction("Index", "Error");

    // OR set the result without redirection:
    filterContext.Result = new ViewResult
    {
        ViewName = "~/Views/Error/Index.cshtml"
    };
}

With the filterContext.ExceptionHandled property you can check if an exception has been handled at an earlier stage (e.g. the HandleErrorAttribute):

if (filterContext.ExceptionHandled)
    return;

Many solutions on the internet suggest to create a base controller class and implement the OnException method in one place to get a global error handler.

However, this is not ideal because the OnException method is almost as limited as the HandleErrorAttribute in its scope. You will end up duplicating your work in at least one other place.

Use case

The Controller.OnException method gives you a little bit more flexibility than the HandleErrorAttribute, but it is still tied to the MVC framework. It is useful when you need to distinguish your error handling between regular and AJAX requests on a controller level.

Application_Error event

The Application_Error method is far more generic than the previous two options. It is not limited to the MVC scope any longer and needs to be implemented in the Global.asax.cs file:

protected void Application_Error(Object sender, EventArgs e)
{
    var raisedException = Server.GetLastError();

    // Process exception
}

If you've noticed it doesn't come from an interface, an abstract class or an overriden method. It is purely convention based, similar like the Page_Load event in ASP.NET Web Forms applications.

Any unhandeled exception within ASP.NET will bubble up to this event. There is also no concept of routes anymore (because it is outside the MVC scope). If you want to redirect to a specific error page you have to know the exact URL or configure it to co-exist with "customErrors" or "httpErrors" in the web.config.

Use case

In terms of global error logging this is a great place to start with! It will capture all exceptions which haven't been handled at an earlier stage. But be careful, if you have set filterContext.ExceptionHandled = true in one of the previous methods then the exception will not bubble up to Application_Error.

However, for custom error pages it is still not perfect. This event will trigger for all ASP.NET errors, but what if someone navigates to a URL which isn't handled by ASP.NET? For example try navigating to http://{your-website}/a/b/c/d/e/f/g. The route is not mapped to ASP.NET and therefore the Application_Error event will not be raised.

customErrors in web.config

The "customErrors" setting in the web.config allows to define custom error pages, as well as a catch-all error page for specific HTTP error codes:

<system.web>
    <customErrors mode="On" defaultRedirect="~/Error/Index">
        <error statusCode="404" redirect="~/Error/NotFound"/>
        <error statusCode="403" redirect="~/Error/BadRequest"/>
    </customErrors>
<system.web/>

By default "customErrors" will redirect a user to the defined error page with a HTTP 302 Redirect response. This is really bad practise because the browser will not receive the appropriate HTTP error code and redirect the user to the error page as if it was a legitimate page. The URL in the browser will change and the 302 HTTP code will be followed by a 200 OK, as if there was no error. This is not only confusing but has also other negative side effects like Google will start indexing those error pages.

You can change this behaviour by setting the redirectMode to "ResponseRewrite":

<customErrors mode="On" redirectMode="ResponseRewrite">

This fixes the initial problem, but will give a runtime error when redirecting to an error page now:

Runtime Error

An exception occurred while processing your request. Additionally, another exception occurred while executing the custom error page for the first exception. The request has been terminated.

This happens because "ResponseRewrite" mode uses Server.Transfer under the covers, which looks for a file on the file system. As a result you need to change the redirect path to a static file, for example to an .aspx or .html file:

<customErrors mode="On" redirectMode="ResponseRewrite" defaultRedirect="~/Error.aspx"/>

Now there is only one issue remaining with this configuration. The HTTP response code for the error page is still "200 OK". The only way to fix this is to manually set the correct error code in the .aspx error page:

<% Response.StatusCode = 404; %>

This is already pretty good in terms of custom error pages, but we can do better!

Noticed how the customErrors section goes into the system.web section? This means we are still in the scope of ASP.NET.

Files and routes which are not handled by your ASP.NET application will render a default 404 page from IIS (e.g. try http://{your-website}/not/existing/image.gif).

Another downside of customErrors is that if you use a HttpStatusCodeResult instead of throwing an actual exception then it will bypass the ASP.NET customErrors mode and go straight to IIS again:

public ActionResult Index()
{
    return HttpNotFound();
    //throw new HttpException(404, "Not found");
}

In this case there is no hack which can be applied to display a friendly error page which comes from customErrors.

Use case

The customErrors setting was for a long time the best solution, but still had its limits. You can think of it as a legacy version of httpErrors, which has been only introduced with IIS 7.0.

The only time when customErrors still makes sense is if you can't use httpErrors, because you are running on IIS 6.0 or lower.

httpErrors in web.config

The httpErrors section is similar to customErrors, but with the main difference that it is an IIS level setting rather than an ASP.NET setting and therefore needs to go into the system.webserver section in the web.config:

<system.webServer>
    <httpErrors errorMode="Custom" existingResponse="Replace">
      <clear/>
      <error
        statusCode="404"
        path="/WebForms/Index.aspx"
        responseMode="ExecuteURL"/>
    </httpErrors>
<system.webServer/>

It allows more configuration than customErrors but has its own little caveats. I'll try to explain the most important settings in a nutshell:

httpErrors can be inherited from a higher level (e.g. set in the machine.config)
Use the <remove/> tag to remove an inherited setting for a specific error code.
Use the <clear/> tag to remove all inherited settings.
Use the <error/> tag to configure the behaviour for one error code.
responseMode "ExecuteURL" will render a dynamic page with status code 200.
- The workaround to set the correct error code in the .aspx page works here as well.
responseMode "Redirect" will redirect with HTTP 302 to a URL.
responseMode "File" will preserve the original error code and output a static file.
- .aspx files will get output in plain text.
- .html files will render as expected.

The main advantage of httpErrors is that it is handled on an IIS level. It will literally pick up all error codes and redirect to a friendly error page. If you want to benefit from master pages I would recommend to go with the ExecuteURL approach and status code fix. If you want to have rock solid error pages which IIS can serve even when everything else burns, then I'd recommend to go with the static file approach (preferably .html files).

Use case

This is currently the best place to configure friendly error pages in one location and to catch them all. The only reason not to use httpErrors is if you are still running on an older version of IIS (< 7.0).

Custom HttpModule

Last but not least I would like to quickly touch on custom HttpModules in ASP.NET. A custom HttpModule is not very useful for friendly error pages, but it is a great location to put global error logging in one place.

With a HttpModule you can subscribe to the OnError event of the HttpApplication object and this event behaves same way as the Application_Error event from the Global.asax.cs file. However, if you have both implemented then the one from the HttpModule gets called first.

The benefit of the HttpModule is that it is reusable in other ASP.NET applications. Adding/Removing a HttpModule is as simple as adding or removing one line in your web.config:

<system.webServer>
    <modules>
        <add name="CustomModule" type="SampleApp.CustomModule, SampleApp"/>
    </modules>
</system.webServer>

In fact someone has already created a powerful and reusable error logging module and it is open source and called ELMAH. Be sure to check out elmah.io as well.

If you need to create application wide error logging, I highly recommend to look at this project!

Final words

I hope this overview was helpful in explaining the different error handling approaches and how they are linked together.

Each of the techniques has a certain use case and it really depends on what requirements you have. If you have any further questions feel free to ask me here or via any of the social media channels referenced on my about page.

EDIT: There is a new blog post on error handling in ASP.NET Core.

Guard clauses without test coverage, a common TDD pitfall

Thu, 19 Mar 2015 00:00:00 +0000

Today I wanted to blog about a little mistake which can easily creep into otherwise good TDD practices.

For this demo I'd like to start with an empty unit test project and initialize a subject under test. Then I add enough code to make the solution build:

namespace MyClassLibrary
{
    public class DomainClass { }

    [TestFixture]
    public class DomainClassTests
    {
        [Test]
        public void Test1()
        {
            var sut = new DomainClass();
        }
    }
}

For this example the method under test will have some very trivial business logic:

Accept an argument
Check if the argument is NULL and return early, otherwise proceed
Call into a dependency
Return a result

Let's finish the first test by checking for the simple case of a NULL argument:

[TestFixture]
public class DomainClassTests
{
    [Test]
    public void DoSomeWork_With_Null_Input_Will_Return_Null()
    {
        // Arrange
        var sut = new DomainClass();

        // Act
        var actual = sut.DoSomeWork(null);

        // Assert
        Assert.IsNull(actual);
    }
}

In true TDD practise you'd write just enough code to compile the project first, then let the test fail and at last return null to make it pass:

public class DomainClass
{
    public object DoSomeWork(object arg)
    {
        return null;
    }
}

However, at this point I have often seen developers implememnt a little bit more code than actually required by the test:

public class DomainClass
{
    public object DoSomeWork(object arg)
    {
        // This guard clause has not been enforced by a particular unit test yet:
        if (arg == null)
            return null;

        return input;
    }
}

It is very tempting to write a bit more code if you already know what the desired end result should look like. Guard clauses are so trivial that it makes it a very common mistake.

Someone might argue that the code is justified, because if you comment out the if statement the test will fail.

This is not true though, because the entire condition as a whole is not required yet and as we have proven above, a simple return null; statement satisfies the test as well.

This little mistake seems very harmless now, but it could inroduce a potential NullReferenceException later in the project when the first developer starts to re-factor code and remove redundant code.

This will essentially happen when we continue implementing the remaining unit tests and business logic for the method under test:

public object DoSomeWork(object arg)
{
    if (arg == null)
        return null;

    var result = _dependecy.ProcessData(arg);
    return result;
}

Now at this point I could delete the if statement together with the "return null" and the first test will still succeed:

public object DoSomeWork(object arg)
{
    // if (input == null)
    //    return null;

    var result = _dependecy.ProcessData(arg);
    return result;
}

Why? Well because in the first test we didn't set up a stub for the dependency yet:

[TestFixture]
public class DomainClassTests
{
    [Test]
    public void DoSomeWork_With_Null_Input_Will_Return_Null()
    {
        // Arrange
        var sut = new DomainClass();

        // Act
        var actual = sut.DoSomeWork(null);

        // Assert
        Assert.IsNull(actual);
    }
}

It means that var result = _dependecy.ProcessData(arg); will return null and make the test go green.

This is why it is best to not introduce little short cuts in TDD and be careful during your code reviews as well!

Making Font Awesome awesome - Using icons without i-tags

Wed, 04 Mar 2015 00:00:00 +0000

Font Awesome is truly an awesome library. It is widely used across many websites and has made a web developer's life so much easier!

There is one thing which I don't find that awesome though. It pollutes your HTML with a lot of styling markup.

Let me explain the problem with an example...

The Problem

The default way to include an icon into your website is by adding an <i> tag into your HTML code; like this:

<i class="fa fa-car"></i>

For example, the car icon will render into:

This is great, but now you end up with many empty <i> tags in your HTML code, which have no structural purpose in the document whatsoever. Even worse, these tags are tightly coupled to a concrete theming technology, Font Awesome in this case.

This is not great and I personally think HTML markup should be theme agnostic.

Solutions out in the wild

A quick Google search brought up some solutions to the problem. None of them without any drawback though.

#1 Shipping a modified copy of Font Awesome

By providing your own copy of Font Awesome you'd obviously be able to do what you like and easily solve the problem, however this appraoch has some major issues:

I don't want to maintain a custom copy of Font Awesome
The library is quite big and I really want to benefit from the public CDN

#2 Add the classes to the target element

The next best suggestion was to attach the classes to the target tag instead of the <i> tag.
For example:

<a class="fa fa-car" href="#">This is a link</a>

Result:
This is a link

As you can see this works, but Font Awesome has become the font for the entire anchor tag now. The browser ends up rendering the text in the default font, which mostly comes from the serif family and is rarely what you want.

#3 Setting the unicode character in your custom CSS

The most popular solution was to set the Unicode character for the content property of the ::before attribute:

a:before {
    font-family: FontAwesome;
    content: "\f1b9";
    display: inline-block;
    padding-right: 3px;
    vertical-align: middle;
}

The result will be visually perfect, but there are a few things which I don't like about this appraoch:

I have to add custom CSS for each tag which I want to decorate
The unicode character can change
It is not readable! CSS code is still code and deserves the same best practices like any other code

Analysing the Font Awesome CSS classes

After I wasn't really satisfied with any of the proposed solutions I tried to find a better one.

Let's break it down

Each icon consists of two CSS classes - the shared "fa" class and an icon-specific class like "fa-car". Setting only the icon-specific class will result in a not very meaningful unicode character:

<a class="fa-car" href="#">This is a link</a>

Result:
This is a link

The icon isn't what we want, but at least the tag's original font remains as is. Using the Google Chrome developer tools I can quickly confirm that the icon-specific class is not doing any harm to the original tag:

Evidently this class only adds the content to the ::before attribute of the target element. The conclusion is that the actual styling gets applied via the "fa" class:

Now this makes sense. The content from the ::before attribute gets rendered inside the original tag and therefore also picks up the styling from the fa class.

Everything from the fa class could equally go into the icon class as part of the ::before attribute, but I can see why the Font Awesome team has extracted it into a shared class, because it is the same for every icon and would be otherwise a maintenance nightmare.

The ultimate solution (or at least the best I came up with)

After inspecting the code I realised that all I need is to create one additional class in a custom style sheet to fix the problem:

.icon::before {
    display: inline-block;
    margin-right: .5em;
    font: normal normal normal 14px/1 FontAwesome;
    font-size: inherit;
    text-rendering: auto;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
    transform: translate(0, 0);
}

It is a copy-paste of the original fa class with 3 changes to it:

I named the class "icon" to avoid a conflict
I attached the ::before attribute
I added a small margin-right, so that the icon doesn't stick to the original tag

With this little trick it seems like I am able to tick all the boxes:

I can continue to use the original Font Awesome CDN library
I don't need to add empty i-tags
It doesn't interfere with other CSS on targeted elements
I don't have to make a custom CSS change for each individual icon (unlike the unicode appraoch)
The code is readable!

Now it can be used like this:

<a class="icon fa-car" href="#">This is a link</a>

Result:
This is a link

Especially the fact that I don't have to use the unicode characters is very appealing. When I read the code "icon fa-car" it is pretty obvious what the rendered result will look like.

Conclusion

At the moment I haven't found a drawback with this appraoch, but there might be something which I have overlooked. If you know of any issues or if you know an even better appraoch, then please let me know! Feedback is much appreciated!

PHP UK Conference 2015

Thu, 26 Feb 2015 00:00:00 +0000

Last week was my first time at the PHP UK Conference in London. As a .NET developer who is very new to the PHP community I didn't have any particular expectations, but I think this year was a great time to be there!

The Venue

The conference took place on Thursday and Friday (apparently for the first time, because in previous years it was Friday and Saturday) at The Brewery, which is a great venue at a very central location in the city of London.

Also worth noting is that this year was the 10^th anniversary of the event with a record high of more than 700 attendees coming from many different countries around the world. The hosting was excellent, food and drinks were available throughout the entire day and in the evenings they had many hundreds of free beer up for grab to celebrate this occasion.

As if this was not enough, the organisers even rented out the All Star Lanes in Brick lane to continue the celebration with some free bowling, free karaoke, more beer, more food and a cake on Thursday night.

I have to admit that due to a light cold I didn't make the most out of it, but I still had a fantastic time!

The Tracks

Both days started off with two great key notes. The first keynote was by coderabbi, who walked us through some of his own experiences, spoke about code reviews and peer coding and eventually spread his wisdom over a packed room.

On Friday Jenny Wong, a developer and (according to her own words) a "community junkie" kicked off the second day with a very light hearted and extremely inspirational speech about bringing developer communities together.

Integrating Communities

The message which I took away is, that it doesn't matter who you are, which technology you use or how experienced you are. We are all connected through the same passion which brought us together - the passion for coding. Sadly we are in an industry where people burn out every day. Rather than belittling and laughing at other developers (even if it is a Wordpress developer :)) we should help each other and help this community to grow and not shrink.

Definitely one of the sessions which got stuck most with me!

Some other sessions which I really enjoyed were "Composer Best Practices", "Application Logging & Logstash", "Modern Software Architectures" and "HHVM at Etsy".

They were perhaps not the most technically advanced sessions, but they all had a great speaker, who was able to present something interesting from a personal experience and give some insights on the topic which were beyond the usual literature which you'll find in books or on the internet.

I won't rehash all of them in this blog post, but I picked a few where I wanted to share some interesting take aways!

Versioning

In the session about Composer best practices Jordi Boggiano started off by explaining the meaning and importance of semantic versioning.

We all know what a build number looks like. It consists at least of three numbers, separated by a dot, where the parts usually stand for the major number, the minor number and a patch or sometimes referred to as the build number.

e.g.: {major}.{minor}.{patch}

What semantic versioning does is to give a clearly defined meaning to each of these numbers with the aim to ease the upgrade and migration pain when developers use 3^rd party code libraries.

In a nutshell, major stands for compatibility breaks (even the smallest!), minor stands for new features (without affecting other features) and patch denotes bug fixes (for existing features). Sounds simple, but this convention really makes a difference when you are dealing with someone else's code and you want to understand the implications of a NuGet package update or similar.

It definitely changed my view on versioning artifacts and if this was new to you as well, then I hope it makes as much sense to you as it did to me :).

HHVM and PHP 7

The other talk I wanted to quickly share with you was HHVM at Etsy by Dan Miller.

For those who don't know what HHVM is, it is a virtual machine which compiles PHP code down into native C++ for better performance. HHVM is written by Facebook in Hack and PHP, and Hack itself is written by Facebook as well.

HHVM stands for HipHop Virtual Machine and is a replacement for the initial HipHop compiler. The major difference is that HHVM is a just-in-time compiler, while HipHop wasn't.

Why a just-in-time compiler for PHP?

Now you probably ask yourself why did Facebook prefer a just-in-time compiler over the other? While there are lots of reasons for it, one of the perhaps minor, but interesting ones was the difficulty to introduce HipHop into the existing development culture. PHP developers are used to drop in replace a file on a server and know that the changes have taken immediate effect. Now with a compilation step in between this was not possible anymore and was a rather huge shift among engineers. A just-in-time compiler on the other hand allowed developers to continue their work flow as they were used to.

HHVM points of interest

Another few interesting things I have learned:

Facebook claims that HHVM increases the throughput between 3 to 6 times. These benchmarks were taken with a much older PHP version though and Etsy's experience was more towards 3x in comparison with PHP 5.
HHVM is used by other big companies like Wikipedia and Baido as well.
HHVM is extremely solid. Etsy had no issues with HHVM whatsoever and did not encounter any bugs, even though their entire code base is in PHP and they expected at least some weird PHP constraints causing errors - but to their pleasant surprise, they didn't.
BUT, they found many issues with 3^rd party modules for HHVM. They had to fix some pretty major bugs for the Memcached and MySQL modules to get them running.
If a HHVM module is not yet in use by another big company, then expect to invest a fair amount of time for bug fixing.
Facebook seems to be very enganged with the project. There were stories told at the conference where people reported a bug and it has been fixed only a few hours later over night. This is kind of cool stuff and music in a developer's ear :)!
HHVM offers a variety of other extremely useful features like:
- HHVM Debugger
  Allows you to set conditional break points, e.g.: only hit the break point for useid=123.
- sgrep
  Great tool for static code analysis which offers a simpler syntax than conventional regex.
  e.g.: sgrep -e 'X && X' will return all the code lines where the left- and right hand statement of a logical AND operator is the same.
- spatch
  Great tool for refactoring your code.
  Good PHP IDEs will offer you refactoring tools as well, but they all rely on text search and replace, why they won't give you 100% confidence that all changes have been made properly in your entire code base.
  e.g.: Remove 2^nd argument from a method, etc.
And last but not least: Some early performance benchmarks between HHVM and PHP 7 showed that PHP 7 gets very close to HHVM. In one benchmark it even outperformed HHVM, but they didn't dived too much into the details and the quality of these figures, so please don't pin this on the wall yet.

Summary

All in all the PHP UK conference was an amazing event and I am glad I had the opportunity to be part of it! Will I go next year again? It is definitely on my list! Hopefully I will see you next year PHP folks!

Hello World

Mon, 16 Feb 2015 00:00:00 +0000

Website has been launched.

More content and small incremental updates will follow over the next few weeks.