Introducing LambdaCD

Sun, 29 Mar 2015 21:50:00 +0000

Preface

In my last post I argued that we should start treating our build pipelines as code, as potentially complex applications that serve a single purpose: To test, publish and deploy your software.

What I didn’t mention was how to get there. Do I want everyone to throw away their beloved Jenkins and go develop their own buildservers?

Well, yes, but I also want to do the heavy lifting for you:

Introducing LambdaCD

LambdaCD is a library that gives you

An execution engine to execute your build steps one after another (and stops on failure)
Persistence and and API to access state and history of your pipeline
A basic UI to visualize and interact with your pipeline.
A few basic building-blocks for your pipeline, e.g. manual triggers, parallel execution of steps, …

What is left for you to do is code the structure of your pipeline and your build-steps. Build steps are regular Clojure functions that take input data and return a map with results. The structure of your pipeline is a nested list of such functions.

In short, this is what you code:

(defn some-step-that-does-nothing [& _]
  {:status :success})

(defn some-step-that-echos-foo [_ ctx]
  (shell/bash ctx "/" "echo foo"))
(defn some-step-that-echos-bar [_ ctx]
  (shell/bash ctx "/" "echo bar"))

(defn some-failing-step [_ ctx]
  (shell/bash ctx "/" "echo \"i am going to fail now...\"" "exit 1"))

(def pipeline-def
  `(
    lambdacd.manualtrigger/wait-for-manual-trigger
    some-step-that-does-nothing
    (in-parallel
      some-step-that-echos-foo
      some-step-that-echos-bar)
    lambdacd.manualtrigger/wait-for-manual-trigger
    some-failing-step
  ))

And this is what you get: (yes, I know this isn’t stellar web design)

Getting started

The easiest way to get started is to use Leiningen to generate a project for you: lein new lambdacd <SOME-PROJECT-NAME>. This will create a project for you that you can run using lein run. lein uberjar creates a self-contained jar-file that can run on any server with java installed. That’s all you need for your very own, custom built build-server!

Now go ahead, have a look at the generated code, play around a bit and then come back, tell me what you think.

Build Pipelines as Code

Sun, 29 Mar 2015 21:39:00 +0000

The problem

Do you have a build pipeline that is more complex than just build, test, deploy? Are you happy with it? Do you fully understand what all those build-steps are doing, what input they take and where this input comes from? Do you like all those fragile plugins you need just to convince your CI server to do the thing you want to do? Do you like clicking your way through a web interface to get stuff done? Is your build pipeline versioned? Is it tested?

Or maybe your organization is trying out microservices. How easy is it for you to set up a pipeline for a new microservice? Have you figured out how to reuse aspects that are common for all services, how to share them between teams and still have stable interfaces, abstractions and dependencies?

An idea

As developers, we know those problems are already solved in other parts of our world: Our code is readable, it has abstractions, testing, dependency management and versioning works effortlessly. Or servers are immutable and their deployment automated, everything built with the same concepts and tools as is our code.

One thing is missing: The glue, the build pipeline between a commit and your critical live systems. The thing that, supposedly, is the highest priority of the development team.

What we need is a way to put our build pipelines where our business logic, our database-layout and our server configuration already is: in code!

What does that get us? It gets us everything we love about code: We can structure it, version it, test it, share it with others any way we like! We can take advantage of all the tools and libraries we know and love!

You want to find out if your deployment really stops when a smoketest fails on staging? A small test with the right mock will tell you!

All your development teams should use the same process to deploy to your private cloud? Just put everything into a small library for everyone to include in their pipelines!

Your management requires your pipeline to behave differently on mondays, except for rainy days? You can build that! (not saying you should)

Simply put: Our build pipeline can be just like any other piece of software we release, with the same tools, methods and quality we are used to!

Next up: A clojure library that let’s you do all that!

Florian Sellmayr

Introducing LambdaCD

Preface

Introducing LambdaCD

Getting started

Build Pipelines as Code

The problem

An idea