Antoine Kalmbach's website

Guile fun times, and blog update

Antoine Kalmbach — Tue, 08 Dec 2020 13:36:12 +0000

Holy crap, it’s been five years! Time passes so quickly. What can happen in five years? Well, during these five years I met my wife, we moved not three but four times, we bought our own apartment, our daughter was born… many things indeed! Here’s to hoping the next five years will be as fun and exciting.

On this site I’ve posted twenty something blog entries plus a smattering of other writings. I’ve picked up the pace this year after having a renewed interest for this site. It’s still running on Jekyll and hosted on GitHub pages, so that hasn’t changed.

I’m still working at the same place as before, though I’ve changed roles and responsibilities some four to five times. I have not had the chance to become bored, so that’s nice.

Lately I’ve been playing around with GNU Recutils and became maintainer of it’s Emacs major mode rec-mode. In passing I’ve also dabbled in writing Guile Scheme bindings for librec, the library powering recutils. Read more about this here!

Recutils, GOOPS and virtual slots

Antoine Kalmbach — Sun, 06 Dec 2020 11:56:09 +0000

For the past month or so I’ve been contributing to GNU Recutils, a set of tools for editing human-readable plain text databases. It’s a cool project in its own right, I’ve been using recutils myself for tracking workouts and storing cooking recipes. The cool part of it is its attempt to be both human-readable and machine-readable, which makes it very easy to use programmatically and then with a simple text editor.

The powerful querying facilities of recutils is what turns it into a thing of beauty. In particular, selection expressions are expressions for querying recfiles. For instance, here’s how I would query exercises in my workout log for squats:

recsel -t Exercise -e "Name ~ 'Squat'" workouts.rec

This would match records of type Exercise where the Name field matches regular expressions, so Squat will match all exercise varieties with the word Squat in it.

The machine readability makes it easy to write programs or tools that interact with recfiles. I’ve become maintainer of the Emacs recfile major mode rec-mode. The major mode makes heavy use of the command line tools of the recutils suite to do provide automatic fixing and parsing of recfiles.

if it’s possible to put Lisp in it, someone will

For fun and profit, I’ve also been writing GNU Guile bindings for librec, the library powering recutils itself. The bindings actually interface with the C library directly using Guile’s amazing C extensions. I was interested in using recfiles in a Guile program, and while it would not have been too difficult to write a parser myself, I thought it was more important to not write one myself. What is more, Guile makes it almost too easy to wrap libraries, I had a functioning Scheme interface for parsing records in less than an hour.

Let’s explore what that interface looks like. We start with the simplest data type in librec, fields.

A recutils record is defined as an ordered collection of fields. Below is a record of three fields:

Book: Structure and Interpretation of Computer Programs
Author: Harold Abelson
Author: Gerald Sussman

The inner field type of librec is defined as rec_field_t, which is an opaque data type wrapping rec_field_s:

typedef struct rec_field_s *rec_field_t;

The underlying rec_field_s structure is a bit more complicated since it includes location data for the field, but for our example imagine it contains just the fields name and value, which are null-terminated strings. You don’t need to know anything about that, since librec offers an extensive API for working with the opaque types.

To make a new field, you would write:

rec_field_t field = rec_field_new("Author", "Harold Abelson");

To get the value and name, you use rec_field_value and rec_field_name:

const char *name = rec_field_name(field); /* "Author" */
const char *value = rec_field_value(field); /* "Harold Abelson */

To modify its name or value, you can use:

rec_field_set_name(field, "Book");
rec_field_set_value(field, "Structure and Interpretation of Computer Programs");

How do we wrap these into Guile, using C extensions? To start with, we can simply make some Scheme methods that work with plain pointers and pass that pointer value around.

SCM_DEFINE (scm_field_new, "new-field", 2, 0, 0, (SCM scm_name, SCM scm_value),
            "Make a new field from a string and value.")
{
  SCM_ASSERT_TYPE(scm_is_string(scm_name), scm_name, 1, "new-field", "string");
  SCM_ASSERT_TYPE(scm_is_string(scm_value), scm_value, 2, "new-field", "string");

  const char *name = scm_to_utf8_string(scm_name);
  const char *value = scm_to_utf8_string(scm_value);

  rec_field_t field = rec_field_new (name, value);

  if (!field)
    return SCM_BOOL_F;

  return scm_from_pointer(field, destroy_field);
}

This defines two functions: destroy_field for letting the garbage collector get rid of unused fields, and then a scm_field_new function defined using the SCM_DEFINE macro. The procedure is straightforward: assert both parameters are strings, convert to const char*, create the field and return it if it was successful, otherwise return Scheme false #f. The last bit creates a pointer object to store the pointer address, and passes the destroy_field as the finalizer parameter for the garbage collector.

In the Guile REPL, it looks like this:

scheme@(recutils)> (new-field "foo" "bar")
$2 = #<pointer 0x7fc0654040f0>

OK, it seems to be a pointer all right. Let’s define some helper methods to work with that:

SCM_DEFINE(scm_field_get_name, "field-name", 1, 0, 0, (SCM ptr),
           "Get the name of a field")
{
  rec_field_t field = (rec_field_t)scm_to_pointer(ptr);

  const char *name = rec_field_name(field);
  
  return scm_from_utf8_string(name);
}

SCM_DEFINE(scm_field_get_value, "field-value", 1, 0, 0, (SCM ptr),
           "Get the value of a field")
{
  rec_field_t field = (rec_field_t)scm_to_pointer(ptr);

  const char *value = rec_field_value(field);
  
  return scm_from_utf8_string(value);
}

Loading this extension into the REPL, we get

scheme@(recutils)> (new-field "foo" "bar")
$1 = #<pointer 0x7fa123d0b980>
scheme@(recutils)> (field-name $1)
$2 = "foo"
scheme@(recutils)> (field-value $1)
$3 = "bar"

What about modifying the field? Well, that’s easy:

SCM_DEFINE(scm_field_set_name, "set-field-name!", 2, 0, 0, (SCM ptr, SCM scm_name),
           "Set the name of a field")
{
  SCM_ASSERT_TYPE(scm_is_string(scm_name), scm_name, 1, "set-field-name!", "string");
  rec_field_t field = (rec_field_t)scm_to_pointer(ptr);

  const char *name = scm_to_utf8_string(scm_name);

  bool result = rec_field_set_name(field, name);

  return scm_from_bool(result);
}


SCM_DEFINE(scm_field_set_value, "set-field-value!", 2, 0, 0, (SCM ptr, SCM scm_value),
           "Set the value of a field")
{
  SCM_ASSERT_TYPE(scm_is_string(scm_value), scm_value, 1, "set-field-value!", "string");
  rec_field_t field = (rec_field_t)scm_to_pointer(ptr);

  const char *value = scm_to_utf8_string(scm_value);

  bool result = rec_field_set_value(field, value);

  return scm_from_bool(result);
}

Using all this in the REPL yields:

scheme@(recutils)> (new-field "foo" "bar")
$1 = #<pointer 0x7ffcac406530>
scheme@(recutils)> (set-field-name! $1 "Blah")
$2 = #t
scheme@(recutils)> (set-field-value! $1 "Test")
$3 = #t
scheme@(recutils)> (field-name $1)
$4 = "Blah"
scheme@(recutils)> (field-value $1)
$5 = "Test"

There we go!

the smell of raw pointers

OK, this looks great. But somehow it feels funny to pass a raw pointer object around as a parameter. Ideally, I’d like to define some sort of structure that wraps the raw pointer into something less raw. Well, turns out Guile has exactly that in the define-wrapped-pointer-type macro! With the above constructor and procedures, we can go further:

(define-wrapped-pointer-type
  field-ptr field-ptr? wrap-field-ptr unwrap-field-ptr
  (lambda (ptr port)
    (format port "#<field-ptr name=~s value=~s 0x~x>"
            (field-name (unwrap-field-ptr ptr))
            (field-value (unwrap-field-ptr ptr))
            (pointer-address (unwrap-field-ptr ptr)))))

What the macro defines are a type name (field-ptr), a predicate (field-ptr?), methods for wrapping and unwrapping, and lastly a printer for pretty printing our pointer. The printer outputs a human readable representation of the printer, in which we leverage the procedures defined above, field-name and field-value.

scheme@(recutils)> (wrap-field-ptr (new-field "Author" "Harold Abelson"))
$2 = #<field-ptr name="Author" value="Harold Abelson" 0x7f8a6950b2d0>
scheme@(recutils)> (field-ptr? $2)
$3 = #t
scheme@(recutils)> (unwrap-field-ptr $2)
$4 = #<pointer 0x7f8a6950b2d0>

This makes it a bit easier to pass around field values so that we can treat them like structures, or records in Scheme parlance. That said, constructing the values is still a bit tedious, especially now that our Scheme user would have to constantly wrap and unwrap values if they are to work with a field.

What if we could work with fields as if they were pure Scheme objects and the underlying machinery – pointers and so forth – would be hidden from us? Well, we can use GOOPS, but first let’s digress into the exciting world of FFI.

why not dynamic FFI?

These days the Guile manual recommends using Dynamic FFI when working with a foreign function interface. That is, the above examples are just C code, but we could have done the same with just regular Scheme using the (system foreign) module. This is what I would do in many other languages (Common Lisp, Python, and so on…). In such a case, I could make my Scheme module completely separate from recutils and librec, since I just need the dynamic library libguile-recutils.so for it to functions. But there are subtle reasons why writing these extensions in C is a good idea.

As I went ahead and wrote the bindings, I had a curious thought: I’m writing functionality for working with recfiles from Guile. But what about adding Guile facilities to recutils? What about letting recutils users extend the programs using Scheme? Wouldn’t it be cool if instead of recutils selection expressions I could pass Scheme programs as the query language? Indeed, this was a topic worth exploring!

The consequence of this was that now I was adding code to recutils itself to link against Guile, which means I will already have a dependency to the Guile C library libguile. So, since I’m now already working with the C API of Guile, limiting myself to the strange world of dynamic FFI was starting to feel rather tedious.

From the start I wanted to work with the real deal: the wrapper types of the Guile extensions would be real wrappers. Each field in Scheme would be represented by a librec C struct underneath. This is so that I can leverage the bidirectional design above, and there is no need to parse or convert values twice when crossing language barriers. So, how do we make a Scheme API that is both nice to use and still C structs underneath? Well, the answer is GOOPS and object-oriented programming!

GOOPS, virtual slots, and you

Working with raw pointers and even pointer records can be painful. It would be much better if we could make fields like this:

(make <field> #:name "Author" #:value "Gerald Sussman")

This is a GOOPS class, of type <field>. The constructor has two keyword arguments #:name and #:value for the rec names.

How can we get a class that has both getters and setters (in terms of slot-ref and slot-set!) that work on the underlying pointer? Easy enough, the answer is virtual slots! If we were to define an ordinary class with slots name and value, Guile would allocate memory for those and if we are to juggle the pointer alongside all of this, both the name and value would be in two places: once, behind the pointer (in C world) and in Scheme, as a slot in the class.

But first, how do we create a class <field> that wraps a pointer? Easy enough, we can use #:init-form as the slot option:

(define-class <field> ()
  ;; Internal pointer type.
  (ptr #:init-form (wrap-rec-ptr (new-field "" ""))
       #:init-keyword #:ptr)

The use of #:init-form causes the following expression to be evaluated every time a new class is instantiated, creating a field with an empty name and value. To get the signature we desire above, we need to use virtual accessors. These let us override the getter and setter #:slot-ref and #:slot-set! respectively which will work on the raw pointer, instead of occupying memory like a normal slot would. This is achieved using #:allocation #:virtual:

(define-class <field> ()
  (ptr #:init-form (wrap-field-ptr (new-field "" ""))
       #:init-keyword #:ptr)

  (name #:init-keyword #:name #:allocation #:virtual
        #:accessor field-name
        #:slot-ref (lambda (field)
                     (%field-name (unwrap-field-ptr (slot-ref field 'ptr))))
        #:slot-set! (lambda (field name)
                      (set-field-name!
                       (unwrap-field-ptr (slot-ref field 'ptr)) name)))
  
  (value #:init-keyword #:value #:allocation #:virtual
         #:accessor field-value
         #:slot-ref (lambda (field)
                      (%field-value (unwrap-field-ptr (slot-ref field 'ptr))))
         #:slot-set! (lambda (field value)
                       (set-field-value! (unwrap-field-ptr (slot-ref field 'ptr)) value))))

Note that the procedures we defined previously in C were renamed to %field-value since it would otherwise conflict with the #:accessor slot option.

So using #:virtual lets us write GOOPS classes and not worry about double allocation. It looks like a regular GOOPS class but actually it is modifying a pointer to a C struct underneath using a C API. Moreover, the biggest benefit of this is the ability to pass values in the constructor. If we didn’t have #:virtual, we’d have to write separate accessor methods like this:

(define-method (f-name (field <field>))
  (%field-name (unwrap-field-ptr (slot-ref field 'ptr))))

(define-method ((setter f-name) (field <field>) name)
  (set-field-name! (unwrap-field-ptr (slot-ref field 'ptr)) name))

But the problem with this and any other approach is that you’d still have memory allocated for the slots. All <field>s will have unnecessary name and value slots allocated. I think the only way to get this behaviour if #:virtual were not available would be to create a custom method for initialize. I think the same applies in other CLOS-like systems (and CLOS itself), but I’m not sure.

context is everything, friends

I don’t think many Guile users will find Scheme bindings for recutils that useful in itself, as a library. Guix uses recfiles in its search output, but its record generation is hand-written, usage not deep enough to warrant using the library.

But I think a case can be made for recutils itself, that is, if recutils were to develop extensibility via Guile, the extension mechanism can load the recutils Scheme module as its base runtime. I discussed the idea over at IRC with Jose Marchesi, the Recutils author and maintainer, and he thought it was a good idea as long as there’s someone there to maintain it.

Maybe this will fly, I don’t know. I don’t see any big technical barriers for it to not work, even if it amounts to just adding Scheme bindings without extending recutils itself. That said, every now and then I’m running into the limitations of selection expressions, so being able to use Scheme as a measure of last resort would be interesting, if nothing else.

As of early December 2020 I have bindings for parsing and creating records and fields, so expect an early release of the Scheme bindings to appear within the next few months.

Have I mentioned I also plan to make Common Lisp bindings as well? Well, now I have, but that’s another story!

A Guile test runner with an exit code

Antoine Kalmbach — Fri, 30 Oct 2020 19:02:40 +0000

The other day I was playing around with the Meson build system for building GNU Guile projects. It was actually a lot of fun, Meson build files are in my opinion much easier to manage than Autoconf, although Autoconf the de facto build system for Guile.

It was all fun and nice until I ran into a problem with running unit tests using meson test. Meson expects unit tests to return exit code 1 to signal a failure, but the standard Guile test runner that implements SRFI-64 does not have an exit code.

So I wrote the following Scheme snippet to make a small script that you can invoke to run tests from an arbitrary Scheme file and it returns 1 if the tests fail. That way it can be used in Meson builds to signal unit test failures.

(define-module (test-driver)
  #:declarative? #f
  #:use-module (srfi srfi-64))

(define* (main _ #:rest files)
  "Load each file in FILES while evaluating them within the context of a test
runner."
  (unless (= 1 (length files))
    (display "Usage: test-driver.scm <TEST-FILE>\n"
             (current-error-port))
    (exit 1))
  
  (let ((file (car files)))
    (unless (file-exists? file)
      (display (format #f "File ~A does not exist!~%" file)
               (current-error-port))
      (exit 1))

    (let* ((full (string-append (getcwd) "/" file))
           (runner (test-runner-simple)))
      (test-with-runner runner
        (load full))
      (if (< 0 (test-runner-fail-count runner))
          (exit 1)
          (exit 0)))))

(apply main (command-line))

Invoking it on a file like simple.scm as follows:

(define-module (simple-tests)
  #:use-module (srfi srfi-64))

(test-begin "simple")

(test-equal "ok" 3 3)

(test-equal "bad" 3 4)

(test-end "simple")

produces the exit code 1 as is expected:

$ guile --no-auto-compile test-driver.scm simple.scm
%%%% Starting test simple  (Writing full log to "simple.log")
/Users/akalmbach/code/guile/simple.scm:8: FAIL bad
# of expected passes      1
# of unexpected failures  1

$ echo $?
1

Working with Git and patches in Emacs

Antoine Kalmbach — Mon, 19 Oct 2020 12:17:06 +0000

Git was initially designed around emails. Modern Git forges have co-opted the term pull request to mean using web-based applications for collaborating using Git. This has become the de facto method of collaboration in the Git world. Before that happened, people actually used plain emails when working with decentralized version control system. There was no notion of requiring a web application for collaboration, you simply mailed your changes to someone and they merged them if they wished.

This document demonstrates the way I use email and Git using Emacs.

Why Git and email?

While working with Git these days you cannot but notice the mainstream collaboration model is completely unlike what this document details. You are expected to use web applications like Github or its copies (Gitlab, etc., see above). Lots of people have written about the pros and cons of each, to highlight a few:

Drew Devault in The advantages of email-driven git workflow
Joe Nelson in Mailing lists vs. Github

My own reason for preferring email is that it promotes discussion. Many forges do not actively promote a discussion forum, which is why things like gitter.im exist. Other alternatives is to have an IRC channel, but those are usually too informal to focus on development.

Development forums are essential

With an email-based workflow the place for collaboration supports both discussion and actual development work and review. A project mailing list offers an environment for discussing the project and for sending and reviewing patches.

When patches are mixed with discussion, it makes it easy for people to follow the current zeitgeist of the project. Changes have more context this way. Contrast this to a forge like GitHub where, if the project doesn’t really state anything, you’re suppose to conduct your discussion in issues before making a pull request. But that can lead to confusion: issues usually are synonymous with bug reports. It would be nicer if issues were, well, issues. In an email-based collaboration model, discussion and development happen in the same place, it’s a vibrant bazaar of development.

Email? Seriously?

Ok, email and git isn’t for everybody. That’s fine. The folks at GitHub rightly understood that if one were to make sending pull requests easier it would increase the popularity of Git. The point isn’t in elitism or setting high barriers of entry. I don’t think the point is in filtering beginners either, since I wouldn’t want to be hostile to beginners.

I suppose advocating Git and email in 2020 is in one sense some kind of activism for simpler solutions from a more elegant time. The argument is also interesting from a technological perspective: git send-email is actually a uniform, standard way of collaborating in Git.

As a result, git send-email is actually quite relevant still despite forges having twisted the meaning of pull request. That is why in every forge a pull request is slightly different. Some forges call them merge requests or something similar, and the user experience for making them varies ever so slightly.

There is no standard pull request any more

There is certain nonobvious elegance in the email-based workflow: it actually standardizes on two tools: git send-email and git am. This standardization means that everyone, regardless of their mail client, will be working with a uniform language for collaboration.

Now, many will say that web applications offer a better user experience than emails. That is, first and foremost, a matter of opinion. While web applications offer more graphical bells and whistles, the problem is that in the modern age of a multitude of Git forges every one of them will have slight differences in the language they use and implement pull requests in. So if you regularly collaborate on many platforms the slight idiosyncracies of each forge will get in the way. Not so with git send-email, it’s the same for everyone!

Barriers of entry

Some forges embrace email based collaboration natively: sourcehut is a newish forge from 2018 which is still in alpha as of October 2020, and it actively promotes an email-based workflow. In fact, the people behind sourcehut are behind the tutorial at git-send-email.io.

Even sourcehut recognizes that setting up sendemail in git can be onerous and a lot of people don’t want to work with email. So they built a webpage to handle that part where you can interactively prepare a patch email as if you were using git send-email. It’s still email at the bottom, that’s the cool part!

So as a result, don’t think of git and email as some sort of activism mixed with retrocomputing or text-based interfaces, rather, the goal is to

rely on a standard way of collaboration, and
natively mix discussion and development in the same environment.

Because git send-email encourages the use of mailing lists, you no longer need an official IRC channel or subreddit or gitter room or whatever. And forges like sourcehut have made it easy to separate patches in the web interface for mailing lists, as well as having CI for mailing lists (seriously, how cool is that?).

Git and email in 5 minutes

Email-based workflows in Git are based on two tools:

git send-email for sending changes to someone
git am for applying changes from someone

git send-email is essentially a wrapper around git format-patch. This wrapper calls git format-patch and mails it to the email address in the --to parameter. So

$ git send-email --to="foo@bar.com@" HEAD^

This will call git format-patch HEAD^ and send it to foo@bar.com. Git includes a Perl-based email client that it uses. This mail includes your commit message and the diff of the changes. When foo@bar.com wants to apply your patch, they save the mail and use git am on it.

$ git am patch.mbox

Now the question is, how do you use git send-email and git am from Emacs so that you don’t have to exit to the shell?

Sending patches from Emacs

Those daring enough who have configured Emacs as their email client probably think, ok, what’s the big deal, let me just call git format-patch and then mail the patch myself. The workflow would be thus:

Prepare some changes.
Commit the changes.
Use git format-patch to prepare a file foo.patch.
Send foo.patch as an email to the intended recipient, e.g. by opening it and then calling M-x message-mode which would open it in a major mode for sending emails.

While this makes sense, you’ll still have to edit the intended recipient manually, ensure your subject is formatted properly, and so on. For basic one-off patches this might be sufficient, but you risk deviating from the standard git send-email format. The following is displayed on git-send-email.io, the git & email tutorial:

Warning! Some people think that they can get away with sending patches through some means other than git send-email, but you can’t. Your patches will be broken and a nuisance to the maintainers whose inbox they land in. Follow the golden rule: just use git send-email.

So it’s safe to say we ought to stick to git send-email. But we can use it interactively from within Emacs.

Calling `git send-email` from Emacs

Calling any shell command is pretty easy in Emacs, just type M-! (or M-x shell-command) and type git send-email -1. This will open up a shell command buffer and git send-email will prompt you for the recipient and so on. You can stop this process at any time using C-c C-c if you’re note happy with it. Here’s an example patch generated using git send-email:

From: Antoine Kalmbach <ane@iki.fi>
Date: Mon, 19 Oct 2020 21:47:50 +0300
Subject: [PATCH] Added a line

This should improve the README experience by quite a bit.
---
Not sure about the final wording.
 README.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index f9e3d61..2e4f197 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,5 @@
 # Hello there!
 
 Blah blah blah.
+
+Let's add this line here.

Sometimes it makes sense to annotate your patch with a message that explains what the patch does, but not in a way that it would end up into the commit history. In the example above, that’s the bit under the three dashes ---.

To add such an annotation, git send-email specifies a --annotate flag that will preview your patch in your editor specified by the $EDITOR environment variable. If you have your EDITOR set correctly to e.g. emacsclient, this will work just fine provided you’re running an Emacs server.

Setting Emacs as `$EDITOR`

If you try --annotate without EDITOR set, git send-email will most likely complain that your terminal has no editor specified, and quit. We’re executing a shell command, so we want to tell the terminal that we’re doing this from the editor. There’s an easy way to do this: Emacs has a with-editor-shell-command which will then set EDITOR to the Emacs process. Just call M-x with-editor-shell-command and type git send-email -1 &. The ampersand tells the shell to execute the command asynchronously, letting us interact with the process while it is running. (If it was executing synchronously we’d have no way of suspending the command execution to spawn an editor buffer.) You can also do directly M-x with-editor-async-shell-command and you can drop the &.

You can also call git format-patch to produce the patch and then send it using git send-email which also accepts patch files. That way you can inspect the patch file before sending it, but this is mostly unnecessary since if you pass --annotate to git send-email it will preview the file anyway. I recommend calling

$ git config --global sendemail.annotate yes

And git send-email will always preview the patch before sending it.

Ok now that we’ve successfully sent a patch, how do we turn this around and apply patches sent to us?

Applying email patches in Emacs

Let’s assume a patch has landed in your email. You need to be able to save this patch in either a mbox or maildir format. If you’re not sure how to do this, you can just copy the mail verbatim into a text file and then pass that to git am. git am will detect the format and use the From: field to set the committer and Subject: and message body will go into the commit message.

How to do this in Emacs? For this, we don’t have to just rely only on shell commands anymore. We can use either

the wonderful Magit package with its interface for git am
opening the patch and sending (piping) the whole buffer to git am

Using Magit, this is as simple as pressing w w (magit-am-apply-patches) or w m (magit-am-apply-maildir). Just find the patch email and you’re set.

Without Magit you can do this just by opening the patch and piping its output to git am. That is as easy as doing M-| git am. But that requires you to be in the same directory as the repository itself. So you’d have to M-x cd into the directory before applying. But all in all, there are several ways for doing this directly from Emacs, and of course as a last resort you can just open the Emacs shell M-x eshell and do it from thecommand line.

Making our own `M-x git-am`

If you’re not too keen on using Magit or cding to the directory every time, I wrote this handy little Emacs Lisp to provide a nice M-x git-am command:

(defun git-am (repository)
  "Run git-am in REPOSITORY with the current buffer.

If REPOSITORY is nil, prompt for a directory.  If a region is
selected, it will pipe the buffer to git am.  If the current
buffer mode is Rmail, Gnus article, or diff mode, it will call
`git am' with the contents of the current buffer.  

If the current buffer isn't appropriate and there is no region selected,
call git am after prompting for a patch/mbox/maildir. 

If the current buffer is a `vc-dir-mode', assume we are in
the repository and we want to look for a patch somewhere else.

If Magit is available and we're in a `magit-status-mode' buffer,
the same treatment as `vc-dir-mode' is given.

With prefix argument ignore the fact we're in a version control
buffer and prompt for the repository anyway."
  (interactive
   (list (if (and (not current-prefix-arg)
                  (memq major-mode '(vc-dir-mode magit-status-mode))
                  (eq 'Git (vc-responsible-backend default-directory)))
             (vc-git-root default-directory)
           (read-directory-name "Repository: "))))
  (when (not repository)
    (error "No repository given."))
  (when (not (eq 'Git (vc-responsible-backend repository)))
    (error "Not a git repository: %s" repository))
  (let ((default-directory (expand-file-name repository)))
    (cond ((mark)
           (shell-command-on-region (region-beginning) (region-end) "git am"))

          ;; Use the whole mail buffer when viewing a patch email or patch file.
          ((memq major-mode '(rmail-mode gnus-article-mode diff-mode))
           (shell-command-on-region (point-min) (point-max) "git am"))
          
          (t (shell-command
              (format "git am %s"
                      (read-file-name "Apply patch (mbox/patch/maildir): " nil nil t)))))))

This command works both in version control status buffers, dired buffers, patch files and patch emails. If the user is looking at some version control status buffer, it assumes the repository is the one we’re in, unless the prefix argument is given by pressing C-u M-x git-am. If we’re in a mail buffer or looking at a diff-mode file (*.patch), we’re going to apply this patch to some unknown repository. And if none of this is true we’re going to prompt for the patch as well.

Emacs has for a couple of major versions supported the idea of version control as projects. That is, a version controlled directory tree is a project, i.e. a collection of related files. This facility leverages ignore mechanisms like .gitignore by providing an interactive file selection interface but with ignored files filtered out from the search. Whenever you interact with such a project, Emacs registers this repository into its list of known projects.

Since these projects map to git repositories, I wrote a wrapper for git-am that will present repositories using project-prompt-project-dir:

(defun project-git-am (&optional use-current-project)
  "Run `git-am' in a project directory.

With a prefix argument run in the current project if found, otherwise
prompt for a project directory.

See the Info node `(emacs) Projects' for what a project is."
  (interactive "P")
  (let ((root (if (and use-current-project (project-current))
                  (project-root (project-current))
                (project-prompt-project-dir))))
    (git-am root)))

Looks like this requires installing project.el from GNU ELPA, or using Emacs 28 from Git, which includes this version of project.el.

Who knows, maybe all of this will end up in its own package one day! Even without this Elisp code you can easily use git am quite readily in Emacs with the tools already available. If nothing else, this code shows how easily you can do powerful customizations for simple shell commands in Emacs!

Conclusion

Git + Email = ❤. It’s nice to see a resurgence of Git and email being boosted by sites like sourcehut. I think the first time I used Git and email must have been in maybe 2008? I don’t remember what it was for, but I do remember mailing it to the author directly from alpine. Not long after that GitHub came and made everything different.

Divergence: A website is born

Antoine Kalmbach — Sun, 11 Oct 2020 10:32:21 +0000

Some readers might have noticed a new section appearing on the front page. It’s just a simple heading with a list of sections outside the blog. This is because I’ve for a long time felt that the blog format doesn’t feel right for informative or lengthy writing. That is, I’d much rather prefer my blog to be just a simple journal I don’t have to think too hard about, since I find it reduces my eagerness to write simple updates.

To me, and I emphasize to me, the blogging format is like a journal, like a .plan file, but on the web. That is why I’m putting new content that doesn’t fit the format to just as plain pages. I’ll set up some sort of index using the Jekyll tags mechanism, it fits well for this purpose.

I’ve also added a change log that lists what has changed on the site. While I write GNU style ChangeLogs in commit messages, this format is not really readable or accessible to readers.

I also removed Bootstrap CSS from the site. Utility CSS is fun, but the site design is so minimal it doesn’t really make any sense. I also added a mailing list to serve as a comment box or discussion forum for the site, but time will tell if that place will see any discussion.

So, all in all, expect to see more interesting content and more blog updates as now I’ve finally made sense of the whole website thing!

The overhead of test-driven development

Antoine Kalmbach — Fri, 09 Oct 2020 08:44:50 +0000

There are significant merits in test-driven development (TDD), but I think its costs outweigh its benefits. The idea in short is that

you write all your software requirements as test cases,
you write the test cases first so that they fail,
you write the program until all test cases pass,
and your program is complete.

Note that the test cases do not necessarily need to be automated, nor do they need to be unit tests, as long as there’s a test–develop–test–develop cycle.

Iterate, iterate, iterate

This can lead to a very short and sweet development cycle, because as soon as all the tests pass, your program is ready–supposedly. But the devil is in the details, because as soon as you lay down those four items, you start asking questions like:

How is a requirement properly captured as a test case?
What is the level of granularity for your tests?
How much work is needed to create working tests?

To answer the first question, you need to properly understand the requirement. You need to be testing the right thing, at the right level. For instance, if a requirement of a library management system says that “the same user cannot make more than one reservation on a single item”, how do you test this? Do you register a user, create an item in the library catalog, make one reservation, then try making another, expecting a rejection? Initially we’d like our fail the test case by not rejecting the double reservation, once the second reservation is rejected the test is considered a success.

The design of the test case itself is quite complex. For the test to be able to work, we now need to have functionality for

registering or loading a preset user,
loading a library catalog (or adding new items to it), and
a mechanism for making reservations, and checking whether they succeed or not.

So obviously the first step is not to get the test to fail but to work: our test program needs to get to the point it has to signal failures or passes. This is where we get to the second question: what do we use to build the test scenario? Do we just write a program that calls some functions in the reservation system, or we do user interface tests where another program interacts with the program? Choosing the testing level is not easy! User interface tests require complex machinery to set up, but they are valuable, because you can treat the underlying implementation as a black box. This is valuable, because this way you cannot make tight coupling between the implementation and the interface. Only the results matter.

First, you must create the universe…

The last question is the most interesting one: we need to be able to create tests that signal whether they have failed or not. For them to be useful, you need the tests to actually interact with the program somehow. This could mean that the tests call the functions you are testing. To get that to work, you need to define function stubs so that the compiler or interpreter you are working with does not reject your code.

For instance, consider this Java code:

@Test
public void testReservation() {
    Reservation reservation = TestData.getReservation();
    
    // some global reservationSystem defined outside this method
    bool success = reservationSystem.makeReservation(reservation);
    bool failure = reservationSystem.makeReservation(reservation);
    Assert.assertFalse(failure);
}

Rather obviously for the Java compiler to accept this we need to have the makeReservation method on the reservationSystem! In the ReservationSystem class we define

// in ReservationSystem.java
public bool makeReservation(Reservation reservation) {
    return false;
}

Not to mention that you’d have to define the Reservation class, and so on.

I think test-driven development can be quite beneficial when you are refactoring your code or adding or modifying functionality. That is, in the above example, suppose we already have a working reservation system and all the necessary scaffolding in place. Implementing the double-reservation prevention in a TDD fashion is quite simple: just add the test above, and fix your code.

But what about when you don’t have a program to modify, or you are writing a suite of test cases for a part that isn’t there? Suppose we don’t yet have a reservation system at all, and this test case is a part of a number of many test cases each and everyone of which needs some kind of functionality in order to compile to get to the failing state?

Futhermore, if you’re starting from scratch, it’s obvious that you need to spend considerable effort on writing the stubs first, and this indirectly guides your development process, and the way you design the program. Here’s where get to the big question: does test-driven development guide your program design in a way that’s appropriate?

The answer is it depends. To be precise, it depends on the way you approach program design. For those who don’t mind writing the stubs and test scaffolding first it can be a hugely productive manner of working, but to those who like to think in small iterations with functionality first, it feels counterintuitive.

When building a bridge it makes a lot of sense to specify up front that it should be robust enough to tolerate the weight of a hundred cows when complete. On the other hand, it would be rather stupid to continuously drive cows on the bridge even before construction has started. The cows would fall into the water and everyone, especially the cows, would find the experience quite displeasing.

Although software is not physical, time is still time and just like the bridge testers would have spend time driving cows into water (initially) and have to rescue them from the water, in the beginning of a TDD process one needs to fight the compiler so that you get the tests to work.

So to me doing the initial grunt work of setting up stubs for test-driven development feels like throwing cows into the water. I like to start with the requirements, figure out a minimum viable design that’s testable, and then write the tests, and repeat until the program satisfies the requirement and the tests are rigorous enough.

But when refactoring programs or adding or modifying functionality I sometimes practice TDD but not religiously. It largely depends on what I’m working on and if the feedback loop between change-test-change-test is fast.

How this site works

Antoine Kalmbach — Wed, 07 Oct 2020 08:54:36 +0000

This site is a fairly standard Jekyll static website. The site is divided into two principal sections: content and blog. Posts are written primarily in Markdown. The site is versioned using Git and hosted at GitHub, a Travis CI job builds the site after every commit and then pushes the contents of the develop branch onto master, which is served by GitHub pages.

An essential part of the content section is the _link function in Jekyll. All the hyperlinks to documents in the site are linked using that, absolute file references are not used. The link function ensures the files are there, this ensures that there are no dead links on the site. I don’t use collections anymore (I used them for the microblog), basic pages are sufficient to maintain the non-blog content on this site. They are not useful for my purposes, perhaps I’ll need them one day.

I once tried to use Org to maintain this site, but it was slow and I saw very little gain in that. It was quite painful to configure as well. Jekyll is fairly easy to hack on and the templating language is quite sufficient.

What this paragraph looks like with markup hidden in Emacs

For editing, I make heavy use of markdown-mode and web-mode. Both are Emacs major modes for editing markdown and HTML, respectively. web-mode in particular is essential for HTML editing since it can handle template engines and has great structural navigation and editing commands. Markdown mode does many things, what I use the most is the ability to hide the markup (C-c C-x C-m), so it just looks like text, and hyperlinks are formatted inline so I don’t see the ugly link syntax. I also insert hyperlinks using C-c C-l since I can use links I’ve previously inserted in the same file.

Origin of this site

Antoine Kalmbach — Wed, 07 Oct 2020 08:52:24 +0000

About five years ago I realized I wanted to have a website again. Yes, again, this wasn’t the first time I had had a website.

I think I had my first website in the late 90s, it was hosted on xoom.com and I had to upload content to it via FTP over a 56k modem, no broadband at that time yet. We had fast internet at our school so sometimes I edited the site there. Later on I switched to hosting provided by a Finnish computer magazine MikroBitti. I stopped maintaining that site around 2005 or so, I must have been 17 or something and had more important things to worry about (like school and World of Warcraft… ok, mostly World of Warcraft).. It wasn’t interesting anymore, it mostly contained some CSS website layouts I had made.

Early 2015 I decided I wanted to write a blog. I had no idea what about, but everyone on the internet seemed to have some sort of verbal outlet for their computer things. So I joined the club, copied some Jekyll theme from somewhere, and wrote a couple of posts. Eventually I ended up deleting those, and I had actually nothing on this site, but in January 2016 I lowered the bar and started again.

It seems the most active year (up to now) was 2016, with the quietest period being the years 2018-2019. I had lots of things going on in my life around that time, and I was still holding the site to too high a standard. I was torn: because I was misusing blogging as a medium, I couldn’t blog as a journal instead I was blogging as a magazine. I’ve come to accept that if I am to blog often it needs to be an informal journal, otherwise I will set the posting standard too high and not post.

Fast forward to 2020 I ended up revitalizing my interest on this site, and this happened almost by accident. I created a created a microblog and then ended it, because I realized it made no sense to have a blog with an arbitrary minimum post length. Then I realized I need to actually have more than just blog posts, many of the blog posts I have are actually quite gargantuan in size.

So I started to transition the site towards the blog being, well, just an online journal of sorts. More structured content is grouped and assembled into the front page, you can read about that over here. Eventually, most of the content will live outside the blog in various sections, because I think blogging as a medium should focus more on a streams-of-though type of posting instead of having informative content. I think combining the two makes it very confusing for readers.

Design of this site

Antoine Kalmbach — Wed, 07 Oct 2020 08:51:53 +0000

This page describes the design of this website. If you come here often you might notice that every now and then something changes. That is normal, because I really enjoy tweaking the smallest of things here, be it adjusting margins, adding padding, changing fonts, and so on. It’s a neverending project, I can’t tell you if the site as you see now will be how it will look like in five years. But here’s a rough idea how it looks like at the time of writing.

Most people will ask, “what design?”, as you’re greeted with the browser default font, probably a serif or sans-serif font, no colors beside the links, barely any borders or visual frills. This is by design, or the design: the design emphasizes this site’s focus on content. I want the text to be as readable and distraction-free as possible. To this end,

the content is center-aligned and constrained to a maximum width on large screens,
the font is black-on-white at about 16px of height,
the navigation is at the top of the page, and also aligned to the center,
there are no colours or background highlights unless absolutely necessary.

Part of the reason why I don’t specify any extra fonts or the like is to make sure the site works on any device possible. I also happen to enjoy browsing “unstyled” pages because it puts emphasis on content over form, although some styling is necessary to make this enjoyable. There is no need to design for responsiveness when the site mostly just text.

The stylesheet is a couple hundred lines of SCSS and mostly just deals with the formating of the navbar and images, and removing some built-in ugliness of the default browser styles, e.g., by thinning out horizontal rules to a single line.

I plan for the central index at the front page to evolve in a tree-like structure of the whole site itself, but as the content outside the blog is rather thin, it’s just a link to, well, this section.

No more microblog

Antoine Kalmbach — Tue, 06 Oct 2020 13:56:28 +0000

The microblog is no more. I got this idea due to a sudden realization: there is no minimum length to a blog post. Any microblog post was just a short blog post. Perhaps the idea with it was to participate in the IndieWeb zeitgeist, but in the end I realized I’m not interested in posting that much short-form content. Twitter is just noise, why would any federated version of it be less noise?

So, the microblog is gone. I’ll keep the page around but this page will be the only place linking to it.

Perhaps this will affect my blogging frequency, perhaps not. One thing that will be different is the blog will focus on informal writings, perhaps even to just say I ate a nice pizza today, and I’ll start working on having a content vs. journal style separation between the blog.

Maybe.

Between two Lisps

Antoine Kalmbach — Mon, 05 Oct 2020 14:03:41 +0000

Out of all Lisps the ones I’ve come to appreciate the most are Scheme and Common Lisp. These two languages are fundamentally very different: Scheme is a minimalist language built on the foundations of lambda calculus, while Common Lisp is a multi-paradigm synthesis of many Lisps before it. Common Lisp is a large standard with many implementations, Scheme is a collection of an evolving but minimalistic standard with many implementations. The core of Scheme is quite small compared to Common Lisp. The latest standard R6RS is about 65 pages, while the ANSI Common Lisp standard from 1994 is about 1100 pages. The Scheme Requests for Implementation process aims to standardize additional features (like test suites) that implementations may implement.

Common Lisp has some wonderful features, conditions and restarts, the Common Lisp Object System (CLOS), the macro system, among many other things. The SLY IDE for Emacs is amazing, and the Steel Bank Common Lisp compiler is really great. It has Quicklisp and ASDF for package and build management, respectively. I find the developer experience of Common Lisp to be superior to almost anything imaginable, and this is not an empty statement: having used all sorts of IDEs and editors for over 25 years, I have seen many.

Tastes differ

That said, Common Lisp is weird. What I find particularly jarring is that functions and variables live in different namespaces: if you put a function into a variable, you can’t just use it like a function, you have to funcall it. Having programmed in lots of languages of the ML family this is just, well, odd; but this is due to historical reasons and there are sound technical reasons for it. There are other oddities, some strange things like (cdr '()) is not an error (in Scheme it is), () and nil are equal (in Scheme #f and () are separate things), and so on.

This isn’t really a fault in Common Lisp: other languages have impacted my taste and preferences to bias me in the direction of Scheme, but that is not to say I cannot work with Common Lisp’s idiosyncracies. Actually, I don’t mind them, I just notice them.

I like the naming styles of Scheme more, as well. It has string? vs string-p for predicate functions, set! for state modifying functions, foo->bar for conversions, these make code quite easier to read. Scheme has hygienic macros, Guile has the traditional defmacro as well.

Many nice Scheme features are available in Common Lisp libraries. Pattern match is available in the trivia library. Named lets are easy to implement with a macro.

Common Lisp aficionados are quick to point out things Scheme doesn’t have: keyword arguments, docstrings, rest arguments, but my Scheme implementation of choice Guile has these built into the language.

Productivity matters

Guile is in a strange niche is that its primary raison d’être is to be an extension language for the GNU project. Like Emacs Lisp is for extending Emacs, Guile is the de facto language for GNU programs for extension and scripting.

Guile doesn’t have Quicklisp and its package manager and build system is basically nonexistent for the first and Autoconf for the second. There is sentiment in the Guile community to have Guix as the package manager for Guile. This might sound a bit onerous, since Guix is also a complete package management for many other things than Guile, but consider this: as Andy Wingo points out in his message that Guile libraries often come with C extensions, Guile packages need some sort of managed build system for building the C extensions. Since it doesn’t have one, to solve the problem of building a package manager you’d also have to build a build system that can manage C code and packages needed by the C code bits. To do this elegantly is a gargantuan task, for instance, chicken-install just asks you to put compiler/linker flags on the command line before calling it, so it obviously is a hard problem.

Now, Guix solves all that, and more, in a manner that is quite elegant and interesting. But it’s not as lightweight as something like Quicklisp or chicken-install from CHICKEN Scheme. What is more, Guix works only on GNU/Linux systems really, so macOS and Windows users won’t be able to do use your library if you plan on distributing it via Guix. Duh, it’s the GNU project, but portability is always nice.

But in Common Lisp I can write (ql:quickload :alexandria) and voilà, it will automatically install the Alexandria library that I can use. Then again, in Common Lisp it’s somewhat rarer to have C extensions, so I don’t know how ASDF handles that.

It is unfair to compare SLY to Geiser, the best Emacs Scheme integration package. SLY is based on SLIME which has decades of man-years of work behind it. Geiser is able to support multiple Scheme implementations and it is quite impressive in this regard. But SLY obviously has much more (e.g. an interactive debugger). That said, Geiser has nice Guile support, and Guile is in general the most Common Lisp-y of all Schemes, in fact, it has

an imitation of CLOS in the form of GOOPS
docstrings and keyword, rest, and optional arguments for function definitions
an interactive REPL and a mutable top-level (unlike many Schemes)
a nice module system
defmacro and exceptions somewhat similar to Common Lisp restarts

and some nice things Common Lisp doesn’t have like first-class continuations. But then again I would use those rarely.

When would I pick one over the other?

If I were to write an extensible C/C++/Rust program that I know will need to use a low-level language, I might do the low level bits using a low level language and then write the rest in Guile. Guile makes it very easy to spin up a REPL socket to do something like myprogram --repl=12345 that you can connect to, and the interop between C and Guile is fantastic, it has to be, as it’s primarily an extension language.

On the other hand, if were to build a wholly standalone application, the choice is not as obvious. I could do the whole thing in either language. Common Lisp can build native executables, although their size will be large (who cares?) since the binary will include the whole Lisp implementation. Guile cannot compile to native code so you’ll have to write a script executable, but that doesn’t really matter.

Scenarios where I would pick Guile:

when I’m making an extensible program that has to have some C/C++ bits but I want to make it scriptable by users
when I want to write a native binary but not write too much C/C++ code
I just want to write Scheme, because Scheme is a bit more elegant
the project has something to do with the GNU project

On the other hand, Common Lisp makes the most sense if I just want to enjoy a seriously rapid development experience, a large standard library and language, and I don’t mind having 50MB binaries (again, who cares?) if I were to write standalone programs. Guile can’t do that, but it’s easy to either write Guile scripts or a C program that essentially bootstraps a binary to load a Scheme runtime. This is how LilyPond is written, for example.

So the answer to which one is decidedly both! Both languages are really fun to write. At the moment I don’t do Lisp at my day job so it’s all for fun anyway. If this were for professional purposes, I don’t know. Very often a strict requirement for professional work is to be able to be productive. In that regard I think Common Lisp has a significant edge, not only due to its superior development experience, but its history as a real production language. There are actual companies doing stuff in Common Lisp. I have not heard of any professional (in the industrial sense) uses of Guile, notwithstanding that many projects powered by Guile (Guix, etc.) are extremely professional in the way they are written and maintained. But Common Lisp has more libraries and companies behind it.

What about Clojure?

I’ve actually had the pleasure to use Clojure for personal fun, and a little bit of professional use. I wrote a couple of libraries (vigil and task) and my experience with has always been positive and its development experience is excellent. Clojure isn’t a true Lisp, or well, it is part of the Lisp family. Its ability to interface with the JVM makes it easy to leverage the thousands of JVM libraries out there.

I’d gladly do it again, though I feel that Common Lisp with CLOS and its module system makes it somewhat easier to practice programming in the large. Clojure explores interesting territories with spec, so it is interesting to see what direction the language will take in the future.

Final words: Emacs Lisp

There’s also a parenthetical elephant in the room here: Emacs Lisp! An old descendant of MacLisp, it’s actually quite fun to write, and the fact that Emacs itself is the interpreter, you have superb introspectability for any Elisp code. Most of the Lisp I write these days is probably Emacs Lisp. It’s very close to Common Lisp. cl-lib adds a sufficient amount of convenience from Common Lisp to make Elisp writing quite enjoyable. EIEIO adds a subset of CLOS that can be used seamlessly with cl-lib. The condition system is missing.

I’ve also done a fair bit of Fennel, a Lisp that compiles to Lua. I used it to write a small game and it was fun to write since you could develop the game in a REPL.

All in all, Lisp in all its variants is the most fun I’ve ever had while programming a computer. Guile and Common Lisp are definitely the most fun I’ve had programming in Lisp.

Back to Rmail

Antoine Kalmbach — Wed, 09 Sep 2020 10:55:56 +0000

While I initially wrote about switching to mu4e, I was initially pleased by it, but then I found out about Rmail. Rmail is kind of like the standard Emacs email reader. Gnus is pretty comprehensive in terms of features but I was kind of put off by its complexity.

So, eventually, my setup became the following:

offlineimap to synchronize an IMAP mailbox on my machine
Rmail configured using a maildir where offlineimap syncs the data
mairix as the search backend, using the Emacs frontend for mairix

Rmail has surprisingly good automatic filing capabilities so that I can keep a mostly empty index and things can be manually filed into separate folders as they appear into the inbox.

The only gripe I have with it is that mairix-search works using mbox files, i.e. when I search for something, the search produces a mailbox result (the same file every time), and to view that I have to open that in Rmail using rmail-input. A possible improvement would be to write an Ivy backend that would update results on the fly.

Inside the machine

Antoine Kalmbach — Thu, 03 Sep 2020 06:34:03 +0000

One of the most interesting aspects of programming is the ability to inspect and modify programs while they are running. We all know debuggers, but there are lots of programs which let you interact with them directly while they are running. Some programs let you run scripts via embedded interpreters, which let you extend the program, but in some instances the programs themselves are the interpreters.

The famous story of the Deep Space 1 probe is an interesting one:

The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA's New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.) — Ron Garret, Lisping at JPL, (2002)

Examples: Emacs, StumpWM, Nyxt

Famous examples of programs being nothing but interpreters with a function are the Emacs family of text editors. Essentially, Emacs itself is a Lisp interpreter acting as a text editor. You may at any time open a read-eval-print loop inside the editor and actually interact with the program directly.

For instance, if I open an Emacs Lisp REPL inside GNU Emacs using M-x ielm, opening and type (next-buffer), I actually end up selecting the next buffer!

Something similar to this is StumpWM, a tiling X11 window manager that is built entirely in Common Lisp. Somewhat similar to Emacs, you can install Swank, a REPL over a network connection, as an extension inside StumpWM, and then you can connect to the REPL using your client of choice. From there, any function inside the StumpWM Lisp image is available to you!

Another program that you can REPL into is Nyxt, an extensible web browser built on WebKit. By running C-h s you can start up a Swank session and then you can connect to it from, e.g., SLIME. Nyxt itself is written in Common Lisp, so offering REPLability via Swank is pretty much a no-brainer. I’ve only tried Nyxt very briefly, but from what I’ve seen it’s pretty interesting!

I actually tried something similar years ago with Common Lisp, it turned out to be super simple. I was building a Lisp-customizable i3 status bar program (I ended up writing it writing it in CHICKEN Scheme, though). Turns it didn’t require much: Swank is available via Quicklisp, so you can just add it to your Common Lisp program and voilà, you have a REPL connection available. I recorded the thing in action via video to demonstrate how simple and powerful this concept is.

These days I work with a multitude of application platforms and the Java Virtual Machine is one of them. Java has powerful introspectability, from its native debugger support to protocols like JMX and even the more modern (and seriously amazing!) Java Flight Recorder. I’ve actually been able to find bugs using JMX, a web server was routinely freezing during high traffic hours – but only for 0.5% of the requests or so, still, this (barely) under the SLA we had for the service. Turns out there was a thread starvation issue: when traffic was sufficiently large, other background processes were exhausting the thread pool reserved for incoming connections, causing the web server to not be able to receive incoming connections for a short period of time. Long story short, I found these incidents via JMX, and was able to pinpoint the issue. The fix was to use separate thread pools for the web server HTTP traffic and then background I/O.

Of course the Java Virtual Machine is not a Lisp interpreter, far from it, but it offers things that are similar. Hot code swapping, dynamic class reloading, etc. make it quite a powerful platform in terms of introspectability. But it’s a far cry from being a Lisp interpreter and a text editor, or a window manager, or a web browser.

Blazing fast development cycles

I think there is a crucial difference between allowing extensibility via scripts (or plugins) and the ability to actually interact completely via a live process. The former usually does not offer total control over the program, the latter offers total control over the process, because the interpreter is the program.

There is something admittedly fascinating about programs being, in a certain manner of speaking, living things. Not only does it give you a feeling of control, but it’s even more personal: you’re not just using the program, you’re inside it.

Having tried Clojure and Figwheel where you can directly REPL into your browser, the speed of development if you’re building some application while it’s running is quite something. I’ve also tried Fennel, a Lisp that compiles to Lua, using the LÖVE game engine using a Fennel REPL connected directly into the LÖVE game engine. The result is quite unfinished but the speed of development was astounding – I could change the game physics on the fly just by evaluating Lisp expressions in Emacs and the game would change instantly!

If I’m ever after building some sort of hyper-customizable program I will definitely try building it around a Lisp interpreter, possibly using Common Lisp.

Winding down

Antoine Kalmbach — Sat, 07 Mar 2020 00:00:00 +0000

I find it curious I can get paid for programming and still practise programming in my spare time and it feels like fun. It doesn’t stress or tire me at all. Superficially, the line between what is work and what is fun is not that obvious. I’m at a computer, bashing its keys, programming it to do what I want. Of course, at work I program the computer to do different things, but you’d have to know what my job entails to know the difference.

I doubt it is possible to enjoy work so much that your work is relaxing, to the extent that it alone can guarantee a means for winding down. In that case, one’s work would be so fulfilling and relaxing that it one is calmed by it. This is not impossible. It is just unlikely.

How does one recognize when you enjoy work or when you enjoy nothing but work? Suppose one eats properly, sleeps properly, and then works all the rest of the time. To me, a common observation for which I offer no scientific basis, is that this is still too much. The brain can’t cope with continuous stimulation, and eventually such a person will either burn out, lose interest, or suffer from some other crash, and end their ability to perform in a job.

The crash comes sometimes from within, sometimes from outside. Families and friends – if you have any – will probably notice before too long that you might be working too much. But a person so engrossed in their passion can sometimes ignore their environment, and power on. Then at some point, their passion crashes. The stack overflows, and everything grinds to a halt.

I’ve been playing with computers for as long as I remember. I began programming at an early age or so, at the age of six, I think. Eventually what was just for fun became not just for fun, I turned my hobby into a profession. I doubt I am unique in this in our field, I know many have started their careers before they were — careers. Many also convert to our profession from other fields by learning the basics in their spare time.

I still program for fun. Occasionally, intermittently, whenever, I don’t keep a stable routine. And I don’t necessarily do anything meaningful. I play with new languages, new technologies, experiment with something vaguely work-related, but in general the striking difference between the programming-as-a-job and programming-as-a-hobby are that in hobby projects I don’t have

deadlines,
commitments,
schedules,
requirements, and
any pressure to achieve anything meaningful.

From that it is glaringly obvious what makes a hobby a hobby and a job a job. You might enjoy your job, but eventually come to realize that you’re obviously not there voluntarily, you have outside commitments, and you are bound to schedules and deadlines. Essentially, fun has fewer rules than work.

This is subjective, however. For some, the absence of such rules is stressful. They might enjoy a structured hobby: learning a new language with a predetermined schedule, follow a training routine to become a better runner, lift weights using a personalized program. So what work is for some is not work to others.

But for me it’s not just computers. I spend time with my wife and our baby daughter, I exercise, I read, cook, play music, whatever. There are many ways for me to wind down. But funnily enough, if someone asks me to list what I do for fun, I can include something that’s very close to my work. This is a common theme in creative professions. Musicians play music for fun, athletes also exercise for fun, graphic designers draw in their spare time, and so on.

A sport, although a hobby, can be stressful. Some years ago I was very into road cycling. I had a rigorous training schedule and was actively considering doing some competition. Before long, I realized I had only room for one rigorous schedule in my head. It was either work or something else. I chose work. Cycling had to be a hobby for me to enjoy it, not something serious. But before I could stop it, I crashed. I completely lost interest in road cycling because it felt too much like work. The distance I have cycled in road since can be counted with a three-digit number. This excludes commuting to work, naturally.

The weird part is that going I knew I didn’t enjoy having a strict structure in my spare time activities. I had figured this out a long time ago, when I was a teenager. Yet, I created a strict structure for a hobby. I lost interest. Perhaps I thought having grown older had given me the ability to manage this structure in such a way it didn’t feel overpowering. It hadn’t.

There is a sweet spot in this. I can find a hobby in which I have some structure but not too much. I’ve been going to the gym for a decade. For that, I’ve found going 2-3 times a week is the sweet spot in which I can keep myself in shape and see continuous improvement, with emphasis in the former. Any gains are just rewards, but not the ultimate objective. I suppose in weightlifting the ultimate objective for me is the physical health improvement. Whether I can add kilograms to my squat is not that interesting in the long run, though I strive to maintain an upward trend. Again, the same patterns emerge: I have no real schedule, no requirements, no commitments.

I guess, for me, personally, the absences of schedules, requirements and commitments is what defines fun for me. But this is just me. A colleague of mine trains in trail running rigorously, with strict schedules and diets. He said the structure helps him relax, because he has a fix point to follow in his spare time. Perhaps one of the strangest discoveries of myself was that a character trait like this ran very deeply. It is pervasive, it affects me in all areas. When travelling for fun, I ask my wife that we do not commit to a serious structure or schedule. We can visit that interesting cultural site, yes, and see that other place, but if you gave me a calendar with meticulously planned daily objectives and places to visit, I would rather stay home.

Some structure in non-work life is inevitable, and more often than not, necessary. Some hobbies require structure, things like choir practise, group sports, dinner with friends occur usually at fixed times. Family life and all it entails requires structure, and so do many aspects of one’s personal life. It’s not a bad thing. But the amount of structure is much smaller than in professional environments.

At the same time, my work is very structured. I organize my work into a to-do list and try to minimize ad-hoc work, instead knowing in advance what I should do the next day. That way I don’t have to think in the previous evening or night what I should do the next day. I get to work, open up the to-do list, and start from there. Of course, with meetings and collegial chats I can’t follow the list to the letter, and this to-do list is not the absolute authority, rather, it is a foundation. With this foundation, I never have to spend time wondering what I should do next. And this is just my work, my calendar is filled with meetings of various sorts. A big part of my job is figuring out the structure of other people’s work.

Knowing what is work and what isn’t makes it very easy for me to keep a good work and life balance. Maintaining that balance is extremely important to me. It is the single most important goal in my professional life to maintain this balance. That way, I can avoid stress and burning out, and enjoy being employed until I’m old. It’s not that I feel I have a duty to enjoy my job, or that enjoying working is somehow a goal in itself, it’s that I don’t want not to enjoy working.

I am incredibly privileged to have had turned my hobby into a profession. Sometimes work is so much fun it literally feels like I get paid to have fun. I am extremely aware that this is rare. Most of humanity considers work to be, well, work. We spend a significant portion of our days, of our lives, doing things we’d rather not do, in order to provide for ourselves.

I don’t think it’s possible for everyone everywhere to enjoy working, or rather, enjoy their job. It would be naive to think otherwise. Most of humanity is employed because they have to be. They are forced into jobs they had no choice in, they work in conditions they can’t control. Similarly, I don’t think it’s even possible for the majority to even enjoy being good at what they do, because for many even the ability to even enjoy a job is such an alien concept.

Even to those like me, who had a choice, I don’t think thoroughly enjoying one’s work is a worthy goal. To me, it’s not a measure of success whether you’re passionate about your job. You don’t need to enjoy your work to be a successful person. In fact, it is very common to be rather neutral about it. I think it is a good objective to not not enjoy your work, rather than to enjoy it. That is, being somewhat neutral, with a slight bias towards the positive, is probably the optimal position. You don’t hate what you do and you don’t love it too much in order to maintain a balance with your other life.

In fact, for me, a measurement of success is how healthy one’s relationship to work is. Fundamentally, work is service. You can serve whomever, even yourself, but should life be service? Maybe work is just some necessary evil we all do to survive? Work is what you do to fund the fun parts of your life. At the end of the day, we all go home, back to our families, friends, and hobbies. Work is transient like that.

Yet, I believe most of us would work even if not forced to. I think humans have an innate need to be useful, to have a purpose. That quest for purpose can be satisfied in many ways, work being one of them. I doubt many of us would work like we currently do, but it would be some sort of service to some entity. But it would be different. It wouldn’t be your average 40 hours a week five days a week. It could be more, it could be less, but I believe the terms on how we do it would be different.

So work is, well, work. It’s not always about what you do, it’s sometimes also the how. An artist could characterize work as the drawing they do primarily for someone else. Some other artist’s characterization might be different. There are most likely as many people as there are characterizations of work.

I’m happy that I have been able to understand what the difference is for me. What lets me enjoy programming in my spare time is that I keep it fun. I don’t do it rigorously, I don’t do it for someone else. Hobby programming is fun because it doesn’t feel like work. Yet, it’s sometimes very close to what I do at work. Most of the time it’s something completely different – embedded stuff, games, interpreters for toy programming languages, and so on. I suppose I could even pull off doing work-related stuff. It’s not the what, but the how.

Embedded Rust and WiFi

Antoine Kalmbach — Wed, 20 Nov 2019 00:00:00 +0000

Earlier this year, in May or so, I started having fun with hobby electronics, and ended up playing around with Rust in embedded devices, microcontrollers and the like. Actually it’s the other way around, I first wanted to write some Rust, and found out that Rust has a thriving embedded ecosystem, and proceeded to buy some microcontroller dev boards (and upgrading my “lab” equipment from 2003), and started writing Rust on them.

It’s fun! The last time I did any embedded was in 2003, when I was 15 or so, and things were way different back then. The affordable microcontrollers were 8-bit turds like PIC8, and the hot stuff was AVR. C was the only language that was around and information on the net was scarce. Then, Arduino happened, and things started improving.

Now in 2019, sixteen years later, things are much easier. You can program MCUs in Python! Clock speeds run in the hundreds of MHz and there are tens of kilobytes of RAM. The internet is full of tutorials.

Rust, given its nice high-level abstractions and package management, has an amazing embedded ecosystem. All you need is their build tool and GDB with ARM support. Using a HAL is actually a real abstraction: implement a single HAL API, like I²C, and your device is usable by anyone. As of writing some things are still missing from the Rust HAL library, like USB and CAN support, but it’s slowly building up.

Anyway, WiFi. Wireless connectivity on a little microcontroller is an extremely nice way of getting your toy device on the internet. Most likely the most popular WiFi microcontroller devboard is the ESP32, which costs about 10 EUR. If you want to write Rust on this thing though, there are problems: the ESP32 is using a Xtensa LX6 processor, which doesn’t (yet) have mainline LLVM support, you’ll have to use C or CircuitPython.

ESP32-WROOM-32

Fine, if your goal is to get something working, it’s probably not a good idea to use Rust for WiFi enabled toy devices. But if you want to do something fun, doing this with Rust is a great idea!

So how to get WiFi working with embedded Rust? The answer is simple: use a microcontroller where you can use Rust and a WiFi coprocessor, where one microcontroller runs your Rust program and the WiFi coprocessor runs its custom firmware which has WiFi support. The firmware has its own TCP stack so the interface between the two processor is fairly high-level, at the level of “here’s a password, connect to this network”. The two microcontrollers would use SPI or some other interface to talk to each other and transfer data.

So you need a microcontroller with WiFi support and a firmware to boot! Turns out there are a few good choices: the ESP-WROOM32 from Espressif running the Adafruit NINA-W102 firmware or ATWINC1500 from Microchip running its own firmware. The nice thing about the Adafruit firmware is that there’s a CircuitPython implementation with SPI, so reimplementing this in Rust looks to be straightforward. The ATWINC1500 libraries are hidden somewhere in Atmel Studio so I’m not going to dig that out just yet!

The ESP32-WROOM-32 is also available as a breakout with SPI support from Adafruit

The alternative would be to build WiFi support yourself on some microcontroller that has WiFi. With the ESP32 being badly supported by LLVM and thus Rust, the other option is ATWINC1500 which is built on the Cortus APS3, building Rust for that is another here-be-dragons I’d rather not explore. Lastly, there’s the WM-N-BM-09 from USI that is a STM32F205RG paired with a Broadcom 43362 WiFi module. This is available in the Adafruit WICED Feather and the Particle Photon. With STM32 being widely supported in embedded Rust, it should be possible to write a WiFi program on such a microcontroller. Then there’s the u-blox NINA W102 which is another ESP32 based WiFi microcontroller, used in various Arduino WiFi products.

That said, building a WiFi “firmware” in Rust is quite a Herculean task, though not impossible! You essentially need a functioning TCP/IP stack and implement a WiFi layer on top. That would be possible with smoltcp, a Rust TCP/IP stack suitable for embedded devices.

However, for the time being, I think I will stick with using ESP32-WROOM or ATWINC1500 with preloaded WiFi ready firmware. Here’s a comparison table of the different WiFi microcontrollers I found:

Microcontroller	Architecture	“Coprocessor” firmware	Interface	Free SDK/library
ESP-WROOM32	Xtensa LX6	Several¹	SPI	WiFiSpi (C), Adafruit CircuitPython
u-blox W102	Xtensa LX6	Several	SPI	ditto
ATWINC1500	Cortus APS3S	Yes, proprietary	SPI, UART	Arduino WiFi101 (C)
WM-N-BM-09	ARM (STM32F205)	Several²	SPI	Several

There's also a breakout for the ATWINC1500.

Based on the chart I would go either with the ESP32-WROOM-32 or the ATWINC1500. The ESP32-WROOM-32 is both cheaper and easier to develop for compared to the ATWINC1500, but the ATWINC1500 seems to be a bit more performant and reliable.

Anyway, this led me to choose ESP32-WROOM-32 as the WiFi “coprocessor”. What’s next is writing a Rust library that can control the ESP32 over SPI. This should be fairly straightforward, since I can essentially reimplement the Adafruit CircuitPython ESP32-SPI driver in Rust. That’s going to be fun! First I’ll need a development board, I’ll most likely use the Adafruit AirLift Shield on top of a Adafruit Metro M0. Why the Metro M0? Well, it has a JTAG header so I can use SWD to debug what’s going on, instead of repeatedly flashing the binary to the bootloader.

Once I have the board, the next step is to build the SPI interface and some sort of socket abstraction that works over the SPI interface. Seems like there is no real abstraction for sockets, and there’s no #![no_std] library for things like HTTP let alone MQTT… this is going to be interesting!

To put it simply, the Rust embedded ecosystem isn’t super mature yet. It’s getting there. That’s not a problem, if I really need to get my toy device (like a temperature sensor over WiFi), I can write the necessary Python/Arduino C++ in a few hours, and I have something that works. But a large part of my resurgent hobby electronics adventures is being able to write Rust in bare-metal systems. The breadboarding and soldering parts are fun as well, but my main interest is in programming. Anyway, this looks like a pleasant challenge!

There’s Arduino NINA (Adafruit’s fork for ESP32-WROOM-32), NodeMCU and so on. ↩
Depends on the vendor, Adafruit is based on Cypress WICED which is proprietary, but the Particle Photon uses FreeRTOS. ↩

Useless interfaces

Antoine Kalmbach — Thu, 23 Mar 2017 00:00:00 +0000

A feature that often frustrates me in object-oriented code is the prevalence of useless interfaces. Interface isn’t meant literally here: this applies to traits of Rust/Scala and protocols of Clojure as well.

The advice of planning for the interface is just and solid, but people tend to follow this tip to the extreme. It is not uncommon to see people design a module or class by defining its interface first, with an actual implementation following later.

One eagerly designs an interface as follows:

trait Mediator {
  def validate(r: Request): Boolean
  def postpone(r: Request, d: Duration): Postponed[Request]
  def frobnicate(a: Request, b: Request): Frobnicated[Request]
}

Then, the implementation, in a class called MainMediator or DefaultMediator, in a separate directory, implements the Mediator interface:

class MediatorImpl extends Mediator {
  def validate(r: Request): Boolean = { ... }
  def postpone(r: Request, d: Duration): Postponed[Request] = { ... }
  def frobnicate(a: Request, b: Request): Frobnicated[Request] = { ... }
}

Dependents of the Mediator trait then naturally get their dependency provided with a constructor argument:

class Resequencer(m: Mediator, c: Clock) {
  def resequence(requests: Seq[Request]): Seq[Postponed[Request]] = 
    requests map { r => m.postpone(r, c.randomDelay()) }
}

val m = Mediator()
val c = Clock()
val foo = new Foo(m)

This pattern is older than the sun, and has been a characteristic of modern, inheritance-based object-oriented programming for ages. Fundamentally, it is alright to separate implementation from specification, but where it goes wrong is the overuse of this paradigm, or when this separation is superfluous.

This separation is superfluous when it serves no purpose. You could as well call it dependency injection on the toilet. There is no fundamental reason why a class or module like Mediator warrants an interface when it is likely that there will never be an alternative implementation.

At a glance, Guy Steele’s influential plan for growth idea from his “Growing a Language” talk seems to contradict what I just said. Shouldn’t defining an interface help us plan for future, alternative implementations of the Mediator? Perhaps a different kind of a Mediator?

Removing the Mediator trait and simply renaming its only implementation will still keep the code working, with the benefit that there are fewer lines of code now, and it isn’t any harder to extend.

This is actually more in line with Steele’s idea. It doesn’t say anywhere a trait or interface cannot be distilled from a set of basic implementations. In other words, when our intuition says to prepare for repetition, we should identify them. The Gang of Four book was never a recipe book for building great programs. It was a catalog! They observed several kinds of large-scale systems and programs and extracted repetetive behaviours in the code, patterns. They never said that to do things right, one ought to use the visitor pattern, or this other pattern, otherwise your programs will be bad.

Back to interface distillation. Programming is about getting rid of repetition. The more experienced the programmer, the better they get at noticing patterns of repetition. The downside is that this may also lead to overengineering for repetition.

So, an experienced programmer thinks, this behaviour I have specificed may be repetitive, let me first create a construct that lets me share the repetition (an interface), and then proceed with the implementation. This is fine if the actual code is known to be repeated, but by seeing interfaces as a hammer and every bit of code as a nail, you will soon bury yourself in pointless dependency injection scenarios.

It may be just as easy to first create a base implementation and once you must duplicate its behaviour, only then create the abstract implementation. You might actually need to spend less total time wiring up the interface, since you observed the repetition. Creating an abstract implementation first always involves a deal of speculation and this is not reliable.

The more experienced programmer understands that you don’t always need to plan for repetition. In fact, repetition is good sometimes. Not every shared functionality needs to be extracted to its own module, because sometimes, shared dependencies will be bad.

The approach I suggest is to in order to produce modularity as a side effect, structure your program into small, reusable pieces. Don’t create huge, monolithic interfaces. Functional programming shows us that dependency injection can be done just by passing a function.

class Foo(postponer: Request => Postponed[Request], delayer: =>Duration) {
  def resequence(requests: Seq[Request]): Seq[Postponed[Request]] = 
    requests map { r => postponer(r, delayer()) }
}

// ...
val m = Mediator()
val c = Clock()
val foo = new Foo(m.postpone, c.randomDelay)

One sees that a higher-order function like the above can be just as well represented by a trait with a single method. If you only need that method, you should depend only on that. With a structural type system it is easy to decompose types. An alternative is to stack traits, and in languages like Scala this is fairly easy. You could as well decompose Mediator into Validator, Postponer, et cetera, ideally interfaces should be fairly homogenous in their purpose: if your interface defines methods for reading, keep that interface separate from the writing interface, and if you need read and write just compose the interfaces together, and so on.

It also helps if your language is powerful enough to do without excessive DI – the reason why horrors like Spring exist is that Java simply wasn’t expressive enough to do dependency injection the traditional way without setting your hair on fire. That, and for some odd reason, people thought writing constructor arguments was so painful it warranted a gigantic framework for it.

Overall, it’s usually a good idea to toy around first with the concrete, and then extract the abstraction. Going the other way around is a dangerous swamp. It’s certainly something I’ve used to do – overengineer for patterns a priori – but I found better results by getting my hands dirty, by writing repetitive code first and then cleaning it up.

Implicit power

Antoine Kalmbach — Wed, 15 Mar 2017 00:00:00 +0000

Scala gets lots of flak for implicits. Some of the feedback is justified: implicits in Scala can be quite intimidating or confusing for beginners. That does not justify their dismissal, as implicits, in all of their flavours, when used correctly, can be actually quite simple and powerful.

I recently had to do a refactoring for large program. The codebase was old, and wasn’t designed to cope with the sort of change I was going to introduce, and I didn’t have much time either. Pressed for time, but compelled by a modicum of professional pride, I didn’t want to half-ass the task by adding jury-rigged solutions that would have left me feeling dirty and empty inside, at worst, leaving a rotting mess to future developers—me. The codebase itself was simple, but large. Its task was more or less to serve as a REST API in front of a high-availability, fast database (Cassandra). One part of the program provided abstractions called collections of database tables. Each collection had a set of methods (such as get) that were then translated to a database query to fetch entities. An entity is a piece of data encapsulating some value. Using a fictional and simplified example of an Bork from a database:

case class Bork(id: Int,
                date: ZonedDateTime,
                frobnicate: Decimal)

trait Borks {
  // find a bork by id
  def get(by: Int): Future[Option[Bork]]
}

// ... in another place, another module

class BorksImpl extends Borks {
  def get(by: Int): Future[Option[Bork]] = {
      // fetch by UUID from the database
  }
}

Each collection trait was implemented by a real class (like RealEntries), hiding database logic behind a concrete implementation. Other parts of the program accessed these entities via the collection trait, like the API front here:

val borks: Borks = ...

// spray dsl
pathNamespace("borks" / IntNumber) { id =>
  get {
    complete {
      borks.get(id).toJson
    }
  }
}

The database in question was Cassandra, in which this database abstraction doesn’t really exist, as databases in Cassandra are actually just prefixes called keyspaces that map to physical directories on the disk. These keyspaces have some properties that separate one keyspace from another, but the point is that they are unlike traditional SQL databases: you need not connect to a database, you can simply switch your query for the table Foo, in keyspace A to B, by switching A.Foo to B.Foo. So, in Cassandra, these keyspaces are opaque and you can simply choose the appropriate keyspace with the right namespace in the table name part of the query.

The task was to support multiple, concurrent databases of entries. Previously, this program operated as a monolith, i.e. there was ever only one database it was operating on. Support was needed for concurrent access to several (possibly non-finite) databases, and the support had to come quickly.

Turns out the simple solution – instantiate one BorksImpl for each keyspace – was not available, as there could be entities in one shared keyspace mapping to other keyspaces. So, one collection like BorksImpl needed to know which keyspaces it was supposed to query, because this information is unavailable to the caller.

A way around the splitting and namespacing was consolidation, but this introduced security problems. We couldn’t simply consolidate all the entries into the same database, as we had access limitations – callers of get acting on keyspace Foo were not allowed to see the data in keyspace Bar. This justified the creation of a split by keyspace, isolating data for the purposes of permission control. This also destroyed the possibility of the above solution, i.e., instantiate one BorksImpl for each keyspace, because one BorksImpl might have needed to query for data from many keyspaces.

So, a request with an id 123 comes in at /borks/123, the application uses the central lookup table to find the target keyspace. The initial implementation looked like this.

trait Borks {
  // find a bork by id
  def get(namespace: String, by: Int): Option[Bork]
}

// ... in another place, another module

class BorksImpl extends Borks {
  def get(namespace: String, by: UUID): Option[Bork] = {
      // fetch by UUID from the database
  }
}

And update the caller API:

val borks: Borks = ...

pathPrefix("borks" / IntNumber) { id =>
  queryNamespace(id) { namespace =>
    get {
      complete {
        borks.get(namespace, borkId).toJson
      }
    }
  }
}

This was fairly simple, but painful, as the get methods of collections like Borks may have called other methods on other collections, nesting calls ever downward, as shown below in the example, where Borks calls barks.get and so forth. As a result, I had to deal with adding the namespace: String parameter to all methods on all collections. Remember, adding the namespace method as a field was not an option – the namespace was an extra parameter to every method invocation.

So I was dealing with transforming code that looked like this:

val barksImpl: Barks = ...
def aggregateWithBarks(id: Int, barks: Set[Int]): Future[Seq[Borks]] = {
   val aggregates = get(id) map { b =>
     b map { bork => 
       barks flatMap { bark =>
         barksImpl.get(bark.id) match {
            ...
         }
       }
     }
   }
   ...
}

and by adding namespace everywhere, I had to transform it into

val barksImpl: Barks = ...
def aggregateWithBarks(namespace: String, id: Int, barks: Set[Int]): Future[Seq[Borks]] = {
   val aggregates = get(namespace, id) map { b =>
     b map { bork => 
       barks flatMap { bark =>
         barksImpl.get(namespace, bark.id) match {
            ...
         }
       }
     }
   }
   ...
}

So I had to add namespace: String to barks.get and borks.aggregateWithBarks. Sounds tedious? Well, imagine there weren’t just one call to barksImpl.get, but tens, and imagine there weren’t just two collections, but a hundred – and tens of thousands of lines to refactor.

Specifically, I didn’t want to keep adding namespace, into every method call inside a method call, but chose to make it implicit instead. This way, I needed only pass the implicit parameter around, and I didn’t need to modify any of the nested method calls. I typed the namespace with a custom case class and added it as an implicit argument:

case class Namespace(namespace: String)

trait Borks {
  def get(id: Int)(implicit namespace: Namespace) = ...
  def aggregateWithBarks(id: Int, barks: Set[Int])(implicit namespace: Namespace) = ...
}

trait Barks {
  def get(id: Int)(implicit namespace: Namespace) = ...
}

So, that was one particularly nice use case for implicit parameters. The good thing is that if the datastore is redesigned cleanly so that you cannot access from one namespace (keyspace) to another, all you need is to instantiate BorksImpl and set implicit val namespace = ... upon instantiation, and the code will work just fine. Implicit parameters let me implement a painful refactoring very quickly.

Naturally, had I had more time, I would’ve done the separation properly, implemented namespacing rules more clearly, completely redesigning the database, and so forth. Anyway, with Scala implicits, I was able to do a non-proper solution in a way that did not elicit a “jesus christ what a hack” feeling. It didn’t pollute my code too much and it will be easy to refactor out when it’s no longer needed.

And, it turned out, I was able to benefit from other implicits: conversions and arguments. I needed the ability to convert from the Namespace entity into a String, as I had in the querybuilder syntax. I needed only to insert namespace instead of having to write namespace.namespace.

object Namespace {
  implicit def namespace2String(n: Namespace): String = n.namespace
}

Session.prepare(QueryBuilder.insertInto(namespace, table).values(...))

Another nice thing was using implicit arguments. The REST API gets the namespace from the URI segment as a parameter to the anonymous function. If I called borks.get I would have needed to put an implicit val n: Namespace = namespace. I avoided that using the implicit argument method:

val borks: Borks = ...

pathPrefix("borks" / IntNumber) { id =>
  queryNamespace(id) { implicit namespace: Namespace =>
    get {
      complete {
        borks.get(borkId).toJson
      }
    }
  }
}

The implicit namespace: Namespace => is equivalent to having namespace => implicit val n: Namespace = namespace; .... Very useful if you’re calling methods requiring implicits in closures, though potentially hazardous, if you’re not typing your implicits! A simpler example:

trait Vyx {
  def frobnicate(num: Int): Int
}

// contrived example, makes no sense
def foo(i: Int)(implicit vyx: Vyx) = {
   i * vyx.frobnicate(num)
}

val foo = Seq("one", "two", "three") map { implicit v: Vyx => foo(1) }

It’s a good idea to type your implicit values as defining an implicit x: X will yoink any implicit X in scope, and if this X happens to be a basic type like String, and you’re not careful, you end up with the wrong implicit value.

Implicits weren’t a new thing to me, this was just a scenario where I was able to simultaneously benefit from many kinds of implicits Scala has to offer (parameters, conversions and arguments). They let me perform an annoying refactoring quickly and painlessly, in a manner that was also future-proof.

Apache Camel and the price of abstractions

Antoine Kalmbach — Wed, 08 Mar 2017 00:00:00 +0000

Apache Camel is a routing and mediation engine. If that doesn’t say anything to you, let’s try this: Camel lets you connect endpoints together. These endpoints can vary. They can simple local components, like files, or external services like ActiveMQ or web services. It has a common language format for the data, so that your data can be protocol agnostic, and an intuitive DSL for specifying the connections and how the data should be processed between messages.

The common language consists of exchanges and messages. These are translated into protocol-specific formats (like a HTTP request) by components, which provide the technical implementation of that service, i.e., the translation of a simple Message into an actual HTTP request.

The connection method is an intuitive DSL that speaks in terms such as from and to. Informally, you can create a route that can, for example, read messages from ActiveMQ, and write them to a file. The language is much richer than this, grouping together things like aggregation, filtering, routing, splitting, load balancing, the list goes on.

Choosing what component to instantiate is done using an URI. An URI will identify the target component, e.g., rabbitmq://myserver:1234/... instantiates the RabbitMQ component, file:... instantiates the file component, netty4:... instantiates the Netty component (version 4.0). As long as the component is available in the classpath, it will be instantiated in the background by Camel. The total number of available components is huge! You have e.g.:

ActiveMQ, RabbitMQ, Kafka, AVRO connectors
Files and directories
REST, SOAP, WSDL, etc.
More esoteric ones like SMPP – yes, you can send SMSes with Camel!

So what’s the point? Let’s assume we need to integrate an upstream system Xyz into Bar. Xyz provides data to you using a binary JSON format, using some known protocol, like ActiveMQ. Then you need to apply some transformations to the data, finally sending it to Bar, which accepts XML, and requires the information to be POSTed to someURL.

In a non-camel setting, using your favorite language, to do this, you

Using an ActiveMQ connector, you build your queue reader and de-serializer
Apply your business logic (whatever that is) to the de-serialized data
Transform into XML
POST the data towards someURL using some HTTP library

Fairly straightforward, right? All you need are an ActiveMQ library, a HTTP library and something that works with JSON and XML.

Here’s where it gets hairy. Three months in, you are informed that the upstream source is converting to RabbitMQ. Oh well, you think, it’s nicer, faster, and implements a saner version of AMQP, why not. So you refactor ActiveMQ to RabbitMQ and there it is.

The point of Camel is this. The previous step requires you to manually refactor your ActiveMQ logic to RabbitMQ. But you’re just sending messages, you don’t really care about the protocol. You’re just sending messages to an endpoint, it’s the data you should care about, nothing else.

So here’s when Apache Camel comes in. It let’s you specify an URL like

rabbitmq://localhost/blah?routingKey=Events.XMC.*

to use the RabbitMQ component, and to painlessly switch to Kafka, you’d add a dependency to the camel-kafka artifact and specify the URL as

kafka:localhost:9092?topic=test

and the Camel Kafka component handles message delivery for you. Since you’re sending canonical camel messages, you needn’t trouble yourself on how this message is already sent. It is likely that you will have to add or remove some message headers though.

Now, you may be asking, is that it? Is it really that simple?

The answer is that it depends. Some components are better than others. If you want to be truly protocol and component agnostic, and you want to refactor from protocol Foo to Bar just by switching the URL of foo://... to bar://, you need to make sure that

You can configure everything for that endpoint using the URI
Message exchanges do not require extra shenanigans to work (no custom headers or a special format required)

Case in point, let’s compare switching from ActiveMQ to RabbitMQ. The first glaring difference is that the ActiveMQ component does not accept the host part in the URI. So we need to do something like

CamelContext ctx = new DefaultCamelContext();
ctx.addComponent("activemq", 
    ActiveMQComponent.activeMQComponent("tcp://USER:PASS@HOSTNAME?broker.persistent=false"));

This makes any activemq:... URI in the context ctx connect to the parameters configured.

Conversely, the RabbitMQ component lets you directly set this in the URI part (multiple addresses can be given with the addresses parameter). So if you’re going with ActiveMQ to RabbitMQ, your code actually becomes simpler, but the complexity merely moves to the URI. The other way around, you have to move your URI-configuration to actual code (or XML, but please, don’t).

So where does this lead us? Ideally, the situation is that given between a choice between three components, you could use an external configuration file that configures a simple URI. The right component is identified based on the URI, pulled out of the classpath. This assumes that, in order of importance,

the endpoints are volatile and finite and can vary between different implementations,
each implementation has a Component which is in the classpath, and
said volatility varies often enough it warrants dynamic configurability via configuration editing and app restarts.

If all of the above hold true, Camel might a good fit for you. Otherwise, I’d be careful: the abstraction isn’t free! What this leads to is a kind of complexity shoveling: although with the RabbitMQ component we don’t need to use code to configure it, we move it to the URI. So it’s still a configuration point. Yet, it’s a nicer configuration point. As in the example above, we see that the connection contains three configurable variables USER, PASS, and HOSTNAME. So, in addition to having to configure the system using code, we have to still configure it otherwise, lest we hard-code the values into the application.

The above approach suffers from decentralization: you now have two places where you customize your system. The first is defining the custom component for a system in code. The second is configuring said custom component via other means.

Our ability to centralize configuration – any configuration, not just that of Camel – depends on the power of the configuration language. Too powerful, you end up in DSL hell. Not powerful enough, people write their own horror shows to add power.

Lastly, we run in the problem of universal pluggability, or universal composition. We imagine that systems like Camel let us “run anything” and “connect everything”, but the reality is different. Systems are usually made of a finite set of components. For practical purposes, it makes no sense to depend on every Camel component. Therefore, you need to pick your dependencies from this finite set of known endpoints. This effectively shatters the myth of universal pluggability.

Most importantly though, nobody really needs this. What really matters is the simplicity of extension. A well designed component is completely configurable through its URI parameters. These are easy to add to your Camel-based system: you only need to understand the new configuration, add the dependency and you’re done.

In summary, if you’re considering Apache Camel, make sure you check both of these, of which the second is most important.

The components are volatile and you need to change them often, so that you can justify the pluggable hole (the changing URI!)
The components you want exist and are completely configurable via that pluggable hole

If you’re unsure of the first item, you can still treat Camel as a lazy way to future-proof the system, e.g., by using one component now, while knowing that another may be used in the future. To that end, you need to make sure that the components fit the above requirements.

I’m currently working on a Clojure library for a Clojure-based routing DSL. It’s shaping up to be quite nice! Here’s an example of the routing DSL:

(route (from "netty4-http:localhost:80/foo")
       (process 
         (comp println body in))
       (to "rabbitmq://localhost:5672/foo"))

My goal is to make the DSL terse and functional (which the current model really isn’t) and to add Akka Camel Consumers and Producers to it. The nice thing about Clojure is that the macro system lets me define these really easily!

Overall, Camel is a nice abstraction, well worth the effort and years that has been put into it. It’s not a free abstraction, since there’s always a slight compatibility or configuration overhead. If it works, it removes programmers from the protocol level, moving them to the data level. This is the level where you should be working at, if your goal is to shuffle data around. For this purpose, when it works, Camel is excellent.

Conversely, if it doesn’t, it puts programmers at an awkward position: you’re still working with both data and protocol, and you have the overhead of the framework to deal with. Worse, your code is now polluted by the requirements of Camel endpoints, when the goal of Camel is to completely remove the requirements imposed by endpoints in general.

That said, in integration scenarios, Camel works most of the time, so you should always have a think about it before you start using it.

Half stack web frameworks

Antoine Kalmbach — Wed, 02 Nov 2016 00:00:00 +0000

In my previous post, I discussed how web development had become weird. In this post, I will discuss what exactly is it that makes it so weird. I will also present an alternative to JavaScript-based SPAs that look and behave like them, yet at the base are built using standard full-stack frameworks. They can leverage modern JavaScript libraries like React and compilers like Babel while simultaneously avoiding the confusing tooling ecosystem and providing a rich and responsive user experience, all the while retaining an pleasant developer experience.

What exactly is wrong with the tooling ecosystem?

I think, largely, the reason why web development has become weird was that front-end development cannot figure itself out. We are wasting effort and time by building more and more elaborate abstractions that fundamentally exist only because of an unhappy accident: the web is the only cross-platform application container. It is also a very accessible medium. To create a web application, fundamentally, one needs to present the right kind of mark-up to a browser that renders it.

Let’s stop here. Just because the web became what it is by accident, doesn’t make it a bad thing in itself. Everybody loves platform independence. Everybody loves accessibility. The web is easy to develop for and it can reach almost everybody. This is a reality we have to deal with, a reality in which web development is (a) popular, (b) ubiquitous and (c) easy.

The combination of those properties creates an interesting melting pot of rapidly evolving technologies. Rapid progress is a nice thing in itself, but a bad thing to the ecosystem when it evolves blindly. Web development doesn’t evolve blindly, rather, it is myopic.

Progress, progress, progress!

To put this into context, we must understand that currently, most software is disposable. Because software is disposable, we eagerly toss a half-functioning solution into the bin and rewrite it, rather than taking it apart and rebuilding a better version. This leads to programs getting rewritten and rewritten, sometimes doing things differently but most of the time it’s just the same thing under a different layer of paint.

But I digress. That is more of a problem with software development in general. We can review a more concrete example: the JavaScript tooling ecosystem. To develop front-end in JS, you need three different tools;

A package manager - npm, bower or yarn
A module bundler - webpack, rollup or browserify
A task runner - gulp, grunt, brunch

Each of these segments work completely differently. So to get started with, that’s three different tooling systems you have to learn. Better yet, each individual tool inside each system is unique in its configuration syntax. So if you learn how to configure Grunt you will have to learn Gulp and Brunch from scratch. Joy.

Yeah, yeah, I get it. Bower was cool because it built flattened trees when npm didn’t. Gulp had a nicer configuration syntax, and Brunch was easy to get started with. Yarn is more secure and more reliable than npm. Webpack can inline your CSS and images and is more configurable than Browserify.

Not about killing innovation

At this point, you may be wondering that I desire a world in which there is but one alternative to every task. This is not the case. I only ask for reservation: if there are no fundamental ideological differences, if there are no personal incompatibilities between the developing organizations, is there any valid argument for building your own version of a tool, instead of contributing to an existing system?

I don’t think this is the case at this scale — obviously, a world with just one kind of tool or library for one thing is stupid, but the sweet spot does definitely not lie at seven.

So if the answer is no, does that mean JavaScript developers are so strange that they cannot get their heads together and agree on something? Do they really think people enjoy keeping up with the Joneses all the time and learning a new tool every year?

When it comes to the first question, remember Joyent and the io.js schism. Oops. As to the second, I doubt it. Still, this is what we have to live with. A guide for building a modern JS front-end app consists of twelve distinct steps, all of them quite elaborate. I applaud the author for the gargantuan effort in that tutorial: it’s the best I’ve seen so far. But seriously, take a look at it! What the fuck?! I could have just rewritten my previous post with a link to that guide and rested my case!

I remember a book about Windows programming using C, and parts of this guide are arcane enough to evoke memories of that. I think one can enumerate the type system of Scala using less. Or how to write a Scheme interpreter.

The usability of the tooling ecosystem is absolutely disgraceful. No other developer segment has this many hoops to jump through and nobody else has to learn so many different tools just to get a simple web application running.

Why do we put up with this? Why isn’t any effort being put into simplifying the tooling stack, instead of making it more elaborate, powerful, and verbose? Consider webpack. It is a powerful utility that is supposed to combine all your assets — that is, code, CSS, images — into a single module that is used in your application. This is a powerful thing. The only problem is that its configuration is hell. I work with SBT every day, and my goodness, even SBT is easier to configure than Webpack. Ask any Scala developer what it means to say that. You will get funny looks. Even Java folks will consider this crazy, although, in fairness, they’ve moved into the post-framework age, and consider us mortals rather quaint.

SPA development is more than just tools

The problems don’t stop here. A SPA must effectively handle client state entirely in the browser, though in ~~isomor~~universal SPA apps part of the rendering and client state is processed on the server. This requires the use of architectural patterns like Redux and React Router.

These libraries are nice and intelligent, but I feel they are a wasted abstraction. Using the trick below I can create React apps that can approximate the performance of a real SPA app, without having to rely on these complicated architectural patterns.

Caveat lector. This is largely a matter of taste. If you really like Redux and React Router, by all means use them, but I find their usability to be sub-par to the MVC architecture of any full-stack framework. The architectural pattern — Flux — is a message-based event loop. The views generate user actions (button clicks) that are dispatched to stores (state containers) which update themselves (increment a number) then deliver state changes (an incremented number) to the views which re-render themselves. If a request is sent to the server, it must be split into two parts: first, a button click is registered, and its effect is rendered; second, a request is sent to the back-end and when it completes, an action describing a completed request is sent to the message dispatcher. So any interaction with the back-end requires two actions. Sounds complicated? Yeah, this is why I prefer a dumb MVC architecture (or Relay).

In summary

So, to put this argument into a more cogent form, I’ll summarize them below.

1. Lack of emphasis on usability, a myopic focus on adding features.

Why doesn’t anyone integrate dependency management, module bundling and task running under the same program? Why do we have to use three different programs that are getting replaced every year? Tool “monoliths” like SBT may be ugly in parts, but they can do package management, compilation, debugging, testing – even if it’s DSL is garish and confusing, still, once you’re familiar with it, you don’t have to master six other horrifying DSLs. Just one.

2. Chasing novelty with little care about its impact on maintainability.

Babel lets us write JS in eleventy different dialects. While that is a cool thing in itself, it a horror show for developers. You ask, who wouldn’t want to use await, or ES6 classes? Well, how about the person who doesn’t want to learn how to use Babel?

With Babel, you can write in any version of JavaScript you want, since it all gets compiled down to ES5 anyway. This is great for building your flavor-of-the-month hack, but it’s also a terrific way of building unmaintainable software. For this zany hack to work, you need ~~tra~~compilers that translate your modern code to old code. The requirement of that tool is too high a price to pay for some fancy language features.

3. Snubbing full-stack frameworks for their want novelty, although they generally feature exemplary usability

Clojure developers have found a way of eschewing frameworks over composable libraries. For some reason, everybody else is really bad at this, so we build frameworks, i.e., sets of libraries that govern the design of your program in a certain way. Monolithic frameworks like Rails or Django are fundamentally dated — though this is easily fixed — but they are usable. Setting up a functional application with these takes a few minutes, and it just works.

A new direction: renovate, not rewrite

In my opinion, front-end development can be done in an alternate, saner way. It doesn’t mean going back to the stone age of Apache or Rails with ActiveRecord. Rather, it means refurbishing these old, battle-tested technologies with modern components without tossing the whole chassis into the bin.

In other words, there is an alternative to the current JavaScript SPA horror show. Using the following technologies, as an example:

A REST API built in a scalable and performant language

Examples: Scala, Haskell, Go, Clojure, Java, Rust, OCaml, Elixir

This gives us a clear advantage when scaling and deploying our application. Data access is made opaque and is in no way tied to the front-end - which is ultimately just presentation and some client state. The language needs the following:
- A stable library ecosystem, especially for data access, e.g., database drivers
- A functioning web server and associated libraries
- Speed, multi-threading, performance
With these properties, you should be quite comfortable in your back-end development.
Client state, presentation and back-end communication handled using a monolithic framework

Examples: Ruby on Rails, Django, Pyramid, MeteorJS, Udash, Play

Rails may be dated in some parts — coupling your front-end with data access is one thing — but as an infrastructure it is functional, mature, easy to understand and stable. The Ruby ecosystem is large and is well documented, even the secondary documentation (StackOverflow etc.) is abundant.
A wrapper that turns ordinary HTTP page requests into XHRs

Examples: Turbolinks (for Ruby on Rails and Django)

Turbolinks is perhaps a hack but it is clever: any HTTP request that would normally cause a page reload, like a link or a form submission, is converted into an XHR. Then, the page redraws itself by swapping out the <body> element from the returned response.

Turbolinks is a “pseudo-SPA” application in that it simply reroutes ordinary page requests (links, form submissions) as XHRs and then from the new page, it merges the <head> element and swaps the <body> element. By using a gem like react-rails you can combine this with react, however, it does not use React’s virtual DOM when redrawing the body content. It only mounts and unmounts the components when the page swaps, retaining the actual DOM bindings.

What?! Your answer is Rails? In 2016?

Just because these frameworks aren’t making headlines doesn’t mean they are stuck in the stone age. These monolithic frameworks still, after years of maturation, possess novelty value in one, unparalleled aspect: usability. These frameworks may not lend themselves to universal applications, but they’re still capable of absorbing new technologies like websockets and GraphQL.

Some parts of them are stuck in the past, of which the most striking one is combining data access with data control and presentation in the same program. This is easily fixed: make your Rails controllers call an external, opaque service to render is data. The job of the full-stack framework is reduced to managing client state and data presentation, which go together.

So, what can be done? Here’s an example.

A REST-backed Rails app with React as the templating engine

react-rails is a Rails gem that gives us React components in the asset pipeline, supporting server-side rendering and Turbolinks (caveat: see above)

Under the hood, when rendering on the server, react-rails uses Babel and ExecJS to prerender the content. Better yet, your content is still rendered by a simple Rails controller like the following.

The controller lives in app/controllers/foos_controller.rb:

class FoosController < ApplicationController
  # maps to GET /foos (on the front-end)
  def index
    # incurs a GET /foos on the back-end
    @foos = Foo.all.to_json
  end 
  
  # maps to POST /foos (on the front-end)
  def create
    # this is a POST /foos on the back-end
    Foo.create(:bar => params['bar'])

    # turbolinks turns this into a XHR
    redirect_to '/foos'
  end
end

The model is just a Her model, an ORM that uses a REST API, which you can customize. In apps/models/foo.rb:

class Foo < Her::Model
  attributes :bar, :id
end

Now Foo.find(1) maps to GET /foos/1 in the back-end, and so forth.

The view is generated by app/views/foos/index.html.erb

<%= 
react_component(
  'Foos', 
  { foos: @foos, token: form_activity_token, action: url_for(action: 'create') }, 
  { prerender: true }
) 
%>

This maps to a React component app/assets/javascripts/components/foos.es6.jsx:

class Foos extends React.Component {
  render() {
    <div>
      <ul>
        {this.props.foos.map((foo) => {
           return <li>{foo.bar}</li>
        })}
      </ul>
      // dataRemote is a Rails trick that makes the form make an XHR
      <form action={this.props.action} method="POST" dataRemote="true">
        <input type="hidden" name="authenticity_token" value={this.props.token} />
        <input type="text" name="bar" value="Blah blah" />
        <input type="submit" value="Add!" />
      </form>
    </div>
  }
}

Try doing that with less code in any JS app! The controller looks like any standard Rails controller. In fact, it is exactly like one, yet the magic of React & Turbolinks lets us wrap this into a SPA-like experience.

Combining these elements, we get an application that can reach nine-tenths of the performance and responsiveness of a 100% JavaScript SPA, while simultaneously avoiding the messy tooling ecosystem.

A total absence of extraneous tooling, the framework has these built-in. No need for Webpack or Babel, these are just another gems you add to your dependency list.
A boring, but familiar, framework that handles routing, message dispatch and API integration for us. Routing and state management are the worst parts of SPA development. Now our state is just another Rails
Responsiveness close enough to that of a real SPA. It will never match a real SPA in speed, since the requests map to Rails controllers, but it will be extremely pleasant to develop in.
A scalable back-end without any data access logic in the front-end (the usual front-end back-end split), the framework handles only UI state and presentation logic.

There are some obvious compromises in such a solution, which are both good and bad.

Compromises made

The biggest compromise is in performance, which is due to the following:

Does not use React’s DOM to its full power. Turbolinks just swaps the body element. This could be improved by making it use the React virtual DOM. This is the bad part. The good part is that we don’t have to create XHRs ourselves in React components.
Forces the user to use JSX, throws ERB/HAML in the bin. It is true that the example application could be indeed built without JSX — just don’t use react-rails — but I find JSX to be a nicer templating syntax than ERB. advantage of React

But it would be naïve to assume this brings us the whole of React. It brings us the templating syntax and binding mechanisms, but since Turbolinks effectively causes a re-rendering of the whole HTML page, this doesn’t fully leverage the server-side rendering aspect of React.

So, overall, the good part of this compromise is that we get to use JSX, which has a nicer, functional approach compared to ERB, but the bad part is that we don’t harness the full power of React.
Turbolinks effectively reverses React server-side rendering. Whereas in a normal SPA app the server-side rendering is the “base” template, in this case a new server-side rendering is produced on every interaction. In a normal SPA app, one just updates the DOM with new state — i.e., props — not with a new DOM.

There is a solution: skip Turbolinks and use XHRs in React components. A simple solution in a controller:
```
def create
    @f = Foo.create(:bar => params[:bar])
    if request.xhr?
      # send a JSON of all the Foos
      render :json => Foo.all.to_json
    else
      # send HTML with a React component
      redirect_to action: 'index'
    end
end
```
If the request is made from a component, it’s can now use setState (or a store) to update its new state. In this paradigm, Rails is acting as the state store.

A better example would be to make the Rails app support GraphQL and use Relay to communicate with the Rails part, see below.

I think, given the simplicity of the above application, I think it’s fair to say that these compromises are warranted. If the actual set-up were any more complicated I wouldn’t be so certain. But, for the simplicity, we must trade performance.

A functioning example

I’ve created a functioning example and put it into two repositories:

Front-end – Rails 5 & react-rails & Her – https://github.com/ane/rails-react-frontend

A Rails 5 app combining react-rails and Her to talk to the back-end.

To install, clone the repo, run bundle install, run foreman start. This will start the Rails server and the live re-loader.
Back-end – https://github.com/ane/rails-react-backend

It’s a dead simple Sinatra REST API that uses Sqlite3. This is obviously not suitable for production.

To install, clone the repo, run bundle install, run rackup.

This application will never match a real SPA. A part of the front-end is not in the browser, so we will rely on a second web-server to run it. So it is an illusion, but as an illusion it is close enough, and it is easy to use.

Conclusion

JavaScript front-end development, as it currently stands, is painful to develop in. One has to master many command line tools that instead of being unified as a single tool, each continue to diverge and grow larger and more powerful. The result is a confusing developer experience.

In this post, I showed that we can scrape the good parts of modern JS developments and use them to modernize an older application stack that mimics the user experience of a SPA, but is not one. The application uses a clever library — Turbolinks — to convert page requests into XHRs, creating an illusion of a single-page application.

The end result is a half stack web framework: we yank data access from a monolithic full-stack framework (Rails) and make it use a REST API and we replace its presentation logic (ERB) with React. The framework is left to handle client state, routing and asset pipelining, which are the painful parts of SPA development, and the UI is rendered using React. So the Model–View–Controller is distributed into three places: Rails for UI state, React for UI rendering, and the REST API is the actual business logic. Effectively, this reduces Rails to a thin SPA-like front-end over a REST API!

Where to go from here? Here are some interesting things that could be explored:

Turbolinks with React. Use React to parse the HTML returned by Turbolinks (if rendered on the server) and use the React virtual DOM to update the DOM, instead of blindly swapping the body element.
GraphQL. Although Her is nice, we could use GraphQL when communicating with the backend and also use it as a communication method between Rails and React.
TypeScript. I like static typing, but currently react-rails doesn’t really work that well with TypeScript.
React On Rails. A different kind of React & Rails integration, which lets you use Webpack. React On Rails is more flexible than react-rails: you get the full power of Webpack and NPM here, so this is both good and bad.

All in all, this solution is a compromise.

Compared to a full-stack Rails app, we have to do extra work in creating a REST API backend, but the result is an app that’s easier to manage due to the separation of concerns. With a separate data access layer — the REST API — complex business logic is contained in a single place. It is easy to couple several clients to such a front-end, and our Rails app is just one of these.

But, compared to a full-fledged SPA, this app will never be as quick, it will never be as fluid, and it may not be what cutting-edge front-end development this day represents, but it is is simple, there is one build tool (bundler), and it is fun to develop in.

I might miss fancy things like state hydration and Redux, but the insanity of Webpack, Gulp, Babel and NPM, I will not miss.

Web development has become weird

Antoine Kalmbach — Tue, 25 Oct 2016 00:00:00 +0000

Call me old-fashioned, call me a curmudgeon, but I think web development has become stupid and superficial. The unending quest towards single-page apps (SPAs) has made web development extremely painful and the current trend is diverging towards seven different directions at once. On one end, we have rich SPAs that can be built as native applications, on the other we have something completely orthogonal, of which a schism is beginning to form.

The underlying problem is unfortunately that the web is being misused as an application container instead of the original text transport protocol it was made to be. It’s no use crying over spilled milk; the web has been subverted, transformed, improved upon, so much so we don’t know what the original even looked like.

How it was

In 2006, the hot new thing was Ruby on Rails or Django. If you weren’t using them, odds were you were using PHP or ASP.NET. Most intranet software ran on SharePoint or, I kid you not, WordPress. Users didn’t really care either way.

People liked Rails and Django because they made web development stupidly simple. No more SQL, just create your models and migrations. An architecture that made sense, MVC, was applied, and web apps became a little bit better. Meanwhile, the overall web development experience got a lot better.

Of course, the web was slower back then. Chrome wasn’t around, so JavaScript usage was very limited. Google began prototyping under-the-hood requests in Gmail around 2006, but before that nobody had heard of AJAX. The concept of doing more than one page request per page load was completely unheard of. The users liked faster page loads, so when Chrome came around with V8, customers started suddenly giving a shit about what browser they used.

Where it all began

On the surface, the appeal in SPAs was obvious. It started with Gmail and AJAX. No more slow page loads, the applications behaved like native applications, and soon they even looked like them! Innovative as that was, now we’re beginning to use so many web applications that are in the web only that we’re slowly starting to forget what the native app experience was.

The problem was that it wasn’t enough, you needed a backend. Before, when there was one application, now there were two, and they usually were completely different from each other. The backend–front-end split was fuzzy to begin with, this introduced an uncertainty and a possibly pointless abstraction. Put the “slow” and “heavy” things to the backend, let the front-end handle rendering and the user interface, all the backend had to do was supply serialized data. Even back then, people started asking questions about the SEO effects of rendering a page entirely in JavaScript. No solution was given, although one solution existed, but was weird.

So while the backend folks built eleventy versions of Sinatra, the front-end folks got busy. In a short time we had Backbone, Angular, and Knockout, then we got frameworks like Durandal and Meteor.js. Finally, Facebook looked at the performance of desktop applications, then looked at the performance of web applications, thought, “holy shit”, and did something about it.

People got scared. It was mixing business and presentation logic, they said. It was mixing JavaScript with something eerily like XML, and everyone said XML sucked. Then people got over their usual trepidation towards $newTechnologyOfTheYear and got on with their lives. Now React is being used left and right.

The only problem was, React was a templating engine at heart. Facebook did not build a bridge for existing front-end frameworks, so that people could have just dropped in React instead of say, Handlebars or even ERB. Facebook did not do this because they already had their own way of rendering content. They didn’t need one. Build your own, they said.

Faced with just a templating engine, developers got confused. “How do I do routes with this?” they asked. So we built routing engines and state containers, and got on with our lives. Soon after that, someone understood React ran quite fine on a Node.js server, and people started rendering pages in two places: the backend and the front-end.

Now, people are using React – a JavaScript library to be run inside a browser – to create native mobile applications. Meanwhile, other folks think, all of this, this excession, is simply too much, and want pages to load quickly.

Couple this with the at least bizarre experience of JavaScript development in 2016, things are looking weird. The tooling iterates at an impossible speed, a new build system emerges every year, and developers must stay on top of things.

Having to stay on top of things is, generally, a good thing. Software progresses, it progresses so fast that we must constantly learn for us to stay employable and the profession to stay enjoyable. But at this speed, when it seems we’re not really learning from the past, it’s not doing anyone any good. React took a good idea from desktop applications, event-driven user interface rendering, and executed it brilliantly as they ported it to the web.

The thing is, it’s still nothing new. Ten years ago we were building crappy and weird-looking software in C#, now we’re building crappy and broken software in a mix of JavaScript and other languages, and they run in the browser, or on smartphones, and they’re responsive, so that when you tilt your tablet sideways, that big fat menu disappears. Huh.

That’s what they call the churn.

The churn. New technologies come and they kill the old technologies, but in the midst of it all, stand you and I, wondering what the hell to do with this mess. From the other side of it all, from the ivory tower of the real world, the business analysts cast their shadow and remind us these technologies are tools, they’re meant to be replaced, they’re disposable. So are we, if we can’t learn new ones, they remind keep reminding us.

So?

I make it sound as if web development is impossible, but that couldn’t be further from the truth. Browsers are getting better and faster. Our applications are prettier, faster, more accessible, more usable. The web is replacing desktop applications and this trend is accelerating – whether this is a good or bad thing, I don’t know.

The only problem is that the development experience keeps reinventing itself at such a pace you may as well put yourself into stasis and wait for things to settle. Wait for front-end development to become boring. Odds are you can sleep for quite a bit until that happens. The second option is just to pick whatever works right now and use it.

The optimistic part is that we, as web developers, are learning, we’re doing some cool things and unifying two halves of the same thing. The backend guys are innovating and tooling progress is insane and exciting. So I cannot state that we haven’t gotten anywhere, we have innovated, learned, and improved the Web. But by how much? Are our end users happier?

A concrete solution

Given the task of implementing a web application, what would I do, given the state of the art in 2016? I spent about four years developing SPAs with many frameworks. I hate them all. Given that sentiment, this is what I would do:

Using a language of your choice, build a business logic API that can be used via REST or some other RPC protocol. The language and its associated tooling should be performant and support rapid iteration.
Use a batteries-included web framework, spiced with a rendering framework of your choice, to create front-end.
Build many front-ends, not just for the web, but for mobile and perhaps even desktop, and keep them thin.
The web front-end can be spiced up (but not replaced) using JavaScript. Come to think of it, I would have done the same thing in 2006.

Point 4. originates from my experiences of creating and maintaining SPA applications. I think SPAs are, by and large, a bogus concept. A web application loading another page isn’t intrinsically a bad idea, if your application is fast enough. Conversely, if your SPA is slow, you’re doing it wrong. SPAs were invented for speed, because conventional web frameworks were slow. This is not the case anymore. Sure, you won’t see Rails, Django or Play beat the TechEmpower benchmarks, but we’ve come a long way from five years ago, which is when people started to play around with SPAs.

Given the speed improvements, why not go full-stack? Why a front-end and a back-end?

The answer for this is not simple. It is because we’re dealing with two incompatible abstractions:

Building your application as an API means you need a client application to provide the user interface.
To build such an interface, your application has to deal with the fact that HTTP, and thus REST, is stateless.
Web applications are usually stateful.
This leads inevitably to the requirement of building an abstraction in the middle that handles client state, which your API does not support.
Building such an abstraction – the front-end – requires a lot of work, e.g. by using a MVC (or MVVM whatever) model. Double the work, half the fun.

So, the back-end abstraction is incompatible with client state, but the front-end application requires client state. Conversely, a full-stack application is often a heavy monolith: it needs to handle data access, its modification and its presentation in the same package. Here, as they say, be dragons. We want to keep business logic and presentation logic separate, hence, a full-stack framework does not work on its own.

As a solution, I offer a synthesis. It’s mixing a REST back-end with a full-stack frontend. The back-end can be built using whatever language is performant and maintainable. Build your front-end with a boring framework like Rails, Django or Pyramid; let it fetch its data from the REST API, i.e., treat the API as the data source. Let the front-end handle client state on its own. What you get in return:

The ease of use of said framework. These frameworks were invented for a reason. You get routing, templating, asset pipelines etc. out-of-the-box.
You can still do AJAX requests easily to build rich user interfaces.
A reusable API in the backend you can use in other applications, keep your web front-end an equal citizen.

If you don’t want to deal with framework bloat, or if you’re scared of non-JavaScript applications, be my guest, build your own front-end using the essentials. Splurge in Gulp, ES6, React, and Redux. Or use TypeScript. But I dare say, after having worked with both full-stack frameworks (e.g. Rails) and SPA+REST frameworks, the compromise above is much more pleasant.

In the end though, it doesn’t really matter: with the exception of a few, our end users couldn’t care less. They really don’t give a shit. So, pick whatever technology works for you and your users. The above is just one option.

Communicators: Actors with purely functional state

Antoine Kalmbach — Fri, 14 Oct 2016 00:00:00 +0000

In Scala, Akka actors, as in the traditional Actor model, may modify private state. The accepted convention is to have a mutable object (e.g. a Map), a var, and mutate it like so:

class Library extends Actor {
  var books = scala.collection.mutable.Map.empty[String, String]
  
  def receive: Receive = {
    case AddBook(isbn, title) => books += (isbn -> title)
  }
}

object Library {
  case class AddBook(isbn: String, title: String)
}

This is a bad idea. There are several reasons for this. First, Scala eschews vars, they should only be used when absolutely necessary (read: never). There is also an additional need for thread-safety for the collection, not because of the receive method itself. The receive method is guaranteed to run inside a single thread. However, an unsuspecting user might still launch a Future and modify the collection, leading to unpredictable behaviour. Such concurrent mutations on a var put strain on the garbage collector, in fact, it often necessitates the existence of a garbage collector.¹ Lastly, as with any mutable state and possible lack of referential transparency, the code can become hard to reason about.

Thankfully, Akka actors offer a possibility to do this completely functionally. The function context.become allows an Actor to change its receive method on-the-fly. In other words, it lets the Actor change its state and communication model. Here’s the above implemented using this paradigm:

class Library extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => {
      // for immutable maps, += returns a new collection
      context.become(active(books += (isbn -> title))) 
    }
  }
}

The active function returns a new Receive, receiving the current actor state as its parameter. Adding logic to it is now easy:

class Library extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn title) => {
      if (books.size < 10) {
        context.become(active(books += (isbn -> title)))
      } else {
        sender() ! "Too many books"
      }
    }
  }
}

The above code is now thread-safe and doesn’t use mutable collections, but what if our logic gets more complicated? What if we need to talk to another Actor, or talk to the sender of the message? This is where we stumble upon a design feature of Akka: all of its Actors are actually compiled down into a callback-based implementation. There is no guarantee that a Future launched in a receive case will be running in the same thread as the next! One could argue that this is not a feature but a flaw, but I won’t go that far. Hence, code dealing with Futures in Akka actors needs to deal with the unforgiving reality that there is no guarantee of thread safety. Case in point:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case AskForBook(isbn) => {
      popReservation(isbn) foreach { i =>
        // AAH!!!
        context.become(active(books - i))
        sender() ! s"Here you go: $i"
      }
    }
  }
}

Why am I screaming in the comments? First, as calling map for our Future launches a new thread, we have no idea whether sender() returns the same value in the new thread, and second, we may be modifying the books collection concurrently with other threads - leaving the garbage collector to collect our mess. So we strain the GC and risk giving the book to the wrong caller!

Since the actual execution of a Future is left to the execution context, which in the case of Actors is the ActorSystems dispatcher, we may or may not be invoking sender() in the right thread — there is simply no guarantee. We can’t reason about it, it has been hidden from us.

To deal with this, Akka has introduced the pipe pattern, which is an implicit given to Futures which solves this:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case AskForBook(isbn) => {
      // launch another thread
      val reservation: Future[String] = popReservation(isbn) map { i =>
        s"Here you go: $i"
        context.become(active(books - isbn)) // AAH!
      }
      // but sender() is still the same
      reservation pipeTo sender
    }
  }
}

Another option is to fix the reference of sender:

val s = sender()
val reservation: Future[String] = popReservation(isbn) map { i =>
  s ! s"Here you go: $i"
  context.become(active(books - isbn)) // AAH!
}

Ok, now we’ve fixed sender(), but what about the books collection? Let’s add a PopBook(isbn: String) case class, and handle that for removals:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case PopBook(isbn) => context.become(active(books - isbn))
    case AskForBook(isbn) => {
      // launch another thread
      val reservation: Future[String] = popReservation(isbn) map { i =>
        s"Here you go: $i"
        self ! PopBook(i)
      }
      // but sender() is still the same
      reservation pipeTo sender
    }
  }
}

Sending messages to self is always thread-safe - the reference does not change over time. So, at this point, it seems clear that making actor code thread-sane involves the use of:

immutable state - call context.become with a closure over the new actor state,
converting asynchronous state modifications as messages to be handled later, and
making sure the sender() reference is consistent

What about complicated states? What if we need to react differently to these messages, e.g., when the library is closed? I sense that you’re about to mention Akka’s FSM construct, which builds a state machine, encapsulating state and transitions to what is essentially syntactic sugar, and on the surface, seems like a good idea.

Enter Akka FSMs

At a closer look, it essentially leads us to repeat the same mistakes as above, and the arguments against it are argumented here. In summary, it boils down to:

Akka FSM’s is too restrictive. You cannot handle multi-step or complicated state transitions, and modeling undeterministic behaviour is impossible.
You are tied to Akka completely, you must use Akka testkit for your tests. Anyone who has worked with testkit knows this to be a burden.
State transitions have identity instead of being truly functional, that is, FSMs alter the current state instead of producing a new one.

Moreover, and I think this is the biggest shortcoming, the Akka FSM are finite-state automata — they are characterised by the state transition function (Input, State) => State. Since we know actors are more about communication than anything else, this model is insufficient, and what we need is a state machine that can produce output: a finite state transducer. Its state transition function has the signature (Input, State) => (Output, State) - every transition produces an output, and Scala can model this efficiently:

trait FSA[State, Input, Output] {
  def transition(s: State, i: Input): (Option[Output], State)
}

With all these flaws, despite being a nice idea at a glance, it’s obvious that for any complicated logic Akka FSM’s aren’t sufficient.

Let’s envision a radical version of actors, accounting for all the flaws described above:

State transitions should be about producing a new state, i.e. (Input, State) => (Output, State)
Actor computations will deal with asynchronous code, we must deal with this intelligently
Keep I/O logic out of actors - the actor only communicates with the external world
Actors should only mutate their state with with context.become

The last bullet point is especially important, as it constrains state changes to be entirely functional, as you can simply make a function def foo(state: State): Receive, and keep calling it recursively, by transitioning states thusly:

def active(state: State): Receive = {
  case someInput: Input => context become active(state)
}

This idea is not new. Erlang actors have worked like this for actual decades, and arguments for using this method in Scala can be found left and right, summarized particularly well in Alexandru Nedelcu’s Scala best practices.

active(Sum) ->
  receive 
    {From, GetValue} -> From ! Sum;
    {n} -> active(Sum + n)
  end.

Putting emphasis on the last point, I’ve come up with a moniker called communicators.

Actor, meet communicator

Let’s define the Communicator trait first independently:

trait Communicator[State, Input, Output] extends Actor {
  /** This is the initial actor state */
  def initial: State

  /** The state transition function */
  def process(state: State, input: Input): Future[(Option[Output], State)]

  /** The output processing function */
  def handle(state: State, output: Output, origin: ActorRef): Future[Unit]
}

initial is simply the initial state machine state, process is the state transition function and handle is the function that will deal with dispatching the result of process. Because we’re producing content in another thread, we want to make sure the reference of sender is fixed, and by using this with the pipeTo pattern, we get thread safety. Let’s extend the Actor trait to get receive

trait Communicator[State, Input, Output] extends Actor {
  /** This is the initial actor state */
  def initial: State

  /** The state transition function */
  def handle(state: State, product: Output, origin: ActorRef): Future[Unit]

  /** The output processing function */
  def process(state: State, input: Input): Future[(Option[Output], State)]
  
  def receive = active(initial)
  
  /** I/O handling which the deriving class must implement */
  def active(newState: State): Receive
}

The active function is the actual output-producing function. The user is left to define three things:

the initial actor state in initial
the output dispatch function handle
the state transition function process
the active function which handles input and output

To see this in action, first, let’s define the application states.

object Library {
  // Library state
  case class LibraryState(open: Boolean, books: Map[String, String])

  // Input alphabet
  sealed trait LibraryInput
  case class SetOpen(o: Boolean)                  extends Input
  case class AddBook(isbn: String, title: String) extends Input
  case class GetBook(isbn: String)                extends Input

  // Output alphabet
  sealed trait LibraryOutput
  case object SorryWeAreClosed                        extends Output
  case object DoNotHaveIt                             extends Output
  case object SorryReserved                           extends Output
  case class Book(isbn: String, title: String)        extends Output
  case class Reservation(isbn: String, title: String) extends Output
}

The actual state is just a case class: this gives us the nice copy function for easy updates. Then we use polymorphism to implement the input and output alphabets. Then we implement the actor itself:

class Library(getReservation: String => Future[Boolean])
    extends Communicator[LibraryState, LibraryInput, LibraryOutput] {

  import Library._

  def initial = State(false, scala.collection.immutable.Map.empty)

  override def active(newState: LibraryState): Receive = {
    case (output: LibraryOutput, origin: ActorRef) => handle(output, origin)

    case input: LibraryInput => {
      val origin = sender()
      process(newState, input) map {
        case (output, state) =>
          output foreach { o =>
            self ! (o, origin)
          }
          self ! state
      }
    }
  }

  override def process(state: State, input: Input): Future[(Option[Output], State)] =
    input match {
      case SetOpen(o) => Future.successful((None, state.copy(open = o)))

      case (GetBook(_) | AddBook(_, _)) if !state.open =>
        Future.successful((Some(SorryWeAreClosed), state))

      case GetBook(isbn) => {
        val book =
          for {
            (isbn, title) <- state.books.get(isbn)
          } yield {
            getReservation(isbn) map { reserved =>
              if (!reserved) {
                (Some(Book(isbn, title)), state.copy(books = state.books - isbn))
              } else {
                (Some(SorryReserved), state)
              }
            }
          }

        book getOrElse Future.successful((Some(DoNotHaveIt), state))
      }

      case AddBook(isbn, title) =>
        Future.successful((None, state.copy(books = state.books + (isbn -> title))))
    }

  override def handle(state: State, output: Output, origin: ActorRef): Future[Unit] = {
    Future {
      origin ! output
    }
  }
}

Decoupling Akka

So, now we’ve made a very thin actor, with little I/O logic inside it, but it’s still an actor. Let’s decouple it entirely from actor semantics. First, we define a StateMachine[I, O] trait:

trait StateMachine[I, O] {
  def process(input: I): Future[(Option[O], StateMachine[I, O])]
}

And excise the state logic from the Communicator, moving it to the State case class:

case class LibraryState(open: Boolean, books: Map[String, String], getReservation: String => Future[Boolean])(
    implicit ec: ExecutionContext)
    extends StateMachine[LibraryInput, LibraryOutput] {
    
  def process(input: LibraryInput): Future[(Option[LibraryOutput], LibraryState)] = {
    input match {
      case SetOpen(o) => Future.successful((None, copy(open = o)))

      case (GetBook(_) | AddBook(_, _)) if !open =>
        Future.successful((Some(SorryWeAreClosed), copy()))

      case GetBook(isbn) => {
        val book =
          for {
            title <- books.get(isbn)
          } yield {
            getReservation(isbn) map { reserved =>
              if (!reserved) {
                (Some(Book(isbn, title)), copy(books = books - isbn))
              } else {
                (Some(SorryReserved), copy())
              }
            }
          }

        book getOrElse Future.successful((Some(DoNotHaveIt), copy()))
      }

      case AddBook(isbn, title) =>
        Future.successful((None, copy(books = books + (isbn -> title))))
    }
  }
}

You may be wondering: wait, where’s the handle implementation? We kept that out from the state machine class since it’s not its responsibility - so we keep that in the Communicator:

class Library(getReservation: String => Future[Boolean])
    extends Communicator[LibraryInput, LibraryOutput, LibraryState] {
  import context.dispatcher

  def initial = LibraryState(false, scala.collection.immutable.Map.empty, getReservation)

  override def handle(output: LibraryOutput, origin: ActorRef): Unit = origin ! output

  override def active(newState: LibraryState): Receive = {
    case (output: LibraryOutput, origin: ActorRef) => handle(output, origin)

    case state: LibraryState => context become active(state)

    case input: LibraryInput => {
      val origin = sender()
      newState.process(input) map {
        case (output, state) => {
          output foreach { o => 
            self ! (o, origin)
          }
          self ! state
        }
      }
    }
  }
}

So, all state is kept neatly in a separate entity that’s entirely unit testable in its own right without having to rely on Akka testkit or the like – input and output dispatch and state transitions are done in the active method.

I know the state case class manipulation introduces more boilerplate, but as long as that boilerplate isn’t complicated, I think this is a fair compromise. Plus, one can use lenses to remove some of the boilerplate, e.g., by defining handy update functions. One could cook up something doggedly interesting using Cats and StateT - as long as you provide a function of the kind (I, S) => (Option[O], S), the sky is the limit.

Thanks to Jaakko Pallari (@jkpl) for previewing this.

This is actually false, as Aaron Turon, a core Rust developer, proves in his article about getting lock-free structures without garbage collection. ↩

Focus

Antoine Kalmbach — Fri, 25 Mar 2016 00:00:00 +0000

Focus is a design element in programming languages that I think deserves more attention than it gets.

A focused language puts emphasis on a set of coherent idioms. Multi-paradigm languages like C++ or C# are unfocused because they lack a certain principle.

Take C, for instance. You can do OOP in C, but it’s awkward. You need structures full of function pointers and the language wasn’t designed for it: it’s not a good idea to do it. The point is that you can but you shouldn’t.

Focus is not so much of what a language has but something that it embodies. A single-paradigm language can still be unfocused, because there can be several ways to wield the singular paradigm. A multi-paradigm language can be focused if the multi-paradigm languages connect at a higher level. Focus can be implemented as a coding standard or it can be something that everybody understands as the idiomatic way of doing things.

Focus is not always a positive trait but it rarely is a negative trait. On the other hand, a lack of focus is more often negative than positive.

Take Haskell, a pure functional language, effectively single-paradigm; it is a very special case. The language itself is absolutely focused to the point of extreme autism, but its flexible type system and vibrant community, there are many ways to program Haskell. Do you absolutely need state? Use state, but be careful. Do you want IO without monads? Well, sure, but be careful.

At a high level, Haskell code is pure. It permits some inconsistencies with its principal paradigm but it eschews them and this is the key difference.

A bigger problem with focus is that it often is intangible. It’s easier to point out languages that are unfocused than those that are. Focus is about philosophy. Some language are very philosophical. For instance, Clojure is just as much about its particular approach to concurrency, state and identity, that is a language implementing those ideas. The language caught on because Rich Hickey, the author, did not market it as the tool that would have solved everybody’s problems, but because he marketed the ideas that Clojure represented as a solution to common programming problems.

“If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.”

Antoine de Saint-Exupéry

In this context, Clojure can be seen as a focused language. These core philosophies are what constitute the language, the fact that Clojure happens to be a Lisp dialect implementing the philosophies is secondary in my mind. With that in mind, I acknowledge being a Lisp is also a core part of Clojure, but its principles about state and identity can be implemented in any language. Clojure does let you do OOP but it feels awkward. When you grok Clojure you understand what that means: the language can be bent for that purpose, but it doesn’t want to be. Its philosophy is like a memory-foam, if you tamper with it, it will coalesce back into its original form. When you see that, it’s the moment you understand what Clojure—or any other language—is about.

Some languages double down on philosophy by making it a part of a coding standard and enforcing it: Go. Go embodies simplicity and intuition, intentionally eschewing things that are not modern, but complex, opting to keep the core language simple. Some chalk this down as a negative trait, others love it; I find it to be both good and bad. Good, because I can jump into any Go codebase and get its purpose in minutes; bad, because sometimes I want to use abstractions for which Go is unsuitable. I respect its design philosophy, because it has one, and absolutely flaunts it. It’s not just a structural type system, it’s an idea.

Scala is another beast. It began as an experiment, trying to augment and fix the deficiencies of Java. It was designed by brilliant PLT theorists and the language is a beautiful and magnanimous behemoth. Scala has so many features that it eschews focus either intentionally or unintentionally. On the other hand, Scala is capable of many great things. But if you ask two Scala programmers what Scala represents to them, you may get different answers.

It can be a technical aspect. To some, it might be about Shapeless or all the cool things that go with it. Macros. DSLs. Code generation. Or it could be how Akka or Spark are amazing tools. It could also be a philosophical difference. Some people want an advanced type system and don’t want to be constrained by the laziness and purity of Haskell. Others want the JVM. Some just want a better Java. Some just happen to use it for Spark.

I would choose the simpler Scala, the better Java. Trait-based generics, sum types, implicits, and functional programming, to name a few. This is not just because it’s less complicated, from a business perspective, it makes it easier to hire new programmers.

As a professional Scala developer and a long-time functional programming enthusiast, I fear that I may never comfortably jump to another company, confident that since I’ve written Scala, I can understand their Scala. That, or years of experience, but who knows what’s enough? Their, whoever they may be, Scala might not be the simple Scala I and my colleagues prefer.

This is scary. For the future of the language, this is an untenable position. While I absolutely enjoy working with the language, I’m afraid that it is fated to be like Macbeth from Shakespeare: “thou shalt get kings, though thou be none”. Thus, Scala will inspire a great language and then die. Maybe it already did, and the clock is ticking. Some purport Kotlin as the successor, but I wouldn’t bet on it just yet.

“Ah, but a man’s reach should exceed his grasp, or what’s a heaven for?”

Robert Browning

The thing about Scala is that this is a conscious design decision. The language is meant to have everything and the kitchen sink. Programming languages don’t have to be simple. Powerful languages are powerful tools. Use them well, you can achieve greatness. You have to choose your tool set and hone it.

But for Haskell, Go, and Clojure, you’re using them, and you’re thinking, what is the natural way to do this? Once you find it, you find yourself implementing ideas using that philosophy, that natural way, and you’re no longer just using a tool. You’re using an idea.

Imprecision and abstraction

Antoine Kalmbach — Thu, 17 Mar 2016 00:00:00 +0000

What is the point of abstractions?

We want to hide things. We want to generalize things. We want to extend things.

Why are mathematical abstractions so intractable? Why is the Wikipedia page on functors incomprehensible to someone not used to mathematical formalisms? Why does it sound so vague?

When approaching abstractions, for educational purposes, it is sometimes easier to think of analogies or similes. We can conceptualize the idea of functors of “procedures” that operate on things inside “boxes”, or we can study relational algebra using Venn diagrams.

These analogies are dangerous, because they are vague. Formalisms leave no room for interpretation because they are exact not whimsically, but because of the pervasive imprecision of the human mind.

Let’s take an analogy that’s very approachable but quite dangerous. Explaining modular arithmetic can done with clocks. The analogy would go like this:

You see, when you have a number modulo 12, think of it as a clock. If x is over 12, think of like like the hand of a clock looping over, and you’re back where you started.

The problem with such an analogy is that not everybody uses 12-hour clocks. Most of Europe uses a 24-hour clock with no distinction between AM and PM. Of course, they are also taught to understand that “sixteen” means “four” since nobody builds 24-hour analog clocks (yet). That aside, it’s still very possible, that when explaining the above analogy to someone accustomed to 24-hour clocks, they’ll get confused since what comes after 12, is 13, not 0.

This is a basic but fundamental example: things like functors, semigroups, monads, and categories, are a bit intractable for a reason: there’s no room left for interpretation.

Mathematical formalisms pare fundamental ideas into pure forms. Your intuition can’t get in the way and corrupt them.

The obvious downside is that these formalisms are harder to understand. I wager that this is for the better because down the road there are concepts so high-level one can’t even begin to think in analogies, and it will only slow one down.

There was a turning point in my math studies when I stopped trying to grok things using analogies. My approach to topology was geometrical. I tried to visualize limit points in my mind and in vain, because the mind can’t bend itself around more than three spatial dimensions. Granted, visualizing hypercubes was possible (“like a cube is made of sides, a hypercube is made of cubes”)… kind of.

Stopping this perilous habit, I started to memorize laws instead. That changed the language of maths for me, forever. I wasn’t understanding relations via shapes or arrows, but by basic axioms and mathematical laws. It wasn’t too long before I started to visualize concepts using these laws.

I stopped staring concepts in the eye, looking for hidden meanings behind bounded sets. I simply read the definition, thought “huh”, and memorized it. Slowly, by building towards other related concepts and set theory I quickly understood what the law meant, without trying to grok the hidden meaning.

Once that became a habit, it became easy and changed my approach forever: let go of intuition. Abstract ideas are hard by definition and they need to be understood piece by piece, from the ground up.

This is why any explanation of a precise thought, like a mathematical formalism, using something imprecise like an analogy, is a fallacy doomed to fail.

When functional programmers try to explain basic ideas like semigroups or functors, they often find themselves in an apologetic mire of simplifications. This is doomed to fail. Give concrete examples of how a functor works. Don’t illustrate them as operations that can map stuff in boxes to other boxes. Invent a problem and solve it using a functor. After all, they’re such a basic concept, even those writing non-functional code end up doing them all the time.

Let the abstraction sink in, it’s the only thing that will survive.

Are my services talking to each other?

Antoine Kalmbach — Tue, 26 Jan 2016 00:00:00 +0000

I am faced with an interesting thought experiment, which asks:

If I can see two of my friends, and I know they should be communicating to each other, what is the simplest way of making sure they are doing so?

Your first instinct is to look at them and listen. What if the communication method is subtler than that? What if you are, metaphorically speaking, deaf, and cannot eavesdrop on their conversation?

A problem like arises when you have a non-trivial amount of distributed components talking to each other, forming a complex network. Let’s start from the basics and consider a simple one:

arrows indicate flows of information, i.e. x → y means x sends information to y

You could assume A is an event log, for example, of financial transactions; B is a message queue and C is a fast queryable cache for the transactions. We want to be able to query the cache quickly for log events and rely on the message queue of transporting them from A to C, while preferably not having a hard software dependency from A to C.

The illusion is that while there are neither code nor protocol dependencies between A and C, a semantic dependency exists: the one in our heads! A is content on dumping information towards B, but what we’re really interested in is messages getting through all the way to C. So in reality, if we superimpose our perceived dependencies on top of information flows, we end up with this:

Tolerating faults

What if the chain breaks? What happens when A can’t push messages onward to B, and we get a blackout? Who gets notified? C doesn’t know what’s happening in A, it’s just not getting information! In line of the original question, if I can see both A and C are doing fine, but they’re not talking to each other, where is or who is the broken phone?

With such a simple case as above, pointing this out is easy, so let’s make our network a bit more complicated.

A - an event log; B - a message queue; C - a cache; E - app back-end; P - a user-facing application; I - a business intelligence system; S - a storage system

Let’s assume each one of these components is an independent service, each load balanced and with redundancies that aren’t visible beyond the node itself¹, and that communication is done over a computer network using some protocol.

The depicted network consists of a set of applications that all in one way or the other build on top of an event log, A. In one branch, there’s a fast queryable cache for the transaction log, the app back-end is an interface for the cache (like a REST API), and the storage acts as a long-term backup system. The second branch consists of a business intelligence system that analyzes the event log data and does something with it.

Indirectly, there are dependency arrows emanating from the root of the network tree (A) to its leaves S, P and I. From an observer’s perspective, these are the relationships that matter. These are the implicit dependencies. Furthermore, we can see those dependencies, but we build the code in such a way that it does not! The event log simply dumps data to a message queue, and that’s it. What is worse, is that the implicit dependencies each propagate up the chain. Not only does the leaf node depend on the root node, it also depends on the intermediate nodes.

Implicit dependencies

The inherent hazard in all this, of course, is that there’s a communication error. Even though we (hopefully) built the system following the robustness principle, data isn’t flowing from the root node to the leaf nodes and we have to quickly identify where the disconnect happened.

Seeing is not enough

Our first instinct is to peer at the logs. So we go through each edge in the network and see if there’s a fault. This means for n nodes looking at least at n-1 edges for each fault! Moreover, the problem isn’t fixed by using something that gives me visibility of the nodes, like ZooKeeper or other service discovery tools. This is because I am interested in the flow of information from one node to another. The thought experiment already assumes that the nodes are there, only the communication between them is broken.

In the Internet world, with the Transmission Control Protocol , communication is made reliable using error-checking and acknowledgments. That means, if A were a network element and wanted to send things over to C, in case of a successful delivery C will acknowledge this back to A.

For various reasons, it may be that in a distributed service network this approach is not feasible. This is the cost of abstractions: when you enforce loose coupling, you have to deal with the consequences of looseness. We could build the transaction log aware of the user-facing Application but that may be overkill.

For the particular problem of acknowledging from a message queue root to a consumer leaf, there are various solutions. You either implement this on your own, which while laborious, essentially follows the principle of error-checking. The caveat is this grows in complexity with every new node. Another option is to use a message queue (one of these things is not like the others) that supports this natively.

The rescue signal

We could build a centralized logging system to which each node logs its events. This centralized system contains all events from all nodes. To make the data meaningful, you need to construct a way to determine the flow of information, that is, grouping events together semantically. Worse, the system will require manual or semi-automated inspection to determine when any event is missing its acknowledgment, that is, A logged an event of sending Foo to message queue but the user application back-end E never processed it.

A system like this could work using a FRP approach: since FRP signals map exactly to discrete events, one could build a rule engine. By integrating time flow and compositional events, a centralized system could use its rule engine to listen to signals. A signal can be any event, e.g., a financial transaction that was logged into the event log. You can combine this signal with another event in a system that consumes transactions and does something with them, like the business intelligence system. The sum of these two signals imply that “a financial transaction was consumed by the business intelligence system”. This is also a signal!

Building a FRP-based rule engine isn’t easy, you’d need to construct a rule engine that can map diverse data events into high-level signals and then create additional logic for summing the signals.

The sum of two signals is another signal. (Oh hey, this makes it a semigroup!)

Once such a system is built, it can be queried to determine the state of the network quite efficiently (and perhaps elegantly), but it does not introduce any fault tolerance and will only tell you where data is moving, but not where it isn’t.

Lurking in the shadows

I guess that most of this stuff underlines the difficulties of unraveling a monolith into a microservice. Keeping track of network traffic is really hard, even at the hardware level (!), so when we push this abstraction to the software level, it is not a surprise that this can cause problems.

Playing with some toy solutions I thought of something I call a shadow network. Let’s say our principal information source is an event monitor X and we have a leaf node in the information dependency tree that is interested in data originating from X.

Each leaf node sends its data to the shadow node. The shadow node understands the data and can tell where it originated from, thereby seeing the implicit dependencies. The shadow node is effectively a mirror of the root node(s).

In the shadow network, X does not receive any new dependencies nor do the intermediaries, but the leaf nodes each push their actions to the shadow node. The shadow node contains a rule engine that can parse leaf events. A rule is something that identifies a source. It could be anything, from a simple parser (“this looks like Apache logs” → “it came from Apache!”) to something more sophisticated. This introduces a dependency only to leaf nodes, but the problem is that the shadow node has to be kept up to date on how to correctly map events to sources. When you change the format of the data traveling across the network, you have to update the rule engine.

Unfortunately, this doesn’t really help us: you can query the shadow node to get the implied dependencies, but that’s it. So while it requires less effort to develop, disregarding cases where creating rules causes difficulties, it suffers from the same flaw than the centralized FRP engine: it can only tell when data is flowing but not when it isn’t.

No easy answers

This makes both solutions rather untenable for monitoring a microservice architecture, but they can be used in cases where the service network grows large and you are working with opaque layers, that is, you don’t know what’s between the leaves and the root, and you want to construct the implicit dependency graph.

Bolting temporal awareness in the shadow network works if the data is supposed to be regular. If the consuming leaf expects a tick from the origin(s) every n seconds, the shadow rule engine can be built to be aware of this. If ticks aren’t happening when they are supposed to, you can create a fault on the implicit dependency. Alas, only regularly occurring data works here, so we’re out of luck for irregular events.

Either way, the original problem is an interesting one. I suppose the only reliable way of doing things is to do what the Internet Protocol does: acknowledgment and error checking. While certainly a lot of work, it will be reliable. We all love reinventing wheels, don’t we?

My opinion? Don’t fix what isn’t broken! While we all benefit from loose coupling, and while microservices definitely are most of the time an improvement over monoliths, both bring hurdles and challenges of their own. The bottom line is that networking is not easy, and if one forgets this, problems will occur.

So for all intents and purposes the nodes represent services as a whole instead of individual physical units, whatever they may be. ↩

The expression problem as a litmus test

Antoine Kalmbach — Fri, 08 Jan 2016 00:00:00 +0000

The expression problem is a famous problem in programming languages.

“The Expression Problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).”

Using interfaces (like in Java) as the datatype example, the problem simply asks whether it is possible to derive the interface and add new methods to the interface, without having to recompile existing code or to resort to using casts.

Obviously, in a OOP language it’s easy to derive interfaces, but the problem uncovers the rigidity of the type system: you can’t modify (i.e. extend) the interface, because you have to modify all the classes classes that implement the interface.

Conversely, in functional programming languages, adding new methods operating on the interface is easy. Consider the canonical OCaml example:

type shape = Circle of float | Rectangle of float * float

let area shp = match shp with
    Circle radius -> 3.14159 *. radius *. radius
  | Rectangle (width, height) -> width *. height
  
let vertices shp = match shp with 
    Circle _ -> infinity
  | Rectangle (_, _) -> 4

So in FP, you could create a function called volume that computes the volume for the existing types, and you needn’t touch the above code. However, as soon as you do that, you realize you’ve made a silly mistake: our shapes are flat so their volume is zero. Quickly, you realize you need a three-dimensional Cube shape.

let volume shp = match shp with
    Circle _ -> 0.
  | Rectangle _ -> 0.
  | Cube s   -> a *. a *. a        (* Cube isn't defined *)

Oops!

Here’s the onion: to get the Cube working, you’ll have to modify the existing code in two places: the definition of shape and both methods area and vertices.

In OOP, this isn’t the case, since you can just derive the new IShape interface and be done with it, but the problem arises when you’re adding the volume function, because you need to modify IShape, and thus every class that derives it.

In FP, adding new functions over the datatypes is easy, but adding new cases to the datatype is tricky, because you have to modify existing functions. In OOP, adding new functions over the datatype is hard, because you need to modify each implementation; adding new cases to the datatype is easy, since all you need to do is derive the interface. FP has invented a multitude of ways to deal with this problem, ranging from type classes, traits to protocols; OOP usually solves with either patterns or open classes. Ruby’s refinements can be used for this purpose as well.

Polymorphic variants

That’s a quick introduction to the problem. I think the expression problem is a perfect litmus test of sorts for programming languages, that is, the measure of the expressive power of the language is the quality of the solutions the language presents to the expression problem.

The expression problem is theoretically solvable in any language, but to varying degrees of elegance. In Java one must resort to using the visitor pattern, and in my mind this is the most inelegant way of going about it. I would rate the solutions on a spectrum: with the most basic solution being the visitor pattern, at the other end we have something like polymorphic variants and type classes. Multimethods and protocols are somewhere in between.

When you compare polymorphic variants of OCaml with Haskell’s type classes, there’s a marked difference in brevity. Polymorphic variants are succincter than type classes but cannot provide the same level of type safety.

type shape = [ `Circle of float | `Rectangle of float * float ]

let area shp = match shp with
    `Circle radius -> radius *. radius *. 3.14159
  | `Rectangle (w, h) -> w *. h

let vertices shp = match shp with
    `Circle radius -> infinity
  | `Rectangle (w, h) -> 4.

Not too different from the above declaration, the type is surrounded with brackets and the types are preceded with backticks. Recreating the volume function is easy.

let volume shp = match shp with
    `Circle _ -> 0.
  | `Rectangle _ -> 0.
  | `Cube a -> a *. a *. a

So now I’ve extended the shape type with another type Cube, and I haven’t touched vertices and area functions. The volume function can be done even more succinctly:

let short_volume shp = match shp with
    (* no volume in two dimensions! *)
    #shape -> 0.
  | `Cube a -> a *. a *. a

It is also possible to constrain the polymorphic variants:

let flatten shp = match shp with
    #shape as x -> x
  | `Cube a -> `Rectangle a

The type of this function is [ < `Circle of float | `Cube of float | `Rectangle of float * float ] -> [> shape]. The [< A | B] means a closed type: it can be only A or B, but nothing else, and [> Foo] means “Foo or something else”. So the flatten function accepts Circle, Rectangle or Cube and returns a shape (or possibly something else). Trying to run flatten (`Sphere 4) produces a type error:

# flatten (`Sphere 3);;
Characters 8-19:
  flatten (`Sphere 3);;
          ^^^^^^^^^^^
Error: This expression has type [> `Sphere of int ]
       but an expression was expected of type
         [< `Circle of float
          | `Cube of float * float
          | `Rectangle of float * float ]
       The second variant type does not allow tag(s) `Sphere

However, the following code compiles:

type polytope = [ shape | `Cube | `Octahedron ]

let frobnicate pt =
  let flattened = flatten pt in
  match flattened with
    #shape -> "Already flaaat!"
  | `Octagon -> "Eight coorneeeeerss"

The compiles, although we didn’t tell the compiler that flatten does not return Octagon. There are two ways to fix this: either explicitly annotate pt to be of type polytope, which produces this error:

Error: This expression has type polytope
       but an expression was expected of type
         [< `Circle of float | `Cube of float | `Rectangle of float * float ]
       The second variant type does not allow tag(s) `Octahedron

It is possible to further constrain the type with type annotations. We can make sure that the flatten function returns only flat shapes:

let safe_flatten shp : [< shape] = match shp with
    #shape as x -> x
  | `Cube a -> `Rectangle a
  | `Sphere r -> `Circle r

This produces the error:

Error: This pattern matches values of type [? `Octagon ]
       but a pattern was expected which matches values of type shape
       The second variant type does not allow tag(s) `Octagon

Not a silver bullet

Unfortunately, polymorphic variants are problematic. The problem with polymorphic variants is you quickly reach an absurd level of complexity and are forced to use annotations or subtyping to ensure maximal type safety. So although polymorphic variants are nice, and they do let us solve the expression problem, they’re an unsteady compromise between type safety and brevity. You can certainly make elegant abstractions with them but they get unwieldy quickly. They aren’t as efficient compared to regular variants either.

So what are the options? In OCaml 4.02, you can use extensible variant types:

type boring_shape = ..
type boring_shape += Circle of float | Square of float
                                                   
let boring_area shp = match shp with
    Circle r -> r *. r *. 3.14159
  | Square a -> a *. a
  | _ -> 0.

type boring_shape += Rectangle of float * float
let radical_area shp = match shp with
    Circle _ as c -> boring_area c
  | Square _ as s -> boring_area s
  | Rectangle (w, h) -> w *. h
  | _ -> 0.

An extensible variant is defined using .., and extension is done with the += operator. The caveat is that you must handle the default _ case in pattern matching. Extensible variants are another neat trick for solving the expression problem.

A measure of expressive power

The expression problem is a great litmus test that measures the expressive power of a programming language. The actual measurement of the test can be either the brevity of the code or its type safety. The solutions range from the clumsy Visitor Pattern in Java to polymorphic and extensible variants in OCaml and to type classes in Haskell. Clojure and Elixir have protocols that are both quite nice but not so type-safe since both are dynamically typed languages. What is more, since the expression problem is also about type safety, then strictly speaking the problem isn’t valid in a dynamic language. Any Lisper knows that Lisps are super expressive anyway.

Before we begin

Antoine Kalmbach — Fri, 01 Jan 2016 00:00:00 +0000

Now that the design of the site is finished, I can finally focus on the essentials.

I’ve decided that this year I will be writing a bit more, here, and elsewhere. To that end, when it comes to this site, I’ve had to perform a simple but challenging task: lowering my standards.

Last year, I did not publish anything because I had absurd standards for content. In my mind, every blog post had to be a thoroughly researched and carefully argued piece, capable of standing the test of time.

This was a monumental mistake.

Researching something thoroughly requires an extraordinary amount of time, and writing opinionated articles that can stand the test of time requires an extraordinary amount of foresight — neither of which I yet have. I have already managed to delete one post which contained opinions I no longer agreed with. I thought its content was rubbish and I was a moron for publishing it, so my only recourse was to delete it, instead of learning from it.

My desire for more content stems from the process of gradual improvement: the more you write the better you get. I cannot do this unless I start from the very basics. I could, of course, simply write guides on how to do XYZ with $THING, but I want to tell stories, not write recipes. This doesn’t mean there won’t be any guides, however!

Another reason for wanting to write more stems from the simple funny fact that somebody reads this. Besides Google Analytics telling me so, a few weeks ago I even received a pull request for typo fixes.

So I know there’s at least one guy who actually reads every word. Disclosure: according to GA, most of the visits are “accidental redirects” (huh?), and that actual, longer visits aren’t common, but there were enough for me to extrapolate that there were, at the very least, two readers.

So, that was the background. And now comes the disclaimer, of sorts.

This site is a blog. As this site is a blog, the writings are, first and foremost, opinion pieces, not research articles, and opinions change. You may find me advocating for stricter type systems one day, and for looser the next. I will only guarantee that, at the time of writing, I will argue my points to the best of my abilities.

Longer, more in-depth stories, if ever completed, will be available under the articles category. This is to mark a distinction. To qualify as an article, the writing will a) contain properly researched writing and references b) be reviewed by somebody else. I’m currently working on something that may one day be considered something like that.

You’re welcome to snoop into the drafts folder of the GitHub repository of the site, but be warned, that stuff is obviously incomplete.

To be continued.

Start-once software

Antoine Kalmbach — Mon, 07 Dec 2015 12:00:00 +0000

Software development tools are in a state of flux. There are two competing directions towards which static analysis tools—like linters and type checkers—are heading.

The traditional direction is to operate in a batch model. Fire up, perform analysis, report results, and die. This is a proven method. Batch-oriented software has been around for ages, and it works really well if the data you’re working with isn’t large.

The new direction is an online model: an analysis tool starts, calculates its data, reports results, and then stays on, monitoring for changes. When changes occur, the program analyzes the changes, and recomputes the effects. This execution model is efficient: incremental updates are easier to calculate than starting from scratch. This approach is arguably more modern¹, since we’re leveraging the full potential of a client-server model.

The obvious caveat is such a process will eat up memory. These days this is becoming less and less of a problem, since a gigabyte of desktop-quality DDR3 memory costs about US$5.

However, if the workload is large, say 100k-1M lines, it actually makes more sense to compute an initial “model” of the system, and then incrementally update the model when the system changes.

My line of reasoning was inspired by this comment on Hacker News:

Over and over I see reports of iteration speed being critical to real-world projects, and that’s certainly my experience.

But it’s something I rarely see featured in tool design. So many of our tools are still batch oriented: they start from scratch, read everything, process everything, and then exit. That made sense in an era where RAM and CPU were expensive. But now that they’re cheap, I want tools that I start when I arrive in the morning and kill when I go home. In between, they should work hard to make my life easy.

If I’m going to use a type checker, I want it to load the AST once, continuously monitor for changes, and be optimized for incremental changes. Ditto compilers, syntax checkers, linters, test runners, debugging tools. Everything that matters to me in my core feedback loop of writing a bit of code and seeing that it works.

For 4% of my annual salary, I can get a machine with 128GB of RAM and 8 3.1 GHz cores. Please god somebody build tools that take advantage of that.

Arguably, the batch mode made sense in the past, when resources weren’t as plentiful as they are now. Now, we can afford a type checker that sits in the background, eating up half a gigabyte memory, and most of us won’t even blink at that, as long as it helps us write better code—and runs quickly.

To this end, one could call such software start-once software², which

Is launched once, via some mechanism and given a target, e.g. a codebase, to monitor for changes
Computes an initial model, the analysis, of the target
Whenever changes occur it instantly recomputes a new model and informs the user of the effects of the changes.

In this case, the target could be a collection of source files (a “project”), the model the compilation errors present in the source files, and the relevance is a way of communicating those errors to the user.

The basic idea can be taken from Facebook’s Flow which does exactly this when you launch it initially, it boots up a server that keeps the state of the AST in memory. The server runs the analysis and the client reports the results. There’s an initial up-front cost to all of this from starting the server but subsequent calls are fast.

There are some pseudo-implementations of this paradigm: you can have a batch process that is run every time the target changes, and then do effectively what is done in step three above. This is kind of what I described above, but the difference is that any model is lost inbetween batch runs.

In fact, file system notification based batch processes remind³ me of the the Chinese room experiment: such tools don’t really have an understanding of the model that is persistent, but due to a simultaneously crude and brilliant approach, we get the subtle impression that such a persistent, evolving understanding actually exists.

In start-once software, the goal is to always keep the model in working memory and update it incrementally. Such a program can react quickly to massive codebase changes. Naturally, speed comes at the cost of memory, but as mentioned in the quote, if it makes development faster, I think this is a perfectly justifiable cost.

This line of thinking doesn’t apply only to static analysis tools. It can work with any streaming data. Memory is cheap, but speed always isn’t. A good example is a log processor that computes the state of some system based on the content of some logs. A start-once processor would continuously monitor the log inputs and update its model of the system. If it has to churn through a lot of logs at the start, it may have an initial delay but because the model is persistent, any changes to the model can be computed quickly.

Storing the model can be done in two ways. If RAM becomes a limitation (will it?), a fast database should be used. AFter all, databases solely exist because of the prohibitively high cost of ephemeral memory compared to the cost of non-volatile memory. Traditionally!

Previously, because of the cost of ephemeral memory, we had to invent a cheaper form of storage. Now that memory is cheap, this isn’t so much of a threat anymore.

When it comes to fault-tolerance, precautions can be taken. The model could be stored as a persistent image—yes, raw memory or an accurate representation thereof, or a database. Once the program recovers, it restores the model immediately, reducing the boot time.

This model also be extended to web servers: instead of recompiling everything at every request (hello PHP), one could compile once and then compile changes incrementally. This idea isn’t new: hot swapping has been around for decades, and its advantages are obvious. Heavy-duty services that take a while to reload and restore benefit massively of hot swapping. This is routine in the JVM, Erlang and Lisp worlds.

Instead of shutting down your engine to upgrade it, you simply replace the parts to be upgraded with new ones.

Extending this to static analysis tools isn’t a massive step. If the philosophy works with server programs I see no reason why it couldn’t work with data analysis tools. At the cost of ephemeral resources like memory, I want static analysis tools that can handle large codebases and compute any change big or small in a fraction of a section.

I hope more tools will adopt this model. We certainly do have the resources now.

Addendum: examples of start-once static analysis tools

Flow

A static analysis tool for JavaScript by Facebook, written in OCaml. Upon launch, it starts a server that runs initial analysis and monitors for changes.

Hack

A PHP-inspired gradually typed programming language for the HHVM virtual machine. It runs a type checker in the background in the same way as Flow does. Also in OCaml!

gocode

Auto-completion service for Go. It uses a client-server model where completion requests are sent to the completion server via a JSON-based RPC. Conversely, every other Go-related static analysis tool (there are lots) is using the batch approach. That’s fine, since Go seems to be vehemently opposed to anything modern and advanced, and sometimes justifiably so. In this regard Gocode is a rare gem among the Go tooling ecosystem.

Yes, I know REPLs have been around since the eighties, but this isn’t exactly the same thing. ↩
Facebook uses the term “online” when describing Flow ↩
They aren’t exactly the same thing. Since I’m talking about computer programs performing some kind of data analysis, and mentioning the Chinese room experiment, I realize that I’m opening a massive can of worms; the comparison is a mere simile here. The users of continuous (i.e. inotify-based) batch analysis tools are on the observing end of the Chinese room experiment. ↩

Antoine Kalmbach's website

Guile fun times, and blog update

Recutils, GOOPS and virtual slots

if it’s possible to put Lisp in it, someone will

the smell of raw pointers

why not dynamic FFI?

GOOPS, virtual slots, and you

context is everything, friends

A Guile test runner with an exit code

Working with Git and patches in Emacs

Why Git and email?

Development forums are essential

Email? Seriously?

There is no standard pull request any more

Barriers of entry

Git and email in 5 minutes

Sending patches from Emacs

Calling git send-email from Emacs

Setting Emacs as $EDITOR

Applying email patches in Emacs

Making our own M-x git-am

Conclusion

Divergence: A website is born

The overhead of test-driven development

Iterate, iterate, iterate

First, you must create the universe…

How this site works

Origin of this site

Design of this site

No more microblog

Between two Lisps

Tastes differ

Productivity matters

When would I pick one over the other?

What about Clojure?

Final words: Emacs Lisp

Back to Rmail

Inside the machine

Examples: Emacs, StumpWM, Nyxt

Blazing fast development cycles

Winding down

Embedded Rust and WiFi

Useless interfaces

Implicit power

Apache Camel and the price of abstractions

Half stack web frameworks

What exactly is wrong with the tooling ecosystem?

Progress, progress, progress!

Not about killing innovation

SPA development is more than just tools

In summary

1. Lack of emphasis on usability, a myopic focus on adding features.

2. Chasing novelty with little care about its impact on maintainability.

3. Snubbing full-stack frameworks for their want novelty, although they generally feature exemplary usability

A new direction: renovate, not rewrite

What?! Your answer is Rails? In 2016?

A REST-backed Rails app with React as the templating engine

Compromises made

A functioning example

Conclusion

Web development has become weird

How it was

Where it all began

So?

A concrete solution

Communicators: Actors with purely functional state

Enter Akka FSMs

Actor, meet communicator

Decoupling Akka

Focus

Imprecision and abstraction

Are my services talking to each other?

Tolerating faults

Seeing is not enough

The rescue signal

Lurking in the shadows

No easy answers

The expression problem as a litmus test

Polymorphic variants

Not a silver bullet

Calling `git send-email` from Emacs

Setting Emacs as `$EDITOR`

Making our own `M-x git-am`