Engineering Blog

Node.js & npm best practices

2023-11-15T00:00:00+01:00

Perhaps you would be surprised to learn that here at Devo we were early adopters of Node.js.

Two of our core propositions and unique features from day one are raw volume of data, and speed in retrieval and processing. Customers know that well: blazingly fast queries over huge data lakes is one of the features that set us apart from our competitors. At the same time, the JavaScript ecosystem is not widely praised for its performance — admittedly, there are languages that have been much “closer to the metal” for ages. Those other platforms are sometimes easier to fine-tune, and organisations with stringent performance requirements have been squeezing every last byte and millisecond out of them for a long time now.

However, we have been defending JS as a language, and Node.js as an environment, from the early days of the company. And we are still doing so!

Turns out that Node.js and npm can form the basis for a friendly toolkit, and also be extensible and support efficient software.

In this post we share some of the realisations and tips that we have discovered and adopted along the way.

Local config files are your friends

We rely extensively on .npmrc and .nvmrc; either locally (sitting at $HOME in each of our workstations), or shared with all other contributors and kept under version control.

Depending on the nature of your project (proprietary or open source, on an internal repository or public on GitHub) you will want your npm dependencies to be fetched from one source or another — and the npm packages you produce to be published to the right place. In fact, the consequences of using one endpoint instead of another might be dire (imagine publishing your precious npm package @corp/internal-trading-strategy to npmjs.com by mistake). Here’s where an .npmrc file comes handy, as it lets you point to a specific set of npm repositories, overriding global configuration, and even temporarily store your authentication credentials for those services.

With regards to nvm, suffice to say that we maintain dozens of npm-based projects with a wide range of requirements. At any given point in time our engineers are working with several different versions of Node.js and npm, so making sure that they always use the right one for each project saves time and prevents mysterious build errors.

Don’t neglect pipelines and know when things are broken

What developer doesn’t have the temptation to mute or simply ignore occasional failures of their Continuous Integration or Continuous Deployment builds? Sometimes overwhelmed by day-to-day work and more pressing issues, it’s difficult to pay attention to each and every failed build — not to mention to humble warnings.

However, we have found that striving to tend to those notifications pays in the long run. Our front-end teams do a deliberate effort to optimise the number of automated checks and to fine-tune the thresholds we set so that there are few false positives and errors are tackled.

It is not that time-consuming to spend ten minutes, every now and then, turning a few knobs and toggling some checkboxes. Let’s see an example.

A pipeline job may be temporarily broken for “good reasons”; those include transient misalignments between projects or dependencies, ongoing work on related pieces of software, and downtime from third-party services (like when a linter or a scanner is under maintenance). Most CI/CD tools provide syntax to keep on running a job that we accept may fail, so that the result of the whole process isn’t affected by that. See eg GitLab’s CI allow_failure keyword.

Preventing false positives in that way increases signal-to-noise ratio and contributes to making actual problems more salient and therefore harder to ignore.

Don’t be afraid to experiment

As you can see, npm and Node.js are at the root of our work with JavaScript.

In the last years we have also explored a few other platforms and dependency managers. Among those, Deno, Yarn or pnpm. Recently, we had been excited about the promises (no pun intended) made by Bun, and the possibility that it alone could replace (and improve) a handful of items in our current tool chain.

Typically at Devo, some colleague conducts early experiments with a pet project of theirs or with some non-critical component. If their experience is positive, those results are usually shared with the front-end architecture chapter, and in turn publicised or even recommended to all other coders.

Other excellent tools that we introduced following that playbook are StrykerJS for mutation testing, release-it to streamline versioning and changelog editions, or Vite for building. We switched from Lerna to native monorepos with npm workspaces. And of course, we have moved more and more of our codebase to TypeScript following the strategy outlined above.

…but don’t chase every bright shiny object

Memes abound about the insane cycle of hype in JS-land. We all know there is a kernel of truth to those funny images and blog posts: JavaScript is a tremendously popular programming language (somewhere between #1 and #6) which makes it very easy for anyone to develop and share anything from a tiny library to a gargantuan framework. It’s sometimes too easy to get excited about so many new techniques and thingies… most of which will be unmaintained in a year’s time.

At Devo, we try to strike a healthy balance between innovation and caution. Having a diverse enough team of programmers helps here, because a wide range of ages, backgrounds, interests and skill sets make for a better debate. All the tools mentioned in the previous paragraph were the result of lively conversations among engineers and of practical experimentation — not top-down impositions or the whim of anyone in particular.

👉 If you or your team are suffering from “JavaScript fatigue”, Axel Rauschmayer has some useful tips, such as “wait for the critical mass” and “don’t use more than 1–2 new technologies per project”.

Streamline maintenance tasks

With large enough codebases, keeping things tidy, clean and updated is a task in itself, something that could keep one person busy full time. JavaScript projects are no exception: if unmaintained, they rot and decay quicker than you can say “unsupported engine” or “critical vulnerability”.

For the purpose of this discussion, let’s break down “routine maintenance” into three different types of activities:

Creating stuff.
Examples: generate an artefact, publish a package, release a version.
Changing stuff.
Examples: update dependencies, fix broken links, amend documentation.
Deleting stuff.
Examples: remove unused dependencies, prune code that can’t be reached, ditch config files that aren’t useful any more.

The first type of maintenance is usually achieved through test suites, documentation comments, programmatic generation of docs, and CI/CD. In an ideal world, no human should ever have to take care of that directly: a new commit being pushed, or the clock striking 3:00 AM, should trigger all changes necessary for the latest version to be linted, tested, built, packaged and deployed automagically.

What fewer JS programmers know is that many of the maintenance tasks in the second category (“changing stuff”) can be automated, too.

We are usually reluctant to let tools update npm dependencies on our behalf, and for good reasons: the most innocent-looking change in our dependency tree could break everything, and in principle there is no way to know. So, how could we entrust that delicate task to an unattended process? Enter packages npm-check-updates (in “doctor mode”) and updtr. These two take care of upgrading all dependencies that can be upgraded, within the semver range specified in package.json, and (most importantly) are able to run an arbitrary npm script and use the result to decide whether each individual upgrade is feasible or not.

For instance:

npx updtr --test "npm t && ./checks.sh && npm run whatever-you-usually-do"

Voilà! Make that part or your scheduled pipeline, and see your dependencies always fresh (assuming that your test suites and your checks are good enough, that is!).

👉 We mentioned broken links above. For that, you could integrate the W3C’s Link Checker into your pipeline.

What about the third type of maintenance, “deleting stuff”? For unused code, there’s code coverage, and most teams use those metrics. For unused npm dependencies, something like npm-check comes to the rescue:

npx npm-check | grep -i 'notused?' | rev | cut -d'?' -f2 | cut -d' ' -f1 | rev

The one-liner above produces a list of packages that probably aren’t used by your software any more. Make sure to review manually though, as there could be false positives; or, if you have enough confidence in your tests, automate the removal of those dependencies iff that change doesn’t break the build.

My brand, my rules tokens

2021-09-29T00:00:00+02:00

Brands have long ceased to be simply the advertised image of a company. Today more than ever, brands interact with their customers in a digital way: social networks, landing pages, email, etc., and maintaining a consistent appearance is vital.

On the other hand, digital product development teams are becoming larger and more specialized, and it is no longer enough to keep the marketing and IT teams synchronized with some corporate rules. Now they need specialized tools that allow them to maintain a brand image consistent and, at the same time, allow them reuse work.

Let’s start from the beginning

Since its proposal in 1994 and its initial release in 1996, Cascading Style Sheets (CSS) evolved from a static description of simple styles to being the key of modern web design. However, CSS was designed to list and specify the styles used in a website and not as a programming language, therefore, its dynamic characteristics are quite limited.

Reusing styles often means copy-pasting or changing parent-child relationships, which can compromise the specificity of the selector. And the lack of variables and functions to reuse parts of the code makes changing the same value in multiple CSS files (such as primary color or font size) very difficult to maintain and there is a high risk of sets the wrong values.

The preprocessors improved the developer experience, turning CSS into a full programming language.

To resolve these shortcomings, the CSS preprocessors such as SASS and LESS were created. The preprocessors improved the developer experience, turning CSS into a full programming language with new functionality like variables, nested selectors, functions, color operations, logical operators, imports, etc. sometimes with a CSS like syntax, which makes it more readable and easier to maintain.

Despite this new advantage, we still have a static code that, once processed and loaded by the browser, can no longer be dynamically modified. This forces us to contemplate all the variations that we want to have available by adding hundreds of classes and modifiers that should be managed with media queries or changing the tags’ classes through JavaScript.

In 2015, the CSS Custom Properties for Cascading Variables Module Level 1 became a W3C Candidate Recommendation, introducing cascading variables that are accepted by all CSS properties. This new feature might seem quite simple but it offers the ability to do many of the things that preprocessors used to do, and also enabled new workflows that were not possible before.

Design Systems, the new era!

Recently, a new concept is becoming popular in the UX and Web Development arena: Design Systems.

Design Systems are tools that provide the essential infrastructure to guarantee design quality and development efficiency. They can range from a few concepts such as branded colors and typographies, to any element that defines the appearance of the application and its interaction with the user: components, templates, usability, tone of voice, etc.

Design Systems are tools that provide the essential infrastructure to guarantee design quality and development efficiency.

A good use of the Design Systems helps to improve production speed and communication, facilitate consistency, simplify and scale the product, achieve visual coherence, build new features more efficiently, keep more focus on UX, bring clarity to developers, reuse code and incorporate new components easily.

The work of designers who create or contribute to a Design System should mimic the world of the developer, doing more technical work and ensuring that they are designing in a reusable and systematic way. There are several areas where designers can contribute: Design Tokens, Components Libraries, Documentation, etc., but from now we are going to focus on Design Tokens.

Configuring your digital brand Design

Tokens are values that define features such as typography, colors, icons, spaces, etc. that will then be used to configure the styles of the application. They are the essence of the brand in a digital context. Design Tokens are intended to be flexible and cross-platform, therefore building it in a common language is paramount.

Figma and Adobe XD have empowered developers to link their code with the designers’ prototypes in a very direct way to keep a single source of truth.

At the same time that designers have been delving into the design of components and templates in an increasingly precise and collaborative way, new applications such as Figma and Adobe XD have also appeared and empowered developers to link their code with the designers’ prototypes in a very direct way.

With applications like Dragoman, Style Dictionary or the Adobe XD extension for VSC, the values used on UI sketches for color, typography, or spacing can be extracted as SCSS variables, JS variables, JSON, custom properties (CSS variables), or any other custom format. This allows maintaining a single source of truth from designs to the variables that configure the application’s style sheets, regardless of the language chosen to generate them.

Let’s rock!

Design Systems help organizations to maintain design coherence among all their front-end applications by sharing the same Design Tokens across the company.

A way to use these Design Tokens is to incorporate them as a dependency. This method, which usually involves a version control, makes it easier to manage changes in the code and helps to avoid errors when CSS files are preprocessed (with SASS or LESS) or exported from JavaScript (with any ‘CSS-in-JS’ technique).

Over time, all the effort that has been invested to keep the organization’s brand ends up in vain.

However, this strategy makes the applications use, gradually, more outdated versions of the libraries than without a scrupulous dependency updating. Over time, all the effort that has been invested to keep the organization’s brand ends up in vain, because a design requirement such as a color change or a logo update can take days, weeks or months to be adopted by all the applications.

How can we take full advantage of Design System?

We will have dynamic styles that can be adapted to different color settings, screen size, device, etc.

As previously described, since 2015 Cascading Style Sheets incorporate Custom Properties that allow us to use CSS variables directly from the browser itself. On the other hand, we have a cross-project that can transform Design Tokens into Custom Properties that can be load in the browser before the own applications’ style sheets.

With that environment, we will have dynamic styles sheets that may adapt all his colors, sizes, fonts, etc. from a few initial variables that define the primary or most abstract level of the brand. And the rest of the components and applications are going to configure his specific variations extending from there.

And the amazing thing is that it is also possible to reset variables, for example within media queries, and have those new cascade values everywhere. Something that is not possible with preprocessor variables. Check out this example:

If you are still not impressed, take a look at another demo where, using JavaScript to reset some CSS variables, the appearance of the page changes on the fly.

Happy coding!

Polymorphic Fixpoints: get rid of inheritance castings!

2021-06-29T00:00:00+02:00

Castings in Java are considered a bad practice.

Java’s strong static typing is a powerful compile-time safeguard. It checks that every method receives and returns objects with the features they are supposed to have. It makes sure that everything is well assembled.

Castings open a backdoor in this mechanism by allowing unchecked type transformations between classes involved in an inheritance. Although castings hardly have any impact in performance, they may lead to runtime errors, which in part defeats the very purpose of strong static typing.

It is commonly agreed that a repeated use of castings denotes a bad design. However, there are some cases in which avoiding them is far from obvious. One of these cases is having several subclasses inheriting from a superclass whose methods have argument or return types referring to the subclasses themselves:

public class A {
  public A method(A other) {...};
}

public class B extends A {
  @Override
  public B method(B other) {...}; // !!
}

Is it possible to avoid castings here?

A Tale of Two Castings

Suppose that we have found a really good algorithm to find long chains of elements. It takes elements from a Set<T>, and for any two of them, it needs to know if they may be chained. With this information, it efficiently builds long chains of elements. For instance, it may find knight tours in a chessboard

or it could create lists of words that change only one letter in each step:

PASS
PAST
CAST
COST
MOST
LOST
LOFT
...

The skeleton of our Chainer would be something like this:

public class Chainer<T> {

  final Set<T> items;

  public Chainer(final Set<T> items) {
    this.items = items;
  }

  public List<T> longChain() {
    // try to build a very long chain
    //
    // during the process, lots of pairs of elements are taken from items;
    // for each pair we need to check whether they may be chained or not
  };

  public List<T> joiningChain(T a, T b) {
    // try to build a chain joining a and b - again, pairs of elements 
    // are checked for "chainability"
  };
}

The type T does not provide the logic of chainable elements. So we build a Chainable interface to provide this logic:

// Chainer.java
public class Chainer<T extends Chainable> { /* ... */ }

// Chainable.java
public interface Chainable {
  public boolean mayBeChainedTo(Chainable other);
}

// CKnight.java
public class CKnight implements Chainable {
  public boolean mayBeChainedTo(Chainable other) { /* ... */ }
}

// CString.java
public class CString implements Chainable {
  public boolean mayBeChainedTo(Chainable other) { /* ... */ }
}

You see, we already have two examples of Chainable: chessboard squares and strings. Two squares may be chained if connected by a single knight jump, and two strings may be chained if only one letter differs. The logic itself is not very hard to implement. However, one may perceive a little subtlety with the types.

// CKnight.java
public boolean mayBeChainedTo(Chainable other) {
  CKnight k = (CKnight)other;
  // ...
}

The right thing would be to compare a CKnight with another CKnight, but the inherited method requires comparing with a Chainable. This forces us to begin the comparison with a downcasting.

But castings make programmers sad

Another possibility would be to provide a BiPredicate<T,T> to the Chainer

public class Chainer<T> {

  final Set<T> items;
  final BiPredicate<T,T> areChainable;
  // ...
}

but this would defeat our aim to encapsulate the behavior in the type itself.

Anoter example, this one a bit more complicated: imagine that we are building an applet that deals with 2-D geometric elements: points, curves, polygons, circles, splines, …

One may conceive different approaches to model 2-D regions. We could think of a numerical approach, where we would model the region using some sort of bitmap, or we could go towards a more analytic approach, keeping track of the atomic components. And perhaps both approaches are useful and can be used simultaneously (one to draw stuff on the screen and the other to perform accurate calculations).

All the approaches would need to implement a similar set of algebraic operations including union, intersection, complement and difference. Some of these operations may be computed in terms of others. This way we create the AlgebraicSet interface. The interface provides a default implementation for intersection and diff, calculated using the other two operations (union and complement).

public interface AlgebraicSet {

  public AlgebraicSet union(AlgebraicSet other);
  public AlgebraicSet complement();
  
  // --------
  
  public default AlgebraicSet intersection(AlgebraicSet other) {
    return this.complement().union(other.complement()).complement();
  }
  
  public default AlgebraicSet diff(AlgebraicSet other) {
    return this.intersection(other.complement());
  }

}

Cool!! Let’s now implement BitMap.

public class BitMap extends AlgebraicSet {

  public BitMap(/* ... */) {
    // perhaps it could be implemented as some sort of double array of booleans?
  }

  public BitMap union(AlgebraicSet other) {
    BitMap bitmapOther = (BitMap)other;
    // bitwise AND
    return new BitMap(/* ... */);
  }
  public BitMap complement() {
    // bitwise NOT
    return new BitMap(/* ... */);
  }

}

Did you notice?

And this time it’s slightly worse: there is a downcasting for the argument type much like in Chainable, but there’s also a difference in the return type. The initial methods to be implemented were AlgebraicSet union(AlgebraicSet other) and AlgebraicSet complement(); Java allows tweaking the return type with BitMap union(AlgebraicSet other) and BitMap complement(), but it’s still ugly nevertheless.

Let’s hope better luck for our symbolic approach in Shape:

public abstract class Shape implements AlgebraicSet {

  // union and complement in terms of certain subclases
  public Shape union(AlgebraicSet other) {
    Shape symbOther = (Shape)other; // Casting :/
    return new Union(this, symbOther);
  }
  public Shape complement() {
    return new Complement(this);
  }

  // checking whether a region contains a point
  // BitMap does not implement it because it's not accurate enough
  public abstract boolean contains(double x, double y);

  // --------

  public static class Union extends Shape {
    protected Shape a;
    protected Shape b;

    public Union(Shape a, Shape b) {
      this.a = a;
      this.b = b;
    }

    public boolean contains(double x, double y) {
      return a.contains(x, y) || b.contains(x, y);
    }

  }

  public static class Complement extends Shape {
    protected Shape a;

    public Complement(Shape a) {
      this.a = a;
    }

    public boolean contains(double x, double y) {
      return !a.contains(x, y);
    }

  }

  // a couple of basic shapes 
  
  public static class Rectangle extends Shape {
    protected double x0;
    protected double x1;
    protected double y0;
    protected double y1;

    public Rectangle(double x0, double x1, double y0, double y1) {
      this.x0 = x0;
      this.x1 = x1;
      this.y0 = y0;
      this.y1 = y1;
    }

    public boolean contains(double x, double y) {
      return x0 <= x && x <= x1 && y0 <= y && y <= y1;
    }

  }

  public static class Circle extends Shape {
    protected double xc;
    protected double yc;
    protected double r;

    public Circle(double xc, double yc, double r) {
      this.xc = xc;
      this.yc = yc;
      this.r = r;
    }

    public boolean contains(double x, double y) {
      return (x-xc)*(x-xc) + (y-yc)*(y-yc) <= r*r;
    }

  }

}

Ok. No problem so far. Let’s check that everything’s peachy.

new Rectangle(-1,1,-1,1).union(new Circle(1,1,1)).contains(0,0); 
// OK --> true

Great.

new Rectangle(-1,1,-1,1).diff(new Circle(1,1,1)).contains(0,0); 
// KO --> The method contains(int, int) is undefined for the type AlgebraicSet

What??!! The method diff was not implemented in Shape, but in AlgebraicSet, and therefore its return type is AlgebraicSet. Although semantically equivalent, union and diff work with different types. So each time we use diff we have to perform another downcasting

((Shape)new Rectangle(-1,1,-1,1).diff(new Circle(1,1,1))).contains(0,0); 
// (╯°□°)╯︵ ┻━┻

and this is too much. We need to find a better solution.

A wannabe type

In both examples, Chainable and AlgebraicSet, the problem is more or less the same: we want to build certain logic that will be inherited by multiple subclasses, and we need to have the current subclass type in the implementation.

How can we do this? Imagine that we have a type Base that will be extended by some type T; inside Base we want to refer to T so… why not? We introduce T as a type parameter.

class Base<T> // Base will eventually become T

Now we may use T to define argument and return types. But, on the other hand, T cannot be anything, it has to extend Base.

class Base<T extends Base>

But Base is no longer a raw type; it needs to be told about T.

class Base<T extends Base<T>>

What’s going on here? Is this a circular definition? Nope, it’s perfect Java syntax. You can impose type predicates on the parameters. If you find some T satistying the predicate, you’ll be able to instantiate a Base<T>. If the predicate is impossible to fulfill, then you won’t be able to instantiate a Base<T>, although everything will compile fine.

Suppose that Derived is to extend Base

class Derived extends Base<Derived>

Look, it works!! Base<Derived> is ok because Derived extends Base<Derived> satisfies the condition T extends Base<T>. And now Derived inherits all the logic in Base with all the T replaced by Derived. Think of Base<T> as being in mid-metamorphosis towards T: that’s why Derived extends Base<Derived>, because Base<Derived> was already prepared to be transformed in Derived.

How does this approach translate in our previous examples?

// Chainable.java
public interface Chainable<C extends Chainable<C>> {
  public boolean mayBeChainedTo(C other);
}

// CKnight.java
public class CKnight implements Chainable<CKnight> {
  public boolean mayBeChainedTo(CKnight other) { /* ... */ };
}

// CString.java
public class CString implements Chainable<CString> {
  public boolean mayBeChainedTo(CString other) { /* ... */ };
}

// Chainer.java
public class Chainer<C extends Chainable<C>> {
  
  final Set<C> items;

  public Chainer(final Set<C> items) {
    this.items = items;
  }

  public List<C> longChain() { /* ... */ };

  public List<C> joiningChain(C a, C b) { /* ... */ };
  
}

Did you see? No castings!

// AlgebraicSet.java
public interface AlgebraicSet<A extends AlgebraicSet<A>> {

  public A union(A other);
  public A complement();

  // --------

  public default A intersection(A other) {
    return this.complement().union(other.complement()).complement();
  }

  public default A diff(A other) {
    return this.intersection(other.complement());
  }

}

// BitMap.java
public class BitMap implements AlgebraicSet<BitMap> {

  // ...

  public BitMap union(BitMap other) {
    // bitwise AND
    return new BitMap(/* ... */);
  }
  public BitMap complement() {
    // bitwise NOT
    return new BitMap(/* ... */);
  }

}

// Shape.java
public abstract class Shape implements AlgebraicSet<Shape> {

  public Shape union(Shape other) {
    return new Union(this, other);
  }
  public Shape complement() {
    return new Complement(this);
  }

  public abstract boolean contains(double x, double y);

  // ...
  
}

Fabulous! Our types match perfectly!

new Rectangle(-1,1,-1,1).union(new Circle(1,1,1)).contains(0,0); // OK
new Rectangle(-1,1,-1,1).diff(new Circle(1,1,1)).contains(0,0);  // OK

Strange? Witchcraft? Well, you’d be surprised to know that this technique is already being used in the core of Java: the keyword enum

public enum Suit {HEART, DIAMOND, CLUB, SPADE};

is indeed syntactic sugar of a class Enum that uses this “recursive definition”

class Enum<E extends Enum<E>> { 
    ... 
}

class Suit extends Enum<Suit> {
  public static final Suit HEART;
  public static final Suit DIAMOND;
  public static final Suit CLUB;
  public static final Suit SPADE;

  // ...
}

This makes sense because all our enum have common properties but we want each subclass to be compared to members of the same subclass, much like with the Chainable elements. Yes, sometimes you have to reason in terms of your subclass types.

Polymorphic Fixpoint

We’re very happy with our trick and ready to move on. Let’s add another common set operation to AlgebraicSet: the symmetric difference.

public default A symDiff(A other) {
  return this.diff(other).union(other.diff(this));
}
// The method diff(A) in the type AlgebraicSet<A> 
// is not applicable for the arguments (AlgebraicSet<A>)

Yikes!! Since this method is implemented in the parent class, this refers to the parent class, and not to the child class. But we need this to be the child class. What can we do?

// Base.java
abstract class Base<T extends Base<T>> {
  abstract public T getChild();
}

// Derived.java
class Derived extends Base<Derived> {
  public Derived getChild() { return this; }
}

Looks good! To formalize this process we’ll create a Fixpoint interface that will be implemented by all the classes using this polymorphic fixpoint trick:

public interface Fixpoint<X extends Fixpoint<X>> {
  public X fixpoint();
}

Let’s introduce Fixpoint in Chainable…

// Chainable.java
public interface Chainable<C extends Chainable<C>> extends Fixpoint<C>

// CKnight.java
public CKnight fixpoint() { return this; }

// CString.java
public CString fixpoint() { return this; }

… and in AlgebraicSet:

// AlgebraicSet.java
public interface AlgebraicSet<A extends AlgebraicSet<A>> extends Fixpoint<A> {

  public A union(A other);
  public A complement();

  // --------

  public default A intersection(A other) {
    return fixpoint().complement().union(other.complement()).complement();
  }

  public default A diff(A other) {
    return fixpoint().intersection(other.complement());
  }

  public default A symDiff(A other) {
    return fixpoint().diff(other).union(other.diff(fixpoint()));
  }

}

// BitMap.java
public BitMap fixpoint() { return this; }

// Shape.java
public Shape fixpoint() { return this; }

Did you see? In AlgebraicSet we are now using fixpoint(), of type A, instead of this, of type AlgebraicSet<A>. Same happens with Chainable.

Why “fixpoint”? Because we’re asking each instance to “evolve” to its “more developed version”, but in the final subclases this evolution is the identity, and thus the return this;. And why “polymorphic”? Well, becuase this “more developed version” takes lots of different forms indeed.

Altogether, we have built the following hierarchy of classes/interfaces:

The dashed objects incorporate the polymorphic fixpoint trick in their type definition and cannot be instantiated until the thick objects implement fixpoint() { return this; } and no longer receive a type parameter. From this moment on, type inheritance is completely standard.

Now we can let our imagination fly and come up with all kinds of fancy stuff, like for example a fluent logger able to write the status of an object at any time

public interface Logger<L extends Logger<L>> extends Fixpoint<L> {

  default public L log() {
    System.out.println(repr());
    return fixpoint();
  }
  
  public String repr();

}

// BitMap.java
public class BitMap implements AlgebraicSet<BitMap>, Logger<BitMap> {
  // ...
  public String repr() { /* ... */ }
}

// some other file
new BitMap(/* ... */).log().union(new BitMap(/* ... */));

Wrapping up

To sum up: the polymorphic fixpoint trick is very useful to avoid castings in situations where a behavior is to be extended by several different implementations but needs to refer to the implementations themselves.

Galidevo Galilei

2021-05-28T00:00:00+02:00

Galileo Galilei was born in Pisa in 1564 and lived a very adventurous life. He studied gravity, motion, inertia and astronomy among others. He was so important in his time that he even got invited to a barbecue that Pope Urban VIII was throwing in his honor. Ended up declining the invitation when he learnt that he was the main dish. Of all his achievements, one was called to change the course of Humanity forever. Galileo is considered to be the father of the scientific method, a sort of magical recipe that empowers whoever makes proper use of it to tell the difference between true, proven facts and the rest. Our society, our technology and our progress is built upon a huge pile of scientific truths that have come out of Galileo’s magic wand.

Here at Devo, we are premium users of the scientific method. Conclusions and proven facts can only be drawn after a meticulous and intensive session of experimenting. The following story is good proof of that. A few weeks ago, a colleague from Professional Services reported a data loss in a new environment they were building. He even pointed the blame at a certain component of the system, because according to his experiments the data loss only occurred in newer versions of that component whilst older versions behaved properly. A true believer in the scientific method since his conclusions were based on experiments, but too hasty drawing his revolver. And maybe too laid-back performing the experiments. Galileo’s recipe is a very powerful tool, but you need to use it carefully and rigorously to make it work.

Solving a complex issue like this one is pretty much like watching a suspense movie. Meeting the characters, understanding the plot, and finally unraveling an unexpected ending.

Back to the Future (1985)

A bit of background first. Devo is a distributed database where the data, instead of being stored in structured, binary files, is spread out in thousands of text files (aka log files). We often refer to the “unit” of data in Devo as an “event”. An event (just an entry in a log file) is the equivalent of a record in a traditional database.

It’s easy to picture Devo as a wide river with a dam in the middle. The dam divides the river in two parts: the ingestion of logs and the exploitation of data. Upstream, the ingestion is in charge of riding the avalanche of log events coming from all our customers into the appropriate area of the dam (aka the hard drives). Downstream, the exploitation deals with understanding the queries, driving them through the appropriate pipes within the dam and returning accurate responses as fast as possible. Both sides communicate through the dam: the ingestion is responsible of storing the data in the right place and with the right format so that the exploitation can locate, understand and dissect what’s in there.

In this context, the concept of “data loss” always happens on the ingestion side. It implies that at least some of the data that we streamed down the river did not make it to the dam. And that’s a big problem because each drop of water only falls once.

The Usual Suspects (1995)

Devo is a complex creature, and only in the ingestion part, it has a few candidates to blame when something fails:

The sender. It’s usually a component called “the relay” (and written in Java), but in this case, it was a custom Python sender built on top of our python-sdk.
Python itself. Compilers, interpreters, runtime environments and standard libraries are also programs, and therefore, they can contain bugs and errors.
The load balancer (LB). Written in JavaScript and run with Nodejs, it receives the stream coming from the sender, frames the events and balances the load among the different syslog collectors.
The Syslog Collectors (SCs), also written in JavaScript and run with Nodejs. They are in charge of manipulating the incoming events and saving them to disk.
Nodejs, the JavaScript VM in charge of running the LB and the SCs.
Docker. Yet another layer on top of the OS. Prone to errors, as everything else.
The infrastructure: the networks, the file system, the disks, the OS, etc…
The relative position of Saturn and Uranus three days after the summer solstice. It changes every year and it deeply affects the behaviour of our software.

Let’s get down to the facts. There was an experiment that was showing a loss of events in a new docker environment featuring v5.12.x of our LB (running on Nodejs 12.x.x), a very recent version of the SCs and a sender agent that was using our python-sdk to send the events. The experiment consisted in sending around 150M events and then checking that all of them were written to disk. The whole thing took around 3 hours to complete, and allegedly lost a few tens (sometimes hundreds) of events in the way. Again, “losing” here means that some events were sent, but never made it to the disk. This “same” experiment was performed in a different environment running v4.11.x of the LB on Nodejs 8.x.x and was not losing events. It looked like the LB was the murderer, but there were still a few surprise witnesses to be called to court.

Finding Nemo (2003)

First thing to do is always finding out what’s wrong. Anything that may remotely account for what’s been observed. Then grab it, and dig deeper.

The experiment that originally led to the reported event loss was way too heavy. Waiting 3 hours for each experiment to finish can exhaust even the most patient scientist. Also, production environments are kind of delicate, and debugging them is not usually an option. Best scenario? Reproduce the event loss in your machine where you can mess up with almost everything with no fear of breaking anything valuable. Drawbacks? Your local environment is probably nothing like the production environments and things that happen there might never reproduce here. Our objective? Lose events in our machine with a simple, light and easy to reproduce experiment. Not an easy task, but it’s where we want to be.

Logs are our friends. Not only because they pay for our bills, but also because they usually contain vital information to find out what’s happening. All we’ve got so far is a very heavy experiment causing a very little event loss only in some selected environment. Unfortunately, the original environment where the issue consistently reproduced was highly secured, and few people had access to it, so we ended up building a replica in a public cloud. And luckily for us, the experiment led to event loss, and looking into the LB logs we found a few traces like this:

TCP source socket error: Error: read ECONNRESET

The sender and the LB communicate via TCP (secured with TLS), and this error is triggered when the connection is forcibly closed by the sender. What does “forcibly” mean in this context? Well, both TCP and TLS are protocols, and their implementations are expected to comply with these protocols. In particular, closure of connections has to follow specific rituals, and errors occur when these rituals are performed poorly. The LB ends up destroying the socket after an error like this, because no more data is expected after receiving an error event.

This trace was a very good scapegoat because:

Abrupt interruptions of TCP/TLS sockets can easily lead to data loss.
The error was triggered 3-4 times per experiment. One time per hour because our python SDK recycles the TCP socket once per hour, and then one time at the end of the experiment. This would perfectly account for the amount of events lost (tens, sometimes hundreds).

Saving Private Ryan (1998)

Gear up because we are getting into the battlefield. Yes, we do have a good candidate to throw to the wolves, but we still have to build a solid case against him. It’s time to simplify things and design much more targeted experiments. Replace the 3 hours experiment with a 3 seconds gig that we can perform again and again. Trigger a scientific method avalanche. We managed to create a repo where we isolated the behaviour of both sender and receiver that was causing the error. And we succeeded in reproducing it with a very simple experiment!

The challenge was then to get rid of the dark clouds approaching from the North. First of all, the error only happened with TLS sockets, never with plain TCP sockets. So it has to be something that the TLS implementation of the receiver is expecting, but not getting. Also, receivers launched with Nodejs 8.x.x never failed, while Nodejs 12.x.x consistently did. Fingers stopped pointing at the LB and started pointing at Nodejs itself. The same (simple) Python client triggered different behaviours in the same (simple) Nodejs receiver depending on the version of Nodejs used.

Was there anything wrong with our sender? Well, in fact there was. The official Python documentation says two things about closing a socket:

If you have a plain TCP socket, you should call socket.shutdown(how) before socket.close() in order to close it in a “timely fashion“.
If your socket is wrapped into a TLS context, calling socket.unwrap() will perform the TLS shutdown handshake. Something that seems very polite before saying goodbye.

We were doing neither of the above. Adding socket.shutdown(how) to our python-sdk seemed to solve the issue, so we assumed that the TLS layer was unwrapping itself, but there were still a few jungles to trim:

Nodejs 8.x.x seemed pretty stable regardless of how abruptly connections were closed from the client side. Nodejs 12.x.x behaved very non-deterministically. Is this something we should report to Nodejs?
Even with Python closing the socket in a “timely fashion”, the bytes received on the Nodejs side fluctuated a bit when the CPU was busy enough, although the ECONNRESET error never showed up. This, of course, only happened with Nodejs v12.x.x.
Using Nodejs also to send events (closing the connections abruptly on purpose) revealed an even weirder behaviour: sending and receiving with Nodejs 12.19.0 seemed pretty seamless, but using Nodejs 12.18.4 to send triggered the ECONNRESET error consistently (!!!!).
We wrote a simple sender in Java and we also reproduced the issue. If we closed the socket by the book, then everything went smooth, both with the receiver in Nodejs 8.x.x and Nodejs 12.x.x. But if instead of closing the socket politely we interrupted the thread that was running the connection, then Nodejs 8.x.x would behave ok, but Nodejs 12.x.x would show data loss and ECONNRESETs.

Jaws (1975)

It’s time to pull out the heavy artillery, and when you are dealing with a TCP/TLS poltergeist, this means calling in the shark. Observing and analyzing what’s going down the wire is a toilsome task. Especially when TLS is involved. Performing the Python experiments with the shark watching revealed that the amount of data interchanged when closing the socket was growing as we refined our manners:

a simple socket.close() produced a minimal amount of data
prefixing socket.shutdown(how) would make the data interchange grow
prefixing socket.unwrap() would make the amount of data reach its maximum

Using Nodejs on the sender side showed that closing the connection with Nodejs 12.18.4 produced less data than doing it with Nodejs 12.19.0. The TLS RFC says that in order to close a connection politely you need to send a close_notify message before attempting to close the TCP connection and it looked like some senders were forgetting to do so, and therefore triggering an error in the receiver. But why didn’t this error show when the receiver was running on Nodejs v8.x.x?

Indetermination is also a potential result of a scientific experiment. Pick a random university and ask anyone in the Quantum Physics department. Observing the wire with the shark altered the results of the experiments. How? Well, the shark is a very powerful tool, but it absorbs a lot of CPU to perform its duties. And this made the experiment fail more often. If we turned it off, then some of the experiments might not even fail. Why was this happening?

Hollywood Ending (2002)

The shark was talking and we just needed to point our ears in the right direction. The combination of circumstances that made our experiment fail had something in common. But it was buried too deep for us to see in the first place. Digging with our scientific shovel was getting us closer to the treasure. Sniffing the wire made us realize that successful experiments showed a couple of extra application packets traveling from the client to the receiver and back. In failing experiments, these strange packets never showed up. We began to think that sometimes the client was failing to perform some of the dances required by the TLS closing ceremony, and the receiver was getting angry, asking for an early divorce and losing some of the progeny on the way. But why? What was the common factor of all the successful/failing experiments? And what was the difference? Suddenly a few bulbs started to turn on in the dark forming the initials T, L and S. Yes, all the experiments that failed were using TLS1.3 to encrypt the wire. Running the receiver with Nodejs 8.x.x guaranteed success because Nodejs 8.x.x does not support TLS1.3. We confirmed this by running experiments using Nodejs 12.x.x in the receiver, but forcing the Python/Nodejs client to use TLS1.2. They all succeeded. Then, where is the bug (if any)? Where is the problem? How can we fix it?

We detected at least a couple of deficiencies that could be improved. And both of them seemed to fall in the “compiler/interpreter“ category. Both, Python 3 and Nodejs <= 12.8.4 forget to send the close_notify message when they are acting as senders in a TLS1.3 play and the show is about to end. Nodejs fixes the issue in v12.19.0, and Python behaves correctly if socket.unwrap() is called before closing the socket (socket.shutdown(how) alone does not do the trick, maybe it’s just a matter of improving the docs). Senders were not closing properly the communication with the receiver when using TLS1.3, but the receiver (Nodejs) could also be a bit more resilient with the small negligence of the sender and obviate the fact that there is a protocol packet missing in exchange of making sure that no data is lost.

And what about the disturbing fact that even with failing experiments, failures do not occur all the time? Why do (failing) experiments fail more often when the CPU is busier? It’s really very hard to tell, but my wild guess would be that it has to do with the order in which the kernel “sees” the incoming data: if it “sees” all the data before the lousy closure, then there will be no error, but if it “sees” a lousy closure when there are still pending data, then it will signal the error. It’s just an hypotheses, and proving it would require diving way deeper than we did.

The Matrix (1999)

Blue pill or red pill? Let’s have both of them. Or neither. Complex problems demand looking at them from every possible angle. From inside and outside the matrix. As Bruce Lee would put it, you need to “become” the application you are trying to debug and travel at the speed of bit through the logic silicon pipes of the computer. Pills are not given away by some Morpheus stuffed in a leather jacket at the beginning of the trip, but they are rather earned during the journey. And they are neither red nor blue. Most of the times they are grayish and they are called wisdom pills. Here are some we’ve collected lately. Use them at your own risk.

Pill 1: well designed experiments extend your knowledge. Lame or biased experiments could reinforce your ignorance. Spend time thinking what you want to prove and how to do it.
Pill 2: truth is a very elusive concept. Paradoxically, now more than ever. In SW engineering, the truth is always written in the code.
Pill 3: if the code is too hard to read, consider it as a black box and perform as many experiments as you need to understand it.
Pill 4: everything is code: our applications, the compilers/interpreters that run them, the runtime environments, the OS libraries, the OS itself, etc…
Pill 5: everything has bugs, even the most proven SW.
Pill 6: when you start a murder investigation, do it with an open mind and start over a blank page. Don’t make any assumptions. Drop all your prejudices.
Pill 7: be a methodical and rigorous investigator. Support all your findings with sound experiments. Then wash, rinse and repeat.
Pill 8: if you happen to find a killer, report it.
Pill 9: if you happen to find a corpse, identify it before dumping it.