Sebastien Dumetz’s Blog

Plugins helpers for Jekyll

2017-08-22T00:00:00+00:00

I use Jekyll a lot at work for various static sites. Throughout it, I’ve developped some handy plugins. While Jekyll’s plugins system is easy and powerful to use, the doc lacks some info on helpers functions, right_way™ to parse arguments, etc… Here’s a short list, half for myself to remember, half for others to learn.

Class documentation

While jekyll is great, developpers might like rubydoc’s page to get a complete Class reference on Jekyll’s internals. The Utils class has some useful methods. Similarly, Liquid gem’s docs are not advertised in any of the mainstream sites. It reveals priceless when digging into templates internals.

The Liquid class has some useful constants, like Liquid::QuotedFragment which will match any quoted string

Helper functions

lookup_variable

lookup_variable(name, context) is the only export of Jekyll::LiquidExtensions, and is a handy shortcut to interpret liquid variables at render time.

Here is the example code for a Liquid::Tag :

require 'jekyll/liquid_extensions'
module Jekyll
  module Tags
    class FooTag < Liquid::Tag
      include Jekyll::LiquidExtensions
      VARIABLE = /\{\{\s*([\w]+\.?[\w]*)\s*\}\}/i
      def initialize(tag_name, markup, tokens)
        super
        @markup = markup
      end

      def interpret(markup, context)
        markup.scan VARIABLE do |variable|
          markup = markup.gsub(VARIABLE, lookup_variable(context, variable.first))
        end
        markup
      end
      def render(context)
        parsed_markup = interpret(@markup,context)
        # actual code here
      end
    end
  end
end

It will replace anything enclosed in “{{ }} “ by it’s variable value within context. If the variable does not exists, it outputs the variable’s name.

sanitized_path

Jekyll require you to “jail” paths to the site’s source directory. Jekyll.sanitized_path(base, file) will effectively ensure this. Complete behaviour tests are available for review here. Practical usage inside your plugin will look like this :

def render(context)
  file_path = Jekyll.sanitized_path(context.registers[:site].source, interpret(@markup,context))
  # actual code here
end

Thus, /foo.txt, foo.txt will resolve relative to the site’s source dir. Furthermore, ../foo.txt and /../foo.txt will also work. I don’t see why you’d do that but it’s always good to have some layer of safety.

Testing

Greatly inspired by work done by others on jekyll-sitemap.

Integration testing

Always check your code actually does something out there in the wild. Fortunately, it’s easy to do just that. Create a fixture/ directory with the required files :

_layouts/default.html
index.html
any necessary file for test purposes

Then test (using rspec here. adapt to your framework of choice)

SOURCE_DIR = File.expand_path("../fixtures", __FILE__)
DEST_DIR   = File.expand_path("../dest",     __FILE__)

def source_dir(*files)
  File.join(SOURCE_DIR, *files)
end

def dest_dir(*files)
  File.join(DEST_DIR, *files)
end

describe "Integration" do
  before(:context) do
    config = Jekyll.configuration({
      "source"      => source_dir,
      "destination" => dest_dir,
      "url"         => "http://example.org",
    })
    site = Jekyll::Site.new(config)
    site.process
  end
  it "render site" do
    expect(File.exist?(dest_dir("index.html"))).to be_truthy
  end
end

Then add test cases to verify if output is as expected.

Hooks

Documentation on Hooks is sparse and it’s a rather underused feature. In my experience, nearly anything you’d want to do with a hook can be done with a template. However :

Hooks can dramatically speed up your rendering

They require no context switch, no bouncing from Liquid components : pure ruby logic. I had this snippet to get a page’s language depending on its url :

    {% unless lang %}
      {% if page.lang %}
        {% assign lang = page.lang %}
      {% elsif page.url contains "/fr" %}
        {% assign lang = "fr" %}
      {% elsif page.url contains "/en" %}
        {% assign lang = "en" %}
      {% else %}
        {% assign lang = "fr" %}
      {% endif %}
    {% endunless %}

This simple snippet was adding 2 seconds of render time on every update, with a site of ~100 pages.

The exact same thing in ruby would take a negligible amount of time :

re = /^\/(?<lang>fr|en)\/(.*)$/
Jekyll::Hooks.register :documents, :pre_render, priority: "high" do |content, doc|
  url = doc.page["url"]
  filepath = doc.page["relative_path"]
  if !  doc.page.key? "lang"
    match = re.match(url)
    if match
      doc.page["lang"] = match["lang"]
    else
      doc.page["lang"] = "fr"
    end
  end
end

Jekyll’s doc already disclose the scopes and events available for use. A few precisions :

One can not set pages variables in the :post_init phase : Those hooks use the Jekyll:Document, Jekyll:Site, etc… Objects. Trying to do so will yield an error :

    Error:  undefined method `[]=' for #<Jekyll::Document:0x00000002fcedb8>

On the other hand, :pre_render hooks are using Jekyll::Drops::DocumentDrop and equivalents, which set the []= operator.

:documents hooks have access to a “content” variable, |content,doc|. It’s defined as “NO CONTENT” in early hook.

The page variable can be modified in :documents, :pre_render hooks using thissyntax : doc.page["foo"]="bar", and will be available in templates as ``. The site variable can be modified at any stage using site.config["foo"] = "bar". Site is generally still accessible in other hooks as a child of the main object.

Jekyll::Hooks.register :documents, :pre_render, priority: "high" do |content, doc|
  url = doc.page["url"]
  doc.page["foo"] = "hello world"
  # ...
end

Handle linux signals in Unity3D

2017-07-01T00:00:00+00:00

Unity3D’s portable runtime based on Mono has become a great “write once run everywhere” tool for interactive apps or video games. However stable it is, there is always some edge cases to portability. The handling of SIGTERM, SIGINT, etc… linux (well, POSIX) signals are one of them.

The bug

Make a dummy unity3D app. Make it do some work in OnApplicationQuit() callback. Get it’s PID and terminate it with kill <pid>. The exit handler never get called.

Other app lifecycle events like GameObject.OnDestroy() obviously won’t get called either because scene cleanup is a consequence of a proper exit sequence.

linux(UNIX/POSIX/whatever) exit signals behaviour is well documented. Generally only two of them will ask for specific exit handlers :

Common name	N°	Action
SIGINT	2	Interruption
SIGTERM	5	Exit request

Lots of ways exists to handle signals under Mono. However Unity3D really is a thing of its own. As such, it doesn’t let us catch those little things.

The fix

As a “temporary” fix, I made a barebones C file to catch the process’s signal :

sighandler.c

#include <signal.h>

void (*c)(int);
void sig_callback(int n){
	c(n);
}
void OnTerm(void (*handler)(int)){
	c = handler;			
	signal(SIGTERM, sig_callback);
}

It compiles easily to a “.so” shared library

    gcc -shared -o libsighandler.so -fPIC sighandler.c

The output file (compiled on debian jessie) can be downloaded here.

We need this file in Assets/Plugins to be automagically exported on build targetting linux platform. Then it needs to be referenced from Unity Scripts. In C# :

using System.Collections;
using UnityEngine;
public class HandleTerm : MonoBehaviour {
	//This is the important part
	public delegate void handlerDelegate();
	[DllImport ("sighandler")]
	private static extern void OnTerm(handlerDelegate handler);
	void ExitHandler(){
		print("onExit");
		//We must call Application.Quit() manually :
		// The default built-in handler is cancelled.
		Application.Quit();
	}
	//Then it's just normal code
	// Use this for initialization
	void Start () {
		//Register handler on initialization
		OnTerm(new handlerDelegate(this.ExitHandler));
	}

	// Update is called once per frame
	void Update () {
	}
	void OnApplicationQuit(){
		//This doesn't get called under normal circumstances when app get a SIGTERM.
		print("Application Quit");
	}

}

Note: print() statements get logged to ~/.config/unity3d/<company_name>/<app_name>/Player.log.

This will override system’s default SIGTERM and SIGINT handlers and let you act accordingly. Note that it’s bad behaviour to take more than a few seconds to exit. As such, most window managers will SIGKILL the process after a few seconds. SIGKILL by itself is interruptible so behave!

JS CustomEvent efficiency

2017-06-20T00:00:00+00:00

CustomEvent in browser-JS is a rarely used feature, while it’s ubiquitous in it’s server-side counterpart. It looks like an elegant solution to dispatch state changes, redux-style.

Let’s see the good in this undervaluated feature

Browser support

Good ol’ caniuse is positive :

Can I Use customevent? Data on support for the customevent feature across the major browsers from caniuse.com.

We’ll need a IE 11 polyfill, which is fortunately simple enough (from mdn):

(function () {

  if ( typeof window.CustomEvent === "function" ) return false;

  function CustomEvent ( event, params ) {
    params = params || { bubbles: false, cancelable: false, detail: undefined };
    var evt = document.createEvent( 'CustomEvent' );
    evt.initCustomEvent( event, params.bubbles, params.cancelable, params.detail );
    return evt;
   }

  CustomEvent.prototype = window.Event.prototype;

  window.CustomEvent = CustomEvent;
})();

Usage

We can easily make helper functions like this :

function dispatchFoo(detail){
  document.dispatchEvent(new CustomEvent("foo",{detail:detail}));
}

function listenFoo(fn){
  document.addEventListener("foo",fn);
  return document.removeEventListener.bind(document,"foo",fn);
}

The listener can unsubscribe itself as it should. It lacks things like a listenOnce() shortcut, but that should be straightforward to implement as needed.

Performance

All’s well, but how fast is it?

The main drawback we read about over the web is the performance penalty of using too much event listeners in your app. However here, we’re not setting loads of handlers all over the place. It’s only using the document element. That change things…

We’ll be testing 3 cases on 10000 iterations each. (complete code here)

Case 1

All events are registered on document global element. 10000 listeners are called when an event is dispatched.

Case 2

10000 DOM nodes are created. Each one gets an element attached. We dispatch 10000 events and each listener gets its own.

Results

type	registration	dispatch	unsubscribe
global listener	135ms	17ms	28ms
multi listeners	13ms	62ms	8ms

Of course it’s hard to get a good measurement on a real world use case (3-4 listeners) because it’s sub-millisecond…

Overall, those results are great. Why so much hate over this function? It comes from a bad practice that frameworks like React and other Shadow DOM based projects tries to solve.

Repaints

Each time something in the DOM that would change appearance is modified, the browser performs a repaint (it’s more of a cascade of operations, but let’s keep it simple).

Except… not really.

To be efficient, the browser does not repaint until enough time has elapsed OR it’s queried for some DOM property and need to recalculate.

Operation	time
update (x1000)	4ms
query+update (x1000)	472ms

Ouch.

What does it means for our little listener? If it does ANY DOM query, it’s going to take ~30ms to complete (depending on page’s complexity).

Conclusion

the CustomEvent mechanism is a great way to propagate, well, events… through your app. However, as always with vanilla JS, it’s easy to hurt performance greatly with no apparent reason.

Providing codecs informations on HTML5 video

2017-05-29T00:00:00+00:00

The HTML5 VideoElement allows for codecs informations in the type property. For once, even safari seems to happily support it. How is it important? Because browsers will choose the first input they can play solely depending on the type property (or file extension, if absent). Once they make the wrong choice (i.e. A too high level mp4 video for iOS/ a low end android), they’ll just hang there forever.

However, the codecs infos are really not so well documented. I made detailed search on the subject and here are all the bytes you could need to describe your h.264 presets.

Introduction

an h.264 video is described as :

  <source src="/path/to/video.mp4" type='video/mp4 codecs="avc1.4D401F"'>

avc1 being h.264’s reference. the 3 hex bytes after the dot are :

profile_idc
constraint_set
level_idc

hereafter you’ll find a complete description of their meaning.

Byte 1 : profile_idc

in h.264 syntax, profile is usually given as a string. For example using libav or ffmpeg, we’ll set it with -profile:v high. libx264’s code has a simple if chain to parse those (in common/common.c)

    static int profile_string_to_int( const char *str )
    {
        if( !strcasecmp( str, "baseline" ) )
            return PROFILE_BASELINE;
        if( !strcasecmp( str, "main" ) )
            return PROFILE_MAIN;
        if( !strcasecmp( str, "high" ) )
            return PROFILE_HIGH;
        if( !strcasecmp( str, "high10" ) )
            return PROFILE_HIGH10;
        if( !strcasecmp( str, "high422" ) )
            return PROFILE_HIGH422;
        if( !strcasecmp( str, "high444" ) )
            return PROFILE_HIGH444_PREDICTIVE;
        return -1;
    }

We can get the corresponding integer in common/set.h

    enum profile_e
    {
        PROFILE_BASELINE = 66,
        PROFILE_MAIN     = 77,
        PROFILE_HIGH    = 100,
        PROFILE_HIGH10  = 110,
        PROFILE_HIGH422 = 122,
        PROFILE_HIGH444_PREDICTIVE = 244,
    };

so here is the complete translation from profile string to profile_idc

profile	enum	idc
baseline	66	42
main	77	4D
high	100	64
high10	110	6E
high422	122	7A
high444	244	7A

Byte 2 : constraint_set

Again from the code :

sps->b_constraint_set0  = sps->i_profile_idc == PROFILE_BASELINE;
/* x264 doesn't support the features that are in Baseline and not in Main,
 * namely arbitrary_slice_order and slice_groups. */
sps->b_constraint_set1  = sps->i_profile_idc <= PROFILE_MAIN;
/* Never set constraint_set2, it is not necessary and not used in real world. */
sps->b_constraint_set2  = 0;
sps->b_constraint_set3  = 0;

sps->i_level_idc = param->i_level_idc;
if( param->i_level_idc == 9 && ( sps->i_profile_idc == PROFILE_BASELINE || sps->i_profile_idc == PROFILE_MAIN ) )
{
    sps->b_constraint_set3 = 1; /* level 1b with Baseline or Main profile is signalled via constraint_set3 */
    sps->i_level_idc      = 11;
}
/* Intra profiles */
if( param->i_keyint_max == 1 && sps->i_profile_idc >= PROFILE_HIGH )
    sps->b_constraint_set3 = 1;

A bit more difficult to read but in the end pretty straightforward. The tricky thing here is given how these are written in headers, constraint_set0 is the highest order bit.

b_constraint_set0 to b_constraint_set5 and 2 reserved bytes. The code tells us constraint_set2 is never used. Also, 4 and 5 are not implemented in libx264. The last 4 bytes are written as :

bs_write( s, 4, 0 );    /* reserved */

so we have this mask : xx0x0000

0 and 1 are easy to figure out :

Having PROFILE_MAIN, we get : constraint_set1=1 any higher profile would yield constraint_set1=0. ``constraint_set0` is set only when using PROFILE_BASELINE.

Then constraint_set3 is set if level_idc is 9 AND profile is BASELINE or MAIN. or : *keyint_max = 1 and profile_idc is HIGH or more**

Easy to figure out if you know your encoding options. If not, any tool would give those results.

A quick look at common/set.c tells us that bits 4 and 5 are treated as the other reserved.

profile	bits	byte
baseline	11000000	0xC0
main	01000000	0x40
high	00000000	0x00
high(intra)	00010000	0x10

Note : ** this changes if your level_idc is **9 and profile <= MAIN . Check later section and add 0x10 if it’s the case.

Byte 3 : level_idc

We have 2 syntaxes for levels. 4.1 or 41 means the same thing. The code speaks for itself :

if( !strcmp(value, "1b") )
            p->i_level_idc = 9;
        else if( atof(value) < 6 )
            p->i_level_idc = (int)(10*atof(value)+.5);
        else
            p->i_level_idc = atoi(value);

1b is a special case that yields a 9. Other values are converted

So we just take our level and convert it to hex.

Conclusion

For a standard h.264 Main Concept level 3.1 video, your codec code would be : avc1.4D401F. For any other stream, you can then infer the bytes from your encoding options.

Encoding videos for the web in 2017

2017-01-12T00:00:00+00:00

I’ve fiddled a bit recently with self-made HTML5 video players. While it’s easy to use and now widely compatible, I’ve had some trouble encoding videos. So here is my cheat sheet for mobile-first video distribution in 2017, including an in-depth look into mp4/webm performance.

Encoding commands

I used libav’s avconv tool on debian Jessie. It uses libav56. older versions might lack the proper options for webm format. In the following examples, I’ll assume usage of a png frames sequence as input. Otherwise, just remove the -f image2 option.

First thing first : mp4.

  avconv -y -f image2 -i path/to/frame-%04d.png -c:v libx264 -c:a copy -pix_fmt yuv420p -profile:v main -level 31  output_file.mp4

Note : The iphone 3 is requiring an even lower set with -profile baseline -level 31. According to apple’s doc on http streaming, earlier iphones support only -level 30. Those settings are for iphone 4 and above. Apple doesn’t advertise which devices support h.264 High profile so I don’t know when it’ll be “safe” to upgrade.

Should we still provide multiple fallbacks now? It depends. MPEG-4/H.264 is clearly the most widely supported format. However a quick look in caniuse.com show that Opera Mini still don’t support MPEG4/H.264. Well, it’s not like it supports any other format though.

So with firefox supporting mp4 since v35 (Jan 2015) and Opera since v25 (Oct 2014), my own video for everybody script just got a lot simpler with no flashPlayer or video/ogv fallbacks.

What of the video/webm?

Pros
- More compact
- Easy seek (more on this later)
Cons
- iOS crash if we put it first
- android seems to ignore it if it’s not first

However, encoding in vp8 is no longer a great trouble. While most earlier guides would make you install some voodoo to be able to export your webm file, a stock avconv will now do :

  avconv -f image2 -i path/to/frame-%04d.png -c:v libvpx -c:a libvorbis -qmin 20 -qmax 30 -threads auto  output_file.webm

Encoding choice

Now is it worth it to replace good ol’ h.264 ? Let’s check the rate/distortion gain. Unlike previous empirical results from me and others, I’ll try to go scientific here. While we’re at it, I’ll also test vp9 encoding.

Since we’re serious about it, let’s compare a few things :

H.264 main profile 31
H.264 high profile 41
VP8 (1 or 2 pass)
VP9 (1 or 2 pass)

I didn’t do 2pass on H.264 as crf (default quality setting) doesn’t support it. To be really complete, I should try with constant bitrate and other quality-control settings.

However, here’s the script I made to test all this :

Give it a png sequence aaaaand it’s done.

Results?

As of now I only measure PSNR (Peak Signal/Noise ratio), added throughout every images. Run the tests at home, YMMV as always.

MP4(main) : 1665kB . PSNR : 15902.2121
MP4(high) : 1705kB . PSNR : 15904.4900
VP8 (1 p) : 1595kB . PSNR : 15848.7263
VP8 (2 p) : 1617kB . PSNR : 15859.8030
VP9 (1 p) : 1614kB . PSNR : 15880.7009
VP9 (2 p) : 1471kB . PSNR : 15908.0548

Higher PSNR = lower distortion.

How much should I trust this?

Not much. Signal/Noise ratio is a thing, but it doesn’t make a difference between this noise in the background and movement noise that makes your video look bad. However, it’s still a hint on real encoding quality.

Conclusion

Providing a bazilion of fallbacks is no longer required if you target mobile devices, which tends to have up to date browsers and can’t be older than 2006 (obviously…), or are not meant to play video anyway.

a fallback to vp8/vp9 encoded video could provide some meaningfull bonus, but is by no means a no-brainer. It depends how much your viewers will rely on seek capacities, for example.

Those measures were enough to convince me to drop ogv support make webm secondary in my online video service.

To be continued : How good are VP8/VP9 when the user seeks a specific time?

Using Webworkers with Webpack

2017-01-02T00:00:00+00:00

I started using web workers for real-worl loads a few weeks ago and like many before me, I found it to be quite easy to use, but a real pain to debug. I’ll gather my thought about them for later use.

The work was pretty simple :

some static assets to always cache upfront
A few API routes I wanted to cache “if possible”.

However, as often with shiny new stuff, documentation can be hard to find.

Setup

webworkers don’t have access to exactly the same runtime as normal scripts. Namely, all window methods and properties are missing. For this reason, webpack’s normal build entries won’t work as webservices. Fortunately, webpack provides a webworker target. One would need 2 config objects to generate basic modules + a web worker :

const config = {
  target: "web", //default
  entry:"main.js"
}
const wsConfig = {
  target:"webworker",
  entry:"sw.js"
}
module.exports = [config, wsConfig];

It took me longer than i’d like to admit to figure this one out, but it’s dead simple and works like a charm. It’s even, documented in the shiny new webpack 2 doc.

However, I could not make it work with hot-reload. I simply disabled hotReplacementPlugin() from my swConfig. It means my page won’t reload automatically when I change sw.js.

Another option, which I finally came to use, is the sw-precache-webpack-plugin, which is just a wrapper around sw-precache. It takes your static files and generates a service file with them.

The service is slightly larger this way, but it save a lot of time and (more importantly), allows readable strategy handles depending on file names.

Update

One main trouble you have, starting a web worker is man,aging updates : You want cache to be invalidated EVERY TIME it should, but your clients should be able work offline for months (okay, hours…) without trouble.

The first trick is to enable clientsClaim:true in sw-precache, or call self.clients.claim() in your worker if you can. It’s not a one-stop-fit-all feature, so read the doc.

The other quick-start improvement is to have a good “install” step. Google’s introduction example doesn’t handle errors and that caused me some trouble :

if ('serviceWorker' in navigator) {
  window.addEventListener('load', function() {
    navigator.serviceWorker.register('/sw.js').then(function(registration) {
      // Registration was successful
      console.log('ServiceWorker registration successful with scope: ', registration.scope);
    }).catch(function(err) {
      // registration failed :(
      console.log('ServiceWorker registration failed: ', err);
    });
  });
}

It’s not able to cleanly handle update events.

I found the registration example from swPrecache to be quite complete :

if ('serviceWorker' in navigator) {
  // Your service-worker.js *must* be located at the top-level directory relative to your site.
  // It won't be able to control pages unless it's located at the same level or higher than them.
  // *Don't* register service worker file in, e.g., a scripts/ sub-directory!
  // See https://github.com/slightlyoff/ServiceWorker/issues/468
  navigator.serviceWorker.register('service-worker.js').then(function(reg) {
    // updatefound is fired if service-worker.js changes.
    reg.onupdatefound = function() {
      // The updatefound event implies that reg.installing is set; see
      // https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#service-worker-container-updatefound-event
      var installingWorker = reg.installing;

      installingWorker.onstatechange = function() {
        switch (installingWorker.state) {
          case 'installed':
            if (navigator.serviceWorker.controller) {
              // At this point, the old content will have been purged and the fresh content will
              // have been added to the cache.
              // It's the perfect time to display a "New content is available; please refresh."
              // message in the page's interface.
              console.log('New or updated content is available.');
            } else {
              // At this point, everything has been precached.
              // It's the perfect time to display a "Content is cached for offline use." message.
              console.log('Content is now available offline!');
            }
            break;

          case 'redundant':
            console.error('The installing service worker became redundant.');
            break;
        }
      };
    };
  }).catch(function(e) {
    console.error('Error during service worker registration:', e);
  });
}

Still not perfect, but it at least clearly says when cache have been updated. If it have not, you know where the trouble is.

cache control

At the end of the day, the only reliable way to achieve good cache control is to uniquely name your scripts so they can not be mistaken.

The sweetness here is either nothing have been updated, and you still have your full old app working (cheers!), or the <html> is new, and it’s script are named differently, and everything is loaded anew. There’s no annoying middle ground.

Creating uniquely named scripts with webpack is easy :

filename: '[name].[hash].js'

there is also a chunkhash variable which is unique to each chunk. hash however is shared

The easy way to use it is to just output your HTML as static files from webpack on each rebuild.

When you can not, because of some dumb dynamic content you decided to add in it (you fool, it’s so 2000’s!), you can make webpack output a stats.json at build time, and read it from your server.

// In webpack.config.js
const fs = require("fs");
config.plugins.push(function() {
  this.plugin("emit", function(stats,callback) {
    fs.writeFile(
      path.resolve(__dirname, "static/build", "stats.json"),
      JSON.stringify(stats.hash),callback);
  });
});

Then you require your file as const hash = "."+require(./static/build/stats.json) or whatever your path is.

Now you can output html dynamically, using the build’s script names.

<!-- in  handlebars template -->
<script src="/static/build/bundle.js"

if you use a different temlplate engine, well… it’s not like it’s difficult to port, is it?

and how will it work in development?

For the development part, I modified a bit my webpack config :

//webpack.config.js
const isProduction = process.env.NODE_ENV === 'production';
//Later in file
filename: isProduction?'[name].[hash].js':'[name].js',

Then in my server, I just have to set hash = ""; in development

A faster Leap Motion experience

2016-12-05T00:00:00+00:00

Like many, I was really enthousiast when the Leap motion was unveiled some years ago. At last, a company that emphasized on gesture recognition’s main problems : Speed and precision. Like most, I have been disappointed. Still not fast enough, still not precise enough. There is still not much to do in the land of gesture control. Or is there?

Why is Leap Motion so bad?

Don’t get me wrong, the leap motion SDK is great and is a big deal in why it got some initial traction. I’m not pretending I would have made a better job or anything. However, let’s face it :

differencing a blob from a finger is hard as hell

To demonstrate it, let’s imagine this common use case :

I have a leap motion demo, running idle on my monitor. Suddenly, I want to show a customer how it works. "Hey, look at how well it recognises movements" * swipes in front of the box*. Aaaand… Nothing. It takes a few frames for the leap motion to detect what just appeared as a finger. Then a few more frames to get a correct velocity approximation. A swipe of the hand is only half a second long, tops!

So I need my hand to stay stable for half a second, be sure the leap motion detect it, don’t move too fast or too far away or withh too strange positions or it will need to re-detect my hand.

Unusable.

Why do I need fingers?

I just need a gesture! Movement! “User ask me to go right” type of thing. Let’s say I don’t care if it’s a figner, or anything reflecting IR light.

If something moves right above my sensor, it’s probably meaningful.

It might not be your use case. For what I know, it might not be anyone else’s use case. But it’s mine.

Extracting data

The Leap SDK won’t help you here. It knows 2 things :

Raw data
Hands (and associated subsystems : fingers, tools, …)

No “Unidentified light blob” here. That’s a shame, because I imagine they have such a map at some point in their internal filtering, and it’s probably way better than mine.

So let’s get raw data from the Leap.

controller.images()[0];

Not frame.images() because as stated in the doc : * the images obtained with this function can be newer than images obtained from the current frame of tracking data.*

Don’t add unnecessary delay.

Here we discover that my monitor is quite IR-reflective

The first thing we’ll do is rule out all unnecessary data. Computations on pixel maps can be quite heavy and it’s a good idea to reduce the dataset early. I just needed to see movement on the X axis so the first step was to linearize the image.

Due to the large viewing angle, the leap lenses make a fisheye effect on images. We first need to rectify them.

It’s either possible to create a shader for it or a function to rectify the full image. The later is fully explained in the doc.

Now that my image is rectified, I’ll just linearize it :

unsigned char * raw_data = controller.images()[0].data();
unsigned char linear_data[640];
for(int x=0;x<640;x++){
  for(int y=90;y=150;y++){
    //prevent overflow by dividing the value by the number of lines.
    linear_data[x] = raw_data[IMAGE_WIDTH*y+x]/60;
  }
}

That’s a 640 bytes array containing all the info we’ll need.

The raw image, and the result of linearisation at the bottom

There’s more !

You can visualize this greyscale data, it’s already great and precise. There’s another great thing we can get out of the leap : Stereovision. You heard Stereo was hard? How about stereo vision computing on a greyscale 1d array? Let’s try.

Just extract data from controller.images()[1] the same way we just did.

We’ll make a multiplication of our values, offsetted by a STEREO factor. This will naturally filter out dots that are not on our target height. The higher the STEREO diff, the lower the points.

This is NOT fool-proof. Imagine you set a diff of 60 and got 2 fingers distant of 60px: It will be well correlated and maybe not at the target height. However like I said, we’re looking for movement, not spacial interpolation precision.

Read a video Backward (Part 2)

2016-10-06T00:00:00+00:00

In previous post, I stated how I made a video player capable of reading a file from back to front. However I’m quite a beginner in video encoding and I felt there was improvements to be made.

video intra modes are a thing for profesionnals. Most consummer applications are doing well with default mode. Here is what I found.

Benchmark method

I needed a benchmark for my particular use case. Two things should be compared :

Decode speed
Video quality

Speed

Decode speed is easy : Just make the decoder loop and log results. I got pretty consistent results out of those simple lines of code (using stingray’s decoder) :

void cycle(decoder::VideoDecoder* decoder,entities::Video* video ){
  int i,c;
  std::chrono::high_resolution_clock::time_point p(std::chrono::high_resolution_clock::now());

  for (c=0;c<10;c++){
    decoder->decodeAndWrite(*video->buffer);
    for(i=0;i<DECODE_SIZE;i++){
      video->buffer->read();
    }
  }
  using dura = std::chrono::duration<double>;
        auto d = std::chrono::high_resolution_clock::now() - p;
        std::cout << "Decode cycle: "
            << std::chrono::duration_cast<dura>(d/10).count()
            <<" s"
            << std::endl;
}

Running it on a highest quality (-q 2) MJPEG video I got 0.165 s/chunk. (Chunk size of 20) – which maps to 121 fps. Real world speed will be a bit less as the system will be busy doing other things (reading inputs, displaying frames).

Lowering the quality helps a lot, obviously. Encoding my test video with -q 10 makes the bitrate drop from 45kbps to 22kbps.

That’s for the old-but-working MJPEG codec. the interesting thing to note here is correlation between bitrate and decode speed.

-q	bitrate	decode time
2	45596.1	0.165s
10	22591.4	0.123s

Quality

Quality is a bit harder. However as avconv quality options -b, -crf, -q, etc. do not behave consistently at all across codecs. For example, h.264 produce good quality results with -crf 20, while vp8 is really bad with -q 10…

I needed a distortion measurement algorithm to get solid results for comparisons. However, keep in mind that even absolute rate/distortion grade is not a good assessment of your encoding’s quality : It does not capture it as perceived by the human eye.

I did not find a good tool to do it: Manual review it is.

h.264 encoding

libx264 has an intra mode. let’s see what is does :

avconv -y -f image2 -i render/frame-%04d.png -vcodec libx264 -an -crf 20 -g 1 -profile:v high test_stingray.mp4

-crf	bitrate	decode time
10	62021.3	0.10s
20	21206.61	0.090s
40	5725.8	0.0676786

A few thing we can already take for granted :

h.264 decoding is faster than MJPEG for the same bitrate.
for lower bitrates (< 30kb/s), h.264 yields better quality.

My quality measurement is empiric. Just run the tests if you want to be sure.

jpeg2000

the problem with jpeg2000 is it’s not really well supported by ffmpeg. While it provides (on linux) a wrapper to encode a video using libopenjpeg, libavcodec doesn’t decode it out of the box. i had to fiddle a bit to find suitable options.

avconv -y -f image2 -i render/frame-%04d.png -vcodec jpeg2000 -compression_level 50 -an -g 1 -pix_fmt yuv420p j2k_test_stingray.mov

compression_level is an int between 0 (lossless) and ?! – I could not find the max, but anything above 100 is clearly unusable.

According to previous research, file size vs quality should be roughly equivalent to h.264 intra. However I could not test it as the decoder refuses to decode it properly (as do most regular video players…).

vp8

Webm is an open video file format sponsored by google. It’s modern and supports mainly vp8 and vp9 video codecs. Only vp8 is supported natively by ffmpeg so let’s try it

avconv -y -f image2 -i render/frame-%04d.png -codec:v libvpx -qmin 1 -qmax 8 -threads auto -codec:a none webm_stingray.webm

Notice how the -g 1 flag is gone? vp8 is natively pure-infra. Fun fact : If you encode the video using -g 1, it will weight 3 times as much. Go figure…

Results :

-q	bitrate	decode time
1-4	12672.8kbits/s	0.098 s
1-8	6501.6kbits/s	0.090 s

Really good. However there is not correlation between libvpx’s -qmin, -qmax and h.264’s -q.

Interestingly, using vp9 though it reduces file size for another 20%, halves decode speed. Further research shows that encoding consistently uses 4 to 5 times more CPU, and decoding 50% more than vp8. Which makes it ineligible for our use case.

Conclusions

The codec war, while not targetted on our use case, provides us with some interesting tools. Video decode speed and file size can be improved significantly over the venerable MJPEG codec.

Writing a native evdev adapter

2016-09-24T00:00:00+00:00

I’ve recently found myself in need of a library to read evdev events. Usually, the X server will retrieve those events, parse them and send them to the concerned X client. In my case though, I needed xpad events from a gamepad. The only existing way to get them through X is the joystick driver, but it has been designed to control X server with a controller instead of a mouse.

All I wanted was to have a way to fire an event saying :

hey, someone pressed button A!

Quite simple. Except I could not find much literature on how to do it, especially in a way I could implement with nodejs.

There are two documented projects that act as evdev event reader helpers : Libevdev and python-evdev. I heard there’s also a ruby one but did not use it.

I could have written a libevdev wrapper, wut it would have required a deep understanding in libav’s internals to work with async I/O.

I decided to write my own simple basic interface. Here is what I learned in the process.

Linux input architecture.

First thing first, let’s have a look at what we’re facing.

Linux kernel is managing input devices with drivers like xpad. Those devices generate evdev events that are exposed through character devices (which are fifo buffers) located in /dev/input.

Those devices are numbered like : /dev/input/event0, /dev/input/event1 and so on.

They are also available in more human-readable ways in /dev/input/by-path/. A joystick will be exposed as : /dev/input/by-path/<usb_bus_id>-event-joystick

When reading this file, kernel’s input.h tell us we’ll get this data structure:

struct input_event {
struct timeval time;
__u16 type;
__u16 code;
__s32 value;
};

timeval being a structure of two int64 on 64bits systems – for 32bits systems, it would be int32 – representing seconds and microseconds.

So an event is a 24 or 16 bytes datastructures. Note that Evdev may send multiple events at once.

Writing the Adapter

With nodejs Examples

Once you understand the event model, this part is in fact really simple. All you need to do is really to open a buffer and read from it.

Fetching messages

That’s the easy part. Reading a buffer and parsing it’s data isn’t what’s going to keep us up all night.

The sweet part here is we can do it all in native javascript.

First you need to find your device, depending on what you’re looking for.

var device = "/dev/input/by-path/your_device_path";
var options = {
        flags: 'r',
        encoding: null,
        fd: null,
        autoClose: true
      };
fd = fs.createReadStream(device,options)
  .on('data', function(buf){
    var i,j, chunk = self.arch === 64 ?24 :16;
    for (i=0,j=buf.length; i<j; i+=chunk) {
      //parse the buffer
    }
  })
  .on("error",function(e){
    console.error(e);
  });

Now for the parsing, the base principle is to just assign bytes to the structure’s fields.

There is a small gotcha with int64 parsing in javascript. However it has already been solved by others :

var ev = {
     time : {}
   }
   if (this.arch === 64){
     low = buf.readInt32LE(0);
     ev.time.tv_sec = buf.readInt32LE(4) * 4294967296.0 + low;
     if (low < 0) time.tv_sec += 4294967296;
     low = buf.readInt32LE(8);
     ev.time.tv_usec = buf.readInt32LE(12) * 4294967296.0 + low;
     if (low < 0) time.tv_usec += 4294967296;
     offset = 16;
   }else{
     ev.time.tv_sec = buf.readInt32LE(0);
     ev.time.tv_usec = buf.readInt32LE(4);
     offset = 8;
   }
   ev.type = buf.readUInt16LE(offset);
   ev.code = buf.readUInt16LE(offset + 2);
   ev.value = buf.readUInt32LE(offset + 4);

Next steps depends on your goal, but if you want to parse event to assign readable strings like “BTN_A”, “ABS_X”, etc. or generate events from node’s EventEmitter interface, you can reuse my node-evdev project.

Querying evdev

Now if we look at the way libevdev is querying informations, here is what the source tell us (in libevdev.c) :

ioctl(fd, EVIOCGID, &dev->ids);
Constant EVIOCGID is defined in input.h as :
#define EVIOCGID _IOR(‘E’, 0x02, struct input_id)

If we follow the lead, it takes us to :

#define	_IOR(g,n,t)	_IOC(IOC_OUT,	(g), (n), sizeof(t))
#define _IOC(inout,group,num,len) \
	(inout | ((len & IOCPARM_MASK) << 16) | ((group) << 8) | (num))
#define	 IOC_OUT 		0x40000000	 /* copy out parameters */

(cherry-picked from linux kernel’s sys/ioctl.h)

Based on this, the only real problem is to find sizeof(t). The structure we used in libevdev is easy to find. However it’s quite complex and would require some time to be ported to javascript with Uint/Int16/longint… adapters. But for now, we only need the input_id structure, which is once again found in the kernel’s input.h :

input_id {
 __u16 bustype;
 __u16 vendor;
 __u16 product;
 __u16 version;
};

4 Uint16. Easy. However there is still one problem to solve : There is no way to make ioctl calls in native js. I could have used an adapter, but it would be overkill for just a single call. Let’s just make a c++ addon.

** The following sample uses nan for portability**

The implementation mimicks libevdev’s method evdev_new_from_fd(). It’s convenient because we already got a file-descriptor in javascript : We can just pass it through this function to get the peripheral’s infos.

#include <nan.h>
#include <sys/ioctl.h>
#include <linux/input.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>

using v8::FunctionTemplate;
using v8::Handle;
using v8::Object;
using v8::String;
using Nan::GetFunction;
using Nan::New;
using Nan::Set;
using Nan::To;


NAN_METHOD(evdev_new_from_fd) {
  int rc;
  struct input_id id;
  rc = ioctl(To<int>(info[0]).FromJust(), EVIOCGID, &id);
  if(rc == -1 ){
    Nan::ThrowError(strerror(errno));
    //Nan::ThrowError(info[0]);
  }
  v8::Local<v8::Object> size = Nan::New<v8::Object>();
  Nan::Set(size,Nan::New("bustype").ToLocalChecked(), Nan::New<v8::Number>((double)id.bustype));
  Nan::Set(size,Nan::New("vendor").ToLocalChecked(), Nan::New<v8::Number>((double)id.vendor));
  Nan::Set(size,Nan::New("product").ToLocalChecked(), Nan::New<v8::Number>((double)id.product));
  Nan::Set(size,Nan::New("version").ToLocalChecked(), Nan::New<v8::Number>((double)id.version));
  info.GetReturnValue().Set(size);
}

NAN_MODULE_INIT(Init) {
  Nan::Set(target, Nan::New("evdev_new_from_fd").ToLocalChecked(),GetFunction(Nan::New<FunctionTemplate>(evdev_new_from_fd)).ToLocalChecked());
}

NODE_MODULE(ioctls, Init)

I think I will never get over v8’s horrible coding style. apart from it, it’s quite straightforward. We fill in an input_id with ioctl() then just convert it to v8 data structures.

Conclusion

The great thing in having a pure-js event parser is that context-switching in nodejs addons are slow. It also keep the code simple and tidy.

While we need to compile an addon to retrieve the peripheral’s metadata, it’s not always required and still quite simple in the end.

I’ve been using this module for one year now with no problem in my electron

How to read a video Backward (Part 1)

2016-09-21T00:00:00+00:00

I’ve recently faced an unexpected challenge in trying to read (decode & display) a video backward at normal speed. For someone who don’t know shit about video encoding, it looks like it should be as straightforward as calling video.previousFrame()… except it’s not. Let’s see why and how we worked around this to create a new generation of holographic applications.

We made a free software based on this : Check out Stingray for the code!

Some background.

Backward reading

Files are not made to be read backward. It’s a fact in computer science, it has been since computer were big lamps that read punch cards.

As far as my reading got, the best way to go back a few bytes in a file is to close it, reopen it, then offet the reader to our known desired position.

Reading frame 9, then 8, then 7… will then require us to open the file at the begining of frame 9, then reopen it on 8, etc…

Video encoding

Most of today’s video codecs use a similar process for compression. Only the underlying compression algorithm differ. The process is quite similar to images compression. Additionally, recent codecs will use the fact that videos are composed of many images that will often have at least some common content (think : moving foreground on fixed background). h.264, MPEG4, etc… are all taking advantage of this to implement a concept of keyframes (or I frame). A keyframe is defined as a complete image stored within a data stream. Every other frames (P frames) only contains incremental changes based on the previous frame.

This is the reason you sometimes get a few seconds of messed up background with just moving parts : Your device just skipped a keyframe.

For the sake of completeness, some algorithms are using B frames that can rely on forward frames for data reference.

A sequence of video frames. Source : Wikipedia

For this reason, seeking precisely in a video is very slow. Backward seek especially as most frames in a video are effectively P frames.

Draft a solution

So I can’t just make up a decode_previous_frame() function to decode frames in the inverse order. As far as I know, there is only one thing to do : Position the read pointer, batch decode some frames and reorder them in a buffer for a reading routine.

To go from frame 20 to frame 0 :

Decode frames [10..20], revert the array and push to buffer
Decode frames [0..10], same thing

And so on. The first batch can be on display while the second one is decoded. To do so, we will need to find balance for our batch size : larger sizes will please the decoder as it doesn’t have to seek_back() too much (which we know forces it to reopen the file), but a smaller size will reduce initial delay and memory footprint as we won’t need to keep as much images.

How to choose the decode batch size? As small as possible as long as decode speed is enough.

Let’s speed it up then.

Speed up

I’m going with libav here, some differences can be observed with other decoding softwares, but mainly they all work the same.

The video encoding problem at its root is unsolvable: Any frame, to be decoded need any amount of previous frames already decoded, each frame only referencing that it needs the previous one. It means if I want to seek to frame 10 and start decoding, the decoder might need to decode every previous frames up to frame 1. Unacceptable.

Enters MJPEG

MJPEG or Motion-JPEG is simply a concatenation of JPEG-Encoded images. It offer the compression ratio of JPEG, around 1:5 which is not bad but clearly less than modern high-compression codecs like MPEG (can go up to 1:100).

MJPEG has been quite forgotten in recent years with the diffusion of high performance codecs, due to hardware accelerated decoding and web diffusion, respectively nullifying the need for an easy-to-decode format and requiring high bandwidth savings. however it’s still supported mainly for professionnal applications that require a fast random access.

Turns out I didn’t even need to re-code anything to make seek fast on MJPEG videos with libav:

avformat_seek_file([...]);

is fast enough to be negligible for chunks as small as 10 frames.

As for the video size, a HD video of 10 seconds weigh in at 30MB. It’s large but not unmanageable.

For reference, here is the CLI options to encode a MJPEG video using avconv :

avconv -r 25 -y -i "<INPUT_VIDEO>" -g 1 -keyint_min 0 -c:v mjpeg -an -q:v 2 -pix_fmt yuvj420p "<OUTPUT_VIDEO>.mov"

The -keyint_min and -g options seems mandatory to prevent avconv from further optimizing the video.

Reduce delays

So we can now decode the video stream backward. However there is still a delay when we want to switch from forward to backward mode. That’s easily solved using a double buffer :

Decoded frames are put in a front_buffer, waiting to be played
Displayed frames are put in a back_buffer
Once the back_buffer overflows, it’s frames are automatically discarded.

When the user hits the switch_direction button, we just have to switch the buffers from front to back.

You can now have high frequency direction switch without any delay or performance penalty.

Conclusion

We got ourselves a decent interactive video player at no cost. I did not describe the code precisely as :

Most of it is boilerplate (keyboard controls, SDL configuration, etc.)
It’s available under GNU-GPL as a free software initiative by holusion names Stingray.

In next part I will talk about our projects to further improve it using next generation video codecs like MJPEG-2000 and H.264/AVC for better compression.