Phil's Musings

Embeddable and Accessible PuzzleScript Games

2018-09-22T00:00:00+00:00

I ❤️ Puzzle Games.

I want kids to play games that don’t talk down to them, encourage them to figure out game mechanics on their own, and I want non-sighted people to play video games.

Introducing: the puzzlescript package.

Many games have puzzles sprinkled in, but I like ones that are distilled into just solving puzzles.

I first realized how engrossing these were while playing The Witness with friends. One person would control the player but everyone would be solving the puzzle in their head just by watching. We played together for days.

Then, I tried it with kids. It turns out, a 6-year-old grasped the puzzle concepts in The Witness faster than her parents and was explaining to her parents how the puzzles worked.

Then, I started looking for more and stumbled upon increpare’s awesome PuzzleScript.

Unlike other video games that tend to teach kids to memorize facts (like “Carmen San Diego”, “Oregon Trail”, or “Math Blaster”), these encourage kids to think critically, and problem-solve in groups.

Specifically, PuzzleScript games are interesting because the whole game is a text file and the levels are typically pretty small. This means it’s easy for kids to explore but it also makes the games playable by people that can’t see. The reason is that all of the sprites have human-readable names. So we can read those out instead of just showing colored pixels.

Background

Inspiration came from both the tons of easter-eggs found in software (Chrome Dino Game, various Microsoft ones and commandline tools) as well as a desire to get more people in general to play video games.

Goals

get kids in classrooms playing these games
get vision-impaired people playing these games
get 404 pages to have these games when something is broken (like the Chrome Dino Game). Example: this website’s 404 page
get people to play these instead of Sudoku to exercise their brains more

Try it out!

Games can run embedded in a webpage or in a commandline terminal. There are over 200 games to choose from!

Terminal

All you need is node 4 or higher and then run the following:

npm install --global puzzlescript
puzzlescript

See philschatz/puzzlescript for examples.

Embed in a Webpage

<body>
  <table id="the-game-ui"></table>
  <script src="https://unpkg.com/[email protected]/lib/webpack-output.js"></script>
  <script>
    const gameSourceString = '...' // Source code for the game
    const table = document.querySelector('#the-game-ui')
    const engine = new PuzzleScript.TableEngine(table)
    engine.setGame(gameSourceString, 0 /*startLevel*/)
    engine.start()
  </script>
</body>

Visit the demo page to play several games (there are over 200), or see this website’s 404 page for an example of using it in a 404 page.

For Next Time

I would like to go through more details on how the code is organized. But if something does not work or is not clear, create an issue.

More Examples

Mirror Isles (original)

More examples can be found in the README or see the demo page to play them in a browser.

Automatically record HTTPS requests instead of manually creating mock files

2017-05-15T00:00:00+00:00

Creating mock HTTP requests seems to be the go-to method for developers.

But wouldn’t is be easier if you could just point your tests to a server and record the HTTP responses and then play them back?

Now there is fetch-vcr!

Since fetch API is in browsers, there is an easier way to do this in JavaScript!

fetch-vcr will record and play back those HTTP requests for you.

How does it work?

Just load the fetch-vcr package in your tests and it will proxy any calls to fetch(url, options).

Each HTTP response is saved to the filesystem as a cassette (also known as a fixture). The cassette contains all of the response headers and the response body.

Depending on the VCR_MODE it will do the following:

playback: (default) it will load the recorded cassette of the request instead of talking to the server
record: it will talk to the server and save the response to a local file (cassette)
cache: it will try to load the cassette and if the cassette is not found, it will do the same thing as record

Because it saves files to the filesystem, it works slightly differently when used in NodeJS and when used in a browser.

How does it work in NodeJS?

It’s super-simple. When running tests:

Make sure your code runs var fetch = require('fetch-vcr'); instead of var fetch = require('node-fetch');
Record your cassettes by running VCR_MODE=record npm test instead of just npm test
Optionally, change the directory that cassettes are saved to by running fetchVCR.configure({fixturePath: '/path/to/folder/'})

Viola! you have just recorded your HTTP requests.

How does it work in a browser?

For browser tests it is a little bit more complicated because browsers do not save files to the filesystem.

Fortunately, browsers like PhantomJS or Selenium allow you to send data from the browser using alert(msg). There are other ways if you would rather do it differently.

You can use the steps listed above but will need to do the following additional steps:

Pass the VCR_MODE environment variable to the browser
Replace fetchVCR.saveFile(rootPath, filename, contents) => Promise with a function that calls alert(JSON.stringify([rootPath, filename, contents]))
Parse the alerts and save them to disk using fs.writeFile(filePath, contents)
- Note: This code does not run in the browser; it runs in the JS file given to PhantomJS (or Selenium if you are using Selenium)

Check out fetch-vcr for more info.

Introducing a Serverless Issue board for GitHub repositories and organizations!

2016-04-06T00:00:00+00:00

At openstax.org different projects use different ticket trackers because different people have different likes and preferences. We use Trello, Pivotal, GitHub Issues, and Wunderlist. But one thing that is common across all of our open source projects is GitHub.

As we’ve grown and as our projects have started to overlap we’ve realized there is value in having a common place to look and see what’s going on in all of the projects.

gh-board does all the things we need and more. If it doesn’t do something, submit a Pull Request and it will!

Other ticket trackers

Any ticket tracker presents additional friction to use with GitHub:

it’s hidden behind a login
- (when a ticket is linked in our IM client we don’t get a nice preview)
the logins are different so @mentioning people is annoying
you have to remember a different type of markup language
you have to remember to link everything twice
- so people looking at the ticket can get to the code and vice-versa
you have to update the tracker when the Pull Request status changes (created, review, tested, merged, etc)
they don’t show the state of Pull Requests so you then have to click to see what the Pull Request status is
URL’s are difficult to share because frequently the state of the page is not in the URL
- ie which milestones, columns, or other filter criteria are being used

As a developer/tester/UX: you’d still have to check multiple places to stay on top of everything (or hope that your email client doesn’t explode!)

GitHub isn’t perfect either

But GitHub Issues is not without its limitations:

Issues are per-repository (we have 100 repositories)
Milestones are per-repository
Labels are per-repository
It is difficult to add additional metadata to a ticket
There is no easy way to have kanban-style columns

How is this different from other GitHub-based trackers?

It has a few features that other ticket trackers lack (like huboard or waffleio):

open source & free!
you can run it anywhere!
- you can still use vanilla GitHub (nothing to import/export and no vendor lock-in)
real-time collaborative editing of Issues
shows the state of related Issues/Pull Requests
shows CI status and merge conflict status
has charts (burndown, gantt, etc)
keeps track of multiple repositories from different organizations
and productivity-enhancing Easter Eggs!

CI Status and Merge Conflict

CI Status shows up as a green check mark or a red x on the top-right corner of a card
Merge conflicts are shown with a yellow warning and have a diagonal striped background

Real-time Collaborative Editing

Issue Images

If an Issue or Pull Request contains an image then it will be shown in the Issue

Easter Eggs

Plus, it comes with productivity-enhancing easter eggs you can unlock!

Charts

Since it stores all the open and closed tickets locally, we can generate all the fancy charts that other ticket trackers generate.

Burnup chart: it clearly shows when new work is added to a Milestone
Gantt chart: shows when milestones are due and colors the bar based on the status of all the Issues

How does it work?

It:

uses octokat.js and polls the GitHub API for changes
uses the 1st repository in the list to find the column labels and milestones
- columns are defined as labels of the form ## - Column Title or you can specify a regular expression
stores all the Issues and Pull Requests in the browser (thanks to IndexedDB)
- think of it like git clone but for Issues & Pull Requests
searches the IndexedDB to find related issues

Introducing the atom pull-requests package!

2016-04-02T00:00:00+00:00

As a programmer that uses GitHub, Pull Requests are a great way to discuss code but whenever I get feedback on code in a large codebase it is annoying to have to find where that change was. Since I use atom.io as my text editor, I decided to write a plugin that adds a great feature of GitHub (Pull Requests) into my text editor (which also happens to be written by GitHub). And viola! pull-requests.

Whenever you check out a branch that has a Pull Request and open it in GitHub, you’ll see the GitHub comment directly on the line of code inside atom. And to help you find it, the Tree view on the left will show how many comments are in the directory so you can find the file.

Here’s a slightly out-of-date screencast showing the whole process (including installing the plugin):

How’s it made?

It uses the octokat.js npm package and the linter atom package. pull-requests checks if your code is in a git repository (actually, a GitHub one) and then checks if the branch corresponds to a Pull Request in the repository (or the parent repository if this is a forked repository) and pulls out the comments on the changes of a file.

Then, it uses linter to add lint messages on the lines of the files. Since linter supports HTML, pull-requests also converts the comment into HTML (complete with emojis) and adds a link to get back to the Pull Request on GitHub so you can continue discussing.

Hope that helps you!

The Death of Openstax?

2015-06-09T00:00:00+00:00

OpenStax isn’t going anywhere; we have a ton of high-quality content and are revolutionizing textbook publishing for the benefit of students. But here are my thoughts on how it could be made a bit more open (and cheaper to boot!).

Problem

As background, openstax books contain a few pieces:

an editor for creating a part of a book (Aloha) and organizing parts of a book into a Table of Contents
rules for attribution (who authored the book)
a way to convert the book into various formats (ePUB and PDF)
a way to read the book online for free
a way to mix-and-match and create a new book
Ideally, openstax will have a way to allow others to suggest edits to a book

Solution

Non-tech-savvy

Use GitHub to store the book content (book viewer, source for openstax books, and blog post)
Use a browser editor to edit the book
Use a little server to automatically create PDFs and ePubs (travis-ci or philschatz/pdf-ci)
Support attribution automatically (example)
Support “Suggested Edits” to content automatically via GitHub’s “Pull Request” (example)
Support “Derived Copies” of content automatically via GitHub’s “Fork”
Development of code and content using GitHub Issues via philschatz/gh-board

Tech-savvy

All book content is stored directly in GitHub (see sources for examples)
Replace the editor with oerpub/github-book-editor which saves directly to GitHub
Replace the web view with Autogenerated Sites using GitHub (see philschatz.com/books and the various book repositories )
Replace PDF (and ePub) generation with travis-ci or philschatz/pdf-ci (Every time GitHub updates, generate a new PDF)
Support Derived copies using GitHub’s “Fork”
Support “Suggested Edits” (which openstax used to have) via GitHub’s Pull Requests
Support a diff of “Suggested Edits” via GitHub’s Markdown diff view (see blog post on books in GitHub)
Support software (and content) development using something on top of GitHub Issues like huboard or ideally philschatz/gh-board (no server/subscription required)

Free Stuff!

No server costs! (except PDF generation which can be minimized)
autogenerated PDFs for every edition (see GitHub Tags and Releases)
easy contributions from the entire world
- travis-ci can make sure the Markdown is well-formed
human-readable changes
Issue tracking for content in each book
Revisions for each book
More reusability/remixability than you could imagine

The Kink

There is a “kink” in this process which I’d be happy to elaborate on:

Each book needs to be a separate repository (cannot “easily” combine multiple books into 1)
Need to learn MarkDown (specifically kramdown) unless the editor is “smart enough”

That’s how I would “kill” OpenStax with minimal effort and stay true to the openness that has helped OpenStax thrive.

For more projects check out my repositories

Openstax Textbooks Reader on GitHub using Kramdown

2014-07-07T00:00:00+00:00

Over the weekend I converted all of our popular textbooks from our new HTML format to kramdown (a Markdown variant that allows classes) and tossed it on GitHub.

Since gh-pages automatically converts kramdown files to HTML using jekyll, I also created a tiny book reader and linked to it in the book. Click the screenshot to read all the books!

GitHub.com Bonus Features

By using a Markdown variant instead of HTML, there are several things GitHub provides for free.

repo pages will render instead of just showing the markup
links between pages just work
diffs and Pull Requests render and add changebars/colors to make the changes clearer

Clickable Table of Contents

Note: {: ...} is kramdown’s way of adding classes

The markup:

---
layout: page
---

1.  {: .preface} [Preface](contents/m46844.md)
2.  {: .part} Unit 1: Levels of Organization
    1.  {: .chapter} [An Introduction to the...](contents/m45981.md)
        1. [Overview of Anatomy and Physiology](contents/m45983.md)
        2. [Structural Organization of the...](contents/m45985.md)
        3. [Functions of Human Life](contents/m45986.md)

    2.  {: .chapter} [The Chemical Level of...](contents/m45996.md)
        1. [Elements and Atoms: The Building...](contents/m45998.md)
        2. [Chemical Bonds](contents/m46000.md)
        3. [Chemical Reactions](contents/m46004.md)

Rendering in normal GitHub.com

The markdown files render on GitHub.com repo pages (see example page on GitHub)

and the raw kramdown:

---
title: "Anatomy of Selected Synovial Joints"
layout: page
---

<div data-type="abstract" markdown="1">
By the end of this section, you will be able to:

* Describe the bones that articulate together to form selected...
* Discuss the movements available at each joint
* Describe the structures that support and prevent excess movements

</div>

# Articulations of the Vertebral Column

In addition to being held together by the intervertebral discs,
vertebrae also articulate with each other at synovial joints formed
between the superior and inferior articular processes called
**zygapophysial joints**{: data-type="term" :} (facet joints) (see
[link text](m46383.md#fig-ch09_01_02)). These are
plane joints that provide for only limited motions between the
vertebrae. The orientation of the articular processes at these joint
varies in different regions of the vertebral column and serves to
determine the types of motions available in each vertebral region.
The cervical and lumbar regions have the greatest ranges of motions.

![Figure alt text](../resource/fig1.jpg "Figure caption"){: data-title="Figure Title" .half-page :}

Rendered Changes

GitHub will also render the changes of kramdown files instead of showing a diff (see example of deleting a chapter)

Features

unobtrusive design
big next/prev buttons
reading progress bar
keyboard left/right buttons work
Table of Contents shows pages you have read
spinner shows when page is loading
just 300 lines of code
works on any book hosted on cnx.org too! Examples of all the Openstax College books
search button reuses the Table of Contents
does not require building the files for gh-pages (unlike GitBook)

Book Reader Links

reader code (it’s in ./dist because I’m lazy)
reader for Openstax books hosted in GitHub
Openstax College books using the same reader code (loads archive.cnx.org)

Notes

the reader works even if files are stored in places other than /contents (thanks to URI.js)
kramdown does not support <figure> (unless you HTML escape it) so I added attributes to the <img> tag (for the title) and JS converts them to a <figure>
internal links do not work on the OSC cnx.org books because I ran out of time and used <base href="archive.cnx.org">
Jekyll requires all Markdown that is converted to HTML must have a YAML header
I removed autogenerated ids on paragraphs and lists so the kramdown is cleaner (id="fs-id*")

Creating a "DSL" for GitHub's API (rewriting philschatz/octokit.js)

2014-05-25T00:00:00+00:00

I originally wrote philschatz/octokit.js to interact with GitHub’s API using Promises. I started by forking an existing API michael/github and rewrote it using CoffeeScript and jQuery Promises. Then, thanks to contributions, they removed the dependence on Underscore and I removed the dependence on jQuery (and added support for several promise implementations).

But the code was getting too large as people incrementally added features so I thought it was time to rewrite with a focus on implementing as much of GitHub’s API concisely and consistently (with the hope of getting it adopted by GitHub officially).

Given my interest in Programming Languages, I decided to make a “Domain Specific Language” for GitHub (it’s in quotes because technically it’s still just Javascript but read on.

The results are at philschatz/octokat.js.

Necessary Features

This new library needed to:

support ~100% of the GitHub API from the start
have as little code as possible to maintain
work in NodeJS and the browser
have multiple source files
have tons of unit tests
support NodeJS callbacks and Promises

Let me quickly go through each of these and the challenges each presented.

Support ~100% of the GitHub API

This library abstracts all the authentication, status codes, caching, headers, HyperMedia (URL templates), and pagination returned from GitHub.

Abstracting Requests and Responses

Understanding how to form a request and how to parse the response is the bulk of the API; the logic is in src/request.coffee and src/replacer.coffee.

The other part is a chaining function that constructs a valid request.

Constructing Valid Requests

In the original philschatz/octokit.js there was a ton of copy/pasta dedicated to just constructing valid URLs.

Instead, this library has a regular expression that validates all URLs before calling GitHub and constructs objects dynamically through chaining (see src/grammar.coffee).

By reading the documentation at https://developer.github.com/v3/ a developer implicitly constructs a URL and then issues a request by calling one of the verb methods.

For example, to list all comments on an issue convert the documentation URL to the following:

GET /repos/:owner/:repo/issues/:number/comments

    .repos(owner, repo).issues(number).comments.fetch()

Here are 3 ways to list all comments on an issue:

octo = new Octokat()

REPO = octo.repos('octokit/octokit.rb') # for brevity

# Option 1: using callbacks
REPO.issues(1).comments.fetch (err, comments) ->
  console.error(err) if err
  console.log(comments) unless err

# Option 2: using Promises
REPO.issues(1).comments.fetch()
.then (comments) ->
  console.log(comments)

# Option 3: using methods on the fetched Repository object
REPO.fetch()
.then (repo) ->
  # `repo` contains returned JSON and additional methods
  repo.issues(1).fetch()
  .then (issue) ->
    # `repo` contains returned JSON and additional methods
    issue.comments.fetch()
    .then (comments) ->
      console.log(comments)

There are several verb methods to choose from:

.fetch() sends a GET and yields an Object
.read() sends a GET and yields raw data
.create(...) sends a POST and yields an Object
.update(...) sends a PATCH and yields an Object
.remove() sends a DELETE and yields a boolean
.add(...) sends a PUT and yields a boolean
.contains(...) sends a GET and yields a boolean

Pagination and HTML templates

GitHub returns headers for lists of results. These are automatically converted to nextPage, previousPage, firstPage, and lastPage methods on the resulting JSON.

Paging through all the issues on a repository looks like this:

# Create an issues object (but does not fetch anything yet)
ISSUES = octo.repos('octokit/octokit.rb').issues

# Option 1: with callbacks
ISSUES.fetch (err, issues) ->
  console.log(issues)
  issues.nextPage (err, moreIssues) ->
    console.log(moreIssues)

# Option 2: with Promises
ISSUES.fetch()
.then (issues) ->
  console.log(issues)
  issues.nextPage()
  .then (moreIssues) ->
    console.log(moreIssues)

Maintain a Minimal Amount of Code

With this API you are prevented from constructing an invalid URL because every request is validated against the URL_VALIDATOR regular expression before sending the request to GitHub.

Instead of writing largely copy/pasta code that constructs classes I opted for a regular expression that represents the entire GitHub API.

This can be found in src/grammar.coffee.

Works in NodeJS and the Browser

Getting this to work was a bit challenging.

NodeJS and AMD have slightly different syntaxes; enough to require adding in some boilerplate code to convert between the two.

Each of the source files contains 2 lines of boilerplate on the top of the file and the bottom.

Attempt 1

Originally, I had the entire library in a single file.

I was able to have very little boilerplate. Just something like the following:

@define ?= (cb) -> cb((dep) -> require(dep.replace('cs!', '')))

define (require) ->

  foo = require 'foo'

  ...

  module?.exports = Octokat # For NodeJS
  window?.Octokat = Octokat # For browsers
  return Octokat            # For browsers using AMD

This worked well but resulted in a single large file.

Attempt 2

I then split up the library into multiple files and got it working but ran into a problem when trying to build everything into one file.

I spent about a week trying to get jrburke/r.js and jrburke/almond to build a single file but could not get NodeJS, AMD, and the single file to all work at the same time.

Attempt 3

Finally, I opted for compiling the coffee files and concatenating them together with a custom define method similar to what other libraries do (see the require() method in less/less.js.

With this option, the tests run:

on the source coffee files in NodeJS
on the source coffee files in the browser
on the built dist/octokat.js file in the browser

Finally, multiple source files and tests running in all 3 environments!

Fixtures and Recording HTTP Requests

The library currently runs about 400 tests. There are about 120 unique tests, 80 are alternates using callbacks instead of promises, and 200 are the same tests but running in the browser.

In order to not pummel GitHub with duplicate requests I use linkedin/sepia to generate “cassettes” to replay the HTTP requests.

Unfortunately, sepia only runs on NodeJS so I wrote philschatz/sepia.js which plays back (and can record) linkedin/sepia tests.

Support NodeJS Callbacks and Promises

In order to support both callbacks and promises, the asynchronous methods (called verb methods) all support a function as the final argument and return a Promise.

This way, you can always end each line with a callback or with a .then() and the code just works.

The client only returns a Promise if one of the supported Promise libraries are detected (jQuery, angular, Q, or native Promises).

If you use one of those libraries but still prefer to use callbacks just make sure the callback is the last argument and the code will just work^TM.

Removing jQuery, and adding Promises

2014-04-14T00:00:00+00:00

This weekend I decided to remove jQuery from 2 of my libraries: css-polyfills.js and octokit.js.

It was motivated by requests from octokit users and by the horrible performance of css-polyfills.js on large textbooks (took ~1.5 hours).

Refactoring css-polyfills.js

I started with you-might-not-need-jquery and made incremental progress through the codebase. Initially my hope was to decrease the time by 50% but the refactor was only providing marginal improvements, until I hit the following line of code:

$el.find('#' + id)

This uses Sizzle to find an element by id. As soon as I turned it into the following…

document.getElementById(id)

BAM! the time dropped from ~1.5 hours down to 4.5 minutes.

The other annoying bit was that the DOM sometimes returns a live list, meaning if you remove an element, it is directly reflected in the NodeList.

To avoid the problem of iterating over live NodeList and removing some of them I wrapped it in _.toArray().

Refactoring octokit.js

This one was a bit easier.

octokit.js uses 2 features in jQuery: jQuery.ajax() and jQuery.Deferred.

ECMAScript 6 has native support for Promise (and the more fun generators) and it seems the XMLHTTPRequest object is not going away any time soon.

Fortunately, it does not do any DOM manipulation so refactoring this only required changing a few isolated spots in the code.

There were some benefits in making the change as well as drawbacks.

As a reward for being almost a decade ahead of its time, jQuery promises are almost, but not quote the same as ECMAScript Promises. They are constructed differently, have different names for functions, and have more features than the native Promises.

For example:

waiting on all Promises to complete in jQuery requires calling jQuery.when
jQuery promises have .notify() and .progress() methods
distinguish between waiting on a promise and extending one (.done() vs .then())
allow multiple arguments to .resolve()

For octokit this would have been a nice way to provide paginated results as a second argument.

Skeleton Auto-generation

2014-04-07T00:00:00+00:00

In Building CSS skeletons and slots I showed how to style a piece of content based on where it occurs (the @contexts... parameters).

The code can be found at philschatz/skeleton-generator.css.

Here’s a discussion on the machinery needed to realize that.

Types and Templates

First, we need a way to describe the selector for a particular type. Let’s use a subfigure (figure inside a figure) for this example.

A typical figure would have styling like:

figure {
  border: 1px solid blue;
  figcaption {
    color: green;
  }
}

When split into slots and skeletons (let’s ignore @kind and @part for now) it would look like:

// The "skeleton":
figure {
  #content>#figure>.style();
  figcaption {
    #content>#figure>.caption();
  }
}

// The "slots":
#content {
  #figure {
    .style()    { border: 1px solid blue; }
    .caption()  { color: green; }
  }
}

Now, we have something to work with. To make a subfigure have a yellow border, the CSS would look something like:

figure {
  figure {                // Note: it's inside another figure
    border: 1px solid yellow;
  }
}

and the skeleton and slots would look like:

// The "skeleton" for a subfigure:
figure {
  figure {
    #content>#figure>.style(figure); // the arg denotes that this is
                                     // _inside_ another `figure`
    figcaption {
      #content>#figure>.caption(figure);
    }
  }
}
// The "slots" for a subfigure:
#content {
  #figure {
    .style(figure) { border: 1px solid yellow; }
  }
}

Now, we could write all possible permutations of figures, notes, tables, examples, etc in the skeleton file but that would be tedious. Instead, we can autogenerate them using the ... list expander in LessCSS.

First, we need to break up the figure into a couple pieces:

the selector that distinguishes a figure from a .note or a .example
the template of the “structure” inside a figure (it has a .style() and .caption())

For each @type (figure, note, table) the selector can be defined as:

#skeleton {
  .selector(figure) {
    figure { .x-template-helper(figure); }
  }
}

Assuming .x-template-helper does something (for now), this would allow us to create:

figure {
  figure {
    ...         // figures all the way down!
  }
}

… since the only thing this mixin defines is the selector figure { ... } (so it can be nested).

Next, we need to define what can be inside a figure (or subfigure). This is defined by the #skeleton>.template(figure) mixin. We can define it as:

#skeleton {
  .template(figure) {
    // All figures have a style and may have a caption.
    #content>#figure>.style();

    figcaption { #content>#figure>.caption(); }
  }
}

This says a figure has a .style() and a style on the caption.

So far we have defined both what a figure is (the .selector(@type)) and what it can contain (the .template(@type)).

Now we need to combine these to generate the expected CSS:

figure {
  figure {
    border: 1px solid yellow;
  }
}

To do it, we need to build a bit more machinery first!

List evaluation in LessCSS

There are several @types we need to permute. In this example there are only figures, but for a book there will be several types (ie figure, note, table, example).

The generator needs to iterate over all these types so for now let’s put them in a variable.

@all-types: figure, note, table, example;

LessCSS has a way way of going through and recursively evaluating a list:

.recursive() { } // base case
.recursive(@head; @tail...) {
  content: @head;
  .recursive(@tail...); // The `@tail` list is expanded to
                        // multiple arguments
}

Using this structure and the .selector(@type) and .template(@type) mixins we can create the following:

.build-children(@type; @rest...) {
  .selector(@type);
  // The recusion step is in `.x-template-helper`
}

// This mixin was "assumed" above but is defined here.
// It is used in the `.selector()` mixin.
.x-template-helper(@type) {
  .template(@type);
  // Recurse!
  .build-children(@all-types...);
}

In order to prevent an infinite loop, we define a @max-depth and add a @depth param to the mixins:

.build-children(0; ...) { } // base case
.build-children(@depth; @type; @rest...) {
  .selector(@type);
  .build-children(@depth; @rest...);
}

.x-template-helper(@type) {
  .template(@type);
  .build-children((@depth - 1); @all-types...);
}

Now, when .build-children(2; @base-types...) is called, we get the following:

figure {
  border: 1px solid blue;       // Recall figures have a blue border
  figcaption {
    color: green;               // Recall figure captions are green
  }
  figure {
    border: 1px solid yellow;   // Recall subfigures have a
                                // yellow border
  }
}

Which is the desired result.

Custom classes

So far, we’ve ignored the @kind and @part parameters to mixins. In this section we can add them back in.

The @part parameter describes which part of the book the content is in. Some examples are preface, chapter, or appendix. The @kind parameter describes the custom class on a piece of content. These are specific to a book (.how-to for physics or .timeline for a history book).

To handle the @part (preface, chapter, appendix, etc) we merely need to add a @part parameter to .build-children().

To handle the @kind parameter we need to let the skeleton-generation code know what are all the possible classes (@kind) for a given type (figure, example, section, etc).

To do this, we add a “function” mixin to the #content namespace which will “return” a list of classes for each type. Because these are specific to each book they are defined as a #content.kinds(@type) mixin that “returns” by setting a @return variable.

For example, we could have several classes on a figure: .full-widthand .half-width.

The expected CSS would be:

figure.full-width { width: 100%; }
figure.half-width { width: 50%;  }

To define these for the skeleton-generator and the add styling in the slots, it would look like:

#content {
  .kinds(figure) {
    // all the possible classes on a `figure` for this book:
    @return: 'full-width', 'half-width';
  }

  #figure {
    .style('full-width') { width: 100%; }
    .style('half-width') { width: 50%; }
  }
}

Then, we can add a @kind parameter to the definition of .selector(@type) and change the corresponding definitions of .build-children() and .x-template-helper().

Custom classes in `@contexts...`

So far, the @contexts... arguments only contain a list of types. For example, styling any exercise inside an example inside a chapter is done like this:

#content {
  #exercise {
    .style(any; chapter; example) { ... }
  }
}

To style an exercise inside a .how-to example we need a bit more information; namely the class .how-to. To do this, we can use the #content>.kinds(@type) mixin defined above when permuting.

It looks something like:

.x-helper-base-types(@depth; @contexts; @type; @rest...) {
  .selector(@type; @depth; any; @contexts...);

  // Loop for any custom `@kind`s
  #content>.kinds(@type);
  @kinds: @return;
  .x-helper-custom-types(@depth; @contexts; @type; @kinds...);

  // recurse
  .x-helper-base-types(@depth; @contexts; @rest...);
}

Now, we can style a figure based on:

the class
the context @types
the context @kinds (classes)

Optimizing

Checking all possible combinations of @types is computation and memory intensive. Fortunately, we can short-cut several of these. For example, a figure can never contain a section or an example

Instead of using a global @all-types for recursion, we can recurse based on valid children by defining a mixin that “returns” the valid children.

This would look like:

#skeleton {
  #content {
    .children(figure) { @return: figure, table; } //style subfigures
                                                  //and tables
                                                  //inside a figure
    // Defines the "root" children for a piece of content
    .children(none) { @return: table, figure, exercise, example; }
  }
}

This mixin is used in .x-helper-base-types to restrict the list of children that are permuted.

Now, we can quickly style a figure based on:

the class
the context @types
the context @kinds (classes)

Building CSS skeletons and slots

2014-03-16T00:00:00+00:00

Simple Slots and Skeletons

We are transitioning from 1 output format for CSS (Docbook HTML) to multiple HTML formats (PDF, EPUB, web).

To reuse the same CSS we split the selectors and rules into mixins that are namespaced based on their logical part in the book. Instead of a .note { color: blue; } we split it into 2 parts:

// The skeleton:
.note { #content>#note>.style(); }

// and the slot:
#content {
  #note {
    .style() { color: blue; }
  }
}

This allows us to reuse the logical styling when the HTML format changes.

Now, styling a book merely involves filling in the right slots.

Slot parameters

Custom classes

Often we have custom “features” in a book. These are often written and as notes or sections with a custom class name.

To style these, every slot has a @kind parameter as the 1st argument to the mixin. Here is an example of a “How To” feature:

#content {
  #note {
    .style('how-to') { color: orange; }
  }
}

Parts of a book

But sometimes it is necessary to style the feature differently based on which part of the book it is in. Most commonly this is used for numbering. For example, a Figure in a chapter would be labeled Figure 4.3 but in an appendix it would be labeled Figure A3.

To support this, the 2nd argument is the @part. For most slot definitions this will just be any.

Contexts

When styling a book, it is often important to know what an element is in. For example, a worked-out example is actually an exercise inside an example. It should not be numbered (since it is not a homework problem) and might be rendered with text other than Problem and Solution.

To handle this case, the final arguments to a mixin are @contexts....

Here is how to style an exercise inside an example:

#content {
  #exercise {
    .style(any; @part; example) { color: green; }
  }
}

This makes any exercise (regardless of class) in any @part that occurs inside an example green.

If you wanted to make any exercise inside a how-to example yellow you would write:

#content {
  #exercise {
    .style(any; @part; example; 'how-to') { color: yellow; }
  }
}

Similarly, any exercise in an example in the review-questions section at the end of a chapter would be something like:

#content {
  #exercise {
    .style(any; chapter; end-of; section; 'review-questions'; example) { color: aqua; }
  }
}

That’s about it for the intro to slots and skeletons. I should have another post on the namespace organization of a book and a post on how the machinery needed to get the @contexts... parameter to work.

Pattern matching with LESS (CSS)

2014-03-08T00:00:00+00:00

Background

Pattern-matching mixins is the distinguishing characteristic that separates Less from other CSS preprocessors like Stylus and Sass.

Let the compiler do what it’s good at: match templates and error early (when one is missing).

LESS is a beautiful language (like CSS) precisely because it is not Turing-complete. Instead, it explores just how far you can take a language with those restrictions.

For example, you can provide conditionals using the pattern-matching when keyword instead of relying explicit control flow like if then else. You can match arguments to mixins either by value or with ... (like Ruby, CoffeeScript, and recent versions of Java).

This frees the author up to put conditional styles elsewhere in a file (much like XSLT templates). Similarly, mixins whose arguments match are always applied. This can initially be an annoying feature but for styling textbooks this is central in our design.

Case Study: Making Textbooks

In textbooks we need to style elements differently depending on where they are (context is crucial).

A subfigure (figure > figure) should render differently than a figure.

A table should be numbered differently when it is in a chapter (ie Table 4.3) than when it is in an appendix (Table A3).

Or take problems and solutions: a problem and solution at the end of a chapter should be “treated” differently than an example with a worked out problem and solution. The former should be numbered (and solution should be moved to the back of a textbook) while the latter should be included in the example (since it is a worked-out example).

Or take a definition of a term. If an equation is used then it should not be numbered; referring to an equation inside a definition does not make a whole lot of sense.

In each of these cases it is the context that is important when deciding how to style an element.

Additionally, the HTML elements differ for different outputs; online the CSS selectors are different than for a single-page book or a multi-page EUPB.

Solution: Slots and Skeletons

To handle multiple HTML formats (EPUB, online, PDF) we came up with the idea of separating the CSS selectors (skeleton) from the rules (slots).

Each book contains a very similar HTML structure (depending on the output format; PDF, EPUB, online) but very different styling (fonts, colors, numbering schemes) so we split the CSS into 2 types of files: slots for the styling and skeletons for the HTML structure and linked them together using a namespaced tree of mixins that represent the logical structure of a book.

In this way a page header would be defined as (the slot):

#page>#top>.outside() { content: 'Chapter 1'; } and used as (the _skeleton_):

@page:left { @top-left: #page>#top>.outside(); }

This allows us to separate the styling (slots) from the content (skeleton).

Context

Getting back to pattern matching; the context of an element (ie “Exercise”) is important for styling that element. To accomplish this, we created rules for defining mixins and special “generators” for all the possible permutations of contexts.

In our system a mixin has roughly the following signature: #logical-type>.mixin(@kind; @part; @contexts...) { }.

Let’s go through the various parts:

#logical-type roughly corresponds to the structural element (and data-type attribute). Some examples are exercise, note, term, example
.mixin(...) is the kind of slot (.style(...) for the whole element, .title(...) for the title, .numbering(...) for how to number the element, etc)
@kind corresponds to a specific class for a particular book (a science book may have experiment while a history book may have did-you-know)
@part represents which part of a book the element occurs in (preface, chapter, appendix, any) and is largely used for numbering an element
@contexts... represents what this logical-type occurs inside (and where all the fun happens). Some examples would be #figure>.style(any; any; figure) { } (a subfigure), or #table>.style(any; any; glossary) (a table inside a glossary)

This gives us the flexibility to style the content of a book based on the logical parts (using namespaces) and the context it occurs in (using @contexts...) and have it work for different output formats (by having different skeleton files).

To accomplish the @contexts... bit we created some mixins that generate all permutations of different contexts. This is accomplished by adding list-expansion to LESS which allows us to expand a list when calling a mixin. For example:

.context-expander(@head; @tail...) {
  .mixin-call(@tail...);        // <-- The new piece added to LESS
}

Note: For most of these permutations, no mixins will match so they will be omitted from the generated CSS.

The result is a single skeleton file for each output format (EPUB, PDF, online) and a slots file (~100 lines) for each book.

css-polyfills for Books

2014-03-07T00:00:00+00:00

“Why CSS instead of Javascript?””

Wouldn’t it be great if authors could customize their books without having them write (or run) arbitrary JavaScript? This post shows a way to do it.

Our Problem

Our books end up being published in various formats with various support for CSS.

We use Docbook for PDFs partly because we need to move content around (ie collating exercises at the end of a chapter, making an index) and XSLT provides a way to move XML around.

Unfortunately, this means 4 things:

developers need to learn XSLT
we must regression-test all of our books whenever we fix a bug or add a feature
CSS for the PDF is different for ePUB and online
numbering things like exercises is different in a PDF than online

Fortunately, there are a few W3C Drafts that help fill in some of the gaps: Generated Content for Paged Media and CSS3 Generated and Replaced Content Module.

Intersection of Some CSS Features:

Feature	EPUB2	Browsers	PrinceXML (PDF)
`::before`	no	yes	yes
`counter-increment:`	no	yes	yes
`content:`	no	partial	yes
`target-text()`	no	no	yes
`page-break-*:`	no	no	yes
`move-to:`	no	no	no
`::outside::before`	no	no	no
`:has()`	no	no	no

To replace Docbook and have one CSS file to style the various formats we need to support all of these features and note a few differences:

PDF is generated using a single large HTML file (CSS needs to operate on all chapters)
ePUB needs to be chunked into multiple HTML files (ideally using CSS page-break-*)
Online, a single HTML file can be viewed outside the context of a book

Browsers

Ideally, we would be able to get access to all of these unsupported selectors and rules using the CSS Document Object Model but browsers only expose the selectors and rules they understand.

The Solution

Enter CSS-Polyfills.

The project uses LessCSS and jQuery to parse the CSS file and “bake” the changes into the HTML.

With it you can do things that are not possible using CSS supported by browsers. For example, you can style an element based on children inside:

.note:has(> .title) { /* Give it a fancier border */ }

Or, you can automatically generate a glossary at the and of a chapter based on terms in the chapter:

.term > .definition { move-to: glossary-area; }
.chapter:after {
  content: pending(glossary-area);
}

You can even style links depending on the target:

a[href] {
  // Use x-target-is as a switch for which link text to use
  content: x-target-is(attr(href), 'figure')   'See Figure';
  content: x-target-is(attr(href), 'table')    'See Table';
  content: x-target-is(attr(href), '.example') 'See Example';
}

In another post, I’ll go over some of the “Freebies” that come out of this project like CSS Coverage and CSS+HTML Diffs for regression testing.

Bonuses

By parsing the CSS file and “baking” the styles into the HTML there are a few “freebies” that come out.

Easy CSS Coverage

As a free perk, you can easily generate Coverage data for your CSS files by transforming a HTML and CSS file from the commandline and filtering stderr.

HTML+CSS Diffs

To do regression tests on books we merely need to generate the “baked” HTML file twice, once with the old CSS and once with the new CSS (all the styles are “baked” into style="..." attributes). Then, a quick XSLT file can compare the two and generate a version of the page with <span> tags marking the places where styling changed.

See https://github.com/philschatz/css-diff.js for a package that does this.

HTML+CSS Diffs

2014-03-02T00:00:00+00:00

CSS Diffs

Our textbooks are frequently hundreds of pages long and use a single CSS file, so making a CSS change can change content in unexpected places.

Again, CSS Polyfills and a little XSLT file makes this easy.

CSS-Diff takes an HTML and CSS file and produces an HTML file with all the styling “baked” in. Then, the provided XSLT file can compare 2 “baked” HTML files and inject <span> tags whenever the styles differ.

See the CSS-Diff project to run it from the commandline.

css-polyfills.js do more with CSS!

2013-12-10T00:00:00+00:00

I often hear “CSS is meant to style and HTML should describe content”, but CSS alone is not enough to make textbooks.

Sometimes you need to:

style an element based on what’s inside (notes containing a title)
change link text based on the target (See Figure 2a vs See Table 4.3)
move elements somewhere else in the book (answers to the back)

Well, now there is css-polyfills.js, based on some great CSS3 specs, Sizzle, and selector-set.

Simple Examples (from Bootstrap)

With css-polyfills.js you can do things that are not possible using CSS in browsers. Bootstrap’s Dismissable Alerts require adding a special class on the alert if it contains a close button. This can be accomplished without adding the alert-dismissable class by using :has:

.alert:has(> .close) { /* Styles for `.alert-dismissable` */ }

Bootstrap Modals and Input Groups require adding wrapper elements in order to style properly; with nested ::before and ::outside these are unnecessary.

You can construct the 2 additional elements (.modal, .modal-dialog) around .modal-content using the following CSS:

.modal-content                  { ... }
.modal-content::outside         { /* Styles for `.modal-dialog` */}
.modal-content::outside::outside{ /* Styles for `.modal` */ }

More Examples

You can even style links depending on the target:

a[href^="#"] {
  // Use target-is as a guard for which link text to use
  content: target-is(attr(href), 'figure')   'See Figure';
  content: target-is(attr(href), 'table')    'See Table';
  content: target-is(attr(href), '.example') 'See Example';
}

Or, you can move elements down the page (ie collate footnotes at the bottom of a wikipedia article):

.footnote { move-to: footnotes-area; }
#footnotes {
  content: pending(footnotes-area);
}

You can also create the linkable footnotes on wikipedia by keeping the references near the content and use CSS to create links and move the references to the bottom of the page (See Clickable Footnotes).

Of course, there’s much more that can be accomplished using css-polyfills.js; check the README.md for more details.

CSS Coverage for Javascript Unit Tests

2013-12-01T00:00:00+00:00

There are many CSS coverage projects but none plug directly into JS unit tests, instrumenting at the same time as the JavaScript coverage.

Assuming you have BlanketJS and Mocha tests, just grab css-coverage.js and add the following into the test harness HTML file :

<link rel="stylesheet/coverage" href="path/to/file.less" />
<script src="css-coverage.js"></script>

Less.js + Mocha + BlanketJS

The coverage script uses the less.tree AST to parse all the selectors and the debugInfo data to get line numbers. It adds a testdone hook to Mocha that runs all the selectors on the page. Then, the line number and coverage counts are given to BlanketJS so LESS/CSS files are included in the coverage report.

Example Screenshot

Check out the Mocha + BlanketJS demo and the github repo for more!

Other Features

Loading LESS files using RequireJS is supported with 0 additional work
Instrumented less files that import other files are automatically instrumented

Building an eBook using GitHub

2013-06-03T00:00:00+00:00

At Connexions we build free textbooks. To do it we have roughly 4 services:

Published Repository (Published Books)
Authoring Repository (Drafts)
Generate exports (EPUB, PDF)
Website

As someone who uses GitHub and Travis daily, I wondered how much of this GitHub (and Travis) can do for us. It turns out, quite a bit. If each book is a repository whose contents is an Unzipped EPUB then amazing things are possible.

Book Publishing = GitHub Repo + GitHub Pages + “Download as Zip” + Service Hooks

the master branch is the published version of the book
tags are various editions of a book
gh-pages branch can be used as a book reader (Website)
“Download as Zip” is the EPUB version of the book
The editor can “save” via the GitHub API
Travis-CI can be used to generate a PDF (and “push” to AWS or some other place)

So, I took our editor and made it save EPUB files to GitHub and viola, a book editor using GitHub!

the WYSIWYG Book Editor and Demo
the PDF-CI server and demo that generates PDFs on commit
example of the book reader using gh-pages and epub.js

Introducing octokit.js

2013-03-10T00:00:00+00:00

octokit.js is a JavaScript client that interacts with GitHub using their API. Try it out with the interactive demo!

Some unique features:

works in a browser or in NodeJS
uses Promises
written in CoffeeScript
Mocha unit tests that actually talk to GitHub (and run in the browser or commandline)
Code Coverage using BlanketJS (that runs in the browser or commandline)
supports Harmony Generators

API Support:

all repo operations (including create/remove)
Teams, Organizations, and Permissions
eTags for caching results
Notifications for rate limit changes
Committing multiple files at once

Finding a test and coverage framework that works both in a browser and on the commandline was a bit challenging but I’ll return to that in another post.

This library also uses jQuery.ajax and jQuery.Deferred.

Phil's Musings

Embeddable and Accessible PuzzleScript Games

Background

Goals

Try it out!

Terminal

Embed in a Webpage

For Next Time

More Examples

Mirror Isles (original)

Automatically record HTTPS requests instead of manually creating mock files

How does it work?

How does it work in NodeJS?

How does it work in a browser?

Introducing a Serverless Issue board for GitHub repositories and organizations!

Other ticket trackers

GitHub isn’t perfect either

How is this different from other GitHub-based trackers?

Related Issues and Pull Requests

CI Status and Merge Conflict

Real-time Collaborative Editing

Issue Images

Easter Eggs

Charts

How does it work?

Introducing the atom pull-requests package!

How’s it made?

The Death of Openstax?

Problem

Solution

Non-tech-savvy

Tech-savvy

Free Stuff!

The Kink

Openstax Textbooks Reader on GitHub using Kramdown

GitHub.com Bonus Features

Clickable Table of Contents

Rendering in normal GitHub.com

Rendered Changes

Features

Book Reader Links

Notes

Creating a "DSL" for GitHub's API (rewriting philschatz/octokit.js)

Necessary Features

Support ~100% of the GitHub API

Abstracting Requests and Responses

Constructing Valid Requests

Pagination and HTML templates

Maintain a Minimal Amount of Code

Works in NodeJS and the Browser

Attempt 1

Attempt 2

Attempt 3

Fixtures and Recording HTTP Requests

Support NodeJS Callbacks and Promises

Removing jQuery, and adding Promises

Refactoring css-polyfills.js

Refactoring octokit.js

Skeleton Auto-generation

Types and Templates

List evaluation in LessCSS

Custom classes

Custom classes in @contexts...

Optimizing

Building CSS skeletons and slots

Simple Slots and Skeletons

Slot parameters

Custom classes

Parts of a book

Contexts

Pattern matching with LESS (CSS)

Background

Case Study: Making Textbooks

Solution: Slots and Skeletons

Context

css-polyfills for Books

“Why CSS instead of Javascript?””

Our Problem

Browsers

The Solution

Custom classes in `@contexts...`