Practicing Ruby

Safely evaluating user-defined formulas and calculations

Thu, 10 Sep 2015 00:00:00 +0000

This article was written in collaboration with Solomon White (@rubysolo). Solomon is a software developer from Denver, where he builds web applications with Ruby and ENV.JAVASCRIPT_FRAMEWORK. He likes code, caffeine, and capsaicin.

Imagine that you’re a programmer for a company that sells miniature zen gardens, and you’ve been asked to create a small calculator program that will help determine the material costs of the various different garden designs in the company’s product line.

The tool itself is simple: The dimensions of the garden to be built will be entered via a web form, and then calculator will output the quantity and weight of all the materials that are needed to construct the garden.

In practice, the problem is a little more complicated, because the company offers many different kinds of gardens. Even though only a handful of basic materials are used throughout the entire product line, the gardens themselves can consist of anything from a plain rectangular design to very intricate and complicated layouts. For this reason, figuring out how much material is needed for each garden type requires the use of custom formulas.

MATH WARNING: You don’t need to think through the geometric computations being done throughout this article, unless you enjoy that sort of thing; just notice how all the formulas are ordinary arithmetic expressions that operate on a handful of variables.

The following diagram shows the formulas used for determining the material quantities for two popular products. Calm is a minimal rectangular garden, while Yinyang is a more complex shape that requires working with circles and semicircles:

In the past, material quantities and weights for new product designs were computed using Excel spreadsheets, which worked fine when the company only had a few different garden layouts. But to keep up with the incredibly high demand for bespoke desktop Zen Gardens, the business managers have insisted that their workflow become more Agile by moving all product design activities to a web application in THE CLOUD™.

The major design challenge for building this calculator is that it would not be practical to have a programmer update the codebase whenever a new product idea was dreamt up by the product design team. Some days, the designers have been known to attempt at least 32 different variants on a “snowman with top-hat” zen garden, and in the end only seven or so make it to the marketplace. Dealing with these rapidly changing requirements would drive any reasonable programmer insane.

After reviewing the project requirements, you decide to build a program that will allow the product design team to specify project requirements in a simple, Excel-like format and then safely execute the formulas they define within the context of a Ruby-based web application.

Fortunately, the Dentaku formula parsing and evaluation library was built with this exact use case in mind. Just like you, Solomon White also really hates figuring out snowman geometry, and would prefer to leave that as an exercise for the user.

First steps with the Dentaku formula evaluator

The purpose of Dentaku is to provide a safe way to execute user-defined mathematical formulas within a Ruby application. For example, consider the following code:

require "dentaku"

calc = Dentaku::Calculator.new
volume = calc.evaluate("length * width * height", 
                       :length => 10, :width => 5, :height => 3)

p volume #=> 150

Not much is going on here – we have some named variables, some numerical values, and a simple formula: length * width * height. Nothing in this example appears to be sensitive data, so on the surface it may not be clear why safety is a key concern here.

To understand the risks, you consider an alternative implementation that allows mathematical formulas to be evaluated directly as plain Ruby code. You implement the equivalent formula evaluator without the use of an external library, just to see what it would look like:

def evaluate_formula(expression, variables)
   obj = Object.new

   def obj.context
     binding
   end

   context = obj.context

   variables.each { |k,v| eval("#{k} = #{v}", context) }
   eval(expression, context)
end

volume = evaluate_formula("length * width * height",
                  :length => 10, :width => 5, :height => 3) 

p volume #=> 150

Although conceptually similar, it turns out these two code samples are worlds apart when you consider the implementation details:

When using Dentaku, you’re working with a very basic external domain specific language, which only knows how to represent simple numbers, variables, mathematical operations, etc. No direct access to the running Ruby process or its data is provided, and so formulas can only operate on what is explicitly provided to them whenever a Calculator object is instantiated.
When using eval to run formulas as Ruby code, by default any valid Ruby code will be executed. Every instantiated object in the process can be accessed, system commands can be run, etc. This isn’t much different than giving users access to the running application via an irb console.

This isn’t to say that building a safe way to execute user-defined Ruby scripts isn’t possible (it can even be practical in certain circumstances), but if you go that route, safe execution is something you need to specifically design for. By contrast, Dentaku is safe to use with minimally trusted users, because you have very fine-grained control over the data and actions those users will be able to work with.

You sit quietly for a moment and ponder the implications of all of this. After exactly four minutes of very serious soul searching, you decide that for the existing and forseeable future needs of our overworked but relentlessly optimistic Zen garden designers… Dentaku should work just fine. To be sure that you’re on the right path, you begin working on a functional prototype to share with the product team.

Building the web interface

You spend a little bit of time building out the web interface for the calculator, using Sinatra and Bootstrap. It consists of only two screens, both of which are shown below:

People who mostly work with Excel spreadsheets all day murmur that you must be some sort of wizard, and compliment you on your beautiful design. You pay no attention to this, because your mind has already started to focus on the more interesting parts of the problem.

SOURCE FILES: app.rb // app.erb // index.erb // materials.erb

Defining garden layouts as simple data tables

With a basic idea in mind for how you’ll implement the calculator, your next task is to figure out how to define the various garden layouts as a series of data tables.

You start with the weight calculations table, because it involves the most basic computations. The formulas all boil down to variants on the mass = volume * density equation:

This material weight lookup table is suitable for use in all of the product definitions, but the quantity value will vary based both on the dimensions of the garden to be built and the physical layout of the garden.

With that in mind, you turn your attention to the tables that determine how much material is needed for each project, starting with the Calm rectangular garden as an example.

Going back to the diagram from earlier, you can see that the quantity of materials needed by the Calm project can be completely determined by the length, width, height, and desired fill level for the sandbox:

You could directly use these formulas in project specifications, but it would feel a little too low-level. Project designers will need to work with various box-like shapes often, and so it would feel more natural to describe the problem with terms like perimeter, area, volume, etc. Knowing that the Dentaku formula processing engine provides support for creating helper functions, you come up with the following definitions for the materials used in the Calm project:

With this work done, you turn your attention to the Yinyang circular garden project. Even though it is much more complex than the basic rectangular design, you notice that it too is defined entirely in terms of a handful of simple variables – diameter, height, and fill level:

As was the case before, it would be better from a product design perspective to describe things in terms of circular area, cylindrical volume, and circumference rather than the primary dimensional variables, so you design the project definition with that in mind:

To make the system easily customizable by the product designers, the common formulas used in the various garden layouts will also be stored in a data table rather than hard-coding them in the web application. The following table lists the names and definitions for all the formulas used in the Calm and Yinyang projects:

Now that you have a rough sense of what the data model will look like, you’re ready to start working on implementing the calculator program. You may need to change the domain model at some point in the future to support more complex use cases, but many different garden layouts can already be represented in this format.

SOURCE FILES: calm.csv // yinyang.csv // materials.csv // common_formulas.csv

Implementing the formula processor

You start off by building a utility class for reading all the relevant bits of project data that will be needed by the calculator. For the most part, this is another boring chore – it involves nothing more than loading CSV and JSON data into some arrays and hashes.

After a bit of experimentation, you end up implementing the following interface:

p Project.available_projects 
#=> ["calm", "yinyang”]

p Project.variables("calm") 
#=> ["length", "width", "height”]

p Project.weight_formulas["black sand"] 
#=> "quantity * 2.000”

p Project.quantity_formulas("yinyang")
          .select { |e| e["name"] == "black sand" } #=>
# [{"name" => "black sand", 
#    "formula" => "cylinder_volume * 0.5 * fill", 
#    "unit" => "cu cm”}]

p Project.common_formulas["cylinder_volume"]
#=> "circular_area * height”

Down the line, the Project class will probably read from a database rather than text files, but this is largely an implementation detail. Rather than getting bogged down in ruminations about the future, you shift your attention to the heart of the problem – the Dentaku-powered Calculator class.

This class will be instantiated with the name of a particular garden layout and a set of dimensional parameters that will be used to determine how much of each material is needed, and how much the entire garden kit will weigh. Sketching this concept out in code, you decide that the Calculator class should work as shown below:

calc = Calculator.new("yinyang", "diameter" => "20", "height" => "5")

p calc.materials.map { |e| [e['name'], e['quantity'].ceil, e['unit']] } #=>
# [["1cm thick flexible strip", 472, "sq cm"],
#  ["granite slab", 315, "sq cm"],
#  ["white sand", 550, "cu cm"],
#  ["black sand", 550, "cu cm"]]

p calc.shipping_weight #=> 4006

With that goal in mind, the constructor for the Calculator class needs to do two chores:

Convert the string-based dimension parameters provided via the web form into numeric values that Dentaku understands. An easy way to do this is to treat the strings as Dentaku expressions and evaluate them, so that a string like "3.1416" ends up getting converted to a BigDecimal object under the hood.
Load any relevant formulas needed to compute the material quantities and weights – relying on the Project class to figure out how to extract these values from the various user-provided CSV files.

The resulting code ends up looking like this:

class Calculator
  def initialize(project_name, params={})
    @params = Hash[params.map { |k,v| [k,Dentaku(v)] }]  #1

    @quantity_formulas = Project.quantity_formulas(project_name)  #2
    @common_formulas   = Project.common_formulas
    @weight_formulas   = Project.weight_formulas
  end

  # ...
end

Because a decent amount of work has already been done to massage all the relevant bits of data into exactly the right format, the actual work of computing required material quantities is surprisingly simple:

Instantiate a Dentaku::Calculator object
Load all the necessary common formulas into that object (e.g. circular_area, cylinder_volume, etc.)
Walk over the various material quantity formulas and evaluate them (e.g. "black sand" => "cylinder_volume * 0.5 * fill")
Build up new records that map the names of materials in a project to their quantities.

A few lines of code later, and you have a freshly minted Calculator#materials method:

# class Calculator

  def materials
    calculator = Dentaku::Calculator.new #1

    @common_formulas.each { |k,v| calculator.store_formula(k,v) }  #2
    
    @quantity_formulas.map do |material|
      amt = calculator.evaluate(material['formula'], @params) #3

      material.merge('quantity' => amt) #4
    end
  end

And for your last trick, you implement the Calculator#shipping_weight method.

Because currently all shipping weight computations are simple arithmetic operations on a quantity for each material, you don’t need to load up the various common formulas used in the geometry equations. You just need to look up the relevant weight formulas by name, then evaluate them for each material in the list to get a weight value for that material. Sum up those values, for the entire materials list, and you’re done!

# class Calculator

  def shipping_weight
    calculator = Dentaku::Calculator.new

    # Sum up weights for all materials in project based on quantity
    materials.reduce(0.0) { |s, e| 
      weight = calculator.evaluate(@weight_formulas[e['name']], e)

      s + weight
    }.ceil
  end

Wiring the Calculator class up to your Sinatra application, you end up with a fully functional program, which looks just the same as it did when you mocked up the UI, but actually knows how to crunch numbers now.

As a sanity check, you enter the same values that you have been using to test the Calculator object on the command line into the Web UI, and observe the results:

They look correct. Mission accomplished!!!

SOURCE FILES: project.rb // calculator.rb

Considering the tradeoffs involved in using Dentaku

It was easy to decide on using Dentaku in this particular project, for several reasons:

The formulas used in the project consist entirely of simple arithmetic operations.
The tool itself is an internal application with no major performance requirements.
The people who will be writing the formulas already understand basic computing concepts.
A programmer will available to customize the workflow and assist with problems as needed.

If even a couple of these conditions were not met, the potential caveats of using Dentaku (or any similar formula processing tool) would require more careful consideration.

Maintainability concerns:

Even though Dentaku’s domain specific language is a very simple one, formulas are still a form of code. Like all code, any formulas that run through Dentaku need to be tested in some way – and when things go wrong, they need to be debugged.

If your use of Dentaku is limited to the sort of thing someone might type into a cell of an Excel spreadsheet, there isn’t much of a problem to worry about. You can fairly easily build some sane error handling, and can provide features within your application to allow the user to test formulas before they go live in production.

The more that user-defined computations start looking like “real programs”, the more you will miss the various niceties of a real programming environment. We take for granted things like smart code editors that understand the languages we’re working in, revision control systems, elaborate testing tools, debuggers, package managers, etc.

The simple nature of Dentaku’s DSL should prevent you from ever getting into enough complexity to require the benefits of a proper development environment. That said, if the use cases for your project require you to run complex user-defined code that looks more like a program than a simple formula, Dentaku would definitely be the wrong tool for the job.

Performance concerns:

The default evaluation behavior of Dentaku is completely unoptimized: simply adding two numbers together is a couple orders of magnitude slower than it would be in pure Ruby. It is possible to precompile expressions by enabling AST caching, and this reduces evaluation overhead significantly. Doing so may introduce memory management issues at scale though, and even with this optimization the evaluator runs several times slower than native Ruby.

None of these performance issues matter when you’re solving a single system of equations per request, but if you need to run Dentaku expressions in a tight loop over a large dataset, this is a problem to be aware of.

Usability concerns:

In this particular project, the people who will be using Dentaku are already familair with writing Excel-based formulas, and they are also comfortable with technology in general. This means that with a bit of documentation and training, they will be likely to comfortably use a code-based computational tool, as long as the workflow is kept relatively simple.

In cases where the target audience is not assumed to be comfortable writing code-based mathematical expressions and working with raw data formats, a lot more in-application support would be required. For example, one could imagine building a drag-and-drop interface for designing a garden layout, which would in turn generate the relevant Dentaku expressions under the hood.

The challenge is that once you get to the point where you need to put a layer of abstraction between the user and Dentaku’s DSL, you should carefully consider whether you actually need a formula processing engine at all. It’s certainly better to go without the extra complexity when it’s possible to do so, but this will depend heavily on the context of your particular application.

Extensibility concerns:

Setting up non-programmers with a means of doing their own computations can help cut down on a lot of tedious maintenance programming work, but the core domain model and data access rules are still defined by the application’s source code.

As requirements change in a business, new data sources may need to be wired up, and new pieces of support code may need to be written from time to time. This can be challenging, because tweaks to the domain model might require corresponding changes to the user-defined formulas.

In practice, this means that an embedded formula processing system works best when either the data sources and core domain model are somewhat stable, or there is a programmer actively maintaining the system that can help guide users through any necessary changes that come up.

With code stored either as user-provided data files or even in the application’s database, there is also a potential for messy and complicated migrations to happen whenever a big change does need to happen. This may be especially challenging to navigate for non-programmers, who are used to writing something once and having it work forever.

NOTE: Yes, this was a long list of caveats. Keep in mind that most of them only apply when you go beyond the “let’s take this set of Excel sheets and turn it into a nicely managed program” use case and venture into the “I want to embed an adhoc SDK into my application” territory. The concerns listed above are meant to help you sort out what category your project falls into, so that you can choose a modeling technique wisely.

Reflections and further explorations

By now you’ve seen that a formula parser/evaluator can be a great way to take a messy ad-hoc spreadsheet workflow and turn it into a slightly less messy ad-hoc web application workflow. This technique provides a way to balance the central management and depth of functionality that custom software development can offer with the flexibility and empowerment of putting computational modeling directly into the hands of non-programmers.

Although this is not an approach that should be used in every application, it’s a very useful modeling strategy to know about, as long as you keep a close eye on the tradeoffs involved.

If you’d like to continue studying this topic, here are a few things to try out:

Grab the source code for the calculator application, and run it on your own machine.
Create a new garden layout, with some new material types and shapes. For example, you could try to create a group of concentric circles, or a checkerboard style design.
Explore how to extend Dentaku’s DSL with your own Ruby functions.
Watch Spreadsheets for developers, a talk by Felienne Hermans on the power and usefulness of basic spreadsheet software for rapid protyping and ad-hoc explorations.

Good luck with your future number crunching, and thanks for reading!

Problem discovery comes before problem solving

Thu, 26 Mar 2015 00:00:00 +0000

Imagine you’re a programmer for a dental clinic, and they need your help to build a vacation scheduling system for their staff. Among other things, this system will display a calendar to staff members that summarizes all of the currently approved and pending vacation requests for the clinic, grouped by role.

The basic idea here is simple: If a dental assistant wants to take a week off some time in July, it’d be more likely to get time off approved for a week where there was only one other assistant out of the office than it would be for a week when five others were on vacation. Rather than waiting for a manager to review their request (which might take a while), this information can be supplied up front to make planning easier for everyone.

Your vacation request system already has been implemented weeks ago, so you can easily get all the data you need on who is requesting what time off, and who has already had their time off approved. Armed with this information, building the request summary calendar should be easy, right? Just take all the requests and then group them by employee roles, and then spit them out in chronological order. You’ll be able to roll out this new feature into production by lunch time!

You grab your morning coffee, and sit down to work. Before you can even open your text editor, an uncomfortable realization weighs heavily upon you: Roles are actually a property of shifts, not employees. Your clinic is understaffed, and so some employees are cross-trained and need to wear multiple hats. To put it bluntly, there’s at least one employee that’s not precisely a receptionist, and would be more adequately described as “receptionish”. She helps out in the billing office at times, and whenever she’s working there, the clinic is down a receptionist.

You do have access to some data about individual shifts, so maybe that could be used to determine roles. By the time a shift is approved, the role is set, and you know for sure what that employee is doing for that day.

You think for a little while. You uncover a few annoying problems that will need to be solved if you decide to go this route.

The shift data is coming from a third party shift planning system, and the import window is set out only ten weeks into the future. In practice, shifts aren’t really firmly committed to until four weeks out, so that makes the practical window even smaller.

The idea that a given employee’s shift in July would be set in stone by March is a fantasy, and so even if you could get at that data, it wouldn’t be perfectly accurate. There’s also no guarantee that attempting to import five times more data than what you’re currently working with won’t cause problems… the whole synchronization system was built in a bit of a hurry, and could be fragile in places.

Feeling the anxiety start to set in, you go for a quick walk around the block, and come to the realization that you’ve gone into problem solving mode already, when you really should be more in the problem discovery phase of things. You haven’t even answered the question of how many employees work in multiple different roles, and you’re already assuming that’s a problem that needs a clear solution.

An idea pops into your head. You rush to your desk, and pop open a Rails console in production. You write a crude query and then massage the data with an ugly chain of Enumerable methods, and end up with a report that looks like this:

 ["Nikko Bergnaum", [["Hygienists", 5]]],
 ["Anderson Miller", [["Billing", 50]]],
 ["Bell Effertz", [["Hygienists", 14]]],
 ["Vicky Okuneva", [["Receptionists", 30]]],
 ["Lavern Von", [["Assistants", 37]]],
 ["Crawford Purdy", [["Receptionists", 40]]],
 ["Valentin Daugherty", [["Hygienists", 61]]],
 ["Eudora Bauch", [["Receptionists", 40]]],
 ["Jaeden Bashirian", [["Assistants", 28]]],
 ["Roel Hammes", [["Dentists", 36]]],
 ["King Schowalter", [["Hygienists", 20]]],
 ["Liam Kovacek", [["Receptionists", 55]]],
 ["Elaina Von", [["Hygienists", 25]]],
 ["Susie Watsica", [["Hygienists", 31]]],
 ["Oswaldo Boyer", [["Hygienists", 20]]],
 ["Gardner Fay", [["Hygienists", 10]]],
 ["Joanny Beatty", [["Assistants", 52]]],
 ["Beth Yost", [["Hygienists", 34]]],
 ["Gerry Torphy", [["Hygienists", 10]]],
 ["Maureen Terry", [["Hygienists", 9]]],
 ["Maritza Kemmer", [["Billing", 25]]],
 ["Morton Hudson", [["Dentists", 61]]],
 ["Santino Parker", [["Hygienists", 49]]],
 ["Jesse Friesen", [["Hygienists", 31]]],
 ["Dillan Krajcik", [["Hygienists", 44]]],
 ["Travon Koch", [["Hygienists", 16]]],
 ["Audreanne Hand", [["Billing", 47]]],
 ["Coralie Predovic", [["Receptionists", 45]]],
 ["Jovani Schulist", [["Management", 50]]],
 ["Tanner D'Amore", [["Dentists", 41]]],
 ["Jace Nitzsche", [["Dentists", 21]]],
 ["Carolina Waters", [["Receptionists", 40]]],
 ["Terence Howell", [["Dentists", 39]]],
 ["Leann Pacocha", [["Assistants", 2]]],
 ["Alvah Rippin", [["Dentists", 50]]],
 ["Lorenzo West", [["Hygienists", 27]]],
 ["Gideon McKenzie", [["Dentists", 41]]],
 ["Katrine O'Reilly", [["Dentists", 51]]],
 ["Briana Ziemann", [["Dentists", 40]]],
 ["Jerome Harris", [["Dentists", 10]]],
 ["Misael Pagac", [["Assistants", 51]]],
 ["Krista Predovic", [["Assistants", 32]]],
 ["Carole O'Hara", [["Assistants", 42]]],
 ["Adalberto Doyle", [["Management", 49], ["Receptionists", 2]]],
 ["Noel Ortiz", [["Management", 28], ["Receptionists", 1]]],
 ["Monique McLaughlin", [["Receptionists", 43], ["Assistants", 1]]],
 ["Jaleel Graham", [["Billing", 50], ["Receptionists", 18]]],
 ["Ned Reilly", [["Receptionists", 50], ["Assistants", 1]]],
 ["Enrico Schowalter", [["Receptionists", 1], ["Assistants", 55]]],
 ["Caesar Goldner", [["Management", 30], ["Receptionists", 16]]],
 ["Kirstin Weissnat", [["Receptionists", 26], ["Assistants", 28]]],
 ["Guillermo Klein",
  [["Assistants", 41], ["Hygienists", 2], ["Receptionists", 3]]]]

This listing shows all the shifts planned for the next ten weeks, with counts for each employee by role. You copy and paste it into a text editor, and delete any of the lines for employees that have a single role. Here’s what you end up with:

   ["Adalberto Doyle", [["Management", 49], ["Receptionists", 2]]],
 ["Noel Ortiz", [["Management", 28], ["Receptionists", 1]]],
 ["Monique McLaughlin", [["Receptionists", 43], ["Assistants", 1]]],
 ["Jaleel Graham", [["Billing", 50], ["Receptionists", 18]]],
 ["Ned Reilly", [["Receptionists", 50], ["Assistants", 1]]],
 ["Enrico Schowalter", [["Receptionists", 1], ["Assistants", 55]]],
 ["Caesar Goldner", [["Management", 30], ["Receptionists", 16]]],
 ["Kirstin Weissnat", [["Receptionists", 26], ["Assistants", 28]]],
 ["Guillermo Klein",
  [["Assistants", 41], ["Hygienists", 2], ["Receptionists", 3]]]]

Now you’ve whittled the list down to only 9 people. In a business that employees over 50 people, this is 20% of the workforce, which isn’t a tiny number. But taking a closer look at the data, you realize something else: even on this list of cross-trained employees, most staff members work in a single role the majority of the time, and very rarely fill in for another role.

Filtering the list again, you remove anyone who works in a single role at least 90% of the time. After this step, only three employees remain on your list:

 ["Jaleel Graham", [["Billing", 50], ["Receptionists", 18]]],
 ["Caesar Goldner", [["Management", 30], ["Receptionists", 16]]],
 ["Kirstin Weissnat", [["Receptionists", 26], ["Assistants", 28]]]]]]]

Because these employees represent only about 5% of the total staff, you’ve revealed this problem as an edge case. For the other six employees that substitute for a different role once in a blue moon, you’d have at least 90% accuracy by always labeling them by their primary role. It’d be confusing to refer to them as anything else, at least for the purposes of vacation planning.

In the process of this ad-hoc exploration, you’ve discovered a reasonably accurate method of predicting employee roles far out into the future: if at least 90% of the shifts they’re assigned are for a particular role, assume that is their primary role. Otherwise, label them as cross-trained, listing out all the roles they commonly fill in for. For example, Jaleel could be listed as “X-Trained (Billing, Receptionist)”, Kirsten as “X-Trained (Receptionist, Assistant)”, and Caesar as “X-Trained (Receptionist, Management)”.

Taking this approach will be at least as reliable as using the raw shift data, and requires no major technical changes to the system’s under-plumbing. It’s also dynamic, in the sense that the system will adaptively relabel employees as cross trained when they’re doing more than one role on a regular basis.

Happy with this re-evaluation of the problem, you start working, and you manage to get the feature into production before lunch after all. In the worst case scenario, you can always come back to this and do more careful analysis, peering into the technological and philosophical rabbit hole that made you nervous in the first place. But there’s a very good chance that this solution will work just fine, and so it’s worth trying it out before venturing out into the darkness.

From this small exercise, you come to a powerful realization:

Software isn’t mathematically perfect reality, it’s a useful fiction meant to capture some aspect of reality that is interesting or important to humans. Although our technical biases may aim for logical purity in the code we write, the humans that use our work mainly care about the story we’re trying to tell them. We should seek the most simple solutions that allow us to tell that story, even if those solutions lack technical elegance.

In other words, feel free to ignore the man behind the curtain.

The anatomy of information in software systems

Mon, 08 Sep 2014 00:00:00 +0000

Suppose that you want catch up with your friends Alice, Bob, and Carol. To do this, you might log into your favorite chat service, join a group chat room, and then type in some sort of friendly greeting and hit the enter key. Moments later, your friends would see your message appear on their screens, and soon after that they would probably send you some sort of response. As long as their reply was somewhat intelligible, you could be reasonably certain that your message was successfully communicated, without giving much thought to the underlying delivery mechanism.

Just beneath the surface of this everyday activity, we find a world of precise rules and constraints governing our communications. In the world of chat clients and servers, the meaning of your message does not matter, but its structure is of critical importance. Protocols define the format for messages to be encoded in, and even small variations will result in delivery failures.

Even though much of this internal complexity is hidden by user interfaces, message format requirements are not a purely technical concern – they can also directly affect human behavior. On Twitter, a message needs to be expressed in 140 characters or less, and on IRC the limit is only a few hundred characters. This single constraint makes Twitter and IRC fundamentally different from email and web forums, so it’s hard to overstate the impact that constraints can have on a communicaitons medium.

In addition to intentional restrictions on message structure, there are always going to be incidental technical limitations that need to be dealt with – the kinds of quirks that arise from having too much or too little expressiveness¹ in a given message format. These unexpected obstacles are among the most interesting problems in information exchange, because they are not an essential part of the job to be done but rather an emergent property of the way we’ve decided to do the job.

As programmers, we’re constantly working to bridge the gap between people and the machines that serve them. This article explores the boundary lines between those two disjoint worlds, and the complicated decisions that need to be made in order to cross the invisible chasm that lies between computational structures and human meaning.

The medium is the message

To see the impact a communication medium can have on its messages, let’s work through a practical example. The line of text below is representative of what an IRC-based chat message look like when it get sent over a TCP socket:

PRIVMSG #practicing-ruby-testing :Seasons greetings to you all!\r\n

Even if you’ve never used IRC before or looked into its implementation details, you can extract a great deal of meaning from this single line of text. The structure is very simple, so it’s fairly obvious that PRIVMSG represents a command, #practicing-ruby-testing represents a channel, and that the message to be delivered is "Seasons greetings to you all!". If I asked you to parse this text to produce the following array, you probably would have no trouble doing so without any further instruction:

["PRIVMSG", "#practicing-ruby-testing", "Seasons greetings to you all!"]

But if this were a real project and not just a thought experiment, you might start to wonder more about the nuances of the protocol. Here are a few questions that might come up after a few minutes of careful thought:

What is the significance of the : character? Does it always signify the start of the message body, or does it mean something else?
Why does the message end in \r\n? Can the message body contain newlines, and if so, should they be represented as \n or \r\n, or something else entirely?
Will messages always take the form "PRIVMSG #channelname :Message Body\r\n", or are their cases where additional parameters will be used?
Can channel names include spaces? How about : characters?

Try as we might, no amount of analyzing this single example will answer these questions for us. That leads us to a very important point: Understanding the meaning of a message doesn’t necessarily mean that we know how to process the information contained within it.

The meaning of a message depends on its level of abstraction

At first glance, the text-based IRC protocol made it easy for us to identify the structure and meaning of the various parts of a chat message. But when we thought a little more about what it would take to actually implement the protocol, we quickly ran into several questions about how to construct well-formed messages.

A lot of the questions we came up with had to do with basic syntax rules, which is only natural when exploring an unfamiliar information format. For example, we can guess that the : symbol is a special character in the following text, but we can’t reliably guess its meaning without reading the formal specification for the IRC protocol:

PRIVMSG #practicing-ruby-testing :Seasons greetings to you all!\r\n

To see the effect of syntax on our interpretation of information formats, consider what happens when we shift the representation of a chat message into a generic structure that we are already familiar with, such as a Ruby array:

["PRIVMSG", "#practicing-ruby-testing", "Seasons greetings to you all!"]

Looking at this information, we still have no idea whether it constitutes a well-formed message to be processed by our hypothetical IRC-like chat system. But because we know Ruby’s syntax, we understand what is being communicated here at a primitive level.

Before when we looked at the PRIVMSG command expressed in the format specified by the IRC protocol, we weren’t able to reliably determine the rules for breaking the message up into its parts by looking at a single example. Because we didn’t already have its syntax memorized, we wouldn’t even be able to reliably parse IRC commands, let alone process them. But as Ruby programmers, we know what array and string literals look like, and so we know how to map their syntax to the concepts behind them.

The mundane observation to be made here is that it’s easier to understand a format you’re familiar with than it is to interpret one you’ve never seen before. A far more interesting point to discover is that these two examples have fundamental differences in meaning, even if they can be interpreted in a way that makes them equivalent to one another.

Despite their superficial similarities, the two examples we’ve looked at operate at completely different levels of abstraction. The IRC-based example directly encodes the concept of a chat message, whereas our Ruby example encodes the concept of an array of strings. In that sense, the former is a direct representation of a domain-specific concept, and the latter is a indirect representation built up from general-purpose data structures. Both can express the concept a chat message, but they’re not cut from the same cloth.

Let’s investigate why this difference in structure matters. Consider what might happen if we attempted to allow whitespace in chat channel names, i.e. #practicing ruby testing instead of #practicing-ruby-testing. By directly substituting this new channel name into our PRIVMSG command example, we get the text shown below:

PRIVMSG #practicing ruby testing :Seasons greetings to you all!\r\n

Here we run into a syntactic hiccup: If we allow for channel names to include whitespace, we need to come up with more complex rules for splitting up the message into its different parts. But if we decide this is an ill-formed string, then we need to come up with a constraint that says that the channel parameter cannot include spaces in it. Either way, we need to come up with a formal rule that will be applied at parse time, before processing even begins.

Now consider what happens when we use Ruby syntax instead:

["PRIVMSG", "#practicing ruby testing", "Seasons greetings to you all!"]

This is without question a well-formed Ruby array, and it will be successfully parsed and turned into an internal data structure. By definition, Ruby string literals allow whitespace in them, and there’s no getting around that without writing our own custom parser. So while the IRC example must consider the meaning of whitespace in channel names during the parsing phase, our Ruby example cannot. Any additional constraints placed on the format of channel names would need to be done via logical validations rather than syntactic rules.

The key insight here is that the concepts we’re expressing when we encode something in one syntax or another have meaning beyond their raw data contents. In the IRC protocol a channel is a defined concept at the symbolic level, with a specific meaning to it. When we encode a channel name as a Ruby string, we can only approximate the concept by starting with a more general structure and then applying logical rules to it to make it a more faithful representation of a concept it cannot directly express. This is not unlike translating a word from one spoken language to another which cannot express the same exact concept using a single word.

Every expressive syntax has at least a few corner cases

Consider once more our fascinating Ruby array:

["PRIVMSG", "#practicing-ruby-testing", "Seasons greetings to you all!"]

We’ve seen that because its structure is highly generic, its encoding rules are very permissive. Nearly any sequence of printable characters can be expressed within a Ruby string literal, and so there isn’t much ambiguity in expression of ordinary strings.

Despite its general-purpose nature, there are edge cases in Ruby’s string literal syntax that could lead to ambiguous or incomprehensible messages. For example, consider strings which have " characters within them:

"My name is: "Gregory"\n"

The above will generate a syntax error in Ruby, becasuse it ends up getting parsed as the string "My name is: ", followed immediately by the constant Gregory, followed by the string "\n". Ruby understandably has no way of interpreting that nonsense, so the parser will fail.

If we were only concerned with parsing string literals, we could find a way to resolve these ambiguities by adding some special parsing rules, but Ruby has a much more complex grammar across its entire featureset. For that reason, it expects you to be a bit more explicit when dealing with edge cases like this one. To get our string to parse, we’d need to do something like this:

"My name is: \"Gregory\"\n"

By writing \" instead of ", we tell the parser to treat the quote character as just another character in the string rather than a symbolic end-of-string marker. The \ acts as an escape character, which is useful for resolving these sorts of ambiguities. The cost of course is that \ itself becomes a potential source of ambiguity, so you end up having to write \\ instead of \ to express backslashes in Ruby string literals.

Edge cases of this sort arise in any expressive text-based format. They are often easy to resolve by adding a few more rules, but in many cases the addition of new processing rules add an even more subtle layer of corner cases to consider (as we’ve seen w. the \ character). Resolving minor ambiguities comes naturally to humans because we can guess at the meaning of a message, but cold-hearted computers can only follow the explicit rules we’ve given them.

Can we free ourselves from the limitations of syntax?

One solution to the syntactic ambiguity problem is to represent information in a way that is convenient for computers, rather than optimizing for human readability. For example, here’s the same array of strings represented as a raw sequence of bytes in MessagePack format:

a7 50 52 49 56 4d 53 47 b8 23 70 72 61 63 74 69 63 69 6e 67 2d 72 75 62 
2d 74 65 73 74 69 6e 67 bd 53 65 61 73 6f 6e 73 20 67 72 65 65 74 69 6e 
73 20 74 6f 20 79 6f 75 20 61 6c 6c 21

At first, this looks like a huge step backwards, because it smashes our ability to intuitively extract meaning from the message by simply reading its contents. But when we discover that the vast majority of these bytes are just encoded character data, things get a little more comprehensible:

"\x93\xA7PRIVMSG\xB8#practicing-ruby-testing\xBDSeasons greetings to you all!"

Knowing that most of the message is the same text we’ve seen in the other examples, we only need to figure out what the few extra bytes of information represent:

Like all binary formats, MessagePack is optimized for ease of processing rather than human readability. Instead using text-based symbols to describe the structure of data, MessagePack uses an entirely numeric encoding format.

By switching away from brackets, commas, and quotation marks to arbitrary values like 93, A7, B8, and BD, we immediately lose the ability to visually distinguish between the different structural elements of the message. This makes it harder to simply look at a message and know whether or not it is well-formed, and also makes it harder to notice the connections between the symbols and their meaning while reading an encoded message.

If you squint really hard at the yellow boxes in the above diagram, you might guess that 93 describes the entire array, and that A7, B8, and BD all describe the strings that follow them. But A7, B8, and BD need to be expressing more than just the concept of a string, otherwise there would be no need to use three different values. You might be able to discover the underlying rule by studying the example for a while, but it doesn’t just jump out at you the way a pair of opening and closing brackets might.

To avoid leaving you in suspense, here’s the key concept: MessagePack attempts to represent seralized data structures using as few bytes as possible, while making processing as fast as possible. To do this, MessagePack uses type headers that tell you exactly what type of data is encoded, and exactly how much space it takes up in the message. For small chunks of data, it conveys both of these pieces of information using a single byte!

Take for example the first byte in the message, which has the hexadecimal value of 93. MessagePack maps the values 90-9F to the concept of arrays with up to 15 elements. This means that an array with zero elements would have the type code of 90 and an array with 15 elements would have the type code of 9F. Following the same logic, we can see that 93 represents an array with 3 elements.

For small strings, a similar encoding process is used. Values in the range of A0-BF correspond to strings with up to 31 bytes of data. All three of our strings are in this range, so to compute their size, we just need to subtract the bottom of the range from each of them:

# note that results are in decimal, not hexadecimal
# String sizes are also computed explicitly for comparison

>> 0xA7-0xA0
=> 7
>> "PRIVMSG".size
=> 7

>> 0xB8-0xA0
=> 24
>> "#practicing-ruby-testing".size
=> 24

>> 0xBD-0xA0
=> 29
>> "Seasons greetings to you all!".size
=> 29

Piecing this all together, we can now see the orderly structure that was previously obfuscated by the compact nature of the MessagePack format:

Although this appears to be superficially similar to the structure of our Ruby array example, there are significant differences that become apparent when attempting to process the MessagePack data:

In a text-based format you need to look ahead to find closing brackets to match opening brackets, to organize quotation marks into pairs, etc. In MessagePack format, explicit sizes for each object are given so you know exactly where its data is stored in the bytestream.
Because we don’t need to analyze the contents of the message to determine how to break it up into chunks, we don’t need to worry about ambiguous interpretation of symbols in the data. This avoids the need for introducing escape sequences for the sole purpose of making parsing easier.
The explicit separation of metadata from the contents of the message makes it possible to read part of the message without analyzing the entire bytestream. We just need to extract all the relevant type and size information, and then from there it is easy to compute offsets and read just the data we need.

The underlying theme here is that by compressing all of the structural meaning of the message into simple numerical values, we convert the whole problem of extracting the message into a series of trivial computations: read a few bytes to determine the type information and size of the encoded data, then read some content and decode it based on the specified type, then rinse and repeat.

Separating structure from meaning via abstract types

Even though representing our message in a binary format allowed us to make information extraction more precise, the data type we used still corresponds to concepts that don’t exactly fit the intended meaning of our message.

One possible way to solve this conceptual mapping problem is to completely decouple structure from meaning in our message format. To do that, we could utilize MessagePack’s application-specific type mechanism; resulting in a message similar to what you see below:

The C7 type code indicates an abstract type, and is followed by two additional bytes: the first provides an arbitrary type id (between 0-127), and the second specifies how many bytes of data to read in that format. After applying these rules, we end up with the following structure:

The contents of each object in the array is the same as it always has been, but now the types have changed. Instead of an array composed of three strings, we now have an array that consists of elements that each have their own type.

Although I’ve illustrated the contents of each object as text-based strings for the sake of readability, the MessagePack format does not assume that the data associated with abstract types will be text-based. The decision of how to process this data is left up to the decoder.

Without getting into too many details, let’s consider how abstract data types might be handled in a real Ruby program² that processed MessagePack-based messages. You’d need to make an explicit mapping between type identifiers and the handlers for each type, perhaps using an API similar to what you see below:

data_types = { 1 => CommandName, 2 => Parameter, 3 => MessageBody }

command = MessagePackDecoder.unpack(raw_bytes, data_types)
#  [ CommandName <"PRIVMSG">, 
#    Parameter   <"#practicing-ruby-testing">, 
#    MessageBody <"Season greetings to you all!"> ]

Each handler would be responsible for transforming raw byte arrays into meaningful data objects. For example, the following class might be used to convert message parameters (e.g. the channel name) into a text-based representation:

class Parameter
  def initialize(byte_array)
    @text = byte_array.pack("C*")

    raise ArgumentError if @text.include?(" ")
  end

  attr_reader :text
end

The key thing to note about the above code sample is that the Parameter handler does not simply convert the raw binary into a string, it also applies a validation to ensure that the string contains no space characters. This is a bit of a contrived example, but it’s meant to illustrate the ability of custom type handlers to apply their own data integrity constraints.

Earlier we had drawn a line in the sand between the array-of-strings representation and the IRC message format because the former was forced to allow spaces in strings until after the parsing phase, and the latter was forced to make a decision about whether to allow them or not before parsing could be completed at all. The use of abstract types removes this limitation, allowing us to choose when and where to apply our validations, if we apply them at all.

Another dividing wall that abstract types seem to blur for us is the question of what the raw contents of our message actually represent. Using our own application-specific type definitions make it so that we never need to consider the contents of our messages to be strings, except as an internal implementation detail. However, we rely absolutely on our decoder to convert data that has been tagged with these arbitrary type identifiers into something that matches the underlying meaning of the message. In introducing abstract types, we have somehow managed to make our information format more precise and more opaque at the same time.

Combining human intuition with computational rigor

As we explored the MessagePack format, we saw that by coming up with very precise rules for processing an input stream, we can interpet messages by running a series of simple and unambiguous computations. But in the process of making things easier for the computer, we complicated things for humans. Try as we might, we aren’t very good at rapidly extracting meaning from numeric sequences like 93, C7 01 07, C7 02 18, and C7 03 1D.

So now we’ve come full circle in our explorations, realizing that we really do want to express ourselves using something like the text-based IRC message format. Let’s look at it one last time to reflect on its strengths and weaknesses:

PRIVMSG #practicing-ruby-testing :Seasons greetings to you all!\r\n

The main feature of representing our message this way is that because we’re familiar with the concept of commands as programmers, it is easy to see the structure of the message without even worrying about its exact syntax rules: we know intuitively that PRIVMSG is the command being sent, and that #practicing-ruby-testing and Seasons greetings to you all! are its parameters. From here, it’s easy to extract the underlying meaning of the message, which is: “Send the message ‘Seasons greetings to you all!’ to the #practicing-ruby-testing channel”.

The drawback is that we’re hazy on the details: we can’t simply guess the rules about whitespace in parameters, and we don’t know exactly how to interpret the : character or the \r\n at the end of the message. Because a correct implementation of the IRC protocol will need to consider various edge cases, attempting to precisely describe the message format verbally is challenging. That said, we could certainly give it a try, and see what happens…

Messages consist of a valid IRC command and its parameters (if any), followed by \r\n.
Commands are either made up solely of letters, or are represented as a three digit number.
All parameters are separated by a single space character.
Parameters may not contain \r\n or the null character (\0).
All parameters except for the last parameter must not contain spaces and must not start with a : character.
If the last parameter contains spaces or starts with a : character, it must be separated from the rest of the parameters by a : character, unless there are exactly 15 parameters in the message.
When all 15 parameters are present, then the separating : character can be omitted, even if the final parameter includes spaces.

This ruleset isn’t even a complete specification of the message format, but it should be enough to show you how specifications written in prose can quickly devolve into the kind of writing you might expect from a tax attorney. Because spoken language is inherently fuzzy and subjective in nature, it makes it hard to be both precise and understandable at the same time.

To get around these communication barriers, computer scientists have come up with metalanguages to describe the syntactic rules of protocols and formats. By using precise notation with well-defined rules, it is possible to describe a grammar in a way that is both human readable and computationally unambiguous.

When we look at the real specification for the IRC message format, we see one of these metalanguages in use. Below you’ll see a nearly complete specification³ for the general form of IRC messages expressed in Augmented Backus–Naur Form:

message    =  command [ params ] crlf
command    =  1*letter / 3digit
params     =  *14( SPACE middle ) [ SPACE ":" trailing ]
           =/ 14( SPACE middle ) [ SPACE [ ":" ] trailing ]

nospcrlfcl =  %x01-09 / %x0B-0C / %x0E-1F / %x21-39 / %x3B-FF
                ; any octet except NUL, CR, LF, " " and ":"

middle     =  nospcrlfcl *( ":" / nospcrlfcl )
trailing   =  *( ":" / " " / nospcrlfcl )

SPACE      =  %x20        ; space character
crlf       =  %x0D %x0A   ; "carriage return" "linefeed"
letter     =  %x41-5A / %x61-7A       ; A-Z / a-z
digit      =  %x30-39                 ; 0-9

If you aren’t used to reading formal grammar notations, this example may appear to be a bit opaque at first glance. But if you go back and look at the rules we listed out in prose above, you’ll find that all of them are expressed here in a way that leaves far less to the imagination. Each rule tells us exactly what should be read from the input stream, and in what order.

Representing syntactic rules this way allows us to clearly understand their intended meaning, but that’s not the only reason for the formality. BNF-based grammar notations express syntactic rules so precisely that we can use them not just as a specification for how to build a parser by hand, but as input data for a code generator that can build a highly optimized parser for us. This not only saves development effort, it also reduces the likelihood that some obscure edge case will be lost in translation when converting grammar rules into raw processing code.

To demonstrate this technique in use, I converted the ABNF representation of the IRC message format into a grammar that is readable by the Citrus parser generator. Apart from a few lines of embedded Ruby code used to transform the input data, the following code look conceptually similar to what you saw above:

grammar IRC
  rule message
    (command params? endline) {
      { :command => capture(:command).value,
        :params  => capture(:params).value }
    }
  end

  rule command
    letters | three_digit_code 
  end

  rule params
    ( ((space middle)14*14 (space ":"? trailing)?) |
      ((space middle)*14 (space ":" trailing)?) ) {
      captures.fetch(:middle, []) + captures.fetch(:trailing, [])
    }
  end

  rule middle
    non_special (non_special | ":")*
  end

  rule trailing
    (non_special | space | ":")+
  end

  rule letters
    [a-zA-Z]+
  end

  rule three_digit_code
    /\d{3}/ { to_str.to_i }
  end

  rule non_special
    [^\0:\r\n ]
  end

  rule space
    " "
  end

  rule endline
    "\r\n"
  end
end

Loading this grammar into Citrus, we end up with a parser that can correctly extract the commands and paramaters from our original PRIVMSG example:

require 'citrus'
Citrus.load('irc')

msg = "PRIVMSG #practicing-ruby-testing :Seasons greetings to you all!\r\n"

data = IRC.parse(msg).value

p data[:command] 
#=> "PRIVMSG"

p data[:params]
#=> ["#practicing-ruby-testing", "Seasons greetings to you all!"]

In taking this approach, we’re forced to accept certain constraints (like a set of complicated rules about where a : character can appear), but we avoid turning our entire message format into meaningless streams of numbers like 93 and C7 01 08. Even if there is a bit more magic going on in the conversion of a Citrus grammar into a functioning parser, we can still see the telltale signs of a deterministic process lurking just beneath the surface.

The decision to express a message in a text-based format or a binary format is one rife with tradeoffs, as we’ve already seen from this single example. Now that you’ve seen both approaches, consider how you might implement a few different types of message formats. Would an audio file be better represented as binary file format, or a text-based format? How about a web page? Before you read this article you probably already knew the answers to those questions, but now hopefully you have a better sense of the tradeoffs involved in how we choose to represent information in software systems.

The philosophical conundrum of information exchange

Computers are mindless automatons, and humans are bad at numbers. This friction between people and their machines runs so deep that it’s remarkable that any software gets built at all. But because there is gold to be found at the other side of the computational tarpit, we muddle through our differences and somehow manage to make it all work.

To work together, computers and humans need a bridge between their mutually exclusive ways of looking at the world. And this is what coding is all about! We encode information into data and source code for computers to process, and then after the work is done, we decode the results of a computation back into a human-friendly message format.

Once everything is wired up, human users of software can think mostly in terms of meaningful information exchange, and software systems only need to worry about moving numbers around and doing basic arithmetic operations. Although it isn’t especially romantic, this is how programmers trick computers and humans into cooperating with each other. When done well, people barely notice the presence of the software system at all, and focus entirely on their job to be done. This suits the computer just fine, as it does not care at all what puny humans think of it.

As programmers, we must concern ourselves with the needs of both people and machines. We are responsible for connecting two seemingly incompatible worlds, each with their own set of rules and expectations. This is what makes our job hard, but is also what makes it rewarding and almost magical at times. We’ve just explored some examples of the sorts of challenges that can arise along the boundary line between people and machines, but I’m sure you can think of many more that are present in your own work.

The next time you come across a tension point in your software design process, take a moment to reflect on these ideas, and see what kind of insights arise. Is the decision you’re about to make meant to benefit the people who use your software, or the machines that run your code? Consider the tradeoffs carefully, but when in doubt, always choose to satisfy the humans. :grin:

NOTE: While writing this article, I was also reading “Gödel, Escher, Bach” in my spare time. Though I don’t directly use any of its concepts here, Douglas Hofstadter deserves credit (and/or blame) for getting me to think deeply on the meaning of meaning and how it relates to software development.

Having too much or too little expressiveness in a format is pretty much a guarantee, because even as we get closer to the Goldilocks Zone, increasingly subtle edge cases tend to proliferate. Since we can’t expect perfection, we need to settle for expressiveness that’s “good enough” and the tradeoffs that come along with it. ↩
The abstract types API shown in this article is only a theoretical example, because the official MessagePack library for Ruby does not support application-specific types as of September 2014, even though they’re documented in the specification. It may be a fun patch to write if you want to explore these topics more, though! ↩
For the sake of simplicity, I omitted the optional prefix in IRC messages which contains information about the sender of a message, because it involves somewhat complicated URI parsing. See page 7 of the IRC specification for details. ↩

A self guided course on streams, files, file formats, and sockets

Mon, 31 Mar 2014 00:00:00 +0000

This self-guided course will help you learn how to work with the low level tools that you’d usually rely on libraries and frameworks to provide. Each of its four parts will give you a feel for a different kind of I/O programming and text processing work:

Standard I/O streams and the filesystem
Encoding and decoding binary files
Parsing text-based file formats
Socket programming and network I/O

In each part of the course, you’ll start by carefully reading a Practicing Ruby article that explores one of the topics listed above. You will then work through a set of review questions to test your grasp of the material. Finally you’ll apply the concepts to realistic problems by working on a set of project-based exercises.

Once you’ve completed the entire study guide, you’ll know how to do all of the following things:

Build command line applications in Ruby that follow the Unix philosophy and work similarly to other console-based applications you use day to day.
Encode and decode binary files at the level of bits and bytes, and understand how primitive data structures are represented in low-level storage formats.
Work with streams in a memory efficient way, whether they’re coming from files, standard I/O, or the internet.
Work with the same parser and compiler technology that is used by serious text processing libraries and programming languages.
Understand the basics behind TCP-level socket programming, and how to build simple client and server software.

We often take these concepts for granted because our libraries and frameworks take care of them for us. But this self-guided tour of Ruby’s basement will help you appreciate the many low-level tools and techniques we have available for solving these problems.

To begin your journey, fork the course git repository and then follow the instructions in its README. Work at your own pace, and don’t hesitate to ask for help when you need it. You can submit issues in our tracker for general questions, and pull requests when you’d like a review of your work. Good luck, and happy hacking!

Sustainable maintenance: Focus on quality first

Mon, 17 Feb 2014 00:00:00 +0000

This article was written in collaboration with Eric Hodel (@drbrain), a developer from Seattle. Eric is a Ruby core team member, and he also maintains RubyGems and RDoc.

A big challenge in managing open source projects is that their codebases tend to decay as they grow. This isn’t due to a lack of technically skilled contributors, but instead is a result of the gradual loss of understandability that comes along with any long-term and open-ended project that has an distributed team of volunteers supporting it.

Once a project becomes more useful, it naturally attracts a more diverse group of developers who are interested in adapting the codebase to meet their own needs. Patches are submitted by contributors who do not fully understand a project’s implementation, and maintainers merge these patches without fully understanding the needs of their contributors. Maintainers may also struggle to remember the reasoning behind any of their own code that they haven’t touched in a while, but they still need to be able to work with it.

As a result of both of these influencing factors, mistaken assumptions tend to proliferate as a project grows, and with them come bugs and undefined behaviors. When direct knowledge of the codebase becomes limited and unreliable, it’s easy to let code quality standards slip without fully realizing the potential for future problems.

If bad code continues to accumulate in this fashion, improving one part of a a project usually means breaking something else in the process. Once a maintainer starts spending most of their time fixing bugs, it gets hard to move their project forward in meaningful ways. This is where open source development stops being fun, and starts feeling like a painful chore.

Not all projects need to end up this way, though. As long as project maintainers make sure to keep the quality arrow pointing upwards over the long haul, any bad code that temporarily accumulates in a project can always be replaced with better code whenever things start getting painful. The real challenge is to establish healthy maintenance practices that address quality issues in a consistent and sustainable way.

Developing a process-oriented approach towards quality

In this article, we’ll discuss three specific tactics we’ve used in our own projects that can be applied at any stage in the software development lifecycle. These are not quick fixes; they are helpful habits that drive up understandability and code quality more and more as you continue to practice them. The good news is that even though it might be challenging to keep up with these efforts on a daily basis, the recommendations themselves are very simple:

Let external changes drive incremental quality improvements
Treat all code with inadequate testing as legacy code
Expand functionality via well-defined extension points

We’ll now take a look at each of these guidelines individually and walk you through some examples of how we’ve put them into practice in RDoc, RubyGems, and Prawn – three projects that have had their own share of quality issues over the years, but continue to serve very diverse communities of users and contributors.

1) Let external changes drive incremental quality improvements

Although there is often an endless amount of cleanup work that can be done in mature software projects, there is rarely enough available development time to invest in these efforts. For programmers working on open source in their spare time, it is hard enough to keep up with new incoming requests, so most preventative maintenance work ends up being deferred indefinitely. When cleanup efforts do happen, they tend to be done in concentrated bursts and then things go back to business-as-usual from there.

A better approach is to pay down technical debts little by little, not as a distinct activity but as part of responding to ordinary change requests. There are only two rules to remember when applying this technique in your daily work:

Try to avoid making the codebase worse with each new change, or at least minimize new maintenance costs as much as possible.
If there is an easy way to improve the code while doing everyday work, go ahead and invest a little bit of effort now to make future changes easier.

The amount of energy spent on meeting these two guidelines should be proportional to the perceived risks and rewards of the change request itself, but typically it doesn’t take a lot of extra effort. It may mean spending an extra 10 minutes on a patch that would take an hour to develop, or an extra hour on a patch that would take a day to prepare. In any case, it should feel like an obviously good investment that is well worth the cost you are paying for it.

There is a great example in Prawn that illustrates this technique being used, and if you want to see it in its raw form, you can check out this pull request from Matt Patterson.

Matt’s request was to change the way that Prawn’s image loading feature detected whether it was working with an I/O object or a path to a file on disk. Initially Prawn assumed that any object responding to read would be treated as an I/O object, but this was too loose of a test and caused some subtle failures when working with Pathname objects.

The technical details of the change are not important here, so don’t worry if you don’t understand them. Instead, just look at the method that would need to be altered to fix this problem, and ask yourself whether you would feel comfortable making a change to it:

def build_image_object(file)
  file.rewind  if file.respond_to?(:rewind)
  file.binmode if file.respond_to?(:binmode)

  if file.respond_to?(:read)
    image_content = file.read
  else
    raise ArgumentError, "#{file} not found" unless File.file?(file)  
    image_content = File.binread(file)
  end
  
  image_sha1 = Digest::SHA1.hexdigest(image_content)

  if image_registry[image_sha1]
    info = image_registry[image_sha1][:info]
    image_obj = image_registry[image_sha1][:obj]
  else
    info = Prawn.image_handler.find(image_content).new(image_content)

    min_version(info.min_pdf_version) if info.respond_to?(:min_pdf_version)

    image_obj = info.build_pdf_object(self)
    image_registry[image_sha1] = {:obj => image_obj, :info => info}
  end

  [image_obj, info]
end

Although this probably isn’t the absolute worst code you have ever seen, it isn’t very easy to read. Because it takes on many responsibilities, it’s hard to even summarize what it is supposed to do! Fortunately for Matt, the part that he would need to change was only the first few lines of the method, which are reasonably easy to group together:

def build_image_object(file)
  file.rewind  if file.respond_to?(:rewind)
  file.binmode if file.respond_to?(:binmode)

  if file.respond_to?(:read)
    image_content = file.read
  else
    raise ArgumentError, "#{file} not found" unless File.file?(file)  
    image_content = File.binread(file)
  end

   # ... everything else 
end

The quick fix would have been to edit these lines directly, but Matt recognized the opportunity to isolate a bit of related functionality and make the code a little bit better in the process of doing so. Pushing these lines of code down into a helper method and tweaking them slightly resulted in the following cleanup to the build_image_object method:

def build_image_object(file)
  io = verify_and_open_image(file)
  image_content = io.read

  # ... everything else
end

In the newly created helper method, Matt introduced his desired change, which is much easier to understand in isolation than it would have been in the original build_image_object method definition. In particular, he changed the duck typing test to look for rewind rather than read, in the hopes that it would be a more reliable way to detect I/O-like objects. Everything else would be wrapped in a Pathname instance:

def verify_and_open_image(io_or_path)
  if io_or_path.respond_to?(:rewind)
    io = io_or_path

    io.rewind

    io.binmode if io.respond_to?(:binmode)
    return io
  end

  io_or_path = Pathname.new(io_or_path)
  raise ArgumentError, "#{io_or_path} not found" unless io_or_path.file?
  
  io_or_path.open('rb')
end

At this point, he could have submitted a pull request, because the tests were still green and the new behavior was working as expected. However, the issue he had set out to fix in the first place wasn’t causing Prawn’s tests to fail, and that was a sign that there was some undefined behavior at the root of this problem. Although Prawn had some tests for reading images referenced by Pathname objects, it only had done its checks at a high level, and did not verify that the PDF output was being rendered correctly.

A test would be needed at the lower level to verify that the output was no longer corrupted, but this kind of testing is slightly tedious to do in Prawn. Noticing this rough spot, Matt created an RSpec matcher to make this kind of testing easier to do in the future:

RSpec::Matchers.define :have_parseable_xobjects do
  match do |actual|
    expect { PDF::Inspector::XObject.analyze(actual.render) }.not_to raise_error
    true
  end
  failure_message_for_should do |actual|
    "expected that #{actual}'s XObjects could be successfully parsed"
  end
end

Finally, he provided a few test cases to demonstrate that his patch fixed the problem he was interested in, and also covered some other common use cases as well:

context "setting the length of the bytestream" do
  it "should correctly work with images from Pathname objects" do
    info = @pdf.image(Pathname.new(@filename))
    expect(@pdf).to have_parseable_xobjects
  end

  it "should correctly work with images from IO objects" do
    info = @pdf.image(File.open(@filename, 'rb'))
    expect(@pdf).to have_parseable_xobjects
  end

  it "should correctly work with images from IO objects not set to mode rb" do
    info = @pdf.image(File.open(@filename, 'r'))
    expect(@pdf).to have_parseable_xobjects
  end
end

When you put all of these changes together, the total value of this patch is much greater than the somewhat obscure bug it fixed. By addressing some minor pain points as he worked, Matt also improved Prawn in the following ways:

The build_image_object method is now more understandable because one of its responsibilities has been broken out into its own method.
The verify_and_open_image method allows us to group together all the basic guard clauses for determining how to read the image data, making it easier to see exactly what those rules are.
The added tests clarify the intended behavior of Prawn’s image loading mechanism.
The newly added RSpec matcher will help us to do more PDF-level checks in future tests.

None of these changes required a specific and focused effort of refactoring or redesign, it just involved a bit of attention to detail and a willingness to make minor improvements that would pay off for someone else in the future.

As a project maintainer, you cannot expect contributors to put this level of effort into their patches – Matt really went above and beyond here. However, you can definitely look for these kind of opportunities yourself during review time, and either ask the contributor to make some revisions, or make them yourself before you merge in new changes. No matter who ends up doing the work, little by little these kinds of incremental cleanup efforts can turn a rough codebase into something pleasant to work with.

2) Treat all code without adequate testing as legacy code

Historically, we’ve defined legacy code as code that was written long before our time, without any consideration for our current needs. However, any untested code can also be considered legacy code¹, because it often has many of the same characteristics that make outdated systems difficult to work with. Open source projects evolve quickly, and even very clean code can cause a lot of headaches if its intended behavior is left undefined.

To guard against the negative impacts of legacy code, it helps to continuously update your project’s test suite so that it constantly reflects your current understanding of the problem domain you are working in. A good starting point is to make sure that your project has good code coverage and that you keep your builds green in CI. Once you’ve done that, the next step is to go beyond the idea of just having lots of tests and start focusing on making your test suite more capable of catching problems before they leak out into released code.

Here are some things to keep in mind when considering the potential impact that new changes will have on your project’s stability:

Any behavior change introduced without test coverage has a good chance of causing a defect or accidentally breaking backwards-compatibility in a future release.
A passing test suite is not proof that a change is well-defined and defect-free.
The only reliable way to verify that existing features have well-defined behavior and good test coverage is to review their code manually.
Contributors often don’t understand your project’s problem domain or its codebase well enough to know how to write good tests for their changes without some guidance.

These points are not meant to imply that each and every pull request ought to be gone over with a fine-tooth comb – they’re only meant to serve as a reminder that maintaining a high quality test suite is a harder problem than we often make it out to be. The same ideas of favoring incremental improvements over heroic efforts that we discussed earlier also apply here. There is no need to rush towards a perfect test suite all at once, as long as it improves on average over time.

We’ll now look at a pull request that Brian Fletcher submitted to RubyGems for a good example of how these ideas can be applied in practice.

Brian’s request was to add support for Base64 encoded usernames and passwords in gem request URLs. Because RubyGems already supported the use of HTTP Basic Auth with unencoded usernames and passwords in URLs, this was an easy change to make. The desired URL decoding functionality was already implemented by Gem::UriFormatter, so the initial commit for this pull request involved changing just a single line of code:

     request = @request_class.new @uri.request_uri
 
     unless @uri.nil? || @uri.user.nil? || @uri.user.empty? then
-      request.basic_auth @uri.user, @uri.password
+      request.basic_auth Gem::UriFormatter.new(@uri.user).unescape,
+                         Gem::UriFormatter.new(@uri.password).unescape
     end
 
     request.add_field 'User-Agent', @user_agent

On the surface, this looks like a fairly safe change to make. Because it only adds support for a new edge case, it should preserve the original behavior for URLs that did not need to be unescaped. No new test failures were introduced by this patch, and a quick look at the test suite shows that Gem::UriFormatter has some tests covering its behavior.

As far as changes go, this one is definitely low risk. But if you dig in a little bit deeper, you can find a few things to worry about:

Even though only a single line of code was changed, that line of code was at the beginning of a method that is almost 90 lines long. This isn’t necessarily a problem, but it should at least be a warning sign to slow down and take a closer look at things.
A quick look at the test suite reveals that although there were tests for the unescape method provided by GemUri::Formatter, there were no tests for the use of Basic Auth in gem request URLs, which means the behavior this patch was modifying was not formally defined. Because of this, we can’t be sure that a subtle incompatibility wasn’t introduced by this patch, and we wouldn’t know if one was introduced later due to a change to GemUri::Formatter, either.
The new behavior introduced by this patch also wasn’t verified, which means that it could have possibly been accidentally removed in a future refactoring or feature patch. Another contributor could easily assume that URL decoding was incidental rather than intentional without tests that indicated otherwise.

These are the kind of problems that a detailed review can discover which are often invisible at the surface level. However, a much more efficient maintenance policy is to simply assume one or more of the above problems exist whenever a change is introduced without tests, and then either add tests yourself or ask contributors to add them before merging.

In this case, Eric asked Brian to add a test after giving him some guidance on how to go about implementing it. For reference, this was his exact request:

Can you add a test for this to test/rubygems/test_gem_request.rb?

You should be able to examine the request object through the block #fetch yields to.

In response, Brian dug in and noticed that the base case of using HTTP Basic Auth wasn’t covered by the tests. So rather than simply adding a test for the new behavior he added, he went ahead and wrote tests for both cases:

class TestGemRequest < Gem::TestCase
 def test_fetch_basic_auth
    uri = URI.parse "https://user:[email protected]/specs." +
                     Gem.marshal_version
    @request = Gem::Request.new(uri, Net::HTTP::Get, nil, nil)
    conn = util_stub_connection_for :body => :junk, :code => 200

    response = @request.fetch

    auth_header = conn.payload['Authorization']

    assert_equal "Basic #{Base64.encode64('user:pass')}".strip, auth_header
  end

  def test_fetch_basic_auth_encoded
    uri = URI.parse "https://user:%7BDEScede%[email protected]/specs." +
                    Gem.marshal_version
    @request = Gem::Request.new(uri, Net::HTTP::Get, nil, nil)
    conn = util_stub_connection_for :body => :junk, :code => 200

    response = @request.fetch

    auth_header = conn.payload['Authorization']

    assert_equal "Basic #{Base64.encode64('user:{DEScede}pass')}".strip, 
                 auth_header
  end
end

It is hard to overstate the difference between a patch with these tests added to it and one without tests. The original commit introduced a new dependency and more complex logic into a feature that lacked formal definition of its behavior. But as soon as these tests are added to the change request, RubyGems gains support for a new special condition in gem request URLs while tightening up the definition of the original behavior. The tests also serve to protect both conditions from breaking without being noticed in the future.

Taken individually, the risks of accepting untested patches are small enough that they don’t seem important enough to worry about when you are pressed for time. But in the aggregate, the effects of untested code will pile up until your codebase really does become unworkable legacy code. For that reason, establishing good habits about reviewing and shoring up tests on each new change can make a huge difference in long-term maintainability.

3) Expand functionality via well-defined extension points

Most open source projects can benefit from having two clearly defined interfaces: one for end-users, and one for developers who want to extend its functionality. This point may seem tangentially related to code quality and maintainability, but a well-defined extension API can greatly increase a project’s stability.

When its possible to add new functionality to a project without patching its codebase directly, it becomes easier to separate essential features that most people will need from features that are only relevant in certain rare contexts. The ability to support external add-ons in a transparent way also makes it possible to try experiments outside of your main codebase and then only merge in features that prove to be both stable and widely used.

Even within the scope of a single codebase, explicitly defining a layer one level beneath the surface forces you to think about what the common points of interaction are between your project’s features. It also makes testing easier, because feature implementations tend to get slimmed down as the extension API becomes more capable. Each part can then be tested in isolation without having to think about large amorphous blobs of internal dependencies.

It may be hard to figure out how to create an extension API when you first start working on a project, because at that time you probably don’t know much about the ways that people will need to extend its core behavior, and you may not even have a good sense of what its core feature set should be! This is completely acceptable, and it makes sense to focus exclusively on your high-level interface at first. But as your project matures, you can use the following guidelines to incrementally bring a suitable extension API into existence:

With each new feature request, ask yourself whether it could be implemented as an external add-on without patching your project’s codebase. If not, figure out what extension points would make it possible to do so.
For any of your features that have become difficult to work with or overly complex, think about what extension points would need to be added in order to extract those features into external add-ons.
For any essential features that have clearly related functionality, figure out what it would take to re-implement them on top of well defined extension points rather than relying on lots of private internal code.

At first, you may start by carrying out these design considerations as simple thought experiments that will indirectly influence the way you implement things. Later, you can take them more seriously and seek to support new functionality via external add-ons rather than merging new features unless there is a very good reason to do otherwise. Every project needs to discover the right balance for itself, but the basic idea is that the value of a clear extension API increases the longer a project is in active use.

Because RDoc has been around for a very long time and has a fairly decent extension API, it is a good library to look at for examples of what this technique has to offer. Without asking Eric for help, I looked into what it would take to autolink Github issues, commits, and version tags in RDoc output. This isn’t something I had a practical use for, but I figured it would be a decent way to test how easily I could extend the RDoc parser.

I started with the following text as my input data:

Please see #125, #127, and #159

Also see @bed324 and v0.14.0

My goal was to produce the following HTML output after telling RDoc what repository that these issues, commits, and tags referred to:

<p>Please see <a href="https://github.com/prawnpdf/prawn/issues/125">#125</a>,
<a href="https://github.com/prawnpdf/prawn/issues/127">#127</a>, and 
<a href="https://github.com/prawnpdf/prawn/issues/159">#159</a></p>

<p>Also see <a
href="https://github.com/prawnpdf/prawn/commit/bed324">@bed324</a> and 
<a href="https://github.com/prawnpdf/prawn/tree/0.14.0">v0.14.0</a></p>

Rendered, the resulting HTML would look like this:

Please see #125, #127, and #159</p>

Also see @bed324 and v0.14.0</p>

I wasn’t concerned about styling or how to fit this new functionality into a full-scale RDoc run. I just wanted to see if I could take my little snippet of sample text and replace the GitHub references with their relevant links. My experiment was focused solely on finding an suitable entry point into the system that supported these kinds of extensions.

After about 20 minutes of research and tinkering, I was able to produce the following example:

require 'rdoc'

REPO_URL = "https://github.com/prawnpdf/prawn"

class GithubLinkedHtml < RDoc::Markup::ToHtml
  def handle_special_ISSUE(special)
    %{<a href="#{REPO_URL}/issues/#{special.text[1..-1]}">#{special.text}</a>}
  end

  def handle_special_COMMIT(special)
    %{<a href="#{REPO_URL}/commit/#{special.text[1..-1]}">#{special.text}</a>}
  end

  def handle_special_VERSION(special)
    tag = special.text[1..-1]

    %{<a href="#{REPO_URL}/tree/#{tag}">#{special.text}</a>}
  end
end

markup = RDoc::Markup.new

markup.add_special(/\s*(\#\d+)/, :ISSUE)
markup.add_special(/\s*(@\h+)/,  :COMMIT)
markup.add_special(/\s*(v\d+\.\d+\.\d+)/, :VERSION)

wh = GithubLinkedHtml.new(RDoc::Options.new, markup)

puts "<body>#{wh.convert ARGF.read}</body>"

Once I figured out the right APIs to use, this became an easy problem to solve. It was clear from the way things were laid out that this sort of use case had already been considered, and a source dive revealed that RDoc also uses these extension points internally to support its own behavior. The only challenge I ran into was that these extension points were not especially well documented, which is unfortunately a more common problem than it ought to be with open source projects.

It is often the case that extension points are built initially to support internal needs rather than external use cases, and so they often lag behind surface-level features in learnability and third-party usability. This is certainly a solvable problem, and is worth considering when working on your own projects. But even without documentation, explicit and stable extension points can be a hugely powerful tool for making a project more maintainable.

Reflections

As you’ve seen from these examples, establishing high quality standards for open source projects is a matter of practicality, not pride. Projects that are made up of code that is easy to understand, easy to test, easy to change, and easy to maintain are far more likely to be sustainable over the long haul than projects that are allowed to decay internally as they grow.

The techniques we’ve discussed in this article are ones that will pay off even if you just apply them some of the time, but the more you use them, the more you’ll get in return. The nice thing about these practices is that they are quite robust – they can be applied in early stage experimental software as well as in projects that have been used in production for years.

The hard part of applying these ideas is not in remembering them when things are painful, but instead in keeping up with them when things are going well with your project. The more contributions you receive, the more important these strategies will become, but it is also hard to keep up with them because they do slow down the maintenance process a little bit. Whenever you feel that pressure, remember that you are looking out for the future of your project by focusing on quality, and then do what you can to educate others so that they understand why these issues matter.

Every project is different, and you may find that there are other ways to keep a high quality standard without following the guidelines we’ve discussed in this article. If you have some ideas to share, please let us know!

The definition of legacy code as code without tests was popularized in 2004 by Michael Feathers, author of the extremely useful Working Effectively with Legacy Code book. ↩

Hunt the Wumpus: An exercise in creative coding

Wed, 11 Dec 2013 00:00:00 +0000

Hunt the Wumpus is a hide-and-seek game that takes place in an underground cave network full of interconnected rooms. To win the game, the player needs to locate the evil Wumpus and kill it while avoiding various different hazards that are hidden within in the cave.

Originally written by Gregory Yob in the 1970s, this game is traditionally played using a text-based interface, which leaves plenty up to the player’s imagination, and also makes programming easier for those who want to build Wumpus-like games of their own.

Because of its simple but clever nature, Hunt the Wumpus has been ported to many different platforms and programming languages over the last several decades. In this article, you will discover why this blast from the past serves as an excellent example of creative computing, and you’ll also learn how to implement it from scratch in Ruby.

Gameplay demonstration

There are only two actions available to the player throughout the game: to move from room to room, or to shoot arrows into nearby rooms in an attempt to kill the Wumpus. Until the player knows for sure where the Wumpus is, most of their actions will be dedicated to moving around the cave to gain a sense of its layout:

You are in room 1.
Exits go to: 2, 8, 5
-----------------------------------------
What do you want to do? (m)ove or (s)hoot? m
Where? 2
-----------------------------------------
You are in room 2.
Exits go to: 1, 10, 3
-----------------------------------------
What do you want to do? (m)ove or (s)hoot? m
Where? 10
-----------------------------------------
You are in room 10.
Exits go to: 2, 11, 9

Even after only a couple actions, the player can start to piece together a map of the cave’s topography, which will help them avoid getting lost as they continue their explorations:

Play continues in this fashion, with the player wandering around until a hazard is detected:

What do you want to do? (m)ove or (s)hoot? m
Where? 11
-----------------------------------------
You are in room 11.
Exits go to: 10, 8, 20
-----------------------------------------
What do you want to do? (m)ove or (s)hoot? m
Where? 20
-----------------------------------------
You are in room 20.
You feel a cold wind blowing from a nearby cavern.
Exits go to: 11, 19, 17

In this case, the player has managed to get close to a bottomless pit, which is detected by the presence of a cold wind emanating from an adjacent room.

Because hazards are sensed indirectly, the player needs to use a deduction process to know for sure which hazards are in what rooms. With the knowledge of the cave layout so far, the only thing that is for certain is there is at least one pit nearby, with both rooms 17 and 19 being possible candidates. One of them might be safe, but there is also a chance that BOTH rooms contain pits. In a literal sense, the player might have reached a dead end:

A risky player might chance it and try one of the two rooms, but that isn’t a smart way to play. The safe option is to backtrack in search of a different path through the cave:

What do you want to do? (m)ove or (s)hoot? m
Where? 11
-----------------------------------------
You are in room 11.
Exits go to: 10, 8, 20
-----------------------------------------
What do you want to do? (m)ove or (s)hoot? m
Where? 8
-----------------------------------------
You are in room 8.
You smell something terrible nearby
Exits go to: 11, 1, 7

Changing directions ends up paying off. Upon entering room 8, the terrible smell that is sensed indicates that the Wumpus is nearby, and because rooms 1 and 11 have already been visited, there is only one place left for the Wumpus to be hiding:

What do you want to do? (m)ove or (s)hoot? s
Where? 7
-----------------------------------------
YOU KILLED THE WUMPUS! GOOD JOB, BUDDY!!!

At the end of the hunt, the player’s map ended up looking like this:

In less fortunate circumstances, the player would need to do a lot more exploration before they could be certain about where the Wumpus was hiding. Other hazards might also be encountered, including giant bats that are capable of moving the player to a random location in the cave. Because all these factors are randomized in each new game, Hunt the Wumpus can be played again and again without ever encountering an identical cave layout.

We will discuss more about the game rules throughout the rest of this article, but the few concepts illustrated in this demonstration are more than enough for us to start modeling some of the key game objects. Let’s get to work!

Implementing “Hunt the Wumpus” from scratch

Like many programs from its era, Hunt the Wumpus was designed to be hackable. If you look at one of the original publications about the game, you can see that the author actively encourages tweaking its rules, and even includes the full source code of the game.

Before you rush off to study the original implementation, remember that it was written four decades ago in BASIC. Unless you consider yourself a technological archaeologist, it’s probably not the best way to learn about the game. With that in mind, I’ve put together a learning exercise that will guide you through implementing some of the core game concepts of Hunt the Wumpus – without getting bogged down in specific game rules or having to write boring user interface code.

In particular, I want you to implement three classes that I have already written the tests for:

A Wumpus::Room class to manage hazards and connections between rooms
A Wumpus::Cave class to manage the overall topography of the cave
A Wumpus::Player class that handles sensing and encountering hazards

You can work through this exercise by cloning its git repository, and following the instructions in the README. I have put the tests for each class on its own branch, so that you can merge them into your own code one at a time until you end up with a complete passing test suite.

Once these three classes are written, you’ll be able to use my UI code and game logic to play a rousing round of Hunt the Wumpus. You’ll also be able to compare your own work to my reference implementation of the game, and discuss any questions or thoughts with me about the differences between our approaches.

Throughout the rest of this article, I will provide design and implementation notes for each class, as well as a brief overview of how the game rules for Hunt the Wumpus can be implemented using these objects. These notes should help you interpret what the test suite is actually asking you to build, and will also help you understand my reference implementation.

NOTE: If you’re short on time or aren’t in the mood for hacking right now, you can still get a lot out of this exercise by simply thinking about how you’d write the code to pass the provided test suite, and then looking my implementation. But it’s definitely better to at least try to write some code yourself, even if you don’t complete the full exercise.

Modeling rooms

Structurally speaking, rooms and their connections form a simple undirected graph:

Our Room class will manage these connections, and also make it easy to query and manipulate the hazards that can be found in a room – including bats, pits, and the wumpus itself. In particular, we will build an object with the following attributes and behaviors:

describe "A room" do
  it "has a number"
  it "may contain hazards"

  describe "with neighbors" do
    it "has two-way connections to neighbors"
    it "knows the numbers of all neighboring rooms"
    it "can choose a neighbor randomly"
    it "is not safe if it has hazards" 
    it "is not safe if its neighbors have hazards"
    it "is safe when it and its neighbors have no hazards"
  end
end

Let’s walk through each of these requirements individually and fill in the necessary details.

1) Every room has an identifying number that helps the player keep track of where they are:

describe "A room" do
  let(:room) { Wumpus::Room.new(12) }

  it "has a number" do
    room.number.must_equal(12)
  end

  # ...
end

2) Rooms may contain hazards, which can be added or removed as the game progresses:

it "may contain hazards" do 
  # rooms start out empty
  assert room.empty?

  # hazards can be added
  room.add(:wumpus)
  room.add(:bats)

  # a room with hazards isn't empty
  refute room.empty?

  # hazards can be detected by name
  assert room.has?(:wumpus)
  assert room.has?(:bats)

  refute room.has?(:alf)

  # hazards can be removed
  room.remove(:bats)
  refute room.has?(:bats)
end

3) Each room can be connected to other rooms in the cave:

describe "with neighbors" do
  let(:exit_numbers) { [11, 3, 7] }

  before do
    exit_numbers.each { |i| room.connect(Wumpus::Room.new(i)) }
  end

   # ...
end

4) One-way paths are not allowed, i.e. all connections between rooms are bidirectional:

it "has two-way connections to neighbors" do
  exit_numbers.each do |i| 
    # a neighbor can be looked up by room number
    room.neighbor(i).number.must_equal(i)

    # Room connections are bidirectional
    room.neighbor(i).neighbor(room.number).must_equal(room)
  end
end

5) Each room knows all of its exits, which consist of all neighboring room numbers:

it "knows the numbers of all neighboring rooms" do
  room.exits.must_equal(exit_numbers)
end

6) Neighboring rooms can be selected at random, which is useful for certain game events:

it "can choose a neighbor randomly" do
  exit_numbers.must_include(room.random_neighbor.number)
end

7) A room is considered safe only if there are no hazards within it or any of its neighbors:

it "is not safe if it has hazards" do
  room.add(:wumpus)

  refute room.safe?
end

it "is not safe if its neighbors have hazards" do
  room.random_neighbor.add(:wumpus)

  refute room.safe?
end

it "is safe when it and its neighbors have no hazards" do
  assert room.safe?
end

Implementation notes

Because this object only handles basic data tranformations, it shouldn’t be hard to implement. But if you get stuck, you can always look at my version of the Wumpus::Room class.

Modeling the cave

Although a game of Hunt the Wumpus can be played with an arbitrary cave layout, the traditional Wumpus cave is based on the dodecahedron. To model things this way, a room is placed at each vertex, and the edges form the connections between rooms. If you squash the structure to fit in a two-dimensional space, you end up with the following graph:

Even though it would be technically possible to construct this structure without a collection object by connecting rooms together in an ad-hoc fashion, traversing the structure and manipulating it would be cumbersome. For that reason, we will build a Wumpus::Cave object with the following properties:

describe "A cave" do
  it "has 20 rooms that each connect to exactly three other rooms" 
  it "can select rooms at random"
  it "can move hazards from one room to another"
  it "can add hazards at random to a specific number of rooms"
  it "can find a room with a particular hazard"
  it "can find a safe room to serve as an entrance"
end

Some of these features a bit tricky to explain comprehensively through tests, but the following examples should give you a basic idea of how they’re meant to work.

1) The cave has 20 rooms, and each room is connected to exactly three other rooms:

describe "A cave" do
  let(:cave)  { Wumpus::Cave.dodecahedron }
  let(:rooms) { (1..20).map { |i| cave.room(i) } }

  it "has 20 rooms that each connect to exactly three other rooms" do
    rooms.each do |room|
      room.neighbors.count.must_equal(3)
      
      assert room.neighbors.all? { |e| e.neighbors.include?(room) }
    end
  end
end

The intent here is to loosly verify that the layout is dodecahedron shaped, but it is more of a sanity check than a strict validation. A stronger check would require us to compute things like minimal cycles for each point, which would make for a much more complicated test.

In my implementation I use a JSON file that hard-codes the connections between each room explicitly rather than trying to automatically generate the layout, so this test is mostly just to catch errors with that configuration file. If you reuse the dodecahredon.json file in your own code, it should make passing these tests easy.

2) Rooms in the cave can be selected randomly:

it "can select rooms at random" do
  sampling = Set.new

  # see test/helper.rb for how this assertion works
  must_eventually("randomly select each room") do
    new_room = cave.random_room 
    sampling << new_room

    sampling == Set[*rooms] 
  end
end

This feature is important for implementing the behavior of giant bats, who move the player to a random location in the cave. It is also useful for hazard placement, as we’ll see later. The way I test the behavior is a bit awkward, but the basic idea is that if you keep selecting rooms at random, you’ll eventually hit every room in the cave.

3) Hazards can be moved from one room to another:

it "can move hazards from one room to another" do
  room      = cave.random_room
  neighbor  = room.neighbors.first

  room.add(:bats)

  assert room.has?(:bats)
  refute neighbor.has?(:bats)

  cave.move(:bats, :from => room, :to => neighbor)

  refute room.has?(:bats)
  assert neighbor.has?(:bats)
end

This test shows bats being moved from a random room to one of its neighbors, but Cave#move can used to move any hazard between any two rooms in the cave, even if they are not adajecent to each other.

4) Hazards can be randomly distributed throughout the cave:

it "can add hazards at random to a specific number of rooms" do
  cave.add_hazard(:bats, 3)

  rooms.select { |e| e.has?(:bats) }.count.must_equal(3)
end

For the most part, the work to be done here is just to pick some rooms at random and add hazards to them. However, because there is no sense in adding a single type of hazard to a room more than once, Cave#add_hazard should take care to select only rooms that do not already have the specified hazard in them. This is hinted at by the specs, but because the check is a loose one, just keep this detail in mind while implementing this method.

5) Rooms can be looked up based on the hazards they contain:

it "can find a room with a particular hazard" do
  cave.add_hazard(:wumpus, 1)

  assert cave.room_with(:wumpus).has?(:wumpus)
end

In my implementation, I just grab the first room that matches the criteria, but any matching room would be acceptable. It would also make sense to have a Cave#all_rooms_with method, but it isn’t needed for a basic implementation of the game.

6) A safe entrance can be located:

it "can find a safe room to serve as an entrance" do
  cave.add_hazard(:wumpus, 1)
  cave.add_hazard(:pit, 3)
  cave.add_hazard(:bats, 3)

  entrance = cave.entrance

  assert entrance.safe?
end

This is where the Wumpus::Room#safe? method comes in handy. Picking any room that passes that condition is enough to get the job done here.

Implementation notes

The desired behavior of the Wumpus::Cave class is admittedly a bit underspecified here, but in many cases minor variations won’t effect gameplay all that much. Some of these operations are also intentionally a bit more general than what is strictly needed for the game, to permit some experimentation with rule changes once you have a working implementation.

This was a challenging object for me to design and test, because many of the features which are intuitively obvious are hard to specify formally. Do the best you can with building it, and refer to my implementation of the Wumpus::Cave class whenever you hit any snags.

Modeling the player

Despite the complexity of the cave layout, most game events in Hunt the Wumpus are triggered by local conditions based on the player’s current room and its direct neighbors. For example, imagine that the player is positioned in Room #1 as shown in following diagram:

With this setup, the player would sense the nearby hazards, resulting in the following output:

You are in room 1.
You hear a rustling sound nearby
You smell something terrible nearby
Exits go to: 2, 3, 4

Ordinarily we’d need to do some investigation work to discover which hazards were where, but because this is a contrived scenario, we don’t need to guess. Knowing the layout of the neighborhood, we can enumerate the possible outcomes for any player action:

The player will encounter the wumpus upon moving into room 2.
The player will encounter bats upon moving into room 3.
The player will not encounter any hazards in room 4.
The player can shoot into room 2 to kill the wumpus.
The player will miss the wumpus by shooting into room 3 or 4.

If you take this single example and generalize it, you’ll find that every turn of Hunt the Wumpus involves only three distinct kinds of events:

describe "the player" do
  it "can sense hazards in neighboring rooms" 
  it "can encounter hazards when entering a room"
  it "can perform actions on neighboring rooms" 
end

With these requirements in mind, it is possible for us to model the Wumpus::Player class as an event-driven object that handles each event type listed above. The only state it needs to explicitly maintain is a reference to the room currently being explored: everything else can be managed externally through callbacks. You’ll see why this is useful when we look at how the game rules are implemented later, but for now just try to follow along as best as you can.

The test setup for the Wumpus::Player class is a bit complicated, mostly because we need to reconstruct something similar to the layout shown in the previous diagram in order to meaningfully test its behavior:

describe "the player" do
  let(:player) { Wumpus::Player.new }

  let(:empty_room) { Wumpus::Room.new(1) }

  let(:wumpus_room) do
    Wumpus::Room.new(2).tap { |e| e.add(:wumpus) }
  end

  let(:bat_room) do
    Wumpus::Room.new(3).tap { |e| e.add(:bats) }
  end

  # ...
end

In addition to wiring up some rooms, I also register all of the events we’re interested in tracking during setup, using some dummy callbacks that are meant to serve as stand-ins for real game logic. This is not an elegant way of building a test harness, but it gets the job done:

let(:sensed)      { Set.new }
let(:encountered) { Set.new }

before do
  empty_room.connect(bat_room)
  empty_room.connect(wumpus_room)

  player.sense(:bats) do
    sensed << "You hear a rustling"
  end

  player.sense(:wumpus) do
    sensed << "You smell something terrible"
  end

  player.encounter(:wumpus) do
    encountered << "The wumpus ate you up!"
  end

  player.encounter(:bats) do
    encountered << "The bats whisk you away!"
  end

  player.action(:move) do |destination|
    player.enter(destination)
  end
end

Once all of that is taken care of, the callbacks can be tested in isolated scenarios:

it "can sense hazards in neighboring rooms" do
  player.enter(empty_room)
  player.explore_room

  sensed.must_equal(Set["You hear a rustling", "You smell something terrible"])
  
  assert encountered.empty?
end

it "can encounter hazards when entering a room" do
  player.enter(bat_room)
  encountered.must_equal(Set["The bats whisk you away!"])
  
  assert sensed.empty? 
end

it "can perform actions on neighboring rooms" do
  player.act(:move, wumpus_room)
  player.room.must_equal(wumpus_room)

  encountered.must_equal(Set["The wumpus ate you up!"])
  assert sensed.empty?
end

These test cases verify that the right callbacks have been called by manipulating simple sets of strings, but the real use case for the Wumpus::Player class is to trigger operations on game objects as well as the user interface. If you are having trouble imagining what that would look like, it may help to read ahead a bit further before attempting to get these tests to pass.

Implementation notes:

Like the Wumpus::Cave class, this object is underspecified, but you probably don’t need to build something identical to my implementation of Wumpus::Player in order to get the game to run. However, you may want to make an effort to ensure that callbacks are triggered in the order that they are registered, otherwise you can run into some interesting edge cases when more than one condition is satisfied at the same time.

Defining the game rules

With a foundation in place, implementing the game logic for Hunt the Wumpus is very easy. My version of the game simplifies the rules, but hopefully still captures the spirit of the original.

As you walk through the following code, you can treat the Wumpus::Narrator object as a black box. This is a boring object that only does some basic I/O under the hood, so your time would be better spent focusing on the game logic.

With that caveat out of the way, let’s take a look at how Hunt the Wumpus can be implemented in terms of the three game objects we just built. To get started, we need a cave!

cave = Wumpus::Cave.dodecahedron

This cave will contain three pits, three giant bats, and the most evil and stinky Wumpus you could ever imagine:

cave.add_hazard(:wumpus, 1)
cave.add_hazard(:pit, 3)
cave.add_hazard(:bats, 3)

We also need a player to navigate the cave, and a narrator to regale us with tales about the player’s adventures:

player    = Wumpus::Player.new
narrator  = Wumpus::Narrator.new

Whenever a player senses a hazard nearby, the narrator will give us a hint of what kind of trouble lurks just around the bend:

player.sense(:bats) do
  narrator.say("You hear a rustling sound nearby") 
end

player.sense(:wumpus) do
  narrator.say("You smell something terrible nearby")
end

player.sense(:pit) do
  narrator.say("You feel a cold wind blowing from a nearby cavern.")
end

If upon entering a room the player encounters the Wumpus, it will become startled. We’ll discuss the detailed consequences of this later, but the basic idea is that it will cause the Wumpus to either run away to an adjacent room, or to gobble the player up:

player.encounter(:wumpus) do
  player.act(:startle_wumpus, player.room)
end

When bats are encountered, the narrator will inform us of the event, then a random room will be selected to drop the player off in. If any hazards are encountered in that room, the effects will be applied immediately, possibly leading to the player’s demise.

But assuming that the player managed to survive the flight, the bats will take up residence in the new location. This can make navigation very complicated, because stumbling back into that room will cause the player to be moved to yet another random location:

player.encounter(:bats) do
  narrator.say "Giant bats whisk you away to a new cavern!"

  old_room = player.room
  new_room = cave.random_room

  player.enter(new_room)

  cave.move(:bats, from: old_room, to: new_room)
end

If the player happens to come across a bottomless pit, the story ends immediately, even though the player’s journey will probably go on forever:

player.encounter(:pit) do
  narrator.finish_story("You fell into a bottomless pit. Enjoy the ride!")
end

The player’s actions are what ultimately ends up triggering game events. The movement action is straightforward: it simply updates the player’s current location and then fires callbacks for any hazards encountered:

player.action(:move) do |destination|
  player.enter(destination)
end

Shooting is more complicated, although the way it is implemented here is still a simplification of how the original game worked. In Gregory Yob’s version, you had only five arrows, but they could travel a distance of up to five rooms, even shooting around corners if you knew the right path. In my version, arrows are unlimited but can only fire into neighboring rooms.

If the player shoots into the room that the Wumpus is hiding in, the beast is slayed and the story ends happily ever after. If instead the player shoots into the wrong room, then no matter where the Wumpus is in the cave, it will be startled by the sound.

player.action(:shoot) do |destination|
  if destination.has?(:wumpus)
    narrator.finish_story("YOU KILLED THE WUMPUS! GOOD JOB, BUDDY!!!") 
  else
    narrator.say("Your arrow missed!")

    player.act(:startle_wumpus, cave.room_with(:wumpus))
  end
end

When the Wumpus is startled, it will either stay where it is or move into one of its neighboring rooms. The player will be able to hear the Wumpus move anywhere in the cave, even if it is not in a nearby room.

If the Wumpus is in the same room as the player at the end of this process, it will gobble the player up and the game will end in sadness and tears:

player.action(:startle_wumpus) do |old_wumpus_room|
  if [:move, :stay].sample == :move
    new_wumpus_room = old_wumpus_room.random_neighbor
    cave.move(:wumpus, from: old_wumpus_room, to: new_wumpus_room)

    narrator.say("You heard a rumbling in a nearby cavern.")
  end

  if player.room.has?(:wumpus)
    narrator.finish_story("You woke up the wumpus and he ate you!")
  end
end

And that pretty much sums it up. I omitted a few lines of boilerplate code that fire up the main event loop, but this pretty much covers all of the code that implements the actual game rules. It is designed to be very hackable, so please do experiment with it however you’d like.

If you want to review the full game executable without the intermingled commentary, please see the bin/wumpus script.

Additional Exercises

Hopefully by working through this article you’ve seen for yourself why Hunt the Wumpus is both fun to play and fun to implement. If you are looking for more things to try, I’d suggest the following activities:

Limit the number of arrows that the player can shoot, and end the game when the player runs out of arrows.
Try implementing the “crooked arrow” behavior of the original Wumpus game. To do this allow the player to specify a path of up to five rooms. Whenever the player guesses an incorrect path, have the arrow to bounce into a random room. If the arrow ends up hitting the player because of this, they lose!
Make it harder to guess the connections between rooms by randomizing the room numbers for each new game while keeping the overall shape the same.
Try out one of the alternative cave layouts described in Gregory Yob’s followup publication about Wumpus 2.
Add new hazards of your own, or other types of game objects that are beneficial, or provide some more depth to the story.
Implement a solver bot that plays the game automatically.
Build a better user interface for the game, either improving the text-based UI or attempting something using a GUI or web-based interface. You should only need to edit the Wumpus::Narrator and Wumpus::Console objects in order to replace the current interface.
Keep the game behavior the same, but try out a different design than the one I provided here and/or improve the test suite.

If you try out any of these extra credit exercises, please share your work. I’d be very interested to see what you come up with. Until then, happy hacking!

Infrastructure automation by example

Tue, 12 Nov 2013 00:00:00 +0000

This issue of Practicing Ruby was a collaboration with Mathias Lafeldt (@mlafeldt), an Infrastructure Developer living in Hamburg, Germany. If Mathias had to choose the one Internet meme that best describes his work, it would certainly be Automate all the things!

For at least as long as Ruby has been popular among web developers, it has also been recognized as a useful tool for system administration work. Although it was first used as a clean alternative to Perl for adhoc scripting, Ruby quickly evolved to the point where it became an excellent platform for large scale infrastructure automation projects.

In this article, we’ll explore realistic code that handles various system automation tasks, and discuss what benefits the automated approach has over doing things the old-fashioned way. We’ll also see first-hand what it means to treat “infrastructure as code”, and the impact it has on building maintainable systems.

Prologue: Why does infrastructure automation matter?

Two massive infrastructure automation systems have been built in Ruby (Puppet and Chef), both of which have entire open-source ecosystems supporting them. But because these frameworks were built by and for system administrators, infrastructure automation is often viewed as a specialized skillset by Ruby programmers, rather than something that everyone should learn. This is probably an incorrect viewpoint, but it is one that is easy to hold without realizing the consequences.

Speaking from my own experiences, I had always assumed that infrastructure automation was a problem that mattered mostly for large-scale public web applications, internet service providers, and very complicated enterprise projects. In those kinds of environments, the cost of manually setting up servers would obviously be high enough to justify using a sophisticated automation framework. But because I never encountered those scenarios in my own work, I was content to do things the old-fashioned way: reading lots of “works for me” instructions from blog posts, manually typing commands on the console, and swearing loudly whenever I broke something. For things that really matter or tasks that seemed too tough for me to do on my own, I’d find someone else to take care of it for me.

The fundamental problem was that my system-administration related pain wasn’t severe enough to motivate me to learn a whole new way of doing things. Because I never got curious enough about the topic, I didn’t realize that infrastructure automation has other benefits beyond eliminating the costs of doing repetitive and error-prone manual configuration work. In particular, I vastly underestimated the value of treating “infrastructure as code”, especially as it relates to creating systems that are abstract, modular, testable, understandable, and utterly hackable. Narrowing the problem down to the single issue of reducing repetitive labor, I had failed to see that infrastructure automation has the potential to eliminate an entire class of problems associated with manual system configuration.

To help me get unstuck from this particular viewpoint, Mathias Lafeldt offered to demonstrate to me why infrastructure automation matters, even if you aren’t maintaining hundreds of servers or spending dozens of hours a week babysitting production systems. To teach me this lesson, Mathias built a Chef cookbook to completely automate the process of building an environment suitable for running Practicing Ruby’s web application, starting with nothing but a bare Ubuntu Linux installation. The early stages of this process weren’t easy: Jordan and I had to answer more questions about our system setup than I ever thought would be necessary. But as things fell into place and recipes started getting written, the benefits of being able to conceptualize a system as code rather than as an amorphous blob of configuration files and interconnected processes began to reveal themselves.

The purpose of this article is not to teach you how to get up and running with Chef, nor is it meant to explain every last detail of the cookbook that Mathias built for us. Instead, it will help you learn about the core concepts of infrastructure automation the same way I did: by tearing apart a handful of real use cases and seeing what you can understand about them. If you’ve never used an automated system administration workflow before, or if you’ve only ever run cookbooks that other people have provided for you, this article will give you a much better sense of why the idea of treating “infrastructure as code” matters. If you already know the answer to that question, you may still benefit from looking at the problem from a beginner’s mindset. In either case, we have a ton of code to work our way through, so let’s get started!

A recipe for setting up Ruby

Let’s take a look at how Chef can be used to manage a basic Ruby installation. As you can see below, Chef uses a pure Ruby domain-specific language for defining its recipes, so it should be easy to read even if you’ve never worked with the framework before:

include_recipe "ruby_build"

ruby_version = node["practicingruby"]["ruby"]["version"]

ruby_build_ruby(ruby_version) { prefix_path "/usr/local" }

bash "update-rubygems" do
  code   "gem update --system"
  not_if "gem list | grep -q rubygems-update"
end

gem_package "bundler"

At the high level, this recipe is responsible for handling the following tasks:

Installing the ruby-build command line tool.
Using ruby-build to compile and install Ruby to /usr/local.
Updating RubyGems to the latest version.
Installing the bundler gem.

Under the hood, a lot more is happening. Let’s take a closer look at each step to understand a bit more about how Chef recipes work.

Installing ruby-build

include_recipe "ruby_build"

Including the default recipe from the ruby_build cookbook in our own code takes care of installing the ruby-build command line utility, and also handles installing a bunch of low-level packages that are required to compile Ruby on an Ubuntu system. But all of this work happens behind the scenes – we just need to make use of the ruby_build_ruby command this cookbook provides and the rest will be taken care of for us.

Compiling and installing Ruby

ruby_version = node["practicingruby"]["ruby"]["version"]

ruby_build_ruby(ruby_version) { prefix_path "/usr/local" }

In our recipe, the version of Ruby we want to install is not specified explicitly, but instead set elsewhere using Chef’s attribute system. In the cookbook’s default attributes file, you’ll find an entry that looks like this:

default["practicingruby"]["ruby"]["version"] = "2.0.0-p247"

Chef has a very flexible and very complicated attribute management system, but its main purpose is the same as any configuration system: to keep source code as generic as possible by not hard-coding application-specific values. By getting these values out of the source file and into well-defined locations, it also makes it easy to see all of our application-specific configuration data at once.

Updating RubyGems

bash "update-rubygems" do
  code   "gem update --system"
  not_if "gem list | grep -q rubygems-update"
end

In this code we make use of a couple shell commands, the first of which is obviously responsible for updating RubyGems. The second command is a guard that prevents the gem update command from running more than once.

Most actions in Chef have similar logic baked into them to make sure operations are only carried out when necessary. These guard clauses are handled internally whenever there is a well defined condition to check for, so you don’t need to think about them often. In the case of shell commands the operation is potentially arbitrary, so a custom guard clause is necessary.

Installing bundler

gem_package "bundler"

This command is roughly equivalent to typing gem install bundler on the command line. Because we installed Ruby into /usr/local, it will be used as our system Ruby, and so we can use gem_package without any additional settings. More complicated system setups would involve a bit more code than what you see above, but for our purposes we’re able to keep things simple.

Putting all of these ideas together, we end up not just with an understanding of how to go about installing Ruby using a Chef recipe, but also a glimpse of a few of the benefits of treating “infrastructure as code”. As we continue to work through more complicated examples, those benefits will become even more obvious.

A recipe for setting up process monitoring

Now that we’ve tackled a simple example of a Chef recipe, let’s work through a more interesting one. The following code is what we use for installing and configuring the God process monitoring framework:

include_recipe "practicingruby::_ruby"

gem_package "god"

directory "/etc/god" do
  owner "root"
  group "root"
  mode  "0755"
end

file "/etc/god/master.conf" do
  owner    "root"
  group    "root"
  mode     "0644"
  notifies :restart, "service[god]"

  home     = node["practicingruby"]["deploy"]["home_dir"] 
  god_file = "#{home}/current/config/delayed_job.god"

  content "God.load('#{god_file}') if File.file?('#{god_file}')"
end

cookbook_file "/etc/init/god.conf" do
  source "god.upstart"
  owner  "root"
  group  "root"
  mode   "0644"
end

service "god" do
  provider Chef::Provider::Service::Upstart
  action   [:enable, :start]
end

The short story about this recipe is that it handles the following tasks:

Installing the god gem.
Setting up some configuration files for god.
Registering god as a service to run at system boot.
Starting the god service as soon as the recipe is run.

But that’s just the 10,000 foot view – let’s get down in the weeds a bit.

Installing god via RubyGems

include_recipe "practicingruby::_ruby"

gem_package "god"

God is distributed as a gem, so we need to make sure Ruby is installed before we can make use of it. To do this, we include the Ruby installation recipe that was shown earlier. If the Ruby recipe hasn’t run yet, it will be executed now, but if it has already run then include_recipe will do nothing at all. In either case, we can be sure that we have a working Ruby configuration by the time the gem_package command is called.

The gem_package command itself works exactly the same way as it did when we used it to install Bundler in the Ruby recipe, so there’s nothing new to say about it.

Setting up a master configuration file

directory "/etc/god" do
  owner "root"
  group "root"
  mode  "0755"
end

file "/etc/god/master.conf" do
  owner    "root"
  group    "root"
  mode     "0644"
  notifies :restart, "service[god]"

  home     = node["practicingruby"]["deploy"]["home_dir"] 
  god_file = "#{home}/current/config/delayed_job.god"

  content "God.load('#{god_file}') if File.file?('#{god_file}')"
end

A master configuration file is typically used with God to load all of the process-specific configuration files for a whole system when God starts up. In our case, we only have one process to watch, so our master configuration is a simple one-line shim that points at the delayed_job.god file that is deployed alongside our Rails application.

Because our /etc/god/master.conf file is so trivial, we directly specify its contents in the recipe itself rather than using one of Chef’s more complicated mechanisms for dealing with configuration files. In this particular case, manually creating the file would certainly involve less work, but we’d lose some of the benefits that Chef is providing here.

In particular, it’s worth noticing that file permissions and ownership are explicitly specified in the recipe, that the actual location of the file is configurable, and that Chef will send a notification to restart God whenever this file changes. All of these things are the sort of minor details that are easily forgotten when manually managing configuration files on servers.

Running god as a system service

God needs to be running at all times, so we want to make sure that it started on system reboot and cleanly terminated when the system is shut down. To do that, we can configure God to run as an Upstart service. To do that, we need to create yet another configuration file:

cookbook_file "/etc/init/god.conf" do
  source "god.upstart"
  owner  "root"
  group  "root"
  mode   "0644"
end

The cookbook_file command used here is similar to the file command, but has a specialized purpose: To copy files from a cookbook’s files directory to some location on the system being automated. In this case, we’re using the files/default/god.upstart cookbook file as our source, and it looks like this:

description "God is a monitoring framework written in Ruby"

start on runlevel [2345]
stop on runlevel [!2345]

pre-start exec god -c /etc/god/master.conf
post-stop exec god terminate

Here we can see exactly what commands are going to be used to start and shutdown God, as well as the runlevels that it will be started and stopped on. We can also see that the /etc/god/master.conf file we created earlier will be loaded by God whenever it starts up.

Now all that remains is to enable the service to run when the system boots, and also tell it to start up right now:

service "god" do
  provider Chef::Provider::Service::Upstart
  action   [:enable, :start]
end

It’s worth mentioning here that if we didn’t explicitly specify the Service::Upstart provider, Chef would expect the service configuration file to be written as a System-V init script, which are written at a much lower level of abstraction. There isn’t anything wrong with doing things that way, but Upstart scripts are definitely more readable.

By this point, we’ve already seen how Chef can be used to install packages, manage configuration files, run arbitrary shell commands, and set up system services. That knowledge alone will take you far, but let’s look at one more recipe to discover a few more advanced features before we wrap things up.

A recipe for setting up an Nginx web server

The recipe we use for configuring Nginx is the most complicated one in Practicing Ruby’s cookbook, but it mostly just combines and expands upon the concepts we’ve already discussed. Try to see what you can understand of it before reading the explanations that follow, but don’t worry if every last detail isn’t immediately clear to you:

node.set["nginx"]["worker_processes"]     = 4
node.set["nginx"]["worker_connections"]   = 768
node.set["nginx"]["default_site_enabled"] = false

include_recipe "nginx::default"

ssl_dir = ::File.join(node["nginx"]["dir"], "ssl")
directory ssl_dir do
  owner "root"
  group "root"
  mode  "0600"
end

domain_name = node["practicingruby"]["rails"]["host"]
bash "generate-ssl-files" do
  cwd   ssl_dir
  flags "-e"
  code <<-EOS
    DOM=#{domain_name}
    openssl genrsa -out $DOM.key 4096
    openssl req -new -batch -subj "/CN=$DOM" -key $DOM.key -out $DOM.csr
    openssl x509 -req -days 365 -in $DOM.csr -signkey $DOM.key -out $DOM.crt
    rm $DOM.csr
  EOS
  notifies :reload, "service[nginx]"
  not_if   { ::File.exists?(::File.join(ssl_dir, domain_name + ".crt")) }
end

template "#{node["nginx"]["dir"]}/sites-available/practicingruby" do
  source "nginx_site.erb"
  owner  "root"
  group  "root"
  mode   "0644"
  variables(:domain_name => domain_name)
end

nginx_site "practicingruby" do
  enable true
end

When you put all the pieces together, this recipe is responsible for the following tasks:

Overriding some default Nginx configuration values.
Installing Nginx and managing it as a service.
Generating a self-signed SSL certificate based on a configurable domain name.
Using a template to generate a site-specific configuration file.
Enabling Nginx to serve up our Rails application.

In this recipe even more than the others we’ve looked at, a lot of the details are handled behind the scenes. Let’s dig a bit deeper to see what’s really going on.

Installing and configuring Nginx

We rely on the nginx cookbook to do most of the hard work of setting up our web server for us. Apart from overriding a few default attributes, we only need to include the nginx:default recipe into our own code to install the relevant software packages, generate an nginx.conf file, and to provide all the necessary init scripts to manage Nginx as a service. The following four lines of code take care of all of that for us:

node.set["nginx"]["worker_processes"]     = 4
node.set["nginx"]["worker_connections"]   = 768
node.set["nginx"]["default_site_enabled"] = false

include_recipe "nginx::default"

The interesting thing to notice here is that unlike the typical server configuration file, only the things we explicitly changed are visible here. All the rest of the defaults are set automatically for us, and we don’t need to be concerned with their values until the time comes when we decide we need to change them. By hiding all the details that do not matter to us, Chef recipes tend to be much more intention revealing than the typical server configuration file.

Generating SSL keys

In a real production environment, we would probably copy SSL credentials into place rather than generating them on the fly. However, since this particular cookbook provides a blueprint for building an experimental testbed rather than an exact clone of our live system, we handle this task internally to make the system a little bit more developer-friendly.

The basic idea behind the following code is that we want to generate an SSL certificate and private key for whatever domain name you’d like, so that it is possible to serve up the application over SSL within a virtualized staging environment. But since that is somewhat of an obscure use case, you can focus on what interesting Chef features are being used in the following code rather than the particular shell code being executed:

ssl_dir = ::File.join(node["nginx"]["dir"], "ssl")
directory ssl_dir do
  owner "root"
  group "root"
  mode  "0600"
end

domain_name = node["practicingruby"]["rails"]["host"]
bash "generate-ssl-files" do
  cwd   ssl_dir
  flags "-e"
  code <<-EOS
    DOM=#{domain_name}
    openssl genrsa -out $DOM.key 4096
    openssl req -new -batch -subj "/CN=$DOM" -key $DOM.key -out $DOM.csr
    openssl x509 -req -days 365 -in $DOM.csr -signkey $DOM.key -out $DOM.crt
    rm $DOM.csr
  EOS
  notifies :reload, "service[nginx]"
  not_if   { ::File.exists?(::File.join(ssl_dir, domain_name + ".crt")) }
end

As you read through this code, you may have noticed that ::File is used instead of File, which looks a bit awkward. The problem here is that Chef defines its own File class that ends up having a naming collision with Ruby’s core class. So to safely make use of Ruby’s File class, we need to explicitly do our constant lookup from the top-level namespace. This is just a small side effect of how Chef’s recipe DSL is implemented, but it is worth noting to clear up any confusion.

With that distraction out of the way, we can skip right over the directory code which we’ve seen in earlier recipes, and turn our attention to the bash command and its options. This example is far more interesting than the one we used to update RubyGems earlier, because in addition to specifying a command to execute and a not_if guard clause, it also does all of the following things:

Switches the working directory to the SSL directory we created within our Nginx directory.
Sets the -e flag, which will abort the script if any command fails to run successfully.
Uses a service notification to tell Nginx to reload its configuration files

From this we see that executing shell code via a Chef recipe isn’t quite the same thing as simply running some commands in a console. The entire surrounding context is also specified and verified, making it a whole lot more likely that things will work the way you expect them to. If these benefits were harder to see in the Ruby installation recipe, they should be easier to recognize now.

Configuring Nginx to serve up Practicing Ruby

Although the nginx cookbook takes care of setting up our nginx.conf file for us, it does not manage site configurations for us. We need to take care of that ourselves and tweak some settings dynamically, so that means telling our recipe to make use of a template:

template "#{node["nginx"]["dir"]}/sites-available/practicingruby" do
  source "nginx_site.erb"
  owner  "root"
  group  "root"
  mode   "0644"
  variables(:domain_name => domain_name)
end

The full template is a rather long file full of the typical Nginx boilerplate, but the small excerpt below shows how it is customized using ERB to insert some dynamic content:

server {
  listen 80;
  server_name <%= "#{@domain_name} www.#{@domain_name}" %>;
  rewrite ^ https://$server_name$request_uri? permanent;
}

Once the configuration file is generated and stored in the right place, we enable it using the following command:

nginx_site "practicingruby" do
  enable true
end

Under the hood, the nxensite script is used to do the actual work of enabling the site, but that implementation detail is deliberately kept hidden from view.

At this point, we have studied enough features of Chef to establish a basic literacy that will facilitate reading a wide range of recipes with only a little bit of effort. At the very least, you now have enough knowledge to make sense of every recipe in Practicing Ruby’s cookbook.

A cookbook for building a (mostly) complete Rails environment

The goal of this article was to give you a sense of what kinds of building blocks that Chef recipes are made up of so that you could see various infrastructure automation concepts in practice. If you feel like you’ve made it that far, you may now be interested in looking at how a complete automation project is sewn together.

The full Practicing Ruby cookbook contains a total of eight recipes, three of which we’ve already covered in this article. The five recipes we did not discuss are responsible for handling the following chores:

Creating and managing a deployment user account to be used by Capistrano.
Installing PostgreSQL and configuring a database for use with our Rails app.
Configuring Unicorn and managing it as an Upstart service.
Setting up some folders and files needed to deploy our Rails app.
Installing and managing MailCatcher as a service, to make email testing easier.

If you are curious about how these recipes work, go ahead and read them! Many are thin wrappers around external cookbook dependencies, and none of them use any Chef features that we haven’t already discussed. Attempting to make sense of how these recipes work would be a great way to test your understanding of what we covered in this article.

If you want to take things a step farther, you can actually try to provision a production-like environment for Practicing Ruby on your own system. The cookbook’s README file is fairly detailed, and we have things set up to work within a virtual machine that can run in isolation without having a negative impact on your own development environment. We also simplify a few things to make setup easier, such as swapping out GitHub authentication for OmniAuth developer mode, making most service integrations optional, and other little tweaks that make it possible to try things out without having to do a bunch of configuration work.

I absolutely recommend trying to run our cookbook on your own to learn a whole lot more about Chef, but fair warning: to do so you will need to become familiar with the complex network of underplumbing that we intentionally avoided discussing in this article. It’s not too hard to work your way through, but expect some turbulence along the way.

Epilogue: What are the costs of infrastructure automation?

The process of learning from Practicing Ruby’s cookbook, and the act of writing this article really convinced me that I had greatly underestimated the potential benefits that infrastructure automation has to offer. However, it is important to be very clear on one point: there’s no such thing as a free lunch.

At my current stage of understanding, I feel the same about Chef as I do about Rails: impressed by its vast capabilities, convinced of its utility, and shocked by its complexity. There are a tremendous amount of moving parts that you need to understand before it becomes useful, and many layers of subsystems that need to be wired up before you can actually get any of your recipes to run.

Another concern is that “infrastructure as code” comes with the drawbacks associated with code and not just the benefits. Third-party cookbooks vary in quality and sometimes need to be patched or hacked to get them to work the way you want, and some abstractions are leaky and leave you doing some tedious work at a lower level than you’d want. Dependency management is also complicated: using external cookbooks means introducing at least one more fragile package installer into your life.

In the case of Chef in particular, it is also a bit strange that although its interface is mostly ordinary Ruby code, it has developed in a somewhat parallel universe where the user is assumed to know a lot about system administration, and very little about Ruby. This leads to some design choices that aren’t necessarily bad, but are at least surprising to an experienced Ruby developer.

And as for infrastructure automation as a whole, well… it doesn’t fully free you from knowing quite a few details about the systems you are trying to manage. It does allow you to express ideas at a higher level, but you still need to be able to peel back the veneer and dive into some low level system administration concepts whenever something doesn’t work the way you expect it would or doesn’t support the feature you want to use via its high level interface. In that sense, an automated system will not necessarily reduce learning costs, it just has you doing a different kind of learning.

Despite all these concerns, I have to say that this is one skillset that I wish I had picked up years ago, and I fully intend to look for opportunities to apply these ideas in my own projects. I hope after reading this article, you will try to do the same, and then share your stories about your experiences.

Recommendations for further reading

Despite having a very complex ecosystem, the infrastructure automation world (and especially the Chef community) have a ton of useful documentation that is freely available and easy to get started with. Here are a few resources to try out if you want to continue exploring this topic on your own:

Opscode Chef documentation: The official Chef documentation; comprehensive and really well organized.
Opscode public cookbooks: You can learn a lot by reading some of the most widely-used cookbooks in the Chef community. For complex examples, definitely check out the apache2 and mysql cookbooks.
#learnchef: A collection of tutorials and screencasts designed to help you learn Chef.
Common Idioms in Chef Recipes: Explanation of (possibly surprising) idioms that sometimes appear in recipe code.
Learning Chef: A friendly introduction to Chef written by Mathias.

If you’ve got some experience with infrastructure automation and have found other tutorials or articles that you like which aren’t listed here, please leave a comment. Mathias will also be watching the comments for this article, so don’t be afraid to ask any general questions you have about infrastructure automation or Chef, too.

Thanks for making it all the way to the end of this article, and happy automating!

Exploring low-level computing concepts with Ruby

Tue, 01 Oct 2013 00:00:00 +0000

This issue of Practicing Ruby was directly inspired by Nick Morgan’s Easy 6502 tutorial. While the Ruby code in this article is my own, the bytecode for the Snake6502 game was shamelessly stolen from Nick. Be sure to check out Easy 6502 if this topic interests you; it’s one of the best programming tutorials I’ve ever seen.

The sea of numbers you see below is about as close to the metal as programming gets:

20 06 06 20 38 06 20 0d 06 20 2a 06 60 a9 02 85 
02 a9 04 85 03 a9 11 85 10 a9 10 85 12 a9 0f 85 
14 a9 04 85 11 85 13 85 15 60 a5 fe 85 00 a5 fe 
29 03 18 69 02 85 01 60 20 4d 06 20 8d 06 20 c3 
06 20 19 07 20 20 07 20 2d 07 4c 38 06 a5 ff c9 
77 f0 0d c9 64 f0 14 c9 73 f0 1b c9 61 f0 22 60 
a9 04 24 02 d0 26 a9 01 85 02 60 a9 08 24 02 d0 
1b a9 02 85 02 60 a9 01 24 02 d0 10 a9 04 85 02 
60 a9 02 24 02 d0 05 a9 08 85 02 60 60 20 94 06 
20 a8 06 60 a5 00 c5 10 d0 0d a5 01 c5 11 d0 07 
06a0: e6 03 e6 03 20 2a 06 60 a2 02 b5 10 c5 10 d0 06 
06b0: b5 11 c5 11 f0 09 e8 e8 e4 03 f0 06 4c aa 06 4c 
06c0: 35 07 60 a6 03 ca 8a b5 10 95 12 ca 10 f9 a5 02 
06d0: 4a b0 09 4a b0 19 4a b0 1f 4a b0 2f a5 10 38 e9 
06e0: 20 85 10 90 01 60 c6 11 a9 01 c5 11 f0 28 60 e6 
06f0: 10 a9 1f 24 10 f0 1f 60 a5 10 18 69 20 85 10 b0 
01 60 e6 11 a9 06 c5 11 f0 0c 60 c6 10 a5 10 29 
1f c9 1f f0 01 60 4c 35 07 a0 00 a5 fe 91 00 60 
a2 00 a9 01 81 10 a6 03 a9 00 81 10 60 a2 00 ea 
ea ca d0 fb 60 

Although you probably can’t tell by looking at it, what you see here is assembled machine code for the venerable 6502 processor that powered many of the classic video games of the 1980s. When executed in simulated environment, this small set of cryptic instructions produces a minimal version of the Snake arcade game, as shown below:

In this article, we will build a stripped down 6502 simulator in JRuby that is complete enough to run this game. If you haven’t done much low-level programming before, don’t worry! Most of what follows is just ordinary Ruby code. I will also be showing you a ton of examples along the way, and those should help keep you on track. You might also want to grab full source code for the simulator, so that you can experiment with it while reading through this article.

Warmup exercise: Reverse engineering Snake6502

An interesting property of machine code is that if you know its structure, you can convert it back into assembly language. Among other things, the ability to disassemble machine code is useful for debugging and exploration purposes. Let’s try this out on Snake6502!

The output below shows memory locations, machine code, and assembly code for the first 28 instructions of the game. These instructions are responsible for initializing the state of the snake and the apple before the main event loop kicks off. You don’t need to understand exactly how they work right now, just try to get a feel for how the code in the hexdump column corresponds to the code in the assembly column:

address  hexdump     assembly
------------------------------
$0600    20 06 06    JSR $0606
$0603    20 38 06    JSR $0638
$0606    20 0d 06    JSR $060d
$0609    20 2a 06    JSR $062a
$060c    60          RTS
$060d    a9 02       LDA #$02
$060f    85 02       STA $02
$0611    a9 04       LDA #$04
$0613    85 03       STA $03
$0615    a9 11       LDA #$11
$0617    85 10       STA $10
$0619    a9 10       LDA #$10
$061b    85 12       STA $12
$061d    a9 0f       LDA #$0f
$061f    85 14       STA $14
$0621    a9 04       LDA #$04
$0623    85 11       STA $11
$0625    85 13       STA $13
$0627    85 15       STA $15
$0629    60          RTS
$062a    a5 fe       LDA $fe
$062c    85 00       STA $00
$062e    a5 fe       LDA $fe
$0630    29 03       AND #$03
$0632    18          CLC
$0633    69 02       ADC #$02
$0635    85 01       STA $01
$0637    60          RTS

If you look at the output carefully, you’ll be able to notice some patterns even if you don’t understand what the instructions themselves are meant to do. For example, each instruction is made up of between 1-3 bytes of machine code. The first byte in each instruction tells us what operation it is, and the remaining bytes (if any) form its operand.

If you take a look at the first four instructions, it is easy to see that the opcode 20 corresponds to the JSR instruction. Forming its operand is similarly straightforward, because it’s the same number in both places, just with opposite byte order:

06 06 -> JSR $0606  
38 06 -> JSR $0638
0d 06 -> JSR $060d
2a 06 -> JSR $062a

If you ignore the symbols in front of the numbers for the moment, mapping single byte operands is even easier, because they’re represented the same way in both the machine code and the assembly code. Knowing that the 85 opcode maps to the STA operation, it should be easy to see how 11, 13, 15 map to $11, $13, $15 in the following example:

11  -> STA $11
13  -> STA $13
15  -> STA $15

But the symbols in front of the numbers in assembly language obviously mean something. If you carefully look at the machine code, you’ll be able to find that the same operation can have multiple different opcodes, each of which identify a particular kind of operand:

a9 0f -> LDA #$0f
a5 fe -> LDA $fe

Without getting into too much detail here, the example above shows us that both a9 and a5 correspond to the LDA instruction. The difference between the two opcodes is that a9 treats its operand as an immediate value, and a5 interprets it as a memory address. In assembly code, this difference is represented syntactically (#$xx vs. $xx), but in the machine code we must rely on numbers alone.

The various ways of interpreting operands (called “addressing modes”) are probably the most confusing part of working with 6502 code. There are about a dozen of them, and to get Snake6502 running, we need to implement most of them. The good news is that every addressing mode is just a roundabout way of converting an operand into a particular address in memory, and once you have that address, the operations themselves do not care about how you computed it. Once you sweep all that stuff under the rug, you can end up with clean operation definitions like this:

# NOTE: 'e' refers to the address that was computed from the instruction's
# operand and addressing mode.

LDA { cpu[:a] = mem[e]  } 
STA { mem[e]  = cpu[:a] }

This realization also tells us that the memory module will not need to take addressing modes into account as long as they’re precomputed elsewhere. With that in mind, let’s get started building a storage model for our simulator. We’ll deal with the hairy problem of addressing modes later.

Memory

Except for a few registers that are used to store intermediate computations, the 6502 processor relies on its memory for pretty much everything. Program code, data, and the system stack all reside in the same 16-bit addressing space. Even flow control is entirely dependent on memory: the program counter itself is nothing more than an address that is used to look up the next instruction to run.

This “all in one bucket” approach is a double-edged sword. It makes it harder to write safe programs, but the tradeoff is that the storage model itself is very simple. Conceptually, the memory module is nothing more than a mapping between 16-bit addresses and 8-bit values:

describe "Storage" do
  let(:mem) { Vintage::Storage.new }

  it "can get and set values" do
    mem[0x1337] = 0xAE

    mem[0x1337].must_equal(0xAE)
  end

  # ...
end

But because the program counter keeps track of a ‘current location’ in memory at any point in time, there is a lot more we can do with this simple structure. Let’s walk through the remaining tests for Vintage::Storage to see what else it implements.

Program loading

When a program is loaded into memory, there is nothing special about the way it is stored, it’s just like any other data. In a real 6502 processer, a register is used to store the address of the next instruction to be run, and that address is used to read an opcode from memory. In our simulator, we can let the Storage class keep track of this number for us, incrementing it whenever we call the Storage#next method.

The following test shows how to load a program and then walk its code one byte at a time:

it "can load a bytecode sequence into memory and traverse it" do
  bytes = [0x20, 0x06, 0x06]

  mem.load(bytes)
  mem.pc.must_equal(program_offset) # load() does not increment counter

  bytes.each { |b| mem.next.must_equal(b) }

  mem.pc.must_equal(program_offset + 3)
end

The starting position of the program can be an arbitrary location, but to maintain compatibility with the simulator from the Easy6502 tutorial, we initialize the program counter to 0x600:

let(:program_offset) { Vintage::Storage::PROGRAM_OFFSET }

it "sets an initial position of $0600" do
  program_offset.must_equal(0x0600)

  mem.pc.must_equal(program_offset)
end

Flow control + branching

Very rudimentary flow control is supported by setting the program counter to a particular address, which causes the processor to jump to the instruction at that address:

it "implements jump" do
  mem.jump(program_offset + 0xAB)

  mem.pc.must_equal(program_offset + 0xAB)
end

Branching can be implemented by only calling jump when a condition is met:

it "implements conditional branching" do
  big   = 0xAB
  small = 0x01

  # a false condition does not affect mem.pc
  mem.branch(small > big, program_offset + 5)
  mem.pc.must_equal(program_offset)

  # true condition jumps to the provided address
  mem.branch(big > small, program_offset + 5)
  mem.pc.must_equal(program_offset + 5)
end

This test case is a bit contrived, so let’s take a look at some real Snake6502 code that illustrates how branching meant to be used:

$064d    a5 ff     LDA $ff      # read the last key pressed on the keyboard
$064f    c9 77     CMP #$77     # check if the key was "w" (ASCII code 0x77)
$0651    f0 0d     BEQ $0660    # if so, jump forward to $0660 
$0653    c9 64     CMP #$64     # check if the key was "d" (ASCII code 0x64)
$0655    f0 14     BEQ $066b    # if so, jump forward to $066b
$0657    c9 73     CMP #$73     # check if the key was "s" (ASCII code 0x73)
$0659    f0 1b     BEQ $0676    # if so, jump forward to $0676
$065b    c9 61     CMP #$61     # check if the key was "a" (ASCII code 0x61)
$065d    f0 22     BEQ $0681    # if so, jump forward to $0681

Presumably, the code at $0660 starts a procedure that moves the snake’s head up, the code at $066b moves it to the right, and so on. In other words, if one of these BEQ instructions finds a match, it will jump to the right place in the code to handle the relevant condition. But if no match is found, the processor will happily continue on to whatever code comes after this set of instructions in the program.

The tricky thing about using instructions that rely on jump (and consequently, branch) is that they are essentially GOTO statements. When you see one of these statements in the code, you know exactly what instruction will be executed next, but there’s no way of telling if it will ever return to the location it was called from. To get around this problem, we need support for subroutines that know how to return to where they’ve been called from. And to implement those, we need a system stack.

Stack operations

Here are the tests for how we’d like our stack to behave:

let(:stack_origin) { Vintage::Storage::STACK_ORIGIN }
let(:stack_offset) { Vintage::Storage::STACK_OFFSET }

it "has a 256 element stack between 0x0100-0x01ff" do
  stack_offset.must_equal(0x0100)
  stack_origin.must_equal(0xff) # this value gets added to the offset
end

it "implements stack-like behavior" do
  mem.sp.must_equal(stack_origin)

  mem.push(0x01)
  mem.push(0x03)
  mem.push(0x05)

  mem.sp.must_equal(stack_origin - 3)

  mem.pull.must_equal(0x05)
  mem.pull.must_equal(0x03)
  mem.pull.must_equal(0x01)

  mem.sp.must_equal(stack_origin)
end

As the tests indirectly suggest, the stack is a region in memory between$0100 and $01ff, indexed by a stack pointer (sp). Each time a value is pushed onto the stack, the value of the stack pointer is decremented, and each time a value is pulled, the pointer is incremented. This makes it so that the stack pointer always tells you where the “top of the stack” is.

Subroutines

With a stack in place, we’ll have most of what we need to implement “Jump to subroutine” (jsr) and “Return from subroutine” (rts) functionality. The behavior of these features will end up looking something like this:

it "implements jsr/rts" do
  mem.jsr(0x0606)
  mem.jsr(0x060d)

  mem.pc.must_equal(0x060d)

  mem.rts
  mem.pc.must_equal(0x0606)

  mem.rts
  mem.pc.must_equal(program_offset)
end

To make the above test pass, jsr needs to push the current program counter onto the stack before executing a jump to the specified address. Later when rts is called, the address is pulled out of the stack, and then another jump is executed to bring you back to where the last jsr command was executed. This works fine even in nested subroutine calls, due to the nature of how stacks work.

The only tricky part is that addresses are 16-bit values, but stack entries are limited to single byte values. To get around this problem, we need a couple helper functions to convert a 16-bit number into two bytes, and vice-versa:

it "can convert two bytes into a 16 bit integer" do
  mem.int16([0x37, 0x13]).must_equal(0x1337)
end

it "can convert a 16 bit integer into two bytes" do
  mem.bytes(0x1337).must_equal([0x37, 0x13])
end

These helpers will also come in handy later, when we need to deal with addressing modes.

Implementation

Behavior-wise, there is a lot of functionality here. In a high level environment it would feel a lot like we were mixing distinct concerns, but at the low level we’re working at it’s understandable that nearly infinite flexibility is desireable.

Despite the conceptual complexity, the Storage class is extremely easy to implement. In fact, it takes less than 80 lines of code if you don’t worry about validations and robustness:

module Vintage
  class Storage
    PROGRAM_OFFSET = 0x0600
    STACK_OFFSET   = 0x0100
    STACK_ORIGIN   = 0xff

    def initialize
      @memory = Hash.new(0)
      @pc     = PROGRAM_OFFSET
      @sp     = STACK_ORIGIN
    end

    attr_reader :pc, :sp

    def load(bytes)
      index = PROGRAM_OFFSET

      bytes.each_with_index { |c,i| @memory[index+i] = c }
    end

    def [](address)
      @memory[address]
    end

    def []=(address, value)
      @memory[address] = (value & 0xff)
    end

    def next
      @memory[@pc].tap { @pc += 1 }
    end

    def jump(address)
      @pc = address
    end

    def branch(test, address)
      return unless test

      @pc = address
    end

    def jsr(address)
      low, high = bytes(@pc)

      push(low)
      push(high)

      jump(address)
    end

    def rts
      h = pull
      l = pull

      @pc = int16([l, h])
    end

    def push(value)
      @memory[STACK_OFFSET + @sp] = value
      @sp -= 1
    end

    def pull
      @sp += 1

      @memory[STACK_OFFSET + @sp]
    end

    def int16(bytes)
      bytes.pack("c*").unpack("v").first
    end

    def bytes(num)
      [num].pack("v").unpack("c*")
    end
  end
end

For such boring code, its a bit surprising to think that it can be a fundamental building block for generic computing. Keep in mind of course that we’re building a simulation and not a real piece of hardware, and we’re doing it in one of the highest level languages you can use.

If it already feels like we’re cheating, just wait until you see the next trick!

Memory-mapped I/O

To implement Snake6502, our simulator needs to be able to generate random numbers, read keyboard input, and also display graphics on the screen. None of these features are directly supported by the 6502 instruction set, so that means that every individual system had to come up with its own way of doing things. This is one of many things that causes machine code (especially old-school machine code) to not be directly portable from one system to another.

Because we’re trying to get Snake6502 to run in our simulator without modifying its bytecode, we’re more-or-less constrained to following the approach used by the Easy6502 simulator: memory-mapped I/O.

This approach is actually very easy to implement in a simulated environment: you add hooks around certain memory addresses so that when they are accessed, they execute some custom code rather than directly reading or writing a value to memory. In the case of Snake6502, we expect the following behaviors:

Reading from $fe returns a random 8-bit integer.
Reading from $ff retrieves the ASCII code of the last key pressed on the keyboard.
Writing to addresses between $0200 to $05ff will render pixels to the screen. ($0200 is the top-left corner of the 32x32 display, and $05ff is the bottom-right corner.)

These features could be added directly to the Storage class, but it would feel a bit awkward to clutter up a generic module with some very specific edge cases. For that reason, it is probably better to implement them as a module mixin:

module Vintage
  module MemoryMap
    RANDOMIZER  = 0xfe
    KEY_PRESS   = 0xff
    PIXEL_ARRAY = (0x0200..0x05ff)

    attr_accessor :ui

    def [](address)
      case address
      when RANDOMIZER
        rand(0xff)
      when KEY_PRESS
        ui.last_keypress
      else
        super
      end
    end

    def []=(k, v)
      super

      if PIXEL_ARRAY.include?(k)
        ui.update(k % 32, (k - 0x0200) / 32, v % 16)
      end
    end
  end
end

You probably already have a good idea of how MemoryMap works from seeing its implementation, but it wouldn’t hurt to see an example of how it is used before we move on. Here’s how to display a single pixel on the screen, randomly varying its color until the spacebar (ASCII code 0x20) is pressed:

mem = Vintage::Storage.new
mem.extend(Vintage::MemoryMap)

mem.ui = Vintage::Display.new 

(mem[0x0410] = mem[0xfe]) until mem[0xff] == 0x20 

It’s worth noting that this is the only code in the entire simulator that directly depends on a connection to some sort of user interface, and the protocol consists of just two methods: ui.update(x, y, color) and ui.last_keypress. In our case, we use a JRuby-based GUI, but anything else could be substituted as long as it implemented these two methods.

At this point, our storage model is pretty much complete. We now can turn our attention to various number crunching features.

Registers and Flags

In order to get Snake6502 to run, we need all six of the programmable registers that the processor provides. We’ve handled two of them already (the stack pointer and the program counter), so we just have four more to implement: A, X, Y, and P. A few design constraints will help make this work go a whole lot faster:

Most of the operations that can be done on A are done the same way on X and Y, so we can implement some generic functions that operate on all three of them.
We can implement the status register (P) as a collection of individual attributes, rather than seven 1-bit flags packs into a single byte.
Because Snake6502 only relies on the (c)arry, (n)egative, and (z)ero flags from the status register, we can skip implementing the other four status flags and still have a playable game.

With those limitations in mind, let’s work through some specs to understand how this model ought to behave. For starters, we’ll be building a Vintage::CPU that implements three registers and three flags, initializing them all to zero by default:

describe "CPU" do
  let(:cpu) { Vintage::CPU.new }

  let(:registers) { [:a, :x, :y] }
  let(:flags)     { [:c, :n, :z] }
  
  it "initializes registers and flags to zero" do
    (registers + flags).each { |e| cpu[e].must_equal(0) }
  end

   #...
end

It will be possible to directly set registers via the #[]= method, because the behavior will be the same for all three registers:

it "allows directly setting registers" do
  registers.each do |e|
    value  = rand(0xff)

    cpu[e] = value
    cpu[e].must_equal(value)
  end
end

However, because flags don’t have the same update semantics as registers, we will not allow directly setting them via #[]=:

it "does not allow directly setting flags" do
  flags.each do |e|
    value  = rand(0xff)

    err = -> { cpu[e] = value }.must_raise(ArgumentError)
    err.message.must_equal "#{e.inspect} is not a register"
  end
end

The carry flag (c) can toggled via the set_carry and clear_carry methods. We’ll need this later for getting the CPU into a clean state whenever we do addition and subtraction operations:

it "allows setting the c flag via set_carry and clear_carry" do
  cpu.set_carry
  expect_flags(:c => 1)

  cpu.clear_carry
  expect_flags(:c => 0)
end

Some other instructions will require us to set the carry flag based on arbitrary conditions, so we’ll need support for that as well:

it "allows conditionally setting the c flag via carry_if" do
  # true condition
  x = 3
  cpu.carry_if(x > 1)

  expect_flags(:c => 1)

  # false condition
  x = 0
  cpu.carry_if(x > 1)

  expect_flags(:c => 0)
end

The N and Z flags are set based on whatever result the CPU last processed:

it "sets z=1 when a result is zero, sets z=0 otherwise" do
  cpu.result(0)
  expect_flags(:z => 1)

  cpu.result(0xcc)
  expect_flags(:z => 0)
end

it "sets n=1 when result is 0x80 or higher, n=0 otherwise" do
  cpu.result(rand(0x80..0xff))
  expect_flags(:n => 1)

  cpu.result(rand(0x00..0x7f))
  expect_flags(:n => 0)
end

The result method also returns a number truncated to fit in a single byte, because pretty much every place we could store a number in this system expects 8-bit integers:

it "truncates results to fit in a single byte" do
  cpu.result(0x1337).must_equal(0x37)
end  

To help keep the CPU in a consistent state and to simplify the work involved in many of the 6502 instructions, we automatically call cpu.result whenever a register is set via CPU#[]=. The tests below show the the effects of that behavior:

  it "implicitly calls result() when registers are set" do
    registers.each do |e|
      cpu[e] = 0x100
      
      cpu[e].must_equal(0)
      expect_flags(:z => 1, :n => 0)

      cpu[e] -= 1
      
      cpu[e].must_equal(0xff)
      expect_flags(:z => 0, :n => 1)
    end
  end

Here’s an implementation that satisfies all of the tests we’ve seen so far:

module Vintage
  class CPU
    def initialize
      @registers = { :a => 0, :x => 0, :y => 0 }
      @flags     = { :z => 0, :c => 0, :n => 0 }
    end

    def [](key)
      @registers[key] || @flags.fetch(key)
    end

    def []=(key, value)
      unless @registers.key?(key)
        raise ArgumentError, "#{key.inspect} is not a register" 
      end

      @registers[key] = result(value)
    end

    def set_carry
      @flags[:c] = 1
    end

    def clear_carry
      @flags[:c] = 0
    end

    def carry_if(test)
      test ? set_carry : clear_carry
    end

    def result(number)
      number &= 0xff

      @flags[:z] = (number == 0 ? 1 : 0)
      @flags[:n] = number[7]

      number
    end
  end
end

Putting it all together, the role of the CPU class is mostly just to do some basic numerical housekeeping that will make implementing 6502 instructions easier. Consider for example, the CMP and BEQ operations, which can be used together to form a primitive sort of if statement. We saw these two operations used together in the earlier example of keyboard input handling:

$064f    c9 77     CMP #$77     # check if the key was "w" (ASCII code 0x77)
$0651    f0 0d     BEQ $0660    # if so, jump forward to $0660 

Using a combination of the CPU and Storage objects we’ve already built, we’d be able to define the CMP and BEQ operations as shown below:

CMP do 
  cpu.carry_if(cpu[:a] >= mem[e])

  cpu.result( cpu[:a] - mem[e] )
end

BEQ { mem.branch(cpu[:z] == 1, e) }

Even if we ignore the cpu.carry_if call, we know from what we’ve seen already that if CPU#result is called with a zero value, it will set the Z flag to 1. We also know that when Storage#branch is called with a true value, it will jump to the specified address, otherwise it will do nothing at all. Putting those two facts together with the Snake6502 shown above tells us that if the value in the A register is 0x77, execution will jump to $0600.

At this point, we’re starting to see how 6502 instructions can be mapped onto the objects we’ve already built, and that means we’re close to the finish line. Before we get there, we only have two obstacles to clear: implementing addressing modes to handle operands, and building a program runner that knows how to map raw 6502 code to the operation definitions shown above.

Addressing Modes

NOTE: The explanation that follows barely scrapes the surface of this topic. If you want to really understand 6502 addressing modes, you should check out the relevant section in the Easy6502 tutorial.

In the very first exercise where we disassembled the first few instructions of Snake6502, we discovered the presence of several addressing modes that cause operands to be interpreted in various different ways. To get the game running, we will need to handle a total of eight different addressing modes.

This is a lot of different ways to generate an address, and its intimidating to realize we’re only implementing an incomplete subset of what the 6502 processor provides. However, its important to keep in mind that the only data structure we have to work with is a simple mapping from 16-bit integers to 8-bit integers. Among other things, clever indexing can give us the functionality we’d expect from variables, references, and arrays – all the stuff that doesn’t have a direct representation in machine code.

I’m going to show the definitions for all of the addressing modes used by Snake6502 below, which probably won’t make much sense at first glance. But try to see if you can figure out what some of this code doing:

module Vintage
  module Operand
    def self.read(mem, mode, x, y)
      case mode
      when "#" # Implicit 
        nil
      when "@" # Relative
        offset = mem.next

        mem.pc + (offset <= 0x80 ? offset : -(0xff - offset + 1)) 
      when "IM" # Immediate
        mem.pc.tap { mem.next }
      when "ZP" # Zero Page
        mem.next
      when "ZX" # Zero Page, X
        mem.next + x
      when  "AB" # Absolute
        mem.int16([mem.next, mem.next])
      when "IX" # Indexed Indirect
        e = mem.next

        mem.int16([mem[e + x], mem[e + x + 1]])
      when "IY" # Indirect Indexed
        e = mem.next

        mem.int16([mem[e], mem[e+1]]) + y
      else
        raise NotImplementedError, mode.inspect
      end
    end
  end
end

Now let’s walk through them one-by-one. You can refer to the source code above as needed to make sense of the following examples.

1) The implicit addressing mode is meant for instructions that either don’t operate on a memory address at all, or can infer the address internally. An example we’ve already seen is the RTS operations that is used to return from a subroutine – it gets its data from the stack rather than from an operand, making it a single byte instruction.

2) The relative addressing mode is used by branches only. Consider the following example:

$0651    f0 0d     BEQ $0660    # if Z=1, jump to $0660

By the time the $0d operand is read, the program counter will be set to $0653. If you add these two numbers together, you get the address to jump to if Z=1: $0660.

3) Immediate addressing is used when you want to have an instruction work on the operand itself. To do so, we return the operand’s address, then increment the program counter as normal. In the example below, the computed address (e) is 0x0650, and mem[e] == 0x77:

$064f    c9 77     CMP #$77

4) Zero page addressing is straightforward, it is simply refers to any address between $00 and $ff. These are convenient for storing program data in, and are faster to access because they do not require combining two bytes into a 16 bit integer. We’ve already seen copious use of this address mode throughout the examples in this article, particularly when working with keyboard input ($ff) and random number generation ($fe).

5) Zero page, X indexing is used for iterating over some simple sequences in memory. For example, Snake6502 stores the position of each part of the snakes body in byte pairs starting at memory location $10. Using this addressing mode, it is possible to walk over the array by simply incrementing the X register as you go.

6) We’ve also seen plenty of examples of absolute addressing, especially when looking at JSR operations. The only complication involved in processing these addresses is that two bytes need to be read and then assembled into a 16bit integer. But since we’ve had to do that in several places already, it should be easy enough to understand.

7) Indexed indirect addressing gives us a way to dynamically compute an address from other addresses that we’ve stored in memory. That sounds really confusing, but the following example should help clear it up. The code below is responsible for moving the snake by painting a white pixel at its updated head position, and painting a black pixel at its old tail position:

$0720    a2 00       LDX #$00
$0722    a9 01       LDA #$01
$0724    81 10       STA ($10,X) 
$0726    a6 03       LDX $03
$0728    a9 00       LDA #$00
$072a    81 10       STA ($10,X) 

The first three lines are hardcoded to look at memory locations $10 and $11 to form an address in the pixel array that refers to the new head of the snake. The next three lines do something similar for the tail of the snake, but with a twist: because the length of the snake is dynamic, it needs to be looked up from memory. This value is stored in memory location $03. So to unpack the whole thing, STA ($10, X) will take the address $10, add to it the number of bytes in the whole snake array, and then look up the address stored in the last position of that array. That address points to the snake’s tail in the pixel array, which ends up getting set to black by this instruction.

8) Indirect indexed addressing gives us yet another way to walk over multibyte structures. In nake6502, this addressing mode is only used for drawing the apple on the screen. Its position is stored in a 16-bit value stored in $00 and $01, and the following code is used to set its color to a random value:

$0719    a0 00       LDY #$00
$071b    a5 fe       LDA $fe
$071d    91 00       STA ($00),Y

There are bound to be more interesting uses of these addressing modes, but we we’ve certainly covered enough ground for now! Don’t worry if you didn’t understand this section that well, it took me many times reading the Easy6502 tutorial and the source code for Snake6502 before I figured these out myself.

6502 Simulator (finally!)

We are now finally at the point where all the hard stuff is done, and all that remains is to wire up the simulator itself. In other words, it’s time for the fun part of the project.

The input for the simulator will be a binary file containing the assembled program code for Snake6502. The bytes in that file not meant to be read as printable characters, but they can be inspected using a hex editor:

$ hexdump examples/snake.rom
0000000 20 06 06 20 38 06 20 0d 06 20 2a 06 60 a9 02 85
0000010 02 a9 04 85 03 a9 11 85 10 a9 10 85 12 a9 0f 85
0000020 14 a9 04 85 11 85 13 85 15 60 a5 fe 85 00 a5 fe
0000030 29 03 18 69 02 85 01 60 20 4d 06 20 8d 06 20 c3
0000040 06 20 19 07 20 20 07 20 2d 07 4c 38 06 a5 ff c9
0000050 77 f0 0d c9 64 f0 14 c9 73 f0 1b c9 61 f0 22 60
0000060 a9 04 24 02 d0 26 a9 01 85 02 60 a9 08 24 02 d0
0000070 1b a9 02 85 02 60 a9 01 24 02 d0 10 a9 04 85 02
0000080 60 a9 02 24 02 d0 05 a9 08 85 02 60 60 20 94 06
0000090 20 a8 06 60 a5 00 c5 10 d0 0d a5 01 c5 11 d0 07
00000a0 e6 03 e6 03 20 2a 06 60 a2 02 b5 10 c5 10 d0 06
00000b0 b5 11 c5 11 f0 09 e8 e8 e4 03 f0 06 4c aa 06 4c
00000c0 35 07 60 a6 03 ca 8a b5 10 95 12 ca 10 f9 a5 02
00000d0 4a b0 09 4a b0 19 4a b0 1f 4a b0 2f a5 10 38 e9
00000e0 20 85 10 90 01 60 c6 11 a9 01 c5 11 f0 28 60 e6
00000f0 10 a9 1f 24 10 f0 1f 60 a5 10 18 69 20 85 10 b0
0000100 01 60 e6 11 a9 06 c5 11 f0 0c 60 c6 10 a5 10 29
0000110 1f c9 1f f0 01 60 4c 35 07 a0 00 a5 fe 91 00 60
0000120 a2 00 a9 01 81 10 a6 03 a9 00 81 10 60 a2 00 ea
0000130 ea ca d0 fb 60
0000135

The challenge that is left to be completed is to process the opcodes and operands in this file and turn them into a running program. To do that, we will make use of a CSV file that lists the operation name and addressing mode for each opcode found in file:

00,BRK,#
10,BPL,@
18,CLC,#
20,JSR,AB
# ... rest of instructions go here ...
E6,INC,ZP
E8,INX,#
E9,SBC,IM
F0,BEQ,@

Once we know the addressing mode for a given operation, we can read its operand and turn it into an address (denoted by e). And once we have that, we can execute the commands that are defined in following DSL:

# NOTE: This file contains definitions for every instruction used 
# by Snake6502. Most of the functionality here is a direct result
# of simple calls to Vintage::Storage and Vintage::CPU instances.

NOP { }
BRK { raise StopIteration }


LDA { cpu[:a] = mem[e] }
LDX { cpu[:x] = mem[e] }
LDY { cpu[:y] = mem[e] }

TXA { cpu[:a] = cpu[:x] }

STA { mem[e] = cpu[:a] }

## Counters

INX { cpu[:x] += 1 }
DEX { cpu[:x] -= 1 }

DEC { mem[e] = cpu.result(mem[e] - 1) }
INC { mem[e] = cpu.result(mem[e] + 1) } 

## Flow control

JMP { mem.jump(e) }

JSR { mem.jsr(e) }
RTS { mem.rts }

BNE { mem.branch(cpu[:z] == 0, e) }
BEQ { mem.branch(cpu[:z] == 1, e) }
BPL { mem.branch(cpu[:n] == 0, e) }
BCS { mem.branch(cpu[:c] == 1, e) }
BCC { mem.branch(cpu[:c] == 0, e) }

## Comparisons

CPX do 
  cpu.carry_if(cpu[:x] >= mem[e])

  cpu.result(cpu[:x] - mem[e]) 
end

CMP do 
  cpu.carry_if(cpu[:a] >= mem[e])

  cpu.result(cpu[:a] - mem[e]) 
end


## Bitwise operations

AND { cpu[:a] &= mem[e] }
BIT { cpu.result(cpu[:a] & mem[e]) }

LSR do
  t = (cpu[:a] >> 1) & 0x7F
 
  cpu.carry_if(cpu[:a][0] == 1)
  cpu[:a] = t
end

## Arithmetic

SEC { cpu.set_carry   }
CLC { cpu.clear_carry }

ADC do 
  t = cpu[:a] + mem[e] + cpu[:c]

  cpu.carry_if(t > 0xff)
  cpu[:a] = t
end

SBC do
  t  = cpu[:a] - mem[e] - (cpu[:c] == 0 ? 1 : 0)

  cpu.carry_if(t >= 0)
  cpu[:a] = t
end

We can treat both the opcode lookup CSV and the instructions definitions DSL as configuration files, to be loaded into the configuration object shown below:

require "csv"

module Vintage
  class Config
    CONFIG_DIR = "#{File.dirname(__FILE__)}/../../config"

    def initialize(name)
      load_codes(name)
      load_definitions(name)
    end

    attr_reader :definitions, :codes

    private

    def load_codes(name)
      csv_data = CSV.read("#{CONFIG_DIR}/#{name}.csv")
                    .map { |r| [r[0].to_i(16), [r[1].to_sym, r[2]]] }

      @codes = Hash[csv_data]
    end

    def load_definitions(name)
      @definitions = {}

      instance_eval(File.read("#{CONFIG_DIR}/#{name}.rb"))
    end

    def method_missing(id, *a, &b)
      return super unless id == id.upcase

      @definitions[id] = b
    end
  end
end

Then finally, we can tie everything together with a Simulator object that instantiates all the objects we need, and kicks off a program execution loop:

module Vintage
  class Simulator
    EvaluationContext = Struct.new(:mem, :cpu, :e)
      
    def self.run(file, ui)
      config = Vintage::Config.new
      cpu    = Vintage::CPU.new
      mem    = Vintage::Storage.new

      mem.extend(MemoryMap)
      mem.ui = ui
      
      mem.load(File.binread(file).bytes)

      loop do
        code = mem.next

        op, mode = config.codes[code]
        if name
          e = Operand.read(mem, mode, cpu[:x], cpu[:y])

          EvaluationContext.new(mem, cpu, e)
                           .instance_exec(&config.definitions[op])
        else
          raise LoadError, "No operation matches code: #{'%.2x' % code}"
        end
      end
    end
  end
end

At this point, you’re ready to play Snake! Or if you’ve been following closely along with this article all the way to the end, you’re probably more likely to have a cup of coffee or take a nap from information overload. Either way, congratulations for making it all the way through this long and winding issue of Practicing Ruby!

Making incremental improvements to legacy systems

Thu, 05 Sep 2013 00:00:00 +0000

When you look at this photograph of highway construction, what do you see?

If your answer was “ugly urban decay”, then you are absolutely right! But because this construction project is only a few miles away from my house, I can tell you a few things about it that reveal a far more interesting story:

On the far left side of the photo, you can see the first half of a newly constructed suspension bridge. At the time this picture was taken, it was serving five lanes of northbound traffic.
Directly next to that bridge, cars are driving southbound on what was formerly the northbound side of our old bridge, serving 3 lanes of traffic.
Dominating the rest of the photograph is the mostly deconstructed southbound side of our old bridge, a result of several months of active work.

So with those points in mind, what you are looking at here is an incremental improvement to a critical traffic bottleneck along the main route between New York City and Boston. This work was accomplished with hardly any service interruptions, despite the incredibly tight constraints on the project. This is legacy systems work at the highest level, and there is much we can learn from it that applies equally well to code as it does to concrete.

Case study: Improving one of Practicing Ruby’s oldest features

Now that we’ve set the scene with a colorful metaphor, it is time to see how these ideas can influence the way we work on software projects. To do that, I will walk you through a major change we made to practicingruby.com that involved a fair amount of legacy coding headaches. You will definitely see some ugly code along the way, but hopefully a bit of cleverness will shine through as well.

The improvement that we will discuss is a complete overhaul of Practicing Ruby’s content sharing features. Although I’ve encouraged our readers to share our articles openly since our earliest days, several awkward implementation details made this a confusing process:

You couldn’t just copy-paste links to articles. You needed to explictly click a share button that would generate a public share link for you.
If you did copy-paste an internal link from the website rather than explicitly generating a share link, those who clicked on that link would be immediately asked for registration information without warning. This behavior was a side-effect of how we did authorization and not an intentional “feature”, but it was super annoying to folks who encountered it.
If you visited a public share link while logged in, you’d see the guest view rather than the subscriber view, and you’d need to click a “log in” button to see the comments, navbar, etc.
Both internal paths and share paths were completely opaque (e.g. “articles/101” and “/articles/shared/zmkztdzucsgv”), making it hard to know what a URL pointed to without visiting it.

Despite these flaws, subscribers did use Practicing Ruby’s article sharing mechanism. They also made use of the feature in ways we didn’t anticipate – for example, it became the standard workaround for using Instapaper to read our content offline. As time went on, we used this feature for internal needs as well, whether it was to give away free samples, or to release old content to the public. To make a long story short, one of our most awkward features eventually also became one of the most important.

We avoided changing this system for quite a long while because we always had something else to work on that seemed more urgent. But after enough time had passed, we decided to pay down our debts. In particular, we wanted to make the following changes:

We wanted to switch to subscriber-based share tokens rather than generating a new share token for each and every article. As long as a token was associated with an active subscriber, it could then be used to view any of our articles.
We wanted to clean up and unify our URL scheme. Rather than having internal path like “/articles/101” and share path like “/articles/shared/zmkztdzucsgv”, we would have a single path for both purposes that looked like this:

/articles/improving-legacy-systems?u=dc2ab0f9bb

We wanted to make sure to be smart about authorization. Guests who visited a link with a valid share key would always see the “guest view” of that article, and logged in subscribers would always see the “subscriber view”. If a key was invalid or missing, the guest would be explicitly told that the page was protected, rather than dropped into our registration process without warning.
We wanted to make sure to make our links easy to share by copy-paste, whether it was from anywhere within our web interface, from the browser location bar, or even in the emails we send to subscribers. This meant making sure we put your share token pretty much anywhere you might click on an article link.

Laying out this set of requirements helped us figure out where the destination was, but we knew intuitively that the path to get there would be a long and winding road. The system we initially built for sharing articles did not take any of these concepts into account, and so we would need to find a way to shoehorn them in without breaking old behavior in any significant way. We also would need to find a way to do this incrementally, to avoid releasing a ton of changes to our system at once that could be difficult to debug and maintain. The rest of this article describes how we went on to do exactly that, one pull request at a time.

NOTE: Throughout this article, I link to the “files changed” view of pull requests to give you a complete picture of what changed in the code, but understanding every last detail is not important. It’s fine to dig deep into some pull requests while skimming or skipping others.

Step 1: Deal with authorization failures gracefully

When we first started working on practicingruby.com, we thought it would be convenient to automatically handle Github authentication behind the scenes so that subscribers rarely needed to explicitly click a “sign in” button in order to read articles. This is a good design idea, but we only really considered the happy path while building and testing it.

Many months down the line, we realized that people would occasionally share internal links to our articles by accident, rather than explicitly generating public links. Whenever that happened, the visitor would be put through our entire registration process without warning, including:

Approving our use of Github to authorize their account
Going through an email confirmation process
Getting prompted for credit card information

Most would understandably abandon this process part of the way through. In the best case scenario, our application’s behavior would be seen as very confusing, though I’m sure for many it felt downright rude and unpleasant. It’s a shame that such a bad experience could emerge from what was actually good intentions both on our part and on whoever shared a link to our content in the first place. Think of what a different experience it might have been if the visitor had been redirected to our landing page where they could see the following message:

Although that wouldn’t be quite as nice as getting free access to an article that someone wanted to share with them, it would at least avoid any confusion about what had just happened. My first attempt at introducing this kind of behavior into the system looked like what you see below:

class ApplicationController < ApplicationController::Base
  # ...
  
  def authenticate
    return if current_authorization 
   
    flash[:notice] = 
      "That page is protected. Please sign in or sign up to continue"
      
    store_location
    redirect_to(root_path)
  end
end 

We deployed this code and for a few days, it seemed to be a good enough stop-gap measure for resolving this bug, even if it meant that subscribers might need to click a “sign in” button a little more often. However, I realized that it was a bit too naive of a solution when I received an email asking why it was necessary to click “sign in” in order to make the “subscribe” button work. My quick fix had broken our registration system. :cry:

Upon hearing that bad news, I immediately pulled this code out of production after writing a test that proved this problem existed on my feature branch but not in master. A few days later, I put together a quick fix that got my tests passing. My solution was to extract a helper method that decided how to handle authorization failures. The default behavior would be to redirect to the root page and display an error message as we did above, but during registrations, we would automatically initiate a Github authentication as we had done in the past:

class ApplicationController < ApplicationController::Base
  # ...
  
  def authenticate
    return if current_authorization 
   
    store_location
    redirect_on_auth_failure
  end
  
  def redirect_on_auth_failure
    flash[:notice] = 
      "That page is protected. Please sign in or sign up to continue"
      
    redirect_to(root_path)
 end
end

class RegistrationController < ApplicationController
  # ...
  
  def redirect_on_auth_failure
    redirect_to login_path 
  end
end 

This code, though not especially well designed, seemed to get the job done without too much trouble. It also served as a useful reminder that I should be on the lookout for holes in the test suite, which in retrospect should have been obvious given the awkward behavior of the original code. As they say, hindsight is 20/20!

HISTORY: Deployed 2013-07-26 and then reverted a few days later due to the registration bug mentioned above. Redeployed on 2013-08-06, then merged three days later.

View complete diff

Step 2: Add article slugs

When we first started working on practicingruby.com, we didn’t put much thought to what our URLs looked like. In the first few weeks, we were rushing to get features like syntax highlighting and commenting out the door while keeping up with the publication schedule, and so we didn’t have much energy to think about the minor details.

Even if it made sense at the time, this is one decision I came to regret. In particular, I disliked the notion that the paths that subscribers saw (e.g. “/articles/101”) were completely different than the ones we generated for public viewing (e.g. “/articles/shared/zmkztdzucsgv”), with no direct way to associate the two. When you add in the fact that both of these URL schemes are opaque, it definitely stood out as a poor design decision on our part.

Technically speaking, it would be possible to unify the two different schemes using subscriber tokens without worrying about the descriptiveness of the URLs, perhaps using paths like “/articles/101?u=dc20f9bb”. However, since we would need to be messing around with article path generation as it was, it seemed like a good idea to make those paths much more attractive by adding slugs. The goal was to have a path like: “/articles/improving-legacy-systems?u=dc2ab0f9bb”.

Because we knew article slugs would be easy to implement, we decided to build and ship them before moving on to the more complicated changes we had planned to make. The pair of methods below are the most interesting implementation details from this changeset:

class Article < ActiveRecord::Base
  # ...

  def self.[](key)
    find_by_slug(key) || find_by_id(key)
  end

  def to_param
    if slug.present?
      slug
    else
      id.to_s
    end
  end
end

The Article[] method is a drop-in replacement for Article.find that allows lookup by slug or by id. This means that both Article[101] and Article['improving-legacy-code'] are valid calls, each of them returning an Article object. Because we only call Article.find() in a few places in our codebase, it was easy to swap those calls out to use Article[] instead.

The Article#to_params method is used internally by Rails to generate paths. So wherever article_url or article_path get called with an Article object, this method will be called to determine what gets returned. If the article has a slug associated, it’ll return something like “/articles/improving-legacy-code”. If it doesn’t have a slug set yet, it will return the familiar opaque database ids, i.e. “/articles/101”.

There is a bit of an inconsistency in this design worth noting: I chose to override the to_params method, but not the find method on my model. However, since the former is a method that is designed to be overridden and the latter might be surprising to override, I felt somewhat comfortable with this design decision.

Although it’s not worth showing the code for it, I also added a redirect to the new style URLs whenever a slug existed for an article. By doing this, I was able to effectively deprecate the old URL style without breaking existing links. While we won’t ever disable lookup by database ID, this at least preserves some consistency at the surface level of the application.

HISTORY: Deployed 2013-08-16 and then merged the next day. Adding slugs to articles was a manual process that I completed a few days after the feature shipped.

View complete diff

In theory it should have been nearly trivial to implement subscriber-based share tokens. After all, we were simply generating a random string for each subscriber and then appending it to the end of article URLs as a GET parameter (e.g. “u=dc20f9bb”). In practice, there were many edge cases that would complicate our implementation.

The ideal situation would be to override the article_path and article_url methods to add the currently logged in user’s share token to any article links throughout the application. However, we weren’t able to find a single place within the Rails call chain where such a global override would make sense. It would easy enough to get this kind of behavior in both our views and controllers by putting the methods in a helper and then mixing that helper into our ApplicationController, but it wasn’t easy to take the same approach in our tests and mailers. To make matters worse, some of the places we wanted to use these path helpers would have access to the ones rails provided by default, but would not include our overrides, and so we’d silently lose the behavior we wanted to add.

We were unable to find an elegant solution to this problem, but eventually settled on a compromise. We built a low level object for generating the URLs with subscriber tokens, as shown below:

class ArticleLink
  include Rails.application.routes.url_helpers

  def initialize(article, params)
    self.article = article
    self.params = params
  end

  def path(token)
    article_path(article, params_with_token(token))
  end

  def url(token)
    article_url(article, params_with_token(token))
  end

  private

  attr_accessor :params, :article

  def params_with_token(token)
    {:u => token}.merge(params)
  end
end

Then in our ApplicationHelper, we added the following bits of glue code:

module ApplicationHelper
  def article_url(article, params={})
    return super unless current_user

    ArticleLink.new(article, params).url(current_user.share_token)
  end

  def article_path(article, params={})
    return super unless current_user

    ArticleLink.new(article, params).path(current_user.share_token)
  end
end

Adding these simple shims made it so that we got the behavior we wanted in the ordinary use cases of article_url and article_path, which were in our controllers and views. In our mailers and tests, we opted to use the ArticleLink object directly, because we needed to explicitly pass in tokens in those areas anyway. Because it was impossible for us to make this code completely DRY, this convention-based design was the best we could come up with.

As part of this changeset, I modified the redirection code that I wrote when we were introducing slugs to also take tokens into account. If a subscriber visited a link that didn’t include a share token, it would rewrite the URL to include their token. This was yet another attempt at introducing a bit of consistency where there previously was none.

HISTORY: Deployed code to add tokens upon visiting an article on 2013-08-20, then did a second deploy to update the archives and library links the next day, merged on 2013-08-23.

View complete diff

Step 4: Redesign and improve broadcast mailer

I use a very basic web form in our admin panel to send email announcements out to Practicing Ruby subscribers. Originally, this feature relied on sending messages in batches, which was the simple thing to do when we assumed we’d be sending an identical message to everyone:

class BroadcastMailer < ActionMailer::Base
  def deliver_broadcast(message={})
    @body = message[:body]

    user_batches(message) do |users|
      mail(
        :to => "[email protected]",
        :bcc => users,
        :subject => message[:subject]
      ).deliver
    end
  end

  private

  def user_batches(message)
    yield(message[:to]) && return if message[:commit] == "Test"

    User.where(:notify_updates => true).to_notify.
      find_in_batches(:batch_size => 25) do |group|
        yield group.map(&:contact_email)
    end
  end
end

Despite being a bit of a hack, this code served us well enough for a fairly long time. It even supported a basic “test mode” that allowed me to send a broadcast email to myself before sending it out everyone. However, the design would need to change greatly if we wanted to include share tokens in the article links we emailed to subscribers. We’d need to send out individual emails rather than sending batched messages, and we’d also need to implement some sort of basic mail merge functionality to handle article link generation.

I don’t want to get too bogged down in details here, but this changeset turned out to be far more complicated than I expected. For starters, the way we were using ActionMailer in our original code was incorrect, and we were relying on undefined behavior without realizing it. Because the BroadcastMailer had been working fine for us in production and its (admittedly mediocre) tests were passing, we didn’t notice the problem until we attempted to change its behavior. After attempting to introduce code that looked like this, I started to get all sorts of confusing test failures:

class BroadcastMailer < ActionMailer::Base
  # NOTE: this is an approximation, but it captures the basic idea...
  def deliver_broadcast(message={})
    @body = message[:body]

    User.where(:notify_updates => true).to_notify.each do |user|
      mail(:to => user.contact_email, :subject => message[:subject]).deliver
    end
  end
end

Even though this code appeared to work as expected in development (sending individual emails to each recipient), in my tests, ActionMailer::Base.deliveries was returning N copies of the first email sent in this loop. After some more playing around with ActionMailer and semi-fruitless internet searches, I concluded that this was because we weren’t using the mailers in the officially sanctioned way. We’d need to change our code so that the mailer returned a Mail object, rather than handling the delivery for us.

Because I didn’t want that logic to trickle up into the controller, and because I expected things might get more complicated as we kept adding more features to this object, I decided to introduce an intermediate service object to handle some of the work for us, and then greatly simplify the mailer object. I also wanted to make the distinction between sending a test message and sending a message to everyone more explicit, so I took the opportunity to do that as well. The resulting code ended up looking something similar to what you see below:

class Broadcaster
  def self.notify_subscribers(params)
    BroadcastMailer.recipients.each do |email|
      BroadcastMailer.broadcast(params, email).deliver
    end
  end

  def self.notify_testers(params)
    BroadcastMailer.broadcast(params, params[:to]).deliver
  end
end

class BroadcastMailer < ActionMailer::Base
  def self.recipients
    User.where(:notify_updates => true).to_notify.map(&:contact_email)
  end

  def broadcast(message, email)
    mail(:to => email,
         :subject => message[:subject])
  end
end

With this code in place, I had successfully converted the batch email delivery to individual emails. It was time to move on to adding a bit of code that would give me mail-merge functionality. I decided to use Mustache for this purpose, which would allow me to write emails that look like this:

Here is an awesome article I wrote:

improving-legacy-systems

Mustache would then run some code behind the scenes and turn that message body into the following output:

Here is an awesome article I wrote:

http://practicingruby.com/articles/improving-legacy-systems?u=dc20f9bb

As a proof of concept, I wrote a bit of code that handled the article link expansion, but didn’t handle share tokens yet. It only took two extra lines in BroadcastMailer#broadcast to add this support:

class BroadcastMailer < ActionMailer::Base
  # ...
  
  def broadcast(message, email)
    article_finder = ->(e) { article_url(Article[e]) }

    @body = Mustache.render(message[:body], :article => article_finder)

    mail(:to => email,
         :subject => message[:subject])
  end
end

I deployed this code in production and sent myself a couple test emails, verifying that the article links were getting expanded as I expected them to. I had planned to work on adding the user tokens immediately after running those live tests, but at that moment realized that I had overlooked an important issue related to performance.

Previous to this changeset, the BroadcastMailer was responsible for sending about 16 emails at a time (25 people per email). But now, it would be sending about 400 of them! Even though we use a DelayedJob worker to handle the actual delivery of the messages, it might take some significant amount of time to insert 400 custom-generated emails into the queue. Rather than investigating that problem right away, I decided to get myself some rest and tackle it the next day with Jordan.

HISTORY: Deployed on 2013-08-22, and then merged the next day.

View complete diff

Step 5: Test broadcast mailer’s performance

Before we could go any farther with our work on the broadcast mailer, we needed to check the performance implications of switching to non-batched emails. We didn’t need to do a very scientific test – we just needed to see how severe the slowdown was. Because our previous code ran without a noticeable delay, pretty much anything longer than a second or two would be concerning to us.

To conduct our test, we first populated our development environment with 2000 users (about 5x as many active users as we had on Practicing Ruby at the time). Then, we posted a realistic email in the broadcast mailer form, and kept an eye on the messages that were getting queued up via the Rails console. After several seconds we hadn’t even queued up 100 jobs, so it became clear that performance very well could be a concern.

To double check our estimates, and to form a more realistic test, we temporarily disabled our DelayedJob worker on the server and then ran the broadcast mailer in our live environment. Although the mailer did finish up queuing its messages without the request timing out, it took about half a minute to do so. With this information in hand, we cleared out the test jobs so that they wouldn’t actually be delivered, and then spent a bit of time lost in thought.

Ultimately, we learned several important things from this little experiment:

The mail building and queuing process was definitely slow enough to worry us.
In the worst case scenario, I would be able to deal with a 30 second delay in delivering broadcasts, but we would need to fix this problem if we wanted to unbatch other emails of ours, such as comment notifications.
The most straightforward way to deal with this problem would be to run the entire mail building and queuing process in the background.

The first two points were not especially surprising to us, but the third concerned us a bit. While we have had good luck using DelayedJob in conjunction with the MailHopper gem to send email, we had some problems in the past with trying to handle arbitrary jobs with it. We suspected this had to do with some of our dependencies being outdated, but never had time to investigate properly. With our fingers crossed, we decided to hope for the best and plan for the worst.

Step 6: Process broadcast mails using DelayedJob

Our first stab at backgrounding the work done by Broadcaster.notify_subscribers was to simply change the call to Broadcaster.delay.notify_subscribers.

In theory, this small change should have done the trick: the method is conceptually nothing more than a “fire and forget” function that did not need to interact in any way with its caller. But after spending a long time staring at an incredibly confusing error log, we realized that it wasn’t safe to assume that DelayedJob would cleanly serialize a Rails params hash. Constructing our own hash to pass into the Broadcaster.notify_subscribers method resolved those issues, and we ended up with the following code in BroadcastsController:

module Admin
  class BroadcastsController < ApplicationController
    def create
      # ...

      # build our own hash to avoid DelayedJob serialization issues
      message = { :subject => params[:subject],
                  :body    => params[:body] } 

      if params[:commit] == "Test"
        message[:to] = params[:to]

        Broadcaster.notify_testers(message)
      else
        Broadcaster.delay.notify_subscribers(message)
      end

      # ...
    end
  end
end

After tweaking our test suite slightly to take this change into account, we were back to green fairly quickly. We experimented with the delayed broadcasts locally and found that it resolved our slowness issue in the UI. The worker would still take a little while to build all those mails and get them queued up, but since it was being done in the background it no longer was much of a concern to us.

We were cautiously optimistic that this small change might fix our issues, so we deployed the code to production and did another live test. Unfortunately, this lead us to a new error condition, and so we had to go back to the drawing board. Eventually we came across this Github issue, which hinted (indirectly) that we might be running into one of the many issues with YAML parsing on Ruby 1.9.2.

We could have attempted to do yet another workaround to avoid updating our Ruby version, but we knew that this was not the first, second, or even third time that we had been bitten by the fact that we were still running an ancient and poorly supported version of Ruby. In fact, we realized that wiping the slate clean and provisioning a whole new VPS might be the way to go, because that way we could upgrade all of our platform dependencies at once.

So with that in mind, Jordan went off to work on getting us a new production environment set up, and we temporarily put this particular changeset on hold. There was still plenty of work for me to do that didn’t rely on upgrading our production environment, so I kept working against our old server while he tried to spin up a new one.

HISTORY: Deployed for live testing on 2013-08-23 but then immediately pulled from production upon failure. Redeployed to our new server on 2013-08-30, then merged the following day.

View complete diff

Now that we had investigated the performance issues with the mailer and had a plan in place to fix them, it was time for me to finish what I had planned to work on in the first place: adding share tokens to article links in emails.

The changes to BroadcastMailer were fairly straightforward: pass a User rather than an email address into the broadcast method, and then use ArticleLink to generate a customized link based on the subscriber’s share token:

class BroadcastMailer < ActionMailer::Base
  def self.recipients
    User.where(:notify_updates => true).to_notify
  end

  def broadcast(message, subscriber)
    article_finder = ->(e) { 
      ArticleLink.new(Article[e]).url(subscriber.share_token) 
    }

    @body = Mustache.render(message[:body], :article => article_finder)

    mail(:to => subscriber.contact_email,
         :subject => message[:subject])
  end
end

The only complication of rewiring BroadcastMailer this way is that it broke our test mailer functionality. Because the test mailer could send a message to any email address (whether there was an account associated with it or not), we wouldn’t be able to look up a valid User record to pass to the BroadcastMailer. The code below shows my temporary solution to this API compatibility problem:

class Broadcaster
   # ...
   
  def self.notify_testers(params)
    subscriber = Struct.new(:contact_email, :share_token)
                       .new(params[:to], "testtoken")

    BroadcastMailer.broadcast(params, subscriber).deliver
  end
end

Using a Struct object to generate an interface shim as I’ve done here is not the most elegant solution, but it gets the job done. A better solution would be to create a container object that could be used by both notify_subscribers and notify_testers, but I wasn’t ready to make that design decision yet.

With these changes in place, I was able to do some live testing to verify that we had managed to get share tokens into our article links. Now all that remained was to add the logic that would allow these share tokens to permit guest access to articles.

HISTORY: Deployed 2013-08-24, then merged on 2013-08-29.

View complete diff

With all the necessary underplumbing in place, I was finally ready to model the new sharing mechanism. The end goal was to support the following behavior:

Subscribers see the full article w. comments whenever they are logged in
With a valid token in the URL, guests see our “shared article” view.
Without a valid token, guests see the “protected page” error
Links that use a token from an expired account are disabled
Old-style share links redirect to the new-style subscriber token links

The main challenge was that there wasn’t an easy way to separate these concepts from each other, at least not in a meaningful way. However, we were able to reuse large chunks of existing code to do this, so most of the work was just tedious rewiring of controller actions while layering in a few more tests here and there.

The changes that needed to be made to support these behaviors were not that hard to make, but I did feel concerned about how complicated our ArticlesController#show action was getting. Including the relevant filters, here is what it looked like after all the changes were made (skim it, but don’t bother trying to understand it!):

class ArticlesController < ApplicationController
  before_filter :find_article, :only => [:show, :edit, :update, :share]
  before_filter :update_url, :only => [:show]
  before_filter :validate_token, :only => [:show]

  skip_before_filter :authenticate, :only => [:show, :shared, :samples]
  skip_before_filter :authenticate_user, :only => [:show, :shared, :samples]

  def show
    store_location
    decorate_article

    if current_user
      mixpanel.track("Article Visit", :title => @article.subject,
                                      :user_id => current_user.hashed_id)

      @comments = CommentDecorator.decorate(@article.comments
                                                    .order("created_at"))
    else
      shared_by = User.find_by_share_token(params[:u]).hashed_id

      mixpanel.track("Shared Article Visit", :title => @article.subject,
                                            :shared_by => shared_by)

      render "shared"
    end
  end

  private

  def find_article
    @article = Article[params[:id]]

    render_http_error(404) unless @article
  end

  def update_url
    slug_needs_updating = @article.slug.present? && params[:id] != @article.slug
    missing_token = current_user && params[:u].blank?

    redirect_to(article_path(@article)) if slug_needs_updating || missing_token
  end

  def validate_token
    return if current_user.try(:active?)

    unless params[:u].present? && 
           User.find_by_share_token_and_status(params[:u], "active")
      attempt_user_login # helper that calls authenticate + authenticate_user
    end
  end
end

This is clearly not a portrait of healthy code! In fact, it looks suspiciously similiar to the code samples that “lost the plot” in Avdi Grimm’s contributed article on confident coding. That said, it’s probably more fair to say that there wasn’t much of a well defined plot when this code was written in the first place, and my attempts to modify it only muddied things further.

It was hard for me to determine whether or not I should attempt to refactor this code right away or wait until later. From a purely technical perspective, the answer was obvious that this code needed to be cleaned up. But looking at it from another angle, I wanted to make sure that the external behavior of the system was what I actually wanted before I invested more time into optimizing its implementation. I didn’t have insight at this point in time to answer that question, so I decided to leave the code messy for the time being until I had a chance to see how well the new sharing mechanism performed in production.

HISTORY: Deployed on 2013-08-26 and then merged on 2013-08-29.

View complete diff

Step 9: Get practicingruby.com running on our new VPS

While I kept working on the sharing mechanism, Jordan was busy setting up a new production environment for us. We set up a temporary subdomain new.practicingruby.com for this purpose, and that allowed us to do some live testing while he got everything up and running.

Technically speaking, we only need to upgrade our Ruby version in order to fix the problems we encountered with DelayedJob, so spinning up a new VPS instance purely for that purpose might sound a bit overkill at first glance. However, starting with a blank slate environment allowed us to upgrade the rest of our serverside dependencies at a relaxed pace, without worrying about potentially causing large amounts of site downtime. Spinning up the new environment in parallel before decommissioning the old one also meant that we could always switch back to our old environment if we encountered any problems during the migration to the new server.

On 2013-08-30, we decided to migrate to the new environment. The first step was to pull in the delayed broadcast mailer code and do live tests similar to the ones we had done earlier. After we found that those went smoothly, we decided to do a complete end-to-end test by using the new system to deliver an announcement about the planned maintenance downtime. That worked without any issues, and so at that time we were ready to perform the cut over.

We made sure to copy over all the data from our old environment immediately after putting it into maintenance mode, and then we updated our DNS entries to point practicingruby.com at the new server. After the DNS records propagated, we used a combination of watching our logs and our analytics dashboard (Mixpanel) to see how things were going. There were only two minor hiccups before everything got back to normal:

We had a small issue with our Oauth configuration on Github, but resolving it was trivial after we realized that it was not in fact a DNS-related problem, but an issue on our end.
We realized as soon as we spun up our cron jobs on the new server that our integration with Mailchimp’s API had broken. The gem we were using was not Ruby 2.0 compatible, but we never realized this in development because we use it only in a tiny cleanup script that runs behind the scenes on the server. Thankfully because we had isolated this dependency from our application code, changing it was very easy.

These two issues were the only problems we needed to debug under pressure throughout all the work we described in this article. Given how much we changed under the hood, I am quite proud of that fact.

Reflections

The state of practicingruby.com immediately after our server migration was roughly comparable to that of the highway construction photograph that you saw at the beginning of this article: some improvements had been made, but there was still plenty of old cruft left around, and lots of work left to be done before things could be considered finished. My goal in writing this article was not to show a beautiful end result, but instead to illustrate a process that is seldom discussed.

In the name of preserving realism, I dragged you through some of our oldest and worst code, and also showed you some newer code that isn’t much nicer looking than our old stuff. Along the way, we used countless techniques that feel more like plumbing work than interesting programming work. Each step along the way, we used a different technique to glue one bit of code to another bit of code without breaking old behaviors, because there was no good one-size fits all solution to turn to. We got the job done, but we definitely got our hands dirty in the process.

I feel fairly confident that some of the changes I showed in this article are ones that I will be thankful for in the long haul, while others I will come to regret. The trouble of course is knowing which will be which, and only time and experience can get me there. But hopefully by sharing my own experiences with you, you can learn something from my mistakes, too!

Special thanks goes to Jordan Byron (the maintainer of practicingruby.com) for collaborating with me on this article, and for helping Practicing Ruby run smoothly over the years.

Using data to explore questions about our lives

Tue, 06 Aug 2013 00:00:00 +0000

This issue was a collaboration with my wife, Jia Wu. Jia is an associate scientist at the Yale Child Study Center, where she spends a good portion of her time analyzing brainwave data from various EEG experiments. Although this article focuses on very basic concepts, her background in statistical programming was very helpful whenever I got stuck on something. That said, if you find any mistakes in this article, you can blame me, not her.

One human quirk that fascinates me is the huge disparity between our moment-to-moment experiences and our perception of past events. This is something that I’ve read about a lot in pop-psych books, and also is one of the main reasons that I practice insight meditation. However, it wasn’t until I read Daniel Kahneman’s book “Thinking, Fast and Slow” that I realized just how strongly separated our experiencing self is from our remembering self.

In both Kahneman’s book and his talk at TED 2010, he uses a striking example comparing two colonoscopy patients who recorded their pain levels periodically throughout their procedure. Although modern sedation techniques have made this a much less painful procedure, no anethesia was used during this study, which pretty much guaranteed that both patients would be in for an unpleasant experience.

From the data Kahneman shows, the first patient had a much shorter procedure and reported much less overall pain than the second patient. However, when asked later about how painful their colonoscopy were, the first patient remembered it to be much more unpleasant than the second patient did. How can that be?

As it turns out, how an event ends has a lot to do with how we will perceive the overall experience when we recall it down the line. In the colonoscopy study, the first patient reported a high pain spike immediately before the end of their procedure, where the second patient had pain that was gradually reduced before the procedure ended. This is the explanation Kahneman offers as to why the first patient remembered their colonoscopy to be far worse of an experience than the second patient remembered it to be.

This disparity between experience and memory isn’t just a one-off observation – it’s a robust finding, and it is has been repeated in many different contexts. The lesson to be learned here is that we cannot trust our remembering mind to give a faithful account of the things we experience day-to-day. The unfortunate cost that comes along with this reality is that we’re not as good about making judgements about our own well being as we could be if we did not have this cognitive limitation.

I thought about this idea for a long time, particularly as it related to my day-to-day happiness. Like most software developers (and probably all writers), my work has a lot of highs and lows to it – so my gut feeling was that my days could be neatly divided into good days and bad days. But because Kahneman had taught me that my intuitions couldn’t be trusted, I eventually set out to turn this psychological problem into an engineering problem by recording and analyzing my own mood ratings over time.

Designing an informal experiment

I wanted my mood study to be rigorous enough to be meaningful on a personal level, but I had no intentions of conducting a tightly controlled scientific study. What I really wanted was to build a simple breadcrumb trail of mood ratings so that I didn’t need to rely on memory alone to gauge my overall sense of well-being over time.

After thinking through various data collection strategies, I eventually settled on SMS messages as my delivery mechanism. The main reason for going this route was that I needed a polling device that could follow me everywhere, but one that wouldn’t badly disrupt whatever I was currently doing. Because I use a terrible phone that pretty much can only be used for phone calls and texting, this approach made it possible for me to regularly update my mood rating without getting sucked into all the things that would distract me on a computer.

To make data entry easy, I used a simple numerical scale for tracking my mood:

Very Happy (9): No desire to change anything about my current experience.
Happy (7-8): Pleased by the current experience, but may still be slightly tired, distracted, or anxious.
Neutral (5-6): Not bothered by my current experience, but not necessarily enjoying it.
Unhappy (3-4): My negative feelings are getting in the way of me doing what I want to do.
Very Unhappy (1-2): Unable to do what I want because I am overwhelmed with negative feelings.

Originally I had intended to collect these mood updates over the course of several weeks without any specific questions in mind. However, Jia convinced me that having at least a general sense of what questions I was interested in would help me organize the study better – so I started to think about what I might be able to observe from this seemingly trivial dataset.

After a short brainstorming session, we settled on the following general questions:

How stable is my mood in general? In other words, how much variance is there over a given time period?
Are there any patterns in the high and low points that I experience each day? How far apart are the two?
Does day of the week and time of day have any effect on my mood?

These questions helped me ensure that the data I intended to collect was sufficient. Once we confirmed that was the case, we were ready to start writing some code!

Building the necessary tools

To run this study, I used two small toolchains: one for data collection, and one for reporting.

The job of the data collection toolchain was primarily to deal with sending and receiving text messages at randomized intervals. It stored my responses into database records similar to what you see below:

[{:id=>485, :message=>"8", :recorded_at=>1375470054},
 {:id=>484, :message=>"8", :recorded_at=>1375465032},
 {:id=>483, :message=>"8", :recorded_at=>1375457397},
 {:id=>482, :message=>"9", :recorded_at=>1375450750},
 {:id=>481, :message=>"8", :recorded_at=>1375411347}, ...]

To support this workflow, I relied almost entirely on external services, including Twilio and Heroku. As a result, the whole data collection toolchain I built consisted of around 80 lines of code spread across two simple rake tasks and a small Sinatra-based web service. Here’s the basic storyline that describes how these two little programs work:

Every ten minutes between 8:00am and 11:00pm each day, the randomizer in the app:remind task gets run. It has a 1:6 chance of triggering a mood update reminder.
Whenever the randomizer sends a reminder, it does so by hitting the /send-reminder route on my web service, which causes Twilio to deliver a SMS message to my phone.
I respond to those messages with a mood rating. This causes Twilio to fire a webhook that hits the /record-mood route on the Sinatra app with the message data as GET parameters. The response data along with a timestamp are then stored in a database for later processing.
Some time later, the reporting toolchain will hit the /mood-logs.csv route to download a dump of the whole dataset, which includes the raw data shown above along with a few other computed fields that make reporting easier.

After a bit of hollywood magic involving a menagerie of R scripts, some more rake tasks, and a bit of Prawn-based PDF generation code, the reporting toolchain ends up spitting out a two-page PDF report that looks like what you see below:

We’ll be discussing some of the details about how the various graphs get generated and the challenges involved in implementing them later on in this article, but if you want to get a sense of what the Ruby glue code looks in the reporting toolchain, I’d recommend looking at its Rakefile. The tasks it provides allow me to type rake generate-report in a console and cause the following chain of events to happen:

The latest mood data get downloaded from the Sinatra app in CSV format.
All of the R-based graphing scripts are run, outputting a bunch of image files.
A PDF is generated to cleanly present those images in a single document.
The CSV data and image files are then be deleted, because they’re no longer needed.

Between this reporting code and the data aggregation toolchain, I ended up with a system that has been very easy to work with for the many weeks that I have been running this study. The whole user experience boils down to pressing a couple buttons on my phone when I’m prompted to do so, and then typing a single command to generate reports whenever I want to take a look at them.

At a first glance, the way this system is implemented may look a bit like its hung together with shoestrings and glue, but the very loose coupling between its components has made it easy to both work on individual pieces in isolation, and to make significant changes without a ton of rework. It seems like the worse is better mantra applies well to this sort of project.

I’d be happy to discuss the design of these two toolchains with you once you’ve finished this article, but for now let’s look at what all those graphs are saying about my mood.

Analyzing the results

The full report for my mood study consists of four different graphs generated via the R stats language, each of which gives us a different way of looking at the data:

Figure 1 provides a view of the average mood ratings across the whole time period
Figure 2 tracks the daily minimum and maximums for the whole time period.
Figure 3 shows the average mood rating and variance broken out by day of week
Figure 4 plots the distribution of the different mood ratings at various times of day.

The order above is the same as that of the PDF report, and it is essentially sorted by the largest time scales down to the shortest ones. Since that is a fairly natural way to look at this data, we’ll discuss it in the same order in this article.

Figure 1 (view source code):

I knew as soon as I started working on this study that I’d want to somehow capture the general trend of the entire data series, but I didn’t anticipate how noisy it would be to plot nearly 500 data points, many of which were too close together to visually distinguish from one another. To lessen the noise, I decided to plot a moving average instead of the individual ratings over time, which is what you see in Figure 1 above.

It’s important to understand the tradeoffs here: by smoothing out the data, I lost the ability to see what the individual ratings were at any given time. However, I gained the ability to easily discern the following bits of useful information:

How my experiences over a period of a couple days compare to the global average (green horizontal line), and to the global standard deviation (gray horizontal lines). This information could tell me whether my day-to-day experience has been improving or getting worse over time, and also how stable the swings in my mood have been recently compared to what might be considered “typical” for me across a large time span.
Whether my recent mood updates indicated that my mood was trending upward or downward, and roughly how long I could expect that to last.

Without rigorous statistical analysis and a far less corruptable means of studying myself, these bits of information could never truly predict the future or even be used as the primary basis for decision making. However, the extra information has been helping me put my mind in a historical perspective that isn’t purely based on my remembered experiences, and that alone has turned out to be extremely useful to me.

Implementation notes (view source code):

I chose to use an exponentially-smoothed weighted average here, mostly because I wanted to see the trend line change direction as quickly as possible whenever new points of data hinted that my mood was getting better or worse over time. There are lots of different techniques for doing weighted averages, and this one is actually a little more complicated than some of the other options out there. If I had to implement the computations myself I may have chosen a more simple method. But since an exponential moving average function already existed in the TTR package, it didn’t really cost me any extra effort to model things this way.

I had first seen this technique used in The Hacker’s Diet, where it proved to be a useful means of cancelling out the noise of daily weight fluctuations so that you could see if you were actually gaining or losing weight. I was hoping it would have the same effect for me with my mood monitoring, and so far it has worked as well as I expected it would.

It’s also worth noting that in this graph, the curve represents something close to a continous time scale. To accomplish this, I converted the UNIX timestamps into fractional days from the moment the study had started. It’s not perfect, but it has the neat effect of making visible changes to the graph after even a single new data point has been recorded.

Figure 2 (view source code):

In a purely statistical sense, the highest and lowest values reported for each day might not be especially relevant. However, the nature of this particular study made me feel it was important to track them. After all, even if the “average” mood for two days were both around 7, a day where the lowest mood rating was a 1 will certainly be different sort of day than one where the lowest rating was a 5! For this reason, Figure 2 shows the extreme high and low for each day in the study. This information is useful for the following purposes:

Determining what my daily peak experiences are like on average. For example, we can see from this data that there was only one day where I didn’t report at least a single rating of 7 or higher, and that most days my high point was either an 8 or 9.
Determining what my daily low points are like on average. Reading the data shown above, we can see that there were only three days in the entire study that I reported a low rating of 1, but that about one in five days had a low rating of 4 or less.
Visualizing the range between high and low points on a daily basis. This can be seen by looking at the space between the two lines: the smaller the distance, the smaller the range of the mood swing for that day.

A somewhat obvious limitation of this visualization is that the range of moods recorded in a day do not necessarily reflect the range of moods actually experienced throughout that day. In most of the other ways I’ve sliced up the dataset, we can hope that averaging will smooth out some of the ill effects of missing information, but this view in particular can be easily corrupted by a single “missed event” per day. The key point here is that Figure 2 can only be viewed as a rough sketch of the overall trend, and not a precise picture of day-to-day experience.

Implementation notes (view source code):

This was an extremely straightforward graph to produce using standard R functions, so there isn’t too much to discuss about it. However, it’s worth pointing out for folks who are unfamiliar with R that the support for data aggregation built into the language is excellent. Here is the code that takes the raw mood log entries and rolls them up by daily minimum and maximum:

data_max <- aggregate(rating ~ day, data, max)

data_min <- aggregate(rating ~ day, data, min)

Because R is such a special-purpose language, it includes many neat data manipulation features similar to this one.

Figure 3 (view source code):

This visualization shows the mean and standard deviation for all mood updates broken out by day of week. Looking at my mood data in this way provides the following information:

Whether or not certain days of the week have better mood ratings on average than others.
Whether or not certain days of the week have more consistent mood ratings than others.
What the general ups-and-downs look like in a typical week in my life

If you look at the data points shown in Figure 3 above, you’ll see that the high points (Monday and Friday) stand out noticeably from the low points (Wednesday and Saturday). However, to see whether that difference is significant or not, we need to be confident that what we’re observing isn’t simply a result of random fluctuations and noise. This is where some basical statistical tests are needed.

To test for difference in the averages between days, we ran a one-way ANOVA test, and then did a pairwise test with FDR correction. Based on these tests we were able to show a significant difference (p < 0.01) between Monday+Wednesday, Monday+Saturday, and Friday+Saturday. The difference between Wednesday+Friday was not significant, but was close (p = 0.0547). I don’t want to get into a long and distracting stats tangent here, but if you are curious about what the raw results of the computations ended up looking like, take a look at this gist.

Implementation notes (view source code):

An annoying thing about R is that despite having very powerful graphing functionality built into the language, it does not have a standard feature for drawing error bars. We use a small helper function to handle this work, which is based on code we found in this blog post.

Apart from the errorbars issue and the calls to various statistical reporting functions, this code is otherwise functionally similar to what is used to generate Figure 2.

Figure 4 (view source code):

The final view of the data shows the distribution of mood ratings broken out by time of day. Because the number of mood ratings recorded in each time period weren’t evenly distributed, I decided to plot the frequency of the mood rating values by percentage rather than total count for each. Presenting the data this way allows the five individual graphs to be directly compared to one another, because it ensures that they all use the same scale.

Whenever I look at this figure, it provides me with the following information:

How common various rating values are, broken out by time of day.
How stable my mood is at a given time of day
What parts of the day are more or less enjoyable than others on average

The most striking pattern I saw from the data shown above was that the percentage of negative and negative-leaning ratings gradually increased throughout the day, up until 8pm, and then they rapidly dropped back down to similar levels as the early morning. In the 8am-11am time period, mood ratings of five or under account for about 7% of the overall distribution, but in the 5pm to 8pm slot, they account for about 20% of the ratings in that time period. Finally, the whole thing falls off a cliff in the 8pm-11pm slot and the ratings of five or lower drop back down to under 7%. It will be interesting to see whether or not this pattern holds up over time.

Implementation notes (view source code):

Building this particular visualization turned out to be more complicated than I had hoped for it to be. It may be simply due to my relative inexperience with R, but I found the hist() function to be cumbersome to work with due to a bunch of awkward defaults. For example, the default settings caused the mood ratings of 1 and 2 to be grouped together, for reasons I still only vaguely understand. Also, the way that I implemented grouping by time period can probably be improved greatly.

Feedback on how to clean up this code is welcome!

Mapping a story to the data

Because this was a very personal study, and because the data itself has very low scientific validity, I shouldn’t embellish the patterns I observed with wild guesses about their root causes. However, I can’t resist, so here are some terrible narrations for you to enjoy!

I learned that although genuine bad days are actually somewhat rare in my life, when they’re bad, they can be really bad:

I learned that I probably need to get better at relaxing during my days off:

I learned that like most people, as I get tired it’s easier for me to get into a bad mood, and that rest helps recharge my batteries:

Although these lessons may not be especially profound, it is fun to see even rudimentary evidence for them in the data I collected. If I keep doing this study, I can use these observations to try out some different things in the hopes of optimizing my day-to-day sense of well being.

Reflections

Given that this article started with a story about a colonoscopy and ended with an animated GIF, I think it’s best to leave it up to you to draw your own conclusions about what you can take away from it. But I would definitely love to hear your thoughts on any part of this project, so please do share them!

Practicing Ruby

Safely evaluating user-defined formulas and calculations

First steps with the Dentaku formula evaluator

Building the web interface

Defining garden layouts as simple data tables

Implementing the formula processor

Considering the tradeoffs involved in using Dentaku

Reflections and further explorations

Problem discovery comes before problem solving

The anatomy of information in software systems

The medium is the message

The meaning of a message depends on its level of abstraction

Every expressive syntax has at least a few corner cases

Can we free ourselves from the limitations of syntax?

Separating structure from meaning via abstract types

Combining human intuition with computational rigor

The philosophical conundrum of information exchange

A self guided course on streams, files, file formats, and sockets

Sustainable maintenance: Focus on quality first

Developing a process-oriented approach towards quality

1) Let external changes drive incremental quality improvements

2) Treat all code without adequate testing as legacy code

3) Expand functionality via well-defined extension points

Reflections

Hunt the Wumpus: An exercise in creative coding

Gameplay demonstration

Implementing “Hunt the Wumpus” from scratch

Modeling rooms

Modeling the cave

Modeling the player

Defining the game rules

Additional Exercises

Infrastructure automation by example

Prologue: Why does infrastructure automation matter?

A recipe for setting up Ruby

A recipe for setting up process monitoring

A recipe for setting up an Nginx web server

A cookbook for building a (mostly) complete Rails environment

Epilogue: What are the costs of infrastructure automation?

Recommendations for further reading

Exploring low-level computing concepts with Ruby

Warmup exercise: Reverse engineering Snake6502

Memory

Memory-mapped I/O

Registers and Flags

Addressing Modes

6502 Simulator (finally!)

Further Reading

Making incremental improvements to legacy systems

Case study: Improving one of Practicing Ruby’s oldest features

Step 1: Deal with authorization failures gracefully

Step 2: Add article slugs

Step 3: Add subscriber share tokens

Step 4: Redesign and improve broadcast mailer

Step 5: Test broadcast mailer’s performance

Step 6: Process broadcast mails using DelayedJob

Step 7: Support share tokens in broadcast mailer

Step 8: Allow guest access to articles via share tokens

Step 9: Get practicingruby.com running on our new VPS

Reflections

Using data to explore questions about our lives

Designing an informal experiment

Building the necessary tools

Analyzing the results

Mapping a story to the data

Reflections