My musings about life, the universe and everything else

Building and distributing a command-line app in Go

mauricio.linhares@gmail.com (Maurício Linhares) — Fri, 01 Apr 2022 00:00:00 UTC

Thanks to its static binaries and cross-compiling, building and distributing command-line apps in Go is a breeze. You can quickly build an app and produce binaries for multiple platforms without leaving your current environment, and we’ll learn how to do just that here.

We’ll build a simple command-line HTTP client, produce binaries for Windows, Mac, and Linux and publish them to Github. You can find the repo for this post here.

We’ll try to build this client by separating the actual command line operation from the rest of the code as much as we can, so to kick it off, let’s look at the code that performs the HTTP request:

type Config struct {
   Headers            http.Header
   UserAgent          string
   Data               string
   Method             string
   Insecure           bool
   Url                *url.URL
   ControlOutput      io.Writer
   ResponseBodyOutput io.Writer
}

The config type here encapsulates the options we offer for the command-line client in a way that is independent of who is making the calls. We allow it to set custom headers, user agent, request body data, HTTP method, if we allow requests to servers with certificates we don’t trust, the URL called, and where we will print details of the process.

Next, the code that takes this config to execute the request:

package gurl

import (
   "bytes"
   "crypto/tls"
   "fmt"
   "github.com/rs/zerolog/log"
   "io"
   "net/http"
   "net/url"
   "strings"
)

type Config struct {
   Headers            http.Header
   UserAgent          string
   Data               string
   Method             string
   Insecure           bool
   Url                *url.URL
   ControlOutput      io.Writer
   ResponseBodyOutput io.Writer
}

func Execute(c *Config) error {
   var r io.Reader
   var tlsConfig *tls.Config

   if c.Data != "" {
      r = bytes.NewBufferString(c.Data)
   }

   if c.Insecure {
      tlsConfig = &tls.Config{
         InsecureSkipVerify: true,
      }
   }

   request, err := http.NewRequest(c.Method, c.Url.String(), r)
   if err != nil {
      return err
   }

   if c.UserAgent != "" {
      request.Header.Set("User-Agent", c.UserAgent)
   }

   for key, values := range c.Headers {
      for _, value := range values {
         request.Header.Add(key, value)
      }
   }

   client := http.Client{
      Transport: &http.Transport{
         TLSClientConfig: tlsConfig,
      },
      CheckRedirect: func(req *http.Request, via []*http.Request) error {
         return http.ErrUseLastResponse
      },
   }

   requestBuilder := &wrappedBuilder{
      prefix: ">",
   }

   requestBuilder.Printf("%v %v", request.Method, request.URL.String())
   requestBuilder.WriteHeaders(request.Header)
   requestBuilder.Println()

   if _, err := io.Copy(c.ControlOutput, strings.NewReader(requestBuilder.String())); err != nil {
      return err
   }

   response, err := client.Do(request)
   if err != nil {
      return err
   }

   defer func() {
      if err := response.Body.Close(); err != nil {
         log.Warn().Err(err).Str("url", c.Url.String()).Msg("failed to close response body")
      }
   }()

   responseBuilder := &wrappedBuilder{
      prefix: "<",
   }

   responseBuilder.Printf("%v %v", response.Proto, response.Status)
   responseBuilder.WriteHeaders(response.Header)
   responseBuilder.Printf("")
   responseBuilder.Println()

   if _, err := io.Copy(c.ControlOutput, strings.NewReader(responseBuilder.String())); err != nil {
      return err
   }

   _, err = io.Copy(c.ResponseBodyOutput, response.Body)
   return err
}

Here we get the config object and set up the HTTP request and client objects to perform it. The code outputs general information to ControlOutput on the config (like what curl does by adding < and > to the output to see the outgoing and incoming data). This function doesn’t care how it gets called, you can call it from a command-line operation or unit tests, and it still works the same way.

We also have a small helper object, the wrappedBuilder, to avoid repeating the prints (and simplify error handling, as it sucks to handle errors on every write to ControlOutput):

type wrappedBuilder struct {
   prefix string
   strings.Builder
}

func (w *wrappedBuilder) WriteHeaders(headers http.Header) {
   for key, values := range headers {
      for _, value := range values {
         w.Printf("%v: %v", key, value)
      }
   }
}

func (w *wrappedBuilder) Println() {
   w.WriteString("\n")
}

func (w *wrappedBuilder) Printf(s string, a ...any) {
   w.WriteString(fmt.Sprintf("%v %v\n", w.prefix, fmt.Sprintf(s, a...)))
}

Building the command line bridge

We’ll use cobra, one of the best command-line libraries available in Go, to parse the commands. In cobra everything is a cobra.Command, and you can have commands with subcommands (like git status, status here is a subcommand of git).

In our specific case, all we need is a single command, so here’s how we’ll build it:

func CreateCommand() *cobra.Command {
   config := &Config{
      Headers:            map[string][]string{},
      ResponseBodyOutput: os.Stdout,
      ControlOutput:      os.Stdout,
   }

   headers := make([]string, 0, 255)

   command := &cobra.Command{
      Use:     `gurl URL`,
      Short:   `gurl is an HTTP client`,
      Long:    `gurl is an HTTP client for a tutorial on how to build command line clients in go`,
      Args:    ArgsValidator(config),
      PreRunE: OptionsValidator(config, headers),
      RunE: func(cmd *cobra.Command, args []string) error {
         return Execute(config)
      },
   }

   command.PersistentFlags().StringSliceVarP(&headers, "headers", "H", nil, `custom headers headers to be sent with the request, headers are separated by "," as in "HeaderName: Header content,OtherHeader: Some other value"`)
   command.PersistentFlags().StringVarP(&config.UserAgent, "user-agent", "u", "gurl", "the user agent to be used for requests")
   command.PersistentFlags().StringVarP(&config.Data, "data", "d", "", "data to be sent as the request body")
   command.PersistentFlags().StringVarP(&config.Method, "method", "m", http.MethodGet, "HTTP method to be used for the request")
   command.PersistentFlags().BoolVarP(&config.Insecure, "insecure", "k", false, "allows insecure server connections over HTTPS")

   return command
}

We start by creating a Config struct with the basic options setup (and pointing to stdout) and then create a cobra.Command with the options and documentation we need. You could add more examples and better documentation to the program here. It will be up to you how much you want to add, but the more, the merrier.

With the command created, we can now add flags. They are special options to change its behavior. These all point back to fields at the Config struct, so you can add more fields to it and then add flags to control the value here. One doesn’t point directly, and it’s the headers. Let’s look at why that is:

func OptionsValidator(c *Config, headers []string) func(cmd *cobra.Command, args []string) error {
   return func(cmd *cobra.Command, args []string) error {
      for _, h := range headers {
         if name, value, found := strings.Cut(h, ":"); found {
            c.Headers.Add(strings.TrimSpace(name), strings.TrimSpace(value))
         } else {
            return newErrorWithCode(3, "header is not a valid http header separated by `:`, value was: [%v]", h)
         }
      }

      return nil
   }
}

Unfortunately, cobra doesn’t offer an easy way to parse map[string][]string from the command line (it does offer map[string]string, though). We use the PreRunE with the OptionsValidator function to parse the headers and add them to the Headers property at Config. If the header values are invalid, we return an error right away before calling the HTTP client code.

The other validation we have in place is for the arguments provided, the program requires a single argument to be called (as in gurl http://example.com/), so we verify there is exactly one parameter, and it is a valid URL before proceeding:

func ArgsValidator(c *Config) func(cmd *cobra.Command, args []string) error {
   return func(cmd *cobra.Command, args []string) error {
      if l := len(args); l != 1 {
         return newErrorWithCode(2, "you must provide a single URL to be called but you provided %v", l)
      }

      u, err := url.Parse(args[0])
      if err != nil {
         return errors.Wrapf(err, "the URL provided is invalid: %v", args[0])
      }

      c.Url = u

      return nil
   }
}

With commands and implementation done, here’s the main function that starts the program (this file lives at cmd/gurl):

package main

import (
   "github.com/mauricio/gurl"
   "github.com/rs/zerolog"
   "os"
)

func main() {
   zerolog.TimeFieldFormat = zerolog.TimeFormatUnix

   if err := gurl.CreateCommand().Execute(); err != nil {
      switch e := err.(type) {
      case gurl.ReturnCodeError:
         os.Exit(e.Code())
      default:
         os.Exit(1)
      }
   }
}

It creates the command, runs it, checks the error, and exits. Here’s the help it generates:

$   go run cmd/gurl/main.go -h
gurl is an HTTP client for a tutorial on how to build command line clients in go

Usage:
 gurl URL [flags]

Flags:
 -d, --data string     data to be sent as the request body
 -H, --headers strings   custom headers headers to be sent with the request, headers are separated by "," as in "HeaderName: Header content,OtherHeader: Some other value"
 -h, --help        help for gurl
 -k, --insecure      allows insecure server connections over HTTPS
 -m, --method string    HTTP method to be used for the request (default "GET")
 -u, --user-agent string  the user agent to be used for requests (default "gurl")

cobra formats and includes all the information we provided when setting up the command and flags. You don’t even have to set up a help command. It just adds it automatically and prints the details.

Building for multiple platforms

Now that we have all the pieces together, it’s time to produce binaries for multiple platforms so we can distribute them. Now it’s easier to clone the repo to follow along, so you don’t have to type everything.

If you don’t have any native dependencies, you can cross-compile your go app to any supported platform. Here’s what it looks like (all commands assume you have cloned the repo and are at the repo’s root folder):

# building the program for intel macs
GOOS=darwin GOARCH=amd64 go build -o gurl-mac-amd64 cmd/gurl/main.go 
# building the program for M1 macs
GOOS=darwin GOARCH=amd64 go build -o gurl-mac-arm64 cmd/gurl/main.go 
# building the program for 64 bits amd/intel linux
GOOS=linux GOARCH=amd64 go build -o gurl-linux-amd64 cmd/gurl/main.go 

You can set the GOOS environment variable to multiple operating systems (like Linux, windows, darwin, and others) and the GOARCH to multiple architectures (like amd64, arm64, and others) and go will produce a binary for that specific version that you can use. So if you’re working on a Linux box, you can make binaries for Macs and Windows without having a Mac or Windows box available. And users don’t have to install virtual machines, language runtimes, or any other dependencies to run these programs. Build and send them the binaries, and they can be used right away.

To automate it to the next level, you can also use a tool like goreleaser that can automatically build for multiple targets and even push the binaries as a release to GitHub, GitLab, or other source control repositories.

Golang’s use of static binaries for its builds makes it an incredible tool to distribute command-line programs. You can build programs for multiple platforms from any platform, and they have no dependencies, making it a breeze to distribute them. Next time you’re planning on building a tool like that, make sure you take a serious look at Go as its language.

Should you document your code?

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 03 Mar 2022 00:00:00 UTC

Time and again, the old “well-written code does not need documentation” rears its ugly head in software circles. Like all the other silver bullets and magical thinking solutions, it ignores context and the realities of building software in the real world. While well-meaning, as good code should be straightforward to follow, it’s hard to express all the decisions that led to that code directly on it, and a couple of weeks from when you wrote, none of it will make sense.

In most codebases, we have tacit and explicit knowledge. Explicit knowledge is what is documented, structured, and well defined. You don’t need to ask other people on the team about it. It’s already there and available for you to consume. Tacit knowledge is the general understanding people on the team have about the code instinctively. It is knowing changing some part of the project usually leads to bugs or that you shouldn’t change some specific configuration because it causes it to break or misbehave in exceptional cases.

The main goal of documenting code is to turn tacit knowledge into explicit, so you and everyone else don’t have to keep all this context in memory. Humans are known to be bad at memorization. We routinely misplace objects we use day-to-day, create false memories because of stories we’ve heard, and forget stuff. As much as you might think you’re going to remember the decisions and reasoning that led to the code you’re writing, you won’t. So, be selfish, think about you a month from now, looking at this code and not knowing why and how it works, you won’t have a great time.

Why did I write this?

One of the most critical effects of documenting code is defining the context you wrote it. It’s easy to look at code today and think who wrote this shit and later running git-blame finding out you wrote it. Why did I do that?

It’s likely whoever wrote the shitty code you’re looking at had good reasons to do so, and documenting such reasons makes them available to whoever is reading and touching it in the future. Document why the code does what it does, what was the context and reasoning to do it, so it’s clear to whoever is reading it why it works the way it does so they know the requirements and expected behavior.

While going through a transition for one of our projects, I noticed code that performed health checking of other services (sending HTTP requests to verify these services are reachable) did something strange. The health checker is configured with multiple endpoints for each service. If one endpoint fails, it goes out of rotation, and traffic is sent to the other endpoints that are available (if a service has HostA, HostB, and HostC and HostA is failing, we only send traffic to HostB and HostC), but if all endpoints fall all hosts go back in the rotation.

There were no comments about why it behaved like this, so it was time to go some git spelunking. An incident a long time ago happened because someone made a mistake on their service configuration and added a health check that would never succeed, taking the whole service down. In response to this incident, we decided that if all endpoints for a service fail, we should assume it was just a misconfiguration and not that it is down and bring all endpoints back. This is now an obvious comment in the code that performs health checking. There is no need to dig into git history or go through JIRA tickets to figure out why something behaves the way it does.

You want to have the reasoning that led to the code explained, so no one needs to perform code archeology to figure out why something works the way it does. Having it clear will lead to fewer incidents and issues when people think they can change the code they see as it might look wrong, stupid, or unnecessary. Until we invent crystal balls where people can find out right away what the effects of changing code will be, it’s best to say why it does what it does.

How does it do it?

Now on to the pernicious explanation of how the code works. People will frequently say you don’t need to say what it does because it is right there in front of you, but there are valid reasons for you to explain the code as it is. One is to make it easier for people to associate the how with the abstraction you’re using.

Is it using a design pattern? Make it part of the name of the class/function and add in the docs that this is the X pattern. Is it using a known algorithm? Say it in the comments that it uses that specific algorithm.

This makes it much easier for whoever is writing the code to dig into it as they don’t have to find out you used specific patterns or algorithms. It’s right there in the comments for them to prime their expectation on what they’re looking at.

Internal use vs library code

Comments also change if you’re dealing with code you’re only using once, internally in a project, or code that will be widely shared inside an organization or to the broader public. If it’s just internal code, it might make sense to mainly document the reasoning and context and be done with it. If what you’re working on is a library that other people on other teams will use, the requirements will change.

A great example is Go’s http package. Most types are documented with examples on using them, like the http.Get function. There is also a full separate package that allows you to unit test code that uses the http package, httptest.

So when writing libraries, you have to think both about how people are going to learn how to use them and what you can do to make the experience more straightforward. Will users need a particular way to perform unit testing? Should you provide mocks? Are there use cases that need specific setups you could simplify with a testing package? These are all issues you have to consider when writing code for other people to consume.

Yes, you should write docs

So, yes, you should document your code. Not only for the people that will be working on it in the future once you’ve left your job but also for you a couple of weeks from now trying to figure out why you wrote these strange lines of code you have no memory of. The less context you have to carry with you while working, the better the experience will be.

Transitioning projects to a new team

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 22 Feb 2022 00:00:00 UTC

If you’ve been long enough on the same team or job, you’ve been either the recipient or the originator of a transition. Maybe the team mission has changed, people have moved, or you are moving somewhere else, a new job or a new team and existing software need a new home. Frequently, this is a short engagement. Someone is leaving, so they schedule a couple of meetings to share knowledge, write a bunch of docs, and that’s it.

But it doesn’t have to be like this. We’re just going through disbanding a team and finding new homes for our existing projects, and we’re trying to make this less frustrating for everyone involved. Our main goal with this process is to make it easier for the other side to learn and maintain these systems.

The process involves: What to do before the transition starts; The actual transition work; What happens once the transition completes;

Pre-transition

The first question you should ask of every project planned for transition is do we need this?. You’re going to invest time in moving a resource to another team that will also have to invest in understanding and maintaining it, most likely without any previous experience. Can you kill it now and not even move?

Can you replace it with some other tool that is already available, a SaaS solution, or some other integration that would require less work than keeping the whole thing running?

Looking for an existing solution for the problem so you don’t have to maintain it could be better than keeping a project alive. You’ll have to invest time in setting up the project for the new team, and they will have to learn how it works and then maintain it (the most expensive part of running software). If you can use a tool that someone else will support at an acceptable price, a better solution might be to go that route instead of keeping the project alive.

If there’s no other available solution or it would be too risky for the business to outsource this to another company, it’s time to prepare the transition work.

Start by building an inventory of all the pieces to be moved. Source code repositories, dashboards, databases, deployments, CI pipelines, service accounts, tokens, and any other resource that the service uses. The more stuff you can find and document, the less risky the move, especially for service accounts and tokens. It’s surprisingly common for service resources to be tied directly to personal accounts, and when you move ownership somewhere else, the service breaks, and no one knows why.

With the inventory in hand, it’s time for a cleanup. Are there any resources that belong to specific people instead of a shared service account or shared roles? If there are, it’s time to move them into a shared service account or roles. This will make the transition more manageable as the new team will join the service account or role and own the resources.

Shore up documentation, playbooks, incident reviews, and standard practices the team performs. All teams have rituals to deal with their day-to-day work that might not be documented. This is the time to turn these rituals into well-defined processes the new people should know. It doesn’t mean they will have to perform the same practices the same way. You’re just letting them know how the team operates right now.

Is there a typical sequence of steps people usually follow when they think an incident is going on? Make it a playbook. Got a specific sequence you perform database maintenances? Document it. Are deployments supposed to run in a particular order? Either encode that on your CI config or document it.

Review existing documentation to verify it’s still valid and up to date. As projects age, explanations, diagrams and the like grow stale, this is the time for you to update or delete documentation that is not valid anymore. Old or wrong documentation is generally worse than not having any documentation at all. Code that performs complex computation should also be documented and thoroughly tested with examples of known failure scenarios. The new people joining have no context on the existing code or the decisions that led to its current state. Make sure it is clear why it does the work the way it does.

Define them now if you don’t have clearly defined SLOs. Odds are you have clear expectations for what the service has to do and how long it takes to do the work (including alerting when it’s not doing that on time), but you might not have them written down. Make them visible now, so the new team knows what to expect when they take this on.

Transitioning

So there’s no way for you to throw this project in the trash, eh? Time to move it elsewhere. Build a timeline with the work that needs to be done for transition. There will be meetings, coding, presentations, so make sure you’re accounting for them and, most importantly, it should have an end date. The time by which the project now belongs to the new team.

While knowledge sharing sessions are somewhat helpful, what is useful is to work on actual code building real stuff on the project. Find features that touch the code and have people on both teams pair up on building them, with the person on the original team as passengers.

You want the new team to get as much exposure to the codebase as possible, so make sure the work is done mainly with guidance from the current team but not active work. This will force the new team to go through a development environment setup and, hopefully, a complete release cycle. Building the feature, producing artifacts, and pushing them to production.

If there are on-call rotations, start with the new folks as secondaries but always reach out to them if incidents happen during working hours. You also want them to have exposure to the project when stuff goes sideways to learn how to debug issues and react to incidents.

Suppose incidents are infrequent enough that you don’t expect them to happen during the transition, roleplay something that has occurred recently. When roleplaying, make the new team the drivers. You provide the problem and let them ask questions on what is going on to get more information and respond to their actions. Your main goal here is to let them feel the system and associate how the pieces fit together to make it work.

Start moving resource ownership to the new team. Source code, machines, tokens, teams, should all either include the new team or be moved to be owned by them. It’s essential to move resources now, during the transition, as outages or issues caused by these moves will be easier to fix while the old team is still active and working on the code. If you let this happen only after all the work is done, there might be surprising behavior lurking around. Also, don’t move it all at once. Move resources one at a time, make sure they work at their new home, and go to the next one. Making multiple changes at once on a project is a surefire way to cause issues.

Present the new people to the teams you’re used to interacting with as the project’s new owners. The existing team needs to make it visible that new people are working on this, and they are the ones that people should communicate with instead of their usual contacts. There is always a lot of informal communication between teams, and it’s crucial to get the new folks into the flow, get them invited to scheduled meetings, and present them to people frequently in contact.

After transitioning

At this point, the old team has disengaged from the project entirely. They should not be on call and should not be involved in the day-to-day operation. You want a clean break for the old team because you want them to focus on the new work they’re doing and not worry about what is going on at the old thing.

That’s why it’s essential to have a firm date to when it ends, so the people leaving don’t feel like they have to wear multiple hats because they left this other project behind. It doesn’t mean they shouldn’t engage when needed, answer questions, or help out, but they should no longer have to worry about what’s going on at the old project.

I hope these tips can lead you to better transitions in and out of projects!

Optimizing your MySQL queries

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 16 Feb 2022 00:00:00 UTC

Odds are, if you’re doing any commercial programming, you’ve had to interact with a SQL database. They’re a staple in programming due to how easy it is to store and retrieve data on them and the many free and open-source options available. Still, people have trouble figuring out how to optimize their queries correctly, so let’s fix that.

While this text is focused on MySQL specifically, most of the discussion is also true for other databases. So, even if you don’t use MySQL, you should learn something new that is going to be helpful on the SQL database you’re using right now.

The database we’ll use here is the employees db from MySQL itself. This repo has a docker-compose setup on how to start and run the database so you can run the commands along.

EXPLAIN to me

The first thing you have to learn to optimize queries in relational databases is to use the EXPLAIN command. EXPLAIN runs the query you’ve created through the database engine to check what it thinks it’s going to take to run the command. It won’t run the command, but it checks how many rows it believes it will need to access to satisfy the query.

So given the following table:

CREATE TABLE employees (
    emp_no      INT             NOT NULL,
    birth_date  DATE            NOT NULL,
    first_name  VARCHAR(14)     NOT NULL,
    last_name   VARCHAR(16)     NOT NULL,
    gender      ENUM ('M','F')  NOT NULL,    
    hire_date   DATE            NOT NULL,
    PRIMARY KEY (emp_no)
);

And a query that finds all the employees named Joe:

SELECT * FROM employees WHERE first_name = "Joe";

You running a query with EXPLAIN before it all, so an EXPLAIN on the previous query would be:

EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G 

The \G here is specific to MySQL to say you want the data printed in rows. Here’s what it prints:

mysql> EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 299645
     filtered: 10.00
        Extra: Using where
1 row in set, 1 warning (0.00 sec)

So when we ask the database what it thinks about this query, its response is a bit harsh. It says it has to look for 299645 rows (this table has 300024 entries, so it’s almost every single row) to find out there are no employees named Joe. There are no keys to be used (keys here would be database indexes), and we’re using a WHERE clause to filter the results.

So, we know we have to find users given the first name, and the EXPLAIN response shows there are no indexes on this specific column, let’s add one:

CREATE INDEX employees_first_name_idx ON employees (first_name);

Now running EXPLAIN again:

mysql> EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_first_name_idx
          key: employees_first_name_idx
      key_len: 58
          ref: const
         rows: 1
     filtered: 100.00
        Extra: NULL
1 row in set, 1 warning (0.00 sec)

A single row checked!

How did this happen?

When you ask for the database to create an index on a column, it creates an optimized structure that allows you to find all rows associated with a specific value quickly. Think about it as if it maps a value (“Joe”) to all rows that have that value on the column you created the index.

Indexes are the magical solution. We should create indexes for every column in the table, and we should be good, right?

“You should have an index for all columns you use for querying a table”

This is a pretty common misconception of how indexes work in relational databases. If you have an index for every column, every query is automatically optimized as the database can use all these indexes to find the rows, but this is not true.

MySQL can, in some cases, use more than one index when querying data. If we create separate indexes on first_name and last_name and try to find a specific user with values for both, we get this:

mysql> EXPLAIN SELECT * FROM employees WHERE last_name = 'Halloran' AND first_name = 'Aleksander'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: index_merge
possible_keys: employees_first_name_idx,employees_last_name_idx
          key: employees_last_name_idx,employees_first_name_idx
      key_len: 66,58
          ref: NULL
         rows: 1
     filtered: 100.00
        Extra: Using intersect(employees_last_name_idx,employees_first_name_idx); Using where

So, we’re still doing good, only one row checked, but the extra field has an interesting reference, intersect(employees_last_name_idx,employees_first_name_idx). MySQL sees two indexes here and decides to use them both for the query. It’s still better than not having an index, but we’re going through two separate data structures to find the value instead of a single one.

If we now get an index on both last_name and first_name, this is the output:

mysql> EXPLAIN SELECT * FROM employees WHERE last_name = 'Halloran' AND first_name = 'Aleksander'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_first_name_idx,employees_last_name_idx,employees_last_first_name_idx
          key: employees_last_first_name_idx
      key_len: 124
          ref: const,const
         rows: 1
     filtered: 100.00
        Extra: NULL
1 row in set, 1 warning (0.00 sec)

The server digs into a single index to find the row that matches the expected value instead of two. Having multiple rows in an index also helps with covering indexes, a feature we’ll discuss later.

Picking index columns

Picking indexes is about how you query the table and what fields are part of the queries. When looking for employees, we want to find them by first and last name, so we have an index on both of them. When creating an index for multiple columns, the order of the columns matters.

Given we have an employee called Georgi Facello, our index would have it referenced under the entry Facello-Georgi (the index is last_name, first_name), so it only works for queries where you are looking for last_name and first_name or last_name alone. A query that only looks for first_name can’t use this index as the index matches left-to-right. For that, you’d need an index that starts with first_name.

It’s also not valid for a query where you need to find someone given a last_name or a first_name, as you’d need both columns to be the leftmost columns in the indexes used, this would be a case where having separate indexes on the columns would be helpful. MySQL would run a union on both indexes to perform the query.

Now, if you’re querying all or most columns in the index, what order should they have?

The columns with the biggest variety in values should come first. You can quickly calculate an average of how common values are in a column with a query like the following:

mysql> SELECT COUNT(DISTINCT emp_no)/COUNT(*) FROM employees;
+---------------------------------+
| COUNT(DISTINCT emp_no)/COUNT(*) |
+---------------------------------+
|             1.0000              |
+---------------------------------+

The primary key is the perfect example. Every row has a unique value, so we get 1. We now want the columns we’ll use in the index to be as close to 1 as we can, let’s look at first_name:

mysql> SELECT COUNT(DISTINCT first_name)/COUNT(*) FROM employees;
+-------------------------------------+
| COUNT(DISTINCT first_name)/COUNT(*) |
+-------------------------------------+
|               0.0042 |
+-------------------------------------+

Then look at last_name:

mysql> SELECT COUNT(DISTINCT last_name)/COUNT(*) FROM employees;
+------------------------------------+
| COUNT(DISTINCT last_name)/COUNT(*) |
+------------------------------------+
|               0.0055               |
+------------------------------------+

So last_name gives us better filtering than first_name, making it the best column on the index’s left side. Still, this is an average, and as you might have found out the hard way in other places, averages are good at hiding outliers. One here would impact query performance directly so we also want to check if there are visible outliers on these columns:

mysql> SELECT last_name, COUNT(*) AS total FROM employees GROUP BY last_name ORDER BY total DESC LIMIT 10;
+-----------+-------+
| last_name | total |
+-----------+-------+
| Baba      |  226  |
| Gelosh    |  223  |
| Coorg     |  223  |
| Sudbeck   |  222  |
| Farris    |  222  |
| Adachi    |  221  |
| Osgood    |  220  |
| Mandell   |  218  |
| Neiman    |  218  |
| Masada    |  218  |
+-----------+-------+

So no outliers here. Values seem to be pretty close to each other. Let’s look at first_name:

mysql> SELECT first_name, COUNT(*) AS total FROM employees GROUP BY first_name ORDER BY total DESC LIMIT 10;
+-------------+-------+
| first_name | total |
+-------------+-------+
| Shahab      |  295 |
| Tetsushi    |  291 |
| Elgin       |  279 |
| Anyuan      |  278 |
| Huican      |  276 |
| Make        |  275 |
| Sreekrishna |  272 |
| Panayotis   |  272 |
| Hatem       |  271 |
| Giri        |  270 |
+-------------+-------+

Not bad either. Values aren’t that far from each other. The order we decided on for the index is a pretty good one.

Now, something else you need to account for when creating indexes is how you will sort the results?

Just like the database uses the index to find rows quickly, it can also sort the results if you’re sorting on the same columns in the index order. For our last_name, first_name index it would mean to ORDER BY last_name or ORDER BY last_name, first_name. If multiple columns are sorted, they all have to be in the same direction, so either all ASC or all DESC. If they’re not all in the same order the database can’t use the index itself to sort the results and would have to resort to temporary tables to load the results and sort them.

Covering indexes

And another important reason to have indexes that hold multiple columns is the covering indexes feature that MySQL provides. When your query only loads the primary key and columns in the index, the database doesn’t even have to look at the tables to read the results. It reads everything from the index alone. Here’s what it looks like:

mysql> explain select emp_no, last_name, first_name from employees where last_name = "Baba"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_last_name_idx,employees_last_first_name_idx
          key: employees_last_first_name_idx
      key_len: 66
          ref: const
         rows: 226
     filtered: 100.00
        Extra: Using index

The hint here is Extra: Using index, which means all the data is read from the index alone. As the index already contains all the information we need (all indexes include the primary key for the table), the database loads everything from it and returns, not even reaching out to the table. This is the best-case scenario for queries, especially if your index fits into memory.

Creating multiple indexes

Creating indexes isn’t free. While they make it faster for us to find data, they also slow down any changes to the table as writing to columns with indexes will cause these indexes to be updated. So you have to strike a balance between making as many queries as possible fast but also allowing for fast INSERT/UPDATE/DELETE commands.

Summary

So, when optimizing, remember to:

Use EXPLAIN to find out how the database thinks your query will perform - check the command docs here;
Create indexes that cover the filtering and ordering you want to perform on your queries;
Evaluate the best order for the columns when creating indexes;
Try to load as much information from covering indexes as possible;

One of the best references to optimizing MySQL databases is the High Performance MySQL that is in its 4th edition and covers from database internals to how to design your database schema to make the most of MySQL. If you’re running apps on MySQL, you should read it. If you’re not using MySQL, it’s very likely there is a book just like this one for it as well, and you should invest some time in reading it.

GoF Design patterns that still make sense in Go

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 07 Feb 2022 00:00:00 UTC

Since its release in 1994, the Design Patterns book continues to be a seminal work in building software. The book created a new shared vocabulary and named these repeated solutions we see all over different codebases. So much so there have been multiple other books on design patterns, documenting even more examples. You can quickly explain that your solution is an Adapter for anyone who has read about it without detailing what an Adapter is.

The material, as expected, isn’t without critique, Peter Novig wrote his analysis on how many of these patterns are unnecessary or replaceable by simpler constructs in dynamic languages. The book was written mainly using C++ (and some Smalltalk), focusing on covering what was available in C++, so dynamic languages change or remove the need for some of these patterns.

Then, a couple of weeks ago, a Twitter thread caught my attention, saying people shouldn’t read this book anymore as many of the patterns are old or don’t make much sense in most mainstream programming languages. I still have fond memories of when I read it back in college (2003, many moons ago) and wondered if it is true? Have languages evolved past this book, or do we still use the patterns and vocabulary defined there?

So, coming from a Golang perspective, what are some of the patterns in the book that are still usable nowadays? Let’s have a look!

Builder

Builders are very much alive. They might have fancy names like functional options or fluent interfaces but the goal is still the same of the honored builder, simplify the creation of a complex object so that you don’t end up with a single function call receiving dozens of parameters.

We’ll now look at how you could set up a builder that produces *http.Request objects:

package gof_go

import (
   "context"
   "io"
   "net/http"
)

// NewBuilder creates a builder given a URL, we're going to use this so we don't leak
// the actual builder and have to worry about null/empty values on the builder itself.
// You could just use a struct directly here but it makes it a bit harder to validate
// defaults so we'll go for the simpler interface based solution.
func NewBuilder(url string) HTTPBuilder {
   return &builder{
      headers: map[string][]string{},
      url:     url,
      body:    nil,
      method:  http.MethodGet,
      ctx:     context.Background(),
      close:   false,
   }
}

// HTTPBuilder defines the fields we want to set on this builder, you could add/remove
// fields here.
type HTTPBuilder interface {
   AddHeader(name, value string) HTTPBuilder
   Body(r io.Reader) HTTPBuilder
   Method(method string) HTTPBuilder
   Close(close bool) HTTPBuilder
   Build() (*http.Request, error)
}

type builder struct {
   headers map[string][]string
   url     string
   method  string
   body    io.Reader
   close   bool
   ctx     context.Context
}

func (b *builder) Close(close bool) HTTPBuilder {
   b.close = close

   return b
}

func (b *builder) Method(method string) HTTPBuilder {
   b.method = method

   return b
}

func (b *builder) AddHeader(name, value string) HTTPBuilder {
   values, found := b.headers[name]

   if !found {
      values = make([]string, 0, 10)
   }

   b.headers[name] = append(values, value)

   return b
}

func (b *builder) Body(r io.Reader) HTTPBuilder {
   b.body = r

   return b
}

func (b *builder) Build() (*http.Request, error) {
   r, err := http.NewRequestWithContext(b.ctx, b.method, b.url, b.body)
   if err != nil {
      return nil, err
   }

   for key, values := range b.headers {
      for _, value := range values {
         r.Header.Add(key, value)
      }
   }

   r.Close = b.close

   return r, nil
}

So we have a builder that sets up some sane defaults (the body is empty, the method is GET, there’s a default context). Here’s what it looks like in use:

func TestBuilder_Build(t *testing.T) {
   request, err := NewBuilder("https://example.com/").
      AddHeader("User-Agent", "Golang patterns").
      Build()

   assert.NoError(t, err)

   assert.Equal(t, "Golang patterns", request.Header.Get("User-Agent"))
   assert.Equal(t, http.MethodGet, request.Method)
   assert.Equal(t, "https://example.com/", request.URL.String())
}

We could go complex with multiple headers, change the method, set a custom context, and it would still look clean and easy to read. That’s the main advantage of using a builder to instantiate complex objects. You get to go as deep as needed but should still be allowed to create something with sane defaults quickly. While this builder follows the fluent API style, it’s not the only way to produce a builder, as long as you find a way to separate the parametrization from the creation of a complex object that would still be a builder (like using the function opts linked above). The main goal here is to make it easier to build complex objects correctly.

Abstract factory/factory method

As the language lacks inheritance and focuses on composition, the primary way you’d represent factories is by having a function that produces an object as a parameter instead of abstract classes or methods.

A common reason you’d do this is to simplify unit tests. Imagine I’m testing a type that opens a socket connection to some other service, while you could use a net.Dialer directly, which would make testing it harder. If I give the object a method that produces a net.Conn object instead, I can easily replace the implementation by changing the factory function.

Here’s what it would look like:

package gof_go

import (
   "bytes"
   "context"
   "fmt"
   "github.com/stretchr/testify/assert"
   "github.com/stretchr/testify/require"
   "net"
   "testing"
   "time"
)

type mockAddress struct {
   network string
   address string
}

func (m *mockAddress) Network() string {
   return m.network
}

func (m *mockAddress) String() string {
   return m.address
}

// mockConnection represents a connection used for testing only
type mockConnection struct {
   closed  bool
   buffer  *bytes.Buffer
   address *mockAddress
}

func (m *mockConnection) Read(b []byte) (n int, err error) {
   return m.buffer.Read(b)
}

func (m *mockConnection) Write(b []byte) (n int, err error) {
   return m.buffer.Write(b)
}

func (m *mockConnection) LocalAddr() net.Addr {
   return m.address
}

func (m *mockConnection) RemoteAddr() net.Addr {
   return m.address
}

func (m *mockConnection) SetDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) SetReadDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) SetWriteDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) Close() error {
   m.closed = true
   return nil
}

type socketClient struct {
   address string
   factory func(ctx context.Context, network, address string) (net.Conn, error)
}

func (s *socketClient) ping(ctx context.Context) error {
   c, err := s.factory(ctx, "tcp4", s.address)
   if err != nil {
      return err
   }

   defer func() {
      if err := c.Close(); err != nil {
         fmt.Printf("failed to close socket: %v\n", err)
      }
   }()

   if _, err := c.Write([]byte("PING")); err != nil {
      return err
   }

   return nil
}

func TestSocketClient(t *testing.T) {
   connection := &mockConnection{
      buffer: bytes.NewBuffer(make([]byte, 0, 1024)),
   }

   c := &socketClient{
      address: "example.com:40",
      factory: func(ctx context.Context, network, address string) (net.Conn, error) {
         connection.address = &mockAddress{
            address: address,
            network: network,
         }

         return connection, nil
      },
   }

   require.NoError(t, c.ping(context.Background()))
   assert.True(t, connection.closed)
   assert.Equal(t, "PING", connection.buffer.String())
}

We need mocks for net.Addr and net.Conn as we want to return in memory structs for those. Then our socketClient type has a factory property with type func(ctx context.Context, network, address string) (net.Conn, error), which is exactly the same function signature as net.Dialer#DialContext so the concrete implementation here would be the net.Dialer function.

Instead of thinking about factories as classes you use or extend, you can have them be a function you call that produces the object. There is no need for inheritance, abstract classes, or interfaces, just a function signature.

Adapter

It continues to be a widely used pattern all over, the database/sql package is an excellent example of the pattern. Every database driver implements the driver.Driver interface and registers it with the database/sql package. So it doesn’t matter what database you’re using, it all looks the same as you’ll be interacting with objects from the database/sql package only most of the time.

We also see this same pattern on libraries like go-cloud where the specific details of cloud providers are hidden behind the standard interface the library presents.

Decorator

Decorators add functionality to an existing object you might have no control over without breaking its interface. You build a decorator by creating an object that wraps another but has the same methods and forwards calls of these methods to the object being decorated, adding some functionality on top of them.

Typical use cases are adding buffers to IO classes (which is the example we’ll see in a bit), adding metrics or tracing to types you don’t have control over, like external libraries, and many more. The main goal is hiding from the calling code that something has changed.

The lack of inheritance makes decorators in Go a bit more problematic, as you can see on this ancient issue on collecting metrics when resolving DNS entries. Given that the code that calls the net.Resolver code expects the exact type and there is no inheritance, we can’t wrap the resolver as we would in languages that allow inheriting. You can only use decorators in Go if you’re dealing with interfaces or function signatures. If you have to wrap a struct, the only way is to change the code that depends on the struct to an interface with the same methods.

Now we’ll look at a decorator that adds a buffer to io.Reader objects (this is not supposed to be the most efficient and optimized version, it’s just an example):

func NewBufferedReader(wrapped io.Reader, length int) io.Reader {
   if length <= 0 {
      length = 1024
   }

   return &bufferedReader{
      currentIndex:  0,
      lastIndex:     0,
      buffer:        make([]byte, length, length),
      wrappedReader: wrapped,
      err:           nil,
   }
}

type bufferedReader struct {
   currentIndex  int
   lastIndex     int
   buffer        []byte
   wrappedReader io.Reader
   err           error
}

func (b *bufferedReader) Read(p []byte) (n int, err error) {
   if len(p) == 0 {
      return 0, errors.New("an empty slice was provided to Read")
   }

   availableBytes := b.lastIndex - b.currentIndex

   if availableBytes == 0 {
      if b.err != nil {
         return 0, b.err
      }

      if read, err := b.wrappedReader.Read(b.buffer); err == nil || err == io.EOF {
         b.err = err
         b.currentIndex = 0
         b.lastIndex = read
         availableBytes = read

         if availableBytes == 0 {
            return 0, b.err
         }
      } else {
         b.err = err
         return 0, err
      }
   }

   expectedBytes := len(p)

   bytesToRead := availableBytes
   if availableBytes > expectedBytes {
      bytesToRead = expectedBytes
   }

   copy(p, b.buffer[b.currentIndex:b.currentIndex+bytesToRead])
   b.currentIndex += bytesToRead

   return bytesToRead, b.err
}

We wrap any existing io.Reader and add a buffer on top of it. For the code that was using a reader before, nothing has changed, it is still an io.Reader object, but we have introduced new functionality without breaking the contract. It’s also how the IO package works, with decorators adding features on top of reader and writer objects. This pattern is still alive in Go, albeit not in as many places as in other languages.

Facade

Still here! Facades hide a complex or extensive API behind a small interface you can interact with without understanding all the details behind the scenes. Like we mentioned before, go-cloud is an excellent example of both adapter and facade, as it hides the complex details of talking to cloud providers behind a straightforward interface.

Proxy

Proxies are an interface to some other object that might be expensive to create or interact with directly. A typical case for using proxies is in ORM tools when loading an association. Think a User has Posts. You don’t necessarily need to load the posts for the user every time you’re loading a user so that ORM tools will hide the Posts behind a proxy object. They will wait until you try to do something with Posts that requires looking at the objects. They’ll now load them from the data source where they live, instantiating the Posts collection.

If you never try to access them, you won’t pay for the cost of loading them, and that’s the advantage of using a proxy. You don’t have to have that object directly available to you all the time. You can apply these lazy load techniques to produce more efficient code and use fewer resources.

Like the Decorator example above, the lack of inheritance also forces proxies to only be available for interface or function types. Reflection in Go also doesn’t offer dynamic proxies (creating a proxy object from scratch in runtime) as you’d find in languages like Java and C# or a method_missing capability as we have in Ruby, you have to generate the proxy code at compilation time.

Chain of responsibility and Command

Two patterns together? Yes!

Almost every single case of Chain of responsibility I’ve ever seen has been with Command. I’m not even sure if it makes sense to implement a chain of responsibility without Command objects (or functions). If you’ve done any HTTP server development in go you’ve seen them together already:

package http

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

This is our Command for handling HTTP requests. It doesn’t matter how you’re going to implement your functionality. As long as it matches this interface, it’s going to work. We also have the same interface defined as a function signature:

package http

type HandlerFunc func(ResponseWriter, *Request)

Both should be interchangeable. The only thing that matters is that they have the same inputs and outputs. Both take an http.ResponseWriter and a http.Request and have no return types.

Good, we have the Command. Where does Chain of Responsibility come from then? Middlewares!

Here’s the middleware interface:

package gof_go

import (
   "fmt"
   "net/http"
)

type HTTPMiddleware func(w http.ResponseWriter, r *http.Request, next http.Handler)

func LoggingMiddleware(w http.ResponseWriter, r *http.Request, next http.Handler) {
   fmt.Printf("REQUEST %v\n", r.URL.String())
   next.ServeHTTP(w, r)
}

func OkHandler(w http.ResponseWriter, r *http.Request) {
   w.WriteHeader(http.StatusOK)
   if _, err := w.Write([]byte("OK")); err != nil {
      fmt.Printf("failed to write to body: %v", err)
   }
}

func MidddlewareToHandler(middleware HTTPMiddleware, next http.Handler) http.Handler {
   return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
      middleware(writer, request, next)
   })
}

Almost the same as the http.Handler, the only difference is that it also takes a next parameter that will be called if the current handler thinks it should. Let’s see what it looks like being used:

package gof_go

import (
   "github.com/stretchr/testify/assert"
   "github.com/stretchr/testify/require"
   "io"
   "net/http"
   "net/http/httptest"
   "testing"
)

func TestOkHandler(t *testing.T) {
   ts := httptest.NewServer(MidddlewareToHandler(LoggingMiddleware, http.HandlerFunc(OkHandler)))
   defer ts.Close()

   res, err := http.Get(ts.URL)
   require.NoError(t, err)

   ok, err := io.ReadAll(res.Body)
   require.NoError(t, err)
   require.NoError(t, res.Body.Close())

   assert.Equal(t, "OK", string(ok))
}

Here we use the MiddlewareToHandler method to add the logging middleware to the OkHandler, but you could add any number of features with other middlewares here. Authorization, feature flipping, rate limiting, and all without changing the commands that handle the requests and in any order you’d like. As the HTTPMiddleware implementors have complete control over if the request continues or not, you can add functionality (almost like decorators) and decide if the request flow continues or not.

While this example only adds a single middleware, the actual stack could go as deep and add as many middlewares as you’d need by making more calls to MiddlewareToHandler. You can even go fancy and use a builder object to create the actual Chain of Responsibility, like the gorilla mux project does.

Iterator

While you can build iterators in Go for specific use cases, the fact that the language doesn’t have a collections library like other languages makes it less common generally. With generics coming along in 1.18 we might see collection libraries showing up and there might be a standard iterator or an easier way for objects that are not in the language itself (like arrays, slices, and maps) to be iterated over using for ... range loops.

The most famous iterator in the language might be sql.Rows, which is an object everyone sees once they’re dealing with SQL in Go.

Observer

Observers are almost first-class citizens in go, given the existence of channels. Most of the code you’d write to support observers, especially one producer, one consumer pattern, is already baked in how we use channels in the language. It starts to get a bit more complicated when you need to publish a message and have multiple consumers see the same message, as you’d need various channels to make a fan-out behavior work.

Strategy

Strategies are known as classes that implement multiple algorithms but expose the same interface. You could have numerous sorting methods (merge sort, bubble sort, quick sort) all implemented in separate classes but with the same interface (a single method that takes an array, for instance).

While you could build strategies in Go using single-method interfaces, that’s not very idiomatic. You’re better off defining a function signature and having each algorithm being a function that implements the same signature.

package gof_go

import "testing"

type Sorter func(a []int)

func MergeSort(a []int) {
   // implementation
}

func QuickSort(a []int) {
   // implementation
}

func TestSorter(t *testing.T) {
   var sorter Sorter
   sorter = MergeSort
   sorter([]int{10, 7, 5, 2, 4})
}

Using functions we remove the need to have classes for every algorithm when it would just be a method in the class anyway.

Template Method

Due to the lack of inheritance and the availability of functions as first-class objects, becomes the Template Function. An abstract method available for you to implement in a subclass becomes a function taken as a parameter just like we had with factories before.

Sorting objects is a pretty typical example of Template Method still in action in Go:

package gof_go

import (
   "github.com/stretchr/testify/assert"
   "sort"
   "testing"
)

type Racer struct {
   Position int
   Name     string
}

func TestSortRacers(t *testing.T) {
   racers := []*Racer{
      {
         Position: 10,
         Name:     "Alonso",
      },
      {
         Position: 2,
         Name:     "Verstappen",
      },
      {
         Position: 1,
         Name:     "Hamilton",
      },
      {
         Position: 12,
         Name:     "Vettel",
      },
   }

   sort.SliceStable(racers, func(i, j int) bool {
      return racers[i].Position < racers[j].Position
   })

   assert.Equal(t, []*Racer{
      {
         Position: 1,
         Name:     "Hamilton",
      },
      {
         Position: 2,
         Name:     "Verstappen",
      },
      {
         Position: 10,
         Name:     "Alonso",
      },
      {
         Position: 12,
         Name:     "Vettel",
      },
   }, racers)
}

Like other implementations, we don’t know what algorithm is used to sort the objects. All we do is provide a way to compare them with a function. It removes the need to use a separate class as you’d do with implementations of Template Method in languages that don’t have functions as first-class citizens.

What Is Dead May Never Die

As we’ve seen, many of the patterns are still here with us, in the standard library, shared third-party libraries, and frameworks. The patterns I haven’t mentioned are sometimes too specific for the problems they’re solving (like interpreter and visitor), so it’s a bit harder to find their usage in the wild, which doesn’t mean they’re unnecessary.

While it might be harder to read the book nowadays if you have no C++ or Smalltalk experience, other books cover the same patterns in a newer format, like the Head First Design Patterns.

So while languages have evolved, they haven’t removed the need for these patterns and the advantages they can bring to your codebase when used correctly. Keep them on your toolbelt, so you know when to use and how to spot them in code, its going to improve your coding vocabulary and the solutions you’ll build.

What to ask at job interviews when you are being interviewed

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 12 Jan 2022 00:00:00 UTC

There are many guides on how to ace job interviews, you’ll find books, videos, courses on multiple subjects covering every aspect possible, including even prep that is just for specific companies (like checking questions Facebook is currently asking), but we talk very little on how you should also interview the people you’re talking to make sure you do want to work at that place. Switching jobs to find that you ended up in a bad work environment on a dead-end team is demoralizing and might affect your future career prospects.

Most of the questions here assume the people you’re talking to will be at least a little bit truthful in their answers. I’d also recommend exercising your network to find friends or friends of friends that work at the place to verify if the answers are accurate or if they’re painting a rosy picture to make sure you accept the offer. Most businesses will have some of the problems we will be discussing next, so it’s going to be up to you to decide if you do want to sign up for these issues or not, there are no perfect places, but you should go in being comfortable on what the challenges are going to be. Glassdoor, Blind, and Fishbowl are usually hit and miss, with stuff being, either too rosy or just hell, so I can’t say I find them very effective.

Also, unless you’re going for a director-level/decision-making position, don’t trick yourself into thinking you’re going to change the whole environment and make it bearable, odds are you won’t and the energy you’ll spend trying to fix it without having the actual power to do so would be better spent on a job that doesn’t require you do pull a Sisyphus every day. The job market is red hot right now, and you should be able to give that bad-looking job a pass.

Now let’s look at a couple of questions you could ask to figure out if it is a good job or a tar pit.

What is the on-call schedule like? How many incidents does the team handle every week on average?

In most sane places, the teams that build apps are also responsible for ensuring they’re running as expected in production. This means that when something bad happens, someone on the team will be called/paged and get the dreaded PagerDuty alert, you have one incident phone call. Most of us have been there, and pager pain is very real. Working on a team that is firefighting all the time is the quickest path to burnout you could take as you’ll be constantly worried about the next time you’ll be fast asleep and the phone is going to ring, destroy your night and the next day.

What you’re looking for here is a clear on-call schedule with multiple people on rotation and a secondary that also rotates. The schedules should usually be a couple of days long (I’d say a week is generally good enough), with at least two weeks off until you’re on call again. Back-to-back on-call schedules are demanding, especially if incidents are frequent because there is very little time you’re not worried about incidents.

How much time does the team spend firefighting or doing unplanned work? If a team spends over 30% of its time just fixing bugs and working through incidents instead of building features, there are deeper issues on the team itself. The root causes here could be lack of investment in reliability, automation, lousy planning, death marches, none of which are a good signs. You could dig deeper here with the people you’re talking to as to what is going to be done to bring the team back into a more manageable schedule and lower the amount of toil/bugs/incidents if there’s no plan or they think this isn’t a problem (it’s just how we do work), it’s a pass.

What is the engineering onboarding process like? What about onboarding on the team itself?

It might sound like a first world problem thing, but having a well-defined onboarding process is critical to having employees that start contributing to the bottom line as fast as possible. Businesses that are hiring but have no plan to get the people they’re hiring into actually doing the job aren’t doing a good job at hiring and might not be doing an excellent job at other places.

Depending on the company’s size, they might not have a general engineering and a specific team onboarding; there could just be one, that’s not a problem. What you want to know is what will happen once you join and the expectations. Is there going to be a buddy taking you through the process, someone you can ask questions and get unstuck? How will you get used to the projects you’ll be working on? Is there documentation? What are the first things you’re going to be working on?

The expectation here is that they won’t just say here’s the source code, now figure it out but that there is some well-defined way for you to go along (preferably with someone else on the team as a pair) to get you used to the tooling, code, development and deployment process. If there is no onboarding process at all, be ready just to be thrown at stuff and be left to fend for yourself and hope for the best. While it’s entirely possible to make this work (as I have had to make this work multiple times) I’d rather be at a place that is going to guide and help me get productive faster instead of having to ask please can you help me to multiple people that might or might not care about helping me out.

How long has the person managing the team been a manager?

It might sound a bit unfair, but, especially as you get more senior, joining a team in a new company with a manager that does not have much experience managing people is risky. While having a manager with a lot of experience isn’t a guarantee that everything will be unicorns and rainbows, a new manager is more likely to lack many skills required to build and lead an effective team.

I’ve heard horror stories from friends that got a new job with a manager that had just been promoted after a stellar stint as an Individual Contributor, which led to a lot of clout in the business, but that couldn’t manage people at all. Given the new manager’s success in his previous career, he was given a lot of leeway on running the team, and most of the criticism would fall on deaf ears, making it hard to move the needle in any direction. This leads to team members not being promoted, not being assigned to important/impactful projects, and not being promoted.

Being under a bad manager is not only bad for your mental health, but it’s also bad for your career as it might stall your professional development and keep you from moving at the pace you expected you’d go. So, beware, joining a team with a newly minted manager carries many risks. Make sure you know what you’re signing up for if this is the case.

What is the vision for the team? What are the important projects it is taking on in the next 12 months? What would be my goals/projects if I joined?

Here what you’re looking for is to make sure the team you’re joining is healthy and doing important work for the business. Teams should have large, impactful projects they have planned and are working on, they should have a vision of what the team’s job is, and as they’re hiring new people, they need to have a reason for why they would be hiring you.

If the team doesn’t have important projects to work on, odds are it might not be an essential part of the business, or the manager hasn’t been able to pull important work or doesn’t want to do meaningful work. It’s not unusual for, in larger companies, there to be teams that are just coasting and delivering small things here and there. If that’s what you’re looking for, the answers to these questions might give you hints this is it.

Otherwise, if you want to do impactful and essential work, you want to make sure the folks you’re talking to have good answers. They should have a clear vision for what the job is, it can’t just be “deliver features”, every team is working on some domain-specific work, so it has to be “improve the way warehouses are managed”, “deliver code faster to production” (if they’re a platform team, for instance). And on that vision, they should also have cool and exciting projects they’re working on that have a tangible impact on the business, like building a new database-as-a-service product to capture a growing market. This kind of team allows you to grow into a better professional and offers challenges that will get you more experience and promotions.

And last but not least, ask why they’re hiring a new person and what would be the work you’d be doing, at what level you’re expected to deliver. Are you here to lead a new project? Continue an existing project? What are the opportunities available for you to grow? Having good answers for these questions means they’ve done their homework on why they need to hire someone else, and you’ll get a feel of the expectations they’re placing on you if you join.

Is there a career development plan, and what does it look like?

You might be in for the love of writing code and stuff, but we all want to be paid and eventually make more money next year than we do right now, and to do so, you have to progress on your career in some way. Ideally, you want the place to have a ladder with multiple levels and well-defined expectations for what you should be doing to get to the next level. Not knowing what you have to do to be promoted leaves you at the whims of your manager, and that is going to be the luck of the draw, the manager might work to get you promoted or not, taking most of the possibility of you influencing this out of your hands.

Having a known set of activities you have to attain to move up allows you to plan your work accordingly, ticking the boxes that need to be ticked instead of just randomly doing stuff and hoping for the best. That also allows you to decide the crucial pieces of work that need to be done and focus on them instead of over-engineering or investing too much time on stuff that won’t matter that much for your career. This gives you some power to decide your path instead of leaving it all in someone else’s hands.

How long does it take for code that is feature complete to reach production in some way?

Here you’re looking for minutes as the answer. In some businesses, hours would be acceptable. More than that is typically bad news unless you’re applying for a job at NASA and your code will ship in a satellite. The speed code arrives in production signals maturity of the business development process. The faster it is, the more mature and uneventful shipping code is, and this is what you’re looking for at your job. The scarier it is for a team to push changes, the more likely they will cause incidents and break stuff.

Teams with a well-oiled deployment process most likely practice continuous integration, have a considerable mass of automated testing (unit, integration, smoke, black box), observability tools, and ways to rollout code that doesn’t update everything all at once. That’s why we have the in some way there in the question, you don’t have to necessarily update everything to everyone at once (it might even be impossible, given the scale of the work), but you still have a way to put that code in front of actual users and verify it is doing the job. This process should also be fully automated and executed from a CI system instead of a developer’s local environment.

Most of the time, not delivering code fast to production happens because the business doesn’t trust the team. New processes or steps are added, like change request approvals, manual QA steps, staging environments, so the changes are less likely to break. Unfortunately, adding more processes instead of improving the engineering practices doesn’t work. You end up with a very long cycle time to get changes out, with larger changesets that are much more likely to break once deployed, and the cycle begins anew, with even more processes to try to build trust.

How do you measure and maintain code quality?

How does the team make sure the code shipped is of quality and meets the standards defined? Are there standards well defined? Is there time for code gardening, refactors, improvements, and upgrades? Are there automated tools running on the codebases to ensure standards are kept?

We expect a review process or pair programming, static code analyzers, automated formatters, test coverage requirements (at different levels, it shouldn’t just be unit or integration tests), and essential documentation. Coding should also be a group exercise, and people should not be working in a vacuum. The team should be working together to deliver on its commitments.

For instance, if there is a pull request review process, how long does it usually take to review a pull request? Minutes? Hours? Days? The longer it takes, the more time wasted waiting, making conflicts more likely. Long review cycles are also a sign the team might not work very well together, and instead of being a team, it might just be a collection of people that happen to be together.

What are the self-development benefits provided?

No tech company survives with a workforce that doesn’t improve, so most of them will offer a stipend or access to education related to the work. You want to be at a place that will buy you that book you need for some new project you’re working on or pay for a course or event you feel like will be essential for your productivity at work.

Places that require you to invest in yourself even when your productivity improvement goes mainly to the business aren’t the best place to stay.

Pick your poison

No place will ever be the perfect workplace; you’re going to find issues, problems, walls that can’t be breached, and people that can’t be reasoned with; still, getting more information allows you to make a better decision when accepting a job or not. You should always ask as many questions as you can about the company, the business, the team, and the people surrounding the team to ensure that it is the place you want to work. Don’t forget your job is most likely the single thing you’re going to spend most of your time in for a long while. You have to make sure this time is worth it.

Why not build a Redis client in Golang?

mauricio.linhares@gmail.com (Maurício Linhares) — Fri, 07 Jan 2022 00:00:00 UTC

Writing applications that talk to each other over the network is something we do almost daily but we usually don’t worry too much about how the data moves along or what the communication protocol looks like, as we use HTTP clients and servers to do most of the work. Still, it’s good to get dirty every once in a while with sockets and byte streams and learn a little about how we communicate with other applications. Today we’re doing that with Redis.

First, why Redis? Redis is a widely used data structures store that has a straightforward protocol format we can cover in a single blog post but that is still very powerful in what it allows us to do. With Redis, you can deal with the common key-value store use case but you can also use other tools, like lists, sets, maps so the simplicity of the protocol also shows that you don’t have to complicate the way you access data even if the implementation you might have on the server-side is indeed complex.

Our main goal here is to understand the basics of network programming in Go, write code that parses messages from and writes them to a Redis server and understand some options we have when writing apps that operate with sockets in the language. We are not going to cover optimizations, TCP/IP, or other advanced concepts, we’re just going to write a simple Redis client. This client also doesn’t work for Redis pub-sub, only for the general request-response interface.

The full source code for this post is available on github.

The protocol

The Redis wire protocol is what we usually call a terminator-based protocol as you know that a message has ended once you find a \r\n. For instance, if you want to define a simple string in Redis it’s going to be written as:

+OK\r\n

The + marks this as a simple string and the content of the simple string is all the bytes until you find a \r\n. The other basic types are errors (-ERR failed to do something\r\n) and integers (:1245\r\n).

But what if the string you want to send has a \r\n in it, you might ask, this is where bulk strings some into place. This type is different than the others as it doesn’t use the \r\n to mark the end alone but it also has a length of bytes that should be read as the value. Here’s an example:

$6\r\ngolang\r\n

A bulk string stats with the $ symbol, then a number as a string until a \r\n that is the length in bytes that you must read until you find another \r\n marking the end of the bulk string. Given we have the length we should read, it doesn’t matter what are the contents inside the bulk string as we’re not checking anything there, we’re just reading everything until the end. So whenever you have to write a string that could have a \r\n in it you have to use bulk strings. Bulk strings have a special case that is a null bulk string, a bulk string with a -1 length should be represented by clients as a null value and not an empty string, these are returned as "$-1\r\n"

And last but not least we have the array type that is a collection of the previously defined types (including arrays of arrays). They start with a * followed by an integer length (the number of items the array holds) and then each item in order, an array of two simple strings would look like "*2\r\n+foo\r\n+bar\r\n", the *2\r\n piece says this is a two items array and then we get two simple strings. Any of the previously defined values are valid on arrays, including arrays themselves, if we wanted to have an array that is two separate arrays of simple strings we could define it like:

*2\r\n*3\r\n+foo\r\n+bar\r\n+sample\r\n*1\r\n+nope\r\n

Which would be equivalent to having:

[
    [ "foo", "bar", "sample"],
    ["nope"]
]

So you can mix and match values for arrays as you see fit. Arrays also have a special case, the null array, which just like the bulk string case is different from having an empty array in Redis, null arrays are defined as *-1\r\n", an array with a -1 length.

Reading data

Now that we’ve covered the protocol basics, we’ll start with the reader part of our implementation. Given we know this is mostly a terminator-based protocol (with only bulk strings as a special case) we can use a bufio.Scanner to parse the stream of data, breaking up the lines on \r\n and if we do see a bulk string do a bit more magic to return it to the code reading from the scanner as a single line.

When using a bufio.Scanner you provide a SplitFunction with the following signature:

type SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)

So you get the data in bytes that are available and a bool value signaling if we’re at the end of the stream or not. You then have to return how many bytes to advance on the stream, the data that should be returned to whoever is reading the scanner, and any error if it has happened. If you still don’t have enough data to read, you just return a nil token value to ask the scanner to read more data (for instance, if there are no \r\n anywhere you know right away this isn’t a full Redis response). Also, atEOF can be true and data not be empty, as you could be at the end of the stream because the server has closed the connection but it did write a full response back so even if you’re at the end of a stream on a scanner you should make sure you check what the data is and return it to whoever is consuming from the scanner.

Our implementation here is done in two parts, one is the split function we use for the scanner, which breaks the Redis responses in lines, and the code that reads the lines, interprets them, and turns them into strings, integers, or arrays. Here’s what it looks like:

package redis_client

import (
	"bufio"
	"bytes"
	"encoding/base64"
	"errors"
	"fmt"
	pkgerrors "github.com/pkg/errors"
	"io"
	"strconv"
)

const (
	defaultBufferLength = 10140
	typeSimpleString    = '+'
	typeErorr           = '-'
	typeInteger         = ':'
	typeBulkString      = '$'
	typeArray           = '*'
)

var (
	separator = []byte("\r\n")
)

type Reader struct {
	scanner *bufio.Scanner
}

func NewReader(r io.Reader) *Reader {
	scanner := bufio.NewScanner(bufio.NewReaderSize(r, defaultBufferLength))
	scanner.Split(redisSplitter)

	return &Reader{
		scanner: scanner,
	}
}

func (r *Reader) Read() (*Result, error) {
	return readRESP(r.scanner)
}

// redisSplitter splits a byte stream into full lines in the redis protocol format. it reads a whole item
// (string, bulk string, error, integer, or array) and then returns it as a line. code reading from scanners
// created with this function can safely assume that if they got a line it's a full line that does not need
// any extra checking.
func redisSplitter(data []byte, atEOF bool) (advance int, token []byte, err error) {
	// a valid redis message has at least 3 characters, if less than that ask for more stuff
	if len(data) < 3 {

		if atEOF {
			return 0, nil, fmt.Errorf("unexpected end of stream, a redis message needs at least 3 characters to be valid, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data))
		}

		return 0, nil, nil
	}

	found := bytes.Index(data, separator)
	// if we could not find a \r\n and the stream is at its end, this stream is broken and can't be recovered
	if found == -1 && atEOF {
		return 0, nil, fmt.Errorf("unexpected end of stream, there should have been a \\r\\n before the end, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data))
	}

	// if there is no \r\n we need to read more data, this means this result isn't finished yet
	if found == -1 {
		return 0, nil, nil
	}

	// bulk strings are special, they change the binary format from a terminator based one (\r\n to close messages)
	// to a length based one. so once we figure out this message is a bulk string (like `$6\r\nfoobar\r\n`),
	// we have to read the length and make sure there are at least length + 2 bytes after the first \r\n to
	// show that we do have the whole bulk string here. it is not safe to just find all \r\n in a bulk string
	// because there could be \r\n tokens as part of the string itself, so we always have to make sure we consume
	// the length and use it to read the whole value.
	if data[0] == typeBulkString {
		length, err := strconv.ParseInt(string(data[1:found]), 10, 64)
		if err != nil {
			return 0, nil, fmt.Errorf("message starts as bulk string but length is not a valid int, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data[0:found]))
		}

		// a -1 length means this is a null string and should be returned as such to clients, null and empty
		// strings are different things in redis. this is the only time we return a bulk string sign ($) here
		// as we'll use it as a marker for null strings. for someone reading from a scanner
		// there should be no difference between a simple or a bulk string as we have already
		// parsed the lengh and we'll return only the actual string contents.
		if length == -1 {
			return 5, []byte("$"), nil
		}

		// a 0 length means an empty string, an empty string is not the same as a null string on redis
		if length == 0 {
			return 6, []byte("+"), nil
		}

		// this is the position of the first \r\n + the expected length + 4 which is the \r\n twice we have on bulk strings
		expectedEnding := found + int(length) + 4
		if len(data) >= expectedEnding {
			// given here we already have all the information we need to return this as a string,
			// we don't return the length anymore, we return this as if it was a normal string.
			// now we set the first `\n` we have to `+` so the code parses it as a simple string
			// as we have already capped the returned slice do the length of the string.

			start := found + 1
			data[start] = '+'
			return expectedEnding, data[start : expectedEnding-2], nil
		}

		if atEOF {
			return 0, nil, fmt.Errorf("unexpected end of stream, stream ends before bulk string has ended, expected there to be %v total bytes but there were only %v, actual content in base64: %v", expectedEnding, len(data), base64.RawStdEncoding.EncodeToString(data))
		}

		// not enough data, ask for more data to be read
		return 0, nil, err
	}

	return found + 2, data[:found], nil
}

// readRESP reads from a scanner that was initiated with `redisSplitter`. it expects every scanned line to be
// a full line of a data type redis supports (unless it's an array, arrays start with the length of the array only).
func readRESP(r *bufio.Scanner) (*Result, error) {

	for r.Scan() {
		line := r.Text()
		switch line[0] {
		case typeSimpleString:
			// if a string, just remove the marker and return it
			return &Result{
				content: line[1:],
			}, nil
		case typeBulkString:
			// a bulk string is only returned if it is nil, otherwise it is turned as a simple string
			return &Result{
				content: nil,
			}, nil
		case typeErorr:
			// if an error just wrap the error and return it
			return &Result{
				content: errors.New(line[1:]),
			}, nil
		case typeInteger:
			content, err := strconv.ParseInt(line[1:], 10, 64)
			if err != nil {
				return nil, fmt.Errorf("failed to parse returned integer: %v (value: %v)", err, line)
			}
			return &Result{
				content: content,
			}, nil
		case typeArray:
			// the first thing to be done when we find an array is to find its length, if not `-1` we then
			// read items from the scanner until we've read all items on the array.
			length, err := strconv.ParseInt(line[1:], 10, 64)
			if err != nil {
				return nil, fmt.Errorf("failed to parse array length: %v (value: %v)", err, line)
			}

			if length == -1 {
				return &Result{content: nil}, nil
			}

			contents := make([]interface{}, 0, length)

			for x := int64(0); x < length; x++ {
				result, err := readRESP(r)
				if err != nil {
					return nil, pkgerrors.Wrapf(err, "failed to read item %v from array", x)
				}

				contents = append(contents, result.content)
			}

			return &Result{
				content: contents,
			}, nil
		}
	}

	if r.Err() == nil {
		return nil, errors.New("scanner was empty")
	}

	return nil, r.Err()
}

Most of the complexity here lies at the redisSplitter function, that is breaking up the bytes into lines with a full item to be read. We also cheat a little on the bulk strings by turning them into simple strings on the fly. This is done mostly because we have the code that splits lines separate from the code that parses the values, if we weren’t using a bufio.Scanner and were reading straight from the byte stream (the io.Reader) we could remove the special case for bulk strings but that would also require us to do a lot of the work that the scanner is doing. Doing it like this pushes the complexity of reading enough bytes and buffering them into the scanner instead of into our code.

We’re also using this Result type here all over but it’s just a wrapper to make it easier to transform a value from Redis if you know what it is supposed to be, here’s the code:

package redis_client

import "fmt"

type Result struct {
	content interface{}
}

func (r *Result) Err() error {
	err, ok := r.content.(error)
	if !ok {
		return nil
	}

	return err
}

func (r *Result) Int64() (int64, error) {
	if err := r.Err(); err != nil {
		return 0, err
	}

	result, ok := r.content.(int64)
	if !ok {
		return 0, fmt.Errorf("content is not an int64: %#v", r.content)
	}

	return result, nil
}

func (r *Result) String() (string, bool, error) {
	if err := r.Err(); err != nil {
		return "", false, err
	}

	if r.content == nil {
		return "", true, nil
	}

	result, ok := r.content.(string)
	if !ok {
		return "", false, fmt.Errorf("content is not an string: %#v", r.content)
	}

	return result, false, nil
}

func (r *Result) Slice() ([]interface{}, error) {
	if err := r.Err(); err != nil {
		return nil, err
	}

	if r.content == nil {
		return nil, nil
	}

	result, ok := r.content.([]interface{})
	if !ok {
		return nil, fmt.Errorf("content is not a slice: %#v", r.content)
	}

	return result, nil
}

func (r *Result) Content() interface{} {
	return r.content
}

It just holds the reponse that was read from redis and makes sure it is the right type before returning it.

Writer

The writer is a simpler inverse version of the reader. Given we already know the protocol, what we have to do here is to write the data in the correct format:

package redis_client

import (
	"encoding/base64"
	"fmt"
	"github.com/pkg/errors"
	"io"
	"strconv"
)

type Writer struct {
	writer io.Writer
}

func NewWriter(w io.Writer) *Writer {
	return &Writer{
		writer: w,
	}
}

func (w *Writer) write(messageType byte, contents ...[]byte) error {
	if _, err := w.writer.Write([]byte{messageType}); err != nil {
		return errors.Wrapf(err, "failed to write message type: %v", messageType)
	}

	for _, b := range contents {
		if _, err := w.writer.Write(b); err != nil {
			return errors.Wrapf(err, "failed to write bytes, content in base64: [%v]", base64.RawStdEncoding.EncodeToString(b))
		}
	}

	return nil
}

func (w *Writer) WriteBulkString(value []byte) error {
	return w.write(
		typeBulkString,
		[]byte(strconv.FormatInt(int64(len(value)), 10)),
		separator,
		value,
		separator,
	)
}

// WriteNil writes a nil bulk string
func (w *Writer) WriteNil() error {
	return w.write(typeBulkString, []byte("-1"), separator)
}

func (w *Writer) WriteInt64(v int64) error {
	return w.write(
		typeInteger,
		[]byte(strconv.FormatInt(v, 10)),
		separator)
}

// WriteArray writes an array that contains int8 to int64, strings, []byte, []interface{} or nil.
// Any other values inside the array will cause this method to return an error.
func (w *Writer) WriteArray(values []interface{}) error {
	if values == nil {
		return w.write(typeArray, []byte("-1"), separator)
	}

	if err := w.write(
		typeArray,
		[]byte(strconv.FormatInt(int64(len(values)), 10)),
		separator,
	); err != nil {
		return err
	}

	for _, v := range values {
		switch t := v.(type) {
		case int8:
			if err := w.WriteInt64(int64(t)); err != nil {
				return err
			}
		case int16:
			if err := w.WriteInt64(int64(t)); err != nil {
				return err
			}
		case int:
			if err := w.WriteInt64(int64(t)); err != nil {
				return err
			}
		case int32:
			if err := w.WriteInt64(int64(t)); err != nil {
				return err
			}
		case int64:
			if err := w.WriteInt64(t); err != nil {
				return err
			}
		case string:
			if err := w.WriteBulkString([]byte(t)); err != nil {
				return err
			}
		case []byte:
			if err := w.WriteBulkString(t); err != nil {
				return err
			}
		case []interface{}:
			if err := w.WriteArray(t); err != nil {
				return err
			}
		case nil:
			if err := w.WriteNil(); err != nil {
				return err
			}
		default:
			return fmt.Errorf("unsupported type: the value [%#v] is not supported by this client, supported types are int8 to int64, strings, []byte, nil, and []interface{} of these same types", v)
		}
	}

	return nil
}

Our writer is a bit strict, it only writes numbers from int8 to int64, strings and arrays with the previous values. We also don’t bother with writing simple strings at all, we just assume all strings are always bulk strings as that allows us to ignore if there is a \r\n at all inside of them and just write them with the length directly. We also introduce methods to write null strings and null arrays as those are both valid values for redis.

The client

So we can read and write the Redis protocol from streams of bytes, the last step is to actually create a client that can open a socket connection given an address and talk to Redis, here’s what it would look like:

package redis_client

import (
	"context"
	"github.com/pkg/errors"
	"io"
	"net"
	"time"
)

var (
	_ io.Closer = &Client{}
)

type Client struct {
	conn   net.Conn
	reader *Reader
	writer *Writer
}

func (c *Client) Close() error {
	return c.conn.Close()
}

func (c *Client) Send(values []interface{}) (*Result, error) {
	c.conn.SetDeadline(time.Now().Add(time.Second * 5))

	if err := c.writer.WriteArray(values); err != nil {
		return nil, errors.Wrapf(err, "failed to execute operation: %v", values[0])
	}

	return c.reader.Read()
}

func Connect(ctx context.Context, address string) (*Client, error) {
	dialer := net.Dialer{
		Timeout:   time.Second * 5,
		KeepAlive: time.Second * 10,
	}

	conn, err := dialer.DialContext(ctx, "tcp4", address)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to connect to %v", address)
	}

	return &Client{
		conn:   conn,
		reader: NewReader(conn),
		writer: NewWriter(conn),
	}, nil
}

Given we’ve done most of the actual protocol parsing work at the Reader and Writer structs the only thing this client does is to create a dialer and open the TCP connection to Redis. The important pieces here are to make sure you setup a timeout on the dialer and a deadline on every operation. Here we’re just hardcoding a 5 seconds deadline on every read/write but you can make this configurable, what is important here is to make sure that we do have timeouts/deadlines configured on every operations so we don’t wait forever or until an error happens on the connection to give up the operation.

And with this you have a fully functional redis client that can send and receive any commands from the server. We built it by creating separate structs for the different operaitons we need (reading and writing) and when composed them into the actual client. By building them as separate pieces we can easily write unit tests for them (check the repo for the tests) and then assemble them into the actual client, this simplifies development and even allows us to build upon these as needed, we could, for instance, create a pooling client that holds onto multiple connections and the only piece that would change is the client itself, the reader and writer code would stay the same, thus requiring less code to be written. Now off you go building more clients for the apps you use!

Rate limiting HTTP requests in Go using Redis

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 30 Dec 2021 00:00:00 UTC

So you’ve created this awesome API, that offers a feature a lot of customers would be interested in using, but they’re using it so much you can’t handle the load in an effective way for everyone. While scaling up and making the service more reliable is an option, just doing that is not enough, the load might be uneven, the usage patterns might not be what your application was made to handle or you might just have limits you can’t fix at this point in time, that’s where rate-limiting comes in.

The idea behind rate limiting is that you’re going to have a maximum amount of requests you’ll be taking from a client and once that limit is reached, over a defined period, you’ll start dropping requests until you reach the end of the period and restart the counter. For instance, clients are allowed to make 60 requests every minute, once they go over 60 requests you start denying the requests letting them know they’re over their quota and need to wait to continue to have their requests served again.

The goal here is to make your service more reliable, rate-limiting is a protection you implement on your side to make sure bad actors or misconfigured clients can’t take the whole service down or cause outages because the service is going over its expected usage limits. In an ideal scenario, you’re not punishing good clients, as you have given them enough requests to do their usual work, but you’d block bad clients from wreaking havoc on your service.

You can find the whole source code for the project we’ll be discussing below here on Github.

Building a rate-limiter

We’ll have multiple implementations for the rate limiter to discuss the pros and cons of the different implementations, let’s start then with the stuff that will be common across the implementations:

package redis_rate_limiter

import (
	"context"
	"time"
)

// Request defines a request that needs to be checked if it will be rate-limited or not.
// The `Key` is the identifier you're using for the client making calls. This could be a user/account ID if the user is
// signed into your application, the IP of the client making requests (this might not be reliable if you're not behind a
// proxy like Cloudflare, as clients can try to spoof IPs). The `Key` should be the same for multiple calls of the
// same client so we can correctly identify that this is the same app calling anywhere.
// `Limit` is the number of requests the client is allowed to make over the `Duration` period. If you set this to
// 100 and `Duration` to `1m` you'd have at most 100 requests over a minute.
type Request struct {
	Key      string
	Limit    uint64
	Duration time.Duration
}

// State is the result of evaluating the rate limit, either `Deny` or `Allow` a request.
type State int64

const (
	Deny  State = 0
	Allow       = 1
)

// Result represents the response to a check if a client should be rate-limited or not. The `State` will be either
// `Allow` or `Deny`, `TotalRequests` holds the number of requests this specific caller has already made over
// the current period and `ExpiresAt` defines when the rate limit will expire/roll over for clients that
// have gone over the limit.
type Result struct {
	State         State
	TotalRequests uint64
	ExpiresAt     time.Time
}

// Strategy is the interface the rate limit implementations must implement to be used, it takes a `Request` and
// returns a `Result` and an `error`, any errors the rate-limiter finds should be bubbled up so the code can make a
// decision about what it wants to do with the request.
type Strategy interface {
	Run(ctx context.Context, r *Request) (*Result, error)
}

A counter-based implementation

So we have the basics in place, what is the input, the output, and a small interface that needs to be implemented by every counter, let’s look at the first counter, that just uses counters on Redis:

package redis_rate_limiter

import (
	"context"
	"github.com/go-redis/redis/v8"
	"github.com/pkg/errors"
	"time"
)

var (
	_ Strategy = &counterStrategy{}
)

const (
	keyWithoutExpire = -1
)

func NewCounterStrategy(client *redis.Client, now func() time.Time) *counterStrategy {
	return &counterStrategy{
		client: client,
		now:    now,
	}
}

type counterStrategy struct {
	client *redis.Client
	now    func() time.Time
}

// Run this implementation uses a simple counter with an expiration set to the rate limit duration.
// This implementation is functional but not very effective if you have to deal with bursty traffic as
// it will still allow a client to burn through its full limit quickly once the key expires.
func (c *counterStrategy) Run(ctx context.Context, r *Request) (*Result, error) {
	// a pipeline in Redis is a way to send multiple commands that will all be run together.
	// this is not a transaction and there are many ways in which these commands could fail
	// (only the first, only the second) so we have to make sure all errors are handled, this
	// is a network performance optimization.

	p := c.client.Pipeline()
	incrResult := p.Incr(ctx, r.Key)
	ttlResult := p.TTL(ctx, r.Key)

	if _, err := p.Exec(ctx); err != nil {
		return nil, errors.Wrapf(err, "failed to execute increment to key %v", r.Key)
	}

	totalRequests, err := incrResult.Result()
	if err != nil {
		return nil, errors.Wrapf(err, "failed to increment key %v", r.Key)
	}

	var ttlDuration time.Duration

	// we want to make sure there is always an expiration set on the key, so on every
	// increment we check again to make sure it has a TTl and if it doesn't we add one.
	// a duration of -1 means that the key has no expiration so we need to make sure there
	// is one set, this should, most of the time, happen when we increment for the
	// first time but there could be cases where we fail at the previous commands so we should
	// check for the TTL on every request.
	if d, err := ttlResult.Result(); err != nil || d == keyWithoutExpire {
		ttlDuration = r.Duration
		if err := c.client.Expire(ctx, r.Key, r.Duration).Err(); err != nil {
			return nil, errors.Wrapf(err, "failed to set an expiration to key %v", r.Key)
		}
	} else {
		ttlDuration = d
	}

	expiresAt := c.now().Add(ttlDuration)

	requests := uint64(totalRequests)

	if requests > r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: requests,
			ExpiresAt:     expiresAt,
		}, nil
	}

	return &Result{
		State:         Allow,
		TotalRequests: requests,
		ExpiresAt:     expiresAt,
	}, nil
}

This implementation is what would usually be called a fixed window strategy, it means that once the expiration has been set, a client that reaches the limit will be blocked from making further requests until the expiration time arrives. If a client has a limit of 50 requests every minute and makes all 50 requests in the first 5 seconds of the minute, it will have to wait 55 seconds to make another request. This is also the main downside of this implementation, it would still let a client burn through its whole limit quickly (bursty traffic) and that could still overload your service, as it could be expecting this traffic to be spread out throughout the whole limiting period.

A sorted set implementation to react better to bursty traffic

A rolling window strategy offers better protection for bursty traffic as it doesn’t reset the counter completely but keeps the history for the duration of the time window. If you have a 5 minutes window, it will always count the amount of traffic that was generated over the last 5 minutes to decide if a client should be blocked or not instead of just waiting for a key to expire. This implementation is more CPU and memory intensive as you have to keep more information in memory(every request with its timestamp) but provides better protection against quick bursts of traffic.

Here’s what it looks like:

package redis_rate_limiter

import (
	"context"
	"github.com/go-redis/redis/v8"
	"github.com/google/uuid"
	"github.com/pkg/errors"
	"strconv"
	"time"
)

var (
	_ Strategy = &sortedSetCounter{}
)

func NewSortedSetCounterStrategy(client *redis.Client, now func() time.Time) Strategy {
	return &sortedSetCounter{
		client: client,
		now:    now,
	}
}

type sortedSetCounter struct {
	client *redis.Client
	now    func() time.Time
}

// Run this implementation uses a sorted set that holds a UUID for every request with a score that is the
// time the request has happened. This allows us to delete events from *before* the current window if the window
// is 5 minutes, we want to remove all events from before 5 minutes ago, this way we make sure we roll old
// requests off the counters creating a rolling window for the rate limiter. So, if your window is 100 requests
// in 5 minutes and you spread the load evenly across the minutes, once you hit 6 minutes the requests you made
// on the first minute have now expired but the other 4 minutes of requests are still valid.
// A rolling window counter is usually never 0 if traffic is consistent so it is very effective at preventing
// bursts of traffic as the counter won't ever expire.
func (s *sortedSetCounter) Run(ctx context.Context, r *Request) (*Result, error) {
	now := s.now()
	// every request needs an UUID
	item := uuid.New()

	minimum := now.Add(-r.Duration)

	p := s.client.Pipeline()

	// we then remove all requests that have already expired on this set
	removeByScore := p.ZRemRangeByScore(ctx, r.Key, "0", strconv.FormatInt(minimum.UnixMilli(), 10))

	// we add the current request
	add := p.ZAdd(ctx, r.Key, &redis.Z{
		Score:  float64(now.UnixMilli()),
		Member: item.String(),
	})

	// count how many non-expired requests we have on the sorted set
	count := p.ZCount(ctx, r.Key, "-inf", "+inf")

	if _, err := p.Exec(ctx); err != nil {
		return nil, errors.Wrapf(err, "failed to execute sorted set pipeline for key: %v", r.Key)
	}

	if err := removeByScore.Err(); err != nil {
		return nil, errors.Wrapf(err, "failed to remove items from key %v", r.Key)
	}

	if err := add.Err(); err != nil {
		return nil, errors.Wrapf(err, "failed to add item to key %v", r.Key)
	}

	totalRequests, err := count.Result()
	if err != nil {
		return nil, errors.Wrapf(err, "failed to count items for key %v", r.Key)
	}

	expiresAt := now.Add(r.Duration)
	requests := uint64(totalRequests)

	if requests > r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: requests,
			ExpiresAt:     expiresAt,
		}, nil
	}

	return &Result{
		State:         Allow,
		TotalRequests: requests,
		ExpiresAt:     expiresAt,
	}, nil
}

The goal here is to use the sorted set with the current timestamp for the request as the sorting key so we can quickly ask for all requests under a specific time range to be deleted, cleaning up requests that were already expired. As we’re using an UUID as the value the odds we’ll have these requests conflict with each other on the sorted set are very low and having a conflict every once in a while wouldn’t be too much trouble, even with millions of requests being served.

Write heavy solutions

Both implementations here are write first, they first write stuff into Redis and then check if the request should be accepted or not. While this makes the code more straightforward and produces less network traffic from the client to the server, they make it much more expensive for the server to operate, as it is always writing something and changing data in memory, even unbounded data, as in the sorted set case. We’re not making sure these sorted sets are cleaned up or that there is a limit on how many requests can be stored and this is a general red flag you should not let happen on the systems you’re working on.

We have to make sure the resource usage here is bounded, that is, it has a well known limit, you should always assume anything you’re writing is going to be abused in some way (either intentionally or by accident) and make sure that your code forces a bound on these resources in some way. On Redis itself, there are many ways to make sure you’re not going to run out of memory by setting a maxmemory value for how much memory it can use and maxmemory-policy to a value that is not no-eviction. For a rate limiter like the one we’re building here allkeys-lru is a pretty decent option.

Bounded counter implementations

The updated counter implementation with a read before write would look like this:

package redis_rate_limiter

import (
	"context"
	"github.com/go-redis/redis/v8"
	"github.com/pkg/errors"
	"time"
)

var (
	_ Strategy = &counterStrategy{}
)

const (
	keyThatDoesNotExist = -2
	keyWithoutExpire    = -1
)

func NewCounterStrategy(client *redis.Client, now func() time.Time) *counterStrategy {
	return &counterStrategy{
		client: client,
		now:    now,
	}
}

type counterStrategy struct {
	client *redis.Client
	now    func() time.Time
}

// Run this implementation uses a simple counter with an expiration set to the rate limit duration.
// This implementation is functional but not very effective if you have to deal with bursty traffic as
// it will still allow a client to burn through its full limit quickly once the key expires.
func (c *counterStrategy) Run(ctx context.Context, r *Request) (*Result, error) {

	// a pipeline in redis is a way to send multiple commands that will all be run together.
	// this is not a transaction and there are many ways in which these commands could fail
	// (only the first, only the second) so we have to make sure all errors are handled, this
	// is a network performance optimization.

	// here we try to get the current value and also try to set an expiration on it
	getPipeline := c.client.Pipeline()
	getResult := getPipeline.Get(ctx, r.Key)
	ttlResult := getPipeline.TTL(ctx, r.Key)

	if _, err := getPipeline.Exec(ctx); err != nil && !errors.Is(err, redis.Nil) {
		return nil, errors.Wrapf(err, "failed to execute pipeline with get and ttl to key %v", r.Key)
	}

	var ttlDuration time.Duration

	// we want to make sure there is always an expiration set on the key, so on every
	// increment we check again to make sure it has a TTl and if it doesn't we add one.
	// a duration of -1 means that the key has no expiration so we need to make sure there
	// is one set, this should, most of the time, happen when we increment for the
	// first time but there could be cases where we fail at the previous commands so we should
	// check for the TTL on every request.
	// a duration of -2 means that the key does not exist, given we're already here we should set an expiration
	// to it anyway as it means this is a new key that will be incremented below.
	if d, err := ttlResult.Result(); err != nil || d == keyWithoutExpire || d == keyThatDoesNotExist {
		ttlDuration = r.Duration
		if err := c.client.Expire(ctx, r.Key, r.Duration).Err(); err != nil {
			return nil, errors.Wrapf(err, "failed to set an expiration to key %v", r.Key)
		}
	} else {
		ttlDuration = d
	}

	expiresAt := c.now().Add(ttlDuration)

	if total, err := getResult.Uint64(); err != nil && errors.Is(err, redis.Nil) {

	} else if total >= r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: total,
			ExpiresAt:     expiresAt,
		}, nil
	}

	incrResult := c.client.Incr(ctx, r.Key)

	totalRequests, err := incrResult.Uint64()
	if err != nil {
		return nil, errors.Wrapf(err, "failed to increment key %v", r.Key)
	}

	if totalRequests > r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: totalRequests,
			ExpiresAt:     expiresAt,
		}, nil
	}

	return &Result{
		State:         Allow,
		TotalRequests: totalRequests,
		ExpiresAt:     expiresAt,
	}, nil
}

Now we read before we increment, making sure we only increment if this is a good request that will be allowed. Otherwise just bail and deny the request before trying anything.

Now we’ll look at the updated sorted set implementation:

package redis_rate_limiter

import (
	"context"
	"github.com/go-redis/redis/v8"
	"github.com/google/uuid"
	"github.com/pkg/errors"
	"strconv"
	"time"
)

var (
	_ Strategy = &sortedSetCounter{}
)

const (
	sortedSetMax = "+inf"
	sortedSetMin = "-inf"
)

func NewSortedSetCounterStrategy(client *redis.Client, now func() time.Time) Strategy {
	return &sortedSetCounter{
		client: client,
		now:    now,
	}
}

type sortedSetCounter struct {
	client *redis.Client
	now    func() time.Time
}

// Run this implementation uses a sorted set that holds an UUID for every request with a score that is the
// time the request has happened. This allows us to delete events from *before* the current window, if the window
// is 5 minutes, we want to remove all events from before 5 minutes ago, this way we make sure we roll old
// requests off the counters creating a rolling window for the rate limiter. So, if your window is 100 requests
// in 5 minutes and you spread the load evenly across the minutes, once you hit 6 minutes the requests you made
// on the first minute have now expired but the other 4 minutes of requests are still valid.
// A rolling window counter is usually never 0 if traffic is consistent so it is very effective at preventing
// bursts of traffic as the counter won't ever expire.
func (s *sortedSetCounter) Run(ctx context.Context, r *Request) (*Result, error) {
	now := s.now()
	expiresAt := now.Add(r.Duration)
	minimum := now.Add(-r.Duration)

	// first count how many requests over the period we're tracking on this rolling window so check wether
	// we're already over the limit or not. this prevents new requests from being added if a client is already
	// rate limited, not allowing it to add an infinite amount of requests to the system overloading redis.
	// if the client continues to send requests it also means that the memory for this specific key will not
	// be reclaimed (as we're not writing data here) so make sure there is an eviction policy that will
	// clear up the memory if the redis starts to get close to its memory limit.
	result, err := s.client.ZCount(ctx, r.Key, strconv.FormatInt(minimum.UnixMilli(), 10), sortedSetMax).Uint64()
	if err == nil && result >= r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: result,
			ExpiresAt:     expiresAt,
		}, nil
	}

	// every request needs an UUID
	item := uuid.New()

	p := s.client.Pipeline()

	// we then remove all requests that have already expired on this set
	removeByScore := p.ZRemRangeByScore(ctx, r.Key, "0", strconv.FormatInt(minimum.UnixMilli(), 10))

	// we add the current request
	add := p.ZAdd(ctx, r.Key, &redis.Z{
		Score:  float64(now.UnixMilli()),
		Member: item.String(),
	})

	// count how many non-expired requests we have on the sorted set
	count := p.ZCount(ctx, r.Key, sortedSetMin, sortedSetMax)

	if _, err := p.Exec(ctx); err != nil {
		return nil, errors.Wrapf(err, "failed to execute sorted set pipeline for key: %v", r.Key)
	}

	if err := removeByScore.Err(); err != nil {
		return nil, errors.Wrapf(err, "failed to remove items from key %v", r.Key)
	}

	if err := add.Err(); err != nil {
		return nil, errors.Wrapf(err, "failed to add item to key %v", r.Key)
	}

	totalRequests, err := count.Result()
	if err != nil {
		return nil, errors.Wrapf(err, "failed to count items for key %v", r.Key)
	}

	requests := uint64(totalRequests)

	if requests > r.Limit {
		return &Result{
			State:         Deny,
			TotalRequests: requests,
			ExpiresAt:     expiresAt,
		}, nil
	}

	return &Result{
		State:         Allow,
		TotalRequests: requests,
		ExpiresAt:     expiresAt,
	}, nil
}

Now before we add a request to the redis server we check whether there have been too many requests for the period already and if there were we just deny the request before adding a new one to Redis.

Integrating it as an HTTP middleware

Now that we have implemented two separate counters, how do we make use of them?

One of the ways of doing it is by creating a middleware handler that wraps an existing HTTP handler and adds the rate limiting capability to it so we can compose the handlers with rate limiting when it’s needed or just leave them without it if we’d like to, here’s what it looks like:

package redis_rate_limiter

import (
	"fmt"
	"net/http"
	"strconv"
	"strings"
	"time"
)

var (
	_            http.Handler = &httpRateLimiterHandler{}
	_            Extractor    = &httpHeaderExtractor{}
	stateStrings              = map[State]string{
		Allow: "Allow",
		Deny:  "Deny",
	}
)

const (
	rateLimitingTotalRequests = "Rate-Limiting-Total-Requests"
	rateLimitingState         = "Rate-Limiting-State"
	rateLimitingExpiresAt     = "Rate-Limiting-Expires-At"
)

// Extractor represents the way we will extract a key from an HTTP request, this could be
// a value from a header, request path, method used, user authentication information, any information that
// is available at the HTTP request that wouldn't cause side effects if it was collected (this object shouldn't
// read the body of the request).
type Extractor interface {
	Extract(r *http.Request) (string, error)
}

type httpHeaderExtractor struct {
	headers []string
}

// Extract extracts a collection of http headers and joins them to build the key that will be used for
// rate limiting. You should use headers that are guaranteed to be unique for a client.
func (h *httpHeaderExtractor) Extract(r *http.Request) (string, error) {
	values := make([]string, 0, len(h.headers))

	for _, key := range h.headers {
		// if we can't find a value for the headers, give up and return an error.
		if value := strings.TrimSpace(r.Header.Get(key)); value == "" {
			return "", fmt.Errorf("the header %v must have a value set", key)
		} else {
			values = append(values, value)
		}
	}

	return strings.Join(values, "-"), nil
}

// NewHTTPHeadersExtractor creates a new HTTP header extractor
func NewHTTPHeadersExtractor(headers ...string) Extractor {
	return &httpHeaderExtractor{headers: headers}
}

// RateLimiterConfig holds the basic config we need to create a middleware http.Handler object that
// performs rate limiting before offloading the request to an actual handler.
type RateLimiterConfig struct {
	Extractor   Extractor
	Strategy    Strategy
	Expiration  time.Duration
	MaxRequests uint64
}

// NewHTTPRateLimiterHandler wraps an existing http.Handler object performing rate limiting before
// sending the request to the wrapped handler. If any errors happen while trying to rate limit a request
// or if the request is denied, the rate limiting handler will send a response to the client and will not
// call the wrapped handler.
func NewHTTPRateLimiterHandler(originalHandler http.Handler, config *RateLimiterConfig) http.Handler {
	return &httpRateLimiterHandler{
		handler: originalHandler,
		config:  config,
	}
}

type httpRateLimiterHandler struct {
	handler http.Handler
	config  *RateLimiterConfig
}

func (h *httpRateLimiterHandler) writeRespone(writer http.ResponseWriter, status int, msg string, args ...interface{}) {
	writer.Header().Set("Content-Type", "text/plain")
	writer.WriteHeader(status)
	if _, err := writer.Write([]byte(fmt.Sprintf(msg, args...))); err != nil {
		fmt.Printf("failed to write body to HTTP request: %v", err)
	}
}

// ServeHTTP performs rate limiting with the configuration it was provided and if there were not errors
// and the request was allowed it is sent to the wrapped handler. It also adds rate limiting headers that will be
// sent to the client to make it aware of what state it is in terms of rate limiting.
func (h *httpRateLimiterHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	key, err := h.config.Extractor.Extract(request)
	if err != nil {
		h.writeRespone(writer, http.StatusBadRequest, "failed to collect rate limiting key from request: %v", err)
		return
	}

	result, err := h.config.Strategy.Run(request.Context(), &Request{
		Key:      key,
		Limit:    h.config.MaxRequests,
		Duration: h.config.Expiration,
	})

	if err != nil {
		h.writeRespone(writer, http.StatusInternalServerError, "failed to run rate limiting for request: %v", err)
		return
	}

	// set the rate limiting headers both on allow or deny results so the client knows what is going on
	writer.Header().Set(rateLimitingTotalRequests, strconv.FormatUint(result.TotalRequests, 10))
	writer.Header().Set(rateLimitingState, stateStrings[result.State])
	writer.Header().Set(rateLimitingExpiresAt, result.ExpiresAt.Format(time.RFC3339))

	// when the state is Deny, just return a 429 response to the client and stop the request handling flow
	if result.State == Deny {
		h.writeRespone(writer, http.StatusTooManyRequests, "you have sent too many requests to this service, slow down please")
		return
	}

	// if the request was not denied we assume it was allowed and call the wrapped handler.
	// by leaving this to the end we make sure the wrapped handler is only called once and doesn't have to worry
	// about any rate limiting at all (it doesn't even have to know there was rate limiting happening for this request)
	// as we have already set the headers, so when the handler flushes the response the headers above will be sent.
	h.handler.ServeHTTP(writer, request)
}

So what do we have here? We start with the Extractor interface that returns a string to be used as the key to a rate limiting check. You could have multiple implementations of this interface, checking HTTP headers or any other fields available on the HTTP request to identify the client, the best way to use this is to have a user/account ID that can be pulled from cookies or headers as IPs change with some frequency, so implement a solution that makes sense for your application.

Then we have the RateLimiterConfig struct that wraps all fields needed to perform rate limiting on requests, an Extractor, a Strategy (our counters), how many requests the client can make and for how long. With all this and an http.Handler to wrap we have a fully functional HTTP rate limiter middleware and you could build the same kind of middleware for any other request/response protocol using this same pattern, just changing how this handler is built.

What should I worry about?

First, as we mentioned before, bursty traffic. Clients that use up their whole limit too fast are still going to be an issue for your systems and while the rolling window limiter diminishes this a little bit it should not be the only thing you’re doing to prevent thundering herds of traffic. Another possible solution is to use smaller durations, so instead of giving clients 10_000 requests every hour, give them 160 requests every minute, this way the worst they could do is 160 requests in a short period of time instead of 10_000.

On the same note, do not set limits for very long periods of time, like 100 requests every day, as that makes using the API very frustrating, you make 100 requests and you now have to wait a whole day for the limit to expire again, use smaller periods so people can spread out the load more evenly.

Dynamic deny lists that completely block traffic from specific clients are also a must have on a system like this. You could make this a configuration you’d use on a proxy that sits before the app that performs rate limiting (like a Nginx/Apache server)or at a CDN you’re using (like Cloudflare) so you can quickly blackhole all traffic from known abusers.

The implementation we have build here is a fail closed solution, which means any error (talking to redis, running extractors)causes the request to be denied. This is mostly because a paranoid implementation is generally safer but might not be the best solution for your service, you might want to have a best effort solution that fails open, once it finds a failure somewhere during rate limiting it lets the request happen.

While this might be more user friendly it also adds to the risk that abuses could overload the rate limiter and then overload the systems behind it because they now have no protection with the rate limiter down. So before moving completely to a fail open solution make sure you have other mitigation measures if the rate limiter fails and there is monitoring on all pieces of the stack, both the app and the Redis servers, so you’re notified quickly if any of them start failing and letting on traffic. You might even add an extra HTTP header to downstream clients to let them know that rate limiting failed but the request was still sent so they can make a decision if they want to accept that request or not.

Again, your main goal with rate limiting should be protecting your systems so that you’re providing the best service possible for most users and preventing that a small number of misbehaving clients wreak havoc on your applications preventing good users from using them.

Explicando optimistic locking

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 12 Sep 2021 00:00:00 UTC

O código listado aqui está nesse repositório do github.

Dia desses depois de ver outra treta na comunidade Golang dizendo que tem que escrever SQL na mão mesmo mandei:

Oi você bichão fodão que não usa ORM porque é ninja demais, lembrou de implementar optimistic locking pros seus objetos ou nem sabe o que é isso? 🤡
— Maurício Linhares (@mauriciojr) September 2, 2021

E tinha gente perguntando o que era optimistic locking, que é um conceito importante pra quem usar bancos de dados, ainda mais nos ambientes de sistemas distribuídos que muita gente trabalha hoje em dia. A idéia de optimistic locking (também conhecido como Optimistic concurrency control - OCC) é que em muitos casos, conflitos na hora de alterar dados em sistemas são incomuns, então incorrer no custo e complexidade de usar transações diretamente nos bancos de dados não vale a pena.

A implementação mais comum é a de que os objetos que você vai alterar durante uma transação tenham um numero/valor de versão associados a eles e sempre que você for fazer uma alteração, você verificaria se o numero de versão que você tem externamente é o mesmo que está salvo no banco de dados. Se ambos forem iguais, você tem a versão atual do objeto e pode alterar ele sem correr o risco de perder dados, se não forem, você sabe que a versão que você tem está desatualizada e não é seguro fazer a alteração.

Um dos usos principais desse tipo de solução é pra aplicações web, onde não é viável manter uma transação do banco de dados aberta enquanto o usuário está entrando informações pra um dado específico. Imagine alguém atualizando os dados de um usuário X no sistema, essa pessoa pode abrir o formulário, sair pra tomar um café, conversar com um colega ou simplesmente ir pra casa no fim do expediente e deixar o formulário aberto no seu navegador. Se você mantivesse uma transação no banco de dados aberta pra esses dados durante esse tempo todo, rapidamente seria impossível fazer qualquer alteração no banco.

Então, transações pra operações longas ou que nós não temos controle sobre quanto tempo elas vão durar são inviáveis e é aqui que temos outro problema, onde quem desenvolve simplesmente ignora a transação longa e simplesmente deixa o código fazer a alteração somente quando o usuário envia o formulário do navegador. Vejamos a sequência:

Usuário 1: Abre o formulário do cliente X;
Usuário 1: Sai pra almoçar;
Usuário 2: Abre o formulário do cliente X;
Usuário 2: Altera informações do cliente X e envia o formulário;
Usuário 1: Volta do almoço, altera informações do cliente X e envia o formulário;

Nesse momento, se você não tem transações longas e simplemente grava tudo na hora que o formulário foi enviado, é bem provável que todas as mudanças feitas pelo Usuário 2 foram perdidas, é um problema comum com bancos de dados que é chamado de atualização perdida (lost update). E é nesse caso que uma solução de optimistic locking é a solução ideal, onde temos uma transação longa fora do banco de dados e ainda queremos perceber quando alguém está mandando informações desatualizadas pro sistema.

Implementação

package optimistic_locking

import (
	"context"
	"io"
)

const (
	PostsTable       = "posts"
	CreatePostsTable = `DROP TABLE IF EXISTS posts;
CREATE TABLE posts (uuid VARCHAR NOT NULL PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL, version VARCHAR NOT NULL);`
)

type Post struct {
	UUID    string
	Title   string
	Content string
	Version string
}

type Posts interface {
	io.Closer
	// Find tries to find a post, returns an error if the post does not exist
	Find(ctx context.Context, uuid string) (*Post, error)
	Save(ctx context.Context, post *Post) error
	List(ctx context.Context) ([]*Post, error)
	// Delete deletes a post and returns whether it actually found the post to be deleted or not.
	Delete(ctx context.Context, uuid string) (bool, error)
	Migrate(ctx context.Context) error
}

Temos aqui a classe que vamos usar como exemplo e a interface que define como vamos mapear essa classe pro banco de dados. O tipo de banco de dados que vai ser utilizado não tem muita relevância, você vai enfrentar esse tipo de problema em praticamente qualquer banco de dados, seja SQL ou NoSQL. Alguns, entretanto, tem suporte direto a optimistic locking então a solução ficaria mais simples.

Vejamos uma implementação que não faz uso de optimistic locking primeiro:

package optimistic_locking

import (
	"context"
	"database/sql"
	"fmt"
	sq "github.com/Masterminds/squirrel"
	"github.com/google/uuid"
	_ "github.com/mattn/go-sqlite3"
	"github.com/pkg/errors"
)

func NewBrokenPosts(path string) (*BrokenSqlitePosts, error) {
	db, err := sql.Open("sqlite3", path)
	if err != nil {
		return nil, err
	}

	return &BrokenSqlitePosts{
		db: db,
	}, err
}

type BrokenSqlitePosts struct {
	db *sql.DB
}

// outros métodos

func findWith(ctx context.Context, runner sq.BaseRunner, uuid string) (*Post, error) {
	scanner := sq.Select("uuid", "title", "content", "version").
		From(PostsTable).
		Where("uuid = ?", uuid).
		RunWith(runner).
		QueryRowContext(ctx)

	post := &Post{}

	if err := scanner.Scan(&post.UUID, &post.Title, &post.Content, &post.Version); err != nil {
		if err == sql.ErrNoRows {
			return nil, errors.Wrapf(err, "could not find post with UUID: %v", uuid)
		}

		return nil, err
	}

	return post, nil
}

func (s *BrokenSqlitePosts) Save(ctx context.Context, post *Post) (err error) {
	if post.UUID == "" {
		post.UUID = uuid.New().String()

		return assertAffected(sq.Insert(PostsTable).
			Columns("uuid", "title", "content", "version").
			Values(post.UUID, post.Title, post.Content, post.Version).
			RunWith(s.db).
			PlaceholderFormat(sq.Dollar).
			ExecContext(ctx))
	}

	return assertAffected(sq.Update(PostsTable).
		Where(sq.Eq{"uuid": post.UUID}).
		Set("title", post.Title).
		Set("content", post.Content).
		Set("version", post.Version).
		RunWith(s.db).
		ExecContext(ctx))
}

func assertAffected(r sql.Result, err error) error {
	if err != nil {
		return err
	}

	affected, err := r.RowsAffected()
	if err != nil {
		return err
	}

	if affected != 1 {
		return fmt.Errorf("expected only %v row to be affected but %v rows were affected", 1, affected)
	}

	return nil
}

O método mais importante aqui é o Save, quando é um objeto sem UUID, colocamos um UUID nele e inserimos ele no banco com um INSERT. Quando ele tem um UUID assumimos que ele já existe no banco de dados e geramos um UPDATE. Em ambos os casos, verificamos que uma linha foi modificada no banco, pra garantir que ambos os comandos realmente causaram uma alteração. Aqui não fazemos nenhuma verificação pra garantir que estamos atualizando a versão correta da linha na tabela, então esse código vai sofrer do problema de atualizações perdidas.

Vejamos então uma implementação que faz a verificação de versões antes de atualizar a linha da tabela:

package optimistic_locking

import (
	"context"
	"database/sql"
	"fmt"
	sq "github.com/Masterminds/squirrel"
	"github.com/google/uuid"
	"github.com/rs/zerolog/log"
)

func NewVersionedPosts(path string) (*VersionedPosts, error) {
	posts, err := NewBrokenPosts(path)
	if err != nil {
		return nil, err
	}

	return &VersionedPosts{
		BrokenSqlitePosts: posts,
	}, nil
}

type VersionedPosts struct {
	*BrokenSqlitePosts
}

func (s *VersionedPosts) Save(ctx context.Context, post *Post) (err error) {
	if post.UUID == "" {
		post.Version = uuid.New().String()
		return s.BrokenSqlitePosts.Save(ctx, post)
	}

	tx, err := s.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
	if err != nil {
		return err
	}

	defer func() {
		if err != nil {
			if txErr := tx.Rollback(); txErr != nil {
				log.Err(txErr).Str("uuid", post.UUID).Msg("failed to rollback transaction")
			}
		}
	}()

	result, err := findWith(ctx, tx, post.UUID)
	if err != nil {
		return err
	}

	if result.Version != post.Version {
		return fmt.Errorf("version mismatch: you're trying to update post with version %v but the current DB version is %v", post.Version, result.Version)
	}

	post.Version = uuid.New().String()

	if err := assertAffected(sq.Update(PostsTable).
		Where(sq.Eq{"uuid": post.UUID}).
		Set("title", post.Title).
		Set("content", post.Content).
		Set("version", post.Version).
		RunWith(tx).
		ExecContext(ctx)); err != nil {
		return err
	}

	return tx.Commit()
}

Aqui, quando vamos executar um UPDATE, iniciamos uma transação no banco, carregamos o objeto atual do banco de dados e verificamos se o valor de Version no banco de dados é o mesmo do objeto que recebemos pra atualizar. Se as versões não forem as mesmas, retornamos um erro e fazemos rollback na transação, se elas forem iguais, alteramos a versão atual e atualizamos o objeto no banco.

A implementação da solução vai ser dependente do banco de dados que você estiver utilizando. Como o SQLite não tem suporte a operações como SELECT ... FOR UPDATE pra travar uma ou um grupo de linhas da tabela durante uma transação, a única opção é fazer com que a transação seja SERIALIZABLE, mas cada banco de dados vai ter uma forma específica de lidar com esse problema. O ideal é procurar como funciona o travamento de linhas/documentos específicos em uma tabela/coleção pro seu banco de dados e usar essa funcionalidade pra implementar esse método.

Podemos verificar o comportamento com um teste de integração:

package optimistic_locking

import (
	"context"
	"encoding/base64"
	"fmt"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"io/ioutil"
	"math/rand"
	"os"
	"testing"
)

var (
	testCtx = context.Background()
)

func withPosts(t *testing.T, factory func(path string) (Posts, error), callback func(t *testing.T, p Posts)) {
	file, err := ioutil.TempFile("", "prefix")
	if err != nil {
		t.Fatalf("failed to create tmp file: %v", err)
	}
	defer os.Remove(file.Name())

	posts, err := factory(file.Name())
	if err != nil {
		t.Fatalf("failed to create Posts: %v", err)
	}

	if err := posts.Migrate(testCtx); err != nil {
		t.Fatalf("failed to migrate database: %v", err)
	}

	callback(t, posts)
}

func randomString(t *testing.T) string {
	bytes := make([]byte, 128)
	_, err := rand.Read(bytes)
	require.NoError(t, err)

	return base64.StdEncoding.EncodeToString(bytes)
}

func samplePost(t *testing.T) *Post {
	return &Post{
		Title:   randomString(t),
		Content: randomString(t),
	}
}

func TestVersionedPosts_Save(t *testing.T) {
	tt := []struct {
		name    string
		err     string
		factory func(path string) (Posts, error)
	}{
		{
			name: "with versioned posts",
			err:  "version mismatch: you're trying to update post with version",
			factory: func(path string) (Posts, error) {
				return NewVersionedPosts(path)
			},
		},
		{
			name: "with broken posts",
			factory: func(path string) (Posts, error) {
				return NewBrokenPosts(path)
			},
		},
	}

	for _, ts := range tt {
		t.Run(ts.name, func(t *testing.T) {
			withPosts(t, ts.factory, func(t *testing.T, p Posts) {
				post := samplePost(t)

				require.NoError(t, p.Save(testCtx, post))

				post.Title = "new title"
				require.NoError(t, p.Save(testCtx, post))

				savedPost, err := p.Find(testCtx, post.UUID)
				require.NoError(t, err)

				savedPost.Content = "new content"
				require.NoError(t, p.Save(testCtx, savedPost))

				post.Content = "this will overwrite"
				err = p.Save(testCtx, post)

				if ts.err != "" {
					require.NotNil(t, err)
					assert.Contains(t, err.Error(), ts.err)
				} else {
					require.NoError(t, err)
				}
			})
		})
	}
}

No teste, quando usamos a implementação VersionedPosts, ele tem que falhar com o erro avisando que as versões não são as mesmas, já na implementação que usa BrokenSqlitePosts ele simplesmente ignora que está atualizando a versão errada do objeto e sobrescreve a alteração que aconteceu antes.

Se você implementou código que faz esse mapeamento de objetos pra qualquer tipo de banco de dados em uma aplicação que tem esse uso de transações longas ou alterações concorrentes dos mesmos recursos (o que é extremamente comum na maior parte dos sistemas atuais), o ideal é ter algum tipo de optimistic locking implementado pra esses objetos. Essa é uma das maiores vantagens de se usar ferramentas de mapeamento objeto-relacional (ou até pra outros bancos de objetos/documentos), já que elas geralmente tem uma implementação desse tipo de solução.

É sempre importante lembrar que existe uma barreira de tradução (impedance mismatch) entre a sua aplicação que geralmente trabalha com objetos/estruturas de dados em memória e onde os dados estão realmente persistidos no banco. O trabalho de escrever uma camada que faz essa tradução é grande e complexo, porque ele precisa considerar todos esses casos especiais que não ficam visíveis quando estamos escrevendo o código que interage com o banco de dados até termos que considerar os diversos modos diferentes de uso que vão acontecer, então no geral é melhor usar uma ferramenta que faça esse mapeamento do que escrever todo o código de tradução, seja SQL ou algum outro banco, na mão, porque muitas vezes podemos não considerar problemas que já foram resolvidos por outras pessoas.

Outras soluções

Apesar de ser uma solução comum, optimistic locking não é a melhor solução pra todos os problemas de concorrência em transações longas, existem outras soluções conhecidas, especialmente os conflict-free replicated data type (CRDTs), que são tipos que operam de forma comutativa, onde a ordem das operações não afeta o produto final. Serviços como Google Docs implementam estruturas de dados parecidas pra garantir a edição paralela de documentos e a maior parte dos carrinhos de compra de comércio eletrônico também precisam implementar soluções parecidas pra que as várias alterações no carrinho pareçam atômicas mesmo que elas não sejam.

Building DigitalOcean's API gateway

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 14 Jan 2021 00:00:00 UTC

TL;DR: this is mostly a text version of a presentation I’ve done a couple times (English or Portuguese) on the history of building DigitalOcean’s API gateway. How we made it easier for folks to build new microservices instead of continuing to add code to our monoliths, the successes, failures and lessons learned.

First, where were we?

DigitalOcean had 3 monoliths back in 2016 when we started (all 3 are still alive today, albeit much smaller than they were before). Why were there 3? There is a shared library that contains most of the logic that all 3 applications use, so in reality we had one “monolith library” that was reused at all 3, most logic changes were made into this library and then it would be upgraded at every single app separately. The library still exists today and continues to be updated every once in a while.

As you can imagine, as more and more changes started to happen, with the company growing incredibly fast both in terms of business and hiring, this wasn’t ideal. Many times someone would change this core library, deploy one of the apps (the control pane you see when you sign in), but not the JSON API. So you could end up with a new feature visible at the control panel but not at the API. There would always be a time when this library version wouldn’t match across all applications, as we were not deploying them all at once every single time, so there was always space for there to be a mismatch of features available.

This also meant that the test suite was growing by leaps and bounds, getting slower, making running the test suite and deployments a pain. There was growing interest in doing something about this, but we couldn’t just migrate everything out. There was also a lot of interest in NOT doing Ruby anymore. There was a growing body of Golang code all over the place and people wanted to use it for their new services instead of building it all inside the old monoliths.

There was a catch here though, how could they possibly do all the things the Rails apps were doing, like authentication, rate limiting, authorization, feature flipping, routing, error handling and all the shared logic, in a new language, without repeating the same thing across all new projects?

We needed something that would be language agnostic and could be run side by side with the existing monoliths. At this point we knew a library ( like Twitter’s Finagle) wasn’t an option. Our solution had to work with the old Ruby code and the new Golang stuff (and JS, maybe Python. It was all up in the air back then). We wouldn’t be able to build a library with all the shared features needed in multiple languages.This is where the API gateway comes in.

What is an API gateway?

The microservices.io patterns list has a great definition for the API gateway:

Implement an API gateway that is the single entry point for all clients. The API gateway handles requests in one of two ways. Some requests are simply proxied/routed to the appropriate service. It handles other requests by fanning out to multiple services. 

It took us multiple meetings and discussions between Joonas Bergius, Nan Zhong, Joe Friedl, me and even Phil Calçado, who was the engineering lead (not sure if this was actually the title but he was the boss), to come to the realization that we were going to build an API gateway.

What came out of the brainstorms:

It was going to be a pure HTTP proxy, no GRPC, GraphQL or other special protocols;
We would not change request or response bodies, so there wouldn’t be any protocol translation. Applications would be responsible for parsing the input and producing valid outputs, in the format the client was expecting;
We’d augment the requests and responses with extra information encoded as HTTP headers, including user information, rate limiting details, tracing metadata, features enabled, so all downstream applications would have to do was looking at the headers, not special exchange format.
Route configuration would be self-service, teams would call a service with their route configuration and the proxy would automatically pick up changes and update it’s routing table. It would perform basic health checking and remove destinations that weren’t reachable, but otherwise would take any request and forward it to the registered services.
It would also include filters that could be run before or after a request was processed, these filters would include authentication, rate limiting, feature flipping and all the other shared pieces teams would need to be available to build apps out of the monoliths.

Does it look a lot like Java’s Servlet API or Ruby’s Rack? Of course, those were the main inspirations for the design. Going for a simple, well known, design would make us move faster as there would be less stuff to do and we could always complicate it in the future. This was one of the best decisions we made and this is still the way it works nowadays.

Why didn’t we pick something that existed? Fair question. There really weren’t many options back then and the ones that did exist did not make it easy to integrate custom code. Our very first challenge was to decrypt and parse Rails sessions to authenticate users, and there was plenty of complex code to decrypt, unmarshal (from Ruby’s serialization format) and make sense of session data.

Nginx, that was one of the options when we started, only offered integration through Lua scripts, so we’d have to write this session decrypting and parsing logic in Lua scripts we’d bundle with our custom Nginx install and that did not feel like a great solution. It was hard to test the scripts, no one had any experience with Lua and the whole setup wasn’t robust, these scripts were seen mostly as doing small changes in the request flow, not for building complex logic.

So, we started building our own API gateway in Golang.

How did we start?

Once we had the basic proxy built, we placed it in front of some of the traffic, to make sure it could proxy correctly. The first two “filters” we built were the authentication filters, for Rails sessions and OAuth tokens. To roll these out, we started by “doing” the filter work for both monoliths but not acting on them, we’d check if the output we came to in our implementation was the same the monoliths decided on and if the response was also the same; if our own service said a request should be denied but the monoliths responded with an OK or the other way around, we’d log that and work on figuring out why it didn’t work.

As we built these filters, another pattern emerged, instead of having all this code inside the API gateway itself, we thought it would be faster to separate the code into a separate service. The initial goal was mostly to make it reusable by other teams if needed and make it easier for us to deploy smaller changes as deploying the gateway itself was a complicated and slow process (due to the way our internal K8s clusters networking was setup back then, we couldn’t run the gateway on them, so it ran on droplets). After that, almost all filters were just glue code to call an external service that actually knew how to get the job done.

While it was a response to environmental constraints and not really something we planned, this design made the gateway itself smaller and more reliable. A lot of the logic would live in these microservices instead of the gateway and updates to them had a much smaller blast radius when something went awry.

@nanzhong finally switched it to serve all traffic sometime in November 2016, a couple months after our announcement, including full support for authentication with the Rails session and OAuth tokens.

Onboarding the first customers

So far we had only done work internally. We weren’t exposing any of this to other teams yet, as we wanted to verify it was all behaving as expected before letting people register their services. The first integrations were bumpy and the experience for the teams we were integrating with was hard.

We lacked documentation and good examples. Teams had to come to us frequently to make sure their services were really up, their configurations were valid, how routing would work (what options are available? Can I use wildcards? Can I use URL parameters? In what order are routes matched?) and how they should integrate route registration into their workflows. This led to a lot of manual labor on our end helping people do stuff they could be doing themselves had we done the work to make it easier to onboard them. The lesson here was clear, if you’re working on infrastructure, make sure you can stay out of the way when people are doing their work, you want them to work on their time and not be an impediment for their work.

We also lacked best practices for what the backend services should look like. Multiple teams were being formed of new developers that didn’t have a lot of experience with building applications with Golang and we did not provide guidance here on how they should set up their applications and what configurations were important.

One recurrent issue we had at the beginning was that Golang’s http.Server class assumes all timeouts are infinite unless you set them (ReadTimeout, ReadHeaderTimeout, WriteTimeout, WriteIdleTimeout) and this would lead to services getting a broken connection from the client (the API gateway) and wouldn’t know what it was until we checked their configs and noticed they just didn’t set a value (while we did have a max timeout for all requests on our end, so we’d close a connection that took too long).

Providing better guidance here for basic options and configurations that teams should be doing in all their apps would have saved a lot of time and effort for everyone.

Next, when registering routes, teams would talk directly to the key-value store we were using to store the routing table (Consul) and as you can imagine this wasn’t ideal. There was very little validation in place and it was super simple for people to register routes that wouldn’t load or that wouldn’t really route anything due to being incomplete. This eventually led to an outage where there was a nil in a place where the gateway did not expect a nil to be that would crash it whenever it loaded routes., Not a good day, I must tell you.

A new service then emerges (microservices all the way) to be the route collector. It would be the bridge between clients and the gateway, validating routes before sending them to Consul and producing a single list of routes with all services the gateway could use to build its routing table. The gateway would poll this service to get the current routing table whenever it changes and update itself if needed, it would still have a local static routing table if the routing service was not available when it started to at least some known destinations. @sivillo led the charge building this one.

Here’s how a service config looks like:

service := &config.Service{

      Name:  "uuidaggregator",

      Routes: []*config.Route{

         {

            Hosts: []string{"uuid-generator-api.digitalocean.com"},

            Paths: []string{“/generate-uuid”},

            BeforeFilters: []string{

               edgeconsts.FilterOauthTokenAuthentication,

               edgeconsts.FilterEnsureAuthentication,

               edgeconsts.FilterPublicAPIRateLimiter,

            },

         },

         {

            Hosts: []string{“cloud.do.com”},

            Paths: []string{“/generate-uuid”},

            // perform cookie session authentication for cloud requests,

            // if it fails deny the request using the ensure auth filter

            BeforeFilters: []string{

               edgeconsts.FilterCookieSessionAuthentication,

               edgeconsts.FilterEnsureAuthentication,

            },

         },

      },

      // Timeout in seconds for the entire request (Default 10s; Max 60s)

      Timeout:  	15,

      MaxIdleConns: 5,

      Destinations: []*config.Destination{

         {

            Scheme: "https",

            Host:   fmt.Sprintf("uuidaggregator-tls.com"),

         },

      },

      // this health check config is optional, the defaults are ok

      HealthCheck: &config.HealthCheck{

         Path:              	"/health",

         IntervalInSeconds: 	5,

         MaxFailedAttempts: 	3,

         MinSuccessfulAttempts: 3,

         TimeoutInSeconds:  	3,

      },

}

When building the routing table, as it’s not made of a single routing file as you’d see in a Rails application, we also had to do some work. We couldn’t just register the routes in the order the services showed up as almost all routers will stop at the first match. That means that a route like:

/api/v1/droplets/*

Could possibly come before something like:

/api/v1/droplets/*/actions

And that would cause the first one to be matched and the second one to be ignored just because of the order they were registered, especially if they were registered in separate services. Our solution here was to first prefer routes that had a domain and/or methods defined and then sort them by longest path, so longest routes, that would effectively produce the longest match, would be found first. We also push full paths before wildcard routes, so the routing table first tries to match the full path matches and only then goes for the wildcard routes that still follow the same pattern of longest matches first, preventing the problem above.

Also, if I were to start this all over, we would not support wildcards for routes at all, in an API gateway almost all times the services know the paths they support already so here’s no need to have this complication on your routes. Just match it all on exact matches and avoid the wildcard nightmare. In a distributed environment with dozens of services like we have today, they make the routing table hard to reason about and don’t really bring any benefits other than making developers type a bit less.

The single entry point for everything

Being the single entry point for all requests that deal with account resources and the accounts themselves, we were in a position to collect data about every single service we had behind us and offer that to the teams that were registering their services. This made the API gateway an important piece when detecting and working through incidents as it had a general view of all services but it also made us a pretty common target for when stuff wasn’t behaving as expected.

Whenever something broke, we’d be the first or one of the first teams to be engaged as we were the entrypoint for everything and we had a better understanding of what was going on. That, obviously, wasn’t ideal but was also another side effect of not thinking about how people out of the team could consume the information we were collecting. We had our own dashboards with metrics and visualizations but they were not easy to understand if you didn’t have all the application and metrics context. It had no explanation for the metrics, thresholds or hints as to what was going on.

Thankfully, our Observability team did actually know how to build dashboards and we shamelessly copied their style building dashboards for our operations team with plenty of explanations, simpler components with coloring schemes that made it easy to check if something wasn’t behaving as expected, and hints. This way they could quickly figure out where an outage was going on, what systems were being affected and how, without having to engage us on every incident.

This also led us to eventually build general dashboards with all metrics we collected for every single service we had registered, automatic alerts for when services were misbehaving (too many 5xx, failures to produce responses, failure to health check), logging trails for all requests that were served, even if the backend service had crashed, so teams could quickly debug what happened on their end even if there were no logs or data on their own metrics or logs.

Adding security measures was also simplified, as our security team would just request or build functionality that made all services more secure by default, introducing filters in all requests to perform basic sanitization and checking. Services didn’t even have to add this to their configs, we’d just introduce them on all routes by default and they’d be none the wiser.

Building features

While we were mostly building new code it was to support or cover existing Ruby libraries or APIs and we had to make it all work seamlessly both for the new apps and for the existing Ruby ones. This led to many implementations that had to either behave the same way or had to expose APIs compatible with what the Ruby apps were doing, otherwise we could end up with a problem where the new microservices and the Ruby app wouldn’t agree with each other.

One of the first examples of this was when @lxfontes was doing our rate limiter. The original solution was based on rack-attack and backed by Redis, as it was shared across the whole API we couldn’t really build something completely new as it would lead to the microservices getting a rate limit that was different than the one you’d see if your call went to the old Ruby API app, the new limiter had to work the same way the Ruby one did so they would both produce the same response. This was both good and bad, as the basic idea of how rate limiting would work was already implemented and we could just read the code in Ruby and port it to Golang but it also made it harder for us to improve upon the solution with a new or better implementation as it would lead to conflicting rate limiting values being returned to clients.

Another case was the feature flipper. We were using jnunemaker/flipper to perform feature flipping but was getting slow for the amount of users we were flipping and we also had to expose it somehow for the new applications. This time we could actually build a completely new service because the library had support for an HTTP API backend, so we built a new, faster, backend and implemented the HTTP API the library expected, so the Ruby apps could continue to call this HTTP API to perform feature flipping while other services connected to the GRPC service that was now the main interface to flip users.

We even had to expose some of the Ruby code, mostly authorization policies, as a GRPC service so people wouldn’t have to rewrite all the policies they already had in place themselves. We just stood up our shared ruby library as a GRPC service and had services call it to run the policies they needed to authorize users. While this wasn’t something that was directly connected to the API gateway work it was still part of our mission to make all the shared infrastructure that was part of the monoliths available for the new microservices that were being built and I think this is a dead simple way to kickstart your move out your existing monolith.

We tried as much as possible not doing big bang migrations and rebuilding everything from scratch with incompatible interfaces. There was no time to migrate everything, including the old Ruby code, so the new implementations always had to maintain some compatibility and this made migrating into the new services much simpler and mostly invisible for people that were using the legacy systems.

Still kicking

As I write this the Edge Gateway, as we call it, is still there serving every single request DigitalOcean’s API and control panel take and some pieces of it seem to have had a bigger impact in what I think is our success so far.

Not being a library

Teams don’t talk to the gateway itself, it’s a somewhat invisible piece of the puzzle for anyone that’s not on our Edge team and this has made our life so much easier. There is no direct dependency between the apps we proxy to and the gateway, they all agree on a set of HTTP headers that will contain extra information and that’s it.

When doing integration testing for the apps, you can send these headers to yourself as if you were the gateway (in production services use mTLS to verify they are indeed getting a request from the gateway) and this relieves us to ask people to upgrade libraries when we add features or update existing functionality. It’s just HTTP, no magic at all.

Being self service

It took us some time to sort out all the kinks and make it easier for people to register their services, but from the get go we decided we did not want to be involved in the way people register their services. This might have led to some more work initially on our end to find the possible ways the configuration could be invalid (and we still find cases every once in a while, like adding a filter that expects the user to be authenticated but not having any authentication filter before it in the route) and prevent taking over routes but has led to teams being able to get their services registered much faster and without having to ask us for “permission” on pull requests.

If you’re working on an infrastructure team, you should do all in your power to avoid being the “gatekeeper” for the teams you service. Also, people often don’t think about User Experience when building infrastructure services and that is a huge mistake, infrastructure services still have user interfaces and you have to think about how you’re going to expose that for users, including providing sane defaults (real timeouts instead of the infinite timeouts in Golang’s default HTTP client and server) and blocking operations that might be syntactically but not logically correct.

Being reliable

Monitors, smoke tests, integration tests, constant verification through internal and external callers that the service is doing what it’s supposed to do. We knew from the get go that getting people to trust us, the new team, building a new application that could eventually become a single point of failure for the control panel and API, would get a lot of effort and would be really hard to maintain if we made too many mistakes causing downtime. Being reliable paid off, even if we did cause incidents here and there :D

Don’t impose

Still today, you can add features to the old monoliths, they’re still there, taking a fraction of the traffic they used to when we started, but they still exist and that’s a good thing. While we wanted to turn them off we knew this wasn’t going to be the end goal, the end goal for us was to make our platform so much better that people wouldn’t even think about building stuff on the monoliths. We offer more features, metrics, dashboards, better error handling, logs and more.

Had there been a mandate for people to “use the Edge gateway” people would feel forced into using us even if it wasn’t the best solution. Internal teams should assume people using them are also customers, because they are, and offer a solution that solves a need they have. A relationship that is forced upon the other team will eventually lead to workarounds and bad outcomes.

The team

We wouldn’t have arrived where we are right now without the multiple people that had a hand directly or indirectly here, Julian Miller that has been on the team for as long as I remember by now and has had his hands in everything we’ve done, Nan Zhong, Louis Sivillo that was also on the team for a long while and led the charge in building our route collection service, migrating routes from the old storage and making sure it was all seamless for the folks using it, Lucas Fontes and Joe Friedl that made important contributions building features people still depend on every day when they were on the team, Dave Worth, Mike Holly and Hugo Corbucci, that had much more context on everything DigitalOcean and were always there to answer questions when I had no idea where something was or how it worked, Joonas Bergius, the first Engineering Manager on the team that was there building the architecture, validating ideas and getting us to deliver on the goal of improving the experience for teams building microservices at DigitalOcean, Steven Black, our second EM, that made sure there was a place for the team to grow and continue to build new features and Nick Silkey, our current EM, master of getting stuff that’s important done and finding budget where there wouldn’t be any. It took many hands to get us to where we are now and many more to keep moving it forward.

Implementando uma hashtable em Java

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 15 Oct 2020 00:00:00 UTC

Você encontra o código fonte completo discutido aqui em github.com/mauricio/hashtable.

Hashtable, aquela estrutura de dados mágica, que a gente usa todo dia mas pensa que é alguma magia que faz ela funcionar, é na verdade bem mais simples de ser implementada do que se imagina. Comentamos um pouco dos usos e também de como se implementa essa estrutura de dados nesse episódio do hipsters.tech então vamos ver agora direto no código como seria implementar isso na mão em Java.

A primeira coisa a fazer é entender a estrutura básica que vamos usar, como falamos no podcast, a implementação mais simples de hashtables é usar um array de arrays, onde o primeiro array define os buckets onde vamos distribuir os objetos e o array interno é o bucket onde adicionamos objetos. Do lado de fora, estamos usando mais memória, já que vamos ter espaços vazios no primeiro array, mas ganhamos na velocidade de se encontrar os objetos dentro da estrutura.

Implementando o objeto entry

Como estamos falando de um objeto que vai salvar pares de chave/valor, precisamos de um objeto que represente esses pares. Java já define uma interface pra isso, java.util.Map.Entry, então vamos implementar essa interface pra representar os nossos pares de chave/valor:

public class Entry<K, V> implements Map.Entry<K, V> {

    private final K key;
    private V value;

    public Entry(K key, V value) {
        this(key);
        this.value = value;
    }

    public Entry(K key) {
        if (key == null) {
            throw new IllegalArgumentException("key can't be null");
        }
        this.key = key;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof Entry)) return false;
        Entry<?, ?> entry = (Entry<?, ?>) o;
        return key.equals(entry.key) &&
                Objects.equals(value, entry.value);
    }

    @Override
    public int hashCode() {
        return this.key.hashCode();
    }

    @Override
    public K getKey() {
        return this.key;
    }

    @Override
    public V getValue() {
        return this.value;
    }

    @Override
    public V setValue(V value) {
        var oldValue = this.value;
        this.value = value;
        return oldValue;
    }

}

A implementação é uma classe com dois atributos, key e value, onde key nunca muda e não pode ser null, a única coisa que vai mudar é o valor pro qual cada chave vai apontar. Essa classe é a que vamos usar como conteúdo da nossa hashtable, então vamos ter um array de arrays de Entry<K,V>.

Iniciando a hashtable

O ponto de entrada principal da nossa hashtable vai ser a implementação do método put, que coloca um par de chave/valor na nossa tabela, então vamos construir ele da forma mais simples possível no momento:

package tech.hipsters;

import java.util.ArrayList;
import java.util.List;

public class Hashtable<K, V> {

    private List<List<Entry<K, V>>> buckets;

    public Hashtable() {
        this(16);
    }

    public Hashtable(int s) {
        this.buckets = new ArrayList<>(s);
        for (int x = 0; x < s; x++) {
            buckets.add(null);
        }
    }

    public V put(K key, V value) {
        throwIfNull(key);

        var entry = this.findOrCreateEntry(key);
        var oldValue = entry.getValue();
        entry.setValue(value);

        return oldValue;
    }

    private int bucketIndexFor(K key) {
        return key.hashCode() % this.buckets.size();
    }

    private Entry<K, V> findOrCreateEntry(K key) {
        var bucketIndex = this.bucketIndexFor(key);
        var bucket = this.buckets.get(bucketIndex);
        if (bucket == null) {
            bucket = new ArrayList<>();
            this.buckets.set(bucketIndex, bucket);
        }

        for (var entry : bucket) {
            if (key.equals(entry.getKey())) {
                return entry;
            }
        }

        var entry = new Entry<K,V>(key);
        bucket.add(entry);
        return entry;
    }

    static void throwIfNull(Object key) {
        if (key == null) {
            throw new NullPointerException("key must not be null");
        }
    }

}

Falamos várias vezes sobre usar arrays de arrays pra nossa tabela e aqui temos um List<List<Entry<K, V>>>, por que?

Porque trabalhar com arrays genéricos (e arrays de arrays genéricos) em Java é terrível, então aqui tomamos um atalho pra deixar o código um pouco mais legível, mas vamos usar as listas quase como se fossem arrays.

Temos dois métodos privados importantes aqui, bucketIndexFor e findOrCreateEntry. O primeiro serve pra descobrir em qual bucket a chave em questão vai ser encontrada. Nossa lógica aqui é usar o resto da divisão do resultado do hashCode da chave pelo tamanho do nosso array de buckets (que nesse momento é 16). Vamos receber um número de 0 a 15, exatamente a posição no array onde a chave deve ficar. Essa é a grande vantagem de usar hashtables, em vez de percorrer as 16 posicões, com uma única operarão matemática (que é barata) nós já sabemos onde procurar a nossa chave.

Já em findOrCreateEntry procuramos se já existe um array na posição onde a chave vai estar, se ele não existir, criamos um novo array e colocamos ele na posição. Depois navegamos o array pra tentar achar um par com a chave que estamos procurando, se ela for encontrada, é retornada. Se sairmos do laço, não encontramos um par pra essa chave então ela precisa ser adicionada no bucket pra ser retornada.

Com isso a implementacão de put fica mínima, usamos findOrCreateEntry pra encontrar o par, colocamos o novo valor nele e retornamos o valor anterior, que vai ser null pra novas chaves. Veja que esse método serve tanto pra novas chaves como chaves que já existem, não precisamos de um caso especial pra novas chaves.

Agora vamos implementar mais alguns métodos necessários a nossa tabela, get (pra pegar um valor dada uma chave), containsKey (pra saber se uma chave existe na tabela) e remove (pra remover uma chave da tabela):

package tech.hipsters;

import java.util.ArrayList;
import java.util.List;

public class Hashtable<K, V> {
    
    // métodos anteriores 

    private Entry<K, V> findEntry(K key) {
        var bucket = this.findBucket(key);
        if (bucket == null || bucket.isEmpty()) {
            return null;
        }

        for (var entry : bucket) {
            if (key.equals(entry.getKey())) {
                return entry;
            }
        }

        return null;
    }

    private List<Entry<K, V>> findBucket(K key) {
        return this.buckets.get(this.bucketIndexFor(key));
    }

    public V get(K key) {
        throwIfNull(key);

        var entry = this.findEntry(key);

        if (entry != null) {
            return entry.getValue();
        }

        return null;
    }

    public boolean containsKey(K key) {
        throwIfNull(key);

        return this.findEntry(key) != null;
    }

    public V remove(K key) {
        throwIfNull(key);

        var bucket = this.findBucket(key);
        if (bucket == null || bucket.isEmpty()) {
            return null;
        }

        for (var it = bucket.iterator(); it.hasNext();) {
            var next = it.next();
            if (key.equals(next.getKey())) {
                it.remove();
                return next.getValue();
            }
        }

        return null;
    }

}

Temos agora dois novos métodos privados, findBucket e findEntry. O primeiro é um atalho pra achar um bucket dado uma chave específica, usando bucketIndexFor. O segundo já é um pouco mais complexo, porque agora queremos encontrar um par dado uma chave mas não queremos adicionar essa chave se ela não existir. Isso serve pra métodos que precisam saber se uma chave está na tabela sem alterá-la como fazemos em findOrCreateEntry.

No método, começamos procurando o bucket, se ele for null ou vazio, já podemos desistir da busca e simplesmente retornar null. Se o bucket não for vazio, navegamos nele pra tentar encontrar a chave e retornar o par se ele for encontrado.

Temos então os pedacos necessários pra construir os métodos containsKey, get e remove na tabela. containsKey é o mais simples, só precisamos saber se já existe um objeto Entry pra chave procurada, então é simplesmente saber se o Entry que findEntry retornou é null ou não.

Já em get temos um pouco mais de lógica porque precisamos saber se o Entry retornado é diferente de null pra poder chamar getValue, senão simplesmente retornamos null. Um efeito colateral do retorno desse método é que é impossível saber se a chave existe ou não usando somente get já que é possível ter uma chave com null como valor, então você teria que usar containsKey e get pra ter certeza que o valor não existe de verdade na tabela.

remove é o caso mais complexo dos 3 métodos que implementamos aqui. Primeiro buscamos pelo bucket onde a chave estaria, se ele existir e não for vazio, usamos um Iterator pra encontrar exatamente onde a chave está e removemos ela, retornando o valor que estava atualmente na chave. Poderíamos usar findEntry e então findBucket pra descobrir onde está o objeto específico e usar remove no próprio bucket se ele existisse, mas isso aumenta as operações que precisamos executar. A implementação usando Iterator é mais rápida (mesmo que seja quase impossível de medir a diferença).

Implementando uma hashtable que cresce dinamicamente

Temos então uma hashtable implementada de forma simples usando um array de arrays de pares chave/valor. Dá pra parar por aqui, mas a nossa tabela tem um problema, ela tem um tamanho fixo e o ideal seria que ela crescesse conforme novos objetos fossem adicionados a ela pra que não tenhamos um excesso de colisões com buckets muito grandes.

Uma das formas de definir isso é usar uma razão entre a quandidade de objetos na tabela e a quantidade de buckets. Aqui temos que considerar mais uma vez o espaco em memória versus a quantidade de operacões que queremos fazer ao chamar métodos na nossa tabela. Por padrão, o HashMap no Java usa 75% como valor base pra essa razão, o que quer dizer que no geral a quantidade de buckets é 75% do total de objetos. Ou seja, se a sua tabela tem 100 objetos, ela teria pelo menos 75 buckets.

Então, sempre que houver adição de chaves na tabela, devemos verificar se é necessário aumentá-la. Nosso código precisa de algumas adicões:

Um contador pra quantidade de chaves que temos na tabela;
Aumentar ou diminuir esse contador sempre que uma alteracão acontecer;
Método pra reconstruir toda a tabela quando a razão entre a quantidade de buckets e chaves na tabela ultrapassar um valor definido;

Vamos assumir o mesmo valor do Java, 75%, como a quantidade de buckets que vamos ter pra quantidade de chaves na tabela, sempre que ultrapassarmos esse valor, a tabela vai ser recriada com o dobro do tamanho atual. Vejamos como seria a implementacão dessas mudancas no código:

package tech.hipsters;

import java.util.ArrayList;
import java.util.List;

public class Hashtable<K, V> {

    private static final float loadFactor = 0.75F;

    private List<List<Entry<K, V>>> buckets;
    private int count;

    public Hashtable(int s) {
        this.buckets = new ArrayList<>(s);
        fill(this.buckets, s);
    }

    public V put(K key, V value) {
        throwIfNull(key);

        var entry = this.findOrCreateEntry(key);
        var oldValue = entry.getValue();
        entry.setValue(value);

        this.rehashIfNeeded();

        return oldValue;
    }

    public V remove(K key) {
        throwIfNull(key);

        var bucket = this.findBucket(key);
        if (bucket == null || bucket.isEmpty()) {
            return null;
        }

        for (var it = bucket.iterator(); it.hasNext();) {
            var next = it.next();
            if (key.equals(next.getKey())) {
                it.remove();
                this.count--;
                return next.getValue();
            }
        }

        return null;
    }

    private Entry<K, V> findOrCreateEntry(K key) {
        var bucketIndex = this.bucketIndexFor(key);
        var bucket = this.buckets.get(bucketIndex);
        if (bucket == null) {
            bucket = new ArrayList<>();
            this.buckets.set(bucketIndex, bucket);
        }

        for (var entry : bucket) {
            if (key.equals(entry.getKey())) {
                return entry;
            }
        }

        var entry = new Entry<K,V>(key);
        bucket.add(entry);
        this.count++;

        return entry;
    }

    private void rehashIfNeeded() {
        if (this.count > 0 && ((this.buckets.size() / (float) this.count) < loadFactor)) {
            var oldBuckets = this.buckets;
            this.count = 0;
            var capacity = oldBuckets.size() * 2;
            this.buckets = new ArrayList<>(capacity);
            fill(this.buckets, capacity);

            for (var bucket : oldBuckets) {
                if (bucket != null) {
                    for (var entry : bucket) {
                        this.put(entry.getKey(), entry.getValue());
                    }
                }
            }
        }
    }

    static <E> void fill(List<E> items, int count) {
        for (int x = 0; x < count; x++) {
            items.add(null);
        }
    }

}

rehashIfNeeded é a mudança principal aqui. O método começa verificando se a razão entre o número de objetos e o número de buckets passa de 0.75, que acontece quando temos 22 objetos adicionados (16 dividido por 22 é 0.72...). Copiamos os buckets atuais pra uma variável temporária, zeramos o contador e criamos uma nova coleção de buckets, com o dobro do tamanho atual, preenchemos ela com nulls, depois adicionamos todos os itens que existiam na coleção anterior nela, um a um.

Fazer o rehash de uma hashtable é uma operação cara, então se você vai trabalhar com tabelas grandes, o ideal é criar elas já com o tamanho que você vai utilizar pra evitar várias operações de rehash conforme os objetos são adicionados na tabela.

E cá estamos com uma tabela de hash funcional, que cresce conforme adicionamos objetos nela!

Access denied instead of Not Found on S3 if you don't have listing rights

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 19 Sep 2015 00:00:00 UTC

The AWS SDK for Ruby (at least the V1) has a really nice interface of interacting with objects. While building a new app using it, I had a use case where I would try to find a file on S3 and if the file didn’t exist the app would follow a different path. Not finding the file wasn’t a bug, it was one of the expected cases for the app and the code would look like this:

s3 = AWS::S3.new
bucket = s3.buckets["some-bucket-name"]
object = s3.objects["some-object-path"]

begin
  contents = object.read
  # do something with the contents
rescue AWS::S3::Errors::NoSuchKey
  # do something else here
end

To do this, I had the following policy for the bucket that this app was using (this is part of a CloudFormation template where an IAM policy is declared):

{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:GetBucketLocation",
    "s3:GetObject",
    "s3:DeleteObject"
  ],
  "Resource": [
    {
      "Fn::Join": [
        "",
        [
          "arn:aws:s3:::",
          {
            "Ref": "S3Bucket"
          },
          "/*"
        ]
      ]
    }
  ]
}

These are the only operations the app actually needed, so I continued to write it until I started doing the tests. When the object wasn’t found, instead of me receiving a NoSuchKey error, I would get an AWS::S3::AccessDenied. Given the policy above, it would seem that the current user had enough rights to issue GetObject requests to the bucket.

After a bunch of head scratching and searching, I thought maybe I should try to give it ListBucket access as well and the policy became:

{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:ListBucket",
    "s3:GetBucketLocation",
    "s3:GetObject",
    "s3:DeleteObject"
  ],
  "Resource": [
    {
      "Fn::Join": [
        "",
        [
          "arn:aws:s3:::",
          {
            "Ref": "S3Bucket"
          },
          "/*"
        ]
      ]
    }
  ]
}

And when you think about it for a bit it kind of makes sense. If you actually return a 404 when an item is not found for someone that has no listing rights it’s as if they actually did have listing rights as they can try and figure out if objects exist or not. If they don’t know what kind of access level they have, an access denied could mean both that they tried to access something that wasn’t there or that they didn’t actually have access.

Kind of weird but at least I won’t be scratching my head again because of this.

Na terra do Tio Sam - Meu primeiro aluguel

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 05 Sep 2015 00:00:00 UTC

Pois é, como todo bom nordestino, chegou a minha hora de fazer as malas e virar retirante também e acabou sendo exatamente pra terra do Mickey (apesar de eu não ter me mudado pra Orlando, pelo menos ainda!) e resolvi que vou documentar as coisas, estranhas ou não, que estou passando por aqui.

Como a empresa pra quem eu trabalho fica na Philadelphia, achei melhor inicialmente pegar um lugar pertinho do escritório, pra que eu não tenha que pegar transporte ou ter carro (andar de carro na cidade é caro, muito caro, mas isso fica pra um post futuro) então passei essas últimas semanas procurando por lugares pra alugar no Zillow, Craigslist e caçando no Google.

Quando cheguei, na segunda feira, 31 de agosto, já havia enviado um email pra um corretor pra visitar um apartamento que estava interessado (o mesmo que acabei alugando) e ele marcou a visita pra ver o lugar na terça, 1 de setembro.

Conhecendo a concorrência

Chego no horário e pra minha surpresa o corretor já estava com outras duas pessoas que também iriam visitar o apartamento. Eu estava então conhecendo os meus concorrentes, dois japoneses, aparentemente estudantes e só um deles falava inglês, então tudo o que era dito era traduzido pro outro, que parecia ser quem realmente tomava as decisões.

Entramos então no prédio todos juntos e o corretor foi mostrando pra nós o apartamento, achei bizarro, mas como estava realmente interessado nesse lugar segui na linha. Mais tarde o corretor explicou que isso é normal (!!!) e as vezes as pessoas até dão um agrado pro dono do imóvel pra facilitar o aluguel. Parece que o jeitinho é uma coisa que não tem nacionalidade.

Uma coisa que no geral é diferente do Brasil é que praticamente todos os apartamentos já vem com a cozinha pronta. Todos os apartamentos/casas tem geladeira, fogão, forno, pia com processador de alimentos e lava-louças, todos embutidos. Nunca vi um anuncio onde isso não estivesse incluso, diferente do que eu via em João Pessoa onde isso no geral é de responsabilidade do locatário.

E as banheiras. Todos os banheiros tem banheiras pro chuveiro. Um dia ainda vou perguntar se americanos realmente usam essas banheiras como banheiras (eu suspeito que não, mas nunca se sabe né) e se é um resto cultural que ficou por aí atrapalhando os banhos alheios. As bordas arredondadas da banheira são tiro e queda pra você escorregar e meter a bunda no chão (não, não fui vítima disso, ainda).

Mas voltando a visita, vemos o apartamento juntos e na saída o corretor explica que se estivermos interessados podemos aplicar pro aluguel. Pra aplicar você dá um mês de aluguel mais taxas necessárias pro corretor e espera o dono avaliar se vai alugar pra você ou não (no meu caso eram 30 dólares pra um credit check).

Enquanto isso o seu dinheiro fica preso na corretora (no meu caso, se eu já tivesse conta bancária poderia ter sido um cheque, sim, as pessoas usam cheques aqui) até a decisão final do dono. Não sei se é um caso específico do imóvel que eu estou, mas o fato da imobiliária não ser quem toma a decisão de alugar ou não é estranho. A preocupação do dono deveria ser receber o dinheiro, não saber quem é que está ou não no imóvel.

Na hora que o corretor explicou isso ficamos todos com aquela cara de e agora José? Será que eu já aplico agora mesmo ou espero pra fazer isso amanhã sem os concorrentes? Liguei o foda-se e disse pro corretor que queria já aplicar agora. Os concorrentes acordaram pra jesus e disseram que iam aplicar agora também. Fomos todos naquele silêncio de cemitério até a imobiliária e começamos a preencher os vários papéis.

Credit e background check

Na hora de preencher os documentos o corretor pergunta como eu vou pagar a aplicação pro aluguel e eu explico a ele que só posso fazer isso em dinheiro porque acabei de chegar no país e não tenho conta em banco e nessa hora a coisa realmente fica interessante. Como eu já havia dito, havia uma taxa de 30 dólares pra uma avaliação do meu crédito (imagine um relatório do SERASA) mas eu ainda não existo oficialmente no país (você precisa de um documento chamado Social Security Number pra entrar no sistema de crédito do país) então qualquer avaliação da minha situação daria desconhecido.

Chegamos então a mais um detalhe do processo. Quando você fecha contratos de aluguel, você paga adiantados o primeiro e o último mês, dependendo do seu histórico de crédito você também pode precisar deixar um mês de depósito de segurança. Como eu não tinha histórico nenhum, já dá pra imaginar o que aconteceu, né?

Fiz a minha aplicação ainda naquele dia e os meus concorrentes acabaram não fazendo porque não estavam nem com dinheiro nem com cheques. Eles terminaram vindo no dia seguinte com cheques pra aplicar mas parece que o dono me achou mais legal e resolveu ficar comigo.

Só que tinha um porém.

Como eu não tinha histórico nenhum, teria que pagar 3 meses adiantados e o depósito de segurança. Ou seja, 4 meses de aluguel numa paulada só. Considerando que eu não tinha lá muito espaço pra negociar, o jeito foi se agarrar no cacto e esperar não pegar em muitos espinhos. Pra completar, a imobiliária não aceitava dinheiro (né?) e tive que ir nos correios trocar o dinheiro por ordens de pagamento (né? 2) pra poder fazer o pagamento.

Com o contrato fechado, fui pra empresa de energia (a PECO) pra pedir a ligação. Comecei o processo pelo site até que aparece o campo pra preencher o bendito Social Security Number (todo mundo quer saber o seu SSN aqui, incrível), quem não tem precisa ir a m dos escritórios da empresa pra pedir a ligação. Felizmente o escritório é a alguns quarteirões do trabalho e consegui ser atendido rapidinho, me deram um dia de prazo pra ligar a energia (isso foi ontem) e hoje já cheguei no apartamento e encontrei tudo ligado.

Aí perguntei ao corretor como é que eu vou fazer os pagamentos, imaginando que eles me mandariam um boleto ou coisa parecida pra pagar e ele responde você manda um cheque pelos correios. Nessa hora eu não pude segurar o riso. Como assim? As pessoas aqui mandam cheques pelos correios?

Pensei comigo, mas que coisa atrasada, isso nunca daria certo no Brasil. Mas aí pensei de novo e tive que parar de rir. Se eles podem mandar cheques pelos correios, é porque ainda se confia no serviço dos correios e não se assume que existem ladrões por todos os lugares que passamos. Nosso sistema bancário realmente é muito avançado, mas no fim das coisas isso não é exatamente um sinal de que as coisas estão bem, mas o contrário, de que fraudes e crimes são tão comuns que tivemos que avançar na marra.

E assim terminou a minha primeira saga, pra arranjar um lugar pra morar. Vamos ver como é que vão ser as cenas dos próximos capítulos.

Respostas

E já chegaram algumas explicações sobre os por quês aqui, vejam só:

@mauriciojr banheiras são usadas pra dar banho nos filhos. Banco automatiza o envio do cheque pra você. SSN é como CPF. Histórico é tudo.
— Bruno Miranda (@brupm) September 6, 2015

Teve da Gisete também:

Muito legal! Depois de muito tempo aqui é interessante ver essas experiencias pelos olhos de outra pessoa, o começo é sempre exciting. :D BTW, EU conheco gente que só toma banho de banheira e nunca chuveiro. É util pra quem tem criança tb.

E do Kung:

a banheira é util para lavar coisas tb (reparou que não tem um tanque?). E de pagar com cheque, relaxa. Quando abrir a conta no banco vai poder automatizar online. Ai eles põem o cheque no correio para vc todo mês smile emoticon hahaha (e não, não é piada)

Building a simple long polling HTTP server with Scala and Netty

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 23 Aug 2015 00:00:00 UTC

You can find the whole source code for this example here.

While messaging solutions are abound, both over HTTP like web sockets and server sent events and using other protocols, sometimes you’re stuck with an HTTP client that can’t really do any of this funky stuff. Firewalls, proxies and many other hurdles along the way between clients and servers will force you to stick with the basic HTTP request-response cycle, but you still need to provide a way for clients to listen to events, this is where long polling comes into play.

The idea of long polling is that the client will perform a request, the server will receive it and leave it hanging for some time until an event arrives or the server decides it was a timeout (last time I checked Dropbox was working with a 20 seconds timeout). Doing something like this means the only thing your HTTP clients have to do is configure a really long timeout for the HTTP response, so they can just pretend this is a common HTTP connection. Proxies and firewalls will also just think this is a slow connection and will leave it be most of the time.

With the dead simple client, all the complication now needs to lie at the server.

Netty comes to the rescue

To build an HTTP server that will suspend requests you’d either need something like the Servlet 3.0 API or just use a low level networking framework like Netty, which is what we will use here. Our solution is going to be quite simple, clients perform a GET request and are suspended, when a POST request arrives for that same path, all suspended clients are notified with its contents as their HTTP response.

The first step here is to build our client’s registry, where all clients will be registered when they make a request, let’s start with its skeleton:

class ClientsRegistry(timeoutInSeconds: Int) {

  private val lock = new ReentrantLock()
  private val pathsToClients = scala.collection.mutable.Map[String, ListBuffer[ClientKey]]()
  private val orderedClients = scala.collection.mutable.TreeSet[ClientKey]()

  private def withLock[R](fn: => R)(implicit executor: ExecutionContext): Future[R] = {
    val p = Promise[R]

    executor.execute(new Runnable {
      override def run(): Unit = {
        lock.lock()
        try {
          p.success(fn)
        } catch {
          case e: Throwable => p.failure(e)
        } finally {
          lock.unlock()
        }
      }
    })

    p.future
  }

}

Our registry contains a lock, two collections and a method to execute a chunk of code, holding the lock, inside an execution context (like a thread pool). The lock exists because this class will be used concurrently by our server, so all access to it has to be thread safe. We could have used concurrent collections here, but it would make the implementation a bit more complicated, so we’ll stick with the common ones and use the lock.

As we have a lock, we can’t force clients to lock their threads so all operations will happen inside a provided execution context, in background, instead of forcing clients of this code to lock their threads (locking the Netty IO thread is a bad idea, so we avoid doing it at all costs). The method also wraps the function given and turns it into a future, so clients can either compose or wait on the future for its result.

But why do we need two collections here instead of just one?

Because of the timeouts problem. If we only had the map that maps paths to collections of clients, how could we figure out which clients need to be timeouted? We’d have to navigate through each one of them, check and if you happen to have lots of clients this is a pretty bad idea.

So we introduce a new collection, a sorted set that will return our clients in ascending order so we can quickly figure out which clients are going to timeout without walking through the whole collection.

Let’s look at our ClientKey implementation:

case class ClientKey(path: String, expiration: Date, ctx : ChannelHandlerContext)
  extends Comparable[ClientKey] {

  override def compareTo(o: ClientKey): Int =
    expiration.compareTo(o.expiration)

  def isExpired : Boolean = new Date().after(expiration)

}

Our implementation is quite simple, it just holds the path, the expiration date and the context for this client (so we can write to it) and it implements Comparable using the expiration as the sorting value. This means all objects inside our orderedClients collection will be sorted by expiration date in ascending order. We’ll see how this affects our timeout implementation soon.

Registering clients

Our fist operation is to receive a client and register it under an specific path, let’s look at it:

def calculateTimeout(): Date = {
  val calendar = Calendar.getInstance
  calendar.add(Calendar.SECOND, timeoutInSeconds)

  calendar.getTime
}

def registerClient(path: String, ctx: ChannelHandlerContext)(implicit executor: ExecutionContext): Future[ClientKey] =
  withLock {
    val client = ClientKey(path, calculateTimeout(), ctx)

    val clients = pathsToClients.getOrElseUpdate(path, ListBuffer[ClientKey]())
    clients += client
    orderedClients += client

    client
  }

When registering clients, we first create a new ClientKey object calculating its timeout (we use the value provided by the registry’s constructor) and append it to both the map and set inside our class. Since the withLock method requires an ExecutionContext our registerClient method will also need one to be provided.

Completing clients

The second operation is to complete a path, it means a POST request came and now all clients for that specific path need to be removed from the registry so we can send them the response. Here’s the implementation:

def complete(path: String)(implicit executor: ExecutionContext): Future[Iterable[ClientKey]] =
  withLock {
    pathsToClients.remove(path).map {
      clients =>
        orderedClients --= clients
        clients
    }.getOrElse(Iterable.empty)
  }

The implementation here is dead simple, first find all clients for an specific path, remove all clients from the timeouts collection (orderedClients) and then return the clients found. If nothing is found just return an empty collection.

Again, as we’re using withLock an execution context must be provided.

Timeouting clients

The last important operation we need is the timeout, let’s look at the code:

def collectTimeouts()(implicit executor: ExecutionContext): Future[Iterable[ClientKey]] = {
  withLock {
    val iterator = orderedClients.iterator
    val timeouts = ListBuffer[ClientKey]()

    var done = false

    while (iterator.hasNext && !done) {
      val next = iterator.next()
      if (next.isExpired) {
        timeouts += next
      } else {
        done = true
      }
    }

    orderedClients --= timeouts

    timeouts.foreach {
      timeout =>
        pathsToClients.get(timeout.path).foreach(b => b -= timeout)
    }

    timeouts
  }
}

You might be looking at that explicit iterator and thinking OMG, why do we need this? Think about it. We can’t be wasting CPU cycles evaluating all clients to verify if they have timeouted or not and our orderedClients set is sorted by expiration in ascending order.

What does that mean?

That as soon as I find an item that is not expired, it means there are no more expired items to be removed. So only one comparison is wasted here, the one that produces the first item that isn’t expired. As long as I’m seeing items that are expired I’m good because I do want to remove these items, once I find the first item that is not expired I break the loop and move on.

So this is why we use a TreeSet here, so we can quickly find the clients to be timeouted in as little comparisons as possible. Later we also remove them from their respective paths (this is why we also save the path at clients) and at the end return them to the caller.

Our client registry is now done, you can find the specs for the registry implememtation here.

Building our Netty handler

In Netty most of your work happens inside of channel handlers, in our case, as a server, we’re going to build a ChannelInboundHandler. As we’re building an HTTP server, we can just assume someone will do all the parsing for us and we’ll receive a FullHttpRequest. Netty also comes with many base classes available so we don’t really need to implement the full ChannelInboundHandler interface, we can inherit from SimpleChannelInboundHandler and implement the channelRead0 method to be done with it, let’s look at our MainHandler implementation:

@Sharable
class MainHandler( registry : ClientsRegistry )(implicit executor: ExecutionContext)
  extends SimpleChannelInboundHandler[FullHttpRequest] {

  import MainHandler.log

  override def channelRead0(ctx: ChannelHandlerContext, msg: FullHttpRequest): Unit = {

    msg.getMethod match {
      case HttpMethod.GET => {
        registry.registerClient(msg.getUri, ctx).onFailure {
          case e => writeError(ctx, e)
        }
      }
      case HttpMethod.POST => {
        ReferenceCountUtil.retain(msg)
        registry.complete(msg.getUri).onComplete {
          result =>
            try {
              result match {
                case Success(clients) => {
                  clients.foreach {
                    client =>
                      client.ctx.writeAndFlush(buildResponse(msg))
                  }
                  ctx.writeAndFlush(new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK))
                }
                case Failure(e) =>
                  writeError(ctx, e)
              }
            } finally {
              ReferenceCountUtil.release(msg)
            }
        }
      }
      case _ =>
        ctx.writeAndFlush(new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.NOT_FOUND))
    }
  }

}

The very first thing here is the @Sharable annotation for our handler, this means it can be used by many different channels at the same time and its really important for you to correctly mark your handlers as Netty will use this information to decide if it can reuse your handler in many different channels and threads.

Just like the client’s registry takes an execution context for its methods, our handler takes one when it’s being created so it can provide it to the registry methods. You can then provide any thread pool you’d like.

Now let’s look at the first case:

case HttpMethod.GET => {
  registry.registerClient(msg.getUri, ctx).onFailure {
    case e => writeError(ctx, e)
  }
}

Since we’re going really low level here, there’s no such thing as a router for our requests, we need to either build one or just match the routes as we’d like to. In this case it’s quite simple, we just use any GET or POST requests and move on. The GET implementation also happens to be the simplest, we just register the client that made it or make sure we provide an error message if we can’t.

This sends us to the meat of the handler, the POST operation:

case HttpMethod.POST => {
  ReferenceCountUtil.retain(msg)
  registry.complete(msg.getUri).onComplete {
    result =>
      try {
        result match {
          case Success(clients) => {
            clients.foreach {
              client =>
                client.ctx.writeAndFlush(buildResponse(msg))
            }
            ctx.writeAndFlush(new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK))
          }
          case Failure(e) =>
            writeError(ctx, e)
        }
      } finally {
        ReferenceCountUtil.release(msg)
      }
  }
}

There’s a lot of stuff going on. First, we have the use of ReferenceCountUtil. In Netty some objects are reference counted inside the framework itself, this is done so they can be reused across different requests, diminishing the pressure at the JVM’s garbage collector. And while this improves throughput it requires us, the developers, to do the bookkeeping.

Most of the time you can ignore these objects are reference counted, but in our case the actual implementation of the handler (the one that writes the response to clients) happens in a different thread than the one that is being used by Netty (the complete method doesn’t return right away, it produces a Future[Iterable[ClientKey]]), if we didn’t call retain here Netty would assume the request was consumed by our code and would gladly reuse it, which would break our server or worse, produce bogus responses.

So, as we need to hold onto this request for a while, we call retain on it and only call release at the finally block of the callback we registered at the future that was produced by the complete call. When we actually enter the callback implementation we can see it’s quite simple, if it finds a list of clients for the given path, it sends the results for each one of them and sends response to the POST client, if it fails the POST client receives an error message.

The interaction between these two HTTP methods (GET and POST) is what is actually building our long polling client. When a GET arrives we don’t answer right away, we register the client and wait for someone to provide a POST, once it arrives we find all registered clients, remove them from the registry and send the response to them. Dead simple.

Another important piece for this handler is the timeouts collector:

def evaluateTimeouts(): Unit = {
  registry.collectTimeouts().onSuccess {
    case clients => clients.foreach {
      client =>
        writeError(client.ctx, new TimeoutException("channel timeouted without a response"))
    }
  }
}

Simple as well, just collect the expired clients and write a timeout error to them.

The other helper methods aren’t that much special:

def writeError(ctx : ChannelHandlerContext, e : Throwable): Unit = {
  val response = new DefaultFullHttpResponse(
    HttpVersion.HTTP_1_1,
    HttpResponseStatus.INTERNAL_SERVER_ERROR,
    Unpooled.wrappedBuffer(e.getMessage.getBytes(CharsetUtil.UTF_8))
  )

  response.headers().add(HttpHeaders.Names.CONTENT_TYPE, "text/plain")

  ctx.writeAndFlush(response)
}

def buildResponse( request : FullHttpRequest ) : FullHttpResponse = {
  val response = new DefaultFullHttpResponse(
    HttpVersion.HTTP_1_1,
    HttpResponseStatus.OK,
    Unpooled.copiedBuffer(request.content())
  )

  if ( request.headers().contains(HttpHeaders.Names.CONTENT_TYPE) ) {
    response.headers().add(HttpHeaders.Names.CONTENT_TYPE, request.headers().get(HttpHeaders.Names.CONTENT_TYPE))
  }

  response
}

One writes an exception as an HTTP server error and the other gets the POST request and creates a new response based on it so we can send it to our registered clients.

You can find tests for the handler here. Our server is almost done!

HTTP housekeeping

As we’re using a low level HTTP framework for our server, it won’t really do anything we don’t ask it to. Stuff like defining the right value for the Connection header, setting Content-Length for responses and using the same HTTP version as the client needs to be done by us. Fortunately, Netty offers a nice way for us to plug a handler that will do this without polluting our main handler, let’s look at it:

object SetHeadersHandler {

  val DefaultServerName = "long-polling-server-example"
  val ConnectionAttribute =
    AttributeKey.valueOf[String](s"${SetHeadersHandler.getClass.getName}.connection")
  val HttpVersionAttribute =
    AttributeKey.valueOf[HttpVersion](s"${SetHeadersHandler.getClass.getName}.version")

}

@Sharable
class SetHeadersHandler extends ChannelDuplexHandler {

  import SetHeadersHandler._

  override def channelRead(ctx: ChannelHandlerContext, msg: scala.Any): Unit = {
    msg match {
      case request: FullHttpRequest => {
        val connection = if (HttpHeaders.isKeepAlive(request))
          HttpHeaders.Values.KEEP_ALIVE
        else
          HttpHeaders.Values.CLOSE
        ctx.channel().attr(ConnectionAttribute).set(connection)
        ctx.channel().attr(HttpVersionAttribute).set(request.getProtocolVersion)
      }
      case _ =>
    }

    super.channelRead(ctx, msg)
  }

  override def write(ctx: ChannelHandlerContext, msg: scala.Any, promise: ChannelPromise): Unit = {

    msg match {
      case response: FullHttpResponse => {
        response.setProtocolVersion(ctx.channel().attr(HttpVersionAttribute).get())
        response.headers().set(HttpHeaders.Names.SERVER, DefaultServerName)
        response.headers().set(HttpHeaders.Names.CONNECTION, ctx.channel().attr(ConnectionAttribute).get())
        response.headers().set(HttpHeaders.Names.CONTENT_LENGTH, response.content().readableBytes())
      }
      case _ =>
    }

    super.write(ctx, msg, promise)
  }
}

A ChannelDuplexHandler is a handler for both inbound and outbound messages, so HTTP request and response objects. Our handler is again marked as @Sharable as it can be reused by many different channels and threads at the same time, to accomplish that we use the channel attributes collection to store the Connection and HTTP version values for the request so we can set them up at the response. This guarantees that any request that comes in will correctly fill in the necessary attributes and we will be able to use these values when building the response.

At the write method we basically pull the fields that were set by channelRead, include a Server and calculate the Content-Length (if you don’t set a content length clients will have trouble figuring out that your response is over) and then wrap it up calling the superclass implementation. This guarantees our HTTP responses are sane and match what the client expected to receive back and we didn’t have to pollute our main handler with any of this. This is one of the main beauties of the Netty API, you can just compose new functionalities on top of your networking code by introducing more handlers along the way.

Check out the tests for the set headers handler.

Starting the server

And as the final step we need an initializer that will start our HTTP server:

class Initializer (timeoutInSeconds : Int, val port: Int) (implicit executor: ExecutionContext)
  extends ChannelInitializer[SocketChannel] {

  import Initializer.log

  private val bossGroup = new NioEventLoopGroup(1)
  private val workerGroup = new NioEventLoopGroup()

  private val serverBootstrap = new ServerBootstrap()
  serverBootstrap.option(ChannelOption.SO_BACKLOG, java.lang.Integer.valueOf(1024))
  serverBootstrap.group(bossGroup, workerGroup)
    .channel(classOf[NioServerSocketChannel])
    .childHandler(this)

  private var serverChannel: Channel = null
  private val setHeadersHandler = new SetHeadersHandler
  private val mainHandler = new MainHandler(new ClientsRegistry(timeoutInSeconds))

  override def initChannel(ch: SocketChannel): Unit = {
    val p = ch.pipeline()

    p.addLast("http-codec", new HttpServerCodec())
    p.addLast("aggregator", new HttpObjectAggregator(Int.MaxValue))
    p.addLast("set-headers-handler", setHeadersHandler)
    p.addLast("handler", mainHandler)
  }

  def start(): Unit = {
    try {
      serverChannel = serverBootstrap.bind(port).sync().channel()
      serverChannel.eventLoop().scheduleAtFixedRate(new Runnable {
        override def run(): Unit =
          mainHandler.evaluateTimeouts()
      },
        timeoutInSeconds,
        timeoutInSeconds,
        TimeUnit.SECONDS
      )

      log.info(s"Starting server ${serverChannel}")
      serverChannel.closeFuture().sync()
    } catch {
      case e: Exception =>
        log.error(s"Server channel failed with ${e.getMessage}", e)
    }
    finally {
      bossGroup.shutdownGracefully()
      workerGroup.shutdownGracefully()
    }
  }

  def stop(): ChannelFuture = {
    log.info(s"Stopping server ${serverChannel}")
    val channelFuture = serverChannel.close().awaitUninterruptibly()
    log.info(s"Closed server channel ${serverChannel}")
    channelFuture
  }

}

The initializer itself is the piece that glues all Netty pieces together. The server backlog (the listener), the event loops and the channel pipeline. Here we setup our server and the pieces it is made of, the most important method is initChannel:

override def initChannel(ch: SocketChannel): Unit = {
  val p = ch.pipeline()

  p.addLast("http-codec", new HttpServerCodec())
  p.addLast("aggregator", new HttpObjectAggregator(Int.MaxValue))
  p.addLast("set-headers-handler", setHeadersHandler)
  p.addLast("handler", mainHandler)
}

This is where we introduce the many pieces that make our pipeline and ordering here is very important. If we change the order of any of the pieces here, we could be breaking our server. Our pipeline starts with an HttpServerCodec, that parses HTTP requests and produces HTTP responses. It then has an HttpObjectAggregator, this aggregator turns the various HTTP messages that Netty produces into the FullHttpRequest we have been working with. Without this you would have to manually handle the many HTTP messages you would receive instead.

Then we start to reach application code with the SetHeadersHandler and finally our MainHandler at the end of the pipeline. This initChannel method is called whenever a new channel is created, the Netty provided HTTP codecs can’t be reused so we need to have one of them for every channel available but our own handlers are all @Sharable so we can just reuse them instead of creating new ones.

The other important piece here is the scheduler:

serverChannel.eventLoop().scheduleAtFixedRate(new Runnable {
  override def run(): Unit =
    mainHandler.evaluateTimeouts()
},
  timeoutInSeconds,
  timeoutInSeconds,
  TimeUnit.SECONDS
)

At our MainHandler earlier we saw the evaluateTimeouts method but we didn’t see it being called, this is because it was supposed to be called here at the initializer using a scheduler. Every Netty event loop is capable of scheduling events and you should just use them here to evaluate the timeouts. In here we’re just using the actual timeout value to perform the evaluations but you could also use a fraction of it to detect timeouts faster.

You can find an initializer test here.

It is alive!

And now you have a basic and functional long polling HTTP server that you can extend to your own needs!

AWS quick tips - taking out your EC2 trash

mauricio.linhares@gmail.com (Maurício Linhares) — Fri, 07 Aug 2015 00:00:00 UTC

If you use EC2 to run your cloud infrastructure, you are most likely making use of pre-made AMIs for faster provisioning of servers inside of auto-scaling groups. Just like any other release material, from time to time you have to cleanup your older stuff both because the cruft accumulates and also because you’re paying for the storage these AMIs take as snapshots at your EC2 account.

Finding unused AMIs

The first cleanup operation is detecting AMIs that are not being used anywhere at your environment, we can do this quickly with a very small Ruby script that uses the aws-sdk-v1 gem :

require 'aws/ec2'
client = AWS::EC2.new # you must have your AWS keys setup correctly for this to work
amis = client.instances.map do |instance|
  begin
    instance.image_id
  rescue AWS::Core::Resource::NotFound
    nil
  end
end.compact.uniq

puts("# Loading available AMIs...")
images = client.images.with_owner("self").sort_by { |i| i.name }
images.each do |image|
  next if amis.include?(image.id)
  fields = []
  fields << image.id
  fields << image.block_device_mappings.values.map(&:snapshot_id).compact.join(',')
  fields << image.name

  puts(fields.join("\t"))
end

This will produce an output like:

# Loading available AMIs...
ami-15903pee	snap-945febdz	java8@2015-08-03-02-50
ami-841928e0	snap-8117t03b	ubuntu_14_base@2015-04-02-21-35
ami-1dqde176	snap-33fh5a54	windows-server2012R2-base-2015-08-07

What this script does is first list all AMIs from running instances in your system and then load all AMIs that you own (AMIs you created yourself using the CreateImage command). We don’t really want to know about public or vendor specific AMIs, only the ones we created ourselves.

Now that you know the AMIs that are not in use, you can just open the AWS console and deregister them, right?

Not really!

Deregistering AMIs

While the AWS console for AMIs will let you deregister AMIs, it does the wrong thing by default, it deregisters the AMI but will not delete the snapshot associated with it. I was really surprised when I looked at our snapshots and we had hundreds of snapshots there from AMIs that were long gone!

So, instead of just using the AWS console for that, we need a script that will actually delete the snapshots associated with them as it makes little sense to deregister the AMI but keep it’s snapshot around.

Here’s how it would look like:

require 'aws/ec2'
ami_id = ARGV[0]
if ami_id.nil? || ami_id.empty?
  puts("you must provide an AMI ID")
  exit(1)
else
  image = client.images[ami_id]
  snapshots = image.block_device_mappings.values.map(&:snapshot_id).compact
  puts "Deregistering AMI [#{image.name}] with snapshots #{snapshots.inspect}"
  image.deregister
  puts "Image deregistered"
  snapshots.each do |snapshot|
    puts "Deleting snapshot #{snapshot}"
    client.snapshots[snapshot].delete
    puts 'Snapshot deleted'
  end
end

And this should finally clean up the huge amount of useless AMIs and snapshots you have at your account. It definitely helped me clean up a lot of storage at our account by removing unused stuff.

If you’re using Chef and Knife

If you happen to be using Chef with your own set of knife plugins, these scripts are actually knife plugins themselves, so you can just copy them to your own repo:

# ami unused
require 'chef/knife'

module KnifeCustom
  class AmiUnused < Chef::Knife

    banner "knife ami unused"

    deps do
      require 'aws/ec2'
    end

    option :progress_comments,
      :long => '--[no-]progress-comments',
      :description => "Include comments about progress, enabled by default",
      :boolean => true,
      :default => true

    option :include_in_use,
      :short => '-U',
      :long => '--include-in-use',
      :description => "Include information about AMIs in-use as well, disabled by default",
      :boolean => true,
      :default => false

    option :snapshot_info,
      :long => '--[no-]snapshot-info',
      :description => "List EBS snapshot information for AMIs, enabled by default",
      :boolean => true,
      :default => true

    option :ami_status,
      :long => '--[no-]ami-status',
      :description => "Show AMI status [in-use|unused], enabled by default",
      :boolean => true,
      :default => true

    def run
      ui.info("# Loading running instances...") if config[:progress_comments]
      amis = client.instances.map do |instance|
        begin
          instance.image_id
        rescue AWS::Core::Resource::NotFound
          nil
        end
      end.compact.uniq

      ui.info("# Loading available AMIs...") if config[:progress_comments]
      images = client.images.with_owner("self").sort_by { |i| i.name }
      images.each do |image|
        next if amis.include?(image.id) && !config[:include_in_use]
        status = amis.include?(image.id) ? "in-use" : "unused"
        fields = []
        fields << status if config[:ami_status]
        fields << image.id
        fields << snapshots_for_image(image).join(',') if config[:snapshot_info]
        fields << image.name

        ui.info(fields.join("\t"))
      end
    end

    def client
      @client ||= AWS::EC2.new
    end

    def snapshots_for_image(image)
      image.block_device_mappings.values.collect(&:snapshot_id).compact
    end
  end
end

And then the command that deregisters AMIs:

require 'chef/knife'

module KnifeCustom

  class AmiDeregister < Chef::Knife

    banner "knife ami deregister AMI_ID"

    deps do
      require 'aws/ec2'
    end

    def run
      ami_id = name_args.first
      if ami_id.nil? || ami_id.empty?
        ui.error("you must provide an AMI ID")
        exit(1)
      else
        image = client.images[ami_id]
        snapshots = image.block_device_mappings.values.map(&:snapshot_id).compact
        ui.info "Deregistering AMI [#{image.name}] with snapshots #{snapshots.inspect}"
        image.deregister
        ui.info "Image deregistered"
        snapshots.each do |snapshot|
          ui.info "Deleting snapshot #{snapshot}"
          client.snapshots[snapshot].delete
          ui.info 'Snapshot deleted'
        end
      end
    end

    def client
      @client ||= AWS::EC2.new
    end

  end

end

Now enjoy the space you now have available and the amount of money you won’t be spending on these useless snapshots!

Fighting procrastination

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 26 Jan 2015 00:00:00 UTC

This is part 3 of a three parts assignment for the Learning how to learn course on Coursera. Also check Part 1 - Your brain on thinking and learning and Part 2 - Chunking concepts to solidify and compress topics.

Procrastination is one of the biggest woes when we’re trying to learn something hard. It’s our escape valve when we don’t want to do something that looks painful or boring. As Parkinson’s law clearly defines it:

Work expands so as to fill the time available for its completion.

We suffer from procrastination until the last minute, when we cram for exams or pull all nighters to deliver that important project that was scheduled for a while. Worse, we avoid taking long term commitments, like writting a book, because it will take too long, all in the name of procrastination.

But what is procrastination?

Procrastination if the act of avoiding something that has to be done because we assume it is going to be painful. It’s a habit we acquire to protect us from the bad feeling we have when we’re about to perform a specific task, like studying for that next calculus exam.

The thing is, studying calculus isn’t going to cause you any pain. In most cases, once you get over the anticipation of doing the work, the pain just goes away. All the supposed suffering you would have by studying calculus just disappears the moment you start doing it. We procrastinate because of the anxiousness we’re feeling before the task, once you go over that hill, you’re good to go.

Simple, isn’t it? Just stop doing it and you’re good to go, right?

Not quite.

Breaking habits

Procrastination wires itself as a habit and habits are naturally hard to break. When performing a habit, your brain is in energy-saving mode, you are almost unconsciously performing it and this is a good thing, it saves energy for other important stuff your brain is doing or you might be focusing on.

Habits are built in 4 pieces:

The Cue - this is the hint that causes your brain to go into auto-pilot, it’s the sign that triggers an automatic response (for instance, thinking about your physics exam next week);
The routine - this is the action you perform in response to the cue, as in surfing the web to drive your attention out of the physics exam;
The reward - habits become hardwired in our brains because they offer a reward (as simple as it might be), when you start surfing the web, your mind drifts away from the painful feeling of studying for the physics exam and you get a little relief. Your brain will remember this and burn this response a little more into itself;
The Belief - we build a firm belief and excuses for the habits we form and the responses we have. Math is hard, I’ll never need statistics, I can just revisit the algorithms book later when I need them are all excuses and these excuses build a shell of protection around the procrastination act, they look logical, but they’re not, they’re just excuses we’re making for our own failings;

What we want to do is to apply our willpower to the routine step and change our response to the cue. The key to fighting procrastination isn’t being it our whole day, no one has enough willpower to wrestle it like that, what we want instead is to apply some of it just after we get the cue to provide a better response.

The first step is to start noticing the cues that send you into procrastination mode. Text messages on your phone, having email/facebook tabs open all the time (with their addicting notification counts), being disrupted while concentrating at work, start noticing how these small things take you out of flow state and send you to do something else. Silence your phone, close the email/facebook/twitter tabs, switch to a less disruptive environment for work or go to the most silent spot at the library to study. Removing the cues that trigger your procrastination is already halfway into effectively fight it.

Then, when you still get a cue after trying to remove them, change your response. This is the only time you have to consciously apply willpower, to change the routine you will perform when a cue comes up. A simple way of doing it is to have fixed time slots for your work/studying tasks, like doing Pomodoros.

The Pomodoro technique was developed by Francesco Cirillo and it consists of having fixed 25 minutes stints of focused work with brief breaks. During a Pomodoro, you can’t do anything other than the work that’s right in front of you, if a cue comes up, you shut it down and continue whatever you’re doing. In a couple minutes you’ll have a break and will be able to procrastinate a bit before starting the next pomodoro, so there is no need to be desperate.

Make sure you also reward yourself when you do that. Creating these small rewards causes your brain to notice there are rewards in doing the actual work as well instead of thinking that rewards only come when it is procrastinating. Doesn’t have to be something big, you could buy that new Blind Guardian album, go for a walk in the park or listen to one of your favorite podcasts.

Focus on process, not product

Many times we avoid long term commitments because we think they’re too hard, will take too long or we will never be able to do it. When I was doing my final college essay, it always felt like I would never finish it, because I couldn’t even start it!

My main problem back then was focusing on having the essay done, the product. If, instead, I had focused on the process of doing it, I wouldn’t have had to pull all nighters to finally deliver it.

How does that work?

Instead of focusing on I have to write the essay, I would think I have an hour to work on the essay every weekday. Look at how this is reframed, instead of forcing myself to think I’ll have to produce something (which would kick in the procrastination response), I just say that I’ll have this whole hour to do work on it, whatever it is. There is no need to produce something out of this hour, but I surely will because there won’t be a reason to run away from it.

This simple change of perspective removes the pain of I have to produce an essay and turns it into I have an hour to work on it. It’s like deciding to read a book every month, you´ll fall behind and give up when you see you can’t keep up with it, if instead you decide to read fifty pages of a book every day you’re much more likely to get it done. How many 1400 pages books do you have hanging around, right?

Go small to go big

The main takeaway you should have is that focusing on small, daily or every couple of days tasks is much more productive and causes much less procrastination than having large, hard to achieve, goals. There is nothing that makes you more driven than constant progress, setting up your goals in small pieces gives you this sense of getting somewhere that is really hard to have when it is about some future product that feels like it will never be finished.

As much as it is hard to fight procrastination, if you start to consciously change your responses to the cues, define clear and quickly attainable goals and give you a couple nice rewards whenever you get stuff done right, you’ll definitely be at a better position. Whenever you think your days aren’t long enought for what you do, think about what you did, which parts of your day were you doing something that wasn’t productive? Could you have used that time in a better way? Could improving your usage of that time give you some more leisure time later to enjoy your friends, family or just hanging around?

Avoiding procrastination isn’t only about being more productive at work, but also about having more time for yourself. If you can be more productive and finish your chores faster, you’ll end up with much more time to do other enjoyable activities.

Chunking concepts to solidify and compress topics

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 22 Jan 2015 00:00:00 UTC

This is part 2 of a three parts assignment for the Learning how to learn course on Coursera. Also check Part 1 - Your brain on thinking and learning and Part 3 - Fighting procrastination.

Chunking is the process of filing information in your brain, it’s how you turn the text, the classes and the practice into mental patterns that can be recalled later when you’re trying to solve a problem. As such, the act of chunking is one of the most important pieces of your learning experience and one that needs careful attention.

The ideas behind it are very old and can be found in many different cultures, each in it’s own specific way. In martial arts, you can see the same concept presented as the Shu-ha-ri, which translates to obey, digress and separate, respectively.

At the Shu step, you are repeating a movement to build the muscle and mental memory of the steps you have to perform. Let’s take the common armbar as an example:

The armbar is one of the most basic submission movements in many martial arts like Judo and Jiu-Jitsu, it’s something you’ll surely learn at your very first weeks of practice. At the beginning, you will be presented to the movement and will be told to repeat it many times. You are focusing your attention on the steps that have to be performed for the move.

You need to remember that if you catch the the right arm, your right leg will be the one lifting you. Many times, when I was learning the armbar, I would hold the right arm and lift my body using the left leg, as you might expect, it didn’t work. As I repeated the process, the focus on performing the armbar and nothing else eventually led me to perform the right movement. I could repeat it correctly (from the mount position, on top of someone) whenever I tried.

Once you reach this point, you move to the Ha step. This is when you understand what you are doing. At this point you think about why the armbar works, how your legs on top of the opponent pins him to the ground and how, if you push your body up and keep your legs down, he has to either give up or risk a broken arm.

Can you still form a chunk if you don’t understand what you’re doing?

Of course, but it will be an useless one, since you won’t be able to relate it to other pieces of your knowledge. As much as being able to perform the movement is important, knowing how it actually works is important for you to reach the next level.

You might be able to perform those statistics exercises you have in the book, but if you don’t understand what the concept means it’s unlikely you’ll be able to link it to other concepts to form a larger (and compressed) chunk of information. You must understand the topic you’re studying for it to be useful in the long term.

And here we get to the last step, the Ri. The separation or transcendence represented by Ri means building a context around this chunk that has been repeated and understood. In an armbar, this would be thinking about the variations and when you can effectively apply it. See, for instance, how Ronda Rousey, an MMA fighter famous for her many armbar submissions, presents a variation:

The movement was learned from a mounted position, but there are many other ways and positions where it can be applied. There are also many moments when you should not try to apply it as it might put you in a bad position and make you lose a battle.

Building context around chunks is needed because we must learn when they should be applied so we can just not bring it to mind when tackling a problem we know can’t be solved using that specific practice. This also helps you file the chunk with other related chunks you already have or, even better, include it inside another chunk that already exists, making it use less space in your mind.

To summarize, the steps to chunking are:

Focus on what you’re trying to chunk, either the information or some martial art movement that you’re doing;
Understand the idea that you’re trying to chunk. If you have trouble, try talking to other people that understand the concept or check how the person who invented it did it;
Build the context on when this specific knowlegde is or isn’t applicable and how it relates to other concepts you already know;

Working memory and chunks

Researchers nowadays believe we have four working memory spots. These spots are the places where you hold chunks when you’re focusing on something. When you’re learning a new concept, all four spots are usually focused on chunking it, which is why it’s hard to learn when you have other distractions going on, they are holding up your working memory spots and leaving less space for your focused mode to grasp and build the chunk.

Once you actually build it, it either forms a new chunk in your brain or is mixed with another chunk you already have. Once you have to recall that chunk, instead of filling up your whole working memory, it will fill a single spot leaving more space for the other stuff you’re focusing on.

For instance, when learning functional programming, you will often need to pull the chunk of recursion and place it in your working memory. You’ll have to remember how you turn imperative loops into their respective recursive versions, how you use helper functions to pass state around and how they relate to the new method you’re learning. Once you understand the basics of functional programming, recursion won’t be sitting alone anymore, it will be accompanied by maps, folds, flat maps, cons lists and other concepts that make extensive use of recursion in functional programming.

Every chunk you build allows you to cram more information into your limited working memory (the memory you use to focus and think now) and also simplifies pulling data from your long term memory as they will be filed together. When you think about a fold/reduce method you’re also pulling the other related methods, like map and flat map since they’re all together inside the same concept.

The illusion of learning

One of the main pitfalls we encounter while learning and building chunks is the illusion of learning. This happens when you think you’ve learned a concept but since you’re really close to the actual material and you can look into the book to remind yoursef, it seems like you have already memorized the concept.

I learned the armbar on a monday, the next wednesday and friday trainings involved other movements other than the armbar, wednesday the other week we started doing it again and I felt helpess as I was making the same mistakes I did when I learned it the first time. What was wrong here? Didn’t I learn this just a week ago? Why am I making the same mistakes again?

Because it was an illusion of learning. While at that moment I could clearly reproduce the movement, it was still a very faint pattern in my brain, I needed more repetition to make it be a strong chunk that could be recalled later.

The best ways to build lasting chunks is by spaced repetition and recall. Since it took me more than a week to practice the armbar again, the original chunk had almost faded away, there were still some traces of it and I could get back to the right movement much faster than the first time, but it still required me to reconnect the brain patterns I had built before.

This is where spaced repetition comes into place. While repeating the same process many times once you have grasped it offers some return, it diminishes the more you repeat in a single session. What you want is to separate this repetition over different periods of time, first a day, then 3 days, then a week, until you have the chunk readily accessible in your brain at any time. Just re-reading some text or re-waching a class many times in a row might help you take a test tomorrow but the knowledge you’ve built will be gone the day after as you stopped practicing it.

Another important technique is recalling the content. Get a pencil and a piece of paper, look away from this screen and start writing the main ideas you can recall of what you have read so far. Now, look away!

Was it hard? What could you remember? Did you remember the Shu-Ha-Ri? Armbars? Ronda Rousey? Chunking?

Recall is a powerful technique because it forces you to go through the patterns in your brain that are being formed by what you are focusing on right now. Since you weren’t looking at the screen, you really had to remember something out of what you’ve read. Whenever you’re studying something new, stop for a bit, look away and try to remember what you’re reading, this will be much more effective than just re-reading everything without any practice.

Interleaving concepts and exercises

Imagine you’re going through a course that is separated in units and each unit has a quizz. Usually, these quizzes include only the concepts presented over that unit, it wouldn’t reference stuff from previous units directly. If it did, students might say hey, we didn’t prepare for this!, they possibly crammed for the previous unit and by now have very little memory out of what was learned back then.

Given we want to solidify the concepts, this is exactly what we have to do. As said earlier, re-reading the same text over and over in a single session (or overlearning) offers very little returns once you’ve grasped the concept, instead, you should interleave the concepts and exercises to force you to exercise your chunks.

This is what improves your context detection, the Ri part of the chunk building process. It’s easy to know which part of the knowledge you must use to solve a problem when you know it’s quizz is specific to unit 3. But if the exercises are mixed between unit 1, 2 and 3, for every problem, you would have to think, which part of the knowledge is this about? How do I apply what I have learned so far to solve it?

Even if your exercises come in sequential lists, you can apply interleaving by doing a couple of exercises in one subject and then jumping to other ones. Try to randomly pick problems to solve so you don’t get used to the steps you’re following. If you always solve the problems in subject sequence, after the second problem solved you’re not even thinking about it anymore, you’re just repeating the same steps you did for the previous one.

You have to keep your brain alive, strengthening the connections you have and avoiding the shortcuts. Your brain is lazy, it will always try to do the least amount of work possible, you have to take the reins and drive it towards your learning objectives.

References

Your brain on thinking and learning

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 17 Jan 2015 00:00:00 UTC

This is part 1 of a three parts assignment for the Learning how to learn course on Coursera. Also check Part 2 - Chunking concepts to solidify and compress topics and Part 3 - Fighting procrastination.

Learning, specially science and math topics, usually ranges from the incredible feeling of understanding to the baffling feeling of I have no idea what I am doing here. While the first feeling is amazing, what gets burned in our brains is the feeling of powerlessness when we aren’t able to understand a specific subject.

I remember that back in college I had this profound difficulty of understanding what a C pointer was when we learned C and data structures during our second semester. I couldn’t wrap my head around the idea of that thing that points to another thing in memory, it was all to fuzzy, abstract. Still, while I didn’t actually understand the concept, I made my way through classes and ended up moving forward.

Coming from high school where I had very little trouble understanding the concepts, this college experience was a bad start. How could I be having so much trouble to understand these topics now if I had always been able to cruise through classes before? Worse, a lot of people in my class weren’t having these issues and had already grokked the concept. Did I make a bad decision going for a programming degree? Was this thing not for me?

Once we went off for winter break (June/July down here at the southern hemisphere) I decided to use the time off to better prepare myself for the new topics we would be learning. At the third semester we would start object oriented programming using Java, I saw Deitel’s Java How to Program at the local library and bought it. Every day I would read a bit, do the exercises and, well, enjoy my vacation. One of these days, a bit after I had learned about references in Java, the C pointers thing clicked.

It was one of those AHA! moments out of nowhere, were all the pieces of the puzzle fell in place and the concept finally solidified in my mind. I had finally figured out what all those stuff was about and why they were there. I jumped back to Turbo C (yes, this was 2003 I guess) and redid the binary trees and linked lists implementations. Now, instead of just repeating what I saw from the samples provided by the teacher, I actually knew what the code I was writing was doing.

It wasn’t the last time I felt like that in college (this sense of feeling dumb and failing are important parts of learning, no pain, no gain), but it was the moment I finally decided that this programming thing was for me. I wasn’t dumb and incapable of learning how to program, as I had considered before, I just needed more effort to wrap my head around all the abstract concepts that were being presented there.

It would take me years to understand how that actually happened.

Focused and diffuse mode thinking

Scientists nowadays believe our brain functions using two different networks, the highly attentive states and the resting state networks. We could call these the focused and the diffuse modes of thinking, respectively.

You enter the focused mode when you focus your brain into an specific problem or activity. When I was reading through Deitel’s book, I was exercising the focused mode, intentionally pushing the information in there into my brain by reading and doing the exercises.

The diffuse mode happens when you are not specifically focusing on anything, when you let your mind drift away and leave it to itself. When I was riding the bike, walking, doing chores, taking a bus to watch a movie or just not doing anything at home my mind would be in diffuse mode (come on, I was a student on vacation, of course there were times of doing nothing).

The moment C pointers clicked for me, I was in diffuse mode, my mind was wandering and connecting the ideas I had learned before while focused and it happened to finally connect the dots to make me understand it. Connecting ideas that are far away from one another or that you didn’t think were connected before is one of the main goals of the diffuse mode, the brain starts making these connections and while not everything is useful, many times you will create or understand something new out of this. If you’re a programmer, you most likely woke up in the wee hours with a solution for a problem you had during your day and scrambled to write it somewhere a couple times, this was your diffuse thinking doing it’s work while you were asleep.

When you are in focused mode, you are working with a small set of information, you’re reading a book, watching a lecture or listening to a podcast. The information in there is usually focused on a single specific subject, and this is great. You don’t want to be watching a lecture that goes from Haskell to ancient aliens and ends up in rocket science, it would be really hard to understand any of it.

This focus also means your mind is mostly thinking on this specific subject alone and not trying to connect it to stuff that is not directly related to it. This leads to problems in itself as the Einstellung effect were you are so focused on solving a problem in a specific way, you fail to see there are many other ways to solve it. When all you think about is hammers, every problem will look like a nail.

I did suffer from this not that long ago, when I had a project built for Windows that we wanted to see if it could be run on Linux. My first instinct was let’s get this to build on Linux and I did start down this path, verifying which pieces would build, which wouldn’t and what we would have to make them actually build. It was definitely going to be a considerable effort but I just couldn’t think of something else.

Thankfully, my boss came in with another idea, what if we tried to run this over Wine?

Guess what?

It did work. While we still want to eventually move everything to also compile on Linux, having the system running on Wine with very little effort was an huge progress (compared to the compiling on Linux project).

And this is when the diffuse mode shows it’s importance.

When your mind is drifting in diffuse mode, it is functioning in a big picture way, it’s not focusing directly at the specificities of a problem, but how it might relate to the other chunks of information you already have in your mind. Think about Newton’s apple incident (yes, it might not be true), Voltaire said:

Sir Isaac Newton walking in his gardens, had the first thought of his system of gravitation, upon seeing an apple falling from a tree.

He was walking in his gardens when the thought reached him. Magic, right?

Not quite. Newton had been thinking about this terrestrial gravity thing for a while already and he had a large body of knowledge in physics and the forces in the universe, which he acquired through focused and diffuse mode thinking before. Maybe all he needed this small glimpse of creativity (yes, science and math also require creativity) to connect the many concepts he had in mind all the time.

He needed to have this big picture view of the ideas to connect them, but he first needed to pump these ideas into his mind using the focused mode. The diffuse mode can only connect information that is already available in your brain, it can’t magically produce knowledge you never had. You’ll never get calculus in some magic moment if you never actually try to understand and study limits, derivatives and integrals, the information is just not there to be connected to anything.

Take it easy

One of the major mistakes we make when studying is studying too hard. You find a hard problem you can’t solve at that moment, you keep banging your head against the wall trying to figure out the solution, but it never comes. Then you go to sleep, for a walk or have a shower and the solution hits you.

This is not an accident. You were in focused mode all the time trying to solve the problem but the actual solution was somewhere you weren’t connecting in focused mode, people will usually say you were thinking inside the box. The actual solution required you to think outside the box but, unfortunately, you can’t consciously do that.

When you find a hard problem you can’t solve after a long stretch using your focused thinking, it might be the sign you need to leave it be for a while so your mind can try and fix it in the background. To do that you need to let the diffuse mode kick in and some of the best ways to do that are:

Sleep on it - this is the most common and powerful way of doing it. Famous creatives like Thomas Edison and Salvador Dali used to sit in a lounge chair holding an object in their hand so it fell and woke them up when they started to drift away, this was just enough time to let the diffuse mode to kick in and connect their ideas;
Go out for a walk, running, swimming or weight lifting - many writers like Jane Austen and Charles Dickens used to go for a walk to get a creativity boost;
Have a shower;
Meditate;
Listen to music;
Talk to others;

Anything that takes your mind away from the problem at hand and doesn’t send you into focused mode again will do, really. The main takeaway here is that forcing yourself to solve a problem when the connections aren’t really there is unlikely going to get you somewhere.

Worse, research has shown that we learn much more by spaced repetition (when you study something now and only study or practice again a day or more later) than by trying to repeat the same study or exercises one after the other. Your mind needs to rest so it can make connections and staying on focused mode all the time going over and over the same information won’t really help you much at it.

To learn effectively, you need to be able to switch to both modes. You need to push the ideas into your brain by using the focused mode but you also need to let it connect with the other concepts you already have in your mind, specially when solving problems, so you can really build upon what you’re learning. Balancing these modes is the key to waste as little time as possible when learning new concepts and solving problems.

References

Understanding Enumerable methods by re-implementing them in Ruby

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 12 Jan 2015 00:00:00 UTC

Ruby’s Enumerable is, by far, one of the greatest examples of how modules should be made. It offers a large collection of methods that are useful for those handling data structures and only requires you to implement a single method, each. So, any class that could behave like a collection and implement an each method can be used as an Enumerable.

A nice exercise to understand how Enumerable works is implementing it’s main methods. By implementing each method ourselves, we understand better what each one of them is doing and how it was possible to build this much functionality requiring only a single method to be implemented.

First, we need a class what will include our CustomEnumerable, let’s define it:

class ArrayWrapper

  include CustomEnumerable

  def initialize(*items)
    @items = items.flatten
  end

  def each(&block)
    @items.each(&block)
    self
  end

  def ==(other)
    @items == other
  end

end

Not much here, it includes CustomEnumerable (our own Enumerable implementation) and is basically a wrapper around an Array. The == method is also implemented here just so we can easily use the eq RSpec matcher when implementing our solution, it is not a requirement for a functional Enumerable implementation.

`map`

The documentation for map says:

Returns a new array with the results of running block once for every element in enum.

So, our code has to call the block given on every item of the collection and then build an array of the results of every call. Let’s implement it:

module CustomEnumerable

  def map(&block)
    result = []
    each do |element|
      result << block.call(element)
    end
    result
  end

end

This will be a pattern for almost all the methods we will be creating, create the destination array, call each on itself and then do the actual work. It’s important to notice that our implementation knows nothing about where it is being included, the only expectation it has is that there is an each method implemented that yields to a block with an object.

To see map in action, let’s see how we could create a new array by multiplying every number on it by 2:

it 'maps the numbers multiplying them by 2' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.map do |n|
    n * 2
  end

  expect(result).to eq([2, 4, 6, 8])
end

`find`

Here is what the docs say about find:

Passes each entry in enum to block. Returns the first for which block is not false. If no object matches, calls ifnone and returns its result when it is specified, or returns nil otherwise.

find is used to find an item inside an Enumerable given it matches the block given to the method. If the item is not found, it returns the default value. Let’s build it:

def find(ifnone = nil, &block)
  result = nil
  found = false
  each do |element|
    if block.call(element)
      result = element
      found = true
      break
    end
  end
  found ? result : ifnone && ifnone.call
end

First we setup the variables that will contain the result if we find it and a signal variable if we have really found the value. Why don’t we just use result with nil to signal that we have not found anything? Because nil might actually be what the user is looking for!

So we really need to know if we have found something (whatever it is) or not before returning. And if we don’t find anything, we call ifnone and use it’s result as the result of the operation, if ifnone is nil we just return it.

There are many cases for find. For instance, we need to be able to find an item:

it 'finds the item given a predicate' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.find do |element|
    element == 3
  end

  expect(result).to eq(3)
end

We could want to change the default value if it does not match anything:

it 'returns the ifnone value if no item is found' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.find(lambda {0}) do |element|
    element < 1
  end
  expect(result).to eq(0)
end

This is useful if you always want to return a value back, even when nothing was found by our search.

And by default if nothing is found, we need to return nil:

it "returns nil if it can't find anything" do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.find do |element|
    element == 10
  end
  expect(result).to be_nil
end

While useful, find returns once it matches the first value inside the collection, what if I wanted to return all values inside the Enumerable that match a criteria?

We use find_all!

`find_all`

The docs say:

Returns an array containing all elements of enum for which the given block returns a true value.

So, now we don’t have defaults anymore, the method always returns an array of all the items that matched the block (or an empty array in case no match happens), let’s build it:

def find_all(&block)
  result = []
  each do |element|
    if block.call(element)
      result << element
    end
  end
  result
end

Since find exits at the very first match, we can’t really reuse it here, the method has to be built on top of each from scratch. We build an array, then traverse our enumerable checking every item, once an item matches the block, we add it to the result. Once we’re done, we return the collection of items that match.

Let’s look at a few examples:

it 'finds all the numbers that are greater than 2' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.find_all do |element|
    element > 2
  end
  expect(result).to eq([3,4])
end

it 'does not find anything' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.find_all do |element|
    element > 4
  end
  expect(result).to be_empty
end

Even in the case of no matches, an array (albeit empty) is returned so the code that uses find_all needs to remember that it has to consider this option and verify if the array has items or not (instead of checking for nil as it would be done with find).

`reduce`

reduce or inject (also known as foldLeft in other languages like OCaml and Scala) is a method that applies a function to an accumulator and an item inside the Enumerable and produces the accumulator object at the end of the run. While it sounds weird, it’s actually a very useful function whenever you need to perform a function that aggregates data inside a collection.

Let’s look at what the docs say about it:

Combines all elements of enum by applying a binary operation, specified by a block or a symbol that names a method or operator.

If you specify a block, then for each element in enum the block is passed an accumulator value (memo) and the element. If you specify a symbol instead, then each element in the collection will be passed to the named method of memo. In either case, the result becomes the new value for memo. At the end of the iteration, the final value of memo is the return value for the method.

If you do not explicitly specify an initial value for memo, then the first element of collection is used as the initial value of memo.

So we have to either take a block or a symbol and we might or might not get an initial value, if we don’t, assume the first item is the initial value. This implementation will actually be a bit tricky, let’s start with the simple case, we give it a block and an initial value:

def reduce(accumulator, &block)
  each do |element|
    accumulator = block.call(accumulator, element)
  end
  accumulator
end

So, this is pretty simple, we just call block with accumulator and element and the next accumulator is whatever the block call produces. Pretty simple implementation, but this abstraction is incredibly powerful and available on all functional programming languages for aggregations (the reduce here is the reduce from the map-reduce paradigm!).

Let’s see it in use:

it 'sums all numbers' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.reduce(0) do |accumulator,element|
    accumulator + element
  end
  expect(result).to eq(10)
end

And here we have a simple reduce function that takes the accumulator and then produces the sum of all items. It’s also important to verify the case of the empty Enumerable, if it happens to be empty, it should just return the initial value provided:

it 'returns the accumulator if no value was provided' do
  items = ArrayWrapper.new
  result = items.reduce(50) do |accumulator,element|
   accumulator + element
  end
  expect(result).to eq(50)
end

Now, let’s include the first optional parameter, the operation symbol that serves as the operation to be applied instead of a block.

def reduce(accumulator, operation = nil, &block)
  if operation && block
    raise ArgumentError, "you must provide either an operation symbol or a block, not both"
  end

  block = case operation
    when Symbol
      lambda { |acc,value| acc.send(operation, value) }
    when nil
      block
    else
      raise ArgumentError, "the operation provided must be a symbol"
  end

  each do |element|
    accumulator = block.call(accumulator, element)
  end
  accumulator
end

The actual implementation here does not change much, we have included a couple validations to make sure the values are the expected ones and when provided a symbol, we build a block to be used ourselves that just uses send to call the method defined by the symbol given as a parameter. The actual loop doesn’t change.

Now let’s look at some usage:

it 'executes the operation provided' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.reduce(0, :+)
  expect(result).to eq(10)
end

First, the basic usage, calling reduce with a symbol that will be applied to the accumulator and every value. This is the same as our first reduce example, but now it uses very little code.

Now let’s look at the failure cases, first one providing both an operation and a block:

it "fails if both a symbol and a block are provided" do
  items = ArrayWrapper.new(1, 2, 3, 4)
  expect do
    items.reduce(0, :+) do |accumulator,element|
      accumulator + element
    end
  end.to raise_error(ArgumentError, "you must provide either an operation symbol or a block, not both")
end

When both are provided, we must fail as it’s impossible to know what the user wanted. Same as if the operation provided is not a symbol:

it 'fails if the operation provided is not a symbol' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  expect do
    items.reduce(0, '+')
  end.to raise_error(ArgumentError, "the operation provided must be a symbol")
end

Not a Symbol? Sorry, can’t use it.

Now for the last step on our implementation, the accumulator parameter is now optional. If it isn’t available, the first element inside the collection should be used. Now we have 4 cases for the reduce call:

accumulator + block
accumulator + operation
operation
no params + block

This time, let’s start defining the two specs that are missing, first, the operation only call:

it 'executes the operation provided without an initial value' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.reduce(:+)
  expect(result).to eq(10)
end

And, finally, the call with a block alone:

it 'executes the block provided without an initial value' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  result = items.reduce do |accumulator,element|
    accumulator + element
  end
  expect(result).to eq(10)
end

Why get these two tests here first?

Look at them, they are actually the same case, which is the case without an accumulator, the only difference is that one provides a block and the other doesn’t, but both will have to pull the first item inside the collection and then run the reduce operation.

If we try to run these specs:

Failures:

1) CustomEnumerable reduce executes the operation provided without an initial value
  Failure/Error: @items.each(&block)
  NoMethodError:
  undefined method `call' for nil:NilClass
  # ./lib/custom_enumerable.rb:47:in `block in reduce'
  # ./spec/custom_enumerable_spec.rb:12:in `each'
  # ./spec/custom_enumerable_spec.rb:12:in `each'
  # ./lib/custom_enumerable.rb:46:in `reduce'
  # ./spec/custom_enumerable_spec.rb:128:in `block (3 levels) in <top (required)>'

2) CustomEnumerable reduce executes the block provided without an initial value
  Failure/Error: result = items.reduce do |accumulator,element|
  ArgumentError:
  wrong number of arguments (0 for 1..2)
  # ./lib/custom_enumerable.rb:32:in `reduce'
  # ./spec/custom_enumerable_spec.rb:134:in `block (3 levels) in <top (required)>'

How do we build this? Most of the code will be handling the parameter juggling. Since reduce was declared long before ruby had named parameters, there is no magic way to decide if the accumulator is an operation or not, we will have to manually check this.

Also, we need a way to get the first item at the collection, otherwise we will end up having to do this inside reduce itself. Let’s start by implementing first then:

def first
  found = nil
  each do |element|
    found = element
    break
  end
  found
end

Using it is quite simple:

it 'returns the first element inside a collection' do
  items = ArrayWrapper.new(1, 2, 3, 4)
  expect(items.first).to eq(1)
end

it 'returns nil if the collection is empty' do
  items = ArrayWrapper.new
  expect(items.first).to be_nil
end

If you’re asking yourself why I’m using break and not just returning from the each block, try changing the code to:

def first
  each do |element|
    return element
  end
end

What happens now?

The second spec, that expects nil to be returned when the collection is empty, fails. Why? Because each returns the collection itself once it runs, since the code was never executed (the collection is empty!) each just returns itself and not nil as our spec expects. So, that’s why we need to explicitly declare our return value instead of relying on the iteration.

Now that first is also implemented, let’s produce the final reduce implementation:

def reduce(accumulator = nil, operation = nil, &block)
  if accumulator.nil? && operation.nil? && block.nil?
    raise ArgumentError, "you must provide an operation or a block"
  end

  if operation && block
    raise ArgumentError, "you must provide either an operation symbol or a block, not both"
  end

  if operation.nil? && block.nil?
    operation = accumulator
    accumulator = nil
  end

  block = case operation
    when Symbol
      lambda { |acc, value| acc.send(operation, value) }
    when nil
      block
    else
    raise ArgumentError, "the operation provided must be a symbol"
  end

  if accumulator.nil?
    ignore_first = true
    accumulator = first
  end

  index = 0

  each do |element|
    unless ignore_first && index == 0
      accumulator = block.call(accumulator, element)
    end
    index += 1
  end
  accumulator
end

Given we don’t know exactly how the parameters will be provided or the collection structure, we can’t really optimize this call (not unless we duplicate the code a bit, for instance, streamlining the implementation if there is an accumulator). But since we want this code to work for all cases, we will hope classes that include this module will provide implementations more aligned with their structure.

The code starts by verifying all parameters, if no parameters were provided, give it up, there’s nothing to do here. Then it starts checking which case we’re taking about here, the first check if operation and block are nil, if both are, it means the accumulator field will have to be the operation and that we don’t have an accumulator.

Then we have the operation validation we had before and we reach another new piece, the check for the accumulator. If the accumulator is nil, we must pull the first item of the collection and we must also instruct the method to ignore the first iteration (as we have manually navigated to it).

Our new each loop now checks these special variables for the empty accumulator case so we can safely process the collection without duplicating the values. This would be a nice place for an optimization were we have the same loop we had before if there is an accumulator, you can definitely improve this method by including this change yourself.

And this concludes the reduce implementation, see if you can come up with better or faster solutions for this, there are definitely better options.

`reduce` magic

Now that we have reduce implemented, there are many methods that can be built around it, like min and max:

def min
  reduce do |accumulator,element|
    accumulator > element ? element : accumulator
  end
end

def max
  reduce do |accumulator,element|
    accumulator < element ? element : accumulator
  end
end

Since reduce already handles the empty case:

it 'produces nil if it is empty' do
  items = ArrayWrapper.new
  expect(items.max).to be_nil
end

And the single element case:

it 'produces 1 as the max result' do
  items = ArrayWrapper.new(1)
  expect(items.max).to eq(1)
end

Our min and max implementations don’t even have to care about it, all they have to do is to provide a block that does the comparison and returns the highest or smallest value found, all the looping and special case handling is done by the reduce function we wrote before. Quite powerful, isn’t it?

There are many other Enumerable methods you can implement just using reduce, like each_with_index, each_with_object, count, max_by, min_by and others, give it a try and complete the enumerable implementation using reduce whenever you can.

And here we end our Enumerable overview by implementing it in Ruby. The full source code for this example is available on github.

Building a prefix tree (or trie) in Scala

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 06 Jan 2015 00:00:00 UTC

Following up on my new year’s resolutions, I’ve been reading Taming Text and there’s quite a lot of interesting stuff there, one of them is the prefix tree.

Prefix trees (or tries) are a very efficient way to store collections of words so you can search for them by their prefixes (or suffixes if you revert the strings). You can use them to produce predictive lists for auto-complete solutions, dictionaries or any solution where you need to quickly access collections of words (or any type of value, as long as it’s keyed by a string) that start with a given prefix.

Defining the operations

Let’s see which operations we would like our trie to implement:

sealed trait Trie extends Traversable[String] {

  def append(key : String)
  def findByPrefix(prefix: String): scala.collection.Seq[String]
  def contains(word: String): Boolean
  def remove(word : String) : Boolean

}

First, our trie is a Traversable[String]. The only thing Traversable[T] requires is a foreach method and we can easily implement one for our data structure. Being a Traversable[T] means our trie can interact with many of the other collections declared at the standard library (and this will be useful at our implementation later). When building new data structures in Scala, it’s often good to make sure they implement one of the base collection traits to simplify their interoperability with the standard library.

Then we have the methods we need for our trie, append to add a new word, findByPrefix to list all words known that start with the given prefix, contains to check if we have the word included at our structure and finally remove to remove a word from it.

The `TrieNode` data structure

The prefix tree, as the name implies, is built in the form of a tree data structure where every node represents either the empty char (the root node) or a single character in a word. The nodes here might or might not contain children and they can also be a word node, meaning that the path of characters that lead to this specific node form an actual word included at the prefix tree.

Let’s look at the structure:

private[trie] class TrieNode(
  val char : Option[Char] = None,
  var word: Option[String] = None) extends Trie {
  private[trie] val children: mutable.Map[Char, TrieNode] =
    new java.util.TreeMap[Char, TrieNode]().asScala

}

Option twice!

Yes, that’s it. The very first node at our tree represents the empty char and since there is no empty char we just make it an Option object since it could then be a None to signal this is the empty char (if you don’t know what Option is, check this out ).

Also, the node might be a word node or not. What does it mean to be a word node? A word node is one that represents a word that was included in the prefix tree. Let’s look at the tree structure we have here:

Here we have a tree with two names, joão and joab, if we include another name like joão paulo, how do we remember that joão was a name as well?

To make sure we don’t forget the names, we have to also mark the tree nodes as word nodes, so we know that we have a word for that part of the tree even if this node has children that forms other words as well. So, that’s the reason why we have this second Option field at the TrieNode object.

The structure also maintains a list of it’s own children so we can continue traversing the tree structure. Given Scala does not provide a mutable TreeMap data structure, we create a Java TreeMap object and wrap it as if it was a Scala mutable map (with the asScala method from scala.collection.JavaConverters). The use of TreeSet here helps us make the traversal in the tree follow the lexicographic order for the stored words.

`append` and `foreach`

We have to talk about these two methods together because it’s really hard to test one without the other. Let’s look at append first:

override def append(key: String) = {

  @tailrec def appendHelper(node: TrieNode, currentIndex: Int): Unit = {
    if (currentIndex == key.length) {
      node.word = Some(key)
      } else {
        val char = key.charAt(currentIndex).toLower
        val result = node.children.getOrElseUpdate(char, {
          new TrieNode(Some(char))
          })

          appendHelper(result, currentIndex + 1)
        }
      }

  appendHelper(this, 0)
}

The first thing to take note here is that since we’re dealing with a recursive data structure, most of our methods will be implemented in a recursive way. The goal of the append method is to create a new node for every letter at our provided word that does not exist at our tree and then mark the last node as a word node.

For instance, imagine our tree only had john included:

j
  o
    h
      n

Once we try to include jane, the tree then becomes:

Both words share the j node, but then they diverge to their own subtrees to form their own words.

And since we’re dealing with recursion, we need to make sure our program won’t blow the stack by doing too many recursive calls. To do that, we’ve included the appendHelper function and tagged it with the @tailrec annotation that will require the method to be written in a way that makes it possible to apply the tail call optimization. The requirement is that the function has to call itself only at a tail position or that it can only call itself as the last operation in a branch.

This is exactly what we do, appendHelper is only called at the end of the else block and hence this method can be optimized and will not cause a stack overflow. Since this function is only usable by the append method, there’s no need to move it out, we can just have it here and not leak this implementation detail.

Now let’s look at the foreach function:

override def foreach[U](f: String => U): Unit = {

  @tailrec def foreachHelper(nodes: TrieNode*): Unit = {
    if (nodes.size != 0) {
      nodes.foreach(node => node.word.foreach(f))
      foreachHelper(nodes.flatMap(node => node.children.values): _*)
    }
  }

  foreachHelper(this)
}

This is much simpler than append, all the code has to do is to take the function provided as parameter and call it on every word node. Since the word object is itself an Option, we can just call foreach at the Option object and it will do the right thing.

Again, we have an internal helper function that actually does the work and is made to be tail call optimized. It’s important to have the stop condition around the actual code here, to make sure we don’t even bother calling foreach if we don’t have any items, otherwise the code would loop forever.

The only interesting bit here is that we make the foreachHelper method take a vararg so we can call it with any number of parameters, this guarantees we can make it optimizable since we can make the first call with a single parameter and then call it again at the tail of the method with the child collection for the current node.

Here’s how these two methods are used:

Here’s how we use it:

"include a word" in {
  val trie = new TrieNode()
  trie.append("Maurício")

  trie must contain("Maurício")
}

Here we can clearly see append in use, but where is foreach?

foreach is in use at the contain matcher here. The contain matcher expects to find a collection where it can iterate and try to find the value we’re asking for. So, while we didn’t implement methods to figure out if we can find that item, the fact that we have implemented foreach and our Trie inherits from the Traversable trait provides us with a lot of functionality for free.

Finding words by their prefix

This is where the prefix tree really becomes useful, when you have to find words by their prefix. What we do here is we take a prefix and find every word that happens to match the prefix given, with a common hash structure you would have an O(1) (a single operation) to find a specific item by key, but you wouldn’t have a way to find all items that have a key that start with bar and this is where tries are useful.

Let’s look at our implementation:

override def findByPrefix(prefix: String): scala.collection.Seq[String] = {

  @tailrec def helper(currentIndex: Int, node: TrieNode, items: ListBuffer[String]): ListBuffer[String] = {
    if (currentIndex == prefix.length) {
      items ++ node
    } else {
      node.children.get(prefix.charAt(currentIndex).toLower) match {
        case Some(child) => helper(currentIndex + 1, child, items)
        case None => items
      }
    }
  }

  helper(0, this, new ListBuffer[String]())
}

Now we are traversing towards a specific path of nodes in our tree, we only want the nodes that match the character we have at every index (all characeters are also lowercased). This could lead us to two cases, one where we have found the character (and then continue deeper) and one were one of the characters in the prefix does not match. As soon as we fail to match (the None case) we return. Otherwise, if we reach the end of the string we append all words under that node to the accumulator list.

Again, this is foreach and Traversable[T] in action. The ++ method at ListBuffer couldn’t possibly know what to do with our TrieNode object, but since we have extended the Traversable[T] trait and implemented foreach it can just use the methods available and pull all the items from that part of the tree and down. We don’t have to actually collect the items, the ++ method will already do the collection work for us and include the words at and below the current item.

Let’s look at an example with a trie that contains johann, john and joan:

j
  o
    a
      n # word node
    h # this is the current node
      a
        n
          n #word node
      n # word node

The node with the h is the one at if (currentIndex == prefix.length) when we make a search by joh, so, once we call foreach on it, this is the beginning of our tree now, only the words that are inside or below it will be returned. In this case, it would return john and johann but not joan because joan is not under the node that contains h.

And again the method uses an internal tail recursive helper function to do it’s job so we don’t blow the stack when trying to find all words that match the prefix.

A not very efficient `contains`

contains is implemented here mostly for completeness, but this is not exactly what you want to do with a prefix tree. If your actual use case is to figure out if whole words are inside a collection, a Set is a much better solution since it will usually require a single operation to give you this information while the worst case scenario for the trie is going to be O(N) where N is the size of the string you’re looking for.

Let’s look at the implementation:

override def contains(word: String): Boolean = {

  @tailrec def helper(currentIndex: Int, node: TrieNode): Boolean = {
    if (currentIndex == word.length) {
      node.word.isDefined
    } else {
      node.children.get(word.charAt(currentIndex).toLower) match {
        case Some(child) => helper(currentIndex + 1, child)
        case None => false
      }
    }
  }

  helper(0, this)
}

As you can see from the code, the best case will be O(1) (a single operation) if not even the first character is included in the tree, but if the word is included or if it is the prefix for a word that is included, the code performs as many operations as there are characters in the string.

A simple solution for a faster contains could be having a Set that holds all words inside the prefix tree, but this is beyond our scope here. Also, we could assume that this implementation could be simplified if we used the findByPrefix method and checked if the word was included in the resulting list. While this is indeed possible, it would require many more operations as findByPrefix finds all words at and below the prefix, while our contains only cares about the exact prefix match.

Still, we could abstract the way they traverse the tree as it is exactly the same and avoid repeating the traversal code, which is something we will do for the remove operation.

Removing items from the trie

This is arguably the most complicated piece of our trie implementation, removing an item from a trie is an expensive operation and at it’s worst case will require O(2N) operations where N is the size of the string being removed. The implementation is complicated because we must build a direct path of nodes until the actual word node we’re looking for and then we must travel this path backwards removing all the empty nodes, usually stopping when we find another word node.

Let’s look at an example:

j
  o
    h
      a
        n
          n #word node
      n # word node
        n
          a
            u
              r #word node

Here we have a tree with the names johann, john and john naur. If we remove john naur we have to remove the whole naur suffix but leave john untouched since we also have john as an actual name, so what needs to happen is:

reach that last r node;
walk up to u;
delete the reference to the r node at it’s children, this will make u not have any children and since it isn’t a word node it means that we have to remove it;

We keep repeating this until we reach the n that is the end of john, we again delete the reference to the n node for naur that is on it and since this node is a word node our delete process has to stop.

As you can see, deleting a word from our prefix tree is a quite complicated operation. As with contains, you’re better off if you build the tree and avoid removing items from it.

The delete operation is divided into two pieces, the first is to build the full path until the word node that is being deleted:

private[trie] def pathTo( word : String ) : Option[ListBuffer[TrieNode]] = {

  def helper(buffer : ListBuffer[TrieNode], currentIndex : Int, node : TrieNode) : Option[ListBuffer[TrieNode]] = {
    if ( currentIndex == word.length) {
      node.word.map( word => buffer += node )
    } else {
      node.children.get(word.charAt(currentIndex).toLower) match {
        case Some(found) => {
          buffer += node
          helper(buffer, currentIndex + 1, found)
        }
        case None => None
      }
    }
  }

  helper(new ListBuffer[TrieNode](), 0, this)
}

We provide a list buffer to be filled with nodes and we then traverse the tree to buid the actual list, if we reach the end of the word we’re looking for and it is indeed a word node then we return the list, if it isn’t we just return None to signal that we haven’t found the full word we’re looking for. It’s important here that we only return if we do find a word node that matches, otherwise we could be deleting parts of other words.

Then, with the method that builds the path, we can implement the remove operation:

override def remove(word : String) : Boolean = {

  pathTo(word) match {
    case Some(path) => {
      var index = path.length - 1
      var continue = true

      path(index).word = None

      while ( index > 0 && continue ) {
        val current = path(index)

        if (current.word.isDefined) {
          continue = false
        } else {
          val parent = path(index - 1)

          if (current.children.isEmpty) {
            parent.children.remove(word.charAt(index - 1).toLower)
          }

          index -= 1
        }
      }

      true
    }
    case None => false
  }

}

The remove method ended up being imperative mostly because it’s easier to understand, what we do here is first mark the last node (the word node) as not being a word node anymore and we keep going for the parent of the current node and removing the current node from it if it does not have any children. This is done because if the current node has no children and it isn’t a word node, it is part of the word we’re removing so it should be taken out as well.

The trie is done

And with this we end our trie (or prefix tree) implementation, as you can see from our implementations, if you need to quickly find words given a prefix, this is a great data structure to use, specially if most of what you will be doing is including words and then trying to find them.

If you actually need to match full words or you need to perform many inclusions and removals, this might not be the best option for you.

The full source for the trie implementation is here and the spec is here.

Technologies I'll learn in 2015

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 25 Dec 2014 00:00:00 UTC

I don’t usually plan years in advance or try to predict what I’m going to learn as work and life priorities change over time, but my goal of building a strong statistics background over 2014 has definitely worked out exactly because I decided to focus on doing it instead of trying to do a lot of different unrelated stuff throughout the year.

With 2015 right at the door, I think it’s time to define what kind of stuff I will be focusing on to keep up the pace.

Distributed systems (or, if you prefer buzzwords, cloud computing)

As the systems grow, more and more of my work involves scaling, integrating and making sure disparate services distributed among many machines stay up and responsive. Unfortunately, most of my distributed systems knowledge has been acquired in the trenches and finding good introductory material to the field isn’t easy.

The goal is to cover most of the topics summarized here and then start going through the actual papers in the field. I’ve already finished Distributed systems for fun and profit by by Mikito Takada and will continue going down the list soon. There is also a Coursera specialization on Cloud Computing by the University of Illinois at Urbana-Champaign that definitely looks promising, gonna enroll and see how it works out. Last but not least, the collection of lectures from Eidgenössische Technische Hochschule Zürich on many different topics about distributed systems has been recommended as a great resource as well.

While gaining experience in the trenches is amazing, understanding the theory, algorithms and options available in the field are also important when making decisions about using the various different systems available, as it can be seen at the current discussion about tools on service discovery . It’s important to make informed decisions and to be able to do that we must understand how distributed systems work.

Machine learning and even more statistics

My main reason to dig deeper into statistics this year was to be able to understand what is going on at the machine learning universe. While it would have been easy to start copying algorithms and running them on data sets and call it all machine learning, I lacked the basic statistics background to understand what was actually going on with those algorithms.

Why did they work? How did they work?

Most of the courses and books on machine learning I found were about writing algorithms and it definitely felt like I was missing something not understanding the statistics that was actually behind most of them. Now that I’ve been able to build this backbone, I’m going back to the machine learning books and courses.

But this doesn’t mean my statistics days are over, I had to pause the Data Science specialization right in the middle because I wanted to take the Introduction to Data Science and Data Analysis and Statistical Inference courses (and both were amazing), so now it’s time to circle back and finish the last courses at the specialization and do the capstone project.

My machine learning days will have to live side by side with my continuing statistics education.

The planned resources so far:

Rust and systems programming

This year I had a new foray into C/C++ development and while I still don’t have considerable experience with these system level languages, it didn’t feel like I would like to devote my study time to them. The tooling feels awkward, C++ as a language seems to give me too much power for no reason (and yes, I want to be constrained and protected from spurious crashes if possible) and it they didn’t click to me.

But this doesn’t mean I’d never do systems programming. Now that Rust is about to reach it’s 1.0 release with a more consistent syntax and standard library, it is definitely time to get serious about it.

The language offers a systems programming environment while still retaining and guaranteeing a lot of the safety you only get at interpreted languages nowadays (or languages that compile to intermediary bytecodes and run on VMs like Java), has a nice functional feel (right from the ML family, which I love by the way) and offers a simpler way to do manual memory management with it’s borrowing and boxing ideas.

The first goal is to build a functional memcached clone. Given this requires using doing threading, data structures, networking and a lot of memory management, it’s definitely going to be a nice tour around the language. Eventually, as the distributed systems education progresses, this memcached clone will become a CRDTs server so I can get some of this distributed systems knowledge out there. Given there aren’t that many CRDT server solutions out there, this might definitely become an actual and useful project in the long run.

Is this doable at all?

I’ve no idea. But having a clearly laid plan for that to study at 2014 did show me that focus was an important part of actually learning a topic. For many years I would be picking up stuff randomly, reading many different chapters in different books on different topics and, as you can imagine, very little of this was actually useful in the long run.

Even if I can’t do all this, I’m sure focusing on these topics and not on everything else that shows up at my twitter feed or hacker news is definitely a better bet at gaining new knowledge.

Do you have any other recommendations for these fields? Hit me on twitter!

May 2015 be an awesome year for all of us :)

Learning R - Part 2 - Making sense of data

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 23 Dec 2014 00:00:00 UTC

This is part of a series of posts introducing the R language

Disclaimer: This text contains spoilers for one of the assignments for the Reproducible Research course at Coursera, if you plan on doing the Data Science specialization you might want to save this to read later or just read it now anyway, it’s just one of the many assignments :)

One of the very first things you’ll learn while working with data is that, many times, you will find that the dataset you have is missing information on important fields, the data dictionary (the piece that explains what each field is) is outdated or incomplete, the values available contain typos, duplication and many other inconsistencies that makes the data not ready for consumption. This data could have been generated by independent computer systems, could have been entered manually by humans or the data collection method might just have been ad hoc, bad data will be something you’ll deal with.

Cleaning and making sense of the data is indeed part of your work doing analysis, whenever you get a dataset, your very first step is to run some exploratory analysis to see what the values look like, find inconsistencies, missing data and perform basic cleanups before starting the actual analysis.

We’ll use the National Climatic Data Center Storms database as the example here. You can go through all files and merge them yourself (not recommended, as you will get newer files with a different format) or you can just download this condensed file with data from 1950 to 2011 about natural disasters that have happened in US territory.

Starting out

As with the previous article, I’d recommend you to use RStudio to follow through the steps we’ll be doing here. Let’s start by loading the file into a data frame:

zip <- bzfile("repdata-data-StormData.csv.bz2")
data <- read.csv(zip,
  strip.white=TRUE,  
  colClasses=c("NULL","character","NULL","NULL","numeric","character",
  "character","character", "NULL",
  "NULL","NULL","character","NULL","NULL","NULL","NULL","NULL","NULL",
  "NULL","NULL","NULL","NULL","numeric","numeric","numeric","character",
  "numeric","character","NULL","NULL","NULL","NULL","NULL","NULL","NULL",
  "NULL","NULL"))

Given the file is large (if you unpack it it will be around half a GB) we don’t want to let R try to figure out the data types or load all columns, we will specifically select the columns we want to use and declare their types so the initial load is faster as R won’t be loading everything or trying to figure out what each field is.

Let’s look at the fields we have here:

names(data)
> [1] "BGN_DATE" "COUNTY" "COUNTYNAME" "STATE" "EVTYPE" "END_DATE"
> "FATALITIES" "INJURIES" "PROPDMG" "PROPDMGEXP" "CROPDMG" "CROPDMGEXP"

Most of the variables here should be self explanatory, let’s take a tour:

BGN_DATE and END_DATE - the start and end of the event;
COUNTY and COUNTYNAME - an unique identifier for the county and the name of the county;
STATE - state of the county where the event happened;
EVTYPE - this is the type of event that has happened, there are many types of events and they will require some clean up, we’ll get back to that in a bit;
FATALITIES and INJURIES - the number of victims (either killed or injured) by the event;
PROPDMG and CROPDMG - these are the cost of the damage caused by the event as property or crop damage;
PROPDMGEXP and CROPDMGEXP - these are the multipliers to be applied to the crop and property damage values provided;

Cleaning up events

We’ll start by looking at the different types of events we have:

unique(data$EVTYPE)

The unique function takes a vector and produces a new vector removing duplicates. This will print a list of 985 different types of events, but once you start looking around you’ll see different events as:

“FLOOD/FLASH FLOODING”
“FLASH FLOOD/FLOOD”
“Flood”
“THUNDERTSORM WIND”
“THUNDESTORM WINDS”
“THUNDERSTROM WINDS”

So we have typos, different casing and switched words all together for events that are supposed to be the same, we need a fix for that. While it would be great if I could tell you there is this magical R function that does this work for us, there isn’t. Due to the many different ways the same event is being presented differently here, the solution will be going through all the event names and changing them to a canonical one.

The first step is to make them all have the same casing and remove extraneous spaces from the beginning or the end of the event names, here’s how we can do it:

data$EVTYPE <- toupper(gsub("^\\s+|\\s+$", "", data$EVTYPE))

Now we start to see the magic, we have a function gsub that takes a regex of what we should replace (the regex is mostly remove all spaces at the beginning or end of the text), the value that will replace (an empty string in this case) and lastly the vector (or string) where we will be replacing stuff.

Here we circle back to what we said in part 1 that everything in R is a vector, the gsub function works if we give it a single string or a vector of strings. Since even the single string is actually a vector of size 1, it doesn’t actually matter if it’s a single one or a collection of them, the function will happily apply the regex at every item and return a new vector (or a single item) with all items processed.

This is an example of vectorization, a very common pattern in R programs. While you would usually use a for/while loop in other languages to accomplish this result, in R most functions will accept a vector and apply it’s operation at every item in the vector automatically. It’s uncommon to see while/for loops in R programs, as most libraries will prefer to use vectorized functions to perform it’s work. Preferring vectorization will usually lead to better performance than trying to loop since libraries will try to optimize for this case.

Going back to our code, we then apply the toupper function to the vector that was produced by gsub which will now produce a new vector with all strings in upper case form.

We then assign the results of the computation to the data$EVTYPE field again, let’s check how many event types we still have:

length(unique(data$EVTYPE))
> [1] 890

This basic cleanup removed 95 duplicate event types, not perfect but still a great start for our job. The next step is, unfortunately, a manual operation, going through every event type and mapping it to a canonical version so we can have less noise when organizing this data. You won’t actually have to do this since I’ve done it already but whenever you find yourself in the same situation, find an expert in the field (which will probably be working with you anyway on the anaylsis) and remove the duplicates.

You should download the replacements.csv that contains the mappings from the original to the canonical event type. Once it’s downloaded and available to be used, load it at your R session:

replacements <- read.csv("replacements.csv", stringsAsFactors=FALSE)

eventFor <- function( evtype ) {
  replacements[replacements$event == evtype,]$actual
}
data$event <- sapply(data$EVTYPE, eventFor)

Running this will take a while since we have to produce results for 902297 rows, just be a little patient. Another option would be using a library like hash (yes, R does not have hashes natively) for faster access, but we can live with this waiting for now.

Going back to the code, we first read the replacements.csv file as usual but now we include the stringsAsFactors=FALSE option to prevent the code from loading everything as factors, which wouldn’t work for our case. By default, read.csv will assume all string fields should be read as factors, so make sure you either declare the types of every column as we have been doing or just disable factors if you don’t want them.

And now for something new, a function declaration!

As you can see, there isn’t much to say here, you declare a variable that will hold the function, use the function keyword, declare the parameters and then {} around the function body. You don’t need to include a return statement if the last line of your function returns the expected value (which happens here) and, perhaps the most interesting part of it, the function has access to all variables/functions declared at it’s external binding.

What does this mean?

It means that the use of replacements inside the function is perfectly valid since replacements is available at the environment where the function was defined. The functions you declare can use any variable declared out of them as long as they are visible when the function is declared. While this isn’t exactly a best practice if you’re coming from other programming languages, it simplifies our life for this case as this function will be called by someone else and not us directly. We know global variables are evil, but let’s can make a small exception here.

And now vectorization happens again. If you’ve been presented to functional languages before (or languages that have map and fold/reduce methods) you already know what sapply is doing. It calls eventFor for every value at data$EVTYPE and since eventFor produces a single value for every call, it merges them all into a new vector, effectively mapping every original value into a new one.

Again, this could be done by a for loop, but using a function like sapply is much simpler, given it also transforms the data into a vector so we can just merge it into our data frame. Which is also something that this code is doing, it is creating a new column, called event at the data frame to hold these new values.

length(unique(data$event))
> [1] 187

And now we only have 187 events to care about, progress!

Parsing dates

The cleanup work never stops, now we’re going to parse the dates:

data$date <- as.Date(data$BGN_DATE, "%m/%d/%Y")
data$year <- as.POSIXlt(data$date)$year+1900

This gives us dates with actual fields (instead of just a string) and the year in separate as we’ll be using it later for other comparisons. Just like the events case, we save them as new columns at the data frame, this allows us to circle back to the original columns and re-process them if necessary. You should avoid overwriting the original fields at your data frames as you might have to re-process the original data later.

Assessing damage

We now consider the damage costs caused by the events. The original data dictionary for this dataset said the multipliers PROPDMGEXP and CROPDMGEXP could assume the values K (for thousands), M (for millions) and B (for billions) of USD. Let’s look at what the actual values at the dataset look like:

unique(data$PROPDMGEXP)
> [1] "K" "M" ""  "B" "m" "+" "0" "5" "6" "?" "4" "2" "3" "h" "7" "H" "-" "1" "8"
unique(data$CROPDMGEXP)
> [1] ""  "M" "K" "m" "B" "?" "0" "k" "2"

Not exactly what we were expecting, right?

Time for another transformation, now with the multipliers.csv file. Here’s the code to load and use it:

data$PROPDMGEXP <- toupper(data$PROPDMGEXP)
data$CROPDMGEXP <- toupper(data$CROPDMGEXP)

multipliers <- read.csv("multipliers.csv", colClasses=c("character", "numeric"))

mapDamage <- function(damage, mapping) {
  damage * multipliers[multipliers$key == mapping,]$number
}
data$property_damage <- mapply(mapDamage, data$PROPDMG, data$PROPDMGEXP)
data$crop_damage <- mapply(mapDamage, data$CROPDMG, data$CROPDMGEXP)
data$total_damage <- data$property_damage + data$crop_damage

Here we’re doing mostly the same of what we did with events, the main difference is that now we’re using mapply instead of sapply to calculate the results. Why is that?

sapply expects to call the given function with a single argument, but to actually perform the calculation here we need both the damage value and the multiplier for that value, so we need to give our functions the pair of damage and multiplier values. mapply comes to the rescue here since it allows us to provide N vectors and it is going to call our function with every sequence of the N values, which in our case is every damage and it’s own multiplier.

Again, no loops are necessary anywhere since all operations are vectorized. At the end we also sum the damage values to calculate the total damage for every event. Since R knows we’re dealing with vectors here it knows what we want is to sum every row and not the whole vectors, so what is happening at that last line is that every row will have it’s property and crop damage summed and included at a new column.

split-apply-combine

This natural disasters database has been collected from various different sources since the 50’, so it might be nice for us to look at how the data collected for every year changes. Our question here will be, how many different events happened on every year so far?

Aggregating values based on categorical variables (which would be our year here) is such a common task on data analysis that it has it’s own name, the split-apply-combine pattern. It’s called like this because you first split the data based on a specific column, then you apply a function to every subset of the data and at the end you combine all the different results into a single data set.

Let’s see how we could build our events per year dataset now:

years <- split(data, data$year)
events_per_year <- lapply(years, function (x) length(unique(x$event)) )
result <- do.call(rbind, events_per_year)

result here is a matrix with the following values:

As you can see, before 1993, very few different events were recorded every year, not because they did not happen, but because the rules to track these natural disasters were different. These discrepancies are going to be all over the place as you work with data and you should always consider what it will mean for your end results, as making direct comparisons between the years before 93 with the ones that come after it will surely generate different results. In this case, we’re better off using either one or the other side of the data set and not everything.

As for the code, the split function takes the data set as the first parameter and the categories as the second. You don’t have to provide unique values here, split is smart enough to figure out what you mean and correctly split the values on every unique option. This produces a list of year -> dataset subset for that year.

Then we use the lapply function at this list, it calls the function you give it for every item at the list and since for our case every item in the list is the subset for a specific year, all we have to do is to select the unique event types and calculate the length of the produced uniques. See how we declared the function right at the parameter? That is an anonymous function, really useful when you want to provide a one-line function to xapply methods.

And at the end we use the do.call function to call rbind on every item of our lists so we can turn them into a matrix. It’s as if we called rbind providing the year and the value on every call, do.call removes the need of manually looping to do it.

And while this is all good pure R standard library programming, you don’t actually want to do this tedious work whenever you need to run an split-apply-combine operation, the plyr library comes with all this baked right inside of it so you don’t waste your time doing it.

First, let’s install it:

install.packages("plyr")

Once the installer is finished, here’s all the code we need:

require(plyr)
events_per_year <- ddply(
  data,
  c("year"),
  summarise,
  count=length(unique(event))
  )

The plyr package contains various methods that do the split-apply-combine scenario on many different kinds of inputs producing different outputs. The ddply method is one that takes a data frame and produces a data frame (that’s why it’s dd), but you could use a method that produces a list or any of the other methods available at the library.

Here we provide our data frame as the first argument, then the list of variables we want to use as the split, then how the result will be built (summarise means we want a new object to be built from scratch) and then the fields we will want included at this new data frame. For this case, all we want is the count field, the value for the field is the operation you want to apply (it does not have to be a function) and it has access to all fields at the data frame you provided, that’s why we just say event here instead of data$event.

The result of this function is a data frame with year and count fields, the same we had above with our manual operation. So, whenever you’re going to play around with these split-apply-combine operations, use plyr and be happy!

Playing around with the data and basic plotting

Given we know the information before 1993 is much different than the one available after it, let’s start by creating a subset of the data that contains events starting at 1993:

filteredData <- data[data$year >= 1993,]

This is known in R parlance as subsetting since you are creating a subset of the data based on some conditional. If you run data$year >= 1993 alone the result will be a logical vector containing one value for every row inside data with either TRUE (if the row matches the condition) or FALSE (if it does not) and when you use this vector as the row part for the [] operator every row that matches a TRUE value will be returned. You can use boolean operators like & and | when performing these comparisons as well for compound matches just like any other programming language.

Let’s group injuries and deaths per event:

healthConsequences <- ddply(
  filteredData,
  c("event"),
  summarise,
  total_deaths=sum(FATALITIES),
  total_injuries=sum(INJURIES)
  )

This gives us the full list of events and their casualty numbers. Now let’s sort this data frame per fatalities:

mostDeadly <- healthConsequences[order(-healthConsequences$total_deaths),]

Now we have a data frame of the most deadly events so far sorted in descending order. Let’s plot the numbers for the top 5 events:

barplot(
  mostDeadly[1:5,2],
  names.arg=mostDeadly[1:5,1],
  col=c("red", "green", "blue", "yellow", "orange"),
  main="Deaths by natural disasters",
  xlab="Event",
  ylab="Deaths")

Since we have one categorical and one numeric variable to use, the most direct option for a plot is a barplot, we grab the top 5 rows and print them. The code itself is self explanatory, first we provide the numeric values, then we set the names.arg to be the top 5 field names, we provide color names for the bars, titles and legends for the generated plot.

We can also do the same for the most injuries:

mostInjuries <- healthConsequences[order(-healthConsequences$total_injuries),]

Let’s look at the top 10 values:

               event total_deaths total_injuries
        TORNADO         1646          23328
            HEAT         3134           9176
           FLOOD          508           6870
      LIGHTNING          817           5232
 TROPICAL STORM          313           4113
   THUNDERSTORM          201           2452
       ICE STORM           89           1977
     FLASH FLOOD         1035           1800
WILD/FOREST FIRE           90           1606
       HIGH WIND          293           1471

And if we plot them to compare:

barplot(
  mostInjuries[1:5,3],
  names.arg=mostInjuries[1:5,1],
  col=c("red", "green", "blue", "yellow", "orange"),
  main="Injuries by natural disasters",
  xlab="Event",
  ylab="Injuries")

We’ll get:

Now start playing around with the damage data to figure out which is the most expensive type of disaster that has happened over these years, there’s a lot of interesting information you can figure out from this simple dataset.

Patience and exploration are key

The data you’ll find out there is unlikely to be clean and clearly defined, you should be able to explore it, find it’s weak points, find it’s missing details and then start your analysis. Ignoring that the data you have might be misleading is a sure path to bad and unreliable analysis and you don’t want to do be doing that, do you?

Growing your experience with community participation

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 03 Dec 2014 00:00:00 UTC

One of the most important parts of being a member in a development community is answering other people’s questions. Whether it’s over mailing lists, IRC channels or StackOverflow, there will always be someone (including me and you) asking questions over these channels and and there will be other people answering them as well.

While getting an answer for these questions is most likely pretty important for those asking them, answering the question also has it’s own benefits that often go unseen for those that are not used to participating in communities like this.

The shared experience

Most of our time is invested in programming using many different tools, from languages to IDEs and frameworks and while a lot of what we do is alike, there are many different applications being built and each of these apps has it’s own team and it’s own experiences. These experiences will lead people in different directions while deciding how the code is written, how the tools are joined together and how the final product behaves.

This leads to very different solutions for the same set of problems and makes different people, while working on the same field, developing completely different styles and experiences on what and how to build their solutions. If you’ve ever been to a busy mailing list you have seen this yourself when someone had trouble using a tool in a completely different setting than yours. I’ve seen people distributing a Rails app inside a VirtualBox VM so it could be run on any OS that supports VirtualBox.

This myriad of experiences leads to people asking stuff you don’t know or wouldn’t consider doing and this is where it gets interesting. You can now tap into their own world for a brief moment and gain experience on something you didn’t think about before but could be useful in the future.

The journey is what’s actually important

A couple months ago while browsing StackOverflow I saw this question about Scala’s DelayedInit and something clicked for me. I did read Scala in Depth a while ago but my memory about this specific section was mostly oh, I remember this was in the book, but I have no idea what it actually is so I couldn’t answer the question right away.

But hey, I could grab the book again, read it to refresh my memory and try to answer it, couldn’t I?

And that’s exactly what I did. Grabbed the book, opened up Eclipse with a Scala worksheet and started to play around with the code until I could actually understand what was going on and answer the question. The time it took me to read the section and build the example could definitely lead someone else to give a complete answer to the question and wouldn’t get the reputation points but I had already done the most important part of it, I stole the experience and learned something I didn’t know (or didn’t pay attention before) about the language.

While learning something you didn’t know before is pretty cool, exercising your find the bug and fix the bug muscles is also a pretty important part of answering other people’s questions. Take this question about very common error for Ruby programs, had I not seen this error before, I’d have to first find what’s leading to this behavior and then later propose a fix for it. A huge part of your career as a developer will be finding and fixing (your own) bugs, having a headstart finding other people’s bugs before they hit you is definitely a great way to fast track your experience.

Seeing this at someone else’s code before seeing this error at my own programs would help me fix it in the future when it happened at my own codebase much faster because I had already solved this issue on someone else’s code.

Let no opportunity be wasted

Whenever you see someone asking a question, specially something you don’t know yet, think if you can invest some time to find an answer. Could be as simple as googling the error message and surfing through the search results (as simple as it sounds if you’re experienced, people aren’t born knowing how to read stack traces or search for errors on google) or reading through the source code of a project you haven’t seen before. Even if someone else answers before you do, your journey to find an answer will teach you something new and keep your knowledge search engines running.

Getting used to finding and fixing bugs, reading through someone else’s code and problem solving skills are qualities every single programmer out there needs have and to practice them all you have to do is to help your community by answering other people’s questions.

Did you answer someone’s question today already?

Learning R - Part 1 - Basics

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 01 Dec 2014 00:00:00 UTC

This is part of a series of posts introducing the R language

R has two very distinct characteristics, one is having a horrible name that makes it hard to search for stuff about it and the other is being an incredible language (and environment) to do statistical computing and plotting. It was meant to be an GNU implementation of S, an older statistical programming language and environment developed at Bell Labs/AT&T.

To follow through this tutorial you’ll have to install R for your operating system and I would also recommend that you install R Studio which is a free GUI tool to work with R.

This tutorial assumes you already know the basics of programming.

Lists and vectors

One of the first distinctions you’ll find in R when compared to general programming languages is that you’ll be working with many values most of the time. Even when working with operations that look like a single variable, you’ll end up with a vector or a list.

Let’s look at an example (you can type the code inside an R console or RStudio to follow the steps):

value <- 10
class(value)
> [1] "numeric"
value <- c(10)
class(value)
> [1] "numeric"  
value <- c(10, 20, 30)
class(value)
> [1] "numeric"

The <- operator is the preferred way to assign variables in R (you could also use =, which has slightly different semantics, but we’ll maintain the usual style here). At the first line we assign 10 to the variable value, when we check type (using the class function), it says it’s numeric.

Later we use the the c function to create a numeric vector and when we make the same class call on it it produces numeric again. Why is that? Because almost everything in R is actually a vector of some kind.

You can also create a vector from an interval using the : operator as in:

interval <- 1:10
> [1]  1  2  3  4  5  6  7  8  9 10

Vectors created like this are always inclusive.

In R a vector is a collection of items all of the same type, so you can’t mix numbers and strings. If you want to have a collection with different types, you’d create a list using the list function:

items <- list(20, "me")
class(items)
> [1] "list"

To access an item inside a vector you use brackets, [ ], and the index of the object you would like to access. Different than other languages you might be used to, in R the first item inside a vector is at the 1 position instead of 0. Let’s look at an example:

item <- 10
item[1]
> [1] 10
items <- c(10, 40, 90)
items[3]
> [1] 90
items[10]
> [1] NA

So the indexes for vectors start at 1 and go until it’s length, if you try to access an item that doesn’t exist, you will get the special value NA, which means there is no such value for this specific vector. While related to NULL, they’re not the same and NA values are much more prevalent in R code than NULL, most of the time you’ll only see NULL values being returned in R when a function doesn’t actually return any value.

Lists are a bit different, if you try to access an element of a list with [] you’ll see something like this:

simple_list <- list("example", 20)
simple_list[1]
> [[1]]
> [1] "example"

You’re still getting a list back, not exactly what you’re looking for, in this case you want to use [[]] to get that first element, as in:

simple_list <- list("example", 20)
simple_list[[1]]
> [1] "example"

When creating lists you can also name the items so you can access them by their names instead of indexes:

named_list <- list(name="John", age=40)
named_list$name
> [1] "John"
named_list$age
> [1] 40

First you create the list providing a name for every item and then you access them using the $ operator. You can still use named_list[[1]] to get the value John here but if you’re naming stuff it wouldn’t make much sense to do that.

Also, the names don’t have to be valid R identifiers, you could do something like this:

special_names = list("Maurício Linhares" = "name")
special_names
> $`Maur\303\255cio Linhares`
> [1] "name"

special_names$"Maurício Linhares"
> [1] "name"

But I’d say there’s very little reason for you to do something like that.

Basic statistical functions

Let’s declare a small vector:

numbers <- c(4, 36, 45, 50, 75)

To calculate the mean of this vector, we do:

mean(numbers)
> [1] 42

The mean function will calculate the arithmetic mean of your vector, in our case it would be:

\frac{4 + 36 + 45 + 50 + 75}{5} = 42

Another common function you could use here is the median:

median(numbers)
> [1] 45

The median produces the number that’s right at the middle of the sorted distribution, in our case, the number right at the middle is 45. What happens if the vector is even?

even_numbers <- c(4, 36, 40, 45, 50, 75)
median(even_numbers)
> [1] 42.5

The two numbers right at the middle of the sorted vector are summed and then divided by 2. It’s important to remember that the median is always calculated from the sorted vector, if you give it an unsorted vector it will sort it and produce the right median:

numbers <- c(36, 4, 75, 50, 45)  
median(numbers)
> [1] 45

How do we decide to use `mean` or `median`?

The median is defined as a robust statistic because outliers (values that are too far away from most of your measurements) have very little effect on it, while the mean is not robust as outliers can greatly affect it’s calculation. Before deciding on which one to use, check the data you have in hand to make sure you’re picking a statistic that makes sense for the data you are working with.

Calculating the mean of an distribution full of outliers will most likely give you a weird value and using median for a distribution that has very little variance might not give you the actual center of the distribution.

Dealing with matrices

Another common data type you’ll find is the matrix. Here’s how you could use a vector to create a matrix:

m <- matrix(c(1, 2, 3, 4), nrow=2, ncol=2)
>       [,1] [,2]
> [1,]    1    3
> [2,]    2    4

Here we use the matrix function to create a 2x2 matrix. nrow is the number of rows and ncol the number of columns, we could have called the function as matrix(c(1, 2, 3, 4), 2, 2) and it would also work but whenever you’re calling a function that takes many parameters in R you’re better off naming the parameters to make sure you’re not making a mistake with the parameters order. It’s also makes it much easier to read and understand which parameter each value is supposed to be.

Also, it’s important to notice that the the matrix will be filled from the vector by column by default, so values are included in the matrix in columns, which is why our 1, 2, 3, 4 vector became the matrix it is now instead of:

     [,1] [,2]
[1,]    1    2
[2,]    3    4

If you wanted the matrix to be filled by row, you have to include the byrow parameter and set it to TRUE as in:

m <- matrix(c(1, 2, 3, 4), nrow=2, ncol=2, byrow=TRUE)
>       [,1] [,2]
> [1,]    1    2
> [2,]    3    4

You can also name the rows and columns of your matrix to make it easier to read:

row_names <- c("Male", "Female")
col_names <- c("Right-Handed", "Left-Handed")
m <- matrix(c(43, 44, 9, 4), nrow=2, ncol=2, dimnames=list(row_names, col_names))
>        Right-Handed Left-Handed
> Male             43           9
> Female           44           4

To access rows, columns and specific items inside a matrix you use the [] operator. First, access full rows:

m[1,]
> Right-Handed  Left-Handed
> 43            9

Then here’s how you access a full column:

m[,1]
> Male Female
>   43     44

And you can also access one specific item:

m[2,2]
> [1] 4

Just like vectors and lists, indexes for matrices in R start at 1.

Factors

Factors are the way we represent categorical variables in R. Categorial variables are those values that instead of being simple numeric values, are states of some variable. Imagine you have patients and you would like to separate them in groups given their current health status and the status are:

Healthy;
Recovering;
Sick;

These values aren’t numeric, but they make sense in our data set and we would like to be able to efficiently use them in our statistics. This is such a common theme in statistics that there is this special type, factor, so we can use categorical variables at our programs.

While you could just use strings for these values (a vector of strings, for instance), using factors identifies this column directly as a categorical variable for R and this leads to better defaults when using statistical analysis and inferencing methods. Factors also use less memory and are faster to process.

Most of the time you won’t be creating factors directly, you’ll be instructing the code that loads your data which of your fields are factors, but let’s see how you can create factors directly in R:

health <- c("Healthy", "Recovering", "Sick")
health_factors <- factor(health)
> [1] "Healthy"    "Recovering" "Sick"
health_factors[1]
> [1] Healthy
> Levels: Healthy Recovering Sick
health_factors[2]
> [1] Recovering
> Levels: Healthy Recovering Sick
health_factors[3]
> [1] Sick
> Levels: Healthy Recovering Sick

Data frames

Data frames are the main data type for R programs, most of the data you’ll be working with is either in this format or will be transformed to a data frame so you can easily work with it. Let’s get started with a data frame using Kaggle’s Titanic test data:

download.file("https://gist.githubusercontent.com/mauricio/f389c162731532e2dea5/raw/5f045d26890255c123ca02a98febf41b8bab085f/titanic.csv", destfile = "titanic.csv", method="curl")
titanic <- read.csv(
  "titanic.csv",
  colClasses=c(
    "integer", // passenger id
    "factor", // survived or not
    "factor", // passenger class
    "character", // name
    "factor", // sex
    "numeric", // age
    "integer", // number of spouse or siblings aboard
    "integer", // number of parents or children aboard
    "character", // ticket
    "numeric", // fare
    "factor", // cabin
    "factor" // port of embarkation - (C = Cherbourg; Q = Queenstown; S = Southampton)
    ))

Our first step here is to download the CSV file that contains the data, R already has a handy download function for that, download.file, so we just use it. It’s important to define the method parameter to curl or wget (you’ll need one of them installed at your system) as the download is using HTTPS and the default handler for file downloads isn’t capable of handling HTTPS connections. The file is downloaded to the titanic.csv file at the same directory as your current R session, you can use the getwd() function to know which directory this is and you can also call setwd("some-path-here") to change the session’s current directory.

Now that we have the file downloaded, we can use one of the many methods to turn a file into a data frame. Since this file is a CSV we’ll use the read.csv function, the first parameter is the path to the file (since the file was downloaded to the current session directory, there’s no need to set the full path) and we’re setting only one parameterm, colClasses. While the read.csv method is smart enough to figure out the types and parse this file correctly, setting the column types gives you fine grained control to the types used and also makes parsing the file much faster as the method won’t have to try to figure out the column types.

Once you have the data frame loaded, let’s check it’s summary:

summary(titanic)

PassengerId    Survived PassengerClass     Name               Sex           Age        SpouseSiblingsAboard
Min.   :  1.0   0:549    1:216          Length:891         female:314   Min.   : 0.42   Min.   :0.000
1st Qu.:223.5   1:342    2:184          Class :character   male  :577   1st Qu.:20.12   1st Qu.:0.000
Median :446.0            3:491          Mode  :character                Median :28.00   Median :0.000
Mean   :446.0                                                           Mean   :29.70   Mean   :0.523
3rd Qu.:668.5                                                           3rd Qu.:38.00   3rd Qu.:1.000
Max.   :891.0                                                           Max.   :80.00   Max.   :8.000
NA's   :177
ParentsChildrenAboard    Ticket               Fare                Cabin     Embarked
Min.   :0.0000        Length:891         Min.   :  0.00              :687    :  2
1st Qu.:0.0000        Class :character   1st Qu.:  7.91   B96 B98    :  4   C:168
Median :0.0000        Mode  :character   Median : 14.45   C23 C25 C27:  4   Q: 77
Mean   :0.3816                           Mean   : 32.20   G6         :  4   S:644
3rd Qu.:0.0000                           3rd Qu.: 31.00   C22 C26    :  3
Max.   :6.0000                           Max.   :512.33   D          :  3
(Other)    :186  

The summary function already provides some basic information about our data set, we can see, for instance, that 342 people survived while 549 died, most people were at the third class and the average fare paid was 32.20. This is just a high level view of the data so you can start digging through the data frame yourself looking at these variables later.

Something else you can do is look right at the data itself using head:

head(titanic, 3)

PassengerId Survived PassengerClass                                                Name    Sex Age
         1        0              3                             Braund, Mr. Owen Harris   male  22
         2        1              1 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female  38
         3        1              3                              Heikkinen, Miss. Laina female  26
SpouseSiblingsAboard ParentsChildrenAboard           Ticket    Fare Cabin Embarked
                  1                     0        A/5 21171  7.2500              S
                  1                     0         PC 17599 71.2833   C85        C
                  0                     0 STON/O2. 3101282  7.9250              S

Or tail:

tail(titanic, 3)

PassengerId Survived PassengerClass                                     Name    Sex Age
       889        0              3 Johnston, Miss. Catherine Helen "Carrie" female  NA
       890        1              1                    Behr, Mr. Karl Howell   male  26
       891        0              3                      Dooley, Mr. Patrick   male  32
SpouseSiblingsAboard ParentsChildrenAboard     Ticket  Fare Cabin Embarked
                  1                     2 W./C. 6607 23.45              S
                  0                     0     111369 30.00  C148        C
                  0                     0     370376  7.75              Q

These functions will print the first rows (for head) and last rows (for tail) so you can inspect a bit of your data.

And just like a matrix, you can also get specific rows, columns and everything else using the [] operator. For instance, here’s how you access the first row:

titanic[1,]

PassengerId Survived PassengerClass                    Name  Sex Age SpouseSiblingsAboard
1           1        0              3 Braund, Mr. Owen Harris male  22                    1
ParentsChildrenAboard    Ticket Fare Cabin Embarked
1                     0 A/5 21171 7.25              S

And now all items at the fourth column:

titanic[,4]

[1] "Braund, Mr. Owen Harris"
[2] "Cumings, Mrs. John Bradley (Florence Briggs Thayer)"
[3] "Heikkinen, Miss. Laina"
[4] "Futrelle, Mrs. Jacques Heath (Lily May Peel)"
[5] "Allen, Mr. William Henry"

The name of the tenth person:

titanic[10,4]
> [1] "Nasser, Mrs. Nicholas (Adele Achem)"

You can use a vector for both indexes as well. For instance, if I wanted rows from 10 to 20 I could do it like:

titanic[10:20,]

And you can also get all values from a single column using the $ operator:

titanic$Name

[1] "Braund, Mr. Owen Harris"
[2] "Cumings, Mrs. John Bradley (Florence Briggs Thayer)"
[3] "Heikkinen, Miss. Laina"
[4] "Futrelle, Mrs. Jacques Heath (Lily May Peel)"
[5] "Allen, Mr. William Henry"

The read.table docs have a full list of formats and parameters supported for reading many different file formats into data frames.

At the next part we’ll part we’ll see how we can subset and plot the data we have collected, stay tuned!

Statistics is fun

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 01 Oct 2014 00:00:00 UTC

Statistics, whether we notice it or not, is part of our everyday lives. Be it when we hear about inflation, unemployment rates, polls, information produced from statistical methods are everywhere and we use it often to make sense of the world and make decisions even when we don’t actually understand how we ended up with those values.

Many years ago (ok, 10 years ago), while in college, I had this statistics class. It wasn’t bad, the teacher was decent and the material, while not the best, wasn’t horrible either. The main issue was that I was learning to program and, unfortunately, statistics wasn’t being presented as an interesting topic that could compete with the universe of programming that was opening up for me at that time.

The information I was presented during these last 8 weeks of Data Analysis and Statistical Inference has been not only valuable in the sense of understanding the tools and basic concepts around machine learning, which was my initial goal, but also gave me tools to better understand the world around me.

The Brazilian election case

Brazil has just gone over one of it’s most competitive president elections of all time, with two sides fighting for every inch of the electorate and, as you can imagine, polls were abound. Before taking the class, I knew that the margin of error was a sign that the actual value was, most likely, inside the interval provided given the confidence level.

If the poll had a 95% confidence level, it means 95% of all polls would produce an interval with the actual value (the distinction here is important, 5% of the polls could still produce an interval that will not include the real population parameter). As a simplification, we say that the actual value will be inside the interval produced by the poll, which when 38% value and a 2% margin of error would be betweem 36% and 40%. This specific interval might not actually have the value, but for now let’s assume it does.

So, I knew what it meant but I had no idea how they got at that specific interval. Going through the classes I then learned that the margin of error for proportions is calculated like:

ME = z* \times \sqrt{\frac{p \times (1 - p)}{n}}

Where p is the probability your poll calculated and n is the number of people you interviewed and z* is a special value we pick to to decide how confident we are about this result, it would be 1.96 for a 95% confidence level.

Why is this interesting?, you might ask, well, in the case of polls for our election here in Brazil it was pretty important because before making a poll you have to register the poll with the local electoral justice department including your margin of error.

Now look back at the ME formula, can you see something weird about the statement above?

You can’t calculate the margin of error without the actual probability so we end up with a chicken and egg problem, you need the ME to register the poll but then you need to actually execute the poll to produce the ME.

What actually happens is that pollsters just include a bogus ME and hope their calculations will be actually close to that. And as you can imagine, many of the polls produced results that were incredibly different than the real election results.

The One Ring to rule them all

The course emphasis on hypothesis testing and bullshit detection (professor Mine never actually says this, but that’s what it is, seriously) has made me look at data and analysis under a very different light. There is a strong focus on making sure you understand what you are saying about the data you have and the analysis you are making so that you don’t imply causation where it can’t be implied and avoid extrapolating models to data they can’t actually work with.

As the saying goes:

Statistics is the art of torturing the numbers until they say what you want them to.

But if you actually understand how these statistics work, you might just be able to detect the torture and avoid being amazed by yet another bogus analysis. A solid understanding of the basics will help you to avoid spurious correlations and making bad decisions based on bad data and biased analysis.

And to the amusement of my old me of 10 years ago, most of this stuff happens in programming environments nowadays, even the learning. The use of R (both running on your own machine or through DataCamp) throughout the course makes the whole learning process much more interactive as you can touch the data and play around with it as much as you’d like to.

Learning statistics for real has definitely had a real impact in my life and my view of the world and even if I don’t actually apply this knowledge directly at work, the effects the critical thinking and evaluation tools I’ve been presented will.

So don’t let your previous bad experiences or prejudice against statistics hold you back, there is a whole world of numbers and data to be explored out there and it keeps growing. All you have to do to tap onto these new fountains of data is understanding statistics.

Enroll and enjoy, I’m sure you’ll have a blast at it as much as I had myself.

Data Analysis and Statistical Inference - Week 1

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 14 Sep 2014 00:00:00 UTC

A couple weeks ago I blogged about my personal experience with Coursera and now that Data Analysis and Statistical Inference has started, I decided it would be great to build small summaries of what was covered on every week of the course as I go by it.

Introduction

Here we’re presented to statistics by using the cigarette research example. Back when we started to research the effects of smoking, many people said they knew people that did smoke but felt great and while this isn’t necessarily untrue, it’s just a single variable at the whole population. That’s what we usually call anedoctal evidence and we can’t really extrapolate this to a whole population.

Going forward, another example is provided, now a research that asked:

Are consumers of certain alcohol brands more likely to end up in the emergency room with injuries?

And while the question seems to be general to the whole population, the sample selected where people that went to ER at the Johns Hopkins Hospital in Baltimore. This makes the answers produced by this research not to be valid to the whole population as the sample was taken from a very small cluster of people and from a single place, so the brands and even the way people drink in Baltimore (and of those that decided to go to the Johns Hopkins ER) coud affect the results of the research.

So we have to carefully consider what kind of answers we can actually build out of the data we have collected.

Data basics

Now we get to understand what is data, starting from what are observations and variables. The example used here comes from Google’s Transparency Report:

period_ending | country   | all_requests | req_complied | hdi
------------- | -------   | ------------ | ------------ | ------
06/30/2013    | Bahrain   | 1            | 0            | very high
06/30/2013    | Brazil    | 321          | 48           | high
06/30/2013    | Guatemala | 2            | 50           | medium
06/30/2013    | Indonesia | 3            | 67           | medium
06/30/2013    | Thailand  | 1            | 0            | high
06/30/2013    | Greece    | 4            | 25           | very high
06/30/2013    | Hong Kong | 1            | 0            | very high

An observation here is a single row of our table. A variable is one of the columns. Every dataset will be comprised of many observations of a collection of variables.

Types of variables

Variables can be of two different types, numerical or categorical.

Numerical variables are, as expected, variables where it would make sense to do math on them like summing, subtracting, calculating averages and the like. These variables are then divided into two other types, continuous or discrete.

Continuous numerical variables are usually measures of something, values that can take an infinite number of values within a range. You’ll see this in hights, weights, distances and the like where you can have any numeric value available.

Discrete numerical variables are usually counts of something, usually around a definite set of values. This could be the count of cards in a household or how many people live a household.

When deciding if a variable is discrete or continuous, think about it’s actual meaning instead of the values you can see at your observations. You could easily make continuous data look like discrete just by rounding it, but it would still be continuous.

Categorical variables are the ones that take a limited number of values. These values could be numbers, but it wouldn’t make sense to do math on them. For instance, we could define sex as a categorical variable with 1 meaning female and 0 meaning male, but it wouldn’t make sense to do math with these 1 and 0 values.

Categorical variables are also divided into two types, ordinal and regular categorical.

ordinal variables are the ones that define an intrinsic order. Imagine you’re collecting people’s responses about a restaurant and the possible values for the feedback variable are very bad, bad, average, good and very good, you can clearly see there is an order for the categories here, so this variable is ordinal categorical.

All other cases are regular categorical.

Going back to Google’s database, let’s look at what kind of variables we have:

country   | all_requests | req_complied | hdi
-------   | ------------ | ------------ | ------
Bahrain   | 1            | 0            | very high
Brazil    | 321          | 48           | high
Guatemala | 2            | 50           | medium
Indonesia | 3            | 67           | medium
Thailand  | 1            | 0            | high
Greece    | 4            | 25           | very high
Hong Kong | 1            | 0            | very high

country - regular categorical, each one is a single country name of the list of all countries in the world;
all_requests - the count of all requests a country has made to remove content from google. Since it’s a counted value, it’s a discrete numeric variable;
req_complied - this is the percentage of requests that google did comply to takedown, this variable takes any value from 0 to 100 (including floating point values), so we have a continuous numeric variable;
hdi - Human Development Index value for each country. We have a collection of values sorted from medium to very high so this one is an ordinal categorical variable.

Relationships between variables

Variables in a dataset can be related as in the value of one seems to go up or down based on the value of the other. When we find something like this, we say the variables are associated or dependent. Looking at the mtcars dataset we can see that there is a relationship between the MPG and weight variables:

In this case, we have a negative association, as while the weight goes up, the MPG goes down. If both variables go up (or down) together, we then have a positive association.

Variables can also be independent, where there is no trend we can detect between their values.

Studies

Studies are the processes by which you will collect data for statistics. They can be observational studies or experiments.

At an observational study, researchers pick samples of the population and observe them doing the object of the research. They do not interact or affect the observed samples in any way other than to collect the data. Observational studies can be retrospective, when they collect data about events that have already happened or prospective, when they collect the data as the events occur.

Due to the way observational studies happen, they can usually only declare there is an association (or not) between variables, but they can not declare a causation. This is due to the fact that these studies won’t usually control for the other variables that could possibly affect the variables that are being measured, as the study only observes what is happening.

The other type of studies are experiments. The main difference here is that researchers will randomly assign subjects to groups to measure the variables taking into account other possible variables that could affect the outcome of the experiment. Since they are controlling for the variables directly, we can declare a causation relationship between what is being measured.

Why is this difference important?

Imagine you want to know if people that do exercise frequently feel more energy. If you’re doing an observational study, you will find people that exercise frequently and peolpe who don’t and observe if the group the exercises feels more energy.

If you find out that those who exercise feel more energy you can’t directly declare practicing exercises causes this effect because they could be exercising exactly because they feel more energy due to some other cause. All you can do is declare that there is an association between these two variables.

On the other side, if you do an experiment and randomly assign people to groups where a group will exercise and another won’t and then measure, you will have people at the exercising group that were not frequently doing it before and people who did exercise frequently at the non-exercising group.

Since the experiment has more control over the variables, if now you reach the same conclusion as at the observational study, you can declare that there is a correlation between these two variables.

Census x Samples

Why do we take samples of the population when running studies instead of going for census?

First, there is the obvious cost of running a census. Finding every single person around is hard, expensive and not necessarily a good idea. People that are hard to find could be too different from the usual population around them and not give you much information.

Then there is the time it takes. Populations rarely stand still and by the time you finish the census it might not be that useful anymore due to people having moved elsewhere.

Sampling bias

So, is sampling perfect?

Not at all, when sampling a population you can be a victim of one of the many sampling biases and this will make it not be a representative part of the population you’re trying to sample.

Some of them are:

Convienience sample - imagine you’re a freshmen in college and you want to find out the average age of your fellow freshmen. To make it a quick exercise, you take a sample of your current class and that’s it. This sample you took won’t be representative of the actual college freshmen as you have only taken data from people near you. Very convenient but not valuable;
Non-response - a non-response bias happens when you do actual random sampling for the population but only a fraction of the sample actually responds and produces data. This fraction that actually did respond might not be representative of the actual population anymore and might produce unreal results;
Voluntary response - this happens when instead of going after people to collect the data, you kind of wait for them to come to you. When you expect people to take action and participate in the sample you’re getting those who voluntarily feel the need to do so and this won’t reflect the actual population you’re trying to sample (unless you’re trying to sample those to volunteer!). This is specially important if the subject causes strong reactions in people;

Sampling methods

There many different ways of sampling a population, three common ones are:

Simple random sample - this is the simplest one, just randomly pick people from the population and you should end up with a sample that represents it. While useful, you might end up in trouble due to the randomness of the selection. Imagine your population is made of 60% females and 40% males but your sample produced 60% males and 40% females, is it representative? Not really, so you need to be careful here or pick the next method;
Stratified sample - now we divide the population in well know groups (like males and females), called strata and then we do the random sampling above at each of these groups. This gives us the benefits of just doing random sampling but since we have grouped the data now it’s much more likely we will end up with a representative sample of the population;
Cluster sample - this is a simplification of stratified samples where instead of sampling from all strata you randomly select only some of those strata and then randomly pick a sample of every strata. This is just like ding the stratified one above but collecting less people from the population. This could be useful if you’re sampling the population of a city, instead of going at every single neighborhood, pick a bunch of them and collect the population out of these groups instead of having to go at each one of them;

Experimental design

Experimental design is made of four specific principles, control, randomize, replicate and block.

To control means always having a group that is not receiving the treatment so you can have a baseline to compare the effects the actual treatment has. If all you had were the main group of peolpe that are going through the treatment it wouldn’t be possible for you to figure out if there really is an actual difference between those who are using it or not. Experiments should always have a treatment group (the one that actually gets whatever is being evaluated) and a control group where they should not get the treatment.

Randomize means, as we’ve said before, sampling people in a way that avoids bias, using some of the methods we have defined already. It’s always important to make sure your experiments are properly randomized so you know your population sample actually represents the population you sampled from.

Replicate means to collect enough observations to make sure there will be enough for the experiment to have any validity. Again, this doesn’t mean you have to make a census, but you shouldn’t be using you and a friend as the only points as well. This could also mean fully replicating another experiment so you can compare your results with the results already available.

And last but not least, blocking. Also related to how we’re building the sample of the population to be at the experiment, we have to consider other variables we’re not measuring directly when building our sample. If doing an experiment of how much spice people will tolerate, you have to consider blocking for people that are used to eat spicy food or not, if you fail to block for a variable like that your experiment could lead to bogus results as people not used to spicy food will have a lower tolerance to it.

When sampling, you have to separate the subjects in blocks first and then start assigning them to the treatment and control groups.

Random sampling and random assignment

While random sampling and random assignment might look the same, they’re actually two different concepts that happen at two different parts of a study.

Random sampling, as we’ve discussed before, is the process by which you take a sample of the population avoiding bias, to produce a small group that is still representative of the population.

Random assignment on the other hand is the process where you randomly assign subjects from the sample you collected to the control and treatment groups of your study. Random assignment is important because you have to make sure that each group contains a valid and representative subset of the sample you have collected, otherwise you could get skewed results that might point to a result only because the group of people selected were more likely to be affected by the treatment.

Failing to do this could lead you to be a victim of confounding variables affecting the result.

What is a confounding variable?

Is a variable that correlates (directly or indirectly) with the variables you’re measuring. That’s why we need to do random assignments (and blocking) to our experiments, to make sure that these things are accounted for.

Now let’s look at a small table that summarizes the importance of random sampling and assignment:

                   | Random Assignment  | No Random Assignment |
-----------------  | ------------------ | -------------------- | ----------------
Random Sampling    | Causal and         | Not causal           | Generalizable
                   | generalizable      | but generalizable    |
-----------------  | ------------------ | -------------------- | ----------------
No Random          | Causal but         | Not causal           | Non-generalizable
Sampling           | not generalizable  | nor generalizable    |
------------------ | ------------------ | -------------------- | ----------------
                   | Causation          | Association          |

As you can see from the table, if you can do both random sampling and random assignment, you get the best of both worlds, you can deduce causation and generalize the results to the whole population. The problem here is that to do random sampling on a huge population is usually not possible, so you end up with a non-random sample but with random assignment.

If you don’t have random assignment (like most observational studies, as we’ve seen before), all you can derive is association (again, as we’ve seen before). It’s not possible to derive causation from observational studies because since you didn’t do random assignment there might be many other confounding variables involved in the process as well.

And as you can see, studies that don’t have random assignment and sampling are mostly useless as you can’t declare an association or generalize it for anything other than the people directly involved.

Visualizing numerical data

Scatterplots

There are many ways to visualize numerical data, one of them is the scatterplot:

Here we can see a scatterplot of GDP per capita vs life expectancy. The graph (that uses data from 2011) shows us that there is a positive association between GDP per capita and life expectancy. When building scatterplots, we will usually make the X axis contain the explanatory variable (the one we think is causing the effect) and the response variable at the Y axis (one one affected by the explanatory) so here we could say that as the GDP grows so does life expectancy.

Since this is an observational study, you already know what comes next, all we can assume here is association, it’s not possible to say that one causes the other but merely that they are positively related.

Histograms

Another way of looking at numerical data is using histograms, let’s look at a histogram of the life expectancies:

Histograms allow you to see the density of your data. Here we can clearly see that most people in the world will live from 65 to 80 years due to the way the data is skewed to the left.

Now let’s look at one that is right skewed:

And now one that is symmetric:

Other than their skewness, histograms can also have a modality, all 3 histograms above are said to be unimodal because they only have one prominent peak. If it has two prominent peaks it would be bimodal as in the example below:

And they can also be uniform (when there is no prominent peak) or multimodal when there are more than two prominent peaks. Modality is important because it shows trends in your data, if the data you collected produces a bimodal histogram it’s very likely that you’re dealing with two clusters of observations, so histograms are very important in exploratory analysis when you’re still trying to figure out the trends and relationships in the data collected.

Boxplots

A generally lesser known visualization is the boxplot. Let’s look at a GDP per capita one:

The box above is the interquartile range (IQR, where 50% of the data falls into) and it’s incredibly useful to find outliers as you can see from the dots to the right of the box. This leads us to believe there are many outliers at our GDP dataset which isn’t exactly unexpected as we know we have much more poor people in the world than rich ones, unfortunately. Also, the notch inside the box is the median of the dataset.

If we build a boxplot out the life expectancy:

And at expectancy we have much less outliers, but we can still see how the values vary there as well.

Measures of center

Measures of center are the values we use to calculate the midpoint or averages of a distribution of values. A common measure of center is the mean that is basically the arithmetic average of all values, sum them all and divide by the count of values and you will have the mean of your distribution.

Another measure is the median that is the midpoint of the distribution. If your distribution was the values [1,2,3,4,5,6,7,8,9] the median would be 5, since it is the value that divides the distribution in two collections of the same size. If we had an even number of values, the median would be the two values right in the middle divided by two.

You can also have the mode of a distribution that is the most frequent value for it. Modes are usually not very useful unless you have a very small subset of possible values that repeat a lot. You can also figure out the mode just by looking at histograms.

Measures of spread

As we measure the center of the data we also measure how the data goes from one end to the other. One common measure of spread is the range of the data. Looking at our life expectancy dataset we see that values range from 43 to 83 so the range is 40 years. While that’s a lot, it doesn’t tell us much, as we’ve seen on boxplots and other examples above, this type of calculation gives too much power to outliers and might hide the actual trends of the data.

To fix that, we have two other related measures of spread, variance and standard deviation. The formula for standard deviation is as follows:

σ = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} {(x_{i} - μ)}^{2}} .

For the variance formula just remove then square root.

So, for our expectancy dataset it would be something along the lines of:

σ = \sqrt{\frac{{(43 - 69.41)}^{2} + {(43 - 69.41)}^{2} + ... + {(83 - 69.41)}^{2}}{192}} .

Which will give us the value of 95.7827 which is correct but it’s not at the same unit as our value. What do we do? We calculate the standard deviation that is just the square root of the variance, which will be 9.786863 which means most of the values will be 9 years away from the mean that is 69.41.

Robust statistics

To decide what kind of measures of center and spread you should use, you need to look at the data you’re working with. We say that robust statistics are those measures that are not greatly affected by outliers at the dataset and non-robust as the ones that are.

Looking at the measures we’ve seen, when measuring center, the median is more robust than the mean because it’s not affected by outliers as the mean is. In a small example given the following values:

[ 1, 2, 3, 4, 5, 6 ]

Our mean is 3.5 as is the median. But if we replace the 6 with 1000:

[ 1, 2, 3, 4, 5, 1000 ]

The mean is now 169 but the median stays at 3.5. Since the mean is the arithmethic average of all values, any outlier will affect it pulling it’s value towards it’s direction while the median stays at the same place.

When measuring spread we can see the same happening at standard deviation and variance because they both use the mean to calculate their values, so they’re not very useful for values with lots of outliers. In this case you would use the interquartile range (IQR) we discussed at boxplots, as it’s less likely to be affected by outliers in the data.

So, when you need robust measures, avoid means, otherwise be happy and use them.

Data transformations

Sometimes it makes sense to transform the data collected to make it simpler to model the relationships we see in it. For instance, we might run a log10 operation at our GDP per capita data to make it smoother so we can have a clear linear relationship between GDP and expectancy. Let’s look at how the plot looks like:

So now instead of a curve, we have a line going along the values. Let’s look at a histogram for the same transformed data:

Now instead of the original right skewed histogram, we have a mostly symmetric one. While this is not representative of the data anymore, it greatly simplifies building and applying models to the data.

You could do many other transformations like squaring values to push outliers even farther, divide 1 by the actual value and so on. But you should only apply transformations when it gives your model or representation some actual value instead of just trying to run a log 10 just because the data is skewed.

Exploring categorical variables

Now that we have seen many different ways to visualize numerical data, we will do the same for categorical variables. One way to look at a single categorical variable is the barplot:

By looking at the plot we can see that the most of the people involved in the survey think it’s hard to save money. While it might look like a histogram, a barplot is different because there is no expected order for it’s bars, different than a histogram that will have each bin ordered by the range of values it represents.

Every single bin in the barplot means the amount of people that gave that same answers and it is ordered the way it is only because the variable is a ordinal categorical so it makes sense for it to be in one specific order, if the variable wasn’t ordinal there wouldn’t be way to order it anyway.

We could also use a contingency table to do the same (but I’d rather use the barplot anyway):

difficulty | counts
---------- | ------
very hard  | 231
somewhat   | 196
not very   | 58
not at all | 14
not sure   | 1
---------- | ------
total      | 500

But our actual dataset offers more data, we can segment the answers by the reported income of the sample surveyed. One way to look at this data is by using a segmented barplot:

Now we can see something else, as the income grows the difficulty to save money goes down, so we have an association between these two variables here. As people make more money, they think saving money is easier, if they make very little saving money is definitely much more complicated.

We can also use mosaic plots to get a better perspective on the percentages of answers for each group as below:

With mosaic plots we can see each group on it’s own (instead of based on the counts as barplots) and this allows us to detect the biases in each group more cleanly. For instance, if we look at the below 40k bracket and compare it with the more than 80k one it’s clearly visible that people above 80k are much more likely to think it’s somewhat difficult than very hard, while most of the 40k respondents think it’s very hard.

Introduction to statistical inference

In this section we’re presented to a case study (made in 1972) that wanted to understand if there was gender discrimination when deciding to promote people or not. The study polled 48 male bank supervisors, they were all given the same personnel file, the only difference was the gender all selected using random assignment. At the end, 35 of the 48 resumes received a promotion.

Let’s look at the data:

gender | promoted | not promoted | total
-----  | -------- | ------------ | -----
male   | 21       | 3            | 24
female | 14       | 10           | 24
------ | -------- | ------------ | -----
total  | 35       | 13           | 48

With the data collected, we have to define our two competing claims. First, there is the null hypotesis, in this case it is that there is nothing going on, gender does not affect the decision to promote a person or not. At the other side there is the alternative hypotesis that claims there is something going on and the gender does affect the decision to promote a person or not.

Now let’s start charting the provided dataset. Since both are categorical variables, we’ll start with a mosaicplot:

From the chart there is a clearly visible difference between males and females being promoted. Inside the male group, more people were promoted than not when compared with the female group, but since this is a mosaicplot (and it compares every group in separate) we don’t know if the amount of applications was the same. To figure that out we need a barplot:

From the barplot we can see that the amount of resumes presented were the same (24) but the difference in the chart is quite visible, much more women weren’t promoted than man, nearly 3 times more.

Now that we have collected and looked at the data, we must evaluate it as the evidence for our case. Could mere chance cause this difference in the experiment? If it could, then our null hypotesis would be the correct one and there wouldn’t be any gender bias when evaluating these resumes.

To figure that out, we need to simulate the probabilities of the outcome, which is reaching out a nearly 42% of non-promotions randomly picking observations from a dataset where we know 35 got promoted but 13 weren’t.

Let’s simulate this right here, we’ll have an array where values 1 to 35 will mean that the person was promoted and from 36 to 48 it wasn’t. We will randomly select numbers from this array, two groups of 24 and we’ll evaluate how the promoted and not-promoted values will get distributed. To verify the null hypotesis our simulation has to consistently produce one group having much more promotions than the other, otherwise it means that the null hypotesis is false and our alternate hypotesis has to be considered.

Use this to execute the experiment: Execute

Current shuffled collection:

Males:

Result values:

Promoted:

Not promoted:

Females:

Result values:

Promoted:

Not promoted:

As you keep clicking on Execute you’ll see that it’s really hard to reach the same distribution as the one provided by the experiment. Most of the time the values will be really close to each other instead of disparate as they are in the study, which leads us to conclude that there actually is an association of gender with getting promotions as our random simulation has made visible that reaching the study values is really hard.

And this concludes the introduction to statistical inference and simulation for this week and the first week summary. Now let’s get to week 2!

Use your MongoDB `_id` field as your created at timestamp

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 04 Sep 2014 00:00:00 UTC

From time to time I have to run queries against our MongoDB collections picking documents created over specific time ranges but not all of those collections have an index at the created_at field. This led to some of them taking too long to run even on our secondaries or the hidden replica.

This changed when I was presented to the fact that the ObjectId values produced by Mongo already include a timestamp on them!

I could just use the collection _id field to do date range queries and since the field is indexed by default I didn’t even have to care about indexing it myself.

If you’re using Mongoid, a simple way of creating these values is using the Moped::BSON::ObjectId.from_time method. Let’s look at an example that finds all items created today:

today_object_id = Moped::BSON::ObjectId.from_time(Time.now.utc.at_beginning_of_day) # 5407ab800000000000000000 for "2014-09-04 00:00:00 UTC"
items_created_today = MongoidModel.where(:_id.gte => today_object_id)

This finds all items that were created after the date you provided (at_beginning_of_day is a method that’s included by ActiveSupport at Time objects) and it should be really quick due to the index at _id that’s already there all the time.

If you have to go through a lot of data and don’t want to keep cursors alive at the server (avoid long running cursors on MongoDB) you can use this same method to build a manual cursor that loads all items for every day for 30 days and does stuff to them. Here’s how it would look like:

last_30_days = 29.times.inject([30.days.ago.utc.at_beginning_of_day]) do |acc,_|
  acc << acc.last.advance(days: 1)
  acc
end

last_30_days.each do |day|
  start_object_id = Moped::BSON::ObjectId.from_time(day)
  end_object_id = Moped::BSON::ObjectId.from_time(day.advance(days: 1))  
  items = MongoidModel.where(:_id.gte => start_object_id).where(:_id.lt => end_object_id)
  # do something with the items here
end

Here we find any items that were created at that day until the beginning of the next one. You can build any kind of date range query against ObjectId fields just by making use of the from_time method, if you can create a Time object, you can produce an ObjectId value to use it as a filter.

This also works if you’re using the main MongoDB driver, just use the BSON::ObjectId.from_time method that works exactly the same way. Give it a Time object and it will give you the ObjectId value that represents that timestamp.

And if you’d like to get an ObjectId without writing any Ruby code at all, use Steve Ridout’s ObjectId generator and you should be good to go.

Now stop indexing your created_at field :)

Make the most of Amazon SQS in Ruby for your background worker needs

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 01 Sep 2014 00:00:00 UTC

While it’s not as popular as other solutions like Resque, Sidekiq or DelayedJob at the Ruby community, using SQS to serve a background processing operation is dead simple and offers a lot of guarantees you might not get from the previous options.

The base SQS model is rather simple, you create queues and you push and pull messages from them. The actual advantages come from the fact that you don’t have to manage the queuing servers yourself (if you ever had to build an HA Redis cluster for Resque workers you know the pain already) and you can have as many producers and consumers as you would like to, there’s no limit on how many messages a queue can hold and how many requests you can make to the SQS servers, so most of the scaling pain you could have setting up a pool of background workers is gone already.

To follow through these examples you have to create an AWS account and setup your AWS keys as environment variables. Check the Ruby’s AWS SDK project for docs on the many ways you can configure your keys.

Creating queues and sending messages

Creating queues and sending messages is dead simple:

sqs = AWS::SQS.new
queue = sqs.queues.create(queue_name, visibility_timeout: 90, maximum_message_size: 262144)
queue.send_message({ file_id: "some-file-id" }.to_json)

Here we get the SQS client, access the queues collection and create a queue. With the queue in hand, we send a message in JSON, but you could use pretty much any text format for your messages, we could be sending YAML messages here and it wouldn’t make any difference for the service, only for your code that handles the messaging.

When you create a queue you can also configure it with some options, one is the maximum_message_size that defines how many bytes your messages could have. The default value is 64k but you can go up to 256k (the value has to be provided in bytes), if your message goes over this limit SQS will reject it with an error so make sure your messages don’t go over the limit you have defined.

Another common parameter is the visibility_timeout that defines how long a message stays invisible to other consumers until it becomes visible again. In SQS we have two types of messages, the available and the in flight ones. Messages that are available are the ones that can be picked up by consumers polling messages from the queue. Messages that are in flight are the ones that have been picked up by one of the consumers but have not been deleted yet. Messages stay in flight until the client that picked them up deletes it or the visibility_timeout has passed.

When configuring the visibility_timeout you should think how long you expect your background workers to take to process the message, the value you set for this field should be considerably longer than the time your workers take to process to make sure this message does not become visible again before your worker can finish processing it. This value is defined in seconds and defaults to 60 seconds.

If the queue is already created, you can pick it up by it’s name with:

sqs = AWS::SQS.new
queue = sqs.queues.named(queue_name)
queue.send_message({ file_id: "some-file-id" }.to_json)

If there is no queue by the name you provided, an AWS::SQS::Errors::NonExistentQueue error is thrown so make sure you catch this error and create the queue if there isn’t an external service creating it for you.

A better way to setup your queues is to use a cloud formation template to create them. This way your apps can focus only on pushing and polling messages instead of having to hold knowledge about how your queues are configured.

Receiving messages from the queue

Building the message receiving part of the process is simple as well, here’s an example:

sqs = AWS::SQS.new

queue = sqs.queues.named(queue_name)
queue.send_message({ file_id: "some-file-id" }.to_json)

received_message = queue.receive_message(wait_time_seconds: 20)
message = JSON.parse(received_message.body)

## do something with the message

received_message.delete

The receive_message method returns from 1 to 10 messages (defaults to 1, but the limit parameter can be set to any value between 1 to 10 to return more messages). In this case we have also declared the wait_time_seconds to be 10, which means the receive_message method will wait up to 20 seconds for a message to arrive to return in a long polling style. Setting this parameter is important to avoid busy loops where your code will be constantly making requests to pick messages when there are none to be received, this value can also be configured when creating the queue.

Now that we have the message we can parse it’s body (it’s just the same JSON we sent before), process and delete it. If you forget to delete messages they will become visible again and another worker would re-process it and the message would never be removed from the queue.

To avoid having such troubles, you can provide a block to the receive_message method that will take care of deleting the message if your block runs successfully so you don’t have to care about it. Here’s an example:

sqs = AWS::SQS.new

queue = sqs.queues.named(queue_name)
queue.send_message({ file_id: "some-file-id" }.to_json)

queue.receive_message(wait_time_seconds: 10) do |received_message|
  message = JSON.parse(received_message.body)
  ## do some processing here
end

And last but not least, you can have a fully functional “forever worker” with just a couple lines of code using the poll method:

sqs = AWS::SQS.new
queue = sqs.queues.named(queue_name)

queue.poll(wait_time_seconds: 10) do |message|
  message = JSON.parse(received_message.body)
  ## do something with the message
end

This loop will run forever processing messages until the process is killed or an exception is thrown so you might want to also wrap it around a begin/rescue block to make sure it continues to run if an error is raised.

Handling errors

While you’d have to include separate plugins or roll your own solution to retries and failure queues, SQS comes with these concepts baked in with the definition of the visibility_timeout value and redrive policies.

The workflow is simple, if one of your workers fail to process a message don’t do anything. Yes, that’s it, don’t do anything. If you use error monitoring software like Rollbar or Exceptional, send the actual message text, the message id and the message receipt_handle field to the service to track the failure. Sending the receipt_handle is extremely important here because if you don’t store it somewhere it’s impossible to delete the message if your system can’t handle it or it’s causing too many failures.

As you didn’t do anything with the message, once it reaches it’s visibility_timeout it will become visible again and one of your workers will automatically pick it up. You don’t have to build fancy retry logic to have this, it just works.

Now, what if a message can’t be processed at all?

If you look at the ReceivedMessage documentation you’ll see there’s a field called approximate_receive_count, we could include at our workers logic that checks the value of this field and if it has been received more than X times we delete the message and move it to a separate failure queue, right?

Wrong!

Half wrong, I mean. That’s exactly what we’re going to do but we won’t write any code for it since it’s already part of SQS, this feature is called redrive policy. At any SQS queue you can declare a redrive policy that takes a receive count and a destination queue, in messaging lingo these destination queues are often called dead letter queues because they receive the messages no one could handle/process, once the message has been received X times, SQS will automatically move it to the dead letter queue and you can run diagnostics and do whatever you would like to with these dead messages.

While you can declare redrive policies for your queues right from the SQS console UI or the command line, ignore both of those options and declare your queues in a cloud formation template. You’ll thank me latter.

Here’s a condensed example of how declaring a queue with a redrive policy looks like:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Creates the queues with a redrive policy",
    "Mappings": {},

    "Resources": {
        "TranscoderDeadQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "ReceiveMessageWaitTimeSeconds": 20,
                "VisibilityTimeout": 600,
                "QueueName": "transcoder_dead-production"
            }
        },
        "TranscoderQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "ReceiveMessageWaitTimeSeconds": 20,
                "VisibilityTimeout": 600,
                "QueueName": "transcoder-production",
                "RedrivePolicy": {
                    "maxReceiveCount": 10,
                    "deadLetterTargetArn": {
                        "Fn::GetAtt": [ "TranscoderDeadQueue", "Arn" ]
                    }
                }
            }
        }
    }
}

At this example, if a message at the transcoder-production queue is received 10 times, it is then sent to the transcoder_dead-queue and won’t be processed by the workers that are polling messages from the transcoder-queue anymore.

And now you have most of the error handling stuff you’d have to care about already done for you.

Declaring custom message attributes

When you want to send metadata about the message, you don’t have to pollute the message itself, you can declare custom attributes when sending the message and receivers can pick up this metadata without having to look at the message itself.

If we wanted to let consumers know the format of the text that is at the message, we could include a content_type attribute to messages and consumers could read that and select the right parser for the format. Let’s look at an example of that:

sqs = AWS::SQS.new

queue = sqs.queues.named(queue_name)
queue.send_message(
  { file_id: "some-file-id" }.to_json,
  message_attributes: {
    "content_type" => {
      "string_value" => "application/json",
      "data_type" => "String",
    }
  }
)

received_message = queue.receive_message(wait_time_seconds: 10, message_attribute_names: ['content_type'])  

message = case received_message.message_attributes["content_type"][:string_value]
when 'application/json'
  JSON.parse(message.body)
else
  raise "Invalid message format #{received_message.message_attributes["content_type"][:string_value]}"
end

The main advantage of using message attributes instead of including these fields at the message itself is that you don’t have to parse the message to be able read the attribute values so you can include information about how the message should be parsed. You could, for instance, include a field that declares if the payload is compressed or not and then decide to unpack the message, all without having to look at the message body at all.

When declaring attributes, you have to define the data_type, which can be String, Number or Binary. You can include an extra tag to one of these types as in Number.Dollars or Binary.Png so you can differentiate the various different numbers you have. The value field name has to match the type you used, if the type is string, the field is string_value, number_value for numbers and binary_value for a binary field. The attribute name is the key you declare for the hash.

To read the value, you have to ask SQS to include the attribute as well, otherwise your message won’t have any metadata. So, when calling the receive_message method include the message_attribute_names option with an array of the metadata fields you expect to find.

The hash returned is in the following format:

{ "content_type" => {
    :string_list_values => [],
    :binary_list_values => [],
    :data_type => "String",
    :string_value => "application/json"}
}

Careful with your `visibility_timeout`

Not that long ago I had a probem where the same message would become visible long before the worker was finished processing it. This made the same message become visible many times from many different workers and, as you might imagine, led to a lot of rework at our servers.

When deciding what should be your visibility_timeout be pessimistic and make sure this same timeout is reflected at your workers. Maybe right now you can’t see how that incredibly simple worker could take more than 1 minute to process the message but network delays, slow machines, too much disk IO and other problems could easily make your worker take much longer to actually perform the operation and delete the message, making it become available for other workers to pick up.

A simple solution for this (if you’re in Ruby 1.9 or newer) is to use the Timeout functionality to make sure your code gives up processing before it reaches the time limit. Still, the timeout should be lower than the visibility_timeout you defined for your queue.

Use batch processing whenever possible

We spoke about the send_message and receive_messsage methods but both of them have their own batch processing versions where you can send at most 10 messages and receive 10 messages. Whenever possible, you should group your messages in groups of 10 to send them in a batch using the batch_send method.

And when it makes sense, you should do the same for the receive_message method as well (that is also aliased to receive_messages) declaring the limit option to 10 to receive 10 messages in a row.

Whenever your worker can do his operations in batches (like indexing stuff in an full text search server) you should make use of this to cram as many updates as possible in a single request instead of forcing yourself to handle only one message at a time. A lot of servers and applications are optimized to perform operations in bulk, so you should make use of this feature to do more work using less resources.

And don’t forget to call batch_delete to delete all messages you have received as well, you don’t want them becoming visible for all the other clients, do you?

Whenever you have to send the same message to many different queues, you don’t want to use SQS directly, you want to use an SNS topic that distributes it’s messages to many SQS queues. Here’s an example:

sns = AWS::SNS.new
sqs = AWS::SQS.new

topic = sns.topics.create(topic_name)
queue = sqs.queues.create(queue_name, visibility_timeout: 90)

subscription = topic.subscribe(queue)
subscription.raw_message_delivery = true

sample_message = "this is a sample message"

topic.publish(sample_message)

received_message = queue.receive_message(wait_time_seconds: 10)

And you could have many other SQS queues subscribed to this SNS topic and receiving messages from it. Imagine you’re building a service that publishes updates to many webhooks (like GitHub!), instead of manually sending a message to each queue of each service, the services would subscribe their queues to a central SNS topic and you would only push the notifications to the SNS topic, the topic would take care of distributing the message to everyone that is subscribed for you.

Always cache the `AWS::SQS::Queue` object

A common mistake when building apps with SQS is calling sqs.queues.named(queue_name) all around instead of caching the returned object. Calling the method named(queue_name) causes you to make a request to SQS to get the queue URL to where requests are sent and build the SQS object.

Whenever possible configure the full queue URL at a config file in your app and use it to build the queue object to avoid making this unnecessary HTTP request whenever you need to push and pull messages to the queue, it’s dead simple to do it:

queue_url = "https://sqs.us-east-1.amazonaws.com/12345678/transcoder-production"
sqs = AWS::SQS.new
queue = sqs.queues[queue_url] # does not make an HTTP request

And now we have the queue object ready to send and receive messages without having to ask SQS for the queue URL.

Give it a try

SQS provides a lot of features and greatly simplifies working with background queues by removing all the queue management from your app. It’s definitely a great option if you have a lot of queues and background processing and don’t want to be responsible for managing all the databases and redis instances required to use the other tools.

Why I am doing Coursera courses

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 28 Aug 2014 00:00:00 UTC

When I finished college at the now distant year of 2006, I thought I might return there for a master degree some time in the future. I did have plans to do it a year later but decided to go down a different path that eventually made it too complicated to consider. I’d have to give up a nice and good paying job doing interesting stuff to live off some government provided stipend, doing research on something that might not even be stuff I’m passionate about.

I can’t say it was a hard decision back then, really.

As the years moved forward, I kept on learning on the job, trying different tools, technologies, paradigms. My list of half-read books kept growing and my interests floated around many different topics, never actually going deep in any of them other than what I might be doing at work for the moment. There was no actual direction, just a learn as you go process.

Then, around 2012, I happened to be using Scala at my job. It was a brand new world, full of new stuff to learn, new concepts, ways of building software that didn’t feel natural to me, so I had a lot of catching up to do. September that year, Martin Odersky would be doing a course on Scala and functional programming on Coursera and while I had already tried in vain to follow through other courses there (I’m still enrolled at the self-paced compilers course) this looked like a good reason to finally take this stuff seriously.

The course started in september and I had already planned a vacation for october (was doing the usual brazillian at disneyland ritual) so I would have a couple weeks of trouble watching the classes and doing the assignments, but it wasn’t impossible, a couple late delivery penalties and I would still be able to do everything.

Rhythm and deadlines

The very first thing that struck me was my lack of rhythm. When you’re in college you get used to setup your week around the lectures, projects and exams you’ll have. I was out of college for 6 years now and was mostly reading and studying when I felt like it. The only deadlines I had were on the job and they were pretty malleable, there was no salary penalty if we missed our deliveries by a day or two.

The course wasn’t this permissive.

I delivered the first assignment (it became available on september 17th) four hours before the september 28th, 4PM deadline. With all the two weeks available to deliver it, I left it to the very last moment at the last day to actually do it.

I was so used to this self paced rhythm of learning I’ve been doing for these past years that it was hard not to keep doing it now as well. It wasn’t as if I was doing crazy hours at work, my life was pretty normal, working from home in a small coastal city in Brazil, hanging out with my girlfriend (now wife), playing boardgames and hanging out with friends.

What I didn’t have was clear priorities and objectives. I would see a book about software architecture (like Beautiful Architectures), would buy it, read a couple chapters and then forget about it and buy another one from another completely different subject.

Don’t get me wrong, I didn’t stop learning new stuff during these years and I’ve built stuff I’m proud for, but if I had been more organized about what I wanted and the stuff I was learning I’m sure a lot of the subjects I just skimmed over in the past wouldn’t be coming back at me to learn them again right now.

Content curation

The clear path of the course, the assignments, the forums, the deadlines, they all made it a different experience that drove me right back at my college years. I didn’t have to look around for all of this stuff myself as I had been after college, someone else had curated the content and organised it in a way I could follow along.

My energy could be invested at learning alone instead of being diverted into finding the material, figuring out how I would exercise that knowledge, how I would validate that my knowledge was indeed good or if I was going at the right direction at all.

Eventually, I went from delivering at the last moment to setting up my time to watch all the material and deliver the assignments right at the day they would be posted. Doing the course was a conscious decision, no one was forcing me to watch the lectures, go through the material or complete the assignments, it was time to act like a grown up and take this seriously as well.

And while the course was mostly introductory material into functional programming, it felt like a refreshingly different approach to what I’ve been doing so far by reading books, blog posts and watching diverse screencasts on the subject.

Getting into the flow

A couple months later I signed up for the Programming Languages course by Dan Grossman to continue the flow around functional programming and programming language concepts in general.

While the method was mostly the same as the Scala one, this one had a twist that wasn’t there back in my college days, peer reviews. The assignments were automatically graded when you uploaded them, but you also had to review the work of other three students and grade them.

Looking at the way people solved the same problems I had solved was an amazing experience. You could spot different techniques, tools and thought processes you wouldn’t be exposed to if you were just submitting your work to some computer to grade.

And while there would be a small number of people trying to troll other students by zeroing their reviews (haters gona hate), the peer reviews weren’t that important to your final grade. What was important was looking at code in languages you had just learned (like Standard ML and LISP), written by people you didn’t know and figuring out what that code was supposed to do and that experience couldn’t be defined by a simple number.

The last LISP assignment, writing a simple expression interpreter, was an amazing source of different solutions and I can’t believe this works moments.

Being objective

Around the same time I also did the Principles of Reactive Programming course and decided it was time for a change of scenery.

I had a special interest in machine learning back from the beginning of my career. At my first job, I ended up implementing a simple recommendation engine using euclidean distance and there was a wealth of machine learning courses available, including the seminal Stanford’s Machine Learning course. I decided to give it a try.

The course itself says that all you need is some basic programming knowledge as you will be programming the algorithms and that’s it. The classes were great, the material is well presented and you can definitely write the algorithms and go through the assignments without much trouble if you know how to program, but I had no idea what I was doing.

It felt like a recipe book, here’s how the algorithm does it’s job, here are the variables, build it. I knew I was missing a lot of the context and the statistics involved in making these decisions so I decided to step back and reevaluate.

If I really wanted to understand this whole machine learning and data science field, I needed to build the foundations were this knowledge would be built upon and again Coursera comes to the rescue with the John Hopkins Data Science specialization, Bill Howe’s Introduction to Data Science (which is wrapping up right now) and Mine Çetinkaya-Rundel’s Data Analysis and statistical inference (which is about to start).

Having a clear objective and being able to work towards it taking these courses has really changed the way I’m looking at education right now and at the value Coursera is providing me with. If you’re having trouble making progress learning, try to fit yourself in a structured couse, with deadlines, assigments and a clear definition of what it wants you to do.

You will be amazed with what a deadline or a grade penalty can make to your procrastinating mind. Time to challenge yourself to make progress and learn new and cool stuff now.

Getting to know AWS CloudFormation and creating custom resources in Ruby

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 16 Aug 2014 00:00:00 UTC

It definitely feels weird to me nowadays that not a lot of people out there are talking about CloudFormation. Templated stacks where you can declare all your dependencies in a file and just bring a stack to life with a single upload is definitely one of the most amazing parts of running stuff on top of AWS.

At Neat we have been building our cloud services using CloudFormation for more than a year now, both new and previous stacks and the improvements in organization and even documentation are visible. You can see the dependencies, configurations, security group and access level rules, databases, cache clusters and even DNS records all configured in a single place.

You can easily change a couple parameters and have a custom environment to try out new stuff and then you can just delete it and have all associated resources deleted as well, there’s no need to hunt down all the changes you have done and delete them one by one, once the template is deleted, all resources declared there are deleted as well. It’s the safe playground where you can try new stuff and have it be cleaned up by someone else once you’re done.

How does a CloudFormation template look like?

Let’s look at a small example, an SQS queue with a dead letter queue attached to it:

{
    "AWSTemplateFormatVersion": "2010-09-09",

    "Description": "Creates our sample queues",

    "Parameters": {
        "Environment": {
            "Description": "Environment in which to manage queues",
            "Type": "String",
            "Default": "qa",
            "AllowedValues": [ "development", "qa", "staging", "production"]
        }
    },

    "Mappings": {
        "EnvironmentOptions": {
            "production": {
              "maxReceiveCount" : 10
            },
            "qa": {
              "maxReceiveCount" : 5
            },
            "development": {
              "maxReceiveCount" : 2
            },
            "staging": {
              "maxReceiveCount" : 10
            }
        }
    },

    "Resources": {
        "IndexerDeadQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "ReceiveMessageWaitTimeSeconds": 20,
                "VisibilityTimeout": 600,
                "QueueName": {
                    "Fn::Join": ["-", ["indexer_dead", {
                        "Ref": "Environment"
                    }]]
                }
            }
        },
        "IndexerQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "ReceiveMessageWaitTimeSeconds": 20,
                "VisibilityTimeout": 600,
                "QueueName": {
                    "Fn::Join": ["-", ["indexer", {
                        "Ref": "Environment"
                    }]]
                },
                "RedrivePolicy": {
                    "maxReceiveCount": {
                        "Fn::FindInMap": [
                            "EnvironmentOptions",
                            {
                                "Ref": "Environment"
                            },
                            "maxReceiveCount"
                        ]
                    },
                    "deadLetterTargetArn": {
                        "Fn::GetAtt": [ "IndexerDeadQueue", "Arn" ]
                    }
                }
            }
        }
    },

    "Outputs": {
        "IndexerQueue": {
            "Value": {
                "Fn::GetAtt": [ "IndexerQueue", "QueueName" ]
            }
        },
        "IndexerDeadLetterQueue": {
            "Value": {
                "Fn::GetAtt": [ "IndexerDeadQueue", "QueueName" ]
            }
        }
    }
}

Templates are usually made of four parts (they don’t have to be in any specific order), Parameters, Mappings, Resources and Outputs.

The Parameters are options you can set when you’re creating or updating the template. These are usually information you’d like to change when creating the stacks, a common option for us is the environment where the stack will run but you could have anything that makes sense here.

Mappings are collections of key/value pairs that you can use with the Fn::FindInMap intrinsic function (intrinsic functions are functions that can be called at the JSON templates to grab values and perform other basic operations), it’s in use here so we can declare a maxReceiveCount value depending on the environment that is being used.

The resources

Then we get to the meat of the template, the Resources. This is where you declare the AWS resources that make your stack, almost all of the available AWS services are available to be used here so you could have EC2 load balancers, auto scaling groups, RDS databases, ElastiCache clusters, DynamoDB nodes, S3 buckets and everything else, all declared at this single template.

In our case here, we’re declaring two SQS queues, let’s look at the first one:

"IndexerDeadQueue": {
    "Type": "AWS::SQS::Queue",
    "Properties": {
        "ReceiveMessageWaitTimeSeconds": 20,
        "VisibilityTimeout": 600,
        "QueueName": {
            "Fn::Join": ["-", ["indexer_dead", {
                "Ref": "Environment"
            }]]
        }
    }
}

The first thing you have is the logical resource id, this is not the actual resource name (or the queue name, in this case), it’s the id for this resource inside this CloudFormation template, this will be useful when we reference this queue at a different place at this sample template. The name has to be unique across all resources declared at this template and must be alphanumeric only ( [a-zA-Z0-9] ).

Once we get inside the resource, we have to declare it’s Type (here’s a full list of all available types) and then we get to the Properties field. Every type has it’s own properties (here are the queue properties) and you don’t have to set an actual value for all of them, you can use parameters, mappings and intrinsic functions to set the property values as well as we’re making at the QueueName field. Instead of declaring a value directly, we use the Fn::Join function to join the base name (indexer_dead) with the Environment parameter to produce the name (to access the parameter we use another function, Ref). If the Environment parameter was set to development, the final queue name would be indexer_dead-development so every environment get’s it’s own queue.

Now that we’ve seen the first, let’s look at the second resource:

"IndexerQueue": {
    "Type": "AWS::SQS::Queue",
    "Properties": {
        "ReceiveMessageWaitTimeSeconds": 20,
        "VisibilityTimeout": 600,
        "QueueName": {
            "Fn::Join": ["-", ["indexer", {
                "Ref": "Environment"
            }]]
        },
        "RedrivePolicy": {
            "maxReceiveCount": {
                "Fn::FindInMap": [
                    "EnvironmentOptions",
                    {
                        "Ref": "Environment"
                    },
                    "maxReceiveCount"
                ]
            },
            "deadLetterTargetArn": {
                "Fn::GetAtt": [ "IndexerDeadQueue", "Arn" ]
            }
        }
    }
}

Again we have an SQS queue being declared but now there is another property, RedrivePolicy. This property allows us to declare the maximum number of times a message can be received and the target queue to where this message should be sent once it reaches this limit. So, if our systems fail to process this message an specific number of times, the queue itself should give up this message and send it to the dead letter queue were we can later verify what’s going on.

To do this we use two intrinsic functions, the Fn::FindInMap function to get the maxReceiveCount value for the current environment and then the Fn::GetAtt function to get the Arn value for the dead letter queue (Arns are the unique identifier almost every AWS resource has). Every resource declares a group of properties that are available for Fn::GetAtt calls, whenever you need to reference specific values from other resources inside your CloudFormation template, check their documentation and see what values they produce when given to a Ref or Fn::GetAtt call.

Creating environment specific mappings helps you fine tune your configuration options for every environment and simplify figuring out which values go where instead of manually updating parameters whenever you create a new CloudFormation. While parameters are useful, it’s much harder to manage and see the changes made when uploading or updating CloudFormation stacks, so prefer mappings whenever possible.

Outputs - publishing interesting information about your stack

And we’re finally at the end of the template, where we declare the Outputs, let’s look at them:

"Outputs": {
    "IndexerQueue": {
        "Value": {
            "Fn::GetAtt": [ "IndexerQueue", "QueueName" ]
        }
    },
    "IndexerDeadLetterQueue": {
        "Value": {
            "Fn::GetAtt": [ "IndexerDeadQueue", "QueueName" ]
        }
    }
}

Outputs are information you publish about your stack, this could be anything, mappings, parameters, resource properties or just a hardcoded value. But the main use of this is publishing information about the resources you created so someone else can consume them.

In our example, we’re exporting the queue names for the queues we’re creating here.

Why?

Because applications that want to consume these queues don’t have to hardcode their names anymore, they can just pull the information out of the CloudFormation stack directly and use that. Let’s look at some Ruby code using the aws-sdk to do it:

cf = AWS::CloudFormation.new
stack = cf.stacks["queues-development"]
outputs = stack.outputs.inject({}) do |acc, output|
  acc[output.key] = output.value
  acc
end
=> {"IndexerQueue" => "indexer-development", "IndexerDeadLetterQueue" => "indexer_dead-development"}

And once they pull the outputs from the CloudFormation, they can grab the actual queues and start pushing and pulling messages to them. This way you could change the queue names and not care about what would happen to the apps, they would rely on the output names (IndexerQueue and IndexerDeadLetterQueue) instead of hardcoding the actual queue names there. You could even just not declare names for these queues at all and the CloudFormation would provide unique names for them for you.

CloudFormation FTW!

Let’s list the advantages first:

Single template to declare your stack (or part of it);
JSON format, fits nicely to version control tools;
Allows for parametrization and specific configurations on a per-environment basis (by using mappings with environment names);
Includes a huge collection of available resources by AWS itself;

The main disadvantage is that not all options are available. For instance, you can’t subscribe an SQS queue to an SNS topic if both are not at the same template. Also, even when you can subscribe (right at the SNS resource declaration) you can’t set parameters to it like the raw subscription that removes the SNS envelope from the message.

And this is where our cfn-bridge project comes into play. When you start building everything with CloudFormation templates, having to manually create resources feels like going back in time but since not every we would have to do it anyway, right?

Wrong!

CloudFormation has support for custom resource creation where you can declare an app that will receive notifications from an SNS topic and will handle the custom resource creation for you and that’s exactly what we did.

In our case, we needed a solution where we could send a message to a single place and have it distribuited to an unknown number of consumers. As expected, SNS was exactly the solution we were looking for, but we needed something that would allow us to subscribe SQS queues to the topic any time instead of having to declare all the subscribers at the same place as the SNS topic. Since these were different applications with their own CloudFormation stacks, we couldn’t merge them all together, we needed something that would allow us to subscribe to the topic after the fact, but there was no resource to do that.

The first option was to give it up and declare all queues together anyway, not ideal or clean, but you have to do what you have to do. After digging a bit around and finding AWS’s own Python custom resource provider we thought, why can’t we do this as well?

And there we went building our own custom provider solution and including the resource that subscribes an SQS queue to an SNS topic after the fact.

Different from AWS’s solution that’s more inclined towards getting the user to define all operations, our goal with this gem is to build a collection of useful custom resources that can be shared for the whole community using CloudFormation. While we’re at only two resources at this point, there are definitely other needs we’re seeing as we progress towards using it more and more (like IAM resources) and we hope this will be useful for others as well.

If you’re using CloudFormation and find functionality that is available at the APIs but not as resources, you’re welcome to contribute them to the project as well. The the gem’s README, fork it and send your contributions.

Quick tips for doing IO with Ruby

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 03 Aug 2014 00:00:00 UTC

Since I always have to go back to the docs to check on most of this stuff, it might just be better to keep it all indexed here so I can just open this blog post instead of going hunting this all again.

And, guess what, this could even be helpful for other people as well, right?

Temporary directories

You might have heard about Tempfile, but did you know you can create temporary directories in Ruby and it’s built in into the standard library?

It’s a bit hidden since it isn’t part of the main Dir documentation, but it’s there, you can create temporary directories and leave the standard library delete them and it’s contents for you by using Dir.mktmpdir:

require 'tmpdir'

Dir.mktmpdir("my-prefix") do |dir|
  File.open(File.join(dir,"text.txt"), 'w') { |f| f.write("this is a test") }
end

Also, always set at least a prefix for your temp folders to make sure you can spot them if they aren’t deleted or if your app crashes and doesn’t remove them for some reason, at least you’ll know which code failed to execute.

This also includes the Dir.tmpdir method that gives you the path to your operating system’s temp directory.

Handling files that fit in memory? Use IO directly

If all you want is write some text to a file, Don’t use File, use IO directly:

IO.write("/path/to/file.txt", "This is my cool text I need to write! Yay!")

This opens the file at /path/to/file.txt, writes the text to it and closes the file. Can’t get much simpler than this.

And just as simple is reading a file:

contents = IO.read("/path/to/file.txt")

This reads the file contents and returns it.

Tempfiles and StringIO

Just like we have temp directories, we also have two classes that can be used as temporary file objects, Tempfile and StringIO. Deciding two use one or the other is rather simple, if your data fits in memory and you don’t care very much about paths, just being able to read and write to an IO like object, StringIO is for you, if you need path-like behavior or if your data doesn’t fit in memory, Tempfile should be the option.

Since they both are IO-like objects you can read and write to them and send them whenever the code expects to receive an IO object. The advantage is that both objects will be cleaned up by the environment once they are garbage collected (but you better play save and unlink tempfiles to avoid having too many file handles open).

In general, prefer StringIO and when you really have to use Tempfile assume it’s just like any other file and make sure you close and unlink the file as soon as you’re done with it.

Use `File.join` instead of manual string concatenation

While the File.join documentation declares the method as simply appending File::SEPARATOR for every item given, the actual implementation does much more than just that and your simple Array.join call won’t be the same as what’s being done there.

Whenever you need to build an actual path, remember to always use File.join:

path = File.join("Users", "mauricio", "projects", "ruby")

Accessing files relative to the current Ruby file

A common problem we see in Ruby code, specially when you’re building gems or writing tests is having to load a file that’s somewhere at your project path, but you obviously can’t set a full path for it as you want it to be usable out of your own machine, you need a relative path for it. A very simple way to do this is to use the __FILE__ special variable.

Let’s look at an example file system structure:

- root
  - lib
    - my_gem
      - operation.rb
  - config
    - items.yml

So, if you’re at operation.rb, you can access items.yml with:

my_gem_directory = File.dir(__FILE__)
File.join(my_gem_directory, "..", "..", "config", "items.yml")

This is basically saying:

Give me the directory where operation.rb is;
Go up a directory (the ..);
Go up another directory;
Now that you’re at root, give me the file config/items.yml;

So you can use the __FILE__ variable as the relative path to load files you know are available at your current file sytem.

If you’re using Ruby 2.x you can also remove the File.dirname(__FILE__) and just use __dir__, as pointed out by brianauton.

Avoid `File.open` without a block

One of the main advantages of using a language with closures is how simple it is to pass code around to be executed by someone else and a very common use case for this is resource management. While in some languages you have to write a huge amount of boileplate to safetly write to a file and not leak the file handle, in Ruby all you have to do is:

File.open("some-file.txt", "w") do |f|
  f.write("this is some text\n")
  f.write("and some more text")
end

The code above will open the file for writing, execute the block setting the actual File object at the f variable and once the code is finished it will flush and close the file handle, making sure I don’t have to care about this.

Whenever doing file operations, always use the block style for open, avoid doing stuff like:

file = File.open("some-file.ext", "w")
file.write("hey, this is bad!")
file.write("where's the exception handling code?")
file.close

While this code might look correct, the lack of exception handling would make the file handle leak and the process running this code could eventually crash with the OS complaining it had too many files open.

We could include the exception handling code and make sure it behaves just like the File.open that takes a block, but why should we? We already have a correct and simpler solution available, don’t reinvent the wheel, just use File.open with blocks and let the Ruby standard library do it’s job.

Prefer `Pathname` for file path and metadata operations

Pathname functions as a nicer interface to Ruby’s path operations and you’re better off getting used to it from now on whenever you need to do stuff with file names as in:

Creating it:

require 'pathname'
path = Pathname.new("README.markdown")

Getting the file extension:

puts path.extname
 => ".markdown"

Expanding the path:

full_path = path.expand_path
 => #<Pathname:/Users/mauricio/projects/ruby/mauricio.github.com/README.markdown>

Getting the directory the file is in:

puts full_path.dirname
 => #<Pathname:/Users/mauricio/projects/ruby/mauricio.github.com>

And what’s really important here is that most of these operations will return a Pathname object instead of String so you can easily chain a sequence of calls all operating on file/directory metadata and they will all function as expected.

`FileUtils` probably already has what you’re looking for

If you’re trying to do something that’s not available at Pathname, File and Dir, what you’re looking for is probably defined at FileUtils.

Many of the operations you’d usually have to manually dive down into a tree of files and directories (like chowning a directory and it’s children) are already defined as single method calls at FileUtils and you should just go there, find the method and call it instead of manually writing code to recurse over the trees and calling methods.

Mutability strikes again

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 27 Jul 2014 00:00:00 UTC

Full source for this example.

While refactoring code and running specs to validate the code was still working, I decided it was also a good moment to refactor the tests as well.

These tests were pretty old and didn’t follow some of the current practices for our test suite (use factories, rspec’s expect syntax and the like) and it was a good opportunity to move object creation logic to a single place, since the objects under test are really important for this specific app and these factories would replace a truckload of copied and pasted code.

The code went from being the full creation declared in many different before blocks like:

before do
  @user = User.create(
    first_name: "John",
    last_name: "Doe",
    options: { hometown: "Tokyo" }
  )
end
# various matchers

To be a single factory:

FactoryGirl.define do

  factory :user do
    first_name "John"
    last_name  "Doe"
    options hometown: "Tokyo"
  end

end

That was simply used with a let block at the spec:

describe User do

  let(:user) { create(:user) }

  it 'should allow setting more options' do
    user.options[:timezone] = "UTC"
    expect(user.options).to eq(hometown: "Tokyo", timezone: "UTC")
  end

  it 'should allow setting yet another option' do
    user.options[:state] = "The Shire"
    expect(user.options).to eq(hometown: "Tokyo", state: "The Shire")
  end

end

So far, so good, running specs one by one shows everything works after the spec refactoring. When I finally run all specs together, surprise, specs start to fail with this weird error:

1) User should allow setting yet another option
   Failure/Error: expect(user.options).to eq(hometown: "Tokyo", state: "The Shire")

     expected: {:hometown=>"Tokyo", :state=>"The Shire"}
          got: {:hometown=>"Tokyo", :timezone=>"UTC", :state=>"The Shire"}

     (compared using ==)

     Diff:
     @@ -1,3 +1,4 @@
      :hometown => "Tokyo",
      :state => "The Shire",
     +:timezone => "UTC",

How come this spec has the timezone field that was set at the other spec? Each spec get’s it’s own user reference created by the factory since we’re using the let block. There’s no way the same user would be reused by both specs.

So, what’s wrong here?

It had to me something I did. Before the refactoring, all specs were green, only after I refactored the user creation code to to live at the factory this started happening, my changes are causing this weird behavior.

Mutable objects at immutable factories

Since running specs one by one did work but running them all together didn’t (one spec was seeing state from the other) I had a shared state issue somewhere. Where could that be? It could be where it was being used or where it was being set.

Then I stared at the factory code again:

FactoryGirl.define do

  factory :user do
    first_name "John"
    last_name  "Doe"
    options hometown: "Tokyo"
  end

end

And BAM. There it was.

options hometown: "Tokyo"

The factory itself is immutable, it never changes after it was built when the code runs and this might have given me the impression that the values I declared there were immutable as well but they’re not.

To make it more visible, the factory could be written like this:

OPTIONS = {hometown: "Tokyo"}
FIRST_NAME = "John"
LAST_NAME = "Doe"

FactoryGirl.define do

  factory :user do
    first_name FIRST_NAME
    last_name  LAST_NAME
    options OPTIONS
  end

end

Now the issue is pretty visible, the hash given to the options attribute is created only once (when the factory is built) and it’s reused for all objects that are created out of that factory. So whenever I did a change at this hash it would be visible by all the other objects that were created from it as well.

And this isn’t just for hashes, any mutable object you declare at your factories, like strings (remember, strings are mutable in Ruby), arrays and the like could suffer from exactly the same effect.

For instance, to make it fail with strings you could use something like this:

it 'should change the full name if first name is changed' do
  user.first_name << "ny"
  expect(user.full_name).to eq("Johnny Doe")
end

it 'should change the full name of the last name is changed' do
  user.last_name << "rn"
  expect(user.full_name).to eq("John Doern")
end

Since the first spec mutates the actual string object, the second one will see Johnny Doern as a result. If you run the second one first, the first one will fail because the name will be Johny Doern instead of the expeted John Doe.

Expect the worst by default

Now when declaring factories for your objects and setting mutable values, it’s definitely simpler to go for the always create a new object solution, your factory doesn’t even have to change that much:

FactoryGirl.define do

  factory :user do
    first_name { "John" }
    last_name  { "Doe" }
    options do
      { hometown: "Tokyo" }
    end
  end

end

Using blocks for computing the values will guarantee that each object will get it’s own set of mutable objects and they will never be reused across different specs.

And with this the test suite runs and shows all tests to be green.

So, when using factories (and even when writing code in general):

Avoid using mutable values if possible;
If they can’t be avoided, replace them fully instead of mutating in place (if I had used user.options = options.merge(timezone: "UTC")) the code would not break the way it did);
If you have to mutate them in place, make sure the factories are always creating new values for every run;

After wasting some of my day trying to figure this out, I have definitely learned yet another lesson as to why I shouldn’t be mutating stuff.

Scala, promises, futures, Netty and Memcached get together to have monads

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 01 May 2014 00:00:00 UTC

Monad all the things!

Continuing our collection of articles about building and using common Scala features, it’s time to talk about the much hyped Futures and Promises. If you look around any community that has embraced async programming (like the JavaScript community) you will see people going crazy about promises. They are, actually, a very simple concept and what most other people will call a Promise is a Future in Scala (and in Java as well). But let’s get started on this, right?

If you haven’t seen the other pieces of this collection of articles, you might want to check them out:

And as usual, the source code for this is at GitHub.

I `Promise[T]` you a good (or bad, I don’t know yet) `Future[T]`

As I said before, what people call promises out there are the futures for us in Scala. Here’s what a Promise[T] is:

Promise is an object which can be completed with a value or failed with an exception.

So what we have to build here is something that can be completed with a success or exception. Can you think of something we have that’s exactly that? Oh yeah, it’s a Try[T]. So, a Promise[T] will be something akin to a Try[T], we’ll see where it becomes different in a minute.

Let’s look at our first stab at implementing a Promise[T]:

package async

import scala.util.{Failure, Success, Try}

object Promise {
  def apply[T]() = new Promise[T]()
}

class Promise[T] {

  @volatile private var result : Try[T] = null

  def isCompleted : Boolean = result != null

  def value : Try[T] = {
    if (this.isCompleted) {
      result
    } else {
      throw new IllegalStateException("this promise is not completed yet")
    }
  }

  def complete(result : Try[T]) : this.type = {
    if ( !this.tryComplete(result) ) {
      throw new IllegalStateException("promise already completed")
    }

    this
  }

  def tryComplete( result : Try[T] ) : Boolean = {
    if ( result == null ) {
      throw new IllegalArgumentException("result can't be null")
    }

    synchronized {
      if ( isCompleted ) {
        false
      } else {
        this.result = result
        true
      }
    }
  }

  def success( value : T ) : this.type = complete(Success(value))
  def trySuccess( value : T ) : Boolean = tryComplete(Success(value))

  def failure(exception : Throwable) : this.type = complete(Failure(exception))
  def tryFailure(exception : Throwable) : Boolean = tryComplete(Failure(exception))

}

The implementation is rather simple, a single instance variable (marked as volatile since we will possibly have many threads checking it) and the tryComplete method where all the action happens. When you have an actual value to complete the promise, you call tryComplete or any of the other utility methods and they will complete the promise if possible.

The completing logic is quite simple. If it isn’t null, set the value and return true signaling that this is the first time someone has tried to complete this promise. Otherwise, return false and don’t do anything. The whole action is wrapped in a synchronized block to make sure many threads trying to complete the same promise won’t leave the promise itself in a bad state.

The other methods are mostly there as validations or shortcuts to our implementation, as is the complete method that can’t be called twice. You might be wondering what the this.type return types mean, well, they mean what you’re thinking already, if you subclass this Promise[T] whenever there is a this.type the type will be your subclassed type and not Promise[T] alone. So you could have your own special promise and it’s type would be visible for callers of these methods as well.

The value method isn’t actually part of the actual Scala’s Promise[T] implementation, it’s there just to simplify our tests at this moment. Here’s how they are looking so far:

package async

import org.specs2.mutable.Specification
import scala.util.{Failure, Try}

class PromiseSpecification extends Specification {

  "promise" should {

    "complete with a value" in {
      val promise = Promise[String]()
      promise.complete(Try("some-value"))
      promise.isCompleted must beTrue
      promise.value.get === "some-value"
    }

    "complete with an error" in {
      val promise = Promise[String]()
      promise.complete(Failure(new Exception()))
      promise.isCompleted must beTrue
      promise.value.isFailure must beTrue
    }

    "tryComplete called many times does not complete twice" in {
      val promise = Promise[String]()
      promise.tryComplete(Try("some-value")) must beTrue
      promise.tryComplete(Try("some-other-value")) must beFalse
      promise.isCompleted must beTrue
      promise.value.get === "some-value"
    }

    "raise an error if completed more than once" in {
      val promise = Promise[String]()
      promise.complete(Try("some-value"))
      promise.complete(Try("some-other-value")) must throwA[IllegalStateException]
    }

    "should not accept null as a value" in {
      val promise = Promise[String]()
      promise.complete(null) must throwA[IllegalArgumentException]
    }

  }

}

Not much to see here so far, we have a simple implementation and a bunch of simple tests. Where’s the magic?

Preparing for the `Future[T]`

Well, as I said before, what people out there call promise for us it will be Future[T]. Our promise object produces a Future[T] object that can be given to other people to work on it’s future result. When you think about it, a Promise[T] is something you use internally to produce a Future[T], clients of your code won’t ever see the promise, all they will see is the Future[T] that is managed by this promise. In short, futures are where all the fun happens in our async code.

Here’s our Future[T]:

trait Future[T] {

  def isCompleted: Boolean

  def value: Option[Try[T]]

  def flatMap[S](f: (T) => Future[S])(implicit executor: ExecutionContext): Future[S]
  def map[S](f: (T) => S)(implicit executor: ExecutionContext): Future[S]

  def foreach[U](f: (T) => U)(implicit executor: ExecutionContext): Unit = map(f)

  def onComplete[U](f: (Try[T]) => U)(implicit executor: ExecutionContext): Unit

  def onFailure[U](pf: PartialFunction[Throwable, U])(implicit executor: ExecutionContext) = onComplete {
    case Failure(e) => pf.apply(e)
    case _ =>
  }


  def onSuccess[U](pf: PartialFunction[T, U])(implicit executor: ExecutionContext): Unit = onComplete {
    case Success(v) => pf.apply(v)
    case _ =>
  }

}

Yes, it’s a trait!

We don’t want people knowing what the actual future looks like, since we might have many implementations for it, so all interactions will be with something that implements Future[T].

Looking at the methods, the first 3 are our old friends, flatMap, map and foreach. But there’s something different here, they also take an ExecutionContext as a parameter. What is that?

Since we’re working on asynchronous code, we don’t actually know when our code will be executed. Different from the usual collection where the flatMap would happen right away, here it will happen sometime in the future when this future is actually completed. The ExecutionContext here serves as a way for us to tell where the flatMap will be executed, this is important because the original code that signals this future to complete itself shouldn’t care if this operation is fast or takes forever, so it must provide it’s own execution context (as if it was a thread pool) to execute itself.

Also, the ExecutionContext parameter is given as a separate parameter list:

(implicit executor: ExecutionContext)

Why is that?

Because we want to allow clients of our code to simplify the way they interact with promises. Scala allows you to declare many parameter list declarations so you can curry your functions, call a function with less parameters than it actually takes and producing a function that takes only the missing parameters. In this case, we’re not interested in currying the function, but in taking the ExecutionContext from the implicit scope. The implicit scope is a special scope in Scala where you can put variables that will be filled in by the compiler instead of yourself.

This will simplify our interactions with Future[T] objects because we will be able to declare a single ExecutionContext in the implicit scope and the compiler will pick it and set it for all calls of these methods. For us, it will be as if this parameter doesn’t even exist, but we could send in a different value if we wanted.

The onComplete is the most important method here, since it is the one that takes your code and runs it on all cases, both success or failure. All other methods revolve around using onComplete to do their jobs. We could even fully implement this trait here and leave only onComplete, value and isCompleted as an abstract methods, but we’ll leave the implementation to our concrete future below.

onFailure and onSuccess are mostly simplifications for our code if we don’t care about an specific outcome, they just do the pattern matching for us so we can give the code a partial function.

Other than that, we have isCompleted and value that will let us now when the future has been completed and what value it holds.

Now let’s get to implementing our DefaultFuture[T]:

package async

import scala.util.{Failure, Success, Try}
import scala.concurrent.ExecutionContext
import scala.collection.mutable.ArrayBuffer
import scala.util.control.NonFatal

class DefaultFuture[T] extends Future[T] {

  class FutureCallback( val function : (Try[T]) => Any, val context : ExecutionContext )

  @volatile private var result : Try[T] = null
  private val callbacks = new ArrayBuffer[FutureCallback]()

  def isCompleted: Boolean = result != null

  def value: Option[Try[T]] = if (this.isCompleted) {
    Some(result)
  } else {
    None
  }

  def complete(value : Try[T]) {
    if (value == null) {
      throw new IllegalArgumentException("A future can't be completed with null")
    }

    synchronized {
      if ( !this.isCompleted ) {
        result = value
        fireCallbacks()
      }
    }
  }

  override def onComplete[U](f: (Try[T]) => U)(implicit executor: ExecutionContext): Unit = {
    val callback = new FutureCallback(f, executor)
    this.synchronized {
      if ( this.isCompleted ) {
        fireCallback(callback)
      } else {
        callbacks += callback
      }
    }
  }

  private def fireCallbacks() {
    callbacks.foreach(fireCallback)
    callbacks.clear()
  }

  private def fireCallback( callback : FutureCallback) {
    callback.context.execute(new Runnable {
      def run() = callback.function(result)
    })
  }

  def flatMap[S](f: (T) => Future[S])(implicit executor: ExecutionContext): Future[S] = {
    val p = Promise[S]()
    onComplete {
      case Success(v) => try {
        f(v).onComplete(p.complete)
      } catch {
        case NonFatal(e) => p.failure(e)
      }
      case Failure(e) => p.failure(e)
    }
    p.future
  }

  def map[S](f: (T) => S)(implicit executor: ExecutionContext): Future[S] = {
    val p = Promise[S]()
    onComplete { v => p complete (v map f) }
    p.future
  }  

}

Wow, now that’s a lot of code!

Our DefaultFuture[T] implementation starts with two variables, the result (as with Promise[T]) and the callbacks collection. This collection keeps all objects that are sent to onComplete, mapping each function received with it’s ExecutionContext. This is important because each callback could use it’s own execution context and we have to make sure we are executing the callbacks at the correct context.

Now we get to isCompleted and value, there isn’t much to see here, the method return types and implementations are simple enough for us to understand what’s going on.

The magic starts at complete. Here we have the code that completes this future with a value, as with Promise[T], the future has to be completed with a Try[T] object and it won’t take null as a value. Everything happens inside a synchronized block because we must make sure no changes happen to the callbacks collection before we are able to fire events and clear it.

The fireCallbacks method will execute each callback in it’s own execution context and then clear the callbacks collection. Clearing the collection in this case is necessary because we could end up with a cycle of futures and promises pointing to each other and causing GC woes, so we just clear them all as soon as we can. And we have a separate fireCallback method because it will be reused at our onComplete as well.

Going down a bit we arrive at onComplete, the main entry point for clients of our Future[T] object. The implementation is dead simple, we create a future callback object, enter a synchronized block (to make sure we don’t conflict with complete) and if we’re not completed yet, we add the callback to our collection of callbacks, if we are completed, we execute the callback right away.

Now, can you see any complications about this? No? It’s because there aren’t. The idea is simple and straightforward, the actual implementation at Scala’s standard library is more complicated because they have to make sure it works in all cases and use as little resources as possible, but the core idea is what we have here, a simple object that can be completed and that will fire events once something arrives.

What about flatMap and map?

Oh, they’re cool as well but they’re cheating. As you can see, they create a promise and return that promise’s future to callers. What they actually do is that they register a callback on the current future and they will execute their given operation once the current future completes. We can see there is a really strong relationship between futures and promises here.

While map is self explanatory, flatMap is a bit different. Why is that? Because the function we take for flatMap has to return a future so we need to evaluate the function, grab the future and then add a callback on it for when it ends. Since we don’t have the Try[T] protection as we have in map when we do (v map f) we also have to handle the case where the function given fails to produce a Future[T] and raises an exception, we have to forward that exception to the future we returned to callers of our code (just as we did on Try[T] on part 4).

Our Promise[T] object has also changed a bit, let’s see where it did change:

class Promise[T] {
	// other implementations

  private val internalFuture = new DefaultFuture[T]()
  def future : Future[T] = internalFuture

  def tryComplete( result : Try[T] ) : Boolean = {
    if ( result == null ) {
      throw new IllegalArgumentException("result can't be null")
    }

    synchronized {
      if ( isCompleted ) {
        false
      } else {
        this.result = result
        this.internalFuture.complete(result) // firing the future callbacks
        true
      }
    }
  }

}

We have included a DefaultFuture object to our promises so they can give this to clients and we have also changed the tryComplete method to also complete the future and fire it’s callbacks when a promise is completed. Now the code we saw at DefaultFuture is definitely correct.

And how do we use it? Well, let’s look at the specs:

package async

import org.specs2.mutable.Specification
import scala.collection.mutable.ArrayBuffer
import scala.util.{Failure, Success}

class DefaultFutureSpecification extends Specification {

  implicit val executionContext = CurrentThreadExecutionContext

  "default future" should {

    "correctly execute the callbacks once completed" in {
      val future = new DefaultFuture[String]()
      val items = new ArrayBuffer[String]()

      future.onSuccess { case value => items += value }
      future.complete(Success("some-value"))

      items === List("some-value")
      future.isCompleted must beTrue
    }

    "correctly execute the callback right away after completion" in {
      val future = new DefaultFuture[String]()
      val items = new ArrayBuffer[String]()

      future.complete(Success("some-value"))
      future.onSuccess { case value => items += value }

      items === List("some-value")
    }

    "correctly execute the many callbacks registered" in {
      val future = new DefaultFuture[String]()
      val items = new ArrayBuffer[String]()

      future.onSuccess { case value => items += value }
      future.onSuccess { case value => items += value }
      future.onSuccess { case value => items += value }
      future.complete(Success("some-value"))

      items === List("some-value", "some-value", "some-value")
    }

    "map the value into something else" in {
      val future = new DefaultFuture[String]()

      val mapped = future.map(s => s.toInt)

      future.complete(Success("1"))

      mapped.value.get.get === 1
    }

    "flatMap the future into another future" in {
      val future = new DefaultFuture[String]()
      val otherFuture = new DefaultFuture[String]()

      val result = for ( first <- future;
                         second <- otherFuture )
                      yield first.toInt + second.toInt

      future.complete(Success("5"))
      otherFuture.complete(Success("3"))

      result.value.get.get === 8
    }

    "should fail callbacks correctly" in {
      val future = new DefaultFuture[String]()
      val exception = new Exception()
      var caughtException : Throwable = null

      future.onFailure { case f =>
        caughtException = f
      }

      future.complete(Failure(exception))

      caughtException === exception
    }

    "should fail right away if failed already" in {
      val future = new DefaultFuture[String]()
      val exception = new Exception()
      var caughtException : Throwable = null

      future.complete(Failure(exception))

      future.onFailure { case f =>
        caughtException = f
      }

      caughtException === exception
    }

    "should return none when not completed" in {
      val future = new DefaultFuture[String]()
      future.value === None
    }

  }

}

As you can see from the spec, we never send in the ExecutionContext parameter we declared in all these methods. As we said above, the compiler will search this parameter in the implicit context for this class before asking me for a value and I have set the value there with this line:

implicit val executionContext = CurrentThreadExecutionContext

This sets an execution context to the implicit context and the compiler happily uses this value all over the place whenever it encounters a method call that requires an execution context with the implicit keyword.

And our execution context implementation is dead simple as well:

package async

import scala.concurrent.ExecutionContext

object CurrentThreadExecutionContext extends ExecutionContext {

  def execute(runnable: Runnable): Unit = runnable.run()
  def reportFailure(t: Throwable): Unit = t.printStackTrace()

}

Just execute whatever you take at the current thread. Not perfect, but simplifies our testing a lot since actually testing async calls would make the tests much more brittle.

Netty comes to the show

Building futures and promises and using them like we did is ok, but if really want to understand how to use them we have to use them in a real async situation and that’s where Netty and Memcached arrive to help us, we will build a dead simple Memcached client with Netty and see how we can use promises and futures to build our APIs.

Let’s look at the operations we want to have declaring a trait for our client:

trait Client {

  def set(key: String, bytes: Array[Byte], flags: Int = 0, expiration: Int = 0): Future[StatusResponse]
  def get(key: String): Future[GetResponse]
  def delete(key: String) : Future[StatusResponse]
  def connect(): Future[Client]
  def close(): Future[Client]

}

These are the basic operations we need, connect, disconnect, set, get and delete. You could easily build all other operations from the codebase we’ll build, but for this example these are enough. The interesting fact about all this is that none of these methods return an actual value, they all return Future[T] objects because this client is async, it won’t block until the memcached server has produced a value so we can’t tell you there is a value but that there will be a value at some point in the future.

Before we dig into the actual networking code, we have to define what our messages will look like. Our client won’t really know much about the binary protocol used to communicate to memcached, all this will be hidden inside our encoders and decoders, all it will know are the high level messages we will encode and decode to/from the binary protocol.

First, we have a couple constants:

package memcached.netty.messages

object Keys {
  // magic
  val RequestKey = 0x80
  val ResponseKey = 0x81

  // requests

  final val Get = 0x00
  final val Set = 0x01
  final val Delete = 0x04
}

These are just a bunch of constants we use when building messages with memcached, the first two are used as markers so network inspection tools can know if it’s a client sending a message to the server or the server sending a response to a client. The other three constants are used to identify which command was sent to the server.

Now let’s look at the messages client can send to servers:

package memcached.netty.messages

sealed abstract class ClientRequest( val code : Int )

class SetRequest( val key : String, val value : Array[Byte], val flags : Int = 0, val expiration : Int = 0 )
  extends ClientRequest(Keys.Set)

class GetRequest( val key : String )
  extends ClientRequest(Keys.Get)

class DeleteRequest( val key : String )
  extends ClientRequest(Keys.Delete)

For every method we have on our client, there’s a message we can use. This is also true for how the binary protocol itself is modeled. Ideally, you would have a class for every message your protocol defines.

Now let’s look at the possible server responses:

package memcached.netty.messages

object ServerResponse {
  final val Ok = 0x0000
  final val NotFound = 0x0001
  final val Exists = 0x0002
  final val ItemNotStored = 0x0005

  final val ValueTooLarge = 0x0003
  final val InvalidArguments = 0x0004
  final val IncrementDecrementNonNumeric = 0x0006
  final val Unknown = 0x0081
  final val OutOfMemory = 0x0082
}

sealed abstract class ServerResponse(
  val command: Int,
  val status: Int,
  val opaque: Int,
  val cas: Long) {

  import ServerResponse._

  def isError: Boolean = status match {
    case Ok | NotFound | Exists | ItemNotStored => false
    case _ => true
  }

}

class StatusResponse(command: Int, status: Int, opaque: Int, cas: Long, val body: Option[String] = None)
  extends ServerResponse(command, status, opaque, cas)

class GetResponse(val value: Option[Array[Byte]], status: Int, val flags: Int, opaque: Int, cas: Long)
  extends ServerResponse(Keys.Get, status, opaque, cas)

In the case of server responses, we have a bit less diversity. We start with a collection of constants, symbolizing the possible status codes we could receive from memcached. The ServerResponse the fields we will always have when we get a memcached response, status, command, CAS and opaque. These fields are part of all responses you will receive from the server.

Here we also define what’s going to be an error for us. It might be weird to think that anything other than Ok are not errors, but NotFound, Exists and ItemNotStored are all expected responses when you’re talking to memcached and clients should handle them, these are not exceptional cases, they are all natural and will happen when you’re talking to the server. On the other hand, status like ValueTooLarge are not expected and will cause the client to throw an exception, clients should correctly abide by memcached requirements when sending messages to it.

Moving on, we have two subclasses for our ServerResponse object. StatusResponse is the catch all case. Most of the time the only response you will get from memcached is a status code about how the operation was executed (or not executed), this is, by far, the most common response we will see.

The other subclass is GetResponse, which is what we receive when we execute a GET request on memcached. This one is different because we have the value that is possibly stored there and we also have flags, opaque and CAS fields that are part of this response.

Encoding messages

So far, so good, we have the client requests and the server responses modeled, now it’s time to start writing Netty code. Let’s start with the request encoder:

package memcached.netty

import io.netty.handler.codec.MessageToByteEncoder
import io.netty.channel.ChannelHandlerContext
import io.netty.buffer.ByteBuf
import memcached.netty.messages._
import scala.annotation.switch
import io.netty.util.CharsetUtil
import memcached.netty.messages.SetRequest
import memcached.netty.messages.GetRequest

class MemcachedEncoder extends MessageToByteEncoder[ClientRequest] {

  def encode(ctx: ChannelHandlerContext, msg: ClientRequest, out: ByteBuf) {
    (msg.code: @switch) match {
      case Keys.Set => encodeSet(out, msg.asInstanceOf[SetRequest])
      case Keys.Get => encodeGet(out, msg.asInstanceOf[GetRequest])
      case Keys.Delete => encodeDelete(out, msg.asInstanceOf[DeleteRequest])
      case _ => throw new UnknownRequestException(msg)
    }
  }

  def encodeSet(buffer : ByteBuf, set: SetRequest) {
    val key = set.key.getBytes(CharsetUtil.US_ASCII)
    buffer
      .writeByte(Keys.RequestKey)
      .writeByte(Keys.Set)
      .writeShort(key.size)
      .writeByte(8) // extras length
      .writeByte(0) // data type
      .writeShort(0) // reserved
      .writeInt(key.size + 8 + set.value.size) // total body size
      .writeInt(0) // opaque
      .writeLong(0) // CAS
      .writeInt(set.flags)
      .writeInt(set.expiration)
      .writeBytes(key)
      .writeBytes(set.value)
  }

  def encodeGet(buffer : ByteBuf, get: GetRequest) {
    encodeKeyMessage(buffer, get.key, Keys.Get)
  }

  def encodeDelete(buffer : ByteBuf, delete: DeleteRequest) {
    encodeKeyMessage(buffer, delete.key, Keys.Delete)
  }

  def encodeKeyMessage( buffer : ByteBuf, keyName : String, code : Int ) {
    val key = keyName.getBytes(CharsetUtil.US_ASCII)
    buffer
      .writeByte(Keys.RequestKey)
      .writeByte(code) // message code
      .writeShort(key.size) // key size
      .writeByte(0) // extras length
      .writeByte(0) // data type
      .writeShort(0) // reserved
      .writeInt(key.size) // total body size
      .writeInt(0) // opaque
      .writeLong(0) // CAS
      .writeBytes(key)
  }

}

And here we are finally digging into Netty. The encoder’s goal is to turn one of our high level ClientRequest messages into a ByteBuf (a collection of bytes) which is what actually gets written to the wire. This class doesn’t know anything about our client or how we’re doing it, all it knows is that it takes a message and turns it into a sequence of bytes. This independence greatly simplifies the interaction between objects in our implementation, they don’t have to know about each other, they just communicate over the high level messages we defined earlier and that’s all.

Given there are some well known patterns when building network messaging apps, Netty comes with a collection of base classes you can inherit when building your own stuff and we’re using one of those here, the MessageToByteEncoder. This class defines an encode method that gives us the ChannelHandlerContext (for now, think about it as the collection of pipes we’re using to communicate), the ClientRequest message and a ByteBuf object were we will write the data.

Our implementation here is just matching on the message code (which is faster than matching on object type) and call the method to turn the message into a collection of bytes. The @switch is there because we want to make sure that the compiler will turn this into a Java switch/case operation, if we make a change to our code that prevents the compiler from generating a switch/case, compilation will fail and we will be able to fix this.

But what are we writing here? Let’s look at how the common memcached packet is organized(the offset are positions in an array):

| offset | description                                                           |
| 0      | magic number indicating if server or client packet                    |
| 1      | message type                                                          |
| 2-3    | size of the key in this message (if there is one)                     |
| 4      | extras length, some messages contain an extra field, that's it's size |
| 5      | data type, not in use                                                 |
| 6-7    | reserved field, not in use                                            |
| 8-11   | total message body size (this includes the key size as well)          |
| 12-15  | opaque field for operations that use it                               |
| 16-23  | CAS field for operations that use it                                  |
| 24-N   | bytes that symbolize the key that is being operated on                |

For both GET and DELETE operations, this is the packet that we write. There’s a bunch of control information at the top and our data starts at the 24th item. All packets sent to/from memcached have at least 24 bytes and all that changes betwen them is if there are extra fields (like we have at the SET message) and if it has a body other than the key value. For the SET operation, the packet would be:

| offset | description                                                           |
| 0      | magic number indicating if server or client packet                    |
| 1      | message type                                                          |
| 2-3    | size of the key in this message (if there is one)                     |
| 4      | extras length, some messages contain an extra data, this is it's size |
| 5      | data type, not in use                                                 |
| 6-7    | reserved field, not in use                                            |
| 8-11   | total message body size (this includes the key size as well)          |
| 12-15  | opaque field for operations that use it                               |
| 16-23  | CAS field for operations that use it                                  |
| 24-27  | flags the client has defined for this key                             |
| 28-31  | expiration defined for this key                                       |
| 24-N   | bytes that symbolize the key that is being set, N is 24 + the key     |
|        | size defined above                                                    |
| N+1-Z  | bytes that represent the value that is being stored, it starts once   |
|        | the key defined above ends and goes until (total size - key size)     |

The message format is mostly the same, the only difference is that now we have two extra fields and the value being set. With that, the code itself is self explanatory, we just turn our messages into bytes following both formats above.

Decoding messages

Now that we know how to turn our messages into bytes, let’s figure out how to do it backwards, turn the bytes written by the server into high level messages we can use. The piece of code responsible for this is the MemcachedDecoder class:

package memcached.netty

import io.netty.handler.codec.ByteToMessageDecoder
import io.netty.channel.ChannelHandlerContext
import io.netty.buffer.ByteBuf
import java.util
import memcached.netty.messages._
import scala.annotation.switch
import io.netty.util.CharsetUtil

class MemcachedDecoder extends ByteToMessageDecoder {

  def decode(ctx: ChannelHandlerContext, in: ByteBuf, out: util.List[AnyRef]) {

    if (in.readableBytes() >= 24) {
      in.markReaderIndex()

      in.readByte() // magic number
      val commandCode = in.readByte()
      val keyLength = in.readUnsignedShort()
      val extrasLength = in.readUnsignedByte()
      val dataType = in.readByte()
      val status = in.readShort()
      val bodyLength = in.readUnsignedInt()
      val opaque = in.readInt()
      val cas = in.readLong()

      if (in.readableBytes() >= bodyLength) {
        (commandCode: @switch) match {
          case Keys.Get => {
            val flags = if ( extrasLength > 0 ) {
              in.readInt()
            } else {
              0
            }

            val bytes = new Array[Byte](bodyLength.toInt - extrasLength)
            in.readBytes(bytes)

            val value = if (status == ServerResponse.Ok) {
              Some(bytes) -> None
            } else {
              None -> Some(new String(bytes, CharsetUtil.US_ASCII))
            }

            out.add(new GetResponse(value._1, status, flags, opaque, cas, value._2))
          }
          case _ if extrasLength == 0 => {
            val body = if (bodyLength > 0) {
              Some(in.toString(CharsetUtil.US_ASCII))
            } else {
              None
            }
            in.readerIndex((in.readerIndex() + bodyLength).toInt)
            out.add(new StatusResponse(commandCode, status, opaque, cas, body))
          }
          case _ => throw new UnknownResponseException(commandCode)
        }

      } else {
        in.resetReaderIndex()
      }
    }

  }

}

This one is also about the same size as the encoder and what it does isn’t much different either. Now we inherit from a ByteToMessageDecoder class since we take a collection of bytes to turn it into a message.

First thing is to make sure we have at least 24 bytes to read. If there are less than 24 bytes we don’t have a full message yet so there’s no need to try and read it, just let the server write a bit more data. Once we get something that has at least 24 bytes, it’s time for the action.

We start by marking the reader index, this means that we want to store at which byte we are before we start to read stuff from this collection. This is necessary because we don’t know if we have a full message yet, we will only know it once we read the body size field so we leave the original index marked so we can get back to it in the future in case the body hasn’t been fully read yet.

Now we read all fields from the packet header (they are the same as the ones we saw above for client packets). Once we finish, we check if the bytes available to read are at least as much as the message body we expect, if they are, we proceed to finish reading the message, if it isn’t we reset the collection to it’s original reader index and let the server write a bit more bytes.

The only special case we have now is the GET case because there are extra fields and the value could be there as well, so we have a special response for it, for all other cases, the StatusResponse is completely fine since all other commands will just check for the status field.

Writing encoders and decoders for well defined binary protocols like memcached’s is dead simple, you just read the bytes and turn them into useful data. Fact that Netty already does all the connection and NIO weight lifting is also a huge advantage as our code gets to be extremely compact and to the point instead of having to handle selector loops, thread pooling and all that.

Getting to the meat of the client

Now that we have the encoder and decoder in place, we can build the actual client that will be used by our code to talk to memcached. Given this class will be larger than all others, we’ll have to break the discussion in two steps, let’s look at the first part:

object NettyClient {

  InternalLoggerFactory.setDefaultFactory(new Slf4JLoggerFactory())
  val DefaultEventLoopGroup = new NioEventLoopGroup()
  val log = LoggerFactory.getLogger(classOf[NettyClient])

  def createBootstrap( handler: ChannelHandler ) = new Bootstrap()
    .group(DefaultEventLoopGroup)
    .channel(classOf[NioSocketChannel])
    .option[java.lang.Boolean](ChannelOption.SO_KEEPALIVE, true)
    .handler(new ChannelInitializer[io.netty.channel.Channel] {
    def initChannel(ch: Channel) {
      ch.pipeline().addLast(
        new MemcachedDecoder,
        new MemcachedEncoder,
        handler
      )
    }
  })

}

Here we define a bunch of the necessary magic for our client to work, first we set a reusable event loop group, using an NioEventLoopGroup. Netty allows you to use many different IO providers, but the most common so far is the NIO based one since if you’re using Netty you usually want to build async networking clients. Here we also set the Netty logger so we can look at stuff that’s happening in there.

Finally, the most important part here, the createBootstrap method. The Bootstrap in Netty serves as the builder object for creating channels that communicate over IO, it holds all the configuration needed to to setup your pipeline, like our encoder and decoder objects, the event loop group, the channel type (an NIO socket channel, in our case) and other options.

The last part of the method is including a ChannelInitializer that declares the order our pipeline works, the encoder/decoder order doesn’t matter since they only work in one way, but the last part is the most important of them all, but it’s important for them to come before our final handler is declared because our handler doesn’t understand ByteBuf objects, all it knows is the high level messages we defined above. So the order will (almost) always have protocol encoders/decoders first and then our actual handler last.

And now the actual client implementation:

class NettyClient(host: String = "localhost", port: Int = 11211)
  extends SimpleChannelInboundHandler[ServerResponse]
  with Client {

  import NettyClient._

  private val bootstrap = createBootstrap(this)

  private val connectPromise = Promise[Client]()
  private var disconnectFuture: Future[Client] = null

  private var currentContext: ChannelHandlerContext = null
  private var commandPromise: Promise[ServerResponse] = null

  def set(key: String, bytes: Array[Byte], flags: Int = 0, expiration: Int = 0): Future[StatusResponse] =
    this.write(new SetRequest(key, bytes, flags, expiration)).castTo[StatusResponse]

  def get(key: String): Future[GetResponse] =
    this.write(new GetRequest(key)).castTo[GetResponse]

  def delete(key: String) : Future[StatusResponse] =
    this.write(new DeleteRequest(key)).castTo[StatusResponse]

  def connect(): Future[Client] = {
    this.bootstrap.connect(host, port).onFailure {
      case e : Throwable => this.connectPromise.failure(e)
    }

    this.connectPromise.future
  }

  def close(): Future[Client] = {
    if (this.currentContext != null && this.currentContext.channel().isActive && this.disconnectFuture == null) {
      this.disconnectFuture = this.currentContext.close().map(v => this)
    }

    if (this.disconnectFuture == null) {
      Promise.success[Client](this).future
    } else {
      this.disconnectFuture
    }
  }

  private def write(request: ClientRequest): Future[ServerResponse] = {
    this.synchronized {
      if (this.commandPromise != null) {
        throw new BusyClientException
      }

      val result = Promise[ServerResponse]()

      this.currentContext.writeAndFlush(request).onFailure {
        case e: Throwable => result.tryFailure(e)
      }

      this.commandPromise = result
      this.commandPromise.future
    }
  }

  def channelRead0(ctx: ChannelHandlerContext, msg: ServerResponse) {
    this.synchronized {
      if (this.commandPromise != null) {
        if (msg.isError) {
          val exception = new CommandFailedException(msg)
          exception.fillInStackTrace()
          this.commandPromise.failure(exception)
        } else {
          this.commandPromise.success(msg)
        }
        this.commandPromise = null
      } else {
        log.error("Received response {} but had no promise to complete", msg)
      }
    }
  }

  override def exceptionCaught(ctx: ChannelHandlerContext, cause: Throwable) {
    log.error("Connection failed", cause)
    this.synchronized {
      if (this.commandPromise != null) {
        this.commandPromise.tryFailure(cause)
      }

      this.connectPromise.tryFailure(cause)
    }
  }

  override def handlerAdded(ctx: ChannelHandlerContext) {
    this.currentContext = ctx
    this.connectPromise.success(this)
  }
}

And here’s the meat of the implementation. Our client inherits from SimpleChannelInboundHandler because it will do most of the event handling and heavy lifting required to make the pipeline work correctly, but you could just write your own handler implementation from scratch here as well, depends on how much you work you want to do. For our simple case, just inheriting from the inbound handler is more than enough, since we’re only overriding 3 methods from the superclass.

Our class starts by declaring the promise fields we will use to handle communication. Again, all of this is async, so all communication and messages has to be handled in an async way as well, with promises and futures being returned and transformed everywhere, even for the connect action.

Let’s look at connect first:

  def connect(): Future[Client] = {
    this.bootstrap.connect(host, port).onFailure {
      case e : Throwable => this.connectPromise.failure(e)
    }

    this.connectPromise.future
  }

Here we use the bootstrap we have to connect to the host and port fields we had defined before and if it fails it will call our onFailure hook and fail the connect promise. The method itself returns the future tied to our connectPromise instance variable.

But hey, how come Netty, a Java project, has an implementation that returns a promise that has an onFailure handler? Well, it doesn’t. The connect method at Boostrap returns a Netty’s channel future but we wrote a nice implicit conversion from the channel future to our Future[T] implementation, let’s check it:

package memcached

import io.netty.channel.{ChannelFutureListener, ChannelFuture}
import async.{Promise, Future}

object ChannelFutureTransformer {

  implicit def toFuture(channelFuture: ChannelFuture): Future[ChannelFuture] = {
    val promise = Promise[ChannelFuture]

    channelFuture.addListener(new ChannelFutureListener {
      def operationComplete(future: ChannelFuture) {
        if ( future.isSuccess ) {
          promise.success(future)
        } else {
          if ( future.cause() != null ) {
            promise.failure(future.cause())
          } else {
            val exception = new FailedFutureException(channelFuture)
            exception.fillInStackTrace()
            promise.failure(exception)
          }
        }
      }
    })

    promise.future
  }

}

While implicits have their own set of issues and complications, this is a powerful way to simplify our client code and make Netty’s interface to be a bit more like what we already have. Instead of having to handle these channel futures in a different way, we just transform them into the Future objects we already know and use. Implicits in Scala are not to be feared, but you should use them judiciously.

So, now we know how our failure to connect to a memcached will fail the Future returned by connect, but where do we succeed the future? Let’s find it!

  override def handlerAdded(ctx: ChannelHandlerContext) {
    this.currentContext = ctx
    this.connectPromise.success(this)
  }

And this is where we succeed our connection promise. When our handler is is finally connected to the server (and is ready to send and receive messages) this method is called with a connected ChannelHandlerContext that represents our full pipeline until the memcached server. Think about it as the pipes (including our encoders and decoders) to the server over the network. We hold this reference here as an instance variable because it’s what we will use to send messages to the server. We write our messages here at the context and they will go through the pipes until the server.

To disconnect from a server, the implementation is a bit simpler:

  def close(): Future[Client] = {
    if (this.currentContext != null && this.currentContext.channel().isActive && this.disconnectFuture == null) {
      this.disconnectFuture = this.currentContext.close().map(v => this)
    }

    if (this.disconnectFuture == null) {
      Promise.success[Client](this).future
    } else {
      this.disconnectFuture
    }
  }

We basically check if we’re connected and ask the handler to close the connection. If we’re already disconnected (or were never connected) we just return an already succeeded promise.

Now let’s get to the part where we actually talk to memcached:

  private def write(request: ClientRequest): Future[ServerResponse] = {
    this.synchronized {
      if (this.commandPromise != null) {
        throw new BusyClientException
      }

      val result = Promise[ServerResponse]()

      this.currentContext.writeAndFlush(request).onFailure {
        case e: Throwable => result.tryFailure(e)
      }

      this.commandPromise = result
      this.commandPromise.future
    }
  }

The write method is the only point where we interact with the channel context. Here we write a message to be sent to the server through our pipeline. First, we make sure we’re not waiting for a response from the server already. Since the protocol itself isn’t multiplexed, we can’t safely send many commands at once to the server from a single client so we just lock it to only one pending command at a time.

Going forward, we create the promise that will hold the response when it comes and write the message. Here we just write one of ClientRequest subclasses, our MemcachedEncoder will do the work and turn it into the actual bytes. Since it’s only one message, we use writeAndFlush and at the same time we register a handler to the channel future that is produced by this write. In this case, the only thing we care about is if we fail to write the message for some reason, so we only register a failure callback.

With that we set the promise as the current promise and return it’s future. Since this method is private, let’s look at who calls it inside our class:

  def set(key: String, bytes: Array[Byte], flags: Int = 0, expiration: Int = 0): Future[StatusResponse] =
    this.write(new SetRequest(key, bytes, flags, expiration)).castTo[StatusResponse]

  def get(key: String): Future[GetResponse] =
    this.write(new GetRequest(key)).castTo[GetResponse]

  def delete(key: String) : Future[StatusResponse] =
    this.write(new DeleteRequest(key)).castTo[StatusResponse]

The methods that call write are just creating the message objects and firing them. Here we also have the new castTo method that simplifies our promise handling. Since the command promise is a Promise[ServerResponse] object, we have to cast it to one of the ServerResponse subclasses and all that castTo does is perform this mapping. It’s implementation at Future[T is extremely simple:

  def castTo[S](implicit executor: ExecutionContext): Future[S] = this.map(v => v.asInstanceOf[S])

We could write this code inside the NettyClient object, but it’s much simpler to just have it at Future.

Now that we know how writes work let’s look at what reads look like:

  def channelRead0(ctx: ChannelHandlerContext, msg: ServerResponse) {
    this.synchronized {
      if (this.commandPromise != null) {
        if (msg.isError) {
          val exception = new CommandFailedException(msg)
          exception.fillInStackTrace()
          this.commandPromise.failure(exception)
        } else {
          this.commandPromise.success(msg)
        }
        this.commandPromise = null
      } else {
        log.error("Received response {} but had no promise to complete", msg)
      }
    }
  }

Even simpler, since all the byte decoding magic happens at MemcachedDecoder here we just grab the message and complete the promise, either with a success or a failure. As Netty allows us to separate our concerns clearly when building networked applications, our objects can focus on doing just one thing, just like our handler here that is mostly an orchestrator between client and server instead of having to care about encoding/decoding and all that.

And the only missing piece is the error handling:

  override def exceptionCaught(ctx: ChannelHandlerContext, cause: Throwable) {
    log.error("Connection failed", cause)
    this.synchronized {
      if (this.commandPromise != null) {
        this.commandPromise.tryFailure(cause)
        this.commandPromise = null
      }

      this.connectPromise.tryFailure(cause)
    }
  }

This method is called if something in our pipeline raises an exception, could be one of the encoders/decoders, a connection failure or anything like that. And what we do here is to try and fail the current command and the connection, since we don’t know at which state this could have been raised. To be safe, just fail it all :)

Talking to memcached

And here we are, let’s look at how we can talk to memcached:

class ClientSpecification extends Specification {

  def toBytes(value: String): Array[Byte] = value.getBytes(CharsetUtil.US_ASCII)

  def fromBytes(bytes: Array[Byte]): String = new String(bytes, CharsetUtil.US_ASCII)

  def withClient[T](f: Client => T): T = {
    val client = new NettyClient()
    await(client.connect())
    try {
      f(client)
    } finally {
      await(client.close())
    }
  }

  def await[T](future: Future[T], seconds: Int = 3): T = {
    var count = 0
    while (!future.isCompleted && count <= seconds) {
      Thread.sleep(1000)
      count += 1
    }

    if (future.isCompleted) {
      future.value.get.get
    } else {
      throw new IllegalStateException(s"Trying to access the future did timeout after ${seconds} seconds")
    }
  }

  "client" should {

    "set a value and get it back correctly" in {
      withClient {
        client =>
          val unique = UUID.randomUUID().toString
          val key = s"hello-${unique}"
          val value = s"hello-world-${unique}"

          val result = await(client.set(key, toBytes(value)))
          result.isError must beFalse
          result.status === ServerResponse.Ok

          val response = await(client.get(key))
          fromBytes(response.value.get) === value
      }

    }

  }

}

You can see the full spec at the project repo, but here we can see how we interact with the promises and future objects we used to build our client.

It’s not actually async, since we have to block for the tests to run, but still we talk to the server and get responses back in an async fashion and you could use this codebase to talk to a memcached server in an extremely simple way.

And now here we are, finally at the end of this tutorial, where you should have learned what and how to use promises and futures in your Scala projects and how you can use a library like Netty to build networked apps in a simple and intuitive fashion.

The "don't depend on a small gem" fallacy

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 13 Apr 2014 00:00:00 UTC

There’s this thing around the Ruby community (it might be at other communities as well, I don’t know) that you should not depend on small gems. If a gem is less than a couple hundred lines of code, you should write it yourself, because, you know, you will have to maintain it.

It isn’t worth it to have a small gem in your codebase because when you upgrade your other dependencies, it might break, the current maintainer might disapear or just stop keeping the gem up to date with the newer Ruby, Rails and other dependencies it might have. And once a gem becomes unmaintained it’s a burden on you and your team to handle it because no one will be able to fix that code anymore.

If you’re talking to people old enough in the Ruby community, they might even invoke the good old Jamis Buck who wrote the following when reviewing a Rails 1.2 book (yeah, I’m old as well):

It shows how to implement a basic user authentication system without resorting to plugins! Far, far too many newcomers to Rails jump on the user-auth plugin bandwagon, which leads to cargo-culting. My advice is: never use a plugin you would not be able to write yourself.

This last piece, never use a plugin you would not be able to write yourself, seems to have been burned into many people’s minds. Whenever they see a small gem, they’ll think (wrongly, that’s not what that phrase meant):

Ah, I don’t need that, I can write that in a couple minutes and I won’t have to depend on that gem anymore, it isn’t worth it.

Now, let me ask you something, what is the difference between that small gem code and your code?

Other than the usual not-invented-here syndrome, your code is as much a liability as other people’s code. It doesn’t matter who wrote it, you, your neighbour, that crazy brazilian guy who lives by the beach (me, by the way). When your application depends on it, it becomes part of the application itself.

Worse, if it’s just a couple hundred lines of code, what kind of trouble are you having trying to understand it? Is there actually an issue or are you just trying to write code for the sake of writing code?

Unless you’re removing code completely, changing from someone else’s code to your code (that will be someone else’s code when you leave the project or job) is changing from 0 to zero. The liability is still there, the code is still there, the only difference is that if it’s out there it might still have the luck of being maintained by someone else other than you. Your private code inside your private repo will only be ever maintained by you and your team, there’s no chance someone from the outside will help with it at all.

As important as understanding when to write code is understanding how to read and reuse code. A couple weeks ago we decided to upgrade mongoid from our old and venerable 2.x to 3.x so we could use the latest replica set related features. While doing this, one of our dependencies, mongoid-sequence, had to be upgraded as well and this upgrade caused issues due to a change in the way the sequence name was generated.

We got this in QA and I could, instead of fixing the gem itself, rewrite the code for it. The gem is dead for many years now, the code we were using was a fork of the original repo already and it’s just a couple lines of code, it wouldn’t take long for me to rewrite it and maintain the same old behavior. But as simple as it would be to rewrite it, it was much simpler to read it, understand what it was doing, what was causing the bug and fix it right there in the first place and that’s what I did (I’m feeling bad I didn’t send a pull request for them now :( ).

Rewriting it, other than taking my time that could be better used working on other important stuff, wouldn’t win us anything. The code was simple, direct and the issue that was happening was clear. Once I knew what was the bug, I fixed it, updated and included tests for the case and moved on. There was no need for me to waste more time into rewriting this if I would end up with something that did the same thing with a slightly different codebase.

I even had a case of this for myself. Many years ago, I wrote a Rails plugin (do people still know what Rails plugins were?) that allowed you to have a master and a slave connection to your database. I wrote it because we were migrating from Rails 1.x to 2.x and it was all in flux, the APIs were still changing and connection pooling had just arrived to ActiveRecord. I could have contributed this to masochism but had no idea if it would be accepted (it probably would, but that’s water under the bridge now) or how much code it would take since I was using a completely different approach for it so I just rolled my own solution.

If you look at it, there’s isn’t much code there, you could definitely rewrite it at your own will and make it do whatever you would want it to do, but people decided it would be simpler just to fork and work on top of it. If you run blame on this SoundCloud fork, it’s unlikely you’ll find any lines written by me, they have changed, fixed and upgrated it to their own needs and this is actually awesome. They have eventually moved out of my code and now it’s their own thing, but this wasn’t a single handed let’s rewrite all the things decision, it was a natural evolution of the codebase just like your own apps evolve.

So, next time someone comes in and says let’s rewrite this gem because it’s small/dead/simple ask them back:

Who maintains the code now? Who will maintain it in the future?

If it’s still you, there’s hardly a reason to do it.

Liked this? Upvote on Hacker News

Public methods are your public API

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 08 Mar 2014 00:00:00 UTC

I was reading Public Methods != Public API and it didn’t really ring to me. The example itself is nice and it makes sense not to assume all public methods are part of your object’s public API but this isn’t what actually happens in the real world. If you use Ruby’s Enumerable frequently, you frequently use methods like map and inject, if any of these methods is renamed to something else or is removed, you will have a hard time.

If you lived through Rails 1.x and 2.x you know this by heart. Most gems were monkeypatching Rails in weird ways to include new features, even Rails itself used this throughout it’s own codebase to implement higher level abstractions like dirty-tracking for ActiveRecord objects (alias_method_chain anyone?). Whenever a new version was published, gems would break and we’d have to go hunting for newer versions of our monkeypatches and hope a hook would come up to save us from all this madness.

Every method that can be seen will be called, no matter what, telling people that you should not call this, it’s not part of the public api has very little value, specially because the code itself isn’t saying this. It will probably be embedded in some kind of documentation and that’s actually not the place for this. Actually, even if the method isn’t public per se, it shouldn’t be there as well:

I almost declared a method private, but then I realized that was the code telling me I needed a new class. Much better result.
— Herr Fauler (@chadfowler) March 3, 2014

While this might seem fatalistic and exaggerated, it’s hard to see cases where this doesn’t apply. When you declare a method private, it usually means you don’t want people that are using that specific object to see that method. It doesn’t mean you don’t want everyone from being able to see it, it’s just that you think whoever is using this object doesn’t need to know about it.

But this current object needs to know about this method, so this method is a public API that someone needs to have access to. And that’s why most of the time when this happens what you want is another object to do the job. And with this we circle back to the beginning, while everything that can be called will be called, if is isn’t part of the current object, it isn’t part of it’s public API because you can’t call it directly. The new level of indirection hides the method from people sending messages to your object.

One advice I always try to keep close to me when building APIs is this from Joshua Block:

When in doubt, leave it out. If there is a fundamental theorem of API design, this is it. It applies equally to functionality, classes, methods, and parameters. Every facet of an API should be as small as possible, but no smaller. You can always add things later, but you can’t take them away. Minimizing conceptual weight is more important than class- or method-count.

Once it’s out there, people will be using it and it’s hard to go back. Hard for who wrote the API, since they will have to maintain or alienate users and hard on users as well since this means they will have to revisit code that was actually working just to honor the new API that is different now.

Careful with what you include

Still in the same vein of public APIs, comes another problem that is more common in dynamic languages like Ruby. When you inherit from or include something in your class/module you’re not taking only what’s directly in what you’re including/extending, you’re taking everything that that object extends/includes as well.

In a language like Ruby, where you can’t easily know at development time all methods that will be included into your object when you inherit/include something so you end up with weird to figure out errors due to methods defined in superclasses/modules you didn’t even know existed.

One of such examples is the process method defined at Mongoid::Attributes::Processing (this was mongoid 2.x). We had no idea there was a process method defined somewhere in mongoid and defined it in our own object. Since this method isn’t called in all code paths, the code worked in some cases and didn’t in others. Figuring out what was going on took some debugging and testing time. process itsef is a rather common method, Rails controllers themselves couldn’t have an action called process since this method was used by ActionPack to send requests to controllers as well.

So, not only it’s important to be careful on what you expose as the public API for your objects that will be consumed but it’s also important to consider what will happen with your classes/modules when they get included into other objects. Will it be like ActiveRecord::Base that includes a couple hundred methods at the included objects? Is it really necessary? Could we move this logic somewhere else?

Most of the time, you can. One of these days we wanted to include feature flagging to accounts in the system. The first idea was declaring the methods right at the Account object itself so we could do stuff like:

account.feature_visible?(:some_feature)
account.show_feature!(:some_other_feature)
account.hide_feature!(:main_feature)

But then account.rb was already big and this code would be used only in some very specific pieces in the system, why not move it somewhere else?

The actual implementation had features that would be enabled system-wide and that could also be enabled only for an specific account. Since we were using MongoDB, the list of features enabled for the account alone became an array at the Account class and the others lived at their own FeatureFlag records, so we needed a solution that had access to the account document and that could search for the FeatureFlag objects as well.

We ended up creating the FeatureFlagging::FeaturesCollection object for it:

module FeatureFlagging
  class FeaturesCollection

    def initialize(account)
      @account = account
    end

    def visible?(name)
      features.include?(name) || FeatureFlag.visible?(name, @account)
    end

    def show!(name)
      self.features = self.features + [name]
    end

    def hide!(name)
      self.features = self.features - [name]
    end

    def features
      @account.features_enabled || []
    end

    def features=(list)
      @account.features_enabled = list.flatten.uniq
      @account.save!
    end

  end
end

And so here we have all methods we need so far to check for features visible or not. Now, we have to hook it somewhere and this somewhere will be the account object itself. Here’s how it looks:

def features
  @features ||= FeatureFlagging::FeaturesCollection.new(self)
end

And this allows me to use code as:

account.features.visible?(:some_feature)
account.features.show!(:some_feature)
account.features.hide!(:some_feature)

Not much of a difference from the original one, is there?

There are two main differences here, the code that does all the feature flagging is now somewhere else and not at the account object. It doesn’t even have to use an account, any object with a features_enabled collection and a save method would do it, so this could be reused in other places/parts of the code as long as the public interface is maintained. This makes it simpler to test and to validate what is going on.

The other difference is that we didn’t pollute the Account object with the feature-related methods. Right now, we only have these 3 specific methods, but they might grow and, more importantly, they’re not inside the already large Account class, not even by including a module, they’re just not part of it in any way.

Classic OOP

There’s no magic here, this is the classic prefer composition over inheritance we see so often out there but don’t practice that much. Using composition in this case now only allows us to slim our Account object but simplifies testing the FeatureFlagging::FeaturesCollection object and allows us to include more methods there as we see fit without getting too worried if these methods will clash with someone or something.

Now let me get back to my codebase and fix all cases of this there as well :D

Going down the rabbit hole of Ruby's object conversions

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 24 Feb 2014 00:00:00 UTC

This new year I decided I’d do a bit less of community work by answering stuff at SO and mailing lists and would contribute more actual code to OSS projects. So, from time to time I wander about projects I use, and try to contribute by fixing stuff and sending PRs.

This has actually led me tqo figure out some interesting stuff and a new bugfix I did for rspec-mocks a couple days ago sent me down the rabbit hole of Ruby’s object conversions.

The Bug

The actual bug was found indirectly by @adamstegman. He was using test doubles in raise statements and they were causing a RuntimeError to be raised instead of the class he was using as a double, so his specs weren’t matching the expected error. In Ruby, you can raise objects of type Exception, strings or objects that act like strings.

These two last cases are the interesting ones here, string or string like objects.

So, what is a string like object?

In Ruby, a string like object is any object that has a to_str method defined on it. So, methods that expect to take a string, will usually check if there is a to_str method defined there if the object isn’t a string.

For rspec doubles, having the to_str method defined made them coercible to string (an unintended side effect, most likely) and raised it’s string representation.

If you try to raise something that isn’t an Exception, string or string like, that’s what you get:

2.1.0 :001 > class Something; end
 => nil
2.1.0 :002 > raise Something.new
TypeError: exception class/object expected
	from (irb):2:in `raise'
	from (irb):2

Given the double shouldn’t really be responding to stuff the user didn’t actually say it should respond to, making it coercible to string could hide weird bugs inside the code, since somewhere along the way the code could convert it to a string and the double itself would be gone.

After figuring out what was going on, the fix was simple, remove the to_str method from TestDouble and now you would see the error above when trying to raise a double, as expected.

And it’s this subtle bug that takes us to the real subject for this blog post, Ruby’s object conversions.

Explicit conversions

No, that’s not what you’re thinking.

Explicit conversions are when objects define methods like to_s, to_i, to_f, to_a and to_h. This means you can call these methods on the objects and they will return a string, an int, a float, an array or a hash, respectively, that represents the object.

These are explicit because the Ruby runtime will not call these methods for you to transform one object into another. The most common example is:

2.1.0 :001 > 1 + "40"
TypeError: String can't be coerced into Fixnum
	from (irb):1:in `+'
	from (irb):1

Well, that doesn’t work, but this works:

2.1.0 :001 > 1 + "40".to_i
 => 41

While String does define a to_i method, Ruby won’t call it for me, I have to manually call the method here to make sure the String object is transformed to an int before summing them.

oh, but that is tedious, isn’t it? you might think. Well, not if you fall for a bug that’s caused by the runtime coercing your objects into something else, just like the bug that was fixed above. Personally, I’d take a well known behavior over magic all the time.

If you have programmed in languages like Java or C#, this might be awkward. In both languages, a String somewhere in a + expression will infect the sum and make it all string concatenation. This leads to subtle bugs and unexpected behavior, you can even see a lengthy discussion at the scala-users about the nightmare it is to have everything magically being turned into strings. Thanks, Matz, you did great!

Now back to explicit conversions, the only special case here is when your object is inside a string interpolation expression. Look at this:

2.1.0 :001 > "me" + 10
TypeError: no implicit conversion of Fixnum into String
	from (irb):1:in `+'
	from (irb):1

Doesn’t work, Fixnum can’t be implicitly converted into String. But if we do this:

2.1.0 :001 > "me #{10}"
 => "me 10"

Perfect! That’s what we’re looking for. In the specific case of string interpolation, Ruby will call the object’s to_s method and use that as the output to be included in the string. That’s the only case you will see the runtime automatically calling one of the explicit conversion methods.

So, if the object you’re working with implements one of these methods and you need to transform it, just call them. For instance, if you have an array of pairs:

2.1.0 :001 > [ ["me",10], ["you", 20] ].to_h
 => {"me"=>10, "you"=>20}

You can easily turn it into a Hash calling to_h on it, as you can turn a Range into an Array by calling to_a on it:

2.1.0 :001 > ('a'..'f').to_a
 => ["a", "b", "c", "d", "e", "f"]

Implicit conversions

Now this is where the magic really starts to show itself. Ruby defines some implicit conversion methods that are called under specific circunstances on objects to check if they can be transformed to something else, they are to_int, to_str, to_ary, to_hash and to_enum (you’ll see some others below).

There isn’t an actual list of where or when these methods are called. Given we don’t annotate variables or methods with types in Ruby (as we do in languages like Java, for instance) the runtime can’t figure out when this would be necessary and the built in funcionality just tries do to this when it thinks it’s necessary. One of the examples of this is exacly the rspec bug above.

Let’s look at the C code that gets called when you try to raise an exception, it starts on rb_f_raise (or Kernel.raise):

static VALUE
rb_f_raise(int argc, VALUE *argv)
{
    VALUE err;
    VALUE opts[raise_max_opt], *const cause = &opts[raise_opt_cause];

    argc = extract_raise_opts(argc, argv, opts);
    if (argc == 0) {
	if (*cause != Qundef) {
	    rb_raise(rb_eArgError, "only cause is given with no arguments");
	}
	err = get_errinfo();
	if (!NIL_P(err)) {
	    argc = 1;
	    argv = &err;
	}
    }
    rb_raise_jump(rb_make_exception(argc, argv), *cause);

    UNREACHABLE;
}

The important piece here is rb_make_exception which, calls make_exception below:

static VALUE
make_exception(int argc, VALUE *argv, int isstr)
{
    VALUE mesg, exc;
    ID exception;
    int n;

    mesg = Qnil;
    switch (argc) {
      case 0:
	break;
      case 1:
	exc = argv[0];
	if (NIL_P(exc))
	    break;
	if (isstr) {
	    mesg = rb_check_string_type(exc);
	    if (!NIL_P(mesg)) {
		mesg = rb_exc_new3(rb_eRuntimeError, mesg);
		break;
	    }
	}
	n = 0;
	goto exception_call;

      case 2:
      case 3:
	exc = argv[0];
	n = 1;
      exception_call:
	if (exc == sysstack_error) return exc;
	CONST_ID(exception, "exception");
	mesg = rb_check_funcall(exc, exception, n, argv+1);
	if (mesg == Qundef) {
	    rb_raise(rb_eTypeError, "exception class/object expected");
	}
	break;
      default:
	rb_check_arity(argc, 0, 3);
	break;
    }
    if (argc > 0) {
	if (!rb_obj_is_kind_of(mesg, rb_eException))
	    rb_raise(rb_eTypeError, "exception object expected");
	if (argc > 2)
	    set_backtrace(mesg, argv[2]);
    }

    return mesg;
}

The piece we’re looking for here is the call to rb_check_string_type, which is the function that converts something that has a to_str method into a real String, let’s see how it’s implemented:

VALUE
rb_check_string_type(VALUE str)
{
    str = rb_check_convert_type(str, T_STRING, "String", "to_str");
    return str;
}

And finally, let’s look at rb_check_convert_type:

static struct conv_method_tbl {
    const char *method;
    ID id;
} conv_method_names[] = {
    {"to_int", 0},
    {"to_ary", 0},
    {"to_str", 0},
    {"to_sym", 0},
    {"to_hash", 0},
    {"to_proc", 0},
    {"to_io", 0},
    {"to_a", 0},
    {"to_s", 0},
    {NULL, 0}
};

static VALUE
convert_type(VALUE val, const char *tname, const char *method, int raise)
{
    ID m = 0;
    int i;
    VALUE r;

    for (i=0; conv_method_names[i].method; i++) {
	if (conv_method_names[i].method[0] == method[0] &&
	    strcmp(conv_method_names[i].method, method) == 0) {
	    m = conv_method_names[i].id;
	    break;
	}
    }
    if (!m) m = rb_intern(method);
    r = rb_check_funcall(val, m, 0, 0);
    if (r == Qundef) {
	if (raise) {
	    rb_raise(rb_eTypeError, "can't convert %s into %s",
		     NIL_P(val) ? "nil" :
		     val == Qtrue ? "true" :
		     val == Qfalse ? "false" :
		     rb_obj_classname(val),
		     tname);
	}
	return Qnil;
    }
    return r;
}

VALUE
rb_check_convert_type(VALUE val, int type, const char *tname, const char *method)
{
    VALUE v;

    /* always convert T_DATA */
    if (TYPE(val) == type && type != T_DATA) return val;
    v = convert_type(val, tname, method, FALSE);
    if (NIL_P(v)) return Qnil;
    if (TYPE(v) != type) {
	const char *cname = rb_obj_classname(val);
	rb_raise(rb_eTypeError, "can't convert %s to %s (%s#%s gives %s)",
		 cname, tname, cname, method, rb_obj_classname(v));
    }
    return v;
}

The code is rather simple, first, it checks if the type already is the type we want to convert to. If it is, return it. Otherwise call convert_type with the value, type and conversion method.

convert_type, in turn, will check if the object implements the conversion method. In our case, it would check if the object implements to_str. Also, it only does the conversion if the method is in the list above, if it isn’t one of those methods it would just ignore it and not perform any conversion.

If we wanted to implement this in pure Ruby, it could be something like:

METHODS = ["to_int", "to_ary","to_str","to_sym","to_hash", "to_proc", "to_io","to_a", "to_s"]
def convert_type( value, type, method, raise_on_error = false )
  result = if value.kind_of?(type) || value.nil?
    value
  elsif METHODS.include?(method) && value.respond_to?(method)
  	value.send(method)
  else
    nil
  end

  if raise_on_error && !result.nil? && result.kind_of?(type)
  	raise TypeError, "can't convert #{value.class.name} to #{type.name} (#{value.class.name}##{method} gives #{result.class.name})"
  end

  result
end

So, while we call these methods implicit converters, they’re not that implicit. The runtime has to manually decide when this is required and call rb_check_string_type to convert what you have into a string or into any of the other types by itself. So, unless the documentation is specific about this or you know the code will make this check, don’t expect your objects to be converted into something else.

Another common built-in conversion is when you’re comparing String, Array and Hash objects with ==. The current implementation will check if the right-hand object is of the same type of the left-hand one and if it isn’t, it will try to convert it. Here’s the Hash#== implementation:

static VALUE
hash_equal(VALUE hash1, VALUE hash2, int eql)
{
    struct equal_data data;

    if (hash1 == hash2) return Qtrue;
    if (TYPE(hash2) != T_HASH) {
	if (!rb_respond_to(hash2, rb_intern("to_hash"))) {
	    return Qfalse;
	}
	if (eql)
	    return rb_eql(hash2, hash1);
	else
	    return rb_equal(hash2, hash1);
    }
    if (RHASH_SIZE(hash1) != RHASH_SIZE(hash2))
	return Qfalse;
    if (!RHASH(hash1)->ntbl || !RHASH(hash2)->ntbl)
        return Qtrue;
    if (RHASH(hash1)->ntbl->type != RHASH(hash2)->ntbl->type)
	return Qfalse;

    data.tbl = RHASH(hash2)->ntbl;
    data.eql = eql;
    return rb_exec_recursive_paired(recursive_eql, hash1, hash2, (VALUE)&data);
}

As you can see, if the object isn’t a Hash, it goes to rb_respond_to(hash2, rb_intern("to_hash")) to check if the object can be converted to a hash, if it can’t, it just returns false right away since you can’t compare some generic object with a hash.

Arithmethic coercion

One little known feature of Ruby numbers is the coerce method, it allows you to mix different types of numbers to do your math correctly. Let’s look at what I would have to do to sum the 1/5 to 10:

2.1.0 :001 > require 'rational'
 => true
2.1.0 :002 > r = Rational(1,5)
 => (1/5)
2.1.0 :003 > result = r.coerce(10)
 => [(10/1), (1/5)]
2.1.0 :004 > sum = result.inject(Rational(0,1), :+)
 => (51/5)

As you can see, I start with the Rational object and then call coerce on it with the integer. As a result of that, the 10 integer is transformed into the 10/1 rational (that is just 10) and we can then sum both of them. I could just manually sum them, but since they are inside an array already it’s much simpler just to use inject to perform the sum.

Boolean coercion

Operators and control structures that expect booleans in Ruby will take any kind of object and use it. There are two cases, false and nil are falsy so they will behave as if it was a real false boolean value (ie. if nil will go to the else piece), and the other case is everything else is truthy.

Empty strings, arrays, hashes, 0, they will all be assumed to be true values. Every single object that is not false or nil is assumed to be true when used in control structures and boolean operators, doesn’t matter what the object is.

This leads to some interesting developments when using boolean operators in Ruby, for instance:

2.1.0 :001 > false || nil
 => nil

Boolean operators in Ruby will not return a boolean, but the last expression that was evaluated by the operator and this is both good a bad. Good, because it lets you write terse statements like the elvis operator:

2.1.0 :003 > me ||= 10
 => 10

This is equivalent to me = me || 10.

And bad because if you actually need something to always be a boolean (maybe you are turning this value to JSON or something else) you need to add a bit more code:

2.1.0 :005 > !!(10 && [])
 => true

Without this, the result of executing that && operation would be [] (the empty array).

Don’t trust magic

The main takeaway I had from all this is that you don’t need magic. Think once, twice, three, four, ten times before you implement one of those implicit conversion methods in your objects, because you never know when it will be called and how this could change the behavior of your system.

If you’re not 100% sure you actually need it, just don’t use it. Stick to the explicit conversion methods, where you know what’s going on and what is going to happen instead of letting your code fly away and your objects be magically transformed into something else.

Scala's Either, Try and the M word

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 17 Feb 2014 00:00:00 UTC

Since we have covered Option previously, it’s time to show you other types just like Option that are prevalent in Scala’s standard library and third party code. We will also see how these types are part of a larger set of objects and they have this name that starts with M that, unfortunately, shuts people off when they hear about it so I won’t type it here just yet.

`Either[T]` one thing or the other

If you look at Either’s Scala Docs you will see it says it is a disjoint union. In math, a disjoint union is a collection of two sets that have no items in common, when we translate this to Scala, imagine that each set is a different type and the items are instances of these two types.

Saying we return an Either[String,Int] means that we could return an Int or a String, they’re both different types and don’t share much. In fact, we usually use them for completely different things, so why would we ever care about using something like this?

One of the most common cases is because we use exceptions when we shouldn’t. If you wrote Objective-C code somewhere in this or in a past life, you have probably seen NSError objects lying around in your codebase. If you have never seen Objective-C, don’t despair, NSError is an object that’s used throughout Objective-C APIs to give you better information about errors that have happened while you have tried to do something.

NSError is not an exception object, you don’t throw, raise, catch or rescue NSError objects (Objective-C has it’s own NSException class for that), they’re not meant for that. They’re meant to convey more information about why your code failed to do whatever you wanted it to do.

And why is it not an exception, you ask?

Because exceptions are for exceptional cases, for when something is really broken, NSError is used when the other case is not an exceptional case but something that happens with some frequency and that you should handle it gracefully. The main goal here is to make it clear from the types you use that a method could have two different outcomes and they are common enough that you should care about them both and make sure your code is capable of handling them.

Let’s get to the code now:

sealed trait Either[+L,+R] {

  def isLeft : Boolean
  def isRight : Boolean

  def fold[X](leftFunction : (L) => X, rightFunction : (R) => X) : X =
    this match {
      case Right(value) => rightFunction(value)
      case Left(value) => leftFunction(value)
    }

}

case class Right[R](value : R) extends Either[Nothing,R] {
  override def isLeft = false
  override def isRight = true
}

case class Left[L](value : L) extends Either[L,Nothing] {
  override def isLeft = true
  override def isRight = false
}

The pattern is mostly what we have seen before, you have the base sealed trait and then you have case classes we can use to pattern match on the values. We also have a fold method that can be used to produce an output to the computation for both cases, mostly a shortcut to pattern matching.

Let’s look at an example of using it:

case class User( id : Int, name : String, password : String )

case class ServiceResponse( status : Int, message : String )

class ServiceClient( val users : Map[Int,User]) {

  def userDetails(id : Int, password : String) : Either[ServiceResponse,User] = {

    users.get(id) match {
      case scala.Some(user) => if ( user.password == password ) {
        Right(user)
      } else {
        Left(ServiceResponse(401, "You can not view this user's details"))
      }
      case scala.None => Left(ServiceResponse(404, "User does not exist"))
    }

  }

}

Here we have a simple ServiceClient class that could be an HTTP client talking to some external service. This external service could respond with specific error codes depending on what the client is asking and how he is asking. These error codes don’t need to be exceptions, that’s just how HTTP works, we could have a success and receive the user data or get one of the non-success HTTP response codes with a message and that’s this message that makes either useful here.

If we were just looking at “user” and “no user” scenarios, we could easily model this interaction using Option, but if we did that we would lose the ability to show the detailed message produced by the server to the client of our app. He would just see that “there’s no user”, without knowing what is actually going on.

Let’s fold on the data now:

class EitherSpecification extends Specification {

  val service = new ServiceClient(
    Map(1 -> User(1, "John Doe", "123456"))
  )

  "either" should {

    "fold to 401" in {

      val message = service.userDetails(1, "none").fold(
        (response) => {
        s"Password does not match ${response.status} - ${response.message}"
      }, (user) => {
        s"User is ${user.name}"
      } )

      message === "Password does not match 401 - You can not view this user's details"
    }

    "fold to 404" in {

      val message = service.userDetails(50, "none").fold(
        (response) => {
          s"User does not exist ${response.status} - ${response.message}"
        }, (user) => {
          s"User is ${user.name}"
        } )

      message === "User does not exist 404 - User does not exist"
    }

    "return the user details" in {

      val message = service.userDetails(1, "123456").fold(
        (response) => {
          "should not have come here"
        }, (user) => {
          s"User is ${user.name}"
        } )

      message === "User is John Doe"
    }

  }

}

As you can see, folding on Either is specially nice if you have to build some output message based on the objects being returned. This could be a command line app, a webapp or anywhere where you had to render some message in all cases.

So, whenever you have a method that could possibly return two different types, Either will be there for you.

Using `Try[T]` to compose on exceptions

Previously, Either was also used to handle the exception case, you call a method that could possibly raise an exception, you would use Either, with Left being the error and Right the expected value. Since Scala 2.10, this usage was replaced by the use of Try. A Try wraps a computation that could either result in a value or in an exception being thrown. Think about it as a specialization for Either, while you would use Either whenever you have to return one thing or the other, with Try it’s always one thing or an error.

Being specific like this, means that using Try for the error case is much simpler than doing the same with Either. Let’s look at how we could implement it:

object Try {
  def apply[T](f: => T): Try[T] =
    try {
      Success(f)
    } catch {
      case NonFatal(e) => Failure(e)
    }
}

sealed trait Try[+T] {

  def flatMap[U](f: (T) => Try[U]): Try[U]
  def map[U](f: (T) => U): Try[U]
  def foreach[U](f: T => U): Unit
  def isSuccess: Boolean
  def isFailure: Boolean
  def getOrElse(f: => T): T = f

}

case class Success[+T](value: T) extends Try[T] {

  override def flatMap[U](f: T => Try[U]): Try[U] =
    try {
      f(value)
    } catch {
      case NonFatal(e) => Failure(e)
    }

  override def map[U](f: T => U): Try[U] = Try[U](f(value))
  override def foreach[U](f: T => U): Unit = f(value)
  override def getOrElse(f: () => T): T = value
  override def isSuccess = true
  override def isFailure = false

}

case class Failure[T <: Nothing](exception: Throwable) extends Try[T] {

  override def flatMap[U](f: (T) => Try[U]): Try[U] = this
  override def map[U](f: (T) => U): Try[U] = this
  override def foreach[U](f: T => U): Unit = {}
  override def isSuccess = false
  override def isFailure = true

}

And again we have that same pattern, one sealed trait and two case classes. Just like Option[T], the main advantage of using Try[T] is being able to compose with the value without caring about what’s inside. It’s just like the Schrodinger’s cat story, you keep working with the box and delay opening it up until it’s completely necessary to know what’s happening in there.

And how do we do that? We map and flatMap with the computation. It doesn’t matter if it’s a Success or a Failure, when we use map and flatMap we compose on the value and wait for it to be something. Let’s look a bit at the usage of Try[T]:

"be a failure when it can't parse the number" in {
  val result = Try( "abc".toInt )
  result.isFailure must beTrue
}

"be a success when it parses the number" in {
  val result = Try( "10".toInt )
  result.isSuccess must beTrue
  result match {
    case Success(v) => v === 10
    case Failure(e) => throw new IllegalStateException("should not have come here")
  }
}

Using Try[T] is a bit different from using Either or Option because you can just wrap a computation around it instead of creating a Success or a Failure manually. Given we know some computation can fail, we can just use the Try[T].apply method at the companion object and it will automatically wrap the correct value or the error and return a Try[T] for us.

While when we write Try( "abc".toInt ) it looks like we are running "abc".toInt before sending it in as a parameter, in reality, the Scala compiler knows that we are expecting a => U function and automatically wraps that code into a closure. This is called thunking or a lazy parameter, since it is not evaluated before being given to the method. In Scala lingo, you will also see people calling this a pass by name, since, instead of sending it the computed value, you are sending in a named function that will, in turn, produce the computed value.

The NonFatal reference at the try/catch block is to mean that we want to catch any exception that is actually recoverable. Some errors (like JVM errors) are not really safe to be caught so we just use this NonFatal here at Try[T] to make sure we don’t try to catch them as well and catch only exceptions that we can recover.

And while this is good and all, you can also manually build a try from Success or Failure as well if you would like to. If you are working with concurrent systems, where you’re handling Future[T] and Promise[T] you will most likely make heavy use of Try[T] results for your computations.

Let’s do some composing:

"be composable with map" in {
  val result = Try("4".toInt).map( v => v * v ).map( v => v / 2 )

  result match {
    case Success(v) => v === 8
    case Failure(e) => throw new IllegalStateException("should not have come here")
  }
}

"be composable with flatMap/for comprehensions" in {

  val result = for (
    v <- Try("5".toInt);
    k <- Try("6".toInt);
    z <- Try("9".toInt)
  ) yield( v + k + z)

  result match {
    case Success(r) => r === 20
    case Failure(e) => throw new IllegalStateException("should not have come here")
  }
}

"fail mapping when one of the tries is a failure" in {
  val result = Try("abc".toInt).map( v => v * v)
  result.isFailure must beTrue
}

"fail flatMap when one of the tries is a failure" in {
  val result = for (
    v <- Try("5".toInt);
    k <- Try("JOE".toInt);
    z <- Try("9".toInt)
  ) yield( v + k + z)

  result.isFailure must beTrue
}

And since we have map and flatMap implemented, we can both manually map the calls and also use for comprehensions (flatMap) on our results.

And other than making it possible for you to wrap a computation that could possibly fail, the most important feature of using Try[T] is that the exception information is stored to whoever needs to unwrap the value at the end of the chain. You don’t have to do the error prone catch(Exception e) throw new SomeOtherException(e) all the way up to the code that actually needs the value (or the error), you just use a Try[T] at all the intermediate steps of the computation and the top level code that will grab the value can look at the real error (and not some infinite chain of bogus exceptions) and behave accordingly.

And what about the `M` word?

Well, if you came this far, you have seen the pattern already. Let’s look at what it looks like:

trait Monad[+T] {

  def flatMap[U]( f : (T) => Monad[U] ) : Monad[U]
  def unit(value : B) : Monad[B]

}

And there it is. Monads are containers for values and the container must have a flatMap (you can also find this being called bind) and a unit(v) operation.

The type declarations are enough for us to figure out what these two methods mean, flatMap means transforming a value that is inside a monad into another value still inside the same type of monad. Why is it necessary to keep ir wrapped? Think about Schrodinger’s cat again, we want to delay the decision of knowing what’s inside the box as much as possible so that I can do something like:

  val result = for (
    v <- Try("5".toInt);
    k <- Try("6".toInt);
    z <- Try("9".toInt)
  ) yield( v + k + z)

The same would be true if we were dealing with Option[T]. When composing many options, we might have a None somewhere but we don’t want to care about that until it is necessary to look inside the box.

Now that we know that flatMap means for a monad, what is this unit(v)? Well, it’s what you use to wrap a value. It’s the Some(v), the Success(r) and even the List(v) (yes, list is a monad as well). In Scala, all monads will have flatMap implemented directly, but each one will have it’s own version of unit(v). Usually, it’s going to be some constructor or apply method at a companion object. Still, they will have have these two operations.

And what about map? Aren’t we using map everywhere?

Well, map is just a special case for flatMap, here’s how we could write map for Option[T]:

  def map[R](f : E => R) : Option[R] = flatMap(v => Some(f(v)))

In our Monad[T] implementation this would look like:

  def map[U](f : (T) => U) : Monad[U] = flatMap(v => unit(f(v)))

Exactly the same code. So we don’t actually need map but having it defined as well is a nice shortcut for some common patterns as well.

To define something to be a monad, you use the 3 monad laws, the first is associativity:

monad.flatMap(f).flatMap(g) == monad.flatMap(v => f(v).flatMap(g)) // associativity

Using Option[T], we could have this as:

"be associative" in {
  val multiplier : Int => Option[Int] = v => Some(v * v)
  val divider : Int => Option[Int] = v => Some(v/2)
  val original = Some(10)

  original.flatMap(multiplier).flatMap(divider) ===
    original.flatMap(v => multiplier(v).flatMap(divider))
}

So, it doesn’t matter if you do the flatMap inside another flatMap or if you do it at the result produced. It has to produce exactly the same values.

The second is the left unit law, here’s the definition:

unit(x).flatMap(f) == f(x)

Which would become the following for Option[T]:

"be left unit" in {
  val multiplier : Int => Option[Int] = v => Some(v * v)
  val item = Some(10).flatMap(multiplier)

  item === multiplier(10)
}

So, getting a monad and running flatMap on it with some function is the same as calling the some function with the wrapped x value.

And the third law is the right unit law, let’s look at the definition:

monad.flatMap(unit) == monad

Which in turn would be defined as follows for Option[T]:

"be right unit" in {
  val value = Some(50).flatMap(v => Some(v))
  value === Some(50)
}

And for this last one, we mean that having a monad, running flatMap on unit, we should have the same as unit(x).

To validate these laws, you can’t just go through these unit tests, you would have to replace all function calls with the actual code (which we won’t do here) and check that the expressions generated are compatible (as you would do in a mathematical equation), but if you can guarantee that your container object respects these 3 laws then you have a monad in place.

As you can see, there isn’t anything specially complicated about monads, they’re just a container type that follows a collection of rules (which are very important, the right unit rule, for instance, allows us to use monads in for-comprehensions) and are used as container types.

As an aside, there is some debate as to if Try[U] is a full monad or not. The problem is that if you think unit(x) is Success(x), then exceptions would be raised when you try to execute the left unit law since flatMap will correctly wrap an exception but the f(x) might not be able to do it. Still, if you assume that the correct unit is Try.apply then this would not be an issue. In any case, while Try might not be a pure monad, it’s close enough so you can use it much the same way.

Didn’t see the other posts about this? Check the whole list:

For more about monads and Scala, check these ones:

Monads are elephants by James Iry - Part 1 - Part 2 - Part 3 - Part 4
Reactive programming course
Functional Programming in Scala

Using foreman and environment variables to isolate and run your apps in development

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 09 Feb 2014 00:00:00 UTC

If you’re building Rails apps you probably found yourself installing many external dependencies to run it. A database, a full text search engine, an in memory cache, background worker processes and many other tools that run as separate processes in your machine. If it’s a mac, you probably used homebrew or MacPorts to install these dependencies and at the end you did something like:

ln -sfv /usr/local/opt/postgresql/*.plist ~/Library/LaunchAgents

To install it as a service that runs when your machine starts.

But think about it for a bit. You’re not actually running this all the time. If you’re maintaining a gem that depends on a tool like that you might not even really need it for anything other than running your test suite, why should you have it running all the time like that?

Well, you don’t!

That’s where foreman comes into action. Foreman is:

a manager for Procfile-based applications. Its aim is to abstract away the details of the Procfile format, and allow you to either run your application directly or export it to some other process management format.

And what is a Procfile-based application? It’s an application that has a Procfile with instructions to start various processes it needs to run correctly. Here’s how a Procfile might look like:

rails: bundle exec rails s
postgres: postgres -D /Users/mauricio/databases/postgresql
elasticsearch: elasticsearch -f

It’s a list of name: command-to-be-run for the dependencies you have. In this case we have the rails app, a PostgreSQL database and then elasticsearch. You could have any other process you depend on running from here and never again install anything to run as a service in your machine.

Procfiles and Heroku

If you're going to deploy your app to Heroku or some other service that uses a Procfile to prepare the environment for your app you should rename your development Procfile to something else like Procfile.local or Procfile.dev so it won't be picked up by your production environment. In this case, instead of foreman start you would do foreman start -f Procfile.local.

Starting out with the database

Now that you’re not installing stuff as services anymore, you can go a step further and have isolated configurations for them as well. Let’s start with a simple Rails app:

rails new foreman-example -T -d postgresql

I’m going to assume you’re using homebrew, if you’re not, just use the command for the package manager you’re using:

brew install postgresql

Once it’s is installed, to go the Rails app directory and run:

pg_ctl init -D vendor/postgresql

This creates a full PostgreSQL database (config files and data files) inside the vendor/postgresql directory (you could even include the db version at the directory name since PG is well known for changing the DB format rather frequently). By default, PG allows access to anyone locally, if you would like to fine tune the access control, check vendor/postgresql/pg_hba.conf and update the configuration as needed. Remember, this is a full PosgreSQL install, you can change anything here and it will be visible only this app.

First service ready, let’s create our Procfile, at the root of the Rails app directory create a file called Procfile with the following content:

postgresql: postgres -D vendor/postgresql

PostgreSQL starts with only your own user included, so if you want to use the default postgres user to sign in, you must create it. First, start PostgreSQL from foreman (foreman start) and once it’s running, sign in:

psql -p 5432 -h localhost -d postgres

You must do this as the same user that created the database (your own user). Once you’re in, just create the postgres user:

CREATE USER postgres SUPERUSER;

You should now be able to sign in as postgres at this server (type \quit to exit psql). Stop foreman and let’s continue.

Redis and a master slave-setup

Now we need some in memory cache, let’s get Redis for this app:

brew install redis

Redis is even simpler than PG, you just have to reference it’s config file, and override the keys you need to change. The file is usually at /usr/local/etc/redis.conf, if it isn’t run brew info redis and check where it is. Let’s prepare the redis folder at our app:

mkdir -p vendor/redis/db

Now let’s create our vendor/redis/redis/redis.conf file:

loglevel notice
logfile ""
dir vendor/redis/db/

This guarantees you’re not writing to other places for this specific redis-server instance. Everything else assumes the defaults, you can see the defaults here.

Now let’s update our Procfile:

postgresql: postgres -D vendor/postgresql
redis: redis-server vendor/redis/redis.conf

Since we’re at it, why don’t we also configure a redis slave? Our app in production might need to send reads to a slave redis and we can just get our config to do the same here, can’t we? Let’s create a separate folder:

mkdir -p vendor/redis-slave/db

And here’s the vendor/redis-slave/redis.conf file:

loglevel notice
logfile ""
dir vendor/redis-slave/db/
slaveof localhost 6379
port 6380

This instructs this redis slave to connect to the master that will be running at the default port and bind itself at port 6380.

Here’s our Procfile updated again:

postgresql: postgres -D vendor/postgresql
redis: redis-server vendor/redis/redis.conf
redis-slave: redis-server vendor/redis-slave/redis.conf
rails: bundle exec rails s

Now we have PG, redis, a redis slave and the Rails webapp process running. Can we do more? Of course, we could include Nginx, another database, anything else you might want to do here and they will all be booted when you type this at the directory where the Procfile is:

foreman start

Given you will be writing this a lot, you should probably include an alias at your shell profile for this command, I did:

alias fs="foreman start"

So it’s just fs and everything is running.

Remember to include the vendor/redis and vendor/postgresql to your .gitignore, you don’t want to push these folders to all your colleagues. And I prefer not to force my own Profile on the whole team as well, it’s simpler to have a Procfile-example file at your repo and let people decide to use it at their own will.

Isolating the configuration

But we’re not done yet, there’s yet another trick at your disposal when using foreman, the .env files. If you’ve heard about The 12 factor app you probably know there are many uses to environment variables, what you might not know is that there are better ways than to declare all environment variables for all apps and gems that you maintain than at your shell’s profile script.

With foreman you can use .env files to declare the environment variables for your app (and it’s dependencies) and maintain them isolated from the rest of your environment. So, if you’re using the aws-sdk gem, your .env file would look like:

AWS_ACCESS_KEY_ID=this-is-some-access-key
AWS_SECRET_ACCESS_KEY=this-is-some-secret
AWS_REGION=us-west-2

And foreman automatically loads the .env file that is at the same directory as your Procfile. This way you can make all environment specific configuration for your app to live at this .env file and let every developer set their own specific configurations here. All variables declared here will be available for all processes started by foreman as environment variables.

Use it everywhere

And while I used a Rails app for this example, foreman doesn’t care about what is being started. You can run anything that can be called from the command line and run in the foreground, so you could possibly use it to run the dependencies for your Java, Ruby, Go, Python or any other language. It’s an awesome tool to have under your toolbelt wherever you go.

So, stop installing stuff as services and use foreman instead.

Foreman offers a bunch of other cool features like running many instances of the same process at the same time, setting automatic values for ports to bind and it also exports your Procfile to many other formats (like Ubuntu’s upstart), so it isn’t just for development, you can actually use it at your production environment as well. Once you’re happy with your Procfile’s, you should go to the website and dig deeper into what else it can do to help you out.

The source code for this example is here.

Never match against Ruby's default exceptions at your tests

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 01 Feb 2014 00:00:00 UTC

While working in an app using Rails 4 and RSpec’s beta version, I came across a weird bug that caused the following backtrace:

Failure/Error: expect {
  expected RSpec::Mocks::VerifyingDoubleNotDefinedError, got #<NoMethodError: undefined method `name' for #<RSpec::Mocks::NamedObjectReference:0x000001026308a0>> with backtrace:
  # ./lib/rspec/mocks/example_methods.rb:182:in `declare_verifying_double'
  # ./lib/rspec/mocks/example_methods.rb:46:in `instance_double'

Going right where the error happened I saw this:

def self.declare_verifying_double(type, ref, *args)
  if RSpec::Mocks.configuration.verify_doubled_constant_names? &&
    !ref.defined?

    raise NameError,
      "#{ref.name} is not a defined constant. " +
      "Perhaps you misspelt it? " +
      "Disable check with verify_doubled_constant_names configuration option."
  end

  declare_double(type, ref, *args)
end

So, pretty obvious, isn’t it?

It’s trying to call a name method at the double/mock ref since it isn’t defined yet, but the double doesn’t have a method called name (the correct method would be description).

Now, before going on to fix the issue, we need to write a spec that shows it happening in a controlled environment. I move on to the specs for this specific file and find this:

describe 'when verify_doubled_constant_names config option is set' do
  it 'prevents creation of instance doubles for unloaded constants' do
    expect {
      instance_double('LoadedClas')
    }.to raise_error(NameError)
  end

  it 'prevents creation of class doubles for unloaded constants' do
    expect {
      class_double('LoadedClas')
    }.to raise_error(NameError)
  end
end

Hey, there is a spec for this behaviour. Why isn’t this spec failing?

Well, that’s the catch, NameError is the superclass for NoMethodError so the match is, in a way, correct. It definitely raises a NameError but not the NameError we expected it would raise.

In this case, there are many possible solutions, we could match on the exception message, to make sure it definitely generates the exception we would want it to or we can create our own error to symbolise this specific error (and that’s what I did when I sent them a PR to fix it).

When you use a custom error to signal that something has gone wrong, it’s much less likely that you will get a false positive like this one. You know only your own code would manually raise that error (given all the other code doesn’t even know it exists) so you would be pretty much safe from falling for a case like this one.

Matching against exception messages is brittle, error prone (you will have to copy and paste the message in many different places) and leads to hard to evolve code. I had a codebase that had matches on Mongoid error messages and once we upgraded to Mongoid 3.x all of these matches failed. Not because the code wasn’t working anymore, mind you, but because the messages had changed. Don’t do it, you don’t want to be there to fix this.

So, avoid using and matching against Ruby’s default exceptions, when you need to raise something, create your own exception classes, it’s absurdly simple:

VerifyingDoubleNotDefinedError = Class.new(StandardError)

And you end up with better documentation, better tests and prevent unexpected errors like this one. Also, always be as specific as possible, if you have different errors that represent different states for your application, make sure your exceptions reflect that as well. Having a single MyAppError class for everything is hardly any better than raising Exception and StandardError all around.

Learning Scala by building Scala - Lists and Option

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 25 Dec 2013 00:00:00 UTC

And off we go to build a couple more list operations and understand new concepts and styles of working with collections and values in Scala. If you didn’t see the fist two parts of of this tutorial, check them below before moving forward:

Finding an item in a list given a predicate

At part 2 we saw how we could filter a list given a predicate function, but what if I wanted to find and get an item given a predicate?

Imagine I have a list of Person instances and I’d like to find someone with the Josh first name. I could use filter for this and grab the first item in the list, but that would be awkward. How are people usually doing it?

Let’s see how this would look like in Ruby:

found = people.find { |p| p.first_name == 'Josh' }
puts "#{found.first_name} #{found.last_name}"

As you can see, just like we had a filter method at our LinkedList, the find method for enumerables in Ruby accept a predicate function and will return the first item for which the block returns true.

But, what if the block never returns true for any of the objects?

Well, then you would get nil (the null in Ruby).

But what if you wanted to find the nil inside an enumerable?

items = [1, nil, 2]
found = items.find { |p| p == nil }
puts found

For this case, the find method accepts a parameter to signal the lack of a value, so we could write the code like this:

items = [1, nil, 2]
found = items.find(0) { |p| p == nil }
if found == 0
  puts "There are no nils"
else
  puts "There is a nil"
end

This special value carries no meaning on itself, it’s just a magical value we decided to use so we could carry on building our code, but it’s an error prone solution and would have to be documented somewhere out of the code so someone calling our code understands what is happening and we know this is a bad sign.

You can see these special values in many places, if you know Java you know the indexOf method in String returns -1 if the string given is not included anywhere and, again, you need to read the docs to know that this is the case and handle it correctly.

Can we do better? Of course we can!

Option comes to the rescue

Our problem is that we need to find a way to say that something has two states, it either has a value or it doesn’t. This idea is prevalent in functional programming in general and the solution is the Option (ML, Scala) or Maybe (Haskell) type. This type encapsulates the idea of having or not having a value. Let’s see how we can implement it in Scala:

sealed trait Option[+E]

case class Some[+E]( element : E ) extends Option[E]
case object None extends Option[Nothing]

Looks a bit like our LinkedList declaration, doesn’t it?

This is almost like declaring a marker type so you can know that what is being returned could either be a value or be nothing and that’s exactly what this is for, to taint your code in a way that you can’t ignore that not having a value is a possibility, it’s encoded right into the types, you don’t have to read the docs to know this, it’s already there in your code to be seen.

Now let’s see how our find method could be implemented:

@tailrec final def find( p : (E) => Boolean  ) : Option[E] = {
  this match {
    case Node( head, tail ) => {
      if ( p(head) ) {
        Some(head)
      } else {
        tail.find(p)
      }
    }
    case Empty => None
  }
}

The implementation is extremely simple, check if the head of the current node matches the predicate, if it matches, return a Some(head), if it doesn’t call find on the tail. If we have reached the end of the list, return None. Let’s see how we can use this in real code:

"find John" in {
  val items = LinkedList("John", "Josh", "Mary")
  items.find( name => name == "John" ) === Some("John")
}

"not find John" in {
  val items = LinkedList("Josh", "Mary")
  items.find( name => name == "John" ) === None
}

"find with pattern matching" in {
  val items = LinkedList("Josh", "Mary")
  items.find( name => name == "Mary" ) match {
    case Some(item) => success
    case None => failure("Should not have come here")
  }
}

And other than the last case, when we pattern match, doesn’t look like this is super cool, does it?

Option as a one item collection

Well, it isn’t. The coolness of Option comes when we think about it as a collection that can hold at most one value and compose on this idea. Let’s add a couple methods to it.

sealed trait Option[+E] {
  def isDefined : Boolean

  def map[R](f : (E) => R) : Option[R]
  def flatMap[R]( f : E => Option[R] ) : Option[R] = if ( isDefined ) f(this.get) else None
  def foreach[U]( f : (E) => U )
  def get() : E
  def getOrElse[B >: E]( f : => B ) : B = if ( isDefined ) get() else f
}

case class Some[+E]( element : E ) extends Option[E] {
  override val isDefined = true
  override def map[R](f : (E) => R) : Option[R] = Some(f(element))
  override def foreach[U]( f : (E) => U ) = f(element)
  override def get() : E = element
}

case object None extends Option[Nothing] {
  override val isDefined = false
  override def map[R](f : (Nothing) => R) : Option[R] = None
  override def foreach[U]( f : (Nothing) => U ) = {}
  override def get() : Nothing = throw new NoSuchElementException("There is no object here, this is a None")
}

Now we clearly have some useful code around here, let’s look at how we could use all this:

"option" should {

  "be something" in {
    val item = Some("10")
    item.get() === "10"
  }

  "be mapped to some value in a for comprehension" in  {
    val upper = for ( name <- Some("Joe") ) yield name.toUpperCase
    upper.get() === "JOE"
  }

  "be mapped to some value manually" in {
    val number = Some("10").map(n => n.toInt)
    number.get() === 10
  }

  "it can't be anything if it is a none" in {
    val something : Option[Int] = None
    val result = something.map( x => x * 5)
    result.isDefined must beFalse
  }

}

The main takeaway here is that by thinking about Option as a collection with at most one element, we can compose on it as if it is any collection, but with some special knowledge.

At the beginning of the Option declaration we have the isDefined method that tells us if this option is a Some or a None, it’s mostly a shortcut to pattern matching. Once we go on, we get to the meat of our implementation, the map method. This is what allows us to compose with the option and ignore if it’s something or empty until we really have to care about it.

This means we can either go for the no-sugar version that calls map directly:

val number = Some("10").map(n => n.toInt)
number.get() === 10

Or we can use the sugarized version with for-comprehensions:

val upper = for ( name <- Some("Joe") ) yield name.toUpperCase
upper.get() === "JOE"

Both have the same effect, but if you have to handle many options, the for comprehension would looks nicer:

"find the numbers with for comprehension" in {
  val numbers = LinkedList(1, 2, 3, 4, 5)

  val result = for {one <- numbers.find(x => x == 1)
                    two <- numbers.find(x => x == 2)
                    moreThan4 <- numbers.find(x => x > 4)
  } yield one + two + moreThan4

  result.get() === 8
}

This comprehension means I want to add the 1, the 2 and the first number that’s bigger than 4 and only that. If any of these numbers aren’t there, give me a none. Think about it a bit, without Option, you would have a collection of nested if statements there, but with Option you don’t have to care about that anymore. If any of the find calls returns None the whole operation becomes a None and all other operations will not happen anymore.

Without for-comprehensions the code is a bit uglier:

"find the numbers flatMapping" in {
  val numbers = LinkedList(1, 2, 3, 4, 5)

  val result = numbers.find(x => x == 1)
    .flatMap(one =>
    numbers.find(x => x == 2)
      .flatMap(
      two => numbers.find(x => x > 4).map(
        moreThan4 =>
          one + two + moreThan4)))

  result.get() === 8
}

Not as good as the original, is it?

There’s also something else here, flatMap. Why is it here? Because we need to flatten our options, otherwise the result of running this operation would be Option[Option[Option[Int]]] and that’s not what we want and to fix this we use flatMap that flattens or unwraps our Option[Option[Int]] to an Option[Int]. Again, using the for-comprehension prevents us from having to write these transformations manually, but map and flatMap are both required if you want to use them, the for-comprehension itself is just a simpler way to write the code above.

And as I mentioned above, if any of the parts becomes None, it all becomes None:

"wont find anything" in {
  val numbers = LinkedList(1, 2, 3, 4, 5)

  val result = for {one <- numbers.find(x => x == 1)
                    two <- numbers.find(x => x == 2)
                    moreThan4 <- numbers.find(x => x > 5)
  } yield one + two + moreThan4

  result === None
}

It doesn’t matter if it’s something or empty, just map on it and keep composing on the options you get back until, eventually, you have to grab a value out of the option. In most cases, you can and should defer the decision to take a value out until there’s no way to send an option and, in this case, you probably want to use getOrElse instead of get. get, as you can see from None’s implementation, isn’t safe and will cause side effects it not handled correctly, so avoid doing it unless it’s really necessary or you don’t care about the possibility of raising an exception.

getOrElse and lazyness

And while we’re at getOrElse, the idea behind it is as simple as it’s implementation. If you really need to have a value out of something that could be empty, you definitely need a default value, by using the getOrElse you can now have this default available directly here instead of waiting somewhere else to be used and you can also make sure that it will be only at the place where it is necessary, and not scattered everywhere in your codebase.

Also, look at the way we have defined getOrElse:

def getOrElse[B >: E]( f : => B ) : B = if ( isDefined ) get() else f

The or else value isn’t a value per se, it’s a function. Why is that? Because we want to be lazy, but in a good way. Think about it, if the option is a Some this value will never be used, does it make sense to create this value in all cases? No, it doesn’t, so, instead of making getOrElse take a value, it takes a function that returns a value. Here’s how it looks when we use it:

"be getOrElse the string" in {
  val item = Some("10")
  item.getOrElse("25") === "10"
}

Well, it doesn’t looks like I’m calling a function, does it? It’s because the Scala compiler is smart enough to understand that even thought I did type just a string, what I mean is that I want a function that returns a string and it will generate just that. So it looks like that string is being created every time, but it isn’t, it will only be created when the option is a None so we don’t incur in the object creation penalty every time we use getOrElse.

And there is more

And let’s not forget foreach that is, again, mostly for side effects. If you don’t care about composing an option and all you want is to run some code if there is something in there, just use foreach and be happy with it. If there is something, your block of code will be called, if there isn’t, nothing will happen.

So, after all that, here are some of the advantages of using Option in your codebase:

You don’t have to document the fact that calling a method could return something or not. When you say it returns Option[Something] whoever is calling it knows it already and will have to handle it carefully. You have baked type safety inside this operation.
You can simplify your conditionals by building your code upon composing options instead infinitely nested operations, this makes for code that’s easier to read and to reason about.
If you are always composing on options and don’t need special values, like null, you can forget null handling! Yes, by using Option, your code should be able to live and run without caring about the existence of null. If you walk around Scala’s standard library and most third party libraries you will see that whenever something would be handled by returning something or null, it will return Option, so this idiom is pervasive in the ecosystem and you should use it as much as possible as well.

Code as the documentation

The real beauty here is that we have encoded something that was usually just some documentation lying around into real code that can be compiled and tested. Now, whenever you see Option being used in Scala, you know what it means and how you are supposed to use it in your code, there’s no need to check the documentation to figure out “what happens if nothing is found” since you already know None happens.

There are a bunch of other objects like Option at the Scala standard library, like Either and Try and we will cover them in the future, for now, enjoy and use Option a lot.

Learning Scala by building Scala - Lists Part 2

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 08 Dec 2013 00:00:00 UTC

Here we continue our journey to learn more about Scala and functional programming using lists. If you haven’t seen the first part of this tutorial, you should probably read it first before reading this part. Now let’s get back to learn more list operations.

The full source code for these examples can be found here.

Check all the other parts for this tutorial below:

Reversing a list

Reversing a linked list is a really simple solution given what we have built already, here’s how it looks being used:

"Reverse a list" in {
  val original = LinkedList(1, 2, 3, 4, 5)
  original.reverse() === LinkedList(5, 4, 3, 2, 1)
}

The use of () here is mostly personal, I like to parenthesise the use of functions when they are building new objects instead of accessing already available information. You could remove the parenthesis from the method definition if you would like to and then you could remove them here when calling it.

Now let’s get to the implementation:

def reverse() : LinkedList[E] = {
  foldLeft(LinkedList[E]()) {
    (acc, item) =>
      Node(item, acc)
  }
}

As you can see, the implementation is quite simple, to reverse a list you just foldLeft on it creating a new list on every element. Since foldLeft goes from left to right and LinkedList objects are built from right to left (first we have a tail and then we have the objects) this is the simplest algorithm possible.

The only weird thing we have is is the call to LinkedList[E](), this is necessary because type inference for the compiler won’t work if you use Empty, so we need to explicitly say we want to build an empty list of type [E].

This is one of the beauties of functional programming, you just compose functions transforming values using the base building blocks provided to create new functionality.

Folding to the right

Just as we had a foldLeft our list also needs a foldRight function that builds composes from right to left. Why would we need something like that?, you might think and the main reason is that not all binary operations are commutative. That is, the order of the operands affects the result of the operation.

When we used foldLeft to sum LinkedList(1, 2, 3, 4, 5) doing the sum from the left or right didn’t matter, both 1 + 2 + 3 + 4 + 5 and 5 + 4 + 3 + 2 + 1 yield the same result. But what if it was a division? Is 4/2 the same as 2/4? They’re not and that’s why we also need a function that composes from the right, we’ll also see that we also need it to implement other functions in an efficient way, otherwise they wouldn’t work for very large inputs and we will later re-implement map in terms of foldLeft for the same reason.

Let’s see how we can implement foldRight:

def foldRight[B]( accumulator : B )(f : (E,B) => B) : B = {
  reverse().foldLeft(accumulator)((acc,item) => f(item,acc))
}

Now you must be thinking come on, this is cheating but that’s the most efficient way to implement a foldRight given our data structure. Since we don’t have fast indexed access to elements (it’s a singly linked list), we would have to recurse N levels (with N being the size of the list) to access the last element and then backtrack until the very first piece of the recursion.

As you can imagine, this would be very inefficient and also fail for large inputs with a stack overflow error so we just reverse the list (which is tail recursive since it uses foldLeft) and we foldLeft on the result. And folding to the left at the reverse of a list is, well, folding right. In this case, two lefts really make a right.

Also, in Scala, the foldRight function for collections reverse the arguments at the binary operation. While in a foldLeft the operation is (accumulator,item) in a foldRight it becomes (item, accumulator) so I have made it the same order here so you don’t get confused when using the real foldRight in collections.

If you’re into algorithms, you might say that instead of doing this stuff in O(N) time we now do it in O(2N) and this is true. Unfortunately, there aren’t many options given the way our data structure is built, we gave up the O(N) time to allow our code to handle large inputs without overflowing. If you had a list with fast indexed access or a doubly linked list you would definitely be able to traverse to the right in a tail recursive way.

Filtering lists

To filter a list we give it a predicate function to create a new list that contains only objects that return true on the predicate. A function that takes another function is so common in functional programming (as you have seen from the previous map, foldLeft and foldRight operations) that there is a special name for it, it’s a higher order function. A function that takes another function and/or produces a function as it’s result is said to be higher order (yeah, being higher order isn’t some absurdly complicated concept).

Let’s look at how we would like it to look:

"filter a list" in {
  val original = LinkedList(1, 2, 3, 4, 5)
  original.filter(x => (x % 2) == 0) === LinkedList(2, 4)
}

So you have a list, call filter on it sending in a function to say what kind of object you want at your filtered list. In this case, we want all even numbers and at the end we get LinkedList(2,4). Let’s look at the implementation now:

def filter( f : (E) => Boolean ) : LinkedList[E] = {
  foldRight(LinkedList[E]()) {
    (item, acc) =>
      if ( f(item) ) {
        Node(item, acc)
      } else {
        acc
      }
  }
}

Look, foldRight again!

Now, why didn’t we do this with a foldLeft in the first place? It’s much more efficient than a foldRight.

The main reason for this is maintaing order. If we used a foldLeft here we would have LinkedList(4,2) as a result and this would change the order the objects were presented on our list and this would be an undesirable side effect. By using a foldRight we now have exactly the output we want, which is the items filtered at the same order they were found in the list.

And our function itself doesn’t do much, it checks if the current item matches the predicate function f, if it does, includes it at the head of the accumulator list, otherwise returns the accumulator itself. Again, we’re just composing stuff and nothing more.

Rebuilding map

And since we now have a foldRight let’s reimplement our map method to use that instead of manually doing the looping, here’s how it would look like:

def map[R](f: E => R): LinkedList[R] = foldRight(LinkedList[R]()) {
  (item, acc) =>
    Node(f(item), acc)
}

Simple, isn’t it?

Again, we use a foldRight here because we need to maintain the order of the elements, we can’t return a reversed result list to users of this method.

Calculating the size of a list

To calculate the size of a linked list you usually need to traverse the whole list counting how many nodes it has, since you don’t know how many nodes were added to it. You could hide this behind a list type that would hide the way the nodes are built and increment the size whenever a new item is added, but since our LinkedList implementation is immutable we can do better, we can just have the size calculated automatically for us in a single operation.

First, we declare a property size at the LinkedList trait:

sealed trait LinkedList[+E] {
  def size : Int 
  // ... other methods implemented here
}

And we include implementations at the Node class:

case class Node[+E](head: E, tail: LinkedList[E]) extends LinkedList[E] {
  val size = 1 + tail.size
}

And in at Empty as well:

case object Empty extends LinkedList[Nothing] {
  val size = 0
}

Since you can’t add items at the end of a LinkedList objects, the size of a list is known at the time it is created, it’s always 1 + tail size and given the tail has already been completely initialised, it has this value already calculated and you can just get it, so, calculating the size of our list is s simple operation.

Now, as you have noticed, I used def to declare the method at the LinkedList trait but at the classes I have used val. Why is that? Declaring val’s in traits is usually frowned upon, you use a trait to include more operations, not to hold state, so while we declare it as a def at the parent trait, our classes define it as vals and for Scala it doesn’t matter, a method that doesn’t take any arguments can safely be replaced by a val anywhere and callers of the code don’t have to know anything about this. Also, if it was a def at Node and Empty then our implementation would not be optimised at all and would still traverse the whole list. Since we made it a val at both ends it is initialised when the object is created and never changed again.

Cons operator (::)

If you hanged around functional code before you probably saw colons around code, both as :: or as :::, in functional programming these are known as the cons operator (this name comes from LISP but ML and Scala refer to it as cons as well). We use this to build lists and it’s special because it’s right associative in Scala. Since it’s right associative, we can build lists like this:

"build lists with cons" in {
  LinkedList(1, 2, 3, 4) == 1 :: 2 :: 3 :: 4 :: Empty
}

This code, if calling the :: directly instead of like an operator would be:

"build lists with cons manually" in {
  LinkedList(1, 2, 3, 4) == Empty.::(4).::(3).::(2).::(1)
}

In Scala, any operator that ends with : (colon) becomes right associative. This way, when you build a list using it, it looks like the final result, given the list is built backwards (starting from the tail) but we traverse it forward. Let’s look at how the method is implemented:

  def ::[B >: E](element : B) : LinkedList[B] = Node(element, this)

Not much there to be said, we could just call this method add, it’s :: mostly to make use of the right associativity to look nicer and to pay homage to the ML language that influenced Scala syntax and style so much. The generic declaration [B >: E] is there to make the Empty case possible given when you do Empty.::(1) you are effectively building a new list of Int out of a LinkedList[Nothing] and you need to say it’s valid to create a new list of type B given B is a super type of E as E in this case is Nothing, the subtype of all types. So [B >: E] means that B is any super type of E.

The other well known case is the ::: operator that includes a list in front of the other, here’s how we use it:

"appending two lists" in {
  val current = LinkedList(1, 2, 3, 4)
  val other = LinkedList(10, 11, 12, 13)

  ( other ::: current ) === LinkedList(10, 11, 12, 13, 1, 2, 3, 4)
}

Just like :: this one is also right associative, so it’s the same as doing current.:::(other), we use the right associativity here mostly to show that other will be prefixed to current.

The implementation includes something new for us:

def :::[B >: E](prefix : LinkedList[B]) : LinkedList[B] = {

  @tailrec def helper(acc : LinkedList[B], other : LinkedList[B]) : LinkedList[B] = {
    other match {
      case Node(head,tail) => helper(head :: acc, tail)
      case Empty => acc
    }
  }

  helper(this, prefix.reverse())
}

Given we can’t make ::: tail recursive per se, what we do here is yet another common idiom you will find in functional languages, an internal helper method. Now the helper method can easily be made tail recursive so we use this to optimize our implementation.

There isn’t anything special about internal helper methods, it’s just like any other method, but it’s only visible here inside our method and not outside of it, given no one really needs to know it exists and it’s only useful in this specific case, that’s the perfect place for it to be.

Looping through the elements of a list

And while we covered most pieces of how you can build new stuff out of lists, we didn’t look at how we can simply loop through a list. If you have a list and all you want to do is loop through the elements without producing a new value or a new list, you’re probably looking at a side effect.

This isn’t inherently bad or evil in any way, it’s just that, different from pure functions, a function that causes a side effect depends on context or information that’s out of the function itself and this usually makes them harder to test and understand that they are doing. Still, side effects are the main way our programs interact with external systems, so there’s no hiding from them, you will eventually need to write code that causes side effects and this is fine, just keep in mind the downsides of it while doing it.

In Scala collections, the method to loop through elements is foreach, here’s how it could be implemented:

def foreach(f : (E) => Unit) {
  @tailrec def loop( items : LinkedList[E] ) {
    items match {
      case Node(head,tail) => {
        f(head)
        loop(tail)
      }
      case Empty => {}
    }
  }

  loop(this)
}

It takes a function as a parameter and calls the function with each element contained in the list itself. Again, we use a helper method here to do the magic and be tail recursive, we could have implemented it with foreach itself being tail recursive but this just looks a bit cooler (yeah, I do like ML a lot).

And since we’re all about side effects now, our test needs to cause side effects as well, let’s look at it:

"foreach implementation" in {
  val items = new ListBuffer[Int]()

  LinkedList(1, 2, 3, 4).foreach( (x) => items += x )

  items === List(1, 2, 3, 4)
}

And here we are accessing a variable out of the loop and mutating it to be able to test our implementation, not ideal, but that’s what we wanted in any case.

Wrapping up

Now we know more about the trade offs we have made when we decided to build our LinkedList object as immutable, we optimised it to be tail recursive everywhere, included a couple new operations like filter, foldRight, foreach and know we know that any operator that ends in : (colon) is right associative.

At the next part, we’ll dig deeper into how lists interact with other functional concepts in the language.

Learning Scala by building Scala - Lists

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 25 Nov 2013 00:00:00 UTC

While trying to come up with a compelling reason to get back at blogging I thought about all I’ve been learning over these last few months and it’s all about functional programming. Coursera courses, books, learning new languages, it all boils down to learning how you can build programs the functional way, so, why not put a bit of all this into a couple blog posts and, better yet, use Scala for it?

After doing a bunch of Standard ML and Racket for Dan Grossman’s Programming Languages course, it’s sad to see that most of the mainstream languages are still so far away in terms of features and functionalities when compared to something as ancient as ML (from the early 70’s!). It’s great to see the resurgence of these ideas in languages like Scala and Clojure and if you haven’t jumped the bandwagon yet, it’s time for you to do it :)

The full source code for these examples can be found here.

Setup

To follow this tutorial you will need SBT and I’d recommend you to use an IDE, but you can definitely use SBT to compile and run the project if you’d prefer to use a basic text editor. You can check a list of IDE’s and plugins available at Typesafe’s website.

Now reate a file at the PROJECT_ROOT/build.sbt path with the following contents (assume PROJECT_ROOT is the folder you created to follow this tutorial):

name := "list-tutorial"

version := "1.0"

scalaVersion := "2.10.3"

libraryDependencies ++= Seq(
    "org.specs2" %% "specs2" % "2.3.4" % "test"
  )

This is our basic project configuration, we will only have Specs2 as a dependency for this tutorial. If you’re using Eclipse or IntelliJ, you should now create the PROJECT_ROOT/project/plugins.sbt file with the following content:

addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")

This includes plugins to generate project files for both IDEs. If you’re using Eclipse, just type sbt eclipse and import the project, if you’re on IntelliJ, just use sbt gen-idea and import the project. If you’re not using any of them, use sbt build and sbt test to compile and run tests for your project.

Lists, camera, action!

Lists are one of the basic constructs of functional programming and it couldn’t be different in Scala. Let’s start by defining our LinkedList object:

sealed trait LinkedList[+E]

case class Node[+E]( val head : E, val tail : LinkedList[E]  ) extends LinkedList[E]

case object Empty extends LinkedList[Nothing]

Now we have a bunch of concepts here, first, there’s the sealed trait. Traits, in Scala, are a mix of Java interfaces and Ruby modules. You can define a collection of methods you expect people to implement for your trait and you can also have base implementations available at your trait so that clients won’t have to implement them all.

In this specific case, I’m using a trait because I don’t want people to be able to create LinkedList objects directly. This could also be done by using an abstract class, but traits are just simpler.

Now, what about sealed? The LinkedList object is not open for extension, we don’t want people to create their own derivations of it. By making it sealed we let the compiler know that only classes inside the same file as the declaration of LinkedList can inherit/implement it. Now the only two possible subclasses for LinkedList are Node[+E] and Empty.

As for the [+E], it’s the type parameter for the LinkedList type so we can make sure we hold only one kind of object there. If you haven’t seen type parameters or generics yet, don’t worry, this won’t be super important for what we’re doing here.

Moving on, now we have Node[+E]. First, what’s a case class? A case class is a class that allows you easily pattern match on it using it’s constructor. You could manually do it, but by defining it as a case class you can forget about all that and just use all the pattern matching magic we will see below.

And while I’m talking about constructors, can you spot the constructor for the Node[E] class? Look again:

( val head : E, val tail : LinkedList[E]  )

This is the constructor for the Node[+E] class. In Scala, the constructor is part of the class declaration itself. We say this is the primary constructor, all other constructors you define for this class will have to call this primary constructor in some way, so you are forced to have a single point of entry for creating your class and this is a good thing, you would usually have to do the same manually if you were in a language like Java or C# and this best practice was translated to a language requirement.

Since linked lists are a recursive structure at heart, our Node[+E] definition reflects just that, you have a head field that is an object of type E and a tail that is just another LinkedList[+E] object. Just like you would have implemented a linked list in any other language.

What about this val thing?

val means this value is immutable. In Scala and in functional programming in general, you will see that there is a lot of value in keeping stuff immutable. It’s much simpler to reason about code when you don’t have to care if the values are changing or not, specially if you’re writing multi-threaded or concurrent code. So, whenever possible, make use of val to declare your variables (yeah, bad name here).

When declaring case classes, all fields are assumed to be vals, so I’m using it here just so you know this is the default. You can just remove it and they will still be immutable.

And if you’re used to C-like languages the way the type is declared for the variable might feel weird, first you say the name of the variable (head in this case) and then you do : E to declare it’s type.

Why is it like that?

Because Scala has local type inference, when you declare a variable inside a method or object body you can do it like this:

val number = 1

Given the right hand side, Scala already knows you have a number there, so you don’t have to type the variable type, but you could if you wanted to:

val number : Int = 1

So declaring the type after the variable makes more sense than doing it before when you have type inference.

The extends just lets us know that we are extending/implementing the LinkedList[E] trait.

Now on to the trickiest part of this all, Empty. Empty is not a class, it’s a singleton object (we do not use class in it’s declaration). In Scala, we don’t have static methods or variables, everything has to be at an instance of something and Empty is the value we will use to implement our list terminator. It would be equivalent of using a global variable or null to signal the end of a list, but without the drawbacks of these options. Imagine you use null as the terminator for a list, what would you do if the user wanted to insert null on it? Yeah, it’s tricky. With this singleton object we wouldn’t have this kind of problem.

As you can see, it doesn’t hold any values and it’s type parameter is Nothing. In Scala, Nothing is the bottom type of all types. It’s meant to be used in these special cases of generics when you want to signal the lack of a value but still be type safe and it works because the compiler assumes Nothing is the subtype for all types. That’s also why our class’ type parameter is defined as [+E] and not [E] only, we accept objects of type E and it’s subclasses (so we can accept Nothing, the subclass of everything).

But don’t worry too much about it, generics won’t affect us much here.

Operation time

Now that we have our data structure defined, let’s implement some operations we would like to see in our list.

Creating lists of items

First, let’s have something to help us create new lists in a simple way, let’s allow the user to create a new list object by calling LinkedList(1, 2, 3, 4). Here’s how you would do it:

object LinkedList {

  def apply[E]( items : E* ) : LinkedList[E] = {
    if (items.isEmpty) {
      Empty
    } else {
      Node( items.head, apply(items.tail : _*) )
    }
  }

}

In Scala, when you do something(), you’re actually calling the apply() method. It’s just syntax sugar around the call. And as we have seen before, since Scala doesn’t have static methods, we declare a singleton object called LinkedList (given there is a trait named LinkedList, we say this is the companion object for LinkedList) and now we can declare any methods that would not make sense at LinkedList instances here.

In Scala, to declare the type a method returns, we use the syntax : Type = which is exactly the same we would use for an assignment. We could remove the : Type here given the compiler can infer the type of the expression, but I personally like to have method return types defined so it’s easier for someone else to look at them and see what they return.

Since Node is a case class we could either call the constructor new Node(head,tail) or we can use the shortcut the case class include that is an apply method that takes the same parameters as the constructor, like we are doing at this example.

The implementation itself is quite simple, we have the parameter items : E*, which is a vararg (the * denotes it’s a sequence of Es) and we use it to recursively build the list. The : _* is required for us to tell the compiler we are not sending in a sequence of E as the first parameter but the sequence is all the parameters (it’s equivalent to the * operator in Ruby).

Mapping one list into another

What is the goal of the map method? To transform a data structure by applying a definite operation on every element of the structure and returning a new structure with the result. So, if we had the following list (2,3,4) and the map operation is squaring an item then it would produce the list (4,9,16).

Let’s start by writing our fist specification for the behaviour we want to see (this class will be created under src/scala/test):

import org.specs2.mutable.Specification

class LinkedListSpecification extends Specification {

  "linked list" should {

    "map correctly" in {
      val original = LinkedList(2, 3, 4)
      original.map(x => x * x) === LinkedList(4, 9, 16)
    }

  }

}

In Specs2 parlance the === operator is equivalent to JUnit’s Assert.assertEqual or RSpec’s should == . If the items are not equal it would raise an exception. You can find a full list of matchers for Specs here.

This is the usage we expect to have, on any LinkedList object we should be able to give it an operation and this operation will be applied to all items and we will get back a new list with the produced objects. One upside we have for free since we made Node and Empty case classes is that they already have an == implementation that compares their internal properties, so as along as our lists contain objects that correct implement the == operator we don’t have to care much about implementing it or the hashCode method.

Now, let’s see how we could implement the map method:

sealed trait LinkedList[+E] {

  def map[R]( f : E => R ) : LinkedList[R] = {
    this match {
      case Node(head, tail) => Node(f(head), tail.map(f))
      case Empty => Empty
    }
  }

}

map takes a single parameter, a function f with type E => R. This means it’s a function that takes an item of type E (the type we hold at this list) and turns it into some type R. The => signals to Scala that we are taking a function here. And the return type is a LinkedList[R] since our function turns each E item into an R item.

At our implementation we see our first case of pattern matching in Scala, using the match keyword. We are matching on the this, the list itself, and if it’s a Node we build a new node by applying f at the head and then mapping again on the tail. If it’s Empty we just return the Empty list since we don’t have more items to apply f.

The case Node(head, tail) is possible because the Node class is a case class that has already implemented the pattern matching logic for us on it’s constructor so the pattern matcher implementation will bind the constructor fields to the variables we declare there. They didn’t have to be called head, tail, I could just call them x and xs and it would have the same effect. What’s important here is that we don’t need to manually switch or check both cases, the match/case construct and our case class will do it all for us.

If we wanted to implement this without pattern matching we could also have done it with subclassing:

sealed trait LinkedList[+E] {
  def map[R]( f : E => R ) : LinkedList[R]
}

case class Node[+E]( val head : E, val tail : LinkedList[E]  ) extends LinkedList[E] {
  override def map[R]( f : E => R ) : LinkedList[R] = Node(f(head), tail.map(f))
}

case object Empty extends LinkedList[Nothing] {
  override def map[R]( f : Nothing => R ) : LinkedList[R] = this
}

It would, mostly, have the same effect, but you’d have to implement the method at both places. Given we already know all subclasses we will have for our LinkedList we will just use pattern matching for all examples here.

Another important thing on this second possible solution is that we did not wrap our method implementations around {}, at Node we just have Node(f(head), tail.map(f)) and that’s it. Scala does not require you to use {} to wrap around a method body as long as each expression evaluates to something that returns the expected value. So, if you’re sick of angle brackets, you can just remove them if the expressions you are using allow you to.

Almost all examples you see here could have their {} removed from the method declarations. Try removing them later and decide what you think is better for your reading.

Folding over data

Imagine you want to sum all numbers in a list of numbers, here’s how you could do it:

def sum( numbers : LinkedList[Int] ) : Int = {
  numbers match {
    case Node(head, tail) => head + sum(tail)
    case Empty => 0
  }
}

Simple, isn’t it? Now what if you wanted to do the same with a list of strings?

def join( numbers : LinkedList[String] ) : String = {
  numbers match {
    case Node(head, tail) => head + join(tail)
    case Empty => ""
  }
}

Can you see the pattern here? We have a neutral value and we take the current item and joint it with the recursive call with the tail of our list. These are classic examples of a fold operation.

Fold is usually defined as a binary operator you apply to an accumulator and to a value in your collection, producing a new accumulator for the next operation. Basically, it’s a way to abstract the algorithm above so you can use it with any type. Let’s look at how we could define a foldLeft method at our LinkedList trait:

sealed trait LinkedList [+E] {

  // map implementation above here

  @tailrec final def foldLeft[B]( accumulator : B )( f : (B,E) => B ) : B = {
    this match {
      case Node(head, tail) => {
        val current = f(accumulator, head)
        tail.foldLeft(current)(f)
      }
      case Empty => accumulator
    }
  }
  
}

We have a bunch of new stuff here. Let’s start with this @tailrec annotation. The JVM doesn’t implement the tail call optimization natively, which means recursive functions like the map above or this foldLeft could cause stack overflow errors for large collections, since each recursive call allocates a new stack element and there is a limit on how deep the stack can get.

So, instead of giving up on recursion completely, if your method is tail-recursive (it calls itself at the end of one of it’s branches) you can annotate it with @tailrec and the Scala compiler will optimize it to a loop for you. Whenever you’re building a recursive function, try to make it tail-recursive so you can let the compiler optimize it for you. One requirement of applying the tail call optimization is that the function itself has to be marked final (and this is yet another reason we need to declare our methods at LikedList and not at it’s subclasses).

Now, getting to the method, instead of just calling it fold, we call it foldLeft because we fold to the left. If we manually applied the calls of summing the list LinkedList(1, 2, 3) they would be f(f(f(0, 1), 2), 3) so we apply the function form the left to the right. Why is it important? Because you could have built a foldRight that applies from right to left and some operations could yield a different result if you’re coming form the right (like merging strings or a division).

Now let’s get to the list of parameters:

( accumulator : B )( f : (B,E) => B )

We have two separate lists of parameters here, the first contains the initial value and the second contains the function we will use to fold over the data. Having separate lists of parameters allows us to curry methods calls, that is applying a function with only part of the parameters producing a new function that takes the missing parameters.

I’ll go deeper into this concept in future posts, this is visible here mostly because this is the same signature that you will see at the default Scala collections and it would probably be weird if I had one signature here and when you started to use real Scala lists you saw a different one.

As for the implementation itself, there isn’t much here. It’s a common recursive function where we compute the next accumulator and send it forward with a recursive call. If we have reached the end of the list, just return the current accumulator itself.

Let’s see how our spec for it would look:

"foldLeft summing all numbers" in {
  val original = LinkedList(2, 3, 4)
  original.foldLeft(0)((acc,item) => acc + item) === 9
}

"foldLeft making a single string from all numbers" in {
  val original = LinkedList(2, 3, 4)
  original.foldLeft(new StringBuilder())((acc,item) => acc.append(item) ).toString() === "234"
}

"map and foldLeft to sum the squares" in {
  val original = LinkedList(2, 3, 4)
  original.map(x => x * x).foldLeft(0)((acc, x) => acc + x) === 29
}

As you can see, we can reuse the foldLeft operation for whenever we need to build one object out of the other. We could even implement the map function above in terms of a fold!

The other important piece here is that we need to give the parameters in separate just like we declare them separately. So it’s foldLeft, the accumulator or neutral object and then the function we are going to apply.

More next time

And we have covered lists, map and foldLeft here, all very important constructs when doing functional programming and working with Scala in general. Next we will move on to a couple other methods on lists and more functional programming constructs. Stay tuned!

Done here? Head to part 2 now!

And all the others as well:

References

If you’d like to dig deeper into all this, check out these resources:

mysql-async and postgresql-async 0.2.3 released!

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 18 May 2013 00:00:00 UTC

And a newer version of the postgresql-async has finally been released and now the project includes mysql-async, an async client for the MySQL database (you can read a bit more about some of the complications of building the MySQL client at this blog post).

Both clients work almost the same way and they both implement the Connection interface so it should not be that complicated to share implementations as long as you pay attention to the database differences. Both projects now have their own README files, this is the PostgreSQL one and this is the MySQL one.

The PostgreSQL version hasn’t seen a lot of action since I was focused on getting the MySQL client to the same level of funcionality, but there is one new feature, that is the support for Array[Byte] parameters for PostgreSQL 9 and above. Support for older versions is planned for future releases. There were also many bugfixes, including contributions from @fwbrazil, check the CHANGELOG for more details.

The Play 2 + postgresql-async on Heroku example has also been updated with the new renamed classes, give it a go and start using async database access in your apps :)

You can add both projects to your app using the dependencies:

"com.github.mauricio" %% "postgresql-async" % "0.2.3"

Or if you are into MySQL:

"com.github.mauricio" %% "mysql-async" % "0.2.3"

If these dependencies are not showing up at Maven Central by the time you try, you can also use the snapshots from Nexus at version 0.2.3-SNAPSHOT for both drivers.

And it goes without saying that contributions, bug fixes, bug reports, benchmarks, feedback are very much welcome.

Turning a synchronous client into async

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 13 May 2013 00:00:00 UTC

DISCLAIMER: this is a rant about MySQL’s network protocol only. I am very grateful for all the work everyone involved in MySQL did and it still is my database of choice if not on Heroku so, don’t take this as another ‘mysql sucks, bla bla bla’ post, I just think it’s important for people to understand how you can complicate someone else’s life with a complicated network protocol.

A week ago I released the first version of the postgresql-async project and got some feedback from people that were looking for something like this but for MySQL. Since I had already planned to do it sometime in the future, why not just start it now? It wouldn’t be much harder than PostgreSQL, would it?

Oh boy, I was never THIS wrong in my life.

As you might have seen from the project description, I’m using Netty as the IO framework for both clients. Netty follows a messaging approach to IO, when something happens, an event is created and eventually your code will receive this event and respond to it in some way. Since all that comes and goes from sockets are bytes, you need to make sense of them in some way and, when using Netty, you can build upon a simple abstraction to do this. Let’s see some code from the PostgreSQLConnectionHandler class:

1
2
3
4
5
6
7
8
9
10
this.bootstrap.setPipelineFactory(new ChannelPipelineFactory() {

  override def getPipeline(): ChannelPipeline = {
    Channels.pipeline(
      new MessageDecoder(configuration.charset, configuration.maximumMessageSize),
      new MessageEncoder(configuration.charset, encoderRegistry),
      PostgreSQLConnectionHandler.this)
  }

})

A channel pipeline, as the name implies, is a sequence of handlers that will process the IO events. If you have done servlets, it’s akin to a filter chain (or a middleware stack on Rack, for instance), each object in the pipeline is invoked with an event and can process, discard or send it forward for someone else to process. In this case, we have a very basic pipeline, a MessageEncoder, a MessageDecoder and the PostgreSQLConnectionHandler itself as the end of the line.

Decoding a message

These classes are exactly what they look like, the MessageDecoder turns a bunch of bytes sent by the server into a meaningful object. If your server emits a warning, it becomes a NoticeMessage, this way, all the code to handle turning collections of bytes into messages live at the MessageDecoder, the driver code itself only works with high level messages, leaving all the heavy lifting for the decoder to do.

PostgreSQL defines a nice format for it’s messages, each message is composed of:

message type = 1 byte
message size (including self) = 4 bytes
message body = (message size) - 4 bytes

In Netty lingo, we call this a frame, it’s a common pattern on network apps communication, you say “this message has N bytes, so wait until you have N bytes available and process it”, this simplifies buffering and processing because you don’t have to wait for a terminator and the sender doesn’t have to escape the data in any way, you just read until N bytes and you’re done.

As this is commonplace in network apps, Netty offers a FrameDecoder class that you can subclass and build your own, here’s how it looks like in the PostgreSQL driver:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class MessageDecoder(charset: Charset, maximumMessageSize : Int = MessageDecoder.DefaultMaximumSize) extends FrameDecoder {

  private val parser = new MessageParsersRegistry(charset)

  override def decode(ctx: ChannelHandlerContext, c: Channel, b: ChannelBuffer): Object = {

    if (b.readableBytes() >= 5) {

      b.markReaderIndex()

      val code = b.readByte()
      val lengthWithSelf = b.readInt()
      val length = lengthWithSelf - 4

      if (b.readableBytes() >= length) {
        code match {
          case ServerMessage.Authentication => {
            AuthenticationStartupParser.parseMessage(b)
          }
          case _ => {
            parser.parse(code, b.readSlice(length))
          }
        }

      } else {
        b.resetReaderIndex()
        return null
      }

    } else {
      return null
    }

  }

}

It’s really simple, wait until we have at least 5 bytes (message type and length), once we have at least 5, read the type, then read size. With size in hand, check if we have at least (size - 4) bytes to read, if we do, read the slice needed and feed it to the message specific decoder so it can send the message forward. If we don’t have enough bytes, just “unread” the buffer and give it back.

These steps are repeated until you disconnect the client. The class is both simple and, most importantly, stateless. This message decoder does not have any mutable state, the state that the connection needs to maintain is kept farther down the pipeline to where it really makes sense to have state.

This makes it really easy to pinpoint bugs and errors, since there is only one possible state for the decoder, if it breaks, it will break on all cases and not in just one very specific and state dependent one. Better yet, since there is no mutable state, you can’t have concurrency issues! This class is thread safe by default and could be shared between different concurrent threads at any moment.

Why am I saying this? Because now we get to see how the MySQL network protocol works.

Decoding MySQL messages

For the MySQL client I built the MySQLFrameDecoder class. The message protocol is a bit different from the PostgreSQL protocol as we work with “packets” and not messages per se (and the protocol uses the little endian byte order). Packets are made of:

packet length - 3 bytes
sequence number - 1 byte
payload - packet length bytes

You might be thinking, “where is the packet type field?”, well, there isn’t one. The closest thing you would have to this are the “generic response packets”:

But this is a very loose assumption since the documentation itself says:

the EOF packet may appear in places where a Protocol::LengthEncodedInteger may appear. You have to check the packet length is less then 9 to make sure it is a EOF packet.

And this is also the case for the OK_Packet since you can get something that looks like an OK while doing prepared statements but it won’t be one, it will be a result set row.

What does this all mean? Different than the PostgreSQL protocol, where the parser knew right away what kind of message it was handling, on MySQL we don’t have this kind of information, the parser needs context, it needs to know “what am I doing now?” to be able to process the messages and this is bad.

First, now we have state in two different places, the message decoder and the MySQLConnectionHandler. Then a query message is received, the connection handler has to ping the decoder and say “hey, we’re in query phase, switch the way you handle messages” and this same decoder also needs to know when this phase ends, clear it’s internal state and then move on to the next sequence of commands. As you might imagine, this is a concurrency nightmare, you have a lot of mutable state and there is no way to hide from it, that’s how the protocol was built and it’s unlikely it will change in the long run.

But it’s still not enough. When you send a statement, MySQL will respond with a text based response, which means all rows will be turned into string values and the driver has to decode them back to their original values. If you use a prepared statement, MySQL will answer with a binary formatted result set. All fields will be encoded as their byte representations and you need to switch to this “decoding” mode by yourself at your parser.

Not only we have a protocol made for sequential programming, the protocol lacks consistency between it’s communication mechanisms. Instead of having only one way to represent a result set during a connection (like PostgreSQL does, you can set if you want all results to be text or binary based) MySQL again puts state into the protocol and forces the driver to understand how to parse data in two different ways. Not funny if you are the guy building the driver.

Why is MySQL like that?

When you switch from a synchronous model to an asynchronous one, the way you think about communication changes. The MySQL network protocol is a classic example of imperative and sequential programming. Here’s how doing a prepared statement in MySQL would look in pseudo code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
 socket.COM_STMT_PREPARE("select 10")
 val prepareResponsePacket = socket.readPacket()

if ( prepareResponsePacket == ERR_Packet ) {
 fail(prepareResponsePacket)
}

val paramsCount = socket.readPacket().numParams
val params = []
val columnsCount = socket.readPacket().numColumns
val columns = []

for ( x = 0; x < paramsCount; x++ ) {
    params.push(socket.readPacket())
}

socket.readPacket() // EOF to parameters

for ( x = 0; x < columnsCount; x++ ) {
    columns.push(socket.readPacket())
}

socket.readPacket() // EOF to columns

socket.COM_STMT_EXECUTE(prepareResponsePacket.statementId)

val executeResponse = socket.readPacket()

if executeResponse == OK_Packet
    emptyResponse()
elsif executeResponse == ERR_Packet
    fail(executeResponse)
else
    processResultSetPackets(socket)
end

When you’re building synchronous code that follows a sequence, it’s rather easy to have it built like that. You are already at the “state” necessary to perform the other actions so it becomes natural for you to keep this state and just go on like that. While in PostgreSQL you need to peek at the message to know what it is, in MySQL you need to understand “where” you are and in an async model this becomes blurry, because you just can’t chain methods one after the other as with the synchronous solution just above.

Also, this kind of solution not only makes it harder for someone to go async, but it also makes it harder to separate the “workflow” of the app from the resources it uses. Since you need to keep a handle to the socket connection available inside the code to keep on reading, it makes it harder to test this stuff in isolation.

A synchronous mind in an asynchronous world

Building an asynchronous solution isn’t exactly something easy from the get go, but a biased protocol like MySQL’s makes it much harder. It’s ambiguous, since messages can not be easily identified unless you know beforehand what you could get, it’s inconsistent, since the same kind of answer (the result set) can come in different formats, again, depending on what kind of state you are and it’s unnecessarily complicated. The prepared statement response could contain a single packet with parameters and column data, but it gets separated into at least 3 packets for no reason other than the possibility of having much more data than the 3 bytes integer used for size (16mb) can handle. If the column data (not the data itself) for the query you are making is more than 16mb, I don’t even want to know what kind of query you’re sending!

All of this made the job of building an async driver for MySQL be much longer than I had planned or hoped for (as much as programmers are bad with estimates). Building a nice network protocol isn’t simple, but making it unnecessarily complex is not the most awesome thing you can do, think about the lives of all the other programmers out there that will try to talk to your system :)

Still, after all that, the driver had it’s first release, so go check it out!

PS: If you use someone else’s MySQL driver and you meet this person, buy him a burrito, he/she definitely deserves it!

Async database access with PostgreSQL, Play, Scala and Heroku

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 29 Apr 2013 00:00:00 UTC

Last year I felt I wanted to contribute in some way to the Scala community. Since I didn’t think I was smart enought to work on the main libraries or compiler, I started to look for something that I could build and would benefit me and the community in general. I started looking around and noticed that there weren’t async drivers for the good old relational databases. If you’re fancy and you’re using one of the new NoSQL ones, you are most likely covered, but if you still rely on databases like PostgreSQL (like I do at my daily job), you have to rely on the official (and blocking) PostgreSQL JDBC driver and so the postgresql-async project was born.

I started looking at what people did in the NodeJS community like node-postgres and I noticed you don’t really need the full JDBC implementation for a usable database driver, as long as you can execute statements and get something back, you probably have all you need so this was the goal. Build something that would allow you to execute queries and get results back.

Connecting to the database

Let’s see some sample usage:

import com.github.mauricio.async.db.postgresql.PostgreSQLConnection
import com.github.mauricio.async.db.util.ExecutorServiceUtils.CachedExecutionContext
import com.github.mauricio.async.db.util.URLParser
import com.github.mauricio.async.db.{RowData, QueryResult, Connection}
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}

object BasicExample {

  def main(args: Array[String]) {

    val configuration = URLParser.parse("jdbc:postgresql://localhost:5233/my_database?username=postgres&password=somepassword")
    val connection: Connection = new PostgreSQLConnection(configuration)

    Await.result(connection.connect, 5 seconds)

    val future: Future[QueryResult] = connection.sendQuery("SELECT 0")

    val mapResult: Future[Any] = future.map(queryResult => queryResult.rows match {
      case Some(resultSet) => {
        val row : RowData = resultSet.head
        row(0)
      }
      case None => -1
    }
    )

    val result = Await.result( mapResult, 5 seconds )

    println(result)

    connection.disconnect

  }


}

The basic usage pattern is quite simple, you ask for something, you get a Future[_] back. In this case, I’m simplifying the code by blocking to get the results, but if you’re using an async framework (like Akka or Play) you can just compose on these futures to do your work.

The PostgreSQLConnection is a real connection to the database, it implements the Connection trait and you should try to use the trait as much as possible. When you create a connection handler, it’s not connected to the database yet, you have to connect it yourself calling connect and waiting for the future to return or composing on the future to do something else.

When you execute a statement, you get back a QueryResult object that might or might not contain a ResultSet. That depends on the kind of statement you executed, if the statement you asked for returns rows, there will be a ResultSet available for you, if it does not then you will have a None in there.

Integrating to a Play 2 application and running it on Heroku

Now if you’re building a webapp and using Play you can avoid all that messy Await.result() stuff and just return the futures themselves (you can see the full source code for this app here).

Let’s start creating a model:

package models

import org.joda.time.LocalDate

case class Message ( id : Option[Long], content : String, moment : LocalDate = LocalDate.now() )

Now let’s wrap the database access in a class to simplify the controllers:

package models

import scala.concurrent.Future
import org.joda.time.LocalDate
import com.github.mauricio.async.db.{RowData, Connection}
import com.github.mauricio.async.db.util.ExecutorServiceUtils.CachedExecutionContext

object MessageRepository {
  val Insert = "INSERT INTO messages (content,moment) VALUES (?,?) RETURNING id"
  val Update = "UPDATE messages SET content = ?, moment = ? WHERE id = ?"
  val Select = "SELECT id, content, moment FROM messages ORDER BY id asc"
  val SelectOne = "SELECT id, content, moment FROM messages WHERE id = ?"
}

class MessageRepository(pool: Connection) {

  import MessageRepository._

  def save(m: Message): Future[Message] = {

    m.id match {
      case Some(id) => pool.sendPreparedStatement(Update, Array(m.content, m.moment, id)).map {
        queryResult => m
      }
      case None => pool.sendPreparedStatement(Insert, Array(m.content, m.moment)).map {
        queryResult => new Message(Some(queryResult.rows.get(0)("id").asInstanceOf[Long]), m.content, m.moment)
      }
    }

  }

  def list: Future[IndexedSeq[Message]] = {

    pool.sendQuery(Select).map {
      queryResult => queryResult.rows.get.map {
        item => rowToMessage(item)
      }
    }

  }

  def find(id: Long): Future[Option[Message]] = {

    pool.sendPreparedStatement(SelectOne, Array[Any](id)).map {
      queryResult =>
        queryResult.rows match {
          case Some(rows) => {
            Some(rowToMessage(rows.apply(0)))
          }
          case None => None
        }
    }

  }

  private def rowToMessage(row: RowData): Message = {
    new Message(
      id = Some(row("id").asInstanceOf[Long]),
      content = row("content").asInstanceOf[String],
      moment = row("moment").asInstanceOf[LocalDate]
    )
  }

}

This database calls wrapper is quite simple but shows the pattern of working with async libraries, we are always composing on futures and returning futures as results, so you should select a framework that fits this kind of usage or at least simplifies it. Another thing to keep in mind is that these map calls require an execution context to be implicitly available somewhere, in this case I’m just using the execution context that comes by default in the driver itself but you might want to provide one from your own app instead.

One way to use this at your Play app is to configure the database at your Global object as in:

object Global extends GlobalSettings {

  private val databaseConfiguration = System.getenv("DATABASE_URL") match {
    case url : String => URLParser.parse(url)
    case _ => new Configuration(
      username = "postgres" ,
      database = Some("postgresql_async_app_development"),
      port = 5433
    )
  }
  private val factory = new PostgreSQLConnectionFactory( databaseConfiguration )
  private val pool = new ConnectionPool(factory, PoolConfiguration.Default)
  val messagesRepository = new MessageRepository( pool )

  override def onStop(app: Application) {
    pool.close
  }

}

Now, instead of using the DatabaseConnectionHandler object directly we use the ConnectionPool object that manages a pool of connections to the database. This connection pool object implements the Connection trait so you can (mostly) assume it works just like a connection, the only difference is that you should not start transactions directly on it. If you need transactions, take a connection from the pool yourself and later remember to give it back to the pool when you’re done. As you can see you don’t need to call connect on the pool but you need to remember to stop it when your application is turning off. Read the class docs for more info on it’s usage.

If you’re finding it weird that we’re parsing the DATABASE_URL environment variable, this is for Heroku support, if you drop this app on Heroku it will just work since the URLParser can also parse Heroku based database URLs (yes, they’re different from JDBC URLs).

Creating the controller

And now to wrap it up, we have the controller that uses our MessagesRepository:

package controllers

import play.api.mvc.{AsyncResult, Action, Controller}
import play.api.data._
import play.api.data.Forms._
import helpers.Global.messagesRepository
import com.github.mauricio.async.db.util.ExecutorServiceUtils.CachedExecutionContext
import models.Message

object Messages extends Controller {

  val messageForm = Form(
    mapping(
      "id" -> optional(longNumber),
      "content" -> text,
      "moment" -> jodaLocalDate
    )(Message.apply)(Message.unapply)
  )

  def index = Action {
    AsyncResult( messagesRepository.list.map {
      messages =>
        Ok(views.html.messages.index(messages))
    } )
  }

  def form = Action {
    Ok(views.html.messages.form(messageForm))
  }

  def edit( id : Long ) = Action {
    AsyncResult {
      messagesRepository.find(id).map {
        messageOption =>
          messageOption match {
            case Some(message) => {
              Ok( views.html.messages.form( messageForm.fill(message) ) )
            }
            case None => Ok( views.html.messages.form( messageForm ) )
          }
      }
    }
  }

  def update = Action { implicit request =>
    messageForm.bindFromRequest().fold(
      form => {
        BadRequest( views.html.messages.form(form) )
      },
      message => {
        AsyncResult {
          messagesRepository.save(message).map {
            message =>
              Redirect(routes.Messages.index())
          }
        }
      } )
  }

}

And now we see another pattern, as you can see from this code snippet:

  def index = Action {
    AsyncResult( messagesRepository.list.map {
      messages =>
        Ok(views.html.messages.index(messages))
    } )
  }

Whenever we use the repository and get a future back, we can just tell play that this is an AsyncResult and it will automatically handle processing our future and generating the response. We just compose again on the future returned and Play will handle the rest of the work for us. And you can just push this app the way it is to Heroku and you will have your fist async PostgreSQL backed Play app.

Pushing to Heroku

Just setup this app on heroku with:

heroku apps:create

And then:

git push heroku master

Once the slug is ready, login to your psql console:

heroku pg:psql

And create the messages table:

CREATE TABLE messages
(
  id bigserial NOT NULL,
  content character varying(255) NOT NULL,
  moment date NOT NULL,
  CONSTRAINT bigserial_column_pkey PRIMARY KEY (id )
);

You should now be able to run the app and enter data correcly.

You can see my app running here.

If you have questions, bug reports or want to help improve the library, hit me on Github or Twitter.

Contributing a new binding for MonoMac and making your MonoMac application updateable using Sparkle

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 23 Sep 2012 00:00:00 UTC

When building MonoMac applications you will eventually have to get your hands dirty interoperating with Cocoa/Objetive-C frameworks, one of the most common examples is updating applications. In the Mac apps world, if you’re not on the app store, you probably want to allow your users to update their applications so they all can start using all the new and shiny features and the most common solution for updating Mac applications is the Sparkle by Andy Matuschak.

It’s such a hassle-free solution for udpates that it almost doesn’t allow for a full blog post about it, but since we’re talking about MonoMac let’s also understand how we can create bindings so that we can call the Sparkle classes from our C# code.

You will need the latest Mono MDK (not the runtime alone) and a sane build environment (Xcode command line tools) to go through all these steps

First, clone MonoMac and MacCore from Github to a common directory as in:

mkdir monomac-build
cd monomac-build
git clone git://github.com/mono/maccore.git
git clone git://github.com/mono/monomac.git

Now that you have both projects, let’s build MonoMac and it’s accessory executables:

cd monomac/src
make

This will build the MonoMac.dll, parse.exe and a bunch of other stuff we won’t talk about here. The build process expects to have the maccore folder at the same level as the monomac folder so don’t rename them to something else as it will probably break the build.

Once you’ve done this, download the Sparkle framework files and get it’s .h files, they will be at the Sparkle.framework/Headers directory (they are Sparkle.h, SUAppcast.h, SUAppcastItem.h, SUUpdater.h, SUVersionComparisonProtocol.h). Copy them somewhere you can easily reference from the command line, I’d recommend placing them at the monomac-build directory.

Now, at the “monomac/src” directory, call parse.exe to generate the basic bindings from the Objective-C .h files:

mono parse.exe ../../SUAppcast.h ../../SUAppcastItem.h ../../SUUpdater.h ../../SUVersionComparisonProtocol.h

This is the output you will receive:

	[BaseType (typeof (NSObject))]
	interface SUAppcast {
		[Export ("fetchAppcastFromURL:")]
		void FetchAppcastFromURL (NSUrl url);

		[Export ("setDelegate:")]
		void SetDelegate ();

		[Export ("setUserAgentString:")]
		void SetUserAgentString (string userAgentString);

		[Export ("items")]
		NSArray Items ();

	}
	interface NSObject {
	}

	[BaseType (typeof (NSObject))]
	interface SUAppcastItem {
		[Export ("dict")]
		NSDictionary Dict ();

		[Export ("title")]
		string Title ();

		[Export ("versionString")]
		string VersionString ();

		[Export ("displayVersionString")]
		string DisplayVersionString ();

		[Export ("date")]
		NSDate Date ();

		[Export ("itemDescription")]
		string ItemDescription ();

		[Export ("releaseNotesURL")]
		NSUrl ReleaseNotesURL ();

		[Export ("fileURL")]
		NSUrl FileURL ();

		[Export ("DSASignature")]
		string DSASignature ();

		[Export ("minimumSystemVersion")]
		string MinimumSystemVersion ();

		[Export ("propertiesDictionary")]
		NSDictionary PropertiesDictionary ();

	}

	[BaseType (typeof (NSObject))]
	interface SUUpdater {
		[Static]
		[Export ("sharedUpdater")]
		SUUpdater SharedUpdater ();

		[Static]
		[Export ("updaterForBundle:")]
		SUUpdater UpdaterForBundle (NSBundle bundle);

		[Export ("hostBundle")]
		NSBundle HostBundle ();

		[Export ("setDelegate:")]
		void SetDelegate (NSObject delegate);

		[Export ("checkForUpdates:")]
		IBAction CheckForUpdates ();

		[Export ("checkForUpdatesInBackground")]
		void CheckForUpdatesInBackground ();

		[Export ("lastUpdateCheckDate")]
		NSDate LastUpdateCheckDate ();

		[Export ("checkForUpdateInformation")]
		void CheckForUpdateInformation ();

		[Export ("resetUpdateCycle")]
		void ResetUpdateCycle ();

		[Export ("updateInProgress")]
		bool UpdateInProgress ();

		//Detected properties
		[Export ("automaticallyChecksForUpdates")]
		bool AutomaticallyChecksForUpdates { get; set; }

		[Export ("updateCheckInterval")]
		double UpdateCheckInterval { get; set; }

		[Export ("feedURL")]
		NSUrl FeedURL { get; set; }

		[Export ("sendsSystemProfile")]
		bool SendsSystemProfile { get; set; }

		[Export ("automaticallyDownloadsUpdates")]
		bool AutomaticallyDownloadsUpdates { get; set; }

	}
	interface NSObject {
		[Export ("updaterShouldPromptForPermissionToCheckForUpdates:")]
		bool UpdaterShouldPromptForPermissionToCheckForUpdates (SUUpdater bundle);

		[Export ("updater:didFinishLoadingAppcast:")]
		void UpdaterdidFinishLoadingAppcast (SUUpdater updater, SUAppcast appcast);

		[Export ("bestValidUpdateInAppcast:forUpdater:")]
		SUAppcastItem BestValidUpdateInAppcastforUpdater (SUAppcast appcast, SUUpdater bundle);

		[Export ("updater:didFindValidUpdate:")]
		void UpdaterdidFindValidUpdate (SUUpdater updater, SUAppcastItem update);

		[Export ("updaterDidNotFindUpdate:")]
		void UpdaterDidNotFindUpdate (SUUpdater update);

		[Export ("updater:willInstallUpdate:")]
		void UpdaterwillInstallUpdate (SUUpdater updater, SUAppcastItem update);

		[Export ("updater:shouldPostponeRelaunchForUpdate:untilInvoking:")]
		bool UpdatershouldPostponeRelaunchForUpdateuntilInvoking (SUUpdater updater, SUAppcastItem update, NSInvocation invocation);

		[Export ("updaterWillRelaunchApplication:")]
		void UpdaterWillRelaunchApplication (SUUpdater updater);

		[Export ("versionComparatorForUpdater:")]
		id <SUVersionComparison> VersionComparatorForUpdater (SUUpdater updater);

		[Export ("pathToRelaunchForUpdater:")]
		string PathToRelaunchForUpdater (SUUpdater updater);

	}

	[BaseType (typeof (NSObject))]
	[Model]
	interface SUVersionComparison {
		[Abstract]
		[Export ("compareVersion:toVersion:")]
		NSComparisonResult CompareVersiontoVersion (string versionA, string versionB);

	}

The parse.exe executable parses the Objective-C header files and generates a C# representation for it’s classes. It does a great job at that but it still requires some manual work to make the code more C#-like and also to remove possible misinterpretations that we’ll see in a minute.

Now, at monomac/src create a file called sparkle.cs and declare a namespace on it, I’m going to use MonoMac.Sparkle:

//
// sparkle.cs: Definitions for the Sparkle Framework
//
using System;
using MonoMac.Foundation;
using MonoMac.ObjCRuntime;
using MonoMac.AppKit;

namespace MonoMac.Sparkle {
	// contents generated from parse.exe
}

We just declared a bunch of namespaces we will need here and copied all contents from the parse.exe output to it. Once all the content is copied, let’s start cleaning it up.

First, remove the empty NSObject reference and comment the interface NSObject and the end of the file. Since Sparkle uses an informal protocol (if you’re not that used to Objective-C, it’s like an anonymous interface, people are not required to fully implement it). Leaving it here serves mostly to help someone trying to implement the delegate.

Now, let’s get to the first class, SUAppcast:

[BaseType (typeof (NSObject))]
interface SUAppcast {
	[Export ("fetchAppcastFromURL:")]
	void FetchAppcastFromURL (NSUrl url);

	[Export ("setDelegate:")]
	void SetDelegate ();

	[Export ("setUserAgentString:")]
	void SetUserAgentString (string userAgentString);

	[Export ("items")]
	NSArray Items ();

}

Here we have two issues, first, the SetDelegate method did not add it’s parameter (most likely because it was called delegate and it’s a keyword in C#) and the Items property was translated as an Items method that returns an NSArray instead of a strongly typed SUAppcastItem array, let’s fix it:

[BaseType (typeof (NSObject))]
interface SUAppcast {
	[Export ("fetchAppcastFromURL:")]
	void FetchAppcastFromURL (NSUrl url);

	[Export ("setDelegate:")]
	void SetDelegate (NSObject delegateHandler);

	[Export ("setUserAgentString:")]
	void SetUserAgentString (string userAgentString);

	[Export ("items")]
	SUAppcastItem[] Items {[Bind("items")] get;}
}

Here we have added the delegateHandler parameter to the SetDelegate method (it has to be an NSObject) and we have also changed the Items method to be an Items property and return an array of SUAppcastItem. The Bind attribute is necessary because the property doesn’t follow the usual get${name}/set${name} naming pattern, so we have to explicitly say how it’s really called.

Now, off to the next one:

[BaseType (typeof (NSObject))]
interface SUAppcastItem {
	[Export ("dict")]
	NSDictionary Dict ();

	[Export ("title")]
	string Title ();

	[Export ("versionString")]
	string VersionString ();

	[Export ("displayVersionString")]
	string DisplayVersionString ();

	[Export ("date")]
	NSDate Date ();

	[Export ("itemDescription")]
	string ItemDescription ();

	[Export ("releaseNotesURL")]
	NSUrl ReleaseNotesURL ();

	[Export ("fileURL")]
	NSUrl FileURL ();

	[Export ("DSASignature")]
	string DSASignature ();

	[Export ("minimumSystemVersion")]
	string MinimumSystemVersion ();

	[Export ("propertiesDictionary")]
	NSDictionary PropertiesDictionary ();

}

This one is mostly making all of it’s properties, well, properties. Here’s how it will be like:

[BaseType (typeof (NSObject))]
interface SUAppcastItem {
	[Export ("dict")]
	NSDictionary Dict { [Bind("dict")] get; }

	[Export ("title")]
	string Title { [Bind("title")] get; }

	[Export ("versionString")]
	string VersionString { [Bind("versionString")] get; }

	[Export ("displayVersionString")]
	string DisplayVersionString { [Bind("displayVersionString")] get; }

	[Export ("date")]
	NSDate Date { [Bind("date")] get; }

	[Export ("itemDescription")]
	string ItemDescription { [Bind("itemDescription")] get; }

	[Export ("releaseNotesURL")]
	NSUrl ReleaseNotesURL { [Bind("releaseNotesURL")] get; }

	[Export ("fileURL")]
	NSUrl FileURL { [Bind("fileURL")] get; }

	[Export ("DSASignature")]
	string DSASignature { [Bind("DSASignature")] get; }

	[Export ("minimumSystemVersion")]
	string MinimumSystemVersion { [Bind("minimumVersion")] get; }

	[Export ("propertiesDictionary")]
	NSDictionary PropertiesDictionary { [Bind("propertiesDictionary")] get; }

}

And we’re done on this, this one was simple, it was just making sure it looked like a C# class.

And we’re off to the last one! It isn’t any different from all the others, it’s just making sure our final class looks like a C# class as much as possible. Here’s how it looks now:

[BaseType (typeof (NSObject))]
interface SUUpdater {
	[Static]
	[Export ("sharedUpdater")]
	SUUpdater SharedUpdater ();

	[Static]
	[Export ("updaterForBundle:")]
	SUUpdater UpdaterForBundle (NSBundle bundle);

	[Export ("hostBundle")]
	NSBundle HostBundle ();

	[Export ("setDelegate:")]
	void SetDelegate (NSObject delegate);

	[Export ("checkForUpdates:")]
	IBAction CheckForUpdates ();

	[Export ("checkForUpdatesInBackground")]
	void CheckForUpdatesInBackground ();

	[Export ("lastUpdateCheckDate")]
	NSDate LastUpdateCheckDate ();

	[Export ("checkForUpdateInformation")]
	void CheckForUpdateInformation ();

	[Export ("resetUpdateCycle")]
	void ResetUpdateCycle ();

	[Export ("updateInProgress")]
	bool UpdateInProgress ();

	//Detected properties
	[Export ("automaticallyChecksForUpdates")]
	bool AutomaticallyChecksForUpdates { get; set; }

	[Export ("updateCheckInterval")]
	double UpdateCheckInterval { get; set; }

	[Export ("feedURL")]
	NSUrl FeedURL { get; set; }

	[Export ("sendsSystemProfile")]
	bool SendsSystemProfile { get; set; }

	[Export ("automaticallyDownloadsUpdates")]
	bool AutomaticallyDownloadsUpdates { get; set; }
}

And it will become this:

[BaseType (typeof (NSObject))]
interface SUUpdater {
	[Static]
	[Export ("sharedUpdater")]
	SUUpdater SharedUpdater { [Bind("sharedUpdater")] get; }

	[Static]
	[Export ("updaterForBundle:")]
	SUUpdater UpdaterForBundle (NSBundle bundle);

	[Export ("hostBundle")]
	NSBundle HostBundle { [Bind("hostBundle")] get; }

	[Export ("setDelegate:")]
	void SetDelegate (NSObject delegateHandler);

	[Export ("checkForUpdates:")]
	void CheckForUpdates ();

	[Export ("checkForUpdatesInBackground")]
	void CheckForUpdatesInBackground ();

	[Export ("lastUpdateCheckDate")]
	NSDate LastUpdateCheckDate { [Bind("lastUpdateCheckDate")] get; }

	[Export ("checkForUpdateInformation")]
	void CheckForUpdateInformation ();

	[Export ("resetUpdateCycle")]
	void ResetUpdateCycle ();

	[Export ("updateInProgress")]
	bool UpdateInProgress {[Bind("updateInProgress")] get;}

	//Detected properties
	[Export ("automaticallyChecksForUpdates")]
	bool AutomaticallyChecksForUpdates { get; set; }

	[Export ("updateCheckInterval")]
	double UpdateCheckInterval { get; set; }

	[Export ("feedURL")]
	NSUrl FeedURL { get; set; }

	[Export ("sendsSystemProfile")]
	bool SendsSystemProfile { get; set; }

	[Export ("automaticallyDownloadsUpdates")]
	bool AutomaticallyDownloadsUpdates { get; set; }
}

First, we make the SharedUpdater method become a static property (that’s what it is, in fact). Then we do the same for the instance method HostBundle, making it an instance property, instead (anything that just gets you a value without side effects should probably become a property).

At the SetDelegate method, we just change the variable name to delegateHandler since delegate is a keyword in C#. Next there is the CheckForUpdates method that returns IBAction but IBAction means voidin Objective-C (it’s a special kind of void as InterfaceBuilder uses this to mark the method as an event handler ), so we change the return from IBAction to void.

And to wrap it all up we make LastUpdateCheckDate and UpdateInProgress properties.

Now, open the Makefile at monomac/src/Makefile and look for this piece:

1
2
3
4
5
6
7
8
9
10
11
12
APIS = \
	appkit.cs		\
	corewlan.cs		\
	foundation-desktop.cs 	\
	growl.cs		\
	imagekit.cs		\
	qtkit.cs		\
	pdfkit.cs		\
	webkit.cs		\
	composer.cs		\
	scriptingbridge.cs	\
	$(SHARED_APIS)

Just add the sparkle.cs file at the end:

1
2
3
4
5
6
7
8
9
10
11
12
13
APIS = \
	appkit.cs		\
	corewlan.cs		\
	foundation-desktop.cs 	\
	growl.cs		\
	imagekit.cs		\
	qtkit.cs		\
	pdfkit.cs		\
	webkit.cs		\
	composer.cs		\
	scriptingbridge.cs	\
    sparkle.cs      \
	$(SHARED_APIS)

Now run make again to rebuild the MonoMac.dll and now there should be a Sparkle directory and it should have four files:

SUAppcast.g.cs
SUAppcastItem.g.cs
SUUpdater.g.cs
SUVersionComparison.g.cs

And this new MonoMac.dll now contains the Sparkle framework classes!

We have finished our API beautifying, let’s start using it in a sample project!

Open MonoDevelop and create a new MonoMac project, I am going to name my project as UpdaterSample. Do not create a directoy for the solution, leave it a the project’s directory. Once you have the project ready, out of MonoDevelop, create a folder called Frameworks at the project’s root. Copy the Sparkle.framework folder to it.

Now, open the project’s options and add a custom command at the before build phase with the following command:

1
rm -rf ${TargetDir}/${ProjectName}.app/Contents/Frameworks; mkdir -p ${TargetDir}/${ProjectName}.app/Contents/Frameworks; cp -a ${SolutionDir}/Frameworks/. ${TargetDir}/${ProjectName}.app/Contents/Frameworks/

This action copies the contens of your Frameworks folder to the Contents/Frameworks folder of your app’s bundle, so you can easily drop new frameworks at this folder and they will all be copied to your bundle before building. Make sure you mark the Run on external console option or it isn’t going to work.

Now that you have the Sparkle framework in place and being copied to your app, let’s add the configuration keys necessary for it, first, set a build for your project (this is the information that will be used by Sparkle to check if the app version is current or not). You can do this by opening your Info.plist and setting it’s Build property as in this image.

Now copy the MonoMac.dll we built into your project folder, remove the reference to the MonoMac.dll that comes with the MonoMac (this one) add in in MonoDevelop and add a reference to your own MonoMac.dll.

Create an SUFeedURL key at your Info.plist file and make it link to your Sparkle file, in this project I’m using:

https://raw.github.com/mauricio/UpdaterSample/master/sparkle.xml

You can check this file and/or check Sparkle’s documentation to understand more about the file format and other options.

With this, lets do a bit of coding in our project to get Sparkle to always download and always perform an update check in background when the application starts:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
using MonoMac.Foundation;
using MonoMac.AppKit;
using MonoMac.Sparkle;

namespace UpdaterSample
{
	public partial class AppDelegate : NSApplicationDelegate
	{
		MainWindowController mainWindowController;
		
		public AppDelegate ()
		{
		}

		public override void FinishedLaunching (NSObject notification)
		{
			mainWindowController = new MainWindowController ();
			mainWindowController.Window.MakeKeyAndOrderFront (this);

			var updater = SUUpdater.SharedUpdater;
			updater.AutomaticallyDownloadsUpdates = true;
			updater.CheckForUpdatesInBackground();
		}
	}
}

I added the namespace to this file with the using MonoMac.Sparkle directive and then I just use the SharedUpdater available to run an update check (forcing a download before showing the screen). You can grab the zip for version 0.0.1 and 0.0.2 to see it in action.

At the end, we need to change our Main.cs file for it to load the Sparkle.framework native library, here’s how it’s done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
using MonoMac.AppKit;
using MonoMac.ObjCRuntime;
using System.IO;

namespace UpdaterSample
{
	class MainClass
	{
		static void Main (string[] args)
		{
			var baseAppPath = Directory.GetParent (Directory.GetParent (System.AppDomain.CurrentDomain.BaseDirectory).ToString ());
			var sparkleFrameworkPath = baseAppPath + "/Frameworks/Sparkle.framework/Sparkle";
			
			Dlfcn.dlopen (sparkleFrameworkPath, 0);

			NSApplication.Init ();
			NSApplication.Main (args);
		}
	}
}

With this the framework is loaded and all of our classes will execute just fine, you can do this with any framework you would like to have loaded by the app.

And with this the app should run and update fine. You can easily add new methods, set delegates and do whatever you would like to do just calling C# code and without having to worry about transforming objects from one to the other.

Now go get your hands dirty wrapping other cool Objective-C libraries. You can check the full source for this project here and you can check my pull request with these bindings to the MonoMac project here.

PS: The original idea of creating the “before build” command to copy frameworks was taken from this blog post by Kenneth Pouncey.

Summary of Working Effecivelly with Legacy Code - Part 2

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 21 Jul 2012 00:00:00 UTC

Chapter 2 - Working with feedback

Part 1 is available here

This chapter starts with something that’s well known to all developers, the two modes we use to make changes in a system, Edit and Pray and Cover and Modify. With such suggestive names, you already know what they mean for real.

While unfortunate, the most common behavior in the industry is Edit and Pray. Whether it happens because people don’t know better, were taught like this or are just being lazy at their jobs, it doesn’t really matter. The most common answer you’re going to get from people that work like this is that they’re being careful while making their changes, as if being careful was enough to do it right. Thing is, even if you are careful, the odds are usually against the developer. He’s could be in a busy environment, he could have external problems that are affecting his performance, it could be a friday afternoon right before a long weekend and all he thinks of is about this small vacation. We’re all human and this is going to happen a lot, that’s why this style of programming is very hard to keep for a long time.

When you’re doing Cover and Modify the order of the words is already a hint to what you’re doing. First, you cover and then you go make your changes. Think about doing a new painting on your car, the very first thing you will do is cover widows, tires, holes and anything that should not be painted. You don’t trust yourself to do a perfect job, instead, you hope for the worst, cover all pieces and then you start doing it. Is it a sign that you have no idea how to paint a car? Probably not, it’s a sign that you understand the risks and instead of tossing a coin you work on mitigating them, making sure you have less stuff to worry about while doing the job.

With software, it’s all the same, when you do Cover and Modify you’re first covering the code with tests, making sure the behavior stays the same as we keep on adding changes. We usually see tests as a way to prove that the program is correct, which is fine and all, but the idea behind this is that we’re adding tests to detect change, whether it’s good or bad.

I had an interesting issue related to this a couple of days ago. I was building a version of a software we already have today running on Windows to run on Mac OS, while doing it, we noticed some of the libraries we were using on Windows were not available or did not work as expected on the Mac and this prompted us to look for Mac specific solutions in this case. One very important piece was a file system watcher, the Windows implementation wasn’t capable of detecting files that were moved from one directory to another but the Mac solution was fully capable of handling this behavior. Thing is, the current application was already built with this in mind, so, what did we do? We wrote tests to make the behavior clear (not generating a rename when between folders) and then got the Mac implementation to do the same.

In the book, Michael defines these tests as a software vise:

Software Vise vise (n.) a clamping device, usually consisting of two jaws closed or opened by a screw or lever, used in carpentry or metalworking to hold a piece in position.

When we have tests that detect changes, it is like having a vise around our code. The behavior of the code is fixed in place. When we make changes, we can know that we are changing only one piece of behavior at a time. In short, we’re in control of our work.

But to be able to use these tests as a vise they have to be as efficient as one, they have to give you feedback right away when you’re building your stuff instead of running in a very long build or QA process. That’s when unit tests come in. Instead of having that large, often manual, test suite to run, you have a bunch of specific tests that exercise the piece of the system you’re working on that run fast and can give you quick feedback if you’re breaking something or not.

Unit tests

The idea behind unit tests is that you are going to test your system’s components in isolation to make sure they do what they’re supposed to do by themselves. They don’t guarantee that your system is going to behave correctly when all different objects are talking between themselves, but that’s not their goal. Their main goal is to make sure the behavior in that most atomic unit in your system is well defined.

Unit tests have three qualities that will make them different from larger integration or acceptance tests, which are:

Error Localization - The further away your test is from the source of the bug, the harder it is to figure out where it is. If you have an acceptance test that exhibits a bug you will have to check all related objects to figure out where the source is. With a unit test, that single unit is possibly already broken and it will be much simpler for you to detect it.
Execition time - Since unit tests are supposed to run really fast (Michael says that a unit test that runs in 1 tenth of a second is already too slow to be called a unit test), you can run them many times and have a very small feedback loop, making sure every small change you make is validated right away instead of waiting forever to find it.
Coverage - When doing large tests, it becomes much more complicated to follow all the paths your code has. To make matters worse, these large tests are also large in size so you would have to build another large test just because there is a small new path that isn’t exercised yet. When you’re working with unit tests it’s much simpler to test this new path atomically.

Moving on, Michael defines stuff that he wouldn’t call as unit tests:

Unit tests run fast. If they don’t run fast, they aren’t unit tests. Other kinds of tests often masquerade as unit tests. A test is not a unit test if:

It talks to a database.

It communicates across a network.

It touches the file system.

You have to do special things to your environment (such as editing configuration files) to run it. Tests that do these aren’t bad. Often they are worth the writing and you generally will write them in unit test harnesses. However, it is important to be able to separate them from true unit tests do that you can keep a set of tests that you can run fast whenever you make changes.

And in here we have a final definition of what a unit test is and isn’t. While I believe it is fitting, we now have a blurry space for what are tests that I write for a single class, in isolation, that talks to a database or makes network requests? I do have to write tests for this class and I want this class to also be tested in isolation (at least as much isolation as possible) to make sure I can also change it and make sure the original behavior is not affected.

I think that, maybe, this definition of unit tests is somewhat too restrictive and we could broaden it to cover these cases when all your class do is access the file system or make network requests. On a system I am currently working on we have a fake web server that is booted with the tests of a class that makes requests to an HTTP service. This class is tested against a fake web server but it makes real HTTP requests against it and it is important for us because it has already found bugs on this code. So, I think this could also be defined as a unit test as I am test the class alone doing it’s job (that is making HTTP requests).

The legacy code dilemma

Now we move on to the most interesting piece of this chapter, dealing with legacy code. With the definition of legacy code from the preface, comes the hardest thing about working on legacy code:

When we change code, we should have tests in place. To put tests in place, we often have to change code.

To be able to write tests for the legacy code we have in our hands currently, we need to start fixing it, but then we will have to make sure our fixes don’t break it even further so we need tests. Yeah, just like that GNU is not UNIX thing. One of the options you have is to use a refactoring tool, but they’re not 100% safe, so there isn’t a perfect solution for this.

Most of the problems you’re going to have when trying to test legacy code is dependencies. Since the code was never unit tested before, it probably exhibits a long list of direct and indirect dependencies, cycles in the object model, stageful singletons and classes that do too much breaking the single responsibility principle. First thing you have to do then is to start breaking these dependencies.

For a very simple example, imagine the following class:

class DocumentProcessor 

  def process(document)
	DocumentStorageService.instance.store( "#{document.id}.bin", document.file )
	if is_processable?( document )
	  DocumentProcessingService.instance.enqueue( document )
	end
  end
  
  def is_processable?( document )
    'application/pdf' == document.content_type
  end
  
end

This class has two direct dependencies, DocumentProcessingService and DocumentStorageService, it knows both classes directly, there isn’t a way to avoid this with the current code and unit testing this code is probably going to generate some problems, as we will have to talk to a storage service (which could be a file system, database or cloud storage for files) and to a queue, again, another external service.

How can we remove these direct dependencies?

class DocumentProcessor 

  def initialize( storage_service, processing_service )
    @storage_service = storage_service
    @processing_service = processing_service
  end

  def process(document)
	@storage_service.store( "#{document.id}.bin", document.file )
	if is_processable?( document )
	  @processing_service.enqueue( document )
	end
  end
  
  def is_processable?( document )
    'application/pdf' == document.content_type
  end
  
end

With the direct dependencies removed, we can now easily test our class as we have moved the dependency selection to someone else and not the executing class itself. While it’s probably not going to be this simple to change in your legacy code, the idea is that since you can’t write unit tests now, you should at least start moving the dependencies out so that you can start covering your classes as expected. Eventually, you will have to make more than simple changes like this and it is ok, once we meet the final goal that is to have unit tests around this code all the investment made in the first place is going to be paid off by a system that’s easier to test and change.

Michael defines the Legacy Code Change Algorithm as a five steps process:

Identify change points.
Find test points.
Break dependencies.
Write tests.
Make changes and refactor.

While it looks simple, once you face a large, unstructured and highly coupled codebase, you will understand that points 1 and 2 are very hard to come by. Figuring out where you need to change and where you need to test is going to be something that depends a lot on your codebase and how the application was built. Once you have this information, you go on removing dependencies to make the code testable, get your tests in place and in the end you will finally have trust that you’re not going to destroy everything once you run this.

Even better, with tests in place, you can move on much faster and make bigger surgeries in the source code to simplify changes in the future and make it much more testable. No one said it would be easy, but you have to make an action.

Summary

This chapter defines more stuff and starts pointing directly on how we are going to work with legacy code, showing us some hard truths and stuff that we should avoid when writing code and also the importance of having unit tests.

Summary of Working Effecivelly with Legacy Code - Part 1

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 21 Jul 2012 00:00:00 UTC

This week we started a book club at OfficeDrop and the first book selected was Working Effectively with Legacy Code by Michael Feathers. Having read it a couple of years ago while in college, going through the chapters again is much more interesting now as at that time I didn’t really understand all that he was talking about in the book. Today the lessons are definitely much more interesting as I see a lot of what he is talking about in code that I have worked with or am working on today.

Why make summaries of the book, you ask. Writing about it makes sure the concepts stick. Teaching is still the best way to learn.

Preface

He opens the book, at the preface, defining what legacy code really means:

Code without tests is bad code. It doesn’t matter how well written it is; it doesn’t matter how pretty or object-oriented or well-encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them, we really don’t know if our code is getting better or worse.

And as much as it did strike me as odd when I first read it, it makes a lot of sense today. Legacy code is not there just because it was written in an ancient language, without any documentation or by team members that are not in the company anymore. It becomes legacy code when no one wants to touch it and if people don’t want to touch it, the most common reason is that it doesn’t have any tests.

You can’t have any confidence on changes you make to the software if you don’t have tests as you can never be sure if your change is going to break something or not. You have no way to verity your changes other than manually testing them and the cycle of writing code, booting up the application and running manual tests is usually not very effective, you will probably waste a lot of time doing it and in the end you will just give up, leave the legacy code alone and build your functionality somewhere else. So, you ran away from the problem.

The code you are writing without tests today is legacy code already, there is no point in hiding it.

Chapter 1 - Changing software

This chapter is mostly about defining what kinds of changes we usually do in software. He then defines them in four kinds:

Adding a feature
Fixing a bug
Improving the design
Optimizing resource usage

Adding a feature and fixing a bug are possibly the most blurry ones. As he walks around defining them, you start to notice that there is a lot of overlap when you’re doing both. Sometimes, when you’re adding a feature, from the customer’s point of view, you’re fixing a bug in fact (the bug of not having this functionality in there already) and sometimes fixing a bug from the developer’s perspective is building a new feature, because the original definition is completely different than the one proposed by the fix.

What we’re doing in both cases could be much more well defined if we used behavior instead of adding features or fixing bugs. When we’re doing any of them we’re adding new behavior and/or changing the existing behavior of the system. In another quote from the book:

Behavior is the most important thing about software. It is what users depend on. Users like it when we add behavior (provided it is what they really wanted) but if we change or remove behavior they depend on (introduce bugs), they stop trusting us.

The other two kinds of change will not alter behavior, they will touch on different qualities of our software. Improving the design will usually be changing the code structure, how classes relate to each other, how they are coupled together to perform the behavior we expect. The most common case of design improvement is by applying the refactorings to your code, this way you do change the structure but maintaining the same behavior.

On optimizing resource usage, instead of changing the externally perceived behavior you change the internal behavior to use less of a resource to make the code go faster, use less memory or anything like that. So, while you’re possibly not changing the output you will most likely change the code internally in a way that will change how it does it’s job.

With this defined, we end up noticing that most of what we want to do when changing software is preserving behavior. Whenever you make a change to your code, you usually want to make sure the old code is not going to break and all the other funcionalities are not going to be affected by this specific change you’re making (whether it’s supposed to add or change any behavior or not).

And this is where the real challenge is, making sure the changes you’re making aren’t going to ripple through the rest of the application breaking or changing behavior that is unrelated to what you’re doing. There are many ways teams deal with this, from declaring “if it isn’t broken, don’t touch it” or just by being cautious. Well, if I pay a lot of attention to what I’m doing here, I’m not going to break anything, am I? And people are still trying really hard to run away and ignore the problem that they are facing instead of just going there and doing something about it.

Conclusion

Preface and first chapter have already started with some interesting ideas, laying the groundwork for what comes next. And now I feel I should probably go back and revisit more books from college since they will probably be much more useful today that they were at that time.

Stay tuned for the next ones :)

Part 2 is available

Using board games to teach object oriented analysis

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 18 Jul 2011 00:00:00 UTC

Just when I arrived from Cambridge back home the coordinator at the college where I usually teach after-grad courses mailed me asking if I’d be available to pick up an “Object Oriented Analysis and Design” class. I was still unpacking my stuff but I just said yes. Why not? It’s something I really like to teach and talk about, could be great to get me back to teaching after 5 months away from classes.

While preparing the material, I started thinking about how to add something unusual. It wasn’t long until I looked at my wardrobe and saw the pile of board game boxes in there. I looked at them and thought, I can get them to model the games themselves!

Meeting board games

[caption id=”attachment_395” align=”alignleft” width=”600” caption=”Ticket to Ride: Europe team learning how the game goes”][/caption]I started on European style board games on 2007, when doing a presentation at the ERECOMP-Alagoas with Phil Calçado. After the event we headed back to the hotel and instead of doing nothing we went out to drink something and talk and he said he had met his wife at a Carcassonne table, an European board game his friends discovered and everyone was having fun.

I can’t say I was board games fan in the past, mostly because all I knew about them was Risk and Monopoly and you just get bored after playing them for a while. But Phil said these European board games were different, so, why not try them? After the event I found a shop and got Settlers of Catan and Carcassonne. When they arrived, it was like an addiction, we were playing Settlers every weekend and from that time to today the board games collection has grown from those two to more than 10 different games games and their expansions (Dominion alone is now at six boxes).

How come you’re using board games on classes?

[caption id=”attachment_397” align=”alignleft” width=”600” caption=”Settlers of Catan team picking up resources”][/caption]Why use board games, you might ask. The course itself is an after-grad course (we call it specialization here in Brazil, it would be something like a graduate school in the US), classes happen every 15 days, on Friday nights from 6:30 to 10:30 and Saturdays from 8:00 to 12:00 and again from 13:00 to 17:00. So students see the same teacher and the same class for 12 hours straight over one weekend.

As you might imagine, 12 hours straight of the same teacher talking about the same subject is not the definition of fun, so you have to come up with unorthodox solutions in your class to keep everyone interested in what you’re talking. You need to engage the students on whatever you’re teaching so they’re not drifting away to sleep after lunch on a Saturday afternoon.

I had a problem, coming up with an interesting topic to use as an example. I couldn’t just bring a shallow model, as it would not fit the idea that was to exercise analysis and design but I couldn’t just bring up something that was too difficult or needed too much background to understand. Also, it had to be something the students could easily relate to and enjoy modeling.

Thing is, board games were the best choice ever.

First contact

[caption id=”attachment_398” align=”alignleft” width=”600” caption=”Modeling, Modeling, Modeling…”][/caption]As expected, the students had no previous experience with any of the two board games I got in, Settlers of Catan and Ticket to Ride: Europe (well, not all of them, one is a close friend of mine that has played most of them, but she has never played “Ticket to Ride”, as I just brought it from the US, so it was her first experience with it also).

The plan is that they have to build a model that’s going to be used to implement these games in the computer, so they have to understand the game rules, themes and development. The fact that they have no previous experience with these games helps simulate the issues we face daily while building software for real. We have to learn the domain as we go, things change and new rules show up at any time.

They had no access to the game rules, I was the only source of information and I intentionally did not give them all the information required to play the game. Information was given in pieces and while they started playing the games to get a feel of them I would come in and bring a new rule. When they got stuck or had no idea how to move forward, they came to me and asked how that would go, as they would ask a client how a feature is supposed to work. While asking and playing the games, they were taking notes and starting to think about how the game is supposed to be modelled. After playing the games for half an hour or a little bit more, they started the real fight, turning those notes into the model.

Modeling in action

[caption id=”attachment_399” align=”alignleft” width=”600” caption=”"This is how the train goes!"”][/caption]As simple as these games look like when you’re playing, bringing this simplicity into a computer is not really that, well, simple. Rules that just make sense when you’re playing the game become sources of complexity in a model, just like the Train Station rule on “Ticket to Ride”. In the game, you’re building rail roads connecting two cities with your own trains, but you can place a train station in a city and use someone else’s trains as if your own if they’re connected to your Train Station. But then these Train Station connections are not valid for the “European Express” objective and they also mean you gave up on points at the end of the game.

Rules that are just common sense for us, players, are not that easy to come by to computers. They have to be told what to do and how to do it and this is where modeling these games is a real challenge that goes really beyond the usual “mapping from the database” we see on nowadays “enterprise” applications. These games have rich models, their own internal language and behaviors, which makes them even more interesting to a design class, as you have many different issues to solve and you’re possibly having fun by doing it about a game.

Does it work?

So far, I would say it met my objectives. The students were highly engaged in trying to figure out how to make those rules become objects in a system (and they also loved to meet and play the games! ☺ ). Some specific cases led to interesting discussions, like how can we implement the “action” during a turn on “Ticket to Ride: Europe”, as each player can perform one of four really different actions (pick cards, pick destination tickets, claim routes or place a train station) and how to model the “Settlers of Catan” game board in a way it would be possible to place terrains and also make sure the settlements and roads would be correctly placed.

They are going to be working on refining the models and in 15 days we will meet again for the final presentation and some “show me the code” about how these games could be implemented. I hope that by the end of this class they will have an improved background on object oriented analysis and built on top of real world knowledge, not only reading books or listening to a teacher talking about objects, relationships and design patterns.

My mother, an elementary school teacher, always tried to do something different for her children, with new games, songs, drawings and activities that went beyond the usual “talk, read and write” and she can barely use a computer. Today we have so many different possibilities and materials but we keep ourselves stuck into the “Power point bullets”, confined in a sequential list of topics.

We can do better. We can improve. We just have to try.

And stay tuned to the results of this crazy experience, I will be sure to post my findings here as soon as the class ends.

Ruby Basics - Equality operators in Ruby

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 30 May 2011 00:00:00 UTC

After Greg Sterndale’s presentation on a boston-rb hackfest earlier this month I noticed that not everyone knew the operators available for equality and comparisons in Ruby. Why not take the dust away from the blog and write about it, then?

Ruby has many equality operators, some of them we use and see everywhere in our applications (like the usual double equal - “==”) and some are also used everywhere but we don’t really get to see them (like the triple equal or case equal operator – “===”). So let’s dig into how Ruby implements comparisons between our objects.

You can see the full source code for this tutorial on Github.

What does it mean to be equal?

Maybe this is something from my Java past, but I find it to be really useful to first define what being “equal” really means. Objects have identities for the Ruby interpreter, you can easy check this by calling the object_id method:

some_string = 'some string'
=> "some string"
another_string = 'another_string'
=> "another_string"
[ some_string.object_id, another_string.object_id ]
=> [2164900860, 2164888440]

As you keep on creating new objects, Ruby itself will give you an object_id for each of them and you could possibly use this to identify all your objects. But there’s a little gotcha:

matz = 'matz'
=> "matz"
matz_2 = 'matz'
=> "matz"
[ matz.object_id, matz_2.object_id ]
=> [2164840660, 2164818480]

The two objects represent exactly the same value, but each of them has it’s own object_id, as the Ruby interpreter has no idea these two objects are the same. So, while object_id could be a shortcut to define the identity of our objects, each object has it’s own way to define what being equal to someone else really means.

Two Strings are equal when they represent exactly the same sequence of characters, two people would be the same if they had the same social security number (or the same CPF if you were in Brazil). To be able to implement this kind of identity the language must offer you with hooks for this and in Ruby these hooks are the equality methods.

”==” - double equals

The double equals method should implement the general identity algorithm to an object, which usually means that you should compare the object attributes and not if they are the same object in memory. And given Ruby is dynamically typed language, you should try not to depend on types, but on methods, instead of checking if the object is from an specific class, check if it responds to an specific method:

class Meter

  def initialize( value )
    @value = value
  end

  def to_meters
    @value.to_f
  end

  def ==( other )
    if other.respond_to?( :to_meters )
      self.to_meters == other.to_meters
    end
  end

end

Instead of checking on the parameter type I check if the object has the method I expect to use, I it has then I do the comparison, if it doesn’t a “false” is sent up, which means that the objects are not equivalent.

The “hash” method

If you’re already implementing “==”, you should also implement “eql?” and “hash” methods, even if you have never seen these methods being called anywhere. The reason is these are the methods the Hash object is going to use to compare your object if you’re using in as a Hash key. The thing is, Hashes have to be fast to figure out if a key is already in there and to be able to do this they just avoid comparing every single object, they just go by “clustering” objects in groups by using the value returned by your object’s “hash” method and then, once in a cluster, they compare the objects themselves using “eql?”.

Then searching for a key in a Hash, they first call “hash” in the key to figure out in which group it would be, then they compare the key with all the other keys in the group using the “eql?” method. In a worst-case scenario, we would compare with 3 objects (and the hash has nine of them), which is quite nice. If you were using an array, your worst case would be 9 comparisons.

For a quick example of how this could affect your code, let’s check this class:

class StringWithoutHash

  def initialize(text)
    @text = text
  end

  def to_text
    @text
  end  
  
  def ==(other_value)
    if other_value.respond_to?(:to_text)
      self.to_text == other_value.to_text
    end
  end

end

The “==” method has been implemented correctly, but we haven’t implemented “hash” and “eql?”, so the two objects will end up in different clusters they we won’t be able to figure out they’re the same object inside the hash.

context 'without hash and eql? methods' do

  it 'should be equal' do
    @first  = StringWithoutHash.new('first')
    @second = StringWithoutHash.new('first')
    @first.should == @second
  end

  it 'should add as two different keys in the hash' do

    @texts = {}

    10.times do
      @texts[ StringWithoutHash.new('first') ] = 'one'
    end

    @texts.keys.size.should == 10

  end

end

Even with @first and @second representing exactly the same text (and being equal) they still generate two keys in the Hash instead of one because we did not implement the “hash” and “eql?” methods. The general rule is that if you override “==” you should also override “hash” and “eql?”. When implementing “hash”, a basic rule to follow is if two objects are “==” they must generate exactly the same hash value (so they can be found in a Hash object), but two objects can have the same hash value but still be different (they belong to the same group but are not the same object).

Here’s a subclass of our StringWithoutHash, StringWithHash, that correctly implements both methods (and the “eql?” method itself is just using the “==”, we don’t even have to bother writting code for it):

class StringWithHash < StringWithoutHash

  def eql?( other )
    self == other
  end  
  
  def hash
    @text.hash
  end

end

And here are some specs showing how it works correctly now:

context 'with hash and eql? methods' do

  it 'should be equal' do
    @first  = StringWithHash.new('first')
    @second = StringWithHash.new('first')
    @first.should == @second
  end

  it 'should add as a single key in the hash' do

    @texts = {}

    50.times do
      @texts[ StringWithHash.new('first') ] = 'one'
    end

    @texts.keys.size.should == 1

  end

end

So, now that we implemented “eql?” and “hash” correctly even trying to add the object 50 times we stil have a single one, because our code can now be sure that the object is there (or not) as the methods are available.

Unless you know exactly what you’re doing (and you know how Ruby implement it’s hashes) you should not implement your own hashing function, just use a hashing function from a basic object like numbers and strings and you’re done. Here’s the example:

class Meter

  # all the other methods


  def hash
    self.to_meters.hash
  end

  def eql?( other )
    self == other
  end 

end

Here, instead of adding my own “hash” function I just reuse the hash method on Float, which already does it the right way. And unless you have some specific needs you should be doing the same.

”===” - triple equals

Triple equals is an interesting operator, it’s everywhere in Ruby code but most people have never seen it in real code out there. But, how come it’s everywhere and no one has ever seen it? It’s hidden inside a common control structure, the “case/when”. Whenever you’re using a “case/when” you’re using, in fact, the “===” operator and this is what makes the case statement on Ruby much more powerful than it’s counterpart in languages like C or Java.

Let’s look at a statement:

age = 19

case age
  when 1..18
    puts 'just out of college'
  when 19..30
    puts 'wild years'
  when 31..40
    puts 'i better find a job in a big corp'
  else
    puts 'retirement plan'
end

This one will print ‘wild years’ and it’s using the ‘===’ operator. Can you see how it’s working? Here’s this same case/when done with if’s:

if 1..18 === age
  puts 'just out of college'
elsif 19..30 === age
  puts 'wild years'
elsif 31..40 === age
  puts 'i better find a job in a big corp'
else
  puts 'retirement plan'
end

In the end, the case/when statement is just a glorified if using the ‘===’ operator to simplify your job (and also make you type less). In the language itself, the triple equals is used mostly to as a “grouping” operator, by getting a value and figuring out at which group it belongs to.

In our example it’s a group of ages represented by Range objects, but you’ll see this being used to figure out if an object is from an specific class, if a string matches a regular expression and the like. And all of this is possible because the method (‘===’) is not called at the object in the “case” definition but on each “when”.

Imagine you have Rectangles and you want to figure out if a specific Point is inside the Rectangle, this is a perfect fit for the triple equals. Let’s look at a sample implementation:

class Point

  include Comparable

  attr_accessor :x, :y

  def initialize(x, y)
    self.x = x
    self.y = y
  end

  def hash
    "#{x}-#{y}".hash
  end

  def <=>(other)

    result = nil

    if other.respond_to?(:x) && other.respond_to?(:y)

      result = if self.x == other.x && self.y == other.y
                 0
               elsif self.x > other.x && self.y > other.y
                 1
               else
                 -1
               end

    end

    result

  end

end

And the Rectangle class:

class Rectangle

  attr_accessor :start, :end

  def initialize( x1, y1, x2, y2 )
    self.start = Point.new( x1, y1 )
    self.end = Point.new( x2, y2 )

    if self.start >= self.end
      raise "Start #{self.start.inspect} should be less than end #{self.end.inspect}"
    end
  end

  def === (other)
    if other.respond_to?(:x) && other.respond_to?( :y )
       other.between?( self.start, self.end )
    else
      self == other
    end
  end

  def == (other)
    if other.respond_to?( :start ) && other.respond_to?( :end )
      self.start == other.start && self.end == other.end
    end
  end

end

And now some usage:

point = Point.new( 2, 4 )

case point
  when Rectangle.new( 0, 3, 5, 8  )
    puts 'found it here'
  when Rectangle.new( 3, 3, 10, 15  )
    puts 'i will not match'
end

The ‘===’ method will be called on Rectangle giving it a Point object and it’s going to figure out if that point is inside the Rectangle or not.

Making your objects Comparable

If you looked closely at the Point object you will notice that it includes the Comparable module and implements a method defined as <=>, the flying saucer operator. The flying saucer operator is to be used as means to sort your objects in a collection, but the Comparable module brings some interesting functionality for classes that implement it.

The idea behind the <=> operator is that when you call it on a object, this object must define it’s position compared to the other object given as a parameter. If the receiver of the call is greater than the argument, it should return 1, if they re the same, it should return 0, if the receiver is less than the argument it should return -1 and if they’re not compatible it should return nil.

Once you implement the flying saucer, you can just include Comparable in your class and the following methods are now implemented for you:

>
>=
<
<=
==
between?

This is why we could use the between? method in Point when implementing the “===” operator on Rectangle. By implementing <=> and including Comparable we get a lot of functionality for our objects for free and we also have a great example of how you should plan to build your own modules on your projects.

Going back to our first example, the Meter class, we could add new classes for Inch and Foot and have them all share the same equality implementation. First, we define that all our classes will have a to_meters method that will return their value in meters. Then we create our module:

module MeterComparator

  include Comparable

  def <=> (other)
    result = nil

    if other.respond_to?(:to_meters)
      receiver_value = self.to_meters
      argument_value = other.to_meters

      result = if receiver_value == argument_value
                 0
               elsif receiver_value < argument_value
                 -1
               else
                 1
               end
    end

    result
  end

  def inspect
    "#<#{self.class}:#{self.hash} size_in_meters=#{self.to_meters}>"
  end

  def hash
    self.to_meters.hash
  end

end

We implemented the <=> operator and (also the “hash” method, don’t forget it!) for this module and included Comparable, which will make all classes including it to be comparable too. Let’s look at how our new Meter, Inch and Foot will look like now:

class Meter

  include MeterComparator

  def initialize( value )
    @value = value
  end

  def to_meters
    @value.to_f
  end

end

And Inch:

class Inch

  include MeterComparator

  def initialize( value )
    @value = value
  end

  def to_meters
    @value.to_f / 39.370
  end

end

And finally Foot:

class Foot

  include MeterComparator

  def initialize( value )
    @value = value
  end

  def to_meters
    @value.to_f / 3.2808
  end

end

All classes share the same comparison methods so we can use them all interchangeably in our code, we can even safely compare them with each other and they’ll yield the correct results:

context 'comparing meters with inches' do

  it 'should be true when they both represent the same distance' do
    @meter = Meter.new(4)
    @inch  = Inch.new(157.48)

    @meter.should == @inch
  end

end

context 'comparing meters to feet' do

  it 'should be true when they both represent the same distance' do
    @meter = Meter.new(8)
    @foot  = Foot.new(26.2464)

    @meter.should == @foot
  end

end

context 'comparing feet to inches' do

  it 'should be true when they both represent the same distance' do
    @foot = Foot.new(26.2464)
    @inch = Inch.new(314.96)

    @foot.should == @inch
  end

end

context 'when sorting objects' do

  before do
    @meter    = Meter.new(1.5)
    @inch     = Inch.new(157.48)
    @foot     = Foot.new(26.2464)
    @measures = [@inch, @meter, @foot].sort
  end

  it 'should order them by size correctly' do
    @measures.first == @meter
    @measures[1] == @inch
    @measures.last == @foot
  end

end

You can mix and mingle different measures and they’ll all play and compare nicely to each other, they just have to implement the “to_meters” method and include the MeterComparator class, simple and right to the point implementation.

Also, once you include the Comparable module your objects become “sortable” in an array, you can use the “sort” and “sort!” methods in Array. The order is ascending as defined by your “<=>” method implementation.

“eql?” and “equal?”

Technically, “eql?” is should behave just like “==” and it’s also the method selected by the Hash class to figure out of your object is already in a “hash cluster” (as we have discussed above). You compare two objects to see if they represent the same values. Usually you can just override “eql?” and delegate it’s call to “==”, as we did in our examples. This is not already done for you, the default “eql?” implementation at the Object class uses comparison between “object_id” values and this is usually NOT what you want, so, make sure that if you implement “==” you also override “eql?” and implement “hash”.

There’s one exception, though, Numeric objects will convert different types when compared using “==” but will not do this when using “eql?”, so:

5 == 5.0 # is true

But:

5.eql?( 5.0 ) # is false

And “equal?” is a little bit exoteric as it will compare if two objects are the same object in memory. You should never ever override this method. In fact, you’re better of ignoring the fact that “equal?” exist at all for your own safety. And don’t say you have not been warned.

Closing thoughts

While there’s a lot to be said about comparing objects in Ruby, the final implementation is quite simple and modules like Comparable make it even simpler as long as you know they exist. Now there’s no reason to correctly implement comparison for your Ruby objects and never forget the “hash” method again! ;)

Criando um programador

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 07 May 2011 00:00:00 UTC

De tempos em tempos surge nas listas de discussão de tecnologia a conversa sobre estágio, como aprender e como se “formar” um programador. Eu tive uma experiência interessante sobre isso e acho que finalmente é a hora de escrever sobre o assunto.

Há algum tempo atrás, quando estava saindo de uma empresa e indo pra outra, resolvi que poderia usar o “downtime” entre uma e outra pra escrever um produto que já tinha na cabeça a algum tempo. Sentei a bunda na cadeira e passei 3 semanas escrevendo muito código, quando finalmente comecei a trabalhar no novo emprego, meu tempo disponível pra ferramenta diminuiu consideravelmente e eu pensei que talvez fosse a hora de encontrar alguém pra me ajudar.

Considerando o meu estado de “empreendedor liso”, não poderia contratar um desenvolvedor mesmo, então fui pra contratar um estagiário. A luta contra os currículos foi dura (deveria ter escrito isso antes ) e no fim só um realmente me interessou, especialmente porque a pessoa dizia que sabia ler e escrever em inglês.

Primeira conversa

Nos encontramos pra uma entrevista formal, conversamos sobre a empresa, meus planos de dominação mundial e o que ele deveria fazer. Descobri que ele havia morado nos EUA durante um tempo, então realmente sabia ler e escrever inglês e pelo papo parecia ser uma pessoa interessada.

A experiência dele era pouca, não teve uma formação legal em orientação a objetos e, mesmo estando cursando a mesma graduação que eu cursei, o fato dele ter tido professores diferentes não deu a ele o conhecimento que eu esperaria que ele já tivesse com o tempo de curso que ele tinha. Além disso ele teria que aprender Rails, HAML, Solr e todas as tecnologias que já estavam sendo usadas na aplicação e isso não seria uma tarefa simples.

Shu-Ha-Ri

Na época, eu estava lendo o Agile Software Development: The Cooperative Game do Cockburn e estava fisgado na idéia do Shu-Ha-Ri, que são os 3 momentos do aprendizado, uma idéia que surgiu no teatro Nô japonês. Segundo o Shu-Ha-Ri, a técnica pode ser dividida em 3 estágios diferentes, Shu ou aprender, Ha ou separar e Ri ou transcender.

No estágio do Shu, o aprendiz somente segue os passos já definidos pelo mestre, sem pensar muito sobre o que está sendo feito. No Ha, o aprendiz já começa a pensar sobre o que ele está fazendo, porque ele deve fazer daquela forma. E no Ri o aprendiz define a sua própria forma, baseado nas suas capacidades, conhecimento e experiências de vida. Ele deixa de ser um aprendiz e agora é um mestre de si mesmo, não mais repetindo o caminho do mestre, mas definindo o seu próprio.

Começando o caminho

Pra começar com o Shu, era necessário aprender e repetir, então a primeira atividade dele no estágio não seria mexer na aplicação, que já estava relativamente pronta. O primeiro mês dele deveria ser criar um blog e escrever resumos com exemplos de código de cada um dos capítulos iniciais do Agile Web Development with Rails.

Na época ele deve ter pensado que eu era meio louco (hoje já tem certeza) e na verdade foi bem um tiro no escuro mesmo, eu nunca tinha feito isso antes com um estagiário, mas eu tinha que arriscar. Outro detalhe importante, ele não ia receber nada por esse primeiro mês, os R$ 500,00 reais que ele iria receber foi comprado em livros (lista no final do post), porque ele ainda tinha muito de teoria a melhorar.

Seguindo o caminho

Depois do primeiro mês só de blog e de ver como a aplicação funcionava, ele começou finalmente a escrever código. Nesse momento, ainda saindo de Shu para Ha, era necessário que eu constantemente explicasse os passos necessários pra que ele executasse as tarefas. Primeiro porque eu tinha um apego especial pelo código da aplicação, que era a minha primeira ferramenta própria, e também porque eu sabia das limitações dele na tecnologia nesse momento.

A maior parte dos trabalhos iniciais era estender funcionalidades já existentes e como a aplicação havia sido escrita a uma única mão, os padrões eram claros e fáceis de se entender. O estilo do código era único e nesse momento a maior parte do trabalho dele era continuar seguindo esse estilo e complementando as funcionalidades com código que fosse sempre parecido ou derivado do que já existia.

Com um pouco mais de tempo, ele já estava escrevendo funcionalidades únicas, criando código do zero e implementando os seus próprios modelos, ainda baseados no que já existia dentro da aplicação, mas já estava claramente em seu momento Ha. Não era mais necessário me perguntar como tudo funcionava, ele já havia criado o seu próprio entendimento sobre como e o que era a aplicação e começava a enxergar ela com os seus próprios olhos. Eu já descrevia as funcionalidades sem explicar detalhes específicos de implementação e ele era capaz de implementá-las sem que houvesse necessidade de ser levado pelo caminho.

Transcendendo o caminho

O estágio acabou, ele tomou o seu próprio caminho e tem agora o seu próprio “estilo”, chegou ao seu momento de Ri, não do conhecimento, mas dessa primeira fase de entendimento do trabalho de um programador. Hoje eu vejo que os primeiros passos, naqueles primeiros meses, foram essenciais pra formação e a prática aliada a teoria foram as principais responsáveis por fazer com que ele se tornasse capaz de escrever código útil, que continua na ferramenta até hoje, com praticamente meses de experiência em programação.

Acho que o grande problema que temos hoje na formação é o exagero do ensino no Shu e também o exagero dos aprendizes no Ri. Os professores se perdem eternamente na fase Shu mas os alunos gostariam de estar já em Ri, eles não querem mais passar pela fase de conhecer, querem passar diretamente pra fase do criar, pensando que a rápida passada por cima do conhecimento já vai lhes dar condições de criar algo por eles mesmos.

Programação se aprende com leitura e, principalmente, prática. A criação do blog não era somente pra que ele lesse o livro naquele momento, mas que ele lesse e repetisse o conhecimento. Nós fixamos o conhecimento por repetição, os músculos de um praticante de artes marciais tem uma memória dos movimentos que ele precisa fazer porque ele os repetiu a exaustão e assim também é programar, você repete os passos seguidamente marcar o seu cérebro com aquele caminho, de forma que você já saiba exatamente pra onde vai assim que começa a escrever.

Implementando

Se você tem aprendizes sob a sua coordenação, aplique o Shu-Ha-Ri com eles. Ajude-os a desenvolver o conhecimento um passo de cada vez, mantenha-os no caminho pra que eles possam apreender o conhecimento de forma que eles apliquem o que foi aprendido corretamente. Você só tem a ganhar ao conseguir formar uma mão de obra mais qualificada e capaz de resolver os problemas.

Se você é um aprendiz, antes de tudo, tenha paciência. Você nunca vai chegar ao Ri sem ter passado pelos estágios anteriores. Não force o seu caminho entre as fases, pois isso só vai garantir que você falhe no futuro. Programar, assim como tudo o mais, exige esforço, repetição e interesse.

A grande vitória dessa minha experiência, não foi minha, mas de Pedro, que se esforçou além do normal pra atingir os objetivos. Leu livros, escreveu sobre as suas experiências e continua fazendo isso (se você continuar seguindo no blog dele, vai ver que ele continuou fazendo resumos sobre os outros livros). Ele poderia ter descansado e feito como todos os outros fazem, se esforçar apenas o suficiente pra ter o básico, mas é esse interesse em ir além que faz as melhores pessoas. Sucesso é muito mais persistência do que talento, já que talento sem ação nunca vai levar ninguém a lugar nenhum.

E você, quais são as suas experiências nisso?

Apêndice

E aqui segue a listinha de livros que ele recebeu (ao menos eu acho que tenham sido esses):

Como ser chutado de uma avaliação pra uma vaga na fase de currículos

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 27 Jan 2011 00:00:00 UTC

Aviso aos navegantes: esse texto foi publicado originalmente em 27/01/2011, a OfficeDrop não está mais recebendo currículos pra essas vagas.

Essa semana anunciamos vagas pra trabalho Home Office pra OfficeDrop:

Recebemos vários currículos, alguns bons e outros inacreditavelmente ruins. Erros crassos na construção do currículo ou completo desleixo com o mesmo fizeram com que vários candidatos fossem desconsiderados simplesmente porque era impossível retirar alguma informação útil do documento digital. O currículo ou continha lixo demais ou era tão direto que era impossível de se adivinhar quais eram as tarefas que a pessoa fazia no trabalho.

Resolvi então que seria uma boa hora pra fazer uma listinha dos erros que nós encontramos pra ficar pra posteridade.

Apresentação

Várias pessoas enviaram currículos como arquivos “.doc” do Office. Se você trabalha com tecnologia a tempo o suficiente pra saber utilizar mouse, teclado e ler ebooks provavelmente sabe que PDF é o formato padrão e mais bem aceito no mercado. Quando você envia um arquivo “.doc” e a pessoa abre no OpenOffice a formatação pode ser perdida ou ele simplesmente não funcionar e se não funcionar, você está fora. Sempre que for enviar um currículo, faça-o em PDF, não use os formatos do Office, OpenOffice e muito menos do Pages no Mac.

Vagas de tecnologia normalmente não exigem que você tenha uma boa aparência, então não é necessário colocar fotos. Sempre que vejo um currículo com fotos lembro do primeiro currículo que fiz aos 14 anos de idade. Também não use imagens ou logotipos de empresas das quais você usa as ferramentas. Quando você tira uma certificação Java, ganha o direito de usar a marca, mas isso não quer dizer que você deve encher o seu currículo com os logos do Java somente pra dizer que você é certificado. Quem é mais importante, a certificação ou você?

Ao enviar o seu currículo, pense que várias outras pessoas vão fazer o mesmo, então mandar um anexo com o nome “Curriculum Vitae” ou “Resume” dá mais trabalho pra o avaliador. Sempre coloque o seu nome no arquivo anexado, de forma que fique claro, mesmo sem abrir o documento, de quem é aquele currículo (como em “Currículo - José da Silva.pdf”).

As vagas pra OfficeDrop foram anunciadas em inglês, a empresa é americana e espera-se que o candidato seja capaz de se comunicar de forma escrita e falada em inglês (está na descrição de cada uma das vagas). Em que língua você acha que o seu currículo deve estar? Quando você não presta atenção na descrição da vaga, já sinaliza que pode ser apenas mais um “atirador de currículos” e provavelmente vai ser sumariamente cortado da avaliação. Se você não é capaz de traduzir o seu currículo pra inglês e recorre a ferramentas como o Google Translate pra fazer isso, você não se encaixa no “ser capaz de ler, escrever e falar inglês”.

Experiência de trabalho

Todas as vagas que nós publicamos exigem experiência prévia com programação e todas elas são pra trabalhar programando em tecnologias atuais, como Ruby, C# e Objective-C (essa nem tão atual né…), então a única coisa que nos interessa no seu currículo são trabalhos relacionados a programação. Se você dá aulas de religião na escolinha da sua igreja, isso não interessa (não nessa parte, vai interessar mais a frente).

O seu currículo deve começar com uma breve descrição das tecnologias que você já trabalhou e os papéis que já desempenhou nos seus diversos trabalhos e projetos.

Ao colocar experiência de trabalho, não escreva somente:

“Janeiro 2008, Dezembro 2010” – Analista de sistemas, FooBar Tecnologia

Descreva em algumas linhas o que você fazia, quais tecnologias você utilizava, o que foi desenvolvido no projeto. Explique quais eram as suas atribuições e, principalmente, tenha certeza de que tudo o que você colocou no seu resumo de experiências esteja citado nas suas experiências de trabalho. Se você coloca que “sabe” Rails e nenhuma das suas experiências diz que você já usou Rails, vou ter que duvidar das suas palavras.

Se você trabalha há muito tempo e teve empregos não relacionados a programação, o melhor é não coloca-los no seu currículo, porque eles vão somente aumentar o tamanho dele e terminam por não dizer nada de você. Quanto maior o seu currículo, maiores as chances do avaliador se chatear e começar a “pular” partes dele.

Eventos

Isso é um dos comentários mais antigos sobre o assunto, mas as pessoas continuam cometendo o mesmo erro, o valor de você “ir” a um evento é basicamente nulo. Por que? Porque é muito difícil de se mensurar o conhecimento adquirido e, principalmente, todo mundo sabe que ninguém vai a evento somente pra assistir palestras. Quer colocar que participou de um evento no seu currículo? Demonstre que ter participado desse evento teve algum efeito na sua formação profissional com código.

Já se você palestrou ou ministrou cursos durante o evento, você deve colocar isso no seu currículo, junto com links pra Slideshare se você tiver usado, projeto no GitHub desenvolvido durante o curso/palestra ou qualquer outro material que você tenha usado durante a apresentação. Dizer que você falou sobre “A dicotomia capitalista de Atirei o Pau no Gato” sem nenhum material de suporte também é inútil.

Cursos complementares

Quando além da sua formação acadêmica você fez cursos complementares, aqui vale a mesma regrinha da experiência de trabalho, só coloque cursos que estejam relacionados ao trabalho que você vai executar na vaga. Vários currículos que nós recebemos tinham “Curso de montagem e manutenção de computadores” ou “Operador Office: Word, Excel e Power Point”.

Se você não fez nenhum curso relacionado ao assunto da vaga, não coloque nada. Controle esse seu ímpeto de querer fazer o maior currículo do mundo, quanto maior o seu currículo, menores as chances de você ser selecionado, porque provavelmente vai ser tachado de enrolão.

Um plus pra cursos é, assim como nos eventos, você ter material pra mostrar com base no que aprendeu no curso. Fez um curso de desenvolvimento pra iOS? Faça um projetinho com o que você aprendeu e coloque no GitHub. Isso demonstra que você não estava somente de corpo presente nas aulas.

Certificações

Em empresas menos enterprisey, certificações já não tem lá muito valor, mas mesmo assim são uma fonte interessante de dados sobre a pessoa no currículo. Todo mundo tem uma certificação SCJP, mas quase ninguém tem uma SCEA, colocar uma SCEA no currículo realmente faz diferença.

Evite colocar certificações pouco conhecidas ou de sites obscuros, como as do “RentACoder.com”, “Brainbench” ou correlatos. Ninguém conhece e, mais uma vez, parece que você quer “gerar conteúdo” pra o seu currículo em vez de realmente demonstrar conhecimentos na área.

Github, side-projets e hobbys

Se você tem conta no GitHub, só coloque ela como “referencia” no seu currículo se você realmente tem alguma coisa própria lá. Se tudo o que você fez foi dar forks em milhares de projetos e nunca deu commit em nenhum deles, é melhor nem colocar o link. Se você tem side projects, é interessante colocar eles também no seu currículo falando o que fez, pra que fez e se ainda está mantendo o projeto.

Algumas empresas também gostam de saber se as pessoas que estão sendo contratadas são normais e fazem outras coisas da vida além de trabalhar. Então, dependendo do perfil da vaga, você pode adicionar uma seção de hobbys no seu currículo, falando que você dá aulas na escola dominical da igreja, gosta de jogar vídeo game, surfar ou whatever.

Fechando

Pense no seu currículo como a primeira impressão que você vai causar e pense que ele nem pode ser pequeno demais pra não dizer nada nem deve ser grande demais pra não parecer enrolação. Currículos com até 3 folhas são aceitáveis, depois disso pode ter certeza de que o avaliador não está mais olhando pro conteúdo e está contando os minutos pra esse sofrimento acabar.

Seja direto, coloque informações que qualifiquem você pra vaga em questão e preste atenção nas entrelinhas da vaga, pra não enviar um currículo em português pra uma empresa americana. Mas não se empolgue, o currículo é somente pra que você seja considerado pra vaga, a entrevista e os assigments que vão ser passados é quem vão dizer se você realmente vai chegar lá ou não, mas é sempre bom causar uma boa primeira impressão.

Handling various rubies at the same time in your machine with RVM – Ruby Version Manager

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 25 Jan 2011 00:00:00 UTC

If you’ve been working in Ruby for more than a year you have probably seen a lot of changes in the landscape. We saw a lot of gems adding compatibility layers to run on Ruby 1.9.2, Rails 3 was finally released (also supporting 1.9.2) and new gems using 1.9.2 features are also showing up.

But guess what? You’re stuck at 1.8 (1.8.7 if you’re lucky) for a lot of projects and you’re in deep fear that installing the latest Ruby 1.9.2 to try all these new fancy things is going to wreak havoc on your environment and you’re going to be FUBAR.

Worse, you still have projects running on Ruby 1.8.6 (with that nasty SMTP-TLS bug) and if you upgrade to a newer Ruby you might have false positives in your codebase and things are going to break in production. You’re already FUBAR, you might think.

But fear not! There’s a knight in shiny armor riding for his damsel in distress! (yes, YOU) And this knight is RVM!

RVM is a set of bash scripts, written by Wayne E. Seguin (and others) that allows you to do just that, install various different Rubies on your machine in a way that they do not conflict with each other. You can even go farther and install different “gem groups” for every project you have so that each one lives in it’s own sandbox without breaking the dependencies or installing gems that are needed for only one application.

Here you’ll learn to setup, install and do the basic workflow when using RVM on your daily development.

Installing RVM

First make sure that your system has a sane build environment by installing compilers/make/whatever it is needed in your OS to build source code. If you’re on an Ubuntu machine, the “build-essential” package should do it, on a Mac it’s the XCode Unix Tools. Once you’re done with that, install Git and from a console type:

bash << ( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

Once finished, open up your “.bash_profile” (or equivalent) and append this to the end of it:

[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"

Close your console and open it again, type “rvm -v” and you should see the version info for your RVM install.

Installing Rubies

Once you have RVM ready, the first thing to do is to start installing the rubies you’ll want to use. Different than what you might be used to, RVM installs the rubies and gems in your user’s directory (under “~/.rvm”), so you will NEVER use “sudo” when installing anything with RVM. Everything is always installed in your user’s home directory (you could possibly install RVM as root, but you DO NOT WANT to do that, trust me).

Let’s start by installing Ruby 1.8.7, Ruby 1.9.2, Ruby Enterprise Edition and JRuby:

rvm install ruby-1.8.7,ruby-1.9.2,ree,jruby

This is going to take a while as RVM is going to pull the source code for all these rubies and build them in your environment. Once it’s over, tell RVM which one is your “default” Ruby (the one that’s going to be loaded by default on your consoles). Let’s use “ruby-1.8.7” as the default:

rvm  --default ruby-1.8.7

Now, every console that opens up will use the Ruby 1.8.7. If you want to switch to another Ruby, here’s how you’d do it:

rvm ruby-1.9.2

To see which Rubies you have installed currently:

rvm list

Installing gems

Before start installing gems you should probably fine tune your “~/.gemrc” file by telling Rubygems not to generate “ri” or “rdoc”, this will absurdly speed up gem installs (unless you really want that). Here’s my “~/.gemrc” file (the most important configs are “install” and “update”):

---
:verbose: true
:bulk_threshold: 1000
install: --no-ri --no-rdoc --env-shebang
update: --no-ri --no-rdoc --env-shebang
:sources:
- http://gemcutter.org
- http://gems.rubyforge.org/
- http://gems.github.com
:benchmark: false
:backtrace: false
:update_sources: true

Now that you’re done, each Ruby that you installed comes with it’s own separate Rubygems environment, which means you can install gems in one Ruby and it isn’t going to affect the other ones. With RVM you can even go further and define “gemsets” that are groups of gems that are independent even from the “main” gems available for that specific Ruby.

The way you’ll work with your rubies and gemsets depends on what you really want. I, personally, love to have a gemset for every single application and that’s what I’m going to talk about now, I hope this workflow works for you too.

First thing to do is to define which are the most common needs you’re going to have using these rubies. In my case, I have Ruby 1.8.7 for my Rails 2.3 projects, so I’ll install Rails and some other common gems on it:

rvm  ruby-1.8.7
gem install rails –v 2.3.10
gem install will_paginate nokogiri mysql sunspot sunspot_rails

Install here only the gems you believe you’re going to use on most of your projects.

Now I’d like to have the Ruby 1.9.2 for Rails 3 development, so I’m going to install the latest Rails on it:

rvm ruby-1.9.2
gem install rails nokogiri mysql2

Once you have the “base” gems ready for every Ruby, you can start defining “gem sets” for your projects. The best way to do this is to create a “.rvmrc” file at the root directory of every project, here’s how the file would look like:

rvm ruby-1.9.2@my_application --create

Once you enter the directory that has the “.rvmrc” file, RVM will look at it and ask you if you want to “trust” the file, once you select “yes”, it’s going to create the gemset named in this file (in our case “my_application” using Ruby 1.9.2). RVM automatically switches to the Ruby and the gemset defined in the file whenever you enter the directory, you don’t have to do anything.

Now, if you run a “gem list” inside this directory you’ll see that there are only two gems in this gemset, “bundler” and “rake”. Not really super, right? But we have already installed Rails 3 in the main Ruby 1.9.2 path, so we can easily migrate them to this new gemset, here’s how you’d do it:

rvm gemset copy ruby-1.9.2 ruby-1.9.2@my_application

This will get all gems we placed in the base ruby-1.9.2 install and copy them over to our application’s gemset. You could even copy the gems from another gemset that you already have.

The beauty in using “.rvmrc” files and gemsets for you projects is that once you push this file to your source control system, everyone that accesses the project will be “forced” to have a gemset and it’s surely going to be easier to manage dependencies.

Also, if you’re combining RVM with Bundler you’ll have a much more reliable and repeatable environment, you can just move this project anywhere, create the gemset, install the necessary gems with Bundler and just start coding.

Conclusion

RVM is one of those tools you MUST have in your toolset when writing Ruby applications, specially if you have to handle many different Rubies, projects or environments and don’t want to start growing white hair (or worse, LOSE your hair) while doing it.

Full text search in in Rails with Sunspot and Solr

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 13 Jan 2011 00:00:00 UTC

[caption id=”attachment_147” align=”alignleft” width=”130” caption=”The book you should get to dig deeper into Solr”][/caption]Click here if you want to see a PDF version of this tutorial.

Full source code for this tutorial is available at GitHub.

Everyone wants to take their databases to run everything as fast as possible. We usually say query less, add more caching mechanisms, add indexes to the columns being searched, but another solution is not to use the database at all and look for better solutions for your querying needs.

When querying for text in our databases, we’re often doing “LIKE” searches. Like searches are only performant if we have an index in that field and the query is written in a way that the index is used. Imagine that you have a field “name” and it contains the text “Battlestar Galactica”. This query would be able to run and use the index:

SELECT p.* FROM products p WHERE p.name LIKE “Battlestar%”

The database would be able to optimize this query and use the index to find the expected row. But, what if the query was like this one:

SELECT p.* FROM products p WHERE p.name LIKE “%Galactica”

[caption id=”attachment_135” align=”alignright” width=”300” caption=”Your DBA getting ready to hit you”][/caption]Database indexes usually match from left to right, so, unless you have a nasty trick under your sleeve, this query will just look at ALL the rows in the products table and perform a match on every “name” column before returning a result. And that’s Really Bad News for you, as the DBA will probably come for you holding a Morning Star to beat you badly. So, querying with “LIKE” when you what you need is full text search isn’t nice.

That’s where full text search based solutions come in for help. Tools like Solr allow you to perform optimized text searches, filter input, categorization and even features like Google’s “Did you mean?”.

In this tutorial you’ll learn how to add full text searching capabilities to your Rails application using Sunpot and Solr. We will also delve a little bit into Solr’s configuration and learn how to use specific tokenizers to clear input, perform partial matching of words and faceting results.

This project uses Rails 3 and Ruby 1.9.2, you’ll find a Gemfile and and “.rvmrc” with all dependencies declared, it should be pretty easy to follow or setup your environment based on it (if you’re not using RVM, that’s a GREAT time to learn using it).

You can possibly follow this tutorial with a previous Rails version and without Bundler or RVM, given all models and most of the code will look exactly the same in Rails 2 and Sunspot is compatible to Rails 2 too.

The source code for this example application is available at GitHub here.

Starting the engines

Download the Sunspot source code from Github.

Enter the project folder and go to “sunspot/solr-1.3”, inside that folder you should see a “solr” folder, copy this folder into your project’s folder. This is where the general Solr configuration is going to live, don’t worry about these files just yet, we’ll get to them later in this tutorial.

Now create a “sunspot.yml” file under your project’s “config” folder, here’s a sample:

Listing 1 – sunspot.yml

development:
  solr:
    hostname: localhost
    port: 8980
    log_level: INFO
  auto_commit_after_delete_request: true

test:
  solr:
    hostname: localhost
    port: 8981
    log_level: OFF

production:
  solr:
    hostname: localhost
    port: 8982
    log_level: WARNING
  auto_commit_after_request: true

You can have different configurations for every environment you’re running. To see all configuration options, go to the Sunspot source code and head to the “sunspot_rails/lib/sunspot/rails/configuration.rb” file.

Now we’ll create two models, Product and Category, so let’s start by creating the migration that will setup them:

rails g migration create_base_tables

Listing 2 – create_base_tables.rb

class CreateBaseTables &lt; ActiveRecord::Migration

  def self.up
    create_table :categories do |t|
      t.string :name, :null =&gt; false
    end

    create_table :products do |t|
      t.string  :name, :null =&gt; false
      t.decimal :price, :scale =&gt; 2, :precision =&gt; 16, :null =&gt; false
      t.text    :description
      t.integer :category_id, :null =&gt; false
    end

    add_index :products, :category_id

  end

  def self.down
    drop_table :categories
    drop_table :products
  end

end

Now we move on to the basic models, starting with the Category model:

Listing 3 – category.rb

class Category &lt; ActiveRecord::Base

  has_many :products

  validates_presence_of :name
  validates_uniqueness_of :name, :allow_blank =&gt; true

  searchable :auto_index =&gt; true, :auto_remove =&gt; true do
    text :name
  end

  def to_s
    self.name
  end

end

Here in the Category class we see our first reference to Sunspot, the “searchable” method, where we configure the fields that should be indexed by Solr. At the Category class, there’s only one field that’s useful at this moment, the “name”, so we tell Sunspot to configure the field name to be indexed as “text” (you usually don’t want your text indexed as “string”, as it will only be a hit in a full match).

The :auto_index and :auto_remove options are there to let Sunspot automatically send your model to be indexed at Solr when it is created/updated/destroyed. The default is “false” for both values, which means you have to manually send your data to Solr and unless you really want to do that, you should keep both of these values as “true” in your models.

Now lets look at the Product class:

Listing 4 – product.rb

class Product &lt; ActiveRecord::Base

  belongs_to :category

  validates_presence_of :name, :description, :category_id, :price
  validates_uniqueness_of :name, :allow_blank =&gt; true

  searchable :auto_index =&gt; true, :auto_remove =&gt; true do
    text :name, :boost =&gt; 2.0
    text :description
    float :price
    integer :category_id
  end

  def to_s
    self.name
  end

end

In our Product class things are a little bit different, we have more fields (and more kinds) being indexed. “float” and “integer” are pretty self explanatory, but the “name” field has some black magic floating around, with the “boost” parameter. Boosting a field when indexing means that if the match is in that specific field, it has more “relevance” than if found somewhere else.

Imagine that you’re looking for Iron Maiden’s “Powerslave” album. You go to Iron Maiden’s Online Store and search for “powerslave”, hoping that the album will be the first hit, but then you see “Live After Dead” before “Powerslave”. Why did it happen? The “Live After Dead” album contains the “Powerslave” song in it’s track listing, so it’s a match as much as the real “Powerslave” album. What we need here is to tell the search tool that if a match is on an album name, it has higher relevance than if the hit is in the track listing.

Boosting allows you to reduce these issues. Some fields are inherently more important than others and you can tell that to Solr by configuring a “:boost” value for them. When something matches on them, the relevance of that match will be improved and it should come up before the other results in search.

Searching

Now let’s take a look at the ProductsController to see how we perform the search:

Listing 4 – products_controller.rb

class ProductsController &lt; ApplicationController

  def index
    @products = if params[:q].blank?
      Product.all :order =&gt; 'name ASC'
    else
      Product.solr_search do |s|
        s.keywords params[:q]
      end
    end
  end

end

As you can see, searching is quite simple, you just call the solr_search method and send in the text to be searched for. One thing that I don’t like about Sunspot is that searches do not return an Array like object, you get a Sunspot::Search::StandardSearch object that has, as a property, the results array which contains the records returned by the search.

Here’s a simple way to fix this issue (I usually place the contents of this file inside an initializer in “config/initializers”):

Listing 5 – sunspot_hack.rb

::Sunspot::Search::StandardSearch.class_eval do

  include Enumerable

  delegate(
    :current_page,
    :per_page,
    :total_entries,
    :total_pages,
    :offset,
    :previous_page,
    :next_page,
    :out_of_bounds?,
    :each,
    :in_groups_of,
    :blank?,
    :[],
    :to =&gt; :results)

end

This simple monkeypatch makes the search object itself behave like an Enumerable/Array and you can use it to navigate directly in the results, without having to call the “results” method. The methods usually used by will_paginate helpers are also included so you can pass this object to a will_paginate call in your view and it’s just going to work.

Indexing

Now that all the models are in place, we can start fine tuning the Solr indexing process. First thing to understand here is what happens when you send text to be indexed by Solr, let’s get into the tool, starting the server:

rake sunspot:solr:run

This rake task starts Solr in the foreground (if you wanted to start it in the background, you’d use “sunspot:solr:start”). With Solr running, you should add some data to the database, this tutorial’s project on Github contains a “seed.rb” file with some basic data for testing, just copy it over your project.

Also copy the “lib/tasks/db.rake” from the project to your project, it contains a “db:prepare” task that truncates the database, seeds it and then indexes all items in Solr and we’re doing to be reindexing data a lot.

With everything copied, run the “db:prepare” task:

rake db:prepare

This will add the categories and products to your database and also index them in Solr. If this task did run successfully, head to the Solr administration interface, at this URL:

http://localhost:8980/solr/admin/schema.jsp

Once you go to it, click on the “FIELDS”, then on “NAME_TEXT”, you should see a screen just like the one in image 1: [caption id=”” align=”alignnone” width=”1023” caption=”Image 1 – Solr schema browser”][/caption]

If you don’t see all the fields that are available in this image, your “rake db:prepare” command has probably failed or Solr wasn’t running when you called it.

What we see here is the information about the fields we’re indexing. This specific field contains all data from the name properties from both Category and Product classes, as you can notice from the top 10 terms.

The name field is not indexed by it’s full content, as a relational database would usually do, the text is broken into tokens, by the solr.StandardTokenizerFactory class in Solr. This class receives our text, like “Battlestar Galactica: The Boardgame” and turns it into:

[“Battlestar”, “Galactica”, “The”, “Boardgame”]

This is what gets indexed and, ultimately, searched by Solr. If you open the web application now and try to search for “battle”, you won’t have any matches. If you search for “Battlestar”, you get the two products that match the name.

Everything when indexing information in Solr revolves around building the best “tokens” available for your input. You have to teach Solr to crunch your data in a way that makes sense and makes it easy to search for, and adding filters to the indexing process does this. While in the same page as Image 1 above, click on the “DETAILS” links as shown in Image 2:

[caption id=”” align=”alignnone” width=”553” caption=”Image 2 – Viewing the analysis and search filters”][/caption]

Each field in Solr has two analyzers, one is the “index” analyzer, that prepares the input to be indexed and the other is the “query” analyzer that prepares the search input to finally perform a search. Unless you have some special need, both of them are usually the same.

In our current configuration, we have the same two filters for both of the analyzers. The StandardFilterFactory filter removes punctuation characters from our input (the “:” in “Battlestar Galactica: The Boardgame” is not in our tokens) and the LowerCaseFilterFactory makes all input lowercased so we can search with “baTTle”, “BATTLE”, “BaTtLe” and they’re all going to work.

Before we move on to add more filters to our analyzers, let’s take a look at the analyzer screen in Solr Admin at - http://localhost:8980/solr/admin/analysis.jsp?highlight=on

In this screen we see how our input is going to be transformed into tokens by the configured analyzers.

[caption id=”” align=”alignnone” width=”1019” caption=”Image 3 – Solr analyzer page”][/caption]

In this screen we have selected the “name_text” field in Solr. In the “Field value (Index)” you enter the values you’re sending to be indexed, just like you would send from your model property, in the “Field value (Query)” you enter the values you’d use to search.

Once you type and hit “Analyze” you should see the output just below the form as we see in Image 3. This output shows how your input is transformed into tokens by the tokenizer and filters, this way you can easily experiment by adding more filters and seeing if the output really matches the way you’d expect it to. This analysis view is your best friend when debugging search/indexing related issues or trying out ways to improve the way Solr indexes and matches your data.

Customizing fields

Now that you have an idea about how the indexing and searching process work, let’s start to customize the fields in Solr, open up the “solr/conf/schema.xml” file and look for this reference:

Listing 6 – solr/conf/schema.xml except

&lt;fieldtype class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot; name=&quot;text&quot;&gt;
      &lt;analyzer&gt;
        &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;
        &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;
        &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;
      &lt;/analyzer&gt;
    &lt;/fieldtype&gt;

If you look at Image 1, where we saw the “name_text” configuration, you’ll see that the field type is “text”, this except above is the configuration for all fields of type “text”, which means that if we add more filters here we’ll affect all fields of this type. This greatly simplifies the way we configure the tool, as we don’t have to define explicit configurations for every single field that our models have, we can just reuse this same “text” config for all fields that are supposed to be indexed as text.

But that’s a lot of talking, let’s get into action!

Let’s start the job by looking at our indexed data from before:

[“battlestar”, “galactica”, “the”, “boardgame”]

The “the” is mostly useless, as it’s going to be available in almost all properties and no one is ever going to search for “the” (oh yeah, there might be that ONE guy that does it). In Information Retrieval lingo, “the” is a stop word, it usually doesn’t have meaning by itself and doesn’t represent valuable information for our indexer, removing all stop words from your input improves performance and the relevance of your results.

Given that this is a common operation, Solr already contains a filter that’s capable of removing all stop words from your data, the solr.StopFilterFactory, let’s see how we can add it to our config:

Listing 7 – solr/config/schema.xml except

&lt;fieldtype class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot; name=&quot;text&quot;&gt;
  &lt;analyzer&gt;
    &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StopFilterFactory&quot; words=&quot;stopwords.txt&quot; ignoreCase=&quot;true&quot;/&gt;
    &lt;filter class=&quot;solr.ISOLatin1AccentFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.TrimFilterFactory&quot; /&gt;
  &lt;/analyzer&gt;
&lt;/fieldtype&gt;

If you look at the “solr/config” folder you’ll se a “stopwords.txt” file that already contains most of the common stop words in English, you can add or remove words from there as needed and if you’re not indexing English text you can just remove the English names and add your language’s stop words. Now change this in your “solr/config/schema.xml” file and stop and start Solr again and open the analyzer:

[caption id=”” align=”alignnone” width=”1011” caption=”Image 4 – Solr analyzer page “][/caption]

As you can see, in the last step, the “the” was removed from both the index input and the query input, we’re maintaining only the pieces of information that are really useful, this makes our index smaller and also speeds up searching.

While you were not looking, we have also added two other filters, solr.ISOLatin1AccentFilterFactory, that removes accents from words in Latin based languages, like Portuguese. If the input is “não”, it becomes “nao”. And after that there’s solr.TrimFilterFactory, that removes unnecessary spaces from our tokens.

Partial matching

Another pretty common need is to be able to match only a part of a word, usually a prefix. In the beginning of the tutorial, we saw that searching for “battle” doesn’t yield any results, while “battlestar” does. This happens because Solr, by default, only sees a match if it’s a full match. The word you entered must be exactly the same as a token that’s available in the index, if there is no exact match, Solr you tell you that there are no results.

If you look at Lucene’s Query Parser Syntax (Solr is somewhat a web interface to Lucene) you’ll see that you can use the “” operator to perform a partial match. We could then search for “battle” and this would yield the results we expect, but doing this kind of partial matching is slow and could possibly become a bottleneck for your application, so we have to figure out another way to do this.

When all you need is prefixed partial matching, the solr.EdgeNGramFilterFactory is your best friend. It will break words into pieces that will then be added to the index, so it looks like you have partial matching, but in fact the partials are tokens by themselves in the index, let’s see how our config would look like in this case:

Listing 8 – solr/config/schema.xml except

&lt;fieldtype class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot; name=&quot;text&quot;&gt;
  &lt;analyzer type=&quot;index&quot;&gt;
    &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StopFilterFactory&quot; words=&quot;stopwords.txt&quot; ignoreCase=&quot;true&quot;/&gt;
    &lt;filter class=&quot;solr.ISOLatin1AccentFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.TrimFilterFactory&quot; /&gt;
    &lt;filter class=&quot;solr.EdgeNGramFilterFactory&quot;
      minGramSize=&quot;3&quot;
      maxGramSize=&quot;30&quot;/&gt;
  &lt;/analyzer&gt;
  &lt;analyzer type=&quot;query&quot;&gt;
    &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.StopFilterFactory&quot; words=&quot;stopwords.txt&quot; ignoreCase=&quot;true&quot;/&gt;
    &lt;filter class=&quot;solr.ISOLatin1AccentFilterFactory&quot;/&gt;
    &lt;filter class=&quot;solr.TrimFilterFactory&quot; /&gt;
  &lt;/analyzer&gt;
&lt;/fieldtype&gt;

As you can see, now we have two sections in our , one of the analyzers is for “index” and the other is for “query”. This is needed because we don’t want to have our search parameters being transformed for a partial match. If the user is searching for “battle”, it doesn’t makes sense to show him results for “bat”, so the generation of pieces of each word should be done only when indexing information.

Now restart your Solr instance and head run again the form we had in the analyzer view, you should see something like Image 5:

[caption id=”” align=”alignnone” width=”1330” caption=”Image 5 – Analyzer output with partial matching enabled”][/caption]

Looking at the output, “battlestar” became:

[“bat”, “batt”, “battl”, “battle”, “battles”, “battlest”, “battlesta”, “battlestar”]

Now, if you search for “battle”, you should find all products that have “battle” as a prefix in any of their words and the search input is not affected by this change.

Faceting

Faceting of results is YACF (Yet Another Cool Feature) that you have when using Solr and Sunspot. “What does that mean?”, you might ask, it means that Solr is able to organize your results based on one of it’s properties and tell you how many results did match for every property value.

“I still don’t get it”, you might be thinking now. In our Product model we’re indexing the “category_id” property, we’ll tell Sunspot to facet our search based on the “category_id” field and Sunspot will tell us how many matches each category had, even if we’re paginating the results. Let’s see how our searching code would change:

Listing 9 – products_controller.rb except

  def index
    @page = (params[:page] || 1).to_i
    @products = if params[:q].blank?
      Product.paginate :order =&gt; 'name ASC', :per_page =&gt; 3, :page =&gt; @page
    else

      result = Product.solr_search do |s|
        s.keywords params[:q]
        unless params[:category_id].blank?
          s.with( :category_id ).equal_to( params[:category_id].to_i )
        else
          s.facet :category_id
        end
        s.paginate :per_page =&gt; 3, :page =&gt; @page
      end

      if result.facet( :category_id )
        @facet_rows = result.facet(:category_id).rows
      end

      result
    end
  end

The search code really changed a lot, now if there’s a “category_id” parameter we will use that to filter our search, if there isn’t we’re going to perform faceting with the “s.facet :category_id” call. There’s also a slight change to the “product.rb” class, let’s see it:

Listing 10 – product.rb except

  searchable :auto_index =&gt; true, :auto_remove =&gt; true do
    text :name, :boost =&gt; 2.0
    text :description
    float :price
    integer :category_id, :references =&gt; ::Category
  end

We’ve added the “:references => ::Category” to the “:category_id” field configuration so Sunspot knows that this field is, in fact, a foreign key to another object, this will allow Sunspot to load the categories in the facets automatically for you.

The “result.facet(:category_id)” asks the search object for the array that contains the facets returned for the :category_id field in this search. Each row in this list contains an “instance” (which, in our case, is an Category object) and a “count”, that’s the number of hits in that specific facet. Once you get your hands at the rows, we can use it in our view, let’s see how we used them:

Listing 11 – products/index.html.haml except

  - if !@facet_rows.blank? &amp;&amp; @facet_rows.size &gt; 1
    %ul
      - for row in @facet_rows
        %li= link_to( "#{row.instance} (#{row.count})", products_path( :q =&gt; params[:q], :category_id =&gt; row.instance ) )

If there are facets available, we use them to add links that will make the user filter based on each specific facet, each row object has an instance and a count, and we use both in the interface to tell the user which category is it and how many hits it had. Look at how our user interface looks like:

[caption id=”” align=”alignnone” width=”376” caption=”Image 6 – Faceting information”][/caption]

And now you finally have search functionality added to a Rails project, with partial matching, faceting, pagination and input cleanup. Just forget that you have ever performed a “SELECT p.* FROM products p WHERE p.name LIKE ‘%battle%’” and be happy to be using a great full text search solution.

Conclusion

Hopefully this tutorial should be enough to get you up and running with Solr, for more advanced features I’d recommend you to search on the Solr wiki and buy “Solr 1.4 – Enterprise Search Server” by David Smiley and Erick Pugh.

Agilidade, snake oil, power balances e a falácia do “nós somos diferentes”

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 05 Jan 2011 00:00:00 UTC

Um companheiro de uma das listas que participo recebeu um email bem interessante, e eu preciso fazer alguns comentários sobre isso.

From: Oleg Puzyreff Date: Tue, Jan 4, 2011 at 9:36 PM Subject: Optimized Scrum training course with Dr. Jeff Sutherland, co founder of Scrum To: fulano@sicrano.com Dear Fulano, qaSignature in partnership with Scrum Inc. is offering Optimized Scrum, a new Scrum training course designed to super-accelerate software development and testing. DO NOT take this class if: • You are happy with your job being outsourced; • You already know everything and do not need to learn the latest trends in Software development; • You love to be busy to the point you work late and on weekends; This course is offered two ways: As a one-day course for CSMs, developers and QA staff: http://scrumtraininginstitute.com/classes/show/458 OR As an optional course* packaged as a third day with Scrim Inc's CSM training: http://scrumtraininginstitute.com/classes/show/459 *This optional course is not part of CSM certification program and not required for ScrumMaster certification. I hope to see you there, and let me know if you have any questions.

Dizer que isso é charlatanismo é chover no molhado. Nós vemos produtos e serviços que prometem soluções mágicas pros nossos problemas diariamente, é só olhar todo o burburinho que foi gerado pelas pulseiras Power Balance.

Marketing é vender sonhos, transformar produtos que fazem a mesma coisa que todos os outros fazem em únicos e exclusivos para os seus clientes. O homem atual já sofreu tanta influência desse mesmo marketing que é capaz de perceber onde termina a realidade e começa o bla-bla-bla publicitário.

Mas se as pessoas já tem conhecimento do que é verdade e mentira em uma campanha de marketing, por que elas se dão ao trabalho de comprar produtos como o Power Balance?

Na mini-série que antecede o seriado Battlestar Galactica, a ciborgue Caprica Six faz um comentário interessante para o humano Gaius Baltar, assim que ele descobre que foi usado como ferramenta de espionagem:

[caption id=”attachment_38” align=”alignright” width=”300” caption=”Gaius Baltar e Caprica Six”][/caption]<blockquote>Gaius Baltar: I had nothing to do with this. You know I had nothing to do with this. Caprica Six: You have an amazing capacity for self-deception. How do you do that?</blockquote>

Gaius tenta, de todas as formas, negar a realidade para evitar ter que lidar com as consequências dos seus atos. Com essa ação, ele não está somente demonstrando uma parte da sua personalidade, mas representando um traço comum pra grande parte dos seres humanos, queremos a solução mais simples para os problemas, mesmo que pra isso tenhamos que fingir que não somos mais seres racionais.

A propaganda do curso de Scrum segue a mesma idéia das propagandas do Power Balance. Em momento nenhum o email fala de fatos, ele apela para os sentimentos do receptor da mensagem, começa por um ponto fraco comum, o medo de perder o emprego, depois pra humildade, pois ninguém sabe de tudo e finaliza com a idéia de que o receptor trabalha muito além do necessário.

Do ponto de vista do marketing, seria uma peça ridícula, típica dos primórdios da profissão ou de profissionais de baixa qualidade. Mas quando essa mensagem atinge as pessoas, ela fala o que elas querem ouvir. Ela leva a mensagem de que, magicamente, todos os problemas do indivíduo vão ser resolvidos, afinal, é um curso com o Doutor Jeff Sutherland, co-fundador do Scrum.

Você tem palavras mágicas, dignas dos melhores – ou piores – livros de auto-ajuda disponíveis e um rockstar dos processos de desenvolvimento, o que poderia dar errado?

Por mais que a propaganda seja completamente irreal e o mais puro e completo snake oil, as pessoas vão correr e se matricular, na esperança de que seja dessa vez que eles finalmente vão encontrar o santo graal do software. A solução definitiva que vai aumentar a produtividade absurdamente e transformar você no novo Gandalf da programação, capaz de rescrever o kernel do Linux todo usando somente o Emacs.

A ilusão de que existe uma forma absurdamente fácil e simples de se desenvolver software, que Fred Brooks já comentou não existir no texto “No Silver Bullet — Essence and Accidents of Software Engineering” de 1986, continua sendo uma realidade. As pessoas continuam buscando essa ilusão e, ainda pior, a comunidade ágil, que um dia brigou ferozmente contra essa concepção, tornou-se o status quo e agora vende o mesmo snake oil que os consultores de processo vendiam nos anos 90.

[caption id=”attachment_39” align=”alignleft” width=”300” caption=”Óleo de Cobra - A solução para todos os seus problemas”][/caption]Hoje os livros e textos sobre desenvolvimento ágil de software são equivalentes aos mais ralos livros de auto-ajuda e motivação pessoal que temos no mercado. Autores ignoram fontes, escondem inspirações e se definem como criadores únicos e intocáveis dos seus processos, negando inclusive que tenham sido influenciados por outros processos semelhantes que já existiam fora da esfera do software, como o Lean.

A assinatura do manifesto ágil foi um passo na direção certa, colocou os processos de software a par das mudanças anteriores (de mais de uma década antes) da gestão de pessoas nas disciplinas de administração de empresas. Antes que acordemos tarde, do que nunca acordar, mas não podemos parar aqui.

A comunidade ágil se apresentou como sendo algo novo, sem os vícios do passado, aprendendo com o que as outras disciplinas, fora da computação, já faziam. Não podemos vender charlatanismo, mentiras, poções mágicas que curam ou ressuscitam projetos. Desenvolver software em equipes é difícil, negar isso é tão surreal quanto negar que a gravidade existe.

Se ainda se espera que um dia a engenharia de software possa ser realmente reconhecida como uma engenharia, é melhor pararmos com essa conversa fiada e voltarmos a desenvolver software, que é o que realmente importa de verdade.

Meu ambiente de desenvolvimento em 7 itens

mauricio.linhares@gmail.com (Maurício Linhares) — Fri, 31 Dec 2010 00:00:00 UTC

Recebi o convite do @cmilfont e agora estou entrando na brincadeira também, vamos lá.

Máquina/SO

Uso um desktop com Ubuntu Linux + VirtualBox (com 3 Windows instalados) pra testes e também pra tocar música no escritório. O trabalho do dia a dia acontece num MacBookPro de 15’ rodando Snow Leopard, sempre ligado num monitor AOC de 24’. Quando preciso usar o Linux, normalmente conecto nele via VNC (ou simplesmente troco os cabos do monitor). Usei o desktop como máquina principal por uns 2 anos, mas desde o meio do ano estou trabalhando somente no Mac e o Linux ficou somente pra testes e rodar as máquinas virtuais.

No Mac eu uso o Ports pra instalar dependências Unix, que são básicas pro ambiente de desenvolvimento, tentei Homebrew mas ele não se deu muito bem na minha máquina.

Editor/IDE

Aqui a salada é grande, trabalhando com Ruby, Java e Objective-C, não tem como ficar em uma só, então eu termino usando várias. Começa com Aptana e NetBeans pra Ruby, hoje eu tenho usado mais o Aptana por causa do suporte melhor pra HAML, mas de vez em quando ainda volto pro NetBeans pra relembrar os velhos tempos.

Quando é hora de trabalhar com Java, fico no Eclipse mesmo, com o mínimo de plug-ins possível (normalmente uso o pacote que vem direto do site mesmo), já que ele está ficando cada dia mais cheio de inutilidades e eu quase não faço mais desenvolvimento web com ele.

E finalmente, na hora de trabalhar com Mac/iPhone/iPad dev eu vou pro Xcode, que, perto do Eclipse, é basicamente um Notepad mais engraçadinho. Foi meio difícil de entender como tudo funciona no ambiente, ele tem vários quirks bizarros, mas o Interface Builder faz a coisa toda valer a pena, se você estiver trabalhando pra desenvolvimento desktop/mobile. Ele é levinho, só dá trabalho mesmo quando você tem que compilar um projeto grande pra 4 arquiteturas diferentes quando vai gerar um release.

Terminal

Meu terminal é comum, a única coisa de especial que ele tem é o RVM (indispensável se você trabalha com Ruby), não gosto muito de firulas nesse pedaço do meu dia a dia.

Browser

Aqui é Chrome na cabeça, pro uso diário. Quando preciso fazer debug de alguma coisa, vou pro Firefox velho de guerra com Firebug, WebDeveloper e YSlow.

Software

Essenciais pro meu dia a dia são o Chicken of the VNC, Office for Mac, Skype, Chrome, Skitch, Tweetie e Witch.

Source code

Não dá pra inventar muito aqui, sempre que posso, mando os projetos pra contas privadas (ou públicas) no github, atualmente tenho sofrido maus bocados com o Mercurial, é bem diferente de se utilizar Git, mas estou sobrevivendo a experiência.

Música

O iTunes não aguentou o tranco de indexar toda a minha lista de músicas, então o trabalho principal do meu desktop hoje é tocar as minhas músicas no Amarok e mandar os scrobbles pro last.fm, lugar onde eu sempre encontro alguma coisa interessante pra ouvir.

E pra continuar a brincadeira eu indico o @pedromtavares, @yurimalheiros e @vitorbaptista.

Deployment Recipes – Deploying, monitoring and securing your Rails application to a clean Ubuntu 10.04 install using Nginx and Unicorn

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 25 Aug 2010 00:00:00 UTC

[caption id=”attachment_152” align=”alignleft” width=”133” caption=”Agile Web Development With Rails”][/caption]

You can get a PDF version of this tutorial here.

If you’re a developer, setting up a server might not look like something you’d do yourself. Most of the time you have a sysadmin in your team/company/business that can do the job for you. But knowing how to configure a server might make your life easier if you’re rolling on your own so let’s take a look at how we can configure a clean slate Ubuntu 10.04 server machine to run a Ruby on Rails application.

This example used a just created Ubuntu 10.04 image from Rackspace, but you should be able to follow it on any Ubuntu 10.04 install, even a local one.

With this tutorial you’ll learn how to:

Install the libraries usually necessary to run Ruby on Rails application;
Setup Nginx as the HTTP server proxy and statics assets server;
Setup Unicorn as the application server that’s going to run your application;
Setup Monit to watch over the processes
Setup some basic firewall rules
And finally deploy your Ruby on Rails application to it;

Remember that this tutorial is given in an “as is” basis and you should backup all system files we’re changing here.

Aliasing the server at your ssh config

Whenever I’m trying to access the server at this tutorial I’ll be using the “rs” alias, this is an SSH defined alias that you can do yourself. First, create a “.ssh” folder at your home directory:

mkdir ~/.ssh

Then, with your favorite text editor, create a file called “config” on it:

mate ~/.ssh/config

The file content for me is as follows (you should change the IP to match your remote server IP):

Host rs
Hostname 184.106.215.175
User root

This will tell the SSH program that whenever I try to access a server named as “rs” it should login into the IP “184.106.215.175” and with the user “root” (don’t worry, we’ll change this later). So, to login to your server I can just type:

ssh rs

And it should ask for your password.

Generating local SSH keys

You can jump to the next section if you already have SSH Keys for your user, if you don’t have or don’t know what they are go on reading. The SSH keys we’re going to generate here will be used to simplify your login over SSH to the remote server.

With the keys in place you won’t need to type in a password to access the server, the remote server will check your keys and and if they match the keys on it you’ll be logged in. Here’s how to generate a key:

ssh-keygen -t dsa

This will create a “.ssh” folder under you home directory with two files:

id_dsa – your private key that should not be shared to ANYONE
id_dsa.pub – your public key, the one you will be copying to your servers and services like GitHub

Logging in to the remote server without typing a password

To be able to login to the remote machine without a password, you have to copy your public to the remote machine into a user’s home directory. The file with the keys should be at “~/.ssh/authorized_keys”.

First, login into the remote server (as root) and create an “.ssh” folder:

mkdir ~/.ssh

Then, log off and, at your local machine, send your id_dsa.pub file to the remote server with an SCP:

scp ~/.ssh/id_dsa.pub root@rs:.ssh/authorized_keys

The “authorized_keys2” file can accept as many keys you’d like, just place each key in a single line.

Installing the required libraries and applications

Logged in as root in the newly created server, you should start by getting the server libraries up to date:

apt-get update
apt-get upgrade –y

After that, it’s time to start installing the Ruby required libraries

apt-get install build-essential ruby-full libmagickcore-dev imagemagick libxml2-dev libxslt1-dev git-core -y

With the basic ruby libraries, it’s time to select your database. If you’re going for MySQL, here’s what you would install:

apt-get install mysql-server libmysqlclient-dev –y

If you’re going for PostgreSQL:

apt-get install postgresql postgresql-client postgresql-server-dev-8.4 –y

Now we have to install the web server, monit and the firewall:

apt-get install nginx monit ufw –y

And the last step is to install Rubygems from the source. The Ubuntu provided “gem” install isn’t updated as often as the real gem is so it’s better to install it from the source. At the time this tutorial was written, Rubygems 1.3.7 was the latest available, you should check at the Rubygems Rubyforge website witch one is the latest version and install it (http://rubyforge.org/projects/rubygems/).

Now let’s download Rubygems:

wget http://rubyforge.org/frs/download.php/70696/rubygems-1.3.7.tgz

Untar it:

tar -xzf rubygems-1.3.7.tgz

And finally install it:

cd rubygems-1.3.7
ruby setup.rb

This will install the executable “gem1.8”, so we just symlink it to be “gem”:

ln –s /usr/bin/gem1.8 /usr/bin/gem

And then clean up the Rubygems files we downloaded:

cd ..
rm –rf rubygems*

Before start installing gems, as this is a server environment, you probably don’t want to have the rdoc and ri files generated for every installed gem as this is just going to slow down the gem installation for nothing, as no one will be calling “ri” in this server, to disable this stuff, fire up a text editor in the server:

nano ~/.gemrc

And put this on it:

---

:verbose: true

:bulk_threshold: 1000

install: --no-ri --no-rdoc --env-shebang

:sources:

- http://gemcutter.org

- http://gems.rubyforge.org/

- http://gems.github.com

:benchmark: false

:backtrace: false

update: --no-ri --no-rdoc --env-shebang

:update_sources: true

This will avoid ri and rdoc generation while installing gems. And now you can start installing the gems your application will need:

gem install rails unicorn will_paginate nokogiri paperclip sunspot sunspot_rails

Add your database gems as needed in this installation too, if you’re on MySQL:

gem install mysql

If you’re on PostgreSQL:

gem install pg

Configuring the default Rails environment

Fire up your editor at the remote server:

nano /etc/environment

And add the following line to it:

export RAILS_ENV=production

This will make all “script” calls assume that the environment is “production”. The server needs to be rebooted for this change to take effect. This is not needed for Capistrano or Unicorn, as they define the environment themselves.

Setting up a common user

All the work we’ve been doing so far is as a root user, but our application will not be run as root, but as a common user. Still as root, create a user, any name will do, I usually go for deployer:

useradd -m -g staff -s /bin/bash deployer

Give this user a password:

passwd deployer

Now open the “/etc/sudoers” file and make the staff user group be able to perform a “dudo” adding the following line to it:

%staff ALL=(ALL) ALL

This will allow users with the staff group call the “sudo” command, but will prompt for a password before that.

With the user created, you should login to the machine as “deployer” (or your username) and again create an “.ssh“ folder, after that, send in the public key again:

scp ~/.ssh/id_dsa.pub deployer@rs:.ssh/authorized_keys

And also create the same “.gemrc” file you created for the root user at the “deployer” user home directory.

Configuring the Nginx server

Our application is going to be deployed by Capistrano at the “deployer” user home directory, in “/home/deployer/shop”. As we’re going to use Capistrano for deployment, the application will always be available at “/home/deployer/shop/current” and that’s the folder we’re going to use for Nginx.

Given that we’re going to run a single application in this server, we can go on and edit the “/etc/nginx/sites-available/default” file directly, if you’re going to host more than one application in this server you should create separate files for each of them and then symlink them to the “/etc/nginx/sites-enabled/” folder.

Get the Nginx host configuration at GitHub.

The comments in the Nginx file make it self explanatory, but what we’re doing is to send all requests that are not for static files directly to the Unicorn backend (using a UNIX socket for that). Once you have configured the “default” file, you should open the “/etc/nginx/nginx.conf” file as we also have some changes to it. The first one is to change the user that Nginx is going to run as. To avoid permission issues we’re going to make Nginx run as “deployer”, the user that’s going to deploy the application code, this way we can rest safely that no “permission denied” on files will ever happen (at least as long as we don’t deploy as another user).

Here is the main nginx.conf file.

In this file we enable GZIP compression for all requests (and disable it if the client is IE6) and we also define more Nginx workers to do the job, this usually helps while serving static content (if you’ve got a lot of static content to serve, if you don’t you can just go on with just 1 worker). We have also changed Nginx to start as “deployer” with the group “staff”, the same user and group of the files he’s going to be serving.

Now start nginx and see if it’s running correctly:

/etc/init.d/nginx restart

ps aux | grep nginx

Setting up Capistrano

Now it’s time to setup Capistrano at your application to perform deployments. In the command line, go to the root folder of your application and (sudo gem install capistrano – if you don’t have it already at your local machine) :

capify .

This will generate a “Capfile” at your project root folder and a “deploy.rb” file at the “config” folder:

The Capfile is just to boot Capistrano, the file we really change is the “deploy.rb”, where we’re going to configure our remote server information and also the deployer user credentials, here’s how the file looks like for the example project:

set :use_sudo,            false
#tell git to clone only the latest revision and not the whole repository

set :git_shallow_clone,   1
set :keep_releases,       5
set :application,         "shop"
set :user,                "deployer"
set :password,            "asdfg"
set :deploy_to,           "/home/deployer/shop"
set :runner,              "deployer"
set :repository,          "git@github.com:mauricio/linuxfi-loja.git"
set :scm,                 :git

#options necessary to make Ubuntu’s SSH happy

ssh_options[:paranoid]    = false
default_run_options[:pty] = true

role :app, "184.106.215.175"
role :web, "184.106.215.175"
role :db,  "184.106.215.175", :primary =&gt; true

In this Capistrano file you have to define your server, the user that Capistrano is going to use to login into it (and also the password) and the location of your source code repository.

Each of the roles defined at “deploy.rb” file can have more than one server and you could possibly separate your components, the “app” role is where your application server lives, the “web” role is where the web server lives and the “db” role is where your database is. The “:primary => true” for the :db role tells Capistrano which one is the “master” database if you’re running in a “master-slave” database setup, as if you’re doing this the migrations should only be run in the “master” and not on the slave databases.

Configuring Unicorn

With the Capistrano file defined, let’s create a Unicorn config file for our application, here’s a simple configuration example (this file is placed at the root of our project and named as “unicorn.rb”):

# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete

# documentation.

worker_processes 4
# Help ensure your application will always spawn in the symlinked

# "current" directory that Capistrano sets up.

working_directory "/home/deployer/shop/current"
# listen on both a Unix domain socket and a TCP port,

# we use a shorter backlog for quicker failover when busy

listen "/tmp/shop.socket", :backlog =&gt; 64
# nuke workers after 30 seconds instead of 60 seconds (the default)

timeout 30
# feel free to point this anywhere accessible on the filesystem

user 'deployer', 'staff'
shared_path = “/home/deployer/shop/shared”
pid "#{shared_path}/pids/unicorn.pid"
stderr_path "#{shared_path}/log/unicorn.stderr.log"
stdout_path "#{shared_path}/log/unicorn.stdout.log"

You should place the number of worker processes based on the memory you have available in your server and also the number of processor cores it has, leaving too little will make your not use the whole server resources and adding too many of it will surely make everyone wait too much on IO access. So, profile, test and reach your best configuration for your current environment.

Unlike using mongrel or thin, you don’t have to define many processes do be run, Unicorn loads a single process that’s going to take care of loading and serving the requests to all of your workers, so you only have to manage a single process instead of a process for each worker, something that’s going to simplify our work a lot.

Now we also need to configure Unicorn to run as a Daemon (a long running process that’s started with Linux itself and not by a user), so it will start itself when the machine boots. To do this we have to create a script that will start/stop/restart our Unicorn server and place it inside the “/etc/init.d” directory, fire up the text editor at the server:

nano /etc/init.d/unicorn

And here are the contents of the file.

As you should have noticed by the file, this unicorn configuration is specific for a single application, if you have many applications, you’ll have to create one file like this one for each of them, which should be something quite easy, as you would only change the values of the PID and DAEMON_OPTS variables.

With the file in place we have to make it executable and tell Linux to load it on startup:

chmod +x /etc/init.d/unicorn
update-rc.d unicorn defaults

Now Unicorn is ready to boot with your server and run your application.

With the Unicorn stuff ready, commit your changes to the project and push them to your version control system, we’re now on to start the deployment phase.

Starting the deployment

Now that almost all the configuration is in place, let’s setup the folders in the remote server. At your local machine in the project root folder, run:

cap  deploy:setup

This will create the whole folder structure for our deployment. After running this, the remote server folder structure should be something like this:

root@rails-tutorial:~# ls -l /home/deployer/shop/
total 8
drwxrwxr-x 2 deployer staff 4096 Aug 25 02:08 releases
drwxrwxr-x 5 deployer staff 4096 Aug 25 02:08 shared

Now we have the environment ready for action and we’re close to our first deployment. You should now create your database in production and configure the “database.yml” accordingly, as the first deployment is going to run the migrations and needs a pre-existing database to be run.

Once you have the database ready we need to add some code to the deploy.rb file to tell Capistrano how our server is started/stopped. You should add the following lines at the end of your deploy.rb file:

namespace :deploy do
  task :start do
    sudo "/etc/init.d/unicorn start"
  end
  task :stop do
    sudo "/etc/init.d/unicorn stop"
  end
  task :restart do
    sudo "/etc/init.d/unicorn reload"
  end
end

These are the calls Capistrano is going to make to handle our Unicorn daemon. Look that at the :restart call we do not send a restart to Unicorn but a “reload”.

Why this?

When you setup a Unicorn server there are two parts involved, the “master” and the “worker” processes. The master process is the one that handles the workers, starting, stopping and restarting them.

The master process usually doesn’t need to be “restarted”, only the workers have to so when you update your application you don’t want to restart Unicorn, all you want is to restart the workers and this “reload” method does exactly that, it sends a “HUP” signal to the Unicorn master process and it will then start to “stop” the old workers and “start” new ones for their places. The biggest advantage for this behavior is that you will have zero downtime as the old workers will be available until the new workers have arrived to serve requests.

The only reason to send a real “restart” to Unicorn is when you’re updating the Unicorn gem itself. Before going on to the next step you should check if the machine that contains the source code for your application allows access from your current server, do an “ssh” connection to if to be sure that everything works (if you’re using GitHub “ssh git@github.com” and read the section just after this one before running the next command).

If you don’t access the source code server at least once your deployment could get stuck at the “add host to known_hosts”:

The authenticity of host 'github.com (207.97.227.239)' can't be established. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.

Enough talking, let’s go on with the deployment, at your local machine, run:

cap deploy:cold

The deploy:cold task does the “first” deployment to the machine, setting up new directories, creating log files and everything it needs to start for the first time. After that you should check your server and you should finally see your application running:

root@rails-tutorial:~# ps aux | grep unicorn

root 21567 0.0 0.9 46792 9480 ? S 03:48 0:00 unicorn_rails master -c /home/deployer/shop/current/unicorn.rb -E production -D

deployer 21568 15.9 5.8 126272 59788 ? S 03:48 0:02 unicorn_rails worker[0] -c /home/deployer/shop/current/unicorn.rb -E production -D

deployer 21569 16.1 5.8 126272 59792 ? S 03:48 0:02 unicorn_rails worker[1] -c /home/deployer/shop/current/unicorn.rb -E production -D

deployer 21570 16.0 5.8 126400 59788 ? S 03:48 0:02 unicorn_rails worker[2] -c /home/deployer/shop/current/unicorn.rb -E production -D

deployer 21571 16.1 5.8 126268 59776 ? S 03:48 0:02 unicorn_rails worker[3] -c /home/deployer/shop/current/unicorn.rb -E production -D

After your “cap deploy:cold” runs correctly, the next deployment should be done with “cap deploy” or “cap deploy:migrations” if there are any migrations to run (it’s even better if you always deploy using “cap deploy:migrations”, this way you never forget there were migrations to run).

Using GitHub as a source code hosting service

If you’re using GitHub with private repos to host your application source code there’s another step that needs to be taken, you will have to create a “deploy key” to your server to make it able to access the private GitHub repo. First, login to the server as “deployer” and run:

ssh-keygen –t dsa

Now, get the contents of the “~/.ssh/id_dsa.pub” (“cat ~/.ssh/id_dsa.pub”) and at your GitHub repo to go “Admin”:

Then go to “Deploy Keys”. You should add the contents of the “id_dsa.pub” file as a deploy key:

Now, your server is ready to pull code from GitHub.

Monitoring your services with Monit

Having Nginx and Unicorn running as daemons that will start if the server reboots is nice, but if something makes Unicorn or Nginx die, we shouldn’t need to restart them ourselves, and this is where Monit enters the show.

Monit is a process monitoring solution for Unix environments, where you can define various parameters of process health and let monit look at the processes and see if they’re behaving as expected. If they’re not doing fine, Monit will try it’s configured actions to make things right. In our example, we’re going to monitor Nginx and Unicorn, but if you had other processes like Resque or Solr you should also add them as resources watched by Monit.

Open the “/etc/monit/monitrc” at the server:

nano /etc/monit/monitrc

And add this line right at the end:

include /etc/monit/conf.d/*

Now we’re going to create the Monit configuration files, first the general configuration:

nano /etc/monit/conf.d/general.conf

And the file content:

set httpd port 6874
      allow localhost
      allow moniter:"165907"

set alert mauricio.linhares@gmail.com
set alert someone-else@gmail.com

set mailserver smtp.gmail.com port 587
        username "some-one@gmail.com" password "123456"
        using tlsv1
        with timeout 30 seconds
set daemon 60
set logfile /var/log/monit.log

This configuration tells monit to open a socket connection inside the server to allow commands to be run (if you don’t set this configuration, you won’t be able to call any commands on it). Then we define to which emails the alerts are going to be sent and finally the email configuration for sending the email alerts.

Now let’s tell monit to watch Nginx:

nano /etc/monit/conf.d/nginx.conf

And the contents:

check process unicorn
  with pidfile /var/run/nginx.pid
  start program = "/etc/init.d/nginx start"
  stop program = "/etc/init.d/nginx stop"
  group nginx

And finally let’s tell monit to watch the Unicorn server:

nano /etc/monit/conf.d/unicorn.conf

And the contents:

check process unicorn
  with pidfile /home/deployer/shop/shared/pids/unicorn.pid
  start program = "/etc/init.d/unicorn start"
  stop program = "/etc/init.d/unicorn stop"
  group unicorn

After doing that, open the “/etc/default/monit” and change this:

startup=0

To this:

startup=1

Now you can start monit (“/etc/init.d/monit start”) and it will start watching over your processes. You can add more tests, like CPU, memory, filesystem usage to your processed, check out the Monit website (http://mmonit.com/) to know more about the tool and the checks you can perform.

Even if you’re using Monit, you should use some kind of external monitoring service like Pingdom (http://pingdom.com/) to be sure that everything is running smoothly at your end.

Securing your web server

If your hosting provider doesn’t have an “emergency” console/terminal for the possible issue that the SSH daemon isn’t running or isn’t accepting connections, you should not perform this firewall setup, as you could become blocked from the machine once the firewall is up and starts blocking all ports. Go on at your own risk ☺

Now that the server is up and monitored, we’ll add the last touch to it by enabling a firewall and fine tuning the SSH access. During the server install we added the “ufw” package that is a simple interface to the Linux firewall, let’s enable it and open the only two ports we need right now, 22 and 80 – you could open 443 if you would use SSL.

As root, run:

ufw enable
ufw allow 22
ufw allow 80

Now check the firewall status:

ufw status

The output should be something like this:

root@rails-tutorial:/etc/monit/conf.d# ufw status
Status: active
To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere

With the firewall in place, we finally go to the last step on our server setup, disabling SSH access by password. Most of the attacks that happen on Linux boxes are brute force attacks to login as root. People use dictionaries and try as many combinations as possible to login to a machine, once you disable SSH access using password (and allow access only using private/public keys) this attack becomes virtually impossible.

The main caveat on this solution is that if you ever lose your private key or has an emergency to access the server, you won’t be able to do it unless your hosting company has an “emergency” console or you’re taking the private/public key pair with you wherever you go.

To disable the password login, open the “/etc/ssh/sshd_config” file and uncomment this line:

# PasswordAuthentication yes

And after uncommenting set the value to “no”:

PasswordAuthentication no

Now restart the SSH server daemon:

/etc/init.d/ssh restart

Closing thoughts

It was a long way, but now you have an Ubuntu server machine ready for the action and running your rails application. This tutorial is by no means complete or defines the “correct” way of doing things, it’s just the way I do my server setups and if you have other best practices, ideas, comments or anything, don’t be shy and hit the comments box bellow, I’d love to hear how people are handling and doing their deployments.

If you have questions or ideas about topics on deployments that I should cover, also post them on the comments as this is just the first post about deployments in a series of posts I’m planning to write.

Enjoy!

Asynchronous email deliveries using Resque and resque_action_mailer_backend

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 28 Jul 2010 00:00:00 UTC

If you have ever sent emails using ActionMailer during a user request you probably noticed that if the email sending fails or takes too long your user might not be really happy with the speed of your application. Making the email sending process an asynchronous one is usually the simplest solution for this problem and there are plenty of tools to do that like ar_mailer, that stores emails in your database and then uses a specific daemon to send them.

ar_mailer and other solutions are quite enough for most of your problems but what if you’re already using Resque to run your asynchronous jobs? Why bother using yet another daemon to do the email sending jobs with all the hassle of managing if you can use your current Resque setup to do it?

With this in mind I sat a bit and wrote this simple gem that lets you do exactly that, send your emails asynchronously using a Resque worker and without changing a single line of your email sending code. Following the same philosophy of ar_mailer, the resque_action_mailer_backend doesn’t require you to call fancy methods on your mailers, you just keep them as they are now and instead of changing them you just change the delivery method to :resque, like this (in a environment file, like “development.rb”):

config.action_mailer.delivery_method = :resque

The gem uses your ActionMailer::Base.smtp_settings configuration to deliver the emails, so you don’t have to change anything in there, just be sure that the SMTP credentials are correct so the worker can deliver emails without issues. The default queue name is “:headbanger_resque_mailer”, but you can change it to whatever you’d like to with this code:

Headbanger::ResqueMailer.queue = :your_email_queue

This single line of code changed will now make all your emails be queued to Resque and they’ll be sent as soon as a worker is available for it. Now you only have to tell your rails application to load the resque_action_mailer_backend gem (in your environment.rb):

config.gem "resque_action_mailer_backend"

You can check a some hints on using this setup at this sample project.

And you’re ready to start sending emails asynchronously using your resque workers!

Enjoy!

Produzindo um avião com métodos ágeis

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 09 Jun 2010 00:00:00 UTC

[caption id=”attachment_19” align=”alignleft” width=”225” caption=”Produto final produzido no curso de CSM”][/caption]Este último sábado foi também o último dia da minha disciplina de Introdução aos Métodos Ágeis e Scrum da especialização em Metodologia para Engenharia de Software na Faculdade iDez. Como projetinho de última aula resolvi roubar a idéia que eu havia visto no curso de Certified Scrum Master que o Michel Goldenberg ministrou em Recife no início do ano.

A idéia básica é que os alunos fariam um produto (que no caso do curso foi um folder com de propaganda) e se utilizariam do Scrum pra produção do mesmo. O trabalho foi simples e exigiu muito mais organização e foco por parte da equipe pra entregar o folder pronto do que trabalho duro propriamente dito.

Muitas idéias diferentes surgiam enquanto estávamos planejando e começando o trabalho, se tivéssemos seguido todas, o produto não teria sido entregue, mas o tempo finito e a comunicação constante da equipe nos mantiveram na linha pra finalizar a tarefa. No fim foi uma ótima forma de cristalizar o conhecimento básico de Scrum que foi passado durante os dois dias de curso e eu sabia que eu não podia terminar essa disciplina de Scrum sem fazer o mesmo com o pessoal.

Será a Embraer?

Coloquei na cabeça que faria a dinâmica, mas faltava a idéia de “o que” produzir. Não queria repetir a idéia dos folders de propaganda, então o que seria um trabalho manual interessante pra colocar o pessoal pra produzir? Aviões, claro!

Na brincadeira, cada equipe ficaria responsável por produzir um avião com o material disponível dentro do tempo estimado de uma hora e meia. Essa hora e meia consistiria tanto no tempo pra planejar a construção do avião (Release e Sprint Planning Meetings) como efetuar a construção em si. Dividi então o tempo nos timeboxes:

20 minutos iniciais - Release Planning - cada equipe tem que fazer o planejamento do release, que vai conter dois sprints e também fazer o planejamento dos dois sprints;
Dois sprints de 30 minutos - aqui é onde eles finalmente executam o trabalho, esses 30 minutos são divididos então em 3 blocos de 10 minutos, com um Daily Scrum entre cada um deles.
Ao fim de cada sprint a equipe deve fazer uma breve apresentação sobre o que eles conseguiram produzir e depois avaliar o que deu certo ou não durante o trabalho (a retrospectiva do sprint);
Terminados os sprints as equipes demonstram o seu produto final pra todos na sala;

A idéia é que o evento todo seja simples, nós fazemos priorização e planning poker durante os 20 minutos iniciais, mas não consideramos velocidade ou fazemos o burndown do projeto. O foco maior é nos timeboxes e no objetivo final que é entregar o avião pronto. Se a equipe percebe que tudo o que eles planejaram não vai ser entregue até o fim do tempo, eles precisam se reorganizar e diminuir o escopo pra poder ter um produto completo ao fim das duas sprints.

[caption id=”attachment_22” align=”alignright” width=”300” caption=”Equipe Boeing planejando o trabalho”][/caption]Montando as equipes, ambiente e materiais

O meu caso eram duas equipes com 8 pessoas cada, das 8 pessoas, 1 era o Scrum Master, 1 o Product Owner e os outros eram a equipe de desenvolvimento em si. O Product Owner não trabalha efetivamente, mas ele pode direcionar a equipe e aceitar ou não o trabalho sendo feito.

O Scrum Master também não deve fazer trabalho diretamente, seu serviço é apenas resolver impedimentos na equipe, como falta de material. Eu também andei atrapalhando as equipes conversando besteira, removendo material, ocupando os desenvolvedores com outros trabalhos, pra manter os Scrum Masters ocupados.

O material necessário pra cada equipe é:

2 garrafas de refrigerante de 2 litros (preferenciamente garrafas que não tenham detalhes no formato, como as de Guaraná Antartica);

2 tesouras;

1 estilete;

2 folhas de cartolina guache/grossa;

2 folhas de cartolina fina;

1 conjunto de canetas hidrocor;

Baralhos pra planning poker;

Post-its e index cards/folhas de resumo;

1 pacote de palitos de churrasco, preferencialmente de bambu;

1 rolo de fita adesiva transparente;

1 régua grande (ao menos 50cm);

[caption id=”attachment_24” align=”alignleft” width=”300” caption=”Os dois produtos finalizados após os 2 sprints”][/caption]Lições da primeira vez

Não levei estiletes, foi bem difícil pro pessoal cortar as garrafas, também levei apenas uma tesoura pra cada grupo e isso foi um problema, já que muita coisa precisava ser cortada, ter duas tesouras teria agilizado muita coisa. Levei apenas um rolo de fita adesiva e ela rodou demais entre as duas equipes, ter um rolo pra cada equipe também teria complicado menos as coisas.

No fim, acho que o saldo final da brincadeira foi bem positivo, os dois grupos produziram os seus aviões e cada um completamente diferente do outro (e eles estavam lado a lado dentro de sala). As equipes mudaram o escopo e a idéia do que iam produzir baseados no caminho que a “construção” estava tomando (a equipe Boeing só resolveu realmente construir um avião com as duas garrafas no segundo sprint, quando a equipe Teco-Teco já estava com o esqueleto do avião quase todo pronto).

Dinâmicas, jogos e atividades interativas no geral deveriam ser mais comuns no ensino em tecnologia, ainda está pra surgir uma forma de ensinar que seja mais completa do que envolver os alunos na atividade de verdade. Clique aqui pra ver a galeria de fotos completa com todas as imagens da aula.

If you’re cleaning up your user’s input in your views you’re doing it wrong

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 08 Nov 2009 00:00:00 UTC

Have you ever found yourself using the “h” view helper all around your views in your applications? Have you ever thought that cleaning up user input in views is a tedious, error prone and cumbersome job?

You’re not alone.

Think with me, the user provides information once to your application, that information could be badly formatted, could be an XSS attack, but you store it as the user provided in your database. When you’re about to show that information, something that could happen once or a hundred of times (you probably would like to have thousands of page views, wouldn’t you?) you finally clean it up, instead of cleaning it up just once when the user provided it.

Insane, heh?

What about stopping with this waste of CPU cycles and cleaning the data once and for all?

Don’t worry, you don’t have to do anything, it’s already done and sorted out for you with this dead simple plugin. The params_sanitizer plugin uses Rails own sanitizers to clean the user input when it’s first provided on form POSTs and PUTs (what? do you change your application/database state with GET calls? OMFG!). You can protect all calls to all controllers, protect all actions in a single controller and even protect specific actions in a single controller, it’s your call!

First step is to install the plugin:

ruby script/plugin install git://github.com/mauricio/params_sanitizer.git

With the plugin installed, you can start cleaning up user input in your application once and forgetting about it forever, here are the examples: Stripping tags from all params in all actions (remember, only POST or PUT actions will really be changed ):

class ApplicationController < ActionController::Base
  strip_tags_from_params #strip_tags users rails full_sanitizer
end

Stripping tags from all params for all actions in a single controller:

class NewsStoriesController < ApplicationController
  strip_tags_from_params
end

Stripping tags from all params for specific actions in a single controller:

class CommentsController < ApplicationController
  strip_tags_from_params :only => [ :create, :update ]
end

If instead of stripping all tags, you’d just like to use the simple sanitizer (it removes bad tags like but would leave others intact, uses rails white_list_sanitizer):

class ApplicationController < ActionController::Base
  sanitize_params
end

class NewsStoriesController < ApplicationController
  sanitize_params
end

class CommentsController < ApplicationController
  sanitize_params :only => [ :create, :update ]
end

This plugin depends only on Rails default sanitizers, so you don’t need to install anything else to have it working.

Now, as the data is cleanly stored in your database, you don’t have to waste CPU cycles cleaning up data in your view layer (and you can even say that you’re more adherent to the MVC, as cleaning up user input was never one of it’s jobs).

Building your own ActiveRecord validation macros with validates_each

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 28 Sep 2009 00:00:00 UTC

A common task when writing your own Rails applications using ActiveRecord is creating your own validations for your models. While it’s perfectly correct to add the validation directly into the model you’re going to need it, sometimes you’d like to reuse the same validation logic in other models and we’re not really going to do a cut-and-paste here are we?

The simplest solution when you’re validating fields (and not the whole model) is to use the validates_each method, as it has some nice features seen in other validations that might interest you as the :if, :unless, :allow_blank and :allow_nil options.

Our custom example validation is to validate that one or more fields are different from one specific field. Imagine that you’re building an invoices application, having the seller to also be the buyer isn’t really what you’re looking for, so that’s why we’re building this validation. Let’s take a look at the validation code:

ActiveRecord::Base.class_eval do

  def self.validates_different( *args )

    options = args.extract_options!
    raise "You must define a :field option to compare to" if options[:field].blank?

    validates_each(*(args << options)) do |record, attribute, value|
      if record.send( options[:field] ) == value
        record.errors.add(
          attribute,
          record.errors.generate_message(
            attribute
            'different',
            :field => record.class.human_attribute_name( options[:field].to_s ) ) )
      end
      true
    end

  end

end

We have inserted a static method inside the ActiveRecord::Base class to be our validation macro, it takes a list of parameters and an options hash at the end, here’s a sample of how it would be used:

class Invoice < ActiveRecord::Base
    validates_different :seller_id, :field => :buyer_id, :allow_blank => true
end

The validation looks just like any other ActiveRecord validation and even uses options well known in them like :allow_blank, keeping the principle of least surprise at bay. It’s also important to notice the use of I18N on the validation message, the “’activerecord.errors.messages” namespace is the ActiveRecord error messages namespace and that’s where you should add your custom validation messages, do not place the messages directly inside your validation or model code. Here’s how the YAML file would look like:

en:
  activerecord:
    errors:
      messages:
        different: “must be different than ”

And there you go, you have built your own validation macro for your ActiveRecord models and even used the I18N helpers to keep the messages away from your model code.

Setting a far future expires header for your Rails app static assets in your Nginx server

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 21 Sep 2009 00:00:00 UTC

Setting a far future expires header for your static assets is one of the first front end performance improvements you must do in your web applications. This will tell your user’s browser to keep the static assets cached and they won’t make unnecessary HTTP requests just to see that the version they currently have is already the latest and this will surely improve your website’s perceived performance (yes, there’s a REAL and a user perceived performance for every application).

Nginx isn’t as well known and used as Apache’s HTTPD but one of the most common deployment environment for Rails applications (specially after Phusion Passenger for Nginx) we clearly needed a simple solution for this expires header. Looking a little bit over the internet you can usually find a code sample just like this one:

location ~* \.(ico|css|js|gif|jpe?g|png)\?[0-9]+$ {
    expires max;
    break;
}

Looking at the regex you can surely figure out that you’re going to match any asset that ends with any of those extensions and has a timestamp (“?123456”) at the end. The timestamps are usually added by Rails asset helpers and they’ll clean up your user’s browser cache when the file is changed (if there’s no timestamp the cache will be cleared only when it’s full or you change the static asset name).

All good, right? Not so easy, cowboy. If you try to use this location directive you’ll never match anything. You might then end up with the following solution:

location ~* \.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ {
    expires max;
    break;
}

Now the timestamp is optional and you’ll finally start matching your static assets. But why wasn’t the previous directive working? Simple, the location directive will match only on the request filename, not on it’s query string. So, as there was never a timestamp, the first location would never set a far future expires header on anything. You should remove the timestamp match completely as it’s completely useless anyway.

But even this simple location directive has it’s own issues and they can easily bite you. We’re matching based on file extensions, so anything that ends with “.js” would be a match and would be served by Nginx, exactly what we want, right?

Maybe not. Imagine that you’re making a controller that responds to an HTML and an RJS view. The HTML view can be called by “/actions.html” and the RJS view can be called by “/actions.js”. When the browser calls “/actions.js”, Nginx looks at the extensions and thinks “hey, it’s a javascript file, I’ll handle it” and it sends a 404 error directly to your client, as this JS file doesn’t exist inside your public folder.

But you could use the HTTP request headers to tell your rails application that it’s a JS request, couldn’t you? Yep, but sometimes you can’t and you’d probably end up digging weird bugs and 404’s in production when you never saw them in your development machine. Not something really nice, I’d tell you.

And how do you get this sorted out? Instead of basing your rules in file extensions, you’ll do it using the folders where they live. In common Rails application images, javascript and stylesheet files have their own default folders and you would use a rule that matched on those folder names, just like the following:

 location ~ ^/(images|javascripts|stylesheets|system)/  {
    expires max;
    break;
  }

The system folder is usually where you keep user generated data, like uploaded images, documents and things alike. With this simple location directive you’ll get all benefits from static asset caching on Nginx without worrying about crazy errors in your application. If you don’t plan to write a controller that answers to “/stylesheets/my_custom_theme.css”, obviously.

Accessing the current request object on your mailer templates to generate links

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 26 Aug 2009 00:00:00 UTC

A common issue with mailer templates is that as they’re not being called from a controller you can’t get your hands on the request object and access properties like host_with_port. While you’re usually calling the mailers inside controllers and you could possibly hand the request as a parameter to it, it isn’t really nice to do this every time you need to send an email.

<%= link_to 'Home', "#{current_request.protocol}#{current_request.host_with_port}/home" %>

So, if you’re looking for a quick and easy solution to this issue, the current_request plugin is your friend, you can install it by calling:

ruby script/plugin install git://github.com/mauricio/current_request.git

The plugin works by setting the current request in a thread local variable that will be available until the end of the request, which means that you can use it safely in your templates, two new methods are added to all views, current_request, that returns, obviously, the current request being answered and current_host that will build the current host with port and protocol for you. Examples:

Or you can just use a shorthand to the current host:

< %= link_to 'Home', "#{current_host}/home" %>

You can also use it wherever you want to access the current request (and not only on templates) by calling:

CurrentRequest::Holder.current_request

Why I am not using Masochism for my master-slave setups and why monkey-patching isn’t the only solution

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 24 Aug 2009 00:00:00 UTC

I got a message this morning from Gregg at Ruby5 asking why I wrote the master_slave_adapter plugin instead of using Technoweenie’s Masochism and I think the answer to this question deserves a little blog post (and the blog really needs some new content :P).

When building the Talkies project we had to setup a master-slave environment using MySQL at the production servers. To get these things up and running I configured the replication on MySQL and set out to find a solution on Rails/ActiveRecord to handle this special need, all SELECT* statements should be sent to the slave db while all other commands should be sent to the master. The only solution available at the time was Masochism (at least it was the only one I could find).

With Rails 2.1, everything looked like we would live happily ever after, but Rails 2.2 brought a lot of changes and many of them on ActiveRecord, the main one being connection pooling and we upgraded. The production server, that wasn’t really live yet, broke badly, the new connection pooling code made the application crazy and the slave was receiving UPDATE* and INSERT* calls ( this was the code at the moment ).

With this new issue showing up I set out to find a solution, the first thing was to hack the plugin itself (as github had no “issues” thing at the moment). Trying it out I couldn’t really find a simple fix and wasn’t really happy with the way the plugin worked, looked a lot like a hack when a hack wasn’t really needed, so I started to write my own solution.

The first requirement was that it should perform no black magic at all, we were burned more than once during the project by plugins that were too clever and relied heavily on monkey-patching, so my solution had to be really straightforward and do as little clever things as possible.

But hey, active_record needs a database adapter, so why not just build a fake database adapter that forwarded the work to a master or slave connection depending on the method called? This way I would never need to hack ActiveRecord, as the thing would just be a common database adapter, like all the others and the plugin would survive to Rails upgrades with little or no changes. And that’s exactly what I did, an ActiveRecord database adapter who’s job is to route method calls to a real master or slave connection.

Why was it an improvement?

By relying on the ActiveRecord database adapter contract I had no need to monkey-patch Rails itself, it would just work, even if Rails or ActiveRecord got upgraded, the only thing that would make me change the plugin was if the database adapter contract got changed and this isn’t really something that changes a lot.

And if there’s one thing that’s burning a lot of people using plugins and Rails itself is clever code and too much monkey-patching. When you’re building a solution that’s going to be “inserted” inside someone else’s codebase that you don’t even know how it’s going to look like, you better try to avoid changing too many things or breaking well known contracts, you might end up with bugs that are hard to discover and kill. And they’ll surely make you waste a lot of your time.

Monkey-patching and class-redefinition are some of the coolest features of Ruby as a language, but they should be used with care and are better avoided if possible.

Setting up your Ruby on Rails application in an Ubuntu Jaunty Jackalope (9.04) server with Nginx, MySQL, Ruby Enterprise Edition and Phusion Passenger

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 09 Jun 2009 00:00:00 UTC

There are many ways to deploy and run Ruby applications with the Ruby on Rails framework but it’s unlikely that you’re going to find a simpler and faster solution than using Ruby Enterprise Edition (REE from now on) with Nginx and Phusion Passenger. Nginx is a fast, scalable and lightweight HTTP server, that is able to serve a lot of content without using up all your memory and Passenger is a module that can be tied into Apache or Nginx to handle your Ruby (and RoR) applications automatically.

When using Passenger you don’t need to worry about managing a pack of Mongrels or use a proxy HTTP server, Passenger lives inside your web server and just takes care of everything for you. Here you’ll learn how to use Passenger in conjunction with Nginx to deploy your applications in the wild.

This tutorial assumes that you’re building a brand new Ubuntu server with none or little custom packages installed. Does this mean you can’t use this with an already customized server? No, but it’s easier if you can follow it step by step to avoid problems, as this has already been tried and tested to be sure that it works. We’re using MySQL here because it’s what I’m using right now but can easily change the apt-get calls to use whatever database you’re using yourself.

Setting up users

If you’re really starting up from a brand new install with no users created beyond the default ones you might want to create a user for yourself so that you don’t need to be logged in as a “root” forever. To create a new user in a Linux box the command is “useradd”:

useradd -m -g staff -s /bin/bash mauricio

This will create a user called “mauricio” with a “/home/mauricio” home directory (as defined by the “-m” param), with “staff” as it’s default group and using the “/bin/bash” shell. After creating a user for yourself you might also want to create a user for the application you’re deploying or a “deployment” user. This is the user that’s going to be used to deploy the application and run all application related processes. Just use the same command above changing the username to your deploy user, this can be the name of the application you’re deploying or just “deploy” (keep all your users belonging to the same “staff” group to avoid file permission issues when you edit or create files).

After doing this you can also make all users that belong to the staff group be able to use the “sudo” command. To do this just open the “/etc/sudoers” file with a text editor (I usually use “nano”) and add this line:

%staff ALL=(ALL) ALL

Setting up your ssh keys

If you’re running in a Linux/Unix box and haven’t generated your SSH keys, it’s time to do it. If you have never heard of them, SSH keys can get you to login into servers where you have a user account without asking you for the password, which is really cool if you have to handle a lot of servers at the same time (and if you don’t want to type passwords every time you do a “git pull

git push”. To generate them do this as your local user in your local machine:

ssh-keygen -t dsa

This command will create a folder called “.ssh” in your home directory (as in “/home/your_user/.ssh” with a bunch of files. It will ask you for a password to protect these files, the password isn’t required but it’s nice to be cautious here as if you don’t set a password anyone with physical access to your machine (or can log in as you) could log in into all machines to where your SSH keys were copied to.

Now that the keys are already generated, you can copy them to the servers you usually log into, to do this, first log in to the server using your account and at your user’s home folder create a .ssh folder:

mkdir ~/.ssh

Log off and, from your local pc, copy the ~/.ssh/id_dsa.pub file to the remote machine using scp:

scp ~/.ssh/id_dsa.pub your_remote_user@host:.ssh/authorized_keys2

This will copy your public key to the remote server and you’ll be able to log into that server from your current local machine and local user to the user you copied the key to in your remote server. You can obviously copy this to as many servers and user accounts as you like and none of them will ask you for a password again.

Getting Ubuntu up-to-date

First thing we need to do is to be sure that our server is up-to-date with the currently installed software:

sudo apt-get update
sudo apt-get upgrade

Then we need to install some basic libraries and MySQL:

sudo apt-get install build-essential mysql-server libmysqlclient15-dev libmagickcore-dev imagemagick  libpcre3 libfcgi-dev libfcgi0ldbl libxml2-dev libxslt1-dev -y

This is going to install the MySQL server, the ImageMagick library to handle image processing and the XML and XSLT libraries needed for some common gems like Nokogiri.

Installing Ruby

We’re not going to use the default Ruby interpreter that comes with Ubuntu but Phusions’s Ruby Enterprise Edition. REE is a reliable and memory friendly fork of the main Ruby interpreter from the Phusion guys. Go to the REE download page and grab the “.deb” files for your architecture (look for the “Ubuntu Linux” tab). Install the “.deb” with:

sudo dpkg -i path-to-the-deb-file

This will get REE installed to “/opt/ruby-enterprise” but the binaries will not be available at your PATH, we’ll need to add the “bin” dir to our PATH variable manually. Open up your “/etc/environment” file with your preferred command line text editor (mine is “nano”):

sudo nano /etc/environment

And add the “/opt/ruby-enterprise/bin” dir to your PATH variable like this:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/ruby-enterprise/bin"

This will get the scripts at the “bin” folder available to your user but not when you use “sudo” calls (Ubuntu just overrides the PATH when you call “sudo” for security reasons) so we’ll need to symlink some of the files to “/usr/bin” to be sure that they’re visible when you’re sudoing:

ln -s /opt/ruby-enterprise/bin/ruby /usr/bin/ruby ln -s /opt/ruby-enterprise/bin/gem /usr/bin/gem ln -s /opt/ruby-enterprise/bin/ri /usr/bin/ri ln -s /opt/ruby-enterprise/bin/rdoc /usr/bin/rdoc ln -s /opt/ruby-enterprise/bin/irb /usr/bin/irb

Now let’s install some gems to be sure that everything is ok:

sudo gem install rails mysql nokogiri rmagick mislav-will_paginate --no-ri --no-rdoc

The “–no-ri –no-rdoc” option is to avoid creating docs that we’re not really going to use and that will take a long time to be generated (also, if you’re into a VPS and don’t have a lot of memory those commands are surely going to throw out of memory errors). If you got no errors here, we’re good to go and install Nginx and Passenger.

Installing Nginx and Passenger

Installing Nginx with Passenger and Ruby EE is as easy as calling this command:

sudo /opt/ruby-enterprise/bin/passenger-install-nginx-module --auto --auto-download

Those “–auto” options are there to tell the installer that we’re saying yes to all defaults and we want it to download a brand new Nginx copy and build it with the Passenger module. The installer is going to ask you where to install Nginx with a default of “/opt/nginx”, just hit enter to get it installed at the default path.

As you can see from the messages printed, Passenger has already generated a sample configuration file with the basic config needed to run the application, here’s an example of how it would look like.

It’s VERY important to set the Nginx user to be the same user that’s going to deploy and create the application files as this will avoid permission issues that are one of the most common problems you’re going to have. With Nginx configured to load your application, start it to be sure that everything is OK again:

sudo /opt/nginx/sbin/nginx

Open up your browser pointing to the server where Nginx is running and you should see your application running correctly. If it isn’t, check the application logs and also Nginx error logs at “/opt/nginx/logs/error.log”. You can kill Nginx with a simple:

sudo pkill nginx

Getting Nginx to run as a Daemon

Now that Nginx is running correctly and serving your application you need to set it up to run as a daemon. To do this we need to create a script that’s going to handle the Nginx daemon and install it using the update-rc.d utility. You can get the script here.

You should save this script at “/etc/init.d/nginx” (sudo to do it), mark it as executable and install it as a daemon:

sudo chmod +x /etc/init.d/nginx
sudo update-rc.d nginx defaults

Now when the machine reboots Nginx will be started automatically. As a last touch, start the Nginx daemon and your server is ready to roll:

sudo /etc/init.d/nginx start

Quick Tip – Using to_s as a label and simplified link_to calls to your ActiveRecord models

mauricio.linhares@gmail.com (Maurício Linhares) — Mon, 08 Jun 2009 00:00:00 UTC

One of the things you’ll find in every rails application is links like this one:

<%= link_to user.login, user %>

Or maybe like this one:

<%= link_to user.login, user_path( user ) %>

Or maybe something ugly like this one:

<%= link_to user.to_label, user_path( user ) %>

How about throwing all of them and just doing it like this:

<%= link_to user %>

Cool, isn’t it?

Now “how can I do that” you ask, it’s dead simple. First, remember that every object responds to a method called “to_s” and this “to_s” method is defined as “a method that returns a string representation of your object” in most programming languages, including Ruby.

A string representation of your object is something human readable that would tell someone else what this object represents. “to_s” isn’t meant to be a debug like method, we already have “inspect” to do that, so why not put it to work and simplify our links?

At every ActiveRecord model in your application you’ll define a to_s method that returns one (or maybe more, if needed) attributes of your object as a string (if they’re not strings, turn them into strings and return). Let’s see how your user would look like:

class User < ActiveRecord::Base
  validates_presence_of :login
  validates_uniqueness_of :login
  def to_s
    self.login
  end
end

I decided that the “login” method is the one that best represents the User object and it’s also the one I want to use then someone else seers users on the website. They won’t see their real names by default, but their logins.

Why is it better to do this with the “to_s” method instead of adding a “to_label” method to all objects? ‘Cos many helpers will already call the to_s method by default on your object (as we’ll see with the link_to helper), so you just get full compatibility for free. Check out our new link_to implementation:

module ApplicationHelper
  #sample link_to override that will generate urls for active_record objects by default

  #if the first parameter is an active_record object and you just want a link to it,

  # you can call it just like this:

  # <%= link_to user %>

  # the helper will take care of generating the correct url

  def link_to( *args )
    options = args.extract_options!
    if args.size == 1 && args.first.is_a?( ActiveRecord::Base )
      super( *([ args.first, args.first ] + [ options ]) )
    else
      super( *( args + [ options ] ) )
    end
  end
end

If there’s only one object (besides the options hash) and it is an ActiveRecord::Base instance, just use the object itself as the url parameter (the real link_to helper will call polymorphic_url on it automatically) and also use the object as the link_to label (the first parameter), this will call the to_s method on our user and the link will use it as the label.

What do you get with this?

A seamless and clear way of defining labels for your models (the “to_s” method is part of the core of the language anyway) and also a simpler way of generating links for your objects. Common methods for object labels are very important because you can never be sure if the property you’re using today as a label will be used forever. Using the “to_s” method as the default “label method” will allow you to change the label property at any time with almost no change to other parts of the code.

Building a I18N aware form builder for your Rails applications

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 06 Jun 2009 00:00:00 UTC

Form builders are one of the coolest features when building Rails applications, they streamline the task of writing complex forms and you can usually write your own form builder to maintain their consistency in your application. One of the most common customized form builders is the one that users the field name to show a label:

< % form_for @user, :builder => SimpleFormBuilder do |f| %>
    < %= f.text_field :date_of_birth %>
< % end %>

We would probably use the “date_of_birth” symbol, turn it into a String and then call “humanize” on it ( you can see a custom form builder example that does exactly this here ):

:date_of_birth.to_s.humanize
=> “Date of birth”

That’s awesome if you’re building a website for a language that uses only ASCII characters, but if you’re building a form in Portuguese you’re doomed. Imagine that my “usuario” (user) has a “profissão” (profession) and I try to use it using a common form builder:

< % form_for @usuario, :builder => SimpleFormBuilder do |f| %>
    < %= f.text_field :profissao %>
< % end %>

What do I get? “profissao”, no tilde. And if I try to create a method called “profissão” at my object it’s just going to break.

So, customized form builders for Rails using funny characters are impossible? Never! With the new I18N support this trouble as been completely removed, let’s see how we can write a customized form builder that uses the attribute names to generate labels and will respect funny Portuguese, Russian or characters from any other language.

Here’s the builder code:

class SimpleFormBuilder < ActionView::Helpers::FormBuilder

  attr_accessor :object_class

  helpers = field_helpers +
            %w{date_select datetime_select time_select} +
            %w{collection_select select country_select time_zone_select} -
            %w{hidden_field label fields_for submit select} # Don't decorate these

  helpers.each do |name|
    class_eval %Q!
    def #{name}(field, *args)
      options = args.extract_options\!
      args << options
      return super if options.delete(:disable_builder)
      @template.content_tag(:p, field_label(field, options) << '' < < super)
    end
    !
  end

  def select(field, choices, options = {}, html_options = {})
    return super if options.delete(:disable_builder) || html_options.delete(:disable_builder)
    @template.content_tag(:p, [field_label(field, options), '', super].join("\n"))
  end

  def submit(value = nil, options = {})
    if self.object && value.nil?
      value = self.object.new_record? ? I18n.t( 'txt.shared.create' ) : I18n.t( 'txt.shared.update' )
    end
    @template.content_tag( :p, super( value, options ) )
  end

  def field_label( field, options )
    self.label( field, options.delete( :label ) || self.object_class.human_attribute_name( field.to_s ), :class => options[:label_class])
  end

  def initialize(object_name, object, template, options, proc)
    super
    self.object_class = self.object.nil? ? self.object_name.to_s.camelize.constantize : self.object.class
  end

end

We’ve created our own form builder that inherits from the ActionView::Helpers::FormBuilder and it redefines all field helpers using a class_eval call (don’t know what class_eval does? Learn here).

Our version isn’t really that different from the usual solution, it looks for a :disable_builder option, if there’s one and it’s true, the builder will just call the original method, without the custom decoration. If there’s no :disable_builder option the builder will set out to do its work. Also, we need the object class to find out the correct attribute names, so your form builder also holds the class of the object that’s being used in the form.

If there’s a :label option available, it’s the one that’s going to be used, if there’s no :label option the builder will access the class of the object that’s being used in the form and call the “human_attribute_name” with the field name as a parameter on it. By default, “human_attribute_name” will just call “humanize” on your field name (same as our code above) but, when we’re using the Rails I18N support things change a bit.

The first thing that changes is that we can use the I18N support to define labels for those fields (and also for the class names). Let’s take a look at the “config/locale/en.yml” to check out how it looks like:

en:
  activerecord:
    models:
      user:
        one: User
        other: Users
    attributes:
      user:
        name: Name
        date_of_birth: Date of birth
        login: Login
        password: Password
        password_confirmation: Confirm Password
        profession: Profession

Under the “activerecord” namespace we have the “models” and “attributes” namespaces. As you might have guessed, the “models” namespace is used to internationalize your model names and the “attributes” to do the same to your object’s attribute names. While we’re using English as the language things aren’t really that interesting, so we’re going to add Portuguese support:

pt-BR:
  activerecord:
    models:
      user:
        one: Usuário
        other: Usuários
    attributes:
      user:
        name: Nome
        date_of_birth: Data de Nascimento
        login: Login
        password: Senha
        password_confirmation: Confirmação da Senha
        profession: Profissão

And now your form builder will use the Portuguese field names on its labels whenever the current locale is set to “pt-BR” (this isn’t the full file, you can check it out at the project repo). The real catch here is to use the “human_attribute_name” instead of just “humanizing“ the field name.

When human_attribute_name is called it will first try to get the attribute name from your I18N files using the current locale and you don’t really need to be writing a Multilanguage application to use the I18N support, whenever you’re using a language that isn’t pure ASCII only you can use the I18N support and have nice default labels for your form fields. Translating your models using the default “models” and “attributes” namespaces will also internationalize Rails default error messages, as they’re going to use the names of the current locale.

You can see this form builder in action at the sample_social_network project.

with_scope and named_scopes ignoring stacked :order clauses

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 04 Jun 2009 00:00:00 UTC

If you’ve been using with_scope and named_scopes a lot with ActiveRecord you have probably noticed that the :order clauses defined at the scopes are lost and only the first :order clause is used. If you defined an :order clause you’d like to have it merged with the other ones already provided. Here’s a simple example:

class User
  named_scope :by_first_name, :order => "#{quoted_table_name}.first_name ASC"
  named_scope :by_last_name, :order => "#{quoted_table_name}.last_name ASC"
end

Our user has two named scopes defined and both of them define an :order clause, if we try to run a finder like this:

User.by_first_name.by_last_name.all

This is the generated query:

SELECT * FROM `users` ORDER BY `users`.first_name ASC

As you’ve noticed, only the first :order clause was used, the last one was lost. Our ideal SQL query would have to look like this, with both :order clauses being used:

SELECT * FROM `users` ORDER BY `users`.last_name ASC , `users`.first_name ASC

That’s why we’re going to hack the with_scope method a litle bit to reach our goal. This issue was already reported to the Rails issue tracker but there’s no fix yet so our only hope is to monkeypatch Rails to behave as we expect it to, so here’s a really simple fix for the problem:

ActiveRecord::Base.class_eval do

  class << self

    def merge_orders( *orders )
      orders.map! do |o|
        if o.blank?
          nil
        else
          o.strip!
          o
        end
      end
      orders.compact!
      orders.join( ' , ' )
    end

    def with_scope_with_hack(method_scoping = {}, action = :merge, &block)
      method_scoping = method_scoping.method_scoping if method_scoping.respond_to?(:method_scoping)

      # Dup first and second level of hash (method and params).

      method_scoping = method_scoping.inject({}) do |hash, (method, params)|
        hash[method] = (params == true) ? params : params.dup
        hash
      end

      method_scoping.assert_valid_keys([ :find, :create ])

      if f = method_scoping[:find]
        f.assert_valid_keys(VALID_FIND_OPTIONS)
        set_readonly_option! f
      end

      # Merge scopings

      if [:merge, :reverse_merge].include?(action) && current_scoped_methods
        method_scoping = current_scoped_methods.inject(method_scoping) do |hash, (method, params)|
          case hash[method]
          when Hash
            if method == :find
              (hash[method].keys + params.keys).uniq.each do |key|
                merge = hash[method][key] && params[key] # merge if both scopes have the same key

                if key == :conditions && merge
                  if params[key].is_a?(Hash) && hash[method][key].is_a?(Hash)
                    hash[method][key] = merge_conditions(hash[method][key].deep_merge(params[key]))
                  else
                    hash[method][key] = merge_conditions(params[key], hash[method][key])
                  end
                elsif key == :include && merge
                  hash[method][key] = merge_includes(hash[method][key], params[key]).uniq
                elsif key == :joins && merge
                  hash[method][key] = merge_joins(params[key], hash[method][key])
                elsif key == :order && merge
                  hash[method][key] = merge_orders(params[key], hash[method][key])
                else
                  hash[method][key] = hash[method][key] || params[key]
                end
              end
            else
              if action == :reverse_merge
                hash[method] = hash[method].merge(params)
              else
                hash[method] = params.merge(hash[method])
              end
            end
          else
            hash[method] = params
          end
          hash
        end
      end

      self.scoped_methods << method_scoping
      begin
        yield
      ensure
        self.scoped_methods.pop
      end
    end

    alias_method_chain :with_scope, :hack

  end

end

You can place this code at an initializer (maybe called with_scope_fix.rb) or at your lib folder and require it in your initializers. And now all your :order clauses defined by named_scope or with_scope calls will be correctly merged and will not be lost in your code.

Understanding class_eval, module_eval and instance_eval

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 04 Jun 2009 00:00:00 UTC

Most of Ruby’s fame is due to it’s dynamic capabilities. In Ruby you can define and redefine methods at runtime, create classes from nowhere and objects from pure dust. Most of these dynamical features are done using one of those methods at the title, class_eval, module_eval and instance_eval, they’re usually the ones responsible for the show and now we’re going to learn a little bit about how they work and how we could use them in our objects.

class_eval and module_eval

These two methods are responsible to granting your access to a class or module definition, as if you were writing their code by yourself. When you do something like this:

Dog.class_eval do
    def bark
        puts “Huf! Huf! Huf!”
    end
end

It’s almost the same as doing this:

class Dog
    def bark
        puts “Huf! Huf! Huf!”
    end
end

What’s the difference?

With the class_eval you’re adding a method to a pre-existing class. If a class called Dog is not defined before our class_eval runs you’d see an “NameError: uninitialized constant Dog”. A class_eval call opens up an existing class for you, it won’t create or open a class that doesn’t exist yet.

And you don’t need to always write real code inside your class_eval calls, you can also send a string object containing the code you want to have ‘evaled inside your class. Let’s see how we could define a macro just like the attr_accessor using class_eval’ed strings:

Object.class_eval do

  class << self

    def attribute_accessor( *attribute_names )

      attribute_names.each do |attribute_name|
        class_eval %Q?
          def #{attribute_name}
              @#{attribute_name}
          end

          def #{attribute_name}=( new_value )
              @#{attribute_name} = new_value
          end
        ?
      end

    end

  end

end

class Dog
  attribute_accessor :name
end

dog = Dog.new
dog.name = 'Fido'

other_dog = Dog.new
other_dog.name = 'Dido'

puts dog.name
puts other_dog.name

As you can see, we used both kinds of class_eval. First we opened up the Object class and added a new class method called attribute_accessor with direct code, but then, at the attribute_acessor I had no way to figure out the method name when I was writing the code, so, instead of just writing the code directly inside the class_eval call I’ve created a string object containing the code that I wanted to have ‘evaled by the class_eval method. The string is then turned into something like this:

def name
    @name
end

def name=( new_value )
    @name = new_value
end

And this is the parameter passed on to the class_eval call. Wrapping up, you can use class_eval to open classes (and modules) and add real code on it as you also can just pass a string containing valid Ruby code and it’s going to be ‘evaled as it was at the class definition body.

The module_eval method is just an alias to class_eval so you can use them both for classes and modules.

The instance_eval method works just like class_eval but it will add the behavior you’re trying to define to the object instance where it was called.

But hey, isn’t this exactly what we were doing with class_eval?

No, it isn’t. With class_eval we opened up a class definition and added code to it’s body. Any kind of code valid inside a class definition was also valid in there. When we’re using instance_eval the rules change a bit ‘cos we’re not opening up a class, but a single object instance.

How’s that? Let’s see an example:

class Dog
  attribute_accessor :name
end

dog = Dog.new
dog.name = 'Fido'

dog.instance_eval do
    #here I am defining a bark method only for this “dog” instance and not for the Dog class

  def bark
   puts 'Huf! Huf! Huf!'
  end

end

other_dog = Dog.new
other_dog.name = 'Dido'

puts dog.name
puts other_dog.name

dog.bark
other_dog.bark #this line will raise a NoMethodError as there’s no “bark” method

                      #at this other_dog object

Not really that interesting, is it? We can also use instance_eval to define methods in Class objects (which in turn will be class methods at that Class object instances) and we can do just that to our attribute_accessor:

Object.instance_eval do

  def attribute_accessor( *attribute_names )

    attribute_names.each do |attribute_name|
      class_eval %Q?
          def #{attribute_name}
              @#{attribute_name}
          end

          def #{attribute_name}=( new_value )
              @#{attribute_name} = new_value
          end
      ?
    end

  end

end

By using instance_eval instead of class_eval we don’t need the “class « self” as the method is defined directly at the Object class and will then be available as a class method for Object instances and Object subclasses instances.

As you might have noticed, these methods are also related to the difference between including and extending modules in Ruby.

SQL functions in WHERE clauses are evil

mauricio.linhares@gmail.com (Maurício Linhares) — Tue, 09 Dec 2008 00:00:00 UTC

Once we get up an running with the basic SQL syntax, doing inserts, updates, deletes and simple selects we start to learn about the SQL functions, the default ones like LOWER, COUNT, AVG and then the functions that are specific for the database you’re using. We learn them and start to feel that your fingers are itching to try them, to use them in the real world. Why would you learn them if you can’t use them anyway?

Well, I can’t tell you that you should never use them, but listen to my advice, do not ever use these SQL functions in WHERE clauses to filter data. They’re evil and they’ll try to kill your database and prevent you from working by having to discover crazy performance bottlenecks and slow queries that don’t look slow at all. At least until you run an “EXPLAIN” on them.

Let’s start with a simple example, we want to know, from our USERS table, which ones where born today so we can send them a beautiful and unpersonal e-mail to remember that they are getting older today. In your USERS table there is a column called “date_of_birth” which is a DATE and obviously it stores the date that the user said he was born at, so we know the day, month and year, we already have all information that we need, now it’s time to write the SQL code to find them.

In our first attempt, we write the following simple SQL:

SELECT * FROM users u where DAYOFMONTH(u.date_of_birth) = 12 AND MONTH(u.date_of_birth) = 1

Pretty simple and does the job perfectly, running it in your test database returns the correct users and it’s definitely fast. Now we get to try it in our production database so you can figure out which users are going to receive the e-mail. We’re running a big website, with a gazillion of users, so there should be some of them that are getting older today, we type in the query at your production database console and…

Wait.

Waiting.

Still waiting and looking at that blank console screen.

Well, you should find yourself something else to read, it’s going to take a while. After a long time waiting you get a list with 10 users with birthdays today. WTF? Why so long just to find 10 users?

A light shines in our heads and we remember that there is no index at the “date_of_birth” column, we never thought about using it in queries so we, as good database guys, did not create an index when it wasn’t needed. But now it is and you just type in the command to create the index for the “date_of_birth” column.

After waiting a little bit to have the index created, we type again our beautiful query and we wait again. This time it seems that’s it’s taking even longer to finish. This is clearly wrong, we have created an index at that database field and queries against that field should use that index. Now we have to bring the most important tool of anyone that has ever used a database, the “explain query” feature, which explains how a query is going to be executed by the database. At your database console, we type (this is for MySQL):

explain select * from users u where DAYOFMONTH(u.date_of_birth) = 12 and MONTH(u.date_of_birth) = 1\G

And here’s our output:

*************************** 1. row ***************************

	id: 1
	select_type: SIMPLE
	table: users
	type: ALL
	possible_keys: NULL
	key: NULL
	key_len: NULL
	ref: NULL
	rows: 4558
	Extra: Using where
1 row in set (0.00 sec)

The most important lines are “possible_keys” and “key”, both of them are NULL, which means that our beautiful query isn’t using any indexes. But another information is even more alarming, the database is looking at 4558 rows to retrieve my results (and this is exactly the count of rows available at the users table). The database is scanning the WHOLE users table to fetch some 8 rows. Can you feel it’s pain?

We’ve created the index and we’re trying to filter just on that column, why is it not being used?

Because we’re using SQL functions, that’s the reason. The DAYOFMONTH function is a transforming function, it takes an argument and generates a value based on that argument and our query is performing it’s filtering based on this generated value. But here’s the problem, the database optimizer doesn’t know the generated value, it has no index for it nor a way to infer which value could be generated because it doesn’t know what this SQL function does. The optimizer can’t perform any optimizations at all.

When faced with such a complex query (from it’s point of view) the optimizer has no option but let the query run against the whole database, selecting every row, applying the function to every column and then finally filtering the result. Every time you use a SQL function that is not native to your database, like DAYOFMONTH, LEFT, RIGHT or MONTH, you might be leading yourself to such a bad query and future bottleneck. When you’re at your development database with a bunch of records, it might not yield any perceivable performance problems but once you reach the production environment with hundreds of rows, your problems will start to rise.

You should avoid filtering based on calculated or transformed data in you queries, as your database optimizer will not be able to give you the best “query plan”. If you’re faced with such a problem, you should create a separate column at your table and generate the value beforehand. In our case, we would need to create two new columns, “day_of_birth” and “month_of_birth”, create an index for both of them and every time a row has it’s “date_of_birth” updated, it should automatically update the “day_of_birth” and “month_of_birth” columns.

From now on, learn the mantra, SQL functions in where clauses are evil :)

Handling database indexes for Rails polymorphic associations

Porque é importante saber como o protocolo HTTP funciona

mauricio.linhares@gmail.com (Maurício Linhares) — Fri, 07 Nov 2008 00:00:00 UTC

É interessante perceber que há tantas pessoas trabalhando escrevendo aplicações web que não entendem o básico sobre a internet e o protocolo HTTP. Você pode encontrar aplicações que demonstram comportamentos bizarros em qualquer lugar, as pessoas simplesmente esquecem de ler as especificações ou dormiram durante as aulas sobre o protocolo HTTP na universidade.

Um dos casos mais infelizes dessa falta de conhecimento é a “febre do POST”. Todos so formulários na aplicação usam o método POST para se comunicar com o servidor, não importa o que ele está fazendo ou se existem ou não “efeitos colaterais” envolvidos no caso. Simplesmente funciona dessa forma e as pessoas aparentemente não tem nenhum motivo pra não fazer dessa forma. Se você pergunta a alguém, eles vão provavelmente soltar a pérola, “ah, me disseram que o GET tem o limite no tamanho dos parâmetros que você pode mandar, então é melhor prevenir do que remediar”.

Mas o que é que há de tão mal nisso?

Se você der uma olhada no RFC do HTTP, vai descrobrir que o método GET é descrito como “seguro” (‘safe’). Seguro, no contexto do HTTP, significa que você deveria ser capaz de fazer vários GETs para uma aplicação web e isso não deveria causar efeitos colaterais, como apagar um cliente ou coisas do gênero, ele não deve causar alterações no recurso que está sendo requisitado, porque toda a idéia do método GET é de que você deveria simplesmente receber uma cópia (bem no estilo copiar-colar) do recurso que está naquela URL específica. Você não está fazendo nada de especial com o recurso, você deveria ser capaz de recebê-lo em qualqure lugar e a qualquer momento que desejasse.

Mas se você olhar a descrição do método POST vai descobrir que ele é um método “inseguro” (‘unsafe’). Se você manda um POST pra uma URL, você pode estar definitivamente alterando alguma coisa no servidor e causando efeitos colaterais malignos, como libertar a Skynet e os Exterminadores que vão trazer o armagedom para a Terra. Ou você pode simplesmente estar criando um novo recurso naquela aplicação, como esse post de blog que você está lendo.

A diferença óbvia é que POSTs podem (e normalmente devem) alterar o estado de alguma coisa do lado do servidor, enquanto um GET nunca deveria ser capaz de fazer uma coisa dessas. Comparando com bancos de dados que usam SQL, GETs seriam como “selects” e POSTs como “inserts”. Você já viu um “insert” que retornava uma tabela de resultados ou um “select” que inseria dados no banco? Nem eu :)

Mas é claro que tudo ainda pode ficar pior. Imagine que você é o dono daquele site maligno que usa apenas POSTs em todos os seus formulários e um desses é um formulário de pesquisa. Usuários vão usá-lo para buscar produtos e adicioná-los as seus carrinhos de compra. Um usuário está interessado em comprar o novo disco do AC/DC, mas ele não tem certeza do nome, então ele simplesmente digita AC/DC e aperta “enter” no teclado.

Voila!

Lá, no topo da página, está “Black Ice”, o novo album deles (Já comprou o seu? Deveria!). Ele clica no link e enquanto ele está vendo a página, se lembra que não comprou o album antes desse, “Stiff Upper Lip”. “Vou apertar no botão voltar e procurar por ele na lista de discos do AC/DC”, pensa o incauto usuário e quando ele clica no botão, o navegador mostra uma mensagem interessante:

“O navegador precisa enviar dados para o servidor para executar esta ação. Você tem certeza que deseja fazer isso?”

O usuário olha aterrorizado para a mensagem. “O que foi que eu fiz? Será que eles vão me cobrar por isso? Vão me mandar o novo disco da Britney Spears porque eu estou tentando apertar no botão de voltar?”.

Como protocolo HTTP define, POSTs não são métodos “seguros” e as ferramentas (normalmente os navegadores) devem avisar o usuário de que alguma coisa ruim pode acontecer se eles tentarem dar um POST por acidente em uma página e é exatamente isso que acontece se você tenta clicar no botão voltar após um POST. Nesse exemplo, o usuário não estaria fazendo nada de errado, mas imagine se em vez de estar voltando pra uma página de busca, ele poderia estar voltando para o formulário de “adicionar cliente” e um “voltar” poderia muito bem fazer com que ele re-criasse o cliente no banco de dados, o que não é exatamente a idéia.

Pior, se você está usando POST em um formulário de busca, os usuários nunca vão poder usar o botão voltar (os mestres da usabilidade dizem que ele é a coisa mais usada nos navegadores) e eles também não vão poder colocar aquela página de resultado nos seus favoritos! Você consegue imaginar algo pior do que isso? Você está evitando que as pessoas possam expressar todo o seu amor pelo seu site postando links pra eles no del.icio.us!

A idéia é bem simples, se você não está alterando nada no servidor, você deveria sempre usar GETs, seja lá o que for. Eles não quebram o botão voltar ou atualizar, deixam os usuários colocar as páginas requisitadas nos favoritos e não vão fazer com que os navegadores mostrem mensagens assustadoras para os usuários. Se você estiver alterando o estado de alguma coisa no servidor, você deve usar POST (e os outros métodos do HTTP que são definidos como “inseguros”, como PUT e DELETE), requisições que usam GET NUNCA deveriam alterar coisas no servidor (sabe aquele link que você fez que apaga um registro no banco de dados? Foi uma péssima idéia!).

E antes que eu me esqueça, depois de um POST com sucesso você deve sempre REDIRECIONAR o usuário para a página de sucesso, nunca, por motivo nenhum do universo, mostre a página de sucesso como resultado do POST. Ao redirecionar o usuário após um POST com sucesso você evita que ele reenvie os dados usando um “atualizar” ou clicando no botão voltar do navegador.

PS: Tradução de “Why learning HTTP does matter”, prometo que eu vou traduzir mais posts de lá, é só ter tempo o suficiente, de qualquer forma, se você sabe inglês, pode ler lá antes :)

Why learning HTTP does matter

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 02 Nov 2008 00:00:00 UTC

It’s interesting to notice that there’s so many people working with web applications that don’t understand the basics of the Internet and the HTTP protocol. You might find applications that exhibit bizarre behaviors anywhere, people just forget to read the specs or sleep during the HTTP protocol classes at college.

One of the most harmful exhibitions of this lack of knowledge is the “POST fever”. Every form in the application performs a POST, no matter what it’s doing or the side effects involved in it, it just works that way and people just don’t have a reason not to go like that, usually, if you ask them, they’ll probably say “oh, someone told me that the GET method has size limit in it’s parameters size”.

But, what’s so bad about it?

If you take a look at the HTTP RFC, you will find that the GET method is described as a “safe” method. Safe, in the HTTP context, means that you should be able to perform GETs to a web application and this should have no side effects on it, it should not change the resource you are requesting, because the whole idea of the GET method is that you should just GET a copy of the resource at that specific URL, you’re not doing anything funny with it, you should just receive it anywhere and anytime you want to.

But if you look at the POST method description, it’s defined as an “unsafe” method. If you send a POST to a URL you might be definitely changing something and generating an evil side effect that might render the whole application useless and bring Skynet and the Terminators to lay Armageddon on Earth. Or you might just be creating a new resource, as a blog posting like this one.

The obvious difference is that POSTs can (and usually should) change the state of something at the server side, while a GET should never do something like that. If you’re keen to SQL databases, GETs are just like “select” commands and POSTs like “insert” commands. Have you ever seen an “insert” returning a result set or a “select” inserting data? Me neither :)

But bear with me, it GETs even worse. Imagine that you’re the owner of that evil website that I said that just uses POSTs in it’s forms and one of those forms is a search form. Users will use it to search for your products and add them to their shopping carts. A user wants to buy the new AC/DC records but he’s not sure about it’s name, so he just types AC/DC and hits enter.

Voila!

There, at the top of the list, is “Black Ice”, their new record (Have you already bought yours?). He clicks on the link and while he’s viewing the CD page he remembers that he hasn’t bought the “Stiff Upper Lip” album. “Let me hit the back button and look for it too”, thinks the poor user and when he hit the button, the browser shows an interesting message:

“The browser will need to send data to the server to perform this action. Are you sure you want to do this?”

The user looks terrified to the message. “What have I done? Will they bill me for this? Are they going to send me the new Britney Spears album ‘cos I’m trying to hit the back button?”

As the HTTP protocol mandates, POSTs are not safe and the tools (usually, our browsers) should tell the user that something bad might happen if they try to POST by accident and that’s exactly what happens if you try to hit the back button after a POST. In this example, the user wouldn’t be doing anything wrong, but instead of coming back to a search page, he could be at a “add client” page and a “back” would make him re-create the last client he sent to the database, which isn’t really interesting.

Worse, if you’re using POST in a search form, they aren’t going to be able to use the back button (and the usability gurus say that it’s the most used feature in browsers) and they aren’t going to be able to bookmark the search results! Can you imagine something worse than that? You are keeping people from expressing their love for you website by posting it in their del.icio.us favorites!

Now, the reasoning is simple, if you’re not changing anything at the server side, you should always perform GETs. They don’t break the back button, they let the users bookmark their pages and they aren’t going to make the browser show the user any funny messages. If you’re changing state at the server side you should definitely use POST (and the other HTTP methods that are designed to change state, like PUT and DELETE), GET requests should NEVER change any state at the server side.

And before I forget, after every successful POST you should REDIRECT the user to a new page and not just render the page for him in response for the POST. Redirecting the user to the “response” page keeps the user from hitting the “back” button and re-entering the data they have already sent during the last POST.

Including and extending modules in Ruby

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 27 Sep 2008 00:00:00 UTC

One of the coolest features in Ruby is the existence of modules and the possibility of including their implementation in any object. This simple behavior is the source of things like the Enumerable module, that gives you a bunch of methods to work with a collection of objects and just expects that the class that included it to define an “each” method. You write a class, define an “each” method, include Enumerable and your’re done, all Enumerable methods are available for you.

Another example is the Comparable module, when you include the Comparable module in your class, you must define the operator (the UFO operator), the Comparable module will give you the implementation of the following operators/methods:

<, <=, >, >=, ==, between?

This is usually why we call them mixins, because they are “mixing in” their behaviors (their methods/messages) into our objects. The idea of mixins serve a purpose similar to that of the multiple inheritance, that is to inherit an implementation from “something” without having to be a direct child of that “something”, in multiple inheritance you would be able to inherit from as many classes as you wanted to. In Ruby we don’t have multiple inheritance, but we can include as many modules as we want, so they give us the same feature, without all the hassle that multiple inheritance usually brings to a language.

The method resolution mechanism is pretty simple, first, if a method in a module that is being included is already defined in the class that is including it, the method of the class has precedence (which means that the method on the module will be ignored). If two modules define a method with the same name, the method on the last module included will be the one available at the class that has included both modules (remember that in Ruby there is no method overloading mechanism). Here’s an example of how it works:

module SimpleModule

  def a_method
    puts 'a_method at module'
  end

  def another_method( parameter )
    puts "Calling another method with parameter -&gt; #{parameter}"
  end

end

module AnotherModule

  def another_method
    puts 'Calling another method without a parameter'
  end

end

class SimpleClass

  include SimpleModule
  include AnotherModule

  def a_method( param )
    puts "a_method at class -&gt; #{param}"
  end

end

instance = SimpleClass.new

#calling the method defined on the class

instance.a_method 'parameter'

#calling method on the AnotherModule

instance.another_method

#this line will throw a 'wrong number of arguments' error

instance.a_method

An ugly example for an ugly practice, don’t rely on these things when you’re writing your own modules, strive to create unique modules that aren’t going to have method names clashing when they are included in other classes. If you have to rely on these rules to write and use your modules, maybe there is a problem in your code or in what you’re trying to do.

Extending methods

As the title of this post says, you can include and also extend modules, but what does it means to extend a module?

When you extend a module, you are adding the methods of that specific module into the object instance you call “extend”. So, the methods of that module will only be available at that specific instance (and not all objects of that class), other objects of the same class will not have the methods of the module available. With this, you can add specific behaviors to just one object of your system, without changing the other ones. Here’s an example:

module InstanceMethods

  def simple_method
    puts "im a method that belongs to an instance"
  end

end

class SimpleObject
end

object = SimpleObject.new
object.extend InstanceMethods
object.simple_method

another_object = SimpleObject.new

#the following line will throw an error, as this instance doesn't extends the module

another_object.simple_method

This might look like a weird feature, how many times have you wanted to introduce a method into a single object?

Not that many, probably, unless this instance is in fact an instance of the Class class (that contains the class methods of your object), and this is where extending modules get interesting and this is how many of the Rails plugins are written, let’s see how we can use this to write our own acts_as_votable plugin.

Rails, extending and including modules

First thing to do is create your Rails project:

rails --database=mysql include_extend_modules

With the project created, we have to create our plugin (enter in your Rails project folder):

script/generate plugin acts_as_votable

This will create a folder called acts_as_votable at the vendor/plugins folder and the plugin skeleton code. The first thing to do is to create our Vote model. It’s a dead simple model, with a polymorphic relationship with a “votable” and a boolean column called “up”, representing if this vote is “up” or “down”. The vote.rb file should live at the vendor/plugins/acts_as_votable/lib folder. Here’s the model code:

#vendor/plugins/acts_as_votable/lib/vote.rb


class Vote < ActiveRecord::Base
  belongs_to :votable, :polymorphic => true
  validates_presence_of :votable
end

Now we have to create a migration to create the votes table at the database:

script/generate migration create_votes

And there is the migration code:

class CreateVotes < ActiveRecord::Migration
  def self.up

    create_table :votes do |t|
      t.integer :votable_id, :null => false
      t.string :votable_type, :limit => 15, :null => false
      t.boolean :up, :default => false, :null => false
      t.timestamps
    end

    add_index :votes, [ :votable_id, :votable_type ]

  end

  def self.down

    drop_table :votes

  end
end

After creating the Vote model and it’s migration, we’ll head to that acts_as_votable.rb file in our plugin folder, it’s where the code that ties the Vote model with the application will live, here’s the code that will be in there:

#vendor/plugins/acts_as_votable/lib/acts_as_votable.rb


module ActsAsVotable

  module ClassMethods

    def acts_as_votable
      has_many :votes, :as => :votable, :dependent => :delete_all
      include InstanceMethods
    end

  end

  module InstanceMethods

    def cast_vote( vote )
      Vote.create( :votable => self, :up => vote == :up )
    end

  end

end

We have created a module called ActsAsVotable to serve as our namespace and in it we have two modules ClassMethods and InstanceMethods. The ClassMethods module defines the methods that we want to introduce at the ActiveRecord::Base class, so that we can just call “acts_as_votable” in any model that inherits from ActiveRecord::Base (just like any other ActiveRecord plugin) and the InstanceMethods module contains the methods that we want an instance that is “votable” to have.

So, if I say that a NewsArticle class is votable, its instances will have the cast_vote method, as the module InstanceMethods was included when they called acts_as_votable. But before creating the NewsArticle model, we have to do some changes in our init.rb file for the acts_as_votable plugin, here’s how it should look like:

#vendor/plugins/acts_as_votable/init.rb


require 'vote'
require 'acts_as_votable'

ActiveRecord::Base.extend ActsAsVotable::ClassMethods

This is where we make the acts_as_votable method available to all classes that inherit from ActiveRecord::Base and this is one of the most common uses of “extending” modules you will see.

Now that we have the code hooked to ActiveRecord, let’s create a simple model to try some tests, create our NewsArticle model:

script/generate model NewsArticle title:string article:text

Now, at the news_article.rb file:

#app/models/news_article.rb


class NewsArticle < ActiveRecord::Base

  acts_as_votable
  validates_presence_of :title, :article

end

We just call the acts_as_votable class method, that is available as we “exetended” the ActsAsVotable::ClassMethods module into the ActiveRecord::Base class, the superclass of our NewsArticle class. Here’s an example of you could do with our models:

article = NewsArticle.create(:title => 'sample', :article => 'sample')

#calling the cast_vote method from the ActsAsVotable::InstanceMethods module

article.cast_vote :up
article.cast_vote :down

#acessing the votes association defined when you called the acts_as_votable method

article.votes

And that’s it, you now know how and when to include or extend modules and even how to build a simple acts_as plugin for your models.

Handling database indexes for Rails polymorphic associations

mauricio.linhares@gmail.com (Maurício Linhares) — Sat, 27 Sep 2008 00:00:00 UTC

One thing that is usually overlooked when defining tables and their associations in a Rails application are the indexes. Usually, this comes from the idea that “my ORM tool does the job” and in fact it might be true sometimes. One of the most successful ORM tools in the Java land, Hibernate, generates a database with indexes for all foreign keys that you have, so Java programmers that use it don’t really worry about these issues (at least not until their database is slowing down to death).

ActiveRecord migrations, on the other side, don’t really worry about these things ( unless you’re using the cool Foreign Key Migrations plugin ), you must define the indexes that you need by yourself. Usually this is done by a simple call like this:

add_index :comments, :user_id

This will create an index for the column :user_id at the :comments table. For simple associations this is straightforward, but ActiveRecord offers goodies that are not so common in other tools and one of them is the “polymorphic associations”. With polymorphic associations you can define an association without defining the kind of the object you will be associated with, you just say that it’s a polymorphic association and you’re done. The code would look like this:

class Comment
    belongs_to :commentable, :polymorphic => true
    belongs_to :user
end

To make this work, at the database level you would need two columns at the :comments table, one called :commentable_id, that will hold the id of the object that owns the comment, and another called :commentable_type, that will hold the full class name of the object that owns the comment. So, if you’re commenting in a Post object with an ID of 1, the commentable_id would be 1 and the commentable_type would be “Post”. At the Post model the association would look like this:

class Post
    has_many :comments, :as =>  :commentable
    has_one :user
end

When you’re using polymorphic associations, your queries for the object will usually contain the commentable_id and the commentable_type in the where clause, as you would be looking for comments for the commentable_id of 1 and commentable_type of “Post”, so it makes sense to create indexes for these columns. As you’ve already saw, you could do this with the following code:

add_index :comments, :commentable_id
add_index :comments, :commentable_type

And now you have two indexes, one for each column, your database searches should fly with this, shouldn’t they?

Well, they will not. You’re defining two different indexes, one for each column, but you almost never search for them in separate, you’re always searching for the :commentable_id and also for the :commentable_type, so you should create an index for both columns and not for each one of them, the call should be something like this:

add_index :comments, [ :commentable_type, :commentable_id]

This is going to generate an index with both columns and your queries for your polymorphic models will now really be faster than before.

Obviously, you can also create indexes for the :commentable_type and :commentable_id columns if you search for them in separate, but having a lot of indexes in your table slows down update calls and might also create big tables in your filesystem. So, when defining polymorphic associations, remember to create an index for both columns and not just one for each of them. Also, if you know that the column will always have a value, make it not null, as searching on indexes of nullable columns in some databases (like MySQL) is slower then searching on not-null column indexes.

And before you go, when ActiveRecord creates a string column at the database level, you can define a :limit option that defines the size of the VARCHAR column at the database. If you don’t give a limit, it’s going to be set as a VARCHAR(255) and I really believe you will not have a model class with a name that has 255 characters, so, instead of creating a column with an unreasonable size (that is going to slow down queries and generate bigger indexes), give it a limit that’s real. Our final table definition would look like this one:

create_table :comments do |t|
  t.integer :user_id, :null => false #indexing nullable columns is slower, try to make all columns that are going to be in indexes not-null
  t.integer :commentable_id, :null => false
  t.string :commetable_type, :null => false, :limit => 20 #could be even less
  t.text :comment
end

add_index :comments, :user_id
add_index :comments, [:commentable_id, :commentable_type]

This post was originally published at the CodeVader weblog.

Unit tests don’t guarantee that your system works

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 05 Mar 2008 00:00:00 UTC

Last week we had an interesting message at the RSpec users list, the most interesting part of it is the following:

“I also had to go into specs on a project I'm not working on, and found an unholy hive of database-accessing specs. It's disheartening. Basically, it's cargo cult development practices - using the "bestpractice" without actually understanding it.”

You might have read this before, “/specs|tests/ that access the database are evil”, but have you ever asked yourself why?

Behavior Driven Development is the next step after Test Driven Development and it borrows many best practices found in the later. The two principles that interest us most in this conversation is test-first development and unit testing.

The idea behind test-first development is that before writing your code, you should write a test stating what you want you “future” code to do. By writing the test before the code you get to work on the public interface provided by your object, the test is the first client of your code, so, if your public interface is cumbersome or difficult to use, this test will be able to catch a bad idea before it’s materialized in your code.

And where is unit testing in all this? You should be doing test-first using unit tests, as unit tests will guarantee that the code you wrote for that single unit (a method, probably) works alone. If you have more objects that need to be used to test this specific behavior, you should use mock objects (fake objects) in their places, so you won’t be testing them in your unit test. Remember, unit tests should only test a unit of code, no more than that. We should do it this way so we don’t get distracted with the other objects implementation, we focus in testing our target, not it’s dependencies.

When we’re writing specs for our objects they should usually work as unit tests, they should only assert the behaviors of a single unit of code, everything else should be done using mocks and stubs. But I said usually.

As I said before, unit tests and your common specs, should only assert the behaviors of a unit of code without considering their relationships with the other objects on the system, but this only guarantees that they work as units. This will never guarantee that they will really work when in real contact with the other objects in the system, unit testing don’t guarantee that your system works, they surely help you to reach this goal, but they aren’t enough.

And what it has to do with that message, anyway?

That spec that access the database is just like an integration test, it asserts that the code being tested works fine when integrated with the database. So, the integration tests are the ones that really show you that your code works as a system, not only as a group of lonely objects.

I’m not saying that you should leave the unit tests behind, because they have a big importance to help you design your code and be sure that it works as a unit, but you shouldn’t rely only in them to test your system, a good suite of integration tests will give you the trust that everything works fine in conjunction.

And sometimes you can’t unit test a functionality, it’s all about integration. Let’s take the “validates_uniqueness_of” validation in ActiveRecord as an example, if you’re writing a spec for your ActiveRecord model, you should add one ‘it’ statement showing that this is needed (you’re specifying how your model behaves, remember?), so here’s how it could look:

it 'Should not be valid if there is another one with the same name' do
           @common_name = 'testuser'
           @user = User.create( :name => @common_name )
           @another_user = User.new( :name => @common_name )
           @another_user.should have(1).error_on(:name)
end

How could you perform this spec without touching the database?

First, you could look ad the “validates_uniqueness_of” source code, figure out how it works and stub it to return what you want, but this is bad because if the framework code changes your specs would break. The other way would be changing the database adapter to a mocked one and send exactly the result you wanted, but this is basically overkill. So why don’t you just leave the “purism” behind, test it in your database and be happy that your code works fine?

One important thing to notice is that integration tests are also slower to run, so you wouldn’t like to wait for the full suit run before performing a commit, usually you would run the unit and integration tests that are most likely to break if you did something wrong, the ones related to what you’re doing now and just be done with it.

So, if you’re in a project that has database accessing specs or specs that are using many real objects (and not mocks), don’t feel bad, but be sure that who wrote it knows that he is doing and that everything that can be unit tested is being unit tested. Integration tests should be written after your functionality is implemented and tested with unit tests, they are not interchangeable, nor you will replace one with the other.

And be sure to never commit your code before running your tests :)

PS: Originally published at the CodeVader blog

Como tratar os seus testes?

mauricio.linhares@gmail.com (Maurício Linhares) — Sun, 27 Jan 2008 00:00:00 UTC

Já percebeu o quanto se fala em qualidade e testes de software nos últimos tempos? Testar software hoje não é mais apenas sentar o seu usuário (vulgo “cliente”) na frente de uma tela rodando o seu sistema pra que ele possa validar se as funcionalidades estão implementadas a contento. A algum o uso de testes unitários, de integração e funcionais se tornaram um fato corriqueiro para equipes que trabalham com software e prezam pela mínima qualidade (até porque qualidade não é apenas falta de bugs) e facilidade de se fazer alterações sem quebrar o sistema todo.

No caminhar da escrita de testes, os desenvolvedores aprenderam muitas coisas, sendo uma delas que o teste que você escreve já é o primeiro cliente do seu código. Quando você está escrevendo um teste pra um método ou função do seu programa, você está escrevendo também o primeiro código que vai fazer o uso dessa funcionalidade e vai poder perceber se é fácil ou difícil fazer uso desse método, se existem muitas dependências e se é necessário carregar estado demais para fazer com que o resultado final aconteça.

Você poderia, inclusive, escrever o teste antes do código que vai ser testado para guiar a sua implementação através do teste, assim você já escreve a sua funcionalidade partindo de um cliente real, que é o próprio teste, e não da sua imaginação, que acha que uma coisa deve ser de um jeito ou de outro. Com desenvolvimento orientado a testes você consegue caminhar mais rápido e escrever código que vai realmente ser utilizado, não apenas o que você “acha” que deveria escrever porque alguém pode vir algum dia a utilizar.

Com esse primeiro “cliente” do seu código você já pode perceber falhas na sua abstração e até mesmo na forma que você pensava que a funcionalidade deveria ser implementada. Você consegue descobrir falhas no seu design, no modo que você pensou que a funcionalidade deveria ser implementada e essa é uma característica que vai levar os nossos testes a um novo patamar, onde eles não vão mais ser testes.

Ora, mas se os meus testes não são testes, eles são o que?

Quando nós pensamos em um “teste”, o que vem a mente é alguma coisa que você faz a um objeto pra descobrir se ele “funciona” ou não. Normalmente você faz testes em um objeto que já está ou acredita-se que esteja “pronto”, você normalmente não testa objetos incompletos ou que já sejam conhecidamente falhos, porque você sabe que eles vão falhar de uma forma ou outra. E, principalmente, você não testa para melhorar, apenas para verificar.

No mundo do software você não precisa ter acesso ao que vai ser testado antes de escrever o teste, você pode já escrever o código do teste antes mesmo de ter a funcionalidade implementada, no seu teste você especificaria o que você espera passar como entrada e o que você espera receber na saída. O seu teste, neste momento, deixa de ser apenas um teste e se transforma em uma especificação, ele descreve como alguma coisa no sistema deve se comportar através de um conjunto finito de entradas e saídas (porque você não tem tempo infinito pra testar).

Ao deixar de encarar os testes como simples validadores de que o seu código funciona e assumir que eles definem as características do programa, você dá um passo além não apenas na questão de garantir a qualidade como também na própria documentação do sistema e gerência do projeto. Se você tem diversas “estórias” (ou “casos de uso”) pra serem implementados no sistema, eles se transformariam em diversas especificações do que o sistema faz e você teria, a cada especificação que passasse, a indicação de que alguma coisa está sendo produzida, você ganha de brinde, além das vantagens de ter software testado, um medidor de funcionalidades embutido.

A questão de deixar de tratar testes apenas como validadores do código chegou ao ponto de que até mesmo as ferramentas de testes unitários estão começando a ser suplantadas por ferramentas de desenvolvimento direcionado a comportamentos (ou Behaviour Driven Development – BDD).

É possível encontrar uma pequena definição sobre BDD na Wikipedia:

“Behavior Driven Development (or BDD) is a software development technique that questions the behavior of an application before and during the development process. Created in response to the failings the founders perceived in Test Driven Development, Behavior Driven Development addresses requirements and specification in a way that is more textual than its predecessor. By asking questions such as “What should this application do?” or “What should this part do?” developers can identify gaps in their understanding of the problem domain and talk to their peers or domain experts to find the answers. By focusing on the behavior of applications, developers try to create a common language that’s shared by all stakeholders: management, users, developers, project management and domain experts.”

A idéia é, além de tornar os testes, que agora são especificações, a documentação natural e a fonte de informações sobre o que o sistema faz e como faz. Cada especificação diz que “o sistema deveria fazer X” e se a especificação passa, é porque o sistema realmente faz “X”. Isso facilita a comunicação dentro da equipe, pois agora eles não se perguntam se o teste “Z” passou, mas se a funcionalidade “Z” foi implementada e também vai facilitar ainda mais para os especialistas de domínio que podem simplesmente chegar pra equipe e explicar o que o sistema deve fazer e as especificações garantindo que ele faz vão ser escritas.

Deixar de pensar em testes como validadores e passar a vê-los como definições facilita a comunicação entre todos os envolvidos no projeto, pois agora eles tendem a falar uma mesma língua (a “linguagem ubíqua” tanto pregada pelos defensores do Domain Driven Design), já que as funcionalidades do sistema vão ser expressas através de especificações definidas pelo especialista do domínio e a equipe de desenvolvimento.

Vejamos um simples exemplo de especificação utilizando o framework RSpec, que é uma ferramenta de BDD para Ruby:

it 'Should not allow a not logged in user send an upload' do
    post :prepare
    response.should redirect_to( :action => 'login' )
end

A nossa especificação diz que “Usuários não logados não devem poder fazer um upload”, então o código dela deve comprovar que é isso que acontece e é exatamente isso que ele faz, ele primeiro faz uma requisição usando o método HTTP “POST” (na primeira linha da especificação) e depois garante que a resposta seja um “redirect” para a “ação” login. Mesmo que você não entenda Ruby ou RSpec, é fácil entender o que esse código quer dizer, as afirmações são claras e é essa facilidade de entendimento que faz com que BDD seja realmente uma ótima maneira de se expressar as funcionalidades que um software deve ter.

Então, se você continua vendo testes apenas como um modo de validar o seu sistema, está na hora de começar a mudar o pensamento e aproveitar melhor esses testes tranformando-os nas especificações de o que o seu sistema faz. Você iria escrever os testes de qualquer maneira, não vai custar nem um pouco a mais, vai?

Referências:

De many-to-many para many-to-one com JPA

mauricio.linhares@gmail.com (Maurício Linhares) — Wed, 02 Jan 2008 00:00:00 UTC

Introdução Relacionamentos em bancos de dados dificilmente são tão simples quanto parecem, especialmente quando você começa a utilizar relacionamentos “N:N” (muitos-para-muitos), esse tipo de relacionamento, um extremamente comum no mundo real, normalmente é um pouco mais complicado quando é abstraído para um banco de dados relacional. Neste material você vai entender como transformar um relacionamento N:N em um N:1 utilizando JPA. Para continuar você deve ter conhecimentos da biblioteca de persistência do Java, a JPA, e do framwework Hibernate. Os exemplos mostrados são, na verdade, testes do JUnit, então ter conhecimento conhecimento básico de o que ele é e para que serve vão lhe ajudar a entender melhor os exemplos. Você pode fazer o download do projeto de exemplo aqui, o projeto é um projeto comum do Eclipse mas também é um projeto do Maven.

Relacionamento “muitos-para-muitos”

Quando estamos iniciando a análise dos nossos sistemas orientados a objetos e começando a montar o banco de dados que vai dar suporte e persistência a esse modelo, é comum que encontremos relacionamentos do tipo “muitos-pra-muitos” (many-to-many – N:N). Nesse tipo de relacionameto entre duas tabelas, nós criamos uma tabela de ligação, que contém sempre um par de colunas, onde cada uma aponta para uma chave primária de uma das tabelas que fazem parte do relacionamento, como no diagrama do nosso exemplo abaixo: Imagem 1 – Diagrama de exemplo com many-to-many O nosso relacionamento demonstra que um cliente pode ter vários produtos, algo que poderia ser utilizado em um sistema de inventário. Vejamos o código em Java utilizando JPA que exemplifica esse diagrama: Listagem 1 – Persistivel.java

package alinhavado;

import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.MappedSuperclass;

@MappedSuperclass
public class Persistivel {
	@Id
	@GeneratedValue(strategy=GenerationType.IDENTITY)
	private Long id;
	// métodos get/set
}

A nossa primeira classe não é exatamente uma classe do sistema, mas uma classe básica para evitar a repetição de código desnecessária, nela nós declaramos o código que define a propriedade “id” que é o identificador de cada linha das tabelas no banco de dados e também definimos o tipo de gerador para a coluna como sendo “identity”, que auto-incrementa automaticamente o valor da coluna. Como essa classe não representa uma tabela ou uma entidade no sistema mas nós queremos que as suas propriedades existam para as suas subclasses, nós definimos ela com a annotation “@MappedSuperclass”, assim, qualquer objeto que herdar dela vai automaticamente herdar os campos que foram definidos com as anotações do JPA, portanto nenhum dos objetos do nosso exemplo precisa definir uma propriedade “id”, ela já foi definida na superclasse. Usando @MappedSuperclass você evita repetição de código desnecessária par seus objetos e ainda garante que todos vão ter as mesmas propriedades e comportamentos, graças a herança. Listagem 2 – Cliente.java

package alinhavado;

import java.util.HashSet;
import java.util.Set;

import javax.persistence.CascadeType;
import javax.persistence.Entity;
import javax.persistence.JoinColumn;
import javax.persistence.JoinTable;
import javax.persistence.ManyToMany;
import javax.persistence.OneToMany;

@Entity
@Table(name=”clientes”)
public class Cliente extends Persistivel {
	private String nome;
	@ManyToMany(cascade=CascadeType.ALL)
	@JoinTable(name="clientes_produtos",
			   joinColumns=  @JoinColumn( name = "cliente_id"),
			   inverseJoinColumns= @JoinColumn(name = "produto_id") )
	private Set produtos = new HashSet();
	// métodos get/set
}

A classe Cliente herda de Persistivel (e consequentemente tem como @Id a propriedade “id” de Persistivel) além de ter uma coleção de produtos relacionados a ela na forma de um relacionamento N:N. Listagem 3 – Produto.java

package alinhavado;

import javax.persistence.Entity;
import javax.persistence.Table;

@Entity
@Table(name="produtos")
public class Produto extends Persistivel {
	private String nome;
	//métodos get/set
}

O nosso objeto Produto também herda de Persistivel e não tem nada além de uma única propriedade, ele não precisa estar relacionado diretamente nem a Cliente nem a Item. Quando estiver fazendo o mapeamento de relacionamentos, evite utilizar relacionamentos bidirecionais, só faça com que os dois lados de um relacionamento se conheçam se isso for estritamente necessário para que o código esteja correto, no nosso exemplo não interessa a um produto saber quais clientes o tem, interessa apenas ao cliente saber quais produtos ele possui, então o melhor a se fazer é não colocar o relacionamento também em Produto.

Transformando um “muitos-para-muitos” em um “muitos-para-um”

Continuando com o nosso exemplo, também é comum que conforme o nosso conhecimento sobre o problema em si aumente e a modelagem evolua, esses relacionamentos N:N comecem a tomar corpo de forma que eles deixam de ser apenas um simples relacionamento e se transformam em uma entidade própria, com suas próprias informações e ciclo de vida. Relacionamentos N:N são, no fim, incomuns em sistemas reais, porque na maior parte das vezes nós temos que guardar informações sobre o relacionamento em si e simplesmente colocar novos atributos na tabela de qualquer forma torna o modelo do banco de dados difícil de se lidar e complica o modelo de objetos que vai agir sobre ele. Vejamos um teste de exemplo do código que faria uso dessa modelagem: Listagem 4 – Teste do relacionamento many-to-many

Cliente cliente = new Cliente();
cliente.setNome( "José" );

Produto produto = new Produto();
produto.setNome("Camisa de banda");

cliente.getProdutos().add(produto);

EntityManager manager = HibernateLoader.createEntityManager();

manager.getTransaction().begin();
manager.persist( cliente );
manager.getTransaction().commit();

Assert.assertTrue(
	 mensagem( "clientes" ) ,
	 contarClientes(manager) > quantidadeDeClientes );

Assert.assertTrue(
	mensagem( "produtos" ) ,
	contarProdutos(manager) > quantidadeDeProdutos );
manager.close();

manager = HibernateLoader.createEntityManager();
Cliente clienteDoBanco = manager.find( Cliente.class , cliente.getId());
Assert.assertTrue(
	"A quantidade de produtos do cliente deve ser maior que zero",
clienteDoBanco.getProdutos().size() > 0 );
manager.close();

Testar o código que se escreve é não apenas normal, como também obrigatório pra que se consiga software de qualidade nos dias de hoje, por isso o nosso exemplo é um teste escrito utilizando a biblioteca de testes JUnit. O código cria um Cliente, um Produto e relaciona o produto ao cliente, após isso nós começamos a testar as funcionalidades implementadas, primeiro nós testamos se a quantidade de clientes e produtos no banco de dados se alterou (os métodos “contarProdutos()“ e “contarClientes()”, “mensagem()” são métodos utilitários da nossa classe de testes que você pode conferir nos arquivos desse tutorial, a classe HibernateLoader é apenas uma classe utilitária criada no exemplo para criar os EntityManagers), após garantir que as quantidades foram alteradas, nós vemos se o produto realmente foi relacionado ao cliente. Para fazer esse último teste, nós criamos um novo EntityManager, isso foi necessário porque algumas implementações da JPA (como o Hibernate) mantém os objetos em um cache no próprio EntityManager, portanto se eu tentasse carregar o Cliente com o mesmo EntityManager que o salvou ele simplesmente me retornaria o objeto “cliente” que estava no seu cache em vez de fazer uma nova consulta no banco de dados. Tomando como base o nosso exemplo anterior, digamos que agora nós precisemos saber exatamente qual a quantidade de um produto específico um cliente tem, com o nosso diagrama anterior nós precisaríamos fazer uma contagem dos produtos relacionados ao cliente específico, o que é possível mas pouco prático, o melhor seria se o próprio relacionamento entre produtos e clientes já trouxesse esse relacionamento, dessa forma nós não precisaríamos ter produtos repetidos no relacionamento como também não seria necessário fazer contagens manuais, no próprio relacionamento a contagem já estaria feita. Vejamos como esse diagrama ficaria agora: Imagem 2 – Diagrama de exemplo com many-to-one Agora nós não temos apenas uma tabela que liga os dois objetos, mas uma entidade própria, que tem seus próprios atributos e representação dentro do sistema. O nosso item representa o relacionamento entre as tabelas clientes e produtos, além de conter informações que caracterizam o relacionamento, que no nosso caso é a quantidade de produtos que o cliente tem. A tabela de relacionamento “clientes_produtos” não precisa mais existir, pois a nova tabela “itens” já faz o trabalho dela. Vejamos agora como ficariam os códigos para esse nosso novo modelo: Listagem 5 – Novo Cliente.java

package alinhavado;

import java.util.HashSet;
import java.util.Set;

import javax.persistence.CascadeType;
import javax.persistence.Entity;
import javax.persistence.JoinColumn;
import javax.persistence.JoinTable;
import javax.persistence.ManyToMany;
import javax.persistence.OneToMany;
import javax.persistence.Table;

@Entity
@Table(name="clientes")
public class Cliente extends Persistivel {
	private String nome;
	@OneToMany(mappedBy="cliente", cascade=CascadeType.ALL)
	private Set items = new HashSet();
	// métodos get/set
}

O nosso cliente agora não mais se relaciona diretamente com os produtos, agora ele se relaciona com os itens, que por fim vão ser o relacionamento com os produtos. E já que falamos neles, vejamos a nossa classe Item: Listagem 6 – Item.java

package alinhavado;

import javax.persistence.CascadeType;
import javax.persistence.Entity;
import javax.persistence.JoinColumn;
import javax.persistence.ManyToOne;
import javax.persistence.Table;
@Entity
@Table(name="itens")
public class Item extends Persistivel {
	private Integer quantidade;
	@ManyToOne(cascade=CascadeType.ALL)
	@JoinColumn(name="produto_id")
	private Produto produto;
	@ManyToOne(cascade=CascadeType.ALL)
	@JoinColumn(name="cliente_id")
	private Cliente cliente;
	// métodos get/set
}

É na classe Item que reside agora o nosso relacionamento, ela contém uma referência para um Cliente e também para um Produto, além de guardar a quantidade de produtos que esse Item representa. Vejamos o exemplo de código que mostra esse relacionamento sendo utilizado: Listagem 7 – Exemplo do relacionamento many-to-one

Cliente cliente = new Cliente();
cliente.setNome( "José" );

Produto produto = new Produto();
produto.setNome("Camisa de banda");

Item item = new Item();
item.setQuantidade( 10 );

item.setCliente(cliente);
item.setProduto(produto);

EntityManager manager = HibernateLoader.createEntityManager();
manager.getTransaction().begin();
manager.persist( item );
manager.getTransaction().commit();

Assert.assertTrue(
	 mensagem( "clientes" ) ,
	 contarClientes(manager) > quantidadeDeClientes );
Assert.assertTrue(
	mensagem( "produtos" ) ,
	contarProdutos(manager) > quantidadeDeProdutos );
Assert.assertTrue(
	mensagem( "itens" ) ,
	contarProdutos(manager) > quantidadeDeItens );

manager.close();

manager = HibernateLoader.createEntityManager();
Cliente clienteDoBanco = manager.find( Cliente.class, cliente.getId());
Assert.assertTrue(
	"A quantidade de itens deve ser maior do que zero",
	clienteDoBanco.getItems().size() > 0);
manager.close();

Como você pode perceber, as diferenças do código são pequenas, nós criamos um Cliente, um Produto e em vez de simplesmente relacionar os dois, nós criamos um novo objeto, o Item, que guarda uma referência para o Cliente e outra para o Produto, além disso ele também conta com uma propriedade, a quantidade. Seguindo no teste nós validamos que agora existem mais clientes e produtos que antes, além de ver se o item foi realmente relacionado ao cliente em questão no último teste.

Conclusão

Relacionamentos N:N podem ser transformados de forma simples em relacionamentos N:1 quando você precisa guardar informações sobre a relação em si, você não deve, em momento algum, criar uma nova coluna em uma tabela de ligação e continuar tratando ela como sendo apenas uma tabela de ligação, se o relacionamento começar a ter propriedades próprias, é porque ele não é mais apenas um relacionamento, mas uma entidade real do seu sistema e deve começar a ser tratado como tal.

Referencias

Documentação oficial do Hibernate. Disponível em: http://hibernate.org/, acesso em 30/12/2007.

Quem quer ser gerente?

mauricio.linhares@gmail.com (Maurício Linhares) — Thu, 20 Dec 2007 00:00:00 UTC

Em uma empresa comum, havia um programador comum.

O programador comum não era tão comum como todos pensavam, ele fazia muito bem o seu trabalho, o código que ele escrevia funcionava corretamente, os problemas eram poucos e os clientes, satisfeitos. Tanto sucesso começou a chamar a atenção dos superiores do programador. Ele começou a subir na empresa, recebendo mais e também podendo tomar decisões mais importantes, ele começava a sentir-se parte de tudo aquilo que o cercava.

Enfim, ele galgou o último degrau que um programador chegaria, havia se tornado ‘arquiteto’ dos sistemas que desenvolvia. Ele continuava fazendo muito bem o seu trabalho, e os seus superiores mais uma vez acharam por bem dar-lhe mais uma promoção, ele agora seria um gerente de projetos. Até então, ele não sabia o que o cargo significava, ninguém havia lhe dito o que ou como as coisas seriam feitas, mas ele aceitou de bom grado, afinal, era uma promoção.

Em poucas semanas, ele percebeu o problema que havia recebido de presente, não podia mais escrever código, seu trabalho agora era lidar com pessoas, interesses, prazos e custos. Ele havia passado anos na universidade, estudado por livros e cursos de extensão para aprender a programar, mas a única coisa que ele recebeu sobre gerenciar foi um pequeno livro, “os 10 ideais do gerente perfeito”, tão gasto que até a capa havia se perdido. Em pouco tempo, a infelicidade e os problemas já haviam extinguido suas forças, os projetos não mais andavam, os resultados eram pífios e seus superiores não paravam de pressionar.

Nesse momento, o simples programador perguntou a si mesmo, “quem disse que eu queria ser gerente?”.

Por mais que pareça um tanto quanto exagerada, a novelização da breve carreira de um programador até as posições de gestão normalmente segue esse caminho, especialmente no Brasil, onde praticamente não existem carreiras diferenciadas para quem faz e quem gerencia o que tem que ser feito. Em grandes empresas por todo mundo existem caminhos diferentes de crescimento, onde um funcinário pode optar por sair da carreira puramente técnica e passar para o lado da gestão de pessoas e projetos, ou ele pode simplesmente continuar evoluindo no caminho técnico e ter os mesmos vencimentos que teria um gerente na sua posição.

A carreira de gerência não é mais nem menos glamurosa do que a do técnico (ou engenheiro, como alguns preferem chamar) e chega a ser esquisito que se entenda que o gerente, especialmente em um lugar onde se faz trabalho intelectual, como desenvolvimento de software, seja visto como alguém que está acima hierarquicamente de toda a equipe. O Google tem engenheiros incríveis como “Crazy” Bob Lee e esses engenheiros valem tanto quanto os gerentes que fazem com que os engenheiros tenham condições de trabalhar. Ao contrário do que a lenda diz, o trabalho do bom gerente não é simplesmente ficar com o açoite na mão pra ter certeza de que todo mundo está trabalhando, mas garantir que todos tem condições de fazer o seu trabalho corretamente. Já pensou o que seria do Google sem os seus engenheiros geniais trabalhando em coisas como o GMail?

Mas os técnicos são tudo? Claro que não. Se não fosse um gerente com visão, o próprio GMail seria um projeto engavetado pelo pessoal do Google. Mas são trabalhos diferentes que existem pessoas e perfis diferentes. Um bom técnico não necessariamente vai dar um bom gerente e vice-versa. E ninguém nunca vai ser um bom gerente se nunca tiver sido treinado pra isso. Na maior parte das empresas, as pessoas simplesmente recebem a notícia de que agora vão “gereciar projetos”, em momento algum se fala em como se gerencia projetos, além daquela “repassada básica” do que precisa ser feito ou dos artefatos que precisam ser gerados e isso é um dos maiores motivos das falhas em posições de gerência, não ter conhecimento o suficiente para assumir o cargo.

Outro grande impedimento é que ninguém sabe tudo e ninguém precisa ser bom gerenciando, bons técnicos que não tem aptidão para gerência deveriam continuar como técnicos mas mesmo assim poder subir na carreira e ter posições equivalentes aos seus companheiros que tomaram o caminho da gerência, pois sem os ténicos, quem é que os gerentes vão gerenciar? Ambos os papéis são necessários, um não pode ser tomado em detrimento do outro. Precisamos entender que pessoas diferentes tem motivações e interesses diferentes, forçar alguém a assimir uma posição ou responsabilidades que não o interessam só torna o problema ainda maior para os dois lados.

Então, antes de simplesmente pegar o seu engenheiro predileto e colocar ele em uma posição de gestão, faça a pergunta mágica a ele, “você quer ser gerente?”.

Referências:

Peopleware: Productive Projects and Teams Behind Closed Doors: Secrets of Great Management