Jason Codes

Installing Exim on Mac OS X

2013-12-31T00:00:00+10:00

Why replace Postfix with Exim when Postfix comes pre-installed with Mac OS X?

For a long time I had been using Postfix on my Macs to forward emails from cron jobs, etc. to my main email account. Since upgrading to Mavericks however, I found this to be less reliable than I would have liked. For various reasons, I send all outbound email via a smart host and Postfix has decided that sometimes it will ignore that setting and try direct delivery. The best part is that it seems to still try sending on the SMTP submission port (TCP 587) when doing this (rather than using TCP 25). Something’s broken and I’ve given up trying to fix it. I’ve decided to replace Postfix with something I know and trust: Exim.

My experience with Exim to date has been almost exclusively on Debian where the packagers have done a great job at making it easy to configure. dpkg-reconfigure exim4-config is pretty awesome. Luckily though, with a little bit of playing around and referencing the manual, it’s not too hard to get Exim going on Mac OS X.

Here we go. :)

Update 2014-01-02: Added section on Enabling IPv6 support

Update 2014-01-02: Added section on Postfix sendmail compatibility

Installing Exim

Creating a service user account

It’s best to run Exim under a dedicated user account. You can create one though the Users & Groups preference pane, but that will leave you with an additional user account showing up in the user interface. Since you’ll never log into this account interactively, it’s better to create a new system account instead. Unfortunately, Mac OS X does not come with a simple command line tool to create user accounts and instead multiple calls to dscl are required. The good news is that I have wrapped this all up into a shell script which I’ve called adduser.

You can install this utility in one of two ways: The first is to download the script file manually, place it somewhere in your path (e.g. ~/bin) and then chmod +x it. The second, easier way is to use fresh. fresh is a tool for managing your dotfiles and it works great for utility scripts. With fresh installed, you can simply run fresh https://github.com/jasoncodes/dotfiles/blob/master/bin/adduser and adduser will be installed.

Homebrew

Homebrew is a package manager for OS X. We could install Exim manually from source but Homebrew makes it so much easier. You probably want to install it now if you haven’t already.

Installing Exim

Create a user account for Exim, brew the formula, and set file permissions for the dedicated user account:

sudo adduser exim
USER=ref:exim brew install exim
sudo chown root /usr/local/etc/exim.conf
sudo cp -ai /usr/local/etc/exim.conf{,.org}
sudo chown exim /usr/local/var/spool/exim
sudo mkdir -p /usr/local/var/spool/exim
sudo chown exim:admin /usr/local/var/spool/exim
sudo chmod 750 /usr/local/var/spool/exim

404 Not Found {#404}

Note: If you get a "Download failed" error when trying to brew install Exim, you can grab a copy of exim-4.80.1.tar.gz from somewhere else (to Google!) and drop it into /Library/Caches/Homebrew. Re-running brew install will then use this pre-cached copy. A great thing to note about Homebrew is that it will checksum the downloaded file to make sure it matches the original source file the formula creator used.

Configuring Exim

Open /usr/local/etc/exim.conf in your preferred text editor. Note that you’ll need to be able to write to this file as root. sudo vim /usr/local/etc/exim.conf is one way to do this but I prefer vim-eunuch’s :SudoWrite.

Local Hostname

For machines which are always on a single network with their hostname configured in DNS, the output of hostname should be both predicable and stable. For mobile machines which roam between networks (e.g. laptops), you’ll probably have better reliability if you tell Exim which hostname you’d like to use when referring to your local machine.

To set the hostname which Exim uses, search for primary_hostname in the configuration file, uncomment the line and set the value to the full hostname of your machine. e.g. example.local.

Block external access

Exim by default denies relay attempts but it’s still good policy to not expose services when you don’t need to. To prevent Exim from listening on all network interfaces, add the following after the primary_hostname entry:

local_interfaces = 127.0.0.1

Postfix `sendmail` compatibility

Exim and Postfix have different default behaviours for sendmail’s -t option which used by default by Rails’ (ActionMailer) sendmail delivery method. This option extracts email addresses from the message headers. When additional email addresses are supplied on the command line, Postfix adds these to the extracted set. Exim’s default is to remove any addresses specified on the command line from the extracted set. Rails expects Postfix’s behaviour. Add the following to the main configuration section (after local_interfaces is fine):

extract_addresses_remove_arguments = false

Exim also adds a Sender header when using sendmail with a custom From address. One generally does not want this behaviour as the Sender header is often displayed in email clients. You can always tell what user account sent an email by examining the Received headers. Add the following to disable this behaviour:

no_local_from_check

Routers

Search for begin routers. This section controls how mail is routed to its destination. We want to send all mail via a smarthost rather than using the default behaviour of delivering directly to destination mail servers via MX entries.

Comment out the existing dnslookup router entry and add a new entry below it to route via a smarthost:

smart_route:
  driver = manualroute
  domains = !+local_domains
  transport = smarthost
  route_list = * smtp.example.net::587

Replace smtp.example.net with your upstream SMTP smarthost server’s hostname.

Transports

Search for begin transports. This section controls how mail is delivered once a destination is found by the router. Notice the "transport" setting in the router configuration. We want to force TLS (encryption) and use authentication (when required) for the target smarthost.

Add a new smarthost transport below the remote_smtp entry:

smarthost:
  driver = smtp
  hosts_require_tls = *
  hosts_require_auth = ${lookup{$host}nwildlsearch{/usr/local/etc/exim/passwd.client}{*}}

Authentication

Search for begin authenticators. This section controls where authentication credentials are retrieved from for both inbound (server) and outbound (client) connections. We're only authenticating as a client here so here's an entry to add which retrieves the username and password from our configuration file:

plain:
  driver = plaintext
  public_name = PLAIN
  client_send = "^${extract{1}{::}{${lookup{$host}lsearch*{/usr/local/etc/exim/passwd.client}{$value}fail}}}\
                 ^${extract{2}{::}{${lookup{$host}lsearch*{/usr/local/etc/exim/passwd.client}{$value}fail}}}"

Next, we’ll create a secured password file to store the credentials for our smarthost.

sudo mkdir /usr/local/etc/exim
sudo touch /usr/local/etc/exim/passwd.client
sudo chmod 600 /usr/local/etc/exim/passwd.client
sudo chown exim /usr/local/etc/exim/passwd.client

Add a line like the following to /usr/local/etc/exim/passwd.client, replacing the placeholders with your smarthost’s hostname, username, and password:

smtp.example.net:username:password

Forwarding local user accounts

Create .forward files in the home directory of any local accounts you want to receive mail for. The file should contain a single line with the destination email address.

Enabling IPv6 support

Exim’s IPv6 support is not enabled out of the box. If you’re interested in this, it’s fairly easy to get going.

We’ll first have to edit the Homebrew formula to compile Exim with IPv6 support enabled. Run brew edit exim and add s << "HAVE_IPV6=yes\n" to the end of the inreplace 'Local/Makefile' block. Run brew uninstall exim to remove the IPv4 version and then re-run USER=ref:exim brew install exim to install the IPv6 enabled version.

Secondly, we’ll add the IPv6 loopback address to the allowed list for relaying. Search for relay_from_hosts and change the value to <; 127.0.0.1 ; ::1.

Finally, we’ll add the IPv6 loopback interface to the list of interfaces to listen to. Search for local_interfaces and change the value to <; 127.0.0.1 ; ::1.

Syntax check config file

Run sudo exim -bV to check the syntax of the config file. Any major errors will be detected by this command. If all is good, you should see Configuration file is /usr/local/etc/exim.conf as the last line of output.

Running Exim on port 25

If you have any other SMTP server running, you should disable it now. If you followed my previous Postfix on OS X guide, you can do this by running sudo launchctl unload -w /Library/LaunchDaemons/org.postfix.master.plist.

Create the following launchd daemon configuration file at /Library/LaunchDaemons/exim.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>exim</string>
  <key>UserName</key>
  <string>root</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/exim</string>
    <string>-bdf</string>
    <string>-q30m</string>
  </array>
  <key>RunAtLoad</key>
  <true />
  <key>KeepAlive</key>
  <true />
</dict>
</plist>

Start the server now by running sudo launchctl load -w /Library/LaunchDaemons/exim.plist.

Run nc -n 127.0.0.1 25 < /dev/null and you should see a 220 banner message confirming the server is now running.

Replace Postfix `sendmail` with Exim

UNIX services such as cron use sendmail rather than using SMTP to deliver mail. In order for these to work, we’ll need to swap out Postfix’s sendmail binary (/usr/sbin/sendmail) for Exim.

sudo mv -i /usr/sbin/sendmail{,.org}
sudo ln -s /usr/local/bin/exim /usr/sbin/sendmail
sudo chown root:wheel /usr/sbin/sendmail
sudo chmod u+s /usr/sbin/sendmail

Log rotation

The main log file for Exim is stored at /usr/local/var/spool/exim/log/mainlog. You can view this file if you wish to see detail on what Exim is doing.

Exim comes with a tool to perform log rotation. Let’s setup a launchd schedule to rotate the logs once a day. Create /Library/LaunchDaemons/exim-logrotate.plist with the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>exim-logrotate</string>
  <key>UserName</key>
  <string>root</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/exicyclog</string>
    <string>-k</string>
    <string>30</string>
  </array>
  <key>RunAtLoad</key>
  <false/>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Hour</key>
    <integer>6</integer>
    <key>Minute</key>
    <integer>25</integer>
  </dict>
</dict>
</plist>

Register the log rotation job with launchctl by running sudo launchctl load -w /Library/LaunchDaemons/exim-logrotate.plist.

Testing

You can test sendmail is working by sending a test message using mail. Assuming you setup a .forward file earlier for your user account, the following should send you an email:

date | mail -s Test $USER

If you receive this test email, you’re done! Yay!

Nested resource URLs without controller names or numeric IDs in Rails 3

2011-04-04T00:00:00+10:00

Let's say you want your app to have URLs like GitHub's: https://github.com/mojombo/jekyll. There are a number of advantages to URLs of this form over more conventional Rails resource URLs:

Anyone with a vague familiarity with GitHub knows that the first path segment is the username and the second segment is the repository name. Without visiting a GitHub URL you can have a good idea of who owns the repository and what the name of the project is. This also makes remembering and typing URLs manually very easy.
Repositories are namespaced under accounts. As all repository URLs contain the account name, repository names need only be unique within a single account.
There are no controller or model names in the URL. A URL such as https://github.com/accounts/mojombo/repositories/jekyll would be overly verbose and would add no value.
Services like GitHub and Twitter allow you to rename accounts. Unfortunately, doing so breaks all existing external links. Out of the box Rails URLs contains numeric IDs which are immutable but aren't overly user friendly. By keeping track of history we can prevent links from breaking when slugs change. FriendlyId solves this problem perfectly.

There are a couple of ways to implement URLs like GitHub's in Rails. One way is with a bunch of custom routes. This can easily result in a complex routes configuration file, especially if we nested further resources underneath repositories (such as issues or wiki pages). Care would also need to be taken for route helpers to have decent names.

A far nicer way to implement these kind of URLs is to define nested resources and hide the relevant controller names from the generated paths. This is the approach we will take here. Source code for this post's example app is available on GitHub.

Note: This post is for Rails 3. For Rails 2.3, you can use default_routing in combination with FriendlyId. I have a fork of default_routing which adds support for Bundler.

Project setup

Generate new Rails app

Create a new directory and setup our RVM gemset:

mkdir example
cd example
git init
echo rvm --create 1.9.2@example > .rvmrc
cd . # trigger RVM to load the rvmrc file

Generate a new Rails 3 app without Test::Unit or Prototype. We'll use RSpec for testing and we can add jquery-rails later when we want to add client-side scripting.

rails new . --skip-test-unit --skip-prototype

Add Inherited Resources and RSpec to the Gemfile. We'll be using these shortly.

gem 'inherited_resources'

group :development, :test do
  gem 'rspec-rails'
end

Run bundle to ensure all the required gems are installed.

Create models

Generate our model files and migrations. We'll be nesting projects underneath accounts and each model will have their own name.

rails generate model account name:string
rails generate model project name:string account:references

At this point we should add database constraints to the migration and validations to the model. For brevity we'll just add the has_many association for Account here:

class Account < ActiveRecord::Base
  has_many :projects
end

And then run the migrations with rake db:migrate.

Controllers and initial routes

Now that our test models are ready, let's add an initial set of routes to config/routes.rb:

Example::Application.routes.draw do
  resources :accounts do
    resources :projects
  end
end

Add a couple of controllers using Inherited Resources:

class AccountsController < ApplicationController
  inherit_resources
end

class ProjectsController < ApplicationController
  inherit_resources
  belongs_to :account
end

We now have URLs of the classic https://example.com/accounts/3141/projects/59265 format. At this point I have also created specs for the routes which I have omitted here for brevity. You can find these in the controllers commit in the example app repo.

Replacing numeric IDs in URLs with slugs

Revealing our surrogate primary keys in user visible URLs is not overly pretty. Usually there is a name field or other suitable text identifier which we could use. Name fields are rarely suitable for use directly in a URL as they typically contain unsafe characters and are often not unique nor immutable. Luckily for us, there's FriendlyId which normalises our name fields and ensures they are unique by adding a sequence number if required. With FriendlyId, we can easily turn URLs from https://example.com/accounts/3141/projects/59265 into https://example.com/accounts/foocorp/projects/widgets.

Add friendly_id to the Gemfile and run bundle:

gem 'friendly_id', '~> 3.2'

Next, create the slugs table. This is where FriendlyId stores information on all current and previous slugs to allow existing URLs to continue to function even if we rename an account or project.

In a production app you should 301 redirect any old slugs to the latest slug by checking resource.friendly_id_status.best? in a before_filter. This will prevent search engines from seeing the same content at different URLs.

rails generate friendly_id

Add a cached slug column to each model table. This is done primarily for performance reasons. This allows FriendlyId to generate URLs without having to recalculate and verify the slug every time.

rails generate migration add_cached_slug_to_accounts cached_slug:string
rails generate migration add_cached_slug_to_projects cached_slug:string

The cached_slug fields will also be preferred for lookups and thus should be indexed together with any parent scope ID. Ensure you add the relevant indexes to the migration.

class AddCachedSlugToAccounts < ActiveRecord::Migration
  def self.up
    add_column :accounts, :cached_slug, :string
    add_index :accounts, :cached_slug, :unique => true
  end

  def self.down
    remove_column :accounts, :cached_slug
  end
end

class AddCachedSlugToProjects < ActiveRecord::Migration
  def self.up
    add_column :projects, :cached_slug, :string
    add_index :projects, [:account_id, :cached_slug], :unique => true
  end

  def self.down
    remove_column :projects, :cached_slug
  end
end

Add has_friendly_id to the models:

class Account < ActiveRecord::Base
  has_friendly_id :name, :use_slug => true
end

class Project < ActiveRecord::Base
  has_friendly_id :name, :use_slug => true, :scope => :account_id
end

Run the migrations and generate slugs for any existing records:

rake db:migrate
rake friendly_id:make_slugs MODEL=Account
rake friendly_id:make_slugs MODEL=Project

Removing the controller names from URLs

Now that we have URLs like https://example.com/accounts/foocorp/projects/widgets, we need to remove the the controller name segments from the path so we end up with URLs like https://example.com/foocorp/widgets.

Updating the specs

First things first, let's update the routing specs to match the URLs we are after. i.e. /foocorp/widgets instead of /accounts/foocorp/projects/widgets.

spec/routing/accounts_routing_spec.rb:

require 'spec_helper'

describe AccountsController do
  describe "routing" do
    it '/ to Accounts#index' do
      path = accounts_path
      path.should == '/'
      { :get => path }.should route_to(
        :controller => 'accounts',
        :action => 'index'
      )
    end

    it '/new to Account#new' do
      path = new_account_path
      path.should == '/new'
      { :get => path }.should route_to(
        :controller => 'accounts',
        :action => 'new'
      )
    end

    it '/:account_id to Account#show' do
      path = account_path 'foocorp'
      path.should == '/foocorp'
      { :get => path }.should route_to(
        :controller => 'accounts',
        :action => 'show',
        :id => 'foocorp'
      )
    end

    it '/:account_id/edit to Account#edit' do
      path = edit_account_path 'foocorp'
      path.should == '/foocorp/edit'
      { :get => path }.should route_to(
        :controller => 'accounts',
        :action => 'edit',
        :id => 'foocorp'
      )
    end
  end
end

spec/routing/projects_routing_spec.rb:

require 'spec_helper'

describe ProjectsController do
  describe "routing" do
    it '/:account_id/new to Projects#new' do
      path = new_account_project_path('foocorp')
      path.should == '/foocorp/new'
      { :get => path }.should route_to(
        :controller => 'projects',
        :action => 'new',
        :account_id => 'foocorp'
      )
    end

    it '/:account_id/:project_id to Projects#show' do
      path = account_project_path 'foocorp', 'widgets'
      path.should == '/foocorp/widgets'
      { :get => path }.should route_to(
        :controller => 'projects',
        :action => 'show',
        :account_id => 'foocorp',
        :id => 'widgets'
      )
    end

    it '/:account_id/:project_id/edit to Projects#edit' do
      path = edit_account_project_path 'foocorp', 'widgets'
      path.should == '/foocorp/widgets/edit'
      { :get => path }.should route_to(
        :controller => 'projects',
        :action => 'edit',
        :account_id => 'foocorp',
        :id => 'widgets'
      )
    end
  end
end

Updating the routes

We can customise the controller names in routes by using the :path option on the resources block. By setting the path to an empty string, we can remove the controller name segment completely from the generated paths.

With no prefix on the nested resource routes, both the show page for accounts (/accounts/foocorp) and the index page for the nested projects (/accounts/foocorp/projects) will end up with the same URL (/foocorp). Since we can't have both of these at the same URL, we should only generate a route for one of these two actions to prevent confusion and possible bugs. In most cases I've found that disabling the nested index route is best as typically you'd want a normal show page for the parent resource. For example, the project listing makes up only part of the account's show page at /foocorp.

Here's the updated routes entries:

Example::Application.routes.draw do
  resources :accounts, :path => '' do
    resources :projects, :path => '', :except => [:index]
  end
end

A small problem

Unfortunately this does not work quite to plan. If you run the specs, you'll see the member actions on the parent resource are not working. Viewing the output of rake routes confirms why:

account_project GET    /:account_id/:id(.:format)      {:action=>"show", :controller=>"projects"}
   edit_account GET    /:id/edit(.:format)             {:action=>"edit", :controller=>"accounts"}

The nested show route is outputted first and catches all member actions on the parent. If you take a look the implementation of resources in action_dispatch/routing/mapper.rb, you'll see that the child block is yielded before any of the resources own routes are outputted.

An easy solution

The workaround for this is to place any nested resources which have empty paths (:path => '') within a second parent resources block. This second block uses :only => [] to prevent it from generating any routes of its own. Any member or collection blocks for resources :accounts (as well as any only/except constraints) can be added as normal to the first resources :accounts entry.

Example::Application.routes.draw do
  resources :accounts, :path => ''
  resources :accounts, :path => '', :only => [] do
    resources :projects, :path => '', :except => [:index]
  end
end

Voilà, all of the specs are now passing.

Upgrading Tomcat with Homebrew

2011-02-14T00:00:00+10:00

If you followed my guide on setting up Tomcat on Mac OS with Homebrew, at some point you may want to update Tomcat to the latest version.

You could stop remove the old version, install the new version, update the symlink then restart the service. You might even get away with leaving the restart to the end. I prefer to install the new version side by side and then flip the symlink just before restarting. This leaves a much smaller downtime window and makes rolling back really easy.

The first step is to update your local formula with the new version info by running the following:

brew edit tomcat

Here's the change I made to move from 7.0.6 to 7.0.8:

diff --git a/Library/Formula/tomcat.rb b/Library/Formula/tomcat.rb
index 90b3443..bded504 100644
--- a/Library/Formula/tomcat.rb
+++ b/Library/Formula/tomcat.rb
@@ -1,9 +1,9 @@
 require 'formula'
 
 class Tomcat <Formula
-  url 'http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.6/bin/apache-tomcat-7.0.6.tar.gz'
+  url 'http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.8/bin/apache-tomcat-7.0.8.tar.gz'
   homepage 'http://tomcat.apache.org/'
-  md5 '1c54578e2e695212ab3ed75170930df4'
+  md5 'b18b0f1d987f82038a7afeb2e3075511'
 
   skip_clean :all

After the formula is updated you can install the new version with:

brew install tomcat

As per the initial install, Homebrew links the keg which results in a handful of generically named scripts being added to your path. We'll need to unlink it.

If you try to unlink with brew unlink tomcat you'll get Error: tomcat has multiple installed versions. It seems the brew command doesn't really like multiple versions of the same formula being installed at the same time and it presently doesn't expose a way to do this other than removing the old version first. We can however unlink it by calling the Homebrew API directly:

ruby -I/usr/local/Library/Homebrew -rglobal -rkeg -e 'puts Keg.new("/usr/local/Cellar/tomcat/7.0.8").unlink'

The next step is to switch out the libexec symlink:

sudo -u tomcat ln -sf /usr/local/Cellar/tomcat/7.0.8/libexec /usr/local/tomcat/

And then restart the service:

sudo launchctl unload -w /Library/LaunchDaemons/org.apache.tomcat.plist
sudo launchctl load -w /Library/LaunchDaemons/org.apache.tomcat.plist

If all goes well, remove the old version (via the API to workaround the brew limitation with multiple versions):

ruby -I/usr/local/Library/Homebrew -rglobal -rkeg -e 'k = Keg.new("/usr/local/Cellar/tomcat/7.0.6"); puts k.unlink; k.uninstall'

Restarting Tomcat automatically on Mac OS with Monit

2011-02-14T00:00:00+10:00

When I setup Tomcat on Mac OS with Homebrew I also setup Monit to monitor if daemons fail. What I didn't do is tell Monit how to restart Tomcat should it fail.

Unfortunately it's often not just a simple case of bouncing Tomcat with launchctl when something goes wrong. This is especially the case with Tomcat as we're running it via a launch script. If launchd times out and kills the script process, any frozen Java process will stay running. That process will still be holding open port 8080 and will prevent a new instance from starting.

I came up with the following shell script to restart Tomcat fully, even if the Java process has frozen. The script asks launchd for what PID it is managing (the catalina.sh script) and then kills its direct children (the Java process). First we try with a normal kill and then fall back on a kill -9 if it won't die. Once the existing process is dead, we can bounce the service with launchctl. Finally, we wait a few seconds to ensure the service has finished starting back up before we return control back to Monit.

Save the following as /usr/local/sbin/tomcat-restart and chmod +x it:

#!/bin/bash -e
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin

TEMPFILE="`mktemp -t tomcat-restart.XXXXXX`"
trap '{ rm -f "$TEMPFILE"; }' EXIT

if launchctl list -x org.apache.tomcat 2> $TEMPFILE
then
  ln "$TEMPFILE" "$TEMPFILE.plist"
  BASE_PID=$(defaults read "$TEMPFILE" PID 2> /dev/null || true)
  rm "$TEMPFILE.plist"
  
  CHILD_PID="$(ps -axo pid,ppid | awk "{ if ( \$2 == \"$BASE_PID\" ) { print \$1 }}")"
  
  if ! [ -z "$CHILD_PID" ]
  then
    echo "Killing Tomcat softly..."
    kill $CHILD_PID
    sleep 2
    if kill -0 $CHILD_PID 2> /dev/null
    then
      sleep 2
    fi
    if kill -0 $CHILD_PID 2> /dev/null
    then
      echo "It's not dead yet. Waiting a little longer..."
      sleep 5
    fi
    if kill -0 $CHILD_PID 2> /dev/null
    then
      echo "Nuking from orbit..."
      kill -9 $CHILD_PID $BASE_PID
    fi
  fi
  
fi

echo "Reversing the polarity..."
sudo launchctl unload -w /Library/LaunchDaemons/org.apache.tomcat.plist || echo "It's dead Jim."
sudo launchctl load -w /Library/LaunchDaemons/org.apache.tomcat.plist || echo "I can't revive it."

if ! curl --connect-timeout 5 --max-time 5 --silent localhost:8080 > /dev/null
then
  echo "Waiting for launch..."
  sleep 5
fi
if ! curl --connect-timeout 5 --max-time 5 --silent localhost:8080 > /dev/null
then
  echo "Not ready yet..."
  sleep 5
fi
if ! curl --connect-timeout 5 --max-time 5 --silent localhost:8080 > /dev/null
then
  echo "It's broken."
  exit 1
fi

echo Done.

Then all we have to do is tell Monit to run this script when the service fails. My updated service entry for Tomcat in /etc/monitrc is as follows:

check host tomcat with address 127.0.0.1
  if failed port 8080
    send "HEAD / HTTP/1.0\r\n\r\n"
    expect "HTTP/1.1"
    with timeout 5 seconds
    then exec "/usr/local/sbin/tomcat-restart"

Check the syntax is valid and then reload Monit:

sudo monit -t
sudo monit reload

CSRF vulnerability in Ruby on Rails 2.3.10 & 3.0.3

2011-02-10T00:00:00+10:00

Yesterday Rails 3.0.4 and Rails 2.3.11 were released with patches for a few security issues. The two you're most likely to be affected by are CSRF Protection Bypass in Ruby on Rails (CVE-2011-0447) and Potential SQL Injection in Rails 3.0.x (CVE-2011-0448). I'll be looking at the CSRF issue here. The Riding Rails blog has a post on the CSRF protection patch which is worth reading.

The problem

It has been discovered that browser plugins such as Flash and Java in some circumstances can bypass the same origin policy and send requests with a X-Requested-With: XMLHttpRequest header and POSTing with a different content type than normally used by forms (i.e. application/javascript instead of application/x-www-form-urlencoded or multipart/form-data). These methods are used by Rails to detect the difference between form POSTs which need the authenticity token field and AJAX requests (from jQuery or Prototype) which do not. This seemed like a good way to prevent cross-site request forgeries since these headers can't be set for a POST via a <form/> and the same origin policy prevents XMLHttpRequest requests from other sites.

For more detail on how this vulnerability works, see CSRF: Flash + 307 redirect = Game Over on the Web Application Security Consortium mailing list.

The fix

Rails 3.0.4 and Rails 2.3.11 have patches for this issue in commits ae19e41 and 7e86f9b respectively. With this patch, Rails now marks all non-GET requests which don't contain a valid authenticity token as unverified. This means AJAX POST requests will now need to pass send the authenticity token in a HTTP header.

The method for handling invalid requests (that is, POSTs without a valid authenticity token) has also changed with this patch. Instead of raising an InvalidAuthenticityToken exception, Rails now calls handle_unverified_request which by default clears your session data. The idea of this is that unverified requests cannot do any damage if they don't have access to any persistent state (like your user session).

A nice thing about this new setup is if you use HTTP authentication or an X-API-Token type header for API requests, they'll continue to function fine as these authentication systems don't use session data.

Authlogic

Authlogic however stores authentication data in a cookie which by default is called user_credentials. This is separate from session data so we have to override handle_unverified_request in the ApplicationController to clear this ourselves.

Update: If you run plugins which initialise current_user before protect_from_forgery runs (such as paper_trail which adds a before_filter to ApplicationController::Base), by the time handle_unverified_request is called Authlogic will have already logged in and it's too late to prevent authentication by just deleting the cookie. We need to clear the cached @current_user and @current_user_session variables to be sure.

def handle_unverified_request
  super
  cookies.delete 'user_credentials'
  @current_user_session = @current_user = nil
end

Please do not blindly upgrade to 2.3.11/3.0.4 without carefully testing your authentication system with POST requests lacking an authentication token. Without adding the above code we would effectively lose CSRF protection as the app will still see the authentication and process the request where there's an invalid or missing authenticity token.

Devise

If you use Devise's "remember me" functionally, you'll need to clear the persistent cookie. By default this is called remember_user_token. If you're making use of multiple Warden scopes, make sure you handle those as well.

def handle_unverified_request
  super
  cookies.delete 'remember_user_token'
  sign_out :user
end

Update: Devise Security Release 1.1.6 patches this issue for Rails 3 by calling sign_out_all_scopes within handle_unverified_request. Devise 1.0.10 for Rails 2.3 has also been released which backports sign_out_all_scopes and calls it from handle_unverified_request. If you're using Devise, upgrade to these versions or later if possible.

Ajax requests (jQuery)

Since X-Requested-With: XMLHttpRequest is no longer enough to verify authenticity to Rails, we'll need to pass the authenticity token with each non-GET Ajax request. New to 2.3.11 is the csrf_meta_tag form helper which has been backported from Rails 3. If you don't already have csrf_meta_tag in your layout, add the following to the %head section of your (hopefully Haml) layouts:

%head
  = csrf_meta_tag

This adds a couple of meta attributes to every page which jQuery can look up to get the authenticity token. The second part of making Ajax requests work again is to set the X-CSRF-Token header on all Ajax requests with the authenticity token.

Update: Mislav Marohnić points out that Rails uses jQuery 1.5's new ajaxPrefilter feature if available in preference to beforeSend. Using ajaxPrefilter instead of beforeSend is preferable as a beforeSend hook would be overridden if any Ajax call uses the beforeSend option itself. ajaxPrefilter allows multiple filters to be added and thus avoids this issue.

If you're using the latest jquery-ujs in your Rails 3 app, you shouldn't need any custom code at all for CSRF protection to work. If you're on Rails 2 (or for some reason stuck using Prototype for UJS), add the following to your main application.js file to send the CSRF token with jQuery Ajax requests:

// Make sure that every Ajax request sends the CSRF token
function CSRFProtection(xhr) {
 var token = $('meta[name="csrf-token"]').attr('content');
 if (token) xhr.setRequestHeader('X-CSRF-Token', token);
}
if ('ajaxPrefilter' in $) $.ajaxPrefilter(function(options, originalOptions, xhr) { CSRFProtection(xhr); });
else $(document).ajaxSend(function(e, xhr) { CSRFProtection(xhr); });

Avoiding authentication denial of service (DoS)

One big thing I don't like about this fix is that it opens up an opportunity for a remote host to kill your login session. This can happen because any POST request with an invalid or missing authenticity token will kill your session data. Any other site with a <script/> block could POST a form to your site to log you out. It's almost as bad as making your logout link a GET instead of a POST.

Because of this I decided I'd rather have the previous behaviour of raising an InvalidAuthenticityToken exception to reject unverified requests completely:

def handle_unverified_request
  raise ActionController::InvalidAuthenticityToken
end

A side affect of this is that any POST requests outside of your web UI (such as to an API) will now fail as they aren't passing in the X-CSRF-Token header. We can get the best of both worlds by raising InvalidAuthenticityToken for web requests and clearing session for all other requests.

def handle_unverified_request
  content_mime_type = request.respond_to?(:content_mime_type) ? request.content_mime_type : request.content_type
  if content_mime_type && content_mime_type.verify_request?
    raise ActionController::InvalidAuthenticityToken
  else
    super
    cookies.delete 'user_credentials'
    @current_user_session = @current_user = nil
  end
end

Hosting Rails apps on a Mac OS X server

2011-02-07T00:00:00+10:00

There are many guides for setting up Rails development environments on various platforms including Mac OS and Ubuntu. I thought I'd mix it up a little with my complete guide on setting up a production Mac OS server.

Update 2011-02-12: Added a note to the backups section on excluding large changing files (such as databases) from Time Machine backups.

Update 2011-03-26 All launchd configuration files for services should be placed in LaunchDaemons not LaunchAgents to run at startup as the correct user account. LaunchAgents are for interactive processes ran under as the logged in console user.

Update 2012-01-04 Added load average and file system monitoring to the example Monit configuration.

Xcode
Homebrew
Sysadmin tweaks
Ruby 1.9.2 via system-wide RVM
Apache & Passenger
User accounts
Email
PostgreSQL
Memcached
ImageMagick
Tomcat
Git Hosting
SSH on an alternate port
Monit
Backups
Applications

Xcode

Download and install Xcode if you haven't already. It provides useful tools like a C compiler. Pretty much nothing is going to work without one.

Homebrew

Homebrew is an awesome package manager for Mac OS. It's superior to MacPorts in many ways. If you're setting up a new machine, there's no decision to be made. If you're already running MacPorts it's still seriously worth switching over.

You can follow the Homebrew installation instructions on the Homebrew wiki or run the steps below to use my formulas:

ruby -e "$(curl -fsSL https://gist.github.com/raw/323731/install_homebrew.rb)"
brew install git
git clone http://github.com/jasoncodes/homebrew.git /tmp/homebrew # substitute your preferred fork
mv /tmp/homebrew/.git /usr/local/
rm -rf /tmp/homebrew
cd /usr/local/
git remote add mxcl http://github.com/mxcl/homebrew.git
git fetch --all

Sysadmin tweaks

GNU command-line utilities

Installing a few GNU utilities makes Mac OS's BSD userland feel more like home. You can skip this step if you're happy with the BSD variants and your apps don't need GNU flavour.

brew install coreutils gnu-sed gawk findutils --default-names

Note: Installing these tools with --default-names will make the GNU variants the default and could possibly cause issues with any scripts that expect BSD versions. The GNU versions generally accept all the options as the BSD versions but there are a few differences. For example: BSD sed uses -E for extended mode and GNU sed uses -r and --regexp-extended. For compatibility with the BSD version of sed I use /usr/bin/sed in this guide.

dot files

I have customized my environment quite a bit and it can be frustrating to use a machine without my settings. As such, I like to install on every machine I use. Everything in this post should work on a bare config (and hopefully under your config as well). Here's what I run to setup my shell config:

curl -sL http://github.com/jasoncodes/dotfiles/raw/master/install.sh | bash
exec bash -i # reload the shell

htop

htop is a top alternative which makes interactive use much easier. It primarily targets Linux but the basic functions work fine on Mac OS. It's far nicer to use than the version of top which comes with Mac OS.

brew install htop

You'll need to sudo htop when running htop in order to see all process information. I also like to enable the Highlight program "basename" setting.

Ruby 1.9.2 via system-wide RVM

Install RVM

sudo bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
echo -e "[[ -s '/usr/local/rvm/scripts/rvm' ]] && source '/usr/local/rvm/scripts/rvm'\n" | sudo bash -c 'cat - /etc/bashrc > /etc/bashrc.new && mv /etc/bashrc{.new,}' # add RVM to global shell config
source '/usr/local/rvm/scripts/rvm' # load RVM in current session

We prepend the RVM loader to /etc/bashrc so it runs on non-interactive shells such as cron. This in combination with /bin/bash -l -c (which is automatically provided by the whenever gem), we can have the RVM provided Ruby 1.9.2 available in cron jobs.

Install Ruby 1.9.2 and set it as default

sudo rvm pkg install readline
brew install libyaml
sudo rvm install 1.9.2 --with-readline-dir=$rvm_path/usr --with-libyaml-dir=/usr/local
sudo rvm --default 1.9.2
rvm default

Update 2011-04-29: These instructions originally installed libyaml for the Psych YAML parser on Ruby 1.9.2. Unfortunately Ruby 1.9.2 up to and including p180 has an issue with Psych where by it fails with merge keys. This can cause problems with certain versions of Bundler and RubyGems, as well as DRY database.yml files. The issue is fixed in Ruby HEAD and there's a now somewhat stale ticket open to backport to 1.9.2. Hopefully we see a fix in the next Ruby 1.9.2 patch release.

Update 2011-07-16: Ruby 1.9.2 p290 was released today and includes the Psych fix for YAML merge keys. As a result, I have re-added libyaml to the Ruby installation instructions. If you're upgrading you'll need to update RVM with rvm get head && rvm reload first. To migrate a gemset over to the latest Ruby, run rvm gemset copy 1.9.2-p{180,290}@example && rvm 1.9.2-p180 gemset delete example.

Lockdown RVM installation

By default RVM will give all users access to modify RVM rubies, gemsets, et al. This is problem as the RVM install is system wide and users should not be able to mess with the environment of others. Luckily, all we need to do is empty out the rvm group which will leave root as the only user allowed to administer RVM:

sudo dscl . delete /Groups/rvm GroupMembership

Fix Homebrew permissions broken by installing RVM system-wide

After installing RVM system-wide you may find /usr/local/lib and /usr/local/bin to be locked down. We can liberate them again without reinstalling Homebrew by coping the owner and permissions from another directory (such as /usr/local/share/man) which is unaffected by the installation of RVM.

sudo chmod -R --reference=/usr/local/lib /usr/local/bin /usr/local/share/man
sudo chown -R --reference=/usr/local/lib /usr/local/bin /usr/local/share/man

Install Bundler

We'll be using Bundler's deployment mode via Capistrano to install and manage gems. This keeps our system gems clean and isolates apps from each other.

sudo gem install bundler

Apache & Passenger

Apache 2 comes standard with Mac OS 10.6. Sprinkle Phusion Passenger on top and we have an app server for Rack apps (including Rails).

sudo gem install passenger
rvmsudo passenger-install-apache2-module

Copy the 3 configuration lines emitted after installation into /etc/apache2/other/passenger.conf:

LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.2/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.2
PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.2-p136/ruby

There are a number of configuration options you can set in passenger.conf to control how it manages worker instances. You can read about these in the Passenger users guide. Here's the settings I'm using:

RailsSpawnMethod smart-lv2
RailsFrameworkSpawnerIdleTime 0
RailsAppSpawnerIdleTime 0
PassengerUseGlobalQueue on
PassengerFriendlyErrorPages off

# recycle instances every so often to keep any leaks under control
PassengerMaxRequests 1000

# default 6
PassengerMaxPoolSize 16

# default 0
PassengerMaxInstancesPerApp 5

# keep at least one instance around per app, let the others time out after 2 minutes
PassengerMinInstances 1
PassengerPoolIdleTime 120

For the configuration changes to take affect you need to restart Apache by running the following:

sudo launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist
sudo launchctl load -w /System/Library/LaunchDaemons/org.apache.httpd.plist

Once restarted you should see Passenger in the server signature:

$ curl -sI localhost | grep ^Server
Server: Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/0.9.8l DAV/2 Phusion_Passenger/3.0.2

HTTP Compression

mod_deflate is loaded by default but it's not configured to compress any responses automatically. Save the following as /etc/apache2/other/deflate.conf to enable HTTP compression for HTML, CSS, Javascript and fonts:

AddOutputFilterByType DEFLATE text/html text/plain text/xml font/ttf font/otf application/vnd.ms-fontobject text/css application/javascript application/atom+xml

Virtual Hosts

It's useful to have a default virtual host to catch any hits that goto any undefined hostnames or direct IP requests. Here's a basic vhost which will act as the default. The key thing to note here is the zeros in the filename which causes it to sort before the other vhost files and Apache's Include to load it first.

/etc/apache2/other/vhosts-000default.conf:

NameVirtualHost *:80
<VirtualHost *:80>
  ServerName _default_
  <Directory /Library/WebServer/Documents>
    AllowOverride All
  </Directory>
</VirtualHost>

Let's use something a bit better than "It works!" as our default page.

/Library/WebServer/Documents/index.html:

<!DOCTYPE html>
<html>
  <head>
    <title></title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  </head>
  <body>
    <p>Move along, nothing to see here&hellip;</p>
  </body>
</html>

We don't want it to be cacheable just in case we screw up and end up serving it instead of a vhost.

/Library/WebServer/Documents/.htaccess:

# Expire default page immediately
ExpiresActive On
ExpiresByType text/html "access"

And finally here's my template for all vhosts which we'll be using a bit later:

/etc/apache2/other/vhosts-example.conf.template:

<VirtualHost *:80>

  ServerName example.com
  ServerAlias www.example.com

  # no-www
  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
  RewriteRule ^(.*)$ http://%1$1 [R=301,L]

  ServerAdmin [email protected]
  DocumentRoot /Users/example/apps/example/production/current/public/

  LogLevel warn
  CustomLog /var/log/apache2/example-production-access.log "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{Host}i\" %D"
  ErrorLog /var/log/apache2/example-production-error.log

  <Directory />
    AllowOverride FileInfo Indexes Options
    Options -Indexes FollowSymLinks -MultiViews
    Order allow,deny
    Allow from all
  </Directory>

</VirtualHost>

To check the configuration syntax and then reload Apache so new vhosts are available, run the following:

sudo apachectl configtest && sudo apachectl graceful

`apache2ctl` errors

If apachectl outputs an error like /usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument, you've ran into a Mac OS 10.6.6 regression. You can patch apachectl by running the following:

sudo /usr/bin/sed -E -i bak 's/^(ULIMIT_MAX_FILES=".*)`ulimit -H -n`(")$/\11024\2/' /usr/sbin/apachectl

Passenger Preference Pane

A quick note on Passenger Preference Pane: I don't recommend you install it. It can be handy in development environments with Passenger for quickly spinning up a new vhost for an app. It does not however allow you to customise vhost settings like logging nor setup a default vhost.

It's a much better idea to create a template and then script the deployment of new applications in production environments. There's more to deploying an app than just creating a new vhost. We also need to create user accounts, databases, configure logging, etc.

Log rotation

Mac OS uses newsyslog to rotate the main log files such as system.log and mail.log. It does not however automatically rotate anything in /var/log/apache2. We could point newsyslog at each log file we want to rotate but logrotate lets us use wildcards.

brew install logrotate
sudo mkdir -p /etc/logrotate.d
sudo bash -c 'cat > /etc/logrotate.conf' <<EOF
compresscmd $(which gzip)
tabooext + template
include /etc/logrotate.d
EOF

Set your Apache log rotate settings. I prefer to rotate my logs weekly and keep 520 rotations (10 years). Disk space is cheap and old logs might be useful. Save the following config as /etc/logrotate.d/apache.conf:

/var/log/apache2/access_log /var/log/apache2/error_log /var/log/apache2/*.log {
  weekly
  missingok
  rotate 520
  compress
  delaycompress
  notifempty
  create 640 root wheel
  sharedscripts
  postrotate
    apachectl graceful
  endscript
}

Since we're going to be reloading Apache after rotating the logs, we can use this opportunity to rotate production.log from our Rails apps. These logs are larger so I only keep 10 rotations. Save the following template as /etc/logrotate.d/vhosts-example.conf.template:

/Users/example/apps/example/production/shared/log/production.log {
  weekly
  missingok
  rotate 10
  compress
  delaycompress
  notifempty
  create 640 example staff
  sharedscripts
}

Setup logrotate to run automatically via launchd. Debian runs logrotate daily at 06:25 which sounds fine by me. See man 5 launchd.plist for details on the StartCalendarInterval option. Save the following as /Library/LaunchDaemons/logrotate.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>logrotate</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/sbin/logrotate</string>
    <string>/etc/logrotate.conf</string>
  </array>
  <key>Disabled</key>
  <false/>
  <key>RunAtLoad</key>
  <false/>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Hour</key>
    <integer>6</integer>
    <key>Minute</key>
    <integer>25</integer>
  </dict>
</dict>
</plist>

And finally run the following to force an initial test rotation and then activate the launchd schedule:

sudo /usr/local/sbin/logrotate -f /etc/logrotate.conf
sudo launchctl load -w /Library/LaunchDaemons/logrotate.plist

User accounts

We want to isolate our services (PostgreSQL, Memcached, etc) and applications in their own user accounts. Unfortunately Mac OS doesn't provide a nice and simple adduser like command but we can make our own. Save the following as /usr/local/bin/adduser and run chmod +x /usr/local/bin/adduser to make it executable:

#!/bin/bash -e
NEW_USERNAME="$1"
if [ -z "$NEW_USERNAME" ]
then
  echo "Usage: $(basename "$0") [username]" >&2
  exit 1
fi
if id "$NEW_USERNAME" 2> /dev/null
then
  echo "$(basename "$0"): User \"$NEW_USERNAME\" already exists." >&2
  exit 1
fi

NEW_UID=$(( $(dscl . -list /Users UniqueID | awk '{print $2}' | sort -n | tail -1) + 1 ))

if ! [ $NEW_UID -gt 500 ]
then
  echo "$(basename "$0"): Could not determine new UID." >&2
  exit 1
fi

dscl . create "/Users/$NEW_USERNAME"
dscl . create "/Users/$NEW_USERNAME" UniqueID $NEW_UID
dscl . create "/Users/$NEW_USERNAME" PrimaryGroupID 20
dscl . delete "/Users/$NEW_USERNAME" AuthenticationAuthority
dscl . create "/Users/$NEW_USERNAME" Password '*'
dscl . create "/Users/$NEW_USERNAME" UserShell /bin/bash
dscl . create "/Users/$NEW_USERNAME" NFSHomeDirectory "/Users/$NEW_USERNAME"
createhomedir -c -u "$NEW_USERNAME"

Now we can simply run sudo adduser foo to create a new account which will not show on the login screen.

Removing a user account later is much easier than creating one. Just run the following:

sudo dscl . delete /Users/foo
sudo rm -rf /Users/foo

Email

Sometimes cron jobs fail. Wouldn't it be nice to hear about it? It's fairly easy to setup postfix to send mail via an external server. You could use Gmail, SendGrid or even your own mail server.

Configure postfix to forward mail to smtp.example.com with SSL and authentication. Replace both instances of smtp.example.com with your SMTP server's hostname and username:password with the username and password for your SMTP account.

sudo bash -c 'umask 0077 > /dev/null && echo "smtp.example.com username:password" >> /etc/postfix/smtp_auth'
sudo postmap hash:/etc/postfix/smtp_auth
sudo postconf -e relayhost=smtp.example.com:submission smtp_use_tls=yes smtp_sasl_auth_enable=yes smtp_sasl_password_maps=hash:/etc/postfix/smtp_auth tls_random_source=dev:/dev/urandom smtp_sasl_security_options=noanonymous

Forward root's mail to an external account. Replace [email protected] with your email address.

sudo cp -ai /etc/postfix/aliases{,.bak} # backup the original aliases file
sudo /usr/bin/sed -i '' 's/^#root.*/root: [email protected]/' /etc/postfix/aliases
grep ^root /etc/aliases # check the replacement worked
sudo postalias /etc/aliases

Set postfix to listen on localhost only. There's no need to give spam zombies the time of day.

sudo postconf -e inet_interfaces=localhost

By default postfix will only start when there's mail in the local queue (typically from calls to sendmail). We'll use our own launchd config which will run postfix all the time. Save the following as /Library/LaunchDaemons/org.postfix.master.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.postfix.master</string>
  <key>Program</key>
  <string>/usr/libexec/postfix/master</string>
  <key>ProgramArguments</key>
  <array>
    <string>master</string>
  </array>
  <key>AbandonProcessGroup</key>
  <true/>
  <key>RunAtLoad</key>
  <true />
  <key>KeepAlive</key>
  <true />
</dict>
</plist>

Start the postfix daemon and check port 25 is now open:

nc -4zv localhost 25 # should error
sudo launchctl unload -w /System/Library/LaunchDaemons/org.postfix.master.plist # stop built in on demand daemon
sudo launchctl load -w /Library/LaunchDaemons/org.postfix.master.plist # load our always running daemon
nc -4zv localhost 25 # should succeed

Forward your email to root (which will forward to your designated email address):

echo root > ~/.forward

Send yourself a test email:

date | mail -s "Test from $(hostname -s)" $USER

PostgreSQL

brew install postgresql

The instructions included in the caveats blurb from Homebrew (which after you run brew install postgresql) are great for setting up a single user development install. However, we want to run a system-wide instance to be used by all our applications. Run the following instead to setup PostgreSQL with ident authentication under the postgres account.

# create user account
sudo adduser postgres
# initialize the database cluster
sudo -u postgres initdb -A ident /usr/local/var/postgres
# set PostgreSQL to run at startup
sudo cp /usr/local/Cellar/postgresql/9.0.1/org.postgresql.postgres.plist /Library/LaunchDaemons/
sudo defaults write /Library/LaunchDaemons/org.postgresql.postgres UserName postgres
sudo plutil -convert xml1 /Library/LaunchDaemons/org.postgresql.postgres.plist
sudo chmod 644 /Library/LaunchDaemons/org.postgresql.postgres.plist
# start the server
sudo launchctl load -w /Library/LaunchDaemons/org.postgresql.postgres.plist

Add yourself as a superuser on the cluster so you can manage it without sudo -u postgres:

sudo -u postgres createuser -s $USER
createdb

The default PostgreSQL memory settings are very conservative. On a production machine you'll want to adjust these to suit your workload. Run sudo cp -ai /usr/local/var/postgres/postgresql.conf{,.org} before editing the config so you have a pristine copy of the config file to reference later.

Below are my current settings for my server which will give you an idea of what settings to play with. See Resource Consumption in the PostgreSQL docs for what each setting means. I recommend you start small and nothing beats trial, error and lots of testing. Don't forget to benchmark with production queries against production data.

shared_buffers = 256MB
work_mem = 64MB
maintenance_work_mem = 128MB
effective_cache_size = 512MB
max_connections = 50

If you're adjusting the shared_buffers setting you will probably run into the following error (in /var/log/messages):

FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=276275200, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMMAX.  To reduce the request size (currently 276275200 bytes), reduce PostgreSQL's shared_buffers parameter (currently 32768) and/or its max_connections parameter (currently 24).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.

The fix is easy. First make sure the quoted request size sounds sane (263 MB in the above error) and then update SHMMAX to be at least as large. I generally to round up to the next power of 2. SHMALL should generally be set to ceil(SHMMAX/PAGE_SIZE) where PAGE_SIZE is 4096 bytes. I'm setting SHMMAX to 512 MB:

sudo sysctl -w kern.sysv.shmmax=$((1048576 * 512))
sudo sysctl -w kern.sysv.shmall=$((1048576 / 4096 * 512))
sysctl -a | egrep '^kern.sysv.shm(max|all)' | /usr/bin/sed 's/: /=/' | sudo tee -a /etc/sysctl.conf

To restart the server run the following:

sudo launchctl unload -w /Library/LaunchDaemons/org.postgresql.postgres.plist
sudo launchctl load -w /Library/LaunchDaemons/org.postgresql.postgres.plist

Finally type psql and you should drop straight into a PostgreSQL prompt.

Granting permissions

Sometimes you may want to give full privileges on a database to a non-superuser account which is not the database owner. For example: you may want to share a database between two apps or let developers play around with data on a staging instance.

To grant full access to a database to a non-superuser you can use the following in a superuser psql prompt:

GRANT ALL ON DATABASE $database TO $user;
\c $database
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO $user;
GRANT ALL ON ALL TABLES IN SCHEMA public TO $user;

Memcached

# install memcached
brew install memcached
# configure to run at startup
sudo adduser memcache
sudo cp /usr/local/Cellar/memcached/1.4.5/com.danga.memcached.plist /Library/LaunchDaemons/
sudo defaults write /Library/LaunchDaemons/com.danga.memcached UserName memcache
sudo plutil -convert xml1 /Library/LaunchDaemons/com.danga.memcached.plist
sudo chmod 644 /Library/LaunchDaemons/com.danga.memcached.plist
# start the service
sudo launchctl load -w /Library/LaunchDaemons/com.danga.memcached.plist

Memcached defaults to a maximum cache size of 64 MB. You can increase this if needed with by adding -m, 128 to ProgramArguments in the launchd plist and restarting the service.

Running echo stats | nc localhost 11211 should give you memcached stats.

Sharing the cache with multiple applications & security issues

If you're configuring multiple applications to use it, make sure you namespace your keys with something like:

config.cache_store = :mem_cache_store, { :namespace => Rails.application.config.database_configuration[Rails.env]['database'] }

If you have untrusted users/applications, you'll probably want to setup multiple instances with SASL authentication.

ImageMagick

If your applications resize images with RMagick, you're going to need the ImageMagick libraries. Installing with --disable-openmp fixes some random crashing issues I was having.

brew install imagemagick --disable-openmp

Tomcat

To run Solr one needs a servlet container. Tomcat is a safe bet here. I recommend the excellent Sunspot library for using Solr in Rails.

Installing

Install Tomcat via Homebrew and then unlink it. Tomcat comes with a number of scripts which have generic names (startup.sh, version.sh, etc). We don't want those in our PATH.

brew install tomcat
brew unlink tomcat

Setup Tomcat to run in its own path (/usr/local/tomcat) under its own username (tomcat):

sudo adduser tomcat
sudo mkdir /usr/local/tomcat
sudo chown tomcat /usr/local/tomcat
sudo -u tomcat ln -s $(brew --prefix tomcat)/libexec /usr/local/tomcat/
sudo -u tomcat ln -s libexec/{bin,lib} /usr/local/tomcat
sudo -u tomcat mkdir /usr/local/tomcat/{logs,temp,webapps,work}
sudo rsync --archive --no-perms --chmod='ugo=rwX' /usr/local/tomcat/{libexec/conf/,conf}
sudo find /usr/local/tomcat/conf -exec chown tomcat:staff {} \;
sudo -u tomcat bash -c 'echo "org.apache.solr.level = WARNING" >> /usr/local/tomcat/conf/logging.properties'

Configure connectors

I recommend editing /usr/local/tomcat/conf/server.xml and replacing all <Connector /> entries with a single HTTP connector for localhost:8080:

<Connector address="127.0.0.1" port="8080" protocol="HTTP/1.1" connectionTimeout="20000" />

Run at startup

Save the following as /Library/LaunchDaemons/org.apache.tomcat.plist to have launchd start Tomcat automatically:

Note: Adjust -Xmx2048M to control how much memory Tomcat can use.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.apache.tomcat</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/tomcat/bin/catalina.sh</string>
    <string>run</string>
  </array>
  <key>UserName</key>
  <string>tomcat</string>
  <key>EnvironmentVariables</key>
  <dict>
    <key>CATALINA_HOME</key>
    <string>/usr/local/tomcat</string>
    <key>JAVA_OPTS</key>
    <string>-Djava.awt.headless=true -Xmx2048M</string>
  </dict>
  <key>Disabled</key>
  <false/>
  <key>RunAtLoad</key>
  <true/>
  <key>HopefullyExitsFirst</key>
  <true/>
  <key>ExitTimeOut</key>
  <integer>60</integer>
</dict>
</plist>

Finally, start Tomcat with the following:

sudo launchctl load -w /Library/LaunchDaemons/org.apache.tomcat.plist

To upgrade Tomcat to a newer version in the future, see my Upgrading Tomcat with Homebrew post.

Git Hosting

Run the following as your admin user on the server to install gitolite:

# create git user account
sudo adduser git
# copy our admin public key over to the git account
sudo cp ~/.ssh/id_rsa.pub ~git/$USER.pub
sudo chown git ~git/$USER.pub
# switch to git user and configure shell
sudo -u git -i
curl -sL http://github.com/jasoncodes/dotfiles/raw/master/install.sh | bash
exec bash -i # reload the shell
# clone gitolite source code
git clone git://github.com/sitaramc/gitolite gitolite-source
# install gitolite
cd gitolite-source
mkdir -p ~/bin ~/share/gitolite/conf ~/share/gitolite/hooks
src/gl-system-install ~/bin ~/share/gitolite/conf ~/share/gitolite/hooks
cd
# configure gitolite with ourselves as the admin
gl-setup $SUDO_USER.pub
# cleanup
rm $SUDO_USER.pub
exit

From your workstation you can then clone the config repo by running:

git clone git@$SERVER:gitolite-admin.git $SERVER-gitolite-admin

The documentation should contain everything you need. If you're new you'll want to read gitolite.conf for permission config and migrate if you're moving from Gitosis.

SSH on an alternate port

I want SSH to be available on both IPv4 and IPv6 on an alternate secondary port. I could use ipfw add 01000 fwd 127.0.0.1,22 tcp from any to me 4242 for IPv4 but ip6fw doesn't support forwarding. We can however just have launchd listen for SSH connections on an alternate port by running the following:

Note: Replace 4242 with your desired alternate port number.

sudo cp /System/Library/LaunchDaemons/ssh.plist /Library/LaunchDaemons/ssh-alt.plist
sudo defaults write /Library/LaunchDaemons/ssh-alt Label com.openssh.sshd-alt
sudo defaults write /Library/LaunchDaemons/ssh-alt Sockets '{ Listeners = { SockServiceName = 4242; }; }'
sudo plutil -convert xml1 /Library/LaunchDaemons/ssh-alt.plist
sudo chmod 644 /Library/LaunchDaemons/ssh-alt.plist
sudo launchctl load -w /Library/LaunchDaemons/ssh-alt.plist

Now with this port opened on my firewall I can use the following in my ~/.ssh/config to easily connect to my server with ssh server:

Host server
  HostName server.example.com
  Port 4242

Monit

Monit is a great tool that lets you monitor processes and make sure they're still serving requests. launchd handles restarting of failed services for us automatically but processes could still hang. This is where monit comes into the picture.

Check out the documentation for what can be monitored. Resources you can monitor include load average, system memory usage, disk space, process memory usage and connectivity.

First thing is to install monit. You'll need at least 5.2.3 as earlier versions are prone to crash on Mac OS. If you're using my Homebrew fork, you're all good to go.

brew install monit

Create /etc/monitrc:

set daemon 30 with start delay 60
set mail-format { from: [email protected] }
set alert [email protected]
set mailserver localhost
set httpd port 2812 allow localhost

check system server
  if loadavg (1min) > 8 then alert
  if loadavg (5min) > 6 then alert

check filesystem rootfs with path /
  if space usage > 90 % then alert

check process apache2 with pidfile /var/run/httpd.pid
  if failed URL http://localhost:80/ with timeout 5 seconds then alert

check host postgresql with address 127.0.0.1
  if failed port 5432 with timeout 5 seconds then alert

check host tomcat with address 127.0.0.1
  if failed port 8080
    send "HEAD / HTTP/1.0\r\n\r\n"
    expect "HTTP/1.1"
    with timeout 5 seconds
    then alert

check host memcached with address 127.0.0.1
  if failed port 11211
    send "stats\n"
    expect "STAT pid"
    with timeout 5 seconds
    then alert

Save the following as /Library/LaunchDaemons/monit.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>UserName</key>
  <string>root</string>
  <key>Label</key>
  <string>monit</string>
  <key>OnDemand</key>
  <false/>
  <key>RunAtLoad</key>
  <true/>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/monit</string>
    <string>-c</string>
    <string>/etc/monitrc</string>
    <string>-I</string>
    <string>-l</string>
    <string>/var/log/monit.log</string>
  </array>
</dict>
</plist>

Secure the config file, check the syntax, and then start monit:

sudo chmod 600 /etc/monitrc
sudo monit -t /etc/monitrc
sudo launchctl load -w /Library/LaunchDaemons/monit.plist

If you kill -STOP or otherwise break a service and you should get an email letting you know. You can view the current status with sudo monit status.

Restarting failed services

A great feature of Monit is that it can run tasks for you when something bad happens. In the case of Tomcat, I have created a script which kills Tomcat and then relaunches it. See my Restarting Tomcat automatically on Mac OS with Monit post for details.

Backups

My current local backup solution consists of both Time Machine backups to a Time Capsule and a weekly startup disk image with Carbon Copy Cloner. A problem with both of these solutions though is that they can't quiesce database writes to allow atomic snapshots. Hopefully Apple's working on their own ZFS/brtfs alternative for Mac OS 10.7 Lion which will allow cheap copy-on-write snapshots.

Until Time Machine can take atomic snapshots and efficiently backup large changing files, we need to make database dumps which can be picked up by our backup system. This applies to both PostgreSQL databases and and other large and changing files such as our Solr indexes.

Update: To prevent Time Machine from backing up these files you can mark them as excluded either in the Time Machine preference pane or with xattr:

sudo xattr -w com.apple.metadata:com_apple_backup_excludeItem com.apple.backupd /usr/local/var/postgres # [...]

If you don't exclude these directories from Time Machine, you'll find that your backup increments will be large and you'll quickly lose your history as Time Machine starts pruning old backups to make room. The worst bit is copying those large files every hour will put a noticeable resource strain on your machine.

If you want to audit what Time Machine is backing up (highly recommended if you suspect your backups are larger than they should be), I've found the easiest way is TimeTracker from CharlesSoft (the makers of the handy Pacifist tool). I found I had to run it under sudo with the Time Machine backup mounted to avoid errors.

I have a backup script which I run daily. It archives PostgreSQL databases and Solr indexes locally and then rsyncs them along with my Git repos to an offsite VPS. Since we're archiving locally, we might as well send these same backups offsite. If your databases are large you might want to look into continuous archiving to create incremental backups. Be sure to perform full base backups regularly with any incremental setup (I recommend weekly).

The script makes use of my lib_exclusive_lock functions to prevent concurrent executions. With sufficiently large databases and a slow enough upstream this becomes a problem.

PostgreSQL databases are detected by querying the pg_database catalog. Solr instances and their paths are detected by looking for the solr/home environment variable in the Tomcat contexts. You'll need to brew install xmlstarlet for this to work.

Note: unlike the rest of this guide, this backup script assumes GNU tools are installed as the default.

To use the script, save it as ~root/bin/backup and add a crontab entry for root (sudo crontab -e) like 15 0 * * * ~/bin/backup. Adjust the configuration variables at the top to suit.

#!/bin/bash -e
set -o pipefail
export PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin

BWLIMIT=30
ARCHIVE_COUNT=5
REMOTE_HOST=[email protected]

# grab lock
cd "`dirname "$0"`"
source "lib_exclusive_lock.sh"
exclusive_lock_require

# prepare backup directory
umask 0077
mkdir -p ~/backups
cd ~/backups

# compact git repos
find ~git/repositories -maxdepth 1 -type d -name '*.git' | while read REPO
do
  (
    cd $REPO
    git gc --auto --quiet
  )
done

# backup PostgreSQL databases
mkdir -p postgres
su - postgres -lc "psql -q -A -t" <<SQL |
SELECT datname
FROM pg_database
WHERE datname NOT LIKE 'template%';
SQL
while read DB_NAME
do
  mkdir -p "postgres/${DB_NAME}"
  FILENAME="postgres/${DB_NAME}/${DB_NAME}_`date +%Y%m%d`.sql.bz2"
  nice su - postgres -lc "pg_dump --format=p $DB_NAME" | nice bzip2 > "${FILENAME}.new"
  mv "${FILENAME}"{.new,}
  find "postgres/${DB_NAME}" -maxdepth 1 -type f -name "${DB_NAME}_*.sql.*" | sort -r | tail -n +$((ARCHIVE_COUNT + 1)) | xargs -r rm
done

# backup Solr indexes
xml sel -t -v "Context/@path " -o " " -v "Context/Environment[@name='solr/home']/@value" /usr/local/tomcat/conf/Catalina/localhost/*.xml | while read CONTEXT_PATH SOLR_HOME
do
  if [ -n "$CONTEXT_PATH" -a -n "$SOLR_HOME" ]
  then
    DB_NAME="$(basename "$CONTEXT_PATH" | sed -e 's/-production-solr$//')"
    mkdir -p ~/"backups/solr/${DB_NAME}"
    FILENAME=~/"backups/solr/${DB_NAME}/${DB_NAME}_$(date +%Y%m%d).tar.bz2"
    rm -rf "$SOLR_HOME".bak
    cp -lr "$SOLR_HOME"{,.bak}
    (cd "${SOLR_HOME}.bak" && nice tar c .) | nice bzip2 > "${FILENAME}.new"
    rm -rf "$SOLR_HOME".bak
    mv "${FILENAME}"{.new,}
    find "solr/${DB_NAME}" -maxdepth 1 -type f -name "${DB_NAME}_*.tar.bz2" | sort -r | tail -n +$((ARCHIVE_COUNT + 1)) | xargs -r rm
  fi
done

# rsync with a retry. sometimes there's intermittent connectivity issues.
function do_rsync()
{
  if ! rsync --archive --delete-after --partial-dir=.partial --bwlimit $BWLIMIT "$@"
  then
    echo rsync failed: "$@"
    sleep 5m
    echo retrying...
    rsync --archive --delete-after --partial-dir=.partial --bwlimit $BWLIMIT "$@"
    echo retried.
  fi
}

# copy backups offsite

do_rsync ~git/repositories/ $REMOTE_HOST:~/backups/git/

find ~/backups/postgres/* -maxdepth 0 -type d | while read DIR
do
  do_rsync "$DIR" "$REMOTE_HOST:~/backups/postgres/"
done

find ~/backups/solr/* -maxdepth 0 -type d | while read DIR
do
  do_rsync "$DIR" "$REMOTE_HOST:~/backups/solr/"
done

Applications

Automating application deployment with Capistrano

The Capistrano Wiki covers the basics on how to setup Capistrano. There are a few gotchas to watch out for however.

The documentation for RVM and Bundler cover setting up Capistrano support pretty well. Other than requiring the right files, the only other thing you should need to do is disable sudo with set :use_sudo, false.

For the remote cache to work (set :deploy_via, :remote_cache), you'll need to enable SSH agent forwarding with ssh_options[:forward_agent] = true. This allows the app user on the server to temporarily use your SSH key to authenticate to your Git repository when deploying.

Here's a complete Capistrano recipe (config/deploy.rb):

require 'bundler/capistrano'

$:.unshift(File.expand_path('./lib', ENV['rvm_path']))
require 'rvm/capistrano'

set :application, 'example' # change me
set :server_name, 'server' # change me
set :user, 'example' # change me
set(:deploy_to) { "~/apps/#{application}/#{stage}" }
set :keep_releases, 5
set(:releases) { capture("ls -x #{releases_path}").split.sort }

set :scm, :git
set :repository, "git@#{server_name}:#{application}.git"
set :branch, "master"
set :deploy_via, :remote_cache

ssh_options[:forward_agent] = true
set :use_sudo, false

role :web, server_name
role :app, server_name
role :db, server_name, :primary => true

namespace :deploy do
  task :start do
  end

  task :stop do
  end

  task :restart do
    run "touch #{current_path}/tmp/restart.txt"
  end
end

before "deploy:symlink", "deploy:migrate"
after "deploy:update", "deploy:cleanup"

Setting up the application environment

With your application configured to deploy via Capistrano, you can prepare the new application environment (user account, database, vhost) with the following script. Save a copy as /usr/local/bin/createapp and chmod +x it:

#!/bin/bash -e
APPNAME=${1:?Specify application name}
DOMAIN=${2:-${APPNAME}.com}

# create user account
sudo adduser $APPNAME
# copy over admin public key
sudo -u $APPNAME -i bash -c 'umask 0077 > /dev/null && mkdir -p ~/.ssh/ && cat >> ~/.ssh/authorized_keys' < ~/.ssh/authorized_keys
# seed SSH known hosts with git server details
sudo -u $APPNAME -i bash -c 'umask 0077 > /dev/null && mkdir -p ~/.ssh/ && echo "$(hostname -s) $(cat /etc/ssh_host_rsa_key.pub)" >> ~/.ssh/known_hosts'
# install dotfiles
sudo -u $APPNAME -i bash < <( curl -sL http://github.com/jasoncodes/dotfiles/raw/master/install.sh )
# forward mail to root
echo root | sudo -u $APPNAME -i bash -c 'cat > ~/.forward'

# create PostgreSQL database
createuser -SDR $APPNAME
createdb -O $APPNAME ${APPNAME}_production

# setup vhost
sudo cp /etc/apache2/other/vhosts-example.conf.template /etc/apache2/other/vhosts-${APPNAME}-production.conf
sudo /usr/bin/sed -i '' -e s/example\.com/${DOMAIN}/g -e s/example/${APPNAME}/g /etc/apache2/other/vhosts-${APPNAME}-production.conf
sudo -e /etc/apache2/other/vhosts-${APPNAME}-production.conf # check config, make any custom tweaks
sudo apachectl configtest && sudo apachectl graceful

# setup log rotation
sudo cp /etc/logrotate.d/vhosts-example.conf.template /etc/logrotate.d/vhosts-${APPNAME}-production.conf
sudo /usr/bin/sed -i '' s/example/${APPNAME}/g /etc/logrotate.d/vhosts-${APPNAME}-production.conf

Deploying your application

From within the app directory on your workstation you can then run the following:

# prepare application directories for deployment
cap deploy:setup
# deploy the application
cap deploy

Clean Google Analytics tracking data from your URLs

2011-01-09T00:00:00+10:00

FeedBurner has a feature which integrates with Google Analytics to tell you how many of your hits come via feed readers. This is rather useful because feed readers typically don't pass any useful referrer information (especially desktop clients). This tracking can be enabled by ticking the "Track clicks as a traffic source in Google Analytics" option in your FeedBurner account.

However, this comes with a disadvantage to those of us who care about clean URLs. I'm sure I'm not the only one who dislikes seeing otherwise clean URLs polluted with Google Analytics tracking data:

http://example.com/posts/foo-bar?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+example

You may even add utm_source=twitter to short URLs when you post to Twitter in addition to tracking your feeds. Either way, a big problem with this is that these URLs are often shared around and not everybody takes the time to remove all the cruft and share the clean canonical URL.

Luckily Google Analytics asynchronous tracking provides a nice little API that lets us queue a function to be ran after the tracking request has been sent. This combined with the history.replaceState method in HTML5 lets us remove the the tracking data from the URL without reloading the page or breaking the browser's history. Another win for modern browsers.

The additional code required to make this happen is one extra statement to be added to your Google Analytics tracking JavaScript:

var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-XXXXX-X']);
_gaq.push(['_trackPageview']);
_gaq.push(function() {
  var newPath = location.pathname + location.search.replace(/[?&]utm_[^?&]+/g, "").replace(/^&/, "?") + location.hash;
  if (history.replaceState) history.replaceState(null, '', newPath);
});

(function() {
  var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
  ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
  var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();

Update 2011-01-26: I originally called replaceState with just location.pathname but that resulted in the removal of anchors within a page (e.g. links to comments). The code has been updated to location.pathname + location.hash to keep anchors. e.g. /foo/bar?utm_medium=example#comment-42 will now be replaced with /foo/bar#comment-42.

Update 2013-01-07: Thanks Henrik Nyh for a patch to preserve parameters other than those used by Google Analytics.

Ruby 1.9.2 encoding issues with Rails 2.3.10

2010-12-30T00:00:00+10:00

While switching my app over from Ruby Enterprise Edition 1.8.7 to Ruby 1.9.2p0, I ran into a few issues with content encodings. The vast majority of these issues could be solved by placing # -*- coding: utf-8 -*- at the top of the source file. A couple of the problems I ran into ended up being a little more complex.

UTF-8 form parameters

The first problem was the params hash was not arriving to my controllers encoding as UTF-8 but instead as ASCII-8BIT. This is not a problem if all your inputs are ASCII-7 but that wasn't so for me. With content such as café I was getting an exception later on during the request that had become all too familiar to me: incompatible character encodings: ASCII-8BIT and UTF-8.

Rails 3 solves this very nicely by doing a number of things including interpreting params as UTF-8 and adding workarounds for Internet Explorer. I'll leave the workarounds and accept-charset="UTF-8" form attributes as an exercise for the reader.

We can force Rails to interpret all string parameters as UTF-8 with a small patch. Save the following as config/initializers/utf8_params.rb:

raise "Check if this is still needed on " + Rails.version unless Rails.version == '2.3.10'

class ActionController::Base

  def force_utf8_params
    traverse = lambda do |object, block|
      if object.kind_of?(Hash)
        object.each_value { |o| traverse.call(o, block) }
      elsif object.kind_of?(Array)
        object.each { |o| traverse.call(o, block) }
      else
        block.call(object)
      end
      object
    end
    force_encoding = lambda do |o|
      o.force_encoding(Encoding::UTF_8) if o.respond_to?(:force_encoding)
    end
    traverse.call(params, force_encoding)
  end
  before_filter :force_utf8_params
  
end

ERB templates

I was having trouble getting the old # -*- coding: utf-8 -*- working in ERB and I was wondering if I'd have to start looking into other options. I'd get an exception any time I outputted a string containing UTF-8 (from params or the model). Luckily I came across the following code in action_view/template_handlers/erb.rb:

magic = $1 if template.source =~ /\A(<%#.*coding[:=]\s*(\S+)\s*-?%>)/
erb = "#{magic}<% __in_erb_template=true %>#{template.source}"

It looks like Rails does support specifying the encoding of ERB templates after all. All you need to do is to add the following to the top of your template:

<%# coding: utf-8 %>

The key part I was missing is to have no whitespace between the <% and #. Additionally, if this is in a plain text template for ActionMailer, you'll want to avoid any newlines between this block and your main content otherwise you'll have a blank line at the top of the email.

Environment variables

I ran into a couple of encoding errors that I was having trouble reproducing locally but were highly reproducible on production (running on Apache with Passenger). It even ran fine when I booted up a RAILS_ENV=production script/server on the production host. This had me stumped for a few minutes until I thought to look at my environment variables. A set | grep UTF revealed the following:

LANG=en_AU.UTF-8
LC_CTYPE=en_US.UTF-8

The bugs reproduced fine once I unset these in my local terminal session. Ruby 1.9 can use these environment variables to set the default encoding to something other than US-ASCII. Handy once you know about it. You may want to try running your tests with these unset to see if you're missing any encoding issues.

If you want Ruby to default to UTF-8 when loading files (i.e File.read) when these environment variables are not set, add the following to an initializer:

Encoding.default_external = 'UTF-8'

Using Sass on Heroku with Hassle

2010-12-12T00:00:00+10:00

A little while ago Lucas Willett and I hacked together Compass, Rails 3 and Heroku without a Hassle. This is a combination of three components:

Render compiled CSS to tmp/stylesheets/ instead of public/stylesheets/ as the file system is read only on Heroku.
Use Rack::Static to mount tmp/stylesheets on /stylesheets so they're accessible via their original URL.
Monkey patch ActionView::Helpers::AssetTagHelper to check tmp/stylesheets/ for stylesheets in addition to the default of public/stylesheets/. This is to ensure cache busting continues to function and you don't unintentionally serve old stylesheets to your users.

The whole reason this workaround came about was that we had some trouble in getting Hassle to work. The problem was around initialisation which wasn't hard to solve in retrospect but I had the hair-brained idea to write the quick workaround above. It worked. The cache busting was an added bonus (of which I would have needed anyway).

One problem though is Rack::Static does not fall-through if it doesn't find a file. As a result, this solution does not work if you have existing files in public/stylesheets/ you want to serve alongside your Sass stylesheets. Luckily for us this wasn't a problem.

However now I am wanting to use this same setup on an existing project which has a mix of both CSS files and Sass stylesheets. The time for the quick hack is over. I have forked Hassle and added a couple of features from our workaround:

Always run Hassle even when not on Heroku. I didn't want to have public/ polluted with generated files so I set Hassle to run all the time. This keeps public/stylesheets/ clean in development mode.
Fix cache busting on Hassle stylesheets. The monkey patch to ensure Sass stylesheets correctly refresh when you redeploy.

Update: I have also made a couple of tweaks to the HTTP response headers to improve the cacheability of the generated stylesheets. By adding a Last-Modified header to the response, browsers can use a If-Modified-Since header in their request to get a much smaller 304 response if nothing has changed. Ideally browsers will just use the existing max-age, but this doesn't happen often enough.

If you'd like to use my fork, add the following to your Gemfile:

gem 'hassle', :git => 'git://github.com/jasoncodes/hassle.git'

We use Hassle in combination with Compass and it's working great.

A free SSL certificate for your web server

2010-12-11T00:00:00+10:00

Update 2017-11-21: StartCom will soon be no longer be issuing certificates. I recommend you look into Let's Encrypt with the Dehydrated client.

No more self-signed certificates

Typically for a low volume site where verified identity is not import one would use a self-signed certificate for SSL. Unfortunately these triggers security warnings in browsers and require you to recognise/remember checksums to prevent man-in-the-middle attacks when accessing your own servers over HTTPS.

StartSSL offers free domain verified SSL server certificates which work pretty much everywhere. The only real downside is they have a 1 year expiry. This means you need to create a new one every year rather than say a 10 year self-signed certificate which will probably last the life of your server.

These free certificates are also chained which means configuration can be a little tricky as there are a few traps you can fall into. If you don't setup the chain correctly on your server you can run into compatibility issues with clients that may not be immediately obvious. These include either forgetting to setup the chain or creating the links in the wrong order. With either of these errors I myself could still access the site fine.

And that's where this post comes in. I need to setup a couple of new certificates and this time I'm making notes on on the steps I'm taking, referencing my existing setup as I go. At worst I'll save some time next year when I have to renew these certificates. Hopefully you'll find this useful as well.

Setting it all up

I run Debian stable on my servers. At the time of writing this is Debian Lenny with Apache 2.2.9. Substitute example.com for your domain name where applicable.

Authenticating with StartSSL

Note: As of the time of writing, Chrome has some issues with SSL client certificates which will cause you problems. I recommend using Safari (or Firefox if that's your thing).

If this is your first time using StartSSL, you'll need to create an account. Click on Control Panel and then on Sign-up. Fill out all the details and you'll get an SSL client certificate which you use to authenticate with the website.

The client certificate expires after a year so you'll have to create a new one when it comes time to renew your server certificate. StartSSL will send you an email when both are coming up for renewal. To create a new client certificate, first reverify your email address under Validations Wizard: Email Address Validation and then create a new certificate under Certificates Wizard

Requesting a server certificate

Validations Wizard: Domain Name Validation

Certificates Wizard: Web Server SSL/TLS Certificate

openssl req -new -newkey rsa:4096 -days 365 -nodes -keyout example.com.key -out example.com.csr

Pick the CSR option when prompted and upload the contents of example.com.csr. You will also be prompted for a hostname underneath your domain. I run a no-www shop so I used my server's hostname (host.example.com). If you want to run www.example.com, enter www here.

As of this point the .csr file is no longer required and can be removed. Alternatively you could generate a CSR with a longer expiry and reuse it next year.

And now we wait for certificate to be issued. This usually happens within the half hour. When you receive the certificate signing confirmation email, download the following certificates:

Toolbox > Retrieve Certificate: You will see your newly created certificate. Save it as example.com.crt.
Toolbox > StartCom CA Certificates: Download "StartCom Root CA (PEM encoded)" (ca.pem)
Toolbox > StartCom CA Certificates: Download "Class 1 Intermediate Server CA" (sub.class1.server.sha2.ca.pem)

Copy the .crt, .key and .pem files to /etc/apache2/ssl on your server.

Configuring Apache

Run the following commands as root:

cd /etc/apache2/ssl
mv ca.pem startssl.ca.crt
mv sub.class1.server.sha2.ca.pem startssl.sub.class1.server.sha2.ca.crt
cat startssl.sub.class1.server.sha2.ca.crt startssl.ca.crt > startssl.chain.class1.server.crt
cat example.com.{key,crt} startssl.chain.class1.server.crt > example.com.pem
ln -sf example.com.pem apache.pem
chown root:ssl *.crt *.key *.pem
chmod 640 *.key *.pem

Edit /etc/apache2/sites-available/ssl and add the following within the <VirtualHost> block:

SSLEngine On
SSLCertificateFile /etc/apache2/ssl/example.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/example.com.key
SSLCertificateChainFile /etc/apache2/ssl/startssl.chain.class1.server.crt

At this point you'll want to configure the rest of Apache for SSL if you haven't already.

Check that your Apache config parses as valid:

apache2ctl -t

And then restart Apache with the new config:

/etc/init.d/apache2 reload

Verifying everything worked

Run the following after restarting Apache to check the certificate chain:

echo HEAD / | openssl s_client -connect localhost:443 -quiet > /dev/null

You should see something like:

depth=2 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0

A depth of 2 and a return value of 0 is good. If the certificate chain is wrong, you'll probably see something like:

depth=0 /description=12345-ABCDEF123456/C=XX/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=host.example.com/[email protected]
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /description=12345-ABCDEF123456/C=XX/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=host.example.com/[email protected]
verify error:num=27:certificate not trusted
verify return:1
depth=0 /description=12345-ABCDEF123456/C=XX/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=host.example.com/[email protected]
verify error:num=21:unable to verify the first certificate
verify return:1

Hosting email services over SSL

I have one host setup to send and receive email using Exim 4 for SMTP and Courier for IMAP. To have these services use your new SSL certificate, point them to your existing certificate files:

ln -sf /etc/apache2/ssl/example.com.pem /etc/courier/imapd.pem
ln -sf /etc/apache2/ssl/example.com.pem /etc/courier/pop3d.pem

ln -sf /etc/apache2/ssl/example.com.pem /etc/exim4/exim.pem
ln -sf /etc/apache2/ssl/example.com.crt /etc/exim4/exim.crt
ln -sf /etc/apache2/ssl/example.com.key /etc/exim4/exim.key

Outlook and Outlook Express are both fairly broken but fortunately I found some workarounds. You'll want to set the following config option for Exim (in /etc/exim4/conf.d/main/03_exim4-config_tlsoptions if you are using split configuration on Debian):

# Outlook Express really sucks
# See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=482012>
MAIN_TLS_TRY_VERIFY_HOSTS=''

MAIN_TLS_CERTKEY = CONFDIR/exim.pem

Outlook does not work properly with the altname in the certificate. You may have to use the servers hostname (host.example.com) rather than the domain name (example.com).

`gup`: A friendlier `git pull --rebase`

2010-11-09T00:00:00+10:00

By now most git users would have heard about rebasing your local commits on top of the remote branch HEAD before you git push them rather than merging to prevent the proliferation of useless same branch merge commits like "Merge remote branch 'origin/topic' into topic".

If one is using git pull, the rebasing can be accomplished by using git pull --rebase instead. This essentially changes git pull from doing git fetch && git merge $TRACKING_BRANCH to git fetch && git rebase $TRACKING_BRANCH.

There's still the inconvenience of having to stash any uncommitted changes before a rebase. If you don't, you'll get messages like "refusing to pull with rebase: your working tree is not up-to-date". This results in a fetch, stash, rebase, pop dance which gets tiring. I think we can do better.

Update 2011-01-11: Another thing to watch out for when using git pull --rebase is merge commits. You cannot preserve merges when rebasing using git pull as it does not let you pass in the --preserve-merges option. This means you could end up losing valuable merge commits. Glen Maddern has a great post on Rebasing Merge Commits in Git over on the Envato Notes blog which covers this in more detail. The good news is that my gup script already handles rebasing merge commits by passing the -p (--preserve-merges) option to git rebase.

Update 2011-09-16: My gup function has had a number of tweaks since I first posted. I removed the quiet flag from git stash pop as late versions of git seem to silence the error when pop fails. The other significant change is that gup will now explicitly fast-forward if it can rather than rebasing. The rest of the changes are minor (e.g. refactoring for style).

Update 2011-09-16: I now prefer git-up when available as it has nicer output and it also has an option to show if one needs to bundle. I still use the gup function as it's handy on foreign systems where I don't have my normal Ruby setup and I like the command name better :).

function gup
{
  # subshell for `set -e` and `trap`
  (
    set -e # fail immediately if there's a problem

    # use `git-up` if installed
    if type git-up > /dev/null 2>&1
    then
      exec git-up
    fi

    # fetch upstream changes
    git fetch

    BRANCH=$(git symbolic-ref -q HEAD)
    BRANCH=${BRANCH##refs/heads/}
    BRANCH=${BRANCH:-HEAD}

    if [ -z "$(git config branch.$BRANCH.remote)" -o -z "$(git config branch.$BRANCH.merge)" ]
    then
      echo "\"$BRANCH\" is not a tracking branch." >&2
      exit 1
    fi

    # create a temp file for capturing command output
    TEMPFILE="`mktemp -t gup.XXXXXX`"
    trap '{ rm -f "$TEMPFILE"; }' EXIT

    # if we're behind upstream, we need to update
    if git status | grep "# Your branch" > "$TEMPFILE"
    then

      # extract tracking branch from message
      UPSTREAM=$(cat "$TEMPFILE" | cut -d "'" -f 2)
      if [ -z "$UPSTREAM" ]
      then
        echo Could not detect upstream branch >&2
        exit 1
      fi

      # can we fast-forward?
      CAN_FF=1
      grep -q "can be fast-forwarded" "$TEMPFILE" || CAN_FF=0

      # stash any uncommitted changes
      git stash | tee "$TEMPFILE"
      [ "${PIPESTATUS[0]}" -eq 0 ] || exit 1

      # take note if anything was stashed
      HAVE_STASH=0
      grep -q "No local changes" "$TEMPFILE" || HAVE_STASH=1

      if [ "$CAN_FF" -ne 0 ]
      then
        # if nothing has changed locally, just fast foward.
        git merge --ff "$UPSTREAM"
      else
        # rebase our changes on top of upstream, but keep any merges
        git rebase -p "$UPSTREAM"
      fi

      # restore any stashed changes
      if [ "$HAVE_STASH" -ne 0 ]
      then
        git stash pop
      fi

    fi

  )
}

Throw the following into your shell's startup script. I keep the script in my dotfiles as it's much easier to bring it along to new machines. Alternatively you could remove the function wrapper and save it as a standalone script in ~/bin.

Once setup you can pull changes for the current tracking branch, rebase any unpushed commits on top of any new ones from upstream, all while preserving anything you have uncommitted (via git stash) with a single command: gup.

Jason Codes

Installing Exim on Mac OS X

Installing Exim

Creating a service user account

Homebrew

Installing Exim

404 Not Found {#404}

Configuring Exim

Local Hostname

Block external access

Postfix sendmail compatibility

Routers

Transports

Authentication

Forwarding local user accounts

Enabling IPv6 support

Syntax check config file

Running Exim on port 25

Replace Postfix sendmail with Exim

Log rotation

Testing

Nested resource URLs without controller names or numeric IDs in Rails 3

Project setup

Generate new Rails app

Create models

Controllers and initial routes

Replacing numeric IDs in URLs with slugs

Removing the controller names from URLs

Updating the specs

Updating the routes

A small problem

An easy solution

Upgrading Tomcat with Homebrew

Restarting Tomcat automatically on Mac OS with Monit

CSRF vulnerability in Ruby on Rails 2.3.10 & 3.0.3

The problem

The fix

Authlogic

Devise

Ajax requests (jQuery)

Avoiding authentication denial of service (DoS)

Hosting Rails apps on a Mac OS X server

Contents

Xcode

Homebrew

Sysadmin tweaks

GNU command-line utilities

dot files

htop

Ruby 1.9.2 via system-wide RVM

Install RVM

Install Ruby 1.9.2 and set it as default

Lockdown RVM installation

Fix Homebrew permissions broken by installing RVM system-wide

Install Bundler

Apache & Passenger

HTTP Compression

Virtual Hosts

apache2ctl errors

Passenger Preference Pane

Log rotation

User accounts

Email

PostgreSQL

Granting permissions

Memcached

Sharing the cache with multiple applications & security issues

ImageMagick

Tomcat

Installing

Configure connectors

Run at startup

Git Hosting

SSH on an alternate port

Monit

Restarting failed services

Backups

Applications

Automating application deployment with Capistrano

Setting up the application environment

Postfix `sendmail` compatibility

Replace Postfix `sendmail` with Exim

`apache2ctl` errors

`gup`: A friendlier `git pull --rebase`