Aaron Van Bokhoven - Software Developer

Request Hole, Testing Endpoints and Webhooks

Thu, 22 Jul 2021 00:00:00 GMT

I’ve been working on a few side projects where I needed to inspect webhooks and initially used RequestBin to do this. However, I had issues with their service being down a few times and had some concerns about privacy. The alternative CLI tools I found required language dependencies and were a bit of a hassle to set up.

This felt like a good excuse to make a CLI tool in Go, which would require no dependencies or versions of languages; just compile, or download, the executable for your operating system.

My initial goal for the tool was to expose an endpoint and print incoming requests to the terminal output and/or write to a log file. This was the first release of Request Hole. (Also known as rh, which is the CLI executable.)

Building the first release was relatively straight forward. It was working great and with the help of pterm I was able to make the terminal output look modern. I later realized it would be a bit more useful to have a web interface that could show me the incoming requests, in real time, similar to RequestBin.

This required a lot of refactoring as the architecture at that point didn’t support having two Goroutines (http servers in our case, one for the incoming requests and one to serve the web UI) running in parallel.

From the very beginning I wanted Request Hole to have an interface for outputs called renderers that could be implemented by any renderer we want to add. This could be a terminal printer, a log writer, or a web UI. To add the web UI, I had to change the way renderers were called, which at the time was synchronous, and decided to move them into individual Goroutines where they could be run in parallel.

By moving the renderers to Go routines, they needed a way to communicate with the incoming request endpoint so I utilized channels. I then moved all the renderer logic into a higher level function that manages all the Goroutines and channels.

After the big refactor, the next step was to start serving the web UI. I decided on create-react-app which provided mostly everything I needed to start building the front-end. Thankfully Go introduced the embed feature in 1.16 which made embedding and serving the front-end static files easy.

The next big decision was to use GraphQL instead of a standard REST API. Pushing incoming requests to the front-end through websockets was a requirement and I had read about GraphQL’s subscriptions, so going with GraphQL felt like a good solution, and a fun experiment. Although not the easiest to setup, compared to a traditional REST API, and facing some weird quirks and nuances, I did manage to get all the plumbing sorted out with gqlgen and Apollo.

One thing I learned was that mapping Go types to GraphQL types don’t always, and most likely won’t, work out. The solution to that is to create custom types using scalars and once I figured that out everything went swimmingly. After getting GraphQL sorted out all that was left to do was to create the UI. I decided on Tailwind so I didn’t have to write any CSS.

You can view the project at https://github.com/aaronvb/request_hole where the source and release files are available to download.

If you’re on a Mac, you can install it with brew install aaronvb/request_hole/rh.

Rails Inspired Logging in Go

Mon, 18 Jan 2021 00:00:00 GMT

One of the few things I miss from Ruby on Rails is the verbose logging the framework provides to you in development. If you’re not coming from Ruby on Rails I believe this will still be applicable and beneficial to your development in Go.

Before we jump into the details, I’d like to preface this by saying this guide to logging is meant to be used in the development environment. While I think logging should be done in production as well, the depth of logging should be limited in any production environment where we need to take security into account. (Side note: The param logging package I’ve released will automatically filter passwords.)

The Rails Log

Below is an example of a standard log output by Rails.

Started GET "/users?foo=bar" for ::1 at 2021-01-18 22:16:58 -1000
Processing by UsersController#index as HTML
  Parameters: {"foo"=>"bar"}
  User Load (0.2ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT ?  [["LIMIT", 1]]
  ↳ app/controllers/users_controller.rb:3:in `index'
Completed 200 OK in 4ms (Views: 0.7ms | ActiveRecord: 0.2ms | Allocations: 973)

Breaking it down we start with the method used in the request, followed by the path and a timestamp.

Started GET "/users?foo=bar" for ::1 at 2021-01-18 22:16:58 -1000

Next we have the parameters passed to the request.

Parameters: {"foo"=>"bar"}

After that we have any queries processed during the request and the line where the query was run.

User Load (0.2ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT ?  [["LIMIT", 1]]
  ↳ app/controllers/users_controller.rb:3:in `index'

And the last line is the end of the request that contains the response status and time to completion.

Completed 200 OK in 4ms (Views: 0.7ms | ActiveRecord: 0.2ms | Allocations: 973)

As you can see, all of these together provide very helpful details into every request.

The Go Log

Out of the box Go won’t give you that kind of logging for web requests.

Enter logrequest and logparams. I wrote these two packages to bring the above logging to your Go project. With one exception, the SQL query logging has to be handled manually, but no worries I’ll explain how I handle that at the end.

app_1      | Started GET "/" 172.19.0.1:64368 HTTP/1.1 at 2021-01-19 08:30:08
app_1      |    SELECT id, name, email, created, active FROM users WHERE id = $1 [8]
app_1      |    ↳ /app/pkg/models/postgres/users.go:75
app_1      | Completed 200 in 2.1124ms

Middleware

To get logrequest and logparams working, we’ll need to use it as middleware within the http request. If you’re not familiar with middleware, Alex Edwards has a great blog post covering it: https://www.alexedwards.net/blog/making-and-using-middleware.

And we’ll be using gorilla/mux to handle the middleware.

Using logrequest and logparams

Below is an example application which I’ll break down.

package main

import (
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/aaronvb/logparams"
    "github.com/aaronvb/logrequest"

    "github.com/gorilla/mux"
)

type application struct {
    errorLog *log.Logger
    infoLog  *log.Logger
}

func main() {
    infoLog := log.New(os.Stdout, "INFO\t", log.Ldate|log.Ltime)
    errorLog := log.New(os.Stderr, "ERROR\t", log.Ldate|log.Ltime|log.Lshortfile)

    app := &application{
        errorLog: errorLog,
        infoLog:  infoLog,
    }

    srv := &http.Server{
        Addr:     ":8080",
        ErrorLog: errorLog,
        Handler:  app.routes(),
    }

    infoLog.Printf("Starting server on %s", ":8080")
    err := srv.ListenAndServe()
    errorLog.Fatal(err)
}

func (app *application) routes() http.Handler {
    r := mux.NewRouter()
    r.HandleFunc("/foobar", app.foobar).Methods("POST")

    // Middleware
    r.Use(app.logRequest)
    r.Use(app.logParams)

    return r
}

func (app *application) foobar(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello world")
}

// Middleware

func (app *application) logRequest(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        lr := logrequest.LogRequest{Request: r, Writer: w, Handler: next, NewLine: 2, Timestamp: true}
        lr.ToLogger(app.infoLog)
    })
}

func (app *application) logParams(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        lp := logparams.LogParams{Request: r}
        lp.ToLogger(app.infoLog)
        next.ServeHTTP(w, r)
    })
}

First we need to make sure we create the log outputs and pass it to the application struct. These will be used by logrequest and logparam. Alternatively, you can have either package return a string instead of printing to a log output.

infoLog := log.New(os.Stdout, "INFO\t", log.Ldate|log.Ltime)
errorLog := log.New(os.Stderr, "ERROR\t", log.Ldate|log.Ltime|log.Lshortfile)

app := &application{
    errorLog: errorLog,
    infoLog:  infoLog,
}

In the routes function, which we use as the Handler, we create a new mux router and pass the two middleware functions to it. Both of those middleware functions will use the libraries.

func (app *application) routes() http.Handler {
    r := mux.NewRouter()
    r.HandleFunc("/foobar", app.foobar).Methods("POST")

    // Middleware
    r.Use(app.logRequest)
    r.Use(app.logParams)

    return r
}

The logRequest middleware function creates a new logrequest struct where we pass the http Request, Writer, Handler, and two optional arguments to configure the output. The reason why we pass the Handler is so that we can hook into the end of the request to show the response status and total time for the request.

Timestamp (boolean) can be used to hide/show the timestamp at the start of the request. (Started GET "/" 172.19.0.1:64368 HTTP/1.1 at 2021-01-19 08:30:08)

NewLine (int) can be used to add line breaks between each request log output(Rails does this by default).

We call ToLogger on the struct to print the request to the logger we created above.

Note: We don’t need to call next like you would normally do with middleware — logrequest package will call this automatically.

func (app *application) logRequest(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        lp := logrequest.LogRequest{Request: r, Writer: w, Handler: next, NewLine: 2, Timestamp: true}
        lp.ToLogger(app.infoLog)
    })
}

The logParams middleware function creates a new logparams struct where we pass just the Request.

By default logparams will filter out password and password_confirmation params. This can be turned off by passing ShowPassword: True (do NOT recommend this).

Also by default, logparams will hide empty parameters. This can be turned off by passing ShowEmpty: true.

We call ToLogger on the struct to print the params to the logger we created above.

func (app *application) logParams(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        lp := logparams.LogParams{Request: r}
        lp.ToLogger(app.infoLog)
        next.ServeHTTP(w, r)
    })
}

SQL Query Logging

The final part of this is going to be subjective based on how you’ve structured your application. In my latest project the structure was influenced by Alex Edwards Let’s Go! book, which is a great dive into building a web application in Go. If you’ve read that, or use a similar structure, this should be easy to apply.

In the main.go we’ll need to create a new log output, and pass that to the application model struct. In this example that’s going to be a User. We’re going to use this log output in the model to print the query.

Note: postgres is a custom postgres package(ORM), that the models belong to and is not related to any official package.

requestLog := log.New(os.Stdout, "", 0)

...

app := &application{
    errorLog:   errorLog,
    infoLog:    infoLog,
    requestLog: requestLog,
    session:    session,
    users: &postgres.UserModel{
        AppModels: postgres.AppModels{
            DB:  db,
            Log: requestLog,
        },
    },
}

In my custom postgres package I’ve created functions that wrap the database/sql query functions which will be used to print to the log output and execute the query.

The line _, file, line, _ := runtime.Caller(1) is what we use to print the file and line number that is calling the function.

// Package postgres is the ORM for the db
package postgres

import (
    "database/sql"
    "log"
    "runtime"
)

// AppModels is the struct that shares the functions between
// each model. Requires DB and Log.
type AppModels struct {
    DB  *sql.DB
    Log *log.Logger
}

func (m *AppModels) queryRow(query string, args ...interface{}) *sql.Row {
    m.Log.Printf("\t\u001b[34;1m%s \u001b[0m%v", query, args)
    _, file, line, _ := runtime.Caller(1)
    m.Log.Printf("\t\u21B3 %s:%d", file, line)
    return m.DB.QueryRow(query, args...)
}

func (m *AppModels) query(query string) (*sql.Rows, error) {
    m.Log.Printf("\t\u001b[34;1m%s", query)
    _, file, line, _ := runtime.Caller(1)
    m.Log.Printf("\t\u21B3 %s:%d", file, line)
    row, err := m.DB.Query(query)
    return row, err
}

func (m *AppModels) exec(query string, args ...interface{}) error {
    m.Log.Printf("\t\u001b[34;1m%s \u001b[0m%v", query, args)
    _, file, line, _ := runtime.Caller(1)
    m.Log.Printf("\t\u21B3 %s:%d", file, line)

    _, err := m.DB.Exec(query, args...)
    return err
}

And now we can use the above functions in the models to do things like find a User.

package postgres

type UserModel struct {
    AppModels
}

func (m *UserModel) Find(id int) (*models.User, error) {
    u := &models.User{}

    stmt := `SELECT id FROM users WHERE id = $1`
    row := m.queryRow(stmt, id)
    err := row.Scan(&u.ID)
    if err != nil {
        if errors.Is(err, sql.ErrNoRows) {
            return nil, models.ErrNoRecord
        } else {
            return nil, err
        }
    }

    return u, nil
}

I hope that helps you get started with some awesome Rails inspired logging. There’s quite a lot going on in this blog post so please comment below or contact me if you have any questions.

Simple Website Text Scraping with Go and AWS Lambda

Sun, 17 Jun 2018 00:00:00 GMT

Recently I needed to know when certain websites were updated with specific text. I decided to utilize AWS Lambda to save on cost of hosting a server, and use Go because it’s fast, and also because it’s one of the supported languages on AWS Lambda. I am also using AWS SES to send me e-mail notifications when results are found.

Bellow I’ll be showing you how to compile the Go script, setup the AWS Lambda function, and configure a cron type job to run the script every hour.

First, clone the repo contains the script.

github.com/aaronvb/aws_lambda_go_scraper

$ git clone git@github.com:aaronvb/aws_lambda_go_scraper.git
$ cd aws_lambda_go_scraper

Then we’ll build the Go script and zip it up for AWS Lambda.

$ GOOS=linux GOARCH=amd64 go build -o main lambda_scraper.go
$ zip main.zip main

Create an AWS Lambda function with the Go runtime, and select or create a role that has access to AWS SES. We’ll be using AWS SES to send out the e-mail notification.

Once the AWS Lambda function is created, upload the zip file and make sure the handler is set to main.

Next, create 3 environment variables: RECIPIENT will be the email which receives the notification, SENDER which will be the email address that sends the notification, and last SES_LOCATION which is the location of your SES(ie: us-west-2).

Don’t forget to add the email address to SES and verify it so it can receive emails.

Now we can create a test event. In the event data pass a JSON hash which has a key urls and a string value with the urls you want to scrape, separated by commas, and a key words, with a string value of comma separated words you wish to scrape.

Example:

{
  "urls": "https://aaronvb.com,https://aaronvb.com/articles/selection-sort-in-ruby.html",
  "words": "ruby,Hawaii,foobar"
}

Click the test button and you should receive a successful function execution with logs and an email.

The logs will contain the results, message ID from SES, and any errors while parsing or sending the email.

Success. The websites were scraped and my words were found!

Let’s Automate This

Now that the AWS Lambda function is working, it’s time to automate this and have it run every hour. We’ll pick different words because we know those exist. Let’s pretend we want to know when my personal website will be updated with the words “swift, java, and angular.”

For this we’ll be using AWS CloudWatch events. So let’s head over there and create a new events rule.

First we set the schedule to a fixed rate of 1 hour. Next, choose the Lambda function we created earlier. And finally, the most important part, select Configure input > Constant (JSON text), and paste in the JSON with the data to send to our function (see code below).

{
  "urls": "https://aaronvb.com",
  "words": "swift,java,angular"
}

Once you fill that in, click Configure details to name the rule and then create it. We now have the script running every hour, scraping our website, searching for the text we provided, and alerting us when it finds it.

Chaining API requests with redux-thunk

Wed, 07 Feb 2018 00:00:00 GMT

I recently came upon a simple problem where I couldn’t find an easy solution to, on the Google or SO. However, I did find the parts to the answer, and put them together to get this working.

My problem was finding a way to chain API requests using redux-thunk and axios. My use case was to gather an array of objects(containing id’s) on the first API request, and then subsequently gather each individual objects’ data from the array above, using the id, through another API request, all asynchronously.

I was able to solve this in redux-thunk by making two actions; one to fetch all, and one to fetch each individually. And, to chain them together, we treat each action as a promise(they return axios promises), then let redux-thunk do it’s magic. To get the array of objects in the first API request, we can use getState() from the first promise.

export const requestIssuesAndIssueData = () => {
  return (dispatch, getState) => {
    dispatch(requestIssues()).then(() => {
      const issuesArr = getState().issues.issuesById;
      issuesArr.forEach(issue => {
        dispatch(requestIssueCommentsHash(issue.number));
      });
    });
  };
};

In the snippet above I am using GitHub issues as an example.

To see the entire example, checkout my repo at https://github.com/aaronvb/redux-thunk-chain-api-requests.

To see it in action I setup a demo project at https://aaronvb.github.io/redux-thunk-chain-api-requests.

Happy hacking!

A Simple React Native Redux Example

Sun, 18 Jun 2017 00:00:00 GMT

Working on my latest React Native project, I decided to make the jump from React’s state to Redux.

I looked through all the examples I could find, the majority were from 2015, and couldn’t find something simple and minimal. Also, a lot of things have changed since 2015, and even 2016 - giving me a good excuse to start from scratch. My first step was to go through the Redux docs and tutorials to give myself a clean slate, instead of just copying code from my previous projects.

http://redux.js.org/docs/basics/ExampleTodoList.html

The above link has a minimal example of a Redux implementation, and is what I used as base for my project.

This is the directory structure of the app which contains Components, Containers, and Reducers, pretty common with a standard React/Redux app.

Simple React Native Redux Example app running in iPhone Simulator.

To show the list of items, I’m using the React Native ListView that is using a Redux Store(items) as its dataSource. There are two actions in the Reducer; ADD_ITEM and REMOVE_ITEM.

// Start the sequence of item ID's at 0
let nextItemId = 0;
// Items reducer
const items = (state = [], action) => {
  switch (action.type) {
    case "ADD_ITEM": {
      return [
        ...state,
        {
          id: nextItemId++,
          name: action.name,
          bgColor: action.bgColor
        }
      ];
    }
    case "REMOVE_ITEM": {
      // Find index of item with matching ID and then
      // remove it from the array by its' index
      const index = state.findIndex(x => x.id === action.id);
      return [...state.slice(0, index), ...state.slice(index + 1)];
    }
    default:
      return state;
  }
};
export default items;

The ListView requires a dataSource, and must be re-rendered when the dataSource is updated. To do this, the dataSource gets mapped to a prop using mapStateToPros, and then we connect the Component to it.

class ItemList extends Component {
  constructor(props) {
    super(props);
    this.handleDestroyItem = this.handleDestroyItem.bind(this);
  }
handleDestroyItem(id) {
    this.props.dispatch({ type: "REMOVE_ITEM", id });
  }
render() {
    return (
      <ListView
        style={styles.container}
        enableEmptySections={true}
        dataSource={this.props.dataSource}
        renderRow={rowData => {
          return (
            <Item
              rowData={rowData}
              handleDestroyItem={id => this.handleDestroyItem(id)}
            />
          );
        }}
      />
    );
  }
}
// Handle data source change from redux store
const dataSource = new ListView.DataSource({
  rowHasChanged: (r1, r2) => r1 !== r2
});
function mapStateToProps(state) {
  return {
    dataSource: dataSource.cloneWithRows(state.items)
  };
}
ItemList.propTypes = {
  dataSource: PropTypes.object,
  dispatch: PropTypes.func
};
export default connect(mapStateToProps)(ItemList);

View the entire code at https://github.com/aaronvb/react-native-redux-example.

A 60% Keyboard w/ Arrow Keys Build Log

Thu, 18 May 2017 00:00:00 GMT

My 60% keyboard with arrow keys XD60 build, with more cat!

With the drop happening right now hopefully I can provide a little insight on this KB.

A little background; I love the idea of arrow keys on 60% keyboards. My daily driver was a modified Infinity with arrow keys:

I was lucky to get 1.25u arrow keys on the first Granite drop to make that build happen. I’ve had my eye on the XD60 for awhile but did not like the idea of moving the ? key or finding a 2u shift so I held off until I had a reason to build another KB.

I picked up the PCB and two plates from Taobao, it took about 2 weeks to get delivered. My original plan was to go with the 2u shift, but at the last moment I decided to try 2x 1u shift. It’s pretty much impossible to get a 2u shift now days. I also decided to try PCB mount switches, my first on all the keyboards I’ve built.

Ordered from Taobao, took about 3 weeks.

Decided to go with PCB mount switches.

After switches soldered I decided to add LED’s. I’m not a huge fan of back-lighting but I had 200 white LED’s from an old an project.

LED’s installed and working. Note that there were 3 switches which had the positive and negative solder points switched. Always double check the LED’s before soldering.

Under-glow LED’s working as well. Too bad I wont see them with my case.

PCB mount stabilizers.

A few of my thoughts so far: The under-glow is neat but I’ll need a new case. It’s weird having 2x 1u shift, still not used to it yet. PCB mount feels more unstable than plate mount, might be louder also. The cat logo is really cool. I should have used SIP sockets. Down and Right arrow LED have opposite positive and negative terminals.

I’ve open-sourced the keymap for my keyboard, check it out at github.com/aaronvb/xd60-keymap.

Street Photography Class

Fri, 05 May 2017 00:00:00 GMT

Last week I taught a photography class at Hawaii Camera aimed at new photographers who were interested in learning the basics of street photography. It was my third photography class, and my first street photography class.

www.eventbrite.com/e/street-photography-an-introduction-for-new-photographers-tickets-33878881647

Overall the turnout was better than I expected and everyone had walked out with a new perspective on photography.

I would like to share the PDF of my presentation: streetphotography by aaronvb.pdf

Ember.js Flash Messages

Wed, 30 Mar 2016 00:00:00 GMT

This is a simple way to create flash messages that are similar to the Ruby on Rails flash messages. I took from Yehuda Katz’s answer on this Stack Overflow post, and extended it to work with the updated controllerFor/needs changes and closing on transitions.

application.handlebars

<div class="container">
  {{#if notification}}
  <div {{bindAttr class=":alert notification.type"}} id="notification">
    <button type="button" class="btn-close" {{action "closeNotification"}}></button>
    {{#if notification.title}}
    <h3>{{notification.title}}</h3>
    {{/if}}
    {{notification.message}}
  </div>
  {{/if}}
  {{outlet}}
</div>

application_controller.js.coffee

App.ApplicationController = Ember.Controller.extend

  # close notification alert
  # bind to action in template, example: {{action "closeNotification"}}
  # detects if persists and closes on next transition
  #
  closeNotification: ->
    notification = @get('notification')
    if notification
      if notification.persists
        console.log "Notification detected, clearing alert notification after next transition"
        notification.persists = null
      else
        console.log "Notification detected, clearing alert notification now"
        this.set('notification', null)

  # notification alert
  # type can be: error, info, success
  # example: @get('controllers.application').notify({title: "Error!", message: "An error occurred in foobar.", type: "alert-error"})
  #
  notify: (options) ->
    this.set('notification', options)

posts_controller.js.coffee

App.PostsNewController = Ember.ObjectController.extend
  needs: ["application"] # allows access to application controller functions

  # post save
  #
  save: ->
    @set('model.title', @get('title'))
    @set('model.description', @get('description'))

    post = @get('model')

    # if post is not valid, show flash message
    #
    post.on 'becameInvalid', this, (obj) ->
      @get("controllers.application").notify({
        title: "Error!",
        message: "There was an error creating this post.",
        type: "alert-error"})

    # if post was created transition to posts index, show flash message and persist it to next transition
    #
    post.on 'didCreate', this, () ->
      @get("controllers.application").notify({
        title: "A new post!",
        message: "#{@get('model.title')} was created.",
        type: "alert-success",
        persists: true})
      @transitionToRoute('posts.index')

    # commit post to server
    #
    @get('model.transaction').commit()

router.js.coffee

App.BeforeRoute = Ember.Route.extend
  # close any open notifcations before a route loads
  #
  activate: ->
    @controllerFor('application').closeNotification()

App.PostsIndexRoute = App.BeforeRoute.extend()

In the application.handlebars template I create a condition that checks if the notification variable is set, and if it is, display it with it’s message and alert-type class(error, success, info, etc).

In the application controller, there’s a function that creates the notification and passes options to it, and there’s also a function that closes the notification. The closeNotification first checks for a ‘persists’ flag and if it doesn’t exist it sets the notification variable to null, which closes it. If a ‘persists’ flag does exist, it sets it the flag to null and skips closing the notification. On the next transition, the ‘persists’ flag wont exist and the notification will be closed.

In the posts new controller, at the top I have a needs array, which let’s the controller know that we need access to the application controller. The notifications are hooked into the ‘on’ callbacks for the post model, and called by using @get(‘controllers.application’).notify(). In earlier versions of Ember.js you would directly use the controllerFor() function, but that was deprecated for ‘needs’.

Lastly in the router, I created an App.BeforeRoute extension of Ember.Route which all of my routes extend from. This way every route will execute the activate function, which executes the closeNotification function, before every route loads.

KWM & Ubersicht Bottom Bar

Wed, 30 Mar 2016 00:00:00 GMT

I recently made the switch to a tiling window manager in OS X. After watching my friend use i3 on his Linux machine, OS X started to feel outdated to me, ironically. OS X really lacks a solid window management experience. I’ve messed around with the built in Spaces, but would revert back to one space, and run every app at max size, relying on cmd+tab to move around.

Spectacle became my pseudo tiling manager for a short while, mainly for split screen work(which I became very fond of). But Spectacle, for me, wasn’t enough. It provided easy to use hotkeys for controlling window sizes, but I wanted it to be automated. I also wanted to control padding around windows, because I like padding!

The same friend who showed me i3 suggested I try KWM, a real tiling manager built for OS X. It took me a bit to get used to the mouse focus control, and KWM can sometimes do weird things, but overall I’m pretty satisfied with it.

Before my switch to KWM, I was using iStat Menus for system information and with the menu bar hidden, it became useless to run. The KWM repo shows an example of nice menu bar replacement, with system information, and I thought that would be nicer if it was at the bottom, like i3. I used that as a starting point, added network traffic and weather data, and ended up with this:

github.com/aaronvb/ubersicht-bottombar

You’ll need to download Übersicht, which is the widget app this uses.

Here’s a screenshot of it in use:

BitBar, Tarsnap, and Ruby

Sat, 27 Feb 2016 00:00:00 GMT

I came across a really cool OSX menu bar library and thought it would be a really good solution to show my Tarsnap backup statuses.

github.com/matryer/bitbar

I love how easy it is to write a plugin for BitBar in Ruby.

It’s as simple as:

#!/usr/bin/ruby
puts "Cool Menubar"
puts "---"
puts "The time is: #{Time.now.strftime('%T')}"

Anyway, I updated my git repo to include a BitBar plugin and also updated the main backup script to output a nice JSON file for it to read: github.com/aaronvb/tarsnap_backup

And here’s a screenshot of it in use:

Ruby on Rails Wrapper for react-stdio

Mon, 01 Feb 2016 00:00:00 GMT

Over the past weekend I was exploring a different method to server side render(aka prerender) react components in Ruby on Rails.

Right now the two most popular Rails + React gems are react-rails and react on rails. Both of them rely on ExecJS(using therubyracer), with react-rails also supporting sprockets, to prerender react components.

If you’re not familiar with prerendering react components on the server, it’s simply Rails rendering the react component in the view before the DOM is loaded. This can be beneficial for SEO reasons.

I came across react-stdio (not to be confused with studio) while working on issues in the reactonrails repo: #183 and decided to see if I could easily implement it without rewriting a lot of the server rendering code. I started with react-rails as they have support for adding your own server renderer class.

Long story short, I was not able to easily integrate react-stdio into either libraries due to the way they render JS code. They both take the compiled components JS file and pass the string of JS code to be evaluated and returned, while also passing polyfills before the JS code. In react-stdio, you can only provide it with a path to the component, not a string of JS code. While I think it’s definitely possible to integrate it react-stdio, I don’t believe it’s worth pursing as the current methods work perfectly fine.

Still, I think there could be uses outside of those libraries, which is why I decided to create this gem.

This is a simple Ruby on Rails wrapper for react-stdio. — railsreactstdio

Using railsreactstdio is actually extremely simple, just like react-stdio!

First make sure you have NPM installed and have react-stdio installed.

Add rails_react_stdio to your gemfile:

gem 'rails_react_stdio', '~> 0.1.0'

If your path to react-stdio is not installed in the default location /usr/local/bin/react-stdio, add a configuration file to your initializers folder.

RailsReactStdio.configure do |config|
  config.react_stdio_path = '/your/path/to/react-stdio'
end

path_to_component = ::Rails.application.assets['components/HelloWorld'].filename

RailsReactStdio::React.render(path_to_component, {message: "aaron"})

> "<p data-reactid=\".1on4o1jtdds\" data-react-checksum=\"359665029\">Hello, aaron</p>"

If you find this useful give me feedback or open a PR. Also, thanks to react-stdio for providing a neat way to render react components.

Automate Tarsnap in OSX with Ruby

Sat, 06 Dec 2014 00:00:00 GMT

I finally found the time to setup proper backups with Tarsnap, tarsnap.com. After following their installation guide, which was very easy to do, I was stuck with a dilemma - how can I automate, rotate, and log this?

My first attempt to automate tarsnap backups was a bash shell script and cron, which worked okay, but still not exactly how I wanted it. For one, I’m not the best at writing bash shell scripts. And two, cron isn’t the best option for automated tasks in OSX. My solution was to use Ruby for the script, and Launchd for automation. What’s really great about Launchd, versus cron, is that if my laptop is sleeping when my script is supposed to run, Launchd will run the job as soon as my laptop awakes.

My github repository for this is at github.com/aaronvb/tarsnap_backup. If you just want to install and run it, head there and clone the repository. Otherwise, I’m going to explain a little about what the script is doing and how Launchd works.

The script is fairly simple and relies on Ruby’s system command call. The back up portion is straight forward, run tarsnap and append a timestamp to the backup name. The prune part is where the script will remove any backups after a certain period, in my case 3 days. It does this by getting a list of backups from tarsnap, and interating over each one, parsing the timestamp and checking if the date is past the prune date(3 days in the past from now).

The plist, which is the Launchd file, automates this script in OSX. The one provided in my github repo is what I use, and you’ll need to modify it to work for you.

Give your job an appropriate, unique, name. This will be used to start/stop your job.

<string>com.aaronvb.tarsnap-backup</string>

Here we have the location to the Ruby exec, and the actual backup.rb file. In my case, I use RVM so my latest Ruby version was in the RVM folder. The backup.rb file is in my tarsnap folder.

<array>
  <string>/Users/aaronvb/.rvm/rubies/ruby-2.1.5/bin/ruby</string>
  <string>/Users/aaronvb/.tarsnap/tasks/backup.rb</string>
</array>

This is the location for your Launchd logs. This is totally optional, and mainly used for debugging. The tarsnap Ruby script will handle it’s own logs.

<key>StandardOutPath</key>
<string>/Users/aaronvb/.tarsnap/logs/launchd.log</string>
<key>StandardErrorPath</key>
<string>/Users/aaronvb/.tarsnap/logs/launchd.log</string>
<key>StartCalendarInterval</key>

This is when Launchd will run this job. I have it set to run every day at Midnight, 8a, 12p, 4p, and 8p. Like I mentioned above, if your computer is sleeping while this job is supposed to run, it will run when it wakes. If you prefer to have it run every N seconds, instead of at exact times, you can use StartInterval.

<key>StartCalendarInterval</key>
<array>
	<dict>
		<key>Hour</key>
		<integer>0</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
	<dict>
		<key>Hour</key>
		<integer>8</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
	<dict>
		<key>Hour</key>
		<integer>12</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
	<dict>
		<key>Hour</key>
		<integer>16</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
	<dict>
		<key>Hour</key>
		<integer>20</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
</array>

StartInterval example. This will run every hour.

<key>StartInterval</key>
<integer>3600</integer>

launchd.info is a great resource for Launchd.

The last part to getting it running is placing the plist in the correct folder and running it properly. I have my plist in /Library/LaunchDaemons because I want it to run system wide and not based on the user. If you place it in this folder, you NEED to load and start the job using sudo.

Selection Sort in Ruby

Sat, 01 Nov 2014 00:00:00 GMT

In my version of Selection Sort in Ruby, I start with the first object in the array and compare it to the next one and so on until I find lowest value, swapping it with the first objects position. I then repeat that on the second object, and the next, until the whole list is sorted.

Here’s an animated sequence of Selection Sort that I found on the Selection Sort Wikipedia.

As always, start with the tests.

require 'test/unit'
require "./selection_sort.rb"

class SortingTests < Test::Unit::TestCase
  def test_selection_sort
    @unsorted = (0..1000).to_a.shuffle
    @sorted = @unsorted.sort
    selection_sort = SelectionSort.new
    result = selection_sort.sort(@unsorted)
    assert_equal @sorted, result
  end
end

And next, the Selection Sort function.

class SelectionSort
  def sort(to_sort)
    arr_to_sort = to_sort
    length_of_sort = arr_to_sort.length
    position_in_array = 0
    while position_in_array < length_of_sort
      @min_val = arr_to_sort[position_in_array]
      (length_of_sort - position_in_array).times.each do |i|
        if arr_to_sort[position_in_array + i] < @min_val
          @min_val = arr_to_sort[position_in_array + i]
          @min_val_index = position_in_array + i
        end
      end
      if @min_val < arr_to_sort[position_in_array]
        @swap_value = arr_to_sort[position_in_array]
        arr_to_sort.delete_at(@min_val_index)
        arr_to_sort.insert(position_in_array, @min_val)
        arr_to_sort.delete_at(position_in_array+1)
        arr_to_sort.insert(@min_val_index, @swap_value)
      end
      position_in_array += 1
    end
    return arr_to_sort
  end
end

Intro to Elasticsearch and Ruby on Rails, Part 1

Fri, 17 Oct 2014 00:00:00 GMT

This will be a two-part guide on my first experiences with Elasticsearch and Ruby on Rails 4. I’ve been a long time fan of Solr, and Sunspot with Ruby on Rails, as Sunspot has always been easy to implement without a big learning curve. There’s also a lot of information out there for Sunspot.

Elasticsearch, on the other hand, seems to be less popular with Ruby on Rails. From my research, there were a few gems that were popular but are no longer active, and any google searches on information regarding them are dated at least a year back. It seems that the go-to gem for Elasticsearch is now from Elasticsearch itself, which actually consist of 3 separate gems.

In the first part, I will setup a basic search using the default Elasticsearch config, while also configuring the search of ActiveRecord associations and indexing a value obtained by a method. In the second part I will cover more advanced searches using custom analyzers and filters, such as ngram, edgengram, and lowercase filters on specific columns.

To get started, grab the latest Ruby on Rails, which at the time of this post is 4.1.6, and add these gems to your Gemfile. Notice we are only using two of the gems.

gem 'elasticsearch-model',
  git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-rails',
  git: 'git://github.com/elasticsearch/elasticsearch-rails.git'

Some information on the two gems we are using taken from their Github repo:

elasticsearch-model, which contains search integration for Ruby/Rails models such as ActiveRecord::Base and Mongoid
elasticsearch-rails, which contains various features for Ruby on Rails applications.

We don’t need to use elasticsearch-persistence because elasticsearch-model will handle everything we need in our Ruby on Rails app.

The easiest way to install Elasticsearch is by using Homebrew, which I hope you are already using.

> brew install elasticsearch
> elasticsearch --config=/usr/local/opt/elasticsearch/config/elasticsearch.yml

By default, our Rails app will connect to Elasticsearch at localhost:9200, but if you need to connect to a different server, create a .rb file in your config/initializers and add this.

# config/initializers/elasticsearch.rb
# ENV['ELASTICSEARCH_ADDRESS_INT'] is the environment variable
# for the elasticsearch server, replace with IP address if not using ENV
User.__elasticsearch__.client = Elasticsearch::Client.new host: ENV['ELASTICSEARCH_ADDRESS_INT']

Alright, so if you haven’t already figured it out from the code above, we’re going to be searching a User model. Create a User table if you don’t already have one, and make sure it has a first name and email column.

> rails g model user first_name:string email:string
> rake db:migrate

Open the model file at app/models/user.rb, and add the Elasticsearch includes. This will allow us to use the Elasticsearch methods on our model, and will also enable the Elasticsearch callbacks for events such as creating, updating, and destroying our model objects, which will keep the Elasticsearch index up to date.

class User < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

That’s pretty much all you need to do to start searching the User model. By default this will search all the columns in this model. Open up the rails console to see how easy it is.

> rails c
Loading development environment (Rails 4.1.6)
irb(main):001:0> User.import
=> 0
irb(main):001:0> User.search("aaron").results.count
=> 10

One thing you’ll notice is the import method that I’m calling on the User model. This is needed because our search index is empty and we need to populate it with our current data. Any subsequent changes to the User table will automatically be added to the index by the Elasticsearch callbacks, so you wont need to run the import method again, unless changes are made outside of the Rails app.

If you read up on the documentation, you’ll be able to do some neat things like find the score each result. One thing I want to mention is that if you use the records method instead of the results method, you will get a collection of ActiveRecord objects instead of Elasticsearch objects.

irb(main):001:0> User.search("aaron").results.first
=> #<Elasticsearch::Model::Response::Result:0x007fd200e71380 @result=#<Hashie::Mash _id="155" _index="users" _score=0.5878618...
> User.search("aaron").results.first._score
=> 0.5878618
> User.search("aaron").records.first
=> #<User id: 155, first_name: "aaron", email: "bokhoven@gmail.com"...

Cool, so now we can search our User table. But what if we have associations that need to be searched as well? Elasticsearch has a model function for that. Let’s start by creating our association model: PhoneNumber, and adding the associations to our models.

> rails g model phone_number number:string user_id:integer
> rake db:migrate

class PhoneNumber < ActiveRecord::Base
  belongs_to :user
end

class User < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  has_many :phone_numbers
end

Now we need to configure how Elasticsearch creates the index document for each object. In our User model add this function.

class User < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  has_many :phone_numbers

  def as_indexed_json(options={})
    as_json(
      only: [:id, :first_name, :email],
      include: [:phone_numbers]
    )
  end
end

I’ll explain what’s going on here. When Elasticsearch indexes our User object, we are telling it that we only want to search the id, first name, and email attribute, and all phone numbers associated with this object. To better visualize what the document looks like, open up the rails console.

> rails c
irb(main):001:0> User.first.as_indexed_json
=> {"id"=>1, "email"=>"bokhoven@gmail.com", "first_name"=>"aaron",
  "phone_numbers"=>[{"id"=>1, "user_id"=>1, "number"=>"123-456-7890"}]}]}

Note, when we search for a Phone Number, we aren’t searching a separate phone number index, we are still searching our User index that has a collection of phone numbers.

Elasticsearch, like most NoSQL databases, treats the world as though it were flat. An index is a flat collection of independent documents. A single document should contain all of the information that is required to decide whether it matches a search request or not.

Elasticsearch Relations

To create a separate Phone Number index and have Elasticsearch emulate the association, look into Nested Objects. While both have their advantages and disadvantages, sticking with the flat document will work well most of the time. Possibly in another article I will go over Nested Objects.

Let’s rebuilt and test our new search index.

irb(main):001:0> User.import
=> 0
> query = User.search("123-456-7890").records.first
=> #<User id: 155, first_name: "aaron", email: "bokhoven@gmail.com"...
> query.phone_numbers.first
=> #<ActiveRecord::Associations::CollectionProxy [#<PhoneNumber id: 1, user_id: 155, number: "123-456-7890">]>

Easy, everything works. Not quite. When using associations, it’s not a good idea to use the records method on the search object. The reason for this is because the collection of records from the search object are ActiveRecord objects, and using the phone numbers method on it will cause ActiveRecord to look up each phone number object for each search record. What you want to use is the results method on the search object, this will use the “flat” document and not create any more queries for the phone numbers as they are included in the results object. To make this clearer, I’ll show you an example.

This is with records.

irb(main):001:0> User.search("aaron").records.first
  User Search (10.1ms) {index: "users", type: "user", q: "aaron"}
  User Load (3.0ms)  SELECT "users".* FROM "users"  WHERE "users"."id" IN (155, 156, 157, 158)
=> #<User id: 155, first_name: "aaron", email: "bokhoven@gmail.com"...

> User.search("aaron").records.first.phone_numbers
  User Search (10.1ms) {index: "users", type: "user", q: "aaron"}
  User Load (3.0ms)  SELECT "users".* FROM "users"  WHERE "users"."id" IN (155, 156, 157, 158)
  PhoneNumber Load (2.9ms)  SELECT "phone_numbers".* FROM "phone_numbers"  WHERE "phone_numbers"."user_id" = $1  [["user_id", 155]]
=> #<ActiveRecord::Associations::CollectionProxy [#<PhoneNumber id: 1, user_id: 155, number: "123-456-7890"...

You can see that another call was made to the PhoneNumber model because we’re calling the phone_numbers method on the User ActiveRecord object which was in the records collection.

This is with results.

irb(main):001:0> User.search("aaron").results.first
  User Search (10.1ms) {index: "users", type: "user", q: "aaron"}
=> #<Hashie::Mash id: 155, first_name: "aaron", email: "bokhoven@gmail.com"...

> User.search("aaron").records.first.phone_numbers
  User Search (10.1ms) {index: "users", type: "user", q: "aaron"}
=> #<Hashie::Mash id: 155, first_name: "aaron", email: "bokhoven@gmail.com", phone_numbers=[#<Hashie::Mash number="123-456-7890"...

With that said, let’s move on to the last topic I wanted to cover, adding values to the search document that are created by a method in the model. An example of this could be a full name method that returns a first and last name column together. It makes more sense to search the full name value instead of both columns separately. We’ll use that example in our app.

> rails g migration add_last_name_to_users last_name:string
> rake db:migrate

class User < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  has_many :phone_numbers

  def full_name
    "#{first_name} #{last_name}"
  end

  def as_indexed_json(options={})
    as_json(
      only: [:id, :full_name, :email],
      include: [:phone_numbers],
      methods: [:full_name]
    )
  end
end

I added the full_name method to our User model, and then modified the as_indexed_json method to include the full_name method, and also added it to the document by including it in the only collection. You can always double check what the document looks like by calling the the as_indexed_json method on a User model, I explained how earlier in this article.

irb(main):001:0> User.search("aaron van bokhoven").results.first
=> #<Hashie::Mash id: 155, full_name: "aaron van bokhoven", email: "bokhoven@gmail.com"...

This is the end of the first part. Elasticsearch is a powerful tool and I’ve only just begun to scratch the surface. Stay tuned for my next part on customizing Elasicsearch to use analyzers and filters.

Bubble Sort in Ruby

Sat, 20 Apr 2013 00:00:00 GMT

Lately during my free time I’ve been reading through sorting algorithms for fun and decided to implement several in Ruby. I’ll be posting each one in a separate article as I go through them.

Here’s a bubble sort in Ruby.

Bubble sort is a pretty fun and easy sorting algorithm. For each pass through an array of values, each value is compared to its adjacent value and swapped into the correct order and so on. Worst case is O(n^2) and best case is O(n) if the array is already sorted.

Let’s start with the test.

require 'test/unit'
require "./bubble_sort.rb"

class SortingTests < Test::Unit::TestCase
  def test_bubble_sort
    @unsorted = (0..1000).to_a.sort{ rand() - 0.5 }[0..1000] # 1000 random integers in an array
    @sorted = @unsorted.sort
    bubble_sort = BubbleSort.new
    result = bubble_sort.sort(@unsorted)
    assert_equal @sorted, result
  end
end

In the test I create an array of random numbers and assert_equal to the result of the bubble sort.

Next the code for the bubble sort.

class BubbleSort
  def sort(to_sort)
    # move the array to sort into a variable, which will be used for recursion
    arr_to_sort = to_sort
    # assume that we haven't swapped any values yet
    swapped = false
    # lower the length by one because we can't compare the last value since it's at the end
    length_of_sort = arr_to_sort.length - 1
    # begin loop through each value
    length_of_sort.times.each do |i|
      # if the value we're on is greater than the value to the left of it, swap
      if arr_to_sort[i] > arr_to_sort[i+1]
        # store values to be swapped
        a,b = arr_to_sort[i],arr_to_sort[i+1]
        # remove value we're on
        arr_to_sort.delete_at(i)
        # insert the value to the right, moving the lesser value to the left
        arr_to_sort.insert(i+1, a)
        # swap is true since we did a swap during this pass
        swapped = true
      end
    end
    if swapped == false
      # no swaps, return sorted array
      return arr_to_sort
    else
      # swaps were true, pass array to sort method
      bubble_sort = BubbleSort.new
      bubble_sort.sort(arr_to_sort)
    end
  end
end

I use recursion for each pass until swapped is equal to false, which means that the array is sorted.

Moving from Linode to Heroku

Wed, 16 Jan 2013 00:00:00 GMT

I recently made the move from Linode to Heroku for my personal sites. The last time I checked out Heroku was about 2 years ago, which I later dismissed because at the time it seemed like having your own VPS was the right thing to do.

Having your own VPS sounds good in theory, but in reality it’s not, for me at least. I liked being able to throw up random projects, host my friends projects, and tinker with new frameworks and languages. I maybe did that once or twice out of the 4 years I had my own VPS. In reality, I’m way to busy to do any of that. I started to get lazy on updates, and security patches. The traffic my personal sites receive are so minimal they don’t need a VPS.

After my week at the Aloha Ruby Conf, I noted to myself to check out Heroku once again, to see what has changed. To my surprise, A LOT has changed. There’s the new cedar stack, updated gem which is really awesome to use, procfile, etc, I could go on.

Anyway, I read through all their documents, which are amazingly good, and then signed up to try their free tier, 1 Dyno, which seems like a perfect size for my site.

The first thing I had to do was switch my applications database from MySQL to PostgreSQL because Heroku has native support for PostgreSQL. You can use MySQL through one of their add-ons if you wish.

# gem 'mysql2'
gem 'pg'

Then I got my site up on Heroku by logging in and deploying.

$ heroku login
$ heroku create
$ git push heroku master
$ heroku run rake:db:migrate

I really like Unicorn(yhbt.net/unicorn/) as my webserver and was excited to see that they fully support it, and all I needed to do was add one line to my Procfile. I also want to mention that even though 1 Dyno is supposed to only have 1 level of concurrency, with Unicorn I can have multiple concurrency.

Add Unicorn the Gemfile.

gem 'unicorn'

I added a Unicorn config file to my project by creating config/unicorn.rb. In the file I can configure the number of workers and timeout.

worker_processes 5
timeout 30
preload_app true

before_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
    Rails.logger.info('Disconnected from ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end

  sleep 1
end

after_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
    Rails.logger.info('Connected to ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

Now that Unicorn is set up, Heroku needs to use Unicorn instead of its default web server. To tell Heroku to use Unicorn, they’ve provided the Procfile. I created a file named Procfile in the project root directory. Inside the web server can be configured.

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

After that’s pushed to Heroku, I can check to make sure Unicorn is running by using $ heroku ps.

$ heroku ps
> === web: `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
   web.1: up 2013/01/16 14:21:20 (~ 1h ago)

I also needed access to cron or something similar to schedule my nightly Rake that updates the photos on the front page to my latest Flickr photos. Once again that took me a few minutes to set up through their web interface.

First I created my rake task in lib/tasks/flickr_update.rake. It’s a simple script that updates a table with the last 3 photos I posted to flickr. I’m using the gem ‘fleakr’ to do this. I have a RecentPhoto model that has the attributes: title, url(url to the photos flickr page), image_url(url to the exact location of the image file).

namespace :flickr  do
  desc "update flickr photos on index page"
  task :update => :environment do
    Fleakr.api_key = 'xxxxxx' # replace with your Flickr API key
    user = Fleakr.user('xxxx') # replace with your Flickr Username
    RecentPhoto.destroy_all # remove all photos from table
    3.times do |num|
      photo = user.photos[num]
      image_url = photo.medium.url
      image_url = image_url[0..-5] << "_q.jpg" # using the q version of this photo which is a square thumbnail.
      RecentPhoto.create(title: photo.title, url: photo.url, image_url: image_url)
    end
  end
end

In the Heroku web application settings add the Scheduler add-on. Then add the heroku run command for that rake task.

That pretty much covers my move from Linode to Heroku. I highly suggest reading through the Heroku documents.

Side note: The only downside to Heroku that I encountered was not being able to write to disk. Since my site is entirely static, the comments are javascript, I relied on page caching to keep the load time fast. When I switched to Heroku I had to disable page caching because I’m not able to write to the cached html pages to disk.

Thunderbolt External Hard Drive with Seagate STAE128 Adapter

Fri, 14 Dec 2012 00:00:00 GMT

Yes, you can use the Seagate STAE128 Thunderbolt Adapter for Backup Plus with any SATA external 2.5” Hard Drive. You can also use the STAE121 but I’ve read that they discontinued that for the newer STAE128.

Here’s a little review on my experience with the Seagate STAE128 Thunderbolt Adapter and a bare 2.5” SATA external hard drive. Since switching over to the Macbook Pro Retina with 256gb SSD, I’ve been a little worried about my hard drive space.

Coming from the regular Macbook Pro with a 750gb hard drive I wasn’t used to worrying about space. I had to move all of my music to an external and reduced the amount of photos I could work on at a time. Since I scan my photos to raw uncompressed TIFF files at a fairly high DPI, the files are quite large. I used to keep about six months to a years worth of scanned negatives on my laptop at a time, and moved anything older to an external. With only 256gb to spare I’m limited to maybe half of that.

So my research started when I was looking into an external in which I could use to edit directly on and was portable. This required something fast and small. I looked into USB 3.0 and Thunderbolt, to which the later provided minimal results. The best I could find were 3 or 4 heavy options for Thunderbolt external hard drives at really steep prices, considering I could buy a 1TB USB 3.0 portal external for a hundred dollars.

After more research I came upon the Seagate Thunder Bolt Adapter, STAE 121, that would allow me to convert the Seagate enclosed external drives to Thunderbolt. A little more googling and I found out I could actually use the STAE 121 with a bare 2.5” SATA hard drive. A little skeptical at first mostly because of the lack of people talking about it on the internet, I went to Best Buy to see if they had it in stock.

When I got there I found the newer version, the Seagate Thunderbolt Adapter For Backup Plus, or STAE128. A little confused, I tried to google it but found nothing. I bought it, and a Thunderbolt plug by Apple.

At first I thought I would need to use a SATA cable to connect my hard drive, but it actually plugged right into it.

I’m using a Seagate Momentus 7200rpm 500gb hard drive which I pulled from my old Macbook Pro. I don’t have any benchmarks from when that was installed internally, but I did find a few through google and this seems to be about the average.

My results using the same hard drive and the Seagate Thunderbolt Adapter STAE128 were very close to that.

Compared to my 2TB Western Digital External HDD USB2.0.

and for fun, compared to the internal 256gb SSD that’s inside my Macbook Retina.

Overall I’m definitely happy with the STAE128. I get to use my SATA 2.5” hard drives that I have laying around, I get speeds that match internal speeds, it’s portable and self powered, and best of all it was only $99(+$ 50 for the Thunderbolt cable). The price alone beats any of the other Thunderbolt options out right now.

Next up, installing and booting Windows 7 off the STAE128 Thunderbolt drive.

UPDATE:

I successfully installed, and am now running, Windows 7 from my Thunderbolt external hard drive. I followed these directions and it worked perfectly.

A Polymorphic Join Table

Thu, 29 Nov 2012 00:00:00 GMT

Here’s an interesting problem I ran into today.

Polymorphic associations in Ruby on Rails are actually quite easy to do, especially in Rails 3.2. If you need a refresher, there’s a great screencast over at Railscasts, which does require a subscription which I highly highly recommend: railscasts.com/episodes/154-polymorphic-association-revised.

However, my problem was a little different, and maybe a special case because I can’t think of many applications this would apply to.

Say I have a Location and a Checkpoint, and the Location and Checkpoint can have notes, posted by Users. I would use a polymorphic association for the Notes to the Location and Checkpoint. But, I also want a single Note, to be posted to many Locations or Checkpoints, which for a single model-to-model relationship I could simply use a join table.

Example, two Locations which are near each other, maybe they’re coordinates, could share a single Note describing the general area, and with the same Note model, one Note may describe multiple Checkpoints.

Solution: Make the join table polymorphic.

Notes model that contains the content and user_id, which I use to associate the User model with.

class CreateNotes < ActiveRecord::Migration
  def change
    create_table :notes do |t|
      t.text :content
      t.integer :user_id
      t.timestamps
    end
  end
end

Note join model that is also polymorphic. notableid and notabletype is the polymorphic attributes used by ActiveRecord.

class CreateNoteJoins < ActiveRecord::Migration
  def change
    create_table :note_joins do |t|
      t.integer :note_id
      t.integer :notable_id
      t.string  :notable_type
      t.timestamps
    end
    add_index :note_joins, [:notable_id, :notable_type]
  end
end

Next I add the model associations to Location and Checkpoint models.

class Location < ActiveRecord::Base
  has_many :note_joins, as: :notable
  has_many :notes, through: :note_joins
end

class Checkpoint < ActiveRecord::Base
  has_many :note_joins, as: :notable
  has_many :notes, through: :note_joins
end

Above I set the notejoins model as the polymorphic association :notable, which will use the notableid and notabletype attributes to assign which model and ID the note join belongs to. Then I set the hasmany association on the Note model, through note_joins. This will let me use such methods as Location.first.notes to pull up all the notes that belong to that location. The same applies to Checkpoint.

In the NoteJoin model I need specify that it belongs to the notable polymorphic association and that it also belongs to a Note. I do so by adding the following.

class NoteJoin < ActiveRecord::Base
  belongs_to :notable, polymorphic: true
  belongs_to :note
end

Now for the Note model, it should belong to the notable polymorphic association, and also belong to a user(through the user_id attribute).

class Note < ActiveRecord::Base
  attr_accessible :content, :user_id
  belongs_to :notable, polymorphic: true
  belongs_to :user

  has_many :note_joins
end

At this point everything should work, and through the NoteJoin model I can have a single Note belong to many different Locations and even Checkpoints.

Let’s quickly test this in console.

> Location.first.notes.create(user_id: 1, content: "foobar")
 => #< Note id: 1, content: "foobar", user_id: 1, created_at: "2012-11-30 02:19:24", updated_at: "2012-11-30 02:19:24" >
> Location.first.notes
 => [#< Note id: 1, content: "foobar", user_id: 1, created_at: "2012-11-30 02:19:24", updated_at: "2012-11-30 02:19:24" >]

Great it works, but now I want to see what Locations or Checkpoints this Note belongs to through NoteJoin. To do that I need to update my Note model to include a hasmany locations and checkpoints. Notice I’m using the source and sourcetype option, to pass my polymorphic association :notable, since that’s how we translate which model it belongs to.

And now I can find the locations which my note belongs to.

> Note.first.locations
 => [#< Location id: 1, name: "foobar", created_at: "2012-11-30 02:19:24", updated_at: "2012-11-30 02:19:24" >]

Another problem came up after this point. If I were to create a Note, how could I easily add many Locations to it? My first thought was to create a method that would update the join table, but I knew there had to be an easier way already built in ActiveRecord to do this.

I did some digging around and found a simple solution:

> note = Note.create(user_id: 1, content: "foobar2")
 => #< Note id: 2, content: "foobar2", user_id: 1, created_at: "2012-11-30 02:19:24", updated_at: "2012-11-30 02:19:24" >
> Location.all.each do |location|
     note.locations << location
   end

I’m using << to push location objects into the note.locations array, and ActiveRecord will handle the creation of the NoteJoin record. Pretty neat.

Coachellagram

Sun, 15 Apr 2012 00:00:00 GMT

My co-worker, Chris, twitter.com/ckalima, and I hacked up a little project hours before Coachella 2012 Weekend 1 started: coachellagram.com . It displays, in real-time, a feed of Instagram photos taken at Coachella.

It was a great experience learning about pubsub technology, pushing data to browsers, and working with the Instagram API. Instagram has a very rich API and I’m excited to see what people are going to with it in the future.

Stay tuned for Weekend 2! at coachellagram.com

UPDATE: Coachellagram has been brought down. Sorry if you missed out. We did get some exposure at TNW: Check out Coachellagram: A rocking collection of Instagram shots from Indio, California

Pagination and page cache sweepers in Rails 3

Sat, 15 Oct 2011 00:00:00 GMT

This is how I handle page cache sweeping in Ruby on Rails 3.1 with pagination. This should work with earlier versions of Ruby on Rails.

First of all, I’m using the gem 'will_paginate', '~> 3.0' and have my routes setup to match the url and replace the ?page= with /page/2.

In routes.rb:

match '/articles/page/:page' => 'articles#index'

So now you want to do page caching on your site, but the problem is you can’t sweep the folder and the page numbers inside of them normally within the sweeper.

For example, you have your index action which uses pagination:

def index
  @articles = Article.paginate(:page => params[:page], :per_page => 5, :conditions => {:published => true}, :order => ('created_at DESC'))
end

and have it properly cached and watched by the sweeper:

caches_page :index
cache_sweeper :article_sweeper

When the pages get cached, they’ll be placed in /public/articles/page/1.html and so forth. The problem is getting the sweeper to delete the pages when there’s a change to the Article model.

Solution: Manually remove each page on the sweep.

class ArticleSweeper < ActionController::Caching::Sweeper
  observe Article # This sweeper is going to keep an eye on the article model

  # If our sweeper detects that a article was created call this
  def after_create(article)
    expire_cache_for(article)
  end

  # If our sweeper detects that a article was updated call this
  def after_update(article)
    expire_cache_for(article)
    expire_cache_for_single(article)
  end

  # If our sweeper detects that a article was deleted call this
  def after_destroy(article)
    expire_cache_for(article)
    expire_cache_for_single(article)
  end

  private
  def expire_cache_for_single(article)
    expire_page(:controller => 'articles', :action => 'show')
  end

  def expire_cache_for(article)
    # Expire the index page now that we added or modified an article
    expire_page(:controller => 'articles', :action => 'index')
    # expire the index.html which is created by root_path
    expire_page '/index.html'

    # manually remove pages(1.html, 2.html, etc)
    if File.exist?("#{Rails.root}/public/articles/page") # check to make sure directory exists
      Dir.foreach("#{Rails.root}/public/articles/page") do |entry|
        unless entry == "." || entry == ".." || entry == ".gitignore" # ignore dot files
          File.delete("#{Rails.root}/public/articles/page/#{entry}")
        end
      end
    end
  end
end

Setting up an Ubuntu server with Ruby 1.9.2 and Rails 3.1

Sat, 24 Sep 2011 00:00:00 GMT

I just recently fired up a new Linode VPS, with Ubuntu 10.04 LTS, to stage a Ruby on Rails 3.1 app running Ruby 1.9.2 and I thought I would share my process. Total time took about 1 hour. I’m also going to share my deploy method using Github and Capistrano.

Just for reference, this is what’s going to be installed:

Git RVM Ruby 1.9.2 Ruby on Rails 3.1 sqlite3 Apache 2 Passenger

So the first thing to do is make sure you have your Ubuntu server setup properly with SSH and and all Ubuntu updates. I followed these steps: articles.slicehost.com/2010/4/30/ubuntu-lucid-setup-part-1. Also make sure you know your root password as we’ll need to su back into root later on.

Once you have that done, the second article should end after you update. Next install Git.

$ sudo aptitude build-dep git-core
$ mkdir ~/sources
$ cd ~/sources
$ wget http://git-core.googlecode.com/files/git-1.7.7.2.tar.gz
$ tar xvzf git-1.7.7.2.tar.gz
$ cd git-1.7.7.2
$ ./configure
$ make
$ sudo make install

You can verify the installation by typing:

$ git --version

After git is installed, we need to get the packages for Ruby 1.9.2 to be compatible with Rails 3.1.

$ sudo apt-get install openssl libssl-dev libreadline5-dev zlib1g-dev libncurses5-dev

Next we need to log into root to install RVM.

$ su
$ bash < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer )

Once RVM is installed, time to install Ruby 1.9.2. Make sure you’re still logged in as root(through su). You’ll need to source the RVM script every time you log into root. Normally you wont be doing this often enough to have it auto load, but you can if you want to.

$ source /usr/local/rvm/scripts/rvm

Now you should have access to the command: rvm

Install packages for Ruby 1.9.2 that Rails 3.1 will require.

$ rvm pkg install readline
$ rvm pkg install iconv
$ rvm pkg install openssl

Now you can install Ruby 1.9.2

$ rvm install 1.9.2 -C --with-openssl-dir=/usr/local/rvm/usr,--with-iconv-dir=/usr/local/rvm/usr,--with-readline-dir=/usr/local/rvm/usr

Once that’s finished, set Ruby 1.9.2 as the system default.

$ rvm use 1.9.2 --default

Then exit root su.

$ exit

You should be back to your normal user. Check to make sure rvm is loaded by typing ‘rvm’. If nothing happens exit SSH and log back in. At this point, Ruby should be installed:

$ ruby -v
-> ruby 1.9.2p290 (2011-07-09 revision 32553) [i686-linux]

Next install Bundler.

$ rvmsudo gem install bundler

Before continuing, make sure your project is on a github repository.

To be able to deploy the code from your github repo to this server, you need to have an rsa key that will be added to your github repository deploy key setting.

$ cd ~/.ssh
$ ssh-keygen -t rsa -C "youremailaddress@whatever.com"

Once that’s done, copy the contents of the id_rsa.pubkey.

$ cat id_rsa.pub

The output should look like: ssh-rsa AASasdnl31nADLKAnrqppq...oeqnAE1== youremailaddress@whatever.com

Copy that to your github project deploy key:

You can verify that you did it correctly:

$ ssh -T git@github.com
-> Hi aaronvb! You've successfully authenticated, but GitHub does not provide shell access.

Next, Ubuntu dependencies for sqlite3:

$ sudo apt-get install sqlite3 libsqlite3-dev

The next steps will be on your LOCAL computer. If you already have your own way of deployment you can skip this. Install capistrano, or update to the latest version, and update to the newest bundler. The reason why you need the latest bundler is so that capistrano can work with the new asset pipeline in Rails 3.1. Let me explain how this can work.

You have two options when deploying your app with the new asset pipeline. Compile your assets locally and include the files in your push to your repository, then deploy normally. Second option, push your code as you normally would to your repository, deploy, and have your assets compile on your server instead of locally. The later is the default way capistrano will handle the deploy, and in my opinion is the easiest way.

On your LOCAL computer:

$ gem install capistrano
$ gem install bundler

Setup capistrano in your app:

$ cd your/app/
$ capify .

Next, change capistrano settings to allow asset pipeline and RVM support. In your app directory, open up the file: Capfileand change it to look like this:

load 'deploy' if respond_to?(:namespace) # cap2 differentiator

# Uncomment if you are using Rails' asset pipeline
load 'deploy/assets'

Dir['vendor/gems/*/recipes/*.rb','vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

load 'config/deploy' # remove this line to skip loading any of the default tasks

$:.unshift(File.expand_path('./lib', ENV['rvm_path'])) # Add RVM's lib directory to the load path.
require "rvm/capistrano"                  # Load RVM's capistrano plugin.

Next configure the capistrano deploy file by opening: your/app/config/deploy.rb. I’ve provided my deploy.rb as a reference:

set :application, "aaronvb"
set :repository,  "git@github.com:aaronvb/aaronvb.git"

set :deploy_to, "~/sites/#{application}"

set :scm, "git"
set :branch, "master"

set :user, "aaronvb"

role :app, "yourserverip:yourserverportforssh"
role :web, "yourserverip:yourserverportforssh"
role :db,  "yourserverip:yourserverportforssh", :primary => true

# if you're still using the script/reaper helper you will need
# these http://github.com/rails/irs_process_scripts

namespace :deploy do
  desc "Restarting Passenger with restart.txt"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "touch #{current_path}/tmp/restart.txt"
  end

  [:start, :stop].each do |t|
    desc "#{t} task is a no-op with mod_rails"
    task t, :roles => :app do ; end
  end
end

Now to setup the directories for capistrano on your server. Still on your LOCAL computer:

$ cap deploy:setup
$ cap deploy:check
-> You appear to have all necessary dependencies installed

Now to deploy JUST the code from your github repository:

$ cap deploy:update_code

Once that finishes, go back to your SERVER. Your app should now be residing in the folder you setup within the capistrano deploy.rb file. Go to that directory. If you used my deploy.rb then it will be in ~/sites/yourappname/.

When you find it, go to /releases/ and go to the only folder in there. It should be a folder named with numbers(datetime stamp of the latest deploy). In that folder you should see your app code from your github repository. The way capistrano works is that every time you deploy your code, it uploads it to a new folder in /releases/ which will be named by a datetime stamp, and then linked to /yourappname/current.

So at this point you should be sitting in your app root directory on your server. The next step is to get your apps gems installed, easy:

$ bundle install

After that finishes, time to install node.js. This is required to compile the assets for the asset pipeline in Rails 3.1. If you decided to compile your assets on your local computer, you can skip this. I followed the instruction from: github.com/joyent/node/wiki/Installing-Node.js-via-package-manager for this part.

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:chris-lea/node.js
$ sudo apt-get update
$ sudo apt-get install nodejs

If you’re using sqlite as the database, you might run into a little problem when deploying. Every time you deploy, your production.db will get removed. This is because your production.db isn’t in your github repository. You have a few options to handle this; have your deploy script copy the production.db from your old release to your latest release, or place your production.db outside of your releases. I use the later option. To do this I created the folder: /sites/yourappname/db/ and in my database.yml config file I changed the path for the production database to: database: /home/yourserverusername/sites/yourappname/db/production.sqlite3.

Time to set the database up in your app. Change directories back to your project.

$ rake db:create RAILS_ENV=production
$ rake db:migrate RAILS_ENV=production

You can test your app now by trying the console: $ rails c production.

Installing Apache 2 and Passenger:

$ sudo aptitude install apache2
$ rvmsudo gem install passenger
$ sudo apt-get install apache2-prefork-dev
$ sudo apt-get install libapr1-dev
$ sudo apt-get install libaprutil1-dev
$ rvmsudo passenger-install-apache2-module

Follow the instructions that Passenger presents after the installation. It should ask you to place a few lines of code at the bottom of /etc/apache2/apache2.conf.

The last part is to create the virtual host for your app:

$ cd /etc/apache2/sites-available
$ sudo nano yoursite.com

Inside that file add. Note: I’m basing this off of my deploy script I provided. If you have your app in a different directory I’m sure you can figure this out. The DocumentRoot and Directory MUST point to the public folder in the /current/ folder of your capistrano deploy directory:

<VirtualHost *:80>
  ServerAdmin admin@yourdomain.com
  ServerName  yourdomain.com
  ServerAlias www.yourdomain.com

  DocumentRoot /home/yourserverusername/sites/yourappname/current/public

  <Directory /home/yourserverusername/sites/yourappname/current/public>
    AllowOverride all
    Options -MultiViews
  </Directory>
</VirtualHost>

Save the file, disable and enable a few things, then restart apache2.

$ sudo a2dissite default
$ sudo a2ensite yoursite.com
$ sudo a2enmod rewrite
$ sudo apache2ctl restart

Hope this helps. It’s quite long and forgive me if I’ve misspelled something or skipped a step. Please let me know if you have any troubles.

Rails caching and undefined class/module

Sun, 17 Jul 2011 00:00:00 GMT

Note: This doesn’t affect anything in production as classes are cached at load, see config/environments/production.rb: config.cache_classes = true

I’ve run into this annoying little problem every time I start work on implementing caching into my rails projects using the built in Rails.cache in Activerecord. If you are using Rails.cache, and you are using it to store objects using Marshal.dump and Marshal.load, which is default, I’m sure you’ve experienced this error. I’m also pretty sure you’ve googled it and were a little confused as to why it keeps happening.

You write an object to the Rails.cache, Rails.cache.write(“post#{@post.id}”, @post), and then read the object, Rails.cache.read(“post#{@post.id}”), you get the object. Now this time you write the object to cache, restart your rails app, and try to read the object, but now you’ve received an error, ArgumentError: undefined class/module YOUR_MODEL. This is because rails is trying to Marshal the object, or model in this case, but rails hasn’t loaded the model yet!

Go ahead, test it:

$ rails c
> @kitten = Kitten.first
> Rails.cache.write("kitten_#{@kitten.id}", @kitten)
=> "OK"
> Rails.cache.read("kitten_1")
=> #<Kitten id: 1, cute: "no">
> exit

$ rails c
> Rails.cache.read("kitten_1")
=> ArgumentError: undefined class/module Kitten
> Kitten
=> Kitten(id: integer, cute: string)
> Rails.cache.read("kitten_1")
=> #<Kitten id: 1, cute: "no">

So the solution is to just load all the damn models at load. I thought about using the production value, config.cache_classes = true, in the development environment, but then I would have to restart my server every time I modified any code. Then I thought why can’t I just require it in an initialize file that only runs if it’s in development mode.

Create a file in your config/initializers folder and put this in:

if Rails.env == "development"
  Dir.foreach("#{Rails.root}/app/models") do |model_name|
    require_dependency model_name unless model_name == "." || model_name == ".."
  end
end

Recurring resque and redis with cron

Thu, 12 May 2011 00:00:00 GMT

Moving one of my projects over from delayed_job to resque/redis, for reasons I wont go into here, I needed to have a few of my workers on a cron job. I was initially going to use the resque-scheduler plugin, but the fact that it runs as a daemon made me a little nervous. I didn’t want to worry about watching the scheduler process for memory leaks and or crashes, and cron is a proven, reliable, scheduling service in itself.

Basically, I took the same concept from my other post, Recurring delayed_job with cron and applied it to redis, instead of using mysql as I did with delayed_job. It works by manually injecting jobs into the queue without using the Rails environment, saving memory, cpu, and time, and also able to be run externally by the system cron.

In this example I’m going to use a Payment model, which holds payment information, and an ‘update’ method which should be run nightly at 0000. I’m using Rails 3 as well, which uses the ‘mysql2’ gem.

This is the resque worker.

class UpdatePayment
  @queue = :update_payment

  def self.perform(payment_id)
    payment = Payment.find_by_id(payment_id)
    if payment
      # update payment logic goes here
    end
  end
end

This is the ruby file that the cron will run. I usually place this in root.rails/lib/crons folder.

require 'redis'
require 'mysql2'

# assuming redis is running on the default port.
# if not, example: redis = Redis.new(:host => "10.0.1.1", :port => 6380)
redis = Redis.new

# Make sure queue exists, if not create it. When clearing a queue with the resque web interface, resque removes the queue, so here we just check to make sure it exists.
if redis.sismember('resque:queues', 'update_payment') == false
  redis.sadd('resque:queues', 'update_payment')
end

# Mysql DB information
client = Mysql2::Client.new(:host => "localhost", :username => "root", :database => 'your_project_development')

# query the db for all payment records
results = client.query("SELECT `payments`.* FROM `payments` ORDER BY id asc")

# create a job in the update_payment queue that will update each payment, pass each payment id
results.each do |row|
  redis.rpush('resque:queue:update_payment', "{\"class\":\"UpdatePayment\",\"args\":[#{row['id']}]}")
end

And this is what goes in the crontab, which can be accessed by typing ‘crontab -e’ in the console.

# m h  dom mon dow   command
0 0 * * * /usr/bin/ruby /path/to/your/file/cron_payment_update.rb

For reference:

Redis - redis.io

Resque - github.com/resque/resque

Resque Intro - github.blog/2009-11-03-introducing-resque

Redis Ruby Gem(should get installed when installing resque) - github.com/redis/redis-rb

Ajax checkbox in Ruby on Rails 2.3

Sun, 06 Mar 2011 00:00:00 GMT

Neat trick if you want to use Ajax on a checkbox and don’t want to create a separate function for onclick.

# Note model has a column, :important, :boolean, :default => false

# This goes in your HTML erb

<% @user.notes.find(:all, :order => 'updated_at DESC').each do |user_note| %>
<% remote_form_for user_note, {:url =>
  remote_toggle_important_user_note_path(:user_id => @user.id, :id => user_note.id)} do |f| %>
<%= f.check_box :important, :onclick => "$('edit_user_note_#{user_note.id}').onsubmit()" %><%= f.label :important %>
<% end %>
<% end %>

# routes

map.resources :users do |user|
  user.resources :notes, :member => { :remote_toggle_important => :put }
end

# notes controller

def remote_toggle_important
  @note = UserNote.find_by_id(params[:id])
  if @note
    @note.update_attributes(params[:user_note])
  end
end

Recurring delayed_job with cron

Wed, 27 Oct 2010 00:00:00 GMT

Do you need a delayed_job job to happen at certain times or intervals?

This one is easy and I use it quite often - particularly to scrape data at certain times during the day, and generate nightly statistics and reports.

Why would I use delayed_job to handle recurring jobs over rake task? Every time a rake task is issued, a new rails instance is spawned which takes time and memory to start and run. If a delayed_job daemon is already running, using that will save both. In my opinion, running a rake task that uses the rails environment is usually very costly in production and should be avoided if possible.

#!/usr/bin/env ruby

# put this somewhere in your project, ie: /lib
# use:
# crontab -e
# */30 * * * * /usr/bin/ruby /your_rails_project/lib/this_file.rb
# that will insert this job into your delayed_job queue every 30 minutes.

require 'rubygems'
require 'mysql'

def parse_args(args)
  if args.empty?
    str = '[]\n\n'
  else
    str = '\n-'
    args.each do |k,v|
      str += ' :' + k.to_s + ': ' + v.to_s + '\n'
    end
  end
  return str
end

# Connect to database
# replace these values with your own:
# DB_USER is your database user
# DB_PASSWORD is your database user password
# DATABASE_NAME is your database name, ie: sample_app_development

dbh = Mysql.real_connect("localhost", "DB_USER", "DB_PASSWORD",
  "DATABASE_NAME")

# Get the current time in db format
db_time = Time.now.strftime("%Y-%m-%d %H:%M:%S")

# Insert data into table
# replace these values with your own:
# YOUR_MODEL is your model
# YOUR_METHOD is your method, or function
# YOUR_ARGUMENTS are your agrument(s), ie {:nws => 4, :asdf => "haha"}
# leave blank if no arguments

model = "YOUR_MODEL"
model_method = "YOUR_METHOD"
args = parse_args({YOUR_ARGUMENTS})

dbh.query("INSERT INTO `delayed_jobs` (`failed_at`, `locked_by`, `created_at`, `handler`, `updated_at`, `priority`, `run_at`, `attempts`, `locked_at`, `last_error`) VALUES(NULL, NULL, '#{db_time}', '--- !ruby/struct:Delayed::PerformableMethod \nobject: LOAD;#{model}\nmethod: :#{model_method}\nargs: #{args}', '#{db_time}', 0, '#{db_time}', 0, NULL, NULL)")

Facebook Connect Javascript Cookie and Ruby on Rails

Mon, 25 Oct 2010 00:00:00 GMT

Update: I’m currently working this into a gem. Feel free to watch/use/contribute on github: github.com/aaronvb/fb_js_connect

I wrote this class to verify(using Facebook’s cookie verification article) and parse the Facebook Javascript Connect cookie.

This can definitely work without Ruby on Rails, but you may need to change a few things.

Reference: Facebook Connect Javascript SDK developers.facebook.com/docs/reference/javascript/

To use this in Ruby on Rails, save this file into your RAILS_ROOT/lib folder. If you alter or use this code please contribute back!

require 'digest/md5'
class FbJsConnect
  # When using the Facebook Javascript SDK to connect users to your site,
  # a cookie will be placed in your applications session store. This class
  # will verify your cookie, tell you if you're connected, and provide
  # you with a User and other information

  #
  # cookie = cookies["fbs_YOUR FACEBOOK APP ID"]
  # fb = FbJsConnect.new(cookie)
  # fb.connected => true or false
  # fb.verified? => true or false
  # fb.user => User or nil (Assuming you have a User class with the
  # attribute fb_uid)
  # fb.uid if fb.connected => uid if connected
  # fb.access_token if fb.connected => access_token if connected

  # Verification works by combining cookie key + values, minus sig,
  # appending app_secret, then MD5 hashing it.
  # The value of the MD5 should equal the sig value.

  attr_accessor :connected

  def initialize(cookie)
    @app_id = "" # YOUR FACEBOOK APP ID
    @app_secret = "" # YOUR FACEBOOK APP SECRET KEY
    fb_cookie = cookie
    if fb_cookie
      # remove any quotes in cookie. Facebook puts it's cookie in
      # quotes for some reason
      fb_cookie.gsub!('"','')
      # split cookie values into array by '&' symbol
      fb_cookie = fb_cookie.split("&")
      fb_cookie_hash = Hash.new
      fb_cookie.each do |c, i|
  	 key_value_arr = c.split("=") # split value into kv hash
  	 fb_cookie_hash[key_value_arr[0]] = key_value_arr[1] # add to new hash
      end

      @fb_cookie_hash = fb_cookie_hash # for verification

      # apply connected and fb_cookie_hash to class attributes IF verified
      if self.verified?
        @attributes = fb_cookie_hash
        self.connected = true
      end
    else
      # no cookie found so return false
      self.connected = false
    end
  end

  def verified?
    Rails.logger.debug("\nFB: digested sig: #{process}")
    Rails.logger.debug("FB: sig: #{@sig}\n\n")
    if process == @sig
      return true
    else
      return false
    end
  end

  def user
    # find user with UID
    #
    user = User.find(:first, :conditions => {:fb_uid =>
      @fb_cookie_hash["uid"]})
    if user
      return user # a user was found with UID
    else
      return nil # a user was not found
    end
  end

  def method_missing(name, *args)
    attribute = name.to_s
    if attribute =~ /=$/
      @attributes[attribute.chop] = args[0]
    else
      @attributes[attribute]
    end
  end

  private
  def process
    @sig = @fb_cookie_hash['sig'] # get sig to compare final hash
    payload = []
    @fb_cookie_hash.each do |k, v|
      payload.push "#{k}=#{v}" unless k == "sig"
    end
    digest = Digest::MD5.hexdigest(payload.sort.to_s + @app_secret)
    return digest
  end

end

PDFKit and wkhtmltopdf without X Server, using qt

Thu, 15 Jul 2010 00:00:00 GMT

If your PDFKit and wkhtmltopdf is working on your development server, probably in OSX, but not working on your production server, which is probably running linux, it’s because you’re missing X Server.

There are a few ways around this, some emulate X Server, but that seems hacky. I read through the wkhtmltopdf docs and read that a patched qt framework will allow you to use wkhtmltopdf without using X Server.

Here’s what I did on my Ubuntu Box to get this to work (note: the compile time for qt was almost 2 hours, and you’ll also need to have git installed):

sudo apt-get build-dep libqt4-gui libqt4-network libqt4-webkit
sudo apt-get install openssl build-essential xorg git-core git-doc libssl-dev

mkdir ~/sources
cd ~/sources
git clone git://gitorious.org/+wkhtml2pdf/qt/wkhtmltopdf-qt.git wkhtmltopdf-qt
cd wkhtmltopdf-qt
./configure -nomake tools,examples,demos,docs,translations -opensource -prefix ../wkqt
make -j3
make install
cd ..

Next install wkhtmltopdf:

git clone git://github.com/antialize/wkhtmltopdf.git wkhtmltopdf
cd wkhtmltopdf
../wkqt/bin/qmake
make -j3
make install

Run ‘wkhtmltopdf’ in shell and you should see it load correctly instead of seeing the x server error. Running ‘wkhtmltopdf-proxy’ should do the same, which is what PDFKit uses.

Ruby Hash to Class Object and OpenStruct

Mon, 12 Jul 2010 00:00:00 GMT

I’ve been working on a class that’s pulling information from an API and wanted to create dynamic attributes from a hash(json).

I came across this link:

pullmonkey.com/2008/01/06/convert-a-ruby-hash-into-a-class-object

It works pretty well, but I haven’t done much benchmarking. I’ve read in the past that define_method can be slow and has memory leaks.

Following that post, I read the comments and came across OpenStruct:

ruby-doc.org/stdlib-2.5.1/libdoc/ostruct/rdoc/OpenStruct.html

I have it working with what I need but there’s still more testing to be done. Just thought I would share.

Ruby Array to Javascript Array

Wed, 10 Mar 2010 00:00:00 GMT

Need to pass a Ruby Array to a Javascript Array? Here’s a little gem I found a while ago that I’m reusing in my current project.

>> a_ruby_array = ["one", "two", "three"]
=> ["one", "two", "three"]
>> "['#{a_ruby_array.join('\',\'')}']"
=> "['one','two','three']"

Validations For Multiple Nested Model Forms

Sat, 06 Mar 2010 00:00:00 GMT

Here’s a little trick I found when handling multiple nested models in a form that require validations for each model.

Spec time. For example, we have a model, Author, which hasmany Books and hasmany Magazines. Book and Magazine has validations. On a single page, we create a three forms for Book, Magazine, and Author. There is also an option select that a user can choose if the Author has a Book or a Magazine.

Pretty straight forward. Let’s do some code.

#models/author.rb
class Author < ActiveRecord::Base
  has_many :books
  has_many :magazines

  accepts_nested_attributes_for :books
  accepts_nested_attributes_for :magazines

  validates_presence_of :name
end

#models/book.rb
class Book < ActiveRecord::Base
  belongs_to :author
  validates_presence_of :title,
                        :genre
end

#models/magazine.rb
class Magazine < ActiveRecord::Base
  belongs_to :author
  validates_presence_of :title,
                        :genre
end

Basic model setup with validations and acceptsnestedattributes_for. More info on that can be found in the Ruby on Rails API - Nested Attributes.

Moving on to the form setup..

#views/authors/new.html.erb
<% form_for @author do |f| %>
<%= f.error_messages %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.radio_button("media_type", "book" %><%= f.label :media_type_book, 'Book' %> [?]
    <%= f.radio_button("media_type", "magazine" %><%= f.label :media_type_magazine, 'Magazine' %> [?]
  </p>
  <% f.fields_for :books do |book| %>
    <p>
      <%= book.label :title %><br />
      <%= book.text_field :title %>
    </p>
    <p>
      <%= book.label :genre %><br />
      <%= book.text_field :genre %>
    </p>
  <% end %>

  <% f.fields_for :magazines do |magazine| %>
    <p>
      <%= magazine.label :title %><br />
      <%= magazine.text_field :title %>
    </p>
    <p>
      <%= magazine.label :genre %><br />
      <%= magazine.text_field :genre %>
    </p>
  <% end %>
  <%= f.submit 'Submit' %>
<% end %>

That sets up the form with the nested model, now for the controller code.

#controllers/authors_controller.rb
class AuthorsController < ApplicationController
  def new
    @author = Author.new
    books = @author.books.build #this builds the nested form in the view
    magazines = @author.magazines.build #this builds the nested form in the view
  end

  def create
    param_hash = params[:author]
    if params[:author][:media_type] == "book"
      param_hash.delete("books_attributes")
    elsif params[:author][:media_type] == "magazine"
      param_hash.delete("magazines_attributes")
    end
    @author = Author.new(param_hash)
    @author.save
  end
end

Instead of passing the params straight to Author.new, it’s put into a hash variable. Then the params get checked for a title and genre, if empty delete the key from the hash and pass to Author.new. Activerecord wont see the book param and will skip the validations for it.

Adding A Simple Search In 5 Minutes to a Ruby on Rails Site

Sat, 24 Oct 2009 00:00:00 GMT

Prototyping a site in Ruby on Rails and need to add a simple search feature in less than 5 minutes?

Note: This should be for smaller projects because I’ll be using regular SQL queries for searching. If you’re looking for something more advanced, try thinking_sphinx(sphinx engine) or sunspot_solr(java lucene solr engine).

I’ll be using Searchlogic and will_paginate in this demo.

#install searchlogic
sudo gem install searchlogic

#install will_paginate
gem sources -a http://gemcutter.org
sudo gem install will_paginate

Then add the gem to the environment file.

#config/environment.rb

config.gem "binarylogic-searchlogic", :lib => "searchlogic"
config.gem 'will_paginate', :version => '~> 2.3.11'

If you prefer to use github or the plugin version, check out their docs. I also suggest reading their docs to learn more about all the neat functions Searchlogic can do. It’s a very powerful and useful plugin.

We’ll be adding the search to Posts, so let’s start by adding the search route to our posts.

#config/routes.rb

map.resources :posts, :collection => { :search => [ :post, :get ] }

This will give you the path /posts/search GET and POST.

Next add the search method to the PostsController.

#app/controllers/posts_controller.rb

class PostsController < ApplicationController

  def index
    @posts = Post.paginate(:all, :order => 'created_at desc', :per_page => 25, :page => params[:page])
  end

  def search
    if request.post?

      #pass the post into a get with the 'q' param
      redirect_to search_posts_path(:q => params[:search][:query])

    elsif request.get?

      unless params[:q].blank?

        #Searchlogics title_like_all to search within titles in the Post model for any of the words in the query.
        #words in the string are split into an array by spaces
        search = Post.title_like_all(params[:q].to_s.split).descend_by_created_at

        #paginate the results
        @posts = search.paginate(:per_page => 25, :page => params[:page])

        #pass the query object to the view to let the user know what they searched for
        @query = params[:q]

      end

      #render the Post index.html.erb
      render 'index'

    end
  end

end

The last part is the search form in your view.

<p>
	<% form_for :search, :url => { :controller => 'posts', :action => 'search' } do |f| %>
		<%= f.text_field :query, :value => @query%>
		<%= f.submit "Search", :disable_with => "Searching..."  %>
	<% end %>
</p>

Reoccur Scheduling With Starling and Workling

Mon, 20 Jul 2009 00:00:00 GMT

One of the main reasons I first used BackgrounDRb was for it’s ability to use a cron like schedule that could reoccur at whatever time or date I specified. It was very easy infact, by just writing the cron schedule in the config file for the worker. I was satisfied and stuck with BDRb for just that reason, until I realized how much memory BDRb was using.

Not only was I relying on BDRb to handle a worker that would reoccur nightly, I had also moved a lot of features that didn’t require immediate user feedback, to the background(ie. the message system). So not only was I looking for an alternative to handle my daily worker, but also everything else that needed regular background processing.

I did some research on the current background processing methods and came across an excellent blog post that provided me with a clean solution to all my problems. transfs.com/devblog/2009/04/06/goodbye-backgroundrb-hello-workling-starling.

Starling and Workling. There’s tons of information on getting it setup. I recommend watching the Railscast by Ryan Bates - Starling and Workling.

Starling is the queue server, which Workling polls and then executes workers depending on the task. Easy enough to understand. Even better, Starling is a separate process outside of your rails application. Meaning, if your rails application gets restarted or stopped, or if Workling is off, everything in the Starling queue gets saved. That’s great and all, but the fact that Starling is a separate process and is using the the same design as memcache, means I can slip anything I want into the Starling queue outside of my rails application! To rephrase that, I’m able to start a background process in my rails application without having to be in my rails application.

Let’s continue with some code examples.

In my project I have a TemporaryShoppingCart model which handles temporary shopping carts that a user might create while visiting my site. At the start of each day(00:00:00 on my server), I want a background worker to clear the temporary shopping carts, but also create a record of the event and record how many carts were removed. The record would be stored in another model, say RemovedTemporaryShoppingCart so I can access that data later through an admin page. Note, I’m not sure how practical this is for a real application, I’m just creating this example off the top of my head.

The first step is to create a method in TemporaryShoppingCart model that will do everything I stated above.

def self.empty_self
  #create a new record in RemovedTemporaryShoppingCart model with the total carts removed
  RemovedTemporaryShoppingCart.create(:total => self.all.length)
  #deletes all records in this model. http://api.rubyonrails.org/classes/ActiveRecord/Base.html#M002275
  delete_all
end

Nothing fancy there. Trying to keep as much code in the model rather than in the worker. Next I create the Workling worker.

#app/workers/daily_worker.rb
class DailyWorker < Workling::Base
  def empty_temporary_shopping_carts(options)
    TemporaryShoppingCart.empty_self
  end
end

Make sure to restart Workling and the rails application to generate the new worker, and that Starling is running as well. Normally I can start the worker through the rails app by ‘DailyWorker.asyncemptytemporaryshoppingcarts()’ but that’s not what I want to do in my case.

If you watched the Starling Workling Railscast by Ryan Bates, near the end of the cast, Ryan shows you how Starling works by connecting to it through a memcached protocol. My next example is the exact same except with real data to start the worker.

Load up irb from the terminal by typing ‘irb’.

>> require 'rubygems'
=> false *returned false because rubygems is already initialized*
>> require 'memcache'
=> true
>> starling = MemCache.new('localhost:22122') *change ip/port if not default*
=> <MemCache: 1 servers, ns: nil, ro: false>
>> starling
=> <MemCache: 1 servers, ns: nil, ro: false>
>> starling.set 'daily_workers__empty_temporary_shopping_carts', {}
=> "STORED\r\n"
>>

There should be a new record in RemovedTemporaryShoppingCart model which means the worker did the job. Great I just started a worker inside my rails application from outside my rails application. So now how do I get it to run on a schedule? Thankfully in that first link that I mentioned, the author realized they could use the built in cron feature in the linux operating system to run a ruby script similar to what we did in the irb above.

#!/usr/bin/env ruby
#lib/nightly_worker_for_cron.rb
require "rubygems"
require "memcache"

host = "localhost"
port = 22122
starling = MemCache.new "#{host}:#{port}"

case ARGV[0]
  when 'empty_temp_carts'
    starling.set 'daily_workers__empty_temporary_shopping_carts', {}
end

This file can be anywhere but for ease of editing access I threw it in my ‘lib’ folder and also named it ‘nightlyworkerfor_cron.rb’.

Now to add it to my crontab I just type ‘crontab -e’ in my console and insert ‘0 0 * * * ruby /thepathtomyrailsapp/lib/nightlyworkerforcron.rb emptytempcarts’ on any line then save. Done!

I wrote this to mainly show people how useful Starling and Workling can be. It might not be as easy to setup as delayedjob or straight forward, but there are a lot of neat things that it can do that delayedjob cannot. Instead of trying to compare which one of these background methods is better, decide on what you NEED your application to do and figure out which one will work for you.

Paperclip Amazon S3 Background Upload Using Starling and Workling

Sun, 19 Jul 2009 00:00:00 GMT

I spent several days searching for ANY information on background uploads to Amazon S3 with Paperclip and I just couldn’t find anything concrete. There were a few posts on the Paperclip Google Code board that talked about it but with no clear examples of how to do it.

Anyway, we know it’s very easy to add S3 support into Paperclip, and there’s tons of information on setting that up. It works flawless and it’s simple, yet your upload now takes twice as long in comparison to using your servers filesystem for storage. The problem is pretty obvious and I’ll explain why in this example in case you’re wondering.

Example A(filesystem)

User uploads a file
Application re sizes file into thumbnails

Example B(Amazon S3)

User uploads a file
Application re sizes file into thumbnails
Application then connects to Amazon S3 and uploads thumbnail
Repeat 3 until each thumbnail is uploaded

As you can see in Example B, the more thumbnails you need the longer your upload is going to take because Paperclip needs to send each file to Amazon S3 one at time. All the while a user is still waiting for the website to respond back. If you use New Relic RPM you’ll notice that your Apdex gets destroyed by this. Which is basically telling you that a user is waiting longer than a second(more like 3-5 seconds) for an upload process.

Although there’s nothing that can be done about the initial upload, why does the user need to wait around while the application uploads to Amazon S3(which could take 2-5 seconds longer)?

I worked on several ways to get Paperclip to process it’s Amazon S3 storage in the background that led me to more complications than a solution. I realized I wanted to try my best to not modify the original Paperclip code so there wouldn’t be any problems updating Paperclip in the future.

The next idea was to have some sort of ‘syncing’ to Amazon S3 system. Let Paperclip upload to the filesystem, then have a background process move the files to Amazon S3, then delete the files off the filesystem.

Let’s assume you already have your rails app, starling/workling installed and running, and paperclip installed and working on the model of your choice. Let’s also assume we’re applying this to a User model and the attachment is an avatar. You’ll also need the aws-s3 gem(http://amazon.rubyforge.org) and add that to your environment.rb.

sudo gem install aws-s3

#environment.rb
config.gem 'aws-s3', :lib => 'aws/s3'

First let’s modify the Paperclip settings in our User model.

has_attached_file   :avatar,
                    :styles => { :large => "800x800&>", :medium => "300x300&>", :thumb => "100x100#", :tiny => "50x50#", :really_tiny => "25x25#"},
                    :storage => :filesystem,
                    :url => "http://s3.amazonaws.com/webapps3bucketname/:attachment/:id/:style/:basename.:extension",
                    :path => "tmp/paperclip_uploads/:attachment/:id/:style/:basename.:extension"

The :url is set to the Amazon’s S3 link so that when we generate urls with object.avatar.url(:medium) it will output the Amazon S3 url instead of the filesystem path. I also set the :path to the location I want the file to be uploaded to and re sized. I chose the tmp folder since it seemed appropriate. (Note: I’m considering moving the location to system because it’s a shared folder across capistrano.)

A new column is needed for this model to let us know when the background uploading is being processed.

script/generate migration add_processing_upload_to_user processing_upload:boolean
rake db:migrate

It’s your choice if you want to set the default to true, assuming upon creation of a User record that an upload will be included. In my case, a user gets created first and if the user decides to upload an avatar, it’s done through the user settings later, so I set the default to false.

Next we’ll setup the worker which will handle the upload to Amazon S3.

#app/workers/paperclip_background_upload_worker.rb
class PaperclipBackgroundUploadWorker < Workling::Base
  def upload_to_amazon_s3(options)
    logger.info("Begin uploading #{options[:type]} to Amazon S3...")
    #find the user and the avatar
    @user = User.find_by_id(options[:id])
    @avatar = @user.avatar

    #connect to amazon s3
    logger.info("Connecting to Amazon S3...")
    AWS::S3::Base.establish_connection!(
                                        :access_key_id     => 'ACCESSKEYID',
                                        :secret_access_key => 'SECRETACCESSKEY'
                                        )
    #first we store the original
    logger.info("Uploading original image...")
    AWS::S3::S3Object.store(@avatar.path_s3(:original), open("#{RAILS_ROOT}/#{@avatar.path(:original)}"), 'YOUR_AMAZONS3_BUCKET_NAME', :access => :public_read)
    #then thumbnails
    @avatar.styles.each do |key, value|
      logger.info("Uploading #{key} image...")
      AWS::S3::S3Object.store(@avatar.path_s3(key), open("#{RAILS_ROOT}/#{@avatar.path(key)}"), 'YOUR_AMAZONS3_BUCKET_NAME', :access => :public_read)
    end
    #start clean up, files first then directories
    logger.info("Removing original from local filesystem...")
    #remove original file
    File.delete("#{RAILS_ROOT}/#{@avatar.path(:original)}")
    #remove thumbnail files
    @avatar.styles.each do |key, value|
      logger.info("Removing #{key} from local filesystem...")
      File.delete("#{RAILS_ROOT}/#{@avatar.path(key)}")
    end
    #delete directories
    logger.info("Removing directories...")
    #remove original folder
    Dir.delete("#{RAILS_ROOT}/tmp/paperclip_uploads/avatars/#{@user.id}/original")
    #remove thumbnail folders
    @avatar.styles.each do |key, value|
      Dir.delete("#{RAILS_ROOT}/tmp/paperclip_uploads/avatars/#{@user.id}/#{key}")
    end
    #remove avatar user_id folder
    Dir.delete("#{RAILS_ROOT}/tmp/paperclip_uploads/avatars/#{@user.id}")

    #set processing to false now that we are finished
    @user.processing_upload = false
    @user.save

    logger.info("Finished uploading to Amazon S3.")
  end
end

This is pretty simple. The worker uses the aws-s3 gem to upload the files and once it finishes they get removed from the filesystem. You’ll need to include your Amazon S3 Access Key ID and Secret Access Key and also fill in your Amazon S3 Bucket name that the files are getting uploaded to where it says ‘YOURAMAZONS3BUCKETNAME”. You may also notice a new method named ’paths3’ which I had to add to Paperclips attachment file.

Open up the ‘attachment.rb’ file in the ‘vendor/plugins/paperclip/lib/paperclip/’ folder. Under ‘def initialize’ add the two ‘@path_s3 =’ lines code(around line 47):

      @dirty             = false
      #a new hash value and key for the s3 path we define in the model
      @path_s3           = options[:path_s3]
      @path_s3           = @path_s3.call(self) if @path_s3.is_a?(Proc)

      normalize_style_definition
      initialize_storage

Then scroll down, in the the same file, to about line 113 where there’s a method named ‘path’(def path style = nil #:nodoc:). Under that method we are going to make a new one.

    def path style = nil #:nodoc:
      original_filename.nil? ? nil : interpolate(@path, style)
    end

    #Added this for background uploading.<br />    #This returns the path to the attachment in the s3 bucket.
    def path_s3 style = nil
      original_filename.nil? ? nil : interpolate(@path_s3, style)
    end

This will let us define a new key value(:paths3) in the hasattachedfile hash in the User model. If you’re wondering what the ‘paths3’ does, it determines the path within the Amazon S3 Bucket that the file gets saved to and by having it in our hasattachedfile options, we can change this if we need to later on. Which I’ll show you next.

Back to the User model, we add a new line in the hasattachedfile hash:

has_attached_file   :avatar,
                    :styles => { :large => "800x800&>", :medium => "300x300&>", :thumb => "100x100#", :tiny => "50x50#", :really_tiny => "25x25#"},
                    :storage => :filesystem,
                    :url => "http://s3.amazonaws.com/webapps3bucketname/:attachment/:id/:style/:basename.:extension",
                    :path => "tmp/paperclip_uploads/:attachment/:id/:style/:basename.:extension",
                    :path_s3 => ":attachment/:id/:style/:basename.:extension"

Don’t forget to restart the server since the Paperclip plugin was modified and also restart workling to generate the new workling we created earlier.

The last part is to call the worker through Starling. In this example, the file upload is being handled normally through the User controller and the Update action.

def update
  @user.attributes = params[:user]
  #only set the processing_upload to true if an avatar file was uploaded
  #if the file doesn't pass valiations, @user wont get saved and processing_upload stays false
  @user.processing_upload = true if params[:user][:avatar]
  if @user.save
    flash[:notice] = "Your profile was successfully updated!"
    #tell starling to start the background worker only if processing_upload is true
    #the id of the user gets passed to let our worker know which files to upload
    PaperclipBackgroundUploadWorker.async_upload_to_amazon_s3(:id => @user.id) if @user.processing_upload
    #redirect to the edit page
    redirect_to edit_user_path
  else
    #save failed
    redirect_to :action => 'edit'
    flash[:notice] = "Something went wrong while trying to update your profile."
  end
end

That’s it! The next step is to create a way to let your users know that the file is being processed. All you have to do is check if processing_upload = true.

This is my first tutorial type of post so bear with me if I missed anything and please let me know asap! Also if you have any experience doing something similar to this, let me know!