The Heart Bit

AWS-SSH - Easy SSH to EC2 servers

Tue, 18 Aug 2015 07:50:00 +0000

Currently I work at Buzzstarter Inc., and I’ve been working on the web app, which infrastructure is hosted in Amazon Web Services. In general, we don’t really need to SSH into EC2 instances because the deployment pipeline is fully automated with OpsWorks, but in many cases it’s still useful to quickly jump into servers console. Here are some reasons you might want to do that:

Check application health in detail when information provided by other services such as NewRelic, AirBrake, CloudWatch, Rollbar were not enough
Perform Quality Assurance at system-level. E.g.: QA deployment changes that cause system packages to be updated
Verify responsiveness of a particular server

The problem

Our environments are very dynamic - we often make changes in our servers configuration and sometimes, due to unexpected issues, we have to shutdown or spin up a new EC2 instance, so our servers are constantly changing IP addresses. A while ago, I used to keep my SSH config (~/.ssh/config) with a mapping of server names with their IP addresses, but I was getting tired of updating that file due to constant changes.

You probably heard about AWS-CLI - a tool that allows you to interact with Amazon infrastucture via command-line. Unfortunately, there’s really no easy way to make use of that to get access via SSH to your instances.

For a while I was really dreaming of being able to SSH to any EC2 instance with just a friendly name in hand. It would be also pretty cool to list the instances per Stack (configured in OpsWorks), so that I could quickly jump into every server of one environment - let’s say I want to SSH into servers of Staging environment.

The solution

Yesteday I launched AWS-SSH - it’s a simple tool that made my dream come true. I think it’s a lot easier to show how useful is that by showing examples, so check out below how to setup and use it.

Setting up the tool

NOTE: the CLI can be run through both aws-ssh and awssh.

Make sure you have aws-cli installed and configured. Follow guide in https://github.com/aws/aws-cli
Install aws-ssh gem: gem install aws-ssh
Create a file named .awssh with default options:

```
profile: your-app-aws-cli-profile-name
region: us-east-1
user: custom-machine-user-for-ssh
```

Examples

List servers filtered by name staging. Here we use option -so, which is the short of --show-only, i.e., it will list only, and not try to SSH.

```
$ awssh -so staging
ssh [email protected] => Buzzstarter Staging - staging-app
ssh [email protected] => Buzzstarter Staging - staging-worker

# All hosts in one-line (suitable for CSSH):
[email protected] [email protected]
```

SSH into first production app server. Here we use regex to filter server hostname.

```
$ awssh prod.*app1
user@prod-rails-app1:~$ # you can now interact with this server from now on
```

In first example, you can observe the last line of the output contains a whitespace-separated list of servers. We can use that as an input to a Cluster-SSH tool in order to SSH into multiple servers at once. I’m gonna use tmux-cssh for example:

```
$ tmux-cssh $(awssh -so prod.*app | tail -1)
```

The result is:

![](/assets/images/tmux-cssh.png)

If you are user of OpsWorks, you can also filter by Stack name: The -s is the short of --stack, which value is a regex:

```
$ awssh -so -s qa
ssh [email protected] => Buzzstarter QA - testing-app
ssh [email protected] => Buzzstarter QA - testing-worker

# All hosts in one-line (suitable for CSSH):
[email protected] [email protected]
```

More information: https://github.com/buzzstarter/aws-ssh

Contributions are welcome!

Happy coding, folks!

SQLite thread-safety check

Sat, 11 Apr 2015 02:50:00 +0000

Before start using an RDMS we should always evaluate which are its constraints and how we are planning to use it.

We often implement multithread processes that access database. One simple example is acessing a database on a multithreaded application server, such as Puma, or on a multithreaded background processing framework, such as Sidekiq.

When running code that access a database on such multithreaded environments, we need to verify if the database is ready to accept connections from multiple threads and, sometimes, if a single connection can be shared with multiple threads.

A real scenario

Recently, I was working on an application that uses in-memory SQLite and the team decided to migrate from Resque - a process-based background processing framework - to Sidekiq, which is thread-based. This change begs the question:

Is SQLite thread-safe?

SQLite was built to work on three diferent modes in this regard and the mode is defined by a compilation flag named THREADSAFE. It can have three values:

THREADSAFE=0: It’s unsafe to use SQLite in a multithreaded application - it ommits all mutexing code on the compiled code;
THREADSAFE=1: It’s safe to use SQLite in a multithreaded application and multiple threads can use the same connection. This is the default option when compiling SQLite;
THREADSAFE=2: It’s safe to use SQLite in a multithreaded application but each connection must be used by a single thread at a time.

Checking thread-safety level of SQLite installation

In Ruby

Open interactive Ruby console: $ irb and execute:

SQLite3::Database.new(":memory:").
  execute("PRAGMA compile_options").
  map(&:first).find { |option| option =~ /THREADSAFE/ }

# This will produce an output like "THREADSAFE=2"

On a C program

Create a file named sqlite_threadsafe.c with following content:

#include <sqlite3.h>

int main() {
  return sqlite3_threadsafe();
}

Then compile with:

$ gcc sqlite3_threadsafe.c -lsqlite3 -o sqlite3_threadsafe

And finally see the thread-safety level executing:

$ ./sqlite_threadsafe
2 # this is the value used on THREADSAFE compilation flag

On SQLite console

Execute PRAGMA compile_options on SQLite console to see all the compile options, including the THREADSAFE option. E.g.:

$ sqlite3

SQLite version 3.8.5 2014-08-15 22:37:57
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

sqlite> PRAGMA compile_options;
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_LOCKING_STYLE=1
ENABLE_RTREE
OMIT_AUTORESET
OMIT_BUILTIN_TEST
OMIT_LOAD_EXTENSION
SYSTEM_MALLOC
THREADSAFE=2

References:

https://www.sqlite.org/compile.html#threadsafe
http://www.sqlite.org/cvstrac/wiki?p=MultiThreading

Why not use Cron in production

Mon, 16 Mar 2015 13:05:00 +0000

Cron is the simplest way to schedule tasks in *nix systems. Since it’s a mature tool, most of developers and companies have been using it in production without analyzing if it’s suitable for distributed systems.

Why not use Cron?

Most of the web applications today try to benefit from distributed systems properties and I dare to say the most important ones on web are scalability and reliability. So, does Cron provide the reliability expected on web applications? See why not:

Cannot run distributed

Cron jobs are setup in one single computer so, if that computer crashes, the application will miss important scheduled tasks until that server is back up or another server with same crontab configuration is spinned up.
Does not feature backfill

When a server hosting Cron is recovered from a crash, it does not automatically execute the jobs missed while the server was down. If a scheduled task is missed while the server is down, a person will need to execute the missed tasks manually.

Anacron is a tool that works in conjunction with Cron and does not assume the system is running continuously. However, Anacron does not run distributed.
Does not retry failures automatically

Cron jobs are not automatically retried on failure. I’ve seen cron jobs that access a database or an external service. When these fail, a reliable solution would retry the same job after some delay - usually an exponential backoff formula is a good choice to define the delay intervals between retries.

Solutions

For Ruby, there is this Sidetiq gem that runs on top of Sidekiq. Here is an ideal setup to avoid the issues mentioned above:
Use Sidekiq’s automatically retries.
Use Sidetiq’s backfill feature.
Have at least two servers running Sidekiq workers.
Notify the application maintainers when the retries are exhausted. Look how to use Sidekiq’s sidekiq_retries_exhausted feature at Sidekiq Error Handling configuration.
There are some SaaS solutions out there. I found this Crondash tool which has retry support. It also reports errors which is useful to monitor the application health. If you search for online cron job on web, you should find other similar SaaS available.

There should be many other solutions to the same problem. Just remember to choose one that supports retry, backfill and running in multiple servers.

Please comment what solutions you are using for cron jobs that fulfills distributed system properties.

Happy coding!

The beginning of an era

Sat, 14 Mar 2015 00:00:55 +0000

Hi folks! Creating a blog is among my New Year’s resolutions. I’ll be writing about software engineering, doing my best to give unique contributions to software development community.

All the code I write will be shared on My GitHub. Please feel free to contribute with any critics, suggestions and pull requests :)

Happy coding!