ABuisman.com

Building a Vector Database in Ruby Using Hash and PStore

2024-09-29T00:00:00+00:00

Today, I’m sharing a very cool piece of code I’ve written. It is a vector database in pure Ruby that you could use for tiny datasets or to test some things in your console quickly.

First, why would we need a vector database?

A quick aside: what are vector databases, you ask? Why would I want to use one? Vector databases have recently become very popular in AI because they allow us to store “embeddings” in them and find other “embeddings” that are close to them. “Embeddings” are vectors that capture the meaning of text or images as numbers. They are designed in such a way that if the texts of two vectors are ‘similar’ to each other, the meaning of the texts is also close to each other.

So, the vectors “I like pizza” and “I like souvlaki” are more “similar” than “The car has a flat tyre.”

In this post, I will show you how we do these calculations and what these embeddings look like.

Why build one in Ruby?

I needed to play with some semantic search queries in a console without setting up a dedicated Vector database. So I thought, if we leave out a few things that a ‘real’ vector database does, we can probably write a bit of Ruby code that stores documents and vectors in an Array that we can loop through to find items closest to our query.

Very quickly, I thought about inheriting from Hash. Hash is already sort of like a database; it allows us to store items based on an ID and key, and it is Enumerable, so we can loop through items.

However, when you use Hash, you can’t persist your data. So, the next step is to port the Hash database to PStore so that everything can be persistent.

I’ll start with Hash, though and before that with generating embeddings.

Generating embeddings

How do we get these embeddings? OpenAI has a practical API for this, and we can use the great ruby-openai client library by Alex Rudall gem to generate the embeddings.

Let’s create a little wrapper for it to isolate the functionality:

# app/lib/openai_client.rb

class OpenaiClient
  def embedding(text, model: 'text-embedding-3-small')
    response = api_client.embeddings(
      parameters: {
        model: model,
        input: text,
      },
    )

    response.dig("data", 0, "embedding")
  end

  private

  def api_client
    @api_client ||= OpenAI::Client.new(access_token: Rails.application.credentials.openai[:api_key])
  end
end

We use the text-embedding-3-small model, which returns vectors with a length of 1536 tokens (numbers).

Replace Rails.application.credentials.openai[:api_key]) with whatever way you prefer to store your API keys.

Let’s take it for a spin now.

oc = OpenaiClient.new

oc.embedding("Crime")

=>
[-0.001625204,
 0.01763012,
 -0.008460364,
 0.09889614,
 0.026863838,
 # ...

 oc.embedding("Criminal")

[0.029961154,
 0.00703114,
 0.0073463763,
 0.0476418,
 -0.00031073904,
 -0.0002184382,
 # ...

Hmm, as they say in Germany, I only understand train station; that is to say, I don’t understand, but the algorithms probably do.

Let’s work with something easier to read:

godfather = oc.embedding("A coming-of-age story of a violent mafia son and his father's unhealthy obsession with oranges.")
meet = oc.embedding("A funny meeting between a father and a man who can milk just about anything with nipples, not having seen this is a crime.")
big = oc.embedding("A man gets his rug soiled by German nihilists who have no regard of the law.")

query = oc.embedding("Crime movie")

Now, we have a few arrays and need to calculate the cosine similarity. To do this, we need a few methods:

  def dot_product(vector1, vector2)
    vector1.zip(vector2).map { |a, b| a * b }.sum
  end

The dot product is the multiplication of two vectors. This can be calculated like so for vectors a and b:

a · b = a[0] × b[0] + a[1] × b[1] ...

In Ruby, we can ‘zip’ two arrays together like this with the zip method:

vec1 = [1, 2, 3, 4]
vec2 = [4, 5, 6, 7]

puts vec1.zip(vec2).inspect
=> [[1, 4], [2, 5], [3, 6], [4, 7]]

We map through this zipped array of the two vectors, multiply them (a * b) and then sum these multiplied numbers.

puts dot_product(vec1, vec2)

=> 60

This would be enough for the OpenAI vectors since they are normalised. This means that all of the magnitudes are 1. This, in turn, means that the dot product of the two vectors will give their similarity. We can’t always assume this will be the case, so we will continue calculating the cosine similarity.

  def magnitude(vector)
    Math.sqrt(vector.map { |component| component**2 }.reduce(:+))
  end

Here is the magnitude of the un-normalised vectors vec1 and vec2

puts "Magnitude vec1: #{magnitude(vec1)}"
puts "Magnitude vec2: #{magnitude(vec2)}"
=>
Magnitude vec1: 5.477225575051661
Magnitude vec2: 11.224972160321824

Cosine similarity normalizes the dot product by the magnitudes of the vectors, effectively focusing on the direction rather than the magnitude.

Here is how we calculate it.

  def cosine_similarity(vector1, vector2)
    dot_prod = dot_product(vector1, vector2)
    magnitude1 = magnitude(vector1)
    magnitude2 = magnitude(vector2)
    dot_prod / (magnitude1 * magnitude2)
  end

puts cosine_similarity(vec1, vec2)
=> 0.9759000729485332

Now let’s try it on our movies dataset with the query: “Crime movie.”

cosine_similarity(query, meet)
=> 0.22359566594195673
cosine_similarity(query, big)
=> 0.21230758115230425
cosine_similarity(query, godfather)
=> 0.3426902243058039

Firstly, these numbers are hard to interpret. Are they close or far apart? They live in 1500+ dimensional space, after all. Bear with me a little bit. Further down, we will try many more queries, which will make things clearer.

We can see here that The Godfather is the closest, probably because it mentions ‘story’ and ‘mafia’. Meet the Parents is the furthest because it doesn’t mention an actual crime but contains a saying with the word ‘crime’.

Nice! It’s a bit like what we expected. Let’s try another:

query = oc.embedding("Gangster")

cosine_similarity(query, meet)
=> 0.18473859726676306
cosine_similarity(query, big)
=> 0.21540343497659717
cosine_similarity(query, godfather)
=> 0.3779570884570852

The word Mafia has a much stronger effect here because members of the mafia are also called gangsters.

Building the database

Now that we’ve figured out how to generate our embeddings let’s start with the Hash-based database.

class VectorDb < Hash
  def add_item(id, content:, embedding: nil)
    self[id] = { content:, embedding: }
  end
end

This will store an item on a given ID along with its content and embedding.

We now want to store the embedding if we haven’t passed one in yet; for this, we need the OpenAI client:

class VectorDb < Hash
  # ...

  def add_item(id, content:, embedding: nil)
    embedding ||= openai_client.embedding(content)
    self[id] = { content:, embedding: }
  end

  private

  def openai_client
    @openai_client ||= OpenaiClient.new
  end
end

We add the same calculation methods as before to calculate distances between vectors:

class VectorDb < Hash
  # ...

  def dot_product(vector1, vector2)
    vector1.zip(vector2).map { |a, b| a * b }.reduce(:+)
  end

  def magnitude(vector)
    Math.sqrt(vector.map { |component| component**2 }.reduce(:+))
  end

  def cosine_similarity(vector1, vector2)
    dot_prod = dot_product(vector1, vector2)
    magnitude1 = magnitude(vector1)
    magnitude2 = magnitude(vector2)
    dot_prod / (magnitude1 * magnitude2)
  end
end

Now, I implement a search method where this all comes together.

The concept here is that we will generate the query embedding, loop through the stored embeddings, and calculate their similarity to that query embedding.

class VectorDb < Hash
  def search(query)
    query_embedding = openai_client.embedding(query)

    result = each_with_object({}) do |(id, item), results|
      results[id] = cosine_similarity(query_embedding, item[:embedding])
    end

    result = result.sort_by { |_id, similarity| similarity }.reverse.to_h
  end

  #...
end

🧙‍♂️ Lo and behold ✨ a vector database.

Let’s try it out:

vb = VectorDb.new

vb.add_item("The Godfather", content: "A coming of age story of a violent mafia son and his father's unhealthy obsession with oranges.")
vb.add_item("Meet the Parents", content: "A funny meeting between a father and a man who can milk just about anything with nipples, not having seen this is a crime.")
vb.add_item("The Big Lebowski", content: "A man gets his rug soiled by German nihilists who have no regard of the law.")

vb.search("Being criminal")
=>
{"The Big Lebowski" => 0.2902403092098413,
 "Meet the Parents" => 0.20513406388011676,
 "The Godfather" => 0.19525673473056426}

vb.search("Movie about breaking the law")
=>
{"The Godfather" => 0.35009263294726534,
 "The Big Lebowski" => 0.2518669715823097,
 "Meet the Parents" => 0.22558495028153422}

vb.search("Farming cattle")
=>
{"Meet the Parents" => 0.22182056416947168,
 "The Godfather" => 0.12570011314672785,
 "The Big Lebowski" => 0.008469368264364811}

Seems like it works. Let’s add an item to skew the results a bit:

vb.add_item("Snatch", content: "A movie about a bunch of gangsters stealing a diamond and a dog.")

# Rerun our query
vb.search("Movie about breaking the law")
=>
{"Snatch"=>0.5023058260371518,
 "The Godfather"=>0.35009263294726534,
 "The Big Lebowski"=>0.2518669715823097,
 "Meet the Parents"=>0.22558495028153422}

I suspect adding “a movie about” has influenced the search quite a bit here. Which is a good thing. The other descriptions in no way reflected that they were about movies. We can test this assumption by adding yet another movie and rerunning the search:

vb.add_item("Snatch v2", content: "A bunch of gangsters stealing a diamond and a dog.")

vb.search("Movie about breaking the law")
=>
{"Snatch"=>0.5023058260371518,
 "The Godfather"=>0.35009263294726534,
 "Snatch v2"=>0.34647917573476067,
 "The Big Lebowski"=>0.2518669715823097,
 "Meet the Parents"=>0.22558495028153422}

The hypothesis was correct. Who would’ve thought adding more context would give better results? It highlights the need to provide the models with high-quality, well-labeled information.

Ruby’s Vector-type

Now that we understand how this all works, I’ll tell you a little secret. Ruby has a Vector data type!

This is how it works:

require 'matrix'

vec1 = Vector[1, 2, 3, 4]
vec2 = Vector[4, 5, 6, 7]

puts vec1.dot(vec2)
=> 60

puts vec1.magnitude
puts vec2.magnitude
=>
Magnitude vec1: 5.477225575051661
Magnitude vec2: 11.224972160321824

# `magnitude` is aliased as `r` and `norm`

def cosine_similarity(vector1, vector2)
  vector1.dot(vector2) / (vector1.norm * vector2.norm)
end

You can create a vector from an embedding like this:

embedding = openai_client.embedding("My text")
vector = Vector[*embedding]
# or
vector = Vector.elements(embedding)

Update VectorDb to use Vector-types.

We can replace all the calculation methods with the following:

def cosine_similarity(vector1, vector2)
  vector1.dot(vector2) / (vector1.norm * vector2.norm)
end

This leaves us with the following final result:

require 'matrix'

class VectorDb < Hash
  def search(query)
    query_embedding = openai_client.embedding(query)
    query_embedding = Vector.elements(query_embedding)

    result = each_with_object({}) do |(id, item), results|
      results[id] = cosine_similarity(query_embedding, item[:embedding])
    end

    result = result.sort_by { |_id, similarity| similarity }.reverse.to_h
  end

  def add_item(id, content:, embedding: nil)
    embedding = openai_client.embedding(content) if embedding.nil?

    self[id] = { content: content, embedding: Vector.elements(embedding) }
  end

  def cosine_similarity(vector1, vector2)
    vector1.dot(vector2) / (vector1.norm * vector2.norm)
  end

  private

  def openai_client
    @openai_client ||= OpenaiClient.new
  end
end

Let’s rerun the first query to double-check the result is the same:

vb.search("Being criminal")
=>
# Before
{"The Big Lebowski"=>0.2902403092098413,
 "Meet the Parents"=>0.20513406388011676,
 "The Godfather"=>0.19525673473056426}

# After
{"The Big Lebowski"=>0.2902403092098413,
 "Meet the Parents"=>0.20513406388011676,
 "The Godfather"=>0.19525673473056426}

Looking very good.

PStore

So far, since we have been using Hash as a basis for our database, we have had to hit the API every time to get the embeddings again. This not only costs time but also money. It isn’t a big problem for testing similarities between a handful of sentences, but if we could store our data, we’d even be able to use it as a real database.

Enter PStore.

PStore implements a file based persistence mechanism based on a Hash. User code can store hierarchies of Ruby objects (values) into the data store file by name (keys). An object hierarchy may be just a single object. User code may later read values back from the data store or even update data, as needed. ruby/pstore: PStore implements a file based persistence mechanism based on a Hash.

Let’s get started with the implementation. I will call the new database Vstore and add the add_item and cosine_similarity methods.

require 'pstore'

class Vstore < PStore
  def add_item(id, content:, embedding: nil)
    transaction do
      embedding = openai_client.embedding(content) if embedding.nil?
      embedding = Vector.elements(embedding)

      self[id] = { content: content, embedding: embedding }
    end
  end

  def cosine_similarity(vector1, vector2)
    vector1.dot(vector2) / (vector1.norm * vector2.norm)
  end

  private

  def openai_client
    @openai_client ||= OpenaiClient.new
  end
end

As you can see, PStore is very similar to Hash, except that we have to wrap operations in a transaction block.

We need to pass a path to a location when initialising the database, like so:

vdb = Vstore.new("my_vector_store.pstore")

The search method will be a bit different, though, because of how we have to loop over the records:

def search(query)
  query_embedding = openai_client.embedding(query)
  query_embedding = Vector.elements(query_embedding)
  result = {}

  transaction(true) do
    roots.each do |id|
      item = self[id]
      next if !item.key?(:embedding) || item[:embedding].nil?

      result[id] = cosine_similarity(query_embedding, item[:embedding])
    end
  end

  result = result.sort_by { |_id, similarity| similarity }.reverse.to_h
end

We start a read-only transaction with transaction(true) , loop over all the keys in the store by calling roots, and fetch the data with self[id]within the each-block.

Apart from this, the implementation is the same as that of the Hash-based one.

Let’s check the results:

vb = Vstore.new('movies.pstore')

vb.add_item("The Godfather", content: "A coming of age story of a violent mafia son and his father's unhealthy obsession with oranges.")
vb.add_item("Meet the Parents", content: "A funny meeting between a father and a man who can milk just about anything with nipples, not having seen this is a crime.")
vb.add_item("The Big Lebowski", content: "A man gets his rug soiled by German nihilists who have no regard of the law.")

vb.search("Being criminal")
=>
{"The Big Lebowski"=>0.2902403092098413,
 "Meet the Parents"=>0.20513406388011676,
 "The Godfather"=>0.19525673473056426}

# With Hash:
{"The Big Lebowski"=>0.2902403092098413,
 "Meet the Parents"=>0.20513406388011676,
 "The Godfather"=>0.19525673473056426}

Beautiful! We’ve created a vector database that generates the embeddings for the items we put into it and allows us to do semantic search!

Want to learn more?

This post is based on a chapter from my new book, “From RAG to Riches: How to Build Insanely Good AI Applications That Use Your Own Data,” which is out now!

I wrote the book to share all the knowledge I gained from building AI applications that use documents and other data to give far better results.

It irks me that big tech companies are trying to sell simple AI features as expensive add-ons or products while the underlying technology is not so complicated that you wouldn’t understand it. It is just so new at the moment that it will take a long time to figure out how it all comes together.

I’ve combined all my learnings in the book, and after reading it, you should be at the top of the industry in using AI in Rails applications.

Get my new book From RAG to Riches.

Language matters: Building blocks

2023-10-12T00:00:00+00:00

I have been working on a big platform with countless pages. Since its beginning, it was built using Bootstrap for the front end, but maintenance has become burdensome over time. This is due to multiple layers of custom CSS and the reuse of CSS, which has caused changes in one place to have unpredictable effects on others.

Our way out of this situation is Tailwind. Since you write HTML and style in the same files, the styling is gone when you remove the HTML. You can still create custom tweaks per page or element without impacting another.

Bootstrap did have its benefits: it provided developers with a nice set of UI elements to use when building screens. So, we have started developing a set of reusable elements.

There are several approaches to creating such a set of elements in Rails. Partials and helpers are part of the framework, so it made sense to start from there. I discussed this with the developers, and they set off exploring our options.

Very quickly, I started hearing them say things like:

“The project’s goal is to build a set of components.”
“Let’s use View Components by GitHub.”
“Have we looked at building components with React?”
“There are also these things called web components.”

Wait a minute?! How did we go from a problem definition of conflicting CSS files and the lack of reusable elements to a whole set of new paradigms? Then it dawned on me. We had named the project “Tailwind Components”.

I then realised that the word “Components” carries so much baggage; people already have a clear picture of what components are in their heads. It might not be the same for everyone, but when you say, “Let’s build components”, it starts having a life of its own.

Components might be the right solution, but I’d like to find out first what we can do by just writing HTML, helpers and partials smarter; we could write “components” afterwards if we still needed them. This whole first phase was skipped because of the choice of words.

The fix was to start calling them ‘building blocks’. This suddenly leaves room for solutions such as writing a modal in a Ruby partial, hooking a stimulus controller to it for the simple dynamics and documenting in Notion, where you can go to copy and paste the modal. This is not the best solution or what we need to go with, but now it is on the table again, along with what might be the real solution.

So, language matters!

Super fast downloading of big files

2022-10-10T00:00:00+00:00

Using curl or wget for downloading big dumps to your virtual machines is slow! Use aria2 instead.

Today I had to download a 49GB Heroku Postgres backup to an EC2 server to import it into an RDS instance. The defacto way to download files is using curl and wget. The problem with these tools is that you download the file using one connection, and connections are often throttled.

You can see that by looking at my first attempt:

curl -o mydump.dump "https://..."

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 49.9G    0  144M    0     0  12.7M      0  1:07:00  0:00:11  1:06:49 16.5M

16.5M?!? One hour and six minutes!? This is silly, we are on a cloud hosted machine that is probably in the same data center as the source.

Luckily I’ve experienced the olden days of the internet where we’d wait for hours on one mp3, and so I have some experience in ways to download things a bit quicker. One such method was splitting up downloads and downloading through multiple connections simultaneously. I used FlashGet and other tools like it for this.

On servers, I’ve used a tool called aget in the past, but I couldn’t install it on my Ubuntu EC2 machine, so I looked around and found aria2. I installed it, looked at the instructions and came up with this command:

aria2c -o mydump.dump --max-connection-per-server=10 -s 10 "https://..."

[#.... 36GiB/49GiB(74%) CN:10 DL:130MiB ETA:1m41s]

130MiB that is more like it!

The command above lets me download ten chunks at the same time. For some reason, the limit with aria2 is 16, so let’s try that:

With 16 parts:

aria2c -o mydump.dump --max-connection-per-server=16 -s 16 "https://..."

[#... 3.0GiB/49GiB(6%) CN:16 DL:222MiB ETA:3m36s]

222MiB, which means about 4 minutes instead of 1 hour and 6 minutes!

Another advantage of using aria2 is that you can continue where you left off if the connection breaks.

Zero downtime credential updates on Heroku.

2022-08-30T00:00:00+00:00

When you use Rails credentials on Heroku, you probably have your key in your environment variables. Probably RAILS_MASTER_KEY. Let’s say you want to rotate this key. You re-encrypt your credentials, push the code (reboot 1) and then add the env-var (reboot 2). You could also flip these two steps around.

In between these two reboots, your app won’t work since you either have a new credentials file with the old key, or the old credentials file with a new key.

What I want to happen is:

Set the new key in a new environment variable: RAILS_MASTER_KEY_NEW
The rails app will boot again since Heroku reboots after each ENV update and see the new key
The rails app checks if the new key can decrypt the credentials, if not, it falls back to the old key

This is how you can do this:

Add this to your config/application.rb:

class Application < Rails::Application # Don't add this obviously

    # When we've defined RAILS_MASTER_KEY_NEW it means we are rotating the encryption key
    # for our credentials. What we want to do then is:
    # 1. Check if we can decrypt the current credentials file with the new key
    # 2. If we can, we will change RAILS_MASTER_KEY to equal RAILS_MASTER_KEY_NEW
    # 3. If not, we will fallback to the old key, thus leave RAILS_MASTER_KEY alone
    if ENV.fetch("RAILS_MASTER_KEY_NEW", false)
      require 'logger'

      cred_logger = Logger.new($stdout) # Rubocop wanted this. He is the boss.
      credential_path = Rails.root.join("config/credentials/#{Rails.env}.yml.enc")
      begin
        Rails.application.encrypted(credential_path, env_key: 'RAILS_MASTER_KEY_NEW').read
        ENV["RAILS_MASTER_KEY"] = ENV.fetch("RAILS_MASTER_KEY_NEW")
        cred_logger.info "application.rb: Using the new credential key, it works!"
      rescue ActiveSupport::MessageEncryptor::InvalidMessage => e
        cred_logger.info "application.rb: Using the old key"
      end
    end

    #...

That’s it.

This needs to be at the top because Rails will pretty quickly try to load the credentials. Some trial and error will probably reveal the sweet spot, but this works just fine here.

After successfully updating your credentials file, you can remove the new key again. Be sure to set the value of RAILS_MASTER_KEY to that of RAILS_MASTER_KEY_NEW though!

Cron monitoring with Blazer

2022-08-23T00:00:00+00:00

Many apps have recurring jobs and we should track them. There are third-party services to do this, but I like to keep things simple and own my data.

I’ll show you how to keep track of cron jobs within your Rails application and report on them through Slack. We collect logs in your database, so you can use them for other things as well.

When done we can write this SQL:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

And get notifications like these:

Using this UI:

There are two steps to this:

We will build a logging endpoint into our Rails app
We will install the blazer gem for building checks and notifications

Logging endpoint

I’ve taken inspiration from Healthchecks.io:

Run our command and then curl to our logging endpoint:

some_command && curl -fsS -m 10 --retry 5 -o /dev/null https://our-domain.com/job_logs/:token/:name

We match a log to a job through the :name. Then we hash a secret and :name into a :token and use it to prevent people from pushing bogus logs to our database.

Let’s create the table we’ll use first:

# The migration

class CreateJobLogs < ActiveRecord::Migration[7.0]
  def change
    create_table :job_logs do |t|
      t.string :name, null: false

      t.string :status, null: false, default: 'success'

      t.string :error
      t.string :duration
      t.string :output

      t.timestamps
    end
  end
end

The fields :status, :error, :duration and :output might be useful in the future.

Now for the controller. We need it to check the token, and create a log record:

# Route - We use GET instead of POST
get '/job_logs/:token/:name', to: 'job_logs#create'

# The controller
class JobLogsController < ApplicationController
  def create
    @job_log = JobLog.new(job_log_params)

    return render json: { error: 'Invalid token' }, status: :unauthorized unless @job_log.valid_token?(params.fetch(:token))

    @job_log.save!
  end

  private

  def job_log_params
    params.permit(:name, :status, :error, :duration, :output)
  end
end

Our model uses the Rails app’s secret key base to generate a token and looks like this:

class JobLog < ApplicationRecord
  validates :name, presence: true
  validates :status, presence: true

  def token
    Digest::SHA1.hexdigest("#{name}-#{secret}")
  end

  def valid_token?(input_token)
    token == input_token
  end

  private

  def secret
    Rails.application.secrets.secret_key_base
  end
end

Let’s try it out:

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/123/ping_site
  curl: (22) The requested URL returned error: 401

# Let's generate a token:
bin/rails r "puts JobLog.new(name: 'ping_site').token"
  40f958deccb8ba624562a7c81e5609ab66a71b4a

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/40f958deccb8ba624562a7c81e5609ab66a71b4a/ping_site

Set up blazer checks

The blazer readme describes how to install it. I’ve mounted it to /blazer.

Take extra care to set up authentication to keep out prying eyes.

To get Slack notifications working, you will have to set the BLAZER_SLACK_WEBHOOK_URL and also set an action_mailer default_url option like so:

# config/environments/development.rb
config.action_mailer.default_url_options = { host: "localhost:3000" }

Let’s write a query to have a look at all our job logs so far:

If we want to run a job daily, we’ll have to truncate the records to their day. We create the following query:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

Now we create a check based on this query. Notice the Slack channel I specified:

Running cron

Finally set up a cron job to run blazer’s checking command:

rake blazer:run_checks SCHEDULE="1 day"

When this runs, Blazer will check if our query returns a result. If it doesn’t, it sends a Slack notification like so:

Once the check starts passing again, we will also get a notification:

Conclusion

There you have it, we have a neat, low-effort way to track cron jobs from within our app. On top of this, we could build extra features such as API throttling or reports.

Blazer enables you two to make something that otherwise requires quite a bit of coding. So think about what else you could do with it.

Enjoy!

Quick page benchmarks

2022-05-29T00:00:00+00:00

I love optimising performance, be it in databases, scripts or webpages. It can be pretty evident in database queries when you’ve improved the performance, with total runtimes of scripts, etc.

Webpages, however, are a bit more tricky when the differences are relatively small. I’ll show you a quick way to run benchmarks that won’t require you to set up a lot of stuff in your framework but instead works with curl and hyperfine, a command-line benchmarking tool.

Hyperfine is an excellent CLI benchmark tool that can benchmark different commands and shows a nice visual result.

What are we testing?

So let’s first have a quick look at the code I want to demonstrate this with:

# The migration for our data
create_table :posts do |t|
  t.string :title
  t.string :tag

  t.timestamps
end

class PostsController < ApplicationController
  def index
    @posts = Post.where(title: 'Faster is more fun', tag: 'ruby')
  end
end

I filled up the database with 440000 records to play with.

I want to figure out whether adding an index to posts on :title and :tag will make the request faster. I am pretty sure it should.

The example is contrived, but let’s explore it anyway. The view looks like this:

<% @posts.each do |post| %>
  <%= post.id %>
<% end %>

Which looks like this:

Imagine this is a webpage with some authorisation in front of it. How can we benchmark it? We might think of a capybara or rack test, which would work. We can hit it a few hundred times in a Benchmark block in Ruby. We’d then have to get the situation in the test suite right, sign in users, etc. Writing this sounds like a lot of work, but there is an easier way!

We will grab the request from the browser as a curl command, which will include all the cookies we need for authentication, and we don’t have to think about the URL; it will be in there already. The test data is in our development database, which we could also point to a follower or some other place with more data.

We will create a command line script to run the curl request and feed it into hyperfine to benchmark it.

We will then make our tweaks and run hyperfine again to see the ‘after’ result and validate that we made things better, not worse.

Ensure you have hyperfine installed: brew install hyperfine.

So the steps are:

Open up the development console
Go to the Network tab
Copy the request to the page as curl
Paste it in a file called whatever you like, such as benchmark.sh
Run chmod +x benchmark.sh
Run hyperfine to capture the before-result: hyperfine --warmup 10 --min-runs 100 ./benchmark.sh
Make our changes and rerun the benchmark to capture the after-result.

So:

Which looks like this:

curl 'http://localhost:3001/' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Accept-Language: en-GB' \
  -H 'Cache-Control: max-age=0' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: _...' \
  -H 'If-None-Match: W/"2474ec9d05466c6ad84b819e7221085b"' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Sec-GPC: 1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36' \
  --compressed

After making the script executable, we can run hyperfine:

hyperfine --warmup 10 --min-runs 100 ./benchmark.sh

The flag --warmup 10 will first run the request 10 times without measuring. I do this to wake up Rails and the database server a bit. --min-runs 100 will ensure we fire at least 100 requests for our benchmark to have some statistical significance.

The result looks as follows:

Or:

Benchmark 1: ./benchmark.sh
  Time (mean ± σ):      57.6 ms ±   8.7 ms    [User: 2.5 ms, System: 3.6 ms]
  Range (min … max):    47.7 ms …  91.5 ms    100 runs

So we can see that without an index, our request has a mean time of 57.6ms.

Let’s add an index and see if it makes it quicker.

-- We run this on the database:
CREATE INDEX index_posts_title_tag ON posts (title, tag);

Now we rerun hyperfine:

Benchmark 1: ./benchmark.sh
  Time (mean ± σ):      29.5 ms ±   7.2 ms    [User: 2.4 ms, System: 3.4 ms]
  Range (min … max):    23.4 ms …  61.8 ms    100 runs

Impressive! Twice as fast! But more importantly, we have validated our improvement!

Note 1: Be sure to have a lot of data!

There is no point in trying this with just 10 posts. With 10 posts, before the index:

Benchmark 1: ./benchmark.sh
  Time (mean ± σ):      18.4 ms ±   5.9 ms    [User: 2.4 ms, System: 3.4 ms]
  Range (min … max):    12.8 ms …  56.3 ms    146 runs

After the index:

Benchmark 1: ./benchmark.sh
  Time (mean ± σ):      15.6 ms ±   2.1 ms    [User: 2.4 ms, System: 3.3 ms]
  Range (min … max):    13.2 ms …  25.8 ms    157 runs

There is some difference, but it is much more visible with more data.

Note 2: This was a back-end test

Curl will only fetch the HTML, and that is it. It won’t load the HTML, extra resources, etc. So this is a back-end test. If you want to test the whole loading of all the front-end assets, etc., you will have to resort to some different command.

I have not looked into the whole cookie copy-ing to make sure I am still signed in on the page I am testing, but running a headless chrome instance can be done like so:

# First run this to have an easy reference to `chrome` if you are on mac:
alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"

Call pages in the headless browser:

chrome --headless --disable-gpu --dump-dom http://localhost:3001/

Taken from here

Put the chrome --headless... command in the benchmark.sh script, and you can use the same basic principles above.

Conclusion

I’ve found this benchmarking method handy because it is quick to do, and the results are pretty reliable.

In my early past, I’ve tried to sort of blindly optimise things and hope for the best in production. Then in production, it is tough to figure out whether you improved performance by 10% or if there is a difference due to other factors such as server load. Doing a benchmark on your computer is relatively controlled, and it is easier to attribute differences to specific changes you made.

You can use this concept for any framework, which makes it very portable.

Simple query optimisation

2022-05-23T00:00:00+00:00

Query optimisation is one of my favourite things to do because you can get extra performance in response times and because we will use fewer resources in your database.

To optimise queries properly, you have to look at the queries but also know something about your application’s domain. I will take you along in a straightforward one I did today.

So starting from the following query:

EXPLAIN ANALYZE SELECT COUNT(*)
  FROM "bookings"
 WHERE "bookings"."state" = 'confirmed' AND (ends_at < '2022-05-24');

Aggregate  (cost=88226.78..88226.79 rows=1 width=8) (actual time=127.844..127.845 rows=1 loops=1)
  ->  Bitmap Heap Scan on bookings  (cost=12225.25..88146.35 rows=32171 width=0) (actual time=55.488..125.832 rows=39106 loops=1)
        Recheck Cond: (((state)::text = 'confirmed'::text) AND (ends_at < '2022-05-24'::date))
        Heap Blocks: exact=30135
        ->  BitmapAnd  (cost=12225.25..12225.25 rows=32171 width=0) (actual time=49.448..49.449 rows=0 loops=1)
              ->  Bitmap Index Scan on index_bookings_on_state  (cost=0.00..2482.18 rows=40768 width=0) (actual time=21.448..21.448 rows=41685 loops=1)
                    Index Cond: ((state)::text = 'confirmed'::text)
              ->  Bitmap Index Scan on index_bookings_on_ends_at  (cost=0.00..9726.74 rows=198709 width=0) (actual time=26.043..26.043 rows=198490 loops=1)
                    Index Cond: (ends_at < '2022-05-24'::date)
Planning time: 0.616 ms
Execution time: 128.003 ms

We can see from the explain-analyse output that we first do a Bitmap Index Scan on index_bookings_on_ends_at. This means that Postgres scans the index to find the records that match the given where condition Index Cond: (ends_at < '2022-05-24'::date), which led to 198490 rows (rows=198490).

Then a scan is done on Bitmap Index Scan on index_bookings_on_state with the condition ((state)::text = 'confirmed'::text) which led to 41685 rows (rows=41685) .

These things happen because there is an index separately on ends_at and state. This means that PostgreSQL has to figure out which records satisfy both conditions and that information is spread out and needs to be combined again.

Combining, at least as I understand it, happens in the BitmapAnd operation. Then the rows are all fetched from their physical storage and scanned in the Bitmap Heap Scan to see if it is true that they satisfy both conditions, hence the Recheck Cond: (((state)::text = 'confirmed'::text) AND (ends_at < '2022-05-24'::date)).

This sounds like a lot of work. You might have guessed how we can improve this: we have to create a single index for state and ends_at. Let’s see what this does.

We create the index like so:

CREATE INDEX index_ends_at_state ON bookings(ends_at date_ops,state text_ops);

Let’s re-run our query:

Aggregate  (cost=5625.65..5625.66 rows=1 width=8) (actual time=35.506..35.506 rows=1 loops=1)
  ->  Index Only Scan using index_ends_at_state on bookings  (cost=0.42..5545.22 rows=32171 width=0) (actual time=0.241..30.923 rows=39106 loops=1)
        Index Cond: ((ends_at < '2022-05-24'::date) AND (state = 'confirmed'::text))
        Heap Fetches: 0
Planning time: 0.280 ms
Execution time: 35.554 ms

Nice! As expected, this is quicker: from 128.003 ms to 35.554 ms, which is 3.6 times faster or 92ms per call less.

Why care?

Whether it should be or not, this query is being called ~70 times per minute at the time of writing. This means 70 * 0,92 = 6,44 seconds/minute of savings. Pretty good, but more broadly speaking, we save some resources.

But wait, there is more!

So here is where a bit of domain knowledge can help us further improve the query’s speed.

We have defined our index with ends_at first and then state, which means that Postgres will first collect all the index entries that match the ends_at condition and then traverse its tree to reach the correct state.

As you’ll recall, our ends_at condition resulted in 198709 rows in the separate index scan, whereas the condition on state resulted in 40768 rows.

This means that ends_at limits the rows less than state.

Let’s swap the columns in the index to see if this will help.

DROP INDEX index_ends_at_state;
CREATE INDEX index_state_ends_at ON bookings(state text_ops,ends_at date_ops);

Now the query plan looks like this:

Aggregate  (cost=1248.27..1248.28 rows=1 width=8) (actual time=24.197..24.198 rows=1 loops=1)
  ->  Index Only Scan using index_ends_at_state on bookings  (cost=0.42..1167.84 rows=32171 width=0) (actual time=0.085..17.707 rows=39106 loops=1)
        Index Cond: ((state = 'confirmed'::text) AND (ends_at < '2022-05-24'::date))
        Heap Fetches: 0
Planning time: 0.255 ms
Execution time: 24.242 ms

Firstly you might look at the new execution time: 24 ms, which is quicker but could be a coincidence. It will go from 17-30 ms execution time every time you run this. So it feels faster, but is it?

Let’s have a look at the costs:

Separate indexes: Aggregate (cost=88226.78..88226.79 rows=1 width=8)
ends_at, state: Aggregate (cost=5625.65..5625.66 rows=1 width=8)
state, ends_at: Aggregate (cost=1248.27..1248.28 rows=1 width=8)

We can see that the calculated execution costs went from 88k to 1,2k, and swapping the columns made it 4,3x cheaper to run.

So here you have it, a quick exploration of how using indexes according to what your application runs them and a bit of domain knowledge can help you squeeze out some extra speed and resources.

Fixing Ruby openssl errors

2022-02-17T00:00:00+00:00

The other day I wanted to try out Rails 7’s import maps on my new M1 MacBook, but I was blocked by an error telling me that Ruby couldn’t make any SSL requests.

I’ll go through how I fixed this.

The command I tried that gave me an error was:

bin/importmap pin @picocss/pico

I got the following error about SSL:

/Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=52.142.124.215:443 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError)
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/protocol.rb:46:in `ssl_socket_connect'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/http.rb:1048:in `connect'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/http.rb:976:in `do_start'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/http.rb:965:in `start'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/3.1.0/net/http.rb:1530:in `request'
  from (irb):25:in `<main>'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/bin/irb:25:in `load'
  from /Users/abuisman/.asdf/installs/ruby/3.1.0/bin/irb:25:in `<main>'

I could reproduce this in a Ruby console with this code:

require 'uri'
require 'net/http'
require 'openssl'

url = URI("https://duckduckgo.com/")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Get.new(url)

response = http.request(request)
puts response.read_body

I’ve seen this error before and usually it happens when you haven’t compiled ruby properly with openssl. So I first reinstalled [email protected] and openssl@3 and a few times again and again, but I couldn’t fix it.

I was ducking around for some info and I sort of picked up in my searching that this could mean that my CA certificates were outdated.

I’ll start with the fix:

brew uninstall [email protected] --force --ignore-dependencies
brew install [email protected]

The --force --ignore-dependencies is critical. I just ran uninstall without it the first 10 times or so and I didn’t notice that nothing was really being removed because of dependencies. It could be that it was clear from the command output but I do not remember seeing anything relating to dependencies. --force --ignore-dependencies was the thing that made a difference for me.

With the following .zshrc configuration for compilation related things:

export OPTFLAGS="-Wno-error=implicit-function-declaration"

# readline
export LDFLAGS="-L/opt/homebrew/opt/readline/lib"
export CPPFLAGS="-I/opt/homebrew/opt/readline/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/readline/lib/pkgconfig"

# openssl config
export PATH="/opt/homebrew/opt/[email protected]:$PATH" # Might be overkill
export PATH="/opt/homebrew/opt/[email protected]/bin:$PATH"

export LIBRARY_PATH="/opt/homebrew/opt/[email protected]:$LIBRARY_PATH"
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=/opt/homebrew/opt/[email protected]"
export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib:$LDFLAGS"
export CPPFLAGS="-I/opt/homebrew/opt/[email protected]/include:$CPPFLAGS"
export PKG_CONFIG_PATH="/opt/homebrew/opt/[email protected]/lib/pkgconfig:$PKG_CONFIG_PATH"

# libffi
export LDFLAGS="$LDFLAGS:-L/opt/homebrew/opt/libffi/lib"
export CPPFLAGS="$CPPFLAGS:-I/opt/homebrew/opt/libffi/include"
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:/opt/homebrew/opt/libffi/lib/pkgconfig"

Then install ruby. For example with asdf:

asdf install ruby 3.1.0

Oh and how do we know it works?

A colleague of mine pointed me to the bundler documentation and there they mention a nice command:

curl -Lks 'https://git.io/rg-ssl' | ruby

Bundler: How to troubleshoot RubyGems and Bundler TLS/SSL Issues

Before the fix the output looked like this:

Afterwards it looks like this:

This means we have a quick way to test the config without having to boot up a ruby console.

We have to could go deeper

In my recent migration from an i7 MacBook to this new M1 Pro I moved over my dotfiles with my .zshrc config. I figured it was wrong at some point so I had a critical look. I must admit looked ridiculous:

export OPTFLAGS="-Wno-error=implicit-function-declaration"

# readline
export LDFLAGS="-L/opt/homebrew/opt/readline/lib"
export CPPFLAGS="-I/opt/homebrew/opt/readline/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/readline/lib/pkgconfig"

# openssl config
export PATH="/opt/homebrew/opt/[email protected]:$PATH"

export LIBRARY_PATH="/opt/homebrew/opt/[email protected]"
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=/opt/homebrew/opt/[email protected]"
export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib"
export PKG_CONFIG_PATH="/opt/homebrew/opt/[email protected]/lib/pkgconfig"

# libffi
export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"
export CPPFLAGS="$-I/opt/homebrew/opt/libffi/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/libffi/lib/pkgconfig"

So many of these variables are being overwritten in every section. It seems like this stuff accumulated while I was trying out things over time or as I was installing libraries. Who knows. Git shows many of these lines as new since moving to the new Mac.

What is the problem? For example LDFLAGS gets overwritten like so:

export LDFLAGS="-L/opt/homebrew/opt/readline/lib"
export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib"
export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"

So in the end the value is just -L/opt/homebrew/opt/libffi/lib. You can see above that I corrected this by appending :$LDFLAGS" each time. This means that it will add things to the LDFLAGS variable instead of overwriting it. It turns out this cleanup didn’t do much, or at least reverting it didn’t break openssl again.

Interestingly, I had already built Ruby 2.7.4 already and it was showing the same issues. After fixing 3.1.0, 2.7.4 was also fixed. I believe this has something to do with new certificate files being downloaded as I mentioned earlier. My general idea is that reinstalling openssl properly also updated some certificate.

If you look here:

You can see that there is a certificate directory and a certificate file:

SSL_CERT_FILE:  /opt/homebrew/etc/[email protected]/cert.pem
SSL_CERT_DIR:   /opt/homebrew/etc/[email protected]/certs

When I look around in the directory it doesn’t contain anything and when I open the certificate file I see the following:

I have no clue what this certificate’s role is to be honest. I might find out and update here.

Conclusion

The fix was for me as described above. I ran into several github issues, stack overflows, etc. that allowed me to piece together these instructions. I hope my gathering of these solutions will be the real fix for you, or for me when I run into this again and forget.

Crazyton - Better than module_function

2021-12-30T00:00:00+00:00

I created a new gem that gives you the nice syntax of module_functions but with the advantages of classes.

An example:

class ApiStub
  include Crazyton

  def sign_up(email, password, status = 200, response = default_sign_up_request)
    stub_request(:post, "https://some-service.com")
      .with(body: { email: email, password: password })
      .to_return(status: status,  body: response.to_json)
  end

  private

  def default_sign_up_request
    {
      event: :signed_up,
      user_id: 123
    }
  end
end

# Invoke

ApiStub.sign_up("[email protected]", "password123")

You can get it here: GitHub - abuisman/crazyton

Why?

Modules with module_function and are a pretty good way to define helpers and make them available in a nice way:

module AppSettings
	module_function

	def google_analytics_enabled?
	  !!ENV.fetch('GOOGLE_ANALYTICS', false)
	end
end

## Usage:

AppSettings.google_analytics_enabled?

But they aren’t great once things get more complex.

Let’s say that I am creating a few interfaces to some websites. Let’s call them crawlers.

This is fine:

require 'uri'
require 'net/http'
require 'openssl'
require 'nokogiri'

module AbuismanCom
  module_function

  def latest_post
    get_selector_text("https://abuisman.com/", ".posts ul li:first-of-type")
  end

  def get_selector_text(url, selector)
    url = URI(url)

    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE

    request = Net::HTTP::Get.new(url)

    response = http.request(request)
    response.read_body
    document = Nokogiri::HTML.parse(response.read_body)
    document.css(selector).text.split("\n").map(&:strip).join(" ")
  end
end

puts AbuismanCom.latest_post
=> Diffing Rails credentials in Rails 6.0 10/05

But when I add another site, I recognise that I want to reuse some of the code and things are going in the wrong direction:

require 'uri'
require 'net/http'
require 'openssl'
require 'nokogiri'

module AbuismanCom
  module_function

  def latest_post
    HttpHelper.get_selector_text("https://abuisman.com/", ".posts ul li:first-of-type")
  end
end

module HackerNews
  module_function

  def top_post
    HttpHelper.get_selector_text("https://news.ycombinator.com/", "#hnmain .itemlist tr:first-of-type")
  end
end

module HttpHelper
  module_function

  def get_selector_text(url, selector)
    url = URI(url)

    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE

    request = Net::HTTP::Get.new(url)

    response = http.request(request)
    response.read_body
    document = Nokogiri::HTML.parse(response.read_body)
    document.css(selector).text.split("\n").map(&:strip).join(" ")
  end
end

puts AbuismanCom.latest_post
=> Diffing Rails credentials in Rails 6.0 10/05
puts HackerNews.top_post
=>  1.      Tokyo police lose 2 floppy disks containing personal info on citizens (mainichi.jp)

I don’t like the call to HttpHelper.get_selector_text. You can fix that by using extend(HttpHelper) on the other two modules, so that is also ok. But I don’t want to do that for every website module that I create.

I will want to add more helpers to the crawler and a caching feature and I don’t want to keep repeating those things in a list of extend calls. I could get a long way with something like extend CrawlerStuff which in turn extends those things, but then I’d just be avoiding inheritance for the sake of avoiding it or for that sweet sweet HackerNews.top_post syntax.

Insert `include Crazyton`

Instead, I can use inheritance from a base class and define whatever I need there:

require 'uri'
require 'net/http'
require 'openssl'
require 'nokogiri'
require 'crazyton'

class BaseCrawler
  include Crazyton

  def get_selector_text(url, selector)
    url = URI(url)

    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE

    request = Net::HTTP::Get.new(url)

    response = http.request(request)
    response.read_body
    document = Nokogiri::HTML.parse(response.read_body)
    document.css(selector).text.split("\n").map(&:strip).join(" ")
  end
end


class AbuismanCom < BaseCrawler
  def latest_post
    get_selector_text("https://abuisman.com/", ".posts ul li:first-of-type")
  end
end

class HackerNews < BaseCrawler
  def top_post
    get_selector_text("https://news.ycombinator.com/", "#hnmain .itemlist tr:first-of-type")
  end
end

Without all the module_function and extend stuff the crawlers look a lot cleaner. The < BaseCrawler is very basic ruby stuff so I don’t need to explain to others in the team why you’d have to do some unfamiliar module setup. Even the include Crazyton happens only once inside of the BaseCrawler.

What IS strange then is that you would have to explain to the others why top_post is suddenly callable like a class method instead of it being an instance method. In the end though I like how little bootstrapping code there is in the classes.

Because we are using classes and inheritance I could also give the BaseCrawler and interface:

require 'uri'
require 'net/http'
require 'openssl'
require 'nokogiri'
require 'crazyton'

class BaseCrawler
  include Crazyton

  def get_selector_text(selector)
    url = URI(base_url)

    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE

    request = Net::HTTP::Get.new(url)

    response = http.request(request)
    response.read_body
    document = Nokogiri::HTML.parse(response.read_body)
    document.css(selector).text.split("\n").map(&:strip).join(" ")
  end

	private

  def base_url
    raise "You have to implement base_url"
  end
end


class AbuismanCom < BaseCrawler
  def latest_post
    get_selector_text(".posts ul li:first-of-type")
  end

  private

  def base_url
    "https://abuisman.com/"
  end
end

class HackerNews < BaseCrawler
  def top_post
    get_selector_text("#hnmain .itemlist tr:first-of-type")
  end

  private

  def base_url
    "https://news.ycombinator.com/"
  end
end

Ah yes, did I mention private methods? Because modules don’t have those, but classes do.

In summary, with Crazyton you get:

smooth syntax
inheritance
standard looking ruby with a twist
private methods
a free Singleton instance

Have fun!

Heroku differ: see which commits would be deployed

2021-05-13T00:00:00+00:00

Before pushing to Heroku, I’d like to know what will be in the release. Let’s use the power of git to answer this question.

The script I created is the following, I’ll go over it, argument, by argument below.

#!/bin/bash

git log —first-parent heroku/master..master —abbrev-commit —date=relative —format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)'

Firstly the output looks like this:

~/Sites/project (master) -> bin/heroku_diff.sh
3820f5782 - (10 hours ago) Merge pull request #1331 from Project/tech/heroku-differ - Achilleas (HEAD -> master, origin/master, origin/HEAD)
996c35a69 - (10 hours ago) Merge pull request #1307 from Project/feature/get-go-leads-form - Other Dev
996c35a69 - (12 hours ago) Merge pull request #1234 from Project/feature/some-feature - Other Dev
~/Sites/project (master) ->

So how did we end up here?

Firstly, let’s check out a project I can use without worrying about sharing things I shouldn’t, I will use the rails codebase for this. In order to have a nice diff, I looked up a PR by David Heinemeier Hansson (DHH), that is about building “a web interface for developing Rails itself. … like with the mailer previews, and now in Rails 6, with the Action Mailbox inbound emails processing.”. Sounds cool, whatever that means, most importantly for us though, it has a few commits that aren’t in the main branch yet. So let us pretend the `conductor

I started out by looking up how to show the commits that are in one branch and not in the other:

~/Projects/rails (conductor6) -> git log main..conductor6
commit 398053bbd7d481ff16cb3fbeeb5fe039fc302fae (HEAD -> conductor6, origin/conductor6)
Merge: c319e5da8d 93dbbe3a81
Author: David Heinemeier Hansson <[email protected]>
Date:   Wed Mar 27 16:49:23 2019 -0700

    Merge branch 'master' into conductor6

commit c319e5da8d3b647f400f4e004b8d7875e9e0768f
Merge: 00297b1891 a04a757e5d
Author: Prathamesh Sonpatki <[email protected]>
Date:   Tue Mar 12 17:40:35 2019 +0530

    Merge branch 'master' into conductor6

commit 00297b18913990af8473d11a3416394bc7e902d8
Merge: 08cfdd94f1 b4bb0c3f5c
Author: Kasper Timm Hansen <[email protected]>
Date:   Tue Mar 12 10:30:01 2019 +0100

    Merge pull request #35580 from prathamesh-sonpatki/conductor6-fixes

    Fix failing tests
#..... etc

Ok, that looks good and it seems to be correct.

ABuisman.com

Building a Vector Database in Ruby Using Hash and PStore

First, why would we need a vector database?

Why build one in Ruby?

Generating embeddings

Building the database

Ruby’s Vector-type

Update VectorDb to use Vector-types.

PStore

Want to learn more?

Language matters: Building blocks

Super fast downloading of big files

Zero downtime credential updates on Heroku.

Cron monitoring with Blazer

Logging endpoint

Set up blazer checks

Running cron

Conclusion

Quick page benchmarks

What are we testing?

Note 1: Be sure to have a lot of data!

Note 2: This was a back-end test

Conclusion

Simple query optimisation

But wait, there is more!

Fixing Ruby openssl errors

We have to could go deeper

Conclusion

Crazyton - Better than module_function

Why?

Insert include Crazyton

Heroku differ: see which commits would be deployed

Insert `include Crazyton`