iridakos - opensource articles

stup - Daily notes in the terminal

Mon, 20 Apr 2020 12:00:00 +0300

Over the past few years I have been participating in Stand-up meetings and it took me some time to find a convenient and effective way for keeping notes about what I was doing every day.

I needed to be able to:

keep notes on the issues I worked on, the meetings I participated in and stuff that blocked my work
easily access these notes based on their date
have an overview of what I did for example the last week
all of the above from inside a terminal without
- having to manually structure and update a single document
- having to manually create a document for each day

stup

I created stup (the name derives from stand-up), an opensource CLI tool that deals with these issues and allows to easily save, access and organize my notes.

The source code of the project is available on GitHub here.

Usage examples

Find below some examples showing the most important usage of stup.

Adding notes

# Adding a note to the default category at current date
stup add -n "Worked on issue #ABC123"

# Adding a note to the default category setting the current date explicitly
stup add today -n "Worked on issue #ABC123"

# Adding a note to the meetings category
stup add today -c "meetings" -n "2 hours with @phoebe for the project kick off"

# Adding a note to the blocking category for April 10th, 2020
stup add @ 2020-04-10 -c "blocking" -n "connectivity issues"

Showing notes

# Showing yesterday's notes
$ stup

# Showing yesterday's notes explicitly setting the date
$ stup yesterday

# Showing today's notes
$ stup today

# Showing notes on a specific date
$ stup show @ 2020-04-18

# Showing notes on a specific date for the meetings category
$ stup show @ 2020-04-18 -c "meetings"

Retrieving all notes for a period of time

# List current week's notes
stup log week

# List previous week's notes
stup log previous-week

# List notes between January 20th, 2020 and March 2nd, 2020
stup log --from 2020-01-20 --to 2020-03-02

# List meeting notes between January 20th, 2020 and March 2nd, 2020
stup log --from 2020-01-20 --to 2020-03-02 -c "meetings"

Behind the scenes

Notes are organized in categories.

When a new note is added, stup creates a markdown file and places it under the category’s directory in a sub-directory with a path based on the date.

CATEGORIES_ROOT_DIRECTORY/CATEGORY/YEAR/MONTH/YYYY-MM-DD.md

# For example, the notes of a category named "programming" April 18th, 2020 are saved under
CATEGORIES_ROOT_DIRECTORY/programming/2020/04/2020-04-18.md

This allows users to retrieve any notes added on a specific date or in a specific period for a specific or for all categories.

# Fetch notes for all categories
$ stup show @ 2020-04-18

# Fetch notes for a specific category for previous week
$ stup log previous-week -c programming

# Fetch notes for all categories for a specific period
$ stup log --from 2020-04-01 --to 2020-04-15

More information

You can:

visit the project repository on Github here.
view the full documentation of stup here.
browse the project’s issues here.

That’s all, thanks for reading! Cat photo.

Find palindrome dates in the Linux command line - explained

Thu, 06 Feb 2020 12:00:00 +0200

Yesterday I saw a great command that finds the palindrome dates between now and x days ago.

The command was posted on Twitter by @climagic (a great account to follow if you want to find awesome command line stuff).

In this post we are going to break it down and explain how it works.

What is a palindrome

Palindrome is a word/number/phrase or other sequence of characters that reads the same backward as forward. – Palindrome - Wikipedia

Few days ago, the date was 02/02/2020 which was a palindrome:

The command

printf "now - %d days\n" {1..332044} |date -f- +%Y%_m%_d$'\n'%Y%m%d |tr -d ' ' > alldates.txt; rev alldates.txt >revdates.txt; paste alldates.txt revdates.txt |awk '$1==$2{print $1}'

And here’s a more readable version using new lines after each pipe.

printf "now - %d days\n" {1..332044} |
date -f- +%Y%_m%_d$'\n'%Y%m%d |
tr -d ' ' > alldates.txt; rev alldates.txt >revdates.txt; paste alldates.txt revdates.txt |
awk '$1==$2{print $1}'

I am going to explain each part of the command separately.

Note that the output of each part is the input of the next part (following the pipe |).

printf "now - %d days\n" {1..332044}

The first part seems pretty straight forward but it does have an interesting part, the brace expansion mechanism, which I didn’t know of.

Open a terminal and type the following commands:

# From 1 to 10
$ echo {1..10}
1 2 3 4 5 6 7 8 9 10

# From a to z
$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z

# From 10 to 20
$ echo Before-{10..20}-After
Before-10-After Before-11-After Before-12-After Before-13-After Before-14-After Before-15-After Before-16-After Before-17-After Before-18-After Before-19-After Before-20-After

As you can see, the brace expansion allows the generation of values based on a “sequence” we set.

Learn more about the Brace expansion mechanism here.

Part explained

we print the phrase “now - x days” so many times as the number of values expanded from the sequence
{1..332044} will generate 332044 values starting from 1 and increasing by 1
we add a new line at the end of each produced string

Output

now - 1 days
now - 2 days
now - 3 days
now - 4 days
now - 5 days
...
now - 332038 days
now - 332039 days
now - 332040 days
now - 332041 days
now - 332042 days
now - 332043 days
now - 332044 days

date -f- `+%Y%\_m%\_d$'\n'%Y%m%d`

In this part we use the date command to parse the previously piped output and print two lines for each date, one with padding zeros and one without zeros for single digit days and months.

Familiarize yourself with the command in your terminal:

# Show current date = now
$ date
Thu Feb  6 07:30:10 EET 2020

# Show yesterday
$ date --date="now - 1 days"
Wed Feb  5 07:31:22 EET 2020

# Show yesterday as dd/mm/yyyy (zero padding)
$ date --date="now - 1 days" +%d/%m/%Y
05/02/2020

# Show yesterday as dd/mm/yyyy (space padded)
$ date --date="now - 1 days" +%_d/%_m/%Y
 5/ 2/2020

# Show yesterday as dd/mm/yyyy (without padding)
$ date --date="now - 1 days" +%-d/%-m/%Y
5/2/2020

Part explained

we use the -f option to define the input of the date to be the lines of a file, followed by a hyphen - to set this file to be the stdin
we set the output format (what follows the plus + symbol) to be the numeric representation of the year (%Y), the month (padded with spaces: %_m, padded with zeros: %m) and the day of the month (padded with spaces: %_d, padded with zeros: %d)
the space padded format is %Y%\_m%\_d and the zero padded format is %Y%m%d
we combine these two formats for each date and we separate them with a new line $'\n' thus, for each date we parse, we produce two lines, one padded with zeros and one padded with spaces

Note: instead of padding with spaces (using the underscore _ in the format), we could use the hyphen - instead which would skip the padding of the field. This way wouldn’t have to remove the spaces as described in the next part.

Output

tr -d ' ' > alldates.txt; rev alldates.txt >revdates.txt; paste alldates.txt revdates.txt

In this part we remove the spaces produced in the previous step and we create two files, one with all the dates and one with all the dates reversed.

Then we combine these two files by merging each line of the first file with the corresponding line of the second file separated with a tab.

Familiarize yourself with the commands of this part in your terminal:

# Remove empty spaces
$ echo "Good night " | tr -d " "
Goodnight

# Remove hyphens
$ echo "Good-night" | tr -d "-"
Goodnight

# Print a-x-z where x is a number between 1 and 9
$ printf "a-%d-z\n" {1..9}
a-1-z
a-2-z
a-3-z
a-4-z
a-5-z
a-6-z
a-7-z
a-8-z
a-9-z

# and then reverse the order of the characters of each line
$ printf "a-%d-z\n" {1..9} | rev
z-1-a
z-2-a
z-3-a
z-4-a
z-5-a
z-6-a
z-7-a
z-8-a
z-9-a

Part explained

tr -d '': we remove the spaces of the input (which is the output of the previous part) and we save the output in a file named alldates.txt
rev: we reverse the order of the characters of each line of the alldates.txt file and we save the output in a new file named revdates.txt
paste: we merge the two files; each line of the alldates.txt file is merged with the corresponding line of the file revdates.txt separated by a tab (the two files have the exact number of lines).

Output

202025	520202
20200205	50200202
202024	420202
20200204	40200202
202023	320202
20200203	30200202
202022	220202
20200202	20200202
202021	120202
20200201	10200202

awk '$1==$2{print $1}'

In this final part, we use awk to print the palindromes.

Familiarize yourself with the usage of awk related to this post, in your terminal:

# Print the whole input (awk's default behavior) if the fields are equal
$ echo "a a" | awk '$1==$2'
a a

# Print the whole input (awk's default behavior) if the fields are not equal
$ echo "a a" | awk '$1!=$2'
# nothing here

# Print the whole input (awk's default behavior) if the fields are not equal
$ echo "a b" | awk '$1!=$2'
a b

# Print the first field if the fields are not equal
$ echo "a b" | awk '$1!=$2 {print $1}'
a

# Print the second field if the fields are not equal
$ echo "a b" | awk '$1!=$2 {print $2}'
b

Part explained

the previous output that is now the input of this part consists of lines and each line has two values separated by a TAB. With this kind of input, awk resolves two fields in each line. We can access these two fields using the $1 and $2 variables.
as described in a previous post, awk statements consist of a pattern-expression and an associated action.
```
<pattern/expression> { <action> }
```
If the pattern succeeds then the associated action is being executed. In this part the expression is translated as “if the two fields are equal” ($1==$2) and the action as “print the first field” (printf $1).
if the fields are equal we do have a palindrome since each pair of fields is actually a date and its reversed value.

Output

Cat photo

That’s all, thanks for reading.

References

Mongoid - Inheritance: change embedded document's type via nested attributes

Fri, 22 Nov 2019 09:00:00 +0200

I recently faced some issues when trying to update a nested document’s type via nested attributes using the mongoid gem. Bare with me if this sentence doesn’t make any sense yet.

Use case

Suppose we have a document named Person that embeds a document that can be any of the types Book or BoardGame which inherit from the class Item.

Item

class Item
  include Mongoid::Document

  embedded_in :person, inverse_of: :item

  field :name
end

Book

class Book < Item
  field :authors, type: Array
end

BoardGame

class BoardGame < Item
  field :players, type: Integer
end

Person

class Person
  include Mongoid::Document

  embeds_one :item, inverse_of: :person
  accepts_nested_attributes_for :item, allow_destroy: true
end

The problems

To simplify the post I won’t go with the scenario of building the forms and the controller actions to demonstrate the problems but instead I will simulate the behavior in the rails console.

Let’s create a Person with a Book as an Item.

>> person = Person.create!({
  item_attributes: {
    _type: "Book",
    name: "Jitterbug Perfume",
    authors: ["Tom Robbins"]
  }
})
=> #<Person _id: 5dd7b01ec2889856ab8a619f, >
>> person.item
=> #<Book _id: 5dd7ae76c2889856ab8a619e, name: "Jitterbug Perfume", _type: "Book", authors: ["Tom Robbins"]>

Failing attribute assignments

Let’s try to change the person’s item from Book to BoardGame with 5 players.

>> person.update_attributes({
  item_attributes: {
    _type: "BoardGame",
    name: "Ticket to Ride",
    players: 5
  }
})
Traceback (most recent call last):
        1: from (irb):9
Mongoid::Errors::UnknownAttribute ()
message:
  Attempted to set a value for 'players' which is not allowed on the model Book.
summary:
  Without including Mongoid::Attributes::Dynamic in your model and the attribute does not already exist in the attributes hash, attempting to call Book#players= for it is not allowed. This is also triggered by passing the attribute to any method that accepts an attributes hash, and is raised instead of getting a NoMethodError.
resolution:
  You can include Mongoid::Attributes::Dynamic if you expect to be writing values for undefined fields often.

This didn’t work, the assignment of the nested attributes item_attributes as you can see from the error message were applied in the model Book.

The fact that we changed the _type attribute of the document doesn’t mean that the object in memory also was magically transformed from a Book to a BoardGame.

Document attribute leftovers

Let’s try to change the person’s item from Book to BoardGame but this time without the players to see if this fixes the issue.

>> person.update_attributes({
  item_attributes: {
    _type: "BoardGame",
    name: "Ticket to Ride"
  }
})
=> true
>> person.item
=> #<Book _id: 5dd7b270c2889858a08a619e, name: "Ticket to Ride", _type: "BoardGame", authors: ["Tom Robbins"]>

This seems to have worked even though the result of the person.item accessor gave a Book document with _type of a BoardGame and with an attribute authors present.

Let’s reload to see if the document is going to be properly instantiated.

>> person.reload.item
=> #<BoardGame _id: 5dd7b270c2889858a08a619e, name: "Ticket to Ride", _type: "BoardGame", players: nil>

Seems to be fine but what is going on with the actual document attributes?

>> person.item.attributes
=> {"_id"=>BSON::ObjectId('5dd7b270c2889858a08a619e'), "_type"=>"BoardGame", "name"=>"Ticket to Ride", "authors"=>["Tom Robbins"]}

As you can see, the authors attribute (that we had initially set) hasn’t been removed when we changed the type of the Item from Book to BoardGame.

Wrong validators

Let’s change the BoardGame document so that it validates the presence of players

class BoardGame < Item
  field :players, type: Integer

  validates :players, presence: true
end

Our person now has an item of type BoardGame. We will now try to change it to a Book

>> person.attributes = {
  item_attributes: {
    _type: "Book",
    name: "Jitterbug Perfume"
  }  
}
=> {:item_attributes=>{:_type=>"Book", :name=>"Jitterbug Perfume"}}
>> person.valid?
=> false
>> person.errors.full_messages
=> ["Item is invalid"]
>> person.item.errors.full_messages
=> ["Players can't be blank"]

As described in the previous section, the fact that the _type attribute changed doesn’t mean that the actual object in memory changed accordingly. Upon save of the parent document, the item is still an instance of the previous _type which was BoadGame thus the validations that take place are not the ones defined in the new Book class but those defined in the BoardGame.

This also means that if we had validations in the Book class too, they wouldn’t execute in this scenario as well.

Investigation

I tried to google the issue (I couldn’t even construct a proper query) to see how others dealt with this but I didn’t find anything helpful and I decided to dive in the source of mongoid in order to locate the code that deals with the nested assignments and possibly create a monkey patch.

First, I searched for the file that defines the accepts_nested_attributes_for method that we used in the Person class.

Found in mongoid/attributes/nested.rb

def accepts_nested_attributes_for(*args)
  options = args.extract_options!.dup
  options[:autosave] = true if options[:autosave].nil?

  options[:reject_if] = REJECT_ALL_BLANK_PROC if options[:reject_if] == :all_blank
  args.each do |name|
    meth = "#{name}_attributes="
    self.nested_attributes["#{name}_attributes"] = meth
    association = relations[name.to_s]
    raise Errors::NestedAttributesMetadataNotFound.new(self, name) unless association
    autosave_nested_attributes(association) if options[:autosave]

    re_define_method(meth) do |attrs|
      _assigning do
        if association.polymorphic? and association.inverse_type
          options = options.merge!(:class_name => self.send(association.inverse_type))
        end
        association.nested_builder(attrs, options).build(self)
      end
    end
  end
end

The line that is mostly related to our issue from the above snippet is this:

association.nested_builder(attrs, options).build(self)

which is executed upon attribute assignment.

Since the item association defined in the Person document is of type embeds_one, I located its code to find the nested_builder method.

Found it at mongoid/association/embedded/embeds_one.rb

def nested_builder(attributes, options)
  Nested::One.new(self, attributes, options)
end

Ok, now we have to see what is going on inside the #build method of the mongoid/association/nested/one.rb file.

def build(parent)
  return if reject?(parent, attributes)
  @existing = parent.send(association.name)
  if update?
    attributes.delete_id
    existing.assign_attributes(attributes)
  elsif replace?
    parent.send(association.setter, Factory.build(@class_name, attributes))
  elsif delete?
    parent.send(association.setter, nil)
  end
end

def replace?
  !existing && !destroyable? && !attributes.blank?
end

def update?
  existing && !destroyable? && acceptable_id?
end

The part that is related to the behavior we want to alter is the one inside the if update? .. elsif replace? block:

...
if update?
  attributes.delete_id
  existing.assign_attributes(attributes)
elsif replace?
  parent.send(association.setter, Factory.build(@class_name, attributes))
...  

This is where the attribute assignment in the existing embedded document takes place.

Desired behavior

Since we are talking about embedded documents, when changing the type instead of updating the existing document’s attributes we can easily replace it at once with a new instance of the proper class.

From that point on, we won’t have to deal with

the invalid attribute assignments
removing the old attributes that are not present in the new document type
fixing the usage of wrong validators.

The replaced object will be properly instantiated and assigned to the parent.

Solution

To change to the desired aforementioned behavior, I patched the Mongoid::Association::Nested::One class.

I created a module (I suggest you read this great article about monkey patching without making a mess) under lib/core_extensions/mongoid/association/nested/one/embedded_one_change_type.rb with the following contents:

module CoreExtensions
  module Mongoid
    module Association
      module Nested
        module One
          module EmbeddedOneChangeType
            attr_accessor :changing_type

            def build(parent)
              return if reject?(parent, attributes)

              @existing = parent.send(association.name)
              # Retrieve the new class
              new_class = attributes.dig(:_type)&.constantize
              # Resolve whether we should be changing type or not
              @changing_type = new_class && @existing.class != new_class && association.embedded?

              if update?
                attributes.delete_id
                existing.assign_attributes(attributes)
              elsif replace?
                parent.send(association.setter, ::Mongoid::Factory.build(@class_name, attributes))
              elsif delete?
                parent.send(association.setter, nil)
              end
            end

            def replace?
              # Replace if the association hasn't been set or if we are changing type.
              (!existing || changing_type) && !destroyable? && !attributes.blank?
            end

            def update?
              # Don't update the document if we are changing type
              existing && !changing_type && !destroyable? && acceptable_id?
            end
          end
        end
      end
    end
  end
end

and inside an initializer I patched the proper mongoid class with:

Mongoid::Association::Nested::One.prepend CoreExtensions::Mongoid::Association::Nested::One::EmbeddedOneChangeType

Explanation

We override the build method in order to additionally resolve if we are changing type in an embedded association or not:

@changing_type = new_class && @existing.class != new_class && association.embedded?

We are changing type if:

there’s a _type attribute present and
the _type’s value is different than the existing object’s one and
the association is embedded

Then we alter the methods that decide whether a document should be updated or replaced, so that they take in consideration if we are @changing_type (as calculated above) or not:

def update?
  # Don't update the document if we are changing type
  existing && !changing_type && !destroyable? && acceptable_id?
end

and

def replace?
  # Replace if the association hasn't been set or if we are changing type.
  (!existing || changing_type) && !destroyable? && !attributes.blank?
end

Feedback and comments

I am not very fond of monkey patching but this was my quick workaround. If you believe that the desired behavior as described here (for the `embedded_one` associations) is invalid or if you deal with this problem in any other way, I'd be glad to here about it.

For suggestions, feedback, comments, typos etc. please use this issue on GitHub.

Thanks for visiting!

Almost forgot, cat photo.

How to manage nested objects in Elasticsearch documents

Thu, 02 May 2019 15:30:00 +0300

In this post we are going to manage nested objects of a document indexed with Elasticsearch.

The nested type is a specialised version of the object datatype that allows arrays of objects to be indexed in a way that they can be queried independently of each other. – Nested datatype - Official Elasticsearch reference

Prerequisites

To follow this post you need:

an up and running Elasticsearch instance
- I use 6.7 here
an up and running Kibana instance to interact with Elasticsearch

Preparation

The document of our index will represent a human and its nested objects will be cats (no surprises).

Create the index

Open your Kibana dev console and type the following to create the index.

# Create the index
PUT iridakos_nested_objects
{
  "mappings": {
    "human": {
      "properties": {
        "name": {
          "type": "text"
        },
        "cats": {
          "type": "nested",
          "properties": {
            "colors": {
              "type": "integer"
            },
            "name": {
              "type": "text"
            },
            "breed": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

Human has:

a name property of type text
a cats property of type nested

Each cat has:

a colors property of type integer
a name property of type text
a breed property of type text

Add a human

In the Kibana console, execute the following to add a human with three cats.

# Index a human
POST iridakos_nested_objects/human/1
{
  "name": "iridakos",
  "cats": [
    {
      "colors": 1,
      "name": "Irida",
      "breed": "European Shorthair"
    },
    {
      "colors": 2,
      "name": "Phoebe",
      "breed": "European"
    },
    {
      "colors": 3,
      "name": "Nino",
      "breed": "Aegean"
    }
  ]
}

Confirm the insertion with:

GET iridakos_nested_objects/human/1

You should see something like this:

{
  "_index": "iridakos_nested_objects",
  "_type": "human",
  "_id": "1",
  "_version": 1,
  "found": true,
  "_source": {
    "name": "iridakos",
    "cats": [
      {
        "colors": 1,
        "name": "Irida",
        "breed": "European Shorthair"
      },
      {
        "colors": 2,
        "name": "Phoebe",
        "breed": "European"
      },
      {
        "colors": 3,
        "name": "Nino",
        "breed": "Aegean"
      }
    ]
  }
}

Done, moving on.

Managing nested objects

Add a new nested object

Suppose that iridakos got a new Persian cat named Leon. To add it in iridakos’ collection of cats we will use the Update API.

In Kibana:

# Add a new cat
POST iridakos_nested_objects/human/1/_update
{
  "script": {
    "source": "ctx._source.cats.add(params.cat)",
    "params": {
      "cat": {
        "colors": 4,
        "name": "Leon",
        "breed": "Persian"
      }
    }
  }
}

Notes:

We accessed the nested cat objects of our human with ctx._source.cats. This gave us a collection
We executed the add method on the collection to add a new cat
The properties of the new cat (params.cat) were passed as parameters in the params attribute of the request under the attribute cat.

Confirm the addition with:

GET iridakos_nested_objects/human/1

Cat added:

{
  "_index": "iridakos_nested_objects",
  "_type": "human",
  "_id": "1",
  "_version": 2,
  "found": true,
  "_source": {
    "name": "iridakos",
    "cats": [
      {
        "colors": 1,
        "name": "Irida",
        "breed": "European Shorthair"
      },
      {
        "colors": 2,
        "name": "Phoebe",
        "breed": "European"
      },
      {
        "colors": 3,
        "name": "Nino",
        "breed": "Aegean"
      },
      {
        "colors": 4,
        "name": "Leon",
        "breed": "Persian"
      }
    ]
  }
}

Remove a nested object

Suppose we want to remove Nino from the human’s cat collection.

In Kibana:

# Remove Nino
POST iridakos_nested_objects/human/1/_update
{
  "script": {
    "source": "ctx._source.cats.removeIf(cat -> cat.name == params.cat_name)",
    "params": {
      "cat_name": "Nino"
    }
  }
}

Notes:

We accessed the nested cat objects of our human with ctx._source.cats. This gave us a collection
We executed the removeIf method on the collection to conditionally remove an item
We provided a Predicate to the removeIf method in which we specify which items we want to remove. This predicate will be executed on each item of the collection and resolves to a Boolean value. If the resolution is true then the item will be removed. In our case, the condition is a simple equality check on the cat’s name attribute.
The cat_name was passed as a parameter (params.cat_name) instead of fixing it to the script source.

Confirm the addition with:

GET iridakos_nested_objects/human/1

Cat removed:

{
  "_index": "iridakos_nested_objects",
  "_type": "human",
  "_id": "1",
  "_version": 3,
  "found": true,
  "_source": {
    "name": "iridakos",
    "cats": [
      {
        "colors": 1,
        "name": "Irida",
        "breed": "European Shorthair"
      },
      {
        "colors": 2,
        "name": "Phoebe",
        "breed": "European"
      },
      {
        "colors": 4,
        "name": "Leon",
        "breed": "Persian"
      }
    ]
  }
}

Update a nested object

Suppose we want to change all cat breeds from European to European Shorthair (Phoebe is the only one in our case).

# Update breed
POST iridakos_nested_objects/human/1/_update
{
  "script": {
    "source": "def targets = ctx._source.cats.findAll(cat -> cat.breed == params.current_breed); for(cat in targets) { cat.breed = params.breed }",
    "params": {
      "current_breed": "European",
      "breed": "European Shorthair"
    }
  }
}

Notes:

We accessed the nested cat objects of our human with ctx._source.cats. This gave us a collection
We executed the findAll method on the collection to select specific items
We provided a Predicate to the findAll method in which we specify which items we want to select. This predicate will be executed on each item of the collection and resolves to a Boolean value. If the resolution is true then the item will be selected. In our case, the condition is a simple equality check on the cat’s breed attribute.
The current_breed was passed as a parameter (params.current_breed) instead of fixing it to the script source.
We then loop on the selected cats (whose breed attribute has value European) and change their breed to the new value which we passed by another parameter breed.

Confirm the change:

GET iridakos_nested_objects/human/1

Cat updated:

{
  "_index": "iridakos_nested_objects",
  "_type": "human",
  "_id": "1",
  "_version": 5,
  "found": true,
  "_source": {
    "name": "iridakos",
    "cats": [
      {
        "colors": 1,
        "name": "Irida",
        "breed": "European Shorthair"
      },
      {
        "colors": 2,
        "name": "Phoebe",
        "breed": "European Shorthair"
      },
      {
        "colors": 4,
        "name": "Leon",
        "breed": "Persian"
      }
    ]
  }
}

Update multiple attributes of nested objects fulfilling multiple conditions

Now, in a more advanced example, we are going to use a more flexible script to:

target objects based on multiple conditions (here colors and breed)
update more than one attributes (here colors and breed)

Suppose we want to change the breed of cats who have 4 colors and their breed is Persian to Aegean and their colors to 3.

The script we will use is the following:

# Update multiple attributes with multiple conditions
POST iridakos_nested_objects/human/1/_update
{
  "script": {
    "source": "def targets = ctx._source.cats.findAll(cat -> { for (condition in params.conditions.entrySet()) { if (cat[condition.getKey()] != condition.getValue()) { return false; } } return true; }); for (cat in targets) { for (change in params.changes.entrySet()) { cat[change.getKey()] = change.getValue() } }",
    "params": {
      "conditions": {
        "breed": "Persian",
        "colors": 4
      },
      "changes": {
        "breed": "Aegean",
        "colors": 3
      }
    }
  }
}

For convenience, here’s the script source with proper indentation.

def targets = ctx._source.cats.findAll(cat -> {
                                         for (condition in params.conditions.entrySet()) {
                                           if (cat[condition.getKey()] != condition.getValue()) {
                                             return false;
                                           }
                                         }
                                         return true; });
for (cat in targets) {
 for (change in params.changes.entrySet()) {
   cat[change.getKey()] = change.getValue()
 }
}

Notes:

We select which cats we want to update by checking that their properties have the value specified in the params.conditions parameter.
For each selected cat, we change its attributes’ values as specified in the params.changes parameter.

Confirm:

GET iridakos_nested_objects/human/1

Cat updated.

{
  "_index": "iridakos_nested_objects",
  "_type": "human",
  "_id": "1",
  "_version": 5,
  "found": true,
  "_source": {
    "name": "iridakos",
    "cats": [
      {
        "colors": 1,
        "name": "Irida",
        "breed": "European Shorthair"
      },
      {
        "colors": 2,
        "name": "Phoebe",
        "breed": "European Shorthair"
      },
      {
        "name": "Leon",
        "colors": 3,
        "breed": "Aegean"
      }
    ]
  }
}

Useful links

That’s all! Cat photo.

Comments and feedback

For feedback, comments, typos etc. please use this issue.

Thanks for visiting!

How to alias and navigate to directories with autocomplete in Linux

Wed, 10 Apr 2019 16:00:00 +0300

I use the terminal a lot and in my day to day work I tend to navigate to the same bunch of directories.

There are some awesome tools out there (like autojump or z) but sometimes, especially when the directories are similarly named, there is a need to be explicit to navigate to the proper one.

I decided to write a script to overcome this issue and to avoid having to edit my .bash* files to manage aliases each time I wanted to add or remove a directory.

goto

goto is a shell utility to quickly navigate to aliased directories with autocomplete (tab completion).

goto repo at GitHub

User registers directory aliases, for example:

goto --register dev /home/iridakos/development

and then cds to that directory with:

goto dev

Find below documentation on the script but make sure you check the script’s documentation page for updates.

Sorry for the name, I know it brings back memories but it’s not what it seems :)

Installation

To install goto all you have to do is clone the repository locally:

git clone https://github.com/iridakos/goto.git

navigate to it

cd goto

and execute:

sudo ./install

You need to restart your shell after installation.

Usage

Change to an aliased directory
Register an alias
Unregister an alias
List aliases
Expand an alias
Cleanup
Help
Version
Extras
- Push before changing directories
- Revert to a pushed directory

Change to an aliased directory

To change to an aliased directory, type:

goto <alias>

Example

goto dev

Register an alias

To register a directory alias, type:

goto -r <alias> <directory>

goto --register <alias> <directory>

Example

goto -r blog /mnt/external/projects/html/blog

goto --register blog /mnt/external/projects/html/blog

Notes

goto expands the directories hence you can easily alias your current directory with:
```
goto -r last_release .
```
and it will automatically be aliased to the whole path.
Pressing the tab key after the alias name, you have the default directory suggestions by the shell.

Unregister an alias

To unregister an alias, use:

goto -u <alias>

goto --unregister <alias>

Example

goto -u last_release

goto --unregister last_release

Notes

Pressing the tab key after the command (-u or --unregister), the completion script will prompt you with the list of registered aliases for your convenience.

List aliases

To get the list of your currently registered aliases, use:

goto -l

goto --list

Expand an alias

To expand an alias to its value, use:

goto -x <alias>

goto --expand <alias>

Example

goto -x last_release

goto --expand last_release

Cleanup

To cleanup the aliases from directories that are no longer accessible in your filesystem, use:

goto -c

goto --cleanup

Help

To view the tool’s help information, use:

goto -h

goto --help

Version

To view the tool’s version, use:

goto -v

goto --version

Extras

Push before changing directories

To first push the current directory onto the directory stack before changing directories, type:

goto -p <alias>

goto --push <alias>

Revert to a pushed directory

To return to a pushed directory, type:

goto -o

goto --pop

Notes

This command is equivalent to popd, but within the goto command.

Behind the scenes

Upon installation, a line is appended to your .bashrc file which sources the script, registering a goto function responsible for changing the directory based on the defined aliases of the database.

The script creates a directory alias database file under ~/.goto. Every register or unregister action modifies this file.

The script also uses the Bash programmable completion feature to define tab completions resolved via extraction of the aliases defined in the goto database file.

That’s all! Cat photo.

Code and comments

You can find the code of goto at https://github.com/iridakos/goto.

For feedback, comments, typos etc. please open an issue in the repository.

Thanks for visiting!

Elasticsearch bucket aggregations and faceted navigation - facets

Mon, 22 Oct 2018 20:00:00 +0300

Introduction

I decided to write a series of tutorials about Elasticsearch aggregations. In this first post of the series, we are going to deal with the bucket aggregations that allow us to implement faceted navigation.

What is Elasticsearch

Elasticsearch is an opensource JSON-based search engine that allows us to search indexed data quickly and with options that are not provided by classic data stores.

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. – official Elasticsearch product page

What is Kibana

Kibana is a tool mainly allowing visualization of Elasticsearch data. We will use Kibana because it also provides a very convenient way for writing and executing queries (with autocomplete).

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you’re getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers. – official Kibana product page

What are Elasticsearch bucket aggregations

Before familiarizing myself with the term aggregations in the Elasticsearch world, what I was actually trying to learn was how to implement the widely known feature of facets for my indexed data.

Chances are that you know about facets, you have seen it in many sites. They are usually placed as a sidebar in search results landing pages and they are rendered as links or checkboxes that act as filters to help you narrow the results based on their properties.

A section of Elasticsearch’s aggregations framework named bucket aggregations provides the functionality we need to implement a faceted navigation.

Bucket aggregations don’t calculate metrics over fields like the metrics aggregations do, but instead, they create buckets of documents. Each bucket is associated with a criterion (depending on the aggregation type) which determines whether or not a document in the current context “falls” into it. In other words, the buckets effectively define document sets. In addition to the buckets themselves, the bucket aggregations also compute and return the number of documents that “fell into” each bucket. – official elasticsearch bucket aggregations reference

Prerequisites

In order to be able to execute the tutorial’s commands and queries, you must first:

Install Elasticsearch (duh)

Providing instructions for installing Elasticsearch is out of scope. I recommend visiting the product’s related documenation site. After the installation, just make sure to start the service if it’s not started. You can check that with

curl http://localhost:9200

It should give something like:

{
  "name" : "blabla",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "blabla",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "04711c2",
    "build_date" : "2018-09-26T13:34:09.098244Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

As you can see, we will use Elasticsearch version 6.4.2 in this tutorial.

Install Kibana

I recommend following the official instructions for installing kibana at your machine’s OS. I followed this guide for installing the product via a repository in Ubuntu.

Start the kibana service and navigate to its home page at http://localhost:5601. We will heavily use the Dev Tools which is a powerful console talking to the Elasticsearch engine.

What is it going to be covered

We are going to deal with:

Terms aggregations
Sub-aggregations
Nested and reverse nested aggregations
Global aggregations

Don’t be afraid, everything will become very clear once we start playing with the data.

Hands on

Let the fun begin.

The model

Suppose that we want to implement a web application for city pet registrations. We will define the following entities:

City office: a city pet registration office
- city: the city in which the office is located
- office type: the type of the office (central, district)
Citizen: a citizen with registered pets
- occupation: the occupation of the citizen
- age: the age of the citizen
Pet: a citizen’s pet
- kind: the kind of the pet (cat, dog etc)
- name: the name of the pet
- age: the age of the pet

Our entities relations are:

One city office has many citizens
One citizen has many registered pets

Prepare the data

Before starting explaining the aggregations, we first have to create the index to store our data and then we have to feed this index with sample data.

Create the index

Navigate to Kibana’s Dev Tools page and execute the following request:

PUT city_offices
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "city": {
          "type": "keyword"
        },
        "office_type": {
          "type": "keyword"
        },
        "citizens": {
          "type": "nested",
          "properties": {
            "occupation": {
              "type": "keyword"
            },
            "age": {
              "type": "integer"
            },
            "pets": {
              "type": "nested",
              "properties": {
                "kind": {
                  "type": "keyword"
                  },
                "name": {
                  "type": "keyword"
                },
                "age": {
                  "type": "integer"
                }
              }
            }
          }
        }
      }
    }
  }
}

In the output section, you should see:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "city_offices"
}

We created an index with name city_offices with the aforementioned properties.

Important note: we chose to define the entity relations as nested objects. Why?

The nested type is a specialized version of the object datatype that allows arrays of objects to be indexed in a way that they can be queried independently of each other … Lucene has no concept of inner objects, so Elasticsearch flattens object hierarchies into a simple list of field names and values – official elasticsearch nested datatype reference

I will explain this with an example.

Suppose we had a city office with two citizens. One 35 year old Dentist and one 30 year old Developer. If we used the object datatype, Elasticsearch would merge all sub-properties of the entity relation resulting to something like this:

{
  "citizens": {
    "occupation": ["Dentist", "Developer"],
    "age": ["35", "30"]
  }
}

Thus, if we wanted to search the index for offices that have a Dentist citizen with age 30, this document would fulfill the criteria even though the Dentist is 35 years old.

Mapping the relation as nested overcomes this problem since

Internally, nested objects index each object in the array as a separate hidden document, meaning that each nested object can be queried independently of the others – official elasticsearch nested datatype reference

Feed sample data

I have created some sample data with random cities, occupations, pet names etc. You can find it at this tutorial’s GitHub repo.

Download the file sample-data.json and from within the download folder execute:

curl -s -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@sample-data.json"; echo

Note: The data were generated with this ruby script. You can alter it as you please and execute it to produce your desired sample data json file.

That’s it. Aggregations time.

Aggregation request format

All aggregation queries are embedded in search requests.

GET <index_name>/_search
{
  "query": { ... },
  "aggs": {
    "<aggregation name>": {
      "<aggregation type>": { <aggregation properties> }
    }
  }
}

where:

aggregation name: is the name we give to our aggregation. This is required in order to be able to parse the search response afterwards and locate the specific aggregation results (you will understand this better after we execute the first aggregation request).
aggregation type: is the type of the aggregation we want to execute. You can browse all available aggregations provided by Elasticsearch here.
aggregation properties: are the available properties specific to the aggregation type.

Terms aggregations

We will use the Terms aggregation in order to find out how many different values our documents have in a specific field.

Navigate to Kibana’s Dev Tools section and in the console (left section of the page) type the following:

GET city_offices/_search
{
  "aggs": {
    "cities": {
      "terms": { "field": "city" }
    }
  }
}

Following the syntax described in the Aggregation request format section above

we are searching in the city_offices index
we are requesting an aggregation of type terms
we specify that we are interested in the different values of the field city

Now execute the request by pressing the play link that you should be seeing given that the query you typed is focused.

Voila!

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 1,
    "hits": [
      {
        ...
      }
    ]
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 37,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8
        },
        {
          "key": "London",
          "doc_count": 8
        },
        {
          "key": "Oslo",
          "doc_count": 8
        },
        {
          "key": "Paris",
          "doc_count": 8
        },
        {
          "key": "San Francisco",
          "doc_count": 8
        },
        {
          "key": "Tokyo",
          "doc_count": 8
        },
        {
          "key": "Athens",
          "doc_count": 7
        },
        {
          "key": "Barcelona",
          "doc_count": 7
        },
        {
          "key": "Chicago",
          "doc_count": 7
        },
        {
          "key": "Madrid",
          "doc_count": 7
        }
      ]
    }
  }
}

Since we are executing a search request, the response contains the hits attribute with all the results matching our query (in this case all documents since we didn’t add a query). It also contains though another attribute named aggregations and that’s where our aggregation results fall under.

Inside the aggregations you can see our cities aggregation and that’s why it is required to name the aggregations you define.

Before explaining the weird sum_other_doc_count 37 value, let’s first examine the buckets.

There’s our different values for the city field of the city offices. “Human reading” the values, we have 8 offices in Amsterdam, London, Oslo, Paris, San Francisco and Tokyo, 7 offices in Athens, Barcelona, Chicago and Madrid.

The total of these offices is 76 but the search response says we have 113. Let’s subtract the bucket’s office total from the actual total offices: (113 - 76 = 37) == sum_other_doc_count. That’s right. Explanation: the aggregation we defined has another property named size which has a default value of 10. Since we didn’t define another value, Elasticsearch limited the buckets to the top 10 different city values based on their occurrences in the index. 37 are the documents whose cities are not listed in the buckets and NOT the number of the unlisted cities.

Let’s change the size to 50 (hoping that we don’t have offices in more than 50 cities).

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    }
  }
}

Hint: In addition to defining the aggregation’s size, I also set in the top level query the size to 0 so that I can browse the response faster since the hits attribute will be empty.

Execute again.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8
        },
        {
          "key": "London",
          "doc_count": 8
        },
        {
          "key": "Oslo",
          "doc_count": 8
        },
        {
          "key": "Paris",
          "doc_count": 8
        },
        {
          "key": "San Francisco",
          "doc_count": 8
        },
        {
          "key": "Tokyo",
          "doc_count": 8
        },
        {
          "key": "Athens",
          "doc_count": 7
        },
        {
          "key": "Barcelona",
          "doc_count": 7
        },
        {
          "key": "Chicago",
          "doc_count": 7
        },
        {
          "key": "Madrid",
          "doc_count": 7
        },
        {
          "key": "New York",
          "doc_count": 7
        },
        {
          "key": "Warsaw",
          "doc_count": 7
        },
        {
          "key": "Berlin",
          "doc_count": 6
        },
        {
          "key": "Budapest",
          "doc_count": 6
        },
        {
          "key": "Melbourne",
          "doc_count": 6
        },
        {
          "key": "Prague",
          "doc_count": 5
        }
      ]
    }
  }
}

Perfect. The previously missed cities are now fetched and the sum_other_doc_count’s 0 value confirms that we haven’t left any document outside the aggregation.

Let’s add another aggregation for the offices, this time for their type.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type"
      }
    }
  }
}

We defined another terms aggregation named office_types and kept the default bucket size since we know that we don’t have more than 10 distinct office types.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        ...
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "primary",
          "doc_count": 57
        },
        {
          "key": "secondary",
          "doc_count": 56
        }
      ]
    }
  }
}

So, we have 57 primary and 56 secondary offices and 0 offices left outside the aggregation’s buckets.

Important note

If we do define a search query along with the aggregations definition, then the aggregations will be applied to the subset of the documents that match the search query.

For example, if we want to see how many different office types are in Athens, we can specify the appropriate search query without altering the aggregations’ one.

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "Athens"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

Execute and…

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 7,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Athens",
          "doc_count": 7
        }
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "secondary",
          "doc_count": 5
        },
        {
          "key": "primary",
          "doc_count": 2
        }
      ]
    }
  }
}

We have 5 secondary and 2 primary offices in Athens.

What if we wanted to have this information for all cities without having to limit the search’s results? In other words, what if we wanted to present the facets like this:

Amsterdam (8)

Primary (4)

Secondary (4)

London (8)

Primary (6)

Secondary (2)

Athens (7)

Primary (2)

Secondary (5)

Read the next section.

Sub-bucket aggregations

The Terms aggregations (and other type of aggregations) allow the definition of sub-aggregations. The sub-aggregations are executed in the documents belonging to the bucket of the parent aggregation. It’s like asking: Give me the different cities. Ok, now give me for each one how many office types it has.

The syntax of sub-aggregations is pretty straight forward.

GET <index_name>/_search
{
  "query": { ... },
  "aggs": {
    "<aggregation name>": {
      "<aggregation type>": { <aggregation properties> },
      "aggs": {
        "<sub-aggregation name>": {
          "<sub-aggregation type>": { <sub-aggregation properties> }
        }
      }
    }
  }
}

Applying the above in our case, all we have to do is to define the search request as follows:

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type"
          }
        }
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

The response:

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        },
        {
          "key": "Oslo",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "Paris",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "San Francisco",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        ...
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "primary",
          "doc_count": 57
        },
        {
          "key": "secondary",
          "doc_count": 56
        }
      ]
    }
  }
}

If we wanted to retrieve even more sub-results for another property of the city offices (for example building type), we could add a sub-aggregation inside the already defined office_types sub-aggregation of the cities aggregation.

Awesome. Time to play with the pets.

Nested aggregations

As previously mentioned, we defined the city office’s relations as nested objects. In order to perform aggregations on these relations we have to follow another approach, the Nested aggregations.

A special single bucket aggregation that enables aggregating nested documents. – official Elasticsearch Nested Aggregation reference

Syntax:

GET <index_name>/_search
{

  "aggs" : {
    "<aggregation-name>" : {
      "nested" : {
        "path" : "<nested-object-path>"
      },
      "aggs" : {
        "<nested-aggregation-name>": {
          "<aggregation-type>" : { <aggregation-properties> }  
        }
      }
    }
  }
}

where nested-object-path is the navigation path of the desired object in the root document. For example, if we want aggregations for the citizens we will set this property to citizens. If we wanted aggregations for the pets, we will set this property to citizens.pets.

Let’s see how we can breakdown the number of citizens per occupation.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens"
      },
      "aggs": {
        "occupations": {
          "terms": {
            "field": "citizens.occupation",
            "size": 50
          }
        }
      }
    }
  }
}

The response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 3966,
      "occupations": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Hairdresser",
            "doc_count": 243
          },
          {
            "key": "Microbiologist",
            "doc_count": 241
          },
          {
            "key": "Farmer",
            "doc_count": 234
          },
          {
            "key": "Marketing Manager",
            "doc_count": 231
          },
          {
            "key": "Clinical Laboratory Technician",
            "doc_count": 230
          },
          {
            "key": "Librarian",
            "doc_count": 230
          },
          {
            "key": "Editor",
            "doc_count": 229
          },
          ...
        ]
      }
    }
  }
}

So, there are 3966 citizens that have registered their pets to our city offices of which 243 are Hairdressers, 241 are Microbiologists etc.

Important note

Mind the field definition of the nested Terms aggregation in the query. It has to be the full path of the nested object.

Ok good, but what if wanted to see in how many offices are these citizens registered. In other words, how many offices have Marketing Managers? How many offices have Librarians?

Reverse nested aggregations

We have to use another type of aggregation, named Reverse nested aggregation.

A special single bucket aggregation that enables aggregating on parent docs from nested documents. Effectively this aggregation can break out of the nested block structure and link to other nested structures or the root document, which allows nesting other aggregations that aren’t part of the nested object in a nested aggregation.

The reverse_nested aggregation must be defined inside a nested aggregation. –official Elasticsearch Reverse nested aggregation reference

Even though it might sound complicated, it’s not. And playing around with the data is the best way for understanding the provided functionality of any feature.

Syntax:

GET <index_name>/_search
{

  "aggs" : {
    "<aggregation-name>" : {
      "nested" : {
        "path" : "<nested-object-path>"
      },
      "aggs" : {
        "<nested-aggregation-name>": {
          "<aggregation-type>" : { <aggregation-properties> },
          "aggs": {
            "in_offices": {
              "reverse_nested": { <reverse-nested-options> }
            }
          }
        }
      }
    }
  }
}

As you can see, the reverse nested aggregation is always defined as a sub-aggregation inside a nested aggregation.

Time to see in how many offices each occupation is spread.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens"
      },
      "aggs": {
        "occupations": {
          "terms": {
            "field": "citizens.occupation",
            "size": 50
          },
          "aggs": {
            "in_offices": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }
}

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 3966,
      "occupations": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Hairdresser",
            "doc_count": 243,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Microbiologist",
            "doc_count": 241,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Farmer",
            "doc_count": 234,
            "in_offices": {
              "doc_count": 99
            }
          },
          {
            "key": "Marketing Manager",
            "doc_count": 231,
            "in_offices": {
              "doc_count": 91
            }
          },
          {
            "key": "Clinical Laboratory Technician",
            "doc_count": 230,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Librarian",
            "doc_count": 230,
            "in_offices": {
              "doc_count": 98
            }
          },
          ...
        ]
      }
    }
  }
}

Translating the response, we have 243 Hairdressers registered in 98 offices, 241 Microbiologists registered in 98 offices, 234 Farmers registered in 99 offices etc.

The reverse nested aggregation accepts a single option named path. This options defines how many steps backwards in the document hierarchy we want Elasticsearch to go to calculate the aggregations. In our case, since citizens is the immediate relation of the city office type we had to leave it undefined implying that we want the aggregations of the occupations to be calculated against the root object a.k.a. the offices. Confused? Don’t worry, it will become clear after playing around with the pets. Now.

How many pets per kind are registered per citizen?

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens.pets"
      },
      "aggs": {
        "kinds": {
          "terms": {
            "field": "citizens.pets.kind",
            "size": 10
          },
          "aggs": {
            "per_citizen": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }
}

Note that we didn’t define the path option and

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 11845,
      "kinds": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Dog",
            "doc_count": 2421,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Hamster",
            "doc_count": 2403,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Cat",
            "doc_count": 2380,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Bird",
            "doc_count": 2330,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Rabbit",
            "doc_count": 2311,
            "per_citizen": {
              "doc_count": 113
            }
          }
        ]
      }
    }
  }
}

So, there are 2421 registered dogs, 2403 registered hamsters, 2380 registered cats etc. The per_citizen bucket info though doesn’t seem right. Does the 113 number ring a bell? Exactly, that’s how many offices we have. Since we didn’t define the path in the reverse nested aggregation, Elasticsearch calculated the count of root documents (a.k.a. the offices) that have each pet kind registered. Let’s fix that.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens.pets"
      },
      "aggs": {
        "kinds": {
          "terms": {
            "field": "citizens.pets.kind",
            "size": 10
          },
          "aggs": {
            "per_citizen": {
              "reverse_nested": {
                "path": "citizens"
              }
            }
          }
        }
      }
    }
  }
}

There it is:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 11845,
      "kinds": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Dog",
            "doc_count": 2421,
            "per_citizen": {
              "doc_count": 1864
            }
          },
          {
            "key": "Hamster",
            "doc_count": 2403,
            "per_citizen": {
              "doc_count": 1852
            }
          },
          {
            "key": "Cat",
            "doc_count": 2380,
            "per_citizen": {
              "doc_count": 1823
            }
          },
          {
            "key": "Bird",
            "doc_count": 2330,
            "per_citizen": {
              "doc_count": 1803
            }
          },
          {
            "key": "Rabbit",
            "doc_count": 2311,
            "per_citizen": {
              "doc_count": 1800
            }
          }
        ]
      }
    }
  }
}

There are 2421 dogs registered by 1864 citizens, 2403 hamsters registered by 1852 citizens, 2380 cats registered by 1823 citizens etc.

Note: Each citizen has more than one pets which can be of different kind that’s why the per_citizen sum of doc_count is way bigger than our citizens.

Before moving on to the last type of aggregations, let’s execute a more advanced nested query answering to the questions:

For each city, how many kind of pets are registered per citizen occupation and in how many offices?

Breaking down the question, we have to think like this:

Since we want results per city, we are going to add a Terms aggregation on the field city
Since we want to have results per citizen occupation, we are going to add a Terms sub-aggregation on the field occupation.
- Since the citizen is a nested object, the aggregation of the previous item has to be a sub-aggregation of a Nested aggregation on the path citizen
Since we want to have results per pet kind, we are going to add a Terms sub-aggregation on the field kind.
- Since the pet is a nested object, the aggregation of the previous item has to be a sub-aggregation of a Nested aggregation on the path citizen.pets

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "citizens": {
          "nested": {
            "path": "citizens"
          },
          "aggs": {
            "occupations": {
              "terms": {
                "field": "citizens.occupation",
                "size": 50
              },
              "aggs": {
                "pets": {
                  "nested": {
                    "path": "citizens.pets"
                  },
                  "aggs": {
                    "kinds": {
                      "terms": {
                        "field": "citizens.pets.kind",
                        "size": 10
                      },
                      "aggs": {
                        "per_occupation": {
                          "reverse_nested": {
                            "path": "citizens"
                          }
                        },
                        "per_office": {
                          "reverse_nested": {}
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

{
  "took": 17,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8,
          "citizens": {
            "doc_count": 230,
            "occupations": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "Dancer",
                  "doc_count": 19,
                  "pets": {
                    "doc_count": 49,
                    "kinds": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": [
                        {
                          "key": "Cat",
                          "doc_count": 13,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 9
                          }
                        },
                        {
                          "key": "Rabbit",
                          "doc_count": 11,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 10
                          }
                        },
                        {
                          "key": "Bird",
                          "doc_count": 9,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        },
                        {
                          "key": "Dog",
                          "doc_count": 8,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        },
                        {
                          "key": "Hamster",
                          "doc_count": 8,
                          "per_office": {
                            "doc_count": 3
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        }
                      ]
                    }
                  }
                },
                ...
              ]
            }
          }
        },
        ...
        ]
      }
    }
  }
}

“Human” reading the response, in Amsterdam there are 8 offices with 230 citizens that have registered pets of which:

19 are Dancers that have registered a total of 49 pets of which
- 13 are cats registered by 9 dancers in 5 offices
- 11 are rabbits registered by 10 dancers in 5 offices
- …
- 8 are hamsters registered by 7 dancers in 3 offices
- I stop this now because I got dizzy

Global aggregations

I will end this tutorial with something easier. Global aggregations.

Defines a single bucket of all the documents within the search execution context. This context is defined by the indices and the document types you’re searching on, but is not influenced by the search query itself. – official Elasticsearch Global aggregation reference

To better explain this, let’s re-think of query:

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type"
          }
        }
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

which would help us build this form:

Amsterdam (8)

Primary (4)

Secondary (4)

London (8)

Primary (6)

Secondary (2)

Athens (7)

Primary (2)

Secondary (5)

When we render such a form, what is the expected behavior once he/she clicks on a checkbox, for example London? Well, we expect the aggregations to be re-applied to the search results that were narrowed by the clicked facet. So the request will become:

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "London"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type",
            "size": 10
          }
        }
      }
    }
  }
}

and the response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        }
      ]
    }
  }
}

If we rendered the form based on the response, it would only have the London entry:

London (8)

Primary (6)

Secondary (2)

It would be better if we could render the previous form with disabled checkboxes for the terms that are no longer valid.

Amsterdam

Primary

Secondary

London

Primary (6)

Secondary (2)

Athens

Primary

Secondary

We could accomplish that if we had the aggregation results of a search request without a query and compared them with the search request that was triggered after user narrows the results by clicking on a checkbox. To avoid executing an additional search request, we can use the Global aggregation.

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "London"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type",
            "size": 10
          }
        }
      }
    },
    "unfiltered": {
      "global": {},
      "aggs": {
        "cities": {
          "terms": {
            "field": "city",
            "size": 50
          },
          "aggs": {
            "office_types": {
              "terms": {
                "field": "office_type",
                "size": 10
              }
            }
          }
        }
      }
    }
  }
}

Now, in the response we have an unfiltered section which we can use to render the form as we please.

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        }
      ]
    },
    "unfiltered": {
      "doc_count": 113,
      "cities": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Amsterdam",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "London",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 6
                },
                {
                  "key": "secondary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Oslo",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Paris",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "San Francisco",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Tokyo",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Athens",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 5
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Barcelona",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Chicago",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 5
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Madrid",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 4
                },
                {
                  "key": "primary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "New York",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Warsaw",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Berlin",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 4
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Budapest",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 3
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Melbourne",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 5
                },
                {
                  "key": "secondary",
                  "doc_count": 1
                }
              ]
            }
          },
          {
            "key": "Prague",
            "doc_count": 5,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 3
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          }
        ]
      }
    }
  }
}

Code and comments

You may find the script generating random data and the already generated sample data here.

You can submit your feedback or any comments you may have here.

Thanks for visiting!

What’s next

My next post will be a tutorial for implementing what we covered in this post within a Rails application using the elasticsearch gem
Next, I might come back with a tutorial covering the metrics aggregations

That’s all! Bucket cat photo.

Change brightness with the function keys on iMac running Ubuntu or Debian

Sun, 24 Jun 2018 18:00:00 +0300

In this post I will describe the steps I had to follow to be able to change the brightness on my iMac using the related keyboard function keys.

Context

After installing Debian 9.4 (stretch) or Ubuntu 18.04 LTS on my iMac, I was no longer able to control the brightness from the keyboard (F1, F2 keys). When pressing these keys, although a notification was shown for increasing/decreasing it, the brightness was not changing.

Identifying the problem

In my case, I had two entries for the backlight class in my system.

$ ls /sys/class/backlight/
acpi_video0  radeon_bl0

The keyboard keys were affecting the brightness value stored in the file brightness located in the acpi_video0 directory.

I tried to change the brightness level manually in the brightness file located in the second directory, the one named radeon_bl0 and the iMac’s brightness did actually change.

echo 190 | sudo tee /sys/class/backlight/radeon_bl0/brightness

Solution

After searching around on how to fix this, I found an answer explaining the meaning of theacpi_backlight kernel parameter.

So, all I had to do was to set my grub configuration to pass the vendor value in the acpi_backlight kernel parameter upon boot so that the vendor specific driver (in my case Radeon) is preferred instead of the default ACPI video.ko.

1. Editing the grub configuration file

I edited the grub configuration file with vim:

sudo vim /etc/default/grub

2. Setting the kernel parameter in the appropriate section

In the line starting with GRUB_CMDLINE_LINUX_DEFAULT I appended the desired kernel parameter acpi_backlight=vendor:

GRUB_CMDLINE_LINUX_DEFAULT="quiet acpi_backlight=vendor"

3. Updating grub

Finally, I had to update grub to produce the actual configuration file being loaded at boot.

sudo update-grub

4. Reboot

After rebooting, I was able to manage the brightness with the brightness function keys. In addition, the /sys/class/backlight directory now lists only one directory, the vendor’s one radeon_bl0.

Caution

The steps described in this post, have been tested on my system with:

Product Name: iMac11,3
VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Broadway PRO [Mobility Radeon HD 5850]

and I can’t know if they are valid for other cases. Follow them at your own risk :)

To find your iMac model, use:

sudo dmidecode | grep -A 9 "System Information"

and your VGA compatible controller:

lspci | grep "VGA compatible controller"

That’s all, cat photo:

Indexing and full text searching Linux man pages

Thu, 12 Apr 2018 19:00:00 +0300

In this post we are going to play around with full text searching Linux man pages using Elasticsearch.

The idea

We will:

setup an Elasticsearch instance locally
create an index for the data
feed the index with the man pages of the OS
create a search method for full text searching
full text search the man pages

The code

I have gathered the code snippets you will find here in a script called elman and it is available on GitHub so after reading this post to get the idea, you can experiment by modifying it as you please.

Let’s start.

Prerequisites

Ruby

You should have ruby installed on your system. I use RVM to manage multiple ruby versions installed on my system. If you want to go with that too, you can find instructions for installing the tool in its homepage and for installing ruby versions (a.k.a. rubies) the related documentation page.

This tutorial is using Ruby 2.4.2. You can check yours using: ruby --version or via RVM with rvm list.

Elasticsearch

Providing instructions for installing Elasticsearch is out of this post’s scope. I recommend visiting the tool’s related documenation site.

After the installation, just make sure to start the service if it’s not started.

You can check that with curl http://localhost:9200.

It should give something like:

{
  "name" : "asdfasdf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "asasdfdf-as",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "asdf",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

As you can see, we will use elasticsearch version 6.0.0 in this tutorial.

Hands on

We will be working in the awesome Ruby interactive shell (irb) but before starting it there’s a gem we need to install.

The elasticsearch gem

The elasticsearch gem is the official library provided by Elastic to interact with Elasticsearch instances in Ruby.

Install the gem and its dependencies use the following command in your terminal:

gem install elasticsearch

Ruby interactive shell

Start the Ruby interactive shell with:

irb

Load the Elasticsearch library we previously installed with:

require 'elasticsearch'

You should see something like the following:

>> require 'elasticsearch'
=> true
>>

From now on, in our session we can use the gem’s provided classes and modules to interact with Elasticsearch.

Creating the client

We have to create an Elasticsearch::Client instance through which we will execute commands on the index. In irb type:

@client = Elasticsearch::Client.new host: 'localhost', port: '9200'

Check the cluster’s health to make sure everything is right:

client.cluster.health
=> {"cluster_name"=>"elasticsearch", "status"=>"yellow", "timed_out"=>false, "number_of_nodes"=>1, "number_of_data_nodes"=>1, "active_primary_shards"=>10, "active_shards"=>10, "relocating_shards"=>0, "initializing_shards"=>0, "unassigned_shards"=>10, "delayed_unassigned_shards"=>0, "number_of_pending_tasks"=>0, "number_of_in_flight_fetch"=>0, "task_max_waiting_in_queue_millis"=>0, "active_shards_percent_as_number"=>50.0}

Make sure that the status from the output hash is not red.

Creating the index

Let’s name the index elastic_manpages and set that it is going to have three properties:

command: the Linux command (even though there are man pages for other stuff like time.conf which is a configuration file)
description: the short description provided in the top of the man page, in the NAME section
manpage: the corresponding man page

Create the index with the following command:

@client.indices.create   index: 'elastic_manpages',
                         body: {
                           mappings: {
                             document: {
                               properties: {
                                 command: {
                                   type: :text
                                 },
                                 description: {
                                   type: :text,
                                   analyzer: :english
                                 },
                                 manpage: {
                                   type: :text,
                                   analyzer: :english
                                 }
                               }
                             }
                           }
                         }

As you can see, we assigned the english analyzer to the description and manpage properties.

Why? Quoting the documentation on Language analyzers:

They are able to take the peculiarities of the specified language into account. For instance, the english analyzer comes with a set of English stopwords (common words like and or the that don’t have much impact on relevance), which it removes. This analyzer also is able to stem English words because it understands the rules of English grammar.

After typing the command, you should get a response like this one:

=> {"acknowledged"=>true, "shards_acknowledged"=>true, "index"=>"elastic_manpages"}

Indexing the man pages

Time to index the man pages. We are going to use the apropos command to list all entries of the mandb.

all_pages = `apropos .`.split "\n"

The all_pages variable now contains a list of all man entries in the following format:

erb (1)              - Ruby Templating

Note: the (1) part above defines what section of the manual the page is from.

For each entry of this list, we need to:

export the command and the description
retrieve the man page for each command
index a new document mapping the values of the previous items to the appropriate properties.

To split the raw apropos entry to the desired field, we will use the following regex:

/(.*)\s\(\d*.*\)\s*-\s*(.*)/

You can check that is working on Rubular.

Back in irb:

apropos_regex = /(.*)\s\(\d*.*\)\s*-\s*(.*)/
all_pages.each do |line|
  matches = apropos_regex.match line
  command = matches[1]
  description = matches[2]
  manpage = `man #{command}`

  @client.index index: 'elastic_manpages',
                type: :document,
                body: {
                  command: command,
                  description: description,
                  manpage: manpage
                }
end

It will take some time depending on your hardware and the amount of man pages on your OS. Wait for it to end. During the process of indexing you might see some warnings for some pages but don’t worry, the process will complete after all and we don’t care that much about them not being indexed correctly for the purpose of this tutorial.

Creating the search method

Now will create a method for full text searching so that we can execute it more than once to search.

In irb:

def search(term)
  result = @client.search  index: 'elastic_manpages',
                           size: 10,
                           body: {
                             query: {
                               multi_match: {
                                 query: term,
                                 type: :cross_fields,
                                 fields: ['command', 'description^3', 'manpage^3'],
                                 operator: :or,
                                 tie_breaker: 1.0,
                                 cutoff_frequency: 0.1
                               }
                             }
                           }

  result['hits']['hits'].map{ |hit| { command: hit['_source']['command'],
                                      description: hit['_source']['description'],
                                      manpage: hit['_source']['manpage'] } }
end

Notes on the search definition

size: 10: set the size of the results to be 10
multi_match: because we want to search on multiple fields (command, description, manpage)
type: :cross_fields: The cross_fields type is particularly useful with structured documents where multiple fields should match (read more here)
tie_breaker: add the scores for each term of the query (read more here)
cutoff_frequency: it prevents scoring / iterating high frequency terms and only takes the terms into account if a more significant / lower frequency term matches a document (read more here)

Play time!

That’s it. Time to check what we have created.

In irb:

search('edit image').each{ |a| puts "#{a[:command].ljust(20)} - #{a[:description]}"}; nil
gimp-2.8             - an image manipulation and paint program.
gimp                 - an image manipulation and paint program.
Image::Magick::Q16   - objected-oriented Perl interface to ImageMagick (Q16). Use it to create, edit, compose, or convert ...
gimp-console-2.8     - an image manipulation and paint program.
gimp-console         - an image manipulation and paint program.
bitmap               - bitmap editor and converter utilities for the X Window System
bmtoa                - bitmap editor and converter utilities for the X Window System
atobm                - bitmap editor and converter utilities for the X Window System
mailcap.order        - the mailcap ordering specifications
xwd                  - dump an image of an X window
=> nil
search('change owner').each{ |a| puts "#{a[:command].ljust(20)} - #{a[:description]}"}; nil
chown                - change file owner and group
chown                - change ownership of a file
fchownat             - change ownership of a file
fchown               - change ownership of a file
lchown32             - change ownership of a file
chown32              - change ownership of a file
fchown32             - change ownership of a file
lchown               - change ownership of a file
fchmod               - change permissions of a file
XSetEventQueueOwner  - set event queue owner on a shared Xlib/XCB connection
=> nil

You can modify the search definition as you please. Check more on the multi_match query here.

As always, cat photo.

Creating a bash completion script

Thu, 01 Mar 2018 02:00:00 +0200

I recently worked on creating a bash completion script for registering directory aliases and navigating to them with autocomplete in Bash and I found the feature really interesting.

In this post I will familiarize you with the process of adding bash completion to your scripts.

What is bash completion?

Bash completion is a functionality through which bash helps users type their commands faster and easier. It accomplishes that by presenting possible options when users press the tab key while typing a command.

$ git<tab><tab>
git                 git-receive-pack    git-upload-archive  
gitk                git-shell           git-upload-pack     
$ git-s<tab>
$ git-shell

How it works

The completion script is code that uses the builtin bash command complete to define which completion suggestions can be displayed for a given executable. The nature of the completion options vary from simple static to highly sophisticated.

Why bother

It is good to provide such a functionality to users:

to save them from typing text when it can be auto-completed
to help them know what are the available continuations to their commands
to prevent errors and improve their experience by hiding or showing options based on what they have already typed

Hands on

Here’s what we are going to do in this tutorial.

We will first create a dummy executable script called dothis. All it does is execute the command which resides on the number, that was passed as an argument, in user’s history. For example, the following command will simply execute the ls -a command given that it exists in history with number 235:

dothis 235

Then we will create a bash completion script that will display commands along with their number from user’s history and we will “bind” it to the dothis executable.

$ dothis <tab><tab>
ls
ls -la
cd ~
man history
git status
history | cut -c 8-

You can see a gif demonstrating the functionality at this tutorial’s code repository on GitHub.

Let the show begin.

Creating the executable script

Create a file named dothis in your working directory and add the following code:

if [ -z "$1" ]; then
  echo "No command number passed"
  exit 2
fi

exists=$(fc -l -1000 | grep ^$1 -- 2>/dev/null)

if [ -n "$exists" ]; then
  fc -s -- "$1"
else
  echo "Command with number $1 was not found in recent history"
  exit 2
fi

Notes:

We first check if the script was called with an argument
We then check if the specific number is included in the last 1000 commands
- if it exists we execute the command using the fc functionality
- otherwise we display an error message

Make the script executable with:

chmod +x ./dothis

We are going to execute this script many times in this tutorial so I suggest you place it in a folder that is included in your path so that we can access it from anywhere with just typing dothis.

I installed it in my home bin folder using:

install ./dothis ~/bin/dothis

You can do the same given that you have a ~/bin folder and that it is included in your PATH variable.

Try to see if it’s working:

dothis

You should see this.

$ dothis
No command number passed

Done.

Creating the completion script

Create a file named dothis-completion.bash. From now on we will be referring to this file with the term completion script.

Once we add some code to it, we will source it to allow the completion to take effect. We must source this file every single time we change something in it.

We will discuss our options for registering this script whenever a bash shell opens later on.

Static completion

Suppose that the dothis program supported a list of commands, for example:

now
tomorrow
never

Let’s use the complete command to register this list for completion. To use the proper terminology, we say we use the complete command to define a completion specification (compspec) for our program.

Add this to the completion script.

#/usr/bin/env bash
complete -W "now tomorrow never" dothis

What did we specify with the complete command above:

we used the -W (wordlist) option to provide a list of words for completion.
we defined to which “program” these completion words will be used (the dothis parameter)

Source the file:

source ./dothis-completion.bash

Now try pressing tab twice in the command line as shown below:

$ dothis <tab><tab>
never     now       tomorrow

Try again after typing the n:

$ dothis n<tab><tab>
never now

Magic! The completion options are automatically filtered to match only those starting with n.

Note: The options are not displayed in the order that we defined them in the word list, they are automatically sorted.

There are a lot of other options to be used instead of the -W that we used in this section. Most of them produce completions in a fixed manner, meaning that we don’t intervene dynamically to filter their output.

For example, if we wanted to have directory names as completion words for the dothis program, we would change the complete command to the following:

complete -A directory dothis

Pressing tab after the dothis program, we would get a list of the directories in the current directory from which we execute the script:

$ dothis <tab><tab>
dir1/ dir2/ dir3/

Find the complete list of the available flags here.

Dynamic completion

We will be producing the completions of the dothis executable with the following logic:

If user presses the tab key right after the command, we will show the last 50 executed commands along with their numbers in history
If user presses the tab key after typing a number that matches more than one commands from history, we will show only those commands along with their numbers in history
If user presses the tab key after a number that matches exactly one command in history, we auto-complete the number without appending the command’s literal (if it’s confusing, you will better understand later on, no worries)

Let’s start by defining a function that will execute each time user requests completion on a dothis command. Change the completion script to this:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY+=("now")
  COMPREPLY+=("tomorrow")
  COMPREPLY+=("never")
}

complete -F _dothis_completions dothis

Note the following:

we used the -F flag in the complete command defining that the _dothis_completions is the function that will provide the completions of the dothis executable
COMPREPLY is an array variable used to store the completions - the completion mechanism uses this variable to display its contents as completions

Now source the script and go for completion:

$ dothis <tab><tab>
never now tomorrow

Perfect. We produce the same completions as in the previous section with the word list. Or not? Try this:

$ dothis nev<tab><tab>
never     now       tomorrow

As you can see, even though we type nev and then we request for completion, the available options are always the same and nothing gets completed automatically. Why is this happening?

The contents of the COMPREPLY variable are always displayed. The function is responsible to add/remove entries from there now.
If the COMPREPLY variable had only one element then that word would be automatically completed in the command. Since current implementation always return the same three words, this will not happen.

Enter compgen: a builtin command that generates completions supporting most of the options of the complete command (ex. -W for word list, -d for directories) and filtering them based on what already has been typed by the user.

Don’t worry if you feel confused, everything will become clear later on.

Type the following in the console to better understand what compgen does:

$ compgen -W "now tomorrow never"
now
tomorrow
never
$ compgen -W "now tomorrow never" n
now
never
$ compgen -W "now tomorrow never" t
tomorrow

So now we can use it but we need to find a way to know what has been typed after the dothis command. We already have the way. The bash completion facilities provide bash variables related to the completion taking place. Here are the more important ones:

COMP_WORDS: an array of all the words typed after the name of the program the compspec belongs to
COMP_CWORD: an index of the COMP_WORDS array pointing to the word the current cursor is at - in other words, the index of the word the cursor was when the tab key was pressed
COMP_LINE: the current command line

To access the word just after the dothis word, we can use the value of COMP_WORDS[1]

Change the completion script again:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY=($(compgen -W "now tomorrow never" "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and there you are:

 $ dothis
never     now       tomorrow  
$ dothis n
never  now

Now, instead of the now, tomorrow, never words, we would like to see actual numbers from the command history.

The fc -l command followed by a negative number -n displays the last n commands. So, we will use:

fc -l -50

which lists the last 50 executed commands along with their numbers. The only manipulation we need to do is replace tabs with spaces in order to be displayed properly from the completion mechanism. sed to the rescue.

Change the completion script as follows:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and test in the console:

$ dothis <tab><tab>
source dothis-completion.bash   649 source dothis-completion.bash   666 cat ~/.bash_profile
clear                           650 clear                           667 cat ~/.bashrc
source dothis-completion.bash   651 source dothis-completion.bash   668 clear
source dothis-completion.bash   652 source dothis-completion.bash   669 install ./dothis ~/bin/dothis
clear                           653 source dothis-completion.bash   670 dothis
source dothis-completion.bash   654 clear                           671 dothis 6546545646
clear                           655 dothis 654                      672 clear
source dothis-completion.bash   656 dothis 631                      673 dothis
source dothis-completion.bash   657 dothis 150                      674 dothis 651
source dothis-completion.bash   658 dothis                          675 source dothis-completion.bash
clear                           659 clear                           676 dothis 651
dothis 623  ls -la              660 dothis                          677 dothis 659
clear                           661 install ./dothis ~/bin/dothis   678 clear
source dothis-completion.bash   662 dothis                          679 dothis 665
clear                           663 install ./dothis ~/bin/dothis   680 clear
source dothis-completion.bash   664 dothis                          681 clear
clear                           665 cat ~/.bashrc

Not bad.

We do have a problem though. Try typing a number as you see it in your completion list and the press the key again.

$ dothis 623<tab>
$ dothis 623  ls 623  ls -la
...
$ dothis 623  ls 623  ls 623  ls 623  ls 623  ls -la

This is happening because in our completion script, we used the ${COMP_WORDS[1]} to always check the first typed word after the dothis command (the number 623 in the above snippet). Hence the completion continues suggesting the same completion again and again while pressing the tab key.

To fix this, we will not allow any kind of completion to take place if at least one argument has already been typed. We will add a condition in our function checking the size of the aforementioned COMP_WORDS array.

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  COMPREPLY=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and retry.

$ dothis 623<tab>
$ dothis 623 ls -la<tab> # SUCCESS: nothing happens here

There is another thing we don’t like though. We do want to display the numbers along with the corresponding commands to help users decide which one is the desired but when there is only one completion suggestion and gets automatically picked by the completion mechanism, we shouldn’t append the command literal too.

In other words, our dothis executable accepts only a number and we haven’t added any functionality checking or expecting other arguments. When our completion function gives only one result, we should trim the command literal and respond only with the command number.

To accomplish this, we will keep the response of the compgen command in an array variable and if its size is 1 we will trim the one and only element to keep just the number. Otherwise, we’ll let the array as is.

Change the completion script to this:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  # keep the suggestions in a local variable
  local suggestions=($(compgen -W "$(fc -l -50 | sed 's/\t/ /')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    # if there's only one match, we remove the command literal
    # to proceed with the automatic completion of the number
    local number=$(echo ${suggestions[0]/%\ */})
    COMPREPLY=("$number")
  else
    # more than one suggestions resolved,
    # respond with the suggestions intact
    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Registering the completion script

If you want to enable the completion just for you on your machine, all you have to do is add a line in your .bashrc file sourcing the script:

source <path-to-your-script>/dothis-completion.bash

If you want to enable the completion for all users, you can just copy the script under /etc/bash_completion.d/ and it will automatically be loaded by Bash.

Fine tuning the completion script

Some extra steps for better results :)

Displaying each entry in a new line

Well, in the bash completion script I was working on, I too had to present suggestions consisting of two parts. I wanted to display the first part with the default color and the second part with a gray color to point out that it was just help text. In this tutorial’s example, it would be nice to present the numbers in the default color but the command literal in another one, less fancy.

Unfortunately, this is not possible at least for the time being because the completions are displayed as plain text and color directives are not processed (for example: \e[34mBlue).

What we can do though to improve user experience (or not :D) is to display each entry in a new line. This solution is not that obvious since again we can’t just append a new line character in each COMPREPLY entry. We will follow a rather hackish way and pad suggestion literals to such a width that fills the terminal’s width.

Enter printf. If you want to display each suggestion on each own line, change the completion script to the following:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  local IFS=$'\n'
  local suggestions=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    local number="${suggestions[0]/%\ */}"
    COMPREPLY=("$number")
  else
    for i in "${!suggestions[@]}"; do
      suggestions[$i]="$(printf '%*s' "-$COLUMNS"  "${suggestions[$i]}")"
    done

    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Source and test:

dothis <tab><tab>
...
499 source dothis-completion.bash                   
500 clear
...       
503 dothis 500

Customizable behavior

In our case, we hard-coded to display the last 50 commands for completion. This is not a good practice. We should first respect what each user might prefer and if he/she hasn’t made any preference then we should default to 50.

To accomplish that we will check if an environment variable DOTHIS_COMPLETION_COMMANDS_NUMBER has been set.

Change the completion script one last time:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  local commands_number=${DOTHIS_COMPLETION_COMMANDS_NUMBER:-50}
  local IFS=$'\n'
  local suggestions=($(compgen -W "$(fc -l -$commands_number | sed 's/\t//')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    local number="${suggestions[0]/%\ */}"
    COMPREPLY=("$number")
  else
    for i in "${!suggestions[@]}"; do
      suggestions[$i]="$(printf '%*s' "-$COLUMNS"  "${suggestions[$i]}")"
    done

    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Source and test:

export DOTHIS_COMPLETION_COMMANDS_NUMBER=5
$ dothis <tab><tab>
clear
source ./dothis-completion.bash
dothis clear
clear
export DOTHIS_COMPLETION_COMMANDS_NUMBER=5

Useful links

Code and Comments

You can find the code of this tutorial on GitHub.

For feedback, comments, typos etc. please open an issue in the repository.

Long post, cat photo

Let me introduce you to my debugger.

That’s all folks!

Creating a Linux Desktop application with Ruby

Thu, 25 Jan 2018 21:00:00 +0200

Lately I was experimenting with GTK and its Ruby bindings and I decided to write a tutorial introducing this functionality. In this post we are going to create a simple ToDo application (something like what we created here with Ruby on Rails) using the gtk3 gem a.k.a. the GTK+ Ruby bindings.

Note: The code of the tutorial is available at GitHub.

What is GTK+

Quoting the toolkit’s page:

GTK+, or the GIMP Toolkit, is a multi-platform toolkit for creating graphical user interfaces. Offering a complete set of widgets, GTK+ is suitable for projects ranging from small one-off tools to complete application suites.

..and about its creation:

GTK+ was initially developed for and used by the GIMP, the GNU Image Manipulation Program. It is called the “The GIMP ToolKit” so that the origins of the project are remembered. Today it is more commonly known as GTK+ for short and is used by a large number of applications including the GNU project’s GNOME desktop.

Prerequisites

GTK+ version

Make sure you have GTK+ installed.

The OS in which I developed the tutorial’s application is Ubuntu 16.04 which has GTK+ installed by default (version: 3.18).

You can check yours with the following command:

dpkg -l libgtk-3-0

Ruby

This tutorial is using Ruby 2.4.2. You can check yours using: ruby --version or via RVM with rvm list.

Glade

Again, quoting the tool’s page

Glade is a RAD tool to enable quick & easy development of user interfaces for the GTK+ toolkit and the GNOME desktop environment

We will use Glade to design the user interface of our application. If you are on Ubuntu, install glade with:

sudo apt install glade

gtk3 gem

This gem provides the Ruby bindings for the GTK+ toolkit. In other words, it allows us to talk to the GTK+ API using the Ruby language.

Install the gem via:

gem install gtk3

The application specs

We will build an application that:

it will have a user interface (desktop application)
it will allow users to set miscellaneous properties to each item (such as priority)
it will allow users to create and edit ToDo items
- all items will be saved as files in the user’s home directory in a folder named .gtk-todo-tutorial
it will allow users to archive ToDo items
- archived items should be put in their own folder archived

The application structure

gtk-todo-tutorial # root directory
  |-- application
    |-- ui # everything related to the ui of the application
    |-- models # our models
    |-- lib # the directory to host any utilities we might need
  |-- resources # directory to host the resources of our application
  gtk-todo # the executable that will start our application

Let’s start!

Building the ToDo application

Initializing the application

Create a directory in which we will save all files that the application will need. As shown in the section above, I named mine gtk-todo-tutorial.

In there create a file named gtk-todo (that’s right, no extension) and add the following:

#!/usr/bin/env ruby

require 'gtk3'

app = Gtk::Application.new 'com.iridakos.gtk-todo', :flags_none

app.signal_connect :activate do |application|
  window = Gtk::ApplicationWindow.new(application)
  window.set_title 'Hello GTK+Ruby!'
  window.present
end

puts app.run

This is going to be the script that will start the application.

Note the shebang in the first line. This is how we define which interpreter must be used to execute the script under UNIX/Linux operating systems. This way, we don’t have to use ruby gtk-todo but just the script’s name gtk-todo.

Don’t try it yet though because we haven’t changed the mode of the file so as to be executable. To do so, type the following command in a terminal after navigating to the application’s root directory:

chmod +x ./gtk-todo # make the script executable

Now from the console execute:

./gtk-todo # execute the script

Ta daaaaaa

Notes

The application object we defined above and all of the GTK+ widgets in general, emit signals to trigger events. Once an application starts running for example, it emits a signal to trigger the activate event. All we have to do is to define what we want to happen when this signal is emitted. We accomplished this by using the signal_connect instance method and passing it a block whose code will be executed upon the given event. We will be doing this a lot throughout the tutorial.
When we initialized the Gtk::Application object we passed two parameters:
- com.iridakos.gtk-todo: this is our application’s id and in general it should be a reverse DNS style identifier. For more information about its usage and best practices check here.
- :flags_none: this is a flag defining the behavior of the application. In our case, we used the default behavior. Check here all the flags and the type of applications they define. You can use the Ruby equivalent flags as defined in Gio::ApplicationFlags.constants. For example, instead of using the :flags_none we could instead use Gio::ApplicationFlags::FLAGS_NONE

Suppose the application object we previously created (Gtk::Application) had a lot of things to do when the activate signal was emitted or that we wanted to connect to more signals. We would end up creating a huge gtk-todo script file making it hard to read/maintain. Time to refactor.

As described in the The application structure section, create a folder named application along with its sub-folders ui, models and lib.

In the ui folder we will place all files related to our user interface.
In the models folder we will place all files related to our models.
In the lib folder we will place all other files that don’t belong to either of the aforementioned folders.

We are going to define a new subclass of the Gtk::Application class for our application. Create a file named application.rb under application/ui/todo with the following contents.

module ToDo
  class Application < Gtk::Application
    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      signal_connect :activate do |application|
        window = Gtk::ApplicationWindow.new(application)
        window.set_title 'Hello GTK+Ruby!'
        window.present
      end
    end
  end
end

and change the gtk-todo script accordingly:

#!/usr/bin/env ruby

require 'gtk3'

app = ToDo::Application.new

puts app.run

Much cleaner, isn’t it? Yeah, but it doesn’t work. You should be getting something like:

./gtk-todo:5:in `<main>': uninitialized constant ToDo (NameError)

The problem is that we haven’t required any of the ruby files placed in the application folder. Change the script file as follows and execute it again.

#!/usr/bin/env ruby

require 'gtk3'

# Require all ruby files in the application folder recursively
application_root_path = File.expand_path(__dir__)
Dir[File.join(application_root_path, '**', '*.rb')].each { |file| require file }

app = ToDo::Application.new

puts app.run

You should be fine.

Resources

At the beginning of this tutorial we said that we would use Glade to design the user interface of the application. Glade actually produces xml files with the appropriate elements and attributes that reflect what we designed via its user interface. We somehow need to make use of these files so that our application gets the UI we designed.

These files are resources for the application and the GResource API provides a way for packing them all together in a binary file and afterwards accessing them from inside the application with advantages as opposed to manually having to deal with already loaded resources, their location on the file system etc. Read more about the API here.

Describing the resources

First, we need to create a file describing the resources of the application. Create a file named gresources.xml and place it directly under the resources folder.

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
  </gresource>
</gresources>

In this “description” we actually say: we have a resource which is located under the ui directory (relative to this xml file) with name application_window.ui. Before loading this resource please remove the blanks. Thanks. Of course this is not going to work now since we haven’t created the resource via Glade yet. Don’t worry though, one thing at a time.

Note: the xml-stripblanks directive will use the xmllint command to remove the blanks. In Ubuntu you have to install the package libxml2-utils to obtain it.

Building the resources binary file

In order to produce the binary resources file, we are going to use another utility of the GLib library called glib-compile-resources. Check if you have it installed with dpkg -l libglib2.0-bin. You should be seeing something like this:

ii  libglib2.0-bin     2.48.2-0ubuntu amd64          Programs for the GLib library

If not, then install the package (sudo apt install libglib2.0-bin in Ubuntu).

Let’s build the file. We will add the code in our script so that the resources are getting built every time we execute it. Change the gtk-todo script as follows.

#!/usr/bin/env ruby

require 'gtk3'
require 'fileutils'

# Require all ruby files in the application folder recursively
application_root_path = File.expand_path(__dir__)
Dir[File.join(application_root_path, '**', '*.rb')].each { |file| require file }

# Define the source & target files of the glib-compile-resources command
resource_xml = File.join(application_root_path, 'resources', 'gresources.xml')
resource_bin = File.join(application_root_path, 'gresource.bin')

# Build the binary
system("glib-compile-resources",
       "--target", resource_bin,
       "--sourcedir", File.dirname(resource_xml),
       resource_xml)

at_exit do
  # Before existing, please remove the binary we produced, thanks.
  FileUtils.rm_f(resource_bin)
end

app = ToDo::Application.new
puts app.run

and execute it. This happens in the console and it’s fine, we’ll fix it later on:

/.../gtk-todo-tutorial/resources/gresources.xml: Failed to locate 'ui/application_window.ui' in any source directory.

What we did:

added a require statement for the fileutils library so that we can use it in the at_exit call
defined the source and target files of the glib-compile-resources command
executed the glib-compile-resources command
set a hook so that before exiting the script (before the application exits) the binary file gets deleted so next time it gets build again

Loading the resources binary file

Ok, we described the resources, we packed them in a binary file. Now we have to load them and register them in the application so that we can use them. This is so easy as adding the following two lines before the at_exit hook.

resource = Gio::Resource.load(resource_bin)
Gio::Resources.register(resource)

That’s it. From now on, we are able to use the resources from anywhere inside the application (we’ll see how later on). Well now the script fails since it can’t load the binary which is not produced but…be patient, we are going to get to the interesting part soon. Actually now.

Designing the main application window

Introducing glade

Open glade.

A quick description of what you see.

On the left section there is a list of widgets which you can drag and drop in the middle section given that a widget can be placed there. For example, you can’t add a top level window inside a label widget. I will be calling this as the Widget section from now on.
On the middle section you see your widgets as they will appear (most of the times) in the application. I will be calling this as the Design section from now on.
On the right section there are two subsections:
- the top section contains the hierarchy of the widgets as added to the resource. I will be calling this as the Hierarchy section from now on.
- the bottom section contains all the properties that you can configure via Glade for a given selected widget of the aforementioned top section. I will be calling this as the Properties section from now on.

I will try to describe the steps for building this tutorial’s UI using Glade but if you are interested in building GTK+ applications you should take a look at the resources & tutorials for using the tool on the official page.

Create the application window design

We are going to create the application window. As you can guess, all we have to do is drag the widget ‘Application Window’ from the widget section to the design section.

Gtk::Builder is an object used in GTK+ applications to read textual descriptions of a user interface (like the one we will build via Glade) and build the described objects-widgets.

In the properties section, the first property is the ID and it has a default value applicationWindow1. If we let this property as is, then later on, through our code we would create a Gtk::Builder that would load the file produced by glade and in order to obtain the application window we would have to use something line:

application_window = builder.get_object('applicationWindow1')

application_window.signal_connect 'whatever' do |a,b|
...

The application_window object would be of class Gtk::ApplicationWindow and thus whatever we had to add to its behavior (like setting its title) would take place out of the original class. Also, as shown in the snippet above, the code to connect to a signal of the window would be placed inside the file that instantiated it.

Good news though, a GTK+ feature introduced in 2013 allows the creation of composite widget templates which among other advantages allows as to define the custom class for the widget (which eventually derives from an existing GTK::Widget class in general). Don’t worry if you are confused. You are going to understand what is going on after we write some code and view the results.

Now, in order to define our design as a template, check the Composite checkbox in the property widget. Note that the ID property changed to Class Name. Fill in there TodoApplicationWindow. This is the class we are going to create in our code to represent this widget.

Save the file with name application_window.ui in a new folder named ui inside the resources. If you open the file from an editor you will see this:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated with glade 3.18.3 -->
<interface>
  <requires lib="gtk+" version="3.12"/>
  <template class="TodoApplicationWindow" parent="GtkApplicationWindow">
    <property name="can_focus">False</property>
    <child>
      <placeholder/>
    </child>
  </template>
</interface>

As you can see, our widget has a class and parent attribute. Following the parent class attribute convention, obviously, our class has to be defined inside a module named Todo. Before getting there, let’s try to start the application by executing the script (./gtk-todo).

Yeah! It starts!

Create the application window class

While running the application, if you check the contents of the application’s root directory you can see the gresource.bin file there. Even though the application starts successfully because the resource bin is present and it can register it, we do not use it yet. We still initiate an ordinary Gtk::ApplicationWindow in our application.rb file and that all we show. Time to create our custom application window class.

Create a file named application_window.rb, place it under application/ui/todo folder and add the following content.

module Todo
  class ApplicationWindow < Gtk::ApplicationWindow
    # Register the class in the GLib world
    type_register

    class << self
      def init
        # Set the template from the resources binary
        set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'
      end
    end

    def initialize(application)
      super application: application

      set_title 'GTK+ Simple ToDo'
    end
  end
end

We defined the init method as a singleton method on the class after opening the eigenclass in order to bind the template of this widget to the previously registered resource file.

Before that, we called the type_register class method which registers and make available our custom widget class to the GLib world.

Finally, each time we create an instance of this window, we set its title to GTK+ Simple ToDo.

Now, let’s go back to the application.rb file and use what we just implemented:

module ToDo
  class Application < Gtk::Application
    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      signal_connect :activate do |application|
        window = Todo::ApplicationWindow.new(application)
        window.present
      end
    end
  end
end

Execute the script.

Define the model

For simplicity, we are going to save the ToDo items in files in json format under a dedicated hidden folder in user’s home directory. Of course, in a real life application we would use a database but this is out of the scope of this tutorial.

Our Todo::Item model will have the following properties:

id: the id of the item
title: the title
notes: any notes
priority: its priority
creation_datetime: the date & time the item was created
filename: the name of the file that an item is saved to

Create a file named item.rb under the application/models directory, with contents:

require 'securerandom'
require 'json'

module Todo
  class Item
    PROPERTIES = [:id, :title, :notes, :priority, :filename, :creation_datetime].freeze

    PRIORITIES = ['high', 'medium', 'normal', 'low'].freeze

    attr_accessor *PROPERTIES

    def initialize(options = {})
      if user_data_path = options[:user_data_path]
        # New item. When saved, it will be placed under the :user_data_path value
        @id = SecureRandom.uuid
        @creation_datetime = Time.now.to_s
        @filename = "#{user_data_path}/#{id}.json"
      elsif filename = options[:filename]
        # Load an existing item
        load_from_file filename
      else
        raise ArgumentError, 'Please specify the :user_data_path for new item or the :filename to load existing'
      end
    end

    # Loads an item from a file
    def load_from_file(filename)
      properties = JSON.parse(File.read(filename))

      # Assign the properties
      PROPERTIES.each do |property|
        self.send "#{property}=", properties[property.to_s]
      end
    rescue => e
      raise ArgumentError, "Failed to load existing item: #{e.message}"
    end

    # Resolves if an item is new
    def is_new?
      !File.exists? @filename
    end

    # Saves an item to its `filename` location
    def save!
      File.open(@filename, 'w') do |file|
        file.write self.to_json
      end
    end

    # Deletes an item
    def delete!
      raise 'Item is not saved!' if is_new?

      File.delete(@filename)
    end

    # Produces a json string for the item
    def to_json
      result = {}
      PROPERTIES.each do |prop|
        result[prop] = self.send prop
      end

      result.to_json
    end
  end
end

As you can see, we defined methods to:

initialize an item
- as new by defining the :user_data_path in which it will be saved later on
- as existing by defining the :filename to be loaded from. The filename must be a json file previously generated by an item
load an item from a file
resolve whether an items is new or not (saved at least once in the :user_data_path or not)
save an item by writing its json string to a file
delete an item
produce the json string of an item as a hash of its properties

Add a new item

Create the button

Let’s add a button to our application window for adding a new item. Open the resources/ui/application_window.ui file in glade.

Drag a Button from the widget section to the design section.
In the properties section, set its ID value to add_new_item_button.
Near the bottom of the General tab in the properties section there’s a text area just below the Label with optional image option. Change its value from button to Add new item
Save the file and execute the script

Don’t worry, we will improve the design later on. Now, let’s see how to connect functionality to our button’s clicked event.

First, we have to update our application window class so that it learns about its new child, the button with id add_new_item_button. Then, we can access the child to alter its behavior.

Change the init method as follows:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'

  bind_template_child 'add_new_item_button'
end

Pretty simple, right? The bind_template_child method does exactly what it says and from now on every instance of our Todo::ApplicationWindow class will have an add_new_item_button method to access the related button. So, let’s alter the initialize method as follows.

def initialize(application)
  super application: application

  set_title 'GTK+ Simple ToDo'

  add_new_item_button.signal_connect 'clicked' do |button, application|
    puts "OMG! I AM CLICKED"
  end
end

As you can see, we access the button by the add_new_item_button method and we define what we want to take place when clicked. Restart the application and try clicking it. In the console you should see the message OMG! I AM CLICKED every time you click the button.

What we want though to happen when we click this button is to show a new window through which we will save a ToDo item. You guessed right. Glade o’clock.

Create the new item window

Create a new project in Glade by pressing the most left icon of the top bar or by selecting File > New from the application menu.
Drag a Window from the widget section to the design area.
Check its Composite property and name the class TodoNewItemWindow.

Drag a Grid from the widget section and place it in the window we added in the previous steps.
Set its rows number to 5 and its columns number to 2 in the window that popped up.
In the General tab of its properties window, set its Rows spacing and Columns spacing to 10 (the numbers are in pixels).
In the Common tab of the properties section, set the Widget Spacing > Margins > Top, Bottom, Left, Right all to 10 so that the contents are not stuck to the borders of the window.

Drag four times a Label widget from the widget section and place them in each row of the Grid.
Change their Label property from top to bottom as:
- Id:
- Title:
- Notes:
- Priority:
In the General tab of the properties section, change the Alignment and Padding > Alignment > Horizontal property from 0.50 to 1 for each property. This will align the label text on the right.
This step is optional but I suggest that you do it: We will not bind these labels in our window since we don’t need to alter their state or behavior. So in this context, we don’t need to set a descriptive id for each of them like we did for the add_new_item_button button in the application window. BUT. We are going to add more elements to our design and the hierarchy of the widgets in Glade will be hard to read with all the label1. label2. So set a descriptive id to each to make our lives easier (like id_label, title_label, notes_label, priority_label). I even set the grid’s id to main_grid cause I don’t like seeing numbers in ids or variable names :)

Drag a Label from the widget section to the second column of the grid’s first row. The id is automatically generated by our model thus we won’t allow editing so a label to display it is more than enough.
Set the ID property to id_value_label.
Set the Alignment and Padding > Alignment > Horizontal property to 0 so that the text aligns on the left.
We are going to bind this widget to our window class so that we can change its text each time we load the window so setting a label through glade is not needed but doing so makes the design look closer to what it’ll look like when rendered with actual data. So you can optionally set a label here to whatever suits you better. I set mine to id-of-the-todo-item-here.

Drag a Text Entry from the widget section to the second column of the second row of the grid.
Set its ‘ID’ property to title_text_entry. As you may have noticed, I prefer obtaining the widget type in the id so that the code in the class becomes more readable later on.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it. This way, the widget will expand horizontally every time its parent (a.k.a. the grid) is resized.

Drag a Text View from the widget section to the second column of the third row of the grid.
Set its ID to notes. Nope, just testing you. Set its ID property to notes_text_view.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal, Vertical checkboxes and turn on the switches which are right next to them. This way, the widget will expand horizontally and vertically every time its parent (a.k.a. the grid) is resized.

Drag a Combo Box from the widget section to the second column of the forth row of the grid.
Set its ID to priority_combo_box.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it. This way, the widget will expand horizontally every time its parent (a.k.a. the grid) is resized.
This widget is actually a drop down element and we are going to populate its values that can be selected by the user when it shows up inside our window class.

Drag a Button Box from the widget section to the second column of the last row of the grid.
On the popped up window select 2 number of items.
In the General tab of the properties section set the Box Attributes > Orientation property to Horizontal.
In the General tab of the properties section set the Box Attributes > Spacing property to 10.
In the Common tab of the properties section set the Widget Spacing > Alignment > Horizontal to Center.
Again, this widget won’t be altered by our code but you can give it a descriptive ID for readability. I named mine actions_box

Drag a Button widget twice and place it to each of the two boxes of the button box widget we added in the previous step.
Set their ID properties to cancel_button & save_button respectively.
In the General tab of the properties window, set their Button Content > Label with option image property to Cancel and Save respectively.

The window is ready. Save the file under resources/ui/new_item_window.ui.

Time to port it in our application.

Implement the new item window class

Before implementing the new class, we must update our GResource description file a.k.a. resources/gresources.xml to obtain the new resource:

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
    <file preprocess="xml-stripblanks">ui/new_item_window.ui</file>
  </gresource>
</gresources>

Now we can create the new window class. Create a file under application/ui/todo named new_item_window.rb and set its contents as follows.

module Todo
  class NewItemWindow < Gtk::Window
    # Register the class in the GLib world
    type_register

    class << self
      def init
        # Set the template from the resources binary
        set_template resource: '/com/iridakos/gtk-todo/ui/new_item_window.ui'
      end
    end

    def initialize(application)
      super application: application
    end
  end
end

Nothing special here. We just changed the template resource to point to the correct file of our resources.

We have to change the add_new_item_button code that executes on the clicked signal to show the new item window. Go ahead and change that code in application_window.rb to this:

add_new_item_button.signal_connect 'clicked' do |button|
  new_item_window = NewItemWindow.new(application)
  new_item_window.present
end

Let’s see what we have done. Start the application and click on the Add new item button. Ta daaaaaaa

Of course, nothing happens when pressing the buttons. We will change that.

First we will bind the ui widgets in the Todo::NewItemWindow class.

Change the init method the to this:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/new_item_window.ui'

  # Bind the window's widgets
  bind_template_child 'id_value_label'
  bind_template_child 'title_text_entry'
  bind_template_child 'notes_text_view'
  bind_template_child 'priority_combo_box'
  bind_template_child 'cancel_button'
  bind_template_child 'save_button'
end

This window is going to be shown either when creating a new Todo item or editing an existing one. Thus the new_item_window naming is not very valid. This was intentional though so that we refactor the code later (No it was not :D I made a mistake when writing the tutorial. In any case, we’ll refactor later on).

For now, we will update the initialize method of the window to require one extra parameter, the Todo::Item to be created or edited. We can then set a more meaningful window title and change the children widgets to reflect the current item.

Change the initialize method to this:

def initialize(application, item)
  super application: application
  set_title "ToDo item #{item.id} - #{item.is_new? ? 'Create' : 'Edit' } Mode"

  id_value_label.text = item.id
  title_text_entry.text = item.title if item.title
  notes_text_view.buffer.text = item.notes if item.notes

  # Configure the combo box
  model = Gtk::ListStore.new(String)
  Todo::Item::PRIORITIES.each do |priority|
    iterator = model.append
    iterator[0] = priority
  end

  priority_combo_box.model = model
  renderer = Gtk::CellRendererText.new
  priority_combo_box.pack_start(renderer, true)
  priority_combo_box.set_attributes(renderer, "text" => 0)

  priority_combo_box.set_active(Todo::Item::PRIORITIES.index(item.priority)) if item.priority
end

and add the constant PRIORITIES in the application/models/item.rb file just below the PROPERTIES constant as:

PRIORITIES = ['high', 'medium', 'normal', 'low'].freeze

What did we do here?

We set the window’s title to a string containing the id of the current item and the mode depending on whether the item is now being created or edited.
We set the id_value_label text to display the current item’s id.
We set the title_text_entry text to display the current item’s title.
We set the notes_text_view text to display the current item’s notes.

We create a model for the priority_combo_box whose entries are going to have only one String value. At a first sight, a Gtk::ListStore model might look a little confusing. I will try to explain how it works now.

Suppose we want to display in a combo box a list of country codes and their respective country names.
We would create a Gtk::ListStore defining that its entries would consist of two string values: one for the country code and one for the country name. Thus we would initialize the ListStore as:
```
model = Gtk::ListStore.new(String, String)
```

In order to fill the model with data we would do something like this (make sure you don’t miss the comments in the snippet):

[['gr', 'Greece'], ['jp','Japan'], ['nl', 'Netherlands']].each do |country_pair|
  entry = model.append
  # Each entry has two string positions since that's how we initialized the Gtk::ListStore
  # Store the country code in position 0
  entry[0] = country_pair[0]
  # Store the country name in position 1
  entry[1] = country_pair[1]
end

We also had to configure the combo box to render two text columns/cells (again, make sure you don’t miss the comments in the snippet):

country_code_renderer = Gtk::CellRendererText.new
# Add the first renderer
combo.pack_start(country_code_renderer, true)
# Use the value in index 0 of each model entry a.k.a. the country code
combo.set_attributes(country_code_renderer, 'text' => 0)

country_name_renderer = Gtk::CellRendererText.new
# Add the second renderer
combo.pack_start(country_name_renderer, true)
# Use the value in index 1 of each model entry a.k.a. the country name
combo.set_attributes(country_name_renderer, 'text' => 1)

I hope I made it a little more clearer…

We add a simple text renderer in the combo box and instruct it to display the one and only value of each model’s entry (a.k.a. position 0). Imagine that our model is something like [['high'],['medium'],['normal'],['low']] and 0 is actually the first element of each sub-array. I will stop with the model-combo-text-renderer explanations now…

Configure the user data path

Remember that when initializing a new Todo::Item (not an existing one) we had to define a :user_data_path in which it would be saved. We are going to resolve this path when the application starts and make it accessible from all the widgets.

All we have to do is check if the .gtk-todo-tutorial path exists inside the user’s home ~ directory. If not, then we will create it. Then we set this as an instance variable of the application. All widgets have access to the application instance. Sooooo….all widgets have access to this user path variable.

Change the application/application.rb file to this:

module ToDo
  class Application < Gtk::Application
    attr_reader :user_data_path

    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      @user_data_path = File.expand_path('~/.gtk-todo-tutorial')
      unless File.directory?(@user_data_path)
        puts "First run. Creating user's application path: #{@user_data_path}"
        FileUtils.mkdir_p(@user_data_path)
      end

      signal_connect :activate do |application|
        window = Todo::ApplicationWindow.new(application)
        window.present
      end
    end
  end
end

One last thing that we have to do before testing what we have done so far is to instantiate the Todo::NewItemWindow when the add_new_item_button is clicked complying with the changes we made a.k.a. change the code in application_window.rb to this:

add_new_item_button.signal_connect 'clicked' do |button|
  new_item_window = NewItemWindow.new(application, Todo::Item.new(user_data_path: application.user_data_path))
  new_item_window.present
end

Start the application and click on the Add new item button. Ta daaaaaa (note the - Create mode part in the title).

Cancel the item creation/update

In order to close the Todo::NewItemWindow window when user clicks the cancel_button all we have to do is to add this to the window’s initialize method:

cancel_button.signal_connect 'clicked' do |button|
  close
end

close is an instance method of the Gtk::Window class that surprisingly enough closes the window.

Save the item

Saving an item involves two steps:

Update the item’s properties based on the widgets’ values
Call the save! method on the Todo::Item instance

Again, our code will be placed in the initialie method of the Todo::NewItemWindow:

save_button.signal_connect 'clicked' do |button|
  item.title = title_text_entry.text
  item.notes = notes_text_view.buffer.text
  item.priority = priority_combo_box.active_iter.get_value(0) if priority_combo_box.active_iter
  item.save!
  close
end

Note that we again close the window after saving the item.

Let’s try that out.

Pressing save and navigating to your ~/.gtk-todo-tutorial folder you should see a file there. Mine had the following contents:

{
	"id": "3d635839-66d0-4ce6-af31-e81b47b3e585",
	"title": "Optimize the priorities model creation",
	"notes": "It doesn't have to be initialized upon each window creation.",
	"priority": "high",
	"filename": "/home/iridakos/.gtk-todo-tutorial/3d635839-66d0-4ce6-af31-e81b47b3e585.json",
	"creation_datetime": "2018-01-25 18:09:51 +0200"
}

Don’t forget to try out the cancel button as well.

Awesome!!!

View ToDo items

We have left the Todo::ApplicationWindow to contain only one button. Time to change that.

We want the window to have the Add new item on the top but below it there should be a list with all of our todo items. To accomplish that we are going to add a Gtk::ListBox in our design which can contain any number of rows.

Update the application window

Open the resources/ui/application_window.ui file in Glade.
If you try to drag a List Box widget from the widget section directly on the window nothing happens. That is normal. First we have to split the window in two parts. One part for the button and one for the list box. Bear with me.
Right click on the new_item_window in the hierarchy section and select Add parent > Box.
In the popped up window, set that you need 2 items.
The orientation of the box is already vertical so we are fine.

Now, drag a List Box and place it on the free are of the previously added box.
Set its ID property to todo_items_list_box
Set is Selection mode to None since we won’t provide such a functionality.

Design the ToDo item list box row

Each row of the list box that we created in the previous step is going to be more complex than a row of text. It is going to contain widgets that will allow the user to expand an item’s notes, and to delete or edit the item.

Create a new project in Glade as we did for the new_item_window.ui. Save it under resources/ui/todo_item_list_box_row.ui.
Unfortunately, at least in my version of Glade, there is no List Box Row widget in the widget section so in order to add one directly as the top level widget of our project, we will do it in a kinda hackish way.
Drag a List Box from the widget section to the design area.
Inside the hierarchy section right click on the List Box and select Add Row

Inside the hierarchy section right click on the newly added List Box Row which is nested under the List Box and select Remove parent. There it is. The List Box Row is the top level widget of the project now.

Check the widget’s Composite property and set its name to TodoItemListBoxRow.
Drag a Box from the widget section to the design area inside our List Box Row.
Set 2 items in the popped up window.
Set its ID property to main_box

Drag another Box from the widget section to the first row of the previously added box.
Set 2 items in the popped up window.
Set its ID property to todo_item_top_box.
Set its Orientation property to Horizontal.
Set its Spacing (General tab) property to 10.

Drag a Label from the widget section to the first column of the todo_item_top_box.
Set its ID property to todo_item_title_label.
Set its Alignment and Padding > Alignment > Horizontal property to 0.00.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it so that the label expands to available space.

Drag a Button from the widget section to the second column of the todo_item_top_box.
Set its ID property to details_button
Check the Button Content > Label with optional image radio and type ... (three dots).

Drag a Revealer widget from the widget section to the second row of the main_box.
Turn off the Reveal Child switch in the General tab.
Set its ID property to todo_item_details_revealer.
Set its Transition type property to Slide Down.

Drag a Box from the widget section to the reveal space.
Set its items to 2 in the popped up window.
Set its ID property to details_box.
In the Common tab, set its Widget Spacing > Margins > Top property to 10.

Drag a Button Box from the widget section to the first row of the details_box.
Set its ID property to todo_item_action_box.
Set its Layout style property to expand.

Drag two Button widgets to the first and second column of the todo_item_action_box respectively.
Set their ID properties to delete_button and edit_button respectively.
Set their Button Content > Label with optional image properties to Delete and Edit respective.

Drag a Viewport widget from the widget section to the second row of the details_box.
Set its ID property to todo_action_notes_viewport.
Drag a Text View widget from the widget section to the todo_action_notes_viewport that we just added.
Set is ID to todo_item_notes_text_view.
Uncheck its Editable property in the General tab of the properties section.

Create the ToDo item list box row class

Now we will create the class reflecting the user interface of the list box row which we just created.

First we have to update our GResource description file to include the newly created design. Change the resources/gresources.xml file as follows:

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
    <file preprocess="xml-stripblanks">ui/new_item_window.ui</file>
    <file preprocess="xml-stripblanks">ui/todo_item_list_box_row.ui</file>
  </gresource>
</gresources>

Create a file named item_list_box_row.rb inside the application/ui folder and add the following content.

module Todo
  class ItemListBoxRow < Gtk::ListBoxRow
    type_register

    class << self
      def init
        set_template resource: '/com/iridakos/gtk-todo/ui/todo_item_list_box_row.ui'
      end
    end

    def initialize(item)
      super()
    end
  end
end

We will not bind any children at the moment here.

When starting the application, we have to search for files in the :user_data_path and for each file we must create a Todo::Item instance. For each instance, we must also add a new Todo::ItemListBoxRow to the Todo::ApplicationWindow’s todo_items_list_box list box. One thing at a time.

First of all, let’s bind the todo_items_list_box in the Todo::ApplicationWindow class. Change the init method as follows:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'

  bind_template_child 'add_new_item_button'
  bind_template_child 'todo_items_list_box'
end

Next, we will add an instance method in the same class that will be responsible to load the todo list items in the related list box. Add this code in Todo::ApplicationWindow.

def load_todo_items
  todo_items_list_box.children.each { |child| todo_items_list_box.remove child }

  json_files = Dir[File.join(File.expand_path(application.user_data_path), '*.json')]
  items = json_files.map{ |filename| Todo::Item.new(filename: filename) }

  items.each do |item|
    todo_items_list_box.add Todo::ItemListBoxRow.new(item)
  end
end

and then call this method at the end of the initialize method.

def initialize(application)
  super application: application

  set_title 'GTK+ Simple ToDo'

  add_new_item_button.signal_connect 'clicked' do |button|
    new_item_window = NewItemWindow.new(application, Todo::Item.new(user_data_path: application.user_data_path))
    new_item_window.present
  end

  load_todo_items
end

Note: We first make sure we empty the list box of its current children rows and we refill it. This way, we will call this method after saving a Todo::Item via the signal_connect of the save_button of the Todo::NewItemWindow and the parent application window will be reloaded! Here’s the updated code (in application/ui/new_item_window.rb):

save_button.signal_connect 'clicked' do |button|
  item.title = title_text_entry.text
  item.notes = notes_text_view.buffer.text
  item.priority = priority_combo_box.active_iter.get_value(0) if priority_combo_box.active_iter
  item.save!

  close

  # Locate the application window
  application_window = application.windows.find { |w| w.is_a? Todo::ApplicationWindow }
  application_window.load_todo_items
end

Previously, we used this code:

json_files = Dir[File.join(File.expand_path(application.user_data_path), '*.json')]

in order to find all the names of the files that exist in the application user data path with json extension.

Let’s see what we’ve created. Start the application and try adding a new ToDo item. After pressing the Save button you should see the parent Todo::ApplicationWindow being automatically updated with the new item!

What’s left to do, is to complete the functionality of the Todo::ItemListBoxRow.

We will first bind the widgets. Change the init method of the Todo::ItemListBoxRow class as follows:

def init
  set_template resource: '/com/iridakos/gtk-todo/ui/todo_item_list_box_row.ui'

  bind_template_child 'details_button'
  bind_template_child 'todo_item_title_label'
  bind_template_child 'todo_item_details_revealer'
  bind_template_child 'todo_item_notes_text_view'
  bind_template_child 'delete_button'
  bind_template_child 'edit_button'
end

Then, we are going to setup the widgets based on the item of each row.

def initialize(item)
  super()

  todo_item_title_label.text = item.title || ''

  todo_item_notes_text_view.buffer.text = item.notes

  details_button.signal_connect 'clicked' do
    todo_item_details_revealer.set_reveal_child !todo_item_details_revealer.reveal_child?
  end

  delete_button.signal_connect 'clicked' do
    item.delete!

    # Locate the application window
    application_window = application.windows.find { |w| w.is_a? Todo::ApplicationWindow }
    application_window.load_todo_items
  end

  edit_button.signal_connect 'clicked' do
    new_item_window = NewItemWindow.new(application, item)
    new_item_window.present
  end
end

def application
  parent = self.parent
  parent = parent.parent while !parent.is_a? Gtk::Window
  parent.application
end

As you can see, when the details_button is clicked, we instruct the todo_item_details_revealer to swap the visibility of its contents.
After deleting an item, we find the application’s Todo::ApplicationWindow in order to call its load_todo_items as we did after saving an item.
When clicking to edit a button, we create a new instance of the Todo::NewItemWindow passing as item the current item. Works like a charm :D
Finally, we had to reach at the application parent of a list box row so we defined a simple instance method application that navigates through the widget’s parents until it reaches a window from which it can obtain the application object.

Save and run the application. There it is.

This has been a really long tutorial and even though there are so many items that we haven’t covered I think we better end it here.

Long post, cat photo.

Useful links

DuckRails v2.0.0 docker image

Thu, 09 Nov 2017 23:05:00 +0200

DuckRails now has its own docker image and so you can run the application without having to setup anything related to databases, secret keys etc. https://hub.docker.com/r/iridakos/duckrails/

Usage

You can now use:

docker pull iridakos/duckrails

to download the image and then you can start the application with:

docker run -p <port>:80 iridakos/duckrails

(replace <port> with the port you want to access the application locally).

Test that everything works as expected by navigating to http://localhost:<port>

You should be seeing the DuckRails home page!

For a detailed guide, check the repository’s wiki page.

Note

This was my first attempt to create a docker image so if you have any notes/improvements, you are more than welcomed to contribute. The Dockerfile is located at the root folder of the DuckRails repo.

DuckRails - Open source development tool for mocking API endpoints

Fri, 01 Apr 2016 10:00:00 +0300

Update - Nov. 18, 2017

Please visit the updated wiki pages of the project at GitHub. This post is old and does not reflect any changes being made to the application.

DuckRails is a development tool.

Its main purpose is to allow developers to quickly and dynamically mock API endpoints.

Why mock?

Some reasons to mock endpoints:

The service you want to consume:
- is still under development
- is not available

and you don’t want to implement the mocking inside your code.

There’s also another interesting point which transforms the DuckRails from a development tool to an actual API (the tool needs a lot of improvements though in order to be used like this in production): Since the mocks can be configured to be dynamic, you can actually wrap your actual endpoint and enhance it with new data, transformations on the original data and whatever else you can think of.

Installation

First of all, you will need to clone the repo:

git clone https://github.com/iridakos/duckrails.git

Then, you have to do some basic stuff pretty common for a Rails application:

cd duckrails
bundle install

This will install all the required gems (libraries) the application needs.

You can use DuckRails either in development (quick setup) or production (better performance) mode.

Development mode

bundle exec rake db:setup

This will setup your database. DuckRails is configured to use MySQL as a database. You can change the file config/database.yml to make it match your custom database setup. Here’s a guide if this is your case.

bundle exec rails server

This will start the application in development mode.

Production mode

For production mode, you will need to do some extra steps.

Your env needs to have a SECRET_KEY_BASE variable (used to secure your sessions from tampering).

To generate a key, you can use:

bundle exec rake secret

Then export the variable with the previous’ command output.

export SECRET_KEY_BASE="your_secret_key_base_here"

Setup the database:

RAILS_ENV=production bundle exec rake db:setup

Precompile the assets (images, stylesheets & javascripts):

RAILS_ENV=production bundle exec rake assets:precompile

Start the server:

bundle exec rails s -e production

That’s all. Navigate to http://localhost:3000 and you should be seeing this:

Using DuckRails

The tool’s functionality is pretty straight forward:

you can view all your mocks
create new mocks
edit existing mocks
delete mocks

Mock configuration

Create a new mock by pressing the related button in the application home page.

There are 4 main sections allowing to configure a mock.

General

Here you can configure the general properties of the mock.

Name: a name for this mock
Description: a general description for the mock, other users using the tool may want to understand your mock and this is a good place to explain it
Method: select the HTTP method this mock matches to. This value along with the route path is going to create a unique “endpoint” serving your mock.
Status: fill in the HTTP Status code this mock will respond with upon call (you can set dynamic status, view the Advance section of this document).
Route path: fill in the path of the mock. You can use path variables, example: /authors/:author_id/posts/:post_id and these will be available later on, when you configure the mock’s body via the @parameters variable (see below).

Response body

In this tab, you configure the response’s body.

Body type: select the type of the body’s content. You can either select:
- Static: implying that the response body will always be the same, no further process
- Embedded ruby: the response’s body will be dynamic and what you fill in the Body content field will be evaluated as embedded ruby script (See the Advanced section of the document for an example).
- Javascript: the response’s body will be dynamic and what you fill in the Body content field will be evaluated as Javascript (See the Advanced section of the document for an example).
Content type: select the content type of the response. The content type can be dynamic, I will explain later on.
Body content: fill in the response’s body (static or embedded ruby depending on your previous selection).

Headers

In this tab you can add headers to the mock’s response.

add header: click the link to add new header
Name: fill in the header’s name
Value: fill in the header’s value
x: remove the header

You can set dynamic headers, see the Advanced section of this document.

Advanced

In this tab you can set some advanced configuration for the mock.

The script field is expected to be evaluated to a JSON. The evaluated JSON’s keys can have values that will override previously (un)defined values:

headers: an array of hashes with keys:
- name: the name of the header
- value: the value of the header
status_code: the HTTP status of the response
content_type: the content type of the response
body: the response body

Besides the evaluated JSON, here is the place to add code to do stuff like simulating a delay.

Here’s an example script:

<% sleep 5 %>
<%=
  {
    headers: [
      {name: 'Dynamic-Header', value: SecureRandom.uuid},
      {name: 'Foo-Date', value: Date.today}
    ],
    status_code: @parameters[:whatever].present? ? 200 : 400,
    content_type: @parameters[:format].present? ? "application/#{@parameters[:format]}" : 'text/plain',
    body: '<h1>New response body</h1>'
  }
%>

The sleep 5 above is going to delay the response for 5 seconds. The evaluated JSON is going to add two dynamic headers and set the status code & content type of the response based on the @parameters variable values.

Embedded ruby

Both in “Body content” of the Response body tab & in “Script” of the Advanced tab, you can access the following variables:

@parameters: a hash containing the request’s parameters (including the path parameters of your route)
@request: the request itself
@response: the response

Example:

Using the following code in the Response body content field:

<div>This is a text mock</div>
<h1>Author id: <%= @parameters[:author_id] %></h1>
<h2>Post id: <%= @parameters[:post_id] %></h2>

and setting your route path to:

/authors/:author_id/posts/:post_id

your mock will be served like this:

Feel free to open issues for the tool here.

If you need some help regarding the documentation, make sure to label your issue as help wanted

iridakos - opensource articles

stup - Daily notes in the terminal

stup

Usage examples

Adding notes

Showing notes

Retrieving all notes for a period of time

Behind the scenes

More information

Find palindrome dates in the Linux command line - explained

What is a palindrome

The command

printf "now - %d days\n" {1..332044}

Part explained

Output

date -f- +%Y%\_m%\_d$'\n'%Y%m%d

Part explained

Output

tr -d ' ' > alldates.txt; rev alldates.txt >revdates.txt; paste alldates.txt revdates.txt

Part explained

Output

awk '$1==$2{print $1}'

Part explained

Output

Cat photo

References

Mongoid - Inheritance: change embedded document's type via nested attributes

Use case

Item

Book

BoardGame

Person

The problems

Failing attribute assignments

Document attribute leftovers

Wrong validators

Investigation

Desired behavior

Solution

Explanation

How to manage nested objects in Elasticsearch documents

Prerequisites

Preparation

Create the index

Add a human

Managing nested objects

Add a new nested object

Remove a nested object

Update a nested object

Update multiple attributes of nested objects fulfilling multiple conditions

Useful links

How to alias and navigate to directories with autocomplete in Linux

goto

Installation

Usage

Change to an aliased directory

Example

Register an alias

Example

Notes

Unregister an alias

Example

Notes

List aliases

Expand an alias

Example

Cleanup

Help

Version

Extras

Push before changing directories

Revert to a pushed directory

Notes

Behind the scenes

Elasticsearch bucket aggregations and faceted navigation - facets

Introduction

What is Elasticsearch

What is Kibana

What are Elasticsearch bucket aggregations

Prerequisites

date -f- `+%Y%\_m%\_d$'\n'%Y%m%d`