Simon's Blog

Let the vibes flow

Wed, 31 Dec 2025 01:00:00 GMT

I happened to have started reading for a Master’s Degree in AI just before the LLM craze started. AI was cool and there were some very impressive use cases but discourse about it was nowhere near as mainstream as it is now.

When the hype took off, it was a very crazy time to be in both industry and academia at the same time. In industry there was a mad rush to build something, anything that used the ChatGPT API. In academia, there was so much research happening that at one point we were being encouraged to not consider any papers older than three months old.

Being one of the few folks who knew how LLMs actually worked, having understanding of their inherent flaws and how they wouldn’t really ever go away, just be managed better (because bias is a part of reality), made me very fun at parties for a while. I got good at bursting people’s bubbles about how no, this is a very bad use case! didn’t make me any friends for a while.

This came to a head when there was a presentation about how a team had integrated the ChatGPT API to run sentiment analysis on one of our products. They were talking about how well it was going (without presenting any sort of objective evaluation against a test set) and how one of their struggles was with inconsistent results.

This rang very weird to my brain. Inconsistent results with an LLM is (a) a feature, not a bug, and (b) configurable! So as the presentation continued I took a look at their source code and found the following: temperature=0.5.

Fixing Google Antigravity IDE Agent crashing immediately on Proxmox

Tue, 30 Dec 2025 01:00:00 GMT

While in my day job I use Cursor, I’ve been trying out Google Antigravity for my personal projects, mainly due to the generous Gemini limits they provide via Google One (which I already pay for separately).

After trying it out on my beefy physical machine with secure mode enabled to make sure it doesn’t wipe my entire hard drive by mistake, I realised that I could trust it in a more hands-off fashion if I gave it its own playground.

I quickly setup a VM on my Proxmox node but as soon as I attempted to connect to my VM via the SSH extension I would get this error:

Antigravity server crashed unexpectedly. Please restart to fully restore AI features.

No other info unfortunately. A restart would lead to the same behaivor immediately. I tried looking into the Antigravity logs but I didn’t find anything useful.

Eventually with some googling I found a Reddit thread which, while the issue there was Mac specific, had the answer for buried deep in a comment. I’m recreating it here to hopefully make it easier for the next person to solve.

Checking a different part of Antigravity error logs revealed the following: FATAL ERROR: This binary was compiled with pclmul enabled, but this feature is not available on this processor (go/sigill-fail-fast)..

The solution on Reddit provided by user ConKernel was: If using proxmox VM, try changing host processor to "host" and reboot the VM. It worked for me

This I can confirm also worked for me and now I can use Antigravity SSHed into my Proxmox VM just fine.

Using a physical machine as if it were an on-demand Virtual Machine

Sun, 07 Dec 2025 17:00:00 GMT

Here are the specs for the best machine I have in the house:

Intel 13th Gen i3-13100
32GB of DDR4 RAM
Nvidia Tesla P40 (24GB of VRAM)

This machine lives in a 4U rack case in my server rack along with the rest of my homelab.

While my 2019 Macbook Pro technically has better specs, as soon as it starts to do any work it attempts to melt the surface it is sitting on (this was the last Intel based model made).

So I figured, using VS Code’s awesome remote development extension, I could use my Macbook as a terminal and push the hard work onto the better machine by SSHing in, especially when I want to do any work involving local LLMs.

To avoid running the machine in the rack 24/7 (especially since I pay the electricity bill), I wondered whether it would be possible to turn the machine on and off only when I wanted to use it. I had already configured it to use Wake On Lan which meant I didn’t need to phyiscally push the power button, but it still meant that I had to trigger the WOL packet from within the local network (which I eventually discovered was an issue when I was using my VPN from a remote location).

I also had left it running without shutting it down a few times, which was frustrating to discover after the fact. I wished it would behave more like an on-demand / ephemeral Virtual Machine, where I only paid for it when it was being used.

To solve this, I wrote ssh-wol-proxy. This is a tiny server written in Go that I run within my homelab. It listens for incoming SSH connections and when a connection is established, it sends a WOL packet to my machine. It then makes multiple attempts to establish a TCP connection on the target SSH port. So as long as your SSH connection timeout is longer than it takes for the machine to start and for sshd to become available, then the proxy will manage to connect and connect you to the machine.

While this was a great solution, it did not solve the part where I left the machine running. For that I configured debian-autoshutdown to check every five minutes whether there is any network traffic on port 22. If after a few checks there is no traffic, it shuts down the machine.

These two solutions together solve my issues. I now have a physical machine that behaves in an on-demand way. I only need to open VS Code and connect to the proxy using the remote development extension and within approximately 45 seconds I’m ready to work.

The code could be greatly improved, it suffers from multiple race conditions, but in a homelab setting it works just fine.

Why I have two LinkedIn accounts

Sat, 06 Dec 2025 17:00:00 GMT

If you search for my name on LinkedIn you’ll find out there are two of me.

Both of these are me. So why do I have two LinkedIn accounts?

This isn’t some elite growth hacking method, its just bad software on LinkedIn’s side.

I don’t have the LinkedIn app on my phone for reason that aren’t relevant here. I do occasionally receive a link to something on LinkedIn I would like to read.

Unfortunately, LinkedIn seems to think that because I have a google account that I want to immediately create an account using my gmail address account. It doesn’t ask whether I want to do this, it just happens. Sometimes I manage to back out of the browser before it can go through but I’m already on the fourth extra account created.

At some point I get annoyed enough to attempt to delete this account, which involes the following:

Logging in with said gmail account (no need to do anything special, I just go to LinkedIn web)
Navigate to Settings & Privacy -> Close & Delete Account
Encounter the following issue:

Go through the process of resetting my password
Go back into LinkedIn, navigate again to Settings & Privacy -> Close & Delete Account
Provide my password and specifically tick the unsubscribe option so I don’t receive marketing emails
All done, until the next time I mistakenly navigate to LinkedIn on my phone

No other service I know of pro-actively signs me up without any confirmation.

permess.mt Update 1

Tue, 22 Apr 2025 19:00:00 GMT

permess.mt is a web-app that allows you to understand Malta’s urban landscape like never before: a fast and precise way of viewing planning permits.

It is now live at https://app.permess.mt/ and I’ll be iterating constantly on it to improve the service. The data is currently updated weekly.

If you’re interested in full dataset access, get in touch!

Breaking the build while my son was being born

Fri, 11 Apr 2025 19:00:00 GMT

The Story

Where were you in February 2024? Here’s where I was:

I worked at Hotjar.
My son is about to be born and I headed out on parental leave on the 26th.
On the 28th I attend a training course because my son hadn’t arrived yet. This was my last contact with my team on Slack.
On the 29th my son is born.
On the 1st of March I logged back on to tell my team.

I saw that nearly no deployments had happened and I found a 60+ message long Slack thread of people attempting to diagnose the broken build. This was clearly serious because Hotjar was always shipping. Easily 10-20 deployments happened on a normal day and it was commonplace for our team (Developer Experience) to get complaints if for any reason the deployment pipeline was closed for more than an hour.

I scrolled to the bottom of the chat to see what the resolution was and what caused it and it turns out… it was my fault.

My fault

I joined Hotjar in May of 2021 as a Backend Engineer, shipping features to our customers, until Jan 2023 where I shifted to the Developer Experience team, where we kept our engineers shipping fast and happy.

At some point during those three years I wrote a very standard looking test that checked whether a repository method (one that interacted with Postgres) returned the expected rows.

This method in particular would filter the rows based on a date range you passed as a paramter, something like this:

def get_foo_between_dates(start_date: datetime, end_date: datetime) -> list[Foo]:
    # Code goes here

Aligning with best practices and the strong testing culture at the time, I wrote both happy path and non-happy path tests. One of them went something like this:

Setup the test: create a Foo for every date in February
Assert that 28 Foos exist
Query for all Foos between 1st and 5th February (inclusive)
Assert that I receive 5 Foos

Pretty simple test, it did its job for three years straight… until finally there were 29 Foos instead of 28 Foos, failing the first assert.

Just to make sure I never forget this, my lovely team at the time wrote this in the card that came with the flowers celebrating my son’s birth:

Worth it.

Why query the last ten when you can query every row

Wed, 02 Apr 2025 19:30:00 GMT

Here’s another SQL query I wrote recently. Lets say you want to query the minimum stock price of a given symbol out of the last ten values. Here was my first take, can you spot the bug?

SELECT MIN(price) FROM data WHERE symbol = 'APPL' ORDER BY id DESC LIMIT 10;

Breaking down my thinking:

I want the minimum price: MIN(price)
For a given symbol: WHERE symbol = 'APPL'
The latest values: ORDER BY id DESC
The last ten values: LIMIT 10

I eventually realised that this query was incorrect. It was returning the minimum price across all rows, not just the last ten.

The answer lies in how the SQL query is parsed. If I had to visualise it, I would put it this way. The query was being executed like so:

[[SELECT MIN(price) FROM data WHERE symbol = 'APPL' ORDER BY id DESC]] LIMIT 10;

In other words, it was grabbing the minimum value over all rows (which returns… a row) and then limiting that result to ten rows, which produces a single row anyway. The correct way would be to query the last ten values, THEN calculate the minimum value:

SELECT MIN(price)
FROM (
    SELECT price
    FROM data
    WHERE symbol = 'APPL'
    ORDER BY id DESC
    LIMIT 10
) AS last_k_values;

Tests would also have covered this issue :)

The JOIN that goes nowhere

Wed, 02 Apr 2025 19:00:00 GMT

Here’s an SQL query I wrote recently.

SELECT DISTINCT pu.id
FROM permit_urls pu
LEFT JOIN permit_details pd ON pu.id = pd.permit_url_id
WHERE pd.permit_type = $1 AND (pd.permit_url_id IS NULL OR pu.decided = 'f')
LIMIT $2;

permit_urls holds URLs that need to be scraped. permit_details holds data of the scraped URL. If a row is added to permit_urls and a relevant permit_details row doesn’t exist, then scraping will occur.

This query worked fine for years until I updated it a few months ago. After that it didn’t break but was consistently returning zero results until I realised my mistake. Can you see the bug?

Here’s where it lived: WHERE pd.permit_type = $1 AND (pd.permit_url_id IS NULL OR pu.decided = 'f').

While that WHERE clause is syntatically valid, the brackets around the latter part was causing the query to short-circuit on the pd.permit_type = $1 check. This is not incorrect in and of itself, but if we think a bit longer about it, pd.permit_type = $1 was always going to resolve as FALSE because the row in the permit_details pd table doesn’t exist yet… so we could never return TRUE from that part of the clause.

The adjustment happened because I wanted to split up the processing by the permit type, and I ended up completely breaking the pipeline in the process.

The obvious solution would be to write tests, but given its a side-project I didn’t see the value in it… until after this bug :)

Automatic Speech Recognition for Maltese - Part 2

Thu, 26 Sep 2024 21:30:00 GMT

This is part 2 of my effort to create an open-weights Automatic Speech Recognition model for my native language Maltese.

Before the how, let’s start with why: Put simply, the Maltese language is dying.

With a fertility rate of 1.1 and an ageing population, every concievable trend shows that the native Maltese population is slowly but surely trending towards 0.

As a result, there is a day in the future where the last Maltese speaker dies. On that day, the language will die with them. Anything that isn’t written down will become akin to an alien language

In recent years the population has actually grown due to (mainly economic) immigation. About 25% of the population is now foreign born. With this increase, the use of English (also an official language of Malta) is becoming the de-facto standard for everyday communication.

These two factors provide a consistent downward pressure on the use of the Maltese language.

So for me this is not just a personal project, but also a national one. We might all pass away, but if a computer can mathematically understand Maltese, then the language will live on for generations to come.

People have asked me whether I’m doing this as part of my Master’s degree in AI. Apart from exploring the area in an assignment, the answer is actually no. There already are researchers much better than me working on this problem, who have had better results than I have.

Academia’s aim tends to be to produce knowledge. This is super commendable and should continue! But unfortunately the fact that the MASRI datasets are not available for commercial use limits the impact of this research. Commercial entities should definitely contribute back to academia to make it sustainable, but without being able to use the result of this research, interest is very quickly lost and in the end, everyone loses and progress stalls.

If we could get a decent ASR model into everyone’s hands, we’d experience a cambrian explosion of applications and services centered around Maltese, offering an upward pressure against the decline of its use:

Transcription of old Reddifusion shows to be able to enjoy them in written form
Accelerate research by speeding up transcription of interviews for theses
Improve the quality of Maltese subtitles for YouTube videos and other media
Provide real time subtitles for live events and podcasts
Build tools around learning Maltese and its proper pronunciation
and much much more!

I’m not limited by compute, but by data. There are a few readily available transcribed audio corpuses, but copyright issues prevent me from using them. An ideal way forward would be to create a new dataset from scratch, one where every Maltese speaker is encouraged to contribute their voice, so that our machines may actually understand them in Maltese.

Automatic Speech Recognition for Maltese - Part 1

Sun, 22 Sep 2024 22:00:00 GMT

As part of my Master’s Degree in AI, one of my assignments was to fine-tune the OpenAI Whisper model for Maltese.

Maltese is a low-resource language spoken by approximately half a million people, most of whom reside on the tiny island of Malta.

I was inspired by this paper that did the same process for Javanese, another low-resource language spoken mainly by residents of the island of Java.

Using my trusty and toasty Nvidia 1080 Ti and a jupyter notebook provided by huggingface I was able to re-train the model in around 20 hours for whisper-large-v2 using 32.9 hours of Maltese transcripted audio, by combining the Mozilla Common Voice and FLEURS datasets.

Using LoRA parameters of r=32 and alpha=64, resulting in approximately 1% of original parameters being retrained, the results were very encouraging:

Model	Maltese WER % on original Whisper	Maltese WER % on fine-tuned Whisper	Diff
Base	114.66	67.88	-46.78
Small	111.18	44.67	-66.51
Large V2	89.67	32.00	-57.67

After completing my assignment, I decided to take it further and see how far I could push LoRA parameters to improve the model. From my research, the current wisdom is to put the alpha value always double the rank value (at least for LLMs). I could not find any supporting evidence for this for ASR, so I varied on that assumption too.

Here are the results tuning whisper-large-v3:

Rank	Alpha	Dropout	WER %	CER %
512	1024	0.05	38.98	11.43
1024	1024	0.05	285.1	210.98
1024	16	0.05	34.74	9.70
1024	24	0.1	32.67	8.93
1024	48	0.1	31.68	8.61
1024	48	0.2	29.59	7.86
1024	64	0.3	29.51	7.94

Increasing the trainable parameters was expected to bring an improvement (rank 1024 results in about 25% of the original model being retrained) but in the end increasing the dropout was what improved both WER and CER the most.

Testing this on data outside of the test data results in something that looks phonetically like Maltese but is not correct. My current intutition is that a lack of data is what is stopping the model from improving further, which will be addressed in part 2.

TP-Link Tapo C110 in Frigate

Wed, 10 Jul 2024 22:00:00 GMT

I recently purchased a TP-Link Tapo C110 for use with Frigate.

Despite my best efforts and a lot of searching online, I was unable to find a single contained config that worked for mse so that I could also view audio as well as video and not just msjpeg.

Here’s my simplified config to get it working. I’m running Frigate on a single host within a Docker container:

mqtt:
  enabled: False

go2rtc:
  streams:
    tp_link_tapo_test: "ffmpeg:rtsp://USERNAME:PASSWORd@IPADDRESS/stream1"

cameras:
  tp_link_tapo_test:
    enabled: True
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/tp_link_tapo_test
          input_args: preset-rtsp-restream
          roles:
            - record
    live:
      stream_name: tp_link_tapo_test

  TP-Link_tapo_c110_0_sub:
    enabled: True
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://USERNAME:PASSWORd@IPADDRESS/stream2
          input_args: preset-rtsp-restream
          roles:
            - record
    live:
      stream_name: TP-Link_tapo_c110_0_sub
    
    detect:
      enabled: False

Re-using a gaming GPU for LLaMa 2

Thu, 02 Nov 2023 22:00:00 GMT

Introduction

I’ve been experimenting with LLMs for a while now for various reasons.

Two of the most useful outcomes have been in text extraction of Maltese road names from unstructured addresses and for code generation of boilerplate scripts (the “write me a script to use GitHub’s API to rename all master branches to main” kind of script).

OpenAI

GPT3.5 Turbo is a great model and very easy to get started with, however with any sort of volume and with a large prompt, the costs begin to add up.

As a result, I began looking into self-hosting my own LLM to try and replace these tasks. Even if the response was slower at least it would be cheaper.

LLaMa

One of OpenAI’s biggest challengers is Meta with their LLaMa model. With the highly permissive licence of LLaMa 2 and the great work done in running these models locally (projects such as llama.cpp), it is now very easy to get started with self-hosting.

If however you have structured your project around OpenAI and their models, it would be much easier to somehow package these models in an API that mimics OpenAI’s API structure and responses. This makes it very easy to swap for a locally hosted model without having to change a lot of code.

Performance

One of OpenAI’s biggest benefits is the performance of the models. While some queries can take a large time to respond, in my experience most of them resolve within two seconds or less.

When self-hosting, especially if you are running on a CPU, performance leaves much to be desired. You can make improvements by using quantized models but you risk missing out on performance and cross-testing multiple quantized versions is a time-prohibitive one, especially for side projects.

Luckily, being a gamer, I have more GPUs than I need in my life and these models run much faster on GPUs.

This blog post covers the steps I went through to self-hosting LLaMa 2 on a gaming PC that I run headlessly in my home.

Steps

1: Put together the correct hardware

While this might seem like an obvious step, you should make sure the hardware you have is compatible with your aim. To keep this simple, the easiest way right now is to ensure you have an NVIDIA GPU with at least 6GB of VRAM that is CUDA compatible. This list can help.

In my case, the RTX 2060 has compute capability 7.5.

2: Install the OS

I installed Ubuntu 22.04 LTS. In general its a good idea to stick to the latest LTS image, it ends in less surprises and is more likely to have matured in terms of installation guides.

3: Install the required NVIDIA drivers/tools

I have used this guide from NVIDIA multiple times to the letter and have never regretted it.

You will also need to install the latest versions of docker, docker compose and the NVIDIA Container Runtime.

Note: This guide assumes you want to run things in a dockerised fashion. You can just as easily do it withuot docker and that’s a valid choice for many use cases. In my case, I followed this way as down the line I plan to run it in a Kubernetes cluster.

4: Confirm all is running well

After finishing installing the drivers, reboot. If the machine boots up normally, congratulations! It isn’t the first time that Ubuntu + NVIDIA has broken for me if I look at it funny.

You can also run nvidia-smi to confirm that the GPU is recognised correctly. This is the output I get from mine:

5: Download the model from huggingface

Go to https://huggingface.co and find a GGUF version of LLaMa-2-7B-Chat.

Let’s break that down:

huggingface is the premier website to find ML models
GGUF is the format used by llama.cpp which is the library we will use to run the model
LLaMa 2 and not 1 because the licencing of the first version was for research only
7B stands for the number of parameters, in this case 7 Billion. We’re opting for the smallest one to get things started
-Chat is the chat dialogue optimised model. You can get the non-chat model later to try it out and find the difference.

Here is a working link at time of writing.

6: Clone the repository for llama-cpp-python

Clone the following repository: abetlen/llama-cpp-python.

This will provide us with the API server.

7: Build the cuda_simple image

Go to docker/cuda_simple and build the Docker image using docker build -t cuda_simple .

Note: You must edit the CUDA_IMAGE argument at the top of this Dockerfile to match the version of CUDA on your machine.

8: Bring up the webserver

For my original usecase (an LLM capable of answering questions about Maltese Law), I used a docker compose file to bring up multiple services. Here is the minimal parts I needed to bring up just the LLM API:

version: '3.5'

services:
  llama-api:
    image: cuda_simple:latest
    ports:
      - 8000:8000
    cap_add:
      - SYS_RESOURCE
    environment:
      USE_MLOCK: 0
      MODEL: /models/7B/llama-2-7b-chat.Q4_0.gguf
      N_GPU_LAYERS: 35
      MAX_TOKENS: 1024
      CONTEXT_WINDOWS: 2048
    volumes:
      - /home/simon/models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

Breaking it down:

llama-2-7b-chat.Q4_0.gguf: this is the filename of the 4 bit quantized model I downloaded from huggingface. This needs to match the filename that you downloaded. A less quantized (meaning 5 bit, 6 bit, 8 bit, etc) version will take more RAM to run but may perform better.
N_GPU_LAYERS: this specifies the nubmer of layers to run on the GPU. Ideally you run all of them to get the best performance, however your VRAM is another limiting factor here.
/home/simon/models: this is the directory where I store models. The full path on the host in my case would be /home/simon/models/7B/llama-2-7b-chat.Q4_0.gguf.

To bring it up, you can do docker-compose up

If all works as expected, you should see the following logs:

Testing

You should be able to access the API docs at <host_ip_address:8000/docs

The docs will also let you try out a query using the Try it out button. Here you can see the response for "The capital of France is:" and the corresponding increase in GPU usage as the LLM processes a response.

Conclusion

The next steps for me are to try out the CodeLlama model, a model optimised to produce Python code and see if I can connect it easily with VS Code, providing a locally hosted version of something like GitHub Copilot without having to pay for the subscription cost.

Mixing datasets using symlinks

Sat, 01 Jul 2023 10:00:00 GMT

I recently had to fine-tune around 100 Computer Vision AI models for an assignment I’m completing as part of my Master’s Degree (the same assignment as this blogpost).

The models were actually the same variants with the difference that they were being trained on different mixes of datasets (specifically, the CamVid dataset, SYNTHIA dataset and the Playing for Data dataset).

In the assignment I needed to generate mixes of the CamVid dataset and another dataset at intervals of 10% of the second dataset added, for example:

CamVid + 10% SYNTHIA
CamVid + 10% PFD
CamVid + 20% SYNTHIA
…etc
CamVid + 100% SYNTHIA

Given that the SYNTHIA and PFD datasets are quite large (in the region of 10,000s of images), creating a copy of each dataset was going to be prohibitively expensive, in terms of both time and storage.

I considered overriding the way that the computer vision models I was using (specifically the YOLO family of models) loads data, however under the time constriants of the assignment this looked like a risky approach. The default way of feeding AI models, especially ones based on images, does tend heavily towards providing a directory of files and the library figures out how to use it.

Here is where symlinks became invaluable.

Symlinks or Symbolic Links are files with the sole purpose of pointing towards another file. Deleting a symbolic link does not affect the file that it links to. Creating one on Linux systems is done using ln and on Windows using mklink.

By leveraging some very simple python, I came up with two scripts which:

Read the file structures of the datasets and created text files for each dataset mixture, such as 100_synthia_20_camvid.txt, indicating the dataset mix was all of SYNTHIA and 20% of Camvid. Each line in the text file contained a file path.
Read each text file and for each line, modify the source file path to create a target file path (so if the path was /foo/bar.png in 100_synthia_20_camvid.txt it would set the target as ./datasets/100_synthia_20_camvid/foo/bar.png) and create a symlink from the source to the target. If the directories did not exist, it would create them.

Using this method I managed to create around 30 different datasets that, despite being made up of only symlinks, occupied around 6GB of disk space. This is a massive improvement over the 82.5GB of the source datasets. Had I copied each file, it would have easily resulted in the single digit terabytes of storage required and been time prohibitive to generate.

For completeness, you can find the python scripts here:

OpenCV vs YOLO Co-ordinates

Tue, 13 Jun 2023 19:00:00 GMT

I recently had to fine-tune around 100 Computer Vision AI models for an assignment I’m completing as part of my Master’s Degree.

OpenCV, to draw a bounding box on an image, expects the top left co-ordinate and the width and height of the box. My annotations however, were in the form of image colour maps, like so (image taken from the CamVid dataset):

Using some OpenCV code, I converted these segment maps to bounding boxes, generating an image like so instead (taking the cars only as an example):

All looked great up to this point.

Unfortunately I was getting pretty bad results from my AI models: we’re talking 4% mAP50. This is the mean average precision of the top 50% of classes that the model was attempting to draw bounding boxes around. Usually in detection models mAP50 isn’t expected to be very high however this result was abysmally low.

Only when I removed 10 of the 11 classes to attempt to train a detector on a single class did I realise that all of the labels were offset 50% to the left and 50% upwards.

This immediately triggered by spidey-senses: it was far too uniform to be a fluke. I eventually realised an important difference between OpenCV and the YOLO text annotation format.

To validate the bounding boxes were correctly I used OpenCV to visually confirm they were correct… after using OpenCV to generate those same bounding boxes. It turns out that OpenCV both expects and returns the following for bounding boxes:

Width and Height of the bounding box
The top left co-ordinate (x, y)

In constrast to this, the YOLO text annotation format expects:

Width and Height of the bound box
The centre co-ordinate (x, y)

The fact that the bounding boxes were all offset upwards and to the left is because to convert from centre to top left, one needs to perform the following calculation: top_left_x = centre_x - (width / 2).

Once I solved this issue and regenerated all of my annotations in the correct format, the mAP50 of my models jumped from 4% to 40%.

Lesson learned :)

The Petoi Bittle Robot Dog - Part 3

Thu, 11 May 2023 12:00:00 GMT

Introduction

This is part 3 of 3 of my review of the Petoi Bittle Robot Dog kit.

This kit was provided to me for free by Petoi in return for a review. I am thankful for the opportunity and had a blast reviewing this product.

In a first for my blog, read till the end to find a special discount code just for my readers: 7% off your entire order until May 14th 2023!

Part 3 of 3

I’m structuring these blog posts as follows:

In part 1, I focused on the kit, its materials and the build process.
In part 2, I will focus on the Python API and explore seeing how to make the dog do a trick from scratch.
In part 3 (this post), I will install the Intelligent Camera Module and see how far I can take its hardware.

Who has eyes to see

The kit I received included the MU Vision Sensor. Given my current study unit for my MSc. in AI is related to computer vision, I thought this would be a great opportunity to stretch my skills.

Following the docs, I updated the firmware of the Bittle to be able to support the camera. The only issue I had here was that I had to update the following line in the code: pathBoardVersion = "NyBoard_V1" for the firmware upload to complete.

Once the firmware was updated, I plugged the camera in and mounted it on the Bittle itself.

Previously I had noted that the Bittle is highly durable. This is still the case, however the camera module is mounted outside of the frame of the dog, using a rubber band to attach it to the plastic bone that comes with it and the bone is held in place by the Bittle’s mouth. Without the camera module I was being pretty liberal in how I treat the Bittle, however with the camera module hanging out front, mostly unprotected, I began being much more careful with it. If there was a protective enclosure I could purchase or 3D print to provide some protection I definitely would consider using one.

Turns out, there is one available, they’re just not listed in the docs. A comment in camera.h within OpenCat points to the following link which is an STL to 3D print a mount for the camera that does not require use of the plastic bone.

Upon turning the Bittle on, despite expecting it to do what it did, I got freaked out! The Bittle instantly recognised me in front of it and started tracking my movements.

Its a bit hard to appreciate the fluidity of the movements in the quality of this gif. I recommend checking out the following YouTube video demonstrating its movement - it is very accurate to my experience!

Trying to make it work

The default behaviour of tracking humans or tennis balls is amusing to see. I wanted to see how easy it would be to try and run an ML model on the image stream coming through the camera.

Here I realised that there are some limitations if you want to continue to operate at a Python level.

I was expecting being able to install a pypi package and begin to print at least some output from the camera at the console, that would then allow me to try out approaches such as running a very light ML model on the ESP8266 that adjusted the behaviour of the Bittle on the fly, or streaming the images over HTTP and controlling the Bittle remotely, allowing me to run larger ML models on my machine.

Unfortunately I was unable to find an easy way forward with this approach. In fact, a question on the forums indicates there is no way of doing it this way at present.

Having said that, it looks like it is possible with more time and effort than I could dedicate. I read through the Arduino sketch available and it is likely not too hard to change it to send these values over i2c to the micro-controller. There’s also a lesson in the Tinkergen Bittle Course covering how to use scratch to make the Bittle react to hand gestures.

Unable to proceed in a practical way, I thought it would be useful to turn this into a system design exercise, theorising the methods I could potentially leverage the camera to adjust the Bittle’s behaviour and the various trade-offs each setup would have.

Computing with one eye and four legs

Here’s the possible ways I would have approached this problem:

Running everything locally on the micro-controller
Running everything locally on a Raspberry Pi
Streaming the image data to another machine through the micro-controller/Raspberry Pi
Streaming the image data directly from the camera to another machine and controlling the Bittle from that machine
A mix of the above

Running everything locally on the micro-controller

This is the setup that I tried to implement above. It is the simplest setup possible whereby the camera is simply feeding what it receives to the controller and the controller decides its output based on that.

There are two main constraints in this kind of setup: CPU clock speed and RAM.

The micro-controller provided looks like an ESP wroom o2d which, according to the dataset, reaches a maximum clock speed of 160MHz (only 80% of which is usable) with <50kB of RAM available.

This is a world away from the kind of resources that are available on SoCs, Laptops and Servers. One might be able to fit a very simple neural network within these constraints, however the behaviour it would be able to exhibit is exceedingly limited.

Running everything locally on a Raspberry Pi

The Bittle has the option to be run from a Raspberry Pi. This would bring it closer to the Freenove Robot Dog. Once you have an ARM CPU, an SD card, Megabytes worth of memory and a standard Linux OS, then running ML models become more viable.

There is however more advanced work involved in getting this to work:

You need to solder a socket to the NyBoard
You will be unable to install the back cover of the Bittle, which makes the RPi susceptible to damage if the Bittle topples itself over
Petoi provide a file for a 3D printable Pi standoff to help keep it steady

Streaming to another machine via the onboard controller

In this scenario, we would take the output from the camera and, using the micro-controller, stream the images to another machine. The machine would then provide instructions to the Bittle over a protocol such as WebSockets

The advantages of this approach are:

We are no longer limited by the compute of the Bittle
We can run any size ML model we want on the other machine
We can potentially control multiple Bittles at once

The disadvantages of this approach are:

We need to maintain the wirless connection (whether Wi-Fi or Bluetooth) with the other machine at all times to provide the upgraded behaviour
If we lose the wireless connection then the Bittle will either freeze or continue whatever it was doing before
The latency may be too high to provide life-like reaction times

Streaming to another machine directly from the camera

It turns out that the MU Vision Sensor 3 allows for direct Wi-Fi connection since it runs an ESP32 micro-controller. It even allows for MicroPython to be used!

In this scenario, we would connect the camera and the Bittle both to the same WiFi network and have the machine ingest the images directly from the camera and then send commands to the Bittle.

The advantages of this approach are:

We may achieve better throughput in terms of reaction time, since the Bittle’s micro-controller is not switching between streaming the images and adjusting its behaviour

The disadvantages of this approach are:

Instead of one wireless connection, we are now using two. This means double the opportunity for flaky or slow connections to interrupt the control of the Bittle

A mix of the above

We can attempt to find compromises between the advantages and disadvantages of the above setups by mixing and matching.

One potential way of doing this would be to have different tiers of behaviour - a simple movement behaviour which is solely controlled by the micro-controller, but is also influenced (but not directly controller) by a separate machine performing computer vision tasks. This way, if either one or both WiFi connections are slow or lost, the Bittle’s core loop keeps running uninterrupted on the micro-controller and then re-synchronises once the connection is restored.

Conclusion

The Petoi Bittle is a great piece of hardware, truly a good product to explore programming, robotics and STEM topics, especially with older children. The price tag (~$300) is higher than other similar products but the quality is very much apparent.

The setup process might need a bit of polish if you are running from a Linux machine - it is my understanding that I could have avoided these issues if I used a Windows or Mac one.

Big thanks to Petoi for entrusting me with this review!

Interested in buying the Petoi Bittle? Go to the Bittle Robot Dog product page and then apply the code Simon7 at checkout to get 7% off your entire order! (Valid till May 14th, 2023).

The Petoi Bittle Robot Dog - Part 2

Mon, 01 May 2023 20:00:00 GMT

Introduction

This is part 2 of 3 of my review of the Petoi Bittle Robot Dog kit.

This kit was provided to me for free by Petoi in return for a review. I am thankful for the opportunity and had a blast reviewing this product.

In a first for my blog, read till the end to find a special discount code just for my readers: 7% off your entire order until May 14th 2023!

Part 2 of 3

I’m structuring these blog posts as follows:

In part 1, I focused on the kit, its materials and the build process.
In part 2 (this post), I will focus on the Python API and explore seeing how to make the dog do a trick from scratch.
In part 3, I will install the Intelligent Camera Module and see how far I can take its hardware.

The dog so far

It turns out, having a robot dog around the house raises questions.

Every person who I showed it to were very interested in seeing how the Bittle handled itself.

What really wowed observers were the tricks that come pre-programmed into the Bittle. It is highly entertaining to see a robot dog doing push ups or pretending to take a pee. The latter one was especially entertaining to my three year old daughter who would start laughing out loud every time I made the dog do the pee trick.

Clarification from Petoi: Uploading the newest firmware will reveal new pre-programmed skills. These were added in anticipation of newer modules which will be released in due time.

The most interesting of them is the “play dead” action. All the other actions keep the dog on its feet whilst “play dead” causes it to topple over. Not only that, but the action then triggers a built-in self righting mechanism which flips the robot dog back onto its feet, ready to perform the next trick.

To trigger these tricks, you can use the provided Infra Red Remote which comes with the kit itself:

Alternatively, you can use the mobile app too:

In general I found the app to be more useful since with the IR remote you need to make sure that you point it at the rear of the dog, which can become obscured relatively easy. It is also easily obscured by the tail that you can attach at the back of the dog, which I ended up removing at some point to make the IR remote more responsive.

Clarification from Petoi: The IR remote was originally intended as a debug interface. The app is recommended for more sensitive and one-to-one control of the Bittle.

Teaching a new dog new tricks

Part of why this product drew my attention was that the company advertised the availability of a Python API. Being a full time Software Engineer who primarily works in Python as well as an ex-organiser of a Python meetup, this was definitely my ideal option to interact with the Bittle and begin to control its behaviour programmatically.

In this part I will briefly explore two methods:

Using the Python SerialMaster approach
Using the ESP8266 + Python Scripts approach

Python SerialMaster

In case it is unclear, as it is not stated in the docs, these instructions require you to find the scripts contained in the OpenCat Repository.

I decided to use the Bluetooth method to avoid having to stay plugging into the Bittle. The docs only show examples for using Windows, so I had to do some extra steps on my Ubuntu machine. First I paired with the Bittle via the GUI which gave me its MAC address:

I then tried to setup rfcomm following these instructions so that I could access the bluetooth connection as a serial port. This however, took much longer that expected.

Using bluetoothctl I could see that I was definitely paired with the Bittle. By running sudo rfcomm bind 0 D8:71:4D:96:C4:3A I could finally see the rfcomm0 file, however the script was failing with a Permission Denied error. With some more googling, I figured out I needed to run sudo chmod a+rw /dev/rfcomm0.

On attempting to run the script, it seemed to always attempt to first connect to /dev/ttyS0, so I changed the script using the following line allPorts = [p for p in allPorts if p != '/dev/ttyS0']. Even then however, connecting to /dev/rfcomm0 still continued to fail.

At that point I switched back to the USB method.

It seemed that the auto-selection of the serial port was never completing, causing me to have to go through manual mode every time:

Even then, the script was throwing exceptions related to tkinter, the GUI framework used for this script:

    tk.messagebox.showwarning(title='Warning', message=txt('Manual mode'))
AttributeError: module 'tkinter' has no attribute 'messagebox'
/dev/ttyACM0
ttyACM0
2023-04-20 18:10:12,095 __main__ - INFO - Connected to serial port: /dev/ttyACM0
Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/simon/Projects/OpenCat-1.0.2/serialMaster/ardSerial.py", line 712, in selectList
    tk.messagebox.showwarning(title='Warning', message=txt('Need to manually select the model type (Nybble/Bittle)'))
AttributeError: module 'tkinter' has no attribute 'messagebox'

At this point I was pretty frustrated with the wired approach, so I decided to move onto the next option. I didn’t regret it. 👇

Clarification from Petoi: Due to the many variants of Linux, testing was mainly done on Windows/Mac. Work will be done in due time to improve the auto-connection algorithm when interacating with the Bittle via bluetooth.

ESP8266 and MicroPython

This option involved the use of the ESP8266 chip which came packaged with the robot, programming it with a version of Python capable of running on micro-controllers: MicroPython.

The first step was to download the latest firmware and update the board as outlined in the docs. This approach was much smoother as all of the steps worked on the first try.

I was reluctant to use Thonny as it would mean having to learn a separate IDE from VS Code, however in the interest of following the instructions comprehensively, I downloaded and installed it. I opted for this approach as I was focusing on using the Python method - there is also the option to use the Arduino environment if one is more familiar with that.

Within ten minutes, I had a python script which made the LED on the board blink. Running this code reminded of the very first time I ran code on an Arduino, several years earlier, also making it simply blink an LED.

Following the high of getting blink to work, I used the example script from the docs to see that the board, now running MicroPython, will be recognised by the Bittle.

Here it is running the example script:

As outlined in the docs, disconnecting and reconnecting the ESP8266 every time you want to change some code and observe the effect is very cumbersome.

The solution presented is to use the WebREPL. I was very impressed that, following the docs, I was now running my blink script over Wi-Fi!

Re-installing the ESP8266 into the robot presented an issue - it started crashing after a while and hanging, playing a beeping sound over and over. Fixing this issue is covered in this part of the docs. After applying the fix I thought I had broken something since my terminal was no longer returning any output. Eventually I realised, it was working as intended! (the fix disables the debug output which eventually crashes the board). This allowed me to run the example script file from my desktop onto the robot itself, wirelessly!

A tip while working with the WebREPL: understanding how to find out whether the board has connected to your WiFi via your router’s web UI is very useful to know whether any issues connecting to the board are because the board is still booting and connecting or in the software.

Now that I could run the WebREPL, I read through the example script to understand how to control the behaviour directly. Using the following code and this list of skills led me to success!

Conclusion

Now that I could control the Bittle programmatically from a language that I was familiar with, I was already beginning to daydream the kind of applications I could use the Bittle for. Given my current focus on AI and Machine Learning, I can see myself experimenting using ML to drive the Bittle and to begin to give it a life of its own. Interestingly, Petoi provide a Universal Robot Description Format of the Bittle, which is a useful resource to run simulations of the Bittle for the purpose of ML training.

Speaking of AI and ML, the final part of this series of blog posts will explore using the Intelligent Camera Module. I intend on seeing how good the eyes of this dog can be.

Interested in buying the Petoi Bittle? Go to the Bittle Robot Dog product page and then apply the code Simon7 at checkout to get 7% off your entire order! (Valid till May 14th, 2023).

The Petoi Bittle Robot Dog - Part 1

Sun, 16 Apr 2023 23:00:00 GMT

Introduction

This is part 1 of 3 of my review of the Petoi Bittle Robot Dog kit.

This kit was provided to me for free by Petoi in return for a review. I am thankful for the opportunity and had a blast reviewing this product.

In a first for my blog, read till the end to find a special discount code just for my readers: 7% off your entire order until May 14th 2023!

Part 1 of 3

I’m structuring these blog posts as follows:

In part 1 (this post), I will focus on the kit, its materials and the build process.
In part 2, I will focus on the Python API and explore seeing how to make the dog do a trick from scratch.
In part 3, I will install the Intelligent Camera Module and see how far I can take its hardware.

The package

I got a real kick out of seeing Simon’s Blog on the box. This blog is little more than a creative and professional outlet and this was a very physical acknowledgement that it has some sort of impact on the world, however small it is.

Inside the box there was a list of parts that it contained. At first I did not understand what some parts are (such as the USB programmer or the “cross shape servo arm”) however the docs on the website provide both pictures and videos that make it easier to follow along.

My Bittle arrived partially assembled so I was able to skip some of the build steps. You can see below how the legs and their servos were already attached together.

I underestimated the force required to put some of the screws in place. I would advise people to be careful not to apply too much pressure and instead patiently tighten the screws.

The instructions are very precise, especially around how the wires should be threaded through the body during assembly. Try not to get ahead of yourself the way I did or you will end up having to undo steps to correct your mistakes 😅.

No corners were cut in the plastic used for this product. It is sleek, sturdy and feels very nice to the touch. At one point I knocked the central body part off the table and it flew across the room. I was already dreading the fact that I would need to ask them to send me another part however the body survived the episode just fine, it didn’t even get scratched.

There were a few confusing moments in the instructions:

At the end of part 4, just as you are instructed to connect the battery, the instructions send you to complete part 5 then return to part 4. Clarification from Petoi: This has been acknowledged and will be rectified
The notes for how the legs are meant to be connected were confusing to me. The picture on the inside of the box helped in this regard.

Calibration instructions were a bit unclear. It seems like you can calibrate it without using the app, however part 6 indicates that the dog should first be calibrated, then the legs need to be installed. The way it works if you are doing it for the first time is to do an initial calibration, then install the legs, then refine that calibration.

Clarification from Petoi: There are two kinds of calibration. First you calibrate the IMU, followed by initially calibrating the joints. Then, you attach the legs and then refine the calibration using the mobile app, Arduino code or desktop app.

Also to be able to perform Calibration I first had to update the firmware by connecting it to a computer.

Here I encountered a few issues due to my setup:

I initially understood that there is a Linux version of the Desktop App, however I could not find a precompiled version of it on their GitHub link.
Buried in the docs was an explanation for the above: you can use the Desktop App on Linux, you just need to download the source code instead of a pre-compiled version and run the script from a terminal after installing a few dependencies.
There were some minor issues in the code: the code assumes that you are either on a Windows or a Mac machine, avrdudeconfPath which you are instructed to modify did not exist and there was no release file for the 1_2 version of the nyboard (the one I was provided). Thankfully knowing a bit of Python I managed to work around these issues and I applied the 1_1 version instead.

Ultimately I was unable to complete the calibration from the desktop app as the program was always picking an incorrect COM port without giving me the option to choose the one I knew was correct (the screen which allows you to update the firmware gives you a choice with a dropdown). Instead I switched to using the provided bluetooth adapter and used their mobile app instead which was a much smoother experience.

When removing the USB cable from the USB adapter, do so CAREFULLY! It resisted me pulling it off and it felt very fragile to put more force on it. Instead, remove the USB adapter from the robot dog first, then remove the USB cable from the adapter.

The dog was really coming together at this point. The last part of the build involves making it look as good as it does in the pictures on the website by covering up the board and by threading the wires of the leg servos through plastic covers that keep them out of the way. By the end, this robot dog looks amazing.

Here it is in comparison to the Freenove Robot Dog kit:

I was curious whether it would perform as good as it does in the website’s videos. The answer is - yes! After two rounds of calibration, the movement is really impressive.

The pack comes with an IR remote and pre-programmed actions which show the dog off doing actions such as pretend to pee or to wave. Here is how it handles itself on our hardwood floor:

Conclusion

This is clearly a product which has been heavily polished for a great user experience. I’ve passed on the issues I faced during the build to Petoi and I look forward to Part 2 where I will put this product through its paces by seeing how far I can push it in terms of speed and usability.

Interested in buying the Petoi Bittle? Go to the Bittle Robot Dog product page and then apply the code Simon7 at checkout to get 7% off your entire order! (Valid till May 14th, 2023).

Irresponsible Denial

Wed, 12 Apr 2023 23:00:00 GMT

The recent story about a group of University of Malta students being arrested for performing a by the book responsible disclosure over a security vulnerability in a popular mobile app used by students, FreeHour is shocking. Everyone I spoke to about it today was similarly shocked or speechless.

The Maltese police force have come across as heavy handed, trigger happy and ignorant, clearly unable to understand how this kind of situation should be addressed.

FreeHour, particularly their CEO Ciappara has revealed incompetence by running a business completely based around a mobile application that is meant to share data, yet clearly not doing so securely or understanding how there was no malicious intent involved from the students.

There are a lot of hot takes going around social media right now and many of them are right, however few are delving into the actual situation and seeing where things went wrong.

It is easy to misunderstand or overlook the nuance of the situation, so let’s start from the beginning.

Responsible Disclosure

What is responsible disclosure?

This is a standard method used within the Information Security (InfoSec) industry which lays out how a security researcher should approach an entity about a security vulnerability they discovered. It is meant to protect the interests of both parties, but most importantly the general wellbeing of the consumers involved.

Let us construct an example situation:

Jane is a security researcher. She investigates a mobile application she uses frequently and discovers that the application is leaking other people’s personal identifying information, such as their personal address, when it is not supposed to.

These kind of issues are usually the result of bugs or misconfigurations, whether by incompetence or by negligence.

Jane decides that, in the interest of the wellbeing of all the users of the mobile application, to contact the company and inform them of what she discovered, so that it can be fixed.

Working under the rules of responsible disclosure, Jane sends an email to the company informing them:

Who she is
What she discovered
How to reproduce it
The approximate time, date and IP Address that she had at the time of discovery
A timeline about when she expects this to be resolved before she proceeds with public disclosure (the standard is 90 days)

In an ideal world, the company:

Understands that Jane is not acting maliciously
Acts as soon as possible to mitigate and resolve the issue
Informs Jane that the issue has been resolved so that she can confirm that the issue has been resolved
Prepares a plan for disclosing the potential data breach to their customers (under Article 33 of GDPR)
Optionally offers compensation to Jane for her services

The idea of offering compensation is that the payout to a security researcher would always be less than the:

Cost of potential litigation in the case of a severe data breach
Material cost of loss of business or abuse of systems by malicious actors
Reputational damage suffered by the business

I state that it is optional because there is no legal requirement for a bug bounty to exist. It exists because it is more profitable to have one than to not have one.

Why set a deadline for public disclosure though?: This is a measure to avoid companies from passing the buck or outright ignoring the issue. Remember that the aim here is safeguard the wellbeing of consumers, not to pretend that everything is fine when in reality it is not.

Doesn’t a company have security experts to stop this from happening in the first place?: In my experience, most do not. Security is sometimes referred to as an “unwritten requirement”. We obviously want our systems to “be secure”, but security is not a single action or characteristic, it is an ongoing mindset and investment.

Some companies will pay for directed penetration testing (or pentests), either because they want to avoid the potential damage mentioned above or because of complaince reasons. The hard cold truth is that most companies just do not invest time and money into security.

Imagine a plumber had to come to your house and offer to ensure all your plumbing is working for 500 Euro. After you paid them 500 Euro, they come back and say, its all OK! If you do not appreciate the potential downsides of having shitty (excuse the pun) plumbing, then it is easy to feel ripped off by the plumber.

This meme sums it up pretty easily:

Ideally companies always set out and make their responsible disclosure policy clear and easy to find on their website. A great example of one is this page by the European Central Bank. It outlines very clearly:

What the security researcher is expected to do
What the security researcher can expect from the ECB
How to inform the ECB and what to include
Which vulnerabilities are considered in scope and which are not
That they do not offer monetary return for any successfully reported vulnerability

This way expectations are managed very well for everyone involved.

An alternative example is the Vulnerability Reward Program by Google/Alphabet. It includes a table stating the potential payout of successful disclosures. The more impactful the issue, the more money a security researcher can expect to earn.

Irresponsible Denial

The above example was the fairy tale ideal scenario. The reality of what happened to these students is unfortunately much worse.

The details in the article by Times of Malta are a bit obscured however from what we can glean, the vulnerability was one of those that you just can’t believe someone would allow to exist (I had found a similar kind of issue myself as detailed here). The amount of data one could extract was appalling, the kind that you sincelery hope that no one else has discovered it yet.

Quoting the article, the CEO of FreeHour claimed that “We are pleased to report that no user data was compromised and the vulnerability was addressed within 24 hours,”. As someone who has a little bit of experience in this field (compared to the clearly no experience that Ciappara has), this statement sounds blatantly false.

Quoting from the above linked ECB responsible disclosure policy, they request the following: the IP address(es) from which the security vulnerability was identified, together with the date and time of the discovery;. The reason they request this is that, without this information, you are categorically unable to tell whether the only request made for the data was made by the security researcher/s.

The article implies that the only response the students received after sending the email was when the police arrested them at home, meaning that the company did not do proper due diligence in determining whether the vulnerability had been exploited before its discovery by the students.

My only criticism of the students would have been to avoid being naive about how Malta works. Everyone knows that things get done by knowing people and by establishing personal relationships. Even in my own experience, I wrote how I was paranoid that someone at the company would not understand where I was coming from and instead I went through a trusted acquitance who could vouch for me, establishing a chain of trust first.

Sending a cold email, using the proper industry standard jargon, to someone who may or may not understand what you are saying or have someone available to them who can explain it, is dangerous. For all we know, the email came across as something along the lines of “if you don’t pay us a bug bounty, then we will tell everyone in 3 months time”. I however repeat: this does not excuse the appalling behaviour that they have been treated with.

The Law

One has to ask: if Ciappara did not know what responsible disclosure is or looks like (which honestly, I can forgive him for not knowing, because from what I understand, he is not a technical person), why didn’t the police know? How did no one in the Cyber Crime Unit, upon reviewing the email sent by the students, realise what was going on?

What is scary for these students is that they are being investigated under Article 337, which very ambiguously states it is illegal to access an application without being “duly authorised by an entitled person”..

Unfortunately this law has not aged well. In this case the system was clearly (mis-)configured to authorise everyone who asks it for the data.

This was not some fancy hack involving zero-days and social engineering whereby the students broke into a super secure system and made off with a treasure trove of data. This was the equivalent of leaving a cabinet of papers out on the street where anyone could open it and look inside.

Do not let the lack of visual element of how computers work and interact with each other make it feel like anything less. People who work in Software… we see the world differently than non-technical people. We understand what happens under the hood and we do not take things at face value, which is why these kind of vulnerabilities are able to be discovered.

The future

This incident has the risk of scaring off anyone who has even the slightest interest in InfoSec or Computing from a very rewarding career and professional life. It will discourage the proper reporting of security vulnerabilities, leaving everyone in Malta worse off.

Seeing the people doing the good work be severely and unfairly punished, while the commercial entities get away scot free is severly unnerving and damaging.

As with anything in Malta, we begin the process of learning when these major screw ups happen. Hopefully the calls for Good Samaritan laws for security researchers are heeded.

I am happy to answer questions (best reach me on LinkedIn) and provide follow up as more details of the case emerge.

Initial review of the Freenove Robot Dog Kit

Fri, 16 Dec 2022 22:00:00 GMT

Introduction

This is my initial review of the Freenove Robot Dog Kit for Raspberry Pi. I purchased this kit from my own funds, thanks to Hotjar’s Personal Development perk.

I am not affiliated with Freenove.

Computing on four legs

This particular kit caught my eye for a few reasons:

I wanted to make use of one of my older Raspberry Pi 3s which had been recently retired from my home compute cluster.
I wanted to expose my daughter to the idea of robotics within the household. We already have a robot vacuum and the robot dog could help expand her understanding of what a robot could be. We had seen Boston Dynamics’ Spot videos many times together.
I wanted to have a platform to experiment with software and AI within a physical space with compute and network constraints present (AKA ‘at the edge’) rather than a big desktop PC with an even bigger GPU and unlimited power supply.

Buying the kit

I purchased the kit from Amazon for around 150 Euros and it arrived pretty quickly.

The batteries were proving difficult to source online as this kit in particular needs high current draw batteries (the instructions include guidelines on which batteries are ideal).

I eventually realised that the batteries ideal for this kit are the same kind of batteries used in Vapes. A local Vape shop had them in stock and delivered them to me in same day for around 25 Euros.

Best consult the document first to make sure you can acquire the batteries easily.

Building the kit

The instructions were available on the kit’s GitHub Repository as a PDF. The document includes instructions for both the physical build and setting up the code on the server (the Raspberry Pi) and the client (in my case, my personal laptop running Ubuntu).

The physical build

The instructions for building the kit are very good. I only experienced two small issues:

I did not realise there was a smaller screw driver in the box and ended up coming close to stripping some of the screw heads (this was my fault)
The acrylic parts of the kit came covered in a brown paper. At first I proceeded to build the kit with the paper attached and then after realised it was just a cover that you can remove with a little force and a finger nail.

I ended up having to disassemble a few of the legs just to remove the cover followed by rebuilding them.

The process took longer than I thought it would, in part because for some parts of the process you need to take out small screws that are in separate small bags next to the servos and this is somewhat time consuming.

The final build quality comes out pretty sturdy at the end. Each of the legs will have a mess of wires passing underneath the belly of the dog which can be somewhat remedied with the cable organizing tube that comes with the box.

This organizer is not listed in the instructions. The ribbon cable for the camera hangs a bit awkwardly over the top of the robot.

I tried to encourage my daughter to do the build with me however the smaller screws and pieces were discouraging for safety reasons, apart from her inability to focus on the build for longer than five minutes at a time since it was taking a while to complete and she just wanted to see it working.

The software

The software for both the client and the server are written in Python and are also available on GitHub in the same repository as the instructions.

There were multiple steps in the instructions about how to set up the Raspberry Pi from scratch and even setup VNC which I initially skipped with the intention of interacting only over ssh.

The entire repository is pretty big (around 544MB) so it took a while to download onto the Pi. This is mainly due to the repository including pre-compiled executables for Mac and Windows and due to git itself. Both of these occupy 532MB:

 du -h Freenove_Robot_Dog_Kit_for_Raspberry_Pi
1.2M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/windows/Face
680K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/windows/Picture
72M     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/windows
1.2M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/mac/Face
680K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/mac/Picture
98M     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application/mac
169M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Application  <----
4.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/branches
8.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/logs/refs/remotes/origin
12K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/logs/refs/remotes
8.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/logs/refs/heads
24K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/logs/refs
32K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/logs
4.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/objects/info
363M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/objects/pack
363M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/objects
8.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/info
64K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/hooks
4.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/refs/tags
8.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/refs/remotes/origin
12K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/refs/remotes
8.0K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/refs/heads
28K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git/refs
363M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/.git  <----
432K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Picture
656K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Datasheet
28K     Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code/Patch
132K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code/Server
1.2M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code/Client/Face
680K    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code/Client/Picture
2.0M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code/Client
2.1M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi/Code
544M    Freenove_Robot_Dog_Kit_for_Raspberry_Pi

Getting the software to run on the Pi was somewhat straightforward and the instructions were pretty clear. Unfortunately picamera2 had just been released and it caused some of the video functionality to break. I put the kit down for a few days only to discover that the kit’s repository was updated in the meantime to support it which unblocked me.

Installing the client was not so easy. The provided instructions did not fully work and failed with incomprehensible errors about segfaults. To get it working I ended up creating an entire python environment using conda and manually installing every dependency within it. I suspect this is due to my laptop running on Ubuntu, whereas using the pre-provided Windows or Mac binaries would have been easier.

Interestingly getting the video functionality to work required me to log into the Pi using VNC instead of ssh and manually starting up the server every time. It does not work if I do the same process via ssh.

Another issue is that the dog is setup to connect to a WiFi network. This introduces both latency and the fact that if I had to take the robot with me elsewhere then I would not be able to connect to it without plugging the Pi into a screen and keyboard and setting up a new WiFi connection.

The above issues were a particular pain whenever I wanted to show the robot to anyone. I would have to get the robot, turn it on, wait for it to finish booting, connect with VNC, start up the server, switch to my laptop, activate the correct conda environment, start the client and finally connect.

All of the functionality worked as expected except for the following:

The kit advertises that it has ‘red ball following’ functionality and actually comes with a red ball in the box, however regardless of what I tried, the dog never reacted to the red ball infront of it.
The kit advertises some facial recognition functionality which also did not work out of the box.

Latency was an issue overall in some instance, especially when the robot moved into certain pockets of theh house where I knew the WiFi signal falters.

Final Thoughts

The kit is great and can get to a working state with only minor tinkering. It is nowhere close to being anything like Boston Dynamic’s Spot in agility and polish but it definitely makes you feel like you have a tiny version of one.

If you are looking for a robot kit that younger children could enjoy and easy to use, this is definitely not the ideal purchase.

If you want a platform open to experimenting and learning with, this is definitely a great option. I highly recommend being at least somewhat experienced in running python scripts and knowing how to install dependencies before considering it.

The future

I think this kit has several avenues with which it could be improved upon:

The observability. The client does not store any data historically (such as the video stream, battery level, gyroscope values or state of the servos). It would be great to have this data exposed via something such as Prometheus or MQTT to make it easier to visualise.
The time to getting started. In my opinion, getting started should be as easy as running something along the lines of docker pull robotdog and docker run robotdog. Some projects such as flightradar24 offer a Raspberry Pi OS image which you can install directly and get started with much faster than installing each part yourself separately.
The ease of use. An alternative to having a QT based client would be to have the Pi host a web based client and the client connects to a WiFi network provided by the Pi. Whilst this approach has its downsides, this would greatly reduce the latency between issuing commands and seeing the result as well as making the robot controllable from any device capable of a WiFi connection and running a conventional web browser.
The ease of extensibility. Currently extending the functionality means having to edit the code and restarting the server. An alternative model could be one of plugins, such as being able to run an image classifier on the video stream and have the results available via REST or MQTT.
An autonomous mode. Right now it only operates in a manually controlled mode. An autonomous mode where it walks and looks around and exhibits some basic behaviour would be a great alternative.

I am planning on using this kit as a test bed for experimenting what I am learning as I read for a Masters in AI from University of Malta so I will likely tackle a number of the above issues as I go along.

Here’s to hoping we don’t end up with this…

Thanks for reading!

So you want to move some data from A to B in AWS

Wed, 14 Dec 2022 22:00:00 GMT

At my day job I’ve been recently thrown head-first into designing, building and improving an AWS setup.

One of our requirements was to have a script in ECS run against some JSON files when these files are either uploaded or updated within S3. Simple enough.

Given AWS has thousands of ways of doing things, I safely assumed there’s probably an easy way to expose S3 to ECS as some sort of filesystem. Turns out there is, but only if you are running ECS on EC2 and not Fargate. Coming from a heavy Docker and Kubernetes background, I figured there must be some way to expose a “volume” to ECS and indeed I found AWS EFS.

AWS EFS which is block (not Object) storage, simply put allows you to connect a filesystem to an ECS task.

Once I figured this out, the next issue was: how do I get my data from S3 to EFS? I thought this would be a good use case for Lambda and in the course of my reserach I discovered AWS DataSync.

DataSync offers a managed way to sync data within AWS. It offers multiple sources and destinations and can even do syncing data in and out of AWS. Both S3 and EFS are supported so this looked like the perfect solution for our problem.

After multiple false starts until I configured it properly, DataSync was working. The next issue I had to deal with: how do I trigger DataSync when the S3 upload happens?

My research led me to S3 Bucket Notifications - you can configure an S3 bucket to notify a Lambda when its contents change. This resulted in the following ~~stack of dominos~~ setup:

S3 Bucket -> S3 Bucket Notifiction -> Lambda -> Lambda uses AWS API to trigger DataSync and wait for completion -> DataSync Task to copy from S3 to EFS -> Lambda uses AWS API to trigger ECS Task -> ECS Task.

This setup worked… but not too well. It had a number of disadvantages:

The S3 bucket notification fires for every file changed and the user was uploading all of them, even if they had not changed.

This was triggering several Lambdas. I resolved this by limiting the labmda concurrency to 1 and I did some reserach on how to buffer and consume bucket notifications through something such as SQS.

The Lambda spent a LONG time waiting.

DataSync tasks, even for tiny amounts of data syncing (we were moving around 50KB), take on the order of several minutes to start. This meant that the lambda had to wait and keep checking via the API whether the DataSync task had completed yet or not, which was all time we were paying for. There was also no easy way to surface this status to our end user, who had no idea whether the process had finished or even started and would likely retry their upload, excacerbating the problem.

Despite being the perfect solution on paper, it felt like I was increasingly trying to convince myself that there is no better solution than DataSync.

Numeric Scripts

Wed, 14 Dec 2022 22:00:00 GMT

I’m often starting new small projects, especially ones that involve web scraping. This habit of gathering one off odd datasets has come in handy a few times when I want to back an opinion up by fact.

Since I only check in on these projects rarely, I typically forget the end to end flow of how to run each step (such as scrape -> parse -> scrape -> parse -> clean up data), which I tend to split into separate scripts.

I’ve started using a “numeric script” approach. scrape_links.py, parselinksfor_urls.py, check_links.py, and extract_links.py on their own have an ambiguous flow, while the following makes it much easier for me to remember:

01scrapelinks.py
02checklinks.py
03extractlinks.py
04parselinksforurls.py

I could of course write a quick README stating the order, but this approach is ✨ self-documenting ✨.

How to talk to the public internet from AWS ECS

Wed, 14 Dec 2022 22:00:00 GMT

At $WORK I’ve been recently thrown head-first into designing, building and improving an AWS setup and this reality with ECS networking really tripped me up.

To get going quickly, we setup Fargate with a Public IP address. That allowed us to verify that our containerised backend was being built, pushed and run correctly in ECS and that we were able to CURL it over the internet - success!

We knew this wasn’t the end goal, so I got to work setting up an ALB, having

Lessons learned about Web Scraping

Wed, 14 Dec 2022 22:00:00 GMT

Introduction

I got my start in software back in 2015. Right out of Codeacademy’s Python guide, I got interested in web scraping and the rest is history.

This blog post is a collection of the lessons learned through the years and multiple projects in this space.

Split your scraping and parsing

Initially my web scrapers would attempt to parse the data (such as extract valuable information from HTML) as soon as the scraping was completed. This approach inevitably led to inefficient scraping as well as the potential for bugs unrelated to scraping causing the scraper to break.

Nowadays I split up the scraping and parsing processes. I decouple them by having the scraper either save the raw data to a database or to files, with the parser being a separate process doing reading of that data and extracting the valuable information.

To keep things simple, these different modes typically live in the same script, with a CLI argument used to run it in scraping mode or in parsing mode. This also allows for easy code re-use between both modes.

Keep your full source content

In a number of projects I have experienced a need to go back and get some new information from a source that I was already scraping. In each case, I ended up having to begin scraping again from scratch.

An alternative to this is to keep all the source content. Whether you do it by storing them as files, or saving to a database, it doesn’t matter as long as it is easy for you to re-inspect it. This approach also allows you to iterate on your parsing without making repeated calls to the same pages over and over.

If storage space is an issue, either because you are retrieving large files (such as wav files) or you expect to scrape large volumes of data, leverage compression techniques such as brotli to minimize the impact.

Respect your scraping targets

In the beginning I would scrape rather liberally. Computers are fast! was my reasoning. Eventually I had one too many run-ins with my IP being flagged or blocked by my scraping targets (either automatically with a tool such as Cloudflare or even manually in some cases).

Whilst there are workarounds (such as rotating your IP Address using a proxy), I have found that scraping respectfully is a better solution.

By implementing large delays between your scrapes, avoiding peak traffic time and lowering my need to have all the data NOW, I’ve largely eliminated these kind of issues.

If you can, scrape asynchronously

If your scrape target has liberal limits, consider scraping asynchronously. What this means is using a distributed task runner (such as Celery for Python or Asnyq for Go). Running multiple workers will reduce the impact of the fact that web scraping tends to be an I/O bound problem.

Handle exceptions and timeouts

Often enough in my first few projects I would leave a scraper running overnight only to discover that at 2am it died of an unhandled exception (such as a TLS handshake failure) or got stuck waiting for a response because requests.get() doesn’t have a default timeout.

Handle exceptions liberally and set sane default timeouts to avoid falling in this pit. You can also work around it by having your process automatically restart if it fails, however your script could get into a loop where it is failing on a specific page and will keep failing over and over. In this case I prefer to have the script skip that page and move on.

Store all your data in one place

This is a more recent decision, inspired by the idea of doing things that don’t scale. There’s likely an academic answer to the question: “which format/database is ideal to store scraped HTML?” however the more realist answer I now prefer is “postgres” regardless of the domain.

This for me is a mix of simplicity (all your data, both raw and parsed is in one place) and a lack of time to dedicate more than is absolutely necessary to side projects. Scaling issues are a good issue to have, it likely means your project is successful by some metric.

That time I discovered a full account takeover vulnerability

Fri, 04 Nov 2022 22:00:00 GMT

Important note

Similar to the accidental pentest, this story happened over two years ago and the details have been fudged to protect both the innocent and the guilty.

Introduction

One of our clients had a successful website and were looking to expand into the mobile space. The task to build this mobile application was assigned to me.

The supplier which built and operated their website for them was also going to provide the backend for us to wire up the App to.

One of the first items I requested was any sort of API documentation they had and a staging environment so that I could test the integration without worrying about affecting their production instance.

The project

The first issue arose when they came back to us stating that they were having difficulty setting up a staging server for us and that we should use the production environment for testing.

This immediately rubbed me the wrong way. Did they not even have a staging environment for their own testing? Turns out, no, they did not.

I did my best to not allow this issue to get to me: I could understand that, on occasion, corners are cut, especially when resources are scarce (whether its time, financial or competence). I decided to, with sufficient CYA, proceed to use their production API whilst exercising caution.

The API documentation (provided in a Word document) was mostly sufficient for us to proceed with a good part of the work, which was a counter-balancing relief to any issues so far.

Working in production

We started to work our way through creating the screens that our designer had given us. We prioritised the work where the API was already available and using mocks for where it was not.

Overall the project proceeded quite smoothly from there.

Testing in production

At one point I wanted to test how the user profile updates itself by changing something on my own profile on the production environment.

I navigated to the user profile page, which had a URL similar to this:

www.example.com/user/profile

…and decided to change my name whilst watching the network tab to observe the behaviour.

Funnily enough, whilst other parts of the website performed asynchronous requests using JavaScript, this one did not. It did not even perform a POST request from a form. This implementation smell triggered the curiosity snowball that eventually formed this entire story.

The smell

My profile had clearly been updated as I could see the new value, both on the website and on the API. So how did the website manage to update it?

Then I realised that the URL had changed: it was no longer www.example.com/user/profile but instead it had become something similar to the following:

www.example.com/user/profile?email=example@example.com&name=Simon&surname=Test%20Surname&gender=M.

This was stinky and smelly. Not only did the website perform an update using a GET request, but as a result of doing so, it had to pass all the values as query parameters. To confirm my suspicion, I changed the surname query parameter value and retried the URL, succeeding in changing the saved value.

Admittedly, while smelly, meaning it is the kind of code that I would have flagged as an issue in a code review, it does the job it is meant to do. This blog post however is not about smelly code, but rather about…

The vulnerability

Between the lack of staging server, the API docs in a Word document and the GET request performing modifications, it was becoming clear to me that corners had been cut on this backend.

From experience I also know that when corners are being cut, security tends to be one of the first that are cut.

So the thought passed through my head: given what I know so far about the supplier, what are the chances that they are ignoring extra keys on this GET request that modifies the User Profile?

It was super easy to test out. I simply typed out the following URL into my browser:

www.example.com/user/profile?email=my-colleague@example.com&password=foobar123

I logged out, then attempted to log into my-colleage@example.com with password foobar123 and sure enough… I got in.

Raising the issue

I had successfully reproduced a full account takeover. I also knew that this was a non-trivally popular website in Malta, with multiple thousands of registered users, each with a certain set of PII and potentially sensitive data in their accounts.

I could not keep this to myself. I raised it through my then manager who raised it with the client who took it up with the supplier. The issue was quickly patched, faster than I had seen any other issue solved in this project so far.

In Conclusion

The client shared with us that they had been having issues with the supplier for a while and wished to switch away, however their hands were tied for now.

I got what amounted to a pat on the back and acquired the knowledge about how to avoid creating this vulnerability in the first place.

This was yet another simple issue that could have been easily solved long before the code hit production with a simple automated test.

Never go hungry again

Mon, 31 Oct 2022 12:20:00 GMT

I was recently stuck in traffic (which is a rare occurrence given I work remotely) so I took the opportunity to observe my surroundings.

The bus stuck in front of me had an advert for one of the food delivery companies here in Malta, with the slogan “Never go hungry again”.

This slogan rubbed me the wrong way and I thought it worthwhile to share why after musing on it for a while.

Here are the issues that I find with this slogan:

It is logically paradoxical

A food delivery company is not offering you food security or even food: it is a highly efficient middleman using supply, demand and technology to get goods from point A to point B. Convenient food just happens to be a good in demand right now.

If all restaurants had to fold, food delivery companies would fold too and the age of convenient food would end.

It is economically paradoxical

Given how obscenely more expensive restaurant food is compared to home cooking (a fact that restaurant owners would hope you conventiently ignore, unsurprising given their vested interests), the idea that a food delivery middleman can assure food security is economically paradoxical.

You’re paying more for convenience today over retaining more funds to ensure you can purchase food tomorrow.

This is re-inforced by the fact that several restaurants were discovered with inflated prices for listings on delivery apps, supposedly to compensate for the commission charged by the delivery company.

It is socially tone deaf

In a time of explosive inflation and rising cost of living, having a slogan referring to food security, even indirectly, is tone deaf, even somewhat insensitive. An increasing number of people in Malta are being classified as at risk of poverty.

Conclusion

My aim is not to take a moral high ground, I make use these services myself, especially when the alternative is much less efficient.

My aim instead is to wonder whether a more humanistic approach could have been taken, one which can exist within the current socio-economic context without coming close to mocking it.

Remote done Well Part 1

Tue, 27 Sep 2022 20:20:00 GMT

Introduction

After I had spent several months working remotely for my previous employer due to pandemic restrictions I realised that:

I wanted to work remotely every day for the forseeable future
I wanted to do so at a company for which remote was a feature not a bug

I have now worked at Hotjar for 1.5 years now. It has been operating fully remotely since it was founded in 2014 and as a result has very strong processes which allow it to continue to operate efficiently, especially as it grows in terms of headcount.

I want to document the more hidden and nuanced parts of working at a company where remote is done well, the kind that you aren’t really exposed to in a job vacancy (we’re hiring by the way).

Meetings are often recorded

You need to miss the weekly all hands because you need to pick up your aunt from the airport? That’s fine, you can watch the recording later.

Not every meeting is recorded. Whilst meetings with large audiences or workshops tend to be recorded, team stand-ups, syncs and refinements typically are not.

This allows for an incredible amount of flexibility with your schedule, especially useful since employees are spread across multiple timezones.

Strong documentation culture

Over my tenure I’ve reversed my sequence of acquiring knowledge. It used to be:

Ask a colleague
Read the code
Check documentation (if it exists)

Now it is:

Check documentation
Read the code
Ask a colleague

Given that colleagues are spread around the world and may be still asleep, just waking up, ramping up focus after their first hot beverage or eating lunch at any time, relying on synchronous communication all the time is highly counter-productive.

As a result, it is more productive to see if it’s been written down somewhere before asking someone about it. Even when I end up asking a colleague, I have a 50/50 chance that their answer will include a link to a document that my confluence-foo was unable to surface.

Occasional, intentional social calls

One of my colleagues once asked me: “do you really sit on a call watching each other eat food at their desk? Isn’t it just awkward munching?“.

It is only awkward if you make it awkward.

Over the past 1.5 years our team has perfected the fortnightly social call:

Order some food (covered by our Work Together perk)
Talk about anything but work (pets, life, achievements, drugs, etc) for the first half hour or so
spend the next 45 minutes playing games together. Scattergories, Skribbl, Gartic Phone and Geoguessr are all awesome options.

Every 2-3 weeks run an optional, no-agenda social call for the last 30 minutes of the day on Friday. Right before the weekend starts there is not anything you can accomplish that cannot wait until Monday morning.

A remote environment does not allow easily for natural conversations and social relationship building. It does not mean you cannot have social relationships with your colleagues, it just means that you need to be intentional about it.

Core hours

There are three hours in the day where there’s an expectation that you can be reached if you are working that day: 14:00 - 17:00 CET.

Other than that, there are no particular restrictions about when you should be ‘butt in seat’.

Teams tend to converge and settle on a steady state of meeting & ceremony times that work for all their members, such as starting a daily sync at 11:00 CET to allow for people who wake up later as well as people in earlier timezones to join comfortably.

Salary Equity

A common criticism of remote work is that it allows work to be farmed out to countries with lower cost of living, resulting in cost saving due to workers more often than not being temporary contractors and accepting of a lower salary.

At Hotjar, the gross salaries are the same for the sheer majority of roles. There’s no reason that someone living in Nigeria should be paid less than someone living in London for the same work. Pay and perks are also the exact same regardless of whether you are an employee or a contractor.

Whether this is a fair approach due to difference in taxation is another debate entirely, however it is definitely more equitable.

This also leads to…

Diversity

I never realised how mono-cultural my previous roles were before I worked at Hotjar. Often enough I find myself on a call with people spanning multiple continents. The focus on equity contributes heavily towards a beautifully diverse team.

etc

I’ll definitely write another post as I continue to muse more about these points.

Did I mention we’re hiring? Come work with me!

Remote done Well Part 2

Tue, 27 Sep 2022 20:20:00 GMT

Introduction

After a good amount of interest and feedback on my first article about Remote done Well, I’ve determined another set of good practices I have observed at Hotjar where I have worked for the past 1.5 years that allow it to succeed as a fully remote company.

Slack Statuses

TODO

Standard Headsets

TODO

Donut Calls

TODO

Helpdesks

TODO

Radio Discipline

TODO

That time I accidentally did a pentest

Fri, 26 Aug 2022 17:00:00 GMT

Important note

This story happened over two years ago and the details have been fudged. I offered the company involved an opportunity to review this blog post before publishing and they approved it without requesting any changes.

Introduction

My wife benefits from a partial refund on some services as a perk provided by her workplace. At the time we were renovating a house so every Euro we could claw back and put into the renovation meant getting one step closer to being able to move in.

To benefit from this refund, we had to provide the receipts of the year before the previous. Meaning that even though we would apply for the refund in (for example) 2016, I would need the receipts of 2014.

The Bills

I logged in to a specific company’s website since the account was in my name and I clicked through to the billing page… only to find the last few bills only available. 🤔. I had opened and been paying my account for longer than that for sure and this would have left me with a few months of bills missing.

I hopped onto their support chat asking for a way to access all of my bills. To authenticate I had to provide my full name, my address and my account number (this is relevant later), only to be told that the system is limited to keeping the last few months of bills only and no more.

This instantly smelled funny to me. As someone who writes software I couldn’t fathom any system let alone a billing system retaining only a few months of data. This data definitely would have been retained, if not for internal purposes then most likely for auditing, reporting and legal reasons.

Frustrated at the prospect of missing out on the full refund, I did what any self respecting programmer does: I pressed F12.

The Network Tab

I started clicking through the billing section of the website again. It was an SPA that was using REST calls to contact the company’s backend, with the responses being encoded in JSON. Simple, solid stuff.

When loading the list of bills the response looked something like this:

{
    "result": [
        {"id": 1000, "month": "Feb", "year": 2022},
        {"id": 5000, "month": "Mar", "year": 2022},
        {"id": 9500, "month": "Apr", "year": 2022},
        ... etc
    ]
}

Very typical response for a list. The most curious part of it however was the id. My bill’s ids were very different from each other, in irregular intervals. This immediately indicated to me that each bill has a globally unique identifier. This is expected… and the curious part is how the id is:

A monotonic increasing integer (meaning 1, 2, 3, etc)
Globally unique across all users (meaning no two users will ever have the same bill id)
Exposed to the Frontend

This meant that I could, in theory, guess all the bill IDs, including the ones that were older than a few months old.

The Bill

To further reinforce my theory that the bill ID was globally unique, I observed that downloading the bill made an API call to an endpoint similar to this: /api/bill/12345/download. The value 12345 always correlated with the id values returned in the list API call. Calling this API endpoint would return data that the browser was turning into a PDF file of the bill.

The Search

Armed with this knowledge, I realised that the best course of action would be:

Determine the average difference between each bill ID
Use that to generate the potential bill IDs that were mine that are older than a few months old
To narrow down the search, attempt to download each bill ID +- 100. It is expected that each billing cycle there will be less bills or more bills.
Use the HTTP Status Code to know whether it is my bill or not
Download all the bills that return an OK status code
Success!

Penny Drop

To find out what status codes would be returned, I attempted to download a bill with a single digit difference (if my bill ID was 123 I attempted to download 124).

It downloaded just fine.

Wait what?

I opened the PDF file quickly and there it was: someone else’s bill.

I had to confirm if it was a fluke.

I changed the ID by one digit again and tried again.

It downloaded just fine.

Awww shit.

The Vulnerability

So it turns out that the API to download a bill did not check whether I owned the account being billed.

You would not think that this is a big deal (especially if you ignore GDPR)… other than the fact that each bill contained the person’s:

Full Name
Address
Account Number

These are the same three data points I had used to authenticate with their support staff. Retrieving a bill could, in theory, escalate to a total account takeover.

The Fix

I remember feeling scared at the time. I had had a negative impression about the attitude of Maltese companies to Information Security for a while, especially after a huge data leak of voter records where the IT professional in charge it was reported to “thought the report was an April Fool’s Joke”.

I needed to somehow contact someone at the company who would trust I was not malicious and action it as soon as possible. After reaching out to a trusted acquaintance of mine, they linked me up with a Senior Manager at the company. They (with my permission) called me on my phone soon after and told me that both an executive and their security team had been informed and asked me for a high level overview before the security team called me for more details.

I was kind of embarrassed to reveal that the entire thing boiled down to, in my opinion, missing validation, rather than some fancy zero day that gave me root access to all their traffic.

After hanging up, the security team called me and we went over reproduction steps together. They thanked me and told me they would be in touch.

The Aftermath

An hour or so later I got an email from them stating that, due to it being a weekend, they went ahead and disabled the download bill feature and would follow up on Monday. Monday evening they communicated again that they had successfully patched the vulnerability. I tried one more time to download a bill that wasn’t mine and got an HTTP Forbidden status code in return.

I eventually spoke again to the senior manager a few days later. I suggested how they could display an email address or some other method for security related escalation more prominently on their website for these kind of issues

I ended up being invited to apply for one of their open positions. The interview went well (and they waived their technical test for me, which was great) however I ultimately did not join the company due to other factors in my life at the time.

In Conclusion

Sometimes even the simplest of issues could escalate into a potentially disastrous data breach. This kind of issue is trivial to track with an automated test.

Also, never get between a motivated software engineer and what is rightfully their data, unless you want them to poke holes in your software 😉

Four day work weeks

Tue, 22 Feb 2022 12:00:00 GMT

For a while in a previous role I worked a 32 work week. I afforded it at the time and preferred to spend a bit more time at home with my newly minted family.

In this blog post I’ll talk a bit about four day weeks, what I managed to achieve, what the benefits are and how I replicated them despite switching back to a 40 hour work week in my current role.

The many four day work weeks

First of all, there are a good number of variants of a four day work week and few people clarify which they are referring to, especially in online conversations. They are:

40 hour weeks in four days, AKA 4x10. This means longer work days, but shorter weeks.
32 hour weeks in four days, AKA 4x8. This means an entire day is no longer dedicated to work. You could start your weekend early, start your week later or have a break somewhere in between.
30 hour weeks in four days, AKA 4x7.5. This is similar to 4x8, with 30 minutes less each day.
32 hour weeks in five days, AKA 5x6.4. This is the model I used, where you’re still working every day, however you leave earlier (or arrive later) than your colleagues.

Each have their pros and cons and are ultimately up to personal preference. While I still worked five days, my day was from 08:00 to 15:00 with a break of an hour. If I wanted I could have worked four 8 hour days, however I felt that working the entire week kept my momentum going for the whole week rather than having a hard stop half way. I do consider 4x10 to be a non-starter, since 8 hour days are already a stretch on the brain, let alone 10 hour days.

The benefits

For me, there were four main benefits:

My wife and I now started and ended at similar times, which meant she could drop me off on her way to work and pick me up on the way back. This reduced our fuel costs drastically and definitely reduced our carbon footprint.
For the times when I drove to work, driving back at 15:00 meant I would get home in 10-15 minutes. The same commute at 17:00 would easily take 40-50 minutes in Malta’s atrocious traffic. That was at least 30 minutes of time saved simply by shifting the time I left the office.
With a shorter day, I cut out a lot of the times I would let my mind wander and a good chunk of time wasting. I spent less time on reddit and more time focused on my tasks, because I knew it would simply be a few more hours before I would be home.
I ended the day with a fresher mind, capable of handling other intellectually stimulating tasks and side projects better.

The downsides

While I did mostly experience upsides, there were a few downsides. The one that comes to mind mostly is that people forget and get surprised that I end the day earlier. I think this created pockets of aminosity from my colleagues at times which I regret causing.

Another one was that I had an agreement with my employer that I could work up to 40 hours in a given week and I would submit a time-sheet before the end of the month in the case that I did. This worked out well for me as I could make up that last 20% of the salary cut I took to work 32 hours if I wanted to, however it essentially turned me into an hourly employee, which meant that no two payslips ever matched fully.

The amount of income tax I was meant to be paying kept varying and it made conversations with a bank (I was looking into taking a loan) annoying complicated.

How I replicated the gains without the 32 hours

I did mention that this was for a previous role. In fact in my current role at Hotjar, I work a full 40 hour work week.

I do miss the slightly extra free time that a 32 hour work week provided however I managed to replicate the gains I had by switching to a completely different paradigm: fully remote work.

Shaving 30 minutes off a 40 minute commute is awesome. Shaving 39.5 minutes off a 40 minute commute is even better. I now have no commute and am completely immune to traffic or parking issues (two very real issues in Malta!).

I am also around my wife more often since she was on maternity leave for the past two years. I sometimes take the opportunity to do some mindless chores while thinking about a difficult problem I’m trying to solve, saving more time for after the day is over. I also have the advantage that software is both my work and my hobby, so I get a good fill of enjoyment at work.

Would I go back to a 32 hour work week while fully remote? I could probably afford it, but in the meantime our combined focus has become more centered around paying off our debts and investing in our future rather than micro-optimising the amount of free time we have now. On the other hand, I could definitely see it working great for people for whom the job is just a job and their passions lie elsewhere.

Scaling Battlesnake testing with Kubernetes

Sun, 14 Nov 2021 20:00:00 GMT

Note: this article assumes familiarity with kubernetes related keywords.

A friend and I are using Battlesnake as a platform for learning new tech skills. In my case, it is to continue to improve my DevOps technical skillset, mainly to finally take the plunge and learn Kubernetes.

Battlesnake is a multiplayer version of the classic game snake, where game engine sends HTTP requests to your web server. Your response simply needs to be up, down, left or right within a deadline of 500ms. How you achieve it and with which technologies are decisions all up to you.

As you iterate on your snake it becomes essential to test it frequently, both to avoid regressions such as causing your snake to prefer to headbutt snakes stronger than it, and to test whether your changes have made it a stronger contender or not.

Initially we achieved this by pushing the snake to our production EC2 instance hosted in AWS us-west-2 (as close as possible to the Battlesnake servers, since every millisecond of latency counts towards the 500ms deadline), however this process was slow, and made even slower by our move to graviton instances.

The Battlesnake team provides a command line version of their rules engine which allows for running of local games. This means that we could run our snake testing locally, ensuring our snake performs the basics (do not hit walls, do not hit yourself, do not run out of health, etc), along with seeing how it performs in a game against another snake (or potentially even against itself).

This is still limiting when it comes to testing two snake variants. You can only easily run the CLI sequentially. If your snakes are also running on the same computer, then they will compete for the same resources (CPU, RAM) that the CLI is using.

Ideally we should be able to run multiple instances of the CLI in parallel, along with multiple, load balanced instances of each snake variant. Here is where kubernetes comes in as a solution. Whilst this can be achived without kubernetes (there are options such as Hashicorp Nomad), k8s offers multiple features we need in a single package whilst also providing a great opportunity for learning.

A kubernetes based approach

Kubernetes provides an opportunity to orchestrate our software (packaged as containers) across a single or multiple nodes. It has existing concepts which aid in achieving outcomes such as service discovery, load balancing and parallelisation of jobs.

Here is what our setup looks like. It runs k3s on a Raspberry Pi based cluster:

Each snake is containerised and deployed to a Docker registry hosted alongside the cluster.

For each snake, we define a deployment. This maintains a replicaset underneath which creates a set of pods running the snake’s webserver. We also define a service which allows each web server’s pod to be reachable within the cluster. To allow reaching it from outside the cluster, we define it as type NodePort and hardcode a port such as 30000, in case we wish to curl it manually.

Here’s a sample:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foobar1
  namespace: snakepit
  labels:
    app: foobar1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: foobar1
  template:
    metadata:
      labels:
        app: foobar1
    spec:
      containers:
        - name: foobar1
          image: "our-registry-url:5000/snake-foobar1:testing-arm"
          ports:
            - containerPort: 8111
          imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: foobar1
  name: foobar1-service
  namespace: snakepit
spec:
  selector:
    app: foobar1
  type: NodePort
  ports:
    - protocol: TCP
      port: 8111
      targetPort: 8111
      nodePort: 30000

The CLI is also containerised. We define a k8s Job where the command is specified to include all CLI args required to include the snakes also deployed within the cluster. We can even define these using the name of the service, since k8s will resolve DNS for us within the cluster (for example: http://foobar1-service:8111). Here’s a sample:

apiVersion: batch/v1
kind: Job
metadata:
  name: bscli
  namespace: snakepit
spec:
  completions: 4
  completionMode: Indexed
  parallelism: 2
  template:
    spec:
      containers:
        - name: bscli
          image: "our-registry-url:5000/bscli:testing-arm"
          imagePullPolicy: IfNotPresent
          command:
            [
              "./battlesnake",
              "play",
              "-W",
              "11",
              "-H",
              "11",
              "--name",
              "foobar1",
              "--url",
              "http://foobar1-service:8111",
              "--name",
              "pastaz2",
              "--url",
              "http://pastaz2-service:8111",
              "-v",
              "-t",
              "500",
            ]
      restartPolicy: Never
  backoffLimit: 1

By adjusting completions and parallelism we can decide how large our testing round should be, along with how many games should run at the same time. In an ideal world: parallelism == completions, however my cluster has limited resources and by running too many pods the end result will slow down the response time of the snake webservers, ending up in a situation where it takes longer to complete a round of testing. By constraining parallelism, we can ensure that we do not saturate the cluster’s resources.

Future work

Our work is not finished yet. Right now, this system does not track the number of wins per snake in a testing round, which is the entire point of testing variants against each other. Here’s what we would need to achieve:

To achieve this, there are number of options:

Tail and grep the logs of the CLI pods (the last line printed by the CLI states the winner).

This option is quick and dirty, however has some constraints, mainly that we would be doing it after the testing round is complete. If we had to interleave the logs from multiple pods from different testing rounds we would not be able to tell which result belongs to which testing round.

Patch the CLI to write the results to a file and upload it to a webserver by including a script of some sort which will run after the CLI exits.

For this, I forked and patched the CLI to export the details of the game to a line delimited JSON file. I also opened a PR which has started the RFC process around this functionality.

The main issue here is that I would have to maintain a fork of the CLI and would I need to run the script after the game ends, potentially complicating how the container is built and how the current system of overriding the container’s command is done.

Divert the CLI Pods’s logs to another Pod by streaming them to a sidecar, followed by uploading the result.

This would be the most “k8s native” way of reaching the goal. This means that there is no need to patch the CLI (for our current needs at least, which is simply to answer the question “did the snake win?”) as that is easily parseable from the current CLI’s output, whilst also keeping the script which will upload the result in a separate container within the Pod, keeping the CLI’s container clean and simple.

Once a solution is picked and implemented I’ll be back with another blog post. In the meantime, feel free to checkout our progress on the bantersnake team page!

Sidenote: The diagrams have been illustrated by myself on a ReMarkable2. Its a great piece of hardware but my lack of decent handwriting is pretty clear. I am hoping this will improve over time.

Docker is a leaky abstraction

Sat, 18 Sep 2021 12:00:00 GMT

This blog post details how I finally understood that Docker is not the perfect abstraction I thought it was and how the hardware your docker image will run on still matters.

In my efforts to improve my Ops skills, I have taken to using Docker wherever possible. By exposing myself continually to it I can understand better how containerisation works, where it is useful and more importantly, where it is not useful along with the pitfalls involved.

In my mind, Docker was this great layer of abstraction that provided value in the form of vague keywords such as reproducible builds and isolated dependencies. Whilst I have definitely reaped those benefits, a recent experience of trying to move some software from a machine with a particular CPU architecture to another has made me realise that Docker (or any flavour of containerisation) does not insulate a developer’s code completely from the machine it runs on. Like any abstraction, it leaks.

For a particular project I am using AWS. Typically it feels over-engineered for my needs which are usually met by Hetzner, Digital Ocean or any smaller VPS provider. In this case, latency was important and the service we were interacting with also runs on AWS, so I wanted to deploy into the same AWS region (more on this project in another blogpost 😉).

I try to run my sideprojects on shoe string budgets. I believe that it helps encourage learning by doing and stimulates creativity over simply throwing money at the problem.

During the course of this project we deployed some code written in Python (rather than Go), and began experiencing CPU saturation on our AWS Lightsail VPS. This was a pressing issue as it was causing us to respond slowly, sometimes exceeding the short timeout provided to us (we have to respond within 500ms).

There were several solutions to solving CPU usage in this case, each with their own tradeoffs:

Moving to a cheaper cloud provider
Optimise the code to make it faster
Switch to a different offering within AWS

Moving to cheaper cloud provider

There are two main issues to this solution:

The amount of vCPUs available to a VPS tend to scale slowly compared to other elements such as RAM or Disk space. For instance, Digital Ocean will only offer more CPU when going from $5 a month to $15 a month.
Moving further away from AWS will introduce network latency, which counted towards our 500ms deadline. I could get 3 AMD vCPUs for around €8 a month from Hetzner, however they only offer datacentres in Europe. From my testing, the latency difference could be up to 100ms from us-west-2 to Hetzner’s Nuremberg Data Centre.

Optimise the code to make it faster

There was one main issue to this solution: we are still heavily in the experimenting and prototyping phase of the project. By spending time on optimising the code (whether the existing code or by rewriting it in a compiled language), we would be burning precious time which would be better spent on more prototyping.

Switch to a different offering within AWS

So far we had used AWS Lightsail to provision a VPS, since it was the offering most familiar to us. Lightsail however offers very little control of the kind of hardware you run on (kind of the point, you shouldn’t need to think about it!).

I had heard about AWS Graviton, which are AWS’ custom ARM processors. When I checked the price, it turned out I could get 2vCPUs on a t4g.nano EC2 instance for the same price as what I was paying for 0.5vCPU on Lightsail.

Moving to Graviton

Moving from Lightsail to EC2 was not particularly diffiicult, as I mostly worked through Terraform and had some prior experience playing around with the EC2 interface. Moving to a graviton instance however is where the learnings came from.

If you ever try to run something compiled for x86 (the CPU architecture of both my laptop, my CI server and our Lightsail instance) on an ARM machine, you’ll get cryptic error messages such as: exec user process caused: exec format error. This counts for Docker too.

The above error had already begun challenging my incorrect understanding of Docker - it was a Docker Image! I shouldn’t need to think about the machine! Well it turns out I did. I did not allow myself to be dissuaded by this and powered on, through several hours of trial and error.

It turns out that Docker has support for multi-CPU architectures on an image level using a command called buildx where one can setup a builder. Customising this builder allows users to compile across CPU architectures. In my case, on Ubuntu, I followed some guides which mentioned qemu and binfmt_misc to enable hardware emulation for buildx to operate. I admit I was mostly flying blindly at this stage. After multiple attempts I was able to manually build an arm64 image on my x86 laptop and verify this by moving the image to our graviton instance and run it as a container.

Great! But holy cow does it come at a cost. A none-too-special multi-stage Docker image packaging a single Go binary which typically took 60 seconds to build (which I personally considered quite slow), around 300s to build for arm64. These kind of long lead times really hampered our development cycle, slowing down our ability to prototype quickly.

The Solution

Not willing to take the slow deployment cycle due to the cost of emulation every time, I began looking into alternatives. One approach suggested to me by more than one person was to build the Docker image on the graviton instance itself.

Whilst a very simple and easy approach, it encourages treating the server as a pet over cattle, along with burning the same compute I wanted to improve. It was plausible that the act of building the Docker image would interfere the running of the code itself.

Another approach would be to either configure my CI/CD server (in my case I use Jenkins) to perform the hardware emulation (I’d still pay the emulation cost, but at least it is not tied to my laptop) or to source some ARM hardware and build the Docker Image on it. After a few more failed attempts, it resulted that the Raspberry Pi 3 or 4, running the 64 bit version of Raspbian (the default one provided on the imager is 32 bit) mirrors Graviton’s arm64 architecture.

Our final setup looks like this:

We write and test the code locally on our x86 laptops
When finished, we merge the code into the main branch
Jenkins (running on an x86 machine), polls the repository and picks up the new code
Jenkins Master hands off the job to a Raspberry Pi running the Jenkins Agent (by restricting the job to a specific label)
The Docker image is built for the arm64 architecture and deployed to the Graviton instance using Ansible

Caveats

Whilst this solution may feel rube-goldberg worthy, I want to reiterate that the aim is to learn over to provide the most cost-effective and bullet-proof solution.

This process also brings into question one of the main benefits of Docker: reproducible builds. A major advantage of Docker is being able to use the image built on the dev’s laptop into a test environment, then onto a staging environment and eventually production. If any of those steps involve a different CPU architecture however and you end up having to rebuild your image, are you still testing/running the same image? How can you be certain that building the image for another CPU architecture will not cause different behaviour?

Dev vs Ops - Go embed

Sun, 21 Mar 2021 17:50:00 GMT

Introduction

In my latest article on my game Cabin Fever. I document how I use Docker to compile and bundle all of the code into a single, deployable package.

With Go 1.16, embedding files has become a first class citizen.

This potentially means that, rather than relying on Docker to keep my binary and the generated CSS and JS files together, I can embed these files directly in the Go binary and ship it out as it is, without the Docker wrapping around it.

This is made possible due to Go’s efficient cross-compilation and the ability to put everything into a single binary, so your mileage may vary with other programming languages.

Using Go embed

To be able to use Go embed, I need to ensure that both my development machine and my CI machine are operating with go version 1.16. At the time of writing, my development machine is on Ubuntu 20.04.2 LTS and the latest Go version available for me is go 1.15.8. Using the instructions available here I was able to install a backport without any hassle.

Switching to Go embed

Working off of the following blogpost, it was very simple to switch to Go embed. I did not include the author’s solution to switch between the normal filesystem and the embedded one depending on the environment at this time, since the hot reload provided by modd already recompiles the binary from scratch.

My next issue was that my go.mod file was still using Go 1.13 and was unable to understand the embed directive. This was quickly solved by updating the version to 1.16.

After resolving a minor issue where the latest golang Docker image was not the 1.16 image (quickly resolved by performing docker pull golang), it looked like the embedding was not working. If I changed index.html after compilation, it was still returning the latest version, meaning that the binary was still retrieving it from my machine’s filesystem rather than the embedded one.

This was resolved by re-reviewing how I served the index.html. It turns out I had written code to read a file from the OS filesystem and had not taken the new embedded file system into account.

Whilst I could have resolved this by directing it to the new embedded filesystem, I decided to use Go embed again, but instead of embedding the file as a filesystem, I used the option to embed the contents of the file as a string, which achieved the same result.

Switching to Go embed actually made the resulting Docker image smaller, reducing it from 10.5MB to 9.27MB. Not a staggeringly large difference, but a difference nonetheless.

Technically I meant to do away with the Docker wrapping completely - however I can still use the Dockerfile as a compilation mechanism that is isolated from the machine I am running it on. According to the following StackOverflow link I can run docker create and docker cp to extract the binary from the Docker image that is created.

Takeaway

This article shows how, despite the latest and greatest Ops tools (such as Docker) technically mean that DevOps practitioners can opt to not care about whatever the code is written in, if we combine Dev knowledge of the underlying code being packaged and deployed we can simplify and streamline the DevOps process even further.

A real-time game from scratch - Continuous Deployment Part 1 [6]

Sat, 20 Mar 2021 12:00:00 GMT

Previous Entries

Introduction

Continuous Deployment is the DevOps discipline of having automated deployments that move our continuously integrated code out to development, staging or even production environments.

The reasoning is that by deploying quickly, often and in small chunks we can avoid the risks associated with large deployments, get feedback quickly and if we encounter issues we can rollback much faster.

Continuous Deployment Steps

Cabin Fever is a game that is played using a browser, so this continuous deployment would involve steps such as the following:

Provision a VPS or some sort of cloud server
Install all dependencies on that server to run the game server
Transfer the game server binary onto the server
Setup or update DNS to point to our new server

The above assumes that I am provisioning a new server every time. An alternative to this would be to provision the server ahead of time, followed by transferring the latest game server binary to the existing server and switching traffic from the old one to the new one.

One thing to deploy

Before I implement deployment, I need to ensure that I have something to deploy. In the previous article I explored the possibility of using Jenkins to build both the server and client, ending with archiving the client bundle and server binary for later use.

An alternative to this approach would be to build a single package which contains everything I need to place on the server to get started. Having worked extensively with Docker, it felt like the easiest way to proceed. It would also make the process simpler, as I could build the image on my CI machine and ship out the image only to the production machine.

Multiple Dockerfiles vs one Mutli-stage Dockerfile

In a project spanning different parts of a stack (in this case, backend and frontend), one will often find Dockerfiles for each part of the project, both for development and deployment purposes. This process helps keep the code both portable across machines (so as long as the docker daemon is available), more reproducible (since all the toolchain required is available in the docker image) and easier to get started with.

Since I am using a mono-repo for Cabin Fever (meaning all the code, regardless of the part of the stack, is in the same git repository), I chose to leverage a single multi-stage Dockerfile over having to wrangle separate ones.

Multi-stage Dockerfiles are considered a good practice even at a single part of the stack - it is a useful tool to avoid shipping larger docker images than are necessary. A smaller image ends up being both cheaper and faster to move around, cutting down on turnaround times.

A psuedo-code example of a multi-stage dockerfile could be the following:

Stage 1: Start from the latest available golang image
Stage 1: Compile the binary using the golang toolchain
Stage 2: Start from the scratch image
Stage 2: Copy the compiled binary from Stage 1
Stage 2: Setup the image to run the binary

As a result, the final image will not be shipped with the golang toolchain (which is not required to be present in the image for our usecase), making it smaller, more secure and more efficient.

Mutli-stage for Cabin Fever

In the final resulting Dockerfile, I used the following pattern:

Stage 1: Using the latest available golang image
Stage 1: Compile the binary using the golang toolchain
Stage 2: Using the latest nodejs image
Stage 2: Download all required dependencies
Stage 2: Build the production bundle
Stage 3: Using the scratch image
Stage 3: Copy the compiled binary from Stage 1
Stage 3: Copy the production bundle from Stage 2
Stage 3: Setup the image to run the server binary

This results in a 10.5MB image, consisting of both the server binary and the frontend bundle. The only two changes I had to do to the code to enable this process were:

serve the index.html file at the root endpoint / and also to serve the static folder through the server so that JS and CSS files can be served to the end user.
set the production bundler (in this case, parcel) to include /static/ as a path under the CSS & JS assets in the generated index.html using the following command line argument: --public-url ./static/. I included this addition in the package.json under the build script to avoid having to include it in the Dockerfile.

Conclusion

Whilst none of the above is technically deployment, it lays the groundwork for the packaging of all the code involved in delivering the full game experience to our end users. The next article will involve more of what I initially wrote about: the actual provision of a new server or usage of an existing one to serve the game experience.

Next Entry: Continuous Deployment Part 2 (Coming soon)

A real-time game from scratch - CI/CD [5]

Mon, 15 Mar 2021 22:00:00 GMT

Previous Entries

Introduction

Up until now, the development of Cabin Fever has all happened on my personal machine, running on localhost. Showing a demo to someone either involves physical proximity to demo it (which is not a good idea given the ongoing pandemic) or recording a video and uploading it to YouTube.

Alternatively, I could push both the back-end and front-end to a server or VPS and allow potentially anyone on the internet to try it out.

By the end of this article, my aim is to have automated the full process of building the software and pushing it out to the server, rather than having to perform those steps manually. I will be applying the DevOos discipline of CI/CD to achieve this.

CI & CD

CI and CD here refer to the following:

Continuous Integration
Continuous Deployment

Continuous Integration

Whenever anyone wants to work on a piece of code, it is a good idea to keep track of the changes using a Version Control System (VCS) such as git. This is software which allows us to keep different versions of the same code, switch between these versions and even bring different versions together.

The idea behind Continuous Integration is to automate checks and balances that the differences between our different version (or branch as git calls it) are still compatible with the original version we branched off from and that the logic of our application still behaves as expected.

In our case, it would involve the following steps:

Check that the code is still valid and can be compiled.
Run automated tests to ensure our code still implements our logic as we expect it to.

The result of this process will either be a compiled binary, ready to be used/deployed, which means we have successfully integrated our code changes. Doing this continuously avoids large changes from piling up which increases the risk of the process failing. By automating it, we take the tedium and human element out of it.

If any of the above steps fail then we should expect some sort of report stating why. This could be that our integration failed, or our new code changes cannot be compiled, or even that some of the tests failed. It is more ideal for us to catch these issues at integration stage rather than when we have already pushed the code out to our production environment.

I plan on tackling the Continuous Deployment part in a future article

DevOps before it was cool

I like to say I did DevOps before DevOps was cool. I’m a DevOps Hipster if you will.

I started my first internship a whole month before the other students. This is because my exams finished sooner and rather than sit and stare at home, the company too me in anyway.

On one of my first days, the Technical Architect outlined one of my first tasks: “Take this thing called Jenkins and link it to this thing called GitLab”. Being a very green programmer, this was my kind of challenge: no idea what I’m doing and no idea where to start from! Nowadays this task takes maybe ten minutes to complete, but circa 5-6 years ago it was not as trivial.

Jenkins

Jenkins is the automation software that the company was using for its Continuous Integration. I decided to use Jenkins again for Cabin Fever as I had already revisited it as I was improving my DevOps skills. I was pleasently happy to find Jenkins had become more versatile over these past several years along with still being as usable as the day I first used it.

Jenkinsfile

The Jenkinsfile is a file describing the pipeline of automated steps that we want to perform. In this case, we would want to perform the following:

Build and package the server binary
Build a production bundle of the frontend
(In the next article): deploy the server and the frontend to a server accessible by our players

Focusing on the first two steps were not particularly difficult as I had setup similar pipelines in the past. It involved:

Starting up a docker container containing the build tools I require (for the server: GoLang and for the frontend: NodeJS)
Running the tests against the built code (if there are any)
Running a command to build the code
Archiving the resulting binary/bundle for later use

I involved docker to keep my builds isolated and repeatable. This means that they do not rely on having toolchains present or configured on the machine that Jenkins is installed on, making it easier to reproduce even if I had to run my Jenkins on another machine.

Conclusion

It took some back and forth because I tried running the server compilation within an alpine image, but after eight attempts I finally completed the Jenkinsfile, meaning I can now build all of Cabin Fever on my personal Jenkins server after every push to the git repository.

Right now the assets are just sitting there, archived within Jenkins, so they aren’t particularly useful. My next article will focus on the Continuous Deployment part - the aim being deploying the latest version of Cabin Fever to my personal server using Jenkins to enable quick playtesting without always running it on my development machine.

Next Entry

6 - Continuous Deployment Part 1

Dev vs Ops - Hide and Seek

Thu, 11 Mar 2021 21:10:00 GMT

Introduction

I received a merge request for review. The changes involved renaming a folder, with no other discernable changes. This MR had the following paraphrased description:

The code works on my laptop, but this is the only way I could get it to build on Jenkins, otherwise the build fails.

🤔

Blurred Line

Similar to my last blog post in the Dev vs Ops series, my mind wandered onto the difference between this dev’s machine (a Macbook similar to mine) and Jenkins. I knew it had something to do with docker.

One of the selling points of containerisation is that the same container that builds on a dev’s machine gets built on your CI and gets shipped out to production, reducing the impact of different environments.

To rule out the environments being the issue, I attempted to build the docker image locally and in fact the same error occurred. Something was failing between our machines and the docker daemon.

Solution

Most dockerfiles will have the following directive or something similar within them:

COPY . .

This line basically says copy everything from the host in this directory into the docker container’s root folder. One might want to only copy a src folder and nest in inside another directory in the container, which would look something like this:

COPY src/ /app/src/

When you run docker build, the docker daemon will actually show you the progress of copying the host’s context into the docker daemon and the amount of data it moves across this boundary.

We do not want to move everything from the host into a container, typically we would only want to move the minimum required to be able to build and run our code.

Rather than specifying exactly what we want to copy into the container (using COPY) in a whitelist fashion, instead we can do this in a blacklist fashion by creating a file called .dockerignore.

Similar to .gitignore, this file instructs the docker daemon to ignore specific files, folders or combination thereof.

It turns out that one of the files was importing a file held inside a specific folder… this same folder was included in the .dockerignore file as it was not required at runtime in production. This meant that the file existed at build time on the dev’s machine, but not within the docker daemon when the container was being built.

Takeaway

Works on my machine needs to be taken with a pinch of salt, or you’ll end up playing hide and seek with missing files.

Dev vs Ops - Case Sensitive Operating Systems

Sun, 07 Mar 2021 12:40:00 GMT

Series Introduction

I am starting a series of articles exploring issues at the line between Development and Operations.

In the past few years this middle ground has started being filled with DevOps Engineers. I believe that whilst DevOps is the logical next step in how we package, run and monitor our software, ever since it has no longer remained part of a software developer’s workflow it has caused Operational and Deployment issues to become “someone else’s problem”.

This encourages developers to care less about how or where their code is run, so as long that it runs on their machine.

Dev vs Ops aims at re-blurring this line, documenting issues I resolve as I work on my DevOps skills. I believe that all software engineers should have at least a cursory understanding of how and where their code will run, without needing to become Kubernetes experts.

Introduction

I was working on a NodeJS project written in TypeScript.

One of the files had an import written as follows:

import shortid from 'shortId';

Nothing out of the ordinary, and the code builds! Works on my machine! I submitted the code for review, it got approved and merged into our develop branch.

Our CI tool (in this case, Jenkins), dutifully detected a new merge on develop and started building the code for deployment onto our development environment.

The build failed and amidst the logs I found the following error (paraphrasing):

No module called shortId.

🤔

I double checked both package file and lock file to be sure I was not being bitten by something at runtime rather than at buildtime, however shortid was clearly present in both.

Blurred Line

It did not feel like a code problem and I did have a sneaking suspicion scratching at the back of my brain that this issue resided at the boundary between Dev and Ops, so I put on my metaphorical debugging hat.

What is the main difference between my machine and Jenkins?

The first that comes to mind is that my machine is a 2019 MacBook running MacOS whilst Jenkins is on a Linux machine. Is there potential for difference in behaviour between these two systems?

I felt close to a solution but also a bit hesitant. I did know that we package all our software as Docker containers and that Jenkins did basically that: build the Docker container and push it out to AWS.

To confirm whether it was the host system or the process of building the container, I decided to build the docker container locally. This was trivial to do as the Dockerfile is in the same repository.

Build failed! Does not work in my container!

From the Dockerfile I could see that we were using the node-alpine Docker image as a basis for our image. This convinced me that the issue was an incompatibility between running it on MacOS and running it in Linux, just not because of the machine the CI was running on as I originally thought was the issue.

Solution

The issue turned out to be case-sensitivity. Whilst MacOS tolerated shortid being written as shortId, Alpine only began accepting the import once I renamed the import to shortid.

Had I kept the mindset that anything that happens that is not on my machine is DevOp’s concern (or fault), I could have easily wasted everyone’s time and caused unnecessary friction between our teams.

Takeaway

Be aware of how your code is packaged and will be run - it is still your responsibility even if its not on your machine anymore!

Kudos to Daniel D’Agostino for the article review.

Visualising Covid-19 in Malta

Thu, 04 Mar 2021 20:00:00 GMT

Introduction

When the pandemic the world became obsessed with numbers. Back then it was new infections, eventually deaths and now vaccinations. A flurry of visualisations popped up, most famously the one from John Hopkins University. To help myself understand the progress of the pandemic in my home country of Malta and as an excuse to learn more about GatsbyJS, I created my own visual tracker.

The main difference between my Covid-19 tracker and the rest is that I wanted to localise it specifically to Malta. Whilst many use WHO, CDC or ECDC data, I based mine on two sources:

The Open Dataset published by the Public Health team in Malta
A collection of events that I maintain myself

By encoding events on the chart alongside official data, I avoid speculation and unreliable reporting whilst also helping to show the effects of events and measures introduced on the situation.

How its made

The data retrieval, parsing and calculation is a set of Python 3 scripts which retrieve from the Public Health open dataset.
The visualisation is generated using GatsbyJS and react-vis, using TypeScript
The automation of updating data depended on GitHub Actions until it was moved to Jenkins instead to avoid having to update the repository with the data generated every day.

You can find the latest version hosted here: https://www.c19.mt/ and more information about the project on the GitHub Repo.

A real-time game from scratch - Physics, TDD and Core Game Loops [4]

Fri, 27 Nov 2020 17:15:00 GMT

Previous Entries

Introduction

Moving a spinning yellow box is not particularly interesting or engaging. I am trying to write a space game after all.

To continue to head in that direction, I will begin implementing some basic physics. The challenge here is two-fold:

Realistic physics as if the player were in space
Having the physics calculated on the server side and propogated to the client

Space Physics

Physics in space is a bit weird if you have never been exposed to the math behind it. Since there is no friction to speak of in the vacuum of space, Newton’s laws of motion display themselves beautifully when compared to how they are exhibited under Earth conditions.

The laws are as follows:

If a body is at rest it will remain at rest. If a body is moving, it will keep moving at constant speed in a straight line. These hold unless the object is acted upon by a force.
The acceleration of a body is proportional to the force applied to it and its mass. Conventionally known as F = m * a.
For every action, there is an equal and opposite reaction.

Newton’s First Law… in space

Imagine a ball sitting in space. It is not moving (it actually is if you look at it on a solar system or galactic scale, but let us keep things simple). Given nothing ever coming into contact with it, it will remain as it is: motionless.

Now imagine an alien passes by and kicks the ball towards our sun. The ball now travels freely and smoothly in a straight line. With no one or nothing else interacting with it, it will happily keep moving towards the sun at the same speed (I am leaving out gravitational pulls for simplicity). There is no friction to take away from the kinetic energy that has been given to the ball.

Newton’s First Law… in code

Currently in our case, the inputs from the keyboard which affect the spinning square are the equivalent of the alien kick mentioned above - we are applying a force to a motionless object which is sitting in a no-friction vaccuum (AKA outer space).

If we leave the ideas of force, mass and acceleration out of the picture for now (we will cover that later when we get to Newton’s Second Law), then the expectation of the movement of our spinning square is very simple: If we move it in a direction, we expect it to keep moving in that direction, until we move it in a different direction.

To write the code implementing the above statement, we can use a software development methodology called Test Driven Development. The idea is that we will write a program which will verify the behaviour of our physics program for us. If the tests pass, then the logic is correct, otherwise we should investigate as we may have introduced a bug, or our expectation of the behaviour of the physics program is incorrect.

For our physics (which only considers the first law), I wrote a function which takes a position, a direction (which may be nil, meaning no direction was passed. nil ).

Accompanying this, I wrote a function which takes a position, the expected position and a direction. This second function simply calls the first one and compares the result with the expected result. If there is a difference, then an error is thrown, otherwise the test passes.

This form of automated testing helps ensure that the logic of my code is correct and remains correct throughout the development of the project. If the test begins failing, then either the code was changed in a way that violates the expected logic or the logic has changed and now the code needs to be changed to match it.

Core Game Loops

To apply the code written for our initial physics engine (which for now covers Newton’s First Law only), we need to reimagine the paradigm of how our server will work.

In its current form, the game state is only ever updated when the client sends data to the server. Now that we are introducing phyics, the game state update should be calculated regardless of whether the client send data or not. The data the client sends should count for the next physics update being calculated.

The above means that we need to implement a core game loop. Different from a core gameplay loop, this is a continuous process which will receive inputs within a specific timeframe, then update the game state and push it out to all clients, even if no inputs are received.

This raises the question: how often should we update? This update time, also called a tick rate determines how responsive a game will be to inputs but also how much data is passed back and forth. The higher the tick rate, the more data will be received, regardless of whether inputs are sent to the server or not.

A useful part of software is that these kind of decisions can be configurable. I do not need to take the decision now and deal with the consequences, but rather I can make the tick rate a parameter and then test out different values to try and find the sweet spot in the ratio of responsiveness to data flow.

Here begins the usage of the concurrency primitives that Go provides. The core game loop will run in its own go routine and on a timer (if our tick rate is 30 ticks per second, then we it would be 1000ms / 30 = every 33.3ms) it will send out a game update. Until that time elapses, the game will accept inputs that the server receives and reccalculate the game state.

Demo

You can find the first video demo of the above logic implemented here:

This would be trivial to implement in a client-side only fashion, however in the above demo all the game state is being calculated server-side, pushed out to the client at the end of every tick.

Conclusion

Whilst the server-side code looks like a bowl of delicious spaghetti and someone’s first concurrent project ever, it works! Whilst it is indeed a small step, it is beginning to feel like proper progress.

Given how long this article is, I have chosen to implement the rest of Newton’s Laws in another article. The next article however, will aim at improving my DevOps skills by implementing Continuous Integration and Continuous Deployment (CI/CD) for the latest version of Cabin Fever on every push to the main branch. This will include:

Building the Frontend
Building the Backend
Ensuring that tests pass
Deploying to a server
Switching over from the old ones to the new

You can find the code for this article at the article-004 tag.

Next Entry

5 - CI/CD

A real-time game from scratch - PixiJS [3]

Thu, 12 Nov 2020 21:00:00 GMT

Previous Entries

Introduction

At some point I need to be able to show something on the client side that is not simply text retrieved from the server.

Short of making some super complex ASCII art based rendering engine I will need to use some library or framework to avoid messing with the canvas manually.

Research

I did not do much research before making my choice.

Admittedly this may be flawed logic if one is considering a long term project, however in my case I would not mind starting with one and switching to the other as it means I would have gained in experience in two tools rather than one.

As a result, my criteria for selection were very simple:

I had heard about it before
It looks vaguely popular (GitHub stars, albeit a noisy indicator of popularity, was a useful consideration here)
Updated semi-recently (This is easy to do with open-source projects since I can take a look at how recent the last commits are)
If open-source, a permissive licence
TypeScript support

Fitting all of the above criteria very well, I chose PixiJS. I did not choose PhaserJS as it appeared to be more featureful but also more restrictive. I do not want to spend my time learning “the Phaser way” of writing a game.

Admittedly in a few article’s time I may bemoan my lack of open mindedness on this matter.

PixiJS

PixiJS is currently sitting at version 5. Whilst getting started was easy, the example on the GitHub readme was not completely “copy-pasteable”. It helps me greatly when my first impression of any library that I want to try out is an example I can simply copy paste onto my machine and get running as fast as possible so that I can verify my setup is correct.

After a bit of digging, I found an examples section on the website which contains both the example running and editable source code I could tinker with to see the effect of my changes.

Soon enough, I had a spinning yellow square on my screen. Initially it was rotating around the origin (top left corner) in and out of the screen. This would have left me very confused had my Bachelor’s Degree in Creative Computing not taught me about global and local co-ordinates and how rotations work in 2D and 3D spaces.

From here it was not difficult to wire up the exchange of data between client and server to the new visual frontend. On every tick, I would update the Yellow Spnning Square’s X and Y position values to whatever was received in the last message from the server.

Whilst a promising start, the box moved too slowly! To get it move faster, I began multiplying the retrieved position by 10 so the box appeared to move faster across the screen.

This however got me thinking: the front-end visualisation is tightly coupled to whatever state is sent back by the server, mapping it 1 to 1 from game state to pixel position on the browser.

In reality, rather than simply using whatever the server sends back, it would make sense for the front-end to map it to the current browser, otherwise a player on desktop with a 4K screen would be able to see more of their surroundings than a player on a laptop with a 1080p screen.

Conclusion

A yellow spinning square is not much of a game, but it beats simply showing text and provides a good foundation to be able to properly visualise changes in game state being sent by the server.

So far any changes are caused by my input from the browser, so in my next article I will attempt to implement a basic physics system, simulating what one could expect if they were in space.

You can find the code for this article at the article-003 tag.

Next Entry

4 - Physics, TDD and Core Game Loops

A real-time game from scratch - WebSockets [2]

Wed, 28 Oct 2020 21:00:00 GMT

Previous Entries

Introduction

Cabin Fever is one of those side-projects which occupies a disproportionate amount of mind share compared to Tempus or C19. Whilst I am making an effort to work on side-projects which provide more end-to-end value over a deep technical dive form of value, the challenge of tackling a realtime entertainment requirement is far too enticing to not devote thinking time to.

Passing data back and forth

We need to be able to send instructions from the browser to the server and data back about the game state. In the first iteration I used the typical HTTP request-response so as to ensure that the setup was working.

This method however is far too inefficient. There is an alternative called WebSockets.

WebSockets

Instead of setting up a connection, sending a request, receing a response, then closing the connection, we can setup the connection, send and receive as much data as we want, then close the connection once we’re done.

Server-side

On the server-side, I considered using the built in support for websockets, however I got started much faster with github.com/gorilla/websocket. Creating the initial websocket handler took only around 20 lines of code that hooked into the same http server I used for serving HTTP requests. My only issue was that I needed to provide a checkOrigin function which is basically the websocket equivalent of CORS.

Client-side

On the client-side, I did not need any custom libraries so far. I simply started using the built-in WebSockets implementation to connect to the new endpoint (in this case, /ws, since / was taken by a standard HTTP handler).

Since we explicitly need to wait for the connection before starting, our code begins to look event-driven. To setup, I hook into onopen on the WebSocket object to ensure that the connection has been setup before we begin to send messages back to the server and the onmessage event to react to messages received from the server.

Towards a game

So far it has been sending datetimes back and forth, which does not amount to much of a game.

The endgoal of this project is to create a realtime browser game, so the next step is to begin introducing features one would find in a game. An easy place to start is capturing keyboard events and sending them to the server, having it update some state on the sever which can be sent back to the client.

This process has an important aim - by performing calculations on the server rather than the client we are aiming for a server-authorative model, which can make cheating more difficult.

Keyboard events

Capturing keyboard input was not difficult at all - it simply involved attaching a listener to the onkeydown event on the <body> tag of the HTML document. This gave me access to e.key which value is a string referring to the key pressed. In my case, I stuck to the classic WASD combination (which is a left hand equivalent of the arrow keys) which are used most of the time in games to control lateral movement.

I only begin listening to keyboard inputs after onopen on the websocket and depending on whether the key was w, a, s or d I translate it to an enum with the values UP, DOWN, LEFT or RIGHT. Notably I do not update the internal state of the client side at this point in time. I do happen to know that some games do both at the same time and then update the state when it receives a response from the backend with the aim of making the game feel smoother and snappier. I plan on applying this optimisation later on.

Game State

On the server-side I initialise a struct called GameState which for now simply tracks the X and Y position of the one player. On receiving of a message, I update the X and Y value according to whether the received string is UP, DOWN, LEFT or RIGHT.

Two points of note here:

The co-ordinate system in the browser is partially inverted from what we learn at school in Maths
The game state has no wrap around validation and I ended up causing an integer underflow

Browser Co-ordinate System vs School Maths Co-ordinates

In school, I learnt that a cartesian co-ordinate system works like this:

When it comes to the browser (or graphics in general), the Y Axis is actually inverted. Increasing the Y axis value means moving downwards rather than upwards.

Put simply:

Moving up: y--
Moving down: y++
Moving left: x--
Moving right: x++

Wrap Around and Underflows

In a game like pong, there are boundaries to the game. The ball will never exit the boundaries (hopefully.) This means that the x and y positions should never go lower than zero, if the origin point is the top left corner.

In my case, whether the final game will have wrap around or not is a design decision for a later stage. An interesting occurrence is that, out of habit, I used the uint type to represent the X and Y values in the game state. uint stands for unsigned integer, meaning a zero or positive whole number, the value cannot be negative.

As I tested the client (which was writing the retrieved game state to the DOM), the Y value suddenly jumped from 0 to 18446744073709552000. In this case, since the value is unable to be negative, it wrapped all the way around the largest possible value that it can store. This is the same reason that Gandhi in the original Civilization game would become aggressive, birthing the meme of Gandhi using Nuclear Weapons: the low aggression number would be pushed into a negative value, however since it was unsigned it would wrap around to the largest possible value.

The solution to this is to provide validation of the values - if it reaches below zero, then switch it back to zero.

Conclusion

So far I have not felt particularly challenged as I have not written any code which I was not already at least familar with. Here are the following avenues I look forward to exploring:

A proper frontend for rendering graphics, using a framework such as PhaserJS or PixiJS
Multiple Clients - how to handle receiving of data from multiple sources, updating a single game state and propogating those changes to all clients
Physics - having the game state update on a core loop and push out updates to all clients regardless of whether they provide input or not

You can find the code for this article at the article-002 tag.

Next Entry

3 - PixiJS

Riċerka rigward Covid Alert Malta

Sun, 25 Oct 2020 10:31:00 GMT

This article is also available in English.

Introduzzjoni

Qiegħed nikteb dan l-artiku ftit iktar minn seba’ xhur wara li l-pandemija tan-Novel Coronavirus waslet f’Malta. Hemm ħafna aspetti tal-pandemija li nixtieq naqsam ħsiebi fuqhom, speċjalment rigward il-veritajiet li kixfet dwarna bħala umani u anke bħala nazzjon, f’sens soċjo-ekonomiku u anke politiku, iżda għal dan l-artiklu ser nżomm ma dak li naf u nifhem l-iktar fih: it-teknoloġija.

Contact tracing f’dawn iż-żminijiet

Il-Gvern Malti ħareg app tal-mowbajls għal-Contact tracing. Din hija application li tikkuntattja mowbajls oħrajn fil-viċinanzi sabiex tgħin fil-mekanżimi tal-contact tracing.

L-ideja hija li, jekk inti kellek kuntatt ma xi ħadd li ġie pożittiv għall-Covid-19, il-proċess sabiex inti tiġi infurmat ssir ħafna ehfef u iktar malajr, ħabba li l-mowbajl tiegħek żamm kont tal-fatt li inti kont fil-viċinanzi ta’ każ pożittiv.

L-ikbar problema għal sistema ta’ dan it-tip dejjem ħa tkun:

Kif tista tippreżerva l-privatezza ta’ individwu filwaqt li żżomm kont tagħhom u tal-kuntatti kollha tagħhom?

L-awtorita’ teknoloġika tal-Gvern il-MITA tisħaq li l-app giet żviluppata mal-linji gwida tal-GDPR u li saret verifika tagħha mill-Awtorita’ tal-Innovazzjoni Diġitali ta’ Malta, l-MDIA.

Sfornutatament nuqqas ta’ trasparenza hija komuni f’xogħol Malti - ġieli huwa diffiċli li tirċievi risposta fuq b’liema mod u sa liema livell sar dan ix-xogħol.

F’azzjoni rari, il-MITA għazlu li l-app ssir open-source. Din hija azzjoni kuraġġjuża u hija l-ahjar metodu li biha l-MITA jistgħu jżidu fiduċja fis-sistema.

Litteralment qegħdin juru li mgħandhom xejn x’jaħbu rigward kif taħdem l-app. Din tippermetti sabiex anke l-iktar nies xettiċi jistgħu jagħmlu riċerka tagħhom u jivverifikaw li s-sistema filfatt tippreserva il-privatezza u li hija għodda tajba sabiex tgħin il-pajjiż ġġib il-pandemija taht kontroll iktar malajr.

Iż-żwieg bejn il-privatezza u Contact tracing

Il-Contact tracing jinħass donnu jaħdem kontra l-privatezza nnifisha. Jikkuntatjak xi ħadd li ma tafx, dwar xiħaga li ma tistax tara b’għajnejk u trid taqsam magħhom fejn kont dan l-aħħar u ma min kont f’kemm tiflaħ dettall.

Il-Contact tracing huwa importanti ħafna sabiex titwaqqaf pandemija, allura akkost ta’ kemm forsi li tmur kontra il-kunċett tal-privatezza, nagħmluha xorta ħalli forsi nħallu din il-pandemija warajna ħarira iktar malajr,

Meta tahseb ftit iktar fuqha, l-ħin u l-post fejn kellek kuntatt ma xi ħadd mhumiex importanti daqs ma min kont f’kuntatt (skont parametri bħad-distanza bejnietkom, kemm domtu ħdejn xulxin u jekk kontux libsin maskra). Ma jkunx ahjar kieku stajna naqsmu din l-informazzjoni mingħajr ma nidħlu f’dettallji dwar fejn u meta?. Bonus jekk tista tinkludi nies li inti qas tafhom personalment.

Problema oħra hija li l-memorja tagħna l-umani ma tantx hija robusta jew b’saħħitha. Ninsew dettalji, nħawdu l-ordni ta’ kif ġraw l-affarijiet jew mingħalina li xihaġa damet inqas jew iktar milli verament damet. Dik il-parlata li qbadt mal-ġirien għand il-grocer? Mingħalik li domt ħames minuti titkellem imma fil-verita’ domt għoxrin minuta. Ma jkunx aħjar kieku stajna niftakru d-dettalji ta’ dawn il-ġrajjiet b’ preċiżjoni assoluta?

Grazzi għal-kollaborazzjoni bejn imħuħ intelliġenti li jaħdmu ma Apple u Google, ammont kbir mill-mowbajls moderni (dawk li għandhom sapport għal Bluetooth Low Energy (BLE) u li għandhom Android 6 “Marhsmallow” jew ogħla jew iOS 13.5 jew ogħla) issa għandhom sistema ta’ Privacy Preserving Contact tracing mibnijja diretta fl-Operating System tagħhom.

Dan l-Artiklu

Dan l-artiklu jiffoka fuq l-implementazzjoni speċifika tal-app li żviluppaw l-MITA li taqbad mas-sitema tal-Apple-Google Privacy-Preserving Contact tracing, minflok is-sistema tal-PPCT innifisha. L-għan huwa li nesplora u nivverifika d-dikjarazzjoni li din l-app ma tiġbor l-ebda tip ta’ data dwar il-location tal-individwu li qiegħed jużahha.

Ser nimxi b’metodu sempliċi ta Mistoqsija u Risposta minflok nirrispondi sett mistoqsijiet fl-aħħar.

Covid Alert Malta

Aħjar nibda b’spjegazzjoni ta’ xinhu open-source.

Software li huwa open-source huwa software fejn il-code tal-programm jista jinqara u jiġi kkompilat, filwaqt li closed-source ma jħallikx taqra l-code tal-programm u min qiegħed jużaħ jista biss juża s-software fil-forma diġa kkompilata (bhal Microsoft Word) jew bħala servizz (bħal Google Maps).

B’open-source software (ġieli mqassar bħala OSS), nistgħu nivverifikaw l-imġieba tal-programm jew billi nħaddmu l-programm jew billi naqraw s-source code, minħabba li jista jkun li jkun hemm imġieba li qiegħda ssir wara l-kwinti imma li mhiex tidher. F’closed-source, nistgħu biss nosservaw l-imġieba tal-programm billi nħaddmuh, u rridu noqogħdu fuq x’jingħad lilna sabiex nkunu nafu x’jagħmel (jew ma jagħmilx) meta nħaddmuħ.

Kieku kelli noffri analoġija, kejk taċ-ċikkulata li huwa closed-source huwa mit-tip li tixtri lest u li tista tġawdi mill-ewwel. Jidher sabiħ u għandu togħma tajba, imma mgħandek l-ebda ideja fuq kif, fejn, meta u b’liema ingredjenti ssajjar. Il-furnar jista jassigurak li l-kejk mgħandux gluten, huwa “vegan” jew ssajjar f’ambjent li huwa “nut free”, imma l-uniku assigurazzjoni li għandek tal-verita hija fil-kelma tal-furnar.

Kejk li huwa open-source jigi bir-riċetta inkluża, b’lista speċifika ħafna tal-ingredjenti, l-ammonti tagħhom, is-sors tagħhom u l-proċess tat-tisjir. Nistgħu nosservaw jew nieklu l-kejk filwaqt li ma nagħtux każ tar-riċetta, imma wkoll nistgħu nuzaw ir-riċetta sabiex nsajru verzjoni tagħna tal-kejk, jew sabiex nbiddlu parti minnu għal xihaġa li nippreferu iktar (per ezepmju, nagħmlu ġamm flok frosting taċ-ċikkulata) jew sabiex naraw jekk nispiċċawx bl-istess kejk ezatt li kien diġa misjur u mogħti lilna jekk insegwu l-istruzzjonijiet.

F’dan il-każ, l-app ta’ Covid Alert Malta, kemm tal-Andriod u anke tal-iOS, flimkien mas-server li tieħu hsieb tinnotifika l-kuntatti pożittivi, huma forks (l-ekwivalenti ta’ remix fis-software) ta’ apps oħrajn tal-Contact tracing li huma open-source, speċifikament is-SwissCovid App.

Mistoqsija

Iġifieri l-MITA serqet l-app SwissCovid u għamlitha daqs li kieku tagħha?

Risposta

Lanqas xejn. Il-liċenzja tal-open-source li taħtha saret l-app SwissCovid tippermetti modifikazzjonijiet u anke tqassim (nikteb f’iktar dettall fuq dan iktar tard fl-artiklu).

Mistoqsija

Għalfejn l-MITA ma kitbux app mill-ġdid?

Risposta

Jiena ma nitkellimx għal-MITA, allura ma nistax nirrispondi f’isimhom, imma nista nispekula.

Qabel ma tikteb kwalunkwe software, wieħed għandu jagħmel evalwazzjoni fuq il-fiżabilita’ tagħha, kemm f’termini finanzjarji u anke ta’ ħin. Il-kitba ta’ app tal-Contact Tracing mill-bidu nett hija possibbli, imma l-ammont ta’ investiment ta’ ħin u għarfien li hemm bzonn huwa għoli ħafna. Fil-każ ta’ pandemija l-ħin huwa ta’ importanza kbira allura kwalunkwe soluzzjoni li tinvolvi investiment għoli ta’ ħin ma tibqax soluzzjoni tajba.

Jkun iktar vijabbli kieku ssib implementazzjoni eżistenti u taddattahha għall-bżonnijiet tiegħek speċifiċi. Wahda mill-idejat prinċipali wara open-source hija li wieħed jevita xogħol doppju. Code jista jigi mqassam f’spirtu ta’ kollaborazzjoni fejn kulħadd jibbenefika u nevitaw li nerggħu nivvintaw ir-rota mill-bidu kull darba.

Mistoqsija

Il-MITA obbligati li jqassmu s-source code?

Risposta

Iva. (Kważi) kull open-source code tiġi b’liċenzja li tispeċifika id-dmirijiet ta’ individwu jew organiżazzjoni meta jagħmlu użu mill-code. F’dan il-każ, l-app SwissCovid app hija liċenzjata taħt il-Mozilla Public Licence 2.0. Nikkwota dirett mill-FAQ tagħhom (bl-ingliż):

Q9: I want to distribute (outside my organization) MPL-licensed source code that I have modified. What do I have to do?

To see the complete set of requirements, read the license. However, generally:

You must inform the recipients that the source code is made available to them under the terms of the MPL (Section 3.1), inċluding any Modifiċations (as defined in Section 1.10) that you have created.

You must make the grants described in Section 2 of the license.

You must respect the restriċtions on removing or altering notiċes in the source code (Section 3.4).

Mistoqsija

App li hija open-source hija inqas sigura?

Risposta

Is-sigurta’ ta’ kull tip ta’ software mhiex marbuta ma jekk hijex open-source jew le, imma hija marbuta mal-kwalita’ tas-software innifsu.

Meta xi ħadd jikteb software li muhiex sigur u jagħzel li ma jiżvelax is-source code, hija l-ekwivalenti ta’ li ssajjar kejk imma tagħżel li ma tiżvelax jekk fihx gluten jew le. Irrispettivament ta’ jekk tiżvelax jew le, l-ammont ta’ gluten ma jinbidilx u jista jwassal xorta għal-ħsara fis-saħħa ta’ min jiekol il-kejk jekk min qiegħed jieklu huwa intolleranti għal-gluten, jew huwa coeliac.

Li tagħzel li ma tizvelax source code fuq bazi ta’ sigurta’ huwa metodu magħruf bħala sigurta’ permezz tal-oskurita’, li huwa metodu skoraġġat minn kwalunke persuna professjonali li taħdem fis-setturi ta’ Software, Sigurta’, jew l-Informatika.

Mistoqsija

Din l-app mhux ġiet ċertifikata mill-MDIA?

Risposta

Wieħed jispera li software bħal Covid Alert Malta, fejn f’dinja idejali hija installata fuq kull mowbajl li hawn fil-pajjiz, tkun għaddiet minn proċess rigoruż ta’ inspezzjoni u verifikazzjoni.

S’issa l-MDIA assigurat li l-app għaddiet mill-proċess ta’ ċertifikażzjoni tagħhom, iżda dan il-proċess muhiex publikku jew trasparenti. Fidi għamja f’instituzzjoni għandha tkun evitata’ minħabba li hija magħmula mill-umani. Jiena ninkoraġġi lill-MDIA sabiex jaqsmu kif u sa liema livell sar dan iċ-ċertifikażzjoni għax dan jżid iktar il-fidi fis-sistema, li għandha twassal għall-aktar nies jinstallaw l-app.

Irrispettivament ta’ dan, il-fatt li l-app hija open-source tħalli lil kulħadd jagħmel verifika tagħha. Irrispettivament ta’ kemm għandek fidi f’kwalunkwe instituzzjoni, kemm jekk inti inqas jew iktar pedantiku mill-instituzzjoni, il-fatt li s-source code huwa pubbliku u tista ssir riċerka fuqu ttik l-opportunita’ sabiex tagħti l-approvazzjoni tiegħek personali.

Is-source code

Il-kompetenza u l-esperjenza tiegħi tinsab l-iktar fl-apps li huma cross platform. Jiena ma niktibx apps f’lingwi speċifiċi ta’ pjattaformi (bhal Java, Kotlin, Objective C jew Swift), iżda għandi espejenza nikteb Java u esperjenza fl-apps in ġenerali u nikkunsidra din l-esperjenza biżżejjed sabiex nagħmel din ir-riċerka.

Is-source code tal-app Covid Alert Malta tinstab f’dan il-link: https://github.com/GOVMT-MITA/dp3t-app-android-ch

Minħabba li hija fork, nistgħu nikkomparaw id-differenzi bejn ir-repositorju oriġinali u dak tal-MITA. Din id-diff, qasir għal differenza, hija lista ġġenerata awtomatikament.

Minn ħarsa ta’ malajr, jidher li l-uniku żieda fl-app mill-MITA kienet l-għażla tal-lingwa meta tużaħħa l-ewwel darba u l-lokaliżżazzjoni minhabba l-lingwa Maltijja. Ma jidher li ma hemm l-ebda differenza fl-imġieba tal-app minn dak provdut minn SwissCovid.

Mistoqsija

L-App tibqa tinsisti li nixgħel is-setting tal-location. Kulħadd jinsisti li l-app ma żżommx kont tal-location tiegħi. Għala hemm din id-diskrepanza? Xi ħadd qiegħed jigdeb?

Risposta

Din id-diskrepanza hija problema sfortunata tal-Esperjenza Umana (jew User Experience / UX bl-ingliż) tas-sistema tal-Android. Tidher biċ-ċar fl-iscreenshot ta’ hawn taħt:

L-ewwelnett, nistgħu nikkonfermaw jekk l-app għandhiex aċċess għad-data tal-location, kemm jekk hux bil-Wi-Fi (magħruf bħala Coarse Location) jew bil-GPS (magħruf bħala Fine Location).

Developer jrid jiddikjara minn qabel xinhuma l-permessi li l-app għandha bżonn sabiex taħdem. F’Android, dawn jinasbu f’file jismu AndroidManifest.xml. F’dan il-każ, nistgħu nsibuħ hawnhekk.

Il-permessi li nsibu huma:

<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.INTERNET" />

Kieku l-app riedet tkun taf l-location tiegħek, konna naraw dawn il-permessi minflok, kif nsibu miktub fid-dokumentazzjoni ta’ Android Developers:

<uses-permission android:name="android.permission.ACCESS_COARSE_location" />
<uses-permission android:name="android.permission.ACCESS_FINE_location" />
<uses-permission android:name="android.permission.ACCESS_BAċKGROUND_location" />

Ċar daqs il-kristall: Covid Alert Malta mihiex kapaċi tkun taf fejn inti. Mhux direttament għallinqas.

Mistoqsija

Mela għaliex l-app għandha bzonn is-setting tal-location mixgħula biex taħdem?

Risposta

Insa Covid Alert Malta għal mument. Informazzjoni ta’ location tista tiġi esposta jekk Bluetooth huwa mixgħul.

Jekk Bluetooth huwa mixgħul, mela l-mowbajl ħa jkun kapaċi kemm jitrasmetti u anke jirċievi data fuq BT. Apparat differenti huwa kapaċi jsib lil xulxin jekk wieħed minnhom huwa ssetjat biex jkun discoverable. Dan huwa l-proċess li jintuża meta tissetja apparat b’kapaċita tal-BT (bħal wireless headset) f’pairable mode. Il-headset jkun qiegħed jgħajjat lil-apparat kollu fil-viċinanza li huwa kapaċi jigi paired u inti tuża l-mowbajl tiegħek sabiex tagħmel pair miegħu, u b’hekk jistgħu jqassmu data ma xulxin, li f’dan il-każ jkun sabiex tisma l-mużika.

Wieħed mill-modi li apparat jista jagħraf apparat ieħor huwa mill-MAC Address tiegħu. Dan qisu l-indirizz tad-dar. Jiena nagħraf id-dar ta’ siehbi Ċikku għax naf li Ċikku jgħix f’numru 24 fi Triq il-Kbira. Bl-istess mod, il-mowbajl tiegħek jagħraf apparat ieħor li jkollu indirizz fil-forma ta’ kelma hekk: D9:79:D4:9C:C9:1C.

Jekk l-mowbajl tiegħek qiegħed jaqsam dan l-indirizz ma kull apparat ieħor fil-viċinanza, mela l-apparat jista juża dan sabiex jagħmel korrelazzjoni bejn l-MAC Address u l-persuna preżenti. Tipikament, dan jsir b’apparat msejjaħ BLE Beacon f’proċess bħal dan:

Sid ta’ ristorant jixtieq jżomm kont ta’ kemm hemm nies fir-ristorant
Is-sid jixtri BLE Beacon u jwaħħalha fuq s-suffitt
Inti tmur regolari fir-ristorant - kull nhar ta’ Ġimgħa wara x-xogħol għal-happy hour.
Il-mowbajl tiegħek qed jgħajjat il-MAC Address tiegħu lil kull apparat ieħor fil-viċinanza
Il-BLE Beacon tieħu nota tal-MAC Address tal-mowbajl tiegħek
Is-sid issa jaf li hemm persuna tiġi kull nhar ta’gimgħa, ħabba li dejjem jara l-istess MAC Address darba fil-ġimgħa fl-istess ħin.

Dan ifisser li s-sid jaf min inti, l-indirizz tad-dar tiegħek, in-numru tal-karta tal-identita’, eċċ? Le, qas xejn.

Dan ifisser li s-sid jaf ta’ min hu l-mowbajl b’dak il-MAC Address? Iva, forsi. Forsi qegħdin jaraw s-CCTV biex jaraw min jidħol mill-bieb meta dik il-MAC Address jiġi rreġistrat mill-BLE Beacon. Din magħrufa bħala attakk tal-korrelazzjoni.

Dan li ridt infisser meta semmejt li l-location tista tiġi esposta. Ma jfissirx illi li jkollok l-BT mixgħul ħa jqassam l-indirizz tad-dar tiegħek, imma tipprovdi biċċa informazzjoni zgħira, li permezz tagħha, jekk xi ħadd lest li jagħmel biżżejjed xogħol biex jiġbor iżjed biċċiet informazzjoni bħal din, jaf jkunu kapaċi jsiru jafu min hu jew hi s-sid tal-mowbajl.

Ifhem din: l-istess jsir bil-Wi-Fi. Il-mowbajl tiegħek qiegħed jagħmel riklam tiegħu nnfisu lil routers fil-viċinanzi (proċess msemmi probing), li fil-proċess jizvela il-MAC Address tiegħu, li jekk xi ħadd jzomm kont tiegħu, jaf jikkawza espożizzjoni tad-data tal-location.

Covid Alert Malta mhiex il-kawża ta’ din l-espożizzjoni, imma din hija r-raġuni li location services trid tkun mixgħula sabiex taħdem l-app, anke jekk Covid Alert Malta muhiex l-app li qiegħed jikkawza din l-espożizzjoni.

Konklużjoni

Jiena tal-parir li wieħed għandu jinstalla Covid Alert Malta għal-dawn ir-raġunijiet:

Is-source code hija pubblika għal kull min jixtieq jagħmel riċerka fuqha
Ma tagħmel l-ebda użu mill-GPS jew minn data tal-location tiegħek
Tgħin sabiex tħaffef il-proċess ta’ Contact Tracing li qiegħed jgħin sabiex tiġi kkontrollata l-pandemija tal-Covid-19
Hemm ħafna apps oħrajn li qegħdin jieħdu d-data tal-location tiegħek f’modijiet iktar ovji, bhal Facebook, TikTok, Google Maps, Revolut, eċċ. Li tkun ikkonċernat fuq Covid Alert Malta imma mbagħad ma tkunx ikkonċernat fuq dawn l-apps l-ohrajn li huma diga insallati turi nuqqas ta’ fehma fuq kemm issir frekwenti dan it-teħid tad-data.

Hemm problema ohra meta tuża l-Bluetooth il-ħin kollu li mgħandhiex x’taqsam ma sigurta’: il-fatt li jista jwaqqa l-batterija iktar malajr. Jiena naħseb li dan huwa prezz zgħir biex inħallsu għal din l-opportunita ghal rapid contact tracing li jżomm il-privatezza tal-individwu, speċjalment meta daż-żminijiet jeżistu battery packs jew quick charging.

Inti tista tniżżel Covid Alert Malta minn hawnhekk:

Grazzi kbira lil-mara tiegħi Rachel tal-paċenzja li ħadet filwaqt ċċekjat dan l-artiklu.

Reviewing Covid Alert Malta

Sun, 25 Oct 2020 10:30:00 GMT

Il-verżjoni bil-Malti ta’ dan l-artiklu jista jinsab hawnekk.

Introduction

I write this article a bit more than seven months after the Novel Coronavirus pandemic arrived onto Maltese shores. There are many aspects of this pandemic that I wish to share thoughts about, especially on the truths that it has revealed about us as a species and as a nation in socio-economic and political realms, however I will, for this article, stick to what I know best, which is the technological.

Contact Tracing in the 21st Century

The Maltese Government has released a contact tracing smartphone application. This is an application which contacts other smartphones in the person’s immediate vicinity, for the purpose of augmenting a contact tracing mechanism.

The idea is that, if someone you came into contact with tests positive for Covid-19, then the process of informing you of this contact becomes much easier, since your smartphone kept track of the fact that you were in the vicinity of a positive case.

The main issue around such a system is and always will be:

How can you preserve the privacy of an individual whilst still keeping track of them and their contacts?

The Maltese Government’s technological authority MITA has claimed that the app was developed in line with GDPR and also had it audited by the Malta Digital Innovation Authority.

Unfortunately, lack of transparency is a persistent issue in Maltese affairs - it is often difficult to get an answer as to how something was performed or to what level.

In a rare move, MITA have opted to open-source the entire stack for this application. This is the boldest and best move that MITA could ever make with the aim of fostering trust in the system.

They are literally stating that they have nothing to hide when it comes to what is going on under the hood. This enables even the most skeptical amongst us to review, audit and verify that the system is indeed preserving privacy and equipping our country to bring the pandemic under control faster.

Marrying privacy and contact tracing

Contact tracing feels like the evil twin of privacy. You are contacted by someone you do not know, about something you cannot see and you have to share where you were and whom you were with within the past few days, in as much detail as possible.

Despite this, it turns out that contact tracing is super important for containing a pandemic, so despite the seemingly inherent privacy nightmare, we do it, so that maybe, just maybe, we can put all this pandemic behind us a little bit sooner.

On closer inspection however, the exact time, date and place you were within proximity to someone is not as important as exactly who you were within proximity to (given parameters such as whether you were within two metres of each other, for longer than fifteen minutes and without any sort of face covering).

Wouldn’t it be great if we could share the information of who we were within proximity of, without going into the other details of where, when and possibly why? Bonus points if it could include people we do not even personally know.

Another issue is that our human memory is not particularly robust or reliable. We forget details, we mix up their order, we perceive time to be longer or shorter than it actually is. That chat you had with your neighbour at the grocer? It felt like five minutes but you actually spent twenty minutes. Wouldn’t it be great if we could recall details of our interactions with the highest precision?

It turns out, thanks to a collaboration between the greatest minds at Apple and Google, most modern smartphones (those supporting Bluetooth Low Energy (BLE) and running Android 6 Marshmallow and above or iOS 13.5 and above) now have Privacy-Preserving Contact Tracing capability built into their respective Operating Systems.

Article Outline

This article will focus heavily on the specific implementation that MITA have released that hooks into the Apple-Google Privacy-Preserving Contact Tracing system rather than the PPCT system itself. The aim is to challenge and verify the specific claims that it does not gather any sort of location data of the individual running it.

I will also follow a simple Question and Answer structure throughout rather than providing an FAQ at the end.

Covid Alert Malta

Let us start with an explanation of what open-source is.

Software which is open-source is software for which the program’s code is available for viewing and compilation, whilst closed-source does not have the program’s code disclosed, and a user can only make use of the software in a pre-compiled manner (like Microsoft Word) or as a service (like Google Maps).

With open-source software (often abbreviated as OSS), we can verify behaviour either by running the program or by reading the source code, since there may be behaviour which occurs but is not disclosed by the program. In closed-source, we can only observe the behaviour of the program by running it, and we have to rely on what we are told it does (or does not) do.

If I had to provide an analogy, a closed-source choclate cake is the kind you buy that is ready to enjoy. It looks delicious and tastes delicious, but you have no idea how, when, where or with what it was baked. The baker may assure you that it is gluten free, vegan or was baked in a nut free environment, but the only assurance of truth in that claim here is to take the baker’s word for it.

An open-source cake would be if the cake came with the recipe listing precisely the ingredients, their amounts, their source and the process of baking. We can observe or eat the cake and not care about the recipe, but we can also use the recipe to bake our own version of the cake, either to mix it up with our own preferences (such as replacing the chocolate frosting with jam) or to see if we end up with the exact same chocolate cake that has been pre-baked for us.

In this case, the Covid Alert Malta Android and iOS apps, along with the server which handles exposure notifications are forks (the equivalent of a software remix) of other open-source Covid-19 contact tracing apps, specifically the SwissCovid App.

Question

Does this mean that MITA stole the SwissCovid app and claimed it as its own?

Answer

Not at all. The specific open-source licence under which the SwissCovid app is made available allows for modifications and re-distribution (more about this later on).

Question

Why didn’t MITA write one from scratch?

Answer

I do not speak for MITA so I cannot answer this in their name, however I can speculate.

Before writing any software one should evaluate the feasability of it, both in terms of financial and time investment. Writing a contact tracing app from scratch is an option, however the time and knowledge investment required can be very high. Taking into consideration the fact that the sooner you release a contact tracing app the better in the case of a pandemic, any large time investment becomes cost-ineffective.

It would be much more feasible to find a pre-existing open-source implementation and adapt it to your specific needs. One of the main ideas behind Open Source is to avoid double-work. Code is shared in the spirit of collaboration and so that everyone benefits from not constantly re-inventing the wheel.

Question

Is MITA obliged to share the source code?

Answer

Yes. (Nearly) all open-source code comes with a licence, which specifies the duties an individual or organisation must assume in order to make use of the code. In this case, the SwissCovid app is licenced under the Mozilla Public Licence 2.0. Quoted directly from the FAQ linked:

Q9: I want to distribute (outside my organization) MPL-licensed source code that I have modified. What do I have to do?

To see the complete set of requirements, read the license. However, generally:

You must inform the recipients that the source code is made available to them under the terms of the MPL (Section 3.1), including any Modifications (as defined in Section 1.10) that you have created.

You must make the grants described in Section 2 of the license.

You must respect the restrictions on removing or altering notices in the source code (Section 3.4).

Question

Does making the source code available make the app less secure?

Answer

The security of any software is not contigent on whether it is open-source or not, but rather it is contingent on the quality of the software itself.

Creating insecure software and choosing not to disclose the source code is the equivalent of baking a cake but not choosing to disclose whether it contains gluten or not. Regardless of disclosure, the amount of gluten does not change and can still cause mild to deadly health issues on consumption of the cake if the consumer is gluten intolerant or a coeliac.

Choosing not to disclose source-code on the basis of security is a method known as security through obscurity, which is highly discouraged by any professional working in Software, Security or Information Systems.

Question

Wasn’t this App reviewed and certified by the MDIA?

Answer

One would hope that software such as Covid Alert Malta, which in an ideal world is running on every single smartphone in the country, would have gone through a rigorous security review.

To this end, the MDIA has claimed that the app passed their certification process, however this certification process is not public or transparent. Blind trust in any institution should be avoided since at the end of the day it is still made up of humans. I highly encourage the MDIA to share how and to what level certification was ensured, especially as this will continue to build trust in the system, which would drive even more installations.

Regardless of the above, open-sourcing the app allows anyone to perform their own audit. Regardless of your level of trust in any institution, whether you are more lax or more pedantic, the fact that the source code is right there for review will enable you to put your own seal of approval on it.

The Source Code

Full Disclosure: My expertise and experience lies mainly in cross-platform apps. I do not write apps in platform specific languages (such as Java, Kotlin, Objective C or Swift). Regardless, I have prior experience in Java and apps in general which I consider sufficient enough to perform this review.

The source code can be found at the following URL: https://github.com/GOVMT-MITA/dp3t-app-android-ch

Given that it is a proper fork, we can actually compare the differences between the original repository and the MITA one. This diff, short for difference, is an auto-generated list of differences on the level of each file.

From a quick skim, it looks like the only major addition to the app was the option to choose language when onboarding and localisation of the app due to the chosen language. There does not seem to have been any change in the behaviour in the app from what SwissCovid provided.

Question

The App keeps insisting that I turn on location settings. Everyone insists that the app never tracks my location. Why is there this disparity? Are they lying?

Answer

This disaprity is an unfortunate User Experience (UX) issue within the Android operating system, encompassed within the screenshot below:

First of all, we can confirm whether the app has access to location data via Wi-Fi (known as Coarse Location) or via GPS (known as Fine Location).

When it comes to Android (this is also the case in iOS), a developer must state up-front what permissions the application requires to function. These are stated in the AndroidManifest.xml file. In our case, we can find it here.

The permissions listed are:

<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.INTERNET" />

If the App wanted to track your location, we would see the following permissions listed, as outlined in the Android Developers Documentation:

<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_BACKGROUND_LOCATION" />

Plain as day: Covid Alert Malta is not able to track your location. Not directly at least.

Question

So then why does the app need location setting turned on?

Answer

Forget Covid Alert Malta for a second. Location information may leak if Bluetooth is enabled.

If Bluetooth is enabled, then your phone is going to be able to both transmit and receive data over BT. Devices are capable of finding each other if one of them is made discoverable. This is the process used when setting a BT device (such as a headset) in pairable mode. The headset shouts to all other devices in the vicinity that it is available for pairing, you use your phone to pair with it and they can now exchange data, which in this case would be audio.

One of the ways devices can recognise each other is via their MAC Address. This is similar to a postal address - I recognise it is John’s House because John lives at number 24 and the house infront of me has number 24. Similarily, your BT device will have an address in the format of D9:79:D4:9C:C9:1C.

If your device is sharing this address with any other device in the vicinity, then devices can use this to correlate your location and have a vague idea that you are the same person. Typically this is done through a device such as a BLE Beacon in a process such as:

Restaurant owner wants to track how many people are in his restaurant
Owner sticks a BLE Beacon above the soffit
You, as a regular, go there every Friday evening for happy hour
Your phone is screaming out its MAC Address to every device in the vicinity
BLE Beacon takes note of your MAC Address
The owner can know that they have a consistent regular every friday night, based on the fact that the same MAC Address turns up at the same time every week

Does this mean that the owner knows your full name, address, ID Card number, etc? No, not at all. Does this mean that they can know which person has that MAC Address? Maybe. Maybe they are watching their security camera footage to see who walks into the door when the MAC Address is registered by the beacon. This is considered a correlation attack.

This is what I meant by location data may leak. It does not mean that having BT enabled will share your home address, but that there can be a nugget of information which, if sufficient nuggets are gathered, can begin to indicate who owns that device.

Get this: the exact same happens with Wi-Fi. Whenever your device is advertising itself to possible routers (in a process calling probing), it reveals its MAC Address, which if you keep track of, can cause location data leakage.

Covid Alert Malta is not the cause of possible location leakage, however this is the reason that location services are required to be enabled, even if it is not Covid Alert Malta which is causing the leakage.

Conclusion

I greatly advise towards installing Covid Alert Malta for the following reasons:

The source code is right there if you want to audit it
It does not actually make use of your GPS or any sort of location data
It will help speed up contact tracing, which should help bring the current Covid-19 outbreak under control
There are many other applications which are siphoning off your location data in more egregious and obvious ways, such as Facebook, TikTok, Google Maps, Revolut, etc. To be concerned about Covid Alert Malta but not about all the other apps that are installed which actually request location permissions shows a fundamental misunderstanding of how widespread data siphoning is.

There are non-security concerns relating to running BT constantly, such as the fact that it can indeed cause higher than expected battery drain. I think that this is a small price to pay for a privacy conscious way of performing rapid contact tracing, especially in the age of battery packs and quick charging.

You can download Covid Alert Malta from the following links:

A real-time game from scratch - Initial Setup [1]

Sun, 04 Oct 2020 19:00:00 GMT

Previous Entries

0 - Motivation

Introduction

So I finally got started on Cabin Fever.

Given that I have chosen to not proceed from an academic route where I learn all the theory then I proceed with the practice, the easiest way to make initial progress is to setup the repository and a Hello World for both the back-end (BE) and front-end (FE) part of the project.

I will not be detailing progress in a step-by-step reproducible fashion, instead I will gloss over details which I feel are not relevant, instead I will focus on the high level concepts. Despite this, the resultant code will be tagged with the number of the article in the title (in this case, the tag will be article-001).

The Repository

The repository is hosted on GitHub under my personal account. I chose the GPLv3 licence since, at the time of writing, the aim of the project is more education focused over commercial.

The folder structure is a very simple monorepo - cross cutting concerns (readme, licence, eventually CI and relevant config files) are stored in the root, whilst BE and FE have their own folders: cabinserver and cabinclient respectively.

The Server

Inside the cabinserver folder I placed a main.go file to allow me to simply run go run main.go.

When it comes to Go & web servers, there is not a decent built-in option to allow for hot-reload development (not in the way one can come to expect in the Frontend world). To achieve this, I use an open source tool called modd (also written in Go) which according to my provided config will watch the filesystem for changes to *.go files and rerun the provided command.

The Client

On the client-side, I chose Parcel as my bundler. I could have used alternaatives such as Webpack, Fuse, or sticking with the plain TS compiler, but I have found parcel to be one of the best bundlers which requires the least/zero config. Within two yarn add commands, I had functioning typescript compilation along with hot-reload and source-map generation.

To ensure that the above works, I simply added an index.html file which runs a TS file, which simply logs Hello, World to console. The “Hello, World” is stored in a strongly typed string, to ensure that the TS features are working as expected. I have currently even gotten away without having to define a tsconfig.json file, however in my experience at some point I will need to definitely provide it.

Linking them together

The simplest and most standard way to send data back and forth would be to use HTTP in the form of REST API using JSON for serialisation and de-serialisation. Whilst I Would definitely not build a real-time game using this option (a turn-based game would most probably work just fine), I will use it to verify that my client and server can at minimum talk to each other on my machine.

The basic setup for now is that on GET to route localhost:8080/, the server returns a string containing the current time. The client is hitting endpoint / once a second, then updating the DOM with the string returned by the server, which contains the timestamp. One can also confirm it is working by looking at the network tab and by seeing no errors in the console.

Conclusion

The next step would be to explore a more event-driven approach through the use of a technology such as Web Sockets. I have experience using SocketIO in Node to simplify development, however since I am now using a non-isomorphic JavaScript stack, it will require going back to basic WebSockets.

You can find the code for this article at the article-001 tag.

Next Entry

2 - WebSockets

Tempus - Rethinking what to Track

Sat, 03 Oct 2020 17:30:00 GMT

Introduction

I am writing an experimental time tracker for software development workflows. It is called Tempus and you can see more information about the motivation behind it in my previous article and the source code on my GitHub.

I have been using the first iteration of Tempus since the 14th of September (nearly a month at time of writing). I am even using it right now as I write this article, as I use it to keep track of how I spend my time on my various side-projects.

In this time I have realised that if I am going to write a time tracker specifically for software development workflows, I cannot write a time tracker which is essentially the same as any of the existing options and state that its for devs simply because I am a software developer myself suffering from Not Invented Here Syndrome. This article is about my thoughts on why one-size-fits-all time trackers aren’t sufficient and how I plan to remedy it in the next iteration of Tempus.

The Issues

I split my Tempus tasks in a fashion similar to the following:

Slack Chats (meant to refer to IM in general, as I tend to use Skype and Teams too)
On a Call
Writing Code
Thinking Time
Architecture
Code Review
Stand up break
Debugging
Dealing with external providers
…etc

Part of what aids adoption of any system is a well designed User Experience (UX). If the UX is not thoroughly planned for, system’s success tends to be despite of its other characteristics rather than because of them.

This is the reason why public transport adoption is better in some countries over others - if the UX of public transport in Malta sucks (and it does), and the only alternative is to drive your own personal car (even with its accompanying issues with parking and cost), then the adoption of public transport will be low.

Any solution to this that does not address the basic UX issues (timeliness, availability, frequency, reliability, etc) is simply a micro-optimisation to a macro problem.

In using Tempus, I (as a User), eXperienced the following main issues:

I often do not know which task to choose. I have a task for being On a Call. I also have a task for Debugging. So if I am debugging a production issue whilst also on a call with a colleague, which one should I choose?
Some context is being lost. I could be Debugging in either our development environment (such as: the build was successful on my machine but not in our CI pipeline), or our production environment (such as: diving into incoming logs to diagnose an urgent ongoing issue)
I would often forget to switch tasks once I have completed them (I would sometimes remain On a Call for over 15 minutes after the call concluded)
I do not have a clear idea of whether I was hitting my ideal time spent ratios. If my current position or the current week calls for a 50:50 split between writing code and working on our architecture, the only way I could reliably know that I was reaching that ratio was if the only two tasks I used were Writing Code and Architecture. Since there were other tasks interleaved with missing context (such as On a Call, which may have been either writing code or architecture), I could not reliably track my progress.

The Solutions

Issues 1 & 2 - Choosing between tasks & lost context

The idea so far to solve these issues is to double down on characteristics unique to software development workflows. After all, Tempus is meant to be a niche tool so I should design and develop it with that aim in mind.

Instead of tracking tasks I suppose that tracking the mode that the task is being performed with. The UI look something like this, where each option is a button. Choosing any option would deselect the other option in the row (behaviour similar to radio buttons).

Mode	Option A	Option B	Option C	etc
Physical	Sitting	Standing	AFK
Workplace	Home	Office	Co-working Space	Other
Task	Developing	Debugging	Documentation
Environment	Development	Staging	Production
Communication	Quiet	On a Call

In the initial idea of Tempus I thought about supporting concurrent tasks, in an effort to support a structure similar to the above, however I decided against it in the interest of simplicity. The difference between free-text concurrent tasks and the above is that one would have to define each mode and the possible values of that mode. It would also make sense to somehow link the values of different modes together via simple rules - for example if I set myself to AFK, then Tempus could remove the selected value for the Task, Communication and Environment modes, whilst retaining the values for Workplace.

Issue 3 - Forgetting to switch tasks

This issue is not particularly addressed by the redesign and would be better addressed by the introduction of some sort of hook mechanism - one such as “if I stand up from my seat, hit the Tempus API to switch my Physical mode to the standing value”.

In the mean time, I would add functionality to change the start and end time that a specific mode’s value was active - for retro-active repair of incorrect data.

Issue 4 - Reliably tracking my progress

The current version shows me a very basic breakdown of how much time I spend on what task. I setup a very basic chart component which literally consists of a div element with a fixed height, max width and then the actual width is set on page render (width: {{ .Task.Percentage }}%).

Here is a screenshot of the breakdown of all my tasks on my production Tempus instance:

To my surprise, 30% of my time is spent solely in some sort of active, synchronous communication. Surprise aside, this only tells me how much (percentage of total) time I spent on a task - it does not tell me whether I am spending either too little or too much time on a specific task type. It would greatly help if I could set daily/weekly targets of, say Not more than 40% of my time should be writing code, to avoid getting stuck trying to optimise my code too much instead of sticking to good enough, alongisde at least 20% of my time should be standing up, to help combat the health issues that come with a sedentary style of work and too much sitting down, encouraging me to make use of my standing desk more.

Conclusion

In conclusion, it looks like the Tempus project is already bearing fruit - helping me flex my UX Design muscles alongside my effort to learn more Go, as also outlined in my previous article about learning to write a real-time game from scratch.

A real-time game from scratch - Motivation [0]

Thu, 01 Oct 2020 17:00:00 GMT

Introduction

In my day to day work, the software I write and architect consists of the standard stateless request & response web server kind. Not particularly exciting or complicated in terms of fundamental structure, although it presents various challenges when running at a certain scale.

In the interest of continued professional development, I have started on the path to writing a real-time game from scratch.

Motivation - Why?

I have been writing programs in Go for a few years now. I love the language and its simplicity, however the applications I have used it for are also quite simplistic - typically REST APIs, small CLI tools and a web scraper. There have been little to no opportunities to use the more sophisticated features of Go, such as its first class concurrency: Go routines and channels, particularly because I aim to pick the simplest tools for the job and the majority APIs I have written have never needed any complex concurrency requirements.

Along with Go, this would also challenge me on my Front-end skills. Most of the FE work that I do is within the context of a framework (typically React) or some minor vanilla JS to add some dynamic interaction to a server side generated web page. A real time game however would require a completely different approach, using tools such as WASM, Canvas or possibly even WebGL to ensure that the refresh rate of the game is as fast as possible.

Another reason is that the more time I spend at my full time job thinking at an architectural level of abstraction, the more I realise the importance of choosing the correct data structures and how to process them, a concern which is less important when you are knee deep in the code. My main concerns are typically how can we reduce the coupling within these two services to avoid development time slowing down or what is the ideal way to upgrade this legacy method of service communication to keep up with our load demand?. These concerns are very different from how do I ensure that, despite unexpected latency from some clients, the game can still run its core loop without blocking all the other clients’ inputs?.

One more reason is that I want to learn how to model systems that are beyond the standard CRUD plus some validation / business logic. A real-time game needs to take several inputs at the same time, ensure that the input is valid, update the game state, synchronise it to all clients and do so fast enough that the game actually feels realtime, despite varying latency, computational power, network availability, etc, all this within a stateful manner.

Finally, I have always found games to be a great entertainment and education tool. Ever since I became a father, being able to sit down for a 3 hour stretch of gaming has become a once in a blue moon treat, causing me to prefer games I can quickly jump into and out of whenever I get a 20 minute window whilst the baby is asleep and all the chores have been finished and I have already spent quality time with my family.

In summary:

Improving my knowledge of Go and its concurrency features
Challenging myself to learn a method of Front-end Development that takes real-time into consideration
Improving my understand of low-level performance and optimisation
Being able to architect a stateful, real-time system which caters for multiple clients
Creating a game that has a short play time and is easy to pick up and drop out of

The Approach - How?

I could plausibly hop onto YouTube or Udemy, follow a game development course and simply copy paste my way to a finished game. Whilst that was a very valid way for me to learn when I did not fully understand what was going on at a code level and allowed me to bootstrap my software career, my aim now is to improve my existing skills, not only in writing the code but also understanding what that code is doing under the hood. Often enough I have spent my time learning the theory, only to come to apply it and get stuck.

Over time I have come to believe that the theory and practice of a topic cannot be divorced from each other, and at least personally, my learning style prefers to start from the practice, learning bits of theory on the way, getting something working, then investing time to seeing whether knowing the full theoretical story could have produced something more practical. In a way, I am taking the agile approach to learning - knowing just enough to get something working, then iterating on that. Adding onto this - I could use tests to confirm my understanding of the theory. If I write a calculation for acceleration on paper, does my answer return the same when I write it a test for the function?

In summary:

Use my existing knowledge to create something working
Document the process, outcomes, areas of improvement
Investigate whether there exist patterns that can improve on the bottlenecks / issues
Refactor existing code to adapt to new found understanding
Repeat until the game can be considered complete

What kind of game?

I have always been an avid fan of Elite Dangerous and space games in general (such as KSP). Inspired by Elite Dangerous Arena, I will be creating a 2D space game, where the aim is to compete in some sort of Deathmatch / Team deathmatch / CTF setup.

This type of game is also interesting as it can involve a simple physics model - namely frictionless movement within a vacuum without gravity, at least initially. It could be interesting to involve elements such as gravitational pulls of objects within the arena, which also affect projectiles. Areas of concern in terms of design would include edges of maps - should they wrap around? Or somehow penalise players who exit the arena bounds?

Conclusion

I do not expect progress to be fast, especially given my packed schedule, but it should be a great learning experience, both for myself and any of my readers who have never delved into games development before, especially if they are coming from a more standard/corporate software development background rather than a pure game development background.

Any and all code I write will be made open-source, under the GPLv3 licence at https://github.com/simonamdev/cabin-fever

Next Entry

1 - Initial Setup

Introducing Tempus

Fri, 18 Sep 2020 20:00:00 GMT

Introduction

Time tracking can be a touchy subject, especially when it comes to Software Development. Often enough it is associated with either tracking time against an estimate of a task or time spent for billing purposes.

The former can cause undue stress and anxiety when a task is still ongoing yet the estimate has been exceeded. The latter can cause undue issues with clients disputing the amount of time that was actually spent.

Regardless of having experienced both of the above in the past and not being the biggest fan of time tracking, over the last year I have found myself tracking the number of hours that I worked in a given month in a Google Keep note.

The Process

Every day for the last 14+ months I have been taking down the time that I:

Started the workday
Stopped for my break
Stopped/Started for any reason (such as doing an errand mid-way through the work-day)
Completed the workday

At the end of every month I would manually calculate the actual number of hours I have worked so that I can adjust to work less/more hours in the next month. After having done this process manually for three times, my developer senses started tingling that this could be easily automated.

Productivity

Recently my role at $WORK has become more vague, causing me to spend a lot of time switching between different areas of concern depending on the priority assigned by the business. I prefer long uninterrupted streches of deep work (up to an hour or two), I find context switches very mentally taxing so this has been a learning experience for me on how to juggle several different contexts.

To this end, I wanted to start to keep track of context switches. This data will eventually lead to a better understanding of when most switches occur or what kind of tasks cause more switches than other.

To clarify, for me a context switch refers not to the type of task but rather the topic / feature being addressed. Switching from a chat over slack with a colleague, moving to a call with that same colleague whilst opening my code editor as we discuss and evaluate a new feature does not involve any context switches. If I am focusing on one area of our system and a colleague asks for a call about a different area, then a context switch occurs.

The Solution

The solution is Tempus. Tempus is my experimental time tracker, mainly aimed at software development and related workflows.

My requirements were simple:

Be able to create a project
Be able to create any kind of task for that project
Be able to start and end a task within a project
Be able to switch from one task to another, whilst also recording if I am making a context switch
Aggregate task data within a project to understand my time investment better (how many hours have I spent today? How many times did I switch contexts?)

The Implementation

To build Tempus, I used the simplest tools I could think of that would allow me access from any device. It is a Web App which is fully server side generated (that is, no JavaScript or AJAX style API calls are involved). Creating Entry types, starting or stopping entries and recording context switches are simple HTML forms which perform POST requests. I used Go as a programming language and SQLite for data persistence, I upload and host the binary on a Digital Ocean VPS behind nginx as a reverse proxy, using letsencrypt for an SSL certificate.

To get an idea of the simplicity, here is a screenshot of tempus in its current state:

The Future

The current iteration is simply for me to try out features as they begin to make sense or as I spot inefficiencies in my usage. One example I quickly spotted was that the project page lists all of the entries under each other (with no sort of filtering or pagination). After the first day of usage I had already clocked in twenty-five unique entries.

Since this list kept on growing, the buttons to choose a new entry type were getting pushed below the fold, which meant I needed to scroll down every time I wanted to switch tasks. To remedy this, I moved the buttons and the current task indicator above the list of entries.

I also see myself changing the front-end to a React-based Single Page Application. This would enable interactions such as a long press to indicate a task switch with a context switch, as well as make charts/analytics super simple to show on screen.

Given that Tempus is aimed primarily at Software Development workflows, it would also make sense to have an easy to use CLI, given that I spend more time living inside a terminal over the browser. Even if I were in a browser, I would need to find the relevant tab again, whilst a terminal is typically a hotkey away. I envisage it would have an API such as tempus start 'Programming' or tempus switch 'Documentation' --context-switch.

Conclusion

Worst case scenario, I learn that I do not need a time tracker or any time tracker could work. I also improve my Go and product development skills. Best case scenario, I create an awesome time tracker. I also improve my Go and product development skills.

Either way, win-win.

Tempus is open-source and is licensed under the MIT Licence. You can find the source code here on GitHub.

Zero to Setup on Ubuntu

Sun, 13 Sep 2020 18:00:00 GMT

Due to a recent purchase of a new laptop (followed by not one but two guarantee covered repairs) as well as refreshing some old hardware, I have had to (re-)install Ubuntu (my OS of choice if the aim of the machine is anything but gaming) more times than I would care to enjoy.

Each (re-)install required the installation of numerous tools which I would need in the course of any software development I perform. In the first two iterations it was a case of downloading and installing the tool whenever I realised that I had not done so yet.

The third time round I realised I should make a list of the tools and install them all upfront.

git

With all of my code hosted on GitHub, I need the git CLI to retrieve and push code.

curl

Since most of my work is Web Development, https://curl.haxx.se/ is an essential CLI tool to quickly check the response of a webserver running on my machine or the response of a remote website/API that I am interacting with.

vim

Whilst not my main text editor, I occasionally want to edit a file quickly from the CLI. vim is my CLI editor of choice. Learning the basic of vim have also saved a good chunk of time when I needed to quickly edit a configuration file on a headless server.

ZSH

ZSH, which stands for Z Shell is an alternative shell to Bash. ZSH along with the plugin framework oh-my-zsh give my terminal superpowers. Completion for various tools, colours in normal tools such as ls, even showing the current branch of the current folder’s git repository.

Visual Studio Code

I used to be an Atom Editor fan which was the free (and slow) alternative to Sublime by GitHub. VS Code however has taken the web development world by storm. With great performance (even for an electron based application) and an ever growing plugin eco-system which can fit nearly anything you throw at it, it has been my daily driver for years.

Bitwarden

In an effort to be more sercure when it comes to passwords, I have moved to using Bitwarden. With a very generous free account option along with being open source, it is the password manager of my choosing.

Docker

I used to be a docker sceptic. Admittedly I started learning how to use docker around the time that it was just getting into vogue, before cloud vendors went all in on containers or when orchestration was still a mostly unsolved problem (not that it is completely solved now, but that is a different story).

But after one too many times of spending more time trying to get a Python script running on a VPS over writing the actual script, the ability to just upload a tarred image and then run a container from it with all dependencies it needed tagging along, it graduated to one of those tools that I wonder how I got any work done before I started using it.

Along with docker I also install Docker Compose

Programming Languages

My go-to stack reflects the fact that I am mainly focused on web development. At this point I am installing:

Go
NodeJS, along with npm and yarn
Python3, although this is typically already installed in the distro that I use

After each one I also enable zsh support and VS Code extensions.

Ulauncher

Ulauncher gives a macOS spotlight like experience to Ubuntu. A great addition to enable the transfer of muscle memory from macOS to Ubuntu

Postman

Postman is a GUI for API development. It is a great tool to refer to when manually crafting a curl request is too daunting.

htop

htop is a terminal based process viewer. It comes with pretty colours and more features over the built-in top. This is useful if coming from a histoy of Windows machines, where Task Manager was always open.

sqlite3browser

Sqlite3Browser is a GUI for interacting with an SQLite Database. I love SQLite due to its simplicity in the way that it creates a flat file that includes all the data relevant to your task. When getting started with any development idea, I always start with SQLite then move to a bigger solution if the idea graduates to a project.

Conclusion

At some point I will probably have to repeat this process as I do experience a few issues with my current laptop that I am associating with LVM and full disk encryption. I forsee the next evolution of this process being the use a tool such as Ansible to automate the process, or possibly using docker for the more often used tools to avoid local installations completely.

O-Level Grading UX Issues

Thu, 14 May 2020 21:30:00 GMT

Within Malta’s Education system, as a secondary school student, you are constantly working towards the O-Levels, which are examinations set by the government’s Ministry of Education, which will award you with the SEC Certificate you need to be able to continue towards higher education.

Grades in the Maltese Education System

Famously these exams award you with grades that are defined by the following: 1, 2, 3, 4, 5, 6, 7 and U.

1 is the highest grade, 7 is the lowest
1 through 5 are considered Passes
6 and 7 are considered Failures
U indicates that the exam was unable to be marked

Covid-19

Due to the Covid-19 pandemic, the inital sitting of the MATSEC exams usually held in April/May was cancelled. Going forward, students have been given the following options:

Sit for their SEC exams in the September Sitting (typically used for resits of the April/May sitting).
Be assessed by the Board based on their mock exam results (typically run by schools, for the purpose of preparing students for the actual SEC exam).

This poses two main issues:

It is unclear so far (in May) that the September sittings will actually be held, due to the dynamic nature of the pandemic. If they are not, then students may lose their one option to be assessed this year.
The student misses on their opportunity to take a resit. I personally have benefitted from resits to help me realise that I was over-confident or under-prepared for the subject I was being examined on

Whilst the latter seems to be a more viable option, its implementation has created a number of UX issues which will continue to plague students, teachers, educators and administrators for a number of years to come, as I will explain throughout this blog post.

According to the following Times of Malta article on the matter, the students who choose the latter option will be awarded “predicted levels” in the following manner:

Level 3 indicates that the student is at the “expected level” (equivalent to grades 1 through 5)
Level 2 indicates that the student is “nearly at the expected level” (equivalent to grades 6 and 7)

When I read the above article, it was quickly obvious to me that whoever or whichever comittee came up with this scheme did not apply any human centric design thinking to this problem. A number of issues are apparent:

Problem 1: Number re-use

By re-using numbers, it will cause confusion in both conversation and administrative tasks. If the response to “What grade did you get?” is “I got a 2”, this no longer means that the student passed and scored quite well, now it can now ambiguously mean that the student failed.

Also consider a simple administrative task: a headmaster requests a report of the distribution of marks attained by the school’s students in a specific subject. If an administrator, using a tool such as Excel, performs the following query: =COUNTIF(A1:A10, "=3"), without taking a separate indicator into consideration that the student opted for a predicted grade, the report will now be factually incorrect.

Problem 2: The use of the predicted marks is inconsistent with the standard marks

In the brain of every teacher, parent, student or administrator there is the following thought: a smaller mark is a better mark. Since 1 is the best mark and the 7 is the worst mark, the smaller your grade, the better your score.

The predicted marks however, follow the opposite system. Marks 1 through 5 are considered Level 3, whilst 6 and 7 are considered Level 2. In the predicted mark system, the smaller mark means that you “failed”, whilst the larger mark means that you “passed”.

Problem 3: The predicted marks are coarser than the standard marks

O-Level grades are often used to differentiate between one student application and another within certain competitive sixth-form instiutions. By making the marking coarser, one cannot tell between a student who would have scored a 5 or a student who would have scored a 1, only that the student “passed” or “failed”.

Proposed solutions

Maltese are known for complaining without wanting to help make the issue better, so here is my contribution on how I would resolve the issues mentioned above:

Solution 1: Use new, unique marks for the predicted levels

By using unique marks (for example: E for Expected Level and N for Nearly Expected Level). The pandemic is an extenuating circumstance, so why not consider extenuating marks? This will avoid:

Confusion between Level 2/3 and the actual grade 2/3
Providing historical context. If a student has an E grade, everyone would know the year when the student had their exam

Solution 2: Avoid providing coarse marks in the first place

From the article, it appears that the MATSEC board is going to be using mark schemes provided by the teachers who would have created the mock exams. If a marking scheme has been provided, a solution would be to mark the exam, then grade the students on a bell curve. If the exam is deliberately easy to allow students to pass or deliberately difficult to attempt to prepare students for the SEC exam, normalising in this way would reduce that bias.

The coarse “pass” or “fail” marks have to be determined somehow. It may simply be that if a student scores more than 50 on 100, then they receive Level 3, otherwise Level 2. This lack of transparency does not inspire confidence that the marking will be fair.

Conclusion

In conclusion, I hope that this article would help bring awareness that, without applying user centric design thinking, even our non-technical or software systems will end up causing extra work or increase the possibility of mistakes which could easily be avoided. If you are in a position of determining any sort of policy or regulation, consider seeing it from the user’s perspective before going ahead.

Kudos to Joseph Trapani for proofreading this article before publication.