janaka.dev

Side Project Intro - EasyRunner

Thu, 03 Jul 2025 00:00:00 GMT

Ever shipped a side project, crossed your fingers, and prayed the infra bill wouldn’t explode overnight?
Yeah, me too. That moment sparked EasyRunner.

EasyRunner.xyz

EasyRunner is a single-server, self-hosted Platform-as-a-Service aimed at indie hackers and solopreneurs building SaaS products. It guarantees no surprise bills and avoids lock-in.

The first release is a CLI that configures an Ubuntu VM (or VPS) on any cloud provider for painless app deployment. Communication and configuration happen agent-lessly over SSH. The hosting stack is Caddy (reverse proxy) and Podman (container runtime). The CLI itself is written in Python.

I’ve been working on EasyRunner on and off since the end of last year. It’s already usable, but a few rough edges need tidying up before launch.

Origin story — scratching my own itch

This project grew out of challenges I hit while building Docq.AI. Our monolithic app relies on an embedded SQLite DB and a handful of ML models, so Azure App Service wasn’t an option — we needed directly attached disk.

Spinning up an Azure VM, automating the infra, bolting on App Gateway … it reminded me that cloud still feels like pulling teeth. Provisioning VMs and load balancers is one task; installing Podman/Docker, hardening the box, and wiring up CI/CD is another.

What I really wanted

The Developer Experience (DX) of Vercel, Render, or Railway plus the portability, cost predictability, and architectural freedom of raw VMs. Direct-attached disk? ARM CPU? Custom networking? No problem.

Hosted PaaS products impose plenty of hidden constraints — you rarely notice until you slam into them. More options mean more room to manoeuvre.

Side-project reality check

Most side projects never need complex, highly available clusters. The trick is staying alive long enough to find traction, without a heart-stopping invoice. Cheap VMs fit the bill: simple, versatile, and you can multi-tenant several apps on one box.

Existing options didn’t click

Some competitors aren’t fully open-source, others lean on Kubernetes, or they try to solve every problem and end up bloated. I’m laser-focusing EasyRunner on hosting the apps you’re building — not your media server or home automation rig.

Licensing & next steps

EasyRunner is currently closed-source. I’ll ship it under a perpetual per-server business licence: buy once for major versions or subscribe for ongoing updates. If revenue clears a healthy threshold, I’ll share a cut with the open-source projects EasyRunner stands on.

Stay tuned — and maybe, just maybe, you can skip that next infra-induced headache.

Goto EasyRunner.xyz and join the waiting list to show your interest and support.

Arduino Desktop Clock - UK Daylightsaving Algorithm

Sat, 09 Apr 2022 00:00:00 GMT

A couple of years back I built this digitial desktop clock. It was mainly for fun but it was July 2020, we were mid pandemic and working from home. I was using a little Casio wrist watch as desktop clock. It sort of did the job but the digits were small and wasn’t readable at night, I like dim lighting. So that was my “real” use case.

I’d been hacking around with Arduino for a few years before this. At first using the original boards but then I came across the ESP boards. First the ESP8266 which was great but the network stack was limitd. For example didn’t support the newer versions of TLS so working with the likes of AWS IoT wasn’t great. IIRC memory was also too limiting for use cases making external calls. Then came the ESP32 which is the platform I’ve been using lately - I’m still using the original release there have been a few iterations with more features. This clock is the only hardware related side-project I have going for the last couple years.

I’ve been hacking on this clock project here and there. To be honest the only time I needed to update it’s firmware is to adjust the daylight saving offset. After doing it three times manually I decided to automate it. The simplest solution would have been to add a physical switch to toggle. But I decided I wanted to take the relatively more complicated route just for the challenge.

So what’s the big deal? I hear you ask.

The ESP32 doesn’t have a built-in RTC and my design doesn’t use an RTC module. I cannot remember why exactly but I suspect that on the weekend I decided to build I wanted an initial version up and running within a few hours so used hardware and code I already had to hack - it’s still held together by a breadboard and jump leads as seen in the photo :). Time is sync’d from Internet time servers on boot up and then once per day to handle drift. So there’s no daylight saving functionality built into the hardware or standard libs. I also have a few requirements: 1) overall optimise for minimal power usage 2) minimise third party libs, the ones I found seem to do more than just calculate if it’s BST or not. 3) minimise external network calls to save power and avoid the extra service binding code. Really it didn’t seem worth an service call for something that should be fairly simple to handle locally.

Honestly, if I found code on the web I could copypaste, I would have. I didn’t find something obvious so “invented” it myself. Highly likely there’s a more efficient way of doing this but this does the job as a first iteration. One day I might optimise, again just for fun.

Here’s the snippet. See the repo for complete code and details.

unsigned int tm_mon = (unsigned int)ti.tm_mon; // month 0=Jan,1=Feb,2=Mar,3=Apr,4=May,5=Jun,6=Jul,7=Aug,8=Sept,9=Oct,10=Nov,11=Dec
  unsigned int tm_mday = (unsigned int)ti.tm_mday; // date 
  unsigned int tm_wday = (unsigned int)ti.tm_wday; // day of week 0=Sun,1=Mon...6=Sat
  Serial.print("mon=");
  Serial.print(tm_mon);
  Serial.print(", mday=");
  Serial.print(tm_mday);
  Serial.print(", wday=");
  Serial.println(tm_wday);

  // Self contained UK daylight saving algo. 
  // Not accurate to the hour of change which is good enough for me. 
  // i.e. switch at 12am rather than BST at 1am and into GMT at 2am. 
  // Use at your own risk 
  if (tm_mon>2 && tm_mon<9) {
     // Apr-Sept is BST for sure  
     daylightOffset_sec= 3600;
     Serial.println("cond1");
  } else if (tm_mon==2 && tm_mday>24 && (tm_mday+(7-tm_wday)) > 31) {
    // BST starts last Sun in March. Cover last Sun to 31st. Earliest Sun is 25th.  0 = Sun. 2 = Feb 
    Serial.println("cond2");
    daylightOffset_sec= 3600;
  } else if (tm_mon==9 && ( tm_mday<24 || (tm_mday+(7-tm_wday)) < 32 )) { 
    // BST ends last Sun in Oct. Cover to the last Sun. Earliest Sun is 25th. 0 = Sun. 9 = Oct
    Serial.println("cond3");
    daylightOffset_sec= 3600;
  } else {
    Serial.println("catch all");
    daylightOffset_sec = 0;
  }

  Serial.print("after logic, daylightOffset_sec=");
  Serial.println(daylightOffset_sec);

What's Platform Engineering?

Wed, 27 Oct 2021 00:00:00 GMT

“Platform Engineering” is the act of developing common internal systems that enable teams working on the customer-facing product (stream-aligned¹ or empowered² teams) to focus on the customer, move faster, and scale efficiently. These systems must address a common need across most, ideally all, of engineering (Total Addressable Market for Platform Engineering). These systems need to have high standards for self-service which create autonomy. Platform team success is measured by high-levels of adoption, usage, and % engineering hours freed up for customer “feature development”. For success, develop, deliver, and support these systems as products by applying a Product Management mindset.

I group platform systems into two categories:

Tools - used to bake and ship a cake, the whisk used to beat the eggs and the oven. These are used at design-time/development-time throughout the development lifecycle (SDLC) aka value stream. You should think of these as the control plane of your system. Examples: testing frameworks, CI/CD systems, IDEs, launch flag systems, logging, and monitoring.

Building blocks - are the ingredients that go into the cake. The customer consumes these things. How well the cake bakes and tastes depends on the quality of the ingredients. These are runtime components, you should think of these together as the data plane. Examples: programming languages, frameworks, libraries, databases, and hosting systems.

Defining platform is for another time but it’s not a singular thing. A SaaS stack is composed of platforms, think of them as layers with hardware all the way at the bottom. Platform engineering in SaaS exists somewhere in the middle with everything below typically bought from cloud infrastructure providers like AWS. The lines are constantly moving as the novel practices become more commonly accepted, then productised, and eventually become a utility like AWS ³. If you are in platform engineering get comfortable with constantly reinventing ones job spec and technology skillset.

What’s the structure of a platform engineering team? this cannot and should not be answered in isolation. It must be designed in context of the wider product engineering organisation and business which includes factors such as scale. It could be a single team, a group of teams under a common platform mission, or multiple teams under independent missions.

Ultimately platform engineering isn’t defined by a universal set of responsibilities anchored around specific technologies. It’s not a bucket you throw everything with labels like AWS, CI/CD, or server. Remember, the top level objective is to enable stream-aligned teams to focus on the customer, move fast, and scale. Therefore, the team structure depends on what platforms are needed which depend on what challenges the stream-aligned teams are/will be facing. There may well be some common practices and patterns that have emerged. Especially has you go lower down the stack, there should be less and less reinventing the wheel. Team Topologies¹ and Accelerate⁴ are some of the best and broadly recognised writing on this topic to-date.

Finally, I remind myself that (internal) platform engineering isn’t about solving problems with just technology, remember the people factor. The culture, incentives, motivations, and team interactions all play an important part.

The above is my current (Oct 2021) best thinking based on experience and observations. If you have any thoughts, questions, or recommended reading hit me up on Twitter @janaka_a.

Introducing Osobisty, My Universal Personal Search Engine

Fri, 01 Oct 2021 00:00:00 GMT

Over the past 3 weeks I built a universal personal search system which I’ve named Osobisty (which means personal or private in Polish). I’ll dive deeper into why I need Osobisty in another post. The short story is I’m looking at ways to improve how I capture information, learn (turn into knowledge), and recall that information+knowledge when needed. This post is about why I decided to build this rather than “buy” along with some technical details.

What is a Universal Personal Search Engine?

It’s a single place to search and access my curated content both public or private. I currently have private notes (Zettlekasten), Kindle highlights, and bookmarked tweets indexed. Bookmarked web pages are next on the list. Contact list, stared emails , and stared Google Drive content might come down the road. You get the idea.

Inspiration

The penny dropped when I heard Linus Lee (aka The Sephist) talk about his Monocle project on the The Changelog E455 - Building Software for Yourself. The problem(s) he’s trying to solve resonated. Apollo is another personal search engine, inspired by Monocle, which I also looked at.

The reasons why Linus builds his own software also resonated. I used to outright dismiss the idea but I’ll give it more consideration going forward. However I don’t intend to build my own solutions to the extent Linus has. I’m definitely not going to build an entire custom stack and tool chain. I might implement a very basic programming language for fun and learning.

As a side note Linus is a side project machine, he’s obviously a very talented engineer. He shares a lot of interesting thoughts on various topics from creative work, productivity, startups, to life. It’s worth following his writing for inspiration if those topics interest you.

Why build my own solution?

Generally my preference is to buy rather than build. Other than websites and this Desktop Clock I’ve never built software for personal use. My important needs and problems don’t tend to be unique so somebody else has already built a solution that is good enough. The same is of course true in this case so here are my reasons.

My main requirements:

Full control over my private data. I can choose to not expose the system over the Internet. If it’s way less likely to get hacked and leaked in a public way compared to a commercial service.
No data and information lock-in (including meta). I want to manage as much of the data as possible in easy to edit Markdown because want it to be super portable over a long period of time (multiple decades into the future).
Support for ingesting my information sources. This meant having the flexibility to extent with custom source formats.

These criteria exclude SaaS solutions. By those requirements Monocle or Apollo are still in the run, they both can meet my main requirements. However, Monocle is built in Linus’ custom programming language (Ink) and UI framework (Torus). Apollo is written in Golang. If I’m going to spend any time coding I want to use languages and technology that are the most broadly applicable in the industry today. That gives me the most return on time-investment because I gain knowledge that helps with my day job. Which gets at the other key reason to build rather than buy. I want a good excuse to code and build, because I enjoy it. Something that adds value to learning and my day job makes it worth while.

I think it’s obvious why I don’t want to learn Ink and Torus (Linus’ reasons are legit). What’s the problem with Golang then? It’s applicable in the industry. But I don’t want to build a web UI using Golang. It’s the wrong tool for the job in my opinion. If I decided to build the search engine backend myself I may have used Golang but I decided not to hand build that (more about that below). Unlike Linus I don’t have a goal to learn the insides of a search engine implementation. It’s something I’ve already dabbled in, though 10-15yrs ago, and I don’t need to understand it any deeper at this time. Of course assuming I could find a suitable search engine to build on top of…

Technologies used to build Osobisty

UI - ReactJS + TypeScript. Crawlers and indexers - NodeJS + TypeScript Search engine - Typesense a fast OSS search engine that’s really easy to work with. I started out considering DuckDB which I’d come across recently (I think on Changelog E454) which supported text indexing. And Lucene was probably one of my backup options. But then I discovered Typesene. I also looked at Meilisearch but Typesense seemed to be better all-round. I didn’t do a deep analysis, just on paper assessment.

React is a very popular industry choice for web UI and I’m familiar with it. I did consider using this as an excuse to try out NextJS but decided against the overhead for now. I wanted to balance productivity and learning. Given the UI was going to be Typescript I decided to use the same for the indexers, again remove the learning/refreshing overheard of two languages rather double down on one.

As a starting point the UI and functonality is clone of Monocle (full credit to Linus in the code) which I plan to Open Source soon.

I’ve also started to work on an accompanying Chrome extension. More on that in a separate post. This is also inspired by Linus, he has a Chrome extension that does semantic search in Monocle to surface related content to a web page is browsing. It looks like I’ll have to implement the semantic search algorithm.

DevSecOps: Taking Snyk and Github Dependabot for a Test Drive

Fri, 03 Sep 2021 10:48:10 GMT

Generally the reason I switched to hosting my own site with Gatsby and play around with various software development tools and technologies is to learn the new. Importantly it also gives me a taste of the latest developer experience. Until recently I’ve not followed the InfoSec area, my focus has been elsewhere. I expect to learn a little more on the InfoSec front as well.

I’ve had Dependabot switched on for this blog (repo) for a while. I wanted to see a) if it would encourage me to keep the software up-to-date, and b) how easy does it make remediation.

This mode of thinking totally applies to commercial software development. Maintenance is absolutely a necessary part of the software life cycle. But the objective is to minimise this burden so teams can focus on delivering new value to customers.

This isn’t going to be a comprehensive analysis. Rather, some notes on what I learn, experience, and observe.

Disclaimer: This is a naive comparison.

High-level

Snyk and Github provide the same code vulnerability detection and remediation functionality. They detect issues with your code and dependencies.

Github
- Dependabot (dependencies) + Code Scanning (your code). Code scanning can be performed by multiple tools. Github CodeQL Analysis + third-party scanning tools. Side note: Dependabot and CodeQL Analysis are acquisitions IIRC.
- No Infra-as-Code (IaC) scanning. But I assume this will come, It’s another lang for CodeQL.
- No container image scanning. I assume this will come at some point.
Snyk
- Snyk Code (your code) + Snyk Open Source (dependencies).
- Snyk also supports container image scanning and Infra-as-Code (like Terraform) scanning. Not sure if the latter is dependency and code also.

Setup

Github
- Dependabot: I can’t remember what I did exactly but IIRC it was very easy. Check a couple of boxes and click and enable button.
- Code Scanning (CodeQL): involves adding a GH Action Workflow to the repo. A wizard in the web UI guides you though adding the YAML file (I accepted the default), then creating and merging the PR.
Snyk
- Sign up for a Snyk account. I used by Github account and was smooth. I do count this as an additional step even thought technically one needs to create a GH account if going from scratch. But typically ones would have a Github, Gitlab, or Bitbucket account forming he base of their development workflow.
- I got the choice of giving perms to just public or public and private reports. I opted for the former for now. In following steps I was given the option to select which repos were to be scanned.
- Completed the wizard and a scan was immediately kicked off.

Scanning

Github
- Dependabot
- seems to perform regular scans and pushes to the repo also trigger scans
- No VS Code extension
- Code Scanning (CodeQL)
- VS Code extension is available for CodeQL but this seems optimised for CodeQL query development rather than vulnerability detection during the app dev workflow.
Snyk
- Auto scan on a daily or weekly frequency. Or manual trigger
- I didn’t time the initial scan. When I did a rescan on this repo today it was reasonably fast, again didn’t time, but in the order or several minutes I feel.
- I assume there’s CI/CD integrations that can trigger scans
- VS Code extension (Snyk Vulnerability Scanner) should give feedback early in the dev lifecycle. Also faster feedback during the dev inner loop. Currently (v1.1.1) only supports Snyk Code. Extension description says Snyk Open Source support is coming out in Q3.

Results

The results between the two are different not surprise.

Github
- Dependabot: 29 issues ( 2 critical, 14 high)
- Snyk did pick up both these critical issues but scored them high
- Did pick up the sanitize-html critical that Snyk reported
- Code Scanning ( CodeQL): 0 issues

Snyk
- 46 issues (1 critical, 19 high)
- picked up the sanitize-html critical which Dependabot did not. However the fix wasn’t obvious. As the screenshot shows, there’s no fix-this-vulnerability button. Reading closer shows it’s a transient dependency of gatsby-transformer-remark@2.16.1 which I assume hasn’t upgraded the sanitize-html dependency yet.
- There’s priority score and severity score. Interestingly the critical has lower priority score. It seems like the maturity of the exploit determines the priority together with the CVSS score. The sanitize-html doesn’t have a known exploit apparently.

Applying Fixes

Both have the create PR with fix feature.
For whatever reason Snyk doesn’t do this for every issue.
Both create several PRs which makes sense. However it also makes it a little mentally hard to parse through and decide which order to apply. Maybe it doesn’t matter most of the time. But I do not yet trust that there aren’t going to be conflicts.

Final Thoughts

Github Dependabot and Code Scanning are obviously integrated into the Github UI which is ideal. However, at this time the gain seems to be marginal. At least for my one off use case test. Snyk does seem to be more comprehensive feature wise. And the scanning itself seems to catch vulnerabilities more completely. But I’m no expert on this front and haven’t analysed deep enough to see if the difference is meaningful. I’m going to see how the two compare overtime while I’ll stick to only applying issues reported by Snyk.

Learnt

CWE (Common Weakness Enumeration) A Community-Developed List of Software & Hardware Weakness Types
CVSS (Common Vulnerability Scoring System)

(Note 10 June 2020) Tracing in .NET and AWS App Mesh with OpenTelemetry

Wed, 10 Jun 2020 23:09:00 GMT

As of this writing OpenTelemetry is still in beta. There isn’t a lot of writing about it yet. Official docs are minimal, especially if you are in .NET Core land. Now throw AWS App Mesh and Fargate into the mix and we are in for a fun ride. Here are notes on what I learnt by messing around with it in a sandbox.

Ideally we would want to use Otlp (OpenTelemetry Protocol) as the export format in the app. But Envoy doesn’t support that yet.
Envoy tracing config changes in newer Envoy versions. App Mesh Envoy is about 2 versions behind right now (1.12.3).
Managing the Envoy tracing config like this isn’t ideal. Envoy version upgrades will involve changes to the tracing config. Appreciate the App Mesh team providing this escape hatch but I’d hope they add first class support. Not having to manually manage Envoy config is one of the main benefits of App Mesh.
The Otel .NET SDK is very immature and APIs are very unstable. They are upfront about this. It’s still in Alpha. It’s also behind the Golang and maybe Java SDK. The differences in Zipkin vs Otlp are shown below in an example. It sounds like there’ll be a significant refactor soon and some bits getting pushed into .NET BCL. Aaron from Petabridge wrote more about all this here and concluded that it’s about a year out still.
The pipeline architecture of the collector/agent is really nice. Allows one to config multiple pipelines of receivers, processors, and exporters. This makes it easy to send traces to multiple backends and options for switching out. Makes vendor evaluation easier and cheaper. Without cost should also be low.
Agent vs Collector - the naming is poor, they are both collectors by function. The name describes the mode it runs in.
- Agent: runs next to the app e.g. sidecar. This will be the core collector. See code repo. It supports only vendor agnostic protocols like Zipkin or OTLP (OpenTelemetry Protocol).
- Collector: Runs stand-alone and aggregates traces from several agents then sends to a backend like Zipkin (self-hosted) or LightStep (cloud). This will be the collector with vendor protocol contributions. See code repo.
- Official docs say “Agent is capable of adding metadata” and “can offer advanced capabilities over the Agent including tail-based sampling”. This implies that there are functional differences between the two binaries.
- See the official explanation here.

High-level view

The agent + collector setup is recommended by OTel.

Important Config Snippets

I’ll link to the working example source code in the future.

startup.cs for Zipkin protocol.

This code propagates trace IDs on outbound calls

using OpenTelemetry.Trace.Configuration; 

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();

    services.AddOpenTelemetry((sp, builder) =>
    {
        builder
            //.SetSampler(Samplers.AlwaysSample)
            
            .UseZipkin(options =>
            {
                options.ServiceName = "hello-world-api";
                options.Endpoint = new Uri(zipkinEndpoint);
            })
            .AddRequestAdapter()
            .AddDependencyAdapter(); 
    });
}

At present the API for Otlp is a little different.

  // ref https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/TracerBuilderExtensions.cs
  services.AddOpenTelemetry((sp, builder) =>
   {
       builder
           .SetResource(Resources.CreateServiceResource("hello-world-api"))
           .UseOpenTelemetryProtocolExporter (options =>
           {
               options.Credentials = ChannelCredentials.Insecure;
               options.Endpoint = otlpEndpoint;
           })
           .AddRequestAdapter()
           .AddDependencyAdapter(); 
  });

Create a custom App Mesh Envoy image with the override config because it’s the easiest way on Fargate. Not ideal for production.

Envoy (v1.12.3) override config envoy-otel-tracer-config.yaml

tracing:
  http:
    name: envoy.zipkin
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v2.ZipkinConfig
      collector_cluster: otel-agent
      collector_endpoint: "/api/v1/spans"
      #collector_endpoint: "/api/v2/spans"
      #collector_endpoint_version: HTTP_JSON

static_resources:
  clusters:
  - name: otel-agent
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: otel-agent
      endpoints:
      - lb_endpoints:
        - endpoint:
           address:
            socket_address:
             address: 127.0.0.1
             port_value: 9411

FROM 840364872350.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.12.3.0-prod
COPY envoy-otel-tracer-config.yaml /tmp/envoy-otel-tracer-config.yaml

Env var that tells the App Mesh Envoy container to use the tracing override config file. The complete task def has 3 containers defs: 1) the app 2) App Mesh Envoy 3) OTel agent.

  {
    "name": "ENVOY_TRACING_CFG_FILE",
    "value": "/tmp/envoy-otel-tracer-config.yaml"
  }

(Note) Using gatsby-plugin-s3 with Github Actions

Mon, 11 May 2020 00:36:00 GMT

I was originally using the Gatsby and S3 sync Github actions together. This worked fine bit shifting. Later I discovered that updates were not showing up when browsing on my mobile. A quick Google and I realised there are some HTTP cache headers that Gatsby recommend which I was not applying. The gatsby-plugin-s3 which implements the recommended caching config. So I decided to switch. I didn’t find Github Action examples online so maybe this post will help somebody save time.

I’m using Cloudflare rather than CloudFront and a pre-created S3 bucket which isn’t managed using code (i.e. Terraform, CloudFormation etc.). It seems they have some workarounds for system like TF and CF that maintain state but maybe it’s best to use the AWS CLI to codify bucket creation in this case.

Here are the relevant snipets. You can view it in context in the repo here.

This is a better setup than what I had before. It’s more Gatsby tuned and much faster. The previous workflow took 2-3mins to execute. This one takes less than 1.5mins. And if you are wondering, my caching issue is resolved. Well at least I’m seeing changes from my mobile. I’m yet to test if what should be cached is being cached.

gatsby-config.js fragment

  plugins: [
    {
      resolve: `gatsby-plugin-s3`,
      options: {
          bucketName: process.env.AWS_S3_BUCKET_NAME,
          region: process.env.AWS_REGION,
          protocol: `https`,
          hostname: `janaka.dev`,
      },
    },
  ]

package.json fragment

  "scripts": {
    "deploy": "npx -n \"-r dotenv/config\" gatsby-plugin-s3 deploy"
  }

Github Actions workflow

.github/workflow/deploy_to_S3_on_push.yaml

name: Deploy Website to S3 Hosting

on:
  push:
    branches:
    - master

jobs:
  deploy:
    name: Build & Deploy
    runs-on: ubuntu-latest
    env:
      AWS_S3_BUCKET_NAME: ${{ secrets.AWS_S3_BUCKET }}
      AWS_REGION: ${{ secrets.AWS_REGION }}
    steps:
    - name: Checkout Repo
      uses: actions/checkout@master
    - name: Install dependencies
      run: yarn install --prod --pure-lockfile
    - name: Build Gatsby Site
      run: yarn build
    - name: Deploy to S3 (with sync)
      run: yarn deploy
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

IAM policy

It seems like they haven’t figured out policies for all the combinations. See more details in this GH issue

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutBucketWebsite",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_NAME_HERE>",
                "arn:aws:s3:::<S3_BUCKET_NAME_HERE>/*"
            ]
        }
    ]
}

(Note) Handling Secrets Correctly in Terraform Config

Fri, 08 May 2020 12:18:00 GMT

There two things that need to be handled.

1. Encrypt the secret in the Terraform config file

Don’t have the secret as plaintext in any of the config file. Encrypt using AWS KMS. Then use awskmssecrets to decrypt at runtime.

Can also use an encrypted secret store like AWS Parameter Store or Hashicorp Vault.

2. Use an encrypted backend for state

Use Terraform Cloud (remote) as the backend.

State files store the plaintext version of secrets. So don’t use local statefiles and check-in to version control. Use a encryted backend like Terraform cloud or AWS S3 to persist state.

Disclaimer: These are notes for myself that I’m sharing. I’m not claiming this is the way or the best way.

Simple example of KMS encrypt and decrypt using AWS CLI v2

Thu, 07 May 2020 23:34:00 GMT

On macOS

Encrypt:

aws kms encrypt --region eu-west-1 / 
--profile <aws_profile_name> / 
--key-id <your_kms_key_here> /
--plaintext fileb://<(echo 'Hello Hello Hello you cheaky secret') / 
--encryption-context somekey=sometoken / 
--query CiphertextBlob / 
--output text

Decrypt:

aws kms decrypt --region eu-west-1 / 
--profile <aws_profile_name> / 
--ciphertext-blob fileb://<(echo '<the_output_from_the_encrypt_command_above>' | base64 -d) / 
--encryption-context somekey=sometoken / 
--output text / 
--query Plaintext | base64 -d

For context, I wanted to quickly encrypt an API token so I could embed it in a Terraform config. Initially, I followed the Terraform doc here. The command in the doc ran successfully but the Terraform config couldn’t make the API call with the token successfully. As a troubleshooting step I wanted to test the decypt using CLI. This didn’t work. Trying to run the command with a sentence as the plaintext errored. It seems because it was a token the command ran successfully but didn’t encrypt the actual token. This post was a great simple example. But it didn’t work either. When encrypting I was getting the error Invalid base64: "Hello Hello Hello you cheaky secret". This Github issue put me on the right track. AWS made some breaking changes in CLI v2.

Hosting this blog on IPFS with DNS and TLS

Fri, 01 May 2020 23:38:00 GMT

I first got this site up and running a couple of weeks back. Originally my goal was to host exclusively on IPFS. This was an exercise to learn about IPFS hands-on. I wanted the site address to be http://janaka.dev. As I started setting that up, initially against the ipfs.io gateway, I learnt that the .dev TLD was TLS only. The ipfs.io gateway didn’t support TLS. So now I had the additional challenge.

CloudFlare

A little Googling and all I could find was CloudFare’s IPFS gateway that supported TLS. I had used textile.io buckets to pin the content. The site was accessible through the address they generated, no problem. But I had real problems getting the site to load reliably via the CloudFlare gateway with DNSLink and my domain. It nearly always didn’t load. So I decided to host https://janaka.dev using S3 and have a site mirror setup at https://ipfs.janaka.dev even though at this point is wouldn’t work. I needed setup to tinker with.

I couldn’t find any formal support from CloudFlare for the IPFS gateway. I wasn’t surprised as I knew it was somewhat an experiment. I Tweeted on the off chance. The discussion put Fleek.co on my radar.

Fleek.co

Fast-Forward to today. I decided randomly to give Fleek a shot. I must say it was pretty quick to get the site moved over and working successfully. Site load performance seems great and works reliably so far.

In my original setup with Textile.io + CloudFlare IPS Gateway I had to do all the work to setup CI/CD. I used Github Actions. Also Textile didn’t have a way to automate the IPNS step. With Fleek, it handles CI/CD soup to nuts (I heard this saying recently and it stuck. Apparently it’s American Google tells me :) ). It’s all wizard driven. Logging with Github. Select the repo, it detected that it was Gatsby, filled out all the build config defaults, TLS cert etc. There were a few manual steps for DNS config of course but this is expected. I presume if my domain was managed by Fleek these would also be automated. So now https://ipfs.janaka.dev is built, deployed, and pinned via Fleek.co

(Collection) IPFS Ecosystem

Wed, 15 Apr 2020 03:08:00 GMT

https://infura.io

IPFS and Ethereum platform with API and SDK
Runs an IPFS gateway https://ipfs.infura.io. But it doesn’t support IPNS (as of 12 April 2020)

https://temporal.cloud

IPFS platform with APIs and SDK
Provides IPFS pinning. Claims to be the cheapest
Public gateway at https://gateway.temporal.cloud
S3X services enables video streaming blog post

https://textile.io

IPFS platform with some abstractions. Makes it easier to work with IPFS.
buckets - an S3 esq abstraction for managing uploaded object content. Includes pinning and versioned endpoints for access over HTTP.
threads - a MongoDB esq abstraction for structured data.

https://pinata.cloud/

IPFS platform with API and toolkit
Provides an IPFS pinning service. You pay them to pin (store) your data

https://www.eternum.io/

IPFS platform
Provides a pinning service

https://fleek.co previously terminal.co

platform for building web sites & apps on IPFS.

https://awesome.ipfs.io/

Currated set of IPFS related resources

https://cloudflare-ipfs.com

Public IPFS Gateway
Supports TLS certs for custom domains. Useful when hosting sites on .dev TLD which is HTTPS only.

https://d.tube/

A Youtube clone. Video is stored on IPFS.

IPLD

IPLD is the latest protocol for linking different hash based structures e.g. link a file IPFS to a commit in Git. Bitcoin and Ethereum also examples of hasg based structured currently supported. IPLD replaces the Merkel-DAG spec in IPFS.
IPLD Explorer in the browser

Hello World

Sat, 28 Mar 2020 18:31:00 GMT

Update [15 April 2020]: I’ve not managed to get IPFS hosting to work reliably yet. Until I do, https://janaka.dev is hosted traditionally (web2). A mirror is hosted on IPFS (web3) at https://ipfs.janaka.dev.

This is my first post hosted on IPFS, the distributed Internet. How exciting!

I wanted an excuse to play with IPFS to learn and thought hosting a blog on IPFS would be a good little project. So over the weekend I had a little hack and here you are.

It also gave me an excuse to play with static site generators. I’ve picked Gatsby. I’m starting simple, using the Gatsby blog starter and markdown files.

In the future I might move the content to a headless CMS like Contentful or Prismic. Why? well not because I think I need anything fancier than MD files. I think a static site with a headless CMS is a strong architecture for a marketing website. Benefits on the technical site are very fast page load times, low hosting costs including maintenance, and static stability. If managing content at scale then content managers still get a powerful and friendly CMS. I’ve not tried Contentful or Prismic yet so that’s a bit of an assumption. Hence why I want to try the setup myself to get some idea how well it all could work.

Delivering on a cloud-first promise

Fri, 04 Jul 2014 09:50:00 GMT

This is one of those promises I made to myself, approximately 5 years ago. I had long since realised that if they can’t see it, they don’t really care. They don’t care until it goes wrong that is. Therefore spending time on what mattered to my colleagues and our customers and outsourcing the rest became the strategy. I’m pretty sure this isn’t revolutionary thinking but how does one execute is always the question, right?

Outsourcing is a concept that had been around for many years but 2009 was different. It was different because cloud computing was starting to gain traction. This method of out sourcing thoroughly suited us. And there started our cloud-first strategy.

Yesterday marked a significant milestone in this journey. We pulled the plug on the final major piece of legacy infrastructure. We decommissioned the final rack in the old data centre which to be honest has largely been a very expensive storage cupboard for the last few years.

In everything we do I ask the question “Would our customers care?” “How much value would this add to our customer?”. I believe we are now firmly into the second iteration. Seeking to push higher up the stack, IaaS to PaaS to SaaS.

Celebrate, it’s all gone!

Addition 23 Jan 2021: This rack was located in a Colt data centre, Wapping, London, UK.