sahansera.dev RSS Feed

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

Sun, 22 Feb 2026 00:00:00 GMT

If you’re anything like me, you’ve probably spent a Friday afternoon trying to remember everything you did that week. Maybe it’s for a standup, a 1:1 with your manager, or just to keep track of your own progress. You end up clicking through PRs, issues, and Slack threads, trying to piece together a coherent story. It’s tedious, and honestly, it’s time you could spend doing actual work.

That’s why I built gh-weekly-updates - a CLI tool that automatically collects your GitHub activity and generates a structured weekly summary using AI.

💡 GitHub repo: github.com/sahansera/gh-weekly-updates. It’s open source and available on PyPI!

The Problem

As engineers, we’re constantly shipping code, reviewing PRs, filing issues, and jumping into discussions. But when it comes time to reflect on the week, all that context is scattered across repos. I wanted something that could:

Pull all my GitHub activity into one place
Summarise it in a way that highlights what actually matters
Run on a schedule so I don’t have to think about it

I couldn’t find anything that did exactly this, so I built it.

What It Does

gh-weekly-updates connects to the GitHub API, collects your activity for a given period, and sends it to an AI model (via GitHub Models) to produce a structured Markdown summary.

Here’s what it picks up:

Pull requests you authored (with merge status, additions/deletions, changed files)
Pull requests you reviewed
Issues you created
Issue comments you left
Discussions you started or participated in

The output is grouped by project or theme and structured into sections like Wins, Challenges, and What’s Next. You can also customise the prompt to match whatever format your team uses.

Getting Started

It’s a Python CLI tool, so you can install it with pip:

pip install gh-weekly-updates

Then just run it:

# If you're already logged in with the GitHub CLI
gh auth login
gh-weekly-updates

That’s it. It will auto-discover repos you contributed to in the past week and generate a summary.

You can also point it at specific repos and date ranges:

gh-weekly-updates --since 2026-02-09 --until 2026-02-16 --repos my-org/my-repo

Configuration

For more control, you can create a config.yaml:

org: my-org

repos:
  - my-org/api-service
  - my-org/web-app

model: openai/gpt-4.1

# Automatically push the summary to a repo
push_repo: my-user/my-weekly-updates

# Customise the AI prompt
prompt_file: my-prompt.txt

The config supports everything from repo lists to custom prompts. You can even swap out the AI model if you have a preference.

Running on a Schedule with GitHub Actions

This is where it gets really useful. You can set up a GitHub Actions workflow to run it every Monday morning and push the summary to a repo automatically:

name: Weekly Summary

on:
  schedule:
    - cron: '0 9 * * 1'  # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  summarise:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install gh-weekly-updates

      - name: Generate summary
        env:
          GITHUB_TOKEN: ${{ secrets.GH_PAT }}  # must be named GITHUB_TOKEN
        run: gh-weekly-updates --config config.yaml

Now every Monday, you get a fresh summary committed to your repo. No manual effort required.

Custom Prompts

The default prompt produces a summary with Wins, Challenges, and What’s Next sections. But you can tailor it to your needs. For example, if your team does impact-style updates, you might want sections like Strategic Influence or Next Steps.

Just create a text file with your prompt and reference it in your config:

prompt_file: my-custom-prompt.txt

The prompt receives all your raw activity data as context, so you can shape the output however you like.

Why Open Source?

I initially built this for myself to automate my own weekly updates at work. But I figured other engineers probably have the same problem, so I cleaned it up and open-sourced it. The tool is intentionally simple - it does one thing and tries to do it well.

If you find it useful, give it a ⭐ on GitHub. And if you have ideas for improvements, PRs and issues are always welcome!

What’s Next

A few things I’m thinking about for future releases:

More activity sources: Picking up commit messages, release notes, and code review comments
Multiple output formats: Slack messages, email digests, Notion pages
Team summaries: Aggregate activity across a whole team, not just one person

If any of these sound interesting to you, feel free to open an issue or start a discussion on the repo.

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

Sun, 27 Jul 2025 00:00:00 GMT

If you followed my last post, Building a Home Lab Kubernetes Cluster with Old Hardware and k3s, you now have a proper x86 Kubernetes cluster humming away on your old laptops. So, what’s next? Time to put that cluster to work—let’s run GitHub Actions jobs on your own hardware!

Why? Because self-hosting your runners is faster, gives you full control (no GitHub minutes limit!), and lets you run bigger jobs (CI, builds, ML, you name it) on your home infra. And with Actions Runner Controller (ARC), managing runners at scale on Kubernetes is surprisingly easy.

Here’s how to set it all up.

What is ARC and Why Should You Care?

ARC (Actions Runner Controller) is an open-source Kubernetes operator from GitHub. It spins up and manages GitHub Actions runners as Kubernetes pods—no more manually registering runners, no more pets, just cattle. Runners auto-scale up and down as jobs arrive. It’s perfect for CI/CD, especially on clusters you own.

Here’s a high level view of how it works under the hood

Prerequisites

Working Kubernetes cluster (see previous post)
kubectl and helm installed on your machine
A GitHub Personal Access Token (PAT) with repo and admin:org scopes

1️⃣ Pre-Setup: Quick Checks

Make sure you have what you need:

which helm
kubectl version --client
helm list -A

If those commands work, you’re good to go.

2️⃣ Install ARC Controller

Let’s install the ARC controller into your control plane namespace:

NAMESPACE="actions-runner-controller"
helm install arc \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

This deploys the controller which manages your runners.

3️⃣ Deploy a Runner Scale Set

Time to create the runners that will actually do the work. Replace the example GitHub URL and PAT with your details:

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/youruser/yourrepo"
GITHUB_PAT="ghp_123456..."

helm install "${INSTALLATION_NAME}" \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
  --set githubConfigSecret.github_token="${GITHUB_PAT}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

GITHUB_CONFIG_URL: The repo or org you want to run jobs for.
GITHUB_PAT: Your Personal Access Token.

4️⃣ Check That It’s Working

Verify the controller and runner pods are up:

# Controller
kubectl get pods -n actions-runner-controller

# Runners
kubectl get pods -n arc-runners

# Runner set status
kubectl get AutoscalingRunnerSet -A
kubectl describe AutoscalingRunnerSet arc-runner-set -n arc-runners

You should see your runners show up as pods. If you trigger a workflow in your GitHub repo, you’ll see a pod spin up, do the job, and then shut down—magic.

5️⃣ Testing It Out

Here’s the fun part. Create a simple GitHub Actions workflow in your repo to test the runners:

name: Test ARC Runners
on: [push]

jobs:
  build:
    runs-on: arc-runners # This tells GitHub to use your self-hosted runners. Use the NAMESPACE name you defined in step 3.
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run a script
        run: echo "Hello from ARC Runner!"
      - name: List files
        run: ls -la

Here’s ci.yaml from my githubstats repo if you need a working example: ci.yaml

Make a commit to trigger the workflow. You should see the runner pod spin up, execute the job, and then terminate.

Here's the workflow running on my ARC self-hosted runner

5️⃣ Monitoring (Bonus: Grafana)

Want to geek out and monitor your runners? If you’ve set up Prometheus/Grafana (see my upcoming post if not!), you can:

Check pod CPU/memory usage
Track how many runners are running
See logs for each pod

Handy queries for Grafana dashboards:

# Number of ARC runner pods
kube_pod_status_phase{namespace="arc-runners", phase="Running"}

# Pod CPU usage
rate(container_cpu_usage_seconds_total{namespace="arc-runners"}[5m])

Here's what the workflow looks like in action

6️⃣ Useful Commands for Day-to-Day Ops

# Watch runner scaling in real-time
kubectl get AutoscalingRunnerSet -n arc-runners -w

# See events and troubleshoot
kubectl get events -n arc-runners --sort-by=.metadata.creationTimestamp

# Pod logs (for a specific runner)
kubectl logs -n arc-runners <pod-name>

Troubleshooting Tips

Runner not connecting? Double-check your PAT and network access.
Pods stuck or crash-looping? Check logs for clues and make sure your cluster has enough resources.
Can’t see runners in GitHub? Make sure the config URL matches your repo/org and the PAT has correct scopes.

That’s It! You’re Running GitHub Actions on Your Own Cluster

You now have GitHub Actions jobs running at home on your cluster, scaling up and down automatically. No more slow or limited runners. Your home lab just levelled up—CI/CD, builds, ML, you name it.

Stay tuned for my next post where I’ll show you how to get beautiful observability dashboards and set up alerting for your home cluster.
Happy automating!

Questions? Want to show off your setup? Ping me on X (Twitter) or drop a comment below!

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Sat, 26 Jul 2025 00:00:00 GMT

If you’ve read my previous post about building a Raspberry Pi k3s cluster, you know I’m a huge fan of home labs. There’s something uniquely satisfying about getting distributed systems running on a bunch of hardware you already own. This time, though, I wanted something a bit more powerful-a cluster that could handle not just learning and tinkering, but also heavier dev, CI/CD, and even ML workloads.

And as it turns out, there’s a ton you can do with a handful of old laptops, a simple switch, and Ubuntu Server. If you’re thinking about upgrading your home cluster or want to avoid vendor lock-in, this one’s for you.

Why Move From Raspberry Pi to x86?

Don’t get me wrong-RPi clusters are awesome for learning, hacking, and even some light home automation. But if you’ve ever tried running real dev pipelines, CI/CD, or any data-heavy ML stuff, you’ll hit those limits fast. Plus, WiFi can get a bit flaky when you’re trying to keep nodes connected under load.

I had a few spare laptops sitting around, and it made perfect sense to give them a second life and push them to their limits. Bonus: with Ethernet and a decent switch, you get much more reliable connectivity.

Network Setup: Keep It Simple (and Reliable)

For this build, I kept things straightforward: all nodes are wired to a simple network switch. This gives much better performance and reliability compared to WiFi, but honestly, you could still pull this off over wireless if needed.

My network switch - how it all started 🛜

A quick tip:

Reserve static IPs for each node using your router’s DHCP reservations.
This makes node management, SSH, and Kubernetes networking so much easier.
Check your DHCP table and make sure every machine has a unique, predictable IP address.

Getting Ubuntu Server Set Up on Each Node

Here’s a high level diagram of what my setup looks like:

Let’s talk about prepping each machine. My old laptops had a new lease on life running Ubuntu Server. I recommend Ubuntu Server LTS for its simplicity and compatibility. If you’re using VMs, just make sure to set the NIC to bridged mode so each VM acts as a full member of your home LAN.

Here’s my go-to checklist for each node:

Update the system and install SSH:

sudo apt update && sudo apt upgrade -y
sudo apt install openssh-server -y
sudo systemctl enable ssh
sudo systemctl start ssh

Set a memorable hostname:

sudo hostnamectl set-hostname master   # or worker-1, etc.

(Optional) Update /etc/hosts for local resolution:
```
sudo nano /etc/hosts
```
Change any lines like 127.0.1.1 old-hostname to your new hostname.
Find your node’s IP address:
```
ip a
```
Use this IP for your DHCP reservation so it always stays the same.
Test SSH from another machine:
```
ssh <username>@<node-ip>
```

Installing k3s: Lightweight, Powerful, Familiar

This process feels pretty magical the first time you see it all come together. I stuck with k3s-lightweight and perfect for home or edge clusters.

On the Control Plane Node (“master”)

Install k3s:
```
curl -sfL https://get.k3s.io | sh -
```
Check that k3s is running:
```
sudo systemctl status k3s
```
Get your cluster join token:
```
sudo cat /var/lib/rancher/k3s/server/node-token
```
Copy this somewhere safe-you’ll need it for your worker nodes.
Get the node’s LAN IP:
```
ip a
```
Let’s say your master node IP is 192.168.0.200.
Check your cluster status:
```
kubectl get nodes
```

If you ever want to reset/reinstall k3s, just run:

sudo /usr/local/bin/k3s-uninstall.sh

On Each Worker Node (“worker-1”, “worker-2”, etc.)

Join the cluster:
```
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.200:6443 K3S_TOKEN=<your-token-here> INSTALL_K3S_EXEC="--node-ip=<worker-ip>" sh -
```
Replace 192.168.0.200 with your control plane’s IP.
Replace <your-token-here> with the token from your master node.
Replace <worker-ip> with this worker’s IP, e.g., 192.168.0.201.
That’s it! The node will auto-register with your cluster. No manual kubeconfig needed.
Check the cluster again from the master:
```
kubectl get nodes
```
You should see both master and your worker(s) as Ready!

Troubleshooting and Gotchas

As with any home lab project, there are always a few snags-here’s what I ran into and how to fix it:

Node not joining?
- Double-check your token (no spaces, copy the whole thing).
- Make sure you can ping the control plane from your worker node.
- Firewalls can get in the way. Ensure port 6443 is open between nodes.
- If you get hostname conflicts, just change the hostname and restart the k3s agent (sudo systemctl restart k3s-agent).
Cluster state looks weird after hostname change?
- You might see both the old and new hostnames in kubectl get nodes. Just delete the old one:
```
kubectl delete node <old-node-name>
```
IP address keeps changing?
- Make a DHCP reservation for each node’s MAC address in your router.
WiFi unreliable?
- Ethernet is always the best bet for clusters, especially for heavy workloads.

What’s Next?

With the basics in place, you’re ready to:

Run and test real-world apps and dev environments
Build your own CI/CD pipelines
Experiment with ML workloads on dedicated nodes

This way, you get all the power of a real Kubernetes lab, but full control and no monthly surprises from a cloud provider.

Lessons Learned

RPi clusters are great for edge computing and learning, but once you need real muscle, x86 hardware makes a huge difference.
WiFi is convenient, but Ethernet is still king for stable clusters-especially if you’re doing anything performance-sensitive.
Old laptops/desktops are a goldmine for home lab builds. Don’t let them collect dust!

Final Thoughts

If you have old machines and a bit of curiosity, you can build a cluster that’s surprisingly capable-no Raspberry Pis required.
And if you’ve already done it with RPis, this is the perfect upgrade path.
Keep an eye out for my next post where I’ll deep-dive into adding observability and tooling!

Happy clustering! 🫡

How Does the Python Virtual Environment Work?

Sun, 06 Jul 2025 00:00:00 GMT

When you start working with Python, one of the first recommendations you’ll hear is to use a “virtual environment.” But what exactly is a Python virtual environment, and how does it work under the hood?

The Problem: Dependency Hell

Python projects often rely on third-party libraries. If you install packages globally, different projects can end up fighting over package versions. This is called “dependency hell.” For example, Project A might require requests==2.25, while Project B needs requests==2.31. Installing both globally can cause conflicts and break your projects.

The Solution: Virtual Environments

A virtual environment is an isolated workspace for your Python project. It lets you install packages locally, so each project can have its own dependencies, regardless of what’s installed elsewhere on your system.

How Does It Work?

When you create a virtual environment (using python -m venv myenv or virtualenv myenv), Python does the following:

Creates a Dedicated Directory Structure

A bin/ or Scripts/ directory with a Python executable and activation scripts
A lib/ directory with a copy of the Python standard library
A pyvenv.cfg config file for metadata

myenv/
├── bin/                # (Note: Scripts\ on Windows)
│   ├── activate        # Shell script to activate the environment (Unix)
│   ├── activate.bat    # Batch script (Windows CMD)
│   ├── Activate.ps1    # PowerShell script (Windows PowerShell)
│   ├── pip             # Environment-specific pip
│   └── python          # Environment-specific Python interpreter
├── lib/
│   └── pythonX.Y/
│       └── site-packages/  # Installed packages go here
├── pyvenv.cfg          # Configuration file with environment metadata

Configures a Standalone Python Interpreter:

The environment includes its own Python executable (or a symlink to it), ensuring that all commands run from within the environment use the correct interpreter. - On most systems, this is a symlink or copy of the base Python binary - This interpreter respects only the packages installed within the environment

    which python
    /path/to/myenv/bin/python

Sets Up Local Package Management:

Each environment gets its own site-packages directory:
- When you run pip install, packages go here instead of the global location
- This isolation prevents version conflicts and makes dependency management predictable
Creates Activation Scripts:

Activation scripts help you enter the environment by:
- Modifying your $PATH so that python and pip point to the virtual environment
- Optionally updating your shell prompt (e.g., showing (myenv))
- Ensuring commands are scoped to the environment
These scripts are OS-specific:
- Unix/macOS: source myenv/bin/activate
- Windows CMD: myenv\Scripts\activate.bat
- PowerShell: myenv\Scripts\Activate.ps1
Includes a Configuration File:

The pyvenv.cfg file records metadata about the environment, including the Python version and the location of the base interpreter.

This file stores:
- The Python version used
- The path to the base interpreter
- Whether system site packages are accessible (default: no)
This metadata is used when running the environment to preserve consistent behavior.

Command Resolution

So what happens when you are in a venv as opposed to running python command globally? The diagram below illustrates how Python and pip commands are resolved with and without a virtual environment. When no virtual environment is active, your system’s PATH directs these commands to the globally installed Python interpreter and packages.

However, once a virtual environment is activated, the PATH is modified to point to the environment’s own executables. This ensures that all Python commands and package installations stay isolated within the virtual environment, avoiding conflicts with system-wide installations.

Why Is This Powerful?

Isolation: Each project gets its own dependencies and versions.
Reproducibility: You can lock dependencies with a requirements.txt or pyproject.toml file, making it easy for others (or yourself in the future) to recreate the environment.
No Admin Rights Needed: You don’t need system-wide permissions to install packages.

Advanced Use Cases

Multiple Python Versions: Use virtual environments to test your code against different Python versions.
Custom Activation Scripts: Modify the activation script to set environment variables specific to your project.
Integration with CI/CD: Virtual environments are essential for setting up isolated builds in CI/CD pipelines.

Under the Hood: What’s Really Happening?

The virtual environment is just a directory with a specific structure.
No containers or VMs are involved-just clever manipulation of paths and environment variables.
Deleting the virtual environment directory removes all installed packages for that project.

Debugging Tips

If activation doesn’t work, check your shell configuration.
Use python -m site to inspect the site-packages directory.
Verify the pyvenv.cfg file for any misconfigurations.

Conclusion

Python virtual environments are a foundational tool for modern Python development. They solve the problem of dependency conflicts, make projects more portable, and keep your system clean. Whether you’re building a quick script or a large application, understanding how virtual environments work will save you countless headaches down the road.

References

Deep Dive - How Chunked Transfer Encoding Works

Wed, 02 Apr 2025 00:00:00 GMT

Chunked transfer encoding is a key HTTP/1.1 feature that allows servers to stream data incrementally without knowing the total size of the response upfront. It’s particularly useful in streaming APIs, live updates, and large or dynamically-generated responses.

In this post, we’ll practically explore how chunked transfer encoding works using the backend we developed in my previous blog post on Streaming APIs with FastAPI and Next.js - Part 2.

🤔 What is Chunked Transfer Encoding?

Chunked transfer encoding modifies HTTP responses into a series of chunks, each prefixed with its size in bytes. It allows servers to start sending response data immediately, without having to calculate the full content length beforehand.

When Transfer-Encoding: chunked is present, the client receives data incrementally and knows the response has ended when a zero-length chunk appears.

💻 Hands-on Example

Let’s use the FastAPI backend we built.

make start-backend

Let’s hit the /stream endpoint with curl to see how chunked transfer encoding works in practice.

curl -i --raw http://localhost:8000/stream

-i: Include headers in output.
--raw: Disable curl’s automatic decoding, revealing raw chunked encoding.

Expected output:

curl -i --raw localhost:8000/stream
HTTP/1.1 200 OK
date: Mon, 31 Mar 2025 09:51:47 GMT
server: uvicorn
content-type: text/plain; charset=utf-8
Transfer-Encoding: chunked

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

30
Simulated log entry at Mon Mar 31 20:21:53 2025

0

Here’s a diagram to help visualize the chunked transfer encoding:

Here’s what’s happening:

Each chunk starts with its length in hexadecimal (1f = 31 bytes).
The data follows the length, and the next chunk starts after a newline.
The chunk with 30 represents a simulated log entry (30 = 48 bytes).
The response ends with a zero-length chunk (0).

Note: This aligns with the techniques demonstrated in my previous blog series Streaming APIs with FastAPI and Next.js (Part 1) and Part 2.

🛠️ Step-by-Step Breakdown of Chunking

We’ll be using the index.py. Here’s exactly what’s happening under the hood:

1. Generator (yield):

Every time yield is executed, the Starlette framework (used internally by FastAPI) receives a new piece of data to stream to the client.
Each yielded data segment corresponds directly to one HTTP chunk.

For example, the yielded line:

yield "Waiting for new log entries...\n"

is packaged into one HTTP chunk.

2. Starlette’s StreamingResponse Handling:

The StreamingResponse from Starlette wraps the async generator.
Starlette doesn’t wait until the generator finishes (which might be infinite). Instead, it immediately pushes each yielded chunk to the underlying ASGI server, typically Uvicorn.

3. Uvicorn’s Chunk Formatting:

Uvicorn (the ASGI server you’re using) receives the yielded chunk from Starlette and formats it according to the HTTP/1.1 chunked transfer encoding specification:

Each chunk is transmitted as follows:

<chunk-size in hexadecimal>\r\n
<chunk-data>\r\n

Here’s how one of your actual data chunks might look:

1f\r\n
Waiting for new log entries...\n\r\n

1f = 31 bytes, the exact length of "Waiting for new log entries...\n"

4. Continuous Chunk Transmission:

Uvicorn immediately sends each formatted chunk down the TCP connection.
Your client (like curl) receives each chunk as soon as it’s sent, which allows incremental processing.

5. Ending the Stream:

If your generator ever completes (or if the server shuts down the connection), Uvicorn sends a special zero-length chunk (0\r\n\r\n) to indicate that transmission has ended.

Example final chunk:

0\r\n
\r\n

🙋 What about HTTP/2 and HTTP/3?

The short answer: HTTP/2+ does not use chunked encoding at all. In fact, the HTTP/2 specification explicitly forbids the use of the Transfer-Encoding: chunked header; if a client incorrectly tries to send it, it’s considered a protocol error.

Instead, HTTP/2 uses a more efficient binary framing layer that allows multiplexing multiple streams over a single connection. This means that chunked transfer encoding is not necessary in HTTP/2 and HTTP/3, as the protocol itself handles streaming more efficiently.

Key Takeaways

Through these practical examples, you’ve seen firsthand how chunked transfer encoding enables incremental streaming of data:

Responses are sent as a series of chunks, each with a defined size.
The end of data transmission is indicated by a zero-length chunk.
Tools like curl, Python frameworks like FastAPI, and browser developer tools help visualize and debug chunked encoding.

Understanding this helps you build better streaming APIs and debug complex HTTP interactions effectively.

Happy Streaming! 🚀

Upgrading sahansera.dev to Gatsby 5

Sun, 30 Mar 2025 00:00:00 GMT

How I Upgraded My Blog to Gatsby V5

The time is now 1:45 AM and I’m finally done upgrading my blog to Gatsby V5. It’s been one heck of a ride, but it was worth it.

This blog post is a reflection of my experience upgrading my blog to Gatsby V5. I’ll cover the challenges I faced, the solutions I found, and the lessons learned along the way.

The Upgrade Journey

After a two year long hiatus from blogging, I decided to dust off my old blog and give it a much-needed facelift. When I tried to run gatsby develop, I was greeted with a slew of warnings and errors. It was clear that my blog was long overdue for an upgrade.

I took a crack at it a few months ago, but I quickly got overwhelmed by the number of breaking changes and decided to put it on hold. This time it’s different because I was determined to fully utilize LLMs for my advantage.

Sneak peak of my last 3 commits 😅

Being a weekend, which is filled with family time and chores, I decided to timebox it just for a few hours. I thought, “How hard can it be? Just a few package updates and some code tweaks, right?” Little did I know that this would turn into a full-blown adventure.

Dependency Overhauls

One of the first tasks was to ensure all my plugins were updated to their latest non-breaking minor versions. I used npm-check-updates to help with this.

ncu --interactive --format group

Example of ncu CLI tool in action

💡 Tip: Always stay on a separate branch, use commits for each change you make, while doing updates.

There were bunch of warnings, but nothing major. Next, I upgraded React and ReactDOM to v18. This was a bit tricky because I had to ensure that all my dependencies were compatible with React 18. I also had to update my Babel configuration to support the new JSX transform.

npm install react@^18.0.0 react-dom@^18.0.0

After that, I bumped up the Gatsby version to ^5.0.0 and ran npm install. This is where the fun really began 😅

Plugin Upgrades

Many of my plugins were still on versions designed for Gatsby V4. I had to update or replace:

gatsby-plugin-catch-links, gatsby-plugin-feed, gatsby-plugin-google-gtag, gatsby-plugin-layout, gatsby-plugin-manifest, gatsby-plugin-offline, gatsby-plugin-react-helmet, gatsby-plugin-sharp, and gatsby-transformer-sharp etc..
Each of these required an upgrade to versions like ^5.14.0 or ^6.14.0 to resolve the peer dependency conflicts with Gatsby V5.
Image Handling:
I migrated from the deprecated gatsby-image to the modern gatsby-plugin-image. This required updating my GraphQL queries from using fluid and fixed fragments to using the gatsbyImageData field. I updated imports, replaced <Img> with <GatsbyImage>, and used getImage() to extract image data.

Much of this is covered in the official migration guide.

Then I ran into the infamous TypeComposer error.

Cannot create as TypeComposer the following value:
  GraphQLScalarType({ name: "Date", description: "A date string, such as 2007-12-03, compliant with the
 ISO 8601 standard for representation of dates and times using the Gregorian calendar.",
specifiedByURL: undefined, serialize: [function String], parseValue: [function String], parseLiteral:
[function parseLiteral], extensions: {  }, astNode: undefined, extensionASTNodes: [] }).

💡 The tl;dr here is there were multiple versions of graphql being used by other dependencies. I had to ensure that all my dependencies were using the same version of graphql. I did this by adding a resolutions field in my package.json:

{
  "resolutions": {
    "graphql": "^16.6.0"
  }
}

Then I ran rm -rf node_modules package-lock.json && npm install again to ensure all dependencies were using the same version of graphql. TypeComposer error was gone, but I still had a few warnings.

Next, I transformed my old GraphQL queries to the new format using the Gatsby codemod.

npx gatsby-codemods@latest sort-and-aggr-graphql .

This worked like a charm and fixed most of the query related issues. Everything else, I just asked ChatGPT to help me with. I had to tweak a few queries here and there, but nothing major.

Styling Challenges

I also relied heavily on styled-jsx for component-scoped CSS. However, Gatsby’s official plugin, gatsby-plugin-styled-jsx, only supports styled-jsx v3-and I needed styled-jsx v5 for React 18 compatibility. After some consideration, I decided to remove the plugin entirely and instead configured Babel to handle styled-jsx directly.

This is where ChatGPT had most of problems. It went through a diamond dependency resolution problem and got stuck in a loop. I had to manually intervene and guide it through the process.

Enabling Nested CSS with styled-jsx

Even after removing the old plugin, I ran into issues when my <style jsx> blocks used nested CSS rules. By default, styled-jsx doesn’t support nesting. I resolved this by integrating styled-jsx-plugin-postcss along with the PostCSS Nested plugin:

Installed the Packages:

npm install styled-jsx-plugin-postcss postcss-nested --save-dev

Created a Configuration File:
I added a styled-jsx.config.js at the project root:
```
module.exports = {
  plugins: [require('postcss-nested')]
};
```

Updated Babel Config:
In my .babelrc, I ensured the styled-jsx plugin was configured to use the PostCSS plugin:

{
  "presets": ["babel-preset-gatsby"],
  "plugins": [
    ["styled-jsx/babel", { "plugins": ["styled-jsx-plugin-postcss"] }]
  ]
}

This allowed my nested CSS in components like the header, footer, and blog items to compile correctly. I would say this the part where LLM helped me the most! 💪

PostCSS Configuration Adjustments

During the upgrade, I also encountered warnings about duplicate autoprefixer instances. My initial postcss.config.js was using postcss-cssnext, which is now deprecated. After some experiments and research, I updated the config to use postcss-preset-env-a modern alternative that handles vendor prefixes efficiently.

Lessons Learned

Upgrade in Small Steps:
Tackling dependency conflicts one by one and verifying functionality helps isolate problems.
Read Plugin Changelogs:
Understanding what’s changed in plugin APIs (like the migration from fluid/fixed to gatsbyImageData) is crucial.
Be Ready to Reconfigure:
Sometimes it’s necessary to remove old plugins (like gatsby-plugin-styled-jsx) and configure Babel or PostCSS directly to maintain compatibility with the latest React and Gatsby versions.
Test Thoroughly:
Both in development mode and via production builds (gatsby build and gatsby serve), to ensure SSR and dynamic behaviors work as expected.

Final Thoughts

Upgrading to Gatsby V5 has not only improved performance and introduced new features, but it also forced me to re-examine and modernize the entire toolchain-from image handling and styling to PostCSS configurations.

If you’re planning a similar upgrade, take your time, tackle one dependency at a time, and don’t hesitate to experiment with configurations until everything clicks.

Prepare to do some A/B testing by having a develop branch and a production branch. This way, you can compare the performance and functionality of your old setup with the new one.

Happy coding, and enjoy your faster, modernized Gatsby blog! 💜

References

Gatsby V5 Migration Guide

Feel free to leave a comment if you have any questions or if you’d like to share your upgrade experiences!

Streaming APIs with FastAPI and Next.js - Part 2

Sat, 29 Mar 2025 00:00:00 GMT

In Part 1, we explored how to stream data into a React component using modern browser APIs. Now it’s time to build the other half: the FastAPI backend that makes it all work.

In this post, we’ll walk through setting up a streaming endpoint using FastAPI and discuss how chunked transfer encoding works on the server side.

💡 Code Repository: The complete code is available on GitHub. You can clone it and run it locally to follow along.

🛠️ Building the Streaming Backend with FastAPI

How the data will flow from Backend to the Frontend

FastAPI makes it really easy to return a streaming response using its StreamingResponse class from starlette.responses.

Let’s build a /stream endpoint in index.py that simulates real-time data like server logs or chat messages.

🚀 Simulating a Real-Time Log Stream

Here’s a minimal FastAPI app with a streaming endpoint:

# backend/index.py
from typing import Any, Generator

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from fastapi.middleware.cors import CORSMiddleware
import time
import os
import uvicorn  # Import uvicorn for running the server
import threading

app = FastAPI()

# Add CORS middleware to allow cross-origin requests
# ...

LOG_FILE_PATH = "logs/server.log"

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            # Move to the end of the file
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n"  # Heartbeat message
                    time.sleep(1)  # Wait for new lines to be written
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"


def simulate_log_generation():
    """Simulate log entries being written to the log file."""
    while True:
        with open(LOG_FILE_PATH, "a") as log_file:
            log_file.write(f"Simulated log entry at {time.ctime()}\n")
        time.sleep(5)  # Write a new log entry every 5 seconds


@app.on_event("startup")
def start_log_simulation():
    """Start the log simulation in a background thread."""
    threading.Thread(target=simulate_log_generation, daemon=True).start()


@app.get("/stream")
def stream():
    return StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

🧠 How It Works

Let’s break it down:

1. Synchronous Generator

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n"
                    time.sleep(1)
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"

We define a synchronous generator function that yields data one chunk at a time. Each yield becomes a chunk sent to the client. The time.sleep(1) simulates delay between events (like logs being written).

2. StreamingResponse

StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

StreamingResponse tells FastAPI to send the data as it becomes available, rather than waiting for the entire response to be generated. The media_type is optional but helps inform the browser how to handle the content.

⚙️ Running the Server

To run the server:

make start-backend

Then hit:

http://localhost:8000/stream

You should see logs appear one line at a time in the terminal or browser, depending on how you call the endpoint.

💡 Tips for Production

✅ Keep the Stream Alive

In a real app, your data stream might be longer-running. Our example already implements this with a heartbeat:

if line:
    yield line
else:
    yield "Waiting for new log entries...\n"  # Heartbeat message
    time.sleep(1)  # Wait for new lines to be written

This ensures the client knows the connection is still alive even when there’s no new data.

🧹 Handle Disconnects Gracefully

If the client disconnects, your generator should stop yielding. Starlette handles this internally, but you can wrap the stream in a try/except block to catch cancellations if needed.

🔐 Secure Your Stream

Add authentication if you’re streaming sensitive data.
Rate-limit the endpoint to avoid abuse.

🧪 Testing with Curl

You can test the streaming endpoint with:

curl http://localhost:8000/stream

You should see each message appear line by line, every second.

🔗 Hooking it up with the Frontend

Now that our backend is streaming correctly, the frontend from Part 1 will handle it smoothly:

const response = await fetch("http://localhost:8000/stream");
const reader = response.body.getReader();
// ...

As each yield in the backend emits data, your React UI will update in near real-time. ✨

🧠 Recap

In this post, we built a real-time streaming API using FastAPI with just a few lines of code. Here’s what we did:

Built a synchronous generator to simulate streaming logs
Used StreamingResponse to stream text over HTTP
Connected it to our frontend from Part 1

Together, the backend and frontend make a simple but powerful full-stack streaming system.

🧱 Next Steps

🔌 Add dynamic data (e.g. logs from a file or DB)
📦 Stream structured data (like JSON Lines)
📈 Use this setup for real-time dashboards or log viewers

🛠️ Useful Links

If you enjoyed this series, feel free to star the repo ⭐ or share it with a friend. Got ideas or feedback? Hit me up on Twitter or drop an issue in the GitHub repo.

Happy streaming! 🚀

Streaming APIs with FastAPI and Next.js - Part 1

Sun, 23 Mar 2025 00:00:00 GMT

Streaming data in the browser is one of those things that feels magical the first time you see it: data appears live - no need to wait for the full response to load. In this two-part series, we’ll walk through building a small full-stack app that uses FastAPI to stream data and a Next.js frontend to consume and render it in real-time.

💡 Code Repository: All the code for this post is available on GitHub. Feel free to explore, clone, and experiment with it!

There are many ways to stream data in the browser, from WebSockets to Server-Sent Events (SSE). But in this post, we’ll focus on a lesser-known method: chunked transfer encoding. This technique allows the server to send data in small, manageable chunks, which the browser can process as they arrive.

If you are interested in learning about Websockets, check out my post on Understanding WebSockets with ASP.NET. It’s targeted at .NET developers, but the concepts are similar. Again, the focus here is on streaming data in the browser and WebSockets is a different protocol altogether.

This post focuses on the frontend bit. In Part 2, we’ll dive into building the FastAPI backend.

🔧 The Setup

How the data will flow from Backend to the Frontend

Let’s say you have a streaming API running locally at:

http://localhost:8000/stream

This endpoint sends back a stream of text data - think server logs, chat messages, or real-time updates. The goal is to connect to this endpoint from a React component and display data as it arrives.

Here’s a simplified version of the key parts of our React component (index.tsx):

// Key imports
import { useEffect, useState, useCallback } from "react";

export default function IndexPage() {
  const [dataChunks, setDataChunks] = useState<{ timestamp: string; log: string }[]>([]);
  const [isLoading, setIsLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);

  const fetchStream = useCallback(async () => {
    // Reset state
    setIsLoading(true);
    setError(null);
    setDataChunks([]);

    try {
      // 1. Connect to the stream
      const response = await fetch("http://localhost:8000/stream");
      
      if (!response.ok || !response.body) {
        throw new Error(`Stream connection failed`);
      }

      // 2. Get a reader from the stream
      const reader = response.body.getReader();
      const decoder = new TextDecoder("utf-8");
      
      // 3. Read chunks until done
      while (true) {
        const { value, done } = await reader.read();
        
        if (done) break;
        
        // 4. Decode and update UI with each chunk
        const chunk = decoder.decode(value);
        setDataChunks((prev) => [
          ...prev, 
          { timestamp: new Date().toLocaleTimeString(), log: chunk }
        ]);
      }
      
      setIsLoading(false);
      
    } catch (error) {
      setError(error instanceof Error ? error.message : String(error));
      setIsLoading(false);
    }
  }, []);

  // Connect to stream on component mount
  useEffect(() => {
    fetchStream();
  }, [fetchStream]);

  // Rendering components (simplified)
  return (
    <div>
      <h1>Server Log Viewer</h1>
      {/* Error handling and rendering of streamed data */}
      {/* See full code on GitHub */}
    </div>
  );
}

🧠 What’s Really Happening?

Let’s break this down and understand the key concepts behind streaming in the browser:

1. Fetching the Stream

const response = await fetch("http://localhost:8000/stream");

Unlike regular fetch() calls that wait for the entire response before handing it over, this gives you access to the ReadableStream - letting you process data chunk-by-chunk.

2. Getting a Reader

const reader = response.body.getReader();

This gives us a ReadableStreamDefaultReader, which lets us manually pull chunks of data from the response. This is part of the Streams API, now supported in all major browsers.

3. Reading Chunks

const { value, done } = await reader.read();

value: a Uint8Array representing a chunk of binary data.
done: true when the stream is finished.

We loop until done becomes true.

4. Decoding the Text

const decoder = new TextDecoder("utf-8");
const chunk = decoder.decode(value, { stream: true });

Streaming responses may split characters across chunks, especially for multi-byte encodings like UTF-8. TextDecoder handles this for us, making sure our text is correctly reconstructed.

5. Updating the UI

setDataChunks((prev) => [...prev, chunk]);

We append each new chunk to our state array. React re-renders the component with every update, giving us that real-time feel.

6. One-Time Effect

useEffect(() => { ... }, []);

The fetch logic runs once when the component mounts. Perfect for one-time side effects like opening a connection.

✅ Recap

With just a few lines of code, we’ve created a streaming experience in the browser using modern Web APIs. The key things were:

fetch() with a streaming response
ReadableStream + TextDecoder
Updating React state to progressively display data

In Part 2, we’ll build the FastAPI backend to power this stream - including how to set up a streaming route and control flush timing for chunked responses.

💡 Gotchas to Watch Out For

CORS: Make sure your FastAPI server has CORS enabled if frontend and backend are on different ports/domains.
Buffering: Some servers (or even browsers like Safari) buffer streaming responses - so test in Chrome/Edge for best results.
Cleanup: If you’re adding WebSocket or long-running fetches, remember to cancel or clean them up in useEffect cleanup.

📚 References

My plans for sahansera.dev in 2024

Sun, 31 Dec 2023 00:00:00 GMT

Reflecting Back on 2023

It’s no secret that my blogging took a backseat this year - becoming a first-time dad has been a joyful, yet the most challenging journey of my life. It has been a year of new beginnings: a precious addition to the family, moving into a new home, and even transitioning to a new job. Through all this, my commitment to being the best dad possible remained unwavering.

Despite the fewer posts, 2023 had its highlights, including some popular posts and milestones that resonated well with you, my readers. Your support and engagement have been the driving force behind every article, and for that, I am incredibly grateful.

Vision for 2024

As I look to 2024, I’m excited to dive back into more consistent and engaging content for sahansera.dev. Here’s what you can look forward to:

Advanced Software Engineering Topics: I’m planning to delve into more complex issues like event-driven systems and distributed computing. Expect in-depth analyses and practical advice drawn from my personal experiences.
Meta Engineering Insights: I’ll share more about career growth, overcoming industry challenges, and the importance of soft skills. This will be a mix of advice, personal stories, and strategies to help you navigate your career path, all shared freely and openly.
Expanding to Visual Tutorials: Leveraging my position at a leading online graphic designing platform, I plan to produce more visual content, possibly reviving my YouTube channel. This means tutorials, walkthroughs, and maybe some behind-the-scenes content!
Gaming and Inspiration: I’ve rekindled my passion for gaming and might share how it intersects with creativity and programming. If you’ve ever been inspired by gaming in your tech journey, you’ll find these anecdotes and insights particularly relatable.

Engagement and Community Building

My goal is to not only grow the blog to 20,000 monthly active users (a 33% increase in MAU) by the end of 2024 but also to foster a more engaged and interactive community. Here’s how I plan to do it:

Consistent and Quality Content: Focusing on the themes mentioned above, I aim to publish regularly and maintain the high-quality, insightful content you expect.
YouTube Channel Growth: While I don’t have a specific subscriber count in mind yet, creating and sharing at least 10 quality videos over the year is a target I’m setting for myself.

Closing Thoughts

As we wrap up another year, I want to reiterate my commitment to this blog and to you, my readers. Your engagement, feedback, and support are what make sahansera.dev thrive.

I’m optimistic about what 2024 holds and excited to embark on this journey with you. Here’s to another year of growth, learning, and sharing in the vast world of software engineering! 🤗

Basics of Apache Kafka - An Overview

Tue, 03 Jan 2023 00:00:00 GMT

A few months ago I wrote an article on setting up a local Apache Kafka instance. In this article, I will briefly introduce Apache Kafka and some of its core concepts. We will look at the these concepts from a high level and understand how they are put together.

What is Kafka?

Apache Kafka is an open-source distributed event streaming platform for building real-time data pipelines and streaming applications. It is designed to handle high volumes of data and enable real-time processing of data streams.

Kafka is typically used in scenarios where data needs to be processed in real-time, such as in real-time analytics, event-driven architectures, and other streaming applications. It can be integrated with various systems, such as databases, storage systems, and stream processing frameworks, to support a wide range of use cases.

Event Streaming vs Messages Queues

I have seen people using Kafka interchangeably with message queues. However, they are two things. Kafka is a distributed event streaming system whereas message queuing systems like RabbitMQ, Amazon SQS, and Azure Queue Storage.

I suppose confusion must have stemmed mainly from the fact that the feature sets of these systems are getting more or less the same and the line between them is getting blurred. However, it is advisable to understand trade-offs among these systems and pick one that suits you best without overcomplicating the overall solution.

Why Kafka?

There are several reasons why Apache Kafka is a popular choice for building real-time data pipelines and streaming applications:

Scalability: Kafka is designed to handle high volumes of data and to scale horizontally, making it suitable for handling large amounts of data.
Real-time processing: Kafka enables real-time processing of data streams, allowing applications to react to new data as soon as it arrives.
Fault-tolerant: Kafka is fault-tolerant and highly available, meaning it can continue operating even if some of its components fail.
Flexibility: Kafka can be integrated with many systems and frameworks, making it a versatile platform for building streaming applications.
High performance: Kafka has been designed to provide high throughput and low latency, making it suitable for applications that require fast processing of data streams.

In a nutshell, imagine you have the following tightly coupled spaghetti of systems (could be deployed internally within your organization or a public-facing app)

For instance, you could imagine these services to be separate microservices or even monoliths that need to defer their workloads to some other systems. As we can see services 1 - 5 talk to services A - E. This becomes extremely difficult to maintain and will cause massive tech debt.

Following is a refactored communication flow utilizing Kafka. Your line of business applications, or microservices can be added or removed as you go. It also provides a consistent API for both produces and consumers. But, could this be a single point of failure? What if Kafka goes down? or can it go down? You’ll find the answers in the upcoming sections, why that’s not the case.

Putting everything together

Below is a diagram that I’ve created to show you a typical request flow in Kafka. The numbers correspond to each subheading that I will describe in much detail below. In each section, I will zoom in on a particular concept to provide any relevant context.

Before we jump in, I need to mention that Kafka exposes a client API that could be used by users to write and read messages. You could use your programming language of choice to interact with the cluster using these API calls.

1. Producers

Producers write data (or events) to topics, and consumers read data from topics. Producers could be anything like an IoT sensor, activity tracking system, a frontend server sending metrics etc. You can specify a key/id in your message in order to make sure that the messages always end up in the same partition. More on these will be covered in Events, Topics and Partitions sections.

Other than that, it’s worth noting that the Producers can choose 3 methods to send messages (fire and forget, Synchronous send & Asynchronous send) and also they can wait for acknowledgements (none, any or all) as well.

2. Events

An event in Apache Kafka is a record of an occurrence or state change in a system. Sometimes it’s also called a record or a message as well. This could be a user interaction, a change in data, or a notification from another system. Events are stored in Kafka topics and are typically published by producers and consumed by consumers. Events in Kafka are immutable, such that they cannot be changed once they have been published.

Data formats

Kafka sees messages are just byte arrays. However, we humans need a way to serialize and deserialize this data. Therefore, you could use a schema for it such as JSON, Protocol Buffers, or Apache Avro (used in Hadoop).

What’s important to understand is that you use a consistent data format in your applications.

Here is an example of a Kafka event with a key and value that are both strings:

Key: "user_id"
Value: "12345"
Timestamp: "20 Dec 2022"
Headers: {}

3. Topics and Partitions

In Kafka, a topic is a way to separate out different messages coming in from different systems. Each topic is partitioned, meaning that the data in a topic is split into multiple partitions distributed across the Kafka cluster. This allows for redundancy and scalability within Kafka.

If a given topic is partitioned across multiple nodes (i.e. servers), how does Kafka know which message goes to which partition also while preserving the order? This is where the key of a message comes in handy. Kafka uses the key of a message and runs it through a hash function and mod (%) the resulting number so that it knows where to write that message.

What if a message doesn’t have a key (key being NULL)? Well, in that case, Kafka defaults to a Round-Robin approach to write the messages to the partitions.

💡 Messages with the same key go to the same partition thus guaranteeing the order.

It’s also worth mentioning that a partition is not strictly tied to a single server. They can also scale out horizontally depending on your requirements.

Brokers

if you start working on Kafka, you’d come across the term brokers. A broker is just a computer, server, container, or cloud instance that’s running a Kafka broker process. So what do they do? They are responsible for managing the partitions that we talked about in the previous section. They also handle any read/write requests.

Remember I mentioned that a partition can be replicated across multiple nodes? Brokers manage that replication too. But there has to be some kind of way to orchestrate this replication, right? In Kafka, they follow a leader-follower approach to tackle that. Generally speaking, reading and writing happens at the leader level. Then the leader and followers work among themselves to get the replication done.

4. Consumers & Consumer Groups

The “consumer” is a complex concept in Kafka. Consumers are applications that subscribe to a set of Kafka topics and read from topic partitions. Consumers track their offsets as they progress through topic partitions and periodically commit offsets to Kafka, which ensures that they can resume from the last processed offset in the event of a restart. These offsets are incremental and the brokers will know which messages to send when the consumer requests more data.

As shown above, same consumer can also listen to mutiple topics as well.

💡 What sets Kafka apart from traditional Message Queues is that reading a message does not destroy it.

Why is Kafka fast?

Apache Kafka uses a combination of in-memory storage and disk storage to achieve high scalability. Kafka stores data in memory in a log structure, which allows for fast access and processing.

However, as the amount of data in a Kafka cluster grows, it can eventually exceed the available memory. In this case, Kafka will automatically flush data to disk to free up memory. This allows Kafka to scale horizontally by adding more nodes to the cluster, which can provide additional memory and disk storage capacity. By combining in-memory and disk storage, Kafka can achieve high scalability while still providing fast access and processing of data.

How do I build a Kafka cluster?

If your organization has a need to deploy your own cluster at a data center you can do that. Kafka’s deployment targets range from bare metal servers to cloud providers. If you are rolling out your own, beware that it’s not easy to set it up it will take time.

Locally (Development)

Using Docker is the easiest I’ve found. Follow this guide to understand how you can do that.
Or you could YOLO and install the binaries yourself 😀

Cloud (Production)

For your production needs, your best bet would be to use

Different cloud providers might charge you differently. It’s always wise to consider the pricing options before locking yourself into a specific cloud provider.

Conclusion

In this article we looked at Apache Kafka is an open-source distributed event streaming platform for building real-time data pipelines and streaming applications. We went over the main concepts that should be enough to get you started. I’m planning to write an article series on implementing producers and consumers in .NET, Go & Python. So stay tuned! Until next time 👋

References

Serverless Go with Azure Functions and GitHub Actions

Mon, 12 Sep 2022 00:00:00 GMT

In this article, we are going to learn how to host a serverless Go web service with Azure Functions leveraging custom handlers in Azure functions. We will also automate the deployments by using GitHub Actions 🚀 You can even apply the same concepts to other languages Rust as long as you can self-contained binary.

Walkthrough video

If you prefer to watch a video, I have created a walkthrough video on my YouTube channel 😊

Introduction

Azure Functions is Microsoft’s offering for serverless computing that enables you to run code on-demand without having to explicitly provision or manage infrastructure. Azure Functions is a great way to run your code in response to events, such as HTTP requests, timers, or messages from Azure services.

As of today, they support multiple runtimes such as .NET, Java, JavaScript, Python, TypeScript. But what if we want to write our app in Go? Well, now you can do that too - by using Custom Handlers.

Before we move on, so what really is a custom handler and how does it work? Custom handlers let your Function app to accept events (eg. HTTP requests) from the Global host (aka Function host - that powers your Function apps) - as long as your chosen language supports HTTP primitives.

Here’s a great overview from Microsoft on how this is achieved.

Source: Microsoft Docs

So, in our case, we are going to wrap a Go binary as a Function app and deploy it to Azure. Sounds good? Let’s jump right into it.

Prerequisites

Make sure you have the following setup locally.

Go SDK
Azure Functions Core Tools
Prefererrably, Azure Functions VS Code extension

The plan

Clone the repo or create one
Create the Azure function resources
Test out the app
Deploy to Azure

1. Clone the repo or create one

Our code is going to be pretty simple. All it does is, whenever we make a request, it will return a random quote on programming.

💡 You can clone the repo I have created from here.

package main

// Removed for brevity

var quotes = []string{
	"Talk is cheap. Show me the code.",
	"First, solve the problem. Then, write the code.",
	"Experience is the name everyone gives to their mistakes.",
	"Any fool can write code that a computer can understand. Good programmers write code that humans can understand.",
}

func quotesHandler(w http.ResponseWriter, r *http.Request) {
	// Get a random quote
	message := quotes[rand.Intn(len(quotes))]

	// Write the response
	fmt.Fprint(w, message)
}

func main() {
	listenAddr := ":8080"
	if val, ok := os.LookupEnv("FUNCTIONS_CUSTOMHANDLER_PORT"); ok {
		listenAddr = ":" + val
	}
	http.HandleFunc("/api/GetQuotes", quotesHandler)
	log.Printf("About to listen on %s. Go to https://127.0.0.1%s/", listenAddr, listenAddr)
	log.Fatal(http.ListenAndServe(listenAddr, nil))
}

Explanation

We setting port 8080 as the default port for our app. But if we are running this in Azure Functions, we will be using a different port. So, we are checking if the FUNCTIONS_CUSTOMHANDLER_PORT environment variable is set. If it is, we will use that port instead.
We are registering a handler for the /api/GetQuotes endpoint. This is the endpoint that we will be using to make requests to our app.
quotesHandler is a simple function that returns a random quote from the quotes array.

2. Create the Azure function resources

You can create your own Azure function from portal.azure.com or using ARM templates. Below are the configuration I chose.

Publish: Code
Runtime Stack: Custom Handler
Operating System: Linux
Plan type: Consumption (Serverless)

3. Test out the app

If you have cloned the project you should see a folder structure similar to what’s shown below.

At the root of the project folder let’s run the following commands.

go build handler.go # To build a binary
func start          # Start the Function app service

You should see the output like so:

Let’s run through each file now.

GetQuotes/function.json: This file defines what happens when a request comes in and what should go out from the function. These are known as bindings. Our function is triggered by HTTP requests and we will return a response
handler.go: This is our Go web service where our main logic lives in. We listen on port 8080 and expose an HTTP endpoint called /api/GetQuotes
host.json: Take note under customHandler.description.defaultExecutablePath is set to handler which says where to find the compiled binary of our Go app and enableForwardingHttpRequest where we tell the function host to forward our traffic

4. Deploy to Azure with GitHub Actions

Now that we have everything ready to go let’s deploy this to Azure! 🚀 To get this done, we are going to use the Azure Functions Action from the GH Actions Marketplace.

From a high-level this is what we need to do in order to deploy this.

Authenticate with Azure
Build the project
Deploy

Authenticate with Azure

Since our app is written in “Go” which is not really supported out of the box, we won’t be able to use the Publish Profile method for this. So we are going to focus on using an Azure Service Principal for RBAC.

💡 Remember to follow the steps according to this guide to create an SP.

Once you have created the service principal, we need to add that as a secret in our repo so that it can be used for authenticating with Azure Resource Manager during the deployment step. Head over to your repo → Settings → Secrets → Actions
Create a secret a called AZURE_RBAC_CREDENTIALS and update its content with what you got when you created the service principal.

Now the that the credentials are in place we can refer to that from the GitHub Action workflow file.

Build the project

This step is pretty straightforward as we are using the Go SDK to build the project with GOOS=linux GOARCH=amd64 config.

This is what the final workflow file should look like:

deploy.yml

name: CI/CD

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
    
env:
  AZURE_FUNCTIONAPP_NAME: azgofuncapp   # set this to your application's name

jobs:

  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: 'Login via Azure CLI'
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_RBAC_CREDENTIALS }}

    - name: 'Set up Go'
      uses: actions/setup-go@v3
      with:
        go-version: 1.18

    - name: Build
      run: GOOS=linux GOARCH=amd64 go build handler.go
      
    - name: 'Deploy to Azure'
      uses: Azure/functions-action@v1
      with:
        app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}

Once everything is done, head over to the following URL and refresh a couple of times to see our programming quotes! 😀

https://<your_app_URL>.azurewebsites.net/api/GetQuotes

Here’s the an example.

Conclusion

Well, that’s it folks! Today we built a small Go web service, wrapped it in an Azure Function and deployed it to Azure by using GitHub Actions!

Troubleshooting

I ran into couple of issues when creating this project.

Are you getting the Value cannot be null. (Parameter 'provider') error?

I was able to resolve it by following the exact config as below.

{
  "version": "2.0",
  "logging": {...},
  "customHandler": {
    "description": {
      "defaultExecutablePath": "handler",
      "workingDirectory": "",
      "arguments": []
    },
    "enableForwardingHttpRequest": true
  }
}

Are you getting the Azure Functions Runtime is unreachable error?

For me, this went away when I did the first deployment. If it still doesn’t go away, check out this link for more info.

References

Deploying a .NET gRPC Server on Azure App Service

Mon, 05 Sep 2022 00:00:00 GMT

In this article we are going to be deploying a gRPC web service written in .NET to an Azure App Service. We will then interact with it and also automate deployment process of the Azure resources and the build.

Introduction

About a few months ago I wrote a blog series on creating a simple gRPC server and a client locally. This time we are going to take the next steps by deploying it to the cloud.

If you missed the series you can get yourself up to speed with the below links 🚀

Walkthrough video

If you like to watch a video walkthrough instead of this article, you can follow along on my Youtube channel too 😊

The Plan

Clone the above repo.
We will create an App Service instance on Azure and deploy it (automated).
Getting ready for deployment.
Deploying the BookshopServer code.
Testing the web service.

1. Clone the repo (or build one)

Since we already have an app in handy we are going to use the same thing and make this work. More specifically we are going to use the exact code we looked at in the Building a gRPC Server in .NET blog post.

💡 The code can be found under this repo.

We are going to be using the BookshopServer as our web service and BookshopClient as the client to interact with it.

Once you have it cloned, you can set the branch to app-service with,

git checkout app-service

2. Create an App Service instance in Azure

You can head over to portal.azure.com and create the App Service instance manually or you could use the ARM template down below.

💡 To save you some time and trouble, I’d stay away from the F1 tier - it was unreliable for me and had a hard time deploying my code. I went ahead with B1 (and you get 30 days free for Linux). It costs you a little bit, but heaps more reliable for Dev/Testing work.

ARM template for easy deployment

If you haven’t got az cli head over here to get it set up. Once installed, do a az login and select the account you want to use for deployments.

I have created a shell script to automate all the steps. This is the usage of the script.

cd BookshopServer/Infrastructure/
./deploy.sh net6-grpc australiasoutheast template.json parameters.json

The parameters to pass in are as follows:

Resource group name (eg: net6-grpc)
Region you want to create the resources in (eg: australiasoutheast)
Name of the ARM template file (eg: template.json)
Name of the parameters file (eg: parameters.json)

If you are unsure, you can see what’s what just by typing ./deploy as well. Once deployed, you should be able to see something like below.

App Service configuration

The ARM template takes care of most of the configuration for us. However there’s one more thing that we need to enabled, called HTTP 2.0 Proxy

This is what your configuration should look like:

3. Getting ready for deployment

Before we do any deployment there are a couple of things we need to change in our code. These are all done for you in my repo. But for the curious,

BookshopServer/Program.cs

Add/Update the configuration to tell Kestrel to listen on all IPs on port 8080 and also on port 8585 but with HTTP 2.0 only.

// ...
builder.WebHost.ConfigureKestrel(options =>
{
  options.ListenAnyIP(8080);
  options.ListenAnyIP(8585, listenOptions => 
  { 
      listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2; 
  }); 
});

// ...

Now that we have that config we no longer need the configuration specified in appSettings.json. Let’s remove the following lines like so.

BookshopServer/appSettings.json

...
// Delete the following lines
"Kestrel": {
    "EndpointDefaults": {
      "Protocols": "Http2"
    }
  }
...

4. Deploying the BookshopServer code

Let’s go ahead and deploy our code! 🚀 There are many ways to do this. If you are setting it up for testing you can simply deploy from your local Git repo. To keep the scope of this blog post small I’m not going to be covering other continous deployment methods. Let me know if you like me to cover them too!

You can follow these steps from the official docs to quickly deploy from locally. There are some configurations that you need to do.

Go to you app service instance > click Deployment Center
Select Local Git from the Source dropdown
Click Local Git/FTPS Credentials
Set a username and a password under User Scope
Hit Save
(Important) Go back to the app service Overview
Copy the URL under Git Clone URL
Go to your local repo
Do a git remote add azure <Git URL you copied before>
Change deployment branch with git push azure <local branch name>:master

This might take a couple of minutes for the initial deployment. After the last step you should be able to see a log like so:

Although it my seem a little daunting at first, this is only a one-off config. For any code changes you make, you just need to commit your changes to your local branch and then do a git push like we saw in step 10 above.

5. Testing the web service

Now to the interesting part. Let’s see how we can interact with our web service 🤔

For some reason I couldn’t get grpcurl to work, thankfully, we already have the BookshopClient project - so let’s use that!

When you open up the project, make sure to update the URL to point to your App service - and that’s all you need to do!

BookshopClient/Program.cs

using var channel = GrpcChannel.ForAddress("https://your_app_service.azurewebsites.net/");

(make sure you add the / at the end of the URL)

and finally let’s run the project.

dotnet run --project BookshopClient

That should return back the list of books as shown below.

That’s it! now you know how to deploy a gRPC .NET web service in Azure 🎉

Conclusion

In this article we looked at how we can leverage Azure App Service to deploy a gRPC server and interact with it. Until next time 👋

References

Setting up a local Apache Kafka instance for testing

Thu, 01 Sep 2022 00:00:00 GMT

I recently started digging into Apache Kafka for a project I’m working on at work. The more I started looking into it I realized I need to get myself a local Kafka cluster to play around with. This article walks you through that process and hopefully save your some time 😄.

If you don’t know what Kafka is, here’s a great video from Confluent.

You could also check out my blog post to understand the main concepts of Kafka if you are interested from here:

Basics of Apache Kafka - An Overview

💡 You can follow along by cloning the repo I have created over here.

Installing the binaries

There’s a quickstart guide over at the official docs. To be honest, I didn’t want to install anything locally in order to keep the Kafka instance fairly separate my dev environment, hence went with the Docker way which you will find below.

The Docker way

This is by far the easiest I have found and saves you ton of time. Once you have cloned the repo just spin it up!

docker-compose up -d

If you want to know what’s going on in here let’s take a look at the docker-compose.yaml file.

docker-compose.yaml

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

We spin up a Zookeeper instance at port 2181 internally within the Docker network (i.e. not accessible from the host)
Next we spin up our broker which is the Kafka instance which has a dependency on Zookeeper. This is exposed at port 9092 and accessible from the host.

Everything else is set to default. If you want to learn what each environment variable does, head over here. You can add multiple brokers (Kafka instances) if you want in the same docker-compose file.

This approach is quite beneficial if you want to stand up an instance for testing on the go or even in a CI environment.

Interacting with the cluster

Before we publish anything, let’s spin up a consumer so that we know our setup is working. To do that, run the following command from the CLI.

docker exec --interactive --tty broker \
kafka-console-consumer --bootstrap-server broker:9092 \
                       --topic example-topic \
                       --from-beginning

Now let’s create some messages and see them in action!

docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server localhost:9092 \
                       --topic example-topic

This is what it will look like from the terminal. To your left is the producer and consumer to your right hand side.

A GUI perhaps?

Not a fan of the CLI? There’s many tools out there. I wanted to find a nice, lightweight, OSS solution for this. I decided to go ahead with Tools for Apache Kafka for VS Code.

Once you have that going you could interact with it from the side bar. There will be a new icon with Kafka’s logo on it!

Here’s what it looks like when it finds your instance.

producer.kafka

PRODUCER keyed-message
topic: example-topic
key: mykeyq
record content

###

PRODUCER non-keyed-json-message
topic: example-topic
{
    "type": "my_test_event-{{random.number}}"
}

consumer.kafka

CONSUMER consumer-group-id
topic: example-topic
partitions: 0
from: 1

You can invoke the producers and consumers right within VS Code, which is pretty easy and cool!

Conclusion

Welll that’s it! Standing up a docker container certainly did save me some time just for testing purposes. Hope you enjoyed this post and see you next time 👋

References

Adding intelligence to React Apps with Azure Cognitive Services

Sun, 21 Aug 2022 00:00:00 GMT

Hi everyone 👋 This blog post is based on a conference talk I delivered at the React Global Summit 2022 on the 21 April 2022. React Global Summit is a global conference that brings together the best minds in the React community to share ideas, learn from each other, and build awesome products. I presented under teh Senior Developer track.

YouTube Recording

This time I don’t have any text to write, because there’s a video instead 😀 Check out the recording below and don’t forget to like and subscribe.

It’s a bit lengthy so I have segmented it into sections with timestamps. Let me know what you think of the format? Should I continue doing this in the long run? I’d love to hear your thoughts. Thanks!

💡 You can follow along with demo code on GitHub

Slide Deck

The slides can be found at my SpeakerDeck profile.

Conclusion

Let me know what you think of this new format. Until next time 👋

Tips for Junior Developers

Tue, 31 May 2022 00:00:00 GMT

Hi everyone 👋 this blog post kind of different from my usual ones since this is based on a recent talk I delivered at the Adelaide Jnr Developer Meetup hosted by juniordev_io on the 18 May 2022.

The slides can be found at my SpeakerDeck profile.

The following is a short summary of my notes for each slide I spoked about. I structured my talk around 3 main topics.

Maintaining Motivation
Dealing with Imposter Syndrome
Staying Productive

Slide Deck

Here are the slides from my talk.

Maintaining Motivation

Put in the work

Developing software is a creative endeavour. This is no secret. Regardless of your background, whether you are coming from a Bootcamp, uni degree, or doing it as a hobby, you have to do the work yourself to gain experience. You can read a 300-page book or binge-watch a 4-hour-long Youtube tutorial on building the next million dollar idea, but it would be a mere waste if you don’t take any action. If you follow a tutorial or do what your professor or instructor is telling you to do, you won’t grow as a developer if you don’t try things on your own.

It’s so easy to “get stuck in a tutorial hell,” which can create a false sense of accomplishment. So what should you do about it? My suggestion is to carve out some of your free time, maybe 30 minutes a day. Start small. It might be a simple to-do app that you want to build with a programming language you want to learn. Learn the basics first, then focus on different building blocks of it. E.g., Think about the UI; how would you make it go live? How would you save the data? How would you handle multiple users accessing your app?

30 minutes/day actually ends up being 14hrs/month which is a considerable amount when you think about it!

💡 Don’t wait until motivation hits you.

Another correlated point to this is waiting until motivation hits you. You may feel overwhelmed and think that the barrier to entry is high. But once you get through that, your curiosity will grow, and you’d learn things quicker. Most importantly, enjoy and have fun.

Adopt a Growth Mindset

Not sure what to focus on next? It is okay not to understand what is going on when you join a company or a new project. That’s fine. It takes time. Usually, you start small - maybe it’s a minor bug or a feature request to change the button’s color. Start from there and try to understand how things tie together. Ask questions, and reach out to your peers. This is also one of the mistakes I made earlier on. I just put my head down and smash through the tickets. But it’s not enough. By doing this, you will begin to understand more and more.

💡 Learning has a compounding effect

What do I mean by compounding effect? Simply put, initially will take a lot of time, might seem slow, and your growth will seem linear. But once it picks up the pace, it will grow exponentially. And when you stop learning, that cadence breaks, and you need to get back up again. So, Learning is a lifelong process.

This doesn’t mean you always have to be learning something every minute. There will be days you will feel like you didn’t know anything, just be answering emails or slack messages. That’s fine. You need to make sure you don’t do that every single day. By the end of the week, I usually just look back and do a small retro to understand what I learned that week. If not, plan for the next week. Consistency is key.

Few more tips:

On-the-job training is great since you will be spending most of your time during the week at work. Read through technical documents that your team has, code reviews are great to get feedback and also understand what others are working on, run pair programming sessions with your team mates
Utilize the Learning and Development budget your company provides
Perhaps you might be thinking of pursuing a certification?
Network with other peers

Imposter Syndrome

You need to be an expert in something to teach others

As mentioned in my previous slide, you have to make learning a critical part of your life. There’s a lot to learn and keep up with. During the last ten years, I have worked with many technologies, from Ruby, JavaScript, Java, Angular, Android, iOS, .NET, SQL, NoSQL database; some technologies don’t even exist now - J2ME Java Micro Edition, ActionScript, Windows Phone 💀. As you can see, it’s a never-ending process. You will pick up new things now and then, from your job, your side project, from your colleagues. Even today, I still need to look up certain things, Google stuff, when I cannot seem to figure out something.

💡 The most important thing is No one knows everything.

How can we overcome this feeling though? When you are learning something new, don’t feel embarrassed to share anything you learned. Don’t think of it as bragging either.

I initially felt this when I started this blog. I didn’t even think whether anyone would want to read my blog. I just did it anyway. I started it about 2.5 years ago; I initially had 1 or 2 readers – now I have around 10,000/month users and recently passed 200,000 users, which is not a massive number, but I feel happy about it. Adding to that, it’s a great feeling when you get to help someone, and they put a comment saying they have been struggling with it for days, and my blog post solved it. You will get sponsorships and get invited to talks, to name a few.

You must’ve figured out a pattern already - gain experience by doing things. You don’t need to be an expert in something to guide others. You can even share how many times you failed to do something and let others know how you overcame that. Put a unique spin on things. Everyone is unique.

Moreover, you will also learn to say “I don’t know” and that’s 100% fine.

Feeling ‘Everyone else is better than me’

On to the second myth. Having unrealistic goals and comparing yourself with others is a recipe for this. It’s in human nature to compare one’s self to another. You wouldn’t know what they have gone through so we can’t really compare ourselves with others. When you set a standard for success, you’d think you have to achieve it, and if you don’t, you’d think you are a failure. Well, that’s not the case. Because there’s no single path to success, there are multiple paths, and your journey might change from time to time, and that’s totally fine. If you feel that you can learn from the people surrounding you, that’s a good sign! Maybe you have an idea to start a side project, a blog, or a Youtube channel to share your learnings. Follow your passion and do what you love.

Staying Productive

Don’t obsess about the programming language

When starting, yes, you need to know some programming language. Maybe it’s the language of choice of your current employer or a module you are taking currently. The reason is that it’s like learning a human language. Once you know the structure of a language, for instance, grammar, syntax, predicate, whatever that might be, it will be reasonably straightforward to understand and transfer the concepts to a different language. Yes, there will be differences here and there, but you can learn them. Just pick one first and run with it. Get comfortable writing code with it. Don’t try to learn as many languages as possible, there’s too many out there. If you don’t know where to start, JavaScript is a good starting point because you can work on both client-side driven applications (vanilla JS then React, Angular when you are comfortable) as we as the server (e.g., with Node.js). Start to identify those building blocks first. You will then form opinions about things, and perhaps start contributing to open source projects

Learn to use a universal IDE

This is where you will spend your time during your day-to-day job. There are many options out there. IDEs range from VS, JetBrains, or even VIM, to EMACS. Sometimes one is better than the other for a particular stack. Usually, pick something you are most comfortable with and learn how to navigate around, interact with the terminal and the key bindings, and configure it the way you want it to function. Usually, when you join a team, you would have a sharable configuration such as an editorconfig file for different IDEs; if not, you can create one. Talk to your peers about what they are using. Not everybody has to use the same IDE too. Also, be mindful of the capabilities of different IDEs as well. First, try the options out there and get a feel for them. It will take time; eventually, you will land on one. It will become a natural part of your development process, and it’s one less thing to worry too. I use VS Code these days since I’m working on multiple projects because I have all the plugins and a familiar environment. Less context switching.

Take regular breaks

This one’s a bit interesting. It may seem a bit counterintuitive as well at first. Research shows that the human mind can juggle seven things at a time. Imagine you have your project opened up, and there are many classes and methods, and you are going through them to find where the bug is? It’s normal to feel overwhelmed. Perhaps you are debugging a notorious bug and where it’s crashing if you feel like you’ve hit a wall. Just take a break. Go for a walk around the block. You will be amazed by the time you resume your work; you will start to see the answer. I do this very often without being too hard on myself. The same applies to taking a break from your work too. Utilize that PTO, go on a vacation, take some downtime, and take care of yourself. It will pay dividends in the long run and avoid burnout too.

Conclusion

Hope these advice and tips will help you to be a better developer. Until next time 👋

Building a gRPC Client in .NET

Fri, 18 Mar 2022 00:00:00 GMT

Introduction

In this article, we will take a look at how to create a simple gRPC client with .NET and communicate with a server. This is the final post of the blog series where we talk about building gRPC services.

Motivation

This is the fifth part of an articles series on gRPC. If you didn’t catch the previous ones please feel free to do so. The links are down below.

Please note that this is intended for anyone who’s interested in getting started with gRPC. If you’re not, please feel free to skip this article.

Plan

The plan for this article is as follows.

Scaffold a .NET console project.
Implementing the gRPC client.
Communicating with the server.

In a nutshell, we will be generating the client for the server we built in our previous post.

💡 As always, all the code samples documentation can be found at: https://github.com/sahansera/dotnet-grpc

Prerequisites

.NET 6 SDK
Visual Studio Code or IDE of your choice
gRPC compiler

Please note that I’m using some of the commands that are macOS specific. Please follow this link to set it up if you are on a different OS.

To install Protobuf compiler:

brew install protobuf

Project Structure

We can use .NET’s tooling to generate a sample gRPC project. Run the following command at the root of your workspace. Remember how we used dotnet new grpc command to scaffold the server project? For this one though, it can simply be a console app.

dotnet new console -o BookshopClient

Your project structure should look like this.

You must be wondering if this is a console app how does it know how to generate the client stubs? Well, it doesn’t. You have to add the following packages to the project first.

dotnet add BookshopClient.csproj package Grpc.Net.Client
dotnet add BookshopClient.csproj package Google.Protobuf
dotnet add BookshopClient.csproj package Grpc.Tools

Once everything’s installed, we can proceed with the rest of the steps.

Generating the client stubs

We will be using the same Protobuf files that we generated in our previous step. If you haven’t seen that already head over to my previous post.

Open up the BookshopClient.csproj file you need to add the following lines:

...
<ItemGroup>
  <Protobuf Include="../proto/bookshop.proto" GrpcServices="Client" />
</ItemGroup>
...

As you can see we will be reusing our Bookshop.proto file. in this example too. One thing to note here is that we have updated the GrpcServices attribute to be Client.

Implementing the gRPC client

Let’s update the Program.cs file to connect to and get the response from the server.

using System.Threading.Tasks;
using Grpc.Net.Client;
using Bookshop;

// The port number must match the port of the gRPC server.
using var channel = GrpcChannel.ForAddress("http://localhost:5000");
var client = new Inventory.InventoryClient(channel);
var reply = await client.GetBookListAsync(new GetBookListRequest { });

Console.WriteLine("Greeting: " + reply.Books);
Console.WriteLine("Press any key to exit...");
Console.ReadKey();

This is based on the example given on the Microsoft docs site btw. What I really like about the above code is how easy it is to read. So here’s what happens.

We first create a gRPC channel with GrpcChannel.ForAddress to the server by giving its URI and port. A client can reuse the same channel object to communicate with a gRPC server. This is an expensive operation compared to invoking a gRPC method on the server. You can also pass in a GrpcChannelOptions object as the second parameter to define client options. Here’s a list for that.
Then we use the auto-generated method Inventory.InventoryClient by leveraging the channel we created above. One thing to note here is that, if your server has multiple services, you can still use the same channel object for all of those.
We call the GetBookListAsync on our server. By the way, this is a Unary call, we will go through other client-server communication mechanisms in a separate post.
Our GetBookList method gets called on the server and returns the list of books.

Now that we know how the requests work, let’s see this in action.

Communicating with the server

Let’s spin up the server that we built in my previous post first. This will be up and running at port 5000.

dotnet run --project BookshopServer/BookshopServer.csproj

For the client-side, we invoke a similar command.

dotnet run --project BookshopClient/BookshopClient.csproj

And in the terminal, we will get the following outputs.

Nice! as you can see it’s not that hard to get everything working 🎉 One thing to note is that we left out the details about TLS and different ways to communicate with the server (i.e. Unary, streaming etc.). I will cover such topics in-depth in the future.

Conclusion

In this article, we looked at how to reuse our Protobuf files to create a client to interact with the server we created in the previous post.

I hope this article series cleared up a lot of confusion that you had about gRPC. Please feel free to share your questions, thoughts, or feedback in the comments section below. Until next time 👋

References

https://docs.microsoft.com/en-us/aspnet/core/tutorials/grpc/grpc-start?view=aspnetcore-6.0&tabs=visual-studio-code

Building a gRPC Client in Go

Sat, 12 Mar 2022 00:00:00 GMT

Introduction

In this article, we will take a look at how to create a simple gRPC client with Go. We will be using this client to interact with the gRPC server that we created in my previous post. So let’s get into it!

Motivation

This is the fourth part of an articles series on gRPC. If you want to jump ahead, please feel free to do so. The links are down below.

Please note that this is intended for anyone who’s interested in getting started with gRPC, therefore, we will keep things simple.

Plan

The plan for this article is as follows.

Scaffold the client-side stubs and go modules
Implementing the gRPC client
Communicating with the server

In a nutshell, we will be generating the client for the server we built in our previous post.

In the above diagram, we will look at how to achieve the components on the left side.

💡 As always, the completed code can be found at: https://github.com/sahansera/go-grpc/tree/main/client

Creating client-side Stubs and Go modules

We will be using the same Protobuf files that we generated in our previous step. If you haven’t seen that already head over to my previous post.

We will create a new folder called client at the root of the project and initialize it with a new Go module.

mkdir client && cd client
go mod init bookshop/client

Once we have the go modules we can now generate the Protobufs for the client side.

protoc --proto_path=proto proto/*.proto --go_out=client --go-grpc_out=client

This is very similar to our previous blog post.

Implementing the gRPC client

Now that we have the modules we can go and implement the code for the client.

In order for us to talk to the server, we first need to create a connection. For that, we can use the grpc.Dial() method.

💡 Note that we are not using TLS here, however, in production environments you must!

The rough skeleton of the client code looks like the following.

main.go

// ...
func main() {
	conn, err := grpc.Dial("localhost:8080", grpc.WithTransportCredentials(insecure.NewCredentials()))

	// Code removed for brevity

	client := pb.NewInventoryClient(conn)

	// Note how we are calling the GetBookList method on the server
	// This is available to us through the auto-generated code
	bookList, err := client.GetBookList(context.Background(), &pb.GetBookListRequest{})

	log.Printf("book list: %v", bookList)
}

The explanation of this code is as follows:

grpc.Dial is a way to create a client connection to a given target. In our case, we can send in the path of our server along with its port. Note how we are passing in an option to turn off TLS by using WithTransportCredentials
We then call the server procedure/method just as you’d normally do when calling a local method in your program. This is the thing that sold me on gRPC because we know exactly what we have to pass and how to invoke the call.
Now on the server side, we have a request handler that will respond to the incoming requests.
We finally log the response that we got from the server.

Communicating with the server

And in the terminal, we will get the following outputs.

Nice! as you can see it’s not that hard to get everything working 🎉 One thing to note is that we left out the details about TLS. But I guess, that will be posted for another day 😊

Conclusion

In this article, we looked at how to reuse our Protobuf files to create a client to interact with the server we created in the previous post.

I hope this article cleared up a lot of confusion that you had about gRPC. Please feel free to share your questions, thoughts, or feedback in the comments section below. Until next time 👋

References

Building a gRPC Server in .NET

Sat, 05 Mar 2022 00:00:00 GMT

Introduction

In this article, we will look at how to build a simple web service with gRPC in .NET. We will keep our changes to minimal and leverage the same Protocol Buffer IDL we used in my previous post. We will also go through some common problems that you might face when building a gRPC server in .NET.

Motivation

For this article also we will be using the Online Bookshop example and leveraging the same Protobufs as we saw before. For those who aren’t familiar with or missed this series, you can find them from here.

We will be covering steps 1 and 2 in the following diagram.

Plan

So this is what we are trying to achieve.

Generate the .proto IDL stubs.
Write the business logic for our service methods.
Spin up a gRPC server on a given port.

In a nutshell, we will be covering the following items on our initial diagram.

💡 As always, all the code samples documentation can be found at: https://github.com/sahansera/dotnet-grpc

Prerequisites

.NET 6 SDK
Visual Studio Code or IDE of your choice
gRPC compiler

Please note that I’m using some of the commands that are macOS specific. Please follow this link to set it up if you are on a different OS.

To install Protobuf compiler:

brew install protobuf

Project Structure

We can use the the .NET’s tooling to generate a sample gRPC project. Run the following command in at the root of your workspace.

dotnet new grpc -o BookshopServer

Once you run the above command, you will see the following structure.

We also need to configure the SSL trust:

dotnet dev-certs https --trust

As you might have guessed, this is like a default template and it already has a lot of things wired up for us like the Protos folder.

Generating the server stubs

Usually, we would have to invoke the protocol buffer compiler to generate the code for the target language (as we saw in my previous article). However, for .NET they have streamlined the code generation process. They use the Grpc.Tools NuGet package with MSBuild to provide automatic code generation, which is pretty neat! 👏

If you open up the Bookshop.csproj file you will find the following lines:

...
<ItemGroup>
  <Protobuf Include="Protos\greet.proto" GrpcServices="Server" />
</ItemGroup>
...

We are going to replace greet.proto with our Bookshop.proto file.

We will also update our csproj file like so:

<ItemGroup>
  <Protobuf Include="../proto/bookshop.proto" GrpcServices="Server" />
</ItemGroup>

Implementing the Server

The implementation part is easy! Let’s clean up the GreeterService that comes default and add a new file called InventoryService.cs

rm BookshopServer/Services/GreeterService.cs
code BookshopServer/Services/InventoryService.cs

This is what our service is going to look like.

InventoryService.cs

Let’s go through the code step by step.

Inventory.InventoryBase is an abstract class that got auto-generated (in your obj/debug folder) from our protobuf file.
GetBookList method’s stub is already generated for us in the InventoryBase class and that’s why we are overriding it. Again, this is the RPC call we defined in our protobuf definition. This method takes in a GetBookListRequest which defines what the request looks like and a ServerCallContext param which contains the headers, auth context etc.
Rest of the code is pretty easy - we prepare the response and return it back to the caller/client. It’s worth noting that we never defined the GetBookListRequest GetBookListResponse types ourselves, manually. The gRPC tooling for .NET has already created these for us under the Bookshop namespace.

Make sure to update the Program.cs to reflect the new service as well.

// ...
app.MapGrpcService<InventoryService>();
// ...

And then we can run the server with the following command.

dotnet run --project BookshopServer/BookshopServer.csproj

We are almost there! Remember we can’t access the service yet through the browser since browsers don’t understand binary protocols. In the next step, we will to test our service 🎉

Common Errors

A common error you’d find on macOS systems with .NET is HTTP/2 and TLS issue shown below.

gRPC template uses TLS by default and Kestrel doesn’t support HTTP/2 with TLS on macOS systems. We need to turn off TLS (ouch!) in order for our demo to work.

💡 Please don’t do this in production! This is intended for local development purposes only.

On local development

// Turn off TLS
builder.WebHost.ConfigureKestrel(options =>
{
  // Setup a HTTP/2 endpoint without TLS.
  options.ListenLocalhost(5000, o => o.Protocols =
      HttpProtocols.Http2);
});

Testing the service

Usually, when interacting with the HTTP/1.1-like server, we can use cURL to make requests and inspect the responses. However, with gRPC, we can’t do that. (you can make requests to HTTP/2 services, but those won’t be readable). We will be using gRPCurl for that.

Once you have it up and running, you can now interact with the server we just built.

grpcurl -plaintext localhost:5000 Inventory/GetBookList

How do we figure out the endpoints of the service? There are two ways to do this. One is by providing a path to the proto files, while the other option enables reflection through the code.

Using proto files

If you don’t want to enable reflection, we can use the Protobuf files to let gRPCurl know which methods are available. Normally, when a team makes a gRPC service they will make the protobuf files available if you are integrating with them. So, without having to ask them or doing trial-and-error you can use these proto files to introspect what kind of endpoints are available for consumption.

grpcurl -import-path Proto -proto bookshop.proto -plaintext localhost:5000 Inventory/GetBookList

Now, let’s say we didn’t have reflection enabled and try to call a method on the server.

grpcurl -plaintext localhost:5000 Inventory/GetBookList

We can expect that it will error out. Cool!

Enabling reflection

While in the BookshopServer folder run the following command to install the reflection package.

dotnet add package Grpc.AspNetCore.Server.Reflection

Add the following to the Program.cs file. Note that we are using the new Minimal API approach to configure these services

// Register services that enable reflection
builder.Services.AddGrpcReflection();

// Enable reflection in Debug mode.
if (app.Environment.IsDevelopment())
{
  app.MapGrpcReflectionService();
}

Conclusion

As we have seen, similar to the Go implementation, we can use the same Protocol buffer files to generate the server implementation in .NET. In my opinion .NET’s new tooling makes it easier to generate the server stubs when a change happens in your Protobufs. However, setting up the local developer environment could be a bit challenging especially for macOS.

Feel free to let me know if you have any questions or feedback. Until next time! 👋

References

Building a gRPC Server in Go

Fri, 25 Feb 2022 00:00:00 GMT

Intro

In this article, we will create a simple web service with gRPC in Go. We won’t use any third-party tools and only create a single endpoint to interact with the service to keep things simple.

Motivation

This is the second part of an articles series on gRPC. If you want to jump ahead, please feel free to do so. The links are down below.

Please note that this is intended for anyone who’s interested in getting started with gRPC. If you’re not, please feel free to skip this article.

The plan

So this is what we are trying to achieve.

Generate the .proto IDL stubs.
Write the business logic for our service methods.
Spin up a gRPC server on a given port.

In a nutshell, we will be covering the following items on our initial diagram.

💡 As always, all the code samples documentation can be found at: https://github.com/sahansera/go-grpc

Prerequisites

This guide targets Go and assumes you have the necessary go tools installed locally. Other than that, we will cover gRPC specific tooling down below. Please note that I’m using some of the commands that are macOS specific. Please follow this link to set it up if you are on a different OS.

To install Protobuf compiler:

brew install protobuf

To install Go specific gRPC dependencies:

go install http://google.golang.org/protobuf/cmd/[email protected]
go install http://google.golang.org/grpc/cmd/[email protected]

Project structure

There is no universally agreed-upon project structure per se. We will use Go modules and start by initializing a new project. So our business problem is this - we have a bookstore and we want to expose its inventory via an RPC function.

Since this talks about the creation of the server, we will call it bookshop/server

We can create a new folder called server for the server and initialize it by so:

go mod init bookshop/server

In a later post, we will also work on the client-side of this app and call it bookshop/client

This is what it’s going to look like at the end of this post.

Creating the service definitions with .proto files

In my previous post, we discussed what are Protobufs and how to write one. We will be using the same example which is shown below.

A common pattern to note here is to keep your .proto files in their separate folder, so that use them to generate the server and client stubs.

bookshop.proto

syntax = "proto3";

option go_package = "bookshop/pb";

message Book {
  string title = 1;
  string author = 2;
  int32 page_count = 3;
  optional string language = 4;
}

message GetBookListRequest {}
message GetBookListResponse { repeated Book books = 1; }

service Inventory {
  rpc GetBookList(GetBookListRequest) returns (GetBookListResponse) {}
}

Note how we have used the option keyword here. We are essentially saying the Protobuf compiler where we want to put the generated stubs. You can have multiple option statements depending on which languages you are using to generate the stubs for.

💡 You can find a full list of allowed values at google/protobuf/descriptor.proto

Other than that, we have 3 messages to represent a Book entity, a request and a response, respectively. Finally we have a service defined called Inventory which has a RPC named GetBookList which can be called by the clients.

If you need to understand how this is structured, please refer to my previous post 🙏

Generating the stubs

Now that we have the IDL created we can generate the Go stubs for our server. It is a good practice to put it under the make gen command so that we can easily generate them with a single command in the future.

protoc --proto_path=proto proto/*.proto --go_out=. --go-grpc_out=.

Once this is done, you will see the generated files under the server/pb folder.

Awesome! 🎉 now we can use these subs in our server to respond to any incoming requests.

Creating the gRPC server

Now, we will create the main.go file to create the server.

main.go

...

type server struct {
	pb.UnimplementedInventoryServer
}

func (s *server) GetBookList(ctx context.Context, in *pb.GetBookListRequest) (*pb.GetBookListResponse, error) {
	return &pb.GetBookListResponse{
		Books: getSampleBooks(),
	}, nil
}

func main() {
	listener, err := net.Listen("tcp", ":8080")
	if err != nil {
		panic(err)
	}

	s := grpc.NewServer()
	pb.RegisterInventoryServer(s, &server{})
	if err := s.Serve(listener); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}
...

We first define a struct to represent our server. The reason why we need to embed pb.UnimplementedInventoryServer is to maintain future compatibility when generating gRPC bindings for Go. You can read more on this in this initial proposal and on README.
As we discussed, GetBookList can be called by the clients, and this is where that request will be handled. We have access to context (such as auth tokens, headers etc.) and the request object we defined.
In the main method, we are creating a listening on TCP port 8080 with the net.Listen method, initialize a new gRPC server instance, register our Inventory service and then start responding to incoming requests.

Interacting with the Server

Once you have it up and running, you can now interact with the server we just built.

grpcurl -plaintext localhost:5000 Inventory/GetBookList

💡 Note: gRPC defaults to TLS for transport. However, to keep things simple, I will be using the -plaintext flag with grpcurl so that we can see a human-readable response.

How do we figure out the endpoints of the service? There are two ways to do this. One is by providing a path to the proto files, while the other option enables reflection through the code.

Enabling reflection

This is a pretty cool feature, where it will give you introspection capabilities to your API.

reflection.Register(gs)

Following screenshot displays some of the commands you can use to introspect the API.

grpcurl -plaintext -msg-template localhost:8080 describe .GetBookListResponse

Using proto files

If you don’t want to to enable it by code we can use the Protobuf files to let gRPCurl know which methods are available. Normally, when a team makes a gRPC service they will make the protobuf files available if you are integrating with them. So, without having to ask them or doing trial-and-error you can use these proto files to introspect what kind of endpoints are available for consumption.

grpcurl -import-path proto -proto bookshop.proto list

gRPCurl is great if you want to debug your RPC calls if you don’t have your client built yet.

Conclusion

In this article we looked at how we can create a simple gRPC server with Go. In the next one we will learn how to do the same with .NET.

Feel free to let me know any feedback or questions. Thanks for reading ✌️

References

Introduction to gRPC

Fri, 18 Feb 2022 00:00:00 GMT

Intro

If you have built RESTful or other OpenAPI-like APIs for some time and wondering what’s next for you, then you have come to the right place. This article series discusses leveraging gRPC to build your next API, even multiple services. We will initially look at the main concepts from a high-level view and then move on to the implementation aspects of it.

Motivation

There are many tutorials on getting started there. But the main issue I faced was that they even made me more confused as there was a lot of contexts lost in the process and brought in way too many third-party libraries or explained a bunch of steps without emphasizing how different pieces work together. Therefore, I thought to create a guide for anyone who’s interested in getting started with gRPC from a hands-on perspective.

This article is the first part of a series on gRPC. The links are down below. If you want to jump ahead, please feel free to do so.

Background

One of the popular choices for building APIs nowadays is creating a RESTful service. However, before even coming to REST API, we need to look back to see other forms we used to have in the past.

SOAP - Popularized back in the late-90s for building service-oriented architectures (SOA) systems which are known for exchanging bloated XMLs. The benefits were detrimental for distributed applications where the schema was rigid.
REST - Promoted the resource-oriented architecture (ROA)-style distributed applications. Often bulky with JSONs, and everything that the service provides is represented as resources (Eg: /api/v1/users/ or api/v1/books/1234 etc.). Sometimes this could result in exposing too much or too little data at the cost of making multiple HTTP calls.
GraphQL - GraphQL takes a step further and exposes a single endpoint that you can use to query or mutate the data through HTTP verbs. It’s still a request-response model and based on the text-based transport protocol, HTTP 1.x.

What if I want to have some bi-directional communication? None of the above solved that. Then we got technologies like WebSockets and Server Sent Eventing.

WebSockets - Built to support bi-directional communication over a single TCP connection. Known to be a very chatty protocol often sending packets back and forth. If you want to know about WebSockets, here is an article that I wrote.
Server Sent Eventing - Another paradigm where the server sends messages to the client once the initial connection has been set up by the client.

There’s a recurring theme going on in the above technologies. It could be summarized as the messages being (bulky, not strongly typed etc.) and inefficient protocols (text-based such as HTTP 1.x etc.).

Hello gRPC

gRPC was born to address some of the challenges we face in the above approaches. In 2015 Google released gRPC to the open-source world. The idea behind gRPC is to enable developers to use RPC-like communications over HTTP/2 while have a single client library. Whoever is maintaining gRPC will maintain the client libraries. Because HTTP/2 works on binary format, gRPC will also abstract away the HTTP/2 stuff from you. The developers only have to define their service contracts through requests, responses and RPC calls, and the gRPC framework will handle the rest for us.

💡 Even before we start learning about gRPC, you must have also thought about what the “g” in gRPC mean? Some say it stands for “Good”; others say it stands for “Google”. You know what? It doesn’t matter as it doesn’t provide more context. You can find all the different variations here, which is, by the way, hilarious! 😂

Source: https://grpc.io/docs/what-is-grpc/introduction/RPC has been around for some time; however, gRPC approaches RPC in a much cleaner way. Another term that you’d come across when learning gRPC is inter-process communication or IPCs. GRPC is mainly built to cater to inter-process communication that lets you connect, invoke, and operate, an excellent choice for micro-services like applications. In distributed computing realm, inter-process communication or, in short, IPC refers to passing messages (synchronously or asynchronously) where any application or a node can act as both client and a server.

Protocol Buffers

They are a language-agnostic way to define what your service does. These are commonly known as “IDLs” or Interface Definition Language(s).

So the steps are,

You write the messages. These messages have statically typed fields.
You write your services by defining what comes in, what goes out.
Compile the proto files and generate the client libraries for your application.

It’s also worth mentioning that Protocol Buffers are not the only way to define our IDLs. There are other formats like FlatBuffers, Bond etc.

Let’s say we want to model a book shop which can be queried to get a list of books available. This is what a protocol buffer would like:

syntax = "proto3";

message Book {
  string title = 1;
  string author = 2;
  int32 page_count = 3;
  optional string language = 4;
}

message GetBookListRequest {}
message GetBookListResponse { repeated Book books = 1; }

service Inventory {
  rpc GetBookList(GetBookListRequest) returns (GetBookListResponse) {}
}

The above is an example that we will use throughout this blog series. Don’t worry if you don’t understand what’s going on here. We will go in-depth in upcoming posts.

The first line specifies the Protobuf version we will be using. It will be set to proto2 if you don’t specify explicitly.
The Book is a message definition with some statically typed fields such as title, author etc.
GetBookListRequest and GetBookListResponse are also messages composed of the Book type we defined above.
Inventory is a service that says what methods we expose to the remotely invoked clients.

There are many advantages of using Protobufs compared to something like JSON. Once you write the definitions for your service, you can share them with other teams and use them to generate stubs/code that can interact with your service.

Another advantage is that Protobufs are binary encoded. The payload is smaller than JSON, which means it would be efficient to send. This also means that it will use fewer CPU cycles to serialize/deserialize the messages.

gRPC Server & Client

Now that we have the Protobuf definitions for our service, we could generate the Server-side and Client-side implementations using the Protoc compiler.

We first create the definition of the service with a .proto file
We then generate the server-side code in our preferred language (Go, C#, Java etc.). This code includes the boilerplate code to serialize, deserialize, functions for receiving and responding to messages.
We then generate the client-side code in our preferred language (doesn’t have to be the same language we chose for the server). This includes methods that we can invoke on our server with additional code to serialize, deserialize messages.
Depending on which gRPC mode we choose, the client-server communication happens over an HTTP/2 connection. We will discuss more on these modes in the next section.

gRPC Modes

There are 4 modes of gRPC communication styles. Following are their brief introductions. Feel free to go more in-depth by reading through the official docs.

Unary RPC - More like our traditional APIs where we send a request and receive a single response.
Server Streaming RPC - Client sends a request and reads until the server stops sending messages via a stream.
Client Streaming - Reverse of the above, the client sends messages through the stream and waits for the server to read and return a response.
Bidirectional streaming RPC - Pretty much both (2) & (3) combined - both client and server streams messages both ways.

Pros and Cons

Any technology comes with a set of advantages and disadvantages. Whether you choose to use gRPC might depend on some of these factors.

Pros

Efficient for inter-process communication with all the good stuff that comes with HTTP/2.
Well defined interfaces to be able to communicate also while supporting polyglot development.
Code-generation of client and server stubs with strong types.

Cons

It may not be suitable for external-facing services since most web browsers’ support is limited.
Changing the service definitions might require rework and regeneration of code.
Could be a steeper learning curve compared to other RESTful or GQL like architectural styles.

Conclusion

In this article, we looked at gRPC from a high level. In the next article, we will look at how we can put these into action and generate a gRPC service with Go. Feel free to let me know any feedback or questions. Thanks for reading ✌️

References

Automating Raspberry Pi K3s provisioning with Ansible

Wed, 15 Dec 2021 00:00:00 GMT

In this article, we will be looking at how we can automate most of the steps involved in installing a Raspberry Pi+K3s cluster by using Ansible.

I explained how to set up a Kubernetes cluster on a Raspberry Pi in my previous article. In case you missed it, you can find it from the following link:

🔗 Building your own private Kubernetes cluster on a Raspberry PI 4 with K3S

If you have a fresh bunch of Raspberry Pis, you might want to follow Step 0 through to Step 8 in the above. However, it’s a one-off setup, so please proceed to the following sections if you already have done them.

In a nutshell, this is what we are trying to automate.

Step 1 - Installing Ansible

So what do we need to get started? You need one or more RPis with Raspbian installed (or an OS supported by K3s). We can get to our desired state by using an Ansible playbook.

Ansible is an industry-leading IT automation tool to provision and manage your infrastructure in a declarative form such YAML.

The easiest way to get the steps to installing Ansible would be to head over to the official docs. For most OSs, it can be just installed via pip.

python -m pip install --user ansible

💡 Note: Use the latest version of Python installed on your machine. In my case, I have it set to python3 since Mac OS comes with an older version of Python installed.

Depending on your system and the internet connection, it will a bit of take time to install Ansible. Sometimes even if you installed it, it would still say that it can’t find ansible command. If that’s the case, feel free to follow this link. If you are on a Mac, you could also run a brew install ansible to install ansible.

Step 2 - K3s Ansible Playbook

Without writing everything from the ground up, we will be using the k3s-ansible repo.

git clone [email protected]:k3s-io/k3s-ansible.git

Once you have cloned the repo, you can run the following command to create a new playbook for your RPi cluster provisioning. Make sure to replace my-cluster with the desired name for your cluster. In my case, I called it rpi-cluster

cp -R inventory/sample inventory/my-cluster

Next, we need to map out the nodes and master them in the inventory/my-cluster/hosts.ini file. Update the [master] and [node] attributes accordingly.

[master]
10.0.0.100

[node]
10.0.0.101

[k3s_cluster:children]
master
node

Under [node] attribute you could even specify a range of IPs if they are consecutive like so.

[node]
10.0.0.[101:105]

Check the inventory/my-cluster/group_vars/all.yml to set variables such as k3s_version and ansible_user, depending on your environment.

Step 3 - Running the Ansible Playbook

ansible-playbook site.yml -i inventory/rpi-cluster/hosts.ini --ask-pass

You can now SSH into the Pi master or nodes and run kubectl commands within it.

if you want to uninstall everything that got installed as part of the playbook:

ansible-playbook reset.yml -i inventory/rpi-cluster/hosts.ini --ask-pass

Conclusion

This article looked at how we can automate provisioning our Raspberry Pi with Ansible. If you are new to DevOps and want to automate provisioning systems, this is a great start. A special shoutout to @itwars for putting this together and doing a lot of heavy lifting creating these scripts.

References

https://github.com/k3s-io/k3s-ansible

Go for .NET Developers

Fri, 10 Dec 2021 00:00:00 GMT

This blog post kind of different from my usual ones since this is based on a recent talk I delivered at the Adelaide .NET User Group on the 09 Dec 2021.

The slides can be found at my SpeakerDeck profile.

The GitHub repo for the demo code can be found here

The talk is structured in to 12 sections,

Conclusion

Hope this will help if you are planning to learn Go as a .NET developer. Until next time 👋

A closer look at commands and args in Kubernetes Pods

Mon, 13 Sep 2021 00:00:00 GMT

In my previous blog post, we looked at how commands and args work in Docker. If you missed that one, the link is down below.

A closer look at commands and args in Docker containers

Today we will be looking at the same but in Kubernetes. To get started, we will be building on top of the following Dockerfile.

FROM debian
ENTRYPOINT ["printf", "Hello from %s\n"]
CMD ["Name Printer"]

This would simply print out Hello from Name Printer when you run it.

As a time-saver, I have already packaged up the image. You can have look at the image layers in Docker Hub.

You could run this with Docker like so.

docker run sahan/name-printer:latest

That should give you Hello from Name Printer as the output.

Now let’s do the same thing in Kubernetes. I’m a big fan of imperative commands in k8s. You can simple create resources without having to write definition files for simple things.

kubectl run test --image=sahan/name-printer:latest

💡 You can quickly delete the pod with kubectl delete pod test --grace-period=0 --force command.

Let’s export the pod definition to a file.

kubectl run test --image=sahan/name-printer:latest --dry-run=client -o yaml > pod.yml

In the pod definition file, we will add a new field called args under spec.containers.args like so:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test
spec:
  containers:
  - image: sahan/name-printer:v1
    name: test
    args: ["Kubernetes"]
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Running and building this gives you Hello from Kubernetes on the console

kubectl create -f pod.yml

This makes sense because we didn’t change/override the ENTRYPOINT. Whatever we define as args simply gets appended to that. So how can we change the ENTRYPOINT’s behaviour?

We can use the command attribute to define whichever command we want to run in it.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test
spec:
  containers:
  - image: sahan/name-printer:latest
    name: test
    command: ["date"]
    args: ["-u"]
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Now when we run the new pod definition it should print out the current date-time in UTC format. As you can see it’s easy to override the ENTRYPOINT’s behaviour in the container.

One thing to take note of is, remember that we had bash as CMD set in our that why we didn’t have to pass in that. If you have a container that and you want to invoke a command, you can simply prepend /bin/bash -c to the command you want to run.

Conclusion

That’s it for this one - hope it’s useful to understand how commands and args work in pods.

Building your own private Kubernetes cluster on a Raspberry PI 4 with K3S

Sat, 14 Aug 2021 00:00:00 GMT

In this article, we will look at setting up your own private Kubernetes cluster on a Raspberry Pi using K3S in your home Wi-Fi network! Our setup will be pretty simple - single master and a single worker node. You can always add more nodes if you like.

Here’s a diagram to give you an idea of what we will be building today.

Topology of what we'll be building today

Basically, we’ll be tying two Raspberry Pis together with K3S and connecting them to a local WLAN. We can then deploy our workloads using a client (your laptop) which will be talking to the Kubernetes API just as you’d normally with any other K8S cluster.

Note that the local addresses shown in this post might be different what you have got in your home network.

Recent updates (15/08/2021)

These updates are primarily courtesy of Chris Carr who suggested this tip!

You can enable ssh, set the hostname and wireless network credentials before flashing the micro SD card. Simply press Cmd+Shift+X (on Mac) or Ctrl+Shift+X (on Windows) to get to the Advanced options window. This will make the steps 1, 3 & 8 redundant now. I have kept them in case you’d want to do them manually.

Step 0 - The initial setup

First, we need to set up the Raspberry Pis to have an OS, enable SSH, little bit of configuration to be able to use K3S.

💡 If you haven’t done a setup on Raspberry Pi in headless mode before, I have a post describing just that. Note that this is for Desktop version if you want to use them for other stuff as well. That post can be found here: https://sahansera.dev/setting-up-raspberry-pi-4-headless-mode/

I have listed down the specs and the OS versions I used for my kit.

2 x Raspberry Pi 4 Model B - 4GB RAM
2 x 128GB Micro SD Cards
Raspbian OS Lite edition (based on Debian Buster)
Stackable case

Here’s a sneak peek of my build ✌️

You can now use Raspberry Pi Imager without using Balena Etcher to flash the micro SD card. This will make selecting the OS version and flashing process easier.

In summary, we will be looking at doing the following steps.

💡 You need to do the following configs to both (or more) of your Raspberry PIs

Enable ssh on both RPIs (manual)
Enable cgroups
Set up wireless (manual)
Enable 64-bit mode at the kernel level
Booting up
Set up IP tables
Assign static IPs
Change the hostnames (manual)
Install K3S Server on Master
Install K3S Agent on Worker

Before booting up, we need to do a couple of configurations as mentioned below.

Step 1 - Enable SSH (Manual)

Open up the root of the micro SD card (this would be mounted as boot) you just flashed. Create a blank file named ssh at the root of that folder.

touch ssh

Step 2 - Enable cgroups

cgroups is an essential kernel level feature which underpins the containerisation technology. This allows the processes to run in isolation with a specific set of resource assigned to it.

Let’s open up the root of the micro SD volume (this would be mounted as boot) you just flashed and edit the cmdline.txt file.

Add the following line to the end of the file.

cgroup_memory=1 cgroup_enable=memory

You might be wondering why are we doing this. Here’s an excerpt from the K3S docs:

💡 Standard Raspbian Buster installations do not start with cgroups enabled. K3S needs cgroups to start the systemd service. cgroups can be enabled by appending cgroup_memory=1 cgroup_enable=memory to /boot/cmdline.txt

Step 3 - Setting up wireless mode (Manual)

I don’t carry around my Raspberry Pi setup 😆, so I opted in to use my home Wi-Fi network instead of ethernet. Here’s how I did it.

While at the root of the /boot/ volume, create a new file called wpa_supplicant.conf file and add the following lines to it.

country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
    ssid="your-networks-SSID"
    psk="your-networks-password"
}

Take a note of the country field (you can find them in here), and make sure to replace that with the corresponding country code that suits you. ssid and psk would be the name of your wireless network and the password, respectively.

Step 4 - Run the kernel in 64-bit mode

Next up, we will tell the Raspbian OS to run the kernel in 64-bit mode. This is required for K3S.

Open the config.txt file and add the following line to the bottom.

arm_64bit=1

That’s it! Now insert the micro SD card to the Raspberry PI and boot it up.

Step 5 - Booting up

When you boot up the RPIs for the first time it will take a couple of minutes to appear on your home network.

If you log in to the router dashboard, you’ll be able to see the RPIs with their IP addresses.

You can log in to them via ssh like so.

ssh [email protected]

Note that pi is the default user and raspberry would be the default password.

💡 Make sure to change the default password with passwd command when you log in for the first time to both nodes

Step 6 - Enabling static-IP configuration

When we boot up the RPIs K3S connects to the worker nodes by using their IPs. Since we will be using WLAN, if we restart our nodes, the IPs would be different and this setup would not work. So we need to add a piece of config to assign static IPs to them.

There are a couple of ways you can do this. Best option would be to use your router’s DHCP server capabilities and make an address reservation. You’d need the device name and MAC address of the RPIs.

If not, you can edit the /etc/dhcpcd.conf file and let the RPI know which IP to assign itself. I would recommend to add this just below “Example static IP configuration” section so that it’d be easier to find it in the future.

interface wlan0
static ip_address=10.0.0.100
static routers=10.0.0.1
static domain_name_servers=8.8.8.8

Once done, it would look like this.

Make sure to update the IP addresses for each RPI.

Step 7 - Setup IP Tables

sudo iptables -F
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Step 8 - Change the hostname to something sensible (Manual)

sudo vi /etc/hostname

This file will only contain a single line, so you can name it something that makes sense to you. I renamed mine as controlplane for the master node and node01 for the worker node.

Next, change the /etc/hosts file where it says raspberrypi to the name of your node.

sudo vi /etc/hosts

Here’s an example for the node01 node.

💡 Make sure to reboot the RPIs once you have done all these changes.

Step 9 - Installing K3S on the Master node

To install K3S on your master node, run the following command. It will do all the bootstrapping it needs to do under the hood.

curl -sfL https://get.k3s.io | sh -

As a side note, I always run a sudo apt update && sudo apt upgrade before installing anything.

Step 10 - The worker node setup & agent registration

Before set up the worker node, we need to take the token from the server. You need to run the following commands on the master node in order to get this token.

sudo cat /var/lib/rancher/k3s/server/token

Then you need run the following command on the worker node. Make sure to update them according to your environment.

curl -sfL https://get.k3s.io | K3S_NODE_NAME="node01" K3S_URL="https://10.0.0.100:6443" K3S_TOKEN="token from above step" sh -

Explanation of the variables:

K3S_NODE_NAME - name of the worker node you are configuring. Remember that we set up the hostnames in step 7.
K3S_URL - the IP address of your master node. The default K3S server port is 6443, so keep it unchanged.
K3S_TOKEN - Token that we received from the from the K3S server. Eg: K10141483xxxxxxxxxx::server:xxxxxxxxxxxx

You’d see [INFO] systemd: Starting k3s-agent message and it’s ready to go!

That’s it! you can now run commands and see it in action. You will need to sudo su in order to run commands on the master node.

kubectl get nodes

Conclusion

By the end of this article, I hope you have your k8s cluster up and running. If you ran into issues, let me know in the comments below. In the next article, we will look at deploying a sample application and see the cluster in action.

I’m hoping to publish a script to automate most if not all of these steps pretty soon. Until next time 👋

References

Tips and Tricks to Ace the Certified Kubernetes Application Developer

Mon, 02 Aug 2021 00:00:00 GMT

I recently passed the Certified Kubernetes Application Developer exam and thought to share some tips and tricks that might come in handy if you are also planning to take the exam in the future.

📔 Background

About a month ago, I decided to learn more about Kubernetes as it would be really useful for the stuff I’m working at GitHub daily. Prior to that, I was always fascinated by Kubernetes but never got the chance to work on an actual system that used it. I knew how it worked from a 10,000 feet view, but didn’t have an idea of core components, basic constructs and literally to be able to do anything with it.

Having taken the exam, I’m quite comfortable navigating through Kubernetes and now it makes sense when I’m doing something with it, rather than merely following some commands.

CKAD is a hands-on exam and managing your time is absolutely crucial. I hope you find the following tips useful✌️

🗒️ Summary of the exam

To summarize the key facts about the CKAD exam,

Passing score is 66%
2 hours duration, comprised of 19 questions
Questions will have varying weights (from 2% - 13%)
You can also open only one tab to browse Kubernetes documentation
Remotely proctored

💻 Aliases and bash tricks

This is a really important first that I can’t recommend enough. I was using the full kubectl command during the study phase but later started using just k by setting up an alias when I was practising simply to cut down the time when typing commands.

alias k=kubectl

Initially, it will take a few seconds to type this out but it will pay dividends throughout the exam. Here are a few more if you are interested. You don’t need to use everything in here though. In fact, I only used the above alias.

Feel free to mix and match the commands you are comfortable with 👍

alias kd='kubectl describe'
alias kr='kubectl run'
alias kc='kubectl create'
alias ke='kubectl explain'
alias kgp='kubectl get pods'
alias kgs='kubectl get svc'

You don’t need to be a Linux guru to take the exam, but, remember you will do it in some Linux env. (potentially Ubuntu). So it helps to know a few basic Bash commands if you are coming from Windows.

cp - Copy files
mv - Move/Rename files
mkdir - Create new folder
ls - List files
rm - Remove/Delete files
grep - Search through text. Useful when you want to filter a list of pods. Eg: kubectl get pods | grep -i status:
Ctrl+R - To do a reverse search to find a command you have previously run

Extra tip: Use short names of resources whenever possible.

Not sure what are the short names? You can check it with kubectl api-resources command.

⌨️ Get a good grasp of VIM

I found having previous experience in VIM came in handy. However, you don’t need to be a master at it. Using nano would be fine too if you are good.

Take the time to set the following to your VIM profile before attempting any questions.

vi ~/.vimrc

Add the following lines and save it.

set expandtab
set tabstop=2
set shiftwidth=2

These commands will save you from having indentation issues and weird syntax issues while working with YAML files during the exam.

Here are some other commands that may be of help if you are not familiar with VIM.

/ - Search through text. Also, use n to go to the next result.
dd - Delete a line
u - Undo
Shift+A - Go to the end of the line and enter the INSERT mode
gg - Go to the beginning of the file
G - Go to the end of the file
o - Go to the next line and enter INSERT mode
v - Enter VISUAL mode. You can select a block of lines with arrow keys or j and k keys. You can copy with y and paste with p . Also, you can indent a block with Shift + > to right and Shift + < to indent to the left

And finally, while you are in NORMAL mode you can type ZZ to quickly save and go back to the terminal without having to type :wq How cool is that? ⚡

☄️ Mastering the imperative commands

You would come across many questions where you would have to create pods, deployments, services etc. In such cases, don’t bother writing up YAML definitions from scratch - or even finding the relevant reference in the k8s docs.

You can save a lot of time by using imperative commands. For instance, if you are tasked to create a pod with nginx as the image, tier:frontend as labels with the port 80 exposed:

kubectl run tmp --image=nginx --labels tier=frontend --port 80

Say you are asked to expose a deployment nginx with a NodePort service called nginx-svc,

kubectl expose deploy nginx --name=nginx-svc --port=80 --target-port=80 --type=NodePort

But what if you can’t get everything included in a single command you can use the --dry-run=client -o yaml > tmp.yaml to export it to a file before creating the resource.

Oh btw, if you need to delete a pod quickly you can use the --grace-period=0 --force command to quickly delete them without waiting.

kubectl delete po <pod name> --grace-period=0 --force

🤔 When in trouble

Pay attention to the weightage of the question and a rough idea of how long it will take you to solve it. I can remember, I was looking at a question that was quite long and had a fair bit of configuration to be done. But the weightage was only 2% 😆 I marked it down on the provided notepad and skipped it (you can also Flag a question). The next question was 4% and was really really easy! I hope you get the point.

💡 Don’t be afraid to skip and revisit questions.

If you forgot how something is placed in a resource definition, you can use kubectl explain <resource name> --recursive | less to find what you are looking for.

Another useful tip I can give you is, the kubectl <resource name> -h command. You can use it like so.

k run -h

☝️ A note on clusters & namespaces

This is also a very important point you should pay attention to. At the top of each question, if you will be given a command to set the current context. Make sure to run it for each question as different questions will be in different clusters.

Another point is, pay attention to any namespaces in the given question text. Sometimes it will be worded within the question. Sometimes it will be at the bottom of the question as a separate note!

In a question where you will have to ssh into servers please make sure to remember (or note it down) which cluster and server you are in. And remember to exit out of it before going to the next question.

📄 Leverage the docs

In certain cases, it’s better to visit the docs than to spend time to figure out what needs to be done. For instance, if there’s a question on setting up a Persistent Volume, the question will also have a section to create a Persistent Volume Claim and to create a Pod to use that.

Go to the docs, type pv at the search bar and click on the link that says “Configure a Pod to Use a PersistentVolume for Storage”. And yes, you need to know where things are at within the K8S docs!

👟 Practice, practice, practice

Speed is key to the exam. Although you get 2 hours, it will just fly! 🦅

When you pay for the exam you will get 2 free mock exam sessions before sitting the real exam.

As Jeremy Clarkson would say, "SPEEEEEEEEED!!!!" 😂

Here are some more exercises I used.

👋 Conclusion

Do you know what’s the hardest thing to do after the exam? waiting for the results! 🤣 It might take up to 24 - 36 hours to get your result. Here’s my certificate if you are interested.

I hope you found these tips helpful. Feel free to comment below if you have got any tips and tricks too! Good luck with your exam!!! 🎉

Setting up Elastic Workplace Search with Docker

Fri, 23 Jul 2021 00:00:00 GMT

Recently I started looking into Elastic Enterprise Search’s Elastic Workplace Search offering. The configuration was not quite obvious when I first started off with the Docker version of it. The goal of this blog post is to walk you through the process of setting it up with Docker for local development.

💡 Note: This is only meant for local development. Consider their cloud offering for production use.

Prerequisites

Docker Desktop
Docker Compose
Allocate docker memory to more than 4GB (really important to check this)

tl;dr

If you are looking for something to get started quickly, all you have to do is the following.

Clone this repo.
Update the environment variables in the .env file. You can run openssl rand -hex 32 to generate values for ENCRYPTION_KEYS variable.
Run docker-compose up
Go to http://localhost:3002 and login as enterprise_search and the password you set in the above step.
[Optional] You can jump ahead to the “Indexing sample data” section to ingest some dummy data into the index.

But, if you are feeling adventurous, read on! 🤘

Creating the docker-compose file

After a bit of trial and error the I came up with the following docker-compose file. The only difference between the above is that this version doesn’t use a .env file and we need to go through some steps to set up each service.

version: "3"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.4
    environment:
      - "node.name=es-node"
      - "discovery.type=single-node"
      - "cluster.name=enterprise-search-docker-cluster"
      - "bootstrap.memory_lock=true"
      - "xpack.security.enabled=true"
      - "xpack.security.authc.api_key.enabled=true"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  enterprisesearch:
    image: docker.elastic.co/enterprise-search/enterprise-search:7.13.4
    environment:
      - "elasticsearch.host=http://elasticsearch:9200"
      - "ent_search.auth.default.source=standard"
      - "elasticsearch.username=elastic"
      - "elasticsearch.password=password"
      - "secret_management.encryption_keys=[changeme Eg: ddc8XXXXXXXXXXXXXXXXXXXXc1157]"
      - "allow_es_settings_modification=true"
      - "JAVA_OPTS=-Xms2g -Xmx2g"
      - "ENT_SEARCH_DEFAULT_PASSWORD=changeme"
    ports:
      - 3002:3002
    depends_on: ['elasticsearch']

Note that I haven’t exposed the elasticsearch service to the host, but feel free to do so by adding ports section under the service definition.

First, set the ENT_SEARCH_DEFAULT_PASSWORD to something you can remember. This will be the default password to be able to login in to the Enterprise Search instance.

For the secret_management.encryption_keys use the following command to generate a secure key. Note that should be at least one value in that list.

openssl rand -hex 32

Don’t worry about setting up elasticsearch.password for now as we will be setting it up in a later step.

Configuring Elasticsearch

If you check the logs for enterprisesearch container now, you’ll see that it’s failing to authenticate against our elasticsearch instance. Don’t worry about it for now. We are going to set it in the below step.

To fix the auth, let’s connect to our elasticsearch container. The easiest way would be to do that from the Docker Desktop interface, or you could use your shell-fu as well 🤺

Once logged in, run the following command.

bin/elasticsearch-setup-passwords auto

Once you get the auto-generated passwords for each user, you will be able to see the following.

Changed password for user apm_system
PASSWORD apm_system = XXXXXXXXXXXXXXXXXXXX

Changed password for user kibana_system
PASSWORD kibana_system = XXXXXXXXXXXXXXXXXXXX

Changed password for user kibana
PASSWORD kibana = XXXXXXXXXXXXXXXXXXXX

Changed password for user logstash_system
PASSWORD logstash_system = XXXXXXXXXXXXXXXXXXXX

Changed password for user beats_system
PASSWORD beats_system = XXXXXXXXXXXXXXXXXXXX

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = XXXXXXXXXXXXXXXXXXXX

Changed password for user elastic
PASSWORD elastic = XXXXXXXXXXXXXXXXXXXX

Make note of the password for elastic as we will be using that to configure the enterprisesearch instance.

Or you could run the following command to quickly generate and grab the password for the elastic user.

bin/elasticsearch-setup-passwords auto -b | grep "PASSWORD elastic =" | cut -d '=' -f2 | xargs

Configuring Elastic Enterprise Search

Since we have already set up the ES instance in the above step, setting up EES will be easy.

This time we need to connect to the enterprisesearch container and update the elasticsearch.password environment variable.

vi config/enterprise-search.yml

Update CHANGE_THIS with the password you received for elastic user in the above step. Make sure to restart the container once that’s done and navigate to localhost:3002 You should be able to see the following welcome screen.

Log in with the enterprise_search and the password that you set for ENT_SEARCH_DEFAULT_PASSWORD in the docker-compose-yml file.

Also, if you have a look at the container logs you will be able to see the following as well. If you can see the below, that means you are good to go!

#########################################################

*** Default user credentials have been setup. These are only printed once, so please ensure they are recorded. ***

username: enterprise_search

password: XXXXXXXXXXXXXXXXXXXX

#########################################################

Once logged in, you will be able to see the following dashboard! 🙌

Indexing sample data

Once you log in there will be many options available for you to import from. To keep things simple, I will be using a custom source. Once you have set it up, you can do a curl request to index some sample data.

curl -X POST http://localhost:3002/api/ws/v1/sources/<put the source id here>/documents/bulk_create \
-H "Authorization: Bearer <put the access token here>" \
-H "Content-Type: application/json" \
-d '[
  {
    "id" : 1234,
    "title" : "The Meaning of Time",
    "body" : "Not much. It is a made up thing.",
    "url" : "https://example.com/meaning/of/time",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "id" : 1235,
    "title" : "The Meaning of Sleep",
    "body" : "Rest, recharge, and connect to the Ether.",
    "url" : "https://example.com/meaning/of/sleep",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "id" : 1236,
    "title" : "The Meaning of Life",
    "body" : "Be excellent to each other.",
    "url" : "https://example.com/meaning/of/life",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

Make sure to change <put the source id here> and <put the access token here> in the above. You can get those values from the Credentials section of the custom source.

You should be able to see the content once the request has been made. That’s it! 🎉

Conclusion

In this article, we looked at setting up Elastic Workplace Search with Docker containers to improve the developer experience without having to use a cloud option for dev work. We looked at two options a shorted version and a longer version where we dived deeped into how to set it up manually as well.

References

A closer look at commands and args in Docker containers

Tue, 22 Jun 2021 00:00:00 GMT

Today we will be looking at how we can make use of commands and args in Docker to run our own processes when we spin up containers. This could be useful if you want to run one-off tasks without having to create your own images.

Let’s imagine that we want run a bash command. What’s the easiest way to do this? Just by spinning up a container using a lightweight image like busybox, ubuntu, debian? For instance, let’s say we use Debian as our base image.

docker run debian echo hello

We will get back “hello”, which is expected. What would happen if we check if the container is running?

The container has stopped running. Containers are not meant to host an operating system compared to a VM. When you run a basic container like Debian or Ubuntu, it will exit once you perform a docker run debian

However, in cases like Nginx, or Redis they will keep running unless you shut it down or it crashes for some reason. How does this happen?

Under the hood

Let’s have a look at their Dockerfiles. Note that these are oversimplified versions of these files (except for Debian).

Each of these files defines a CMD instruction at the bottom which tells them to run some command. In the case of Debian, it issues just bash which is just a shell, not a process. If you already spotted, both Redis and nginx Dockerfiles have definitions to spin up their processes - the Redis server and Nginx web server, respectively.

The `CMD` instruction

You would be wondering how would one go about customising which commands to run when you spin up a container. One way to specify commands when you start a container is by issuing the name of the executable when we run the container. For example,

docker run debian echo hello-world

If you check the container status, you will see that your container has stopped. Can we make this permanent? Yes, that’s why we need to have a Dockerfile to specify what’s our desired state when the container runs.

Say we use the CMD instruction in our Dockerfile, there are 3 ways to specify this according to Docker docs.

CMD ["executable","param1","param2"] (exec form, this is the preferred form)
CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
CMD command param1 param2 (shell form)

💡 There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.

Test1.Dockerfile

FROM debian
CMD ["printf", "Hello from Test1"]

How can we dynamically define what gets printed? One caveat here is that if we are to pass in command line args it will not get appended to the existing CMD instruction we have set. Docker will try to execute as a command and our CMD instruction will get replaced. Let’s give this a test.

If you run docker run test1 hello, it will result in a “starting container process caused: exec: “hello”: executable file not found in $PATH: unknown.” error.

We can still pass in a command like so: docker run test1 printf hello but that beats the purpose of having a Dockerfile in the first place.

The `ENTRYPOINT` instruction

How can we specify something to run always so that we only need to specify its arguments. Let’s create another file called Test2.Dockerfile.

Test2.Dockerfile

FROM debian
ENTRYPOINT ["printf", "Hello from %s\n"]
CMD ["Test2"]

The ENTRYPOINT instruction says Docker that whatever we have specified there, must be run when the container starts. In other words, if we are to pass in arguments, they will get appended to what we have specified there as the ENTRYPOINT.

docker build -t test -f Test2.Dockerfile .

If we run this now, we get the following.

Cool! If we pass in a parameter now, it gets picked up and replaces the CMD instruction we had at line no. 3; but it will still show the output as we have defined in our ENTRYPOINT. This makes sense since there can be only one CMD instruction per container.

You would be thinking what’s the purpose of having CMD if it gets replaced? It’s really handy when you want to specify a default value.

What would happen if we specify multiple arguments?

What happend here? Let me know in the comments below what are your thoughts on this 😊

Conclusion

In this article we looked at passing in commands and args to container when we spin them up. In the next article, we will compare this to a Kubernetes Pod definition and figure out how we can do the same.

References

https://docs.docker.com/engine/reference/builder/

ASP.NET Core Health Checks

Thu, 25 Mar 2021 00:00:00 GMT

When you are developing a project where you have multiple services talking to each other, it would be hard to see their service health instantly.

This article will look at how we can configure ASP.NET Core health checks and look into what kind of different metrics we can gather from it.

💡 Follow along with the code samples for this blog post from this repo.

The setup

I will be using ASP.NET’s MVC and template, as most of you are familiar with it. However, you can also use other project types such as API, Console or even Blazor.

Let’s scaffold an MVC app.

dotnet new mvc -n Monitor
dotnet new sln
dotnet sln add Monitor

First, we need to install the HealthChecks package.

cd Monitor/
dotnet add package AspNetCore.HealthChecks.UI

Now we are ready to hook it up to the middleware pipeline.

Application Configuration

Navigate to the Startup.cs and, let’s add the following in the ConfigureServices method.

services.AddHealthChecks();

For starters, we will add an endpoint to spit out a JSON to show the app’s current status.

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

		// Mapping a new endpoint to see the health
    endpoints.MapHealthChecks("/health");
});

Well, that isn’t particularly useful. This health checks package we are using provides many different output formats. They have made it extensible so that you can even use a custom output format. If you have many services talking to each other, it will make sense to use a format like JSON.

💡 Feel free to skip this section and jump ahead to “Registering health checks” section to see the final result.

Health checks UI dashboard

We are interested in seeing is a Dashboard that comes out-of-the-box. We will use our current project as a monitoring app to probe in and check the health status of some other applications that are running. The following diagram explains our desired state.

Let’s get to a point we can see something visually and start extending our solution on top of that. To be able to use the dashboard, it needs to have a backing store. We need to bring in a package called AspNetCore.HealthChecks.UI.InMemory.Storage

dotnet add package AspNetCore.HealthChecks.UI.InMemory.Storage

Let’s change the code in our Startup/ConfigureServices class. Replace the previous code with the following.

// ..
services.AddHealthChecksUI()
        .AddInMemoryStorage();
// ..

Next, let’s map a new endpoint to be able to see the dashboard.

// ...
app.UseEndpoints(endpoints =>
{
    endpoints.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    endpoints.MapHealthChecksUI();
});

//...

Now we are ready to run the application and see what we get.

dotnet new webapi -n Service1
dotnet new webapi -n Service2
dotnet sln add Service1 Service2

We need to add the health checks middleware to the two projects we created in the above step.

Startup.cs

public void ConfigureServices(IServiceCollection services)
{
		// ...
    services.AddHealthChecks();
		// ...
}

app.UseEndpoints(endpoints =>
{
    // ...
    endpoints.MapHealthChecks("/health", new HealthCheckOptions()
    {
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    });
		// ...
});

Just copy and add the following lines to the .csproj files of the two services we created.

<PackageReference Include="AspNetCore.HealthChecks.UI" Version="5.0.1" />
<PackageReference Include="AspNetCore.HealthChecks.UI.Client" Version="5.0.1" />

Finally, you need to add the endpoints of the services to our Monitor project. Let’s add the following bit to the appsettings.json file of the Monitor project. Head over to the official docs to learn more about the configuration.

"HealthChecksUI": {
  "HealthChecks": [
    {
      "Name": "Service 1",
      "Uri": "https://localhost:5011/health"
    },
    {
      "Name": "Service 2",
      "Uri": "https://localhost:5021/health"
    }
  ],
  "EvaluationTimeInSeconds": 10
},

💡 Don’t forget to take the port numbers your servers are running on from the corresponding launchSettings.json files. This could change depending on whether you are using IIS Express or Kestrel for development.

If you fire up the 3 projects now, you will be able to see the health checks dashboard.

The default URI for the dashboard UI is https://localhost:<app port>/healthchecks-ui

Under the hood, this small SPA dashboard polls the /healthchecks-api URI which returns a JSON.

Registering health checks

Without bringing in any other dependencies let’s simulate the 3 states that it will report to us:

Head over here to see a complete list of supported health checks. Let’s add some dummy health checks to see them in action in the dashboard.

We will replace the code in Service1/Startup.cs with the following.

services.AddHealthChecks()
    .AddCheck("Foo", () =>
        HealthCheckResult.Healthy("Foo is OK!"), tags: new[] { "foo_tag" })
    .AddCheck("Bar", () =>
        HealthCheckResult.Unhealthy("Bar is unhealthy!"), tags: new[] { "bar_tag" })
    .AddCheck("Baz", () =>
        HealthCheckResult.Degraded("Baz is degraded!"), tags: new[] { "baz_tag" });

In my previous blog post, I showed you how we could easily integrate with the Redis. I will be using AspNetCore.HealthChecks.Redis package to configure the health checks.

dotnet add package AspNetCore.HealthChecks.Redis

We will also add this configuration bit to its appsettings.json file.

"Redis": {
  "ConnectionString":  "localhost:5002" 
},

Let’s spin up a Redis docker container on port 5002.

docker run --name redis-cache -p 5002:6379 -d redis

Dashboard:

Feel free to stop the docker container and see how the errors get displayed in the dashboard.

Adding custom health checks

Now let’s switch to our Service2 project and do something interesting with it as well. We are going to be adding our custom health check.

public class RemoteHealthCheck : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new CancellationToken())
    {
        var isHealthy = CheckRemoteEndpointHealth();

        return Task.FromResult(isHealthy ? 
            HealthCheckResult.Healthy("Remote endpoint is healthy.") :
            HealthCheckResult.Unhealthy("Remote endpoint is unhealthy"));
    }

    private bool CheckRemoteEndpointHealth()
    {
        // Just stubbing it out for demo
        var rnd = new Random().Next(1, 5);
        return rnd % 2 != 0;
    }
}

We can now register RemoteHealthCheck in the Startup class like so.

services.AddHealthChecks()
        .AddCheck<RemoteHealthCheck>(nameof(RemoteHealthCheck));

Now the dashboard will show you the status of both services.

You’ll see that the RemoteHealthCheck will be going down from time to time because we have set it to return an unhealthy randomly. You can also set the EvaluationTimeInSeconds setting to like 2s to see the result quickly.

Conclusion

Today we looked at how you can improve your developer experience by leveraging ASP.NET’s health checks. There are so many providers and configurations to play around with. Feel free to head over to the docs to learn more.

References

Distributed Caching in ASP.NET Core with Redis

Sun, 28 Feb 2021 00:00:00 GMT

About a year ago, I wrote a blog post on simple In-Memory Caching in ASP.NET Core with IMemoryCache. This article mainly introduced the concept of caching and how we can store stuff in the server’s memory for simple tasks. Today’s objective is to leverage the IDistributedCache to do some distributed caching so that we can horizontally scale out our web app.

💡 UPDATE on Aug 2022: I have recently migrated this project to .NET 6 also with its minimal hosting model and also added a docker-compose.yaml file for better developer experience 🎉. You can still access the .NET Core 5 (works with .NET Core 3.1+ too) version from this branch.

For this specific tutorial, I will be using Redis as my cache provider. Redis is a battle-tested, fast memory cache that can store many types of objects. Redis is being used by giants such as Twitter, Github, Instagram, Stackoverflow, Medium, Airbnb etc.

💡 You can find the accompanying code for this blog post from here.

Here’s a snapshot of what we are going to be building.

User requests a user object.
App server checks if we already have a user in the cache and return the object if present.
App server makes a HTTP call to retrieve the list of users.
Users service returns the users list to the app server.
App server sends the users list to the distributed (Redis) cache.
App server gets the cached version until it expires (TTL).
User gets the cached user object.

The main reason why we call this a distributed cache is that it lives outside of our application server (as opposed to traditional in-memory caching) and we have the flexibility of scaling it horizontally (when operating in the cloud), if need be. Head over here to have a look at how this could be useful in enterprise applications.

The IDistributedCache interface provides us with a bunch of methods to manipulate your cache. And the actual implementation is specific to the technology we want to use. Here’s a summary of different ways you can do this.

Technology	NuGet package	Notes
Distributed Memory Cache	-	This is only recommended for dev and testing purposes. This is not an actual distributed cache.
Distributed SQL Server Cache	Microsoft.Extensions.Caching.SqlServer	Use SQL Server instance as a cache (locally or in cloud with Azure SQL Server).
Distributed Redis Cache	Microsoft.Extensions.Caching.StackExchangeRedis	Use Redis as a backing store (locally or in cloud with Azure Redis Cache)client package is Developed by peeps at StackExchange.
Distributed NCache Cache	NCache.Microsoft.Extensions.Caching.OpenSource	Wrapper around the NCache Distributed Cache

Scaffolding a sample app

We will create a Web MVC app in ASP.NET Core 6.

dotnet new mvc -n DistributedCache
dotnet new sln
dotnet sln add DistributedCache

Let’s go ahead and add the Redis client package from NuGet.

dotnet add package Microsoft.Extensions.Caching.StackExchangeRedis --version 6.0.0

Creating a Redis docker container

For this step, I assume that you have already installed Docker on your machine. It’s handy to have this so that you can spin up your own Redis container whenever you want for development purposes.

💡 For this section you can run docker-compose.yaml at the root of the project to spin up a local Redis container.

docker run --name redis-cache -p 5002:6379 -d redis

We are telling docker to use the official redis image and spin up a container with the name redis-cache and bind port 6379 of the container to the port 5002 of your host machine. Why I chose port 5002 is that it might be open as it’s a less obvious port number.

If you haven’t got the Redis image locally, it will fetch that from the DockerHub and spin up a new container under the name redis-cache. Next let’s verify that our docker instance is up and running. You could do so with,

docker ps -a

or alternatively with docker ps -a | grep redis-cache to filter our the output, if you have a bunch of containers running in the background like I do 😅

Now that we have the Redis container up and running let’s configure our web app to use it.

Application Configuration

For .NET Core 3.1 - 5 with generic host model:

Startup.cs

For .NET 6+ with minimal hosting model:

Program.cs

Since we have already added the required NuGet package, we only need to register its service in our app’s DI container and tell it where to find our Redis instance.


// Register the RedisCache service
services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = Configuration.GetSection("Redis")["ConnectionString"];
});

When we call AddStackExchangeRedisCache on the services object, it registers a singleton of RedisCache class against the IDistributedCache interface under the covers. This is what it looks like in the source:


// ...
services.Add(ServiceDescriptor.Singleton<IDistributedCache, RedisCache>());
// ..

appsettings.json

Since we have the Docker instance up and running at port 5002, we can mention that for development settings.

"Redis": {
  "ConnectionString": "localhost:5002"
}

I have brought across the service from my previous tutorial and added them to this project. You can find them under the Services folder. In fact, I have made the code to look more bit simpler as well.

Implementation

The functionality is pretty simple, and here’s what we going to do:

Get the cached user (if any) and display its email address
A button to invoke a HTTP call and cache a list of users
A button to clear the cache

The UI would look something like the following.

Let’s look at the main entry point of the actions, the HomeController class.

HomeController.cs

public async Task<IActionResult> Index()
{
    var users = (await _cacheService.GetCachedUser())?.FirstOrDefault();
    return View(users);
}

public async Task<IActionResult> CacheUserAsync()
{
    var users = await _usersService.GetUsersAsync();
    var cacheEntry = users.First();
    return View(nameof(Index), cacheEntry);
}

public IActionResult CacheRemoveAsync()
{
    _cacheService.ClearCache();
    return RedirectToAction(nameof(Index));
}

The code here is pretty self-explanatory, and we implement the 3 features we discussed in the Index, CacheUserAsync and CacheRemoveAsync actions.

💡 Tip: You would ideally want to decorate the UsersService class with CacheService by using a DI container such as Scrutor. You don’t want to write the plumbing code we have written here to emulate a similar thing as the default DI container doesn’t support the behaviour. Refer to Andrew Lock’s excellent article on this topic.

I’m going to skip all the other plumbing code and show you how we Get and Set values with the Redis cache. The real magic happens in the ICacheProvider class.

The code itself it pretty self-explanatory. In the GetFromCache method, we call the GetStringAsync with a given key (_Users in this case). It’s worth noting that we need to deserialise it to the type we want before returning it to the caller. Similarly, we serialise our users list and save it as a string in the Redis cache under the _Users key.

CacheProvider.cs

public class CacheProvider : ICacheProvider
{
    private readonly IDistributedCache _cache;

    public CacheProvider(IDistributedCache cache)
    {
        _cache = cache;
    }
    
    public async Task<T> GetFromCache<T>(string key) where T : class
    {
        var cachedResponse = await _cache.GetStringAsync(key);
        return cachedResponse == null ? null : JsonSerializer.Deserialize<T>(cachedResponse);
    }

    public async Task SetCache<T>(string key, T value, DistributedCacheEntryOptions options) where T : class
    {
        var response = JsonSerializer.Serialize(value);
        await _cache.SetStringAsync(key, response , options);
    }

    public async Task ClearCache(string key)
    {
        await _cache.RemoveAsync(key);
    }
}

So what gets saved under the covers?

We can connect to the container and open up the redis-cli to see what’s inside. To do that, you could run the following command.

docker exec -it redis-cache redis-cli

Once you are in, you could issue a hgetall _Users command to inspect what’s inside the hash that got saved in our request.

If you like to use a GUI, here’s a nice representation of what our web app saved under the hood. I used RedisInsight tool for this.

Demo

Here’s a working demo when you run the code from my repo:

As you can see, it will only fetch the users list only the first time we click the “Cache It” button. Every subsequent requests will fetch the users list from the Redis cache and serve to our app. The cache expiry can be configured by setting a sliding window or an absolute expiry by passing in configuration. In this demo I have set a sliding expiry for 2 minutes.

Conclusion

In this article, we converted our previous In-Memory example to use the IDistributedCache interface provided by ASP.NET Core and used Redis as a backing store. This approach can be utilised to leverage cloud service such as Azure Redis Cache for use-cases such as response caching, session storage etc.

Hope you enjoyed this article and feel free to share your thoughts and feedback. Until next time 👋

References

Publishing Android Apps to Microsoft App Center from Azure DevOps

Tue, 16 Feb 2021 00:00:00 GMT

In my previous article, we looked at how we can set up multi-stage builds for an Android app with Azure DevOps. In this article, we will be looking at distributing this app to App Center for internal testing.

💡 Note: that this a continuation of my previous blog article, Multi-stage builds for Ionic Apps with Azure Pipeline Templates. I suggest you go through it if you haven’t already.

Let’s look at what we are going to be building today. As always, you can find the complete source code in here.

What is Microsoft (Visual Studio) App Center?

Microsoft App Center is a platform where you can build, test, distribute and monitor your mobile apps. The beauty of this is that you can use it as a place to distribute your apps for a target user group for beta testing your app - similar to TestFlight but you can do it for multiple apps.

Creating a new app in App Center

To get started, head over to https://appcenter.ms/apps. If you haven’t already got an app created in App Center, we first need to do that. You could either create it under an Organization or a User. In my case, I have created it under my user account. The details look like this.

Creating a Service Connection

Before pushing our signed APK to the App Center, we need to create a link between the Azure DevOps instance and the App Center tenant. For that, we will be creating a service connection so that the build agent has necessary credentials to make this link work and where to put the latest build.

There are two parts to getting a service connection.

First, we need to generate an API token which will be used by the AppCenterDistribute task. Let’s do the following steps in the App Center.

Click on Profile Icon (top-right corner)→ Account Settings → User API Tokens.
Click on New API token button.
Give it a memorable description
Select Full Access as for Access option.

💡 Tip: Make sure to copy the API token as we will need it for our next step.

Once that’s done, let create the service connection in Azure DevOps.

Navigate to your project’s settings → Scroll down until you find Service connections. At the time of writing, it could be found under https://dev.azure.com/{your directory}/{your project name}/_settings/adminservices
Click on Create service connection.
Type in “App Center” in the search box and select Visual Studio App Center from the list.
Full in the details of the service connection pop up, as shown below. Make sure to keep the Server URL as it is.

Now we can use this service connection to deploy our app from Azure DevOps to the App Center.

Configuring the deploy task

We will first look at configuring the deploy task for the Debug build. We will add a new “Deploy” stage and continue from there.

The latest additions to the azure-piplines.yml will look like follows.

...
- stage: Deploy
    displayName: Deploy Debug and Release Apps to App Center
    dependsOn: Build
    jobs:
      - deployment: Deploy_Android_Debug
        variables:
          buildConfiguration: Debug
        displayName: Deploy Android - Debug App
        pool:
          vmImage: $(vmImageName)
        environment: Staging
        strategy:
          runOnce:
            deploy:
              steps:
                - template: ionic-android-debug-deploy.yml
                  parameters:
                    appCenterServiceConnection: 'App Center Integration for Sample App'
                    appSlug: '{username}/{appname in App Center}'
                    appFile: '$(Pipeline.Workspace)/$(projectName)/app-$(buildConfiguration).apk'
                    releaseNotes: 'Debug App'

I’m not going to do this for release build as it will be the same configuration but a different app slug.

💡 How do I find the app slug? Go to your app’s overview page in App Center and take note of the URL; the app slug would be {your org or user name}/{app name}

Let’s create a new file for the ionic-android-release-deploy.yml for the App Center deployment.

parameters:
  - name: appCenterServiceConnection
    displayName: 'Name of the service connection with App Center'
  - name: appSlug
    displayName: 'Path to application in AppCenter'
  - name: appFile
    displayName: 'Path of the apk to deploy'
  - name: releaseNotes
    displayName: 'Release notes'
  - name: deployMobileAssets
    displayName: "Deploy Mobile Assets"

steps:    
  - task: AppCenterDistribute@3
    inputs:
      serverEndpoint: '${{ parameters.appCenterServiceConnection }}'
      appSlug: '${{ parameters.appSlug }}'
      appFile: '${{parameters.appFile}}'
      symbolsOption: 'Android'
      destinationType: 'groups'
      releaseNotesOption: 'input'
      releaseNotesInput: '${{parameters.releaseNotes}}'
    displayName: 'Distribute to App Center'

The configuration for the release app will be the same in our case. So I am going to skip to keep things simple.

Once the pipeline runs, you should be able to see your brand new app release in the App Center 😎 Pat yourself on the back! Rest is easy.

Deploying our app for internal testing

App Center has the concept of adding collaborators to your app. This means that you can share your app with others by inviting them via email addresses. The official docs explains the concepts behind creating teams for your app.

When you have a release under your app, you could then distribute it to your team members by going to the “Distribute” on the left sidebar.

You can either create a new release from an existing release artifact or decide to distribute the release you have already got there.

When adding user groups for testing, you can add the teams you have already created for your app or directly share it with their email address. Once you hit Submit, they will receive an email to directly download the APK to their phone, install and run the app.

Conclusion

This article continued building upon our previous azure-pipelines.yml file and looked at how we can deploy our app to App Center for distribution. I hope this was helpful. Until next time 👋

References

Multi-stage builds for Ionic Apps with Azure Pipeline Templates

Tue, 26 Jan 2021 00:00:00 GMT

In this article, we will create an Azure Pipeline for an Ionic app using the new Azure pipelines YAML templates. We will also cover signing a release build of the published artifacts (APK signing). I have summarised the goals of this blog post in the diagram below.

A picture is worth a thousand words. Any points for my diagramming skills? 💯😅

Our main goals are to;

Scaffold an Ionic app
Generate the Android app
Set up Azure pipelines
Sign the release APKs

💡 You can follow along with my completed project repository in here.

Scaffold an Ionic App

Let’s get the initial setup out of the way! Ionic docs already have got a great getting started guide. In this step, we are not going to do anything fancy in Ionic’s side of things. We will just scaffold a basic app and use Capacitor to generate an Android app. If you have already got an existing Ionic app, you can skip ahead to the “Create the Azure Pipeline” section.

1. Creating a blank Ionic app

ionic start sampleapp blank --capacitor --type=ionic-angular

As you can see, there’s nothing fancy here. Just a blank Ionic+Angular project with Capacitor integration.

2. Set Project ID (optional) - You don’t need to change the appId for this tutorial, but I like to set it anyway.

{
...
  "appId": "com.sahan.sampleapp",
...
}

Open up capacitor.config.json file and change the appId property to whatever suits you.

3. Add Capacitor - Android Project

If you haven’t set up your environment for Android development, please follow this guide to do so.

ionic build && ionic cap add android

Note the ionic build command here. We first need to issue the ionic build command before adding the Android project for the first time. With the initial ionic cap add android command it will copy the required web assets into the native project.

💡 It’s now recommended to keep your native projects in source control.

If you make any changes to your Ionic project, you can use the ionic cap copy command to copy any changed web assets or ionic cap sync if you are using/updated any native dependencies.

After these steps, your project structure should look like the following.

Create the Azure Pipeline

For this tutorial, I will create a single azure-pipelines.yml file and build-publish two Android apps, one of Debug configuration and another for Release.

1. Creating the initial structure of the azure-pipelines.yml

Let’s first create a folder called infrastructure at the root of our solution structure. We will use this folder to keep all our Azure Pipeline line YAML files. We will then add two additional YAML files called ionic-android-debug-build.yml and ionic-android-release-build.yml. I will explain why we do this in a second. Your file structure should look like this.

For our initial step, we will create the barebones structure of our azure-pipelines.yml file

trigger:
- main

variables:
  vmImageName: 'windows-latest'
  projectName: 'SampleApp'

stages:
  - stage: Build
    displayName: Build Ionic - Android projects
    jobs:
      # Debug build
      - job: Build_Ionic_Android_Debug
        variables:
          - name: buildConfiguration
            value: Debug
        displayName: Build Debug
        pool:
          vmImage: $(vmImageName)
        steps:
          - template: ionic-android-debug-build.yml

      # Release build
      - job: Build_Ionic_Android_Release
        variables:
          - name: buildConfiguration
            value: Release
        displayName: Build Release
        pool:
          vmImage: $(vmImageName)
        steps:
          - template: ionic-android-release-build.yml

Here’s a rough explanation of the structure of the above file.

trigger: We trigger our pipeline whenever a change happens to the main branch
variables: We have defined two global variables which can be reused in our stages
stages: This is where we define our stages. Initially, we have added Build stage. Later we will add another stage called Deploy
- jobs: We have two jobs called Debug and Release .
- jobs:steps: Note how we give our template path with a template attribute. This way, we don’t need to keep everything in one YAML file.

💡 Tip: If you are using the script task you will sometimes find that any multiline commands won’t run. Only the first one would get picked up. You can append your commands with && or use a PowerShell task as shown in this tutorial.

2. Create ionic-android-debug-build.yml

steps:    
  - script: npm install -g @ionic/cli
    displayName: 'Install Ionic CLI'

  - task: Npm@1
    inputs:
      workingDir: '$(Build.SourcesDirectory)/$(projectName)'
      command: install
    displayName: 'NPM Install'

  - powershell: |
      ionic cap build android --no-open
      npx cap sync
      cd android
      ./gradlew assemble$(buildConfiguration)
    workingDirectory: $(Build.SourcesDirectory)/$(projectName)
    displayName: 'Build Android Project'

  - task: CopyFiles@2
    inputs:
      SourceFolder: '$(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)'
      contents: "**/app-$(buildConfiguration).apk"
      targetFolder: "$(Build.ArtifactStagingDirectory)/$(projectName)"
    displayName: "Copy unsigned APK to staging directory"

  - task: PublishBuildArtifacts@1
    inputs:
      PathtoPublish: "$(Build.ArtifactStagingDirectory)/$(projectName)"
      ArtifactName: "$(projectName)"
      publishLocation: "Container"
    displayName: "Publish artifacts"

We use the gradelw utility that comes with the android project to build it. This is a wrapper around Gradle which invokes it with any parameters we have given (in our case assembleDebug or assembleRelease). If Gradle is not found in your system (or in a build agent) it will also download it from a distribution server.

The ionic-android-release-build.yml is pretty much the same configuration except for some extra flags to do the release build. We will go through this file in the Signing step.

Now, let go ahead and commit the changes. You might want to create a new Azure Pipeline if you haven’t already. Once the pipeline is run, you will be able to see all green (hopefully). 😀

If you have a look at the build artifacts, you will find the debug and release build APKs all in one place now. Next, we will have a look at signing the APK files.

Signing the APK

If you are publishing your app to the Play Store, we need to consider signing our APKs. Ionic’s official documentation already covers this step, so we will look at how we can automate this step.

1. Generating the .keystore file

If you are doing this for the first time, you need to create the private .keystore file used for signing the APK. You can use the keytool file that comes with the Android SDK:

keytool -genkey -v -keystore my-release-key.keystore -alias alias_name -keyalg RSA -keysize 2048 -validity 10000

Remember to take note of the following when you are generating this file.

Keystore Password
Keystore Alias
Key Password

We will be putting these into pipeline variables as our next step.

2. Creating the pipeline variables

Once you have generated the keystore file, you can navigate to the Library section of your project in Azure DevOps.

Make sure to put in the keystorePassword, keyAlias and keyPassword variables which we created in the above step.

After that, you need to upload your keystore file to Secure files section. We will later access this file in the Android signing task.

3. Add the signing task

We will be using the AndroidSigning@3 task to sign our release build. It’s pretty straight forward task to use and the parameters we need to send are as follows.

- task: AndroidSigning@3
    inputs:
      apkFiles: '$(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)/*.apk'
      apksign: true
      apksignerKeystoreFile: '${{ parameters.keystoreFileName }}'
      apksignerKeystorePassword: '${{ parameters.keystorePassword }}'
      apksignerKeystoreAlias: '${{ parameters.keyAlias }}'
      apksignerKeyPassword: '${{ parameters.keyPassword }}'
      apksignerArguments: --out $(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)/$(projectName)-$(buildConfiguration).apk --verbose
      zipalign: true
    displayName: 'Sign the APK'

Notice how we are linking the variables we created in the Library section. Now, our final ionic-android-release-build.yml file would look like the following.

parameters:
  - name: keystoreFileName
    displayName: "The keystore file name for signing the apk"
    type: string
  - name: keystorePassword
    displayName: "Password for the keystore"
    type: string
  - name: keyAlias
    displayName: "Key alias"
    type: string
  - name: keyPassword
    displayName: "Key password"

steps:    
  - script: npm install -g @ionic/cli
    displayName: 'Install Ionic CLI'

  - task: Npm@1
    inputs:
      workingDir: '$(Build.SourcesDirectory)/$(projectName)'
      command: install
    displayName: 'NPM Install'

  - powershell: |
      ionic cap build android --prod --no-open
      npx cap sync
      cd android
      ./gradlew assemble$(buildConfiguration)
    workingDirectory: $(Build.SourcesDirectory)/$(projectName)
    displayName: 'Build Android Project'

  - task: AndroidSigning@3
    inputs:
      apkFiles: '$(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)/*.apk'
      apksign: true
      apksignerKeystoreFile: '${{ parameters.keystoreFileName }}'
      apksignerKeystorePassword: '${{ parameters.keystorePassword }}'
      apksignerKeystoreAlias: '${{ parameters.keyAlias }}'
      apksignerKeyPassword: '${{ parameters.keyPassword }}'
      apksignerArguments: --out $(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)/$(projectName)-$(buildConfiguration).apk --verbose
      zipalign: true
    displayName: 'Sign the APK'

  - task: CopyFiles@2
    inputs:
      SourceFolder: '$(Build.SourcesDirectory)/$(projectName)/android/app/build/outputs/apk/$(buildConfiguration)'
      contents: "**/$(projectName)-$(buildConfiguration).apk"
      targetFolder: "$(Build.ArtifactStagingDirectory)/$(projectName)"
    displayName: "Copy unsigned APK to staging directory"

  - task: PublishBuildArtifacts@1
    inputs:
      PathtoPublish: "$(Build.ArtifactStagingDirectory)/$(projectName)"
      ArtifactName: "$(projectName)"
      publishLocation: "Container"
    displayName: "Publish artifacts"

💡 Tip: I found that I can’t make the file name coming from the variable group. Which means you need to have it hardcoded in the azure-pipelines.yml file

First, we need to tell the azure-pipeline.yml which variable group to use. Here’s a small excerpt of that.

# other tasks removed for brevity
- job: Build_Ionic_Android_Release
  variables:
    - group: SampleApp-Release
    - name: buildConfiguration
      value: Release

💡 Note that the notation for defining a variable can take two variants. If you are not using a group you could simply write it as buildConfiguration: Release. If you are using the group attribute, you need to use the notation I have shown in the above snippet. I had to learn it the hard way 😂

You would have also noticed that we are using parameters here. These will be injected in the azure-pipeline.yml file.

# other tasks removed for brevity
steps:
  - template: ionic-android-release-build.yml
    parameters:
      keystoreFileName: 'sampleapp-release-key.keystore'
      keystorePassword: $(keystorePassword)
      keyAlias: $(keyAlias)
      keyPassword: $(keyPassword)

Once you push your changes to the repo, Azure DevOps would ask you to permit your pipeline to access the secure file which we added in the above step. Click on View and permit it. Alternatively, you could also use Azure KeyVault for storing secrets.

The completed YAML files can be found in my repo over here. Once the pipeline is completed, you can have a look at the artifacts. Voilà! We have the signed APKs

Conclusion

In summary, we created a blank Ionic app, set up Azure DevOps pipeline YAML files and published the signed artifacts. You can use the same structure to add an iOS project as well. However, the build steps could be more involved and cumbersome to get right initially. In my next article, we will look at how we can integrate with Microsft App Center for distribution.

If you have any feedback or questions, let me know in the comments below. Until next time 👋

References

Understanding WebSockets with ASP.NET

Fri, 01 Jan 2021 00:00:00 GMT

In this article, we will go through RFC 6455 WebSockets specification and configure a generic ASP.NET (Core) 7 application to communicate over WebSockets connection with SignalR. We will dive into the underlying concepts to understand what happens under the covers.

💡 UPDATE on March 2023: I have recently migrated this project to .NET 7 🎉. You can still access the .NET 5 version from this branch.

Walkthrough video

If you like to watch a video walkthrough instead of this article, you can follow along on my Youtube channel too 😊

A bit about WebSockets

WebSockets was introduced to enable two-way communication between a client and a server. One of the pain points with HTTP 1.0 was creating and closing a connection each time we send a request to the server. With HTTP 1.1 however, persistent connections (RFC 2616) were introduced by making using of a keep-alive mechanism. With this, connections could be reused for more than one request - which will reduce latency as the server knows about the client and they do not need to start over the handshake process per request.

💡 When you are learning about protocols, a good place to start is to read its corresponding RFC specification.

WebSockets is built on top of HTTP 1.1 spec as it allows persistent connections. So, when you are making a WebSocket connection for the first time, it is essentially an HTTP 1.1 request (more on this later). This enables real-time communication between a client and a server. In a nutshell, the following diagram depicts what happens during the initiation (handshake), data transfer and closing of a WS connection. We will dive deeper into these concepts later.

The protocol has two parts to it; Handshake and Data Transfer.

Handshake

Let’s talk about the opening handshake first. From the spec,

The opening handshake is intended to be compatible with HTTP-based

server-side software and intermediaries, so that a single port can be used by both HTTP clients talking to that server and WebSocket clients talking to that server.

Simply put, a WebSocket connection is based on HTTP (and TCP as transport) over a single port. Here’s the summary of steps.

A server must be listening for incoming TCP socket connections. This could be any port you have assigned - normally this would be 80 or 443.
The client initiates the opening handshake (otherwise the server wouldn’t know who to talk to) with an HTTP GET request - This is the “Web” part in “WebSockets”. In the headers, the client will ask the server to Upgrade the connection to a WebSocket.
The server sends a handshake response telling the client that it will be changing the protocol from HTTP to WebSocket.
Both client and server negotiate the connection details. Either of the parties can back out if the terms are unfavourable.

Here’s what a typical opening (client) handshake request looks like.

GET /ws-endpoint HTTP/1.1
Host: example.com:80
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: L4kHN+1Bx7zKbxsDbqgzHw==
Sec-WebSocket-Version: 13

Note how the client sends out Connection: Upgrade and Upgrade: websocket headers in the request.

And, the server handshake response,

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: CTPN8jCb3BUjBjBtdjwSQCytuBo=

Note how the server sends out HTTP/1.1 101 Switching Protocols in the response headers. Anything other than a 101 indicates that the opening handshake was not completed.

The closing handshake is pretty simple. Either the client or server can send out a closing handshake request. From the spec,

It is safe for both peers to initiate this handshake simultaneously.

The closing handshake is intended to complement the TCP closing handshake (FIN/ACK), on the basis that the TCP closing handshake is not always reliable end-to-end, especially in the presence of intercepting proxies and other intermediaries.

We will talk about these in action when we jump over to the demo section.

Data Transfer

The next key concept we need to understand is Data Transfer. Either of the parties can send messages at any given time - as it is a Full Duplex communication protocol.

The messages are composed of one or more frames. A frame can be of type text (UTF-8), binary, and control frames (such as 0x8 (Close), 0x9 (Ping), and 0xA (Pong)).

If you are interested, you can read the full RFC spec from here.

Setup

Let’s put this into action and see how it works.

💡 Follow along with the completed code from my repository here

First create a new ASP.NET 5 WebAPI app.

dotnet new webapi -n WebSocketsTutorial
dotnet new sln
dotnet sln add WebSocketsTutorial

Now we will add SignalR to our app.

dotnet add WebSocketsTutorial/ package Microsoft.AspNet.SignalR

Sample code explanation

We will start by adding the WebSockets middleware to our WebAPI app. Head over to the Startup.cs file and add the following line inside the Configure method.

I like to keep things simple for this tutorial. Therefore, I’m not going to talk about SignalR (Hubs and stuff). It would be purely based on WebSocket communication. You can also achieve the same with raw WebSockets, you don’t have to use SignalR if you want to keep things even more simpler.

...

app.UseWebSockets();

...

Next, we will delete the default WeatherForecastController and add a new controller called WebSocketsController. Note that we will be just using a controller action instead of intercepting the request pipeline

The full code for this controller will look like this. This code is based on Microsoft’s official docs’ example.

WebSocketsController.cs

using System;
using System.Net.WebSockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace WebSocketsTutorial.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class WebSocketsController : ControllerBase
    {
        private readonly ILogger<WebSocketsController> _logger;

        public WebSocketsController(ILogger<WebSocketsController> logger)
        {
            _logger = logger;
        }

        [HttpGet("/ws")]
        public async Task Get()
        {
          if (HttpContext.WebSockets.IsWebSocketRequest)
          {
              using var webSocket = await HttpContext.WebSockets.AcceptWebSocketAsync();
              _logger.Log(LogLevel.Information, "WebSocket connection established");
              await Echo(webSocket);
          }
          else
          {
              HttpContext.Response.StatusCode = 400;
          }
        }
        
        private async Task Echo(WebSocket webSocket)
        {
            var buffer = new byte[1024 * 4];
            var result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
            _logger.Log(LogLevel.Information, "Message received from Client");

            while (!result.CloseStatus.HasValue)
            {
                var serverMsg = Encoding.UTF8.GetBytes($"Server: Hello. You said: {Encoding.UTF8.GetString(buffer)}");
                await webSocket.SendAsync(new ArraySegment<byte>(serverMsg, 0, serverMsg.Length), result.MessageType, result.EndOfMessage, CancellationToken.None);
                _logger.Log(LogLevel.Information, "Message sent to Client");

                buffer = new byte[1024 * 4];
                result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
                _logger.Log(LogLevel.Information, "Message received from Client");
                
            }
            await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
            _logger.Log(LogLevel.Information, "WebSocket connection closed");
        }
    }
}

Here’s what we did,

Add a new route called ws/
Check if the current request is via WebSockets otherwise throw a 400.
Wait until client initiates a request. L:40
Going into a loop until the client closes the connection. L:43
Within the loop, we will prepend “Server: Hello. You said: <client’s message>” to the message and send it back to the client.
Wait until the client send another request.

Note that the server does not need to wait until the client sends a request to push messages to the client, after the initial handshake. Let’s run the application and see whether it works.

dotnet run --project WebSocketsTutorial

Once you run the application, head over to https://localhost:5001/swagger/index.html. You should see the Swagger UI.

We will now see how we can get the client and server to talk to each other. For the purpose of this demo, I will be using Chrome’s DevTools (Open new tab → Inspect or press F12 → Console tab). But, you can use any client of your choice.

First, we will create a WebSocket connection to our server endpoint.

let webSocket = new WebSocket('wss://localhost:5001/ws');

What this does is, it initiates a connection between the client and the server. wss:// is the WebSockets Secure protocol since our WebAPI app is served via TLS.

You can then send messages by calling webSocket.send() method. Your console should look similar to the one below.

A closer look at the WebSocket connection

if you go to the Network tab, filter out the requests by the WS tab and click on the last request called ws.

Click on the Messages tab and examine the message passed back and forth. During this time, if you invoke the following command, you will be able to see “This was sent from the Client!” appearing in this box. Give it a try!

webSocket.send("Client: Hello");

As you can see, the server does need to wait for the client to send a response (that is, after the initial handshake), and the client can send the messages without being blocked. This is Full Duplex communication. We have covered the Data Transfer aspect of WebSocket communication. As an exercise you could run a loop to push messages to the client to see it in action.

In addition to this, the server and client will have ping-pongs to see if the client is still alive. This is an actual feature in WebSockets! If you really want to have a look at these packets, you can use a tool like WireShark to get an idea.

How does it do the Handshake? Well, if you jump over to the Headers tab, you will be able to see the request-response headers we talked about in the first section of this post 🙌

Have a play around with webSocket.close() too so that we can fully cover the open-data-close loop.

Conclusion

If you are interested in having a look at the RFC for WebSockets, head over to RFC 6455 and have a read. This post only scratches the surface of WebSockets, and there are many other things that we could discuss such as Security, Load Balancing, Proxies etc.

Don’t forget to let me know any feedback or comments. Until next time ✌️

References

How to ramp up in a new codebase with NDepend

Wed, 21 Oct 2020 00:00:00 GMT

Reading code is good. But, visualising code is even better. I recently wrapped up a gig where we had to perform a .NET Core upgrade (from .NET Framework 4.5 to .NET Core 3.1) which also had quite a bit of tech debt. It could be a bit challenging to ramp up quickly in a codebase that is entirely new to you and get the preferred outcomes of it.

In this post, we will look at how we can use static analysis and dependency graphs with NDepend to quickly ramp up and even identify issues in a real project.

Installation

You can download a free trial from here if you haven’t already got NDepend. Once you have downloaded it, there’s nothing there to do in terms of installation. Just extract it to a folder of your choice.

VisualNDepend.exe is the standalone application and you can install the VS extension with NDepend.VisualStudioExtension.Installer.exe.

We will be using eShopOnWeb project for this demo.

git clone https://github.com/dotnet-architecture/eShopOnWeb

Once you have cloned the repo, you should have the following folder structure.

Before we begin, we need to build the application so that NDepend can analyse the assemblies.

dotnet build .\eShopOnWeb.sln

Now we are ready to use NDepend on our solution. You can either use NDepend standalone app or use VS extension.

Analysing the assemblies

If you choose to do a one-off analysis you will get to select which assemblies you would want to analyse. You can select the .sln file of your solution and NDepend will pick up the assemblies.

Once you have analyzed your solution, you will get the following options.

The dashboard gives an excellent breakdown of insights like LoC, tech debt, types, quality gates etc.

This is quite useful since it gives you a birds-eye view of the solution of what you are working with.

Interactive Dependency Graph

The most exciting feature for me was the dependency graph. For our .NET Core upgrade, it was quite easy to understand which projects should be migrated first with NDepend. So, I believe if you are new to our codebase, this is the best option to start with to get a feel for it.

At a glance, it will give you a nice visualisation of the dependencies of your solution. You can zoom in/out of any dependency in your solution structure. If you have got a big monitor, you are in for a treat! 😊 For any migration effort, it’s necessary to identify which projects depend on which.

💡 This is just a demo of what you can quickly achieve when you make use of dependency graphs in practice. When you are doing an actual migration though, you need to take small steps carefully and focus on the migration with the least amount of code changes. Keep other refactorings once you are through that phase.

Let’s have some fun with the eShopOnWeb project now. Looking closer at the ApplicationCore project, you will find the following (unless they have changed it while at the time you are reading this post)

Can you spot there’s a weird-looking namespace in there? Ardalis.GuardClauses doesn’t sound like it’s supposed to be there. But let’s have a look in the class.

If you have a look at that class, it’s, presumably, in a wrong folder and also a different namespace that to other classes we have in that project. If you shift the file to Extensions folder, fix the namespace (in VS) and press F5 in NDepend. This will do another run, and the latest changes will be available. Press Alt+G to quickly go to the dependency graph.

💡 Don’t forget to install the VS Extension that comes with NDepend so that you can quickly dive into code from the dependency graph!

We found our first issue within a matter of seconds! 👊

Now the file seems to be in the correct place. If you scroll down a little bit in that dependency graph, you would quickly realize that there’s another file called JsonExtensions.cs which happens to be also some extensions but in a namespace that is not related to what it represents.

If you have a quick look at the folder structure it will tell you a different story,

We won’t refactor this now, but you get the idea. This is just a simple example but you would find these types of issues even in code that’s in production, that could sometimes give you a lot of headaches. But the point I wanted to make was that by using static analysis and dependency graphs you could get a better understanding of what’s going on in your code rather than merely going through folders and files.

💡 If you are not sure of what those arrows mean, you can use the Context-Sensitive Help which can be enabled by clicking on the ? mark.

You can also do other cool stuff like finding out who are the callers, callees for a given type. You can even write LINQ queries and drill down further.

How does it stack up against Visual Studio 2019’s tools?

I have used the inbuilt dependency graph that comes with VS 2019 for some time now and it’s good for basic stuff. Since I’m using VS Professional edition, I can’t explain its full capabilities. You can have a look here to see which editions support dependency diagrams.

Here’s a screenshot from VS’s dependency diagram feature.

You can dive into your projects, but you will soon end up with a huge mess of classes, and you will find yourself turning the level of detail up/down and settings rather than going through the code. Another thing that you need to be aware of is that it could become quite slow or sometimes not even render the dependency graph when you throw a large project at it.

When compared with NDepend, I found that NDepend provides an intuitive user experience with excellent breadth and depth into your solution. Best of all, it works all the time with lightning speed! ⚡️ This is why NDepend now has become my go-to tool for working with large projects.

Conclusion

As we have seen in this post, it is quite beneficial to have an overall view of the solution you are dealing with by visualising it. This post only scratches the surface of what NDepend is capable of and there are many other features of it that I haven’t covered.

Let me know in the comments below what other tools that you use to quickly ramp up on a codebase. If you want to learn more, you can checkout the links in the below section. Until then 👋

References

Having Fun with Microsoft IoC Container for .NET Core

Fri, 21 Aug 2020 00:00:00 GMT

The objective of this post is to configure and use Microsoft’s default dependency injection container from scratch to understand how it all hangs together when in action. There are many other great articles explaining Dependency Injection and Inversion of Control (DI & IoC from now on) in ASP.NET Core out there. This article assumes that you understand those principles. So I will not be covering those.

We will start with a simple console application, configure an IoC container, and have some fun with it by diving into the .NET Core DI Extensions’ source code.

💡 Follow along with the code from my repository

Microsoft’s IoC Container in .NET Core

The .NET Core IoC container is located in Microsoft.Extensions.DependencyInjection namespace. Let’s look at what are the steps involved in order to use this in our application.

We need to add this assembly via NuGet.
Types must be registered with ServiceCollection
Types are retrieved from ServiceProvider

We will start off with a simple console application and see how we can achieve the above steps.

Setup

Let’s start by creating a console application.

dotnet new console -n IoCTutorial
dotnet new sln
dotnet sln add IoCTutorial

Open it up in your favourite IDE and you are ready to follow along. I won’t be using any other dependency (such as logging) just to keep this tutorial simple.

Let’s go ahead and add the DI assembly from NuGet.

dotnet add package Microsoft.Extensions.DependencyInjection

Your project structure should now look like this.

Let’s add two classes into our project; MyService and MyDependency. Here’s what’s gonna go into those two classes.

MyService.cs

public class MyService
{
    private readonly MyDependency _myDependency;
    public MyService(MyDependency myDependency)
    {
        Console.WriteLine("Constructed MyService");
        _myDependency = myDependency;
    }
    
    public void DoSomething()
    {
        _myDependency.DoWork();
    }
}

MyDependency.cs

public class MyDependency
{
    public MyDependency()
    {
        Console.WriteLine("Constructed MyDependency");
    }
    
    public void DoWork()
    {
        Console.WriteLine("Doing some work in MyDependency");
    }
}

As we can see, MyService has an (aptly named 😅 ) dependency on MyDependency. In order to invoke our service, we need to pass an instance of the required dependency into its constructor. In our driver code (Program.cs) without any IoC stuff, we could do something like this.

var myService = new MyService(new MyDependency());
myService.DoSomething();

This is baaaaad 💀. Problem with this is, now we are responsible for managing the dependencies, their lifetimes (and also violates SOLID principles) etc. Let’s fix this by using the IoC container. We will simply give them a scoped lifetime.

// Register our types in the container
var container = new ServiceCollection();
container.AddScoped<MyDependency>();
container.AddScoped<MyService>();

// Build the IoC and get a provider
var provider = container.BuildServiceProvider();

If you aren’t familiar with service lifetimes, it’s best if you refer to the official documentation before continuing on.

💡 Note that you don’t have to instantiate ServiceCollection in an ASP.NET Core Web or Worker Service application. You would ideally register your services using the IServiceCollection in the Startup class.

The quick & dirty service registration

Once we have set this up, we need to call BuildServiceProvider method on the container. This returns us a ServiceProvider as a result. Remember that we first need to register our types in the container (ServiceCollection) and then retrieve them using the provider. Bear with me on this one as we are using the concrete implementations rather than interfaces for this initial cut.

// Get our service and call DoSomething()
var myService = provider.GetService<MyService>();
myService.DoSomething();

Now when we run this, it should create an instance of MyDependency, construct an instance of MyService and run the DoSomething() method.

Notice how we never constructed MyDependency because the DI framework resolved it and did the constructor injection for us. Nice!

Internals of ServiceCollection and ServiceProvider

Looking at our code, the ServiceCollection holds a bunch of ServiceDescriptors and provides some utility methods to manipulate it. ServiceDescriptors are really the objects that describe (type, implementation, lifetime etc.) our service registrations.

public class ServiceCollection : IServiceCollection
{
    private readonly List<ServiceDescriptor> _descriptors = new List<ServiceDescriptor>();

    public int Count => _descriptors.Count;
    public bool IsReadOnly => false;

    public ServiceDescriptor this[int index]
    {
        get
        {
            return _descriptors[index];
        }
        set
        {
            _descriptors[index] = value;
        }
    }

    public void Clear() {...}

    public bool Contains(ServiceDescriptor item) {...}

    public bool Remove(ServiceDescriptor item) {...}

    void ICollection<ServiceDescriptor>.Add(ServiceDescriptor item) {..}

    // Other methods removed for brevity
}

In the subsequent sections, we will look at how it gets utilised.

The AddScoped (and other service registration extension methods) method just adds our service registration (as a ServiceDescriptor) into the above _descriptors list.

Looking at the internals of the BuildServiceProvider extension method, we can see that it actually instantiates a new service provider with our given service registrations. As we saw earlier these service registrations are passed in as a ServiceCollection and if you put debug this, you will be able to see the following:

If you are interested, one observation I made was to get an idea of when it actually instantiates our types. In our Program.cs if you update it as below,

var provider = container.BuildServiceProvider();
Console.WriteLine("-----------");
var myService = provider.GetService<MyService>();
Console.WriteLine("-----------");
myService.DoSomething();

Should give you,

See the dotted line before “Doing some work in MyDependency”? I initially thought that these types would have been constructed when we built the service provider. However, it seems that they are instantiated when we call provider.GetService<MyService>(); instead. This should, however, not to be confused with service lifetimes in an ASP.NET Core web application where service lifetimes play a major role in the request pipeline.

💡 The IoC container will instantiate our implementation types when the GetService() method is called, not when the IoC container is built.

Looking at the implementation of the ServiceProviderEngine’s code you will see that the data structure that holds all our type registrations is really a ConcurrentDictionary. If you drill down in the call chain of GetService far enough, you would come across the following class in the source of DI extensions.

So, we have an idea of how we can register services in the IoC container. Nevertheless, this can still be improved by mapping interfaces rather than using concrete classes. We will look at it in the next section.

Replacing Concrete Classes with Interfaces

Let’s extract interfaces for MyService and MyDependency

public interface IMyService
{
    void DoSomething();
}

public interface IMyDependency
{
    void DoWork();
}

Nothing interesting here. Let’s just implement these two interfaces in their corresponding classes.

Our MyService will now use the interface instead of the concrete class.

public class MyService : IMyService
{
    private readonly IMyDependency _myDependency;
    
    public MyService(IMyDependency myDependency)
    {
        Console.WriteLine("Constructed MyService");
        _myDependency = myDependency;
    }
    
    public void DoSomething()
    {
        _myDependency.DoWork();
    }
}

In our Program.cs, we will update the following lines to use the abstractions instead of concrete classes,

container.AddScoped<IMyDependency, MyDependency>();
container.AddScoped<IMyService, MyService>();

Now when we run the application, we will get the same result. However, to make things interesting, let’s add another MyDependency class that implements IMyDependency interface. Without spending too much time on a name, let’s name that MyDependency2 and register it in the IoC container.

container.AddScoped<IMyDependency, MyDependency>();
container.AddScoped<IMyDependency, MyDependency2>(); // New implementation
container.AddScoped<IMyService, MyService>();

What do we get now?

It’s no longer using MyDependency, but using MyDependency2 instead.

💡 Remember, if you add multiple concrete implementations for the same interface, it will always pick up the last one that got registered, by default.

Unravelling why we get the last registered implementation type

So, how do we register multiple implementations for the same service type? It’s not supported as you might already know or have come across this discussion on Github. Looking at the internals of the GetService() method we can uncover why this happens.

If you debug the ServiceCollection, you will be able to see that our two implementation types for IMyDependency are still there.

So there has to be something in the call stack which hands over the last registered implementation type to the service provider. If you start off from GetService() method in ServiceProviderServiceExtensions class, you would end up in ServiceProviderEngine’s GetService() method’s implementation.

internal object GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
{
    if (_disposed)
    {
        ThrowHelper.ThrowObjectDisposedException();
    }

    var realizedService = RealizedServices.GetOrAdd(serviceType, _createServiceAccessor);
    _callback?.OnResolve(serviceType, serviceProviderEngineScope);
    DependencyInjectionEventSource.Log.ServiceResolved(serviceType);
    return realizedService.Invoke(serviceProviderEngineScope);
}

RealizedServices is a ConcurrentDictionary that keeps track of our service types and a Func to retrieve the implementation type by visiting the corresponding call sites in the dependency tree. The _createServiceAccessor actually refers to CreateServiceAccessor which does the above.

To find the “why” of this, we need to dig into GetCallSite method. This will hopefully navigate you through to CreateCallSite method in CallSiteFactory.cs class. If you find the following lines, that could give you a hint where to look at next. In our case, it will call the TryCreateExact method because it’s neither an Open Generic nor an Enumerable.

...
var callSite = TryCreateExact(serviceType, callSiteChain) ??
                TryCreateOpenGeneric(serviceType, callSiteChain) ??
                TryCreateEnumerable(serviceType, callSiteChain);
...

Taking a peek inside TryCreateExact you will be able to see the following like which passes in the last implementation type of the service type we are looking for. There you have the answer for what we were searching for from a more granular level of detail.

A way to access multiple implementations of same service type

Let’s say in the rare situation where you would want to invoke both dependencies somewhere in your code; how would you achieve that?

I have changed the MyService class a little bit to give you an idea of how you can sort of do that. The key here is, we need to inject our dependencies as an IEnumerable which will give us access to the service registrations of a given type.

public class MyService : IMyService
{
    private readonly IEnumerable<IMyDependency> _myDependencies;
    
    public MyService(IEnumerable<IMyDependency> myDependencies)
    {
        Console.WriteLine("Constructed MyService");
        _myDependencies = myDependencies;
    }
    
    public void DoSomething()
    {
        foreach (var dependency in _myDependencies)
        {
            dependency.DoWork();
        }
    }
}

Or better yet, you can specifically select the dependency you want with a little bit of LINQ by doing something like:

_myDependencies.FirstOrDefault(x => x.GetType() == typeof(MyDependency))?.DoWork();

💡 Note that the IoC container will only resolve and inject the collection of services when the parameter type is an IEnumerable and not any other type such as IList, Array etc. Otherwise, you can expect an InvalidOperationException.

Not the world’s best solution, but gets the job done 🙂 Another downside to this is having to manually register the services rather than making use of assembly scanning. You can also play around with TryAddEnumerable extension method to see how you can improve this implementation depending on your use case.

A quick detour: how does it resolve an IEnumerable?

Taking a quick detour, let’s see how we got both our MyDependency and MyDependency2 when we injected an IEnumerable.

When it does the service lookup, depending on the lifetime of the service, it will access different caches of the service provider engine. This happens in the VisitCallSite in CallSiteVisitor class. During this detour, you will also encounter some impressive locking code with Monitor 👀. Once you run through some more hoops you will end up in VisitCallSiteMain method which will call the VisitIEnumerable method.

If you carefully look at the ServiceCallSites in the above screenshot (highlighted in blue) you can see it has references to our MyDependency and MyDependency2 types under ImplementationType params in their corresponding sections. Since we requested an IEnumerable it knows that it needs to resolve all the dependencies we have registered under IMyDependency type.

Now, if you have a look VisitIEnumerable method you would see that it visits each implementation type’s constructor in a for-loop.

Now that we are iterating through each of the implementation types we will again go through the VisitCallSite method (doing the service lookup in the cache etc.) call chain as before and end up in the VisitCallSiteMain method.

Déjà vu? yet again we are visiting the VisitCallSiteMain method as before because now we are resolving each dependency of type IMyDependency by visiting the ctors in the above loop. In the VisitConstructor method, by using reflection (ConstructorInfo.Invoke() method) it will call the constructor on our implemented type and return it. Note that this will happen only during the initial call to the lookup the service and subsequent calls depends on the lifetime (by using the internal cache).

As we can see, Microsoft’s default IoC container has all the essential capabilities but also lacks some advanced features like assembly scanning, service decorators that you would get in frameworks like Autofac, Scrutor etc.

Summary

To summarise, we first looked at how we can easily bring in Microsoft’s IoC container into a console application. Then we had a look at service registration and retrieving the services using a provider. Finally, we had some fun with it by using multiple implementations of a given service type and some of the learnings along the way. This is just the tip of the iceberg though. There’s more stuff to dive into, which I am hoping to cover in the future.

Hope you enjoyed this post and see you next time. Cheers! 👋

References

Understanding .NET Generic Host Model

Thu, 13 Aug 2020 00:00:00 GMT

In this article, we will concentrate on how the Generic Host model hosts ASP.NET Core 3.x Web app and a Worker Service. We will first discuss the definition of a Host and its configuration. In the subsequent sections, we will dive into the implementation details from a higher level.

So what’s the deal with the Generic Host

With the separation of execution and initialisation, Generic Host provides us with a cleaner way to configure and start up our apps. By default, when you create an ASP.NET Core app now, your application will be hosted using the Generic Host model. If you create a new worker service app, it will be hosted the same way.

Not only that, but this model also provides you standardised configuration, DI, logging, and many more. You can even create a traditional console app, beef it up and make use of Generic Host.

💡 Follow along with the code from this repository

The Host

According to the official documentation, a Host is,

ASP.NET Core apps configure and launch a host. The host is responsible for app startup and lifetime management. At a minimum, the host configures a server and a request processing pipeline. The host can also set up logging, dependency injection, and configuration.

Let’s create a new .NET 3.1 WebAPI and a Worker Service project

dotnet new webapi -n WebApplication
dotnet new worker -n WorkerService
dotnet new sln
dotnet sln add WebApplication WorkerService

If you open up the solution in an IDE, you will see the following project structure.

They both have a Program.cs which takes care of setting up a host. In the case of the WebApplication project, it sets up a request processing pipeline defined in a Startup.cs and in the WorkerService project, sets a new hosted service which is an essentially an IHostedService.

In the WebApplication project, when you open up the Program.cs file, you will find the following boilerplate code has been added by the template:

public static void Main(string[] args)
{
    CreateHostBuilder(args).Build().Run();
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });

And, in the WorkerService project we have the following code:

public static void Main(string[] args)
{
    CreateHostBuilder(args).Build().Run();
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddHostedService<Worker>();
        });

Except for the ConfigureWebHostDefaults() and ConfigureServices(), everything else is the same.

Host Configuration

If you look at the CreateHostBuilder method in the above code, it calls a CreateDefaultBuilder static method from Host coming from Microsoft.Extensions.Hosting namespace. It looks like that when we scaffold an ASP.NET Core app, it gives us a .NET Generic Host by default now. We used to have Web Host in ASP.NET Core 2.x, which was made deprecated since ASP.NET Core 3.0. For any future applications, it is recommended to use the .NET Generic Host.

This does a few things under the covers by wrapping,

Dependency Injection services
HTTP Server implementation (such as Kestrel)
Logging
Configuration etc.

In order to get an idea what the above methods do, I looked into the source code on Github.

We will start off with CreateDefaultBuilder method first.

Host.CreateDefaultBuilder()

public static IHostBuilder CreateDefaultBuilder(string[] args)
{
    // Initialize a new HostBuilder object
    var builder = new HostBuilder();

    // Specify the content root directory
    builder.UseContentRoot(Directory.GetCurrentDirectory());

    // Host Configuration : Add environment variables starting with DOTNET_
    // and add any command line args passed
    builder.ConfigureHostConfiguration(config => ... );

    // App Configuration : Add appsettings.json (depending on the env.) files
    // and add user secrets if in development mode
    builder.ConfigureAppConfiguration((hostingContext, config) => ... )

    // Config logging
    .ConfigureLogging((hostingContext, logging) => ... )

    // Use default DI provider
    .UseDefaultServiceProvider((context, options) => ... );

    return builder;
}

As you can see, it pretty much configures a HostBuilder object and returns it. There’s nothing really specific to web hosting in here. This is why it’s common to both HTTP and non-HTTP workloads.

Taking a step further, let’s look at how the web host gets configured. We will now look through ConfigureWebHostDefaults method.

GenericHostBuilderExtensions.ConfigureWebHostDefaults()

public static IHostBuilder ConfigureWebHostDefaults(this IHostBuilder builder, Action<IWebHostBuilder> configure)
{
    return builder.ConfigureWebHost(webHostBuilder =>
    {
        WebHost.ConfigureWebDefaults(webHostBuilder);

        configure(webHostBuilder);
    });
}

Remember that ConfigureWebHostDefaults is used only for HTTP workloads and let’s see what we get as the default web host configuration.

WebHost.ConfigureWebDefaults()

internal static void ConfigureWebDefaults(IWebHostBuilder builder)
{
	// Configure static web assets if in Development mode
    builder.ConfigureAppConfiguration((ctx, cb) => ... );

	// Configure Kestrel
    builder.UseKestrel((builderContext, options) => ... )

	// Configure the default services
    .ConfigureServices((hostingContext, services) => ... )

	// Configure IIS for Windows
    .UseIIS()
    .UseIISIntegration();
}

So far, we have seen that both approaches use the same Generic Host paradigm in the two projects. If you are interested in customising the default configuration, head over to Microsoft Docs’ official documentation.

Finally, how does it all run?

Now comes the interesting part.

In both cases, after the configuration sections, we finally call the Run() on IHost object implemented in HostingAbstractionsHostExtensions. This will run the app and block the calling thread until the host is shut down. This is enabled by WaitForShutdownAsync which is called at the beginning of the start-up process, which can be triggered by Ctrl+C/SIGTERM or SIGINIT.

Let’s look at how both web hosts and worker services run.

For a worker service, remember how we registered our Worker class by passing it into ConfigureServices method. This Worker class extends BackgroundService which in turn implements IHostedService. IHostedService provides 2 methods, namely, StartAsync and StopAsync. So when we run our host, it must be retrieving our Worker service and invoking these methods.

Source: Microsoft

In the Host.cs there’s a separate StartAsync method and we can find the following lines inside it.

_hostedServices = Services.GetService<IEnumerable<IHostedService>>();

foreach (var hostedService in _hostedServices)
{
    // Fire IHostedService.Start
    await hostedService.StartAsync(combinedCancellationToken).ConfigureAwait(false);
}

So our guess was correct. It certainly invokes the StartAsync method of BackgroundService, that calls ExecuteAsync method in which we have ultimately implemented in our Worker class.

For a web host, there’s a little bit of abstraction on top of this before it hits the above section. A summary of how it reaches this as follows;

In Program.cs, configure a new webhost builder object in ConfigureWebHostDefaults
Register Startup class
GenericHostBuilderExtensions.ConfigureWebHostDefaults method gets called
GenericHostWebHostBuilderExtensions.ConfigureWebHost gets called
Register a GenericWebHostService service

public static IHostBuilder ConfigureWebHost(this IHostBuilder builder, Action<IWebHostBuilder> configure)
{
    var webhostBuilder = new GenericWebHostBuilder(builder);
    configure(webhostBuilder);
    builder.ConfigureServices((context, services) => services.AddHostedService<GenericWebHostService>());
    return builder;
}

So what is a GenericWebHostService ? It’s an IHostedService 🤩. Rest of the story is as above as we looked at in the worker service scenario. Because of this nicely decoupled initialisation we are able to run both ASP.NET Core and Worker services on the Generic Host.

Summary

To summarise, we looked at what makes the Generic Host generic and dug deeper into the implementation details in .NET Github repo. We also looked at what makes an ASP.NET Core web application and a worker service different, configuration-wise. This post became a bit longer than I initially I thought it would be 😅 Nevertheless, hope you picked up a thing or two.

Cheers!

References

D3.js Creating a Bar Chart from Ground Up

Sun, 05 Jul 2020 00:00:00 GMT

Creating a bar chart is not that difficult, or is it? Today, we are going to dissect the basic elements of a bar chart and create it from the ground up using D3.js. It’s pretty easy to copy bits and pieces and construct a bar chart. But, the motivation behind this post is to cover the concepts behind creating a bar chart.

You can clone the finished version of this tutorial - D3 Bar Chart repo

I will be using D3.js v5.16.0 which is the latest version of as of yet. Before jumping into coding, first, we need to understand what is the anatomy of a D3 chart.

Anatomy of a D3 Bar Chart

Again, I’m going to keep this as simple as possible. We won’t be covering stuff like calling a Web API, loading a CSV, filtering, cleaning, sorting etc. D3 uses SVG and its coordinate system under the hood - i.e. 0px, 0px are at top left corner.

So, let’s start with a blank SVG and set its width and height.

HTML Structure

<!DOCTYPE html>
<html lang="en">
<head>
  <title>D3 Playground</title>
  <style>
    svg {
      background-color: #ccc;
    }

    rect {
      stroke: black;
      stroke-width: 0.5px;
    }
  </style>
</head>
<body>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.16.0/d3.min.js"></script>
  <svg></svg>
  <script>// Our code goes here</script>
</body>
</html>

Now we will add our initial JS code to set things up.

var data = [1, 2, 3, 4, 5];
var width = 800, height = 300;
var margins = {top: 20, right: 20, bottom: 20, left: 20};

// Create the SVG canvas
var svg = d3.select('svg')
            .attr('width', width)
            .attr('height', height);

Following diagram shows the key elements we will be having in our bar chart.

Scales

Now we let’s set a scale to our data across x and y axes. By using scales, we can define how each data element can be mapped to pixels on the screen.

Let’s create our scale for x axis,

var xScale = d3.scaleBand()
	.domain([0, 1, 2, 3, 4])
	.range([0, width - (margins.left+margins.right)]);

scaleBand is used when you have ordinal values for your axis. So, it will take any amount of ordinal values in the domain function and spit out values specified in the range function. The reason why we deduct the margins is that we need our bars to fit within the margins of our chart. We will now get values from 0px to 760px.

And the scale for y axis,

var yScale = d3.scaleLinear()
	.domain([1, 5])
	.range([margins.top, 100]);

Since our y axis is going to have quantitative continuous values, we choose the scaleLinear function to map our y values. In our dataset, the min is 1 and the max is 5. So we provide 1 and 5 as an array into the domain. Now our range is from 10px to 100px. Why 100px? Just bear with me on this one.

Now let’s add some margins around in our SVG canvas. Otherwise, you will see clipping and other sorts of problems when you have data on your chart. For this we can use a SVG group element and a transformation.

svg.append('g')
  .attr('transform', 'translate('+ margins.top +','+ margins.left +')')

This is clearly visualised in Mike Bostock’s Observable notebook.

Let’s add the rest of the code to draw the bars.

svg.append('g')
  .attr('transform', 'translate('+ margins.top +','+ margins.left +')')
  .selectAll('rect')
  .data(data)
  .enter()
  .append('rect')
    .attr('x', function(d, i) {
      return xScale(i); // We only need the index. i.e. Ordinal
    })
    .attr('y', function(d, i) {
      return yScale(d); // We need to pass in the data item
    })
    .attr('width', xScale.bandwidth()) // Automatically set the width
    .attr('height', function(d, i) { return yScale(d); })
    .attr('fill', 'lightblue');

In the above code, we have put our bars into a element to group them so that we can transform them easily. Since we are using translate method, it will add 10px to x and y coordinates of each element we will be drawing inside it. The rest of the code works according to D3 data joins.

Let’s run this and see,

Our DOM looks likes this now,

<svg width="800" height="300">
  <g transform="translate(20,20)">
    <rect x="0" y="20" width="152" height="20" fill="lightblue"></rect>
    <rect x="152" y="40" width="152" height="40" fill="lightblue"></rect>
    <rect x="304" y="60" width="152" height="60" fill="lightblue"></rect>
    <rect x="456" y="80" width="152" height="80" fill="lightblue"></rect>
    <rect x="608" y="100" width="152" height="100" fill="lightblue"></rect>
  </g>
</svg>

Oops, why is it like upside down? Remember that the SVG coordinates start from the top-left corner. So everything gets drawn relative to that point. Which means we need to change the range of our y values. Let’s fix this.

var yScale = d3.scaleLinear()
  .domain([1, 5])
  .range([height - (margins.top+margins.bottom)*2, 0]);

Wait, what’s this calculation? We are basically setting the max and min values for our y range. In other words, we need our max y value to go up until 220px because we need to account for the height of the bar as well.

Almost there, but the heights look weird. That’s because we changed our y scale. Now, let’s fix the height.

.attr('height', function(d, i) { 
  return height - (margins.top+margins.bottom) - yScale(d); 
})

Remember, we need to deduct the top and bottom margins from the total height so that whatever the value we get from yScale will not exceed that boundary.

Cool, now we are getting somewhere 😁

Axes

D3’s axes API is pretty straight forward. You can utilise that to add horizontal and vertical axes to any graph. To wrap up our bar chart, let’s add the axes.

X axis

svg.append('g')
  .attr('transform', 'translate('+ margins.left +','+ (height - margins.top) +')')
  .call(d3.axisBottom(xScale));

Y axis

svg.append('g')
  .attr('transform', 'translate('+ margins.left +','+ margins.top +')')
  .call(d3.axisLeft(yScale));

Looks okay, but the axes are a bit off. So let’s fix that.

var margins = {top: 30, right: 30, bottom: 30, left: 30};

Simple! When creating a graph in D3 always remember to use variables when possible so that you can easily fix if something’s not looking good.

And we are done!

Great! and we are done ✅

References

D3.js Join Semantics - A Conceptual Look

Tue, 30 Jun 2020 00:00:00 GMT

It’s really easy to get started with D3 when you go through their excellent documentation. However, the goal of this post is to give you a sense of what patterns it uses underneath for joining DOM elements and data items.

The key element of D3 is that it treats data you want to visualise as a database. We can also think that D3 treats the elements you have on your webpage as a database.

Data Join Semantics

Let’s say we have the following SVG elements in our DOM and a variable data with some values we want to bind it with.

<svg x="100" y="100">
	<rect></rect>
	<rect></rect>
	<rect></rect>
	<rect></rect>
	<rect></rect>
</svg>

Now let’s see how D3 joins the data with DOM elements. We will use the above structure as a reference.

Update

d3.selectAll('rect').data([1, 2, 3, 4, 5])

So we have a database of DOM elements and a database of data items. When we have a 1:1 mapping with them we call it the Update section. Once this association is formed internally in D3, we can manipulate our DOM elements using methods such as .attr().

This situation is called “Update” stage. Think of this as a natural join between two database tables.

Enter

What if we have more data elements than our DOM elements?

For such cases that go into a different area in the selection called Enter area. That’s when we can access that area with the .enter()method in D3. We can then tell D3 to what to do with those extra data items. For instance, if we want to create new ‘rect’ DOM elements, we can do a,

d3.selectAll('rect')
	.data([1, 2, 3, 4, 5, **6**])
	.enter() // accessing the Enter area
	.append('rect') // telling what to do with it
	.attr('height', ...) // rest of the manipulations

Exit

Finally, we have the case of having more DOM elements than our data elements.

Imagine now we have 5 data elements but only 4 DOM elements. So these extra UI elements go into Exit area. Just like before, we just need to tell D3 what to do with those extra items? Remove you say? 👍

d3.selectAll('rect')
	.data([1, 2, 3, 4, 5])
	.exit()
	.remove();

Mapping with Relational Algebra

Remember we are now thinking about D3 in terms of database. Let’s see how we can map this into relational algebra.

As we can see, when we have UI elements and data with 1:1 mapping, it like a natural join. When we have more data than UI elements, it like an anti-join where we get the extra data elements in Enter stage. As with the Exit stage, we get the extra UI elements that do not map with data.

Now we know the underpinning principles of D3 join semantics. This helps us to dive into a D3.js code without worrying too much about syntax.

What would happen if we do a .selectAll() on an empty set of DOM elements? Can we still use .enter()? There’s a well-known pattern for that as well 😊 I will explain that in a separate blog post.

References

Securing Hangfire Dashboard in ASP.NET Core with a Custom Auth Policy

Fri, 15 May 2020 00:00:00 GMT

This blog post assumes that you have a basic understanding of Hangfire. If not, you can follow their excellent guide on how to get started. The scope of this blog post is to show you how to secure the Hangfire Dashboard and only authorise certain users to be able to access it without opening it up to the general public.

By default, if you try to access your Hangfire dashboard, it will work perfectly fine on your localhost because local requests are allowed. However, things can get a bit tricky when you want to secure your Dashboard.

For starters, Hangfire provides us with a IDashboardAuthorizationFilter to perform our custom authorisation when deployed. But, what if we want to do the authorisation at an endpoint level rather than using a custom filter?

Step 1 - Initial Setup

To get started, you can clone the repo I have put together:

Securing Hangfire Dashboard

To summarise, the above solution is a .NET Core 3.1 Web API project and has the following dependencies.

We use MemoryStorage since we don’t want to persist anything for this demo. This will work just fine with SQLStorage. To keep things simple on the authentication side of things, I have used AzureAD.UI library.

Once you have cloned the repo, do not forget to create an app registration in the Azure portal, under Active Directory. Make sure you have set the correct Redirect URIs (Note: we need to add hangfire URI) and selected ID tokens.

You also need to copy and paste the ClientId and TenantId from your app registration to appsettings.json

"AzureAd": {
  "ClientId": "Enter_the_Application_Id_here",
  "TenantId": "Enter_the_Tenant_Info_Here"
}

Once you have done this setup, you can just press F5 and head over to https://localhost:44317/hangfire to see if everything is working.

Step 2 - Adding a custom authorisation policy

For our server to tell which policy to use to authorise users to access the Hangfire dashboard, we need to define a custom authorisation policy.

Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // Add Authentication
    
    ...Code removed for brevity

    // Add a new policy for hangfire
    services.AddAuthorization(options =>
    {
        // Policy to be applied to hangfire endpoint
        options.AddPolicy(HangfirePolicyName, builder =>
        {
            builder
                .AddAuthenticationSchemes(AzureADDefaults.AuthenticationScheme)
                .RequireAuthenticatedUser();
        });
    });

    ...Code removed for brevity

    // Bootstrap Hangfire
    services.AddHangfire(configuration => configuration
        .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
        .UseSimpleAssemblyNameTypeSerializer()
        .UseRecommendedSerializerSettings()
        .UseMemoryStorage());
}

Step 3 - Securing the Dashboard

Now that we have defined our custom authorisation policy, let’s configure our Hangfire dashboard endpoint to require authorisation with our HangfirePolicyName policy.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ...Code removed for brevity

    // Hangfire Settings
    app.UseHangfireServer();
    app.UseHangfireDashboard();

    // Configure endpoints
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
        endpoints.MapHangfireDashboard("/hangfire", new DashboardOptions()
        {
            Authorization = new List<IDashboardAuthorizationFilter> { }
        })
        .RequireAuthorization(HangfirePolicyName);
    });

    //Register our background job
    RecurringJob.AddOrUpdate("some-id", () => Console.WriteLine(), Cron.Minutely);
}

It is important to remember that we need to pass in an empty list of IDashboardAuthorizationFilter because otherwise, Hangfire will block all your requests to the dashboard when deployed (i.e. works only on localhost if we don’t override it like this).

Now head over to https://localhost:44317/hangfire you will be asked to login since our new hangfire dashboard endpoint is secured with Azure AD. Once you log in, you will see the dashboard.

Step 4 - Requiring a Specific Role

We can also add custom role checks inside our AddPolicy() call. This is quite useful and a more secure way to protect your Hangfire Dashboard endpoint rather than granting access all authenticated users of your system.

...
services.AddAuthorization(options =>
    {
      // Policy to be applied to hangfire endpoint
      options.AddPolicy(HangfirePolicyName, builder =>
      {
        builder
          .AddAuthenticationSchemes(AzureADDefaults.AuthenticationScheme)
          .RequireAuthenticatedUser();

        // Your custom role check
        if (env.IsProduction())
        {
          builder.RequireRole("AdministratorRoleName");
        }
      });
    });
...

Conclusion

With this approach, you can have a nicely decoupled way of protecting your Hangfire dashboard route. You can also move the authorisation logic to a custom extension method and inject your custom services as opposed to using a Hangfire’s authorisation filter.

~~There’s a pull request in Hangfire repository to add an extension method to add authorisation policy support by a colleague of mine. Feel free to upvote it 🙂~~

💡 UPDATE: The above PR has gone in and the new extension method will be available as part of 1.7.24 release.

References

Multi-stage Image Builds with Docker

Sun, 19 Apr 2020 00:00:00 GMT

What and Why Behind Multistage Builds

In a Dockerfile, each statement adds up a new layer to the image. It could be counterproductive if you also build your application package when building your Docker image. This could increase the size of our docker images substantially. That’s when we need to leverage multi-stage builds feature.

Simply put, multistage builds are useful when we want to clean up and reduce the image size without keeping unwanted artifacts lying around in our image.

Let’s understand this through an example.

A Concrete Example

I have created a sample React app and 2 Dockerfiles to demonstrate this, namely, Dev.Dockerfile and Prod.Dockerfile

Typical Dockerfile without stages

Dev.Dockerfile

FROM node:13.13.0-alpine
WORKDIR /app

COPY package.json ./
RUN npm install --silent
RUN npm install [email protected] -g --silent
COPY . ./

CMD ["npm", "start"]

Let’s build and run this Docker image:

docker build -t multi-stage-demo:dev -f Dev.Dockerfile .

In the lightweight version, we use the same base image to build the application, but we are using nginx base image to run our application since we already have our application built.

Prod.Dockerfile

# Stage 1 - the build process
FROM node:13.13.0-alpine as build-deps
WORKDIR /app

COPY package.json ./
RUN npm install --silent
RUN npm install [email protected] -g --silent
COPY . ./
RUN npm run build

# Stage 2 - the deploy process
FROM nginx
COPY --from=build-deps /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Here, you’d notice that we have two FROM directives. The first one will be used for the base image for building the application and the second one for the final image. The built binaries are copied over from the first image over to the second and we expose port 80 to the host.

Now, let’s give this a go and build it:

docker build -t multi-stage-demo:prod -f Prod.Dockerfile .

Time to check our image sizes

docker images | grep multi-stage-demo

If we compare dev to prod image, that’s a ~73% reduction! 😍This can also benefit you in reduced deployment times.

The Github repo with code is available here

Where to From Here?

There are some nice examples and best practices defined in the official Docker documentation.

For example, if your build contains several layers, you can order them from the less frequently changed (to ensure the build cache is reusable) to the more frequently changed:

Install tools you need to build your application

Install or update library dependencies

Generate your application

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

References

Deploying Multi-Container Services on Azure Service Fabric

Mon, 13 Apr 2020 00:00:00 GMT

To give you a bit of context on what we will be looking at today, we will have one goal - to deploy a microservices application to Azure with Service Fabric. My preferred choice of a cloud these days would be Azure since I have been working on it for a while now.

To keep things simple, we will be looking at the deployment of a multi-container application rather than worrying too much about what the application does.

Prerequisites

If you followed my previous post, you would already have the below mentioned installed on your developer workstation.

Docker Desktop
Azure account
Azure Service Fabric SDK and tooling

Creating a demo app

Now that we have some basic understanding of what Azure Sevice Fabric is capable of, let’s look at what we are going to building today

Create a containersed React app
Create a containerised .NET Core service
Create a Service Fabric project
Wrap our above mentioned service in the Service Fabric project
Build a CI/CD pipeline in Azure DevOps

At a glance, this will be the architecture of our demo project.

Scaffolding our multi-container application

You can clone my Github repository to get started.

git clone https://github.com/sahansera/deploying-containers-on-service-fabric.git

The folder structure is as follows:

For each project, I have created a Dockerfile. What they basically do is, build each project and expose a port from their respective containers.

Web - Dockerfile

# Stage 1 - the build process
FROM node as build-deps
WORKDIR /usr/src/app
COPY web-client/package.json web-client/yarn.lock ./
RUN yarn --no-bin-links
COPY ./web-client/. ./
RUN yarn build

# Stage 2 - the deploy process
FROM nginx
COPY --from=build-deps /usr/src/app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

The reason for having stages here is because we don’t want to include the source code in the container since we only need to copy the executable or the build output over to the container.

ProductCatalog - Dockerfile

For the .NET Core project, we have a bit more involved since we need to have the .NET Core 3.1 SDK to build the project and .NET Core 3.1 runtime to run the project.

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY ./*.sln .
COPY ./ProductCatalog/*.csproj ./ProductCatalog/
RUN dotnet restore

# Copy everything else and build
COPY ./ProductCatalog/. ./ProductCatalog
WORKDIR /app/ProductCatalog
RUN dotnet publish -c Release -o app

# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS runtime
WORKDIR /app
COPY --from=build-env /app/ProductCatalog/app .
ENTRYPOINT ["dotnet", "ProductCatalog.dll"]

Building and tagging images

We can run the following command to test out our images locally.

If you haven’t used Docker Hub before. Here’s a quick intro to creating a repository and pushing images.

In a terminal, navigate to ProductCatalog/ folder and run the following command

docker build -f . -t <your_docker_hub_username>/service-fab:product-catalog-img

Now change directory to Web/ folder, let’s build and tag backend service’s Docker image

docker build -f . <your_docker_hub_username>/service-fab:web-shop-img

Pushing the images to Docker Hub

In order for Service Fabric to fetch and build our container images, we need to first push them to a container registry. I’m using Docker Hub here as an example. But you can use any other registry such as GCR, ACR etc. These repositories can be either public or private.

Make sure you are already logged in to Docker service from the command line,

docker login --username=yourhubusername --email=[email protected]

Check the image ID using,

docker images

This will come in handy in the following command.

Now let’s push the images with,

docker push <image_name>

We need to run this command twice for both frontend and backend images we created.

Configuring Service Fabric project

Let’s open up WebShop solution in Visual Studio. You can configure the credentials in your Service Fabric project.

Frontend configuration WebShopClientPkg/ServiceManifest.xml

<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
  <EntryPoint>
    <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
    <ContainerHost>
      <ImageName>**index.docker.io/sahan/service-fab:web-shop-img**</ImageName>
    </ContainerHost>
  </EntryPoint>
  <!-- Pass environment variables to your container: -->
  <!--
  <EnvironmentVariables>
    <EnvironmentVariable Name="VariableName" Value="VariableValue"/>
  </EnvironmentVariables>
  -->
</CodePackage>

Backend service configuration WebShopApiPkg/ServiceManifest.xml

<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
  <EntryPoint>
    <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
    <ContainerHost>
      <ImageName>**index.docker.io/sahan/service-fab:product-catalog-img**</ImageName>
    </ContainerHost>
  </EntryPoint>
  <!-- Pass environment variables to your container: -->
  <!--
  <EnvironmentVariables>
    <EnvironmentVariable Name="VariableName" Value="VariableValue"/>
  </EnvironmentVariables>
  -->
</CodePackage>

Note how I have replaced the path to the Docker Hub registry in both configs.

Creating a Service Fabric cluster

Before deploying our services, we need to create a Service Fabric cluster in Azure. In the Azure portal, just type in service fabric and click on Create service fabric cluster.

Now, fill in the cluster info as shown below.

In the cluster configuration step, you will be asked to set the Node type count. Just set it to 2 to denote frontend and backend services. In the same page, you will be asked how many nodes you want to create for each node type. Select Single node cluster, for now, otherwise, you will be charged for the total number of VMs. However, this is not recommended for production-grade apps.

In order to expose our frontend and backend services let’s enter endpoints as port 80 and 81 respectively.

Next up, we need to create a key vault for our service fabric cluster. This is a very important step and make sure to select the Azure Virtual Machines for deployment checkbox. You will also need to enter a name for the certificate. We will cover this in a bit. Give it some time to create the key vault and proceed to the next step.

Once it’s up, you will see you cluster like so,

It may take up to 15 minutes until your VMs start to appear under the Nodes list.

Installing client certificate

In order to be able to access the Service Fabric dashboard, you need to install the client certificate from the key vault.

Head over to Azure Portal → ’Key Vault’ → Select the key vault we created in the previous step’s Certificates
Select the certificate (there should be only 1 at this stage)
Click Download PFX/PEM format

Once downloaded double click the certificate. You don’t need to enter anything in the Password field and click Next and complete the wizard.

Now, you should be able to access the dashboard of your cluster. The link will be in Service Fabric home page in Azure Portal.

Deploying to Azure

The fast and the quickest way to deploy your app is by using Visual Studio. Just right click on the service fabric project and select Publish

After about 5 minutes you should be able to see that our services have got deploy to the cluster. You can check that from the following URL:

http://<yourclustername>.<yourclusterregion>.cloudapp.azure.com/

Or, you can get the URL from service fabric portal and remember to delete the port number and revert it to http://

Our frontend running at port 80

Our backend service running at port 81

In my next post, we will look at how to set up an Azure DevOps build pipeline to deploy your services to a designated Service Fabric cluster. ✌

References

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-overview

Setting up your Raspberry PI 4 - Headless mode

Tue, 31 Mar 2020 00:00:00 GMT

When I first got my Raspberry PI 4 (RPI for the rest of the article), I was so excited to get started. However, there was one problem. How do I get started? To make matters worse it only had 2 micro-HDMI ports and I couldn’t connect it to a monitor. After a while I found myself wading through link after link.

I am going to show you what worked best for me and a recipe you can also use to get started with your RPI 4 quickly - without an external monitor 🙂

Specs

Source: https://www.raspberrypi.org/products/raspberry-pi-4-model-b/

Step 1 - Installing an OS

My choice of OS for the RPI is Raspbian since it’s easy to get started with and comes with a lot of tools out of the box. I would recommend selecting “with Desktop” option if you are planning to use it with a window manager.

Once you have got the ISO downloaded we need to flash it to a microSD card. I used balenaEtcher since it was pretty straight forward to use.

If you are using Windows, it will ask you to format the drive when opened with Windows Explorer. Don’t do that 😀 Otherwise, you will find yourself doing this step over and over again.

Now, open the microSD drive in your Windows Explorer or Finder. For us to be able to connect via ssh later, we need to create an empty file called ssh at the root of the drive (without any extension).

Safely eject, unplug the microSD card and chuck the microSD card in to the RPI’s slot.

Step 2 - Connecting the RPI to LAN

This is going to be a one-off step. We need to connect the RPI to the router with an ethernet cable (RJ45) and boot it up. Give it a minute to settle down.

After that, we need to find the local IP of our RPI for us to be able to SSH into it. Normally you can head over to 192.168.1.1 or routerlogin.net if you are using a NETGEAR router.

You need to find the connected list of devices of your router. Now, copy the IP of your RPI

Step 3 - SSH in to your RPI (Headless mode)

You can wither use PuTTY if you are on windows or your terminal on a *nix system.

ssh pi@your_rpi_local_ip

In default settings, the password would be just raspberry

There’s only a single step remaining to get in to your RPI’s desktop environment.

Step 4 - Connecting to RPI with VNC

Now that you know how to ssh into the RPI, we need to use it in GUI mode. We are going to use a VNC provider for that. VNC is a graphical desktop sharing system that allows us to remotely control the desktop interface of our RPI (running VNC Server) from another computer or mobile device (running VNC Viewer).

While you are connected your RPI, let’s run the following commands in the terminal

  sudo apt update
  sudo apt install realvnc-vnc-server realvnc-vnc-viewer

You can enable VNC Server at the command line using raspi-config:

  sudo raspi-config

Now, enable VNC Server by doing the following:

Navigate to Interfacing Options.
Scroll down and select VNC > Yes.

For our changes to take effect, let’s reboot the RPI

  sudo reboot

While it’s rebooting, let’s download the RealVNC viewer,

Once the viewer is installed on your local machine, let’s open it up and enter the IP address of your RPI, username & password.

Don’t forget to connect the RPI to your home’s Wifi network. Now you can unplug the RJ45 from the RPI and keep it anywhere you want.

Hope this article helped you. In my next article, we will see how you can configure your own Kubernetes cluster on your RPI!

Enjoy your new Raspberry PI 🥧

References

Getting Started with Azure Service Fabric

Mon, 23 Mar 2020 00:00:00 GMT

tl;dr - if you only want to get started quickly, just jump ahead to “Setting up a Development Environment” section

I have been experimenting with container orchestrators to help me deploy a microservices-style hobby project I have been working on (more on this in a separate blog post) to the cloud. If there’s one thing I learnt from this project, there are a lot of moving parts to it and might not be easy to manage a distributed cluster with hundreds of services that are fast, reliable and scaling.

While I was on the journey in finding a platform/framework with a good balance of container orchestration, managing and monitoring infrastructure, I came across Azure Service Fabric which looked promising.

The following definition is the one that caught my eye,

The aim of Service Fabric is to solve the hard problems of building and running a service and to use infrastructure resources efficiently, so teams can solve business problems by using a microservices approach. Service Fabric is agnostic about how you build your service, and you can use any technology. But it does provide built-in programming APIs that make it easier to build microservices.

For starters, you can have a quick look at Microsoft’s official page to learn more about Azure Service Fabric.

Source: https://docs.microsoft.com/en-au/azure/service-fabric/service-fabric-overview

Options in Azure for Microservices

Here’s a quick summary of different options you can use to build microservices type applications in Azure,

AKS: Microsoft’s Managed Kubernetes service with Docker as runtime
Service Fabric: Another step forward from AKS. All about microservices and, - Service communication - Service discovery - Monitoring and Telemetry - Provision and upgrade - Testing locally - Scalability - Manage downtimes
- Focus on business objectives and infrastructure. In AKS you need to build your infrastructure. Build and deploy from day1, but still be able to customize.
- Proven tech - Many azure services run on ASF. Cosmos DB, Azure SQL, Power BI, Skype, IoT Hub, Cortana, ACR, Azure DevOps
Azure Functions - Mini version of microservices. Event-driven (triggers, scheduled). Easier to get started than Azure Service Fabric and Azure Kubernetes Service

Supported Environments

Public Clouds

Service Fabric is designed to run on any cloud platform other than Azure, such as AWS, GCP etc. However, as you might have already guessed, Azure has first-class support.
On Premise

If you want to manage your own premise cluster, ASF supports Windows Server 2012 R2, 2016 or higher, Linux Ubuntu 16.04 and 18.04. However, when it comes to Macs, you are kind of on your own since you will have to rely on an emulator for that.
Developer Workstation

Service Fabric does not have a dependency on Visual Studio since it’s designed to be modular. This is great because you can have a similar setup to your production environment in your development machine. It also brings you much good stuff such as simulating real-life disasters and fixing issues before they go into production!

Programming Models

Azure service fabric provides us with different ways to wrap our services in. Here’s a summary of programming models used in Service Fabric.

Reliable Service: Used to address single or multiple business concerns.
- Stateless - Acts like a console application
- Stateful - With its own transactional replicated storage
Reliable Actors: Service Fabric uses virtual actors to handle communication between servers. Here’s an excellent article on Actor Model
Guest Executables: run any executable without any code changes
Containers: Separate blog post will be published on this (I promise!)

Setting up a Development Environment

To set up your local environment, we need to first install the necessary tooling. You can also have a look at Service Fabric’s official documentation on getting started.

Make sure you follow these steps,

Install Visual Studio 2019 - Community edition is fine. In fact, you can use an editor such as Visual Studio Code. But it would be easier to use VS to get something up and running.
Install Service Fabric tooling
Install Service Fabric SDK - If you run Visual Studio and try to create a new Service Fabric project it will ask you whether you want to install the Service Fabric. However, you can also install the SDK with the Microsoft Web Platform Installer (don’t forget to scroll down to Download Web PI section)

Verifying your local setup

Once you have installed the prerequisites, you can open up Visual Studio and select Create a new project

In the Create a new project dialog, filter by typing “fabric” and select Service Fabric Application

Enter a project name. You don’t need to worry about Framework version unless you are working on a classic ASP application.

You will get another popup to select a project type. For this demo, we will select Stateless Service and enter a name for our project.

In your solution explorer you will find 2 projects like so, you can also double check with my repo if you managed to get this far.

The first project is just a wrapper around the Verification Project. If you open up Verification.cs you will see that in RunAsync method is the main entry point of our stateless console app. Hit F5 and see whether your local cluster is working fine. It may take about 2 - 3 mins initially.

Once everything is up and running, you will see the following output in Visual Studio. Put a breakpoint in your code and have a play around it.

Now open up a browser and navigate to http://localhost:19080/

You will be able to see that your application is running in a healthy state in your local cluster.

Congratulations for making it this far 🙌! Stay tuned for my next post on deploying a multi-container application to Azure with Service Fabric.

References

Simple In-Memory Caching in .Net Core with IMemoryCache

Wed, 15 Jan 2020 00:00:00 GMT

Caching is the process of storing the data that’s frequently used so that data can be served faster for any future requests.

💡 UPDATE on April 2025: I have recently migrated this project to .NET 9.0 with modern C# features, improved architecture, and a comprehensive test suite. You can find the previous updates at the bottom of this post.

Suppose we have a very lightweight process which talks to another server whose data is not going to change frequently; “Our service” and “Users Service” (which returns an array of users) respectively.

Without any caching in place, we would be making multiple requests which will ultimately result in timeouts or making the remote server unnecessarily busy.

Walkthrough video

If you like to watch a video walkthrough instead of this article with a thorough explanation, you can follow along on my Youtube channel too 😊

Introduction to IMemoryCache

Let’s have a look at how we can improve the performance of these requests by using a simple caching implementation. .NET Core provides 2 cache implementations under Microsoft.Extensions.Caching.Memory library:

IMemoryCache - Simplest form of cache which utilises the memory of the web server.
IDistributedCache - Usually used if you have a server farm with multiple app servers.

In this example we will be using the IMemoryCache along with latest version of .NET (Core) as of yet, which is version 7.

These are the steps we are going to follow:

Create/Clone a sample .NET Core app
Naive implementation
Refactoring our code to use locking

1. Create/Clone a sample .NET Core app

You can simply clone In-memory cache sample code repo I have made for the post. If not, make sure that you scaffold a new ASP.NET Core MVC app to follow along.

First, we need to inject the in-memory caching service into the constructor of the HomeController

public class HomeController : Controller
{
    private readonly IMemoryCache _cache;

    public HomeController(ILogger<HomeController> logger, IMemoryCache memoryCache)
    {
        _cache = memoryCache;
    }

Note: With .Net Core 3.1 you don’t need to specifically register the memory caching service. However, if you are using a prior version such as 2.1, then you will need to add the following line in the Startups.cs:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMemoryCache(); // Add this line for .NET 2.1

        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
    }
}

2. Naive implementation

For the sake of this tutorial, we’ll use a free external API such as reqres.in/api/users. Let’s imagine we want to cache the response of the API. For simplicity, I have used the example code provided by Microsoft Docs.


// Code removed for brevity
...

// Look for cache key.
if (!_cache.TryGetValue(CacheKeys.Entry, out cacheEntry))
{
    // Key not in cache, so get data.
    cacheEntry = DateTime.Now;

    // Set cache options.
    var cacheEntryOptions = new MemoryCacheEntryOptions()
        // Keep in cache for this time, reset time if accessed.
        .SetSlidingExpiration(TimeSpan.FromSeconds(3));

    // Save data in cache.
    _cache.Set(CacheKeys.Entry, cacheEntry, cacheEntryOptions);
}

Explanation

The code is pretty straightforward. We first check whether we have the value for the given key present in our in-memory cache store. If not, we do the request to get the data and store in our cache. What SetSlidingExpiration does is that as long as no one accesses the cache value, it will eventually get deleted after 10 seconds. But if someone accesses it, the expiration will get renewed.

Suppose we want to get a list of users as per our use case. Here’s my implementation with a bit of code re-structuring:

var users = _cacheProvider.GetFromCache<IEnumerable<User>>(cacheKey);
if (users != null) return users;

// Key not in cache, so get data.
users = await func();

var cacheEntryOptions = new MemoryCacheEntryOptions()
    .SetSlidingExpiration(TimeSpan.FromSeconds(10)); 

_cacheProvider.SetCache(cacheKey, users, cacheEntryOptions);

return users;

await func() will wait and return the response from our external API endpoint (provided that it’s passed into our method) so that we can use that value to store in our cache.

Have a look at the full implementation of my GetCachedResponse() in CachedUserService.cs for a more generic solution to handle any type of data.

This gets the job done for a very simple workload. But how can we make this more reliable if there are multiple threads accessing our cache store? Let’s have a look in our next step.

3. Refactoring our code to use locking

Now, let’s assume that we have several users accessing our service which means there could be multiple requests accessing our in-memory cache. One way to make sure that no two different users get different results is by utilising .Net locks. Refer the following scenario:

💡 Note: I need to mention that the IMemoryCache is thread-safe by design. However, if you are using doing computationally heavy task when creating the initial cache entry it’s advisable to use a Semaphore to synchronize the threads accessing the same logic, as shown in my examples.

Let’s breakdown the sequence of requests and responses:

User A makes a request to our web service
In-memory cache doesn’t have a value in place, it enters in to lock state and makes a request to the Users Service
User B makes a request to our web service and waits till the lock is released
This way, we can reduce the number of calls being made to the external web service. returns the response to our web service and the value is cached
Lock is released, User A gets the response
User B enters the lock and the cache provides the value (as long it’s not expired)
User B gets the response

The above depiction is a very high-level abstraction over all the awesome stuff that happens under the covers. Please use this as a guide only. Let’s implement this!

...
var semaphore = new SemaphoreSlim(1, 1);
var users = _cacheProvider.GetFromCache<IEnumerable<User>>(cacheKey);

if (users != null) return users;
try
{
    await semaphore.WaitAsync();

    // Recheck to make sure it didn't populate before entering semaphore
    users = _cacheProvider.GetFromCache<IEnumerable<User>>(cacheKey);

    if (users != null) return users;
    users = await func();
    
    var cacheEntryOptions = new MemoryCacheEntryOptions()
        .SetSlidingExpiration(TimeSpan.FromSeconds(10)); 

    _cacheProvider.SetCache(cacheKey, users, cacheEntryOptions);
}
finally
{
    // It's important to do this, otherwise we'll be locked forever
    semaphore.Release();
}

return users;

Explanation

Same as in our previous example we first check our cache for the presence of the value for a key provided. if not, we then asynchronously wait to enter the Semaphore. Once our thread has been granted access to the Semaphore, we recheck if the value has been populated previously for safety. If we still don’t have a value, we then call our external service and store the value in the cache.

Have a look at the CachedUserService.cs for the full implementation.

Demo

Hope you enjoyed this tutorial. Happy to know your thoughts! 🙂

Updates

💡 UPDATE on March 2023: I have recently migrated this project to .NET 7 and refactored the code to make it more readable. You can find the previous updates at the bottom of this post. You can still access the .NET Core 3.1 version from this branch

💡 UPDATE on Aug 2022: I have recently migrated this project to .NET 6 also with its minimal hosting model 🎉. You can still access the .NET Core 3.1 version from this branch.

💡 UPDATE on Feb 2021: Head over to my blog post on IDistributedCache if you are interested to see how to integrate with Redis as a distributed caching solution.

References

Understanding Nullable Value Types in C#

Sat, 28 Dec 2019 00:00:00 GMT

I recently started picking out different topics that I like to learn more about. One such thing I wanted to have a closer look at is “Nullable value types in C#“.

We first need to understand the ‘why’ behind nullable value types.

Let’s have a look at this class:

public class User
{
    public string Name { get; set; }
    public int DaysSinceLastLogin { get; set; }
    public DateTime RegistrationTimestamp { get; set; }

    public User()
    {
        DaysSinceLastLogin = -1; // Magic Number
        RegistrationTimestamp = DateTime.MinValue; // Magic Number
    }
}

When designing a class like this, we often come across situations where we want to default a value type to a null or tell the program that there’s no actual value set.

In the above example, since RegistrationTimestamp and DaysSinceLastLogin are value types, we can’t represent that an actual value exists, we need to default them to the magic numbers we chose. Two major drawbacks of this approach are, we wouldn’t know what the actual magic numbers represent and if we decide to change the value we need to do some plumbing to get around it.

For a much cleaner implementation we can make use of C#‘s out of the box Nullable Value Types.

Definition of the Nullable Value Types

The MSDN definition goes as follows:

“A nullable value type T? represents all values of its underlying value type T and an additional null value.”

Let’s think of DaysSinceLastLogin from the above example. Its type is Int32 which spans from -2,147,483,648 to 2,147,483,647. Obviously, in our context we don’t need negative numbers to represent how many days have elapsed since the last login of the user (we could have used uint instead, but let’s not get bogged down with such details). For the sake of the argument, let’s look at how to can convert it to a Nullable value Type.

What this does is, essentially, it wraps our value type of int in a Nullable struct; which allows us to assign it a integer value or a null

Refactoring our code with Magic Numbers

public class User
{
    public string Name { get; set; }
    public Nullable<int> DaysSinceLastLogin { get; set; }
    public Nullable<DateTime> RegistrationTimestamp { get; set; }

    public User()
    {
        DaysSinceLastLogin = null;
        RegistrationTimestamp = null;
    }
}

Now we can simply assign nulls to our DaysSinceLastLogin and RegistrationTimestamp. But, C# is known for its nice syntactic sugar. Let’s sugarcoat it like so:

public class User
{
    public string Name { get; set; }
    public int? DaysSinceLastLogin { get; set; }
    public DateTime? RegistrationTimestamp { get; set; }
}

This is beautiful, and behind the scenes it still wraps then in a Nullable<T> struct. Another good thing is that, now we don’t need to explicitly specify null to DaysSinceLastLogin and RegistrationTimestamp. We all know that nulls are a billion dollar mistake, but still, could have been worse 😁

Convenience properties and methods of Nullable

.HasValue - true if it has a value; false if null
.Value - Gives us the underlying value. However, this will throw an InvalidOperationException if you try to access a value whose .Value property is false
.GetValueOrDefault() - Underlying value or default. Eg: for an integer whose value has not been set, will return 0

-.GetValueOrDefault(T) - Value or a specified default value.

In my next post we will look at how we can access and check for null values in C#. Until then ✌️

References

https://docs.microsoft.com/en-us/dotnet/api/system.nullable-1?view=netcore-3.1

Deploying an ASP.NET Core App on Google Kubernetes Engine

Wed, 25 Dec 2019 00:00:00 GMT

I recently started playing around with the Google Cloud Platform. Since ASP.NET Core now runs on virtually any major OS, I wanted to give it a go with GCP’s Google Kubernetes Engine offering to see whether we can use it to host a .NET Core application.

So here are the steps we are going to follow:

Create/Clone a sample ASP.NET Core app
Build the docker image
Push the containerized app to Google Container Registry (GCR)
Deploy the app to Google Kubernetes Engine (GKE)

First things first, what are the prerequisites?

VS Code or Visual Studio 2019
.NET Core SDK v3.1+
GCloud CLI Tools v274.0.0
Docker and Kubectl

1. Create/Clone a sample ASP.NET Core app

You can grab the ASP.NET Sample app from my Github Repo:

ASP.NET Core Food App

This app has nothing to do with food though 😁 Let’s just say I was hungry when creating it.

You can clone the above repo into your local machine and navigate to the FoodApp folder and publish the app to app/ folder from a command line:

cd FoodAppCore/FoodApp
dotnet restore
dotnet publish -o app

2. Build the docker image

Don’t forget to replace PROJ_ID with your GCP project’s ID

Let’s build the docker image:

docker build -t gcr.io/PROJ_ID/food-app .

If you want to test out your docker build locally, try it by doing so:

docker run --rm -p 8080:8080 gcr.io/PROJ_ID/food-app:latest

3. Push the containerized app to Google Container Registry (GCR)

Let’s push our image to GCR:

docker push gcr.io/PROJ_ID/food-app

If you haven’t configured gcloud CLI to authenticate to GCR you need to run the following command (you need to this only once):

gcloud auth configure-docker

4. Deploy the app to Google Kubernetes Engine (GKE)

Next, we need to create a cluster in GKE to deploy our app to:

gcloud container clusters create foodapp-cluster --num-nodes=3

Finally, let’s deploy it to GKE:

kubectl create deployment food-app --image=gcr.io/PROJ_ID/food-app:latest

If the above command gives you an error, don’t forget to install kubectl for gcloud:

gcloud components install kubectl

Wait, now the app is deployed, how can our clients access it? This final command will expose our app through a load balancer binding port 8080 to 80

kubectl expose deployment food-app --type="LoadBalancer" --port=80 --target-port=8080

Well, that’s it! you have your app running in GKE! If you are wondering how to access it, run a kubectl get service and grab its external IP and paste it in a browser.

References

Demystifying Async & Await in C#

Mon, 02 Dec 2019 00:00:00 GMT

You might have come across the two keywords async and await many times while working on .NET projects. Although it’s a fascinating thing to use, it’ll also be beneficial in knowing what happens under the hood when dealing with such code.

Before getting bogged down with in-depth details what’s what, we need to clarify some common misconceptions about Tasks and Threads used in C#.

🤔 Tasks Vs Threads

Simply put, Tasks are not Threads. If you have ever worked with Promises in other languages such as JavaScript, you will be quite comfortable in using them. Tasks are essentially Promises, which would be completed in a later point in time which lets you deal with the result returned from an asynchronous action. This also means that Tasks can be faulted and queried to know whether they are completed or not. Threads, on the other hand, are a more lower-level implementation of OS-level code executions.

It is really important to understand that tasks are not an abstraction over threads, and we should think of tasks as an abstraction over some work that’s intended to be happening asynchronously. In summary:

Tasks are not Threads
Tasks are similar to Promises providing you with a clean API to handle async code
Tasks do not guarantee parallel execution
Tasks can be explicitly requested to run on a separate thread via the Task.Run API.
Tasks are scheduled to be run by a TaskScheduler

🔮 The Magic of the ‘await’ keyword

As Stephen Cleary clearly mentions in his excellent blog post, async keyword only enables await. So, an async method would simply run like any other synchronous method until it sees an await keyword.

The await keyword is where the magic happens. It gives back control to the caller of the method that performed await and it ultimately allows an IO-bound (eg. calling a web service) or a CPU bound task (eg: such as a CPU-heavy calculation) to be responsive. All async and await does is providing us with some nice syntactic sugar to write cleaner code.

Let’s consider the following simple example:

public async Task<string> DownloadString(string url)
{
   var client = new HttpClient();
   var request = await client.GetAsync(url);
   var download = await request.Content.ReadAsStringAsync();
   return download;
}

The same code can be equivalent to what gets unfolded into under the hood:

public Task<String> DownloadString(string url) 
{ 
    var client = new HttpClient();
    var request = client.GetAsync(url); 
    var download = request.ContinueWith(http => 
        http.Result.Content.ReadAsStringAsync()); 
    return download.Unwrap(); 
}

As you can see, it pretty much creates a chain of tasks that needs to execute and gets queued up with the TaskScheduler. The above code is somewhat okay, but it is also beneficial to always use the language constructs provided by C# without doing everything manually!

The sequence of how this gets executed is as follows:

Calling client.GetAsync(url) will create a request behind the scenes by calling lower-level .NET libraries.
Some part of its underlying code may run synchronously until it delegates its work from the networking APIs to the OS
At this point, a Task gets created and gets bubbled up to the original caller of the asynchronous code. This is still an unfinished Task!
During this time, the caller may query the status of the Task
Once the network request is completed by the OS level, the response gets returned via an IO completion port and CLR gets notified by a CPU interrupt that the work is done
The response gets scheduled to be handled by the next available thread to unwrap the data
The remainder of your async method continues running synchronously

The key takeaway here is that there won’t be any dedicated threads to complete a Task. Which also means that your task continuation/completion isn’t guaranteed to run on the same thread that started it.

☠️ Common gotchas when using async and await

You would be tempted to parallelise everything that you see in your code when you start using async and await (well, I did 😁). But there are some pitfalls that you should avoid in your code.

Making methods void methods async void

This is the first mistake I did when I got my hands dirty with async-await! I had a void method which called an async method within itself. Since you can’t use await without making the calling method to be async, I made the caller method async void !

public async void GetResult() 
{
    await SomeAsyncMethod();
}

The problem here is, the caller of the GetResult method doesn’t have any kind of control over the result of SomeAsyncMethod() This also causes other side-effects such as lack of a call stack to debug if something goes wrong, application crashes if an exception occurs etc.

However, there are times that you can’t avoid this though. If you open up AsyncWpfApp in our code example you would find code like the one below:

private async void AsyncParallelBtn_Click(object sender, RoutedEventArgs e)
{
    ...
    var output = await RunAsyncParallelDemo.Start();
    ...
}

It’s not necessarily a problem if you find code like the one above since you can’t change the method signatures of the event handlers in WPF.

Nevertheless, it’s recommended to avoid using async void where possible.

Ignoring or Forgetting to ‘await’

It is an easy mistake to forget to await on an async task since the compiler would happily compile your code without complaining. In the below example we simply call an async function without awaiting on it.

public void Caller()
{
    Console.WriteLine("Before");

    // Deliberately forgetting to await
    DoSomeBackgroundWorkAsync();

    Console.WriteLine("After");
}

DoSomeBackgroundWorkAsync() method will return a Task (either a Task or a Task depending on the implementation) to the caller method without actually executing it.

Therefore, be mindful when you are dealing with async methods (especially with third party libraries) not to forget to use await

Blocking async tasks with .Result() and .Wait()

This is another common trap developers jump into (unknowingly 😋) when they need to run some async code in a synchronous method. Let’s consider the below example:

public void DoSomeWork()
{
    var result = DoAsyncWork().Result();
}

At a glance, it may look pretty convenient to use the .Result() method. However, this could cause serious deadlock problems. In order to avoid this, you should usually make your caller method async. This could be lot of work as it creates a cascading effect on your changes. But that’s usually preferable.

More ways to write non-blocking code as clearly depicted in MSDN:

It’s recommended to make the caller async where you would end up making a lot of cascading changes to your codebase although it could be painful if you are working on a legacy project.

Conclusion

I hope you enjoyed this article as much as I did writing it. The bottom line is that Task is almost always the best option; it provides a much more powerful API and avoids wasting OS threads.

Ready for more stuff? Head over to my Github repo to work through some sample code.

https://github.com/sahansera/c-sharp-tasks

Suggestions or Found a Bug?

If you think there’s a misunderstanding in any of the things I explained, please feel free to comment down below :) Also, if you found a bug or have any suggestions, please open a pull request in my Github repo. Cheers!

References

How to Pre-Configure Services Before Registering with ASP.NET Core's Built-in Service Container

Sun, 01 Dec 2019 00:00:00 GMT

In ASP.NET, if you are using the default DI service container, registering services is done in the ConfigureServices(IServiceCollection) method in your Startup.cs class.

In some special situations, you may want to take control of the instantiation of some service. The idea here is to inject any other dependencies that a service would require before registering with your DI container; ASP.NET Core’s default service container in our example.

Here is an example:

services.AddScoped<IComputerVisionClient, ComputerVisionClient>((serviceProvider) =>
{
    ConfigService configService = serviceProvider.GetService<ConfigService>();
    return new ComputerVisionClient(
        new ApiKeyServiceClientCredentials(configService.SubscriptionKey),
        new System.Net.Http.DelegatingHandler[] { })
    {
        Endpoint = configService.Endpoint
    };
});

The above code sits in your ConfigureServices(IServiceCollection services) method when registering your service.

In simple terms, it instantiates ComputerVisionClient by using another component. We can obtain a reference to another service ConfigService which is used to keep our configurations in.

Now you should be able to inject that service in your controllers/services 😊

sahansera.dev RSS Feed

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

The Problem

What It Does

Getting Started

Configuration

Running on a Schedule with GitHub Actions

Custom Prompts

Why Open Source?

What’s Next

Links

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

What is ARC and Why Should You Care?

Prerequisites

1️⃣ Pre-Setup: Quick Checks

2️⃣ Install ARC Controller

3️⃣ Deploy a Runner Scale Set

4️⃣ Check That It’s Working

5️⃣ Testing It Out

5️⃣ Monitoring (Bonus: Grafana)

6️⃣ Useful Commands for Day-to-Day Ops

Troubleshooting Tips

That’s It! You’re Running GitHub Actions on Your Own Cluster

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Why Move From Raspberry Pi to x86?

Network Setup: Keep It Simple (and Reliable)

Getting Ubuntu Server Set Up on Each Node

Installing k3s: Lightweight, Powerful, Familiar

On the Control Plane Node (“master”)

On Each Worker Node (“worker-1”, “worker-2”, etc.)

Troubleshooting and Gotchas

What’s Next?

Lessons Learned

Final Thoughts

How Does the Python Virtual Environment Work?

The Problem: Dependency Hell

The Solution: Virtual Environments

How Does It Work?

Command Resolution

Why Is This Powerful?

Advanced Use Cases

Under the Hood: What’s Really Happening?

Debugging Tips

Conclusion

References

Deep Dive - How Chunked Transfer Encoding Works

🤔 What is Chunked Transfer Encoding?

💻 Hands-on Example

🛠️ Step-by-Step Breakdown of Chunking

🙋 What about HTTP/2 and HTTP/3?

Key Takeaways

Further Reading

Upgrading sahansera.dev to Gatsby 5

How I Upgraded My Blog to Gatsby V5

The Upgrade Journey

Dependency Overhauls

Plugin Upgrades

Styling Challenges

Enabling Nested CSS with styled-jsx

PostCSS Configuration Adjustments

Lessons Learned

Final Thoughts

References

Streaming APIs with FastAPI and Next.js - Part 2

🛠️ Building the Streaming Backend with FastAPI

🚀 Simulating a Real-Time Log Stream

🧠 How It Works

1. Synchronous Generator

2. StreamingResponse

⚙️ Running the Server

💡 Tips for Production

✅ Keep the Stream Alive

🧹 Handle Disconnects Gracefully

🔐 Secure Your Stream

🧪 Testing with Curl

🔗 Hooking it up with the Frontend

🧠 Recap

🧱 Next Steps

🛠️ Useful Links

Streaming APIs with FastAPI and Next.js - Part 1