Notes on theAEYDR - Architectural Decision Record

Inverting IAM Permissions for System-Centric Ownership

Fri, 09 Jan 2026 00:00:00 +0000

Today I Explained

One of the most common ways organizations model IAM and role-based access control is around teams. You start with the assumption that “this team owns this thing,” so it makes sense to give them permissions scoped to it. If I own Service A, then of course I should be the one who can restart it, update the code, change its database configuration, and so on. That logic extends out to build systems, artifact stores, CI pipelines. It’s a reasonable-enough model.

Stale Code and the Cost of Forgetting

Tue, 06 Jan 2026 00:00:00 +0000

Today I Explained

Stale code, or code rot, which can be generally described as a whole category of long-lingering issues that show up with old code sitting around in the codebase. For example, there was a time when our organization was really into Ansible. That led us to spin up a bunch of reusable repositories full of Ansible roles that were basically little packages you could plug into your playbooks. These were for installing stuff like Java runtimes, jq, and all sorts of other tiny setup utilities. We went really minimal and micro with it.

Letting the Repo Explain Itself

Sat, 03 Jan 2026 00:00:00 +0000

Today I Explored

Code archaeology, at least the way I’m playing with it, is the idea that a repository can be wired so cleanly that the repo itself starts narrating what’s inside. If the metadata is rich enough with sensible folder names and the odd tag sprinkled in, then tools can roam through and surface the big picture without me hand-stitching an overview every time.

Deliberate Slack Bots as Cultural Infrastructure

Fri, 02 Jan 2026 00:00:00 +0000

Today I Explained

One of the proposals to build more deliberate Slack bots as part of our onboarding flow and general culture. It came up while brainstorming ways we could make culture more of a natural part of how we work, instead of something that only exists in a stale wiki page somewhere. The idea is that these bots would enable small processes that reflect our culture.

Using AWS IoT Certificates for Safer External Access

Mon, 29 Dec 2025 00:00:00 +0000

Today I Explored

I looked into a pattern of using AWS IoT certificates as a way of granting permissions to automated processes and services that live outside AWS. This ties into the broader conversation around how we typically provision credentials to CI systems either by stuffing IAM access keys into some secrets manager or by piping in temporary credentials from another service that rotates them regularly. Those are the common paths. But I’ve been digging into this alternative: using AWS IoT certificates to grant access via IAM roles.

Grafana Dashboards via S3 and CSI

Sun, 28 Dec 2025 00:00:00 +0000

Today I Explained

Grafana supports defining dashboards as config JSON files. You can bundle these directly into the Grafana container image, or load them at runtime through a config map. You can even go further and use a volume mount that stays up to date, with Grafana periodically reloading itself to pick up any new changes. The idea I’ve been proposing is to take that and pair it with the AWS S3 CSI driver. We’d publish our dashboards into an S3 bucket as part of our CI workflow, and then use the CSI driver to mount that bucket into the Grafana pod. Grafana would then keep syncing from that S3 bucket and refreshing dashboards as needed.

A Simple ALB and S3 Setup for Network Reachability Testing

Thu, 25 Dec 2025 00:00:00 +0000

Today I Explained

How you can build a really simple ALB setup using just S3 and DNS to serve a JSON endpoint that you can hit to test basic network reachability. This is one of those things where, whether you’re trying to validate something programmatically or just open a browser and check, it’s useful to have a quick way to confirm, yep, I can reach this endpoint from wherever I’m running. And doing it with just an ALB and an S3 bucket is pretty lightweight.

Kubernetes Operators, Ownership and Blue-Green Infrastructure Implications

Wed, 24 Dec 2025 00:00:00 +0000

Today I Explained

How one of the ways Kubernetes operators persist their state is a little different from something like Terraform. There’s no centralized state file that they’re managing. Instead, they tend to rely on annotations, tags, and other attached metadata to keep track of the resources they’re responsible for. A good example of this is external-dns, which can manage Route 53 records in AWS (or DNS records in other clouds). It uses DNS TXT records to embed identifying metadata so that it knows which records it owns. That way, it can safely find, update, or delete them. Same thing happens with the AWS Load Balancer Controller. It tags the load balancers to claim ownership, so it knows what it can safely manage or remove.

Exploring API Procedures as a Pattern in REST APIs

Mon, 22 Dec 2025 00:00:00 +0000

Today I Explored

Using procedures in REST APIs, similar to something you’d see within a database’s stored procedures (or GraphQL on the client). It’s something that’s come up frequently as I’ve been working with APIs that require me to do round-trips for retrieving metadata about resources like tags, annotations, or labels. This was partly inspired by my experience with GraphQL on the client side, and stored procedures/views on the database side.

Organizing Private DNS in AWS

Sat, 20 Dec 2025 00:00:00 +0000

Today I Explained

There are a few different ways you can manage private DNS inside AWS, and one of the first things I think is important is making sure you actually own the domain you’re planning to use. Not because you want to make it public or publish anything externally, but just so you have full control over it. You want to be able to define how it’s used, make sure it’s not getting misused for things like spoofing or weird email routing or whatever kind of trickery someone might try.

Why We Mirror Artifacts Across AWS Regions

Fri, 19 Dec 2025 00:00:00 +0000

Today I Explained

In AWS, you’ll run into plenty of situations where artifacts S3 buckets, CodeArtifact, or ECR are mirrored across multiple regions. Maybe you’re primarily deploying out of ca-central-1, but you’ve also mirrored those same artifacts into eu-central-1 to support some EU workloads. Part of that is just convenience: having a local copy in-region cuts down on latency and dependency complexity.

Why point-and-click access control slows everything down

Mon, 15 Dec 2025 00:00:00 +0000

Today I Explained

Some thoughts on point-and-click operations, even when it’s just in isolated groups or limited scopes. I feel like the kind of bottlenecks this creates negatively impact the organization in ways beyond just speed.

The Well-Architected Repository For Automated Changes

Sat, 13 Dec 2025 00:00:00 +0000

Today I Explained

The “Well-Architected Repository” model, in the context of automated code changes. At different points, we’ve talked about this idea of well-architected repositories, borrowing from AWS’s well-architected infrastructure concepts. The core idea is having a consistent set of files or structures in place across repositories to make it easier when someone’s jumping between different repositories or even different parts of a monorepo. It started off as just a guideline, more for when people were doing manual changes to make the developer experience consistent. It was just a helpful way to organize things, define templates, and make onboarding smoother.

Mirroring GitHub Actions with Terraform

Fri, 12 Dec 2025 00:00:00 +0000

Today I Explained

A GitHub Actions cache that’s managed and enabled through Terraform. One of the core issues with GitHub Actions is that you’re depending on external repositories to be available for your workflows to succeed. If those go away or change, your reliability goes with them. Importantly, though, GitHub Actions often uses floating versions, which means anything like v4 or v4.3 is actually referring to a specific patch release (v4.3.7). These all point to a specific commit in GitHub.

Using a Standby OU to Dodge SCP Bootstrap Traps

Tue, 09 Dec 2025 00:00:00 +0000

Today I Explained

When juggling AWS Organizations, the moment Service Control Policies become truly useful is also when they start breaking the bootstrap workflow. I absolutely want the guardrails of no deleting VPCs, no tinkering with an account alias, and keep the org-access role cemented in place. All of these are great because those moves protect the critical plumbing that keeps everything else upright. The snag is that slapping those restrictions at the root level means the very pipeline that spins an account up (or tears one down) can’t finish its own work.

API-derived status checks for GitHub repos via Terraform

Sat, 06 Dec 2025 00:00:00 +0000

Today I Explained

How an API-derived feedback loop can help manage GitHub repository status checks using Terraform. One of the things GitHub supports is managing your repositories with Terraform, which lets you do things like set required status checks directly in the repo. The tricky part, though, is that there’s this weird chicken-and-egg problem. If you enable a required status check before that check actually exists in the repository, then every pull request just gets blocked.

The Language of Slack Emojis

Tue, 02 Dec 2025 00:00:00 +0000

Today I Explained

We use Slack emojis a lot. Both for being funny and sometimes for more functional reasons. You’ll see the usual fun stuff like :sadcat: or :thisisfine: floating around, which are mostly just for tone or jokes. But we’ve also got emojis that actually serve a functional purpose. Some are used in automated messages or show up in workflows. Others are deliberately chosen to act as visual cues in specific channels to provide context, help with scanning, or just generally keep things organized. It’s about vibes.

Useful GitHub Organization Patterns

Sun, 30 Nov 2025 00:00:00 +0000

Today I Explained

One of the patterns I sometimes use for managing GitHub organizations which is splitting them up into a couple of distinct types.

Centralizing Slack Notifications from Alerts with EventBridge

Fri, 28 Nov 2025 00:00:00 +0000

Today I Explained

Using a centralized EventBridge model, you can make it way easier for services to emit Slack messages. Not by pushing all the logic into the services themselves, but by making their only real responsibility just sending something off to SNS, SQS, or EventBridge inside the account. In some cases, you might not even need to have your code explicitly do that part. You can model it so that the event is derived from metric changes, or a side effect of some other event in the system.

Publishing Cross-Account Deployment Metrics with EventBridge

Wed, 26 Nov 2025 00:00:00 +0000

Today I Explained

When you’re using a centralized EventBridge model, it’s actually pretty straightforward to pass around custom metrics or event signals across accounts. This is especially useful when you’re dealing with things that don’t really fall neatly into traditional application monitoring. These aren’t your standard CPU usage or request latency numbers. They’re usually events that reflect some infrastructural or environmental state. Stuff that’s meaningful to the broader system but kind of outside the app’s own little bubble.

Treating Secrets as Disposable

Tue, 25 Nov 2025 00:00:00 +0000

Today I Explained

How I’ve been hung up on the idea that secrets are fundamentally disposable, just like binary artifacts. We ship binaries, container layers, Lambda zips, VM images, as long as the checksum matches, the code runs the same whether the bits came from us-east-1 or a mirror on the far side of the planet. Infrastructure costs, latency, or cache warm-up still exist, but because the bytes are identical, there’s zero functional change to the code.

Musings on Hirings and the User Experience

Mon, 24 Nov 2025 00:00:00 +0000

Today I Explained

Some of my thoughts on hiring practices in software organizations. I’ve got opinions on basically every part of it like most people do, but the one I wanted to focus on is the general flow someone goes through when they’re trying to join an organization. It’s usually something like:

Injecting Chaos to Expose Rotten Foundations

Fri, 21 Nov 2025 00:00:00 +0000

Today I Explored

I keep circling around the idea of chaos engineering, but not the technical-style that’s common but more of an organizational version that stresses our documentation and institutional knowledge. The spark was simple enough: our runbooks look authoritative until someone follows them to the letter and, boom, nothing works. The fix gets pasted into Slack, everybody nods, and the documentation itself stays stale. That tiny cycle made me think about the ways the organization operates on shared memory rather than written truth.

What's in the !artifacts Directory?

Tue, 18 Nov 2025 00:00:00 +0000

Today I Explained

What the !artifacts directory is, why it shows up in our repo after a build, and what those common files like manifest, status, and build are doing in there. In our setup, the build process has this packaging step that’s responsible for gathering up everything we’re creating and sticks it all into one place. That place might have a few subfolders, but the idea is that everything under !artifacts represents the complete output of the build, fully baked and self-contained. The whole point is that you can take this directory and enumerate everything in it for publishing to artifact stores.

Why One Might Proxy Secrets Manager Names Through SSM

Sat, 15 Nov 2025 00:00:00 +0000

Today I Explained

Sometimes it’s useful to proxy the name of an AWS Secrets Manager secret using an SSM parameter. The main reason for this is how CloudFormation handles references. It has pretty decent support for importing values from SSM parameters, which makes SSM a pretty effective way to pull in something from your infrastructure since it allows other infrastructure-as-code to be responsible for writing the SSM parameter in the first place, ensuring it’s current.

Why We Separate InternalOps from CorpOps

Fri, 14 Nov 2025 00:00:00 +0000

Today I Explained

Why it makes sense to distinguish between InternalOps and CorpOps. CorpOps is just shorthand for corporate operations, which is basically the stuff managed by IT. They’re the ones responsible for systems like Okta, which in turn governs access to things like GitHub, payroll, expense reports, all of our internal tools. If there’s a system that handles operations for the company as a whole, that’s CorpOps territory, and it usually routes through Okta.

Reasonable Early Security Controls in AWS

Tue, 11 Nov 2025 00:00:00 +0000

Today I Explored

When you’re spinning up an AWS account for a new project, you usually don’t need the full buffet of security tooling AWS offers. But some controls, if added early, can save you a ton of pain later. Not because they solve everything upfront but because they prevent you from having to retrofit controls after you’ve already scaled. It’s not about locking things down on day one. It’s more like: “You’re probably going to need this eventually,” and it’s easier to set it up now than unwind mistakes later. Think of it like YAGNI, but with security guardrails.

Using GUIDs for Terraform State file paths

Sat, 08 Nov 2025 00:00:00 +0000

Today I Explained

Why the organisation stores every Terraform state file under a generated GUID instead of a descriptive path.

Remote backends such as S3 or Azure Blob normally let you pick any object key. For example, networking/prod/vpc.tfstate. At first glance, human-readable keys seem convenient. But in a GitOps repository, it’s not ideal to have a fixed repository structure, as any shifting of stack locations will result in drift between where the state is stored and where it lives in the repository.

Minimal state machines within Terraform

Wed, 05 Nov 2025 00:00:00 +0000

Today I Explained

A less-than-ideal pattern for creating a state machine within Terraform. For most development cases, Terraform is only triggered when inputs change, such as a new module version, a new artifact version, or an explicit decision to update the variables in the Terraform stack. You aren’t limited to that, though.

Terraform Secrets Retrieval with HCP Vault or SaaS Providers

Sun, 02 Nov 2025 00:00:00 +0000

Today I Explained

Why one might adopt a lightweight but suboptimal infrastructure code pattern for retrieving secrets from a password manager (e.g. LastPass, Keeper, 1Password, etc.) and publishing those secrets into something like AWS SecretsManager. In many organisations, there’s a password manager-type solution. These are typically intended for use by people which will access them via a browser. This is also used to pass secrets between people within the organisation, such as when the owner of a service gives access tokens to a consumer of the service within the organisation.

Challenges with Nested Data-Rich Structures in Terraform

Thu, 30 Oct 2025 00:00:00 +0000

Today I Explained

When building pipeline-style infrastructure, you sometimes need to describe an entire state machine or multi-step workflow as a single, deeply nested object, which is much richer than the flat variables Terraform typically expects. These definitions capture everything from conditional branches to user-interaction steps, and the shape often resembles any NodeRED, AWS Step Functions, or GitHub Actions YAML.

Cloud agnostic patterns using build artifact metadata

Fri, 24 Oct 2025 00:00:00 +0000

Today I Explained

Designing infrastructure as code to be cloud agnostic is often a loss-based effort that sees us trying to encode conditional logic, favour a single cloud environment, or lose out on cloud provider-specific patterns that are better suited to our use case.

Deletion, Terraform and Infrastructure as Code Lifecycles

Sun, 19 Oct 2025 00:00:00 +0000

Today I Explained

Infrastructure as code written in Terraform or CloudFormation often ends up not handling deletion scenarios gracefully. For example, an AWS S3 bucket requires the force_destroy flag to delete a bucket that still contains objects; otherwise, deletion fails. In CloudFormation, a stack with an S3 bucket containing objects cannot be deleted.

Building Infrastructure with Deletion in Mind

Mon, 13 Oct 2025 00:00:00 +0000

Today I Explained

Designing infrastructure with deletion in mind is an often ignored and surprisingly important consideration when it comes to infrastructure as code. When building systems, it can be easy to focus solely on how to create and change components without considering the implications of tearing them down.

AWS CodePipeline for airgapped-ish Terraform deployments

Wed, 08 Oct 2025 00:00:00 +0000

Today I Explained

That you can use AWS CodePipeline as a Terraform executor to create pseudo-airgapped deployments of Terraform resources. The basic idea is to execute Terraform entirely inside the VPC by packaging every required provider version and module into a single archive bundle. With the CodeBuild jobs provisioned in the VPC, this locks execution into the VPC, giving you controls to restrict access to internet resources.

A Discouragement for Account-ID Conditionals in Terraform

Sun, 05 Oct 2025 00:00:00 +0000

Today I Explained

You should avoid, whenever possible, using the region or the account ID as a conditional within Terraform or any infrastructure code. Relying on identifiers such as “if the ID equals X, enable this” or “if the region is Y, create that bucket” seems convenient until one wishes to leverage the code of your Terraform.

Atlantis configuration within Terraform Stacks for distributed configuration

Mon, 29 Sep 2025 00:00:00 +0000

Today I Explained

Why the organisation splits Atlantis project definitions into each Terraform stack directory and then regenerates the top-level atlantis.yaml using scripts.

Why We Use Structured Slack Messages for Deployment Events

Wed, 24 Sep 2025 00:00:00 +0000

Today I Explained

Since different workloads carry varying levels of risk, we structure Slack messages for deployment events to match the needs of different audiences. For example, infrastructure-as-code modules that manage secrets pose relatively low risk and can be updated frequently. It’s straightforward to validate that secrets are intact, so we can release updates as needed. Deployment notifications are therefore rather to the point:

SQL Bundles for Infrastructure as Code Databases

Fri, 19 Sep 2025 00:00:00 +0000

Today I Explored

When using the C# EntityFramework for provisioning databases, this relies on a bootstrap step to spin up the database, handled by the application code. While exploring this workflow, the idea emerged that rather than letting application code create tables and views on first run, an AWS RDS instance receives its entire schema from the infrastructure layer. Terraform applies a bundled SQL artifact that declares tables, views, and stored procedures upfront. The database is ready before a single line of application code executes.

Scaling Gomplate by avoiding logic in templates with module-driven patterns

Sun, 14 Sep 2025 00:00:00 +0000

Today I Explained

When you mix logic with presentation, you often end up with templates that are difficult to read and maintain. For example, consider a scenario where you need to perform set operations on a list of items. If you were to include this logic directly in your Gomplate template, your code might look something like this:

Gomplate, escape hatches, and the limits of tools

Sun, 07 Sep 2025 00:00:00 +0000

Today I Explained

One of the ways that you can extend the viability of a scripting (or generic) solution is by going beyond the limit of a tool. When you work with languages like PowerShell, Bash, or even CI pipelines written in GitHub Actions YAML, you eventually reach a point where the tool’s simplicity no longer meets your needs. Tasks that involve richer data management or complex logic are better suited to a full programming language. When you reach that boundary, there are techniques that let you extend the tool’s usefulness a bit further without requiring you to perform the migration.

Structuring Documentation Before Writing It

Wed, 03 Sep 2025 00:00:00 +0000

Today I Explained

Although the structure of documentation can influence its accessibility and readability, when you have nothing to organise, it isn’t worth focusing on frameworks and layout. Discussions over folder hierarchies, documentation locations, and layout frameworks can yield exquisitely organised documentation, but they won’t reveal the gaps in your documentation. Structure alone produces nothing. Only writing yields documentation.

Bazel Toolchains as Files Outside of Bazel Code

Fri, 29 Aug 2025 00:00:00 +0000

Today I Explained

In Bazel, when developing rules that wrap an existing CLI tool (for example, jq, gator, or terraform), you can declare the toolchain alongside the rules. Typically, you use http_archive (or a similar repository rule) to fetch the tool binaries, then register a toolchain that makes them available to Bazel:

Bazel Toolchains as Interface Instead of Embedded in Rules

Fri, 22 Aug 2025 00:00:00 +0000

Today I Explained

Bazel, RulesOCI, and Docker Load for Running Container Images

Sun, 17 Aug 2025 00:00:00 +0000

Today I Explained

With the deprecation of rules_docker, Bazel no longer provides a maintained container run-and-extract pattern. Its successor, rules_oci, treats containers primarily as an archive format rather than an environment in which you can execute builds. The old container_run_and_extract rule looked something like this:

Leveraging bazel query for generic conditional build workflows

Sun, 10 Aug 2025 00:00:00 +0000

Today I Explained

Bazel supports the query subcommand, allowing inspection of the build graph from build scripts. Both single targets and complex patterns can be matched, enabling scripts to make data-driven decisions rather than having strict requirements for build actions to be defined.

Bazel workspace directories for runtime emitted files

Mon, 04 Aug 2025 00:00:00 +0000

Today I Explained

This was written before Bazel 5.0; newer versions (or rules) may better support this pattern.

When you run a Bazel target with bazel run, you can use the BUILD_WORKSPACE_DIRECTORY environment variable to write outputs to a predictable location inside your workspace. For example:

An idea on Bazel & Variant-agnostic configurations

Wed, 30 Jul 2025 00:00:00 +0000

Today I Explored

The idea came from reflecting on how Terraform handles modular reuse through its module structure. In Terraform, defining a module is straightforward, and looping through different configurations is seamless. It got me wondering if a similar approach was achievable in Bazel, particularly when managing multi-platform or multi-architecture builds.

Reusable Shell-Based Workflows with `bazel run`

Sun, 27 Jul 2025 00:00:00 +0000

Today I Explained

Bazel can actually be leveraged to make reusable build scripts available that don’t necessarily integrate within Bazel itself. By defining custom rules, it becomes possible to wrap standalone shell scripts and run them with bazel run.

Using Bazel as an Organisation's Non-Bazel Build System

Mon, 21 Jul 2025 00:00:00 +0000

Today I Explored

One of the reasons why the organisation chose Bazel is, surprisingly, not actually related to caching, remote execution, or speed. The compelling draw was Bazel’s knack for acting as a library of build actions that can be invoked and parameterised just like functions in code. In practice, that means a shell script, static-analysis tool, or container build can live within a library rather than the source it touches.

Dedicated Node Groups for Shared Services for Cost Optimisation

Wed, 16 Jul 2025 00:00:00 +0000

Today I Explored

Dedicated Node Groups for Shared Services within Node Group-based Kubernetes Clusters in AWS could be a practical way to tame operating costs. When an Amazon EKS cluster relies on a single, catch-all node group, most workloads end up on AMD64 on-demand instances due to limited ARM support. That default blocks the cluster from taking advantage of cheaper capacity, particularly ARM64 Graviton nodes on Spot.

Chasing Cost Savings with Spot, Graviton, and Resource Reservations

Sat, 12 Jul 2025 00:00:00 +0000

Today I Explored

When AWS announced the release of Graviton, the roughly 30% compute discount proved extremely appealing for many organisations, triggering a rush toward adoption that often felt myopic. Given how essential compute resources are within AWS, that enthusiasm made sense on paper. Yet I also noticed how many of the pre-existing invisible x86 assumptions continued on. The single-architecture container images, schedulers defaulting to AMD64, metadata not carrying over from artifacts to infrastructure as code. These were quietly carried over as the organisation transitioned towards Graviton machines.

Attribute-Aware Deployments with AWS Account Tags

Sun, 06 Jul 2025 00:00:00 +0000

Today I Explained

Attribute-aware deployment patterns are one of the reasons to take the effort to add annotations and metadata to AWS accounts. Tagging each account at the Organisations level means the information lives close to the infrastructure it describes, rather than in a scattered trail of wikis and runbooks.

Terraform-Integrated AWS Workflows for Terraform (CodePipeline, Step Functions)

Mon, 30 Jun 2025 00:00:00 +0000

Today I Explained

You can leverage built-in technologies of AWS like CodePipeline or Step Functions to create deployment workflows for Terraform. By combining these technologies, you can achieve a self-contained deployment pattern within AWS that works with private networking, air-gapped deployments, and strictly controlled artifacts.

Testing Terraform with Conftest, OPA, and Bazel

Thu, 26 Jun 2025 00:00:00 +0000

Today I Explained

Conftest is a small command-line wrapper around Open Policy Agent that lets you write policies that test configuration files and run them as assertions. Because Terraform state is declared in HCL, you can point Conftest at a *.tf file and ask, for example, whether every aws_s3_bucket block turns on server-side encryption. In practice, that feels like writing unit tests for infrastructure.

Quick AWS CLI Multi-Region (and Multi-Account) Queries

Wed, 25 Jun 2025 00:00:00 +0000

Today I Explained

Bash supports a neat pattern for quickly running AWS CLI queries across multiple regions using argument expansion. When working with AWS services, it’s common to gather information from different regions or accounts. Instead of manually switching regions, you can automate this process with a Bash script.

Using ASDF for Operations, and Devcontainers for Build

Sat, 21 Jun 2025 00:00:00 +0000

Today I Explained

Why the organisation would leverage both asdf and Devcontainers at the same time. To review, a Devcontainer is a development environment running within a container image that allows you to clone a repository and start coding within an environment suited to the repository.

Idiomatic code and an SQL-compatible domain model for codegen

Mon, 16 Jun 2025 00:00:00 +0000

Today I Explored

Working with sqlc, the Go tool that converts handwritten SQL into type-safe code, sparked a new line of thinking: could the process be entirely codegen? I wondered whether a reusable domain model, defined once in a neutral format, might generate every SQL artifacts, the matching domain classes layer and the database layer. The idea dovetailed with a long-standing fascination I have with Postgres stored functions (& procedures): let the model emit the functions, views, and idiomatic code so that storage details stay fluid while the interfaces remain stable.

Internal Keys, User-Facing URNs and exposing the database to users

Sat, 14 Jun 2025 00:00:00 +0000

Today I Explored

While working on backend services, I kept seeing the same pattern: database primary keys bubbled up the entire stack until they surfaced in bookmarkable URLs, shell scripts, Terraform configs, and even the user interface. Because those numeric IDs became part of application behaviours, any schema migration switching the primary key poses the risk of disrupting application behaviour even when the IDs are generally treated as existing just for database relational purposes.

Unified Toolchains with an Ad-Hoc `third_party/toolchains`

Sat, 14 Jun 2025 00:00:00 +0000

Today I Explained

Repositories have a number of ways in which they can support development environment setup. One of the more common recent additions is DevContainers, but many other systems exist, like language-specific install scripts, Nix, or just the tried-and-true Bash script. These systems will lean on version files such as .python-version or .bazel-version to pin runtimes.

Organisation-Wide Toolchain Management with ASDF

Sun, 08 Jun 2025 00:00:00 +0000

Today I Explained

You’ll notice that many repositories contain a .tool-versions file at the root. Inside this file are space-delimited toolchain names and their versions, like so:

Using ASDF in CI to Pin Tool Versions

Thu, 05 Jun 2025 00:00:00 +0000

Today I Explained

You don’t have to pigeon-hole ASDF into a desktop nicety for juggling tool versions on a laptop. The basic idea behind the tool is simple: it installs every binary into a predictable directory and chooses which one runs based on a version you declare. Any environment that can execute those binaries and tolerate a quick ASDF bootstrap can benefit from the same mechanism, whether that’s a CI runner or within a production server. Drop the familiar .tool-versions file into the pipeline, run a short setup script, and the workflow comes up with exactly the versions your local shell expects.

CloudFormation, Applications & Separation of Concerns

Mon, 15 Apr 2024 00:00:00 +0000

Today I Explained

CloudFormation supports the resource type AWS::CloudFormation::Stack, which allows a CloudFormation stack to provision another CloudFormation stack. This is known as a nested stack. These nested stacks allow re-using an already defined CloudFormation stack when deploying a stack, or just as a mechanism for grouping.

Quickly initializing tools with ASDF

Sat, 19 Aug 2023 00:00:00 +0000

Today I Explained

If you happen to be leveraging a single plugin strategy for asdf, as a way of consistently managing toolchains, a quick helper that you can make use of to quickly set up plugins is running the command:

A strict execution pipeline with Terraform

Fri, 18 Aug 2023 00:00:00 +0000

Today I Explained

Terraform support the concept of the “plan”, a preview of the speculative actions that Terraform will take to modify your infrastructure. This is used in most Terraform implementations, performing a plan that will be reviewed before being executed by an apply.

ASDF, Toolchains and a Single Plugin as Package Database

Fri, 18 Aug 2023 00:00:00 +0000

Today I Explained

asdf is a CLI tool that manages multiple language-runtime versions on a per-project basis. It does this through per-tool plugins, which are Bash scripts that install the runtimes onto a developer’s workstation. When asdf is installed, this works by first installing the plugin from a Git URL:

Predictable naming of resources in CloudFormation

Thu, 17 Aug 2023 00:00:00 +0000

Today I Explained

Some AWS Services have naming uniqueness requirements, which are typically within an AWS Account Region, or within an AWS Account. Which you’ll typically find out about when encountering an error message that sounds similar to:

Packaged bundles of Infrastructure through Terraform

Wed, 16 Aug 2023 00:00:00 +0000

Today I Explained

Terraform configuration files have three different ways in which they are able to impose some requirements about dependencies they have within the terraform ecosystem. The first is the lockfile, which is used to pin the specific versions of the Terraform Providers to use when executing Terraform. This can be fairly important for both performance and security, as it allows more providers to make use of the same precached version of a given Terraform provider.

Terraform, AWS & Sourcing Secrets

Mon, 14 Aug 2023 00:00:00 +0000

Today I Explained

It isn’t always the case that you are solely working with the Terraform AWS provider, and in the cases in which you are working with providers for external services, you don’t necessarily have access to the same authentication pattern as IAM. Often you connect to these external services using a URL and a token credential. For working with these kind of terraform modules, the token credential isn’t going to be readily available on the workstation. Sometimes a helper may exists within /usr/secrets/bin to dynamically pull the secret from an external provider, but this isn’t always the case.

Tags in Terraform, overriding defaults and module-level tags

Sun, 13 Aug 2023 00:00:00 +0000

Today I Explained

The Terraform AWS Provider supports a field known as default_tags which can significantly cutdown on the amount of copying & pasting when it comes to apply tags on all resources within a deployment. This is especially useful for Terraform modules, as they don’t require passing along tags to each sub-module contained within. This makes the overhead of setting up metadata within a Terraform module more practical, allowing for things like licensing, documentation URL or compliance-driven tags to be defined inside of the module itself.

Postgres Delete/Restore by Table Properties

Sat, 12 Aug 2023 00:00:00 +0000

Today I Explained

You’ve likely encountered tables where the records within the table are treated as immutable, creating entirely new entries when a change is made. These are versioned tables, and are one way to provide a versioning scheme within a database like Postgres. Typically this is implemented within the table itself, by having a table in which all versions exist, using a VersionID (or similar ID) to distinguish between the versions:

Soft Deletes as the responsibility of the Database

Fri, 11 Aug 2023 00:00:00 +0000

Today I Explained

You’ve likely heard of the concept of “soft delete” within tables. These is where a flag exists on a row called deleted (or similarly named), that can be toggled to indicate whether an entry has been “deleted”. This doesn’t remove the row from the table, and keeps it around allowing a user to restore it (set the flag to false) at a future date. Should it not be toggled back, after some time it will be removed from the database, based on a deleted-at and expiration policy.

Dynamic templating facilitated through gomplate

Thu, 10 Aug 2023 00:00:00 +0000

Today I Explained

gomplate is a template renderer CLI that can fit nicely within the niche of lightweight templating solution, as it’s support for datasources (like JSON & YAML) and conditional execution pushes it a step up above hand-crafting something in PowerShell or Bash. One of the frustrations that arises with this method of templating is the difficulty with debugging the execution of the conditional logic.

Bazel, Container Push & Infrastructure in Builds

Wed, 09 Aug 2023 00:00:00 +0000

Today I Explained

A pattern that I generally discouraged is the encoding of artifact stores within build systems, such as having a container registry hardcoded into the docker push (or equivalent actions) in the build system. This isn’t to say that the container registry can’t be specified within the continuous integration pipeline, but that it should be avoided within the build system.

Cost Categories, Infrastructure and Tags

Tue, 08 Aug 2023 00:00:00 +0000

Today I Explained

Tags are an extremely useful mechanism within AWS for managing both permissions, categorization, associations & relationships within infrastructure. As IAM supports policies that permit or restrict permissions based on the presence of a tag, it can be tempting to make use of it. However, this can be a risky approach as it runs the risk of creating IAM policies that are difficult to make attestations about.

Pseudo Content Addressable Storage within S3

Mon, 07 Aug 2023 00:00:00 +0000

Today I Explained

While reviewing an AWS S3 bucket you may have comes across files named similar to 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7. This being a file, sometimes a zip archive, with a named composed of what appears to be a random set of numbers and letters. These files aren’t actually published into S3 with a random name, but rather are published with their checksum as their name.

Denying point & click EC2s within AWS Accounts

Sun, 06 Aug 2023 00:00:00 +0000

Today I Explained

When spinning up an EC2, the AWS Web Console offers an interface wizard, known as Launch Wizard. This wizard offers a guided workflow for configuring & starting an EC2 within an AWS Region. Yet sometimes you may be prevented or strongly discouraged from making use of this wizard when wishing to provision an EC2. Why?

Prebuilt IAM policies, and least privilege in AWS

Sat, 05 Aug 2023 00:00:00 +0000

Today I Explained

You may find repositories for which the test pipeline failed when you had recently made a modification to an IAM Policy or Role, which saw the addition of an IAM policy. Specifically, the managed access policies which don’t grant least privilege, but assist with covering common use cases. These errors sometimes look like:

AWS Accounts and the hidden costs of compliance

Fri, 04 Aug 2023 00:00:00 +0000

Today I Explained

Even with no deployed applications, an AWS Account ~~will~~ may still incur costs.

The above statement is a bit of a special case. As organization grows they adopt many of AWS’s built-in services for meeting compliance expectations, such as CloudTrail, GuardDuty, Inspector & AWS Config. They’ll implement things like IAM Roles in accounts that permit tooling to scan infrastructure to identify potential malicious activity within the accounts.

Prefix Lists for Common Third Party Services

Thu, 03 Aug 2023 00:00:00 +0000

Today I Explained

When developing a service within Amazon Web Services (AWS), it isn’t always intended to be consumed by any machine in the world wide web. It may only be responsible for communicating with a single third party service, or an internal service hosted in a different part of the company’s cloud. For these kind of services, they often have a declared set of CIDR ranges (IPs) that you can expect to receive & interact with for network traffic.

Message prompts when connecting to EC2s

Wed, 02 Aug 2023 00:00:00 +0000

Today I Explained

When you connect to instances within the organization, you’ll notice that it sometimes includes a prompt that includes the company logo & an advisory message, this looks something like the following:

Why are Lambda zip names sometimes random strings in CloudFormation?

Tue, 01 Aug 2023 00:00:00 +0000

Today I Explained

Deployment keys for unique deployments Terraform deployments

Mon, 31 Jul 2023 00:00:00 +0000

Today I Explained

Resources within Terraform can be sometimes be created with the name_prefix field set. Instead of using the fixed name field which leaves room for collisions, the name_prefix will generate a unique suffix, typically sourced from the current timestamp. For a stack with multiple resources using these fields, it can generate a large number of uniquely named resources. For resources that don’t support this kind of field, the option still exists for using the random provider to generate a prefix or suffix.

A workflow for creating new AWS Accounts

Sun, 30 Jul 2023 00:00:00 +0000

Today I Explained

Before automating the creation a new AWS Account using the API or through the AWS Console, there are some processes that should be considered beforehand.

Third party secrets management with dedicated vault AWS Accounts

Sat, 29 Jul 2023 00:00:00 +0000

Today I Explained

Organizations don’t typically grant every individual administrator permissions in third party services, or the ability to create new application integrations. The ability to create slack apps, github apps, oauth clients or service accounts is locked down, typically to the IT or CorpOps groups.

Root domain Hosted Zones within dedicated AWS Accounts

Fri, 28 Jul 2023 00:00:00 +0000

Today I Explained

When a subdomain hosted zone, such as subdomain.aeydr.dev, is created within an AWS Account it isn’t automatically setup for resolution in the wider AWS Organization or internet. This is because the root domain aeydr.dev hosted zone isn’t delegating to this new subdomain hosted zone for managing those records. Within AWS, subdomain delegation is necessary to enable another hosted zone to be responsible for DNS records on a domain.

Managing an AWS Organization using Terraform

Thu, 27 Jul 2023 00:00:00 +0000

Today I Explained

AWS Organizations & accounts are the de-facto approach for separating workloads based on function, compliance requirements, or facilitating an organization’s logical architecture separations. This is because an AWS account is a hard boundary, for which it requires explicit permissions to cross this boundary. For isolating workloads from other areas of the organization (production from staging), it is an effective solution.

A re-usable help command for Makefiles

Wed, 26 Jul 2023 00:00:00 +0000

Today I Explained

Makefiles are a solid approach for having a lightweight task runner within a repository. Other tools for task runners exist that can be used for running common actions or useful scripts, but Makefiles have an appealing labyrinthine simplicity to them. One of these benefits is just familiarity with the syntax:

Default provisioning of infrastructure in a standby state

Tue, 25 Jul 2023 00:00:00 +0000

Today I Explained

Shared infrastructure as code packages often have the number of replicas or minimum size configured with a value of at least 1, or an equivalent to ensure that the service is running when first setup. This is a default behaviour of plug & play to reduce the time to value (TTV). As after running the initial setup, it is possible to begin interacting with the service immediately. This doesn’t have to be the default behaviour. One of the consequences of choosing this behaviour is that infrastructure is provisioned without consideration for the expected workload traffic. This results in numerous infrastructure with resource reservations far exceeding the kind of traffic you are actually working with. Especially in the prototyping or development case, it causes the problem of many small wasteful expenses that have the cumulative effect of a significant budget expenditure.

Minimal AWS Accounts for permission restrictions

Mon, 24 Jul 2023 00:00:00 +0000

Today I Explained

When viewing an AWS Organization you may have come across organization units or AWS Accounts that are named after technologies or responsibilities. Some of these might be similar to the below:

Self-registration of infrastructure with Terraform

Sun, 23 Jul 2023 00:00:00 +0000

Today I Explained

Although Terraform uses the concept of modules as a container for multiple resources, this doesn’t mean that the entire infrastructure of an application must be contained within a single state. In fact, for practical reasons it is often necessary to split resources across multiple state files.

Pullthrough cache for mirrored public ECR images

Fri, 21 Jul 2023 00:00:00 +0000

Today I Explained

When working with container images from third-party registries (DockerHub, Quay, GitHub Container Registry), it can sometimes result in you failing to pull the container images. One such example of this error is:

Centralizing artifacts for an AWS Organization

Thu, 20 Jul 2023 00:00:00 +0000

Today I Explained

Container images aren’t always located within the same AWS Account as the workload, which is why it is possible that you can receive errors when communicating with AWS Elastic Container Registry (ECR) about insufficient permissions to pull from the repository:

Entrypoint AWS Accounts for third party services

Wed, 19 Jul 2023 00:00:00 +0000

Today I Explained

When setting up third party services with AWS, using technologies like OpenID Connect¹ ², if your AWS organization has many AWS Accounts it can result in many ways in which third party services has access into your infrastructure. Without proper tagging or audit controls, this can become a genuine pain point during compliance & audits, as tracking them all down will be an exercise.

Restricted IAM Roles within continuous integration

Tue, 18 Jul 2023 00:00:00 +0000

Today I Explained

When setting up continuous integration, one of the first challenges faced by newcomers is providing credentials to the service to authenticate with AWS. Typically this will be done with IAM Access Keys, potentially something with IAM IoT Devices, or the more recent direction of OpenID Connect if your continuous integration service supports it ¹ ².

Restricted SSH for instances using session manager & IAM

Mon, 17 Jul 2023 00:00:00 +0000

Today I Explained

AWS Systems Manager Session Manager] allows for connecting to EC2s running within the AWS cloud. Provided that these instances have been configured with systems manager, it provides a lightweight way of enabling keyless SSH for servers.

SSM & Pre-defined users for least privilege SSH

Sun, 16 Jul 2023 00:00:00 +0000

Today I Explained

By default, AWS Systems Manager Session Manager uses the ssm-user role when connecting to EC2s that has been configured with systems manager. This user can be disabled, but is enabled by default.

Precomputed APIs using AWS S3 buckets

Sat, 15 Jul 2023 00:00:00 +0000

Today I Explained

Internal datasets in organizations can be tricky, because the internal nature of the data means that approaches for making them accessible are as needs arise. This can mean that internl datasets are distributed in ways such as:

Infrastructure guards to prevent accidental destruction

Fri, 14 Jul 2023 00:00:00 +0000

Today I Explained

It isn’t always desirable for the lifecycles of resources to be fully managed with single execution within infrastructure as code. The possibility of accidently deleting a database or storage bucket containing product data, or immediately revoking a secret that grants access to resources is less than ideal. These kind of events can cause frustrating production incidents.

StackSets with Parameter Store for AWS Resource Access Manager Shares

Thu, 13 Jul 2023 00:00:00 +0000

Today I Explained

When constructing infrastructure that is intended to be leveraged across multiple AWS Accounts, AWS supports resource sharing through the resource access manager. This can be especially useful when working with EC2 Prefix Lists, which allow sharing a group of CIDR blocks for security groups and route tables.

Empty CloudFormation Stacks using WaitConditionHandle

Wed, 12 Jul 2023 00:00:00 +0000

Today I Explained

CloudFormation Templates have a requirement that at least one resource exists within the CloudFormation template, which prevents stacks from being created that contain only outputs. An alternative to creating a dummy resource like AWS::IAM::Role is to make use of WaitConditionHandle.

Kubernetes, CloudFormation and OIDC Magic Numbers

Tue, 11 Jul 2023 00:00:00 +0000

Today I Explained

Magic numbers in programming are numeric-or-other values that appear in code without any explanation of their meaning or any guidance on where they came from. A good example arises with Kubernetes and IAM OIDC identity providers. These provider ARNs look like this:

Mirroring external containers images to internal registries

Mon, 10 Jul 2023 00:00:00 +0000

Today I Explained

Using empty CloudFormation Stacks to publish outputs for CloudFormation

Sun, 09 Jul 2023 00:00:00 +0000

Today I Explained

Terraform can be an excellent tool for provisioning infrastructure, as it supports cross-region & cross-account patterns which aren’t doable within CloudFormation without the use of Stacksets. This can present challenges when needing to expose outputs from this infrastructure to be consumable by CloudFormation.

Using Conditions within CloudFormation Templates for property overrides

Sat, 08 Jul 2023 00:00:00 +0000

Today I Explained

CloudFormation doesn’t support any mechanisms that allows for modifying individual properties of resources in a deployed stack within AWS, which is why in the case of hotfixes you’ll often see a CloudFormation Template modified using the CloudFormation Web Designer. This constraint is why you’ll often see CloudFormation Templates with blank parameters (or a constant like None) for things like (Min|Max)Size, InstanceType, or naming parameters such as fully qualified domain names (FQDNs).

Using CloudFormation Mappings to define preset resource reservations for rightsizing

Fri, 07 Jul 2023 00:00:00 +0000

Today I Explained

When developing within AWS (& the cloud in-general), it is often the case that multiple environments of an applications exist. This can be for the purpose of prototyping, staging environments for ensuring new changes are working as intended, or taking advantage of computing savings that can exist within other AWS Regions.

Using multiple Terraform AWS Providers for global infrastructure

Thu, 06 Jul 2023 00:00:00 +0000

Today I Explained

Sometimes when working within Cloud you’ll have requirements to ensure data redundancy or operate services within specific geographic areas. When working with AWS, this will mean that you’ll need to provision resources within multiple AWS Regions. For applications, this can mean just deploying a lambda or machine within a certain region. For infrastructure, this can present challenges when the infrastructure needs to span across regions.

Using Feature Flags within Terraform for conditional infrastructure

Wed, 05 Jul 2023 00:00:00 +0000

Today I Explained

Resources within Terraform can be conditionally created but not by a toggle like enabled or disabled, but instead using the count meta-argument. This argument takes in whole numbers (>= 0) to determine how many of a given resource to create, and using a ternary operator you can convert a boolean to a integer for conditional resources:

Using built-in CloudFormation macros to source AMI IDs by a friendly identifier

Tue, 04 Jul 2023 00:00:00 +0000

Today I Explained

CloudFormation doesn’t support any look up mechanisms for Amazon Machine Images (AMIs), which is why you’ll often see AMIs for EC2s specified either as:

Using built-in CloudFormation variables to generate unique resource names

Sat, 17 Jun 2023 00:00:00 +0000

Today I Explained

When working within AWS, eventually you will encounter a service that needs unique names, and typically you learn this by an error message that sounds similar to: