<![CDATA[Chamber 🏰 of Tech Secrets]]>https://brianchambers.substack.comhttps://substackcdn.com/image/fetch/$s_!qlyH!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbrianchambers.substack.com%2Fimg%2Fsubstack.pngChamber 🏰 of Tech Secretshttps://brianchambers.substack.comSubstackFri, 24 Apr 2026 14:52:25 GMT<![CDATA[Chamber 🏰 of Tech Secrets #55: Issue-Driven Development]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-55-issuehttps://brianchambers.substack.com/p/chamber-of-tech-secrets-55-issueMon, 17 Nov 2025 10:30:42 GMTThe State of AI Coding in the Chamber 🏰 of Tech Secrets

Over the past 4+ months, I have continued to leverage AI coding tools at an often feverish pace and continue to learn and refine my process for maximizing productivity and minimizing tail-chasing and frustration. While frustrations do occur (see my scornful linkedIn posts), I still find these tools to provide a meaningful productivity boost. 🚀

Claude Code remains an indispensable part of my personal process for…

  • Building REST APIs (including database SQL, DDL)

  • Designing and developing front-end web apps

  • Synthetic Systems (like a synthetic PLC I made recently for testing)

  • Code Reviews

  • Managing non-prod infrastructure deployments (iteration through errors)

  • Doing 60-80% of the scaffolding for edge projects

Changes to my process

Back in July, I wrote about my spec-driven development process for AI coding. That process was designed to achieve the goals of maximizing productivity and minimize the tail-chasing of one-shoting apps or features. You can read that here.

Since my last post, the amount of AI code projects I am working on has 5x’d, resulting in some new learnings and meaningful refinement of my processes.

In my previous post, I wrote about how I used a specification document (architecture.md) and a list of tasks (tasks.md) that needed to be read into context and executed by an AI coding agent.

While this worked out great for a lot of projects, there were also some issues I bumped into…

  • Design drift: No matter how thorough a design, there will always be some changes as you iterate through execution. Practically, remembering to keep the architecture.md up to date through these iterations requires extraordinary discipline. Just one miss and your design will start to drift over time, which poses problems when the old approaches continue to be read in for planning and executing future tasks.

  • Attention Skew: LLMs process all tokens in a sequence (the context window) on every request, calculate the best responses, and return them to the user. A heavy and broad specification has the potential to skew the model’s quality of output on niche task execution, resulting in subpar intent following or poor design outputs. Reading the entire architecture document every time is not ideal. Reading all tasks in a tasks.md is not ideal either. Which leads us to…

  • Context Window Management: Quality in → quality out. “In” is everything in your context window, so managing it to include only the essential is very important. It is also essential for personal sanity. AI tools like Claude Code perform auto-compaction of the context window as it nears its limit, which is a pain to wait for when you are iterating through an issue. Furthermore, during compaction, most of your “conversation history” is lost and the agent will have to rediscover basic things like where your local database is (again!). I have found that focusing on just enough detail in the context window (with frequent `/clear` commands) works best. Using the minimal MCP tools (not just servers…tools) can also save meaningful space in the context window (every tool comes with some token overhead).

Note how a fresh Claude Code w/ Sonnet 4.5 and a single MCP tool (GitHub) has. already consumed 45% of the available context window

What can be done about these issues?

Issue-Driven Development

To address these challenges, I have shifted to what I call “issue-driven development”. The guts of the process are similar but in practice I find this approach to yield better results and to be cognitively easier to manage.

In an issue-driven development approach, we shift from managing architecture.md and tasks.md files to managing only rich tasks which are stored in an “issue”. In my case, these are GitHub issues, but you can use any issue tracking system you wish.

Rules Files

The first thing we need to ensure is that our agent remembers what we are doing in our project and what the basic structures of our project are. Per the context window management idea above, we want to keep this minimal.

I have found rules files (claude.md, .cursor/rules, etc) to be a good way to inject the minimal “memories” into the context window to prohibit wasting tokens and time understanding the project after a /clear or /compact occurs.

I like to keep the rules files minimal and focus on…

  1. what is the project doing

  2. what is the tech stack

  3. where are the basic things that need to be known / reused (database, env vars, etc)

  4. what are the basics guidelines for development

As your project develops, you will likely have additional things you want to add to your rules file to help the coding agent efficiently navigate the project. That’s great, just keep it to the essential.

Here is a sample from a recent project.

# CLAUDE.md

## Project Overview

What We’re Building...
A simple relationship management system for managing farmer relationships and measuring community impact over time.

## Project Overview

/crm-api - contains the golang REST code
/crm-ui - contains the frontend React/NEXTJS code

## Database

Database: `oicrmdb` running locally in Docker. Find ports and credentials in /crm-api/.env.example.

All migrations are in /crm-api/migrations and should follow the convention 001_overview_of_change.sql. Sample data scripts must follow the same convention and be stored in /crm-api/migrations/sample_data.

Ports:
* REST API runs on 8099
* UI runs on 3039


## Project Overview

Tech Stack:

* Frontend: Next.js with TypeScript, Tailwind CSS + shadcn/ui components
* Backend: Golang REST API using standard HTTP library (no frameworks)
* Database: PostgreSQL with direct SQL queries (no ORM)
* Auth: OIDC via Keycloak for authentication at https://...

Development Philosophy:

* Verbosity and cleverness is bad, simplicity is good
* Idiomatic, simple Go code with minimal dependencies
* Behavior-focused testing over code coverage metrics
* Minimize mocks in tests - prefer integration tests where practical
* API abstracts storage implementation from UI
* Small, incremental changes that can be developed and tested independently
* Reuse UI components (data tables, etc)
* Never disable auth, TLS, or do other similar things to “get it working”. Always work the problem thoroughly until completion

I always hand-roll this file to keep it minimal and specific, but may use an AI agent to update it later.

Requirements Development

How do issues get developed and stored?

As in my prior approach, I usually start by thinking through a design with a model… typically Opus 4.1 via Claude Desktop for me, but use whatever you want. No reason not to do this with Claude Code, either (can write files more easily or reference other local projects for patterns).

I still like to create an architecture.md file as an output but I like to keep it minimal.

In light of that architecture context (in the 1M token window w/ Opus) I create a list of tasks or issues. I have the GitHub MCP Server installed with Claude Desktop and Claude Code (using a PAT scoped down to only what I need). Once I am happy with the design and list of tasks, I ask Claude to load these as new Git issues.

Prompt: I like this design. Please take each of these tasks and create a GitHub issue for them one-by-one. Add them to the project named "<project name>". 

*** I say “one-by-one” because batch loads never seem to work and I want to avoid wasting my time and tokens.

Now we have a set of baseline issues we can work with and a visual on what we are building and where we are (again this would also work with jira, asana, trello, etc. provided they have an MCP server).

An example from a recent project

In all of this, what we are trying to achieve is the composition of a perfect task that carries all necessary architectural context, all necessary outcome context, and all necessary “how we work” context without being overly complicated or verbose, such that the task can be executed successfully from a single issue. This is hard to get right across a large number of tasks, so in practice, I have another step I typically do.

Prompt in Plan Mode: Pull issue #107, read it thoroughly, and make a plan for how to implement it in light of our architecture (see architecture.md). 

<Review the Plan>

Prompt: Please update issue #107 with this plan and move it to state of "Ready". 

The most important thing in this phase is knowing what you want to build and how you want to build it. Precision really matters.

Task Execution

Assuming we get the requirements stage right and have a robust task, execution becomes very straightforward.

Prompt: Pull issue #107 and read it thoroughly. Write behavioral tests first and write code until they pass. Do the work in a new branch, execute the task completely, and then push the changes and create a Pull 
Request. Move the task to "In Review" when complete.

If you wish you can avoid the dual issue pull here and just implement the plan directly. I like to log the plan back to the issue for posterity (why did I make that decision again?).

Code Reviews + Additional Issues

We must, of course, review the code generated by our agents.

There are multiple options for Code Reviews. For a live production system, you likely want to review every one of these features with an AI code review and a human eyes review. If you are building a new system from scratch and are working on a lot of issues in a short time, you might be okay with batch code reviewing (AI + human again). For enterprises, go with the former.

For AI code reviews, I prompt a few times (in parallel) with different perspectives.

  • Simple, clean code

  • Architecture

  • Security

I execute these reviews with standard prompts I have created using Claude Code Subagents. Outputs are grouped by priority.

For each, I review the results manually, and then ask the agent to open GitHub issue for the ones I deem legitimate (it is never all of them).

Another thing I like about this issue-driven approach is dealing with bugs and defects I find while reviewing code or the functional system’s behavior. I often find a slew of semi-related or unrelated defects while working an issue. Personally, that is a big cognitive tax… I know there are many things wrong and I feel like I am quickly losing track or getting distracted chasing them down. With an issue-driven approach, we can easily 1) manually in GitHub or 2) via Claude Code open a new issue via MCP. I do both depending on the nature of what I find. Using the agent approach can help inject useful context from code into the issue which can be helpful when you interpret it later.

Cloud Deployments

Most of the work I am doing right now is for a non-profit where I serve as a Volunteer CIO/CTO/Software Engineer. Given that they have a footprint in Google Workspaces, I decided to use Google Cloud for all of my cloud deployments (Google Cloud friends, email me with non-profit credits! 😂).

Claude Code has been really helpful in working through live development environment configuration issues with Cloud Build, Cloud Run, and other services using the gcloud utility locally. I set up secrets and build triggers and such myself, but when build triggers were failing for various reasons early, I was able to have Claude tail the logs and keep iterating on small changes to my cloudbuild.yaml until everything was right.

Final Thoughts

A few final thoughts…

  • Obviously, review the outputs at some point. These tools are great but not perfect. They are great at getting things working but don’t always follow your design parameters. They will take shortcuts like disabling authentication completely “because that will be a simpler approach”. They will create some slop which you can have them fix, but you have to be aware of it.

  • For lower level systems work where precision matters, you might get some benefit from scaffolding but may be better off hand-rolling the more important elements of the code (maybe a 60/40 split).

  • There is no magic in the prompts I shared. I often change them slightly.

  • I haven’t fully automated my workflow yet but still think that would be awesome.

No approach is perfect. Mine is not. If you know what you want to build and how, it can work quite well and scale quite nicely.

I hope it helps you on your journey. I remain slightly averse to other people’s frameworks (Kiro, etc.) that force me to stop working and learn a new tool… but I think I am just busy and therefore lazy.

What is working for you? I’d love to hear in the comments.


Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #54: Spec-driven Development]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-54-spec-drivenhttps://brianchambers.substack.com/p/chamber-of-tech-secrets-54-spec-drivenTue, 08 Jul 2025 09:30:42 GMTVibe Coding 🏄‍♂️

Can you really vibe code an application that works without writing any lines of code yourself? We seem to be in a weird place right now. Some engineers dismiss AI coding tools with a hand wave and a thought of “it can’t do what I do”. Other “engineers” are one-shotting apps into “production” with no understanding of what an HTTP header or a container is.

What a time to be alive!

Allow me to tell you about my experience vibe coding, the process I have been using, what I’ve learned, and what I plan to do in the very near future.


My Experience

I have vibe coded quite a bit over the past 4 months and have experimented with a number of tools and approaches. Tools like Cursor, Windsurf, and Claude Code. Approaches like one-shot “build me an app that does X, Y, Z… <long prompts>” or “build me a feature that does X”.

In my first attempt, I finagled my way to a working application with a simple REST API in golang (a language I’m very comfortable in) and a frontend written in React/NextJS (an entire programming paradigm I avoid whenever possible) via Cursor. Given the misguided paths along the way, that application took me about 4-5 hours over a few days to get working the way I wanted with pretty good quality.

Needless to say, I was hooked! 👏

A screenshot of my vibe-coded Tech Landscape app (inspired by CNCF Landscape)

Then, a few months ago, I came across the idea of spec-driven development which was quickly validated and expounded on by a lot of the content at the AI Engineer World’s Fair event I attended in San Francisco in early June.

My Process

There is a lot of good information available about using AI tools to write software, but I thought it would be good to consolidate my own workflow here. Here is what I do currently.

We’ll walk through the process step-by-step.

Design Loops

The first thing I do is iteratively develop a specification file named architecture.md. Ideally, this artifact lives longer than the code and enables re-generation of the code base in the future.

Practical? 🤷‍♂️ Useful to get a good v 1.0? ✅

Architecture Specification

Here’s how I build my specification, which is a mix of steps, tools, and thought processes co-mingled.

  • Tooling: I master my specification in markdown in Obsidian so that I can jump around the process in my conversation with the friendly LLM. I name this file architecture.md.

  • LLM help: I usually start with a prompt describing what I want to build and ask the model to generate an architecture specification in markdown. Reasoning models are great for helping with the architecture.md creation process. I am pretty happy using Gemini 2.5-pro, Claude Opus 4, or OpenAI GPT 4-o or o3.

  • Handcrafting: Having done this a few times, I have learned that sometimes it is often worth the time to put more manual work into this file. Don’t shortcut here — describe what you want. Be extremely clear about the features you want, how you want them to work, what you want the tech stack to look like (uv for python, standard go http vs gorilla mux, postgreSQL, etc.), and what “done” looks like.

  • Other stuff: You may want to put the DDL for databases (I usually generate that with the model iteratively) or the REST API endpoints you want to see built in the spec. You might want to put references to particular specifications (use MCP, use the Google API design specification, etc.) you want to follow. You might want to put a generated project structure in too. Honestly, more seems to be better.

  • Write as if for a child: “you must use MCP and follow the specification found at <link> exactly. Do not deviate from the specification under any circumstance and ensure the code implementation is compliant with the spec.” It is essential to be very specific about your expectations, much as if you were helping someone on their first programming project.

This specification development step takes about 50% of the total project time with my current process.

Here is an example of the architecture.md for a simple task management API and MCP server I put together for niche use on a personal project.

Initial version of the architecture.md for the taskman project

Task List

The next step is to create a list of tasks that, once completed, result in a codebase that meet my functional and technical requirements.

To do this, we take our architecture.md and leverage an LLM (I like Claude Opus 4 or Gemini 2.5-pro for this purpose) to create a comprehensive task list. Here is a prompt similar to what I use when performing this task:

Prompt: “Your job is to create a task list to complete the project that is defined in the file `architecture.md`. Each task should be small, highly testable, and designed for a large language model to complete. Ensure the foundational tasks (database setup, project setup, foundation code components like servers) are defined first so that the following tasks can leverage them. Provide the task list in markdown format.

The resulting tasks.md is usually pretty good, but I open it in Obsidian and do some adjusting to make sure I like the way everything comes out. As I read the tasks, I usually think of things I should have added in architecture.md at the start, so I will make those changes (and regenerate tasks) too.

Initial tasks.md for the taskman project, generated by Claude Opus 4

This process usually takes about 5% of the total time on a given project (10-15 minutes tops).

We now have our two essential artifacts: architecture.md defining our solution requirements and application features and tasks.md defining the list of tasks that must be completed to

Development Loops

Now let’s talk about how development loops work, which is our implementation phase.

In this phase, we iterate through our tasks with identical (or nearly so) prompts like this one.

Please familiarize yourself with the architecture in architecture.md. Once you are ready, create a plan for executing Task 11 in tasks.md. Always write minimal code. Ensure everything developed is thoroughly tested and that all tests pass before the task is considered complete. Ensure that everything written for the MCP server is compliant with the MCP spec, per the architecture.md instructions.

Note that our architecture.md specification is going to get read on each prompt (by design) so it will be freshly injected into our context window. We’ll do the same for the specific task we’re working on.

At the end of each task—assuming we like what we see—we’ll ask the model to git add and commit because this is far too much work for us to do ourselves.

This is also a great time to ask questions about the current code base if you want to make sure that things are on track. Take a peep at the code too and make sure the structure is looking good.

From a tool perspective, my current process leverages:

  1. Agentic Coding Tool: Obviously we’ll need an AI-assisted or driven coding tool. My favorite has quickly become Claude Code over the past month or two, driving me to the “max” subscription so I can jam away as much as I want. I was running out of tokens before, hence the switch. For those unfamiliar, Claude Code is entirely terminal based. I have my own claude-yolo alias so I can bypass questions and permissions when needed and let it work a task from end-to-end.

  2. Visual Studio Code: I keep VSCode open so I can look at code changes here and there and make tweaks if needed (very rare).

Remember that AI Engineering is Context Engineering, so manage your context window thoughtfully. Claude Code tells you what percentage remains before auto-compaction. It is often prudent to proactively compact the context at the end of a feature, or clear it altogether so that you can run through each task and any subsequent iteration without running into the end of the context window, which always results in… well… lost context. 🤷‍♂️ Carry over context is not that important since everything that matters should be in the architecture.md and tasks.md files.

Maintaining the spec

Naturally, no specification is written perfectly and there inevitable misunderstandings as tasks get completed, tested, and reviewed (by me). What to do about the architecture.md? What has worked well for me is working iteratively (2-3 loops) on changes that are needed to the code, then asking the LLM to also make updates to the architecture.md file to reflect the design changes. In fact the architecture.md you saw earlier was actually co-edited by me and Claude Sonnet 4. This keeps our specification in good standing but lets us account for shortsightedness in design thinking.

Code Review Loops

Intermediate AI Code Reviews

As the development process proceeds, I like to interject a code review every 3-4 tasks to see how things are looking. Here’s what that looks like:

Prompt: You are a deeply experienced, highly critical software engineer. Please conduct a deep and thoughtful code review of the code base you find here. Only tasks 1-3 have been completed so far, so you will need to ignore other features and focus on the quality of what has been implemented. Be sure to root in the architecture.md for project goals, tech stack and guidance. You may find the tasks in tasks.md at the root of the project.

This often yields some findings which I will address iteratively. If there is anything that justifies an architecture.md update, we’ll make that change as well.

Final AI Code Review

Prompt: Please complete a final code review of this project. Be very diligent about software quality, testability, security, specification compliance, good REST practices, and minimal lines of code. Call out anything that is not well done or up to standards. Make sure all of the architecture.md features are implemented and the coding practices were followed. Provide recommendations for anything that should be improved, removed, or cleaned up. Ensure we have good logging and error handling throughout. Think long and deeply about this review.

Claude Code in yolo mode performing a code review

Final Human Code Review

Finally, it is time for a thorough human review to make sure everything looks good. I typically find that Clade Code has left a few linting errors behind, so I’ll use Cursor to fix those. It is really good resolving these types of issues and it’s as simple as clicking on the code line and then “fix in chat” and hitting enter.

Lessons Learned

  • Be explicit: Being explicit about the behaviors you want is essential. Logging, error handling methodology, use of an LLM… If you are not clear, you’ll get something that is not what you want. Be clear about tech stack, minimal code, consistent logging and error handling, and everything else that is normally in your head when you work on a project yourself.

  • Active the key terms: Prompts like “minimal code” and “easily readable” and “simple as possible” seem to yield good quality results within this process. Claude models also react to “think about this”, “think

  • Think product, think architecture: I name the file architecture.md for a reason. I think architecture thinking will increase in importance over the next several years as mass producing code becomes so easy. Good ideas + good explanations + good understandings of how to design things + good process = good outcomes. 🚀

  • n tools > 1: Use the right tools for the job. My spec-driven development involves Claude Code, Obsidian, Google Gemini or ChatGPT o3, VS Code, terminals with tmux for testing, and Cursor.

  • Commit early, commit often: If things go sideways, this is your safety net. It’s good practice anyway.

  • Code Review Frequently: Prompt for code reviews every 3-4 tasks.

  • Update the specs: Update architecture.md and tasks.md if you discover new architecture needs, features, or tasks that should be completed. Well, have the LLM do it, of course.

  • Integration tests are life: Integration tests are the best way to tell if you code is really working properly so make sure to have the model write them as early as possible—preferably first—and then iterate until they pass. I do not always do this but I do ensure the tests are good and pass before moving on (usually).

  • Make a plan, review the plan: I usually put Claude Code in plan mode when giving it a task, and I do review these plans before switching back to “yolo” mode. This has allowed me to catch the model over-engineering with features like in-memory caching for a super low-volume REST API that I am running on my laptop.

  • Yolo mode: Speaking of, yolo mode is a must unless you want to stare at the terminal and wait for a request to run a cd or docker compose command. There are obvious risks here, but they are acceptable given my current use. And I have a future plan!

My Future Plan

I have plans.

  1. The task management API and MCP you see alongside the example architecture.md from early in this article supports this next step. Put projects and decomposed tasks here.

  2. Create an ambient agent that watches for new tasks for a given project that need to be completed.

  3. Allow the ambient agent to farm the task to an instance of Claude Code running in a docker container in a git working-tree, powered by container-use (thanks Dagger/Solomon Hykes!). This agent will manage what tasks can be worked on in parallel vs linearly.

  4. Have the agent working on the task submit a PR for the working tree when the task is complete.

  5. Have a team of agents perform a code review on the submitted code. Three agents is my current thinking: 1) bias for architecture 2) bias for software engineering best practices and minimal code and 3) bias for cybersecurity. These will likely have standardized, structured prompts that can be fetched from an MCP server on-demand for consistency across projects.

  6. Revise the code based on the feedback from the reviewers.

  7. Assign to Brian for final human review.

  8. Merge and trigger CI pipeline and deploy to next stage environments.

  9. Profit.

This should let me start building solid tasks (perhaps taking more requirements away from architecture.md and keeping it more as guardrails) and then completely automate the in parallel. 🚀 Let’s go!

Will there be challenges with large or complex projects? Absolutely. Will this build you a competitor to google search? No. But for many business or personal applications, this flow can work quite well. Keep it simple and take small steps one step at a time. The models and tools are only going to get better. Happy vibing!


Are you vibe coding? Or have you moved to specification-driven development? What has your experience been? I’d love to hear from you in the comments or on LinkedIn. Until next time, keep the vibes going 🏄‍♂️… but don’t delete all the files on your laptop or your companies get repo. 🤪

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #53: Never forget]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-53-neverhttps://brianchambers.substack.com/p/chamber-of-tech-secrets-53-neverFri, 23 May 2025 09:30:39 GMTNever forget that…

  • Little smiles are better than the best of Agentic AI.

  • Hugs are better than one-shot vibe coding an entire project.

  • Rolling a ball together is better than any new LLM release.

  • Laughing at a sneeze together is better than completing 1,000,000 Trello tasks.

  • Reading a 6-page book together is better than finishing your own reading list.

It’s hard to remember when things are busy and changing so fast. It’s easy to forget what really matters. Don’t forget.

I hate to put my little guy on the internet but we’re all friends here.
]]>
<![CDATA[Chamber 🏰of Tech Secrets #52: Edge Device Onboarding]]>https://brianchambers.substack.com/p/secure-device-onboardinghttps://brianchambers.substack.com/p/secure-device-onboardingTue, 13 May 2025 09:31:11 GMTThe Chamber 🏰 of Tech Secrets is open!


Secure Edge Onboarding

Welcome to Living on the Edge Part II, where the goal is to go from zero to production with an edge computing environment.

First things first… we need to design an architecture where we can securely onboard devices into our lab environment… and this requires a significant lift! 🏋️‍♀️


📦 Plug it in and let’s go!

When a new edge device shows up in a box, it’s time to open it up, plug it in, and get excited!

We’ll need to use an approach often referred to as zero touch provisioning, meaning that we do not expect a skilled IT technician to be present to complete setup… we simply plug and play and the rest of the things take care of themselves.

Before we go too far down that road, what should we have already done?

  • Secure Supply Chain: did the devices come from a trusted vendor? In my case, they came from my trusted friends at SimplyNUC. ✅

  • Do you know where you are on the risk spectrum?: How risky is this environment and what security controls do we need to have in place? In my case, this is a home lab environment running non-critical applications and I will not allow for inbound web traffic (a best practice), so our risk profile is very low. That said, my purpose here is to explore the challenges of a true enterprise environment, so we’ll “over-engineer” a little for the sake of learning. If this device was going to a Google datacenter or a tactical application, it would need more rigor, perhaps even a custom TPM (what Google does). ✅

  • Architecture & System Requirement: What is the environment’s architecture, including the network topology, hardware profile, and security requirements? My requirements are simply to learn and brute force through challenges with the help of AI, so ✅ 😀.

  • Attestation Server & Certificate Authority: We will need an attestation server and CA that we can depend on. To learn more about how this works, check out Chamber 🏰 of Tech Secrets #50.

For a more complete and thorough analysis on the above topic, check out the recent blog post on Secure Device Onboarding from me and my friends at Edge Monsters.


A quick note: the flow we will describe is essentially the same that you would use in a full datacenter or cloud environment if you were a major provider like AWS or Google. The datacenter paradigm is essentially the same at the edge when it comes to establishing a secure and trusted environment, which is cool.

First Boot

Let’s get started!

On first-boot, our device will be configured to PXE boot into a minimal mode with the base code for facilitating the process of establishing trust, which we’ll dive into below. If all steps pass, the device will boot a complete, first-class image that allows it to participate in the “datacenter” environment it is present in.

What are those steps that must pass to establish trust? Enter the TPM.

What is a TPM?

Each of my edge devices is equipped with a special chip called a Trusted Platform Module (TPM). A TPM is a small hardware chip on a computer's motherboard that provides hardware-based security functions such as securely storing encryption keys, passwords, and certificates.

Once the TPM module connector has been aligned, insert it into the socket until resistance is felt.
TPM Chip being placed on motherboard. Source Open Computing Project

TPMs help us establish what we call a “hardware root of trust”. What are the benefits of a TPM, you ask?

  • Secure Physical Design: TPM chips are built to make physical attacks really hard. They're packaged in ways that protect against drilling, cutting, or probing. Some even have special coatings or sensors that detect if someone tries to open the chip — and if tampering is detected, the chip can zero out sensitive data.

  • Internal Encryption: Sensitive data inside a TPM is encrypted within the chip itself. The keys never leave the TPM unencrypted, so even if someone somehow intercepts communications with the TPM, they can't grab the secrets it possesses.

  • Limited Interfaces: TPMs have very strict, minimal ways you can interact with them. You can't just send random commands or dump memory.

  • Independent, Isolated Execution: The TPM operates separately from the main CPU and memory, providing a highly secure environment for select operations such as creating key pairs or writing digital signatures.

Now that we understand what a TPM is and what value it adds, let’s get into the technical weeds on how to go about working with one.

Establishing Trust

Our goal is to establish trust. To do that, we need to start by ensuring the can trust our TPM chip itself, since it is at the root of everything that follows. If we trust something in the chain that has been compromised, we lose.

We then want to assert an identity (often called remote attestation) to the server in our environment that is responsible for new device onboarding and configuration. That trust process requires that we:

  1. Prove that the TPM is trustworthy and has not been tampered with.

  2. Prove that the device is trustworthy by having the TPM sign its configuration.

  3. Prove that the device is in the intended boot state and has not been tampered with.

Only when these conditions area met will we consider an edge node to be trustworthy and allow it to run production workloads. Let’s dig into how these trust proofs work technically!

Endorsement Key

TPM’s are typically shipped with an embedded asymmetric key pair called an Endorsement Key (EK) pair. The private key in this pair lives inside the TPM and is never released or shared. 🔐 Typically, the device manufacturer will digitally sign the EK public key with their own certificate authority, producing an EK Certificate, which can be used to validate the “identity” of the TPM.

If we are able to validate the manufacturer’s signature wrapping our EK public key, we can be highly confident that the TPM is trustworthy for subsequent operations, given that we trust the manufacturer. This establishes our baseline level of trust.

The EK is extremely sensitive material and is at the root of our trust topology, so we will use it for the fewest operations possible to minimize risk.

Attestation Key

Next, we’ll generate a new asymmetric key pair in the TPM on the device called an Attestation Key (AK), which we use to assert the identity and state of our device in subsequent boots. The private key for the AK pair will stay locked away in the TPM forever 🔐. We can retrieve our AK public key from the TPM and use it in our next step.

Certificate Signing Request

On the edge device, we’ll also generate a Certificate Signing Request (CSR) for the Attestation Key (AK) pair. This operation will be orchestrated by our initialization software running in the minimal boot OS on the node, but the digital signature will take place within the TPM. Remember that the private keys generated in the TPM stay in the TPM.

Something like this…

When the attestation server (also acting as a Certificate Authority—whether cloud or local) validates the CSR, it confirms that the device controls the private key—stored inside the TPM and therefore safe—that was used to sign the request and can thus be trusted in subsequent operations.

A certificate request might look like something like this.

-----BEGIN CERTIFICATE REQUEST-----\nMIICVzCCAT8CAQAwYTELMAkGA1UEBhMCVVMxEzARBgNVBAoMCk15Q29tcGFueTEZMBcGA1UECwwQVHJ1c3RlZERldmljZXMxFzAVBgNVBAMMDlRQTURldmljZS1TRjEwMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC+JFiqqE0B5ZojHnKURD4AaScs5f4nPAV6/0KL7ldN3ojYK0ZIf5d6H5M5TK+lb7QKPr8mg+WECiSMgxxAnb9MZPVZReKpc6hivDWX4knsydJq1lB3nQ6TLh+EvRzSvupcDWBMI8Z7nNpiL6U5fRkQ1EMt+z9IM2f63D+nk7ePwiwIDAQABoAAwDQYJKoZIhvcNAQELBQADgYEAG1MTQJg2ct9tivlguXNRljIl6pRPYe5zEwFElU32v9s4rRWc5PXx4+/5N0ByNHX5l7z90a/bAFT9gnHKZ0ZhUkUIor4DLcycQ3HOj6LiwbI3xMklGMOMXnF9RUy5hYz5AdA12Pf8DklZW0tivBLrGbT3Wh7sbU5TZVa5upLfY8=\n-----END CERTIFICATE REQUEST-----

That’s a lot! Let’s do a quick recap. At this stage of the process, we have our Endorsement Key (EK) Pair and Endorsement Certificate. The EK Private Key is safely locked away in the TPM. We also have an Attestation Key (AK) Pair. The AK Private Key is also locked away in the TPM. Finally, we have our hands on a Certificate Signing Request for the AK so that we can prove this is a trusted device.

Measured Boot + PCR

Before the attestation server is willing to trust this device, it will need to make sure it is booting in the manner expected, once again ensuring there has been no tampering. This is where Measured Boot and Platform Configuration Registers (PCRs) come into play.

A device could have malware installed before shipping, so verifying boot integrity is critical for a secure edge environment.

What is Measured Boot?

In a “Measured Boot”, each component—BIOS, firmware, bootloader, kernel, etc.—measures itself (i.e., hashes its code) and stores the hash inside TPM registers called PCRs (Platform Configuration Registers).

Measured boots can be used to prohibit the release of material from the TPM in a process called credential sealing, which we’ll discuss in a future post.

Measured Boot can also be used by the attestation server verify that the device booted into a known, trusted state by checking a hash of the chain of PCR values against a previously established “golden” state stored on the server side.

Platform Configuration Registers (PCRs)

TPM chips have a small number of PCRs (typically 24). As the system boots, hash measurements from each stage are extended into specific PCRs. Each extension is cryptographic hash (likely SHA-256 or similar) of the previous values and the new value concatenated together, creating a tamper-evident chain.

Example PCRs:

  • PCR 0–7: firmware and BIOS stages

  • PCR 8–15: OS loader and OS measurements

  • PCR 16+: Application measurements (not often used)

It is important to note that Measurement != Enforcement. In measured boot, we record what happened but do not block progress (that's called Secure Boot). These measurements give us a verifiable history of what code executed from power-on up to OS start.

Contacting the Attestation Server

Now we are ready to contact the attestation server and begin the process of onboarding the device.

The first request to the attestation server will be a simple nonce. For our purposes, we’ll send the device’s serial number in the request and the attestation server will send us a reply containing a random nonce. We do this to protect the server-side system from potential replay attacks.

HTTP Request: POST /nonce
Body: {"serialNumber":"SN9753124680"}

HTTP Response: 
Body: {
  "nonce": "f8a9d3b2e1c447d29f7be7a4e9623f75",
  "nonce_validity_seconds": 300,
  "attestation_server_time": "2025-04-26T20:00:00Z",
  "device_serial_number": "SN9753124680",
  "request_id": "9d4b03a1-0e29-4d7a-b6ea-c94b13eaf65f"
}

It is also important to ensure that the edge device is communicating with a legitimate attestation server before sending any sensitive onboarding data or taking configuration instructions.

Upon initial connection, the server will present its certificate, which the device can validate against the trusted Root CA certificate. We can provide this to the device at initial image time or in the minimal OS PXE-boot. This enables us to verify the identity of the attestation server and establish secure communications with TLS/HTTPS.

Finally, a secure (or perhaps measured if lower risk) boot on the edge device can validate that the PXE boot—discovered via DHCP—put the device in an expected state that is trustworthy.

Next, our edge device can request that the TPM create a quote, which includes a digest of select PCR values combined with our aforementioned Nonce, which is signed with the AK Private Key (trusted by the attestation server).

Next, we can send a bundled request to the attestation server including the following components:

  • EK Certificate (EK Cert)

  • AK Certificate Signing Request (CSR)

  • PCR Quote (includes nonce)

  • Device metadata

An HTTP request to the attestation server will look something like this…

HTTP REQUEST: POST /v1/attest
BODY
{
  "device_serial_number": "SN9753124680",
  "nonce": "f8a9d3b2e1c447d29f7be7a4e9623f75",
  "csr": "-----BEGIN CERTIFICATE REQUEST-----\nMI....",
  "pcrQuote": "<<Base64-encoded TPM Quote Structure>>",
  "ek_certificate": "-----BEGIN CERTIFICATE-----\nM.....
}

Certificate Issuance

Our attestation server will field this request and validate the components of the payload…

  • EK Certificate: Assess the EK Certificate to see who signed it (the manufacturer we hope). The server will validate the digital signature using the full certificate chain from the manufacturers root CA to confirm hardware provenance and trust. This certificate chain will need to be acquired from the manufacturer(s) out-of-band or on-the-fly at request time within the attestation server implementation.

  • PCR Quote: Check the PCR values against the “golden” state values (computed previously out-of-band) to ensure there was no tampering, plus validate the Nonce that was issued to ensure the quote is appropriately recent.

  • Device Metadata: Check device metadata against an out-of-band registry or store the device record in a database for future asset management purposes.

  • CSR: Unpack the public key from the CSR and validate the digital signature. Additionally, we can validate other CSR fields such as serial number or MAC Address against a registry if desired.

The combination of these values tells the server that the TPM is trusted, that the AK private key is safe in the TPM, and that the device has booted into a trusted state. Under these circumstances, we consider the device trustworthy and can proceed with fulfilling the CSR by issuing an X.509 AK Certificate, which will represent the device’s identity ongoing.

Our attestation server will store the following device identity attributes in a database:

  • Device serial number

  • AK Certificate serial number

  • AK Public Key fingerprint (useful for lookups, etc.)

  • Certificate validity (not before, not after)

  • Certificate PEM (optional)

  • Device Status (active/revoked)

The edge device will store the issued AK Certificate on the file system (not inside the TPM).

Note that we can reissue new certificates for the device over time by repeating the attestation process. Device metadata can help deduplicate records across future attestations. This process may occur hourly, daily, or monthly—or perhaps every boot—depending on your security needs.

The AK certificate typically contains:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0x0d2341be3
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Local TPM Attestation CA, O=MyCompany, C=US
        Validity
            Not Before: Apr 26 12:00:00 2025 GMT
            Not After : Apr 27 12:00:00 2025 GMT
        Subject: CN=device-01234, O=MyCompany, OU=TrustedDevices, C=US
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            Public-Key: (2048 bit)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation
            X509v3 Extended Key Usage:
                1.3.6.1.5.5.7.3.9 (Custom for TPM attestation)
            1.3.6.1.4.1.45724.2.1.1 (TPM Manufacturer Info Extension - Optional)
    Signature Algorithm: sha256WithRSAEncryption
         <CA Signature bytes>

Additional Considerations

  • Time matters ⏰: Before initial attestation, devices should sync with the local NTP server to ensure their time is accurate. If they have drifted meaningfully or been misconfigured, the nonce check is likely to fail.

  • Human-in-the-loop 🧔: Do you want or need to know that a trusted person is present in the attestation sequence? If this is important for the environment, a protocol like FIDO with a YubiKey is a great solution. In this case, we can add an extra layer of cryptography to the equation prior to the initial release of the Attestation Key Certificate (AK Cert). This adds an extra level of trust, ensuring the device is in the presence of a trusted human.

A YubiKey in case you are not familiar

In my lab environment, I’ll implement the FIDO + YubiKey flow because… why not? Another opportunity to learn! I’ll use a simple NTP configuration since I do not have possession of highly accurate atomic clocks.

Conclusion

By completing all of these steps, we can ensure that an edge (or datacenter) server is completely trustworthy 🔐, has not been tampered with 🦹, and is running the configuration we desire (vs. something compromised) ✅ upon arrival.

We also learned how the incoming device can ensure it is talking to a legitimate attestation server.

Generated image
Massively simplified process

We are now one step closer to being ready for the next steps of device configuration and enterprise application workloads. 🚀

This was a lot, so we’ll circle back next time to discuss TPM-based Credential Sealing with Measured Boot. Leave a comment if this is helpful and if you would like to see the code and the command by command breakdown for how this works. Cheers!

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #51: The "Ready, First" Pattern]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-51-the-readyhttps://brianchambers.substack.com/p/chamber-of-tech-secrets-51-the-readyThu, 08 May 2025 09:31:05 GMTThe Chamber 🏰 of Tech Secrets is open.

The “Ready, First” Pattern

The “X-Ready, X-First” pattern is one I’ve observed to be successful a number of times over the course of my career as major landscape shifts have happened in the technology industry.

Allow me to share a few examples.

Mobile-Ready, Mobile-First

The first major disruption to occur in my career was the proliferation of mobile devices.

History of iPhone 3GS: Faster and more powerful | iMore
Remember when the iPhone had a button?!

Like many others, I observed our organization go through a Mobile Ready phase. We had a bunch of things to do like…

  • Build APIs to serve data and functionality to apps

  • Ensure we had mobile-friendly authentication and authorization paradigms

  • Figure out how to deploy devices to public and private app stores

  • Build teams with the appropriate development skills

  • Learn to be successful with customer-facing mobile applications

During the “Mobile Ready” phase, we thought we might have a single “mobile team” that did all the things. As we moved towards mobile first, we realized we had to democratize the technology to maximize the opportunity.

A bit later, it was time to go mobile-first. While it is a bit hyperbolic (we still had good reason to build web apps), it did change the way we thought about what sort of experience to build for users of our applications. Mobile became a primary thought and often the default as consumer (and therefore enterprise) preference moved towards native mobile experiences.

Cloud-Ready, Cloud-First

Next was the cloud transformation journey, which started with a project that was literally named “Cloud Ready”. We had started our journey with cloud on two parallel paths: 1) What would come to be known as the CFA One mobile app and all its associated services and 2) our “Big Data and Analytics Platform”.

Remember these days? Source: AWS Blog

The cloud ready phase helped bolster these efforts while preparing the organization for larger scale. We had to account for things like:

  • What cloud provider will we use at scale?

  • What will our patterns be for all the things we had always done in a managed datacenter environment, including AuthN/Z, application deployments, databases, operational services, etc.?

  • We had to get comfortable with security posture in he cloud.

  • We had to solve all the typical stuff like logging, monitoring, tracing, billing, account management, account access with SSO, and a slew of other things.

  • We embraced both cloud and a DevOps model at the same time, so we had a lot to learn there too.

  • We rallied a lot of of EA, Cybersecurity, and other supporting teams around the cloud ready vision on a single new project, launched it, and then started to think about how to scale.

  • We really built our software engineering muscle and culture at this time since custom-written software made a lot of sense in our cloud strategy (vs just hosting COTS).

As you might expect, we followed the next year with a project named? Yes! You nailed it! 🧠 “Cloud First”. At that point, we shifted to fully embracing cloud as the default approach and have never looked back. We moved from a small handful of teams to nearly one hundred (some have come and gone and count in my number) over the following years.

The “-first” phase was—once again—all about democratization and default-approach transformation.

AI-Ready, AI-First

Today, we stand on the cusp of embracing the next disruptive wave of change: AI. Once again, we’ll run the same play, starting with AI-Ready. We’ll need to solve a lot of challenges, which we are starting now:

  • People: We’ll need to up-skill people and help them maximize the AI opportunity in their jobs.

  • Process: We’ll need to understand and document our business processes so we know where AI can play and where we need humans in the loop for compliance or other reasons (perhaps in the future, AI can only spend $XXX before a human validates?).

  • Data: We’ll need to have our data catalogued and ready for transformation and usage, whether that is for fine-tuning a model or injecting important, high-quality data into a context window or via RAG.

  • Technology: We’ll need to figure out all the tech for delivering AI, all the architecture patterns, and all the developer tools. We’ll need to double down on our API implementations so we can empower models and agents with MCP-fronted tools. We’ll need specifications, standards, frameworks, registries, and more.

Thankfully, we have a great team of people across the organization to make these happen, and we have the benefit of learning from our past.

At some point, we’ll be ready to turn our eyes to “AI-First”, a term I am already seeing in blog posts and LinkedIn on a regular basis.

Ready or not, AI-First is coming.

Takeaways

So what can you take from this anecdotal experience and apply in your company?

  1. Going straight to “first” is a bad idea. For major disruptions, it takes time to prepare, to build organizational support, to get funding and people, and to manage change. Embrace the “ready” phase and know that “first” is coming. Be patient. There is clear value in the readiness work (think of the AI examples above) and I think some degree of value in the anticipation of transformation that is ahead when we embrace “first”. In my career, the one time we jumped to a “-first” without a “-ready” was “API-First”, and it has not been the greatest success of my career. In hindsight, I think a patient readiness phase would have helped.

  2. Use “X-first” in your internal vision casting, but not your external communications. “What might it look like if we were AI-First?” is a worthwhile question to ask to help shape the readiness journey. In my experience, we did not jump to the “-first” until teams were effectively clamoring for it and it was an obvious next step. This restraint creates anticipation, buyin, and support when the first model is embraced and requires organizational change, which is honestly the hardest part of any job.

  3. Don’t overuse the “ready, first” pattern. It applies to major disruptors but not all changes. I would not recommend “SaaS-Ready, SaaS-First” or “Docker-Ready, Docker-First” or “Kubernetes-Ready, Kubernetes-First”. You get the idea. API-First might have been too small a change, which could be another reason for its’ lackluster transformational power.

  4. Know the value of the transformation you’re engaging in. For mobile, it was about access to the business anywhere and everywhere. For cloud it was about empowering teams to manage their own infrastructure through APIs. For AI, the promise is radical productivity and work product increase. Following this guidance will help keep you on track on the previous three points, and ensure that you can set a bearing, take some steps, re-orient, and repeat.

I hope this pattern serves you well in your transformational endeavors. Agree or disagree, drop me a note and let me know what your experience has been. Thanks for reading! 🙏

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #50: Living on the Edge Part I]]>https://brianchambers.substack.com/p/living-on-the-edge-part-i-establishinghttps://brianchambers.substack.com/p/living-on-the-edge-part-i-establishingTue, 06 May 2025 09:30:41 GMTThe Chamber 🏰 of Tech Secrets is open for the 50th time! A quick note of thanks to all the readers and commenters. I am grateful you take the time to read and engage. 🙏

Designing a Custom Certificate Authority and Attestation Server

Highly distributed and rapidly developed applications are wonderful… but they also open lots of new attack vectors for the proverbial “bad guys”. 🦹🏻

In the world of edge computing and the emerging AI-dominated world, the basics of security will matter more than ever. One of the most essential components of securing the modern internet is certificates, issued by certificate authorities (aka “CA”).

In this post, we’ll take a look at what certificates are, how they come to be and how they work in practice. Then we’ll explore why they are essential to edge environments, and how I designed a custom CA and attestation server for my personal edge lab. Let’s go!

🔐 Certificates and Certificate Authorities

First, we’ll ensure we have a baseline understanding of the certificate concepts and how they apply to security and identity in the era of the internet.

What is a certificate?

Certificates are built on Public Key Infrastructure (PKI) which uses asymmetric cryptography, a technique in which a public key is shared widely and an associated private key is kept secret.

A certificate is a digitally signed document that binds a public key to the identity of a system, device, or person. It is signed by an authority's private key, allowing others to verify the identity using the authority’s public key as a foundation of trust.

A typical certificate contains:

  • Public Key – The key used to establish secure connections.

  • Identity Information – Details like a domain name, device name, or organization.

  • Issuer Signature – The CA’s digital signature verifying authenticity.

  • Validity Period – Defines the window during which the certificate is valid.

  • Extensions – Additional fields like usage constraints or Subject Alternative Names (SANs).

What is a certificate authority?

A Certificate Authority (CA) is a trusted entity that issues and signs certificates. The CA vouches for the authenticity of the identity information included in the certificates it signs. This trust model applies to both public internet CAs (like DigiCert or Let's Encrypt) and private, internal CAs (self-signed) within organizations.

When a client—such as a web server or edge device—wants a certificate, it generates a key pair and sends a Certificate Signing Request (CSR) to the CA. The CA verifies the attributes in the request (through automated checks or manual validation), and if approved, returns a certificate signed with its own private key.

To protect its root private key, which acts as the foundation of trust, most Certificate Authorities (CAs) do not use it directly to sign end-entity certificates. Instead, they create one or more intermediate CAs, each with its own private key and certificate. The root CA signs the certificate of the intermediate, delegating trust. This layered model reduces risk while maintaining a verifiable chain of trust back to the root.

Now that we know how certificates get created, let’s look at how they work in practice.

🌎 How Certificates Work on the Web

When you visit a secure website like https://brianchambers.substack.com, the site presents a TLS certificate as part of the initial connection handshake. This certificate includes the domain name and a digital signature from a trusted public Certificate Authority (CA), such as DigiCert or Let’s Encrypt.

Web browsers come preloaded with a list of trusted root certificates, each of which contains the public key of a known and trusted CA. When the browser receives a website’s certificate, it performs a cryptographic signature validation using the public key from the issuing CA (which may be an intermediate certificate signed by a root CA) to validate the signature and build a chain of trust. If the signature is valid, the certificate has not expired, and the domain name matches, the browser proceeds with establishing the secure connection.

Monitor TLS/SSL: Certificates, Ciphers, Expiration and Spoofing | Exoprise
Source: https://www.exoprise.com/2019/07/29/monitor-ssl-expiration-spoofing-changes/

In short, the certificate proves the identity of the website and enables the browser to trust that it is safe to proceed with communications.

🖥️ How Certificates Work for Devices

Certificates work similarly for devices, such as edge compute nodes. The goal is to ensure that when a device connects to a network or service, you can trust who it claims to be.

Here’s how that typically works:

  • The device generates a key pair (public/private).

  • It sends its public key, along with identity metadata (e.g., serial number, device type), in a Certificate Signing Request (CSR) to a CA.

  • The CA issues a certificate for the device, signing its public key and identity data.

  • The device can now authenticate to systems, access APIs, or join secure networks—presenting this certificate as proof of identity.

In this model, the certificate is endorsing the device's identity (like its serial number, hardware ID, or assigned hostname). In some cases, this also attests to to the device’s measured boot state, which proves it booted into a known and trusted configuration. (more on that in a future post).

🤝 The Unifying Concept: Certificates Prove Identity

Whether it’s a website or a device, a certificate is doing the same fundamental thing: It cryptographically proves the identity of a digital entity and enables others to trust it.

If you just wanted to learn about the basics, you can sign off now. If you’d like to read about how I designed my own certificate authority and attestation server for onboarding edge devices, read on!

Building a Custom Certificate Authority

Certificates are an important construct for establishing trust at the edge. In order to do attestation—the process of ensuring a device is in a known, trusted, and unmodified state—we need an attestation server to and a CA to issue certificates.

In support of my Edge Lab project, I decided to write my own certificate authority, which will have authority over my local edge domain. This affords me the opportunity to shape a REST API around my edge device attestation requirements as well. This is a logical approach for an “enterprise” of my size (1 employee, $0 revenue) where the one employee is just trying to learn by doing. 😎

High Level Architecture

We’ll unpack this target architecture as we go.

System Design & Requirements

Here are the key system design attributes for the attestation server:

  • Goals

    • Establish a hardware root of trust using the device’s onboard TPM. We won’t get into these details until the next post, but our server plays a critical role.

    • Facilitate secure edge node onboarding and identity management using Public Key Infrastructure (PKI) patterns and certificates.

    • Enable measured boot at attestation time to ensure zero tampering has taken place and the device is in a trusted state.

    • Facilitate zero touch provisioning, meaning no human has to be present to complete any steps on the edge device.

    • Handle ongoing attestations, key rotations, and other necessary features.

    • Stop the device from participating in the edge cluster if it is tampered with, if certificates expire, or if it boots in an unexpected state.

    • Enable certificate revocation from edge devices or edge attestation servers when needed.

  • Requirements

    • Handle device attestation bundle inputs, which are subsequently validated, resulting in certificate outputs.

    • Serve root and intermediate certificates for dynamic signature validation. In other words, come to the attestation server if you need the certificates to validate something signed by the CA.

    • Sign intermediate certificates for edge site local CAs. Since we contemplate having 1..n clusters (I’ll have 2+ at my house over time), we need to be able to issue intermediates so that we can run local attestation servers (the ideal model, though those must bootstrap themselves by talking to our central attestation server).

    • Support deployment of cloud or edge environments.

    • Support certificate renewal using previously issued certificate and PCR + nonce combo (see Chamber of Tech Secrets #51).

    • The root private key for the CA must be air-gapped and stored completely offline. Intermediate private keys for cloud deployments will be stored in Google Cloud KMS (you could use AWS or another service as well). Edge sub-intermediate private keys will be generated in the TPM on the edge attestation server and send to the cloud attestation server for signature.

    • Allow configuration to determine where the attestation server looks for its signing key since it could reside in the TPM or in a cloud service (KMS).

  • Implementation Details

    • REST API implementation following Google API design patterns w/ OpenAPI 3.1 specification.

    • Implemented in Golang w/ bias for idiomatic practices.

    • PostgreSQL backend stores attestation records, certificate metadata, and device registration state.

    • Containerized deployment targeting Google Cloud Run (Cloud) and K3s (Edge).

    • Possible reverse proxy or API Gateway to offload concerns. Architecture TBD.

API Design 0.0.1

This RESTful API will cover our initial requirements but will likely undergo some iteration over the course of the project. I am sure I forgot something…hence the version 0.0.1 indicating an unstable API (< version 1.0).

*** revocation endpoint is not currently defined and will be added later.

Certificate Hierarchy

Finally, let’s recap how this certificate hierarchy works between cloud and edge.

🏠 Root Certificate Authority (CA) — Offline

☁️ Cloud Intermediate CA — GCP KMS

  • Private Key stored in GCP KMS.

  • Signs intermediate certs for local attestation servers (delegated intermediates)

  • Signs edge device certificates if they participate in direct-to-cloud attestation

🖥️ Local Attestation Server (Edge "Sub-Intermediate")

  • Has a private key generated & stored in a TPM on the local server

  • CSR is sent to cloud intermediate CA, which issues an intermediate cert

  • That intermediate cert:

    • Identifies this as an authorized signing authority

    • Includes name constraints or other policy constraints to limit scope. “These constraints may include name constraints, Subject DNs, or certificate policy OIDs that identify the delegated scope of each attestation server. For example:

      • OID = 1.2.3.4.5.100 for cloud-issued device certs

      • OID = 1.2.3.4.5.XXXXX for site-issued device certs, where XXXXX indicates our site unique identifier in integer form (.1, .4365, .50042, etc.)

      • In this case we’ll use the sample OID “1.2.3.4.5.XX” since we are not expecting to share our certs externally and do not want to bother registering for an enterprise OID arc with IANA.

  • The server:

    • Acts as the CA of record and signs CSRs from edge devices in its environment

    • Validates measured boot values

📦 Edge Devices

  • Each device generates its own attestation key in TPM

  • Sends a CSR to the local attestation server

  • Receives a cert signed by that local server’s TPM-held key

  • Uses that cert for secure identity and attestation

Conclusion

In this post, we learned how certificates get created and how they work as vouchers for a device or domain identity. We also learned how a Certificate Authority and Attestation Server are foundational building blocks for secure, scalable edge computing deployments, and reviewed the design specification for the one I’ll use in my edge lab.

So, what’s next?

  • I’ll finish implementation of the Attestation Server + CA and I’ll publish the code to GitHub for those that are interested. I have a 16 month old, so I am not as fast as I once was.

  • The next blog post in the series is about how to establish a hardware root of trust for a new edge node and implement measured boot.

  • I’ll implement an initial device config and the edge node code (hey, a rhyme!) that talks to our attestation server(s) on boot. This will get delivered via a minimal PXE boot when the device boots the first time.

  • I’ll write another blog post about how credential sealing works, which will provide another layer of security for our device, once again depending on our trusty TPM.

  • I have yet to mention networking, which is pretty important. I’ll write a separate post about that soon enough!

  • Then we’re on to the easy stuff… PXE booting a “permanent” image, K3s install, GitOps, observability, WASM, GPUs, radar IoT devices, and more.

Thanks for reading — this one gets into a lot of sticky technical design details and probably is tough reading, but I hope you stuck with it, learned something new, and tool the time to understand the concepts.

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Superpowers in the AI Age (and how to develop them)]]>https://brianchambers.substack.com/p/superpowers-in-the-ai-age-and-howhttps://brianchambers.substack.com/p/superpowers-in-the-ai-age-and-howTue, 22 Apr 2025 09:30:41 GMTThe Chamber 🏰 of Tech Secrets is open. Due to reader feedback, we’ll be abandoning the “Chamber of Tech Secrets #X” title and go to a real title for easier search in the Substack app and elsewhere. Title inversion begins now!


Superpowers in the AI Age

Superpowers are rare or unique characteristics—often attributed to heroes—that set them apart from the common man: incredible speed, the ability to fly, unworldly strength, teleportation, or a cool bat suit with utility belt. 😜

Superpowers also exist in the real world. With the advent of the AI age, some powers are bubbling to the top of the list in important. By cultivating them intentionally, we can set ourselves up for outsized success and drive incredible outcomes (maybe even save the world!).

Superpowers | Superpower Wiki | Fandom
Photo credit; https://powerlisting.fandom.com/wiki/Superpowers

Many Heroes Have Humble Beginnings

Have you ever felt “behind” in your area of expertise? Have you looked around at some of the elite thinkers in your domain and told yourself “there is no way I can ever do what they do… they know so much. They seem to have a superpower that I don’t”? Being around such people is a great learning opportunity, but it can also be intimidating and even demoralizing. I remember many times where I was talking with someone who bounded deep into the technical weeds and I wondered to myself “how do they know all this stuff?”.

The answer to this question has always been, in some form or fashion, “experience”. Not in the “x years of doing y” that pads resumes… the real kind of experience. Real experience is where the [for our purposes] technology has been rigorously tried and applied over many hours of time, often in substitute for sleep. Adamant determination to “figure it out”, rigorous trial and error, and thoughtful application of learnings builds something over time—expertise—that is hard to duplicate otherwise. I call the person that has all of this experience the “Tech Genius”.

Let me be clear — I am not the “Tech Genius”.

Have you ever had thoughts like…

  • “I could never start an open source project… people would just laugh at my code”.

  • “TPM, endorsement key, attestation key, measured boot, platform configuration registers… what exactly is happening here?”

  • “Models invoke tools via the Model Context Protocol (MCP) — think of it as USB-C for models”. [sounds simple, but the model doesn’t actually invoke the tool, so there ya go]"

  • “I could never make my own linux distribution”

I have good news. Regardless of the doubts you may have, the legacy barriers to learning are rapidly being removed.

[[ Enter AI ]]

The AI Mentor

Alongside experience, the next best way to learn is through great teachers and mentors. But the great ones are not equally distributed.

AI assistants — specifically via the chat (or voice) interface — unlock the potential for incredibly high-speed, high-quality learning via a digital mentor or teacher. If invited to do so, they can play a technical mentor role on topics of interest and take you down the proverbial rabbit hole as deeply as you desire to go. I believe this technique works extremely well for understanding how technical things work since that is, by nature, something that is already present in the corpus of human knowledge and is often well-documented.

There have always been a great resources available in the form of technical books or awesome resources like Kubernetes the Hard Way. What is absent in those is the ability to ask questions as you go, and the ability to shape the material in the way your brain works. This is where an AI mentor has huge potential to help. While AI experiences have a tendency to spew a lot of text, you can ask things like “lets walk through this step by step to make sure I really understand”, and then you can move at the speed of your own questions and processing.

Here are a few of my recent experiences:

  • AI as Teacher/Mentor: I wanted to understand every detail of how a device with a TPM chip can be securely on-boarded and trusted in a datacenter or edge environment. I wanted to understand every key exchange and certificate signing request. I wanted to understand how to validate that the machine was running the software and configuration expected (measured boot). I wanted to understand how the inbound device trusts the server that configures it. So I spent about 90 minutes one evening asking question after question until there were no remaining mysteries. My next step is to apply this to my Edge Lab project (starting soon!).

  • AI as Co-Architect: Last week, I was working on a personal project to build out a series of AI agents to help me with some financial investment research. I used a few AI assistants to help me think through my architectural design, challenge some of my ideas, and ultimately arrive at a better overall architecture. I am also working on a potential open source project that I built to help me with agent discovery. AI was a big help crafting and refining the design there, too. It offered me several feature ideas I didn’t think about on my own.

  • AI as Engineer: Sometimes there are small hurdles in your project that get you stuck from moving forward. While working on a project a few weeks ago, I needed OAuth, specifically Auth Code Flow. Rather than spending several hours scouring for open source projects and getting them running, I “vibe coded” a mock OAuth server that just did Auth Code flow in about 2 minutes and used that for testing (all local so no worries). There is much more to say about vibe coding in a future post.

The Superpowers

With AI being here, what do we need to bring to the table? I submit that there are three key characteristics that stand out as superpowers in this new era.

  • Curiosity: Those who are actively curious—those who ask questions until achieving understanding and who put the learnings into practice—will learn much faster than those who don’t. Technical knowledge has a compounding effect—the more you have learned, the easier it is to learn more. Said another way, your learning potential (considering the corpus of what is already known) will be capped only by your curiosity and resources you have available to you (most of us won’t be setting up clusters of Nvidia H100s). Rapid learning is one of the most important characteristics of a human in a rapidly changing world.

  • Creativity: AI models are not intrinsically creative. From : “… AI is really amplified intelligence rather than truly agentic intelligence, since it requires the creative prompt to get started”. My thesis is that AI will be most valuable to those who have an abundance of creative ideas for using it. I mean creative ideas very broadly: “How can I contribute to shaping reality?”. This appears in the form of questions, business ideas, open source projects, ways to automate workflows, and much more (yes, I did leave out images, music, video—also relevant).

  • Courage: We must do something with what we learn. We must create or we have simply entertained ourselves and occupied our time. It takes courage to reinvent oneself and to create.

Paired with a rapidly-improving AI complement, these three superpowers will propel us into shaping the future we want to see.


Putting the ideas to work

Okay, so what can you do with this information? A few suggestions:

  1. Start a list of topics you are interested in but don’t fully understand. Add to it often. Pick one topic from the list each week and work through it to the level of your satisfaction under the tutelage of your favorite AI assistant. I suggest just picking one at a time as it is easy to get distracted by too many new ideas (guilty!).

  2. Put ego aside. You may feel like you should already know something and feel a little funny about admitting to yourself that you knowledge is more shallow than you think it should be. Don’t get caught in the trap of thinking you are better than you are.

  3. Go beyond conceptual understanding to doing. Doing is where the experience will come from. Don’t be afraid of doing something wrong or having to start over many times. Doing will show you where the gaps in your understanding are. Don’t concern yourself with if the thing is already a “solved problem”… build it again yourself for understanding alone. Don’t forget take time to celebrate what you do and what you learn.

  4. Share what you build and what you learn publicly. Put the code on your GitHub. Write a blog post. Share the parts of your thinking that are still cloudy. Explain why you decided to take path X over path Y. The human feedback you get, even if it comes across as judgmental or “jerky” at times (and sometimes it will though most will be friendly and positive), will be useful to validate your learnings and refine your thinking.

  5. Develop the characteristics of curiosity (see above), creativity (change inputs and practice divergent thinking / steel manning), and courage (best learned by taking action).

Let’s wrap by recalling that the purpose of these superpowers are to accelerate learning—which is to say intelligence, since intelligence is simply the speed of applied learning (via Alex Hormozi)—for the purpose of doing something good in the world.

Pick a subject you’d like to understand, give this a try, and let me know how it goes in the comments or on LinkedIn. Happy learning friends!


Yes, real experience still matters. But you can close the knowledge and understanding gaps must faster than ever before by trying, failing, and fixing in a lower-stakes environment. No, you can’t blindly trust everything AI says, but it can be a very useful teacher and assistant. No, you shouldn’t vibe code up a critical business application and sling it into production if you have no idea what you are doing. The horror stories are already out there. Yes, it will still matter what we know as AI continues to “do all the things”. The nature of those things may change, but they will still matter. None of this matters if you don’t act on it.

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #48]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-48https://brianchambers.substack.com/p/chamber-of-tech-secrets-48Tue, 25 Mar 2025 09:45:48 GMTThe Chamber 🏰 of Tech Secrets is open. If you are wondering where the edge compute lab posts are, hang in there. I am still waiting on some new hardware and doing some research on design. Check out Edge Monsters if you’re hungry for that in the meantime. Today, I want to talk about job titles.


Chief Optimist

I have a new title on LinkedIn. I hope it shows up in everyone’s feed to congratulate me on. 😜

This feels like a good place to mention that it makes me happy that I don’t have a professional LI photo

I didn’t get a new job at a new company or a new role at my current company. Nobody told me to do anything differently than before, actually.

Internally, my title is the same as it has been for two years: “Sr. Director, Enterprise Architecture”. We don’t use words like “Chief”… we don’t have a CISO title even though we have a CISO role. But, I am the senior-most Enterprise Architect, which is typically called a Chief Architect… so I use that to make my job easier to understand for folks outside the company. No ego.

Now you know why “Chief Architect” was there… but about the Optimist Part?


Do titles matter? I don’t think they matter much…

Where they matter:

  • Communicating what you are likely responsible for to the outside world

  • Finding the right people to network with

  • I don’t know… that’s all I can think of except for…

  • How you think of yourself (in terms of the responsibilities you have and the attitude you bring)

The last one is the most important one, and this is is why I changed my title to Chief Architect / Optimist.


Since my role is “enterprise” (aka the entire organization) in nature, I see a lot of stuff. Most stuff. It is easy to get very hung up on the things that aren’t going particularly well or on decisions that trend in a negative direction or thing that are utterly irrational. If there are poor architectures, I usually see them. If there is organizational politics, I see it. If there is bureaucracy, I see it. If there is a lack of agility, I see it. If there is a failing project, I see it. You get the idea.

All of these things have the ability to create a downward spiral of pessimism, which is quite frankly a terrible place to find oneself on a daily basis and a poor position to stay in for any length of time, especially as a leader. Especially as a leader that wants to inspire positive change.

For this reason, I have decided to think of myself as the Chief Optimist for my company. Optimistic about what?

I am optimistic about the way that technology can impact our business and help us fulfill our mission, which is the best mission of any company I know. I am optimistic that we can build things that are great, run projects that succeed, use the best available tools and systems, manage our data, harness the potential of AI, and more.


Why did you put this on LinkedIn Brian? Well, sometimes you need accountability to help you get where you want to go. By making myself “Chief Optimist”, people are likely to ask me about it. Feel free to. Most importantly, by putting it out there as an idea, I force myself to think about it and internalize it… and here I am writing about it… so now I can act on it.


Yes, I know unbridled optimism is not that useful.

Yes, I know this sounds naive.

Yes, I know this doesn’t mean everything will now magically go according to patterns or plans and will still be messy.

Yes, I know this looks SUPER cheesy.

Yes, I know I can’t just pretend that everything is awesome all the time.

Yes, I know I have to acknowledge failures and I know I still have to meet people where they are.

Yes, I know this is not a real title (but how many LI titles actually are real anyway?)

But now I can face a challenging or frustrating situation with the mindset of “we are going to do this and do it well” and ask others to help me make that a reality. Instead of bringing “whats wrong” to the table, I’ll focus on the shared “right” that we are aiming towards. Instead of talking about whats frustrating around me, I can bring optimism to the table and we can figure out, together, how we can go forward.

I’ll be more helpful and less defensive. And those are better qualities in a leader. I want to get great stuff done. I love what technology can do for any and every business. Watch SpaceX catch a rocket or the pace of change in AI… there is every reason for techno-optimism. It is that attitude that I wish to embody.

The Chief Optimist, at your service. 🫡

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #47]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-47https://brianchambers.substack.com/p/chamber-of-tech-secrets-47Tue, 25 Feb 2025 10:01:46 GMTThe Chamber 🏰 of Tech Secrets is open.

Today I’ll share some of my insights from reading an internal memo from Beast Productions (production company of YouTuber Mr. Beast) that are immediately relevant and applicable to anyone looking to succeed in the business world, including in enterprise technology.

3 Lessons from the Mr. Beast Memo

Mr. Beast sends a memo to all his new employees since he doesn’t always get to spend significant time with them personally. Believe it or not, there are a lot of fantastic lessons for enterprise technology professionals within! I suggest reading the entire thing, but if that’s too much (it’s 36 pages, but some are pictures), you’ll at least get to digest a few of my takeaways here.

MrBeast to Host and EP Reality Competition Series 'Beast Games' on Prime  Video
Photo Credit: https://www.thewrap.com/mrbeast-reality-show-amazon-beast-games/

“What is your goal here?”

What is your goal here? Your goal here is to make the best YOUTUBE videos possible. That’s the number one goal of this production company. It’s not to make the best produced videos. Not to make the funniest videos. Not to make the best looking videos. Not the highest quality videos.. It’s to make the best YOUTUBE videos possible. Everything we want will come if we strive for that. Sounds obvious but after 6 months in the weeds a lot of people tend to forget what we are actually trying to achieve here.

Can you state what your team does this simply? Sometimes I struggle to do this. This reminds me how important it is to leave nothing open to interpretation when it comes to a team’s goal (or better, purpose). You are here to __________.

The most successful members of the team are obsessed with achieving the team’s purpose. If the purpose is clear, you make it possible for each individual to channel their passion into the right actions, and the team will move together. If the purpose is unclear or askew, the passion of each person will pull the team in different directions, and it will likely struggle or fail. Or the purpose will be confusing and won’t resonate, and individual passion will diminish.

As a leader, your responsibility is making the purpose of the team crystal clear. As a member of a team, it’s your job to make sure you understand the team’s purpose and bring your passion to it. Any failures here other than the individual bringing their passion to a clear purpose are the leader’s fault.


“Watch our videos”

If you’re going to be working for the Beast brand you should be a fan of the Beast brand. A lot of very valuable knowledge comes with watching vast amounts of our videos. I feel silly for having to write this but all the time I talk to 32 new people that have at most seen like 5 or 6 of our videos and it’s mind blowing that they don’t see a problem with that lol. As I said earlier I’d love for you to watch all the channels but especially the main channel seeing how you’re working on it. To get 60% up to speed I'd watch our last 50ish videos, if you’re a monster and really want to understand the history of the company and the innovations we’ve been through, I'd recommend you watch every video back until you hit the 10 million subscriber special. (anything before that is a waste of time in my opinion) If you’ve seen every video we’ve made since 10 mil then you will have a lot of context and information others don’t and it will make you that much much much more valuable.

Context. Context. Context.

If you know where we came from and how we got to where we are, you will be far better positioned to figure out which problems to solve in your work and then do them in a way that is aligned… and that will co-create a future we all want.

Context makes you valuable. Of course you must apply it.

What we have produced shows a lot about who we are.

How do people get context about our systems legacy (not to be confused with our legacy systems, though there is overlap)? Where is the “10 million subscriber special” in our history so that we can point people to the things worth knowing and understanding?

I have been thinking about this for my Enterprise Architecture team and have decided to add a “history lesson” segment to our monthly meetings so that we can have someone share about something we did in the past or currently have so that our team can be more equipped to apply those things (usually tools, platforms, applications) or understand why people who have been around a long time might be thinking a certain way. We probably won’t go back and spend a lot of time talking about the days where everything we did was built on top of an Oracle ERP (cause it doesn’t matter), but our last 5-7 years matter quite a bit (and much of our team is newer to our company than that). We’ll also work on expanding our shared understanding of the existing systems and integration landscape.

What we have made is a reflection of who we are… and if we want to help shape that in the future, we must first understand it.


Acting on Feedback

He will give you a list of things that you need to improve to become what we need and if you actually listen and master those things, we will give you a shot at the role [you want]. (only problem is most people think they are better than they really are and don’t take seriously when we give them things to improve and then wonder why they never move up).

Most people think they are better than they are and don’t take feedback seriously.

This reminds me of an Instagram post from Chadd Wright (former Navy Seal, ultrarunner, all-around beast)…

You are in one of three categories.

1. You are as good as you think you are.
2. You are better than you think you are.
3. You are not as good as you think you are.

Most people spend their entire life in category 3. The first 2 categories must be constantly proven…. Open competition is the litmus test. Make a commitment, train, show up, go hard and see where the chips fall. If you’re like me, you usually learn you’re not as good as you thought you were. That is what drives progress. It never ends. It’s painful. It’s humbling. It’s rare. It’s what sets true winners apart.

Ask for feedback from your boss. Ask your friends. Test yourself with challenges and give yourself feedback. Do after-action reports on your work, your workouts, or whatever else you do. It takes courage. Focus on what you can do to get into buckets 1 and 2.

Don’t get trapped ignoring or discounting feedback though. That is a sure way to end up in bucket 3. People in bucket 3 have a cap on their personal and career growth.


Bonus Takeaways

A few more from my notes…

  • Use the right level of communication for the thing you are trying to communicate. You didn’t communicate if you didn’t confirm they read it.

  • Nothing comes before your priorities! If you didn’t get your priorities completed, there is nobody to blame but yourself.

  • Don’t just stop because one person told you no, stop when all conceivable options are exhausted.

  • Consume things on a daily basis that help you get better at what you do – in his case its stupid media stuff to be clever and funny… in our case its things that help us understand tech, whats happening, and help us influence people better.


I hope you enjoyed these takeaways. Write something down and act on it. Go read the entire Mr. Beast memo and let me know what jumps out to you. See ya next week!

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #46]]>https://brianchambers.substack.com/p/the-future-with-agentshttps://brianchambers.substack.com/p/the-future-with-agentsTue, 18 Feb 2025 10:30:49 GMTThe Chamber 🏰 of Tech Secrets is open.

I’m traveling this week for work, but I want to share some thoughts I had over the weekend and see how they resonate. Once I get back home, it will be time to dig into the Edge Build series, so stay tuned for that.


Our Agentic Future

I saw two things during my Saturday morning reading that led to a bit of an epiphany for me (and I am probably way behind) as it relates to AI Agents.

Let’s start by diving into the two ideas individually, and then make the connection.

Idea 1: Triangle of Talent

I came across the “triangle of talent” from Shaan Puri. It looks like this…

Credit: Shaan Puri / https://x.com/ShaanVP/status/1887967867951530417

This post was pointing towards the minimal mix of talent levels based on organizational size. When you are an early startup, you need 100% of people in L5. As you grow, the ratio changes in Shaan’s opinion (and mine—you may need people who just complete tasks if you tell them WHAT/HOW/WHEN).

While my own version of this might be less blunt (“L1 - Useless” and “L2 - Task Monkey”) 🤣, I think the framework is strong and forces thinking about the strengths and capabilities of your own team. Do you have the right mix? Who has potential to move to Level 4 or 5 with more active coaching and mentoring?

Okay let’s continue to the second idea…


Idea 2: Three Observations

Shortly after, I read the blog post, Three Observations by Sam Altman. I’d recommend reading it, but in short, Sam makes 3 observations about AI.

  1. “The intelligence of an AI model roughly equals the log of the resources used to train and run it (training compute, data, inference compute)… It appears that you can spend arbitrary amounts of money and get continuous and predictable gains; the scaling laws that predict this are accurate over many orders of magnitude.”

  2. “The cost to use a given level of AI falls about 10x every 12 months, and lower prices lead to much more use”. I’m calling this “Altman’s Law”.

  3. The socioeconomic value of linearly increasing intelligence is super-exponential in nature.

In summary, AI is going to get smarter with more investment, it’s going to get cheaper, more people will use it, it is going to make things better. So the investment is going to continue.

Then he turned to the idea of agents [emphasis mine]…

“Let’s imagine the case of a software engineering agent, which is an agent that we expect to be particularly important. Imagine that this agent will eventually be capable of doing most things a software engineer at a top company with a few years of experience could do, for tasks up to a couple of days long. It will not have the biggest new ideas, it will require lots of human supervision and direction, and it will be great at some things but surprisingly bad at others.

Still, imagine it as a real-but-relatively-junior virtual coworker. Now imagine 1,000 of them. Or 1 million of them. Now imagine such agents in every field of knowledge work.”

Altman’s vision of a world filled with AI agents—thousands or millions of them working as junior assistants—raises an interesting question… How do we ensure these agents are actually useful? Done poorly, we risk creating thousands of inefficient, poorly managed agents rather than a true productivity revolution, and the potential for a management nightmare.

This leads me to my weekend idea…

Interlude: What is an agent anyway?

Since we are going to talk about agents, we should be clear about what they are.

An AI agent is a system that autonomously takes actions toward achieving a goal within a defined environment. It typically…

  • Receives input (natural language, structured data, etc.)

  • Processes the input using reasoning, planning, and memory

  • Acts on the world by making API calls, triggering workflows, or interacting with other agents/humans

  • Iterates and adapts based on feedback

Agents are software. And to be an agent, it must have agency.

Idea 3: Triangle of Agents

Now that we agree what an agent is, we can connect the previous two ideas together, and reveal the “Triangle of Agents”, which I propose as a way to think about the capability of agents and their usefulness in organizations.

Behold! The Triangle of Agents! 🙇‍♂️

Yes, you can make fun of my imperfect triangle. It is harder to make a segmented triangle in miro than I expected! If only I had an AI Agent to do it… 🤔

Let’s unpack each of the triangle levels…

Level 1: Brittle and Useless

These are agents that require extensive supervision and frequently fail to complete assigned tasks. Many enterprises will produce Level 1 agents in the short term as they experiment with AI, often driven by cost-saving initiatives or hype rather than robust strategy.

Many of these agents will be based on strong business ideas but will fail due to poor integration patterns and architectures, such as over-reliance on point-to-point integrations or brittle workflows that require direct database manipulations. These fragile solutions will ultimately add little value, frustrating both customers and employees.

Examples include things like customer support chatbots that step through a complex customer support journey but fail to resolve the issue for the user due to something that could have been discovered at the beginning OR A workflow automation tool that crashes when encountering unexpected data formats.

Level 2: Task Completers

“Tell them what, how and when and they will perform a task”.

Task Completers are highly structured and valuable for automating routine workflows, but they lack adaptability and independent problem-solving capabilities. They are good at understanding well-defined user instructions and execute them reliably while functioning within strict operational boundaries. This is great for repetitive, rule-based tasks with precision.

Examples could include an AI-powered scheduling assistant that books meetings based on predefined rules or a customer service bot that processes refunds following a strict workflow.

Level 3: Problem Solvers

“Tell them what to do, and they will figure out how.”

Problem-solving agents bring a major leap in AI capability. They not only execute tasks but also determine the best approach to completion. They dynamically adjust their methods based on context, even if the specific steps weren’t predefined. These agents are particularly effective in augmenting human workflows and increasing productivity.

In the near term, these agents will likely operate with human-in-the-loop oversight to build trust, but they may eventually work autonomously in many cases.

These agents are defined by their ability to adapt to new or slightly ambiguous situations without explicit step-by-step guidance. They utilizing reasoning capabilities/models to optimize task execution, and then perform multiple steps independently within a given goal framework.

Examples could include a coding AI that not only suggests fixes but also runs tests and deploys patches or an AI-powered research assistant that autonomously gathers, summarizes, and analyzes relevant information for decision-making

Level 4: Systems Thinkers

“Tell them the problem, and they will figure it out using people and processes.”

Systems Thinking agents move beyond individual task execution and begin to coordinate across multiple teams, processes, and systems. These agents don’t just complete tasks; they identify dependencies, optimize workflows, and orchestrate both human and AI resources to solve complex business problems.

For these agents to function effectively, enterprises need to invest in some foundational components

  • API & Event Registry: What APIs and enterprise messages are available? What do they do? How do you authenticate to them? How do they translate into business terms?

  • Agent Identity Management: Defining least-privileged authentication, authorization, and resource access for AI agents

  • Agent Registry: Repository of existing agents and their respective capabilities

  • Organizational Structure: who is responsible for human approvals? Who are the people to escalate to in the event of an issue?

Systems Thinkers are defined by their ability to coordinate multiple AI agents and human workers to achieve a higher-level business goal. They can learn to understand an organizations available resources and apply them to making decisions that impact multiple systems or business areas / departments.

One example would be a business operations AI that optimizes workforce distribution, supply chain logistics, and inventory management in real-time.

Level 5: Artificial General Intelligence Agents

“They identify the right problem and get it solved.”

AGI agents are autonomous decision-makers capable of identifying opportunities, predicting challenges, and proactively solving problems. Unlike Level 4 agents that require a clearly defined problem statement, AGI agents determine what needs solving and take action accordingly.

This level requires the highest degree of autonomy, reasoning, and context-awareness. Full realization of this level is likely years away (if ever).

It is defined by the ability to learn and maintain a near-complete context of an organization and then apply reasoning to prioritize impactful problems, research solutions, provide recommendations, and farm out action.


People and Agents

For most organizations, the lowest hanging fruit is implementing Level 2 and 3 agents, with some possible opportunities for Level 4 emerging. This will be a massive productivity boon for adopters.

We must remember that all of these agents will do what they are asked within their capability (which will increase) and permissions set (which is probably poorly managed today).

This means that agents give us the potential to complete massive numbers of tasks very quickly, which is great. It also means that agents give us the potential to create massive problems — at scale — very quickly if we ask for the wrong things, which is not great. You will get what you ask for, so the context (aka data) you possess in your head and your ability to translate need into agentic action will be extremely important.

The skill of asking for the right things in the right way will become increasingly important. So… the era of Agentic AI may mean we need more Level 4 and Level 5 people on our teams, farming tasks to Level 2 and Level 3 agents. A Level 5 human paired with Level 3 and Level 4 agents will have massive productivity potential.


What is Brian doing today?

  • Personal: Use reasoning models to research things quickly — I run a lot of ideas through Google Gemini thinking models for personal purposes. I allow thinking models to have agency over creation of a report (say on market research for investment purposes) or research output of an idea (best approaches for bootstrapping an edge node) and then I digest it to make decisions.

  • Personal: I use multiple language / thinking models in parallel to refine and enrich blog posts. I don’t write these things by myself anymore… I have a team of AI friends from OpenAI and Google Gemini helping me flesh out my ideas and enrich them with context and examples, and they work for free! The agency is over a draft which is followed by my feedback and refinement… but it would be fun to set up a game of pinball between the two and see what they create (and how the loop ends).

  • Work: Learn as much as I can about agents, agent-frameworks, and agent-tangential opportunities (especially in knowledge/system/api discovery and agent idm). I have yet to endow agency to anything work-related, but hopefully opportunities come soon!

Conclusion

Now we have a decent model for thinking about the usefulness and productivity of AI agents, and some ideas on how they might be able to help us at each level.

Remember: People are still going to be critical!

Asking for the right things in the right way will matter more than ever. Leadership will matter more than ever. Having a large context will matter more than ever. Coaching people to develop themselves into context-equipped Superstars (Level 5) will matter more than ever.

The organizations that win will be the organizations that harness the power of AI at Level 4 (and maybe 5), but to do so, they will need Level 5 leaders.

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #45]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-45https://brianchambers.substack.com/p/chamber-of-tech-secrets-45Tue, 11 Feb 2025 10:15:53 GMTThe Chamber 🏰 of Tech Secrets is open.

So much is going on in the tech world right now. DeepSeek R1 launched and destroyed public markets about two weeks ago. Markets bounce right back as it looks like DeepSeek was innovative, but still expensive ($1b estimated TCO). Google’s earnings report disappoints investors, but they continue forward with a plan to invest $75 billion in AI and Cloud infrastructure. Wow. Gemini 2.0 launches as well and I am interested to get my hands on Deep Research. OpenAI released OpenAI Deep Research, their reasoning model. Palantir (PLTR) rides the AI wave with continued earnings smashes and rockets 🚀 to all time highs. Lots of AI everywhere!

At the same time, the blocking and tackling of IT remains critical to business success and transformation. And, as I have always thought, many of the critical and data-intensive apps are going to continue their move towards the edge. And that will be our topic today.


Edge Computing

In 2018, I was deep in the weeds of our first iteration of Edge Computing in my architecture role at Chick-fil-A. I stumbled into a new frontier, applying lots of new approaches to solving edge problems (like GitOps) while taking inspiration from the past (like using commodity hardware from Google’s early days). Over the past 7 years, I have shared a lot about the approaches we took to build out a large-scale edge deployment and what I learned from the experience (K3s clusters, zero-touch provisioning, GitOps, importance of automation, support models, telemetry management, edge ingress, and more). You can find all of those on the wonderful Chick-fil-A Tech Blog.

However, the time has come to drop the assumptions of the past and build a new Edge environment using the best from my lessons learned and by leveraging the many architecture patterns and open source projects that have developed since, including some great thinking from my friends at Edge Monsters.

In this series, I’ll be going from zero to “production” with a new edge lab environment in my home. Expect posts on this topic to be scattered across the year as I build the core and then add other capabilities to my environment to mimic real world use cases that I find interesting.

I’d love for you to follow along if you are interested and work through the process with me — it will be a great learning experience for all of us. For those that want to follow along, I may create some sort of collaboration forum to share progress (likely a free slack team).

Gear List

The gear list is likely to evolve over time as the lab builds out to include more components… or I find that something in the list was a mistake, but to start we’ll need the following…

  • Raspberry Pi 0 ($39.98) — this will serve as a sidecar to our environment and provide cluster provisioning services, help establish root of trust, perform cluster management, possibly PXE boot nodes, hold a docker image repository, and do other things that serve the edge environment standup and management processes. I haven’t tried any of this, so some parts may not work or require additional components. All part of the fun!

  • Intel NUC— of course I’ll be using an Intel NUC from my friends at SimplyNUC. Why would I use anything else? I’ll start with a single node to keep things simple, but am likely to add more nodes in the future.

  • Ubiquiti EdgeRouter X ($134.99) — we need a separate LAN for our lab environment, and I wanted some programability and automation capabilities via CLI or REST API. This router comes highly recommended from an Edge Monster amigo.

Bonus Item / Substitutions:

  • Raspberry Pi 5 Starter Kit Pro ($159.99)— I have one of these to play with a heterogeneous cluster, introduce an ARM chip into a clustered architecture, or to replace my Pi 0 if i find that I need more resources (hopefully not). This is also another option if the NUC is too expensive for your lab environment, but there will be some differences as we progress as my primary computing device will be using an Intel i7 x86 processor, and the Raspberry uses ARM.

I wanted this image here, but it’s a NUC… not a Raspberry Pi 5

Development Environment

I’ll be using the following to build out my components. If you aren’t a developer and want to follow along, I’ll probably make my code available on GitHub with instructions on how to run it (maybe a little delayed from the blog posts but will do my best). Here’s what I’ll be working on…

  • Macbook Pro 2021 version—not a requirement but just lets you know what I’m using.

  • An AWS Account—I will be sending data from my edge environment to the cloud, and using the cloud to aid (minimally) in the cluster registration process. It will be important to see the status and health of our cluster over time. We probably won’t exceed the free tier but just be aware that we might over time (or if you want to extend and run something bigger).

  • Golang environment—on your laptop, a VM, or whatever else you prefer. Golang will be the language of choice for this project.

  • Ansible — I’m thinking about using ansible to manage all of the edge configuration

  • Other stuff — lots more things will be necessary. Ethernet cables for connecting to our network. Monitor and keyboard for initial pi config / testing. We’ll likely play with different approaches to managing and deploying apps (K3s, nomad, etc.). I haven’t thought this all out yet, so we’ll add dependencies as we go while keeping them minimal.

The Application(s)

What sorts of applications will make sense to run in my house? I don’t know yet, but here are a few thoughts…

  • IOT — I am thinking I’ll get some sort of IOT sensors as part of the solution and run some sort of framework at the edge. This probably means we’ll need a gateway device eventually that can talk protocols other than wifi, but that will come later. We have a lot of work to do before that.

  • AI — At some point, I’ll likely do something with cameras (another component to get unless Blink has an API I can use.. prob not?) and something with an LLM just for fun.

  • Controlling APIs — I’ll probably come up with some edge-deployed APIs that can control things in my home… maybe the pool management? Will have to see what sorts of programability exist and what is most interesting.

  • Inventory App — while it doesn’t need edge, maybe an inventory app that lets me increment and decrement certain supplies so I always know what I have? This could be useful in a lot of places around the house.

  • What application would you like to build? I am open to ideas, so respond in the comments or reply to this email (if you’re reading via an email client).

Many of the things I am thinking could be achieved direct-to-cloud, but the point here is to exercise the edge development muscle, and to do that and learn from it, we’ll want some apps. So share your ideas.


I’ll be using Google Gemini 2.0 services as a friendly assistant to help me work through unexpected challenges and will share how that goes as well.

So that’s it! We’ll get started in the next few weeks when all of the hardware arrives and once I work through some initial challenges, so stay tuned for the next post in the Chamber 🏰 of Tech Secrets Edge Build Series.

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work, and take some time to share with a friend.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #44]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-44https://brianchambers.substack.com/p/chamber-of-tech-secrets-44Thu, 06 Feb 2025 10:30:49 GMTThe Chamber 🏰 of Tech Secrets is open. Today the world of software and systems engineering meets self-help as we talk about resilience in systems and in life.


Resilience is a common theme in systems engineering. I like the definition: “the ability of a system to maintain an acceptable level of service despite failures, challenges, and unexpected conditions”. This is accomplished through architectural and engineering techniques that anticipate, respond to, and recover from failures.

Those with a systems background will be familiar with many of these techniques including monitoring for early detection, elasticity, scaling, backups, hot-standby environments, regionally distributed workloads, rapid recovery, failovers, and the like.

When failures do occur, we are typically very intentional about learning from them, employing tactics like blameless postmortems.

I often like to look at metaphors in the world and bring them into tech, but today let’s go the opposite direction.


A few months ago, NVidia CEO Jensen Huang gave a talk in which he said a few very interesting things about resilience. Give it a watch.

One that stand out to me: “I don’t know how to teach it to you except for I hope suffering happens to you.”

Is Jensen just a meanie? No. He understands that adversity, challenge, frustration, struggles, hardships, and failures are simply part of life, but also are often the very best of teachers. More so, they are the catalyst for improvement and change.

I think this is an important lesson to take to heart in the business world. In my experience, the higher level your role, the more you have to influence and align with others. As your role expands, there are by nature more groups you must align with to be successful. This means things often won’t go your way.

How will you respond? Do you choose to live in frustration or resign yourself to giving a lesser effort? Or do you choose to treat each work frustration or failure to influence as a chance to self-assess, refocus on what is most important, and get better?

Do you choose to give up, or are you resilient?

Here’s 3 tips from systems that may help you choose resilience.

  • Invest in resilience: In software and infrastructure, resilience isn’t the default—it requires deliberate investment in redundancy, failover mechanisms, and recovery processes. The same is true for personal resilience. You have to invest in habits, relationships, and mental frameworks that help you recover from setbacks. One example would be Failure Testing. Just as engineers run chaos experiments to test system stability under stress, you can proactively put yourself in controlled challenges—hard workouts, “scary” activities, new challenges, learning new skills—to build resilience. Personally, I try and have a challenge every 2 months that gets me out of my comfort zone, pushes me in a new area, or forces a beginner’s mindset. Just for fun:

    • January / February - Try Brazilian Jiu Jitsu (first drop in today, actually)

    • March / April - Run a sub-6 minute mile. I want to see 5:XX on the clock. I am not sure if I’ve ever done that.

    • May / June - Hyrox, Tactical Games

    • July / August - 50k run / hike on the Appalachian Trail in GA (6500k vertical up/down)

    • September / October - Climb Grand Teton from car-to-car in a day (7k vertical up/down), possible solo backcountry elk hunt

    • November / December - TBD

  • Observability: In systems, we have lots of tools that create data that we can use to understand what is taking place in near-real-time so that we can quickly react to issues. Do you have any data about you? What does nominal and optimal performance look like at work, in your health and fitness, and in your relationships? Do you have a vision for what you want life to look like? Goals? Do you have any time to reflect on those and measure if things are on par or not? Personally, I like to have goals for each month and have a weekly and monthly review process I go through. My wife and I also write down answers to ~10 questions each month about how things are going between us to ensure we’re communicating about what’s important. This creates data and a chance to reflect.

  • Blameless post-mortem: In software, teams often conduct “blameless post-mortems” or retrospectives to look back on an incident or outage and see, objectively, what can be done better in the future to avoid a repeat? Take the time to analyze your decisions and resulting outcomes. Don’t blame others and don’t blame yourself (I always fail at these things, nobody listens to me, blah blah blah). I often use a simple set of questions:

    1. What went well?

    2. What didn’t go well?

    3. What did I learn?


Being resilient does not mean pretending all is well. You can limp along through a struggle and still be resilient if you are operating at “an acceptable level of service”. Software outages hurt. Suffering in life hurts. We can and must choose resilience. What will you do when you experience failures, challenges, or unexpected conditions? Resilience is a choice that takes investment, and the choice is yours.

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #43]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-43https://brianchambers.substack.com/p/chamber-of-tech-secrets-43Tue, 14 Jan 2025 10:01:03 GMTThe Chamber 🏰 of Tech Secrets is open.


A CR, or Continuing Resolution, is a type of legislation passed by the U.S. Congress to fund federal government operations temporarily when the formal appropriations process has not been completed by the start of the fiscal year. It allows government agencies to continue functioning at current or specified funding levels for a limited period, preventing a government shutdown.

The U.S. Congress has relied on CRs to fund government operations in every fiscal year since 1997. This means that for over 28 consecutive years, at least one CR has been enacted annually to prevent government shutdowns due to delays in passing regular appropriations bills. Between fiscal years 2010 and 2022, Congress passed 37 CRs.

Continuing Resolutions are the proverbial “kicking the can”. This has minimal perceived short term consequences, but long-term impact in the form of a growing national debt that effects everyone.

Source: US Treasury

In 2024, 17.9% of US government revenue (income from taxes) went to servicing this existing debt. When compounding works for you, its magic; when it works against you, its merciless.


The organizational equivalent of national debt is what I call “Enterprise Debt”.

Enterprise Debt is the accumulated cost or burden of suboptimal decisions in an organization’s enterprise architecture.

What does your organization’s equivalent to this chart look like on the technical side? Do you operate with a CR mindset?

Some examples of Enterprise Debt:

  • Silo Debt: This project has to get done so we take the path of least resistance on everything, including integrations, and create a reality where anyone who needs to consume anything from us is going to have to wait for something point-to-point to be developed for them in the future. The data and business capabilities in this area are effectively hostages that need to be rescued by name on a project-by-project basis.

  • Short-term thinking Debt: We could move this application to the cloud, but it feels risky, so we’ll just upgrade in place and leave that cloud problem to someone else in the future. <One year later> This application went to end-of-life on-prem and is going to be offered cloud only… begin migration to cloud!

  • Data Management Debt: We need tons of people and money and tools to locate, understand, catalog, manage, transform, and re-share data because we didn’t make this a priority as we designed and delivered our systems landscape in the first place.

  • Micro Perspective Debt: We are having challenges with thing X, so we buy a system for that. Then we have a challenge with thing Y so we buy a system for that. Thing Z… repeat. We don’t like Part A of thing X anymore so we buy system XX but don’t retire X because we still need function B and C from there. We never pause to take a holistic view of a domain and revisit how well technology systems are supporting the business.

  • Undisciplined Pursuit of More Debt: Using this tool is hard, and at my last company we used that tool and it was easy. Buy, implement, repeat.

  • Institutional Knowledge Debt: We haven’t touched those systems in ages… in fact I don’t think anyone who worked on those even works here anymore.

  • No Docs Debt: We don’t have time to make that architecture diagram. We don’t have time to register our APIs in the developer portal. We just have to launch this thing.

As you see, Enterprise Debt can be the result of short-term thinking, difficult tradeoff decisions, real constraints, or rapid growth (people, processes, technology — could come through the business growth or M&A).

Impacts

Consider the Iron Triangle in software: Fast, cheap, or good? Pick 2.

Enterprise debt tends to make things slower, more expensive, and more risky (Cybersecurity in particular) over time. That often results in a compounding effect with more outcomes that are “we chose fast but it was slow” and “we chose cheap but it was expensive”.

Let’s talk about AI. Organizations with brittle systems, data or integration landscapes are very unlikely to succeed with AI. How will the mighty AI help you operate when your operations depend on systems the AI can’t talk to?

Ending the Cycle

Ending the CR cycle in the US will either take heroic leadership or the occurrence of some catastrophic financial event. A mindset has developed that says “this is normal”… “this is ok”. “This is how we operate”. We even celebrate the passing of each CR as a major victory of bi-partisan government.

What can we do to change the Enterprise Debt mindset, if it is present in our organization?

  • Stop the debt growth: The US has an issue with paying down the national debt because the annual deficit continues to grow year-over-year. Is your Enterprise Debt growing each year? Consider processes and solutions that can keep the debt from growing.

  • Understand where the debt is: Map the organization and catalog where Enterprise Debt is present.

  • Select areas of debt that matter: Use the beautiful map to determine which debts have the biggest current or potential impact. Being selective is key to building trust and credibility; nobody trusts the person who acts like everything should be perfect. Help identify and execute paths forward using architecture techniques.

  • Tell the story: Most of the work is here. Tell the story about the perils and impacts of the Enterprise Debt deficit / debt. Do it over and over again until people understand. Most likely, nobody is trying to destroy the company’s systems landscape with bad decisions, so help them see the impact.

  • Help: Can you help make some of the current challenges better by rolling up your sleeves and creating a shared win?

What to do when they don’t listen to you? Repeat the above. Hope and pray. 🙏


Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #42]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-42https://brianchambers.substack.com/p/chamber-of-tech-secrets-42Tue, 07 Jan 2025 12:50:04 GMTThe Chamber 🏰 of Tech Secrets is open! In this post, we’ll look at an “emerging technology” that is not new at all, but rather re-emerging. That would be Nuclear Energy.


The Surging Demand for Electricity

Technologies like cloud computing, AI, and Electric Vehicles are driving increased demand for energy… electricity in particular. While residential energy usage is and has been experiencing marginal growth over time, the demands of these new and emerging technologies are challenging our capacity outlook, and sometimes current capacity. Add a potential comeback of American manufacturing (driven by attempts to decouple from China and/or tariff policies) and increased humanoid robotics or automation and we could see even larger increases in demand in the near future.

Cloud and Data Center energy usage — driven by AI — is on the rise, reaching 4.4% of total US energy consumed in 2024 (176 TWh). If current trends continue, this could reach upwards of 12% of current total energy capacity by 2028.

The Competitive Landscape

For the United States to remain competitive on the global stage (cost of goods, technological innovation, application of technology to more verticals), it will be extremely important to have competitive energy prices in the US.

Based on a fairly recent International Monetary Fund (IMF) paper, my assistant was able to put together the following chart showing electricity cost per Kilowatt hour by country, backing out government subsidies as best as possible since the goal here is to explore the real costs.

As you can see, China currently has the lowest energy cost per kilowatt hour and is poised to dramatically improve that position in the coming years with twenty nine nuclear reactors currently under construction, as of January 2025.

Electricity in the US currently costs nearly double that of China, even before these new reactors come online.

Why does this matter?

As we inferred above, electricity is a key component of many of the emerging technologies that are predicted to dominate the future. A cursory look at most “what to watch in 2025” lists will point you to the continued AI revolution (electricity), Electric Vehicles (charged with more electricity), humanoid and other robotics (running off electricity or batteries charged by electricity), cryptocurrency or stable coins (common complaint is they use too much… electricity), automated manufacturing (electricity), and of course “computing everywhere in everything” (electricity).


The Historical Fall of Nuclear

Nuclear power currently is rapidly emerging as a “no-brainer”… but it’s hardly a new technology. How did we get here?

Once seen as the energy of the future, nuclear power fell out of favor over the years due to a mix of safety concerns, high-profile reactor accidents, political shifts, and a general public apprehension. The most familiar incidents—Three Mile Island, Chernobyl, and Fukushima Daiichi—were significant in shaping public perception. These accidents, coupled with the challenges of high construction costs, extreme regulatory complexities, long-lasting waste management concerns, and NIMBYism (Not In My Back Yard), led many countries to pause or scale back their nuclear ambitions, including the United States.

These high-profile incidents occurred with older-generation reactors, specifically Generation II designs, which, while incorporating certain safety improvements, lacked the advanced features seen in more modern designs. Here is a recap of those narrative-changing events over a ~30 year period.

  • Three Mile Island (1979, USA): A partial meltdown occurred due to mechanical failures and operator error, leading to the release of a small amount of radioactive gas. Although contained, this event sparked widespread fear and led to major regulatory reforms in the U.S. nuclear industry.

  • Chernobyl (1986, USSR): This catastrophic explosion was caused by a combination of flawed reactor design and inadequate safety protocols, resulting in the most severe nuclear disaster in history. Chernobyl released a massive amount of radioactive material into the environment, affecting large parts of Europe and heightening global fear of nuclear energy.

  • Fukushima Daiichi (2011, Japan): Triggered by an earthquake and tsunami, this incident led to multiple reactor meltdowns and released significant radioactive material. It exposed vulnerabilities in nuclear plant designs for natural disasters and prompted a worldwide reevaluation of nuclear safety standards. Many countries begun discussing plans to deactivate reactors and move to different energy sources after this incident.

These incidents underscored the potential risks of nuclear power and led many countries to prioritize renewable sources or rely more heavily on natural gas and other fossil fuels. Some countries stopped new nuclear projects, while others shut their existing reactors down systematically (see Germany).


Recent Resurgence

Very recently, nuclear energy has started to make a resurgence. As we mentioned, China has embraced nuclear technology and has 29 nuclear reactors under construction currently. This may lower their cost per kilowatt hour to as low as $0.0165 USD when the projects are complete.

Two new GenIII+ nuclear reactors were put into service at Plant Vogtle in Georgia in 2023 and represent the first new nuclear energy generation in the US since 2016 when the Watts-Bar GenII reactor came online in Tennessee. Currently, there are no other projects that have started construction. There are a lot of plans, though.

As of Jan 3, 2025, three advanced reactor developers are working towards submitting construction permits. From the US Department of Energy:

  • TerraPower plans to build its sodium-cooled fast reactor and molten salt energy storage system near a retiring coal plant in Wyoming.

  • X-energy plans to build a four-unit, high-temperature gas reactor plant at Dow’s Seadrift manufacturing site in Texas.

  • Tennessee Valley Authority (TVA) also plans to submit a construction permit application by late summer 2024.

New Developments: Gen IV Reactors

A big part of the resurgence of nuclear is the modern reactors. China just placed the first operational Gen IV reactor into commercial service in late 2023 and many refer to these as “meltdown proof".

Gen IV reactors rely on natural physical laws (like gravity or convection) for cooling and shutdown, reducing the need for human intervention or complex mechanical systems. Many designs are "walk-away safe," meaning they can safely shut down without human action even in the event of a total power loss. Some of the designs, such as MSRs, use coolants that operate at lower pressure, minimizing the risk of catastrophic failure or core meltdown.

Some designs, like fast reactors, can "burn" nuclear waste from earlier reactors as fuel, reducing the amount and longevity of radioactive waste. This also results in a shorter half-life for the waste byproducts, meaning less need for long-term waste storage.

Another interesting concept in Gen IV designs is “Breeder Reactors” which can produce more fuel than they consume.

Unlike traditional reactors that use water, Gen IV reactors employ coolants like Helium, liquid metals, or molten salts. These coolants allow operation at higher temperatures and lower pressures, improving safety and efficiency.

There are six designs that are considered to be under the umbrella of Gen IV reactors:

  1. Sodium-Cooled Fast Reactors (SFRs): Use liquid sodium as a coolant, operate at high temperatures, and can recycle nuclear waste.

  2. Molten Salt Reactors (MSRs): Use molten salt as both a coolant and a medium for fuel, offering exceptional safety and fuel flexibility.

  3. Gas-Cooled Fast Reactors (GFRs): Use helium as a coolant, allowing very high-temperature operation.

  4. Lead-Cooled Fast Reactors (LFRs): Use liquid lead for cooling, offering high safety and efficiency.

  5. Very High-Temperature Reactors (VHTRs): Designed for hydrogen production and industrial heat applications

  6. Supercritical-water-cooled Reactor (SCWR): Uses supercritical water as both coolant and working fluid to achieve higher thermal efficiency, simpler design, and reduced waste, while operating at extremely high temperatures and pressures. Supercritical water is water that is heated above its critical temperature (374°C or 705°F) and critical pressure (22.1 MPa or 3,212 psi), at which it exists in a unique state that is neither a distinct liquid nor gas but exhibits properties of both.

And I thought software people loved acronyms…


Small Module Reactors (SMRs)

Of particular note are Small Modular Reactors (SMRs). SMRs are generally smaller in size and capacity (see photo from Oklo below), producing 10-300 MWe per unit as opposed to traditional reactors that product 1,000 MWe, or more (for the math averse, that’s 10-30% of the output).

Oklo to deploy two Aurora plants in Ohio -- ANS / Nuclear Newswire
Oklo Aurora Powerhouse — it looks like a modern version of a cabin

As the name implies these reactors are also modular, which enables off-site construction and transportation to the installation site. In theory, this will keep standard designs (easier approval) and allow modular scale-up in the future with as demand grows in a given area, and all at a more manageable cost. This is in contrast to previous reactors, which are effectively snowflakes ❄️ due to unique considerations at each site.

There are several companies developing SMRs including and Kairos Energy, NuScale Power ($SMR), X-Energy, and Oklo ($OKLO).

Since many technology readers will be familiar with Sam Altman of OpenAI, I’ll mention his investment in Oklo, a company working on an SMR project called the Aurora Powerhouse (pictured above).

The Aurora is a compact, liquid-metal-cooled fast reactor designed to generate between 15 to 50 megawatts of electrical power. Notably, it can operate for up to a decade before requiring refueling, enhancing its reliability and reducing operational costs. The Aurora utilizes advanced fuel recycling technologies, allowing it to use nuclear waste as fuel, thereby addressing waste management challenges. The first commercial reactor is planned at the Idaho National Laboratory, beginning operation in or by 2027.


Big Tech Plans

Big Tech companies see where the puck is headed and are already skating there.

  • In September 2024, Microsoft entered a 20-year agreement with Constellation Energy (CEG) to purchase power from the planned reopening of the Three Mile Island Unit 1 reactor in Pennsylvania. This initiative aims to supply Microsoft's data centers with carbon-free nuclear energy.

  • Google's agreement with Kairos Power involves developing multiple small modular reactors (SMRs) to supply clean electricity to its data centers. The first reactor is expected to be operational by 2030, with additional deployments through 2035, collectively providing up to 500 megawatts of electricity. These reactors will be located in relevant service areas to supply clean electricity to Google data centers.

  • Amazon is investing in X-energy, a company specializing in high-temperature gas-cooled reactors (HTGRs). These reactors use helium as a coolant and are designed for enhanced safety and efficiency. Amazon's plans include developing SMRs to power its data centers, aiming to meet its sustainability goals.


Wrap up and hypothesis

As we have observed, the next wave of innovation and human progress is tightly coupled to electricity. While other approaches may develop (solar panels in space directed to earth, maybe?), the best and least intrusive solution that we can currently scale is nuclear power. I suppose the contrarian take is that renewables should be the answer, but I struggle to see it.

My hypothesis is that we see a resurgence of nuclear projects in 2025 (already proving true) and a large build out of SMRs that are in ubiquitous operation across the US by 2035. Investment dollars are and will continue to flow into companies developing these projects. The big questions are, will the regulatory landscape cooperate and how long will it take? Will there be major setbacks that destroy confidence in these new reactors, despite their enticing promises?

I am bullish on the companies that are building the next generation of nuclear reactors and excited about the positive benefits of low-cost electricity for the US (and global) economy. Like we said at the start: energy is one of the key components of producing things and the cheaper we can get it, the better. Driving the marginal cost of energy down will benefit everyone and create a better world for all of us. It is one of the keys to entering another “golden age”.

And if you end up with a Gen IV SMR reactor down the road from your house, you might not even care. Or know.


This post took a lot of time to research and write. I had to cut a lot of details and it’s likely imperfect on those I did include, but I hope it helped you learn something new or sent you searching for more information about something. Let me know what you thought of it.


Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #41]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-41https://brianchambers.substack.com/p/chamber-of-tech-secrets-41Fri, 03 Jan 2025 16:03:07 GMT<Squeaky door creaks> The Chamber 🏰 of Tech Secrets is open.

2024 flew by, and mostly without any action in the Chamber 🏰 of Tech Secrets. My apologies to my many faithful readers for that. I will try my best to be more consistent in 2025! This post is a reflection on 2024. I hope you find it useful and that it sends you down some rabbit 🐰 holes. Happy New Year!

AI Takeover

In addition to my failure to prioritize writing publicly in 2024, I struggled to find topics to write about as it often seemed that Generative AI dominated the landscape. A few brief thoughts on that…

  • AI seems promising, but it is still very early. A lot of the news has been related to developments in chips (NVDA +190%) and large infrastructure build outs by cloud providers. Most companies I know are investing in Generative AI and are optimistic about the future, but the work in 2024 was yet more infrastructure buildout (albeit internal — legal, compliance, privacy, data, talent, etc.). As those buildouts complete, will the value start flowing?

  • I’ve observed some degree of business value in co-pilots and add-ons to SaaS applications, improved chatbot experiences, or AI generated feedback summaries (Amazon), but the overall market still seems to be anticipatory. When will the “killer app” arrive? Or will there be a complete paradigm shift that comes (agents instead of apps?). That remains to be seen.

Regardless of where you stand on AI, at least one thing remains true: The need for business-critical SaaS applications, quality cloud infrastructure services, talented software engineers equipped with best practices, a good organizational architecture, a strong integration capability / API strategy, mutually beneficial partnerships, robust identity management and security services, and strong data management practices has not gone away. It has, perhaps, become even more critical. Will AI change any of this in the future? Maybe. But today, these things remain essential to an organization that wants to thrive in their applications of technology. And I suspect these same things will be necessary for organizations that want to maximize AI as it develops.

2024 Reflection

I like to do some reflection throughout and at the end of the year, so here are a few key items that may be useful / informative.

Key Highlights

2024 was a great year. Our little boy, Jackson, turned One the week of Christmas (12/22). It has been an amazing year to be his dad, to watch him grow physically, to watch him learn about how things work, and to enjoy all the unique little things about him and memories with him. This is the unquestionable highlight of the year.

We made some fun trips this year. In the summer we went to the PacNW to visit Mt. Rainier National Park, Hood River, Crater Lake, and the Oregon coast. In the fall we spent about 10 days in Jackson, WY, Grand Teton National Park, Yellowstone, and the Bitterroot Valley in MT. I love getting out west, especially in the fall.

Bonus: I did watch the first third or so of the Harry Potter movies with the kids (parts 1-3). And the Lord of the Rings series (it has been years).

Some favorites and takeaways

Favorite Podcast Episode: I listened to Alex Hormozi on Modern Wisdom while doing a solo 20-mile hike in Grand Teton National Park last fall. I actually started it a second time, and listened to it a third time later. I’ve become a big fan of Alex’s podcast The Game as well. I had so many takeaways from this episode, I won’t write them here. Perhaps that will be my next post, though.

Favorite Quote: From the above podcast episode, “The magic you’re looking for is in the thing you’ve been avoiding”. I’ve applied this in many areas, but the easiest example is in fitness. I had been avoiding carrying heavy sandbags over distance or holding them for extended times because, well… it sucks. Same for stretching / mobility work. It is easy to avoid “deeper work” at work (the kind that requires sitting and thinking vs just responding to urgent matters)… if that’s the thing you’ve avoiding, just go there. The magic is there.

Favorite Book: Comfort Crisis (listened to it on Audible 3 times)

Best Habit: I think I have a tie here between 11) Daily Journaling and 2) working out at lunch. I haven’t written publicly a lot, but I did write about my key priorities, my workouts, and my thoughts on many things in a Google Doc journal every day in 2024. It was fun to read back through them and remember moments from the year, and to be reminded of ideas that were beginning to develop (to see how they’ve grown) or to revisit them again. Working out at lunch breaks the day in half, which is awesome, and ensures I maintain a focus on my fitness goals.

Efforts I really enjoyed: Edge Monsters and Chief Architect Network

Getting out of comfort zone: I try and challenge myself or learn new things so I can maintain a beginner’s mindset often. This helps with staying humble, but also just keeps life interesting. This year I learned a lot about hunting… specifically that it can be HARD. I learned about climbing trees with climbing stands, using saddles to hunt, managing the wind, scouting for places to hunt, and more. Not to mention learning to shoot a bow effectively. I also walked around in the woods a lot and in some new terrain (north-central WY), which was awesome. So far, I haven’t succeeded but hopefully that will change before the end of January.

Most Interesting Tech Trend: The juxtaposition of the continued miniaturization of things (enabling powerful edge applications on small form factors with low energy usage) while also seeing the growth of supercomputing (entire racks with huge form factors and massive energy demands leading to big nuclear power investments).

Things I enjoyed building: 2024 saw the launch of the Chief Architect Network (CAN) and Edge Monsters, and I am thankful to be a co-founder in both. CAN is all about creating the world’s best network of Chief Architects and Enterprise Architecture leaders. Edge Monsters brings together the world’s best Edge Computing practitioners with production experience. As always, I love being the least-smart person in the room.


I could go on and on. 2024 was an exciting and busy year. Stick with me in 2025 as there is more to come. Future Chamber 🏰 of Tech Secrets topics are likely to include more thoughts around miniaturization and edge computing, managing technical debt, enterprise architecture, work-life balance, the resurgence of nuclear energy, and more. If there is something else you’d like me to write about, just let me know. I am open to ideas. Until next time…

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #40]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-40https://brianchambers.substack.com/p/chamber-of-tech-secrets-40Mon, 29 Jul 2024 09:31:01 GMTThe Chamber 🏰 of Tech Secrets is open. The summer is in full swing and I have been busy with all kinds of things: lots of work, fitness, and our first vacation involving air travel with 7-month old Jackson. We took a nice 12-day trip and made stops in Portland, Mt. Rainier National Park, Hood River, Bend, Crater Lake and Cape Kiwanda.

This week, we’ll explore one of my favorite topics: Principles. Instead of talking about principles that I like or use, we’ll explore how to write a great principle. I hope you find this useful at home and in your professional endeavors. Let’s jump in!


Anatomy of a Great Principle

I am a big advocate for principles. I shared some of my own in Chamber of Tech Secrets #24: Principles and wrote about our first version of Enterprise Architecture Principles for Chick-fil-A on our Chick-fil-A Tech Blog.

Principles are important because they have the potential to become an oral or written tradition that shapes the way people think: personally, as a family, or organizationally. When the foundation for thinking is right, creative and congruent ideas can emerge and lead to actions that are “right”.

Our first round of EA principles for Chick-fil-A were good and very useful, but not great. After applying them for three years and reflecting on them, I started to ask myself what made a great principle? In re-writing a new version, I have arrived at what I believe are some great tips for writing a good principle.

Let’s dig into some principles for writing great principles, if you will.

Great principles…

  • Are understandable and clear: This likely goes without saying, but it is worth wrestling over each and every word of a principle to ensure its understandable and clear to a reader. Think about the least common denominator in your base of potential readers and start with their level of understanding. Ask yourself “what behaviors do I want to emerge from this principle?” Build to your key point rapidly but make sure to connect to the relevant first principles, which takes us to our second point…

  • Start from first principles: Great principles appeal to first principles (from the relevant discipline or context), shared cultural contexts, or other universal truths.

  • Are short and quippy: A great principle needs to be easy to remember and recite. It takes lots of repetition for principles to embed in people’s brains and show up in how they think. If the principle is long or complex, it is easily forgotten and loses its power.

  • Focus on one idea: It is easy to start “principle jamming”; stuffing many different ideas into a single principle. Avoid this. The best principles convey one key message. This will be more compelling and easier for people to remember. You can still make the principle tangible and actionable, but ensure your top level idea is the key thing the reader takes away.

  • Are easily expoundable: A great principle is indeed short and quippy, but may need a “double click” to help the reader internalize the idea. The presence of this accompanying material makes the principle less open to interpretation (when needed) and ensure that principles…

  • Are deterministic: A great principle holds up to testing from many different potential scenarios and is reinforced by these examples, not weakened by them. When approaching a principle with a question of “how should I think?” or “how should I act?”, one should emerge with a clear directional answer.


Making it tangible

Let’s make this tangible with an example. To do so, I’ll share a preview of one of our new Enterprise Architecture principles.

Understandable and clear: No matter who you are, if you read this principle you will emerge thinking that you should “think enterprise” and that two things you can do to implement that principle tangibly are to create technical interfaces and consider an enterprise stakeholder base. Why do we have this principle? Sometimes we observe that people forget to think enterprise and get overly focused on their own product or their primary business partners at the potential expense of the success of the larger organization.

Start from first principles: First, we frame the context for the principle, which is about how we think about our technology systems. Then we appeal to the idea of 1) winning in the marketplace and 2) “winning hearts every day” (internal lingo for connecting with and serving people with honor, dignity, and respect).

Be quippy: “Think Enterprise” — this is short enough to remember and quippy enough to stick, and conveys the entire spirit of the principle.

Focus on one idea: This principle tells you to remember to “think enterprise” and explains why. If this phrase sticks in people’s minds and changes the way they think, we’ve “won”.

Be expoundable: We can easily expand the idea of thinking enterprise into some ideas that are easy to execute, specifically thinking about technical interfaces,

Be deterministic: Try testing this against your own example, particularly if it’s system-related. I am trying to do X. I am worried about Y and Z. Does the principle hold up? Let me know!


Conclusion

To do all this, you need to know what you want to say and what behaviors you want the principle to drive, whether in thinking or action. If you put these tips into practice, you can create great principles that you help you influence people by changing the way people think and act, or by memorializing ideas that are important to your organization. Go forth and principle!


Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #39]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-39https://brianchambers.substack.com/p/chamber-of-tech-secrets-39Mon, 08 Jul 2024 09:30:23 GMTThe Chamber 🏰 of Tech Secrets is open. This week I want to share some reflections on public speaking. I find it pretty natural and low-stress today, but that has not always been the case.


Public Speaking Woes

When I was in elementary school, I was extremely shy. My mom would introduce me to other adults and I would hide behind her. I wouldn’t speak much to most people, just my close friends and family. I dreaded talking in front of the class through school. I was often shy and socially awkward.

As I grew up and became more confident, these patterns faded some, but I remained aware of and nervous about the thoughts and expectations of others. Particularly dreaded was “public speaking”, generally performed in the context of oral reports in school. Anticipating these events filled me with a strange sense of dread—the stomach turning kind (iykyk). I tried to focus on getting everything right, not losing my train of thought, and not getting a “frog in my throat” that would interrupt my words. I experienced all of these “failures”, and more. I always survived, but emerged relieved and sweaty at the end, wondering what everyone else thought of me. This was even true in the earlier days of my working career in meetings and departmental presentations.

Over the past 8 years, I have done more public speaking engagements than I can count, most of which are technical or leadership oriented. People tell me that I do a great job, which always strikes me as humorous given the above history. If I can change…

Cracked.com on X: "Rocky Wisdom! 28 Movie And TV Quotes So Stupid They're  Sort Of Brilliant - https://t.co/o3GSATIZ3V https://t.co/vgJcd1ePFN" / X
Photo Credit: https://x.com/cracked/status/829877305358946305

Secrets of successful conference presentations

Here are a few things I think you should consider when doing public speaking engagements, particularly technical ones.

Make no assumptions about your audience: One thing I have observed is that a lot of people assume that others have the same context they do. This is wrong. People have come to your talk interested to learn, but if you assume too much and don’t create a shared context, you’ll notice them shift their attention to their phones. So how do you create that shared context?

Most of my talks have been about technology at Chick-fil-A… so I like to start by helping people understand our business and our challenges, and then explain why we wanted a technical solution like the one I am presenting. Hopefully, this means that the audience is clear on the WHY so we can turn our attention together to WHAT and HOW.

Start with a story: I used to start my presentations with an agenda and by introducing myself. The best advice I ever read about public presentations was to NEVER do that. It’s boring. And its a presenter-centric tactic. It puts you in a mindset of saying “I am the presenter. Here is who I am. Here is what I am going to teach you”. It is far better to start with a story that engages the audience and gets them thinking about how they might address the problems you experienced as they come on the journey with you. Yes — this is true even for the most technical of topics. The story can be short… but it is your chance to extend a hand to the audience and invite them to think about this problem with you, which is far more fun and engaging than the alternative.

Be the master of your content: This is the most obvious thing ever, but make sure you are the master of your content. If you know your topic in and out, there is really nothing that can surprise you or go wrong during your talk. You might not nail every point you intended, but nobody notices because they never saw your notes.

Present the content in the best way you possibly can and don’t worry about the expectations of the audience: After a lot of talk about connecting with the audience, remember that you are not presenting to impress anyone. The audience is not constantly judging your presentation, your delivery, how you look, how pretty your slides are. Think about your own experience. When you attend a talk, you are trying to learn something. If you are presenting and you make a mistake, nobody cares other than you.

It. Is. Ok.

Most people don’t remember most talks they listened to anyway. They will remember things they did that may have been a result of the talk, whether they can attribute it back or not. Don’t waste time trying to be flashy and impressive. Interesting may draw people in, but sharing experiences is what will make your talk great.

Be available: People often follow up after talks, whether in-person or on LinkedIn. Be as available as you can. A lot of the value creation for others, and you, will come from the relationships and conversations that come out of your presentation.

Use lots of slides: If you are nervous about losing your train of thought, use lots of slides. This approach works great for me. Some people do amazing presentations with no slides or just a few slides. I like to use lots of slides, but not many words. There are a few reasons for this approach…

  • When I have a lot of slides, the flow of the presentation takes care of itself. If I happen to lose my train of thought (which has happened), the slides are there to keep me moving down the right path. This takes away a lot of anxiety about getting lost or missing key points.

  • People like to engage many senses at once — I try and keep my slides to mostly pictures or diagrams so that they support what is being said, but don’t say what is being said. This gets the eyes and ears engaged and can stir an internal dialogue in the listener where they start to think about what is being said as well.

    Here is an example: The purpose of this slide is to explain our GitOps architecture for container deployments to thousands of edge locations. Without some narration, you may have some ideas of what we do… but its not entirely clear. You need to listen to what I am saying. Without the picture, this becomes a pretty complicated topic to explain.

  • If there is a key phrase you need to really nail, put the text on the slide. You can read it and remember it, and the audience will read it and perhaps remember it. One that I use regularly in the context of edge computing is “we should run out workloads as close to the user as necessary, but no closer”.

Demos: Are live demos useful? Reflecting on my live demos in the past, I think my motivation was to validate myself, to make people think what I did was cool, and to “wow” the audience. If that is your reason for a live demo, just skip it. The presentation is for the audience to learn, not for self-validation. If your demo is able to help people get the point of your talk or truly learn to do something themselves, have at it.

Bring a friend: Nervous? Don’t feel like you have mastery over certain components of the topic? Bring a friend. You can always co-present. Don’t transition back and forth too much, but two-presenter events can be really great since they bring two different ways of thinking and communicating to the content. If you are nervous that you will lose your train of thought or make a mistake, you have a backstop up there with you.

Anatomy of a great presentation

What makes a great talk? Reflecting on years of attending conferences presentations, I think the best talks have the following elements:

  • They tell and story and invite the listener to think along with them about the problem and how it might be solved.

  • They are clearly rooted in the experiences of the speaker (rich with stories).

  • They are honest and transparent — things that go poorly often make better talks than things that went perfectly.

  • They leave time for questions, which can often be the best part of a presentation since its a chance to really engage with the audience. “No questions” is usually a sign of a talk that could have gone better.

  • They are filled with humility — this is what we did and why and it may not work for everyone, but here’s why it made sense for us.

  • They don’t share too much and are actionable — we don’t need to know everything you’ve ever thought about. We just need something we can take away and act on.

If you are nervous about presenting publicly, I’d urge you to take some steps to overcome that fear. One of the best ways to become an active participant in the tech community is to contribute directly. Don’t be shy. Give it a try. 😀


Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #38]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-38https://brianchambers.substack.com/p/chamber-of-tech-secrets-38Mon, 10 Jun 2024 09:00:38 GMTThe Chamber 🏰 of Tech Secrets is open. Thanks to everyone who sent an email or made a comment here or on LinkedIn to say congrats about Jackson. I really appreciate the community that exists around the Chamber 🏰 of Tech Secrets… I always read, appreciate, and generally reply to everything you guys say. Thanks for being here! This week is not really about tech, but rather about what I have learned as I reflect on my 20 years at Chick-fil-A. I hope you find it beneficial.

Reflecting on 20 years at Chick-fil-A

During my college years, I worked at a local Chick-fil-A restaurant as a Team Leader, managing the front counter area, working the drive thru, cleaning the dining room, and restocking sauces. At that time, Chick-fil-A was still fairly regional. Most payments were cash as we did not yet take credit cards (but we did take personal checks). We still hand cupped coleslaw each morning. We did not sell milkshakes. There was no Chick-fil-A One mobile experience. I think I capped out my hourly earnings at $6/hour.

During college, I only interviewed with two companies: Accenture and Chick-fil-A. I got offers from both on the same day and told Chick-fil-A “yes”. I showed up a few months later fresh out of college with minimal real-world experience. The most useful skill I had was relational database modeling and SQL. I was far less mature than I thought I was at the time. I knew far less (about anything and everything) than I thought I did. I was nervous to share my thoughts, apprehensive around people with big titles, and diligently avoided confrontation. I was slow to speak in meetings. People used terms I didn’t understand and I would patiently wait for context to emerge naturally — sometimes over weeks — before figuring out what they were talking about. I remember the this distinctly with “cloud computing”, which I found very confusing at the time. 😜 Overall, I was simply hoping to do a good job, avoid getting noticed and hopefully not get exposed as someone who didn’t really belong.

As of Friday, June 7, I have been working at Chick-fil-A for 20 years. 😱

I literally cannot disconnect Chick-fil-A’s influence on my life from the person I am today.

Hes Literally A Ray Of Sunshine GIFs - Find & Share on GIPHY

The things I value, the way I think, how I work, what I know, what I do, the way I have grown… all have been highly influenced by this wonderful organization.

For fun, here are a few things I have worked on over the years:

  • third-level escalated support for all restaurant systems (in-store client server and web apps)

  • data replication from stores to corporate for supply chain orders, daily sales data, etc. using Sybase database replication technologies over dial-up internet connections

  • simple bug fixes and code changes in Delphi for our locally-hosted “ERP” for restaurants

  • developer productivity tools and code release automation (in like 2008)

  • lots of SQL in Oracle and Sybase databases for countless purposes

  • solution architecture using enterprise middleware tools like Oracle SOA, Mulesoft, etc.

  • implementing the user profile APIs for Chick-fil-A One in java for the first time

  • leading the move to AWS Cloud for the enterprise (along with others, I take no credit)

  • helping shift our organization to a DevOps / Product Team focused model

  • launching our data lake infrastructure

  • creating our IOT strategy and Edge Compute capabilities

  • leading the Enterprise Architecture practice and a fantastic team of wonderful people who try to make sure that technology helps us win

There are many, many more. A lot of the work is meaningful, but I am even more thankful that some of my best friends have come from working on some of these projects.

Reflections on 20 Years

I want to share a few reflections from my 20 years, both to celebrate and hopefully help you think about your own career.

  • Gratitude: I have to start with gratitude. The business I work for and the family that owns it are nothing short of outrageously generous. They are committed to slow and conservative growth so that we can be a company that doesn’t do layoffs. Ever hire is long-term. We share wonderful ideas like “value results and relationships” and “be the world’s most caring company” and we actually pursue them. Everyone that I have ever worked for has invested in me and cared for me as a person. To any of you who read this, you know who you are: Thank You! I would not be the person I am today without your influences on my life. So what action you take away from this? Two things:

    • Express your gratitude for people who invest in you to those people. They will appreciate it and it will make your relationship stronger. And its the least you can do. Do it early and often.

    • Become a person who invests in others such that they are going to think of you when they reflect on their growth and are grateful. Do it for them, not for you.

  • Purpose: Chick-fil-A has a Corporate Purpose. I see it on a plaque any time I enter our Support Center’s main building. I know many companies have the same. What I think is rare about Chick-fil-A is that the Corporate Purpose actually drives everything that we do. It reads: “to glorify God by being a faithful steward of all we have been entrusted with and to have a positive influence on all who come in contact with Chick-fil-A”.

    Business Strategy Case Studies of Chick-fil-A and Costco | by Joshua  Chandiramani | Medium
    Photo credit: Wall Street Journal

    Stewardship (thinking like an owner) and positive influence are two core things I can rally around. What action can you take? If you can, attach yourself to a place that operates by a purpose (I know they are rare) and make sure its one that resonates with you. It will make your work so much more meaningful and will enrich your life.

  • Play the long game: I almost left Chick-fil-A at one time over what seemed like a lot of money but was trivial in the long run. That would have been one of the biggest mistakes of my life. In those conversations, someone told me that I needed to “play the long game”. It was not what I wanted to hear at the time, but it was so true. I was thinking short-term. Another time, I remember looking around the department I work in and thinking “wow there are already leaders and potential future leaders ahead of me for all these area, I just don’t see where I will ever have any opportunity to move any further”. Short-term thinking. There have been countless opportunities since. I already wrote about the benefits of job-hopping vs staying long term, so I’ll stop here, but I definitely have learned that, at any given time, my perspective about what is possible in the long-term is far too narrow. “We overestimate what we can do in the short term, but underestimate what we can do long term” — I think this says it well. By playing the long-game, I can overcome my own limitations in vision for the future and allow time to work for me instead. Speaking of time…

  • Personal growth compounds: Each bit of personal growth you experience —whether intentional or absorbed via osmosis — is like adding a few dollars to your high-yield savings account (for simplicity of metaphor — obviously you would 401k match first, then max your ROTH, build an emergency fund, etc etc 😜). Without even knowing it, you will exercise that growth on a daily basis, resulting in a compounding effect. You’ll make intentional deposits too through challenging projects or personal development. A few more pennies in your account. Then you experience the benefits of time and compounding. When you look in your account after 5, 10, or 20 years you will find that you are far more “rich” than you ever expected was possible. This is how I feel and what I have experienced. I entered Chick-fil-A as a mere shadow of my current self, and I am not shy to say that I am pleased with who I have become because it honors those who made it possible (and most of those people are not me). What can you do with this? INVEST. IN. YOURSELF. Sometimes you will benefit from the culture around you. Other times you will have to devote some time to grow yourself and set yourself up for future opportunity. Start early because time is your friend. Deposit often because the larger the base, the greater the compounding effect.

  • It takes more than “tech skills”: I started my career with a few tech skills. I grew some more over time with different experiences and roles. Some were on the job and some were personal development investments on my own time (learning programming languages, cloud services, etc.). Most of the things I learned early on are irrelevant and obsolete now (Sybase database replication over dial up internet, anyone?). More important than each specific technical skill, I built what I think is a pretty solid foundation over time such that new things that come along mostly make sense to me and I can see how they would fit into our enterprise architecture. Tech changes fast and will keep changing fast so, if you want to be able to grow with your business, you need to focus not on mastering a single thing, but mastering how to add new things to your current things. This may not be perfect advice for everyone but I think it would be useful to many. Get curious and stay curious and don’t get lazy on learning new things. At some point though, knowing all the tech becomes not enough. So my advice is to make sure you 1) build organizational context so you can understand how things fit together 2) develop soft skills so you can be influential and build good relationships and 3) be okay to steadily plod towards “better” because change is hard and it takes time.

  • Everything else: I could go on and on about things I have learned but I would never hit them all because I have been shaped in so many positive ways slowly over time — like a pebble that gets smooth in a river — that attribution is impossible. I am confident saying that, if I had accepted an offer from Accenture, I would not be who I am today.

There are thousands of phrases running through my head — “here to serve”, “if you aren’t selling chicken, you better be helping someone who is”, “create raving fans”, and of course “my pleasure”.

It is easy to get frustrated with work for any number of reasons. I have my frustrations on a regular basis (often daily). Our former Chick-fil-A President retired last year and in his last talk, he said this about Chick-fil-A as a company: “Is it perfect? No way… But is it special? Absolutely”. I love this perspective and try to remember it every day. No place is perfect and we can always improve on many things… but the place I have spent the last 20 years is truly a special place filled with incredible people pursuing a meaningful mission and purpose, and that is more than I could have ever asked for and more than I deserve. I am grateful for 20 years, and looking forward to spending the rest of my career right here. 🙏

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #37]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-37https://brianchambers.substack.com/p/chamber-of-tech-secrets-37Thu, 06 Jun 2024 09:31:06 GMTThe Chamber 🏰 of Tech Secrets is open!

Extended Absence

After a prolonged absence, I suppose I should share where I’ve been and what I’ve been up to.

On December 22, 2023 my wife gave birth to our son, Jackson Brian Chambers. We have a 3-generation tradition in our family where the first boy has taken dad’s first name as middle name, so I am happy to continue that into the next generation. He came a little early, but thankfully were back home on the 23rd and enjoyed the Christmas and New Year holidays at home together. Jackson is healthy, happy, and sleeping great. I have been learning about life as a new dad and polishing my dad joke skills (obviously the most important thing). Beyond that, I’ve been keeping busy with active and outdoors activities like mountain bike rides with friends and family, trail runs and races, archery (3D targets with a compound bow), and of course CrossFit (who did Murph over Memorial Day weekend?). The year is off to a great start for me and I hope yours is too!

Turning to our topic for today… this post is a written version of a conversation I had on the Chief Architect Network LIVE podcast with my co-host, Grant Ecker. You can watch the entire conversation here and hear our dialogue and Grant’s four ideas.


Four things that could level up your leadership

If you are an Enterprise Architect, Chief Architect, or a technology leader of any sort, there are certain repeatable strategies and behaviors you can employ to take your leadership, influence, and success reaching outcomes to the next level. Driving change in an organization — while delivering cohesion instead of chaos — is a big challenge.

I am reluctant to use the word “should”… let’s consider these opportunities you could take advantage of. In my experience, they work when applied with consistency over time.

Think fast, think slow

Borrowing from the excellent book “Thinking, fast and slow” by Daniel Kahneman (recommend), the first thing to do is ensure you understand your thought model.

Kahneman breaks down two key systems that our brains use.

  • System 1: The fast, intuitive, and emotional system. It operates automatically and quickly, with little or no effort and no sense of voluntary control. It helps us make quick judgments and decisions but can be prone to biases and systematic errors.

  • System 2: The slow, deliberate, and logical system. It allocates attention to effortful mental activities that demand it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice, and concentration.

Thinking, Fast and Slow" by Daniel Kahneman: Book Summary, Review

As an Enterprise Architect (in particular), it is critical to be able to employ System 1 (hereafter “fast”) thinking. We often need to think on our feet, help people get unstuck, or make a judgement when asked to answer a question by another leader. I tend to find that mastery of one’s domain makes this system 1 thinking fairly easy for a lot of architecture leaders. Of course, we must remember that these thoughts are subject to our biases, gut feelings, and the like. I think of this type of thinking taking place in meetings, forums, and leadership presentations where things are quick and snappy. We use fast thinking all the time, and it’s a good thing. We need it.

Then there is System 2 (hereafter “slow”) thinking. This type of thinking requires carving out quiet time to think deeply. This slow thinking is where some very important things happen: strategies form, first-principles are considered, ideas are distilled to their simplest form, deliverables are crafted, presentations are rehearsed, feedback is developed.

Scrolling social media, news, LinkedIn has made it very challenging to be a slow thinker — our brains are changing and we seem to struggle more and more with things that take dedicated time. But hear me on this: you can’t be successful long term if you do not make time for slow thinking.

Brain 🧠 Tip #1: New to Chamber of Tech Secrets are Brain Tips. I often get my name - Brian - misspelled as “Brain” in solicitations, internal emails, and even once on a credit card! Let’s go with it and have some fun. This week’s brain tip is to fix this problem now before it starts to catch up with you. Go as many weeks into the future on your calendar as you must and find a time that you like for slow thinking. Block. Your. Calendar. You have my permission. Treat that time as sacred and don’t give it up for meetings or other easy-to-do instant gratification activities, ever if you can help it but at least not easily. I have what I call “heads down Wednesday” blocked from 12-5pm every week for the next 15 years.

Build Scalable, Shareable Mental Models

Enterprises have a lot going on across one to many businesses. Departments, teams, and other structures we create employ people who tend to focus on whatever outcomes they own and charge after them relentlessly. This is good — we want a world where teams can push the proverbial gas pedal and the car accelerates. We want a world where they can be decoupled from everyone else and focus on their work. At the time time, if a subset of teams are crushing it but ignoring everyone else around them completely, those other teams are likely to suffer. Then the company doesn’t win… and when the company doesn’t win, it doesn’t matter which sub-teams did.

We have a challenging goal as Enterprise Architects. We need to keep things loosely coupled, decentralized, and autonomous as much as possible (at least that’s my goal) AND we need to make sure that one team’s success doesn’t come at the expense of other teams and ultimately cause the company to lose.

Central to this is having a “shared context”, and we want to make sure that our teams have “just enough” of it. Just enough context means understanding the landscape around you so that you can act like a lego block in the larger organization while not having to understand all the things that make up the other blocks.

Building scalable, shareable mental models is how I attempt to navigate this challenge. These are things that others can “think on top of”. What is such a model? Let me give you two examples.

First is Business Capability Models, which are a Business Architecture technique for building shared context about what an organization does and how it is supported by people, process, technology, and data. In Business Architecture, we use industry standard models as much as possible and avoid making the model a copy of the organizational chart. The model reflects what things the company does, not how it is currently structured. We can use this model to layer in systems, people, vendors, financial spend, technical components, operational criticality tiering, risk considerations, and more. When we create these types of models we build shared context. On top of that, we allow others to extend our model for their needs. For example, we are currently discussing tracking our progress on each capability across the spectrum of Manual, System-enabled, Automated, AI-Enabled. Those may not be the end-state words we use, but you get the idea.

Second is one of my favorite things: Principles. Principles are hard to do well, but when you get them right they can be a powerful mechanism for creating a scalable and shareable mental model. Principles encapsulate things we want everyone to put at the bottom of their pyramid as they think about solving a business problem. Your culture, operating model, and many other factors will determine what you want to put in this wonderful layer, but I highly recommend having a strong set of principles that you can share in text form and conversationally. For the latter, you have to take time to internalize them… and I recommend doing that while you write them so you get it right. Remember to start from a least common denominator of organizational understanding, whatever that is.

I have written about our EA principles on the Chick-fil-A Tech Blog extensively if you want to learn more.

Brain 🧠 Tip #2: To share context, we need to have context… so this all starts with listening.

Write

Writing is a fantastic way to force yourself to crystalize your thoughts. Writing forces context-building, idea cohesion, and conciseness in communication. This is probably why some highly effective companies like Amazon have used it over slides in meetings to build appropriate context for decision making.

Writing publicly can take your thinking to the next level. If you can explain an idea to those who don’t have the same shared context as you, how much more effectively will you be able to communicate your ideas internally to people who share more context? How much better will you understand all of the intricacies of the topic if you explore it in writing, which forces you to face your own limitations in understanding?

I have written about writing before, so I’ll leave it at that.

I encourage every Chief Architect, Enterprise Architect and… well… human to start writing today. The benefits are incredible and the cost is simply making time (think System 2, slow thinking).

Brain 🧠 Tip #3: WRITE!

Be macro focused, not micro

It is easy to get focused on the little details of every project and if they are on or off the rails. There is a time and place for that, but as Enterprise Architects, we need to look at the bigger picture. Is the organization developing its capabilities in the right areas to succeed and win in the marketplace? Are we researching and exploring new and emerging technologies that can help our business? Are we making the right investments? Do we have good architecture? Not perfect architecture… that will never happen. Are we winning?

Focus on these things and you’ll likely be more effective, less stressed, and happier.


Thanks for reading. I hope the 6+ month wait between posts was worth it. Have a great week!

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>
<![CDATA[Chamber 🏰 of Tech Secrets #36]]>https://brianchambers.substack.com/p/chamber-of-tech-secrets-36https://brianchambers.substack.com/p/chamber-of-tech-secrets-36Mon, 11 Dec 2023 10:00:36 GMTThe Chamber 🏰 of Tech Secrets is open.


Any year that passes in which you don’t destroy one of your best-loved ideas is a wasted year. - Charlie Munger

Charles Munger
Photo credit: https://www.forbes.com/profile/charles-munger/?sh=60a3ab04697a

One of my favorite experiences is watching things that I designed or built be retired and replaced by something new and different, led by someone new and different.

This generally means that things continue to progress and improve in our organization, getting simpler and better over time. As a bonus, it means I am no longer responsible for those things. 😁


  1. The best line of code I have ever written is the one that was most recently deleted and replaced.

  2. The best architecture I have designed is one that is actively evolving into something new and better.

  3. The best leadership I have provided is giving others the freedom to destroy the things that got me to where I am.

These ideas only become tragic when nothing replaces my work, which means I worked on something that did not truly matter.


A few reflections on this idea…

  • There is always room for any idea, system, or architecture to improve and evolve to become better. Even if it is already simple and “just works”.

  • Being a stepping stone on the endless path of improvement is a high honor.

  • The hard break from one of your “best loved ideas” is destructive to the ego, which is good for the soul. This is particularly true if you have an identity (even just in work) that is associated with your discreet successes.

  • Breaking up with your “best loved ideas” creates margin for your next great idea. When we destroy and demolish, we make room for the new and better.

  • In my experience, successes are often bred of past failures and learnings. As things that were mine become someone else’s, give these new creators the grace I wanted as I iterated through the failure-to-success journey.


Which of your best-loved ideas have you let go of or destroyed this year? In that newfound margin, what is your next great and best-loved idea?

RIP Charlie. ❤️

Thanks for reading Chamber 🏰 of Tech Secrets! Subscribe for free to receive new posts and support my work.

]]>